IBM z/VM ("VM") is the most "fun" operating system running on modern mainframes if you are a programmer/expert-user type of computer person. Although senior to Unix, it has been kept up-to-date and is a fully modern operating system with Java and TCP/IP which grew out of something much like a PROM monitor.
VM (or CP-66/67 at the time) was the invention of an approximately 20-person conspiracy within IBM to write a programmer's dream operating system for mainframes. They told senior management that they were writing a debugger for a management-designed OS that was going down the tubes at the time because it was awful and didn't work.
VM did in fact debug the target OS, because VM means Virtual Machines. VM divides up a mainframe into CPU-supported isolated virtual machines. Each one "thinks" it's the entire mainframe, and each one itself can likewise divide its own virtual mainframe into any number of virtual mainframes. A virtual machine can launch a different mainframe operating system in any one or more of its own virtual machines, so debugging other mainframe OSes is easy, since the child OS can't crash the parent who launched it.
Yes, VM debugged the management OS, which promptly sank with barely a trace (you never heard of it). VM itself was introduced commercially in 1972. A rather scathing witicism within IBM circles is that "VM has outlived many of the executives that tried to kill it." A more complete history of VM is found on Melinda Varian's Home Page, http://pucc.princeton.edu:80/~melinda/
VM was and is used run multiple mainframe operating system instances actively in production for thousands of users on a single machine. VM underlies the port of Linux to the mainframe. You may already be using VM if that web server you love turns out to be Apache on GNU Linux on VM.