Operating Systems: Personal histories • Operating systems innovation is as old as computing, and has enjoyed many distinct eras • During each era, there have been those who have pronounced operating systems finished — often with the bromide “commoditized” • Over our careers (20 years and 5 years, respectively), we have seen OS innovation continue unabated • To understand our perspective on the future, it is worth exploring our perspectives on the past... • We are (of course) victims of our own biases; these histories must be viewed as personal, not authoritative
1993-1996: The Architectural Wilderness • By 1993, the Unix wars of the 1980s were broadly over, but the systems that survived were proprietary — and the Unix abstractions themselves were thought dated • Such as it existed, operating systems innovation was thought to be had not in the implementation, but rather in the programmatic abstractions • An era ruled by experimentation (e.g., object oriented operating systems like Spring) and debate (monolithic kernels vs. microkernels; microkernels vs. exokernels) • In the end, Microsoft NT was going to vanquish all...
1997-2000: The Unix Renaissance • The internet boom of the late 1990s brought with it a Unix renaissance: the internet and Unix had grown up together, and were culturally intertwined • Unix dominated the symmetric multiprocessing systems that were used for scale-up workloads, belying the old belief that operating systems “couldn’t scale” • When the price/performance of x86 displaced RISC, focus began to shift from proprietary Unix vendors to the open source Unix variants: Linux and the BSDs
2001-2005: A New Golden Age • After the nuclear winter of 2001, open source systems like Linux become economically vital — and these systems focused on catching up to proprietary kin • Meanwhile, Sun’s critical mass of operating systems talent yielded a remarkable set of OS technologies, e.g. ZFS, DTrace, Zones, SMF, etc. • Some of these had antecedents in other systems (e.g., FreeBSD jails, IBM System z fault management), but the state of the art was advanced considerably • In 2005, these technologies were open sourced; over the next five years, many were ported to (or aped by) other systems
2006-2009: Ascendancy of Virtualization • Prior to VMware’s work in the early 2000s, it was commonly thought that x86 “couldn’t be” virtualized • Thanks to VMware’s innovative approach to the problem, hardware virtualization became mainstream with workload consolidation (i.e., Windows) • Microprocessor support (AMD SVM, Intel VMX) + open source technologies (KVM, Xen) accelerated and broadened this trend to “real” workloads • Where hypervisors were once lean, they began to need the full functionality of operating systems — creating a new crop of tough, interesting problems
2010-2013: Elastic Infrastructure • The rise of virtualization allowed for the realization of a vision that dated to the late 1950s: centralized, wide- area, shared-infrastructure computing • Known as utility grid cloud computing, the rise of elastic infrastructure has effected an economic revolution as well as a technical one • Cloud computing is a broad-based systems revolution, including programming languages, databases, and (of course) distributed systems • All create fresh problems for the operating system!
2014 & beyond: The Container Revolution • As applications and services have grown more complicated, packaging + configuration management have become increasingly burdensome • At the same time, the performance and cost penalties of hardware virtualization are increasingly clear • Lighter weight operating system-based virtualization — coupled with image management — allows for containers to become the unit of software deployment • Enthusiasm for Docker shows that the time for this idea has come: ease of platform-as-a-service coupled with the flexibility of infrastructure-as-a-service
Joyent and OS innovation • Joyent is a San Francisco-based cloud computing company that pioneered OS-based virtualization in the cloud (circa 2005) • We’re not afraid to innovate: we have our own operating system (SmartOS) and our own runtime (node.js) • Upon these, we have built distributed systems for elastic compute (SmartDataCenter) and big data (Manta) • OS innovation is in our marrow: we seek to build a best- of-all-worlds system to help solve tomorrow’s problems • And did we mention that we’re hiring? ;)
Operating systems: The road ahead • The operating system remains our most elemental software abstraction — and a great opportunity for attacking higher-level problems • And where abstractions become sedimented there is (paradoxically) more room for innovation, not less! • We believe that new problems and technologies will continue to open up new opportunities for OS innovation • And in operating systems, an idea is only as robust as the implementation — it is no mere detail • There remains much work to be done; the primacy of OS innovation continues!