CoreOS + Digitial Ocean Meetup 9/8/14 Darren Shepherd STAMPEDE.IO
About Me – Darren Shepherd @ibuildthecloud, darren0 Fancy Title Engineer @ Citrix Previously Cloud Architect @ GoDaddy 3 yrs Building IaaS systems for past 5 yrs Apache CloudStack – Committer OpenStack 2 from scratch proprietary clouds Stampede.io is the 4th orchestration platform I’ve built
What is Stampede.io Hybrid IaaS/Docker orchestration platform Can run both VMs and Containers in a consistent fashion Share approaches where makes sense, but stil respects that Containers != VMs Easy to install/upgrade and use Tailored installation for CoreOS Cattle.io is the more raw framework under the hood
Why you’d build this?
Boring Reason Personal R&D project @ Citrix Just me, my opinions, and a MacBook Pro (running Linux) Loose upfront goals Make IaaS somehow better ○ Apply 5 years of lessons learned Include Docker/Containers ○ Shiny new tech is fun Spent 6 months locked in a closet Hacking – ~50k lines of code Staring at wall – “How do I think containers can be useful?”
My home office is a closet
UI Developed by Vincent Fiduccia (vincent99 on Github) Node.js and Ember Event driven using WebSockets
Either changing the world or world domination are expected outcomes Grandiose Reason
Container Playbook Containers are portable Containers can run on a laptop, in a VM, or on Bare metal Bare metal becomes more attractive Faster and cheaper Bare metal + Container == world domination Hypervisors, Virtualization, VMware, AWS, etc are doomed
But there’s a problem Containers are portable Containers are a compute technology Storage and Networking are not portable EBS – Reliable storage w/ snapshots VPC – IP addressing, firewall, L2
Nothing we can’t Architect around Ephemeral apps Distributed storage NoSQL - Cassandra Architect for the cloud!
Darren’s brief history of EC2 2006 – Ephemeral only VMs 2008 – EBS 2009 – VPC 2010-2014 – Crazy exponential growth
EBS and VPC are essential You can architect for the cloud But many won’t Amazon didn’t convince everyone to re-architect for the cloud Amazon supported legacy architectures Containers are currently EC2 2006 Most people wil continue to run on the infrastructure they already have Not a game changer
Containers won’t change the infrastructure world unless we tackle storage and networking
The theory Container are portable because Linux is ubiquitous Great idea, why didn’t I think of that We can build a portable EBS and VPC with just Linux Linux has the majority of the technology needed, we just need to piece it together.
Stampede is about building a portable cloud – compute, storage, and networking
What does this mean? Infrastructure providers only need to provide Linux Simple block storage L3 connectivity If VT-x/SVM is available, VMs can be launched Stampede provides everything else Stampede can be provided “as a Service” Normalize the infrastructure market Massive scale infrastructure provider is not needed
Fun Launching Containers on Digital Ocean
Default Stampede Deployment CoreO CoreO CoreO S Node S Node S Node 1 2 3 Controller Agent Agent Agent Libvirt Libvirt
One node becomes Libvirt controller
Logical Components in Controller
Deployed in single process for Controller simplicity Database
Should scale for Lock Manager clouds <50 servers Event Bus API Server Process Server Agent Server
Fully Distributed Controller
Stampede on Stampede
Digital Ocean Deployment 3 Etcd clusters w/ Fleet Management Stack – 3 x 12GB SFO1 Nodes – 100 x 2GB NYC3 Nodes – 100 x 2GB
Tests Sustained rate 6-7 containers per second Not tuned for raw throughput (naively stupid scheduler) Gigantic batches 10000 containers in one API call (count=10000) Restart everything during deployments Agents, Servers, Database, ZooKeeper, etc Just tried to break things
How Many Containers?
A lot – 127,884 127,884 Running Containers ~600 per VM 17 failures (0.01%) ~6 hours Was not testing for throughput Focus on reliability Note: This was done with unmanaged networking