Impending Dooooom WE COULDN’T SCALE THE STATUS QUO 5 stable apps over several years, with little churn Dev teams are doing their own ops but can’t have access to everything Everyone has access to all DB secrets Deployment tooling is all hand-crafted (with a bird on it)
MOAR Challenge ORGANIZATIONAL CHANGES We’re doubling developer count every year SOA: Adding up to 30 services in next 12 months Early part of the year we launched major product: Insights Increasingly heterogeneous apps
The Simplest Thing THAT COULD POSSIBLY WORK PaaS would be great, but doesn’t fix our immediate need Solve deployment problem first, then logging, then discovery and dynamic configuration Start with standard web apps, not high volume data apps: do the easiest things first
What We Did AND WHY Separate builds from deploys Hand built docker files at first, then automated configuration Deployment via in-house tool (Centurion) Health check end point, configs via ENV vars
Lowering The Bar AS IN HIGH JUMP NOT AS IN LIMBO Builds based on standard base layers Tool for automatic docker files (Shipright) Retained Dockerfile flexibility
Centurion CONTROLLED DEPLOYMENTS WITH DOCKER Rake application with tasks and dependencies Talk to a fleet of Docker servers at once Does rolling deployment Guarantees run environment Extensible
Productionize That Thing MONITORING OH-MY Monitoring (check_docker) Nagios New Relic
Operating the Machine WHAT WE DO NOW Configuration is in git (etcd soon) Builds and deployments via jenkins Debugging via ssh, Papertrail. (log routing soon) Developers update dependencies at will Test, staging, integration and production are identical
Developers Love It AND WE WERE SURPRISED? We expected resistance, instead we got enthusiasm. Telling developers to pull request against your puppet repo is a world of hurt. At a recent SOA kickoﬀ devs were able to get things up an running in hours instead of a week.
A bright, shining future WHERE TO FROM HERE? Centralized configuration via etcd Discovery through etcd, too Dynamic allocation of hosts by Centurion Dynamic Nagios, load balancing configs Containerized high volume apps and databases.