Thanks a lot for sharing us such a GREAT Open- Source Pipeline-based CI!
Finally we have a CI supporting pipelines as first-class citizens!!
That’s all. Good-bye!
Is Concourse a production- ready pipeline-based CI for a startup/small team?
Production readiness for CI Something that: * Just works™ * Costs as low as possible * Is maintainable
Just works! * `vagrant up concourse/lite` to bootstrap * `vagrant box update && vagrant destroy -f && vagrant up` to update * Deployment w/ standalone binaries just works. The deployment will be more manual than using BOSH (Note: It works as far as you use Ubuntu trusty or later with updated kernel as described in Concourse’s and Docker’s official documentation) * A lot of builtin/community resources_types * Concourse CI’s concourse pipeline is a greatest example
Costs as low as possible? * Wasting money for running idling CI workers when e.g. you and your colleagues drinking beer is unacceptable * Scale out my CI workers only when someone in your team is working. * Scalable/Elastic Concourse * Can be done on my side
Maintainability for a small team? * I am the only one available to introduce Concourse to my organization * No one (including me) in my org has experiences in CloudFoundry, BOSH * BOSH/CloudFoundry/Pivotal all looks GREAT * Have no time learning it (I wish I got paid to learn it) * Need Concourse today, but it comes with BOSH
Recap: Issues? > Last one-mile to production- readiness for us = Low cost and BOSH
Production-ready pipeline-based CI for me = AWS + Auto-scaling + Concourse = concourse-aws
1. Build AMI (Based on Ubuntu trusty, Docker/ Concourse baked) 2. Provision Concourse cluster providing configuration params interactively
Features * (Almost) one-command to bootstrap Concourse CI on AWS * Supports both time-based and utilization-based auto-scaling (e.g. zero workers in after-hours, 80+ CPU usage for several minutes to trigger a scale-out) * Multi-AZ (Single AZ failure doesn’t bring down the whole cluster. Running jobs will fail though)
Implementation * Terraform for provisioning AWS resources * Some bash scripts to wrap terraform calls * Some golang apps to interactively generate configuration files and passes it to terraform
TODO * CoreOS support(for faster, easier deployment, customization, maintainability, and for fun)? * Automatic VPC creation?(Do you need a separate VPC for Concourse?) * Separate terraform template for DB(Postgres)? * CloudFormation?
I wish Concourse had… Any official(or documented) way to: * gracefully stop workers when an EC2 instance is shutting-down or ASG is going to stop it * Currently we have to utilize AutoScaling Lifecycle Hooks + SQS + `fly pause-pipeline <EVERY PIPELINE>` + `fly builds | grep started | wc -l` to gracefully stop Concourse without killing running jobs
I wish Concourse had… Any official(or documented) way to: * cache pulled docker images to reduce build times immediately after scaling out * Docker’s “local registry as a pull-through cache” is good for DockerHub. Can’t be used with AWS ECR or Quay.io though
Use mumoshu/concourse-aws to bootstrap a production-ready Concourse cluster w/o learning BOSH You can try it today!