Consul Results amazing automatically growing and shrinking Consul clusters health checks to prevent unhealthy secondaries from acquiring locks
Consul until, we ran into massive swap allocation. 40 GB swap allocation. fine for prototypes, not for production.
Results from Consul HA PostgreSQL is possible but, we need a tool which uses our resources more wisely.
Switch to etcd because of what we’d learned in Consul, the switch to etcd took a day to have a working sample
Modern etcd diagram Start Start-up Process Connect to wait 5 etcd? seconds no yes Is data Win race to set directory yes initialization Initialize yes empty? key? database Take over no lead TTL key Leader wait 5 Start owns key? seconds PostgreSQL as a leaderless Start yes no Secondary Postgres pg_basebackup from leader Start Postgres Do I own leader Running Loop key? no yes Is leader key Update owned? leader TTL lock yes no Am I following Am I the the correct healthiest leader? member? Am I the no leader? yes follow proper leader Acquire leader lock? no no yes yes yes Promote to leader Wait 30 seconds