Fun with Deployments and Zero Downtime (ZDT)

Background concepts:

1. To deploy ZDT: upload_capi_release && ~/workspace/capi-release/scripts/deploy -o ~/workspace/capi-ci/cf-deployment-operations/temporary/add-deployment-updater.yml

   Process 'cc_deployment_updater' running . -- lives on the scheduler vm

2. It's invoked from /var/vcap/jobs/cc_deployment_updater/bin/cc_deployment_updater by running rake deployment_updater:start and this starts a loop that sleeps for cc.deployment_updater.update_frequency_in_seconds (default 5) and then runs one orchestration step on all active deployments (e.g. ones in state DEPLOYING).

To make ZDT happen:

1. First, cf v3-push APP && cf v3-scale APP -i N an app (like APP=dora and N maybe 4 or 5)

2. Modify the code in a simple way

3. Stage a new package. (These commands come from How to Create an App Using V3 of the CC API):

   CF_API_ENDPOINT=$(cf api | grep -i "api endpoint" | awk '{print $3}')
   
   APP_GUID=$(cf v3-app dora --guid  2>/dev/null)
   
   PACKAGE_GUID=$(cf curl /v3/packages -X POST -d "$(printf '{"type":"bits", "relationships": {"app": {"data": {"guid": "%s"}}}}' "$APP_GUID")" | tee /dev/tty | jq -r .guid)
   
   zip -r my-app-v2.zip * # ( -x *.zip if there are old zip files hanging around)
   
   curl -k "$CF_API_ENDPOINT/v3/packages/$PACKAGE_GUID/upload" -F bits=@"my-app-v2.zip" -H "Authorization: $(cf oauth-token | grep bearer)"
   
   rm my-app-v2.zip  # clean up after yaself
   
   BUILD_GUID=$(cf curl /v3/builds -X POST -d "$(printf '{ "package": { "guid": "%s" }}' "$PACKAGE_GUID")" | tee /dev/tty | jq -r .guid)
   
   watch cf curl /v3/builds/$BUILD_GUID
   
   

4. Get the new package's droplet and associate it with the app:

   At some point the build should be STAGED. If it's FAILED or still staging after an unreasonable amount of time, you have a problem more severe than trying to ZDT-update an app. Otherwise, stop watch and continue:

   DROPLET_GUID=$(cf curl /v3/builds/$BUILD_GUID | jq -r '.droplet.guid')
   
   cf curl /v3/apps/$APP_GUID/relationships/current_droplet -X PATCH -d "$(printf '{"data": {"guid": "%s"}}' "$DROPLET_GUID")"
   
   

5. Finally, create a deployment for the app with the new droplet:

   cf curl /v3/deployments -d "$(printf '{ "relationships":{ "app": { "data": { "guid": "%s" }}}' $APP_GUID)"
   
   

6. Repeatedly hit the app and verify that the output contains an ever-increasing proportion of the new output until the old output completely disappears and the webish process instances increase and then drop away so you're back to N web instances.

   for x in {1..10000} ; do curl "dora.${CF_API_ENDPOINT#*.}"; sleep 0.5; done
   
   

Verifying the v2 app doesn't blik

1. Add a sleep 60 to the updater with log statements

2. Tail logs in one window

3. When prompted, run commands like

   `cf curl /v2/apps/[APP_GUID]