The Question
I saw a question pop up in a work chat recently.
We want to automate scheduling within Cloud Build. Our goal is to tag releases and have them deploy according to a schedule. Cloud Build’s current configuration triggers an immediate deployment—this is highly problematic and not what we want.
This is an interesting question. I prefer to ship deployments as soon as possible; however, I can see why people would prefer to queue them. Maybe you’d prefer to ship updates during the night when your users are asleep? You do you. But if I were to schedule a release, I know when I’d want to.
Friday evening after I’ve logged off for the weekend.
So how do I make sure my application is deployed at 6 PM on a Friday and then turn on my Out of Office settings in Gmail afterwards so I won’t be disrupted if something breaks?
The Tech Stack
Cloud Deploy
While the original question was about Cloud Build’s immediacy, the immediacy may not be the issue per se. You probably do want to run tests, build the container, etc. immediately. It’s the orchestration to actually deploy the application that we want to control. For that, we’ll use Google’s Continuous Deployment (CD) tool.
Workflows
I like Cloud Deploy but it doesn’t do everything, a common problem with any service. Sometimes you need to write code to solve a very specific challenge but you don’t want to maintain the code long term. Workflows lets us declare our logic in YAML and Google can manage the underlying code for us.
Cloud Functions
But if I do need to write some code, in this case I’ll pick the solution that’ll continously build and maintain my environment. I’ll need to monitor my packages for security compromises but we’ll try to minimize our dependencies here.
Domain-Wide Delegation
This isn’t a specific feature but I was really surprised that this wasn’t documented clearly. Thank you Johannes Passing for blogging about this previously. It was the clearest description I found for how to do this without service account keys.
Getting the next deployment time
We’ll start with a Cloud Function. There’s an incredibly convenient package called cronexpr that will return the next timestamp that satisfies the cron expression. In this case, I live in Vancouver, Canada so I’m timing this for 6 PM on a Friday for me. Or 1 AM UTC on a Saturday.
I’ll create a Cloud Function with the following code:
function.go
package p
import (
"log"
"net/http"
"time"
"github.com/aptible/supercronic/cronexpr"
)
func NextRelease(w http.ResponseWriter, r *http.Request) {
cron, err := cronexpr.Parse("0 1 * * 7")
if err != nil {
log.Default().Printf("Error parsing cron expression: %s", err)
w.Write([]byte(time.Now().Format(time.RFC3339)))
}
w.Write([]byte(cron.Next(time.Now()).Format(time.RFC3339)))
}go.mod
module example.com/cloudfunction
require github.com/aptible/supercronic v0.2.29Now if I run the following, I get the next time stamp at the time of writing.
curl -X GET https://us-central1-PROJECT.cloudfunctions.net/next-deploy-window
---
2024-04-07T01:0:00ZSetting up Cloud Deploy
Cloud Deploy has four components that matter to us.
- A delivery pipeline consisting of one or more targets, which reference our Skaffold profiles.
- A target, which requires approval before a rollout can occur.
- A Knative spec that defines our Cloud Run service.
- A Skaffold file with a profile mapped to our Knative spec.
clouddeploy.yaml
apiVersion: deploy.cloud.google.com/v1
kind: DeliveryPipeline
metadata:
name: my-run-demo-app-1
description: main application pipeline
serialPipeline:
stages:
- targetId: deploy-prod
profiles: [prod]
---
apiVersion: deploy.cloud.google.com/v1
kind: Target
metadata:
name: deploy-prod
run:
location: projects/PROJECT/locations/us-central1
requireApproval: truegcloud deploy apply --file=clouddeploy.yaml --region=us-central1 --project=PROJECTFor Skaffold, we’ll go into our application’s repository and create two files.
resources/cloud-run.yaml
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
name: my-important-app-prod
spec:
template:
spec:
containers:
- image: appskaffold.yaml
apiVersion: skaffold/v4beta2
kind: Config
profiles:
- name: prod
manifests:
rawYaml:
- resources/cloud-run.yaml
deploy:
cloudrun: {}And then we’ll update our Cloud Build pipelines’ build trigger to call Cloud Deploy at the end.
steps:
- name: 'docker'
args: [ 'buildx', 'create', '--name', 'mybuilder', '--use' ]
- name: 'docker'
args: [ 'buildx', 'build', '--platform', 'linux/amd64', '-t', '$LOCATION-docker.pkg.dev/$PROJECT_ID/app/my-super-app:$SHORT_SHA', '--push', '.' ]
- name: 'gcr.io/cloud-builders/gcloud'
args: ['deploy', 'releases', 'create', 'commit-release-$SHORT_SHA', '--project', '$PROJECT_ID', '--region', '$LOCATION', '--delivery-pipeline', 'my-run-demo-app-1', '--images', 'app=$LOCATION-docker.pkg.dev/$PROJECT_ID/my-super-app:$SHORT_SHA' ]Domain Wide Delegation
An integral step in this process is turning on my Vacation Settings in Gmail. I wanted a way for a GCP Service Account to edit my Gmail Account. And I don’t want a prompt to authorize this application to view or edit my account settings.
We can authorize GCP Service Accounts to perform this task through Domain Wide Delegation.
We’ll create two service accounts. One for GCP workflow and one that’ll be used for Domain Wide Delegation. The former will be able to sign JWT tokens with the private key of the latter. We’ll want the OAuth2 Client Id of the latter.
gcloud iam service-accounts create workflow-account
gcloud iam service-accounts create update-gmail
gcloud iam service-accounts add-iam-policy-binding update-gmail@PROJECT.iam.gserviceaccount.com --member serviceAccount:workflow-account@PROJECT.iam.gserviceaccount.com --role roles/iam.serviceAccountTokenCreator
gcloud iam service-accounts describe update-gmail@PROJECT.iam.gserviceaccount.com --format="value(oauth2ClientId)"Write down that ID.
We’ll go to our Google Workspace Domain-Wide Delegation page and we’ll add a new client ID that’s authorized to call the https://www.googleapis.com/auth/gmail.settings.basic scope.
This setting may take up to 24 hours to propagate.
Our Workflow
This workflow is how we’ll wrap the various steps.
Before we dive into the workflow steps, let’s cover what it does.
- Gets the next deployment time.
- Sleeps until workflow until then.
- Approves a Cloud Deploy rollout to advance.
- Setups a JWT claim that’ll be used to create an access token for my Gmail account.
- Signs the JWT.
- Requests an access token.
- Uses that access token to interact with the Gmail REST API.
Of note, GCP Workflows won’t charge us based on how long we sleep for. We don’t need to continuously poll an endpoint, it’ll wait without incurring additional charges until the next steps are ready to occur.
Steps four to six are the most important, in my opinion. Oftentimes, the documented steps use self managed private keys. We don’t want to manage or secure these keys ourselves. Instead, we’ll use the signJwt API to use a service account’s system-managed private key. Let Google manage a service account’s private key for you!
main:
params: [args]
steps:
- get_next_deploy_time:
call: http.get
args:
url: https://us-central1-PROJECT.cloudfunctions.net/next-deploy-window
result: deploy_time_response
- wait_until_deploy:
call: sys.sleep_until
args:
time: ${deploy_time_response.body}
- next_step:
call: http.post
args:
url: ${"https://clouddeploy.googleapis.com/v1/projects/PROJECT/locations/us-central1/deliveryPipelines/"+args.deliverypipeline+"/releases/"+args.release+"/rollouts/"+args.rollout+":approve"}
auth:
type: OAuth2
body:
approved: True
result: rollout_resp
- set_jwt:
assign:
- jwt_claim:
iss: "1111111111111111111"
scope: https://www.googleapis.com/auth/gmail.settings.basic
aud: https://oauth2.googleapis.com/token
exp: ${sys.now() + 60}
iat: ${sys.now()}
sub: taylor@taylorstacey.ca
- prep_jwt_payload:
call: json.encode_to_string
args:
data: ${jwt_claim}
result: jwt_claim_string
- sign_jwt:
call: http.post
args:
url: https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/update-gmail@PROJECT.iam.gserviceaccount.com:signJwt
body:
payload: ${jwt_claim_string}
auth:
type: OAuth2
result: signed_jwt
- get_access_token:
call: http.post
args:
url: https://oauth2.googleapis.com/token
body:
grant_type: urn:ietf:params:oauth:grant-type:jwt-bearer
assertion: ${signed_jwt.body.signedJwt}
result: token_resp
- update_vacation:
call: http.put
args:
url: https://gmail.googleapis.com/gmail/v1/users/taylor@taylorstacey.ca/settings/vacation
body:
enableAutoReply: True
responseSubject: Out of Office
responseBodyPlainText: Is production down? Yikes but not my problem!
headers:
Authorization: ${"Bearer " + token_resp.body.access_token}
result: vacation_settings
- success:
call: sys.log
args:
text: SuccessEventarc
We’ll want our Workflow to run whenever a rollout is created.
gcloud eventarc triggers create on-rollout \
--location=us-central1 \
--service-account=1111111111-compute@developer.gserviceaccount.com \
--destination-workflow=deploy-on-friday \
--destination-workflow-location=us-central1 \
--event-filters="type=google.cloud.deploy.rollout.v1.created"If all is configured correctly, our Workflow will kick off whenever our Cloud Build job is run.
Summary
This example is cheeky but flexible. Change the cron schedule for nightly or weekend deployments. Use the JWT signing steps to sign JWTs without private service account keys. Incorporate other Workspace APIs (maybe Google Chat messages?).
Any questions? Hit me up!