I have finally built my own Kubernetes bare-metal playground and I'm ready to dive into some nice use cases. If you're curious about how I set up my environment you can check previous articles from this series.
The first issue I wish to tackle is the CI/CD pipeline for a monorepo hosted on Github.
Disclaimer: I will be focusing in this initial phase more on the happy flow and won't pay too much attention on error cases.
What do I wish to accomplish in this first iteration?
By the end of this article, I want to successfully release and deploy a new version for 2 Golang services in my k8s staging environment upon successful pull request merge to the main branch.
Short description of the workflow I envision:
Each service in the monorepo must have:
- Dockerfile -> builds & runs the app
- Makefile -> manage versioning, trigger docker build, push image to DockerHub & push helm chart to Chartmuseum
- Helm chart -> used by Flux to deploy to k8s
Upon pull request approval a github action will trigger a bash script that identifies and runs the Makefile build phase of the changed service.
Upon pull request merge to main branch a github action will trigger a bash script that identifies and runs the Makefile release phase of the changed service.
Let's breakdown the work:
- Create 2 services with Dockerfile, Makefile, version file and helm chart
- Git action for build phase
- Git action for deploy phase
1 - Create the services
I have created 2 simple Golang ping and pong services.
With Dockerfile
FROM golang:1.18-alpine as build
WORKDIR /app
# file context should be root of repository
COPY golang/go.mod ./
COPY golang/go.sum ./
RUN go mod download
COPY golang/ping-service/ ./
RUN go build -o ping-service
FROM alpine:latest
WORKDIR /svc
COPY --from=build /app .
CMD [ "./ping-service" ]
EXPOSE 8080
And a VERSION.ver file to track the service version.
I have created a deploy folder in which I ran helm create ping-service to initialize the helm chart.
And finally the Makefile
# Get current version from VERSION.ver file.
GETVERSION = $(shell cat ${VERSION_FILE})
# CONSTS
IMAGE_NAME = ping-service
IMAGE_FULL_NAME = alexc94/$(IMAGE_NAME)
CHART_FULL_NAME= $(IMAGE_NAME)-chart
build:
echo "Build $(IMAGE_NAME) started."
docker build --tag $(IMAGE_FULL_NAME):$(GETVERSION) -f $(DIR)/Dockerfile .
echo "Build $(IMAGE_NAME) finished."
release:
echo "Start $(IMAGE_NAME) release phase."
# increment current service version from VERSION.ver file
$(eval VERSION=$(shell sudo ./semver.sh $(VERSION_FILE) release-patch))
# replace both appversion & chart version
sudo yq e '.appVersion = "$(GETVERSION)"' -i $(DIR)/deploy/Chart.yaml
sudo yq e '.version = "$(GETVERSION)"' -i $(DIR)/deploy/Chart.yaml
git config --global user.email "service@account.net"
git config --global user.name "service_account"
git add -A '*.ver'
git add -A '*.yaml'
git commit -m "version bump $(GETVERSION)"
git push origin HEAD
echo "${DOCKER_TOKEN}" | docker login --username ${DOCKER_USERNAME} --password-stdin
docker build --tag $(IMAGE_FULL_NAME):$(GETVERSION) -f $(DIR)/Dockerfile .
docker push $(IMAGE_FULL_NAME):$(GETVERSION)
# this is my ChartMuseum service hosted in my cluster
helm repo add alex-charts http://alexpc.ddnsking.com:8080
helm repo update
helm package $(DIR)/deploy --dependency-update
wget --post-file $(CHART_FULL_NAME)-$(GETVERSION).tgz http://alexpc.ddnsking.com:8080/api/charts
# this sets output like "ping-service=ping-service:1.0.0" which will be used by deploy script in git action.
echo "::set-output name=$(IMAGE_NAME)::$(IMAGE_NAME):$(GETVERSION)"
echo "Finish $(IMAGE_NAME) release phase."
2. Build phase
Typically we expect the CI to build the service during a pull request.
We could trigger this on each commit pushed to the branch, but personally, I don't think it's the best approach with regards to efficiency... because in the end, Github actions cost money.
So I've chosen to trigger the build when the pull request is created and on pull request approval.
The following is the pr_review.yaml github action file.
# This is a basic workflow to help you get started with Actions
name: PR review
# Controls when the workflow will run
on:
# action is triggered when pull request is opened
pull_request:
types: [opened]
# and when a reviewer approves the changes
pull_request_review:
types: [submitted]
# Allows you to run this workflow manually from the Actions tab
workflow_dispatch:
# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
# This workflow contains a single job called "build"
build:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v2
with:
fetch-depth: 2
token: ${{ secrets.GH_TOKEN }}
- name: Build monorepo
env:
ACTION: build
shell: bash
run: |
chmod +x ./build.sh
./build.sh
The action itself is pretty simple because the logic for finding and executing the Makefile is in the build.sh file.
#!/bin/bash
# set -x
# set -f
# create tmp file to track built services.
> BUILT_LIST
build () {
echo "Build input = $1"
DIRNAME=`echo $1`
MKFILE=`echo "${DIRNAME}/Makefile"`
# depth represents the number of folders in the path e.g: /golang/ping-service/deploy will have a depth of 3
DEPTH=${1//[^\/]/}
# try to find a makefile starting from the last folder and "walking" to the first one
for (( n=${#DEPTH}; n>0; --n )); do
if [ -f $MKFILE ]; then
break
else
DIRNAME="${DIRNAME}/.."
MKFILE=`echo "${DIRNAME}/Makefile"`
fi
done
# get the full path of the makefile.
MKFILE_FULL=`readlink -e ${MKFILE}`
# build only if it's not on our list of built makefiles.
BUILT=$(<BUILT_LIST)
if [[ $BUILT != *"${MKFILE_FULL}"* ]]; then
echo "Build ${DIRNAME} (${MKFILE_FULL})"
DIR=`dirname ${MKFILE_FULL}`
# by convention Version.ver should be in the same location as the Makefile
VERSION_FILE=`echo "${DIR}/VERSION.ver"`
# execute the Makefile
make -f $MKFILE ${ACTION} VERSION_FILE=${VERSION_FILE} DIR=${DIR}
if [ $? -ne 0 ]; then
echo "Build failed"
exit 1
fi
# add makefile path to the already built list
echo "${MKFILE_FULL}" > BUILT_LIST
else
echo "Skip ${MKFILE_FULL}; Already built"
fi
}
# script starts here
# get list of all modified files for the current branch
git diff-tree --name-only -r HEAD HEAD^ | while read line; do
echo "processing modified file: $line"
build `dirname $line`
echo "------------------------------"
done
Deploy phase
For this phase we will rely on Flux to do the hard work and safely deploy the services in the cluster.
Before I can talk about the deploy action you must first understand how I set up the initial configuration files for the 2 services to be used by Flux.
In base dir I created the helm release definitions for both services and in staging I have the values which will be used for this particular 'staging' environment.
I have also instructed Flux to sync these services by creating the apps.yaml file.
In order for Flux to deploy a new version for "ping-service" we simply need to change the version in this values.yaml file.
Now that hopefully you have an idea of how Flux synchronizes the staging environment we can talk about the pr_merge.yaml git action.
# This is a basic workflow to help you get started with Actions
name: PR merge
# Controls when the workflow will run
on:
# triggers when pull request is closed
pull_request:
branches: [main]
types: [closed]
# does not trigger if pull request contains changes in these directories because it's not relevant
paths-ignore:
- 'clusters/**'
- 'apps/**'
# Allows you to run this workflow manually from the Actions tab
workflow_dispatch:
# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
deploy:
# run steps only if pull request was merged and not simply closed
if: github.event.pull_request.merged == true
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v2
with:
fetch-depth: 2
token: ${{ secrets.GH_TOKEN }}
- name: Install dependencies
run: |
sudo wget -O /usr/local/bin/yq https://github.com/mikefarah/yq/releases/download/v4.6.3/yq_linux_amd64
sudo chmod +x /usr/local/bin/yq
sudo curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
sudo chmod +x get_helm.sh
./get_helm.sh
- name: Build & release
id: build_and_release
env:
DOCKER_USERNAME: ${{ secrets.DOCKER_HUB_USERNAME }}
DOCKER_TOKEN: ${{ secrets.DOCKER_HUB_ACCESS_TOKEN }}
ACTION: release
shell: bash
run: |
chmod +x ./semver.sh
chmod +x ./build.sh
./build.sh
- name: Deploy
shell: bash
env:
GH_SVC_ACCOUNT_TOKEN: ${{ secrets.GH_SVC_ACCOUNT_TOKEN }}
run: |
chmod +x ./deploy.sh
./deploy.sh ${{join(steps.build_and_release.outputs.*, ',')}}
This action has a bit more instructions than the previous one so I will try to break it down into a few lines:
- triggers only for merged pull requests into main branch
- triggers only if there are files changed in directories other than /clusters and /apps
- installs yq dependency for easily modifying yaml files.
- installs helm to generate packages upon release phase
- executes build script with ACTION environment variable set to 'release' which will be used to trigger the release step in the service Makefile
- executes the deploy script which has the responsibility to change the helm chart version
Let's talk a bit more about deploy.sh.
Maybe for staging environment this is not mandatory, but I was imagining that for production environment we shouldn't just simply deploy a new version once code is merged into main... so I thought that a good approach would be for the deploy script to modify the chart version in a separate branch and create a pull request to main in order for someone to review the changes and decide whether it's a good time to deploy to production or not.
So how it works?
In the release section of each service's Makefile, after docker image and helm chart are pushed I set an output in the form of "ping-service=ping-service:1.0.0" to signal next step actions that ping-service version 1.0.0 is ready to be deployed.
# this sets output like "ping-service=ping-service:1.0.0" which will be used by deploy script in git action.
echo "::set-output name=$(IMAGE_NAME)::$(IMAGE_NAME):$(GETVERSION)"
And now going back to pr_merge.yaml git action which will use this output and pass it to the deploy script as a comma separated string. I join all the outputs, because we can deploy more than 1 service at a time.
./deploy.sh ${{join(steps.build_and_release.outputs.*, ',')}}
And in deploy.sh we simply iterate over the services, create a new branch, change the chart version and create a pull request.
SERVICES=$1
APPS_PATH="apps/staging"
createPullRequest(){
svc_name_with_version=$1
#service should be in the form name:version -> split by :
arr=(${svc_name_with_version//:/ })
svc_name=${arr[0]}
svc_version=${arr[1]}
echo "Service processed is $svc_name with version $svc_version"
values_path=`echo $APPS_PATH/$svc_name/values.yaml`
branch_name=`echo release-$svc_name-$svc_version-to-staging`
if ! [ -f $values_path ]; then
exit 1
fi
git config --global user.email "service@account.net"
git config --global user.name "service_account"
git checkout -b $branch_name
yq e ".spec.chart.spec.version = \"$svc_version\"" -i $values_path
git add -A $values_path
git commit -m "Version bump $svc_name $svc_version"
git push -u origin $branch_name
curl --request POST \
--url https://api.github.com/repos/cioti/devops-chapter/pulls \
--header "authorization: Bearer $GH_SVC_ACCOUNT_TOKEN" \
--header 'content-type: application/json' \
--data "{\"title\":\"Release $svc_name v$svc_version to staging\",\"body\":\"Please review these changes in order for flux to sync the changes in staging!\",\"head\":\"cioti:$branch_name\",\"base\":\"main\"}"
git checkout main
}
# split services by ','
for svc in ${SERVICES//,/ } ; do
createPullRequest $svc
done
And that's it, done😍 CI/CD with GitOps, Flux and ChartMuseum in a few steps.
Well of course there's a bit more than that which I haven't covered in this post because it's already too long and I don't wanna bore you that much, but if you are interested, drop me a message and we can chat.
I'm sure there are better and nicer ways to achieve this so feel free to share💡
Thank you for reading and see you in the next one✌