Monorepo CI/CD with GitOps, Flux and ChartMuseum

Monorepo CI/CD with GitOps, Flux and ChartMuseum

Alex Ciotirla's photo
Alex Ciotirla
·

9 min read

I have finally built my own Kubernetes bare-metal playground and I'm ready to dive into some nice use cases. If you're curious about how I set up my environment you can check previous articles from this series.

The first issue I wish to tackle is the CI/CD pipeline for a monorepo hosted on Github.

Disclaimer: I will be focusing in this initial phase more on the happy flow and won't pay too much attention on error cases.

What do I wish to accomplish in this first iteration?

By the end of this article, I want to successfully release and deploy a new version for 2 Golang services in my k8s staging environment upon successful pull request merge to the main branch.

Short description of the workflow I envision:

Each service in the monorepo must have:

  • Dockerfile -> builds & runs the app
  • Makefile -> manage versioning, trigger docker build, push image to DockerHub & push helm chart to Chartmuseum
  • Helm chart -> used by Flux to deploy to k8s

Upon pull request approval a github action will trigger a bash script that identifies and runs the Makefile build phase of the changed service.

Upon pull request merge to main branch a github action will trigger a bash script that identifies and runs the Makefile release phase of the changed service.

Let's breakdown the work:

  1. Create 2 services with Dockerfile, Makefile, version file and helm chart
  2. Git action for build phase
  3. Git action for deploy phase

1 - Create the services

I have created 2 simple Golang ping and pong services.

With Dockerfile

FROM golang:1.18-alpine as build

WORKDIR /app

# file context should be root of repository
COPY golang/go.mod ./
COPY golang/go.sum ./
RUN go mod download

COPY golang/ping-service/ ./

RUN go build -o ping-service

FROM alpine:latest
WORKDIR /svc

COPY --from=build /app .

CMD [ "./ping-service" ]
EXPOSE 8080

And a VERSION.ver file to track the service version.

I have created a deploy folder in which I ran helm create ping-service to initialize the helm chart.

And finally the Makefile

# Get current version from VERSION.ver file.
GETVERSION = $(shell cat ${VERSION_FILE})

# CONSTS
IMAGE_NAME = ping-service
IMAGE_FULL_NAME = alexc94/$(IMAGE_NAME)
CHART_FULL_NAME= $(IMAGE_NAME)-chart

build:
    echo "Build $(IMAGE_NAME) started."
    docker build --tag $(IMAGE_FULL_NAME):$(GETVERSION) -f $(DIR)/Dockerfile .
    echo "Build $(IMAGE_NAME) finished."
release:
    echo "Start $(IMAGE_NAME) release phase."
#   increment current service version from VERSION.ver file
    $(eval VERSION=$(shell sudo ./semver.sh $(VERSION_FILE) release-patch))
#    replace both appversion & chart version
    sudo yq e '.appVersion = "$(GETVERSION)"' -i $(DIR)/deploy/Chart.yaml
    sudo yq e '.version = "$(GETVERSION)"' -i $(DIR)/deploy/Chart.yaml
    git config --global user.email "service@account.net"
    git config --global user.name "service_account"
    git add -A '*.ver'
    git add -A '*.yaml'
    git commit -m "version bump $(GETVERSION)"
    git push origin HEAD
    echo "${DOCKER_TOKEN}" | docker login --username ${DOCKER_USERNAME} --password-stdin
    docker build --tag $(IMAGE_FULL_NAME):$(GETVERSION) -f $(DIR)/Dockerfile .
    docker push $(IMAGE_FULL_NAME):$(GETVERSION)
#   this is my ChartMuseum service hosted in my cluster
    helm repo add alex-charts http://alexpc.ddnsking.com:8080
    helm repo update
    helm package $(DIR)/deploy --dependency-update
    wget --post-file $(CHART_FULL_NAME)-$(GETVERSION).tgz http://alexpc.ddnsking.com:8080/api/charts
#    this sets output like "ping-service=ping-service:1.0.0" which will be used by deploy script in git action.
    echo "::set-output name=$(IMAGE_NAME)::$(IMAGE_NAME):$(GETVERSION)"
    echo "Finish $(IMAGE_NAME) release phase."

2. Build phase

Typically we expect the CI to build the service during a pull request.

We could trigger this on each commit pushed to the branch, but personally, I don't think it's the best approach with regards to efficiency... because in the end, Github actions cost money.

So I've chosen to trigger the build when the pull request is created and on pull request approval.

The following is the pr_review.yaml github action file.

# This is a basic workflow to help you get started with Actions

name: PR review

# Controls when the workflow will run
on:
 # action is triggered when pull request is opened
  pull_request:
   types: [opened] 
 # and when a reviewer approves the changes
  pull_request_review:
   types: [submitted]

  # Allows you to run this workflow manually from the Actions tab
  workflow_dispatch:

# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
  # This workflow contains a single job called "build"
  build:
    runs-on: ubuntu-latest
    steps:
    - name: Checkout code
      uses: actions/checkout@v2
      with:
        fetch-depth: 2
        token: ${{ secrets.GH_TOKEN }}
    - name: Build monorepo
      env:
        ACTION: build
      shell: bash
      run: |
          chmod +x ./build.sh
          ./build.sh

The action itself is pretty simple because the logic for finding and executing the Makefile is in the build.sh file.

#!/bin/bash
# set -x
# set -f

# create tmp file to track built services.
> BUILT_LIST

build () {
  echo "Build input = $1"  
  DIRNAME=`echo $1`
  MKFILE=`echo "${DIRNAME}/Makefile"`
  # depth represents the number of folders in the path e.g: /golang/ping-service/deploy will have a depth of 3
  DEPTH=${1//[^\/]/}

  # try to find a makefile starting from the last folder and "walking" to the first one
  for (( n=${#DEPTH}; n>0; --n )); do    
    if [ -f $MKFILE ]; then      
      break
    else      
      DIRNAME="${DIRNAME}/.."
      MKFILE=`echo "${DIRNAME}/Makefile"`
    fi
  done

  # get the full path of the makefile.
  MKFILE_FULL=`readlink -e ${MKFILE}`
  # build only if it's not on our list of built makefiles.    
  BUILT=$(<BUILT_LIST)    
  if [[ $BUILT != *"${MKFILE_FULL}"* ]]; then
    echo "Build ${DIRNAME} (${MKFILE_FULL})"
    DIR=`dirname ${MKFILE_FULL}`
  # by convention Version.ver should be in the same location as the Makefile
    VERSION_FILE=`echo "${DIR}/VERSION.ver"`
  # execute the Makefile
    make -f $MKFILE ${ACTION} VERSION_FILE=${VERSION_FILE} DIR=${DIR}

    if [ $? -ne 0 ]; then
      echo "Build failed"         
      exit 1
    fi

  # add makefile path to the already built list
    echo "${MKFILE_FULL}" > BUILT_LIST
  else
    echo "Skip ${MKFILE_FULL}; Already built"
  fi
}

# script starts here
# get list of all modified files for the current branch 
git diff-tree --name-only -r HEAD HEAD^ | while read line; do  
    echo "processing modified file: $line" 
    build `dirname $line`
    echo "------------------------------"
done

Deploy phase

For this phase we will rely on Flux to do the hard work and safely deploy the services in the cluster.

Before I can talk about the deploy action you must first understand how I set up the initial configuration files for the 2 services to be used by Flux.

In base dir I created the helm release definitions for both services and in staging I have the values which will be used for this particular 'staging' environment.

I have also instructed Flux to sync these services by creating the apps.yaml file.

In order for Flux to deploy a new version for "ping-service" we simply need to change the version in this values.yaml file.

Now that hopefully you have an idea of how Flux synchronizes the staging environment we can talk about the pr_merge.yaml git action.

# This is a basic workflow to help you get started with Actions

name: PR merge

# Controls when the workflow will run
on:
# triggers when pull request is closed
  pull_request:
   branches: [main]
   types: [closed]
# does not trigger if pull request contains changes in these directories because it's not relevant
   paths-ignore:
    - 'clusters/**'
    - 'apps/**'

  # Allows you to run this workflow manually from the Actions tab
  workflow_dispatch:

# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
  deploy:
# run steps only if pull request was merged and not simply closed
    if: github.event.pull_request.merged == true
    runs-on: ubuntu-latest
    steps:
    - name: Checkout code
      uses: actions/checkout@v2
      with:
        fetch-depth: 2
        token: ${{ secrets.GH_TOKEN }}
    - name: Install dependencies
      run: |
        sudo wget -O /usr/local/bin/yq https://github.com/mikefarah/yq/releases/download/v4.6.3/yq_linux_amd64
        sudo chmod +x /usr/local/bin/yq
        sudo curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
        sudo chmod +x get_helm.sh
        ./get_helm.sh
    - name: Build & release
      id: build_and_release
      env:
        DOCKER_USERNAME: ${{ secrets.DOCKER_HUB_USERNAME }}
        DOCKER_TOKEN: ${{ secrets.DOCKER_HUB_ACCESS_TOKEN }}
        ACTION: release
      shell: bash
      run: |
          chmod +x ./semver.sh
          chmod +x ./build.sh
          ./build.sh
    - name: Deploy
      shell: bash
      env:
        GH_SVC_ACCOUNT_TOKEN: ${{ secrets.GH_SVC_ACCOUNT_TOKEN }}
      run: |
        chmod +x ./deploy.sh
        ./deploy.sh ${{join(steps.build_and_release.outputs.*, ',')}}

This action has a bit more instructions than the previous one so I will try to break it down into a few lines:

  • triggers only for merged pull requests into main branch
  • triggers only if there are files changed in directories other than /clusters and /apps
  • installs yq dependency for easily modifying yaml files.
  • installs helm to generate packages upon release phase
  • executes build script with ACTION environment variable set to 'release' which will be used to trigger the release step in the service Makefile
  • executes the deploy script which has the responsibility to change the helm chart version

Let's talk a bit more about deploy.sh.

Maybe for staging environment this is not mandatory, but I was imagining that for production environment we shouldn't just simply deploy a new version once code is merged into main... so I thought that a good approach would be for the deploy script to modify the chart version in a separate branch and create a pull request to main in order for someone to review the changes and decide whether it's a good time to deploy to production or not.

So how it works?

In the release section of each service's Makefile, after docker image and helm chart are pushed I set an output in the form of "ping-service=ping-service:1.0.0" to signal next step actions that ping-service version 1.0.0 is ready to be deployed.

# this sets output like "ping-service=ping-service:1.0.0" which will be used by deploy script in git action.
echo "::set-output name=$(IMAGE_NAME)::$(IMAGE_NAME):$(GETVERSION)"

And now going back to pr_merge.yaml git action which will use this output and pass it to the deploy script as a comma separated string. I join all the outputs, because we can deploy more than 1 service at a time.

./deploy.sh ${{join(steps.build_and_release.outputs.*, ',')}}

And in deploy.sh we simply iterate over the services, create a new branch, change the chart version and create a pull request. 

SERVICES=$1
APPS_PATH="apps/staging"

createPullRequest(){
    svc_name_with_version=$1
    #service should be in the form name:version -> split by :
    arr=(${svc_name_with_version//:/ })
    svc_name=${arr[0]}
    svc_version=${arr[1]}

    echo "Service processed is $svc_name with version $svc_version"
    values_path=`echo $APPS_PATH/$svc_name/values.yaml`
    branch_name=`echo release-$svc_name-$svc_version-to-staging`
    if ! [ -f $values_path ]; then  
        exit 1
    fi
    git config --global user.email "service@account.net"
    git config --global user.name "service_account"
    git checkout -b $branch_name
    yq e ".spec.chart.spec.version = \"$svc_version\"" -i $values_path

    git add -A $values_path
    git commit -m "Version bump $svc_name $svc_version"
    git push -u origin $branch_name

    curl --request POST \
        --url https://api.github.com/repos/cioti/devops-chapter/pulls \
        --header "authorization: Bearer $GH_SVC_ACCOUNT_TOKEN" \
        --header 'content-type: application/json' \
        --data "{\"title\":\"Release $svc_name v$svc_version to staging\",\"body\":\"Please review these changes in order for flux to sync the changes in staging!\",\"head\":\"cioti:$branch_name\",\"base\":\"main\"}"
    git checkout main
}

# split services by ','
for svc in ${SERVICES//,/ } ; do 
    createPullRequest $svc
done

And that's it, done😍 CI/CD with GitOps, Flux and ChartMuseum in a few steps.

image.png

Well of course there's a bit more than that which I haven't covered in this post because it's already too long and I don't wanna bore you that much, but if you are interested, drop me a message and we can chat.

I'm sure there are better and nicer ways to achieve this so feel free to share💡

Thank you for reading and see you in the next one✌

KubernetesCI/CDci-cdgithub-actions