Background:

Two months ago I started to work on migration from Bamboo CI/CD to Gitlab.com for my team in a big enterprise company. Our project has microservices architecture, which are running as docker containers (around 80 containers) on our servers.

Our plan was to have our repositories in Gitlab and use Gitlab CI/CD pipelines to build and deploy our apps. We will use self hosted Gitlab Runners as shell executors. (no k8s is used)

I am very happy with the result we could achieve and that's why I want to share our best practices with you 💪.

Here are main concepts of our CI/CD process in Gitlab:

Shared Pipeline Templates:

As you probably guessed, having some many microservices means their build and deploy pipelines are almost identical. So we had to find a way to reuse some .yml templates. For this we created a new project in gitlab called ci-cd-assets. This project contains default pipeline for all our apps and reusable parts of pipeline like common jobs as well.

We tried to achieve individual .gitlab-ci.yml file as minimal as possible in every project to keep all pipeline code in central place. In case you want to change how pipelines work you simply change it in ci-cd-assetsproject and its applied everywhere.

Here is an example of .gitlab-ci.yml file:

variables:  
  APP_NAME: account-service

include:
  - project: $CI_PROJECT_NAMESPACE/ci-cd-assets
    file: /templates/app-pipeline.yml
    ref: master

☝️ For this you need to whitelist all other projects in settings of ci-cd-assets

Even though Gitlab now has a feature called CI/CD components and Catalogs, I didnt see any benefit of this approach and simply having shared repo felt like a better idea.

Shared Dockerfile and shell scripts:

Beside sharing .yml templates we also have default Dockerfiles and other scripts used inside pipelines.

As you probably guess all of them also sit in shared project ci-cd-assets. As a first step in every piepeline we checkout this shared repo files and save them as artifacts. By saving them as artifacts you can share this files with all consequent jobs inside pipeline. Here is an example:

get-shared-assets:
  variables:
    CI_CD_ASSETS_REPOSITORY_URL: https://gitlab-ci-token:$CI_JOB_TOKEN@gitlab.com/$CI_PROJECT_NAMESPACE/ci-cd-assets.git

  before_script:
    - echo "Checking out CI/CD assets..."
  script:
    - git clone $CI_CD_ASSETS_REPOSITORY_URL ./build
  artifacts:
    paths:
      - ./build/*

☝️ All files of shared project will be available during builds under ./build folder.

Versioning with Git Tags:

We develop using feature branches. New code is pushed to feature branch, after merge request is approved its merged in master and then latter deployed to staging and production environments. Deployment to staging happens automatically but manually to production after testing.

We decided to separate build pipelines and deploy pipelines by presence of a git tag. This means when commit is pushed or merged to master branch without a tag a build pipeline will start.

During this build pipeline we run unit tests, linting, building a docker image and publishing a docker image to container registry. This docker image has a tag which is an application version e.g v10.0.0. As a last step of build pipeline we create a git tag with the same version as docker image. Now we have git tag v10.0.0

This relationship tells us which commit produced which docker image and if this code is deployed to an environment this application version is actually used.

You can think that every git tag created in repo is a potential release. Then the same git tag (or version) of the application can be released to staging or production.

Build Pipeline:

Push to a branch (feature or default) will trigger a build pipeline. Green status of build pipeline tells us that build was successful and a new tag is created for a release.

This is a basic example of build pipeline yml:

stages:
  - prepare
  - build
  - post-build

include:
  - '/templates/get-ci-cd-assets.yml'

build-and-test:
  stage: build
  before_script:
    - echo "Start building and testing..."
  script:
    - ./build/scripts/docker-build.sh
    - ./build/scripts/docker-push.sh 

tagging:
  stage: post-build
  before_script:
    - echo "Tagging the release with version..."
  script:
    - ./build/scripts/create-tag.sh

Deploy Pipeline:

Git tag creation will trigger a deploy pipeline. In Gitlab tag creation is one of the pipeline triggers. Deploy pipeline will read a tag for what it is currently running (which indicates a docker image version) and use this version inside a deploy script.

By this approach we could separate build and deploy pipelines. 😊 This means a commit to master branch (or merge) will first trigger a build pipeline, when build finishes a tag is created with a version. This tag will trigger a deploy pipeline to release this version so some environment.

This is an example of app pipeline yml file:

workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE != "merge_request_event"  # do not run MR creation
  
include:
  - local: '/templates/build-pipeline.yml'
    rules:
      - if: $CI_COMMIT_TAG == null
  - local: '/templates/deploy-pipeline.yml'
    rules:
      - if: $CI_COMMIT_TAG

☝️ Simply includes different pipeline templates based on git tag presence.

Lets say we merged a MR to master and deployed it to staging and production. This process produce 3 pipelines:

1. Build pipeline (runs for a branch)

2. Deploy to Staging pipeline (runs automatically for a tag)

3. Deploy to Production pipeline (runs manually for same tag)

Another useful feature of Gitlab is Environments. When you attach an environment keyword to a job it becomes a deploy job and gives you an deployment history in Gitlab.

This is a basic example of deploy pipeline yml:

variables:
 ENVIRONMENT: 'staging'

deploy-staging:
  stage: deploy
  variables:
    VERSION: $CI_COMMIT_TAG # read version from tag
  environment: staging
  before_script:
    - echo "Deploying to staging..."
  script:
    - ./build/scripts/deploy.sh $APP_NAME $VERSION $ENVIRONEMNT
  rules:
    - if: $ENVIRONMENT == 'staging'
      
deploy-production:
  stage: deploy
  variables:
    VERSION: $CI_COMMIT_TAG # read version from tag
  environment: production
  before_script:
    - echo "Deploying to production..."
  script:
    - ./build/scripts/deploy.sh $APP_NAME $VERSION $ENVIRONEMNT
  rules:
    - if: $ENVIRONMENT == 'production'

☝️ Default value of $ENVIRONMENT is 'staging', so deploy to staging happens automatically. To deploy a tag to production, you can manually start a pipeline, select a desired tag and set overwrite variable ENVIRONMENT to 'production'.

Any opinions are welcomed! Enjoy! 🎉