devops - What is gitlab runner - Stack Overflow
GitLab: Is it possible to run pipeline on a specific runner? - Stack Overflow
Running the entire GitLab CI pipeline locally with external GitLab runners
Building for Windows in GitLab CI
What is runner actually for?
You have your project along with a .gitlab-ci.yml file. .gitlab-ci.yml defines what stages your CI/CD pipeline has and what to do in each stage. This typically consists of a build,test,deploy stages. Within each stage you can define multiple job. For example in build stage you may have 3 jobs to build on debian, centos and windows (in GitLab glossary build:debian, build:centos, build:windows). A GitLab runner clones the project read the gitlab-ci.yaml file and do what he is instructed to do. So basically GitLab runner is a Golang process that executes some instructed tasks.
where is it meant to be installed?
You can install a runner in your desired environment listed here. https://docs.gitlab.com/runner/install/ or you can use a shared runner that is already installed on GitLab's infrastructure.
Does it care which directory it is run in?
Yes. Every task executed by runner is relativly to CI_PROJECT_DIR defined in https://gitlab.com/help/ci/variables/README. But you can alter this behaviour.
where does it execute it's script commands? At root?
Do I need to set my executor to docker? Shouldn't I just set it to shell, pull the image, and build it?
A runner can have mutiple executors such as docker, shell, virtualbox etc but docker being the most common one. If you use docker as the executor you can pull any image from docker hub or your configured registry and you can do loads of stff with docker images. In a docker environment normally you run them as the root user. https://docs.gitlab.com/runner/executors/README.html
GitLab runner is a build instance which is used to run the jobs over multiple machines and send the results to GitLab and which can be placed on separate users, servers, and local machine. You can register the runner as shared or specific after installing it.
Shared Runners : These runners are useful for jobs multiple projects which have similar requirements. Instead of using multiple runners for many projects, you can use a single or a small number of Runners to handle multiple projects which will be easy to maintain and update.
Specific Runners : These runners are useful to deploy a certain project, if jobs have certain requirements or specific demand for the projects. Specific runners use FIFO (First In First Out) process for organizing the data with first-come first-served basis.
Protected Runners : The runners can be protected to save the important information.
You have two mechanisms by which you can attempt to isolate a new runner for testing:
- use tags and private runner attachment (already called out)
- use the gitlab-runner exec verb directly on the runner
- canary the runner for a single build only
Option 1
use tags and private runner attachment (already called out).
To further expand on this... even in a draconian setup where you can't alter tags and whatnot -- you can always FORK the project.
In your new private fork, you can go to Settings >> CI/CD and override the .gitlab-ci.yml file in the Custom CI Configuration Path under the General Pipelines Settings. This allows you to git cp .gitlab-ci.yml .mycustomgitlab-ci.yml and then simply git add/git commit/git push and you're in business.
Opinion: If you cannot use the mechanisms in place to adjust the tags on the questionable runner and isolate a new forked project, this isn't a technical problem, it is a political problem.
Option 2
Gitlab-runner exec....
Assuming you're using a shell gitlab runner...
- SSH to the questionable gitlab runner box you're trying to test
- Clone the repo for the project in question to ... say ...
/tmp/myrepo - Execute Gitlab-Runner:
/path/to/gitlab-runner exec shell {.gitlab-ci.yml target}
See https://docs.gitlab.com/runner/commands/#gitlab-runner-exec and a blog about it at https://substrakt.com/how-to-debug-gitlab-ci-builds-locally/
Option 3
Canary the gitlab-runner for a single build.
You can spin up the gitlab-runner process to do N number of builds and then go back offline. See: https://docs.gitlab.com/runner/commands/#gitlab-runner-run-single
... This is not zero-impact, but will definitely limit the blast radius of any problems.
There currently isn't a solution for building on a specific runner in GitLab, but there is an issue open for Sticky Runners, which hopefully will be out in the next 3-6 months according to the Milestones!
The work around I've done so far to build a project on a specific runner is to use the GitLab Runner API, in a rather hacky way, along the lines of:
- Get all project runners
- As I know I've deployed the latest runner, that would have the highest runner "number"
- Pause all the other runners associated with the project in question
- Trigger the pipeline to build on the latest runner
- Poll the GitLab API to get the status of the pipeline
- Once that succeeds, resume all other runners!
- If the pipeline fails, remember to resume the paused runners...
Is there a way to run the entire GitLab CI pipeline on the local host, where it uses the same GitLab runners that are located on external compute resources, such as an external build server?
I know of the gitlab-runner exec command, but as I understand it, that command uses the local host as the GitLab runner. Is there a way of using the command the same way, but with the external GitLab runners that are already used in the CI pipeline when pushing to the cloud? A thought that came to mind is to use a script to ssh into an external host and then use the GitLab runner command to perform all jobs in the gitlab-ci.yml file. However, this will require the local host to link the repository to the external host somehow before executing the pipeline. This method also seems tedious when having multiple external computers with a GitLab runner. Does anyone have any better suggestions?