Since a few months ago this is possible using gitlab-runner:
gitlab-runner exec docker my-job-name
Note that you need both docker and gitlab-runner installed on your computer to get this working.
You also need the image key defined in your .gitlab-ci.yml file. Otherwise won't work.
Here's the line I currently use for testing locally using gitlab-runner:
gitlab-runner exec docker test --docker-volumes "/home/elboletaire/.ssh/id_rsa:/root/.ssh/id_rsa:ro"
Note: You can avoid adding a
--docker-volumeswith your key setting it by default in/etc/gitlab-runner/config.toml. See the official documentation for more details. Also, usegitlab-runner exec docker --helpto see all docker-based runner options (like variables, volumes, networks, etc.).
Due to the confusion in the comments, I paste here the gitlab-runner --help result, so you can see that gitlab-runner can make builds locally:
gitlab-runner --help
NAME:
gitlab-runner - a GitLab Runner
USAGE:
gitlab-runner [global options] command [command options] [arguments...]
VERSION:
1.1.0~beta.135.g24365ee (24365ee)
AUTHOR(S):
Kamil Trzciński <ayufan@ayufan.eu>
COMMANDS:
exec execute a build locally
[...]
GLOBAL OPTIONS:
--debug debug mode [$DEBUG]
[...]
As you can see, the exec command is to execute a build locally.
Even though there was an issue to deprecate the current gitlab-runner exec behavior, it ended up being reconsidered and a new version with greater features will replace the current exec functionality.
Note that this process is to use your own machine to run the tests using docker containers. This is not to define custom runners. To do so, just go to your repo's CI/CD settings and read the documentation there. If you wanna ensure your runner is executed instead of one from gitlab.com, add a custom and unique tag to your runner, ensure it only runs tagged jobs and tag all the jobs you want your runner to be responsible of.
Unofficial alternative gitlab-ci-local
There's an unofficial package which looks promising, with more features than the official gitlab-runner exec: https://github.com/firecow/gitlab-ci-local
It allows you to run gitlab pipelines locally as shell executor or docker executor.
Answer from elboletaire on Stack OverflowSince a few months ago this is possible using gitlab-runner:
gitlab-runner exec docker my-job-name
Note that you need both docker and gitlab-runner installed on your computer to get this working.
You also need the image key defined in your .gitlab-ci.yml file. Otherwise won't work.
Here's the line I currently use for testing locally using gitlab-runner:
gitlab-runner exec docker test --docker-volumes "/home/elboletaire/.ssh/id_rsa:/root/.ssh/id_rsa:ro"
Note: You can avoid adding a
--docker-volumeswith your key setting it by default in/etc/gitlab-runner/config.toml. See the official documentation for more details. Also, usegitlab-runner exec docker --helpto see all docker-based runner options (like variables, volumes, networks, etc.).
Due to the confusion in the comments, I paste here the gitlab-runner --help result, so you can see that gitlab-runner can make builds locally:
gitlab-runner --help
NAME:
gitlab-runner - a GitLab Runner
USAGE:
gitlab-runner [global options] command [command options] [arguments...]
VERSION:
1.1.0~beta.135.g24365ee (24365ee)
AUTHOR(S):
Kamil Trzciński <ayufan@ayufan.eu>
COMMANDS:
exec execute a build locally
[...]
GLOBAL OPTIONS:
--debug debug mode [$DEBUG]
[...]
As you can see, the exec command is to execute a build locally.
Even though there was an issue to deprecate the current gitlab-runner exec behavior, it ended up being reconsidered and a new version with greater features will replace the current exec functionality.
Note that this process is to use your own machine to run the tests using docker containers. This is not to define custom runners. To do so, just go to your repo's CI/CD settings and read the documentation there. If you wanna ensure your runner is executed instead of one from gitlab.com, add a custom and unique tag to your runner, ensure it only runs tagged jobs and tag all the jobs you want your runner to be responsible of.
Unofficial alternative gitlab-ci-local
There's an unofficial package which looks promising, with more features than the official gitlab-runner exec: https://github.com/firecow/gitlab-ci-local
It allows you to run gitlab pipelines locally as shell executor or docker executor.
Deprecated since 2024-05 (see edit)
Only for gitlab-runner < 17.0.
- See gitlab issue.
- See gitlab docu
I use this docker-based approach:
Edit: 2024-05
docker run --entrypoint bash --rm -w "PWD:$PWD" -v /var/run/docker.sock:/var/run/docker.sock gitlab/gitlab-runner:v15.11.1 -c 'git config --global --add safe.directory "*";gitlab-runner exec docker test'
Thanks to user Ivo Smits who reported a successful test with gitlab-runner:v15.11.1
Edit: 2022-10
For all git versions > 2.35.2. You must add safe.directory within the container to avoid fatal: detected dubious ownership in repository at.... This also true for patched git versions < 2.35.2. The old command will not work anymore.
Details
0. Create a git repo to test this answer
mkdir my-git-project
cd my-git-project
git init
git commit --allow-empty -m"Initialize repo to showcase gitlab-runner locally."
1. Go to your git directory
cd my-git-project
2. Create a .gitlab-ci.yml
Example .gitlab-ci.yml
image: alpine
test:
script:
- echo "Hello Gitlab-Runner"
3. Create a docker container with your project dir mounted
docker run -d \
--name gitlab-runner \
--restart always \
-v "
PWD" \
-v /var/run/docker.sock:/var/run/docker.sock \
gitlab/gitlab-runner:latest
(-d) run container in background and print container ID
(--restart always) or not?
(-v ") Mount current directory into the current directory of the container - Note: On Windows you could bind your dir to a fixed location, e.g. PWD"
-v "${PWD}:/opt/myapp". Also $PWD will only work at powershell not at cmd
(-v /var/run/docker.sock:/var/run/docker.sock) This gives the container access to the docker socket of the host so it can start "sibling containers" (e.g. Alpine).
(gitlab/gitlab-runner:latest) Just the latest available image from dockerhub.
4. Execute with
Avoid fatal: detected dubious ownership in repository at... More info
docker exec -it -w $PWD gitlab-runner git config --global --add safe.directory "*"
Actual execution
docker exec -it -w $PWD gitlab-runner gitlab-runner exec docker test
# ^ ^ ^ ^ ^ ^
# | | | | | |
# (a) (b) (c) (d) (e) (f)
(a) Working dir within the container. Note: On Windows you could use a fixed location, e.g. /opt/myapp.
(b) Name of the docker container
(c) Execute the command "gitlab-runner" within the docker container
(d)(e)(f) run gitlab-runner with "docker executer" and run a job named "test"
5. Prints
...
Executing "step_script" stage of the job script
$ echo "Hello Gitlab-Runner"
Hello Gitlab-Runner
Job succeeded
...
Note: The runner will only work on the commited state of your code base. Uncommited changes will be ignored. Exception: The .gitlab-ci.yml itself does not have be commited to be taken into account.
Note: There are some limitations running locally. Have a look at limitations of gitlab runner locally.
Worth checking:
GitLab CI/CD Pipeline: Run Script via SSH to remote server
Better way for multiline ssh command
TL;DR
Create SSH keys. You'll get 2 keys: one private-key (ex: id_ed25519) and one public-key (ex: id_ed25519.pub)
Add the private-key as a GitLab Variable: Settings -> CI/CD -> Variables -> Expand -> Add Variable (GitLab’s variable is a key-value pair. Name the key SSH_PRIVATE_KEY and paste your private-key in the value field. Then Click Add Variable.)
Add one more Gitlab variable: DServerIP as the IP address of the remote server (in your case, the server hosting the gitLab-runner)
Add the public-key to the remote server under the .ssh folder (it should be /home/username/.ssh or /root/.ssh)
Paste the public-key into authorized_keys file (it should be /home/username/.ssh/authorized_keys or /root/.ssh/authorized_keys, but if you don’t have authorized_keys file, create it.)
use the following script in your .gitlab-ci file
image: alpine:latest
stages:
- Execute_script_on_remote_host
update:
stage: Execute_script_on_remote_host
before_script:
- 'command -v ssh-agent >/dev/null || ( apk add --update openssh-client )'
- eval $(ssh-agent -s)
- echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- ssh-keyscan $DServerIP >> ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts
script:
- scp ./auto_update_test.sh root@$DServerIP:/opt/splunktest
- |
ssh -tt root@$DServerURL /bin/bash -s << EOT
cd /opt/splunktest
chmod +x auto_update_test.sh
bash auto_update_test.sh
exit
EOT
Notes:
- make sure that the file auto_update_test.sh is in the same directory as .gitlab-ci.yml (or just change its path in the script)
- you can put all your commands into a shell script (ex: remote_update.sh) in your project and change the SSH command:
remote_update.sh:
cd /opt/splunktest
chmod +x auto_update_test.sh
bash auto_update_test.sh
exit
SSH command in the script:
script:
- scp ./auto_update_test.sh root@$DServerIP:/opt/splunktest
- ssh -tt root@$DServerIP "/bin/bash -s" <remote_update.sh
- you even can connect via ssh and execute the commands of the auto_update_test.sh file without the need to copying it to the remote server.
SSH command in the script:
script:
- ssh -tt root@$DServerIP "/bin/bash -s" <auto_update_test.sh
See also:
- https://superuser.com/questions/649186/how-can-i-get-the-scp-command-to-overwrite-the-destination-folder
- Pseudo-terminal will not be allocated because stdin is not a terminal
Your runner is using the docker executor. When you run jobs with this runner, it creates new docker containers for each job. The container has its own filesystem separate from your 'baremetal' host where the runner is installed.
Therefore you have two problems:
- you won't find
/opt/splunkin your container filesystem (the immediate cuase of your error) - Running the script inside your job container may not have the effect you're expecting it to have on the host system, since it's running inside a container.
If you only need to update files on the host in the /opt/splunk directory, you can configure your runner to mount this directory from the host into your job containers, so that /opt/splunk from the host will be available in /opt/splunk in the job containers and changes within that directory will persist on the host.
In your runner's /etc/gitlab-runner/config.toml, you can add this volumes key:
# ...
[runners.docker]
# ...
volumes = ["/opt/splunk"]
Then restart your runner.
You could also consider using a shell executor instead, which would run jobs (and commands therein) directly on the same host and filesystem(s) where the runner is installed.