Read more

How to checkout submodules in Gitlab CI

Emanuel
June 10, 2021Software engineer at makandra GmbH

Please prefer https://docs.gitlab.com/ee/ci/git_submodules.html with relative submodule paths

Accessing other repositories in Gitlab CI is not straight forward, since the access rights of the current pipeline might not be sufficient enough.

Illustration web development

Do you need DevOps-experts?

Your development team has a full backlog? No time for infrastructure architecture? Our DevOps team is ready to support you!

  • We build reliable cloud solutions with Infrastructure as code
  • We are experts in security, Linux and databases
  • We support your dev team to perform
Read more Show archive.org snapshot

One approach is to use project access tokens and clone the repositories via HTTPS.

  • Create a project access token Show archive.org snapshot for all submodules you want to have access to with the setting read_repository
  • Add the secrets as environment variable to the main project you want to have access to submodules:
    • Protected false (depending on your security settings), masked true
    • Add the project access token

Example configuration for the project manager with two submodules customer_1 and customer_2:


image:
  name: "example.com/manager/ci:v1"

before_script:
  # git clone without --recursive already happend
  - git submodule deinit --all --force # Enforce that the submodules are not cached before changing them
  - sed -i "s/git@code.example.com\/dev\/customer-1.git/https:\/\/gitlab-ci-token:$CI_CUSTOMER_1_ACCESS_TOKEN@code.example.com\/dev\/customer-1.git/" .gitmodules
  - sed -i "s/git@code.example.com\/dev\/customer-2/https:\/\/gitlab-ci-token:$CI_CUSTOMER_2_ACCESS_TOKEN@code.example.com\/dev\/customer-2.git/" .gitmodules
  - git submodule init
  - git submodule update
  - bundle install

rspec:
  script:
    - bundle exec rake db:create db:migrate
    - bundle exec rspec

This approach requires you to touch your CI configuration every time the gitmodules change. You also might prefer to remove the .gitmodules file entirely after the deinit and add the submodules one per line instead of using sed:

git submodule deinit --all --force

rm .gitmodules
git submodule add https://gitlab-ci-token:$CI_CUSTOMER_1_ACCESS_TOKEN@code.example.com/dev/customer-1.git customer-1
git submodule add https://gitlab-ci-token:$CI_CUSTOMER_2_ACCESS_TOKEN@code.example.com/dev/customer-2.git customer-2
git submodule update

Example error message when you try to clone via SSH with insufficient access rights:

Cloning into '/builds/manager/customer-1'...
Host key verification failed.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
fatal: clone of 'git@code.example.com/dev/customer-1.git' into submodule path '/builds/manager/customer-1' failed
Posted by Emanuel to makandra dev (2021-06-10 16:02)