Skip to content

Docker-in-Docker service and Docker Hub ratelimit

The recommended way of running Docker commands in a GitLab CI Kubernets runner is to use a service to run Docker-in-Docker container (docker:dind). By default the docker:dind container pulls container images from Docker Hub without caching. This can cause problems as pulls from Docker Hub are rate-limited.

This tutorial tests how docker pull ubuntu:24.04 commands consume Docker Hub ratelimit and provides example on how to configure pull-through cache to an Kubernetes cluster working as a GitLab runner.

How to check if docker pull commands are cached

Create a new GitLab project with .gitlab-ci.yml and scripts/check-docker-hub-ratelimit.sh files.

The scripts/check-docker-hub-ratelimit.sh script prints current Docker Hub ratelimit and also writes the result to a file if filename is given as first parameter. See Docker Hub usage and rate limits article in Docker documentation for more details.

scripts/check-docker-hub-ratelimit.sh
#!/bin/sh -e

fetch_ratelimit() {
  curl -I -H "Authorization: Bearer $1" https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest 2>&1 | grep -i ratelimit
}

target=$1
token=$(curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:ratelimitpreview/test:pull" | jq -r .token)

if [ -n "$target" ]; then
  fetch_ratelimit $token | tee $target;
else
  fetch_ratelimit $token;
fi

The pipeline defined by .gitlab-ci.yml tries to pull the same Docker image twice and checks if the rate limit headers are different after first and second pull.

.gitlab-ci.yml
default:
  image: docker:cli
  services:
    - name: "docker:dind"
      command: ["--tls=false", "--host=tcp://0.0.0.0:2375"]

variables:
  DOCKER_TLS_CERTDIR: ""
  DOCKER_HOST: tcp://docker:2375

stages:
  - test

test-caching:
  stage: test
  before_script:
    - apk add curl jq
  script:
    - docker pull ubuntu:24.04
    - ./scripts/check-docker-hub-ratelimit.sh 1st.txt
    - docker rmi ubuntu:24.04
    - docker pull ubuntu:24.04
    - ./scripts/check-docker-hub-ratelimit.sh 2nd.txt
    - diff 1st.txt 2nd.txt

The pipeline should fail, if images are pulled directly from Docker Hub.

How to setup pull-through cache to Kubernetes runner

Manifests for configuring pull-through cache and configmap are available in the repository that provides this website.

git clone https://github.com/cicd-tutorials/cicd-tutorials.net.git
cd docs/tutorials/gitlab-ci/docker-hub-ratelimit

Configure pull-through cache and configmap for GitLab runner by running kubectl apply -f manifests/.

kubectl apply -f manifests/

These manifests assume that GitLab runner is using gitlab-runner namespace. Edit the namespace value in docker-config.yaml if this is not the case.

Modify the GitLab runner configuration so that the configmap defined in docker-config.yaml is mounted to all containers launched by the GitLab runner. Example of a full values.yaml help input file below.

gitlabUrl: # Your instance URL
rbac: { create: true }
runnerToken: # Your runner token
runners:
  config: |
    [[runners]]
      executor = "kubernetes"
      [runners.kubernetes]
        image = "alpine:3.12"
        privileged = true
        [[runners.kubernetes.volumes.config_map]]
          name = "dockerd-config"
          mount_path = "/etc/docker/daemon.json"
          sub_path = "daemon.json"