Saturday, January 15, 2022

[FIXED] Connecting to Docker Daemon from jenkins running inside a docker container

January 15, 2022 continuous-integration, docker, docker-compose, jenkins

Issue

I have a jenkins running inside a docker container and a docker running inside a different docker container.

I have map the /var/run/docker.sock file of local machine to the docker container and am able to execute docker commands inside docker container. Both the docker container and jenkins container are on the same network. But while connecting to docker container from jenkins, am getting connection refused. I have given 666 permissions to /var/run/docker.sock file but yet am not able to connect between the two. Both the container can ping each other sucessfully.

Solution

TL;DR

You could connect to the Docker in Docker environment over tcp or by sharing the docker socket between the containers.

This example is with everything in docker orchestrated using docker-compose.

.
├── docker-compose.yaml
├── Dockerfile
├── etc
│   └── nginx
│       └── conf.d
│           └── default.conf
└── plugins.txt

The docker-compose.yaml sets up jenkins behind nginx and a docker:20.10.5-dind service.

tcp

version: '3.7'
services:
  nginx:
    image: 'nginx:1.19'
    container_name: 'nginx'
    restart: 'always'
    depends_on:
    - 'jenkins'
    ports:
    - '80:80'
    volumes:
    - 'jenkins:/var/jenkins_home'
    - './etc/nginx/conf.d/default.conf:/etc/nginx/conf.d/default.conf'
  jenkins:
    build:
      context: '.'
    container_name: 'jenkins'
    restart: 'always'
    expose:
    - '50000'
    - '8080'
    environment:
    - 'DOCKER_HOST=tcp://docker:2376'
    - 'DOCKER_CERT_PATH=/certs/client'
    - 'DOCKER_TLS_VERIFY=1'
    volumes:
    - 'jenkins:/var/jenkins_home'
    - 'certs:/certs:ro'
  docker:
    image: 'docker:20.10.5-dind'
    container_name: 'docker'
    privileged: true
    volumes:
    - 'certs:/certs'
volumes:
  jenkins:
  certs:

Note: the docker client certificates are shared between the docker and the jenkins containers and the environment is set in the jenkins container to connect to the docker service.

The nginx config is slightly modified from the doc:

upstream jenkins {
  keepalive 32;
  server jenkins:8080 max_fails=3;
}
map $http_upgrade $connection_upgrade {
  default upgrade;
  '' close;
}
server {
  listen *:80;
  listen [::]:80;
  server_name _;

  charset utf-8;
  ignore_invalid_headers off;

  error_page   500 502 503 504  /50x.html;
  location = /50x.html {
      root   /usr/share/nginx/html;
  }
  location ~ "^/static/[0-9a-fA-F]{8}\/(.*)$" {
    rewrite "^/static/[0-9a-fA-F]{8}\/(.*)" /$1 last;
  }
  location /userContent {
    root /var/jenkins_home/;
    if (!-f $request_filename){
      rewrite (.*) /$1 last;
      break;
    }
    sendfile on;
  }
  location / {
    sendfile off;
    proxy_pass         http://jenkins;
    proxy_redirect     default;
    proxy_http_version 1.1;

    proxy_set_header   Connection        $connection_upgrade;
    proxy_set_header   Upgrade           $http_upgrade;

    proxy_set_header   Host              $host;
    proxy_set_header   X-Real-IP         $remote_addr;
    proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header   X-Forwarded-Proto $scheme;
    proxy_max_temp_file_size 0;

    client_max_body_size       10m;
    client_body_buffer_size    128k;

    proxy_connect_timeout      90;
    proxy_send_timeout         90;
    proxy_read_timeout         90;
    proxy_buffering            off;
    proxy_request_buffering    off;
    proxy_set_header Connection "";
  }
}

The jenkins service is a custom built image pre-baked with the docker client and the default suggested jenkins plugins plus the Docker and Docker Pipeline plugins:

FROM docker:20.10.5-dind as docker
FROM jenkins/jenkins:alpine
USER root
COPY --from=docker /usr/local/bin/docker /usr/local/bin/docker
COPY plugins.txt /usr/share/jenkins/plugins.txt
RUN /usr/local/bin/install-plugins.sh < /usr/share/jenkins/plugins.txt
USER jenkins

github:1.33.1
pipeline-model-api:1.8.4
scm-api:2.6.4
mailer:1.32.1
workflow-support:3.8
font-awesome-api:5.15.2-2
pipeline-milestone-step:1.3.2
git:4.6.0
plain-credentials:1.7
resource-disposer:0.15
jackson2-api:2.12.1
jquery3-api:3.5.1-3
gradle:1.36
credentials:2.3.15
docker-workflow:1.26
workflow-scm-step:2.12
display-url-api:2.3.4
bootstrap4-api:4.6.0-2
antisamy-markup-formatter:2.1
command-launcher:1.5
pipeline-stage-tags-metadata:1.8.4
snakeyaml-api:1.27.0
pipeline-stage-view:2.19
script-security:1.76
okhttp-api:3.14.9
pipeline-stage-step:2.5
workflow-step-api:2.23
timestamper:1.11.8
pipeline-github-lib:1.0
token-macro:2.13
pam-auth:1.6
workflow-cps-global-lib:2.18
ws-cleanup:0.39
pipeline-model-definition:1.8.4
workflow-aggregator:2.6
jsch:0.1.55.2
matrix-auth:2.6.5
ssh-credentials:1.18.1
ant:1.11
jjwt-api:0.11.2-9.c8b45b8bb173
momentjs:1.1.1
trilead-api:1.0.13
durable-task:1.35
workflow-job:2.40
git-server:1.9
ssh-slaves:1.31.5
plugin-util-api:2.0.0
git-client:3.6.0
lockable-resources:2.10
checks-api:1.5.0
pipeline-input-step:2.12
cloudbees-folder:6.15
pipeline-build-step:2.13
popper-api:1.16.1-2
pipeline-graph-analysis:1.10
matrix-project:1.18
workflow-api:2.41
github-branch-source:2.9.7
workflow-basic-steps:2.23
apache-httpcomponents-client-4-api:4.5.13-1.0
workflow-multibranch:2.22
workflow-cps:2.90
ldap:1.26
build-timeout:1.20
echarts-api:5.0.1-1
pipeline-model-extensions:1.8.4
structs:1.22
junit:1.48
docker-java-api:3.1.5.2
docker-plugin:1.2.2
workflow-durable-task-step:2.38
credentials-binding:1.24
jdk-tool:1.5
bouncycastle-api:2.20
docker-commons:1.17
github-api:1.123
authentication-tokens:1.4
email-ext:2.82
branch-api:2.6.2
pipeline-rest-api:2.19
ace-editor:1.1
handlebars:1.1.1

After the initial jenkins setup, create the X.509 Client Certificate Server Credentials then configure the Docker Cloud with the docker service using tcp.

Note: you can get the client cert, client key and server ca cert for creating the X.509 Client Certificate Server Credentials using the below commands:

docker exec docker cat /certs/client/key.pem
docker exec docker cat /certs/client/cert.pem
docker exec docker cat /certs/server/ca.pem

socket

version: '3.7'
services:
  nginx:
    image: 'nginx:1.19'
    container_name: 'nginx'
    restart: 'always'
    depends_on:
    - 'jenkins'
    ports:
    - '80:80'
    volumes:
    - 'jenkins:/var/jenkins_home'
    - './etc/nginx/conf.d/default.conf:/etc/nginx/conf.d/default.conf'
  jenkins:
    build:
      context: '.'
    container_name: 'jenkins'
    restart: 'always'
    expose:
    - '50000'
    - '8080'
    volumes:
    - 'jenkins:/var/jenkins_home'
    - 'socket:/var/run'
  docker:
    image: 'docker:20.10.5-dind'
    container_name: 'docker'
    privileged: true
    volumes:
    - 'socket:/var/run'
volumes:
  jenkins:
  socket:

Note: the docker socket is shared between the docker and the jenkins containers in the socket volume.

By default the docker socket is owned by root:root, the jenkins user is not able to connect to the shared socket, you can change the sockets group ownership to the GID of the jenkins user: docker exec docker chown 0:1000 /var/run/docker.sock.

After the initial jenkins setup configure the Docker Cloud with the docker service using the shared unix socket.

Answered By - masseyb
Answer Checked By - Marilyn (JavaFixing Volunteer)

This Answer collected from stackoverflow and tested by JavaFixing community admins, is licensed under cc by-sa 2.5 , cc by-sa 3.0 and cc by-sa 4.0