Docker

Setting up docker on Ubuntu. See getting started.

Danger

Do NOT expose docker.sock to containers, even in read-only.

Install Docker Service

Add docker repository for latest docker packages.

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu xenial stable"
apt update && apt install docker-ce

Containers operate with their own internal UID when accessing volume mounts, most allow you to specify UID and GID of the running services for proper access. Create a docker user with no privileges so we can explicitly set file restrictions in mounted volumes for containers.

Add docker user for containers that can set effective UID.

adduser --system --home /etc/docker --shell /bin/false docker

Create A Standalone Container

Enables the use of host network without additional networking options (e.g. exposed docker ports appear as if they are on the host). Needs to be manully run every time to create a container.

Great for one-time usage; however compose is preferred in almost all cases.

Create a standalone docker container.

docker run -t -d \
  --name {CONTAINER NAME} \
  --network host \
  --restart unless-stopped \
  -p 3000:3000 \
  -p 4000:4000/udp \
  -v /etc/localtime:/etc/localtime:ro \
  {REPO}/{CONTAINER}:{TAG}

Runs a container detached, using host network exposing port 3000 and port 4000 UDP.
Always restarts the container, unless explicitly stopped.
Map /etc/localtime to set the containers timezone properly.
If the container is not found, it will automatically be pulled.
Using create instead of run -d will create the container but not start it in the background automatically.
Daemon mode -t -d is needed to keep the container in interactive mode; otherwise as soon as the container is idle it will sleep, stopping the background running services.

Compose with Containers 

Enables the management of multiple docker containers within a single YAML file to manage them as a service unit. Additionally eases modification and updates of those containers. This is preferred to running standalone containers. Commands mirror standard docker commands but are called with docker-compose.

Important

Expose ports are visible internally on the docker network for the container.

Ports are ports that are publicly visible from outside the docker network.

Generally with compose only reverse proxy ports are set as Ports and the containers Expose port internally for the proxy.

Each YAML file represents a service and is generally stored in separate directory representing the service name.

0640 root staff {SERVICE NAME}/docker-compose.yaml

version: "3"

services:
  mycontainer:
    image: {REPO}/{CONTAINER}:{TAG}
    network_mode: host
    restart: unless-stopped
    stop_grace_period: 1m
    ports:
      - '3000:3000'
      - '4000:4000/udp'
    environment:
      - ENV_VAR=value
    volumes:
      - /etc/localtime:/etc/localtime:ro
  mycontainer2:
    image: {REPO}/{CONTAINER}:{TAG}
    restart: unless-stopped
    expose:
      - '3000'
    depends_on:
      - mycontainer
    volumes:
      - /etc/localtime:/etc/localtime:ro

All ports need quotes to be interpreted correctly, as YAML interprets (LOL) values in XX:XX as Time.
Expose explicitly exposes ports within the isolated services networks for other services to use, it is not publically accessible.
depends_on will require other named docker container to start before this container.
stop_grace_period is optional, but enables a longer shutdown time for a given docker container, if shutdown requires more than 10 seconds (e.g. writing data to database, etc).

Turnup All Composed Services

docker-compose up -d

--remove-orphans will delete containers removed from the config.
Use in conjunction with docker-compose pull to automatically update images, then restart containers with the new images.
Any changes to containers will be re-configured automatically, including new images.
services are auto labeled as {SERVICE}_{CONTAINER}_{INSTANCE}. Service being the directory name, container being the container name and instance being the specific numbered instance (typically 1).
Ensure you are the right user; standard sudo will launch jobs with your username. Switch Users or sudo su - root -c "docker-compose up -d".

Common Management Tasks

See cheetsheet. Compose commands mirror standard docker commands but are called with docker-compose. Individual compose containers may be managed with the docker command as well using the {SERVICE}_{CONTAINER}_{INSTANCE} moniker.

Pull new images for composed Services.

docker-compose pull

List Running Services.

docker ps

Pull a docker container to use.

docker pull {REPO}/{CONTAINER}:{TAG}

Pulls a copy of the container to local docker image store.
Can store multiple tagged versions of a container.

List containers.

docker container ls -a

-a is used to list non-running containers as well.

Show stored images.

docker images

Remove an image from local storage.

docker rmi {REPO}/{CONTAINER}:{TAG}

Remove a container.

docker rm {NAME}

This removes a created container. It does not remove the image the container is based on.

Open shell on running container.

docker exec -it {NAME} /bin/bash

Get all current configuration settings for a given container.

docker inspect {NAME}

Follow logs for running container.

docker logs -f {NAME}

Update all docker images already downloaded.

docker images | grep -v REPOSITORY | awk '{print $1}' | xargs -L1 docker pull

Stop all docker containers.

docker stop $(docker ps -aq)

Show docker container stats.

docker stats {NAME}

Remove {NAME} to display all running services.

Interactive Docker Shell that Respects Terminal Size 

Dynamically re-size docker container shell terminal to whatever terminal you are using.

0700 user user ~/.bash_profile

docker-shell() {
  sudo docker exec -it -u $2 $1 /bin/bash -c "stty cols $COLUMNS rows $LINES && /bin/bash";
}
export -f docker-shell

usage: docker-shell {INSTANCE} {USER}.

Docker Bridged Adapters

By default Docker will add -P FORWARD DROP rule to iptables to prevent specific exploitation vectors for containers. Unfortunately, this is applied to all interfaces, regardless of whatever interface docker uses; this rule is re-applied everytime the service is started. Iptables by default filters bridged interfaces.

This will result in KVM virtual machines on a system with Docker to not be able to use a Bridge for network communication. As a bridge is a layer 2 device, it really shouldn’t be filtering IP packets anyways. You can just disable bridged adapters from applying the iptables. If you still use the bridge adapter for system traffic, consider munging the filter instead.

Disable IP filtering on bridged interfaces.

echo "0" /proc/sys/net/bridge/bridge-nf-call-iptables
echo "0" /proc/sys/net/bridge/bridge-nf-call-ip6tables
echo "0" /proc/sys/net/bridge/bridge-nf-call-arptables

This will validate bridging is fixed but not persist across reboots.

Update settings for sysctl as well as UFW sysctl:

0644 root root /etc/sysctl.conf

net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0

0644 root root /etc/ufw/sysctl.conf

net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0

There is a longstanding bug bug with sysctl in debian/ubuntu not applying sysctl.conf properly with network settings. This can be resolved using a root cronjob:

sudo crontab -e

@reboot sleep 15; /sbin/sysctl -p

Ensure settings are applied by rebooting and checking settings are set.

reboot
sysctl -a | grep bridge

Compose Containers on Different Networks 

Setup network isolation of compose containers to minimize exposure. By default all containers will end up on the same default network. This enables network isolation of containers.

Create a custom network named custom_net_name on the subnet 172.40.0.0/16 for this compose container. Containers will automatically receive an IP on this network when turning up.

0640 root staff {SERVICE}/docker-compose.yml

networks:
  custom_net_name:
    driver: bridge
    ipam:
      config:
        - subnet: 172.40.0.0/16

Containers can be specified with static IP’s within the compose definition.

0640 root staff {SERVICE}/docker-compose.yml

services:
  my_container:
    ...
    networks:
      custom_net_name:
        ipv4_address: 172.40.1.1

Accessing Networks from Other Compose Containers

Custom networks may be explicitly accessed by other containers (e.g. a reverse proxy) by explicitly defining them within the compose definition.

0640 root staff {SERVICE}/docker-compose.yml

networks:
  custom_net_name:
    external: true
...
services:
  my_proxy:
    networks:
      my_proxy_network:
      custom_net_name:

custom_net_name is a network defined in another container. Once this is added, the proxy container will be able to do DNS resolution of container names as usual, including proxying traffic to that network.

Default Gateway is Not Correct

Docker does not provide a way to set the appropriate default gateway for multi-network containers. This results in non-deterministic source IP routing.

Warning

When a container is connected to multiple networks, its external connectivity is provided via the first non-internal network, in lexical order.

The current fix is to inspect the container and find the first gateway listed in the connected networks. This will be the default gateway for the container.

There is currently no clean way to set a default gateway via compose.

Inspect docker container to show networks and gateway.

docker inspect {NAME}

Forward Traffic via Specific Interfaces

Nginx can forward traffic via specific interfaces for location definitions which may workaround this issue for specific containers.

0640 root staff {SERVICE}/docker-compose.yml

networks:
  custom_net_name:
    external: true
...
services:
  my_proxy:
    networks:
      my_proxy_network:
        ipv4_address: 172.1.1.1
      custom_net_name:
        ipv4_address: 172.2.1.1

custom_net_name is a network defined in another container. Once this is added, the proxy container will be able to do DNS resolution of container names as usual, including proxying traffic to that network.

Use IPv4 address for proxy_bind command for specific nginx locations.

location / {
  proxy_bind 172.2.1.1;
  proxy_pass ...
}

UFW & Docker

Docker manually manipulates iptables to set container rules for expose and ports. Any firewall program currently breaks Docker routing when used.

Danger

Do not enable UFW or other firewalls on Docker host until Docker Bypasses Firewall is resolved.

If UFW has been enabled and Docker services stop responding, fix by:

Restore Docker services after UFW accidentally enabled.

ufw disable
service docker stop
service docker start

Explore Image Filesystem 

Container filesystems can be explored without launching the container by specifying a replacement entrypoint. This is helpful for debugging issues.

Explore container filesystem.

docker run --rm -it --entrypoint=/bin/sh {IMAGE}

--rm will automatically remove the container when finished executing.
-it will launch an interactive terminal.
-entrypoint will override the existing entrypoint for the image.

Copy Data From Container

Files can be copied directly out of containers.

Copy data from docker container.

docker cp {NAME}:/local/container/file .

Tagging Docker Images

Docker images can be tagged with multiple tags for the same image.

Create two tages for the same image.

docker tag {ID} {USER}/{IMAGE}:0.2
docker tag {ID} {USER}/{IMAGE}:latest

Multiple tags during docker image build.

docker build -t {USER}/{IMAGE}:0.2 -t {USER}/{IMAGE}:latest .
docker push {USER}/{IMAGE}

Push Image to Docker Hub

Login to Docker Hub and push via the correct tag.

Login and push image based on tags.

docker login
docker push {USER}/{IMAGE}:{TAG}

Docker Container Not Getting Interrupt Signals

Caused by the container Dockerfile not properly using the Exec specification for the entrypoint script. Exec will hand over the process and enable signals to propagate into the container when docker stop is issued.

Specify entrypoint properly in Dockerfile.

ENTRYPOINT ["/my/entrypoint/script/with/signals"]

GDBus.Error:org.freedesktop.DBus.Error.ServiceUnknown Error 

Docker push requires gnome-keyring to login by default and will fail producing the error: GDBus.Error:org.freedesktop.DBus.Error.ServiceUnknown. This can be bypassed by replacing the functionality with gnupg2 and pass.

Install gnupg2 and pass to prevent error.

apt install gnupg2 pass
docker login
docker push

References

Docker

Install Docker Service

Create A Standalone Container

Compose with Containers

Common Management Tasks

Interactive Docker Shell that Respects Terminal Size

Docker Bridged Adapters

Compose Containers on Different Networks

Accessing Networks from Other Compose Containers

Default Gateway is Not Correct

Forward Traffic via Specific Interfaces

UFW & Docker

Explore Image Filesystem

Copy Data From Container

Tagging Docker Images

Push Image to Docker Hub

Docker Container Not Getting Interrupt Signals

GDBus.Error:org.freedesktop.DBus.Error.ServiceUnknown Error

Compose with Containers 

Interactive Docker Shell that Respects Terminal Size 

Compose Containers on Different Networks 

Explore Image Filesystem 

GDBus.Error:org.freedesktop.DBus.Error.ServiceUnknown Error 