Moving from Vagrant to Docker and Docker Compose: Docker compose

On the last part , i talked about my try using Kubernetes to orchestrate our local development environment and how it was no good at least at this stage since there was a lot of problems related to mounted volumes, and how pods are communicating so it made more sense to switch to a simpler orchestrator which is Docker Compose.

The compose file

version: "3.2"
services:
  mongo:
    container_name: mg_mongo
    build:
      context: ./config/mongo/
    restart: always
    networks:
      default:
        aliases:
          - mrgeek_mongo
    ports:
      - "27017:27017"
    volumes:
      # Mongo config
      - ./config/mongo/mongod.conf:/etc/mongod.conf
      # Mongo data to be persistent between reboots
      - mongo-data:/data/db


  rabbitmq:
    container_name: mg_rabbitmq
    build:
      context: ./config/rabbitmq/
    restart: always
    networks:
      default:
        aliases:
          - mrgeek_rabbitmq
    ports:
      - "5672:5672"
      - "15672:15672"
    volumes:
      # RabbitMQ configuration
      - ./config/rabbitmq/:/etc/rabbitmq/

  elasticsearch:
    container_name: mg_elasticsearch
    build:
      context: ./config/elasticsearch/
    restart: unless-stopped
    networks:
      default:
        aliases:
          - mrgeek_elasticsearch
    ports:
      - "9200:9200"
    volumes:
      # Elasticsearch data
      - elastic-data:/var/lib/elasticsearch
      # Elasticsearch configuration files
      - ./config/elasticsearch/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
      - ./config/elasticsearch/jvm.options:/usr/share/elasticsearch/config/jvm.options

  kibana:
    container_name: mg_kibana
    build:
      context: ./config/kibana/
    restart: unless-stopped
    depends_on:
      - elasticsearch
    environment:
      ELASTICSEARCH_URL: http://mrgeek_elasticsearch:9200
      ELASTICSEARCH_HOSTS: http://mrgeek_elasticsearch:9200
    ports:
      - "5601:5601"
    volumes:
      # Kibana configuration files
      - ./config/kibana/kibana.yml:/usr/share/kibana/config/kibana.yml
      - ./config/kibana/node.options:/usr/share/kibana/config/node.options
      - ./config/nginx/ssl/:/usr/share/kibana/config/ssl/

  main:
    container_name: mg_main
    build:
      # define parent directory as context so we can get things from other
      # repositories
      context: ./../
      dockerfile: ./mg_docker/Dockerfile
    depends_on:
      - mongo
      - rabbitmq
      - elasticsearch
    ports:
      - "443:443"
      - "80:80" 
      - "5222:5222" 
      - "22:22"
    extra_hosts:
      # extra hosts
      - "mrgeek_api:127.0.0.1"
      - "mrgeek_xmpp:127.0.0.1"
      - "mrgeek_file_upload:127.0.0.1"
      - "mrgeek_reporting:127.0.0.1"
      - "host.docker.internal:host-gateway"
    volumes:
      # Bashrc
      - ./config/.bashrc:/root/.bashrc

      # dnsmasq.conf
      - ./config/dnsmasq/dnsmasq.conf:/etc/dnsmasq.conf

      # Nginx configuration from our nginx-edge repo
      - ./config/nginx/ssl/:/etc/nginx/ssl/
      - ./../mg-nginx/src/conf.d/:/etc/nginx/conf.d/

      # Kibana configuration
      - ./config/kibana/:/etc/kibana/

      # Ejabberd
      - ./config/ejabberd/conf/:/opt/ejabberd/conf/
      - ejabberd-data:/opt/ejabberd/database/
      - ./config/ejabberd/ssl/:/opt/ejabberd/ssl/

      # PHP fpm conf
      - ./config/api/php-fpm.conf:/etc/php/fpm/php-fpm.conf
      - ./config/api/www.conf:/etc/php/fpm/pool.d/www.conf
      - ./config/api/opcache.ini:/etc/php/fpm/conf.d/05-opcache.ini
      - ./config/api/php.ini:/etc/php/fpm/php.ini
      - ./config/api/php.ini:/etc/php/cli/php.ini

      #
      # Main services
      #

      # api
      - ./../mg_api/:/var/www
      - ./../mg_api/composer.lock:/var/www/composer.lock

      # mg_xmpp_node
      - ./../mg_xmpp/:/var/app/mg_xmpp/

      # mg_reporting
      - ./../mg_reporting/:/var/app/mg_reporting/

      # mg_file_upload
      - ./../mg_file_upload/:/var/app/mg_file_upload/

volumes:
  mongo-data:
  elastic-data:
  ejabberd-data:

The first 4 services have common attributes as you can see all of their config we get from a ./config directory were it has sub-directories for each service it has its own configuration files plus its own Dockerfile as you can see here:

mg_docker/
├─ config/
│  ├─ api/
│  │  ├─ php-fpm.conf
│  │  ├─ www.conf
│  │  ├─ opcache.ini
│  │  ├─ php.ini
│  ├─ dnsmasq/
│  │  ├─ dnsmasq.conf
│  ├─ elasticsearch/
│  │  ├─ elasticsearch.yml
│  │  ├─ jvm.options
│  │  ├─ Dockerfile
│  ├─ kibana/
│  │  ├─ Dockerfile
│  │  ├─ kibana.yml
│  │  ├─ node.options
│  ├─ ejabberd/
│  │  ├─ conf/
│  │  │  ├─ ejabberd.yml
│  │  │  ├─ ejabberdctl.cfg
│  ├─ rabbitmq/
│  │  ├─ Dockerfile
│  │  ├─ enabled_plugins
│  ├─ mongo/
│  │  ├─ Dockerfile
│  │  ├─ mongod.conf
├─ Dockerfile
├─ docker-compose.yml
├─ docker-entrypoint.sh
mg_api/
mg_xmpp/
mg_reporting/
mg_file_upload/

Important thing to note here is the networks aliases for each service because that's how the services are going to communicate between each other, a good example for this is elasticsearch container aliases:

    networks:
      default:
        aliases:
          - mrgeek_elasticsearch

and we're using this alias in kibana container passing it to it's configuration as environment variables

    environment:
      ELASTICSEARCH_URL: http://mrgeek_elasticsearch:9200
      ELASTICSEARCH_HOSTS: http://mrgeek_elasticsearch:9200

Now for the main service it has more configuration as you can see, but good thing to notice is that the context is set with context: ./../ and since the context is changed we're referring to the docker file with this dockerfile: ./mg_docker/Dockerfile

Context is changed so we can refer to other services which exist on the parent directory of this one ../

And we're also setting the extra hosts so that services can communicate to each other inside the same container

    extra_hosts:
      # extra hosts
      - "mrgeek_api:127.0.0.1"
      - "mrgeek_xmpp:127.0.0.1"
      - "mrgeek_file_upload:127.0.0.1"
      - "mrgeek_reporting:127.0.0.1"
      - "host.docker.internal:host-gateway"

Also we're setting the volumes as well where you can see we're mounting other services

dnsmasq is used to act as a DHCP server inside the container so we can reference these extra hosts inside the container

The docker files

Docker files for our services are really simple and they actually might be redundant and we can even use docker compose directly for this, but for better and clear control we went with docker files

The one hell of a Dockerfile that we rely on is the main docker file for our main service which has:

Nginx
PHP & PHP-FPM
Ejabberd
NodeJs

It is a big one i know, and later on this one will be splitted into multiple services of course, so here is it:

FROM debian:10

###############################################################################
# Install basic software
#
RUN set -x \
    && apt-get update \
    && apt-get install -y --no-install-recommends \
        systemd \
        sudo \
        build-essential \
        nano \
        openssh-server \
        apt-transport-https \
        ca-certificates \
        gnupg \
        curl \
        software-properties-common \
        locales \
        python-pip \
        python-setuptools \
        python \
        unzip

###############################################################################
# Install nginx
#
RUN curl -fsSL https://nginx.org/keys/nginx_signing.key | apt-key add -

RUN add-apt-repository 'deb https://nginx.org/packages/mainline/debian/ buster nginx'

RUN set -x \
    && apt-get update \
    && apt-get install -y nginx \
    && rm -rf /var/lib/apt/lists/*

###############################################################################
# Install ejabberd
#
ENV EJABBERD_HOME=/opt/ejabberd
ENV EJABBERD_CTL=/opt/ejabberd/bin/ejabberdctl
ENV NGINX_INTERNAL_SERVICE_HOST=localhost

COPY ./mg_ejabberd/ejabberd.deb /root/ejabberd.deb
RUN dpkg -i /root/ejabberd.deb

RUN mkdir -p /var/log/ejabberd
RUN touch /var/log/ejabberd/extauth.log
RUN chown -R ejabberd:ejabberd /var/log/ejabberd
RUN chown -R ejabberd:ejabberd ${EJABBERD_HOME}/

###############################################################################
# Install dnsmasq for nginx resolver
#
RUN apt-get update
RUN apt-get install -y dnsmasq dnsutils net-tools


###############################################################################
# Install php, php-fpm and composer
#
RUN apt-get update
RUN apt-get install -y \
      php-cli \
      php-fpm \
      php-xml \
      php-bcmath \
      php-mbstring \
      php-gd \
      php-curl \
      php-mongodb \
      php-pear \
      php-dev \
      php-xdebug \
      php-apcu

# Setup xdebug
COPY ./mg_docker/config/api/xdebug.ini /etc/php/mods-available/xdebug.ini

# Install mongo extenstion
RUN pecl install mongodb

RUN curl -sS https://getcomposer.org/installer -o /root/composer-setup.php
RUN php /root/composer-setup.php --install-dir=/usr/local/bin --filename=composer

RUN echo "extension=mongodb.so" > /etc/php/cli/conf.d/20-mongodb.ini
RUN echo "extension=mongodb.so" >/etc/php/fpm/conf.d/20-mongodb.ini

###############################################################################
# Some cleanup to reduce space
#
RUN apt-get clean && apt-get autoremove
RUN rm -rf /root/ejabberd.deb
RUN rm -rf /root/composer-setup.php

###############################################################################
# Nginx main configuration
#
COPY ./mg-nginx/src/nginx.conf /etc/nginx/nginx.conf

ARG USER_ID=1000
ARG GROUP_ID=1000

# Remove www-data user
RUN userdel -f www-data &&\
    if getent group www-data ; then groupdel www-data; fi

# Generate a new www-data user with IDs 1000
# Will be used by API to create cache & logs
RUN groupadd -g ${GROUP_ID} www-data
RUN useradd -l -u ${USER_ID} -g www-data www-data
RUN usermod -G sudo www-data
RUN install -d -m 0755 -o www-data -g www-data /home/www-data /var/run/php /var/run/nginx
RUN chown --changes --no-dereference --recursive \
        ${USER_ID}:${GROUP_ID} \
        /home/www-data \
        /var/lib/php/sessions \
        /etc/nginx /var/run/nginx /var/run/php \
        /var/cache/nginx \
        /var/log/nginx \
        /var/run/php \
        /var/run/nginx \
        /etc/php/
RUN sed  -i 's/^\%sudo.*/%sudo   ALL=(ALL) NOPASSWD:ALL/g' /etc/sudoers

###############################################################################
# Entrypoint
#
COPY ./mg_docker/docker-entrypoint.sh /tmp/docker-entrypoint.sh
RUN chmod +x /tmp/docker-entrypoint.sh
USER www-data

###############################################################################
# Install nvm and node v14.15.4 and npm v6.14.10
#
ENV NODE_VERSION=14.15.4
RUN curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.37.2/install.sh | bash
ENV NVM_DIR=/home/www-data/.nvm
RUN . "$NVM_DIR/nvm.sh" && nvm install ${NODE_VERSION}
RUN . "$NVM_DIR/nvm.sh" && nvm use v${NODE_VERSION}
RUN . "$NVM_DIR/nvm.sh" && nvm alias default v${NODE_VERSION}
ENV PATH="/home/www-data/.nvm/versions/node/v${NODE_VERSION}/bin/:${PATH}"
RUN node --version
RUN npm --version


###############################################################################
# Install pm2
#
RUN npm install -g pm2

ENTRYPOINT ["./tmp/docker-entrypoint.sh"]

###############################################################################
# Expose needed ports
#
EXPOSE 433 80 5222 22

STOPSIGNAL SIGQUIT

CMD ["nginx"]

It's pretty straight forward docker file the only thing to note is this part

ARG USER_ID=1000
ARG GROUP_ID=1000

# Remove www-data user
RUN userdel -f www-data &&\
    if getent group www-data ; then groupdel www-data; fi

# Generate a new www-data user with IDs 1000
# Will be used by API to create cache & logs
RUN groupadd -g ${GROUP_ID} www-data
RUN useradd -l -u ${USER_ID} -g www-data www-data
RUN usermod -G sudo www-data
RUN install -d -m 0755 -o www-data -g www-data /home/www-data /var/run/php /var/run/nginx
RUN chown --changes --no-dereference --recursive \
        ${USER_ID}:${GROUP_ID} \
        /home/www-data \
        /var/lib/php/sessions \
        /etc/nginx /var/run/nginx /var/run/php \
        /var/cache/nginx \
        /var/log/nginx \
        /var/run/php \
        /var/run/nginx \
        /etc/php/
RUN sed  -i 's/^\%sudo.*/%sudo   ALL=(ALL) NOPASSWD:ALL/g' /etc/sudoers

Here we're creating new www-data user with the same UID and GID as the host user which are both 1000 then we're changing the owner of most of services' directories to the new user, so whenever for example PHP is generating logs, which are mounted on the machine, host user can still control them without using sudo

This is also used to run our docker entrypoint as you can see

COPY ./mg_docker/docker-entrypoint.sh /tmp/docker-entrypoint.sh
RUN chmod +x /tmp/docker-entrypoint.sh
ENTRYPOINT ["./tmp/docker-entrypoint.sh"]
`

In which we're running all the services of this container.

This way it was easier to mount files, and even the mounted volumes will have the same exact user as on the host machine. So we will never have to use sudo to remove a directory for example or change a file.

Final thoughts

It was quite a journey, which is still ongoing BTW but to reach this point where at least the software is up & running and is development friendly is quite amazing

The directory structure we have plays an important role to have this working, where each service is a repository on its own, and exists on the parent directory of this docker project, that's why it was easier to reuse them in docker containers

DevOps indeed helped and fixed some hard issues, and i learned a lot by just touching this bare topic, i was exposed to so many technologies so deep more than i would ever dreamed.

Moving from Vagrant to Docker & Docker Compose: Docker compose

Finally making it work with Docker Compose

The compose file

The docker files

Final thoughts