Local Development Environment

AGIL Ops Hub maintains a set of docker-compose configuration files in the dev-containers repository. If you wish to run any module locally, the easiest way to get all their dependencies up and running is by using those compose files.

The following document explains how you can use the dev-containers repository to quickly get a local development environment up and running.

info

We use Podman Desktop because it works across Windows, macOS, and Linux, and its free - so the rest of the instructions here will be with podman. You can replace docker... with podman... if you're using Docker.

note

The Docker Compose files in dev-containers are designed to pull the container images from AGIL Ops Hub private GitHub container registry. If you have move the container images to your own project Container Registry, you will need to modify the Docker Compose files correspondingly.

Basically, you will need to change all the image field to point to your Container Registry instead of ghcr.io/mssfoobar

Guide

1. Clone the repository

1.1 - Clone the git repository

git clone https://github.com/mssfoobar/dev-containers

1.2 - Navigate into the root folder

cd dev-containers

2. Set up the environment variables

By default, the environment variable file already contains all the default values you need to get up and running except for the DEV_PASSWORD & DEV_PASSWORD_URL_ENCODED. Choose a good password and fill them in accordingly.

tip

DEV_PASSWORD_URL_ENCODED is required for services that need to use a database connection string with credentials to establish connections. Some passwords might have special symbols in them, such as "@", causing database connection strings to fail. For example, if your DEV_PASSWORD is "strong_p@ssword", in DEV_PASSWORD_URL_ENCODED, it must be "strong_p%40ssword".

.env.template

# Root domain to use for all services
DEV_DOMAIN=127.0.0.1.nip.io

# Global username and password used for everything
DEV_USER=admin
DEV_PASSWORD=

# Global username and password used for everything - but URL encoded
DEV_USER_URL_ENCODED=admin
DEV_PASSWORD_URL_ENCODED=

2.1 - Make a copy of the file and rename it to .env

cp .env.template .env

2.2 - Choose a suitable password and set it for both variables.

.env

# Root domain to use for all services
DEV_DOMAIN=127.0.0.1.nip.io

# Global username and password used for everything
DEV_USER=admin
DEV_PASSWORD=myfavp@ssword

# Global username and password used for everything - but URL encoded
DEV_USER_URL_ENCODED=admin
DEV_PASSWORD_URL_ENCODED=myfavp%40ssword

3. Run the command for the module you want

Every compose.yml file in each folder is designed to already include the services they need, simply start it up with the .env file and apply any overrides you need:

3.1 - Example, for the IAN module, you can run it like so.

podman compose --env-file .env -f ian/compose.yml up -d

tip

An compose.override.sample.yml exists which as a sample compose override file you can use to fill in configuration overrides (such as for port mapping to directly access the databases with your database management client).

cp compose.override.sample.yml override.yml

podman compose --env-file .env -f ian/compose.yml -f ./override.yml up -d

warning

For Docker users, you must use the override to override the default traefik settings as it is assuming you are using Podman - the volumes: entry is different to bind to the appropriate docker.sock.

In the future, we intend to provide tooling with this information integrated. For now, each module states all their dependencies in their respective folders, so if you want to include the GIS (Geospatial Information Services) module, look at the README.md in it's folder and run that command.

4. To stop the module

To stop the module, simply execute the command in step 3 with the up -d replaced with down.

For example, to stop the running IAN module, execute the following:

podman compose --env-file .env -f ian/compose.yml down

Additional Configuration

Wildcard DNS vs Host file

Custom hostname resolution is required for proper development - the system relies on cookies being forwarded to the front-facing servers for authentication.

You can use a wildcard DNS service like https://nip.io or modify your /etc/hosts file.

Wildcard DNS (nip.io)

If you're using nip.io, set the DEV_DOMAIN environment variable to get nip.io to resolve it to the loopback address:

DEV_DOMAIN=127.0.0.1.nip.io

For your web services, make sure the PUBLIC_DOMAIN is set appropriately (to nip.io)

PUBLIC_DOMAIN=nip.io

This requires no additional changes to your system - you may then access services via their appropriate subdomain, for example, to visit the Keycloak admin site, you will go to http://iams-keycloak.127.0.0.1.nip.io

Host file

Alternatively, you can modify your hosts file to allow your machine to resolve the addresses appropriately, you can do this by adding the values below to your hosts file in C:\Windows\System32\Drivers\etc\hosts in Windows, or /etc/hosts in Unix systems.

## -------------------- AOH Local Development --------------------
## ========== Traefik ==========
127.0.0.1 traefik.localhost

## ========== IAMS ==========
127.0.0.1 iams-keycloak.localhost iams-aas.localhost iams-web.localhost iams-db.localhost

## ========== Web Base ==========
127.0.0.1 web-base.localhost

## ========== RTUS ==========
127.0.0.1 rtus-seh.localhost rtus-pms.localhost rtus-db.localhost

## ========== TAG ==========
127.0.0.1 tag.localhost tag-web.localhost tag-db.localhost

## ========== UNH ==========
127.0.0.1 unh.localhost unh-web.localhost unh-db.localhost

## ========== IAN ==========
127.0.0.1 ian.localhost ian-web.localhost ian-db.localhost

## ========== DASH ==========
127.0.0.1 dash.localhost dash-web.localhost dash-db.localhost

## ========== GIS ==========
127.0.0.1 gis.localhost gis-web.localhost gis-db.localhost

## ========== WFE ==========
127.0.0.1 wfe.localhost wfe-web.localhost wfe-db.localhost

For your web services, make sure the PUBLIC_DOMAIN is left blank, this is a mandatory cookie setting for browsers when the domain is meant to be localhost, you can use the compose.override.yml to achieve this.

name: aoh

services:
    iams-web:
        environment:
            - PUBLIC_DOMAIN=

    ...

Using a Local Configuration File for Container Settings

For greater development flexibility, we provide a customizable YAML file that allows you to adjust certain container settings. This file is intended for your personal development environment and is not part of the project's shared configuration. This file will overwrite any specified parameters that is in the ./ims/compose.yml

To customize, please modify the following file with your desired IDE

vi ./overwrite.yml

Below is an snippet of overwrite.yml

<module>-db:
    ports:
        - 5432:5432

For this specific scenario, we are adjusting the ports parameter to redirect port 5432 in ./ims/compose.yml

Database Volumes

By default, the dev-containers repository create named volumes to store the database data files.

To view the volumes created, you can execute the following command:

podman volume ls

To reset the database, simplest way is to shutdown the container, remove the correspondingly named volume, and restart the container. The Docker Compose file will reinitialize the database.

To delete the volume, you can execute the following commands:

$ podman volume rm volume-name

• Replace volume-name with the volume name obtained to previous step

for example, to delete the volume for IAMS database, execute the followings:

podman volume rm aoh_iams-db-volume

After deleted the volume, you should run the following command to clean up your development environment and reclaim storage space. It removes all unused volumes not associated with containers, including those that Docker Compose creates.

podman volume prune

Troubleshooting

Updating Container Images

By default, the dev-containers repository uses latest-dev for all our tags. Docker/Podman will not automatically pull the newest image as we're re-using the 'latest-dev' tag. You'll have to either delete the image locally or manually pull the latest[-dev] tagged image to receive the newest updated images.

Changing Env Vars

The env vars are used as part of the initialization process - it includes generating users, setting credentials, etc. and all these are persisted in the .data folder as it is used in the Docker volumes. If you need to change any of the environment variables (DEV_DOMAIN, DEV_USER or DEV_PASSWORD), do not expect the values to be reflected without re-initializing everything (you will need to delete .data and restart everything).

Unable to login to Keycloak

Since Keycloak 26, the KEYCLOAK_ADMIN and KEYCLOAK_PASSWORD have been deprecated and replaced with KC_BOOTSTRAP_ADMIN_USERNAME and KC_BOOTSTRAP_ADMIN_PASSWORD. If you are having difficulties logging in, you might be using an older container.

Podman

Credential issues when migrating from Docker

There is a common issue you might face when Docker was installed and you are switching over to podman. If you are able to pull the containers manually, but not when you run podman compose ... please visit the following link to see how you can resolve it: https://github.com/containers/podman/issues/22682

Traefik

For Traefik to perform routing, it needs access to the Docker API via the docker.sock, for Podman, that is available at /run/podman/podman.sock by default. If you're having problems with Traefik look for the remoteSocket config:

podman info

remoteSocket:
    exists: true
    path: /run/podman/podman.sock

Under "path": You might see something other than /run/podman/podman.sock, in that scenario, use the override.yml file to modify it appropriately:

cp override.sample.yml override.yml

## Use this override file to override any default configuration
## Example Use Case: Forward ports to a database you want to access
name: aoh

services:
    traefik:
        volumes:
            - /run/user/example/podman.sock:/var/run/docker.sock

Then, run the command with the module you want and try again:

podman compose --env-file .env -f example_module/compose.yml -f override.yml