Development Conventions

This section provides general guidelines for development as well as their justification. It also deals with one of the two hardest problems in computer science. Namely, naming.

Note that these are guidelines to facilitate development - if any tooling is to be built around the convention, we'll upgrade it to an internal "standard" which then MUST be followed for applications to function properly.

Standards are marked with a *.

Naming Git Repositories

For all source code stored in Git repositories, services will typically namespace them under organizations, or projects (e.g. GitHub and GitLab respectively). This means we do not need to add additional prefixes for our repositories.

To keep things easy to remember and associate with naming in code, we shall name all repositories based on the abbreviation of the module they're for, and the repository description must continue the full module name at the beginning.

For example, for Unified Notification Hub the repository name shall be:

unh

and a sample description would be:

Unified Notification Hub - supports email, sms, push notifications, and customized notification channels.

Database

Naming schemas

Schema names MUST BE CONFIGURABLE as projects might want to have schemas sit in the same database, and their schema names might clash with ours. Services within our own system might have to share database with other external services as well.

Internally, we ensure all our modules do not have conflicting short names; we'll use these short names for schema names (which should also be the same name as the repository).

Example for a service named Unified Notification Hub, the repository abbreviation and schema will be named unh.

Naming tables

Tables should be named after the entity they represent, in singular form (so a table representing notifications should be called notification)

warning

That's notification WITHOUT the "S"!

Naming views

Views shall be prefixed with v_, this allows us to clearly see which tables are views. Database management UI's would also typically sort tables by name, allowing all views to be grouped together neatly.

v_[view name]

Example:

v_notification_template

note

To be discussed: naming materialized views

Naming association tables

Association tables are commonly used to map the relationship between entities, especially for many-to-many relationships. We'll add two underscores to concatenate the two table names for this:

[table 1]__[table 2]

Example:

user__notification

The rationale for this is that sometimes, tables can be named with two word, for example, user_info, a single underscore would lead to ambiguity in this scenario - user_info_ntofication.

Naming the database user

Direct database access for each service is via a unique database account. We do this instead of having a shared account for the purpose having the ability to apply access control between services. We'll also have to namespace this user account as we might share the database with other external services (as is often the case with projects being deployed into existing infrastructure)

aoh_[repo name]_user

Example:

aoh_unh_user

Mandatory database columns

The following fields (database columns) are mandatory and must be NOT NULLABLE.

Name	Type	Default	Description
`id`	`uuid`	`gen_random_uuid()`	We use a `uuid` as a primary key for all tables as a best practice. Having universally unique synthetic primary keys (as opposed to sequential ints) allows us to avoid a large class of common errors that might arise from unintended references. It is also better from a security perspective. If your module requires human-readable ID's, create another column for it and assign a unique constraint across that column + `tenant_id`.
`created_at`	`timestamp with timezone`	`now()`	The UTC timestamp when this record was first created
`updated_at`	`timestamp with timezone`	`now()`	The UTC timestamp when this record was last updated, use database triggers
`created_by`	`text`		The reference to the user who created this record - do not apply database constraints
`updated_by`	`text`		The reference to the user who last modified this record - do not apply database constraints
`tenant_id`	`text`		The reference to the tenant this row belongs to - do not apply database constraints
`occ_lock`	`int`	`0`	This integer must match on all update queries to ensure the user is not trying to update a row with outdated data (because another user might have already updated it, changing the number). The purpose is for optimistic currency control but you can think of this as a version number.

note

We only apply database constraints (e.g. foreign key constraints) when the reference is within the same service (schema). This is to avoid coupling the services at the database level.

PostgreSQL-specific types and functions

Some of the types and functions (like gen_random_uuid) might be a PostgreSQL-specific function. This will have to vary based on your actual database product - use the appropriate alternatives instead.

To Be Decided:

Discussion required to determine if the soft delete capability is required, and if the implementation should be via extra fields or using archive tables.

Golang Mandatory tables

Modules should be separated by their schema, and within their own schemas, they should have a dedicated module_info table. This table exists for services to be able to query the current state of the schema, as well as any other information that might be useful for services. tenant_id, created_by and updated_by is not required for this table.

Module Info Table:

Table Name

module_info

This table is designed to arbitrary store key values pairs.

Table Schema:

Name	Type	Default	Description
`id`	`uuid`	`gen_random_uuid()`	Mandatory field, see mandatory database colunmns
`created_at`	`timestamp`	`now()`	Mandatory field, see mandatory database colunmns
`updated_at`	`timestamp`	`now()`	Mandatory field, see mandatory database colunmns
`occ_lock`	`int`	`0`	Mandatory field, see mandatory database colunmns
`key`	`text`		The key of the pair - `NOT NULL`, `UNIQUE`
`value`	`text`	`NULL`	The value of the pair
`comment`	`text`	`NULL`	A descriptive comment about the key-value pair

Recommended Initial Data:

Key	Value	Comment	Description
`INITIAL_SCHEMA_VERSION`	`0.0.1`	`initial schema version`	Initial schema version for the service to check against
`CURRENT_SCHEMA_VERSION`	`0.0.1`	`current schema version`	Current schema version for the service to check against
`INITIAL_APPLICATION_VERSION`	`0.0.1`	`initial app version`	Current application version populated by the service if the row doesn't exist
`CURRENT_APPLICATION_VERSION`	`0.0.1`	`current schema version`	Current application version, upserted by the service
`DEPLOYMENT_TIME`	`2024-08-12T18:32:42`	`the time this service was started`	The time when the service was run

Note that the values above will not be accurate when multiple replicas of a service is being run. In such a scenario, Kubernetes will have to be the source of truth. Otherwise, these fields are useful for the developers during development time - the particularly important keys are the SCHEMA_VERSION, which must be checked against before the application proceeds.

API, Routes, and Endpoints

Kubernetes Liveness and Readiness probes

All services must provide a liveness and readiness endpoint for Kubernetes to call. As a rule, we'll use livez and readyz at the root path, which is common with Google's internal practices. However, certain frameworks might already provide such endpoints for the same purpose, requiring you to utilize their naming scheme; in such a scenario, you should stick with the framework's built-in endpoints (e.g. SpringBoot's /actuator/health/liveness & /actuator/health/readiness).

Liveness Endpoint: /livez
Readiness Endpoint: /readyz

Web Pages

The routing of web pages shall also be namespaced - first by the project, then by the module:

/[project]/[module]/...

Examples:

View all incidents page: /aoh/incidents/
View specific incident: /aoh/incidents/inc-20240607-0001
Dashboard page: /aoh/dashboard?name=My+First+Dashboard

Data Response Format

The following is a recommended response format for all your API responses. These are general guidelines and might not apply to all cases. For example, this should be the structur of your HTTP response body, or messages sent through Kafka etc.

Key	Type	Optional	Description
`data`	`object` or `array`	`yes`	The actual response data for the request.
`message`	`string`	`yes`	An accompanying message for additional information or debugging.
`sent_at`	`string` formatted as `ISO8601`	`yes`	The time this message was sent.
`errors`	`array` of `{ message: string }`	`yes`	An aggregation of errors if error feedback is necessary.

Example response body
{
    data: {...}, // arbitrary format - recommend array for lists, object for individual records
    message: "...", // string
    sent_at: "", //iso8601
    errors: [ {
        message: "....", // string
        ...
    } ]
}

Pagination

Querying for entities almost always requires pagination to pull data effectively (applications should almost never pull an entire table of data).

Since pagination is a very common requirement, and different frameworks in different languages might support different out-of-the-box implementations of pagination, we have to allow for some API to be flexible. However, the guidelines to follow for paginated endpoints should be:

Pagination arguments should be URL query params
Requests: page, size and sort should be supported
- page specifies what page the caller is currently at (the offset cursor) - only positive integers starting from 1
- size specifies how many rows are in each page - only positive integers starting from 1
- sort is a list of tuples containing field and direction
  - The field refers to the column to sort by
  - The direction refers to whether the sort is ascending or descending
    - ASC for ascending
    - DESC for descending (default)
Response: data, page: { number }, page: { size }, and page: { total_records } information should be returned
- data is the result of the query
- number is the current page of response
- size is the number of records per page
- total_records is the total number of records in the table
- count is an optional field, specifying the number of elements in the current page
- sort is an optional field, specifying the sort parameters used for to retrieve current page

Request Pagination Params

Name	Type	Default	Description
`page`	`int`	`1`	The number of elements in each page. Invalid values will be ignored and default will be used
`size`	`int`	`10`	The current page - pages start at 1. Invalid values will be ignored and default will be used
`sort`	`string,string`	`created_at,DESC`	A list of sort columns and direction. Invalid values will be ignored and default will be used

Response Pagination Data

Name	Type	Optional	Description
`page: { number }`	`int`	`no`	The current page - pages start at 1
`page: { size }`	`int`	`no`	The number of elements in `each` page
`page: { total_records }`	`int`	`no`	The number of records in each page
`page: { count }`	`int`	`yes`	The number of elements in the `current` page
`page: { sort }`	`string,string`	`yes`	The sort column and direction

These fields need not strictly follow the same name (due to possible framework limitations), but the pagination API should follow the specs listed above. The page number, page size, and total_records are typically required by the caller in order to understand where the cursor is in relation to the rest of the table. If the API allows specifying the sort parameters, sort should be returned as well.

Example paginated API call:

Example request - page 1, default page size, and default sort
example.agilopshub.com/user

Example request - page 3, 2 records per page, default sort
example.agilopshub.com/user?page=3&size=2

Example request - page 3, 2 records per page, sort by username, descending
example.agilopshub.com/user?&page=3&size=2&sort=username

Example request - page 3, 2 records per page, sort by email - descending, then sort by username - ascending
example.agilopshub.com/user?page=3&size=2&sort=email,desc&sort=username,asc

Example response
{
    "data": [
        {
            "username": "coolguy",
            "email": "iamcool@gmail.com"
        },
        {
            "username": "example",
            "email": "example@gmail.com"
        },
    ],
    "page": {
        "number": 3,
        "size": 2,
        "total_records": 35,
        "sort": "username,asc"
    }
    ...
}

Example response 2
{
    "data": [
        {
            "username": "a coolguy",
            "email": "zzz@gmail.com"
        },
    ],
    "page": {
        "number": 4,
        "size": 3,
        "total_records": 10,
        "count": 1,
        "sort": [
            "username,desc",
            "email,asc"
        ]
    }
    ...
}

Log levels

Logging is covered in the Logging & Exception Handling guidelines section.

As a quick rule of thumb, use DEBUG log level to trace inputs (and potentially outputs) to function calls that are useful for debugging the system, and leave these logs there. For INFO log level, use it to trace key events in the system to indicate it is functionally correctly (such as a successful database connection event).

Container Images

Adding container labels

Users might often need to pull, scan and retag containers, and upload them to their own container registry for their own production environment. To avoid confusion on what version of code each container contains, we add a revision annotation in each container, along with a few other important information, including and especially the URL.

LABEL org.opencontainers.image.source REPOSITORY_URL
LABEL org.opencontainers.image.title MODULE_TITLE
LABEL org.opencontainers.image.description DESCRIPTION
LABEL org.opencontainers.image.authors AUTHORS
LABEL org.opencontainers.image.revision REVISION

Replace REPOSITORY_URL with the URL of the source code repository.
Replace MODULE_TITLE with the full name of the module.
Replace DESCRIPTION with a brief description of the container's purpose in the module.
Replace AUTHORS with links to authors involved in creating this container.
Replace REVISION the revision which should be passed in as a build ARG in the CI

For the REVISION, we use the [branch]-[commithash] , for example: develop-c3h0c69. This, combined with the URL, allows us to track where the original code for this container resides.

note

These labels are official annotations defined by the OCI annotations specification.

Tagging

Our standard for tagging built container images follows a version number it is working towards, followed by an optional short tag to represent its purpose, followed by a dash and the git commit hash of the source repo:

Container Tag Regex
/v[\d]+.[\d]+.[\d]+(dev|beta|rc)-[\da-f]{7}/

Example Tag:

v1.12.0rc-710093d

The latest release image must also always be tagged with latest.

latest

The latest development release image must also always be tagged with latest-dev.

latest-dev

The images should also follow their repository structure - because we use monorepos and keep source code for each repository together, their tags should include the short name of the module, followed the short name of the module again, and the sub-name for the container in the module. This is required so that GitHub can properly segregate different containers linked to the same repository.

Example latest-dev tags for the IMS module's app and web containers:

Latest Development images for the IMS module
ghcr.io/mssfoobar/ims/ims-app:latest-dev
ghcr.io/mssfoobar/ims/ims-web:latest-dev

Full example images and their tags:

Latest
ghcr.io/mssfoobar/unh/unh-app:latest

Latest Development Image
ghcr.io/mssfoobar/unh/unh-app:latest-dev

Released v3.0.1
ghcr.io/mssfoobar/unh/unh-app:v3.0.1

Release candidate for v4.0.0
ghcr.io/mssfoobar/unh/unh-app:v4.0.0-78cdaeb

Refer to the post on versioning scheme for more information on the version numbers.

Naming Git Repositories​

Database​

Naming schemas​

Naming tables​

Naming views​

Naming association tables​

Naming the database user​

Mandatory database columns​

Golang Mandatory tables​

Module Info Table:​

Table Schema:​

Recommended Initial Data:​

API, Routes, and Endpoints​

Kubernetes Liveness and Readiness probes​

Web Pages​

Data Response Format​

Pagination​

Request Pagination Params​

Response Pagination Data​

Log levels​

Container Images​

Adding container labels​

Tagging​

Naming Git Repositories

Database

Naming schemas

Naming tables

Naming views

Naming association tables

Naming the database user

Mandatory database columns

Golang Mandatory tables

Module Info Table:

Table Schema:

Recommended Initial Data:

API, Routes, and Endpoints

Kubernetes Liveness and Readiness probes

Web Pages

Data Response Format

Pagination

Request Pagination Params

Response Pagination Data

Log levels

Container Images

Adding container labels

Tagging