Here are some details about the workflow.

1. The developer creates a Pull Request(PR) and attaches a "preview" label to it. This label is customizable.

2. PR creation triggers the Continuous Integration build process using GitHub Actions. A Docker image artifact is created and pushed to the AWS Elastic Container Registry.

3. The ArgoCD ApplicationSet controller detects the PR and triggers the application deployment in a separate namespace.

4. In this namespace, ArgoCD provisions the service-specific Kubernetes resources such as - Ingress, Service, and Deployment.

5. The developers and QA team can access the Preview environment via the Ingress URL.

6. Once the PR is closed/merged, or the "preview" label is removed, ArgoCD automatically removes all the resources of the Preview environment.

ArgoCD provides a general framework for creating Preview environments. However, here are some challenges we had to solve when building Preview environments on top of ArgoCD.

• Dependency Management for Services

• Seed Data Management

• Keeping costs low for Preview environments

Dependency Management for Services

For frontend services, setting up Preview environments is easy. For backend services, we have to also set up its dependent components such as database, queue, cache, and other backend services.

For this, we chose a hybrid approach where the database, queue, and cache were provisioned specifically for the Preview environment in the same namespace. The dependent backend services were used from the shared Staging environment. This allowed us to save the resources cost on shared backend services yet not share the database, queue, and cache.

We introduced configuration overrides to solve an interesting scenario. Consider two services - Service A and Service B, where Service A depends on Service B. A developer working on a Pull Request of Service A, can use Service B either from the Staging environment or from any Preview environment of Service B. We enabled developers to point to any version of Service B (either Staging or another Preview environment) by changing the configuration in ArgoCD.

Seed Data Management

Each backend service in the Preview environment has its own newly created database. To test the service in Preview environment, the team had to create and update database records. This was time-consuming. We solved this by creating customized Docker images with pre-seeded data. We also made it easy for developers to update the seed data by replacing a dump file in a Git repository. This allowed us to version control the seed data as well. With this, the team would not have to start from scratch, effectively reducing the time to test the Preview environment.

Keeping costs low for Preview environments

As Preview environments are created for every Pull Request with a specific label, it can quickly grow into many environments being provisioned. To ensure this doesn’t add up to too much infrastructure cost, we suggested running the Preview environments using spot instances in Kubernetes. Spot instances are perfect for these non-critical workloads, like Preview environments.

If a Pull Request doesn’t have any new activity (commits, comments, etc.) for a certain period of time, the preview label will automatically be removed. This, in turn, would delete the Preview environment to save costs.

In summary, we were able to roll out Preview environments for all of Manatal’s services (both frontend and backend) on their existing tech stack. All this work was carried out by just one senior engineer in less than 2 months.