Automated preview environments from microservices in multiple repositories?

Question

I have multiple microservices (with Dockerfiles) in GitHub repositories.

Although I do unit testing on each microservice, I want to spin up all the services as a preview environment for E2E testing (review environment in Heroku lingo).

Locally, I have a docker-compose.yml file that can create this environment. However, I would like to run this in an automated manner for each PR in each repository.

What CI pipelines/tools are there to enable this workflow that supports:

• Creating a new preview environment for each new PR (and destroying it when PR is closed)
• Supports running E2E tests
• Can post test results on the GitHub pull request
• As simple as possible. The services already have self-contained Docker images, so only configuration is needed through environment variables.
• Big bonus if the environment can be accessed with a public URL

Answer 1

I understand you're looking for a way to create automated preview environments for your microservices in multiple repositories. It's great that you're already doing unit testing on each microservice, and now you're looking to run end-to-end (E2E) tests in a preview environment for each new pull request (PR) opened in each repository.

To achieve this, you can use a container orchestration tool like Kubernetes or Docker Swarm to manage the deployment and scaling of your microservices. You can then use a continuous integration (CI) tool like GitHub Actions, GitLab CI/CD, CircleCI, or Jenkins to trigger the creation of preview environments when a new PR is opened.

Your CI pipeline can be configured to build and deploy the microservices using their Docker images and environment variables, and then run E2E tests against the preview environment. Test reporting tools like pytest or JUnit can be used to generate test reports that can be uploaded to the PR on GitHub for easy review.

It's important to store environment variables securely using a secret management tool like HashiCorp Vault or AWS Secrets Manager to prevent sensitive data from being exposed in the code repository or preview environment.

For accessing the preview environment with a public URL, you can use a tool like ngrok or a cloud provider like AWS Elastic Beanstalk or Google App Engine to expose the environment. This makes it easy for developers and stakeholders to access the preview environment for review and testing.

By automating as much as possible using tools that integrate well with each, you can create a streamlined workflow for creating and managing preview environments for your microservices that can help you catch issues early in the development process.