I'm looking for advice on what the best practice is for implementing a containerized CI/CD environment within GitHub Actions, which supports Windows, MacOS & Linux (Ubuntu) environments, used for building & testing a large amount of Java projects. Ideally something along the lines of:

  1. A PR/merge is made
  2. Whatever jobs required by the GitHub Actions workflows are triggered, inside a fresh container (that's either Windows, MacOS or Ubuntu depending on what the job needs) that's created just for the job.
  3. When the job finishes, the container is destroyed. Maybe with a limit on the maximum amount containers can run at the same time.

Our CI system currently uses a bunch of Ubuntu & Windows Virtual Machines (hosted on Azure, which frankly is proving a pain as we often get random network errors sometimes, causing maven dependcies to be mis-downloaded/corrupted, which causes maven to randomly crash) configured upon startup to register themselves as a self-hosted runner within GitHub Actions, as well as a handful of physical Mac Minis hosted in our office which are configured as self-hosted runners also.

We can't use github-hosted runners because of the cost, and the fact that they don't go as powerful as we need. Our CI system includes a lot of end-to-end tests across Windows, Mac & Ubuntu, in a variety of configurations, each of which require at least 16GB of RAM and 8 vCPUs. Each CI run will on average tend to use around 20-30 of these jobs, running anywhere from 15 to 90 minutes.

Because we're currently hosting these Windows & Ubuntu virtual machines as scale sets (allowing us to use a dedicated image with any dependencies preloaded), we can use some pretty basic scheduling to reduce the amount of VMs at night, yet increase them during the day when people are working, however this is inconvenient and often leads to VMs being destroyed at night in the middle of a job. So something ephemeral would be better for us.

GitHub Actions does have support for ephemeral runners now, which seems like what we'd need, and in conjunction with a suitable docker image for each OS and a Kubernetes cluster to run them on, seems like the direction we should be moving towards. This would be great if we just needed to do our end-to-end testing on Linux, however i'm not sure how well docker supports emulating Windows & MacOS, and I would like some advice on what would be the best approach here.

We did consider moving towards Azure DevOps Pipeline entirely for our CI, as that works better with dynamically-scaling on Azure's scale sets, however they informed us we couldn't really have more than 20 simultaenous jobs running at once across our entire CI pipeline, which when you consider we may have 10 seperate branches running CI, each of which uses a minumum of 10 jobs, wasn't really an option for us. Hosting CI infrastructure on codebuild isn't an option for us either unfortunately.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy