I am working on a product where we have 3 dev teams. 1 for UI and 2 for backend. Each team requires a complete env for their own testing. There 10+ backend services and many more other components and 4 UI components. We have following env: 3 Dev env one for each teams 1 Dev Integration 2 for QA (Automation and manual teasting)
All env are using Kubernetes, all services deployed as containers. Each env having their own DB server. 3 dev env and 1 dev int environment are deployed on single K8s Cluster and 2 QA env deployed on another cluster.
Each env has its own set of jenkins jobs for all modules (Both CI and CD). So we have total 6 copies of all jobs in various jenkins folders (1 for each env). They all use common svc and dpl file (Stored as template) from our devops repo to deploy services on their corresponding kubernetes env. and CD Jobs have all the necessary variables and values used with template files for corresponding env).
Now when any module has deployment related changes, we have to make common changes in dpl and svc files but any new config variables or change in existing config parameters need to be done in each env jobs. With Multiple version of product support, I foresee maintenance of more copies of complete set of jenkins jobs per version.
I have couple of ideas to reduce duplicate work but wanted to know What are my best options to reduce replication of changes and maintenance of so many sets of jobs. Please assist and let me know if more details are required.