We are currently migrating many of our production data ingress/processing scripts/processes into a kubernetes cluster. Each "customer" or company that we interface with has a bunch of data and we want to process all of this data on a per customer basis. I have created a container with a script where I pass a single customerID into.
E.G. One container will run and process one customers data. This will need to happen for one or more customers on a schedule.
I have been using the following definition to work specifically for customer 756, and it is working as expected.
apiVersion: batch/v1beta1 kind: CronJob metadata: name: image-puller-756 spec: schedule: "30 * * * *" concurrencyPolicy: Forbid jobTemplate: spec: template: spec: containers: - name: image-puller-756 image: index.docker.io/my_company/image_puller:latest imagePullPolicy: Always env: - name: ALLOW_TTY value: "true" - name: PYTHONUNBUFFERED value: "0" - name: prodDbUser valueFrom: secretKeyRef: name: image-creds key: username - name: prodDbPass valueFrom: secretKeyRef: name: image-creds key: password command: ["/bin/bash"] args: ["-c", "/usr/bin/python3 -u /go/bin/main.py -c 756"] restartPolicy: Never imagePullSecrets: - name: myregistrykey
Once scheduled it will run hourly which is perfect, however I now need a way to duplicate this definition for ~500 different customers and apply it to the cluster. Currently I am using a template file and doing a Find and replace with a list of the customers I'm testing, but I'm looking for a better way to manage it. The customers will only have certain processes ran for them based on some logic, and I would like to be able to run another process that will act as a schedule and dynamically add or remove jobs for the customers.
I thought that there might be a way using yaml or k8s to pass a list and a template and have it create them. The approach just seems very antiquated and I'm struggling to think that this is the correct one.