We are currently migrating many of our production data ingress/processing scripts/processes into a kubernetes cluster. Each "customer" or company that we interface with has a bunch of data and we want to process all of this data on a per customer basis. I have created a container with a script where I pass a single customerID into. E.G. One container will run and process one customers data.
This will need to happen for one or more customers on a schedule.
I have been using the following definition to work specifically for customer 756, and it is working as expected.
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: image-puller-756
spec:
schedule: "30 * * * *"
concurrencyPolicy: Forbid
jobTemplate:
spec:
template:
spec:
containers:
- name: image-puller-756
image: index.docker.io/my_company/image_puller:latest
imagePullPolicy: Always
env:
- name: ALLOW_TTY
value: "true"
- name: PYTHONUNBUFFERED
value: "0"
- name: prodDbUser
valueFrom:
secretKeyRef:
name: image-creds
key: username
- name: prodDbPass
valueFrom:
secretKeyRef:
name: image-creds
key: password
command: ["/bin/bash"]
args: ["-c", "/usr/bin/python3 -u /go/bin/main.py -c 756"]
restartPolicy: Never
imagePullSecrets:
- name: myregistrykey
Once scheduled it will run hourly which is perfect, however I now need a way to duplicate this definition for ~500 different customers and apply it to the cluster. Currently I am using a template file and doing a Find and replace with a list of the customers I'm testing, but I'm looking for a better way to manage it. The customers will only have certain processes ran for them based on some logic, and I would like to be able to run another process that will act as a schedule and dynamically add or remove jobs for the customers.
I thought that there might be a way using yaml or k8s to pass a list and a template and have it create them. The approach just seems very antiquated and I'm struggling to think that this is the correct one.