Kubernetes create multiple jobs based on template and list

Question

We are currently migrating many of our production data ingress/processing scripts/processes into a kubernetes cluster. Each "customer" or company that we interface with has a bunch of data and we want to process all of this data on a per customer basis. I have created a container with a script where I pass a single customerID into. E.G. One container will run and process one customers data. This will need to happen for one or more customers on a schedule.

I have been using the following definition to work specifically for customer 756, and it is working as expected.

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: image-puller-756
spec:
  schedule: "30 * * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: image-puller-756
            image: index.docker.io/my_company/image_puller:latest
            imagePullPolicy: Always
            env:
              - name: ALLOW_TTY
                value: "true"
              - name: PYTHONUNBUFFERED
                value: "0"
              - name: prodDbUser
                valueFrom:
                  secretKeyRef:
                    name: image-creds
                    key: username
              - name: prodDbPass 
                valueFrom:
                  secretKeyRef:
                    name: image-creds
                    key: password
            command: ["/bin/bash"]
            args: ["-c", "/usr/bin/python3 -u /go/bin/main.py -c 756"]
          restartPolicy: Never
          imagePullSecrets:
          - name: myregistrykey

Once scheduled it will run hourly which is perfect, however I now need a way to duplicate this definition for ~500 different customers and apply it to the cluster. Currently I am using a template file and doing a Find and replace with a list of the customers I'm testing, but I'm looking for a better way to manage it. The customers will only have certain processes ran for them based on some logic, and I would like to be able to run another process that will act as a schedule and dynamically add or remove jobs for the customers.

I thought that there might be a way using yaml or k8s to pass a list and a template and have it create them. The approach just seems very antiquated and I'm struggling to think that this is the correct one.

Answer 1

Have you checked the documentation for cron-jobs? You can use the run command.

$ kubectl run hello --schedule="*/1 * * * *" --restart=OnFailure --image=busybox -- /bin/sh -c "date; echo Hello from the Kubernetes cluster"
cronjob "hello" created

This way you can loop through your customers in a bash script to deploy your containers.

Not knowing the big picture its hard to answer completely, but I also think ansible templates would work for this. You could turn your above config file into a jinja2 template. You could pass your customers in as a JSON dict and use with_items to loop through them or treat your customers as an inventory. This would give you an easy way to deal with certain "snowflake" customers.

EDIT: as far as doing this with Vanilla Kubernetes, I don't think it's possible. You could add your own custom controller and extend the kubernetes API. This would allow you to creat an API object that defines a customer, and then have a custom controller pick that up and create your cron job. Check out https://github.com/coreos/prometheus-operator as they have implemented such a solution to allow multiple teams to run there own Prometheus solution on the same cluster.