Cheapest and easiest way to run a periodic Docker process

Question

I have a dockerized Python process that a client would like to run once or twice a month. It calls a number of Google APIs and stores the results in a Google Sheet. It does not accept network requests. I want to provide one "magic button" that will spin up the Docker container and then shut everything down.

Some background: I had previously deployed this to an AWS EC2 micro instance via Docker Cloud. My client got a Docker Cloud account and was able to run the container on demand, relatively painlessly. There are two problems with this workflow:

1. The provisioned EC2 instance (Docker host) was running 24/7, whether the Docker container was run up or not. This became expensive.
2. Docker cloud shut down a month ago.

Alternatives seem to be Google and AWS Kubernetes services. My reservation is that they will be too complicated for my client to use. Advice?

Answer 1

AWS offers fargate now, which is managed by them rather than being a service like ECS which you ran the servers for and they manage the control plane. Fargate is basically "serverless" ECS. You can use a cloudwatch scheduled even to run something occasionally on a schedule on a fargate ECS cluster. Therefore you're only paying for what you're using. Something to consider is fargate is quite a bit more expensive than ECS running on your own EC2 instance, but if you're running in frequently you'll likely see a pretty marginal savings.

Another option would be to automate something similar. So create an ECS cluster with an ASG backing it, and adjust scale to match ECS demand, and then use cloudwatch to launch the container. ECS can then scale up and down, but this can be tricky because this method is usually used to autoscale ECS clusters based on cloutwatch triggers, so you might have to hack around in cloudwatch and possibly hack together a lambda script, since you're technically not using it the way it's intended. In the end this would probably be the most cost effective, but in your case the cost savings could be negligible.

In my opinion Kubernetes is a ten ton hammer that you'd be using to hang a picture on the wall. It's way overkill.

You can explore those 2 options and see if they are cost effective, and easy enough to manage. Other container orchestration is usually going to be overkill, ECS has the advantage(usually a disadvantage) of being stupid simple and you pay very little and have basically no operational overhead for the control plane.

Answer 2

The cheapest way is to use Heroku's Container Registry.

It's entirely free, easy to use and deploy (You don't even need the heroku CLI , just docker push your image to registry.heroku.com with your auth token)

It may not be the best though but based on your comments I think it should be fine for you, see cons here.

Another cheap alternative is hyper.sh.

Answer 3

You can mix the powerful of Zappa and Hug to convert your code in a serverless AWS Lambda function adding only a decorator to your python process main function

import hug
[...]

@hug.get('/your_endpoint_name')
def your_function_name():
    """Here goes your code"""
    [...]
    return "Function finished sucesfully"

After this you can deploy to AWS Lambda with zappa deploy prod and you will only need to call the returned URL twice a month.

Answer 4

While I'd normally agree with the other answers that Kubernetes is overkill, KubeSail simplifies it as much as possible in order to make it easy to run tasks like yours. There is a free tier that should let you run your job indefinitely. You just need to log in with GitHub, get your Kube config, and then you can use the following:

apiVersion: batch/v1
kind: Job
metadata:
  name: my-python-script
spec:
  template:
    spec:
      containers:
      - name: my-python-script
        image: asciimo/my-python-image

Just save out the above to my-job.yaml, replacing image: asciimo/my-python-image with the name of your image on dockerhub (or another registry), then run

kubectl apply -f my-job.yaml

If you wanted to learn more about Jobs, the Kubernetes docs contain a lot more info and options: https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/

Full disclosure, I am one of the KubeSail Founders

Answer 5

Amazon's ECS service is really simple and allows you to schedule container tasks

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/scheduling_tasks.html

Answer 6

Ansible ec2 module have required functionality (launch instances, runs some tasks and then terminate them) out-of-the-box. And even have sample playbook.