Can I use AWS Step Functions to launch many different fargate 'runs'?

Question

I am building a service that has two independent steps. The first step is a data transformation step that has to happen and is common to all downstream processes, then the second is starting a few dozen independent parallel processes that use the same input data, save for a few different runtime specified parameters.

I know I can use a step function to set off the data transformation step and wait for it to complete before starting off a single fargate task, but can I get it to start a few dozen (or more) in parallel using a common docker object but pass each of those runs slightly different parameter values.

I want to use fargate because we are a small team and I don't want to have to spend team resources on maintaining compute resources, and I can't use AWS Lambda because we expect the run times will often exceed 15 minutes, and there isn't an easy or obvious way to break the problem down any further to be certain the jobs will be able to execute within 15 minutes.

Answer 1

Yes, you can start ECS Fargate tasks from Step Functions in the way you've described.

Step Functions allows you to pass Parameters to the underlying service API.

https://docs.aws.amazon.com/step-functions/latest/dg/connect-parameters.html

[sidebar: that doc also specifies how to pass state inputs, which you've stated is a requirement]

For ECS Fargate, a single state machine with no unique configuration could look something like this:

 "States": {
    "${name}": {
      "Type": "Task",
      "Resource": "arn:aws:states:::ecs:runTask.sync",
      "Parameters": {
        "LaunchType": "FARGATE",
        "Cluster": "${cluster}",
        "TaskDefinition": "${task_definition}",
        "NetworkConfiguration": {
          "AwsvpcConfiguration": {
            "Subnets": ["${subnet_1}", "${subnet_2}"],
            "SecurityGroups": ["${security_group}"],
            "AssignPublicIp": "DISABLED"
          }
        }
      },

Now, you've asked about passing each task slightly different parameter values...

Thankfully, the ECS RunTask API has a helpful overrides: { containerOverrides: [] } parameter.

https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_RunTask.html

Overriding the command (for task execution) or environment (for OS variables) should provide the customization you require without needing multiple ECS Task Definitions.

So, add each distinct task to your state machine, each ingesting state inputs and with its own unique override, and that should do it!