Pattern(s) to scale node.js based ETL

Question

I have a node.js ETL application that transforms data from 1..n locations and from each location there will be 0..m transactions. The application is started via cron on a configurable interval, currently 15 minutes. Currently, the pattern employed is to use Promise.map({concurrency:x}) at both the location and transaction level to obtain a combined degree of concurrency to achieve the maximum throughput (constrained by DB I/O.

Right now the app runs on a single Ubuntu VM, processing 6 locations and ~ 1000 transactions/location. So minimal data processing overhead. The pattern works. But when there are 1000+ locations and possibly more transactions per interval, the pattern will still work of course, but the time to complete all locations will grow. Being node, the process is single threaded and so the pattern is not leveraging the available cores.

So I’m thinking about how to change the pattern to scale. Options I’m considering (not mutually exclusive)

• containerise the app, start one for every n locations - based on some manual determination of # of locations to process per container.
• remain non containerised but use fork() to leverage additional CPU’s and partition on n locations

Based on my reading to date, my approach is:

• forget “fork"
• containerise the app
  • spin up x apps, x being determined based on manual determination since the parameters are known vs dynamic
  • each container would be responsible for n locations
  • each app would set its own “cron” to orchestrate its batch processing

Not being an expert in docker (but reasonably familiar) and certainly a newbie to orchestration, I would appreciate any guidance on how containers / orchestration might help.

Answer 1

I ended up following this path

Rather than use cron, i used a scheduling framework (in my case Bree.js) which allowed me to "jobify" the tasks and also execute these tasks in separate processes - this allowed me the vertical expansion needed (ie: using more processor cores)

A docker container contains the dispatch (scheduling) and jobs - so once the container is started, jobs (and associated processes) are created/managed within that container

Eventually, when I need to scale horizontally, the "dispatch" function will morph to running a job whose sole purpose is to post an event (eg: Azure service bus). I will have an orchestrator running various containers who read from the queue and process accordingly.