I'm currently trying to scale a crawler in the cloud.
I'm wondering what would be the best way to automate the work-flow, considering the following conditions:
- Guaranteed bandwidth, CPU and memory per crawler, no matter how many crawls are running in parallel.
- Sometimes there are 3 crawlers running at the same time, sometimes 10000.
- There is no need of communication between crawlers, and no shared queue.
- The crawl should start quickly after the demand of the user.

The potential "work-flow" could be:
1 - A user pushes a button to start a crawl.
2 - An API receives the request.
3 - The API asks if there is a container available to start a crawl.
Here 2 options:
3a - A container was in stand-by, ready to start .
3b - No container available, so a new container is started .
4 - The IP:PORT of the container is sent to the API, and tagged as "reserved" .
5 - The API receives the IP:PORT .
6 - The API sends the request to http://IP:PORT with the required parameters to start the crawl (URL, etc.).
Here 2 options:
6a - Expected behavior: The crawl starts.
=> A new container is started and is tagged as "available" mode for potential new crawl to come.
6b - Unexpected behavior: The request to start the crawl doesn't reach the target.
=> The container is tagged as "available" after n seconds without being used to start a crawl.
7 - The container used for the crawl is deleted.

  • Does each user only trigger 1 crawler, or do they need to trigger n number of concurrent crawlers? Will crawlers take >15 minutes? – user2640621 Jan 16 at 1:13
  • Each user needs to trigger at least 1 crawler (crawling one website) making potentially n concurrent request (it stays on the same website). Crawl can last from 1 minute to up to several days, depending of the size of the website, the n concurrent request, speed of the website, etc. – Quentin Jan 16 at 12:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.