I'm currently trying to scale a crawler in the cloud.
I'm wondering what would be the best way to automate the work-flow, considering the following conditions:
- Guaranteed bandwidth, CPU and memory per crawler, no matter how many crawls are running in parallel.
- Sometimes there are 3 crawlers running at the same time, sometimes 10000.
- There is no need of communication between crawlers, and no shared queue.
- The crawl should start quickly after the demand of the user.

The potential "work-flow" could be:
1 - A user pushes a button to start a crawl.
2 - An API receives the request.
3 - The API asks if there is a container available to start a crawl.
Here 2 options:
3a - A container was in stand-by, ready to start .
3b - No container available, so a new container is started .
4 - The IP:PORT of the container is sent to the API, and tagged as "reserved" .
5 - The API receives the IP:PORT .
6 - The API sends the request to http://IP:PORT with the required parameters to start the crawl (URL, etc.).
Here 2 options:
6a - Expected behavior: The crawl starts.
=> A new container is started and is tagged as "available" mode for potential new crawl to come.
6b - Unexpected behavior: The request to start the crawl doesn't reach the target.
=> The container is tagged as "available" after n seconds without being used to start a crawl.
7 - The container used for the crawl is deleted.

  • Does each user only trigger 1 crawler, or do they need to trigger n number of concurrent crawlers? Will crawlers take >15 minutes? – user2640621 Jan 16 at 1:13
  • Each user needs to trigger at least 1 crawler (crawling one website) making potentially n concurrent request (it stays on the same website). Crawl can last from 1 minute to up to several days, depending of the size of the website, the n concurrent request, speed of the website, etc. – Quentin Jan 16 at 12:07

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.