I need to simulate many connections. Currently using curl. What is the best way to deploy thousands of such requests with different IPs using some script?

  • What did you try? – 030 Apr 14 '18 at 7:16
  • I don't know much about microservices and dockers. currently only one machine starting many requests but all requests are with the same IP. – nmnir Apr 14 '18 at 8:06
  • Have you looked at Kubernetes? – dwjohnston Apr 15 '18 at 9:43
  • @dwjohnston why Kubernetes should be better compared with the other suggestions? – nmnir Apr 15 '18 at 12:43
  • Could you clarify what you're trying to achieve? Is this for a load balancing/stress test? – dwjohnston Apr 15 '18 at 21:40

Linux, without docker

In Linux, you can create arbitrarily many IP address for each interface. A comment in https://serverfault.com/questions/328146/max-number-virtual-ip-addresses-per-nic reports success with 2000, and hearsay of 5000 successful IPs.

So. Pick any Linux box on your intranet, which could of course be a VM, and create as many IPs on its single ethernet interface (MAC address) as you wish.

Say you're using the private network, then you can execute lots of these:

ifconfig eth0:1 netmask
ifconfig eth0:2 netmask
ifconfig eth0:260 netmask

You should be able to use curl --interface eth0:3 (or maybe --interface to use a specific source address. For wget, it's --bind-address= etc. Start the curls in the background ("&") and they will run more or less concurrently - or look for some test driver (different question) to run them for you.

This will give you lots of requests from separate IP addresses, but all conveniently running on one machine.

Docker, trivially

With docker: just make an image, as small as humanly possible, which connects to your server. You can use curl without any special options.

Then fire up your 1000 docker containers, just like you usually would. Each will automatically get an IP address on the internal docker network (172.x.0.0/16 by default).

https://stackoverflow.com/questions/21799382/is-there-a-maximum-number-of-containers-running-on-a-docker-host indicates that it is possible with hardware that is not too uncommon today; a naive 1000 docker containers used roughly 3 GB of RAM for them. That question also has some hints on options you can use to decrease the memory needs. Aside from memory, there won't be too much overhead, i.e. CPU wise it should comparable.

Docker, with special networking

docker network opens up manually created networking for you. You should be able to cobble something together (just check the individual --help pages, it's all pretty self explanatory), and combine that with the ifconfig approach explained above. This would mean that you only need one container for your curls, and start 1000's of them inside that container.

I would try to just run your 1000 containers, first, though, unless you are limited in the RAM department, for ease of use.

About the curl...

Whatever you do, don't just start a single curl in each instance, but have each runner be an endless loop of back-to-back curl calls. This way, you avoid long gaps inbetween where your individual jobs are being started and stopped; this is obviously especially important for the "1000 docker containers" variation. In the answer linked above, they note that it took them some good amount of minutes to start up those thousands of containers, so you do not want to do that over and over again.

And you might as well get rid of curl altogether and create small networking app in the language of your choice (Perl, Ruby, C, Java...) to avoid process startup overhead.

  • 1
    To get a better distribution in case 1, gatling could be a solution also. Beware that a single machine will hardly make more simultaneous requests than its number of core, deferring network io to the ethernet card may help a little but I doubt the 1k request will be simultaneous (probably not a problem but it worth being said I think) – Tensibai Apr 14 '18 at 15:49
  • 1
    @Tensibai, I assume the question is not so much about having traffic comparable to 1000 real users, but literally about testing the capability of some special program to keep track of 1000 IPs... whatever that may be good for. But I'll put in a note about doing loops instead of one-offs for the curl processes so it's more likely to get a lot of concurrent requests by avoid the setup overhead. – AnoE Apr 14 '18 at 19:09
  • @AnoE Thanks! Just to add to the disscusion: It is both about traffic and about IPs. servers (TCP stacks) behave different when there are 1000 requests from one IP or 1000 IP each making 1 request. – nmnir Apr 15 '18 at 7:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.