I need to simulate many connections. Currently using curl. What is the best way to deploy thousands of such requests with different IPs using some script?
Linux, without docker
In Linux, you can create arbitrarily many IP address for each interface. A comment in https://serverfault.com/questions/328146/max-number-virtual-ip-addresses-per-nic reports success with 2000, and hearsay of 5000 successful IPs.
So. Pick any Linux box on your intranet, which could of course be a VM, and create as many IPs on its single ethernet interface (MAC address) as you wish.
Say you're using the 10.0.0.0/8 private network, then you can execute lots of these:
ifconfig eth0:1 10.0.0.2 netmask 255.0.0.0
ifconfig eth0:2 10.0.0.3 netmask 255.0.0.0
ifconfig eth0:260 10.0.1.6 netmask 255.0.0.0
You should be able to use
curl --interface eth0:3 (or maybe
--interface 10.0.0.3) to use a specific source address. For
--bind-address=10.0.0.3 etc. Start the curls in the background ("&") and they will run more or less concurrently - or look for some test driver (different question) to run them for you.
This will give you lots of requests from separate IP addresses, but all conveniently running on one machine.
With docker: just make an image, as small as humanly possible, which connects to your server. You can use
curl without any special options.
Then fire up your 1000 docker containers, just like you usually would. Each will automatically get an IP address on the internal docker network (172.x.0.0/16 by default).
https://stackoverflow.com/questions/21799382/is-there-a-maximum-number-of-containers-running-on-a-docker-host indicates that it is possible with hardware that is not too uncommon today; a naive 1000 docker containers used roughly 3 GB of RAM for them. That question also has some hints on options you can use to decrease the memory needs. Aside from memory, there won't be too much overhead, i.e. CPU wise it should comparable.
Docker, with special networking
docker network opens up manually created networking for you. You should be able to cobble something together (just check the individual
--help pages, it's all pretty self explanatory), and combine that with the
ifconfig approach explained above. This would mean that you only need one container for your
curls, and start 1000's of them inside that container.
I would try to just run your 1000 containers, first, though, unless you are limited in the RAM department, for ease of use.
Whatever you do, don't just start a single
curl in each instance, but have each runner be an endless loop of back-to-back
curl calls. This way, you avoid long gaps inbetween where your individual jobs are being started and stopped; this is obviously especially important for the "1000 docker containers" variation. In the answer linked above, they note that it took them some good amount of minutes to start up those thousands of containers, so you do not want to do that over and over again.
And you might as well get rid of
curl altogether and create small networking app in the language of your choice (Perl, Ruby, C, Java...) to avoid process startup overhead.