How to generate a list of every external URL accessed during docker build?

Question

Is there a straight forward way to generate a .txt file containing each URL (or corresponding IP) visited during a docker build process?

Example

An example list might look like this:

# urls.txt
https://hub.docker.com/
https://www.npmjs.com/

(but a real case using an extensive Dockerfile may be more complex and contain many more URLs/IPs).

Things I tried so far

1. Installing and using an intrusion detection engine

IDSs I attempted to install and use were feature rich and probably more complex than required for what I'm attempting. Configuring IDS engines is not trivial (especially for someone with little network knowledge), so I'd like to explore simpler solutions, if possible.

2. Collecting and parsing docker build logs

I captured build logs with this:

docker build --no-cache --progress=plain  . &> build.log

and extracted lines containing a URL with this regular expression:

(http|ftp|https):\/\/([\w_-]+(?:(?:\.[\w_-]+)+))([\w.,@?^=%&:\/~+#-]*[\w@?^=%&\/~+#-])

A lot of URLs are found, but... i) it doesn't find URLs which aren't explicitly stated in the build logs, and ii) it finds URLs which aren't actually accessed but which are simply stated in logs (i.e. logs along the lines of "For more info, visit https://www.example.com".

Question

How can I generate a nice clean list of each external URL visited during a docker build process?

Answer 1

One thing that pops my mind is using either of Wireshark, tshark, or tcpdump for capturing network packets originating at docker daemon while docker build is running.

After recording the traffic, need to filter & process the captured data.

Answer 2

Maybe you could use some local redirecting proxy service and patch your localhosts file to get all requests caught there?

Answer 3

You could start with something like this (tcpdump):

tcpdump -q ip and ip6 and dst port 80 or 443 > tcpdump.log

This should capture all outgoing ipv4 and ipv6 traffic, to destiation ports 80 and 443 (http and https). You can add -i eth1 to capture data on a specific interface only. You can play with filters. You should get something like this (results from curling google and facebook) - notice that you won't get actual google.com domain, but the results could still help to see which domains are blocked by your firewall:

19:07:58.915115 IP pmsbuild02.4752 > bud02s38-in-f4.1e100.net.http: tcp 0
19:07:58.915125 IP pmsbuild02.4752 > bud02s38-in-f4.1e100.net.http: tcp 0
19:07:58.947639 IP pmsbuild02.4752 > bud02s38-in-f4.1e100.net.http: tcp 0
19:07:58.947869 IP pmsbuild02.4752 > bud02s38-in-f4.1e100.net.http: tcp 0
19:07:58.948406 IP pmsbuild02.4752 > bud02s38-in-f4.1e100.net.http: tcp 0
19:08:04.608537 IP pmsbuild02.10146 > edge-star-mini-shv-02-frt3.facebook.com.http: tcp 0
19:08:04.610415 IP pmsbuild02.10146 > edge-star-mini-shv-02-frt3.facebook.com.http: tcp 0
19:08:04.610494 IP pmsbuild02.10146 > edge-star-mini-shv-02-frt3.facebook.com.http: tcp 80
19:08:04.656677 IP pmsbuild02.10146 > edge-star-mini-shv-02-frt3.facebook.com.http: tcp 0
19:08:04.656887 IP pmsbuild02.10146 > edge-star-mini-shv-02-frt3.facebook.com.http: tcp 0
19:08:04.657372 IP pmsbuild02.10146 > edge-star-mini-shv-02-frt3.facebook.com.http: tcp 0

You can then easily filter out the domains with awk (get the 5th section, delimited by spaces) - note that there's still the http/https appended to the end of the domain, and the colon sign:

awk '{print $5}' tcpdump.log

Result:

edge-star-mini-shv-01-vie1.facebook.com.https: