2

Is there a straight forward way to generate a .txt file containing each URL (or corresponding IP) visited during a docker build process?

Example

An example list might look like this:

# urls.txt
https://hub.docker.com/
https://www.npmjs.com/

(but a real case using an extensive Dockerfile may be more complex and contain many more URLs/IPs).

Things I tried so far

1. Installing and using an intrusion detection engine

IDSs I attempted to install and use were feature rich and probably more complex than required for what I'm attempting. Configuring IDS engines is not trivial (especially for someone with little network knowledge), so I'd like to explore simpler solutions, if possible.

2. Collecting and parsing docker build logs

I captured build logs with this:

docker build --no-cache --progress=plain  . &> build.log

and extracted lines containing a URL with this regular expression:

(http|ftp|https):\/\/([\w_-]+(?:(?:\.[\w_-]+)+))([\w.,@?^=%&:\/~+#-]*[\w@?^=%&\/~+#-])

A lot of URLs are found, but... i) it doesn't find URLs which aren't explicitly stated in the build logs, and ii) it finds URLs which aren't actually accessed but which are simply stated in logs (i.e. logs along the lines of "For more info, visit https://www.example.com".

Question

How can I generate a nice clean list of each external URL visited during a docker build process?

1
  • Note: wasn't sure if this is best here on or on SO. Glad for it to be moved if it's more appropriate elsewhere.
    – stevec
    Feb 5, 2023 at 2:22

3 Answers 3

3

One thing that pops my mind is using either of Wireshark, tshark, or tcpdump for capturing network packets originating at docker daemon while docker build is running.

After recording the traffic, need to filter & process the captured data.

1

Maybe you could use some local redirecting proxy service and patch your localhosts file to get all requests caught there?

1

You could start with something like this (tcpdump):

tcpdump -q ip and ip6 and dst port 80 or 443 > tcpdump.log

This should capture all outgoing ipv4 and ipv6 traffic, to destiation ports 80 and 443 (http and https). You can add -i eth1 to capture data on a specific interface only. You can play with filters. You should get something like this (results from curling google and facebook) - notice that you won't get actual google.com domain, but the results could still help to see which domains are blocked by your firewall:

19:07:58.915115 IP pmsbuild02.4752 > bud02s38-in-f4.1e100.net.http: tcp 0
19:07:58.915125 IP pmsbuild02.4752 > bud02s38-in-f4.1e100.net.http: tcp 0
19:07:58.947639 IP pmsbuild02.4752 > bud02s38-in-f4.1e100.net.http: tcp 0
19:07:58.947869 IP pmsbuild02.4752 > bud02s38-in-f4.1e100.net.http: tcp 0
19:07:58.948406 IP pmsbuild02.4752 > bud02s38-in-f4.1e100.net.http: tcp 0
19:08:04.608537 IP pmsbuild02.10146 > edge-star-mini-shv-02-frt3.facebook.com.http: tcp 0
19:08:04.610415 IP pmsbuild02.10146 > edge-star-mini-shv-02-frt3.facebook.com.http: tcp 0
19:08:04.610494 IP pmsbuild02.10146 > edge-star-mini-shv-02-frt3.facebook.com.http: tcp 80
19:08:04.656677 IP pmsbuild02.10146 > edge-star-mini-shv-02-frt3.facebook.com.http: tcp 0
19:08:04.656887 IP pmsbuild02.10146 > edge-star-mini-shv-02-frt3.facebook.com.http: tcp 0
19:08:04.657372 IP pmsbuild02.10146 > edge-star-mini-shv-02-frt3.facebook.com.http: tcp 0

You can then easily filter out the domains with awk (get the 5th section, delimited by spaces) - note that there's still the http/https appended to the end of the domain, and the colon sign:

awk '{print $5}' tcpdump.log

Result:

edge-star-mini-shv-01-vie1.facebook.com.https:

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.