85

I want to manually download a Docker Image from Docker Hub. More specifically, I want to download a Docker Image from Docker Hub on a machine in a restricted environment which does not (and cannot) have the Docker client software installed. I would have thought that this would be possible using the official API, but this does not appear to be the case - see the following discussion:

Is it really the case that the API doesn't support downloading images? Is there a way to work around this?


UPDATE 1:

I came across the following ServerFault post:

The accepted solution uses the docker save command, which doesn't help in my situation. But another solution posted there cites the following StackOverflow post:

One of the solutions there refers to a command-line tool called docker-registry-debug which, among other things, can generate a curl command for downloading an image. Here is what I got:

user@host:~$ docker-registry-debug curlme docker ubuntu

# Reading user/passwd from env var "USER_CREDS"
# No password provided, disabling auth
# Getting token from https://index.docker.io
# Got registry endpoint from the server: https://registry-1.docker.io
# Got token: signature=1234567890abcde1234567890abcde1234567890,repository="library/docker",access=read
curl -i --location-trusted -I -X GET -H "Authorization: Token signature=1234567890abcde1234567890abcde1234567890,repository="library/docker",access=read" https://registry-1.docker.io/v1/images/ubuntu/layer

user@host:~$ curl \
-i --location-trusted -I -X GET \
-H "Authorization: Token signature=1234567890abcde1234567890abcde1234567890,repository="library/docker",access=read" 

https://registry-1.docker.io/v1/images/ubuntu/layer
HTTP/1.1 404 NOT FOUND
Server: gunicorn/18.0
Date: Wed, 29 Nov 2017 01:00:00 GMT
Expires: -1
Content-Type: application/json
Pragma: no-cache
Cache-Control: no-cache
Content-Length: 29
X-Docker-Registry-Version: 0.8.15
X-Docker-Registry-Config: common
Strict-Transport-Security: max-age=31536000

So unfortunately it looks like the curl command generated does not work.


UPDATE 2:

It looks like I'm able to download layer blobs from Docker Hub. Here is how I'm currently going about it.

Get an authorization token:

user@host:~$ export TOKEN=\
"$(curl \
--silent \
--header 'GET' \
"https://auth.docker.io/token?service=registry.docker.io&scope=repository:library/ubuntu:pull" \
| jq -r '.token' \
)"

Pull an image manifest:

user@host:~$ curl \
--silent \
--request 'GET' \
--header "Authorization: Bearer ${TOKEN}" \
'https://registry-1.docker.io/v2/library/ubuntu/manifests/latest' \
| jq '.'

Pull an image manifest and extract the blob sums:

user@host:~$ curl \
--silent \
--request 'GET' \
--header "Authorization: Bearer ${TOKEN}" \
'https://registry-1.docker.io/v2/library/ubuntu/manifests/latest' \
| jq -r '.fsLayers[].blobSum'

sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
sha256:be588e74bd348ce48bb7161350f4b9d783c331f37a853a80b0b4abc0a33c569e
sha256:e4ce6c3651b3a090bb43688f512f687ea6e3e533132bcbc4a83fb97e7046cea3
sha256:421e436b5f80d876128b74139531693be9b4e59e4f1081c9a3c379c95094e375
sha256:4c7380416e7816a5ab1f840482c9c3ca8de58c6f3ee7f95e55ad299abbfe599f
sha256:660c48dd555dcbfdfe19c80a30f557ac57a15f595250e67bfad1e5663c1725bb

Download a single layer blob and write it to a file:

user@host:~$ BLOBSUM=\
"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"

user@host:~$ curl \
--silent \
--location \
--request GET \
--header "Authorization: Bearer ${TOKEN}" \
"https://registry-1.docker.io/v2/library/ubuntu/blobs/${BLOBSUM}" \
> "${BLOBSUM/*:/}.gz"

Write all of the blob sums to a file:

user@host:~$ curl \
--silent \
--request 'GET' \
--header "Authorization: Bearer ${TOKEN}" \
'https://registry-1.docker.io/v2/library/ubuntu/manifests/latest' \
| jq -r '.fsLayers[].blobSum' > ubuntu-blobsums.txt

Download all of the layer blobs from the manifest:

user@host:~$ while read BLOBSUM; do
curl \
--silent \
--location \
--request 'GET' \
--header "Authorization: Bearer ${TOKEN}" \
"https://registry-1.docker.io/v2/library/ubuntu/blobs/${BLOBSUM}" \
> "${BLOBSUM/*:/}.gz"; \
done < blobsums.txt

Now I have a bunch of layer blobs and I need to recombine them into an image - I think.


Related Links:

7
  • 1
    "More specifically, I want to download a Docker Image from Docker Hub on a machine in a restricted environment which does not (and cannot) have the Docker client software installed." => What's the point to have the images on this machine then ? (easier workaround is using a pivot host, one where you acn docker pull from dockerhub and docker save/ docker push to an internal registry after)
    – Tensibai
    Commented Nov 30, 2017 at 11:06
  • @Tensibai To copy it to another machine that does have Docker but doesn't have internet access.
    – igal
    Commented Nov 30, 2017 at 13:46
  • 1
    Did you had a look at the docker pull code? It sounds the way to go to build something like this from basic http calls
    – Tensibai
    Commented Nov 30, 2017 at 19:04
  • @Tensibai I think I figured it out. I also think I got a solution from the Docker community. I'll come back and post the solution later today.
    – igal
    Commented Nov 30, 2017 at 20:57
  • @Tensibai I posted a solution with a shell script that solves the problem.
    – igal
    Commented Dec 3, 2017 at 15:04

9 Answers 9

56

It turned out that the Moby Project has a shell script on the Moby Github which can download images from Docker Hub in a format that can be imported into Docker:

The usage syntax for the script is given by the following:

download-frozen-image-v2.sh target_dir image[:tag][@digest] ...

The image can then be imported with tar and docker load:

tar -cC 'target_dir' . | docker load

To verify that the script works as expected, I downloaded an Ubuntu image from Docker Hub and loaded it into Docker:

user@host:~$ bash download-frozen-image-v2.sh ubuntu ubuntu:latest
user@host:~$ tar -cC 'ubuntu' . | docker load
user@host:~$ docker run --rm -ti ubuntu bash
root@1dd5e62113b9:/#

In practice I would have to first copy the data from the internet client (which does not have Docker installed) to the target/destination machine (which does have Docker installed):

user@nodocker:~$ bash download-frozen-image-v2.sh ubuntu ubuntu:latest
user@nodocker:~$ tar -C 'ubuntu' -cf 'ubuntu.tar' .
user@nodocker:~$ scp ubuntu.tar user@hasdocker:~

and then load and use the image on the target host:

user@hasdocker:~ docker load ubuntu.tar
user@hasdocker:~ docker run --rm -ti ubuntu bash
root@1dd5e62113b9:/#
4
  • 1
    The machine with internet connectivity does not and cannot have Docker installed. but you apply docker load
    – 030
    Commented Dec 3, 2017 at 15:11
  • 1
    @030 Just to test/demonstrate that the script works and that the download image data can be imported into Docker. In practice I would first have to copy the data to a machine with Docker installed.
    – igal
    Commented Dec 3, 2017 at 15:13
  • Perhaps you could add that part for clarification
    – 030
    Commented Dec 3, 2017 at 15:13
  • 2
    @030 I added an example session illustrating what the workflow would look like in practice.
    – igal
    Commented Dec 3, 2017 at 15:20
20

There is a tool called Skopeo which can retrieve Docker images from a repository and save them in several formats.

For example:

  1. Download the image and save the layers as a tarball: skopeo copy docker://ubuntu docker-archive:/tmp/ubuntu.tar:ubuntu

  2. Transfer /tmp/ubuntu.tar to another machine if desired.

  3. Load the image on a Docker instance which does not have internet connection: docker load --input /tmp/ubuntu.tar

It is available in CentOS 7 repo with the package name skopeo. There are no Debian or Ubuntu packages at this time (but it is easy to compile).

1
5

thanks for motivation. I made a powershell version of it. Check it out... With it you can move in dockerhub containers to a restricted docker networks with a windows desktop and an ssh-scp tool to docker machine without root or administrator rights

https://gitlab.com/Jancsoj78/dockerless_docker_downloader a new hacker tool :)

$image = "ubuntu"
$tag = "latest"
$imageuri = "https://auth.docker.io/token?service=registry.docker.io&scope=repository:library/"+$image+":pull"
$taguri = "https://registry-1.docker.io/v2/library/"+$image+"/manifests/"+$tag
$bloburi = "https://registry-1.docker.io/v2/library/"+$image+"/blobs/sha256:"

#token request
$token = Invoke-WebRequest -Uri $imageuri | ConvertFrom-Json | Select -expand token

#pull image manifest
$blobs = $($(Invoke-Webrequest -Headers @{Authorization="Bearer $token"} -Method GET -Uri $taguri | ConvertFrom-Json | Select -expand fsLayers ) -replace "sha256:" -replace "@{blobSum=" -replace "}")

#download blobs
for ($i=0; $i -lt $blobs.length; $i++) {
    $blobelement =$blobs[$i]

    Invoke-Webrequest -Headers @{Authorization="Bearer $token"} -Method GET -Uri $bloburi$blobelement -OutFile blobtmp

    $source = "blobtmp"
    $newfile = "$blobelement.gz"

#overwrite
Copy-Item $source $newfile -Force -Recurse
#source blobs
ls *.gz
}
#postprocess
echo "copy these .gz to your docker machine"
echo "docker import .gz backward one by one"
echo "lastone with ubuntu:latest"
echo "after docker export and reimport to make a simple layer image"
0
3

Here is an adapted python script, thus having an OS independant solution: docker-drag

Use it like that, and it will create a TAR archive that you will be able to import using docker load :

python docker_pull.py hello-world
python docker_pull.py alpine:3.9
python docker_pull.py kalilinux/kali-linux-docker
4
  • 2
    If you close your github account there's nothing left, if you're ok with sharing it, keep the link but paste the script here also please. You can edit your answer, paste your code and then select it and type ctrl+K or the {} (code) button in the editor top bar to format it.
    – Tensibai
    Commented Apr 9, 2019 at 13:16
  • I would love to paste the code here but it is 100 lines long and I don't think it will be readable. Nevertheless, you can fork the code to save your own copy of the script.
    – Dutfaz
    Commented Apr 9, 2019 at 13:19
  • That's not for me, that to have an answer which is self sustaining, if the link break, do you really think this may help someone reading this answer in a few months ? (btw the maximum size for an answer is 30k characters)
    – Tensibai
    Commented Apr 9, 2019 at 13:39
  • You could put this in pip. I find it very helpful.
    – Anoyz
    Commented Apr 18, 2021 at 18:24
1

To me it is not completely clear what you are trying to achieve and why the attempts are not a solution for the problem. If I would need to solve this issue I would like @Tensibai and other Q&As indicated, do a docker pull first on a system with internet connectivity, save the docker image, copy it to the machine without internet connectivity, load the image and run it.

Demonstration

There are no images on system A:

userA@systemA ~ $ docker images
REPOSITORY        TAG               IMAGE ID          CREATED             SIZE
userA@systemA ~ $

Pull an image from dockerhub:

userA@systemA ~ $
docker pull nginx
Using default tag: latest
latest: Pulling from library/nginx
bc95e04b23c0: Pull complete 
f3186e650f4e: Pull complete 
9ac7d6621708: Pull complete 
Digest: sha256:b81f317384d7388708a498555c28a7cce778a8f291d90021208b3eba3fe74887
Status: Downloaded newer image for nginx:latest
userA@systemA ~ $ docker images
REPOSITORY        TAG               IMAGE ID            CREATED             SIZE
nginx             latest            9e7424e5dbae        10 days ago         108MB

Save docker image:

userA@systemA ~ $ docker save nginx -o nginx.tar

Copy docker image to systemB and load it.

userB@systemB ~ $ docker load -i nginx.tar
cec7521cdf36: Loading layer  58.44MB/58.44MB
350d50e58b6c: Loading layer  53.76MB/53.76MB
63c39cd4a775: Loading layer  3.584kB/3.584kB
Loaded image: nginx:latest
userB@systemB ~ $ docker images
REPOSITORY        TAG               IMAGE ID            CREATED             SIZE
nginx             latest            9e7424e5dbae        10 days ago         108MB
1
  • 2
    The machine with internet connectivity does not and cannot have Docker installed. The question is asking for a way to download an image without using the Docker client. See my solution.
    – igal
    Commented Dec 3, 2017 at 15:03
1

I didn't really understand Jancsó's way of postprocessing, so I've spent some time to modify his script and here's what I came with:

https://github.com/meetyourturik/dockerless-docker-downloader

upd: apparently, 'a link only answer...' is something bad, so here's a whole script:

# Workaround for SelfSigned Cert an force TLS 1.2
add-type @"
    using System.Net;
    using System.Security.Cryptography.X509Certificates;
    public class TrustAllCertsPolicy : ICertificatePolicy {
        public bool CheckValidationResult(
        ServicePoint srvPoint, X509Certificate certificate,
        WebRequest request, int certificateProblem) {
            return true;
        }
    }
"@
[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy
[System.Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12

# use 'library/' prefix for 'official' images like postgres 
$image = "atlassian/jira-software" 
$tag = "8.13.2" 

$imageuri = "https://auth.docker.io/token?service=registry.docker.io&scope=repository:${image}:pull" 
$taguri = "https://registry-1.docker.io/v2/${image}/manifests/${tag}"
$bloburi = "https://registry-1.docker.io/v2/${image}/blobs/" 

# generating folder to save image files 
$path = "$image$tag" -replace '[\\/":*?<>|]'
if (!(test-path $path)) { 
    New-Item -ItemType Directory -Force -Path $path 
} 

# token request 
$token = Invoke-WebRequest -Uri $imageuri | ConvertFrom-Json | Select -expand token 

# getting image manifest 
$headers = @{} 
$headers.add("Authorization", "Bearer $token") 
# this header is needed to get manifest in correct format: https://docs.docker.com/registry/spec/manifest-v2-2/ 
$headers.add("Accept", "application/vnd.docker.distribution.manifest.v2+json") 
$manifest = Invoke-Webrequest -Headers $headers -Method GET -Uri $taguri | ConvertFrom-Json 

# downloading config json 
$configSha = $manifest | Select -expand config | Select -expand digest 
$config = ".\$path\config.json" 
Invoke-Webrequest -Headers @{Authorization="Bearer $token"} -Method GET -Uri $bloburi$configSha -OutFile $config 

# generating manifest.json 
$manifestJson = @{} 
$manifestJson.add("Config", "config.json") 
$manifestJson.add("RepoTags",@("${image}:${tag}")) 

# downloading layers 
$layers = $manifest | Select -expand layers | Select -expand digest 
$blobtmp = ".\$path\blobtmp" 

#downloading blobs 
$layersJson = @() 
foreach ($blobelement in $layers) { 
    # making so name doesnt start with 'sha256:' 
    $fileName = "$blobelement.gz" -replace 'sha256:' 
    $newfile = ".\$path\$fileName" 
    $layersJson += @($fileName) 

    # token expired after 5 minutes, so requesting new one for every blob just in case 
    $token = Invoke-WebRequest -Uri $imageuri | ConvertFrom-Json | Select -expand token 
    
    Invoke-Webrequest -Headers @{Authorization="Bearer $token"} -Method GET -Uri $bloburi$blobelement -OutFile $blobtmp 
    
    Copy-Item $blobtmp $newfile -Force -Recurse 
} 

# removing temporary blob 
Remove-Item $blobtmp 

# saving manifest.json 
$manifestJson.add("Layers", $layersJson) 
ConvertTo-Json -Depth 5 -InputObject @($manifestJson) | Out-File -Encoding ascii ".\$path\manifest.json" 

# postprocessing
echo "copy generated folder to your docker machine" 
echo "tar -cvf imagename.tar *" 
echo "docker load < imagename.tar"

after sctipt downloads blobs and generates config and manifest jsons download it to docker machine and execute two following commands:

tar -cvf imagename.tar *
docker load < imagename.tar

first creates an archive, 2nd uploads image archive to docker

0
0

Another tool that could be used to download docker images is P2IWD.

2
  • There doesn’t appear to be any code in that repository.
    – igal
    Commented Mar 6, 2022 at 13:24
  • @igal version 0.1.0 has been released. It is capable to download docker images from a Nexus3 server and to upload them to another one.
    – 030
    Commented Mar 6, 2022 at 22:11
0

You can manually download a docker image with curl by:

  1. getting a free auth token
  2. download the manifest
  3. download the layers (blob hashes specified in the manifest)

Example

For example, I'll show how you can download the hitch package from hub.docker.com.

Get an Auth Token

Execute the following command to get an authentication token from Docker Hub within the scope of the 'hitch' package's namespace.

# get a JSON with an anonymous token
curl -so "token.json" "https://auth.docker.io/token?service=registry.docker.io&scope=<resourcetype>:<component>/<component>:<action>";

# extract token from JSON
token=$(cat token.json | jq -jr ".token")

ⓘ INFO: If you're like me and wondering where the heck is the OCI spec that defines this /token endpoint for authentication, know that it doesn't exist 🤦

To learn more about the syntax of this URL and 'scope' GET variable, see docker's Token Scope Documentation.

The above commands will get a free/temporary token that you can use in subsequent API calls. If all went well, there will be no output from these commands. Here's an example execution

user@disp7456:~$ curl -so "token.json" "https://auth.docker.io/token?service=registry.docker.io&scope=repository:library/hitch:pull";
user@disp7456:~$ 

user@disp7456:~$ token=$(cat token.json | jq -jr ".token")
user@disp7456:~$ 

Download the Manifest

We can download the manifest for the '1.8.0-1' tag of the 'hitch' package with the 'GET /v2/<name>/manifests/<reference>' API endpoint.

curl -o manifest.json -s -H "Authorization: Bearer ${token}" https://registry-1.docker.io/v2/library/<package_name>/manifests/<tag>

And here's an example execution that downloads the manifest for the '1.8.0-1' tag of the 'hitch' package


user@disp7456:~$ curl -o manifest.json -s -H "Authorization: Bearer ${token}" https://registry-1.docker.io/v2/library/hitch/manifests/1.8.0-1
user@disp7456:~$ 

user@disp7456:~$ ls
manifest.json
user@disp7456:~$ 

Parse the Manifest

The previous step downloaded a file named 'manifest.json'. This 'manifest.json' file lists all of the "layers" that make up the image of the 'hitch' package.

Each "layer" consists of a tarball and some metadata about the layer in json format. The information that we need to download the layer's tar file is located in the 'manifest.json' file. And the metadata about each layer is also in the 'manifest.json' file.

The format of the 'manifest.json' file is 'vnd.docker.distribution.manifest.v1+json', which is defined in Image Manifest Version 2, Schema 1.

Most importantly, the 'manifest.json' file contains two parallel arrays of the same length:

  1. fsLayers[]
  2. history[]

Consider this truncated snippet of the manifest for the 'hitch' package's '1.8.0-1' tag:

{
   "schemaVersion": 1,
   "name": "library/hitch",
   "tag": "1.8.0-1",
   "architecture": "amd64",
   "fsLayers": [
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:3148f4af0a813bcff0a3ed2562aabfb1b596b52ef36eb5eb4d82ce836350b73a"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:a0e9543db8c1238572466cf00b55436bc7b7e849f7cb305128f391a94b75c2fc"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:728328ac3bde9b85225b1f0d60f5c149f5635a191f5d8eaeeb00e095d36ef9fd"
      }
   ],
   "history": [
      {
         "v1Compatibility": "{\"architecture\":\"amd64\",\"config\":{\"Hostname\":\"\",\"Domainname\":\"\",\"User\":\"\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"ExposedPorts\":{\"443/tcp\":{}},\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\"],\"Cmd\":[],\"Image\":\"sha256:996009c7d7eb032c9ea750e5decc1f8aedbf4530b892cf4ebc7716e1458f36d9\",\"Volumes\":null,\"WorkingDir\":\"/etc/hitch\",\"Entrypoint\":[\"docker-hitch-entrypoint\"],\"OnBuild\":null,\"Labels\":null},\"container\":\"0ff54ee96c4bbfe77da3b2124720ef95c6154d3bc1d3e40a168920dd818367c4\",\"container_config\":{\"Hostname\":\"0ff54ee96c4b\",\"Domainname\":\"\",\"User\":\"\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"ExposedPorts\":{\"443/tcp\":{}},\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\"],\"Cmd\":[\"/bin/sh\",\"-c\",\"#(nop) \",\"CMD []\"],\"Image\":\"sha256:996009c7d7eb032c9ea750e5decc1f8aedbf4530b892cf4ebc7716e1458f36d9\",\"Volumes\":null,\"WorkingDir\":\"/etc/hitch\",\"Entrypoint\":[\"docker-hitch-entrypoint\"],\"OnBuild\":null,\"Labels\":{}},\"created\":\"2024-05-14T05:23:11.666992342Z\",\"docker_version\":\"20.10.23\",\"id\":\"6703605aae83084affcafb4abcc7c556f0e436c4992ae224f1f58e88242328cb\",\"os\":\"linux\",\"parent\":\"c48ca3d95161bbcdfcaa2e016a675965d55f4f06147ef4445c69347c5965f188\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"c48ca3d95161bbcdfcaa2e016a675965d55f4f06147ef4445c69347c5965f188\",\"parent\":\"1d2af5a156bbc461d98824c2f6bfe295327d4419105c0b7f88f14cb28d0bb240\",\"created\":\"2024-05-14T05:23:11.581588417Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop)  EXPOSE 443\"]},\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"1d2af5a156bbc461d98824c2f6bfe295327d4419105c0b7f88f14cb28d0bb240\",\"parent\":\"8f914c821cbe154cc6677bceb043669e4295ad5cfb6409efa9c2ec1beba75fad\",\"created\":\"2024-05-14T05:23:11.489285564Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop)  ENTRYPOINT [\\\"docker-hitch-entrypoint\\\"]\"]},\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"8f914c821cbe154cc6677bceb043669e4295ad5cfb6409efa9c2ec1beba75fad\",\"parent\":\"a8d8314458142ee2a4ebccb19f48b6f9c696100103c3d49cbbe7ecd2575120e5\",\"created\":\"2024-05-14T05:23:11.403178706Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) COPY file:1abf3c94dce5dc9f6617dc8d36a6fe6f4f7236189d4819f16cefb54288e80e0d in /usr/local/bin/ \"]}}"
      },
      {
         "v1Compatibility": "{\"id\":\"a8d8314458142ee2a4ebccb19f48b6f9c696100103c3d49cbbe7ecd2575120e5\",\"parent\":\"5a78b0e89bbae2390b83e60174ae1efc583f766eff7dffaffa747ccb67472d0f\",\"created\":\"2024-05-14T05:23:11.304477182Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) WORKDIR /etc/hitch\"]},\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"5a78b0e89bbae2390b83e60174ae1efc583f766eff7dffaffa747ccb67472d0f\",\"parent\":\"5a12a2c67ff9b5bfad288a4ede18d08c259c301efb85403d08a40ea2ad0eb1f8\",\"created\":\"2024-05-14T05:23:11.160227264Z\",\"container_config\":{\"Cmd\":[\"|5 DISTVER=bullseye PKGCOMMIT=f12ab7958bc4885f3f00311cbca5103d9e6ba794 PKGVER=1 SHASUM=62b3554d668c9d17382415db10898bf661ee76343e4ee364f904457efda6cb1eeee7cb81d7a3897734024812b64b1c0e2dc305605706d81a0c1f6030508bf7e2 SRCVER=1.8.0 /bin/sh -c set -ex;     BASE_PKGS=\\\"apt-utils curl dirmngr dpkg-dev debhelper devscripts equivs fakeroot git gnupg pkg-config\\\";     export DEBIAN_FRONTEND=noninteractive;     export DEBCONF_NONINTERACTIVE_SEEN=true;     tmpdir=\\\"$(mktemp -d)\\\";     cd \\\"$tmpdir\\\";     apt-get update;     apt-get install -y --no-install-recommends $BASE_PKGS;     git clone https://github.com/varnish/pkg-hitch.git;     cd pkg-hitch;     git checkout ${PKGCOMMIT};     rm -rf .git;     curl -Lf https://hitch-tls.org/source/hitch-${SRCVER}.tar.gz -o $tmpdir/orig.tgz;     echo \\\"${SHASUM}  $tmpdir/orig.tgz\\\" | sha512sum -c -;     tar xavf $tmpdir/orig.tgz --strip 1;     sed -i         -e \\\"s/@SRCVER@/${SRCVER}/g\\\"         -e \\\"s/@PKGVER@/${PKGVER:-1}/g\\\"         -e \\\"s/@DISTVER@/$DISTVER/g\\\" debian/changelog;     mk-build-deps --install --tool=\\\"apt-get -o Debug::pkgProblemResolver=yes --yes\\\" debian/control;     sed -i '' debian/hitch*;     dpkg-buildpackage -us -uc -j\\\"$(nproc)\\\";     apt-get -y purge --auto-remove hitch-build-deps $BASE_PKGS;     apt-get -y --no-install-recommends install ../*.deb;     sed -i 's/daemon = on/daemon = off/' /etc/hitch/hitch.conf;     rm -rf /var/lib/apt/lists/* \\\"$tmpdir\\\"\"]}}"
      },
      {
         "v1Compatibility": "{\"id\":\"5a12a2c67ff9b5bfad288a4ede18d08c259c301efb85403d08a40ea2ad0eb1f8\",\"parent\":\"c03ad9230005f64133de4501e14a882ef25f03443da4da55ca002d5619f998be\",\"created\":\"2024-05-14T05:21:33.061082853Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop)  ARG SHASUM=62b3554d668c9d17382415db10898bf661ee76343e4ee364f904457efda6cb1eeee7cb81d7a3897734024812b64b1c0e2dc305605706d81a0c1f6030508bf7e2\"]},\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"c03ad9230005f64133de4501e14a882ef25f03443da4da55ca002d5619f998be\",\"parent\":\"24e7aee556d6a38bfa2e13430db8a998c023a2920017eabc0b3bf0dd7661bf7d\",\"created\":\"2024-05-14T05:21:32.967727298Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop)  ARG PKGCOMMIT=f12ab7958bc4885f3f00311cbca5103d9e6ba794\"]},\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"24e7aee556d6a38bfa2e13430db8a998c023a2920017eabc0b3bf0dd7661bf7d\",\"parent\":\"f0d07a99d7d1f0b849a4cbe8fc4552d374f4448c2e7f8bfd908aa43132c4ec34\",\"created\":\"2024-05-14T05:21:32.875807605Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop)  ARG DISTVER=bullseye\"]},\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"f0d07a99d7d1f0b849a4cbe8fc4552d374f4448c2e7f8bfd908aa43132c4ec34\",\"parent\":\"65c7b6d17437bf7a3216e2fea283071e9b5c0d71c6b97472baa8807a30b5d9d8\",\"created\":\"2024-05-14T05:21:32.781941821Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop)  ARG PKGVER=1\"]},\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"65c7b6d17437bf7a3216e2fea283071e9b5c0d71c6b97472baa8807a30b5d9d8\",\"parent\":\"863a608d086b1bcf7f9b30ccf57260e6cb5d3d793b4e1131aa8f6041b07a7270\",\"created\":\"2024-05-14T05:21:32.682503634Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop)  ARG SRCVER=1.8.0\"]},\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"863a608d086b1bcf7f9b30ccf57260e6cb5d3d793b4e1131aa8f6041b07a7270\",\"parent\":\"e00e363f3a25341591a5a5e724e20ae3e70f0396be8483a07c0b39d25d33fecd\",\"created\":\"2024-05-14T01:28:27.043980081Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop)  CMD [\\\"bash\\\"]\"]},\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"e00e363f3a25341591a5a5e724e20ae3e70f0396be8483a07c0b39d25d33fecd\",\"created\":\"2024-05-14T01:28:26.699066026Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) ADD file:9b38b383dd93169a663eed88edf3f2285b837257ead69dc40ab5ed1fb3f52c35 in / \"]}}"
      }
   ],
   ...
   ]
}

The sha256sum used to download the blob of the first layer is found at the first element of the fsLayers[] array (fsLayers[0]). The metadata about this first layer is found at the first element of the history[] array (history[0]).

The sha256sum used to download the blob of the second layer is found at second element of the fsLayers[] array (fsLayers[1]). The metadata about this second layer is found at the second element of the history[] array (history[1]).

Et cetera...

Download the Layers

So how do we download each of these layers separately, yet organize them such that we can later import them as a single image into docker? The answer to that lies in the Docker Image Specification v1.0.0.

The above spec provides an example tree of the files:

For example, here's what the full archive of library/busybox is (displayed in tree format):


.
├── 5785b62b697b99a5af6cd5d0aabc804d5748abbb6d3d07da5d1d3795f2dcc83e
│ ├── VERSION
│ ├── json
│ └── layer.tar
├── a7b8b41220991bfc754d7ad445ad27b7f272ab8b4a2c175b9512b97471d02a8a
│ ├── VERSION
│ ├── json
│ └── layer.tar
├── a936027c5ca8bf8f517923169a233e391cbb38469a75de8383b5228dc2d26ceb
│ ├── VERSION
│ ├── json
│ └── layer.tar
├── f60c56784b832dd990022afc120b8136ab3da9528094752ae13fe63a2d28dc8c
│ ├── VERSION
│ ├── json
│ └── layer.tar
└── repositories\

There are one or more directories named with the ID for each layer in a full image. Each of these directories contains 3 files:

  • `VERSION` - The schema version of the `json` file
  • `json` - The JSON metadata for an image layer
  • `layer.tar` - The Tar archive of the filesystem changeset for an image
    layer.

The content of the VERSION files is simply the semantic version of the JSON metadata schema:

1.0\

And the repositories file is another JSON file which describes names/tags:

{  
    "busybox":{  
        "latest":"5785b62b697b99a5af6cd5d0aabc804d5748abbb6d3d07da5d1d3795f2dcc83e"
    }
}

Every key in this object is the name of a repository, and maps to a collection of tag suffixes. Each tag maps to the ID of the image represented by that tag.

So, as shown in the quote from the spec above, what we need to do is to create a set of directories -- one for each layer -- named after the layer's ID. Each of these layer-specific directories must contain 3 files:

  1. The actual tarball of the layer (named 'layer.tar'),
  2. The metadata of the layer (in a file literally named 'json' -- with no file extension)
  3. A file named 'VERSION' whose contents is literally just '1.0'

All of the information that we need is in the 'manifest.json' file that we already downloaded. Let's just loop through each layer that it defines, download the layer.tar tarball, create the 'json' metadata file, and hard-code the 'VERSION' file with the following BASH snippet

num_layers=$(cat manifest.json | jq -r ".history | length")

for ((i = 0 ; i < $num_layers ; i++)); do
    layer_blobSum=$(cat manifest.json | jq -r ".fsLayers[$i].blobSum")
    layer_metadata=$(cat manifest.json | jq -r ".history[$i].v1Compatibility")
    layer_id=$(echo $layer_metadata | jq -r ".id")

    echo $layer_id
    echo $layer_blobSum

    mkdir -p "layers/$layer_id"
    echo "1.0" > "layers/$layer_id/VERSION"
    echo $layer_metadata > "layers/$layer_id/json"
    curl -o "layers/$layer_id/layer.tar" -#LH "Authorization: Bearer ${token}" "https://registry-1.docker.io/v2/library/<package_name>/blobs/${layer_blobSum}"
done

And here's an example execution that executes the above snippet to download all of the layers onto disk in a set of directories as defined by the Docker Image Specification v1.0.0.

user@disp7456:~$ num_layers=$(cat manifest.json | jq -r ".history | length")
user@disp7456:~$ 

user@disp7456:~$ for ((i = 0 ; i < $num_layers ; i++)); do
    layer_blobSum=$(cat manifest.json | jq -r ".fsLayers[$i].blobSum")
    layer_metadata=$(cat manifest.json | jq -r ".history[$i].v1Compatibility")
    layer_id=$(echo $layer_metadata | jq -r ".id")

    echo $layer_id
    echo $layer_blobSum

    mkdir -p "layers/$layer_id"
    echo "1.0" > "layers/$layer_id/VERSION"
    echo $layer_metadata > "layers/$layer_id/json"
    curl -o "layers/$layer_id/layer.tar" -#LH "Authorization: Bearer ${token}" "https://registry-1.docker.io/v2/library/hitch/blobs/${layer_blobSum}"
done
6703605aae83084affcafb4abcc7c556f0e436c4992ae224f1f58e88242328cb
sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
################################################################ 100.0%
c48ca3d95161bbcdfcaa2e016a675965d55f4f06147ef4445c69347c5965f188
sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
################################################################ 100.0%
1d2af5a156bbc461d98824c2f6bfe295327d4419105c0b7f88f14cb28d0bb240
sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
################################################################ 100.0%
8f914c821cbe154cc6677bceb043669e4295ad5cfb6409efa9c2ec1beba75fad
sha256:3148f4af0a813bcff0a3ed2562aabfb1b596b52ef36eb5eb4d82ce836350b73a
################################################################ 100.0%
a8d8314458142ee2a4ebccb19f48b6f9c696100103c3d49cbbe7ecd2575120e5
sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
################################################################ 100.0%
5a78b0e89bbae2390b83e60174ae1efc583f766eff7dffaffa747ccb67472d0f
sha256:a0e9543db8c1238572466cf00b55436bc7b7e849f7cb305128f391a94b75c2fc
################################################################ 100.0%
5a12a2c67ff9b5bfad288a4ede18d08c259c301efb85403d08a40ea2ad0eb1f8
sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
################################################################ 100.0%
c03ad9230005f64133de4501e14a882ef25f03443da4da55ca002d5619f998be
sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
################################################################ 100.0%
24e7aee556d6a38bfa2e13430db8a998c023a2920017eabc0b3bf0dd7661bf7d
sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
################################################################ 100.0%
f0d07a99d7d1f0b849a4cbe8fc4552d374f4448c2e7f8bfd908aa43132c4ec34
sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
################################################################ 100.0%
65c7b6d17437bf7a3216e2fea283071e9b5c0d71c6b97472baa8807a30b5d9d8
sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
################################################################ 100.0%
863a608d086b1bcf7f9b30ccf57260e6cb5d3d793b4e1131aa8f6041b07a7270
sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
################################################################ 100.0%
e00e363f3a25341591a5a5e724e20ae3e70f0396be8483a07c0b39d25d33fecd
sha256:728328ac3bde9b85225b1f0d60f5c149f5635a191f5d8eaeeb00e095d36ef9fd
################################################################ 100.0%
user@disp7456:~$ 

user@disp7456:~$ tree layers/
layers/
├── 1d2af5a156bbc461d98824c2f6bfe295327d4419105c0b7f88f14cb28d0bb240
│   ├── json
│   ├── layer.tar
│   └── VERSION
├── 24e7aee556d6a38bfa2e13430db8a998c023a2920017eabc0b3bf0dd7661bf7d
│   ├── json
│   ├── layer.tar
│   └── VERSION
├── 5a12a2c67ff9b5bfad288a4ede18d08c259c301efb85403d08a40ea2ad0eb1f8
│   ├── json
│   ├── layer.tar
│   └── VERSION
├── 5a78b0e89bbae2390b83e60174ae1efc583f766eff7dffaffa747ccb67472d0f
│   ├── json
│   ├── layer.tar
│   └── VERSION
├── 65c7b6d17437bf7a3216e2fea283071e9b5c0d71c6b97472baa8807a30b5d9d8
│   ├── json
│   ├── layer.tar
│   └── VERSION
├── 6703605aae83084affcafb4abcc7c556f0e436c4992ae224f1f58e88242328cb
│   ├── json
│   ├── layer.tar
│   └── VERSION
├── 863a608d086b1bcf7f9b30ccf57260e6cb5d3d793b4e1131aa8f6041b07a7270
│   ├── json
│   ├── layer.tar
│   └── VERSION
├── 8f914c821cbe154cc6677bceb043669e4295ad5cfb6409efa9c2ec1beba75fad
│   ├── json
│   ├── layer.tar
│   └── VERSION
├── a8d8314458142ee2a4ebccb19f48b6f9c696100103c3d49cbbe7ecd2575120e5
│   ├── json
│   ├── layer.tar
│   └── VERSION
├── c03ad9230005f64133de4501e14a882ef25f03443da4da55ca002d5619f998be
│   ├── json
│   ├── layer.tar
│   └── VERSION
├── c48ca3d95161bbcdfcaa2e016a675965d55f4f06147ef4445c69347c5965f188
│   ├── json
│   ├── layer.tar
│   └── VERSION
├── e00e363f3a25341591a5a5e724e20ae3e70f0396be8483a07c0b39d25d33fecd
│   ├── json
│   ├── layer.tar
│   └── VERSION
└── f0d07a99d7d1f0b849a4cbe8fc4552d374f4448c2e7f8bfd908aa43132c4ec34
    ├── json
    ├── layer.tar
    └── VERSION

14 directories, 39 files
user@disp7456:~$ 

Finally, besides these layer-specific dirs, we need one additional file (named simply 'repository', with no file extension) at the same height as these dirs. As defined by the Docker Image Specification v1.0.0, this file should state the name & tag of the image, and it points to the first layer of the image.

Note the 0th item in the history[] array is the first layer of the image, so we can create this file with the following command

start_image=$(cat manifest.json | jq -r ".history[0].v1Compatibility")
start_image_id=$(echo $start_image | jq -r ".id")

cat > layers/repositories <<EOF
{
    "<image_name>": { "stable": "$start_image_id" }
}
EOF

And here's an example execution to create our 'repository' file for the 'hitch' package.

user@disp7456:~$ start_image=$(cat manifest.json | jq -r ".history[0].v1Compatibility")
user@disp7456:~$ 

user@disp7456:~$ start_image_id=$(echo $start_image | jq -r ".id")
user@disp7456:~$ 

user@disp7456:~$ cat > layers/repositories <<EOF
{
    "hitch": { "stable": "$start_image_id" }
}
EOF
user@disp7456:~$

user@disp7456:~$ ls layers
1d2af5a156bbc461d98824c2f6bfe295327d4419105c0b7f88f14cb28d0bb240
24e7aee556d6a38bfa2e13430db8a998c023a2920017eabc0b3bf0dd7661bf7d
5a12a2c67ff9b5bfad288a4ede18d08c259c301efb85403d08a40ea2ad0eb1f8
5a78b0e89bbae2390b83e60174ae1efc583f766eff7dffaffa747ccb67472d0f
65c7b6d17437bf7a3216e2fea283071e9b5c0d71c6b97472baa8807a30b5d9d8
6703605aae83084affcafb4abcc7c556f0e436c4992ae224f1f58e88242328cb
863a608d086b1bcf7f9b30ccf57260e6cb5d3d793b4e1131aa8f6041b07a7270
8f914c821cbe154cc6677bceb043669e4295ad5cfb6409efa9c2ec1beba75fad
a8d8314458142ee2a4ebccb19f48b6f9c696100103c3d49cbbe7ecd2575120e5
c03ad9230005f64133de4501e14a882ef25f03443da4da55ca002d5619f998be
c48ca3d95161bbcdfcaa2e016a675965d55f4f06147ef4445c69347c5965f188
e00e363f3a25341591a5a5e724e20ae3e70f0396be8483a07c0b39d25d33fecd
f0d07a99d7d1f0b849a4cbe8fc4552d374f4448c2e7f8bfd908aa43132c4ec34
repositories
user@disp7456:~$

user@disp7456:~$ cat layers/repositories
{
    "hitch": { "stable": "6703605aae83084affcafb4abcc7c556f0e436c4992ae224f1f58e88242328cb" }
}
user@disp7456:~$ 

Your 'layers/' directory should now be prepared-to-spec for importing the entire image into docker.

For reference, here's the contents of just one of the layers:

user@disp7456:~$ ls layers/8f914c821cbe154cc6677bceb043669e4295ad5cfb6409efa9c2ec1beba75fad/
json  layer.tar  VERSION
user@disp7456:~$ 

user@disp7456:~$ cat layers/8f914c821cbe154cc6677bceb043669e4295ad5cfb6409efa9c2ec1beba75fad/VERSION 
1.0
user@disp7456:~$ 

user@disp7456:~$ cat layers/8f914c821cbe154cc6677bceb043669e4295ad5cfb6409efa9c2ec1beba75fad/json 
{"id":"8f914c821cbe154cc6677bceb043669e4295ad5cfb6409efa9c2ec1beba75fad","parent":"a8d8314458142ee2a4ebccb19f48b6f9c696100103c3d49cbbe7ecd2575120e5","created":"2024-05-14T05:23:11.403178706Z","container_config":{"Cmd":["/bin/sh -c #(nop) COPY file:1abf3c94dce5dc9f6617dc8d36a6fe6f4f7236189d4819f16cefb54288e80e0d in /usr/local/bin/ "]}}
user@disp7456:~$ 

user@disp7456:~$ sha256sum layers/8f914c821cbe154cc6677bceb043669e4295ad5cfb6409efa9c2ec1beba75fad/layer.tar 
3148f4af0a813bcff0a3ed2562aabfb1b596b52ef36eb5eb4d82ce836350b73a  layers/8f914c821cbe154cc6677bceb043669e4295ad5cfb6409efa9c2ec1beba75fad/layer.tar
user@disp7456:~$ 

user@disp7456:~$ tar -tvf layers/8f914c821cbe154cc6677bceb043669e4295ad5cfb6409efa9c2ec1beba75fad/layer.tar 
drwxr-xr-x 0/0               0 2024-05-12 19:00 usr/
drwxr-xr-x 0/0               0 2024-05-12 19:00 usr/local/
drwxr-xr-x 0/0               0 2024-05-14 00:23 usr/local/bin/
-rwxrwxr-x 0/0             319 2024-05-14 00:21 usr/local/bin/docker-hitch-entrypoint
user@disp7456:~$ 

Load the Image

Finally, you can copy the whole layers directory into your restricted environment and then load the layers as one image into docker with docker image load

tar -cC layers . | docker image load

Here's an example execution

user@disp7456:~$ docker image ls
REPOSITORY   TAG       IMAGE ID   CREATED   SIZE
user@disp7456:~$ 

user@disp7456:~$ tar -cC layers . | docker load
e00e363f3a25: Loading layer [==================================================>]  31.43MB/31.43MB
863a608d086b: Loading layer [==================================================>]      32B/32B
65c7b6d17437: Loading layer [==================================================>]      32B/32B
f0d07a99d7d1: Loading layer [==================================================>]      32B/32B
24e7aee556d6: Loading layer [==================================================>]      32B/32B
c03ad9230005: Loading layer [==================================================>]      32B/32B
5a12a2c67ff9: Loading layer [==================================================>]      32B/32B
5a78b0e89bba: Loading layer [==================================================>]  1.573MB/1.573MB
a8d831445814: Loading layer [==================================================>]      32B/32B
8f914c821cbe: Loading layer [==================================================>]     415B/415B
1d2af5a156bb: Loading layer [==================================================>]      32B/32B
c48ca3d95161: Loading layer [==================================================>]      32B/32B
6703605aae83: Loading layer [==================================================>]      32B/32B
user@disp7456:~$ 

user@disp7456:~$ docker image ls
REPOSITORY   TAG       IMAGE ID       CREATED       SIZE
hitch        stable    f07eadb841be   3 weeks ago   85.1MB
user@disp7456:~$ 

Attribution

The example above was copied from Manually Downloading Container Images (Docker, Github Packages)

-1

If your company has nexus or a similar repository, they may already have set up nexus to pull in what you need. You could try adding your repo's host/ip in front of the relative docker hub path.

Examples:

docker pull nexus.example.com:18443/nginx
docker pull nexus.example.com:18443/selenium/node-chrome
docker pull nexus.example.com:18443/postgres:9.4

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.