Awesome
diffoci: diff for Docker and OCI container images
diffoci compares Docker and OCI container images for helping reproducible builds.
[!NOTE] "OCI" here refers to the "Open Container Initiative", not to the "Oracle Cloud Infrastructure".
$ diffoci diff --semantic alpine:3.18.2 alpine:3.18.3
TYPE NAME INPUT-0 INPUT-1
File lib/libcrypto.so.3 4518b7d5f6563f81ca1b3d857bbd7008d686545d8211283240506449618b2858 d21ccbb32c1e98340027504e4d951ab4c38450f1f211a4157fa9e962216ac625
File usr/bin/iconv 8c54b8962ef07047032facf57cacb2f19eecfbefe03ad001e8c6c2b332e0d334 07115db0a4bd082f746ddbc6433b96397adcb523956417cb8376cd1ab4faf3d6
File usr/lib/engines-3/capi.so 1b0508fa6be2efe1412f59da46b3963c7611696563b6b2b699b1aa9dba447b5a cc03f5d58b206389f9560e8e7f15495d2bd2f384793549f63f3cb3cb9c71b466
File bin/busybox 6bf2a4358d5c26033d2d2e30ef9ce0ba9e3a9b34d81f0964f1d127123ae3e827 89f85a8b92bdb5ada51bd8ee0d9d687da1a759a0d80f8d012e7f8cbb3933a3a0
File etc/os-release fb844374742438cf1b4e675dcd7d87c2fd6fbdb7cc7be30c62d4027240474aaf e08e943282c5d38f99bfde311c7d5759a4578f92fca5943e5b1351e8cd472892
File usr/lib/engines-3/padlock.so 670f9a084f97974b12c81cc7a88fce5ec9f47dcee00da3192a17577a5298e62f 15fb67a501ad1293034dd16106edbdee0c07803ebcbbeb5a8123b4e784c1491f
File usr/bin/getconf d314880d6288d48e8223b3da02d00605f6a993b669593c1c1a0fa5fd95339d48 040fc0ddb83193809919381bb23d89e4efefdc207e18ef0f3d0e0d9977bf2b58
File lib/ld-musl-x86_64.so.1 c6b3288ba48945a21ede7ccf6f7a257d41bacf2c061236726ff9b5def383a766 35002c47957674f588389c0f3ca23b01adab091ea5773326e1c7782e5ece1207
File usr/lib/engines-3/afalg.so 2aaa6a94f0e07f556e77faa8bec55321d0fe138f27193fdcd6ef8ccef051c7ba 9ef6555b3594798ce5a05dfd79b996397642b367c75b5cab0c752050d28e6138
File lib/apk/db/installed d8b49a733ba6590033ae75f27a52f83281b693ba46751a9e488bf12dbdb06c61 c5cd4d9ed0a78abe602cc1e5fb29c84bef033c99d7c7f5e15fdb76b6c6cf8ad0
File lib/apk/db/scripts.tar fbc51430c9f8b6a1a547281858fb36dff7947a2660a8bb6937b48f12361e3bb1 7d28a05788bda8fc5ec835257d5204d6d45c002458b04ad4981bce3be733c80f
File etc/alpine-release 8fc1c65f6ef13f8ef573d59100eb183cf5a66aa7e94a6e6cba1dd22e7a0af51b 7cc26f207ef55a1422daecc84d155e9fa50fd170574807e73e55823dd81407d9
File etc/ssl/misc/tsget.pl 559f94eb47d2d6f81c361e334fd882596858c7934a8b0111177f535a910f2990 3feab60fb8d102da8746aa2a422ba77c60dda6b4ae9ac33056fe82bfc071969d
File lib/apk/db/triggers f78b6968b2c85c453a9dd89368a9fc78788d7c34aeb202544a41714e90544a37 ff37f223b335fc42d5a563dd268cbf476411d8fdd2b88addade5b5f715f0425a
File lib/libssl.so.3 c22f9f45c1266ebdcc5f89c2f3471e89ffc5b2a4cd299f672156723270994133 91ad7b1cc8cf5575afeeb202c0fd1fefb63ceea1492491218f3132813eb04e49
File usr/lib/engines-3/loader_attic.so 4f419a09b9f608e753045d83c255818c793913c274f724eb10d8ca8f474ae457 959301d5c88fab262a0631f1c87d02e1ef52215de35e0316af4513014a449b7f
File usr/bin/getent 1c6aa066998ddd019b6df0142ffcf8b1f4307593dc42b92e7fc82cd601905f75 c22523bc4d2208c8df51b9c7b87ad5596aaf171a4e69d169f007ada8241c5d09
File usr/lib/ossl-modules/legacy.so 5b677eca0c3a3ac53c1a49fbac49534ea2862c62416d14ec6053f2080d5bae50 65b55bc81cbab89e7d2d7e5636455ef86642ae8c377dec3924103d3513ede888
Installation
Binary
Binaries are available for Linux and macOS: https://github.com/reproducible-containers/diffoci/releases
Source
Needs Go 1.21 or later.
go install github.com/reproducible-containers/diffoci/cmd/diffoci@latest
Basic usage
Strict mode
diffoci diff IMAGE0 IMAGE1
[!TIP] Non-Linux users typically have to specify
--platformexplicitly. e.g.,diffoci diff --platform=linux/amd64 IMAGE0 IMAGE1.
The strict mode is often too strict. Consider using the non-strict mode (see below).
Non-strict (aka "semantic") mode
diffoci diff --semantic IMAGE0 IMAGE1
Non-strict mode ignores:
- Timestamps
- Build histories
- File ordering in tar layers
- Image name annotations
Advanced usage
Dumping conflicting files
Set --report-dir=DIR to dump conflicting files in the specified dir.
$ diffoci diff --semantic --report-dir=/tmp/diff alpine:3.18.2 alpine:3.18.3
TYPE NAME INPUT-0 INPUT-1
...
File etc/alpine-release 8fc1c65f6ef13f8ef573d59100eb183cf5a66aa7e94a6e6cba1dd22e7a0af51b 7cc26f207ef55a1422daecc84d155e9fa50fd170574807e73e55823dd81407d9
...
$ diff -ur /tmp/diff/input-0/ /tmp/diff/input-1/
Binary files /tmp/diff/input-0/manifests-0/layers-0/bin/busybox and /tmp/diff/input-1/manifests-0/layers-0/bin/busybox differ
diff -ur /tmp/diff/input-0/manifests-0/layers-0/etc/alpine-release /tmp/diff/input-1/manifests-0/layers-0/etc/alpine-release
--- /tmp/diff/input-0/manifests-0/layers-0/etc/alpine-release 2023-06-15 00:03:14.000000000 +0900
+++ /tmp/diff/input-1/manifests-0/layers-0/etc/alpine-release 2023-08-07 22:09:12.000000000 +0900
@@ -1 +1 @@
-3.18.2
+3.18.3
...
Accessing containerd images
diffoci uses the containerd image store by default when containerd v1.7 or later is running.
The default namespace is default.
To explicitly enable containerd:
diffoci --backend=containerd --containerd-socket=SOCKET --containerd-namespace=NAMESPACE
To explicitly disable containerd:
diffoci --backend=local
Accessing Docker images
To access Docker images that are not pushed to a registry, prepend docker:// to the image name:
docker build -t foo ~/foo
docker build -t bar ~/bar
diffoci diff docker://foo docker://bar
You do NOT need to specify a custom --backend to access Docker images.
Accessing Podman images
To access Podman images that are not pushed to a registry, prepend podman:// to the image name.
See the docker:// example above, and read docker as podman.
Accessing private images
To access private images, create a credential file as ~/.docker/config.json using docker login.
Examples
Non-reproducible Docker Hub images
golang:1.21-alpine3.18
The sources of the official Docker Hub images are available at https://github.com/docker-library.
For example, the source of golang:1.21-alpine3.18
can be found at https://github.com/docker-library/golang/blob/d1ff31b86b23fe721dc65806cd2bd79a4c71b039/1.21/alpine3.18/Dockerfile.
The source can be built as follows:
$ DOCKER_BUILDKIT=0 docker build -t my-golang-1.21-alpine3.18 'https://github.com/docker-library/golang.git#d1ff31b86b23fe721dc65806cd2bd79a4c71b039:1.21/alpine3.18'
...
Successfully tagged my-golang-1.21-alpine3.18:latest
[!NOTE]
DOCKER_BUILDKIT=0is specified here because the officialgolang:1.21-alpine3.18image is currently built with the legacy builder. A future revision of the official image may be built with BuildKit, and in such a case,DOCKER_BUILDKIT=1will rather need to be specified here.
The resulting image binary (my-golang-1.21-alpine3.18) can be compared with the official image binary (golang:1.21-alpine3.18) as follows:
$ diffoci diff docker://golang:1.21-alpine3.18 docker://my-golang-1.21-alpine3.18 --semantic --report-dir=~/diff
INFO[0000] Loading image "docker.io/library/golang:1.21-alpine3.18" from "docker"
docker.io/library/golang:1.21 alpine3.18 saved
Importing elapsed: 2.6 s total: 0.0 B (0.0 B/s)
INFO[0004] Loading image "docker.io/library/my-golang-1.21-alpine3.18:latest" from "docker"
docker.io/library/my golang 1.21 alpine3 saved
Importing elapsed: 2.6 s total: 0.0 B (0.0 B/s)
TYPE NAME INPUT-0 INPUT-1
Layer ctx:/layers-1/layer length mismatch (457 vs 454)
File lib/apk/db/scripts.tar eef110e559acb7aa00ea23ee7b8bddb52c4526cd394749261aa244ef9c6024a4 342eaa013375398497bfc21dff7dd017a647032ec5c486011142c576b7ccc989
Layer ctx:/layers-1/layer name "usr/local/share/ca-certificates/.wh..wh..opq" only appears in input 0
Layer ctx:/layers-1/layer name "usr/share/ca-certificates/.wh..wh..opq" only appears in input 0
Layer ctx:/layers-1/layer name "etc/ca-certificates/.wh..wh..opq" only appears in input 0
Layer ctx:/layers-2/layer length mismatch (13927 vs 13926)
Layer ctx:/layers-2/layer name "usr/local/go/.wh..wh..opq" only appears in input 0
File lib/apk/db/scripts.tar 073bb5094fc5bba800f06661dc7f1325c5cb4250b13209fb9e3eaf4e60e4bfc4 1369581b62bd60304c59556ea85f585bd498040c8fa223243622bb7990833063
Layer ctx:/layers-3/layer length mismatch (4 vs 3)
Layer ctx:/layers-3/layer name "go/.wh..wh..opq" only appears in input 0
[!NOTE] The
--semanticflag is specified to ignore differences of timestamps, image names, and other "boring" attributes. Without this flag, thediffocicommand may print an enourmous amount of output.
In the my-golang-1.21-alpine3.18 image, special files called "Opaque whiteouts" (.wh..wh..opq)
are missing due to filesystem difference between Docker Hub's build machine and the local machine.
Also, the lib/apk/db/scripts.tar file in the layer 1 is not reproducible due to the timestamps of the tar entries inside it.
The differences can be inspected by running the diffoscope command for ~/diff/input-{0,1}/layers-1/lib/apk/db/scripts.tar:
$ sudo apt-get install -y diffoscope
$ diffoscope ~/diff/input-0/layers-1/lib/apk/db/scripts.tar ~/diff/input-1/layers-1/lib/apk/db/scripts.tar
--- /home/suda/diff/input-0/layers-1/lib/apk/db/scripts.tar
+++ /home/suda/diff/input-1/layers-1/lib/apk/db/scripts.tar
├── file list
│ @@ -1,9 +1,9 @@
│ --rwxr-xr-x 0 root (0) root (0) 56 2023-08-09 03:36:47.000000 alpine-baselayout-3.4.3-r1.Q1zwvKMnYs1b6ZdPTBJ0Z7D5P3jyA=.pre-install
│ --rwxr-xr-x 0 root (0) root (0) 983 2023-08-09 03:36:47.000000 alpine-baselayout-3.4.3-r1.Q1zwvKMnYs1b6ZdPTBJ0Z7D5P3jyA=.post-install
│ --rwxr-xr-x 0 root (0) root (0) 755 2023-08-09 03:36:47.000000 alpine-baselayout-3.4.3-r1.Q1zwvKMnYs1b6ZdPTBJ0Z7D5P3jyA=.pre-upgrade
│ --rwxr-xr-x 0 root (0) root (0) 983 2023-08-09 03:36:47.000000 alpine-baselayout-3.4.3-r1.Q1zwvKMnYs1b6ZdPTBJ0Z7D5P3jyA=.post-upgrade
│ --rwxr-xr-x 0 root (0) root (0) 139 2023-08-09 03:36:47.000000 busybox-1.36.1-r2.Q1gQ/L3UBnSjgkFWEHQaUkUDubqdI=.post-install
│ --rwxr-xr-x 0 root (0) root (0) 1239 2023-08-09 03:36:47.000000 busybox-1.36.1-r2.Q1gQ/L3UBnSjgkFWEHQaUkUDubqdI=.post-upgrade
│ --rwxr-xr-x 0 root (0) root (0) 546 2023-08-09 03:36:47.000000 busybox-1.36.1-r2.Q1gQ/L3UBnSjgkFWEHQaUkUDubqdI=.trigger
│ --rwxr-xr-x 0 root (0) root (0) 137 2023-08-09 03:36:47.000000 ca-certificates-20230506-r0.Q1FG8M+7w+dkjV9Vy0mGFWW2t4+Do=.post-deinstall
│ --rwxr-xr-x 0 root (0) root (0) 63 2023-08-09 03:36:47.000000 ca-certificates-20230506-r0.Q1FG8M+7w+dkjV9Vy0mGFWW2t4+Do=.trigger
│ +-rwxr-xr-x 0 root (0) root (0) 56 2023-08-24 07:50:41.000000 alpine-baselayout-3.4.3-r1.Q1zwvKMnYs1b6ZdPTBJ0Z7D5P3jyA=.pre-install
│ +-rwxr-xr-x 0 root (0) root (0) 983 2023-08-24 07:50:41.000000 alpine-baselayout-3.4.3-r1.Q1zwvKMnYs1b6ZdPTBJ0Z7D5P3jyA=.post-install
│ +-rwxr-xr-x 0 root (0) root (0) 755 2023-08-24 07:50:41.000000 alpine-baselayout-3.4.3-r1.Q1zwvKMnYs1b6ZdPTBJ0Z7D5P3jyA=.pre-upgrade
│ +-rwxr-xr-x 0 root (0) root (0) 983 2023-08-24 07:50:41.000000 alpine-baselayout-3.4.3-r1.Q1zwvKMnYs1b6ZdPTBJ0Z7D5P3jyA=.post-upgrade
│ +-rwxr-xr-x 0 root (0) root (0) 139 2023-08-24 07:50:41.000000 busybox-1.36.1-r2.Q1gQ/L3UBnSjgkFWEHQaUkUDubqdI=.post-install
│ +-rwxr-xr-x 0 root (0) root (0) 1239 2023-08-24 07:50:41.000000 busybox-1.36.1-r2.Q1gQ/L3UBnSjgkFWEHQaUkUDubqdI=.post-upgrade
│ +-rwxr-xr-x 0 root (0) root (0) 546 2023-08-24 07:50:41.000000 busybox-1.36.1-r2.Q1gQ/L3UBnSjgkFWEHQaUkUDubqdI=.trigger
│ +-rwxr-xr-x 0 root (0) root (0) 137 2023-08-24 07:50:41.000000 ca-certificates-20230506-r0.Q1FG8M+7w+dkjV9Vy0mGFWW2t4+Do=.post-deinstall
│ +-rwxr-xr-x 0 root (0) root (0) 63 2023-08-24 07:50:41.000000 ca-certificates-20230506-r0.Q1FG8M+7w+dkjV9Vy0mGFWW2t4+Do=.trigger
These differences are boring, but not filtered out by the --semantic flag of the diffoci command, because diffoci is not aware of the formats of the files inside the image layers.
The lib/apk/db/scripts.tar file in the layer 2 has the same issue:
$ diffoscope ~/diff/input-0/layers-2/lib/apk/db/scripts.tar ~/diff/input-1/layers-2/lib/apk/db/scripts.tar
--- /home/suda/diff/input-0/layers-2/lib/apk/db/scripts.tar
+++ /home/suda/diff/input-1/layers-2/lib/apk/db/scripts.tar
├── file list
│ @@ -1,9 +1,9 @@
│ --rwxr-xr-x 0 root (0) root (0) 56 2023-08-09 04:41:27.000000 alpine-baselayout-3.4.3-r1.Q1zwvKMnYs1b6ZdPTBJ0Z7D5P3jyA=.pre-install
│ --rwxr-xr-x 0 root (0) root (0) 983 2023-08-09 04:41:27.000000 alpine-baselayout-3.4.3-r1.Q1zwvKMnYs1b6ZdPTBJ0Z7D5P3jyA=.post-install
│ --rwxr-xr-x 0 root (0) root (0) 755 2023-08-09 04:41:27.000000 alpine-baselayout-3.4.3-r1.Q1zwvKMnYs1b6ZdPTBJ0Z7D5P3jyA=.pre-upgrade
│ --rwxr-xr-x 0 root (0) root (0) 983 2023-08-09 04:41:27.000000 alpine-baselayout-3.4.3-r1.Q1zwvKMnYs1b6ZdPTBJ0Z7D5P3jyA=.post-upgrade
│ --rwxr-xr-x 0 root (0) root (0) 139 2023-08-09 04:41:27.000000 busybox-1.36.1-r2.Q1gQ/L3UBnSjgkFWEHQaUkUDubqdI=.post-install
│ --rwxr-xr-x 0 root (0) root (0) 1239 2023-08-09 04:41:27.000000 busybox-1.36.1-r2.Q1gQ/L3UBnSjgkFWEHQaUkUDubqdI=.post-upgrade
│ --rwxr-xr-x 0 root (0) root (0) 546 2023-08-09 04:41:27.000000 busybox-1.36.1-r2.Q1gQ/L3UBnSjgkFWEHQaUkUDubqdI=.trigger
│ --rwxr-xr-x 0 root (0) root (0) 137 2023-08-09 04:41:27.000000 ca-certificates-20230506-r0.Q1FG8M+7w+dkjV9Vy0mGFWW2t4+Do=.post-deinstall
│ --rwxr-xr-x 0 root (0) root (0) 63 2023-08-09 04:41:27.000000 ca-certificates-20230506-r0.Q1FG8M+7w+dkjV9Vy0mGFWW2t4+Do=.trigger
│ +-rwxr-xr-x 0 root (0) root (0) 56 2023-08-24 07:50:52.000000 alpine-baselayout-3.4.3-r1.Q1zwvKMnYs1b6ZdPTBJ0Z7D5P3jyA=.pre-install
│ +-rwxr-xr-x 0 root (0) root (0) 983 2023-08-24 07:50:52.000000 alpine-baselayout-3.4.3-r1.Q1zwvKMnYs1b6ZdPTBJ0Z7D5P3jyA=.post-install
│ +-rwxr-xr-x 0 root (0) root (0) 755 2023-08-24 07:50:52.000000 alpine-baselayout-3.4.3-r1.Q1zwvKMnYs1b6ZdPTBJ0Z7D5P3jyA=.pre-upgrade
│ +-rwxr-xr-x 0 root (0) root (0) 983 2023-08-24 07:50:52.000000 alpine-baselayout-3.4.3-r1.Q1zwvKMnYs1b6ZdPTBJ0Z7D5P3jyA=.post-upgrade
│ +-rwxr-xr-x 0 root (0) root (0) 139 2023-08-24 07:50:52.000000 busybox-1.36.1-r2.Q1gQ/L3UBnSjgkFWEHQaUkUDubqdI=.post-install
│ +-rwxr-xr-x 0 root (0) root (0) 1239 2023-08-24 07:50:52.000000 busybox-1.36.1-r2.Q1gQ/L3UBnSjgkFWEHQaUkUDubqdI=.post-upgrade
│ +-rwxr-xr-x 0 root (0) root (0) 546 2023-08-24 07:50:52.000000 busybox-1.36.1-r2.Q1gQ/L3UBnSjgkFWEHQaUkUDubqdI=.trigger
│ +-rwxr-xr-x 0 root (0) root (0) 137 2023-08-24 07:50:52.000000 ca-certificates-20230506-r0.Q1FG8M+7w+dkjV9Vy0mGFWW2t4+Do=.post-deinstall
│ +-rwxr-xr-x 0 root (0) root (0) 63 2023-08-24 07:50:52.000000 ca-certificates-20230506-r0.Q1FG8M+7w+dkjV9Vy0mGFWW2t4+Do=.trigger
Depending on the time to build the image, more differences may happen, especially when the Alpine packages on the internet are bumped up.
Conclusion
This example indicates that although the official golang:1.21-alpine3.18 image binary is not fully reproducible, its non-reproducibility is practically negligible, and
this image binary can be assured to be certainly buildable from with the published source.
If the published source is trustable, this image binary can be trusted too.