Containers are pretty amazing. They allow simple processes to act like virtual machines. Underneath this elegance is a set of patterns and practices that ultimately makes everything work. At the root of the design is layers. Layers are the fundamental way of storing and distributing the contents of a containerized file system. The design is both surprisingly simple and at the same time very powerful. In today’s post, I’ll explain what layers are and how they work conceptually.
Building layered images
When you create an image, you typically use a
Dockerfile to define the contents of the container. It contains a series of commands, such as:
2RUN echo "hello" > /work/message.txt
3COPY content.txt /work/content.txt
4RUN rm -rf /work/message.txt
Under the covers, the container engine will execute these commands in order, creating a “layer” for each. But what is really happening? It’s easiest to think of each layer as a directory that holds all of the modified files.
Let’s step through an example of a possible implementation approach.
FROM scratchindicates that this container is starting with no contents. This is the first layer, and it could be represented by an empty directory,
- Create a second directory,
/img/layer2and copy everything from
/img/layer1into it. Then, execute the next command from the Dockerfile (which writes a file to
/work/message.txt). Those contents are written to
/img/layer2/work/message.txt. This is the second layer.
- Create a third directory,
/img/layer3copying everything from
img/layer2into it. The next Dockerfile command requires copying
content.txtfrom the host to that directory. That file is written to
/img/layer3/work/content.txt. This is the third layer.
- Finally, create a fourth directory,
/img/layer4copying everything from
img/layer3into it. The next command deletes the message file,
img/layer4/work/message.txt. This is the fourth layer.
To share these layers, the easiest approach is to create a compressed
.tar.gz for each directory. To reduce the total file size, any files that are unmodified copies of data from a previous layer would be removed. To make it clear when a file was deleted, a “whiteout file” could be used as a placeholder. The file would simply prefix
.wh. to the original filename. For example, the fourth layer would replace the deleted file with a placeholder named
.wh.message.txt. When a layer is unpacked, any files that start with
.wh. can be deleted.
Continuing our example, the compressed files would contain:
message.txt was not modified)
message.txt was deleted).
content.txt was not modified, so it is not included.
Building lots of images this way would result in lots of “layer1” directories. To make sure the name is unique, the compressed file is named based on a digest of the contents. This is similar to how Git works. It has the benefit of identifying identical content while identifying any corruption of the files while downloading. If the digest of the contents does not match the file name, the file is corrupt.
To make the results reproducible, one more thing is required — a file that explains how to order the layers (a manifest). The manifest would identify which files to be downloaded the order for unpacking them. This enables recreating the directory structures. It also provides an important benefit: layers can be reused and shared between images. This minimizes the local storage requirements.
In practice, there are more optimizations available. For example,
FROM scratch really means there is no parent layer, so our example really starts with the contents of
layer2. The engine can also look at the files used in the build to determine whether or not a layer needs to be recreated. This is the basis for layer caching, which minimizes the need to build or recreate layers. As an additional optimizing, layers that don’t depend on the previous layer can use
COPY --link to indicate that the layer won’t need to delete or modify any files from the previous layer. This allows the compressed layer file to be created in parallel to the other steps.
Before a container can run, it needs a file system to mount. In essence, it needs a directory with all of the files that need to be available. The compressed layer files contain the components of the file system, but they can’t be directly mounted and used. Instead, they need to be unpacked and organized into a file system. This unpacked directory is called a snapshot (well, it’s one of a few things with that name 😄).
The process of creating a snapshot is the opposite of image building. It starts by downloading the manifest and building a list of layers to download. For each layer, a directory is created with the contents of the layer’s parent. This directory is called the active snapshot. Next, a diff applier is responsible for unpacking the compressed layer file and applying the changes to the active snapshot. The resulting directory is then called a committed snapshot. The final committed snapshot is the one that is mounted as the container’s file system.
Using our earlier example:
- The initial layer,
FROM scratch, means that we can start with the next layer and an empty directory. There is no parent.
- A directory for
layer2is created. This empty directory is now an active snapshot. The file
layer2.tar.gzis downloaded, validated (by comparing the digest to the file name), and unpacked into the directory. The result is a directory containing
/work/message.txt. This is the first committed snapshot.
- A directory for
layer3is created, and the contents of
layer2are copied into it. This is a new active snapshot. The file
layer3.tar.gzis downloaded, validated, and unpacked. The result is a directory containing
/work/content.txt. This is now the second committed snapshot.
- A directory for
layer4is created, and the contents of
layer3are copied into it. The file
layer4.tar.gzis downloaded, validated, and unpacked. The diff applier recognizes the whiteout file,
/work/.wh.message.txt, and deletes
/work/message.txt. This leaves just
/work/content.txt. This is the third committed snapshot.
layer4was the last layer, it is the basis for a container. To enable it to support read and write operations, a new snapshot directory is created and the contents of
layer4are copied into it. This directory is mounted as the container’s file system. Any changes made by the running container will happen in this directory.
If any of those directories already exist, it indicates that another image had the same dependency. As a result, the engine can skip the download and diff applier. It can use the layer as-is. In practice, each of these directories and files is named based on the digest of the contents to make that easier to identify. For example, a set of snapshots might look like this:
3 ├─ 635944d2044d0a54d01385271ebe96ec18b26791eb8b85790974da36a452cc5c
4 ├─ 9de59f6b211510bd59d745a5e49d7aa0db263deedc822005ed388f8d55227fc1
5 ├─ fb0624e7b7cb9c912f952dd30833fb2fe1109ffdbcc80d995781f47bd1b4017f
6 └─ fb124ec4f943662ecf7aac45a43b096d316f1a6833548ec802226c7b406154e9
The actual snapshot system supports plugins that can improve some of these behaviors. For example, it can allow the snapshots to be pre-composed and unpacked, speeding up the process. This allows the snapshots to be stored remotely. It also allows for special optimizations, such as just-in-time downloading of the needed files and layers.
While it’s easy to mount, the snapshot approach we just described creates a lot of file churn and lots of duplicate files. This slows down the process of starting a container the first time and wastes space. Thankfully, this is one of many aspects of the containerization process that can be handled by the file system. Linux natively supports mounting directories as overlays, implementing most of the process for us.
In Linux (or a Linux container running as
--privileged or with
tmpfsmount (memory-based file system that will be used to explore the overlay process)
1mkdir /tmp/overlay 2mount -t tmpfs tmpfs /tmp/overlay
Create directories for our process. We’ll use
lowerfor the lower (parent) layer,
upperfor the upper (child) layer,
workas a working directory for the file system, and
mergedto contain the merged file system.
Create some files for the experiment. Optionally, you can add files in
1cd /tmp/overlay 2echo hello > lower/hello.txt 3echo "I'm only here for a moment" > lower/delete-me.txt 4echo message > upper/upper-message.txt
Mount these directories as an
overlaytype file system. This will create a new file system in the
mergeddirectory that contains the combined contents of the
workdirectory will be used to track changes to the file system.
1mount -t overlay overlay -o lowerdir=lower,upperdir=upper,workdir=work merged
Explore the file system. You’ll notice that
mergedcontains the combined contents of
lower. Then, make some changes:
1rm -rf merged/delete-me.txt 2echo "I'm new" > merged/new.txt 3echo world >> merged/hello.txt
delete-me.txtis removed from
mergedand a new file,
new.txtis created in the same directory. If you
treethe directories, you’ll see something interesting:
|-- lower | |-- delete-me.txt | `-- hello.txt |-- merged | |-- hello.txt | |-- new.txt | `-- upper-message.txt |-- upper | |-- delete-me.txt | |-- hello.txt | |-- new.txt | `-- upper-message.txt
ls -l uppershows
1total 12 2c--------- 2 root root 0, 0 Jan 20 00:17 delete-me.txt 3-rw-r--r-- 1 root root 12 Jan 20 00:20 hello.txt 4-rw-r--r-- 1 root root 8 Jan 20 00:17 new.txt 5-rw-r--r-- 1 root root 8 Jan 20 00:17 upper-message.txt
merged shows the effects of our changes,
upper (as the parent layer) stores the changes similar to the example in our manual process. It contains the new file,
new.txt and the modified
hello.txt. It has also created a whiteout file. For the overlay filesystem, this involves replacing the file with a character device (and a 0, 0 device number). In short, it has everything we need to be able to package up the directories!
You can see how this approach could also be used to implement a snapshot system. The
mount command can natively take a colon (
:) delimited list of
lowerdir paths, all of which are unioned together into a single file system. This is part of the nature of modern containers – the containers are composed using native operating system features.
That’s really all there is to creating a basic system. In fact, the
containerd runtime used by Kubernetes (and the recently release Docker Desktop 4.27.0) uses a similar approach to build and manage its images (with the deeper details covered in
Content Flow). Hope this has helped to demystify the way container images work!