{"id":11055,"date":"2018-03-31T16:30:56","date_gmt":"2018-03-31T14:30:56","guid":{"rendered":"https:\/\/www.dbi-services.com\/blog\/docker-efficiently-building-images-for-large-software\/"},"modified":"2018-03-31T16:30:56","modified_gmt":"2018-03-31T14:30:56","slug":"docker-efficiently-building-images-for-large-software","status":"publish","type":"post","link":"https:\/\/www.dbi-services.com\/blog\/docker-efficiently-building-images-for-large-software\/","title":{"rendered":"Docker: efficiently building images for large software"},"content":{"rendered":"<h2>By Franck Pachot<\/h2>\n<p>.<br \/>\nI see increasing demand to build a Docker image for the Oracle Database. But the installation process for Oracle does not really fit the Docker way to install by layers: you need to unzip the distribution, install from it to the Oracle Home, remove the things that are not needed, strop the binaries,&#8230; Before addressing those specific issues, here are the little tests I&#8217;ve done to show how the build layers increase the size of the image.<br \/>\n<!--more--><br \/>\nI&#8217;m starting with an empty docker repository on XFS filesystem:<\/p>\n<pre><code>\n[root@VM121 docker]# df -hT \/var\/lib\/docker\nFilesystem     Type  Size  Used Avail Use% Mounted on\n\/dev\/sdc       xfs    80G   33M   80G   1% \/var\/lib\/docker\n<\/code><\/pre>\n<h3>add, copy, rename and append<\/h3>\n<p>For the example, I create a 100MB file in the context:<\/p>\n<pre><code>\n[root@VM121 docker]# mkdir -p \/var\/tmp\/demo\n[root@VM121 docker]# dd if=\/dev\/urandom of=\/var\/tmp\/demo\/file0.100M count=100 bs=1M\n<\/code><\/pre>\n<p>Here his my docker file:<\/p>\n<pre><code>\n FROM alpine:latest as staging\n WORKDIR \/var\/tmp\n ADD  file0.100M .\n RUN  cp file0.100M file1.100M\n RUN  rm file0.100M\n RUN  mv file1.100M file2.100M\n RUN  dd if=\/dev\/urandom of=file2.100M seek=100 count=100 bs=1M\n<\/code><\/pre>\n<p>The 1st step starts with an alpine image<br \/>\nThe 2nd step sets the working directory<br \/>\nThe 3rd step adds a 100M file from the context<br \/>\nThe 4th step copies the file, so that we have 200M in two files<br \/>\nThe 5th step removes the previous file, so that we have 100M in one file<br \/>\nThe 6th step renames the file, staying with only one 100M file<br \/>\nThe 7th step appends 100M to the file, leaving 200M in one file<\/p>\n<p>Here is the build with default option:<\/p>\n<pre><code>\n[root@VM121 docker]# docker image build -t franck\/demo \/var\/tmp\/demo\n<\/code><\/pre>\n<p>The context, my 100M files is send first:<\/p>\n<pre><code>\nSending build context to Docker daemon  104.9MB\n<\/code><\/pre>\n<p>And here are my 7 steps:<\/p>\n<pre><code>\nStep 1\/7 : FROM alpine:latest as staging\nlatest: Pulling from library\/alpine\nff3a5c916c92: Pull complete\nDigest: sha256:7df6db5aa61ae9480f52f0b3a06a140ab98d427f86d8d5de0bedab9b8df6b1c0\nStatus: Downloaded newer image for alpine:latest\n ---&gt; 3fd9065eaf02\nStep 2\/7 : WORKDIR \/var\/tmp\nRemoving intermediate container 93d1b5f21bb9\n ---&gt; 131b3e6f34e7\nStep 3\/7 : ADD  file0.100M .\n ---&gt; 22ca0b2f6424\nStep 4\/7 : RUN  cp file0.100M file1.100M\n ---&gt; Running in b4b1b9c7e29b\nRemoving intermediate container b4b1b9c7e29b\n ---&gt; 8c7290a5c87e\nStep 5\/7 : RUN  rm file0.100M\n ---&gt; Running in 606e2c73d456\nRemoving intermediate container 606e2c73d456\n ---&gt; 5287e66b019c\nStep 6\/7 : RUN  mv file1.100M file2.100M\n ---&gt; Running in 10a9b379150e\nRemoving intermediate container 10a9b379150e\n ---&gt; f508f426f70e\nStep 7\/7 : RUN  dd if=\/dev\/urandom of=file2.100M seek=100 count=100 bs=1M\n ---&gt; Running in 9dcf6d80642c\n100+0 records in\n100+0 records out\nRemoving intermediate container 9dcf6d80642c\n ---&gt; f98304641c54\nSuccessfully built f98304641c54\nSuccessfully tagged franck\/demo:latest\n<\/code><\/pre>\n<p>So, what&#8217;s the size of my docker repository after my image with this 200M file?<\/p>\n<pre><code>\n[root@VM121 docker]# df -hT \/var\/lib\/docker\nFilesystem     Type  Size  Used Avail Use% Mounted on\n\/dev\/sdc       xfs    80G  538M   80G   1% \/var\/lib\/docker\n<\/code><\/pre>\n<p>I have more than 500MB here.<\/p>\n<p>Actually, besides the alpine image downloaded, which is only 4MB, the image I have build is 538MB:<\/p>\n<pre><code>\n[root@VM121 docker]# docker image ls\nREPOSITORY          TAG                 IMAGE ID            CREATED                  SIZE\nfranck\/demo         latest              f98304641c54        Less than a second ago   528MB\nalpine              latest              3fd9065eaf02        2 months ago             4.15MB\n<\/code><\/pre>\n<p>We can better understand this size by looking at intermediate images:<\/p>\n<pre><code>\n[root@VM121 docker]# docker image ls -a\nREPOSITORY          TAG                 IMAGE ID            CREATED             SIZE\nfranck\/demo         latest              f98304641c54        1 second ago        528MB\n&lt;none&gt;              &lt;none&gt;              f508f426f70e        27 seconds ago      319MB\n&lt;none&gt;              &lt;none&gt;              5287e66b019c        36 seconds ago      214MB\n&lt;none&gt;              &lt;none&gt;              8c7290a5c87e        37 seconds ago      214MB\n&lt;none&gt;              &lt;none&gt;              22ca0b2f6424        42 seconds ago      109MB\n&lt;none&gt;              &lt;none&gt;              131b3e6f34e7        47 seconds ago      4.15MB\nalpine              latest              3fd9065eaf02        2 months ago        4.15MB\n<\/code><\/pre>\n<p>The first one, &#8217;22ca0b2f6424&#8242; is from the step 3 which added the 100MB file<br \/>\nThe second one &#8216;8c7290a5c87e&#8217; is from the 4th step which copied the file, bringing the image to 200MB<br \/>\nThe third one &#8216;5287e66b019c&#8217; is from the 5th step which removed the file. I didn&#8217;t increase the size but didn&#8217;t remove anything either.<br \/>\nThe fourth one &#8216;f508f426f70e&#8217; is from the 6th step which renamed the file. But this, for docker, is like copying to a new layer and that adds 100MB<br \/>\nFinally, the 7th step appended only 100MB, but this finally resulted to copy the full 200MB file to the new layer<\/p>\n<p>We can see all those operations, and size added at each step, from the image history:<\/p>\n<pre><code>\n[root@VM121 docker]# docker image history franck\/demo\nIMAGE               CREATED             CREATED BY                                      SIZE                COMMENT\nf98304641c54        1 second ago        \/bin\/sh -c dd if=\/dev\/urandom of=file2.100M \u2026   210MB\nf508f426f70e        27 seconds ago      \/bin\/sh -c mv file1.100M file2.100M             105MB\n5287e66b019c        36 seconds ago      \/bin\/sh -c rm file0.100M                        0B\n8c7290a5c87e        37 seconds ago      \/bin\/sh -c cp file0.100M file1.100M             105MB\n22ca0b2f6424        42 seconds ago      \/bin\/sh -c #(nop) ADD file:339435a18aeeb1b69\u2026   105MB\n131b3e6f34e7        47 seconds ago      \/bin\/sh -c #(nop) WORKDIR \/var\/tmp              0B\n3fd9065eaf02        2 months ago        \/bin\/sh -c #(nop)  CMD [\"\/bin\/sh\"]              0B\n&lt;missing&gt;           2 months ago        \/bin\/sh -c #(nop) ADD file:093f0723fa46f6cdb\u2026   4.15MB\n<\/code><\/pre>\n<h3>All in one RUN<\/h3>\n<p>One workaround is to run everything in the same layer. Personally, I don&#8217;t like it because I don&#8217;t get the point of using a Dockerfile for just running one script.<br \/>\nSo, here is the Dockerfile with only one RUN command:<\/p>\n<pre><code>\n FROM alpine:latest as staging\n WORKDIR \/var\/tmp\n ADD  file0.100M .\n RUN  cp file0.100M file1.100M                                      \n  &amp;&amp;  rm file0.100M                                                 \n  &amp;&amp;  mv file1.100M file2.100M                                      \n  &amp;&amp;  dd if=\/dev\/urandom of=file2.100M seek=100 count=100 bs=1M\n<\/code><\/pre>\n<p>The build is similar except that there are fewer steps:<\/p>\n<pre><code>\n[root@VM121 docker]# docker image build -t franck\/demo \/var\/tmp\/demo\nSending build context to Docker daemon  104.9MB\nStep 1\/4 : FROM alpine:latest as staging\nlatest: Pulling from library\/alpine\nff3a5c916c92: Pull complete\nDigest: sha256:7df6db5aa61ae9480f52f0b3a06a140ab98d427f86d8d5de0bedab9b8df6b1c0\nStatus: Downloaded newer image for alpine:latest\n ---&gt; 3fd9065eaf02\nStep 2\/4 : WORKDIR \/var\/tmp\nRemoving intermediate container 707644c15547\n ---&gt; d4528b28c85e\nStep 3\/4 : ADD  file0.100M .\n ---&gt; e26215766e75\nStep 4\/4 : RUN  cp file0.100M file1.100M                                        &amp;&amp;  rm file0.100M                                                     &amp;&amp;  mv file1.100M file2.100M                                        &amp;&amp;  dd if=\/dev\/urandom of=file2.100M seek=100 count=100 bs=1M\n ---&gt; Running in 49c2774851f4\n100+0 records in\n100+0 records out\nRemoving intermediate container 49c2774851f4\n ---&gt; df614ac1b6b3\nSuccessfully built df614ac1b6b3\nSuccessfully tagged franck\/demo:latest\n<\/code><\/pre>\n<p>This leaves us with a smaller space usage::<\/p>\n<pre><code>\n[root@VM121 docker]# df -hT \/var\/lib\/docker\nFilesystem     Type  Size  Used Avail Use% Mounted on\n\/dev\/sdc       xfs    80G  340M   80G   1% \/var\/lib\/docker\n<\/code><\/pre>\n<p>The image is smaller, but still larger than the final state (a 300MB image for only one 200MB file):<\/p>\n<pre><code>\n[root@VM121 docker]# docker image ls\nREPOSITORY          TAG                 IMAGE ID            CREATED                  SIZE\nfranck\/demo         latest              df614ac1b6b3        Less than a second ago   319MB\nalpine              latest              3fd9065eaf02        2 months ago             4.15MB\n<\/code><\/pre>\n<p>This is because we have grouped the RUN steps, but the ADD has its own layer, adding a file that is removed later:<\/p>\n<pre><code>\n[root@VM121 docker]# docker image ls -a\nREPOSITORY          TAG                 IMAGE ID            CREATED                  SIZE\nfranck\/demo         latest              df614ac1b6b3        Less than a second ago   319MB\n&lt;none&gt;              &lt;none&gt;              e26215766e75        20 seconds ago           109MB\n&lt;none&gt;              &lt;none&gt;              d4528b28c85e        22 seconds ago           4.15MB\nalpine              latest              3fd9065eaf02        2 months ago             4.15MB\n&nbsp;\n[root@VM121 docker]# docker image history franck\/demo\nIMAGE               CREATED                  CREATED BY                                      SIZE                COMMENT\ndf614ac1b6b3        Less than a second ago   \/bin\/sh -c cp file0.100M file1.100M         \u2026   210MB\ne26215766e75        20 seconds ago           \/bin\/sh -c #(nop) ADD file:fe0262a4b800bf66d\u2026   105MB\nd4528b28c85e        22 seconds ago           \/bin\/sh -c #(nop) WORKDIR \/var\/tmp              0B\n3fd9065eaf02        2 months ago             \/bin\/sh -c #(nop)  CMD [\"\/bin\/sh\"]              0B\n&lt;missing&gt;             2 months ago             \/bin\/sh -c #(nop) ADD file:093f0723fa46f6cdb\u2026   4.15MB\n<\/code><\/pre>\n<p>This is the kind of issue we have when building an Oracle Database image. We need to ADD the zip file for the database distribution, and the latest bundle patch. It is removed later but still takes space on the image. Note that one workaround to avoid the ADD layer can be to get the files from an NFS or HTTP server with wget or curl in a RUN layer rather than an ADD one. There&#8217;s an example on <a href=\"http:\/\/www.oradba.ch\/2018\/03\/smaller-oracle-docker-images\/\" target=\"_blank\" rel=\"noopener noreferrer\">Stefan Oehrli blog post<\/a>.<\/p>\n<h3>&#8211;squash<\/h3>\n<p>With the latest versions of docker, there&#8217;s an easy way to flatten all those intermediary images at the end.<br \/>\nHere I&#8217;ve 18.03 and enabled experimental features:<\/p>\n<pre><code>\n[root@VM121 docker]# docker info\nContainers: 0\n Running: 0\n Paused: 0\n Stopped: 0\nImages: 8\nServer Version: 18.03.0-ce\nStorage Driver: overlay2\n Backing Filesystem: xfs\n...\n&nbsp;\n[root@VM121 docker]# cat \/etc\/docker\/daemon.json\n{\n  \"experimental\": true\n}\n<\/code><\/pre>\n<p>I start with the same as before but just add &#8211;squash to the build command<\/p>\n<pre><code>\n[root@VM121 docker]# docker image build --squash -t franck\/demo \/var\/tmp\/demo\n<\/code><\/pre>\n<p>The output is similar but the image is an additional one, reduced down to the size of my final state (with one 200MB file):<\/p>\n<pre><code>\n[root@VM121 docker]# docker image ls\nREPOSITORY          TAG                 IMAGE ID            CREATED                  SIZE\nfranck\/demo         latest              2ab439a723c4        Less than a second ago   214MB\n&lt;none&gt;              &lt;none&gt;              c3058e598b0a        3 seconds ago            528MB\nalpine              latest              3fd9065eaf02        2 months ago             4.15MB\n<\/code><\/pre>\n<p>The intermediate image list shows that all was done as without &#8216;&#8211;squash&#8217; but with an additional set which reduced the size:<\/p>\n<pre><code>\n[root@VM121 docker]# docker image ls -a\nREPOSITORY          TAG                 IMAGE ID            CREATED                  SIZE\nfranck\/demo         latest              2ab439a723c4        Less than a second ago   214MB\n&lt;none&gt;              &lt;none&gt;              c3058e598b0a        3 seconds ago            528MB\n&lt;none&gt;              &lt;none&gt;              1f14d93a592e        23 seconds ago           319MB\n&lt;none&gt;              &lt;none&gt;              7563d40b650b        27 seconds ago           214MB\n&lt;none&gt;              &lt;none&gt;              8ed15a5059bd        28 seconds ago           214MB\n&lt;none&gt;              &lt;none&gt;              24b11b9026ce        31 seconds ago           109MB\n&lt;none&gt;              &lt;none&gt;              382bb71a6a4a        33 seconds ago           4.15MB\nalpine              latest              3fd9065eaf02        2 months ago             4.15MB\n<\/code><\/pre>\n<p>This step is visible in the image history as a &#8216;merge&#8217; step:<\/p>\n<pre><code>\n[root@VM121 docker]#  docker image history franck\/demo\nIMAGE               CREATED                  CREATED BY                                      SIZE                COMMENT\n2ab439a723c4        Less than a second ago                                                   210MB               merge sha256:c3058e598b0a30c606c1bfae7114957bbc62fca85d6a70c2aff4473726431394 to sha256:3fd9065eaf02feaf94d68376da52541925650b81698c53c6824d92ff63f98353\n&lt;missing&gt;             3 seconds ago            \/bin\/sh -c dd if=\/dev\/urandom of=file2.100M \u2026   0B\n&lt;missing&gt;             23 seconds ago           \/bin\/sh -c mv file1.100M file2.100M             0B\n&lt;missing&gt;             27 seconds ago           \/bin\/sh -c rm file0.100M                        0B\n&lt;missing&gt;             28 seconds ago           \/bin\/sh -c cp file0.100M file1.100M             0B\n&lt;missing&gt;             31 seconds ago           \/bin\/sh -c #(nop) ADD file:14cef588b48ffbbf1\u2026   0B\n&lt;missing&gt;             33 seconds ago           \/bin\/sh -c #(nop) WORKDIR \/var\/tmp              0B\n&lt;missing&gt;             2 months ago             \/bin\/sh -c #(nop)  CMD [\"\/bin\/sh\"]              0B\n&lt;missing&gt;             2 months ago             \/bin\/sh -c #(nop) ADD file:093f0723fa46f6cdb\u2026   4.15MB\n<\/code><\/pre>\n<p>However, even if I have a smaller final image, my filesystem usage is even larger with this additional 210MB:<\/p>\n<pre><code>\n[root@VM121 docker]# df -hT \/var\/lib\/docker\nFilesystem     Type  Size  Used Avail Use% Mounted on\n\/dev\/sdc       xfs    80G  739M   80G   1% \/var\/lib\/docker\n<\/code><\/pre>\n<p>Let&#8217;s prune it to get rid of those intermediate images:<\/p>\n<pre><code>\n[root@VM121 docker]# docker image prune -f\nDeleted Images:\ndeleted: sha256:c3058e598b0a30c606c1bfae7114957bbc62fca85d6a70c2aff4473726431394\ndeleted: sha256:37ed4826d70def1978f9dc0ddf42618d951f65a79ce30767ac3a5037d514f8af\ndeleted: sha256:1f14d93a592eb49a210ed73bf65e6886fcec332786d54b55d6b0e16fb8a8beda\ndeleted: sha256:c65cf4c70aed04e9b57e7a2a4fa454d3c63f43c32af251d8c86f6f85f44b1757\ndeleted: sha256:7563d40b650b2126866e8072b8df92d5d7516d86b25a2f6f99aa101bb47835ba\ndeleted: sha256:31ee5456431e903cfd384b1cd7ccb7918d203dc73a131d4ff0b9e6517f0d51cd\ndeleted: sha256:8ed15a5059bd4c0c4ecb78ad77ed75da143b06923d8a9a9a67268c62257b6534\ndeleted: sha256:6be91d85dec6e1bda6f1c0d565e98dbf928b4ea139bf9cb666455e77a2d8f0d9\ndeleted: sha256:24b11b9026ce738a78ce3f7b8b5d86ba3fdeb15523a30a7c22fa1e3712ae679a\ndeleted: sha256:c0984945970276621780a7888adfde9c6e6ca475c42af6b7c54f664ad86f9c9f\ndeleted: sha256:382bb71a6a4a7ddec86faa76bb86ea0c1a764e5326ad5ef68ce1a6110ae45754\n&nbsp;\nTotal reclaimed space: 524.3MB\n<\/code><\/pre>\n<p>Now having only the squashed image:<\/p>\n<pre><code>\n[root@VM121 docker]# docker image ls -a\nREPOSITORY          TAG                 IMAGE ID            CREATED             SIZE\nfranck\/demo         latest              2ab439a723c4        32 minutes ago      214MB\nalpine              latest              3fd9065eaf02        2 months ago        4.15MB\n&nbsp;\n[root@VM121 docker]# df -hT \/var\/lib\/docker\nFilesystem     Type  Size  Used Avail Use% Mounted on\n\/dev\/sdc       xfs    80G  237M   80G   1% \/var\/lib\/docker\n<\/code><\/pre>\n<h3>multi-stage build<\/h3>\n<p>Finally, you can do something similar to an intermediate squash using multi-stage build.<\/p>\n<p>Here is my Dockerfile:<\/p>\n<pre><code>\n FROM alpine:latest as staging\n WORKDIR \/var\/tmp\n ADD  file0.100M .\n RUN  cp file0.100M file1.100M\n RUN  rm file0.100M\n RUN  mv file1.100M file2.100M\n RUN  dd if=\/dev\/urandom of=file2.100M seek=100 count=100 bs=1M\n&nbsp;\n FROM alpine:latest\n WORKDIR \/var\/tmp\n COPY --from=staging \/var\/tmp .\n<\/code><\/pre>\n<p>With multi-stage build, we can start the second stage from a different image, and add more steps, but here I just start with the same alpine image and copy the final layer of the previous build.<\/p>\n<p>We see something very similar to the &#8211;squash one:<\/p>\n<pre><code>\n[root@VM121 docker]# docker image ls\nREPOSITORY          TAG                 IMAGE ID            CREATED                  SIZE\nfranck\/demo         latest              55f329385f8c        Less than a second ago   214MB\n&lt;none&gt;              &lt;none&gt;              fd26a00db784        8 seconds ago            528MB\nalpine              latest              3fd9065eaf02        2 months ago             4.15MB\n&nbsp;\n[root@VM121 docker]# docker image ls -a\nREPOSITORY          TAG                 IMAGE ID            CREATED             SIZE\nfranck\/demo         latest              55f329385f8c        1 second ago        214MB\n&lt;none&gt;              &lt;none&gt;              fd26a00db784        9 seconds ago       528MB\n&lt;none&gt;              &lt;none&gt;              9bf5be367b63        32 seconds ago      319MB\n&lt;none&gt;              &lt;none&gt;              531d78833ba8        35 seconds ago      214MB\n&lt;none&gt;              &lt;none&gt;              05dd68114743        36 seconds ago      214MB\n&lt;none&gt;              &lt;none&gt;              b9e5215a9fc8        39 seconds ago      109MB\n&lt;none&gt;              &lt;none&gt;              ab332f486793        41 seconds ago      4.15MB\nalpine              latest              3fd9065eaf02        2 months ago        4.15MB\n<\/code><\/pre>\n<p>The history of the last stage shows the copy of 210MB from the previous one:<\/p>\n<pre><code>\n[root@VM121 docker]# docker image history franck\/demo\nIMAGE               CREATED             CREATED BY                                      SIZE                COMMENT\n55f329385f8c        1 second ago        \/bin\/sh -c #(nop) COPY dir:2b66b5c36eff5b51f\u2026   210MB\nab332f486793        41 seconds ago      \/bin\/sh -c #(nop) WORKDIR \/var\/tmp              0B\n3fd9065eaf02        2 months ago        \/bin\/sh -c #(nop)  CMD [\"\/bin\/sh\"]              0B\n&lt;missing&gt;           2 months ago        \/bin\/sh -c #(nop) ADD file:093f0723fa46f6cdb\u2026   4.15MB\n<\/code><\/pre>\n<p>The usage of filesystem is similar to the &#8211;squash one. Even if we reduced the final image, all the intermediate states had to be stored:<\/p>\n<pre><code>\n[root@VM121 docker]# df -hT \/var\/lib\/docker\nFilesystem     Type  Size  Used Avail Use% Mounted on\n\/dev\/sdc       xfs    80G  737M   80G   1% \/var\/lib\/docker\n<\/code><\/pre>\n<p>That looks good, if you accept to use a large intermediate space while building the image, which gives you the possibility to debug without re-running from the beginning, thanks to the layers in cache. However, you have still the inefficiency that each time you try the build, the context will be sent again even when not needed. And that is long with a 3GB .zip in the case of Oracle Database installation. Unfortunately, if you add the file to the .dockerignore once you know you have the ADD steps in cache, the next build attempt will not use the caches anymore. I would love to see a per-stage .dockerignore file for multi-stage builds. Or simply have docker realize that some files in the context will not be needed by the COPY or ADD that are not in cache yet.<\/p>\n<p>Sending the whole context at each build attempt, when debugging your Dockerfile, is not efficient at all and looks like punch-card time compilation where people sent the cards to be compiled during the night. One syntax error on the first line and you go for another day.<\/p>\n<p>One solution is to have all the required files in an NFS or HTTPd server and get them with ADD from the URL as mentioned earlier. <\/p>\n<h3>Multi-stage with multi-contexts<\/h3>\n<p>Another solution is to put all COPY or ADD from context in one Dockerfile to build the image containing all required files, and then build your image from it (and squash it at the end).<\/p>\n<p>Here is my first Dockerfile, just adding the files from the context:<\/p>\n<pre><code>\n[root@VM121 docker]# ls \/var\/tmp\/demo\nDockerfile  file0.100M  nocontext\n[root@VM121 docker]# cat \/var\/tmp\/demo\/Dockerfile\n FROM alpine:latest as staging\n WORKDIR \/var\/tmp\n ADD  file0.100M .\n<\/code><\/pre>\n<p>I build this &#8216;staging&#8217; image:<\/p>\n<pre><code>\n[root@VM121 docker]# docker image build -t franck\/stage0 \/var\/tmp\/demo\nSending build context to Docker daemon  104.9MB\nStep 1\/3 : FROM alpine:latest as staging\nlatest: Pulling from library\/alpine\nff3a5c916c92: Pull complete\nDigest: sha256:7df6db5aa61ae9480f52f0b3a06a140ab98d427f86d8d5de0bedab9b8df6b1c0\nStatus: Downloaded newer image for alpine:latest\n ---&gt; 3fd9065eaf02\nStep 2\/3 : WORKDIR \/var\/tmp\nRemoving intermediate container 0eeed8e0cfd2\n ---&gt; a5db3b29c8e1\nStep 3\/3 : ADD  file0.100M .\n ---&gt; 2a34e1e981be\nSuccessfully built 2a34e1e981be\nSuccessfully tagged franck\/stage0:latest\n<\/code><\/pre>\n<p>This one is the minimal one:<\/p>\n<pre><code>\n[root@VM121 docker]# docker image ls\n+ docker image ls\nREPOSITORY          TAG                 IMAGE ID            CREATED                  SIZE\nfranck\/stage0       latest              2a34e1e981be        Less than a second ago   109MB\nalpine              latest              3fd9065eaf02        2 months ago             4.15MB\n&nbsp;\n[root@VM121 docker]# df -hT \/var\/lib\/docker\nFilesystem     Type  Size  Used Avail Use% Mounted on\n\/dev\/sdc       xfs    80G  139M   80G   1% \/var\/lib\/docker\n<\/code><\/pre>\n<p>Now, I don&#8217;t need to send this context anymore during further development of my Dockerfile.<br \/>\nI&#8217;ve added the following steps to a Dockerfile in another directory:<\/p>\n<pre><code>\n[root@VM121 docker]# ls \/var\/tmp\/demo\/nocontext\/\nDockerfile\n[root@VM121 docker]# cat \/var\/tmp\/demo\/nocontext\/Dockerfile\n FROM franck\/stage0 as stage1\n WORKDIR \/var\/tmp\n RUN  cp file0.100M file1.100M\n RUN  rm file0.100M\n RUN  mv file1.100M file2.100M\n RUN  dd if=\/dev\/urandom of=file2.100M seek=100 count=100 bs=1M\n FROM alpine:latest\n WORKDIR \/var\/tmp\n<\/code><\/pre>\n<p>Here is the build, using multi-stage to get a squashed final image (you can also use &#8211;squash)<\/p>\n<pre><code>\n[root@VM121 docker]# docker image build -t franck\/demo \/var\/tmp\/demo\/nocontext\n&nbsp;\nSending build context to Docker daemon  2.048kB\nStep 1\/9 : FROM franck\/stage0 as stage1\n ---&gt; 2a34e1e981be\nStep 2\/9 : WORKDIR \/var\/tmp\nRemoving intermediate container eabf57a8de05\n...\nSuccessfully built 82478bfa260d\nSuccessfully tagged franck\/demo:latest\n<\/code><\/pre>\n<p>At that point, there&#8217;s no advantage on space used as I keep all layers for easy Dockerfile development:<\/p>\n<pre><code>\n[root@VM121 docker]# df -hT \/var\/lib\/docker\nFilesystem     Type  Size  Used Avail Use% Mounted on\n\/dev\/sdc       xfs    80G  738M   80G   1% \/var\/lib\/docker\n&nbsp;\n[root@VM121 docker]# docker image ls\nREPOSITORY          TAG                 IMAGE ID            CREATED              SIZE\nfranck\/demo         latest              82478bfa260d        About a minute ago   214MB\n&lt;none&gt;              &lt;none&gt;              5772ad68d208        About a minute ago   528MB\nfranck\/stage0       latest              2a34e1e981be        About a minute ago   109MB\nalpine              latest              3fd9065eaf02        2 months ago         4.15MB\n<\/code><\/pre>\n<p>But now, if I want to add an additional step:<\/p>\n<pre><code>\n[root@VM121 docker]# cat &gt;&gt; \/var\/tmp\/demo\/nocontext\/Dockerfile &lt;&lt;&lt; 'RUN chmod a+x \/var\/tmp'\n<\/code><\/pre>\n<p>I can re-build quickly, using cached layers, and without the need to send the context again:<\/p>\n<pre><code>\n[root@VM121 docker]# docker image build -t franck\/demo \/var\/tmp\/demo\/nocontext\nSending build context to Docker daemon  2.048kB\nStep 1\/10 : FROM franck\/stage0 as stage1\n ---&gt; 2a34e1e981be\nStep 2\/10 : WORKDIR \/var\/tmp\n ---&gt; Using cache\n ---&gt; fa562926cc2b\nStep 3\/10 : RUN  cp file0.100M file1.100M\n ---&gt; Using cache\n ---&gt; 31ac716f4d61\nStep 4\/10 : RUN  rm file0.100M\n ---&gt; Using cache\n ---&gt; d7392cf51ad9\nStep 5\/10 : RUN  mv file1.100M file2.100M\n ---&gt; Using cache\n ---&gt; 4854e503885b\nStep 6\/10 : RUN  dd if=\/dev\/urandom of=file2.100M seek=100 count=100 bs=1M\n ---&gt; Using cache\n ---&gt; 5772ad68d208\nStep 7\/10 : FROM alpine:latest\n ---&gt; 3fd9065eaf02\nStep 8\/10 : WORKDIR \/var\/tmp\n ---&gt; Using cache\n ---&gt; a5db3b29c8e1\nStep 9\/10 : COPY --from=stage1 \/var\/tmp .\n ---&gt; Using cache\n ---&gt; 82478bfa260d\nStep 10\/10 : RUN chmod a+x \/var\/tmp\n ---&gt; 4a69ee40a938\nSuccessfully built 4a69ee40a938\nSuccessfully tagged franck\/demo:latest\n<\/code><\/pre>\n<p>Once I&#8217;m ok with my final image, I can remove the intermediate ones:<\/p>\n<pre><code>\n[root@VM121 docker]# docker image prune -f\nDeleted Images:\ndeleted: sha256:5772ad68d20841197d1424f7c64edd21704e4c7b470acb2193de51ae8741385d\ndeleted: sha256:bab572d749684d126625a74be4f01cc738742f9c112a940391e3533e61dd55b9\ndeleted: sha256:4854e503885b4057809fe2867a743ae7898e3e06b329229519fdb5c9d8b10ac1\ndeleted: sha256:de4acb90433c30d6a21cc3b4483adbd403d8051f3c7c31e6bc095a304606355a\ndeleted: sha256:d7392cf51ad99d5d0b7a1a18d8136905c87bc738a5bc94dec03e92f5385bf9c8\ndeleted: sha256:f037e7f973f4265099402534cd7ba409f35272701166d59a1be8e5e39508b07c\ndeleted: sha256:31ac716f4d61f0048a75b8de6f18757970cf0670a0a3d711e4386bf098b32041\ndeleted: sha256:2dccb363c5beb4daf45586383df6454b198f824d52676f70318444c346c0fe9a\ndeleted: sha256:fa562926cc2b3cb56400e1068984bb4048f56713a3cf6dcfa3cf6d945023ebc4\n&nbsp;\nTotal reclaimed space: 419.4MB\n<\/code><\/pre>\n<p>And the staging one:<\/p>\n<pre><code>\n[root@VM121 docker]# docker image rm franck\/stage0\nUntagged: franck\/stage0:latest\nDeleted: sha256:2a34e1e981be9154c31c5ee7eb942cc121267f4416b6fe502ab93f2dceafd98c\nDeleted: sha256:b996a1bdc829167f16dcbe58b717284764470661c3116a6352f15012e1dff07c\n<\/code><\/pre>\n<p>Finally, I optimized the developement of the Dockerfile and finished with the minimal size.<\/p>\n<pre><code>\n[root@VM121 docker]# df -hT \/var\/lib\/docker\nFilesystem     Type  Size  Used Avail Use% Mounted on\n\/dev\/sdc       xfs    80G  237M   80G   1% \/var\/lib\/docker\n<\/code><\/pre>\n<h3>So what?<\/h3>\n<p>I&#8217;m always surprised by the lack of efficiency when building an image with a Dockerfile. Any serious application deployment involves several intermediate files and the way docker build is layered inflates the size and the time required. Efficient layering and snapshotting work at block level. Here, at file level, any byte of data modified in a file, even metadata such as the file name, is a whole file copy. But for common applications, the installation steps are not as simple adding new files. You may have files appended, object files added to libraries, then compiled, the stripped&#8230;<\/p>\n<p>In this post, I tested some recent features, such as multi-stage build and the experimental &#8211;squash, as well as a simple manual multi-stage build. Of course, you can do everything in the same layers, and even not use Dockerfiles at all, but then why using Docker? There&#8217;s also the <a href=\"https:\/\/www.packer.io\/docs\/builders\/docker.html\" target=\"_blank\" rel=\"noopener noreferrer\">Packer<\/a> approach that I&#8217;ve not tested yet. However, I like the Docker approach, but only when used correctly. Deploying an application, like Oracle Database, should use the layered build in the following way: additional steps for new options or new updates. This means that the files must be built elsewhere, in a staging container, and added in one step. And to be efficient, the context should be sent only when needed: when a non-cached ADD or COPY requires it.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>By Franck Pachot . I see increasing demand to build a Docker image for the Oracle Database. But the installation process for Oracle does not really fit the Docker way to install by layers: you need to unzip the distribution, install from it to the Oracle Home, remove the things that are not needed, strop [&hellip;]<\/p>\n","protected":false},"author":27,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[59],"tags":[601,96,1171],"type_dbi":[],"class_list":["post-11055","post","type-post","status-publish","format-standard","hentry","category-oracle","tag-docker","tag-oracle","tag-oracle-19c"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v27.2 (Yoast SEO v27.2) - https:\/\/yoast.com\/product\/yoast-seo-premium-wordpress\/ -->\n<title>Docker: efficiently building images for large software - dbi Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.dbi-services.com\/blog\/docker-efficiently-building-images-for-large-software\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Docker: efficiently building images for large software\" \/>\n<meta property=\"og:description\" content=\"By Franck Pachot . I see increasing demand to build a Docker image for the Oracle Database. But the installation process for Oracle does not really fit the Docker way to install by layers: you need to unzip the distribution, install from it to the Oracle Home, remove the things that are not needed, strop [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.dbi-services.com\/blog\/docker-efficiently-building-images-for-large-software\/\" \/>\n<meta property=\"og:site_name\" content=\"dbi Blog\" \/>\n<meta property=\"article:published_time\" content=\"2018-03-31T14:30:56+00:00\" \/>\n<meta name=\"author\" content=\"Oracle Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Oracle Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"21 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/docker-efficiently-building-images-for-large-software\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/docker-efficiently-building-images-for-large-software\/\"},\"author\":{\"name\":\"Oracle Team\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/66ab87129f2d357f09971bc7936a77ee\"},\"headline\":\"Docker: efficiently building images for large software\",\"datePublished\":\"2018-03-31T14:30:56+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/docker-efficiently-building-images-for-large-software\/\"},\"wordCount\":1455,\"commentCount\":0,\"keywords\":[\"Docker\",\"Oracle\",\"Oracle 19c\"],\"articleSection\":[\"Oracle\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.dbi-services.com\/blog\/docker-efficiently-building-images-for-large-software\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/docker-efficiently-building-images-for-large-software\/\",\"url\":\"https:\/\/www.dbi-services.com\/blog\/docker-efficiently-building-images-for-large-software\/\",\"name\":\"Docker: efficiently building images for large software - dbi Blog\",\"isPartOf\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#website\"},\"datePublished\":\"2018-03-31T14:30:56+00:00\",\"author\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/66ab87129f2d357f09971bc7936a77ee\"},\"breadcrumb\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/docker-efficiently-building-images-for-large-software\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.dbi-services.com\/blog\/docker-efficiently-building-images-for-large-software\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/docker-efficiently-building-images-for-large-software\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Accueil\",\"item\":\"https:\/\/www.dbi-services.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Docker: efficiently building images for large software\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#website\",\"url\":\"https:\/\/www.dbi-services.com\/blog\/\",\"name\":\"dbi Blog\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.dbi-services.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/66ab87129f2d357f09971bc7936a77ee\",\"name\":\"Oracle Team\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/secure.gravatar.com\/avatar\/f711f7cd2c9b09bf2627133755b569fb5be0694810cfd33033bdd095fedba86d?s=96&d=mm&r=g\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/f711f7cd2c9b09bf2627133755b569fb5be0694810cfd33033bdd095fedba86d?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/f711f7cd2c9b09bf2627133755b569fb5be0694810cfd33033bdd095fedba86d?s=96&d=mm&r=g\",\"caption\":\"Oracle Team\"},\"url\":\"https:\/\/www.dbi-services.com\/blog\/author\/oracle-team\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Docker: efficiently building images for large software - dbi Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.dbi-services.com\/blog\/docker-efficiently-building-images-for-large-software\/","og_locale":"en_US","og_type":"article","og_title":"Docker: efficiently building images for large software","og_description":"By Franck Pachot . I see increasing demand to build a Docker image for the Oracle Database. But the installation process for Oracle does not really fit the Docker way to install by layers: you need to unzip the distribution, install from it to the Oracle Home, remove the things that are not needed, strop [&hellip;]","og_url":"https:\/\/www.dbi-services.com\/blog\/docker-efficiently-building-images-for-large-software\/","og_site_name":"dbi Blog","article_published_time":"2018-03-31T14:30:56+00:00","author":"Oracle Team","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Oracle Team","Est. reading time":"21 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.dbi-services.com\/blog\/docker-efficiently-building-images-for-large-software\/#article","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/docker-efficiently-building-images-for-large-software\/"},"author":{"name":"Oracle Team","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/66ab87129f2d357f09971bc7936a77ee"},"headline":"Docker: efficiently building images for large software","datePublished":"2018-03-31T14:30:56+00:00","mainEntityOfPage":{"@id":"https:\/\/www.dbi-services.com\/blog\/docker-efficiently-building-images-for-large-software\/"},"wordCount":1455,"commentCount":0,"keywords":["Docker","Oracle","Oracle 19c"],"articleSection":["Oracle"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.dbi-services.com\/blog\/docker-efficiently-building-images-for-large-software\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.dbi-services.com\/blog\/docker-efficiently-building-images-for-large-software\/","url":"https:\/\/www.dbi-services.com\/blog\/docker-efficiently-building-images-for-large-software\/","name":"Docker: efficiently building images for large software - dbi Blog","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/#website"},"datePublished":"2018-03-31T14:30:56+00:00","author":{"@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/66ab87129f2d357f09971bc7936a77ee"},"breadcrumb":{"@id":"https:\/\/www.dbi-services.com\/blog\/docker-efficiently-building-images-for-large-software\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.dbi-services.com\/blog\/docker-efficiently-building-images-for-large-software\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.dbi-services.com\/blog\/docker-efficiently-building-images-for-large-software\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Accueil","item":"https:\/\/www.dbi-services.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Docker: efficiently building images for large software"}]},{"@type":"WebSite","@id":"https:\/\/www.dbi-services.com\/blog\/#website","url":"https:\/\/www.dbi-services.com\/blog\/","name":"dbi Blog","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.dbi-services.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/66ab87129f2d357f09971bc7936a77ee","name":"Oracle Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/f711f7cd2c9b09bf2627133755b569fb5be0694810cfd33033bdd095fedba86d?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/f711f7cd2c9b09bf2627133755b569fb5be0694810cfd33033bdd095fedba86d?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/f711f7cd2c9b09bf2627133755b569fb5be0694810cfd33033bdd095fedba86d?s=96&d=mm&r=g","caption":"Oracle Team"},"url":"https:\/\/www.dbi-services.com\/blog\/author\/oracle-team\/"}]}},"_links":{"self":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/11055","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/users\/27"}],"replies":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/comments?post=11055"}],"version-history":[{"count":0,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/11055\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/media?parent=11055"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/categories?post=11055"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/tags?post=11055"},{"taxonomy":"type","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/type_dbi?post=11055"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}