Docker slowness on jessie & trusty

[dachary]

Bonjour,

In the past few months I’ve observed that jessie & trusty are slow in Docker although stretch & xenial are fast. I’m running stretch on a 4.9.0-6-amd64 #1 SMP Debian 4.9.82-1+deb9u3 (2018-03-02) x86_64 GNU/Linux and Docker version 17.05.0-ce, build 89658be

• time docker run ubuntu:xenial apt-get update => real 0m3.935s
• time docker run ubuntu:trusty apt-get update => real 0m36.203s
• time docker run debian:jessie apt-get update => real 0m33.171s
• time docker run debian:stretch apt-get update => real 0m2.774s

Is it just me?

[conorsch]

Yes, I’m seeing similar results:

• time docker run ubuntu:xenial apt-get update => real 0m19.181s
• time docker run ubuntu:trusty apt-get update => real 1m33.617s
• time docker run debian:jessie apt-get update => real 1m25.184s
• time docker run debian:stretch apt-get update => real 0m8.093s

Although each command takes significantly longer on my host than yours. Still, the gap between older and newer distros is comparable. Haven’t dug to investigate.

[dachary]

@dachary

ocker run ubuntu:trusty apt-get update

apt-get updates appear to be running slower now

I was able to confirm what @gavanderhoorn mentioned about the spectre and meltdown mitigations. Disabling them does resolve the apt-get performance issues in Trusty containers running on Xenial hosts with the linux-aws kernel.

[dachary]

loic@raise:~$ time sudo docker run ubuntu:xenial apt-get -qq update

real	0m4.470s
user	0m0.082s
sys	0m0.014s
loic@raise:~$ time sudo docker run ubuntu:trusty apt-get -qq update

real	0m47.783s
user	0m0.082s
sys	0m0.025s
loic@raise:~$ docker --version
Docker version 18.03.1-ce, build 9ee9f40
loic@raise:~$ uname -a
Linux raise 4.16.0-0.bpo.1-amd64 #1 SMP Debian 4.16.5-1~bpo9+1 (2018-05-06) x86_64 GNU/Linux

[conorsch]

For those playing along at home, make sure you see the excellent debugging posts linked to the GH thread above:

• https://hackernoon.com/another-reason-why-your-docker-containers-may-be-slow-d37207dec27f
• https://sysdig.com/blog/container-isolation-gone-wrong/

Great reading. Thanks for sharing, @dachary!

[dachary]

I am no longer able to reproduce the above results for trusty. But they stay the same for jessie. The kernel & docker versions are the same but the Docker images are not.

loic@raise:~$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
ubuntu              xenial              5e8b97a2a082        6 days ago          114MB
ubuntu              trusty              578c3e61a98c        6 days ago          223MB
loic@raise:~$ time sudo docker run ubuntu:xenial apt-get -qq update

real	0m4.600s
user	0m0.075s
sys	0m0.039s
loic@raise:~$ time sudo docker run ubuntu:trusty apt-get -qq update

real	0m4.529s
user	0m0.088s
sys	0m0.020s

loic@raise:~$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
debian              stretch             8626492fecd3        6 weeks ago         101MB
debian              jessie              4eb8376dc2a3        6 weeks ago         127MB
loic@raise:~$ time sudo docker run debian:jessie apt-get -qq update

real	0m47.211s
user	0m0.077s
sys	0m0.027s
loic@raise:~$ time sudo docker run debian:stretch apt-get -qq update

real	0m3.366s
user	0m0.086s
sys	0m0.024s

[dachary]

$ time docker run --rm ubuntu@sha256:885bb6705b01d99544ddb98cbe4e4555d1efe1d052cef90832e72a0688ac6b37 apt-get -qq update

real	0m4.138s
$ time docker run --rm ubuntu@sha256:084989eb923bd86dbf7e706d464cf3587274a826b484f75b69468c19f8ae354c apt-get -qq update

real	0m37.338s