In my post Forensicating Docker, Part 1 back in March (yes, I promise a Part 2 in the next couple of months, the $dayjob has slowed that down a bit), I talked a little about the AUFS layered filesystem that was used by the docker install on the system I was investigating. While I was forensicating the case I talked about in that diary, I wanted to see what the container filesystem looked like from my SIFT VM so I wrote a script to do the mounting the same way docker does (except for forensic purposes the mount is read-only). The script can be found here. Unfortunately, docker can use multiple storage drivers.  So far, I've adapted the script to handle two/three of them, AUFS and Overlay/Overlay2. AUFS is the default on (older?) versions of Ubuntu, but AUFS isn't included by default in RedHat (or derivates), you would have to compile your own kernel. Overlay2 is included in newer kernels (pretty much anything after 3.18), so I suspect it may become the default at some point in the future. These are the storage drivers that handle so called Union filesystems to handle the layering. The btrfs, zfs, and devicemapper storage drivers are all block-level rather than file-level storage drivers. In effect, they require separate devices/partitions/loop-mounted files taking advantage of filesystem features such as snapshots in the underlying filesystem drivers to handle the layering.  While I think I can get btrfs into the script, I haven't looked at zfs and I've had difficutly with devicemapper, so I may not be able to get all of these. See [3] and [4] below for more background on docker storage drivers.

Having gone through all of that, for the purpose of forensication, it is important to remember that changes made within a container will all be captured in the "top" layer of these layered or union filesystems. To find that top layer (for docker >= 1.9), you need to look at /var/lib/docker/image/storagedriver/layerdb/mounts/containerid/mount-id to find the id for the container storage. Then, the changes will be taking place in /var/lib/docker/storagedriver/containermedia-init/diff, so that is where you should expect to find any indications of malicious activity within the container.

References:
[1] https://isc.sans.edu/forums/diary/Forensicating+Docker+Part+1/20835/
[2] https://github.com/clausing/docker-scripts
[3] https://docs.docker.com/engine/userguide/storagedriver/imagesandcontainers/
[4] https://integratedcode.us/2016/08/30/storage-drivers-in-docker-a-deep-dive/

---------------
Jim Clausing, GIAC GSE #26
jclausing --at-- isc [dot] sans (dot) edu