Memory dumping for incident response is nothing new, but ever since they locked down access to direct memory (/dev/mem) on Linux, I’ve had bad experiences dumping memory. I usually end up crashing the server about 60 percent of the time while collecting data with Fmem.
A new version of Linux memory dumping utility rekall (previous called Winpmem) has recently came out. I’ve been testing it on the latest versions of Ubuntu and Redhat EL 5 and have not run into any issues with collection.
If you are fortunate enough to have an environment where you have groups of servers with the same patch levels, you should run the following steps on a non-compromised server. Additionally, if the compromised system is a VM, you can clone it and perform these actions on the clone. Make sure you collect all other volatile data (MACtimes, LSOF, PS ect..) before you dump memory as this may still cause instability to the system and you do not want to lose this data.
Preparing for collectionInstall Linux Kernel Headers Ubuntu >sudo apt-get install linux-headers-server zip CentOS/Redhat >yum install kernel-headers gcc
Download and Compile rekallWhen you run the makefile, it will automatically create part of the profile for the server. This will need to be copied off the server for analysis. >wget http://downloads.rekall.googlecode.com/git/Linux/linux_pmem_1.0RC1.tgz >tar -zxvf linux_pmem_1.0RC1.tgz >cd linux >make
Note:For Redhat/CentOS systems you will need to adjust the Makefile KHEADER variable.
Copy this file to your Volatility analysis machine under your volatility directory /plugins/overlays/linux/. Load the Kernel Driver>sudo insmod pmem.ko >sudo lsmod |grep pmem pmem 12680 0
Collect MemoryNow that the drive is loaded, a new device is accessible /dev/pmem. We want to copy the memory to an external device/share. #Items in {} need to be changed per incident to be useful for analysis >dcfldd if=/dev/pmem bs=512 conv=noerror,sync of=/{USBDRIVE}/ mount/{servername.date}.memory.dd hash=md5,sha256 hashlog=/{USBDRIVE}/{servername.date}.memory.dd-hash.log
Unload driver >sudo rmmod pmem.ko Analysis using RekallNow that collection is completed, we need to be able to examine the memory dump. Copy the memory image to your analysis workstation.
Install Rekall >sudo apt-get install python-pip python-dev >sudo pip install rekall
Build Rekall Profile We now need to create a profile that will work with Rekall. Convert the file that was copied from the server and name it something useful for future analysis. >rekal.py convert_profile 3.5.0-45-generic.zip Ubuntu3.5.0-45-generic.zip >rekal.py --profile ./Ubuntu3.5.0-45-generic.zip -f /media/mem.dd pslist
To enter the interactive shell, you do not add a plugin on the commandline >rekal.py --profile ./Ubuntu3.5.0-45-generic.zip -f /media/mem.dd
To list the available plugins, use the interactive shell: >rekal.py info[tab][tab]
plugins.arp plugins.check_idt plugins.convert_profile plugins.dwarfparser plugins.info plugins.lsof plugins.null plugins.psaux plugins.vmscan plugins.banner plugins.check_modules plugins.cpuinfo plugins.fetch_pdb …..
To get more info about a specific plugin use a ? after plugin name mem 12:38:31>plugins.pslist?
Some of the more useful plugins are:
Optional (If you want to use Volatility for analysis)I haven’t spent much time on this, but Volatility will not be able to use the rekall default profile. You also have to do the steps below to read the memory dump with Volatility. I’m guessing only a small change in the file is needed, but I have dug any deeper at this time.
>sudo apt-get install dwarfdump >wget https://volatility.googlecode.com/files/volatility-2.3.1.tar.gz >tar -zxvf volatility-2.3.1.tar.gz >cd volatility-2.3.1/tools/linux >make >zip Ubuntu{Kernel ver}.zip ./module.dwarf /boot/System.map-`uname -r`
For more information on Rekall http://docs.rekall.googlecode.com/git/tutorial.html
For more info on Volatility Linux analysis https://code.google.com/p/volatility/wiki/LinuxMemoryForensics
-- Tom Webb |
Tom 59 Posts ISC Handler Mar 7th 2014 |
Thread locked Subscribe |
Mar 7th 2014 8 years ago |
Hi Tom,
Another tool you may wish to look at for Linux memory forensics (both acquisition and analysis) is Second Look. It's got reliable memory acquisition, with pre-compiled modules for over 10000 kernels covering the major distros for the past 7+ years. Similarly, reference data is already there for you -- no need to find a clean system on which to create a profile. And it does integrity verification of the kernel and processes in memory -- so it can find much malware that Volatility and Rekall miss. Disclaimer: I am the product lead for Second Look. Andrew http://secondlookforensics.com/ |
Anonymous |
Quote |
Jun 4th 2014 8 years ago |
Sign Up for Free or Log In to start participating in the conversation!