Recently, I was involved in a network outage caused by a defective pfSense firewall appliance. Due to storage issues (with the onboard flash), the firewall did not boot anymore. This can be quickly solved from a hardware point of view because this firewall model has a slot to install an M2-compatible flash device and boot from it. But, there was a problem with the configuration. The last backup they had was pretty old, and they made a lot of changes. No debate about the fact that a robust backup process should have been implemented. Let's focus on the challenge of recovering the last configuration from the firewall. Challenge accepted!

First, I booted the firewall on an emergency USB stick and serial console access. First tip: always keep your console cables and emergency boot devices in a safe place. Once on the firewall, I tried to access the last configuration (stored as a big XML file) without luck. It was impossible to mount the corrupted filesystem. Because the file system was too big, it was impossible to take an image and store it on a USB key. Let's dump it through the network! I manually configured a NIC to connect to a server and used our best friend: netcat!

# dd if=/dev/mmcsd0 | nc 192.168.254.8 8888

Let's boot a SIFT Workstation to start a listener:

# nc -l -p 8888 >pfsense.raw

After a long time, I had an image of the corrupted file system on a remote host, ready to be investigated. I did not change the default block size ("bs") parameter to ensure a safe copy and avoid errors.

Now, let's find a way to extract interesting information from the disk image (remember, I need to extract the last configuration). I tried to mount the disk but again, no luck. So I decided to speed up the analysis and try to perform data carving. When data carving is mentioned, many people think about the tool bulk_extractor[1]. This tool tries to find interesting pieces of information from a disk image. It looks for "structured information" (email addresses, credit card numbers, URLs, images, ...). In my case, I was looking for a specific file and decided to use another tool: Scalpel[2]. This one performs file carving operations based on patterns that describe particular files or data fragment "types". You may define these patterns based on fixed strings or regular expressions. The tool is pretty old but you can find any modern file due to its flexible configuration. The good news is that Scalpel is installed on the SIFT Workstation[3]. By default, XML files are not carved. Let's enable this in the configuration file (/etc/scalpel/scalpel.conf). Just add the following line:

xml     n   10000000    <?xml    </pfsense>

This instructs Scalpel to search for XML files starting with "<?xml" and ending with "</pfsense>" (That's how pfSense configurations are stored). The file size is a maximum of 10M, and we don't verify the case ('n') in patterns.

Now, let's cross our fingers and scan the disk image:

# scalpel -v -c /etc/scalpel/scalpel.conf -o /tmp/carved pfsense.raw

Once the scan is completed, an audit file is generated:

Scalpel version 1.60 audit file
Started at Thu May 22 12:28:06 2023
Command line:
scalpel -c /etc/scalpel/scalpel.conf -o /tmp/carved pfsense.raw 

Output directory: /tmp/carved
Configuration file: /etc/scalpel/scalpel.conf

Opening target "H="

The following files were carved:
File          Start            Chop        Length        Extracted From
00000003.xml    156532736        NO          8384365        pfsense.raw
00000002.xml    156368896        NO          8548205        pfsense.raw
00000001.xml    156303360        NO          8613741        pfsense.raw
...

This time, it was successful, and 52 files were carved. Which one is the right configuration? You need to remove the false positive ones (for example, the smallest ones). You can also grep for interesting strings. After a few extra checks, we were confident about the right file. We copied back to the firewall with the fresh install, rebooted, and the system was back alive!

In conclusion, DFIR techniques can be very helpful when your infrastructure is down, but it will cost time (read: money!). The best is, of course, to have a strong backup/restore process.

[1] https://github.com/simsong/bulk_extractor
[2] https://github.com/sleuthkit/scalpel
[3] https://www.sans.org/tools/sift-workstation/

Xavier Mertens (@xme)
Xameco
Senior ISC Handler - Freelance Cyber Security Consultant
PGP Key