Collecting IOCs from IMAP Folder
I've plenty of subscriptions to "cyber security" mailing lists that generate a lot of traffic. Even if we try to get rid of emails, that's a fact: email remains a key communication channel. Some mailing lists posts contain interesting indicators of compromize. So, I searched for a nice way to extract them in an automated way (and to correlate them with other data). I did not find a solution ready to use that matched my requirements:
- Connect to any mailbox (preferably via IMAP)
- Produce data easy to process (JSON)
- Be easy to deploy (Docker)
So, I built my own Docker image... It is based on the following components:
- procmail
- getmail
- some Python libraries
- The project es_mail_intel[1]
The last tool is an old project that achieves exactly why I expect: It extracts IOCs from emails and stores them in ElasticSearch. But, if you don't want ElasticSearch, it can also produce a JSON file! Parsing emails is a pain! So, I did not want to write my own parser.
Data are processed in this way: Emails are fetched via IMAP at regular intervals by getmail and pushed to procmail. It pushes them to the Python script that extracts interesting data.
IMAP data >> getmail >> procmail >> mail_parser2json_extract.py >> JSON data
Here is my Dockerfile:
FROM ubuntu:18.04 MAINTAINER Xavier Mertens <xavier@rootshell.be> RUN apt-get update && \ DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends -y \ ca-certificates \ getmail \ procmail \ git \ python \ python-ipaddress \ python-pdfminer \ python-elasticsearch \ python-xlrd \ && \ apt-get clean && \ rm -rf /var/lib/apt/lists/* RUN mkdir -p /root/.getmail RUN git clone https://github.com/clverhack/es_email_intel.git /opt/es_email_intel COPY getmail.conf / RUN echo ":0" >>/procmailrc RUN echo "|/opt/es_email_intel/mail_parser2json_extract.py 2 >>/log/ioc.json" >>/procmailrc COPY run.sh / RUN chmod u+x /run.sh RUN touch /tmp/firstboot CMD ["/run.sh"]
It needs a getmail.conf with the parameters of the mailbox you'd like to monitor:
[retriever] type = SimpleIMAPSSLRetriever server = CONF_SERVER username = CONF_LOGIN password = CONF_PASSWORD [destination] type = MDA_external path = /usr/bin/procmail user = getmail group = getmail arguments = ('/procmailrc', ) [options] verbose=1 read_all=false delete=false delete_after=0 delete_bigger_than=0 max_bytes_per_session=0 max_message_size=0 max_messages_per_session=0 delivered_to=false received=false message_log=/log/getmail.log message_log_syslog=false message_log_verbose=true
getmail is a very powerful tool with plenty of options. Just have a look at the documentation[2] to find your best way to interact with your mailboxes. The script 'run.sh' will be executed by the container and, at first boot, configure your credentials:
#!/bin/bash if [ -r /tmp/firstboot ]; then sed -i "s|CONF_SERVER|$IMAP_SERVER|g" /getmail.conf sed -i "s|CONF_LOGIN|$IMAP_USER|g" /getmail.conf sed -i "s|CONF_PASSWORD|$IMAP_PASS|g" /getmail.conf groupadd getmail useradd -u $UID -g getmail -d /home/getmail getmail mkdir /home/getmail && chown getmail:getmail /home/getmail test -d /log || mkdir /log touch /log/getmail.log /log/ioc.json chown -R root:getmail /log chmod -R g+w /log rm /tmp/firstboot fi while true do /usr/bin/getmail -r /getmail.conf sleep $IMAP_WAIT done
And, finally, my docker-compose.yml file:
version: '3' services: iocollector: build: . image: "xme/iocollector" restart: always hostname: iocollector container_name: iocollector volumes: - /etc/localtime:/etc/localtime:ro - /data/iocollector/log:/log environment: - UID=1000 - IMAP_SERVER=<server_ip_or_fqdn> - IMAP_USER=<username> - IMAP_PASS=<password> - IMAP_WAIT=30 network_mode: bridge
Start your docker and it will populate the mapped /log directory with an 'ioc.json' file:
{ "bitcoin_wallet": [ "1ncgvmeqno77x5pqbp7yi4779si", "3k04tl9upoapvqrcbntcsnglvcr5yn", "366vraic2k6ocpol7q9ck8vs2o5by0dp", "1fgtytuwedagp1krbukpd1dkkpfs", "3pmusoxmgn1qmrzedtxsqbua6dua6t", "34pg4dq162op34bd1aospshvsyxwhsxyo", "38wvzcnz49gvuwo29iuf8vxyssvmfi66w", "1nvgzipjvq5hdd16iszjdx9jzvf3", "1igrmo3w477lnb4e3tsz3gfdrciwo", "1yenq1jfsx54ptvvdtbplcqumu2dz3we", "3wusjglqxkkro3nfzd1mkwx1qqf", "1tjkoizmzrof4kgkok3jciwysmrxprj7" ], "ctime": "Thu Mar 5 17:54:23 2020", "domain": [ "xpressvpngoodforpsdgo.duckdns.org", "windowsfirewallsecurityauthorise.duckdns.org", "cartonal.net", "asco.az", "duckdns.org", "dnverificationfailedbymicrosoftthanks.duckdns.org", "saranadinamika.co.id", ... "windowsdefenderserversecuresofficeiq.duckdns.org", "tecnol2.com", "synergiespakistan.com", "chxsndy3manufacturingandinsurancebusines.duckdns.org", "ecomglobalpurchasecentercom.duckdns.org", "globalsharesecurefilesgood.duckdns.org", "ewrt55.com", "matriximportsusa.com", "pstv-energy.com", "msofficecloudtransferfileprotocolsys.duckdns.org", "workshop002.duckdns.org", "globalfbdnsaddressgoogle.duckdns.org", "www.virustotal.com", "marroiq.com", "challengerevertprocessupdate.duckdns.org", "windowsdefenderserversecureserver.duckdns.org", "windowsdefenderserversecuresofficew.duckdns.org" ], "email": [ "1042@home7.solarlog-web.de", ... ], "epoch": "1583427263", "filename": [ "64443.exe", "soa.xlsx", "copy.xlsx", "p.o.xlsx", "agenttesla.xlsx", "vbc.exe", "products.xlsx", "quote.pdf" ], "ipv4": [ ... ], "md5": [], "message_text": "...", "mutex": [], "sha1": [], "sha256": [ "3a0965bd363e46e8c30590d7ddfd22bfc353728c6311c3ca61479da6f874bd5f", "f3a98da7e0a2c7cfe0907f5b0b769a4ff3bad45d7dce7ed905d6c3c2bf835201", "cb01d834237a37d3825f4075961cf6eda5f856af13526d699e4f5c93edf00c15", "2ef88915bb08f6d0d4c5fb2a90777930ad4e0d0b757e4b2b51988813338470f9", "df852235358fe361ba0577ad4c064b642148b2ac52d5857316d20bca33ea3d04", "172542fe3f7aae11deaf6960652d0074a1a7d83ae5596951ad69ba151a9f9630" ], "ssdeep": [], "url": [ "windowsdefenderserversecureserver.duckdns.org/mich/vbc.exe", "globalsharesecurefilesgood.duckdns.org/love/vbc.exe", "windowsfirewallsecurityauthorise.duckdns.org/big/vbc.exe", "workshop002.duckdns.org/vbc.exe", "windowsdefenderserversecuresofficew.duckdns.org/windows/vbc.exe", "dnverificationfailedbymicrosoftthanks.duckdns.org/lee/vbc.exe", "dnverificationfailedbymicrosoftthanks.duckdns.org/lavin/vbc.exe", "ecomglobalpurchasecentercom.duckdns.org/love/vbc.exe", "windowsdefenderserversecuresofficew.duckdns.org/reg/vbc.exe", ... "globalfbdnsaddressgoogle.duckdns.org/lvc/vbc.exe" ] }
Note: The complete is email is parsed. You will find in the JSON file all SMTP headers, the email body, etc. Less relevant for IOC's but still interesting in some cases (by example, to analyze spam).
Here is a recap of the data flow:
[1] https://github.com/clverhack/es_email_intel.git
[2] http://pyropus.ca/software/getmail/
Xavier Mertens (@xme)
Senior ISC Handler - Freelance Cyber Security Consultant
PGP Key
Comments
www
Nov 17th 2022
6 months ago
EEW
Nov 17th 2022
6 months ago
qwq
Nov 17th 2022
6 months ago
mashood
Nov 17th 2022
6 months ago
isc.sans.edu
Nov 23rd 2022
6 months ago
isc.sans.edu
Nov 23rd 2022
6 months ago
isc.sans.edu
Dec 3rd 2022
6 months ago
isc.sans.edu
Dec 3rd 2022
6 months ago
<a hreaf="https://technolytical.com/">the social network</a> is described as follows because they respect your privacy and keep your data secure. The social networks are not interested in collecting data about you. They don't care about what you're doing, or what you like. They don't want to know who you talk to, or where you go.
<a hreaf="https://technolytical.com/">the social network</a> is not interested in collecting data about you. They don't care about what you're doing, or what you like. They don't want to know who you talk to, or where you go. The social networks only collect the minimum amount of information required for the service that they provide. Your personal information is kept private, and is never shared with other companies without your permission
isc.sans.edu
Dec 26th 2022
5 months ago
isc.sans.edu
Dec 26th 2022
5 months ago