Threat Level: green Handler on Duty: Xavier Mertens

SANS ISC: SANS Internet Storm Center SANS Internet Storm Center


Sign Up for Free!   Forgot Password?
Log In or Sign Up for Free!

Latest Diaries

Easy Access to the NIST RDS Database

Published: 2021-06-19
Last Updated: 2021-06-19 10:27:32 UTC
by Xavier Mertens (Version: 1)
0 comment(s)

When you're facing some suspicious files while performing forensic investigations or analyzing malware components, it's always interesting to know these files are legit or malicious/modified. One of the key sources to verify hashes is provided by NIST and is called the NSLR project ("National Software Reference Library")[1]. They build "Reference Data Set" (RDS) of information that can be queried to verify a file hash. These RDS are available to download[2] but, as you may expect, there are huge (they are provided as ISO files between 500MB to 4GB!)

CIRCL[3], the Luxembourg CERT, has a good reputation to offer/participate in services like MISP, a passive DNS service, etc. They are now offering an API to query the NIST RDS via HTTP or DNS requests!

How does it work? You can use the FQDN hashlookup.circl.lu like this:

PS C:\Users\xavie> curl.exe -X GET https://hashlookup.circl.lu/lookup/md5/8ED4B4ED952526D89899E723F3488DE4 -H 'Accept: application/json'
{
  "CRC32": "7A5407CA", 
  "FileName": "wow64_microsoft-windows-i..timezones.resources_31bf3856ad364e35_10.0.16299.579_de-de_f24979c73226184d.manifest", 
  "FileSize": "2520", 
  "MD5": "8ED4B4ED952526D89899E723F3488DE4", 
  "OpSystemCode": {"MfgCode": "1006", "OpSystemCode": "362", "OpSystemName": "TBD", "OpSystemVersion": "none"},
  "ProductCode": {
    "ApplicationType": "Security", 
    "Language": "Multilanguage", 
    "MfgCode": "608", 
    "OpSystemCode": "868", 
    "ProductCode": "190742", 
    "ProductName": "Cumulative Update for Windows Server 2016 for x64 (KB4338817)", 
    "ProductVersion": "1709"
  },
  "SHA-1": "00000079FD7AAC9B2F9C988C50750E1F50B27EB5", "SpecialCode": ""
}

You can also query the database via a DNS request (my preferred method!):

root@remnux:/# dig +short -t TXT 8ED4B4ED952526D89899E723F3488DE4.dns.hashlookup.circl.lu | jq -r . | jq .
{
  "CRC32": "7A5407CA",
  "FileName": "wow64_microsoft-windows-i..timezones.resources_31bf3856ad364e35_10.0.16299.579_de-de_f24979c73226184d.manifest",
  "FileSize": "2520",
"MD5": "8ED4B4ED952526D89899E723F3488DE4",
"OpSystemCode": {"MfgCode": "1006", "OpSystemCode": "362", "OpSystemName": "TBD", "OpSystemVersion": "none"},
"ProductCode": {
  "ApplicationType": "Security",
  "Language": "Multilanguage",
  "MfgCode": "608",
  "OpSystemCode": "868",
  "ProductCode": "190742",
  "ProductName": "Cumulative Update for Windows Server 2016 for x64 (KB4338817)",
  "ProductVersion": "1709"
  },
  "SHA-1": "00000079FD7AAC9B2F9C988C50750E1F50B27EB5", "SpecialCode": ""
}

SHA1 and MD5 hashes are supported and you can also submit bulk requests. More documentation is available here[4].

If you need to scan a complete filesystem, my recommendation is of course to use your own local copy of the NIST databases! Many forensic tools allow you to compare hashes against a local database. But when you quickly need to check a single file or a restricted set of files, this service is perfect! Thank you to CIRCL for providing this to the community! The website says also that more databases will be added in the future!

[1] https://www.nist.gov/itl/ssd/software-quality-group/national-software-reference-library-nsrl
[2] https://www.nist.gov/itl/ssd/software-quality-group/national-software-reference-library-nsrl/nsrl-download/current-rds
[3] https://circl.lu
[4] https://gist.github.com/adulau/4191d44e30fc01df38f1d5fe605fa920#file-hashlookup-circl-lu-md

Xavier Mertens (@xme)
Senior ISC Handler - Freelance Cyber Security Consultant
PGP Key

0 comment(s)

If you have more information or corrections regarding our diary, please share.

Recent Diaries

Open redirects ... and why Phishers love them
Jun 18th 2021
1 day ago by Daniel (0 comments)

Network Forensics on Azure VMs (Part #2)
Jun 18th 2021
1 day ago by Daniel (0 comments)

Network Forensics on Azure VMs (Part #1)
Jun 17th 2021
2 days ago by Daniel (0 comments)

June 2021 Forensic Contest
Jun 16th 2021
3 days ago by Brad (0 comments)

Multi Perimeter Device Exploit Mirai Version Hunting For Sonicwall, DLink, Cisco and more
Jun 15th 2021
4 days ago by Johannes (0 comments)

Update: mac-robber.py
Jun 13th 2021
6 days ago by Jim (0 comments)

Fortinet Targeted for Unpatched SSL VPN Discovery Activity
Jun 12th 2021
1 week ago by Guy (0 comments)

View All Diaries →

Latest Discussions

Dshield Sensor
created Jun 8th 2021
1 week ago by Rick (0 replies)

API port data
created Apr 25th 2021
1 month ago by JJ (1 reply)

RSS feed containing non-XML compatible characters
created Apr 14th 2021
2 months ago by Anonymous (1 reply)

Handler's Diary (Full text) RSS Feeds stopt working due to a typo
created Mar 5th 2021
3 months ago by bas.auer@auerplace.nl (0 replies)

port_scan issue in Snort3
created Feb 23rd 2021
3 months ago by astraea (0 replies)

View All Forums →

Latest News

Top Diaries

Maldocs: Protection Passwords
Feb 28th 2021
3 months ago by DidierStevens (0 comments)

An infection from Rig exploit kit
Jun 17th 2019
2 years ago by Brad (0 comments)

Qakbot infection with Cobalt Strike
Mar 3rd 2021
3 months ago by Brad (0 comments)

Adversary Simulation with Sim
Mar 2nd 2021
3 months ago by Russ McRee (0 comments)

Fun with DNS over TLS (DoT)
Mar 1st 2021
3 months ago by Rob VandenBrink (0 comments)