SANS ISC: Domain Whitelisting With Alexa and Umbrella Lists - update - Internet Security | DShield

• SANS Site Network
  • Current Site
  • Internet Storm Center
  • Other SANS Sites Help
  • Graduate Degree Programs
  • Security Training
  • Security Certification
  • Security Awareness Training
  • Penetration Testing
  • Industrial Control Systems
  • Cyber Defense Foundations
  • DFIR
  • Software Security
  • Government OnSite Training

SANS ISC InfoSec Forums

A was asked if I could share the files of my last diary entry: Domain Whitelisting With Alexa and Umbrella Lists.

You can find the files on my site here. And to teach you how to fish :-), here are the commands I used to produce these lists:

 

csv-cut.py -s "\t" 1 emd.txt > blacklist.txt
csv-lookup.py -s , -e blacklist.txt 0 top-1m-umbrella.csv 1 0 blacklist-umbrella.csv
csv-lookup.py -s , -e blacklist.txt 0 top-1m-alexa.csv 1 0 blacklist-alexa.csv

My csv tools can be found on my Beta GitHub repository.

My assumption when I read this blog post, was that the blacklisted domains would rank low in the Alexa and Umbrella lists. They don't, look at the histograms of the rankings.

Blacklisted domains with Alexa rank:

Blacklisted domains with Umbrella rank:

These long tail distributions indicate that blacklisted domains with higher ranks are more prevalent than those with lower ranks. This is also reflected in the ranking median (287,251 for Alexa and 393,879 for Umbrella) and average (350,553 for Alexa and 420,846 for Umbrella).

Conclusion: don't use Alexa and Umbrella top 1,000,000 lists as whitelists blindly, even if you just use the top 1000 or 10000.

Didier Stevens
Microsoft MVP Consumer Security
blog.DidierStevens.com DidierStevensLabs.com