Threat Level: green Handler on Duty: Manuel Humberto Santander Pelaez

SANS ISC: When Google isn't Google SANS ISC InfoSec Forums

Watch ISC TV. Great for NOCs, SOCs and Living Rooms: https://isctv.sans.edu

Sign Up for Free!   Forgot Password?
Log In or Sign Up for Free!
When Google isn't Google

Like many other exploit scripts, the recent "Plesk" exploit used a fake user agent of "Googlebot". Attackers assume that most web applications are happy to be indexed by Google and possibly ably no or less stringent filters. For example, some applications will show more content to Google that is not readily displayed to normal users unless these users sign up, solve a captcha or even pay.

Google however makes it pretty easy to distinguish "real" Google bots from fake once. The IP address used by Google will reverse resolve to crawl-a-b-c-d.googlebot.com, where a-b-c-d is the IP address of the bot. In addition, this host name will resolve to the IP address used. In order to validate if a google bot is "real", the lookup against .googlebot.com is required. An attacker could fake the reverse lookup if the attacker can provide reverse DNS for the IP address used by the attacker.

Personally, I use  a little shell script to extract "fake google" spiders from my logs:

 

#!/bin/sh
 
# you may need to adjut the "cut" parameter and file name to match your own log format.
for b in `grep 'Googlebot' /var/log/httpd/*access_log | cut -f 2 -d' ' | sort -u`; do
  h=`host $b`
  if echo $h | grep -e ' crawl-.*\.googlebot\.com\.$'; then
    h=`echo $h | cut -f5 -d' '`
    n=`host $h | cut -f4 -d' '`
    if [ "$n" = "$b" ] ; then
      echo ok $n $h $b
    else
      echo fake $b;
    fi
  else
    echo fake $b;
  fi
done

 

 

------

Johannes B. Ullrich, Ph.D.
SANS Technology Institute
Twitter

I will be teaching next: Defending Web Applications Security Essentials - SANS San Francisco Winter 2019

Johannes

3679 Posts
ISC Handler
Thought this would be relevant for this discussion:
https://github.com/kolodny/is_googlebot.php

Came across is the other day after reading the previous plesk posts.
Anonymous

Sign Up for Free or Log In to start participating in the conversation!