A Tale of Two Admins (and no Change Control)

A Tale of Two Admins (and no Change Control)
I have a client who's done the right thing, they've broken out their test environment from their production environment. The production environment is in a colocation facility, and uses a different firewall. The test environment is in the office location, and shares the office subnet and the office firewall. So sort-of the right thing, they're moving in the right direction - I would have given the test lab it's own firewalled DMZ subnet. About two years ago, one of the server admins asked the office firewall administrator to open port 3389 (RDP) to a test box, so that they could continue their build at home. Not a great solution - I would have told him to VPN in and do it without changing the firewall - but it was done, the build got done and life moved on. Unfortunately, the firewall change was not documented, was not remembered and was not backed out. Fast forward 2 years. The two folks from 2 years back have both moved on to other positions and/or companies, and a new server admin is building a new Hyper-V server in the test environment. They're just about to deploy to producion when he notices RDP connections to it from our friends in China. Yes, that undocumented change had come home to roost! So, after we did the post mortem, what did they learn? There's no fixing a compromised hypervisor - NFO (Nuke from Orbit) - repartition the RAID Array and starting over is always the best advice. Hypervisors don't need a GUI - they shouldn't be RDP'ing into that box for admin in the first place. DOCUMENT ALL FIREWALL CHANGES. HAVE A CHANGE CONTROL PROCESS. Happily, they've got a formal change control process now. On the firewall, there's an assessment step on all changes, to decide if the requested change is a good idea in the first place (open RDP was a singularly BAD idea). Finally, they now run a basic NMAP scan (all addresses in the range, all ports) of the office environment from the colo, and the results are run through diff, comparing it for changes against yesterday's results. This client is lucky in this regard because they have 2 separate locations that can scan each other, but in a more typical situation, the folks responsible for security might do this from their laptop, scanning from home after work or before driving in each day. You'd be surprised what a full port scan might find - those issues we're stuck with on open ports on home firewalls (https://isc.sans.edu/forums/diary/Scans+Increase+for+New+Linksys+Backdoor+32764+TCP+/17336 and https://isc.sans.edu/diary/Exposed+UPNP+Devices/15040 for instance) would have been caught a long time ago if more folks scanned their infrastructure from the untrusted outside network! Mind you, typically home users never patch their firewalls anyway, so all those open PNP and other backdoor ports are with us for the long haul now. Do you regularly scan your firewall from the outside? Does your scan highlight changes, or are you looking for just vulnerabilities (using Nessus or similar) rather than changes? Let us know in our comment form below. =============== Rob VandenBrink Metafore	Rob VandenBrink 578 Posts ISC Handler Feb 10th 2014
Thread locked Subscribe	Feb 10th 2014 8 years ago
Where they attempts to connect on 3389 or actual established authenticated sessions? Either way I wouldn't be happy but they are significantly different.	Anonymous
Quote	Feb 11th 2014 8 years ago
I find the Chinese attempts to "hack" open ports relatively half-hearted. Rarely does a Chinese IP try to seriously brute-force a password. Maybe a Chinese reader can verify this, but I suspect a standard cybersecurity syllabus being taught widely in junior colleges, etc, country wide. The probes come from IP blocks "from everywhere" in China. Although they could come from a bot net as well, but a bot net that's largely concentrated inside the country. I think they're doing both themselves and the rest of the world a favor with these basic obvious probes - they learn cybersecurity, and maybe all of us learn to configure and tune our IDS techniques. It would be interesting to snare one of these probes with a honeypot and see what they are after, if anything. "Free Pen Tests from our friends in China"!	Anonymous
Quote	Feb 11th 2014 8 years ago

I have a client who's done the right thing, they've broken out their test environment from their production environment. The production environment is in a colocation facility, and uses a different firewall. The test environment is in the office location, and shares the office subnet and the office firewall. So sort-of the right thing, they're moving in the right direction - I would have given the test lab it's own firewalled DMZ subnet.

About two years ago, one of the server admins asked the office firewall administrator to open port 3389 (RDP) to a test box, so that they could continue their build at home. Not a great solution - I would have told him to VPN in and do it without changing the firewall - but it was done, the build got done and life moved on.

Unfortunately, the firewall change was not documented, was not remembered and was not backed out.

Fast forward 2 years. The two folks from 2 years back have both moved on to other positions and/or companies, and a new server admin is building a new Hyper-V server in the test environment. They're just about to deploy to producion when he notices RDP connections to it from our friends in China. Yes, that undocumented change had come home to roost!

So, after we did the post mortem, what did they learn?

There's no fixing a compromised hypervisor - NFO (Nuke from Orbit) - repartition the RAID Array and starting over is always the best advice.
Hypervisors don't need a GUI - they shouldn't be RDP'ing into that box for admin in the first place.
DOCUMENT ALL FIREWALL CHANGES. HAVE A CHANGE CONTROL PROCESS. Happily, they've got a formal change control process now. On the firewall, there's an assessment step on all changes, to decide if the requested change is a good idea in the first place (open RDP was a singularly BAD idea).
Finally, they now run a basic NMAP scan (all addresses in the range, all ports) of the office environment from the colo, and the results are run through diff, comparing it for changes against yesterday's results. This client is lucky in this regard because they have 2 separate locations that can scan each other, but in a more typical situation, the folks responsible for security might do this from their laptop, scanning from home after work or before driving in each day.

You'd be surprised what a full port scan might find - those issues we're stuck with on open ports on home firewalls (https://isc.sans.edu/forums/diary/Scans+Increase+for+New+Linksys+Backdoor+32764+TCP+/17336 and https://isc.sans.edu/diary/Exposed+UPNP+Devices/15040 for instance) would have been caught a long time ago if more folks scanned their infrastructure from the untrusted outside network! Mind you, typically home users never patch their firewalls anyway, so all those open PNP and other backdoor ports are with us for the long haul now.

Do you regularly scan your firewall from the outside? Does your scan highlight changes, or are you looking for just vulnerabilities (using Nessus or similar) rather than changes? Let us know in our comment form below.

===============
Rob VandenBrink
Metafore

Rob VandenBrink

578 Posts
ISC Handler

Feb 10th 2014

Where they attempts to connect on 3389 or actual established authenticated sessions? Either way I wouldn't be happy but they are significantly different.

Anonymous

I find the Chinese attempts to "hack" open ports relatively half-hearted. Rarely does a Chinese IP try to seriously brute-force a password. Maybe a Chinese reader can verify this, but I suspect a standard cybersecurity syllabus being taught widely in junior colleges, etc, country wide. The probes come from IP blocks "from everywhere" in China. Although they could come from a bot net as well, but a bot net that's largely concentrated inside the country.

I think they're doing both themselves and the rest of the world a favor with these basic obvious probes - they learn cybersecurity, and maybe all of us learn to configure and tune our IDS techniques. It would be interesting to snare one of these probes with a honeypot and see what they are after, if anything.

"Free Pen Tests from our friends in China"!

Anonymous