Outsourcing critical infrastructure (such as DNS)
Migrating everything to “cloud” or various online services is becoming increasingly popular in last couple of years (and will probably not stop). However, leaving our most valuable jewels with someone else makes a lot of security people (me included) nervous.
During some of the latest external penetration tests I noticed an increasing trend of companies moving some of their services to various cloud solutions or to their providers.
When performing the reconnaissance phase of a penetration test, a very important part is to try to map the target assets/network as much as possible. Of course, DNS is one of the most important services, which must be available publicly and which sometimes gets misconfigured (how many times have you seen DNS information about internal assets that gets published on the Internet).
So, a simple dig query will tell us what the DNS servers for our target client are:
$ dig ns target.com
;; QUESTION SECTION:
;target.com. IN NS
;; ANSWER SECTION:
target.com. 1365 IN NS zion.target.com.
target.com. 1365 IN NS morpheus.target.com.
;; ADDITIONAL SECTION:
zion.target.com. 1366 IN A 212.71.248.24
morpheus.target.com. 9018 IN A 88.198.75.37
Now what do we have here? Things look generally OK – there are two DNS servers for our target domain, at two different hosting companies (or, for the sake of this article, we can pretend that they are at the target company’s ISP).
The problem here is that the trust for our most critical infrastructure now completely lays with the ISP (or a hosting company). Why is that a problem? Well – remember all those attacks that happen when an account at a registrar gets hacked and domain information (including DNS servers) gets changed? The same thing applies here – DNS servers are the key to our kingdom.
I recently had to work on an incident that included such an attack where the NS records were modified silently by an attacker that got access to the hosting company. And that attack was very sneaky – the attacker modified only selected DNS records: the MX records. So, for couple of hours during a business day, the attacker changed the MX records (only) to point to his SMTP servers. Those servers were configured just to relay e-mail (and additionally, a specific version of an SMTP server was used to prevent adding headers) to the real destination. This was a very simple Man-in-the-Middle attack that was, unfortunately, very successful for the attacker as he was able to collect and analyze absolutely all e-mail sent to the victim company. While he was not able to see the outgoing e-mails, just remember how many times you’ve seen people actually remove the original e-mail (or reply inline) when replying? This is indeed very rare these days although those older will remember that once upon a time it was part of netiquette.
Lessons learned here? While outsourcing DNS servers is not necessarily a bad thing, be aware of the risks that come with it (and with cloud usage in general). For this particular case, depending on the business the target company is in, I most of the times recommend that the DNS servers, as critical infrastructure, are kept on premises and managed by local staff. This way, you decrease the risk of the hosting company getting pwned, or simply risk of a disgruntled employee at the hosting company.
If you do decide to outsource DNS anyway, ask yourself first if you would detect the attack I mentioned? What controls do you have in place for detecting such an attack?
Implementation of additional monitoring controls such as regularly checking your critical DNS records (such as NS, MX and possibly A records for critical names) can go a long way and is very inexpensive. For this particular case, SPF would help as well, but unfortunately the majority of servers will simply use SPF information for spam detection and only very rare MUA’s will warn users when SPF records do not match the sending IP address.
Have similar outsourcing war stories? Let us know!
Comments
Yep, when "our" data is out of "our" control it is no longer "our" data. The cost of "mobility" will result in fighting "immobility"
Anonymous
Aug 19th 2015
9 years ago
Also, we go to pains to watch for, log, and mitigate brute force attacks on accounts. Does your cloudy app provider do this? Probably not. We log access to various apps - does user A login to service B from more than one geography in a short time, or are there multiple failed logins, etc. Does your cloud app provider log access and can you see/audit those logs? Probably not.
This isn't to say nothing should ever be moved to a cloud service. We just need to think a bit - what data is going to be in this cloud app and how big a disaster would it be if some no-goodnik or a competitor could login to this cloud app as one of your users.
I'm seeing a lot of companies outsource their email services. Now anyone, anywhere, with any device can access their email. Yay cloud! And boy is it embarrassing when dozens of their users' email accounts are compromised and start sending spam/phish to everyone in their addressbooks, all their customers, prospects, etc. Yay cloud...
Anonymous
Aug 19th 2015
9 years ago
In two consecutive phrases first the NS records are changed, then the MX records.
Changing the NS records may be a result of a breakin at the registrar;
the party where you buy the domain name and which is intermediary between domain owner and registry,
operator of the top level domain.
Such incidents have indeed happened, but are unrelated to the fact that your DNS is or is not outsourced.
Changing (only) the MX records suggest the breakin occured at the master name server,
because it implies a change to the content of a zone file.
But outsourcing DNS name servers does not imply the content is no longer under your control.
In fact, outsourced name servers should be slaves;
the modifiable content should stay on a - hidden - master, still under your control.
Or does the article suggest that both cloud providers have broken into,
their name servers roles changed for the domain - from slave to master;
and, in the local copies, the MX records got changed ?
The shown NS and A records look generally OK ?
I see only 2 IPv4 addresses ! North Korea, .kp, has 3 and they went down for hours, late 2014...
Missing also are IPv6 addresses.
On the other hand, one or both of the IPv4 addresses could be "anycast",
which changes the picture altogether, because an attack is diverted to the nearest server.
Monitoring what authoritative name servers publish is indeed wise.
But I don't leave my customers in the dark :
my suggestion is scripts, executed by some network monitoring server,
that alert when they find published data to be different from expected data.
Is outsourcing to the cloud bad or good ?
I see hosting in the cloud as a design element to increase robustness against (D)DOS.
And one important criterium you should consider is DDOS mitigation capabilities of the cloud provider.
They are mandatory !
Because they will protect you when another of their customers is being attacked;
and, of course, your own outsourced services, if they are focussed.
Anonymous
Aug 20th 2015
9 years ago