CenturyLink Outage Causing Internet Wide Problems
Update From Centrylink at approx 15:00 UTC / 11:00 EDST:
The IP NOC with the assistance of the Operations Engineering team confirmed a routing issue to be preventing BGP sessions from establishing correctly. A configuration adjustment was deployed at a high level, and sessions began to re-establish with stability. As the change propagates through the affected devices, service affecting alarms continue to clear.Due to the nature of this outage, it may be necessary to reset your services locally at your equipment, or manually reset your BGP session. If after that action has been performed a service issue prevails, please contact the CenturyLink Repair Center for troubleshooting assistance.
Early this morning (US East Coast time), CenturyLink started having problems with routes passing AS3356. This network is central in routing a large part of internet traffic, and the outage is still causing problems for many services like for example OpenDNS, Duo Security, Cloudflare, Imperva (a service SANS, and isc.sans.edu uses).
At this point, there is no indication that this is an attack. This looks so far like a misconfiguration or maybe a hardware failure.
If a network like AS3356 has problems handling traffic, a typical response is to route traffic via a different network. As a customer of CenturyLink, you would disconnect from CenturyLink, and instead, advertise your IP address space via a different backup ISP. It looks like this failed for two reasons in many cases:
- AS3356 itself did not withdraw these routes once the customer disconnected. So the rest of the internet still continued to believe CenturyLink, and is sending traffic to them vs sending it to the backup ISP
- Which backup ISP? AS3356 used to belong to Level 3. Level 3 was purchased by CenturyLink. CenturyLink also merged with other ISPs/NSPs like for example Qwest. This is another example of how the Internet has long been much less diverse than it should be.
What can you do about this as an end-user? Not much. Wait for CenturyLink to find a network engineer who is fluent enough in BGP to fix this. Some customers of CenturyLink report estimated times to resolution quoted at 1 pm ET. But there is no public acknowledgment of this time. I have seen some traffic come back to ISC/Imperva. For ISC, we also have dshield.org which does not appear to be affected (different ISP setup).
You may want to disable affected services like OpenDNS as they may make things worse. Google DNS appears to be working. You could also decide to not require 2FA if you rely on a service like Duo. But I will live that risk decision up to you. And attackers could take advantage of widespread disabling of Duo.
Also: the companies I named here are just some notable once I ran across as affected. There are likely more.
---
Johannes B. Ullrich, Ph.D. , Dean of Research, SANS Technology Institute
Twitter|
Comments
www
Nov 17th 2022
6 months ago
EEW
Nov 17th 2022
6 months ago
qwq
Nov 17th 2022
6 months ago
mashood
Nov 17th 2022
6 months ago
isc.sans.edu
Nov 23rd 2022
6 months ago
isc.sans.edu
Nov 23rd 2022
6 months ago
isc.sans.edu
Dec 3rd 2022
6 months ago
isc.sans.edu
Dec 3rd 2022
6 months ago
<a hreaf="https://technolytical.com/">the social network</a> is described as follows because they respect your privacy and keep your data secure. The social networks are not interested in collecting data about you. They don't care about what you're doing, or what you like. They don't want to know who you talk to, or where you go.
<a hreaf="https://technolytical.com/">the social network</a> is not interested in collecting data about you. They don't care about what you're doing, or what you like. They don't want to know who you talk to, or where you go. The social networks only collect the minimum amount of information required for the service that they provide. Your personal information is kept private, and is never shared with other companies without your permission
isc.sans.edu
Dec 26th 2022
5 months ago
isc.sans.edu
Dec 26th 2022
5 months ago