An interesting blog post by Kristian Kielhofer describes how a specific SPI packet can "kill" an Intel Gigabit ethernet card [1]. If a card is exposed to this traffic, the system has to be physically power cycled. A reboot will not recover the system. The network card crashed whenever the value 0x32 or 0x33 was found at offset 0x47f. Kristian first noticed this happening for specific SIP packets, but in the end, it turned out that any packet with 0x32 at 0x47f caused the crash. Intel traced the problem to an EEPROM used in this specific card (82574L). There are some links in the comment to the blog suggesting that others have run into this problem before. For example, the commend: "ping -p 32 -s 1110 x.x.x.x" can crash an affected card remotely. [1] http://blog.krisk.org/2013/02/packets-of-death.html
------ |
Johannes 4479 Posts ISC Handler Feb 7th 2013 |
Thread locked Subscribe |
Feb 7th 2013 9 years ago |
Actually, it's just receiving a few particular values, in a particular position in the frame, - so a web page, or even just a modified image could trigger this.
Nasty, but probably worse case is as a DoS attack tool, as a crafted request could trigger the bug. His case was well, really obscure (a VoIP packet happened to trigger it), but only makes the story more interesting. |
DomMcIntyreDeVitto 45 Posts |
Quote |
Feb 6th 2013 9 years ago |
Some of the newer posts in Kristians link above refer to similar problems with ZTE MMC 3G USB Mobile Broadband dongles and their drivers. I have had similar things occur (BSOD driver failures) lately that require a full reboot and they seemed to have stopped after a Java update. There still seems to be some partial issues where an individual page on a website blows up IE but just closing down the offending page and not IE fixes the problem. Weird to say the least.
|
Anonymous |
Quote |
Feb 7th 2013 9 years ago |
Actually the ZTE MMC is my USB not the net card driver failures referred to in the link FYI.
|
Anonymous |
Quote |
Feb 7th 2013 9 years ago |
If it is only on a plug-in NIC, one can always change the NIC, but is there any chance the same bug exists on motherboard mounted NICs also? That would require replacing the entire motherboard -- unless it is on-site reflashable and Intel has a patch.
|
Moriah 133 Posts |
Quote |
Feb 7th 2013 9 years ago |
It surely can't be that easy? Or else a stream of encrypted (pseudorandom) data would trigger the bug after approx. 128 packets.
One of my server rentals has this model of NIC onboard. I sure hope they're not vulnerable, or else my provider has a whole datacentre full of these... |
Steven C. 171 Posts |
Quote |
Feb 7th 2013 9 years ago |
- http://www.securitytracker.com/id/1028089
"... A power cycle is required to return the system to normal operation. The original advisory is available at: http://blog.krisk.org/2013/02/packets-of-death.html Impact: A remote user can cause the target controller to crash. Solution: The vendor has issued a fix, available via customer support... [Rots o' ruck finding that number.] . |
Jack 160 Posts |
Quote |
Feb 7th 2013 9 years ago |
FYI:
I used the two pcaps ( pod-http-post.pcap & pod-icmp-ping.pcap ) from http://www.kriskinc.com/intel-pod using the tcpdump-edit instructions provided. I also added "-l 30" to retry. I have an Intel 82574L Gigabit CT adapter on an ASUS X79 board running CentOS 6.3, fully patched. The box says "Version E39199-008" "Date: 09/08/2012". I have been unable to duplicate the crash. I "ifconfig up"ed the adapter (without an IP). "tcpdump -p -i eth3" shows the machine receiving the frames, but it doesn't stop working. |
MarkJx 5 Posts |
Quote |
Feb 7th 2013 9 years ago |
Intel technical support number: 916-377-7000
|
JeffSoh 31 Posts |
Quote |
Feb 7th 2013 9 years ago |
We uploaded the offending packet to our online packet viewer, CloudShark, with some notes on what to look for and links to Kristian's articles on the topic. Check it out here:
http://appliance.cloudshark.org/news/cloudshark-in-the-wild/intel-packet-of-death-capture/ |
JeffSoh 1 Posts |
Quote |
Feb 7th 2013 9 years ago |
I wonder if there's any chance the scope of this problem is bigger than 82574L
The symptoms sound very similar to an issue I have been having with an Intel 82567LM card over the passed 2 years. (In a Dell Optiplex 960) It just randomly just quits talking on the network....and to fix it you can either reboot, or unplug the NIC cable for 10 seconds and then plug it back in. The problem seems to semi-randomly happen after the sleeping computer wakes up While it is in this strange state, sometimes I can PING and sometimes I can't. If I can't ping, then if I IPCONFIG...the static IP Address no longer shows it is registered. IT shows 0.0.0.0 Very Strange! Motherboard has been changed out 3-4 times (by Dell....who sends me the same motherboard and embedded NIC I'm assuming) Network cable replaced Cable run to the closet tested Different network port used Drivers/BIOS/chipset drivers updated I also seem to be having sporadic problems with 82579LM cards that are in systems that were brought online last year. After the system wakes up for its nightly backup, the backup server can not contact the system. Doesn't happen all the time tho.... pretty random and sporadic.... |
K-Dee 68 Posts |
Quote |
Feb 8th 2013 9 years ago |
Thanks for updating the article; but if there are 2 values that could trigger this and one that would prevent it, I think it would be quickly triggered on 2 out of 3 hosts. I imagine there's some other requirement for the ethernet frame, and that might limit exposure to being link-local.
Anyway I'm concerned anything like this was even possible. Why is software hidden in firmware blobs inspecting payload data? Remotely-triggerable bugs of this sort could allow DoS of whole data centres, customer premises equipment (modem/routers), or ISPs' core infrastructure. |
Steven C. 171 Posts |
Quote |
Feb 8th 2013 9 years ago |
Re: 'innoculation', Kris K's blog later mentions that most values other than 0x30,31,32 will do that. The chance of hitting the error becomes more like 1 in 128 and probably depends what type of traffic is sent/received during startup, or incoming broadcasts on your network.
|
Steven C. 171 Posts |
Quote |
Feb 8th 2013 9 years ago |
I guess I'm a bit more cynical than others, looking at a weak link that was produced back in 2012 and Intel's pass the buck response. Personally I've seen bad code in their tethered and non-tethered products through HP, Dell.
Intel "worked with the author as well as the original motherboard manufacturer to investigate and determine root cause"? Intel caused the issue, not the MB manufactures with a flawed EEPROM image that could have been fixed. We had 5 servers that ran dual NICS with this chipset.. Bravo! A lot of us have been around this industry for years, Sadly what is becoming a trend as new products come out, there is some security flaw to them. If we did work like this, we would not be working. |
scsmith59 6 Posts |
Quote |
Feb 10th 2013 9 years ago |
Sign Up for Free or Log In to start participating in the conversation!