[Voyage-linux] ICMP dest unreachable broadcast storm on wlan0
Andrew Niemantsverdriet
(spam-protected)
Fri Sep 29 03:20:32 HKT 2006
Does this client happen to have an older linksys or dlink SOHO router? We
have seen this (although not with voyage) with people that have older
routers. It is a known problem and a new router or firmware upgrade has
fixed it. It has been a while since I have had to deal with this problem
so the details are a bit fuzzy but basiclly the router just starts
freaking out for some reason and then it is a chain reaction where all the
other routers freak out as well. Not sure if this is your exact problem
but it is something we had to deal with once.
> The "boot everyone off and add back one at a time" approach seems to
> have gotten things working again though it has done so without
> specifically identifying the culprit, i.e. *all* customers' radios have
> now re-associated and the storm has not reappeared.
>
> My procedure was as follows:
>
> iwlist wlan0 scan|grep Address|tr -s ' ' ' '|cut -d' ' -f6
>
> returns the list of associated clients so the command:
>
> for mac in `iwlist wlan0 scan|grep Address|tr -s ' ' ' '|cut -d' ' -f6`;
> do iwpriv wlan0 kickmac $mac;
> done
>
> kicks everyone off.
>
> Using the maccmd tools: (http://madwifi.org/wiki/UserDocs/iwpriv)
>
> iwpriv wlan0 maccmd 3
> iwpriv wlan0 maccmd 1 (whitelist)
>
> run the "for" loop above and Voila!, no associated clients (and no
> broadcast storm either)
>
> iwpriv wlan0 addmac <mac address> (cut and paste each MAC in turn from
> the scan command above run in a different window)
>
> all the while runnng tpcdump -i wlan0 proto ICMP in yet another window
> and see which client added back in causes the storm to reappear.
>
> If/when it reappears, run
>
> iwpriv wlan0 delmac <mac address you just added>
>
> follwed by the kickmac for loop to see if the problem goes away (it
> should).
>
> Add all the MAC addresses until you are back up.
>
> The real puzzle remains however in that while kicking all the clients
> off and adding them back one at a time allowed me to finally isolate the
> source of the traffic, I did actually add the offending client a few
> minutes later and the problem did *not* reappear. Why? This particular
> client actually has another AP behind it with about 20 clients so it was
> important to get the connection there back up.
>
> I'd like to find out why, of course, but in any case, I've learned a
> method to get things back on line and hope it might help someone else as
> well.
>
> Edwin
>
> Edwin Whitelaw wrote:
>> The DoS ICMP storm has now been going for 4+ hrs so I've had the
>> opportunity to collect data and try different things.
>>
>> 1) Tried several versions of the Prism firmware, including 1.7.4 and
>> 1.8.4. I have been running 1.8.2 for almost a year w/o problems.
>> Iptraf shows the incoming rate at ~150kbs and it was interesting that
>> 1.8.4 slowed that to ~10kbs but functionally still did not allow
>> association from my clients. Broadcast storm aside, I have not found
>> 1.8.4 to work, hence the 1.8.2 currently installed.
>>
>> 2) iwlist wlan0 scan shows everyone there and with good signal.
>>
>> 3) tcpdump -i wlan0 proto ICMP shows around 650 packets/second
>>
>> 4) Here are some snippets from iptraf, then tcpdump, and finally
>> iptraf in LAN Station monitor mode
>>
>> ICMP dest unrch (proto) (56 bytes) from 0.0.0.0 to 172.16.8.1 on wlan0 ?
>> ? ICMP dest unrch (proto) (56 bytes) from 0.0.0.0 to 172.16.8.1 on
>> wlan0 ?
>> ? ICMP dest unrch (proto) (56 bytes) from 0.0.0.0 to 172.16.8.1 on
>> wlan0 ?
>> ? ICMP dest unrch (proto) (56 bytes) from 0.0.0.0 to 172.16.8.1 on
>> wlan0 ?
>> ? ICMP dest unrch (proto) (56 bytes) from 0.0.0.0 to 172.16.8.1 on
>> wlan0 ?
>> ? ICMP dest unrch (proto) (56 bytes) from 0.0.0.0 to 172.16.8.1 on
>> wlan0 ?
>> ? ICMP dest unrch (proto) (56 bytes) from 0.0.0.0 to 172.16.8.1 on
>> wlan0 ?
>> ? ICMP dest unrch (proto) (56 bytes) from 0.0.0.0 to 172.16.8.1 on
>> wlan0 ?
>> ? ICMP dest unrch (proto) (56 bytes) from 0.0.0.0 to 172.16.8.1 on wlan0
>>
>> tcpdump -i wlan0 proto ICMP
>>
>> 12:10:32.286597 IP 0.0.0.0 > 172.16.8.1: ICMP OSPF-ALL.MCAST.NET
>> protocol 89 unreachable, length 36
>> 12:10:32.286679 IP 0.0.0.0 > 172.16.8.1: ICMP OSPF-ALL.MCAST.NET
>> protocol 89 unreachable, length 36
>> 12:10:32.287238 IP 0.0.0.0 > 172.16.8.1: ICMP OSPF-ALL.MCAST.NET
>> protocol 89 unreachable, length 36
>> 12:10:32.287317 IP 0.0.0.0 > 172.16.8.1: ICMP OSPF-ALL.MCAST.NET
>> protocol 89 unreachable, length 36
>> 12:10:32.288810 IP 0.0.0.0 > 172.16.8.1: ICMP OSPF-ALL.MCAST.NET
>> protocol 89 unreachable, length 36
>> 12:10:32.288886 IP 0.0.0.0 > 172.16.8.1: ICMP OSPF-ALL.MCAST.NET
>> protocol 89 unreachable, length 36
>> 12:10:32.293616 IP 0.0.0.0 > 172.16.8.1: ICMP OSPF-ALL.MCAST.NET
>> protocol 89 unreachable, length 36
>> 12:10:32.293711 IP 0.0.0.0 > 172.16.8.1: ICMP OSPF-ALL.MCAST.NET
>> protocol 89 unreachable, length 36
>> 12:10:32.294479 IP 0.0.0.0 > 172.16.8.1: ICMP OSPF-ALL.MCAST.NET
>> protocol 89 unreachable, length 36
>> 12:10:32.294613 IP 0.0.0.0 > 172.16.8.1: ICMP OSPF-ALL.MCAST.NET
>> protocol 89 unreachable, length 36
>> 12:10:32.296556 IP 0.0.0.0 > 172.16.8.1: ICMP OSPF-ALL.MCAST.NET
>> protocol 89 unreachable, length 36
>>
>> Note: I have both shutdown quagga/ospfd and also disabled it and
>> rebooted without affecting the broadcast storm so do not believe it is
>> my own ospf process at fault.
>>
>> tcpdump with -v (verbose)
>>
>> 12:40:15.750062 IP (tos 0x0, ttl 122, id 49021, offset 0, flags
>> [none], proto: ICMP (1), length: 56) 0.0.0.0 > 172.16.8.1: ICMP
>> OSPF-ALL.MCAST.NET protocol 89 unreachable, length 36
>> IP (tos 0xc0, ttl 1, id 49021, offset 0, flags [none], proto: OSPF
>> (89), length: 64) 172.16.8.1 > OSPF-ALL.MCAST.NET: [|ospf]
>> 12:40:15.751592 IP (tos 0x0, ttl 123, id 49021, offset 0, flags
>> [none], proto: ICMP (1), length: 56) 0.0.0.0 > 172.16.8.1: ICMP
>> OSPF-ALL.MCAST.NET protocol 89 unreachable, length 36
>> IP (tos 0xc0, ttl 1, id 49021, offset 0, flags [none], proto: OSPF
>> (89), length: 64) 172.16.8.1 > OSPF-ALL.MCAST.NET: [|ospf]
>>
>> iptraf LAN station monitor (sorry the formatting got scrogged. This
>> is sorted by byte count. The important part is the all 1s MAC address
>> of all f's and the next one is the MAC of the wlan0 radio itself)
>>
>> IPTraf
>> ??????? PktsIn ???????? IP In ????? BytesIn ????? InRate ???? PktsOut
>> ??????? IP Out ???? BytesOut ??? OutRate ??????
>> ? Ethernet HW addr: ffffffffffff on wlan0 ?
>> ? 113097 0 6339844 147.6 0 0 0 0.0 ?
>> ? Ethernet HW addr: 00026f42bca9 on wlan0 ?
>> ? ? 9771 0 583293 8.4 1088 0 220688 0.4 ?
>> ? Ethernet HW addr: 00026f3ab2f3 on wlan0 ?
>> ? ? 102 0 57119 0.0 55 0 2641 0.0 ?
>> ? Ethernet HW addr: 00026f3aaf4c on wlan0 ?
>> ? ? 148 0 48995 0.0 153 0 19278 0.0 ?
>> ? Ethernet HW addr: 00026f40e125 on wlan0 ?
>> ? ? 127 0 24565 0.0 1818 0 113522 1.4 ?
>>
>> 5) Neither
>>
>> ebtables -A INPUT -i wlan0 -s FF:FF:FF:FF:FF:FF -j DROP
>>
>> or
>>
>> iptables -A INPUT -p ICMP -s 0/0 -d 172.16.8.1 -j DROP
>>
>> have any effect.
>>
>> 6) Tcpdump on the 5GHz uplink interface shows no problems.
>>
>> 7) I have disconnected the antenna cable from wlan0 to the omni and
>> all ICMP traffic ceases so it would appear to be externally
>> generated. Reconnecting immediately displays the problem.
>> As you read this, I will be using iwpriv maccmd tools to knock off
>> each radio in an attempt to identify/isolate the source. I suppose
>> it's also possible it could be coming from a non-customer in the area?
>>
>> Can anyone think of what or how a wireless client would be generating
>> the traffic shown?
>>
>> Thanks,
>>
>> Edwin
>>
>>
>> Punky Tse wrote:
>>> Hi Edwin,
>>>
>>> - did you box expose to Internet that everyone can reach your box?
>> the 5GHz PtP link does have a real IP but the wlan0 side (where the
>> problem exists is all private. Sniffing the 5GHz side shows no
>> problems.
>>> - did you check /var/log/ to see if any strange things happens?
>> I have my logging redirected to a normal PC. Tailing that
>> /var/log/messages shows only normal DHCP traffic.
>> /var/log/daemon is pretty full with
>> Sep 24 20:03:01 rinerwrap zebra[2189]: netlink_parse_info:
>> netlink-listen type RTM_NEWLINK(16), seq=0, pid=0
>> Sep 24 20:03:01 rinerwrap zebra[2189]: netlink_link_change: ignoring
>> IFLA_WIRELESS message
>>
>> entries but they pre-date the storm and look no different now. There
>> are also some fairly normal looking OSPF traffic from the other parts
>> of my net but note that the storm exists even when all quagga/ospf
>> processes on this box are shutdown.
>>
>>> - did you run vmstat to see the CPU consumption? If you can tell
>>> whether it is consuming CPU on system or userland program it could
>>> help.
>> riner:~# vmstat 2
>> procs -----------memory---------- ---swap-- -----io---- -system--
>> ----cpu----
>> r b swpd free buff cache si so bi bo in cs us sy
>> id wa
>> 0 0 0 86836 2184 24036 0 0 5 0 1072 49 3 17
>> 80 0
>> 0 0 0 86836 2184 24036 0 0 0 0 1199 24 0 17
>> 83 0
>> 0 0 0 86836 2184 24036 0 0 0 0 1184 14 1 17
>> 83 0
>> 0 0 0 86800 2184 24036 0 0 0 0 1196 19 0 19
>> 81 0
>> 0 0 0 86800 2184 24036 0 0 0 0 1124 13 1 14
>> 85 0
>> 0 0 0 86800 2184 24036 0 0 0 0 1086 13 0 16
>> 84 0
>> 0 0 0 86800 2184 24036 0 0 0 0 1103 21 0 16
>> 83 0
>>
>> top similarly shows no runaway process
>>
>> riner:/var/log# w
>> 12:55:47 up 1:12, 2 users, load average: 0.00, 0.00, 0.00
>>
>>
>>>
>>>
>>> Punky
>>>
>>> Edwin Whitelaw wrote:
>>>> Running Voyage 0.2 fully updated on WRAP 2C. Two radios, one 5GHz
>>>> (SR5)for backhaul and an NL2611 for the AP. Firmware on the AP
>>>> radio is
>>>>
>>>> wifi0: NIC: id=0x8013 v1.0.0
>>>> wifi0: PRI: id=0x15 v1.1.1
>>>> wifi0: STA: id=0x1f v1.8.2
>>>> wifi0: Intersil Prism2.5 PCI: mem=0xa0000000, irq=9
>>>> wifi0: registered netdevice wlan0
>>>>
>>>> I'm only recently getting occasional (every few days) ICMP dest
>>>> unreachable broadcast storms that are effectively DoS attacks on the
>>>> system though at this point I'm not sure whether it's a
>>>> rogue/defective hardware issue, misbehaving software or a deliberate
>>>> attack from an infected customer's site. Unfortunately, it has been
>>>> difficult to determine the origin since the source IP address is
>>>> 0.0.0.0 and the source MAC shows as all "f"s. Iptables entries to
>>>> block all ICMP from 0.0.0.0 incoming on wlan0 has no effect.
>>>>
>>>> The storms last from just a few minutes to 10s of minutes though if
>>>> I am not actually at the console when they occur it is difficult to
>>>> get an exact read on the duration.
>>>>
>>>> The clients on this AP are a mix of Engenius CB3s, Tranzeo CPEs
>>>> (basically the same radio) and a few smartbridges.
>>>>
>>>> iptraf shows the storms as ICMP dest unreachble and tcpdump shows
>>>> ICMP and OSPF as the protocol. We do run OSPF but I have shut down
>>>> quagga during one of these storms with no effect and would expect it
>>>> to stop if OSPF were the cause.
>>>>
>>>> Anyone else experiencing this problem or have a suggestion on how to
>>>> protect against it? I will try and capture some tcpdump output the
>>>> next time and regret not having it at this point though to my eyes,
>>>> it doesn't offer much information beyond this verbal description.
>>>>
>>>> Edwin
>>>>
>>>>
>>
>
> --
> <=+=+=+==+=+=+==+=+=+=+=+=+=+=+=>
> Edwin Whitelaw, P.E.
> New River Valley Unwired, LLC
> 2200 Lonesome Dove Dr
> Christiansburg, VA 24073
> 540-239-0318
>
>
> _______________________________________________
> Voyage-linux mailing list
> Voyage-linux at list.voyage.hk
> http://list.voyage.hk/mailman/listinfo/voyage-linux
>
More information about the Voyage-linux
mailing list