[Voyage-linux] ICMP dest unreachable broadcast storm on wlan0
Edwin Whitelaw
(spam-protected)
Fri Sep 29 01:01:06 HKT 2006
The DoS ICMP storm has now been going for 4+ hrs so I've had the
opportunity to collect data and try different things.
1) Tried several versions of the Prism firmware, including 1.7.4 and
1.8.4. I have been running 1.8.2 for almost a year w/o problems.
Iptraf shows the incoming rate at ~150kbs and it was interesting that
1.8.4 slowed that to ~10kbs but functionally still did not allow
association from my clients. Broadcast storm aside, I have not found
1.8.4 to work, hence the 1.8.2 currently installed.
2) iwlist wlan0 scan shows everyone there and with good signal.
3) tcpdump -i wlan0 proto ICMP shows around 650 packets/second
4) Here are some snippets from iptraf, then tcpdump, and finally iptraf
in LAN Station monitor mode
ICMP dest unrch (proto) (56 bytes) from 0.0.0.0 to 172.16.8.1 on wlan0 ?
? ICMP dest unrch (proto) (56 bytes) from 0.0.0.0 to 172.16.8.1 on wlan0 ?
? ICMP dest unrch (proto) (56 bytes) from 0.0.0.0 to 172.16.8.1 on wlan0 ?
? ICMP dest unrch (proto) (56 bytes) from 0.0.0.0 to 172.16.8.1 on wlan0 ?
? ICMP dest unrch (proto) (56 bytes) from 0.0.0.0 to 172.16.8.1 on wlan0 ?
? ICMP dest unrch (proto) (56 bytes) from 0.0.0.0 to 172.16.8.1 on wlan0 ?
? ICMP dest unrch (proto) (56 bytes) from 0.0.0.0 to 172.16.8.1 on wlan0 ?
? ICMP dest unrch (proto) (56 bytes) from 0.0.0.0 to 172.16.8.1 on wlan0 ?
? ICMP dest unrch (proto) (56 bytes) from 0.0.0.0 to 172.16.8.1 on wlan0
tcpdump -i wlan0 proto ICMP
12:10:32.286597 IP 0.0.0.0 > 172.16.8.1: ICMP OSPF-ALL.MCAST.NET
protocol 89 unreachable, length 36
12:10:32.286679 IP 0.0.0.0 > 172.16.8.1: ICMP OSPF-ALL.MCAST.NET
protocol 89 unreachable, length 36
12:10:32.287238 IP 0.0.0.0 > 172.16.8.1: ICMP OSPF-ALL.MCAST.NET
protocol 89 unreachable, length 36
12:10:32.287317 IP 0.0.0.0 > 172.16.8.1: ICMP OSPF-ALL.MCAST.NET
protocol 89 unreachable, length 36
12:10:32.288810 IP 0.0.0.0 > 172.16.8.1: ICMP OSPF-ALL.MCAST.NET
protocol 89 unreachable, length 36
12:10:32.288886 IP 0.0.0.0 > 172.16.8.1: ICMP OSPF-ALL.MCAST.NET
protocol 89 unreachable, length 36
12:10:32.293616 IP 0.0.0.0 > 172.16.8.1: ICMP OSPF-ALL.MCAST.NET
protocol 89 unreachable, length 36
12:10:32.293711 IP 0.0.0.0 > 172.16.8.1: ICMP OSPF-ALL.MCAST.NET
protocol 89 unreachable, length 36
12:10:32.294479 IP 0.0.0.0 > 172.16.8.1: ICMP OSPF-ALL.MCAST.NET
protocol 89 unreachable, length 36
12:10:32.294613 IP 0.0.0.0 > 172.16.8.1: ICMP OSPF-ALL.MCAST.NET
protocol 89 unreachable, length 36
12:10:32.296556 IP 0.0.0.0 > 172.16.8.1: ICMP OSPF-ALL.MCAST.NET
protocol 89 unreachable, length 36
Note: I have both shutdown quagga/ospfd and also disabled it and
rebooted without affecting the broadcast storm so do not believe it is
my own ospf process at fault.
tcpdump with -v (verbose)
12:40:15.750062 IP (tos 0x0, ttl 122, id 49021, offset 0, flags [none],
proto: ICMP (1), length: 56) 0.0.0.0 > 172.16.8.1: ICMP
OSPF-ALL.MCAST.NET protocol 89 unreachable, length 36
IP (tos 0xc0, ttl 1, id 49021, offset 0, flags [none], proto: OSPF (89),
length: 64) 172.16.8.1 > OSPF-ALL.MCAST.NET: [|ospf]
12:40:15.751592 IP (tos 0x0, ttl 123, id 49021, offset 0, flags [none],
proto: ICMP (1), length: 56) 0.0.0.0 > 172.16.8.1: ICMP
OSPF-ALL.MCAST.NET protocol 89 unreachable, length 36
IP (tos 0xc0, ttl 1, id 49021, offset 0, flags [none], proto: OSPF (89),
length: 64) 172.16.8.1 > OSPF-ALL.MCAST.NET: [|ospf]
iptraf LAN station monitor (sorry the formatting got scrogged. This is
sorted by byte count. The important part is the all 1s MAC address of
all f's and the next one is the MAC of the wlan0 radio itself)
IPTraf
??????? PktsIn ???????? IP In ????? BytesIn ????? InRate ???? PktsOut
??????? IP Out ???? BytesOut ??? OutRate ??????
? Ethernet HW addr: ffffffffffff on wlan0 ?
? 113097 0 6339844 147.6 0 0 0 0.0 ?
? Ethernet HW addr: 00026f42bca9 on wlan0 ?
? ? 9771 0 583293 8.4 1088 0 220688 0.4 ?
? Ethernet HW addr: 00026f3ab2f3 on wlan0 ?
? ? 102 0 57119 0.0 55 0 2641 0.0 ?
? Ethernet HW addr: 00026f3aaf4c on wlan0 ?
? ? 148 0 48995 0.0 153 0 19278 0.0 ?
? Ethernet HW addr: 00026f40e125 on wlan0 ?
? ? 127 0 24565 0.0 1818 0 113522 1.4 ?
5) Neither
ebtables -A INPUT -i wlan0 -s FF:FF:FF:FF:FF:FF -j DROP
or
iptables -A INPUT -p ICMP -s 0/0 -d 172.16.8.1 -j DROP
have any effect.
6) Tcpdump on the 5GHz uplink interface shows no problems.
7) I have disconnected the antenna cable from wlan0 to the omni and all
ICMP traffic ceases so it would appear to be externally generated.
Reconnecting immediately displays the problem.
As you read this, I will be using iwpriv maccmd tools to knock off each
radio in an attempt to identify/isolate the source. I suppose it's also
possible it could be coming from a non-customer in the area?
Can anyone think of what or how a wireless client would be generating
the traffic shown?
Thanks,
Edwin
Punky Tse wrote:
> Hi Edwin,
>
> - did you box expose to Internet that everyone can reach your box?
the 5GHz PtP link does have a real IP but the wlan0 side (where the
problem exists is all private. Sniffing the 5GHz side shows no problems.
> - did you check /var/log/ to see if any strange things happens?
I have my logging redirected to a normal PC. Tailing that
/var/log/messages shows only normal DHCP traffic.
/var/log/daemon is pretty full with
Sep 24 20:03:01 rinerwrap zebra[2189]: netlink_parse_info:
netlink-listen type RTM_NEWLINK(16), seq=0, pid=0
Sep 24 20:03:01 rinerwrap zebra[2189]: netlink_link_change: ignoring
IFLA_WIRELESS message
entries but they pre-date the storm and look no different now. There
are also some fairly normal looking OSPF traffic from the other parts of
my net but note that the storm exists even when all quagga/ospf
processes on this box are shutdown.
> - did you run vmstat to see the CPU consumption? If you can tell
> whether it is consuming CPU on system or userland program it could help.
riner:~# vmstat 2
procs -----------memory---------- ---swap-- -----io---- -system--
----cpu----
r b swpd free buff cache si so bi bo in cs us sy
id wa
0 0 0 86836 2184 24036 0 0 5 0 1072 49 3 17
80 0
0 0 0 86836 2184 24036 0 0 0 0 1199 24 0 17
83 0
0 0 0 86836 2184 24036 0 0 0 0 1184 14 1 17
83 0
0 0 0 86800 2184 24036 0 0 0 0 1196 19 0 19
81 0
0 0 0 86800 2184 24036 0 0 0 0 1124 13 1 14
85 0
0 0 0 86800 2184 24036 0 0 0 0 1086 13 0 16
84 0
0 0 0 86800 2184 24036 0 0 0 0 1103 21 0 16
83 0
top similarly shows no runaway process
riner:/var/log# w
12:55:47 up 1:12, 2 users, load average: 0.00, 0.00, 0.00
>
>
> Punky
>
> Edwin Whitelaw wrote:
>> Running Voyage 0.2 fully updated on WRAP 2C. Two radios, one 5GHz
>> (SR5)for backhaul and an NL2611 for the AP. Firmware on the AP radio is
>>
>> wifi0: NIC: id=0x8013 v1.0.0
>> wifi0: PRI: id=0x15 v1.1.1
>> wifi0: STA: id=0x1f v1.8.2
>> wifi0: Intersil Prism2.5 PCI: mem=0xa0000000, irq=9
>> wifi0: registered netdevice wlan0
>>
>> I'm only recently getting occasional (every few days) ICMP dest
>> unreachable broadcast storms that are effectively DoS attacks on the
>> system though at this point I'm not sure whether it's a
>> rogue/defective hardware issue, misbehaving software or a deliberate
>> attack from an infected customer's site. Unfortunately, it has been
>> difficult to determine the origin since the source IP address is
>> 0.0.0.0 and the source MAC shows as all "f"s. Iptables entries to
>> block all ICMP from 0.0.0.0 incoming on wlan0 has no effect.
>>
>> The storms last from just a few minutes to 10s of minutes though if I
>> am not actually at the console when they occur it is difficult to get
>> an exact read on the duration.
>>
>> The clients on this AP are a mix of Engenius CB3s, Tranzeo CPEs
>> (basically the same radio) and a few smartbridges.
>>
>> iptraf shows the storms as ICMP dest unreachble and tcpdump shows
>> ICMP and OSPF as the protocol. We do run OSPF but I have shut down
>> quagga during one of these storms with no effect and would expect it
>> to stop if OSPF were the cause.
>>
>> Anyone else experiencing this problem or have a suggestion on how to
>> protect against it? I will try and capture some tcpdump output the
>> next time and regret not having it at this point though to my eyes,
>> it doesn't offer much information beyond this verbal description.
>>
>> Edwin
>>
>>
--
<=+=+=+==+=+=+==+=+=+=+=+=+=+=+=>
Edwin Whitelaw, P.E.
New River Valley Unwired, LLC
2200 Lonesome Dove Dr
Christiansburg, VA 24073
540-239-0318
More information about the Voyage-linux
mailing list