Slow Internet link with wrong MTU size in the ISP backbone

So slow Internet connection after scheduled maintenance on upstream provider(ISP). It could be multiple reasons such as software issue, hardware failure, bad patch cable , dirty fiber and trunk issue. Those are very rare case. Lately I noticed more trouble with MTU mis-setting on backbone routers. When ISP expends their backbone trunk, they put TenGigE card instead of SONET in these days. No wonder why they do that b/c GigE or TenGigE card is much cheaper then SONET card.

Here is common mistake is that the forget to change default MTU size(1500) of the fresh card from the box. Most of ISP set MTU on their backbone as upto 9180 or at least bigger than 1524 bytes (due to VLAN tag for internal use). Again, this is very rare case, but it happens. See how to verify MTU size wrong in the ISP backbone.

 

 

 

 

[ Backgroud ]

IP address

PC : 100.100.100.1
Server : 200.200.200.1

CE1 : E0 : 100.100.100.2 / Serial1 : 100.20.1.1
ISP PE1 : E0 :  100.30.1, Serial1 : 100.20.1.2
ISP Backbone Router : E0 : 100.30.1.2, E1 : 200.30.1.2
ISP PE2 : E1 : 200.30.1.1, Serial1 : 200.20.1.2
CE2 : E0 : 200.200.200.2, Serial1 : 200.20.1.1

[ Symptom ]

Browsing and downloading files are slow slow from PC to Server. Ping testing is fine.

[ Achievement ]

Send ICMP packet with 1500 bytes + DF bit to see which device is response with "Fragmentation needed" message. Figure the hop using Wireshark or TCPDump.

 

[ Step 1 ]

Check your ACL or firewall if security policy is blocking ICMP type3, 4. Just for testing remove the ACL or open for full ICMP messages types.

[ Step 2 ]

Send ping(ICMP packet) with 1500 bytes + DF bit.

From PC

c:\> ping 200.200.200.1 -l 1401 -f

Packet needs to be fragmented but DF set
Packet needs to be fragmented but DF set
Packet needs to be fragmented but DF set
Packet needs to be fragmented but DF set
Packet needs to be fragmented but DF set

From Cisco router

CE1#ping 200.200.200.1 size 1401 df-bit

Type escape sequence to abort.
Sending 5, 1500-byte ICMP Echos to 200.200.200.1, timeout is 2 seconds:
Packet sent with the DF bit set
M.M.M
Success rate is 0 percent (0/5)
CE1#

CE1#ping 200.200.200.1 size 1400 df-bit

Type escape sequence to abort.
Sending 5, 1400-byte ICMP Echos to 200.200.200.1, timeout is 2 seconds:
Packet sent with the DF bit set
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 12/27/48 ms
CE1#

As you can see above, packet size smaller than 1400 bytes is okay, but bigger than the packet was failed.

 

[ Step 3 Capturing packet ]

You can us whatever you are familar with. In this case Wireshark is used. As you can see below, PC is receiving ICMP message type 3 "Fragmentation needed" from IP 100.30.1.2 that is IP of "Backbone Router", Ethernet interface. 
 

Now, we know which router might have improper MTU size on it.

For the testing, I set IP mtu 1400 on the "Backbone Router"

Backbone_Router#sh run int fa0/1
Building configuration…

Current configuration : 110 bytes
!
interface FastEthernet0/1
 ip address 200.30.1.2 255.255.255.252
 ip mtu 1400
 duplex auto
 speed auto
end

Backbone_Router#

** Do not forget allow ICMP messages from your ACL. ICMP message will be dropped before captured by packet analyzer.

[ Step 4 ]

Save all the evidence and call ISP tech support.