[Cisco] High Process CPU time on Cisco router

   July 18, 2009  Leave a comment


When you see a latency or slow response to your destination, check your router CPU time. If it’s high, it means something abnormal or something is happening. Well, it could be also one of attacks, route leak or etc. From the output below, see each PID in order to find which process caused the high CPU time.


Router#sh processes cpu
CPU utilization for five seconds: 65%/79%; one minute: 82%; five minutes: 91%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
1 0 1 0 0.00% 0.00% 0.00% 0 Chunk Manager
2 584 1758822 0 0.00% 0.00% 0.00% 0 Load Meter
3 0 1 0 0.00% 0.00% 0.00% 0 OSPF Hello
4 3355180 893075 3756 0.00% 0.03% 0.00% 0 Check heaps
5 68 1845 36 0.00% 0.00% 0.00% 0 Pool Manager
6 0 2 0 0.00% 0.00% 0.00% 0 Timers
7 0 2 0 0.00% 0.00% 0.00% 0 ATM Idle Timer
8 0 2 0 0.00% 0.00% 0.00% 0 Serial Backgroun
9 136 293117 0 0.00% 0.00% 0.00% 0 AAA high-capacit
:
:
:

1. Clear counters so that it will be easy to determine the line card that has the process level switching.

Router# clear counters
Clear "show interface" counters on all interfaces [confirm]
Router#


2. Check interface status

Router#sh int stat
Serial1/0
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 128 10168 128 10184
Route cache 0 0 0 0
Distributed cef 2 120 3 312
Total 130 10288 131 10496
Interface Serial1/1 is disabled

Serial1/2
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 41223 324498 2545 84498
Route cache 0 0 0 0
Distributed cef 0 0 0 0
Total 41223 324498 2545 84498
Interface Serial1/3 is disabled

Interface Serial1/4 is disabled

:
:
:

PPKts In = 41223 and Chars In = 324498 are high numbers

3. Go to attached slot card and check any incomplete adjacencies for the interfaces for the slot card that crashed. The below is sample of output

Router#execute-on slot 1 show adj det
========= Line Card (Slot 1) =======

Protocol Interface Address
IP POS7/0 point2point(23)
105 packets, 7319 bytes
FF030021
FIB LC 00:00:00
IP Serial1/8.500 point2point(20)
16936 packets, 1131316 bytes
7C4103CC
FIB LC 00:00:00
IP Serial1/10 point2point(19)
123388 packets, 9156159 bytes
0F000800
FIB LC 00:00:00
IP ATM5/0.102 point2point(47)
412342 packets, 30029441 bytes
00660100AAAA030000000800
FIB LC 00:00:00
IP Serial1/2 point2point(22)
324394 packets, 22287249 bytes
0F000800
FIB LC 00:00:00
IP Serial1/0 point2point(1771)
1707806 packets, 238755375 bytes
0F000800
FIB LC 00:00:00
IP Serial 1/3 point2point(18989) (incomplete)
0 packets, 0 bytes
IP Serial 1/5 point2point(4) (incomplete)
0 packets, 0 bytes
IP Serial 1/7 point2point(276) (incomplete)
0 packets, 0 bytes

4. Execute a microcode reload on the line card in slot 1.

Router(config) #microcode reload 1


5. After microcode reloaded, check CPU time again.

Router#sh processes cpu
CPU utilization for five seconds: 6%/7%; one minute: 8%; five minutes: 9%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
1 0 1 0 0.00% 0.00% 0.00% 0 Chunk Manager
2 584 1758822 0 0.00% 0.00% 0.00% 0 Load Meter
3 0 1 0 0.00% 0.00% 0.00% 0 OSPF Hello
4 3355180 893075 3756 0.00% 0.03% 0.00% 0 Check heaps
5 68 1845 36 0.00% 0.00% 0.00% 0 Pool Manager
6 0 2 0 0.00% 0.00% 0.00% 0 Timers
7 0 2 0 0.00% 0.00% 0.00% 0 ATM Idle Timer
8 0 2 0 0.00% 0.00% 0.00% 0 Serial Backgroun
9 136 293117 0 0.00% 0.00% 0.00% 0 AAA high-capacit

 


Leave a Reply