Troubleshooting example using Splunk – Syslog

Tweet

Splunk was configured as a Syslog server on my lab network for few weeks now. Since I modified few parameters on service side and configured network devices, firewall, switches and router, which sent syslog message to the Splunk server. Past few days it built pretty thick index database. Well my index size is way below than 500Mbyte daily limitation for free version of Splunk. I am not expecting huge errors and critical messages because it is small Lab and also I am only monitoring network devices.

Today, while I was playing and reviewing Splunk, I found some of device within network sending spoofing traffic which is sending packet to unknown network, 192.168.100.15(RFC1918). As far as I know I didn’t set it up within the lab network.

Let’s find out what’s going on.

 

 

From main page, search ‘192.168.100.15’. Many packets destined to 192.168.100.15 are keeping forwarded to 192.168.77.1 which is my firewall. And it dropped those. The packets were origin from 192.168.77.9 that is my wireless access point at lab.

 

Here is my lab network layout. As I mentioned, those ghost packets are form coming from access point (192.168.77.9). There are three candidates who might generate the packets. I got lucky. First one I checked is the one.

 

 

From DOS prompt, type ‘netstat -a’ to see TCP session. That’s it. Notebook/ibm is sending syn_sent to destination IP, 192.168.100.15:9100. Okay now we know which device is generating the useless traffic. How we can stop it? 

 

In order to figure out what processor is involved with the behavior, type ‘netstat -b’ command.
PID stand for Processor Identification will be shown up. That is 324.

 

Now, open ‘Window Task Manager’ and find a PID number that is associated with program or service. In this case, printer spooler service is the one. You can kill the processor, but I had to check one more think to kill the processor in permanent. 

 

Looked Printer folder and figured which printer kept sending data to the network. "Dell Laser Printer 3000cn" is the one and I remembered I used the notebook for church event last year. I setup network printer which has 192.168.100.15. Wow, one of printing job is hanging on the printer queue since then. Finally it is removed.

 

Nice Splunk! The traffic was not critically affected to my network, but it was good to be removed.