Identifying network latency / jitter issues w/ Wireshark

Issue : sporadically and randomly clients would see jitter (picture freezing for a small number of seconds) when viewing live video streams such as BBC news

To start tackling the above issue it is first important to observe the normal behaviour of the video stream. In BBC’s case, the video stream is using RTMP (port 1935), however it is just as common to see video streams using port 80, 443 or other ports. Make sure to note which IPs and ports are generating the video traffic. With this information in hand, we can proceed to perform a packet capture (ideally on both the end client and the gateway).

As a starting point, I usually get a general “feel” for the network traffic by using wireshark’s in-built I/O graphs (Input / Output graphs). This graph is indispensable when dealing with suspected network bandwidth issues. To open the IO graphs, go to the menu “Statistics > IO graphs”

jitter1

This will produce a graph showing (by default) the number of packets per second. I almost always change this to show the number of bytes per second. To do this, simply change the “Unit” option for the “Y Axis” at the bottom right corner to “Bytes/Tick”. If you would like to “zoom in” or “stretch” the graph, change the “Pixels per tick” in the X Axis options to a higher number.

Notice also that you can apply normal wireshark display filters at the bottom, causing the graph to change and show throughput for only the traffic of interest. You can also have multiple filters to compare different traffics’ throughput.

In the case of the network with video jitter issue, I filtered on port 1935 and the relevant IP, which gave us this result n the IO graph:

jitter2

This is not very good performance wise. Basically the intervals between the spikes represent intervals of time when the client is not communicating at all with the video server. This of course the jitter. Compare the above graph with the one below taken from a good performance video stream:

jitter3

Note how the spikes are not very pronounced and the graph only ever drops to 0 bytes when I actually turn off the stream towards the end of the capture.

Most often such a pattern is again either due to processing issues or network issues. I think the major issue is evident from the packet capture itself. Grabbing the initial I/O graph and zooming in to the first two spikes we see:

jitter4

So basically the first “dip” is at approximately 28 seconds into the packet capture. Turning back to the actual capture and scrolling down to this time we see:

jitter5

Note how at 26 seconds, the proxy sends an ACK to the server, but the server only responds back with some data at 29seconds, a whole 3 seconds delay. This is what’s causing your jitters. The next step would be to see where along the network path between the node where the packet capture was taken, and the server is this delay introduced? Either take more packet captures (for example at the gateway) to make sure all packets are forwarded in a timely fashion, or check the CPU and interface utilization at all intermediate nodes to make sure no delay is introduced

The packet captures usually help in deducing what is not causing the delay. In the above example for instance, we know the problem is somewhere between client and server. Checking a pcap on the gateway, we observed that the client’s ACK was forwarded instantly, but the delay from the server persisted. So the delay is between the gateway and the video server… that is, the ISP.