TechComm Labs

The popular RV forums RV.net – Open Roads Forum are sometimes very very slow in Linux systems. It looks like RFC 1323 is the primary resource and that some routers don’t follow the standard – even after many years. On windows up through XP, the TCP window scaling is not enabled. Later version use auto-tuning like modern versions of the Linux kernel. Ohio State has help file for setting TCP window scaling in Vista. A general purpose Linux Tweaking page is at speedguide.net or scribd.

A way to fix this problem (see buglist) is to change the parameters for the network receive and send buffers (see John’s Tidbits). On my Ubuntu 10.10, the pertinent collection defaults are:


$ cat /proc/sys/net/ipv4/tcp_wmem
4096 16384 4194304
$ cat /proc/sys/net/ipv4/tcp_rmem
4096    87380    4194304


From various discussions, it appears that these modified values will speed up rv.net.


# echo "4096 16384 131072" > /proc/sys/net/ipv4/tcp_wmem
# echo "4096 87380 174760" > /proc/sys/net/ipv4/tcp_rmem


Note that you’ll need a root prompt for this, use ‘sudo -s’ to get one.

For a more permanent solution, add the following to /etc/sysctl.conf


net.ipv4.tcp_wmem = 4096 16384 131072
net.ipv4.tcp_rmem = 4096 87380 174760

Use ‘sysctl -p’ if needed to get these values implemented in the current session. – I note that my system has “net.ipv4.conf.all.log.martians = 1” commented out. After watching the History Channel and their fixation on UFO’s and alien life, I am wondering if I need to change that, too …

From the tcp man page, tcp_rmem is the receive buffer minimum, default, and maximum sizes that depend upon available system memory. Minimum is usually the 4k standard memory page size. Default is usually 87380. tcp_wmem is the send buffer. Minimum is memory page size. Default is 16k, maximum is calculated.

So the suggestion for rv.net is to reduce the receive buffer maximum to 174,760 and the send buffer to 131,072 (128MB) from their initial 4,194,304 (4MB). Since this does seem to make a dramatic change in response, it raises the question about why and where the numbers came from, especially the receive buffer max as it isn’t an even multiple of the memory page size.

From Pittsburgh Supercomputing Center:

Recent versions of Linux (version 2.6.17 and later) have full autotuning with 4 MB maximum buffer sizes. Except in some rare cases, manual tuning is unlikely to substantially improve the performance of these kernels over most network paths, and is not generally recommended

What seems to be a factor is what is called TCP window scaling. The protocol allows for a 64kb address space for a window into the transmitted data that allows buffering to fix errors that may occur. Window scaling is a workaround to create a bigger buffer for faster networks with large files. Some routers mess up the workaround and that causes problems like seen with rv.net. kerneltrap has a discussion about this. “cat /proc/sys/net/ipv4/tcp_window_scaling” comes up with 1. echo this to 0 to eliminate window scaling. The reduction of the maximum buffer size also ameliorates the problem because it, too, limits the window scaling.

From there, it gets to bandwidth delay products. That is a calculation about how much data is in transit at one time. The buffers need to be big enough to handle about twice what is in transit to detect errors and request a resend if necessary. Modern networks have more in transit and that means larger buffers are needed which is why the RFC hack was created. But that causes other problems. The bufferbloat episode is related but in this case it is router firmware that does not implement the RFC correctly even after more than ten years.

And it is things like this that put fear in the heart of network administrators looking at the transition to IPv6.