Yenya's World

Fri, 09 Sep 2005

TCP segmentation offload

Last week LWN had an interesting article about TCP offload engines in the Linux kernel. This week they added a follow-up to that article written by a vendor of TOE-equipped hardware. I don't have a TOE-capable NIC myself, but in the comments section people discussed the whether a full TOE gives any significant advantage when compared to a simple TCP segmentation offload. Mingo said that any modern-enough NIC which supports TSO is also supported under Linux, including tg3 which we use in most of our servers.

I wondered whether my version of the tg3 hardware supports TSO, and if so, whether it is better than no TSO at all. I found that according to ethtool -k eth0 the TSO is switched off at Odysseus. So I tried to switch it on (altough I wondered why it is not switched on by default, provided that the hardware supports this feature).

I tried to measure the difference by downloading an ISO image of FC4 i386 CD1 (665434112 bytes) from two hosts connected to the same switch. I did 10 transfers of the same file with each settings, and took the average and maximum of the last five transfers only (to avoid any start-up temporary conditions). The client Alpha was dual Opteron 248 with Tyan S2882 board, and the client Beta was quad Opteron 848 on HP DL-585 board. The FTP server is dual Opteron 244 on Tyan S2882 board, and ProFTPD with sendfile(2) enabled.

ClientTSOAverage speedMax speed
Alphaoff108.7 MB/s110.5 MB/s
Alphaon100.9 MB/s101.2 MB/s
Betaoff102.1 MB/s102.4 MB/s
Betaon93.2 MB/s95.5 MB/s

Surprisingly enough, the tests without TSO were faster than with it enabled. Looking at tcpdump it seems that the system with TSO enabled sends only a 15 KB-sized frames to the NIC instead of full 64 KB-sized ones:

18:45:38.993150 IP odysseus.ftp-data > alpha.33125: P 127424:143352(15928) ack 1 win 1460 <nop,nop,timestamp 2408404 698142942>
18:45:38.993203 IP odysseus.ftp-data > alpha.33125: P 143352:159280(15928) ack 1 win 1460 <nop,nop,timestamp 2408404 698142942>

So I wonder what is wrong with TSO on my hardware. Any clues? I think I should poke the netdev mailing list.

Section: /computers (RSS feed) | Permanent link | 0 writebacks


Yenya's World: Linux and beyond - Yenya's blog.


RSS feed

Jan "Yenya" Kasprzak

The main page of this blog



Blog roll:

alphabetically :-)