0

Need to make some optimizations and have been reading articles such as linux-tcp-tuning

In a world of load balancers and revers proxy is there any value in tuning TCP connections that are inside the network? As I understand (please correct me if I'm wrong) there is almost zero latency and no packet loss between the load balancer and the servers.

So I wonder if changing parameters such as net.ipv4.tcp_window_scaling, net.ipv4.tcp_timestamps, net.ipv4.tcp_sack will have any effect if not set on the load balancer. Thanks!

EDIT:

Other parameters improved my server in simulation. I set net.ipv4.tcp_fin_timeout=5, net.ipv4.tcp_keepalive_probes=5 and it served more clients (without a load balancer). I just don't know if it will have the same effect with a load balancer. There are so many TCP parameters under /proc/sys/net/ipv4/. Testing them on a single server is one thing. Testing them with a load balancer and many servers is a huge effort. I just don't know if it worth the effort of testing those parameters with many severs and a load balancer.

5

Tuning the parameters you have identified will certainly have an effect - to degrade performance. For example, net.ipv4.tcp_window_scaling is an on or off option. Similarly, net.ipv4.tcp_sack is either on or off. Both default to a value of on on Linux and on all load balancers. Both of these were defined in RFC1323 which was published in order to give us the ability to achieve 10/100 Ethernet networks. Without this RFC, networks were limited to 10 Mbps a second, so disabling these will result in a huge, immediate and noticable performance hit.

These performance benefits come because there is a huge difference between "almost" zero latency and zero latency. Using a sliding TCP window allows equipment to use Selective Acknowledgements (SACKs) to acknowledge the receipt of hundreds of packets simultaneously. So, if latency is 10ms on a network, without SACKs, every packet has to be acknowledged and that 10ms latency adds up very quickly - even at theoretical minimum latencies. This means that I send packet A, then have to wait for a response to packet A before sending packet B. But as it turns out I can send the rest of the alphabet and then some while waiting for the ACK to packet B. Enter SACKs - These allow the receiver to signal that it is has successfully received up to and including packet Z. Meanwhile, the sender is already sending packet AA-ZZ.

This results in something known as a "TCP Window Size" which is the maximum amount of data the client can buffer/handle. Since memory can be a concern, if the recipient is only capable of storing and processing Packets A-F, if we send A-Z some amount of data is going to be lost. Therefore, it becomes necessary for the recipient to signal their ability to receive data - that is "I can only buffer this many packets). This is really CPU dependent on how fast the recipient can empty their buffers and transmit the data up to layer 7. It is not uncommon to see TCP windows scale from empty to full (Max window size goes from 0 indicating the window is completely used to max indicating the window is completely available) and this is one way throughput bottlenecks can be identified on your network.

TCP Window Scaling offers two primary benefits:

  1. Less unused time on the wire - we aren't sitting around waiting for an ACK and we have successfully defeated the prop. delay (latency).
  2. Less time devoted congestion control = more time for data. Congestion control messages can eat into your transmission time and bandwidth. Even with TCP Window Scaling, Congestive Collapse can still occur in which an entire TCP window may need to be retransmitted and congestion control may be required.

Finally, the only real reason to update net.ipv4.tcp_timestamps is to provide additional security as it is otherwise based off of server uptime. Since you are proxying behind a load balancer anway, adjusting this setting is of little value unless you are using something like a Performance Layer 4 profile. I doubt adjusting this will make much difference, though again it is only likely to degrade, not improve performance.

Far and away, what is more likely to help is adjusting your send and receive buffer sizes and using Jumbo Frames. this will allow you to process more data with the same amount of overhead (a 75% reduction in processing over a standard MTU of 1500. These are the primary changes required to reach a 10 Gbps connection. Otherwise, just make sure you have enough CPU to ingest your data from the TCP stack and be aware of the pattern of your traff (eg, avoid things like the C10K problem).

EDIT:

tcp_fin_timeout doesn't actually improve performance, instead it defeats resource constraints. This setting could actually be a major player in a load balancer implementation. The problem is that for any given client and server, the server will be hosting a web server (for example) on 192.168.1.5:80. Every TCP connection must have both a source IP and source port as well as a destination port and destination IP. Source ports are chosen by the client from a pool of "ephemeral" ports. This results in a theoretical maximum of 65534 connections (the maximum number of ports) between a given client to port 80 on that server. When using load balancers, this can result in "port exhaustion" when the pool of ephemeral ports is depleted when SNAT (Secure Network Address Translation) is used on the load balancer, which results in a theoretical maximum of 65534 between the load balancer and the server. There are two basic solutions to this problem: 1) use a SNAT pool where 2 source IPs for the load balancer * 65534 = 131068 connections and 4 IPs X 65534 = 262136 max connections, and so forth or 2) don't use a SNAT. For this to work, the load balancer needs to be part of the default route back to your clients. If you are testing with a single actual client simulating multiple clients, this is a poor real-world test and you may actually support many more clients than you think because you are exhausting your ephemeral ports, where real clients would never do that because they come from unique IPs and two clients can use the same ephemeral port if you are not using SNAT on the load balancer.

Now, I keep mentioning that this is the theoretical maximum but the reality is that on Linux servers, there is a defined range of ephemeral ports and so typically you will see port exhaustion at around half of the theoretical maximum. You can see results similar to adjusting tcp_fin_timeout by adjusting the ephemeral port range in /proc/sys/net/ipv4/ip_local_port_range.

This brings me to the final piece, the TIME_WAIT state governed by tcp_fin_timeout and defaults to 60 seconds. This gives every port used a 60-second "cooling off period" before it is returned to the ephemeral port pool. This is why you saw a performance boost in your testing. This setting is supposed to be 2 times your maximum segment lifetime as defined by the TCP protocol, as it is theoretically possible that for a FIN packet with this setting below below 1 X maximum segment lifetime (30 seconds) to still be in transit. In reality, this is very unlikely, but technically you are breaking RFC compliance by adjusting this setting. If you adjust this setting downward however, you would want to be sure to adjust it downward on the Load Balancer's TCP profile (so tcp_fin_timeout corresponds to Time Wait, not FIN Wait on an F5, for example).

Ultimately, this is a fantastic example of where testing with a load balancer and many servers would result in a huge performance difference - because your testing probably doesn't reflect the real-world architechure. It isn't a good simulation of the customer experience.

However, tweaking both ephemeral port settings and using keepalives are two divergent traffic management strategies. In fact, if you use a setting like OneConnect, there simply is no longer any need to tweak your ephemeral port ranges and timeouts because connections are recycled and multiple clients can be funneled over the same connection (note, this only works for HTTP traffic, but similar can be achieved with tcp_keepalive_probes if you want to configure it that way because clients no longer have to wait for the three-way-handshake - they can simply start transmitting.)

Conclusion

There is no pat answer here. You are asking "Is it worth doing all this work?" - I don't know. Is it? What is the acceptable level of business risk? What does it cost your business if you get it wrong and bottleneck?

Ultimately, I feel you are basically asking two things 1) what do all these settings do and what is the effect of tweaking them? And 2) are there any shortcuts I can take?

Unfortunately, you have not provided enough information to answer either question (which is why your last question was closed and this one will be downvoted.) First, I don't have the time to explain what every TCP setting in proc does (nor do I know off the top of my head, I'm actually googling these on the fly.) but secondly (and more importantly) it is impossible to determine their effect without knowing how your service is architected. There is probably performance that can be squeezed out of your system, but kernel TCP parameters are pretty much tuned to the best possible configuration that can be safely made without breaking the TCP/IP RFC. What you are doing is pushing into territory that can be safe, but only if you understand the architecture and the TCP/IP protocol. And it would behoove you to learn it. If you want to be the smartest guy in the room, learn TCP/IP. Pick up a copy of TCP/IP Illustrated and/or Internetworking with TCP/IP and commit the basics to memory. Learn how HTTP requests and SSL handshakes work. Learn how to read a pcap in wireshark and tell the story of what is happening according to these protocols. This will take you very far in life whether you go in to System Administration, Networking, or Programming (I have used this knowledge in all three realms). Simply put, there is no substitute for knowing what the hell you are doing and this just isn't something you are going to be able to answer off of Stack Exchange. This is why you pay consultants - because if they are good they know the ins-and-outs of this. Once you understand 1) your architecture and 2) what the settings actually do and what effect they will have you will better be able to tell if you can take any shortcuts (that is, I don't need to tweak this setting; it is irrelevant. This testing will be worthless because it doesn't simulate the customer experience.

  • Thanks. Now I understand that the parameters that I choose as an example will degrade performance. Please see me as it expand the question and make it clearer. – nmnir Apr 2 '18 at 19:22
  • Post Edit comment: Trust me that I know much about pcap, wireshark, etc.. I was not asking about each and everyone parametrs. The question was clear and the parametres ware just as an example. Lot's of manualls about improving servers were written and I wanted to know if they are relevant. In short you say maybe or that you don't know. DV if you want, hope that someone will hire you as a consultant fiven the effort you put (though I will not) – nmnir Apr 18 '18 at 19:10

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.