Publications

Enhanced control path for repeated TCP connections

2021 12th ACM SIGOPS Asia-Pacific Workshop on Systems (APSys '21), Virtual, Association for Computing Machinery, New York, NY, USA, 131–137. https://doi.org/10.1145/3476886.3477512

Junho Lee, Gyeongsik Yang, Zhixiong Niu, Peng Cheng, Yongqiang Xiong, Chuck Yoo

Download PDF

Abstract

This paper presents FALTCON that enhances the control path for repeated TCP connections. First, we measure and find that the control path of TCP stack consumes as many CPU cycles as that of the data path, which brings up the importance of optimizing the control path. Yet, to the best of our knowledge, there has been little research effort on investigating the control path. Also, we observe that a significant portion of TCP traffic (e.g., HTTP) is not only short-lived but also repeated for a server and client pair. We design FALTCON to take advantage of the property of being repeated. Specifically, FALTCON re-designs the control path to remove the duplicate allocation of the structures and redundant operations over them. FALTCON is implemented in Linux 5.1 that has the latest and highly efficient networking stack. Furthermore, we optimize FALTCON to be lockless entirely and to work per-core. The experiment results show that FALTCON achieves a higher number of connections than Linux, up to 19%, and with much less CPU cycles up to 31%.

Paper summary

Reuse the setup work of repeated TCP connections

Repeated connections can avoid rebuilding the same kernel structures and state every time.

TCP connection establishment allocates structures and initializes state even when endpoints repeatedly reconnect. FALTCON keeps reusable objects from those connections to reduce duplicate work in the control path.

Keep reusable state close to the CPU core

An archive and two keepers preserve connection structures and state. Per-core archives and receive-side scaling keep a connection’s packets on the same core, reducing shared-state synchronization in the implementation.

How FALTCON retains connection state while keeping reuse local to a CPU core.
MechanismPurpose
Archive and keepersPreserve connection structures and state
Per-core archivesLimit shared-state synchronization
Receive-side scalingKeep a connection’s packets on the same core

Measure connection setup and web-server capacity

Tests use Nginx without HTTP keep-alive and clients that repeatedly establish connections. Compared with Linux, FALTCON reduces connection-establishment CPU cycles by up to 31%. For 64-byte requests across the tested CPU-core counts, it handles 9–19% more connections per second.

Original Figure 4(a): Linux and FALTCON connections per second across 1–18 CPU cores.View full-size figure
Connections per second as CPU cores increase. FALTCON handles more connections than Linux under the tested 64-byte HTTP workload over 10-Gbps Ethernet.

Key takeaways

  1. Repeated connections repeat control-path work even when the application exchange is small. Reusing setup structures targets that cost before packet processing begins.
  2. Reuse must avoid becoming a synchronization bottleneck. FALTCON keeps archived state local to each core, linking its reuse strategy to the multicore execution model.