Some Congestion Experienced One and Two-Flow TestsRedactedLiberec 30463 11Czech Republicpete@heistp.netRedactedPortland97217United StatesORrgrimes@freebsd.orgKokkonranta 21Pitkajarvi31520Finland+358 44 927 2377chromatix99@gmail.com
Internet
Transport Working GroupInternet-DraftSCEThis note presents one and two-flow test results for the SCE (Some Congestion
Experienced) reference implementation. These tests are not intended to be a
comprehensive real-world evaluation of SCE, but an illustration of SCE's
influence on basic TCP metrics in a controlled environment.IntroductionSCE provides early and proportional feedback to the CC (congestion control)
algorithms for transport protocols, including but not limited to TCP. The
is a Linux kernel modified to support SCE, including:
Enhancements to Linux's Cake (Common Applications Kept Enhanced) AQM to
support SCE signaling
Modifications to the TCP receive path to reflect SCE signals back to the
sender
The addition of three new TCP CC algorithms that modify the originals to add
SCE support: Reno-SCE, DCTCP-SCE and Cubic-SCE (work in progress as of this
writing)
In this note we run one and two-flow TCP tests across a range of simulated path
bandwidths and RTTs. One-flow tests measure SCE's impact on TCP throughput and
TCP RTT. Two-flow tests evaluate fairness between and among several SCE and
non-SCE TCP implementations, while making several adjustments to Cake's SCE and
fair queueing parameters.It is recognized that these tests do not simulate real-world conditions, and
will not be an indication of how SCE will perform in all situations. However,
they serve as fundamental tests for the SCE reference implementation. Once the
behavior in these tests is well understood and theory and experiment are in
agreement, additional complexity can be added to the test procedures with the
confidence that the reference implementation's fundamentals are sound.TerminologyThe following terminology is used in this document:
Path Bandwidth or Cake-limited Bandwidth: The available bandwidth between
the sender and receiver, as controlled by Cake on the middlebox, and set by
Cake's bandwidth parameter.
Path RTT or just RTT (in context): The approximate minimum round-trip time
of packets that go from the sender to receiver and back when the path is
unloaded.
Netem bi-directional delay: The total path RTT added by netem in both
directions.
TCP throughput and TCP RTT: Well-known terms that apply specifically to
the flows under test.
Test Tools and EnvironmentThe tool is used for all tests. Flent uses netperf for its TCP tests,
and allows for test batches, plotting, the recording of results and the
collection of metadata in JSON format . Flent both captures the
measured TCP throughput from netperf, and simultaneously uses the tool in
Linux to passively monitor TCP RTT.All tests are performed using a three node dumbbell topology:Test topology
+--------+ +-----------+ +----------+
| Sender |--------------| Middlebox |--------------| Receiver |
+--------+ +-----------+ +----------+
Sender: Runs Flent and sends data to the receiver
Middlebox:
Acts as a router between the sender and receiver
Runs Cake on egress of both interfaces for queue management and SCE signaling
Runs netem on the ingress of both interfaces for delay simulation, splitting
the total delay in half for each interface
Receiver: Receives data from and reflects SCE signals back to the sender via
the ESCE (Echo Some Congestion Experienced) bit
All nodes run the SCE reference implementation kernel as of commit 56915a82
(2019-06-20), and are connected directly via Gigabit Ethernet.TestsThe tests are implemented with a Flent batch file to drive netperf and
re-configure Cake and netem on the middlebox with various parameters. Scripts
post-process the results and create csv and markdown tables for external use,
including by this document.Unless otherwise mentioned, measurements are obtained from TCP flows from start
to finish, not at steady state. This allows for some discussion of the
differences in TCP CC algorithm behavior during slow start and congestion
avoidance. Typically, each test is run long enough to obtain a reasonable
approximation of steady state throughput, but in a few high BDP cases slow start
accounts for a significant portion of the test length. When relevant to the
analysis of the results, this is stated in the text.Results and AnalysisOne-Flow TestsThe goal of the one-flow tests is to analyze the impact of SCE on the TCP
throughput and TCP RTT of single TCP flows across a range of simulated path
bandwidths and RTTs. What follows is an analysis of the results. See
and for the raw results for TCP
throughput and TCP RTT, respectively.Reno-SCE TCP ThroughputThe following table shows the difference in TCP throughput for Reno-SCE vs Reno
across the tested range of simulated path bandwidths and RTTs:
Difference in TCP Throughput (reno-sce - reno), normalized to
Cake-limited Bandwidth; Columns: netem bi-directional Delay (ms);
Rows: Cake-limited Bandwidth (Mbit)
0
2
5
10
20
40
80
160
0.5
0.000
0.000
0.000
0.000
0.000
-0.100
-0.180
-0.160
1
0.000
0.000
0.000
0.000
-0.110
-0.210
-0.280
-0.280
2
0.000
0.000
-0.010
-0.120
-0.210
-0.375
-0.370
-0.405
5
0.000
-0.052
-0.142
-0.228
-0.178
-0.032
-0.006
0.016
10
0.000
-0.071
-0.139
-0.143
-0.046
0.062
0.116
0.168
25
0.000
-0.003
-0.006
0.010
0.073
0.143
0.196
0.259
50
0.000
0.000
0.001
0.029
0.102
0.169
0.235
0.272
100
0.000
0.000
0.004
0.043
0.103
0.215
0.238
0.289
From the above TCP throughput differences we can observe:
Improved utilization for SCE at sufficiently high BDPs. This is due to SCE's
proportional congestion signals, which can significantly reduce the classic
Reno throughput sawtooth by making drops or CE marks rare to non-existent.
The utilization improvement increases with BDP largely because the TCP window
recovery time after a drop or CE mark increases with BDP, deepening the
sawtooth.
Significant under-utilization at bandwidths <= 10Mbit, which tends to worsen
as path RTT increases. Investigation is underway as to the source of this.
These drops in utilization are however also accompanied by drops in TCP RTT
(see below).
Reno-SCE TCP RTTThe following table shows the difference in TCP RTT for Reno-SCE vs Reno
across the tested range of simulated path bandwidths and RTTs:
From the above TCP RTT differences we can observe:
Significant reductions in TCP RTT for Reno-SCE at most BDPs. SCE's
proportional congestion signals are aiding the sender in managing queue
lengths.
Greater reductions in TCP RTT at lower path bandwidths and RTTs. This is
because Reno-linear growth becomes relatively larger as the path bandwidth
decreases. This results in greater queue growth in the interval before AQM
activates, and a subsequently longer drain time after congestion is signaled.
A slight increase in TCP RTT for SCE at high BDPs. This has been observed to
have a possible connection to the ESCE (Echo Some Congestion Experienced)
feedback strategy implemented in on the TCP receive side. Research and
optimization in this area is ongoing.
DCTCP-SCE TCP ThroughputThe following table shows the difference in TCP throughput for DCTCP-SCE vs
DCTCP across the tested range of simulated path bandwidths and RTTs:
Difference in TCP Throughput (dctcp-sce - dctcp), normalized to
Cake-limited Bandwidth; Columns: netem bi-directional Delay (ms);
Rows: Cake-limited Bandwidth (Mbit)
0
2
5
10
20
40
80
160
0.5
0.000
0.000
0.000
0.000
0.000
-0.120
-0.220
-0.400
1
0.000
0.000
0.000
0.000
-0.120
-0.260
-0.420
-0.570
2
0.000
0.000
-0.015
-0.130
-0.285
-0.505
-0.565
-0.705
5
0.000
-0.038
-0.150
-0.270
-0.374
-0.588
-0.652
-0.620
10
0.000
-0.087
-0.197
-0.265
-0.266
-0.257
-0.271
-0.134
25
-0.006
-0.060
-0.131
-0.167
-0.154
-0.133
-0.067
0.110
50
0.000
-0.001
-0.006
-0.008
0.005
0.012
0.143
0.180
100
0.000
0.007
0.018
0.031
0.040
0.087
0.151
0.246
From the above TCP throughput differences we can observe:
Improved utilization for SCE at higher BDPs. At first glance we might assume
that this is due to SCE's feedback signals improving upon DCTCP's window
recovery time, as with Reno, but another significant part of this increase is
due to DCTCP-SCE's steeper ramp during slow start, and a test length that is
short relative to the time spent in slow start. The test is 300 seconds long,
and at a path bandwidth of 100Mbit and RTT of 160ms, DCTCP takes a full 225
seconds to ramp up to the BDP, while DCTCP-SCE, with its steeper ramp, takes
only 80 seconds. That said, steady state throughput at this BDP is around
87Mbit for DCTCP, and around 96Mbit for DCTCP-SCE, an increase of around 10%,
so the proportional congestion control signals also play a part in increasing
utilization.
Significant under-utilization at bandwidths <= 25Mbit, which tends to worsen
as path RTT increases. Investigation is underway as to the source of this.
These drops in utilization are however also accompanied by drops in TCP RTT
(see below).
DCTCP-SCE TCP RTTThe following table shows the difference in TCP RTT for DCTCP-SCE vs DCTCP
across the tested range of simulated path bandwidths and RTTs:
From the above TCP RTT differences we can observe:
A nearly across the board reduction in TCP RTT for DCTCP-SCE. SCE's
proportional congestion signals are aiding the sender in managing queue
lengths.
Greater reductions in TCP RTT at lower bandwidths and path RTTs. See
for an explanation of this.
A slight increase in TCP RTT for DCTCP-SCE at 100Mbit / 160ms. See
for a likely explanation of this.
Two-Flow TestsThe goal of the two-flow tests is to measure fairness between and among
SCE and non-SCE TCP flows, through either a single queue or with fair queueing.
What follows is a partial analysis of the results, and see
through
for the raw results tables.Single Queue (Cake "flowblind")Cake's flowblind parameter disables fair queueing, so that Cake uses only a
single queue. This can be used to evaluate single-queue throughput fairness
between SCE and non-SCE flows.Reno vs RenoIt might be useful to remind ourselves that competition between two (or more)
flows of the same TCP CC algorithm is usually designed to yield a high degree of
throughput fairness in a single queue. Looking at Jain's fairness index
for two flows across a range of simulated path bandwidths and RTTs
shows this to be true for two TCP Reno flows, for example:
From the above Jain's fairness index numbers we can observe that while there can
be reasonable fairness at very low BDPs, and while there is no starvation,
fairness degrades quickly at higher throughputs and RTTs. This is due to the
Cake's default SCE signaling ramp being tuned to provide an early signal of
congestion, to avoid CE marks and packet drops. As a result, SCE enabled flows
back off in the face of competition, whereas non-SCE flows fill the queue until
a drop or CE mark occurs.Cubic vs DCTCP-SCEOf particular interest to the congestion control community is competition
between the commonly used TCP Cubic algorithm and DCTCP-SCE. It is a
well-established fact that classic DCTCP will typically out-compete Cubic in a
single queue. It would be valuable if there were a way to improve that fairness
with SCE.When comparing Cubic vs DCTCP-SCE using Cake's default SCE signaling ramp, we
can see that while there is no starvation, fairness does degrade fairly rapidly
as BDP increases:
As of today, SCE by default does not lead to fairness at all BDPs between SCE
and non-SCE flows. However, efforts are ongoing to improve this, and as we can
see in , Cake's signaling ramp can be tuned to improve this
fairness.Single Queue (Cake "sce-single")As we saw in , there is room to improve SCE vs non-SCE fairness in a
single queue. One way to do this is to change the SCE signaling ramp to reduce
or delay SCE signals until closer to the point where CE signals occur. This is
the motivation behind the sce-single and sce-thresh Cake parameters.By default, Cake begins proportionally signaling SCE when a packet's sojourn
time in the queue is greater than half the CoDel target, and reaches 100% SCE
signaling when the sojourn time is greater than or equal to the CoDel target.
The sce-single parameter delays the start of the ramp until the sojourn
reaches the CoDel target itself, while keeping the ramp slope the same. The
sce-thresh parameter, while not evaluated here, allows an intermediate ramp
between the default and sce-single, using values from 2-1024, with
sce-thresh set to 8 yielding a ramp that's halfway in-between.It is expected that the result of signaling SCE later is a subsequent increase
in TCP RTT, including for single flows. Thus, the adjustment of the SCE
signaling ramp is a tradeoff between the increased utilization and reduced TCP
RTT possible with SCE, and fairness with non-SCE flows in a single queue.
By using sce-single, we can show the maximum fairness that can be achieved
by tuning the SCE signaling ramp in this way.Reno vs Reno-SCERevisiting the Reno vs Reno-SCE comparison that we did in
, we now run the same test with Cake's sce-single
parameter set:
We can see that single queue fairness has improved considerably. While there are
results at a few bandwidth-delay combinations that are still under
investigation, single queue fairness between Reno and Reno-SCE has largely been
achieved.Cubic vs DCTCP-SCERevisiting the Cubic vs DCTCP-SCE comparison that we did in
, we now run the same test with Cake's sce-single
parameter set:
While it can be seen that there is an almost across the board improvement vs
Cake's default SCE ramp, there are still bandwidth-delay combinations that do
not yield a sufficient level of fairness in a single queue. Work is ongoing to
improve this.Fair Queue (Cake "triple-isolate")Cake's default triple-isolate fairness mode provides fairness among flows and a
combination of source and destination IP addresses. For our purposes, this will
effectively serve as fair queueing among flows, as we are only using one IP on
each of the source and destination hosts.With fair queueing, we expect to achieve a high level of throughput fairness at
most bandwidth-delay combinations.Cubic vs DCTCP-SCERevisiting the Cubic vs DCTCP-SCE comparisons that we did in
and , we now run
the same test with Cake's fair queueing enabled via triple-isolate:
With fair queueing, fairness is achieved for a broad range of bandwidth-delay
combinations. We do see a consistent and narrow deviation at around 160ms
between 5 and 10Mbit, which happens in competition between SCE and non-SCE
flows, regardless of the exact algorithms in use. The cause of this will be
investigated.DCTCP vs DCTCP-SCEAs another example of SCE vs non-SCE competition with fair queueing enabled,
here we compare DCTCP vs DCTCP-SCE:
As with Cubic vs DCTCP-SCE, a high degree of fairness is achieved for most
bandwidth-delay combinations, yet we do see the same small gap in fairness at
around 160ms between 5 and 10Mbit as we see with Cubic vs DCTCP-SCE and other
SCE vs non-SCE flow tests with fair queueing. It is expected that further
investigation into this will lead to a solution.Security ConsiderationsThere are no known security considerations introduced by this note.IANA ConsiderationsThis document has no IANA actions.AcknowledgmentsMany thanks go out to Toke Hoiland-Jorgensen for making several key changes to the
Flent tool.Informative ReferencesThe FLExible Network Tester Home Pagess man pageSome Congestion Experienced Reference Implementation GitHub RepositoryAppendix (Raw Results Tables)One-Flow TCP Throughput