When many systems talk about low latency, they focus almost entirely on Layer 4.
- Faster send and receive paths
- Less queue buildup
- Shorter network round trips
- A more efficient kernel transmit path
All of that matters. But if the goal is truly low latency and extreme performance, focusing only on Layer 4 is not enough.
The reason is simple:
A request is never completed by Layer 4 alone.
In a real edge system, the final latency outcome of Layer 4 is often magnified directly by the hot paths above it.
- Layer 5 session establishment and connection governance
- Layer 6 identity checks, object preparation, and state transitions
- Layer 7 access control, outbound decisions, and runtime object reuse
If those paths remain heavy, serial, blocking, or full of global contention, then a fast Layer 4 will still not feel fast in the real product. What you end up with is a platform whose lower transport layer is fast on paper, while the system as a whole is not.
So our goal was never simply to make Layer 4 faster. Our goal from the beginning was this:
Start with low-latency Layer 4 as the target, then keep reducing Layers 5 through 7 so upper-layer handling no longer eats back the gains from the transport layer.
That is what this article is about.
1. Why Layer-4 low latency so often loses to Layers 5 through 7
If you break down a real edge request, you quickly find that the time spent is not limited to network send and receive.
Along the real hot path, the system also has to do work like this:
- Judge whether a session should be allowed during connection establishment.
- Determine whether a newly seen source falls inside a device-limit boundary.
- Decide whether a given service carries extra outbound capability and whether the matching runtime object needs to be prepared.
- Check whether source-address rules, access rules, or runtime identity have changed.
- Register the current connection into structures used later for governance.
A lot of that is not traditionally labeled as Layer 4 work, yet it directly determines some of the things users actually feel.
- Whether the first packet has to wait
- Whether the handshake path starts to stutter
- Whether new connections are delayed by external decisions
- Whether the system burns itself out in user-space management logic when peaks arrive
So a truly low-latency system cannot optimize transport alone. It also has to optimize everything that happens around transport.
That is exactly what we did.
2. Layer one of optimization: fully separate session gating from throughput control so the first-packet path does not pay for bulk traffic
When latency rises at peak load, a common reason is not that bandwidth has truly run out. It is that two fundamentally different concerns were mixed together.
- The cadence of new session establishment
- The cadence of sustained data throughput
Once those two concerns get forced into one unified limiter, the result is usually predictable.
- The system intends to control large transfer, but slows handshakes at the same time.
- The system experiences a burst of new connections, but the entire transfer scheduler gets polluted with it.
- What users feel is not that bandwidth is controlled. What they feel is that even connection establishment begins to lag.
We did not build it that way.
Inside the kernel design, session establishment and sustained throughput are two different paths.
- Session establishment goes through an independent handshake gate.
- Sustained transfer goes through independent directional rate control.
There is one especially important detail here.
Handshake gating is not a crude sleep. It is atomic time gating
The system maintains an independent handshake time gate for each service. It does not depend on heavy locks or one global queue. It uses an atomic time-window decision path instead.
That produces two important benefits.
- The first-packet path is shorter and less likely to slow down because of global contention.
- At peak load, the system can decide quickly whether to allow or reject, instead of piling up all new sessions in a waiting line.
At the same time, keep a very small and explicit burst window
This is a detail many people overlook.
Real traffic naturally contains small legitimate bursts: first-page loading, concurrent asset fetches, short fan-out, and similar behavior. If the system refuses all such burstiness completely, it turns normal product behavior into something stiff and unnatural.
So we preserve a controlled and very short burst window. That lets the system contain abnormal session storms without harming normal fan-out too early.
The value of that design is not that it looks smarter on paper. The value is this:
The first-packet path of low-latency Layer 4 should not carry the blame for sustained-throughput governance.
3. Layer two of optimization: a fast Layer 4 does not mean user-space sleep loops still deserve a place in the hot path
If a system genuinely cares about low latency, then once a Layer-4 task can be handled by the kernel, it should not remain in user space by habit.
That is especially obvious on the TCP downstream path.
Our approach is not to make user-space throttling try harder. It is to avoid making user space do the job whenever possible
On Linux, we prefer to push downstream pacing into the kernel:
- Prepare `fq` queue semantics on eligible interfaces.
- Set `SO_MAX_PACING_RATE` on connections that support it.
The result is not just more accurate rate limiting. It also means:
- Less user-space scheduling participation
- Less tail-latency jitter caused by sleep and wake batching
- Less chance that bursts will slow down user space simply because too many connections need pacing work there
The logic is straightforward. If the system wants extreme Layer-4 latency, then a large amount of download pacing should not still be wandering around in user space when it could live closer to the socket path.
Even the user-space fallback must not become a heavyweight scheduler
Not every path can be handed to the kernel completely, so we keep a very restrained user-space fallback throttler. But its value lies less in broad functionality and more in three concrete properties.
- A lightweight CAS-driven timeline
- No accumulation of large implicit burst credit
- A maximum wait bound so old connection debt does not keep polluting later hot paths
If the goal of low-latency Layer 4 is to get as close to the kernel as possible, then the goal here is just as important:
Even when a path must stay in user space, user space itself must not become the latency hotspot.
4. Layer three of optimization: what hurts the Layer-5 hot path most is not judgment itself, but forcing the first request to wait for external judgment
In real systems, Layers 5 and 6 often damage Layer-4 latency not because their logic is inherently complicated, but because they assume the first request must wait until all checks are complete.
Device limits are a classic example.
A common implementation, when a public source appears for the first time, synchronously asks the control plane questions like these:
- Is this source allowed?
- Has the current device count already exceeded the limit?
- Can traffic continue to pass?
That is defensible from a feature perspective. From a performance perspective, it is almost the same as inserting external RTT directly into the first-packet path.
We explicitly avoid that.
Latency first, not synchronous waiting first
In our hot path, when a new source arrives for the first time, the current connection thread is not blocked immediately on an external check. The system performs quick local decisions first.
- Does a short local allow or deny cache already exist?
- If not, can the connection enter a short grace-allow window first?
- Is an asynchronous device check already in flight?
Only then does the background layer launch the real device validation concurrently.
A same-source burst triggers only one external check
If sixteen concurrent requests arrive from the same service and the same source in a short interval, the system does not blindly issue sixteen identical checks. It deduplicates them and leaves only one in-flight validation.
That matters enormously for burst performance because it prevents two secondary failures.
- The edge node does not create its own control-plane flood.
- A burst from one source does not drag down the first-packet path for the whole machine.
Results are written back and only then converged locally
When the background result arrives, the system writes it back into local allow and deny cache. If the final result is denial, the system evicts matching connections by service and IP instead of reacting with a vague global cleanup.
The core purpose of this full path is simple:
External judgment in Layers 5 and 6 should not be billed directly to the Layer-4 first-packet latency budget.
5. Layer four of optimization: source rules are not parsed on demand in the hot path. They are precompiled into structures that match quickly
Another common mistake is to assume that because the rules themselves are not complicated, parsing them on every request must be harmless.
That may be tolerable under light traffic. Under high concurrency and a low-latency target, it becomes expensive very quickly.
That is especially true for source-address allow rules.
- If raw rule text is reparsed every time,
- or if each request performs repeated string-heavy judgment,
- or if the rule representation itself is not directly matchable,
then the low-latency gains from Layer 4 are quickly consumed by rule handling above it.
So our design does not merely store rules. It does this instead:
Normalize rules in advance into exact-address sets and CIDR-prefix structures that are built for fast matching.
That means runtime processing is reduced to:
- Exact hits
- CIDR-prefix checks
- And only a very small amount of necessary logic
It is no longer about repeatedly reinterpreting raw text.
That difference may be easy to miss under low concurrency. Under real latency pressure and peak load, it becomes critical. Every "small" decision in the hot path eventually becomes part of the total latency bill once it is multiplied by scale.
6. Layer five of optimization: if Layer-7 outbound capability is not prewarmed, the first real request will always pay the cold-start cost
If a platform supports richer outbound capabilities, such as advanced egress objects, second-layer outbound paths, or user-level outbound policy, then one of the most common Layer-7 performance traps is this:
The first real request ends up doing all the platform's preparation work for it.
The typical sequence looks something like this:
- Validate whether the target host is legal
- Resolve DNS
- Create the runtime outbound object
- Establish identity mapping
- Write the result into cache
If all of that happens inside the first real request, then a low-latency Layer 4 no longer matters very much, because the request has already been slowed by a Layer-7 cold start.
Our optimization path here is very explicit.
1. Cache host-validation results
Outbound target hosts are not revalidated from scratch every time. The system maintains TTL-based validation cache and caps cache size so validation logic itself does not become a hotspot.
2. Deduplicate DNS resolution
Concurrent lookups for the same target host are not allowed to fan out repeatedly. They are deduplicated through singleflight semantics.
3. Prewarm outbound objects
When the system sees that a service has been configured with extra outbound capability, it does not wait for the first real request to create the object. It prewarms in the background so the hot path is moved forward in time.
4. Reuse outbound-object identity
The system gives runtime outbound objects stable identity and does not recreate or replace them unnecessarily on every request.
The value of this optimization is straightforward:
No matter how powerful a Layer-7 capability is, the first real business request should not be the thing that initializes it for the platform.
If the system really wants low latency, this kind of preparation work has to be moved out of the request hot path.
7. Layer six of optimization: separate authentication changes from policy changes so Layer 7 does not rebuild everything every time
There is another latency problem that low-latency systems often overlook.
The moment anything changes in the control plane, many systems reflexively rebuild all runtime identity and policy state together.
That is simple to implement in the short term, but costly in practice.
- User-space objects are rebuilt too often.
- Restrictions that could have been updated locally are forced down a full refresh path.
- Hot-path cache hit rates fall.
- Runtime stability is disturbed for no good reason.
So we split user-facing changes into two categories:
- Authentication-state changes
- Policy-state changes
If the change is only a policy change, such as:
- Rate limits changed,
- device limits changed,
- source rules changed,
- or outbound policy changed,
then the system pushes only those changes into the corresponding runtime objects instead of rebuilding the whole identity layer from scratch.
That may not look like network optimization at first glance, but it matters enormously to latency and performance. If every Layer-7 change triggers a full reset, then the stability of Layer 4 keeps getting disturbed from above.
True low latency is not just about a faster network. It is also about this:
When upper-layer state changes, only the layer that truly needs to move should move.
8. Layer seven of optimization: connection governance must stay local. Anomalies should not trigger global scans
When the system needs to govern abnormal sources, such as cases where:
- a source is denied by device limits,
- an identity changes,
- or a runtime object needs to converge,
the most dangerous possible response is this:
- Scan every connection globally,
- then inspect one by one to decide what should be cleaned.
That is barely tolerable at low concurrency. At peak load it immediately becomes a hotspot.
So our design follows two rules.
1. Shard connection registration
Connections do not all live inside one giant structure. They are registered in shards by user key. That keeps concurrent registration and teardown from all fighting for one hotspot.
2. Maintain a local `service + ip` index
The system does not merely know how many connections exist. It can quickly locate the local connection set for a given service and a given source.
That allows many governance actions to stay local:
- Evict only the connections for one source under one service,
- avoid scanning every connection on the machine,
- and keep anomalies from dragging down unrelated healthy traffic.
The meaning of this design is clear:
When Layer 7 needs to govern an object, it should not shake the global state of Layer 4 in the process.
9. The final layer: background convergence should handle only the objects that truly became hot
If a system genuinely cares about low latency, then background threads cannot behave carelessly either.
A lot of platforms carry the same long-term problem here.
- The hot path gets faster,
- then a background cycle scans every object anyway,
- and CPU plus cache are taken back by background work.
We explicitly avoid that.
Traffic and online-state construction use a dirty-mark and hot-set model:
- Only service instances that truly changed enter the hot set.
- Objects that did not change are not rebuilt repeatedly.
- Once convergence completes, the object exits the hot set quickly.
That may sound like a background implementation detail, but it is directly tied to low latency. End users do not care if the hot path got three milliseconds faster while background work destroyed the cache behavior of the whole machine.
Real extreme performance requires both foreground and background work to avoid doing meaningless things.
10. Why these Layer-5-through-7 optimizations are actually the prerequisite for Layer-4 low latency to become real
Once you assemble the design as a whole, the conclusion becomes very clear.
Layer 4 performance is responsible for making the road smooth
- Kernel-level pacing
- Lighter send rhythm
- Less user-space interference
Layer 5 optimization is responsible for keeping session establishment from stalling itself
- Independent handshake gating
- A small-window burst allowance
- Sharded connection registration
Layer 6 optimization is responsible for keeping validation and state transitions from blocking the first packet
- Asynchronous device checks
- Grace allow
- Deduplicated external validation
- Precompiled source rules
Layer 7 optimization is responsible for keeping application objects from wasting the lower-layer gains through cold start and full refresh
- Outbound-object prewarming
- Host-validation cache
- DNS deduplication
- Object-identity reuse
- Separate updates for authentication state and policy state
So we are not merely saying this:
"We optimized Layer 4, and we also optimized Layers 5, 6, and 7."
What we actually mean is this:
If Layers 5 through 7 are not light enough, Layer-4 low latency never becomes real in the final product.
That is why our way of chasing extreme Layer-4 latency has never been to stare only at Layer 4. It has always meant continuing to remove weight from the entire hot path above it.
Closing
A mature edge kernel should never treat low latency as a single network parameter, and it should never treat extreme performance as an isolated optimization in one layer.
It should have a different kind of capability:
Make Layer 4 fast first, then make sure Layers 5 through 7 do not give those gains back.
That is the cross-layer optimization work we keep doing inside the edge kernel.
It is not only about transport, not only about control, and not only about application objects. It is about making sure that:
- The session path is light enough,
- the validation path is short enough,
- outbound objects are warm enough,
- rule matching is fast enough,
- and state convergence is restrained enough.
That is how low Layer-4 latency becomes real, end-to-end low latency that users can actually feel.
If we had to summarize the capability in one sentence, we would define it like this:
Extreme Layer-4 latency requires hot paths in Layers 5 through 7 that are just as light, just as restrained, and just as controllable.
