Runtime Optimization in Java Systems

In high-throughput Java services, performance problems often appear in the right tail of the latency distribution rather than in the mean. Network waits, queue accumulation, lock contention, garbage collection, and downstream variability shape P99 and higher percentiles. With one million requests in an hour, the P99 region represents roughly ten thousand requests, so mean and median values alone are operationally incomplete.

In multi-layer call chains, end-to-end latency cannot be derived from the percentile value of a single service. If each of five independent components has a 1 percent probability of exceeding a given latency threshold, the probability that at least one component in the call chain exceeds that threshold is approximately 4.9 percent, calculated as 1 - 0.99^5. This calculation depends on the assumption of independence. In real systems, latencies may be correlated because of a shared database, the same network path, shared processors, or concurrent GC activity. In such cases, the simple probability calculation may remain optimistic.

Accurate tail-latency measurement depends directly on the traffic model used by the load generator. In a closed-loop test, a virtual user sends the next request only after receiving the response to the previous one. When the server slows down, the client also reduces its request generation rate. Consequently, the constant or externally driven traffic pressure that would enlarge the queue in production decreases automatically during the test.

This measurement error is known as coordinated omission. For example, when the server cannot respond for one second, closed-loop clients do not create new requests during that interval. Because the histogram measures only requests that were actually sent, the real queue that would have formed during the pause remains invisible. In open-loop or constant-arrival-rate tests, request generation time is decoupled from server response time. When the system exceeds capacity, queue growth, rejection, and timeout behavior become observable. If production traffic is externally driven and independent of server speed, capacity testing must preserve the same property.

High-traffic Java services require latency distributions, allocation, GC, lock contention and downstream waiting to be measured together. A healthy mean can coexist with a growing P99 tail, so optimisation decisions should be driven by end-to-end queue and resource behaviour rather than microbenchmark results alone.

Queueing Theory and Pool Sizing

Little's Law describes the relationship among average concurrency, average arrival rate, and average time spent in a stable system:

L = λ × W

Here, L is the average number of jobs or requests simultaneously present in the system. λ, represented by the Greek letter lambda, is the average number of arrivals per unit of time. W is the average time between a request entering and leaving the system.

For example, consider a system receiving an average of 2,000 requests per second with an average response time of 40 milliseconds:

L = 2,000 × 0.040 = 80

This result indicates that, in a stable state, an average of 80 requests will be executing or waiting. It is not, however, a direct thread-pool or connection-pool size. Little's Law relates averages; by itself, it does not model P99 latency, traffic bursts, the service-time distribution, or resource contention.

A database connection pool must not grow without limit according to the number of concurrent application requests. Each connection consumes a session, memory, lock state, execution context, and processor time on the database side. Even when the application opens hundreds of connections, the database can execute only a limited number of queries efficiently. Beyond that boundary, throughput may stop increasing while context switching, lock contention, cache loss, and query queues continue to grow.

The formula ((core count × 2) + effective disk count) is a historical starting heuristic used for some traditional database workloads. It does not define a universal optimum for systems using SSDs, remote databases, multi-socket processors, mixed query profiles, or high network latency. Pool size must be established through controlled load tests while database CPU use, active sessions, query wait events, lock duration, and the application queue are observed together.

When the pool is exhausted, the system should apply a time-bounded admission policy rather than wait indefinitely. A bounded queue, fast-fail behavior, backpressure, or load shedding should be selected according to the system contract. connectionTimeout, transaction timeout, and driver-level network timeout values must be mutually consistent. Lifetime settings such as maxLifetime should also be shorter than connection limits enforced by the database, firewall, or load balancer, reducing the effect of connections that are silently closed while in use.

Threading Models

Platform threads are scheduled by the operating system, and each requires a native stack and kernel resources. Mapping a large number of concurrent, mostly waiting connections one-to-one onto platform threads increases memory consumption and scheduler overhead.

Virtual threads are lightweight threads managed by the Java runtime. Application code can remain sequential and blocking, while the JVM runs virtual threads on a smaller number of carrier platform threads. When a virtual thread waits in a supported blocking I/O operation, it can detach from its carrier. Its call stack is retained in heap-based stack chunks managed by the JVM, and the virtual thread is mounted on an appropriate carrier again when execution can continue.

This model does not increase CPU capacity. In a CPU-bound workload, raising the number of runnable tasks far above the core count does not create throughput and may increase scheduler pressure. The main benefit of virtual threads appears in systems where a substantial portion of many concurrent tasks wait on network, file, or database I/O.

Virtual threads also do not remove limits imposed by scarce resources such as database connection pools. The ability to create one hundred thousand virtual threads does not imply that one hundred thousand concurrent database queries are safe. Scarce resources must still be constrained through semaphores, connection pools, bounded queues, or equivalent admission-control mechanisms.

In the Java 21 period, some waits occurring inside synchronized blocks could pin a virtual thread to its carrier. For this reason, older guidance often included broad recommendations to use ReentrantLock. Starting with Java 24, pinning caused by object monitors was largely eliminated. In these versions, replacing all synchronized code with ReentrantLock merely because virtual threads are used is not a valid optimization strategy. Lock selection should be based on fairness, interruptible waiting, timed acquisition, condition support, and measured contention behavior.

Running long CPU computations while holding a lock enlarges the critical section regardless of whether virtual or platform threads are used. Performing I/O while holding a lock can also prevent other tasks from progressing. The priority should not be changing the lock type, but reducing shared mutable state and minimizing the scope of the critical section.

For highly contended statistics counters, the single memory location used by AtomicLong can become a bottleneck. LongAdder distributes updates across multiple cells and thereby reduces write contention. Its sum() result, however, is not an atomic snapshot during concurrent updates. Business rules requiring an exact and linearizable counter should use an appropriate atomic or locked structure instead of LongAdder.

Garbage Collector Selection

Garbage collector selection is a measurable trade-off among throughput, pause time, CPU cost, and memory reserve. There is no exact and universally valid rule that permits optimizing only two of three variables. The suitable choice depends on the workload's allocation rate, live-data size, heap capacity, latency objective, and processor reserve.

G1 divides the heap into regions and selects regions for evacuation according to predicted pause targets. It provides a balanced starting point between throughput and reasonable pause times for general-purpose server applications. However, MaxGCPauseMillis is not a strict upper bound. A high allocation rate, humongous objects, insufficient heap reserve, or a delayed concurrent-marking cycle can cause the target to be exceeded.

ZGC performs most marking, relocation, and reference-processing work concurrently with application threads. It is therefore suitable for systems targeting low pauses with large heaps. The fact that low pause times do not grow directly with heap size does not mean that pauses are constant or guaranteed under every condition. Root-set size, operating-system scheduling, memory pressure, and insufficient CPU reserve can still produce measurable deviations.

In Java 24 and later releases, ZGC operates only in generational mode. Tracking objects in young and old generations aims to collect short-lived objects at lower cost. Even so, using ZGC does not eliminate allocation-rate problems. If the application creates objects faster than GC threads can process them, the heap expands, CPU consumption rises, and allocation stalls or out-of-memory conditions may eventually occur.

Parallel GC focuses on increasing aggregate throughput by using multiple GC threads during stop-the-world pauses. It may be suitable for batch operations in which longer pauses are acceptable. In latency-sensitive online services, the same behavior can degrade P99 and P99.9 values. Instead of choosing by collector name alone, GC time, CPU use, allocation rate, promotion rate, live-data size, and tail latency should be compared under the same production-like load.

Allocation reduction must not be reduced to removing uses of new. Allocation of small, short-lived objects through TLABs can be very inexpensive. The larger costs arise from high allocation volume, large objects, unnecessary copying, long-lived intermediate data structures, and objects promoted into the old generation. Rather than introducing object pools without an allocation profile, optimization should target the paths responsible for the largest total number of allocated bytes.

Profiling and Safepoint Bias

JVM performance problems cannot be diagnosed through CPU samples alone. CPU time, wall-clock time, lock wait duration, file and network I/O events, object allocation, GC activity, and thread parking time are different data sets. A method appearing expensive in wall-clock time does not necessarily consume substantial CPU; the elapsed time may have been spent waiting for a downstream service.

Safepoint-based samplers can observe a thread stack only at locations considered safe by the JVM. If some regions of compiled code contain safepoints less frequently than others, the sample distribution can diverge from actual CPU consumption. This effect is known as safepoint bias. Long loops or heavily compiled hot paths may appear less frequently than they should, while code following the loop may appear disproportionately prominent.

On supported platforms, async-profiler can use perf_events and asynchronous stack-walking mechanisms to produce CPU, allocation, lock, and wall-clock profiles. Java Flight Recorder provides long-running event recording from within the JVM at low overhead. Java 25 also includes cooperative sampling improvements intended to increase the safety of JFR sampling while reducing safepoint bias. Tool selection alone is insufficient; the sampling source, frequency, recording duration, and workload must be stated explicitly.

In a flame graph, horizontal width represents the sample count or weight for the selected sample type. In a CPU flame graph, that width is approximately related to CPU consumption; in an allocation graph, it may represent allocated bytes or object count. Horizontal position does not represent chronological order. The vertical axis represents the call chain, and broad frames near the top commonly identify leaf or near-leaf methods generating direct cost.

JSON serialization, character encoding, temporary String creation, collection copying, and regular-expression processing may look moderate in a CPU profile while dominating an allocation profile. These allocations can indirectly degrade latency by increasing GC frequency. CPU and allocation profiles should therefore be evaluated together for the same load window.

JIT and Hot-Path Behavior

HotSpot may first interpret bytecode and then compile it into machine code at different tiers according to runtime profiles. C1 emphasizes fast compilation and profile collection, while C2 applies more expensive and advanced optimizations. Performance observed before the application has gathered sufficient profile data is not equivalent to steady-state performance.

Inlining does more than remove method-call overhead; it enables later optimizations such as constant propagation, devirtualization, and dead-code elimination. Not every method is inlined. Method size, call-site polymorphism, compiler budgets, and runtime profiles affect the decision. If an interface call repeatedly observes only one concrete type, the call site may be treated as monomorphic and converted into a direct call. If new types later appear, the JVM may invalidate the speculative code and recompile it.

Escape analysis examines whether an allocated object escapes the method or thread in which it is created. For suitable objects, HotSpot C2 can apply scalar replacement, separate the object into its fields, and eliminate the actual heap allocation entirely. This behavior is often described as stack allocation, but HotSpot does not generally move the complete object into a local stack allocation. It is more accurate to state that allocation is eliminated and the fields are represented as registers or compiler-managed temporary values.

When a speculative assumption is violated, the JVM may deoptimize a compiled frame. Required objects can be materialized again, and execution may continue at a lower compilation tier. If such events become frequent, they can create temporary spikes in the latency distribution. Without JIT logs, JFR compilation events, and code-cache metrics, this behavior cannot be determined reliably from source code alone.

Microbenchmark results can be distorted by JIT optimizations such as dead-code elimination, constant folding, or excessive inlining. The Java Microbenchmark Harness should be used to control warm-up, forks, result consumption, and measurement iterations. Even then, a microbenchmark does not represent queueing, network, GC, and database interactions in a production system. The effect of a local optimization on end-to-end latency must be measured separately.

GraalVM Native Image can reduce JVM startup and JIT warm-up costs by compiling the application into machine code ahead of time. Reflection does not disappear entirely; much of the dynamic access must be identified during the build or described through metadata. The Native Image runtime does not use the same dynamic optimization model as HotSpot C2, although PGO can be applied with workload profiles collected in advance. The decision should therefore consider throughput, memory use, build complexity, observability, and dynamic-feature requirements rather than startup time alone.

End-to-End Optimization

Serialization formats, compression, and zero-copy mechanisms are not automatic optimizations for every system. Replacing JSON with a binary format such as Protocol Buffers can reduce message size and parsing cost, but it introduces schema-management, debugging, and client-compatibility costs. Formats such as FlatBuffers can reduce copying for particular access patterns, yet their overall complexity may outweigh the benefit for small messages or simple workflows.

Compression should not be selected by applying a fixed threshold to payload size alone. Content compressibility, network bandwidth, processor reserve, latency objective, and client support must be measured together. Zstandard, Brotli, and gzip have different compression-ratio and CPU-cost profiles. Recompressing already compressed audio, image, or video data usually provides little benefit.

Zero-copy paths such as FileChannel.transferTo can reduce copying of file contents into user space. Their actual behavior depends on the operating system, file system, TLS use, and JDK implementation. When TLS termination or application-level transformation is required, a complete zero-copy path may not be available.

Runtime optimization cannot be completed through a single JVM flag, collector choice, or coding pattern. The service-level objective must first be defined, after which throughput and P50, P95, P99, and P99.9 latency should be measured under open-loop load. Queue length, active connections, rejected work, CPU use, allocation rate, GC events, lock waits, and downstream latency should be recorded over the same interval.

Changes should be applied individually and compared under the same traffic profile. A change that improves the mean while degrading tail latency, or increases throughput while raising the error rate, should not be considered successful. Java runtime optimization is not an exercise in setting flags by assumption; it is a systems-engineering process driven by measurable hypotheses, controlled experiments, and production feedback.

The Optimization Boundary Is a Distribution, Not an Average

An average runtime can hide GC pauses, JIT warm-up, lock contention, and scheduler outliers. In a production service the interesting question is often not how fast the best iteration runs, but how the latency distribution changes as concurrency and resource pressure rise.

That is where P99 Latency, Queueing Delay, CPU Affinity, and False Sharing connect micro-level observations to service behaviour. A change may improve throughput and still make tail latency worse; the workload's actual SLO decides whether it is an optimization.

References

• John D. C. Little. (1961). A Proof for the Queuing Formula: L = λW. Operations Research, 9(3), 383-387. doi:10.1287/opre.9.3.383

• Patricio Chilano Mateo; Alan Bateman. (2024). JEP 491: Synchronize Virtual Threads without Pinning. OpenJDK. URL

• Ron Pressler; Alan Bateman. (2023). JEP 444: Virtual Threads. OpenJDK. URL

• Stefan Karlsson. (2023). JEP 439: Generational ZGC. OpenJDK. URL