Real-time AI does not always need a GPU.

There is a recurring assumption in production AI: if the experience has a strict real-time requirement, the model belongs on a GPU.

Sometimes that is true. Often it is an expensive shortcut.

The better question is not, "Is a GPU faster?" It is, "What is the least expensive serving design that can meet the production target and service-level objective under realistic load?"

I had to answer that question while building EAD Mynd, a multilingual conversational avatar for Environment Agency - Abu Dhabi. The platform combined grounded retrieval, Arabic and English text-to-speech, zero-shot response classification, speech alignment, and lip-sync timing in a public-facing experience. It ultimately served more than 3,700 launch interactions, maintained 99.9% uptime, and delivered a time-to-first-byte below four seconds.

LLM response generation used a separate serving path; the CPU optimization discussed here applied to response classification and speech alignment.

The infrastructure was deliberately CPU-oriented. That meant the inference path could not depend on keeping an expensive GPU available for every live session. I used two complementary runtime strategies: ONNX Runtime for the zero-shot response classifier and OpenVINO for the speech-alignment models. Around them, I added quantized model variants, warm starts, bounded concurrency, caching, chunking, and streaming.

The result was not one magical optimization. It was a stack of small decisions that removed enough work, made the remaining work cheaper, and delivered useful output early enough for the avatar to still feel live.

Start With The Metric, Not The Runtime

Inference optimization becomes confused when "faster" is treated as one number.

For an offline batch job, throughput may be the priority: finish the largest number of inputs per second. For a live avatar, the user experiences latency. More specifically, the user notices how long the system appears silent before something useful arrives.

That makes at least four measurements important:

• cold-start latency: the first request after a process or model starts
• warm p50 latency: the normal middle of the distribution
• p95 and p99 latency: the slow requests that determine whether the experience is dependable
• time-to-first-byte: how soon the application can begin returning a useful response

Throughput still matters because concurrent users share the same CPU. Cost matters because a fast design that requires an underutilized GPU for every replica may not be economically attractive. Accuracy matters because a model that is fast but changes the product's behavior is not an optimization; it is a different model.

So the target I optimize is not minimum kernel time in isolation. It is an operating envelope:

Meet the quality threshold
+ meet the tail-latency target
+ sustain expected concurrency
+ minimize cost per successful request

That framing is what made CPU inference a real option for this project.

The Main Inference Optimization Levers

ONNX and OpenVINO are important, but they sit inside a larger toolbox. In practice, I think about inference optimization in layers.

The order matters. The cheapest inference is the inference that never runs, which is why cache design and request routing can beat a low-level kernel improvement. The next best move is often reducing the amount of data or model work. Runtime conversion should be part of the solution, not the entire performance strategy.

Graph optimization

A training framework needs flexibility for gradients, debugging, dynamic control flow, and research. A serving runtime can make stronger assumptions. Once a model is exported, it can fold constants, remove redundant nodes, fuse compatible operations, and choose layouts or kernels suited to the target hardware.

ONNX Runtime's graph optimizer includes basic, extended, and layout transformations. These range from eliminating identity and dropout nodes to fusing patterns such as matrix multiplication plus addition, GELU, attention, and layer normalization.

This is why exporting the same learned weights can improve inference even before quantization. The model's behavior is intended to stay the same, but the execution plan becomes leaner.

Reduced precision and quantization

FP32 is a safe default, not a universal production requirement. Lower-precision representations reduce model size and memory bandwidth, and supported processors can execute them more efficiently.

There are several different choices hiding behind the word quantization:

• post-training quantization applies lower precision after training
• dynamic quantization calculates activation parameters during inference
• static quantization calculates them ahead of time from representative calibration data
• quantization-aware training exposes the model to quantization effects during training
• mixed precision keeps sensitive operations at higher precision while reducing the rest

ONNX Runtime's quantization guide recommends evaluating method choice by model type and hardware. Its general guidance favors dynamic quantization for transformer and RNN workloads and static quantization for CNNs, but the benchmark and accuracy check on the target machine remain decisive. OpenVINO's NNCF workflow similarly uses representative calibration data for post-training INT8 quantization and supports an accuracy-aware path when a simple conversion loses too much quality.

Quantization is not free speed. Unsupported kernels, quantize/dequantize transitions, old CPU instruction sets, or a poorly calibrated dataset can make the result slower or less accurate. I treat every precision change as a new artifact that must pass the same functional evaluation as the original model.

Smaller models, pruning, and distillation

If runtime optimization is not enough, change the amount of model being served.

Pruning removes parameters or structures with limited contribution. Distillation trains a smaller student to reproduce the behavior of a larger teacher. Architecture replacement is simpler still: if a smaller model meets the business quality threshold, serving the larger one is often waste.

These techniques can produce bigger gains than runtime tuning because they reduce the work at its source. Their cost is that they move the problem back toward training and evaluation. For a production system already approaching launch, exporting and optimizing an accepted model may carry less delivery risk than training a replacement.

Batching, concurrency, and threading

Batching improves throughput by amortizing overhead and feeding larger operations to the hardware. It can also damage an interactive SLO if the service waits to fill a batch. Dynamic batching helps, but it still introduces a queueing tradeoff.

Threading has a similar trap. More threads do not automatically mean more throughput. A web server may have worker processes, the inference runtime may have its own thread pool, and BLAS or OpenMP may create more threads underneath that. Multiplying all three can oversubscribe the CPU, increase context switching, inflate memory use, and make p99 latency worse.

The right unit of tuning is the whole serving process: number of replicas, processes per replica, runtime threads per process, request concurrency, and memory per loaded model.

Caching, warmup, and streaming

Caching skips repeated work. Warmup moves one-time compilation, allocation, and kernel selection out of the first live request. Streaming does not necessarily make the full computation finish sooner, but it changes perceived latency by returning useful partial output before the entire job completes.

For conversational products, that last distinction is crucial. A response that begins in two seconds and finishes in six often feels faster than one that arrives all at once after five.

ONNX, ONNX Runtime, And OpenVINO Are Different Things

The names are often used interchangeably, but they solve different parts of the deployment problem.

ONNX is an open model representation: a portable computation graph, operator definitions, standard data types, weights, and metadata. It is the artifact and interoperability layer.

ONNX Runtime is an execution engine for ONNX models. It optimizes the graph and assigns supported subgraphs to execution providers such as CPU, CUDA, TensorRT, OpenVINO, and other hardware-specific backends. The execution provider is what connects the portable graph to a concrete device implementation.

OpenVINO is a model optimization and inference toolkit with strong CPU support and device-aware compilation. It can consume models from several ecosystems, compile them for a selected device, apply performance hints, and work with lower-precision artifacts. OpenVINO's LATENCY and THROUGHPUT hints express two different serving goals; its documentation recommends treating thread count, stream count, core type, hyper-threading, and pinning as coordinated scheduling controls rather than isolated knobs.

TensorRT occupies a similar hardware-specific role for NVIDIA GPUs. It is the natural candidate when the selected deployment target is NVIDIA and GPU cost is justified by model size, concurrency, or latency. It was not the right default for a CPU-first deployment.

The practical selection is:

Training framework
        |
        v
Portable/exported graph (for example ONNX)
        |
        +--> ONNX Runtime + CPUExecutionProvider
        +--> ONNX Runtime + CUDA/TensorRT/OpenVINO provider
        +--> OpenVINO Runtime on CPU/GPU/NPU
        +--> TensorRT engine on NVIDIA GPU

ONNX is therefore not a synonym for quantization, and exporting to ONNX does not by itself turn FP32 weights into INT8. This distinction mattered in this project because I used ONNX export for one service and separately preferred quantized OpenVINO artifacts in another.

How I Used ONNX

One of the avatar pipeline's small but latency-sensitive services classified generated responses into behavioral categories such as greeting, explanation, listing, limitation, and affirmation. The implementation used a multilingual zero-shot classifier.

The baseline path loaded the model through Transformers and PyTorch on CPU. I then exported the sequence-classification model with Hugging Face Optimum's ORTModelForSequenceClassification, explicitly selecting ONNX Runtime's CPUExecutionProvider. I saved the exported model and tokenizer together so production workers could load a ready serving artifact instead of performing conversion during startup.

In simplified form, the conversion looked like this:

from optimum.onnxruntime import ORTModelForSequenceClassification

onnx_model = ORTModelForSequenceClassification.from_pretrained(
    model_name,
    export=True,
    provider="CPUExecutionProvider",
)

onnx_model.save_pretrained(output_dir)
tokenizer.save_pretrained(output_dir)

The production API then loaded that directory directly with the same CPU provider and placed the model behind a normal Transformers zero-shot pipeline. That kept the application-facing classification contract familiar while replacing the underlying execution path.

The production engineering around the model mattered as much as the export:

• each worker loaded a clean ONNX Runtime model after process creation instead of inheriting runtime state across a fork
• model warmup ran representative classifications before serving normal traffic
• Gunicorn used synchronous, single-thread workers for the CPU-bound endpoint
• OpenMP, MKL, and OpenBLAS thread counts were capped to prevent nested thread pools from oversubscribing the machine
• the API exposed warm latency, p95, p99, throughput, and batch measurements rather than relying on one timing sample
• the classifier's application-level executor was bounded instead of growing with request volume

This is an important detail: the classifier optimization in the production branch is an ONNX export and ONNX Runtime CPU-serving path. It should not be described as INT8 merely because quantization was one of the broader experiments. Conversion, graph optimization, and quantization are separate steps, and production documentation should say which ones actually happened.

Where OpenVINO Fit

The speech-alignment service had a different workload. It used separate Arabic and English acoustic alignment models to turn synthesized speech into the character timing needed for avatar lip-sync.

For this service I used OpenVINO IR models and compiled them for CPU inference. The production path:

• loaded separate Arabic and English processors and alignment models
• preferred quantized OpenVINO model artifacts when available
• fell back to FP32 artifacts if a quantized model could not be loaded
• allowed dynamic audio lengths instead of forcing every request into one fixed shape
• enabled CPU pinning and a compiled-model cache
• warmed each language model with one-, two-, and three-second dummy inputs
• kept audio preprocessing minimal and passed contiguous NumPy arrays into inference

The fallback was part of the optimization design, not an admission of failure. Lower precision should be preferred only when it remains operationally and functionally valid. A known-good FP32 path provides a safe recovery option if a quantized artifact is missing or incompatible.

OpenVINO also made the latency-versus-throughput choice explicit. A single interactive request benefits from latency-oriented scheduling, while a shared alignment service under concurrent avatar sessions may benefit from multiple inference streams. The correct configuration comes from load testing the real concurrency pattern, not from setting every performance flag to its largest value.

The End-To-End Optimizations Around The Models

Neither runtime operated alone. The full speech path was designed to get work off the critical path and expose partial results early.

Long answers were split into short, language-aware chunks, normally capped around 80 characters and broken at natural sentence boundaries. Each chunk could be synthesized, aligned, encoded, and emitted as newline-delimited JSON without waiting for the entire answer.

When both alignment and response classification were required, those independent operations were submitted in parallel for the current chunk. The Azure speech synthesizer was reused behind a lock so the service did not rebuild its client and connection state for every request. A startup warmup exercised the SDK before live traffic.

Redis cached completed requests using the text and relevant synthesis options as part of the key. Repeated content could therefore bypass TTS, alignment, and classification rather than merely running them faster. A separate raw-audio path skipped decoding, fades, silence insertion, re-encoding, and alignment when the caller explicitly did not need those features.

These choices attacked different portions of latency:

Model execution + serving-path controls

The latency result came from coordinating model execution with the serving path. Runtime conversion reduced model cost, while warmup, caching, concurrency control, parallel work, streaming, and request-specific fast paths protected the end-to-end production SLO.

That combination is how CPU-first inference supported the product requirement. The documented result is an end-to-end time-to-first-byte below four seconds for the live avatar, not a claim that one ONNX call took four seconds or that ONNX alone met the production SLO.

What The CPU Decision Saved

The infrastructure decision was not only technically viable; it had a measurable cost consequence. Using UAE North rates retrieved from Microsoft's Azure Retail Prices API, a one-year savings plan prices a 64-vCPU VM at approximately $2.30656 per hour and one NVIDIA A10 VM at approximately $3.8090624 per hour.

At 730 operating hours per month:

64-vCPU CPU VM
= $2.30656 × 730
= $1,683.79 per month

Full-A10 GPU VM
= $3.8090624 × 730
= $2,780.62 per month

Monthly saving
= $2,780.62 - $1,683.79
= $1,096.83

The CPU configuration therefore cost about 60% of the full-A10 alternative and reduced the monthly compute bill by approximately 40%. That is about $13,161.92 in annualized compute savings for each continuously running instance.

The saving did not come from accepting a lower production target. The optimized CPU deployment met the required peak throughput, peak concurrent-load envelope, and sub-four-second time-to-first-byte SLO. Once those requirements were satisfied, paying for the full dedicated A10 would have bought unused acceleration rather than a better production outcome.

This comparison uses current public Azure retail pricing as a normalized reference, not a reconstruction of a confidential client invoice. It excludes storage, networking, managed services, support, tax, and migration engineering, all of which would need to be held constant in a formal total-cost-of-ownership comparison.

How I Would Benchmark The Decision

A credible CPU-versus-GPU decision needs a reproducible benchmark matrix. I would record at least:

The inputs should reflect production lengths and languages. Warmup requests must be excluded from warm latency but reported separately as cold-start cost. CPU and GPU runs need the same correctness suite and the same response contract. Tail latency should be collected under sustained load, because an impressive single-request measurement can hide queueing and thread contention.

The final comparison is not "CPU milliseconds versus GPU milliseconds." It is:

monthly serving cost
--------------------
successful requests meeting quality and latency targets

For this project, the CPU path met the product's real-time response target and removed the need for a GPU-heavy serving design. That made it materially cheaper at the infrastructure level while keeping the system deployable on the available environment. I would only publish an exact percentage saving if the GPU baseline, utilization assumptions, cloud SKU, traffic profile, and measurement period were available alongside it.

When I Would Still Choose A GPU

CPU-first should not become another dogma.

I would move toward a GPU runtime when the model cannot meet its tail-latency target after responsible optimization, when concurrency is high enough to keep batching efficient, when the workload is dominated by large matrix multiplications, or when model size and generation length make memory bandwidth decisive. Large generative models often sit in this category.

I would favor ONNX Runtime on CPU when portability matters, the exported operators are well supported, and I want to preserve a familiar Transformers-style application interface. I would favor OpenVINO when CPU or Intel device optimization, compiled artifacts, quantization workflows, and detailed scheduling controls are central to the deployment. I would favor TensorRT when NVIDIA GPUs are already the chosen target and their utilization can justify the additional cost and build complexity.

The runtime follows the workload and hardware. It should not decide them in advance.

What I Learned

The strongest lesson from this project was that inference optimization is a systems problem.

Exporting a model can help. Quantizing it can help more. But either gain can disappear if every web worker creates a large runtime thread pool, if cold processes receive live traffic before warmup, if the application waits for a full response before streaming, or if it recomputes identical work that should have been cached.

The production sequence I trust is:

1. Define the end-to-end quality, p95/p99 latency, concurrency, and cost targets.
2. Profile the whole request and locate the critical path.
3. Remove unnecessary work and add safe cache or fast paths.
4. Export to a serving graph and validate output parity.
5. Test reduced precision with representative calibration and evaluation data.
6. Tune processes, runtime threads, pinning, and batching together.
7. Warm the real shapes and cache compiled artifacts where appropriate.
8. Stream useful output as early as the product permits.
9. Load-test the target hardware and compare cost per successful SLO-compliant request.

GPUs are extraordinary inference devices. They are also not a substitute for architecture.

For this project, ONNX Runtime and OpenVINO made CPU execution capable enough. The surrounding decisions made it production-ready. Together they let the avatar remain responsive while avoiding a GPU dependency that the workload did not require.