We Made Our AI 3x Faster
by Making It Dumber

The Problem: 448MB of Neural Networks on a Modest VPS

FaceVault runs on a single modest VPS. Not a GPU cluster. Not a Kubernetes fleet. One box with a handful of cores and a tight RAM budget. This is deliberate — we keep infrastructure costs minimal, and it forces us to be efficient.

Every verification loads four ONNX models:

With a tight container memory limit, those 448MB of model weights were eating a big chunk of our budget before a single request came in. Each uvicorn worker loads its own copy of the models into memory, so scaling to multiple workers would have multiplied that footprint — a non-starter without shrinking the models first.

We needed the models to get smaller. Ideally a lot smaller. Without retraining them, without switching architectures, and without losing the accuracy we'd spent weeks tuning.

What INT8 Quantization Actually Is

Neural networks store their learned knowledge as millions of floating-point numbers called weights. By default, most models use 32-bit floats (FP32) — each weight takes 4 bytes, can represent values with ~7 decimal digits of precision, and covers a range from roughly ±3.4 × 10³⁸.

INT8 quantization converts those 32-bit floats into 8-bit integers. Each weight drops from 4 bytes to 1 byte. That's a 75% reduction in model size, with a proportional reduction in the amount of data the CPU needs to move through its caches during inference.

FP32 weight:  0.0372914671897888  (32 bits, 4 bytes)
INT8 weight:  9                   (8 bits, 1 byte)

Scale factor: 0.00414349...
Zero point:   0

Reconstruction: 9 * 0.00414349 = 0.03729...
                                  ≈ original value

The trick is that neural networks are remarkably tolerant of imprecision. A face embedding doesn't need 7 decimal digits of precision per weight to distinguish your face from someone else's. The learned patterns survive quantization even when individual weights are rounded.

There are two flavors of quantization:

The Entire Script Is 6 Lines

This is not a metaphor. The quantization script that gave us 2–3x faster inference is genuinely this short:

from onnxruntime.quantization import quantize_dynamic, QuantType

models = [
    ("w600k_r50.onnx",              "w600k_r50_int8.onnx"),
    ("depth_anything_vits.onnx",     "depth_anything_vits_int8.onnx"),
    ("db_resnet50-69ba0015.onnx",    "db_resnet50-69ba0015_int8.onnx"),
    ("parseq-00b40714.onnx",         "parseq-00b40714_int8.onnx"),
]

for src, dst in models:
    quantize_dynamic(src, dst, weight_type=QuantType.QInt8)

That's it. quantize_dynamic from ONNX Runtime reads each model, maps every FP32 weight to its nearest INT8 representation (with per-channel scale factors stored in the model), and writes a new .onnx file. No training data. No GPU. No hyperparameter tuning. It took about 30 seconds per model.

The only dependency is the onnx Python package (not installed in production — we ran this inside the container, then copied the quantized models into the persistent volume). ONNX Runtime, which we already use for inference, loads INT8 models transparently. No code change needed on the inference side — just point the model path to the new file.

# Before
_ARCFACE_MODEL_PATH = os.environ.get(
    'ARCFACE_MODEL_PATH', '/models/w600k_r50.onnx'
)

# After
_ARCFACE_MODEL_PATH = os.environ.get(
    'ARCFACE_MODEL_PATH', '/models/w600k_r50_int8.onnx'
)

We kept the originals on disk. If something goes wrong, reverting is changing one string.

Before and After

Three workers loading INT8 models use 375MB for weights. That's less than what a single worker used to need with FP32. This is what unlocked our move from 1 uvicorn worker to 3 — previously impossible within our original container memory limit.

What About Accuracy?

This is the question that matters. If quantization made our face matching unreliable or our OCR miss MRZ fields, the speed improvement would be worthless.

We ran our full test suite — 286 tests covering face matching, anti-spoofing, document fraud detection, OCR extraction, trust engine scoring, webhooks, billing, and end-to-end session flows. All 286 passed. No threshold adjustments needed. No new edge cases.

This isn't surprising from an information theory perspective. Here's why:

The Other 6 Optimizations We Shipped

Quantization was the headline, but it was part of a broader "Phase 1" optimization push. All changes were free (no infrastructure cost), took a single afternoon, and shipped without any API contract changes. Here's the full list:

Every one of these changes was under 10 lines of code. Most were 1–3 lines. The combined effect is multiplicative, not additive — faster models × more workers × less contention × fewer wasted reads = significantly more than the sum of its parts.

By the Numbers

Zero dollars spent. One afternoon of work. Roughly 3x the verification capacity on the same server. The single most impactful change was INT8 quantization — not because it was the cleverest optimization, but because it was the one that unlocked all the others. Without smaller models, we couldn't add workers. Without more workers, the semaphore and thread pool changes wouldn't matter.

Further Reading