Why We Rebuilt Our OCR Pipeline
From Scratch

The Problem With "Just Use Tesseract"

When we first built FaceVault's document intelligence layer, the plan was simple: extract the MRZ (Machine Readable Zone) from passports using PassportEye, and fall back to Tesseract OCR for everything else. MRZ extraction was rock solid. Passports have that lovely two-line code at the bottom with standardised formatting. Parse it, done.

Then the real world showed up.

A Singapore NRIC doesn't have an MRZ. Neither does a Malaysian MyKad, an Irish PPSN card, most EU national IDs, or the majority of driver's licences worldwide. For these documents, we were relying entirely on Tesseract — and it was struggling. The extraction would come back garbled, fields would bleed into each other, and names would get mixed up with document headers. We were seeing match rates that weren't acceptable for a product people would trust with their identity.

We tried the obvious fixes. We tweaked Tesseract's page segmentation modes. We added image preprocessing — CLAHE enhancement, adaptive thresholding, denoising. We tried different binarisation strategies. Each tweak helped a little, but the fundamental problem remained: Tesseract was giving us a wall of text with no structure, and we were using fragile regex patterns to fish out names and dates from that wall. It was a house of cards.

The Conversation That Changed Everything

I was stuck. The kind of stuck where you've been staring at OCR output logs for hours and everything looks like MRZ filler characters. So I did what any engineer does when they've exhausted their own ideas — I called a trusted old buddy.

The conversation was one of those rare ones where the solution doesn't come from a single brilliant insight, but from someone asking the right questions. "Why are you merging all the OCR text into one blob before extracting fields?" — that one hit different. "Why are you using the same preprocessed image for both engines?" — that one too.

And then, with the blessing of the great Lord and a fresh perspective, the architecture of the new pipeline crystallised. Not a tweak. Not a patch. A complete rethink of how we approach document text extraction.

The New Pipeline: Four Layers Deep

Our new OCR pipeline runs in four stages. Each stage is designed to succeed where the previous one might fail, and the confidence-weighted reconciliation at the end picks the best result for each individual field.

Label-Aware Extraction: Reading Like a Human

The single biggest accuracy improvement came from label-value association. Think about how you read an ID card. You don't scan every word and try to figure out which one is the name. You look for the label "SURNAME" and then read the text next to it or below it. That's exactly what our pipeline does now.

We maintain a pattern library covering labels in English, Malay, French, German, Spanish, Italian, Irish, and more. The system recognises that "NAMA PENUH" on a Malaysian MyKad means the same thing as "FULL NAME" on a UK driving licence, or "SLOINNE" on an Irish PPSN card means "SURNAME".

Critically, the system handles split name labels. Many European and Irish IDs don't have a single "FULL NAME" field — they have separate "SURNAME" and "FORENAME" labels. The old pipeline would find "SURNAME", grab the value, call it the full name, and move on. The new pipeline extracts surname and forename independently, then combines them: forename + surname = full name. Simple, but it was the difference between a failed match and a passed verification for every Irish, German, French, and many other EU documents.

The Invisible Label Problem

Just when we thought we'd cracked it, we hit another wall. The Irish PPSN card is green. A nice, official, government-issued green. And the field labels — "SURNAME", "FORENAME", "DATE OF BIRTH" — are also printed in green. A slightly different shade, barely distinguishable to the eye, and completely invisible to standard OCR preprocessing.

The actual values (the person's name, their date of birth) are printed in black. Those come through perfectly. But without the labels, our label-value extraction strategy had nothing to anchor to. It was like having a perfectly good map but no street signs.

The fix required understanding why the labels were invisible. Our standard preprocessing converts to LAB colour space and applies CLAHE (Contrast Limited Adaptive Histogram Equalisation) to the L channel — the lightness channel. This works brilliantly for most documents because text is usually darker or lighter than the background. But when the label and background have the same lightness and only differ in hue or saturation, the L channel shows zero contrast.

This second OnnxTR pass only triggers when the first pass finds no labels at all — so there's no performance penalty for documents that work fine with standard preprocessing. When it does trigger, it specifically targets the label extraction, feeding the newly visible labels back into the same label-value association logic. The result: documents that were completely unreadable now extract cleanly.

What This Means For You

If you're a CTO or product lead evaluating KYC providers, here's what this pipeline delivers in practical terms:

If you're a developer integrating FaceVault, nothing changes in your API calls. The OCR pipeline runs automatically when a user uploads an ID document. You get the same data_match, name_match, dob_match, and nationality_match fields in the session response — they're just far more likely to be correct now.

Being Honest About Our Approach

We could have taken the easier route. Cloud OCR APIs from the big providers would have given us decent results with a fraction of the engineering effort. But that would mean sending your users' identity documents to a third party. Every passport photo, every selfie, every name and date of birth — routed through someone else's infrastructure. For a product built on the promise of privacy, that was a non-starter.

So we built our own. OnnxTR running locally on ONNX Runtime, Tesseract as a specialist backup, and a custom extraction layer that understands what identity documents actually look like. It took longer. It was harder. But now every byte of your users' data stays within FaceVault's infrastructure, processed in memory, never touching an external API, and purged on schedule.

Is it perfect? No. There will always be edge cases — damaged documents, unusual layouts, handwritten fields. But the architecture is designed to improve incrementally. Every new document type we encounter teaches the system something new, and the layered approach means we can add new extraction strategies without rewriting the pipeline.

Sometimes the best engineering comes from the simplest realisations: give each tool the input it was designed for, read documents the way a human would, and trust the math to pick the best answer. The rest is just persistence — and having good people to talk to when you're stuck.