Trusted data in, before any model sees it

An AI application is only as good as the data it receives. In practice, data engineering takes more effort than prompt engineering and model selection combined. Chapter 3 covers the pipeline that turns emails, documents, databases and recordings into clean, validated input.

Sources and connectors

Business data arrives in every shape: structured records in databases and APIs, semi-structured forms and invoices, unstructured emails and contracts, images, audio and video. Each needs a connector: an API client, a webhook receiver, change data capture on a database, a mailbox poller or a file watcher.

Choose connectors by freshness, volume and permission needs. Above all, carry the source system's access rights along as metadata. If a user may not see a document in SharePoint, the AI must not show it to them either.

Processing: from raw to structured

The processing stages are:

• extraction: text and tables from PDFs, Office files and HTML;
• OCR for scans and photos, with confidence scores;
• structuring: an LLM turns free text into typed records against a schema;
• summarization, normalization (dates, currencies, names), deduplication and filtering.

Use the simplest reliable extractor for each content type. Keep vision LLMs for what classic tools cannot do, and cross-check critical fields such as amounts and policy numbers.

Validate in two passes

Validate deterministically first: schemas, required fields, formats, ranges, checksums and cross-field rules. Only then validate semantically with an LLM, for checks that are hard to write as rules, such as whether a description fits the declared damage type or whether two documents contradict each other.

Records that fail do not disappear. They go to quarantine with a reason, where someone can fix them or the pipeline can retry.

Measure data quality continuously

Data quality is a metric, not a one-off clean-up. Track completeness, validity, consistency and freshness per source, and alert when they drift. Anything an LLM inferred rather than read should be labelled as inferred.

Keep originals and lineage

Store raw originals unchanged, record lineage from file to parsed text to record, and make every stage re-runnable. When you improve an extractor next month, you can then reprocess history instead of starting from scratch.

Key takeaways

• Keep raw originals, record lineage, make every stage re-runnable.
• Validate deterministically first, semantically second.
• Measure data quality continuously and quarantine failures.