AI systems that survive contact with production.
LLM applications, RAG systems, and classic ML — built with evaluation harnesses, cost discipline and monitoring, because a demo is not a product.
- Use-case triage
- Evaluated prototype
- Production engineering
- Operate & improve
If it isn’t evaluated, it isn’t done.
The distance between an impressive demo and a dependable AI product is engineering: evaluation sets, guardrails, fallback paths, cost budgets, drift monitoring. A demo is measured by whether it worked in the room. A product is measured by what it does on the tenth thousand request, on the input nobody anticipated, at three in the morning when the provider is degraded and the answer still has to be either correct or an honest refusal.
Most stalled AI projects are not stalled on model quality. They are stalled because nobody can say whether the new prompt is better than the old one, because per-request cost was discovered after launch, because the retrieval corpus was never measured and the answers quietly drifted with it, or because there is no defined behaviour for the case where the model is wrong. We build AI systems the way we build everything — to be operated, not just launched — which means the unglamorous parts are in scope from the start, not deferred to a hardening phase that never gets funded.
Every AI behaviour ships with a test set and a score to beat — and a change that lowers the score does not merge, whoever proposed it.
Four practices, one engineering standard.
We run these on our own products as well as for clients — the scoring and recommendation models inside Loyalty & Retention are operated by the same practice, against the same evaluation discipline.
LLM applications
Copilots, extraction, summarisation and conversational systems — prompt pipelines, tool use and structured outputs engineered properly. Prompts are versioned artefacts in the repository with tests, not strings pasted into a console. Outputs that feed another system are schema-validated and repaired or refused, never parsed hopefully. Tool use is bounded by explicit permissions, so an agent that can read cannot quietly write. See LLMs & RAG for how we build the pieces.
RAG done right
Retrieval quality measured, chunking tuned, citations verifiable — grounded answers instead of confident noise. Retrieval is evaluated as its own component before generation is tuned at all, because a system that retrieves the wrong passage cannot be fixed by a better prompt. Chunking, embedding choice, hybrid keyword blending and re-ranking are decided on measured recall against a labelled set, and every claim in an answer carries a citation a user can open.
Classic ML
Prediction, scoring and optimisation — the tier-prediction and next-best-offer models inside our own loyalty platform are this practice. Forecasting, churn and propensity scoring, routing and assignment optimisation: problems where a gradient-boosted model on clean features beats a language model on cost, latency and explainability, and where the real work is the feature pipeline that Data Engineering keeps honest.
LLMOps & evaluation
Eval suites, regression gates on model changes, token-cost budgets, latency SLOs and drift alerts. Every behaviour has a frozen set with a score to beat; every prompt, model or retrieval change runs against it in CI. Cost is a budget per request with an alert, not a surprise on the monthly invoice. Drift is watched on inputs and on outcomes, so quality decay is detected before a user reports it — the operating side of MLOps.
How the work runs, day to day.
A score to beat, before anything scales.
The first artefact of any engagement is an evaluation set: real inputs from your domain, expected behaviours agreed with the people who own the outcome, and a scoring method that matches what “good” means for the use case — exact match for extraction, faithfulness and citation accuracy for retrieval, rubric scoring with human calibration for open text. Once it exists, every argument about prompts, models and retrieval settings becomes an experiment instead of an opinion, and no change reaches production without moving the number in the right direction.
- Sets frozen and versioned; new failure cases are added, never quietly removed
- Regression gate in CI — a prompt, model or index change that lowers a score fails the build
- Human calibration on a sample, so the automated judge is trusted for a reason
Token economics decided before launch, not after.
An AI feature has a unit cost, and it is discoverable from the start. We model cost per request across the realistic input distribution, then engineer it down: prompt caching for the stable prefix, retrieval that sends a few relevant passages instead of many hopeful ones, smaller models on the classification steps that do not need a large one, and structured outputs that stop a model narrating its work. The result is a budget per request with an alert wired to it — and a system whose economics are modelled for traffic growth.
- Cost model published before launch, with the assumptions written down
- Caching, routing and context trimming measured against the eval set, so quality is not traded away silently
- Budget alerts on cost per request and per tenant, not only on the monthly total
The parts that make it dependable when the model is not.
Every production AI system needs defined behaviour for the day the provider is degraded, the retrieval index is stale, the output fails validation or the user asks something the system should not answer. We design those paths explicitly — a smaller fallback model, a cached or deterministic response, a human escalation queue, an honest refusal — and test them like any other failure path, because they are the difference between a degraded feature and a broken product.
- Provider abstraction so a model can be swapped after an evaluation, not after an outage
- Validation, repair and refusal on every structured output
- Escalation to a human queue with the context attached, where the stakes require it
Three people have to be satisfied.
A use case triaged honestly, including the ones we decline.
Value multiplied by feasibility, scored with your team — and an explicit column for “do not use AI here”. Deterministic rules beat a model on many problems that get proposed as AI work, and saying so early is cheaper for everyone than discovering it in the second sprint.
- Each candidate scored on value, data readiness, tolerance for error and cost
- Acceptance behaviour agreed in writing before build — including what the system must refuse
- Sequencing that puts a measurable, low-blast-radius use case first
An AI service that behaves like the rest of your estate.
Versioned prompts, typed interfaces, schema-validated outputs, tracing through retrieval and generation, and a regression gate in the same pipeline as everything else. Deployed on infrastructure your team already operates, with the runbooks to match.
- Provider-abstracted so a model change is a configuration decision
- Traces across retrieval, generation and tool calls for a single request
- Deployed on Kubernetes or serverless in your own cloud account
Sensitive data handled as a design input, not an afterthought.
What leaves your boundary, to which provider, under what retention, and what happens when a subject asks. Redaction in the pipeline, in-region or self-hosted models where policy requires, and a record of what the system was asked and what it answered.
- PII detection and redaction before anything leaves the estate
- Zero-retention endpoints, or self-hosted models where KVKK / GDPR requires
- Request and response logging with retention agreed, for audit and incident review
What is standard and what is scoped per client.
How the work runs.
Use-case triage
Value × feasibility scoring — including the honest “don’t use AI here” column. We look at the data that exists rather than the data that was promised, at the tolerance for a wrong answer, and at what the process does today when a human gets it wrong. Candidates come out ranked, with the acceptance behaviour written down and a first evaluation set sketched, so the build starts against a definition of good rather than a demo.
Evaluated prototype
A working system against a frozen eval set — quality is a number before scale is a plan. Retrieval is measured before generation is tuned, model and provider choices are benchmarked on your data at your cost budget, and the failure cases found in this phase become permanent members of the set. The phase ends with a score, a cost per request and a written recommendation — including, occasionally, that the use case is not ready.
Production engineering
Guardrails, fallbacks, cost controls, monitoring — the unglamorous work that makes it dependable. Input validation and PII handling, output schema validation with repair and refusal paths, rate limiting and provider abstraction, tracing across the request, budget alerts and latency SLOs. The regression gate goes into CI so the eval set defends the system from the next well-meaning prompt edit.
Operate & improve
Drift tracked, evals rerun on every change, user feedback looped into the sets. Input distributions and outcome quality are watched, corpora are refreshed on a schedule rather than when someone notices, and provider releases are evaluated before adoption instead of arriving by surprise. Your team runs this with us until it is plainly theirs, with runbooks for drift, outage and quality incidents.
What you keep.
- Deployed AI system with monitoring and cost dashboards
- Evaluation suites with baseline scores and regression gates
- Guardrail and fallback architecture, documented
- Model/provider decision record with benchmarks on your data
- Runbooks for drift, outage and quality incidents
When this is the right call.
Choose this when an AI capability has to be operated rather than demonstrated: a copilot inside a product real customers pay for, extraction that feeds a system of record, retrieval over a corpus your staff rely on, or scoring that changes what a business does with a customer. It fits organisations with a prototype that impressed everyone and has not moved since, teams whose AI costs surprised them, and regulated environments where an answer needs a citation and a refusal path. It pairs with Data Engineering when the inputs need building first, and with Product Discovery when the use case itself is still an argument.
Look elsewhere when the problem is well served by rules or a query — plenty of “AI projects” are a report and a threshold, and we will tell you that before you spend a budget on it. This is also the wrong engagement when the data does not exist yet: no evaluation set can be built from a corpus nobody has assembled, so the first work is a data platform, not a model. And if what you want is faster software delivery rather than an AI feature, that is AI-First Engineering.
Frequently asked questions
Services that pair with this.
Tell us what you’re building.
Describe what exists today and what you need. An engineer reads it and replies in writing with how we would approach it, and the model that fits.