LLM & agentic pipelines
Agentic LLM pipelines — systems where models plan, call tools and execute multi-step work — are the highest-leverage and highest-risk pattern in applied AI. The math is unforgiving: chain ten steps that are each 95% reliable and the pipeline succeeds only ~60% of the time, unless validation gates catch and repair errors between steps. NorthSight Technologies engineers multi-agent pipelines to a ~90% successful-run rate, measured by evaluation harnesses rather than claimed.
- Agentic pipelines chain many model decisions — per-step errors compound unless engineered against.
- The reliability toolkit: structured outputs, validation gates between steps, guardrails, retry/repair logic, and a failure taxonomy.
- Measured result on our pipelines: ~90% successful-run rate, ~60% faster delivery, ~40% of outputs first-pass merge-ready (requirements→code case).
- Framework choice matters far less than validation discipline.
Where agentic systems break
Multi-step reasoning fails in characteristic, catchable ways. Across the systems we've built and audited, the recurring failure modes are:
- Compounding step errors — a slightly wrong plan makes every downstream step wronger.
- Loops — an agent retries a failing approach indefinitely instead of escalating.
- Bad tool calls — malformed arguments, wrong tool for the job, or results misread.
- Context blowups — accumulated history crowds out the instructions that matter.
- Unvalidated hand-offs — step N's plausible-but-wrong output becomes step N+1's trusted input.
We maintain a failure taxonomy for every pipeline we build: each observed failure is classified, counted and mapped to a specific defence. That is what turns "it usually works" into a number.
The reliability layer
- Structured outputs. Every step emits schema-validated data, not free prose — so errors are caught at the boundary where they occur, not three steps later.
- Validation gates. Between steps: schema checks, business-rule checks, and LLM-as-judge review where rules can't be written. Failing outputs are repaired or retried, not passed along.
- Guardrails & budgets. Loop limits, cost ceilings, tool allow-lists, and escalation paths to humans for the cases the pipeline shouldn't decide alone.
- Evaluation harness. End-to-end success measured against defined criteria on a golden set of real tasks — every prompt or model change is scored before it ships.
Case study: requirements → PR-ready code
A genericized example from our work: a planner → implementer → reviewer pipeline that turns requirements into production-grade code, tests and documentation. With structured outputs, validation gates, a failure taxonomy and an evaluation harness, the measured results were ~60% faster delivery, a ~90% successful-run rate, and ~40% of outputs merge-ready on the first pass — with quality measured, not guessed. We're happy to walk through the real architecture and trade-offs on a call.
Frequently asked questions
What is an agentic LLM pipeline?
A system where LLM agents plan, call tools and execute multi-step work — e.g., requirements to reviewed code, or a document to a completed workflow. Because it chains many model decisions, reliability has to be engineered between the steps, not hoped for.
Why does our agent demo fall apart in production?
Demos exercise the happy path once; production runs the unhappy paths thousands of times. Without validation gates, per-step error rates compound across the chain. The fix is structural: schema-validated outputs, checks between steps, and measured end-to-end success.
What does ~90% successful-run rate actually mean?
Roughly nine of ten end-to-end runs complete to defined success criteria without human rescue, as measured by the evaluation harness on real tasks. The remaining runs fail loudly and route to a human — never silently.
Which agent framework should we use?
The honest answer: it matters less than people think. Plain Python orchestration, provider SDKs, or an established framework can all reach production quality. We design the failure handling and evaluation first, then choose the lightest tooling that supports it.
Can you make our existing agent system reliable?
Usually. We start with an audit: trace real failing runs, build the failure taxonomy, then retrofit the reliability layer. Full rebuilds are the exception.