Skip to main content

Fast feedback: observability-driven engineering

An overnight batch system taught me what slow feedback costs. When a job failed, the team found out the next morning when operations called the on-call engineer. By then, downstream systems had consumed partial data, reconciliation had broken, and remediation took the rest of the day. The feedback loop ran for hours. Each incident cost hundreds of thousands of dollars.

Across 25 years of engineering leadership in banking and financial services, the speed of feedback loops has proven to be the single strongest predictor of team performance. This is not just intuition. The DORA research programme, published in Accelerate, demonstrated empirically that elite teams have feedback cycles orders of magnitude faster than their low-performing counterparts. During the DevSecOps transformation at a Tier-1 bank, the first capability invested in was not security tooling or deployment automation, it was observability. You cannot improve what you cannot see, and you cannot fix what you do not know is broken.

Why fast feedback matters​

Fast feedback loops let teams find and fix issues quickly, which is how they learn. In financial services, where regulatory obligations demand auditability and system reliability, detecting and remediating issues in minutes rather than hours is a risk control, not just an efficiency gain.

The research is unambiguous. Forsgren, Humble, and Kim found that high performing teams have a Mean Time to Recovery (MTTR) measured in minutes, not days. Google's Site Reliability Engineering discipline formalised this with error budgets, Service Level Objectives (SLOs), and structured on-call practices. The thread connecting all of this work is the same: shorten the time between a change being introduced and the signal that tells you whether it worked.

In AI-assisted engineering, the feedback loop includes the agent itself. Teams need to know whether the agent retrieved the right context, whether its memory was current, whether its model backend was healthy, and whether its tool-use trajectory supports the final recommendation. The latest AI research makes this explicit: agent memory, repository retrieval, and execution traces are observable production surfaces, not hidden implementation details.

The feedback system​

The feedback system has four parts. Continuous integration gives early feedback on every commit. Shift-left testing moves testing earlier in the delivery lifecycle so defects are caught when they are cheapest to fix. Monitoring and observability correlate logs, metrics, and traces to explain system behaviour in production. Agent observability captures prompts, retrieval context, memory writes, tool calls, model route, reasoning budget, and human overrides for AI-assisted workflows.

Continuous integration​

Continuous integration (CI) means regularly merging code changes into a shared repository and running automated tests against every change, so integration problems surface while they are still small. The practice rests on automated builds (in regulated environments, build provenance and reproducibility are auditable controls, not nice-to-haves), automated testing covering unit tests, contract tests, and static analysis, and frequent commits, at least daily to trunk. Long-lived feature branches are where feedback goes to die.

Example: CI in a banking environment​

At the bank, we implemented a CI pipeline that ran on every pull request. Within eight minutes of a developer pushing code, they received feedback on compilation, unit tests, static analysis (SonarQube), dependency vulnerability scanning (Snyk), and secrets detection. Before this pipeline existed, developers would wait until a nightly build to discover failures. The reduction in cycle time was dramatic: defect escape rate to integration environments dropped by over 60% in the first quarter.

The key insight was that speed matters as much as coverage. A test suite that takes 45 minutes to run will be bypassed by developers under deadline pressure. We invested heavily in parallelisation, test pyramid optimisation, and caching to keep the feedback loop under ten minutes.

Shift-left testing​

Shift-left testing is the practice of moving testing earlier in the development lifecycle. Rather than treating testing as a gate at the end of a sprint, testing becomes a continuous activity that begins at the moment a requirement is written.

  • Static analysis: run linters and static analysis tools (SonarQube, ESLint, Checkmarx) in the IDE and the CI pipeline, catching code quality and security issues before code review.
  • Contract testing: use tools like Pact to verify API contracts between services independently, without requiring a full integration environment.
  • BDD and specification by example: write executable specifications (Cucumber, SpecFlow) that serve as both requirements documentation and automated acceptance tests.
  • Threat modelling: run lightweight threat models during design, not after deployment. In the DevSecOps transformation, we embedded threat modelling into the definition of ready for any feature involving data flows or authentication changes.

Example: shift-left security in financial services​

One of the most impactful shift-left initiatives involved embedding SAST (Static Application Security Testing) directly into the developer's IDE. Previously, security scans ran in a separate pipeline stage, and findings arrived days after the code was written. Developers had already moved on to other work and had lost the mental context. By moving the scan to the IDE (using Checkmarx or Semgrep plugins), developers saw security findings in real time, alongside their compiler warnings. The fix rate for critical findings improved from under 40% to over 85% within six months.

Claude Code hooks push this further than a scanner plugin can: a hook that runs on every edit, not just on save or commit, gives feedback before the developer has mentally moved past the change, and because it is an agent reading the diff in context rather than a pattern-matcher, it can propose the fix rather than only flag the line.

Monitoring and observability​

Monitoring tells you when something is broken. Observability tells you why. As Charity Majors, Liz Fong-Jones, and George Miranda argue in Observability Engineering, traditional monitoring based on predefined dashboards and alerts is insufficient for modern distributed systems. You need the ability to ask arbitrary questions of your production telemetry without having anticipated the question in advance.

The three pillars of observability are:

  • Logs: Structured, contextual event records. In banking, every log entry must be correlated to a transaction ID for audit purposes.
  • Metrics: Numeric time-series data (latency percentiles, error rates, throughput). These power your SLOs and error budgets.
  • Traces: Distributed traces that follow a request across service boundaries. When a payment transaction crosses six microservices, traces are the only way to understand where latency is being introduced.

Working practices​

  • Log aggregation: collect logs from various sources for analysis using tools like the ELK Stack (Elasticsearch, Logstash, Kibana) or Splunk.
  • Metrics collection: gather system performance metrics such as response times, error rates, and saturation, using Prometheus, Datadog, or similar platforms.
  • Distributed tracing: instrument services with OpenTelemetry to produce traces that can be analysed in Jaeger, Zipkin, or Honeycomb.
  • Alerting on SLOs: alert on Service Level Objectives rather than arbitrary static thresholds. An alert that fires when you are burning through your error budget at an unsustainable rate is far more actionable than an alert that fires when CPU exceeds 80%.
  • Agent trace capture: record the files, sources, tools, commands, memory entries, and model fallbacks an agent used before it produces a recommendation or takes action.
  • AI-serving SLOs: measure LLM platform health directly: queue depth, prefill latency, decode latency, timeout rate, KV-cache pressure, context-cache hit rate, and fallback activation.

Example: observability in a payment system​

We operated a real time payment processing system that handled millions of transactions daily. Traditional monitoring would alert us when error rates exceeded a static threshold. The problem was that error rates fluctuated naturally with traffic patterns, a 1% error rate at 3am with low volume was catastrophic, while a 1% rate during peak hours might be within normal variance.

We moved to SLO-based alerting using burn-rate windows. The system calculated how quickly we were consuming our monthly error budget and alerted only when the burn rate indicated we would breach the SLO before the end of the window. This reduced alert noise by over 70% while simultaneously catching genuine incidents faster.

Example: log aggregation in regulated environments​

In banking, log aggregation is not just an engineering practice, it is a regulatory requirement. The challenge is that compliance teams need immutable, tamper-evident log storage with retention periods measured in years, while engineering teams need fast, queryable access to recent logs. We implemented a tiered architecture: hot storage in Elasticsearch for the last 30 days (fast queries for incident response), warm storage in object storage for 6 months (compliance queries), and cold archival for 7 years (regulatory retention).

What fast feedback changes​

Fast feedback changes both delivery speed and operational risk:

Defects get caught early, when they are cheap to fix; the IBM Systems Sciences Institute found that defects found in production cost 6x more to fix than those found during implementation. Mean Time to Recovery drops, and elite performers in the DORA research achieve MTTR under one hour. Systems become more stable because anomalies are detected before they cascade into customer-facing incidents. In regulated environments there is a further benefit: the ability to demonstrate rapid detection and remediation is itself a control, and regulators want to see that you can detect and contain issues, not just that you prevent them. For AI-assisted work, teams can distinguish a good AI recommendation from a lucky one by inspecting retrieval quality, trajectory completeness, and human override history.

Tools for fast feedback​

The toolchain spans delivery pipelines and production telemetry:

  • Jenkins: An open-source automation server for continuous integration and continuous delivery (CI/CD).
  • GitHub Actions: A CI/CD tool that allows you to automate workflows directly from your GitHub repository.
  • ELK Stack: A set of tools for log aggregation and analysis, including Elasticsearch, Logstash, and Kibana.
  • Prometheus: An open-source monitoring and alerting toolkit, purpose-built for reliability and dimensional data.
  • Grafana: An open-source platform for monitoring and observability, used to visualise metrics collected by Prometheus and other sources.
  • OpenTelemetry: A vendor-neutral observability framework for generating, collecting, and exporting telemetry data (traces, metrics, logs).
  • Honeycomb: A modern observability platform built on high-cardinality, high-dimensionality data, designed for debugging distributed systems.
  • Jaeger / Zipkin: Open-source distributed tracing systems for monitoring and troubleshooting microservice architectures.

References​

  1. Forsgren, N., Humble, J., & Kim, G. (2018). Accelerate: The Science of Lean Software and DevOps. IT Revolution Press. The definitive research on what drives software delivery performance, including the statistical link between feedback speed and organisational outcomes.

  2. Majors, C., Fong-Jones, L., & Miranda, G. (2022). Observability Engineering: Achieving Production Excellence. O'Reilly Media. Establishes the distinction between monitoring and observability, and provides practical guidance on instrumenting modern distributed systems.

  3. Beyer, B., Jones, C., Petoff, J., & Murphy, N.R. (2016). Site Reliability Engineering: How Google Runs Production Systems. O'Reilly Media. Introduces SLOs, error budgets, and structured approaches to production reliability at scale. Available at sre.google/sre-book.

  4. Humble, J. & Farley, D. (2010). Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation. Addison-Wesley. The foundational text on deployment pipelines and the mechanics of fast, reliable software delivery.

  5. Kim, G., Humble, J., Debois, P., & Willis, J. (2016). The DevOps Handbook. IT Revolution Press. Practical implementation guidance for the Three Ways of DevOps, including amplifying feedback loops.

  6. arXiv. (2026). "Agent Trajectories as Programs." arXiv:2606.16988. https://arxiv.org/abs/2606.16988

  7. arXiv. (2026). "Agent Memory: Characterization and System Implications of Stateful Long-Horizon Workloads." arXiv:2606.06448. https://arxiv.org/abs/2606.06448

  8. arXiv. (2026). "RTP-LLM: High-Performance Alibaba LLM Inference Engine." arXiv:2605.29639. https://arxiv.org/abs/2605.29639