AI-Native Observability | Build Software That AI Agents Can Debug Too!
AI coding agents let us develop software very quickly.
But after deploying that software to production, how do we identify and resolve the issues that appear when real users start using it?
This is particularly important when building durable software that needs to be maintained for a long time.
This is where observability matters.
What is observability?
Put simply, observability is the ability to understand what is happening inside a production system, and why, using the data the system produces.
There are three main pillars:
- Metrics: Measure things such as error rates, latency and system performance.
- Logs: Provide details about events and errors that occurred within the system.
- Traces: Show how a request travels across multiple services.
Collecting these types of data separately is not enough.
This is where wide events become useful: recording many details about a request in a single structured event, such as its service, trace ID, duration, error, deployment version and business context.
Connecting and examining these details makes it easier to identify where a problem occurred.
What is an SLO (service level objective)?
CPU and memory usage alone cannot tell us how healthy a system is. The reliability of the service users receive matters too.
For example, we could define a target that 99.5% of checkout requests must complete successfully within one second over a month.
That is an SLO. Targets like this help us identify issues that affect the actual user experience and prioritize alerts.
Where do AI agents fit in?
Imagine an issue occurs in a regular production application at 2 a.m.
An AI agent with access to observability data, the code repository and deployment history could:
- Detect the alert and analyze logs, metrics and traces.
- Examine code changes and dependencies to try to identify the root cause.
- Propose a fix and run tests and validation.
The concepts of one-way and two-way doors, and blast radius from an earlier article matter here.
For changes that are low risk and easy to reverse, an agent could be allowed to deploy a fix and notify the engineer, subject to predefined safeguards, automated validation and rollback mechanisms.
For high-risk changes, it could prepare the root cause analysis, evidence and proposed fix, then request human approval.
The engineer can then make a decision with the necessary context, instead of starting the investigation from scratch. Critical incidents still require human escalation.
What should software engineers focus on?
Observability should be considered from the beginning of software design, rather than added only after a production issue occurs.
Observability-driven development means planning structured telemetry, correlation IDs, meaningful alerts, SLOs and the necessary business context early.
AI-native software engineering extends beyond generating code with AI.
It also includes designing software so that both AI and humans can understand the production system, identify issues and resolve them safely.
This article focuses on production observability for regular deterministic software, and using AI agents to help resolve its issues. In a future article, we will discuss agentic AI observability: observing the behavior, decisions and tool calls of non-deterministic AI agents.