Building an Evidence Layer for Complex Automation

For a while, one of the frustrating things about my home automation system was that it could be doing exactly what I had designed it to do and still be surprisingly difficult to explain.

The system had evolved from relatively simple automation into something more layered. There were eligibility rules, demand calculations, guards, retries and reconciliation. Manual or external actions could intervene too.

That complexity was there for good reasons. Real environments are messy.

But it created a new problem: when something unexpected happened, the current state stopped telling the story.

Seeing an AC as active told me what the system believed at that moment. It did not necessarily tell me why it was active, who had changed it, whether a retry was involved, or whether the physical device had actually done what the software thought it had done.

A simple question—

Why did this happen?

—could turn into a few hours of investigation.

I would move between traces, state history, controller decisions, metrics and provenance records, lining up timestamps and reconstructing what had happened.

What bothered me was that I kept having to do it again.

At some point, repeatedly becoming faster at manual investigation stopped looking like the right optimization.

So I started building helpers.

Given a question and a time range, they could collect the relevant evidence from several sources, normalize timestamps and put the records into sequence.

That saved time almost immediately.

It also exposed a more interesting problem.

Collecting the evidence did not mean I understood what the evidence was entitled to tell me.

A controller decision could tell me what the controller decided. It could not prove that the physical device complied.

A trace could show which path executed. It could not prove the external outcome.

Recorded state could tell me what an integration believed. That belief could still be stale or wrong.

I found myself asking the same question repeatedly:

What is this evidence actually allowed to prove?

That question changed what I was building.

The helpers evolved from collectors into something closer to an evidence layer. They began preserving not only the records, but where they came from, what period they covered, what evidence should have been present and what was missing.

The output became a machine-readable contract for the investigation.

It described the question, the exact time window, relevant evidence, provenance, expected and observed behavior, known gaps and deterministic findings.

Instead of giving the next stage a pile of logs, I could give it a structured account of the problem.

And then the contract caught me making a mistake.

I asked the investigation tooling what had happened during a specific recent interval. It returned a plausible explanation.

When I inspected the evidence underneath, I noticed that the interval I had asked about and the interval it had actually examined were different.

The answer might still have been correct.

But that was no longer enough.

If I could not tell exactly what evidence the explanation was based on, I could not tell exactly how much confidence to place in it.

So I changed the design.

The requested interval and the resolved interval became explicit. Context around the incident could still be included, but it could no longer quietly become evidence from inside the requested window.

Then I validated the corrected behavior against a real incident interval.

That small correction captured the larger point:

a convincing explanation is only as trustworthy as the evidence contract underneath it.

The same discipline helped with another deceptively simple result: nothing found.

Sometimes “nothing found” meant the investigation ran normally and found no issue.

Sometimes it meant evidence was missing and the investigation could not really evaluate the question.

Those are very different outcomes.

So I separated the health of the investigation machinery from the condition it was trying to diagnose.

The system gained the ability to say, in effect:

I cannot evaluate this reliably yet.

I came to value that answer.

It was much more useful than silently turning missing evidence into certainty.

By this point, I was also changing how I used AI.

The obvious approach would have been to give a model access to a large amount of operational history and ask it to work out what happened.

But by then I understood how much hidden work was contained in that request.

Before anyone could reason about cause, somebody still had to decide which records belonged to the incident, align them correctly, distinguish context from evidence, understand what was missing and know which source was authoritative for which question.

I did not want AI rediscovering those rules every time.

So I pushed as much of that work as possible below the AI boundary.

Collection stayed deterministic. Provenance was explicit. Time-window handling was enforced by code. Conditions that could be expressed clearly stayed in deterministic detectors and verdict logic.

AI received the result of that work.

The question changed from something like:

Here are a lot of logs. What happened?

to:

Here is the precise question. Here is the bounded evidence. Here is what each source represents. Here is what is missing. Given those constraints, which explanations fit best, and what should I check next?

That turned out to be a much better problem for AI.

Consider a simplified, public-safe example.

The controller expects an AC to remain in hold, but the observed state changes to active. There is evidence of external activity near the transition, while part of the guard evidence is missing.

The useful conclusion is not:

The AC turned on because of X.

There is not enough evidence for that.

But “insufficient information” is not very useful either.

A better answer is:

The state changed unexpectedly. External input is plausible, confidence is limited by the missing guard evidence, and these are the records to check next.

Sometimes that was enough to reach a credible root cause.

Sometimes it narrowed the possibilities to a strong hypothesis.

And sometimes the biggest benefit was simply reducing an open-ended investigation to a few precise discovery steps.

That was when the larger payoff became clear.

I could have kept taking one difficult incident at a time to AI and benefiting from each answer.

Instead, I had built the machinery that made the next incident easier too.

The benefit compounded. I was not just getting a faster answer; I was making the next investigation faster.

For some difficult cases, work that had previously taken me a few hours of searching, sequencing and reconstructing evidence could be narrowed to minutes.

Not because AI suddenly understood the system better than I did.

Because I no longer made either of us reconstruct the system from scratch.

That changed my view of where AI is most useful in complex operational work.

Ambiguity is tempting territory for AI, but ambiguity often hides several different problems: unclear scope, incomplete evidence, conflicting sources and assumptions about what a particular record actually proves.

Giving all of that ambiguity to AI does not make it disappear.

What helped was doing almost the opposite.

I became stricter about the question, the time window, provenance, evidence authority and what remained unknown.

Only then did I ask AI to reason.

The breakthrough was not adding AI to debugging.

It was building enough trustworthy machinery around the problem that AI could help me investigate repeatedly without becoming the source of truth.

---

My role

I designed and built the investigation and observability capability around the automation system, using AI-assisted development where it was useful: the evidence-collection helpers, machine-first evidence contract, provenance and time-boundary handling, deterministic interpretation layers, investigation-health semantics, and the boundary between deterministic processing and AI-assisted reasoning.

Why it mattered

For some difficult cases, investigations that had previously required a few hours of manual reconstruction could be narrowed to minutes. When the evidence did not support a root-cause conclusion, the system could still narrow the hypothesis and identify the remaining discovery steps.

The public technical case study contains the deeper architecture, representative evidence packet and implementation detail.