Passing Tests Are Not Proven Behaviour: The Composition Blind Spot in AI-Assisted Engineering

By Michael Mifsud·16 July 2026

The Green Test Mirage

We have built a software engineering workspace that feels incredibly secure. Our project repository, LangChain on Steroids, was designed with what we believed to be a state-of-the-art governance model. We had a ticket-driven workflow managed by an autonomous Engineering Steward, high-level behavioral integration tests driven by Gherkin features and pytest-bdd, a dedicated Code Smell Auditor running against the Python codebase, strict local-liveness timers, and a direct unit-testing baseline targeting a high coverage standard across every production module.

By any standard metrics, the project’s health was impeccable. All tests were green.

Yet, a major production defect still escaped.

The defect centered around output persistence. The system has multiple data-acquisition strategies used to fetch raw documents before passing them to downstream parsers and persistence layers. When executing our standard network fetcher, persistence worked flawlessly. But when we activated a specific, cloud-based acquisition strategy—Firecrawl—the data retrieved was never written to disk. It vanished from the pipeline silently, with no errors, no exceptions, and absolutely no failing tests.

In a traditional development cycle, the immediate, almost instinctive reaction to a bug like this is:

“We need another test.”

We write a quick regression test for Firecrawl persistence, watch it pass, commit the fix, and carry on. But this response misses a much deeper, more uncomfortable architectural lesson.

The problem was not that our code was broken. The problem was that our behavioral specifications were shallow. Every relevant test we had was already passing, but we had never actually proven—or even claimed—that persistence was guaranteed to work across every acquisition strategy. We had only asserted its behavior along one specific execution path.

The green test was a mirage. It was passing not because the system was complete, but because nobody had asked the question.

Component Correctness Is Not Composition Correctness

This blind spot led to a simple, yet vital engineering principle:

Component correctness does not imply composition correctness.

When we broke down the failure, we found that every individual capability in our system was structurally and logically correct.

• The acquisition component worked. It connected to Firecrawl, fetched the document, and returned a payload.
• The persistence component worked. It took an input string, initialized a file target, and wrote the bytes to disk.

Both modules had direct unit tests proving their isolated behavior at their stable public boundaries. However, we had never proven the composition of those two capabilities:

$$\text{Firecrawl Acquisition} \longrightarrow \text{Persistence}$$

We assumed that because the components were individually verified, their combination was safe.

In modern software engineering—especially when using AI coding assistants to generate boilerplate, refactor modules, or write tests—this assumption is highly dangerous. LLMs are exceptional at optimizing isolated blocks of code to satisfy immediate, narrow constraints. They are equally adept at manufacturing mock boundaries to make unit tests pass with minimum effort.

If your testing architecture relies on proving isolated components while ignoring how those components compose under varying strategies, you are not validating system behavior. You are merely validating implementation details in a sandbox.

Untangling the Vocabulary of Behaviour

To reason about these blind spots clearly, we have to establish a sharper vocabulary. In many teams, terms like “feature,” “strategy,” and “implementation” are treated as interchangeable synonyms. This blurring makes behavioral analysis almost impossible.

To bring order to our specifications, we find it useful to separate these concepts into five distinct layers:

1. Capabilities: The high-level promises the system makes to its consumers. They represent the “what.” For example: “Retrieve raw HTML content” or “Persist structured markdown payloads.” Capabilities are strategy-agnostic; they should remain stable even if the entire underlying technology stack is swapped.
2. Strategies: Interchangeable implementations of a capability. They represent the “how.” For example, the capability to retrieve a document can be executed via a standard requests HTTP strategy, a headless browser strategy, or a Firecrawl API strategy.
3. Behavioural Dimensions: Orthogonal variables of behavior that apply across strategies. These are the edge cases, environmental contexts, and constraints—such as network timeouts, rate limits, empty responses, or invalid inputs.
4. Policies: System-wide rules that govern how capabilities and strategies interact. A policy defines the choices made when strategies compose. For example: “If the primary acquisition strategy fails with a rate-limit dimension, fallback to the secondary strategy and always persist the final result.”
5. Implementation Details: The private, inside-the-box code choices (variable names, local loops, internal helpers) that do not affect the public contract. Tests should almost never bind themselves to this layer.

When we review our tests with this vocabulary in mind, the composition blind spot becomes obvious. Our original Gherkin feature promised the Capability of output persistence. But instead of defining a Policy that applied across all acquisition Strategies, the specification bound the capability of persistence exclusively to the standard HTTP strategy.

Because the Gherkin files never declared a policy for the Firecrawl strategy, our coding agents had no behavioral contract to implement, and our testing agents had no contract to assert.

The Feature Smell

A key insight from this incident is that we should not need to read production code or inspect mock configurations to discover these gaps. A well-written, executable specification should contain enough behavioral context to allow an experienced engineer to infer the necessary combinations.

If a feature file claims:

“We persist tool output so that workflows are auditable.”

and the same document lists multiple acquisition strategies, a reviewer should immediately ask:

“Does this capability hold for every strategy listed?”

The absence of that combination in the feature file is not just a missing test; it is a Feature Smell.

Just as code smells indicate underlying structural issues in your Python classes, feature smells indicate logical gaps, unproven compositions, or overreaching claims in your behavioral contracts. Examples of common feature smells include:

• The Strategy Slip: Proving a capability using only the easiest strategy (e.g., a local mock) while claiming the behavior is generic.
• The Silent Path: Defining multiple strategies but only asserting integration and policies on one.
• The Floating Claim: Making a broad, high-level behavioral promise in the Gherkin text that is never backed up by an executable scenario binding or step assertion.

Experimenting with a Feature Smell Auditor

To address these blind spots, we began an engineering experiment. We designed a new, specialized governance agent called the Feature Smell Auditor.

Unlike standard static analysis tools or our existing Code Smell Auditor (which looks for circular package imports, mutable state, or exception-swallowing in our Python files), the Feature Smell Auditor works entirely at the level of behavioral specifications. It analyzes Gherkin .feature files and their corresponding scenario bindings.

It does not act as an infallible, all-knowing authority. Instead, it serves as a skeptical peer that asks difficult, structured questions of our behavioral contracts:

• What capability does this feature file actually claim to prove?
• Which behavioral dimensions are outlined in the text but never exercised by any scenario?
• Which compositions of strategies are left completely unproven?
• Are our capability claims significantly broader than our executable evidence?

By running this auditor against our Gherkin specifications before we write a single line of production code, we can identify gaps in our reasoning early. If the auditor flags that we have defined a “Firecrawl Strategy” and a “Persistence Capability” but have zero scenarios demonstrating their composition, it flags an unproven composition smell. It forces us to resolve our behavioral requirements before we begin implementing them.

The Division of Disciplines: Feature-to-Test

This experiment also forced us to rethink how we delegate tasks to our AI development agents.

Originally, we tasked our primary Python coding agent with both generating the production implementation and translating our Gherkin feature files into pytest-bdd step definitions. It seemed like a highly efficient workflow.

But this, too, was the wrong abstraction.

Writing clean, performant, and maintainable production code is a different discipline from translating a behavioral contract into a rigorous, skeptical suite of assertions. When the same agent is responsible for both, it naturally designs its tests to accommodate its implementation choices. It builds mocks that mirror its internal functions, reinforcing its own blind spots and ensuring that its tests will pass. The test suite ceases to be independent evidence; it becomes a secondary reflection of the production code.

To break this loop, we separated these roles and introduced a dedicated Feature-to-Test Agent.

The responsibility of this agent is not to make the tests pass. Its sole, unyielding mandate is to protect the integrity of the behavioral contract. It reads the feature files, analyzes the strategies and policies, and designs the step definitions and mocks with absolute skepticism.

By separating the Implementation Agent (which writes the production code) from the Feature-to-Test Agent (which translates the behavioral contract), we establish a healthy, adversarial balance. The testing agent remains objective, ensuring that our executable evidence is credible and that our boundaries are honestly defended.

Tomorrow’s Behaviour

As software engineering increasingly relies on autonomous agents to write, refactor, and maintain our systems, our traditional reliance on “passing tests” as a proxy for correctness must evolve. A green test run is only as valuable as the behavioral questions it asks.

If we allow our agents to write code and tests in isolation, without auditing the integrity of our behavioral compositions, we will continue to build systems where every individual component is correct, yet the application itself fails silently in production.

We must shift our focus from code coverage to contract composition. Our governance must start at the behavioral layer, identifying feature smells and proving strategy interactions before implementation begins.

Ultimately, we have to remember the fundamental purpose of our testing suites:

The purpose of a test is not to confirm today’s implementation. It is to prove tomorrow’s behaviour.

When we treat our feature specifications as active, audited, and executable evidence of composition, we stop chasing the mirage of green tests and begin building genuinely predictable systems.