LangChain on Steroids with BDD: When a Feature Becomes Executable Evidence

By Michael Mifsud

The Pipeline Already Worked

In my previous post, Declarative AI Pipelines: Architecting a Decoupled, YAML-Driven LLM Orchestration Engine, I detailed how we solved the brittle coupling of traditional LLM chains by creating langchain-on-steroids (LOS). We isolated orchestration logic into a dedicated context. By introducing a centralized DeclarativeEngine as our Domain-Driven Design (DDD) Aggregate Root, a decorator-registered tool registry, and a stateless context dictionary, we proved that complex, multi-step LLM workflows could be modified, swapped, or re-sequenced entirely in YAML without deploying new Python orchestration code.

The pipeline worked. But as any software engineer knows, a system that “works” on a developer’s machine or passes a few hand-crafted execution scripts is only half-built. To confidently maintain and distribute an orchestration library, we need more than manual execution checks; we need a testing suite that acts as verifiable behavioral evidence.

More importantly, we need to answer a deeper architectural question: Are our specifications actually proving system behavior, or are they merely passive documentation that looks executable?

But Were the Features Really Executable?

The central argument of this evolution is simple:

A behavioral specification is not executable evidence merely because a .feature file and some step definitions exist.

Previously, our codebase included Gherkin feature files and step definitions written for the Behave testing framework. On the surface, the presence of these files gave the impression of active Behavioral-Driven Development (BDD). However, their existence did not prove that our test suites actually collected or ran them during standard local development or continuous integration.

In practice, we found that Behave step definitions could sit silently in our directories without any guarantee that Pytest collected them. It is incredibly easy to fall into the trap of writing beautiful Gherkin scenarios that describe “system behavior” but rot because they are never hooked into the active test runner. They become documentation that looks executable, failing to provide actual evidence of correctness.

The migration to pytest-bdd forced a critical distinction. Under pytest-bdd, shared step definitions in a conftest.py file do not collect feature scenarios by themselves. Each scenario must be explicitly bound to the Pytest execution lifecycle. This is achieved through scenario bindings using either @scenario decorators or scenarios block declarations:

from pytest_bdd import scenarios

# Explicitly bind all scenarios within the target feature file
scenarios("features/declarative_pipeline.feature")

This structural bind establishes a verifiable chain of custody:

Feature Specification

(.feature file)

Scenario Binding

(scenarios() or @scenario)

pytest-bdd Steps

(Given / When / Then)

Real DeclarativeEngine

(Abstract Context Execution)

Controlled External Boundaries

(Mocks: requests, LLM)

Observable Behavioural Evidence

(Assertions)

Without this binding, there is no execution. Pytest will simply skip over the Gherkin files, leaving your specifications as nothing more than glorified markdown comments.

Clarifying the Feature Boundaries

Before we could run these scenarios deterministically, we had to clean up our feature boundaries. A major smell in our original BDD layout was the feature file generalized_network_extraction.feature.

The name implied a generic, highly abstract network crawling capability. Yet, when we inspected the actual step definitions and configuration options, the behavior was tightly coupled to scraping LinkedIn job boards and parsing job postings with predefined templates and regex patterns.

To make our boundaries honest, we refactored this layout:

  1. generalized_network_extraction.feature was renamed to parameterized_job_network_extraction.feature. This made it explicit that the behavior being exercised was a platform-agnostic, parameter-driven job-network pipeline, not a generic, raw crawler.
  2. direct_url_network_extraction.feature was introduced as a separate, clean contract. This feature defines the behavior for retrieving raw content directly from any valid, absolute URL.

The architectural value of separating Generic Direct URL Retrieval from Parameterized Job-Network Extraction is substantial. It ensures that our core network transport logic remains highly reusable and completely decoupled from specialized scraping strategies. We avoid contaminating langchain-on-steroids with the structural requirements of downstream consumer projects, keeping our boundaries clean and our library generic.

Collection Is Part of the Test

With our features clarified and bound to pytest-bdd, we established a strict verification process. We checked collection independently of execution:

poetry run pytest --collect-only -q

Why check collection first? The objective was not simply to secure a green test run. It was to prove that:

Our independent collection check confirmed that 11 BDD/E2E tests were successfully gathered. Running the suite proved that all 11 BDD tests passed successfully.

By ensuring that the test runner explicitly collects every scenario, we convert passive Gherkin text into active, executable evidence. If a step definition is missing or a scenario is unbound, the collection step flags it immediately, preventing silent test bypasses.

Real Capability, Controlled Boundaries

A common failure in behavioral testing is the tendency to mock too much. When integration tests mock the system under test, they become tautological—they prove only that your mocks behave like your mocks, not that your production engine actually works.

To prevent this, our BDD integration tests run the real capability of langchain-on-steroids. We execute the actual DeclarativeEngine and allow it to coordinate real tools and flow data top-to-bottom. We do not mock our own orchestration loop, our tool registration mappings, or our DOM parsing logic.

Instead, we control only the external, non-deterministic boundaries of the system:

Feature Scenario
       ↓
pytest-bdd Scenario Binding
       ↓
Given / When / Then Step Definitions
       ↓
Real DeclarativeEngine execution
       ↓
Controlled External Boundary (Mocks: requests.Session.get & ChatOpenAI.invoke)
       ↓
Observable Result

By intercepting outgoing connections at requests.Session.get and modeling LLM responses inside ChatOpenAI.invoke, we run the entire codebase under pure local conditions. This is fundamentally different from mocking the system under test. Controlling the external boundary preserves the execution path of our internal components, while replacing the non-deterministic external networks with reliable, reproducible local test fixtures.

BDD and Unit Tests Answer Different Questions

While BDD integration tests prove that our features behave as promised, they are not a replacement for direct unit-test evidence. They answer fundamentally different questions.

We do not write a BDD scenario for every edge case; doing so would bloat our feature files and slow down our high-level integration loops. Instead, we maintain a strict separation. Our high-level behaviors are proven by 11 BDD scenarios, while our low-level code robustness is verified by a separate unit-test suite located under tests/unit/.

One Test per Module Is Only a Baseline

To ensure our namespaced package layout (src/langchain_on_steroids/) has robust coverage, we expanded our direct unit-test suite. We treated “one test per module” as a minimum visibility baseline—not as proof of test quality, but as a starting point to ensure every module has a corresponding test file.

Our production package is structured cleanly:

src/langchain_on_steroids/
├── __init__.py
├── extraction/
│   ├── __init__.py
│   ├── html.py
│   ├── network.py
│   └── strings.py
├── orchestration/
│   ├── __init__.py
│   ├── engine.py
│   └── registry.py
└── validation/
    ├── __init__.py
    └── prompt_rules.py

By expanding our unit tests, we mapped every single one of these 10 production modules to direct unit-test files (such as test_html.py, test_network.py, test_strings.py, test_engine.py, test_registry.py, test_prompt_rules.py, and package import boundary assertions in test_init.py).

Running this direct unit-test suite independently collected and executed:

When we combine our unit tests and our BDD integration tests under a single pytest command, the test runner collects and executes:

This dual-layer testing architecture ensures that we have fast, deterministic unit feedback on our isolated functions, backed by comprehensive behavioral validation of our declarative engine.

Introducing Shared Governance Agents

Writing clean code and comprehensive tests is a continuous battle. To help us maintain our standards, we introduced the concept of shared governance agents.

The Code Smell Auditor is a specialized AI agent designed to review code for signs of unnecessary complexity, poor separation of responsibility, weak boundaries, duplication, and excessive coupling. Crucially, the auditor does not belong solely to the langchain-on-steroids project. It resides in the shared agentic environment, exposed to our project through a symbolic link:

.agent/governance-agents/code-smell-auditor/SKILL.md

This shared approach ensures that reusable engineering governance travels across repositories, rather than being reinvented or copied into every codebase.

We did not want to present the auditor as an infallible, all-knowing authority, nor did we want to bloat its instructions with a massive, rigid ruleset from day one. Instead, we kept its initial skill minimal. We wanted to answer an experimental question:

What does the agent already know, and what do we genuinely need to teach it?

We ran the auditor with a completely open instruction: “Go smell the code.” We did not give it a predefined answer sheet. We let it analyze our production package and unit tests independently to see what architectural risks it could identify.

Go Smell the Code

The Code Smell Auditor performed an independent review and highlighted 5 non-blocking diagnostic signals:

Finding 1: Circular/Implicit Package Coupling via Side-Effect Imports

Finding 2: Global Mutable State Registry

Finding 3: Loss of Diagnostic Detail in Parser Boundaries

Finding 4: Hardcoded Magic Network Timeouts

Finding 5: Heavy Mock Coupling to Internal Chain DSL

The auditor’s report was highly practical. It did not treat these findings as automatic bugs or blocker failures. Instead, it framed them as legitimate engineering trade-offs, helping us understand our structural risks while respecting our conscious design decisions.

Who Audits the Auditor?

While the Code Smell Auditor successfully identified several high-value structural smells, the experiment also highlighted a key truth about automated governance:

Governance agents also need governance.

Comparing the auditor’s findings against an independent expert review exposed several blind spots:

  1. Production-Biased Reasoning: The auditor was highly effective at spotting production-code smells (like global state or side-effect imports) but was significantly weaker at analyzing test-specific smells.
  2. Acceptance of Incidental Assertions: It failed to challenge unit-test patterns that asserted exact collection sizes, or package-docstring wordings used as executable assertions.
  3. Coverage-Driven Testing: It did not raise questions about whether certain tests were written primarily to satisfy our “one test per module” coverage rule, rather than targeting actual, high-risk business behaviors.

These are not proven defects in the auditor; they are valuable indicators of where we need to refine its capabilities. It shows that rather than expanding the auditor’s instructions into a massive, bloated manual, we should evaluate and tune its reasoning over time using an independent framework, such as the SOLID Book Test.

Teach the Agent Only What It Does Not Already Know

Our experimental loop for improving governance agents is designed to prevent instruction bloat:

Minimal Skill

(Baseline instructions)

Run Audit

(Identify code smells)

Independent Evaluation

(Expose blind spots)

Identify Blind Spots

(Compare with expert)

Minimal Skill Update

(Add smallest useful instruction)

By running a minimal skill, comparing its output against human analysis, and adding only the specific missing context, we keep our shared skills lightweight and maintainable. The objective is not to teach an agent everything we know; it is to discover what it already knows and supply only the unique project-level context it lacks.

What This Changes

This engineering cycle has fundamentally changed how we evaluate correctness in langchain-on-steroids:

By treating tests as executable evidence and governance as a shared, continuous agentic skill, we ensure that our declarative AI pipeline engine remains robust, decoupled, and prepared for future consumer scaling.