Declarative AI Pipelines: Architecting a Decoupled, YAML-Driven LLM Orchestration Engine

By Michael Mifsud

The Brittle Chain Problem

LLM frameworks like LangChain made it incredibly easy to start chaining prompts together. However, as AI applications migrate from experimental scripts to production systems, a major structural flaw emerges: brittle coupling.

In typical Python-based LLM architectures, the execution flow, the tool invocations, the prompt templates, and the data-extraction logic are deeply intertwined in procedural code. A simple change—such as modifying a prompt template, swapping out a web scraper tool, or re-sequencing an intermediate formatting step—requires a full code deployment and triggers a cascading wave of regression tests.

To build resilient, maintainable AI applications, we must separate pipeline orchestration (the structure of the workflow) from implementation details (the execution of individual tools and LLM calls).

We solved this coupling problem by architecting langchain-on-steroids: a decoupled, enterprise-grade declarative AI pipeline engine in Python that choreographs sophisticated multi-step workflows through simple, version-controlled YAML configuration files.

Architecture: The Declarative Engine

By adopting Domain-Driven Design (DDD), we isolated the execution logic inside a dedicated Orchestration Bounded Context. At the center of this architecture sits the DeclarativeEngine, which serves as the Aggregate Root coordinating steps, tools, and dynamic prompt templating.

Rather than binding functions together using rigid signatures, the engine models execution using an Abstract State Context Dictionary. The context dictionary maintains a single source of truth that is fed top-to-bottom through the pipeline sequence:

context = {
    "input": initial_input,
    "parameters": global_parameters,
    "options": step_specific_options
}

This stateful dictionary approach ensures that tools and LLM prompt steps can be updated or swapped transparently without modifying a single line of orchestration code.

The Decoupled Execution Model

LLM Bounded Context

Registry Bounded Context

Declarative Pipeline Engine (DDD Aggregate Root)

Input Interface

Safe Load config

Execute

Initializes

Step 1: 'tool'

Bypasses blocks & returns raw HTML

Step 2: 'tool'

Strips DOM scripts/CSS bloat

Step 3: 'llm_step'

Render prompt_template with input

Generate structured markdown

Final Output

linkedin_job_pipeline.yaml

Initial Payload (e.g., Job URL)

DeclarativeEngine

State Context Dict

- input

- parameters

- options

TOOL_REGISTRY

(Decorator-Registered Plugins)

fetch_raw_html

extract_html_text

execute_dynamic_llm_step

Target LLM (Claude / Gemini)

Clean Markdown Specification

Modeling Workflows in YAML

The pipeline’s sequence is completely decoupled from the runtime codebase. For example, to scrape, parse, and analyze a complex job posting, we map the entire execution path inside a single, declarative YAML configuration:

workflow:
  name: "LinkedIn Job Specification Extractor"

  # Global canvas variables to feed contextual data into steps
  parameters:
    target_job: "https://www.linkedin.com/jobs/view/4012345678"
    focus_keyword: "Python AI Engineer"
    geo_location: "London, UK"

  # Sequential execution track (Data pipelines top-to-bottom)
  steps:
    - name: "network_fetcher"
      type: "tool"
      target: "fetch_raw_html"  # Bypasses blocks and streams response

    - name: "dom_element_cleaner"
      type: "tool"
      target: "extract_html_text"  # Wipes script/CSS token bloat

    - name: "job_specification_mapper"
      type: "llm_step"
      # The output from the cleaner step becomes the dynamic {input} below
      prompt_template: |
        You are an expert technical recruiter analyzing a raw structural job posting payload.
        
        Using the provided semantic element text, perform the following tasks:
        1. Extract the core job title, company name, and location.
        2. Isolate and list all mandatory 'Skills' (languages, frameworks, tools).
        3. Summarize the hard 'Requirements' (experience level, certifications).
        
        Format the entire final response as a clean, highly scannable Markdown specification block.

        Job Content Payload Elements:
        {input}

Technical Implementation Details

1. Dynamic Tool Registry

To prevent tools from being tightly coupled to the engine, we designed a decorator-based registration system:

# src/orchestration/registry.py
from typing import Callable, Dict, Any

TOOL_REGISTRY: Dict[str, Callable[Dict[str, Any](/posts/Dict[str, Any/), Any]] = {}

def register_tool(name: str):
    def decorator(func: Callable[Dict[str, Any](/posts/Dict[str, Any/), Any]):
        TOOL_REGISTRY[name] = func
        return func
    return decorator

Individual tools are declared as independent modular plugins. For example, the HTML-to-text extractor isolates DOM cleaning logic entirely:

# src/extraction/html.py
from src.orchestration.registry import register_tool

@register_tool("extract_html_text")
def extract_html_text(context: dict) -> str:
    raw_html = context["input"]
    # ElementParser strips away script tags, styles, and empty spaces
    clean_text = clean_dom_elements(raw_html)
    return clean_text

2. Sequential Execution Loop

The engine parses the workflow and executes steps linearly. Because the context is updated after each step, the output of step $N$ seamlessly becomes the input for step $N+1$:

# src/orchestration/engine.py
for step in self.steps:
    step_type = step.get("type")
    step_name = step.get("name")

    if step_type == "tool":
        target = step.get("target")
        if target not in TOOL_REGISTRY:
            raise ValueError(f"Tool '{target}' is not registered.")

        # Pass the unified context down to the decoupled tool
        context["options"] = step.get("options", {})
        context["input"] = TOOL_REGISTRY[target](context)

    elif step_type == "llm_step":
        prompt_template = step.get("prompt_template")
        # Render the template dynamically and execute the gateway call
        context["input"] = execute_dynamic_llm_step(prompt_template, context["input"])

Verifying Robustness via Behavior-Driven Development (BDD)

To prove that the orchestrator behaves deterministically across different schemas, we wrote BDD integration tests using Behave. This tests the engine’s loading and execution capabilities without coupling our assertions to a live, non-deterministic LLM output:

Feature: Declarative Pipeline Execution Engine
  As a system architect
  I want to orchestrate AI components using a YAML configuration file
  So that workflows can be updated declaratively without changing code

  Scenario: Execute a minimal tool and LLM configuration sequence
    Given the engine initializes with the blueprint "config/minimal_pipeline.yaml"
    When the engine processes the initial text string "Hello World"
    Then the input passes through the configured uppercase utility tool
    And the modified text is successfully processed by the dynamic LLM prompt step
    And the final output is returned as a valid string response

The Operational Yield

By shifting from procedural chaining code to a configuration-driven pipeline architecture, we achieved major operational improvements: