Designing for Operational Clarity: A Reusable Terraform and AWS Gateway

By Michael Mifsud·12 July 2026

The Edge Gateway Challenge

Ingesting telemetry data from distributed edge devices requires a stable, highly available, and secure entry point. When designing a “Telemetry Gateway” to act as this ingestion edge server, the infrastructure must be reliable, repeatable, and easily managed by small operations teams.

Using an AWS Lightsail instance offers a lightweight, cost-effective alternative to a full EC2 and VPC setup. However, moving this configuration into an automated, stateless pipeline reveals several operational challenges:

• Resource Volatility: How do we prevent automated infrastructure teardowns (terraform destroy) from accidentally destroying a critical, hardcoded static IP address?
• Provisioning Race Conditions: How do we coordinate port and firewall creation when the underlying virtual machine takes several seconds to boot and register?
• Rate-Limit Exhaustion: How do we automate SSL certificate generation via Let’s Encrypt in CI/CD without hitting API rate limits during repeated runs?
• Silent Failures: How do we detect, diagnose, and recover from runtime service failures without demanding constant operator intervention?

This article explores how a reusable Terraform infrastructure project was designed from the ground up to solve these problems through deployment safety, idempotent provisioning, and operational clarity.

Architectural Blueprint

The system splits responsibilities clearly between Infrastructure as Code (IaC) using Terraform, continuous delivery using a multi-stage GitLab CI/CD pipeline, and server configuration using an idempotent, Docker-based provisioning payload.

1. Deployment Safety by Architecture

Deployment safety is not merely a matter of checking code syntax. It must be built directly into the resource lifecycle, state management, and orchestration stages.

Decoupling the Static IP

A classic failure mode in automated environments is the accidental destruction of public entry points. If a static IP is managed within the core Terraform configuration, running terraform destroy releases the IP back to AWS’s pool, breaking edge device configurations.

To prevent this, the static IP is decoupled entirely from Terraform’s lifecycle. It is provisioned and managed manually in the AWS Console, and passed to the CI/CD pipeline as an environment variable ($TF_VAR_STATIC_IP). Terraform manages only the compute instance and firewall rules. If the instance is torn down, the static IP remains stable, ready to be attached to the next provisioned instance.

Race Condition Prevention via time_sleep

When provisioning AWS Lightsail compute instances, the virtual machine is created and reported as active by the AWS API before its internal virtualization hypervisor has fully registered the instance’s firewall subsystem. If Terraform attempts to create the firewall port mappings immediately after instance creation, the API returns a race condition error.

To solve this, the configuration introduces a deliberate, declarative delay using the hashicorp/time provider:

resource "time_sleep" "wait_30_seconds" {
  depends_on      = [aws_lightsail_instance.telemetry_server]
  create_duration = "30s"
}

resource "aws_lightsail_instance_public_ports" "web_ports" {
  instance_name = aws_lightsail_instance.telemetry_server.name
  depends_on    = [time_sleep.wait_30_seconds]

  port_info {
    protocol  = "tcp"
    from_port = 80
    to_port   = 80
  }
  port_info {
    protocol  = "tcp"
    from_port = 443
    to_port   = 443
  }
  port_info {
    protocol  = "tcp"
    from_port = 22
    to_port   = 22
  }
}

This ensures port configuration is executed only after the instance is stabilized, removing timing flakes from the pipeline.

State Locking and Isolation

To prevent team members from stepping on each other’s deployments, the pipeline leverages GitLab’s built-in HTTP Backend for remote state storage.

Crucially, the pipeline dynamically isolates states per Git branch using the commit slug (${CI_COMMIT_REF_SLUG}):

TF_STATE_NAME: ${CI_COMMIT_REF_SLUG}
TF_HTTP_ADDRESS: "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/terraform/state/${TF_STATE_NAME}"

This guarantees that feature branches maintain separate, isolated environments. Concurrent runs on the same branch are protected by native HTTP locking and unlocking endpoints (POST / DELETE), preventing state file corruption.

2. The Verification Pipeline

The GitLab CI/CD pipeline follows a strict, progressive flow designed to catch failures early and ensure that what is applied matches exactly what was reviewed.

Ephemeral Integration Testing

On every commit, before generating a plan for the target branch, the pipeline executes the terraform_test job. This job provisions a temporary, isolated test instance utilizing an ephemeral state file:

terraform_test:
  stage: test
  variables:
    TF_STATE_NAME: "test-state-${CI_COMMIT_REF_SLUG}"
    TF_VAR_instance_name: "test-telemetry-${CI_COMMIT_REF_SLUG}"
  script:
    - terraform -chdir=terraform test
    - chmod +x scripts/test-provision.sh
    - ./scripts/test-provision.sh

By executing automated integration tests (terraform test) and a dry-run provisioning script (test-provision.sh) on actual, temporary cloud resources, changes are fully validated on live AWS infrastructure before any production-grade configurations are altered.

Immutable Plans

To prevent configuration skew during execution, the pipeline separates planning and application:

1. The automated plan stage generates an execution plan and saves it as an immutable artifact: terraform plan -out=tfplan.
2. The manual apply stage relies entirely on this artifact: terraform apply -auto-approve tfplan.

This ensures that only the exact changes inspected during the planning phase are applied, protecting against race conditions where the state or AWS environment might change mid-run.

3. Idempotent Provisioning & SSL Lifecycle

Once the infrastructure is ready, the pipeline executes a manual provision_edge job. Rather than depending on heavy configuration management tools, the server is configured via an idempotent bash script (provision.sh) piped over SSH.

Robust SSH Handshake

Stateless CI/CD runners often execute the provisioning job before the server’s SSH daemon is ready to receive commands. The pipeline implements a highly resilient alpine-based handshake retry loop:

echo "⏳ Waiting for Ubuntu to wake up on $TF_VAR_STATIC_IP..."
for i in $(seq 1 12); do
  ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ConnectTimeout=5 ubuntu@$TF_VAR_STATIC_IP "echo '✅ SSH is alive!'" && break || (echo "Wait... ($i/12)" && sleep 10)
done

This wait-and-retry mechanism protects the pipeline from immediate connection dropouts.

The Dual-Phase SSL Bootstrap

To obtain Let’s Encrypt certificates, the server must pass the HTTP-01 ACME challenge, which requires Let’s Encrypt to make an HTTP request to the server on port 80. However, a production Nginx setup requires SSL certificates to exist before it can start on port 443.

The provision.sh script resolves this chicken-and-egg problem using a dual-phase bootstrap:

The script verifies if certificates exist using a symbolic link check (sudo test -L "$CERT_PATH/fullchain.pem"). If they are missing, it spins up a minimal port-80 bootstrap configuration, runs the Certbot docker container, obtains the certs, and then overwrites the configuration with the full SSL setup.

Avoiding Staging Rate Limits

Let’s Encrypt enforces strict weekly rate limits on production certificates. To keep development and test pipelines running safely, the script checks the branch context using the USE_PROD_SSL environment variable:

CERTBOT_FLAGS="--non-interactive --agree-tos --email $EMAIL"
if [ "$USE_PROD_SSL" != "true" ]; then
    echo "🧪 Non-production mode detected. Using Let's Encrypt STAGING environment to avoid rate limits."
    CERTBOT_FLAGS="$CERTBOT_FLAGS --staging"
fi

This simple check restricts production certificates to the main integration branch, while feature branches utilize the staging environment to test provisioning safely.

4. Operational Clarity: The “Hammock Rule”

Once deployed, the Telemetry Gateway must remain visible and maintainable. This project institutes strict operational boundaries and an automated forensic diagnostic standard known as The Hammock Rule.

Transparent Port Exposure

In line with container best practices, all internal services defined in the orchestration blueprints expose their standard ports directly to the host (e.g., PostgreSQL on 5432, Content Repositories on 8080). This design decision ensures that the container stack is entirely transparent, allowing operator diagnostics, testing, and routing checks (check-routing.sh) to execute locally without hiding the stack behind a network abstraction layer.

The Hammock Rule: Autonomous Log Audits

If a deployed application service returns an unexpected 404 Not Found or 500 Internal Server Error, or if a database validation check fails, operations teams are often forced to manually inspect individual containers to diagnose the error.

The Hammock Rule is an emergency protocol that automates this forensic diagnostic loop. Whenever a service monitoring check fails, the gateway immediately runs a targeted log grep before calling for operator support:

docker logs --tail 1000 alfresco | grep -iE "SEVERE|ERROR|Exception|startup failed"

The system analyzes the log output (e.g., catching common failure modes like connection timeouts or missing database keystores) and attempts to resolve the issue autonomously using the predefined service blueprints before escalating.

Conclusion: Reusability is Operational

A truly reusable infrastructure project is not just a collection of Terraform files. It is an operational system designed to survive the realities of stateless execution, unstable APIs, network lag, and rate limits.

By decoupling critical state (the Static IP), isolating environments cleanly in GitLab, structuring a progressive verification pipeline, and designing provisioning scripts to be completely idempotent, this Telemetry Gateway demonstrates how cloud systems can be built to be safe, self-documenting, and operationally resilient.