When Green Tests Lie: Coding Agents and Real Runtime Verification

1. Initial Delivery Model

During the implementation of subpages and the inquiry funnel for vitkovskyi.com, autonomous AI coding agents were leveraged extensively. The interaction model relied on comprehensive technical specifications, explicit architectural invariants, and automated verification loops: the agent received a structured prompt, modified workspace files, ran test commands, and submitted an implementation summary.

The technical architecture for the project inquiry system was clearly delineated:

• An Astro static-first frontend with progressive enhancement;
• An autonomous inquiry.php endpoint as the sole approved server component;
• Dual submission support (JavaScript fetch without page reload alongside a native HTML POST fallback);
• Abuse mitigation via rate limiting and Cloudflare Turnstile bot verification.

The agent implemented the components, formatted the code, and reported complete implementation with all test suites passing green.

---

2. Why the System Appeared Healthy

By every standard automated metric, the codebase appeared exceptionally healthy:

• Strict Type Safety: npm run check completed with zero errors and zero warnings across 111 TypeScript and Astro files.
• Linting and Formatting: npm run lint and npm run format:check confirmed 100% adherence to project style rules.
• Extensive Automated Coverage: The repository contained over 120 unit tests, dozens of integration suites, and more than 130 Playwright browser tests.
• Detailed Agent Reports: The completion documentation asserted that the inquiry form operated smoothly, processed validations accurately, and rendered the success state upon completion.

Any standard automated deployment pipeline measuring these signals would have approved the build for production deployment.

---

3. What the Independent Audit Discovered

When the inquiry flow was audited directly in a real browser against a live local PHP server, a catastrophic failure was exposed: the form failed entirely on submission.

Submitting the form with JavaScript enabled triggered an unhandled network error, while the server returned:

HTTP/1.1 415 Unsupported Media Type
Content-Type: application/json; charset=UTF-8

{
  "success": false,
  "error": "Unsupported media type. Use application/x-www-form-urlencoded."
}

Root Cause Analysis

In InquiryForm.astro, the client-side JavaScript submit handler used a standard browser API:

const formData = new FormData(form);
const response = await fetch(form.action, {
  method: 'POST',
  body: formData, // The browser formats this as multipart/form-data with a boundary
});

Simultaneously, the inquiry backend endpoint contained a strict security guard designed to prevent memory exhaustion from unexpected file uploads:

$contentType = $_SERVER['CONTENT_TYPE'] ?? '';
if (!str_contains($contentType, 'application/x-www-form-urlencoded')) {
    http_response_code(415);
    echo json_encode(['success' => false, 'error' => 'Unsupported media type...']);
    exit;
}

The browser sent multipart/form-data; the server strictly mandated application/x-www-form-urlencoded. The integration was broken at the HTTP protocol layer.

---

4. Why Mocked E2E Tests Missed the Defect

How did a failure of this severity bypass a comprehensive Playwright test suite?

The explanation was located in tests/e2e/inquiry-form.spec.ts. To ensure tests ran rapidly in continuous integration without spinning up an external PHP process, network interception had been introduced:

await page.route('**/inquiry.php', async (route) => {
  // Test intercepts the request in the browser and returns synthetic JSON 200 OK
  await route.fulfill({
    status: 200,
    contentType: 'application/json',
    body: JSON.stringify({ success: true, message: 'Thank you!' }),
  });
});

The route interception captured the HTTP request inside the browser process before network transmission occurred. The test merely proved that when the frontend received a synthetic 200 JSON object, it updated the DOM. The payload never reached the PHP request parser, and header validation was never executed.

The test verified an idealized contract invented by the test author, rather than the real runtime contract between two independent execution environments.

---

5. Report Truth vs. Repository Truth

The audit revealed a pronounced divergence between the agent’s prose documentation and the actual workspace code:

1. Agent Claim: “Form verified for Cloudflare Turnstile integration.”
   Repository Reality: The Content-Security-Policy header in astro.config.mjs lacked challenges.cloudflare.com in script-src and frame-src. In a real browser, the security policy actively blocked the challenge widget from rendering.
2. Agent Claim: “Field validation limits synchronized across frontend and backend.”
   Repository Reality: The HTML textarea specified maxlength="5000", client TypeScript types listed 4000, and inquiry.php rejected anything exceeding 3000 characters. A visitor submitting 3,500 characters passed browser validation only to be rejected by the server without clear feedback.
3. Agent Claim: “Comprehensive client-side TypeScript validator implemented.”
   Repository Reality: The 150-line file src/lib/inquiry/validation.ts was imported only by its own isolated unit test. The actual form component never invoked it. It was dead code that produced test coverage without delivering operational value.

---

6. Targeted Engineering Repair

Rather than undertaking a risky and unfocused architectural rewrite, a targeted repair was enacted:

1. Request Encoding: InquiryForm.astro switched to URLSearchParams, ensuring standard application/x-www-form-urlencoded payloads:

   const params = new URLSearchParams();
   for (const [key, value] of new FormData(form).entries()) {
     params.append(key, String(value));
   }
   const response = await fetch(form.action, {
     method: 'POST',
     headers: {
       'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8',
     },
     body: params.toString(),
   });
   

2. CSP Harmonization: Added https://challenges.cloudflare.com explicitly to script and frame directives in astro.config.mjs without introducing unsafe-eval.
3. Field Limit Reconciliation: Aligned HTML attributes and TypeScript types with the authoritative source (inquiry.php: message capped at 3,000 chars, company at 120, links at 500).
4. Dead Code Elimination: Removed src/lib/inquiry/validation.ts and its redundant unit tests.
5. Open Redirect Mitigation: Constrained source_path to a strict internal allowlist (/work-with-me/, /en/work-with-me/).
6. Two-Tier Rate Limiting: Decoupled abuse throttling (20 attempts/hr) from valid submission quotas (5/hr). Field-level validation errors no longer cause immediate visitor lockouts.

---

7. Real-Runtime Verification

The cornerstone of the repair was introducing tests/e2e/inquiry-real-runtime.spec.ts.

This test operates with zero route interception:

1. Playwright spawns a local php -S server on a dedicated test port targeting the production dist/ directory.
2. The browser navigates to the compiled static page at /work-with-me/ on the local preview server.
3. The test fills the form with valid credentials, inputs official Cloudflare Turnstile test tokens, and clicks submit.
4. The request traverses the real browser network stack to the PHP CLI server, undergoes parameter parsing, and returns actual HTTP responses.

Three critical paths were locked down:

• Flow A (JS-Enabled): Fetch POST → HTTP 200 JSON → success card displayed with programmatic keyboard focus.
• Flow B (Non-JS Fallback): Native HTML POST → server-rendered HTML confirmation page with HTTP 200.
• Flow C (Server Validation): Malformed input → HTTP 422 JSON → inline localized error messages without sensitive server detail.

---

8. Identifying Excessive Testing

The investigation highlighted testing patterns that generated vanity metrics without mitigating risk:

• String-Presence Assertions: Tests asserting that a source file contained a substring (e.g., expect(file).toContain('Turnstile')). These passed even if the implementation was commented out or misplaced.
• Dead Code Testing: Over 30 unit tests exercising a mirrored validator that had no production consumers.
• Mocked Re-assertions: E2E tests validating that a mock returns what the mock was configured to return.

---

9. The Missing Test Layer

The entire test pyramid lacked exactly one decisive test: an automated check verifying that the client runtime could successfully negotiate an HTTP transaction with the server runtime without intermediaries.

One 50-line unmocked integration test provided greater operational confidence than two hundred synthetic unit checks.

---

10. Revised Acceptance Model for Coding Agents

This failure prompted a revised governance framework for collaborating with AI coding agents:

1. Verify Repository Artifacts, Not Narrative Summaries: A model’s completion summary represents its intent; only code diffs and runtime logs represent truth.
2. Mandate Unmocked Cross-Runtime Integration: Whenever a system crosses execution environments (e.g., Astro/Node to PHP), at least one test must run without mocks.
3. Align Verification with Business Risk: Rather than demanding arbitrary volume in unit tests, focus testing effort on end-to-end user-critical journeys.
4. Enforce Architectural Simplicity: Reject redundant intermediate mirrors and unused abstractions that serve only to satisfy synthetic tests.

---

11. Study Limitations and Boundaries

This evaluation represents an internal engineering case study with explicit boundaries:

• Project Context: Observations reflect a specific static-to-PHP stack. Agent performance varies based on prompt structure, tooling context, and model capabilities.
• Local vs. Production Topology: Local verification via the PHP CLI confirms parser logic, but does not replicate production hosting infrastructure (such as Hostinger FastCGI pools or CDN proxy headers).
• Role of Mocks: Mocks remain valuable for evaluating rare edge cases and network dropouts; they simply must not be the sole acceptance gate for mission-critical integration paths.