Simple Before Distributed: When an AI System Actually Needs Workflow Orchestration

The Premature Distribution Trap

In modern AI engineering, there is an understandable tendency to adopt hyper-scale cloud patterns prematurely. The introduction of an LLM or OCR capability is often immediately coupled with distributed message brokers (Kafka, RabbitMQ), vector stores, asynchronous task queues (Celery, BullMQ), and dedicated durable workflow engines (Temporal, AWS Step Functions).

However, every distributed component introduces real engineering tax:

• Network partition handling and distributed consensus overhead;
• Inability to run or test workflows easily in a clean local development environment;
• Serialization and queue dispatch latencies;
• Operational overhead: monitoring dead-letter queues, handling poison pills, and managing worker lifecycle;
• Complex distributed tracing and error attribution.

The Simple Before Distributed principle states that an AI system should remain at the simplest viable execution archetype until the physical characteristics of the workload demand the next tier of complexity.

Below, we examine two contrasting first-party case studies and formalize a 12-factor decision framework.

---

A Tale of Two Workload Topologies

To understand where complexity belongs, consider two contrasting systems with opposing operational requirements.

Case A: The Perfume Engine Validation Prototype

The initial validation prototype for Perfume Engine evaluated fragrance-note compatibility using deterministic scoring over an curated, normalized dataset.

Operational characteristics:

1. Data volume: A compact, bounded catalog (dozens of notes and fragrance records).
2. Compute time: Graph matching and projection calculations executed in under 50 milliseconds in memory.
3. Failure probability: Zero third-party API dependencies during runtime evaluation.
4. Restart cost: Negligible; any request or test could be recomputed from scratch instantly.

Architectural Choice: The system was implemented as a pure, deterministic in-memory function with no runtime database, no asynchronous workers, and no message queues. Data was loaded as an immutable build artifact, and queries were evaluated synchronously. Introducing Redis, Kafka, or background workers into this loop would have added infrastructure latency, configuration complexity, and cost without improving reliability or user experience.

Case B: The Big PDF Translator Architecture Study

In our architecture study for a high-volume document translation system, the operational profile was completely inverted.

Operational characteristics:

1. Work duration: Documents ranged from 100 to 1,000+ pages; processing a full document (OCR, structural extraction, multi-stage LLM translation, layout reconstruction) took from 15 minutes to several hours.
2. External dependencies: Hundreds of API requests to commercial OCR and machine translation services subject to rate limits, connection resets, and HTTP 429/502/504 errors.
3. Failure granularity: An API timeout on page 850 must not invalidate the successful work completed on pages 1 through 849.
4. Restart cost: Re-running a 1,000-page job from the beginning would cost significant API fees and waste hours of processing time.

Architectural Choice: Synchronous execution or simple background threads are structurally inadequate for this problem. The architecture requires a canonical intermediate representation (DocumentIR), persistent step-level checkpoints, and durable workflow orchestration capable of pausing, resuming, retrying individual units, and surviving process restarts.

---

The 12-Factor Orchestration Matrix

To guide architecture choices objectively, we evaluate workloads across 12 distinct dimensions:

#	Dimension	Favors Simplicity (Sync / Local Background)	Demands Durable Orchestration (Queues / Workflows)
1	Execution Duration	Under 5 seconds (comfortably fits HTTP timeout)	Minutes to hours (requires persistent execution state)
2	Step Count	1–2 atomic operations	Multi-step graph with conditional branching
3	Restart Cost	Trivial (CPU milliseconds, zero financial cost)	Prohibitive (expensive GPU/LLM tokens, hours of work)
4	Failure Probability	Near zero (pure local deterministic computation)	High (unstable external APIs, throttling, quotas)
5	Retry Granularity	All-or-nothing retry is acceptable	Fine-grained retry per paragraph, page, or token batch
6	Progress Persistence	Intermediate state has no residual value	Intermediate artifacts must be inspectable and durable
7	Cancellation & Pause	Not needed, or simple socket disconnection	Explicit pause, user cancellation, and resume semantics
8	External Dependencies	None, or internal low-latency database	3+ external SaaS endpoints with divergent rate limits
9	Concurrency & Fan-Out	Sequential or small in-process thread pool	Massive parallel fan-out (thousands of sub-tasks)
10	Human-in-the-Loop	Fully automated completion	Workflow must pause for human review across days
11	Recovery & RPO	Client simply re-submits if node crashes	Process resumes from last checkpoint on worker reboot
12	Audit & Debugging	Standard application logging (ELK / CloudWatch)	Full event-sourced replay and step-by-step history

---

Five Architectural Execution Archetypes

Evaluating these factors leads to five distinct architectural tiers:

[1. Synchronous Request-Reply]
       │ (HTTP timeouts exceeded, latency variance increases)
       ▼
[2. In-Memory Background Job] (task runner, thread pool)
       │ (risk of job loss on process restart, status polling needed)
       ▼
[3. Database-Backed Job State] (relational polling, transactional state machine)
       │ (high concurrency, specialized workers, queue throughput)
       ▼
[4. Distributed Queue & Worker Pools] (RabbitMQ, Redis Streams, Celery)
       │ (multi-step dependencies, cross-service sagas, long-running durability)
       ▼
[5. Durable Workflow Orchestration] (Temporal, AWS Step Functions)

1. Synchronous Request-Reply

• Best for: Deterministic evaluations, input validation, lightweight embedding calls where p99 < 1–2s.
• Example: Perfume Engine validation prototype; short query classification.
• Benefits: Minimal cognitive overhead, zero state infrastructure, rapid testability.

2. In-Memory Background Job

• Best for: Non-critical fire-and-forget operations where losing a single task during deployment is tolerable (e.g., telemetry emission, cache pre-warming).
• Benefits: No external database or message broker required.

3. Database-Backed Job State

• Best for: Asynchronous tasks lasting 5–60 seconds, volume of dozens of tasks per minute, where users need progress tracking in a web UI (pending, processing, completed).
• Benefits: Leverages existing ACID transactions in PostgreSQL/MySQL without managing external broker infrastructure.

4. Distributed Queue & Worker Pools

• Best for: High-throughput, homogeneous tasks requiring independent node scaling (e.g., dedicated GPU workers processing audio transcription).
• Trade-off: Coordinating multi-step dependencies or compensation logic solely via queue topics requires complex custom state machines.

5. Durable Workflow Orchestration

• Best for: Long-running, multi-day, or multi-step processes (as modeled in the Big PDF Translator study) requiring granular retry policies, human approvals, or multi-stage external coordination.
• Benefits: Immunity to process failure, event-sourced auditability, and deterministic step replays.

---

The Rule of Thumb for AI Workloads

For founders and engineering leads, the guiding heuristic is simple:

If a failure can be resolved by restarting the entire job at negligible cost, keep the architecture simple.

If restarting an operation from scratch wastes more than $10 in API costs or 30 minutes of user waiting, durable orchestration pays for itself from day one.

Architectural sophistication is not measured by the number of distributed services in a diagram, but by how closely infrastructure complexity reflects actual workload requirements.