TL;DR
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
A recent discussion advocates replacing external workflow orchestrators with Postgres to simplify durable workflows. This approach leverages Postgres’s scalability, observability, and security features, potentially streamlining reliable program execution.
Developers are increasingly advocating for using Postgres as the primary tool for implementing durable workflows, replacing complex external orchestrators. This shift aims to simplify workflow management while leveraging Postgres’s proven scalability and reliability, making durable program execution more accessible and manageable.
Traditionally, durable workflows rely on external orchestrators like Temporal or Airflow to manage step execution and checkpointing. These systems coordinate workflow steps through a central server, which can introduce complexity and points of failure. Recent discussions suggest that since durable workflows fundamentally depend on a database to checkpoint state, it’s more efficient to eliminate the orchestrator and use Postgres directly for coordination.
In this model, application servers interact directly with Postgres, creating workflow entries and polling for tasks. Workers execute steps and checkpoint progress by updating Postgres tables. If a worker crashes, other servers can recover workflows from the checkpoints stored in the database. This approach uses Postgres’s native features—such as locking and constraints—to manage concurrency and ensure consistency, removing the need for a separate orchestrator server.
Proponents highlight that Postgres’s scalability allows systems to handle tens of thousands of workflows per second, especially when combined with distributed or sharded setups. Its high availability options, including streaming replication and managed multi-AZ deployments, further enhance system resilience. Additionally, storing workflow data in relational tables enables powerful real-time observability through SQL queries, facilitating monitoring and debugging without additional tooling.
Why It Matters
This approach could significantly reduce the complexity of building reliable workflows, lowering operational overhead and points of failure. By leveraging Postgres’s mature ecosystem, teams can address scalability, availability, security, and observability issues using well-understood solutions, potentially making durable workflows more accessible to a broader range of developers and organizations.
However, this shift also challenges the prevailing reliance on specialized orchestration tools, raising questions about scalability limits, best practices for concurrency control, and handling complex workflows in large-scale environments. The community is still evaluating whether Postgres-based workflows can replace dedicated orchestrators in all scenarios.
Products Worth Considering
LG UltraGear 32GS60QC-B 32-inch Curved QHD Gaming Monitor (180Hz, 1ms, 1000R, AMD FreeSync, HDR10)
LG UltraGear 34GX90SA-W 34-inch WQHD Curved OLED Gaming Monitor
Sony WH-1000XM4 Wireless Noise Cancelling Headphones, Black
As an affiliate, we earn on qualifying purchases.
Background
Durable workflows have traditionally been implemented with external orchestrators like Temporal, Airflow, and AWS Step Functions, which manage step coordination outside the database. These systems are designed to handle complex dependencies, retries, and observability but add architectural complexity.
Recent discussions, including prominent developer forums and blogs, have questioned whether this complexity is necessary, given that the core requirement—checkpointing state—is inherently a database concern. The idea of using Postgres directly as the orchestrator has gained traction, especially among teams seeking simpler, more integrated solutions.
“If durable workflows are just about databases, then why have a separate orchestrator? Using Postgres directly simplifies the architecture and leverages its robustness.”
— Developer community member on Hacker News
“Postgres’s scalability and high availability features are mature and well-understood, making it a reliable backbone for durable workflows.”
— Postgres expert
What Remains Unclear
It remains unclear whether Postgres-based workflows can match the flexibility and complexity of dedicated orchestrators in all use cases, particularly for highly intricate or large-scale workflows. Operational best practices and performance limits are still being explored, and more real-world testing is needed to validate this approach at scale.
What’s Next
Developers and organizations are beginning to experiment with Postgres-backed workflows in production environments. Future steps include benchmarking performance, developing best practices for concurrency and failure recovery, and creating tooling to support observability and debugging. Community discussions and case studies will help determine whether this approach can replace traditional orchestrators broadly.
Key Questions
Can Postgres handle complex workflows with many dependencies?
While Postgres can manage simple to moderately complex workflows effectively, handling highly intricate dependencies may require careful schema design and application logic. Ongoing research and testing are assessing its limits for complex scenarios.
Is this approach suitable for large-scale production systems?
It is still an emerging approach. Many developers see promise in using Postgres for durability and observability, but comprehensive benchmarks and operational best practices are still developing before it can be widely adopted at scale.
What are the main advantages over traditional external orchestrators?
The primary benefits include reduced architectural complexity, leveraging existing database infrastructure, improved observability via SQL, and potentially lower operational costs. However, trade-offs may exist depending on workflow complexity and scale.
Source: Hacker News
