Memory Efficiency and Performance

At Ettaflow, our control plane needs to manage thousands of concurrent data pipelines. Reliability is non-negotiable. If a worker crashes, the pipeline must pick up exactly where it left off.

Why Go?

We chose Go (Golang) for our core infrastructure services to help us Escape the Volume Tax on infrastructure costs.

• Concurrency: Goroutines allow us to handle thousands of concurrent connections with minimal memory overhead.
• Performance: Near C-level performance with the safety of a managed language. Used in tandem with Apache Arrow for data handling.
• Simplicity: New engineers can ramp up quickly.

Orchestration with Temporal

Stateless services are easy. Stateful workflows—like a long-running ETL job that waits for an API webhook—are hard.

We use Temporal to handle workflow orchestration.

The Problem with Queues

Using standard queues (Kafka/RabbitMQ) for complex workflows often leads to “queue hell” where you have to manage state manually in a database, dealing with timeouts, retries, and dead-letter queues.

The Temporal Way

Temporal allows us to write workflow code as if it were synchronous.

func (w *Workflow) Run(ctx Context) error {
    // This looks synchronous, but it persists state!
    data := cv.ExecuteActivity(ctx, FetchData, src).Get()
    cv.ExecuteActivity(ctx, WriteData, dest, data).Get()
    return nil
}

This abstraction allows us to build complex, reliable pipelines that survive process restarts and infrastructure failures without losing a beat.