Microservices patterns

The saga pattern

Keeping data consistent across services without one big transaction. Taught twice: first straight, then in the kitchen.

Think Agentic

In a monolith, this was easy

One application. One database.

Create the order, charge the card, reserve the stock: one transaction.

Anything fails? The database rolls it all back. All or nothing.

Microservices broke the easy part

🧾

Order

Owns the orders database

πŸ’³

Payment

Owns the payments database

πŸ“¦

Inventory

Owns the stock database

🚚

Shipping

Owns the shipments database

"Place an order" now touches four databases. Who guarantees all or nothing?

Why not one big distributed transaction?

Two-phase commit (2PC) can make several databases commit together. In microservices it's usually avoided:

  • βœ—Everything stays locked while every service answersOne slow service blocks all the others.
  • βœ—All services must be up at the same momentThat defeats the point of independent services.
  • βœ—Many modern databases and message brokers don't support it

The saga idea

Break the big transaction into a chain of small, local ones.Step 1: Order service creates the order, marked PENDING, and commits.Step 2: Payment service charges the card and commits.Step 3: Inventory service has no stock. This step fails.Steps 1 and 2 are already committed. Nothing rolls back by itself, so we compensate: refund the card.Then cancel the order. Compensations run in reverse order.The data is consistent again. That's a saga: local transactions forward, compensations backward.

🧾Create orderPENDING
β†’
πŸ’³Charge card$420
β†’
πŸ“¦Reserve stockout of stock
β†’
🚚Shipnever reached
↩️Cancel orderCANCELLED
←
↩️Refund card+$420
←

Compensation is not rollback

A rollback erases. A compensation is a new action that cancels the effect.

You don't delete the payment. You issue a refund, and the customer sees both on their statement.

Some things can't be undone at all, like an email already sent. Then the compensation is a follow-up: "Sorry, your order was cancelled."

For a moment, the data disagrees

Between steps, the card is charged but the order isn't confirmed. That's not a bug. It's eventual consistency, and you design for it:

  • 1Use in-between states: PENDING, not CONFIRMEDOther screens and services can see a saga is still running.
  • 2Tell the customer the truth"We're processing your order," not "Order confirmed."
  • 3Every saga ends in a clear final stateCompleted, or fully compensated. Never stuck halfway.

Two ways to run a saga

πŸ”” Choreography

  • No one in charge
  • Each service announces events: "PaymentCompleted"
  • Other services listen and react

πŸ§‘β€βœˆοΈ Orchestration

  • One coordinator is in charge
  • It sends commands: "Charge this card"
  • It tracks every step and runs the undo

Same goal, same compensations. The difference is who knows the plan.

βœ“1

Checkpoint 1

What did we learn about sagas? Three quick situations.

🎯

Checkpoint 1 Β· Question 1 of 3

The payment succeeded, then the inventory step failed.What should happen to the payment?

Issue a refund. The money really moved, so you compensate with a new action. Deleting the row would hide what happened and leave the customer out of pocket.

Checkpoint 1 Β· Question 2 of 3

Step 2 of a saga emailed the customer a confirmation. Then step 4 fails.How do you compensate step 2?

Send a follow-up. Some actions can't be undone. The compensation is a new action that corrects the effect.

Checkpoint 1 Β· Question 3 of 3

Two "services" share the same database and are owned by the same team.An update needs to change both of their tables.

No saga needed. One database means one ordinary transaction does the job. Sagas are for data in separate databases.
🍳

Now, in the kitchen

The same pattern, as a catering order moving through a busy restaurant.

πŸ‘¨β€πŸ³

Five stations, five notebooks

πŸ›ŽοΈ

Host stand

Takes orders

Order service
πŸ’΅

Cashier

Takes deposits

Payment service
πŸ₯«

Pantry

Reserves ingredients

Inventory service
🍳

Line cook

Cooks the food

Kitchen service
🚐

Driver

Delivers it

Delivery service

Each station writes only in its own notebook. Nobody can lock all five notebooks at once.

The order

🧾 "Dinner for 20. Salmon. Delivered Friday at 7."

It needs all five stations, one after another.

If any station can't do its part, the stations that already did theirs must undo it. The question is how the stations coordinate.

1

The kitchen that listens

Choreography: no one in charge. Each station rings a bell when it's done, and the next station reacts.

πŸ””

Choreography: when it all works

πŸ›ŽοΈHost stand
πŸ’΅Cashier
πŸ₯«Pantry
🍳Line cook
🚐Driver

THE BELL BOARD (event messages)

  1. πŸ›ŽοΈ Host writes the order as PENDING, rings OrderPlaced
  2. πŸ’΅ Cashier hears it, takes the deposit, rings DepositPaid
  3. πŸ₯« Pantry hears it, reserves salmon, rings SalmonReserved
  4. 🍳 Line cook hears it, cooks, rings FoodReady
  5. 🚐 Driver hears it, delivers, rings Delivered
  6. πŸ›ŽοΈ Host hears it and marks the order CONFIRMED. Nobody gave a single command.

Choreography: the pantry is out of salmon

πŸ›ŽοΈHost stand
πŸ’΅Cashier
πŸ₯«Pantry
🍳Line cook
🚐Driver

THE BELL BOARD

  1. πŸ›ŽοΈ OrderPlaced
  2. πŸ’΅ DepositPaid
  3. πŸ₯« Pantry can't reserve salmon, rings SalmonUnavailable
  4. πŸ’΅ Cashier hears it, refunds the deposit, rings DepositRefunded
  5. πŸ›ŽοΈ Host hears it and marks the order CANCELLED

Every station had to know which bells mean "undo your part."

Choreography: when to use it

Great for

  • Short flows: 2 to 4 steps
  • Stations that stay independent
  • No central piece to build or run

Watch out

  • Nobody can see the whole order at a glance
  • Adding a step means changing several stations
  • Debugging means asking every station what it heard
  • Easy to create loops of bells reacting to bells

Elegant for small flows. Hard to follow once the flow grows.

βœ“2

Checkpoint 2

Choreography: the kitchen that listens. Two situations.

🎯

Checkpoint 2 Β· Question 1 of 2

A choreography saga: Shipping fails right after StockReserved.Who has to react?

Every earlier station. There's no orchestrator. Shipping announces the failure, and Inventory, Payment and Order each listen and undo their own part.

Checkpoint 2 Β· Question 2 of 2

You need a fraud check between payment and stock reservation.The saga uses choreography. What changes?

Rewire the listeners. Adding a step changes who listens to what in several services. That's choreography's main cost as flows grow.
2

The expediter at the pass

Orchestration: one person holds the ticket and tells each station what to do next.

πŸ§‘β€βœˆοΈ

Orchestration: when it all works

πŸ§‘β€βœˆοΈExpediter πŸ›ŽοΈ Host πŸ’΅ Cashier πŸ₯« Pantry 🍳 Line cook 🚐 Driver

THE TICKET BOARD (saga state)

  1. Host: create order (PENDING)βœ“
  2. Cashier: take depositβœ“
  3. Pantry: reserve salmonβœ“
  4. Line cook: cookβœ“
  5. Driver: deliverβœ“
  6. Host: confirm orderβœ“

One place shows the whole plan and exactly where the order is.

Orchestration: the pantry is out of salmon

πŸ§‘β€βœˆοΈExpediter πŸ›ŽοΈ Host πŸ’΅ Cashier πŸ₯« Pantry 🍳 Line cook 🚐 Driver

THE TICKET BOARD

  1. Host: create order (PENDING)βœ“
  2. Cashier: take depositβœ“
  3. Pantry: reserve salmonβœ—
  4. Undo: Cashier refunds deposit↩
  5. Undo: Host cancels order↩

The expediter reads the board and runs the undo steps, newest first. The stations just follow orders.

Orchestration: when to use it

Great for

  • Long or branching flows
  • Seeing exactly where every order is
  • Changing the flow in one place

Watch out

  • One more component to build and keep running
  • The expediter can start doing the stations' jobs: keep only the sequence there
  • A poorly built expediter becomes a bottleneck

Tools that play the expediter: Temporal, Camunda, AWS Step Functions.

Side by side

πŸ”” ChoreographyπŸ§‘β€βœˆοΈ Orchestration
Who knows the plan?Nobody, as a wholeThe orchestrator
How steps startReacting to eventsReceiving commands
Adding a stepChange several servicesChange the orchestrator
DebuggingTrace events across servicesRead one saga's state
Best forShort, stable flowsLong, complex, branching flows
βœ“3

Checkpoint 3

Orchestration, and choosing between the two. Three situations.

🎯

Checkpoint 3 Β· Question 1 of 3

A new user signs up, gets a welcome email, and is added to the CRM.Three steps, one small team, and it rarely changes.

Choreography. Short and stable: a few events and listeners, with no extra component to run.

Checkpoint 3 Β· Question 2 of 3

Loan approval: nine steps, several branches, and compliance must see every application's status.The flow changes every quarter.

Orchestration. Long, branching, changing, and it needs visibility: that's the expediter's job.

Checkpoint 3 Β· Question 3 of 3

The deposit must be 20% of the order total.Where should that rule live?

In the Payment service. The orchestrator owns the sequence, not the business rules. Otherwise the expediter ends up doing the cooking.
!

The hard parts

Three kitchen accidents every saga has to survive, whichever style you pick.

🚨

The point of no return

🐟

The salmon is already cooked when the driver's van breaks down.

You can't un-cook salmon. Cancelling everything now wastes the food and the customer's evening.

So you don't cancel. You find another driver.

The rule: every saga has a point of no return (the pivot step). Before it, failures are compensated. After it, steps are retried until they succeed, so design those steps to be retryable.

The bell rang twice

πŸ””

The "order placed" bell gets rung twice. It happens: messaging systems usually deliver at least once.

The cashier takes the deposit twice.

The fix: check the ticket number. Each station records which tickets it already handled and ignores repeats. This is called an idempotent consumer.

The note that never left the kitchen

πŸ“

The cashier writes "deposit paid" in the notebook, then gets interrupted before ringing the bell.

The notebook says paid. Nobody else ever finds out. The order is stuck forever.

The fix: in the same motion as the notebook entry, drop the note in the station's own out-tray. A runner empties every out-tray and rings the bells. This is the transactional outbox: the database update and the outgoing message are saved together.
βœ“4

Checkpoint 4

The hard parts. Three situations from real systems.

🎯

Checkpoint 4 Β· Question 1 of 3

Some customers were charged twice.The "OrderPlaced" message was delivered twice during a network hiccup.

Make payment idempotent. Duplicates will happen. The payment service records which order IDs it already charged and ignores repeats.

Checkpoint 4 Β· Question 2 of 3

A custom-engraved watch has been engraved. Then creating the shipping label fails.The engraving can't be reversed.

Retry shipping. Engraving was the point of no return. After the pivot, steps are retried until they succeed, not undone.

Checkpoint 4 Β· Question 3 of 3

Payment saved PAID, then crashed before publishing DepositPaid.The order is stuck: no other service ever heard about it.

The outbox. It's the note that never left the kitchen. Publishing first just flips the problem; saving both in one transaction removes it.
3

Building it for real

The same sagas on AWS and in Spring Boot: architecture first, then the code that matters.

πŸ—οΈ

Orchestration on AWS

Step Functions plays the expediter. Each service keeps its own table.The client calls API Gateway, which routes to the Order service. It saves the order as PENDING and starts a Step Functions execution.Step Functions calls the Payment Lambda. The deposit is charged and recorded in Payment's own table.The Inventory Lambda fails: out of stock. Step Functions catches the error.The Catch sends it to compensation states: refund, then cancel. Every step appears in the execution history.The kitchen, in AWS terms: Step Functions is the expediter, its execution history is the ticket board.

πŸ“± Client πŸšͺ API GW 🧾 OrderECS Β· own table πŸ§‘β€βœˆοΈ StepFunctionsthe expediter πŸ’³ PaymentLambda Β· DynamoDB πŸ“¦ InventoryLambda Β· DynamoDB 🚚 ShippingLambda Β· DynamoDB ↩ refund deposit ↩ cancel order

The expediter's rules, in Step Functions

// One step of the state machine (Amazon States Language)
"ReserveStock": {
  "Type": "Task",
  "Resource": "arn:aws:states:::lambda:invoke",
  "Parameters": { "FunctionName": "reserve-stock", "Payload.$": "$" },
  "Retry": [{
    "ErrorEquals": ["Lambda.ServiceException"],
    "MaxAttempts": 3, "BackoffRate": 2
  }],
  "Catch": [{
    "ErrorEquals": ["States.ALL"],
    "Next": "RefundPayment"   // start compensating
  }],
  "Next": "ShipOrder"
},
"RefundPayment": { "Type": "Task", ..., "Next": "CancelOrder" }
  1. Retry handles blips: a timeout isn't a business failure.
  2. Catch is where real failures turn into compensation.
  3. Compensations are just more states, chained in reverse order.
  4. Steps after the pivot get Retry only, no Catch to compensation.

Choreography on AWS

EventBridge plays the bell board. No service is in charge.Order saves the order and an outbox record in one transaction. A stream publishes OrderPlaced to EventBridge.A rule delivers it to Payment's SQS queue. Payment charges once (checking the event ID) and publishes DepositPaid.Inventory hears DepositPaid, reserves stock, publishes StockReserved.Shipping publishes Shipped. Order hears it and marks the order CONFIRMED.Failure works the same way: Inventory publishes StockUnavailable, and Payment and Order react by compensating.

SQSSQSSQSSQS 🧾 Orderown table + outbox πŸ’³ Paymentown table + outbox πŸ“¦ Inventoryown table + outbox 🚚 Shippingown table + outbox πŸ“£ Amazon EventBridge: the bell board OrderPlaced DepositPaid StockReserved Shipped

Orchestration in Spring Boot, with Temporal

// The workflow is the expediter, written as plain Java
public void placeOrder(Order order) {
  Saga saga = new Saga(new Saga.Options.Builder().build());
  try {
    payments.charge(order);
    saga.addCompensation(payments::refund, order);

    inventory.reserve(order);
    saga.addCompensation(inventory::release, order);

    shipping.ship(order);  // pivot: retried, never undone
  } catch (ActivityFailure e) {
    saga.compensate();     // runs the undos, newest first
    throw e;
  }
}
  1. Each call is an activity, often a call to another Spring Boot service.
  2. Register an undo only after its step succeeds.
  3. Temporal saves progress after every step, so a crash resumes instead of restarting.
  4. Same idea: Camunda, Axon, Eventuate Tram.

Choreography in Spring Boot, with Kafka

// Payment service: hear OrderPlaced, react, announce
@KafkaListener(topics = "order-events")
@Transactional
public void on(OrderPlaced event) {
  if (processed.existsById(event.eventId())) {
    return;                 // bell rang twice: ignore
  }
  payments.save(Payment.forOrder(event));
  outbox.save(OutboxEvent.of("DepositPaid", event.orderId()));
  processed.save(new ProcessedEvent(event.eventId()));
}  // all three rows commit together, or none do

// A relay (Debezium, or Spring Modulith's event
// externalization) moves outbox rows to Kafka.
  1. Idempotency: the processed-events table is the ticket-number check.
  2. Outbox: the event is a row in the same transaction, never a separate send.
  3. Compensation is just another listener, e.g. on StockUnavailable.

The hard parts, in practice

ProblemOn AWSIn Spring Boot
Retry after the pivotStep Functions Retry with backoffTemporal retry policies, Spring Retry
Bell rang twiceDynamoDB conditional write on the event ID; Powertools idempotencyProcessed-events table with a unique key
Note never leftDynamoDB Streams β†’ EventBridge PipesOutbox table + Debezium, or Spring Modulith
Finding a stuck sagaStep Functions execution history, X-RaySaga ID in every message, OpenTelemetry

Choosing a stack

StackStylePick it when
Step Functions + Lambda + DynamoDBOrchestrationYou're on AWS and want managed, visual workflows
EventBridge + SQS + LambdaChoreographyShort flows, AWS-native events
Spring Boot + TemporalOrchestrationCode-first and portable across clouds
Spring Boot + Kafka + DebeziumChoreographyYou already run Kafka
CamundaOrchestrationThe business wants to see and change the flow as a diagram

Other clouds: Azure Durable Functions + Service Bus. Google Cloud Workflows + Pub/Sub.

Don't build your own orchestrator. And test the failure path, not just the happy one.

βœ“5

Checkpoint 5

Building it for real. Three design decisions, then your turn.

🎯

Checkpoint 5 Β· Question 1 of 3

Your team is on AWS. The order flow has 7 steps with branches.Operations wants to see exactly where each order is stuck.

Step Functions. Long, branching, and needs visibility: orchestration. Every execution shows its current step and history.

Checkpoint 5 Β· Question 2 of 3

Spring Boot + Kafka choreography: sometimes the order is saved, but OrderPlaced never reaches Kafka.No errors in the logs.

Outbox + a relay. Kafka isn't part of your database transaction. Write the event as a row in the same transaction, and let Debezium or Spring Modulith publish it.

Checkpoint 5 Β· Question 3 of 3

In Step Functions, ShipOrder times out.Shipping comes after the point of no return.

Retry with backoff. After the pivot you retry, not undo. Alert a person only if the retries run out.

Your turn

Design the trip-booking saga

A customer books a flight, a hotel and a rental car, paid by card. Flight tickets are non-refundable once issued. Support must see every booking's status.

  • 1List the steps, and an undo for each
  • 2Find the point of no return
  • 3Put the steps in the best order
  • 4Choose choreography or orchestration, and a stack

One good answer: the steps

StepUndoWhy here
1. Hold the hotel roomRelease the holdCheap and fully reversible
2. Reserve the carCancel the reservationCheap and fully reversible
3. Charge the cardRefundReversible, but costs fees
4. Issue the flight ticketNone: the pivotIrreversible, so it goes last
5. Confirm hotel and carRetry until doneAfter the pivot
6. Email the itineraryRetry until doneAfter the pivot

One good answer: the decisions

  • 1Reversible steps first, the irreversible one lastThe best saga design is often just the right order.
  • 2OrchestrationSix steps, and support needs to see status: Step Functions on AWS, or Spring Boot + Temporal.
  • 3The booking ID as the idempotency keyEspecially on the card charge and the ticket issue.
  • 4The trip shows PENDING until the ticket is issuedThen CONFIRMED, even while steps 5 and 6 retry.

At a glance

Real nameIn the kitchenRemember
SagaOne catering order across stationsA chain of local transactions
CompensationThe refund, not tearing out the pageA new action that cancels an effect
ChoreographyStations reacting to bellsEvents, no one in charge
OrchestrationThe expediter with the ticketCommands, one place knows the plan
Pivot stepThe salmon is cookedAfter it: retry, don't undo
IdempotencyCheck the ticket numberHandle each message once
OutboxNote in the out-traySave the change and the message together

No single transaction across services: a saga is a chain of local ones.

Every step needs an undo, until the point of no return.

Short flows: choreography. Complex flows: orchestration.

Think Agentic

Learning
β†’ / Space next Β· ← back Β· M slide menu Β· F fullscreen