The salmon is already cooked when the driver's van breaks down.
You can't un-cook salmon. Cancelling everything now wastes the food and the customer's evening.
So you don't cancel. You find another driver.
Microservices patterns
Keeping data consistent across services without one big transaction. Taught twice: first straight, then in the kitchen.
Think Agentic
One application. One database.
Create the order, charge the card, reserve the stock: one transaction.
Anything fails? The database rolls it all back. All or nothing.
Owns the orders database
Owns the payments database
Owns the stock database
Owns the shipments database
"Place an order" now touches four databases. Who guarantees all or nothing?
Two-phase commit (2PC) can make several databases commit together. In microservices it's usually avoided:
Break the big transaction into a chain of small, local ones.Step 1: Order service creates the order, marked PENDING, and commits.Step 2: Payment service charges the card and commits.Step 3: Inventory service has no stock. This step fails.Steps 1 and 2 are already committed. Nothing rolls back by itself, so we compensate: refund the card.Then cancel the order. Compensations run in reverse order.The data is consistent again. That's a saga: local transactions forward, compensations backward.
A rollback erases. A compensation is a new action that cancels the effect.
You don't delete the payment. You issue a refund, and the customer sees both on their statement.
Some things can't be undone at all, like an email already sent. Then the compensation is a follow-up: "Sorry, your order was cancelled."
Between steps, the card is charged but the order isn't confirmed. That's not a bug. It's eventual consistency, and you design for it:
Same goal, same compensations. The difference is who knows the plan.
What did we learn about sagas? Three quick situations.
π―Checkpoint 1 Β· Question 1 of 3
The payment succeeded, then the inventory step failed.What should happen to the payment?
Checkpoint 1 Β· Question 2 of 3
Step 2 of a saga emailed the customer a confirmation. Then step 4 fails.How do you compensate step 2?
Checkpoint 1 Β· Question 3 of 3
Two "services" share the same database and are owned by the same team.An update needs to change both of their tables.
The same pattern, as a catering order moving through a busy restaurant.
π¨βπ³Takes orders
Order serviceTakes deposits
Payment serviceReserves ingredients
Inventory serviceCooks the food
Kitchen serviceDelivers it
Delivery serviceEach station writes only in its own notebook. Nobody can lock all five notebooks at once.
π§Ύ "Dinner for 20. Salmon. Delivered Friday at 7."
It needs all five stations, one after another.
If any station can't do its part, the stations that already did theirs must undo it. The question is how the stations coordinate.
Choreography: no one in charge. Each station rings a bell when it's done, and the next station reacts.
πTHE BELL BOARD (event messages)
OrderPlacedDepositPaidSalmonReservedFoodReadyDeliveredTHE BELL BOARD
OrderPlacedDepositPaidSalmonUnavailableDepositRefundedEvery station had to know which bells mean "undo your part."
Elegant for small flows. Hard to follow once the flow grows.
Choreography: the kitchen that listens. Two situations.
π―Checkpoint 2 Β· Question 1 of 2
A choreography saga: Shipping fails right after StockReserved.Who has to react?
Checkpoint 2 Β· Question 2 of 2
You need a fraud check between payment and stock reservation.The saga uses choreography. What changes?
Orchestration: one person holds the ticket and tells each station what to do next.
π§ββοΈTHE TICKET BOARD (saga state)
One place shows the whole plan and exactly where the order is.
THE TICKET BOARD
The expediter reads the board and runs the undo steps, newest first. The stations just follow orders.
Tools that play the expediter: Temporal, Camunda, AWS Step Functions.
| π Choreography | π§ββοΈ Orchestration | |
|---|---|---|
| Who knows the plan? | Nobody, as a whole | The orchestrator |
| How steps start | Reacting to events | Receiving commands |
| Adding a step | Change several services | Change the orchestrator |
| Debugging | Trace events across services | Read one saga's state |
| Best for | Short, stable flows | Long, complex, branching flows |
Orchestration, and choosing between the two. Three situations.
π―Checkpoint 3 Β· Question 1 of 3
A new user signs up, gets a welcome email, and is added to the CRM.Three steps, one small team, and it rarely changes.
Checkpoint 3 Β· Question 2 of 3
Loan approval: nine steps, several branches, and compliance must see every application's status.The flow changes every quarter.
Checkpoint 3 Β· Question 3 of 3
The deposit must be 20% of the order total.Where should that rule live?
Three kitchen accidents every saga has to survive, whichever style you pick.
π¨The salmon is already cooked when the driver's van breaks down.
You can't un-cook salmon. Cancelling everything now wastes the food and the customer's evening.
So you don't cancel. You find another driver.
The "order placed" bell gets rung twice. It happens: messaging systems usually deliver at least once.
The cashier takes the deposit twice.
The cashier writes "deposit paid" in the notebook, then gets interrupted before ringing the bell.
The notebook says paid. Nobody else ever finds out. The order is stuck forever.
The hard parts. Three situations from real systems.
π―Checkpoint 4 Β· Question 1 of 3
Some customers were charged twice.The "OrderPlaced" message was delivered twice during a network hiccup.
Checkpoint 4 Β· Question 2 of 3
A custom-engraved watch has been engraved. Then creating the shipping label fails.The engraving can't be reversed.
Checkpoint 4 Β· Question 3 of 3
Payment saved PAID, then crashed before publishing DepositPaid.The order is stuck: no other service ever heard about it.
The same sagas on AWS and in Spring Boot: architecture first, then the code that matters.
ποΈStep Functions plays the expediter. Each service keeps its own table.The client calls API Gateway, which routes to the Order service. It saves the order as PENDING and starts a Step Functions execution.Step Functions calls the Payment Lambda. The deposit is charged and recorded in Payment's own table.The Inventory Lambda fails: out of stock. Step Functions catches the error.The Catch sends it to compensation states: refund, then cancel. Every step appears in the execution history.The kitchen, in AWS terms: Step Functions is the expediter, its execution history is the ticket board.
// One step of the state machine (Amazon States Language) "ReserveStock": { "Type": "Task", "Resource": "arn:aws:states:::lambda:invoke", "Parameters": { "FunctionName": "reserve-stock", "Payload.$": "$" }, "Retry": [{ "ErrorEquals": ["Lambda.ServiceException"], "MaxAttempts": 3, "BackoffRate": 2 }], "Catch": [{ "ErrorEquals": ["States.ALL"], "Next": "RefundPayment" // start compensating }], "Next": "ShipOrder" }, "RefundPayment": { "Type": "Task", ..., "Next": "CancelOrder" }
EventBridge plays the bell board. No service is in charge.Order saves the order and an outbox record in one transaction. A stream publishes OrderPlaced to EventBridge.A rule delivers it to Payment's SQS queue. Payment charges once (checking the event ID) and publishes DepositPaid.Inventory hears DepositPaid, reserves stock, publishes StockReserved.Shipping publishes Shipped. Order hears it and marks the order CONFIRMED.Failure works the same way: Inventory publishes StockUnavailable, and Payment and Order react by compensating.
// The workflow is the expediter, written as plain Java public void placeOrder(Order order) { Saga saga = new Saga(new Saga.Options.Builder().build()); try { payments.charge(order); saga.addCompensation(payments::refund, order); inventory.reserve(order); saga.addCompensation(inventory::release, order); shipping.ship(order); // pivot: retried, never undone } catch (ActivityFailure e) { saga.compensate(); // runs the undos, newest first throw e; } }
// Payment service: hear OrderPlaced, react, announce @KafkaListener(topics = "order-events") @Transactional public void on(OrderPlaced event) { if (processed.existsById(event.eventId())) { return; // bell rang twice: ignore } payments.save(Payment.forOrder(event)); outbox.save(OutboxEvent.of("DepositPaid", event.orderId())); processed.save(new ProcessedEvent(event.eventId())); } // all three rows commit together, or none do // A relay (Debezium, or Spring Modulith's event // externalization) moves outbox rows to Kafka.
StockUnavailable.| Problem | On AWS | In Spring Boot |
|---|---|---|
| Retry after the pivot | Step Functions Retry with backoff | Temporal retry policies, Spring Retry |
| Bell rang twice | DynamoDB conditional write on the event ID; Powertools idempotency | Processed-events table with a unique key |
| Note never left | DynamoDB Streams β EventBridge Pipes | Outbox table + Debezium, or Spring Modulith |
| Finding a stuck saga | Step Functions execution history, X-Ray | Saga ID in every message, OpenTelemetry |
| Stack | Style | Pick it when |
|---|---|---|
| Step Functions + Lambda + DynamoDB | Orchestration | You're on AWS and want managed, visual workflows |
| EventBridge + SQS + Lambda | Choreography | Short flows, AWS-native events |
| Spring Boot + Temporal | Orchestration | Code-first and portable across clouds |
| Spring Boot + Kafka + Debezium | Choreography | You already run Kafka |
| Camunda | Orchestration | The business wants to see and change the flow as a diagram |
Other clouds: Azure Durable Functions + Service Bus. Google Cloud Workflows + Pub/Sub.
Don't build your own orchestrator. And test the failure path, not just the happy one.
Building it for real. Three design decisions, then your turn.
π―Checkpoint 5 Β· Question 1 of 3
Your team is on AWS. The order flow has 7 steps with branches.Operations wants to see exactly where each order is stuck.
Checkpoint 5 Β· Question 2 of 3
Spring Boot + Kafka choreography: sometimes the order is saved, but OrderPlaced never reaches Kafka.No errors in the logs.
Checkpoint 5 Β· Question 3 of 3
In Step Functions, ShipOrder times out.Shipping comes after the point of no return.
Your turn
A customer books a flight, a hotel and a rental car, paid by card. Flight tickets are non-refundable once issued. Support must see every booking's status.
| Step | Undo | Why here |
|---|---|---|
| 1. Hold the hotel room | Release the hold | Cheap and fully reversible |
| 2. Reserve the car | Cancel the reservation | Cheap and fully reversible |
| 3. Charge the card | Refund | Reversible, but costs fees |
| 4. Issue the flight ticket | None: the pivot | Irreversible, so it goes last |
| 5. Confirm hotel and car | Retry until done | After the pivot |
| 6. Email the itinerary | Retry until done | After the pivot |
| Real name | In the kitchen | Remember |
|---|---|---|
| Saga | One catering order across stations | A chain of local transactions |
| Compensation | The refund, not tearing out the page | A new action that cancels an effect |
| Choreography | Stations reacting to bells | Events, no one in charge |
| Orchestration | The expediter with the ticket | Commands, one place knows the plan |
| Pivot step | The salmon is cooked | After it: retry, don't undo |
| Idempotency | Check the ticket number | Handle each message once |
| Outbox | Note in the out-tray | Save the change and the message together |
No single transaction across services: a saga is a chain of local ones.
Every step needs an undo, until the point of no return.
Short flows: choreography. Complex flows: orchestration.
Think Agentic