AI has made webhook integration look deceptively simple. Create a controller, accept a JSON body, parse a few fields, and call the service that updates your database.
That is not webhook processing. That is receiving a request.
The real work starts after the payload arrives. External systems retry, Queues redeliver, and Schedulers overlap, and as a result Payloads omit fields. A message that looks like a complete update is often only a signal that something changed somewhere else.
I have seen teams treat webhooks as synchronous API calls in disguise. That works in demos. It fails in production, especially when the webhook is coming from an external system that has its own retry rules, its own event IDs, and its own idea of what "updated" means.
This article is about the architecture line I now draw around webhooks: receive, store, claim, dispatch, and sync.
A Webhook Is Not the Source of Truth
The most common lie is assuming the webhook payload is the complete truth.
An external CRM, payment gateway, or risk system sends a payload that says an entity changed. It may include useful fields. It may include enough context to display a notification. But that does not mean the payload is the right place to rebuild your local domain record.
For many operational systems, the safer pattern is:
external system says: record 12345 changed
↓
application extracts record ID
↓
application runs its existing sync flow
↓
application fetches the full latest record
↓
application updates local state through the normal path
That is less glamorous than mapping every field in the webhook JSON. It is also more reliable.
This matters because the sync flow usually already contains the hard parts: field normalization, relationship handling, encryption boundaries, audit behavior, retry handling, and database update semantics. If you bypass that path just because a webhook payload arrived, you create a second write path that will drift.
Related Articles
Shared topics and tags
Newsletter
Expert notes in your inbox
Subscribe for new articles.
A webhook should wake up the right workflow. It should not invent a new one.
Store First, Process Later
Another lie is treating webhook handling as a single request-response operation.
The external system does not care whether your downstream database update, enrichment call, or business workflow completed. It only cares that your receiver accepted the event. If your app tries to do all processing during the inbound request, every downstream dependency becomes part of the webhook acknowledgement path.
That is fragile.
The pattern I prefer is simple:
receive payload
↓
store payload durably
↓
acknowledge receipt
↓
process asynchronously
This gives the system room to breathe. A temporary downstream failure does not force the external system to keep hammering the receiver. A parsing bug can be fixed and reprocessed. Operators can see what arrived, what failed, and what completed.
The database row becomes the operational record of the webhook. Not just a log line. Not just a transient message.
That row needs a status:
NEW → PROCESSING → COMPLETE
↘
FAILED
This status model looks small, but it is the difference between "we hope the webhook ran" and "we can explain what happened."
Claim Before You Process
Asynchronous processing introduces its own lie: if the scheduler sees a row, it owns the row.
It does not.
In real systems, multiple app instances may run the same scheduler. A job may overlap with itself. A queue may deliver the same item twice. Two workers can read the same NEW row at almost the same time.
The fix is a claim step.
Do not process a webhook just because you selected it. First, atomically change it from NEW to PROCESSING only if it is still NEW.
worker A: update row 10 from NEW to PROCESSING
worker B: update row 10 from NEW to PROCESSING
Only one should win.
The worker that wins the claim processes the event. The worker that loses skips it. This is not overengineering. This is basic respect for concurrency.
The claim protects against duplicate local processing. It does not replace external idempotency.
External Request IDs Are Not Optional
Most serious webhook providers include an event ID, request ID, delivery ID, or some equivalent identifier. Use it.
Store the event type and external request ID together. Add a uniqueness rule around that pair where possible.
The external request ID protects against duplicate deliveries from the external system. The claim step protects against duplicate processing by your own workers.
Without the external ID, a retried webhook looks like a new event. Without the claim, the same stored row can be processed twice by your own app. These are different failure modes, and pretending one solves the other is another way teams lie to themselves.
Dispatch, Do Not Rebuild
Once the webhook is claimed and identified, the next decision is where it goes.
A good webhook processor should dispatch into an existing workflow whenever possible.
For example:
record.updated webhook
↓
extract record ID
↓
publish "sync this record"
↓
existing lead sync listener handles the real update
That preserves one write path.
It also keeps the webhook layer narrow. The webhook processor is responsible for event interpretation and idempotency. The sync layer is responsible for knowing how to fetch and persist the domain object.
When those responsibilities blur, webhook code slowly becomes a parallel integration layer. It starts with one field. Then three fields. Then relationship updates. Then special cases. Eventually the system has two definitions of how external data enters the database.
That is when small webhook changes become dangerous.
Unknown Events Should Not Break the Pipeline
External systems evolve. New webhook types appear. Payload shapes change. Fields become nullable that were previously always present.
Your processor should not assume every stored payload is one of the events you currently handle.
The better behavior is:
Parse metadata
Identify the event type
Process supported events
Mark unsupported but valid events complete or ignored according to your policy
Mark malformed events failed
That keeps the pipeline moving.
An unknown event is not the same as a broken event. A missing required hook field is not the same as an unsupported event type. A duplicate request ID is not the same as a processing failure.
Those distinctions matter when someone has to debug the system later.
Tests Must Cover the Operational Contract
Webhook tests often stop at "can deserialize the sample payload." That is necessary, but far from enough.
The tests that matter cover the operational contract:
Can the sample payload be parsed?
Are old rows with missing new metadata still safe after migration?
Does the uniqueness rule reject duplicate external request IDs?
Does the claim step allow only one processor to win?
Does the event dispatch call the existing sync workflow?
Does a malformed payload become failed instead of crashing the scheduler?
These are not decorative tests. They prove the system can survive the ways webhooks actually behave.
Summary
Treat the webhook as a signal, not the complete source of truth: Keeps the existing sync/write path authoritative
Store payloads before processing: Separates external acknowledgement from internal workflow success
Add external event/request ID tracking: Prevents duplicate provider deliveries from becoming duplicate work
Claim rows before processing: Prevents multiple workers from processing the same stored event
Dispatch into existing workflows: Avoids creating a second domain write path
Test migration and duplicate behavior: Verifies production-facing guarantees, not just happy-path parsing
Final Thoughts
AI can generate webhook DTOs quickly. It can create a controller quickly. It can even write a parser test from a sample payload. What it will often miss is the operational contract around that payload: retries, duplicates, concurrency, incomplete data, unknown event types, and the difference between a signal and a source of truth.
As AI accelerates implementation, engineers need to be more precise about these boundaries, not less. The value is no longer in typing the DTO. The value is in deciding what the DTO is allowed to do.
That is still engineering.
If you are building webhook integrations that quietly assume one delivery, one worker, and one perfect payload, share this with your team. More in the "Stop Lying to Your Stack" series.
