Back to blog

Talking Webhooks on Ship It Weekly

webhooksreliabilityproduct

A while back, Brian Teller invited me onto Ship It Weekly for one of his Ship It Conversations episodes. We spent about half an hour talking about webhooks: why they break, what to do about it, and what it takes to run them in production without losing sleep.

I wanted to say thanks to Brian for having me on. He asked good questions, he clearly knew the territory, and he let me go on about retries for longer than most people would. If you work in DevOps or platform engineering, the show is worth a subscribe. Brian mixes weekly news recaps with longer conversations like this one.

You can listen to the episode here:

Here’s a short recap of what we covered, with links to our older posts if you want to go deeper on any of it.

A webhook is just a POST, until it isn’t

Brian opened with the point that a webhook looks like the simplest thing in the world. One app sends an HTTP POST to another app. Done.

The trouble starts when the receiving side is down, or slow, or gets the same event twice, or gets events in a different order than they happened. All of that lands on whoever owns the receiver. We talked about how a simple POST slowly grows into queues, retries and someone getting paged, and how most teams don’t notice that happening until something breaks. We wrote more about this in why webhook delivery is harder than you think.

Retries and duplicates

Most webhook senders promise at-least-once delivery. If they don’t hear back in time, they send it again. That’s good, because it means you don’t lose events. It also means you will get duplicates, and your handler needs to be fine with that.

We talked about deduplicating on an event ID, and about answering quickly and doing the slow work afterward, so the sender doesn’t time out and retry something you already got. Our posts on retry strategy and idempotency keys cover both sides of this.

Signatures and replay

Next was HMAC signatures. A signature tells you the request really came from the sender and wasn’t changed on the way. Adding a timestamp and an event ID on top of that stops someone from capturing a real request and sending it to you again later.

One point I tried to make: a valid signature only proves who sent the message. It doesn’t prove the thing in the message should happen. You still need to check that the order exists, the amount makes sense, and so on. If you’ve never set up signature checks, start with our HMAC post.

Testing on your own machine

Webhooks are annoying to test locally because the sender can’t reach localhost. People get by with tunnels, shared staging servers, or pasting copied payloads into a test. All of those work some of the time.

I talked about how the Hookbridge CLI handles this. You get a URL that doesn’t change, you point the sender at it, and the CLI forwards each request to your machine. Restart your laptop and the URL is still the same, so you don’t have to go update the webhook settings every morning.

What people get wrong in production

Brian asked what the common mistakes are. My short list was signature checks, being able to see what happened to every delivery, and keeping the receiver up. When a customer says “I never got that event,” you want to be able to look it up and answer in a minute, not dig through logs for an afternoon. We also touched on keeping those logs useful without storing sensitive data you didn’t need to keep. There’s more in our post on webhook observability.

Why I built Hookbridge

Brian also asked where Hookbridge came from. The honest answer is that every company I’ve worked at that used webhooks ran into the same problems, and each time we built the same fixes from scratch. After enough rounds of that, I decided to build it once, properly, so other teams wouldn’t have to.

We finished by talking about who uses Hookbridge today, including people running n8n and self-hosted agents like OpenClaw, and how pull endpoints let you receive webhooks without opening anything up to the internet.

Brian’s wrap-up summed it up better than I did: most webhook pain comes from only building the happy path. Plan for the failures first.

Thanks again

Thanks to Brian and Ship It Weekly for having me. It was a fun conversation, and it’s always nice to talk with someone who has been paged for the same kinds of problems.

We’ve got a couple of bigger things in the works that we’ll be sharing soon. In the meantime, if any of this sounds familiar, you can sign up for a free Hookbridge account and try it out.