How to Design Reliable and Scalable Webhooks with RabbitMQ - FullStackMastery
  • Home  / 
  • RabbitMQ
  •  /  How to Design Reliable and Scalable Webhooks with RabbitMQ

How to Design Reliable and Scalable Webhooks with RabbitMQ

By Jim Liao / February 22, 2017

Webhooks are user-defined HTTP callbacks that are used to invoke behavior on another site when an event occurs. Webhooks are great for creating loosely coupled architectures for integrating multiple heterogenous systems. Unfortunately implementing webhooks requires you to handle multiple scenarios related to external system failures. In this video, I walk you through how to design scalable and reliable webhooks with RabbitMQ.

The challenge with implementing webhooks is that the external systems that you are calling can either be offline or unresponsive. In all failure cases, it is the responsibility of the webhook implementation to keep the event message and retry calling the destination system at a later time. By using RabbitMQ, you gain reliability by using message queues. By not removing messages from the message queue until it has been successfully processed by the external systems, you can guarantee that the event messages will not be dropped. By running multiple processes as consumers against the same message queue, you can gain scalability.

When you are considering implementing your own webhooks, consider using RabbitMQ or another software that provides message queues as a way to make your webhooks reliable.

I hope you enjoyed this presentation and make sure to download the slides using the link below.


Anonymous - February 26, 2017

Your style is unique compared to other folks I have read stuff from. Thank you for posting when you’ve got the opportunity,

Anonymous - February 26, 2017

My brother suggested I might like this blog. He was totally right. This post actually made my day. You can not imagine simply how much time I had spent for this info! Thanks!

Fredrik - November 30, 2017

This pattern chould be indeed useful of several reasons.

Let say that the site A is instead the client side and internal in the site A you have two stages, first one is the deliver/source of the events which could be fields/forms and the other one part is what you already have descibe with some sort of a queue delivery system in site A.
Then I could with this pattern now secure that if I would in “realtime” (danger word to use here) , update my changnings from my applications let say from the profile data I’ve changed in my app, which would be the payload and I send it to site B, throw this message queue (event queue), then I could be sure that the message is upload to the server (site B). If it fails ? well then I would recieve some fail message back and could store it in a retry later with reschedule mechanism that push it to the server again, when the server are “ready”.
Other reason for which the server is not ready is that the message was broken on the way from site A to site B, it need not only an ACK or NACK it also need some sort of confirm that the data was delivered correctly and fully with perhaps some sort of a checksum, perhaps.

What you think about that ?

Click here to add a comment

Leave a comment: