April 2022 – madhukar gilla

infrastructure failures/lease expiry/exception : if the receiver is not able to ack the receipt of a message, though it processed it successfully, most of the message brokers will re-enlist the message back into the queue again because of the “AtLeastOnceDelivery” model, as a result the receiver will re-process the duplicate message again !!! (ghost message).
Say in a method if we have the following logic a) SaveToDb b) Send an OrderSaved message, now what if one of them fails ? We would have published an OrderSaved message though our SaveToDb call failed/ SaveToDb was succeded but OrderSave message was never sent. (phantom record).

so how do we solve it ?

Make all the receiver Idempotent, basically if we process a message N no of times the result should be the same. Maintain a list of processed message Id’s or some property in a cache or a DB and before doing processing the message, check if its listed in the cache or DB, if it is don’t process it.
Use the OutBox pattern : in a nutshell the message to be sent is logged into a table (OutBox) using the same single DataStore transaction as SaveToDb. So if the SaveToDb failed the message will never be sent out. If the DataStore transaction succeeded the the message is then published in its own transaction along with marking the message table (OutBox) row as sent. If a failure happens then in the same transaction the message table row will be marked as not sent.