ActiveMQ (prefetch and asyncDispatch) negative destination inflight values in Jconsole explained

While tracking down an issue for a customer over the past few days I noticed the inflight count for my destination in jconsole has a negative value. On closer inspection, I found that the value was fluctuating wildly with negative values before settling down again to a more reasonable positive range. I took a detour to investigate and it turns out this behavior is expected. The negative values are the result of prefetch and asyncDispatch, let me explain with a little note to self:

The use case included a pre filled queue with ~30k messages and multiple(10) consumers which dequeued a small amount(again 10) of messages before disconnecting and immediately reconnecting. In this case, a prefetch value of 10 is ideal, but with the default prefetch value of 1000, the broker is busy dispatching messages to the consumer long after it has decided to quit. In addition, with asyncDispath, while dispatch to a consumer is instigated by the broker, the actual delivery is delegated, to the broker transport connection worker thread. This means that the delivery attempts back up on the individual transport connections rather than slowing down the broker.

The destination inflight count is a measure of the number of messages that have been dispatched by the broker but not yet acknowledged by any consumer. On each dispatch completion by the worker thread, the inflight value is incremented. The decrementing normally happens on a message acknowledge. In the event of a consumer closure with unconsumed messages, the remaining value is decremented when the consumer closes.

This is the crux. From the broker perspective, on consumer closure, it has dispatched 1000 messages and got an ack for 10 so it needs to decrement the inflight by 1090 990 (thanks for the correction Arjan). But from the perspective of the worker thread, busy doing the actual dispatch, it still has a lot of incrementing to do. The negative values arise from the consumer closure occurring before async dispatch is complete. When there are many concurrent consumers, the negative swing can be quite noticeable and quite large.

The good news is that this is perfectly fine, the books are kept in balance and there is eventual consistency. In addition, using either of a prefetch value of 10 or asyncDispatch=false ensures that the negative values do not occur as the broker is kept directly in step with message delivery to the consumer. In general though, the an appropriate value of prefetch is the correct solution if it is known in advance that a consumer will do work in batches.