Persist pending requests on start up. Make Request Node Pending Store persistent through restarts.

[MantisClone]

Problem

When restarting a Request Node, all pending requests from before the restart are dropped, and none of them are persisted.

Details

There have been incidents where the Request Node will fail to persist requests for extended periods of time. When this happens, pending requests stay in the Pending Store. The common fix for these incidents is to restart the Request Node, but the pending requests are dropped as a result.

Proposed Solution

• Re-implement the pendingStore as an external Redis-based (in-memory) queue.
  • This ensures that requests are persisted in the order they are received and that nonces are not reused.
  • By moving it to an external queue, the Request Node can be restarted without dropping pending requests.

Implementation Options

• BeeQueue (Redis-based)
• BullMQ (Redis-based)
• Kafka Stream

Rejected Options

• Google PubSub - We don't want to vendor lock the Request Node implementation.