Skip to content

Conversation

@FGasper
Copy link
Collaborator

@FGasper FGasper commented Nov 12, 2025

This adds an opt-in mode to tail the oplog directly instead of reading a change stream. Oplog tailing is less resource-intensive on the server than a change stream, which yields better performance and allows verification of more migrations.

Oplog tailing carries a few caveats (or else it would be the default when possible):

  • The oplog’s format is undocumented, by design. A future MongoDB version may not work with the present logic. Change streams, by contrast, are fully documented and intended for use in applications like Migration Verifier.
  • To capture transactions that are in progress when verification starts, oplog tailing actually starts from the start optime of the oldest transaction that’s in progress when verification starts. This means a few extra rechecks may be enqueued at the start of a verification.

Additional notes:

  • Since migration-verifier always reads with majority read concern, we can ignore the term and hash that normally, with the timestamp, comprise an oplog entry’s “optime”.
  • ChangeStreamReader.start() is now part of ChangeReaderCommon so that its retry bits can be shared with the new OplogReader type.
  • This includes a nascent agg library, which simplifies aggregation usage in Go by providing types for various expressions.
  • The retryer now wraps errors with the function description message.
  • To ensure that OplogReader doesn’t read too far “back in time”, the integration test suite now kills all transactions between test methods.
  • Some of the tests are reordered a bit so that document insertion happens before verifier creation. This was originally done to troubleshoot test failures but seems worth retaining for general clarity.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant