-
Notifications
You must be signed in to change notification settings - Fork 116
Description
Reconciliation Refactor
The current reconciliation pipeline has accumulated complexity after several rounds of fixes for truncate behaviour, optimistic rebasing, and duplicate-suppression bugs. Maintaining it has become risky: much of the flow is synchronous/CPU-bound but still entwined with batching flags and per-key delete replays for truncates, making the code brittle even without heavy IO. This plan aligns the system around a simpler contract—explicit truncate events, a single optimistic rebase path, and a shared change stream—so we can reason about the data flow, reduce footguns, and unblock future improvements.
1. Goals & Scope
- Reduce reconciliation complexity while preserving correctness guarantees: no missing/phantom events, consistent indexes, stable lifecycle state.
- Unify the change stream for live queries and index maintenance, adding explicit
truncatesemantics. - Keep optimistic UI responsive with a predictable, easy-to-audit rebasing routine.
- Make the observable behaviour for
insert/update/deletecompatible with today’s tests, while documenting any intentional relaxations (e.g. allowing extraupdatenoise when values truly change).
Out of scope: rewriting transaction orchestration, changing persistence APIs, or altering subscription filtering beyond what the new event contract requires.
2. Change-Event Contract
- Extend
ChangeMessage.typetoinsert | update | delete | truncate.truncatehas shape{ type: 'truncate', metadata? }with nokey,value, orpreviousValue.insert/update/deleteretain their current fields.
CollectionChangesManager.emitEventsandCollectionSubscription.filterAndFlipChangestreattruncateas a stream-wide reset:- emit it immediately (bypass deferral—don't wait for pending sync transactions),
- skip key-based flipping logic,
- allow subsequent inserts in the same batch (from the same commit),
- expect subscriptions to clear caches and live queries to resnapshot on receiving
truncate.
- Indexes gain a
reset()entry point so a singletruncateclears their internal state before replaying subsequent inserts. - Live-query consumers clear their local caches and resnapshot on
truncaterather than depending on synthetic delete batches. Add regression tests for this behaviour.
3. Core State & Simplifications
- Retain
syncedData,syncedMetadata,transactions,optimisticUpserts,optimisticDeletes, andsize. - Drop persistent
recentlySyncedKeys/lastSyncContext. Callers passsyncedKeysintorebaseOptimistic; for sync commits this set equals the keys present insyncedDiff. - During a sync commit build
syncedDiff: Map<TKey, { before?: TOutput; after?: TOutput }>while applying server operations. This map drives event generation, duplicate suppression, and providessyncedKeysfor the optimistic pass. - Capture
visibleBeforeByKeyper commit (by callingthis.get(key)before applying changes). No long-lived map required.
4. High-Level Flow
4.1 Local transaction changes
- Transaction mutates →
onTransactionStateChangefires. shouldBatchEvents = pendingSyncedTransactions.length > 0(unchanged heuristic).- Invoke
rebaseOptimistic({ reason: 'local-change', truncate: false, syncedKeys: emptySet, syncedDiff: null }). - Using the returned events: update
size, callindexes.updateIndexes(events), then emit (forceEmit = shouldBatchEvents === false).
4.2 Sync transaction commit
- Collect & capture
- Collapse committed pending sync transactions into an ordered list of operations plus
truncateSeen. - Build
visibleBeforeByKeyby callingthis.get(key)for each touched key before mutating state.
- Collapse committed pending sync transactions into an ordered list of operations plus
- Apply authoritative changes
- If
truncateSeen, push atruncateevent intosyncEvents, clearsyncedData,syncedMetadata, andsyncedKeys, and mark that indexes mustreset()before processing subsequent events. - For each operation:
- Update
syncedMetadataandsyncedData(respectingrowUpdateMode). - Record
{ before, after }insyncedDiffusingvisibleBeforeByKeyand the mutated authoritative value. - Add key to
syncedKeys. - If key has an optimistic overlay (in
optimisticDeletesoroptimisticUpserts): skip creating asyncEvent. The rebase in phase 3 will handle this key since it's insyncedKeysand will generate the appropriate event for the user-visible change. - Otherwise push
{ type, key, value: type === 'delete' ? before : after, previousValue: before }intosyncEvents.
- Update
- If
- Reapply optimistic layer & emit once
- Call
rebaseOptimistic({ reason: 'sync-commit', truncate: truncateSeen, syncedKeys, syncedDiff })to rebuild optimistic state and obtainoptimisticEvents. - Concatenate
syncEvents+optimisticEventsinto a single ordered batch. - Use that batch to: (a) reset indexes if needed, (b) call
indexes.updateIndexes(batch), (c) adjustsize(truncate⇒ reset to 0 first, then apply per-event deltas), and finally (d) emit withforceEmit = true. - Lifecycle hooks (
markReady, first-commit callbacks, GC timers) run after the combined batch is emitted.
- Call
5. rebaseOptimistic Specification
Inputs
reason:'local-change' | 'sync-commit'(diagnostic only).truncate: boolean.syncedKeys:Set<TKey>(empty for local-only recomputes).syncedDiff:Map<TKey, { before?: TOutput; after?: TOutput }>ornull.
Algorithm
-
Snapshot previous optimistic state:
- If
truncate, treatpreviousUpserts/previousDeletesas empty. - Otherwise reference the current
optimisticUpserts/optimisticDeletes(no cloning yet).
- If
-
Seed new maps:
nextUpsertsstarts as a shallow clone ofpreviousUpserts, excluding keys insyncedKeys(because these keys' authoritative base just changed and must be recomputed).nextDeletesclonespreviousDeletesand drops entries insyncedKeysfor the same reason.- Keys NOT in
syncedKeyspreserve their object references, keeping the "same ref ⇒ no event" fast path.
-
Apply active transactions (in
createdAtorder viaSortedMap).- Optimization: when
reason === 'sync-commit', only reprocess mutations whose keys are insyncedKeys(their authoritative base changed). Keys outside that set were already cloned verbatim in step 2. - Inserts/updates: merge with authoritative base (
syncedData) when needed and write intonextUpserts. We do not run a deep equality check afterwards—the merged object becomes the new reference. Trade-off: this can emit extraupdateevents compared to today’s deep-equals suppression, but it removes one of the most expensive comparisons from the hot path. Existing tests will highlight any unacceptable noise. - Deletes: remove from
nextUpserts, add tonextDeletes.
Frequency note:
rebaseOptimisticruns after each transaction mutate/cleanup and directly after each sync commit. By restricting the sync-commit case tosyncedKeys, we avoid full rescans of unrelated optimistic data. - Optimization: when
-
Swap
this.optimisticUpserts/this.optimisticDeleteswith the new maps. -
Derive events using the matrix below. Evaluate
authoritativeAfter(key)assyncedDiff?.get(key)?.after ?? this.syncedData.get(key)andauthoritativeBefore(key)assyncedDiff?.get(key)?.before ?? this.syncedData.get(key).Note: For sync commits,
syncedDiffcontains the capturedbeforefromvisibleBeforeByKey; for local changes, both fall back to the unchangedthis.syncedData.
| Previous \ New | new upsert present | new delete present | not in new optimistic state |
|---|---|---|---|
| previous upsert present | • same ref ⇒ no event • different ref ⇒ update (value: new upsert, previousValue: old upsert) |
delete (value: previous upsert) |
• key ∈ syncedKeys AND previous upsert === authoritativeAfter ⇒ no event • authoritativeAfter exists ⇒ update (value: authoritativeAfter, previousValue: previous upsert)• no authoritative ⇒ delete (value: previous upsert) |
| previous delete present | insert (value: new upsert) |
no event | • authoritativeAfter exists ⇒ insert (value: authoritativeAfter)• no authoritative ⇒ no event |
| not previously optimistic | • authoritativeBefore exists ⇒ update (value: new upsert, previousValue: authoritativeBefore)• no authoritativeBefore ⇒ insert (value: new upsert) |
• authoritativeBefore exists ⇒ delete (value: authoritativeBefore)• no authoritativeBefore ⇒ no event |
— |
Notes:
- The matrix shows which event type to emit and what values to use based on the before/after state
- "Same ref" comparison uses
===(reference equality) - The deduplication check for synced keys (row 1, col 3) prevents duplicate events when an optimistic change matches what the server committed
- "no event" cases occur when there's no user-visible change (undefined → undefined, or same reference)
- Return
{ events, touchedKeys: allTouched }, whereallTouchedis the union of all keys frompreviousUpserts,nextUpserts,previousDeletes,nextDeletes, andsyncedKeys. The caller uses this for index updates and size deltas.
6. Metadata, Indexes, and Size
- Metadata updates stay in the authoritative apply phase.
truncateclears the entire map; inserts/updates refresh entries; deletes remove them. - Index workflow per emission:
- Inspect the batch; if it contains a
truncate, callindexes.reset()once before applying subsequent events. - Call
indexes.updateIndexes(events)with the full ordered batch (local or sync).
- Inspect the batch; if it contains a
sizemaintenance mirrors the index workflow:- When a batch contains
truncate, resetsize = 0before applying deltas. - For each event in the batch:
insert→+1,delete→-1,update→0. - Apply the net delta once per batch (local or sync) before emitting to listeners.
- When a batch contains
7. Duplicate Suppression Workflow
- Authoritative phase: we omit per-key sync events when there is an active optimistic overlay, ensuring the optimistic layer owns emission for those keys.
- Optimistic phase: during matrix evaluation we skip events for
syncedKeyswhen the optimistic state now matches the authoritativeafter. This covers cases where a server commit completes an optimistic mutation without re-emitting the same payload. - Combined effect: when the server value differs from the optimistic one, the optimistic pass emits the change; when they match, neither phase emits duplicates. Tests should include examples such as an optimistic insert followed by an identical server insert.
8. Testing & Validation
- Unit tests for
rebaseOptimisticcovering:- Insert/update/delete permutations, rollback scenarios, truncate rebuild.
- Behaviour when
syncedKeysis empty vs populated. - Cases where optimistic overlays exist during sync commits to confirm the deduplication flow described above.
- Integration tests for sync flow:
- Server-only updates.
- Server updates clashing with optimistic inserts and deletes.
- Truncate followed by rebuild: subscribers see
[truncate, …inserts], indexes reset once,sizereturns to match the new data.
- Regression tests for transaction rollback and concurrent mutations to confirm we still emit the correct rebased sequences.
- Subscription tests verifying live queries clear and resnapshot on
truncate.
9. Implementation Order
- Land the event-contract updates (
ChangeMessagetype, subscription/index handling, live-query truncate support) with accompanying tests. - Refactor live query/subscription plumbing to honour
truncate(see §10) while keeping existing reconciliation logic. - Replace the reconciliation internals (phased commit + new
rebaseOptimistic) once the new event contract and live-query plumbing are in place.
This sequencing keeps the blast radius manageable: we teach every consumer about truncate first, then simplify the reconciliation logic.
10. Live Query & Subscription Refactor
CollectionChangesManager adjustments
- Detect
truncatein every emission. When present, bypass batching (forceEmitsemantics) and deliver the event immediately after notifying subscribers to reset state. - Introduce a
handleTruncate()helper that iterates active subscriptions before emitting. This helper should:- call each subscription’s
handleTruncatehook (see below), - reuse the existing
cleanup()logic for batching (clearbatchedEvents, setshouldBatchEvents = false) instead of duplicating that reset code, - treat the emission as a synchronous flush (no microtasks) so subsequent inserts flow through as a fresh stream.
- call each subscription’s
- When
truncateis emitted alongside other changes in the same batch, ensure the single-event appears first so subscribers reset prior to applying new inserts. - Keep the reset logic synchronous—no microtasks between
handleTruncate()and event emission—so there is no window where consumers see post-truncate inserts before their local caches clear.
CollectionSubscription adjustments
- Add a
handleTruncate()method that reuses the same state-reset logic we hit during unsubscribe/cleanup (factor that code into a shared helper):- clear
sentKeys, - reset
snapshotSentandloadedInitialStateflags so the next snapshot behaves like an initial load, - forward the
truncatechange unfiltered to the callback (skipfilterAndFlipChanges).
- clear
- Update
emitEventsto handletruncatespecially when present in a batch:- call
handleTruncate()once, - emit the batch as-is (truncate should already be first), bypassing standard flip/filter logic,
- subsequent inserts in the same batch flow through unfiltered since subscribers just reset their state.
- call
- Ensure ordered snapshot helpers (
requestSnapshot,requestLimitedSnapshot) remain valid by recognising a truncate-induced reset and re-requesting data when necessary.
Live query consumers
- The
Collectionsurface that exposes live queries should react totruncateby:- invoking the existing cleanup/reset helpers (the same ones used when the collection is GC’d) to clear cached results or memoized selectors,
- prompting a resubscription/snapshot if auto-loading is enabled,
- exposing the raw
truncateevent to consumers who wish to show loading indicators.
- Update docs and typings so downstream code knows
ChangeMessage['type']includestruncatewith no key/value payload.
Testing checklist
- Unit test
CollectionSubscription.emitEventscovering truncate-only, truncate+insert, and subsequent snapshot requests. - Integration test change streams to ensure indexes, subscribers, and live queries all observe
[truncate, insert…]and rehydrate correctly. - Regression test batching logic: pending sync transactions should not defer a truncate emission.
Appendix A – Legacy Code to Retire
Once the refactor lands, these pieces of the current implementation become obsolete and should be removed or folded into the new flow:
CollectionStateManager.recentlySyncedKeysand all logic manipulating it (state.ts:51, 272-280, 780-787, 835-838). Per-callsyncedKeysreplaces this set.CollectionStateManager.preSyncVisibleStateandcapturePreSyncVisibleState(state.ts:50, 464-475, 777-858). Visibility is now captured inline during each commit.CollectionStateManager.isCommittingSyncTransactionsguard (state.ts:52, 219-222, 657-659). Sequential processing makes the flag redundant.- Legacy
commitPendingTransactionsbody (state.ts:411-792), including truncate reapply logic andcompletedOptimisticOpsdedupe (state.ts:480-757). The new three-phase commit replaces it outright. CollectionStateManager.recomputeOptimisticState,collectOptimisticChanges, andgetPreviousValuehelpers (state.ts:216-408). These give way to the newrebaseOptimisticimplementation.CollectionStateManager.calculateSizehelper (state.ts:333-346). Size will be maintained via per-event deltas.deepEqualsimport and reconciler-only comparisons (state.ts:1, 718-751). Remove or relocate if no other call sites remain.CollectionChangesManagerbatching state beyond a simple flush trigger (changes.ts:24, 57-86, 161-164). Single-batch emission lets us collapse this.- Transaction hook that eagerly calls
commitPendingTransactionswhen sync transactions exist (transactions.ts:396-420). After the refactor,touchCollectionshould only trigger the optimistic rebase. - Tests or utilities referencing the removed fields (e.g.
recentlySyncedKeys,preSyncVisibleState). Update or drop them alongside the code changes.
Take care to audit tests for references to these structures so they are updated or removed alongside the implementation changes.
Appendix B – Critical Comparison
Advantages of the proposed design
- Unified change stream: explicit
truncateevents and a single optimistic rebase pipeline simplify reasoning and remove the need for ad-hoc per-key delete batches. - Deterministic batching: emitting one consolidated batch per sync/local change eliminates race-prone batching flags and microtask juggling.
- Lower state surface area: transient structures (
recentlySyncedKeys,preSyncVisibleState) disappear, reducing the chance of stale bookkeeping bugs. - Testability: clearer phases and smaller helpers make it easier to unit-test rebase logic, truncate handling, and change emission in isolation.
- Explicit contracts: downstream consumers (indexes, live queries) receive a documented
truncatesignal instead of relying on incidental behaviour.
Drawbacks / risks introduced
- Behavioural changes: accepting extra
updatenoise (no deep-equality guard) may break assumptions in downstream consumers; we gain performance and simplicity but risk potential UI churn. - Plan complexity: the new logic still requires careful coordination between sync events and optimistic rebase; implementing the matrix correctly is non-trivial and mistakes could regress visibility.
- Transition cost: migrating live queries, indexes, and tests to the new event contract demands significant refactoring before the benefits appear.
- Synchronous flush requirement: ensuring truncate resets run synchronously tightens timing constraints; missing a synchronous path could reintroduce the race conditions we are trying to avoid.
- New abstractions: introducing shared helpers (e.g.,
handleTruncate) adds indirection; if not carefully factored, we may end up with another layer of hard-to-follow logic.
Advantages of the current implementation
- Battle tested: the existing code reflects multiple rounds of bug fixes and is currently passing all production tests.
- Granular deduplication: deep-equality checks and multiple suppression layers avoid emitting redundant updates, keeping downstream churn low.
- Incremental rollouts: the current structure supports partial fixes without rewriting the whole pipeline, which can be safer for urgent patches.
Drawbacks of the current implementation
- Diffuse responsibilities: reconcile logic spans
recomputeOptimisticState,commitPendingTransactions, and transaction hooks, making reasoning difficult. - Hidden coupling: batching flags (
shouldBatchEvents,isCommittingSyncTransactions) and microtask clears (recentlySyncedKeys) act as implicit coordination mechanisms that are easy to break. - Truncate fragility: simulating truncates through per-key deletes causes large event bursts and requires careful reapplication of optimistic state, a common source of bugs.
- State sprawl: numerous maps/sets must stay in sync; drift between them leads to subtle visibility issues.
- Testing blind spots: with responsibilities split across several entry points, it is difficult to isolate behaviour in tests, leading to regression risks.
The choice is between living with today’s complexity and ad-hoc fixes, or investing in a clearer contract that may introduce short-term churn (extra updates, migration cost) but promises simpler reasoning and future enhancements.