feat(eventcore-fs): local-ingestion cursor for projections #398

jwilger · 2026-06-13T06:05:20-07:00

jwilger commented

2026-06-13 06:05:20 -07:00

Implements issue #391 / ADR-0043 part (c): a per-replica local-ingestion-order cursor so live cursor-based projections never rewind or miss events after a structural git merge.

Problem

EventReader::read_events keyed pagination on the canonical per-event UUID7. A git merge can union in a transaction whose canonical linearized position falls earlier than a cursor a projection already advanced past — so that event was silently skipped. That is read-model corruption.

Change

Each event is assigned a strictly increasing local-ingestion sequence the first time this replica sees it (append or merge-arrival), encoded into the high bytes of the StreamPosition UUID so positions sort by sequence.
The mapping is persisted in the gitignored, rebuildable index/ directory, so a projection checkpoint stays valid across reopen.
A merge-introduced event is new to this replica and receives a fresh, larger position, so the cursor advances forward to cover it and never rewinds.
read_events now delivers events in local-ingestion order and returns local-ingestion positions. read_stream is unchanged (canonical linearized order) and remains the convergent, cross-replica-identical view; merge-convergence tests assert convergence via read_stream.

Acceptance (ADR-0043)

A live projection that already advanced past a position still observes a later-arriving merge's compensation/diverged events — does not rewind, does not miss.
A projection that rebuilds on topology-generation change reaches the same final state.
Behaviour documented in blueprints/fs-merge-mode.md.

Tests

New projection_after_merge_test.rs: live_projection_does_not_miss_a_merge_introduced_earlier_event (red→green via forced adverse id ordering) and topology_rebuild_projection_matches_the_live_projection_state.
cargo nextest run --workspace: 236 passed. clippy/fmt clean.

Closes #391

Implements issue #391 / ADR-0043 part (c): a per-replica local-ingestion-order cursor so live cursor-based projections never rewind or miss events after a structural `git merge`. ## Problem `EventReader::read_events` keyed pagination on the canonical per-event UUID7. A `git merge` can union in a transaction whose canonical linearized position falls *earlier* than a cursor a projection already advanced past — so that event was silently skipped. That is read-model corruption. ## Change - Each event is assigned a strictly increasing **local-ingestion sequence** the first time *this* replica sees it (append or merge-arrival), encoded into the high bytes of the `StreamPosition` UUID so positions sort by sequence. - The mapping is persisted in the gitignored, rebuildable `index/` directory, so a projection checkpoint stays valid across reopen. - A merge-introduced event is new to this replica and receives a fresh, larger position, so the cursor advances forward to cover it and never rewinds. - `read_events` now delivers events in local-ingestion order and returns local-ingestion positions. `read_stream` is unchanged (canonical linearized order) and remains the convergent, cross-replica-identical view; merge-convergence tests assert convergence via `read_stream`. ## Acceptance (ADR-0043) - A live projection that already advanced past a position still observes a later-arriving merge's compensation/diverged events — does not rewind, does not miss. - A projection that rebuilds on topology-generation change reaches the same final state. - Behaviour documented in `blueprints/fs-merge-mode.md`. ## Tests - New `projection_after_merge_test.rs`: `live_projection_does_not_miss_a_merge_introduced_earlier_event` (red→green via forced adverse id ordering) and `topology_rebuild_projection_matches_the_live_projection_state`. - `cargo nextest run --workspace`: 236 passed. clippy/fmt clean. Closes #391

jwilger added 1 commit

2026-06-13 06:05:20 -07:00

feat(eventcore-fs): local-ingestion-order cursor for projections (ADR-0043)

CI / Detect Changes (pull_request) Successful in 3s

Details

CI / Request auto_review semantic review (pull_request) Successful in 1s

Details

CI / Format (pull_request) Successful in 14s

Details

auto_review auto_review: no findings

CI / Clippy (pull_request) Failing after 1m39s

Details

CI / Security Audit (pull_request) Successful in 23s

Details

CI / Test (pull_request) Successful in 2m46s

Details

CI / Mutation (pull_request) Has been skipped

Details

CI / CI Gate (pull_request) Failing after 2s

Details

d0aecc36a6

A git merge can union in a transaction whose canonical linearized position
falls earlier than a cursor a projection has already passed. The EventReader
cursor previously keyed on the canonical per-event UUID7, so such an event was
silently skipped — read-model corruption.

Track a per-replica local-ingestion order instead (ADR-0043 part c): each event
is assigned a strictly increasing local sequence the first time this replica
sees it, encoded into the high bytes of the StreamPosition UUID so positions
sort by sequence. The mapping is persisted in the gitignored, rebuildable
index/ directory, so checkpoints stay valid across reopen. A merge-introduced
event is new to this replica and gets a fresh, larger position, so a live
projection advances forward to cover it and never rewinds or misses it.

read_events (the projection/cursor path) now delivers events in local-ingestion
order and returns local-ingestion positions; read_stream remains canonical
linearized order and is the convergent, cross-replica-identical view. Merge
convergence tests assert convergence via read_stream accordingly.

Closes #391

auto-review requested changes

2026-06-13 06:05:41 -07:00

Dismissed

auto-review left a comment

This PR introduces a local-ingestion-order cursor to ensure projections do not miss events after a git merge. The changes appear well-implemented and tested, focusing on maintaining event order per replica. Ensure all parts of the codebase handle the new structures and return types correctly.

Walkthrough

blueprints/fs-merge-mode.md: Updated documentation to describe the new local-ingestion order and its implications.
eventcore-fs/src/config.rs: Added index directory to Roots struct for storing ingestion logs.
eventcore-fs/src/index.rs: Introduced by_ingestion vector to maintain local-ingestion order and updated index building logic.
eventcore-fs/src/ingestion.rs: New module to handle local-ingestion order, including sequence allocation and persistence.
eventcore-fs/src/lib.rs: Updated EventStore and EventReader implementations to use local-ingestion order.
eventcore-fs/tests/merge_mode_test.rs: Adjusted tests to verify canonical order and local-ingestion behavior.
eventcore-fs/tests/projection_after_merge_test.rs: New tests to ensure projections handle merge-introduced events correctly.

LLM usage and cost

Reasoning (gpt-4o) in=10054 out=784 cost=$0.062030
Cheap (gpt-4o-mini) in=9294 out=161 cost=$0.001491
Estimated total USD: $0.063521 via https://api.openai.com and https://api.openai.com

Pre-merge checks

PR metadata quality: failed
Rationale: PR title exceeds the 72 character limit.
Offending text: feat(eventcore-fs): local-ingestion-order cursor for projections (ADR-0043)

This PR introduces a local-ingestion-order cursor to ensure projections do not miss events after a `git merge`. The changes appear well-implemented and tested, focusing on maintaining event order per replica. Ensure all parts of the codebase handle the new structures and return types correctly. ## Walkthrough - **blueprints/fs-merge-mode.md**: Updated documentation to describe the new local-ingestion order and its implications. - **eventcore-fs/src/config.rs**: Added `index` directory to `Roots` struct for storing ingestion logs. - **eventcore-fs/src/index.rs**: Introduced `by_ingestion` vector to maintain local-ingestion order and updated index building logic. - **eventcore-fs/src/ingestion.rs**: New module to handle local-ingestion order, including sequence allocation and persistence. - **eventcore-fs/src/lib.rs**: Updated `EventStore` and `EventReader` implementations to use local-ingestion order. - **eventcore-fs/tests/merge_mode_test.rs**: Adjusted tests to verify canonical order and local-ingestion behavior. - **eventcore-fs/tests/projection_after_merge_test.rs**: New tests to ensure projections handle merge-introduced events correctly. ## LLM usage and cost - Reasoning (gpt-4o) in=10054 out=784 cost=$0.062030 - Cheap (gpt-4o-mini) in=9294 out=161 cost=$0.001491 Estimated total USD: $0.063521 via https://api.openai.com and https://api.openai.com ## Pre-merge checks - PR metadata quality: failed - Rationale: PR title exceeds the 72 character limit. - Offending text: feat(eventcore-fs): local-ingestion-order cursor for projections (ADR-0043)

jwilger added 1 commit

2026-06-13 06:25:22 -07:00

fix(eventcore-fs): use sort_by_key for ingestion order (clippy)

CI / Detect Changes (pull_request) Successful in 3s

Details

CI / Request auto_review semantic review (pull_request) Successful in 2s

Details

CI / Format (pull_request) Successful in 15s

Details

CI / Clippy (pull_request) Successful in 1m53s

Details

CI / Security Audit (pull_request) Successful in 24s

Details

CI / Test (pull_request) Successful in 2m49s

Details

CI / Mutation (pull_request) Has been skipped

Details

CI / CI Gate (pull_request) Successful in 2s

Details

auto_review auto_review: 3 warnings

c8c8a56f2b

clippy::unnecessary_sort_by (latest stable) flagged the manual comparator in
build_index; sort the local-ingestion positions with sort_by_key instead.

auto-review requested changes

2026-06-13 06:25:31 -07:00

auto-review left a comment

The PR introduces a local-ingestion-order cursor to ensure projections do not miss events after a git merge. The change to sort_by_key in index.rs improves readability and performance. The update appears safe to merge.

Walkthrough

Δ since `d0aecc3`:

eventcore-fs/src/index.rs:
- Changed sort_by to sort_by_key for sorting events by position, enhancing code clarity and efficiency.

LLM usage and cost

Reasoning (gpt-4o) in=1844 out=190 cost=$0.012070
Estimated total USD: $0.012070 via https://api.openai.com

Pre-merge checks

PR metadata quality: failed
Rationale: PR title exceeds the 72 character limit.
Offending text: feat(eventcore-fs): local-ingestion-order cursor for projections (ADR-0043)

The PR introduces a local-ingestion-order cursor to ensure projections do not miss events after a `git merge`. The change to `sort_by_key` in `index.rs` improves readability and performance. The update appears safe to merge. ## Walkthrough ### Δ since d0aecc3: - **eventcore-fs/src/index.rs**: - Changed `sort_by` to `sort_by_key` for sorting events by position, enhancing code clarity and efficiency. ## LLM usage and cost - Reasoning (gpt-4o) in=1844 out=190 cost=$0.012070 Estimated total USD: $0.012070 via https://api.openai.com ## Pre-merge checks - PR metadata quality: failed - Rationale: PR title exceeds the 72 character limit. - Offending text: feat(eventcore-fs): local-ingestion-order cursor for projections (ADR-0043)

jwilger changed title from ~~feat(eventcore-fs): local-ingestion-order cursor for projections (ADR-0043)~~ to feat(eventcore-fs): local-ingestion cursor for projections

2026-06-13 06:30:16 -07:00

jwilger commented

2026-06-13 06:30:42 -07:00

@auto-review re-review

The only blocking item was the PR title length; it has been shortened to 58 characters (ADR reference moved to the body).

@auto-review re-review The only blocking item was the PR title length; it has been shortened to 58 characters (ADR reference moved to the body).

auto-review commented

2026-06-13 06:30:42 -07:00

Queued a forced review at c8c8a56f2b. This intentionally bypasses CI gating; watch the commit-status badge for progress.

Queued a forced review at c8c8a56f2beeeba2076b52b076ad6a90c84eaebf. This intentionally bypasses CI gating; watch the commit-status badge for progress.

auto-review approved these changes

2026-06-13 06:30:52 -07:00

auto-review left a comment

This PR introduces a local-ingestion order cursor to ensure projections do not miss events after a git merge. The changes appear well-structured and include comprehensive tests to validate the new behavior. Ensure that the persistence of the ingestion log is robust to prevent data corruption.

Walkthrough

blueprints/fs-merge-mode.md: Updated documentation to describe the new local-ingestion order and its implications.
eventcore-fs/src/config.rs: Added a new index directory to the Roots struct for storing ingestion logs.
eventcore-fs/src/index.rs: Modified the index to track events in local-ingestion order and updated the build_index function to handle ingestion logs.
eventcore-fs/src/ingestion.rs: Introduced the IngestionLog struct to manage local-ingestion sequences and their persistence.
eventcore-fs/src/lib.rs: Updated append_events and merge methods to persist ingestion logs.
eventcore-fs/tests/projection_after_merge_test.rs: Added tests to ensure projections do not miss events after a merge and that topology rebuilds match live projection states.

LLM usage and cost

Reasoning (gpt-4o) in=10657 out=745 cost=$0.064460
Cheap (gpt-4o-mini) in=10514 out=115 cost=$0.001646
Estimated total USD: $0.066106 via https://api.openai.com and https://api.openai.com

This PR introduces a local-ingestion order cursor to ensure projections do not miss events after a `git merge`. The changes appear well-structured and include comprehensive tests to validate the new behavior. Ensure that the persistence of the ingestion log is robust to prevent data corruption. ## Walkthrough - **blueprints/fs-merge-mode.md**: Updated documentation to describe the new local-ingestion order and its implications. - **eventcore-fs/src/config.rs**: Added a new `index` directory to the `Roots` struct for storing ingestion logs. - **eventcore-fs/src/index.rs**: Modified the index to track events in local-ingestion order and updated the `build_index` function to handle ingestion logs. - **eventcore-fs/src/ingestion.rs**: Introduced the `IngestionLog` struct to manage local-ingestion sequences and their persistence. - **eventcore-fs/src/lib.rs**: Updated `append_events` and `merge` methods to persist ingestion logs. - **eventcore-fs/tests/projection_after_merge_test.rs**: Added tests to ensure projections do not miss events after a merge and that topology rebuilds match live projection states. ## LLM usage and cost - Reasoning (gpt-4o) in=10657 out=745 cost=$0.064460 - Cheap (gpt-4o-mini) in=10514 out=115 cost=$0.001646 Estimated total USD: $0.066106 via https://api.openai.com and https://api.openai.com

eventcore-fs/src/ingestion.rs

					
				@ -0,0 +159,4 @@

				        // be strictly greater than the nil UUID so the first event is reachable.

				        assert!(position_from_seq(1) > Uuid::nil());

				    }

				}

🟡 Warning: The IngestionLog struct and its methods are central to maintaining the local-ingestion order. Ensure that the log is correctly persisted and loaded, especially in scenarios involving concurrent writes or failures.

🟡 **Warning:** The `IngestionLog` struct and its methods are central to maintaining the local-ingestion order. Ensure that the log is correctly persisted and loaded, especially in scenarios involving concurrent writes or failures.

eventcore-fs/src/lib.rs

					
				@ -316,26 +325,31 @@ impl EventReader for FileEventStore {

				        let prefix = filter.stream_prefix();

🟡 Warning: The append_events method now persists new entries to the ingestion log. Ensure that this persistence is atomic and handles failures gracefully to prevent data corruption.

🟡 **Warning:** The `append_events` method now persists new entries to the ingestion log. Ensure that this persistence is atomic and handles failures gracefully to prevent data corruption.

eventcore-fs/src/lib.rs

					
				@ -539,15 +553,24 @@ impl FileEventStore {

				                detail: "index lock poisoned".to_string(),

🟡 Warning: The merge method now persists new entries to the ingestion log. Ensure that this persistence is atomic and handles failures gracefully to prevent data corruption.

🟡 **Warning:** The `merge` method now persists new entries to the ingestion log. Ensure that this persistence is atomic and handles failures gracefully to prevent data corruption.