feat(events): migrate event store to eventcore 0.9 with eventcore-fs #117

jwilger · 2026-06-13T09:19:43-07:00

jwilger commented

2026-06-13 09:19:43 -07:00

Summary

Replaces EMC's bespoke filesystem event log (model/events/v1/*.json, a custom
parent-linked event DAG with hand-rolled conflict detection) and the
external SQLite operational store with a single committed, git-mergeable
eventcore_fs::FileEventStore rooted at model/events. Upgrades to
eventcore 0.9 and drops eventcore-sqlite.

Dependencies: eventcore = 0.9, add eventcore-fs, eventcore-types;
remove eventcore-sqlite.
Storage: commands execute against a FileEventStore; only
model/events/events/*.jsonl (immutable per-transaction files) is committed.
The store writes its own .gitignore/.gitattributes; tmp/, index/,
.eventcore/, locks/, and .lock stay machine-local. No more SQLite cache
outside the repo, and no more custom JSON export.
Projection: ProjectedModel::from_events now folds the persisted
EmcEvent stream read back from the store; Lean/Quint/review artifacts are
regenerated exactly as before.
Conflicts: EMC's semantic conflict (two concurrent edits of the same
workflow/slice) maps onto eventcore-fs's structural fork (two transactions
extending a stream from the same base version). emc list conflicts reports
forks; emc resolve conflict <stream-id> <branch-transaction-id> reconciles by
keeping the chosen branch (a recorded N-parent merge transaction).
Fidelity: EmcEvent::WorkflowConnected and
EmcEvent::WorkflowTransitionEvidenceAdded were lossy relative to the rich
exported payloads (they dropped source_control/target_view). Both are now
enriched so navigation transitions survive the store round-trip — this was the
whole event type that previously only round-tripped through the JSON log.
Removes the now-dead ExportedEvent envelope/DAG/bespoke-conflict code while
keeping ExportedEventBody and its codecs (still used on the write path and
inside SliceFactEvent).

Rationale

eventcore 0.9 ships eventcore-fs, a file-based, git-mergeable event store that
provides exactly the offline-collaboration merge semantics EMC had been
approximating with its own committed JSON DAG. Adopting it removes a whole
parallel persistence layer (the SQLite operational store plus the custom export
format and conflict detector), so there is one source of truth that both
validates commands (eventcore command invariants) and drives projection, and the
merge story is owned by the library.

Verification

cargo fmt --all --check, cargo clippy --all-targets --all-features -- -D warnings, cargo build, and the copyright-headers check all pass.
Full cargo test suite is green, including the cucumber and MCP integration
suites.
New behavioral coverage: a two-replica git-merge scenario that produces a fork
and reconciles it via the chosen branch; the existing check_rebuilds_* suite
verifies emc check regenerates every artifact family from the store.

🤖 Generated with Claude Code

## Summary Replaces EMC's bespoke filesystem event log (`model/events/v1/*.json`, a custom parent-linked event DAG with hand-rolled conflict detection) **and** the external SQLite operational store with a single committed, git-mergeable `eventcore_fs::FileEventStore` rooted at `model/events`. Upgrades to eventcore 0.9 and drops `eventcore-sqlite`. - **Dependencies:** `eventcore = 0.9`, add `eventcore-fs`, `eventcore-types`; remove `eventcore-sqlite`. - **Storage:** commands `execute` against a `FileEventStore`; only `model/events/events/*.jsonl` (immutable per-transaction files) is committed. The store writes its own `.gitignore`/`.gitattributes`; `tmp/`, `index/`, `.eventcore/`, `locks/`, and `.lock` stay machine-local. No more SQLite cache outside the repo, and no more custom JSON export. - **Projection:** `ProjectedModel::from_events` now folds the persisted `EmcEvent` stream read back from the store; Lean/Quint/review artifacts are regenerated exactly as before. - **Conflicts:** EMC's semantic conflict (two concurrent edits of the same workflow/slice) maps onto eventcore-fs's structural **fork** (two transactions extending a stream from the same base version). `emc list conflicts` reports forks; `emc resolve conflict <stream-id> <branch-transaction-id>` reconciles by keeping the chosen branch (a recorded N-parent merge transaction). - **Fidelity:** `EmcEvent::WorkflowConnected` and `EmcEvent::WorkflowTransitionEvidenceAdded` were lossy relative to the rich exported payloads (they dropped `source_control`/`target_view`). Both are now enriched so navigation transitions survive the store round-trip — this was the whole event type that previously only round-tripped through the JSON log. - Removes the now-dead `ExportedEvent` envelope/DAG/bespoke-conflict code while keeping `ExportedEventBody` and its codecs (still used on the write path and inside `SliceFactEvent`). ## Rationale eventcore 0.9 ships `eventcore-fs`, a file-based, git-mergeable event store that provides exactly the offline-collaboration merge semantics EMC had been approximating with its own committed JSON DAG. Adopting it removes a whole parallel persistence layer (the SQLite operational store plus the custom export format and conflict detector), so there is one source of truth that both validates commands (eventcore command invariants) and drives projection, and the merge story is owned by the library. ## Verification - `cargo fmt --all --check`, `cargo clippy --all-targets --all-features -- -D warnings`, `cargo build`, and the copyright-headers check all pass. - Full `cargo test` suite is green, including the cucumber and MCP integration suites. - New behavioral coverage: a two-replica git-merge scenario that produces a fork and reconciles it via the chosen branch; the existing `check_rebuilds_*` suite verifies `emc check` regenerates every artifact family from the store. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

jwilger added 2 commits

2026-06-13 09:19:43 -07:00

feat(events): migrate event store to eventcore 0.9 + eventcore-fs (WIP: source) b97400351d

Replace the custom model/events/v1 JSON event DAG and the external SQLite
operational store with a single committed, git-mergeable
eventcore_fs::FileEventStore rooted at model/events.

- Cargo: eventcore 0.9, eventcore-fs, eventcore-types; drop eventcore-sqlite.
- event_runtime: execute commands against FileEventStore; read_all_emc_events,
  list_forks, reconcile_choose_branch; remove SQLite/sync/repair.
- shell: ExportEvent runs the command (store writes files); no custom JSON.
- events: ProjectedModel::from_events consumes EmcEvent; conflicts via forks;
  fingerprint from sorted per-event content digests; delete ExportedEvent
  envelope/DAG/bespoke-conflict code (ExportedEventBody codecs retained).
- EmcEvent::WorkflowConnected enriched with source_control/target_view so the
  projection keeps full navigation-transition fidelity.

Library and binary build green with no warnings. Test suites still reference
the removed JSON/SQLite layer and are rewritten in a follow-up commit.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

test(events): rewrite event/runtime tests for the eventcore-fs store

CI / Nix flake check (pull_request) Successful in 3m31s

Details

CI / Request auto_review semantic review (pull_request) Successful in 2s

Details

auto_review auto_review: 5 warnings

CI / Rust CI (pull_request) Failing after 14m44s

Details

c3e332a0f1

Replace tests coupled to the removed model/events/v1 JSON envelope and SQLite
cache with behavioral coverage of the eventcore-fs-backed runtime.

- event_runtime_external_tests: FileEventStore behavior (in-repo store, append +
  read-back ordering, command invariants) and a two-replica git-merge fork that
  exercises list_forks + reconcile_choose_branch.
- internal_semantic_tests: drop obsolete ExportedEvent-JSON and stream-id
  parsing tests; carry the enriched WorkflowConnected/WorkflowTransitionEvidence
  navigation fields.
- event_log_export: drop format-assertion/SQLite/legacy-conflict tests; keep the
  behavioral check_rebuilds_* suite that verifies `emc check` regenerates
  artifacts from the store.
- init_project: artifact-only recovery now asserts the committed
  model/events/events/ store rather than model/events/v1.

Fix a second lossy event variant uncovered by the rebuild tests:
WorkflowTransitionEvidenceAdded now carries source_control/target_view so
navigation-transition evidence survives the store round-trip.

Ignore a stray model/ store dir if emc runs in the repo root.

All local gates pass: copyright-headers, fmt, clippy (-D warnings), build, test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

auto-review approved these changes

2026-06-13 09:27:39 -07:00

Dismissed

auto-review left a comment

This PR migrates the event store to use eventcore-fs instead of a custom filesystem event log and SQLite operational store. The changes appear well-structured, but careful attention should be paid to the handling of new fields in events and the transition from SQLite to ensure no loss of functionality. Overall, the migration seems beneficial for simplifying the persistence layer.

Walkthrough

.gitignore: Added an entry to ignore /model/ directory, which is relevant for the new event store setup.
Cargo.toml: Updated dependencies to use eventcore-fs and removed eventcore-sqlite. This is a significant change that requires ensuring all functionalities are preserved.
docs/plans/eventcore-fs-migration.md: New documentation file outlining the migration plan. This is crucial for understanding the rationale and steps involved in the migration.
src/core/event_commands.rs: Enriched EmcEvent variants with additional fields. This change requires ensuring these fields are correctly populated and used.
src/core/event_runtime.rs: Major refactor to replace SQLite with eventcore-fs. This involves changes in how events are stored and retrieved, impacting the entire event handling logic.
src/shell.rs: Adjusted to work with the new event store setup, removing the need for exporting events to JSON files.

LLM usage and cost

Reasoning (gpt-4o) in=23311 out=879 cost=$0.129740
Cheap (gpt-4o-mini) in=13183 out=144 cost=$0.002064
Estimated total USD: $0.131804 via https://api.openai.com and https://api.openai.com

This PR migrates the event store to use `eventcore-fs` instead of a custom filesystem event log and SQLite operational store. The changes appear well-structured, but careful attention should be paid to the handling of new fields in events and the transition from SQLite to ensure no loss of functionality. Overall, the migration seems beneficial for simplifying the persistence layer. ## Walkthrough - **.gitignore**: Added an entry to ignore `/model/` directory, which is relevant for the new event store setup. - **Cargo.toml**: Updated dependencies to use `eventcore-fs` and removed `eventcore-sqlite`. This is a significant change that requires ensuring all functionalities are preserved. - **docs/plans/eventcore-fs-migration.md**: New documentation file outlining the migration plan. This is crucial for understanding the rationale and steps involved in the migration. - **src/core/event_commands.rs**: Enriched `EmcEvent` variants with additional fields. This change requires ensuring these fields are correctly populated and used. - **src/core/event_runtime.rs**: Major refactor to replace SQLite with `eventcore-fs`. This involves changes in how events are stored and retrieved, impacting the entire event handling logic. - **src/shell.rs**: Adjusted to work with the new event store setup, removing the need for exporting events to JSON files. ## LLM usage and cost - Reasoning (gpt-4o) in=23311 out=879 cost=$0.129740 - Cheap (gpt-4o-mini) in=13183 out=144 cost=$0.002064 Estimated total USD: $0.131804 via https://api.openai.com and https://api.openai.com

Cargo.toml Outdated

🟡 Warning: The eventcore dependency version has been updated from 0.8.0 to 0.9.0. Ensure that all breaking changes in the new version are handled appropriately in the codebase.

🟡 **Warning:** The `eventcore` dependency version has been updated from 0.8.0 to 0.9.0. Ensure that all breaking changes in the new version are handled appropriately in the codebase.

Cargo.toml

🟡 Warning: The eventcore-sqlite dependency has been removed and replaced with eventcore-fs. Ensure that all functionalities provided by eventcore-sqlite are now correctly handled by eventcore-fs.

🟡 **Warning:** The `eventcore-sqlite` dependency has been removed and replaced with `eventcore-fs`. Ensure that all functionalities provided by `eventcore-sqlite` are now correctly handled by `eventcore-fs`.

src/core/event_commands.rs Outdated

🟡 Warning: The EmcEvent::WorkflowTransitionEvidenceAdded and EmcEvent::WorkflowConnected events have been enriched with source_control and target_view fields. Ensure that these fields are correctly populated and utilized throughout the application.

🟡 **Warning:** The `EmcEvent::WorkflowTransitionEvidenceAdded` and `EmcEvent::WorkflowConnected` events have been enriched with `source_control` and `target_view` fields. Ensure that these fields are correctly populated and utilized throughout the application.

src/core/event_commands.rs Outdated

🟡 Warning: Lines 454–458: The ConnectWorkflowCommand now includes source_control and target_view. Ensure these fields are correctly handled in all relevant parts of the application.

🟡 **Warning:** **Lines 454–458:** The `ConnectWorkflowCommand` now includes `source_control` and `target_view`. Ensure these fields are correctly handled in all relevant parts of the application.

src/core/event_runtime.rs Outdated

					
				@ -91,751 +73,355 @@ pub(crate) fn lock_project_runtime(project_root: &Path) -> Result<ProjectRuntime

				    Ok(ProjectRuntimeLock { file })

🟡 Warning: The ensure_event_store function replaces the previous SQLite cache provisioning. Ensure that this new implementation correctly initializes the event store as expected.

🟡 **Warning:** The `ensure_event_store` function replaces the previous SQLite cache provisioning. Ensure that this new implementation correctly initializes the event store as expected.

jwilger added 1 commit

2026-06-13 10:29:34 -07:00

fix(events): drop obsolete event-store paths and port conflict/readiness tests

CI / Nix flake check (pull_request) Successful in 2m39s

Details

CI / Request auto_review semantic review (pull_request) Successful in 2s

Details

auto_review auto_review: 1 warning

CI / Rust CI (pull_request) Successful in 9m20s

Details

f3d85a4995

- project: stop creating the dead model/events/v1 directory on init; the
  eventcore-fs store provisions its own model/events/events/ layout.
- mcp_stdio: build a real two-replica git-merge fork for the list/resolve
  conflict tools and assert the fork-based output (conflict <stream> base <n>
  branches <tx>; resolve by stream id + chosen branch transaction).
- verify_project: assert workflow-readiness behavior through the store and
  `emc list workflows` staleness instead of reading the removed v1 JSON log.

Full `cargo nextest run` (the CI test command) passes 423/423; fmt and clippy
(-D warnings, all targets) are clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

jwilger dismissed auto-review's review

2026-06-13 10:29:34 -07:00

Reason:

New commits pushed, approval review dismissed automatically according to repository settings

auto-review approved these changes

2026-06-13 10:32:26 -07:00

auto-review left a comment

The PR migrates the event store to use eventcore_fs::FileEventStore, removing the custom filesystem event log and SQLite store. The changes appear well-structured, but ensure that all directory references and test setups are correctly updated.

Walkthrough

Δ since `c3e332a`:

src/core/project.rs:
- Removed the creation of the model/events/v1 directory. Ensure no dependencies on this directory remain.
tests/mcp_stdio.rs:
- Updated tests to use create_slice_update_fork instead of create_concurrent_slice_update_conflict. Verify the new function sets up the test scenarios correctly.
tests/verify_project.rs:
- Removed checks for WorkflowReadinessDeclared events, indicating a change in verification logic. Ensure all necessary verification steps are still covered.

LLM usage and cost

Reasoning (gpt-4o) in=6115 out=532 cost=$0.038555
Cheap (gpt-4o-mini) in=5793 out=159 cost=$0.000964
Estimated total USD: $0.039519 via https://api.openai.com and https://api.openai.com

The PR migrates the event store to use `eventcore_fs::FileEventStore`, removing the custom filesystem event log and SQLite store. The changes appear well-structured, but ensure that all directory references and test setups are correctly updated. ## Walkthrough ### Δ since c3e332a: - **src/core/project.rs**: - Removed the creation of the `model/events/v1` directory. Ensure no dependencies on this directory remain. - **tests/mcp_stdio.rs**: - Updated tests to use `create_slice_update_fork` instead of `create_concurrent_slice_update_conflict`. Verify the new function sets up the test scenarios correctly. - **tests/verify_project.rs**: - Removed checks for `WorkflowReadinessDeclared` events, indicating a change in verification logic. Ensure all necessary verification steps are still covered. ## LLM usage and cost - Reasoning (gpt-4o) in=6115 out=532 cost=$0.038555 - Cheap (gpt-4o-mini) in=5793 out=159 cost=$0.000964 Estimated total USD: $0.039519 via https://api.openai.com and https://api.openai.com

tests/mcp_stdio.rs

🟡 Warning: Lines 603–607: The function create_slice_update_fork replaces create_concurrent_slice_update_conflict. Ensure that the new function accurately reflects the intended test setup and that all necessary conditions for the test are met.

🟡 **Warning:** **Lines 603–607:** The function `create_slice_update_fork` replaces `create_concurrent_slice_update_conflict`. Ensure that the new function accurately reflects the intended test setup and that all necessary conditions for the test are met.

jwilger commented

2026-06-13 10:40:23 -07:00

Addressing the auto_review 🟡 warnings (both reviews approved; these are advisory "ensure X" notes). Each is verified:

WorkflowTransitionEvidenceAdded / WorkflowConnected enriched with source_control/target_view — these fields were the reason the migration needed them: EmcEvent was lossy vs the old exported payload, so navigation transitions would have lost their endpoints. They are populated in the commands (ConnectWorkflowCommand, AddWorkflowFactCommand::to_event) from the source WorkflowConnection/WorkflowTransitionEvidenceRecord, and reconstructed in ProjectedModel::apply_event (using WorkflowTransitionRecord::new_with_navigation_endpoints / WorkflowTransitionEvidenceRecord::new_with_navigation_endpoints). Coverage: check_rebuilds_workflow_transition_evidence_from_exported_events and check_rebuilds_workflow_connections_from_exported_events rebuild artifacts from the store and assert the navigation endpoints survive the round-trip; internal_semantic_tests asserts the new fields serialize.
ConnectWorkflowCommand source_control/target_view handling — same as (1); the command copies them from connection.source_control()/target_view() and the projection emits navigation transitions only when both are present.
ensure_event_store replacing SQLite provisioning — it opens (creating if absent) the FileEventStore at model/events, which materializes events/ + git metadata. Verified by event_store_lives_inside_the_project_repository and init_recovers_artifact_only_project_by_exporting_initial_event.
eventcore 0.8 → 0.9 — the command API used here (execute, CommandLogic::{handle,apply}, NewEvents, require!, RetryPolicy) is unchanged across 0.9; the whole suite compiles and 423/423 tests pass.
eventcore-sqlite → eventcore-fs — SQLite was a derived validation cache; its functionality (command validation + projection input + conflict detection) is now served by the single committed FileEventStore: commands execute against it, projection reads EmcEvent back via read_all_emc_events, and conflicts use eventcore-fs forks (list_forks / reconcile). Covered by the check_rebuilds_* suite and the two-replica fork/reconcile tests.
create_slice_update_fork test helper — it builds a genuine eventcore-fs fork by cloning the project into a second replica (stripping replica-local store state so it mints a fresh replica_id), making divergent update slice commits in each, and unioning the committed transactions — exactly how a git merge of two offline clones produces a fork. The MCP list_conflicts/resolve_conflict tools then operate on that real fork.

Full cargo nextest run (the CI test command) passes 423/423; clippy -D warnings and fmt are clean.

Addressing the `auto_review` 🟡 warnings (both reviews approved; these are advisory "ensure X" notes). Each is verified: 1. **`WorkflowTransitionEvidenceAdded` / `WorkflowConnected` enriched with `source_control`/`target_view`** — these fields were the *reason* the migration needed them: `EmcEvent` was lossy vs the old exported payload, so navigation transitions would have lost their endpoints. They are populated in the commands (`ConnectWorkflowCommand`, `AddWorkflowFactCommand::to_event`) from the source `WorkflowConnection`/`WorkflowTransitionEvidenceRecord`, and reconstructed in `ProjectedModel::apply_event` (using `WorkflowTransitionRecord::new_with_navigation_endpoints` / `WorkflowTransitionEvidenceRecord::new_with_navigation_endpoints`). Coverage: `check_rebuilds_workflow_transition_evidence_from_exported_events` and `check_rebuilds_workflow_connections_from_exported_events` rebuild artifacts from the store and assert the navigation endpoints survive the round-trip; `internal_semantic_tests` asserts the new fields serialize. 2. **`ConnectWorkflowCommand` source_control/target_view handling** — same as (1); the command copies them from `connection.source_control()/target_view()` and the projection emits navigation transitions only when both are present. 3. **`ensure_event_store` replacing SQLite provisioning** — it opens (creating if absent) the `FileEventStore` at `model/events`, which materializes `events/` + git metadata. Verified by `event_store_lives_inside_the_project_repository` and `init_recovers_artifact_only_project_by_exporting_initial_event`. 4. **eventcore 0.8 → 0.9** — the command API used here (`execute`, `CommandLogic::{handle,apply}`, `NewEvents`, `require!`, `RetryPolicy`) is unchanged across 0.9; the whole suite compiles and 423/423 tests pass. 5. **`eventcore-sqlite` → `eventcore-fs`** — SQLite was a derived validation cache; its functionality (command validation + projection input + conflict detection) is now served by the single committed `FileEventStore`: commands `execute` against it, projection reads `EmcEvent` back via `read_all_emc_events`, and conflicts use eventcore-fs forks (`list_forks` / `reconcile`). Covered by the `check_rebuilds_*` suite and the two-replica fork/reconcile tests. 6. **`create_slice_update_fork` test helper** — it builds a genuine eventcore-fs fork by cloning the project into a second replica (stripping replica-local store state so it mints a fresh `replica_id`), making divergent `update slice` commits in each, and unioning the committed transactions — exactly how a `git merge` of two offline clones produces a fork. The MCP `list_conflicts`/`resolve_conflict` tools then operate on that real fork. Full `cargo nextest run` (the CI test command) passes 423/423; clippy `-D warnings` and fmt are clean.

jwilger merged commit 9f8b4c2911 into main

2026-06-13 10:40:58 -07:00

jwilger deleted branch feat/eventcore-fs-migration

2026-06-13 10:40:58 -07:00

jwilger referenced this pull request from a commit

2026-06-13 10:40:59 -07:00

feat(events): migrate event store to eventcore 0.9 with eventcore-fs (#117)

release-plz-bot referenced this pull request

2026-06-13 10:41:27 -07:00

chore(release): v0.1.1 #118

release-plz-bot referenced this pull request

2026-06-13 11:13:41 -07:00