Stress test projection scenario fails on PostgreSQL due to shared database state #371

New issue

Closed

opened 2026-04-15 10:55:05 -07:00 by jwilger-ai-bot · 0 comments

jwilger-ai-bot commented

2026-04-15 10:55:05 -07:00

(Migrated from github.com)

Summary

The projection stress test scenario reports Correctness: FAIL on the PostgreSQL backend because it shares a database with contract tests and previous stress test runs. Memory and SQLite backends pass because they create fresh in-memory stores per run.

Root Cause

The PostgreSQL eventcore_events table accumulates events across all test runs (contract tests, prior stress runs). When the projection scenario runs in batch mode:

Phase 1 writes 900 BankAccountEvent events via execute() — these succeed
Phase 2 calls run_projection() in batch mode (default ProjectionConfig)
Batch mode reads the first 1000 events ordered by event_id (UUID v7, time-ordered)
The oldest events in the table are from contract tests with schemas like {"data": "to be deleted"}, {"n": 1} — not BankAccountEvent variants
serde_json::from_value::<BankAccountEvent>() fails for all 1000 rows
The .ok() in read_events silently drops all of them → 0 events returned
The pipeline sees 0 events and considers the projection complete
The projector's apply() is never called → balances map is empty → correctness check fails

Observed: After a stress test run, the database had 181,130 total events, with 3,237 non-stress-test events (from contract tests) holding the oldest UUID v7 values. The first 1000 rows were 100% contract test events.

Reproduction

# Run contract tests against postgres first
docker-compose up -d
cargo nextest run --workspace  # populates eventcore_events

# Then run stress tests
./target/release/eventcore-stress run-all --backend postgres --concurrency 50 --duration 60s
# Projection scenario will report FAIL

Fix

The stress test's postgres setup should isolate itself from other test data. Options:

Truncate the table before running (simplest, but destructive if sharing a dev database)
Use a separate database (e.g., postgres_stress created via docker-compose)
Use a stream prefix filter in the projection so it only reads stress test events
Reset the database schema via DROP TABLE + migrate() at the start of each run-all

Option 4 is probably the most appropriate for a stress testing tool — start clean each run.

Files

eventcore-stress/src/scenarios/projection.rs — the projection scenario
eventcore-stress/src/main.rs — run-all command orchestration
eventcore-postgres/src/lib.rs:267-349 — read_events with silent .ok() deserialization

## Summary The projection stress test scenario reports `Correctness: FAIL` on the PostgreSQL backend because it shares a database with contract tests and previous stress test runs. Memory and SQLite backends pass because they create fresh in-memory stores per run. ## Root Cause The PostgreSQL `eventcore_events` table accumulates events across all test runs (contract tests, prior stress runs). When the projection scenario runs in batch mode: 1. Phase 1 writes 900 `BankAccountEvent` events via `execute()` — these succeed 2. Phase 2 calls `run_projection()` in batch mode (default `ProjectionConfig`) 3. Batch mode reads the **first 1000 events** ordered by `event_id` (UUID v7, time-ordered) 4. The oldest events in the table are from contract tests with schemas like `{"data": "to be deleted"}`, `{"n": 1}` — not `BankAccountEvent` variants 5. `serde_json::from_value::<BankAccountEvent>()` fails for all 1000 rows 6. The `.ok()` in `read_events` silently drops all of them → 0 events returned 7. The pipeline sees 0 events and considers the projection complete 8. The projector's `apply()` is never called → `balances` map is empty → correctness check fails **Observed:** After a stress test run, the database had 181,130 total events, with 3,237 non-stress-test events (from contract tests) holding the oldest UUID v7 values. The first 1000 rows were 100% contract test events. ## Reproduction ```bash # Run contract tests against postgres first docker-compose up -d cargo nextest run --workspace # populates eventcore_events # Then run stress tests ./target/release/eventcore-stress run-all --backend postgres --concurrency 50 --duration 60s # Projection scenario will report FAIL ``` ## Fix The stress test's postgres setup should isolate itself from other test data. Options: 1. **Truncate the table** before running (simplest, but destructive if sharing a dev database) 2. **Use a separate database** (e.g., `postgres_stress` created via docker-compose) 3. **Use a stream prefix filter** in the projection so it only reads stress test events 4. **Reset the database schema** via `DROP TABLE` + `migrate()` at the start of each `run-all` Option 4 is probably the most appropriate for a stress testing tool — start clean each run. ## Files - `eventcore-stress/src/scenarios/projection.rs` — the projection scenario - `eventcore-stress/src/main.rs` — `run-all` command orchestration - `eventcore-postgres/src/lib.rs:267-349` — `read_events` with silent `.ok()` deserialization