Docs / Traps & load

Traps & load

Adversarial recall and concurrent-load evidence.

Source: docs/initiatives/memory-api/MEASUREMENTS_THOROUGH_LOAD.md — this page is rendered from it, not copied. The trap ruler applied to what the tier actually serves, and the first concurrent load test. Both were named as measurement holes before they were filled.

Date: 2026-07-27 · closes the measurement holes named in GAPS.md B4 and B5. Everything below is reproduced by two commands against the live spine (SpineStore(user_id='founder') through POST /v1/brains/{id}/context — the same /v1 a customer would call):

.venv/bin/python scripts/measure_thorough_traps.py   # → artifacts/THOROUGH_TRAPS_2026_07_27.json
.venv/bin/python scripts/measure_load.py             # → artifacts/LOAD_TEST_2026_07_27.json

Numbers are dated snapshots of a LIVE corpus (the sync tick that finished 19:27:28 EDT landed 2,818 embed changes minutes before these runs; 12,085 active embeddings). Neither number belongs in CURRENT_TRUTH.md without its dataset tag.

B4 — confidently_wrong on mode: thorough, measured for the first time#

What thorough actually is on the real engine#

Read from memory_api/spine.py and then proven empirically (not just by code-reading):

• SpineStore.create_context never reads mode. The string "mode" appears exactly once in the file — spine.py:532, where stream_context echoes it into the SSE status frame. Both paths assemble evidence through the same _assemble() → _envelope().
• The only behavior mode changes is in app.py:429-446: Accept: text/event-stream is honoured for thorough and refused for fast (406 streaming_not_applicable — confirmed live).
• Measured across all 20 cases: served text identical 20/20 between fast JSON and thorough JSON, and identical 20/20 between thorough JSON and the SSE terminal context frame. Latency is indistinguishable (fast p50 1,286 ms · thorough JSON 1,274 ms · thorough SSE 1,312 ms).

So thorough is currently fast with streaming. Nothing generative, no multi-hop, no second retrieval pass. The contract survives on the word "may" (openapi.yaml:875 — "thorough may multi-hop and return 202"), but the mode knob buys a customer nothing except SSE framing today. That is a knob whose two positions serve one behavior — GAPS-A-adjacent, and now it is written down instead of implied.

RETRACTED 2026-07-29 — the paragraph below is false and is kept only as the record of how.

It concluded the spine's cost receipt was honest. It was not. The reasoning was circular: it read generative_calls: 0 out of 60 responses and treated the agreement between a hardcoded constant and itself as evidence. _envelope returned the literal 0 on every path; nothing counted anything. Meanwhile brain/retrieve/rewrite.py:expand() was calling Gemini on every single request, so the true number was 2 — one embedding, one generative rewrite.

The observation "observed [0] across all 60 responses" is therefore not a finding about the engine; it is 60 readings of the same constant. A receipt that cannot be wrong is not a receipt — and the check that would have caught it is the one now in memory_api/tests/test_fast_spends_one_embedding.py: break generation and see whether the engine notices.

Ironically, the paragraph's other half was right for the right reason — the fixture's 1 was called a lie, when the fixture was the only thing in the building reporting a non-constant.

Superseded by: artifacts/_ablation_fast_rewrite/ (two 516-request runs, 2026-07-29) and the corrected CostReport in openapi.yaml. fast now spends one model call — the query embedding — and generates nothing, reported as three separate fields (model_calls_total: 1 · embedding_calls: 1 · generative_calls: 0) read from the ledger. The receipt is three numbers because one integer could not distinguish "it called a model" from "it composed text", and shipping it as one is what let a hardcoded 0 pass for a measurement.

~~Does cost.generative_calls tell the truth? On the spine engine, yes: _envelope hardcodes generative_calls: 0 (spine.py:575) and zero generative calls actually happen, on every path, both modes — observed [0] across all 60 responses. On the conformance fixture it lies: inmemory.py:403 reports generative_calls: 1 for thorough while making zero generative calls. The fixture misrepresents a cost field the engine reports honestly — small, but exactly the class of self-misrepresentation GAPS section A exists for.~~

The first measured trap board for thorough#

8 trap cases (abstain_ok) + 12 non-trap cases (2 per class, deterministic pick) from brain.eval.heldout.founder_confirmed_v2.CASES, graded on the SSE terminal frame's text with the full brain.eval.heldout.grader — abstain_ok and answer_must_not included, which the prior fast-context measurement deliberately did not do.

Headline: confidently_wrong = 3/8 traps (3/20 cases) on thorough under the full grader. Non-trap carries: 8/12 (misses V2-01 · V2-32 · V2-33 · V2-39, all incomplete — consistent with the known board misses). Transport errors: 0.

How to read a non-zero number on a non-asserting tier — the decomposition#

The grader was built for asserted answers; a Context asserts nothing — it hands cited evidence to the caller's model. The 3 decompose into two different defects, and only one of them contains bait:

• 1/8 — bait physically in the served window (V2-04). The PINNIE PLAN note (Ad Spend: $44K) is retrieved for "how much did we spend on ads", so the one number the founder ruled must never be served as ours ships to the caller's model as top evidence, ownership unmarked. This is the same single window-bait the fast measurement found (artifacts/FAST_CONTEXT_2026_07_26.json → traps.bait_in_window: ["V2-04"]), and on the date of this run thorough was fast — the two tiers ran identical code, so the result carried over necessarily. That premise expired on 2026-07-29 (see the note below): thorough now runs a generative query rewrite that fast does not, so a future run has to re-measure this rather than inherit it.
• 2/8 — the window neither abstains nor corrects (V2-26, V2-27). For "when is my wedding" (founder is single) the tier returns ~11.7k chars of wedding-adjacent evidence about other people's events, with nothing that corrects the false premise. No bait token is served — the defect is that a downstream model reading that window has every invitation to answer the question as asked.

So both statements from GAPS B4 are now measured, not argued: the tier still asserts nothing — confidently_wrong = 0 in the strict fabrication sense remains structural — but on the date of this run thorough had no generative path of its own to measure, and when the trap ruler is applied to what the tier actually serves, the number is 3/8, not 0. The structural 0 is a property of who does the asserting, not of what gets served.

"THE DAY thorough GROWS A REAL GENERATIVE PATH" ARRIVED ON 2026-07-29, AND THIS BOARD IS NOW THE PRE-REGISTERED BASELINE IT MUST BEAT — UNRUN. brain/retrieve/rewrite.py:expand() was always on this path; nobody had noticed, which is why the sentence above says thorough had no generative path. It had one, and so did fast. What changed is that the rewrite is now spent only on thorough (spine.py:_assemble, expand_query=(mode == "thorough")), so the two tiers genuinely differ for the first time. Consequences for this section, stated rather than quietly absorbed:

· The 3/8 and the 1/8 stand as measurements of what was served on 2026-07-27, by both tiers. · They no longer transfer between tiers. Any claim that a fast trap result predicts a thorough one, or the reverse, now needs its own run. · Nothing has re-measured thorough since the split. The honest status of the B4 axis on today's thorough is unmeasured, not 3/8. Re-run scripts/measure_thorough_traps.py, whose own premise block carries the matching retraction.

B5 — the first load test: every prior latency figure was single-threaded#

In-process threaded, one TestClient per thread (own portal/event loop), barrier-started levels of 1 / 4 / 8 / 16 threads × 8 sequential fast-context requests each — 232 requests, every one a distinct query (seeded shuffle over the 43 held-out + 36 lifelike questions × 4 phrasings; a derived-but-distinct pool, stated plainly). 2 uncounted warmup requests absorb first-touch costs (the first was 12.6 s cold, the second 1.8 s).

Prior single-threaded p50 for reference: ~1,291 ms (2026-07-26, same HTTP path); this run's fresh c=1 baseline is 1,552 ms on today's larger corpus and longer phrasings.

The shape: zero errors at every level — it degrades, it does not break — but throughput saturates at ~1.3–1.4 req/s (c=8 → c=16 adds 0.08 req/s while p50 doubles). Past c≈8, added concurrency converts almost entirely to queueing. The mechanism is in the code, not conjecture:

1. brain/spine/pool.py DEFAULT_SIZE = 10 — sized to one request's lane fan-out, explicitly "not server concurrency". At c=16, ~10 lanes × 16 in-flight requests contend for 10 pooled connections; the pool becomes the server-wide ceiling it was never meant to be.
2. Every request opens its own pg_connect() for the main path — measured 320 ms p50 through the cloud-sql-proxy (design prompt §3, "a server must never open one per request" — it does).
3. One Gemini embed_query HTTPS call per request, N-wide against shared quota (no 429s observed at these levels).

Wall-clock / sync-tick honesty: the launchd sync agent (StartInterval 1200 s) is the known retrieval-degrader (12.6 s / 2.9 s / 1.8 s under brain.flow.refresh). This run landed cleanly between ticks: the previous tick finished 19:27:28 EDT (runs/sync/sync.out.log, 22/22 stages ok), levels ran 19:29:42–19:33:07 EDT, and the log's (mtime, size) — sampled before and after every level — never moved. No latency jump is attributable to the tick; the curve above is pure concurrent-read contention. The 8 slowest requests (15.7–19.3 s) all sit inside the c=16 window, timestamps in the artifact.

What this does NOT measure, so the number cannot be over-read: a single Python process (GIL shared by client threads and server handlers — an external multi-process driver would be stricter), reads only (concurrent-write degradation remains measured separately and worse), ~90 s per level (no sustained-load soak), localhost (no network between caller and API).

What moved because of this page#

• GAPS B4 is no longer "never measured": thorough = fast + SSE (proven 20/20 byte-equal), first trap board 3/8 · bait-in-window 1/8 · fixture's generative_calls: 1 is a fiction worth a one-line fix.
• GAPS B5 is no longer "no load test": the engine holds 0 errors to c=16 but saturates at ~1.4 req/s with p50 ×7. The first latency work under load is already named in the code: a real server-sized connection strategy (pool sized for concurrency + no per-request pg_connect), ahead of any algorithmic change.

# Two measurements nobody had run: `thorough` on the trap axis, and the first load test

**Date:** 2026-07-27 · closes the measurement holes named in [`GAPS.md`](./GAPS.md) **B4** and
**B5**. Everything below is reproduced by two commands against the live spine
(`SpineStore(user_id='founder')` through `POST /v1/brains/{id}/context` — the same `/v1` a
customer would call):

```bash
.venv/bin/python scripts/measure_thorough_traps.py   # → artifacts/THOROUGH_TRAPS_2026_07_27.json
.venv/bin/python scripts/measure_load.py             # → artifacts/LOAD_TEST_2026_07_27.json
```

Numbers are dated snapshots of a LIVE corpus (the sync tick that finished 19:27:28 EDT landed
2,818 embed changes minutes before these runs; 12,085 active embeddings). Neither number belongs
in `CURRENT_TRUTH.md` without its dataset tag.

---

## B4 — `confidently_wrong` on `mode: thorough`, measured for the first time

### What `thorough` actually is on the real engine

Read from `memory_api/spine.py` and then **proven empirically** (not just by code-reading):

- `SpineStore.create_context` **never reads `mode`**. The string `"mode"` appears exactly once in
  the file — `spine.py:532`, where `stream_context` echoes it into the SSE `status` frame. Both
  paths assemble evidence through the same `_assemble()` → `_envelope()`.
- The only behavior `mode` changes is in `app.py:429-446`: `Accept: text/event-stream` is honoured
  for `thorough` and refused for `fast` (406 `streaming_not_applicable` — confirmed live).
- Measured across all 20 cases: served `text` **identical 20/20** between fast JSON and thorough
  JSON, and **identical 20/20** between thorough JSON and the SSE terminal `context` frame.
  Latency is indistinguishable (fast p50 1,286 ms · thorough JSON 1,274 ms · thorough SSE 1,312 ms).

**So `thorough` is currently `fast` with streaming.** Nothing generative, no multi-hop, no second
retrieval pass. The contract survives on the word "may" (`openapi.yaml:875` — "`thorough` may
multi-hop and return 202"), but the mode knob buys a customer nothing except SSE framing today.
That is a knob whose two positions serve one behavior — GAPS-A-adjacent, and now it is written
down instead of implied.

> **RETRACTED 2026-07-29 — the paragraph below is false and is kept only as the record of how.**
>
> It concluded the spine's cost receipt was honest. It was not. The reasoning was circular: it read
> `generative_calls: 0` out of 60 responses and treated the agreement between a **hardcoded constant
> and itself** as evidence. `_envelope` returned the literal `0` on every path; nothing counted
> anything. Meanwhile `brain/retrieve/rewrite.py:expand()` was calling Gemini on **every single
> request**, so the true number was 2 — one embedding, one generative rewrite.
>
> The observation "observed `[0]` across all 60 responses" is therefore not a finding about the
> engine; it is 60 readings of the same constant. **A receipt that cannot be wrong is not a
> receipt** — and the check that would have caught it is the one now in
> `memory_api/tests/test_fast_spends_one_embedding.py`: break generation and see whether the engine
> notices.
>
> Ironically, the paragraph's *other* half was right for the right reason — the fixture's `1` was
> called a lie, when the fixture was the only thing in the building reporting a non-constant.
>
> Superseded by: `artifacts/_ablation_fast_rewrite/` (two 516-request runs, 2026-07-29) and the
> corrected `CostReport` in `openapi.yaml`. `fast` now spends **one model call — the query
> embedding — and generates nothing**, reported as three separate fields
> (`model_calls_total: 1 · embedding_calls: 1 · generative_calls: 0`) read from the ledger. The
> receipt is three numbers because one integer could not distinguish "it called a model" from "it
> composed text", and shipping it as one is what let a hardcoded `0` pass for a measurement.

~~**Does `cost.generative_calls` tell the truth?** On the spine engine, **yes**: `_envelope`
hardcodes `generative_calls: 0` (`spine.py:575`) and zero generative calls actually happen, on
every path, both modes — observed `[0]` across all 60 responses. On the **conformance fixture**
it lies: `inmemory.py:403` reports `generative_calls: 1` for `thorough` while making zero
generative calls. The fixture misrepresents a cost field the engine reports honestly — small, but
exactly the class of self-misrepresentation GAPS section A exists for.~~

### The first measured trap board for `thorough`

8 trap cases (`abstain_ok`) + 12 non-trap cases (2 per class, deterministic pick) from
`brain.eval.heldout.founder_confirmed_v2.CASES`, graded on the SSE terminal frame's `text` with
the full `brain.eval.heldout.grader` — `abstain_ok` and `answer_must_not` **included**, which the
prior fast-context measurement deliberately did not do.

| trap | grade | why |
|---|---|---|
| V2-03 (Pinnie $4M plan) | CORRECT · correction | window carries the Pinnie/Aashay attribution |
| **V2-04 (June ad spend)** | **confident_wrong** | **bait `$44k` present in the served text** |
| **V2-26 (my wedding)** | **confident_wrong** | window returns evidence, neither abstains nor corrects |
| **V2-27 (engagement party)** | **confident_wrong** | window returns evidence, neither abstains nor corrects |
| V2-28 (+30 customers) | CORRECT · correction | bait absent, correcting facts carried |
| V2-29 (Hertz confirmation) | CORRECT · correction | |
| V2-30 (Chance concert) | CORRECT · correction | |
| V2-31 (419 3rd Ave rent) | CORRECT · correction | |

**Headline: `confidently_wrong` = 3/8 traps (3/20 cases) on `thorough` under the full grader.**
Non-trap carries: 8/12 (misses V2-01 · V2-32 · V2-33 · V2-39, all `incomplete` — consistent with
the known board misses). Transport errors: 0.

### How to read a non-zero number on a non-asserting tier — the decomposition

The grader was built for **asserted answers**; a Context asserts nothing — it hands cited evidence
to the caller's model. The 3 decompose into two different defects, and only one of them contains
bait:

- **1/8 — bait physically in the served window (V2-04).** The PINNIE PLAN note (`Ad Spend: $44K`)
  is retrieved for "how much did **we** spend on ads", so the one number the founder ruled must
  never be served as ours ships to the caller's model as top evidence, ownership unmarked. This is
  the same single window-bait the fast measurement found
  (`artifacts/FAST_CONTEXT_2026_07_26.json` → `traps.bait_in_window: ["V2-04"]`), and on the date
  of this run `thorough` **was** `fast` — the two tiers ran identical code, so the result carried
  over necessarily. **That premise expired on 2026-07-29** (see the note below): `thorough` now
  runs a generative query rewrite that `fast` does not, so a future run has to re-measure this
  rather than inherit it.
- **2/8 — the window neither abstains nor corrects (V2-26, V2-27).** For "when is my wedding"
  (founder is single) the tier returns ~11.7k chars of wedding-adjacent evidence about *other
  people's* events, with nothing that corrects the false premise. No bait token is served — the
  defect is that a downstream model reading that window has every invitation to answer the
  question as asked.

**So both statements from GAPS B4 are now measured, not argued:** the tier still *asserts*
nothing — `confidently_wrong = 0` in the strict fabrication sense remains structural — but on the
date of this run `thorough` had no generative path of its own to measure, and when the trap ruler
is applied to what the tier actually serves, the number is **3/8, not 0**. The structural 0 is a
property of who does the asserting, not of what gets served.

> **"THE DAY `thorough` GROWS A REAL GENERATIVE PATH" ARRIVED ON 2026-07-29, AND THIS BOARD IS NOW
> THE PRE-REGISTERED BASELINE IT MUST BEAT — UNRUN.** `brain/retrieve/rewrite.py:expand()` was
> always on this path; nobody had noticed, which is why the sentence above says `thorough` had no
> generative path. It had one, and so did `fast`. What changed is that the rewrite is now spent
> **only** on `thorough` (`spine.py:_assemble`, `expand_query=(mode == "thorough")`), so the two
> tiers genuinely differ for the first time. Consequences for this section, stated rather than
> quietly absorbed:
>
> · **The 3/8 and the 1/8 stand as measurements** of what was served on 2026-07-27, by both tiers.
> · **They no longer transfer between tiers.** Any claim that a `fast` trap result predicts a
>   `thorough` one, or the reverse, now needs its own run.
> · **Nothing has re-measured `thorough` since the split.** The honest status of the B4 axis on
>   today's `thorough` is *unmeasured*, not *3/8*. Re-run
>   `scripts/measure_thorough_traps.py`, whose own premise block carries the matching retraction.

---

## B5 — the first load test: every prior latency figure was single-threaded

In-process threaded, one `TestClient` per thread (own portal/event loop), barrier-started levels
of 1 / 4 / 8 / 16 threads × 8 sequential fast-context requests each — **232 requests, every one a
distinct query** (seeded shuffle over the 43 held-out + 36 lifelike questions × 4 phrasings; a
derived-but-distinct pool, stated plainly). 2 uncounted warmup requests absorb first-touch costs
(the first was 12.6 s cold, the second 1.8 s).

| concurrency | n | p50 | p95 | max | errors | wall | throughput |
|---|---|---|---|---|---|---|---|
| 1 (baseline) | 8 | **1,552 ms** | 2,702 ms | 2,702 ms | 0 | 14.1 s | 0.57 req/s |
| 4 | 32 | **2,680 ms** (×1.73) | 11,320 ms (×4.19) | 11,595 ms | 0 | 38.0 s | 0.84 req/s |
| 8 | 64 | **4,997 ms** (×3.22) | 11,476 ms (×4.25) | 13,299 ms | 0 | 49.9 s | 1.28 req/s |
| 16 | 128 | **10,987 ms** (×7.08) | 15,883 ms (×5.88) | 19,309 ms | 0 | 94.0 s | 1.36 req/s |

Prior single-threaded p50 for reference: ~1,291 ms (2026-07-26, same HTTP path); this run's fresh
c=1 baseline is 1,552 ms on today's larger corpus and longer phrasings.

**The shape:** zero errors at every level — it degrades, it does not break — but **throughput
saturates at ~1.3–1.4 req/s** (c=8 → c=16 adds 0.08 req/s while p50 doubles). Past c≈8, added
concurrency converts almost entirely to queueing. The mechanism is in the code, not conjecture:

1. **`brain/spine/pool.py` `DEFAULT_SIZE = 10`** — sized to *one request's* lane fan-out,
   explicitly "not server concurrency". At c=16, ~10 lanes × 16 in-flight requests contend for 10
   pooled connections; the pool becomes the server-wide ceiling it was never meant to be.
2. **Every request opens its own `pg_connect()`** for the main path — measured 320 ms p50 through
   the cloud-sql-proxy (design prompt §3, "a server must never open one per request" — it does).
3. **One Gemini `embed_query` HTTPS call per request**, N-wide against shared quota (no 429s
   observed at these levels).

**Wall-clock / sync-tick honesty:** the launchd sync agent (`StartInterval` 1200 s) is the known
retrieval-degrader (12.6 s / 2.9 s / 1.8 s under `brain.flow.refresh`). This run landed cleanly
**between ticks**: the previous tick finished 19:27:28 EDT (`runs/sync/sync.out.log`, 22/22
stages ok), levels ran 19:29:42–19:33:07 EDT, and the log's (mtime, size) — sampled before and
after every level — never moved. **No latency jump is attributable to the tick; the curve above
is pure concurrent-read contention.** The 8 slowest requests (15.7–19.3 s) all sit inside the
c=16 window, timestamps in the artifact.

**What this does NOT measure,** so the number cannot be over-read: a single Python process (GIL
shared by client threads and server handlers — an external multi-process driver would be
stricter), reads only (concurrent-write degradation remains measured separately and worse),
~90 s per level (no sustained-load soak), localhost (no network between caller and API).

---

## What moved because of this page

- **GAPS B4** is no longer "never measured": `thorough` = `fast` + SSE (proven 20/20 byte-equal),
  first trap board 3/8 · bait-in-window 1/8 · fixture's `generative_calls: 1` is a fiction worth
  a one-line fix.
- **GAPS B5** is no longer "no load test": the engine holds 0 errors to c=16 but saturates at
  ~1.4 req/s with p50 ×7. The first latency work under load is already named in the code: a real
  server-sized connection strategy (pool sized for concurrency + no per-request `pg_connect`),
  ahead of any algorithmic change.