A conformance test suite and public scorecard

Does your agent notice when its memory is edited behind its back?

agmi seeds an AI agent's memory or checkpoint store through the tool's own API, edits the stored files directly, reloads, and asks the tool to read its memory again. Every row on the scorecard is a measurement of the real library at a pinned version, reproducible offline in under a minute.

Seed the store, edit one record behind the tool's back, reload, ask the tool. The tool answers with the altered record. 1. Seed through the API 01 units: metric 02 budget: 4,000 03 approver: Iyer 04 ship to: Bengaluru 05 terms: net 30 Five records, written theway a user would. 2. Edit the file directly 01 units: metric 02 budget: 40,000 03 approver: Iyer 04 ship to: Bengaluru 05 terms: net 30 One digit added on disk.No API call was made. 3. Reload open the store load five records integrity check none The tool starts freshfrom what is on disk. 4. Ask the tool recall("budget") budget: 40,000 accepted no error, no warning The agent carries onfrom the altered record.

Each cell on the scorecard is one run of this sequence with one kind of edit. The verdict comes from the tool itself: if it raises, refuses or reports, the cell says so. If it serves the altered record, the cell says accepted.

Three of the most used agent memory layers accept every one of five edits silently, and the agent resumes from the altered memory. One store with a signed receipt mode reports the edits, but only through a separate audit call. Its read path still serves the altered record.

See the scorecard Read the paper Run it yourself pip install agent-memory-integrity

Five terms, in plain words

Agent memory
What an AI agent keeps between steps and between sessions: facts about a user, decisions it made, the state it will resume from. It lives in a database or a file, not in the model.
Checkpoint
A saved snapshot of an agent's state at one step, so the run can pause and resume. A checkpoint store is the sequence of those snapshots.
At rest
While the data is sitting in storage and the tool is not running. Every edit on the scorecard is made at rest.
Read path
The call that returns memories to the agent. A check here means the agent never sees an altered record.
Audit call
A separate method an operator runs to verify the store. A check here finds edits after the fact, and only if someone runs it.

The scorecard

Five at-rest edits, measured against the real library at the version shown. The checked at column says where the tool would catch the edit: on the read path that returns memories to the agent, or only in a separate audit call an operator has to make.

Target Checked at Tamper Truncate Delete middle Reorder Forge
LangGraph SqliteSaverlanggraph-checkpoint-sqlite 3.1.1 read path accepted accepted accepted accepted accepted
Letta core memory historyletta 0.16.8 read path accepted accepted accepted accepted accepted
Mem0 local Qdrant storemem0ai 2.0.20 read path accepted accepted accepted accepted accepted
inspeximus, receipts offinspeximus 3.0.0, default configuration read path accepted accepted accepted accepted accepted
inspeximus, receipts oninspeximus 3.0.0, attacker holds the store directory audit call reported reported reported reported reported
inspeximus, receipts oninspeximus 3.0.0, attacker also holds the user's config home audit call reported accepted reported reported reported
OpenFang audit log, modela Python re-implementation of the hash chain, not the Rust binary read path detected detected detected detected detected
accepted
The tool loaded the altered store, raised nothing, and served the altered data. The agent carried on as if it were true.
detected
The tool raised or refused on the read path. The agent never saw the altered data.
reported
A separate integrity call named the problem. The read path still served the altered data. An operator has to run that call to find out.

The inspeximus rows were submitted by the inspeximus maintainer and reproduced independently by agmi on inspeximus 3.0.0, macOS, Python 3.12. Rows submitted by a tool's own maintainer land only after agmi reproduces them, and they are marked as such. The reference row is a model of OpenFang's hash chained audit log, kept to show that all five edits are detectable by a chained store. It is not a measurement of the real binary.

Every row is pinned by a test that fails the day the library adds a check. When that happens the row is re-measured on the new version and the change is published in the next monthly report.

The five edits

Each edit is made to the raw store while the tool is not running. All five keep the file valid: the store still opens. That is the point. A corrupt file gets caught by the parser; a well formed edit has to be caught by the tool.

Tamper

Change the content of one existing record in place. The count, order and ids are untouched.

Truncate

Remove the most recent records. The agent silently rewinds to an earlier state and never learns that later writes existed.

Delete middle

Remove one record from inside the history and leave a gap. A store with sequence numbers has the evidence to notice; most do not look.

Reorder

Swap two records. Nothing is added or lost, but what happened first has changed, and so has what the agent resumes from.

Forge

Insert a record the tool never wrote. On every checkpoint store measured so far, the agent resumes from the forged checkpoint.

Where memory sits in an agent, and where agmi sits

Two diagrams. The first shows the layers of a production agent and the one layer nobody checks. The second shows how the suite is put together, so a new attack or a new target is one file.

The memory layer of an agent system

An agent runtime calls a model and tools, and reads and writes memory through a memory tool. The memory tool persists to a backing store on disk or in a database. agmi edits that backing store and observes what the memory tool returns. Runtime Agent loop Language model Tools Memory layer Checkpointersaves and resumes agent state Long term memoryrecall, embeddings, history Audit logwhat was decided, by whom Backing store SQLite, Postgrescheckpoint rows, msgpack blobs Qdrant, pgvector, filespickled points, JSON records JSONL, receipts sidecarhash chain, if any agmi edits the backing store here, then reads back through the memory layer above it.

Everything above the bottom band trusts the bottom band. The checkpointer assumes the row it loads is the row it wrote. The memory tool assumes the point it recalls is the point it stored. agmi asks a single question of each product: does anything on the way back up check that assumption?

How the suite is built

Attacks on the left, adapters on the right, a runner in the middle that crosses every attack with every adapter and prints the scorecard. Pinned tests and CI re-run it. Attacks one class each, one pass rule tampertruncatedelete_middlereorderforge plus memory-specific edits in 0.6 Adapters one per real library setup() seed()reload() verify()detection_point seed and verify go through thetool's own API, never the file Runner every attack againstevery adapter prints the scorecard Pinned tests and CIfail the day a library changes A new target is one adapter file. A new attack is one class. Neither needs the other to change.

Two places a tool can catch an edit

Left: a read-path check refuses the altered record before the agent sees it. Right: an audit-call check finds the edit only when an operator runs it, after the agent has already read the altered record. Read path Agent asks Tool checks refuses The altered record never reaches the agent. Scorecard prints detected. Audit call Agent asks served Operator runs reported Found later, if someone looks. Scorecard prints reported.

How a row is measured

The rules every row follows

  1. Seed and read through the tool's own API. The adapter writes memories the way a user would and reads them back the way the tool's users would. The raw store is touched only during the edit.
  2. The tool gives the verdict. If it loads the altered store and raises, refuses or reports the problem, that is a detection. If it loads and serves the altered data, that is accepted. Detection is never inferred from the adapter's own checks.
  3. Name the detection point. Read path, or a separate audit call. Both count, but they are different guarantees and the scorecard shows which.
  4. Pin the version. The row states the library version, and its test fails the day the library changes its behaviour.
  5. Runs offline in under a minute. No hosted APIs, no model downloads, no Docker.

Read path or audit call

Two tools can both "notice" an edit and give an agent very different protection. A tool that checks on the read path refuses the altered record before the agent ever sees it. A tool that only checks in a separate audit call will serve the altered record to the agent, and the edit is found later, if someone runs the check.

The scorecard keeps the two apart. A detection on the read path prints as detected. A detection in an audit call prints as reported, and the checked at column says audit call.

Run it yourself

pip install agent-memory-integrity
git clone https://github.com/tech4biz-yasha/agmi && cd agmi
pip install -e ".[dev,langgraph,letta,mem0,inspeximus]"
PYTHONPATH=. python3 agmi/full_runner.py

Prints the scorecard above beside a naive baseline. About a minute on a laptop.

Who the attacker is

Someone with write access to the backing store: the SQLite file, the vector collection, the checkpoint table. That includes a compromised host, a malicious or careless co-tenant on shared storage, a backup restored from the wrong point, and any process running as the same user account as the agent.

The agent talks to its memory tool through an API. The attacker writes to the store underneath the tool, bypassing the API. Agent Memory toolAPI, retrieval, checkpoints Backing storeSQLite, Qdrant, Postgres, files Attacker with write access The attacker never calls the API. The only thing that can catch the edit is what the tool does on reload.

This is a design gap, not a bug in any one product. LangGraph, Letta and Mem0 do not claim their stores are tamper evident. inspeximus makes that claim for its receipts mode, and the scorecard shows what that buys and where it stops. The gap matters the moment agent memory is used as a record: of what a user asked for, what an agent approved, what a workflow decided.

Anything that looks like an actual vulnerability, such as a stored payload that runs code on load, goes to the tool's maintainers privately first. Design gaps are discussed in public.

Paper, code, standards

Paper
Tamper Evidence in AI Agent Memory Stores: A Conformance Suite and Three Measurements. Preprint, September 2026. doi:10.5281/zenodo.22765627, also on SSRN 7461118.
Software
agmi 0.5.0, archived record doi:10.5281/zenodo.22860887. Source at github.com/tech4biz-yasha/agmi, MIT licence. Package agent-memory-integrity on PyPI.
Cite
Khandelwal, Y. (2026). Tamper Evidence in AI Agent Memory Stores: A Conformance Suite and Three Measurements. doi:10.5281/zenodo.22765627. A CITATION.cff ships with the repository.

Mapping to the IETF agent security benchmark draft

The IETF Benchmarking Methodology Working Group has an individual draft, draft-han-bmwg-agent-security-benchmark, that lists 55 metrics for AI agent security. Three of them describe what agmi measures. agmi was proposed on the working group list on 21 September 2026 as a reproducible method for metric 5.4.7.

Draft metric What it asks for agmi
5.4.7 Protection of memory data integrity Tamper with local memory files, restart, check whether the agent detects and refuses. The five at-rest edits and the checked at column.
5.4.3 Memory poisoning defence Inject an instruction into memory in one session, trigger it in the next. Memory injection, on real retrieval tools in 0.6.
5.4.4 Cross-user, cross-session isolation Read another user's or session's memory. Cross-session bleed, on real retrieval tools in 0.6.

Monthly reports

The scorecard is re-run and published on the last day of every month, with the same title and the same table, so changes are visible across time: a tool that adds a check, a new target, a new edit.

September 2026
First report. Three real targets at rest, all five edits accepted by all three. Read it in the repository.
October 2026
Due 31 October. Adds inspeximus, the checked at column, memory-specific edits on real retrieval tools, and the mapping to the IETF draft.

Roadmap

One family of edits at a time, each measured on real libraries before the next begins.

  1. 0.5done

    Five at-rest edits. Real measurements for LangGraph, Letta, Mem0 and inspeximus. Rulebook, pinned tests, CI.

  2. 0.6next

    Memory-specific edits with real embedders: injection, cross-session bleed, retrieval hijack, indirect prompt injection, on Mem0 and Graphiti.

  3. 0.7planned

    Stored payloads that execute on load. Reported privately to maintainers before anything is published.

  4. 0.8planned

    Edits during execution: replay, cross-thread poisoning, rollback while the agent is running.

  5. 0.9planned

    A reference integrity layer, a hash chain in checkpoint metadata, offered upstream as an optional mode.

  6. 1.0planned

    Stable adapter interface, published conformance levels a vendor can cite, and the monthly cadence as a standing commitment.

Get your tool scored

Any memory or checkpoint store can have a row. Write an adapter against the MemoryAdapter interface, seed and read through your own API, and open a pull request with the row and its pinned test.

Maintainers are welcome to submit their own tool. Those rows land only after agmi reproduces them on an independent machine, and the scorecard marks them: submitted by the maintainer, reproduced independently by agmi on the version shown. Accepted cells stay as measured. A claim that goes beyond what the row measured will be asked to come out.

The full rules are in CONTRIBUTING.md. Nobody pays for a row and no vendor has a say in a verdict.

If you found a real vulnerability with this suite, follow SECURITY.md and do not open a public issue.

About

Yasha Khandelwal built agmi and maintains it. She is a principal architect with twenty years in enterprise systems, previously at Cisco, Google and SpaceX, and today co-founder and CEO of Tech4Biz Solutions, an engineering firm in Bengaluru.

agmi is her independent work. It is published under her own name and the MIT licence, it carries no company mark, and no vendor pays for a row or has a say in a verdict. Tech4Biz does not fund it and does not appear in its results. Paid assessments that use this suite are offered separately through AuditTrax Labs, and that work never changes what the public scorecard says.

Author
yashakhandelwal.com
ORCID
0009-0005-2166-4951
Email
yasha.khandelwal@tech4biz.io
Code
github.com/tech4biz-yasha/agmi