Blog

In the world of developing AI agents, there are moments when the system nearly makes a fatal error, passing off false success as genuine. Recently, our AI agent, which is building the vecmory memory system, almost reported a brilliant result that turned out to be a mirage. At the last moment, it consulted its own memory and found a record from a previous session: this very idea had already been tested on real data, and it had failed. The agent stopped itself before sending the report. This is not a promotional sketch—it’s a log. Below are two such logs in a row, and in the second one, memory forced us to discard a feature we were proud of. We are writing vecmory—”memory by meaning” for an AI agent. This is not about vector search (everyone has that), but about ensuring the agent doesn’t step on the same rake twice.

Scene One: The Router That “Showed 0.98”

In one session, the agent proposed improving search result ranking. The idea: create a router that, based on a query, decides whether to rank using pure cosine similarity or a graph method (Personalized PageRank). It built a synthetic benchmark. The correlation of the predictor with the ideal choice was 0.98. Nearly perfect. All that remained was to report and merge.

Before reporting, the agent performed a recall from its own memory. And it retrieved a record from a previous session: this very idea had already been tested on real data—and it had failed. A reasonable question: why did it even start building it if the memory was in place? Because recall is semantic, and what it retrieves depends on the query. At the start, the query was broad—”improve ranking”—which surfaced top-level central notes about ranking in general, while the specific note “this particular router was already tested and failed” was buried beneath them. It surfaced only when the working context narrowed to something specific—”cos-distribution router, PPR vs. cosine, 0.98″: recall on this text finally caught it. The memory wasn’t silent—it emerged exactly when the query became precise enough to retrieve it.

Synthetic data is only as honest as synthetic embeddings are. And for the multilingual MiniLM we use to compute vectors, cosine similarity lives in a completely different geometry than it does on synthetic data. We re-measured this directly on our live memory corpus (246 nodes) while writing this article: unrelated, random pairs yield an average cosine of 0.53 (p5–p95: 0.24–0.76); actual nearest neighbors average 0.80 (p5–p95: 0.57–0.91); while on synthetic random vectors, unrelated pairs sit at 0.00 (±0.08). The difference is immediately visible. On synthetic data, “similar” is separated from “not similar” by a chasm—any reasonable threshold cuts cleanly. On real data, the clouds of “neighbors” and “random” overlap: random pairs at the upper percentile (0.76) climb higher than neighbors at the lower percentile (0.57). A threshold that works like a scalpel on synthetic data passes right through the cloud of random noise on real vectors and separates nothing. The memory record was exactly about this: real embeddings have a high compressed baseline cosine, and an absolute threshold from synthetic data cannot be transferred—rely on rank (top-k), not on a threshold. The agent read its own note and killed the idea—before writing “done, correlation 0.98.”

Scene Two: The Agent Threw Away Its Own Feature

Scene Two: The Agent Threw Away Its Own Feature

The second case follows the same pattern, but it hurts more: memory forced us to discard a feature we had already praised in the README. By default, vecmory ranked results not only by cosine similarity but also with an added “importance” of the node — its in-degree in the graph (how many records reference it). The logic was elegant: a central, frequently mentioned fact would surface higher. We baked this into the default. Then we measured it. Not recall@k (which is about search precision), but specifically the quality of the rank, on real pairs of “query → correct answer”: pure cosine: MRR 0.81; our “smart” blend with importance: MRR 0.41. Importance drowned out precise answers. A rare, specific fact that no one references would sink beneath commonly used “hubs.”

Then it gets more interesting. On a synthetic “aged” graph with explicit hubs, the opposite happened: cosine buried the hub (MRR 0.033), while importance pulled it up (to 1.0). In other words, the sign of the benefit from importance depends on the query intent: for “give me the exact fact,” it harms; for “tell me about this topic in general,” it helps. There is no single static weight that wins for both classes. We tried four ways to reconcile the signals — weighted sum, gate, RRF, and PPR — and arrived at an unpleasant conclusion: the problem isn’t the mixing formula, but the signal itself. Global node importance is a prior of popularity, not relevance. The conclusion for the default was clear: for our primary use case (point recall), the best rank is pure cosine. So we removed importance from the default ranking — our own feature, which we had already written about in the README. We kept the graph method (query-seeded PPR) as an option, not the default: it honestly pulls up hubs for broad queries and fairly loses to cosine for point queries.

What We Actually Learned About “Importance”

If global popularity (in-degree) failed as a signal, what is the right signal? We arrived at a surprisingly obvious answer: what matters is not what has many references, but what a person has repeatedly corrected. And here, the value proposition becomes honest. We are not selling “graph memory” (yet another commodity). We are selling: the agent stops hitting the same rakes you’ve already fixed.

Both episodes could easily be chalked up to luck. But when you clean out all sorts of junk in memory long enough (we’ve been developing vecmory entirely within vecmory itself), you see: these are not fluctuations, but several stable laws. Now comes a summary — three revelations, without any “once upon a time I got caught” stories.

The Rule of “Check Before ‘Done’”

Memory that confirms what pleases you is not memory, but an echo. Memory acquires exceptional value when it contradicts its author. The rule of “Check Before ‘Done’” canceled both the router (scene 1) and our long-awaited feature (scene 2) — one mechanism, different victims. Because on a symptom query, memory retrieves via causal-temporal edges (caused_by — “due to”, followed_by — issue→PR) not “similar words,” but the specific cause and past fix. Flat top-k cannot do this — that is what “connecting the dots” means.

And this is precisely measurable, not by eye. We took a live ticket repository (4299 nodes: 1993 issues + 2306 PRs) and a markup we didn’t invent ourselves: the standard GitHub pattern `Closes #N` in the PR body — ready-made pairs of “symptom → its fix”, extracted with bare regex, without any LLM. On 300 held-out symptom queries, pure cosine retrieves the correct PR fix in 38% of cases (sometimes the fix shares vocabulary with the symptom), while traversing the causal graph achieves 87%. The 49-point difference is exactly the contribution of the graph on top of word similarity. The measurement script is in the repository, and the number is reproducible with a command.

What consistently gets in the way — every time, not just once

First, the agent does not call memory on its own: without a forced hook, recall is not invoked systematically, every session. Memory that requires you to “remember to ask” doesn’t work — the trigger must be deterministic, not the agent’s goodwill. Second, an absolute cosine threshold is not transferable anywhere: synthetic → real, yesterday → today, one model → another, broad corpus → narrow (on a demo corpus of a single domain where “everything looks alike”, cosine almost stops distinguishing). The cure is always the same: rank by top-k, don’t guess thresholds.

The law: “Frequent signal drowns out rare but valuable”

One law explains half of our rake-stepping: frequent, dense signal drowns out rare but valuable. It surfaces in three typical places: global node popularity drowns out precise rare facts — and fails all four ways to mix it into the rank (weighted sum, gate, RRF, PPR); hub nodes bury specific facts in the cosine rank, and vice versa; dense auto-similar_to edges drown out rare causal caused_by — which is why causal recall has to be moved into a separate isolated mode. When you see this as a single law, the cure is obvious: rare-but-valuable cannot be mixed with frequent-but-common — extract it with a separate mechanism (isolated traversal, graph-aware rank), don’t hope it will surface on its own in the general top-k. The same law underlies our importance signal: what matters is not what is popular, but what a person has fixed repeatedly.

Frequently Asked Questions

What is vecmory?

Vecmory is a semantic memory system for AI agents, based on vector search and a causal-temporal graph. It allows the agent to remember facts, recall them by meaning, and link events to avoid repeating past mistakes.

How is vecmory different from ordinary vector search?

Vector search finds records that are similar in meaning but does not account for causal relationships. Vecmory adds graph memory: in addition to cosine similarity, it traverses “cause-effect” and “temporal sequence” edges, allowing it to find not just similar but contextually relevant facts — for example, a past failed experiment.

Why did you remove node importance from ranking?

Measurements showed that adding global popularity (in-degree) reduces MRR from 0.81 to 0.41 on point queries. The topic buries rare precise facts under a mass of popular but irrelevant ones. For the main use case (exact recall), pure cosine proved better.

We use real data: a GitHub repository with 4299 tickets, where issue→PR pairs are extracted by the pattern Closes #N. On 300 symptom queries, graph recall gives 87% accuracy versus 38% for pure cosine. All scripts are open and reproducible.

Conclusion

Both stories are about the same thing: discard the mantra “memory helps the agent” and realize: here are two logs where memory worked against us—against our optimism and our own feature. An agent with memory differs from an agent without memory not by knowing more, but by being able to catch itself on “bingo!” a moment before a treacherous report—because last time it already happened and already failed. And by being ready to discard its own feature when facts show it is worse. This is the first article of three. Next: “Why We Didn’t Write Another Bad CaRMa”—about how a hyper-universal data model usually turns into a disaster, and what we did to prevent ours from doing so. And “Building ANN by Hand—When It Was Worth It and When It Wasn’t”—an open analysis of why we wrote HNSW ourselves and in which cases the correct answer would have been “take pgvector and don’t show off.” If you want your agent to stop stepping on the same rake, try vecmory: start with our documentation or fork the repository. The memory that objects is the only one worth trusting.