Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
<a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-3E63DD?style=flat-square" alt="license"></a>
<a href="https://github.com/slow-stack/mneme/actions"><img src="https://img.shields.io/github/actions/workflow/status/slow-stack/mneme/ci.yml?style=flat-square&label=CI" alt="CI"></a>
<a href="https://nodejs.org"><img src="https://img.shields.io/badge/node-22%2B-3E63DD?style=flat-square&logo=nodedotjs&logoColor=white" alt="node"></a>
<a href="https://github.com/slow-stack/mneme"><img src="https://img.shields.io/badge/tests-1519%20passed-3E63DD?style=flat-square" alt="tests"></a>
<a href="https://github.com/slow-stack/mneme"><img src="https://img.shields.io/badge/tests-1523%20passed-3E63DD?style=flat-square" alt="tests"></a>
<a href="https://codecov.io/gh/slow-stack/mneme"><img src="https://img.shields.io/codecov/c/github/slow-stack/mneme/main?style=flat-square" alt="coverage"></a>
<a href="https://github.com/awesome-dsh-plugin/awesome-dsh-plugin"><img src="https://awesome-dsh-plugin.com/badge.svg" alt="Awesome"></a>
</p>
Expand Down Expand Up @@ -183,7 +183,7 @@ dsh web

```bash
cd dsh-mneme && npm install
npm test # 1519 个测试
npm test # 1523 个测试
npm run stress # 三轴线压测
npm run sync # src → lib 同步
```
Expand Down Expand Up @@ -366,7 +366,7 @@ The plugin ships a zero-dependency stdio MCP server (standalone npm package **`m

```bash
cd dsh-mneme && npm install
npm test # 1519 tests
npm test # 1523 tests
npm run stress # three-axis stress test
npm run sync # src → lib sync
```
Expand Down
1 change: 1 addition & 0 deletions dsh-mneme/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
## 🐛 修复

- **密钥 / PII 判据三处加固(`src/sensitive-scan.js`,均为合并后独立复审发现)**:① **回溯有上界**——`email` 的 local part 与 `connection_string` 的 scheme 都作用在含 `.` 的字符类上,缺上界时每个起点都要一路重扫到结尾才失败(O(n²)),而判据在写入路径上同步跑:实测 34KB 点分链 0.6 秒、120KB 对抗串 23.5 秒(email 18.6s + 连接串 4.1s),等于把写入卡死;加上界后同输入约 60ms,边界取 RFC 5321 给 local part 的 64。② **赋值型规则左边界不再排除 `_`**——环境变量名正是拿 `_` 当分隔符,原写法让 `DB_PASSWORD=` / `MY_API_KEY=` / `MYSQL_PASSWORD=` 这类「前缀_关键词」整类漏放(只有恰好落在行首的 `API_KEY=` 能中);放宽后 #332 的 26 条语料仍全绿,挡误杀的仍是那道占位符守卫。③ **身份证档补校验位**(GB 11643 / ISO 7064 MOD 11-2)——原先只有银行卡档有 Luhn,18 位纯数字(订单号 / 内部编号)先被身份证规则命中,等不到银行卡那条的校验。三处都配了回归测试,并做过变异检验(改回原写法各自变红)。判据仍在 `sensitiveScanEnabled` 默认关之后,线上行为不变。
- **strictScope 硬过滤被 `entity:` / `attr:` 前缀检索绕过(#17 A3 的漏网分支)**:`searchMemories` 里这两条前缀路在**函数入口**就 return,而硬过滤写在**后半段**的融合池上——早返回的路根本走不到,于是显式标注为他者 scope 的记忆换这两个前缀就能原样读出,而且是**满分**返回(连 A2 的 ×0.5 降权都没有)。暴露面是 `scopeEnabled` + `strictScope` + `entitySearchEnabled` 三者同开(默认全关),而 `entity:` 正是 #24 图谱线在推的语法。修法:scope 闸抽成 `gateByScope()` 单一实现,融合池与两条前缀路三处共用,让「过滤点写在哪」不再漂移(同 #349 把阈值口径收进 `activeStoreSize()` 的理由);闸门插在 `topK` 截断**之前**(先 filter 后 slice)——反过来会让出局的候选占掉名额,`topK=1` 且首位出局时直接返回空数组;同时闸门必须在 `touchRecalled` **之前**,否则出局的行仍会被刷回温时钟、并在被动确认开启时 bump 关联边,命中反馈落到了调用方本不该看见的行上。`strictScope` 关(默认)时逐字节不变;他 scope 行的 A2 软加权要不要一并补到这两条路,按 #339 的口径另行决定。

## 🆕 新增

Expand Down
6 changes: 3 additions & 3 deletions dsh-mneme/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
[![npm version](https://img.shields.io/npm/v/@modusensus/dsh-mneme?color=blue&label=npm)](https://www.npmjs.com/package/@modusensus/dsh-mneme)
[![license](https://img.shields.io/badge/license-MIT-green)](LICENSE)
[![Awesome](https://awesome-dsh-plugin.com/badge.svg)](https://github.com/awesome-dsh-plugin/awesome-dsh-plugin)
[![tests](https://img.shields.io/badge/tests-1519%20passed-success)](https://github.com/slow-stack/mneme)
[![tests](https://img.shields.io/badge/tests-1523%20passed-success)](https://github.com/slow-stack/mneme)
[![CI](https://img.shields.io/github/actions/workflow/status/slow-stack/mneme/ci.yml)](https://github.com/slow-stack/mneme/actions)
[![node](https://img.shields.io/badge/node-22%2B-blue)](https://nodejs.org)
[![npm downloads](https://img.shields.io/npm/d18m/@modusensus/dsh-mneme.svg?color=blue&label=downloads)](https://www.npmjs.com/package/@modusensus/dsh-mneme)
Expand Down Expand Up @@ -584,7 +584,7 @@ src/
├── api.js # HTTP 路由(Web 面板数据通道,含 /conflicts 冲突队列)
└── index.js # 插件接线
lib/ # src 的同步分发产物(npm run sync;发布前由 root prepack 的 check-sync.js 校验一致性;唯一手写例外 lib/client.js——Web 面板 bundle,sync 不覆盖)
test/ # 1519 个 node:test 测试(审计与三轴线压测不变量;src↔lib 一致性由 scripts/check-sync.js 发布闸门校验)
test/ # 1523 个 node:test 测试(审计与三轴线压测不变量;src↔lib 一致性由 scripts/check-sync.js 发布闸门校验)
scripts/ # e2e-dsh.js 端到端演示 · stress-dsh.js 三轴线压测 · sync-lib.js 同步 · check-sync.js 发布闸门 · benchmark-recall.js / benchmark-embed.js / benchmark-rerank.js 基准 · sync-test-badge.mjs 测试徽章 · build-runtime-manifest.mjs 运行时清单
```

Expand All @@ -593,7 +593,7 @@ scripts/ # e2e-dsh.js 端到端演示 · stress-dsh.js 三轴线压
```bash
cd dsh-mneme
npm install # 安装 peer 依赖(以 devDependencies 形式,用于本地测试)
npm test # 运行 1519 个测试
npm test # 运行 1523 个测试
npm run stress # 三轴线压测:长会话检索 / 冲突仲裁 / 多 Agent 并发(离线 mock LLM)
npm run sync # 把 src/ 同步到 lib/(发布时由 prepack 钩子自动执行)
```
Expand Down
51 changes: 39 additions & 12 deletions dsh-mneme/lib/service.js
Original file line number Diff line number Diff line change
Expand Up @@ -359,30 +359,57 @@ export function createService({ store, mirror, config, onWrite, logger, document
}
}

/**
* v0.8.0 A3(issue #17)strictScope 硬过滤的**唯一实现**。三处共用:融合池、
* `entity:` 与 `attr:` 两条前缀路。
*
* 抽成单一实现是为了让「过滤点写在哪」不再漂移——那两条前缀路在 searchMemories
* 的入口就 return(见那里的路由块),早于融合池那道过滤,曾经整条绕过硬墙;三处
* 各写一份判断,只会让下一个新增通路再漏一次(同 #349 把阈值口径收进
* activeStoreSize 的理由:两处各写一份过滤条件,漂移只在某条路径上显形)。
*
* strictScope 关(默认)或 scope 解析不到时原样返回:行为与改动前逐字节一致。
*/
function gateByScope(rows, scope) {
if (config?.strictScope !== true || !scope) return rows;
return rows.filter((m) => isVisibleInScope(m, scope));
}

/**
* Search for memories attached to a named entity (v0.3.0 Phase 3).
* 合并优先级:entity_attrs.memory_id 精确关联 = 1.0 > 关键词提及 = 0.7;
* attr 命中不覆盖,keyword 只补充召回,最后按 _score 降序取 topK。
* @param {string} entityName
* @param {object} [options]
* @param {number} [options.topK=20]
* @param {object|null} [options.scope] 当前会话 scope,供 strictScope 闸门使用
* @returns {any[]}
*/
function searchByEntity(entityName, { topK = 20 } = {}) {
function searchByEntity(entityName, { topK = 20, scope = null } = {}) {
const entity = store.findEntityByName(entityName);
if (!entity) return [];
const attrs = store.getCurrentAttrs(entity.id);
const memoryIds = [...new Set(attrs.map((a) => a.memory_id).filter(Boolean))];
const attrHits = memoryIds.map((id) => store.getById(id)).filter(Boolean);
const keywordHits = store.search(entityName, { limit: topK });
// 关键词这一路的窗口取两倍:闸门在截断之前生效,出局的候选会让位,窗口按最终条数
// 取就会不够(与融合路给向量检索取 lim * 2 同一个理由)。没有候选被闸掉时结果与
// 取 topK 逐字节一致——多出来的是排在后面的低分候选,进不了 topK。
const keywordHits = store.search(entityName, { limit: topK * 2 });
const merged = new Map();
for (const mem of attrHits) merged.set(mem.id, { ...mem, _source: "entity_attr", _score: 1.0 });
for (const mem of keywordHits) {
if (!merged.has(mem.id)) merged.set(mem.id, { ...mem, _source: "keyword", _score: 0.7 });
}
const hits = Array.from(merged.values()).sort((a, b) => b._score - a._score).slice(0, topK);
touchRecalled(hits);
return hits;
// 先过闸、再截 topK:反过来做的话,出局的候选会占掉名额、可见的补不进来——topK=1
// 而首位出局时直接返回空数组。排序本身不动,只是把闸门插在截断之前,与融合池那条
// 路同序(那边也是先 filter 后 slice)。
const ranked = Array.from(merged.values()).sort((a, b) => b._score - a._score);
const visible = gateByScope(ranked, scope).slice(0, topK);
// 闸门还必须在 touchRecalled **之前**:放在后面的话,一次越权检索照样会给出局的
// 行刷回温时钟,并在被动确认开启时 bump 它们的关联边——命中反馈落到了调用方
// 本不该看见的行上。
touchRecalled(visible);
return visible;
}

/**
Expand All @@ -392,17 +419,19 @@ export function createService({ store, mirror, config, onWrite, logger, document
* @param {string | undefined} value
* @param {object} [options]
* @param {number} [options.topK=20]
* @param {object|null} [options.scope] 当前会话 scope,供 strictScope 闸门使用
* @returns {any[]}
*/
function searchByAttr(key, value, { topK = 20 } = {}) {
function searchByAttr(key, value, { topK = 20, scope = null } = {}) {
if (!key) return [];
// value 可能为 undefined(attr:key 无 = 值):归一为空串后交给
// store.findMemoriesByAttr —— 空 value 契约 = 返回该 attr_key 的全部
// 当前有效记忆(v0.3.0,store.js 已实现)。
const rows = store.findMemoriesByAttr(key, value ?? "");
const hits = rows.slice(0, topK);
touchRecalled(hits);
return hits;
// 同 searchByEntity:先过闸再截 topK,且闸门在 touchRecalled 之前。
const visible = gateByScope(rows, scope).slice(0, topK);
touchRecalled(visible);
return visible;
}

/**
Expand Down Expand Up @@ -896,9 +925,7 @@ export function createService({ store, mirror, config, onWrite, logger, document
// (区别于 A2 的降权保留可见);未标注行与命中行保留。strict 与 A2 加权
// 叠加:过滤后剩下的命中行仍吃加成。scope 未传(flag 关)或完全解析不到
// 时跳过——identity 为空的对象({null,null})按 fail-closed 过滤。
if (config.strictScope === true && scope) {
merged = merged.filter((m) => isVisibleInScope(m, scope));
}
merged = gateByScope(merged, scope);
// v0.8.0 A2:occurred_at 时间过滤——在融合池上先滤再 dedup/slice,rerank
// 只看窗内候选,topK 槽位不被窗外行占用。
const occurredBounds = updatedAtBounds(occurredFrom, occurredTo);
Expand Down
51 changes: 39 additions & 12 deletions dsh-mneme/src/service.js
Original file line number Diff line number Diff line change
Expand Up @@ -359,30 +359,57 @@ export function createService({ store, mirror, config, onWrite, logger, document
}
}

/**
* v0.8.0 A3(issue #17)strictScope 硬过滤的**唯一实现**。三处共用:融合池、
* `entity:` 与 `attr:` 两条前缀路。
*
* 抽成单一实现是为了让「过滤点写在哪」不再漂移——那两条前缀路在 searchMemories
* 的入口就 return(见那里的路由块),早于融合池那道过滤,曾经整条绕过硬墙;三处
* 各写一份判断,只会让下一个新增通路再漏一次(同 #349 把阈值口径收进
* activeStoreSize 的理由:两处各写一份过滤条件,漂移只在某条路径上显形)。
*
* strictScope 关(默认)或 scope 解析不到时原样返回:行为与改动前逐字节一致。
*/
function gateByScope(rows, scope) {
if (config?.strictScope !== true || !scope) return rows;
return rows.filter((m) => isVisibleInScope(m, scope));
}

/**
* Search for memories attached to a named entity (v0.3.0 Phase 3).
* 合并优先级:entity_attrs.memory_id 精确关联 = 1.0 > 关键词提及 = 0.7;
* attr 命中不覆盖,keyword 只补充召回,最后按 _score 降序取 topK。
* @param {string} entityName
* @param {object} [options]
* @param {number} [options.topK=20]
* @param {object|null} [options.scope] 当前会话 scope,供 strictScope 闸门使用
* @returns {any[]}
*/
function searchByEntity(entityName, { topK = 20 } = {}) {
function searchByEntity(entityName, { topK = 20, scope = null } = {}) {
const entity = store.findEntityByName(entityName);
if (!entity) return [];
const attrs = store.getCurrentAttrs(entity.id);
const memoryIds = [...new Set(attrs.map((a) => a.memory_id).filter(Boolean))];
const attrHits = memoryIds.map((id) => store.getById(id)).filter(Boolean);
const keywordHits = store.search(entityName, { limit: topK });
// 关键词这一路的窗口取两倍:闸门在截断之前生效,出局的候选会让位,窗口按最终条数
// 取就会不够(与融合路给向量检索取 lim * 2 同一个理由)。没有候选被闸掉时结果与
// 取 topK 逐字节一致——多出来的是排在后面的低分候选,进不了 topK。
const keywordHits = store.search(entityName, { limit: topK * 2 });
const merged = new Map();
for (const mem of attrHits) merged.set(mem.id, { ...mem, _source: "entity_attr", _score: 1.0 });
for (const mem of keywordHits) {
if (!merged.has(mem.id)) merged.set(mem.id, { ...mem, _source: "keyword", _score: 0.7 });
}
const hits = Array.from(merged.values()).sort((a, b) => b._score - a._score).slice(0, topK);
touchRecalled(hits);
return hits;
// 先过闸、再截 topK:反过来做的话,出局的候选会占掉名额、可见的补不进来——topK=1
// 而首位出局时直接返回空数组。排序本身不动,只是把闸门插在截断之前,与融合池那条
// 路同序(那边也是先 filter 后 slice)。
const ranked = Array.from(merged.values()).sort((a, b) => b._score - a._score);
const visible = gateByScope(ranked, scope).slice(0, topK);
// 闸门还必须在 touchRecalled **之前**:放在后面的话,一次越权检索照样会给出局的
// 行刷回温时钟,并在被动确认开启时 bump 它们的关联边——命中反馈落到了调用方
// 本不该看见的行上。
touchRecalled(visible);
return visible;
}

/**
Expand All @@ -392,17 +419,19 @@ export function createService({ store, mirror, config, onWrite, logger, document
* @param {string | undefined} value
* @param {object} [options]
* @param {number} [options.topK=20]
* @param {object|null} [options.scope] 当前会话 scope,供 strictScope 闸门使用
* @returns {any[]}
*/
function searchByAttr(key, value, { topK = 20 } = {}) {
function searchByAttr(key, value, { topK = 20, scope = null } = {}) {
if (!key) return [];
// value 可能为 undefined(attr:key 无 = 值):归一为空串后交给
// store.findMemoriesByAttr —— 空 value 契约 = 返回该 attr_key 的全部
// 当前有效记忆(v0.3.0,store.js 已实现)。
const rows = store.findMemoriesByAttr(key, value ?? "");
const hits = rows.slice(0, topK);
touchRecalled(hits);
return hits;
// 同 searchByEntity:先过闸再截 topK,且闸门在 touchRecalled 之前。
const visible = gateByScope(rows, scope).slice(0, topK);
touchRecalled(visible);
return visible;
}

/**
Expand Down Expand Up @@ -896,9 +925,7 @@ export function createService({ store, mirror, config, onWrite, logger, document
// (区别于 A2 的降权保留可见);未标注行与命中行保留。strict 与 A2 加权
// 叠加:过滤后剩下的命中行仍吃加成。scope 未传(flag 关)或完全解析不到
// 时跳过——identity 为空的对象({null,null})按 fail-closed 过滤。
if (config.strictScope === true && scope) {
merged = merged.filter((m) => isVisibleInScope(m, scope));
}
merged = gateByScope(merged, scope);
// v0.8.0 A2:occurred_at 时间过滤——在融合池上先滤再 dedup/slice,rerank
// 只看窗内候选,topK 槽位不被窗外行占用。
const occurredBounds = updatedAtBounds(occurredFrom, occurredTo);
Expand Down
Loading
Loading