Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 7 additions & 6 deletions .github/workflows/e2e-harness.yml
Original file line number Diff line number Diff line change
Expand Up @@ -427,9 +427,9 @@ jobs:
- name: Test long-query retrieval and admission without skips
env:
POWERCONTEXT_TEST_ATOMIC_MEMORY_FTS_PATH: legacy
# The count cap keeps 13 cases within 7 per shard, independent of duration estimates.
# 7 * (300s + 10s grace) + 190s collection < 40m.
timeout-minutes: 40
# The count cap keeps 16 cases within 8 per shard, independent of duration estimates.
# 8 * (300s + 10s grace) + 190s collection < 45m.
timeout-minutes: 45
run: |
uv run --locked python scripts/run_atomic_memory_clean_migration.py \
oceanbase --test-timeout 300 --shard "${{ matrix.shard }}/2" --durations .github/oceanbase-migration-durations.json --output "$MIGRATION_REPORTS/long-query" --tests \
Expand Down Expand Up @@ -486,8 +486,8 @@ jobs:
${{ !cancelled() && contains(fromJSON('["schedule","workflow_dispatch"]'), github.event_name) && (github.event_name != 'workflow_dispatch' || inputs.suite != 'recall-effort') &&
((inputs.suites == '' && needs.changes.result != 'success') || fromJSON(inputs.suites || needs.changes.outputs.suites || '{}').database) }}
runs-on: ${{ (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository) && 'powercontext-ci-standard' || 'ubuntu-24.04' }}
# Five cases (including three parser fallbacks), bounded setup, and diagnostics.
timeout-minutes: 65
# Eight cases (including three parser fallbacks), bounded setup, and diagnostics.
timeout-minutes: 80
env:
OCEANBASE_CI_CONTAINER: atomic-memory-native-fts-oceanbase
OCEANBASE_IMAGE: ghcr.io/oceanbase/oceanbase-ce:4.4.2.1-101000022026050611
Expand Down Expand Up @@ -528,7 +528,8 @@ jobs:
--output "$MIGRATION_REPORTS/startup"

- name: Test native FTS retrieval and admission without skips
timeout-minutes: 30
# 8 * (300s + 10s grace) + 190s collection < 45m.
timeout-minutes: 45
run: |
uv run --locked python scripts/run_atomic_memory_clean_migration.py \
oceanbase --test-timeout 300 --output "$MIGRATION_REPORTS/native-fts" --tests \
Expand Down
10 changes: 9 additions & 1 deletion docs/en/docs/operate/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,11 @@ budget does not replay a write whose commit outcome is unknown after a connectio

Source processing generates candidates, then enumerates related active memories satisfying permission, tag and
retrieval eligibility. Fts enumerates all qualifying matches; vector enumerates all results within its threshold;
hybrid combines both. Neither comparison batch size nor token budget truncates the total candidates. Oversized
hybrid combines both. Lexical relatedness counts both sides: a memory qualifies when the distinct candidate terms it
contains meet the lexical admission threshold for the candidate's distinct terms, or number at least the minimum match
count and cover the coverage fraction of the memory's own indexed words, including its kind. A long candidate therefore
still finds a short memory of the same fact. Ordinary search and `prepare_context` keep the query-only threshold.
Neither comparison batch size nor token budget truncates the total candidates. Oversized
comparison input reduces the batch until every candidate is processed. A single object or complete extraction input
that exceeds the budget fails the window without partial publication or cursor advancement. Fallback changes recall
evidence and requires an explicit choice; by default errors retain the window for diagnosis and retry.
Expand Down Expand Up @@ -650,6 +654,10 @@ Atomic Memory uses native `minimum_should_match` filtering when the connected Oc
Older engines retain the whole-word `LOCATE` coverage check. Capability detection does not treat query timeouts,
permission failures, or connection errors as an unsupported feature. Both paths keep the same analyzed, deduplicated
query terms and admission threshold, including recovery probes and complete related-memory enumeration.
For related-memory enumeration, the native path also matches memories that share at least the minimum match count of
terms and are short enough to qualify below the candidate's threshold, then applies whole-word coverage to them. For a
long candidate almost every memory is that short, so the whole-word check runs on nearly every memory sharing that
many terms.
Queries with more than 256 distinct terms use native filtering only on identified OceanBase 4.6 or newer engines.
Earlier OceanBase versions, seekDB, and unrecognized version strings retain whole-word coverage for these queries:
OceanBase CE 4.4.2.1 accepts the native syntax but can ignore the threshold above that term count.
Expand Down
18 changes: 10 additions & 8 deletions docs/en/docs/workflows/atomic-memory.md
Original file line number Diff line number Diff line change
Expand Up @@ -267,14 +267,16 @@ Client methods for creation, replacement and exact historical reads.
Search and list require `scope.read`, following the Memory and Topic Memory Scope boundary. An individual
Artifact share permits its exact get and history reads; it does not grant a Scope inventory or search.
If Scope read is revoked before reranking or context assembly, the request returns `403` even when an individual share remains.
Source extraction enumerates the same-Scope threshold set owned by its principal in pages until enumeration
ends: the first call ranks the complete set once, and later calls only read that ranking's page bodies. The
configured authorizer then checks Artifact read and write before comparing a memory. The page size limits one
page's bodies and authorization batch only, never the total; the fixed ranking is not recomputed when another
Scope changes the corpus. Each page's read and authorization share one consistent snapshot, and compared read
dependencies survive release until commit rechecks their body revisions. A body revision published after ranking
makes the window conflict and retry, and an exhausted retry budget fails explicitly. Denied candidates are
skipped; authorization failures stop processing. The immutable Owner columns are only an extraction prefilter.
Source extraction enumerates every active same-Scope memory that meets the related threshold, without an owner
prefilter, in pages until enumeration ends: the first call ranks the complete set once, and later calls only read
that ranking's page bodies. The configured authorizer then checks Artifact read and write before comparing a memory.
The page size limits one page's bodies and authorization batch only, never the total; the fixed ranking is not
recomputed when another Scope changes the corpus. Each page's read and authorization share one consistent snapshot,
and compared read dependencies survive release until commit rechecks their body revisions. A body revision
published after ranking makes the window conflict and retry, and an exhausted retry budget fails explicitly. Denied
candidates are skipped; authorization failures stop processing. In fts and hybrid related modes the lexical
threshold also measures the memory's own indexed words, so a long candidate still reaches a short memory of the same
fact.

Scope checks using the same database with supported Builtin or Casbin providers share the business snapshot.
Independent decision stores and custom providers retain their configured read consistency contracts.
Expand Down
10 changes: 8 additions & 2 deletions docs/zh/docs/operate/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,10 @@ PowerContext 进程启动时从环境变量读取配置。当前工作目录存
带 token 的恢复遇到变化会返回 `preview_stale`,不会自动重新解释意图;重试预算也不处理提交结果未知的连接失败。

Source 处理先生成候选,再枚举满足权限、标签及检索资格的相关 active 记忆。fts 枚举全部合格匹配,vector 枚举
阈值内全部结果,hybrid 合并两者。总候选不会按 comparison batch size 或 token 限额截断。
阈值内全部结果,hybrid 合并两者。词法相关性同时衡量两侧:记忆包含的候选不同词数达到候选不同词数的词法
准入阈值,或者不少于最少匹配词数且覆盖该记忆自身索引词(含 kind)的覆盖比例,即算相关,因此较长的候选
仍能找到同一事实的较短记忆。普通搜索和 `prepare_context` 仍只按查询词计算阈值。总候选不会按 comparison
batch size 或 token 限额截断。
比较输入过大时缩小单批,直到所有候选都处理完;连单个对象或完整抽取输入都超预算时窗口失败,不部分提交或推进 cursor。
开启 fallback 会改变相关召回依据,应显式选择;默认错误会保留当前窗口供排查和重试。

Expand Down Expand Up @@ -587,7 +590,10 @@ URL 必须使用 `mysql+aoceanbase` 驱动、明确的端口,并设置 `charse

Atomic Memory 会在连接的 OceanBase 或 seekDB 引擎支持时,使用原生 `minimum_should_match` 过滤;旧版引擎继续使用
完整词 `LOCATE` 覆盖检查。能力检测不会把查询超时、权限错误或连接错误当作功能不支持。两条路径使用相同的分析、去重后的
查询词和准入阈值,也适用于恢复探测与完整关联记忆枚举。超过 256 个不同词的查询,仅在确认版本为 OceanBase 4.6 或更新版本时使用
查询词和准入阈值,也适用于恢复探测与完整关联记忆枚举。关联记忆枚举的原生路径还会匹配与候选共享至少
最少匹配词数、且短到可低于候选阈值入选的记忆,再对这些记忆做完整词覆盖检查。候选较长时,几乎所有记忆
都满足这个长度条件,因此几乎每条共享这么多词的记忆都要做完整词覆盖检查。
超过 256 个不同词的查询,仅在确认版本为 OceanBase 4.6 或更新版本时使用
原生过滤;更早的 OceanBase、seekDB 和无法识别版本的引擎继续使用完整词覆盖检查,因为 OceanBase CE 4.4.2.1 虽然接受原生语法,
但超过该词数时可能忽略匹配阈值。查询包含 `com` 等被 SPACE 分词器省略的 Analyzer 词时,检索、完整枚举和恢复探测
也会保留完整词覆盖检查,阈值仍按所有不同的 Analyzer 词计算。该选择无需新增配置、表或索引迁移,沿用现有 `SPACE` 全文索引。
Expand Down
14 changes: 7 additions & 7 deletions docs/zh/docs/workflows/atomic-memory.md
Original file line number Diff line number Diff line change
Expand Up @@ -247,13 +247,13 @@ unsupported 通过 `ServerResponseError.status_code`、`.code`、`.details` 检

搜索和列表沿用 Memory、Topic Memory 的 Scope 边界,要求 `scope.read`。单条 Artifact 分享允许
精确读取和历史读取,不授予整个 Scope 的列表或搜索权限。重排或上下文组装前若 Scope 读取权限已被
撤销,请求返回 `403`,即使单条分享仍有效。Source 提取会在同 Scope 中枚举满足阈值、属于当前主体的
记忆(首次调用一次性排出完整轻量排名,后续只按该排名逐页读取正文,直到枚举结束),再由实际配置的授权服务
检查 Artifact 读取和写入权限后交给模型比较。页大小只限制单页正文与鉴权批量,不限制总数;排名固定后
不因其他 Scope 的语料变化重算。每页读取与鉴权在同一一致性快照内完成,已比较身份的读取依赖保留到提交,
提交时重新校验其正文 revision。排名之后若某条记忆发布新的正文 revision,该窗口以冲突重试,
重试上限用尽即显式失败。明确拒绝的候选会被跳过;授权服务故障会终止处理。不可变的 Owner 列只用于
提取时的所有者预筛选。
撤销,请求返回 `403`,即使单条分享仍有效。Source 提取会在同 Scope 中枚举满足相关阈值的全部 active
记忆,不按所有者预筛选(首次调用一次性排出完整轻量排名,后续只按该排名逐页读取正文,直到枚举结束),再由
实际配置的授权服务检查 Artifact 读取和写入权限后交给模型比较。页大小只限制单页正文与鉴权批量,不限制总数;
排名固定后不因其他 Scope 的语料变化重算。每页读取与鉴权在同一一致性快照内完成,已比较身份的读取依赖保留到
提交,提交时重新校验其正文 revision。排名之后若某条记忆发布新的正文 revision,该窗口以冲突重试,重试上限
用尽即显式失败。明确拒绝的候选会被跳过;授权服务故障会终止处理。在 fts 和 hybrid 关联模式下,词法阈值也会
衡量记忆自身的索引词,较长的候选仍能找到同一事实的较短记忆。

使用同一数据库的受支持 Builtin 或 Casbin 服务进行 Scope 检查时,授权读取与业务数据共享快照。
独立的决策存储和自定义服务保留各自配置的读取一致性契约。
Expand Down
4 changes: 2 additions & 2 deletions release/upgrade-support.json

Large diffs are not rendered by default.

23 changes: 14 additions & 9 deletions src/powercontext/builtin/artifacts/search.py
Original file line number Diff line number Diff line change
Expand Up @@ -381,17 +381,21 @@ def fts_query_requirements(value: str, /, *, floor: AdmissionFloor | None = None
query_terms = tuple(sorted(set(analyze_fts_query(value).split())))
if not query_terms:
return (), 0
return query_terms, fts_required_matches(len(query_terms), floor=floor)


def fts_required_matches(term_count: int, /, *, floor: AdmissionFloor | None = None) -> int:
"""Return how many shared terms the admission threshold requires of a set of ``term_count`` terms.

A set of at most two terms requires one match; a larger set requires
``max(lexical_min_matched_terms, ceil(lexical_coverage * term_count))``.
"""

if term_count <= _FTS_SHORT_QUERY_MAX_TERMS:
return 1
coverage = _FTS_MIN_QUERY_COVERAGE if floor is None else floor.lexical_coverage
min_matched = _FTS_MIN_MATCHED_TERMS if floor is None else floor.lexical_min_matched_terms
required_matches = (
1
if len(query_terms) <= _FTS_SHORT_QUERY_MAX_TERMS
else max(
min_matched,
math.ceil(len(query_terms) * coverage),
)
)
return query_terms, required_matches
return max(min_matched, math.ceil(term_count * coverage))


def fts_match_query(value: str) -> str | None:
Expand Down Expand Up @@ -434,6 +438,7 @@ def admits_fts_projection(query: str, projection: str, /, *, floor: AdmissionFlo
"analyze_text_with_spans",
"fts_match_query",
"fts_query_requirements",
"fts_required_matches",
"lexical_search_score",
"unit_l2_cosine_similarity",
]
53 changes: 53 additions & 0 deletions src/powercontext/builtin/persistence/atomic_memory_index.py
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@
RecallChannelWeights,
analyze_text,
fts_query_requirements,
fts_required_matches,
lexical_search_score,
)
from powercontext.builtin.inference import EmbeddingModel
Expand Down Expand Up @@ -474,6 +475,56 @@ def oceanbase_fts_coverage_sql(terms: tuple[str, ...], /) -> tuple[str, dict[str
return coverage[0] if coverage else "0", parameters


def _memory_words_requirement(word_count: int, floor: AdmissionFloor | None) -> int:
floor = DEFAULT_ADMISSION_FLOOR if floor is None else floor
return max(floor.lexical_min_matched_terms, math.ceil(word_count * floor.lexical_coverage))


def atomic_memory_related_requirement(
term_count: int, word_count: int, /, *, floor: AdmissionFloor | None = None
) -> int:
"""Return how many distinct candidate terms a stored memory must contain to be related.

The shared terms must meet the lexical admission threshold for the candidate's ``term_count`` distinct terms, or
number at least the minimum match count and cover the coverage fraction of the memory's own ``word_count`` indexed
words. The requirement never grows beyond what a short memory can supply, so a long candidate still finds a short
memory of the same fact. Prepare-time search keeps the query-only threshold.
"""

return min(fts_required_matches(term_count, floor=floor), _memory_words_requirement(word_count, floor))


def atomic_memory_related_requirement_sql(
term_count: int, dialect: Literal["sqlite", "mysql"], /, *, floor: AdmissionFloor | None = None
) -> tuple[str, dict[str, object], int]:
"""Render ``atomic_memory_related_requirement`` over each row's indexed words.

Returns the SQL expression, its parameters, and the lowest requirement any row can have. A shared-term count is an
integer, so comparing it with ``words * coverage`` equals comparing it with that product's ceiling. Both backends
multiply in double precision, as Python does, so SQL and Python admit the same rows.
"""

floor = DEFAULT_ADMISSION_FLOOR if floor is None else floor
required = fts_required_matches(term_count, floor=floor)
parameters: dict[str, object] = {"fts_related_required": required}
lowest = min(required, floor.lexical_min_matched_terms)
if lowest == required:
return ":fts_related_required", parameters, lowest
length, least, greatest = ("length", "MIN", "MAX") if dialect == "sqlite" else ("CHAR_LENGTH", "LEAST", "GREATEST")
# Analyzed text joins encoded words with single spaces; encoded words never contain one.
words = (
f"(CASE WHEN searchable_text = '' THEN 0 ELSE {length}(searchable_text) - "
f"{length}(REPLACE(searchable_text, ' ', '')) + 1 END)"
)
parameters.update(
fts_related_min_matched=floor.lexical_min_matched_terms, fts_related_coverage=float(floor.lexical_coverage)
)
expression = (
f"{least}(:fts_related_required, {greatest}(:fts_related_min_matched, {words} * :fts_related_coverage))"
)
return expression, parameters, lowest


def atomic_memory_filter_sql(
filters: AtomicMemoryIndexFilter, dialect: Literal["sqlite", "mysql"], /
) -> tuple[str, dict[str, object]]:
Expand Down Expand Up @@ -1117,6 +1168,8 @@ def validate_request(request: AtomicMemorySearchRequest | AtomicMemoryRelatedReq
"atomic_memory_page_from_ranks",
"atomic_memory_page_slice",
"atomic_memory_profile_fingerprint",
"atomic_memory_related_requirement",
"atomic_memory_related_requirement_sql",
"combine_atomic_memory_channels",
"combine_atomic_memory_related_ranks",
"load_atomic_memory_tags",
Expand Down
Loading
Loading