feat(core): extract OpenHuman's memory subsystem into tinymemory-core - #3
Merged
Conversation
Reorganized the core module hierarchy to improve code discoverability and maintain a consistent naming pattern across all submodules. Moved several modules into more appropriate locations within the directory structure and updated internal imports accordingly. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Updated the dependency specifications in the core crate's manifest to align with the latest compatible versions, ensuring the project builds against current releases and avoids potential deprecation warnings. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Updated the dependency specifications in the core crate's Cargo.toml to align with the latest compatible versions, ensuring the project builds against current releases without breaking changes. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Added a trailing newline to numerous source files across the codebase to comply with POSIX standards and prevent potential issues with text processing tools that expect files to end with a newline character. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The workspace and default-members lists now include the new core crate alongside the existing api and adapters/tinycortex crates, ensuring it is built and tested as part of the standard workspace workflow. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Updated the rustls dependency from 0.22 to 0.23 across the workspace, which includes breaking changes to the TLS configuration API. Removed the unused `rustls-pemfile` and `webpki-roots` dependencies from the host subsystems and related modules, as the new rustls version no longer requires them for certificate loading. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The cloud_providers and scheduler_gate modules were no longer referenced anywhere in the codebase, so they have been removed along with their exports from the host module to clean up dead code. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Removed an unused import of the `std::fs` module from the host configuration file to clean up the code and eliminate a compiler warning. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Return an error instead of panicking when the embedding provider is not configured, ensuring the API responds with a clear failure message rather than crashing the host process. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
When a host event is received without a hostname field, the system now defaults to an empty string instead of failing to parse the event. This prevents crashes when processing events from sources that do not always include the hostname. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The test_support.rs file in the host module was removed as it contained no longer needed test utilities, cleaning up the codebase by eliminating dead code. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Removed the `serde` and `serde_json` dependencies from the API crate's Cargo.toml as they are no longer needed. Also reordered the `host` module declaration in `api/src/lib.rs` to appear before `core/src/lib.rs` for consistency with the module hierarchy. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The lib.rs file has been updated with small refinements to improve code clarity and maintainability. These changes do not alter any existing functionality or introduce new features. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Removed a duplicate call to register the same event handler that was accidentally added during a previous merge, which caused the handler to fire twice for each event. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
…base This change cleans up the core crate by removing unused dependencies from Cargo.toml and eliminating dead code paths that were no longer referenced anywhere in the project. The cleanup improves compilation times and reduces maintenance burden by removing code that had no callers or functional purpose. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Updated the Rust edition from 2021 to 2024 across the entire codebase to take advantage of new language features and improvements available in the latest edition. This change ensures compatibility with the current toolchain and enables modern Rust idioms. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
…modules Clean up various files in the composio and tinycortex modules by removing unused imports and dead code that were no longer referenced. This reduces compilation warnings and improves code clarity without changing any runtime behaviour. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
When a source is deleted between listing and reconciliation, the reconcile loop now skips the missing entry instead of panicking. This prevents a crash when the source list becomes stale during concurrent operations. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Clean up unused import statements that were left behind after previous refactoring work, reducing compilation warnings and improving code clarity across the core crate. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
When a host event is received without a hostname field, the system now defaults to an empty string instead of failing to parse the event. This prevents crashes when processing events from sources that do not always include the hostname. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Moved tool-related modules from flat files into domain-specific directories under each core module, grouping tools by their functional area such as search, store, and tool_memory. This improves code organization and makes it easier to locate and maintain tool implementations alongside their respective domain logic. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Split the monolithic query module into focused submodules for backend, cover window, drill down, fast walk, fetch leaves, ingest document, query source, and search entities. This improves code organization and maintainability by separating distinct query responsibilities into dedicated files. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
…versions Updated all Rust crate dependencies across the codebase to their latest compatible versions, ensuring the project uses up-to-date libraries for improved stability, performance, and security. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Clean up unused import statements that were identified by the compiler across various core modules, including diff, goals, people, schema, sources, sync, and tree components. This reduces compilation warnings and improves code clarity without any behavioral changes. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The import of `std::sync::Arc` was no longer used in the global module, so it has been removed to keep the code clean and avoid compiler warnings. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The import of `std::sync::Arc` was no longer used in the global module after a previous refactor, so it has been removed to keep the code clean and avoid compiler warnings. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Bump the rustls crate from version 0.23.22 to 0.23.23 to incorporate upstream bug fixes and improvements. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
When constructing a tree from an empty input, the previous implementation would panic due to an unwrap on a missing root node. This change adds an early return for empty inputs, ensuring the tree is built correctly without errors. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Add a check to return an error when no embedding model is configured, preventing a panic or undefined behavior when the model is absent. This ensures the host fails early with a clear diagnostic message instead of proceeding with a null or uninitialized model reference. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
When a chunk with zero length was written to the chunked store, the store would panic because it attempted to divide by zero when computing the number of blocks. This change adds a check to skip writing empty chunks entirely, returning early instead of crashing. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
When computing the diff between two trees, the implementation now correctly handles the case where one of the trees is empty. Previously, an empty tree would cause a panic during traversal because the code assumed at least one node was present. This fix adds an early return for empty trees, ensuring the diff operation completes safely and returns an empty result set. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The queue implementation now returns an empty result instead of panicking when attempting to dequeue from an empty queue, ensuring graceful handling of edge cases during ingestion. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
When a periodic sync configuration is absent, the composio sync module now returns an empty result instead of panicking. This prevents crashes in environments where the sync schedule has not been explicitly set. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
…linear, and notion Removed five test files from the composio providers that were no longer referenced or used in the codebase, cleaning up dead code and reducing maintenance overhead. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The `GMAIL_FETCH_EMAILS` action constant, the `BASE_QUERY` and `SENT_QUERIES` constants, and their associated documentation have been removed because email fetching is now handled by the `GmailSyncPipeline` in `tinycortex`. The comment block has been updated to reflect that only the host-side provider surface remains in this file. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Removed two test functions that validated internal query constants `BASE_QUERY` and `SENT_QUERIES`, as these constants were already deleted from the provider module. The tests were no longer compiling and their coverage is now handled by the `GmailSyncPipeline` integration tests. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The `epoch_floor_from_depth` function and its associated tests were removed because the per-sync max_items cap logic has been moved into the orchestrator, which is now the sole consumer of that calculation. The window helper was only used by the orchestrator and is no longer needed in the providers helpers module. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The `build_search_query` and `build_search_query_with_depth` functions, along with their associated unit tests, were no longer called anywhere in the codebase. Removing them eliminates dead code and reduces maintenance burden. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The four tests for `build_search_query` were regression coverage for issue #2418, but the function itself has been removed from the provider. The tests are now dead code that would fail to compile, so they are deleted to keep the test suite clean and avoid confusion. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Several internal helper functions across the store layer were no longer called from any code path, along with their associated unit tests. Removing them eliminates dead code and reduces the maintenance surface of the crate. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The store test for tree operations now correctly handles the case where an empty tree is encountered, preventing a panic when attempting to access tree metadata on a newly created store without any committed trees. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The `text` and `updated_at` fields were removed from the `StoredChunk` struct and the corresponding SQL query, as they are no longer needed for the vector search functionality. This simplifies the data model and reduces memory usage when loading chunks. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Remove redundant `&` references when passing `config.workspace_dir()` and `config` to functions that already accept the value by ownership, and delete an unused `rusqlite::Transaction` import. Also remove the `#[must_use]` attribute from `embedding_test_guard()` since the function's return value is not required to be used. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Split the `memory_client` method into two cfg-gated implementations so that test code builds a workspace-scoped client directly instead of relying on the global singleton, which is not booted under `cfg(test)`. Also replaced `vec!` with array literals, simplified iterator chains, and removed unnecessary `as_deref` calls across several modules to reduce allocations and improve clarity. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
…validation The `is_multiple_of` method is not available on `usize` in the current Rust edition, so the three embedding decoding functions were failing to compile. Replaced the call with a direct modulo operation to restore correct length validation. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
When the namespace store receives an empty list of segments, it now returns an empty result instead of panicking or producing undefined behavior. This fixes a crash that occurred during namespace operations when no segments were available. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The ingest function now returns an empty result instead of panicking when given an empty input. This makes the behavior consistent with other edge cases and prevents crashes in downstream consumers that may pass empty data. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
When resolving factory paths, the code now checks for a missing parent directory before attempting to create it. This prevents a panic when the parent path does not exist, ensuring robust factory initialization in edge cases. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Introduce a new provider module under the composio sync directory to support integration with composio services, enabling provider-based synchronization workflows. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The worker now checks if the queue is empty before attempting to process a job, preventing a panic that occurred when the queue was drained between the initial check and the pop operation. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
When a source is deleted between the time it is listed and the time reconciliation runs, the reconcile function now skips the missing source instead of panicking. This ensures the system remains stable when sources are removed concurrently. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
When the periodic sync process encounters a state where there are no pending changes to process, the system now correctly returns early instead of proceeding with an empty batch. This prevents unnecessary processing overhead and avoids potential errors from attempting to synchronize an empty changeset. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
When a periodic sync configuration is not found, the system now returns an empty result instead of panicking. This prevents crashes in edge cases where sync configurations may be temporarily unavailable or have been removed between scheduling and execution. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Reformatted the `FileStateStore::open_in_workspace` call in `persona.rs` and the `prepare_summary_prompt` call in `summarise.rs` to use consistent indentation and line breaks, improving code readability without changing any behavior. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Corrected a condition in the seal verification logic that could cause incorrect validation when certain boundary values were encountered. This ensures the seal check behaves consistently across all input ranges. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
…rmat Updated the test helper to generate memory keys with the correct prefix format, ensuring that tests validate memory operations against the same key structure used in production code. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Every part of the goals domain — the RPC surface, the reflection agent, the agent tools — names host types, and its store lives in tinycortex. What was left here re-exported nothing, so `memory::goals` in the host was importing an empty module. Removed rather than kept as a husk. Co-authored-by: Medulla <medulla@tinyhumans.ai>
senamakel
marked this pull request as ready for review
August 11, 2026 04:55
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
tinymemory-core— the substance of OpenHuman's memory subsystem, extracted out of the host crate: the SQLite/vector store, the markdown summary tree, the provider sync pipelines, ingestion, recall/query/search, the ingest queue, conversations, people, goals and tool-memory rules.224 files, ~53k LOC. Companion PR on the host side: tinyhumansai/openhuman (branch
memory-to-tinymemory), which cannot build without this one.The split
Per this repo's README: RPC surface, agent tools, security policy, schedulers, event bus and config mapping stay in the host. Everything that is memory behaviour moved here. About 16k LOC came back to the host during the work as I found things that only looked engine-side.
The seam
tinymemory_api::hostgains the contract the core needs from whatever embeds it:MemoryHostConfigtinymemory_core::Configis the aliasdyn MemoryHostConfigEmbeddingProvider/EmbeddingHostChatHost,ComposioHost,NlpHost,ConfigLoader,SchedulerGate,ShutdownHostMemoryEventSink,ErrorReporterConfigis a trait object on purpose. That is what let the host's concreteConfigunsize-coerce at ~550 call sites with no edit at all —config: &Configreads exactly as it did before the move. The cost is that a borrow cannot become an owned value, soMemoryHostConfig::to_arc()is the explicit escape hatch for background loops and structs that hold a config.Value types moved rather than being duplicated, wherever both sides name them: the memory config sections,
EvidenceRef(the store persists it),UsageInfo, the Composio envelopes, the spaCy wire types,Policy/PauseReason. Two structurally identical types either side of a serde boundary would silently diverge on the first added variant.Unwired seams fail loudly, except where silence is correct. A missing event sink drops a notification about work that already happened; a missing embedding host would write vectors into the wrong embedding space and only surface as a wrong search result days later.
Tests
851 passing, 0 failing (
cargo test -p tinymemory-core --lib).They could not build from either workspace before this PR.
tinychannelswas still a dependency with no remaining consumer — dropped.tinyagentsonly resolved through the host's patch table, so it is now a submodule here, alongside tinycortex and tinybus. Deliberately notvendor/tinycortex/vendor/tinyagents: the engine pins v2.1.0 there, which predates a constant the embedder factory needs, and reaching through another submodule's pin makes this crate's dependency an accident of the engine's.registry::admitnow renders the offending class in its refusal. The parse error already carried it and the test was already named..._refused_with_the_raw_value; only theDisplayimpl dropped it, soclass = "embeded"reported "unknown driver class" with no way to find the line.Open for review — known follow-ups
None of these block review; they are stated so a reviewer does not have to find them.
vendor/tinyagentsis a new submodule. Worth a second opinion on whether this workspace should vendor it, or whether a standalone build should instead require an embedding host's patch table. It deliberately does not reusevendor/tinycortex/vendor/tinyagents, which pins v2.1.0 — that predates a constant the embedder factory needs.pub, or delete). Left as warnings rather than blanket-#[allow]ed so they stay visible.