Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions specs/catalog/omni-graph-live-watcher-fidelity-parity.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,14 @@ A recursive divide-and-conquer code and database audit of `tools/omni-graph/src/
- **AC-058**: Method invocations with receiver targets (e.g. `FilterEngine.calculate(x)`) preserve root receiver objects as `REFERENCES` edges.
- **AC-059**: MATLAB `import` statements (e.g. `import control.guidance.NavSystem;` or `import math.filters.*;`) emit `import` nodes and `IMPORTS` edges.
- **AC-060**: Package directory hierarchies prefixed with `+` (e.g. `+control/+guidance/NavSystem.m`) synthesize `package` nodes (`control.guidance`) with `DECLARES` edges to contained classes and functions.
- **AC-061**: An `.slx` Open Packaging Conventions ZIP archive is decompressed in-memory to stream `simulink/blockdiagram.xml` with zero temporary disk writes and bounded memory consumption (< 5MB).
- **AC-062**: Top-level model declaration emits a `model` node, and nested subsystems emit `subsystem` nodes with hierarchical `CONTAINS_BLOCK` edges.
- **AC-063**: Individual computational and I/O blocks (`Inport`, `Outport`, `Gain`, `Sum`, `Integrator`, `MATLABFunction`) emit `block` nodes indexed by unique Simulink Identifier (`SID`).
- **AC-064**: Direct and branched signal connection lines (`<Line Src="SID#out:port" Dst="SID#in:port">` and `<Branch>`) emit directed `SIGNAL_TO` edges connecting output ports to input ports.
- **AC-065**: Model reference blocks (`<Block BlockType="ModelReference">`) emit `model_reference` nodes and cross-file `REFERENCES_MODEL` edges targeting the referenced `.slx` / `.mdl` model.
- **AC-066**: High-fanout broadcast signal lines (deg(v) > 50, such as power rails or global CAN buses) are classified into `DATAFLOW_BROADCAST` to preserve modularity cluster boundaries.
- **AC-067**: Legacy plain-text Simulink `.mdl` files are parsed with identical topological node and edge parity to `.slx` archives.
- **AC-068**: Both full directory ingestion (`execute_ingest`) and the incremental live watcher (`reindex_file`) support `.slx` and `.mdl` files with transactional atomic swap fidelity.

### 2.3 Non-Functional Requirements (NF)

Expand Down
3 changes: 2 additions & 1 deletion tools/omni-graph/CONTEXT.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,8 @@ blockers: []
| High-Throughput Parallel Batch Persistence (R-100) | Complete | — | 🟢 tokio::join! on backbone storage and community inference; omni-mcp.py handles HTTP 423 gracefully |
| Filesystem Symlink Cycle Guard & Navigation (R-109) | Complete | — | 🟢 Symlink metadata interception, canonical target resolution, LINKS_TO edge emission, zero duplicate AST clones |
| MathWorks MATLAB AST Ingestion (R-107) | Complete | — | 🟢 Vendored ABI 14 tree-sitter-matlab; extracted classdef, multiple inheritance EXTENDS, properties, constructors, methods, receiver preservation, import commands, package hierarchy |
| Test Coverage | 118/118 Unit Tests | 118 | 🟢 100% Verified (0.05s, 0 clippy warnings) |
| MathWorks Simulink MBSE Block Diagram Ingestion (R-108) | Complete | — | 🟢 Open Packaging Conventions (OPC) ZIP extraction of simulink/blockdiagram.xml, streaming XML & legacy MDL parsing, block containment, model references (REFERENCES_MODEL), dataflow signal wires (SIGNAL_TO), and high-degree broadcast attenuation (DATAFLOW_BROADCAST) |
| Test Coverage | 122/122 Unit Tests | 122 | 🟢 100% Verified (0.05s, 0 clippy warnings) |
| Live Watcher In-Memory SAIR Acceleration (R-078, R-091) | Complete | — | 🟢 Integrated in-memory SymbolTable into IncrementalPipeline; eliminated 392s subquery bottleneck; live sync completes in <250ms with dedicated 1s quiescent trigger |
| AI-Native Cosmology & Modularity Guard (R-086, R-087) | Complete | — | 🟢 Purged Miller's Law clamp; enforced Affinity-Floor Invariant and delta Q guard |
| Scale-Adaptive Subsystem Sizing (R-088) | Complete | — | 🟢 Sub-linear dynamic ceiling K_adaptive(N) bounding natural domains without forced merges |
Expand Down
2 changes: 2 additions & 0 deletions tools/omni-graph/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,8 @@ hex = "0.4"
chrono = { version = "0.4", features = ["serde"] }
notify = "6.1"
tokio-stream = { version = "0.1", features = ["sync"] }
zip = { version = "9.4.1", default-features = false, features = ["deflate"] }
quick-xml = "0.42.0"

[dev-dependencies]
tokio-test = "0.4"
32 changes: 32 additions & 0 deletions tools/omni-graph/DEVLOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,35 @@
### 2026-11-06 — MathWorks Simulink (`.slx` / `.mdl`) MBSE Block Diagram Ingestion Engine (R-108)

**Agent/Author**: Antigravity (Google DeepMind) & `@vadde`
**SDLC Phase**: `review` (MBSE Graph Integration & Verification)
**Branch**: `feat/r108-simulink-mbse-ingestion`
**Requirement**: `R-108` ([`specs/catalog/omni-graph-live-watcher-fidelity-parity.md`](file:///Users/aparv/Library/CloudStorage/OneDrive-Personal/G-Drive/Interviews/knowledge/PolyTuring/specs/catalog/omni-graph-live-watcher-fidelity-parity.md))
**Duration**: Complete

#### Executive Summary & Key Milestones Completed
1. **Model-Based Systems Engineering (MBSE) Ingestion Architecture (`src/parser/simulink.rs`)**:
- Implemented `parse_slx` Open Packaging Conventions (OPC) ZIP decompression: extracted `simulink/blockdiagram.xml` in-memory with bounded buffer memory footprint (< 5MB) and zero temporary disk writes.
- Implemented streaming XML parser using `quick-xml 0.42` handling both `Event::Start` and `Event::Empty` tags for `<Block BlockType="..." Name="..." SID="...">`, `<System>`, `<Line>`, `<Branch>`, and `<P Name="...">`.
- Implemented legacy plain-text Simulink `.mdl` tokenizer and parser with identical topological parity for pre-2012 models (`Model { System { Block { ... } Line { ... } } }`).
2. **Topological Block, Signal Wire, and Cross-Model Linkage**:
- Hierarchical Containment: Emitted root `model` node, `subsystem` nodes, and `CONTAINS_BLOCK` edges tracking the system hierarchy stack.
- SIDs and Ports: Indexed blocks (`Inport`, `Outport`, `Gain`, `DiscreteFilter`, `SubSystem`, `ModelReference`) by their unique Simulink Identifier (`SID`).
- Signal Dataflow: Emitted directed `SIGNAL_TO` edges connecting output ports (`SID#out:port`) to input ports (`SID#in:port`), supporting branched fan-out topologies.
- Cross-Model Architecture: Emitted `REFERENCES_MODEL` edges from `ModelReference` blocks targeting external `.slx` and `.mdl` models across the workspace.
- High-Degree Broadcast Attenuation: High-fanout broadcast lines (deg(v) > 50, e.g. 28V power buses, CAN telemetry buses) are automatically classified into `DATAFLOW_BROADCAST` to preserve community modularity clustering.
3. **Pipeline & Live-Watcher Integration**:
- Symmetrically wired into batch ingestion (`execute_ingest` in `src/ingestion/mod.rs`) and incremental watcher (`reindex_file` in `src/watcher/pipeline.rs`).
- Binary `.slx` files are intercepted before null-byte binary filtering, computing SHA-256 hashes, updating the in-memory `SymbolTable`, and executing transactional atomic swaps in SurrealDB.
4. **Verification & Testing (`tests/unit_tests.rs`)**:
- Added 4 dedicated MBSE unit tests:
- `test_r108_simulink_slx_opc_zip_extraction`: In-memory ZIP archive generation and blockdiagram.xml extraction verifying `Inport`, `Gain`, `SubSystem`, and `SIGNAL_TO` dataflow edges.
- `test_r108_simulink_model_reference_and_fanout`: Model reference cross-model links (`REFERENCES_MODEL`) and branched fan-out signal lines.
- `test_r108_simulink_legacy_mdl_parity`: Full parsing of legacy plain-text `.mdl` files verifying identical topological structure.
- `test_r108_simulink_broadcast_bus_degree_attenuation`: High-degree bus fanout (56 receivers > 50) verified to be tagged as `DATAFLOW_BROADCAST`.
- All 122 unit tests pass (100% green, 0 clippy warnings).

---

### 2026-11-06 — MathWorks MATLAB (`.m`) AST Ingestion Engine (R-107)

**Agent/Author**: Antigravity (Google DeepMind) & `@vadde`
Expand Down
7 changes: 4 additions & 3 deletions tools/omni-graph/STATUS.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,15 +54,16 @@ fidelity_spec: ../../specs/catalog/omni-graph-live-watcher-fidelity-parity.md
| 2026-11-06 | `review` | `in-progress` | Launch v0.7.0 Live Watcher Fidelity Parity & Concurrency Overhaul (ADR-002, SPEC-OMNI-005, R-093 to R-109): Closing 14 forensic loopholes across transactional atomic batch swap, reactive reverse dependencies, unresolved retention pool, deduplication, and non-blocking Louvain actor. |
| 2026-11-06 | `in-progress` | `testing` | Completed Milestones 1 through 6 (R-093 to R-106): SymbolTable hardening, directory backbone synthesis, transactional atomic swap, edge deduplication, dual-tier modularity, non-blocking Louvain actor, convergence barrier (wait_for_sync), nanometer fidelity diagnostics, and lock-free generational versioning. 111 unit tests pass (100% green, 0 clippy warnings). |
| 2026-11-06 | `testing` | `testing` | Completed R-109 (Filesystem Symlink Cycle Guarding & Alias Navigation) and R-107 (MathWorks MATLAB AST Ingestion): Vendored ABI 14 tree-sitter-matlab grammar, extracted classdef, multiple inheritance EXTENDS, properties, constructors, methods, receiver preservation, import commands, and package hierarchy. 118 unit tests pass (100% green, 0 clippy warnings). |
| 2026-11-06 | `testing` | `review` | Completed R-108 (MathWorks Simulink .slx / .mdl MBSE Ingestion): In-memory OPC ZIP extraction of simulink/blockdiagram.xml, streaming XML & legacy .mdl parser, block containment, cross-model references (REFERENCES_MODEL), dataflow signal wiring (SIGNAL_TO), and high-degree broadcast bus attenuation (DATAFLOW_BROADCAST). 122 unit tests pass (100% green, 0 clippy warnings). |

---

## Implementation Progress

| Category | Done | Total | Percentage |
|----------|------|-------|------------|
| Requirements (R-XXX) | 80 | 81 | 98.8% |
| Acceptance Criteria (AC-XXX) | 44 | 45 | 97.8% |
| Requirements (R-XXX) | 81 | 81 | 100% |
| Acceptance Criteria (AC-XXX) | 45 | 45 | 100% |
| Non-Functional (NF-XXX) | 15 | 15 | 100% |

---
Expand Down Expand Up @@ -176,7 +177,7 @@ fidelity_spec: ../../specs/catalog/omni-graph-live-watcher-fidelity-parity.md
| R-105 | Zero-Delay Community Inference for New Files | `src/analysis/mod.rs`, `src/watcher/pipeline.rs` | `tests/unit_tests.rs:test_r105_zero_delay_community_inference_for_new_files` | ✅ |
| R-106 | Language-Aware SAIR Scoping & Exact Match Fast-Path | `src/ingestion/symbol_table.rs`, `src/db/mod.rs` | `tests/unit_tests.rs:test_r106_language_aware_sair_scoping` | ✅ |
| R-107 | MathWorks MATLAB (`.m`) AST & Classdef Ingestion | `src/parser/mod.rs`, `grammars/tree-sitter-matlab` | `tests/unit_tests.rs:test_r107_matlab_*` | ✅ |
| R-108 | MathWorks Simulink (`.slx` / `.mdl`) Block Diagram MBSE Ingestion | `src/parser/simulink.rs`, `src/ingestion/mod.rs` | Spec/Research spike complete, queued for Phase 3 | ⬜ |
| R-108 | MathWorks Simulink (`.slx` / `.mdl`) Block Diagram MBSE Ingestion | `src/parser/simulink.rs`, `src/ingestion/mod.rs`, `src/watcher/pipeline.rs` | `tests/unit_tests.rs:test_r108_*` | ✅ |
| R-109 | Filesystem Symlink Cycle Guarding & Alias Navigation | `src/ingestion/mod.rs`, `src/watcher/pipeline.rs` | `tests/unit_tests.rs:test_r109_symlink_cycle_guard_and_alias_navigation` | ✅ |

**Status Legend**: ⬜ Not started · 🔨 In progress · ✅ Implemented · 🧪 Tested · ❌ Blocked
Expand Down
122 changes: 122 additions & 0 deletions tools/omni-graph/src/ingestion/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -736,6 +736,128 @@ async fn execute_ingest(
Ok(b) => b,
Err(_) => continue,
};

// Intercept MathWorks Simulink .slx OPC ZIP archives before null-byte check (R-108)
if ext == "slx" {
let rel_path = path
.strip_prefix(root)
.unwrap_or(path)
.to_string_lossy()
.to_string();

let mut hasher = Sha256::new();
hasher.update(&raw_bytes);
let hash = hex::encode(hasher.finalize());

if !cache.is_stale(&rel_path, &hash).await {
files_skipped += 1;
continue;
}

files_scanned += 1;

if let Some(parse_res) = CodeParser::parse_simulink_slx(workspace_name, &rel_path, &raw_bytes) {
let filename = Path::new(&rel_path)
.file_name()
.map(|n| n.to_string_lossy().to_string())
.unwrap_or_else(|| rel_path.clone());
let file_node_id = format!("{}:file:{}", workspace_name, rel_path);
let file_node = ExtractedNode {
id: file_node_id.clone(),
workspace: workspace_name.to_string(),
label: filename.clone(),
kind: "file".to_string(),
file_path: rel_path.clone(),
language: "simulink".to_string(),
line_start: 1,
line_end: 1,
text: format!("Simulink Model Archive: {}", filename),
};

let parent_dir = Path::new(&rel_path)
.parent()
.map(|p| p.to_string_lossy().to_string())
.unwrap_or_default();
let parent_id = if parent_dir.is_empty() || parent_dir == "." {
format!("{}:workspace", workspace_name)
} else {
format!("{}:dir:{}", workspace_name, parent_dir)
};

let cont_edge = ExtractedEdge {
workspace: workspace_name.to_string(),
source_id: parent_id,
target_label: filename,
target_id: Some(file_node_id.clone()),
edge_type: "CONTAINS".to_string(),
category: "CONTAINMENT".to_string(),
};

let mut file_containment_edges = vec![cont_edge];
for node in &parse_res.nodes {
if node.kind == "model" {
file_containment_edges.push(ExtractedEdge {
workspace: workspace_name.to_string(),
source_id: file_node_id.clone(),
target_label: node.label.clone(),
target_id: Some(node.id.clone()),
edge_type: "CONTAINS".to_string(),
category: "CONTAINMENT".to_string(),
});
}
}

// Collect directory hierarchy for Topological Containment Backbone (R-063)
let mut cur_dir = Path::new(&rel_path).parent();
while let Some(parent) = cur_dir {
let p_str = parent.to_string_lossy().to_string();
if p_str.is_empty() || p_str == "." {
break;
}
discovered_dirs.insert(p_str);
cur_dir = parent.parent();
}

let mut all_nodes = Vec::with_capacity(parse_res.nodes.len() + 1);
all_nodes.push(file_node);
all_nodes.extend(parse_res.nodes);

let mut all_edges = Vec::with_capacity(parse_res.edges.len() + file_containment_edges.len());
all_edges.extend(file_containment_edges);
all_edges.extend(parse_res.edges);

// Register nodes in in-memory symbol table (R-078 Pass 1)
symbol_table.register_file_nodes(&all_nodes);

let node_texts: Vec<String> = all_nodes.iter().map(|n| n.text.clone()).collect();
pending_texts.extend(node_texts);
pending_files.push(PendingBatchFile {
rel_path: rel_path.clone(),
hash: hash.clone(),
nodes: all_nodes,
containment_edges: all_edges,
text_count: pending_texts.len(),
});

if pending_files.len() >= 64 {
flush_vector_batch(
workspace_name,
&mut pending_files,
&mut pending_texts,
embedder,
db,
cache,
&mut files_indexed,
&mut total_nodes,
&mut total_edges,
)
.await;
}

continue;
}
}

let peek_len = raw_bytes.len().min(512);
if raw_bytes[..peek_len].contains(&0) {
// Binary file containing null bytes (ELF, Mach-O, raw data), skip cleanly
Expand Down
22 changes: 22 additions & 0 deletions tools/omni-graph/src/parser/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,9 @@
// Implements: R-003 (Deterministic AST parsing)
// Extracts functions, structs, classes, imports, and calls

pub mod simulink;
pub use simulink::SimulinkParser;

use serde::{Deserialize, Serialize};
use std::collections::HashSet;
use std::path::Path;
Expand Down Expand Up @@ -67,6 +70,9 @@ impl CodeParser {
if extension == "toml" {
return Self::parse_toml(workspace, file_path, content);
}
if extension == "mdl" {
return Self::parse_simulink_mdl(workspace, file_path, content);
}
let (language_name, mut parser) = match extension {
"rs" => {
let mut p = Parser::new();
Expand Down Expand Up @@ -173,6 +179,22 @@ impl CodeParser {
Some(ParseResult { nodes, edges })
}

/// Ingests a Simulink .slx archive in-memory (R-108)
pub fn parse_simulink_slx(workspace: &str, file_path: &str, raw_bytes: &[u8]) -> Option<ParseResult> {
SimulinkParser::parse_slx(workspace, file_path, raw_bytes)
}

/// Ingests a Simulink blockdiagram.xml payload (R-108)
#[allow(dead_code)]
pub fn parse_simulink_xml(workspace: &str, file_path: &str, xml_content: &str) -> Option<ParseResult> {
SimulinkParser::parse_blockdiagram_xml(workspace, file_path, xml_content)
}

/// Ingests a legacy plain-text Simulink .mdl model (R-108)
pub fn parse_simulink_mdl(workspace: &str, file_path: &str, content: &str) -> Option<ParseResult> {
SimulinkParser::parse_mdl(workspace, file_path, content)
}

/// Extracts function identifier from C/C++ function_definition declarator chain
fn extract_c_function_name<'a>(node: Node<'a>, content: &'a str) -> Option<&'a str> {
let mut curr = node.child_by_field_name("declarator")?;
Expand Down
Loading
Loading