Graph-Based Data Modeling and Query System

This system ingests SAP Order-to-Cash dataset, constructs a graph of interconnected entities, visualizes it, and allows natural language queries.

Architecture Overview

The system follows a modular architecture with clear separation of concerns:

• Data Layer: JSONL file ingestion and SQLite database for efficient querying
• Graph Layer: NetworkX-based graph construction with entity relationships
• Visualization Layer: PyVis interactive graph rendering
• Query Layer: LLM-powered natural language to SQL translation
• UI Layer: Streamlit web interface with tabbed navigation

Key Components

• app.py: Main Streamlit application
• sap-order-to-cash-dataset/: Raw SAP data files
• sap_data.db: SQLite database for queries
• graph.html: Generated graph visualization

Database Choice: SQLite

Why SQLite?

• Lightweight: No server required, embedded database
• Read-optimized: Perfect for analytical queries on static datasets
• Python-native: Excellent pandas integration via to_sql() and read_sql_query()
• ACID compliance: Reliable transactions for data integrity
• File-based: Easy deployment and backup

Data Processing:

• JSONL files loaded into pandas DataFrames
• Dict columns converted to JSON strings for SQLite compatibility
• Tables created with df.to_sql(if_exists='replace')
• Foreign key relationships preserved through JOIN queries

LLM Prompting Strategy

The system uses a hybrid approach combining LLM power with rule-based fallbacks:

Primary Strategy: Schema-Aware Prompting

You are a SQL expert. Given the following database schema:
{table1}: {columns}
{table2}: {columns}
...

Translate the following natural language question into a SQL query. The database is SQLite.

Question: {user_question}

Return only the SQL query, no explanation.

Benefits:

• Provides full context of available tables and columns
• Ensures generated SQL is syntactically correct
• Leverages LLM's understanding of SQL patterns

Fallback Strategy: Rule-Based Templates

For common query patterns when API is unavailable:

• Aggregation queries: "highest number of X" → SELECT col, COUNT(*) FROM table GROUP BY col ORDER BY count DESC
• Flow tracing: Complex JOINs for order-to-cash pipeline
• List queries: "list all customers" → SELECT * FROM business_partners
• Integrity checks: UNION queries for broken flows

Why Hybrid?

• Reliability: Works without internet/API dependency
• Cost-effective: Reduces API calls for predictable queries
• Fast: Rule-based responses are instant
• Comprehensive: Covers most user needs

Guardrails Implementation

The system implements multiple layers of guardrails to ensure safe, relevant responses:

1. Keyword Filtering

dataset_keywords = ['order', 'delivery', 'billing', 'payment', 'customer', 'product', 'invoice', 'journal', 'sales', 'material', 'plant', 'partner']
if not any(kw in question.lower() for kw in dataset_keywords):
    return "This system is designed to answer questions related to the provided dataset only."

Purpose: Restricts queries to SAP Order-to-Cash domain only.

2. Schema Confinement

• LLM prompted with exact table/column names
• No access to external data or general knowledge
• SQL generation limited to provided schema

3. Query Validation

• Generated SQL executed against known database
• Error handling for malformed queries
• Fallback to safe default queries

4. Response Grounding

• All answers based on actual dataset queries
• No hallucinated information
• Transparent SQL display for user verification

Setup

1. Install dependencies: pip install pandas networkx pyvis streamlit openai

2. Set your OpenAI API key in app.py (replace "your-api-key-here")

3. Run: python -m streamlit run app.py

Features

• Graph Construction: Nodes for entities (Sales Orders, Deliveries, Billings, etc.), edges for relationships.

• Visualization: Interactive graph using PyVis.

• Query Interface: Natural language to SQL translation using OpenAI GPT, with fallbacks.

• Guardrails: Restricts to dataset-related queries.

Example Queries

• "Which products are associated with the highest number of billing documents?"

• "Trace the full flow of billing document 90504248"

• "Identify sales orders that have broken or incomplete flows"