Structurify

Live Demo: structurify.web.app

Structurify is a production-ready, event-driven B2B SaaS platform that automates the transformation of unstructured, messy spreadsheet data (CSV, XLSX) into a standardized master schema using Google Gemini 2.5 Flash.

Built on a completely decoupled Serverless Fan-Out Architecture on Google Cloud Platform (GCP), Structurify ensures 0% web server blocking, high resilience under burst loads, and highly cost-effective scaling. It utilizes a LangGraph Map-Reduce pipeline to gracefully process files of immense scale by dynamically batching and error-correcting the LLM output.

---

Core Features

• Strict Schema Enforcement: Define exactly the JSON/Excel schema you need, and Structurify will enforce strict type-casting and structure.
• Auto-Clean Mode: Don't know the schema? Structurify will automatically infer the schema from the file headers and repair capitalization, trim whitespace, and standardize date formats across the board.
• Email Notifications: Upload a massive dataset (over 5MB), and Structurify will immediately email you a tracking link to watch the live progress, followed by a final success email with your secure download URL.
• Massive Scalability: The backend acts as a lightweight router while heavy data processing is handled by scalable workers via Cloud Pub/Sub, preventing Gateway Timeouts on long jobs.

---

System Architecture

The architecture utilizes a robust asynchronous data pipeline:

1. Next.js Frontend: The user drops a messy spreadsheet and defines a target JSON schema (or leaves it blank for Auto-Clean).
2. Direct to GCS: The file is securely uploaded directly from the browser to a Google Cloud Storage bucket via a presigned URL generated by the Backend Gateway.
3. API Gateway (FastAPI): Logs the job in Firestore and dispatches a lightweight event to Google Cloud Pub/Sub.
4. Cloud Run Worker (FastAPI/Python): A massively scalable, asynchronous worker consumes the Pub/Sub push notification.
5. LangGraph Map-Reduce Pipeline:
   • Split: The worker chunks the file (e.g. 500 rows at a time).
   • Map (LangGraph): Each chunk is pushed to Gemini 2.5 Flash using strict response_schema parameters. If Gemini hallucinates or encounters an error, a LangGraph state-machine automatically retries the extraction up to 3 times.
   • Reduce: A transaction counter monitors the chunks. When all are complete, a final Reducer service compiles them into a pristine Excel (.xlsx) workbook.
6. Real-time UI: The frontend listens to Firestore via onSnapshot and instantly provides the user with a real-time progress bar and a secure download URL.

---

Tech Stack

• Frontend: Next.js 14, React, TailwindCSS, Firebase Client SDK
• Backend API Gateway: Python, FastAPI, Uvicorn
• Asynchronous Worker: Python, FastAPI, Pandas, LangGraph, Google GenAI SDK (Gemini 2.5 Flash), Tenacity
• Cloud Infrastructure: Google Cloud Platform (Cloud Run, Cloud Storage, Cloud Pub/Sub, Firestore, Artifact Registry, Secret Manager)
• Architecture Standard: Clean Architecture / Domain Driven Design (DDD)
• Testing: pytest, httpx, jest, React Testing Library

---

Google Cloud Platform (GCP) Services Used

Structurify heavily leverages GCP's serverless ecosystem to achieve massive scalability and low operational overhead. Here is how each service is utilized:

1. Cloud Run: Hosts both the FastAPI Gateway (structurify-backend) and the LangGraph Processing Engine (structurify-worker) as serverless containers. The backend is public-facing, while the worker is private and triggered internally.
2. Cloud Storage (GCS): Provides highly durable object storage.
   • raw-uploads bucket stores incoming, messy spreadsheets.
   • processed-outputs bucket stores the final, clean .xlsx and .csv files.
3. Cloud Pub/Sub: The backbone of the asynchronous event-driven architecture. The backend publishes events to the schema-transformation-jobs topic, which then securely pushes the workload to the Cloud Run Worker without holding open synchronous HTTP connections.
4. Firestore (Datastore): The primary NoSQL database. It logs every job's status, tracking progress in real-time. The frontend subscribes to these Firestore documents via onSnapshot to render the live loading timeline.
5. Artifact Registry: Acts as the secure, private container image registry. It stores the Docker images for both the backend and worker before they are deployed to Cloud Run.
6. Secret Manager: Securely stores sensitive credentials like the GEMINI_API_KEY. The Cloud Run worker pulls these directly into environment variables at runtime, ensuring keys are never exposed in plaintext.
7. Firebase Hosting: Serves the Next.js static frontend application globally with low-latency CDN caching.

---

Project Structure

The codebase strictly adheres to Clean Architecture principles across all microservices:

Structurify/
├── backend/                  # API Gateway Service
│   ├── src/
│   │   ├── api/routers       # FastAPI endpoints
│   │   ├── core/             # Environment configs (pydantic-settings)
│   │   ├── models/           # Pydantic request/response schemas
│   │   └── services/         # GCP abstractions (Storage, PubSub, Firestore)
│   └── tests/                # Pytest suites
│
├── worker/                   # Asynchronous Processing Engine
│   ├── src/
│   │   ├── api/              # PubSub Push endpoints (Map / Reduce routes)
│   │   └── services/         # LangGraph Chunk Processor, Reducer, File Parser, Email Service
│   └── tests/                # Pytest suites (Mocks GCP & Gemini)
│
├── frontend/                 # Next.js Application
│   ├── src/
│   │   ├── app/              # Next.js App Router orchestration
│   │   ├── components/       # Atomic UI (SchemaBuilder, Timeline, UploadZone)
│   │   └── hooks/            # Decoupled business logic (useFileUpload, useJobListener)
│   └── __tests__/            # Jest component & hook tests
│
├── deploy.sh                 # Fully automated CI/CD to Cloud Run
└── setup_gcp_infrastructure.sh # Automated IaC for entire GCP environment

---

Local Development Setup

1. Prerequisites

• Node.js >= 18
• Python >= 3.10
• Google Cloud CLI (gcloud) installed and authenticated
• A Firebase Project (for the frontend client)

2. Environment Configuration

Navigate into the respective folders and copy the .env.example to .env (or .env.local for frontend).

Backend (backend/.env)

GOOGLE_CLOUD_PROJECT=your-gcp-project-id
RAW_BUCKET_NAME=raw-uploads-your-gcp-project-id
PUBSUB_TOPIC_ID=schema-transformation-jobs

Worker (worker/.env)

GOOGLE_CLOUD_PROJECT=your-gcp-project-id
RAW_BUCKET_NAME=raw-uploads-your-gcp-project-id
PROCESSED_BUCKET_NAME=processed-outputs-your-gcp-project-id
GEMINI_API_KEY=your-gemini-api-key
FRONTEND_URL=http://localhost:3000
SMTP_SERVER=smtp.gmail.com
SMTP_PORT=587
SMTP_USERNAME=your-email@gmail.com
SMTP_PASSWORD=your-app-password
SMTP_FROM_EMAIL=your-email@gmail.com

Frontend (frontend/.env.local)

NEXT_PUBLIC_FIREBASE_API_KEY=...
NEXT_PUBLIC_FIREBASE_AUTH_DOMAIN=...
NEXT_PUBLIC_FIREBASE_PROJECT_ID=...
# ... other firebase config
NEXT_PUBLIC_BACKEND_URL=http://localhost:8000

3. Running Locally

Start the Backend (Port 8000)

cd backend
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
uvicorn src.main:app --port 8000 --reload

Start the Worker (Port 8080)

cd worker
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
uvicorn src.main:app --port 8080 --reload

Start the Frontend (Port 3000)

cd frontend
npm install
npm run dev

4. Running Tests

• Backend: cd backend && PYTHONPATH=backend pytest backend/tests/
• Worker: cd worker && PYTHONPATH=worker pytest worker/tests/
• Frontend: cd frontend && npm run test

---

Cloud Deployment

Structurify is designed for seamless, automated deployment to Google Cloud Run.

1. Initial Infrastructure Setup

Run the infrastructure shell script once to provision Artifact Registry, Storage Buckets, Firestore, Secret Manager, Pub/Sub, and IAM rules.

chmod +x setup_gcp_infrastructure.sh
./setup_gcp_infrastructure.sh your-gcp-project-id us-central1 your-gemini-api-key

2. Service Deployment

Whenever you make changes to the code, simply run the deployment script to build the Docker images via Cloud Build and deploy them directly to Cloud Run.

chmod +x deploy.sh
./deploy.sh your-gcp-project-id us-central1

(Once deployed, update your frontend/.env.local's NEXT_PUBLIC_BACKEND_URL to point to the live Backend Cloud Run URL).