Master Plan Intelligence reads a real-estate project's master plan image and turns it into structured, auditable intelligence: a spatial map of every drawn entity (towers and wings, amenities, legend markers, roads, water, open space) reconciled against the printed legend; executive dashboards for land use, amenity mix, spatial access and architectural block plans; key-value site metrics with density benchmarks when project records are supplied; and range-based Green Area % and Open Area % estimates with a confidence score, reasons and hard guardrails. A second layer compares three projects side by side on buyer-facing categories.
Every analysis is checked before it is returned by three backend QA agents (reviewer, fixer, teacher) that keep a self-tuning playbook, and an offline engineer agent turns what they could not fix into pull requests with tests and a fixture-plan regression gate. The output contract is typed and identical across plans.
The engine is deliberately honest: it returns ranges, never fake precision; it never uses "official" language unless the value is in RERA, brochure or structured project data; and every output carries the reasons behind its confidence.
- Why this exists
- What it does
- Locked definitions
- Guardrails
- Confidence scoring
- Three-project master plan comparison
- Master plan entity map and hover viewer
- Backend QA agents (reviewer, fixer, teacher, and the engineer that opens pull requests)
- Architecture
- Repository structure
- Getting started
- API reference
- CLI
- Output example
- Environment variables
- Testing
- Status and roadmap
- Disclaimer
- License
Project comparison modules on real-estate portals rarely have structured values for green cover and open space. Developers do not disclose them consistently, and the one artefact that shows the layout, the master plan, is almost always just an image.
Buyers still want a quick, understandable signal about openness, density and family-friendly livability. This engine produces that signal from what is actually available, and tells you how much to trust it.
The pipeline runs in stages. Each stage writes a typed JSON artefact so every result is auditable.
| Stage | Module | What happens |
|---|---|---|
| 1. Parse | app/services/json_parser.py |
Normalises Datacenter JSON into project metadata: area, units, amenities, RERA records, wow factors, unit mix, master plan description, and any direct green/open claims found in text ("70% open space", "3 acres central green"). |
| 2. Preprocess | app/services/image_preprocessor.py |
Validates the master plan image, writes a normalised PNG, and scores image quality from sharpness, brightness and colourfulness. |
| 3. Vision reasoning | app/services/vision_analyzer.py |
Sends the normalised image plus project context to Gemini 3.5 Flash and gets back structured JSON: boundary confidence, visible green / open / built zones, a qualitative estimate, and warnings such as "stylised, not to scale". |
| 4. Segmentation | app/services/segmentation_engine.py |
OpenCV HSV masks for green landscape, water / pool, built structures and paved hardscape; class-wise pixel percentages, an unknown / unclassified share, and overlay images. |
| 5. Estimate | app/services/area_calculator.py |
Combines segmentation lower bounds, the vision estimate, JSON signals and image quality into range-based Green and Open Area estimates. Applies guardrails and confidence caps. |
| 6. Explain | validation_engine.py, confidence_engine.py, explanation_engine.py |
Validates ranges, computes the confidence score and label, and produces human-readable reasons and warnings. |
| 7. Category insights | app/services/category_insight_engine.py |
Scores the project on buyer-facing categories (green planning, openness, built density, amenities, sports, family living, circulation, sustainability) with positives, risks, evidence and tooltips. |
| 8. Report | app/services/report_writer.py |
Writes a Markdown report and a compare-module JSON contract ready for a product surface. |
Rendered from real runs with the production build (headless Chrome, 1440px wide, scaled down here).
 |
 |
| Birla Taranya, plan URL + Datacenter JSON | Sattva Lumina, plan URL only |
 |
 |
| Mahindra Rainforest Phase-2, an architectural block plan | Lodha Corinthia, plan URL only |
 |
 |
| The hover viewer on Lodha Corinthia: marker-placed courts carry their drawn area | Kalpataru Vian: blocks named from the printed key note, no water |
The backend agents have no UI; what they produce is a backlog, a trail and, from the engineer, pull requests. Real output from this repository, captured the same way:
 |
 |
qa_agents.py backlog: tolerances the teacher tuned from evidence, the engineer's queue, the open proposal |
qa_agents.py propose B3: one attempt, every gate passed, pull request opened |
 |
|
| The pull request the engineer opened (#1): evidence, rationale, tests, guard result and the regression table, waiting for a human. See the first live proposal for why it should be closed rather than merged. |
Green Area is landscaped, planted, garden, lawn, tree-covered or green recreational area that is visible or declared in project data. Roads, driveways, parking, pools, concrete decks, bare paving, clubhouses, retail and non-grass sports courts are excluded.
Open Area is the total non-building footprint inside the project boundary. It includes green area, roads, pathways, paved open areas, sports courts, pools, decks, setbacks and open recreational zones. Tower, clubhouse, retail and utility footprints are excluded.
Full definitions, inclusions and exclusions live in PRODUCT_BIBLE.md.
These rules are mandatory and enforced in code, not left to the model.
- Green Area % must be ≤ Open Area %. If an estimate violates this, green is adjusted downward and a warning is attached.
- Ranges, not points. Raw values are converted to bands whose width depends on confidence. Low-confidence results never show numeric precision.
- No "official" language unless the value comes directly from RERA, a brochure or structured project data.
- Confidence drops for a weak or missing boundary, poor image quality, JSON / image conflict, missing project area, or a high unknown segmentation share.
- Confidence caps: unknown area above 70% caps the score at 0.74; combined with a stylised / not-to-scale warning it caps at 0.69; a final range wider than 20 points caps at 0.72.
- Manual boundary wins. If a user-corrected boundary polygon is supplied, it overrides the AI-detected boundary (schema supported, see DATA_CONTRACTS.md).
- All outputs include reasons and warnings.
Comparison outputs have their own copy guardrails (see below): tied projects never show a "Clear Winner", sustainability is always described as declared features rather than environmental performance, and evidence chips are normalised and length-capped before they reach the frontend.
Every output carries a score between 0 and 1, a label, and the reasons that produced it.
| Label | Score | Range width shown |
|---|---|---|
| High | 0.80 to 1.00 | ±3% |
| Medium | 0.55 to 0.79 | ±5% |
| Low | 0.30 to 0.54 | ±8 to 10% |
| Insufficient | below 0.30 | No numeric range. "Data not sufficient for reliable estimate." |
Scoring starts at a base of 0.50 and adds or subtracts evidence signals: project JSON and project area available, master plan present, boundary detected, manual boundary supplied, green amenity signals, direct declared values, and agreement between the vision and segmentation estimates all add; unclear boundary, low image quality, high unknown area, JSON / image conflict and guardrail adjustments subtract. The exact weights are in CONFIDENCE_SCORING.md.
POST /compare-master-plans takes JSON + master plan for three projects and returns a category comparison matrix built for a buyer-facing compare page.
- Categories include green landscape planning, open space and breathing room, built density and tower footprint, amenity richness, sports and active lifestyle, kids / family / community living, access and internal circulation, pool and water leisure, environment and sustainability, plus an evidence-reliability signal.
- Status per category:
Clear Winner,Close Call,WeakorInsufficient, decided deterministically from score gaps, rating bands and cleaned evidence counts. Ties and near-ties are never shown as decisive. - Evidence chips: master plan components and Datacenter entities that support each category, de-duplicated, normalised and capped in length.
- Master plan understanding (
master_plan_understanding_engine.py): optional Gemini structure extraction of legends, towers, amenities and zones. - AI narrative (
master_plan_narrative_engine.py): optional Gemini-written buyer summary, generated after the deterministic matrix and constrained by it. - Frontend contract validator (
frontend_contract_validator.py): the last gate before the response. Rewrites over-claiming copy, enforces the tie rules, strips unsupported winner badges and keeps sustainability language to declared features.
The comparison is served by the API (POST /compare-master-plans) and covered by tests; the React frontend is now dedicated to single-plan analysis (next section).
POST /master-plan/entities turns one master plan image into a spatial entity map: every wing, commercial block, clubhouse, gate, road, parking cluster, court, pool, garden, play area, game and utility yard with a bounding box, plus the legend as printed and a few site-level facts. The React app (frontend/, Vite + React + TypeScript) takes the plan either as a public image URL (fetched server-side, so CDN CORS and hotlink rules never block it; the stored copy is served back from GET /master-plan/runs/{run_id}/image) or as an upload, with the project's Datacenter JSON optional. From the plan alone it builds an executive dashboard: a composition card (features by type, residential blocks, entry points, parking, plan thumbnail), a land use card (drawing area by use, summed from the located boxes; on architectural block plans also the printed plot area, sanctioned BUA and derived FSI, and a chart of the reservations printed on the sheet), an amenities and lifestyle section (amenity mix by lifestyle category, legend coverage meter, legend promise vs plan evidence per category, walk to the nearest amenity per block) and a spatial and access section (where amenities and green sit on a 3x3 grid of the drawing, block exposure counts, and a block-by-block access matrix of amenities and green nearby plus entry gate, external road, parking and utility-yard proximity). On architectural block plans the amenity section gives way to a block plan analysis section: reservations by type (green and open space, civic, setbacks, future development), phase towers and drawn ground coverage, and road frontages by side and width. Expanding the plan opens the hover viewer: the boxes are drawn over the image, hovering a region shows the feature's full metadata, clicking pins it, filters and search scope the overlay, and a collapsed inventory table is the chart's accessible twin. With the Datacenter JSON the land-use card becomes the green and open area estimate (low to high ranges, OpenCV pixel evidence, confidence reasons), a home mix card is added (carpet area by configuration, green per home, declared vs located amenities), and the zoning, density and land allocation dashboards follow.
How the extraction works (app/services/master_plan_entity_extractor.py):
- Focused Gemini calls run in parallel, each with a strict
response_schemaand thinking at its minimum (thinking_budget=0on Gemini 2.x,thinking_level=minimalon 3.x): a wings call for residential building footprints (re-asked once if the boxes look fabricated), a structures call for the other large features, a site call for the legend text, plot shape, external roads and drawing annotations, and a marker call per image tile (2x2 with overlap) for the small numbered circles that key features to the legend. - Prefixed legend codes resolve from bare numbers. Legends that prefix their rows by category (GR-12, RE-27, PT-44) while the circles on the plan print only the number are matched through a bare-number alias whenever the number is unique across the prefixes; the targeted re-scan asks for both forms. On Birla Taranya this took legend coverage from 15 to 57 of 58 rows.
- Marker-placed areas are boxed over the drawing. A marker places a court, pool, lawn, play area, clubhouse or parking cluster reliably but only as a marker-sized disc; one focused crop call per such feature returns the drawn area's box, adopted only when it contains the marker and is sized like a feature. The hover map then covers the court itself (Lodha Corinthia's futsal court was a disc until this step).
- Small sheets get a named-code marker pass. When the image's longest side is under 1000px the circles are a few pixels across and the open marker pass pads them with invented codes; every legend code is then also asked for by name on each tile, and the two marker lists are merged by position. Markers whose codes are not in the legend are kept only as annotations.
- A misread marker never erases a drawn feature. A drawn box that disagrees with a marker read elsewhere for the same feature used to be discarded; it is now kept and checked on its own crop, and dropped only when the crop does not show that feature (Kalpataru Vian's two multiplay courts came back this way).
- Numbered legend markers are treated as ground truth. A marker whose code matches a legend row becomes that feature at that spot. A drawn box that agrees with a marker is merged with it (
located_by: both); a drawn box far from the marker for the same feature is discarded. - Water features need evidence: a pool or water body that no legend row, declared amenity or legend marker vouches for must be painted in saturated blue or cyan (tower plates and lobbies drawn in pale grey-blue are a known confusion); the ones with blue ink get a second look on their own crop, and only a clear "not water" answer removes them. Plans whose legend or records list water are left untouched.
- Blocks named from printed key notes: when letter keys that are not legend codes are drawn beside the buildings (for example lobby markers C and D with a note "C Lobby- Wing A&B | D Lobby- Wing C&D"), one extra focused call transcribes the notes and the blocks are named from the markers they contain ("Tower 1 (Wing A & B)"); an umbrella box around several buildings is dropped. Plans with numbered markers never trigger this call.
- Architectural block plans (CAD line drawings that print no legend, e.g. a municipal block plan) get two more focused calls, and only they do: one transcribes the printed site figures (title, plot area, sanctioned BUA, scale; FSI is derived) and every labelled parcel with its printed area (setbacks, R.G. area, amenity open space, school, cultural centre, garden/park), and one boxes the tower footprint outlines on the full sheet (two temperatures) and on 2x2 tiles, merged by overlap, because small outlines at the plot edges only show up on the tiles. New footprints are kept only when they do not overlap a known block and are sized like the other towers on the sheet; a per-crop "building outline" check can reject one, but only when a second, wider look agrees, and a rejected outline that sits in a row with two accepted towers is kept regardless. Offset boxes of one tower are merged by centre distance, and a wing box that spans two towers is dropped (when both are boxed inside it) or split into verified halves. Footprints are named from the printed text when the crop reads a name that is not already in use, otherwise "Footprint N". Measured on Mahindra Rainforest Phase-2 (12 towers): before these safeguards repeated runs gave 10, 12, 12 and 14; after, 12 every time. Which towers belong to this phase is measured from the fill colour of each box (hatched or solid saturated fill), not taken from the model. The site summary carries
drawing_style: "block_plan"andprinted_records, and the green/open estimate is marked Low confidence because a line drawing carries little pixel signal. - Known model failure modes are filtered: identical boxes reused across features, evenly spaced same-size runs, boxes covering most of the drawing, and circles detected on blank paper (a pixel check).
- Derived metadata: category key for the nine comparison categories, nearest residential block and distance, per-block proximity notes ("Wing E sits closest to Entry Plaza, Outdoor Gym and Obstacle Park; ..."), legend coverage, entity and group counts.
All coordinates are fractions of the image (0 to 1), so the frontend overlays them on the original upload without any server-side image serving. The same map can be produced for all three projects during a compare run with use_ai_entity_extraction=true; the explorer then shows the maps without a second call.
The entity map carries no model-written prose: the plot shape is an enum (shape_class), per-block proximity is a set of lists and flags, and processing notes are machine-readable diagnostics (code, count, items) that the frontend turns into sentences.
POST /master-plan/analysis adds key-value site metrics (app/services/site_metrics_engine.py) on top of the entity map and renders three dashboards in the explorer:
- Master Plan & Site Zoning: parcel area, open realm and building footprint, homes per acre against micro-market and city medians, sanctioned FSI, land and open realm per home, with the plan thumbnail opening the hover viewer.
- Density & Spaciousness Profile: density curves (kernel density of homes per acre) for the micro-market and the city with the project marked, privacy stat tiles, a land-area-versus-unit-count scatter with a least-squares trendline, and a density-parity callout.
- Land & Open Space Allocation: a flow diagram from site parcel to open realm and building plinth and on to land per home, outdoor amenities and built-up envelope; hovering a flow shows how the figure is derived.
Every figure is a MetricValue with value, unit, source (project_records, master_plan_estimate, master_plan_entities, derived, benchmark_dataset, unavailable), an optional confidence_label and a basis formula key. Figures that only exist in sanction records (FSI, UDS, clubhouse area) stay unavailable unless supplied via site_records_json. Benchmarks come from a JSON dataset at DENSITY_BENCHMARKS_PATH; the repository ships a synthetic sample (data/benchmarks/density_benchmarks.sample.json, is_sample: true) that the UI badges as sample data. Every chart has a table view, hover tooltips with hit targets larger than the marks, and a colour set validated for colour-vision safety.
Entity vocabulary: residential_block, commercial_block, clubhouse, security_cabin, utility_yard, entry_exit_gate, external_road, internal_road, driveway, walkway_pathway, parking, swimming_pool, water_feature, sports_court, kids_play_area, outdoor_game, outdoor_gym, plaza, pet_park, working_pods, other_feature, lawn_garden, tree_canopy_buffer, legend_marker, legend_block, title_block, north_arrow, site_boundary. Boxes are model estimates on a stylised, not-to-scale drawing: use them to locate features, never to measure them.
Every analysis is checked by three backend agents before the response is built. They run inside the API, write their trail to qa_trail.json beside the run artifacts, and add nothing to the API contract or the UI. A fourth agent, the engineer, runs offline and turns what the first three could not fix into pull requests.
request ─► extractor ─► reviewer ─► fixer ─► reviewer ─► best version ─► response
│ │
└──── trail (qa_trail.json) ─────┘
│
teacher ─► playbook: rules, cases, recipes, lessons,
tolerances (self-tuned), backlog
│
offline, by a developer: `qa_agents.py propose <id>` ▼
engineer ─► patch + test ─► unit suite ─► test guards the change
─► five fixture plans ─► pull request (human merges)
| Agent | Module | What it does |
|---|---|---|
| Reviewer | app/agents/reviewer.py |
Deterministic rule checks first (unique block labels, water needs legend/record/marker support or blue ink, no box over 35% of the sheet, legend coverage, printed records on block plans, green never above open, Low confidence on line drawings), then two model calls: an independent count of towers, pools and gates that sees only the drawing, and a judgement that reads the playbook, the count and the extraction summary. Produces a pass/fail verdict, a score (blocker 0.5, major 0.2, minor 0.05 off) and typed findings. |
| Fixer | app/agents/fixer.py |
Maps finding codes to recipes that reuse the extractor's verified tools: tiled footprint recovery for missed towers on block plans, blue-ink and crop checks for suspect water, crop checks for suspect blocks, duplicate-label renaming, oversized-box drops, estimate clamping and downgrading. A model opinion alone never removes a feature. |
| Teacher | app/agents/teacher.py |
Maintains the playbook (app/agents/playbook.py): acceptance rules, rejection cases and fix recipes, seeded from the Sattva Lumina, Birla Taranya, Kalpataru Vian and Mahindra Rainforest cases. After a cycle with findings it distils new entries from the trail with the text model, deduplicates them against the existing text (at most three per kind per cycle) and always records a deterministic lesson line. It also tunes the reviewer's tolerances from evidence and keeps the backlog of what only code can fix. The runtime copy lives at data/agents/playbook.json; the seed in code never changes, and seed entries added later are merged into the runtime copy on load. |
| Engineer | app/agents/engineer.py |
Runs offline, never inside a request. Takes one backlog item, drafts a minimal patch plus a unit test with the text model (it sees the item, the QA trails behind it, the confirmed facts of the fixture plans and the source it may touch), applies it in a throwaway git worktree and opens a pull request only when every gate passes. A human reviews and merges; the agent never merges. |
What runs inside a request and what does not. The fixer only runs recipes that exist as tested code, and the teacher writes the playbook, never the pipeline's source: agents editing the extraction code while a request is served would put untested changes into production. Every finding the fixer could not act on is therefore recorded by the teacher as a backlog item in the playbook, keyed by finding code and plan style, with occurrence counts, example run ids and the model's suggested code change. python scripts/qa_agents.py backlog lists it; python scripts/qa_agents.py teach <run_dir>... feeds stored trails to the teacher. The backlog is the engineer's queue.
python scripts/qa_agents.py backlog # the queue, the tolerances and the pull requests already open
python scripts/qa_agents.py propose B7 # one item (or `next` for the most frequent open one)
python scripts/qa_agents.py propose B7 --skip-plans --no-pr # unit tests only, local branch only
python scripts/qa_agents.py propose B7 --from-report data/agents/proposals/B7-<stamp>.json # re-verify a stored patch, no new draft
python scripts/qa_agents.py regression # re-run the five fixture plans by handThe engineer's gates, in order. A proposal that fails any of them is not published; the attempt, its reason and the patch are kept under data/agents/proposals/ and on the backlog item (attempts, last_attempt), so the next attempt or the next developer starts from the evidence.
- Allowed shape. Edits only under
app/agents,app/services,app/schemas,app/utilsandtests, neverapp/config.py, the frontend, docs or dependencies; everyold_textmust occur exactly once; at most 16 edits and 400 changed lines; at least one new test function; no test function is ever removed. The model gets the validation errors back and tries once more. - Unit suite. The whole pytest suite passes in the worktree with the patch applied. A failure is fed back to the model for a second draft.
- The test guards the change. The new tests are run alone on the base code with only the test edits applied; they must fail there and pass with the change. A test that passes either way is reported and the proposal is refused.
- Fixture plans.
docs/regression/plans.jsonlists the five plans with the facts confirmed by hand (12 towers on the Mahindra block plan, 2 towers and no water on Kalpataru Vian, 8 wings and one pool on Sattva Lumina, 6 towers and a boxed court on Lodha Corinthia, 58 legend rows at 80%+ coverage on Birla Taranya) as inclusive ranges. The pipeline is re-run on each plan from the worktree with the patched code; every fact must still hold. A plan that fails is run once more before it counts as a regression, because the vision model varies run to run; a pass on the retry is reported as "pass (on attempt 2)" so the reviewer sees the flakiness. The worktree carries the runtime playbook, so the QA loop behaves as it does in production; each run leaves its extracted map and QA trail underdata/output/regression_runs/. The stored input images are used when present, otherwise the public URLs.
Only then does it commit on a qa/engineer-<item>-<finding>-<date> branch, push, and open the pull request with gh, whose body carries the backlog evidence, the rationale, the new tests, the guard result and the regression table. The backlog item becomes proposed with the pull request URL; merging it (or closing it) is a human decision, after which teach on new runs stops recording the finding.
Why this shape rather than self-modifying code: the pipeline's quality was won by hand through many probe runs, and run-to-run variance on the vision model is large. An agent that edits production code on its own would have no way to tell a real fix from a lucky run; the regression manifest is the only ground truth, and a pull request is the only place a human can weigh the trade-off. The engineer needs Gemini and a configured gh; --no-pr keeps everything local.
The engineer's first real run took backlog item B3 (marker-placed courts and lawns left as discs on rendered plans, seen on five runs) and opened pull request #1 after one attempt: the suite passed, the new test failed on the base branch and passed with the change, and all five fixture plans held, 12 towers on Mahindra and no water on Kalpataru included. The run is shown in the screenshots.
Every gate passed and the pull request should still be closed, not merged. The patch only marks the marker-expansion recipe as "checked" when it boxes nothing: safe, but it changes nothing on the plan, and it would count those runs as verified clean when the court simply could not be boxed. The model took the easiest route to a green build. That is exactly the judgement the gates cannot make and the reason a human stays in the loop. The prompt now requires a proposal to change what the extractor or the fixer produces for the example runs, and to decline otherwise; the pull request is kept as the worked example.
Two harness lessons from the same run, both fixed: the worktree must carry the git-ignored test fixtures (data/input/342956.json and the master plan image) and the credentials, otherwise the API tests fail and the failure is blamed on the patch; and the suite is run once on the base commit before any model call is spent.
The second run for B3, under the tightened prompt, drafted a real change: it relaxed the acceptance of a marker's drawn area (the marker may sit up to 4% of the sheet outside the box instead of 1%, and the box may be only 10% larger than the disc instead of 20%), with a test that fails on the base. The suite passed, the guard passed, and the fixture gate refused it because the Mahindra block plan came back with 11 towers. That plan has no legend markers, so the patch cannot have touched it. Two things followed: the regression now re-runs a failing plan once before it counts, and a proposal's patch is stored in its report so --from-report can re-verify the same patch without a new draft.
The replay failed on Mahindra again, twice in a row, which ruled out variance. The first suspect was the harness: the worktree had no runtime playbook, so the regression ran with the seed rather than the tolerances the teacher had learned. That was a real gap and is fixed (the worktree now carries data/agents/playbook.json with the credentials and the stored fixture images, and every regression run writes its extracted map and QA trail under data/output/regression_runs/), but the trails then showed it was not the cause: the extractor itself returned 11 towers before the QA loop ran, and a clean worktree without the patch did the same. The pipeline had drifted with no code change. The wing pass had started returning one box over the two narrow bottom-left towers at 1.55 times the median tower width, just under the 1.6 umbrella threshold, and the two real outlines the footprint pass found inside it were discarded as duplicates of that box. A width threshold cannot separate a two-tower umbrella from one large tower on this sheet (both about 0.14 of the sheet wide); two verified outlines inside the box can. The extractor now keeps footprint candidates that sit inside a wider block, and when two of them pass the crop check side by side with a gap between them, they replace the block (umbrella_blocks_split_by_footprints). Mahindra was back to 12 towers on three consecutive runs. The gate was right to refuse each time; what it could not tell was why, and the preserved trails could. With the extractor fixed, the same stored patch was replayed through the gates and passed all five plans, Mahindra at 12 towers on the first attempt; it is open as pull request #2, this time a real change for a human to judge on quality: the relaxed thresholds could let an area jump to a neighbouring feature, which only the overlays show.
What the agents do learn on their own. Three things change without a developer, all of them in the playbook file and all of them bounded. (1) Rules, rejection cases and lessons: the teacher adds a rule only when a trail shows the same fix working, so the reviewer's prompt and the fixer's prompt grow from evidence. (2) Reviewer tolerances: each fixer recipe reports whether it verified the disputed items and found nothing wrong (checked); the teacher counts those into the backlog item (verified_clean), and after three such runs for the same finding and plan style it raises the matching tolerance by one tower, never above two, and logs the step in tuning_history. The Mahindra block plan is the worked example: the independent count kept saying 11 towers against 12 verified footprints, and after three verified-clean rounds the reviewer stops raising that over-count. (3) The backlog itself, which is the list of things only code can fix. python scripts/qa_agents.py backlog prints the tolerances and their history above the open items.
The orchestrator (app/agents/orchestrator.py) runs review -> fix -> review for at most QA_MAX_ROUNDS rounds; a failed review always goes to the fixer, and so does a pass scoring below 0.9 so that fixable major findings still get repaired. The version with the best review score is adopted, so the loop can only improve an analysis or leave it as extracted. The fixer never acts on the reviewer's opinion alone: water backed by the legend, the records or a marker is never re-checked, block crop checks run on block plans only, and a one-tower difference in the independent count is treated as noise. python scripts/qa_agents.py playbook prints the playbook; python scripts/qa_agents.py review <run_dir> re-reviews a stored run.
Datacenter JSON ──► json_parser ─────────────────────────────┐
│
Master plan image ─► image_preprocessor ──┬─► vision_analyzer (Gemini 3.5 Flash) ──┐
└─► segmentation_engine (OpenCV) ────────┤
▼
area_calculator ─► validation ─► confidence ─► explanation
│
┌──────────────────────────────────────────────┤
▼ ▼
category_insight_engine report_writer
│ (Markdown + compare contract)
×3 projects ▼
master_plan_compare_engine ─► master_plan_narrative_engine (Gemini) ─► frontend_contract_validator
│
▼
master_plan_comparison_report_writer ─► React frontend
Vision and segmentation are deliberately separate sources of evidence. The vision model reasons about what is in the plan; OpenCV measures how much; neither is trusted alone, and disagreement lowers confidence.
master-plan-intelligence/
├── app/
│ ├── config.py # pydantic-settings; reads .env
│ ├── main.py # FastAPI app: /health, /estimate, /compare-master-plans, /master-plan/entities, /master-plan/analysis
│ ├── rules/ # class definitions, green/open rules, guardrails, confidence rules
│ ├── schemas/ # Pydantic models for every stage's input and output
│ ├── agents/ # backend QA agents: reviewer, fixer, teacher, playbook, orchestrator, engineer, regression
│ ├── services/ # pipeline stages (see table above)
│ ├── storage/ # SQLAlchemy scaffolding (persistence not yet wired)
│ └── utils/ # file, image and Gemini client helpers
├── frontend/ # Vite + React + TypeScript master plan analysis app (URL or upload)
├── docs/ # contract schema, sample responses, screenshots, regression manifest (fixture plans)
├── scripts/ # run_single_project.py, qa_agents.py, export_contract.py
├── tests/ # pytest suite (109 tests)
├── data/
│ ├── benchmarks/ # density benchmark dataset (synthetic sample shipped)
│ ├── input/ # your project JSON + master plan images (git-ignored)
│ └── output/ # json/, overlays/, reports/, api_runs/, compare_runs/ (git-ignored)
├── PRODUCT_BIBLE.md # vision, definitions, guardrails, roadmap
├── MODEL_LOGIC.md # hybrid model logic, classes, formulas, boundary priority
├── CONFIDENCE_SCORING.md # score weights, labels, range widths
├── DATA_CONTRACTS.md # input/output JSON contracts
├── CODE_DEPENDENCY_MAP.md # module responsibilities and dependencies
└── IMPLEMENTATION_TRACKER.md # step-by-step build log
- Python 3.10+
- Node.js 18+ (frontend only)
- Gemini access: a Google Cloud service account with Vertex AI enabled (preferred) or an AI Studio API key
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env
# edit .env: either point GOOGLE_APPLICATION_CREDENTIALS at a Vertex AI
# service-account JSON (kept outside the repo) or set GEMINI_API_KEY
uvicorn app.main:app --reload --port 2345
# Swagger UI: http://127.0.0.1:2345/docscd frontend
npm install
npm run dev
# http://localhost:5173 (expects the API at http://127.0.0.1:2345; override with VITE_API_BASE_URL)Returns app name, environment and the default port.
Single-project estimate. multipart/form-data.
| Field | Type | Default | Notes |
|---|---|---|---|
project_json_file |
file | required | Datacenter JSON |
master_plan_image_file |
file | required | PNG / JPG / JPEG / WEBP |
run_vision |
bool | false |
Gemini vision analysis |
run_segmentation |
bool | true |
OpenCV segmentation |
run_area_estimate |
bool | false |
Requires vision and segmentation |
run_category_comparison |
bool | false |
Requires area estimate |
write_report |
bool | false |
Requires area estimate |
Response: run_id, run_artifacts (paths to every JSON / overlay / report written under data/output/api_runs/<run_id>/), project_metadata, image_metadata, vision_analysis, segmentation_result, area_estimate, category_comparison, final_outputs.
Three-project comparison. multipart/form-data with project_{a,b,c}_json_file and project_{a,b,c}_master_plan_image_file, plus use_ai_narrative and use_ai_structure_extraction (both default true) and use_ai_entity_extraction (default false; adds a spatial entity map per project at the cost of extra Gemini calls). All pipeline stages run for each project.
Response: comparison_run_id, projects, category_comparison_matrix, category_winners, overall_recommendation, user_friendly_summary, internal_evidence_quality, project_entity_maps ({"A": map | null, "B": ..., "C": ...}), run_artifacts, notes.
Entity map plus key-value site metrics for the dashboards (see above). multipart/form-data.
| Field | Type | Default | Notes |
|---|---|---|---|
master_plan_image_file |
file | one of file / URL | PNG / JPG / JPEG / WEBP |
master_plan_image_url |
string | one of file / URL | Public http(s) image URL, fetched server-side (size-capped, public hosts only) |
project_json_file |
file | optional | Datacenter JSON; without it the entity map and an image-only green / open estimate are returned, but no site metrics |
run_area_estimate |
bool | true |
Runs Gemini vision + OpenCV segmentation to estimate open and green area (with or without project records; records raise the confidence) |
site_records_json |
string | optional | JSON with any of sanctioned_fsi, built_up_area_sq_m, clubhouse_area_sq_m, undivided_land_share_sq_m, open_area_percent, building_footprint_sq_m, homes_per_floor_min, homes_per_floor_max |
Response: run_id, image_url (backend path serving the stored plan), source_image_url, entity_map, site_metrics (or null), area_estimate (or null), image_metadata, diagnostics, run_artifacts. Typical latency is 40 to 70 seconds with the area estimate, 20 to 40 without.
Spatial entity map for one master plan (see above). multipart/form-data.
| Field | Type | Default | Notes |
|---|---|---|---|
master_plan_image_file |
file | one of file / URL | PNG / JPG / JPEG / WEBP |
master_plan_image_url |
string | one of file / URL | Public http(s) image URL, fetched server-side |
project_json_file |
file | optional | Datacenter JSON; seeds the model with declared amenities and tags the result with the project id |
Serves the master plan stored for an analysis or entities run, so the UI can display a URL-sourced plan without hotlinking the original.
Response: run_id, entity_map, image_metadata, run_artifacts. The entity_map carries image_width, image_height, coordinate_system: "fraction_of_image", extraction_confidence_label, site, legend, entities, entity_counts, group_counts, block_proximity, warnings and notes. Each entity has entity_type, entity_group, label, legend_code, legend_label, category_key, box {x_min, y_min, x_max, y_max}, center_x, center_y, relative_area, confidence_label, located_by, attributes, adjacent_to, nearest_residential_block and nearest_residential_distance. Typical latency is 20 to 30 seconds for a 1800 px plan.
POST /master-plan/analysis returns the same shape for every plan. A figure that could not be produced is null, never a missing key, and entity_counts / group_counts carry every key of the vocabulary with zeros, so the frontend and the project detail page can bind to fixed paths. The response is built from app/schemas/analysis_response_schema.py; its JSON Schema is in docs/contract/ (python scripts/export_contract.py regenerates it) and a test asserts the shape is identical with and without project records and on legend-less plans.
Full responses of four real runs are in docs/samples/ (local paths replaced by <repo>):
| Sample | Plan | Input | What it exercises |
|---|---|---|---|
| sattva-lumina.json | Sattva Lumina, Bengaluru | plan URL | rendered plan with a 38-row numbered legend, wings A to H, real pool |
| kalpataru-vian.json | Kalpataru Vian, Mumbai | plan URL | letter keys explained by a printed note, water guardrail (no pools) |
| mahindra-rainforest.json | Mahindra Rainforest Phase-2, Mumbai | plan URL | architectural block plan: printed records, 11 parcels, 12 tower footprints |
| lodha-corinthia.json | Lodha Corinthia, Mumbai | plan URL | six towers, a 16-row legend fully placed, marker-placed courts boxed over their drawn areas |
| birla-taranya-with-records.json | Birla Taranya, Thane | plan URL + Datacenter JSON | site metrics, density benchmarks, land allocation; a GR/RE/PT prefixed legend whose circles print bare numbers |
Shape, with the sub-keys that are always present:
python scripts/run_single_project.py \
--json data/input/342956.json \
--image data/input/master_plan.jpg \
--run-vision \
--run-segmentation \
--run-area-estimate \
--run-category-comparison \
--write-reportFlags are cumulative; omit later ones to stop at an earlier stage. Outputs land in data/output/json/, data/output/overlays/ and data/output/reports/.
{
"green_area_percentage": { "min": 28, "max": 35, "display": "~28–35%" },
"open_area_percentage": { "min": 62, "max": 70, "display": "~62–70%" },
"confidence": {
"score": 0.74,
"label": "Medium",
"reasons": [
"Project area is available in Datacenter JSON.",
"Master plan image is available.",
"Green amenity signals are present.",
"Boundary was detected by AI but not manually verified."
]
},
"guardrails": { "passed": true, "warnings": [], "adjustments": [] },
"compare_module_display": {
"green_space_label": "Estimated Green Space",
"green_space_value": "~28–35%",
"open_area_label": "Estimated Open Area",
"open_area_value": "~62–70%",
"confidence_label": "Medium Confidence",
"tooltip": "Estimated using the project master plan and available project data. Actual values may vary from official layout or RERA disclosures."
}
}| Variable | Default | Purpose |
|---|---|---|
GOOGLE_APPLICATION_CREDENTIALS |
none | Path to a Vertex AI service-account JSON. When set, Gemini runs through Vertex AI |
GOOGLE_CLOUD_PROJECT |
from key file | GCP project for Vertex AI; defaults to the key file's project_id |
GOOGLE_CLOUD_LOCATION |
global |
Vertex AI location. Gemini 3.x models are served from the global endpoint (and asia-south1); us-central1 returns 404 for them |
GEMINI_API_KEY |
none | Gemini Developer API key; used only when no service-account path is set |
GEMINI_VISION_MODEL |
gemini-3.5-flash |
Vision reasoning model |
GEMINI_TEXT_MODEL |
gemini-3.5-flash |
Narrative model |
QA_AGENTS_ENABLED |
true |
Run the reviewer -> fixer -> reviewer loop after every extraction |
QA_MAX_ROUNDS |
2 |
Review rounds per analysis |
QA_TEACHER_ENABLED |
true |
Let the teacher update the playbook after cycles with findings |
QA_PLAYBOOK_PATH |
data/agents/playbook.json |
Runtime playbook (git-ignored; seeded from code) |
DENSITY_BENCHMARKS_PATH |
data/benchmarks/density_benchmarks.sample.json |
Project parcel/unit dataset for density benchmarks; the default is a labelled synthetic sample |
APP_ENV |
local |
Environment label |
APP_NAME |
Master Plan Intelligence |
Reported by /health |
DATABASE_URL |
sqlite:///./goa_engine.sqlite3 |
Reserved for the storage layer |
MAX_IMAGE_SIZE_MB |
30 |
Upload size gate |
DEFAULT_CONFIDENCE_THRESHOLD |
0.65 |
Threshold used by compare-module display logic |
pytest -qThe suite covers the JSON parser, image preprocessor, vision analyzer, segmentation engine, area calculator, confidence engine, validation engine, report writer, category insight engine, master plan compare engine, the master plan entity extractor (payload parsing, marker reconciliation, pattern filters, tiling, truncated-response salvage), the site metrics engine (unit conversions, provenance, allocation flow, density benchmarks), the QA agents (rule checks, severities, fixer recipes, teacher learning and tuning, playbook merge), the engineer (patch validation, the full propose flow on a throwaway git repository with fake tools, the regression manifest) and all API endpoints. No test needs network access; Gemini calls are stubbed.
The fixture-plan regression is the one check that does need Gemini: python scripts/qa_agents.py regression re-runs the five plans in docs/regression/plans.json and prints a pass/fail table against their confirmed facts. Run it before merging any change to the extractor; the engineer runs it for you before it opens a pull request.
Frontend type-check and build:
cd frontend && npm run buildImplemented: full single-project pipeline (parse, preprocess, Gemini vision, OpenCV segmentation, range estimate, guardrails, confidence, explanation, report), category insight engine, three-project master plan comparison API with deterministic statuses and AI narrative, frontend contract validator, master plan spatial entity map with legend-marker reconciliation, key-value site metrics with density benchmarks, FastAPI endpoints, and a React master plan analysis app (plan by URL or upload) with composition, amenity-mix, legend-coverage and block-access dashboards from the plan alone, a hover-to-inspect entity viewer, and zoning / density / allocation dashboards when project records are supplied. Backend QA agents (reviewer, fixer, teacher) run on every analysis with a self-tuning playbook, and an offline engineer agent turns their backlog into gated pull requests.
Planned: reviewer UI with manual polygon correction, batch processing, a trained segmentation model to shrink the unknown share, a human-reviewed evaluation dataset, and wiring the SQLAlchemy storage layer (scripts/init_db.py and scripts/generate_sample_report.py are placeholders today).
Outputs are estimates derived from a master plan image and available project data. They are not official measurements, not RERA disclosures, and must not be presented as certified green or open area figures.
MIT © 2026 Deepesh Kumar Gupta. See LICENSE.
{ "run_id": "uuid", "image_url": "/master-plan/runs/<run_id>/image", "source_image_url": "https://..." , // null for uploads "entity_map": { "project_id": null, "project_name": null, // from the Datacenter JSON when given "image_width": 1113, "image_height": 800, "coordinate_system": "fraction_of_image", "extraction_confidence_label": "High", "site": { "shape_class": "irregular", "north_arrow_present": true, "external_roads": [ { "label", "width_text", "side" } ], "entry_exit_count": 5, "residential_block_labels": [], "commercial_block_count": 2, "parking_cluster_count": 0, "utility_yard_count": 0, "drawing_style": null, // "block_plan" on architectural sheets "printed_records": null // { title, plot_area_text, plot_area_sq_m, sanctioned_bua_text, // sanctioned_bua_sq_m, fsi_text, derived_fsi, scale_text, // parcels: [ { label, area_text, area_sq_m, box } ] } on block plans }, "legend": [ { "code", "label", "normalized_label", "category_key", "located_on_plan", "marker_count" } ], "entities": [ { "entity_id", "entity_type", "entity_group", "label", "legend_code", "legend_label", "category_key", "box": { "x_min", "y_min", "x_max", "y_max" }, "center_x", "center_y", "relative_area", "confidence_label", "located_by", "attributes": {}, "adjacent_to": [], "nearest_residential_block", "nearest_residential_distance" } ], "entity_counts": { "<every entity type>": 0 }, "group_counts": { "structure", "circulation", "amenity", "landscape", "water", "annotation" }, "block_proximity": [ { "block_label", "nearby_amenities", "nearby_green", "near_entry_gate", "near_external_road", "near_utility_yard", "near_parking", "nearest_amenity_distance" } ], "diagnostics": [ { "code", "count", "items" } ] }, "site_metrics": null, // with records: every metric as { value, unit, source, confidence_label, basis }, // unit_mix, allocation_nodes, allocation_links, density_benchmarks, constants "area_estimate": { // null only when run_area_estimate=false or the pipeline failed "green_area_percentage": { "min", "max", "display", "basis" }, "open_area_percentage": { ... }, "confidence": { "score", "label", "reasons" }, "guardrails": { "passed", "warnings", "adjustments" }, "compare_module_display": { ... }, "evidence_summary": { ... }, "warnings": [], "notes": [] }, "image_metadata": { ... }, "diagnostics": [ { "code", "count", "items" } ], "run_artifacts": { "input_master_plan_image_path", "input_project_json_path", "normalized_image_path", "master_plan_entity_map_json_path", "area_estimate_json_path", "site_metrics_json_path", "qa_trail_json_path" } }