Skip to content
View prabhathv07's full-sized avatar

Block or report prabhathv07

Block user

Prevent this user from interacting with your repositories and sending you notifications. Learn more about blocking users.

You must be logged in to block users.

Content in all repositories owned by your account will be closed.
Maximum 250 characters. Please don’t include any personal information such as legal names or email addresses. Markdown is supported. This note will only be visible to you.
Report abuse

Contact GitHub support about this user’s behavior. Learn more about reporting abuse.

Report abuse
prabhathv07/README.md

Prabhath Vipparthi

Data Scientist and AI Engineer
NLP, embeddings, retrieval systems, and the pipelines that keep them running
Applied AI Engineer at Scale AI · M.S. Data Science, NJIT (GPA 3.7)

Portfolio · LinkedIn · Email

Profile views


About

I have four years of experience split across two kinds of work: enterprise data science, and applied AI.

The first stretch was at HCL Tech, where I built churn models, demand forecasts, and customer analytics for enterprise clients. The modeling was rarely the hard part. The hard part was turning a loose business goal into a question a model could answer, and then keeping the outputs trustworthy once other people started making decisions from them.

After that I did my master's in data science at NJIT and moved toward NLP, embeddings, and retrieval. That is the work I do now at Scale AI, where I prepare training and evaluation datasets, engineer features, evaluate classification and language models, and support deployment and monitoring for ML and AI pipelines in production.

I move between traditional statistical modeling and newer AI engineering depending on what the problem needs. A gradient boosting model with well-chosen features often beats something more fashionable, and knowing when that is true saves a lot of time.

The projects below are the ones I would want you to open rather than take my word for. They run on real public datasets, they have test suites, and the CI badges point at workflows that execute the pipelines.

Experience

Applied AI Engineer, Scale AI (January 2026 to present)

Building Python, SQL, and PySpark pipelines for dataset preparation and feature engineering. Training classification models in scikit-learn and PyTorch with cross-validation and threshold tuning to improve minority-class detection. Using Hugging Face transformers and embeddings for text classification and semantic evaluation, and implementing embedding-based retrieval workflows behind Python REST APIs for similarity search and LLM evaluation. Deployment and monitoring run through MLflow, Docker, Kubernetes, Azure, and Databricks, with tracking for model quality, data drift, and latency.

Data Scientist, HCL Tech (June 2021 to July 2024)

Churn and demand forecasting for enterprise clients. Engineered 30 or more customer-level features covering purchase frequency, recency, transaction value, and engagement, then compared logistic regression, random forest, and gradient boosting under cross-validation and ROC-AUC, which improved recall on high-risk customers by 18 percent. Forecasting models cut error by roughly 15 percent. Recurring workflows ran on Airflow, AWS, and MLflow, with Tableau dashboards for the business side.

Projects

CI

A briefing system that runs unattended every weekday morning and is live here. It pulls about 130 symbols across futures, macro indicators, sector ETFs, a mover universe, and a watchlist using yfinance with a Polygon.io fallback, computes RSI-14 and moving averages with 20-day sparklines, sends a compact structured summary to Gemini 2.5 Flash, and emails the result to confirmed subscribers.

The part I would point a reviewer at is the retrieval layer. Each briefing is chunked and embedded with text-embedding-004 into a pgvector column, and POST /ask answers plain-English questions about the archive with inline date citations. The same retrieval interface sits over two backends: Python cosine on SQLite locally, and native HNSW embedding <=> query on Postgres in production.

Every LLM call is logged to Postgres with its input, output, model name, latency, and status. That was a deliberate choice after an early version passed raw JSON to the model and responses started truncating. With the logs, a regression shows up as a query I can run instead of an alert that never fires. There are 68 tests across 9 modules, all offline against SQLite and deterministic fake providers, so the suite needs no network access or API keys.

FastAPI PostgreSQL pgvector Gemini RAG SQLAlchemy Docker Render GitHub Actions

My graduate capstone, built for NJIT's Learning and Development Initiative to replace manual taxonomy tagging of digital badges. It is in use by NJIT staff.

The classifier is deliberately rule-based rather than a fine-tuned model. Staff needed to see why a badge landed in a category and be able to override it, so a multi-layer signal extraction pipeline combines 130 or more lexicon phrase patterns, 44 regex rules, and a spaCy verb matcher, feeding a three-stage engine that assigns category, then type, then cognitive level. Each decision carries a confidence score, a plain-English explanation, and an audit log entry.

On the human-labeled evaluation set it classified all 20 badges correctly. The test suite has 351 tests at a 100 percent pass rate and over 85 percent coverage, running in GitHub Actions.

FastAPI React/Vite spaCy SQLite Python TDD

An adaptation of the self-instruct method to TypeScript. The pipeline streams source files from The Stack v2, extracts functions with tree-sitter, and filters seed examples through automated validation before generation. A multi-stage run on StarCoder2-3B through vLLM produces instruction-response pairs, with an LLM-as-a-judge stage dropping low-confidence outputs.

Most of the effort went into the filtering rather than the generation. Unfiltered seeds produce fluent instruction pairs that are quietly wrong, which is worse than producing fewer of them.

Python vLLM StarCoder2 Hugging Face PyTorch tree-sitter

CI

A bronze, silver, and gold pipeline over 5,972,150 real NYC TLC yellow taxi trips from January and February 2024. The PySpark cleaning job drops 528,764 rows, or 8.85 percent, for bad fares, zero distance, inverted timestamps, and impossible occupancy counts, leaving 5,443,386 clean trips. dbt builds four gold marts and runs 23 data-quality tests, all passing, and the whole thing is chained into a daily Airflow DAG. CI runs the pipeline end to end on every push.

What the data actually says:

  • Manhattan produces 75 percent of revenue, $111.9M of $149.1M, so yellow taxis are mostly a Manhattan and airport product
  • Credit card accounts for $128.3M, or 86 percent of revenue, against $19.3M in cash
  • Airport-run hours between 4 and 6 AM carry average fares of $23 to $28

There is also a Streamlit dashboard for revenue and payment trends by borough and time period.

PySpark dbt DuckDB Airflow Streamlit Docker GitHub Actions

CI

A hypothesis test on 96,096 real Olist customers and 99,441 orders from 2016 to 2018, asking whether customers who first pay by credit card come back more often than those who first pay by boleto. Six DuckDB window-function queries build a cohort retention matrix, and a two-proportion z-test settles the question.

They do not. Repeat rates were 2.07 percent for credit card against 2.10 percent for boleto, a difference of 0.03 percentage points with p = 0.76. The power analysis matters more than the result: the test can detect gaps of 0.34 percentage points or larger at 80 percent power, so this is a confident null rather than an underpowered one.

The more useful finding was incidental. Only 3.1 percent of customers ever place a second order at all, which puts the real opportunity in the first-to-second purchase, not in payment method.

Python DuckDB SQL statsmodels scipy pandas

Three MapReduce jobs on a 4-node AWS EC2 Hadoop cluster, processing 2 GB of OHLCV tick data across more than 100 trading pairs. The jobs compute volatility rankings, open-to-close performance, and cumulative volume with peak timestamps. Each uses a single reducer, which forces a global top 10 instead of per-node local rankings.

Java Apache Hadoop MapReduce HDFS AWS EC2

Tech stack

Languages and core

Python SQL Java TypeScript pandas NumPy

NLP, LLMs, and applied AI

PyTorch Transformers spaCy vLLM RAG pgvector Google GenAI

ML and statistics

scikit-learn TensorFlow statsmodels SciPy

Data engineering

PySpark Databricks dbt Airflow DuckDB Hadoop

MLOps, infrastructure, and visualization

MLflow Docker Kubernetes Azure AWS FastAPI PostgreSQL GitHub Actions Power BI Streamlit

Education

M.S. Data Science, computational track, New Jersey Institute of Technology, Ying Wu College of Computing. GPA 3.7.

Contact

Popular repositories Loading

  1. event_manager event_manager Public

    Forked from kaw393939/event-manager-qa-onboarding

    Python

  2. user_management user_management Public

    Forked from WISClub/user_management

    FastAPI + PostgreSQL user management service with JWT OAuth2, role-based access control, 138 automated tests, and GitHub Actions CI/CD

    Python

  3. Data-Science-Projects Data-Science-Projects Public

    Jupyter Notebook

  4. Frequent-Itemset-Mining Frequent-Itemset-Mining Public

    Market basket analysis comparing Brute Force, Apriori, and FP-Growth across five retail transaction datasets

    Python

  5. Heart_Failure_Prediction Heart_Failure_Prediction Public

    Binary heart disease prediction using Random Forest, LSTM & KNN — 86.8% accuracy, AUC 0.94, 10-fold cross-validation

    Jupyter Notebook

  6. prabhathv07.github.io prabhathv07.github.io Public

    TypeScript