LLM-ML-Engineering-Lab

English | 简体中文

A hands-on machine learning learning repository, organized chapter by chapter: from math preliminaries and feature engineering through classical models (KNN, linear / logistic regression, perceptron) to clustering, with runnable notebooks and scripts for every topic. Deep learning / LLM dependencies (PyTorch, Transformers) are declared but experiments have not started yet.

🎯 Learning Goals

• Build ML intuition from the math up: functions, vectors, matrices, probability
• Understand and implement core training mechanics by hand (e.g., gradient descent for linear regression)
• Master the scikit-learn workflow: data splitting, scaling, model fitting, metric evaluation
• Recognize underfitting / overfitting and counter them with polynomial degree control and L1/L2 regularization
• Prepare the environment for later deep learning and LLM engineering experiments

📚 Learning Roadmap

Fundamentals

• Functions, vectors, matrices, probability (ch01)
• Feature engineering: variance filter, Pearson / Spearman correlation, PCA (ch02)
• Model fitting: underfitting / overfitting, Lasso & Ridge regularization
• Gradient descent (notebooks + hand-written implementation)

Classical Machine Learning

• Evaluation metrics: precision / recall / F1, classification_report (ch03)
• KNN: classification, regression, feature scaling, heart disease case (ch04)
• Linear regression: API usage, gradient descent, SGD, advertising case (ch05)
• Logistic regression: heart disease classification, MNIST-style digit recognizer (ch06)
• Perceptron: AND / NAND logic gates (ch07, partial)
• Information theory: binary entropy curve (ch08, basics)
• Unsupervised learning: K-Means, inertia, silhouette & Calinski-Harabasz scores (ch09)

Deep Learning / LLM (planned)

• Neural networks with PyTorch
• Tokenization & embeddings
• Fine-tuning with Transformers / Datasets / Accelerate
• Chinese text segmentation experiments with jieba

🧠 Topics Covered

Chapter	Directory	Contents
ch01	source/notebook/ch01_preliminary_knowledge/	Functions, vectors, matrices, probability
ch02	source/notebook/ch02_feature_engineering/feature/	Variance filter, Pearson correlation, Spearman correlation, PCA
ch03	source/notebook/ch03_metrics/	Classification metrics, classification_report
ch04	source/notebook/ch04_knn/	KNN classification & regression, scalers, heart disease dataset
ch05	source/notebook/ch05_linear_regression/	Linear regression, hand-written gradient descent, SGD, advertising dataset
ch06	source/notebook/ch06_logsitic_regression/	Logistic regression API, heart disease, digit recognizer (MNIST-style train.csv)
ch07	source/notebook/ch07_perception/	Perceptron logic gates (AND / NAND)
ch08	source/notebook/ch08_other_supervised/	Binary entropy plot
ch09	source/notebook/ch09_unsupervised/	K-Means on make_blobs, clustering metrics
—	source/notebook/1_fitting_test.py, 2_regularization.py	Polynomial fitting, Lasso / Ridge regularization
—	source/notebook/3_gradient_descent1.ipynb, 4_gradient_descent2.ipynb	Gradient descent notebooks

🧪 Experiments

Polynomial Fitting & Regularization

1_fitting_test.py fits a noisy sine curve with polynomial features of increasing degree to show underfitting, good fit, and overfitting side by side; 2_regularization.py then applies Lasso and Ridge to pull the overfitted high-degree polynomial back.

Hand-Written Gradient Descent

ch05_linear_regression/2_gradient_descent.py implements batch gradient descent from scratch with NumPy (cost function J(beta), gradient, iterative update) on simulated study-hours → score data, and compares against the scikit-learn LinearRegression result.

Heart Disease Classification

data/heart_disease.csv (Chinese column headers, 13 features + diagnosis label) is used with both KNN (ch04) and logistic regression (ch06) to compare classical classifiers on the same tabular problem, including feature scaling.

Digit Recognizer

ch06_logsitic_regression/3_digit_recognizer.py trains LogisticRegression on data/train.csv (MNIST-style 28×28 pixel images with labels), with MinMaxScaler preprocessing and an 80/20 train/test split.

K-Means Clustering

ch09_unsupervised/ clusters make_blobs data with KMeans, visualizes raw data vs. cluster assignments and centroids, and evaluates with inertia_, silhouette score, and Calinski-Harabasz score.

🛠️ Tech Stack

• Language: Python ≥ 3.14
• ML: scikit-learn, NumPy, Pandas
• Visualization: Matplotlib, Seaborn, Jupyter Notebook
• Declared for future LLM work (not yet used in code): PyTorch (+ torchvision / torchaudio), Transformers, Datasets, Tokenizers, Accelerate, jieba
• Environment: managed with uv (uv.lock), requirements.txt also provided

📂 Project Structure

LLM-ML-Engineering-Lab/
├── source/
│   └── notebook/
│       ├── 1_fitting_test.py               # underfitting / overfitting demo
│       ├── 2_regularization.py             # Lasso / Ridge regularization
│       ├── 3_gradient_descent1.ipynb       # gradient descent notebook I
│       ├── 4_gradient_descent2.ipynb       # gradient descent notebook II
│       ├── ch01_preliminary_knowledge/     # functions, vectors, matrices, probability
│       ├── ch02_feature_engineering/       # variance filter, correlation, PCA
│       ├── ch03_metrics/                   # classification metrics
│       ├── ch04_knn/                       # KNN classification & regression
│       ├── ch05_linear_regression/         # linear regression & gradient descent
│       ├── ch06_logsitic_regression/       # logistic regression cases
│       ├── ch07_perception/                # perceptron logic gates
│       ├── ch08_other_supervised/          # entropy basics
│       ├── ch09_unsupervised/              # K-Means & clustering metrics
│       └── data/
│           ├── advertising.csv             # TV/Radio/Newspaper → Sales
│           ├── heart_disease.csv           # heart disease classification data
│           └── train.csv                   # MNIST-style digit data
├── pyproject.toml                          # uv project definition (Python ≥ 3.14)
├── requirements.txt
├── uv.lock
└── README.md

🚀 Getting Started

Requirements

• Python ≥ 3.14 (declared in pyproject.toml)
• uv (recommended) or pip

Installation

With uv:

uv sync

Or with pip:

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Running Examples

Notebooks:

uv run jupyter notebook source/notebook

Scripts (note the relative CSV paths assume running from the chapter directory):

cd source/notebook/ch06_logsitic_regression
uv run python 3_digit_recognizer.py

📝 Notes

• Notebook file paths use relative data references such as ../data/train.csv, so run scripts from their own chapter directory.
• heart_disease.csv uses Chinese column headers.
• No test suite or CI is configured — this is a learning repository, examples are verified by running them directly.
• The directory name ch06_logsitic_regression contains a typo ("logsitic"), kept as-is.

🗺️ Roadmap

• Complete the perceptron chapter (XOR and multi-layer extension)
• Decision trees / information-gain-based models building on the entropy notebook
• Start PyTorch neural network experiments (dependencies already declared)
• Move into LLM engineering: tokenization, embeddings, fine-tuning, RAG

📄 License

This project is licensed under the MIT License — see the LICENSE file for details.