MathOCR — Handwritten Math Recognition

MathOCR is a local Tkinter application that recognizes handwritten math symbols and produces Unicode and LaTeX. Training, inference, and saved samples stay on your machine.

Quick start

MathOCR supports Python 3.10–3.12 with the dependency ranges in requirements.txt.

python -m venv .venv
.venv\Scripts\activate
pip install -r requirements.txt
python app.py

On Linux or macOS, activate with source .venv/bin/activate.

The included .keras model can be loaded from the Training tab. To train a digit-only model, choose Train on MNIST. MNIST is downloaded by TensorFlow on first use.

Training the math-symbol model

The training loader expects one folder per class:

dataset/
├── 0/
├── 1/
├── plus/
├── int/
├── sqrt/
└── ...

Choose Load CROHME dataset… and select the parent folder. Every class needs at least two readable PNG, JPEG, or BMP images. The application validates the dataset, creates a stratified split, applies balanced class weights, checkpoints the best validation model, reduces the learning rate when needed, and restores the best weights.

After training, the log reports the weakest validation classes. Full per-class accuracy and the confusion matrix are stored in model metadata.

Drawing and personalization

• Draw one line of math on the ruled canvas.
• Auto-recognition is debounced and runs through one background inference worker.
• Low-confidence symbols appear as ? with alternative candidates.
• Add labeled samples from your handwriting, then choose Fine-tune on samples.
• Fine-tuning mixes personal samples with a balanced replay set to reduce catastrophic forgetting.
• Use Cancel training for a graceful stop at the next epoch boundary.

The segmenter groups connected strokes using adaptive size and vertical overlap. It handles multi-stroke symbols such as =, i, and ÷ better than a fixed horizontal gap. Expression output remains left-to-right; full two-dimensional fraction, root, and superscript parsing is a future model-level feature.

Saving models

Saving creates a pair:

mathocr_model.keras
mathocr_model.meta.json

Metadata is versioned and records the class order, preprocessing version, image size, random seed, training history, user samples, and replay samples. Loading validates both files, their schema, array shapes, labels, and the model output count before changing the active model.

Older schema-v1 metadata remains supported when it matches the current 28×28 preprocessing contract.

Code structure

app.py                Tkinter UI and background-job coordination
model.py              Lazy TensorFlow loading, training, inference, persistence
image_processing.py   TensorFlow-free preprocessing and segmentation
formatting.py         Unicode and LaTeX formatting
symbols.py            Class aliases and symbol mappings
test_*.py             Fast unit and regression tests

Importing the image and formatting modules does not initialize TensorFlow, so the unit suite runs in a fraction of a second.

Development

Install development tools:

pip install -r requirements-dev.txt

Run verification:

python -m unittest discover -v
ruff check .
mypy image_processing.py formatting.py symbols.py model.py

GitHub Actions runs the fast suite and static checks on Python 3.10, 3.11, and 3.12. Training is intentionally excluded from CI because it requires downloading datasets and substantial compute.

Model architecture

Input 28×28×1
├── translation, zoom, and rotation augmentation
├── Conv2D(32) ×2 + BatchNorm + MaxPool + Dropout
├── Conv2D(64) ×2 + BatchNorm + MaxPool + Dropout
├── Conv2D(128) ×2 + BatchNorm
├── GlobalAveragePooling
├── Dense(256, ReLU) + Dropout
└── Dense(N classes, Softmax)

Training seeds NumPy/TensorFlow operations where supported and requests deterministic TensorFlow operations when the platform provides them.