Interactive Handwritten Digit Recognizer Canvas

An interactive desktop drawing canvas application powered by a deep Convolutional Neural Network (CNN) that achieves 99.65% top-1 accuracy on the MNIST handwritten digit benchmark. Built with Pygame and TensorFlow/Keras, this project features dynamic freehand stroke rasterization, sub-millisecond tensor downsampling interpolation, and an audio-reactive state machine user interface.

---

📸 Interactive Desktop Experience

The application provides a responsive drawing board where users sketch digits in real time:

┌─────────────────────────────────────────────────────────────────────────┐
│                                                                         │
│   ┌───────────────────────────┐         ┌───────────────────────────┐   │
│   │                           │         │   PREDICTED RESULT:       │   │
│   │                           │         │                           │   │
│   │       [ 8 ]               │         │           [ 8 ]           │   │
│   │                           │         │                           │   │
│   │    Freehand Canvas        │         └───────────────────────────┘   │
│   │    (392 x 392 px)         │                                         │
│   │                           │         ┌───────────────────────────┐   │
│   └───────────────────────────┘         │  [ RESET CANVAS ]         │   │
│                                         ├───────────────────────────┤   │
│   • Sub-millisecond raster              │  [ PREDICT DIGIT ]        │   │
│   • Downsampled to 28x28 tensor         └───────────────────────────┘   │
│                                                                         │
└─────────────────────────────────────────────────────────────────────────┘

Key Highlights

• High-Accuracy CNN Architecture: Achieves 99.65% test accuracy through dual multi-scale convolutional blocks, Batch Normalization after every layer, strided downsampling, and heavy spatial dropout regularization.
• Real-Time Stroke Ingestion: Converts continuous mouse brush inputs into discrete intensity distributions using spatial downsampling algorithms without external rasterization libraries.
• Coordinate Space Harmonization: Bridges the coordinate gap between Pygame's top-left pixel space and MNIST's Cartesian array orientation via 90-degree matrix rotation and vertical axis flipping.
• Multi-State Audio-Visual Engine: Features a full-screen GUI with DPI awareness, smooth button animations, acoustic feedback (click.wav, building.wav), and state-driven transitions (Home $\to$ Predicting $\to$ Building $\to$ Result $\to$ Reset).

---

🧠 System Architecture & Preprocessing Flow

The pipeline translates freehand cursor strokes into high-confidence classification predictions:

[ Freehand Cursor Input on Canvas ]
                │
                ▼
   ┌──────────────────────────┐
   │    pygame.PixelArray     │ ──> Ingests raw RGB pixel buffer (392 x 392 px)
   └──────────────────────────┘
                │
                ▼
   ┌──────────────────────────┐
   │ Intensity Grid Sampling  │ ──> Integrates stroke density into 14x14 sub-blocks
   └──────────────────────────┘
                │
                ▼
   ┌──────────────────────────┐
   │  Tensor Harmonization    │ ──> Applies np.rot90() and np.flipud() to match MNIST
   └──────────────────────────┘
                │
         Normalized Tensor (1, 28, 28, 1)
                │
                ▼
 ┌────────────────────────────────────────────────────────┐
 │                   Deep CNN Architecture                │
 │                                                        │
 │ Block 1: Conv2D(32, 3x3) -> Conv2D(32, 3x3)            │
 │          -> Conv2D(32, 5x5, s=2) + BN + Dropout(0.4)   │
 │                                                        │
 │ Block 2: Conv2D(64, 3x3) -> Conv2D(64, 3x3)            │
 │          -> Conv2D(64, 5x5, s=2) + BN + Dropout(0.4)   │
 │                                                        │
 │ Head:    Flatten() -> Dense(128) + BN + Dropout(0.4)   │
 │          -> Dense(10, Softmax)                         │
 └────────────────────────────────────────────────────────┘
                │
                ▼
   ┌──────────────────────────┐
   │ argmax() Output (0 - 9)  │ ──> Rendered to Pygame Results HUD
   └──────────────────────────┘

---

📐 Mathematical Formulation & Preprocessing

1. Spatial Downsampling & Intensity Integration

The interactive canvas has a native resolution of $392 \times 392$ pixels, which scales dynamically with display resolution. To downsample this canvas into a standardized $28 \times 28$ MNIST tensor, the drawing region is partitioned into an $28 \times 28$ grid of local cells, each measuring $S_x \times S_y$ pixels ($14 \times 14$ at 1080p).

For each grid cell $(i, j)$, the intensity value $\mathbf{A}[i, j]$ is computed by integrating the proportion of drawn black pixels: $$\mathbf{A}[i, j] = \frac{1}{S_x \cdot S_y} \sum_{u=0}^{S_x-1} \sum_{v=0}^{S_y-1} \mathbb{I}\Big(\text{PixelArray}[i \cdot S_x + u, ; j \cdot S_y + v] = \mathbf{0}_{\text{RGB}}\Big)$$ where $\mathbb{I}(\cdot)$ is the indicator function evaluating to $1$ for drawn stroke pixels and $0$ for white canvas background.

2. Coordinate Transformation & Orientation Alignment

Pygame surfaces are indexed using screen coordinates where $(0, 0)$ is the top-left corner and the $y$-axis increases downward, whereas NumPy and image classification frameworks expect standard Cartesian matrix indexing. To align user sketches with the MNIST orientation: $$\mathbf{T} = \text{flipud}\big(\text{rot90}(\mathbf{A})\big)$$ The resulting array $\mathbf{T}$ is reshaped to shape (1, 28, 28, 1) and forwarded directly to the network.

---

🔬 CNN Model Architecture

The neural network (CNN.py) is designed with deep convolutional stages that prioritize spatial hierarchy extraction and robust invariance against stroke thickness:

Layer	Type	Specifications	Output Shape	Parameters
Input	InputLayer	Grayscale MNIST image	(28, 28, 1)	0
Block 1	Conv2D	32 filters, 3×3, ReLU	(26, 26, 32)	320
	BatchNormalization	Axis=-1	(26, 26, 32)	128
	Conv2D	32 filters, 3×3, ReLU	(24, 24, 32)	9,248
	BatchNormalization	Axis=-1	(24, 24, 32)	128
	Conv2D (Strided)	32 filters, 5×5, stride=2, 'same', ReLU	(12, 12, 32)	25,632
	BatchNormalization	Axis=-1	(12, 12, 32)	128
	Dropout	Rate = 0.4	(12, 12, 32)	0
Block 2	Conv2D	64 filters, 3×3, ReLU	(10, 10, 64)	18,496
	BatchNormalization	Axis=-1	(10, 10, 64)	256
	Conv2D	64 filters, 3×3, ReLU	(8, 8, 64)	36,928
	BatchNormalization	Axis=-1	(8, 8, 64)	256
	Conv2D (Strided)	64 filters, 5×5, stride=2, 'same', ReLU	(4, 4, 64)	102,464
	BatchNormalization	Axis=-1	(4, 4, 64)	256
	Dropout	Rate = 0.4	(4, 4, 64)	0
Head	Flatten	Flatten feature map	(1024)	0
	Dense	128 units, ReLU	(128)	131,200
	BatchNormalization	Axis=-1	(128)	512
	Dropout	Rate = 0.4	(128)	0
	Dense (Output)	10 units, Softmax	(10)	1,290

• Total Parameters: ~327,242 trainable parameters.
• Strided Convolutions vs Max-Pooling: Using $5 \times 5$ convolutions with stride 2 allows the network to learn its own downsampling kernel rather than applying static max pooling.
• Batch Normalization & Dropout: Aggressive regularization ensures the network does not overfit to specific font artifacts and generalizes smoothly to human freehand input.

---

📁 Repository Structure

digit-recognizer/
├── assets/                   # Graphical UI assets and audio effects
│   ├── building.wav          # Acoustic feedback during prediction processing
│   ├── click.wav             # Button click sound effect
│   ├── home.jpg              # Idle ready canvas UI skin
│   ├── predict.jpg           # Predict button hover/active texture
│   ├── reset.jpg             # Canvas reset button active texture
│   ├── result.jpg            # Prediction results HUD background
│   └── building1-3.jpg       # Three-frame animated calculation sequence
├── CNN.py                    # CNN architecture definition & MNIST training pipeline
├── main.py                   # Pygame interactive desktop canvas & inference loop
├── model.h5                  # Pre-trained CNN weights (~4 MB, 99.65% test accuracy)
├── requirements.txt          # Python dependencies
└── LICENSE                   # GNU General Public License v3.0

---

⚡ Quickstart & Installation

1. Environment Setup

Clone the repository and set up a clean Python virtual environment:

git clone https://github.com/FortunateSpy5/digit-recognizer.git
cd digit-recognizer

# Create and activate virtual environment
python -m venv venv

# On Windows:
.\venv\Scripts\activate

# On Linux / macOS:
source venv/bin/activate

2. Install Dependencies

pip install -r requirements.txt

---

🚀 Running the Interactive Application

Launch the desktop drawing canvas:

python main.py

User Controls

• Draw: Left-click and drag inside the central square canvas to sketch any digit from 0 to 9.
• Predict: Click the Predict button on the right-hand panel. The application will play a transition sound, animate the processing state, and display the predicted digit in bold typography.
• Reset: Click the Reset button to wipe the canvas and sketch a new digit.
• Exit: Press ESC on your keyboard or close the window.

---

🏋️ Retraining the Model from Scratch

If you wish to retrain the CNN model on the MNIST dataset:

python CNN.py

• Downloads the latest MNIST train (60,000) and test (10,000) sets.
• Normalizes grayscale float tensors $[0, 1]$ and one-hot encodes targets.
• Trains over 60 epochs with Adam optimization (batch_size=64).
• Saves the newly optimized weights directly to model.h5.

---

👥 Authors & Acknowledgements

• Soumotanu Mazumdar (@FortunateSpy5)
• Sayantan Mondal
• MNIST Database: Yann LeCun, Corinna Cortes, Christopher Burges (yann.lecun.com/exdb/mnist)

---

📄 License

This repository is distributed under the GNU General Public License v3.0. See the LICENSE file for complete details.