🤖 PROTO-X-ROVA: Advanced AI Humanoid Assistant

Voice Interaction • Computer Vision • Autonomous Navigation • IoT Robotics

---

🌟 Overview

ROVA (Robot with Voice and Vision Analysis) is a state-of-the-art, open-source, AI-powered robotic platform. Developed by a team of visionary students from Takshashila Academy, ROVA integrates high-torque mechanical engineering with a modern, local-first AI software stack.

This NPR 75,000 project was funded by Takshashila Academy, Kathmandu.

Designed to run completely autonomously without relying on cloud APIs, ROVA combines local Large Language Models (LLMs), real-time computer vision, and edge computing (ESP32) to create a robot that can see, hear, speak, and interact with its environment.

The real PROTO-X-ROVA physical prototype in action

---

✨ Key Features

• 🗣️ Natural Voice Interaction: Uses faster-whisper for offline Speech-to-Text and pyttsx3 for Text-to-Speech. Real-time communication using local or cloud LLMs.
• 🧠 Local AI Brain: Powered by local models via Ollama (Qwen2.5), ensuring complete privacy and zero API costs.
• 👁️ Real-Time Vision: Utilizes ultralytics YOLO models for real-time object detection and obstacle avoidance via a smartphone IP webcam.
• 🦾 Precision Control: Heavy-duty 4-wheel drive system with high-current BTS7960 motor drivers and dual ESP32 architecture for controlling multi-axis servo arms.
• 🌐 Web Dashboard: A sleek local web interface for manual control, live telemetry, video streaming, and remote interaction.

---

🏗️ Architecture

ROVA operates on a decentralized, microservice-like architecture designed for low-latency robotics:

graph TD
    subgraph "Main Server (Laptop/PC)"
        A[Voice & Brain Node\n'esp_voice_server.py']
        B[Vision Node\n'vision_node.py']
        O[Ollama Server\nLLM Engine]
    end

    subgraph "Edge Hardware (ESP32)"
        C[Chassis & Audio ESP32\n'sketch_jan13a.ino']
        D[Arm Controller ESP32\n'arm_controller.ino']
    end
    
    subgraph "External Sensors"
        P[Smartphone\nIP Webcam App]
    end

    A <-->|Audio Stream & TCP Commands| C
    A <-->|TCP Commands| D
    A <-->|Prompts & Responses| O
    P -->|Video Stream| B
    B -->|Obstacle/Detection Events| A

Loading

---

🛠️ Hardware Stack

To build your own ROVA, you will need:

Component	Specification
Microcontrollers	2x ESP32 Dev Boards (Chassis & Arms)
Drive Motors	4x GH37-385 (12V DC, 100 RPM, Geared)
Motor Drivers	2x BTS7960 (43A High-Current Bridge)
Arm Servos	SG90 / MG996R + PCA9685 PWM Controller
Power System	12V LiPo/Lead-acid with LM2596 Buck Converters
Audio Input	I2S Microphone (INMP441)
Audio Output	I2S Amplifier (MAX98357A) + Speaker
Vision	Smartphone running IP Webcam / ESP32-CAM

---

🚀 Getting Started

Follow these instructions to set up the software environment for ROVA.

Documentation and Configuration

The repository includes detailed guides for the hardware and software layers:

• Complete wiring guide - power, ESP32, motors, audio, sensors, and arm wiring.
• BTS7960 motor-driver guide - motor power and GPIO connections.
• ESP32 and INMP441 notes - microphone and ESP32 connection details.
• Speech recognition uses Faster-Whisper and downloads the tiny.en model on first run; set LOCAL_WHISPER_MODEL_PATH in .env.local to use a local model instead.
• Environment template - local IP addresses and optional model settings.

ROVA runs several local services. The voice server listens on TCP port 4000, the arm controller on 4001, the vision node sends events to port 5000, and the web dashboard is served on port 8080. Keep the laptop, ESP32 boards, phone camera, and Ollama host on the same trusted LAN.

1. Prerequisites & Cloning

• Python 3.10+
• Arduino IDE / PlatformIO
• Ollama (For local AI)

git clone https://github.com/Susant07-star/PROTO-X-ROVA.git
cd PROTO-X-ROVA/Robot

2. Python Environment Setup

python -m venv venv
# Windows:
venv\Scripts\activate
# Linux/Mac:
source venv/bin/activate

# Install dependencies
pip install -r requirements.txt

3. Environment Configuration

ROVA uses a .env.local file to manage IP addresses securely without hardcoding them into the repository.

# Run this from the Robot/ directory
# Windows PowerShell:
Copy-Item .env.example .env.local
# Linux/macOS:
cp .env.example .env.local

Open .env.local in a text editor and fill in your specific IP addresses:

• CONTROL_NODE_IP: The IP of the laptop running the Python server.
• PHONE_CAM_URL: The URL of your smartphone's IP Webcam app.
• OLLAMA_IP: The IP of the machine running the Ollama server.

4. Install Ollama (The Brain)

1. Install Ollama.
2. Pull the model used by the current server configuration:

ollama run qwen2.5:7b-instruct

For lower-memory systems, use a smaller Qwen2.5 model and set `OLLAMA_MODEL` in `.env.local` to match.

---

## 💻 Running the System

You must start the system in a specific order to ensure all nodes connect successfully. All commands should be run from the `Robot/` directory.

### Step 1: Start the Voice & Brain Server
```bash
python esp_voice_server.py

You should see a message indicating the TCP server is listening on port 4000.

Step 2: Start the Vision Node

In a new terminal window (with your virtual environment activated):

python vision_node.py

Ensure your smartphone IP Webcam app is running before starting this script.

Step 3: Power on the Robot

Turn on the ESP32 microcontrollers. They will automatically connect to your WiFi network and then connect to the Python server via TCP.

Step 4: Open the Web Dashboard

Navigate to http://localhost:8080 in your web browser. You can manually drive the robot, view the live camera feed, and chat with the AI!

---

🔌 Flashing the ESP32 Firmware

Before powering on the robot, flash the C++ firmware to the ESP32 boards using the Arduino IDE.

1. Open Robot/sketch_jan13a/sketch_jan13a.ino (Main Chassis).
2. Ensure you have the ESP32 Board Manager installed.
3. Update the WiFi configuration at the top of the file:

   const char *ssid     = "YOUR_WIFI_SSID";
   const char *password = "YOUR_WIFI_PASSWORD";
   const char *serverIP = "YOUR_LAPTOP_IP"; // Must match CONTROL_NODE_IP

4. Compile and Upload to your primary ESP32.
5. Repeat the process for the Robot/arm_controller/arm_controller.ino sketch.

---

🛡️ Security Note

Warning

ROVA is designed to run on a trusted, secure local network (LAN). The web interface and TCP sockets currently do not implement strict authentication. Do not port-forward ROVA to the public internet without implementing a reverse proxy (like Nginx) with SSL and Basic Authentication.

---

👥 The Team

Developed at Takshashila Academy, Kathmandu, Nepal.

• Team Leader: Sushant (Surya) Bhandari
• Members: Neema Sherpa, Uttam Shrestha, Nishant Bhandari, Prithivi Raiymajhi, Subash Shrestha
• Principal: Mrs. Netra Neupane

---

📜 License

This project is open-source and available under the MIT License.