Local AI Worker

Local inference worker for running GGUF models with llama.cpp and exposing the worker API remotely through Cloudflare Tunnel.

The worker runs entirely on a local machine. The public endpoint is provided by Cloudflare Tunnel, while the model server remains accessible only from the local worker.

Architecture

Client
  │
  │ HTTPS
  ▼
llama.codetlab.com
  │
  ▼
Cloudflare
  │
  │ Tunnel
  ▼
Worker API :3000
  │
  │ localhost
  ▼
llama.cpp :8080
  │
  ▼
Local GGUF Model

The public client communicates only with the Worker API.

llama.cpp is not exposed directly to the Internet.

---

Components

Worker API

TypeScript/Node.js API responsible for receiving requests and delegating inference to the local llama.cpp server.

Local endpoint:

http://localhost:3000

Main endpoint:

POST /v1/chat

Example request:

{
  "message": "Hola, ¿qué podés hacer?"
}

Example response:

{
  "answer": "..."
}

The Worker API communicates with llama.cpp through its local HTTP API.

---

llama.cpp

llama.cpp provides the local inference server for the GGUF model.

Current model:

qwen2.5-coder-3b.gguf

The model server runs locally on port 8080:

http://localhost:8080

It is started with:

~/.llama-app/llama serve \
  --model ~/models/qwen2.5-coder-3b.gguf \
  --host 127.0.0.1 \
  --port 8080

The OpenAI-compatible chat endpoint is:

http://localhost:8080/v1/chat/completions

The Worker API is responsible for communicating with this endpoint.

---

Cloudflare Tunnel

The worker is exposed through Cloudflare Tunnel instead of directly exposing the machine's public IP or opening the model server to the Internet.

Public endpoint:

https://llama.codetlab.com

Routing:

llama.codetlab.com
        │
        ▼
Cloudflare
        │
        ▼
Cloudflare Tunnel
        │
        ▼
http://localhost:3000

The tunnel connects outbound from the worker to Cloudflare.

This means the local machine does not need to expose llama.cpp:8080 publicly.

---

Local Startup

The development server can start both the Worker API and llama.cpp.

npm run dev

Conceptually:

npm run dev
      │
      ├── Worker API
      │      └── :3000
      │
      └── llama.cpp
             └── :8080

The Worker API communicates with:

http://127.0.0.1:8080

---

Environment

The Worker API uses:

PORT=3000
LLAMA_URL=http://127.0.0.1:8080

Example .env:

PORT=3000
LLAMA_URL=http://127.0.0.1:8080

Environment files containing secrets should not be committed.

---

API

Health

GET /health

Example:

curl http://localhost:3000/health

Expected response:

{
  "status": "ok"
}

---

Chat

POST /v1/chat

Request:

{
  "message": "Explicame qué es un puntero en C++"
}

Example:

curl -X POST http://localhost:3000/v1/chat \
  -H "Content-Type: application/json" \
  -d '{"message":"Explicame qué es un puntero en C++"}'

The request flow is:

Client
  │
  ▼
POST /v1/chat
  │
  ▼
Worker API
  │
  ▼
llama.cpp
  │
  ▼
GGUF Model
  │
  ▼
Worker API
  │
  ▼
Client

---

Public Request

Once the Cloudflare Tunnel is running, the same API is available through:

https://llama.codetlab.com/v1/chat

Example:

curl -X POST https://llama.codetlab.com/v1/chat \
  -H "Content-Type: application/json" \
  -d '{"message":"Hola"}'

The public request does not communicate directly with llama.cpp.

---

Worker Availability

The worker is physically hosted on the local machine.

PC ON
  │
  ▼
cloudflared
  │
  ▼
Cloudflare
  │
  ▼
llama.codetlab.com
  │
  ▼
Worker API
  │
  ▼
llama.cpp

When the PC is offline:

PC OFF
  │
  X
cloudflared unavailable

The public hostname remains configured, but the worker is unavailable.

When the PC is powered on again and cloudflared reconnects:

PC ON
  │
  ▼
cloudflared
  │
  ▼
Cloudflare
  │
  ▼
llama.codetlab.com
  │
  ▼
Worker API

No DNS change is required when the local worker goes offline and comes back online.

---

Security Boundary

The intended network boundary is:

                    PUBLIC
                       │
                       ▼
             llama.codetlab.com
                       │
                       ▼
                Cloudflare
                       │
                 Tunnel
                       │
                       ▼
              Worker API :3000
                       │
                 localhost
                       │
                       ▼
              llama.cpp :8080
                       │
                       ▼
                  GGUF Model

llama.cpp should remain bound to localhost unless there is a specific reason to expose it to another trusted machine.

The public API should be responsible for:

• request validation
• authentication/authorization
• rate limiting
• request IDs
• timeouts
• worker health
• error handling

---

Worker Model

This machine is the first worker in a larger inference architecture.

Current state:

                 API
                  │
                  ▼
             Worker 01
                  │
                  ▼
              llama.cpp
                  │
                  ▼
          qwen2.5-coder-3b

Future architecture:

                    API
                     │
                Worker Router
                     │
          ┌──────────┼──────────┐
          ▼          ▼          ▼
      Worker 01  Worker 02  Worker 03
          │          │          │
      llama.cpp  llama.cpp  llama.cpp

Clients should communicate with the API rather than directly with individual workers.

Workers are responsible for model inference.

The router is responsible for selecting an available worker.

---

Current Status

Implemented

• Local llama.cpp inference
• GGUF model execution
• TypeScript Worker API
• HTTP communication between Worker API and llama.cpp
• /health endpoint
• /v1/chat endpoint
• Cloudflare Tunnel
• Public worker hostname
• Local model server isolated from public Internet

Planned

• Request validation
• Authentication
• Rate limiting
• Request IDs
• Conversation IDs
• Worker authentication
• Worker health checks
• Worker heartbeat
• Streaming responses
• Multiple inference workers
• Worker routing
• Load balancing
• Model/worker capability discovery

---

Design Principle

The client should know the API, not the workers.

Client
  │
  ▼
Public API
  │
  ▼
Worker Router
  │
  ├── Worker 01
  ├── Worker 02
  └── Worker 03