Skip to content

Implement voice-driven coding interface: Python orchestration layer + Swift TTS component - #3

Merged
tathagata merged 2 commits into
mainfrom
copilot/implement-voice-driven-coding-interface
Mar 1, 2026
Merged

tathagata merged 2 commits into
mainfrom
copilot/implement-voice-driven-coding-interface

Conversation

Copilot AI commented Mar 1, 2026 •

Copy link
Copy Markdown
Contributor

Adds the remaining core pieces of the voice-driven coding interface for blind developers: a Python orchestration layer that wires all existing modules into a single pipeline, and a macOS-native Swift TTS tool for spoken feedback.

Python Orchestration (src/orchestrator.py)

  • TTSBridge — Spawns the Swift TTS subprocess and pipes text via stdin. Raises TTSError for empty input or missing executable.
  • Orchestrator.run_once() — Full pipeline in one call:
    VoiceParser → ClaudeAPI → TestingGuardrails → TTSBridge
    
orchestrator = Orchestrator(voice_parser, claude_api, guardrails, tts_bridge)
result = orchestrator.run_once()
# result = {"command": "...", "code": "...", "summary": "..."}

Swift TTS (swift_tts/)

Swift Package Manager CLI targeting macOS 12+. Accepts text from stdin (Python pipe) or CLI args, speaks via AVSpeechSynthesizer, and blocks until speech completes via a DispatchSemaphore/delegate pattern.

echo "Your code passed all 3 tests." | swift_tts
# or: swift_tts "Your code passed all 3 tests."

Build with: swift build -c release inside swift_tts/.

Tests (tests/test_orchestrator.py)

14 unit tests covering TTSBridge edge cases (empty text, missing exe, subprocess errors) and Orchestrator call-through and error propagation — all using mocks, no real subprocess or API calls.

Original prompt

Implement a voice-driven coding interface harness utilizing Python for the Claude Agent SDK (claude-agent-sdk) and Swift for macOS-native Text-to-Speech (TTS) with AVSpeechSynthesizer. This tool is intended to empower blind developers with a natural, voice-operated interface for coding.

Key Features:

  1. Python Backend (Agent Brain + Glue):

    • Use Claude Agent SDK for interacting with Claude API to handle user prompts, generate code, and refine outputs.
    • Implement a "VoiceParser" class to capture and preprocess spoken user commands using Python (e.g., speech_recognition).
    • Build the orchestration layer to communicate results between the Claude API and the Swift TTS system.
    • Add testing guardrails using pytest to validate the correctness of Claude-generated code autonomously, with post-test refinements handled in Python.
  2. Swift TTS (Interaction Layer):

    • Create a lightweight Swift command-line tool for converting Python-piped text into high-quality macOS speech using AVSpeechSynthesizer.
    • Synchronize Python → Swift communication.
  3. System Workflow:

    • Input: User gives a voice command (e.g., "Create a function to fetch stock prices.").
    • Processing: Python captures the speech, sends the intent to Claude via its SDK, validates the result against generated pytest tests, and refines the code if necessary.
    • Output: Provide spoken feedback of the results (e.g., code summaries, test status) using the Swift TTS component via AVSpeechSynthesizer.
  4. Accessibility and Efficiency:

    • Ensure seamless speech-to-text and response feedback by leveraging macOS Accessibility Features.
    • Provide summaries and updates at each step in a concise, voice-friendly manner.
  5. Testing:

    • Write robust unit and integration tests for Python modules and validate Swift TTS functionality with mocked inputs for voice outputs.

This system should be modular, maintainable, and prioritize correctness and voice accessibility in all workflows.

This pull request was created from Copilot chat.


🔒 GitHub Advanced Security automatically protects Copilot coding agent pull requests. You can protect all pull requests by enabling Advanced Security for your repositories. Learn more about Advanced Security.

Co-authored-by: tathagata <68151+tathagata@users.noreply.github.com>
Copilot AI changed the title [WIP] Implement voice-driven coding interface using Python and Swift Implement voice-driven coding interface: Python orchestration layer + Swift TTS component Mar 1, 2026

@tathagata tathagata left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lgtm

@tathagata
tathagata marked this pull request as ready for review March 1, 2026 15:47
@tathagata
tathagata merged commit 1ad9815 into main Mar 1, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants