MrHuang Listener

MrHuang Listener is a hardware voice-input front end for MrHuang: an ESP32-S3 that captures speech from two microphones and streams it over WiFi to a server, which forwards it to the MrHuang backend for processing. It is a work-in-progress, prealpha learning project — the goal is to get spoken input into the existing assistant without typing, and most of the work so far is firmware bring-up and getting a clean audio path off the device.

Architecture

The device runs ESP-IDF firmware written in C on an ESP32-S3. Two INMP441 MEMS microphones are wired in a stereo I2S configuration; the firmware reads the I2S samples and streams them as raw PCM over WiFi, as UDP datagrams, to a server. On the host side, a Python asyncio UDP server accepts the stream, then hands the audio off to the MrHuang backend for AI processing. Opus encoding on the device — to cut the WiFi bitrate — is the planned next step.

 INMP441 ×2 (stereo I2S)
          │
          ▼
 ┌─────────────────────────┐
 │  ESP32-S3               │
 │  ESP-IDF (C)            │
 │  I2S capture → UDP/WiFi │
 └─────────────────────────┘
          │  raw PCM · UDP
          ▼
 ┌─────────────────────────┐
 │  Python UDP server      │
 │  receive · hand off     │
 └─────────────────────────┘
          │
          ▼
   MrHuang backend

The split keeps the embedded side narrow: the ESP32-S3 is responsible only for capturing the two I2S channels and getting them onto the network. Stream handling, buffering, and the handoff to the existing MrHuang assistant all live in the Python server, where they are easier to iterate on than on-device. It is the embedded voice-ingress front end for MrHuang's voice-first direction — the hardware counterpart to a desktop mic daemon, both built to feed the same self-hosted speech-to-text layer.

Status

This is prealpha and in active development. The firmware bring-up and the device-to-server audio path are the current focus; the integration with the MrHuang backend is the connection it is built toward rather than a finished pipeline, and the audio path itself is still stubbed. It is a way to learn ESP-IDF, I2S microphone capture, and UDP audio streaming in C while extending an assistant that already exists, so the writeup describes the intended shape, not a shipped product.