MrHuang

MrHuang is a self-hosted AI companion that runs entirely on owned hardware — a local open-weight model on a self-hosted GPU, with no cloud LLM and no fallback, so a conversation never leaves the box and never bills per token. Every message is stored locally on self-hosted infrastructure, which turns day-to-day chat into a persistent, private knowledge base — the opposite of a stateless chat window that forgets everything between sessions.

It didn't start there. MrHuang began as a TypeScript/Express service that ran its agent loop against a hosted model through a third-party Agent SDK. The current build is the result of a full backend rewrite to Python and a brain transplant onto self-hosted inference — both shipped, both live, both covered below.

Architecture

A message arrives over WhatsApp (via the Meta Cloud API) or the web UI, passes through a gateway that authenticates and rate-limits it, gets assembled into a token-budgeted prompt, and is run through an in-house agent loop against a local open-weight model. Responses come back; the full exchange is persisted to a local SQLite store.

The brain is a self-hosted open-weight Mixture-of-Experts model (a ~35B model with ~3B active params, FP8 weights, large native context) served on vLLM over its OpenAI-compatible API. Inference is 100% local — there is no cloud LLM and no fallback path. Chain-of-thought reasoning is on by default on every turn. The agent loop is in-house and stateless per turn, built on the OpenAI Python SDK pointed straight at the local model; tools run in-process rather than over a separate transport (the autonomous tool-loop itself stays gated — see Security).

flowchart TD
    WA["WhatsApp<br/>Meta Cloud API"] -->|webhook| GW
    UI["Web UI"] --> GW
    GW["Gateway<br/>signature verify · allowlist · rate-limit"]
    GW --> CA["ContextAssembler<br/>markdown KB → token-budgeted prompt<br/>hot-reloadable persona"]
    CA -.->|semantic search| EMB["Embeddings sidecar<br/>local · vLLM"]
    CA --> AL["Agent loop<br/>in-house · in-process tools<br/>concurrency · timeouts · backoff · streaming"]
    AL -->|OpenAI-compatible| LLM["Open-weight MoE<br/>self-hosted GPU · vLLM"]
    AL --> DB[("SQLite<br/>local history + knowledge base")]
    AL -.->|response| WA
    AL -.->|response| UI

Agent loop — an in-house, stateless per-turn step loop that talks to the local model over the OpenAI-compatible API. It owns the production concerns the model server leaves to the caller: concurrency is bounded, soft and hard timeouts cap latency, backoff absorbs transient failures, and responses stream rather than blocking on the full completion. Tools run in-process (the email and calendar tools live in an in-process registry rather than over a separate tool-server transport — MCP was dropped); the autonomous tool-loop itself stays gated until the sandbox ships (see Security).

ContextAssembler builds each system prompt from a markdown knowledge base, fitting the most relevant context into a fixed token budget. Retrieval is single-vector semantic search backed by a self-hosted embeddings model — a small embedder running on the same vLLM box, so nothing about the user's context leaves the hardware to be retrieved. The personality spec is hot-reloadable, so behaviour can be tuned without a redeploy.

Durable job spine — the proactive and scheduled side of MrHuang runs on an own-built, crash-safe job spine: a scheduler, a worker, and an idempotent delivery ledger with leases and retries that survive a crash or a redeploy. It replaced the old cron-based jobs and is now the substrate every scheduled task rides on. The same spine enforces a fail-closed permission ceiling on every action (see Security).

Here is the full lifecycle of a single message:

sequenceDiagram
    actor U as User · WhatsApp / Web
    participant G as Gateway
    participant A as ContextAssembler
    participant E as Agent loop
    participant M as Open-weight model · vLLM
    participant D as SQLite
    U->>G: message + webhook signature
    G->>G: verify · allowlist · rate-limit
    G->>A: authenticated message
    A->>A: assemble token-budgeted prompt
    A->>E: prompt
    E->>M: run turn (OpenAI-compatible, local)
    M-->>E: reasoning + response
    E-->>U: response
    E->>D: persist exchange

How it got here

The original design ran a TypeScript/Express backend whose agent loop called a hosted model through a third-party Agent SDK. Two things changed, and both are now the live system:

The brain transplant — the move from a hosted model to self-hosted inference — has landed. A local open-weight MoE on vLLM is the only brain; there is no cloud dependency, no auth, no billing, no break-glass fallback. A second vLLM instance serves the retrieval embeddings on the same GPU. A self-hosted speech-to-text sidecar also runs on that GPU — today it serves desktop dictation; MrHuang's own in-app voice notes still transcribe through a cloud service, with the local swap queued (see What's Next). The brain itself — the part the previous version of this page filed under "What's Next" — is now production.

The backend rewrite — TypeScript/Express to Python 3.13 / FastAPI with the in-house agent loop — was a parity-verified, one-way cutover. The third-party Agent SDK and the separate tool transport are gone; tools run in-process. The cutover was independently peer-reviewed (8.1/10) against its headline parity claims, and main is kept in sync with production. The old TypeScript tree is retired.

Security

A personal assistant reachable from the public internet is an attractive target, so the trust boundary is treated as hostile:

• Webhook authentication — inbound messages are cryptographically signature-verified against the raw request body and checked against a trusted-sender allowlist before any work happens.
• Prompt-injection defense — untrusted message content is never allowed to override system instructions or escalate what an action is permitted to do.
• Tiered permission ceiling — every side effect runs through a three-tier egress ladder (auto / gated-and-logged / human-approved), and the spine enforces a fail-closed ceiling bounded by the role and channel that produced the job. An action that needs more authority than its origin grants is refused, not downgraded silently.
• Tool-boundary jail (designed, not yet shipped) — the autonomous tool-loop is hard-refused in code today. The defense it's gated behind is an OS-level jail at the tool boundary — realpath and root containment, symlink-escape rejection, and a network-restricted exec container — so a jailbroken prompt that emits a write to a system path is rejected before any byte hits disk. It is designed and specified, and the loop stays refused until it ships.
• Zero-trust networking — there is no open inbound port. Ingress is via a Cloudflare Tunnel; the box itself is only reachable over a Tailscale mesh, and operator access goes the same way.

These controls compose into one hostile-by-default path: nothing acts until it has cleared the edge, the gateway, and the permission ceiling.

flowchart LR
    A["Inbound message<br/>untrusted"]
    A -->|"Cloudflare Tunnel<br/>no open inbound port"| GW
    subgraph trust["Trust boundary (treated as hostile)"]
        direction TB
        GW["Gateway<br/>signature verify · allowlist · rate-limit"]
        AG["Agent<br/>prompt-injection defense"]
        CEIL["Permission ceiling<br/>three-tier ladder · fail-closed"]
        JAIL["Tool jail<br/>designed · loop refused until shipped"]
        GW --> AG --> CEIL --> JAIL
    end
    OP["Operator"] -.->|Tailscale mesh| GW

Deployment

MrHuang ships as Docker containers on self-hosted infrastructure and runs in daily production. Updates land through an automated, health-gated deployment pipeline that keeps production current on a known-good build without manual intervention. Rollback is a revert-and-rebuild, with a pre-cutover data snapshot as the safety net.

A full self-hosted observability stack rides alongside the app, deployed out of band from its deploy job: metrics collection, dashboards provisioned as code, alerting with push notifications, uptime monitoring, and a GPU metrics exporter for the inference box. Keeping it all on owned hardware is the point: the data, the model, and the uptime are under direct control. Test coverage is large and green — thousands of backend tests plus the retained regression suite from the old stack.

What's Next

The brain transplant is done, so the frontier moves. With inference fully self-hosted and the durable spine load-bearing, the next milestones turn MrHuang from a text assistant into an always-available home voice assistant and, eventually, a safe autonomous actor. Everything below is direction — specified, partly de-risked, but not shipped.

The honest hard part is no longer the plumbing. Self-hosted speech-to-text already runs on the box (it serves desktop dictation today; wiring it into the assistant's in-app voice notes, which still transcribe through a cloud service, is the near-term swap). What's hard is making spoken interaction feel like a presence rather than a command line, and unrefusing autonomy without giving a jailbroken prompt the keys.

Voice round-trip — Epic P1 ~L

Make spoken input and output round-trip entirely on owned hardware. Stories: stand up a self-hosted text-to-speech sidecar on the GPU; add a voice-ingress path and a separate mic daemon so a wake word and local transcription happen at the edge; finish swapping in-app voice notes onto the self-hosted STT sidecar so no audio reaches a cloud service. The MrHuang Listener hardware mic plugs in here as the embedded counterpart to the desktop mic daemon — both feed the same self-hosted speech-to-text layer.

Proactive wake-up — Epic P1 ~M

Let MrHuang reach out unprompted instead of only answering. Stories: a scheduled morning check-in that rides the durable spine and the same text-to-speech effector; a crisis-guard so a proactive message can never make a bad moment worse. The spine already survives crashes and redeploys, which is what makes an unattended scheduled action safe to trust.

Sandboxed agency — Epic P2 ~L · gated on the tool jail

Unrefuse the agent tool-loop — but only inside a real sandbox. Stories: ship the OS-level jail (path and root containment, symlink-escape rejection, a network-restricted exec container); keep the already-shipped tiered permission ceiling in front of it; enforce that WhatsApp and voice channels may only ever act inside the sandbox, never on the host. The tool-loop stays hard-refused in code until the jail is real.

The load-bearing constraint across all three is the same one that made the brain transplant safe: a side effect should never be able to exceed the authority of the channel that asked for it — whether that side effect is a spoken reminder, an unprompted morning message, or a file write.