LifeOS — with voice

LifeOS is live and open source, 284 passing tests across 27 files, a voice- and photo-first personal health tracker I talk to or photograph a plate at, built on a provider-agnostic realtime voice layer that runs OpenAI Realtime or Gemini Live behind one UI, and a storage seam that lets the identical codebase run as a self-hosted single-user app or a multi-tenant hosted product.

role

Designer & engineer

when

2025 – present

stack

Next.js 16, OpenAI Realtime, Gemini Live, Claude vision, Notion/Supabase, PWA

status

Live & open source

What it is

I wanted daily nutrition and health tracking (meals, cycle, wellness, skincare, fitness) without the friction of typing into a food-logging app after every meal, which is the thing that kills adherence to nutrition trackers in the first place. Existing tools make you type or scan a barcode; none combine voice-first logging, photo-based portion estimation, and cycle-aware guidance in one private, self-hostable system. So I built it for my own daily use first, then generalized it into an OSS project other people can self-host or use hosted.

Open the PWA, installed on phone or desktop, and either speak ("I had rice and dal for lunch") or photograph a plate. A review card lets you confirm or edit the AI's estimate before anything gets logged, with instant-log and log-again shortcuts for repeat meals. It's been in daily use for four months of continuous development — 179 commits on the private repo, spanning March to July 2026.

These are unstaged captures from the instance I use every day — the trends page is honest about a patchy logging streak, which is itself the design problem the voice-first input exists to solve.

How it's built

Voice: two providers, one schema. The app supports OpenAI Realtime and Google Gemini Live behind one UI and one tool-calling surface. The hard part isn't picking a provider, it's that their function-calling schemas are incompatible at the type level. OpenAI expects JSON Schema types in lowercase; Gemini Live expects the same tool definitions with uppercase type names (OBJECT, STRING, NUMBER). LifeOS keeps one canonical lowercase schema and per-provider adapters that transform it, a recursive uppercase-types walk for Gemini, a straight wrap for OpenAI, so the integration guide and the actual tool logic never diverge. Gemini Live also needed infrastructure OpenAI Realtime doesn't: Vercel and Netlify can't hold a persistent WebSocket, so a small Node proxy sits between the browser and Google, holding the API key server-side and relaying raw audio frames.

Vision: portion estimation with an honest confidence field. Photos go through Claude vision, which returns structured JSON (food identification, portion description, estimated serving weight in grams, full macro/micronutrient breakdown) and, deliberately, a confidence rating of high/medium/low rather than pretending every estimate is exact. Menu photos get handled separately from food photos, with a best-macro-choice recommendation layered on top.

Storage: one codebase, two tenancy models. A StorageAdapter interface is implemented by both a Notion adapter and a Supabase adapter. The backend is selected purely by environment — a Supabase URL present means hosted multi-tenant with billing; absent means self-hosted single-tenant with bring-your-own LLM keys. Route handlers stay thin and call the resolved adapter, so the same API routes serve both deployment modes without branching per route. It's backed by 284 passing tests across 27 files.

The bug where it talked to itself

Building the Gemini Live voice integration surfaced a bug the integration guide calls "our biggest bug": a self-inflicted audio feedback loop. Without gating microphone input, the sequence was — you speak, Gemini responds through the device speakers, the open microphone picks its own voice back up, Gemini receives its own audio as new "user" input, and starts responding to itself. The symptom was double responses and phantom food items appearing in the log from a conversation that was really the AI talking to itself.

The fix was a single boolean gate: track isSpeaking, flip it true when Gemini starts streaming audio out, flip it false when the turn completes, and simply don't forward mic audio to the WebSocket while it's true.

A smaller, related bug: the WebSocket proxy server forwards messages between the browser and Google, but the browser's initial "setup" message would sometimes arrive before the proxy's own connection to Google had finished its handshake, getting silently dropped. The fix was a queue — buffer any client messages that arrive early, then flush once the upstream socket reports open.

Open source

LifeOS is public at github.com/deepika-builds/life-os-oss. The same codebase runs two ways: self-hosted and single-tenant with your own LLM keys, or hosted and multi-tenant with billing, selected purely by whether a Supabase URL is present in the environment. The public snapshot currently trails the private repo while a few things finish: an identity migration (the app originally hardcoded one person's name, diet, and cuisine preference into the LLM prompts; voice paths now read from a per-user profile store, vision prompts still don't) and the unglamorous work of shipping OSS responsibly. The lesson from both bugs above generalizes: a realtime bidirectional integration finds every race condition and feedback path you didn't explicitly close off, and the fix is almost always a small, boring piece of state rather than a rewrite.