How to Integrate Gemini 3.1 Flash Live Into Your App — The Complete Guide

A no-BS guide based on actually building real-time two-way voice into a production health tracking app. Every mistake, dead end, and "aha" moment included.

What is Gemini 3.1 Flash Live?

Google's real-time, bidirectional audio AI model. You stream audio in, it streams audio back — like a phone call with AI. It does speech recognition, understanding, reasoning, and voice response ALL in one model. No separate STT → LLM → TTS pipeline needed.

Key specs:

• Audio in: raw PCM16, 16kHz, little-endian
• Audio out: raw PCM16, 24kHz, little-endian
• Protocol: WebSocket (persistent connection)
• API version: v1alpha (NOT v1beta — critical!)
• Model ID: gemini-3.1-flash-live-preview
• Supports: function calling, system instructions, input/output transcription, barge-in (interrupting the AI mid-sentence)

How is it different from regular Gemini?

Regular Gemini models (gemini-2.5-flash, etc.) support generateContent — you send text/images, get text back via REST API. The Live model ONLY supports bidiGenerateContent — bidirectional streaming over WebSocket. You can't use curl or a regular API call. This tripped us up early.

Step 0: Getting Your API Key (Google Cloud Console)

This is where most people get confused. Here's the exact flow:

1. Go to console.cloud.google.com
2. Create a project (or select existing one)
3. Go to APIs & Services → Library → search "Generative Language API" → Enable it
4. Go to APIs & Services → Credentials → Create Credentials → API Key
5. Click your new API key → under "API restrictions" select "Restrict key"
6. From the dropdown, select "Generative Language API" → Save
7. IMPORTANT: Go to Billing → make sure a payment method is linked. Without billing, WebSocket connections are blocked (REST still works, which is confusing)

How to verify your key works:

curl "https://generativelanguage.googleapis.com/v1beta/models?key=YOUR_KEY"

If you see the Live model in the list but WebSocket still fails with "API key not valid" — it's the billing. We spent hours on this.

The confusing part: Your key will work perfectly for REST API calls (generateContent) but fail on WebSocket (bidiGenerateContent) if billing isn't linked. There's no clear error message about billing — it just says "API key not valid."

Step 1: Understanding the Architecture

Browser (mic audio)
    → WebSocket → Your Proxy Server (Railway/Fly.io)
        → WebSocket → Google Gemini Live API
        ← WebSocket ← Audio response
    ← WebSocket ←
Browser (plays audio)

Why do you need a proxy server?

Three reasons we discovered the hard way:

1. API key security — You can't put your Google API key in browser JavaScript. Anyone can view source and steal it. The proxy holds the key server-side.
2. Hosting limitations — Vercel, Netlify, Cloudflare Pages — none of them support WebSocket connections. They're built for request/response, not persistent connections. You need an actual server.
3. Browser restrictions — We hit issues where the browser's WebSocket connection to Google was silently failing. Routing through our own proxy fixed this.

Where to host the proxy:

• Railway (what we used) — free tier, deploys in 2 minutes
• Fly.io — good free tier, global edge deployment
• Render — free tier with some sleep limitations

NOT Vercel/Netlify — they literally can't run WebSocket servers.

Step 2: Build the WebSocket Proxy

This is a ~40 line Node.js server. Its only job: relay messages between browser and Google.

const { WebSocketServer, WebSocket } = require("ws");

const PORT = process.env.PORT || 3001;
const API_KEY = process.env.GEMINI_API_KEY;

// v1alpha — NOT v1beta!
const GEMINI_URL = `wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1alpha.GenerativeService.BidiGenerateContent?key=${API_KEY}`;

const wss = new WebSocketServer({ port: PORT });

wss.on("connection", (clientWs) => {
  const geminiWs = new WebSocket(GEMINI_URL);
  let geminiReady = false;
  const buffer = []; // CRITICAL — buffer until Google connects

  geminiWs.on("open", () => {
    geminiReady = true;
    for (const msg of buffer) geminiWs.send(msg);
    buffer.length = 0;
  });

  clientWs.on("message", (data) => {
    const str = data.toString();
    if (geminiReady) geminiWs.send(str);
    else buffer.push(str);
  });

  geminiWs.on("message", (data) => {
    if (clientWs.readyState === WebSocket.OPEN)
      clientWs.send(data.toString());
  });

  clientWs.on("close", () => geminiWs.close());
  geminiWs.on("close", (code, reason) => {
    if (clientWs.readyState === WebSocket.OPEN)
      clientWs.close(code, reason?.toString());
  });
});

Why the message buffer matters

Without it:

1. Browser connects to your proxy → instant
2. Browser sends setup message → arrives at proxy
3. Proxy tries to forward to Google... but Google WebSocket isn't open yet (~200ms)
4. Setup message silently dropped
5. Connection hangs forever

Step 3: Capture Mic Audio in the Browser

Gemini needs raw PCM16 audio at 16kHz. Use an AudioWorklet:

const stream = await navigator.mediaDevices.getUserMedia({
  audio: { sampleRate: 16000, channelCount: 1, echoCancellation: true, noiseSuppression: true }
});

const audioContext = new AudioContext({ sampleRate: 16000 });
const source = audioContext.createMediaStreamSource(stream);

const workletCode = `
  class PCMProcessor extends AudioWorkletProcessor {
    process(inputs) {
      const ch = inputs[0]?.[0];
      if (ch) {
        const pcm = new Int16Array(ch.length);
        for (let i = 0; i < ch.length; i++) {
          const s = Math.max(-1, Math.min(1, ch[i]));
          pcm[i] = s < 0 ? s * 0x8000 : s * 0x7FFF;
        }
        this.port.postMessage(pcm.buffer, [pcm.buffer]);
      }
      return true;
    }
  }
  registerProcessor("pcm16", PCMProcessor);
`;

await audioContext.audioWorklet.addModule(URL.createObjectURL(new Blob([workletCode], { type: "application/javascript" })));
const worklet = new AudioWorkletNode(audioContext, "pcm16");
source.connect(worklet);

Why AudioWorklet, not MediaRecorder? MediaRecorder outputs WebM/Opus. Gemini needs raw PCM. No conversion possible.

Step 4: Connect to Gemini Live

const ws = new WebSocket("ws://localhost:3001"); // Your proxy

ws.onopen = () => {
  ws.send(JSON.stringify({
    setup: {  // "setup" NOT "config" — v1alpha specific
      model: "models/gemini-3.1-flash-live-preview",
      generationConfig: {
        responseModalities: ["AUDIO"],
        temperature: 0.3,
      },
      systemInstruction: {
        parts: [{ text: "You are a helpful assistant." }],
      },
      tools: [{ functionDeclarations: [/* your functions */] }],
    },
  }));
};

Setup message cheat sheet

• Wrapper: use setup: {}, not config: {} — wrong key throws "Unknown name 'config'"
• API version: use v1alpha, not v1beta — wrong version throws "API key not valid"
• Speech config: don't include speechConfig: {} — it throws "Unknown name"

Step 5: Stream Audio (with mic muting!)

let isSpeaking = false; // CRITICAL

worklet.port.onmessage = (e) => {
  if (ws.readyState === WebSocket.OPEN && !isSpeaking) {
    const pcm = new Uint8Array(e.data);
    let binary = "";
    for (let i = 0; i < pcm.length; i++) binary += String.fromCharCode(pcm[i]);
    ws.send(JSON.stringify({
      realtimeInput: {
        audio: { data: btoa(binary), mimeType: "audio/pcm;rate=16000" }
      }
    }));
  }
};

The isSpeaking check is the most important line. Without it, Gemini hears itself through your speakers, thinks you're speaking, and responds to its own voice. Double responses, phantom inputs, AI talking in circles.

Step 6: Handle Responses

ws.onmessage = (event) => {
  const data = JSON.parse(event.data);
  if (data.setupComplete) { /* ready */ return; }
  if (data.serverContent?.inputTranscription?.text) { /* what user said */ }
  if (data.serverContent?.outputTranscription?.text) { /* what Gemini said */ }
  if (data.serverContent?.modelTurn?.parts) {
    isSpeaking = true; // MUTE MIC
    // play audio chunks
  }
  if (data.serverContent?.turnComplete) {
    isSpeaking = false; // UNMUTE MIC
  }
  if (data.toolCall?.functionCalls) { /* handle function calls */ }
};

Step 7: Play Audio Response

Gemini sends base64 PCM16 at 24kHz. Queue chunks and play sequentially:

async function playChunk(base64) {
  const ctx = new AudioContext({ sampleRate: 24000 });
  const binary = atob(base64);
  const pcm16 = new Int16Array(binary.length / 2);
  for (let i = 0; i < pcm16.length; i++)
    pcm16[i] = binary.charCodeAt(i*2) | (binary.charCodeAt(i*2+1) << 8);
  const float32 = new Float32Array(pcm16.length);
  for (let i = 0; i < pcm16.length; i++) float32[i] = pcm16[i] / 32768;
  const buffer = ctx.createBuffer(1, float32.length, 24000);
  buffer.copyToChannel(float32, 0);
  const source = ctx.createBufferSource();
  source.buffer = buffer;
  source.connect(ctx.destination);
  source.start();
  await new Promise(r => { source.onended = r; });
}

Queue is essential. Playing chunks simultaneously = garbled audio.

Step 8: Deploy

App → Vercel/Netlify (any static host)

Proxy → Railway (easiest):

npm install -g @railway/cli
railway login && railway init
railway variables set GEMINI_API_KEY=your_key
railway up && railway domain

Comparisons

Gemini Live

• Best for: all-in-one voice
• Two-way audio: yes
• Function calling: built-in
• Hinglish: struggles
• Free tier: yes

Nova Sonic (AWS)

• Best for: AWS ecosystem
• Two-way audio: yes
• Function calling: limited
• Hinglish: similar struggles
• Free tier: no

Sarvam AI

• Best for: Indian languages
• Two-way audio: needs separate TTS
• Function calling: no
• Hinglish: excellent
• Free tier: limited

Cerebras — fastest text gen (~2000 tok/sec) but no real-time voice.

One-Way vs Two-Way Voice

Two-way: Speak → AI talks back → conversation → confirm. Natural but slower, more expensive.

One-way: Speak → transcribe → show on screen → tap confirm. Faster, cheaper, more reliable.

For simple data entry, one-way is honestly better. Two-way shines for conversations with corrections and follow-ups.

Every Gotcha (Save Yourself Hours)

1. "API key not valid" — Use v1alpha, enable billing. (2 hrs wasted)
2. SDK crashes bundler — Skip @google/genai, use raw WebSocket. (1 hr)
3. Double responses — Mute mic while AI speaks. (1 hr)
4. Setup message dropped — Buffer in proxy until Google connects. (30 min)
5. "Unknown name 'config'" — Use setup not config. (30 min)
6. "Unknown name 'speechConfig'" — Don't include it, not supported. (20 min)
7. AI says "logged" but doesn't — Tell it "only call the function, don't say you did it". (15 min)
8. Missing API fields — Fill defaults for fields Gemini doesn't return. (15 min)

The Vibe-Coding Disclaimer

Full transparency: this was vibe-coded. I described what I wanted to Claude Code, and it built everything. I directed design decisions, picked the UI, and debugged alongside AI — but didn't write the code myself. Think architect + AI builder.

The voice feature took ~15 iterations and multiple dead ends before landing on the raw WebSocket approach.

Cost & Token Usage

Gemini Live pricing is based on audio tokens:

• ~1 token per 0.04 seconds of audio
• Both input (your speech) and output (Gemini's response) count
• A 30-second conversation ≈ 750 input tokens + 750 output tokens

Check usage at: Google Cloud Console → APIs & Services → Generative Language API → Metrics

For casual personal use, it's negligible. For production with many users, monitor and set billing alerts.

Built with Gemini 3.1 Flash Live + Next.js + Railway.