Google

Speech

The Google Voice implementation in Mastra provides both text-to-speech (TTS) and speech-to-text (STT) capabilities using Google Cloud services. It supports multiple voices, languages, advanced audio configuration options, and both standard API key authentication and Vertex AI mode for enterprise deployments.

Usage example

import { GoogleVoice } from '@mastra/voice-google'

// Initialize with default configuration (uses GOOGLE_API_KEY environment variable)
const voice = new GoogleVoice()

// Text-to-Speech (plain text)
const audioStream = await voice.speak('Hello, world!', {
  languageCode: 'en-US',
  audioConfig: {
    audioEncoding: 'LINEAR16',
  },
})

// Text-to-Speech with SSML
const ssmlStream = await voice.speak('ignored', {
  input: {
    ssml: '<speak>Take <say-as interpret-as="unit">5 mg</say-as> daily.</speak>',
  },
})

// Text-to-Speech with Gemini-TTS model
const geminiStream = await voice.speak('Hello from Gemini TTS!', {
  voice: { name: 'Kore', modelName: 'gemini-2.5-flash-preview-tts' },
  input: { prompt: 'Warm, calm tone.' },
})

// Speech-to-Text
const transcript = await voice.listen(audioStream, {
  config: {
    encoding: 'LINEAR16',
    languageCode: 'en-US',
  },
})

// Get available voices for a specific language
const voices = await voice.getSpeakers({ languageCode: 'en-US' })

Constructor parameters

speechModel?:

GoogleModelConfig

= { apiKey: process.env.GOOGLE_API_KEY }

Configuration for text-to-speech functionality

GoogleModelConfig

apiKey?:

string

Google Cloud API key. Falls back to GOOGLE_API_KEY environment variable. Not used when vertexAI is true.

keyFilename?:

string

Path to service account JSON key file. Falls back to GOOGLE_APPLICATION_CREDENTIALS environment variable.

credentials?:

object

In-memory service account credentials object with client_email and private_key properties.

listeningModel?:

GoogleModelConfig

= { apiKey: process.env.GOOGLE_API_KEY }

Configuration for speech-to-text functionality

GoogleModelConfig

apiKey?:

string

Google Cloud API key. Falls back to GOOGLE_API_KEY environment variable. Not used when vertexAI is true.

keyFilename?:

string

Path to service account JSON key file. Falls back to GOOGLE_APPLICATION_CREDENTIALS environment variable.

credentials?:

object

In-memory service account credentials object with client_email and private_key properties.

speaker?:

string

= 'en-US-Casual-K'

Default voice ID to use for text-to-speech

vertexAI?:

boolean

= false

Enable Vertex AI mode for enterprise deployments. Uses project-based authentication instead of API keys. Requires 'project' to be set.

project?:

string

Google Cloud project ID (required when vertexAI is true). Falls back to GOOGLE_CLOUD_PROJECT environment variable.

location?:

string

= 'us-central1'

Google Cloud region for Vertex AI. Falls back to GOOGLE_CLOUD_LOCATION environment variable.

Methods

speak()

Converts text to speech using Google Cloud Text-to-Speech service.

input:

string | NodeJS.ReadableStream

Text to convert to speech. If a stream is provided, it will be converted to text first.

options?:

object

Speech synthesis options

GoogleSpeakOptions

speaker?:

string

Voice ID to use for this request.

languageCode?:

string

Language code for the voice (e.g., 'en-US'). Defaults to the language code derived from the speaker ID, or 'en-US'.

input?:

ISynthesizeSpeechRequest['input']

Rich input object passed through to the Google Cloud TTS API. Supports ssml, markup, prompt (Gemini-TTS style steering), customPronunciations, and multiSpeakerMarkup. When provided without text, ssml, markup, or multiSpeakerMarkup, the positional input argument is used as the text field automatically.

voice?:

ISynthesizeSpeechRequest['voice']

Voice configuration merged on top of defaults (name and languageCode). Supports modelName (e.g., 'gemini-2.5-flash-preview-tts') and multiSpeakerVoiceConfig.

audioConfig?:

ISynthesizeSpeechRequest['audioConfig']

Audio configuration options from Google Cloud Text-to-Speech API.

Returns: Promise<NodeJS.ReadableStream>

listen()

Converts speech to text using Google Cloud Speech-to-Text service. Supports both v1 (default) and v2 APIs. The v2 API adds support for AAC-in-MP4 audio (iOS Safari) via auto-decoding.

v1 (default)

audioStream:

NodeJS.ReadableStream

Audio stream to transcribe

options?:

GoogleListenOptionsV1

v1 recognition options

GoogleListenOptionsV1

config?:

IRecognitionConfig

v1 recognition configuration from Google Cloud Speech-to-Text API

v2

Pass v2: true to use the Cloud Speech-to-Text v2 API, which supports additional audio formats like AAC-in-MP4 (iOS Safari).

const transcript = await voice.listen(iosSafariAacStream, {
  v2: true,
  config: {
    autoDecodingConfig: {},
  },
})

audioStream:

NodeJS.ReadableStream

Audio stream to transcribe

options:

GoogleListenOptionsV2

v2 recognition options

GoogleListenOptionsV2

v2:

true

Enables the v2 API path

config?:

v2.IRecognitionConfig

v2 recognition configuration. Defaults to auto-decoding with languageCodes: ['en-US'] and model: 'long'. Set autoDecodingConfig: {} to auto-detect the audio format, or use explicitDecodingConfig to specify an encoding like MP4_AAC, M4A_AAC, or MOV_AAC.

recognizer?:

string

v2 recognizer resource path. Defaults to projects/{project}/locations/global/recognizers/_ where {project} is resolved from the constructor project option, GOOGLE_CLOUD_PROJECT, or the client's default project.

Returns: Promise<string>

getSpeakers()

Returns an array of available voice options, where each node contains:

voiceId:

string

Unique identifier for the voice

languageCodes:

string[]

List of language codes supported by this voice

isUsingVertexAI()

Checks if Vertex AI mode is enabled.

Returns: boolean - true if using Vertex AI, false otherwise

getProject()

Gets the configured Google Cloud project ID.

Returns: string | undefined - The project ID or undefined if not set

getLocation()

Gets the configured Google Cloud location/region.

Returns: string - The location (default: 'us-central1')

Authentication

The Google Voice provider supports two authentication methods:

Standard Mode (API Key)

Uses a Google Cloud API key for authentication. Suitable for development and basic use cases.

// Using environment variable (GOOGLE_API_KEY)
const voice = new GoogleVoice()

// Using explicit API key
const voice = new GoogleVoice({
  speechModel: { apiKey: 'your-api-key' },
  listeningModel: { apiKey: 'your-api-key' },
  speaker: 'en-US-Casual-K',
})

Vertex AI Mode (Service Account)

Uses Google Cloud project-based authentication with service accounts. Recommended for production and enterprise deployments.

Benefits:

• Better security (no API keys in code)
• IAM-based access control
• Project-level billing and quotas
• Audit logging
• Enterprise features

Configuration Options:

// Using Application Default Credentials (ADC)
// Set GOOGLE_APPLICATION_CREDENTIALS and GOOGLE_CLOUD_PROJECT env vars
const voice = new GoogleVoice({
  vertexAI: true,
  project: 'your-gcp-project',
  location: 'us-central1', // Optional, defaults to 'us-central1'
})

// Using service account key file
const voice = new GoogleVoice({
  vertexAI: true,
  project: 'your-gcp-project',
  speechModel: {
    keyFilename: '/path/to/service-account.json',
  },
  listeningModel: {
    keyFilename: '/path/to/service-account.json',
  },
})

// Using in-memory credentials
const voice = new GoogleVoice({
  vertexAI: true,
  project: 'your-gcp-project',
  speechModel: {
    credentials: {
      client_email: 'service-account@project.iam.gserviceaccount.com',
      private_key: '-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----',
    },
  },
})

Required Permissions

IAM Roles

For Text-to-Speech:

• roles/texttospeech.admin - Text-to-Speech Admin (full access)
• roles/texttospeech.editor - Text-to-Speech Editor (create and manage)
• roles/texttospeech.viewer - Text-to-Speech Viewer (read-only)

For Speech-to-Text:

• roles/speech.client - Speech-to-Text Client

OAuth Scopes

For synchronous Text-to-Speech synthesis:

• https://www.googleapis.com/auth/cloud-platform - Full access to Google Cloud Platform services

For long-audio Text-to-Speech operations:

• locations.longAudioSynthesize - Create long-audio synthesis operations
• operations.get - Get operation status
• operations.list - List operations

Important notes

1. Authentication: Either a Google Cloud API key (standard mode) or service account credentials (Vertex AI mode) is required.
2. Environment Variables:
   • GOOGLE_API_KEY - API key for standard mode
   • GOOGLE_CLOUD_PROJECT - Project ID for Vertex AI mode
   • GOOGLE_CLOUD_LOCATION - Location for Vertex AI mode (defaults to 'us-central1')
   • GOOGLE_APPLICATION_CREDENTIALS - Path to service account key file
3. The default voice is set to 'en-US-Casual-K'.
4. Both text-to-speech and speech-to-text services use LINEAR16 as the default audio encoding.
5. The speak() method supports advanced audio configuration through the Google Cloud Text-to-Speech API.
6. The listen() method supports various recognition configurations through the Google Cloud Speech-to-Text API.
7. Available voices can be filtered by language code using the getSpeakers() method.
8. Vertex AI mode provides enterprise features including IAM control, audit logs, and project-level billing.

Gemini Live

The GeminiLiveVoice class provides real-time voice interaction capabilities using Google's Gemini Live API. It supports bidirectional audio streaming, tool calling, session management, and both standard Google API and Vertex AI authentication methods.

Usage example

import { GeminiLiveVoice } from '@mastra/voice-google-gemini-live'
import { playAudio, getMicrophoneStream } from '@mastra/node-audio'

// Initialize with Gemini API (using API key)
const voice = new GeminiLiveVoice({
  apiKey: process.env.GOOGLE_API_KEY, // Required for Gemini API
  model: 'gemini-2.0-flash-exp',
  speaker: 'Puck', // Default voice
  debug: true,
})

// Or initialize with Vertex AI (using OAuth)
const voiceWithVertexAI = new GeminiLiveVoice({
  vertexAI: true,
  project: 'your-gcp-project',
  location: 'us-central1',
  serviceAccountKeyFile: '/path/to/service-account.json',
  model: 'gemini-2.0-flash-exp',
  speaker: 'Puck',
})

// Or use the VoiceConfig pattern (recommended for consistency with other providers)
const voiceWithConfig = new GeminiLiveVoice({
  speechModel: {
    name: 'gemini-2.0-flash-exp',
    apiKey: process.env.GOOGLE_API_KEY,
  },
  speaker: 'Puck',
  realtimeConfig: {
    model: 'gemini-2.0-flash-exp',
    apiKey: process.env.GOOGLE_API_KEY,
    options: {
      debug: true,
      sessionConfig: {
        interrupts: { enabled: true },
      },
    },
  },
})

// Establish connection (required before using other methods)
await voice.connect()

// Set up event listeners
voice.on('speaker', audioStream => {
  // Handle audio stream (NodeJS.ReadableStream)
  playAudio(audioStream)
})

voice.on('writing', ({ text, role }) => {
  // Handle transcribed text
  console.log(`${role}: ${text}`)
})

voice.on('turnComplete', ({ timestamp }) => {
  // Handle turn completion
  console.log('Turn completed at:', timestamp)
})

// Convert text to speech
await voice.speak('Hello, how can I help you today?', {
  speaker: 'Charon', // Override default voice
  responseModalities: ['AUDIO', 'TEXT'],
})

// Process audio input
const microphoneStream = getMicrophoneStream()
await voice.send(microphoneStream)

// Update session configuration
await voice.updateSessionConfig({
  speaker: 'Kore',
  instructions: 'Be more concise in your responses',
})

// When done, disconnect
await voice.disconnect()
// Or use the synchronous wrapper
voice.close()

Configuration

Constructor options

apiKey?:

string

Google API key for Gemini API authentication. Required unless using Vertex AI.

model?:

GeminiVoiceModel

= 'gemini-2.0-flash-exp'

The model ID to use for real-time voice interactions.

speaker?:

GeminiVoiceName

= 'Puck'

Default voice ID for speech synthesis.

vertexAI?:

boolean

= false

Use Vertex AI instead of Gemini API for authentication.

project?:

string

Google Cloud project ID (required for Vertex AI).

location?:

string

= 'us-central1'

Google Cloud region for Vertex AI.

serviceAccountKeyFile?:

string

Path to service account JSON key file for Vertex AI authentication.

serviceAccountEmail?:

string

Service account email for impersonation (alternative to key file).

instructions?:

string

System instructions for the model.

sessionConfig?:

GeminiSessionConfig

Session configuration including interrupt and context settings.

GeminiSessionConfig

interrupts?:

object

Interrupt handling configuration.

interrupts.enabled?:

boolean

Enable interrupt handling.

interrupts.allowUserInterruption?:

boolean

Allow user to interrupt model responses.

contextCompression?:

boolean

Enable automatic context compression.

inputAudioTranscription?:

GeminiAudioTranscriptionConfig

Transcription settings for the user's speech. Set languageCodes to BCP-47 hints (for example ['es-ES']) so short or accented utterances aren't transcribed in the wrong language. Without hints, the server auto-detects the language. Applied when the session connects.

GeminiAudioTranscriptionConfig

languageCodes?:

string[]

BCP-47 language codes present in the audio.

outputAudioTranscription?:

GeminiAudioTranscriptionConfig

Transcription settings for the model's speech. Takes the same languageCodes hints as inputAudioTranscription. Applied when the session connects.

debug?:

boolean

= false

Enable debug logging for troubleshooting.

Methods

connect()

Establishes a connection to the Gemini Live API. Must be called before using speak, listen, or send methods.

requestContext?:

object

Optional request context for the connection.

returns:

Promise<void>

Promise that resolves when the connection is established.

speak()

Converts text to speech and sends it to the model. Can accept either a string or a readable stream as input.

input:

string | NodeJS.ReadableStream

Text or text stream to convert to speech.

options?:

GeminiLiveVoiceOptions

Optional speech configuration.

GeminiLiveVoiceOptions

speaker?:

GeminiVoiceName

Voice ID to use for this specific speech request.

languageCode?:

string

Language code for the response.

responseModalities?:

('AUDIO' | 'TEXT')[]

Response modalities to receive from the model.

Returns: Promise<void> (responses are emitted via speaker and writing events)

sendContext()

Sends conversation history into the live session without triggering a model response. Use this to seed prior turns (e.g. from Mastra Memory) on a cold connect so the model has context before the user speaks.

await voice.sendContext([
  { role: 'user', content: 'What is the weather?' },
  { role: 'assistant', content: 'It is 72°F in San Francisco.' },
])

// Model stays silent until the user actually speaks.
await voice.send(micStream)

turns:

IncrementalTurn[]

Prior conversation turns to seed into the session. Each turn has a role ("user" or "assistant") and content string. Both roles are supported on newer models (e.g. gemini-2.5-flash-native-audio-preview-12-2025). Some older models only accept user-role turns.

options?:

object

Optional configuration.

object

turnComplete?:

boolean

Whether to mark the turn as complete and trigger a model response.

Returns: Promise<void>

listen()

Processes audio input for speech recognition. Takes a readable stream of audio data and returns the transcribed text.

audioStream:

NodeJS.ReadableStream

Audio stream to transcribe.

options?:

GeminiLiveVoiceOptions

Optional listening configuration.

Returns: Promise<string> - The transcribed text

send()

Streams audio data in real-time to the Gemini service for continuous audio streaming scenarios like live microphone input.

audioData:

NodeJS.ReadableStream | Int16Array

Audio stream or buffer to send to the service.

Returns: Promise<void>

updateSessionConfig()

Updates the session configuration at runtime. This can modify voice settings and speaker selection. It can also modify other runtime configurations.

config:

Partial<GeminiLiveVoiceConfig>

Configuration updates to apply.

Returns: Promise<void>

addTools()

Adds a set of tools to the voice instance. Tools allow the model to perform additional actions during conversations. When GeminiLiveVoice is added to an Agent, any tools configured for the Agent will automatically be available to the voice interface.

tools:

ToolsInput

Tools configuration to equip.

Returns: void

addInstructions()

Adds or updates system instructions for the model.

instructions?:

string

System instructions to set.

Returns: void

answer()

Triggers a response from the model. This method is primarily used internally when integrated with an Agent.

options?:

Record<string, unknown>

Optional parameters for the answer request.

Returns: Promise<void>

getSpeakers()

Returns a list of available voice speakers for the Gemini Live API.

Returns: Promise<Array<{ voiceId: string; description?: string }>>

disconnect()

Disconnects from the Gemini Live session and cleans up resources. This is the async method that properly handles cleanup.

Returns: Promise<void>

close()

Synchronous wrapper for disconnect(). Calls disconnect() internally without awaiting.

Returns: void

on()

Registers an event listener for voice events.

event:

string

Name of the event to listen for.

callback:

Function

Function to call when the event occurs.

Returns: void

off()

Removes a previously registered event listener.

event:

string

Name of the event to stop listening to.

callback:

Function

The specific callback function to remove.

Returns: void

Events

The GeminiLiveVoice class emits the following events:

speaker:

event

Emitted when audio data is received from the model. Callback receives a NodeJS.ReadableStream.

speaking:

event

Emitted with audio metadata. Callback receives { audioData?: Int16Array, sampleRate?: number }.

writing:

event

Emitted when transcribed text is available. Callback receives { text: string, role: 'assistant' | 'user' }. On native-audio models the assistant transcript is driven by the server's output_audio_transcription channel rather than modelTurn.parts.text.

thinking:

event

Emitted on native-audio models with the model's chain-of-thought / reasoning text from modelTurn.parts.text. Callback receives { text: string }. Does not fire on non-native-audio models, where modelTurn.parts.text is the spoken response and is emitted as writing instead.

session:

event

Emitted on session state changes. Callback receives { state: 'connecting' | 'connected' | 'disconnected' | 'disconnecting' | 'updated', config?: object }.

turnComplete:

event

Emitted when a conversation turn is completed. Callback receives { timestamp: number }.

toolCall:

event

Emitted when the model requests a tool call. Callback receives { name: string, args: object, id: string }.

usage:

event

Emitted with token usage information. Callback receives { inputTokens: number, outputTokens: number, totalTokens: number, modality: string }.

error:

event

Emitted when an error occurs. Callback receives { message: string, code?: string, details?: unknown }.

interrupt:

event

Emitted on barge-in when the user starts speaking over an in-flight model response. The server cancels any further audio for the current turn. Callback receives { type: 'user', timestamp: number }.

Native-audio behavior

Native-audio Gemini Live models (any model whose ID contains native-audio, such as gemini-2.5-flash-native-audio-preview-12-2025) split text output across two channels:

• The model's spoken reply is delivered as audio plus an output_audio_transcription transcript and surfaced as writing with role: 'assistant'.
• The model's internal reasoning is delivered as modelTurn.parts.text and surfaced as thinking.

On non-native-audio models there is no output_audio_transcription channel, so modelTurn.parts.text is the spoken response itself and is emitted as writing. The thinking event doesn't fire.

Input transcription, output transcription, and barge-in detection (realtime_input_config.activity_handling = 'START_OF_ACTIVITY_INTERRUPTS') are enabled automatically in the setup payload. You don't need extra configuration.

Available models

The following Gemini Live models are available:

• gemini-2.0-flash-exp (default)
• gemini-2.0-flash-exp-image-generation
• gemini-2.0-flash-live-001
• gemini-live-2.5-flash-preview-native-audio
• gemini-2.5-flash-exp-native-audio-thinking-dialog
• gemini-live-2.5-flash-preview
• gemini-2.6.flash-preview-tts

Available voices

The following voice options are available:

• Puck (default): Conversational, friendly
• Charon: Deep, authoritative
• Kore: Neutral, professional
• Fenrir: Warm, approachable

Authentication methods

Gemini API (Development)

The simplest method using an API key from Google AI Studio:

const voice = new GeminiLiveVoice({
  apiKey: 'your-api-key', // Required for Gemini API
  model: 'gemini-2.0-flash-exp',
})

Vertex AI (Production)

For production use with OAuth authentication and Google Cloud Platform:

// Using service account key file
const voice = new GeminiLiveVoice({
  vertexAI: true,
  project: 'your-gcp-project',
  location: 'us-central1',
  serviceAccountKeyFile: '/path/to/service-account.json',
})

// Using Application Default Credentials
const voice = new GeminiLiveVoice({
  vertexAI: true,
  project: 'your-gcp-project',
  location: 'us-central1',
})

// Using service account impersonation
const voice = new GeminiLiveVoice({
  vertexAI: true,
  project: 'your-gcp-project',
  location: 'us-central1',
  serviceAccountEmail: 'service-account@project.iam.gserviceaccount.com',
})

Advanced features

Session Management

The Gemini Live API supports session resumption for handling network interruptions:

voice.on('sessionHandle', ({ handle, expiresAt }) => {
  // Store session handle for resumption
  saveSessionHandle(handle, expiresAt)
})

// Resume a previous session
const voice = new GeminiLiveVoice({
  sessionConfig: {
    enableResumption: true,
    maxDuration: '2h',
  },
})

Tool Calling

Enable the model to call functions during conversations:

import { z } from 'zod'

voice.addTools({
  weather: {
    description: 'Get weather information',
    parameters: z.object({
      location: z.string(),
    }),
    execute: async ({ location }) => {
      const weather = await getWeather(location)
      return weather
    },
  },
})

voice.on('toolCall', ({ name, args, id }) => {
  console.log(`Tool called: ${name} with args:`, args)
})

Notes

• The Gemini Live API uses WebSockets for real-time communication
• Audio is processed as 16kHz PCM16 for input and 24kHz PCM16 for output
• The voice instance must be connected with connect() before using other methods
• Always call close() when done to properly clean up resources
• Vertex AI authentication requires appropriate IAM permissions (aiplatform.user role)
• Session resumption allows recovery from network interruptions
• The API supports real-time interactions with text and audio