How I Built an AI Voice Commerce System with Twilio & Gemini

How to Build an AI Voice Commerce System with Twilio, Gemini, FastAPI, and PostgreSQL

A modern phone call can do far more than route a customer through an interactive voice response menu. With real-time media streaming, speech recognition, a language model, controlled backend tools, and transactional services, a call can become a conversational interface for product discovery, order management, customer support, and commerce.

This article presents a production-oriented reference architecture for an AI voice commerce system built with:

Twilio Programmable Voice and Media Streams
Google Cloud Speech-to-Text
the Gemini API
Python and FastAPI
PostgreSQL with pgvector
Redis
AWS ECS on Fargate
OpenTelemetry
a PCI-compliant payment provider

The goal is not to present a universal benchmark or a copy-and-paste production system. Real latency, accuracy, throughput, availability, and cost depend on call geography, model choice, network conditions, audio quality, traffic patterns, and implementation details. Instead, this guide focuses on architectural patterns that can be tested, measured, and hardened for a specific deployment.

What Is AI Voice Commerce?
System Goals and Non-Goals
High-Level Architecture
Technology Stack
Twilio Call Ingress and Media Streaming
Real-Time Speech-to-Text
Gemini as a Controlled Reasoning Layer
Conversation State and Orchestration
Semantic Product Search with PostgreSQL and pgvector
Safe Tool Invocation
Order and Payment Architecture
Latency Engineering
Security and Abuse Prevention
AWS Deployment Architecture
Scaling and High Availability
Observability and Monitoring
Cost Modeling
Common Implementation Mistakes
Production Readiness Checklist
Final Thoughts

What Is AI Voice Commerce?

AI voice commerce combines telephony, speech processing, language models, business logic, data systems, and payment infrastructure to let customers complete commercial tasks through natural conversation.

A typical system includes the following layers:

Component	Responsibility
Telephony platform	Receives and controls PSTN or VoIP calls
Media streaming	Sends call audio to the application and, for bidirectional streams, sends generated audio back
Speech-to-text	Converts live audio into interim and final transcripts
Language model	Classifies intent, extracts entities, generates responses, or requests approved tools
Orchestrator	Maintains state, applies business rules, handles retries, and coordinates services
Product search	Retrieves relevant products using structured filters, full-text search, vectors, or a hybrid approach
Transaction services	Checks inventory, creates orders, applies idempotency, and handles payment workflows
Security controls	Validates requests, limits abuse, constrains model actions, and protects sensitive data
Observability	Captures traces, metrics, logs, business outcomes, and security events

The important distinction is that the language model should not be the system of record. It can interpret a request and propose an action, but deterministic application code must validate and execute that action.

System Goals and Non-Goals

A practical AI voice commerce platform may support:

conversational product discovery
product comparison and question answering
authenticated order status checks
cart creation and order confirmation
payment handoff or tokenized payment flows
CRM and support-system updates
escalation to a human agent
interruption handling and conversation recovery
multilingual or locale-specific experiences

The architecture should also define what the assistant cannot do. Examples include changing prices without authorization, exposing internal prompts, reading full payment credentials, issuing unrestricted refunds, or executing arbitrary functions.

A production target should be written as a set of measurable service-level indicators rather than a marketing claim. Useful indicators include:

time to first transcript
final-transcript delay after the user stops speaking
model time to first token
tool execution latency
text-to-speech time to first audio
end-to-end response-start latency
task-completion rate
transfer-to-human rate
order error rate
payment failure rate
security challenge and block rates

High-Level Architecture

The core call path looks like this:

Caller
  |
  v
Twilio Programmable Voice
  |
  |  TwiML + secure WebSocket
  v
FastAPI Voice Gateway
  |
  +--> Streaming Speech-to-Text
  |
  +--> Conversation Orchestrator
  |       |
  |       +--> Gemini reasoning / structured decision
  |       +--> Customer and session data
  |       +--> Product search
  |       +--> Inventory and order services
  |       +--> Payment provider
  |       +--> CRM or support platform
  |
  +--> Text-to-Speech or audio-generation service
  |
  v
Twilio bidirectional Media Stream
  |
  v
Caller

Twilio's TwiML instruction sends call audio to a WebSocket server in near real time. creates a unidirectional stream, while creates a bidirectional stream that can receive audio from the application. Twilio requires a secure wss URL for Media Streams. Twilio documents these behaviors in its reference.

For bidirectional streams, Twilio sends inbound call audio as base64-encoded audio/x-mulaw at 8 kHz with one channel, and the application must use the same format when sending audio back. The WebSocket message reference documents the event types and media format.

Technology Stack

Layer	Suggested technology	Why it fits
Telephony	Twilio Programmable Voice	PSTN connectivity, TwiML, webhooks, call control, and Media Streams
API and WebSocket gateway	FastAPI	Python async support, HTTP endpoints, and WebSocket handling
Speech recognition	Google Cloud Speech-to-Text	Streaming recognition and interim results
Reasoning	Gemini API	Structured outputs and function-calling support
Primary database	PostgreSQL	Transactional consistency and relational modeling
Vector search	pgvector	Vector similarity search inside PostgreSQL
Session and hot cache	Redis	Fast external session storage and rate-limit counters
Runtime	Amazon ECS on AWS Fargate	Managed container execution without managing EC2 hosts
Load balancing	Application Load Balancer	HTTP/HTTPS routing to ECS services
Observability	OpenTelemetry plus a telemetry backend	Vendor-neutral traces, metrics, and logs
Payments	Stripe or another PCI-compliant provider	Tokenized payment collection and payment orchestration

Model names and API versions change. For production, pin an explicitly supported stable model instead of relying on an experimental alias. Google's model documentation recommends specific stable model identifiers for production applications and lists preview, latest, experimental, and deprecated variants separately. Review the current Gemini models before deployment.

Twilio Call Ingress and Media Streaming

1. Receive the call webhook

When a customer calls a Twilio number, Twilio sends an HTTP request to the configured voice webhook. The application responds with TwiML that controls the call.

A simplified FastAPI handler can return a bidirectional stream:

import os

from fastapi import FastAPI, HTTPException, Request
from fastapi.responses import Response
from twilio.request_validator import RequestValidator

app = FastAPI()


def external_request_url(request: Request) -> str:
    """Return the exact public URL Twilio used.

    In deployments behind a reverse proxy or load balancer, configure trusted
    forwarded headers and reconstruct the public scheme and host correctly.
    """
    return str(request.url)


@app.post("/twilio/voice")
async def handle_voice_webhook(request: Request) -> Response:
    form = await request.form()
    signature = request.headers.get("X-Twilio-Signature", "")

    validator = RequestValidator(os.environ["TWILIO_AUTH_TOKEN"])
    is_valid = validator.validate(
        external_request_url(request),
        dict(form),
        signature,
    )

    if not is_valid:
        raise HTTPException(status_code=403, detail="Invalid Twilio signature")

    twiml = """<?xml version="1.0" encoding="UTF-8"?>

  
    
  
"""

    return Response(content=twiml, media_type="application/xml")

Twilio cryptographically signs webhook requests and sends the signature in the X-Twilio-Signature header. Twilio recommends validating the signature with an official SDK and using the exact URL and request parameters seen by Twilio. Reverse proxies can change the scheme or host visible to the application, so proxy configuration must be tested carefully. Twilio's security guide explains the validation process, and its Python tutorial uses RequestValidator. See the official Python validation example.

2. Process WebSocket events

Twilio sends connected, start, media, and stop events to the WebSocket server. A minimal receiver looks like this:

import base64
from fastapi import WebSocket, WebSocketDisconnect


@app.websocket("/ws/media")
async def media_stream(websocket: WebSocket) -> None:
    await websocket.accept()
    call_sid: str | None = None
    stream_sid: str | None = None

    try:
        while True:
            message = await websocket.receive_json()
            event = message.get("event")

            if event == "start":
                call_sid = message["start"]["callSid"]
                stream_sid = message["start"]["streamSid"]
                await start_call_session(call_sid, stream_sid)

            elif event == "media":
                audio_bytes = base64.b64decode(
                    message["media"]["payload"]
                )
                await enqueue_audio(call_sid, audio_bytes)

            elif event == "stop":
                await close_call_session(call_sid)
                break

    except WebSocketDisconnect:
        await close_call_session(call_sid)

In a real implementation, keep the receive loop lightweight. Decode, validate, timestamp, and enqueue audio; do not block the WebSocket while waiting for speech recognition, model inference, database queries, or text-to-speech.

3. Support interruption

A voice agent should stop speaking when the caller interrupts. For a bidirectional Twilio Media Stream, the application can send a clear message to remove buffered outbound audio. Use mark messages to correlate sent audio with playback completion. These message types are documented in Twilio's Media Streams WebSocket reference.

Real-Time Speech-to-Text

Batch transcription is a poor fit for conversational telephony because the application needs partial results while the caller is speaking. Google Cloud Speech-to-Text supports streaming recognition, which returns recognition results as audio is processed. Google documents streaming recognition and provides language-specific samples.

Twilio's inbound media format is μ-law, 8 kHz, mono. Google Cloud Speech-to-Text supports MULAW, and its documentation describes μ-law as 8-bit companded audio. See Google's audio-encoding documentation.

A simplified configuration is:

from google.cloud import speech_v1 as speech

recognition_config = speech.RecognitionConfig(
    encoding=speech.RecognitionConfig.AudioEncoding.MULAW,
    sample_rate_hertz=8000,
    language_code="en-US",
    enable_automatic_punctuation=True,
    model="phone_call",
)

streaming_config = speech.StreamingRecognitionConfig(
    config=recognition_config,
    interim_results=True,
    single_utterance=False,
)

Interim versus final transcripts

Use interim transcripts for low-risk speculative work, such as:

detecting likely intent
prefetching a customer profile
warming a product search
preparing a likely response template

Do not place an order, charge a payment method, cancel an order, or change customer data from an interim transcript. Commit side effects only after the final transcript is available and the business rules are satisfied.

Improve recognition quality

Telephony audio is narrowband and lossy, so accuracy varies with line quality, background noise, accent, language, and product vocabulary. Improve results by:

using the audio's native sample rate instead of unnecessary resampling
selecting a suitable recognition model and locale
supplying phrase hints or adaptation for product names where supported
tracking confidence and re-prompting when confidence is low
adding DTMF or human-agent fallback for critical identifiers
testing with real call recordings collected under an appropriate consent and retention policy

Avoid publishing a universal word-error-rate claim unless it comes from a reproducible evaluation set that matches the deployment environment.

Gemini as a Controlled Reasoning Layer

The language model should interpret conversation, not own business state or directly execute unrestricted actions.

Gemini supports structured outputs that follow a JSON Schema and function calling that lets the model request application-defined tools. Google's documentation distinguishes the two: structured output is suitable when the final model response must match a schema, while function calling is designed for requesting actions or external data. The application remains responsible for executing the function. See the Gemini structured-output guide and function-calling guide.

Use a stable SDK and model

Google's current Python SDK is google-genai:

pip install google-genai pydantic

The SDK is maintained in Google's official googleapis/python-genai repository. Review the official SDK repository and release notes.

Define a structured decision contract

import os
from typing import Literal

from google import genai
from google.genai import types
from pydantic import BaseModel, Field


class ProductFilters(BaseModel):
    category: str | None = None
    minimum_price: float | None = Field(default=None, ge=0)
    maximum_price: float | None = Field(default=None, ge=0)
    requested_features: list[str] = Field(default_factory=list)


class VoiceDecision(BaseModel):
    intent: Literal[
        "product_search",
        "product_question",
        "select_product",
        "place_order",
        "check_order_status",
        "cancel_order",
        "request_human",
        "clarify",
    ]
    confidence: float = Field(ge=0, le=1)
    filters: ProductFilters | None = None
    product_id: int | None = None
    response_text: str


client = genai.Client(api_key=os.environ["GEMINI_API_KEY"])

SYSTEM_INSTRUCTION = """
You are a voice-commerce reasoning component.
Return only information allowed by the supplied response schema.
Never claim that an action succeeded until the application provides a tool result.
Never request a tool outside the approved workflow.
For payments, refunds, cancellations, or order placement, require explicit confirmation.
Keep spoken responses concise and easy to understand over a phone call.
"""


def decide(transcript: str, context_summary: str) -> VoiceDecision:
    response = client.models.generate_content(
        model=os.environ.get("GEMINI_MODEL", "gemini-3.6-flash"),
        contents=(
            f"Conversation context:\n{context_summary}\n\n"
            f"Customer transcript:\n{transcript}"
        ),
        config=types.GenerateContentConfig(
            system_instruction=SYSTEM_INSTRUCTION,
            response_mime_type="application/json",
            response_json_schema=VoiceDecision,
            temperature=0.2,
        ),
    )
    return VoiceDecision.model_validate_json(response.text)

Before deploying, verify that the selected model identifier is available in the target API and region. Model availability and deprecation status change over time.

Do not expose private chain-of-thought

A production schema does not need an unrestricted reasoning field. Store auditable decision data such as intent, confidence, selected policy, tool requested, validation result, and tool outcome. This provides operational traceability without relying on hidden model reasoning.

Conversation State and Orchestration

Voice conversations are not linear. Callers interrupt, revise constraints, ask follow-up questions, and change goals. A finite-state machine or workflow engine makes allowed transitions explicit.

from enum import StrEnum


class CallState(StrEnum):
    GREETING = "greeting"
    LISTENING = "listening"
    SEARCHING = "searching"
    REVIEWING = "reviewing"
    AWAITING_CONFIRMATION = "awaiting_confirmation"
    PROCESSING_ORDER = "processing_order"
    COMPLETED = "completed"
    ESCALATING = "escalating"
    ENDED = "ended"


ALLOWED_TRANSITIONS: dict[CallState, set[CallState]] = {
    CallState.GREETING: {CallState.LISTENING},
    CallState.LISTENING: {
        CallState.SEARCHING,
        CallState.REVIEWING,
        CallState.ESCALATING,
        CallState.ENDED,
    },
    CallState.SEARCHING: {
        CallState.REVIEWING,
        CallState.LISTENING,
        CallState.ESCALATING,
    },
    CallState.REVIEWING: {
        CallState.SEARCHING,
        CallState.AWAITING_CONFIRMATION,
        CallState.LISTENING,
        CallState.ESCALATING,
    },
    CallState.AWAITING_CONFIRMATION: {
        CallState.PROCESSING_ORDER,
        CallState.REVIEWING,
        CallState.ENDED,
    },
    CallState.PROCESSING_ORDER: {
        CallState.COMPLETED,
        CallState.REVIEWING,
        CallState.ESCALATING,
    },
    CallState.COMPLETED: {CallState.LISTENING, CallState.ENDED},
    CallState.ESCALATING: {CallState.ENDED},
    CallState.ENDED: set(),
}

The orchestrator should own:

the current state
a compact conversation summary
selected products and quantities
authentication status
confirmation status
idempotency keys
tool outcomes
retry counts
risk decisions
human-handoff context

Keep state outside individual application containers. Redis is useful for short-lived call sessions, while PostgreSQL should store durable business records and audit events.

Semantic Product Search with PostgreSQL and pgvector

Voice queries often describe needs rather than exact product names:

“I need a lightweight laptop for travel with strong battery life and a budget under $1,000.”

A robust search flow combines:

deterministic filters for price, availability, brand, region, and policy constraints
full-text or lexical search for exact terms and identifiers
vector similarity for semantic relevance
optional re-ranking using business rules or a dedicated ranking model

pgvector adds vector similarity search to PostgreSQL and supports exact and approximate nearest-neighbor search, including cosine distance. Its <=> operator represents cosine distance, and cosine similarity can be calculated as 1 - distance. See the official pgvector documentation.

SELECT
    id,
    name,
    description,
    price,
    1 - (embedding <=> $1::vector) AS cosine_similarity
FROM products
WHERE is_active = TRUE
  AND inventory_available > 0
  AND price >= $2
  AND price <= $3
ORDER BY embedding <=> $1::vector
LIMIT 5;

Do not tell a caller that a product is available or at a specific price based only on a vector index. Read authoritative inventory and pricing data before presenting or confirming the result.

Hybrid-search design

A useful ranking formula might combine:

final_score =
    semantic_similarity
  + lexical_match_weight
  + availability_weight
  + business_priority_weight
  - policy_penalty

Document every weight, test it on a labeled query set, and monitor whether ranking changes hurt conversion, relevance, or fairness.

Safe Tool Invocation

The model may select an action, but backend code must authorize it.

from collections.abc import Awaitable, Callable
from typing import Any

Tool = Callable[..., Awaitable[dict[str, Any]]]


class ToolRegistry:
    def __init__(self) -> None:
        self._tools: dict[str, Tool] = {
            "search_products": search_products,
            "get_product_details": get_product_details,
            "create_order": create_order,
            "check_order_status": check_order_status,
        }

        self._intent_acl: dict[str, set[str]] = {
            "product_search": {"search_products"},
            "product_question": {"get_product_details"},
            "place_order": {"create_order"},
            "check_order_status": {"check_order_status"},
        }

    async def dispatch(
        self,
        *,
        intent: str,
        tool_name: str,
        arguments: dict[str, Any],
        session: "CallSession",
    ) -> dict[str, Any]:
        if tool_name not in self._intent_acl.get(intent, set()):
            raise PermissionError("Tool is not allowed for this intent")

        validated = validate_tool_arguments(tool_name, arguments)
        enforce_session_permissions(tool_name, session)
        enforce_confirmation_policy(tool_name, session)

        tool = self._tools.get(tool_name)
        if tool is None:
            raise LookupError("Unknown tool")

        return await tool(**validated)

Required guardrails include:

strict tool allowlists
typed argument validation
authorization based on the authenticated customer
explicit confirmation for consequential actions
quantity and value limits
idempotency keys
timeouts and bounded retries
safe error mapping
audit logging without sensitive payloads

OWASP identifies prompt injection, insecure output handling, and excessive agency as important risks in LLM applications. The practical response is layered control: treat model output as untrusted, validate it, minimize available privileges, and require deterministic authorization before executing actions. See the OWASP GenAI Security Project.

Order and Payment Architecture

Order placement should be a transactional workflow, not a single model response.

1. Re-read authoritative product price and availability
2. Confirm product, quantity, delivery details, and total
3. Create an idempotency key for the confirmed action
4. Reserve inventory or use an equivalent concurrency control
5. Create the payment request through the payment provider
6. Commit the order only after the required payment state is reached
7. Release inventory when the workflow fails or expires
8. Generate the receipt from persisted order data

Never collect raw card data in the model context

Use the payment provider's approved collection and tokenization flow. Stripe describes tokenization as collecting sensitive payment details directly and exchanging them for a token so sensitive card data does not touch the merchant's server. PCI compliance remains a shared responsibility between the payment provider and the business. See Stripe's token documentation and integration security guide.

Depending on jurisdiction, provider capability, and risk appetite, a voice workflow may:

send a secure payment link by SMS
transfer the caller to a PCI-compliant payment flow
use a previously stored payment method after strong customer verification
collect non-sensitive confirmation while the provider handles sensitive fields

Consult qualified security, privacy, payments, and legal specialists before processing payment information over voice.

Latency Engineering

There is no universal latency number that every voice commerce system can guarantee. A meaningful latency budget must separate user speech time, endpointing, network transport, recognition, model inference, tool execution, synthesis, and playback.

Track at least these timestamps:

t0: last inbound audio frame for the utterance
t1: final transcript emitted
t2: model request started
t3: first model output or decision available
t4: tool request started
t5: tool result available
t6: first TTS/audio chunk available
t7: first outbound audio sent
t8: first outbound audio played to caller

Useful derived metrics include:

endpointing_delay = t1 - t0
model_latency = t3 - t2
tool_latency = t5 - t4
tts_time_to_first_audio = t6 - t5
response_start_latency = t8 - t0

Techniques that reduce perceived delay

Stream each stage

Use streaming recognition, streaming model output where appropriate, and streaming audio generation. Google's Gemini documentation supports streaming responses, while Twilio's bidirectional stream can receive outbound media frames.

Parallelize independent work

Fetch customer context and prepare likely product data while the intent decision is running, provided the work has no side effects.

profile_task = asyncio.create_task(load_customer_profile(customer_id))
decision_task = asyncio.create_task(classify_intent(transcript, context))

profile, decision = await asyncio.gather(profile_task, decision_task)

Keep connections warm

Reuse HTTP clients, database pools, Redis connections, and model clients. Avoid loading large local embedding models inside a request handler.

Use short spoken turns

A phone response should usually present one decision or a small set of options at a time. Shorter output improves comprehension, interruption handling, and time to first useful information.

Prefetch carefully

Interim transcripts can trigger read-only prefetching, but side effects must wait for final input, authorization, and confirmation.

Benchmark honestly

Publish latency percentiles only when the test method is documented. Include:

test region and caller geography
number of calls and duration
warm versus cold traffic
audio format and language
concurrency distribution
exact model and API versions
whether values are server-side or caller-perceived
P50, P95, and P99
failure and timeout definitions

Security and Abuse Prevention

An internet-facing telephony application has several attack surfaces: public webhooks, WebSockets, the model interface, tool APIs, payment workflows, customer identity flows, and operational dashboards.

Defense-in-depth flow

Incoming call or webhook
  |
  +--> TLS and Twilio signature validation
  +--> account, number, and network-level rate controls
  +--> session and replay checks
  +--> customer authentication when required
  +--> transcript and input policy checks
  +--> constrained model decision
  +--> schema validation
  +--> tool allowlist and authorization
  +--> explicit confirmation for consequential actions
  +--> idempotent transaction execution
  +--> redacted audit and security logging

Prompt injection

Prompt injection cannot be solved by a system prompt alone. Treat every transcript, retrieved product description, CRM note, and tool result as untrusted data.

Practical controls include:

separate instructions from user and retrieved data
provide the model with the minimum necessary context
use typed outputs or function declarations
reject unknown intents, fields, and tools
validate model output before use
restrict tool permissions by task and customer identity
require confirmation for purchases, cancellations, refunds, and account changes
cap tool loops, token use, and execution time
monitor unusual tool-request patterns
keep secrets and payment data out of prompts

Rate limiting and abuse controls

Rate limits should not rely on a single identifier. Combine signals such as:

account or tenant
destination number
source number, where appropriate
call velocity
repeated failed authentication
repeated payment failures
repeated identical requests
unusual order value or quantity
infrastructure-level network signals

A risk score should guide proportional actions such as monitoring, requiring additional verification, limiting high-risk tools, transferring to a human, or blocking. Test the system for false positives and review the privacy and fairness implications of any behavioral features.

Sensitive-data handling

redact or avoid storing full transcripts when they are not required
never log card details, secrets, access tokens, or authentication answers
encrypt data in transit and at rest
use a documented retention schedule
limit employee and service access by role
record administrative access to sensitive systems
provide appropriate call-recording and AI-use disclosures
comply with applicable consent, privacy, consumer-protection, and telemarketing laws

AWS Deployment Architecture

A straightforward AWS deployment can use:

Route 53
  |
AWS WAF
  |
Application Load Balancer
  |
Amazon ECS Service on AWS Fargate
  |        |         |
  |        |         +--> OpenTelemetry Collector / telemetry backend
  |        +------------> ElastiCache for Redis
  +---------------------> Amazon RDS for PostgreSQL
                           |
                           +--> Multi-AZ high availability
                           +--> optional read replicas for read scaling

Amazon ECS services can maintain a desired number of tasks, replace failed tasks, run behind a load balancer, and use Application Auto Scaling. AWS documents Application Load Balancers for HTTP/HTTPS routing to ECS tasks and Network Load Balancers for layer-4 TCP or UDP requirements. See the ECS load-balancing guide.

AWS Fargate charges for requested compute, memory, operating system, architecture, and storage resources for running tasks. See the official AWS Fargate pricing page.

Correctly separate high availability from read scaling

For Amazon RDS for PostgreSQL:

a Multi-AZ deployment provides high availability and failover
a traditional standby in a Multi-AZ DB instance deployment does not serve read traffic
read replicas are used to scale read-heavy workloads and use asynchronous replication

AWS documents these distinctions in its RDS Multi-AZ guide and read-replica guide.

Scaling and High Availability

Keep application tasks stateless

Do not depend on one container's memory for durable call state. Store session state in Redis or another shared store and persist business state in PostgreSQL.

import json


class SessionStore:
    def __init__(self, redis_client) -> None:
        self.redis = redis_client

    async def save(self, call_sid: str, payload: dict) -> None:
        await self.redis.set(
            f"call-session:{call_sid}",
            json.dumps(payload),
            ex=3600,
        )

    async def load(self, call_sid: str) -> dict | None:
        value = await self.redis.get(f"call-session:{call_sid}")
        return json.loads(value) if value else None

Scale on workload indicators

CPU alone may not reflect voice workload pressure. Consider a combination of:

active WebSocket sessions per task
event-loop lag
queued audio duration
speech-recognition backlog
model request concurrency
tool-call latency
memory use
error and timeout rate

ECS Service Auto Scaling uses Application Auto Scaling and CloudWatch metrics to adjust the desired task count. AWS supports target tracking and step scaling. See the ECS auto-scaling guidance.

Design for failure

use timeouts around every external call
use bounded retries with jitter only for retryable operations
make order and payment actions idempotent
provide a deterministic fallback response
support DTMF for critical flows
transfer to a human when confidence or system health is poor
test dependency failures, partial outages, and deployment rollbacks
drain WebSocket sessions during deployments where possible

Observability and Monitoring

OpenTelemetry is a vendor-neutral framework for generating, collecting, and exporting traces, metrics, and logs. It is not an observability backend by itself. See the OpenTelemetry documentation.

Trace one conversational turn

A trace can include spans for:

voice.turn
  +-- stt.finalize
  +-- context.load
  +-- model.decide
  +-- product.search
  +-- inventory.check
  +-- order.create
  +-- tts.first_audio
  +-- twilio.send_audio

Do not attach raw payment data or unrestricted transcripts to spans. Prefer identifiers, durations, statuses, error classes, and redacted metadata.

Essential technical metrics

active calls and streams
call setup failure rate
WebSocket disconnect rate
audio queue depth
time to interim and final transcript
model latency and error rate
schema-validation failure rate
tool-call count, latency, and error rate
TTS time to first audio
end-to-end response-start latency
Redis and database latency
rate-limit and security-rule activations

Essential business metrics

task-completion rate
product-search success rate
order-confirmation rate
purchase conversion rate
average order value
human-transfer rate
caller abandonment rate
cancellation and refund rate
repeat-call rate

Use separate dashboards for system health, conversation quality, business outcomes, and security. Correlate them by privacy-safe call and trace identifiers.

Cost Modeling

Do not publish a universal cost per call. Twilio pricing varies by country, number type, call direction, and features. Speech-to-Text pricing depends on processed audio and tier. Gemini pricing depends on the selected model, input and output tokens, modality, caching, and service tier. AWS cost depends on region and requested resources. Review Twilio Voice pricing for the target country, Google Cloud Speech-to-Text pricing, Gemini API pricing, and AWS Fargate pricing when building the estimate.

A transparent monthly model is:

telephony_cost = inbound_minutes * telephony_rate
stt_cost = processed_audio_minutes * stt_rate
llm_cost = input_tokens * input_rate + output_tokens * output_rate
tts_cost = synthesized_units * tts_rate
compute_cost = task_vcpu_time + task_memory_time + storage
platform_cost = database + cache + load_balancer + WAF + secrets
observability_cost = ingest + retention + queries
network_cost = data_transfer + cross-region traffic
support_cost = provider_support + operational staffing

total_monthly_cost = sum(all cost categories)
unit_cost = total_monthly_cost / successfully_completed_calls

Track both cost per connected call and cost per successful outcome. A cheap call that fails to complete the task may be more expensive to the business than a slightly longer successful call.

Practical optimization levers

route simple intents to cheaper, faster models when quality is acceptable
cap conversation history and use a compact state summary
cache stable catalog and FAQ data with appropriate invalidation
avoid sending unnecessary fields to the model
stop model generation when enough response text is available
keep spoken answers concise
batch offline embedding generation
right-size Fargate, database, and Redis capacity
use autoscaling based on actual workload signals
sample high-volume debug telemetry while retaining errors and security events

Common Implementation Mistakes

1. Giving the model excessive authority

Risk: The model can call any function with arbitrary arguments.

Better design: Use a small tool allowlist, typed schemas, authorization, value limits, confirmation, and idempotency.

2. Treating an interim transcript as final

Risk: The system acts on words that the recognizer later changes.

Better design: Use interim results only for read-only prediction and prefetching.

3. Claiming success before a tool succeeds

Risk: The assistant says an order was placed even though the transaction failed.

Better design: Generate the confirmation from persisted tool output.

4. Mixing catalog search with authoritative commerce data

Risk: Semantic search returns an outdated price or unavailable product.

Better design: Use vectors for candidate retrieval, then re-read authoritative price and inventory.

5. Storing sessions only in process memory

Risk: A deployment or task failure loses the conversation.

Better design: Externalize session state and persist durable events.

6. Logging too much

Risk: Transcripts, payment data, or identity information leak into logs and traces.

Better design: Minimize, redact, encrypt, restrict, and expire sensitive data.

7. Using a fixed latency claim without a test method

Risk: A server-side microbenchmark is presented as caller-perceived performance.

Better design: Define timestamps, publish percentile distributions, and document conditions.

8. Confusing Multi-AZ with read scaling

Risk: The architecture assumes a standby replica handles queries.

Better design: Use Multi-AZ for availability and explicit read replicas for read scaling where appropriate.

9. Hard-coding experimental or deprecated model names

Risk: The application breaks when a model is removed.

Better design: Pin a stable supported model, monitor deprecation notices, and test upgrades.

Production Readiness Checklist

Telephony and media

Twilio webhook signatures are validated
the public webhook URL is reconstructed correctly behind proxies
WebSocket authentication and origin controls are defined
inbound media format is handled correctly
outbound audio matches Twilio's required μ-law format
interruption, clear, and playback tracking are tested
call recording and AI disclosure requirements are reviewed

AI and orchestration

model output follows a typed schema
every intent and action is allowlisted
consequential actions require confirmation
the model cannot directly access secrets or unrestricted tools
low-confidence and ambiguous requests have fallback paths
human escalation is available
model and SDK versions are pinned and monitored

Commerce and payments

price and inventory are revalidated before confirmation
orders and payments are idempotent
raw card data never enters prompts or application logs
the payment flow follows provider and PCI guidance
failed workflows release reservations safely
receipts are generated from persisted transaction data

Reliability and security

external calls have timeouts and bounded retries
session state survives task replacement
rate limits use multiple signals
sensitive data is minimized, redacted, encrypted, and expired
dependency failures and partial outages are tested
audit events are immutable enough for investigation needs

Observability and cost

end-to-end traces cover each conversational turn
latency is measured at P50, P95, and P99
business outcomes are correlated with technical performance
security events are monitored separately from normal errors
provider usage and spend have budgets and alerts
cost is measured per successful outcome, not only per call

Final Thoughts

AI voice commerce is best understood as a distributed transaction system with a conversational interface.

The phone and media layer carries audio. Speech recognition converts audio into text. Gemini interprets the request or proposes an approved action. The orchestrator enforces state and policy. PostgreSQL and commerce services remain authoritative. The payment provider handles sensitive payment workflows. Security controls constrain every transition, and observability shows whether the system is fast, reliable, safe, and commercially useful.

The most important production principle is simple:

Let the model interpret. Let deterministic systems authorize, execute, and record.

That separation makes the platform easier to test, safer to operate, and more resilient as telephony providers, speech models, language models, and infrastructure continue to evolve.

Verified Sources

Topics

AI Voice Commerce AI Telephony

Md Bazlur Rahman Likhon

Senior Cloud and AI Engineer

Generative AI expert with 6+ years experience and 300+ certifications. Building LLM, RAG systems, and multi-cloud AI solutions.

[email protected]