Serverless LLM Inference: The Real Cost of AWS Bedrock vs. GCP Vertex vs. Azure OpenAI in 2026

Most enterprises overspend 32–57% on LLM inference due to invisible serverless costs—cold starts, throttling, and orchestration overhead that never appear on pricing pages. A recent IDC survey found that 96% of organizations deploying GenAI report costs higher than expected, with 71% admitting they have little to no control over where those costs originate. This isn't a budget variance problem. It's an architectural blind spot that scales exponentially as workloads move from pilot to production. datarobot

This analysis dissects the true economics of serverless LLM inference across AWS Bedrock, Google Cloud Vertex AI, and Azure OpenAI—not the marketing collateral, but the operational reality uncovered through enterprise deployment data, compliance frameworks, and production cost breakdowns. If you're a cloud architect, CTO, or FinOps lead evaluating these platforms for 2026 deployments, this report will save you from the $1.2M cost overrun that 24% of enterprises experience when they miss forecasts by more than 50%. cio

Why 2026 Serverless LLM Inference Is Different from Traditional Serverless

Traditional serverless compute (Lambda, Cloud Functions) scales predictably: cold starts add 100-500ms, concurrency limits are transparent, and pricing is linear. Serverless LLM inference shatters these assumptions.

Stateless is a lie. LLMs maintain pseudo-stateful context through conversation history, retrieved documents, and tool execution chains. A "stateless" RAG query carries 1,000+ tokens of retrieved context, 50 tokens of user input, and generates 200 tokens of output—1,250 tokens billed as a single request, but architecturally resembling a distributed transaction across vector databases, embedding models, and generation endpoints. appinventiv

Token pricing is noise. Token costs represent approximately 5% of total serverless LLM spend. The remaining 95% hides in vector storage (20-40%), query infrastructure (30-50%), embedding generation (10-20%), monitoring, and cross-region egress. A production RAG system processing 50,000 daily queries doesn't cost $200/month in tokens—it costs $800-3,500/month when you include OpenSearch Serverless, CloudWatch logs at $0.50/GB ingestion, and inter-region data transfer at $0.02/GB. cloudzero

Cold starts break production SLOs. In production environments measured at scale, serverless LLM cold starts average 40+ seconds before the first token appears—versus 30ms per token for warm instances. For customer-facing chatbots or real-time analytics, this latency gulf is operationally unacceptable. Salesforce documented that moving to AWS Bedrock Custom Model Import reduced cold start impact but still required extensive testing to ensure scaling from zero to multiple model copies didn't introduce throttling. engineering.salesforce

Agentic workflows compound costs super-linearly. A ten-step agent workflow with 10% inefficiency per step doesn't cost 10% more—it costs 3-10× more because each tool call expands context, generates intermediate reasoning tokens, and triggers retry storms when external APIs fail. Function calling overhead alone can add 5,000 tokens per request when ten tools are defined with 500-token schemas—before the user even asks a question. codeant

Architecture Primer: What "Serverless LLM Inference" Actually Means

Serverless LLM inference platforms abstract GPU provisioning, model loading, and auto-scaling behind managed APIs. You submit prompts via HTTP, the platform routes requests to warm instances (or cold-starts new ones), and you're billed per token processed. This model works beautifully for unpredictable workloads—until you examine what happens under load.

Queueing Behavior and Backpressure

When request volume exceeds provisioned capacity, platforms queue incoming requests. AWS Bedrock's on-demand mode throttles with 429 errors when burst limits are hit. Azure OpenAI enforces both TPM (tokens per minute) and RPM (requests per minute) limits, with RPM evaluated over 1-10 second windows. If you send 15 requests in one second to a 600 RPM deployment, you're throttled immediately—even if you're nowhere near your monthly token quota. techcommunity.microsoft

Google Vertex AI takes a different approach: batch inference jobs share a dynamic resource pool with no predefined quota limits, but requests queue when model capacity saturates across all customers. This means your batch job that took 2 hours last week might take 6 hours today because another customer is running a massive fine-tuning job. docs.cloud.google

Streaming vs. Blocking and Session Memory

Streaming responses reduce time-to-first-token (TTFT) by 10-100× but slightly increase total generation time. For conversational applications, streaming is non-negotiable—human visual reaction time averages 200ms, so TTFT below this threshold makes interactions feel instant. Non-streaming requests wait for the entire response to generate before returning, which is acceptable for batch processing but catastrophic for user-facing applications. baseten

Session memory poses a subtle challenge. Most serverless platforms are stateless between requests, requiring applications to resend full conversation history with each call. A 10-turn conversation accumulates 5,000+ tokens of history that must be transmitted and billed on every subsequent request. Vertex AI's Gemini 2.0 Flash Live API addresses this with a Session Context Window that charges per turn for accumulated tokens—meaning past turns are reprocessed and billed in each new turn, up to your configured context window size. cloud.google

Tool Loops and Orchestration Overhead

When LLMs invoke external tools (databases, APIs, search engines), each invocation adds:

• JSON schema parsing cost: The model evaluates tool definitions on every request
• Network round-trip latency: External API calls add 50-500ms depending on geography
• Token amplification: Tool responses expand context for subsequent reasoning steps
• Retry storms: Poor error handling triggers exponential backoff loops that consume tokens without producing value codeant

A customer service agent that should make 2 API calls often makes 12 due to inefficient orchestration—burning tokens, adding latency, and pushing costs higher while users wait. codeant

Deep Cost Breakdown: Provider-by-Provider Analysis

AWS Bedrock: The Model Marketplace with Hidden Infrastructure Costs

AWS Bedrock positions itself as a managed model marketplace, offering Claude (Anthropic), Llama (Meta), Mistral, Cohere, and Amazon's Titan models through a unified API. Pricing is consumption-based with two primary modes: on-demand and provisioned throughput.

Cost Structure

On-Demand Pricing (Claude 3.5 Sonnet):

• Input: $0.003 per 1K tokens
• Output: $0.015 per 1K tokens
• Batch mode (50% discount): $0.0015 / $0.0075 aws.amazon

Provisioned Throughput:

• Hourly commitment: ~$39.60 per model unit
• Monthly cost for continuous availability: ~$28,000 xenoss
• Reserved capacity with predictable latency (<200ms first token) wezom

The on-demand/provisioned decision is critical. A consistent workload processing 100M tokens/month (50M input, 50M output) costs:

• On-demand: (50M × $0.003) + (50M × $0.015) = $150 + $750 = $900/month
• Provisioned (1 unit): $28,000/month with zero per-token charges

Provisioned throughput makes economic sense only above ~3-5 billion tokens/month for Claude 3.5 Sonnet, or for latency-critical applications where <200ms first token justifies the premium. Most enterprises discover this threshold through painful trial: they commit to provisioned capacity based on projected volume, then realize actual usage is 60% lower, resulting in $16,800/month of wasted spend. smiansh

Hidden Traps

Cross-region inference is free—but not really. AWS Bedrock's cross-region failover carries no additional token charges, which sounds generous. However, the underlying data transfer between regions incurs standard AWS egress fees: $0.02/GB for inter-region traffic. For a high-throughput application processing 10TB of request/response data monthly, that's $200K in "free" cross-region inference. cloudzero

CloudWatch logging compounds fast. Enabling detailed request logging (essential for debugging and cost attribution) ingests logs at $0.50/GB and stores them at $0.03/GB/month. A production chatbot handling 1M requests/day with 2KB average log payload generates 2TB monthly—$1,000 ingestion + $60 storage = $1,060/month just for observability. cloudzero

Embeddings are billed separately. RAG architectures require embeddings for every document chunk and query. Cohere Embed English costs $0.10 per 1M tokens. Processing 10M document tokens for initial indexing costs $1, but embedding 100K daily queries (20 tokens average) adds $60/month recurring—often overlooked in TCO calculations. cloudexmachina

Best-Fit Workloads

AWS Bedrock excels when:

• Your infrastructure already runs on AWS (native integration with S3, Lambda, DynamoDB)
• You need model diversity (Claude for reasoning, Titan for embeddings, Mistral for multilingual)
• Compliance requires AWS-specific certifications (FedRAMP High in GovCloud, HIPAA with KMS encryption) milvus
• You plan to use Amazon Bedrock Knowledge Bases with S3 Vectors for 90% vector storage cost reduction aws.amazon

Avoid Bedrock if:

• You need the absolute lowest per-token cost for high-volume production (GCP Vertex AI's Gemini 2.0 Flash at $0.15/$0.60 per 1M tokens is 50-75% cheaper) cloud.google
• Your application requires multi-modal generation with audio or video (Gemini's native modality support is superior) cloud.google
• You want transparent, real-time cost visibility (Bedrock's cost attribution lags by hours, making optimization reactive) cloudexmachina

GCP Vertex AI: The Multimodal Leader with Aggressive Batch Discounts

Google Vertex AI's Gemini models dominate multimodal use cases and offer the most aggressive batch pricing in the industry. Vertex AI integrates tightly with BigQuery, Dataflow, and Looker, making it the natural choice for data-intensive enterprises already running analytics on GCP.

Cost Structure

Gemini 2.5 Pro (Standard):

• Input: $1.25 per 1M tokens (≤200K context), $2.50 (>200K)
• Output: $10 per 1M tokens (≤200K), $15 (>200K)
• Batch API: 50% discount → $0.625 / $5 per 1M tokens costgoat

Gemini 2.5 Flash (Speed-Optimized):

• Input: $0.30 per 1M tokens
• Output: $2.50 per 1M tokens
• Batch API: $0.15 / $1.25 per 1M tokens cloud.google

Gemini 2.0 Flash (Cost-Optimized):

• Input: $0.15 per 1M tokens
• Output: $0.60 per 1M tokens
• Batch API: $0.075 / $0.30 per 1M tokens cloud.google

Context Caching: Gemini offers 90% cost reduction on cached tokens. For applications with large system prompts or document context reused across requests, caching delivers extraordinary savings. A 50,000-token document cached across 1,000 queries:

• Without caching: 50,000 × 1,000 × $1.25/1M = $62.50
• With caching: (50,000 × $1.25 × 1.25/1M) first request + (50,000 × 1,000 × $0.125/1M) subsequent = $0.078 + $6.25 = $6.33 total costgoat

This 90% reduction makes RAG architectures economically viable at scale.

Hidden Traps

Grounding costs stack unexpectedly. Gemini's grounding with Google Search adds $35 per 1,000 prompts after the free 1,500 daily limit. An enterprise chatbot handling 10,000 daily queries that ground 30% of responses incurs 3,000 grounding calls/day × 30 days = 90,000 calls/month. After the 45,000 free tier (1,500/day × 30), you're billed for 45,000 calls: 45 × $35 = $1,575/month for grounding alone—often exceeding token costs. cloud.google

Rate limits are opaque. Unlike AWS or Azure, Vertex AI doesn't publish hard TPM limits. Instead, batch jobs queue dynamically when model capacity saturates. This makes capacity planning frustrating: a batch job that completed in 2 hours last week might take 8 hours today with no visibility into why. For time-sensitive analytics or daily reporting pipelines, this unpredictability forces over-provisioning of buffer time. docs.cloud.google

Long-context pricing doubles. Processing >200K token contexts (Gemini 2.5 Pro's strength) costs 2× the base rate: $2.50 input / $15 output per 1M tokens vs. $1.25 / $10 for ≤200K. An application processing 1M-token documents pays $2.50 input for each, which quickly eclipses the value of long-context support. Smart architects chunk documents into <200K segments to avoid this penalty, sacrificing the "long context window" marketing promise for economic sanity. finout

Best-Fit Workloads

GCP Vertex AI excels when:

• You need multimodal capabilities (audio, video, image alongside text) without separate API calls cloud.google
• Your data lives in BigQuery and you want to run inference without data movement (Vertex AI in BigQuery eliminates egress) xenoss
• Batch processing dominates your workload (50% discounts + 90% context caching = unbeatable economics for async pipelines) costgoat
• GDPR compliance requires EU data residency (europe-west12, de-central1 with regional processing guarantees) datastudios

Avoid Vertex AI if:

• You need transparent capacity guarantees for real-time applications (dynamic batch queuing is unacceptable for user-facing services)
• Your application relies on OpenAI-specific function calling conventions (Gemini supports 1,024 functions vs. OpenAI's 128, but schema compatibility differs) ruh
• You're cost-sensitive to grounding/search augmentation (Google Search grounding costs add up faster than RAG with self-hosted vector DBs)

Azure OpenAI: The Enterprise Fortress with PTU Lock-In

Azure OpenAI provides exclusive access to OpenAI models (GPT-4o, o1, DALL-E 3) within Microsoft's enterprise cloud, wrapped with SOC 2, ISO 27001, and HIPAA compliance guarantees. For organizations standardized on Microsoft 365 and Azure services, this integration is seamless—but the pricing complexity is unmatched.

Cost Structure

GPT-4o (Pay-as-You-Go):

• Standard: $0.005 / $0.015 per 1K tokens azure-noob
• Global deployment: $2.50 / $10 per 1M tokens finout
• Batch API: 50% discount helicone

GPT-4o mini (Cost-Optimized):

• Standard: $0.00015 / $0.0006 per 1K tokens azure-noob

Provisioned Throughput Units (PTU):

• Minimum commitment: $2,448/month azure-noob
• Hourly pricing available for flexibility iaxservices
• Reservations (monthly/yearly): Up to 70% savings vs. pay-as-you-go azure.github

PTU economics are deceptive. Microsoft positions PTU as "predictable performance for production workloads," which is true—but the breakeven analysis is brutal. A GPT-4o deployment processing 100M tokens/month (50M input, 50M output):

• Pay-as-you-go: (50M × $2.50/1M) + (50M × $10/1M) = $125 + $500 = $625/month
• PTU (minimum): $2,448/month

You need to process ~400M tokens/month just to break even on the minimum PTU commitment. Enterprises consistently over-provision PTU capacity because forecasting is guesswork. One data point: 85% of organizations misestimate AI costs by more than 10%, with 24% missing by over 50%. These organizations commit to PTU reservations based on pilot usage, then face 75% idle capacity when production ramps slower than expected. cio

Hidden Traps

Premium APIM Gateway costs are shocking. Deploying Azure OpenAI behind Azure API Management (recommended for rate limiting, caching, and multi-region failover) requires Premium tier for VNET integration: $2,795/month per unit. This fixed cost dwarfs token charges for most applications and is rarely mentioned in TCO discussions. truefoundry

Data residency isn't automatic. Azure OpenAI's HIPAA compliance requires deploying in US regions and configuring private endpoints—not default behavior. Many enterprises assume Azure OpenAI is "HIPAA-compliant by default" and discover during audits that their deployment is non-compliant because they didn't explicitly enable the BAA through the Data Protection Addendum. learn.microsoft

TPM and RPM limits interact unexpectedly. Azure enforces both TPM (tokens per minute) and RPM (requests per minute) limits, with RPM set at 6 per 1,000 TPM. A 100,000 TPM deployment gets 600 RPM. If your application sends 15 requests in one second (15 req/sec = 900 RPM), you're throttled immediately—even if each request is only 100 tokens. This interaction catches developers by surprise during load testing. clemenssiebler

Best-Fit Workloads

Azure OpenAI excels when:

• Your organization is a Microsoft shop (Teams, Power Platform, Azure DevOps integration is unmatched) dev
• You require GPT-4o or o1 specifically (exclusive access unavailable elsewhere) umbrellacost
• HIPAA compliance is non-negotiable and you have budget for Premium APIM + BAA configuration learn.microsoft
• You can accurately forecast >400M tokens/month to justify PTU reservations techcommunity.microsoft

Avoid Azure OpenAI if:

• You're cost-sensitive and usage is unpredictable (PTU lock-in + minimum commitments + APIM Gateway costs spiral fast)
• You need model diversity (Azure only offers OpenAI models; no Claude, Gemini, or Mistral access) intellias
• Your application requires sub-200ms first token latency at scale (pay-as-you-go mode doesn't guarantee capacity like AWS Provisioned Throughput) wezom

Cold Start, Concurrency & Latency Analysis: What Really Happens Under Load

Marketing materials promise "elastic scaling" and "millisecond latency." Production reality is messier.

Cold Start Latency Breakdown

Platform	Cold Start (Typical)	Warm Latency (P50)	First Token Guarantee	Mitigation
AWS Bedrock On-Demand	3-10 seconds linkedin	30-50ms per token arxiv	None	Provisioned Throughput (<200ms) wezom
AWS Bedrock Provisioned	<200ms wezom	30-50ms per token	<200ms TTFT SLA	Pre-warmed capacity
Vertex AI Gemini	2-8 seconds ardor	25-40ms per token ardor	None	Batch API (async queuing)
Azure OpenAI Pay-As-You-Go	2-7 seconds clemenssiebler	30-60ms per token learn.microsoft	None	PTU (reserved capacity)
Azure OpenAI PTU	<500ms iaxservices	30-60ms per token	Capacity reserved	Pre-allocated compute

Cold starts matter most for:

• Customer-facing chatbots: 3-10 second delays before first response feel broken to users
• Real-time analytics dashboards: Queries triggered by user clicks can't wait 8 seconds for model warm-up
• Alert systems: Payment failure notifications delayed by 3 seconds undermine trust linkedin

Salesforce documented that onboarding models to AWS Bedrock reduced cold start impact from 6-8 weeks (provisioning GPUs manually) to 1-2 weeks (managed service), but cold starts remained a challenge requiring extensive testing and provisioned capacity planning. engineering.salesforce

Burst Handling and Throttling

AWS Bedrock throttles with HTTP 429 errors when burst concurrency exceeds allocated capacity. Cross-region inference provides overflow capacity from multiple regions, but the failover logic is opaque—you discover throttling through monitoring, not predictive capacity planning. aws.amazon

Azure OpenAI evaluates RPM over 1-10 second windows. If your traffic is bursty (common in B2B SaaS with enterprise customers batch-processing reports), you hit 429 errors even when your average RPM is 50% below the limit. The solution: implement client-side queueing with exponential backoff, adding 3-6 seconds of delay between requests to smooth bursts. techcommunity.microsoft

Vertex AI queues batch requests dynamically when model capacity saturates. Unlike AWS/Azure, there's no 429 error—your job just waits. One production team reported their nightly ETL pipeline (expected: 2 hours) took 9 hours because Vertex AI queued requests behind another customer's workload. There's no visibility into queue position or expected wait time. docs.cloud.google

Retry Storms and Token Waste

Poor error handling creates retry storms: a single failed tool call triggers exponential backoff that makes 5-10 retry attempts, each consuming tokens without producing value. In agentic workflows, this compounds: one failed retrieval step can cause the entire agent loop to retry, multiplying token consumption by 3-5×. codeant

Production guidance:

• Implement circuit breakers that fail fast after 2-3 retries
• Use idempotency tokens to prevent duplicate processing
• Log failed requests with full context (prompt, error code, latency) for post-mortem analysis codeant

Function Calling & Agentic Cost Explosion: The Hidden Budget Killer

Function calling (also called tool use) lets LLMs invoke external APIs, databases, and search engines. It's the difference between a chatbot that answers questions and an agent that takes action. It's also where costs explode.

Token Overhead Anatomy

Every tool defined in your function calling schema consumes input tokens on every request—even when the tool isn't invoked. Ten tools with 500-token schemas each = 5,000 tokens of overhead before the user's question is processed. codeant

Example: Customer Service Agent with 8 Tools

• Order lookup (600 tokens)
• Inventory check (450 tokens)
• Shipping status (550 tokens)
• Return policy (700 tokens)
• Escalation workflow (650 tokens)
• Knowledge base search (800 tokens)
• CRM update (500 tokens)
• Email notification (400 tokens)

Total schema overhead: 4,650 tokens per request

At GPT-4o pricing ($2.50/1M input tokens), this overhead costs $0.011625 per request. For 100,000 daily requests, that's $1,162.50/day = $34,875/month just to define tools that might not even be called.

Agentic Loop Amplification

Agent workflows chain multiple tool calls together. Each call generates intermediate reasoning tokens, expands context for subsequent steps, and risks retry storms when tools fail. A seemingly simple agent task:

User request: "Find the cheapest hotel in Tokyo for next weekend and book it"

Agent execution:

1. Tool call: Search hotels (200 tokens reasoning + 2,000 tokens results)
2. Tool call: Check availability (150 tokens reasoning + 500 tokens response)
3. Tool call: Compare prices (300 tokens reasoning + 1,500 tokens comparison)
4. Tool call: Validate credit card (100 tokens reasoning + 50 tokens response)
5. Tool call: Submit booking (200 tokens reasoning + 300 tokens confirmation)

Total tokens: ~5,300 tokens for a task that would take a human 3 minutes

If step 4 fails (credit card API timeout), the agent retries—but most implementations naively restart from step 1, wasting tokens on steps 1-3 that already succeeded. Three retries = 15,900 tokens = $0.040 per failed booking attempt at GPT-4o pricing.

Industry data confirms this: agentic AI systems cost 3-10× more than simple chat applications because of token amplification, retry overhead, and orchestration complexity. One enterprise AI lead reported: "Our chatbot cost $2,000/month. When we added agent capabilities, the bill jumped to $14,000/month with no change in user volume." linkedin

JSON Schema Inflation

LLMs are verbose when generating tool call JSON. OpenAI's function calling requires:

{
  "name": "search_hotels",
  "arguments": {
    "location": "Tokyo, Japan",
    "check_in": "2026-01-25",
    "check_out": "2026-01-27",
    "guests": 2,
    "sort_by": "price_ascending"
  }
}

This 150-token JSON structure is generated as output tokens (billed at 4× the input rate for GPT-4o: $10 vs. $2.50 per 1M tokens). Ten tool calls in an agent workflow = 1,500 output tokens = $0.015 in JSON overhead alone—before processing any tool results. finout

Optimization Strategies

1. Reduce tool schema verbosity: Strip descriptions to bare minimums. "Searches hotels by location and date" can be "Search hotels" without accuracy loss, saving 20-40 tokens per tool.

2. Implement tool result caching: If an agent calls "get_weather('Tokyo')" twice in one conversation, cache the result and reuse it instead of calling the API again. requesty

3. Batch tool calls: Design APIs that accept multiple operations in one request. Instead of three separate database queries (3 tool calls), send one batched query (1 tool call). codeant

4. Use smaller models for tool orchestration: GPT-4o mini ($0.15/$0.60 per 1M tokens) handles tool orchestration adequately for 80% of cases. Reserve GPT-4o for final response generation. ai.koombea

5. Implement checkpoint-resume logic: When retries occur, resume from the last successful step instead of restarting the entire chain. codeant

Compliance & Data Residency: Where Your Data Actually Goes

Enterprise AI deployments face regulatory requirements that constrain architecture choices. Marketing promises of "GDPR compliance" obscure critical implementation details.

Data Processing vs. Data Storage

All three platforms process data in-region when configured correctly, but logging and model training data follow different rules:

Requirement	AWS Bedrock	GCP Vertex AI	Azure OpenAI
Input data processing	Regional (us-east-1, eu-west-1, etc.) aws.amazon	Regional with EU lock (europe-west12, de-central1) datastudios	Regional (must select US for HIPAA) learn.microsoft
Output data storage	Transient (not stored post-response) milvus	Transient (not stored unless opted into feedback) alumio	Transient (30-day abuse monitoring unless disabled) umbrellacost
Log storage	CloudWatch in same region cloudzero	Cloud Logging in same region finout	Azure Monitor in same region azure.microsoft
Model training data	Not used for training unless opted into custom models aws.amazon	Not used for training (enterprise Vertex AI) datastudios	Not used for training (Azure OpenAI) learn.microsoft

HIPAA Configuration Reality

All three platforms claim HIPAA eligibility, but:

AWS Bedrock: Requires signing a BAA, enabling KMS encryption for data at rest, configuring VPC endpoints for private networking, and deploying in FedRAMP High regions for government workloads. The BAA doesn't activate automatically—you must request it through your AWS account team. aws.amazon

GCP Vertex AI: Requires enabling the regulated-data flag at the project level, pairing with a Google Cloud BAA, and using Private Service Connect (PSC) to isolate network traffic. The flag must be set before deployment; retrofitting existing projects is unsupported. datastudios

Azure OpenAI: Requires deploying in US regions, enabling private endpoints, and validating that your licensing agreement (Enterprise Agreement or CSP) includes the BAA via Microsoft's Data Protection Addendum. The BAA is automatic if your licensing is correct, but many enterprises assume compliance without verifying DPA coverage—then fail audits. learn.microsoft

GDPR Data Residency

AWS Bedrock: Supports EU regions (eu-west-1 Frankfurt) with data processing guarantees, but cross-region inference may route requests to other regions for capacity. For strict GDPR compliance, disable cross-region inference and accept throttling risk. aws.amazon

GCP Vertex AI: Provides the strongest guarantees with region-locking to europe-west12 (Belgium) or de-central1 (Germany). Enterprise Workspace plans enable data residency controls that prevent data from leaving configured regions—critical for GDPR's data localization requirements. alumio

Azure OpenAI: Offers regional deployments but doesn't publish explicit data residency guarantees for all compliance frameworks. GDPR compliance is supported through Microsoft's DPA, but verifying that logs, telemetry, and temporary data stay in-region requires manual validation with Azure support. learn.microsoft

Compliance Certification Matrix

Standard	AWS Bedrock	GCP Vertex AI	Azure OpenAI
HIPAA	Eligible with BAA + KMS milvus	Supported with BAA + regulated flag datastudios	Eligible with BAA via DPA learn.microsoft
GDPR	Compliant with regional controls milvus	Compliant with EU region-locking datastudios	Compliant via DPA learn.microsoft
SOC 2 Type II	In scope binyam	Certified datastudios	Certified learn.microsoft
ISO 27001/27701	In scope milvus	Certified datastudios	Certified learn.microsoft
FedRAMP High	Authorized (GovCloud only) aws.amazon	Authorized (Vertex AI + Search) executivebiz	In progress (Azure Gov) learn.microsoft

Decision Matrix: Matching Platforms to Enterprise Requirements

Requirement	Best Choice	Why
Lowest cost at scale (>1B tokens/month)	GCP Vertex AI (Gemini 2.0 Flash)	$0.15/$0.60 per 1M tokens is 50-75% cheaper than alternatives; batch API + context caching compound savings cloud.google
Best EU compliance (GDPR)	GCP Vertex AI	Region-locking to europe-west12/de-central1 with no cross-border transfers datastudios
Lowest latency APAC	AWS Bedrock (ap-southeast-1 Singapore)	Provisioned Throughput guarantees <200ms TTFT; Bedrock available in more APAC regions than Vertex AI aws.amazon
Best for agentic workflows	Anthropic Claude via Bedrock	Interleaved thinking + parallel tool execution reduces orchestration overhead; 72.7% SWE-bench score ruh
Best for RAG at scale	AWS Bedrock + S3 Vectors	90% vector storage cost reduction vs. specialized vector DBs; subsecond query performance aws.amazon
Microsoft 365 integration	Azure OpenAI	Native Teams, Power Platform, Azure DevOps integration dev
Multimodal (audio/video/image)	GCP Vertex AI (Gemini 2.5 Pro)	Native multimodal support without separate API calls cloud.google
Predictable performance (SLA)	AWS Bedrock Provisioned	<200ms TTFT guarantee with reserved capacity wezom
Cost-sensitive pilot/MVP	GCP Vertex AI (Gemini 2.0 Flash free tier)	1,500 requests/day free; batch mode for production scale cloud.google
HIPAA healthcare	Azure OpenAI (if already on Azure)	Automatic BAA via DPA for existing EA customers learn.microsoft

Cost Simulation Examples: Real Numbers from Production Workloads

Scenario 1: Startup Chatbot (10,000 Daily Queries)

Workload:

• 10,000 queries/day × 30 days = 300,000 queries/month
• Average: 500 input tokens + 150 output tokens per query
• Total: 150M input + 45M output tokens/month

AWS Bedrock (Claude 3.5 Sonnet on-demand):

• Input: 150M × $0.003/1K = $450
• Output: 45M × $0.015/1K = $675
• Total: $1,125/month (tokens only)
• Add CloudWatch logs (600MB): +$300
• All-in: $1,425/month

GCP Vertex AI (Gemini 2.0 Flash standard):

• Input: 150M × $0.15/1M = $22.50
• Output: 45M × $0.60/1M = $27
• Total: $49.50/month (tokens only)
• Add Cloud Logging (600MB): +$300
• All-in: $349.50/month

Azure OpenAI (GPT-4o mini pay-as-you-go):

• Input: 150M × $0.00015/1K = $22.50
• Output: 45M × $0.0006/1K = $27
• Total: $49.50/month (tokens only)
• Add Azure Monitor: +$300
• All-in: $349.50/month

Winner: Tie between GCP Vertex AI and Azure OpenAI at ~$350/month

Scenario 2: Mid-Scale SaaS RAG System (500,000 Daily Queries)

Workload:

• 500,000 queries/day × 30 days = 15M queries/month
• Average: 1,050 input tokens (50 query + 1,000 retrieved context) + 200 output tokens
• Total: 15.75B input + 3B output tokens/month
• Vector storage: 10M documents, 500M embeddings
• Nightly re-indexing: 500M embedding tokens/month

AWS Bedrock (Claude 3.5 Sonnet batch mode):

• Input: 15.75B × $0.0015/1K = $23,625
• Output: 3B × $0.0075/1K = $22,500
• Embeddings (Cohere): 500M × $0.10/1M = $50
• S3 Vectors storage + queries: $500 (vs. $5,000 for OpenSearch Serverless) aws.amazon
• CloudWatch (30GB): $15,300
• Total: $61,975/month

GCP Vertex AI (Gemini 2.5 Flash batch + context caching):

• Input (90% cached): 15.75B × 0.1 × $0.15/1M = $236
• Input (10% uncached): 15.75B × 0.1 × $0.15/1M × 1.25 = $295
• Output: 3B × $1.25/1M = $3,750
• Embeddings (Vertex text-embedding): 500M × $0.0001/1K = $50
• Vertex Vector Search: $1,500
• Cloud Logging (30GB): $15,300
• Total: $21,131/month

Azure OpenAI (GPT-4o batch + PTU mix):

• Input: 15.75B × $1.25/1M = $19,687.50
• Output: 3B × $5/1M = $15,000
• Embeddings (text-embedding-3-small): 500M × $0.00001/1K = $5
• Vector storage (Cosmos DB): $2,000
• Azure Monitor (30GB): $15,300
• Total: $51,992.50/month

Winner: GCP Vertex AI at $21,131/month (66% cheaper than Azure, 66% cheaper than AWS)

The context caching and batch discount combination is economically unbeatable for RAG workloads with repeated context.

Scenario 3: Enterprise Agentic System (1M Tool Calls/Month)

Workload:

• 200,000 agent sessions/month
• Average: 5 tool calls per session = 1M tool calls/month
• Tool schema overhead: 5,000 tokens per request
• Per tool call: 200 reasoning tokens + 800 result tokens
• Total per session: 5,000 (schema) + 5 × (200 + 800) = 10,000 tokens input + 5 × 300 = 1,500 tokens output
• Monthly: 2B input + 300M output tokens

AWS Bedrock (Claude 3.5 Sonnet on-demand with prompt caching):

• Input (90% schema cached): 2B × 0.1 × $0.003/1K = $600
• Input (10% uncached): 2B × 0.1 × $0.003/1K × 1.25 = $750
• Output: 300M × $0.015/1K = $4,500
• Total: $5,850/month

GCP Vertex AI (Gemini 2.5 Flash + context caching):

• Input (90% schema cached): 2B × 0.1 × $0.30/1M = $60
• Input (10% uncached): 2B × 0.1 × $0.30/1M × 1.25 = $75
• Output: 300M × $2.50/1M = $750
• Total: $885/month

Azure OpenAI (GPT-4o + prompt caching):

• Input (90% cached): 2B × 0.1 × $2.50/1M = $500
• Input (10% uncached): 2B × 0.1 × $2.50/1M × 1.25 = $625
• Output: 300M × $10/1M = $3,000
• Total: $4,125/month

Winner: GCP Vertex AI at $885/month (78% cheaper than Azure, 85% cheaper than AWS)

Final Verdict: Opinionated Recommendations

Best overall for cost-conscious enterprises: GCP Vertex AI. The combination of aggressive batch discounts (50%), context caching (90% savings), and industry-leading per-token pricing makes Vertex AI the economic winner for production workloads processing >100M tokens/month. The multimodal capabilities and BigQuery integration are bonuses. The tradeoff: you accept dynamic batch queuing and opaque capacity planning.

Best for AWS-native architectures: AWS Bedrock. If your infrastructure already runs on AWS, Bedrock's native integration with S3, Lambda, CloudWatch, and DynamoDB eliminates inter-cloud egress costs and simplifies compliance. Provisioned Throughput is expensive but delivers guaranteed <200ms TTFT—essential for latency-critical applications. S3 Vectors with Bedrock Knowledge Bases provides 90% vector storage cost reduction, making RAG economically viable at scale. The tradeoff: you pay a 2-3× premium over Vertex AI for on-demand token processing.

Best for Microsoft-centric enterprises: Azure OpenAI. If your organization lives in Teams, Power Platform, and Azure DevOps, Azure OpenAI's native integration justifies the cost premium. PTU reservations with 70% discounts can match Vertex AI economics if you forecast accurately (big if). HIPAA compliance via automatic BAA is the smoothest of the three platforms for healthcare workloads. The tradeoff: you're locked into OpenAI models (no Claude, Gemini, Mistral), and Premium APIM Gateway adds $2,795/month fixed cost for production-grade deployments.

Best for agentic workflows: Anthropic Claude via AWS Bedrock. Claude's interleaved thinking and parallel tool execution reduce orchestration overhead, and the 72.7% SWE-bench Verified score demonstrates superior coding and reasoning capabilities. Prompt caching with 90% savings on tool schemas makes agentic cost explosions manageable. The tradeoff: Claude 3.5 Sonnet costs 2× more than Gemini 2.5 Flash per token, so the efficiency gains must offset higher unit costs.

Avoid if budget is your primary constraint: Azure OpenAI pay-as-you-go. GPT-4o at $2.50/$10 per 1M tokens is 16-66× more expensive than Gemini 2.0 Flash ($0.15/$0.60). Unless you require GPT-4o specifically (OpenAI alignment, brand recognition, ecosystem tooling), you're overpaying for capabilities that Gemini 2.5 Flash delivers at a fraction of the cost.

Conversion: Next Steps for Your Architecture

Building production-grade serverless LLM infrastructure requires aligning technical capabilities with business constraints. The platforms analyzed here represent different architectural philosophies: AWS prioritizes integration depth, Google optimizes for cost efficiency and multimodal flexibility, and Azure delivers enterprise compliance and Microsoft ecosystem lock-in.

For most enterprises, the right answer is a hybrid strategy: Use GCP Vertex AI for high-volume batch processing and RAG workloads where 50% batch discounts and 90% context caching deliver unbeatable economics. Deploy AWS Bedrock for latency-critical user-facing applications where Provisioned Throughput's <200ms TTFT guarantees justify the premium. Reserve Azure OpenAI for Microsoft-native use cases where Teams/Power Platform integration creates disproportionate value.

If you're evaluating these platforms for 2026 deployments and need help with:

• Architecture audits: Reviewing existing LLM infrastructure for hidden cost drivers (vector storage, monitoring, egress, retry storms)
• Cost modeling: Building detailed TCO projections that account for infrastructure, not just token pricing
• Migration planning: Designing phased transitions that minimize risk while capturing immediate cost savings

The 96% cost overrun rate for GenAI deployments isn't inevitable—it's the result of treating LLM inference like traditional serverless compute. Enterprises that invest in accurate cost modeling, compliance-aware architecture, and continuous optimization from day one avoid the $1.2M budget blowouts that plague late movers. The gap between leaders and laggards in 2026 won't be determined by model selection—it will be defined by who understood the real cost structure and architected accordingly.

The platforms have done their part: they've built the infrastructure. Now it's your turn to build it right.