API Landscape & Architecture

1. The LLM API Ecosystem

The modern LLM API ecosystem is built around a surprisingly simple pattern: you send a sequence of messages (a conversation) to an HTTP endpoint, and the model returns a completion. Despite this conceptual simplicity, each provider has evolved distinct conventions, capabilities, and pricing models that shape how you build applications.

The major commercial providers include OpenAI, Anthropic, and Google. Each offers hosted inference with usage-based pricing, typically measured in tokens. Beyond these, cloud platforms like AWS Bedrock and Azure OpenAI provide enterprise wrappers that add compliance, networking, and billing features on top of the same underlying models. For open-source models, serving frameworks like vLLM, TGI, and Ollama expose OpenAI-compatible endpoints, creating a de facto standard.

2. OpenAI Chat Completions API

The OpenAI Chat Completions API established the dominant pattern for LLM APIs. You send a list of messages, each with a role (system, user, or assistant) and content, along with generation parameters. The API returns a completion object containing the model's response, token usage counts, and metadata.

2.1 Core Request Structure

The key parameters that control generation behavior are:

• model: The model identifier (e.g., gpt-4o, o4-mini)
• messages: The conversation history as an array of role/content objects
• temperature: Controls randomness (0.0 = deterministic, 2.0 = maximum randomness)
• top_p: Nucleus sampling threshold (alternative to temperature)
• max_tokens: Maximum number of tokens to generate
• frequency_penalty: Penalizes tokens based on how often they have appeared (range: -2.0 to 2.0)
• presence_penalty: Penalizes tokens that have appeared at all (range: -2.0 to 2.0)

from openai import OpenAI

client = OpenAI()  # reads OPENAI_API_KEY from environment

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": "You are a helpful coding assistant."},
        {"role": "user", "content": "Explain Python list comprehensions in two sentences."}
    ],
    temperature=0.7,
    max_tokens=150
)

print(response.choices[0].message.content)
print(f"Tokens used: {response.usage.prompt_tokens} input, "
      f"{response.usage.completion_tokens} output")

2.2 Streaming Responses with Server-Sent Events

For interactive applications, waiting for the entire response to complete before displaying anything creates a poor user experience. Streaming delivers tokens as they are generated using the Server-Sent Events (SSE) protocol. Each chunk arrives as a small JSON object containing the delta (the new content since the last chunk).

from openai import OpenAI

client = OpenAI()

stream = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "user", "content": "Write a haiku about APIs."}
    ],
    stream=True
)

collected_content = ""
for chunk in stream:
    delta = chunk.choices[0].delta
    if delta.content:
        collected_content += delta.content
        print(delta.content, end="", flush=True)

print(f"\n\nFull response: {collected_content}")

2.3 The Batch API

OpenAI's Batch API allows you to submit large collections of requests (up to 50,000) that are processed asynchronously within a 24-hour window. The key advantage is a 50% cost reduction compared to synchronous API calls. This makes it ideal for tasks like dataset labeling, bulk classification, or evaluation pipelines where real-time responses are unnecessary.

from openai import OpenAI
import json

client = OpenAI()

# Step 1: Create a JSONL file with batch requests
requests = [
    {
        "custom_id": f"review-{i}",
        "method": "POST",
        "url": "/v1/chat/completions",
        "body": {
            "model": "gpt-4o-mini",
            "messages": [
                {"role": "system", "content": "Classify the sentiment as positive, negative, or neutral."},
                {"role": "user", "content": review}
            ],
            "max_tokens": 10
        }
    }
    for i, review in enumerate(["Great product!", "Terrible service.", "It was okay."])
]

with open("batch_input.jsonl", "w") as f:
    for req in requests:
        f.write(json.dumps(req) + "\n")

# Step 2: Upload and submit the batch
batch_file = client.files.create(file=open("batch_input.jsonl", "rb"), purpose="batch")
batch_job = client.batches.create(input_file_id=batch_file.id, endpoint="/v1/chat/completions",
                                   completion_window="24h")
print(f"Batch submitted: {batch_job.id}, status: {batch_job.status}")

3. Anthropic Messages API

Anthropic's Messages API follows a similar conversational pattern but introduces several distinctive design choices. System prompts are a separate top-level parameter rather than a message role. The API supports prompt caching (which can reduce costs by up to 90% for repeated prefixes), extended thinking for complex reasoning, and a robust tool use system.

3.1 Core Differences from OpenAI

The most notable architectural differences include:

• System prompt: Passed as a top-level system parameter, not inside the messages array
• Max tokens: Required (not optional), specified as max_tokens
• Content blocks: Responses use content blocks (an array of typed objects) rather than a single string
• Prompt caching: Explicit cache control markers let you cache expensive prefixes
• Extended thinking: Built-in chain-of-thought reasoning that exposes the model's thinking process

import anthropic

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from environment

response = client.messages.create(
    model="claude-sonnet-4-20250514",
    max_tokens=200,
    system="You are a helpful coding assistant. Be concise.",
    messages=[
        {"role": "user", "content": "Explain Python list comprehensions in two sentences."}
    ]
)

print(response.content[0].text)
print(f"Tokens: {response.usage.input_tokens} input, "
      f"{response.usage.output_tokens} output")
print(f"Stop reason: {response.stop_reason}")

3.2 Prompt Caching

Anthropic's prompt caching feature is particularly valuable when you repeatedly send requests with a shared prefix (for example, a long system prompt or a large document in context). By marking content blocks with cache_control, you tell the API to cache that prefix. Subsequent requests that share the same cached prefix receive a significant discount on input token costs and reduced latency.

response = client.messages.create(
    model="claude-sonnet-4-20250514",
    max_tokens=300,
    system=[
        {
            "type": "text",
            "text": "You are an expert on the Python standard library. " * 50,  # Long system prompt
            "cache_control": {"type": "ephemeral"}  # Cache this prefix
        }
    ],
    messages=[
        {"role": "user", "content": "What is the difference between os.path and pathlib?"}
    ]
)

# Check cache performance
print(f"Cache creation tokens: {response.usage.cache_creation_input_tokens}")
print(f"Cache read tokens: {response.usage.cache_read_input_tokens}")

4. Google Gemini API

Google's Gemini API uses a generateContent endpoint that shares the same conversational pattern but employs different terminology. Messages are called "contents," roles are "user" and "model" (not "assistant"), and the API supports unique features like grounding (connecting to Google Search), code execution, and native multimodal input.

from google import genai

client = genai.Client()  # reads GOOGLE_API_KEY from environment

response = client.models.generate_content(
    model="gemini-2.5-flash",
    contents="Explain Python list comprehensions in two sentences.",
    config=genai.types.GenerateContentConfig(
        temperature=0.7,
        max_output_tokens=150,
        system_instruction="You are a helpful coding assistant."
    )
)

print(response.text)
print(f"Tokens: {response.usage_metadata.prompt_token_count} input, "
      f"{response.usage_metadata.candidates_token_count} output")

5. Enterprise Wrappers: AWS Bedrock and Azure OpenAI

Enterprise cloud providers wrap the underlying models with additional infrastructure for security, compliance, and billing. AWS Bedrock provides access to models from Anthropic, Meta, Mistral, and others through a unified API with IAM authentication, VPC endpoints, and consolidated AWS billing. Azure OpenAI deploys the same OpenAI models within Microsoft's cloud, adding features like content filtering, private networking, and regional data residency.

The following example shows how to call Claude on AWS Bedrock using boto3. The key difference from the direct Anthropic API is authentication (IAM credentials instead of an API key) and the endpoint configuration:

import boto3
import json

# Bedrock uses IAM credentials (configured via AWS CLI or environment variables)
bedrock = boto3.client(
    service_name="bedrock-runtime",
    region_name="us-east-1"
)

body = json.dumps({
    "anthropic_version": "bedrock-2023-05-31",
    "max_tokens": 256,
    "messages": [
        {"role": "user", "content": "Explain what an API gateway does in two sentences."}
    ]
})

response = bedrock.invoke_model(
    modelId="anthropic.claude-sonnet-4-20250514-v1:0",
    body=body,
    contentType="application/json",
    accept="application/json"
)

result = json.loads(response["body"].read())
print(result["content"][0]["text"])

6. Open-Source Serving: The OpenAI-Compatible Pattern

A powerful convention has emerged in the open-source ecosystem: serving frameworks expose an API that mimics the OpenAI Chat Completions format. This means you can point the OpenAI Python SDK at any compatible server by changing the base_url, and your existing code works without modification. Frameworks like vLLM, Text Generation Inference (TGI), and Ollama all support this pattern.

from openai import OpenAI

# Point the OpenAI client at a local vLLM server
client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="not-needed"  # Local servers often skip auth
)

response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[
        {"role": "user", "content": "Explain Python list comprehensions in two sentences."}
    ],
    temperature=0.7,
    max_tokens=150
)

print(response.choices[0].message.content)

7. Provider Comparison

The following table summarizes the key differences across the major providers. Understanding these differences helps you choose the right provider for your use case and design portable abstractions.

8. Authentication and Rate Limits

Every provider requires authentication, typically via an API key passed in the Authorization header. Enterprise providers like Azure OpenAI also support Azure Active Directory tokens. Rate limits are enforced at multiple levels: requests per minute (RPM), tokens per minute (TPM), and sometimes requests per day (RPD). Exceeding these limits results in HTTP 429 (Too Many Requests) responses.

Rate limit headers returned with each response tell you how much capacity remains. Monitoring these headers allows you to implement proactive throttling rather than waiting for 429 errors:

• x-ratelimit-remaining-requests: Requests remaining in the current window
• x-ratelimit-remaining-tokens: Tokens remaining in the current window
• x-ratelimit-reset-requests: Time until the request limit resets
• x-ratelimit-reset-tokens: Time until the token limit resets

9. Choosing Between Providers

The right provider depends on your specific requirements. Consider these factors when making your decision:

• Model capability: For the most complex reasoning tasks, compare the latest models from each provider on your specific use case. Benchmarks help, but real evaluation on your data is essential.
• Pricing: Input and output tokens are priced differently. Anthropic and Google tend to offer competitive pricing for large-context workloads. OpenAI's Batch API cuts costs by 50% for non-real-time tasks.
• Context window: Gemini supports up to 1M tokens; Anthropic's Claude supports 200K; OpenAI's GPT-4o supports 128K. Longer contexts enable retrieval-free processing of large documents.
• Compliance: Enterprise wrappers (Bedrock, Azure) offer HIPAA, SOC 2, and data residency guarantees that direct API access may not.
• Latency: Test time-to-first-token and total generation time from your deployment region. Geographic proximity to provider data centers matters.