Structured Output & Tool Integration

1. The Structured Output Problem

Consider a simple task: extract the name, email, and sentiment from a customer support message. If you ask an LLM to do this with a plain text prompt, you might get the information scattered across prose, formatted inconsistently, or wrapped in unnecessary explanation. To build reliable pipelines, you need the model to return a specific JSON structure, every time, without deviation.

There are three levels of structured output enforcement, each with increasing reliability:

1. Prompt-based: You ask the model to return JSON in the system prompt. This works most of the time, but the model can still return malformed JSON or add commentary outside the JSON block.
2. JSON mode: The API guarantees the response is valid JSON, but does not enforce a specific schema. The model chooses the keys and structure.
3. Schema-constrained: The API guarantees the response matches a specific JSON Schema. This is the most reliable approach, as it enforces both validity and structure at the decoding level.

2. JSON Mode and Response Schemas

2.1 OpenAI JSON Mode

OpenAI's simplest structured output option is JSON mode, activated by setting response_format={"type": "json_object"}. This guarantees the response is valid JSON but does not enforce any particular schema. You must include the word "JSON" in your prompt for this mode to work reliably.

from openai import OpenAI

client = OpenAI()

response = client.chat.completions.create(
    model="gpt-4o",
    response_format={"type": "json_object"},
    messages=[
        {"role": "system", "content": "Extract contact info. Return JSON with keys: name, email, sentiment."},
        {"role": "user", "content": "Hi, I'm Sarah Chen (sarah@example.com) and I love your product!"}
    ]
)

import json
data = json.loads(response.choices[0].message.content)
print(json.dumps(data, indent=2))

2.2 OpenAI Structured Outputs with JSON Schema

For maximum reliability, OpenAI's Structured Outputs feature lets you provide a complete JSON Schema. The model is constrained at the decoding level to produce output that conforms exactly to your schema. This means required fields are always present, types are always correct, and enum values are always valid.

from openai import OpenAI

client = OpenAI()

response = client.chat.completions.create(
    model="gpt-4o",
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "contact_extraction",
            "strict": True,
            "schema": {
                "type": "object",
                "properties": {
                    "name": {"type": "string", "description": "Full name of the person"},
                    "email": {"type": "string", "description": "Email address"},
                    "sentiment": {
                        "type": "string",
                        "enum": ["positive", "negative", "neutral"],
                        "description": "Overall sentiment"
                    },
                    "confidence": {
                        "type": "number",
                        "description": "Confidence score between 0 and 1"
                    }
                },
                "required": ["name", "email", "sentiment", "confidence"],
                "additionalProperties": False
            }
        }
    },
    messages=[
        {"role": "user", "content": "Hi, I'm Sarah Chen (sarah@example.com) and I love your product!"}
    ]
)

import json
data = json.loads(response.choices[0].message.content)
print(json.dumps(data, indent=2))
print(f"Type of confidence: {type(data['confidence']).__name__}")
print(f"Sentiment is valid enum: {data['sentiment'] in ['positive', 'negative', 'neutral']}")

3. Pydantic and the Instructor Library

While raw JSON Schema works, defining schemas as dictionaries is verbose and error-prone. The Instructor library bridges this gap by letting you define your output schema as a Pydantic model (a Python class with typed fields) and automatically converting it to the appropriate API format. Instructor works with OpenAI, Anthropic, Google, and many other providers.

3.1 Basic Instructor Usage

import instructor
from pydantic import BaseModel, Field
from openai import OpenAI

# Patch the OpenAI client with Instructor
client = instructor.from_openai(OpenAI())

# Define your schema as a Pydantic model
class ContactInfo(BaseModel):
    name: str = Field(description="Full name of the person")
    email: str = Field(description="Email address")
    sentiment: str = Field(description="positive, negative, or neutral")
    confidence: float = Field(ge=0.0, le=1.0, description="Confidence score")

# Extract structured data; the response is a Pydantic object, not raw JSON
contact = client.chat.completions.create(
    model="gpt-4o",
    response_model=ContactInfo,
    messages=[
        {"role": "user", "content": "Hi, I'm Sarah Chen (sarah@example.com) and I love your product!"}
    ]
)

print(f"Name: {contact.name}")
print(f"Email: {contact.email}")
print(f"Sentiment: {contact.sentiment}")
print(f"Confidence: {contact.confidence}")
print(f"Type: {type(contact).__name__}")

3.2 Nested Models and Enums

Pydantic models support rich type hierarchies, including nested objects, lists, enums, and optional fields. This allows you to define complex extraction schemas that would be tedious to express as raw JSON Schema.

import instructor
from pydantic import BaseModel, Field
from enum import Enum
from typing import Optional
from openai import OpenAI

class Sentiment(str, Enum):
    POSITIVE = "positive"
    NEGATIVE = "negative"
    NEUTRAL = "neutral"

class Issue(BaseModel):
    category: str = Field(description="Issue category like billing, technical, shipping")
    description: str = Field(description="Brief description of the issue")
    severity: int = Field(ge=1, le=5, description="Severity from 1 (low) to 5 (critical)")

class TicketExtraction(BaseModel):
    customer_name: str
    customer_email: str
    sentiment: Sentiment
    issues: list[Issue] = Field(description="List of issues mentioned")
    requires_escalation: bool = Field(description="Whether this needs manager attention")
    summary: str = Field(max_length=200, description="Brief summary of the ticket")

client = instructor.from_openai(OpenAI())

ticket = client.chat.completions.create(
    model="gpt-4o",
    response_model=TicketExtraction,
    messages=[
        {"role": "user", "content": (
            "From: alex.wong@company.com\n"
            "Subject: Multiple issues with my order\n\n"
            "Hi, I'm Alex Wong. My order #4521 arrived damaged and I was "
            "charged twice on my credit card. The box was completely crushed "
            "and two items were broken. I need this resolved immediately. "
            "This is the third time I've had shipping problems."
        )}
    ]
)

print(f"Customer: {ticket.customer_name} ({ticket.customer_email})")
print(f"Sentiment: {ticket.sentiment.value}")
print(f"Escalation needed: {ticket.requires_escalation}")
print(f"Issues found: {len(ticket.issues)}")
for issue in ticket.issues:
    print(f"  [{issue.severity}/5] {issue.category}: {issue.description}")

4. Function Calling and Tool Use

Function calling (also called "tool use") is a mechanism that lets the model indicate it wants to invoke an external function rather than produce a text response. The model does not actually execute the function; instead, it generates a structured JSON object containing the function name and arguments. Your application code executes the function and sends the result back to the model, which then incorporates it into its response.

4.1 OpenAI Function Calling

In the OpenAI API, you define available tools in the tools parameter, each with a name, description, and a JSON Schema for its parameters. When the model decides a tool is needed, it returns a response with finish_reason="tool_calls" instead of producing text.

from openai import OpenAI
import json

client = OpenAI()

# Define the available tools
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get current weather for a city",
            "parameters": {
                "type": "object",
                "properties": {
                    "city": {"type": "string", "description": "City name"},
                    "units": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "Temperature unit"
                    }
                },
                "required": ["city"]
            }
        }
    }
]

# Simulated weather function
def get_weather(city: str, units: str = "celsius") -> str:
    data = {"Paris": "22C, sunny", "London": "15C, cloudy", "Tokyo": "28C, humid"}
    return data.get(city, f"No data for {city}")

# Step 1: Send user message with tools
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "What's the weather in Paris?"}],
    tools=tools
)

# Step 2: Check if model wants to call a tool
message = response.choices[0].message
if message.tool_calls:
    tool_call = message.tool_calls[0]
    args = json.loads(tool_call.function.arguments)
    print(f"Model wants to call: {tool_call.function.name}({args})")

    # Step 3: Execute the function
    result = get_weather(**args)

    # Step 4: Send result back to model
    final_response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "user", "content": "What's the weather in Paris?"},
            message,  # The assistant's tool_call message
            {"role": "tool", "tool_call_id": tool_call.id, "content": result}
        ],
        tools=tools
    )
    print(f"Final answer: {final_response.choices[0].message.content}")

4.2 Anthropic Tool Use

Anthropic's tool use follows the same conceptual pattern but with different API conventions. Tools are defined with input_schema (instead of parameters), and the response uses content blocks with a tool_use type. The tool result is sent back as a tool_result content block.

import anthropic
import json

client = anthropic.Anthropic()

tools = [
    {
        "name": "get_weather",
        "description": "Get current weather for a city",
        "input_schema": {
            "type": "object",
            "properties": {
                "city": {"type": "string", "description": "City name"},
                "units": {"type": "string", "enum": ["celsius", "fahrenheit"]}
            },
            "required": ["city"]
        }
    }
]

def get_weather(city, units="celsius"):
    data = {"Paris": "22C, sunny", "London": "15C, cloudy"}
    return data.get(city, f"No data for {city}")

# Step 1: Send request with tools
response = client.messages.create(
    model="claude-sonnet-4-20250514",
    max_tokens=300,
    tools=tools,
    messages=[{"role": "user", "content": "What's the weather in Paris?"}]
)

# Step 2: Find the tool_use block
for block in response.content:
    if block.type == "tool_use":
        print(f"Tool call: {block.name}({block.input})")
        result = get_weather(**block.input)

        # Step 3: Send tool result back
        final = client.messages.create(
            model="claude-sonnet-4-20250514",
            max_tokens=300,
            tools=tools,
            messages=[
                {"role": "user", "content": "What's the weather in Paris?"},
                {"role": "assistant", "content": response.content},
                {"role": "user", "content": [
                    {"type": "tool_result", "tool_use_id": block.id, "content": result}
                ]}
            ]
        )
        print(f"Final: {final.content[0].text}")

4.3 Google Gemini Function Calling

Google Gemini uses function_declarations for tool definitions and returns function calls as structured parts in the response. The syntax differs from both OpenAI and Anthropic, but the conceptual loop (define tools, receive a call, execute, return results) is identical.

from google import genai
from google.genai import types

client = genai.Client()

# Define tools using function_declarations
weather_tool = types.Tool(
    function_declarations=[
        types.FunctionDeclaration(
            name="get_weather",
            description="Get current weather for a city",
            parameters=types.Schema(
                type="OBJECT",
                properties={
                    "city": types.Schema(type="STRING", description="City name"),
                },
                required=["city"],
            ),
        )
    ]
)

def get_weather(city: str) -> str:
    data = {"Paris": "22C, sunny", "Tokyo": "28C, humid"}
    return data.get(city, f"No data for {city}")

# Step 1: Send request with tools
response = client.models.generate_content(
    model="gemini-2.5-flash",
    contents="What's the weather in Paris?",
    config=types.GenerateContentConfig(tools=[weather_tool]),
)

# Step 2: Extract the function call from the response
part = response.candidates[0].content.parts[0]
print(f"Function call: {part.function_call.name}({dict(part.function_call.args)})")
result = get_weather(**dict(part.function_call.args))

# Step 3: Send function response back
from google.genai.types import Content, Part

final = client.models.generate_content(
    model="gemini-2.5-flash",
    contents=[
        Content(parts=[Part(text="What's the weather in Paris?")], role="user"),
        response.candidates[0].content,
        Content(parts=[Part(function_response=types.FunctionResponse(
            name="get_weather", response={"result": result}
        ))], role="user"),
    ],
    config=types.GenerateContentConfig(tools=[weather_tool]),
)
print(f"Final: {final.text}")

5. Parallel and Sequential Tool Calls

Modern LLMs can request multiple tool calls in a single response. For instance, if a user asks "What is the weather in Paris and Tokyo?", the model may emit two get_weather calls simultaneously. Your application should detect multiple tool calls and execute them in parallel for better performance.

import asyncio
from openai import OpenAI
import json

client = OpenAI()

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get current weather for a city",
            "parameters": {
                "type": "object",
                "properties": {
                    "city": {"type": "string"}
                },
                "required": ["city"]
            }
        }
    }
]

def get_weather(city):
    data = {"Paris": "22C, sunny", "Tokyo": "28C, humid", "London": "15C, cloudy"}
    return data.get(city, "Unknown")

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Weather in Paris and Tokyo?"}],
    tools=tools
)

message = response.choices[0].message
print(f"Number of tool calls: {len(message.tool_calls)}")

# Execute all tool calls and collect results
tool_messages = []
for tc in message.tool_calls:
    args = json.loads(tc.function.arguments)
    result = get_weather(**args)
    print(f"  {tc.function.name}({args}) = {result}")
    tool_messages.append({
        "role": "tool",
        "tool_call_id": tc.id,
        "content": result
    })

# Send all results back at once
final = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "user", "content": "Weather in Paris and Tokyo?"},
        message,
        *tool_messages
    ],
    tools=tools
)
print(f"\nFinal: {final.choices[0].message.content}")

6. Cross-Provider Tool Use Comparison