ROI Measurement & Value Attribution

1. The LLM ROI Framework

LLM ROI is calculated as the net benefit (value generated minus total cost) divided by total cost, expressed as a percentage. The challenge is that both the numerator and denominator contain components that are difficult to measure precisely. The framework below structures costs and benefits into measurable categories.

from dataclasses import dataclass

@dataclass
class LLMROIModel:
    """Generic ROI model for LLM projects over a given time horizon."""
    name: str
    horizon_months: int

    # Costs (all in USD)
    dev_cost: float            # one-time development
    infra_monthly: float      # monthly infrastructure
    api_monthly: float        # monthly API charges
    maintenance_monthly: float # monthly maintenance

    # Value (all in USD)
    labor_savings_monthly: float
    speed_value_monthly: float
    quality_value_monthly: float
    revenue_impact_monthly: float

    def total_cost(self) -> float:
        recurring = (self.infra_monthly + self.api_monthly
                     + self.maintenance_monthly) * self.horizon_months
        return self.dev_cost + recurring

    def total_value(self) -> float:
        monthly = (self.labor_savings_monthly + self.speed_value_monthly
                   + self.quality_value_monthly + self.revenue_impact_monthly)
        return monthly * self.horizon_months

    def roi_percent(self) -> float:
        cost = self.total_cost()
        return ((self.total_value() - cost) / cost) * 100

    def payback_months(self) -> float:
        monthly_net = (self.labor_savings_monthly + self.speed_value_monthly
                       + self.quality_value_monthly + self.revenue_impact_monthly
                       - self.infra_monthly - self.api_monthly
                       - self.maintenance_monthly)
        if monthly_net <= 0:
            return float("inf")
        return self.dev_cost / monthly_net

    def summary(self) -> str:
        return (f"{self.name} ({self.horizon_months}mo horizon)\n"
                f"  Total Cost:  ${self.total_cost():>12,.0f}\n"
                f"  Total Value: ${self.total_value():>12,.0f}\n"
                f"  ROI:         {self.roi_percent():>11.1f}%\n"
                f"  Payback:     {self.payback_months():>11.1f} months")

2. Coding Assistant ROI

Coding assistants (GitHub Copilot, Cursor, Cody) are among the most widely deployed LLM applications in enterprises. Their ROI is driven primarily by developer productivity gains, measured as reduced time on routine coding tasks, fewer context switches, and faster onboarding.

# ROI model for a coding assistant deployment (100 developers)
coding_assistant = LLMROIModel(
    name="Coding Assistant (100 devs)",
    horizon_months=12,
    dev_cost=15_000,          # setup, SSO integration, policy config
    infra_monthly=0,         # SaaS, no self-hosting
    api_monthly=3_900,       # 100 devs x $39/seat/month
    maintenance_monthly=500, # admin time, policy updates
    labor_savings_monthly=8_333,  # ~10% productivity gain on $1M annual salary
    speed_value_monthly=4_167,   # faster feature delivery (est. 5% revenue impact)
    quality_value_monthly=2_000,  # fewer bugs in production
    revenue_impact_monthly=0,    # indirect, hard to measure
)
print(coding_assistant.summary())

3. Customer Support ROI

Customer support is the second most common enterprise LLM use case. The ROI model for support differs from coding assistants because it involves a mix of full automation (chatbot deflection) and human augmentation (agent copilot). Each channel has different economics.

# ROI model for AI-powered customer support
support_ai = LLMROIModel(
    name="Customer Support AI",
    horizon_months=12,
    dev_cost=150_000,          # RAG pipeline, fine-tuning, integration
    infra_monthly=4_500,       # vector DB, inference GPU, monitoring
    api_monthly=6_000,         # LLM API calls (200K tickets/yr)
    maintenance_monthly=3_000,  # knowledge base updates, model retraining
    labor_savings_monthly=19_250,   # 55% cost reduction on $420K annual support
    speed_value_monthly=3_000,     # faster resolution, fewer escalations
    quality_value_monthly=2_500,   # higher CSAT, fewer repeat contacts
    revenue_impact_monthly=1_500,  # reduced churn from better support
)
print(support_ai.summary())

4. Attribution Challenges

Value attribution is the hardest part of LLM ROI. When a customer support team implements an AI copilot, improves their training program, and hires two senior agents in the same quarter, how much of the improvement should be attributed to the AI system? There are three common attribution approaches, each with tradeoffs.

5. Lab: Building a Conversational AI Agent ROI Model

In this lab, you will build a complete ROI model for a conversational AI agent that handles first-line customer inquiries. The model accounts for variable costs (per-conversation API charges), fixed costs (infrastructure and maintenance), and multiple value streams.

from dataclasses import dataclass
import json

@dataclass
class ConversationalAgentROI:
    """Detailed ROI model for a conversational AI agent.

    Handles per-conversation variable costs and multiple value streams.
    """
    # Volume assumptions
    monthly_conversations: int
    avg_turns_per_conversation: int
    deflection_rate: float          # fraction handled without human

    # Per-conversation costs
    avg_input_tokens: int
    avg_output_tokens: int
    input_price_per_million: float   # USD per 1M tokens
    output_price_per_million: float

    # Fixed monthly costs
    infra_monthly: float
    maintenance_monthly: float

    # One-time costs
    development_cost: float

    # Value parameters
    cost_per_human_conversation: float  # fully loaded agent cost
    csat_revenue_impact_monthly: float  # reduced churn value

    def monthly_api_cost(self) -> float:
        total_turns = self.monthly_conversations * self.avg_turns_per_conversation
        input_cost = (total_turns * self.avg_input_tokens / 1_000_000
                      * self.input_price_per_million)
        output_cost = (total_turns * self.avg_output_tokens / 1_000_000
                       * self.output_price_per_million)
        return input_cost + output_cost

    def monthly_total_cost(self) -> float:
        return self.monthly_api_cost() + self.infra_monthly + self.maintenance_monthly

    def monthly_labor_savings(self) -> float:
        deflected = self.monthly_conversations * self.deflection_rate
        return deflected * self.cost_per_human_conversation

    def annual_roi_report(self) -> dict:
        annual_cost = self.development_cost + self.monthly_total_cost() * 12
        annual_savings = self.monthly_labor_savings() * 12
        annual_revenue = self.csat_revenue_impact_monthly * 12
        annual_value = annual_savings + annual_revenue
        roi = ((annual_value - annual_cost) / annual_cost) * 100

        return {
            "annual_api_cost": round(self.monthly_api_cost() * 12),
            "annual_infra_cost": round(self.infra_monthly * 12),
            "annual_maintenance": round(self.maintenance_monthly * 12),
            "development_cost": round(self.development_cost),
            "total_annual_cost": round(annual_cost),
            "annual_labor_savings": round(annual_savings),
            "annual_revenue_impact": round(annual_revenue),
            "total_annual_value": round(annual_value),
            "roi_percent": round(roi, 1),
            "cost_per_ai_conversation": round(
                self.monthly_total_cost() / self.monthly_conversations, 3
            ),
            "cost_per_human_conversation": self.cost_per_human_conversation,
        }

# Build the model
agent_roi = ConversationalAgentROI(
    monthly_conversations=25_000,
    avg_turns_per_conversation=4,
    deflection_rate=0.45,
    avg_input_tokens=800,
    avg_output_tokens=300,
    input_price_per_million=3.0,    # GPT-4o-mini pricing
    output_price_per_million=12.0,
    infra_monthly=2_500,
    maintenance_monthly=2_000,
    development_cost=120_000,
    cost_per_human_conversation=8.50,
    csat_revenue_impact_monthly=3_000,
)

report = agent_roi.annual_roi_report()
print(json.dumps(report, indent=2))