LLM Product Management

1. Translating Business Problems to LLM Requirements

The first step in LLM product management is converting a vague business request ("we want an AI chatbot") into a precise product requirements document that engineering can build against. This translation requires understanding both the business context and the capabilities and limitations of current models.

The Requirements Translation Framework

from dataclasses import dataclass, field
from typing import List, Optional
from enum import Enum

class RiskLevel(Enum):
    LOW = "low"        # Wrong answer is inconvenient
    MEDIUM = "medium"  # Wrong answer causes rework
    HIGH = "high"      # Wrong answer causes financial or legal harm
    CRITICAL = "critical"  # Wrong answer endangers safety

@dataclass
class LLMProductSpec:
    """Structured LLM product specification."""
    name: str
    user_persona: str
    job_to_be_done: str

    # Functional requirements
    input_types: List[str]          # text, image, document, audio
    output_format: str              # free text, structured JSON, classification
    max_latency_seconds: float
    context_window_needs: str       # "short" (<4K), "medium" (4-32K), "long" (>32K)

    # Risk and quality
    hallucination_risk: RiskLevel
    requires_citations: bool
    human_review_required: bool
    accuracy_target: float          # 0.0 to 1.0

    # Scale
    daily_requests_estimate: int
    concurrent_users_peak: int

    # Constraints
    data_residency: Optional[str] = None  # "US", "EU", etc.
    pii_handling: str = "none"             # "none", "redact", "allowed"

    def model_tier_recommendation(self) -> str:
        if self.hallucination_risk in (RiskLevel.HIGH, RiskLevel.CRITICAL):
            return "Frontier model (GPT-4o, Claude 3.5 Sonnet) with guardrails"
        elif self.context_window_needs == "long":
            return "Long-context model (Gemini 1.5 Pro, Claude 3.5)"
        elif self.daily_requests_estimate > 10000:
            return "Fine-tuned small model (Llama 3.1 8B) for cost efficiency"
        else:
            return "Mid-tier model (GPT-4o-mini, Claude 3.5 Haiku)"

# Example: Customer support copilot
spec = LLMProductSpec(
    name="Support Copilot",
    user_persona="Tier-1 support agent handling billing and account inquiries",
    job_to_be_done="Draft accurate responses to customer tickets using knowledge base",
    input_types=["text", "document"],
    output_format="free text with inline citations",
    max_latency_seconds=3.0,
    context_window_needs="medium",
    hallucination_risk=RiskLevel.HIGH,
    requires_citations=True,
    human_review_required=True,
    accuracy_target=0.92,
    daily_requests_estimate=5000,
    concurrent_users_peak=200,
    data_residency="US",
    pii_handling="redact",
)

print(f"Product: {spec.name}")
print(f"Model recommendation: {spec.model_tier_recommendation()}")
print(f"Accuracy target: {spec.accuracy_target:.0%}")

2. Success Metrics for LLM Products

LLM products require a layered metrics framework that captures quality at the model level, product level, and business level. Focusing on only one layer leads to blind spots: a model with 95% accuracy is useless if users do not trust it, and high user satisfaction means nothing if the product does not reduce costs.

from dataclasses import dataclass

@dataclass
class LLMProductMetrics:
    """Weekly metrics dashboard for an LLM product."""
    # Model quality
    accuracy: float
    hallucination_rate: float
    latency_p95: float

    # Product usage
    csat: float
    adoption_rate: float
    edit_rate: float

    # Business impact
    resolution_rate: float
    deflection_rate: float
    cost_per_resolution: float

    def health_check(self) -> dict:
        return {
            "accuracy": "PASS" if self.accuracy >= 0.85 else "FAIL",
            "hallucination": "PASS" if self.hallucination_rate < 0.05 else "FAIL",
            "csat": "PASS" if self.csat >= 4.0 else "WARN",
            "adoption": "PASS" if self.adoption_rate >= 0.60 else "WARN",
            "deflection": "PASS" if self.deflection_rate >= 0.40 else "WARN",
        }

# Week 8 metrics for Support Copilot
week8 = LLMProductMetrics(
    accuracy=0.91, hallucination_rate=0.03, latency_p95=2.8,
    csat=4.2, adoption_rate=0.72, edit_rate=0.24,
    resolution_rate=0.76, deflection_rate=0.38, cost_per_resolution=4.20
)

for metric, status in week8.health_check().items():
    print(f"  {metric:15s} {status}")

3. Hallucination Risk Management

Hallucination is the defining risk of LLM products. Unlike bugs in traditional software, hallucinations are not deterministic: the same input can produce correct output 99 times and a confidently stated falsehood on the 100th. Product managers must design systems that minimize hallucination occurrence and mitigate its impact when it does occur.

4. UX Design for LLM Products

LLM products need UX patterns that manage uncertainty, set appropriate expectations, and give users control over AI-generated content. The following patterns have emerged as best practices across successful AI products.

Core UX Patterns

from dataclasses import dataclass
from typing import List, Optional

@dataclass
class AIResponse:
    """Structured AI response with UX metadata."""
    content: str
    confidence: float                    # 0.0 to 1.0
    citations: List[dict]              # [{source, page, text}]
    suggested_actions: List[str]
    requires_human_review: bool

    def confidence_label(self) -> str:
        if self.confidence >= 0.85:
            return "high_confidence"
        elif self.confidence >= 0.60:
            return "medium_confidence"
        else:
            return "low_confidence"

    def ux_treatment(self) -> dict:
        label = self.confidence_label()
        return {
            "high_confidence": {
                "border_color": "#27ae60",
                "icon": "check_circle",
                "disclaimer": None,
            },
            "medium_confidence": {
                "border_color": "#f39c12",
                "icon": "info",
                "disclaimer": "This response may need verification.",
            },
            "low_confidence": {
                "border_color": "#e94560",
                "icon": "warning",
                "disclaimer": "Low confidence. Please verify before using.",
            },
        }[label]

response = AIResponse(
    content="Based on your policy, the refund window is 30 days from purchase.",
    confidence=0.72,
    citations=[{"source": "refund_policy_v3.pdf", "page": 2}],
    suggested_actions=["Send to customer", "Edit draft", "Escalate"],
    requires_human_review=True,
)
print(f"Confidence: {response.confidence_label()}")
print(f"UX treatment: {response.ux_treatment()}")

5. Iterative Delivery for LLM Products

LLM products benefit from a delivery cadence that is faster and more experimental than traditional software. Because model behavior can change with prompt modifications (no code deploy required), product teams can iterate on quality much faster than conventional feature development allows.

6. Stakeholder Communication

Communicating LLM product progress to non-technical stakeholders requires translating model metrics into business language. The following template provides a structure for weekly stakeholder updates that avoids jargon while maintaining technical accuracy.

def generate_stakeholder_update(metrics: LLMProductMetrics, week: int) -> str:
    """Generate a non-technical stakeholder update from product metrics."""
    health = metrics.health_check()
    passing = sum(1 for v in health.values() if v == "PASS")
    total = len(health)

    update = f"""
WEEKLY UPDATE: Support Copilot (Week {week})

STATUS: {passing}/{total} metrics on target

HIGHLIGHTS:
- Customer satisfaction: {metrics.csat}/5.0 (target: 4.0)
- Tickets resolved without escalation: {metrics.resolution_rate:.0%}
- AI-assisted deflection: {metrics.deflection_rate:.0%} (target: 40%)
- Cost per resolution: ${metrics.cost_per_resolution:.2f}

QUALITY:
- Response accuracy: {metrics.accuracy:.0%}
- Factual errors (hallucinations): {metrics.hallucination_rate:.1%}
- Agent edit rate: {metrics.edit_rate:.0%} of AI drafts modified

ACTIONS:
"""
    if metrics.deflection_rate < 0.40:
        update += "- Deflection below target; expanding knowledge base coverage\n"
    if metrics.edit_rate > 0.30:
        update += "- High edit rate; investigating prompt quality for top categories\n"
    if metrics.hallucination_rate >= 0.05:
        update += "- Hallucination rate elevated; adding source verification step\n"

    return update.strip()

print(generate_stakeholder_update(week8, 8))