Chain-of-Thought & Reasoning Techniques

1. Chain-of-Thought Prompting

Chain-of-Thought prompting works by encouraging the model to generate intermediate reasoning steps before producing a final answer. The mechanism is straightforward: each reasoning step conditions the generation of subsequent steps, allowing the model to "carry" information forward through the computation. Without CoT, the model must compress all reasoning into a single forward pass through the network. With CoT, the model effectively uses its own generated text as a scratchpad, offloading intermediate computation into the token sequence.

1.1 Zero-Shot CoT

The simplest form of CoT requires no examples at all. Kojima et al. (2022) discovered that appending the phrase "Let's think step by step" to a prompt is sufficient to trigger reasoning behavior in large models. This zero-shot CoT approach is remarkably effective, improving accuracy on GSM8K (grade school math) from 17.7% to 78.7% for PaLM 540B.

import openai

client = openai.OpenAI()

def solve_with_cot(problem: str) -> str:
    """Solve a problem using zero-shot Chain-of-Thought."""
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system",
             "content": """Solve the problem step by step.
Show your reasoning clearly, then provide the final answer
on a separate line starting with "ANSWER: "."""},
            {"role": "user",
             "content": problem}
        ],
        temperature=0.0
    )
    return response.choices[0].message.content

problem = """A store sells notebooks for $4 each. If you buy 5 or more,
you get a 20% discount on the entire purchase. Sarah buys 7 notebooks
and pays with a $50 bill. How much change does she receive?"""

print(solve_with_cot(problem))

1.2 Few-Shot CoT

For tasks where zero-shot CoT is insufficient, providing examples of step-by-step reasoning significantly improves performance. The key insight is that the examples teach the model not just the answer format, but the style of reasoning to apply. Different reasoning styles suit different tasks: arithmetic problems benefit from sequential calculation steps, logic problems benefit from listing premises and drawing inferences, and coding problems benefit from planning before implementing.

import openai

client = openai.OpenAI()

# Few-shot CoT: provide exemplar reasoning chains
FEW_SHOT_EXAMPLES = """
Example 1:
Q: A train travels 120 miles in 2 hours. It then travels 90 miles in 1.5 hours. What is the average speed for the entire trip?
A: Let me work through this step by step.
Total distance = 120 + 90 = 210 miles
Total time = 2 + 1.5 = 3.5 hours
Average speed = total distance / total time = 210 / 3.5 = 60 mph
ANSWER: 60 mph

Example 2:
Q: A store has a "buy 2, get 1 free" deal on $6 shirts. How much do 7 shirts cost?
A: Let me work through this step by step.
For every 3 shirts, you pay for 2: that is 2 x $6 = $12 per group of 3.
7 shirts = 2 complete groups (6 shirts) + 1 remaining shirt.
Cost for 6 shirts = 2 x $12 = $24
Cost for 1 extra shirt = $6
Total = $24 + $6 = $30
ANSWER: $30
"""

def solve_with_few_shot_cot(problem: str) -> str:
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system",
             "content": f"""Solve problems step by step, following the style shown in these examples:
{FEW_SHOT_EXAMPLES}
Show your reasoning clearly, then provide the final answer on a line starting with "ANSWER: "."""},
            {"role": "user", "content": problem}
        ],
        temperature=0.0
    )
    return response.choices[0].message.content

problem = """A bakery sells cupcakes for $3 each. If you buy a dozen (12),
you get a 25% discount. Maria buys 15 cupcakes. How much does she pay?"""

print(solve_with_few_shot_cot(problem))

2. Self-Consistency

Self-consistency, introduced by Wang et al. (2022), builds on CoT by sampling multiple reasoning paths and selecting the answer that appears most frequently. The intuition is that while any single reasoning chain might contain errors, different chains are likely to make different errors. When multiple independent chains converge on the same answer, confidence in that answer increases.

2.1 Implementation

import openai
from collections import Counter

client = openai.OpenAI()

def solve_with_self_consistency(problem: str, n_samples: int = 5) -> dict:
    """Sample multiple CoT paths and take majority vote."""
    answers = []

    for _ in range(n_samples):
        response = client.chat.completions.create(
            model="gpt-4o",
            messages=[
                {"role": "system",
                 "content": """Solve step by step. End with ANSWER: <number>"""},
                {"role": "user", "content": problem}
            ],
            temperature=0.7  # Higher temperature for diverse paths
        )
        text = response.choices[0].message.content
        # Extract the final answer
        if "ANSWER:" in text:
            answer = text.split("ANSWER:")[-1].strip()
            answers.append(answer)

    # Majority vote
    vote_counts = Counter(answers)
    best_answer, count = vote_counts.most_common(1)[0]

    return {
        "answer": best_answer,
        "confidence": count / len(answers),
        "all_answers": dict(vote_counts)
    }

result = solve_with_self_consistency(
    "If a train travels at 60 mph for 2.5 hours, then at 80 mph for 1.5 hours, "
    "what is the total distance traveled?"
)
print(f"Answer: {result['answer']}")
print(f"Confidence: {result['confidence']:.0%}")
print(f"All votes: {result['all_answers']}")

3. Tree-of-Thought (ToT)

Tree-of-Thought (Yao et al., 2023) extends CoT by exploring multiple reasoning paths in a structured tree, with the ability to evaluate and backtrack. While CoT follows a single linear chain and self-consistency samples independent chains, ToT builds a branching exploration tree where the model can evaluate partial solutions, prune unpromising branches, and explore alternatives.

The ToT framework has three core components:

1. Thought generation: At each step, the model proposes multiple possible next steps (branches).
2. Thought evaluation: The model (or a separate evaluator) scores each branch on how promising it looks.
3. Search strategy: A search algorithm (typically breadth-first or depth-first) navigates the tree, expanding the most promising nodes and pruning dead ends.

3.1 Simplified ToT Implementation

import openai

client = openai.OpenAI()

def generate_thoughts(problem: str, context: str, n: int = 3) -> list[str]:
    """Generate n candidate next steps."""
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user",
            "content": f"""Problem: {problem}
Progress so far: {context}
Generate {n} different possible next steps. Number them 1, 2, 3.
Each should be a single reasoning step."""}],
        temperature=0.8
    )
    text = response.choices[0].message.content
    return [line.strip() for line in text.split("\n") if line.strip()]

def evaluate_thought(problem: str, thought: str) -> float:
    """Score a thought from 0 to 1."""
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user",
            "content": f"""Problem: {problem}
Proposed reasoning step: {thought}
Rate this step from 0.0 to 1.0 based on correctness and
usefulness. Respond with only a number."""}],
        temperature=0.0
    )
    try:
        return float(response.choices[0].message.content.strip())
    except ValueError:
        return 0.5

# Example: solve with tree exploration
problem = "Find two numbers that multiply to 36 and add to 13."
thoughts = generate_thoughts(problem, "(starting)")
for t in thoughts[:3]:
    score = evaluate_thought(problem, t)
    print(f"  [{score:.2f}] {t}")

4. Step-Back Prompting

Step-back prompting (Zheng et al., 2023) takes the opposite approach from diving into details. Instead of reasoning through the specific problem, it first asks the model to identify the abstract principle or high-level concept, then applies that principle to solve the specific problem. This is particularly effective for science, math, and policy questions where the correct reasoning requires recalling a general rule before applying it.

The two-phase approach works as follows. First, generate a "step-back" question: "What general principle or concept is relevant to this problem?" Then, use the model's answer to that abstract question as context for solving the original specific problem. This prevents the model from getting lost in surface-level details and grounds its reasoning in correct foundational knowledge.

import openai

client = openai.OpenAI()

def step_back_solve(question: str) -> str:
    """Two-phase step-back prompting: abstract first, then solve."""

    # Phase 1: Generate the step-back (abstract) question
    abstraction = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system",
             "content": "Given a specific question, identify the general principle or foundational concept needed to answer it. Respond with ONLY the principle, not the answer to the original question."},
            {"role": "user", "content": question}
        ],
        temperature=0.0
    )
    principle = abstraction.choices[0].message.content
    print(f"Step-back principle: {principle[:120]}...")

    # Phase 2: Solve using the principle as context
    solution = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system",
             "content": f"Use this foundational principle to answer the question:\n\n{principle}"},
            {"role": "user", "content": question}
        ],
        temperature=0.0
    )
    return solution.choices[0].message.content

answer = step_back_solve(
    "If the temperature of an ideal gas is doubled while keeping "
    "the volume constant, what happens to its pressure?"
)
print(f"\nAnswer: {answer}")

5. The ReAct Framework

ReAct (Yao et al., 2022) combines Reasoning and Acting in an interleaved loop. The model alternates between generating reasoning traces (thinking about what to do) and taking actions (calling tools, searching databases, executing code). This pattern is fundamental to modern LLM agents and represents the bridge between prompt engineering and agentic AI.

5.1 The ReAct Loop

import openai, json

client = openai.OpenAI()

TOOLS = [
    {"type": "function",
     "function": {
         "name": "search",
         "description": "Search for information on a topic",
         "parameters": {
             "type": "object",
             "properties": {
                 "query": {"type": "string", "description": "Search query"}
             },
             "required": ["query"]
         }
     }}
]

REACT_SYSTEM = """You are a research assistant. For each question:
1. THINK: Reason about what information you need
2. ACT: Use the search tool to find information
3. OBSERVE: Analyze the search results
4. Repeat THINK/ACT/OBSERVE until you have enough information
5. Provide a final, well-sourced answer"""

def react_agent(question: str) -> str:
    messages = [
        {"role": "system", "content": REACT_SYSTEM},
        {"role": "user", "content": question}
    ]

    for step in range(5):  # Max 5 iterations
        response = client.chat.completions.create(
            model="gpt-4o",
            messages=messages,
            tools=TOOLS,
            tool_choice="auto"
        )
        msg = response.choices[0].message

        if msg.tool_calls:
            # Model wants to use a tool (ACT)
            messages.append(msg)
            for call in msg.tool_calls:
                result = execute_search(
                    json.loads(call.function.arguments)["query"]
                )
                messages.append({
                    "role": "tool",
                    "tool_call_id": call.id,
                    "content": result
                })
        else:
            # Model has enough info, return final answer
            return msg.content

    return "Max iterations reached without final answer."

6. Comparison of Reasoning Techniques

7. Choosing a Reasoning Technique: Decision Flowchart

With multiple reasoning techniques available, selecting the right one for a given task is itself a decision problem. The following flowchart provides a practical starting point. Start at the top and follow the questions to arrive at a recommended technique.