227 Reflection and Self-Improvement in LLM Agents

Human writers revise. Programmers debug. Scientists replicate experiments and update hypotheses. The first attempt at any complex cognitive task is rarely the best one; iterative refinement against feedback is how high-quality work emerges. Large language models, however, are traditionally used in a single forward pass: one prompt, one response, done. This chapter examines a cluster of techniques that break that assumption by building feedback loops directly into the inference process, allowing a model to critique, reflect on, and refine its own outputs without any gradient update.

227.2 2. Self-Refine: Iterative Refinement with Self-Feedback

Madaan et al. (2023) introduced Self-Refine as the clearest instantiation of the generate-critique-refine loop. The same pretrained model plays all three roles: generator, critic, and refiner, with no additional training, no external environment, and no separate model.

227.2.1 2.1 The Loop

The procedure is:

Generate $y^{(0)} = \text{LLM}(p_\text{gen}, x)$ where $p_\text{gen}$ is a task-specific generation prompt.
Feedback $c^{(t)} = \text{LLM}(p_\text{fb}, x, y^{(t)})$ where $p_\text{fb}$ asks the model to critique $y^{(t)}$ with respect to specific quality dimensions (correctness, clarity, style, constraint satisfaction).
Refine $y^{(t+1)} = \text{LLM}(p_\text{ref}, x, y^{(t)}, c^{(t)})$.
Repeat until a stopping criterion is met.

Stopping can be a fixed iteration count or a self-confidence signal: the feedback step includes a question asking whether the output is satisfactory, and the loop terminates when the model answers affirmatively.

flowchart LR
    X["Input x"] --> G["Generate\ny(0)"]
    G --> F["Feedback\nc(t)"]
    F --> R["Refine\ny(t+1)"]
    R --> S{"Stop?"}
    S -- "No" --> F
    S -- "Yes" --> O["Output y(T)"]

227.2.2 2.2 Empirical Results

Madaan et al. evaluated Self-Refine on seven tasks including math reasoning, code optimization, constrained generation, and dialogue response rewriting. Using GPT-3.5 and GPT-4 as the base model, they observed average absolute improvements of approximately 20 percentage points across tasks, with the largest gains on tasks where the quality criterion can be expressed precisely in natural language. Code optimization (measured by execution speedup) improved by 13% for GPT-3.5 and 17% for GPT-4. Sentiment reversal and acronym generation, where success criteria are crisp, saw larger gains still.

The critical design insight is that the feedback prompt must be specific. A generic “is this good?” elicits vague responses. A prompt that asks the model to evaluate correctness of each mathematical step, flag ambiguous variable names, or identify constraint violations produces actionable critiques that the refinement step can act on.

227.2.3 2.3 A Runnable Self-Refine Skeleton

The following implementation demonstrates the loop structure using a stub LLM client. Replacing the call_llm function with a real API call (Anthropic, OpenAI, or a local model) makes it fully operational.

import re

def call_llm(prompt: str) -> str:
    """Stub: replace with real LLM API call."""
    return f"[LLM response to: {prompt[:60]}...]"

GENERATE_PROMPT = """\
Solve the following math word problem step by step.
Problem: {problem}
Solution:"""

FEEDBACK_PROMPT = """\
Review the following solution to a math word problem.
Problem: {problem}
Solution: {solution}

Identify any errors in reasoning, arithmetic, or problem setup.
Then state whether the solution is SATISFACTORY or needs REVISION.
Feedback:"""

REFINE_PROMPT = """\
Revise the solution based on the feedback below.
Problem: {problem}
Previous solution: {solution}
Feedback: {feedback}
Improved solution:"""

def is_satisfactory(feedback: str) -> bool:
    return "SATISFACTORY" in feedback.upper() and "REVISION" not in feedback.upper()

def self_refine(problem: str, max_iterations: int = 3) -> dict:
    history = []
    solution = call_llm(GENERATE_PROMPT.format(problem=problem))
    history.append({"iteration": 0, "solution": solution, "feedback": None})

    for t in range(max_iterations):
        feedback = call_llm(
            FEEDBACK_PROMPT.format(problem=problem, solution=solution)
        )
        history[-1]["feedback"] = feedback

        if is_satisfactory(feedback):
            break

        solution = call_llm(
            REFINE_PROMPT.format(
                problem=problem, solution=solution, feedback=feedback
            )
        )
        history.append({"iteration": t + 1, "solution": solution, "feedback": None})

    return {"final_solution": solution, "iterations": len(history), "history": history}

result = self_refine("A train travels 120 km in 1.5 hours. What is its average speed?")
print(f"Iterations: {result['iterations']}")
print(f"Final solution excerpt: {result['final_solution'][:120]}")

227.3 3. Reflexion: Verbal Reinforcement Learning

Self-Refine operates in a single session with no memory beyond the current context window. Shinn et al. (2023) introduced Reflexion to handle sequential decision-making tasks where an agent interacts with an environment over multiple episodes, accumulating experience that should inform future attempts.

227.3.1 3.1 Architecture and Formalism

Reflexion extends a standard actor-environment loop with two additional components: an Evaluator that scores outcomes and a Reflector that converts scores and trajectories into natural language summaries stored in episodic memory.

Let episode $k$ consist of a sequence of state-action pairs $(s_0^k, a_0^k, s_1^k, a_1^k, \ldots, s_T^k)$. Define:

\[r_k = \text{Evaluator}(s_T^k) \in \{0, 1\}\]

for binary task success, or a scalar reward signal for graded tasks. The Reflector generates a verbal reflection:

\[\rho_k = \text{Reflect}(s_0^k, a_0^k, \ldots, s_T^k, r_k)\]

This reflection is appended to episodic memory $\mathcal{M}$:

\[\mathcal{M}_{k+1} = \mathcal{M}_k \cup \{\rho_k\}\]

On episode $k+1$, the Actor conditions on the full memory:

\[a_t^{k+1} \sim \pi_\theta(\cdot \mid s_t^{k+1}, \mathcal{M}_{k+1})\]

This is “verbal reinforcement learning” in the sense that reward signals are translated into natural language policy improvements rather than parameter gradients. The reflection $\rho_k$ plays the role that a policy gradient update would play in standard RL, but operates entirely through in-context learning.

227.3.2 3.2 Verbal Memory as Policy Improvement

The key insight is that LLMs are few-shot learners: conditioning on past failures described in natural language steers the model away from those failure modes. A reflection like “I tried searching for the answer in Wikipedia directly but the page did not exist; next time I should first check whether the entity name is spelled correctly using a broader search” directly encodes a strategy update that the model can apply.

This bypasses several challenges of standard RL for language agents: sparse rewards are acceptable since a single binary success/failure signal is enough to generate a useful reflection; credit assignment over long action sequences is handled implicitly by the LLM’s ability to reason about which actions likely caused the failure; and catastrophic forgetting is avoided because memory persists across episodes.

227.3.3 3.3 Empirical Results

Shinn et al. (2023) evaluated Reflexion on three benchmarks. On AlfWorld, a text-based household task environment, Reflexion with GPT-4 reached 97% success rate versus 85% for the ReAct baseline and 74% for the plain GPT-4 agent. On HotpotQA multi-hop question answering, Reflexion improved exact-match accuracy from 30% (chain-of-thought) to 51%. On HumanEval coding benchmarks, Reflexion combined with code execution feedback reached 91% pass@1, outperforming GPT-4’s baseline of 67%.

The gains are largest on tasks where failures are recoverable, success signals are unambiguous, and the root cause of failure can be identified from the trajectory. Tasks requiring precise numerical computation or memorized facts benefit less, since the reflection cannot inject information the model does not already have.

from dataclasses import dataclass, field
from typing import Optional

@dataclass
class EpisodicMemory:
    reflections: list[str] = field(default_factory=list)
    max_entries: int = 5

    def add(self, reflection: str) -> None:
        self.reflections.append(reflection)
        if len(self.reflections) > self.max_entries:
            self.reflections.pop(0)

    def format_for_context(self) -> str:
        if not self.reflections:
            return ""
        entries = "\n".join(f"- {r}" for r in self.reflections)
        return f"Past reflections:\n{entries}\n"

def simulate_environment(action: str, target: str) -> tuple[bool, str]:
    """Stub: replace with real task environment."""
    success = target.lower() in action.lower()
    obs = "Task completed." if success else f"Failed. Expected to address: {target}"
    return success, obs

def reflexion_episode(
    task: str,
    target: str,
    memory: EpisodicMemory,
    max_steps: int = 5,
) -> tuple[bool, EpisodicMemory]:
    context = memory.format_for_context()
    action = call_llm(
        f"{context}Task: {task}\nYour action:"
    )
    success, observation = simulate_environment(action, target)

    if not success:
        reflection = call_llm(
            f"Task: {task}\nAction taken: {action}\nResult: {observation}\n"
            "What went wrong and how would you approach this differently next time? "
            "Be specific and concise."
        )
        memory.add(reflection)

    return success, memory

memory = EpisodicMemory()
for episode in range(3):
    success, memory = reflexion_episode(
        task="Find the capital of the country where the Eiffel Tower is located.",
        target="Paris",
        memory=memory,
    )
    print(f"Episode {episode + 1}: {'Success' if success else 'Failure'}, "
          f"memory entries={len(memory.reflections)}")

227.4 4. CRITIC: Tool-Interactive Critiquing

Both Self-Refine and Reflexion rely on the model’s own judgment to evaluate outputs. Gou et al. (2023) identified this as the fundamental limitation: a model that produces an incorrect mathematical derivation is unlikely to reliably identify that derivation as incorrect, because the same reasoning process that generated the error will evaluate it.

CRITIC (Critiquing with Tools for Iterative Correction) breaks this circular dependency by grounding critiques in external tools that can verify claims objectively.

227.4.1 4.1 Tool-Grounded Verification

The CRITIC pipeline extends the generate-critique-refine loop by inserting a tool-use step into the critique phase:

\[c^{(t)} = \text{Critique}\!\left(y^{(t)},\; \text{Tool}\!\left(q^{(t)}, y^{(t)}\right)\right)\]

where $q^{(t)}$ is a verification query generated by the model from $y^{(t)}$ and $\text{Tool}(\cdot)$ is an external verifier. The choice of tool depends on the claim type:

Mathematical claims: a symbolic math engine (WolframAlpha, SymPy) evaluates expressions and checks derivations.
Code: a Python interpreter executes the code and returns stdout, stderr, and exit codes.
Factual claims: a search engine retrieves current evidence against which the claim is checked.

The model’s critique is then conditioned on the concrete tool output rather than the model’s unaided assessment. If WolframAlpha returns that $\int_0^1 x^2\,dx = 1/3$ but the model’s solution states $1/2$, the critique can note the specific discrepancy with a concrete numerical ground truth.

227.4.2 4.2 Formal Grounding Advantage

Let $\epsilon_\text{gen}$ be the model’s error rate on a task and $\epsilon_\text{eval}$ be its error rate at evaluating its own outputs on the same task. Self-Refine improves over a single pass only when $\epsilon_\text{eval} < \epsilon_\text{gen}$: the model must be better at spotting errors than at avoiding them. For factual and mathematical tasks, empirical evidence suggests this condition often fails; the model’s self-evaluation error rate approaches its generation error rate.

External tools invert this relationship. A Python interpreter has $\epsilon_\text{eval} \approx 0$ for functional correctness: if the code runs and produces the right output, it is correct. Symbolic computation engines have near-zero error on arithmetic and algebraic simplification. Web search introduces latency and relevance noise but still substantially reduces $\epsilon_\text{eval}$ for factual verification. CRITIC therefore reliably improves over Self-Refine precisely where self-correction tends to fail.

227.4.3 4.3 Results

Gou et al. (2023) evaluated CRITIC on free-form question answering (TriviaQA, AmbigNL), mathematical reasoning (GSM8K), and program synthesis (MBPP, HumanEval). On GSM8K with GPT-4, CRITIC reached 95.0% accuracy versus 91.6% for chain-of-thought alone. On HumanEval, it reached 85.1% pass@1 versus 67.0% for unassisted GPT-4. On TriviaQA, accuracy improved from 75.3% to 82.4%, with the search tool providing concrete evidence the model could not hallucinate past.

227.5 5. When Self-Correction Helps Versus Hurts

Sun et al. (2023) posed the most important cautionary question in this space: is self-correction actually reliable, or does it degrade outputs when the model is miscalibrated?

Their controlled study asked GPT-4 to solve reasoning problems, then instructed it to reconsider and potentially revise its answer. Without any external signal, self-correction made outputs worse on reasoning tasks. The model would revise a correct answer to an incorrect one because the instruction to reconsider suggested that revision was expected. This is not a failure of intelligence but of calibration: the model cannot distinguish between “I made an error and should revise” and “the prompt is asking me to revise regardless.”

The findings crystallize into a principle: self-correction without external feedback is unreliable; self-correction with external feedback is consistently beneficial.

External feedback can take many forms:

Unit test results (code generation)
Symbolic computation output (mathematics)
Search engine results (factual claims)
Human evaluation scores (open-ended quality)
Environmental reward signals (sequential decision-making)
Constrained optimization feasibility checks (structured generation)

The signal need not be rich. Even binary pass/fail from a test suite is enough to direct improvement. What matters is that the feedback is generated by a process that is independent of the model’s reasoning and therefore not subject to the same systematic biases.

227.6 6. The Reflection Design Pattern in Practice

The three frameworks discussed above map onto a decision tree based on what feedback signals are available in a given deployment context.

flowchart TD
    A["New agent task"] --> B{"External tool\ncan verify?"}
    B -- "Yes" --> C["CRITIC\nTool-grounded critique"]
    B -- "No" --> D{"Environment gives\nsuccess signal?"}
    D -- "Yes" --> E["Reflexion\nVerbal RL with memory"]
    D -- "No" --> F["Self-Refine\nPure self-feedback"]
    C --> G["Refined output"]
    E --> G
    F --> G

Several practical considerations govern implementation:

Prompt engineering for critique specificity. Vague feedback (“this could be better”) produces vague refinements. Effective critique prompts enumerate specific dimensions: for code, check correctness, efficiency, edge cases, and readability separately. For mathematical proofs, check each inference step. Structured feedback with labeled sections transfers more reliably to the refinement step.

Stopping criteria. Fixed iteration counts are simple but wasteful when the output converges early. Self-reported confidence is cheap but unreliable for the same reasons self-correction is unreliable without external signals. The most robust stopping criterion for tool-verified tasks is the external tool’s pass/fail signal. For Self-Refine, a practical heuristic is to stop when the feedback step produces a critique shorter than a threshold length, indicating diminishing marginal criticism.

Memory management in Reflexion. Episodic memory grows with episodes and can exhaust the context window. Sliding window truncation (keeping only the $k$ most recent reflections) is the simplest approach. More sophisticated systems compress older reflections or index them for retrieval, treating the memory as a retrieval-augmented knowledge base rather than a flat list.

Computational cost. Each refinement iteration multiplies inference cost. For a $T$-iteration loop, a task that normally costs $C$ tokens now costs approximately $(1 + 2T) \cdot C$ tokens: one generation, $T$ feedbacks, and $T$ refinements. The cost-quality trade-off must be evaluated per application; for tasks where correctness is critical and inference is cheap relative to human review, even five iterations may be cost-effective.

from typing import Callable

def run_critic_loop(
    problem: str,
    generate_fn: Callable[[str], str],
    verify_fn: Callable[[str, str], tuple[bool, str]],
    refine_fn: Callable[[str, str, str], str],
    max_iterations: int = 4,
) -> dict:
    """
    Generic CRITIC loop: generate, verify with external tool, critique, refine.
    generate_fn(problem) -> initial solution
    verify_fn(problem, solution) -> (passed: bool, tool_output: str)
    refine_fn(problem, solution, tool_output) -> refined solution
    """
    solution = generate_fn(problem)
    trace = [{"step": 0, "solution": solution}]

    for i in range(1, max_iterations + 1):
        passed, tool_output = verify_fn(problem, solution)
        trace[-1]["tool_output"] = tool_output
        trace[-1]["passed"] = passed

        if passed:
            break

        solution = refine_fn(problem, solution, tool_output)
        trace.append({"step": i, "solution": solution})

    return {"solution": solution, "iterations": len(trace), "trace": trace}

def stub_generate(problem: str) -> str:
    return f"Initial attempt at: {problem}"

def stub_verify(problem: str, solution: str) -> tuple[bool, str]:
    passed = "correct" in solution.lower()
    return passed, ("OK" if passed else "Solution does not contain verification keyword.")

def stub_refine(problem: str, solution: str, tool_output: str) -> str:
    return f"Revised (correct) solution for: {problem}. Tool said: {tool_output}"

result = run_critic_loop(
    problem="Compute the derivative of x^3 at x=2.",
    generate_fn=stub_generate,
    verify_fn=stub_verify,
    refine_fn=stub_refine,
)
print(f"Converged in {result['iterations']} iteration(s).")
print(f"Final: {result['solution'][:100]}")

227.7 7. Connections to Broader Agent Architectures

Reflection mechanisms are modular and compose naturally with other agent components. A ReAct agent (Yao et al., 2022) that interleaves reasoning and tool use can be augmented with Reflexion-style episodic memory across trials. A chain-of-thought reasoner can wrap its inner reasoning loop with Self-Refine. A code generation pipeline can insert CRITIC’s execution-based verification before committing generated code to a codebase.

The deepest connection is to the idea of test-time compute scaling: given a fixed model, better answers can be obtained by spending more inference-time computation. Reflection is one architecture for this; others include best-of-$n$ sampling, Monte Carlo tree search over reasoning steps, and process reward models. What distinguishes reflection from pure sampling approaches is that it uses the model’s own structured feedback to direct the search, rather than sampling blindly and selecting post hoc. This directed search is more compute-efficient when the model’s critiques are informative, and less efficient when they are not, which returns us to the calibration problem identified by Sun et al.

Understanding when model self-evaluation is trustworthy, and designing hybrid architectures that route to external verification when it is not, is an active research frontier. The frameworks covered in this chapter represent the foundational vocabulary for that ongoing work.

227.8 References

Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., … & Clark, P. (2023). Self-Refine: Iterative Refinement with Self-Feedback. arXiv preprint arXiv:2303.17651. https://arxiv.org/abs/2303.17651
Shinn, N., Cassano, F., Labash, B., Gopinath, A., Narasimhan, K., & Yao, S. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning. arXiv preprint arXiv:2303.11366. https://arxiv.org/abs/2303.11366
Gou, Z., Shao, Z., Gong, Y., Shen, Y., Yang, Y., Duan, N., & Chen, W. (2023). CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing. arXiv preprint arXiv:2305.11738. https://arxiv.org/abs/2305.11738
Sun, Z., Shen, Y., Zhou, Q., Zhang, H., Chen, Z., Cox, D., … & Gan, C. (2023). Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision. arXiv preprint arXiv:2310.01798. https://arxiv.org/abs/2310.01798

# Reflection and Self-Improvement in LLM Agents {#sec-reflection-agents} Human writers revise. Programmers debug. Scientists replicate experiments and update hypotheses. The first attempt at any complex cognitive task is rarely the best one; iterative refinement against feedback is how high-quality work emerges. Large language models, however, are traditionally used in a single forward pass: one prompt, one response, done. This chapter examines a cluster of techniques that break that assumption by building feedback loops directly into the inference process, allowing a model to critique, reflect on, and refine its own outputs without any gradient update. ## 1. The Case for Iterative Refinement A useful framing: LLM outputs are samples from a distribution $p_\theta(y \mid x)$ where $x$ is the prompt and $\theta$ are frozen weights. The first sample is not necessarily the mode of that distribution, nor the highest-quality reachable point. Conditioning on richer context, including a critique of the first sample, shifts the effective sampling distribution toward better regions. More precisely, let $y^{(0)} \sim p_\theta(\cdot \mid x)$ be the initial output and $c^{(0)}$ be a critique generated by querying the same model with $(x, y^{(0)})$. The refined output is then $y^{(1)} \sim p_\theta(\cdot \mid x, y^{(0)}, c^{(0)})$. This extended context acts as a soft form of test-time compute scaling: more tokens are consumed in exchange for higher output quality, with no parameter update required. Why does this work at all? A model that has seen extensive data about what good outputs look like can often identify flaws in a specific output more reliably than it can produce a perfect output from scratch. Critiquing is a different, often easier, cognitive task than generating. A code reviewer spots the off-by-one error that the original author missed. A proof-reader catches the ambiguous pronoun reference the writer glossed over. The same principle applies to language models trained on human-written text that includes revision histories, code reviews, and editorial feedback. The central question is not whether refinement can help in principle, but when it helps in practice, what feedback signal drives it, and how to prevent it from degrading quality when the model's self-evaluation is uncalibrated. ## 2. Self-Refine: Iterative Refinement with Self-Feedback Madaan et al. (2023) introduced Self-Refine as the clearest instantiation of the generate-critique-refine loop. The same pretrained model plays all three roles: generator, critic, and refiner, with no additional training, no external environment, and no separate model. ### 2.1 The Loop The procedure is: 1. **Generate** $y^{(0)} = \text{LLM}(p_\text{gen}, x)$ where $p_\text{gen}$ is a task-specific generation prompt. 2. **Feedback** $c^{(t)} = \text{LLM}(p_\text{fb}, x, y^{(t)})$ where $p_\text{fb}$ asks the model to critique $y^{(t)}$ with respect to specific quality dimensions (correctness, clarity, style, constraint satisfaction). 3. **Refine** $y^{(t+1)} = \text{LLM}(p_\text{ref}, x, y^{(t)}, c^{(t)})$. 4. Repeat until a stopping criterion is met. Stopping can be a fixed iteration count or a self-confidence signal: the feedback step includes a question asking whether the output is satisfactory, and the loop terminates when the model answers affirmatively. ```{mermaid} flowchart LR X["Input x"] --> G["Generate\ny(0)"] G --> F["Feedback\nc(t)"] F --> R["Refine\ny(t+1)"] R --> S{"Stop?"} S -- "No" --> F S -- "Yes" --> O["Output y(T)"] ``` ### 2.2 Empirical Results Madaan et al. evaluated Self-Refine on seven tasks including math reasoning, code optimization, constrained generation, and dialogue response rewriting. Using GPT-3.5 and GPT-4 as the base model, they observed average absolute improvements of approximately 20 percentage points across tasks, with the largest gains on tasks where the quality criterion can be expressed precisely in natural language. Code optimization (measured by execution speedup) improved by 13% for GPT-3.5 and 17% for GPT-4. Sentiment reversal and acronym generation, where success criteria are crisp, saw larger gains still. The critical design insight is that the feedback prompt must be specific. A generic "is this good?" elicits vague responses. A prompt that asks the model to evaluate correctness of each mathematical step, flag ambiguous variable names, or identify constraint violations produces actionable critiques that the refinement step can act on. ### 2.3 A Runnable Self-Refine Skeleton The following implementation demonstrates the loop structure using a stub LLM client. Replacing the `call_llm` function with a real API call (Anthropic, OpenAI, or a local model) makes it fully operational. ```python import re def call_llm(prompt: str) -> str: """Stub: replace with real LLM API call.""" return f"[LLM response to: {prompt[:60]}...]" GENERATE_PROMPT = """\ Solve the following math word problem step by step. Problem: {problem} Solution:""" FEEDBACK_PROMPT = """\ Review the following solution to a math word problem. Problem: {problem} Solution: {solution} Identify any errors in reasoning, arithmetic, or problem setup. Then state whether the solution is SATISFACTORY or needs REVISION. Feedback:""" REFINE_PROMPT = """\ Revise the solution based on the feedback below. Problem: {problem} Previous solution: {solution} Feedback: {feedback} Improved solution:""" def is_satisfactory(feedback: str) -> bool: return "SATISFACTORY" in feedback.upper() and "REVISION" not in feedback.upper() def self_refine(problem: str, max_iterations: int = 3) -> dict: history = [] solution = call_llm(GENERATE_PROMPT.format(problem=problem)) history.append({"iteration": 0, "solution": solution, "feedback": None}) for t in range(max_iterations): feedback = call_llm( FEEDBACK_PROMPT.format(problem=problem, solution=solution) ) history[-1]["feedback"] = feedback if is_satisfactory(feedback): break solution = call_llm( REFINE_PROMPT.format( problem=problem, solution=solution, feedback=feedback ) ) history.append({"iteration": t + 1, "solution": solution, "feedback": None}) return {"final_solution": solution, "iterations": len(history), "history": history} result = self_refine("A train travels 120 km in 1.5 hours. What is its average speed?") print(f"Iterations: {result['iterations']}") print(f"Final solution excerpt: {result['final_solution'][:120]}") ``` ## 3. Reflexion: Verbal Reinforcement Learning Self-Refine operates in a single session with no memory beyond the current context window. Shinn et al. (2023) introduced Reflexion to handle sequential decision-making tasks where an agent interacts with an environment over multiple episodes, accumulating experience that should inform future attempts. ### 3.1 Architecture and Formalism Reflexion extends a standard actor-environment loop with two additional components: an Evaluator that scores outcomes and a Reflector that converts scores and trajectories into natural language summaries stored in episodic memory. Let episode $k$ consist of a sequence of state-action pairs $(s_0^k, a_0^k, s_1^k, a_1^k, \ldots, s_T^k)$. Define: $$r_k = \text{Evaluator}(s_T^k) \in \{0, 1\}$$ for binary task success, or a scalar reward signal for graded tasks. The Reflector generates a verbal reflection: $$\rho_k = \text{Reflect}(s_0^k, a_0^k, \ldots, s_T^k, r_k)$$ This reflection is appended to episodic memory $\mathcal{M}$: $$\mathcal{M}_{k+1} = \mathcal{M}_k \cup \{\rho_k\}$$ On episode $k+1$, the Actor conditions on the full memory: $$a_t^{k+1} \sim \pi_\theta(\cdot \mid s_t^{k+1}, \mathcal{M}_{k+1})$$ This is "verbal reinforcement learning" in the sense that reward signals are translated into natural language policy improvements rather than parameter gradients. The reflection $\rho_k$ plays the role that a policy gradient update would play in standard RL, but operates entirely through in-context learning. ### 3.2 Verbal Memory as Policy Improvement The key insight is that LLMs are few-shot learners: conditioning on past failures described in natural language steers the model away from those failure modes. A reflection like "I tried searching for the answer in Wikipedia directly but the page did not exist; next time I should first check whether the entity name is spelled correctly using a broader search" directly encodes a strategy update that the model can apply. This bypasses several challenges of standard RL for language agents: sparse rewards are acceptable since a single binary success/failure signal is enough to generate a useful reflection; credit assignment over long action sequences is handled implicitly by the LLM's ability to reason about which actions likely caused the failure; and catastrophic forgetting is avoided because memory persists across episodes. ### 3.3 Empirical Results Shinn et al. (2023) evaluated Reflexion on three benchmarks. On AlfWorld, a text-based household task environment, Reflexion with GPT-4 reached 97% success rate versus 85% for the ReAct baseline and 74% for the plain GPT-4 agent. On HotpotQA multi-hop question answering, Reflexion improved exact-match accuracy from 30% (chain-of-thought) to 51%. On HumanEval coding benchmarks, Reflexion combined with code execution feedback reached 91% pass@1, outperforming GPT-4's baseline of 67%. The gains are largest on tasks where failures are recoverable, success signals are unambiguous, and the root cause of failure can be identified from the trajectory. Tasks requiring precise numerical computation or memorized facts benefit less, since the reflection cannot inject information the model does not already have. ```python from dataclasses import dataclass, field from typing import Optional @dataclass class EpisodicMemory: reflections: list[str] = field(default_factory=list) max_entries: int = 5 def add(self, reflection: str) -> None: self.reflections.append(reflection) if len(self.reflections) > self.max_entries: self.reflections.pop(0) def format_for_context(self) -> str: if not self.reflections: return "" entries = "\n".join(f"- {r}" for r in self.reflections) return f"Past reflections:\n{entries}\n" def simulate_environment(action: str, target: str) -> tuple[bool, str]: """Stub: replace with real task environment.""" success = target.lower() in action.lower() obs = "Task completed." if success else f"Failed. Expected to address: {target}" return success, obs def reflexion_episode( task: str, target: str, memory: EpisodicMemory, max_steps: int = 5, ) -> tuple[bool, EpisodicMemory]: context = memory.format_for_context() action = call_llm( f"{context}Task: {task}\nYour action:" ) success, observation = simulate_environment(action, target) if not success: reflection = call_llm( f"Task: {task}\nAction taken: {action}\nResult: {observation}\n" "What went wrong and how would you approach this differently next time? " "Be specific and concise." ) memory.add(reflection) return success, memory memory = EpisodicMemory() for episode in range(3): success, memory = reflexion_episode( task="Find the capital of the country where the Eiffel Tower is located.", target="Paris", memory=memory, ) print(f"Episode {episode + 1}: {'Success' if success else 'Failure'}, " f"memory entries={len(memory.reflections)}") ``` ## 4. CRITIC: Tool-Interactive Critiquing Both Self-Refine and Reflexion rely on the model's own judgment to evaluate outputs. Gou et al. (2023) identified this as the fundamental limitation: a model that produces an incorrect mathematical derivation is unlikely to reliably identify that derivation as incorrect, because the same reasoning process that generated the error will evaluate it. CRITIC (Critiquing with Tools for Iterative Correction) breaks this circular dependency by grounding critiques in external tools that can verify claims objectively. ### 4.1 Tool-Grounded Verification The CRITIC pipeline extends the generate-critique-refine loop by inserting a tool-use step into the critique phase: $$c^{(t)} = \text{Critique}\!\left(y^{(t)},\; \text{Tool}\!\left(q^{(t)}, y^{(t)}\right)\right)$$ where $q^{(t)}$ is a verification query generated by the model from $y^{(t)}$ and $\text{Tool}(\cdot)$ is an external verifier. The choice of tool depends on the claim type: - **Mathematical claims**: a symbolic math engine (WolframAlpha, SymPy) evaluates expressions and checks derivations. - **Code**: a Python interpreter executes the code and returns stdout, stderr, and exit codes. - **Factual claims**: a search engine retrieves current evidence against which the claim is checked. The model's critique is then conditioned on the concrete tool output rather than the model's unaided assessment. If WolframAlpha returns that $\int_0^1 x^2\,dx = 1/3$ but the model's solution states $1/2$, the critique can note the specific discrepancy with a concrete numerical ground truth. ### 4.2 Formal Grounding Advantage Let $\epsilon_\text{gen}$ be the model's error rate on a task and $\epsilon_\text{eval}$ be its error rate at evaluating its own outputs on the same task. Self-Refine improves over a single pass only when $\epsilon_\text{eval} < \epsilon_\text{gen}$: the model must be better at spotting errors than at avoiding them. For factual and mathematical tasks, empirical evidence suggests this condition often fails; the model's self-evaluation error rate approaches its generation error rate. External tools invert this relationship. A Python interpreter has $\epsilon_\text{eval} \approx 0$ for functional correctness: if the code runs and produces the right output, it is correct. Symbolic computation engines have near-zero error on arithmetic and algebraic simplification. Web search introduces latency and relevance noise but still substantially reduces $\epsilon_\text{eval}$ for factual verification. CRITIC therefore reliably improves over Self-Refine precisely where self-correction tends to fail. ### 4.3 Results Gou et al. (2023) evaluated CRITIC on free-form question answering (TriviaQA, AmbigNL), mathematical reasoning (GSM8K), and program synthesis (MBPP, HumanEval). On GSM8K with GPT-4, CRITIC reached 95.0% accuracy versus 91.6% for chain-of-thought alone. On HumanEval, it reached 85.1% pass@1 versus 67.0% for unassisted GPT-4. On TriviaQA, accuracy improved from 75.3% to 82.4%, with the search tool providing concrete evidence the model could not hallucinate past. ## 5. When Self-Correction Helps Versus Hurts Sun et al. (2023) posed the most important cautionary question in this space: is self-correction actually reliable, or does it degrade outputs when the model is miscalibrated? Their controlled study asked GPT-4 to solve reasoning problems, then instructed it to reconsider and potentially revise its answer. Without any external signal, self-correction made outputs worse on reasoning tasks. The model would revise a correct answer to an incorrect one because the instruction to reconsider suggested that revision was expected. This is not a failure of intelligence but of calibration: the model cannot distinguish between "I made an error and should revise" and "the prompt is asking me to revise regardless." The findings crystallize into a principle: **self-correction without external feedback is unreliable; self-correction with external feedback is consistently beneficial.** External feedback can take many forms: - Unit test results (code generation) - Symbolic computation output (mathematics) - Search engine results (factual claims) - Human evaluation scores (open-ended quality) - Environmental reward signals (sequential decision-making) - Constrained optimization feasibility checks (structured generation) The signal need not be rich. Even binary pass/fail from a test suite is enough to direct improvement. What matters is that the feedback is generated by a process that is independent of the model's reasoning and therefore not subject to the same systematic biases. ## 6. The Reflection Design Pattern in Practice The three frameworks discussed above map onto a decision tree based on what feedback signals are available in a given deployment context. ```{mermaid} flowchart TD A["New agent task"] --> B{"External tool\ncan verify?"} B -- "Yes" --> C["CRITIC\nTool-grounded critique"] B -- "No" --> D{"Environment gives\nsuccess signal?"} D -- "Yes" --> E["Reflexion\nVerbal RL with memory"] D -- "No" --> F["Self-Refine\nPure self-feedback"] C --> G["Refined output"] E --> G F --> G ``` Several practical considerations govern implementation: **Prompt engineering for critique specificity.** Vague feedback ("this could be better") produces vague refinements. Effective critique prompts enumerate specific dimensions: for code, check correctness, efficiency, edge cases, and readability separately. For mathematical proofs, check each inference step. Structured feedback with labeled sections transfers more reliably to the refinement step. **Stopping criteria.** Fixed iteration counts are simple but wasteful when the output converges early. Self-reported confidence is cheap but unreliable for the same reasons self-correction is unreliable without external signals. The most robust stopping criterion for tool-verified tasks is the external tool's pass/fail signal. For Self-Refine, a practical heuristic is to stop when the feedback step produces a critique shorter than a threshold length, indicating diminishing marginal criticism. **Memory management in Reflexion.** Episodic memory grows with episodes and can exhaust the context window. Sliding window truncation (keeping only the $k$ most recent reflections) is the simplest approach. More sophisticated systems compress older reflections or index them for retrieval, treating the memory as a retrieval-augmented knowledge base rather than a flat list. **Computational cost.** Each refinement iteration multiplies inference cost. For a $T$-iteration loop, a task that normally costs $C$ tokens now costs approximately $(1 + 2T) \cdot C$ tokens: one generation, $T$ feedbacks, and $T$ refinements. The cost-quality trade-off must be evaluated per application; for tasks where correctness is critical and inference is cheap relative to human review, even five iterations may be cost-effective. ```python from typing import Callable def run_critic_loop( problem: str, generate_fn: Callable[[str], str], verify_fn: Callable[[str, str], tuple[bool, str]], refine_fn: Callable[[str, str, str], str], max_iterations: int = 4, ) -> dict: """ Generic CRITIC loop: generate, verify with external tool, critique, refine. generate_fn(problem) -> initial solution verify_fn(problem, solution) -> (passed: bool, tool_output: str) refine_fn(problem, solution, tool_output) -> refined solution """ solution = generate_fn(problem) trace = [{"step": 0, "solution": solution}] for i in range(1, max_iterations + 1): passed, tool_output = verify_fn(problem, solution) trace[-1]["tool_output"] = tool_output trace[-1]["passed"] = passed if passed: break solution = refine_fn(problem, solution, tool_output) trace.append({"step": i, "solution": solution}) return {"solution": solution, "iterations": len(trace), "trace": trace} def stub_generate(problem: str) -> str: return f"Initial attempt at: {problem}" def stub_verify(problem: str, solution: str) -> tuple[bool, str]: passed = "correct" in solution.lower() return passed, ("OK" if passed else "Solution does not contain verification keyword.") def stub_refine(problem: str, solution: str, tool_output: str) -> str: return f"Revised (correct) solution for: {problem}. Tool said: {tool_output}" result = run_critic_loop( problem="Compute the derivative of x^3 at x=2.", generate_fn=stub_generate, verify_fn=stub_verify, refine_fn=stub_refine, ) print(f"Converged in {result['iterations']} iteration(s).") print(f"Final: {result['solution'][:100]}") ``` ## 7. Connections to Broader Agent Architectures Reflection mechanisms are modular and compose naturally with other agent components. A ReAct agent (Yao et al., 2022) that interleaves reasoning and tool use can be augmented with Reflexion-style episodic memory across trials. A chain-of-thought reasoner can wrap its inner reasoning loop with Self-Refine. A code generation pipeline can insert CRITIC's execution-based verification before committing generated code to a codebase. The deepest connection is to the idea of test-time compute scaling: given a fixed model, better answers can be obtained by spending more inference-time computation. Reflection is one architecture for this; others include best-of-$n$ sampling, Monte Carlo tree search over reasoning steps, and process reward models. What distinguishes reflection from pure sampling approaches is that it uses the model's own structured feedback to direct the search, rather than sampling blindly and selecting post hoc. This directed search is more compute-efficient when the model's critiques are informative, and less efficient when they are not, which returns us to the calibration problem identified by Sun et al. Understanding when model self-evaluation is trustworthy, and designing hybrid architectures that route to external verification when it is not, is an active research frontier. The frameworks covered in this chapter represent the foundational vocabulary for that ongoing work. ## References 1. Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., ... & Clark, P. (2023). Self-Refine: Iterative Refinement with Self-Feedback. *arXiv preprint arXiv:2303.17651*. https://arxiv.org/abs/2303.17651 2. Shinn, N., Cassano, F., Labash, B., Gopinath, A., Narasimhan, K., & Yao, S. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning. *arXiv preprint arXiv:2303.11366*. https://arxiv.org/abs/2303.11366 3. Gou, Z., Shao, Z., Gong, Y., Shen, Y., Yang, Y., Duan, N., & Chen, W. (2023). CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing. *arXiv preprint arXiv:2305.11738*. https://arxiv.org/abs/2305.11738 4. Sun, Z., Shen, Y., Zhou, Q., Zhang, H., Chen, Z., Cox, D., ... & Gan, C. (2023). Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision. *arXiv preprint arXiv:2310.01798*. https://arxiv.org/abs/2310.01798

227.1 1. The Case for Iterative Refinement