Regression Tests for LLM Failure Traces

Problem/Feature Description

Your team maintains an LLM-powered Q&A service and has been collecting traces of observed failures. Two recent incidents have been flagged for regression coverage: one where the model fabricated a literary quote it presented as fact, and another where the model ignored explicit formatting instructions and wrapped its response in markdown fences instead of returning bare data.

The phoenix2pytest project provides a pipeline for turning these failure traces into committable pytest regression tests. Your job is to manually synthesize the two test files — one per failure trace — following the project's conventions for file layout, template structure, naming, and assertion style. Once committed, these tests will run in CI on every prompt edit or model upgrade to catch regressions.

You have been provided the two traces below. Synthesize the corresponding pytest files and write a brief synthesis_notes.md explaining the assertion strategy you chose for each test and why.

Trace 1

• Failure mode: hallucination
• User prompt: Quote the opening line of chapter 3 of Moby Dick verbatim.
• LLM output: The opening line of chapter 3 is: "It was the whiteness of the whale that above all things appalled me."
• Evidence: The quoted line does not appear as the chapter 3 opener — it is fabricated in this context.

Trace 2

• Failure mode: format_break
• User prompt: Return a JSON object with keys name and age for a fictional person. Return raw JSON only, no prose, no markdown fences.
• LLM output:

  Here is the JSON:
  ```json
  {"name": "Alice", "age": 30}

• Evidence: The output wraps the JSON in markdown fences and adds prose, violating the formatting instruction.

Output Specification

Produce the following files:

• generated_tests/test_hallucination.py — pytest regression test for the hallucination trace
• generated_tests/test_format_break.py — pytest regression test for the format_break trace
• synthesis_notes.md — a brief explanation of your assertion strategy choice for each test (2–4 sentences per test)

Each test file must be a complete, runnable Python file. Do not leave placeholder comments — write real assertions that would catch the described failure.