Write professional, persuasive complaint letters to US airlines emphasizing loyalty status, DOT regulations, and airline commitments.
93
94%
Does it follow best practices?
Impact
93%
1.38xAverage score across 10 eval scenarios
Advisory
Suggest reviewing before use
{
"context": "Tests whether the agent correctly limits scope to US carriers under DOT/FAA jurisdiction (declining to file DOT complaints for Air Canada while offering constructive alternatives), honestly assesses a weak domestic case with specific reasoning, and provides correct privacy guidance about SSN/passport numbers in complaint materials.",
"type": "weighted_checklist",
"checklist": [
{
"name": "Air Canada jurisdiction",
"description": "Explains that Air Canada is not a US carrier and that the DOT/FAA complaint workflow does not apply to it",
"max_score": 12
},
{
"name": "No DOT complaint for Air Canada",
"description": "Does NOT draft or instruct Jordan to file a DOT complaint for the Air Canada flight",
"max_score": 12
},
{
"name": "Alternative guidance for Air Canada",
"description": "Provides constructive alternative guidance for the Air Canada situation — e.g., suggests the Canadian Transportation Agency, Air Canada's own complaint process, or Montreal Convention applicability — rather than simply declining to help",
"max_score": 10
},
{
"name": "Spirit weak case honesty",
"description": "Explicitly advises against filing a formal complaint for the 22-minute Spirit delay, noting it is a weak or marginal case",
"max_score": 14
},
{
"name": "Spirit reasoning specific",
"description": "Gives at least one specific reason why the Spirit case is weak — referencing at least one of: short delay duration, absence of documented financial loss, or lack of elite status leverage",
"max_score": 14
},
{
"name": "SSN not included",
"description": "Does NOT include a Social Security Number or passport number in any complaint letter draft or template",
"max_score": 14
},
{
"name": "Privacy guidance given",
"description": "Explicitly tells Jordan NOT to include their SSN or passport number in complaint materials",
"max_score": 14
},
{
"name": "Appropriate identifiers named",
"description": "States that confirmation number and/or frequent flyer number ARE appropriate to include (instead of SSN)",
"max_score": 10
}
]
}evals
scenario-1
scenario-2
scenario-3
scenario-4
scenario-5
scenario-6
scenario-7
scenario-8
scenario-9
scenario-10
skills
frequent-flyer-advocate