Pragmatic Testing Framework for Python with BDD-style syntax and pluggable architecture
{
"context": "Evaluates how well the solution leverages Vedro's built-in selection and ordering controls to match the spec. Scoring rewards correct use of CLI filters, decorators, and plugins rather than general test structure. Only package-level behaviors that affect which scenarios run and in what order are considered.",
"type": "weighted_checklist",
"checklist": [
{
"name": "Path filter",
"description": "Uses Vedro path targeting (e.g., passing scenario directories/files to `vedro run` or configuring `ScenarioFinder`) so only the requested `paths` are discovered and executed.",
"max_score": 20
},
{
"name": "Subject filter",
"description": "Applies Vedro's subject matching (such as the `--subject` CLI argument or equivalent runner configuration) so only scenarios whose `subject` contains the provided substring are executed.",
"max_score": 15
},
{
"name": "Tag expressions",
"description": "Relies on Vedro's tag filters (`--tags` with logical expressions and `--exclude-tags`) via the Tagger plugin instead of custom filtering, ensuring include/exclude behavior matches the spec.",
"max_score": 25
},
{
"name": "Focus and skip",
"description": "Implements execution control with Vedro's `@only` focus marker and `@skip`/`@skip_if` decorators so focused scenarios run exclusively and skipped ones report as skipped without executing steps.",
"max_score": 20
},
{
"name": "Ordering and seed",
"description": "Configures Vedro's Orderer plugin (`--order` values declaration/reversed/random) and pairs random ordering with Seeder support (`--seed` or fixed_seed) to produce repeatable random sequences.",
"max_score": 20
}
]
}tessl i tessl/pypi-vedro@1.14.0docs
evals
scenario-1
scenario-2
scenario-3
scenario-4
scenario-5
scenario-6
scenario-7
scenario-8
scenario-9
scenario-10