Pragmatic Testing Framework for Python with BDD-style syntax and pluggable architecture
{
"context": "Evaluates how the solution uses Vedro's seeding controls to achieve deterministic runs, scenario-level overrides, and explicit seed reporting for reproducibility. Focuses solely on correct application of run-level flags and @seed overrides rather than general code quality.",
"type": "weighted_checklist",
"checklist": [
{
"name": "Run seed flag",
"description": "Invokes Vedro with `--seed <value>` (or equivalent run arguments) so rerunning with the same seed yields identical scenario outputs.",
"max_score": 25
},
{
"name": "Generated seed reuse",
"description": "When no seed is provided, enables `--show-seeds` or Seeder config to surface the generated run seed and demonstrates that rerunning with that seed reproduces the prior outputs.",
"max_score": 15
},
{
"name": "Scenario override",
"description": "Marks at least one scenario with `@seed(<int>)` to give it a deterministic seed independent of the run-level seed.",
"max_score": 25
},
{
"name": "Override precedence",
"description": "Shows that a scenario decorated with `@seed` keeps its override even when the run is started with `--seed`, producing output that differs from run-seeded scenarios while remaining repeatable.",
"max_score": 15
},
{
"name": "Seed reporting",
"description": "Run output/report explicitly includes both the run seed and any scenario-level seed values (e.g., via `--show-seeds` or reporter extras) rather than keeping them implicit.",
"max_score": 20
}
]
}tessl i tessl/pypi-vedro@1.14.0docs
evals
scenario-1
scenario-2
scenario-3
scenario-4
scenario-5
scenario-6
scenario-7
scenario-8
scenario-9
scenario-10