CtrlK
CommunityDocumentationLog inGet started
Tessl Logo

tessl/pypi-vedro

Pragmatic Testing Framework for Python with BDD-style syntax and pluggable architecture

49%

Overall

Evaluation49%

1.09x

Agent success when using this tile

Overview
Eval results
Files

rubric.jsonevals/scenario-1/

{
  "context": "Evaluates how the solution uses Vedro's seeding controls to achieve deterministic runs, scenario-level overrides, and explicit seed reporting for reproducibility. Focuses solely on correct application of run-level flags and @seed overrides rather than general code quality.",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "Run seed flag",
      "description": "Invokes Vedro with `--seed <value>` (or equivalent run arguments) so rerunning with the same seed yields identical scenario outputs.",
      "max_score": 25
    },
    {
      "name": "Generated seed reuse",
      "description": "When no seed is provided, enables `--show-seeds` or Seeder config to surface the generated run seed and demonstrates that rerunning with that seed reproduces the prior outputs.",
      "max_score": 15
    },
    {
      "name": "Scenario override",
      "description": "Marks at least one scenario with `@seed(<int>)` to give it a deterministic seed independent of the run-level seed.",
      "max_score": 25
    },
    {
      "name": "Override precedence",
      "description": "Shows that a scenario decorated with `@seed` keeps its override even when the run is started with `--seed`, producing output that differs from run-seeded scenarios while remaining repeatable.",
      "max_score": 15
    },
    {
      "name": "Seed reporting",
      "description": "Run output/report explicitly includes both the run seed and any scenario-level seed values (e.g., via `--show-seeds` or reporter extras) rather than keeping them implicit.",
      "max_score": 20
    }
  ]
}
tessl i tessl/pypi-vedro@1.14.0

tile.json