CtrlK
CommunityDocumentationLog inGet started
Tessl Logo

tessl/pypi-vedro

Pragmatic Testing Framework for Python with BDD-style syntax and pluggable architecture

49%

Overall

Evaluation49%

1.09x

Agent success when using this tile

Overview
Eval results
Files

rubric.jsonevals/scenario-5/

{
  "context": "Evaluates how well the solution leverages Vedro's built-in selection and ordering controls to match the spec. Scoring rewards correct use of CLI filters, decorators, and plugins rather than general test structure. Only package-level behaviors that affect which scenarios run and in what order are considered.",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "Path filter",
      "description": "Uses Vedro path targeting (e.g., passing scenario directories/files to `vedro run` or configuring `ScenarioFinder`) so only the requested `paths` are discovered and executed.",
      "max_score": 20
    },
    {
      "name": "Subject filter",
      "description": "Applies Vedro's subject matching (such as the `--subject` CLI argument or equivalent runner configuration) so only scenarios whose `subject` contains the provided substring are executed.",
      "max_score": 15
    },
    {
      "name": "Tag expressions",
      "description": "Relies on Vedro's tag filters (`--tags` with logical expressions and `--exclude-tags`) via the Tagger plugin instead of custom filtering, ensuring include/exclude behavior matches the spec.",
      "max_score": 25
    },
    {
      "name": "Focus and skip",
      "description": "Implements execution control with Vedro's `@only` focus marker and `@skip`/`@skip_if` decorators so focused scenarios run exclusively and skipped ones report as skipped without executing steps.",
      "max_score": 20
    },
    {
      "name": "Ordering and seed",
      "description": "Configures Vedro's Orderer plugin (`--order` values declaration/reversed/random) and pairs random ordering with Seeder support (`--seed` or fixed_seed) to produce repeatable random sequences.",
      "max_score": 20
    }
  ]
}
tessl i tessl/pypi-vedro@1.14.0

tile.json