CtrlK
CommunityDocumentationLog inGet started
Tessl Logo

tessl/pypi-vedro

Pragmatic Testing Framework for Python with BDD-style syntax and pluggable architecture

49%

Overall

Evaluation49%

1.09x

Agent success when using this tile

Overview
Eval results
Files

rubric.jsonevals/scenario-6/

{
  "context": "This checklist evaluates whether the suite runner leans on Vedro's interruption and exit-shaping plugins to enforce fail-fast thresholds, respond to signals, and surface the framework's exit codes instead of reinventing them. It assumes scenarios are executed through Vedro's runner so the plugin behaviors are authoritative.",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "Interrupter hook",
      "description": "Registers and enables vedro.plugins.interrupter.Interrupter (via config or CLI) so interruption logic runs on every execution instead of custom stop checks.",
      "max_score": 20
    },
    {
      "name": "Fail-fast threshold",
      "description": "Maps the spec's max_failures argument to the Interrupter's fail-fast configuration/CLI flag (e.g., --fail-fast count) so runs stop after the configured number of scenario failures.",
      "max_score": 25
    },
    {
      "name": "Signal stop",
      "description": "Uses the Interrupter's built-in signal subscription (SIGINT/SIGTERM support) rather than manual os.signal handling to halt the run when the provided intercept_signal is emitted.",
      "max_score": 20
    },
    {
      "name": "Terminator exit codes",
      "description": "Keeps vedro.plugins.terminator.Terminator active so success, failure, and interruption return the framework's exit codes instead of hand-crafted integers.",
      "max_score": 20
    },
    {
      "name": "Allow-empty switch",
      "description": "Toggles Terminator's empty-suite handling (e.g., --no-scenarios-ok/NoScenariosOk setting) based on allow_empty so no-scenario runs yield code 0 only when explicitly allowed.",
      "max_score": 15
    }
  ]
}
tessl i tessl/pypi-vedro@1.14.0

tile.json