CtrlK
CommunityDocumentationLog inGet started
Tessl Logo

tessl/pypi-vedro

Pragmatic Testing Framework for Python with BDD-style syntax and pluggable architecture

49%

Overall

Evaluation49%

1.09x

Agent success when using this tile

Overview
Eval results
Files

rubric.jsonevals/scenario-8/

{
  "context": "Evaluates how well the solution leverages Vedro's plugin management CLI and configuration validation features. Focus is on using Vedro's plugin subcommands plus computed config fields to manage plugins safely and let Vedro reject bad plugin settings.",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "Plugin install",
      "description": "Uses Vedro's CLI (`vedro plugin install` and `vedro plugin enable` as needed) to provision requested plugins instead of stubbing or bypassing the command.",
      "max_score": 30
    },
    {
      "name": "Plugin listing",
      "description": "Calls `vedro plugin list` and parses its output or return data to report plugin names with enabled/disabled state.",
      "max_score": 15
    },
    {
      "name": "Computed options",
      "description": "Builds plugin configuration values through Vedro's `@computed` decorator (or equivalent computed config helper) rather than static literals.",
      "max_score": 20
    },
    {
      "name": "Validation flag",
      "description": "Keeps `Config.validate_plugins_configs` enabled (or explicitly sets it) so Vedro validates plugin settings before running scenarios.",
      "max_score": 20
    },
    {
      "name": "Rejects unknown",
      "description": "Relies on Vedro's plugin config validation to raise when unknown plugin options are supplied, instead of manual filtering or silent ignores.",
      "max_score": 15
    }
  ]
}
tessl i tessl/pypi-vedro@1.14.0

tile.json