Pragmatic Testing Framework for Python with BDD-style syntax and pluggable architecture
{
"context": "Evaluates how the solution drives Vedro's CLI runner using vedro.cfg.py and the plugin registry. Checks that suites run through official commands, config wires plugins, and plugin toggling/listing is done via the package tools.",
"type": "weighted_checklist",
"checklist": [
{
"name": "vedro run",
"description": "run_suite launches the suite through the `vedro run` CLI (or vedro.run entrypoint) and passes `--project-dir`/path arguments so execution relies on the framework runner instead of manual scenario execution.",
"max_score": 25
},
{
"name": "Config loading",
"description": "Execution picks up settings from `project_dir/vedro.cfg.py`, using a Config subclass to define reporters and plugins via Config.plugins rather than hard-coding them in code.",
"max_score": 20
},
{
"name": "Plugin wiring",
"description": "The audit plugin subclass extends `vedro.core.Plugin` and is registered through a `PluginConfig` (or added to Config.plugins) so it is visible to the CLI registry and runs when enabled.",
"max_score": 20
},
{
"name": "Enable/disable",
"description": "Plugin state changes rely on `vedro plugin enable` / `vedro plugin disable` commands (or equivalent plugin manager toggles) and runs show the audit log when enabled and no log when disabled.",
"max_score": 20
},
{
"name": "Registry listing",
"description": "Solution surfaces the output of `vedro plugin list` (or `vedro plugin`) and shows the audit plugin with correct enabled/disabled status after toggling.",
"max_score": 15
}
]
}tessl i tessl/pypi-vedro@1.14.0docs
evals
scenario-1
scenario-2
scenario-3
scenario-4
scenario-5
scenario-6
scenario-7
scenario-8
scenario-9
scenario-10