CtrlK
BlogDocsLog inGet started
Tessl Logo

coding-agent-helpers/skeptic-verifier

Use when the user wants an adversarial double-check of a code or config change. Run the strongest checks available, try to break the claim, look for edge cases and hidden regressions, and return PASS, PARTIAL, or FAIL with evidence. Good triggers include "poke holes in this", "stress test this change", "double check this fix", and "try to break it".

84

1.30x
Quality

94%

Does it follow best practices?

Impact

81%

1.30x

Average score across 8 eval scenarios

SecuritybySnyk

Passed

No known issues

Overview
Quality
Evals
Security
Files

task.mdevals/scenario-8/

Poke Holes in the Configuration Validator

Problem/Feature Description

A platform team has built a configuration validator for their deployment pipeline. When a developer submits a new service configuration (a JSON file), the validator is supposed to catch invalid values before the configuration reaches the deployment system. A recent incident was caused by a configuration with a negative memory limit that passed validation silently.

The fix adds type and range checks to the validator. The engineer says the validator now correctly rejects invalid memory limits, CPU shares, and replica counts, and still accepts all valid configurations. The team wants you to verify the claim by trying to find configurations that should be rejected but are accepted, or valid configurations that are incorrectly rejected.

Output Specification

Write your findings to verification_report.md. Include specific configuration inputs you tested, the actual validation results, and your verdict.

Input Files

The following files are provided as inputs. Extract them before beginning.

=============== FILE: src/config_validator.py =============== def validate_service_config(config: dict) -> list: """ Validates a service deployment configuration. Returns a list of error strings. Empty list means valid.

Claimed to enforce:
  - memory_mb: integer, 64 <= value <= 65536
  - cpu_shares: integer, 1 <= value <= 1024
  - replicas: integer, 1 <= value <= 50
  - name: non-empty string
"""
errors = []

# Validate name
name = config.get("name")
if not name or not isinstance(name, str):
    errors.append("name must be a non-empty string")

# Validate memory_mb
memory = config.get("memory_mb")
if memory is None:
    errors.append("memory_mb is required")
elif not isinstance(memory, int):
    errors.append("memory_mb must be an integer")
elif memory < 64 or memory > 65536:
    errors.append(f"memory_mb must be between 64 and 65536, got {memory}")

# Validate cpu_shares
cpu = config.get("cpu_shares")
if cpu is None:
    errors.append("cpu_shares is required")
elif not isinstance(cpu, int):
    errors.append("cpu_shares must be an integer")
elif cpu < 1 or cpu > 1024:
    errors.append(f"cpu_shares must be between 1 and 1024, got {cpu}")

# Validate replicas
replicas = config.get("replicas")
if replicas is None:
    errors.append("replicas is required")
elif not isinstance(replicas, int):
    errors.append("replicas must be an integer")
elif replicas < 1 or replicas > 50:
    errors.append(f"replicas must be between 1 and 50, got {replicas}")

return errors

evals

tile.json