CtrlK
CommunityDocumentationLog inGet started
Tessl Logo

tessl/pypi-toil

Pipeline management software for clusters.

Agent Success

Agent success rate when using this tile

67%

Improvement

Agent success rate improvement when using this tile compared to baseline

1.05x

Baseline

Agent success rate without this tile

64%

Overview
Eval results
Files

rubric.jsonevals/scenario-9/

{
  "context": "This criteria evaluates an engineer's ability to use Toil's job graph construction and validation capabilities. It focuses on proper use of Job class hierarchy, dependency management methods (addChild, addFollowOn), and graph validation techniques for cycle detection and connectivity verification.",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "Job class usage",
      "description": "Uses Job class or Job.wrapFn() to define workflow jobs. Jobs inherit from Job or are properly wrapped as Toil job objects.",
      "max_score": 15
    },
    {
      "name": "addChild dependency setup",
      "description": "Uses addChild() method to establish parent-child relationships where children execute after parent completes. Correctly models the parallel processing jobs as children of the root job.",
      "max_score": 20
    },
    {
      "name": "addFollowOn dependency setup",
      "description": "Uses addFollowOn() method to establish follow-on relationships where follow-ons execute after all children and descendants complete. Correctly models aggregation and merge jobs as follow-ons.",
      "max_score": 20
    },
    {
      "name": "Job graph construction",
      "description": "Constructs a complete job graph matching the specification with proper dependency ordering: root -> parallel processors -> aggregators -> final merge.",
      "max_score": 15
    },
    {
      "name": "Cycle detection validation",
      "description": "Implements or uses Toil's capabilities to detect cycles in the job dependency graph. The validation correctly identifies when circular dependencies exist.",
      "max_score": 15
    },
    {
      "name": "Connectivity validation",
      "description": "Implements or uses Toil's capabilities to verify all jobs are reachable from the root job. The validation correctly identifies orphaned or unreachable jobs.",
      "max_score": 15
    }
  ]
}
tessl i tessl/pypi-toil@9.0.0

tile.json