CtrlK
CommunityDocumentationLog inGet started
Tessl Logo

tessl/pypi-toil

Pipeline management software for clusters.

Agent Success

Agent success rate when using this tile

67%

Improvement

Agent success rate improvement when using this tile compared to baseline

1.05x

Baseline

Agent success rate without this tile

64%

Overview
Eval results
Files

rubric.jsonevals/scenario-10/

{
  "context": "This criteria evaluates how well the engineer uses Toil's promise/future system to pass data between jobs in a multi-stage pipeline. The focus is on proper use of job.rv() for returning values, promise path selection for accessing nested data structures, and constructing workflows where jobs receive data via promises from predecessor jobs.",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "Job return values",
      "description": "All three job types (splitter, calculator, aggregator) properly return dictionaries using standard Python return statements, making values accessible via job.rv() promises",
      "max_score": 15
    },
    {
      "name": "Promise creation",
      "description": "Uses job.rv() method to create promises from job return values that can be passed to successor jobs",
      "max_score": 20
    },
    {
      "name": "Promise path selection",
      "description": "Uses promise path indexing (e.g., promise['even_indices'], promise['odd_indices']) to access nested data from the splitter job's dictionary return value",
      "max_score": 25
    },
    {
      "name": "Promise as arguments",
      "description": "Passes promises as arguments when creating calculator and aggregator jobs, enabling data flow from predecessor jobs",
      "max_score": 20
    },
    {
      "name": "Job relationships",
      "description": "Correctly establishes job dependencies using addChild() or addFollowOn() methods to ensure proper execution order (splitter -> calculators -> aggregator)",
      "max_score": 15
    },
    {
      "name": "Parallel execution",
      "description": "Structures the workflow so both calculator jobs can execute in parallel (both as children/follow-ons of the splitter, not sequentially dependent on each other)",
      "max_score": 5
    }
  ]
}
tessl i tessl/pypi-toil@9.0.0

tile.json