CtrlK
CommunityDocumentationLog inGet started
Tessl Logo

tessl/pypi-toil

Pipeline management software for clusters.

Agent Success

Agent success rate when using this tile

67%

Improvement

Agent success rate improvement when using this tile compared to baseline

1.05x

Baseline

Agent success rate without this tile

64%

Overview
Eval results
Files

rubric.jsonevals/scenario-7/

{
  "context": "This criteria evaluates how well the engineer uses Toil's job dependency management features (addChild and addFollowOn) to create a multi-stage data processing pipeline with correct execution order and parallelization.",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "Job class definition",
      "description": "Defines job classes that inherit from toil.job.Job for the different pipeline stages (e.g., DownloadJob, ProcessJob, AggregateJob, CleanupJob or similar)",
      "max_score": 15
    },
    {
      "name": "addChild usage",
      "description": "Uses Job.addChild() method to add parallel jobs (download jobs and/or processing jobs) that should execute concurrently",
      "max_score": 25
    },
    {
      "name": "addFollowOn usage",
      "description": "Uses Job.addFollowOn() method to create sequential stages that execute after children complete (e.g., processing coordinator after downloads, aggregation after processing, cleanup after aggregation)",
      "max_score": 25
    },
    {
      "name": "Correct execution order",
      "description": "Implements the job hierarchy such that stages execute in the correct order: downloads in parallel, then processing in parallel, then aggregation, then cleanup",
      "max_score": 20
    },
    {
      "name": "Workflow initialization",
      "description": "Uses toil.common.Toil class or similar to initialize and start the workflow with the root job and job store",
      "max_score": 10
    },
    {
      "name": "Root job coordination",
      "description": "Creates a root/coordinator job that properly sets up the job dependency chain using addChild and addFollowOn to coordinate the entire workflow",
      "max_score": 5
    }
  ]
}
tessl i tessl/pypi-toil@9.0.0

tile.json