CtrlK
CommunityDocumentationLog inGet started
Tessl Logo

tessl/pypi-toil

Pipeline management software for clusters.

Agent Success

Agent success rate when using this tile

67%

Improvement

Agent success rate improvement when using this tile compared to baseline

1.05x

Baseline

Agent success rate without this tile

64%

Overview
Eval results
Files

rubric.jsonevals/scenario-6/

{
  "context": "This criteria evaluates how well the engineer uses Toil's API to implement conditional workflow execution. The focus is on proper use of Job classes, the promise/return value system for passing data between jobs, and conditional job graph construction.",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "Job class definition",
      "description": "All processing components (QualityCheckJob, FullProcessingJob, PartialProcessingJob, ErrorReportJob, DataWorkflow) are defined as classes that inherit from toil.job.Job",
      "max_score": 15
    },
    {
      "name": "Job.run() implementation",
      "description": "Each job class properly overrides the run(self, fileStore) method to implement its logic",
      "max_score": 10
    },
    {
      "name": "Return value usage",
      "description": "QualityCheckJob returns a quality score value from its run() method that can be accessed by successor jobs",
      "max_score": 15
    },
    {
      "name": "Promise system",
      "description": "Uses job.rv() or promise mechanism to pass the quality score from QualityCheckJob to DataWorkflow for conditional branching decisions",
      "max_score": 20
    },
    {
      "name": "Conditional job addition",
      "description": "DataWorkflow conditionally adds child jobs (addChild() or addFollowOn()) based on the quality score value: FullProcessingJob for score >= 80, PartialProcessingJob for 50-79, ErrorReportJob for < 50",
      "max_score": 25
    },
    {
      "name": "Workflow execution",
      "description": "Uses Toil context manager (with Toil(options) as toil:) and toil.start() method to execute the workflow with the root job",
      "max_score": 10
    },
    {
      "name": "Correct test outcomes",
      "description": "The workflow produces correct outputs for all three test cases: returns appropriate result string based on which job path was executed",
      "max_score": 5
    }
  ]
}
tessl i tessl/pypi-toil@9.0.0

tile.json