CtrlK
CommunityDocumentationLog inGet started
Tessl Logo

tessl/pypi-toil

Pipeline management software for clusters.

Agent Success

Agent success rate when using this tile

67%

Improvement

Agent success rate improvement when using this tile compared to baseline

1.05x

Baseline

Agent success rate without this tile

64%

Overview
Eval results
Files

rubric.jsonevals/scenario-1/

{
  "context": "This evaluation assesses how well the engineer uses Toil's FileStore API for file management operations. The focus is on proper usage of import, export, read, and write methods, as well as temporary file handling within a multi-job workflow.",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "File Import",
      "description": "Uses fileStore.import_file() to import files from external URLs into the job store, correctly passing source URIs and storing the returned file IDs",
      "max_score": 20
    },
    {
      "name": "File Export",
      "description": "Uses fileStore.export_file() to export files from the job store to external destinations, correctly passing file IDs and destination URIs",
      "max_score": 20
    },
    {
      "name": "Read from Job Store",
      "description": "Uses fileStore.readGlobalFile() to download files from the job store to local filesystem for processing",
      "max_score": 20
    },
    {
      "name": "Write to Job Store",
      "description": "Uses fileStore.writeGlobalFile() to upload processed files from local filesystem to the job store and returns file IDs",
      "max_score": 20
    },
    {
      "name": "Temporary File Management",
      "description": "Uses fileStore.getLocalTempDir() or fileStore.getLocalTempFile() to create temporary storage locations for intermediate processing",
      "max_score": 10
    },
    {
      "name": "Job Chaining",
      "description": "Properly chains jobs using addChild() or addFollowOn() to establish dependencies and pass file IDs between jobs",
      "max_score": 10
    }
  ]
}
tessl i tessl/pypi-toil@9.0.0

tile.json