Query and download public cancer imaging data from NCI Imaging Data Commons using idc-index. Use for accessing large-scale radiology (CT, MR, PET) and pathology datasets for AI training or research. No authentication required. Query by metadata, visualize in browser, check licenses.
Install with Tessl CLI
npx tessl i github:K-Dense-AI/claude-scientific-skills --skill imaging-data-commons89
Quality
92%
Does it follow best practices?
Impact
73%
1.35xAverage score across 3 eval scenarios
Use the idc-index Python package to query and download public cancer imaging data from the National Cancer Institute Imaging Data Commons (IDC). No authentication required for data access.
Current IDC Data Version: v23 (always verify with IDCClient().get_idc_version())
Primary tool: idc-index (GitHub)
CRITICAL - Check package version and upgrade if needed (run this FIRST):
import idc_index
REQUIRED_VERSION = "0.11.10" # Must match metadata.idc-index in this file
installed = idc_index.__version__
if installed < REQUIRED_VERSION:
print(f"Upgrading idc-index from {installed} to {REQUIRED_VERSION}...")
import subprocess
subprocess.run(["pip3", "install", "--upgrade", "--break-system-packages", "idc-index"], check=True)
print("Upgrade complete. Restart Python to use new version.")
else:
print(f"idc-index {installed} meets requirement ({REQUIRED_VERSION})")Verify IDC data version and check current data scale:
from idc_index import IDCClient
client = IDCClient()
# Verify IDC data version (should be "v23")
print(f"IDC data version: {client.get_idc_version()}")
# Get collection count and total series
stats = client.sql_query("""
SELECT
COUNT(DISTINCT collection_id) as collections,
COUNT(DISTINCT analysis_result_id) as analysis_results,
COUNT(DISTINCT PatientID) as patients,
COUNT(DISTINCT StudyInstanceUID) as studies,
COUNT(DISTINCT SeriesInstanceUID) as series,
SUM(instanceCount) as instances,
SUM(series_size_MB)/1000000 as size_TB
FROM index
""")
print(stats)Core workflow:
client.sql_query()client.download_from_selection()client.get_viewer_URL(seriesInstanceUID=...)Core Sections (inline):
Reference Guides (load on demand):
| Guide | When to Load |
|---|---|
index_tables_guide.md | Complex JOINs, schema discovery, DataFrame access |
use_cases.md | End-to-end workflow examples (training datasets, batch downloads) |
sql_patterns.md | Quick SQL patterns for filter discovery, annotations, size estimation |
clinical_data_guide.md | Clinical/tabular data, imaging+clinical joins, value mapping |
cloud_storage_guide.md | Direct S3/GCS access, versioning, UUID mapping |
dicomweb_guide.md | DICOMweb endpoints, PACS integration |
digital_pathology_guide.md | Slide microscopy (SM), annotations (ANN), pathology workflows |
bigquery_guide.md | Full DICOM metadata, private elements (requires GCP) |
cli_guide.md | Command-line tools (idc download, manifest files) |
IDC adds two grouping levels above the standard DICOM hierarchy (Patient → Study → Series → Instance):
tcga_luad, nlst). A patient belongs to exactly one collection.Use collection_id to find original imaging data, may include annotations deposited along with the images; use analysis_result_id to find AI-generated or expert annotations.
Key identifiers for queries:
| Identifier | Scope | Use for |
|---|---|---|
collection_id | Dataset grouping | Filtering by project/study |
PatientID | Patient | Grouping images by patient |
StudyInstanceUID | DICOM study | Grouping of related series, visualization |
SeriesInstanceUID | DICOM series | Grouping of related series, visualization |
The idc-index package provides multiple metadata index tables, accessible via SQL or as pandas DataFrames.
Complete index table documentation: Use https://idc-index.readthedocs.io/en/latest/indices_reference.html for quick check of available tables and columns without executing any code.
Important: Use client.indices_overview to get current table descriptions and column schemas. This is the authoritative source for available columns and their types — always query it when writing SQL or exploring data structure.
| Table | Row Granularity | Loaded | Description |
|---|---|---|---|
index | 1 row = 1 DICOM series | Auto | Primary metadata for all current IDC data |
prior_versions_index | 1 row = 1 DICOM series | Auto | Series from previous IDC releases; for downloading deprecated data |
collections_index | 1 row = 1 collection | fetch_index() | Collection-level metadata and descriptions |
analysis_results_index | 1 row = 1 analysis result collection | fetch_index() | Metadata about derived datasets (annotations, segmentations) |
clinical_index | 1 row = 1 clinical data column | fetch_index() | Dictionary mapping clinical table columns to collections |
sm_index | 1 row = 1 slide microscopy series | fetch_index() | Slide Microscopy (pathology) series metadata |
sm_instance_index | 1 row = 1 slide microscopy instance | fetch_index() | Instance-level (SOPInstanceUID) metadata for slide microscopy |
seg_index | 1 row = 1 DICOM Segmentation series | fetch_index() | Segmentation metadata: algorithm, segment count, reference to source image series |
ann_index | 1 row = 1 DICOM ANN series | fetch_index() | Microscopy Bulk Simple Annotations series metadata; references annotated image series |
ann_group_index | 1 row = 1 annotation group | fetch_index() | Detailed annotation group metadata: graphic type, annotation count, property codes, algorithm |
contrast_index | 1 row = 1 series with contrast info | fetch_index() | Contrast agent metadata: agent name, ingredient, administration route (CT, MR, PT, XA, RF) |
Auto = loaded automatically when IDCClient() is instantiated
fetch_index() = requires client.fetch_index("table_name") to load
Key columns are not explicitly labeled, the following is a subset that can be used in joins.
| Join Column | Tables | Use Case |
|---|---|---|
collection_id | index, prior_versions_index, collections_index, clinical_index | Link series to collection metadata or clinical data |
SeriesInstanceUID | index, prior_versions_index, sm_index, sm_instance_index | Link series across tables; connect to slide microscopy details |
StudyInstanceUID | index, prior_versions_index | Link studies across current and historical data |
PatientID | index, prior_versions_index | Link patients across current and historical data |
analysis_result_id | index, analysis_results_index | Link series to analysis result metadata (annotations, segmentations) |
source_DOI | index, analysis_results_index | Link by publication DOI |
crdc_series_uuid | index, prior_versions_index | Link by CRDC unique identifier |
Modality | index, prior_versions_index | Filter by imaging modality |
SeriesInstanceUID | index, seg_index, ann_index, ann_group_index, contrast_index | Link segmentation/annotation/contrast series to its index metadata |
segmented_SeriesInstanceUID | seg_index → index | Link segmentation to its source image series (join seg_index.segmented_SeriesInstanceUID = index.SeriesInstanceUID) |
referenced_SeriesInstanceUID | ann_index → index | Link annotation to its source image series (join ann_index.referenced_SeriesInstanceUID = index.SeriesInstanceUID) |
Note: Subjects, Updated, and Description appear in multiple tables but have different meanings (counts vs identifiers, different update contexts).
For detailed join examples, schema discovery patterns, key columns reference, and DataFrame access, see references/index_tables_guide.md.
# Fetch clinical index (also downloads clinical data tables)
client.fetch_index("clinical_index")
# Query clinical index to find available tables and their columns
tables = client.sql_query("SELECT DISTINCT table_name, column_label FROM clinical_index")
# Load a specific clinical table as DataFrame
clinical_df = client.get_clinical_table("table_name")See references/clinical_data_guide.md for detailed workflows including value mapping patterns and joining clinical data with imaging.
| Method | Auth Required | Best For |
|---|---|---|
idc-index | No | Key queries and downloads (recommended) |
| IDC Portal | No | Interactive exploration, manual selection, browser-based download |
| BigQuery | Yes (GCP account) | Complex queries, full DICOM metadata |
| DICOMweb proxy | No | Tool integration via DICOMweb API |
| Cloud storage (S3/GCS) | No | Direct file access, bulk downloads, custom pipelines |
Cloud storage organization
IDC maintains all DICOM files in public cloud storage buckets mirrored between AWS S3 and Google Cloud Storage. Files are organized by CRDC UUIDs (not DICOM UIDs) to support versioning.
| Bucket (AWS / GCS) | License | Content |
|---|---|---|
idc-open-data / idc-open-data | No commercial restriction | >90% of IDC data |
idc-open-data-two / idc-open-idc1 | No commercial restriction | Collections with potential head scans |
idc-open-data-cr / idc-open-cr | Commercial use restricted (CC BY-NC) | ~4% of data |
Files are stored as <crdc_series_uuid>/<crdc_instance_uuid>.dcm. Access is free (no egress fees) via AWS CLI, gsutil, or s5cmd with anonymous access. Use series_aws_url column from the index for S3 URLs; GCS uses the same path structure.
See references/cloud_storage_guide.md for bucket details, access commands, UUID mapping, and versioning.
DICOMweb access
IDC data is available via DICOMweb interface (Google Cloud Healthcare API implementation) for integration with PACS systems and DICOMweb-compatible tools.
| Endpoint | Auth | Use Case |
|---|---|---|
| Public proxy | No | Testing, moderate queries, daily quota |
| Google Healthcare | Yes (GCP) | Production use, higher quotas |
See references/dicomweb_guide.md for endpoint URLs, code examples, supported operations, and implementation details.
Required (for basic access):
pip install --upgrade idc-indexImportant: New IDC data release will always trigger a new version of idc-index. Always use --upgrade flag while installing, unless an older version is needed for reproducibility.
IMPORTANT: IDC data version v23 is current. Always verify your version:
print(client.get_idc_version()) # Should return "v23"If you see an older version, upgrade with: pip install --upgrade idc-index
Tested with: idc-index 0.11.10 (IDC data version v23)
Optional (for data analysis):
pip install pandas numpy pydicomDiscover what imaging collections and data are available in IDC:
from idc_index import IDCClient
client = IDCClient()
# Get summary statistics from primary index
query = """
SELECT
collection_id,
COUNT(DISTINCT PatientID) as patients,
COUNT(DISTINCT SeriesInstanceUID) as series,
SUM(series_size_MB) as size_mb
FROM index
GROUP BY collection_id
ORDER BY patients DESC
"""
collections_summary = client.sql_query(query)
# For richer collection metadata, use collections_index
client.fetch_index("collections_index")
collections_info = client.sql_query("""
SELECT collection_id, CancerTypes, TumorLocations, Species, Subjects, SupportingData
FROM collections_index
""")
# For analysis results (annotations, segmentations), use analysis_results_index
client.fetch_index("analysis_results_index")
analysis_info = client.sql_query("""
SELECT analysis_result_id, analysis_result_title, Subjects, Collections, Modalities
FROM analysis_results_index
""")collections_index provides curated metadata per collection: cancer types, tumor locations, species, subject counts, and supporting data types — without needing to aggregate from the primary index.
analysis_results_index lists derived datasets (AI segmentations, expert annotations, radiomics features) with their source collections and modalities.
Query the IDC mini-index using SQL to find specific datasets.
First, explore available values for filter columns:
from idc_index import IDCClient
client = IDCClient()
# Check what Modality values exist
modalities = client.sql_query("""
SELECT DISTINCT Modality, COUNT(*) as series_count
FROM index
GROUP BY Modality
ORDER BY series_count DESC
""")
print(modalities)
# Check what BodyPartExamined values exist for MR modality
body_parts = client.sql_query("""
SELECT DISTINCT BodyPartExamined, COUNT(*) as series_count
FROM index
WHERE Modality = 'MR' AND BodyPartExamined IS NOT NULL
GROUP BY BodyPartExamined
ORDER BY series_count DESC
LIMIT 20
""")
print(body_parts)Then query with validated filter values:
# Find breast MRI scans (use actual values from exploration above)
results = client.sql_query("""
SELECT
collection_id,
PatientID,
SeriesInstanceUID,
Modality,
SeriesDescription,
license_short_name
FROM index
WHERE Modality = 'MR'
AND BodyPartExamined = 'BREAST'
LIMIT 20
""")
# Access results as pandas DataFrame
for idx, row in results.iterrows():
print(f"Patient: {row['PatientID']}, Series: {row['SeriesInstanceUID']}")To filter by cancer type, join with collections_index:
client.fetch_index("collections_index")
results = client.sql_query("""
SELECT i.collection_id, i.PatientID, i.SeriesInstanceUID, i.Modality
FROM index i
JOIN collections_index c ON i.collection_id = c.collection_id
WHERE c.CancerTypes LIKE '%Breast%'
AND i.Modality = 'MR'
LIMIT 20
""")Available metadata fields (use client.indices_overview for complete list):
Note: Cancer type is in collections_index.CancerTypes, not in the primary index table.
Download imaging data efficiently from IDC's cloud storage:
Download entire collection:
from idc_index import IDCClient
client = IDCClient()
# Download small collection (RIDER Pilot ~1GB)
client.download_from_selection(
collection_id="rider_pilot",
downloadDir="./data/rider"
)Download specific series:
# First, query for series UIDs
series_df = client.sql_query("""
SELECT SeriesInstanceUID
FROM index
WHERE Modality = 'CT'
AND BodyPartExamined = 'CHEST'
AND collection_id = 'nlst'
LIMIT 5
""")
# Download only those series
client.download_from_selection(
seriesInstanceUID=list(series_df['SeriesInstanceUID'].values),
downloadDir="./data/lung_ct"
)Custom directory structure:
Default dirTemplate: %collection_id/%PatientID/%StudyInstanceUID/%Modality_%SeriesInstanceUID
# Simplified hierarchy (omit StudyInstanceUID level)
client.download_from_selection(
collection_id="tcga_luad",
downloadDir="./data",
dirTemplate="%collection_id/%PatientID/%Modality"
)
# Results in: ./data/tcga_luad/TCGA-05-4244/CT/
# Flat structure (all files in one directory)
client.download_from_selection(
seriesInstanceUID=list(series_df['SeriesInstanceUID'].values),
downloadDir="./data/flat",
dirTemplate=""
)
# Results in: ./data/flat/*.dcmDownloaded file names:
Individual DICOM files are named using their CRDC instance UUID: <crdc_instance_uuid>.dcm (e.g., 0d73f84e-70ae-4eeb-96a0-1c613b5d9229.dcm). This UUID-based naming:
s3://idc-open-data/<crdc_series_uuid>/<crdc_instance_uuid>.dcm)To identify files, use the crdc_instance_uuid column in queries or read DICOM metadata (SOPInstanceUID) from the files.
The idc download command provides command-line access to download functionality without writing Python code. Available after installing idc-index.
Auto-detects input type: manifest file path, or identifiers (collection_id, PatientID, StudyInstanceUID, SeriesInstanceUID, crdc_series_uuid).
# Download entire collection
idc download rider_pilot --download-dir ./data
# Download specific series by UID
idc download "1.3.6.1.4.1.9328.50.1.69736" --download-dir ./data
# Download multiple items (comma-separated)
idc download "tcga_luad,tcga_lusc" --download-dir ./data
# Download from manifest file (auto-detected)
idc download manifest.txt --download-dir ./dataOptions:
| Option | Description |
|---|---|
--download-dir | Output directory (default: current directory) |
--dir-template | Directory hierarchy template (default: %collection_id/%PatientID/%StudyInstanceUID/%Modality_%SeriesInstanceUID) |
--log-level | Verbosity: debug, info, warning, error, critical |
Manifest files:
Manifest files contain S3 URLs (one per line) and can be:
Format (one S3 URL per line):
s3://idc-open-data/cb09464a-c5cc-4428-9339-d7fa87cfe837/*
s3://idc-open-data/88f3990d-bdef-49cd-9b2b-4787767240f2/*Example: Generate manifest from Python query:
from idc_index import IDCClient
client = IDCClient()
# Query for series URLs
results = client.sql_query("""
SELECT series_aws_url
FROM index
WHERE collection_id = 'rider_pilot' AND Modality = 'CT'
""")
# Save as manifest file
with open('ct_manifest.txt', 'w') as f:
for url in results['series_aws_url']:
f.write(url + '\n')Then download:
idc download ct_manifest.txt --download-dir ./ct_dataView DICOM data in browser without downloading:
from idc_index import IDCClient
import webbrowser
client = IDCClient()
# First query to get valid UIDs
results = client.sql_query("""
SELECT SeriesInstanceUID, StudyInstanceUID
FROM index
WHERE collection_id = 'rider_pilot' AND Modality = 'CT'
LIMIT 1
""")
# View single series
viewer_url = client.get_viewer_URL(seriesInstanceUID=results.iloc[0]['SeriesInstanceUID'])
webbrowser.open(viewer_url)
# View all series in a study (useful for multi-series exams like MRI protocols)
viewer_url = client.get_viewer_URL(studyInstanceUID=results.iloc[0]['StudyInstanceUID'])
webbrowser.open(viewer_url)The method automatically selects OHIF v3 for radiology or SLIM for slide microscopy. Viewing by study is useful when a DICOM Study contains multiple Series (e.g., T1, T2, DWI sequences from a single MRI session).
Check data licensing before use (critical for commercial applications):
from idc_index import IDCClient
client = IDCClient()
# Check licenses for all collections
query = """
SELECT DISTINCT
collection_id,
license_short_name,
COUNT(DISTINCT SeriesInstanceUID) as series_count
FROM index
GROUP BY collection_id, license_short_name
ORDER BY collection_id
"""
licenses = client.sql_query(query)
print(licenses)License types in IDC:
Important: Always check the license before using IDC data in publications or commercial applications. Each DICOM file is tagged with its specific license in metadata.
The source_DOI column contains DOIs linking to publications describing how the data was generated. To satisfy attribution requirements, use citations_from_selection() to generate properly formatted citations:
from idc_index import IDCClient
client = IDCClient()
# Get citations for a collection (APA format by default)
citations = client.citations_from_selection(collection_id="rider_pilot")
for citation in citations:
print(citation)
# Get citations for specific series
results = client.sql_query("""
SELECT SeriesInstanceUID FROM index
WHERE collection_id = 'tcga_luad' LIMIT 5
""")
citations = client.citations_from_selection(
seriesInstanceUID=list(results['SeriesInstanceUID'].values)
)
# Alternative format: BibTeX (for LaTeX documents)
bibtex_citations = client.citations_from_selection(
collection_id="tcga_luad",
citation_format=IDCClient.CITATION_FORMAT_BIBTEX
)Parameters:
collection_id: Filter by collection(s)patientId: Filter by patient ID(s)studyInstanceUID: Filter by study UID(s)seriesInstanceUID: Filter by series UID(s)citation_format: Use IDCClient.CITATION_FORMAT_* constants:
CITATION_FORMAT_APA (default) - APA styleCITATION_FORMAT_BIBTEX - BibTeX for LaTeXCITATION_FORMAT_JSON - CSL JSONCITATION_FORMAT_TURTLE - RDF TurtleBest practice: When publishing results using IDC data, include the generated citations to properly attribute the data sources and satisfy license requirements.
Process large datasets efficiently with filtering:
from idc_index import IDCClient
import pandas as pd
client = IDCClient()
# Find chest CT scans from GE scanners
query = """
SELECT
SeriesInstanceUID,
PatientID,
collection_id,
ManufacturerModelName
FROM index
WHERE Modality = 'CT'
AND BodyPartExamined = 'CHEST'
AND Manufacturer = 'GE MEDICAL SYSTEMS'
AND license_short_name = 'CC BY 4.0'
LIMIT 100
"""
results = client.sql_query(query)
# Save manifest for later
results.to_csv('lung_ct_manifest.csv', index=False)
# Download in batches to avoid timeout
batch_size = 10
for i in range(0, len(results), batch_size):
batch = results.iloc[i:i+batch_size]
client.download_from_selection(
seriesInstanceUID=list(batch['SeriesInstanceUID'].values),
downloadDir=f"./data/batch_{i//batch_size}"
)For queries requiring full DICOM metadata, complex JOINs, clinical data tables, or private DICOM elements, use Google BigQuery. Requires GCP account with billing enabled.
Quick reference:
bigquery-public-data.idc_current.*dicom_all (combined metadata)dicom_metadata (all DICOM tags)OtherElements column (vendor-specific tags like diffusion b-values)See references/bigquery_guide.md for setup, table schemas, query patterns, private element access, and cost optimization.
Before using BigQuery, always check if a specialized index table already has the metadata you need:
client.indices_overview or the idc-index indices reference to discover all available tables and their columnsclient.fetch_index("table_name")client.sql_query() (free, no GCP account needed)Common specialized indices: seg_index (segmentations), ann_index / ann_group_index (microscopy annotations), sm_index (slide microscopy), collections_index (collection metadata). Only use BigQuery if you need private DICOM elements or attributes not in any index.
| Task | Tool | Reference |
|---|---|---|
| Programmatic queries & downloads | idc-index | This document |
| Interactive exploration | IDC Portal | https://portal.imaging.datacommons.cancer.gov/ |
| Complex metadata queries | BigQuery | references/bigquery_guide.md |
| 3D visualization & analysis | SlicerIDCBrowser | https://github.com/ImagingDataCommons/SlicerIDCBrowser |
Default choice: Use idc-index for most tasks (no auth, easy API, batch downloads).
Integrate IDC data into imaging analysis workflows:
Read downloaded DICOM files:
import pydicom
import os
# Read DICOM files from downloaded series
series_dir = "./data/rider/rider_pilot/RIDER-1007893286/CT_1.3.6.1..."
dicom_files = [os.path.join(series_dir, f) for f in os.listdir(series_dir)
if f.endswith('.dcm')]
# Load first image
ds = pydicom.dcmread(dicom_files[0])
print(f"Patient ID: {ds.PatientID}")
print(f"Modality: {ds.Modality}")
print(f"Image shape: {ds.pixel_array.shape}")Build 3D volume from CT series:
import pydicom
import numpy as np
from pathlib import Path
def load_ct_series(series_path):
"""Load CT series as 3D numpy array"""
files = sorted(Path(series_path).glob('*.dcm'))
slices = [pydicom.dcmread(str(f)) for f in files]
# Sort by slice location
slices.sort(key=lambda x: float(x.ImagePositionPatient[2]))
# Stack into 3D array
volume = np.stack([s.pixel_array for s in slices])
return volume, slices[0] # Return volume and first slice for metadata
volume, metadata = load_ct_series("./data/lung_ct/series_dir")
print(f"Volume shape: {volume.shape}") # (z, y, x)Integrate with SimpleITK:
import SimpleITK as sitk
from pathlib import Path
# Read DICOM series
series_path = "./data/ct_series"
reader = sitk.ImageSeriesReader()
dicom_names = reader.GetGDCMSeriesFileNames(series_path)
reader.SetFileNames(dicom_names)
image = reader.Execute()
# Apply processing
smoothed = sitk.CurvatureFlow(image1=image, timeStep=0.125, numberOfIterations=5)
# Save as NIfTI
sitk.WriteImage(smoothed, "processed_volume.nii.gz")See references/use_cases.md for complete end-to-end workflow examples including:
client.get_idc_version() at the start of a session to confirm you're using the expected data version (currently v23). If using an older version, recommend pip install --upgrade idc-indexlicense_short_name field and respect licensing terms (CC BY vs CC BY-NC)citations_from_selection() to get properly formatted citations from source_DOI values; include these in publicationsLIMIT clause when exploring to avoid long downloads and understand data structure%collection_id/%PatientID/%ModalityIssue: ModuleNotFoundError: No module named 'idc_index'
pip install --upgrade idc-indexIssue: Download fails with connection timeout
dirTemplate to organize downloads by batchIssue: BigQuery quota exceeded or billing errors
references/bigquery_guide.md for cost optimization tipsIssue: Series UID not found or no data returned
LIMIT 5 to test query firstIssue: Downloaded DICOM files won't open
pydicom.dcmread(file, force=True)See references/sql_patterns.md for quick-reference SQL patterns including:
For segmentation and annotation details, also see references/digital_pathology_guide.md.
The following skills complement IDC workflows for downstream analysis and visualization:
See references/digital_pathology_guide.md for DICOM-compatible tools (highdicom, wsidicom, TIA-Toolbox, Slim viewer).
Always use client.indices_overview for current column schemas. This ensures accuracy with the installed idc-index version:
# Get all column names and types for any table
schema = client.indices_overview["index"]["schema"]
columns = [(c['name'], c['type'], c.get('description', '')) for c in schema['columns']]See the Quick Navigation section at the top for the full list of reference guides with decision triggers.
This skill version is available in skill metadata. To check for updates:
8158971
If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.