data-engineering
Method for scraping, transforming, validating and loading data through NDJSON. Use when building a scraper or ETL step, validating an NDJSON feed, or importing data into a CMS or database.
68
86%
Does it follow best practices?
Run evals on this skill
Adds up to 20 points to the overall score
Low
Low-risk findings worth noting
Data Engineering
The project's sources, record schema, scripts and commands live in .opencastle/stack/data-pipeline-config.md. Use those names; when a script it lists does not exist, write it rather than guessing at one. Starter scraper and validator: REFERENCE.md.
Scraper
Headless browser (Playwright, or Puppeteer Cluster for volume) with retries and a per-page timeout, so one slow page does not stall the run.
NDJSON Output
One record per line, valid JSON, fields as the project's schema names them. Every record carries a source-unique ID (e.g. source + sourceId) to deduplicate and import on; keep text in its original encoding.
Pipeline
- Scrape a sample of 50–200 records first; check it has the schema's required fields. Missing fields → fix the extractor selectors and re-run the sample.
- Validate every line (JSON parse + schema): 0 parse errors, all required fields. Isolate failing lines and inspect their source HTML.
- Dry-run the import against a staging target: counts within ±5% of expectation, no duplicates. Otherwise reset staging and adjust the dedupe key.
- Snapshot the target (timestamped export) before writing.
- Import with idempotent upserts keyed on the source ID; restore the snapshot on failure.
- Repository
monkilabs/opencastle
- Revision
3942f46
- Last updated
- First committed
Is this your skill?
If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.