Robust Data Processor

Build a resilient data processing service that streams JSON from an API and continues processing even when individual callbacks fail.

Background

Your service receives a large JSON array of user records from a streaming API. Some records may trigger errors during processing (invalid format, missing required fields, etc.), but the service must continue processing all remaining records and collect both successful results and error information.

Requirements

Data Processing Pipeline

Create a function processUserStream(url, onComplete) that:

1. Fetches JSON data from the provided URL using a streaming approach
2. Processes each user record as it arrives
3. Handles processing errors gracefully without stopping the stream
4. Collects successfully processed records
5. Tracks which records caused errors and why
6. Calls onComplete(results, errors) when done, where:
   • results is an array of successfully processed user records
   • errors is an array of objects containing error information (e.g., {index, message})

Record Processing Logic

For each user record in the stream at path users.*, apply this transformation:

• Convert the name field to uppercase (call .toUpperCase() on it)
• Add a processed: true flag
• Collect the transformed record with id, name, and email fields

Note: Some records may have missing or invalid name fields, which will cause .toUpperCase() to throw an error. Your solution must handle this gracefully.

Error Handling Requirements

• Transformation errors (e.g., calling .toUpperCase() on undefined) must NOT stop the stream
• Errors in processing one record must NOT prevent other records from being processed
• The stream must continue and complete successfully even when transformation errors occur
• Track which records failed to process and capture the error information
• Leverage the streaming library's error resilience capabilities

Test Cases

• It successfully processes valid user records and transforms names to uppercase @test
• It continues processing after a record with missing fields causes an error @test
• It captures error information for failed records without stopping the stream @test
• It processes all remaining records after multiple errors occur @test

Expected Input Format

{
  "users": [
    {"id": 1, "name": "alice", "email": "alice@example.com"},
    {"id": 2, "email": "bob@example.com"},
    {"id": 3, "name": "charlie", "email": "charlie@example.com"}
  ]
}

Expected Output Format

For the above input (where the second record has no name field), the function should produce:

results = [
  {id: 1, name: "ALICE", email: "alice@example.com", processed: true},
  {id: 3, name: "CHARLIE", email: "charlie@example.com", processed: true}
]

errors = [
  {index: 1, message: "Cannot read property 'toUpperCase' of undefined"} // or similar
]

Implementation

@generates

API

/**
 * Processes a stream of user records from a JSON API with error resilience.
 *
 * @param {string} url - The URL to fetch JSON data from
 * @param {Function} onComplete - Callback invoked when processing completes: onComplete(results, errors)
 * @returns {void}
 */
function processUserStream(url, onComplete) {
  // IMPLEMENTATION HERE
}

module.exports = { processUserStream };

Dependencies { .dependencies }

oboe { .dependency }

Provides streaming JSON parsing support.