Data Ingestion Monitor

Build a system that ingests temperature sensor data into a time-series database and monitors the ingestion process. The system should track successful writes, handle errors gracefully, and provide visibility into the ingestion pipeline.

Requirements

Data Model

The system should ingest temperature readings with the following structure:

• Measurement name: temperature
• Tags: sensor_id (string), location (string)
• Fields: value (float, in Celsius)
• Timestamp: current time

Monitoring and Statistics

Implement a monitoring system that tracks:

1. Total number of successful batch writes
2. Total number of failed batch writes
3. Total number of retry attempts
4. A list of error messages from failed writes

The monitoring statistics should be accessible via a get_stats() function that returns a dictionary with keys: successful_batches, failed_batches, retry_attempts, and errors (list of error strings).

Batching Configuration

Configure the write API to:

• Write data in batches of 5 points
• Flush batches every 1000 milliseconds
• Use appropriate write options for batching

Data Generation

Create a function generate_sensor_data(count) that generates count temperature readings with:

• Random sensor IDs (choose from: "sensor_001", "sensor_002", "sensor_003")
• Random locations (choose from: "floor_1", "floor_2", "floor_3")
• Random temperature values between 18.0 and 26.0 degrees Celsius
• Current timestamp

Main Workflow

1. Initialize the database client with appropriate configuration
2. Set up monitoring callbacks to track write operations
3. Generate and write 15 temperature readings
4. Ensure all buffered data is written before closing
5. Print the monitoring statistics

Dependencies { .dependencies }

influxdb-client { .dependency }

Provides time-series database client functionality.

Test Cases

Test 1: Monitor successful writes { .test }

File: test_monitor.py

Description: Verify that successful batch writes are counted correctly.

Setup:

• Mock the write operation to always succeed
• Generate and write 15 data points with batch size of 5

Expected Result:

• successful_batches should be 3 (15 points / 5 per batch)
• failed_batches should be 0
• retry_attempts should be 0

Test 2: Track errors { .test }

File: test_monitor.py

Description: Verify that write errors are captured and counted.

Setup:

• Mock the write operation to fail on the second batch
• Generate and write 15 data points

Expected Result:

• failed_batches should be at least 1
• errors list should contain at least one error message
• Error tracking should not prevent processing of remaining batches

Test 3: Count retry attempts { .test }

File: test_monitor.py

Description: Verify that retry attempts are counted when writes fail with retryable errors.

Setup:

• Configure retry behavior
• Mock a retryable error scenario (e.g., rate limit)
• Trigger a write operation

Expected Result:

• retry_attempts should be greater than 0 when a retryable error occurs
• The retry callback should be invoked for retryable failures