0
# Iterator Patterns and Data Analysis
1

2
Comprehensive iteration capabilities for detailed exploration and analysis of histogram data. HdrHistogram provides multiple iterator types for different analysis patterns, from percentile exploration to linear bucket analysis.
3

4
## AbstractHistogramIterator
5

6
Base class for all histogram iterators providing common iteration functionality.
7

8
```java { .api }
9
public abstract class AbstractHistogramIterator implements Iterator<HistogramIterationValue> {
10
    
11
    // Iterator interface
12
    boolean hasNext();
13
    HistogramIterationValue next();
14
    void remove();  // Unsupported operation
15
    
16
    // Reset and reuse
17
    void reset();
18
    void reset(AbstractHistogram histogram);
19
    
20
    // Current state access
21
    long getValueIteratedTo();
22
    long getValueIteratedFrom();
23
    long getCountAtValueIteratedTo();
24
    long getCountAddedInThisIterationStep();
25
    long getTotalCountToThisValue();
26
    long getTotalValueToThisValue();
27
    double getPercentileIteratedTo();
28
    double getPercentileLevelIteratedTo();
29
}
30
```
31

32
## PercentileIterator
33

34
Iterates through histogram values according to percentile levels with exponentially decreasing resolution, focusing detail on higher percentiles.
35

36
```java { .api }
37
public class PercentileIterator extends AbstractHistogramIterator {
38
    
39
    // Constructor
40
    public PercentileIterator(AbstractHistogram histogram, int percentileTicksPerHalfDistance);
41
    
42
    // Reset for reuse
43
    void reset(int percentileTicksPerHalfDistance);
44
    void reset(AbstractHistogram histogram, int percentileTicksPerHalfDistance);
45
}
46
```
47

48
### Usage Examples
49

50
```java
51
// Create percentile iterator with 5 ticks per half distance
52
// This provides more detail at higher percentiles
53
PercentileIterator iterator = new PercentileIterator(histogram, 5);
54

55
System.out.println("Percentile Distribution:");
56
System.out.println("Value(μs) | Percentile | Count | TotalCount");
57
System.out.println("----------|------------|-------|------------");
58

59
while (iterator.hasNext()) {
60
    HistogramIterationValue iterationValue = iterator.next();
61
    
62
    System.out.printf("%8d | %9.4f%% | %5d | %10d%n",
63
        iterationValue.getValueIteratedTo(),
64
        iterationValue.getPercentileIteratedTo(),
65
        iterationValue.getCountAddedInThisIterationStep(),
66
        iterationValue.getTotalCountToThisValue());
67
}
68
```
69

70
### Percentile Resolution Control
71

72
The `percentileTicksPerHalfDistance` parameter controls resolution:
73

74
```java
75
// Low resolution - fewer data points, faster iteration
76
PercentileIterator coarse = new PercentileIterator(histogram, 1);
77

78
// Medium resolution - good balance  
79
PercentileIterator balanced = new PercentileIterator(histogram, 5);
80

81
// High resolution - more data points, detailed analysis
82
PercentileIterator detailed = new PercentileIterator(histogram, 10);
83

84
// Compare percentile coverage
85
System.out.println("Percentile Resolution Comparison:");
86
analyzePercentileResolution("Coarse (1)", coarse);
87
analyzePercentileResolution("Balanced (5)", balanced); 
88
analyzePercentileResolution("Detailed (10)", detailed);
89
```
90

91
### SLA Analysis with Percentiles
92

93
```java
94
public void analyzeSLACompliance(AbstractHistogram histogram, long slaThreshold) {
95
    PercentileIterator iterator = new PercentileIterator(histogram, 5);
96
    
97
    System.out.printf("SLA Analysis (threshold: %d μs):%n", slaThreshold);
98
    
99
    while (iterator.hasNext()) {
100
        HistogramIterationValue value = iterator.next();
101
        long latency = value.getValueIteratedTo();
102
        double percentile = value.getPercentileIteratedTo();
103
        
104
        // Highlight key percentiles and SLA violations
105
        if (percentile >= 50.0) {  // Focus on P50 and above
106
            String status = latency <= slaThreshold ? "✓" : "✗ VIOLATION";
107
            System.out.printf("P%.1f: %d μs %s%n", percentile, latency, status);
108
        }
109
        
110
        // Stop at 99.99th percentile for most analyses
111
        if (percentile >= 99.99) break;
112
    }
113
}
114
```
115

116
## LinearIterator
117

118
Iterates through histogram values using linear steps of equal size, ideal for uniform bucket analysis.
119

120
```java { .api }
121
public class LinearIterator extends AbstractHistogramIterator {
122
    
123
    // Constructor  
124
    public LinearIterator(AbstractHistogram histogram, long valueUnitsPerBucket);
125
    
126
    // Reset for reuse
127
    void reset(long valueUnitsPerBucket);
128
    void reset(AbstractHistogram histogram, long valueUnitsPerBucket);
129
}
130
```
131

132
### Usage Examples
133

134
```java
135
// Create linear iterator with 1000μs (1ms) buckets
136
LinearIterator iterator = new LinearIterator(histogram, 1000);
137

138
System.out.println("Linear Distribution (1ms buckets):");
139
System.out.println("Range(ms) | Count | Density");
140
System.out.println("----------|-------|--------");
141

142
while (iterator.hasNext()) {
143
    HistogramIterationValue iterationValue = iterator.next();
144
    
145
    if (iterationValue.getCountAddedInThisIterationStep() > 0) {
146
        long fromValue = iterationValue.getValueIteratedFrom();
147
        long toValue = iterationValue.getValueIteratedTo();
148
        long count = iterationValue.getCountAddedInThisIterationStep();
149
        
150
        System.out.printf("%3d - %3d | %5d | %s%n",
151
            fromValue / 1000, toValue / 1000, count,
152
            "*".repeat((int)(count / 100)));  // Simple ASCII histogram
153
    }
154
}
155
```
156

157
### Bucket-Based Analysis
158

159
```java
160
public void analyzeDistributionBuckets(AbstractHistogram histogram, long bucketSize) {
161
    LinearIterator iterator = new LinearIterator(histogram, bucketSize);
162
    
163
    List<BucketInfo> buckets = new ArrayList<>();
164
    
165
    while (iterator.hasNext()) {
166
        HistogramIterationValue value = iterator.next();
167
        
168
        if (value.getCountAddedInThisIterationStep() > 0) {
169
            buckets.add(new BucketInfo(
170
                value.getValueIteratedFrom(),
171
                value.getValueIteratedTo(),
172
                value.getCountAddedInThisIterationStep()
173
            ));
174
        }
175
    }
176
    
177
    // Find peak bucket
178
    BucketInfo peakBucket = buckets.stream()
179
        .max(Comparator.comparing(b -> b.count))
180
        .orElse(null);
181
        
182
    if (peakBucket != null) {
183
        System.out.printf("Peak distribution: %d-%d range with %d samples%n",
184
            peakBucket.fromValue, peakBucket.toValue, peakBucket.count);
185
    }
186
    
187
    // Analyze distribution spread
188
    long totalRange = histogram.getMaxValue() - histogram.getMinNonZeroValue();
189
    long activeRanges = buckets.size();
190
    
191
    System.out.printf("Distribution spread: %d active ranges out of %d total range%n",
192
        activeRanges, totalRange / bucketSize);
193
}
194
```
195

196
## LogarithmicIterator
197

198
Iterates through histogram values at logarithmically increasing levels, providing higher resolution at lower values.
199

200
```java { .api }
201
public class LogarithmicIterator extends AbstractHistogramIterator {
202
    
203
    // Constructor
204
    public LogarithmicIterator(AbstractHistogram histogram, 
205
                              long valueUnitsInFirstBucket, 
206
                              double logBase);
207
    
208
    // Reset for reuse
209
    void reset(long valueUnitsInFirstBucket, double logBase);
210
    void reset(AbstractHistogram histogram, long valueUnitsInFirstBucket, double logBase);
211
}
212
```
213

214
### Usage Examples
215

216
```java
217
// Create logarithmic iterator: first bucket = 10μs, base = 2.0  
218
LogarithmicIterator iterator = new LogarithmicIterator(histogram, 10, 2.0);
219

220
System.out.println("Logarithmic Distribution:");
221
System.out.println("Bucket Range | Count | Cumulative%");
222
System.out.println("-------------|-------|------------");
223

224
long totalCount = histogram.getTotalCount();
225

226
while (iterator.hasNext()) {
227
    HistogramIterationValue iterationValue = iterator.next();
228
    
229
    if (iterationValue.getCountAddedInThisIterationStep() > 0) {
230
        long fromValue = iterationValue.getValueIteratedFrom();
231
        long toValue = iterationValue.getValueIteratedTo();
232
        long count = iterationValue.getCountAddedInThisIterationStep();
233
        double cumulative = 100.0 * iterationValue.getTotalCountToThisValue() / totalCount;
234
        
235
        System.out.printf("%5d - %5d | %5d | %9.2f%%n",
236
            fromValue, toValue, count, cumulative);
237
    }
238
}
239
```
240

241
### Logarithmic Base Analysis
242

243
Different logarithmic bases provide different resolution patterns:
244

245
```java
246
public void compareLogarithmicBases(AbstractHistogram histogram) {
247
    double[] bases = {1.5, 2.0, 3.0, 10.0};
248
    
249
    for (double base : bases) {
250
        LogarithmicIterator iterator = new LogarithmicIterator(histogram, 1, base);
251
        
252
        int bucketCount = 0;
253
        while (iterator.hasNext()) {
254
            HistogramIterationValue value = iterator.next();
255
            if (value.getCountAddedInThisIterationStep() > 0) {
256
                bucketCount++;
257
            }
258
        }
259
        
260
        System.out.printf("Base %.1f: %d active buckets%n", base, bucketCount);
261
    }
262
}
263
```
264

265
## RecordedValuesIterator
266

267
Iterates through all recorded histogram values using finest granularity steps, including only non-zero counts.
268

269
```java { .api }
270
public class RecordedValuesIterator extends AbstractHistogramIterator {
271
    
272
    // Constructor
273
    public RecordedValuesIterator(AbstractHistogram histogram);
274
    
275
    // Reset for reuse
276
    void reset();
277
    void reset(AbstractHistogram histogram);
278
}
279
```
280

281
### Usage Examples
282

283
```java
284
// Iterate through all recorded values
285
RecordedValuesIterator iterator = new RecordedValuesIterator(histogram);
286

287
System.out.println("All Recorded Values:");
288
System.out.println("Value | Count | Cumulative Count | Percentile");
289
System.out.println("------|-------|------------------|------------");
290

291
while (iterator.hasNext()) {
292
    HistogramIterationValue iterationValue = iterator.next();
293
    
294
    long value = iterationValue.getValueIteratedTo();
295
    long count = iterationValue.getCountAtValueIteratedTo();
296
    long totalCount = iterationValue.getTotalCountToThisValue();
297
    double percentile = iterationValue.getPercentileIteratedTo();
298
    
299
    System.out.printf("%5d | %5d | %14d | %9.4f%%%n",
300
        value, count, totalCount, percentile);
301
        
302
    // Limit output for large histograms
303
    if (totalCount > histogram.getTotalCount() * 0.999) {
304
        System.out.println("... (showing first 99.9% of data)");
305
        break;
306
    }
307
}
308
```
309

310
### Exact Value Analysis
311

312
```java
313
public void findExactValues(AbstractHistogram histogram, long... searchValues) {
314
    RecordedValuesIterator iterator = new RecordedValuesIterator(histogram);
315
    Set<Long> searchSet = Arrays.stream(searchValues).boxed().collect(Collectors.toSet());
316
    
317
    System.out.println("Exact Value Search:");
318
    
319
    while (iterator.hasNext() && !searchSet.isEmpty()) {
320
        HistogramIterationValue value = iterator.next();
321
        long currentValue = value.getValueIteratedTo();
322
        
323
        if (searchSet.contains(currentValue)) {
324
            System.out.printf("Value %d: count=%d, percentile=%.4f%%%n",
325
                currentValue,
326
                value.getCountAtValueIteratedTo(), 
327
                value.getPercentileIteratedTo());
328
            searchSet.remove(currentValue);
329
        }
330
    }
331
    
332
    // Report missing values
333
    if (!searchSet.isEmpty()) {
334
        System.out.println("Values not found: " + searchSet);
335
    }
336
}
337
```
338

339
## AllValuesIterator
340

341
Iterates through all possible histogram values using finest granularity, including zero counts.
342

343
```java { .api }
344
public class AllValuesIterator extends AbstractHistogramIterator {
345
    
346
    // Constructor
347
    public AllValuesIterator(AbstractHistogram histogram);
348
    
349
    // Reset for reuse  
350
    void reset();
351
    void reset(AbstractHistogram histogram);
352
}
353
```
354

355
### Usage Examples
356

357
```java
358
// Analyze value coverage and gaps
359
AllValuesIterator iterator = new AllValuesIterator(histogram);
360

361
long consecutiveZeros = 0;
362
long maxGap = 0;
363
long gapStart = 0;
364

365
while (iterator.hasNext()) {
366
    HistogramIterationValue iterationValue = iterator.next();
367
    long value = iterationValue.getValueIteratedTo();
368
    long count = iterationValue.getCountAtValueIteratedTo();
369
    
370
    if (count == 0) {
371
        if (consecutiveZeros == 0) {
372
            gapStart = value;
373
        }
374
        consecutiveZeros++;
375
    } else {
376
        if (consecutiveZeros > maxGap) {
377
            maxGap = consecutiveZeros;
378
            System.out.printf("Largest gap: %d values (%d - %d)%n", 
379
                maxGap, gapStart, gapStart + maxGap - 1);
380
        }
381
        consecutiveZeros = 0;
382
    }
383
}
384
```
385

386
## HistogramIterationValue
387

388
Container class for histogram iteration results with comprehensive value information.
389

390
```java { .api }
391
public class HistogramIterationValue {
392
    
393
    // Value information
394
    long getValueIteratedTo();
395
    long getValueIteratedFrom();
396
    
397
    // Count information
398
    long getCountAtValueIteratedTo();
399
    long getCountAddedInThisIterationStep();
400
    long getTotalCountToThisValue();
401
    long getTotalValueToThisValue();
402
    
403
    // Percentile information
404
    double getPercentile();
405
    double getPercentileLevelIteratedTo();
406
    
407
    // Utility methods
408
    String toString();
409
}
410
```
411

412
### Comprehensive Value Analysis
413

414
```java
415
public void analyzeIterationValue(HistogramIterationValue value) {
416
    System.out.printf("Iteration Value Analysis:%n");
417
    System.out.printf("  Current value: %d%n", value.getValueIteratedTo());
418
    System.out.printf("  Value range: %d - %d%n", 
419
        value.getValueIteratedFrom(), value.getValueIteratedTo());
420
    System.out.printf("  Count at value: %d%n", value.getCountAtValueIteratedTo());
421
    System.out.printf("  Count in step: %d%n", value.getCountAddedInThisIterationStep());
422
    System.out.printf("  Cumulative count: %d%n", value.getTotalCountToThisValue());
423
    System.out.printf("  Cumulative value: %d%n", value.getTotalValueToThisValue());
424
    System.out.printf("  Percentile level: %.4f%%%n", value.getPercentileLevelIteratedTo());
425
}
426
```
427

428
## Double Histogram Iterators
429

430
HdrHistogram provides corresponding iterator types for DoubleHistogram values.
431

432
### DoublePercentileIterator
433

434
```java { .api }
435
public class DoublePercentileIterator implements Iterator<DoubleHistogramIterationValue> {
436
    
437
    public DoublePercentileIterator(DoubleHistogram histogram, int percentileTicksPerHalfDistance);
438
    
439
    boolean hasNext();
440
    DoubleHistogramIterationValue next();
441
    void reset(int percentileTicksPerHalfDistance);
442
}
443
```
444

445
### DoubleLinearIterator
446

447
```java { .api }
448
public class DoubleLinearIterator implements Iterator<DoubleHistogramIterationValue> {
449
    
450
    public DoubleLinearIterator(DoubleHistogram histogram, double valueUnitsPerBucket);
451
    
452
    boolean hasNext();
453
    DoubleHistogramIterationValue next();
454
    void reset(double valueUnitsPerBucket);
455
}
456
```
457

458
### Usage Examples for Double Iterators
459

460
```java
461
// Analyze double histogram with percentile iterator
462
DoubleHistogram responseTimeHist = getResponseTimeHistogram();
463
DoublePercentileIterator iterator = new DoublePercentileIterator(responseTimeHist, 5);
464

465
System.out.println("Response Time Percentile Analysis:");
466
while (iterator.hasNext()) {
467
    DoubleHistogramIterationValue value = iterator.next();
468
    
469
    double responseTime = value.getValueIteratedTo();
470
    double percentile = value.getPercentileLevelIteratedTo();
471
    
472
    if (percentile >= 50.0) {  // Focus on P50+
473
        System.out.printf("P%.1f: %.3f seconds%n", percentile, responseTime);
474
    }
475
}
476
```
477

478
### DoubleHistogramIterationValue
479

480
```java { .api }
481
public class DoubleHistogramIterationValue {
482
    
483
    double getValueIteratedTo();
484
    double getValueIteratedFrom();
485
    long getCountAtValueIteratedTo();
486
    long getCountAddedInThisIterationStep();
487
    long getTotalCountToThisValue();
488
    double getPercentileLevelIteratedTo();
489
}
490
```
491

492
## Advanced Iterator Patterns
493

494
### Multi-Iterator Comparison
495

496
```java
497
public void compareIteratorTypes(AbstractHistogram histogram) {
498
    // Setup different iterator types
499
    PercentileIterator percentile = new PercentileIterator(histogram, 5);
500
    LinearIterator linear = new LinearIterator(histogram, 1000);
501
    LogarithmicIterator logarithmic = new LogarithmicIterator(histogram, 100, 2.0);
502
    
503
    System.out.println("Iterator Type Comparison:");
504
    
505
    // Count steps for each iterator type
506
    int percentileSteps = countSteps(percentile);
507
    int linearSteps = countSteps(linear);
508
    int logSteps = countSteps(logarithmic);
509
    
510
    System.out.printf("Percentile iterator: %d steps%n", percentileSteps);
511
    System.out.printf("Linear iterator: %d steps%n", linearSteps);
512
    System.out.printf("Logarithmic iterator: %d steps%n", logSteps);
513
}
514

515
private int countSteps(Iterator<HistogramIterationValue> iterator) {
516
    int steps = 0;
517
    while (iterator.hasNext()) {
518
        iterator.next();
519
        steps++;
520
    }
521
    return steps;
522
}
523
```
524

525
### Custom Analysis Functions
526

527
```java
528
public class HistogramAnalyzer {
529
    
530
    public void findOutliers(AbstractHistogram histogram, double outlierThreshold) {
531
        PercentileIterator iterator = new PercentileIterator(histogram, 10);
532
        
533
        System.out.printf("Outlier Analysis (threshold: %.1f%%):%n", outlierThreshold);
534
        
535
        while (iterator.hasNext()) {
536
            HistogramIterationValue value = iterator.next();
537
            double percentile = value.getPercentileIteratedTo();
538
            
539
            if (percentile >= outlierThreshold) {
540
                long latency = value.getValueIteratedTo();
541
                long count = value.getCountAddedInThisIterationStep();
542
                
543
                System.out.printf("P%.4f: %d μs (%d samples)%n", 
544
                    percentile, latency, count);
545
            }
546
        }
547
    }
548
    
549
    public void analyzeDistributionShape(AbstractHistogram histogram) {
550
        LinearIterator iterator = new LinearIterator(histogram, 500);  // 500μs buckets
551
        
552
        List<Long> bucketCounts = new ArrayList<>();
553
        
554
        while (iterator.hasNext()) {
555
            HistogramIterationValue value = iterator.next();
556
            bucketCounts.add(value.getCountAddedInThisIterationStep());
557
        }
558
        
559
        // Calculate distribution metrics
560
        long maxBucket = Collections.max(bucketCounts);
561
        double mean = bucketCounts.stream().mapToLong(Long::longValue).average().orElse(0);
562
        
563
        System.out.printf("Distribution shape analysis:%n");
564
        System.out.printf("  Buckets: %d%n", bucketCounts.size());
565
        System.out.printf("  Peak bucket count: %d%n", maxBucket);
566
        System.out.printf("  Mean bucket count: %.1f%n", mean);
567
        System.out.printf("  Peak-to-mean ratio: %.2f%n", maxBucket / mean);
568
    }
569
}
570
```
571

572
### Iterator Performance Optimization
573

574
```java
575
public class OptimizedIteratorUsage {
576
    
577
    // Reuse iterators to minimize object creation
578
    private final PercentileIterator percentileIter;
579
    private final LinearIterator linearIter;
580
    
581
    public OptimizedIteratorUsage(AbstractHistogram templateHistogram) {
582
        this.percentileIter = new PercentileIterator(templateHistogram, 5);
583
        this.linearIter = new LinearIterator(templateHistogram, 1000);
584
    }
585
    
586
    public void analyzeManyHistograms(List<AbstractHistogram> histograms) {
587
        for (AbstractHistogram histogram : histograms) {
588
            // Reset and reuse existing iterators
589
            percentileIter.reset(histogram, 5);
590
            linearIter.reset(histogram, 1000);
591
            
592
            analyzeWithPercentiles(percentileIter);
593
            analyzeWithLinearBuckets(linearIter);
594
        }
595
    }
596
    
597
    private void analyzeWithPercentiles(PercentileIterator iterator) {
598
        // Analysis implementation using reused iterator
599
        while (iterator.hasNext()) {
600
            HistogramIterationValue value = iterator.next();
601
            // Process value...
602
        }
603
    }
604
    
605
    private void analyzeWithLinearBuckets(LinearIterator iterator) {
606
        // Analysis implementation using reused iterator  
607
        while (iterator.hasNext()) {
608
            HistogramIterationValue value = iterator.next();
609
            // Process value...
610
        }
611
    }
612
}
613
```
614

615
## Choosing the Right Iterator
616

617
### Iterator Selection Guide
618

619
| Analysis Need | Recommended Iterator | Configuration |
620
|---------------|---------------------|---------------|
621
| SLA compliance | PercentileIterator | percentileTicksPerHalfDistance = 5-10 |
622
| Distribution shape | LinearIterator | valueUnitsPerBucket = expected_range/100 |
623
| Scale analysis | LogarithmicIterator | logBase = 2.0 or 10.0 |
624
| Exact value lookup | RecordedValuesIterator | N/A |
625
| Gap analysis | AllValuesIterator | N/A (use sparingly) |
626
| Response time analysis | DoublePercentileIterator | For double histograms |
627
628
### Performance Characteristics
629

630
| Iterator Type | Performance | Memory | Use Case |
631
|--------------|-------------|---------|----------|
632
| PercentileIterator | Fast | Low | High-level percentile analysis |
633
| LinearIterator | Fast | Low | Uniform bucket analysis |
634
| LogarithmicIterator | Fast | Low | Scale-aware analysis |
635
| RecordedValuesIterator | Medium | Low | Detailed value exploration |
636
| AllValuesIterator | Slow | Medium | Complete coverage analysis |
637
638
### Memory and Performance Tips
639

640
- **Reuse iterators** when analyzing multiple histograms
641
- **Choose appropriate granularity** to balance detail vs. performance
642
- **Limit iteration scope** for large histograms (break early when possible)
643
- **Use percentile iterators** for most analysis scenarios
644
- **Avoid AllValuesIterator** for large value ranges unless necessary