tessl/maven-org-datavec--datavec-local

DataVec integration library providing data loading, transformation, and Spark processing capabilities for DeepLearning4j

—

Pending

Overview

Eval results

Files

Sequence Processing

Name: tessl/maven-org-datavec--datavec-local
Author: tessl

Time series and sequential data processing capabilities for DeepLearning4j. The SequenceRecordReaderDataSetIterator handles variable-length sequences with configurable alignment modes, making it ideal for RNN, LSTM, and other sequence-based neural network architectures.

Capabilities

SequenceRecordReaderDataSetIterator

Main class for converting SequenceRecordReader data into DataSet objects for sequence-based neural network training.

public class SequenceRecordReaderDataSetIterator implements DataSetIterator {
    // Single reader constructors (features and labels from same reader)
    public SequenceRecordReaderDataSetIterator(SequenceRecordReader reader, 
                                              int miniBatchSize, 
                                              int numPossibleLabels, 
                                              int labelIndex);
    public SequenceRecordReaderDataSetIterator(SequenceRecordReader reader, 
                                              int miniBatchSize, 
                                              int numPossibleLabels, 
                                              int labelIndex, 
                                              boolean regression);
    
    // Separate readers constructors (separate readers for features and labels)
    public SequenceRecordReaderDataSetIterator(SequenceRecordReader featuresReader, 
                                              SequenceRecordReader labelsReader, 
                                              int miniBatchSize, 
                                              int numPossibleLabels);
    public SequenceRecordReaderDataSetIterator(SequenceRecordReader featuresReader, 
                                              SequenceRecordReader labelsReader, 
                                              int miniBatchSize, 
                                              int numPossibleLabels, 
                                              boolean regression);
    public SequenceRecordReaderDataSetIterator(SequenceRecordReader featuresReader, 
                                              SequenceRecordReader labelsReader, 
                                              int miniBatchSize, 
                                              int numPossibleLabels, 
                                              boolean regression, 
                                              AlignmentMode alignmentMode);
    
    // Iterator methods
    public boolean hasNext();
    public DataSet next();
    public DataSet next(int num);
    public void remove();
    
    // Configuration methods
    public void setPreProcessor(DataSetPreProcessor preProcessor);
    public DataSetPreProcessor getPreProcessor();
    public void setCollectMetaData(boolean collectMetaData);
    public boolean getCollectMetaData();
    
    // Information methods
    public int totalExamples();
    public int inputColumns();
    public int totalOutcomes();
    public int batch();
    public int cursor();
    public int numExamples();
    public List<String> getLabels();
    
    // Reset and async support
    public boolean resetSupported();
    public boolean asyncSupported();
    public void reset();
    
    // Metadata support
    public DataSet loadFromMetaData(RecordMetaData recordMetaData) throws IOException;
    public DataSet loadFromMetaData(List<RecordMetaData> recordMetaDatas) throws IOException;
}

AlignmentMode Enum

Defines how sequences of different lengths are aligned within a batch.

public enum AlignmentMode {
    EQUAL_LENGTH,    // All sequences must be the same length (throws exception if not)
    ALIGN_START,     // Align sequences at the start, pad shorter sequences at the end
    ALIGN_END        // Align sequences at the end, pad shorter sequences at the start
}

Constructor Parameters

Single Reader Parameters

reader: SequenceRecordReader containing both features and labels
miniBatchSize: Number of sequences per batch
numPossibleLabels: Number of possible label classes
labelIndex: Column index containing the label within each time step
regression: true for regression tasks, false for classification

Separate Readers Parameters

featuresReader: SequenceRecordReader for input features
labelsReader: SequenceRecordReader for target labels
miniBatchSize: Number of sequences per batch
numPossibleLabels: Number of possible label classes
regression: true for regression tasks, false for classification
alignmentMode: How to handle sequences of different lengths

Usage Examples

Basic Time Series Classification

import org.datavec.api.records.reader.SequenceRecordReader;
import org.datavec.api.records.reader.impl.csv.CSVSequenceRecordReader;
import org.datavec.api.split.NumberedFileInputSplit;
import org.deeplearning4j.datasets.datavec.SequenceRecordReaderDataSetIterator;

// Setup sequence readers
SequenceRecordReader featuresReader = new CSVSequenceRecordReader();
SequenceRecordReader labelsReader = new CSVSequenceRecordReader();

// Initialize with file splits (one file per sequence)
featuresReader.initialize(new NumberedFileInputSplit("features_%d.csv", 0, 999));
labelsReader.initialize(new NumberedFileInputSplit("labels_%d.csv", 0, 999));

// Create sequence iterator
SequenceRecordReaderDataSetIterator iterator = 
    new SequenceRecordReaderDataSetIterator(
        featuresReader,         // features reader
        labelsReader,          // labels reader
        32,                    // miniBatchSize
        10,                    // numPossibleLabels (10 classes)
        false,                 // regression = false (classification)
        AlignmentMode.ALIGN_START  // alignment mode
    );

// Use iterator for training
while (iterator.hasNext()) {
    DataSet sequenceData = iterator.next();
    // Features shape: [batchSize, numFeatures, maxSequenceLength]
    // Labels shape: [batchSize, numLabels, maxSequenceLength]
    System.out.println("Features shape: " + Arrays.toString(sequenceData.getFeatures().shape()));
}

Single Reader Example

// When features and labels are in the same sequence file
SequenceRecordReader reader = new CSVSequenceRecordReader();
reader.initialize(new NumberedFileInputSplit("data_%d.csv", 0, 99));

// Last column (index 5) contains labels
SequenceRecordReaderDataSetIterator iterator = 
    new SequenceRecordReaderDataSetIterator(
        reader,     // single reader
        16,         // miniBatchSize
        3,          // numPossibleLabels
        5           // labelIndex (last column)
    );

Regression Example

// Time series regression (e.g., stock price prediction)
SequenceRecordReaderDataSetIterator regressionIterator = 
    new SequenceRecordReaderDataSetIterator(
        featuresReader,         // features reader
        labelsReader,          // labels reader (continuous values)
        64,                    // miniBatchSize
        1,                     // numPossibleLabels (1 for regression)
        true,                  // regression = true
        AlignmentMode.EQUAL_LENGTH  // all sequences must be same length
    );

Different Alignment Modes

// ALIGN_START: Pad shorter sequences at the end with zeros
SequenceRecordReaderDataSetIterator alignStartIterator = 
    new SequenceRecordReaderDataSetIterator(
        featuresReader, labelsReader, 32, 5, false, 
        AlignmentMode.ALIGN_START);

// ALIGN_END: Pad shorter sequences at the beginning with zeros  
SequenceRecordReaderDataSetIterator alignEndIterator = 
    new SequenceRecordReaderDataSetIterator(
        featuresReader, labelsReader, 32, 5, false, 
        AlignmentMode.ALIGN_END);

// EQUAL_LENGTH: Throw exception if sequences have different lengths
SequenceRecordReaderDataSetIterator equalLengthIterator = 
    new SequenceRecordReaderDataSetIterator(
        featuresReader, labelsReader, 32, 5, false, 
        AlignmentMode.EQUAL_LENGTH);

Sequence Preprocessing

import org.nd4j.linalg.dataset.api.preprocessor.NormalizerMinMaxScaler;

// Create iterator
SequenceRecordReaderDataSetIterator iterator = 
    new SequenceRecordReaderDataSetIterator(featuresReader, labelsReader, 32, 5);

// Add sequence-aware preprocessing  
NormalizerMinMaxScaler scaler = new NormalizerMinMaxScaler();
iterator.setPreProcessor(scaler);

// Fit normalizer on sequence data
scaler.fit(iterator);
iterator.reset();

// Use normalized sequence data
while (iterator.hasNext()) {
    DataSet normalizedSequences = iterator.next();
    // Each sequence is normalized independently
}

Sequence Data Format

Input Data Shape

Features: [batchSize, numFeatures, maxSequenceLength]
Labels: [batchSize, numLabels, maxSequenceLength]
Feature Mask: [batchSize, maxSequenceLength] (indicates valid time steps)
Label Mask: [batchSize, maxSequenceLength] (indicates valid labels)

Masking

The iterator automatically creates masks for variable-length sequences:

Feature masks mark valid input time steps (1.0 = valid, 0.0 = padding)
Label masks mark valid output time steps for training
Masked time steps are ignored during training and evaluation

Error Handling

Common Exceptions

ZeroLengthSequenceException: Thrown when a sequence has zero length
IllegalArgumentException: Invalid constructor parameters
IOException: File reading errors from SequenceRecordReader

Validation

miniBatchSize must be positive
numPossibleLabels must be positive
labelIndex must be valid for single-reader mode
Sequence readers must be properly initialized
EQUAL_LENGTH mode requires all sequences to have identical length

Install with Tessl CLI

npx tessl i tessl/maven-org-datavec--datavec-local

docs

dataset-iteration.md

index.md

multi-input-output.md

sequence-processing.md

spark-integration.md

tile.json

tessl/maven-org-datavec--datavec-local