0
# Multi-Input/Output Support
1

2
Advanced multi-modal data processing for complex neural network architectures with multiple inputs and outputs. The RecordReaderMultiDataSetIterator uses a flexible builder pattern to configure multiple data sources and map them to different network inputs and outputs.
3

4
## Capabilities  
5

6
### RecordReaderMultiDataSetIterator
7

8
Main class for creating MultiDataSet objects from multiple RecordReaders and SequenceRecordReaders.
9

10
```java { .api }
11
public class RecordReaderMultiDataSetIterator implements MultiDataSetIterator {
12
    // Iterator methods
13
    public boolean hasNext();
14
    public MultiDataSet next();
15
    public MultiDataSet next(int num);
16
    public void remove();
17
    
18
    // Configuration methods
19
    public void setPreProcessor(MultiDataSetPreProcessor preProcessor);
20
    public MultiDataSetPreProcessor getPreProcessor();
21
    public void setCollectMetaData(boolean collectMetaData);
22
    public boolean getCollectMetaData();
23
    
24
    // Reset and async support
25
    public boolean resetSupported();
26
    public boolean asyncSupported();
27
    public void reset();
28
    
29
    // Metadata support
30
    public MultiDataSet loadFromMetaData(RecordMetaData recordMetaData) throws IOException;
31
    public MultiDataSet loadFromMetaData(List<RecordMetaData> recordMetaDatas) throws IOException;
32
}
33
```
34

35
### Builder Class
36

37
Fluent API for configuring multi-input/output data processing.
38

39
```java { .api }
40
public static class Builder {
41
    public Builder(int batchSize);
42
    
43
    // Add data readers
44
    public Builder addReader(String readerName, RecordReader recordReader);
45
    public Builder addSequenceReader(String seqReaderName, SequenceRecordReader seqRecordReader);
46
    
47
    // Configure sequence alignment
48
    public Builder sequenceAlignmentMode(AlignmentMode alignmentMode);
49
    
50
    // Add inputs (features)
51
    public Builder addInput(String readerName);
52
    public Builder addInput(String readerName, int columnFirst, int columnLast);
53
    public Builder addInputOneHot(String readerName, int column, int numClasses);
54
    
55
    // Add outputs (labels)
56
    public Builder addOutput(String readerName);
57
    public Builder addOutput(String readerName, int columnFirst, int columnLast);
58
    public Builder addOutputOneHot(String readerName, int column, int numClasses);
59
    
60
    // Time series configuration
61
    public Builder timeSeriesRandomOffset(boolean timeSeriesRandomOffset, long rngSeed);
62
    
63
    // Build the iterator
64
    public RecordReaderMultiDataSetIterator build();
65
}
66
```
67

68
### AlignmentMode Enum
69

70
```java { .api }
71
public enum AlignmentMode {
72
    EQUAL_LENGTH,    // All sequences must be the same length
73
    ALIGN_START,     // Align sequences at start, pad end
74
    ALIGN_END        // Align sequences at end, pad start
75
}
76
```
77

78
## Usage Examples
79

80
### Basic Multi-Input Example
81

82
```java
83
import org.datavec.api.records.reader.impl.csv.CSVRecordReader;
84
import org.datavec.api.split.FileSplit;
85
import org.deeplearning4j.datasets.datavec.RecordReaderMultiDataSetIterator;
86

87
// Setup multiple record readers
88
RecordReader featuresReader = new CSVRecordReader();
89
featuresReader.initialize(new FileSplit(new File("features.csv")));
90

91
RecordReader labelsReader = new CSVRecordReader();
92
labelsReader.initialize(new FileSplit(new File("labels.csv")));
93

94
// Build multi-input iterator
95
RecordReaderMultiDataSetIterator iterator = 
96
    new RecordReaderMultiDataSetIterator.Builder(32) // batch size
97
        .addReader("features", featuresReader)
98
        .addReader("labels", labelsReader)
99
        .addInput("features")           // All columns from features as input
100
        .addOutput("labels")            // All columns from labels as output
101
        .build();
102

103
// Use iterator
104
while (iterator.hasNext()) {
105
    MultiDataSet multiDataSet = iterator.next();
106
    INDArray[] inputs = multiDataSet.getFeatures();    // Array of input arrays
107
    INDArray[] outputs = multiDataSet.getLabels();     // Array of output arrays
108
}
109
```
110

111
### Multiple Inputs and Outputs
112

113
```java
114
// Setup readers for different data modalities
115
RecordReader imageReader = new ImageRecordReader();
116
RecordReader textReader = new CSVRecordReader();
117
RecordReader metadataReader = new CSVRecordReader();
118

119
// Initialize readers
120
imageReader.initialize(new FileSplit(new File("images/")));
121
textReader.initialize(new FileSplit(new File("text_features.csv")));
122
metadataReader.initialize(new FileSplit(new File("metadata.csv")));
123

124
// Configure multi-modal architecture
125
RecordReaderMultiDataSetIterator iterator = 
126
    new RecordReaderMultiDataSetIterator.Builder(16)
127
        // Add all readers
128
        .addReader("images", imageReader)
129
        .addReader("text", textReader)
130
        .addReader("metadata", metadataReader)
131
        
132
        // Configure multiple inputs
133
        .addInput("images")                    // Input 0: Image data
134
        .addInput("text", 0, 99)              // Input 1: Text features (columns 0-99)
135
        .addInput("metadata", 0, 9)           // Input 2: Metadata (columns 0-9)
136
        
137
        // Configure multiple outputs
138
        .addOutputOneHot("metadata", 10, 5)   // Output 0: Classification (column 10, 5 classes)
139
        .addOutput("metadata", 11, 13)        // Output 1: Regression (columns 11-13)
140
        .build();
141

142
// Process multi-modal data
143
while (iterator.hasNext()) {
144
    MultiDataSet batch = iterator.next();
145
    
146
    // Access multiple inputs
147
    INDArray imageInput = batch.getFeatures(0);      // Images
148
    INDArray textInput = batch.getFeatures(1);       // Text features  
149
    INDArray metadataInput = batch.getFeatures(2);   // Metadata
150
    
151
    // Access multiple outputs
152
    INDArray classificationOutput = batch.getLabels(0);  // One-hot classification
153
    INDArray regressionOutput = batch.getLabels(1);      // Regression targets
154
}
155
```
156

157
### Sequence Multi-Input Example
158

159
```java
160
import org.datavec.api.records.reader.impl.csv.CSVSequenceRecordReader;
161

162
// Setup sequence readers
163
SequenceRecordReader audioReader = new CSVSequenceRecordReader();
164
SequenceRecordReader videoReader = new CSVSequenceRecordReader();
165
SequenceRecordReader labelReader = new CSVSequenceRecordReader();
166

167
// Initialize sequence readers
168
audioReader.initialize(new NumberedFileInputSplit("audio_%d.csv", 0, 999));
169
videoReader.initialize(new NumberedFileInputSplit("video_%d.csv", 0, 999));
170
labelReader.initialize(new NumberedFileInputSplit("labels_%d.csv", 0, 999));
171

172
// Configure multi-modal sequence processing
173
RecordReaderMultiDataSetIterator iterator = 
174
    new RecordReaderMultiDataSetIterator.Builder(8)
175
        // Add sequence readers
176
        .addSequenceReader("audio", audioReader)
177
        .addSequenceReader("video", videoReader)  
178
        .addSequenceReader("labels", labelReader)
179
        
180
        // Configure sequence alignment
181
        .sequenceAlignmentMode(AlignmentMode.ALIGN_START)
182
        
183
        // Add sequence inputs
184
        .addInput("audio")        // Audio features
185
        .addInput("video")        // Video features
186
        
187
        // Add sequence output
188
        .addOutputOneHot("labels", 0, 10)  // 10-class classification per time step
189
        
190
        .build();
191
```
192

193
### Column Range Selection
194

195
```java
196
// Fine-grained control over which columns to use
197
RecordReader dataReader = new CSVRecordReader();
198
dataReader.initialize(new FileSplit(new File("complex_data.csv")));
199

200
RecordReaderMultiDataSetIterator iterator = 
201
    new RecordReaderMultiDataSetIterator.Builder(64)
202
        .addReader("data", dataReader)
203
        
204
        // Use specific column ranges for different inputs
205
        .addInput("data", 0, 49)      // Input 0: Columns 0-49 (features)
206
        .addInput("data", 50, 99)     // Input 1: Columns 50-99 (different feature type)
207
        
208
        // Use specific columns for outputs
209
        .addOutputOneHot("data", 100, 3)  // Output 0: Column 100, 3 classes
210
        .addOutput("data", 101, 105)      // Output 1: Columns 101-105 (regression)
211
        
212
        .build();
213
```
214

215
### Time Series Random Offset
216

217
```java
218
// Add random time offset for data augmentation
219
RecordReaderMultiDataSetIterator iterator = 
220
    new RecordReaderMultiDataSetIterator.Builder(32)
221
        .addSequenceReader("sequences", sequenceReader)
222
        .addInput("sequences")
223
        .addOutput("sequences", 10, 12)
224
        
225
        // Enable random time series offset
226
        .timeSeriesRandomOffset(true, 12345L)  // enabled, with seed
227
        
228
        .build();
229
```
230

231
### Preprocessing Multi-Modal Data
232

233
```java
234
import org.nd4j.linalg.dataset.api.preprocessor.MultiDataSetPreProcessor;
235
import org.nd4j.linalg.dataset.api.preprocessor.MultiNormalizerMinMaxScaler;
236

237
// Create iterator
238
RecordReaderMultiDataSetIterator iterator = 
239
    new RecordReaderMultiDataSetIterator.Builder(32)
240
        .addReader("input1", reader1)
241
        .addReader("input2", reader2)
242
        .addInput("input1")
243
        .addInput("input2")
244
        .addOutput("input1", 10, 12)
245
        .build();
246

247
// Add multi-input preprocessing
248
MultiNormalizerMinMaxScaler scaler = new MultiNormalizerMinMaxScaler();
249
iterator.setPreProcessor(scaler);
250

251
// Fit preprocessor
252
scaler.fit(iterator);
253
iterator.reset();
254

255
// Use preprocessed data
256
while (iterator.hasNext()) {
257
    MultiDataSet preprocessed = iterator.next();
258
    // All inputs normalized independently
259
}
260
```
261

262
## Advanced Configuration
263

264
### Builder Method Details
265

266
#### Reader Management
267
- `addReader(name, reader)`: Add a RecordReader with a string identifier
268
- `addSequenceReader(name, reader)`: Add a SequenceRecordReader with identifier
269

270
#### Input Configuration  
271
- `addInput(readerName)`: Use all columns from reader as input
272
- `addInput(readerName, start, end)`: Use columns start to end (inclusive) as input
273
- `addInputOneHot(readerName, column, numClasses)`: Convert column to one-hot encoding
274

275
#### Output Configuration
276
- `addOutput(readerName)`: Use all columns from reader as output
277
- `addOutput(readerName, start, end)`: Use columns start to end (inclusive) as output  
278
- `addOutputOneHot(readerName, column, numClasses)`: Convert column to one-hot encoding
279

280
## Error Handling
281

282
### Common Exceptions
283
- **IllegalArgumentException**: Invalid builder configuration
284
- **IllegalStateException**: Builder used incorrectly (e.g., no inputs defined)
285
- **IOException**: RecordReader I/O errors
286

287
### Validation Rules
288
- At least one input must be defined
289
- At least one output must be defined  
290
- Reader names must be unique
291
- Column indices must be valid for their respective readers
292
- Batch size must be positive
293
- Sequence readers require consistent sequence alignment mode