0
# Layers and Components
1

2
Reusable neural network layers and components for building custom models or extending existing architectures. Keras Hub provides both general-purpose layers and specialized components for transformer models.
3

4
## Capabilities
5

6
### Transformer Layers
7

8
Core transformer architecture components for building attention-based models.
9

10
```python { .api }
11
class TransformerEncoder:
12
    """Transformer encoder block with multi-head attention and feed-forward network."""
13
    def __init__(
14
        self,
15
        intermediate_dim: int,
16
        num_heads: int,
17
        dropout: float = 0.0,
18
        activation: str = "relu",
19
        layer_norm_epsilon: float = 1e-05,
20
        kernel_initializer: str = "glorot_uniform",
21
        bias_initializer: str = "zeros",
22
        **kwargs
23
    ): ...
24

25
class TransformerDecoder:
26
    """Transformer decoder block with masked multi-head attention."""
27
    def __init__(
28
        self,
29
        intermediate_dim: int,
30
        num_heads: int,
31
        dropout: float = 0.0,
32
        activation: str = "relu",
33
        layer_norm_epsilon: float = 1e-05,
34
        kernel_initializer: str = "glorot_uniform",
35
        bias_initializer: str = "zeros",
36
        **kwargs
37
    ): ...
38

39
class CachedMultiHeadAttention:
40
    """Multi-head attention with key/value caching for efficient generation."""
41
    def __init__(
42
        self,
43
        num_heads: int,
44
        key_dim: int,
45
        value_dim: int = None,
46
        dropout: float = 0.0,
47
        use_bias: bool = True,
48
        kernel_initializer: str = "glorot_uniform",
49
        bias_initializer: str = "zeros",
50
        **kwargs
51
    ): ...
52
```
53

54
### Embedding Layers
55

56
Various embedding strategies for representing tokens and positions.
57

58
```python { .api }
59
class PositionEmbedding:
60
    """Learnable position embeddings."""
61
    def __init__(
62
        self,
63
        sequence_length: int,
64
        initializer: str = "glorot_uniform",
65
        **kwargs
66
    ): ...
67

68
class TokenAndPositionEmbedding:
69
    """Combined token and position embeddings."""
70
    def __init__(
71
        self,
72
        vocabulary_size: int,
73
        sequence_length: int,
74
        embedding_dim: int,
75
        mask_zero: bool = False,
76
        **kwargs
77
    ): ...
78

79
class ReversibleEmbedding:
80
    """Reversible token embeddings that can map back from embeddings to tokens."""
81
    def __init__(
82
        self,
83
        input_dim: int,
84
        output_dim: int,
85
        tie_weights: bool = True,
86
        embeddings_initializer: str = "uniform",
87
        **kwargs
88
    ): ...
89

90
class RotaryEmbedding:
91
    """Rotary position embeddings (RoPE)."""
92
    def __init__(
93
        self,
94
        max_wavelength: int = 10000,
95
        scaling_factor: float = 1.0,
96
        **kwargs
97
    ): ...
98

99
class SinePositionEncoding:
100
    """Sinusoidal position encoding as used in original Transformer."""
101
    def __init__(
102
        self,
103
        max_wavelength: int = 10000,
104
        **kwargs
105
    ): ...
106

107
class AlibiBias:
108
    """Attention bias for ALiBi (Attention with Linear Biases) positional encoding."""
109
    def __init__(
110
        self,
111
        num_heads: int,
112
        **kwargs
113
    ): ...
114
```
115

116
### Normalization Layers
117

118
Normalization techniques used in modern transformer architectures.
119

120
```python { .api }
121
class RMSNormalization:
122
    """Root Mean Square (RMS) normalization layer."""
123
    def __init__(
124
        self,
125
        epsilon: float = 1e-06,
126
        **kwargs
127
    ): ...
128
```
129

130
### Task-Specific Heads
131

132
Specialized output layers for different tasks.
133

134
```python { .api }
135
class MaskedLMHead:
136
    """Output head for masked language modeling tasks."""
137
    def __init__(
138
        self,
139
        vocabulary_size: int,
140
        token_embedding: ReversibleEmbedding = None,
141
        activation: str = None,
142
        **kwargs
143
    ): ...
144
```
145

146
### Preprocessing Layers
147

148
Layers for data preprocessing and augmentation.
149

150
```python { .api }
151
class ImageConverter:
152
    """Base class for image data conversion and preprocessing."""
153
    def __init__(
154
        self,
155
        height: int,
156
        width: int,
157
        crop_to_aspect_ratio: bool = True,
158
        interpolation: str = "bilinear",
159
        data_format: str = None,
160
        **kwargs
161
    ): ...
162

163
class AudioConverter:
164
    """Base class for audio data conversion and preprocessing."""
165
    def __init__(
166
        self,
167
        sample_rate: int = 16000,
168
        **kwargs
169
    ): ...
170

171
class MaskedLMMaskGenerator:
172
    """Generate masks for masked language modeling tasks."""
173
    def __init__(
174
        self,
175
        mask_selection_rate: float = 0.15,
176
        mask_token_rate: float = 0.8,
177
        random_token_rate: float = 0.1,
178
        mask_token_id: int = None,
179
        vocabulary_size: int = None,
180
        **kwargs
181
    ): ...
182

183
class MultiSegmentPacker:
184
    """Pack multiple text segments with appropriate separators."""
185
    def __init__(
186
        self,
187
        start_value: int,
188
        end_value: int,
189
        sep_value: int = None,
190
        pad_value: int = 0,
191
        sequence_length: int = None,
192
        **kwargs
193
    ): ...
194

195
class StartEndPacker:
196
    """Pack text sequences with start and end tokens."""
197
    def __init__(
198
        self,
199
        start_value: int,
200
        end_value: int,
201
        pad_value: int = 0,
202
        sequence_length: int = None,
203
        return_padding_mask: bool = False,
204
        **kwargs
205
    ): ...
206

207
class RandomDeletion:
208
    """Randomly delete tokens from sequences for data augmentation."""
209
    def __init__(
210
        self,
211
        rate: float = 0.1,
212
        max_deletions: int = None,
213
        skip_list: list = None,
214
        seed: int = None,
215
        **kwargs
216
    ): ...
217

218
class RandomSwap:
219
    """Randomly swap adjacent tokens for data augmentation."""
220
    def __init__(
221
        self,
222
        rate: float = 0.1,
223
        max_swaps: int = None,
224
        skip_list: list = None,
225
        seed: int = None,
226
        **kwargs
227
    ): ...
228
```
229

230
### Object Detection Layers
231

232
Specialized layers for object detection tasks.
233

234
```python { .api }
235
class AnchorGenerator:
236
    """Generate anchor boxes for object detection."""
237
    def __init__(
238
        self,
239
        bounding_box_format: str,
240
        min_level: int,
241
        max_level: int,
242
        num_scales: int,
243
        aspect_ratios: list,
244
        anchor_size: int,
245
        **kwargs
246
    ): ...
247

248
class BoxMatcher:
249
    """Match ground truth boxes to anchor boxes."""
250
    def __init__(
251
        self,
252
        thresholds: list,
253
        match_values: list,
254
        force_match_for_each_col: bool = False,
255
        **kwargs
256
    ): ...
257

258
class NonMaxSuppression:
259
    """Non-maximum suppression for object detection post-processing."""
260
    def __init__(
261
        self,
262
        bounding_box_format: str,
263
        from_logits: bool = False,
264
        iou_threshold: float = 0.5,
265
        confidence_threshold: float = 0.05,
266
        max_detections: int = 100,
267
        max_detections_per_class: int = 100,
268
        **kwargs
269
    ): ...
270
```
271

272
### Specialized Model Components
273

274
Components specific to certain model architectures.
275

276
```python { .api }
277
class FNetEncoder:
278
    """F-Net encoder using Fourier transforms instead of attention."""
279
    def __init__(
280
        self,
281
        intermediate_dim: int,
282
        activation: str = "relu",
283
        dropout: float = 0.0,
284
        layer_norm_epsilon: float = 1e-05,
285
        **kwargs
286
    ): ...
287

288
class SAMMaskDecoder:
289
    """Mask decoder for Segment Anything Model."""
290
    def __init__(
291
        self,
292
        num_multimask_outputs: int = 3,
293
        iou_head_depth: int = 3,
294
        iou_head_hidden_dim: int = 256,
295
        **kwargs
296
    ): ...
297

298
class SAMPromptEncoder:
299
    """Prompt encoder for Segment Anything Model."""
300
    def __init__(
301
        self,
302
        embed_dim: int = 256,
303
        image_embedding_size: tuple = (64, 64),
304
        input_image_size: tuple = (1024, 1024),
305
        mask_in_chans: int = 16,
306
        **kwargs
307
    ): ...
308
```
309

310
## Usage Examples
311

312
### Building Custom Transformer Model
313

314
```python
315
import keras_hub
316
import keras
317

318
# Create custom model using Keras Hub layers
319
def create_custom_transformer(
320
    vocab_size: int,
321
    sequence_length: int,
322
    num_layers: int = 6,
323
    hidden_dim: int = 512,
324
    num_heads: int = 8,
325
    intermediate_dim: int = 2048
326
):
327
    # Input
328
    token_ids = keras.Input(shape=(sequence_length,), dtype="int32")
329
    
330
    # Token and position embeddings
331
    embeddings = keras_hub.layers.TokenAndPositionEmbedding(
332
        vocabulary_size=vocab_size,
333
        sequence_length=sequence_length,
334
        embedding_dim=hidden_dim
335
    )(token_ids)
336
    
337
    # Stack transformer encoder layers
338
    x = embeddings
339
    for _ in range(num_layers):
340
        x = keras_hub.layers.TransformerEncoder(
341
            intermediate_dim=intermediate_dim,
342
            num_heads=num_heads,
343
            dropout=0.1
344
        )(x)
345
    
346
    # Output layer for classification
347
    outputs = keras.layers.Dense(2, activation="softmax")(x[:, 0, :])  # Use [CLS] token
348
    
349
    return keras.Model(token_ids, outputs)
350

351
# Create and compile model
352
model = create_custom_transformer(vocab_size=10000, sequence_length=128)
353
model.compile(optimizer="adam", loss="sparse_categorical_crossentropy")
354
```
355

356
### Using Preprocessing Layers
357

358
```python
359
import keras_hub
360

361
# Create mask generator for MLM training
362
mask_generator = keras_hub.layers.MaskedLMMaskGenerator(
363
    mask_selection_rate=0.15,
364
    mask_token_rate=0.8,
365
    random_token_rate=0.1,
366
    mask_token_id=103,  # [MASK] token ID
367
    vocabulary_size=30000
368
)
369

370
# Apply masking to token sequences
371
tokens = [[101, 2054, 2003, 2115, 2171, 102]]  # Example tokenized text
372
masked_tokens, mask_positions, mask_ids = mask_generator(tokens)
373
print("Masked tokens:", masked_tokens)
374
print("Mask positions:", mask_positions)
375
```
376

377
### Custom Position Embedding
378

379
```python
380
import keras_hub
381
import keras
382

383
# Use rotary embeddings in custom attention
384
class CustomAttentionWithRoPE(keras.layers.Layer):
385
    def __init__(self, num_heads, key_dim, **kwargs):
386
        super().__init__(**kwargs)
387
        self.num_heads = num_heads
388
        self.key_dim = key_dim
389
        self.rotary_embedding = keras_hub.layers.RotaryEmbedding()
390
        self.attention = keras.layers.MultiHeadAttention(
391
            num_heads=num_heads,
392
            key_dim=key_dim
393
        )
394
    
395
    def call(self, inputs):
396
        # Apply rotary embeddings
397
        rotary_embedded = self.rotary_embedding(inputs)
398
        # Apply attention
399
        return self.attention(rotary_embedded, rotary_embedded)
400

401
# Use in model
402
attention_layer = CustomAttentionWithRoPE(num_heads=8, key_dim=64)
403
```
404

405
### Data Augmentation with Text Layers
406

407
```python
408
import keras_hub
409

410
# Create augmentation pipeline
411
random_deletion = keras_hub.layers.RandomDeletion(rate=0.1)
412
random_swap = keras_hub.layers.RandomSwap(rate=0.05)
413

414
# Apply augmentations
415
original_tokens = [[1, 2, 3, 4, 5, 6, 7, 8]]
416
augmented_tokens = random_deletion(original_tokens)
417
augmented_tokens = random_swap(augmented_tokens)
418

419
print("Original:", original_tokens[0])
420
print("Augmented:", augmented_tokens[0])
421
```
422

423
### Sequence Packing
424

425
```python
426
import keras_hub
427

428
# Pack multiple segments with separators
429
packer = keras_hub.layers.MultiSegmentPacker(
430
    start_value=101,  # [CLS]
431
    end_value=102,    # [SEP]
432
    pad_value=0,      # [PAD]
433
    sequence_length=128
434
)
435

436
# Pack two text segments
437
segment1 = [2054, 2003, 2115, 2171]  # "what is your name"
438
segment2 = [2026, 2572, 2017, 2009]  # "my name is"
439

440
packed = packer([segment1, segment2])
441
print("Packed sequence:", packed)
442
```
443

444
### Object Detection Components
445

446
```python
447
import keras_hub
448

449
# Create anchor generator for object detection
450
anchor_generator = keras_hub.layers.AnchorGenerator(
451
    bounding_box_format="xyxy",
452
    min_level=3,
453
    max_level=7,
454
    num_scales=3,
455
    aspect_ratios=[0.5, 1.0, 2.0],
456
    anchor_size=4
457
)
458

459
# Generate anchors for feature maps
460
feature_map_shapes = [(32, 32), (16, 16), (8, 8)]
461
anchors = anchor_generator(feature_map_shapes)
462
print("Generated anchors shape:", anchors.shape)
463

464
# Apply non-max suppression
465
nms = keras_hub.layers.NonMaxSuppression(
466
    bounding_box_format="xyxy",
467
    iou_threshold=0.5,
468
    confidence_threshold=0.05,
469
    max_detections=100
470
)
471
```
472

473
### Normalization Layers
474

475
```python
476
import keras_hub
477
import keras
478

479
# Use RMS normalization in custom model
480
class CustomLayer(keras.layers.Layer):
481
    def __init__(self, **kwargs):
482
        super().__init__(**kwargs)
483
        self.dense = keras.layers.Dense(512)
484
        self.rms_norm = keras_hub.layers.RMSNormalization()
485
    
486
    def call(self, inputs):
487
        x = self.dense(inputs)
488
        return self.rms_norm(x)
489

490
# Use in model
491
layer = CustomLayer()
492
```