tessl/pypi-keras
Multi-backend deep learning framework that provides a unified, high-level API for building and training neural networks across JAX, TensorFlow, PyTorch, and OpenVINO backends.
—
Pending
regularizers.mddocs/
Regularizers
Regularizers apply penalties to layer parameters during training to reduce overfitting by constraining the complexity of the model. They add terms to the loss function that penalize large weights.
Capabilities
Base Regularizer
The abstract base class for all regularizers, providing the interface for regularization penalties.
class Regularizer:
"""
Base class for weight regularizers.
All regularizers should inherit from this class and implement the __call__ method.
"""
def __call__(self, x):
"""
Compute the regularization penalty.
Parameters:
- x: Weight tensor to regularize
Returns:
Scalar tensor representing the regularization penalty
"""L1 Regularization
L1 regularization adds a penalty term proportional to the sum of the absolute values of the weights, promoting sparsity.
class L1(Regularizer):
"""
L1 regularization penalty.
Adds a penalty term proportional to the sum of absolute values of weights.
Promotes sparsity by driving some weights to exactly zero.
Usage:
```python
layer = Dense(10, kernel_regularizer=L1(0.01))
# or
layer = Dense(10, kernel_regularizer='l1')
```
"""
def __init__(self, l1=0.01):
"""
Initialize the L1 regularizer.
Parameters:
- l1: L1 regularization factor (default: 0.01)
"""
def l1(l1=0.01):
"""
Create an L1 regularizer.
Parameters:
- l1: L1 regularization factor (default: 0.01)
Returns:
L1 regularizer instance
"""L2 Regularization
L2 regularization adds a penalty term proportional to the sum of squares of the weights, promoting small weights.
class L2(Regularizer):
"""
L2 regularization penalty.
Adds a penalty term proportional to the sum of squares of weights.
Promotes small weights and smooth solutions.
Usage:
```python
layer = Dense(10, kernel_regularizer=L2(0.01))
# or
layer = Dense(10, kernel_regularizer='l2')
```
"""
def __init__(self, l2=0.01):
"""
Initialize the L2 regularizer.
Parameters:
- l2: L2 regularization factor (default: 0.01)
"""
def l2(l2=0.01):
"""
Create an L2 regularizer.
Parameters:
- l2: L2 regularization factor (default: 0.01)
Returns:
L2 regularizer instance
"""Combined L1L2 Regularization
Combines both L1 and L2 regularization penalties, providing benefits of both sparsity and weight decay.
class L1L2(Regularizer):
"""
Combined L1 and L2 regularization penalty.
Combines both L1 and L2 penalties, providing both sparsity (L1) and
weight decay (L2) effects.
Usage:
```python
layer = Dense(10, kernel_regularizer=L1L2(l1=0.01, l2=0.01))
# or
layer = Dense(10, kernel_regularizer='l1_l2')
```
"""
def __init__(self, l1=0.0, l2=0.0):
"""
Initialize the L1L2 regularizer.
Parameters:
- l1: L1 regularization factor (default: 0.0)
- l2: L2 regularization factor (default: 0.0)
"""
def l1_l2(l1=0.01, l2=0.01):
"""
Create a combined L1L2 regularizer.
Parameters:
- l1: L1 regularization factor (default: 0.01)
- l2: L2 regularization factor (default: 0.01)
Returns:
L1L2 regularizer instance
"""Orthogonal Regularization
Encourages weight matrices to be orthogonal, which can help with gradient flow and representational diversity.
class OrthogonalRegularizer(Regularizer):
"""
Orthogonal regularization penalty.
Encourages weight matrices to be orthogonal by penalizing the deviation
from orthogonality. Useful for maintaining diverse representations and
improving gradient flow.
Usage:
```python
layer = Dense(10, kernel_regularizer=OrthogonalRegularizer(factor=0.01))
```
"""
def __init__(self, factor=0.01, mode='rows'):
"""
Initialize the OrthogonalRegularizer.
Parameters:
- factor: Regularization strength (default: 0.01)
- mode: 'rows' or 'columns' - which dimension to orthogonalize (default: 'rows')
"""
def orthogonal_regularizer(factor=0.01, mode='rows'):
"""
Create an orthogonal regularizer.
Parameters:
- factor: Regularization strength (default: 0.01)
- mode: 'rows' or 'columns' - which dimension to orthogonalize (default: 'rows')
Returns:
OrthogonalRegularizer instance
"""Utility Functions
Helper functions for regularizer management and serialization.
def serialize(regularizer):
"""
Serialize a regularizer to a string or config dict.
Parameters:
- regularizer: Regularizer to serialize
Returns:
String identifier or config dictionary
"""
def deserialize(config, custom_objects=None):
"""
Deserialize a regularizer from a string or config dict.
Parameters:
- config: String identifier or config dictionary
- custom_objects: Optional dict mapping names to custom objects
Returns:
Regularizer instance
"""
def get(identifier):
"""
Retrieve a regularizer by string identifier.
Parameters:
- identifier: String name or regularizer instance
Returns:
Regularizer instance
"""Usage Examples
Basic Regularization
import keras
from keras import regularizers
# Using string identifiers
model = keras.Sequential([
keras.layers.Dense(64, kernel_regularizer='l2', activation='relu'),
keras.layers.Dense(32, kernel_regularizer='l1', activation='relu'),
keras.layers.Dense(10, activation='softmax')
])
# Using regularizer classes directly
model = keras.Sequential([
keras.layers.Dense(64,
kernel_regularizer=regularizers.L2(0.01),
bias_regularizer=regularizers.L1(0.01),
activation='relu'),
keras.layers.Dense(32,
kernel_regularizer=regularizers.L1L2(l1=0.01, l2=0.01),
activation='relu'),
keras.layers.Dense(10, activation='softmax')
])Advanced Regularization
import keras
from keras import regularizers
# Orthogonal regularization for maintaining diverse representations
layer = keras.layers.Dense(
128,
kernel_regularizer=regularizers.OrthogonalRegularizer(factor=0.01),
activation='relu'
)
# Different regularizers for different parts of the layer
layer = keras.layers.Dense(
64,
kernel_regularizer=regularizers.L2(0.01), # Weight regularization
bias_regularizer=regularizers.L1(0.01), # Bias regularization
activity_regularizer=regularizers.L1(0.01), # Output regularization
activation='relu'
)
# Custom regularization strength
strong_l2 = regularizers.L2(0.1) # Strong regularization
weak_l1 = regularizers.L1(0.001) # Weak regularization
model = keras.Sequential([
keras.layers.Dense(128, kernel_regularizer=strong_l2, activation='relu'),
keras.layers.Dropout(0.5), # Combine with dropout for better regularization
keras.layers.Dense(64, kernel_regularizer=weak_l1, activation='relu'),
keras.layers.Dense(10, activation='softmax')
])Regularization in Different Layer Types
import keras
from keras import regularizers
# Convolutional layers
conv_model = keras.Sequential([
keras.layers.Conv2D(32, 3,
kernel_regularizer=regularizers.L2(0.01),
activation='relu'),
keras.layers.Conv2D(64, 3,
kernel_regularizer=regularizers.L1L2(l1=0.01, l2=0.01),
activation='relu'),
keras.layers.GlobalAveragePooling2D(),
keras.layers.Dense(10, activation='softmax')
])
# Recurrent layers
rnn_model = keras.Sequential([
keras.layers.LSTM(64,
kernel_regularizer=regularizers.L2(0.01),
recurrent_regularizer=regularizers.L1(0.01),
return_sequences=True),
keras.layers.LSTM(32,
kernel_regularizer=regularizers.OrthogonalRegularizer(0.01)),
keras.layers.Dense(10, activation='softmax')
])Regularization Guidelines
When to Use Each Type:
- L1 Regularization: Use when you want sparse weights (feature selection)
- L2 Regularization: Use for general overfitting prevention and smooth solutions
- L1L2 Regularization: Use when you want both sparsity and weight decay
- Orthogonal Regularization: Use when you want diverse, uncorrelated representations
Typical Regularization Strengths:
- Light regularization: 0.001 - 0.01
- Moderate regularization: 0.01 - 0.1
- Strong regularization: 0.1 - 1.0
Best Practices:
- Start with L2 regularization (0.01) as a baseline
- Combine with dropout for better regularization
- Use different strengths for different layers
- Monitor validation loss to tune regularization strength
- Apply regularization primarily to dense layers rather than convolutional layers
Install with Tessl CLI
npx tessl i tessl/pypi-keras