tessl/pypi-keras-nightly

Multi-backend deep learning framework providing a unified API for building and training neural networks across JAX, TensorFlow, PyTorch, and OpenVINO backends

—

Pending

Overview

Eval results

Files

Activation Functions

Name: tessl/pypi-keras-nightly
Author: tessl

Comprehensive collection of activation functions for neural networks. Activation functions introduce non-linearity into neural networks, enabling them to learn complex patterns and representations. Keras provides a wide range of activation functions from traditional sigmoid and tanh to modern alternatives like GELU and Swish.

Capabilities

Standard Activation Functions

Traditional activation functions commonly used in neural networks.

def relu(x, negative_slope=0.0, max_value=None, threshold=0.0):
    """
    Rectified Linear Unit activation function.
    
    Args:
        x: Input tensor
        negative_slope: float, slope for values below threshold (default: 0.0)
        max_value: float, saturation threshold (default: None)
        threshold: float, threshold value below which values are damped (default: 0.0)
    
    Returns:
        Tensor with same shape and dtype as input
    """

def sigmoid(x):
    """
    Sigmoid activation function.
    
    Args:
        x: Input tensor
    
    Returns:
        Tensor with values in range (0, 1)
    """

def tanh(x):
    """
    Hyperbolic tangent activation function.
    
    Args:
        x: Input tensor
    
    Returns:
        Tensor with values in range (-1, 1)
    """

def softmax(x, axis=-1):
    """
    Softmax activation function.
    
    Args:
        x: Input tensor
        axis: int, axis along which to apply softmax (default: -1)
    
    Returns:
        Tensor with values summing to 1 along specified axis
    """

def linear(x):
    """
    Linear (identity) activation function.
    
    Args:
        x: Input tensor
    
    Returns:
        Input tensor unchanged
    """

Modern Activation Functions

Contemporary activation functions that often provide better performance than traditional alternatives.

def gelu(x, approximate=False):
    """
    Gaussian Error Linear Unit activation function.
    
    Args:
        x: Input tensor
        approximate: bool, whether to use approximation (default: False)
    
    Returns:
        Tensor with GELU activation applied
    """

def silu(x):
    """
    Sigmoid Linear Unit (SiLU/Swish) activation function.
    
    Args:
        x: Input tensor
    
    Returns:
        Tensor with SiLU activation applied
    """

def swish(x):
    """
    Swish activation function (alias for SiLU).
    
    Args:
        x: Input tensor
    
    Returns:
        Tensor with Swish activation applied
    """

def mish(x):
    """
    Mish activation function.
    
    Args:
        x: Input tensor
    
    Returns:
        Tensor with Mish activation applied
    """

Exponential and Logarithmic Functions

Activation functions based on exponential and logarithmic operations.

def elu(x, alpha=1.0):
    """
    Exponential Linear Unit activation function.
    
    Args:
        x: Input tensor
        alpha: float, scale for negative part (default: 1.0)
    
    Returns:
        Tensor with ELU activation applied
    """

def selu(x):
    """
    Scaled Exponential Linear Unit activation function.
    
    Args:
        x: Input tensor
    
    Returns:
        Tensor with SELU activation applied
    """

def celu(x, alpha=1.0):
    """
    Continuously Differentiable Exponential Linear Unit.
    
    Args:
        x: Input tensor
        alpha: float, scale parameter (default: 1.0)
    
    Returns:
        Tensor with CELU activation applied
    """

def exponential(x):
    """
    Exponential activation function.
    
    Args:
        x: Input tensor
    
    Returns:
        Tensor with exponential activation applied
    """

def log_sigmoid(x):
    """
    Logarithm of sigmoid activation function.
    
    Args:
        x: Input tensor
    
    Returns:
        Tensor with log-sigmoid activation applied
    """

def log_softmax(x, axis=-1):
    """
    Log-softmax activation function.
    
    Args:
        x: Input tensor
        axis: int, axis along which to apply log-softmax (default: -1)
    
    Returns:
        Tensor with log-softmax activation applied
    """

Specialized and Experimental Functions

Less common activation functions for specific use cases and experimental applications.

def relu6(x):
    """
    ReLU6 activation function (ReLU capped at 6).
    
    Args:
        x: Input tensor
    
    Returns:
        Tensor with ReLU6 activation applied
    """

def leaky_relu(x, negative_slope=0.3):
    """
    Leaky ReLU activation function.
    
    Args:
        x: Input tensor
        negative_slope: float, slope for negative values (default: 0.3)
    
    Returns:
        Tensor with Leaky ReLU activation applied
    """

def hard_sigmoid(x):
    """
    Hard sigmoid activation function.
    
    Args:
        x: Input tensor
    
    Returns:
        Tensor with hard sigmoid activation applied
    """

def hard_silu(x):
    """
    Hard SiLU activation function.
    
    Args:
        x: Input tensor
    
    Returns:
        Tensor with Hard SiLU activation applied
    """

def hard_swish(x):
    """
    Hard Swish activation function (alias for hard_silu).
    
    Args:
        x: Input tensor
    
    Returns:
        Tensor with Hard Swish activation applied
    """

def hard_tanh(x):
    """
    Hard hyperbolic tangent activation function.
    
    Args:
        x: Input tensor
    
    Returns:
        Tensor with hard tanh activation applied
    """

def softplus(x):
    """
    Softplus activation function.
    
    Args:
        x: Input tensor
    
    Returns:
        Tensor with softplus activation applied
    """

def softsign(x):
    """
    Softsign activation function.
    
    Args:
        x: Input tensor
    
    Returns:
        Tensor with softsign activation applied
    """

def glu(x, axis=-1):
    """
    Gated Linear Unit activation function.
    
    Args:
        x: Input tensor
        axis: int, axis along which to split input (default: -1)
    
    Returns:
        Tensor with GLU activation applied
    """

Sparse and Shrinkage Functions

Specialized activation functions for sparsity and value shrinkage.

def sparsemax(x, axis=-1):
    """
    Sparsemax activation function.
    
    Args:
        x: Input tensor
        axis: int, axis along which to apply sparsemax (default: -1)
    
    Returns:
        Tensor with sparsemax activation applied
    """

def sparse_plus(x):
    """
    Sparse plus activation function.
    
    Args:
        x: Input tensor
    
    Returns:
        Tensor with sparse plus activation applied
    """

def sparse_sigmoid(x):
    """
    Sparse sigmoid activation function.
    
    Args:
        x: Input tensor
    
    Returns:
        Tensor with sparse sigmoid activation applied
    """

def squareplus(x, b=4.0):
    """
    Squareplus activation function.
    
    Args:
        x: Input tensor
        b: float, smoothness parameter (default: 4.0)
    
    Returns:
        Tensor with squareplus activation applied
    """

def hard_shrink(x, lambd=0.5):
    """
    Hard shrinkage activation function.
    
    Args:
        x: Input tensor
        lambd: float, threshold parameter (default: 0.5)
    
    Returns:
        Tensor with hard shrinkage applied
    """

def soft_shrink(x, lambd=0.5):
    """
    Soft shrinkage activation function.
    
    Args:
        x: Input tensor
        lambd: float, threshold parameter (default: 0.5)
    
    Returns:
        Tensor with soft shrinkage applied
    """

def tanh_shrink(x):
    """
    Tanh shrinkage activation function.
    
    Args:
        x: Input tensor
    
    Returns:
        Tensor with tanh shrinkage applied
    """

def threshold(x, threshold_value=0.0, value=0.0):
    """
    Threshold activation function.
    
    Args:
        x: Input tensor
        threshold_value: float, threshold value (default: 0.0)
        value: float, replacement value below threshold (default: 0.0)
    
    Returns:
        Tensor with threshold activation applied
    """

Utility Functions

Functions for serialization, deserialization, and retrieval of activation functions.

def get(identifier):
    """
    Retrieve an activation function by name or function.
    
    Args:
        identifier: str or callable, activation function identifier
    
    Returns:
        Activation function
    """

def serialize(activation):
    """
    Serialize an activation function to a configuration.
    
    Args:
        activation: Activation function to serialize
    
    Returns:
        Configuration dictionary
    """

def deserialize(config, custom_objects=None):
    """
    Deserialize an activation function from configuration.
    
    Args:
        config: Configuration dictionary
        custom_objects: dict, custom objects for deserialization
    
    Returns:
        Activation function
    """

Usage Examples

Basic Usage

import keras
from keras import activations

# Apply activation to tensor
x = keras.ops.convert_to_tensor([-2.0, -1.0, 0.0, 1.0, 2.0])

# ReLU activation
relu_output = activations.relu(x)  # [0., 0., 0., 1., 2.]

# Sigmoid activation
sigmoid_output = activations.sigmoid(x)  # [0.119, 0.269, 0.5, 0.731, 0.881]

# GELU activation
gelu_output = activations.gelu(x)  # [-0.045, -0.159, 0., 0.841, 1.955]

In Layer Definitions

from keras import layers

# Using activation as string
dense_layer = layers.Dense(64, activation='relu')

# Using activation function directly
dense_layer = layers.Dense(64, activation=activations.gelu)

# Using activation with parameters
def custom_relu(x):
    return activations.relu(x, negative_slope=0.1, max_value=6.0)

dense_layer = layers.Dense(64, activation=custom_relu)

Comparing Activation Functions

import numpy as np
import matplotlib.pyplot as plt

x = np.linspace(-5, 5, 100)
x_tensor = keras.ops.convert_to_tensor(x)

# Compare different activations
relu_out = activations.relu(x_tensor)
gelu_out = activations.gelu(x_tensor) 
silu_out = activations.silu(x_tensor)
tanh_out = activations.tanh(x_tensor)

Install with Tessl CLI