tessl/pypi-pyllamacpp

Python bindings for llama.cpp enabling efficient local language model inference without external API dependencies

—

Pending

Overview

Eval results

Files

Web User Interface

Name: tessl/pypi-pyllamacpp
Author: tessl

Streamlit-based web interface for interactive model testing and development. The web UI provides a browser-based chat interface with configurable parameters, real-time model interaction, and an intuitive graphical interface for users who prefer web-based interaction over command-line tools.

Capabilities

Web UI Launch

Launch the Streamlit web interface for interactive model testing and configuration.

def webui() -> None:
    """
    Launch the Streamlit web user interface.
    
    Creates a browser-based interface for interacting with PyLLaMACpp models,
    providing chat functionality, parameter configuration, and real-time
    text generation monitoring.
    """

def run():
    """
    Run the web UI application.
    
    Alternative entry point for launching the web interface,
    typically used for programmatic access or custom deployments.
    """

Basic Usage

Launch the web interface directly from Python:

from pyllamacpp.webui import webui

# Launch the web interface
webui()

Or use the run function:

from pyllamacpp.webui import run

# Alternative launch method
run()

The web interface will start a local Streamlit server accessible through your browser, typically at http://localhost:8501.

Web UI Features

The web interface provides:

Model Selection: Browse and select GGML model files
Parameter Configuration: Adjust generation parameters with sliders and inputs
Interactive Chat: Real-time chat interface with the loaded model
Response Streaming: Live display of generated tokens as they are produced
Session History: Maintain conversation history within the session
Parameter Presets: Save and load commonly used parameter configurations
Model Information: Display model metadata and performance statistics

Interface Components

The web UI includes several key components:

Model Configuration Panel

Model Path Selection: File browser for selecting GGML model files
Context Parameters: Configure n_ctx, seed, f16_kv, and other model settings
GPU Settings: Configure GPU layer offloading and memory options

Generation Parameters Panel

Sampling Controls: Temperature, top-p, top-k sliders
Length Controls: n_predict, n_keep configuration
Advanced Parameters: Mirostat, repetition penalty, frequency penalty
Performance Settings: Thread count, batch size optimization

Chat Interface

Message Input: Text area for entering prompts and questions
Response Display: Real-time streaming output with syntax highlighting
Conversation History: Scrollable chat history with user/AI distinction
Action Buttons: Send, clear, reset context controls

Configuration Schema Integration

The web UI uses the same parameter schemas as the CLI for consistency:

# Integration with CLI parameter schemas
from pyllamacpp.constants import GPT_PARAMS_SCHEMA

# Dynamic UI generation based on parameter schema
def _generate_config_ui(model_name, config_schema):
    """Generate UI components based on parameter schema."""
    
def _config_ui(config_name: str, key: str, config: dict):
    """Generate individual parameter UI component."""
    
def _get_config_from_session_state(model_name: str, config_schema: dict) -> dict:
    """Extract configuration from Streamlit session state."""

Advanced Usage

Custom Web UI Deployment

Deploy the web UI with custom configuration:

import streamlit as st
from pyllamacpp.webui import webui
from pyllamacpp.model import Model

# Custom Streamlit configuration
st.set_page_config(
    page_title="Custom PyLLaMACpp Interface",
    page_icon="🦙",
    layout="wide"
)

# Launch with custom settings
webui()

Programmatic Integration

Integrate web UI components into custom applications:

import streamlit as st
from pyllamacpp.webui import _create_model, _get_config_from_session_state
from pyllamacpp.constants import GPT_PARAMS_SCHEMA

def custom_model_interface():
    st.title("Custom Model Interface")
    
    # Model selection
    model_path = st.file_uploader("Select GGML Model", type=['bin', 'ggml'])
    
    if model_path:
        # Create model with UI configuration
        model = _create_model(model_path.name, n_ctx=2048)
        
        # Get parameters from UI
        params = _get_config_from_session_state("custom_model", GPT_PARAMS_SCHEMA)
        
        # Chat interface
        if prompt := st.chat_input("Enter your message"):
            with st.chat_message("user"):
                st.write(prompt)
            
            with st.chat_message("assistant"):
                response_placeholder = st.empty()
                full_response = ""
                
                for token in model.generate(prompt, **params):
                    full_response += token
                    response_placeholder.write(full_response)

custom_model_interface()

Session State Management

The web UI manages state across user interactions:

# Session state initialization
def _init_session_state(key: str, val: any):
    """Initialize session state variable if not exists."""

# Key generation for unique UI components
def _get_key(model_name: str, config_name: str) -> str:
    """Generate unique key for UI components."""
    return f"{model_name}-{config_name}"

Real-time Features

Live Token Streaming

The web UI displays tokens as they are generated:

def new_text_generated(text):
    """Callback function for new text generation."""
    # Real-time display in Streamlit interface
    st.write(text, end="")

Performance Monitoring

Display real-time performance metrics:

Tokens per second: Generation speed monitoring
Memory usage: RAM and GPU memory consumption
Model load time: Initial model loading performance
Response latency: Time from prompt to first token

Web UI Configuration

Streamlit Configuration

The web UI can be configured through Streamlit's config system:

# .streamlit/config.toml
[server]
port = 8501
address = "0.0.0.0"

[theme]
primaryColor = "#FF6B6B"
backgroundColor = "#FFFFFF"
secondaryBackgroundColor = "#F0F2F6"
textColor = "#262730"

[browser]
gatherUsageStats = false

Custom Themes

Apply custom styling to the web interface:

import streamlit as st

# Custom CSS styling
st.markdown("""
<style>
.stChatMessage {
    padding: 1rem;
    border-radius: 0.5rem;
    margin-bottom: 1rem;
}

.user-message {
    background-color: #E3F2FD;
    border-left: 4px solid #2196F3;
}

.assistant-message {
    background-color: #F3E5F5;
    border-left: 4px solid #9C27B0;
}
</style>
""", unsafe_allow_html=True)

Deployment Options

Local Development

Run the web UI for local development:

# Direct Python execution
python -c "from pyllamacpp.webui import webui; webui()"

# Or using Streamlit directly
streamlit run -m pyllamacpp.webui

Production Deployment

Deploy the web UI in production environments:

# Docker deployment
docker run -p 8501:8501 -v /models:/models streamlit-pyllamacpp

# Cloud deployment with specific configuration
streamlit run --server.port 8080 --server.address 0.0.0.0 -m pyllamacpp.webui

Security Considerations

When deploying the web UI:

Model Access: Restrict access to model files and directories
Network Security: Use HTTPS in production deployments
Resource Limits: Configure memory and CPU limits
Authentication: Implement user authentication for sensitive deployments
Input Validation: Sanitize user inputs and prompts

Integration with External Tools

The web UI can be extended with additional functionality:

# Integration with monitoring tools
import streamlit as st
import plotly.graph_objects as go

def add_performance_dashboard():
    """Add performance monitoring dashboard."""
    col1, col2 = st.columns(2)
    
    with col1:
        st.metric("Tokens/sec", "45.2", "2.1")
    
    with col2:
        st.metric("Memory Usage", "3.2 GB", "-0.1 GB")
    
    # Performance graph
    fig = go.Figure()
    fig.add_trace(go.Scatter(y=[40, 42, 45, 43, 47], name="Tokens/sec"))
    st.plotly_chart(fig)

Dependencies

The web UI requires additional dependencies:

# Required packages
import streamlit as st
from streamlit import runtime
from streamlit_ace import st_ace, KEYBINDINGS, LANGUAGES, THEMES
from streamlit.web import cli as stcli

Install web UI dependencies: