0
# Generator Components
1

2
Text generation components for creating answers, summaries, and other text outputs using Large Language Models (LLMs). The primary generation component in Haystack is PromptNode, which provides a unified interface for multiple LLM providers and supports prompt templates for various NLP tasks.
3

4
## Core Imports
5

6
```python
7
from haystack.nodes import PromptNode, PromptTemplate, PromptModel
8
from haystack.nodes.prompt import BaseOutputParser, AnswerParser
9
from haystack.nodes.prompt.invocation_layer import PromptModelInvocationLayer
10
```
11

12
## Capabilities
13

14
### PromptNode
15

16
The central abstraction for LLM support in Haystack, providing unified access to multiple model providers including OpenAI, Azure OpenAI, Cohere, and Hugging Face transformers.
17

18
```python { .api }
19
class PromptNode(BaseComponent):
20
    def __init__(
21
        self,
22
        model_name_or_path: Union[str, PromptModel] = "google/flan-t5-base",
23
        default_prompt_template: Optional[Union[str, PromptTemplate]] = None,
24
        output_variable: Optional[str] = None,
25
        max_length: Optional[int] = 100,
26
        api_key: Optional[str] = None,
27
        api_base: Optional[str] = None,
28
        timeout: Optional[float] = None,
29
        use_auth_token: Optional[Union[str, bool]] = None,
30
        use_gpu: Optional[bool] = None,
31
        devices: Optional[List[Union[str, "torch.device"]]] = None,
32
        stop_words: Optional[List[str]] = None,
33
        top_k: int = 1,
34
        debug: Optional[bool] = False,
35
        model_kwargs: Optional[Dict] = None,
36
        truncate: bool = True,
37
    ):
38
        """
39
        Creates a PromptNode instance for text generation tasks.
40
        
41
        Args:
42
            model_name_or_path: Model identifier or PromptModel instance
43
            default_prompt_template: Default template for prompts
44
            output_variable: Variable name for storing results 
45
            max_length: Maximum tokens in generated output
46
            api_key: API key for external services
47
            api_base: Base URL for API endpoints
48
            timeout: Request timeout in seconds
49
            use_auth_token: Authentication token for Hugging Face
50
            use_gpu: Whether to use GPU acceleration
51
            devices: Specific devices to use
52
            stop_words: Words that stop generation
53
            top_k: Number of outputs to generate per prompt
54
            debug: Enable debug logging
55
            model_kwargs: Additional model parameters
56
            truncate: Whether to truncate long inputs
57
        """
58
    
59
    def predict(self, query: str, documents: List[Document] = None, **kwargs) -> List[str]:
60
        """Generate text based on query and optional documents."""
61
    
62
    def prompt(self, prompt_template: Union[str, PromptTemplate], **kwargs) -> List[str]:
63
        """Generate text using a specific prompt template."""
64
```
65

66
### PromptTemplate
67

68
Template system for defining reusable prompts with placeholders and output parsing.
69

70
```python { .api }
71
class PromptTemplate:
72
    def __init__(
73
        self,
74
        prompt: str,
75
        output_parser: BaseOutputParser = None,
76
        output_variable: str = "results",
77
        name: str = None
78
    ):
79
        """
80
        Create a prompt template with placeholder variables.
81
        
82
        Args:
83
            prompt: Template string with {variable} placeholders
84
            output_parser: Parser for structured output extraction
85
            output_variable: Name for the output variable
86
            name: Template identifier
87
        """
88
    
89
    def fill(self, **kwargs) -> str:
90
        """Fill template placeholders with provided values."""
91
```
92

93
### PromptModel
94

95
Shared model instance that can be used across multiple PromptNodes to save memory and loading time.
96

97
```python { .api }
98
class PromptModel:
99
    def __init__(
100
        self,
101
        model_name_or_path: str = "google/flan-t5-base",
102
        max_length: Optional[int] = 100,
103
        api_key: Optional[str] = None,
104
        use_auth_token: Optional[Union[str, bool]] = None,
105
        use_gpu: Optional[bool] = None,
106
        devices: Optional[List[Union[str, "torch.device"]]] = None,
107
        model_kwargs: Optional[Dict] = None,
108
    ):
109
        """
110
        Create a reusable model instance for multiple PromptNodes.
111
        
112
        Args:
113
            model_name_or_path: Model identifier
114
            max_length: Maximum output length
115
            api_key: API key for external services
116
            use_auth_token: Hugging Face authentication
117
            use_gpu: GPU usage preference
118
            devices: Specific devices to use
119
            model_kwargs: Additional model parameters
120
        """
121
```
122

123
## Usage Examples
124

125
### Basic Text Generation
126

127
```python
128
from haystack.nodes import PromptNode
129

130
# Initialize with OpenAI GPT
131
generator = PromptNode(
132
    model_name_or_path="text-davinci-003",
133
    api_key="your-openai-key",
134
    max_length=200
135
)
136

137
# Generate text
138
result = generator.predict("Explain quantum computing in simple terms")
139
print(result[0])  # Generated explanation
140
```
141

142
### Question Answering Generation
143

144
```python
145
from haystack.nodes import PromptNode, PromptTemplate
146
from haystack import Document
147

148
# Create QA template
149
qa_template = PromptTemplate(
150
    prompt="Given the context below, answer the question.\n\nContext: {join(documents)}\n\nQuestion: {query}\n\nAnswer:",
151
    output_variable="answer"
152
)
153

154
# Initialize generator
155
generator = PromptNode(
156
    model_name_or_path="google/flan-t5-large",
157
    default_prompt_template=qa_template
158
)
159

160
# Generate answer with context
161
docs = [Document(content="Python is a programming language known for its simplicity.")]
162
answer = generator.predict("What is Python known for?", documents=docs)
163
print(answer[0])  # "Python is known for its simplicity."
164
```
165

166
### Summarization
167

168
```python
169
from haystack.nodes import PromptNode, PromptTemplate
170

171
# Create summarization template
172
summary_template = PromptTemplate(
173
    prompt="Summarize the following text in 2-3 sentences:\n\n{documents}\n\nSummary:",
174
    output_variable="summary"
175
)
176

177
# Initialize summarizer
178
summarizer = PromptNode(
179
    model_name_or_path="google/flan-t5-base",
180
    default_prompt_template=summary_template,
181
    max_length=150
182
)
183

184
# Summarize documents
185
docs = [Document(content="Long text content to be summarized...")]
186
summary = summarizer.predict(documents=docs)
187
print(summary[0])
188
```
189

190
### Using Multiple Model Providers
191

192
```python
193
# OpenAI GPT
194
openai_generator = PromptNode(
195
    model_name_or_path="gpt-3.5-turbo",
196
    api_key="your-openai-key"
197
)
198

199
# Hugging Face model
200
hf_generator = PromptNode(
201
    model_name_or_path="google/flan-t5-base",
202
    use_gpu=True
203
)
204

205
# Azure OpenAI
206
azure_generator = PromptNode(
207
    model_name_or_path="gpt-35-turbo",
208
    api_key="your-azure-key",
209
    api_base="https://your-resource.openai.azure.com/"
210
)
211
```
212

213
## Types
214

215
```python { .api }
216
from typing import Union, List, Dict, Optional, Any
217
from haystack.schema import Document
218

219
# Output parser base class
220
class BaseOutputParser:
221
    def parse(self, model_output: str, prompt_template: PromptTemplate) -> Any:
222
        """Parse structured output from model response."""
223

224
# Answer-specific parser
225
class AnswerParser(BaseOutputParser):
226
    def parse(self, model_output: str, prompt_template: PromptTemplate) -> List[Answer]:
227
        """Parse model output into Answer objects."""
228

229
# Model invocation layer for custom providers
230
class PromptModelInvocationLayer:
231
    def invoke(self, prompt: str, **kwargs) -> List[str]:
232
        """Invoke model with prompt and return generated text."""
233
```