0
# Extension Interfaces
1

2
Specialized interfaces for extension functionality, image analysis, and advanced processing operations. These classes provide structured access to extension-specific APIs and specialized processing tools.
3

4
## Capabilities
5

6
### Model Keywords Interface
7

8
Interface for retrieving model-specific keywords and metadata.
9

10
```python { .api }
11
class ModelKeywordInterface:
12
    """Interface for model keyword generation and lookup."""
13
    
14
    def __init__(self, webuiapi: WebUIApi):
15
        """
16
        Initialize the interface with a WebUIApi instance.
17
        
18
        Parameters:
19
        - webuiapi: WebUIApi client instance
20
        """
21
    
22
    def get_keywords(self, model_name: str) -> ModelKeywordResult:
23
        """
24
        Get keywords associated with a specific model.
25

26
        Parameters:
27
        - model_name: Name of the model to query
28

29
        Returns:
30
        ModelKeywordResult containing keywords and metadata
31
        """
32

33
class ModelKeywordResult:
34
    """Result container for model keyword operations."""
35
    
36
    def __init__(self):
37
        self.keywords: List[str] = []  # Associated keywords
38
        self.model: str = ""  # Model name
39
        self.oldhash: str = ""  # Model hash
40
        self.match_source: str = ""  # Source of keyword match
41
```
42

43
### Image Processing Interfaces
44

45
Specialized interfaces for advanced image processing operations.
46

47
```python { .api }
48
class InstructPix2PixInterface:
49
    """Interface for instruction-based image editing using InstructPix2Pix."""
50
    
51
    def __init__(self, webuiapi: WebUIApi):
52
        """
53
        Initialize the interface with a WebUIApi instance.
54
        
55
        Parameters:
56
        - webuiapi: WebUIApi client instance
57
        """
58
    
59
    def img2img(
60
        self,
61
        prompt: str,
62
        images: List[Image.Image],
63
        negative_prompt: str = "",
64
        **kwargs
65
    ) -> WebUIApiResult:
66
        """
67
        Edit images using natural language instructions.
68

69
        Parameters:
70
        - prompt: Natural language editing instruction
71
        - images: List of input images to edit
72
        - negative_prompt: What to avoid in editing
73
        - **kwargs: Additional img2img parameters
74

75
        Returns:
76
        WebUIApiResult containing edited images
77
        """
78

79
class RemBGInterface:
80
    """Interface for background removal operations."""
81
    
82
    def __init__(self, webuiapi: WebUIApi):
83
        """
84
        Initialize the interface with a WebUIApi instance.
85
        
86
        Parameters:
87
        - webuiapi: WebUIApi client instance
88
        """
89
    
90
    def rembg(
91
        self,
92
        input_image: str,
93
        model: str = "u2net",
94
        return_mask: bool = False,
95
        alpha_matting: bool = False,
96
        alpha_matting_foreground_threshold: int = 270,
97
        alpha_matting_background_threshold: int = 10,
98
        alpha_matting_erode_size: int = 10
99
    ) -> Dict:
100
        """
101
        Remove background from images.
102

103
        Parameters:
104
        - input_image: Base64-encoded input image
105
        - model: Background removal model ("u2net", "u2netp", "silueta", etc.)
106
        - return_mask: Return the segmentation mask
107
        - alpha_matting: Apply alpha matting for better edges
108
        - alpha_matting_foreground_threshold: Foreground threshold for alpha matting
109
        - alpha_matting_background_threshold: Background threshold for alpha matting  
110
        - alpha_matting_erode_size: Erosion size for alpha matting
111

112
        Returns:
113
        Dictionary containing processed image and optional mask
114
        """
115
```
116

117
### Segment Anything Interface
118

119
Comprehensive interface for Segment Anything Model (SAM) operations including detection, segmentation, and mask processing.
120

121
```python { .api }
122
class SegmentAnythingInterface:
123
    """Interface for Segment Anything Model operations."""
124
    
125
    def __init__(self, webuiapi: WebUIApi):
126
        """
127
        Initialize the interface with a WebUIApi instance.
128
        
129
        Parameters:
130
        - webuiapi: WebUIApi client instance
131
        """
132
    
133
    def heartbeat(self) -> Dict:
134
        """
135
        Check if SegmentAnything extension is responsive.
136

137
        Returns:
138
        Status dictionary indicating extension health
139
        """
140
    
141
    def get_sam_models(self) -> List[str]:
142
        """
143
        Get list of available SAM models.
144

145
        Returns:
146
        List of SAM model names
147
        """
148
    
149
    def sam_predict(
150
        self,
151
        sam_model_name: str,
152
        input_image: str,
153
        sam_positive_points: List[List[int]] = None,
154
        sam_negative_points: List[List[int]] = None,
155
        sam_bbox: List[int] = None,
156
        **kwargs
157
    ) -> SegmentAnythingSamResult:
158
        """
159
        Perform SAM prediction with point or bounding box prompts.
160

161
        Parameters:
162
        - sam_model_name: SAM model to use for prediction
163
        - input_image: Base64-encoded input image
164
        - sam_positive_points: List of [x, y] positive prompt points
165
        - sam_negative_points: List of [x, y] negative prompt points
166
        - sam_bbox: Bounding box as [x1, y1, x2, y2]
167

168
        Returns:
169
        SegmentAnythingSamResult containing segmentation masks
170
        """
171
    
172
    def dino_predict(
173
        self,
174
        dino_model_name: str,
175
        input_image: str,
176
        dino_text_prompt: str,
177
        dino_box_threshold: float = 0.3,
178
        dino_preview_checkbox: bool = False,
179
        **kwargs
180
    ) -> SegmentAnythingGinoResult:
181
        """
182
        Perform DINO prediction for object detection.
183

184
        Parameters:
185
        - dino_model_name: DINO model to use
186
        - input_image: Base64-encoded input image
187
        - dino_text_prompt: Text description of objects to detect
188
        - dino_box_threshold: Detection confidence threshold
189
        - dino_preview_checkbox: Generate preview visualization
190

191
        Returns:
192
        SegmentAnythingGinoResult containing detection results
193
        """
194
    
195
    def dilate_mask(
196
        self,
197
        input_image: str,
198
        mask: str,
199
        dilate_amount: int = 5,
200
        **kwargs
201
    ) -> SegmentAnythingDilationResult:
202
        """
203
        Dilate/erode segmentation masks.
204

205
        Parameters:
206
        - input_image: Base64-encoded input image
207
        - mask: Base64-encoded mask image
208
        - dilate_amount: Dilation amount (positive=dilate, negative=erode)
209

210
        Returns:
211
        SegmentAnythingDilationResult containing processed mask
212
        """
213
    
214
    def generate_semantic_segmentation(
215
        self,
216
        input_image: str,
217
        category_names: List[str],
218
        **kwargs
219
    ) -> Dict:
220
        """
221
        Generate semantic segmentation with category labels.
222

223
        Parameters:
224
        - input_image: Base64-encoded input image
225
        - category_names: List of category names to segment
226

227
        Returns:
228
        Dictionary containing semantic segmentation results
229
        """
230
    
231
    def sam_and_semantic_seg_with_cat_id(
232
        self,
233
        input_image: str,
234
        category_id: int,
235
        **kwargs
236
    ) -> SegmentAnythingSemanticSegWithCatIdResult:
237
        """
238
        Combine SAM with semantic segmentation using category ID.
239

240
        Parameters:
241
        - input_image: Base64-encoded input image
242
        - category_id: Category ID for segmentation
243

244
        Returns:
245
        SegmentAnythingSemanticSegWithCatIdResult containing combined results
246
        """
247
```
248

249
### Image Tagging Interface
250

251
Interface for automatic image tagging and interrogation using various models.
252

253
```python { .api }
254
class TaggerInterface:
255
    """Interface for automatic image tagging and analysis."""
256
    
257
    def __init__(self, webuiapi: WebUIApi):
258
        """
259
        Initialize the interface with a WebUIApi instance.
260
        
261
        Parameters:
262
        - webuiapi: WebUIApi client instance
263
        """
264
    
265
    def tagger_interrogate(
266
        self,
267
        image: str,
268
        model: str = "wd14-vit-v2-git",
269
        threshold: float = 0.35,
270
        additional_tags: str = "",
271
        exclude_tags: str = "",
272
        sort_by_alphabetical_order: bool = False,
273
        add_confident_as_weight: bool = False,
274
        replace_underscore: bool = False,
275
        replace_underscore_excludes: str = "",
276
        escape_tag: bool = False
277
    ) -> Dict:
278
        """
279
        Generate tags for images using trained tagging models.
280

281
        Parameters:
282
        - image: Base64-encoded input image
283
        - model: Tagging model to use ("wd14-vit-v2-git", "deepdanbooru", etc.)
284
        - threshold: Confidence threshold for tag inclusion
285
        - additional_tags: Additional tags to include
286
        - exclude_tags: Tags to exclude from results
287
        - sort_by_alphabetical_order: Sort tags alphabetically
288
        - add_confident_as_weight: Add confidence as weight to tags
289
        - replace_underscore: Replace underscores in tags
290
        - replace_underscore_excludes: Tags to exclude from underscore replacement
291
        - escape_tag: Escape special characters in tags
292

293
        Returns:
294
        Dictionary containing generated tags and confidence scores
295
        """
296
    
297
    def tagger_interrogators(self) -> List[str]:
298
        """
299
        Get list of available tagging models.
300

301
        Returns:
302
        List of available interrogator/tagger model names
303
        """
304
```
305

306
**Usage Examples:**
307

308
```python
309
import webuiapi
310
from PIL import Image
311

312
api = webuiapi.WebUIApi()
313

314
# Model keywords lookup
315
keyword_interface = webuiapi.ModelKeywordInterface(api)
316
result = keyword_interface.get_keywords("realistic_vision_v40")
317
print(f"Keywords: {result.keywords}")
318

319
# Background removal
320
rembg_interface = webuiapi.RemBGInterface(api)
321
image = Image.open("portrait.jpg")
322
image_b64 = webuiapi.raw_b64_img(image)
323

324
bg_result = rembg_interface.rembg(
325
    input_image=image_b64,
326
    model="u2net",
327
    return_mask=True,
328
    alpha_matting=True
329
)
330

331
# Instruction-based editing
332
instruct_interface = webuiapi.InstructPix2PixInterface(api)
333
edit_result = instruct_interface.img2img(
334
    prompt="make the sky more dramatic",
335
    images=[image],
336
    negative_prompt="unrealistic, fake"
337
)
338

339
# Segment Anything operations
340
sam_interface = webuiapi.SegmentAnythingInterface(api)
341

342
# Check available models
343
sam_models = sam_interface.get_sam_models()
344
print(f"Available SAM models: {sam_models}")
345

346
# SAM prediction with point prompts
347
sam_result = sam_interface.sam_predict(
348
    sam_model_name="sam_vit_h_4b8939.pth",
349
    input_image=image_b64,
350
    sam_positive_points=[[250, 300], [280, 320]],  # Click points
351
    sam_negative_points=[[100, 100]]  # Avoid this area
352
)
353

354
# DINO object detection
355
dino_result = sam_interface.dino_predict(
356
    dino_model_name="GroundingDINO_SwinT_OGC",
357
    input_image=image_b64,
358
    dino_text_prompt="person, face",
359
    dino_box_threshold=0.3
360
)
361

362
# Image tagging
363
tagger_interface = webuiapi.TaggerInterface(api)
364

365
# Get available taggers
366
taggers = tagger_interface.tagger_interrogators()
367
print(f"Available taggers: {taggers}")
368

369
# Generate tags
370
tag_result = tagger_interface.tagger_interrogate(
371
    image=image_b64,
372
    model="wd14-vit-v2-git",
373
    threshold=0.35,
374
    sort_by_alphabetical_order=True,
375
    add_confident_as_weight=True
376
)
377

378
print(f"Generated tags: {tag_result}")
379
```
380

381
## Types
382

383
```python { .api }
384
class ModelKeywordResult:
385
    """Container for model keyword lookup results."""
386
    keywords: List[str]  # Associated keywords
387
    model: str  # Model name
388
    oldhash: str  # Model hash
389
    match_source: str  # Source of keyword match
390

391
class SegmentAnythingSamResult:
392
    """Result container for SAM prediction operations."""
393
    # Contains segmentation masks and prediction metadata
394

395
class SegmentAnythingGinoResult:
396
    """Result container for DINO prediction operations."""
397
    # Contains object detection boxes and confidence scores
398

399
class SegmentAnythingDilationResult:
400
    """Result container for mask dilation operations."""
401
    # Contains processed masks after dilation/erosion
402

403
class SegmentAnythingControlNetSegRandomResult:
404
    """Result container for ControlNet segmentation (random mode)."""
405
    # Contains segmentation results with random sampling
406

407
class SegmentAnythingControlNetSegNotRandomResult:
408
    """Result container for ControlNet segmentation (non-random mode)."""
409
    # Contains deterministic segmentation results
410

411
class SegmentAnythingSemanticSegWithCatIdResult:
412
    """Result container for semantic segmentation with category ID."""
413
    # Contains semantic segmentation with category labels
414
```