0
# Data Types and Response Objects
1

2
Comprehensive type system for Google Cloud Vision API, including request/response objects, annotation results, geometry types, configuration parameters, and enums. This provides strong typing for all API interactions with detailed structured data for analysis results.
3

4
## Request and Response Types
5

6
### Core Annotation Types
7

8
Primary request and response types for image annotation operations.
9

10
```python { .api }
11
class AnnotateImageRequest:
12
    """Request for annotating a single image."""
13
    image: Image
14
    features: List[Feature]
15
    image_context: Optional[ImageContext]
16

17
class AnnotateImageResponse:
18
    """Response from annotating a single image."""
19
    face_annotations: List[FaceAnnotation]
20
    landmark_annotations: List[EntityAnnotation]  
21
    logo_annotations: List[EntityAnnotation]
22
    label_annotations: List[EntityAnnotation]
23
    text_annotations: List[EntityAnnotation]
24
    full_text_annotation: Optional[TextAnnotation]
25
    safe_search_annotation: Optional[SafeSearchAnnotation]
26
    image_properties_annotation: Optional[ImageProperties]
27
    crop_hints_annotation: Optional[CropHintsAnnotation]
28
    web_detection: Optional[WebDetection]
29
    product_search_results: Optional[ProductSearchResults]
30
    localized_object_annotations: List[LocalizedObjectAnnotation]
31
    error: Optional[Status]
32

33
class BatchAnnotateImagesRequest:
34
    """Request for batch annotation of multiple images."""
35
    requests: List[AnnotateImageRequest]
36
    parent: Optional[str]
37

38
class BatchAnnotateImagesResponse:
39
    """Response from batch annotation of multiple images."""
40
    responses: List[AnnotateImageResponse]
41

42
class BatchAnnotateFilesRequest:
43
    """Request for batch annotation of files (PDF/TIFF)."""
44
    requests: List[AnnotateFileRequest]
45
    parent: Optional[str]
46

47
class BatchAnnotateFilesResponse:
48
    """Response from batch annotation of files."""
49
    responses: List[AnnotateFileResponse]
50

51
class AnnotateFileRequest:
52
    """Request for annotating a file."""
53
    input_config: InputConfig
54
    features: List[Feature]
55
    image_context: Optional[ImageContext]
56
    pages: Optional[List[int]]
57

58
class AnnotateFileResponse:
59
    """Response from annotating a file."""
60
    input_config: InputConfig
61
    responses: List[AnnotateImageResponse]
62
    total_pages: int
63
    error: Optional[Status]
64
```
65

66
### Async Operation Types
67

68
Types for long-running asynchronous operations.
69

70
```python { .api }
71
class AsyncAnnotateFileRequest:
72
    """Request for asynchronous file annotation."""
73
    input_config: InputConfig
74
    features: List[Feature]
75
    image_context: Optional[ImageContext]
76
    output_config: OutputConfig
77

78
class AsyncAnnotateFileResponse:
79
    """Response from asynchronous file annotation."""
80
    output_config: OutputConfig
81

82
class AsyncBatchAnnotateImagesRequest:
83
    """Request for asynchronous batch image annotation."""
84
    requests: List[AnnotateImageRequest]
85
    output_config: OutputConfig
86
    parent: Optional[str]
87

88
class AsyncBatchAnnotateImagesResponse:
89
    """Response from asynchronous batch image annotation."""
90
    output_config: OutputConfig
91

92
class AsyncBatchAnnotateFilesRequest:
93
    """Request for asynchronous batch file annotation."""
94
    requests: List[AsyncAnnotateFileRequest]
95
    parent: Optional[str]
96

97
class AsyncBatchAnnotateFilesResponse:
98
    """Response from asynchronous batch file annotation."""
99
    responses: List[AsyncAnnotateFileResponse]
100
```
101

102
## Detection Result Types
103

104
### Face Detection
105

106
Detailed face detection results with landmarks and attributes.
107

108
```python { .api }
109
class FaceAnnotation:
110
    """Face detection result with detailed attributes."""
111
    bounding_poly: BoundingPoly
112
    fd_bounding_poly: BoundingPoly
113
    landmarks: List[Landmark]
114
    roll_angle: float
115
    pan_angle: float
116
    tilt_angle: float
117
    detection_confidence: float
118
    landmarking_confidence: float
119
    joy_likelihood: Likelihood
120
    sorrow_likelihood: Likelihood
121
    anger_likelihood: Likelihood
122
    surprise_likelihood: Likelihood
123
    under_exposed_likelihood: Likelihood
124
    blurred_likelihood: Likelihood
125
    headwear_likelihood: Likelihood
126

127
class Landmark:
128
    """Individual facial landmark."""
129
    type_: LandmarkType
130
    position: Position
131
```
132

133
### Entity Detection
134

135
General entity detection results for landmarks, logos, and labels.
136

137
```python { .api }
138
class EntityAnnotation:
139
    """Generic entity detected in image."""
140
    mid: str
141
    locale: str
142
    description: str
143
    score: float
144
    confidence: float
145
    topicality: float
146
    bounding_poly: Optional[BoundingPoly]
147
    locations: List[LocationInfo]
148
    properties: List[Property]
149
```
150

151
### Object Localization
152

153
Localized object detection results.
154

155
```python { .api }
156
class LocalizedObjectAnnotation:
157
    """Localized object detection result."""
158
    mid: str
159
    language_code: str
160
    name: str
161
    score: float
162
    bounding_poly: BoundingPoly
163
```
164

165
### Text Detection
166

167
Comprehensive text detection and OCR results.
168

169
```python { .api }
170
class TextAnnotation:
171
    """Full text annotation result."""
172
    pages: List[Page]
173
    text: str
174

175
class Page:
176
    """Page of text with structured hierarchy."""
177
    property: Optional[TextProperty]
178
    width: int
179
    height: int
180
    blocks: List[Block]
181
    confidence: float
182

183
class Block:
184
    """Block of text (e.g., paragraph)."""
185
    property: Optional[TextProperty]
186
    bounding_box: BoundingPoly
187
    paragraphs: List[Paragraph]
188
    block_type: BlockType
189
    confidence: float
190

191
class Paragraph:
192
    """Paragraph within a block."""
193
    property: Optional[TextProperty]
194
    bounding_box: BoundingPoly
195
    words: List[Word]
196
    confidence: float
197

198
class Word:
199
    """Word within a paragraph."""
200
    property: Optional[TextProperty]
201
    bounding_box: BoundingPoly
202
    symbols: List[Symbol]
203
    confidence: float
204

205
class Symbol:
206
    """Individual symbol/character."""
207
    property: Optional[TextProperty]
208
    bounding_box: BoundingPoly
209
    text: str
210
    confidence: float
211
```
212

213
### Safe Search Detection
214

215
Content safety classification results.
216

217
```python { .api }
218
class SafeSearchAnnotation:
219
    """Safe search detection result."""
220
    adult: Likelihood
221
    spoof: Likelihood
222
    medical: Likelihood
223
    violence: Likelihood
224
    racy: Likelihood
225
```
226

227
### Image Properties
228

229
Image color and visual properties analysis.
230

231
```python { .api }
232
class ImageProperties:
233
    """Properties of an image."""
234
    dominant_colors: DominantColorsAnnotation
235

236
class DominantColorsAnnotation:
237
    """Dominant color information."""
238
    colors: List[ColorInfo]
239

240
class ColorInfo:
241
    """Color information for detected objects."""
242
    color: Color
243
    score: float
244
    pixel_fraction: float
245

246
class Color:
247
    """RGB color representation."""
248
    red: float
249
    green: float
250
    blue: float
251
    alpha: Optional[float]
252
```
253

254
### Crop Hints
255

256
Suggested image cropping information.
257

258
```python { .api }
259
class CropHintsAnnotation:
260
    """Collection of crop hints."""
261
    crop_hints: List[CropHint]
262

263
class CropHint:
264
    """Suggested crop for an image."""
265
    bounding_poly: BoundingPoly
266
    confidence: float
267
    importance_fraction: float
268
```
269

270
### Web Detection
271

272
Web entity and similar image detection results.
273

274
```python { .api }
275
class WebDetection:
276
    """Web detection results."""
277
    web_entities: List[WebEntity]
278
    full_matching_images: List[WebImage]
279
    partial_matching_images: List[WebImage]
280
    pages_with_matching_images: List[WebPage]
281
    visually_similar_images: List[WebImage]
282
    best_guess_labels: List[WebLabel]
283

284
class WebEntity:
285
    """Entity from the web."""
286
    entity_id: str
287
    score: float
288
    description: str
289

290
class WebImage:
291
    """Metadata for online image."""
292
    url: str
293
    score: float
294

295
class WebPage:
296
    """Metadata for web page."""
297
    url: str
298
    score: float
299
    page_title: str
300
    full_matching_images: List[WebImage]
301
    partial_matching_images: List[WebImage]
302

303
class WebLabel:
304
    """Label from web detection."""
305
    label: str
306
    language_code: str
307
```
308

309
## Geometry Types
310

311
Spatial positioning and boundary information.
312

313
```python { .api }
314
class BoundingPoly:
315
    """Bounding polygon for detected objects."""
316
    vertices: List[Vertex]
317
    normalized_vertices: List[NormalizedVertex]
318

319
class Vertex:
320
    """2D vertex coordinates."""
321
    x: int
322
    y: int
323

324
class NormalizedVertex:
325
    """Vertex coordinates normalized to [0,1]."""
326
    x: float
327
    y: float
328

329
class Position:
330
    """3D position coordinates."""
331
    x: float
332
    y: float
333
    z: float
334

335
class LatLongRect:
336
    """Latitude/longitude rectangle."""
337
    min_lat_lng: LatLng
338
    max_lat_lng: LatLng
339

340
class LatLng:
341
    """Latitude/longitude pair."""
342
    latitude: float
343
    longitude: float
344
```
345

346
## Configuration Types
347

348
### Feature Configuration
349

350
Configuration for detection features and processing options.
351

352
```python { .api }
353
class Feature:
354
    """Detection feature configuration."""
355
    type_: FeatureType
356
    max_results: int
357
    model: str
358
    
359
    class Type:
360
        """Feature type enumeration."""
361
        TYPE_UNSPECIFIED = 0
362
        FACE_DETECTION = 1
363
        LANDMARK_DETECTION = 2
364
        LOGO_DETECTION = 3
365
        LABEL_DETECTION = 4
366
        TEXT_DETECTION = 5
367
        DOCUMENT_TEXT_DETECTION = 11
368
        SAFE_SEARCH_DETECTION = 6
369
        IMAGE_PROPERTIES = 7
370
        CROP_HINTS = 9
371
        WEB_DETECTION = 10
372
        PRODUCT_SEARCH = 12
373
        OBJECT_LOCALIZATION = 19
374

375
class ImageContext:
376
    """Context and region hints for image analysis."""
377
    lat_long_rect: Optional[LatLongRect]
378
    language_hints: List[str]
379
    crop_hints_params: Optional[CropHintsParams]
380
    product_search_params: Optional[ProductSearchParams]
381
    web_detection_params: Optional[WebDetectionParams]
382
    text_detection_params: Optional[TextDetectionParams]
383

384
class CropHintsParams:
385
    """Parameters for crop hint detection."""
386
    aspect_ratios: List[float]
387

388
class ProductSearchParams:
389
    """Parameters for product search."""
390
    bounding_poly: Optional[BoundingPoly]
391
    product_set: str
392
    product_categories: List[str]
393
    filter: str
394

395
class WebDetectionParams:
396
    """Parameters for web detection."""
397
    include_geo_results: bool
398

399
class TextDetectionParams:
400
    """Parameters for text detection."""
401
    enable_text_detection_confidence_score: bool
402
    advanced_ocr_options: List[str]
403
```
404

405
### Image Input Types
406

407
Types for specifying image input sources.
408

409
```python { .api }
410
class Image:
411
    """Image data for analysis."""
412
    content: Optional[bytes]
413
    source: Optional[ImageSource]
414

415
class ImageSource:
416
    """Source location of image."""
417
    gcs_image_uri: Optional[str]
418
    image_uri: Optional[str]
419

420
class ImageAnnotationContext:
421
    """Context for image annotation."""
422
    uri: str
423
    page_number: int
424
```
425

426
## Product Search Types
427

428
### Product Catalog Types
429

430
Types for managing product catalogs and search functionality.
431

432
```python { .api }
433
class Product:
434
    """Product definition."""
435
    name: str
436
    display_name: str
437
    description: str
438
    product_category: str
439
    product_labels: List[KeyValue]
440

441
class ProductSet:
442
    """Product set definition."""
443
    name: str
444
    display_name: str
445
    index_time: Timestamp
446
    index_error: Optional[Status]
447

448
class ReferenceImage:
449
    """Reference image for products."""
450
    name: str
451
    uri: str
452
    bounding_polys: List[BoundingPoly]
453

454
class ProductSearchResults:
455
    """Results from product search."""
456
    index_time: Timestamp
457
    results: List[Result]
458
    product_grouped_results: List[GroupedResult]
459
    
460
class Result:
461
    """Individual search result."""
462
    product: Product
463
    score: float
464
    image: str
465

466
class GroupedResult:
467
    """Grouped search results."""
468
    bounding_poly: BoundingPoly
469
    results: List[Result]
470
    object_annotations: List[ObjectAnnotation]
471
```
472

473
### Product Management Request Types
474

475
Request types for product catalog management operations.
476

477
```python { .api }
478
class CreateProductSetRequest:
479
    """Request to create product set."""
480
    parent: str
481
    product_set: ProductSet
482
    product_set_id: str
483

484
class ListProductSetsRequest:
485
    """Request to list product sets."""
486
    parent: str
487
    page_size: int
488
    page_token: str
489

490
class ListProductSetsResponse:
491
    """Response from listing product sets."""
492
    product_sets: List[ProductSet]
493
    next_page_token: str
494

495
class GetProductSetRequest:
496
    """Request to get product set."""
497
    name: str
498

499
class UpdateProductSetRequest:
500
    """Request to update product set."""
501
    product_set: ProductSet
502
    update_mask: FieldMask
503

504
class DeleteProductSetRequest:
505
    """Request to delete product set."""
506
    name: str
507

508
class CreateProductRequest:
509
    """Request to create product."""
510
    parent: str
511
    product: Product
512
    product_id: str
513

514
class ListProductsRequest:
515
    """Request to list products."""
516
    parent: str
517
    page_size: int
518
    page_token: str
519

520
class ListProductsResponse:
521
    """Response from listing products."""
522
    products: List[Product]
523
    next_page_token: str
524

525
class GetProductRequest:
526
    """Request to get product."""
527
    name: str
528

529
class UpdateProductRequest:
530
    """Request to update product."""
531
    product: Product
532
    update_mask: FieldMask
533

534
class DeleteProductRequest:
535
    """Request to delete product."""
536
    name: str
537

538
class CreateReferenceImageRequest:
539
    """Request to create reference image."""
540
    parent: str
541
    reference_image: ReferenceImage
542
    reference_image_id: str
543

544
class ListReferenceImagesRequest:
545
    """Request to list reference images."""
546
    parent: str
547
    page_size: int
548
    page_token: str
549

550
class ListReferenceImagesResponse:
551
    """Response from listing reference images."""
552
    reference_images: List[ReferenceImage]
553
    page_size: int
554
    next_page_token: str
555

556
class GetReferenceImageRequest:
557
    """Request to get reference image."""
558
    name: str
559

560
class DeleteReferenceImageRequest:
561
    """Request to delete reference image."""
562
    name: str
563

564
class AddProductToProductSetRequest:
565
    """Request to add product to set."""
566
    name: str
567
    product: str
568

569
class RemoveProductFromProductSetRequest:
570
    """Request to remove product from set."""
571
    name: str
572
    product: str
573

574
class ListProductsInProductSetRequest:
575
    """Request to list products in set."""
576
    name: str
577
    page_size: int
578
    page_token: str
579

580
class ListProductsInProductSetResponse:
581
    """Response from listing products in set."""
582
    products: List[Product]
583
    next_page_token: str
584
```
585

586
### Import and Batch Operations
587

588
Types for bulk operations and data import.
589

590
```python { .api }
591
class ImportProductSetsRequest:
592
    """Request to import product sets."""
593
    parent: str
594
    input_config: ImportProductSetsInputConfig
595

596
class ImportProductSetsResponse:
597
    """Response from importing product sets."""
598
    reference_images: List[ReferenceImage]
599
    statuses: List[Status]
600

601
class ImportProductSetsInputConfig:
602
    """Input configuration for importing."""
603
    gcs_source: ImportProductSetsGcsSource
604

605
class ImportProductSetsGcsSource:
606
    """GCS source for importing product sets."""
607
    csv_file_uri: str
608

609
class PurgeProductsRequest:
610
    """Request to purge products."""
611
    parent: str
612
    product_set_purge_config: Optional[ProductSetPurgeConfig]
613
    delete_orphan_products: bool
614
    force: bool
615

616
class ProductSetPurgeConfig:
617
    """Configuration for purging product sets."""
618
    product_set_id: str
619

620
class BatchOperationMetadata:
621
    """Metadata for batch operations."""
622
    state: State
623
    submit_time: Timestamp
624
    end_time: Timestamp
625
```
626

627
## File and Storage Types
628

629
Types for file processing and cloud storage integration.
630

631
```python { .api }
632
class InputConfig:
633
    """Input configuration for batch operations."""
634
    gcs_source: Optional[GcsSource]
635
    content: Optional[bytes]
636
    mime_type: str
637

638
class OutputConfig:
639
    """Output configuration for batch operations."""
640
    gcs_destination: GcsDestination
641
    batch_size: int
642

643
class GcsSource:
644
    """Google Cloud Storage source."""
645
    uri: str
646

647
class GcsDestination:
648
    """Google Cloud Storage destination."""
649
    uri: str
650

651
class OperationMetadata:
652
    """Metadata for long-running operations."""
653
    state: OperationState
654
    create_time: Timestamp
655
    update_time: Timestamp
656
```
657

658
## Enums and Constants
659

660
### Likelihood Enumeration
661

662
Standard likelihood levels for various detections.
663

664
```python { .api }
665
class Likelihood:
666
    """Likelihood levels for detection confidence."""
667
    UNKNOWN = 0          # Unknown likelihood
668
    VERY_UNLIKELY = 1    # Very unlikely
669
    UNLIKELY = 2         # Unlikely
670
    POSSIBLE = 3         # Possible
671
    LIKELY = 4           # Likely
672
    VERY_LIKELY = 5      # Very likely
673
```
674

675
### Utility Types
676

677
Common utility types used throughout the API.
678

679
```python { .api }
680
class Property:
681
    """Generic name/value property pair."""
682
    name: str
683
    value: str
684
    uint64_value: int
685

686
class KeyValue:
687
    """Key-value pair."""
688
    key: str
689
    value: str
690

691
class LocationInfo:
692
    """Location information."""
693
    lat_lng: LatLng
694

695
class Status:
696
    """Error status information."""
697
    code: int
698
    message: str
699
    details: List[Any]
700

701
class Timestamp:
702
    """Timestamp representation."""
703
    seconds: int
704
    nanos: int
705

706
class FieldMask:
707
    """Field mask for partial updates."""
708
    paths: List[str]
709
```
710

711
## Usage Examples
712

713
### Working with Detection Results
714

715
```python
716
from google.cloud.vision import ImageAnnotatorClient, Image, Likelihood
717

718
client = ImageAnnotatorClient()
719
image = Image(source={'image_uri': 'https://example.com/photo.jpg'})
720
response = client.annotate_image({'image': image})
721

722
# Face detection results
723
for face in response.face_annotations:
724
    print(f"Face bounds: {face.bounding_poly.vertices}")
725
    print(f"Joy: {face.joy_likelihood}")
726
    print(f"Confidence: {face.detection_confidence}")
727
    
728
    # Check likelihood levels
729
    if face.joy_likelihood >= Likelihood.LIKELY:
730
        print("Happy face detected!")
731

732
# Text detection results  
733
if response.text_annotations:
734
    full_text = response.text_annotations[0].description
735
    print(f"Detected text: {full_text}")
736
    
737
    # Structured text from full_text_annotation
738
    if response.full_text_annotation:
739
        for page in response.full_text_annotation.pages:
740
            for block in page.blocks:
741
                for paragraph in block.paragraphs:
742
                    para_text = ''.join([
743
                        symbol.text for word in paragraph.words 
744
                        for symbol in word.symbols
745
                    ])
746
                    print(f"Paragraph: {para_text}")
747

748
# Label detection results
749
for label in response.label_annotations:
750
    print(f"Label: {label.description}")
751
    print(f"Score: {label.score}")
752
    print(f"Topicality: {label.topicality}")
753

754
# Object localization results
755
for obj in response.localized_object_annotations:
756
    print(f"Object: {obj.name}")
757
    print(f"Confidence: {obj.score}")
758
    vertices = obj.bounding_poly.normalized_vertices
759
    print(f"Bounds: ({vertices[0].x}, {vertices[0].y}) to ({vertices[2].x}, {vertices[2].y})")
760
```
761

762
### Creating Structured Requests
763

764
```python
765
from google.cloud.vision import (
766
    AnnotateImageRequest, Feature, Image, ImageContext, 
767
    TextDetectionParams, ProductSearchParams
768
)
769

770
# Create structured request
771
image = Image(source={'image_uri': 'https://example.com/document.jpg'})
772

773
# Configure text detection with confidence scores
774
text_params = TextDetectionParams(
775
    enable_text_detection_confidence_score=True
776
)
777

778
image_context = ImageContext(
779
    text_detection_params=text_params,
780
    language_hints=['en']
781
)
782

783
features = [
784
    Feature(type_=Feature.Type.DOCUMENT_TEXT_DETECTION, max_results=1),
785
    Feature(type_=Feature.Type.LABEL_DETECTION, max_results=10)
786
]
787

788
request = AnnotateImageRequest(
789
    image=image,
790
    features=features,
791
    image_context=image_context
792
)
793

794
response = client.annotate_image(request)
795

796
# Access confidence scores in text detection
797
if response.full_text_annotation:
798
    for page in response.full_text_annotation.pages:
799
        print(f"Page confidence: {page.confidence}")
800
        for block in page.blocks:
801
            print(f"Block confidence: {block.confidence}")
802
```
803

804
### Batch Processing with Type Safety
805

806
```python
807
from google.cloud.vision import BatchAnnotateImagesRequest
808

809
# Create multiple requests
810
requests = []
811
image_urls = [
812
    'https://example.com/image1.jpg',
813
    'https://example.com/image2.jpg', 
814
    'https://example.com/image3.jpg'
815
]
816

817
for url in image_urls:
818
    image = Image(source={'image_uri': url})
819
    request = AnnotateImageRequest(
820
        image=image,
821
        features=[Feature(type_=Feature.Type.LABEL_DETECTION, max_results=5)]
822
    )
823
    requests.append(request)
824

825
# Create batch request
826
batch_request = BatchAnnotateImagesRequest(requests=requests)
827
batch_response = client.batch_annotate_images(batch_request)
828

829
# Process results with type safety
830
for i, response in enumerate(batch_response.responses):
831
    print(f"Image {i+1} results:")
832
    if response.error:
833
        print(f"Error: {response.error.message}")
834
        continue
835
        
836
    for label in response.label_annotations:
837
        print(f"  {label.description}: {label.score}")
838
```