0
# Azure Cognitive Services Custom Vision
1

2
Microsoft Azure Custom Vision Client Library for Python provides powerful machine learning capabilities for training custom computer vision models without requiring deep ML expertise. This library enables developers to build domain-specific image classification and object detection models through Azure's cloud infrastructure, offering programmatic access to the complete machine learning pipeline from data upload to model deployment.
3

4
## Package Information
5

6
- **Package Name**: azure-cognitiveservices-vision-customvision
7
- **Language**: Python
8
- **Installation**: `pip install azure-cognitiveservices-vision-customvision`
9
- **Dependencies**: `msrest>=0.6.21`, `azure-common~=1.1`, `azure-mgmt-core>=1.2.0,<2.0.0`
10

11
## Core Imports
12

13
```python
14
from azure.cognitiveservices.vision.customvision.training import CustomVisionTrainingClient
15
from azure.cognitiveservices.vision.customvision.prediction import CustomVisionPredictionClient
16
```
17

18
Authentication:
19
```python
20
from msrest.authentication import ApiKeyCredentials
21
```
22

23
## Basic Usage
24

25
```python
26
from azure.cognitiveservices.vision.customvision.training import CustomVisionTrainingClient
27
from azure.cognitiveservices.vision.customvision.prediction import CustomVisionPredictionClient
28
from msrest.authentication import ApiKeyCredentials
29

30
# Initialize training client
31
training_key = "your-training-key"
32
training_endpoint = "https://southcentralus.api.cognitive.microsoft.com"
33
credentials = ApiKeyCredentials(in_headers={"Training-key": training_key})
34
training_client = CustomVisionTrainingClient(training_endpoint, credentials)
35

36
# Create a new project
37
project = training_client.create_project("My Custom Vision Project")
38

39
# Initialize prediction client
40
prediction_key = "your-prediction-key"
41
prediction_endpoint = "https://southcentralus.api.cognitive.microsoft.com"
42
prediction_credentials = ApiKeyCredentials(in_headers={"Prediction-key": prediction_key})
43
prediction_client = CustomVisionPredictionClient(prediction_endpoint, prediction_credentials)
44

45
# Make predictions (after model is trained and published)
46
with open("test_image.jpg", "rb") as image_data:
47
    results = prediction_client.classify_image(project.id, "published_model_name", image_data)
48
    for prediction in results.predictions:
49
        print(f"{prediction.tag_name}: {prediction.probability:.2%}")
50
```
51

52
## Architecture
53

54
The Azure Custom Vision API follows a two-client architecture that separates model development from inference:
55

56
- **Training Client**: Manages the complete ML pipeline including project creation, data management, model training, performance evaluation, and export capabilities
57
- **Prediction Client**: Handles inference operations on trained and published models for both classification and object detection tasks
58
- **Rich Model System**: Comprehensive set of data classes representing projects, iterations, tags, images, predictions, and performance metrics
59
- **Multi-Platform Export**: Support for TensorFlow, ONNX, CoreML, Docker containers, and mobile platforms
60

61
This design enables scalable workflows where models are developed using the training client and deployed for production inference through the prediction client, supporting both cloud-based and edge deployment scenarios.
62

63
## Capabilities
64

65
### Training Operations
66

67
Complete machine learning pipeline management including project creation, data upload and management, model training, performance evaluation, and export capabilities. Supports both image classification and object detection workflows with advanced features like custom domains, batch operations, and AI-assisted labeling.
68

69
```python { .api }
70
class CustomVisionTrainingClient:
71
    def __init__(self, endpoint: str, credentials): ...
72
    
73
    # Project Management
74
    def get_projects(self) -> list: ...
75
    def create_project(self, name: str, **options) -> Project: ...
76
    def get_project(self, project_id: str) -> Project: ...
77
    def delete_project(self, project_id: str): ...
78
    def update_project(self, project_id: str, updated_project: Project) -> Project: ...
79
    def export_project(self, project_id: str) -> ProjectExport: ...
80
    def import_project(self, token: str, name: str = None) -> Project: ...
81
    
82
    # Domain Management
83
    def get_domains(self) -> list: ...
84
    def get_domain(self, domain_id: str) -> Domain: ...
85
    
86
    # Training Operations
87
    def train_project(self, project_id: str, **options) -> Iteration: ...
88
    def get_iterations(self, project_id: str) -> list: ...
89
    def get_iteration(self, project_id: str, iteration_id: str) -> Iteration: ...
90
    def delete_iteration(self, project_id: str, iteration_id: str): ...
91
    def update_iteration(self, project_id: str, iteration_id: str, name: str) -> Iteration: ...
92
    
93
    # Image Management
94
    def create_images_from_data(self, project_id: str, image_data: bytes, tag_ids: list = None) -> ImageCreateSummary: ...
95
    def create_images_from_files(self, project_id: str, batch: ImageFileCreateBatch) -> ImageCreateSummary: ...
96
    def create_images_from_urls(self, project_id: str, batch: ImageUrlCreateBatch) -> ImageCreateSummary: ...
97
    def create_images_from_predictions(self, project_id: str, batch: ImageIdCreateBatch) -> ImageCreateSummary: ...
98
    def get_images(self, project_id: str, **options) -> list: ...
99
    def get_images_by_ids(self, project_id: str, image_ids: list, iteration_id: str = None) -> list: ...
100
    def get_tagged_images(self, project_id: str, **options) -> list: ...
101
    def get_untagged_images(self, project_id: str, **options) -> list: ...
102
    def delete_images(self, project_id: str, image_ids: list = None, **options): ...
103
    def update_image_metadata(self, project_id: str, image_ids: list, metadata: dict) -> ImageMetadataUpdateSummary: ...
104
    def get_image_count(self, project_id: str, **options) -> int: ...
105
    def get_tagged_image_count(self, project_id: str, **options) -> int: ...
106
    def get_untagged_image_count(self, project_id: str, **options) -> int: ...
107
    
108
    # Tag Management
109
    def get_tags(self, project_id: str, iteration_id: str = None) -> list: ...
110
    def create_tag(self, project_id: str, name: str, **options) -> Tag: ...
111
    def get_tag(self, project_id: str, tag_id: str, iteration_id: str = None) -> Tag: ...
112
    def delete_tag(self, project_id: str, tag_id: str): ...
113
    def update_tag(self, project_id: str, tag_id: str, updated_tag: Tag) -> Tag: ...
114
    
115
    # Performance Analysis
116
    def get_iteration_performance(self, project_id: str, iteration_id: str, **options) -> IterationPerformance: ...
117
    def get_image_performances(self, project_id: str, iteration_id: str, **options) -> list: ...
118
    def get_image_performance_count(self, project_id: str, iteration_id: str, **options) -> int: ...
119
    
120
    # Publishing
121
    def publish_iteration(self, project_id: str, iteration_id: str, publish_name: str, prediction_id: str, **options) -> bool: ...
122
    def unpublish_iteration(self, project_id: str, iteration_id: str): ...
123
    
124
    # Export Management
125
    def get_exports(self, project_id: str, iteration_id: str) -> list: ...
126
    def export_iteration(self, project_id: str, iteration_id: str, platform: str, **options) -> Export: ...
127
    
128
    # Quick Testing
129
    def quick_test_image(self, project_id: str, image_data: bytes, **options) -> ImagePrediction: ...
130
    def quick_test_image_url(self, project_id: str, url: str, **options) -> ImagePrediction: ...
131
    
132
    # Additional Operations (see training.md for complete signatures)
133
    # Image regions, tagging, predictions, suggestions, and more...
134
```
135

136
[Training Operations](./training.md)
137

138
### Prediction Operations
139

140
Inference operations for trained custom vision models supporting both image classification and object detection. Provides multiple prediction methods with options for storing prediction results or running predictions without storage.
141

142
```python { .api }
143
class CustomVisionPredictionClient:
144
    def __init__(self, endpoint: str, credentials): ...
145
    
146
    # Image Classification (with storage)
147
    def classify_image(self, project_id: str, published_name: str, image_data: bytes, application: str = None) -> ImagePrediction: ...
148
    def classify_image_url(self, project_id: str, published_name: str, url: str, application: str = None) -> ImagePrediction: ...
149
    
150
    # Image Classification (no storage)
151
    def classify_image_with_no_store(self, project_id: str, published_name: str, image_data: bytes, application: str = None) -> ImagePrediction: ...
152
    def classify_image_url_with_no_store(self, project_id: str, published_name: str, url: str, application: str = None) -> ImagePrediction: ...
153
    
154
    # Object Detection (with storage)
155
    def detect_image(self, project_id: str, published_name: str, image_data: bytes, application: str = None) -> ImagePrediction: ...
156
    def detect_image_url(self, project_id: str, published_name: str, url: str, application: str = None) -> ImagePrediction: ...
157
    
158
    # Object Detection (no storage)
159
    def detect_image_with_no_store(self, project_id: str, published_name: str, image_data: bytes, application: str = None) -> ImagePrediction: ...
160
    def detect_image_url_with_no_store(self, project_id: str, published_name: str, url: str, application: str = None) -> ImagePrediction: ...
161
```
162

163
[Prediction Operations](./prediction.md)
164

165
## Core Types
166

167
```python { .api }
168
class Project:
169
    """Custom Vision project with settings and metadata."""
170
    id: str
171
    name: str
172
    description: str
173
    domain_id: str
174
    classification_type: str
175
    target_export_platforms: list
176
    created: datetime
177
    last_modified: datetime
178
    thumbnail_uri: str
179
    drain_mode: str
180
    image_processing_settings: dict
181
    is_flagship: bool
182
    status: str
183

184
class ImagePrediction:
185
    """Main prediction result containing predictions and metadata."""
186
    id: str
187
    project: str
188
    iteration: str
189
    created: datetime
190
    predictions: list
191

192
class Prediction:
193
    """Individual prediction with tag, probability, and optional bounding box."""
194
    probability: float
195
    tag_id: str
196
    tag_name: str
197
    bounding_box: BoundingBox = None
198

199
class BoundingBox:
200
    """Image region coordinates for object detection."""
201
    left: float
202
    top: float
203
    width: float
204
    height: float
205

206
class Iteration:
207
    """Trained model iteration with performance metrics."""
208
    id: str
209
    name: str
210
    created: datetime
211
    last_modified: datetime
212
    trained_at: datetime
213
    status: str
214
    is_default: bool
215
    domain_id: str
216
    classification_type: str
217
    platform: str
218
    export_model_container_uri: str
219
    reserved_budget_in_hours: int
220
    publish_name: str
221

222
class Tag:
223
    """Image classification tag."""
224
    id: str
225
    name: str
226
    description: str
227
    image_count: int
228
    type: str
229

230
class Domain:
231
    """Available domain for different model types."""
232
    id: str
233
    name: str
234
    type: str
235
    exportable: bool
236
    enabled: bool
237

238
class CustomVisionErrorException(Exception):
239
    """Exception raised for all API operation failures."""
240
    error: CustomVisionError
241

242
class CustomVisionError:
243
    """Detailed error information."""
244
    code: str
245
    message: str
246

247
# Training-specific Model Classes
248
class ImageCreateSummary:
249
    """Summary of image creation operation."""
250
    is_batch_successful: bool
251
    images: list
252
    duplicate_count: int
253

254
class ImageFileCreateBatch:
255
    """Batch for creating images from files."""
256
    images: list
257
    tag_ids: list
258

259
class ImageFileCreateEntry:
260
    """Single file entry for batch creation."""
261
    name: str
262
    contents: bytes
263
    tag_ids: list
264
    regions: list
265

266
class ImageUrlCreateBatch:
267
    """Batch for creating images from URLs."""
268
    images: list
269
    tag_ids: list
270

271
class ImageUrlCreateEntry:
272
    """Single URL entry for batch creation."""
273
    url: str
274
    tag_ids: list
275
    regions: list
276

277
class ImageIdCreateBatch:
278
    """Batch for creating images from predictions."""
279
    images: list
280
    tag_ids: list
281

282
class ImageIdCreateEntry:
283
    """Single prediction entry for batch creation."""
284
    id: str
285
    tag_ids: list
286
    regions: list
287

288
class Image:
289
    """Image information with metadata."""
290
    id: str
291
    created: datetime
292
    width: int
293
    height: int
294
    image_uri: str
295
    thumbnail_uri: str
296
    metadata: dict
297
    tags: list
298
    regions: list
299
    predictions: list
300

301
class ImageRegion:
302
    """Image region information."""
303
    region_id: str
304
    tag_name: str
305
    created: datetime
306
    tag_id: str
307
    left: float
308
    top: float
309
    width: float
310
    height: float
311

312
class ImageRegionCreateBatch:
313
    """Batch for creating regions."""
314
    regions: list
315

316
class ImageRegionCreateEntry:
317
    """Single region entry for batch creation."""
318
    image_id: str
319
    tag_id: str
320
    left: float
321
    top: float
322
    width: float
323
    height: float
324

325
class ImageRegionCreateSummary:
326
    """Region creation summary."""
327
    created: list
328
    duplicated: list
329
    exceeded: list
330

331
class ImageRegionProposal:
332
    """Region proposal."""
333
    project_id: str
334
    image_id: str
335
    proposals: list
336

337
class ImageTag:
338
    """Image tag information."""
339
    image_id: str
340
    tag_id: str
341
    created: datetime
342

343
class ImageTagCreateBatch:
344
    """Batch for creating tags."""
345
    tags: list
346

347
class ImageTagCreateEntry:
348
    """Single tag entry for batch creation."""
349
    image_id: str
350
    tag_id: str
351

352
class ImageTagCreateSummary:
353
    """Tag creation summary."""
354
    created: list
355
    duplicated: list
356
    exceeded: list
357

358
class ImageMetadataUpdateEntry:
359
    """Metadata update entry."""
360
    image_id: str
361
    metadata: dict
362

363
class ImageMetadataUpdateSummary:
364
    """Metadata update summary."""
365
    updated: list
366
    limit_exceeded: list
367
    errors: list
368

369
class Export:
370
    """Export details."""
371
    platform: str
372
    status: str
373
    download_uri: str
374
    flavor: str
375
    newer_version_available: bool
376

377
class ProjectExport:
378
    """Project export information."""
379
    token: str
380

381
class ProjectSettings:
382
    """Project settings."""
383
    domain_id: str
384
    classification_type: str
385
    target_export_platforms: list
386
    use_negative_set: bool
387
    detection_parameters: str
388
    image_processing_settings: dict
389

390
class ImageProcessingSettings:
391
    """Image processing settings."""
392
    augmentation_methods: dict
393

394
class IterationPerformance:
395
    """Iteration performance metrics."""
396
    per_tag_performance: list
397
    precision: float
398
    precision_std_deviation: float
399
    recall: float
400
    recall_std_deviation: float
401
    average_precision: float
402

403
class TagPerformance:
404
    """Tag performance metrics."""
405
    id: str
406
    name: str
407
    precision: float
408
    precision_std_deviation: float
409
    recall: float
410
    recall_std_deviation: float
411
    average_precision: float
412

413
class ImagePerformance:
414
    """Image performance metrics."""
415
    id: str
416
    created: datetime
417
    width: int
418
    height: int
419
    image_uri: str
420
    thumbnail_uri: str
421
    predictions: list
422
    tags: list
423

424
class PredictionQueryResult:
425
    """Prediction query result."""
426
    results: list
427
    token: dict
428

429
class PredictionQueryToken:
430
    """Prediction query token."""
431
    session: str
432
    continuation: str
433
    max_count: int
434
    order_by: str
435
    tags: list
436
    iteration_id: str
437
    start_time: datetime
438
    end_time: datetime
439
    application: str
440

441
class PredictionQueryTag:
442
    """Prediction query tag."""
443
    id: str
444
    min_threshold: float
445
    max_threshold: float
446

447
class StoredImagePrediction:
448
    """Stored image prediction."""
449
    id: str
450
    project: str
451
    iteration: str
452
    created: datetime
453
    predictions: list
454
    image_uri: str
455
    thumbnail_uri: str
456

457
class SuggestedTagAndRegion:
458
    """Tag and region suggestion."""
459
    id: str
460
    tag: dict
461
    region: dict
462
    confidence: float
463

464
class SuggestedTagAndRegionQuery:
465
    """Suggestion query result."""
466
    results: list
467
    token: dict
468

469
class SuggestedTagAndRegionQueryToken:
470
    """Suggestion query token."""
471
    session: str
472
    continuation: str
473
    max_count: int
474
    order_by: str
475
    tag_ids: list
476
    threshold: float
477

478
class StoredSuggestedTagAndRegion:
479
    """Stored suggestion."""
480
    id: str
481
    tag_id: str
482
    region: dict
483
    confidence: float
484

485
class TagFilter:
486
    """Tag filter criteria."""
487
    tag_id: str
488
    min_threshold: float
489
    max_threshold: float
490

491
class ModelInformation:
492
    """Model information."""
493
    model_id: str
494
    model_name: str
495
    created_date: datetime
496
    domain_id: str
497

498
class CustomBaseModelInfo:
499
    """Custom base model information."""
500
    name: str
501
    version: int
502

503
class TrainingParameters:
504
    """Training parameters."""
505
    training_type: str
506
    reserved_budget_in_hours: int
507
    force_train: bool
508
    notification_email_address: str
509
    selected_tags: list
510
    custom_base_model_info: dict
511

512
class Region:
513
    """Region information."""
514
    left: float
515
    top: float
516
    width: float
517
    height: float
518

519
class RegionProposal:
520
    """Region proposal."""
521
    confidence: float
522
    bounding_box: dict
523

524
class ImageUrl:
525
    """Image URL information."""
526
    url: str
527
```
528

529
## Constants and Enumerations
530

531
```python { .api }
532
# Classification Types
533
class Classifier:
534
    MULTICLASS = "Multiclass"
535
    MULTILABEL = "Multilabel"
536

537
# Training Types  
538
class TrainingType:
539
    REGULAR = "Regular"
540
    ADVANCED = "Advanced"
541

542
# Export Platforms
543
class ExportPlatform:
544
    COREML = "CoreML"
545
    TENSORFLOW = "TensorFlow"
546
    DOCKERFILE = "DockerFile"
547
    ONNX = "ONNX"
548
    VAIDK = "VAIDK"
549
    OPENVINO = "OpenVino"
550

551
# Export Status
552
class ExportStatus:
553
    EXPORTING = "Exporting"
554
    FAILED = "Failed"
555
    DONE = "Done"
556

557
# Domain Types
558
class DomainType:
559
    CLASSIFICATION = "Classification"
560
    OBJECT_DETECTION = "ObjectDetection"
561

562
# Tag Types
563
class TagType:
564
    REGULAR = "Regular"
565
    NEGATIVE = "Negative"
566
    GENERAL_PRODUCT = "GeneralProduct"
567

568
# Export Flavors
569
class ExportFlavor:
570
    LINUX = "Linux"
571
    WINDOWS = "Windows"
572
    ONNX10 = "ONNX10"
573
    ONNX12 = "ONNX12"
574
    ARM = "ARM"
575
    TENSOR_FLOW_NORMAL = "TensorFlowNormal"
576
    TENSOR_FLOW_LITE = "TensorFlowLite"
577

578
# Image Creation Status
579
class ImageCreateStatus:
580
    OK = "OK"
581
    OK_DUPLICATE = "OKDuplicate"
582
    ERROR_SOURCE = "ErrorSource"
583
    ERROR_IMAGE_FORMAT = "ErrorImageFormat"
584
    ERROR_IMAGE_SIZE_BYTES = "ErrorImageSizeBytes"
585
    ERROR_STORAGE = "ErrorStorage"
586
    ERROR_LIMIT_EXCEED = "ErrorLimitExceed"
587
    ERROR_TAG_LIMIT_EXCEED = "ErrorTagLimitExceed"
588
    ERROR_REGION_LIMIT_EXCEED = "ErrorRegionLimitExceed"
589
    ERROR_UNKNOWN = "ErrorUnknown"
590

591
# Image Metadata Update Status
592
class ImageMetadataUpdateStatus:
593
    OK = "OK"
594
    ERROR_IMAGE_NOT_FOUND = "ErrorImageNotFound"
595
    ERROR_LIMIT_EXCEED = "ErrorLimitExceed"
596
    ERROR_UNKNOWN = "ErrorUnknown"
597

598
# Project Status
599
class ProjectStatus:
600
    SUCCEEDED = "Succeeded"
601
    IMPORTING = "Importing"
602
    FAILED = "Failed"
603

604
# Order By Options
605
class OrderBy:
606
    NEWEST = "Newest"
607
    OLDEST = "Oldest"
608
    SUGGESTED = "Suggested"
609

610
# Sort By Options
611
class SortBy:
612
    UNCERTAINTY_ASCENDING = "UncertaintyAscending"
613
    UNCERTAINTY_DESCENDING = "UncertaintyDescending"
614

615
# Custom Vision Error Codes (major categories)
616
class CustomVisionErrorCodes:
617
    NO_ERROR = "NoError"
618
    BAD_REQUEST = "BadRequest"
619
    BAD_REQUEST_EXPORT_PLATFORM_NOT_SUPPORTED = "BadRequestExportPlatformNotSupported"
620
    BAD_REQUEST_INVALID_IDS = "BadRequestInvalidIds"
621
    BAD_REQUEST_INVALID_IMPORT_TOKEN = "BadRequestInvalidImportToken"
622
    BAD_REQUEST_LIMITED_TRIAL_MODEL = "BadRequestLimitedTrialModel"
623
    BAD_REQUEST_NOT_LIMITED_TRIAL = "BadRequestNotLimitedTrial"
624
    BAD_REQUEST_PARAMETER_MISSING = "BadRequestParameterMissing"
625
    BAD_REQUEST_PROJECT_NOT_TRAINED = "BadRequestProjectNotTrained"
626
    BAD_REQUEST_PROJECT_TRAINED = "BadRequestProjectTrained"
627
    BAD_REQUEST_REGION_PROPOSAL_NOT_SUPPORTED = "BadRequestRegionProposalNotSupported"
628
    BAD_REQUEST_TRAINING_NOT_NEEDED = "BadRequestTrainingNotNeeded"
629
    BAD_REQUEST_UNSUPPORTED_DOMAIN = "BadRequestUnsupportedDomain"
630
    BAD_REQUEST_UNSUPPORTED_EXPORT_PLATFORM = "BadRequestUnsupportedExportPlatform"
631
    CONFLICT = "Conflict"
632
    FORBIDDEN = "Forbidden"
633
    FORBIDDEN_DOMAIN_NOT_SUPPORTED_FOR_ADVANCEDTRAINING = "ForbiddenDomainNotSupportedForAdvancedTraining"
634
    FORBIDDEN_EXPORT_VALIDATION_FAILED = "ForbiddenExportValidationFailed"
635
    FORBIDDEN_INVALID_SUBSCRIPTION_API = "ForbiddenInvalidSubscriptionApi"
636
    FORBIDDEN_LIMITED_TRIAL_UPGRADE_REQUIRED = "ForbiddenLimitedTrialUpgradeRequired"
637
    FORBIDDEN_MULTI_CLASS_LABEL_LIMIT_EXCEEDED = "ForbiddenMultiClassLabelLimitExceeded"
638
    FORBIDDEN_MULTI_LABEL_LABEL_LIMIT_EXCEEDED = "ForbiddenMultiLabelLabelLimitExceeded"
639
    FORBIDDEN_PROJECT_EXISTS = "ForbiddenProjectExists"
640
    FORBIDDEN_QUERY_TOO_LONG = "ForbiddenQueryTooLong"
641
    FORBIDDEN_USER_RESOURCE_ACCESS = "ForbiddenUserResourceAccess"
642
    INTERNAL_SERVER_ERROR = "InternalServerError"
643
    NOT_FOUND = "NotFound"
644
    NOT_SUPPORTED = "NotSupported"
645
    QUOTA_EXCEEDED = "QuotaExceeded"
646
    REQUEST_ENTITY_TOO_LARGE = "RequestEntityTooLarge"
647
    REQUEST_TIMEOUT = "RequestTimeout"
648
    TOO_MANY_REQUESTS = "TooManyRequests"
649
    UNAUTHORIZED = "Unauthorized"
650
    UNSPECIFIED = "Unspecified"
651
    UNSUPPORTED_MEDIA_TYPE = "UnsupportedMediaType"
652
    ERROR_INVALID = "ErrorInvalid"
653
    # ... and 120+ additional specific error codes
654
```