0
# Metadata and Configuration Streams
1

2
This section covers streams that extract Pipedrive configuration metadata, custom field definitions, and system configuration data.
3

4
## Overview
5

6
Metadata streams provide configuration information, custom field definitions, and system setup data from Pipedrive. These streams typically use full refresh synchronization as the configuration data changes infrequently.
7

8
## Field Definition Streams
9

10
These streams extract custom field definitions for different entity types in Pipedrive.
11

12
### Deal Fields
13

14
Extracts custom field definitions for deals.
15

16
```yaml { .api }
17
deal_fields:
18
  type: DeclarativeStream
19
  name: deal_fields
20
  retriever:
21
    type: SimpleRetriever
22
    requester:
23
      $ref: "#/definitions/base_requester"
24
      path: v1/dealFields
25
      http_method: GET
26
      request_parameters:
27
        api_token: "{{ config['api_token'] }}"
28
        limit: "50"
29
    record_selector:
30
      type: RecordSelector
31
      extractor:
32
        type: DpathExtractor
33
        field_path: ["data"]
34
    paginator:
35
      type: DefaultPaginator
36
      page_token_option:
37
        type: RequestOption
38
        inject_into: request_parameter
39
        field_name: start
40
      pagination_strategy:
41
        type: CursorPagination
42
        cursor_value: "{{ response['additional_data']['pagination']['next_start'] }}"
43
  schema_loader:
44
    type: InlineSchemaLoader
45
    schema:
46
      $ref: "#/schemas/deal_fields"
47
```
48

49
**Sync Modes**: full_refresh
50
**API Endpoint**: `v1/dealFields`
51

52
### Organization Fields
53

54
Extracts custom field definitions for organizations.
55

56
```yaml { .api }
57
organization_fields:
58
  type: DeclarativeStream
59
  name: organization_fields
60
  retriever:
61
    requester:
62
      path: v1/organizationFields
63
```
64

65
**Sync Modes**: full_refresh
66
**API Endpoint**: `v1/organizationFields`
67

68
### Organizations
69

70
Extracts organization/company data from Pipedrive.
71

72
```yaml { .api }
73
organizations:
74
  type: DeclarativeStream
75
  name: organizations
76
  primary_key: [id]  
77
  retriever:
78
    requester:
79
      path: v1/organizations
80
      request_parameters:
81
        items: organization
82
    record_selector:
83
      extractor:
84
        type: DpathExtractor
85
        field_path: ["data"]
86
```
87

88
**Sync Modes**: full_refresh
89
**API Endpoint**: `v1/organizations?items=organization`
90

91
### Person Fields
92

93
Extracts custom field definitions for persons/contacts.
94

95
```yaml { .api }
96
person_fields:
97
  type: DeclarativeStream
98
  name: person_fields
99
  retriever:
100
    requester:
101
      path: v1/personFields
102
```
103

104
**Sync Modes**: full_refresh
105
**API Endpoint**: `v1/personFields`
106

107
### Activity Fields
108

109
Extracts custom field definitions for activities.
110

111
```yaml { .api }
112
activity_fields:
113
  type: DeclarativeStream
114
  name: activity_fields
115
  retriever:
116
    requester:
117
      path: v1/activityFields
118
```
119

120
**Sync Modes**: full_refresh
121
**API Endpoint**: `v1/activityFields`
122

123
### Product Fields
124

125
Extracts custom field definitions for products.
126

127
```yaml { .api }
128
product_fields:
129
  type: DeclarativeStream
130
  name: product_fields
131
  retriever:
132
    requester:
133
      path: v1/productFields
134
```
135

136
**Sync Modes**: full_refresh
137
**API Endpoint**: `v1/productFields`
138

139
## System Configuration Streams
140

141
These streams extract system-level configuration and metadata.
142

143
### Activity Types
144

145
Extracts available activity types configured in Pipedrive.
146

147
```yaml { .api }
148
activity_types:
149
  type: DeclarativeStream
150
  name: activity_types
151
  primary_key: [id]
152
  retriever:
153
    type: SimpleRetriever
154
    requester:
155
      path: v1/activityTypes
156
      request_parameters:
157
        api_token: "{{ config['api_token'] }}"
158
    record_selector:
159
      type: RecordSelector
160
      extractor:
161
        type: DpathExtractor
162
        field_path: ["data"]
163
```
164

165
**Sync Modes**: full_refresh
166
**API Endpoint**: `v1/activityTypes`
167

168
### Pipelines
169

170
Extracts sales pipeline configurations.
171

172
```yaml { .api }
173
pipelines:
174
  type: DeclarativeStream
175
  name: pipelines
176
  primary_key: [id]
177
  retriever:
178
    requester:
179
      path: v1/pipelines
180
    record_selector:
181
      extractor:
182
        type: CustomRecordExtractor
183
        class_name: source_declarative_manifest.components.NullCheckedDpathExtractor
184
        field_path: ["data", "*"]
185
        nullable_nested_field: data
186
```
187

188
**Sync Modes**: full_refresh
189
**API Endpoint**: `v1/pipelines`
190

191
### Stages
192

193
Extracts pipeline stage configurations.
194

195
```yaml { .api }
196
stages:
197
  type: DeclarativeStream
198
  name: stages
199
  primary_key: [id]
200
  retriever:
201
    requester:
202
      path: v1/stages
203
    record_selector:
204
      extractor:
205
        type: CustomRecordExtractor
206
        class_name: source_declarative_manifest.components.NullCheckedDpathExtractor
207
        field_path: ["data", "*"]
208
        nullable_nested_field: data
209
```
210

211
**Sync Modes**: full_refresh
212
**API Endpoint**: `v1/stages`
213

214
### Users
215

216
Extracts user account information and permissions.
217

218
```yaml { .api }
219
users:
220
  type: DeclarativeStream
221
  name: users
222
  primary_key: [id]
223
  retriever:
224
    requester:
225
      path: v1/recents
226
      request_parameters:
227
        items: user
228
    record_selector:
229
      extractor:
230
        type: DpathExtractor
231
        field_path: ["data", "*", "data", "*"]
232
```
233

234
**Sync Modes**: full_refresh
235
**API Endpoint**: `v1/recents?items=user`
236

237
### Roles
238

239
Extracts user role definitions and permissions.
240

241
```yaml { .api }
242
roles:
243
  type: DeclarativeStream
244
  name: roles
245
  primary_key: [id]
246
  retriever:
247
    requester:
248
      path: v1/roles
249
```
250

251
**Sync Modes**: full_refresh
252
**API Endpoint**: `v1/roles`
253

254
### Permission Sets
255

256
Extracts permission set configurations.
257

258
```yaml { .api }
259
permission_sets:
260
  type: DeclarativeStream
261
  name: permission_sets
262
  primary_key: [id]
263
  retriever:
264
    requester:
265
      path: v1/permissionSets
266
```
267

268
**Sync Modes**: full_refresh
269
**API Endpoint**: `v1/permissionSets`
270

271
### Currencies
272

273
Extracts supported currency configurations.
274

275
```yaml { .api }
276
currencies:
277
  type: DeclarativeStream
278
  name: currencies
279
  primary_key: [code]
280
  retriever:
281
    requester:
282
      path: v1/currencies
283
```
284

285
**Sync Modes**: full_refresh
286
**API Endpoint**: `v1/currencies`
287

288
### Lead Labels
289

290
Extracts lead label configurations for lead management.
291

292
```yaml { .api }
293
lead_labels:
294
  type: DeclarativeStream
295
  name: lead_labels
296
  primary_key: [id]
297
  retriever:
298
    requester:
299
      path: v1/leadLabels
300
```
301

302
**Sync Modes**: full_refresh
303
**API Endpoint**: `v1/leadLabels`
304

305
### Goals
306

307
Extracts goal and target configurations.
308

309
```yaml { .api }
310
goals:
311
  type: DeclarativeStream
312
  name: goals
313
  primary_key: [id]
314
  retriever:
315
    requester:
316
      path: v1/goals
317
```
318

319
**Sync Modes**: full_refresh
320
**API Endpoint**: `v1/goals`
321

322
### Filters
323

324
Extracts saved filter configurations.
325

326
```yaml { .api }
327
filters:
328
  type: DeclarativeStream
329
  name: filters
330
  primary_key: [id]
331
  retriever:
332
    requester:
333
      path: v1/filters
334
    record_selector:
335
      extractor:
336
        type: CustomRecordExtractor
337
        class_name: source_declarative_manifest.components.NullCheckedDpathExtractor
338
        field_path: ["data", "*"]
339
        nullable_nested_field: data
340
```
341

342
**Sync Modes**: full_refresh
343
**API Endpoint**: `v1/filters`
344

345
## Common Configuration
346

347
All metadata streams share these characteristics:
348

349
### Base Request Pattern
350
```yaml { .api }
351
retriever:
352
  type: SimpleRetriever
353
  requester:
354
    $ref: "#/definitions/base_requester"
355
    path: "v1/{endpoint}"
356
    http_method: GET
357
    request_parameters:
358
      api_token: "{{ config['api_token'] }}"
359
      limit: "50"
360
```
361

362
### Pagination
363
```yaml { .api }
364
paginator:
365
  type: DefaultPaginator
366
  page_token_option:
367
    type: RequestOption
368
    inject_into: request_parameter
369
    field_name: start
370
  pagination_strategy:
371
    type: CursorPagination
372
    cursor_value: "{{ response['additional_data']['pagination']['next_start'] }}"
373
```
374

375
### Data Extraction
376

377
Most metadata streams use the standard `DpathExtractor`:
378
```yaml { .api }
379
record_selector:
380
  type: RecordSelector
381
  extractor:
382
    type: DpathExtractor
383
    field_path: ["data"]
384
```
385

386
Some streams (pipelines, stages, filters) use `NullCheckedDpathExtractor` to handle potential null data fields.
387

388
## Schema Loading
389

390
All metadata streams use inline schema definitions:
391
```yaml { .api }
392
schema_loader:
393
  type: InlineSchemaLoader
394
  schema:
395
    $ref: "#/schemas/{stream_name}"
396
```
397

398
Schemas are defined in the `schemas` section of the manifest file and specify the expected structure of each metadata stream's records.