0
# Configuration and Data Management
1

2
System for managing dust map data storage, downloading map files, and configuring package behavior. dustmaps provides centralized configuration management and standardized data fetching across all dust map implementations.
3

4
## Capabilities
5

6
### Configuration Management
7

8
Global configuration system for managing data directories, API endpoints, and package behavior.
9

10
```python { .api }
11
class Configuration:
12
    def __init__(self, fname):
13
        """
14
        Initialize configuration from file.
15

16
        Parameters:
17
        - fname (str): Path to configuration file
18
        """
19

20
    def __getitem__(self, key):
21
        """
22
        Get configuration value using dictionary syntax.
23

24
        Parameters:
25
        - key (str): Configuration key
26

27
        Returns:
28
        - str | float | bool: Configuration value
29
        """
30

31
    def __setitem__(self, key, value):
32
        """
33
        Set configuration value using dictionary syntax.
34

35
        Parameters:
36
        - key (str): Configuration key
37
        - value (str | float | bool): Configuration value
38
        """
39

40
    def get(self, key, default=None):
41
        """
42
        Get configuration value with default fallback.
43

44
        Parameters:
45
        - key (str): Configuration key
46
        - default: Default value if key not found
47

48
        Returns:
49
        - Configuration value or default
50
        """
51

52
    def load(self):
53
        """Load configuration from file."""
54

55
    def save(self, force=False):
56
        """
57
        Save configuration to file.
58

59
        Parameters:
60
        - force (bool): Overwrite existing file without confirmation
61
        """
62

63
    def remove(self, key):
64
        """
65
        Remove configuration key.
66

67
        Parameters:
68
        - key (str): Configuration key to remove
69
        """
70

71
    def reset(self):
72
        """Reset configuration to default values."""
73

74
# Global configuration instance
75
config: Configuration
76
```
77

78
**Usage Example:**
79

80
```python
81
from dustmaps.config import config
82

83
# Set data directory for large map files
84
config['data_dir'] = '/path/to/large/data/directory'
85

86
# Get current data directory
87
data_dir = config['data_dir']
88
print(f"Data directory: {data_dir}")
89

90
# Get with default value
91
api_timeout = config.get('api_timeout', 30)
92

93
# Save configuration to file
94
config.save()
95
```
96

97
### Configuration Options
98

99
```python { .api }
100
# Standard configuration keys
101
ConfigKeys = {
102
    'data_dir': str,      # Directory for storing large map data files
103
    'api_timeout': float, # Web API request timeout (seconds)
104
    'max_memory': float,  # Maximum memory usage for map loading (GB)
105
    'cache_enabled': bool # Enable/disable local caching
106
}
107
```
108

109
### Exception Classes
110

111
Configuration-related exceptions for error handling.
112

113
```python { .api }
114
class ConfigError(Exception):
115
    """Exception raised for configuration errors."""
116

117
class ConfigWarning(UserWarning):
118
    """Warning issued for configuration issues."""
119
```
120

121
### Path Management
122

123
Utilities for managing file paths and directories used by dustmaps.
124

125
```python { .api }
126
def fix_path(path):
127
    """
128
    Expand user home and environment variables in file paths.
129

130
    Parameters:
131
    - path (str): File path potentially containing ~ or environment variables
132

133
    Returns:
134
    - str: Expanded absolute path
135
    """
136

137
def data_dir():
138
    """
139
    Get the configured data directory for storing large map files.
140

141
    Returns:
142
    - str: Absolute path to data directory
143
    """
144

145
def output_dir():
146
    """
147
    Get directory for temporary output files.
148

149
    Returns:
150
    - str: Absolute path to output directory
151
    """
152

153
# Module-level path constants
154
script_dir: str           # dustmaps package installation directory
155
data_dir_default: str     # Default data directory
156
test_dir: str            # Test files directory
157
output_dir_default: str   # Default output directory
158
```
159

160
### Data Fetching Utilities
161

162
Common utilities for downloading and validating dust map data files.
163

164
```python { .api }
165
def get_md5sum(fname, chunk_size=1024):
166
    """
167
    Calculate MD5 checksum of a file.
168

169
    Parameters:
170
    - fname (str): Path to file
171
    - chunk_size (int): Size of chunks to read (bytes)
172

173
    Returns:
174
    - str: MD5 checksum as hexadecimal string
175
    """
176

177
def h5_file_exists(fname, size_guess=None, rtol=0.1, atol=1.0, dsets={}):
178
    """
179
    Check if HDF5 file exists and has expected properties.
180

181
    Parameters:
182
    - fname (str): Path to HDF5 file
183
    - size_guess (int, optional): Expected file size in bytes
184
    - rtol (float): Relative tolerance for size check
185
    - atol (float): Absolute tolerance for size check (MB)
186
    - dsets (dict): Expected datasets and their properties
187

188
    Returns:
189
    - bool: True if file exists and meets requirements
190
    """
191

192
def dataverse_download_doi(doi, local_fname, file_requirements={}):
193
    """
194
    Download file from Harvard Dataverse using DOI.
195

196
    Parameters:
197
    - doi (str): Digital Object Identifier for the dataset
198
    - local_fname (str): Local filename to save downloaded file
199
    - file_requirements (dict): Expected file properties for validation
200

201
    Raises:
202
    - DownloadError: If download fails or file validation fails
203
    """
204

205
class DownloadError(Exception):
206
    """Exception raised during data download operations."""
207

208
# Dataverse base URL
209
dataverse: str = 'https://dataverse.harvard.edu'
210
```
211

212
### Individual Map Fetch Functions
213

214
Each dust map module provides a standardized `fetch()` function for downloading map data.
215

216
```python { .api }
217
# Available in dustmaps.sfd
218
def fetch():
219
    """Download SFD'98 dust map data (north and south galactic pole files)."""
220

221
# Available in dustmaps.bayestar
222
def fetch(version='bayestar2019'):
223
    """
224
    Download Bayestar 3D dust map data.
225

226
    Parameters:
227
    - version (str): Map version ('bayestar2015', 'bayestar2017', 'bayestar2019')
228
    """
229

230
# Available in dustmaps.planck
231
def fetch(which='2013'):
232
    """
233
    Download Planck dust map data.
234

235
    Parameters:
236
    - which (str): Map version ('2013' or 'GNILC')
237
    """
238

239
# Available in dustmaps.marshall
240
def fetch(clobber=False):
241
    """
242
    Download Marshall et al. 2006 dust map data.
243

244
    Parameters:
245
    - clobber (bool): Overwrite existing files
246
    """
247

248
# Available in dustmaps.iphas
249
def fetch(clobber=False):
250
    """
251
    Download IPHAS dust map data.
252

253
    Parameters:
254
    - clobber (bool): Overwrite existing files
255
    """
256

257
# Similar fetch() functions available in:
258
# dustmaps.csfd, dustmaps.chen2014, dustmaps.chen2018,
259
# dustmaps.lenz2017, dustmaps.pg2010, dustmaps.leike_ensslin_2019,
260
# dustmaps.leike2020, dustmaps.edenhofer2023, dustmaps.gaia_tge,
261
# dustmaps.decaps
262
```
263

264
## Configuration Workflow
265

266
### Initial Setup
267

268
```python
269
from dustmaps.config import config
270

271
# Configure data directory (first-time setup)
272
config['data_dir'] = '/home/user/dustmaps_data'
273

274
# Optional: Configure other settings
275
config['api_timeout'] = 60.0  # seconds
276
config['max_memory'] = 8.0    # GB
277

278
# Save configuration
279
config.save()
280
```
281

282
### Environment Variable Override
283

284
```bash
285
# Override config file location
286
export DUSTMAPS_CONFIG_FNAME="/custom/path/dustmapsrc"
287

288
# dustmaps will use this file instead of default ~/.dustmapsrc
289
```
290

291
### Data Download Workflow
292

293
```python
294
# Step 1: Configure data directory
295
from dustmaps.config import config
296
config['data_dir'] = '/path/to/data'
297

298
# Step 2: Download specific maps
299
import dustmaps.sfd
300
import dustmaps.bayestar
301
import dustmaps.planck
302

303
dustmaps.sfd.fetch()                    # Download SFD map
304
dustmaps.bayestar.fetch()               # Download Bayestar 2019
305
dustmaps.planck.fetch(which='GNILC')    # Download Planck GNILC
306

307
# Step 3: Use maps
308
from dustmaps.sfd import SFDQuery
309
from dustmaps.bayestar import BayestarQuery
310
from dustmaps.planck import PlanckGNILCQuery
311

312
sfd = SFDQuery()
313
bayestar = BayestarQuery()
314
planck = PlanckGNILCQuery()
315
```
316

317
### Batch Download
318

319
```python
320
# Download multiple maps programmatically
321
maps_to_fetch = [
322
    ('sfd', lambda: __import__('dustmaps.sfd', fromlist=['fetch']).fetch()),
323
    ('planck', lambda: __import__('dustmaps.planck', fromlist=['fetch']).fetch()),
324
    ('bayestar', lambda: __import__('dustmaps.bayestar', fromlist=['fetch']).fetch()),
325
    ('marshall', lambda: __import__('dustmaps.marshall', fromlist=['fetch']).fetch()),
326
]
327

328
for map_name, fetch_func in maps_to_fetch:
329
    try:
330
        print(f"Downloading {map_name}...")
331
        fetch_func()
332
        print(f"✓ {map_name} download complete")
333
    except Exception as e:
334
        print(f"✗ {map_name} download failed: {e}")
335
```
336

337
### Error Handling
338

339
```python
340
from dustmaps.config import ConfigError, ConfigWarning
341
from dustmaps.fetch_utils import DownloadError
342
from dustmaps.dustexceptions import Error
343

344
try:
345
    # Configuration operations
346
    config['data_dir'] = '/invalid/path'
347
    config.save()
348

349
    # Data download
350
    import dustmaps.sfd
351
    dustmaps.sfd.fetch()
352

353
    # Map usage
354
    from dustmaps.sfd import SFDQuery
355
    sfd = SFDQuery()
356

357
except ConfigError as e:
358
    print(f"Configuration error: {e}")
359
except DownloadError as e:
360
    print(f"Download failed: {e}")
361
except Error as e:
362
    print(f"dustmaps error: {e}")
363
```
364

365
### Configuration File Format
366

367
The configuration file (`~/.dustmapsrc` by default) is stored in JSON format:
368

369
```json
370
{
371
  "data_dir": "/path/to/dustmaps/data",
372
  "api_timeout": 30.0,
373
  "max_memory": 4.0,
374
  "cache_enabled": true
375
}
376
```