0
# Data Quality and Calibration
1

2
Tools for managing data quality flags, photometric equalization, and applying calibration corrections to HST observations. These utilities ensure data integrity and enable accurate photometric and astrometric analysis.
3

4
## Capabilities
5

6
### Data Quality Bit Management
7

8
Reset specified data quality (DQ) bits in HST FLT files to enable proper processing of flagged pixels.
9

10
```python { .api }
11
def reset_dq_bits(input, bits, extver=None, extname='dq'):
12
    """
13
    Reset specified DQ values to 0 in FLT files.
14

15
    Parameters:
16
    - input: str or list, input FLT files to modify
17
    - bits: int or str, DQ bit values to reset (comma-separated string or integer)
18
    - extver: int, extension version to modify (default: all)
19
    - extname: str, extension name containing DQ array
20

21
    Returns:
22
    None (modifies input files in place)
23
    """
24

25
def run(configobj=None):
26
    """
27
    Processing function for resetbits task.
28

29
    Parameters:
30
    - configobj: ConfigObj, configuration with DQ bit parameters
31

32
    Returns:
33
    None (modifies input files)
34
    """
35

36
def main():
37
    """
38
    Command-line entry point for resetbits script.
39

40
    Returns:
41
    None
42
    """
43
```
44

45
### Photometric Equalization
46

47
Equalize sensitivities of images across different chips and observation epochs to ensure consistent photometry.
48

49
```python { .api }
50
def photeq(files='*_flt.fits', sciext='SCI', errext='ERR',
51
          readnoise=0.0, gain=1.0, platescale=1.0, exptime=1.0,
52
          expstart=0.0, **kwargs):
53
    """
54
    Equalize sensitivities across chips and epochs.
55

56
    Parameters:
57
    - files: str or list, input files for photometric equalization
58
    - sciext: str, science extension name
59
    - errext: str, error extension name
60
    - readnoise: float, detector read noise value
61
    - gain: float, detector gain value
62
    - platescale: float, pixel scale in arcsec/pixel
63
    - exptime: float, exposure time for normalization
64
    - expstart: float, exposure start time
65
    - **kwargs: dict, additional calibration parameters
66

67
    Returns:
68
    None (modifies input files with equalized values)
69
    """
70

71
def run(configObj):
72
    """
73
    Processing function for photeq task.
74

75
    Parameters:
76
    - configObj: ConfigObj, configuration with photometric parameters
77

78
    Returns:
79
    None (applies photometric equalization)
80
    """
81
```
82

83
### Distortion Reference File Updates
84

85
Update ACS headers with new distortion reference file information and residual distortion corrections.
86

87
```python { .api }
88
def update(input, refdir="jref$", local=None, interactive=False,
89
          wcsupdate=True):
90
    """
91
    Update headers with ACS distortion reference files.
92

93
    Parameters:
94
    - input: str or list, input ACS files to update
95
    - refdir: str, directory containing reference files
96
    - local: str, local directory for reference files
97
    - interactive: bool, interactive mode for user prompts
98
    - wcsupdate: bool, update WCS after reference file update
99

100
    Returns:
101
    None (updates file headers)
102
    """
103

104
def run(configobj=None, editpars=False):
105
    """
106
    Processing function for updatenpol task.
107

108
    Parameters:
109
    - configobj: ConfigObj, configuration with update parameters
110
    - editpars: bool, allow parameter editing
111

112
    Returns:
113
    None (applies reference file updates)
114
    """
115

116
def main():
117
    """
118
    Command-line entry point for updatenpol script.
119

120
    Returns:
121
    None
122
    """
123
```
124

125
### Pixel Value Replacement
126

127
Replace problematic pixel values such as NaNs in images with user-specified replacement values.
128

129
```python { .api }
130
def replace(input, **pars):
131
    """
132
    Replace pixel values like NaNs with other values.
133

134
    Parameters:
135
    - input: str or list, input images to modify
136
    - **pars: dict, replacement parameters including:
137
        - mask_value: value to search for replacement
138
        - fill_value: replacement value
139
        - extensions: list of extensions to process
140

141
    Returns:
142
    None (modifies input images)
143
    """
144

145
def run(configobj):
146
    """
147
    Processing function for pixreplace task.
148

149
    Parameters:
150
    - configobj: ConfigObj, configuration with replacement parameters
151

152
    Returns:
153
    None (applies pixel value replacements)
154
    """
155
```
156

157
## Usage Examples
158

159
### Resetting Data Quality Bits
160

161
```python
162
from drizzlepac import resetbits
163

164
# Reset cosmic ray and hot pixel flags
165
resetbits.reset_dq_bits('j8bt06nyq_flt.fits', bits='4096,8192')
166

167
# Reset specific bits for multiple files
168
input_files = ['file1_flt.fits', 'file2_flt.fits', 'file3_flt.fits']
169
resetbits.reset_dq_bits(input_files, bits='4096')  # Reset CR flag
170

171
# Command-line usage equivalent
172
# resetbits j8bt06nyq_flt.fits 4096,8192
173
```
174

175
### Photometric Equalization
176

177
```python
178
from drizzlepac import photeq
179

180
# Basic photometric equalization
181
photeq.photeq(files='*_flt.fits',
182
             readnoise=3.1,
183
             gain=2.0,
184
             platescale=0.05)  # ACS/WFC pixel scale
185

186
# Multi-chip equalization with specific parameters
187
acs_files = ['j8bt06nyq_flt.fits', 'j8bt06nzq_flt.fits']
188
photeq.photeq(files=acs_files,
189
             sciext='SCI',
190
             errext='ERR',
191
             readnoise=3.1,
192
             gain=2.0)
193
```
194

195
### Updating ACS Distortion References
196

197
```python
198
from drizzlepac import updatenpol
199

200
# Update ACS files with latest distortion references
201
updatenpol.update('j8bt*_flt.fits',
202
                 refdir='jref$',
203
                 interactive=False,
204
                 wcsupdate=True)
205

206
# Update with local reference files
207
updatenpol.update('observation_flt.fits',
208
                 local='/path/to/local/refs/',
209
                 wcsupdate=True)
210
```
211

212
### Replacing NaN Values
213

214
```python
215
from drizzlepac import pixreplace
216

217
# Replace NaN values with zero
218
pixreplace.replace('image_with_nans.fits',
219
                  mask_value=float('nan'),
220
                  fill_value=0.0)
221

222
# Replace specific problematic values
223
pixreplace.replace('problematic_image.fits',
224
                  mask_value=-999.0,
225
                  fill_value=0.0,
226
                  extensions=['SCI', 'ERR'])
227
```
228

229
## Data Quality Bit Definitions
230

231
### Standard HST DQ Bit Values
232

233
| Bit | Value | Description |
234
|-----|-------|-------------|
235
| 0 | 1 | Reed-Solomon decoding error |
236
| 1 | 2 | Data lost in transmission |
237
| 2 | 4 | Bad detector pixel or beyond aperture |
238
| 3 | 8 | Masked by aperture feature |
239
| 4 | 16 | Hot pixel |
240
| 5 | 32 | CTE tail |
241
| 6 | 64 | Warm pixel |
242
| 7 | 128 | Bad pixel in bias |
243
| 8 | 256 | Bad pixel in dark |
244
| 9 | 512 | Bad pixel in flat field |
245
| 10 | 1024 | Impacted by cosmic ray |
246
| 11 | 2048 | Impacted by detector artifact |
247
| 12 | 4096 | Pixel saturated |
248
| 13 | 8192 | Bad pixel flagged by user |
249
250
### Common DQ Bit Operations
251

252
```python
253
# Reset cosmic ray flags (bit 10, value 1024)
254
resetbits.reset_dq_bits(files, bits='1024')
255

256
# Reset multiple flags
257
resetbits.reset_dq_bits(files, bits='1024,4096')  # CR + saturation
258

259
# Reset all user-flagged bad pixels
260
resetbits.reset_dq_bits(files, bits='8192')
261
```
262

263
## Calibration Parameters
264

265
### Photometric Parameters
266

267
Key parameters for photometric equalization:
268

269
- **Read Noise**: Detector-specific noise characteristics
270
- **Gain**: Conversion factor from electrons to DN
271
- **Plate Scale**: Angular pixel size for flux normalization
272
- **Exposure Time**: Temporal normalization factor
273
- **Filter Corrections**: Wavelength-dependent adjustments
274

275
### Instrument-Specific Values
276

277
#### ACS/WFC
278
```python
279
photeq_pars = {
280
    'readnoise': 3.1,    # e-/pixel
281
    'gain': 2.0,         # e-/DN
282
    'platescale': 0.05   # arcsec/pixel
283
}
284
```
285

286
#### WFC3/UVIS
287
```python
288
photeq_pars = {
289
    'readnoise': 3.1,    # e-/pixel
290
    'gain': 1.5,         # e-/DN
291
    'platescale': 0.04   # arcsec/pixel
292
}
293
```
294

295
#### WFC3/IR
296
```python
297
photeq_pars = {
298
    'readnoise': 2.5,    # e-/pixel
299
    'gain': 2.5,         # e-/DN
300
    'platescale': 0.13   # arcsec/pixel
301
}
302
```
303

304
## Reference File Management
305

306
### ACS NPOL File Updates
307

308
The `updatenpol` task manages:
309
- **NPOLFILE**: Non-polynomial distortion corrections
310
- **D2IMFILE**: Detector-to-image coordinate transformations
311
- **Residual corrections**: Additional distortion terms
312

313
### Reference File Locations
314

315
- **CRDS Cache**: Calibration Reference Data System
316
- **JREF**: HST reference file directory
317
- **Local Files**: User-provided reference files
318

319
## Error Handling
320

321
### Common Calibration Errors
322

323
- **IOError**: Missing input files or reference files
324
- **ValueError**: Invalid DQ bit values or calibration parameters
325
- **KeyError**: Missing required FITS header keywords
326
- **PermissionError**: Write access issues for file modifications
327

328
### Quality Checks
329

330
```python
331
# Verify successful DQ bit reset
332
from astropy.io import fits
333

334
with fits.open('processed_flt.fits') as hdul:
335
    dq_data = hdul['DQ', 1].data
336
    cr_pixels = (dq_data & 1024) > 0  # Check for remaining CR flags
337
    print(f"Remaining CR pixels: {cr_pixels.sum()}")
338
```
339

340
## Best Practices
341

342
### DQ Bit Management
343
1. **Backup original files** before resetting DQ bits
344
2. **Document bit modifications** for reproducibility
345
3. **Validate results** after DQ modifications
346
4. **Use conservative approach** - reset only necessary bits
347

348
### Photometric Equalization
349
1. **Apply before image combination** for consistent photometry
350
2. **Use instrument-specific parameters** for accuracy
351
3. **Validate equalization results** across multiple exposures
352
4. **Consider temporal variations** for long-term observations
353

354
### Distortion Updates
355
1. **Use latest reference files** for best accuracy
356
2. **Update WCS after reference changes**
357
3. **Validate astrometric precision** after updates
358
4. **Document reference file versions** used