0
# Exception Handling
1

2
Comprehensive exception handling system providing standardized error classes with contextual information, error classification, and localized message support for debugging distributed Spark operations.
3

4
## Capabilities
5

6
### SparkException
7

8
Main exception type thrown by Spark operations with support for error classes, message parameters, and query context.
9

10
```scala { .api }
11
/**
12
 * Main exception class thrown by Spark with structured error information
13
 * @param message Human-readable error message
14
 * @param cause Root cause exception (optional)
15
 * @param errorClass Standardized error class identifier (optional)
16
 * @param messageParameters Template parameters for localized messages
17
 * @param context Query context information for debugging
18
 */
19
class SparkException(
20
  message: String,
21
  cause: Throwable,
22
  errorClass: Option[String],
23
  messageParameters: Map[String, String],
24
  context: Array[QueryContext] = Array.empty
25
) extends Exception(message, cause) with SparkThrowable
26

27
object SparkException {
28
  /** Create internal error exception with message */
29
  def internalError(msg: String): SparkException
30
  
31
  /** Create internal error exception with message and cause */
32
  def internalError(msg: String, cause: Throwable): SparkException
33
  
34
  /** Create internal error exception with context and summary */
35
  def internalError(
36
    msg: String, 
37
    context: Array[QueryContext], 
38
    summary: String,
39
    category: Option[String]
40
  ): SparkException
41
}
42
```
43

44
**Usage Examples:**
45

46
```scala
47
import org.apache.spark.{SparkException, QueryContext}
48

49
// Basic exception creation
50
val basicError = new SparkException("Operation failed")
51

52
// Exception with cause
53
val errorWithCause = new SparkException("Database connection failed", sqlException)
54

55
// Exception with error class and parameters
56
val structuredError = new SparkException(
57
  errorClass = "INVALID_PARAMETER_VALUE",
58
  messageParameters = Map("parameter" -> "timeout", "value" -> "-1"),
59
  cause = null
60
)
61

62
// Internal error creation
63
val internalError = SparkException.internalError("Unexpected state in executor")
64

65
// Handling exceptions
66
try {
67
  // Some Spark operation
68
} catch {
69
  case e: SparkException =>
70
    println(s"Error class: ${e.getErrorClass}")
71
    println(s"Parameters: ${e.getMessageParameters}") 
72
    if (e.isInternalError) {
73
      logger.error("Internal Spark error occurred", e)
74
    }
75
}
76
```
77

78
### SparkThrowable Interface
79

80
Standardized interface for all Spark exceptions providing consistent error information access.
81

82
```java { .api }
83
/**
84
 * Interface mixed into Throwables thrown from Spark
85
 * Provides standardized access to error classification and context
86
 */
87
public interface SparkThrowable {
88
  /** Returns succinct, human-readable error category identifier */
89
  String getErrorClass();
90
  
91
  /** Returns portable error identifier across SQL engines (SQLSTATE) */
92
  default String getSqlState() {
93
    return SparkThrowableHelper.getSqlState(this.getErrorClass());
94
  }
95
  
96
  /** Returns true if this error is an internal Spark error */
97
  default boolean isInternalError() {
98
    return SparkThrowableHelper.isInternalError(this.getErrorClass());
99
  }
100
  
101
  /** Returns message template parameters for localization */
102
  default Map<String, String> getMessageParameters() {
103
    return new HashMap<>();
104
  }
105
  
106
  /** Returns query context information for debugging */
107
  default QueryContext[] getQueryContext() { 
108
    return new QueryContext[0]; 
109
  }
110
}
111
```
112

113
### QueryContext Interface
114

115
Provides contextual information about where errors occurred during query execution.
116

117
```java { .api }
118
/**
119
 * Query context information for SparkThrowable exceptions
120
 * Helps users understand where errors occur while executing queries
121
 */
122
public interface QueryContext {
123
  /** 
124
   * The object type of the query which throws the exception
125
   * Empty string for main query, otherwise object type in upper case (e.g., "VIEW")
126
   */
127
  String objectType();
128
  
129
  /** 
130
   * The object name of the query which throws the exception  
131
   * Empty string for main query, otherwise the object name (e.g., view name "V1")
132
   */
133
  String objectName();
134
  
135
  /** The starting index in the query text (0-based) */
136
  int startIndex();
137
  
138
  /** The stopping index in the query text (0-based) */
139
  int stopIndex();
140
  
141
  /** The corresponding fragment of the query which throws the exception */
142
  String fragment();
143
}
144
```
145

146
### Specialized Exception Types
147

148
Spark provides specialized exception types for specific error scenarios while maintaining the SparkThrowable contract.
149

150
```scala { .api }
151
/** Exception for driver process execution failures */
152
private[spark] class SparkDriverExecutionException(cause: Throwable) 
153
  extends SparkException("Execution error", cause)
154

155
/** Exception for child process exit codes in PySpark */  
156
private[spark] case class SparkUserAppException(exitCode: Int)
157
  extends SparkException(s"User application exited with $exitCode")
158

159
/** Exception for dead executor access attempts */
160
private[spark] case class ExecutorDeadException(message: String)
161
  extends SparkException(
162
    errorClass = "INTERNAL_ERROR_NETWORK",
163
    messageParameters = Map("message" -> message), 
164
    cause = null
165
  )
166

167
/** Exception for version upgrade compatibility issues */
168
private[spark] class SparkUpgradeException private(
169
  message: String,
170
  cause: Option[Throwable], 
171
  errorClass: Option[String],
172
  messageParameters: Map[String, String]
173
) extends RuntimeException with SparkThrowable
174

175
/** Arithmetic exception with error classification */
176
private[spark] class SparkArithmeticException private(
177
  message: String,
178
  errorClass: Option[String],
179
  messageParameters: Map[String, String],
180
  context: Array[QueryContext]  
181
) extends ArithmeticException(message) with SparkThrowable
182

183
/** Other specialized exceptions for specific domains */
184
private[spark] class SparkUnsupportedOperationException // For unsupported operations
185
private[spark] class SparkClassNotFoundException // For missing classes
186
private[spark] class SparkConcurrentModificationException // For concurrency issues
187
private[spark] class SparkDateTimeException // For date/time errors
188
private[spark] class SparkFileNotFoundException // For missing files
189
private[spark] class SparkNumberFormatException // For number parsing errors
190
private[spark] class SparkIllegalArgumentException // For invalid arguments
191
private[spark] class SparkRuntimeException // For runtime errors
192
private[spark] class SparkNoSuchElementException // For missing elements
193
private[spark] class SparkSecurityException // For security violations
194
private[spark] class SparkArrayIndexOutOfBoundsException // For array access errors
195
private[spark] class SparkSQLException // For SQL-related errors
196
private[spark] class SparkSQLFeatureNotSupportedException // For unsupported SQL features
197
```
198

199
## Error Handling Best Practices
200

201
### Exception Creation Patterns
202

203
```scala
204
// Prefer error classes over plain messages for new code
205
val standardError = new SparkException(
206
  errorClass = "INVALID_PARAMETER_VALUE",
207
  messageParameters = Map("parameter" -> "partitions", "value" -> "0"),
208
  cause = null
209
)
210

211
// Use internal error factory methods for internal failures
212
val internalError = SparkException.internalError(
213
  "Unexpected null value in partition metadata"
214
)
215

216
// Include query context for user-facing errors
217
val contextualError = new SparkException(
218
  errorClass = "SYNTAX_ERROR", 
219
  messageParameters = Map("message" -> "Invalid column reference"),
220
  cause = null,
221
  context = Array(queryContext)
222
)
223
```
224

225
### Exception Handling Patterns
226

227
```scala
228
try {
229
  // Spark operations
230
} catch {
231
  case e: SparkException if e.isInternalError =>
232
    // Log internal errors for debugging
233
    logger.error(s"Internal Spark error: ${e.getErrorClass}", e)
234
    throw e
235
    
236
  case e: SparkException =>
237
    // Handle user errors gracefully
238
    val errorInfo = s"Error: ${e.getErrorClass} - ${e.getMessage}"
239
    logger.warn(errorInfo)
240
    // Potentially recoverable - retry or provide user feedback
241
    
242
  case e: SparkThrowable =>
243
    // Handle other Spark throwables
244
    if (e.getSqlState != null) {
245
      // SQL-compatible error handling
246
      handleSqlError(e.getSqlState, e.getMessage)
247
    }
248
}
249
```
250

251
### Error Classification
252

253
The exception system supports error classification through several mechanisms:
254

255
- **Error Classes**: Standardized identifiers like `INVALID_PARAMETER_VALUE`, `INTERNAL_ERROR`
256
- **SQL States**: Standard SQL error codes for database compatibility  
257
- **Internal vs External**: Distinguishes Spark internal bugs from user errors
258
- **Message Parameters**: Template parameters for consistent, localizable error messages
259
- **Query Context**: Source location information for debugging query-related errors