0
# Testing Utilities
1

2
Numerical testing utilities with tolerance-based comparisons for vectors, matrices, and doubles. Essential for testing numerical algorithms where exact equality is impractical due to floating-point precision.
3

4
## Capabilities
5

6
### Double Comparison Utilities
7

8
Enhanced comparison operators for double values with absolute and relative tolerance support.
9

10
```scala { .api }
11
object TestingUtils {
12
  /** Helper case class for double comparisons */
13
  case class CompareDoubleRightSide(
14
    fun: (Double, Double, Double) => Boolean, 
15
    y: Double, 
16
    eps: Double, 
17
    method: String
18
  )
19

20
  /** 
21
   * Implicit class adding tolerance-based comparison operators to Double values
22
   */
23
  implicit class DoubleWithAlmostEquals(val x: Double) {
24
    /** Returns true if values are within tolerance */
25
    def ~=(r: CompareDoubleRightSide): Boolean
26
    
27
    /** Returns true if values are NOT within tolerance */
28
    def !~=(r: CompareDoubleRightSide): Boolean
29
    
30
    /** Throws exception if values are NOT within tolerance, otherwise returns true */
31
    def ~==(r: CompareDoubleRightSide): Boolean
32
    
33
    /** Throws exception if values ARE within tolerance, otherwise returns true */
34
    def !~==(r: CompareDoubleRightSide): Boolean
35
    
36
    /** Creates absolute tolerance comparison */
37
    def absTol(eps: Double): CompareDoubleRightSide
38
    
39
    /** Creates relative tolerance comparison */
40
    def relTol(eps: Double): CompareDoubleRightSide
41
  }
42
  
43
  /** Message constants */
44
  val ABS_TOL_MSG: String = " using absolute tolerance"
45
  val REL_TOL_MSG: String = " using relative tolerance"
46
}
47
```
48

49
**Usage Examples:**
50

51
```scala
52
import org.apache.spark.ml.util.TestingUtils._
53

54
val x = 1.0000001
55
val y = 1.0000002
56

57
// Absolute tolerance comparison
58
val closeEnough = x ~= y absTol 1e-6
59
println(s"Close enough with absolute tolerance: $closeEnough")  // true
60

61
val tooFar = x ~= y absTol 1e-8  
62
println(s"Close enough with stricter tolerance: $tooFar")      // false
63

64
// Relative tolerance comparison  
65
val relativelyClose = x ~= y relTol 1e-6
66
println(s"Relatively close: $relativelyClose")                // true
67

68
// Assertion-style comparisons (throw exceptions on failure)
69
try {
70
  x ~== y absTol 1e-6  // Passes
71
  println("Absolute tolerance assertion passed")
72
} catch {
73
  case e: org.scalatest.exceptions.TestFailedException => 
74
    println(s"Assertion failed: ${e.getMessage}")
75
}
76

77
try {
78
  x ~== y absTol 1e-8  // Fails and throws exception
79
} catch {
80
  case e: org.scalatest.exceptions.TestFailedException => 
81
    println(s"Strict assertion failed: ${e.getMessage}")
82
}
83

84
// Negation comparisons
85
val notClose = x !~= y absTol 1e-8
86
println(s"Not close with strict tolerance: $notClose")        // true
87
```
88

89
### Vector Comparison Utilities
90

91
Tolerance-based comparisons for Vector objects, comparing element-wise with specified tolerance.
92

93
```scala { .api }
94
object TestingUtils {
95
  /** Helper case class for vector comparisons */
96
  case class CompareVectorRightSide(
97
    fun: (Vector, Vector, Double) => Boolean, 
98
    y: Vector, 
99
    eps: Double, 
100
    method: String
101
  )
102

103
  /** 
104
   * Implicit class adding tolerance-based comparison operators to Vector values
105
   */
106
  implicit class VectorWithAlmostEquals(val x: Vector) {
107
    /** Returns true if all vector elements are within tolerance */
108
    def ~=(r: CompareVectorRightSide): Boolean
109
    
110
    /** Returns true if any vector elements are NOT within tolerance */
111
    def !~=(r: CompareVectorRightSide): Boolean
112
    
113
    /** Throws exception if any elements are NOT within tolerance, otherwise returns true */
114
    def ~==(r: CompareVectorRightSide): Boolean
115
    
116
    /** Throws exception if all elements ARE within tolerance, otherwise returns true */
117
    def !~==(r: CompareVectorRightSide): Boolean
118
    
119
    /** Creates absolute tolerance comparison for vectors */
120
    def absTol(eps: Double): CompareVectorRightSide
121
    
122
    /** Creates relative tolerance comparison for vectors */
123
    def relTol(eps: Double): CompareVectorRightSide
124
  }
125
}
126
```
127

128
**Usage Examples:**
129

130
```scala
131
import org.apache.spark.ml.linalg.Vectors
132
import org.apache.spark.ml.util.TestingUtils._
133

134
val v1 = Vectors.dense(1.0, 2.0, 3.0)
135
val v2 = Vectors.dense(1.000001, 1.999999, 3.000001)
136
val v3 = Vectors.dense(1.1, 2.1, 3.1)
137

138
// Absolute tolerance comparison
139
val closeVectors = v1 ~= v2 absTol 1e-5
140
println(s"Vectors close with absolute tolerance: $closeVectors")  // true
141

142
val farVectors = v1 ~= v3 absTol 1e-5
143
println(s"Vectors close with strict tolerance: $farVectors")      // false
144

145
// Relative tolerance comparison
146
val relativelyCloseVectors = v1 ~= v2 relTol 1e-5
147
println(s"Vectors relatively close: $relativelyCloseVectors")     // true
148

149
// Assertion-style comparisons
150
try {
151
  v1 ~== v2 absTol 1e-5  // Passes
152
  println("Vector absolute tolerance assertion passed")
153
} catch {
154
  case e: org.scalatest.exceptions.TestFailedException => 
155
    println(s"Vector assertion failed: ${e.getMessage}")
156
}
157

158
// Working with sparse vectors
159
val sparse1 = Vectors.sparse(5, Array(0, 2, 4), Array(1.0, 3.0, 5.0))
160
val sparse2 = Vectors.sparse(5, Array(0, 2, 4), Array(1.000001, 2.999999, 5.000001))
161

162
val sparsesClose = sparse1 ~= sparse2 absTol 1e-5
163
println(s"Sparse vectors close: $sparsesClose")                  // true
164

165
// Mixed dense/sparse comparison
166
val dense = Vectors.dense(1.0, 0.0, 3.0, 0.0, 5.0)
167
val mixedClose = dense ~= sparse1 absTol 1e-10
168
println(s"Dense and sparse vectors close: $mixedClose")          // true
169
```
170

171
### Matrix Comparison Utilities
172

173
Tolerance-based comparisons for Matrix objects, comparing element-wise with specified tolerance.
174

175
```scala { .api }
176
object TestingUtils {
177
  /** Helper case class for matrix comparisons */
178
  case class CompareMatrixRightSide(
179
    fun: (Matrix, Matrix, Double) => Boolean, 
180
    y: Matrix, 
181
    eps: Double, 
182
    method: String
183
  )
184

185
  /** 
186
   * Implicit class adding tolerance-based comparison operators to Matrix values
187
   */
188
  implicit class MatrixWithAlmostEquals(val x: Matrix) {
189
    /** Returns true if all matrix elements are within tolerance */
190
    def ~=(r: CompareMatrixRightSide): Boolean
191
    
192
    /** Returns true if any matrix elements are NOT within tolerance */
193
    def !~=(r: CompareMatrixRightSide): Boolean
194
    
195
    /** Throws exception if any elements are NOT within tolerance, otherwise returns true */
196
    def ~==(r: CompareMatrixRightSide): Boolean
197
    
198
    /** Throws exception if all elements ARE within tolerance, otherwise returns true */
199
    def !~==(r: CompareMatrixRightSide): Boolean
200
    
201
    /** Creates absolute tolerance comparison for matrices */
202
    def absTol(eps: Double): CompareMatrixRightSide
203
    
204
    /** Creates relative tolerance comparison for matrices */
205
    def relTol(eps: Double): CompareMatrixRightSide
206
  }
207
}
208
```
209

210
**Usage Examples:**
211

212
```scala
213
import org.apache.spark.ml.linalg.Matrices
214
import org.apache.spark.ml.util.TestingUtils._
215

216
val m1 = Matrices.dense(2, 2, Array(1.0, 2.0, 3.0, 4.0))
217
val m2 = Matrices.dense(2, 2, Array(1.000001, 1.999999, 3.000001, 3.999999))
218
val m3 = Matrices.dense(2, 2, Array(1.1, 2.1, 3.1, 4.1))
219

220
// Absolute tolerance comparison
221
val closeMatrices = m1 ~= m2 absTol 1e-5
222
println(s"Matrices close with absolute tolerance: $closeMatrices")  // true
223

224
val farMatrices = m1 ~= m3 absTol 1e-5
225
println(s"Matrices close with strict tolerance: $farMatrices")      // false
226

227
// Relative tolerance comparison
228
val relativelyCloseMatrices = m1 ~= m2 relTol 1e-5
229
println(s"Matrices relatively close: $relativelyCloseMatrices")     // true
230

231
// Assertion-style comparisons
232
try {
233
  m1 ~== m2 absTol 1e-5  // Passes
234
  println("Matrix absolute tolerance assertion passed")
235
} catch {
236
  case e: org.scalatest.exceptions.TestFailedException => 
237
    println(s"Matrix assertion failed: ${e.getMessage}")
238
}
239

240
// Working with sparse matrices
241
val sparse1 = Matrices.sparse(3, 3, Array(0, 1, 2, 3), Array(0, 1, 2), Array(1.0, 2.0, 3.0))
242
val sparse2 = Matrices.sparse(3, 3, Array(0, 1, 2, 3), Array(0, 1, 2), Array(1.000001, 1.999999, 3.000001))
243

244
val sparsesClose = sparse1 ~= sparse2 absTol 1e-5
245
println(s"Sparse matrices close: $sparsesClose")                  // true
246

247
// Mixed dense/sparse comparison  
248
val denseEquiv = Matrices.dense(3, 3, Array(1.0, 0.0, 0.0, 0.0, 2.0, 0.0, 0.0, 0.0, 3.0))
249
val mixedClose = denseEquiv ~= sparse1 absTol 1e-10
250
println(s"Dense and sparse matrices close: $mixedClose")          // true
251
```
252

253
### Practical Testing Examples
254

255
#### Testing Vector Operations
256

257
```scala
258
import org.apache.spark.ml.linalg.Vectors
259
import org.apache.spark.ml.util.TestingUtils._
260

261
def testVectorOperations(): Unit = {
262
  val v1 = Vectors.dense(1.0, 2.0, 3.0)
263
  val v2 = Vectors.dense(4.0, 5.0, 6.0)
264
  val expected = 32.0  // 1*4 + 2*5 + 3*6
265
  
266
  // Calculate dot product using vector operations
267
  val result = v1.toArray.zip(v2.toArray).map{ case (a, b) => a * b }.sum
268
  
269
  // Test with tolerance
270
  assert(result ~== expected absTol 1e-10)
271
  println("Vector dot product test passed")
272
}
273

274
def testVectorNorms(): Unit = {
275
  val vector = Vectors.dense(3.0, 4.0)
276
  val expectedL2Norm = 5.0  // sqrt(3^2 + 4^2)
277
  
278
  val l2Norm = Vectors.norm(vector, 2.0)
279
  
280
  // Test with tolerance
281
  assert(l2Norm ~== expectedL2Norm absTol 1e-10)
282
  println("Vector norm test passed")
283
}
284

285
testVectorOperations()
286
testVectorNorms()
287
```
288

289
#### Testing Matrix Operations
290

291
```scala
292
import org.apache.spark.ml.linalg.Matrices
293
import org.apache.spark.ml.util.TestingUtils._
294

295
def testMatrixMultiplication(): Unit = {
296
  val A = Matrices.dense(2, 2, Array(1.0, 2.0, 3.0, 4.0))  // [[1,3],[2,4]]
297
  val B = Matrices.dense(2, 2, Array(5.0, 6.0, 7.0, 8.0))  // [[5,7],[6,8]]
298
  val expected = Matrices.dense(2, 2, Array(19.0, 22.0, 43.0, 50.0))
299
  
300
  val result = A.multiply(B)
301
  
302
  // Test with tolerance
303
  assert(result ~== expected absTol 1e-10)
304
  println("Matrix multiplication test passed")
305
}
306

307
testMatrixMultiplication()
308
```
309

310
### Advanced Tolerance Considerations
311

312
#### Absolute vs Relative Tolerance
313

314
```scala
315
import org.apache.spark.ml.util.TestingUtils._
316

317
// For values near zero, use absolute tolerance
318
val nearZero1 = 1e-10
319
val nearZero2 = 2e-10
320
assert(nearZero1 ~== nearZero2 absTol 1e-9)  // Good
321
// nearZero1 ~== nearZero2 relTol 1e-9  // Would throw exception - relative tolerance meaningless
322

323
// For larger values, relative tolerance is often better
324
val large1 = 1e6
325
val large2 = 1.000001e6  
326
assert(large1 ~== large2 relTol 1e-5)  // Good
327
// large1 ~== large2 absTol 1e-5  // Would fail - absolute difference is 1.0
328
```
329

330
#### NaN Handling
331

332
```scala
333
import org.apache.spark.ml.util.TestingUtils._
334

335
val nan1 = Double.NaN
336
val nan2 = Double.NaN
337

338
// NaN values are considered equal in tolerance comparisons
339
assert(nan1 ~== nan2 absTol 1e-10)  // Passes
340
assert(nan1 ~== nan2 relTol 1e-10)  // Passes
341

342
println("NaN comparison tests passed")
343
```
344

345
## Types
346

347
```scala { .api }
348
// Helper case classes for comparison operations
349
case class CompareDoubleRightSide(
350
  fun: (Double, Double, Double) => Boolean, 
351
  y: Double, 
352
  eps: Double, 
353
  method: String
354
)
355

356
case class CompareVectorRightSide(
357
  fun: (Vector, Vector, Double) => Boolean, 
358
  y: Vector, 
359
  eps: Double, 
360
  method: String
361
)
362

363
case class CompareMatrixRightSide(
364
  fun: (Matrix, Matrix, Double) => Boolean, 
365
  y: Matrix, 
366
  eps: Double, 
367
  method: String
368
)
369

370
// Implicit classes that add comparison operators
371
implicit class DoubleWithAlmostEquals(val x: Double)
372
implicit class VectorWithAlmostEquals(val x: Vector) 
373
implicit class MatrixWithAlmostEquals(val x: Matrix)
374
```