t

com.holdenkarau.spark.testing

ScalaDataFrameSuiteBase

trait ScalaDataFrameSuiteBase extends AnyFunSuite with DataFrameSuiteBase

Base trait for testing Spark DataFrames in Scala.

Linear Supertypes
DataFrameSuiteBase, DataFrameSuiteBaseLike, SharedSparkContext, SparkContextProvider, BeforeAndAfterAll, SuiteMixin, TestSuite, TestSuiteLike, AnyFunSuite, AnyFunSuiteLike, Documenting, Alerting, Notifying, Informing, TestRegistration, org.scalatest.TestSuite, Suite, Serializable, Serializable, Assertions, TripleEquals, TripleEqualsSupport, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. ScalaDataFrameSuiteBase
  2. DataFrameSuiteBase
  3. DataFrameSuiteBaseLike
  4. SharedSparkContext
  5. SparkContextProvider
  6. BeforeAndAfterAll
  7. SuiteMixin
  8. TestSuite
  9. TestSuiteLike
  10. AnyFunSuite
  11. AnyFunSuiteLike
  12. Documenting
  13. Alerting
  14. Notifying
  15. Informing
  16. TestRegistration
  17. TestSuite
  18. Suite
  19. Serializable
  20. Serializable
  21. Assertions
  22. TripleEquals
  23. TripleEqualsSupport
  24. AnyRef
  25. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Type Members

  1. class CheckingEqualizer[L] extends AnyRef
    Definition Classes
    TripleEqualsSupport
  2. class Equalizer[L] extends AnyRef
    Definition Classes
    TripleEqualsSupport
  3. trait NoArgTest extends () ⇒ Outcome with TestData
    Attributes
    protected
    Definition Classes
    TestSuite

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. def !==[T](right: Spread[T]): TripleEqualsInvocationOnSpread[T]
    Definition Classes
    TripleEqualsSupport
  3. def !==(right: Null): TripleEqualsInvocation[Null]
    Definition Classes
    TripleEqualsSupport
  4. def !==[T](right: T): TripleEqualsInvocation[T]
    Definition Classes
    TripleEqualsSupport
  5. final def ##(): Int
    Definition Classes
    AnyRef → Any
  6. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  7. def ===[T](right: Spread[T]): TripleEqualsInvocationOnSpread[T]
    Definition Classes
    TripleEqualsSupport
  8. def ===(right: Null): TripleEqualsInvocation[Null]
    Definition Classes
    TripleEqualsSupport
  9. def ===[T](right: T): TripleEqualsInvocation[T]
    Definition Classes
    TripleEqualsSupport
  10. def afterAll(): Unit
    Definition Classes
    DataFrameSuiteBase → SharedSparkContext → BeforeAndAfterAll
  11. def alert: Alerter
    Attributes
    protected
    Definition Classes
    AnyFunSuiteLike → Alerting
  12. def appID: String
    Definition Classes
    SparkContextProvider
  13. def appName: String
    Definition Classes
    SparkContextProvider
  14. def approxEquals(r1: Row, r2: Row, tol: Double, tolTimestamp: Duration): Boolean
    Definition Classes
    DataFrameSuiteBaseLike
  15. def approxEquals(r1: Row, r2: Row, tolTimestamp: Duration): Boolean
    Definition Classes
    DataFrameSuiteBaseLike
  16. def approxEquals(r1: Row, r2: Row, tol: Double): Boolean
    Definition Classes
    DataFrameSuiteBaseLike
  17. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  18. def assert[U](message: String, expected: U, actual: U)(implicit CT: ClassTag[U]): Unit
    Definition Classes
    TestSuite → TestSuiteLike
  19. def assert[U](expected: U, actual: U)(implicit CT: ClassTag[U]): Unit
    Definition Classes
    TestSuite → TestSuiteLike
  20. macro def assert(condition: Boolean, clue: Any)(implicit prettifier: Prettifier, pos: Position): Assertion
    Definition Classes
    Assertions
  21. macro def assert(condition: Boolean)(implicit prettifier: Prettifier, pos: Position): Assertion
    Definition Classes
    Assertions
  22. macro def assertCompiles(code: String)(implicit pos: Position): Assertion
    Definition Classes
    Assertions
  23. def assertDataFrameApproximateEquals(expected: DataFrame, result: DataFrame, tol: Double, tolTimestamp: Duration, customShow: (DataFrame) ⇒ Unit = _.show()): Unit

    Compares if two DataFrames are equal, checks that the schemas are the same.

    Compares if two DataFrames are equal, checks that the schemas are the same. When comparing inexact fields uses tol & tolTimestamp.

    tol

    max acceptable numeric tolerance, should be less than 1.

    tolTimestamp

    max acceptable timestamp tolerance.

    customShow

    unit function to customize the show method when dataframes are not equal. IE: df.show(false) or df.toJSON.show(false).

    Definition Classes
    DataFrameSuiteBaseLike
  24. def assertDataFrameDataEquals(expected: DataFrame, result: DataFrame): Unit

    Compares if two DataFrames are equal without caring about order of rows, by finding elements in one DataFrame that is not in the other.

    Compares if two DataFrames are equal without caring about order of rows, by finding elements in one DataFrame that is not in the other. The resulting DataFrame should be empty inferring the two DataFrames have the same elements. Does not compare the schema.

    Definition Classes
    DataFrameSuiteBaseLike
  25. def assertDataFrameEquals(expected: DataFrame, result: DataFrame, customShow: (DataFrame) ⇒ Unit = _.show()): Unit

    Compares if two DataFrames are equal, checks the schema and then if that matches checks if the rows are equal.

    Compares if two DataFrames are equal, checks the schema and then if that matches checks if the rows are equal.

    customShow

    unit function to customize the show method when dataframes are not equal. IE: df.show(false) or df.toJSON.show(false).

    Definition Classes
    DataFrameSuiteBaseLike
  26. def assertDataFrameNoOrderEquals(expected: DataFrame, result: DataFrame): Unit

    Compares if two DataFrames are equal without caring about order of rows, by finding elements in one DataFrame that is not in the other.

    Compares if two DataFrames are equal without caring about order of rows, by finding elements in one DataFrame that is not in the other. The resulting DataFrame should be empty inferring the two DataFrames have the same elements. Also verifies that the schema is identical.

    Definition Classes
    DataFrameSuiteBaseLike
  27. macro def assertDoesNotCompile(code: String)(implicit pos: Position): Assertion
    Definition Classes
    Assertions
  28. def assertEmpty[U](arr: Array[U])(implicit CT: ClassTag[U]): Unit
    Definition Classes
    TestSuite → TestSuiteLike
  29. def assertResult(expected: Any)(actual: Any)(implicit prettifier: Prettifier, pos: Position): Assertion
    Definition Classes
    Assertions
  30. def assertResult(expected: Any, clue: Any)(actual: Any)(implicit prettifier: Prettifier, pos: Position): Assertion
    Definition Classes
    Assertions
  31. def assertSchemasEqual(expected: StructType, result: StructType): Unit

    Compare if two schemas are equal, ignoring autoGeneratedAlias magic

    Compare if two schemas are equal, ignoring autoGeneratedAlias magic

    Definition Classes
    DataFrameSuiteBaseLike
  32. def assertSmallDataFrameDataEquals(expected: DataFrame, result: DataFrame): Unit

    Compares if two DataFrames are equal without caring about order of rows, by finding elements in one DataFrame that is not in the other.

    Compares if two DataFrames are equal without caring about order of rows, by finding elements in one DataFrame that is not in the other. Similar to the function assertDataFrameDataEquals but for small DataFrame that can be collected in memory for the comparison.

    Definition Classes
    DataFrameSuiteBaseLike
  33. def assertThrows[T <: AnyRef](f: ⇒ Any)(implicit classTag: ClassTag[T], pos: Position): Assertion
    Definition Classes
    Assertions
  34. def assertTrue(expected: Boolean): Unit
    Definition Classes
    TestSuite → TestSuiteLike
  35. macro def assertTypeError(code: String)(implicit pos: Position): Assertion
    Definition Classes
    Assertions
  36. macro def assume(condition: Boolean, clue: Any)(implicit prettifier: Prettifier, pos: Position): Assertion
    Definition Classes
    Assertions
  37. macro def assume(condition: Boolean)(implicit prettifier: Prettifier, pos: Position): Assertion
    Definition Classes
    Assertions
  38. def beforeAll(): Unit
    Definition Classes
    DataFrameSuiteBase → SharedSparkContext → BeforeAndAfterAll
  39. def builder(): Builder

    Constructs a configuration for hive or iceberg, where the metastore is located in a temp directory.

    Constructs a configuration for hive or iceberg, where the metastore is located in a temp directory.

    Definition Classes
    DataFrameSuiteBaseLike
  40. def cancel(cause: Throwable)(implicit pos: Position): Nothing
    Definition Classes
    Assertions
  41. def cancel(message: String, cause: Throwable)(implicit pos: Position): Nothing
    Definition Classes
    Assertions
  42. def cancel(message: String)(implicit pos: Position): Nothing
    Definition Classes
    Assertions
  43. def cancel()(implicit pos: Position): Nothing
    Definition Classes
    Assertions
  44. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native() @IntrinsicCandidate()
  45. def conf: SparkConf
    Definition Classes
    SparkContextProvider
  46. def convertEquivalenceToAToBConstraint[A, B](equivalenceOfB: Equivalence[B])(implicit ev: <:<[A, B]): CanEqual[A, B]
    Definition Classes
    TripleEquals → TripleEqualsSupport
  47. def convertEquivalenceToBToAConstraint[A, B](equivalenceOfA: Equivalence[A])(implicit ev: <:<[B, A]): CanEqual[A, B]
    Definition Classes
    TripleEquals → TripleEqualsSupport
  48. def convertToCheckingEqualizer[T](left: T): CheckingEqualizer[T]
    Definition Classes
    TripleEquals → TripleEqualsSupport
  49. implicit def convertToEqualizer[T](left: T): Equalizer[T]
    Definition Classes
    TripleEquals → TripleEqualsSupport
  50. def defaultEquality[A]: Equality[A]
    Definition Classes
    TripleEqualsSupport
  51. def enableHiveSupport: Boolean
    Attributes
    protected
    Definition Classes
    DataFrameSuiteBaseLike
  52. def enableIcebergSupport: Boolean
    Attributes
    protected
    Definition Classes
    DataFrameSuiteBaseLike
  53. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  54. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  55. final def execute(testName: String, configMap: ConfigMap, color: Boolean, durations: Boolean, shortstacks: Boolean, fullstacks: Boolean, stats: Boolean): Unit
    Definition Classes
    Suite
  56. def expectedTestCount(filter: Filter): Int
    Definition Classes
    Suite
  57. def fail(message: String): Unit
    Definition Classes
    TestSuite → TestSuiteLike
  58. def fail(cause: Throwable)(implicit pos: Position): Nothing
    Definition Classes
    Assertions
  59. def fail(message: String, cause: Throwable)(implicit pos: Position): Nothing
    Definition Classes
    Assertions
  60. def fail(message: String)(implicit pos: Position): Nothing
    Definition Classes
    Assertions
  61. def fail()(implicit pos: Position): Nothing
    Definition Classes
    Assertions
  62. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @IntrinsicCandidate()
  63. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @IntrinsicCandidate()
  64. val icebergWarehouse: String
    Definition Classes
    DataFrameSuiteBaseLike
  65. def ignore(testName: String, testTags: Tag*)(testFun: ⇒ Any)(implicit pos: Position): Unit
    Attributes
    protected
    Definition Classes
    AnyFunSuiteLike
  66. implicit def impSqlContext: SQLContext
    Attributes
    protected
    Definition Classes
    DataFrameSuiteBaseLike
  67. def info: Informer
    Attributes
    protected
    Definition Classes
    AnyFunSuiteLike → Informing
  68. def intercept[T <: AnyRef](f: ⇒ Any)(implicit classTag: ClassTag[T], pos: Position): T
    Definition Classes
    Assertions
  69. val invokeBeforeAllAndAfterAllEvenIfNoTestsAreExpected: Boolean
    Definition Classes
    BeforeAndAfterAll
  70. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  71. lazy val localMetastorePath: String
    Definition Classes
    DataFrameSuiteBaseLike
  72. lazy val localWarehousePath: String
    Definition Classes
    DataFrameSuiteBaseLike
  73. def lowPriorityTypeCheckedConstraint[A, B](implicit equivalenceOfB: Equivalence[B], ev: <:<[A, B]): CanEqual[A, B]
    Definition Classes
    TripleEquals → TripleEqualsSupport
  74. def markup: Documenter
    Attributes
    protected
    Definition Classes
    AnyFunSuiteLike → Documenting
  75. val maxUnequalRowsToShow: Int
    Definition Classes
    DataFrameSuiteBaseLike
  76. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  77. def nestedSuites: IndexedSeq[Suite]
    Definition Classes
    Suite
  78. def note: Notifier
    Attributes
    protected
    Definition Classes
    AnyFunSuiteLike → Notifying
  79. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @IntrinsicCandidate()
  80. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @IntrinsicCandidate()
  81. def pending: Assertion with PendingStatement
    Definition Classes
    Assertions
  82. def pendingUntilFixed(f: ⇒ Unit)(implicit pos: Position): Assertion with PendingStatement
    Definition Classes
    Assertions
  83. final def registerIgnoredTest(testText: String, testTags: Tag*)(testFun: ⇒ Any)(implicit pos: Position): Unit
    Definition Classes
    AnyFunSuiteLike → TestRegistration
  84. final def registerTest(testText: String, testTags: Tag*)(testFun: ⇒ Any)(implicit pos: Position): Unit
    Definition Classes
    AnyFunSuiteLike → TestRegistration
  85. def rerunner: Option[String]
    Definition Classes
    Suite
  86. implicit def reuseContextIfPossible: Boolean
    Attributes
    protected
    Definition Classes
    SharedSparkContext
  87. def run(testName: Option[String], args: Args): Status
    Definition Classes
    BeforeAndAfterAll → SuiteMixin
  88. def runNestedSuites(args: Args): Status
    Attributes
    protected
    Definition Classes
    Suite
  89. def runTest(testName: String, args: Args): Status
    Attributes
    protected
    Definition Classes
    AnyFunSuiteLike → TestSuite → Suite
  90. def runTests(testName: Option[String], args: Args): Status
    Attributes
    protected
    Definition Classes
    AnyFunSuiteLike → Suite
  91. def sc: SparkContext
    Definition Classes
    SharedSparkContext → SparkContextProvider
  92. def setup(sc: SparkContext): Unit

    Setup work to be called when creating a new SparkContext.

    Setup work to be called when creating a new SparkContext. Default implementation currently sets a checkpoint directory.

    This _should_ be called by the context provider automatically.

    Definition Classes
    SparkContextProvider
  93. lazy val spark: SparkSession
    Definition Classes
    DataFrameSuiteBaseLike
    Annotations
    @transient()
  94. def sqlBeforeAllTestCases(): Unit
    Definition Classes
    DataFrameSuiteBaseLike
  95. lazy val sqlContext: SQLContext
    Definition Classes
    DataFrameSuiteBaseLike
    Annotations
    @transient()
  96. final val succeed: Assertion
    Definition Classes
    Assertions
  97. def suiteId: String
    Definition Classes
    Suite
  98. def suiteName: String
    Definition Classes
    Suite
  99. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  100. def tags: Map[String, Set[String]]
    Definition Classes
    AnyFunSuiteLike → Suite
  101. lazy val tempDir: File
    Definition Classes
    DataFrameSuiteBaseLike
  102. def test(testName: String, testTags: Tag*)(testFun: ⇒ Any)(implicit pos: Position): Unit
    Attributes
    protected
    Definition Classes
    AnyFunSuiteLike
  103. def testCodegenOnly(testName: String, testTags: Tag*)(testFun: ⇒ Any)(implicit pos: Position): Unit
  104. def testCombined(testName: String, testTags: Tag*)(testFun: ⇒ Any)(implicit pos: Position): Unit
  105. def testDataFor(testName: String, theConfigMap: ConfigMap): TestData
    Definition Classes
    AnyFunSuiteLike → Suite
  106. def testNames: Set[String]
    Definition Classes
    AnyFunSuiteLike → Suite
  107. def testNonCodegen(testName: String, testTags: Tag*)(testFun: ⇒ Any)(implicit pos: Position): Unit
  108. def testsFor(unit: Unit): Unit
    Attributes
    protected
    Definition Classes
    AnyFunSuiteLike
  109. def toString(): String
    Definition Classes
    AnyFunSuite → AnyFunSuiteLike → AnyRef → Any
  110. def typeCheckedConstraint[A, B](implicit equivalenceOfA: Equivalence[A], ev: <:<[B, A]): CanEqual[A, B]
    Definition Classes
    TripleEquals → TripleEqualsSupport
  111. implicit def unconstrainedEquality[A, B](implicit equalityOfA: Equality[A]): CanEqual[A, B]
    Definition Classes
    TripleEquals → TripleEqualsSupport
  112. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  113. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  114. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  115. def withClue[T](clue: Any)(fun: ⇒ T): T
    Definition Classes
    Assertions
  116. def withFixture(test: NoArgTest): Outcome
    Attributes
    protected
    Definition Classes
    TestSuite
  117. def withSQLConf(pairs: (String, String)*)(f: ⇒ Unit): Unit

    Sets all SQL configurations specified in pairs, calls f, and then restores all SQL configurations.

    Sets all SQL configurations specified in pairs, calls f, and then restores all SQL configurations. Taken from Spark SQLHelper.

    Attributes
    protected
    Definition Classes
    DataFrameSuiteBase

Deprecated Value Members

  1. def assertDataFrameApproximateEquals(expected: DataFrame, result: DataFrame, tol: Double): Unit

    Compares if two DataFrames are equal, checks that the schemas are the same.

    Compares if two DataFrames are equal, checks that the schemas are the same. When comparing inexact fields uses tol.

    tol

    max acceptable tolerance for numeric (between(0, 1)) & timestamp (millis).

    Definition Classes
    DataFrameSuiteBaseLike
    Annotations
    @deprecated
    Deprecated

    (Since version 1.5.0) Use assertDataFrameApproximateEquals with timestamp tolerance

  2. def conversionCheckedConstraint[A, B](implicit equivalenceOfA: Equivalence[A], cnv: (B) ⇒ A): CanEqual[A, B]
    Definition Classes
    TripleEquals → TripleEqualsSupport
    Annotations
    @deprecated
    Deprecated

    (Since version 3.1.0) The conversionCheckedConstraint method has been deprecated and will be removed in a future version of ScalaTest. It is no longer needed now that the deprecation period of ConversionCheckedTripleEquals has expired. It will not be replaced.

  3. def convertEquivalenceToAToBConversionConstraint[A, B](equivalenceOfB: Equivalence[B])(implicit ev: (A) ⇒ B): CanEqual[A, B]
    Definition Classes
    TripleEquals → TripleEqualsSupport
    Annotations
    @deprecated
    Deprecated

    (Since version 3.1.0) The convertEquivalenceToAToBConversionConstraint method has been deprecated and will be removed in a future version of ScalaTest. It is no longer needed now that the deprecation period of ConversionCheckedTripleEquals has expired. It will not be replaced.

  4. def convertEquivalenceToBToAConversionConstraint[A, B](equivalenceOfA: Equivalence[A])(implicit ev: (B) ⇒ A): CanEqual[A, B]
    Definition Classes
    TripleEquals → TripleEqualsSupport
    Annotations
    @deprecated
    Deprecated

    (Since version 3.1.0) The convertEquivalenceToBToAConversionConstraint method has been deprecated and will be removed in a future version of ScalaTest. It is no longer needed now that the deprecation period of ConversionCheckedTripleEquals has expired. It will not be replaced.

  5. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] ) @Deprecated
    Deprecated
  6. def lowPriorityConversionCheckedConstraint[A, B](implicit equivalenceOfB: Equivalence[B], cnv: (A) ⇒ B): CanEqual[A, B]
    Definition Classes
    TripleEquals → TripleEqualsSupport
    Annotations
    @deprecated
    Deprecated

    (Since version 3.1.0) The lowPriorityConversionCheckedConstraint method has been deprecated and will be removed in a future version of ScalaTest. It is no longer needed now that the deprecation period of ConversionCheckedTripleEquals has expired. It will not be replaced.

  7. final val styleName: String
    Definition Classes
    AnyFunSuiteLike → Suite
    Annotations
    @deprecated
    Deprecated

    (Since version 3.1.0) The styleName lifecycle method has been deprecated and will be removed in a future version of ScalaTest with no replacement.

Inherited from DataFrameSuiteBase

Inherited from DataFrameSuiteBaseLike

Inherited from SharedSparkContext

Inherited from SparkContextProvider

Inherited from BeforeAndAfterAll

Inherited from SuiteMixin

Inherited from TestSuite

Inherited from TestSuiteLike

Inherited from AnyFunSuite

Inherited from AnyFunSuiteLike

Inherited from Documenting

Inherited from Alerting

Inherited from Notifying

Inherited from Informing

Inherited from TestRegistration

Inherited from org.scalatest.TestSuite

Inherited from Suite

Inherited from Serializable

Inherited from Serializable

Inherited from Assertions

Inherited from TripleEquals

Inherited from TripleEqualsSupport

Inherited from AnyRef

Inherited from Any

Ungrouped