t

com.holdenkarau.spark.testing

ScalaDataFrameSuiteBase

trait ScalaDataFrameSuiteBase extends AnyFunSuite with DataFrameSuiteBase

Base trait for testing Spark DataFrames in Scala.

Linear Supertypes
DataFrameSuiteBase, DataFrameSuiteBaseLike, SharedSparkContext, SparkContextProvider, BeforeAndAfterAll, SuiteMixin, TestSuite, TestSuiteLike, AnyFunSuite, AnyFunSuiteLike, Documenting, Alerting, Notifying, Informing, TestRegistration, org.scalatest.TestSuite, Suite, Serializable, Serializable, Assertions, TripleEquals, TripleEqualsSupport, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. ScalaDataFrameSuiteBase
  2. DataFrameSuiteBase
  3. DataFrameSuiteBaseLike
  4. SharedSparkContext
  5. SparkContextProvider
  6. BeforeAndAfterAll
  7. SuiteMixin
  8. TestSuite
  9. TestSuiteLike
  10. AnyFunSuite
  11. AnyFunSuiteLike
  12. Documenting
  13. Alerting
  14. Notifying
  15. Informing
  16. TestRegistration
  17. TestSuite
  18. Suite
  19. Serializable
  20. Serializable
  21. Assertions
  22. TripleEquals
  23. TripleEqualsSupport
  24. AnyRef
  25. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Type Members

  1. class CheckingEqualizer[L] extends AnyRef
    Definition Classes
    TripleEqualsSupport
  2. class Equalizer[L] extends AnyRef
    Definition Classes
    TripleEqualsSupport
  3. trait NoArgTest extends () ⇒ Outcome with TestData
    Attributes
    protected
    Definition Classes
    TestSuite

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. def !==[T](right: Spread[T]): TripleEqualsInvocationOnSpread[T]
    Definition Classes
    TripleEqualsSupport
  3. def !==(right: Null): TripleEqualsInvocation[Null]
    Definition Classes
    TripleEqualsSupport
  4. def !==[T](right: T): TripleEqualsInvocation[T]
    Definition Classes
    TripleEqualsSupport
  5. final def ##(): Int
    Definition Classes
    AnyRef → Any
  6. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  7. def ===[T](right: Spread[T]): TripleEqualsInvocationOnSpread[T]
    Definition Classes
    TripleEqualsSupport
  8. def ===(right: Null): TripleEqualsInvocation[Null]
    Definition Classes
    TripleEqualsSupport
  9. def ===[T](right: T): TripleEqualsInvocation[T]
    Definition Classes
    TripleEqualsSupport
  10. def afterAll(): Unit
    Definition Classes
    DataFrameSuiteBase → SharedSparkContext → BeforeAndAfterAll
  11. def alert: Alerter
    Attributes
    protected
    Definition Classes
    AnyFunSuiteLike → Alerting
  12. def appID: String
    Definition Classes
    SparkContextProvider
  13. def approxEquals(r1: Row, r2: Row, tol: Double): Boolean
    Definition Classes
    DataFrameSuiteBaseLike
  14. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  15. def assert[U](message: String, expected: U, actual: U)(implicit CT: ClassTag[U]): Unit
    Definition Classes
    TestSuite → TestSuiteLike
  16. def assert[U](expected: U, actual: U)(implicit CT: ClassTag[U]): Unit
    Definition Classes
    TestSuite → TestSuiteLike
  17. macro def assert(condition: Boolean, clue: Any)(implicit prettifier: Prettifier, pos: Position): Assertion
    Definition Classes
    Assertions
  18. macro def assert(condition: Boolean)(implicit prettifier: Prettifier, pos: Position): Assertion
    Definition Classes
    Assertions
  19. macro def assertCompiles(code: String)(implicit pos: Position): Assertion
    Definition Classes
    Assertions
  20. def assertDataFrameApproximateEquals(expected: DataFrame, result: DataFrame, tol: Double): Unit

    Compares if two DataFrames are equal, checks that the schemas are the same.

    Compares if two DataFrames are equal, checks that the schemas are the same. When comparing inexact fields uses tol.

    tol

    max acceptable tolerance, should be less than 1.

    Definition Classes
    DataFrameSuiteBaseLike
  21. def assertDataFrameDataEquals(expected: DataFrame, result: DataFrame): Unit

    Compares if two DataFrames are equal without caring about order of rows, by finding elements in one DataFrame that is not in the other.

    Compares if two DataFrames are equal without caring about order of rows, by finding elements in one DataFrame that is not in the other. The resulting DataFrame should be empty inferring the two DataFrames have the same elements. Does not compare the schema.

    Definition Classes
    DataFrameSuiteBaseLike
  22. def assertDataFrameEquals(expected: DataFrame, result: DataFrame): Unit

    Compares if two DataFrames are equal, checks the schema and then if that matches checks if the rows are equal.

    Compares if two DataFrames are equal, checks the schema and then if that matches checks if the rows are equal.

    Definition Classes
    DataFrameSuiteBaseLike
  23. def assertDataFrameNoOrderEquals(expected: DataFrame, result: DataFrame): Unit

    Compares if two DataFrames are equal without caring about order of rows, by finding elements in one DataFrame that is not in the other.

    Compares if two DataFrames are equal without caring about order of rows, by finding elements in one DataFrame that is not in the other. The resulting DataFrame should be empty inferring the two DataFrames have the same elements. Also verifies that the schema is identical.

    Definition Classes
    DataFrameSuiteBaseLike
  24. macro def assertDoesNotCompile(code: String)(implicit pos: Position): Assertion
    Definition Classes
    Assertions
  25. def assertEmpty[U](arr: Array[U])(implicit CT: ClassTag[U]): Unit
    Definition Classes
    TestSuite → TestSuiteLike
  26. def assertResult(expected: Any)(actual: Any)(implicit prettifier: Prettifier, pos: Position): Assertion
    Definition Classes
    Assertions
  27. def assertResult(expected: Any, clue: Any)(actual: Any)(implicit prettifier: Prettifier, pos: Position): Assertion
    Definition Classes
    Assertions
  28. def assertSmallDataFrameDataEquals(expected: DataFrame, result: DataFrame): Unit

    Compares if two DataFrames are equal without caring about order of rows, by finding elements in one DataFrame that is not in the other.

    Compares if two DataFrames are equal without caring about order of rows, by finding elements in one DataFrame that is not in the other. Similar to the function assertDataFrameDataEquals but for small DataFrame that can be collected in memory for the comparison.

    Definition Classes
    DataFrameSuiteBaseLike
  29. def assertThrows[T <: AnyRef](f: ⇒ Any)(implicit classTag: ClassTag[T], pos: Position): Assertion
    Definition Classes
    Assertions
  30. def assertTrue(expected: Boolean): Unit
    Definition Classes
    TestSuite → TestSuiteLike
  31. macro def assertTypeError(code: String)(implicit pos: Position): Assertion
    Definition Classes
    Assertions
  32. macro def assume(condition: Boolean, clue: Any)(implicit prettifier: Prettifier, pos: Position): Assertion
    Definition Classes
    Assertions
  33. macro def assume(condition: Boolean)(implicit prettifier: Prettifier, pos: Position): Assertion
    Definition Classes
    Assertions
  34. def beforeAll(): Unit
    Definition Classes
    DataFrameSuiteBase → SharedSparkContext → BeforeAndAfterAll
  35. def builder(): Builder

    Constructs a configuration for hive or iceberg, where the metastore is located in a temp directory.

    Constructs a configuration for hive or iceberg, where the metastore is located in a temp directory.

    Definition Classes
    DataFrameSuiteBaseLike
  36. def cancel(cause: Throwable)(implicit pos: Position): Nothing
    Definition Classes
    Assertions
  37. def cancel(message: String, cause: Throwable)(implicit pos: Position): Nothing
    Definition Classes
    Assertions
  38. def cancel(message: String)(implicit pos: Position): Nothing
    Definition Classes
    Assertions
  39. def cancel()(implicit pos: Position): Nothing
    Definition Classes
    Assertions
  40. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native() @HotSpotIntrinsicCandidate()
  41. def conf: SparkConf
    Definition Classes
    SparkContextProvider
  42. def convertEquivalenceToAToBConstraint[A, B](equivalenceOfB: Equivalence[B])(implicit ev: <:<[A, B]): CanEqual[A, B]
    Definition Classes
    TripleEquals → TripleEqualsSupport
  43. def convertEquivalenceToBToAConstraint[A, B](equivalenceOfA: Equivalence[A])(implicit ev: <:<[B, A]): CanEqual[A, B]
    Definition Classes
    TripleEquals → TripleEqualsSupport
  44. def convertToCheckingEqualizer[T](left: T): CheckingEqualizer[T]
    Definition Classes
    TripleEquals → TripleEqualsSupport
  45. implicit def convertToEqualizer[T](left: T): Equalizer[T]
    Definition Classes
    TripleEquals → TripleEqualsSupport
  46. def defaultEquality[A]: Equality[A]
    Definition Classes
    TripleEqualsSupport
  47. def enableHiveSupport: Boolean
    Attributes
    protected
    Definition Classes
    DataFrameSuiteBaseLike
  48. def enableIcebergSupport: Boolean
    Attributes
    protected
    Definition Classes
    DataFrameSuiteBaseLike
  49. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  50. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  51. final def execute(testName: String, configMap: ConfigMap, color: Boolean, durations: Boolean, shortstacks: Boolean, fullstacks: Boolean, stats: Boolean): Unit
    Definition Classes
    Suite
  52. def expectedTestCount(filter: Filter): Int
    Definition Classes
    Suite
  53. def fail(message: String): Unit
    Definition Classes
    TestSuite → TestSuiteLike
  54. def fail(cause: Throwable)(implicit pos: Position): Nothing
    Definition Classes
    Assertions
  55. def fail(message: String, cause: Throwable)(implicit pos: Position): Nothing
    Definition Classes
    Assertions
  56. def fail(message: String)(implicit pos: Position): Nothing
    Definition Classes
    Assertions
  57. def fail()(implicit pos: Position): Nothing
    Definition Classes
    Assertions
  58. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  59. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  60. val icebergWarehouse: String
    Definition Classes
    DataFrameSuiteBaseLike
  61. def ignore(testName: String, testTags: Tag*)(testFun: ⇒ Any)(implicit pos: Position): Unit
    Attributes
    protected
    Definition Classes
    AnyFunSuiteLike
  62. implicit def impSqlContext: SQLContext
    Attributes
    protected
    Definition Classes
    DataFrameSuiteBaseLike
  63. def info: Informer
    Attributes
    protected
    Definition Classes
    AnyFunSuiteLike → Informing
  64. def intercept[T <: AnyRef](f: ⇒ Any)(implicit classTag: ClassTag[T], pos: Position): T
    Definition Classes
    Assertions
  65. val invokeBeforeAllAndAfterAllEvenIfNoTestsAreExpected: Boolean
    Definition Classes
    BeforeAndAfterAll
  66. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  67. lazy val localMetastorePath: String
    Definition Classes
    DataFrameSuiteBaseLike
  68. lazy val localWarehousePath: String
    Definition Classes
    DataFrameSuiteBaseLike
  69. def lowPriorityTypeCheckedConstraint[A, B](implicit equivalenceOfB: Equivalence[B], ev: <:<[A, B]): CanEqual[A, B]
    Definition Classes
    TripleEquals → TripleEqualsSupport
  70. def markup: Documenter
    Attributes
    protected
    Definition Classes
    AnyFunSuiteLike → Documenting
  71. val maxUnequalRowsToShow: Int
    Definition Classes
    DataFrameSuiteBaseLike
  72. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  73. def nestedSuites: IndexedSeq[Suite]
    Definition Classes
    Suite
  74. def note: Notifier
    Attributes
    protected
    Definition Classes
    AnyFunSuiteLike → Notifying
  75. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  76. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  77. def pending: Assertion with PendingStatement
    Definition Classes
    Assertions
  78. def pendingUntilFixed(f: ⇒ Unit)(implicit pos: Position): Assertion with PendingStatement
    Definition Classes
    Assertions
  79. final def registerIgnoredTest(testText: String, testTags: Tag*)(testFun: ⇒ Any)(implicit pos: Position): Unit
    Definition Classes
    AnyFunSuiteLike → TestRegistration
  80. final def registerTest(testText: String, testTags: Tag*)(testFun: ⇒ Any)(implicit pos: Position): Unit
    Definition Classes
    AnyFunSuiteLike → TestRegistration
  81. def rerunner: Option[String]
    Definition Classes
    Suite
  82. implicit def reuseContextIfPossible: Boolean
    Attributes
    protected
    Definition Classes
    SharedSparkContext
  83. def run(testName: Option[String], args: Args): Status
    Definition Classes
    BeforeAndAfterAll → SuiteMixin
  84. def runNestedSuites(args: Args): Status
    Attributes
    protected
    Definition Classes
    Suite
  85. def runTest(testName: String, args: Args): Status
    Attributes
    protected
    Definition Classes
    AnyFunSuiteLike → TestSuite → Suite
  86. def runTests(testName: Option[String], args: Args): Status
    Attributes
    protected
    Definition Classes
    AnyFunSuiteLike → Suite
  87. def sc: SparkContext
    Definition Classes
    SharedSparkContext → SparkContextProvider
  88. def setup(sc: SparkContext): Unit

    Setup work to be called when creating a new SparkContext.

    Setup work to be called when creating a new SparkContext. Default implementation currently sets a checkpoint directory.

    This _should_ be called by the context provider automatically.

    Definition Classes
    SparkContextProvider
  89. lazy val spark: SparkSession
    Definition Classes
    DataFrameSuiteBaseLike
    Annotations
    @transient()
  90. def sqlBeforeAllTestCases(): Unit
    Definition Classes
    DataFrameSuiteBaseLike
  91. lazy val sqlContext: SQLContext
    Definition Classes
    DataFrameSuiteBaseLike
    Annotations
    @transient()
  92. final val succeed: Assertion
    Definition Classes
    Assertions
  93. def suiteId: String
    Definition Classes
    Suite
  94. def suiteName: String
    Definition Classes
    Suite
  95. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  96. def tags: Map[String, Set[String]]
    Definition Classes
    AnyFunSuiteLike → Suite
  97. lazy val tempDir: File
    Definition Classes
    DataFrameSuiteBaseLike
  98. def test(testName: String, testTags: Tag*)(testFun: ⇒ Any)(implicit pos: Position): Unit
    Attributes
    protected
    Definition Classes
    AnyFunSuiteLike
  99. def testCodegenOnly(testName: String, testTags: Tag*)(testFun: ⇒ Any)(implicit pos: Position): Unit
  100. def testCombined(testName: String, testTags: Tag*)(testFun: ⇒ Any)(implicit pos: Position): Unit
  101. def testDataFor(testName: String, theConfigMap: ConfigMap): TestData
    Definition Classes
    AnyFunSuiteLike → Suite
  102. def testNames: Set[String]
    Definition Classes
    AnyFunSuiteLike → Suite
  103. def testNonCodegen(testName: String, testTags: Tag*)(testFun: ⇒ Any)(implicit pos: Position): Unit
  104. def testsFor(unit: Unit): Unit
    Attributes
    protected
    Definition Classes
    AnyFunSuiteLike
  105. def toString(): String
    Definition Classes
    AnyFunSuite → AnyFunSuiteLike → AnyRef → Any
  106. def typeCheckedConstraint[A, B](implicit equivalenceOfA: Equivalence[A], ev: <:<[B, A]): CanEqual[A, B]
    Definition Classes
    TripleEquals → TripleEqualsSupport
  107. implicit def unconstrainedEquality[A, B](implicit equalityOfA: Equality[A]): CanEqual[A, B]
    Definition Classes
    TripleEquals → TripleEqualsSupport
  108. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  109. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  110. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  111. def withClue[T](clue: Any)(fun: ⇒ T): T
    Definition Classes
    Assertions
  112. def withFixture(test: NoArgTest): Outcome
    Attributes
    protected
    Definition Classes
    TestSuite
  113. def withSQLConf(pairs: (String, String)*)(f: ⇒ Unit): Unit

    Sets all SQL configurations specified in pairs, calls f, and then restores all SQL configurations.

    Sets all SQL configurations specified in pairs, calls f, and then restores all SQL configurations. Taken from Spark SQLHelper.

    Attributes
    protected
    Definition Classes
    DataFrameSuiteBase

Deprecated Value Members

  1. def conversionCheckedConstraint[A, B](implicit equivalenceOfA: Equivalence[A], cnv: (B) ⇒ A): CanEqual[A, B]
    Definition Classes
    TripleEquals → TripleEqualsSupport
    Annotations
    @deprecated
    Deprecated

    (Since version 3.1.0) The conversionCheckedConstraint method has been deprecated and will be removed in a future version of ScalaTest. It is no longer needed now that the deprecation period of ConversionCheckedTripleEquals has expired. It will not be replaced.

  2. def convertEquivalenceToAToBConversionConstraint[A, B](equivalenceOfB: Equivalence[B])(implicit ev: (A) ⇒ B): CanEqual[A, B]
    Definition Classes
    TripleEquals → TripleEqualsSupport
    Annotations
    @deprecated
    Deprecated

    (Since version 3.1.0) The convertEquivalenceToAToBConversionConstraint method has been deprecated and will be removed in a future version of ScalaTest. It is no longer needed now that the deprecation period of ConversionCheckedTripleEquals has expired. It will not be replaced.

  3. def convertEquivalenceToBToAConversionConstraint[A, B](equivalenceOfA: Equivalence[A])(implicit ev: (B) ⇒ A): CanEqual[A, B]
    Definition Classes
    TripleEquals → TripleEqualsSupport
    Annotations
    @deprecated
    Deprecated

    (Since version 3.1.0) The convertEquivalenceToBToAConversionConstraint method has been deprecated and will be removed in a future version of ScalaTest. It is no longer needed now that the deprecation period of ConversionCheckedTripleEquals has expired. It will not be replaced.

  4. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] ) @Deprecated
    Deprecated
  5. def lowPriorityConversionCheckedConstraint[A, B](implicit equivalenceOfB: Equivalence[B], cnv: (A) ⇒ B): CanEqual[A, B]
    Definition Classes
    TripleEquals → TripleEqualsSupport
    Annotations
    @deprecated
    Deprecated

    (Since version 3.1.0) The lowPriorityConversionCheckedConstraint method has been deprecated and will be removed in a future version of ScalaTest. It is no longer needed now that the deprecation period of ConversionCheckedTripleEquals has expired. It will not be replaced.

  6. final val styleName: String
    Definition Classes
    AnyFunSuiteLike → Suite
    Annotations
    @deprecated
    Deprecated

    (Since version 3.1.0) The styleName lifecycle method has been deprecated and will be removed in a future version of ScalaTest with no replacement.

Inherited from DataFrameSuiteBase

Inherited from DataFrameSuiteBaseLike

Inherited from SharedSparkContext

Inherited from SparkContextProvider

Inherited from BeforeAndAfterAll

Inherited from SuiteMixin

Inherited from TestSuite

Inherited from TestSuiteLike

Inherited from AnyFunSuite

Inherited from AnyFunSuiteLike

Inherited from Documenting

Inherited from Alerting

Inherited from Notifying

Inherited from Informing

Inherited from TestRegistration

Inherited from org.scalatest.TestSuite

Inherited from Suite

Inherited from Serializable

Inherited from Serializable

Inherited from Assertions

Inherited from TripleEquals

Inherited from TripleEqualsSupport

Inherited from AnyRef

Inherited from Any

Ungrouped