Type Members

1.  class BatchCountListener extends StreamingListener
2.  class Column extends ColumnGenerator

   Previously ColumnGenerator.

   Previously ColumnGenerator. Allows the user to specify a generator for a specific column.

3.  abstract class ColumnGenerator extends Serializable

   ColumnGenerator - prevously Column; it is now the base class for all ColumnGenerators.

4.  class ColumnList extends ColumnGenerator

   ColumnList allows users to specify custom generators for a list of columns inside a StructType column.

5.  trait DataFrameSuiteBase extends TestSuite with SharedSparkContext with DataFrameSuiteBaseLike

   :: Experimental :: Base class for testing Spark DataFrames.

6.  trait DataFrameSuiteBaseLike extends SparkContextProvider with TestSuiteLike with Serializable
7.  trait DatasetSuiteBase extends DataFrameSuiteBase with DatasetSuiteBaseLike
8.  trait DatasetSuiteBaseLike extends DataFrameSuiteBaseLike
9.  class JavaDataFrameSuiteBase extends SharedJavaSparkContext with DataFrameSuiteBaseLike with JavaTestSuite
10.  class JavaDatasetSuiteBase extends JavaDataFrameSuiteBase with DatasetSuiteBaseLike with Serializable
11.  class JavaStreamingSuiteBase extends JavaSuiteBase with StreamingSuiteCommon

    This is the base trait for Spark Streaming testsuite.

    This is the base trait for Spark Streaming testsuite. This provides basic functionality to run user-defined set of input on user-defined stream operations, and verify the output matches as expected.

    This implementation is designed to work with JUnit for java users.

    Note: this always uses the manual clock to control Spark Streaming's batches.

12.  class JavaSuiteBase extends SharedJavaSparkContext
13.  trait JavaTestSuite extends TestSuiteLike
14.  trait LocalSparkContext extends BeforeAndAfterEach with BeforeAndAfterAll

    Manages a local sc SparkContext variable, correctly stopping it after each test.

15.  trait PerTestSparkContext extends LocalSparkContext with BeforeAndAfterEach with SparkContextProvider

    Provides a local sc SparkContext variable, correctly stopping it after each test.

    Provides a local sc SparkContext variable, correctly stopping it after each test. The stopping logic is provided in LocalSparkContext.

16.  class PerfListener extends SparkListener

    This listener collects basic execution time information to be used in micro type performance tests.

    This listener collects basic execution time information to be used in micro type performance tests. Be careful imposing strict limits as there is a large amount of variability.

17.  trait Prettify extends AnyRef
18.  trait RDDComparisons extends RDDComparisonsLike with TestSuite
19.  trait RDDComparisonsLike extends TestSuiteLike
20.  trait ScalaDataFrameSuiteBase extends AnyFunSuite with DataFrameSuiteBase

    Base trait for testing Spark DataFrames in Scala.

21.  class SharedJavaSparkContext extends SparkContextProvider

    Shares a local SparkContext between all tests in a suite and closes it at the end

22.  trait SharedSparkContext extends BeforeAndAfterAll with SparkContextProvider

    Shares a local SparkContext between all tests in a suite and closes it at the end.

    Shares a local SparkContext between all tests in a suite and closes it at the end. You can share between suites by enabling reuseContextIfPossible.

23.  trait SparkContextProvider extends AnyRef
24.  trait StreamingActionBase extends StreamingSuiteBase

    Methods for testing Spark actions.

    Methods for testing Spark actions. Because actions don't return a DStream, you will need to verify the results of your test against mocks.

25.  trait StreamingSuiteBase extends BeforeAndAfterAll with Logging with StreamingSuiteCommon with SharedSparkContext

    This is the base trait for Spark Streaming testsuites.

    This is the base trait for Spark Streaming testsuites. This provides basic functionality to run user-defined set of input on user-defined stream operations, and verify the output.

26.  trait StructuredStreamingBase extends DataFrameSuiteBase with StructuredStreamingBaseLike

    Early Experimental Structured Streaming Base.

27.  trait StructuredStreamingBaseLike extends SparkContextProvider with TestSuiteLike with Serializable
28.  class TestInputStream[T] extends FriendlyInputDStream[T]

    This is a input stream just for the testsuites.

    This is a input stream just for the testsuites. This is equivalent to a checkpointable, replayable, reliable message queue like Kafka. It requires a sequence as input, and returns the i_th element at the i_th batch under manual clock.

    Based on TestInputStream class from TestSuiteBase in the Apache Spark project.

29.  class TestOutputStream[T] extends Serializable

    This is a output stream just for testing.

    This is a output stream just for testing.

    The buffer contains a sequence of RDD's, each containing a sequence of items

30.  trait TestSuite extends TestSuiteLike
31.  trait TestSuiteLike extends AnyRef