c

com.holdenkarau.spark.testing

JavaStreamingSuiteBase

class JavaStreamingSuiteBase extends JavaSuiteBase with StreamingSuiteCommon

This is the base trait for Spark Streaming testsuite. This provides basic functionality to run user-defined set of input on user-defined stream operations, and verify the output matches as expected.

This implementation is designed to work with JUnit for java users.

Note: this always uses the manual clock to control Spark Streaming's batches.

Linear Supertypes
StreamingSuiteCommon, Logging, Logging, JavaSuiteBase, SharedJavaSparkContext, SparkContextProvider, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. JavaStreamingSuiteBase
  2. StreamingSuiteCommon
  3. Logging
  4. Logging
  5. JavaSuiteBase
  6. SharedJavaSparkContext
  7. SparkContextProvider
  8. AnyRef
  9. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new JavaStreamingSuiteBase()

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. def actuallyWait: Boolean
    Definition Classes
    StreamingSuiteCommon
  5. def appID: String
    Definition Classes
    SparkContextProvider
  6. def appName: String
    Definition Classes
    SparkContextProvider
  7. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  8. def batchDuration: Duration
    Definition Classes
    StreamingSuiteCommon
  9. def beforeAllTestCasesHook(): Unit

    Hooks for setup code that needs to be executed/torn down in order with SparkContexts

    Hooks for setup code that needs to be executed/torn down in order with SparkContexts

    Attributes
    protected[testing]
    Definition Classes
    SharedJavaSparkContext
  10. lazy val checkpointDir: String
    Definition Classes
    StreamingSuiteCommon
  11. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native() @IntrinsicCandidate()
  12. def compareArrays[U](i1: Array[U], i2: Array[U], sorted: Boolean = false)(implicit arg0: ClassTag[U]): Unit

    Utility wrapper around assertArrayEquals that resolves the types

    Utility wrapper around assertArrayEquals that resolves the types

    Definition Classes
    JavaSuiteBase
  13. def conf: SparkConf
    Definition Classes
    JavaStreamingSuiteBase → StreamingSuiteCommon → SparkContextProvider
  14. def copyAndSort[U](array: Array[U])(implicit arg0: ClassTag[U]): Array[U]
    Definition Classes
    JavaSuiteBase
  15. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  16. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  17. val eventuallyTimeout: Timeout
    Definition Classes
    StreamingSuiteCommon
  18. def framework: String
    Definition Classes
    StreamingSuiteCommon
  19. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @IntrinsicCandidate()
  20. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @IntrinsicCandidate()
  21. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  22. def initializeLogIfNecessary(isInterpreter: Boolean): Unit
    Attributes
    protected
    Definition Classes
    Logging
  23. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  24. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  25. def jsc(): JavaSparkContext
    Definition Classes
    SharedJavaSparkContext
  26. def log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  27. def logDebug(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  28. def logDebug(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  29. def logError(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  30. def logError(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  31. def logInfo(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  32. def logInfo(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  33. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  34. def logTrace(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  35. def logTrace(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  36. def logWarning(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  37. def logWarning(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  38. def master: String
    Definition Classes
    StreamingSuiteCommon
  39. def maxWaitTimeMillis: Int
    Definition Classes
    StreamingSuiteCommon
  40. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  41. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @IntrinsicCandidate()
  42. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @IntrinsicCandidate()
  43. def numInputPartitions: Int
    Definition Classes
    StreamingSuiteCommon
  44. def runBefore(): Unit
    Definition Classes
    SharedJavaSparkContext
    Annotations
    @Before()
  45. def sc(): SparkContext
    Definition Classes
    SharedJavaSparkContext → SparkContextProvider
  46. def setup(sc: SparkContext): Unit

    Setup work to be called when creating a new SparkContext.

    Setup work to be called when creating a new SparkContext. Default implementation currently sets a checkpoint directory.

    This _should_ be called by the context provider automatically.

    Definition Classes
    SparkContextProvider
  47. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  48. def testOperation[U, V, W](input1: List[List[U]], input2: List[List[V]], operation: Function2[JavaDStream[U], JavaDStream[V], JavaDStream[W]], expectedOutput: List[List[W]], ordered: Boolean): Unit

    Test binary DStream operation with two lists of inputs, with number of batches to run same as the number of input values.

    Test binary DStream operation with two lists of inputs, with number of batches to run same as the number of input values. The size of the two input lists should be equal.

    Each input micro-batch is a list of values or as null to simulate empty batch.

    input1

    First sequence of input collections

    input2

    Second sequence of input collections

    operation

    Binary DStream operation to be applied to the 2 inputs

    expectedOutput

    Sequence of expected output collections

    ordered

    Compare output values with expected output values within the same output batch ordered or unOrdered. Comparing doubles may not work well in case of unordered.

  49. def testOperation[U, V, W](input1: List[List[U]], input2: List[List[V]], operation: Function2[JavaDStream[U], JavaDStream[V], JavaDStream[W]], expectedOutput: List[List[W]]): Unit

    Test binary DStream operation with two lists of inputs, with number of batches to run same as the number of input values.

    Test binary DStream operation with two lists of inputs, with number of batches to run same as the number of input values. The size of the two input lists should be equal.

    Each input micro-batch is a list of values or as null to simulate empty batch.

    input1

    First sequence of input collections

    input2

    Second sequence of input collections

    operation

    Binary DStream operation to be applied to the 2 inputs

    expectedOutput

    Sequence of expected output collections

  50. def testOperation[U, V](input: List[List[U]], operation: Function[JavaDStream[U], JavaDStream[V]], expectedOutput: List[List[V]], ordered: Boolean): Unit

    Test unary DStream operation with a list of inputs, with number of batches to run same as the number of input values.

    Test unary DStream operation with a list of inputs, with number of batches to run same as the number of input values.

    Each input micro-batch is a list of values or as null to simulate empty batch.

    input

    Sequence of input collections

    operation

    Binary DStream operation to be applied to the 2 inputs

    expectedOutput

    Sequence of expected output collections

    ordered

    Compare output values with expected output values within the same output batch ordered or unordered. Comparing doubles may not work well in case of unordered.

  51. def testOperation[U, V](input: List[List[U]], operation: Function[JavaDStream[U], JavaDStream[V]], expectedOutput: List[List[V]]): Unit

    Test unary DStream operation with a list of inputs, with number of batches to run same as the number of input values.

    Test unary DStream operation with a list of inputs, with number of batches to run same as the number of input values.

    Each input micro-batch is a list of values or as null to simulate empty batch.

    input

    Sequence of input collections

    operation

    Binary DStream operation to be applied to the 2 inputs

    expectedOutput

    Sequence of expected output collections

  52. def toString(): String
    Definition Classes
    AnyRef → Any
  53. def useManualClock: Boolean
    Definition Classes
    StreamingSuiteCommon
  54. def verifyOutput[V](output: Seq[Seq[V]], expectedOutput: Seq[Seq[V]], ordered: Boolean)(implicit arg0: ClassTag[V]): Unit

    Verify whether the output values after running a DStream operation is same as the expected output values, by comparing the output collections either as lists (order matters) or sets (order does not matter)

  55. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  56. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  57. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] ) @Deprecated
    Deprecated

Inherited from StreamingSuiteCommon

Inherited from Logging

Inherited from Logging

Inherited from JavaSuiteBase

Inherited from SharedJavaSparkContext

Inherited from SparkContextProvider

Inherited from AnyRef

Inherited from Any

Ungrouped