co.cask.cdap.etl.spark.streaming.function

Class DynamicJoinMerge<JOIN_KEY,INPUT_RECORD,OUT>

java.lang.Object

co.cask.cdap.etl.spark.streaming.function.DynamicJoinMerge<JOIN_KEY,INPUT_RECORD,OUT>

Type Parameters:

JOIN_KEY - type of join key

INPUT_RECORD - type of input object

OUT - type of output object

All Implemented Interfaces:

Serializable, org.apache.spark.api.java.function.Function2<org.apache.spark.api.java.JavaPairRDD<JOIN_KEY,List<JoinElement<INPUT_RECORD>>>,org.apache.spark.streaming.Time,org.apache.spark.api.java.JavaRDD<OUT>>

public class DynamicJoinMerge<JOIN_KEY,INPUT_RECORD,OUT>
extends Object
implements org.apache.spark.api.java.function.Function2<org.apache.spark.api.java.JavaPairRDD<JOIN_KEY,List<JoinElement<INPUT_RECORD>>>,org.apache.spark.streaming.Time,org.apache.spark.api.java.JavaRDD<OUT>>

Serializable function that can be used to perform the merge part of a Joiner. Dynamically instantiates the relevant Joiner plugin to ensure that code changes are picked up and to ensure that macro substitution occurs.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
DynamicJoinMerge(DynamicDriverContext dynamicDriverContext)

Method Summary

Modifier and Type	Method and Description
org.apache.spark.api.java.JavaRDD<OUT>	call(org.apache.spark.api.java.JavaPairRDD<JOIN_KEY,List<JoinElement<INPUT_RECORD>>> input, org.apache.spark.streaming.Time batchTime)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DynamicJoinMerge

public DynamicJoinMerge(DynamicDriverContext dynamicDriverContext)

Method Detail

call

public org.apache.spark.api.java.JavaRDD<OUT> call(org.apache.spark.api.java.JavaPairRDD<JOIN_KEY,List<JoinElement<INPUT_RECORD>>> input,
                                                   org.apache.spark.streaming.Time batchTime)
                                            throws Exception

Specified by:

call in interface org.apache.spark.api.java.function.Function2<org.apache.spark.api.java.JavaPairRDD<JOIN_KEY,List<JoinElement<INPUT_RECORD>>>,org.apache.spark.streaming.Time,org.apache.spark.api.java.JavaRDD<OUT>>

Throws:

Exception