co.cask.cdap.etl.spark

Class Compat

java.lang.Object

co.cask.cdap.etl.spark.Compat

public final class Compat
extends Object

Utility class to handle incompatibilities between Spark1 and Spark2. All hydrator-spark-core modules must have this class with the exact same method signatures. Incompatibilities are in a few places. Should not contain any classes from Spark streaming. FlatMapFunction and PairFlatMapFunction in Spark2 was changed to return an Iterator instead of an Iterable This class contains convert methods to change a FlatMapFunc and PairFlatMapFunc into the corresponding Spark version specific class. Outer join methods in Spark1 use guava's Optional whereas Spark2 uses its own Optional.

Field Summary

Fields

Modifier and Type	Field and Description
static String	SPARK_COMPAT

Method Summary

Modifier and Type	Method and Description
static <T,R> org.apache.spark.api.java.function.FlatMapFunction<T,R>	convert(FlatMapFunc<T,R> func)
static <T,K,V> org.apache.spark.api.java.function.PairFlatMapFunction<T,K,V>	convert(PairFlatMapFunc<T,K,V> func)
static <T> void	foreachRDD(org.apache.spark.streaming.api.java.JavaDStream<T> stream, org.apache.spark.api.java.function.Function2<org.apache.spark.api.java.JavaRDD<T>,org.apache.spark.streaming.Time,Void> func)
static <K,V1,V2> org.apache.spark.api.java.JavaPairRDD<K,scala.Tuple2<com.google.common.base.Optional<V1>,com.google.common.base.Optional<V2>>>	fullOuterJoin(org.apache.spark.api.java.JavaPairRDD<K,V1> left, org.apache.spark.api.java.JavaPairRDD<K,V2> right)
static <K,V1,V2> org.apache.spark.api.java.JavaPairRDD<K,scala.Tuple2<com.google.common.base.Optional<V1>,com.google.common.base.Optional<V2>>>	fullOuterJoin(org.apache.spark.api.java.JavaPairRDD<K,V1> left, org.apache.spark.api.java.JavaPairRDD<K,V2> right, int numPartitions)
static <K,V1,V2> org.apache.spark.api.java.JavaPairRDD<K,scala.Tuple2<V1,com.google.common.base.Optional<V2>>>	leftOuterJoin(org.apache.spark.api.java.JavaPairRDD<K,V1> left, org.apache.spark.api.java.JavaPairRDD<K,V2> right)
static <K,V1,V2> org.apache.spark.api.java.JavaPairRDD<K,scala.Tuple2<V1,com.google.common.base.Optional<V2>>>	leftOuterJoin(org.apache.spark.api.java.JavaPairRDD<K,V1> left, org.apache.spark.api.java.JavaPairRDD<K,V2> right, int numPartitions)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

SPARK_COMPAT

public static final String SPARK_COMPAT

See Also:

Constant Field Values

Method Detail

convert

public static <T,R> org.apache.spark.api.java.function.FlatMapFunction<T,R> convert(FlatMapFunc<T,R> func)

convert

public static <T,K,V> org.apache.spark.api.java.function.PairFlatMapFunction<T,K,V> convert(PairFlatMapFunc<T,K,V> func)

foreachRDD

public static <T> void foreachRDD(org.apache.spark.streaming.api.java.JavaDStream<T> stream,
                                  org.apache.spark.api.java.function.Function2<org.apache.spark.api.java.JavaRDD<T>,org.apache.spark.streaming.Time,Void> func)

leftOuterJoin

public static <K,V1,V2> org.apache.spark.api.java.JavaPairRDD<K,scala.Tuple2<V1,com.google.common.base.Optional<V2>>> leftOuterJoin(org.apache.spark.api.java.JavaPairRDD<K,V1> left,
                                                                                                                                    org.apache.spark.api.java.JavaPairRDD<K,V2> right)

leftOuterJoin

public static <K,V1,V2> org.apache.spark.api.java.JavaPairRDD<K,scala.Tuple2<V1,com.google.common.base.Optional<V2>>> leftOuterJoin(org.apache.spark.api.java.JavaPairRDD<K,V1> left,
                                                                                                                                    org.apache.spark.api.java.JavaPairRDD<K,V2> right,
                                                                                                                                    int numPartitions)

fullOuterJoin

public static <K,V1,V2> org.apache.spark.api.java.JavaPairRDD<K,scala.Tuple2<com.google.common.base.Optional<V1>,com.google.common.base.Optional<V2>>> fullOuterJoin(org.apache.spark.api.java.JavaPairRDD<K,V1> left,
                                                                                                                                                                     org.apache.spark.api.java.JavaPairRDD<K,V2> right)

fullOuterJoin

public static <K,V1,V2> org.apache.spark.api.java.JavaPairRDD<K,scala.Tuple2<com.google.common.base.Optional<V1>,com.google.common.base.Optional<V2>>> fullOuterJoin(org.apache.spark.api.java.JavaPairRDD<K,V1> left,
                                                                                                                                                                     org.apache.spark.api.java.JavaPairRDD<K,V2> right,
                                                                                                                                                                     int numPartitions)