co.cask.cdap.etl.spark

Interface SparkPairCollection<K,V>

Type Parameters:

K - type of key in the collection

V - type of value in the collection

All Known Implementing Classes:

PairDStreamCollection, PairRDDCollection

public interface SparkPairCollection<K,V>

Abstraction over different types of spark pair collections with common shared operations on those collections. For example, both JavaPairRDD and JavaPairDStream support the flatMap operation, but don't share a higher interface. This allows us to perform a common operation and have the implementation take care of whether it happens on a PairRDD, PairDStream, or whatever other object is being used.

Method Summary

Modifier and Type	Method and Description
<T> SparkCollection<T>	flatMap(org.apache.spark.api.java.function.FlatMapFunction<scala.Tuple2<K,V>,T> function)
<T> SparkPairCollection<K,scala.Tuple2<com.google.common.base.Optional<V>,com.google.common.base.Optional<T>>>	fullOuterJoin(SparkPairCollection<K,T> other)
<T> SparkPairCollection<K,scala.Tuple2<com.google.common.base.Optional<V>,com.google.common.base.Optional<T>>>	fullOuterJoin(SparkPairCollection<K,T> other, int numPartitions)
<C> C	getUnderlying()
<T> SparkPairCollection<K,scala.Tuple2<V,T>>	join(SparkPairCollection<K,T> other)
<T> SparkPairCollection<K,scala.Tuple2<V,T>>	join(SparkPairCollection<K,T> other, int numPartitions)
<T> SparkPairCollection<K,scala.Tuple2<V,com.google.common.base.Optional<T>>>	leftOuterJoin(SparkPairCollection<K,T> other)
<T> SparkPairCollection<K,scala.Tuple2<V,com.google.common.base.Optional<T>>>	leftOuterJoin(SparkPairCollection<K,T> other, int numPartitions)
<T> SparkPairCollection<K,T>	mapValues(org.apache.spark.api.java.function.Function<V,T> function)

Method Detail

getUnderlying

<C> C getUnderlying()

flatMap

<T> SparkCollection<T> flatMap(org.apache.spark.api.java.function.FlatMapFunction<scala.Tuple2<K,V>,T> function)

mapValues

<T> SparkPairCollection<K,T> mapValues(org.apache.spark.api.java.function.Function<V,T> function)

join

<T> SparkPairCollection<K,scala.Tuple2<V,T>> join(SparkPairCollection<K,T> other)

join

<T> SparkPairCollection<K,scala.Tuple2<V,T>> join(SparkPairCollection<K,T> other,
                                                  int numPartitions)

leftOuterJoin

<T> SparkPairCollection<K,scala.Tuple2<V,com.google.common.base.Optional<T>>> leftOuterJoin(SparkPairCollection<K,T> other)

leftOuterJoin

<T> SparkPairCollection<K,scala.Tuple2<V,com.google.common.base.Optional<T>>> leftOuterJoin(SparkPairCollection<K,T> other,
                                                                                            int numPartitions)

fullOuterJoin

<T> SparkPairCollection<K,scala.Tuple2<com.google.common.base.Optional<V>,com.google.common.base.Optional<T>>> fullOuterJoin(SparkPairCollection<K,T> other)

fullOuterJoin

<T> SparkPairCollection<K,scala.Tuple2<com.google.common.base.Optional<V>,com.google.common.base.Optional<T>>> fullOuterJoin(SparkPairCollection<K,T> other,
                                                                                                                             int numPartitions)