co.cask.cdap.etl.spark.streaming

Class PairDStreamCollection<K,V>

java.lang.Object

co.cask.cdap.etl.spark.streaming.PairDStreamCollection<K,V>

Type Parameters:

K - type of key in the collection

V - type of value in the collection

All Implemented Interfaces:

SparkPairCollection<K,V>

public class PairDStreamCollection<K,V>
extends Object
implements SparkPairCollection<K,V>

JavaPairDStream backed SparkPairCollection

Constructor Summary

Constructors

Constructor and Description
PairDStreamCollection(JavaSparkExecutionContext sec, org.apache.spark.streaming.api.java.JavaPairDStream<K,V> pairStream)

Method Summary

Modifier and Type	Method and Description
<T> SparkCollection<T>	flatMap(org.apache.spark.api.java.function.FlatMapFunction<scala.Tuple2<K,V>,T> function)
<T> SparkPairCollection<K,scala.Tuple2<com.google.common.base.Optional<V>,com.google.common.base.Optional<T>>>	fullOuterJoin(SparkPairCollection<K,T> other)
<T> SparkPairCollection<K,scala.Tuple2<com.google.common.base.Optional<V>,com.google.common.base.Optional<T>>>	fullOuterJoin(SparkPairCollection<K,T> other, int numPartitions)
org.apache.spark.streaming.api.java.JavaPairDStream<K,V>	getUnderlying()
<T> SparkPairCollection<K,scala.Tuple2<V,T>>	join(SparkPairCollection<K,T> other)
<T> SparkPairCollection<K,scala.Tuple2<V,T>>	join(SparkPairCollection<K,T> other, int numPartitions)
<T> SparkPairCollection<K,scala.Tuple2<V,com.google.common.base.Optional<T>>>	leftOuterJoin(SparkPairCollection<K,T> other)
<T> SparkPairCollection<K,scala.Tuple2<V,com.google.common.base.Optional<T>>>	leftOuterJoin(SparkPairCollection<K,T> other, int numPartitions)
<T> SparkPairCollection<K,T>	mapValues(org.apache.spark.api.java.function.Function<V,T> function)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

PairDStreamCollection

public PairDStreamCollection(JavaSparkExecutionContext sec,
                             org.apache.spark.streaming.api.java.JavaPairDStream<K,V> pairStream)

Method Detail

getUnderlying

public org.apache.spark.streaming.api.java.JavaPairDStream<K,V> getUnderlying()

Specified by:

getUnderlying in interface SparkPairCollection<K,V>

flatMap

public <T> SparkCollection<T> flatMap(org.apache.spark.api.java.function.FlatMapFunction<scala.Tuple2<K,V>,T> function)

Specified by:

flatMap in interface SparkPairCollection<K,V>

mapValues

public <T> SparkPairCollection<K,T> mapValues(org.apache.spark.api.java.function.Function<V,T> function)

Specified by:

mapValues in interface SparkPairCollection<K,V>

join

public <T> SparkPairCollection<K,scala.Tuple2<V,T>> join(SparkPairCollection<K,T> other)

Specified by:

join in interface SparkPairCollection<K,V>

join

public <T> SparkPairCollection<K,scala.Tuple2<V,T>> join(SparkPairCollection<K,T> other,
                                                         int numPartitions)

Specified by:

join in interface SparkPairCollection<K,V>

leftOuterJoin

public <T> SparkPairCollection<K,scala.Tuple2<V,com.google.common.base.Optional<T>>> leftOuterJoin(SparkPairCollection<K,T> other)

Specified by:

leftOuterJoin in interface SparkPairCollection<K,V>

leftOuterJoin

public <T> SparkPairCollection<K,scala.Tuple2<V,com.google.common.base.Optional<T>>> leftOuterJoin(SparkPairCollection<K,T> other,
                                                                                                   int numPartitions)

Specified by:

leftOuterJoin in interface SparkPairCollection<K,V>

fullOuterJoin

public <T> SparkPairCollection<K,scala.Tuple2<com.google.common.base.Optional<V>,com.google.common.base.Optional<T>>> fullOuterJoin(SparkPairCollection<K,T> other)

Specified by:

fullOuterJoin in interface SparkPairCollection<K,V>

fullOuterJoin

public <T> SparkPairCollection<K,scala.Tuple2<com.google.common.base.Optional<V>,com.google.common.base.Optional<T>>> fullOuterJoin(SparkPairCollection<K,T> other,
                                                                                                                                    int numPartitions)

Specified by:

fullOuterJoin in interface SparkPairCollection<K,V>