co.cask.cdap.etl.spark

Interface SparkCollection<T>

Type Parameters:

T - type of elements in the spark collection

All Known Implementing Classes:

DStreamCollection, RDDCollection

public interface SparkCollection<T>

Abstraction over different types of spark collections with common shared operations on those collections. For example, both JavaRDD and JavaDStream support the flatMap operation, but don't share a higher interface. This allows us to perform a common operation and have the implementation take care of whether it happens on an RDD, DStream, DataFrame, etc. Also handles other Hydrator specific operations, such as performing a SparkCompute operation.

Method Summary

Modifier and Type	Method and Description
SparkCollection<RecordInfo<Object>>	aggregate(StageSpec stageSpec, Integer partitions, StageStatisticsCollector collector)
SparkCollection<T>	cache()
<U> SparkCollection<U>	compute(StageSpec stageSpec, SparkCompute<T,U> compute)
<U> SparkCollection<U>	flatMap(StageSpec stageSpec, org.apache.spark.api.java.function.FlatMapFunction<T,U> function)
<K,V> SparkPairCollection<K,V>	flatMapToPair(org.apache.spark.api.java.function.PairFlatMapFunction<T,K,V> function)
<C> C	getUnderlying()
SparkCollection<RecordInfo<Object>>	multiOutputTransform(StageSpec stageSpec, StageStatisticsCollector collector)
void	publishAlerts(StageSpec stageSpec, StageStatisticsCollector collector)
void	store(StageSpec stageSpec, org.apache.spark.api.java.function.PairFlatMapFunction<T,Object,Object> sinkFunction)
void	store(StageSpec stageSpec, SparkSink<T> sink)
SparkCollection<RecordInfo<Object>>	transform(StageSpec stageSpec, StageStatisticsCollector collector)
SparkCollection<T>	union(SparkCollection<T> other)
SparkCollection<T>	window(StageSpec stageSpec, Windower windower)

Method Detail

getUnderlying

<C> C getUnderlying()

cache

SparkCollection<T> cache()

union

SparkCollection<T> union(SparkCollection<T> other)

transform

SparkCollection<RecordInfo<Object>> transform(StageSpec stageSpec,
                                              StageStatisticsCollector collector)

multiOutputTransform

SparkCollection<RecordInfo<Object>> multiOutputTransform(StageSpec stageSpec,
                                                         StageStatisticsCollector collector)

flatMap

<U> SparkCollection<U> flatMap(StageSpec stageSpec,
                               org.apache.spark.api.java.function.FlatMapFunction<T,U> function)

aggregate

SparkCollection<RecordInfo<Object>> aggregate(StageSpec stageSpec,
                                              @Nullable
                                              Integer partitions,
                                              StageStatisticsCollector collector)

flatMapToPair

<K,V> SparkPairCollection<K,V> flatMapToPair(org.apache.spark.api.java.function.PairFlatMapFunction<T,K,V> function)

compute

<U> SparkCollection<U> compute(StageSpec stageSpec,
                               SparkCompute<T,U> compute)
                        throws Exception

Throws:

Exception

store

void store(StageSpec stageSpec,
           org.apache.spark.api.java.function.PairFlatMapFunction<T,Object,Object> sinkFunction)

store

void store(StageSpec stageSpec,
           SparkSink<T> sink)
    throws Exception

Throws:

Exception

publishAlerts

void publishAlerts(StageSpec stageSpec,
                   StageStatisticsCollector collector)
            throws Exception

Throws:

Exception

window

SparkCollection<T> window(StageSpec stageSpec,
                          Windower windower)