co.cask.cdap.etl.spark.batch

Class BatchSparkPipelineDriver

java.lang.Object

co.cask.cdap.etl.spark.SparkPipelineRunner

co.cask.cdap.etl.spark.batch.BatchSparkPipelineDriver

All Implemented Interfaces:

JavaSparkMain, TxRunnable, Serializable

public class BatchSparkPipelineDriver
extends SparkPipelineRunner
implements JavaSparkMain, TxRunnable

Batch Spark pipeline driver.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
BatchSparkPipelineDriver()

Method Summary

Modifier and Type	Method and Description
protected SparkPairCollection<Object,Object>	addJoinKey(StageSpec stageSpec, String inputStageName, SparkCollection<Object> inputCollection, StageStatisticsCollector collector)
protected SparkCollection<RecordInfo<Object>>	getSource(StageSpec stageSpec, StageStatisticsCollector collector)
protected SparkCollection<Object>	mergeJoinResults(StageSpec stageSpec, SparkPairCollection<Object,List<JoinElement<Object>>> joinedInputs, StageStatisticsCollector collector)
void	run(DatasetContext context)
void	run(JavaSparkExecutionContext sec)

Methods inherited from class co.cask.cdap.etl.spark.SparkPipelineRunner

runPipeline

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BatchSparkPipelineDriver

public BatchSparkPipelineDriver()

Method Detail

getSource

protected SparkCollection<RecordInfo<Object>> getSource(StageSpec stageSpec,
                                                        StageStatisticsCollector collector)

Specified by:

getSource in class SparkPipelineRunner

addJoinKey

protected SparkPairCollection<Object,Object> addJoinKey(StageSpec stageSpec,
                                                        String inputStageName,
                                                        SparkCollection<Object> inputCollection,
                                                        StageStatisticsCollector collector)
                                                 throws Exception

Specified by:

addJoinKey in class SparkPipelineRunner

Throws:

Exception

mergeJoinResults

protected SparkCollection<Object> mergeJoinResults(StageSpec stageSpec,
                                                   SparkPairCollection<Object,List<JoinElement<Object>>> joinedInputs,
                                                   StageStatisticsCollector collector)
                                            throws Exception

Specified by:

mergeJoinResults in class SparkPipelineRunner

Throws:

Exception

run

public void run(JavaSparkExecutionContext sec)
         throws Exception

Specified by:

run in interface JavaSparkMain

Throws:

Exception

run

public void run(DatasetContext context)
         throws Exception

Specified by:

run in interface TxRunnable

Throws:

Exception