co.cask.cdap.etl.spark.streaming.function

Class DynamicSparkCompute<T,U>

java.lang.Object

co.cask.cdap.etl.api.batch.SparkCompute<T,U>

co.cask.cdap.etl.spark.streaming.function.DynamicSparkCompute<T,U>

Type Parameters:

T - type of object in the input collection

U - type of object in the output collection

All Implemented Interfaces:

PipelineConfigurable, Serializable

public class DynamicSparkCompute<T,U>
extends SparkCompute<T,U>

This class is required to make sure that macro substitution occurs each time a pipeline is run instead of just the first time the pipeline is run. Without this, the SparkCompute plugin gets serialized into the checkpoint and re-loaded every subsequent time with the properties from the first run.

See Also:

Serialized Form

Field Summary

Fields inherited from class co.cask.cdap.etl.api.batch.SparkCompute

PLUGIN_TYPE

Constructor Summary

Constructors

Constructor and Description
DynamicSparkCompute(DynamicDriverContext dynamicDriverContext, SparkCompute<T,U> compute)

Method Summary

Modifier and Type	Method and Description
void	initialize(SparkExecutionPluginContext context)
org.apache.spark.api.java.JavaRDD<U>	transform(SparkExecutionPluginContext context, org.apache.spark.api.java.JavaRDD<T> input)

Methods inherited from class co.cask.cdap.etl.api.batch.SparkCompute

configurePipeline

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DynamicSparkCompute

public DynamicSparkCompute(DynamicDriverContext dynamicDriverContext,
                           SparkCompute<T,U> compute)

Method Detail

initialize

public void initialize(SparkExecutionPluginContext context)
                throws Exception

Overrides:

initialize in class SparkCompute<T,U>

Throws:

Exception

transform

public org.apache.spark.api.java.JavaRDD<U> transform(SparkExecutionPluginContext context,
                                                      org.apache.spark.api.java.JavaRDD<T> input)
                                               throws Exception

Specified by:

transform in class SparkCompute<T,U>

Throws:

Exception