com.stratio.deep.context

Class DeepSparkContext

java.lang.Object

org.apache.spark.api.java.JavaSparkContext

com.stratio.deep.context.DeepSparkContext

public class DeepSparkContext
extends org.apache.spark.api.java.JavaSparkContext

Entry point to the Cassandra-aware Spark context.

Author:

Luca Rosellini

Constructor Summary

Constructors

Constructor and Description
DeepSparkContext(org.apache.spark.SparkContext sc) Overridden superclass constructor.
DeepSparkContext(String master, String appName) Overridden superclass constructor.
DeepSparkContext(String master, String appName, String sparkHome, String jarFile) Overridden superclass constructor.
DeepSparkContext(String master, String appName, String sparkHome, String[] jars) Overridden superclass constructor.
DeepSparkContext(String master, String appName, String sparkHome, String[] jars, Map<String,String> environment) Overridden superclass constructor.

Method Summary

Methods

Modifier and Type	Method and Description
<T extends IDeepType> CassandraRDD<T>	cassandraEntityRDD(IDeepJobConfig<T> config) Builds a new testentity based CassandraEntityRDD.
CassandraRDD<Cells>	cassandraGenericRDD(IDeepJobConfig<Cells> config) Builds a new generic (cell based) CassandraGenericRDD.
<T> CassandraJavaRDD<T>	cassandraJavaRDD(IDeepJobConfig<T> config) Builds a new CassandraJavaRDD.
org.apache.spark.api.java.JavaDoubleRDD	union(org.apache.spark.api.java.JavaDoubleRDD... arg0)
<K,V> org.apache.spark.api.java.JavaPairRDD<K,V>	union(org.apache.spark.api.java.JavaPairRDD<K,V>... arg0)
<T> org.apache.spark.api.java.JavaRDD<T>	union(org.apache.spark.api.java.JavaRDD<T>... arg0)

Methods inherited from class org.apache.spark.api.java.JavaSparkContext

accumulable, accumulator, accumulator, accumulator, addFile, addJar, broadcast, cancelAllJobs, cancelJobGroup, checkpointFile, clearCallSite, clearFiles, clearJars, clearJobGroup, doubleAccumulator, env, fromSparkContext, getCheckpointDir, getConf, getLocalProperty, getSparkHome, hadoopConfiguration, hadoopFile, hadoopFile, hadoopRDD, hadoopRDD, intAccumulator, jarOfClass, jarOfObject, newAPIHadoopFile, newAPIHadoopRDD, objectFile, objectFile, parallelize, parallelize, parallelizeDoubles, parallelizeDoubles, parallelizePairs, parallelizePairs, sc, sequenceFile, sequenceFile, setCallSite, setCheckpointDir, setJobGroup, setLocalProperty, stop, textFile, textFile, toSparkContext, union, union, union

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DeepSparkContext

public DeepSparkContext(org.apache.spark.SparkContext sc)

Overridden superclass constructor.

Parameters:

sc - an already created spark context.

DeepSparkContext

public DeepSparkContext(String master,
                String appName)

Overridden superclass constructor.

Parameters:

master - the url of the master node.

appName - the name of the application.

DeepSparkContext

public DeepSparkContext(String master,
                String appName,
                String sparkHome,
                String jarFile)

Overridden superclass constructor.

Parameters:

master - the url of the master node.

appName - the name of the application.

sparkHome - the spark home folder.

jarFile - the jar file to serialize and send to all the cluster nodes.

DeepSparkContext

public DeepSparkContext(String master,
                String appName,
                String sparkHome,
                String[] jars)

Overridden superclass constructor.

Parameters:

master - the url of the master node.

appName - the name of the application.

sparkHome - the spark home folder.

jars - the jar file(s) to serialize and send to all the cluster nodes.

DeepSparkContext

public DeepSparkContext(String master,
                String appName,
                String sparkHome,
                String[] jars,
                Map<String,String> environment)

Overridden superclass constructor.

Parameters:

master - the url of the master node.

appName - the name of the application.

sparkHome - the spark home folder.

jars - the jar file(s) to serialize and send to all the cluster nodes.

environment - a map of environment variables.

Method Detail

cassandraJavaRDD

public <T> CassandraJavaRDD<T> cassandraJavaRDD(IDeepJobConfig<T> config)

Builds a new CassandraJavaRDD.

Parameters:

config - the deep configuration object to use to create the new RDD.

Returns:

a new CassandraJavaRDD

cassandraEntityRDD

public <T extends IDeepType> CassandraRDD<T> cassandraEntityRDD(IDeepJobConfig<T> config)

Builds a new testentity based CassandraEntityRDD.

Parameters:

config - the deep configuration object to use to create the new RDD.

Returns:

a new entity-based CassandraRDD

cassandraGenericRDD

public CassandraRDD<Cells> cassandraGenericRDD(IDeepJobConfig<Cells> config)

Builds a new generic (cell based) CassandraGenericRDD.

Parameters:

config - the deep configuration object to use to create the new RDD.

Returns:

a new generic CassandraRDD.

union

public <T> org.apache.spark.api.java.JavaRDD<T> union(org.apache.spark.api.java.JavaRDD<T>... arg0)

union

public org.apache.spark.api.java.JavaDoubleRDD union(org.apache.spark.api.java.JavaDoubleRDD... arg0)

union

public <K,V> org.apache.spark.api.java.JavaPairRDD<K,V> union(org.apache.spark.api.java.JavaPairRDD<K,V>... arg0)