org.apache.spark.launcher

Class SparkLauncher

java.lang.Object

org.apache.spark.launcher.SparkLauncher

public class SparkLauncher
extends java.lang.Object

Launcher for Spark applications.

Use this class to start Spark applications programmatically. The class uses a builder pattern to allow clients to configure the Spark application and launch it as a child process.

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	DRIVER_EXTRA_CLASSPATH Configuration key for the driver class path.
static java.lang.String	DRIVER_EXTRA_JAVA_OPTIONS Configuration key for the driver VM options.
static java.lang.String	DRIVER_EXTRA_LIBRARY_PATH Configuration key for the driver native library path.
static java.lang.String	DRIVER_MEMORY Configuration key for the driver memory.
static java.lang.String	EXECUTOR_CORES Configuration key for the number of executor CPU cores.
static java.lang.String	EXECUTOR_EXTRA_CLASSPATH Configuration key for the executor class path.
static java.lang.String	EXECUTOR_EXTRA_JAVA_OPTIONS Configuration key for the executor VM options.
static java.lang.String	EXECUTOR_EXTRA_LIBRARY_PATH Configuration key for the executor native library path.
static java.lang.String	EXECUTOR_MEMORY Configuration key for the executor memory.
static java.lang.String	SPARK_MASTER The Spark master.

Constructor Summary

Constructors

Constructor and Description
SparkLauncher()
SparkLauncher(java.util.Map<java.lang.String,java.lang.String> env) Creates a launcher that will set the given environment variables in the child.

Method Summary

Methods

Modifier and Type	Method and Description
SparkLauncher	addAppArgs(java.lang.String... args) Adds command line arguments for the application.
SparkLauncher	addFile(java.lang.String file) Adds a file to be submitted with the application.
SparkLauncher	addJar(java.lang.String jar) Adds a jar file to be submitted with the application.
SparkLauncher	addPyFile(java.lang.String file) Adds a python file / zip / egg to be submitted with the application.
SparkLauncher	addSparkArg(java.lang.String arg) Adds a no-value argument to the Spark invocation.
SparkLauncher	addSparkArg(java.lang.String name, java.lang.String value) Adds an argument with a value to the Spark invocation.
java.lang.Process	launch() Launches a sub-process that will start the configured Spark application.
SparkLauncher	setAppName(java.lang.String appName) Set the application name.
SparkLauncher	setAppResource(java.lang.String resource) Set the main application resource.
SparkLauncher	setConf(java.lang.String key, java.lang.String value) Set a single configuration value for the application.
SparkLauncher	setDeployMode(java.lang.String mode) Set the deploy mode for the application.
SparkLauncher	setJavaHome(java.lang.String javaHome) Set a custom JAVA_HOME for launching the Spark application.
SparkLauncher	setMainClass(java.lang.String mainClass) Sets the application class name for Java/Scala applications.
SparkLauncher	setMaster(java.lang.String master) Set the Spark master for the application.
SparkLauncher	setPropertiesFile(java.lang.String path) Set a custom properties file with Spark configuration for the application.
SparkLauncher	setSparkHome(java.lang.String sparkHome) Set a custom Spark installation location for the application.
SparkLauncher	setVerbose(boolean verbose) Enables verbose reporting for SparkSubmit.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

SPARK_MASTER

public static final java.lang.String SPARK_MASTER

The Spark master.

See Also:

Constant Field Values

DRIVER_MEMORY

public static final java.lang.String DRIVER_MEMORY

Configuration key for the driver memory.

See Also:

Constant Field Values

DRIVER_EXTRA_CLASSPATH

public static final java.lang.String DRIVER_EXTRA_CLASSPATH

Configuration key for the driver class path.

See Also:

Constant Field Values

DRIVER_EXTRA_JAVA_OPTIONS

public static final java.lang.String DRIVER_EXTRA_JAVA_OPTIONS

Configuration key for the driver VM options.

See Also:

Constant Field Values

DRIVER_EXTRA_LIBRARY_PATH

public static final java.lang.String DRIVER_EXTRA_LIBRARY_PATH

Configuration key for the driver native library path.

See Also:

Constant Field Values

EXECUTOR_MEMORY

public static final java.lang.String EXECUTOR_MEMORY

Configuration key for the executor memory.

See Also:

Constant Field Values

EXECUTOR_EXTRA_CLASSPATH

public static final java.lang.String EXECUTOR_EXTRA_CLASSPATH

Configuration key for the executor class path.

See Also:

Constant Field Values

EXECUTOR_EXTRA_JAVA_OPTIONS

public static final java.lang.String EXECUTOR_EXTRA_JAVA_OPTIONS

Configuration key for the executor VM options.

See Also:

Constant Field Values

EXECUTOR_EXTRA_LIBRARY_PATH

public static final java.lang.String EXECUTOR_EXTRA_LIBRARY_PATH

Configuration key for the executor native library path.

See Also:

Constant Field Values

EXECUTOR_CORES

public static final java.lang.String EXECUTOR_CORES

Configuration key for the number of executor CPU cores.

See Also:

Constant Field Values

Constructor Detail

SparkLauncher

public SparkLauncher()

SparkLauncher

public SparkLauncher(java.util.Map<java.lang.String,java.lang.String> env)

Creates a launcher that will set the given environment variables in the child.

Parameters:

env - Environment variables to set.

Method Detail

setJavaHome

public SparkLauncher setJavaHome(java.lang.String javaHome)

Set a custom JAVA_HOME for launching the Spark application.

Parameters:

javaHome - Path to the JAVA_HOME to use.

Returns:

This launcher.

setSparkHome

public SparkLauncher setSparkHome(java.lang.String sparkHome)

Set a custom Spark installation location for the application.

Parameters:

sparkHome - Path to the Spark installation to use.

Returns:

This launcher.

setPropertiesFile

public SparkLauncher setPropertiesFile(java.lang.String path)

Set a custom properties file with Spark configuration for the application.

Parameters:

path - Path to custom properties file to use.

Returns:

This launcher.

setConf

public SparkLauncher setConf(java.lang.String key,
                    java.lang.String value)

Set a single configuration value for the application.

Parameters:

key - Configuration key.

value - The value to use.

Returns:

This launcher.

setAppName

public SparkLauncher setAppName(java.lang.String appName)

Set the application name.

Parameters:

appName - Application name.

Returns:

This launcher.

setMaster

public SparkLauncher setMaster(java.lang.String master)

Set the Spark master for the application.

Parameters:

master - Spark master.

Returns:

This launcher.

setDeployMode

public SparkLauncher setDeployMode(java.lang.String mode)

Set the deploy mode for the application.

Parameters:

mode - Deploy mode.

Returns:

This launcher.

setAppResource

public SparkLauncher setAppResource(java.lang.String resource)

Set the main application resource. This should be the location of a jar file for Scala/Java applications, or a python script for PySpark applications.

Parameters:

resource - Path to the main application resource.

Returns:

This launcher.

setMainClass

public SparkLauncher setMainClass(java.lang.String mainClass)

Sets the application class name for Java/Scala applications.

Parameters:

mainClass - Application's main class.

Returns:

This launcher.

addSparkArg

public SparkLauncher addSparkArg(java.lang.String arg)

Adds a no-value argument to the Spark invocation. If the argument is known, this method validates whether the argument is indeed a no-value argument, and throws an exception otherwise.

Use this method with caution. It is possible to create an invalid Spark command by passing unknown arguments to this method, since those are allowed for forward compatibility.

Parameters:

arg - Argument to add.

Returns:

This launcher.

addSparkArg

public SparkLauncher addSparkArg(java.lang.String name,
                        java.lang.String value)

Adds an argument with a value to the Spark invocation. If the argument name corresponds to a known argument, the code validates that the argument actually expects a value, and throws an exception otherwise.

It is safe to add arguments modified by other methods in this class (such as setMaster(String) - the last invocation will be the one to take effect.

Use this method with caution. It is possible to create an invalid Spark command by passing unknown arguments to this method, since those are allowed for forward compatibility.

Parameters:

name - Name of argument to add.

value - Value of the argument.

Returns:

This launcher.

addAppArgs

public SparkLauncher addAppArgs(java.lang.String... args)

Adds command line arguments for the application.

Parameters:

args - Arguments to pass to the application's main class.

Returns:

This launcher.

addJar

public SparkLauncher addJar(java.lang.String jar)

Adds a jar file to be submitted with the application.

Parameters:

jar - Path to the jar file.

Returns:

This launcher.

addFile

public SparkLauncher addFile(java.lang.String file)

Adds a file to be submitted with the application.

Parameters:

file - Path to the file.

Returns:

This launcher.

addPyFile

public SparkLauncher addPyFile(java.lang.String file)

Adds a python file / zip / egg to be submitted with the application.

Parameters:

file - Path to the file.

Returns:

This launcher.

setVerbose

public SparkLauncher setVerbose(boolean verbose)

Enables verbose reporting for SparkSubmit.

Parameters:

verbose - Whether to enable verbose output.

Returns:

This launcher.

launch

public java.lang.Process launch()
                         throws java.io.IOException

Launches a sub-process that will start the configured Spark application.

Returns:

A process handle for the Spark app.

Throws:

java.io.IOException