org.apache.spark.sql

Class SparkSession

Object

org.apache.spark.sql.SparkSession

All Implemented Interfaces:

java.io.Closeable, java.io.Serializable, AutoCloseable

@InterfaceStability.Stable
public class SparkSession
extends Object
implements scala.Serializable, java.io.Closeable

The entry point to programming Spark with the Dataset and DataFrame API.

In environments that this has been created upfront (e.g. REPL, notebooks), use the builder to get an existing session:

   SparkSession.builder().getOrCreate()
 

The builder can also be used to create a new session:

   SparkSession.builder()
     .master("local")
     .appName("Word Count")
     .config("spark.some.config.option", "some-value")
     .getOrCreate()
 

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	SparkSession.Builder Builder for SparkSession.
class	SparkSession.implicits$ :: Experimental :: (Scala-specific) Implicit methods available in Scala for converting common Scala objects into DataFrames.

Method Summary

Methods

Modifier and Type	Method and Description
Dataset<Row>	baseRelationToDataFrame(BaseRelation baseRelation) Convert a BaseRelation created for external data sources into a DataFrame.
static SparkSession.Builder	builder() Creates a SparkSession.Builder for constructing a SparkSession.
Catalog	catalog() Interface through which the user may create, drop, alter or query underlying databases, tables, functions etc.
static void	clearActiveSession() Clears the active SparkSession for current thread.
static void	clearDefaultSession() Clears the default SparkSession that is returned by the builder.
void	close() Synonym for stop().
RuntimeConfig	conf() Runtime configuration interface for Spark.
Dataset<Row>	createDataFrame(JavaRDD<?> rdd, Class<?> beanClass) Applies a schema to an RDD of Java Beans.
Dataset<Row>	createDataFrame(JavaRDD<Row> rowRDD, StructType schema) :: DeveloperApi :: Creates a DataFrame from a JavaRDD containing Rows using the given schema.
Dataset<Row>	createDataFrame(java.util.List<?> data, Class<?> beanClass) Applies a schema to a List of Java Beans.
Dataset<Row>	createDataFrame(java.util.List<Row> rows, StructType schema) :: DeveloperApi :: Creates a DataFrame from a List containing Rows using the given schema.
Dataset<Row>	createDataFrame(RDD<?> rdd, Class<?> beanClass) Applies a schema to an RDD of Java Beans.
<A extends scala.Product> Dataset<Row>	createDataFrame(RDD<A> rdd, scala.reflect.api.TypeTags.TypeTag<A> evidence$2) :: Experimental :: Creates a DataFrame from an RDD of Product (e.g.
Dataset<Row>	createDataFrame(RDD<Row> rowRDD, StructType schema) :: DeveloperApi :: Creates a DataFrame from an RDD containing Rows using the given schema.
<A extends scala.Product> Dataset<Row>	createDataFrame(scala.collection.Seq<A> data, scala.reflect.api.TypeTags.TypeTag<A> evidence$3) :: Experimental :: Creates a DataFrame from a local Seq of Product.
<T> Dataset<T>	createDataset(java.util.List<T> data, Encoder<T> evidence$6) :: Experimental :: Creates a Dataset from a List of a given type.
<T> Dataset<T>	createDataset(RDD<T> data, Encoder<T> evidence$5) :: Experimental :: Creates a Dataset from an RDD of a given type.
<T> Dataset<T>	createDataset(scala.collection.Seq<T> data, Encoder<T> evidence$4) :: Experimental :: Creates a Dataset from a local Seq of data of a given type.
Dataset<Row>	emptyDataFrame() Returns a DataFrame with no rows or columns.
<T> Dataset<T>	emptyDataset(Encoder<T> evidence$1) :: Experimental :: Creates a new Dataset of type T containing zero elements.
ExperimentalMethods	experimental() :: Experimental :: A collection of methods that are considered experimental, but can be used to hook into the query planner for advanced functionality.
SparkSession.implicits$	implicits() Accessor for nested Scala object
ExecutionListenerManager	listenerManager() :: Experimental :: An interface to register custom QueryExecutionListeners that listen for execution metrics.
SparkSession	newSession() Start a new session with isolated SQL configurations, temporary tables, registered functions are isolated, but sharing the underlying SparkContext and cached data.
Dataset<Long>	range(long end) :: Experimental :: Creates a Dataset with a single LongType column named id, containing elements in a range from 0 to end (exclusive) with step value 1.
Dataset<Long>	range(long start, long end) :: Experimental :: Creates a Dataset with a single LongType column named id, containing elements in a range from start to end (exclusive) with step value 1.
Dataset<Long>	range(long start, long end, long step) :: Experimental :: Creates a Dataset with a single LongType column named id, containing elements in a range from start to end (exclusive) with a step value.
Dataset<Long>	range(long start, long end, long step, int numPartitions) :: Experimental :: Creates a Dataset with a single LongType column named id, containing elements in a range from start to end (exclusive) with a step value, with partition number specified.
DataFrameReader	read() Returns a DataFrameReader that can be used to read non-streaming data in as a DataFrame.
DataStreamReader	readStream() :: Experimental :: Returns a DataStreamReader that can be used to read streaming data in as a DataFrame.
static void	setActiveSession(SparkSession session) Changes the SparkSession that will be returned in this thread and its children when SparkSession.getOrCreate() is called.
static void	setDefaultSession(SparkSession session) Sets the default SparkSession that is returned by the builder.
SparkContext	sparkContext()
Dataset<Row>	sql(String sqlText) Executes a SQL query using Spark, returning the result as a DataFrame.
SQLContext	sqlContext() A wrapped version of this session in the form of a SQLContext, for backward compatibility.
void	stop() Stop the underlying SparkContext.
StreamingQueryManager	streams() :: Experimental :: Returns a StreamingQueryManager that allows managing all the StreamingQuerys active on this.
Dataset<Row>	table(String tableName) Returns the specified table as a DataFrame.
<T> T	time(scala.Function0<T> f) Executes some code block and prints to stdout the time taken to execute the block.
UDFRegistration	udf() A collection of methods for registering user-defined functions (UDF).
String	version() The version of Spark on which this application is running.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

builder

public static SparkSession.Builder builder()

Creates a SparkSession.Builder for constructing a SparkSession.

Returns:

(undocumented)

Since:

2.0.0

setActiveSession

public static void setActiveSession(SparkSession session)

Changes the SparkSession that will be returned in this thread and its children when SparkSession.getOrCreate() is called. This can be used to ensure that a given thread receives a SparkSession with an isolated session, instead of the global (first created) context.

Parameters:

session - (undocumented)

Since:

2.0.0

clearActiveSession

public static void clearActiveSession()

Clears the active SparkSession for current thread. Subsequent calls to getOrCreate will return the first created context instead of a thread-local override.

Since:

2.0.0

setDefaultSession

public static void setDefaultSession(SparkSession session)

Sets the default SparkSession that is returned by the builder.

Parameters:

session - (undocumented)

Since:

2.0.0

clearDefaultSession

public static void clearDefaultSession()

Clears the default SparkSession that is returned by the builder.

Since:

2.0.0

sparkContext

public SparkContext sparkContext()

version

public String version()

The version of Spark on which this application is running.

Returns:

(undocumented)

Since:

2.0.0

sqlContext

public SQLContext sqlContext()

A wrapped version of this session in the form of a SQLContext, for backward compatibility.

Returns:

(undocumented)

Since:

2.0.0

conf

public RuntimeConfig conf()

Runtime configuration interface for Spark.

This is the interface through which the user can get and set all Spark and Hadoop configurations that are relevant to Spark SQL. When getting the value of a config, this defaults to the value set in the underlying SparkContext, if any.

Returns:

(undocumented)

Since:

2.0.0

listenerManager

public ExecutionListenerManager listenerManager()

:: Experimental :: An interface to register custom QueryExecutionListeners that listen for execution metrics.

Returns:

(undocumented)

Since:

2.0.0

experimental

public ExperimentalMethods experimental()

:: Experimental :: A collection of methods that are considered experimental, but can be used to hook into the query planner for advanced functionality.

Returns:

(undocumented)

Since:

2.0.0

udf

public UDFRegistration udf()

A collection of methods for registering user-defined functions (UDF).

The following example registers a Scala closure as UDF:

   sparkSession.udf.register("myUDF", (arg1: Int, arg2: String) => arg2 + arg1)
 

The following example registers a UDF in Java:

   sparkSession.udf().register("myUDF",
       new UDF2<Integer, String, String>() {
           @Override
           public String call(Integer arg1, String arg2) {
               return arg2 + arg1;
           }
      }, DataTypes.StringType);
 

Or, to use Java 8 lambda syntax:

   sparkSession.udf().register("myUDF",
       (Integer arg1, String arg2) -> arg2 + arg1,
       DataTypes.StringType);
 

Returns:

(undocumented)

Since:

2.0.0

Note:

The user-defined functions must be deterministic. Due to optimization, duplicate invocations may be eliminated or the function may even be invoked more times than it is present in the query.

streams

public StreamingQueryManager streams()

:: Experimental :: Returns a StreamingQueryManager that allows managing all the StreamingQuerys active on this.

Returns:

(undocumented)

Since:

2.0.0

newSession

public SparkSession newSession()

Start a new session with isolated SQL configurations, temporary tables, registered functions are isolated, but sharing the underlying SparkContext and cached data.

Returns:

(undocumented)

Since:

2.0.0

Note:

Other than the SparkContext, all shared state is initialized lazily. This method will force the initialization of the shared state to ensure that parent and child sessions are set up with the same shared state. If the underlying catalog implementation is Hive, this will initialize the metastore, which may take some time.

emptyDataFrame

public Dataset<Row> emptyDataFrame()

Returns a DataFrame with no rows or columns.

Returns:

(undocumented)

Since:

2.0.0

emptyDataset

public <T> Dataset<T> emptyDataset(Encoder<T> evidence$1)

:: Experimental :: Creates a new Dataset of type T containing zero elements.

Parameters:

evidence$1 - (undocumented)

Returns:

2.0.0

createDataFrame

public <A extends scala.Product> Dataset<Row> createDataFrame(RDD<A> rdd,
                                                     scala.reflect.api.TypeTags.TypeTag<A> evidence$2)

:: Experimental :: Creates a DataFrame from an RDD of Product (e.g. case classes, tuples).

Parameters:

rdd - (undocumented)

evidence$2 - (undocumented)

Returns:

(undocumented)

Since:

2.0.0

createDataFrame

public <A extends scala.Product> Dataset<Row> createDataFrame(scala.collection.Seq<A> data,
                                                     scala.reflect.api.TypeTags.TypeTag<A> evidence$3)

:: Experimental :: Creates a DataFrame from a local Seq of Product.

Parameters:

data - (undocumented)

evidence$3 - (undocumented)

Returns:

(undocumented)

Since:

2.0.0

createDataFrame

public Dataset<Row> createDataFrame(RDD<Row> rowRDD,
                           StructType schema)

:: DeveloperApi :: Creates a DataFrame from an RDD containing Rows using the given schema. It is important to make sure that the structure of every Row of the provided RDD matches the provided schema. Otherwise, there will be runtime exception. Example:

  import org.apache.spark.sql._
  import org.apache.spark.sql.types._
  val sparkSession = new org.apache.spark.sql.SparkSession(sc)

  val schema =
    StructType(
      StructField("name", StringType, false) ::
      StructField("age", IntegerType, true) :: Nil)

  val people =
    sc.textFile("examples/src/main/resources/people.txt").map(
      _.split(",")).map(p => Row(p(0), p(1).trim.toInt))
  val dataFrame = sparkSession.createDataFrame(people, schema)
  dataFrame.printSchema
  // root
  // |-- name: string (nullable = false)
  // |-- age: integer (nullable = true)

  dataFrame.createOrReplaceTempView("people")
  sparkSession.sql("select name from people").collect.foreach(println)
 

Parameters:

rowRDD - (undocumented)

schema - (undocumented)

Returns:

(undocumented)

Since:

2.0.0

createDataFrame

public Dataset<Row> createDataFrame(JavaRDD<Row> rowRDD,
                           StructType schema)

:: DeveloperApi :: Creates a DataFrame from a JavaRDD containing Rows using the given schema. It is important to make sure that the structure of every Row of the provided RDD matches the provided schema. Otherwise, there will be runtime exception.

Parameters:

rowRDD - (undocumented)

schema - (undocumented)

Returns:

(undocumented)

Since:

2.0.0

createDataFrame

public Dataset<Row> createDataFrame(java.util.List<Row> rows,
                           StructType schema)

:: DeveloperApi :: Creates a DataFrame from a List containing Rows using the given schema. It is important to make sure that the structure of every Row of the provided List matches the provided schema. Otherwise, there will be runtime exception.

Parameters:

rows - (undocumented)

schema - (undocumented)

Returns:

(undocumented)

Since:

2.0.0

createDataFrame

public Dataset<Row> createDataFrame(RDD<?> rdd,
                           Class<?> beanClass)

Applies a schema to an RDD of Java Beans.

WARNING: Since there is no guaranteed ordering for fields in a Java Bean, SELECT * queries will return the columns in an undefined order.

Parameters:

rdd - (undocumented)

beanClass - (undocumented)

Returns:

(undocumented)

Since:

2.0.0

createDataFrame

public Dataset<Row> createDataFrame(JavaRDD<?> rdd,
                           Class<?> beanClass)

Applies a schema to an RDD of Java Beans.

WARNING: Since there is no guaranteed ordering for fields in a Java Bean, SELECT * queries will return the columns in an undefined order.

Parameters:

rdd - (undocumented)

beanClass - (undocumented)

Returns:

(undocumented)

Since:

2.0.0

createDataFrame

public Dataset<Row> createDataFrame(java.util.List<?> data,
                           Class<?> beanClass)

Applies a schema to a List of Java Beans.

WARNING: Since there is no guaranteed ordering for fields in a Java Bean, SELECT * queries will return the columns in an undefined order.

Parameters:

data - (undocumented)

beanClass - (undocumented)

Returns:

(undocumented)

Since:

1.6.0

baseRelationToDataFrame

public Dataset<Row> baseRelationToDataFrame(BaseRelation baseRelation)

Convert a BaseRelation created for external data sources into a DataFrame.

Parameters:

baseRelation - (undocumented)

Returns:

(undocumented)

Since:

2.0.0

createDataset

public <T> Dataset<T> createDataset(scala.collection.Seq<T> data,
                           Encoder<T> evidence$4)

:: Experimental :: Creates a Dataset from a local Seq of data of a given type. This method requires an encoder (to convert a JVM object of type T to and from the internal Spark SQL representation) that is generally created automatically through implicits from a SparkSession, or can be created explicitly by calling static methods on Encoders.

== Example ==

   import spark.implicits._
   case class Person(name: String, age: Long)
   val data = Seq(Person("Michael", 29), Person("Andy", 30), Person("Justin", 19))
   val ds = spark.createDataset(data)

   ds.show()
   // +-------+---+
   // |   name|age|
   // +-------+---+
   // |Michael| 29|
   // |   Andy| 30|
   // | Justin| 19|
   // +-------+---+
 

Parameters:

data - (undocumented)

evidence$4 - (undocumented)

Returns:

(undocumented)

Since:

2.0.0

createDataset

public <T> Dataset<T> createDataset(RDD<T> data,
                           Encoder<T> evidence$5)

:: Experimental :: Creates a Dataset from an RDD of a given type. This method requires an encoder (to convert a JVM object of type T to and from the internal Spark SQL representation) that is generally created automatically through implicits from a SparkSession, or can be created explicitly by calling static methods on Encoders.

Parameters:

data - (undocumented)

evidence$5 - (undocumented)

Returns:

(undocumented)

Since:

2.0.0

createDataset

public <T> Dataset<T> createDataset(java.util.List<T> data,
                           Encoder<T> evidence$6)

:: Experimental :: Creates a Dataset from a List of a given type. This method requires an encoder (to convert a JVM object of type T to and from the internal Spark SQL representation) that is generally created automatically through implicits from a SparkSession, or can be created explicitly by calling static methods on Encoders.

== Java Example ==

     List<String> data = Arrays.asList("hello", "world");
     Dataset<String> ds = spark.createDataset(data, Encoders.STRING());
 

Parameters:

data - (undocumented)

evidence$6 - (undocumented)

Returns:

(undocumented)

Since:

2.0.0

range

public Dataset<Long> range(long end)

:: Experimental :: Creates a Dataset with a single LongType column named id, containing elements in a range from 0 to end (exclusive) with step value 1.

Parameters:

end - (undocumented)

Returns:

(undocumented)

Since:

2.0.0

range

public Dataset<Long> range(long start,
                  long end)

:: Experimental :: Creates a Dataset with a single LongType column named id, containing elements in a range from start to end (exclusive) with step value 1.

Parameters:

start - (undocumented)

end - (undocumented)

Returns:

(undocumented)

Since:

2.0.0

range

public Dataset<Long> range(long start,
                  long end,
                  long step)

:: Experimental :: Creates a Dataset with a single LongType column named id, containing elements in a range from start to end (exclusive) with a step value.

Parameters:

start - (undocumented)

end - (undocumented)

step - (undocumented)

Returns:

(undocumented)

Since:

2.0.0

range

public Dataset<Long> range(long start,
                  long end,
                  long step,
                  int numPartitions)

:: Experimental :: Creates a Dataset with a single LongType column named id, containing elements in a range from start to end (exclusive) with a step value, with partition number specified.

Parameters:

start - (undocumented)

end - (undocumented)

step - (undocumented)

numPartitions - (undocumented)

Returns:

(undocumented)

Since:

2.0.0

catalog

public Catalog catalog()

Interface through which the user may create, drop, alter or query underlying databases, tables, functions etc.

Returns:

(undocumented)

Since:

2.0.0

table

public Dataset<Row> table(String tableName)

Returns the specified table as a DataFrame.

Parameters:

tableName - (undocumented)

Returns:

(undocumented)

Since:

2.0.0

sql

public Dataset<Row> sql(String sqlText)

Executes a SQL query using Spark, returning the result as a DataFrame. The dialect that is used for SQL parsing can be configured with 'spark.sql.dialect'.

Parameters:

sqlText - (undocumented)

Returns:

(undocumented)

Since:

2.0.0

read

public DataFrameReader read()

Returns a DataFrameReader that can be used to read non-streaming data in as a DataFrame.

   sparkSession.read.parquet("/path/to/file.parquet")
   sparkSession.read.schema(schema).json("/path/to/file.json")
 

Returns:

(undocumented)

Since:

2.0.0

readStream

public DataStreamReader readStream()

:: Experimental :: Returns a DataStreamReader that can be used to read streaming data in as a DataFrame.

   sparkSession.readStream.parquet("/path/to/directory/of/parquet/files")
   sparkSession.readStream.schema(schema).json("/path/to/directory/of/json/files")
 

Returns:

(undocumented)

Since:

2.0.0

time

public <T> T time(scala.Function0<T> f)

Executes some code block and prints to stdout the time taken to execute the block. This is available in Scala only and is used primarily for interactive testing and debugging.

Parameters:

f - (undocumented)

Returns:

(undocumented)

Since:

2.1.0

implicits

public SparkSession.implicits$ implicits()

Accessor for nested Scala object

Returns:

(undocumented)

stop

public void stop()

Stop the underlying SparkContext.

Since:

2.0.0

close

public void close()

Synonym for stop().

Specified by:

close in interface java.io.Closeable

Specified by:

close in interface AutoCloseable

Since:

2.1.0