(Scala-specific) Implicit methods available in Scala for converting common Scala objects into DataFrames.

val sqlContext = new SQLContext(sc)
import sqlContext.implicits._

Convert a BaseRelation created for external data sources into a DataFrame.

Caches the specified table in-memory.

Removes all cached tables from the in-memory cache.

Applies a schema to a List of Java Beans.

WARNING: Since there is no guaranteed ordering for fields in a Java Bean, SELECT * queries will return the columns in an undefined order.

Applies a schema to an RDD of Java Beans.

WARNING: Since there is no guaranteed ordering for fields in a Java Bean, SELECT * queries will return the columns in an undefined order.

Applies a schema to an RDD of Java Beans.

WARNING: Since there is no guaranteed ordering for fields in a Java Bean, SELECT * queries will return the columns in an undefined order.

:: DeveloperApi :: Creates a DataFrame from a java.util.List containing Rows using the given schema. It is important to make sure that the structure of every Row of the provided List matches the provided schema. Otherwise, there will be runtime exception.

:: DeveloperApi :: Creates a DataFrame from a JavaRDD containing Rows using the given schema. It is important to make sure that the structure of every Row of the provided RDD matches the provided schema. Otherwise, there will be runtime exception.

:: DeveloperApi :: Creates a DataFrame from an RDD containing Rows using the given schema. It is important to make sure that the structure of every Row of the provided RDD matches the provided schema. Otherwise, there will be runtime exception. Example:

import org.apache.spark.sql._
import org.apache.spark.sql.types._
val sqlContext = new org.apache.spark.sql.SQLContext(sc)

val schema =
  StructType(
    StructField("name", StringType, false) ::
    StructField("age", IntegerType, true) :: Nil)

val people =
  sc.textFile("examples/src/main/resources/people.txt").map(
    _.split(",")).map(p => Row(p(0), p(1).trim.toInt))
val dataFrame = sqlContext.createDataFrame(people, schema)
dataFrame.printSchema
// root
// |-- name: string (nullable = false)
// |-- age: integer (nullable = true)

dataFrame.createOrReplaceTempView("people")
sqlContext.sql("select name from people").collect.foreach(println)

Creates a DataFrame from a local Seq of Product.

Creates a DataFrame from an RDD of Product (e.g. case classes, tuples).

Creates a Dataset from a java.util.List of a given type. This method requires an encoder (to convert a JVM object of type T to and from the internal Spark SQL representation) that is generally created automatically through implicits from a SparkSession, or can be created explicitly by calling static methods on Encoders.

Java Example

List<String> data = Arrays.asList("hello", "world");
Dataset<String> ds = spark.createDataset(data, Encoders.STRING());

Creates a Dataset from an RDD of a given type. This method requires an encoder (to convert a JVM object of type T to and from the internal Spark SQL representation) that is generally created automatically through implicits from a SparkSession, or can be created explicitly by calling static methods on Encoders.

Creates a Dataset from a local Seq of data of a given type. This method requires an encoder (to convert a JVM object of type T to and from the internal Spark SQL representation) that is generally created automatically through implicits from a SparkSession, or can be created explicitly by calling static methods on Encoders.

Example

import spark.implicits._
case class Person(name: String, age: Long)
val data = Seq(Person("Michael", 29), Person("Andy", 30), Person("Justin", 19))
val ds = spark.createDataset(data)

ds.show()
// +-------+---+
// |   name|age|
// +-------+---+
// |Michael| 29|
// |   Andy| 30|
// | Justin| 19|
// +-------+---+

Drops the temporary table with the given table name in the catalog. If the table has been cached/persisted before, it's also unpersisted.

Returns a DataFrame with no rows or columns.

:: Experimental :: A collection of methods that are considered experimental, but can be used to hook into the query planner for advanced functionality.

Return all the configuration properties that have been set (i.e. not the default). This creates a new copy of the config properties in the form of a Map.

Return the value of Spark SQL configuration property for the given key. If the key is not set yet, return defaultValue.

Return the value of Spark SQL configuration property for the given key.

Returns true if the table is currently cached in-memory.

An interface to register custom org.apache.spark.sql.util.QueryExecutionListeners that listen for execution metrics.

Returns a new HiveContext as new session, which will have separated SQLConf, UDF/UDAF, temporary tables and SessionState, but sharing the same CacheManager, IsolatedClientLoader and Hive client (both of execution and metadata) with existing HiveContext.

Creates a DataFrame with a single LongType column named id, containing elements in an range from start to end (exclusive) with an step value, with partition number specified.

Creates a DataFrame with a single LongType column named id, containing elements in a range from start to end (exclusive) with a step value.

Creates a DataFrame with a single LongType column named id, containing elements in a range from start to end (exclusive) with step value 1.

Creates a DataFrame with a single LongType column named id, containing elements in a range from 0 to end (exclusive) with step value 1.

Returns a DataFrameReader that can be used to read non-streaming data in as a DataFrame.

sqlContext.read.parquet("/path/to/file.parquet")
sqlContext.read.schema(schema).json("/path/to/file.json")

Returns a DataStreamReader that can be used to read streaming data in as a DataFrame.

sparkSession.readStream.parquet("/path/to/directory/of/parquet/files")
sparkSession.readStream.schema(schema).json("/path/to/directory/of/json/files")

Invalidate and refresh all the cached the metadata of the given table. For performance reasons, Spark SQL or the external data source library it uses might cache certain metadata about a table, such as the location of blocks. When those change outside of Spark SQL, users should call this function to invalidate the cache.

Set the given Spark SQL configuration property.

Set Spark SQL configuration properties.

Executes a SQL query using Spark, returning the result as a DataFrame. This API eagerly runs DDL/DML commands, but not for SELECT queries.

Returns a StreamingQueryManager that allows managing all the StreamingQueries active on this context.

Returns the specified table as a DataFrame.

Returns the names of tables in the given database as an array.

Returns the names of tables in the current database as an array.

Returns a DataFrame containing names of existing tables in the given database. The returned DataFrame has three columns, database, tableName and isTemporary (a Boolean indicating if a table is a temporary one or not).

Returns a DataFrame containing names of existing tables in the current database. The returned DataFrame has three columns, database, tableName and isTemporary (a Boolean indicating if a table is a temporary one or not).

A collection of methods for registering user-defined functions (UDF).

The following example registers a Scala closure as UDF:

sqlContext.udf.register("myUDF", (arg1: Int, arg2: String) => arg2 + arg1)

The following example registers a UDF in Java:

sqlContext.udf().register("myUDF",
    (Integer arg1, String arg2) -> arg2 + arg1,
    DataTypes.StringType);

Removes the specified table from the in-memory cache.

(Scala-specific) Create an external table from the given path based on a data source, a schema and a set of options. Then, returns the corresponding DataFrame.

Create an external table from the given path based on a data source, a schema and a set of options. Then, returns the corresponding DataFrame.

(Scala-specific) Creates an external table from the given path based on a data source and a set of options. Then, returns the corresponding DataFrame.

Creates an external table from the given path based on a data source and a set of options. Then, returns the corresponding DataFrame.

Creates an external table from the given path based on a data source and returns the corresponding DataFrame.

Creates an external table from the given path and returns the corresponding DataFrame. It will use the default data source configured by spark.sql.sources.default.

Construct a DataFrame representing the database table accessible via JDBC URL url named table. The theParts parameter gives a list expressions suitable for inclusion in WHERE clauses; each one defines one partition of the DataFrame.

Construct a DataFrame representing the database table accessible via JDBC URL url named table. Partitions of the table will be retrieved in parallel based on the parameters passed to this function.

Construct a DataFrame representing the database table accessible via JDBC URL url named table.

Loads a JSON file (one object per line) and applies the given schema, returning the result as a DataFrame.

Loads a JSON file (one object per line), returning the result as a DataFrame. It goes through the entire dataset once to determine the schema.

Loads a JavaRDD[String] storing JSON objects (one object per record) inferring the schema, returning the result as a DataFrame.

Loads an RDD[String] storing JSON objects (one object per record) inferring the schema, returning the result as a DataFrame.

Loads an JavaRDD[String] storing JSON objects (one object per record) and applies the given schema, returning the result as a DataFrame.

Loads an RDD[String] storing JSON objects (one object per record) and applies the given schema, returning the result as a DataFrame.

Loads an RDD[String] storing JSON objects (one object per record), returning the result as a DataFrame. It goes through the entire dataset once to determine the schema.

Loads an RDD[String] storing JSON objects (one object per record), returning the result as a DataFrame. It goes through the entire dataset once to determine the schema.

(Scala-specific) Returns the dataset specified by the given data source and a set of options as a DataFrame, using the given schema as the schema of the DataFrame.

(Java-specific) Returns the dataset specified by the given data source and a set of options as a DataFrame, using the given schema as the schema of the DataFrame.

(Scala-specific) Returns the dataset specified by the given data source and a set of options as a DataFrame.

(Java-specific) Returns the dataset specified by the given data source and a set of options as a DataFrame.

Returns the dataset stored at path as a DataFrame, using the given data source.

Returns the dataset stored at path as a DataFrame, using the default data source configured by spark.sql.sources.default.

Loads a Parquet file, returning the result as a DataFrame. This function returns an empty DataFrame if no paths are passed in.