SQLConf (Spark 1.3.1 JavaDoc)

Object
- org.apache.spark.sql.SQLConf

All Implemented Interfaces:

java.io.Serializable
```
public class SQLConf
extends Object
implements scala.Serializable
```
A class that enables the setting and getting of mutable config parameters/hints.
In the presence of a SQLContext, these can be set and queried by passing SET commands into Spark SQL's query functions (i.e. sql()). Otherwise, users of this class can modify the hints by programmatically calling the setters and getters of this class.
SQLConf is thread-safe (internally synchronized, so safe to be used in multiple threads).

See Also:
Serialized Form

Nested Class Summary

Nested Classes
Modifier and Type Class and Description

static class SQLConf.Deprecated$

Nested Classes
Modifier and Type	Class and Description
`static class`	`SQLConf.Deprecated$`

Constructor Summary

Constructors
Constructor and Description

SQLConf()

Constructors
Constructor and Description
`SQLConf()`

Method Summary

Methods
Modifier and Type	Method and Description
`static String`	`AUTO_BROADCASTJOIN_THRESHOLD()`
`int`	`autoBroadcastJoinThreshold()` Upper bound on the sizes (in bytes) of the tables qualified for the auto conversion to a broadcast value during the physical executions of join operations.
`static String`	`BROADCAST_TIMEOUT()`
`int`	`broadcastTimeout()` Timeout in seconds for the broadcast wait time in hash join
`void`	`clear()`
`static String`	`CODEGEN_ENABLED()`
`boolean`	`codegenEnabled()` When set to true, Spark SQL will use the Scala compiler at runtime to generate custom bytecode that evaluates expressions found in queries.
`static String`	`COLUMN_BATCH_SIZE()`
`static String`	`COLUMN_NAME_OF_CORRUPT_RECORD()`
`int`	`columnBatchSize()` The number of rows that will be
`String`	`columnNameOfCorruptRecord()`
`static String`	`COMPRESS_CACHED()`
`static String`	`DATAFRAME_EAGER_ANALYSIS()`
`boolean`	`dataFrameEagerAnalysis()`
`static String`	`DEFAULT_DATA_SOURCE_NAME()`
`static String`	`DEFAULT_SIZE_IN_BYTES()`
`String`	`defaultDataSourceName()`
`long`	`defaultSizeInBytes()` The default size in bytes to assign to a logical operator's estimation statistics.
`String`	`dialect()` The SQL dialect that is used when parsing queries.
`static String`	`DIALECT()`
`static String`	`EXTERNAL_SORT()`
`boolean`	`externalSortEnabled()` When true the planner will use the external sort, which may spill to disk.
`scala.collection.immutable.Map<String,String>`	`getAllConfs()` Return all the configuration properties that have been set (i.e.
`String`	`getConf(String key)` Return the value of Spark SQL configuration property for the given key.
`String`	`getConf(String key, String defaultValue)` Return the value of Spark SQL configuration property for the given key.
`static String`	`IN_MEMORY_PARTITION_PRUNING()`
`boolean`	`inMemoryPartitionPruning()` When set to true, partition pruning for in-memory columnar tables is enabled.
`boolean`	`isParquetBinaryAsString()` When set to true, we always treat byte arrays in Parquet files as strings.
`boolean`	`isParquetINT96AsTimestamp()` When set to true, we always treat INT96Values in Parquet files as timestamp.
`int`	`numShufflePartitions()` Number of partitions to use for shuffle operators.
`static String`	`PARQUET_BINARY_AS_STRING()`
`static String`	`PARQUET_CACHE_METADATA()`
`static String`	`PARQUET_COMPRESSION()`
`static String`	`PARQUET_FILTER_PUSHDOWN_ENABLED()`
`static String`	`PARQUET_INT96_AS_TIMESTAMP()`
`static String`	`PARQUET_USE_DATA_SOURCE_API()`
`String`	`parquetCompressionCodec()` The compression codec for writing to a Parquetfile
`boolean`	`parquetFilterPushDown()` When true predicates will be passed to the parquet record reader when possible.
`boolean`	`parquetUseDataSourceApi()` When true uses Parquet implementation based on data source API
`static String`	`SCHEMA_STRING_LENGTH_THRESHOLD()`
`int`	`schemaStringLengthThreshold()`
`void`	`setConf(java.util.Properties props)` Set Spark SQL configuration properties.
`void`	`setConf(String key, String value)` Set the given Spark SQL configuration property.
`static String`	`SHUFFLE_PARTITIONS()`
`static String`	`THRIFTSERVER_POOL()`
`void`	`unsetConf(String key)`
`boolean`	`useCompression()` When true tables cached using the in-memory columnar caching will be compressed.

Methods inherited from class Object
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Constructor Detail
  - SQLConf
```
public SQLConf()
```
- Method Detail
  - COMPRESS_CACHED
```
public static String COMPRESS_CACHED()
```
  - COLUMN_BATCH_SIZE
```
public static String COLUMN_BATCH_SIZE()
```
  - IN_MEMORY_PARTITION_PRUNING
```
public static String IN_MEMORY_PARTITION_PRUNING()
```
  - AUTO_BROADCASTJOIN_THRESHOLD
```
public static String AUTO_BROADCASTJOIN_THRESHOLD()
```
  - DEFAULT_SIZE_IN_BYTES
```
public static String DEFAULT_SIZE_IN_BYTES()
```
  - SHUFFLE_PARTITIONS
```
public static String SHUFFLE_PARTITIONS()
```
  - CODEGEN_ENABLED
```
public static String CODEGEN_ENABLED()
```
  - DIALECT
```
public static String DIALECT()
```
  - PARQUET_BINARY_AS_STRING
```
public static String PARQUET_BINARY_AS_STRING()
```
  - PARQUET_INT96_AS_TIMESTAMP
```
public static String PARQUET_INT96_AS_TIMESTAMP()
```
  - PARQUET_CACHE_METADATA
```
public static String PARQUET_CACHE_METADATA()
```
  - PARQUET_COMPRESSION
```
public static String PARQUET_COMPRESSION()
```
  - PARQUET_FILTER_PUSHDOWN_ENABLED
```
public static String PARQUET_FILTER_PUSHDOWN_ENABLED()
```
  - PARQUET_USE_DATA_SOURCE_API
```
public static String PARQUET_USE_DATA_SOURCE_API()
```
  - COLUMN_NAME_OF_CORRUPT_RECORD
```
public static String COLUMN_NAME_OF_CORRUPT_RECORD()
```
  - BROADCAST_TIMEOUT
```
public static String BROADCAST_TIMEOUT()
```
  - EXTERNAL_SORT
```
public static String EXTERNAL_SORT()
```
  - THRIFTSERVER_POOL
```
public static String THRIFTSERVER_POOL()
```
  - DEFAULT_DATA_SOURCE_NAME
```
public static String DEFAULT_DATA_SOURCE_NAME()
```
  - SCHEMA_STRING_LENGTH_THRESHOLD
```
public static String SCHEMA_STRING_LENGTH_THRESHOLD()
```
  - DATAFRAME_EAGER_ANALYSIS
```
public static String DATAFRAME_EAGER_ANALYSIS()
```
  - dialect
```
public String dialect()
```
    The SQL dialect that is used when parsing queries. This defaults to 'sql' which uses a simple SQL parser provided by Spark SQL. This is currently the only option for users of SQLContext.
    When using a HiveContext, this value defaults to 'hiveql', which uses the Hive 0.12.0 HiveQL parser. Users can change this to 'sql' if they want to run queries that aren't supported by HiveQL (e.g., SELECT 1).
    Note that the choice of dialect does not affect things like what tables are available or how query execution is performed.
  - useCompression
```
public boolean useCompression()
```
    When true tables cached using the in-memory columnar caching will be compressed.
  - parquetCompressionCodec
```
public String parquetCompressionCodec()
```
    The compression codec for writing to a Parquetfile
  - columnBatchSize
```
public int columnBatchSize()
```
    The number of rows that will be
  - numShufflePartitions
```
public int numShufflePartitions()
```
    Number of partitions to use for shuffle operators.
  - parquetFilterPushDown
```
public boolean parquetFilterPushDown()
```
    When true predicates will be passed to the parquet record reader when possible.
  - parquetUseDataSourceApi
```
public boolean parquetUseDataSourceApi()
```
    When true uses Parquet implementation based on data source API
  - externalSortEnabled
```
public boolean externalSortEnabled()
```
    When true the planner will use the external sort, which may spill to disk.
  - codegenEnabled
```
public boolean codegenEnabled()
```
    When set to true, Spark SQL will use the Scala compiler at runtime to generate custom bytecode that evaluates expressions found in queries. In general this custom code runs much faster than interpreted evaluation, but there are significant start-up costs due to compilation. As a result codegen is only beneficial when queries run for a long time, or when the same expressions are used multiple times.
    Defaults to false as this feature is currently experimental.
  - autoBroadcastJoinThreshold
```
public int autoBroadcastJoinThreshold()
```
    Upper bound on the sizes (in bytes) of the tables qualified for the auto conversion to a broadcast value during the physical executions of join operations. Setting this to -1 effectively disables auto conversion.
    Hive setting: hive.auto.convert.join.noconditionaltask.size, whose default value is 10000.
  - defaultSizeInBytes
```
public long defaultSizeInBytes()
```
    The default size in bytes to assign to a logical operator's estimation statistics. By default, it is set to a larger value than autoBroadcastJoinThreshold, hence any logical operator without a properly implemented estimation of this statistic will not be incorrectly broadcasted in joins.
  - isParquetBinaryAsString
```
public boolean isParquetBinaryAsString()
```
    When set to true, we always treat byte arrays in Parquet files as strings.
  - isParquetINT96AsTimestamp
```
public boolean isParquetINT96AsTimestamp()
```
    When set to true, we always treat INT96Values in Parquet files as timestamp.
  - inMemoryPartitionPruning
```
public boolean inMemoryPartitionPruning()
```
    When set to true, partition pruning for in-memory columnar tables is enabled.
  - columnNameOfCorruptRecord
```
public String columnNameOfCorruptRecord()
```
  - broadcastTimeout
```
public int broadcastTimeout()
```
    Timeout in seconds for the broadcast wait time in hash join
  - defaultDataSourceName
```
public String defaultDataSourceName()
```
  - schemaStringLengthThreshold
```
public int schemaStringLengthThreshold()
```
  - dataFrameEagerAnalysis
```
public boolean dataFrameEagerAnalysis()
```
  - setConf
```
public void setConf(java.util.Properties props)
```
    Set Spark SQL configuration properties.
  - setConf
```
public void setConf(String key,
           String value)
```
    Set the given Spark SQL configuration property.
  - getConf
```
public String getConf(String key)
```
    Return the value of Spark SQL configuration property for the given key.
  - getConf
```
public String getConf(String key,
             String defaultValue)
```
    Return the value of Spark SQL configuration property for the given key. If the key is not set yet, return defaultValue.
  - getAllConfs
```
public scala.collection.immutable.Map<String,String> getAllConfs()
```
    Return all the configuration properties that have been set (i.e. not the default). This creates a new copy of the config properties in the form of a Map.
  - unsetConf
```
public void unsetConf(String key)
```
  - clear
```
public void clear()
```

Class SQLConf

Nested Class Summary

Constructor Summary

Method Summary

Methods inherited from class Object

Constructor Detail

SQLConf

Method Detail

COMPRESS_CACHED

COLUMN_BATCH_SIZE

IN_MEMORY_PARTITION_PRUNING

AUTO_BROADCASTJOIN_THRESHOLD

DEFAULT_SIZE_IN_BYTES

SHUFFLE_PARTITIONS

CODEGEN_ENABLED

DIALECT

PARQUET_BINARY_AS_STRING

PARQUET_INT96_AS_TIMESTAMP

PARQUET_CACHE_METADATA

PARQUET_COMPRESSION

PARQUET_FILTER_PUSHDOWN_ENABLED

PARQUET_USE_DATA_SOURCE_API

COLUMN_NAME_OF_CORRUPT_RECORD

BROADCAST_TIMEOUT

EXTERNAL_SORT

THRIFTSERVER_POOL

DEFAULT_DATA_SOURCE_NAME

SCHEMA_STRING_LENGTH_THRESHOLD

DATAFRAME_EAGER_ANALYSIS

dialect

useCompression

parquetCompressionCodec

columnBatchSize

numShufflePartitions

parquetFilterPushDown

parquetUseDataSourceApi

externalSortEnabled

codegenEnabled

autoBroadcastJoinThreshold

defaultSizeInBytes

isParquetBinaryAsString

isParquetINT96AsTimestamp

inMemoryPartitionPruning

columnNameOfCorruptRecord

broadcastTimeout

defaultDataSourceName

schemaStringLengthThreshold

dataFrameEagerAnalysis

setConf

setConf

getConf

getConf

getAllConfs

unsetConf

clear