org.apache.spark.sql.parquet

Class AppendingParquetOutputFormat

Object

org.apache.hadoop.mapreduce.OutputFormat<K,V>

org.apache.hadoop.mapreduce.lib.output.FileOutputFormat<Void,T>

parquet.hadoop.ParquetOutputFormat<Row>

org.apache.spark.sql.parquet.AppendingParquetOutputFormat

public class AppendingParquetOutputFormat
extends parquet.hadoop.ParquetOutputFormat<Row>

TODO: this will be able to append to directories it created itself, not necessarily to imported ones.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.mapreduce.lib.output.FileOutputFormat

org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.Counter

Field Summary

Fields inherited from class parquet.hadoop.ParquetOutputFormat

BLOCK_SIZE, COMPRESSION, DICTIONARY_PAGE_SIZE, ENABLE_DICTIONARY, ENABLE_JOB_SUMMARY, PAGE_SIZE, VALIDATION, WRITE_SUPPORT_CLASS, WRITER_VERSION

Constructor Summary

Constructors

Constructor and Description
AppendingParquetOutputFormat(int offset)

Method Summary

Methods

Modifier and Type	Method and Description
void	checkOutputSpecs(org.apache.hadoop.mapreduce.JobContext job)
org.apache.hadoop.fs.Path	getDefaultWorkFile(org.apache.hadoop.mapreduce.TaskAttemptContext context, String extension)

Methods inherited from class parquet.hadoop.ParquetOutputFormat

getBlockSize, getBlockSize, getCompression, getCompression, getDictionaryPageSize, getDictionaryPageSize, getEnableDictionary, getEnableDictionary, getOutputCommitter, getPageSize, getPageSize, getRecordWriter, getRecordWriter, getRecordWriter, getValidation, getValidation, getWriterVersion, getWriteSupport, getWriteSupportClass, isCompressionSet, isCompressionSet, setBlockSize, setCompression, setDictionaryPageSize, setEnableDictionary, setPageSize, setValidation, setValidation, setWriteSupportClass, setWriteSupportClass

Methods inherited from class org.apache.hadoop.mapreduce.lib.output.FileOutputFormat

getCompressOutput, getOutputCompressorClass, getOutputPath, getPathForWorkFile, getUniqueFile, getWorkOutputPath, setCompressOutput, setOutputCompressorClass, setOutputPath

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

AppendingParquetOutputFormat

public AppendingParquetOutputFormat(int offset)

Method Detail

checkOutputSpecs

public void checkOutputSpecs(org.apache.hadoop.mapreduce.JobContext job)

Overrides:

checkOutputSpecs in class org.apache.hadoop.mapreduce.lib.output.FileOutputFormat<Void,Row>

getDefaultWorkFile

public org.apache.hadoop.fs.Path getDefaultWorkFile(org.apache.hadoop.mapreduce.TaskAttemptContext context,
                                           String extension)

Overrides:

getDefaultWorkFile in class org.apache.hadoop.mapreduce.lib.output.FileOutputFormat<Void,Row>