org.apache.spark.internal.io

Class SparkHadoopMapReduceWriter

Object

org.apache.spark.internal.io.SparkHadoopMapReduceWriter

public class SparkHadoopMapReduceWriter
extends Object

A helper object that saves an RDD using a Hadoop OutputFormat (from the newer mapreduce API, not the old mapred API).

Constructor Summary

Constructors

Constructor and Description
SparkHadoopMapReduceWriter()

Method Summary

Modifier and Type	Method and Description
static <K,V> void	write(RDD<scala.Tuple2<K,V>> rdd, org.apache.hadoop.conf.Configuration hadoopConf, scala.reflect.ClassTag<V> evidence$1) Basic work flow of this command is: 1.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SparkHadoopMapReduceWriter

public SparkHadoopMapReduceWriter()

Method Detail

write

public static <K,V> void write(RDD<scala.Tuple2<K,V>> rdd,
                               org.apache.hadoop.conf.Configuration hadoopConf,
                               scala.reflect.ClassTag<V> evidence$1)

Basic work flow of this command is: 1. Driver side setup, prepare the data source and hadoop configuration for the write job to be issued. 2. Issues a write job consists of one or more executor side tasks, each of which writes all rows within an RDD partition. 3. If no exception is thrown in a task, commits that task, otherwise aborts that task; If any exception is thrown during task commitment, also aborts that task. 4. If all tasks are committed, commit the job, otherwise aborts the job; If any exception is thrown during job commitment, also aborts the job.

Parameters:

rdd - (undocumented)

hadoopConf - (undocumented)

evidence$1 - (undocumented)