parquet.hadoop

Class ParquetWriter<T>

java.lang.Object

parquet.hadoop.ParquetWriter<T>

All Implemented Interfaces:

Closeable, AutoCloseable

public class ParquetWriter<T>
extends Object
implements Closeable

Write records to a Parquet file.

Field Summary

Fields

Modifier and Type	Field and Description
static int	DEFAULT_BLOCK_SIZE
static CompressionCodecName	DEFAULT_COMPRESSION_CODEC_NAME
static boolean	DEFAULT_IS_DICTIONARY_ENABLED
static boolean	DEFAULT_IS_VALIDATING_ENABLED
static int	DEFAULT_PAGE_SIZE
static ParquetProperties.WriterVersion	DEFAULT_WRITER_VERSION

Constructor Summary

Constructors

Constructor and Description
ParquetWriter(org.apache.hadoop.fs.Path file, org.apache.hadoop.conf.Configuration conf, WriteSupport<T> writeSupport)
ParquetWriter(org.apache.hadoop.fs.Path file, WriteSupport<T> writeSupport) Create a new ParquetWriter.
ParquetWriter(org.apache.hadoop.fs.Path file, WriteSupport<T> writeSupport, CompressionCodecName compressionCodecName, int blockSize, int pageSize) Create a new ParquetWriter.
ParquetWriter(org.apache.hadoop.fs.Path file, WriteSupport<T> writeSupport, CompressionCodecName compressionCodecName, int blockSize, int pageSize, boolean enableDictionary, boolean validating) Create a new ParquetWriter.
ParquetWriter(org.apache.hadoop.fs.Path file, WriteSupport<T> writeSupport, CompressionCodecName compressionCodecName, int blockSize, int pageSize, int dictionaryPageSize, boolean enableDictionary, boolean validating) Create a new ParquetWriter.
ParquetWriter(org.apache.hadoop.fs.Path file, WriteSupport<T> writeSupport, CompressionCodecName compressionCodecName, int blockSize, int pageSize, int dictionaryPageSize, boolean enableDictionary, boolean validating, ParquetProperties.WriterVersion writerVersion) Create a new ParquetWriter.
ParquetWriter(org.apache.hadoop.fs.Path file, WriteSupport<T> writeSupport, CompressionCodecName compressionCodecName, int blockSize, int pageSize, int dictionaryPageSize, boolean enableDictionary, boolean validating, ParquetProperties.WriterVersion writerVersion, org.apache.hadoop.conf.Configuration conf) Create a new ParquetWriter.

Method Summary

Methods

Modifier and Type	Method and Description
void	close()
void	write(T object)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_BLOCK_SIZE

public static final int DEFAULT_BLOCK_SIZE

See Also:

Constant Field Values

DEFAULT_PAGE_SIZE

public static final int DEFAULT_PAGE_SIZE

See Also:

Constant Field Values

DEFAULT_COMPRESSION_CODEC_NAME

public static final CompressionCodecName DEFAULT_COMPRESSION_CODEC_NAME

DEFAULT_IS_DICTIONARY_ENABLED

public static final boolean DEFAULT_IS_DICTIONARY_ENABLED

See Also:

Constant Field Values

DEFAULT_IS_VALIDATING_ENABLED

public static final boolean DEFAULT_IS_VALIDATING_ENABLED

See Also:

Constant Field Values

DEFAULT_WRITER_VERSION

public static final ParquetProperties.WriterVersion DEFAULT_WRITER_VERSION

Constructor Detail

ParquetWriter

public ParquetWriter(org.apache.hadoop.fs.Path file,
             WriteSupport<T> writeSupport,
             CompressionCodecName compressionCodecName,
             int blockSize,
             int pageSize)
              throws IOException

Create a new ParquetWriter. (with dictionary encoding enabled and validation off)

Parameters:

file - the file to create

writeSupport - the implementation to write a record to a RecordConsumer

compressionCodecName - the compression codec to use

blockSize - the block size threshold

pageSize - the page size threshold

Throws:

IOException

See Also:

ParquetWriter(Path, WriteSupport, CompressionCodecName, int, int, boolean, boolean)

ParquetWriter

public ParquetWriter(org.apache.hadoop.fs.Path file,
             WriteSupport<T> writeSupport,
             CompressionCodecName compressionCodecName,
             int blockSize,
             int pageSize,
             boolean enableDictionary,
             boolean validating)
              throws IOException

Create a new ParquetWriter.

Parameters:

file - the file to create

writeSupport - the implementation to write a record to a RecordConsumer

compressionCodecName - the compression codec to use

blockSize - the block size threshold

pageSize - the page size threshold (both data and dictionary)

enableDictionary - to turn dictionary encoding on

validating - to turn on validation using the schema

Throws:

IOException

See Also:

ParquetWriter(Path, WriteSupport, CompressionCodecName, int, int, int, boolean, boolean)

ParquetWriter

public ParquetWriter(org.apache.hadoop.fs.Path file,
             WriteSupport<T> writeSupport,
             CompressionCodecName compressionCodecName,
             int blockSize,
             int pageSize,
             int dictionaryPageSize,
             boolean enableDictionary,
             boolean validating)
              throws IOException

Create a new ParquetWriter.

Parameters:

file - the file to create

writeSupport - the implementation to write a record to a RecordConsumer

compressionCodecName - the compression codec to use

blockSize - the block size threshold

pageSize - the page size threshold

dictionaryPageSize - the page size threshold for the dictionary pages

enableDictionary - to turn dictionary encoding on

validating - to turn on validation using the schema

Throws:

IOException

See Also:

#ParquetWriter(Path, WriteSupport, CompressionCodecName, int, int, int, boolean, boolean, WriterVersion)

ParquetWriter

public ParquetWriter(org.apache.hadoop.fs.Path file,
             WriteSupport<T> writeSupport,
             CompressionCodecName compressionCodecName,
             int blockSize,
             int pageSize,
             int dictionaryPageSize,
             boolean enableDictionary,
             boolean validating,
             ParquetProperties.WriterVersion writerVersion)
              throws IOException

Create a new ParquetWriter. Directly instantiates a Hadoop Configuration which reads configuration from the classpath.

Parameters:

file - the file to create

writeSupport - the implementation to write a record to a RecordConsumer

compressionCodecName - the compression codec to use

blockSize - the block size threshold

pageSize - the page size threshold

dictionaryPageSize - the page size threshold for the dictionary pages

enableDictionary - to turn dictionary encoding on

validating - to turn on validation using the schema

writerVersion - version of parquetWriter from ParquetProperties.WriterVersion

Throws:

IOException

See Also:

#ParquetWriter(Path, WriteSupport, CompressionCodecName, int, int, int, boolean, boolean, WriterVersion, Configuration)

ParquetWriter

public ParquetWriter(org.apache.hadoop.fs.Path file,
             WriteSupport<T> writeSupport,
             CompressionCodecName compressionCodecName,
             int blockSize,
             int pageSize,
             int dictionaryPageSize,
             boolean enableDictionary,
             boolean validating,
             ParquetProperties.WriterVersion writerVersion,
             org.apache.hadoop.conf.Configuration conf)
              throws IOException

Create a new ParquetWriter.

Parameters:

file - the file to create

writeSupport - the implementation to write a record to a RecordConsumer

compressionCodecName - the compression codec to use

blockSize - the block size threshold

pageSize - the page size threshold

dictionaryPageSize - the page size threshold for the dictionary pages

enableDictionary - to turn dictionary encoding on

validating - to turn on validation using the schema

writerVersion - version of parquetWriter from ParquetProperties.WriterVersion

conf - Hadoop configuration to use while accessing the filesystem

Throws:

IOException

ParquetWriter

public ParquetWriter(org.apache.hadoop.fs.Path file,
             WriteSupport<T> writeSupport)
              throws IOException

Create a new ParquetWriter. The default block size is 50 MB.The default page size is 1 MB. Default compression is no compression. Dictionary encoding is disabled.

Parameters:

file - the file to create

writeSupport - the implementation to write a record to a RecordConsumer

Throws:

IOException

ParquetWriter

public ParquetWriter(org.apache.hadoop.fs.Path file,
             org.apache.hadoop.conf.Configuration conf,
             WriteSupport<T> writeSupport)
              throws IOException

Throws:

IOException

Method Detail

write

public void write(T object)
           throws IOException

Throws:

IOException

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException