Package org.apache.beam.sdk.io.parquet

Class ParquetIO.Sink

  • All Implemented Interfaces:
    java.io.Serializable, org.apache.beam.sdk.io.FileIO.Sink<org.apache.avro.generic.GenericRecord>
    Enclosing class:
    ParquetIO
    public abstract static class ParquetIO.Sink
extends java.lang.Object
implements org.apache.beam.sdk.io.FileIO.Sink<org.apache.avro.generic.GenericRecord>
    Implementation of ParquetIO.sink(org.apache.avro.Schema).
    See Also:
    Serialized Form

    • Constructor Detail

      • Sink

        public Sink()

    • Method Detail

      • withCompressionCodec

        public ParquetIO.Sink withCompressionCodec​(org.apache.parquet.hadoop.metadata.CompressionCodecName compressionCodecName)
        Specifies compression codec. By default, CompressionCodecName.SNAPPY.

      • withConfiguration

        public ParquetIO.Sink withConfiguration​(java.util.Map<java.lang.String,​java.lang.String> configuration)
        Specify Hadoop configuration for ParquetWriter.

      • withConfiguration

        public ParquetIO.Sink withConfiguration​(org.apache.hadoop.conf.Configuration configuration)
        Specify Hadoop configuration for ParquetWriter.

      • withRowGroupSize

        public ParquetIO.Sink withRowGroupSize​(int rowGroupSize)
        Specify row-group size; if not set or zero, a default is used by the underlying writer.

      • withPageSize

        public ParquetIO.Sink withPageSize​(int pageSize)
        Specify the page size for the Parquet writer. Defaults to 1 MB.

      • withDictionaryEncoding

        public ParquetIO.Sink withDictionaryEncoding​(boolean enableDictionary)
        Enable or disable dictionary encoding. Enabled by default.

      • withBloomFilterEnabled

        public ParquetIO.Sink withBloomFilterEnabled​(boolean enableBloomFilter)
        Enable or disable bloom filters. Disabled by default.

      • withMinRowCountForPageSizeCheck

        public ParquetIO.Sink withMinRowCountForPageSizeCheck​(int minRowCountForPageSizeCheck)
        Specify the minimum number of rows to write before a page size check is performed. The writer buffers at least this many rows before checking whether the page size threshold has been reached. With large rows, the default (100) can cause excessive memory use; set a lower value (e.g. 1) to flush pages more frequently.

      • withMinRowCountForPageSizeCheck

        public ParquetIO.Sink withMinRowCountForPageSizeCheck​(org.apache.beam.sdk.options.ValueProvider<java.lang.Integer> minRowCountForPageSizeCheck)
        Like withMinRowCountForPageSizeCheck(int), but accepts a ValueProvider so the value can be supplied at runtime (required for classic Dataflow templates).

      • withMaxRowCountForPageSizeCheck

        public ParquetIO.Sink withMaxRowCountForPageSizeCheck​(int maxRowCountForPageSizeCheck)
        Specify the maximum number of rows to buffer before a page size check is forced. By default Parquet estimates the next check from the average row size and may defer it (up to 10000 rows); a run of small rows followed by large rows can then let the column page buffer overflow Integer.MAX_VALUE before the deferred check fires. Setting this (e.g. 1) caps the interval so a check -- and flush -- happens at least this often regardless of the estimate. Pair it with withMinRowCountForPageSizeCheck(int) to bound the buffer for tables whose row sizes vary widely.

      • withMaxRowCountForPageSizeCheck

        public ParquetIO.Sink withMaxRowCountForPageSizeCheck​(org.apache.beam.sdk.options.ValueProvider<java.lang.Integer> maxRowCountForPageSizeCheck)
        Like withMaxRowCountForPageSizeCheck(int), but accepts a ValueProvider so the value can be supplied at runtime (required for classic Dataflow templates).

      • withAvroDataModel

        public ParquetIO.Sink withAvroDataModel​(org.apache.avro.generic.GenericData model)
        Define the Avro data model; see AvroParquetWriter.Builder.withDataModel(GenericData).

      • open

        public void open​(java.nio.channels.WritableByteChannel channel)
          throws java.io.IOException
        Specified by:
        open in interface org.apache.beam.sdk.io.FileIO.Sink<org.apache.avro.generic.GenericRecord>
        Throws:
        java.io.IOException

      • write

        public void write​(org.apache.avro.generic.GenericRecord element)
           throws java.io.IOException
        Specified by:
        write in interface org.apache.beam.sdk.io.FileIO.Sink<org.apache.avro.generic.GenericRecord>
        Throws:
        java.io.IOException

      • flush

        public void flush()
           throws java.io.IOException
        Specified by:
        flush in interface org.apache.beam.sdk.io.FileIO.Sink<org.apache.avro.generic.GenericRecord>
        Throws:
        java.io.IOException