Package org.apache.beam.sdk.io.parquet

Class ParquetIO.Sink

  • All Implemented Interfaces:
    java.io.Serializable, org.apache.beam.sdk.io.FileIO.Sink<org.apache.avro.generic.GenericRecord>
    Enclosing class:
    ParquetIO
    public abstract static class ParquetIO.Sink
extends java.lang.Object
implements org.apache.beam.sdk.io.FileIO.Sink<org.apache.avro.generic.GenericRecord>
    Implementation of ParquetIO.sink(org.apache.avro.Schema).
    See Also:
    Serialized Form

    • Constructor Detail

      • Sink

        public Sink()

    • Method Detail

      • withCompressionCodec

        public ParquetIO.Sink withCompressionCodec​(org.apache.parquet.hadoop.metadata.CompressionCodecName compressionCodecName)
        Specifies compression codec. By default, CompressionCodecName.SNAPPY.

      • withConfiguration

        public ParquetIO.Sink withConfiguration​(java.util.Map<java.lang.String,​java.lang.String> configuration)
        Specify Hadoop configuration for ParquetWriter.

      • withConfiguration

        public ParquetIO.Sink withConfiguration​(org.apache.hadoop.conf.Configuration configuration)
        Specify Hadoop configuration for ParquetWriter.

      • withRowGroupSize

        public ParquetIO.Sink withRowGroupSize​(int rowGroupSize)
        Specify row-group size; if not set or zero, a default is used by the underlying writer.

      • withPageSize

        public ParquetIO.Sink withPageSize​(int pageSize)
        Specify the page size for the Parquet writer. Defaults to 1 MB.

      • withDictionaryEncoding

        public ParquetIO.Sink withDictionaryEncoding​(boolean enableDictionary)
        Enable or disable dictionary encoding. Enabled by default.

      • withBloomFilterEnabled

        public ParquetIO.Sink withBloomFilterEnabled​(boolean enableBloomFilter)
        Enable or disable bloom filters. Disabled by default.

      • withMinRowCountForPageSizeCheck

        public ParquetIO.Sink withMinRowCountForPageSizeCheck​(int minRowCountForPageSizeCheck)
        Specify the minimum number of rows to write before a page size check is performed. The writer buffers at least this many rows before checking whether the page size threshold has been reached. With large rows, the default (100) can cause excessive memory use; set a lower value (e.g. 1) to flush pages more frequently.

      • withMinRowCountForPageSizeCheck

        public ParquetIO.Sink withMinRowCountForPageSizeCheck​(org.apache.beam.sdk.options.ValueProvider<java.lang.Integer> minRowCountForPageSizeCheck)
        Like withMinRowCountForPageSizeCheck(int), but accepts a ValueProvider so the value can be supplied at runtime (required for classic Dataflow templates).

      • withAvroDataModel

        public ParquetIO.Sink withAvroDataModel​(org.apache.avro.generic.GenericData model)
        Define the Avro data model; see AvroParquetWriter.Builder.withDataModel(GenericData).

      • open

        public void open​(java.nio.channels.WritableByteChannel channel)
          throws java.io.IOException
        Specified by:
        open in interface org.apache.beam.sdk.io.FileIO.Sink<org.apache.avro.generic.GenericRecord>
        Throws:
        java.io.IOException

      • write

        public void write​(org.apache.avro.generic.GenericRecord element)
           throws java.io.IOException
        Specified by:
        write in interface org.apache.beam.sdk.io.FileIO.Sink<org.apache.avro.generic.GenericRecord>
        Throws:
        java.io.IOException

      • flush

        public void flush()
           throws java.io.IOException
        Specified by:
        flush in interface org.apache.beam.sdk.io.FileIO.Sink<org.apache.avro.generic.GenericRecord>
        Throws:
        java.io.IOException