Package com.jerolba.carpet

Class CarpetWriter.Builder<T>

java.lang.Object
com.jerolba.carpet.CarpetWriter.Builder<T>
Enclosing class:
CarpetWriter<T>
public static class CarpetWriter.Builder<T> extends Object

  • Constructor Details

    • Builder

      public Builder(org.apache.parquet.io.OutputFile path, Class<T> recordClass)

    • Builder

      public Builder(OutputStream outputSrream, Class<T> recordClass)

  • Method Details

    • withConf

      public CarpetWriter.Builder<T> withConf(org.apache.hadoop.conf.Configuration conf)
      Set the Configuration used by the constructed writer.
      Parameters:
      conf - a Configuration
      Returns:
      this builder for method chaining.

    • withConf

      public CarpetWriter.Builder<T> withConf(org.apache.parquet.conf.ParquetConfiguration conf)
      Set the ParquetConfiguration used by the constructed writer.
      Parameters:
      conf - a ParquetConfiguration
      Returns:
      this builder for method chaining.

    • withWriteMode

      public CarpetWriter.Builder<T> withWriteMode(org.apache.parquet.hadoop.ParquetFileWriter.Mode mode)
      Set the write mode used when creating the backing file for this writer.
      Parameters:
      mode - a ParquetFileWriter.Mode
      Returns:
      this builder for method chaining.

    • withCompressionCodec

      public CarpetWriter.Builder<T> withCompressionCodec(org.apache.parquet.hadoop.metadata.CompressionCodecName codecName)
      Set the compression codec used by the constructed writer.
      Parameters:
      codecName - a CompressionCodecName
      Returns:
      this builder for method chaining.

    • withCodecFactory

      public CarpetWriter.Builder<T> withCodecFactory(org.apache.parquet.compression.CompressionCodecFactory codecFactory)
      Set the codec factory used by the constructed writer.
      Parameters:
      codecFactory - a CompressionCodecFactory
      Returns:
      this builder for method chaining.

    • withEncryption

      public CarpetWriter.Builder<T> withEncryption(org.apache.parquet.crypto.FileEncryptionProperties encryptionProperties)
      Set the file encryption properties used by the constructed writer.
      Parameters:
      encryptionProperties - a FileEncryptionProperties
      Returns:
      this builder for method chaining.

    • withRowGroupSize

      public CarpetWriter.Builder<T> withRowGroupSize(long rowGroupSize)
      Set the Parquet format row group size used by the constructed writer.
      Parameters:
      rowGroupSize - an integer size in bytes
      Returns:
      this builder for method chaining.

    • withMaxPaddingSize

      public CarpetWriter.Builder<T> withMaxPaddingSize(int maxPaddingSize)
      Set the maximum amount of padding, in bytes, that will be used to align row groups with blocks in the underlying filesystem. If the underlying filesystem is not a block filesystem like HDFS, this has no effect.
      Parameters:
      maxPaddingSize - an integer size in bytes
      Returns:
      this builder for method chaining.

    • enableValidation

      public CarpetWriter.Builder<T> enableValidation()
      Enables validation for the constructed writer.
      Returns:
      this builder for method chaining.

    • withValidation

      public CarpetWriter.Builder<T> withValidation(boolean enableValidation)
      Enable or disable validation for the constructed writer.
      Parameters:
      enableValidation - whether validation should be enabled
      Returns:
      this builder for method chaining.

    • withPageSize

      public CarpetWriter.Builder<T> withPageSize(int pageSize)
      Set the Parquet format page size used by the constructed writer.
      Parameters:
      pageSize - an integer size in bytes
      Returns:
      this builder for method chaining.

    • withPageRowCountLimit

      public CarpetWriter.Builder<T> withPageRowCountLimit(int rowCount)
      Sets the Parquet format page row count limit used by the constructed writer.
      Parameters:
      rowCount - limit for the number of rows stored in a page
      Returns:
      this builder for method chaining

    • withDictionaryPageSize

      public CarpetWriter.Builder<T> withDictionaryPageSize(int dictionaryPageSize)
      Set the Parquet format dictionary page size used by the constructed writer.
      Parameters:
      dictionaryPageSize - an integer size in bytes
      Returns:
      this builder for method chaining.

    • enableDictionaryEncoding

      public CarpetWriter.Builder<T> enableDictionaryEncoding()
      Enables dictionary encoding for the constructed writer.
      Returns:
      this builder for method chaining.

    • withDictionaryEncoding

      public CarpetWriter.Builder<T> withDictionaryEncoding(boolean enableDictionary)
      Enable or disable dictionary encoding for the constructed writer.
      Parameters:
      enableDictionary - whether dictionary encoding should be enabled
      Returns:
      this builder for method chaining.

    • withByteStreamSplitEncoding

      public CarpetWriter.Builder<T> withByteStreamSplitEncoding(boolean enableByteStreamSplit)
      Enable or disable BYTE_STREAM_SPLIT encoding for FLOAT and DOUBLE columns.
      Parameters:
      enableByteStreamSplit - whether BYTE_STREAM_SPLIT encoding should be enabled
      Returns:
      this builder for method chaining.

    • withDictionaryEncoding

      public CarpetWriter.Builder<T> withDictionaryEncoding(String columnPath, boolean enableDictionary)
      Enable or disable dictionary encoding of the specified column for the constructed writer.
      Parameters:
      columnPath - the path of the column (dot-string)
      enableDictionary - whether dictionary encoding should be enabled
      Returns:
      this builder for method chaining.

    • withWriterVersion

      public CarpetWriter.Builder<T> withWriterVersion(org.apache.parquet.column.ParquetProperties.WriterVersion version)
      Set the format version used by the constructed writer.
      Parameters:
      version - a WriterVersion
      Returns:
      this builder for method chaining.

    • enablePageWriteChecksum

      public CarpetWriter.Builder<T> enablePageWriteChecksum()
      Enables writing page level checksums for the constructed writer.
      Returns:
      this builder for method chaining.

    • withPageWriteChecksumEnabled

      public CarpetWriter.Builder<T> withPageWriteChecksumEnabled(boolean enablePageWriteChecksum)
      Enables writing page level checksums for the constructed writer.
      Parameters:
      enablePageWriteChecksum - whether page checksums should be written out
      Returns:
      this builder for method chaining.

    • withMaxBloomFilterBytes

      public CarpetWriter.Builder<T> withMaxBloomFilterBytes(int maxBloomFilterBytes)
      Set max Bloom filter bytes for related columns.
      Parameters:
      maxBloomFilterBytes - the max bytes of a Bloom filter bitset for a column.
      Returns:
      this builder for method chaining

    • withBloomFilterNDV

      public CarpetWriter.Builder<T> withBloomFilterNDV(String columnPath, long ndv)
      Sets the NDV (number of distinct values) for the specified column.
      Parameters:
      columnPath - the path of the column (dot-string)
      ndv - the NDV of the column
      Returns:
      this builder for method chaining.

    • withBloomFilterFPP

      public CarpetWriter.Builder<T> withBloomFilterFPP(String columnPath, double fpp)

    • withAdaptiveBloomFilterEnabled

      public CarpetWriter.Builder<T> withAdaptiveBloomFilterEnabled(boolean enabled)
      When NDV (number of distinct values) for a specified column is not set, whether to use `AdaptiveBloomFilter` to automatically adjust the BloomFilter size according to `parquet.bloom.filter.max.bytes`
      Parameters:
      enabled - whether to write bloom filter for the column

    • withBloomFilterCandidateNumber

      public CarpetWriter.Builder<T> withBloomFilterCandidateNumber(String columnPath, int number)
      When `AdaptiveBloomFilter` is enabled, set how many bloom filter candidates to use.
      Parameters:
      columnPath - the path of the column (dot-string)
      number - the number of candidate

    • withBloomFilterEnabled

      public CarpetWriter.Builder<T> withBloomFilterEnabled(boolean enabled)
      Sets the bloom filter enabled/disabled
      Parameters:
      enabled - whether to write bloom filters
      Returns:
      this builder for method chaining

    • withBloomFilterEnabled

      public CarpetWriter.Builder<T> withBloomFilterEnabled(String columnPath, boolean enabled)
      Sets the bloom filter enabled/disabled for the specified column. If not set for the column specifically the default enabled/disabled state will take place. See withBloomFilterEnabled(boolean).
      Parameters:
      columnPath - the path of the column (dot-string)
      enabled - whether to write bloom filter for the column
      Returns:
      this builder for method chaining

    • withMinRowCountForPageSizeCheck

      public CarpetWriter.Builder<T> withMinRowCountForPageSizeCheck(int min)
      Sets the minimum number of rows to write before a page size check is done.
      Parameters:
      min - writes at least `min` rows before invoking a page size check
      Returns:
      this builder for method chaining

    • withMaxRowCountForPageSizeCheck

      public CarpetWriter.Builder<T> withMaxRowCountForPageSizeCheck(int max)
      Sets the maximum number of rows to write before a page size check is done.
      Parameters:
      max - makes a page size check after `max` rows have been written
      Returns:
      this builder for method chaining

    • withColumnIndexTruncateLength

      public CarpetWriter.Builder<T> withColumnIndexTruncateLength(int length)
      Sets the length to be used for truncating binary values in a binary column index.
      Parameters:
      length - the length to truncate to
      Returns:
      this builder for method chaining

    • withStatisticsTruncateLength

      public CarpetWriter.Builder<T> withStatisticsTruncateLength(int length)
      Sets the length which the min/max binary values in row groups are truncated to.
      Parameters:
      length - the length to truncate to
      Returns:
      this builder for method chaining

    • config

      public CarpetWriter.Builder<T> config(String property, String value)
      Set a property that will be available to the read path. For writers that use a Hadoop configuration, this is the recommended way to add configuration values.
      Parameters:
      property - a String property name
      value - a String property value
      Returns:
      this builder for method chaining.

    • withExtraMetaData

      public CarpetWriter.Builder<T> withExtraMetaData(Map<String,String> extraMetaData)
      Add to writer metadata to include in the generated parquet file.
      Parameters:
      extraMetaData - to add
      Returns:
      this builder for method chaining.

    • withExtraMetaData

      public CarpetWriter.Builder<T> withExtraMetaData(String key, String value)
      Add to writer metadata to include in the generated parquet file.
      Parameters:
      key - of the metadata to add
      value - of the metadata to add
      Returns:
      this builder for method chaining.

    • withAllocator

      public CarpetWriter.Builder<T> withAllocator(org.apache.parquet.bytes.ByteBufferAllocator allocator)
      Sets the ByteBuffer allocator instance to be used for allocating memory for writing.
      Parameters:
      allocator - the allocator instance
      Returns:
      this builder for method chaining

    • withLevelStructure

      public CarpetWriter.Builder<T> withLevelStructure(AnnotatedLevels annotatedLevels)
      Set the type of collections type that will be generated following the LogicalTypes definition
      Parameters:
      annotatedLevels - an Enum configuring the number of levels
      Returns:
      this builder for method chaining.

    • withColumnNamingStrategy

      public CarpetWriter.Builder<T> withColumnNamingStrategy(ColumnNamingStrategy columnNamingStrategy)
      Set the strategy to use generating message and record column names
      Parameters:
      columnNamingStrategy - an Enum configuring the strategy to use
      Returns:
      this builder for method chaining.

    • withDefaultTimeUnit

      public CarpetWriter.Builder<T> withDefaultTimeUnit(TimeUnit defaultTimeUnit)
      Set the time unit resolution writing TIME or TIMESTAMP fields:
      • milliseconds
      • microseconds
      • nanoseconds
      Parameters:
      defaultTimeUnit - an Enum configuring the resolution to use
      Returns:
      this builder for method chaining.

    • withDefaultDecimal

      public CarpetWriter.Builder<T> withDefaultDecimal(int precision, int scale)
      Set Decimal precision and scale
      Returns:
      this builder for method chaining.

    • build

      public CarpetWriter<T> build() throws IOException
      Throws:
      IOException