Package io.confluent.parallelconsumer.state

Class PartitionStateManager<K,V>

java.lang.Object
io.confluent.parallelconsumer.state.PartitionStateManager<K,V>
All Implemented Interfaces:
org.apache.kafka.clients.consumer.ConsumerRebalanceListener
public class PartitionStateManager<K,V> extends Object implements org.apache.kafka.clients.consumer.ConsumerRebalanceListener
In charge of managing PartitionStates.

This state is shared between the BrokerPollSystem thread and the AbstractParallelEoSStreamProcessor.

See Also:

  • Field Details

    • USED_PAYLOAD_THRESHOLD_MULTIPLIER_DEFAULT

      public static final double USED_PAYLOAD_THRESHOLD_MULTIPLIER_DEFAULT
      See Also:

  • Constructor Details

  • Method Details

    • getPartitionState

      public PartitionState<K,V> getPartitionState(org.apache.kafka.common.TopicPartition tp)

    • onPartitionsAssigned

      public void onPartitionsAssigned(Collection<org.apache.kafka.common.TopicPartition> assignedPartitions)
      Load offset map for assigned assignedPartitions
      Specified by:
      onPartitionsAssigned in interface org.apache.kafka.clients.consumer.ConsumerRebalanceListener

    • onPartitionsRevoked

      public void onPartitionsRevoked(Collection<org.apache.kafka.common.TopicPartition> partitions)
      Clear offset map for revoked partitions

      AbstractParallelEoSStreamProcessor.onPartitionsRevoked(java.util.Collection<org.apache.kafka.common.TopicPartition>) handles committing off offsets upon revoke

      Specified by:
      onPartitionsRevoked in interface org.apache.kafka.clients.consumer.ConsumerRebalanceListener
      See Also:

    • onPartitionsLost

      public void onPartitionsLost(Collection<org.apache.kafka.common.TopicPartition> partitions)
      Clear offset map for lost partitions
      Specified by:
      onPartitionsLost in interface org.apache.kafka.clients.consumer.ConsumerRebalanceListener

    • onOffsetCommitSuccess

      public void onOffsetCommitSuccess(Map<org.apache.kafka.common.TopicPartition,org.apache.kafka.clients.consumer.OffsetAndMetadata> committed)
      Truncate our tracked offsets as a commit was successful, so the low water mark rises, and we dont' need to track as much anymore.

      When commits are made to broker, we can throw away all the individually tracked offsets before the committed offset.

    • getEpochOfPartitionForRecord

      public Long getEpochOfPartitionForRecord(org.apache.kafka.clients.consumer.ConsumerRecord<K,V> rec)
      Returns:
      the current epoch of the partition this record belongs to

    • getEpochOfPartition

      public Long getEpochOfPartition(org.apache.kafka.common.TopicPartition partition)
      Returns:
      the current epoch of the partition

    • isRecordPreviouslyCompleted

      public boolean isRecordPreviouslyCompleted(org.apache.kafka.clients.consumer.ConsumerRecord<K,V> rec)

    • isAllowedMoreRecords

      public boolean isAllowedMoreRecords(org.apache.kafka.common.TopicPartition tp)
      Check we have capacity in offset storage to process more messages

    • isAllowedMoreRecords

      public boolean isAllowedMoreRecords(WorkContainer<?,?> wc)
      See Also:

    • hasWorkInCommitQueues

      public boolean hasWorkInCommitQueues()

    • getNumberOfEntriesInPartitionQueues

      public long getNumberOfEntriesInPartitionQueues()

    • getHighestSeenOffset

      public long getHighestSeenOffset(org.apache.kafka.common.TopicPartition tp)

    • addWorkContainer

      public void addWorkContainer(WorkContainer<K,V> wc)

    • isBlocked

      public boolean isBlocked(org.apache.kafka.common.TopicPartition topicPartition)
      Checks if partition is blocked with back pressure.

      If false, more messages are allowed to process for this partition.

      If true, we have calculated that we can't record any more offsets for this partition, as our best performing encoder requires nearly as much space is available for this partitions allocation of the maximum offset metadata size.

      Default (missing elements) is true - more messages can be processed.

      See Also:

    • isPartitionRemovedOrNeverAssigned

      public boolean isPartitionRemovedOrNeverAssigned(org.apache.kafka.clients.consumer.ConsumerRecord<?,?> rec)

    • onSuccess

      public void onSuccess(WorkContainer<K,V> wc)

    • onFailure

      public void onFailure(WorkContainer<K,V> wc)

    • collectDirtyCommitData

      public Map<org.apache.kafka.common.TopicPartition,org.apache.kafka.clients.consumer.OffsetAndMetadata> collectDirtyCommitData()

    • couldBeTakenAsWork

      public boolean couldBeTakenAsWork(WorkContainer<?,?> workContainer)

    • getUSED_PAYLOAD_THRESHOLD_MULTIPLIER

      public static double getUSED_PAYLOAD_THRESHOLD_MULTIPLIER()
      Best efforts attempt to prevent usage of offset payload beyond X% - as encoding size test is currently only done per batch, we need to leave some buffer for the required space to overrun before hitting the hard limit where we have to drop the offset payload entirely.

    • setUSED_PAYLOAD_THRESHOLD_MULTIPLIER

      public static void setUSED_PAYLOAD_THRESHOLD_MULTIPLIER(double USED_PAYLOAD_THRESHOLD_MULTIPLIER)
      Best efforts attempt to prevent usage of offset payload beyond X% - as encoding size test is currently only done per batch, we need to leave some buffer for the required space to overrun before hitting the hard limit where we have to drop the offset payload entirely.