Package io.confluent.parallelconsumer.internal

Class ExternalEngine<K,V>

java.lang.Object
io.confluent.parallelconsumer.internal.AbstractParallelEoSStreamProcessor<K,V>
io.confluent.parallelconsumer.internal.ExternalEngine<K,V>
All Implemented Interfaces:
DrainingCloseable, ParallelConsumer<K,V>, Closeable, AutoCloseable, org.apache.kafka.clients.consumer.ConsumerRebalanceListener
public abstract class ExternalEngine<K,V> extends AbstractParallelEoSStreamProcessor<K,V>
Overrides key aspects required in common for other threading engines like Vert.x and Reactor

  • Constructor Details

  • Method Details

    • getTargetOutForProcessing

      protected int getTargetOutForProcessing()
      Overrides:
      getTargetOutForProcessing in class AbstractParallelEoSStreamProcessor<K,V>
      Returns:
      the number of records to try to get, based on the current count of records outstanding - but unlike core, we don't pipeline messages into the executor pool for processing.

    • checkPipelinePressure

      protected void checkPipelinePressure()
      Description copied from class: AbstractParallelEoSStreamProcessor
      Checks the system has enough pressure in the pipeline of work, if not attempts to step up the load factor.
      Overrides:
      checkPipelinePressure in class AbstractParallelEoSStreamProcessor<K,V>

    • setupWorkerPool

      protected ThreadPoolExecutor setupWorkerPool(int poolSize)
      The vert.x module doesn't use any thread pool for dispatching work, as the work is all done by the vert.x engine. This thread is only used to dispatch the work to vert.x.

      TODO optimise thread usage by not using any extra thread here at all - go straight from the control thread to vert.x.

      Overrides:
      setupWorkerPool in class AbstractParallelEoSStreamProcessor<K,V>

    • onUserFunctionSuccess

      protected void onUserFunctionSuccess(WorkContainer<K,V> wc, List<?> resultsFromUserFunction)
      With Vertx and Reactor, a function hasn't succeeded until the inner vertx function has also succeeded logging
      Overrides:
      onUserFunctionSuccess in class AbstractParallelEoSStreamProcessor<K,V>

    • addToMailBoxOnUserFunctionSuccess

      protected void addToMailBoxOnUserFunctionSuccess(WorkContainer<K,V> wc, List<?> resultsFromUserFunction)
      With Vertx and Reactor, a function hasn't succeeded until the inner vertx function has also succeeded no op
      Overrides:
      addToMailBoxOnUserFunctionSuccess in class AbstractParallelEoSStreamProcessor<K,V>

    • isAsyncFutureWork

      protected abstract boolean isAsyncFutureWork(List<?> resultsFromUserFunction)
      System dependent way for the core engine to know if an external scheduler needs to be relied on for completion, as opposed to the internal threading system in the core module.
      Parameters:
      resultsFromUserFunction - the results to test
      Returns:
      true if the work needs special treatment