Package org.opensearch.client.opensearch._helpers.bulk

Class BulkIngester<Context>

java.lang.Object
org.opensearch.client.opensearch._helpers.bulk.BulkIngester<Context>
Type Parameters:
Context - optional context type to associate with each bulk operation
All Implemented Interfaces:
AutoCloseable
public class BulkIngester<Context> extends Object implements AutoCloseable
A bulk ingester for efficiently indexing large volumes of documents to OpenSearch.

The BulkIngester buffers bulk operations and automatically flushes them based on configurable thresholds:

  • Number of operations (maxOperations)
  • Total size in bytes (maxSize)
  • Time interval (flushInterval)

It also provides:

  • Backpressure control via maxConcurrentRequests to prevent overwhelming the cluster
  • Automatic retries with configurable backoff policies for failed operations
  • Per-request context tracking via the optional Context type parameter
  • Event notifications through the BulkListener interface

The ingester is thread-safe and can be used concurrently from multiple threads. It must be closed when no longer needed to flush any buffered operations and release resources.

Example usage: 


 BulkIngester<Void> ingester = BulkIngester.of(b -> b
     .client(client)
     .maxOperations(1000)
     .flushInterval(5, TimeUnit.SECONDS)
 );

 // Add operations
 ingester.add(op -> op.index(i -> i.index("my-index").id("1").document(myDoc)));

 // Close when done (flushes remaining operations)
 ingester.close();

  • Method Details

    • maxOperations

      public int maxOperations()
      The configured max operations to buffer in a single bulk request.

    • maxSize

      public long maxSize()
      The configured maximum size in bytes for a bulk request. Operations are added to the request until adding an operation leads the request to exceed this size.

    • maxConcurrentRequests

      public int maxConcurrentRequests()
      The configured maximum number of concurrent request sent to OpenSearch.

    • flushInterval

      public Duration flushInterval()
      The configured flush period.

    • pendingOperations

      public int pendingOperations()
      The number of operations that have been buffered, waiting to be sent.

    • pendingOperationsSize

      public long pendingOperationsSize()
      The size in bytes of operations that have been buffered, waiting to be sent.

    • pendingRequests

      public int pendingRequests()
      The number of in flight bulk requests.

    • operationsCount

      public long operationsCount()
      Statistics: the number of operations that were added to this ingester since it was created.

    • operationContentionsCount

      public long operationContentionsCount()
      Statistics: the number of operations that had to wait before being added because the operation buffer was full and the number of http requests in flight exceeded the configured maximum number.
      See Also:

    • requestCount

      public long requestCount()
      Statistics: the number of bulk requests that were produced by this ingester since it was created.

    • requestContentionsCount

      public long requestContentionsCount()
      Statistics: the number of bulk requests that could not be sent immediately because the number of http requests in flight exceeded the configured maximum number.
      See Also:
      • BulkIngester.Builder.maxConcurrentRequests

    • flush

      public void flush()
      Manually flush any buffered operations, sending them to OpenSearch immediately.

      This method is non-blocking and returns immediately. Flushing happens asynchronously, subject to the maxConcurrentRequests limit. Operations that are scheduled for retry will only be sent once their retry delay has elapsed.

      This method is useful when you want to ensure operations are sent without waiting for automatic flush triggers (maxOperations, maxSize, or flushInterval).

    • add

      public void add(BulkOperation operation, Context context)
      Add a bulk operation to the ingester with an associated context.

      The operation will be buffered and sent to OpenSearch when any of the configured flush thresholds is reached (maxOperations, maxSize, or flushInterval), or when flush() or close() is called.

      This method blocks if adding the operation would exceed the maxConcurrentRequests limit, providing backpressure to prevent overwhelming the cluster.

      Parameters:
      operation - the bulk operation to add
      context - optional context to associate with this operation for tracking purposes
      Throws:
      IllegalStateException - if the ingester has been closed

    • add

      public void add(BulkOperation operation)
      Add a bulk operation to the ingester without an associated context.

      Equivalent to calling add(operation, null).

      Parameters:
      operation - the bulk operation to add
      Throws:
      IllegalStateException - if the ingester has been closed
      See Also:

    • add

      public void add(Function<BulkOperation.Builder,ObjectBuilder<BulkOperation>> f)
      Add a bulk operation to the ingester using a builder function, without an associated context.

      This is a convenience method that accepts a function to build the bulk operation inline.

      Parameters:
      f - the function to build the bulk operation
      Throws:
      IllegalStateException - if the ingester has been closed
      See Also:

    • add

      public void add(Function<BulkOperation.Builder,ObjectBuilder<BulkOperation>> f, Context context)
      Add a bulk operation to the ingester using a builder function, with an associated context.

      This is a convenience method that accepts a function to build the bulk operation inline.

      Parameters:
      f - the function to build the bulk operation
      context - optional context to associate with this operation for tracking purposes
      Throws:
      IllegalStateException - if the ingester has been closed
      See Also:

    • close

      public void close()
      Close this ingester, first flushing any buffered operations. This does not close the underlying OpenSearchClient and Transport.
      Specified by:
      close in interface AutoCloseable

    • of

      public static <Context> BulkIngester<Context> of(Function<BulkIngester.Builder<Context>,BulkIngester.Builder<Context>> f)