Package org.apache.beam.sdk.io.gcp.bigquery

Class SchemaUpdateHoldingFn<DestinationT extends @NonNull java.lang.Object,​ElementT>

  • All Implemented Interfaces:
    java.io.Serializable, org.apache.beam.sdk.transforms.display.HasDisplayData
    public class SchemaUpdateHoldingFn<DestinationT extends @NonNull java.lang.Object,​ElementT>
extends org.apache.beam.sdk.transforms.DoFn<org.apache.beam.sdk.values.KV<org.apache.beam.sdk.util.ShardedKey<DestinationT>,​@Nullable ElementT>,​org.apache.beam.sdk.values.KV<DestinationT,​StorageApiWritePayload>>
    This is a stateful DoFn that buffers elements that triggered table schema update. Once the table schema has been updated, this reprocesses the messages and allows them to continue on through the sink. This DoFn receives messages from both ConvertMessagesDoFn and PatchTableSchemaDoFn. ConvertMessagesDoFn sends elements to be buffered. PatchTableSchemaDoFn sends a null element as a sentinal to indicate that the table has recently been patched, which triggers us to immediately try and reprocess the buffered elements.
    See Also:
    Serialized Form

    • Nested Class Summary

      • Nested classes/interfaces inherited from class org.apache.beam.sdk.transforms.DoFn

        org.apache.beam.sdk.transforms.DoFn.AlwaysFetched, org.apache.beam.sdk.transforms.DoFn.BoundedPerElement, org.apache.beam.sdk.transforms.DoFn.BundleFinalizer, org.apache.beam.sdk.transforms.DoFn.CurrentRecordId, org.apache.beam.sdk.transforms.DoFn.CurrentRecordOffset, org.apache.beam.sdk.transforms.DoFn.Element, org.apache.beam.sdk.transforms.DoFn.FieldAccess, org.apache.beam.sdk.transforms.DoFn.FinishBundle, org.apache.beam.sdk.transforms.DoFn.FinishBundleContext, org.apache.beam.sdk.transforms.DoFn.FireTimestamp, org.apache.beam.sdk.transforms.DoFn.GetInitialRestriction, org.apache.beam.sdk.transforms.DoFn.GetInitialWatermarkEstimatorState, org.apache.beam.sdk.transforms.DoFn.GetRestrictionCoder, org.apache.beam.sdk.transforms.DoFn.GetSize, org.apache.beam.sdk.transforms.DoFn.GetWatermarkEstimatorStateCoder, org.apache.beam.sdk.transforms.DoFn.Key, org.apache.beam.sdk.transforms.DoFn.MultiOutputReceiver, org.apache.beam.sdk.transforms.DoFn.NewTracker, org.apache.beam.sdk.transforms.DoFn.NewWatermarkEstimator, org.apache.beam.sdk.transforms.DoFn.OnTimer, org.apache.beam.sdk.transforms.DoFn.OnTimerContext, org.apache.beam.sdk.transforms.DoFn.OnTimerFamily, org.apache.beam.sdk.transforms.DoFn.OnWindowExpiration, org.apache.beam.sdk.transforms.DoFn.OnWindowExpirationContext, org.apache.beam.sdk.transforms.DoFn.OutputReceiver<T extends java.lang.Object>, org.apache.beam.sdk.transforms.DoFn.ProcessContext, org.apache.beam.sdk.transforms.DoFn.ProcessContinuation, org.apache.beam.sdk.transforms.DoFn.ProcessElement, org.apache.beam.sdk.transforms.DoFn.RequiresStableInput, org.apache.beam.sdk.transforms.DoFn.RequiresTimeSortedInput, org.apache.beam.sdk.transforms.DoFn.Restriction, org.apache.beam.sdk.transforms.DoFn.Setup, org.apache.beam.sdk.transforms.DoFn.SideInput, org.apache.beam.sdk.transforms.DoFn.SplitRestriction, org.apache.beam.sdk.transforms.DoFn.StartBundle, org.apache.beam.sdk.transforms.DoFn.StartBundleContext, org.apache.beam.sdk.transforms.DoFn.StateId, org.apache.beam.sdk.transforms.DoFn.Teardown, org.apache.beam.sdk.transforms.DoFn.TimerFamily, org.apache.beam.sdk.transforms.DoFn.TimerId, org.apache.beam.sdk.transforms.DoFn.Timestamp, org.apache.beam.sdk.transforms.DoFn.TruncateRestriction, org.apache.beam.sdk.transforms.DoFn.UnboundedPerElement, org.apache.beam.sdk.transforms.DoFn.WatermarkEstimatorState, org.apache.beam.sdk.transforms.DoFn.WindowedContext

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      org.joda.time.Duration getAllowedTimestampSkew()  
      void onPollTimer​(org.apache.beam.sdk.util.ShardedKey<DestinationT> key, org.apache.beam.sdk.options.PipelineOptions pipelineOptions, org.apache.beam.sdk.state.BagState<org.apache.beam.sdk.values.TimestampedValue<ElementT>> bag, org.apache.beam.sdk.state.CombiningState<java.lang.Long,​long[],​java.lang.Long> minBufferedTimestamp, org.apache.beam.sdk.state.ValueState<java.lang.Long> timerTs, org.apache.beam.sdk.state.Timer pollTimer, org.apache.beam.sdk.transforms.windowing.BoundedWindow window, org.apache.beam.sdk.transforms.DoFn.MultiOutputReceiver o)  
      void onTeardown()  
      void onWindowExpiration​(org.apache.beam.sdk.util.ShardedKey<DestinationT> key, org.apache.beam.sdk.options.PipelineOptions pipelineOptions, org.apache.beam.sdk.state.BagState<org.apache.beam.sdk.values.TimestampedValue<ElementT>> bag, org.apache.beam.sdk.state.CombiningState<java.lang.Long,​long[],​java.lang.Long> minBufferedTimestamp, org.apache.beam.sdk.transforms.DoFn.MultiOutputReceiver o)  
      void processElement​(org.apache.beam.sdk.values.KV<org.apache.beam.sdk.util.ShardedKey<DestinationT>,​@Nullable ElementT> element, org.joda.time.Instant timestamp, org.apache.beam.sdk.state.BagState<org.apache.beam.sdk.values.TimestampedValue<ElementT>> bag, org.apache.beam.sdk.state.CombiningState<java.lang.Long,​long[],​java.lang.Long> minBufferedTimestamp, org.apache.beam.sdk.state.ValueState<java.lang.Long> timerTs, org.apache.beam.sdk.state.Timer pollTimer, org.apache.beam.sdk.transforms.DoFn.ProcessContext context, org.apache.beam.sdk.transforms.windowing.BoundedWindow window, org.apache.beam.sdk.transforms.DoFn.MultiOutputReceiver o)  
      void startBundle()  
      boolean tryFlushBuffer​(DestinationT destination, org.apache.beam.sdk.options.PipelineOptions pipelineOptions, org.apache.beam.sdk.state.BagState<org.apache.beam.sdk.values.TimestampedValue<ElementT>> bag, org.apache.beam.sdk.state.CombiningState<java.lang.Long,​long[],​java.lang.Long> minBufferedTimestamp, org.apache.beam.sdk.transforms.DoFn.MultiOutputReceiver o)  

      • Methods inherited from class org.apache.beam.sdk.transforms.DoFn

        getInputTypeDescriptor, getOutputTypeDescriptor, populateDisplayData, prepareForProcessing

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Method Detail

      • startBundle

        @StartBundle
public void startBundle()

      • onTeardown

        @Teardown
public void onTeardown()

      • processElement

        @ProcessElement
public void processElement​(@Element
                           org.apache.beam.sdk.values.KV<org.apache.beam.sdk.util.ShardedKey<DestinationT>,​@Nullable ElementT> element,
                           @Timestamp
                           org.joda.time.Instant timestamp,
                           @StateId("bufferedElements")
                           org.apache.beam.sdk.state.BagState<org.apache.beam.sdk.values.TimestampedValue<ElementT>> bag,
                           @StateId("minBufferedTimestamp")
                           org.apache.beam.sdk.state.CombiningState<java.lang.Long,​long[],​java.lang.Long> minBufferedTimestamp,
                           @StateId("timerTimestamp")
                           org.apache.beam.sdk.state.ValueState<java.lang.Long> timerTs,
                           @TimerId("pollTimer")
                           org.apache.beam.sdk.state.Timer pollTimer,
                           org.apache.beam.sdk.transforms.DoFn.ProcessContext context,
                           org.apache.beam.sdk.transforms.windowing.BoundedWindow window,
                           org.apache.beam.sdk.transforms.DoFn.MultiOutputReceiver o)
                    throws java.lang.Exception
        Throws:
        java.lang.Exception

      • getAllowedTimestampSkew

        public org.joda.time.Duration getAllowedTimestampSkew()
        Overrides:
        getAllowedTimestampSkew in class org.apache.beam.sdk.transforms.DoFn<org.apache.beam.sdk.values.KV<org.apache.beam.sdk.util.ShardedKey<DestinationT extends @NonNull java.lang.Object>,​@Nullable ElementT>,​org.apache.beam.sdk.values.KV<DestinationT extends @NonNull java.lang.Object,​StorageApiWritePayload>>

      • onPollTimer

        @OnTimer("pollTimer")
public void onPollTimer​(@Key
                        org.apache.beam.sdk.util.ShardedKey<DestinationT> key,
                        org.apache.beam.sdk.options.PipelineOptions pipelineOptions,
                        @StateId("bufferedElements")
                        org.apache.beam.sdk.state.BagState<org.apache.beam.sdk.values.TimestampedValue<ElementT>> bag,
                        @StateId("minBufferedTimestamp")
                        org.apache.beam.sdk.state.CombiningState<java.lang.Long,​long[],​java.lang.Long> minBufferedTimestamp,
                        @StateId("timerTimestamp")
                        org.apache.beam.sdk.state.ValueState<java.lang.Long> timerTs,
                        @TimerId("pollTimer")
                        org.apache.beam.sdk.state.Timer pollTimer,
                        org.apache.beam.sdk.transforms.windowing.BoundedWindow window,
                        org.apache.beam.sdk.transforms.DoFn.MultiOutputReceiver o)
                 throws java.lang.Exception
        Throws:
        java.lang.Exception

      • onWindowExpiration

        @OnWindowExpiration
public void onWindowExpiration​(@Key
                               org.apache.beam.sdk.util.ShardedKey<DestinationT> key,
                               org.apache.beam.sdk.options.PipelineOptions pipelineOptions,
                               @StateId("bufferedElements")
                               org.apache.beam.sdk.state.BagState<org.apache.beam.sdk.values.TimestampedValue<ElementT>> bag,
                               @StateId("minBufferedTimestamp")
                               org.apache.beam.sdk.state.CombiningState<java.lang.Long,​long[],​java.lang.Long> minBufferedTimestamp,
                               org.apache.beam.sdk.transforms.DoFn.MultiOutputReceiver o)
                        throws java.lang.Exception
        Throws:
        java.lang.Exception

      • tryFlushBuffer

        public boolean tryFlushBuffer​(DestinationT destination,
                              org.apache.beam.sdk.options.PipelineOptions pipelineOptions,
                              @StateId("bufferedElements")
                              org.apache.beam.sdk.state.BagState<org.apache.beam.sdk.values.TimestampedValue<ElementT>> bag,
                              @StateId("minBufferedTimestamp")
                              org.apache.beam.sdk.state.CombiningState<java.lang.Long,​long[],​java.lang.Long> minBufferedTimestamp,
                              org.apache.beam.sdk.transforms.DoFn.MultiOutputReceiver o)
                       throws java.lang.Exception
        Throws:
        java.lang.Exception