Package org.apache.beam.sdk.io.gcp.bigquery

Class ConvertMessagesDoFn<DestinationT extends @NonNull java.lang.Object,​ElementT>

  • All Implemented Interfaces:
    java.io.Serializable, org.apache.beam.sdk.transforms.display.HasDisplayData
    public class ConvertMessagesDoFn<DestinationT extends @NonNull java.lang.Object,​ElementT>
extends org.apache.beam.sdk.transforms.DoFn<org.apache.beam.sdk.values.KV<DestinationT,​ElementT>,​org.apache.beam.sdk.values.KV<DestinationT,​StorageApiWritePayload>>
    DoFn that interacts with the StorageApiDynamicDestinations instance to convert messages to StorageApiWritePaylod. Messages that fail to convert are routed to the dead-letter PCollection. If schemaUpdateOptions are set, then messages that fail to convert due to missing columns are routed to a buffering transform that holds them until the table's schema has been updated.
    See Also:
    Serialized Form

    • Nested Class Summary

      • Nested classes/interfaces inherited from class org.apache.beam.sdk.transforms.DoFn

        org.apache.beam.sdk.transforms.DoFn.AlwaysFetched, org.apache.beam.sdk.transforms.DoFn.BoundedPerElement, org.apache.beam.sdk.transforms.DoFn.BundleFinalizer, org.apache.beam.sdk.transforms.DoFn.CurrentRecordId, org.apache.beam.sdk.transforms.DoFn.CurrentRecordOffset, org.apache.beam.sdk.transforms.DoFn.Element, org.apache.beam.sdk.transforms.DoFn.FieldAccess, org.apache.beam.sdk.transforms.DoFn.FinishBundle, org.apache.beam.sdk.transforms.DoFn.FinishBundleContext, org.apache.beam.sdk.transforms.DoFn.FireTimestamp, org.apache.beam.sdk.transforms.DoFn.GetInitialRestriction, org.apache.beam.sdk.transforms.DoFn.GetInitialWatermarkEstimatorState, org.apache.beam.sdk.transforms.DoFn.GetRestrictionCoder, org.apache.beam.sdk.transforms.DoFn.GetSize, org.apache.beam.sdk.transforms.DoFn.GetWatermarkEstimatorStateCoder, org.apache.beam.sdk.transforms.DoFn.Key, org.apache.beam.sdk.transforms.DoFn.MultiOutputReceiver, org.apache.beam.sdk.transforms.DoFn.NewTracker, org.apache.beam.sdk.transforms.DoFn.NewWatermarkEstimator, org.apache.beam.sdk.transforms.DoFn.OnTimer, org.apache.beam.sdk.transforms.DoFn.OnTimerContext, org.apache.beam.sdk.transforms.DoFn.OnTimerFamily, org.apache.beam.sdk.transforms.DoFn.OnWindowExpiration, org.apache.beam.sdk.transforms.DoFn.OnWindowExpirationContext, org.apache.beam.sdk.transforms.DoFn.OutputReceiver<T extends java.lang.Object>, org.apache.beam.sdk.transforms.DoFn.ProcessContext, org.apache.beam.sdk.transforms.DoFn.ProcessContinuation, org.apache.beam.sdk.transforms.DoFn.ProcessElement, org.apache.beam.sdk.transforms.DoFn.RequiresStableInput, org.apache.beam.sdk.transforms.DoFn.RequiresTimeSortedInput, org.apache.beam.sdk.transforms.DoFn.Restriction, org.apache.beam.sdk.transforms.DoFn.Setup, org.apache.beam.sdk.transforms.DoFn.SideInput, org.apache.beam.sdk.transforms.DoFn.SplitRestriction, org.apache.beam.sdk.transforms.DoFn.StartBundle, org.apache.beam.sdk.transforms.DoFn.StartBundleContext, org.apache.beam.sdk.transforms.DoFn.StateId, org.apache.beam.sdk.transforms.DoFn.Teardown, org.apache.beam.sdk.transforms.DoFn.TimerFamily, org.apache.beam.sdk.transforms.DoFn.TimerId, org.apache.beam.sdk.transforms.DoFn.Timestamp, org.apache.beam.sdk.transforms.DoFn.TruncateRestriction, org.apache.beam.sdk.transforms.DoFn.UnboundedPerElement, org.apache.beam.sdk.transforms.DoFn.WatermarkEstimatorState, org.apache.beam.sdk.transforms.DoFn.WindowedContext

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      void finishBundle​(org.apache.beam.sdk.transforms.DoFn.FinishBundleContext c)  
      java.lang.Iterable<org.apache.beam.sdk.values.TimestampedValue<org.apache.beam.sdk.values.KV<DestinationT,​ElementT>>> handleProcessElements​(org.apache.beam.sdk.io.gcp.bigquery.StorageApiDynamicDestinations.MessageConverter<ElementT> messageConverter, java.lang.Iterable<org.apache.beam.sdk.values.TimestampedValue<org.apache.beam.sdk.values.KV<DestinationT,​ElementT>>> values, org.apache.beam.sdk.transforms.DoFn.MultiOutputReceiver o, TableRowToStorageApiProto.ErrorCollector errorCollector)  
      void onTeardown()  
      void processElement​(org.apache.beam.sdk.transforms.DoFn.ProcessContext c, org.apache.beam.sdk.options.PipelineOptions pipelineOptions, org.apache.beam.sdk.values.KV<DestinationT,​ElementT> element, org.joda.time.Instant timestamp, org.apache.beam.sdk.transforms.DoFn.MultiOutputReceiver o)  
      void startBundle()  

      • Methods inherited from class org.apache.beam.sdk.transforms.DoFn

        getAllowedTimestampSkew, getInputTypeDescriptor, getOutputTypeDescriptor, populateDisplayData, prepareForProcessing

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Method Detail

      • onTeardown

        @Teardown
public void onTeardown()

      • startBundle

        @StartBundle
public void startBundle()

      • processElement

        @ProcessElement
public void processElement​(org.apache.beam.sdk.transforms.DoFn.ProcessContext c,
                           org.apache.beam.sdk.options.PipelineOptions pipelineOptions,
                           @Element
                           org.apache.beam.sdk.values.KV<DestinationT,​ElementT> element,
                           @Timestamp
                           org.joda.time.Instant timestamp,
                           org.apache.beam.sdk.transforms.DoFn.MultiOutputReceiver o)
                    throws java.lang.Exception
        Throws:
        java.lang.Exception

      • finishBundle

        @FinishBundle
public void finishBundle​(org.apache.beam.sdk.transforms.DoFn.FinishBundleContext c)
                  throws java.lang.Exception
        Throws:
        java.lang.Exception

      • handleProcessElements

        public java.lang.Iterable<org.apache.beam.sdk.values.TimestampedValue<org.apache.beam.sdk.values.KV<DestinationT,​ElementT>>> handleProcessElements​(org.apache.beam.sdk.io.gcp.bigquery.StorageApiDynamicDestinations.MessageConverter<ElementT> messageConverter,
                                                                                                                                                         java.lang.Iterable<org.apache.beam.sdk.values.TimestampedValue<org.apache.beam.sdk.values.KV<DestinationT,​ElementT>>> values,
                                                                                                                                                         org.apache.beam.sdk.transforms.DoFn.MultiOutputReceiver o,
                                                                                                                                                         TableRowToStorageApiProto.ErrorCollector errorCollector)
                                                                                                                                                  throws java.lang.Exception
        Throws:
        java.lang.Exception