co.cask.cdap.data.stream

Class StreamDataFileWriter

java.lang.Object

co.cask.cdap.data.stream.StreamDataFileWriter

All Implemented Interfaces:

FileWriter<co.cask.cdap.api.flow.flowlet.StreamEvent>, TimestampCloseable, Closeable, Flushable, AutoCloseable

@NotThreadSafe
public final class StreamDataFileWriter
extends Object
implements TimestampCloseable, Flushable, FileWriter<co.cask.cdap.api.flow.flowlet.StreamEvent>

File format Stream event file:

 event_file = <header> <data>* <end_marker>
 header = "E" "1" <properties>
 properties = Avro encoded with the properties schema
 data = <timestamp> <length> <stream_event>+
 timestamp = 8 bytes int64 for timestamp in milliseconds
 length = Avro encoded int32 for size in bytes for all <stream_event>s
 stream_event = Avro encoded bytes according to the StreamData schema
 end_marker = 8 bytes int64 with value == -(close_timestamp)

 
 

Stream index file:

 meta_file = <header> <index>*
 header = "I" "1" <properties>
 properties = Avro encoded with the properties schema
 index = <timestamp> <offset>
 timestamp = 8 bytes int64 for timestamp in milliseconds
 offset = 8 bytes int64 for offset to data block in the event file

 
 

Constructor Summary

Constructors

Constructor and Description
StreamDataFileWriter(com.google.common.io.OutputSupplier<? extends OutputStream> eventOutputSupplier, com.google.common.io.OutputSupplier<? extends OutputStream> indexOutputSupplier, long indexInterval) Constructs a new instance that writes to given outputs.
StreamDataFileWriter(com.google.common.io.OutputSupplier<? extends OutputStream> eventOutputSupplier, com.google.common.io.OutputSupplier<? extends OutputStream> indexOutputSupplier, long indexInterval, Map<String,String> properties) Constructs a new instance that writes to given outputs.

Method Summary

Methods

Modifier and Type	Method and Description
void	append(co.cask.cdap.api.flow.flowlet.StreamEvent event) Appends an event to the file.
void	appendAll(Iterator<? extends co.cask.cdap.api.flow.flowlet.StreamEvent> events) Writes multiple events to the stream file.
void	close()
void	flush()
long	getCloseTimestamp() Returns timestamp in milliseconds when the close operation happened.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

StreamDataFileWriter

public StreamDataFileWriter(com.google.common.io.OutputSupplier<? extends OutputStream> eventOutputSupplier,
                    com.google.common.io.OutputSupplier<? extends OutputStream> indexOutputSupplier,
                    long indexInterval)
                     throws IOException

Constructs a new instance that writes to given outputs. Same as calling StreamDataFileWriter(OutputSupplier, OutputSupplier, long, Map) with an empty property map.

Throws:

IOException

StreamDataFileWriter

public StreamDataFileWriter(com.google.common.io.OutputSupplier<? extends OutputStream> eventOutputSupplier,
                    com.google.common.io.OutputSupplier<? extends OutputStream> indexOutputSupplier,
                    long indexInterval,
                    Map<String,String> properties)
                     throws IOException

Constructs a new instance that writes to given outputs.

Parameters:

eventOutputSupplier - the provider of the OutputStream for writing events

indexOutputSupplier - the provider of the OutputStream for writing the index

indexInterval - the time interval in milliseconds for emitting a new index entry

properties - the property set that will be stored as file properties

Throws:

IOException - if there is an error in preparing the output streams

Method Detail

append

public void append(co.cask.cdap.api.flow.flowlet.StreamEvent event)
            throws IOException

Description copied from interface: FileWriter

Appends an event to the file.

Specified by:

append in interface FileWriter<co.cask.cdap.api.flow.flowlet.StreamEvent>

Parameters:

event - event to append

Throws:

IOException - if fail to append

appendAll

public void appendAll(Iterator<? extends co.cask.cdap.api.flow.flowlet.StreamEvent> events)
               throws IOException

Writes multiple events to the stream file. Events provided by the iterator must be sorted by timestamp. This method guarantees events with the same timestamp are written in the same data block. Note that events of the same timestamp are all buffered in memory before writing to disk, since the data block length needs to be known before it can be written to disk.

Specified by:

appendAll in interface FileWriter<co.cask.cdap.api.flow.flowlet.StreamEvent>

Parameters:

events - an Iterator that provides events to append

Throws:

IOException

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException

flush

public void flush()
           throws IOException

Specified by:

flush in interface Flushable

Throws:

IOException

getCloseTimestamp

public long getCloseTimestamp()

Description copied from interface: TimestampCloseable

Returns timestamp in milliseconds when the close operation happened.

Specified by:

getCloseTimestamp in interface TimestampCloseable