co.cask.cdap.data.stream

Class TimePartitionedStreamFileWriter

java.lang.Object

co.cask.cdap.data.file.PartitionedFileWriter<co.cask.cdap.api.flow.flowlet.StreamEvent,TimePartitionedStreamFileWriter.TimePartition>

co.cask.cdap.data.stream.TimePartitionedStreamFileWriter

All Implemented Interfaces:

FileWriter<co.cask.cdap.api.flow.flowlet.StreamEvent>, Closeable, Flushable, AutoCloseable

@NotThreadSafe
public class TimePartitionedStreamFileWriter
extends PartitionedFileWriter<co.cask.cdap.api.flow.flowlet.StreamEvent,TimePartitionedStreamFileWriter.TimePartition>

Stream file path format:

Each file has path pattern

     [streamName]/[partitionName]/[bucketName].[dat|idx]
 

Where .dat is the event data file, .idx is the accompany index file.

The partitionName is formatted as

   [partitionStartTime].[duration]
 

with both partitionStartTime and duration in seconds.

The bucketName is formatted as

   "bucket".[bucketId].[seqNo]
 

where the bucketId is an integer. The seqNo is a strictly increasing integer for the same bucketId.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	TimePartitionedStreamFileWriter.TimePartition Uses timestamp to represent partition information.

Nested classes/interfaces inherited from class co.cask.cdap.data.file.PartitionedFileWriter

PartitionedFileWriter.PartitionedFileWriterFactory<T,P>

Constructor Summary

Constructors

Constructor and Description
TimePartitionedStreamFileWriter(Location streamLocation, long partitionDuration, String fileNamePrefix, long indexInterval)

Method Summary

Methods

Modifier and Type	Method and Description
protected TimePartitionedStreamFileWriter.TimePartition	getPartition(co.cask.cdap.api.flow.flowlet.StreamEvent event) Returns the partition ID for the given event.
protected void	partitionChanged(TimePartitionedStreamFileWriter.TimePartition oldPartition, TimePartitionedStreamFileWriter.TimePartition newPartition) Invoked when partition changed (different from the last event get appended).

Methods inherited from class co.cask.cdap.data.file.PartitionedFileWriter

append, appendAll, close, closePartitionWriter, flush

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TimePartitionedStreamFileWriter

public TimePartitionedStreamFileWriter(Location streamLocation,
                               long partitionDuration,
                               String fileNamePrefix,
                               long indexInterval)

Method Detail

getPartition

protected TimePartitionedStreamFileWriter.TimePartition getPartition(co.cask.cdap.api.flow.flowlet.StreamEvent event)

Description copied from class: PartitionedFileWriter

Returns the partition ID for the given event.

Specified by:

getPartition in class PartitionedFileWriter<co.cask.cdap.api.flow.flowlet.StreamEvent,TimePartitionedStreamFileWriter.TimePartition>

Parameters:

event - The event for computing a partition ID.

Returns:

A long representing the partition ID.

partitionChanged

protected void partitionChanged(TimePartitionedStreamFileWriter.TimePartition oldPartition,
                    TimePartitionedStreamFileWriter.TimePartition newPartition)
                         throws IOException

Description copied from class: PartitionedFileWriter

Invoked when partition changed (different from the last event get appended).

Overrides:

partitionChanged in class PartitionedFileWriter<co.cask.cdap.api.flow.flowlet.StreamEvent,TimePartitionedStreamFileWriter.TimePartition>

Throws:

IOException