Package org.apache.iotdb.tsfile.write.writer

Class TsFileIOWriter

java.lang.Object

org.apache.iotdb.tsfile.write.writer.TsFileIOWriter

All Implemented Interfaces:

java.lang.AutoCloseable

Direct Known Subclasses:

ForceAppendTsFileWriter, RestorableTsFileIOWriter

public class TsFileIOWriter
extends java.lang.Object
implements java.lang.AutoCloseable

TsFileIOWriter is used to construct metadata and write data stored in memory to output stream.

Field Summary

Fields

Modifier and Type	Field	Description
protected boolean	canWrite
static java.lang.String	CHUNK_METADATA_TEMP_FILE_SUFFIX
protected java.util.List<ChunkGroupMetadata>	chunkGroupMetadataList
protected java.util.List<ChunkMetadata>	chunkMetadataList
protected java.io.File	chunkMetadataTempFile
protected static TSFileConfig	config
protected ChunkMetadata	currentChunkMetadata
protected long	currentChunkMetadataSize
protected boolean	enableMemoryControl
protected java.util.LinkedList<java.lang.Long>	endPosInCMTForDevice
protected java.io.File	file
protected boolean	hasChunkMetadataInDisk
protected static byte[]	MAGIC_STRING_BYTES
protected long	maxMetadataSize
protected TsFileOutput	out
protected int	pathCount
protected LocalTsFileOutput	tempOutput
static byte	VERSION_NUMBER_BYTE

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	TsFileIOWriter()	empty construct function.
	TsFileIOWriter(java.io.File file)	for writing a new tsfile.
	TsFileIOWriter(java.io.File file, boolean enableMemoryControl, long maxMetadataSize)	for write with memory control
	TsFileIOWriter(TsFileOutput output)	for writing a new tsfile.
	TsFileIOWriter(TsFileOutput output, boolean test)	for test only

Method Summary

Modifier and Type	Method	Description
boolean	canWrite()
int	checkMetadataSizeAndMayFlush()	Check if the size of chunk metadata in memory is greater than the given threshold.
void	close()	close the outputStream or file channel without writing FileMetadata.
void	endChunkGroup()	end chunk and write some log.
void	endCurrentChunk()	end chunk and write some log.
void	endFile()	write TSFileMetaData to output stream and close it.
void	filterChunks(java.util.Map<Path,java.util.List<java.lang.Long>> chunkStartTimes)	Remove such ChunkMetadata that its startTime is not in chunkStartTimes
void	flush()
java.util.List<ChunkGroupMetadata>	getChunkGroupMetadataList()
java.util.List<ChunkMetadata>	getChunkMetadataListOfCurrentDeviceInMemory()	This method should be called before flushing chunk group metadata list, otherwise, it will return null.
java.lang.String	getCurrentChunkGroupDeviceId()
java.util.Map<java.lang.String,java.util.List<ChunkMetadata>>	getDeviceChunkMetadataMap()
java.util.Map<java.lang.String,java.util.List<TimeseriesMetadata>>	getDeviceTimeseriesMetadataMap()	this function is for Upgrade Tool and Split Tool.
java.io.File	getFile()
TsFileOutput	getIOWriterOut()	this function is only for Test.
long	getMaxPlanIndex()
long	getMinPlanIndex()
long	getPos()	get the length of normal OutputStream.
boolean	isWritingChunkGroup()	For TsFileReWriteTool / UpgradeTool.
void	mark()
void	reset()
void	setFile(java.io.File file)
void	setMaxPlanIndex(long maxPlanIndex)
void	setMinPlanIndex(long minPlanIndex)
protected int	sortAndFlushChunkMetadata()	Sort the chunk metadata by the lexicographical order and the start time of the chunk, then flush them to a temp file.
int	startChunkGroup(java.lang.String deviceId)
protected void	startFile()
void	startFlushChunk(java.lang.String measurementId, CompressionType compressionCodecName, TSDataType tsDataType, TSEncoding encodingType, Statistics<? extends java.io.Serializable> statistics, int dataSize, int numOfPages, int mask)	start a ChunkMetaData.
void	truncate(long offset)
void	writeBytesToStream(PublicBAOS bytes)	Writes given bytes to output stream.
void	writeChunk(Chunk chunk)
void	writeChunk(Chunk chunk, ChunkMetadata chunkMetadata)	Write a whole chunk in another file into this file.
void	writeEmptyValueChunk(java.lang.String measurementId, CompressionType compressionType, TSDataType tsDataType, TSEncoding encodingType, Statistics<? extends java.io.Serializable> statistics)	Write an empty value chunk into file directly.
void	writePlanIndices()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

MAGIC_STRING_BYTES

protected static final byte[] MAGIC_STRING_BYTES

VERSION_NUMBER_BYTE

public static final byte VERSION_NUMBER_BYTE

config

protected static final TSFileConfig config

out

protected TsFileOutput out

canWrite

protected boolean canWrite

file

protected java.io.File file

currentChunkMetadata

protected ChunkMetadata currentChunkMetadata

chunkMetadataList

protected java.util.List<ChunkMetadata> chunkMetadataList

chunkGroupMetadataList

protected java.util.List<ChunkGroupMetadata> chunkGroupMetadataList

maxMetadataSize

protected long maxMetadataSize

currentChunkMetadataSize

protected long currentChunkMetadataSize

chunkMetadataTempFile

protected java.io.File chunkMetadataTempFile

tempOutput

protected LocalTsFileOutput tempOutput

hasChunkMetadataInDisk

protected volatile boolean hasChunkMetadataInDisk

pathCount

protected int pathCount

enableMemoryControl

protected boolean enableMemoryControl

endPosInCMTForDevice

protected java.util.LinkedList<java.lang.Long> endPosInCMTForDevice

CHUNK_METADATA_TEMP_FILE_SUFFIX

public static final java.lang.String CHUNK_METADATA_TEMP_FILE_SUFFIX

See Also:

Constant Field Values

Constructor Detail

TsFileIOWriter

protected TsFileIOWriter()

empty construct function.

TsFileIOWriter

public TsFileIOWriter(java.io.File file)
               throws java.io.IOException

for writing a new tsfile.

Parameters:

file - be used to output written data

Throws:

java.io.IOException - if I/O error occurs

TsFileIOWriter

public TsFileIOWriter(TsFileOutput output)
               throws java.io.IOException

for writing a new tsfile.

Parameters:

output - be used to output written data

Throws:

java.io.IOException

TsFileIOWriter

public TsFileIOWriter(TsFileOutput output,
                      boolean test)

for test only

TsFileIOWriter

public TsFileIOWriter(java.io.File file,
                      boolean enableMemoryControl,
                      long maxMetadataSize)
               throws java.io.IOException

for write with memory control

Throws:

java.io.IOException

Method Detail

writeBytesToStream

public void writeBytesToStream(PublicBAOS bytes)
                        throws java.io.IOException

Writes given bytes to output stream. This method is called when total memory size exceeds the chunk group size threshold.

Parameters:

bytes - - data of several pages which has been packed

Throws:

java.io.IOException - if an I/O error occurs.

startFile

protected void startFile()
                  throws java.io.IOException

Throws:

java.io.IOException

startChunkGroup

public int startChunkGroup(java.lang.String deviceId)
                    throws java.io.IOException

Throws:

java.io.IOException

endChunkGroup

public void endChunkGroup()
                   throws java.io.IOException

end chunk and write some log. If there is no data in the chunk group, nothing will be flushed.

Throws:

java.io.IOException

isWritingChunkGroup

public boolean isWritingChunkGroup()

For TsFileReWriteTool / UpgradeTool. Use this method to determine if needs to start a ChunkGroup.

Returns:

isWritingChunkGroup

startFlushChunk

public void startFlushChunk(java.lang.String measurementId,
                            CompressionType compressionCodecName,
                            TSDataType tsDataType,
                            TSEncoding encodingType,
                            Statistics<? extends java.io.Serializable> statistics,
                            int dataSize,
                            int numOfPages,
                            int mask)
                     throws java.io.IOException

start a ChunkMetaData.

Parameters:

measurementId - - measurementId of this time series

compressionCodecName - - compression name of this time series

tsDataType - - data type

statistics - - Chunk statistics

dataSize - - the serialized size of all pages

mask - - 0x80 for time chunk, 0x40 for value chunk, 0x00 for common chunk

Throws:

java.io.IOException - if I/O error occurs

writeChunk

public void writeChunk(Chunk chunk,
                       ChunkMetadata chunkMetadata)
                throws java.io.IOException

Write a whole chunk in another file into this file. Providing fast merge for IoTDB.

Throws:

java.io.IOException

writeEmptyValueChunk

public void writeEmptyValueChunk(java.lang.String measurementId,
                                 CompressionType compressionType,
                                 TSDataType tsDataType,
                                 TSEncoding encodingType,
                                 Statistics<? extends java.io.Serializable> statistics)
                          throws java.io.IOException

Write an empty value chunk into file directly. Only used for aligned timeseries.

Throws:

java.io.IOException

writeChunk

public void writeChunk(Chunk chunk)
                throws java.io.IOException

Throws:

java.io.IOException

endCurrentChunk

public void endCurrentChunk()

end chunk and write some log.

endFile

public void endFile()
             throws java.io.IOException

write TSFileMetaData to output stream and close it.

Throws:

java.io.IOException - if I/O error occurs

getPos

public long getPos()
            throws java.io.IOException

get the length of normal OutputStream.

Returns:

- length of normal OutputStream

Throws:

java.io.IOException - if I/O error occurs

getDeviceChunkMetadataMap

public java.util.Map<java.lang.String,java.util.List<ChunkMetadata>> getDeviceChunkMetadataMap()

canWrite

public boolean canWrite()

mark

public void mark()
          throws java.io.IOException

Throws:

java.io.IOException

reset

public void reset()
           throws java.io.IOException

Throws:

java.io.IOException

close

public void close()
           throws java.io.IOException

close the outputStream or file channel without writing FileMetadata. This is just used for Testing.

Specified by:

close in interface java.lang.AutoCloseable

Throws:

java.io.IOException

getFile

public java.io.File getFile()

setFile

public void setFile(java.io.File file)

filterChunks

public void filterChunks(java.util.Map<Path,java.util.List<java.lang.Long>> chunkStartTimes)

Remove such ChunkMetadata that its startTime is not in chunkStartTimes

writePlanIndices

public void writePlanIndices()
                      throws java.io.IOException

Throws:

java.io.IOException

truncate

public void truncate(long offset)
              throws java.io.IOException

Throws:

java.io.IOException

getIOWriterOut

public TsFileOutput getIOWriterOut()

this function is only for Test.

Returns:

TsFileOutput

getChunkMetadataListOfCurrentDeviceInMemory

public java.util.List<ChunkMetadata> getChunkMetadataListOfCurrentDeviceInMemory()

This method should be called before flushing chunk group metadata list, otherwise, it will return null.

getDeviceTimeseriesMetadataMap

public java.util.Map<java.lang.String,java.util.List<TimeseriesMetadata>> getDeviceTimeseriesMetadataMap()

this function is for Upgrade Tool and Split Tool.

Returns:

DeviceTimeseriesMetadataMap

getMinPlanIndex

public long getMinPlanIndex()

setMinPlanIndex

public void setMinPlanIndex(long minPlanIndex)

getMaxPlanIndex

public long getMaxPlanIndex()

setMaxPlanIndex

public void setMaxPlanIndex(long maxPlanIndex)

checkMetadataSizeAndMayFlush

public int checkMetadataSizeAndMayFlush()
                                 throws java.io.IOException

Check if the size of chunk metadata in memory is greater than the given threshold. If so, the chunk metadata will be written to a temp files. Notice! If you are writing a aligned device in row, you should make sure all data of current writing device has been written before this method is called. For writing not aligned series or writing aligned series in column, you should make sure that all data of one series is written before you call this function.

Throws:

java.io.IOException

sortAndFlushChunkMetadata

protected int sortAndFlushChunkMetadata()
                                 throws java.io.IOException

Sort the chunk metadata by the lexicographical order and the start time of the chunk, then flush them to a temp file.

Throws:

java.io.IOException

getCurrentChunkGroupDeviceId

public java.lang.String getCurrentChunkGroupDeviceId()

getChunkGroupMetadataList

public java.util.List<ChunkGroupMetadata> getChunkGroupMetadataList()

flush

public void flush()
           throws java.io.IOException

Throws:

java.io.IOException