Package org.apache.iotdb.tsfile.read

Class TsFileSequenceReader

  • All Implemented Interfaces:
    java.lang.AutoCloseable
    Direct Known Subclasses:
    TsFileRestorableReader, UnClosedTsFileReader
    public class TsFileSequenceReader
extends java.lang.Object
implements java.lang.AutoCloseable

    • Field Detail

      • file

        protected java.lang.String file

      • fileMetadataPos

        protected long fileMetadataPos

      • fileMetadataSize

        protected int fileMetadataSize

    • Constructor Detail

      • TsFileSequenceReader

        public TsFileSequenceReader​(java.lang.String file)
                     throws java.io.IOException
        Create a file reader of the given file. The reader will read the tail of the file to get the file metadata size.Then the reader will skip the first TSFileConfig.MAGIC_STRING.getBytes().length + TSFileConfig.NUMBER_VERSION.getBytes().length bytes of the file for preparing reading real data.
        Parameters:
        file - the data file
        Throws:
        java.io.IOException - If some I/O error occurs

      • TsFileSequenceReader

        public TsFileSequenceReader​(java.lang.String file,
                            boolean loadMetadataSize)
                     throws java.io.IOException
        construct function for TsFileSequenceReader.
        Parameters:
        file - -given file name
        loadMetadataSize - -whether load meta data size
        Throws:
        java.io.IOException

      • TsFileSequenceReader

        public TsFileSequenceReader​(java.lang.String file,
                            boolean loadMetadata,
                            boolean cacheDeviceMetadata)
                     throws java.io.IOException
        Throws:
        java.io.IOException

      • TsFileSequenceReader

        public TsFileSequenceReader​(TsFileInput input)
                     throws java.io.IOException
        Create a file reader of the given file. The reader will read the tail of the file to get the file metadata size.Then the reader will skip the first TSFileConfig.MAGIC_STRING.getBytes().length + TSFileConfig.NUMBER_VERSION.getBytes().length bytes of the file for preparing reading real data.
        Parameters:
        input - given input
        Throws:
        java.io.IOException

      • TsFileSequenceReader

        public TsFileSequenceReader​(TsFileInput input,
                            boolean loadMetadataSize)
                     throws java.io.IOException
        construct function for TsFileSequenceReader.
        Parameters:
        input - -given input
        loadMetadataSize - -load meta data size
        Throws:
        java.io.IOException

      • TsFileSequenceReader

        public TsFileSequenceReader​(TsFileInput input,
                            long fileMetadataPos,
                            int fileMetadataSize)
        construct function for TsFileSequenceReader.
        Parameters:
        input - the input of a tsfile. The current position should be a marker and then a chunk Header, rather than the magic number
        fileMetadataPos - the position of the file metadata in the TsFileInput from the beginning of the input to the current position
        fileMetadataSize - the byte size of the file metadata in the input

    • Method Detail

      • loadMetadataSize

        public void loadMetadataSize()
                      throws java.io.IOException
        Throws:
        java.io.IOException

      • getFileMetadataPos

        public long getFileMetadataPos()

      • getTsFileMetadataSize

        public int getTsFileMetadataSize()

      • getFileMetadataSize

        public long getFileMetadataSize()
                         throws java.io.IOException
        Return the tsfile meta data size of this tsfile.
        Throws:
        java.io.IOException

      • getAllMetadataSize

        public long getAllMetadataSize()
                        throws java.io.IOException
        Return the whole meta data size of this tsfile, including ChunkMetadata, TimeseriesMetadata and etc.
        Throws:
        java.io.IOException

      • readTailMagic

        public java.lang.String readTailMagic()
                               throws java.io.IOException
        this function does not modify the position of the file reader.
        Throws:
        java.io.IOException

      • isComplete

        public boolean isComplete()
                   throws java.io.IOException
        whether the file is a complete TsFile: only if the head magic and tail magic string exists.
        Throws:
        java.io.IOException

      • readHeadMagic

        public java.lang.String readHeadMagic()
                               throws java.io.IOException
        this function does not modify the position of the file reader.
        Throws:
        java.io.IOException

      • readVersionNumber

        public byte readVersionNumber()
                       throws java.io.IOException
        this function reads version number and checks compatibility of TsFile.
        Throws:
        java.io.IOException

      • readFileMetadata

        public TsFileMetadata readFileMetadata()
                                throws java.io.IOException
        this function does not modify the position of the file reader.
        Throws:
        java.io.IOException - io error

      • readBloomFilter

        public BloomFilter readBloomFilter()
                            throws java.io.IOException
        this function does not modify the position of the file reader.
        Throws:
        java.io.IOException - io error

      • readDeviceMetadata

        public java.util.Map<java.lang.String,​TimeseriesMetadata> readDeviceMetadata​(java.lang.String device)
                                                                            throws java.io.IOException
        this function reads measurements and TimeseriesMetaDatas in given device Thread Safe
        Parameters:
        device - name
        Returns:
        the map measurementId -> TimeseriesMetaData in one device
        Throws:
        java.io.IOException - io error

      • readTimeseriesMetadata

        public TimeseriesMetadata readTimeseriesMetadata​(java.lang.String device,
                                                 java.lang.String measurement,
                                                 boolean ignoreNotExists)
                                          throws java.io.IOException
        Throws:
        java.io.IOException

      • readITimeseriesMetadata

        public ITimeSeriesMetadata readITimeseriesMetadata​(Path path,
                                                   boolean ignoreNotExists)
                                            throws java.io.IOException
        Throws:
        java.io.IOException

      • readTimeseriesMetadata

        public java.util.List<TimeseriesMetadata> readTimeseriesMetadata​(java.lang.String device,
                                                                 java.lang.String measurement,
                                                                 java.util.Set<java.lang.String> allSensors)
                                                          throws java.io.IOException
        Throws:
        java.io.IOException

      • readITimeseriesMetadata

        public java.util.List<ITimeSeriesMetadata> readITimeseriesMetadata​(java.lang.String device,
                                                                   java.util.Set<java.lang.String> measurements)
                                                            throws java.io.IOException
        Throws:
        java.io.IOException

      • binarySearchInTimeseriesMetadataList

        protected int binarySearchInTimeseriesMetadataList​(java.util.List<TimeseriesMetadata> timeseriesMetadataList,
                                                   java.lang.String key)

      • getAllDevices

        public java.util.List<java.lang.String> getAllDevices()
                                               throws java.io.IOException
        Throws:
        java.io.IOException

      • getAllDevicesIteratorWithIsAligned

        public TsFileDeviceIterator getAllDevicesIteratorWithIsAligned()
                                                        throws java.io.IOException
        Returns:
        an iterator of "device, isAligned" list, in which names of devices are ordered in dictionary order, and isAligned represents whether the device is aligned. Only read devices on one device leaf node each time to save memory.
        Throws:
        java.io.IOException

      • getDevicesAndEntriesOfOneLeafNode

        public void getDevicesAndEntriesOfOneLeafNode​(java.lang.Long startOffset,
                                              java.lang.Long endOffset,
                                              java.util.Queue<Pair<java.lang.String,​long[]>> measurementNodeOffsetQueue)
                                       throws java.io.IOException
        Get devices and first measurement node offset.
        Parameters:
        startOffset - start offset of device leaf node
        endOffset - end offset of device leaf node
        measurementNodeOffsetQueue - device -> first measurement node offset
        Throws:
        java.io.IOException

      • readChunkMetadataInDevice

        public java.util.Map<java.lang.String,​java.util.List<ChunkMetadata>> readChunkMetadataInDevice​(java.lang.String device)
                                                                                              throws java.io.IOException
        read all ChunkMetaDatas of given device
        Parameters:
        device - name
        Returns:
        measurement -> ChunkMetadata list
        Throws:
        java.io.IOException - io error

      • getAllPaths

        public java.util.List<Path> getAllPaths()
                                 throws java.io.IOException
        this function return all timeseries names
        Returns:
        list of Paths
        Throws:
        java.io.IOException - io error

      • getPathsIterator

        public java.util.Iterator<java.util.List<Path>> getPathsIterator()
                                                          throws java.io.IOException
        Returns:
        an iterator of timeseries list, in which names of timeseries are ordered in dictionary order
        Throws:
        java.io.IOException - io error

      • isAlignedDevice

        public boolean isAlignedDevice​(MetadataIndexNode measurementNode)
        Check whether the deivce is aligned or not.
        Parameters:
        measurementNode - the next measurement layer node of specific device node

      • getTimeseriesMetadataOffsetByDevice

        public java.util.Map<java.lang.String,​Pair<java.util.List<IChunkMetadata>,​Pair<java.lang.Long,​java.lang.Long>>> getTimeseriesMetadataOffsetByDevice​(MetadataIndexNode measurementNode,
                                                                                                                                                                      java.util.Set<java.lang.String> excludedMeasurementIds,
                                                                                                                                                                      boolean needChunkMetadata)
                                                                                                                                                               throws java.io.IOException
        Get the measurements of current device by its first measurement node. Also get the chunk metadata list and timeseries metadata offset.
        Parameters:
        measurementNode - first measurement node of the device
        excludedMeasurementIds - do not deserialize chunk metadatas whose measurementId is in the set. Notice: It only takes effect when the needChunkMetadata parameter is true.
        needChunkMetadata - need to deserialize chunk metadatas or not
        Returns:
        measurement -> chunk metadata list -> timeseries metadata
        Throws:
        java.io.IOException

      • getChunkMetadataListByTimeseriesMetadataOffset

        public java.util.List<IChunkMetadata> getChunkMetadataListByTimeseriesMetadataOffset​(long startOffset,
                                                                                     long endOffset)
                                                                              throws java.io.IOException
        Get chunk metadata list by the start offset and end offset of the timeseries metadata.
        Parameters:
        startOffset - the start offset of timeseries metadata
        endOffset - the end offset of timeseries metadata
        Throws:
        java.io.IOException

      • getDeviceTimeseriesMetadata

        public void getDeviceTimeseriesMetadata​(java.util.List<TimeseriesMetadata> timeseriesMetadataList,
                                        MetadataIndexNode measurementNode,
                                        java.util.Set<java.lang.String> excludedMeasurementIds,
                                        boolean needChunkMetadata)
                                 throws java.io.IOException
        Get timeseries metadata under the measurementNode and put them into timeseriesMetadataList. Skip timeseries whose measurementId is in the excludedMeasurementIds.
        Parameters:
        measurementNode - next layer measurement node of specific device leaf node
        excludedMeasurementIds - skip timeseries whose measurementId is in the set
        Throws:
        java.io.IOException

      • getAllTimeseriesMetadata

        public java.util.Map<java.lang.String,​java.util.List<TimeseriesMetadata>> getAllTimeseriesMetadata​(boolean needChunkMetadata)
                                                                                                  throws java.io.IOException
        Throws:
        java.io.IOException

      • getMetadataAndEndOffset

        protected Pair<MetadataIndexEntry,​java.lang.Long> getMetadataAndEndOffset​(MetadataIndexNode metadataIndex,
                                                                                java.lang.String name,
                                                                                boolean isDeviceLevel,
                                                                                boolean exactSearch)
                                                                         throws java.io.IOException
        Get target MetadataIndexEntry and its end offset
        Parameters:
        metadataIndex - given MetadataIndexNode
        name - target device / measurement name
        isDeviceLevel - whether target MetadataIndexNode is device level
        exactSearch - whether is in exact search mode, return null when there is no entry with name; or else return the nearest MetadataIndexEntry before it (for deeper search)
        Returns:
        target MetadataIndexEntry, endOffset pair
        Throws:
        java.io.IOException

      • readChunkGroupHeader

        public ChunkGroupHeader readChunkGroupHeader()
                                      throws java.io.IOException
        read data from current position of the input, and deserialize it to a CHUNK_GROUP_FOOTER.
        This method is not threadsafe.
        Returns:
        a CHUNK_GROUP_FOOTER
        Throws:
        java.io.IOException - io error

      • readChunkGroupHeader

        public ChunkGroupHeader readChunkGroupHeader​(long position,
                                             boolean markerRead)
                                      throws java.io.IOException
        read data from current position of the input, and deserialize it to a CHUNK_GROUP_FOOTER.
        Parameters:
        position - the offset of the chunk group footer in the file
        markerRead - true if the offset does not contains the marker , otherwise false
        Returns:
        a CHUNK_GROUP_FOOTER
        Throws:
        java.io.IOException - io error

      • readPlanIndex

        public void readPlanIndex()
                   throws java.io.IOException
        Throws:
        java.io.IOException

      • readChunkHeader

        public ChunkHeader readChunkHeader​(byte chunkType)
                            throws java.io.IOException
        read data from current position of the input, and deserialize it to a CHUNK_HEADER.
        This method is not threadsafe.
        Returns:
        a CHUNK_HEADER
        Throws:
        java.io.IOException - io error

      • readChunk

        public java.nio.ByteBuffer readChunk​(long position,
                                     int dataSize)
                              throws java.io.IOException
        notice, this function will modify channel's position.
        Parameters:
        dataSize - the size of chunkdata
        position - the offset of the chunk data
        Returns:
        the pages of this chunk
        Throws:
        java.io.IOException

      • readMemChunk

        public Chunk readMemChunk​(ChunkMetadata metaData)
                   throws java.io.IOException
        read memory chunk.
        Parameters:
        metaData - -given chunk meta data
        Returns:
        -chunk
        Throws:
        java.io.IOException

      • readMemChunk

        public Chunk readMemChunk​(CachedChunkLoaderImpl.ChunkCacheKey chunkCacheKey)
                   throws java.io.IOException
        read memory chunk.
        Parameters:
        chunkCacheKey - given key of chunk LRUCache
        Returns:
        chunk
        Throws:
        java.io.IOException

      • readTimeseriesCompressionTypeAndEncoding

        public Pair<CompressionType,​TSEncoding> readTimeseriesCompressionTypeAndEncoding​(TimeseriesMetadata timeseriesMetadata)
                                                                                throws java.io.IOException
        read the CompressionType and TSEncoding of a timeseries. This method will skip the measurement id, and data type. This method will change the position of this reader.
        Parameters:
        timeseriesMetadata - timeseries' metadata
        Returns:
        a pair of CompressionType and TSEncoding of given timeseries.
        Throws:
        java.io.IOException

      • getMeasurementSchema

        public MeasurementSchema getMeasurementSchema​(java.util.List<IChunkMetadata> chunkMetadataList)
                                       throws java.io.IOException
        Get measurement schema by chunkMetadatas.
        Throws:
        java.io.IOException

      • readPageHeader

        public PageHeader readPageHeader​(TSDataType type,
                                 boolean hasStatistic)
                          throws java.io.IOException
        not thread safe.
        Parameters:
        type - given tsfile data type
        Throws:
        java.io.IOException

      • position

        public long position()
              throws java.io.IOException
        Throws:
        java.io.IOException

      • position

        public void position​(long offset)
              throws java.io.IOException
        Throws:
        java.io.IOException

      • skipPageData

        public void skipPageData​(PageHeader header)
                  throws java.io.IOException
        Throws:
        java.io.IOException

      • readCompressedPage

        public java.nio.ByteBuffer readCompressedPage​(PageHeader header)
                                       throws java.io.IOException
        Throws:
        java.io.IOException

      • readPage

        public java.nio.ByteBuffer readPage​(PageHeader header,
                                    CompressionType type)
                             throws java.io.IOException
        Throws:
        java.io.IOException

      • readMarker

        public byte readMarker()
                throws java.io.IOException
        read one byte from the input.
        this method is not thread safe
        Throws:
        java.io.IOException

      • close

        public void close()
           throws java.io.IOException
        Specified by:
        close in interface java.lang.AutoCloseable
        Throws:
        java.io.IOException

      • getFileName

        public java.lang.String getFileName()

      • fileSize

        public long fileSize()
              throws java.io.IOException
        Throws:
        java.io.IOException

      • readData

        protected java.nio.ByteBuffer readData​(long position,
                                       int totalSize)
                                throws java.io.IOException
        read data from tsFileInput, from the current position (if position = -1), or the given position.
        if position = -1, the tsFileInput's position will be changed to the current position + real data size that been read. Other wise, the tsFileInput's position is not changed.
        Parameters:
        position - the start position of data in the tsFileInput, or the current position if position = -1
        totalSize - the size of data that want to read
        Returns:
        data that been read.
        Throws:
        java.io.IOException

      • readData

        protected java.nio.ByteBuffer readData​(long start,
                                       long end)
                                throws java.io.IOException
        read data from tsFileInput, from the current position (if position = -1), or the given position.
        Parameters:
        start - the start position of data in the tsFileInput, or the current position if position = -1
        end - the end position of data that want to read
        Returns:
        data that been read.
        Throws:
        java.io.IOException

      • readRaw

        public int readRaw​(long position,
                   int length,
                   java.nio.ByteBuffer target)
            throws java.io.IOException
        notice, the target bytebuffer are not flipped.
        Throws:
        java.io.IOException

      • selfCheck

        public long selfCheck​(java.util.Map<Path,​IMeasurementSchema> newSchema,
                      java.util.List<ChunkGroupMetadata> chunkGroupMetadataList,
                      boolean fastFinish)
               throws java.io.IOException
        Self Check the file and return the position before where the data is safe.
        Parameters:
        newSchema - the schema on each time series in the file
        chunkGroupMetadataList - ChunkGroupMetadata List
        fastFinish - if true and the file is complete, then newSchema and chunkGroupMetadataList parameter will be not modified.
        Returns:
        the position of the file that is fine. All data after the position in the file should be truncated.
        Throws:
        java.io.IOException

      • selfCheckWithInfo

        public long selfCheckWithInfo​(java.lang.String filename,
                              boolean fastFinish,
                              java.util.Map<java.lang.Long,​Pair<Path,​TimeseriesMetadata>> timeseriesMetadataMap)
                       throws java.io.IOException,
                              TsFileStatisticsMistakesException
        Self Check the file and return whether the file is safe.
        Parameters:
        filename - the path of file
        fastFinish - if true, the method will only check the format of head (Magic String TsFile, Version Number) and tail (Magic String TsFile) of TsFile.
        Returns:
        the status of TsFile
        Throws:
        java.io.IOException
        TsFileStatisticsMistakesException

      • checkChunkAndPagesStatistics

        public long checkChunkAndPagesStatistics​(IChunkMetadata chunkMetadata)
                                  throws java.io.IOException
        Throws:
        java.io.IOException

      • getChunkMetadataList

        public java.util.List<ChunkMetadata> getChunkMetadataList​(Path path,
                                                          boolean ignoreNotExists)
                                                   throws java.io.IOException
        get ChunkMetaDatas of given path, and throw exception if path not exists
        Parameters:
        path - timeseries path
        Returns:
        List of ChunkMetaData
        Throws:
        java.io.IOException

      • getIChunkMetadataList

        public java.util.List<IChunkMetadata> getIChunkMetadataList​(Path path)
                                                     throws java.io.IOException
        Throws:
        java.io.IOException

      • getChunkMetadataList

        public java.util.List<ChunkMetadata> getChunkMetadataList​(Path path)
                                                   throws java.io.IOException
        Throws:
        java.io.IOException

      • getAlignedChunkMetadata

        public java.util.List<AlignedChunkMetadata> getAlignedChunkMetadata​(java.lang.String device)
                                                             throws java.io.IOException
        Get AlignedChunkMetadata of sensors under one device
        Parameters:
        device - device name
        Throws:
        java.io.IOException

      • readChunkMetaDataList

        public java.util.List<ChunkMetadata> readChunkMetaDataList​(TimeseriesMetadata timeseriesMetaData)
                                                    throws java.io.IOException
        get ChunkMetaDatas in given TimeseriesMetaData
        Returns:
        List of ChunkMetaData
        Throws:
        java.io.IOException

      • getAllMeasurements

        public java.util.Map<java.lang.String,​TSDataType> getAllMeasurements()
                                                                    throws java.io.IOException
        get all measurements in this file
        Returns:
        measurement -> datatype
        Throws:
        java.io.IOException

      • getFullPathDataTypeMap

        public java.util.Map<java.lang.String,​TSDataType> getFullPathDataTypeMap()
                                                                        throws java.io.IOException
        get all types of measurements in this file
        Returns:
        full path -> datatype
        Throws:
        java.io.IOException

      • getDeviceMeasurementsMap

        public java.util.Map<java.lang.String,​java.util.List<java.lang.String>> getDeviceMeasurementsMap()
                                                                                                throws java.io.IOException
        Throws:
        java.io.IOException

      • getDeviceNameInRange

        public java.util.List<java.lang.String> getDeviceNameInRange​(long start,
                                                             long end)
                                                      throws java.io.IOException
        get device names which has valid chunks in [start, end)
        Parameters:
        start - start of the partition
        end - end of the partition
        Returns:
        device names in range
        Throws:
        java.io.IOException

      • getMetadataIndexNode

        public MetadataIndexNode getMetadataIndexNode​(long startOffset,
                                              long endOffset)
                                       throws java.io.IOException
        get metadata index node
        Parameters:
        startOffset - start read offset
        endOffset - end read offset
        Returns:
        MetadataIndexNode
        Throws:
        java.io.IOException

      • getMinPlanIndex

        public long getMinPlanIndex()

      • getMaxPlanIndex

        public long getMaxPlanIndex()

      • getMeasurementChunkMetadataListMapIterator

        public java.util.Iterator<java.util.Map<java.lang.String,​java.util.List<ChunkMetadata>>> getMeasurementChunkMetadataListMapIterator​(java.lang.String device)
                                                                                                                                   throws java.io.IOException
        Returns:
        An iterator of linked hashmaps ( measurement -> chunk metadata list ). When traversing the linked hashmap, you will get chunk metadata lists according to the lexicographic order of the measurements. The first measurement of the linked hashmap of each iteration is always larger than the last measurement of the linked hashmap of the previous iteration in lexicographic order.
        Throws:
        java.io.IOException

      • readMetadataIndexNode

        public MetadataIndexNode readMetadataIndexNode​(long start,
                                               long end)
                                        throws java.io.IOException
        Read MetadataIndexNode by start and end offset.
        Parameters:
        start - the start offset of the MetadataIndexNode
        end - the end offset of the MetadataIndexNode
        Returns:
        MetadataIndexNode
        Throws:
        java.io.IOException - IOException

      • equals

        public boolean equals​(java.lang.Object o)
        Overrides:
        equals in class java.lang.Object

      • hashCode

        public int hashCode()
        Overrides:
        hashCode in class java.lang.Object