Package org.h2.mvstore

Class FileStore<C extends Chunk<C>>

java.lang.Object
org.h2.mvstore.FileStore<C>
Direct Known Subclasses:
AppendOnlyMultiFileStore, RandomAccessStore
public abstract class FileStore<C extends Chunk<C>> extends Object
Class FileStore is a base class to allow for different store implementations. FileStore concept revolves around notion of a "chunk", which is a piece of data written into the store at once.

  • Field Details

    • META_ID_KEY

      public static final String META_ID_KEY
      The key for the entry within "layout" map, which contains id of "meta" map. Entry value (hex encoded) is usually equal to 1, unless it's a legacy (upgraded) database and id 1 has been taken already by another map.
      See Also:

    • readCount

      protected final AtomicLong readCount
      The number of read operations.

    • readBytes

      protected final AtomicLong readBytes
      The number of read bytes.

    • writeCount

      protected final AtomicLong writeCount
      The number of write operations.

    • writeBytes

      protected final AtomicLong writeBytes
      The number of written bytes.

    • lastChunk

      protected volatile C extends Chunk<C> lastChunk
      The newest chunk. If nothing was stored yet, this field is not set.

    • saveChunkLock

      protected final ReentrantLock saveChunkLock

    • storeHeader

      protected final HashMap<String,Object> storeHeader

    • recoveryMode

      protected final boolean recoveryMode

    • PIPE_LENGTH

      public static final int PIPE_LENGTH
      See Also:

  • Constructor Details

  • Method Details

    • open

      public abstract void open(String fileName, boolean readOnly, char[] encryptionKey)

    • open

      public abstract FileStore<C> open(String fileName, boolean readOnly)

    • init

      protected final void init(String fileName, boolean readOnly)

    • bind

      public final void bind(MVStore mvStore)

    • stop

      public final void stop(long allowedCompactionTime)

    • close

      public void close()

    • getMetaMapId

      public final int getMetaMapId(IntSupplier nextIdSupplier)

    • getLayoutMap

      public final Map<String,String> getLayoutMap()
      Get this store's layout map. This data is for informational purposes only. The data is subject to change in future versions.

      The data in this map should not be modified (changing system data may corrupt the store).

      The layout map contains the following entries: 

       chunk.{chunkId} = {chunk metadata}
 root.{mapId} = {root position}
      Returns:
      the metadata map

    • isRegularMap

      public final boolean isRegularMap(MVMap<?,?> map)

    • getRootPos

      public final long getRootPos(int mapId)
      Get "position" of the root page for the specified map
      Parameters:
      mapId - to get root position for
      Returns:
      opaque "position" value, that should be used to read the page

    • deregisterMapRoot

      public final boolean deregisterMapRoot(int mapId)
      Performs final stage of map removal - delete root location info from the layout map. Specified map is supposedly closed, is anonymous and has no outstanding usage by now.
      Parameters:
      mapId - to deregister
      Returns:
      true if root was removed, false if it is not there

    • hasChangesSince

      public final boolean hasChangesSince(long lastStoredVersion)
      Check whether there are any unsaved changes since specified version.
      Parameters:
      lastStoredVersion - version to take as a base for changes
      Returns:
      if there are any changes

    • lastChunkVersion

      public final long lastChunkVersion()

    • getMaxPageSize

      public final long getMaxPageSize()

    • getRetentionTime

      public final int getRetentionTime()

    • setRetentionTime

      public final void setRetentionTime(int ms)
      How long to retain old, persisted chunks, in milliseconds. Chunks that are older may be overwritten once they contain no live data.

      The default value is 45000 (45 seconds) when using the default file store. It is assumed that a file system and hard disk will flush all write buffers within this time. Using a lower value might be dangerous, unless the file system and hard disk flush the buffers earlier. To manually flush the buffers, use MVStore.getFile().force(true), however please note that according to various tests this does not always work as expected depending on the operating system and hardware.

      The retention time needs to be long enough to allow reading old chunks while traversing over the entries of a map.

      This setting is not persisted.

      Parameters:
      ms - how many milliseconds to retain old chunks (0 to overwrite them as early as possible)

    • shouldSaveNow

      public abstract boolean shouldSaveNow(int unsavedMemory, int autoCommitMemory)
      Decision about autocommit is delegated to store
      Parameters:
      unsavedMemory - amount of unsaved memory, so far
      autoCommitMemory - configured limit on amount of unsaved memory
      Returns:
      true if commit should happen now

    • getAutoCommitDelay

      public final int getAutoCommitDelay()
      Get the auto-commit delay.
      Returns:
      the delay in milliseconds, or 0 if auto-commit is disabled.

    • setAutoCommitDelay

      public final void setAutoCommitDelay(int millis)
      Set the maximum delay in milliseconds to auto-commit changes.

      To disable auto-commit, set the value to 0. In this case, changes are only committed when explicitly calling commit.

      The default is 1000, meaning all changes are committed after at most one second.

      Parameters:
      millis - the maximum delay

    • isKnownVersion

      public final boolean isKnownVersion(long version)
      Check whether all data can be read from this version. This requires that all chunks referenced by this version are still available (not overwritten).
      Parameters:
      version - the version
      Returns:
      true if all data can be read

    • rollbackTo

      public final void rollbackTo(long version)

    • initializeCommonHeaderAttributes

      protected final void initializeCommonHeaderAttributes(long time)

    • processCommonHeaderAttributes

      protected final void processCommonHeaderAttributes()

    • hasPersistentData

      protected final boolean hasPersistentData()

    • isIdle

      protected final boolean isIdle()

    • setLastChunk

      protected final void setLastChunk(C last)

    • registerDeadChunk

      protected final void registerDeadChunk(C chunk)

    • dropUnusedChunks

      public final void dropUnusedChunks()

    • writeFully

      protected abstract void writeFully(C chunk, long pos, ByteBuffer src)
      Write to the file.
      Parameters:
      chunk - to write
      pos - the write position
      src - the source buffer

    • readFully

      public abstract ByteBuffer readFully(C chunk, long pos, int len)
      Read data from the store.
      Parameters:
      chunk - that owns data to be read
      pos - the read "position"
      len - the number of bytes to read
      Returns:
      the byte buffer with data requested

    • readFully

      protected final ByteBuffer readFully(FileChannel file, long pos, int len)

    • allocateChunkSpace

      protected abstract void allocateChunkSpace(C chunk, WriteBuffer buff)
      Allocate logical space and assign position of the buffer within the store.
      Parameters:
      chunk - to allocate space for
      buff - to allocate space for

    • writeChunk

      protected abstract void writeChunk(C chunk, WriteBuffer buffer)
      Write buffer associated with chunk into store at chunk's allocated position
      Parameters:
      chunk - chunk to write
      buffer - to write

    • writeCleanShutdownMark

      protected abstract void writeCleanShutdownMark()
      Performs final preparation before store is closed normally

    • adjustStoreToLastChunk

      protected abstract void adjustStoreToLastChunk()
      Make persistent changes after lastChunk was reset

    • getStoreHeader

      public Map<String,Object> getStoreHeader()
      Get the store header. This data is for informational purposes only. The data is subject to change in future versions. The data should not be modified (doing so may corrupt the store).
      Returns:
      the store header

    • createChunk

      protected abstract C createChunk(int id)

    • createChunk

      public abstract C createChunk(String s)
      Build a Chunk from the given string.
      Parameters:
      s - the string
      Returns:
      the Chunk created

    • createChunk

      protected abstract C createChunk(Map<String,String> map)

    • writeCleanShutdown

      protected void writeCleanShutdown()

    • saveChunkMetadataChanges

      public void saveChunkMetadataChanges(C chunk)
      Store chunk's serialized metadata as an entry in a layout map. Key for this entry would be "chunk.<id>"
      Parameters:
      chunk - to save

    • freeChunkSpace

      protected abstract void freeChunkSpace(Iterable<C> chunks)
      Mark the space occupied by specified chunks as free.
      Parameters:
      chunks - chunks to be processed

    • validateFileLength

      protected abstract boolean validateFileLength(String msg)

    • compact

      public boolean compact(int targetFillRate, int write)
      Try to increase the fill rate by re-writing partially full chunks. Chunks with a low number of live items are re-written.

      If the current fill rate is higher than the target fill rate, nothing is done.

      Please note this method will not necessarily reduce the file size, as empty chunks are not overwritten.

      Only data of open maps can be moved. For maps that are not open, the old chunk is still referenced. Therefore, it is recommended to open all maps before calling this method.

      Parameters:
      targetFillRate - the minimum percentage of live entries
      write - the minimum number of bytes to write
      Returns:
      if any chunk was re-written

    • compactStore

      public void compactStore(long maxCompactTime)

    • compactStore

      protected abstract void compactStore(int thresholdFillRate, long maxCompactTime, int maxWriteSize, MVStore mvStore)
      Compact store file, that is, compact blocks that have a low fill rate, and move chunks next to each other. This will typically shrink the file. Changes are flushed to the file, and old chunks are overwritten.
      Parameters:
      thresholdFillRate - do not compact if store fill rate above this value (0-100)
      maxCompactTime - the maximum time in milliseconds to compact
      maxWriteSize - the maximum amount of data to be written as part of this call
      mvStore - that owns this FileStore

    • doHousekeeping

      protected abstract void doHousekeeping(MVStore mvStore) throws InterruptedException
      Throws:
      InterruptedException

    • start

      public MVMap<String,String> start()

    • initializeStoreHeader

      protected abstract void initializeStoreHeader(long time)

    • readStoreHeader

      protected abstract void readStoreHeader(boolean recoveryMode)

    • discoverChunk

      protected final C discoverChunk(long block)
      Discover a valid chunk, searching file backwards from the given block
      Parameters:
      block - to start search from (found chunk footer should be no further than block-1)
      Returns:
      valid chunk or null if none found

    • findLastChunkWithCompleteValidChunkSet

      protected final boolean findLastChunkWithCompleteValidChunkSet(Comparator<C> chunkComparator, Map<Long,C> validChunksByLocation, boolean afterFullScan)

    • getChunksFromLayoutMap

      protected Iterable<C> getChunksFromLayoutMap()

    • readChunkHeaderAndFooter

      protected final C readChunkHeaderAndFooter(long block, int expectedId)
      Read a chunk header and footer, and verify the stored data is consistent.
      Parameters:
      block - the block
      expectedId - of the chunk
      Returns:
      the chunk, or null if the header or footer don't match or are not consistent

    • readChunkHeaderOptionally

      protected final C readChunkHeaderOptionally(long block, int expectedId)

    • readChunkHeaderOptionally

      protected final C readChunkHeaderOptionally(long block)

    • readChunkFooter

      protected final C readChunkFooter(long block)
      Try to read a chunk footer.
      Parameters:
      block - the index of the next block after the chunk
      Returns:
      the chunk, or null if not successful

    • getWriteBuffer

      public WriteBuffer getWriteBuffer()
      Get a buffer for writing. This caller must synchronize on the store before calling the method and until after using the buffer.
      Returns:
      the buffer

    • releaseWriteBuffer

      public void releaseWriteBuffer(WriteBuffer buff)
      Release a buffer for writing. This caller must synchronize on the store before calling the method and until after using the buffer.
      Parameters:
      buff - the buffer than can be re-used

    • getCreationTime

      public long getCreationTime()
      The time the store was created, in milliseconds since 1970.
      Returns:
      creation time

    • getAutoCompactFillRate

      protected final int getAutoCompactFillRate()

    • sync

      public void sync()

    • getFillRate

      public abstract int getFillRate()

    • shrinkStoreIfPossible

      protected abstract void shrinkStoreIfPossible(int minPercent)
      Shrink store if possible, and if at least a given percentage can be saved.
      Parameters:
      minPercent - the minimum percentage to save

    • size

      public long size()
      Get the file size.
      Returns:
      the file size

    • setSize

      protected final void setSize(long size)

    • getWriteCount

      public long getWriteCount()
      Get the number of write operations since this store was opened. For file based stores, this is the number of file write operations.
      Returns:
      the number of write operations

    • getReadCount

      public long getReadCount()
      Get the number of read operations since this store was opened. For file based stores, this is the number of file read operations.
      Returns:
      the number of read operations

    • getReadBytes

      public long getReadBytes()
      Get the number of read bytes since this store was opened.
      Returns:
      the number of write operations

    • isReadOnly

      public boolean isReadOnly()

    • getDefaultRetentionTime

      public int getDefaultRetentionTime()
      Get the default retention time for this store in milliseconds.
      Returns:
      the retention time

    • clear

      public void clear()

    • getFileName

      public String getFileName()
      Get the file name.
      Returns:
      the file name

    • getMvStore

      protected final MVStore getMvStore()

    • markUsed

      protected abstract void markUsed(long pos, int length)
      Mark the space as in use.
      Parameters:
      pos - the position in bytes
      length - the number of bytes

    • backup

      public abstract void backup(ZipOutputStream out) throws IOException
      Throws:
      IOException

    • getChunks

      protected final ConcurrentMap<Integer,C> getChunks()

    • getRewriteCandidates

      protected Collection<C> getRewriteCandidates()

    • isSpaceReused

      public boolean isSpaceReused()

    • setReuseSpace

      public void setReuseSpace(boolean reuseSpace)

    • store

      protected final void store()

    • getChunksFillRate

      public int getChunksFillRate()
      Get the current fill rate (percentage of used space in the file). Unlike the fill rate of the store, here we only account for chunk data; the fill rate here is how much of the chunk data is live (still referenced). Young chunks are considered live.
      Returns:
      the fill rate, in percent (100 is completely full)

    • getCacheSize

      public int getCacheSize()
      Get the maximum cache size, in MB. Note that this does not include the page chunk references cache, which is 25% of the size of the page cache.
      Returns:
      the cache size

    • getCacheSizeUsed

      public int getCacheSizeUsed()
      Get the amount of memory used for caching, in MB. Note that this does not include the page chunk references cache, which is 25% of the size of the page cache.
      Returns:
      the amount of memory used for caching

    • setCacheSize

      public void setCacheSize(int mb)
      Set the read cache size in MB.
      Parameters:
      mb - the cache size in MB.

    • populateInfo

      public void populateInfo(BiConsumer<String,String> consumer)

    • getCacheHitRatio

      public int getCacheHitRatio()

    • getTocCacheHitRatio

      public int getTocCacheHitRatio()

    • rewriteChunks

      protected boolean rewriteChunks(int writeLimit, int targetFillRate)

    • executeFileStoreOperation

      public void executeFileStoreOperation(Runnable operation)

    • accountForRemovedPage

      public void accountForRemovedPage(long pos, long version, boolean pinned, int pageNo)
      Remove a page.
      Parameters:
      pos - the position of the page
      version - at which page was removed
      pinned - whether page is considered pinned
      pageNo - sequential page number within chunk