org.deeplearning4j.api.storage

Interface StatsStorage

All Superinterfaces:

StatsStorageRouter

public interface StatsStorage
extends StatsStorageRouter

A general-purpose stats storage mechanism, for storing stats information (mainly used for iteration listeners).

Key design ideas: (a) Two types of storable objects: i. Persistable objects, for once per session objects ("static info") and also for periodically reported data ("updates") ii. StorageMetaData objects, for (b) There are 4 types of things used to uniquely identify these Persistable objects:
i. SessionID: A unique identifier for a single session
ii. TypeID: A unique identifier for the listener or type of data
For example, we might have stats from 2 (or more) listeners with identical session and worker IDs
This is typically hard-coded, per listener class
iii. WorkerID: A unique identifier for workers, within a session
iv. Timestamp: time at which the record was created
For example, single machine training (with 1 listener) would have 1 session ID, 1 type ID, 1 worker ID, and multiple timestamps.
Distributed training multiple listeres could have 1 session ID, multiple type IDs, and multiple worker IDs, and multiple timestamps for each
A hyperparameter optimization job could have multiple session IDs on top of that.

Note that the StatsStorage interface extends StatsStorageRouter

Author:

Alex Black

Method Summary

Modifier and Type	Method and Description
void	close() Close any open resources (files, etc)
void	deregisterStatsStorageListener(StatsStorageListener listener) Remove the specified listener, if it is present.
List<Persistable>	getAllStaticInfos(String sessionID, String typeID) Get all static informations for the given session and type ID
List<Persistable>	getAllUpdatesAfter(String sessionID, String typeID, long timestamp) Get all updates for the given session ID (all worker IDs), that occur after (not including) the given timestamp.
List<Persistable>	getAllUpdatesAfter(String sessionID, String typeID, String workerID, long timestamp) Get all updates for the given session and worker ID, that occur after (not including) the given timestamp.
long[]	getAllUpdateTimes(String sessionID, String typeID, String workerID) List the times of all updates for the specified sessionID, typeID and workerID
Persistable	getLatestUpdate(String sessionID, String typeID, String workerID) Get the latest update record (i.e., update record with the largest timestamp value) for the specified session and worker IDs
List<Persistable>	getLatestUpdateAllWorkers(String sessionID, String typeID) Get the latest update for all workers, for the given session ID
List<StatsStorageListener>	getListeners() Get a list (shallow copy) of all listeners currently present
int	getNumUpdateRecordsFor(String sessionID) Return the number of update records for the given session ID (all workers)
int	getNumUpdateRecordsFor(String sessionID, String typeID, String workerID) Return the number of update records for the given session ID and worker ID
Persistable	getStaticInfo(String sessionID, String typeID, String workerID) Get the static info for the given session and worker IDs, or null if no such static info has been reported
StorageMetaData	getStorageMetaData(String sessionID, String typeID) Get the session metadata, if any has been registered via StatsStorageRouter.putStorageMetaData(StorageMetaData)
Persistable	getUpdate(String sessionID, String typeId, String workerID, long timestamp) Get the specified update (or null, if none exists for the given session/worker ids and timestamp)
List<Persistable>	getUpdates(String sessionID, String typeID, String workerID, long[] timestamps) Get updates for the specified times only
boolean	isClosed()
List<String>	listSessionIDs() Get a list of all sessions stored by this storage backend
List<String>	listTypeIDsForSession(String sessionID) Get the list of type IDs for the given session ID
List<String>	listWorkerIDsForSession(String sessionID) For a given session ID, list all of the known worker IDs
List<String>	listWorkerIDsForSessionAndType(String sessionID, String typeID) For a given session ID and type ID, list all of the known worker IDs
void	registerStatsStorageListener(StatsStorageListener listener) Add a new StatsStorageListener.
void	removeAllListeners() Remove all listeners from the StatsStorage instance
boolean	sessionExists(String sessionID) Check if the specified session ID exists or not

Methods inherited from interface org.deeplearning4j.api.storage.StatsStorageRouter

putStaticInfo, putStaticInfo, putStorageMetaData, putStorageMetaData, putUpdate, putUpdate

Method Detail

close

void close()
    throws IOException

Close any open resources (files, etc)

Throws:

IOException

isClosed

boolean isClosed()

Returns:

Whether the StatsStorage implementation has been closed or not

listSessionIDs

List<String> listSessionIDs()

Get a list of all sessions stored by this storage backend

sessionExists

boolean sessionExists(String sessionID)

Check if the specified session ID exists or not

Parameters:

sessionID - Session ID to check

Returns:

true if session exists, false otherwise

getStaticInfo

Persistable getStaticInfo(String sessionID,
                          String typeID,
                          String workerID)

Get the static info for the given session and worker IDs, or null if no such static info has been reported

Parameters:

sessionID - Session ID

workerID - worker ID

Returns:

Static info, or null if none has been reported

getAllStaticInfos

List<Persistable> getAllStaticInfos(String sessionID,
                                    String typeID)

Get all static informations for the given session and type ID

Parameters:

sessionID - Session ID to get static info for

typeID - Type ID to get static info for

Returns:

All static info instances matching both the session and type IDs

listTypeIDsForSession

List<String> listTypeIDsForSession(String sessionID)

Get the list of type IDs for the given session ID

Parameters:

sessionID - Session ID to query

Returns:

List of type IDs

listWorkerIDsForSession

List<String> listWorkerIDsForSession(String sessionID)

For a given session ID, list all of the known worker IDs

Parameters:

sessionID - Session ID

Returns:

List of worker IDs, or possibly null if session ID is unknown

listWorkerIDsForSessionAndType

List<String> listWorkerIDsForSessionAndType(String sessionID,
                                            String typeID)

For a given session ID and type ID, list all of the known worker IDs

Parameters:

sessionID - Session ID

typeID - Type ID

Returns:

List of worker IDs, or possibly null if session ID is unknown

getNumUpdateRecordsFor

int getNumUpdateRecordsFor(String sessionID)

Return the number of update records for the given session ID (all workers)

Parameters:

sessionID - Session ID

Returns:

number of update records

getNumUpdateRecordsFor

int getNumUpdateRecordsFor(String sessionID,
                           String typeID,
                           String workerID)

Return the number of update records for the given session ID and worker ID

Parameters:

sessionID - Session ID

workerID - Worker ID

Returns:

number of update records

getLatestUpdate

Persistable getLatestUpdate(String sessionID,
                            String typeID,
                            String workerID)

Get the latest update record (i.e., update record with the largest timestamp value) for the specified session and worker IDs

Parameters:

sessionID - session ID

workerID - worker ID

Returns:

UpdateRecord containing the session/worker IDs, timestamp and content for the most recent update

getUpdate

Persistable getUpdate(String sessionID,
                      String typeId,
                      String workerID,
                      long timestamp)

Get the specified update (or null, if none exists for the given session/worker ids and timestamp)

Parameters:

sessionID - Session ID

workerID - Worker ID

timestamp - Timestamp

Returns:

Update

getLatestUpdateAllWorkers

List<Persistable> getLatestUpdateAllWorkers(String sessionID,
                                            String typeID)

Get the latest update for all workers, for the given session ID

Parameters:

sessionID - Session ID

Returns:

List of updates for the given Session ID

getAllUpdatesAfter

List<Persistable> getAllUpdatesAfter(String sessionID,
                                     String typeID,
                                     String workerID,
                                     long timestamp)

Get all updates for the given session and worker ID, that occur after (not including) the given timestamp. Results should be sorted by time.

Parameters:

sessionID - Session ID

workerID - Worker Id

timestamp - Timestamp

Returns:

List of records occurring after the given timestamp

getAllUpdatesAfter

List<Persistable> getAllUpdatesAfter(String sessionID,
                                     String typeID,
                                     long timestamp)

Get all updates for the given session ID (all worker IDs), that occur after (not including) the given timestamp. Results should be sorted by time.

Parameters:

sessionID - Session ID

timestamp - Timestamp

Returns:

List of records occurring after the given timestamp

getAllUpdateTimes

long[] getAllUpdateTimes(String sessionID,
                         String typeID,
                         String workerID)

List the times of all updates for the specified sessionID, typeID and workerID

Parameters:

sessionID - Session ID to get update times for

typeID - Type ID to get update times for

workerID - Worker ID to get update times for

Returns:

Times of all updates

getUpdates

List<Persistable> getUpdates(String sessionID,
                             String typeID,
                             String workerID,
                             long[] timestamps)

Get updates for the specified times only

Parameters:

sessionID - Session ID to get update times for

typeID - Type ID to get update times for

workerID - Worker ID to get update times for

timestamps - Timestamps to get the updates for. Note that if one of the specified times does not exist, it will be ommitted from the returned results list.

Returns:

List of updates at the specified times

getStorageMetaData

StorageMetaData getStorageMetaData(String sessionID,
                                   String typeID)

Get the session metadata, if any has been registered via StatsStorageRouter.putStorageMetaData(StorageMetaData)

Parameters:

sessionID - Session ID to get metadat

Returns:

Session metadata, or null if none is available

registerStatsStorageListener

void registerStatsStorageListener(StatsStorageListener listener)

Add a new StatsStorageListener. The given listener will called whenever a state change occurs for the stats storage instance

Parameters:

listener - Listener to add

deregisterStatsStorageListener

void deregisterStatsStorageListener(StatsStorageListener listener)

Remove the specified listener, if it is present.

Parameters:

listener - Listener to remove

removeAllListeners

void removeAllListeners()

Remove all listeners from the StatsStorage instance

getListeners

List<StatsStorageListener> getListeners()

Get a list (shallow copy) of all listeners currently present

Returns:

List of listeners