org.deeplearning4j.datasets.datavec

Class RecordReaderDataSetIterator

java.lang.Object

org.deeplearning4j.datasets.datavec.RecordReaderDataSetIterator

All Implemented Interfaces:

Serializable, Iterator<org.nd4j.linalg.dataset.DataSet>, org.nd4j.linalg.dataset.api.iterator.DataSetIterator

Direct Known Subclasses:

CifarDataSetIterator, LFWDataSetIterator

public class RecordReaderDataSetIterator
extends Object
implements org.nd4j.linalg.dataset.api.iterator.DataSetIterator

Record reader dataset iterator

Author:

Adam Gibson

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected int	batchNum
protected int	batchSize
protected org.datavec.api.io.WritableConverter	converter
protected int	labelIndex
protected int	labelIndexTo
protected org.nd4j.linalg.dataset.DataSet	last
protected int	maxNumBatches
protected int	numPossibleLabels
protected org.nd4j.linalg.dataset.api.DataSetPreProcessor	preProcessor
protected org.datavec.api.records.reader.RecordReader	recordReader
protected boolean	regression
protected Iterator<List<org.datavec.api.writable.Writable>>	sequenceIter
protected boolean	useCurrent

Constructor Summary

Constructors

Constructor and Description
RecordReaderDataSetIterator(org.datavec.api.records.reader.RecordReader recordReader, int batchSize)
RecordReaderDataSetIterator(org.datavec.api.records.reader.RecordReader recordReader, int batchSize, int labelIndex, int numPossibleLabels) Main constructor for classification.
RecordReaderDataSetIterator(org.datavec.api.records.reader.RecordReader recordReader, int batchSize, int labelIndexFrom, int labelIndexTo, boolean regression) Main constructor for multi-label regression (i.e., regression with multiple outputs)
RecordReaderDataSetIterator(org.datavec.api.records.reader.RecordReader recordReader, int batchSize, int labelIndex, int numPossibleLabels, int maxNumBatches)
RecordReaderDataSetIterator(org.datavec.api.records.reader.RecordReader recordReader, org.datavec.api.io.WritableConverter converter, int batchSize)
RecordReaderDataSetIterator(org.datavec.api.records.reader.RecordReader recordReader, org.datavec.api.io.WritableConverter converter, int batchSize, int labelIndex, int numPossibleLabels)
RecordReaderDataSetIterator(org.datavec.api.records.reader.RecordReader recordReader, org.datavec.api.io.WritableConverter converter, int batchSize, int labelIndex, int numPossibleLabels, boolean regression)
RecordReaderDataSetIterator(org.datavec.api.records.reader.RecordReader recordReader, org.datavec.api.io.WritableConverter converter, int batchSize, int labelIndex, int numPossibleLabels, int maxNumBatches, boolean regression)
RecordReaderDataSetIterator(org.datavec.api.records.reader.RecordReader recordReader, org.datavec.api.io.WritableConverter converter, int batchSize, int labelIndexFrom, int labelIndexTo, int numPossibleLabels, int maxNumBatches, boolean regression) Main constructor

Method Summary

Modifier and Type	Method and Description
boolean	asyncSupported()
int	batch()
int	cursor()
List<String>	getLabels()
boolean	hasNext()
int	inputColumns()
org.nd4j.linalg.dataset.DataSet	loadFromMetaData(List<org.datavec.api.records.metadata.RecordMetaData> list) Load a multiple examples to a DataSet, using the provided RecordMetaData instances.
org.nd4j.linalg.dataset.DataSet	loadFromMetaData(org.datavec.api.records.metadata.RecordMetaData recordMetaData) Load a single example to a DataSet, using the provided RecordMetaData.
org.nd4j.linalg.dataset.DataSet	next()
org.nd4j.linalg.dataset.DataSet	next(int num)
int	numExamples()
void	remove()
void	reset()
boolean	resetSupported()
void	setCollectMetaData(boolean collectMetaData) When set to true: metadata for the current examples will be present in the returned DataSet.
void	setPreProcessor(org.nd4j.linalg.dataset.api.DataSetPreProcessor preProcessor)
int	totalExamples()
int	totalOutcomes()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

getPreProcessor

Methods inherited from interface java.util.Iterator

forEachRemaining

Field Detail

recordReader

protected org.datavec.api.records.reader.RecordReader recordReader

converter

protected org.datavec.api.io.WritableConverter converter

batchSize

protected int batchSize

maxNumBatches

protected int maxNumBatches

batchNum

protected int batchNum

labelIndex

protected int labelIndex

labelIndexTo

protected int labelIndexTo

numPossibleLabels

protected int numPossibleLabels

sequenceIter

protected Iterator<List<org.datavec.api.writable.Writable>> sequenceIter

last

protected org.nd4j.linalg.dataset.DataSet last

useCurrent

protected boolean useCurrent

regression

protected boolean regression

preProcessor

protected org.nd4j.linalg.dataset.api.DataSetPreProcessor preProcessor

Constructor Detail

RecordReaderDataSetIterator

public RecordReaderDataSetIterator(org.datavec.api.records.reader.RecordReader recordReader,
                                   org.datavec.api.io.WritableConverter converter,
                                   int batchSize)

RecordReaderDataSetIterator

public RecordReaderDataSetIterator(org.datavec.api.records.reader.RecordReader recordReader,
                                   int batchSize)

RecordReaderDataSetIterator

public RecordReaderDataSetIterator(org.datavec.api.records.reader.RecordReader recordReader,
                                   int batchSize,
                                   int labelIndex,
                                   int numPossibleLabels)

Main constructor for classification. This will convert the input class index (at position labelIndex, with integer values 0 to numPossibleLabels-1 inclusive) to the appropriate one-hot output/labels representation.

Parameters:

recordReader - RecordReader: provides the source of the data

batchSize - Batch size (number of examples) for the output DataSet objects

labelIndex - Index of the label Writable (usually an IntWritable), as obtained by recordReader.next()

numPossibleLabels - Number of classes (possible labels) for classification

RecordReaderDataSetIterator

public RecordReaderDataSetIterator(org.datavec.api.records.reader.RecordReader recordReader,
                                   org.datavec.api.io.WritableConverter converter,
                                   int batchSize,
                                   int labelIndex,
                                   int numPossibleLabels,
                                   boolean regression)

RecordReaderDataSetIterator

public RecordReaderDataSetIterator(org.datavec.api.records.reader.RecordReader recordReader,
                                   org.datavec.api.io.WritableConverter converter,
                                   int batchSize,
                                   int labelIndex,
                                   int numPossibleLabels)

RecordReaderDataSetIterator

public RecordReaderDataSetIterator(org.datavec.api.records.reader.RecordReader recordReader,
                                   int batchSize,
                                   int labelIndex,
                                   int numPossibleLabels,
                                   int maxNumBatches)

RecordReaderDataSetIterator

public RecordReaderDataSetIterator(org.datavec.api.records.reader.RecordReader recordReader,
                                   int batchSize,
                                   int labelIndexFrom,
                                   int labelIndexTo,
                                   boolean regression)

Main constructor for multi-label regression (i.e., regression with multiple outputs)

Parameters:

recordReader - RecordReader to get data from

labelIndexFrom - Index of the first regression target

labelIndexTo - Index of the last regression target, inclusive

batchSize - Minibatch size

regression - Require regression = true. Mainly included to avoid clashing with other constructors previously defined :/

RecordReaderDataSetIterator

public RecordReaderDataSetIterator(org.datavec.api.records.reader.RecordReader recordReader,
                                   org.datavec.api.io.WritableConverter converter,
                                   int batchSize,
                                   int labelIndex,
                                   int numPossibleLabels,
                                   int maxNumBatches,
                                   boolean regression)

RecordReaderDataSetIterator

public RecordReaderDataSetIterator(org.datavec.api.records.reader.RecordReader recordReader,
                                   org.datavec.api.io.WritableConverter converter,
                                   int batchSize,
                                   int labelIndexFrom,
                                   int labelIndexTo,
                                   int numPossibleLabels,
                                   int maxNumBatches,
                                   boolean regression)

Main constructor

Parameters:

recordReader - the recordreader to use

converter - the batch size

maxNumBatches - Maximum number of batches to return

labelIndexFrom - the index of the label (for classification), or the first index of the labels for multi-output regression

labelIndexTo - only used if regression == true. The last index _inclusive_ of the multi-output regression

numPossibleLabels - the number of possible labels for classification. Not used if regression == true

regression - if true: regression. If false: classification (assume labelIndexFrom is the class it belongs to)

Method Detail

setCollectMetaData

public void setCollectMetaData(boolean collectMetaData)

When set to true: metadata for the current examples will be present in the returned DataSet. Disabled by default.

Parameters:

collectMetaData - Whether to collect metadata or not

next

public org.nd4j.linalg.dataset.DataSet next(int num)

Specified by:

next in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

totalExamples

public int totalExamples()

Specified by:

totalExamples in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

inputColumns

public int inputColumns()

Specified by:

inputColumns in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

totalOutcomes

public int totalOutcomes()

Specified by:

totalOutcomes in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

resetSupported

public boolean resetSupported()

Specified by:

resetSupported in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

asyncSupported

public boolean asyncSupported()

Specified by:

asyncSupported in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

reset

public void reset()

Specified by:

reset in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

batch

public int batch()

Specified by:

batch in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

cursor

public int cursor()

Specified by:

cursor in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

numExamples

public int numExamples()

Specified by:

numExamples in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

setPreProcessor

public void setPreProcessor(org.nd4j.linalg.dataset.api.DataSetPreProcessor preProcessor)

Specified by:

setPreProcessor in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

hasNext

public boolean hasNext()

Specified by:

hasNext in interface Iterator<org.nd4j.linalg.dataset.DataSet>

next

public org.nd4j.linalg.dataset.DataSet next()

Specified by:

next in interface Iterator<org.nd4j.linalg.dataset.DataSet>

remove

public void remove()

Specified by:

remove in interface Iterator<org.nd4j.linalg.dataset.DataSet>

getLabels

public List<String> getLabels()

Specified by:

getLabels in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

loadFromMetaData

public org.nd4j.linalg.dataset.DataSet loadFromMetaData(org.datavec.api.records.metadata.RecordMetaData recordMetaData)
                                                 throws IOException

Load a single example to a DataSet, using the provided RecordMetaData. Note that it is more efficient to load multiple instances at once, using loadFromMetaData(List)

Parameters:

recordMetaData - RecordMetaData to load from. Should have been produced by the given record reader

Returns:

DataSet with the specified example

Throws:

IOException - If an error occurs during loading of the data

loadFromMetaData

public org.nd4j.linalg.dataset.DataSet loadFromMetaData(List<org.datavec.api.records.metadata.RecordMetaData> list)
                                                 throws IOException

Load a multiple examples to a DataSet, using the provided RecordMetaData instances.

Parameters:

list - List of RecordMetaData instances to load from. Should have been produced by the record reader provided to the RecordReaderDataSetIterator constructor

Returns:

DataSet with the specified examples

Throws:

IOException - If an error occurs during loading of the data