org.nd4j.imports.graphmapper.tf

Class TFGraphMapper

java.lang.Object

org.nd4j.imports.graphmapper.BaseGraphMapper<GraphDef,NodeDef,AttrValue,NodeDef>

org.nd4j.imports.graphmapper.tf.TFGraphMapper

All Implemented Interfaces:

GraphMapper<GraphDef,NodeDef,AttrValue,NodeDef>

public class TFGraphMapper
extends BaseGraphMapper<GraphDef,NodeDef,AttrValue,NodeDef>

Map tensorflow graph protos to the intermediate representation for samediff.

Author:

Adam Gibson

Field Summary

Fields

Modifier and Type	Field and Description
static String	SHAPE_KEY
static String	VALUE_ATTR_KEY

Method Summary

Modifier and Type	Method and Description
boolean	alreadySeen(NodeDef nodeDef)
static DataType	convertType(DataType tfType)
DataType	dataTypeForTensor(NodeDef tensorProto, int outNum)
void	dumpBinaryProtoAsText(File inputFile, File outputFile) Dump a binary proto file representation as a plain string in to the target text file
void	dumpBinaryProtoAsText(InputStream inputFile, File outputFile) Dump a binary proto file representation as a plain string in to the target text file
INDArray	getArrayFrom(NodeDef nodeDef, GraphDef graph)
Map<String,AttrValue>	getAttrMap(NodeDef nodeDef) Get the attribute map for given node
String	getAttrValueFromNode(NodeDef nodeDef, String key)
List<String>	getControlDependencies(NodeDef node) Get the list of control dependencies for the current node (or null if none exist)
String	getInputFromNode(NodeDef node, int index) Get the input node for the given node
static TFGraphMapper	getInstance() Singleton.
DifferentialFunction	getMappedOp(String name) Get the mapped op name for a given op relative to the type of node being mapped.
String	getName(NodeDef nodeDef) Get the name of the node
INDArray	getNDArrayFromTensor(String tensorName, NodeDef node, GraphDef graph)
org.nd4j.shade.protobuf.Message.Builder	getNewGraphBuilder() Returns a graph builder for initial definition and parsing.
List<NodeDef>	getNodeList(GraphDef graphDef)
String	getNodeName(String name) Map a tensorflow node name to the samediff equivalent for import
NodeDef	getNodeWithNameFromGraph(GraphDef graph, String name) Get the node from the graph
String	getOpType(NodeDef nodeDef)
long[]	getShape(NodeDef nodeDef)
long[]	getShapeFromAttr(AttrValue attr) Get the shape of the attribute value
long[]	getShapeFromAttribute(AttrValue attrValue)
long[]	getShapeFromTensor(NodeDef tensorProto) Get the shape for the given tensor type
String	getTargetMappingForOp(DifferentialFunction function, NodeDef node) Get the target mapping key (usually based on the node name) for the given function
boolean	hasShape(NodeDef nodeDef)
protected void	importCondition(String conditionName, NodeDef tfNode, ImportState<GraphDef,NodeDef> importState)
void	initFunctionFromProperties(DifferentialFunction on, Map<String,AttrValue> attributesForNode, NodeDef node, GraphDef graph) Calls initFunctionFromProperties(DifferentialFunction, Map, NodeDef, GraphDef) using DifferentialFunction.tensorflowName()
void	initFunctionFromProperties(String mappedTfName, DifferentialFunction on, Map<String,AttrValue> attributesForNode, NodeDef node, GraphDef graph) Init a function's attributes
boolean	isConstant(NodeDef nodeDef) Returns true if the given node is a constant
boolean	isControlDependency(String name)
boolean	isOpIgnoreException(NodeDef node) Returns true if this node is a special case (maybe because of name or other scenarios) that should override GraphMapper.opsToIgnore() in certain circumstances
boolean	isPlaceHolder(NodeDef nodeDef) Returns true if the given node is a place holder type (think a yet to be determined shape)_
boolean	isPlaceHolderNode(NodeDef node) Returns true if the given node is a place holder
boolean	isStringType(NodeDef tensorProto)
boolean	isVariableNode(NodeDef nodeDef)
void	mapNodeType(NodeDef tfNode, ImportState<GraphDef,NodeDef> importState, OpImportOverride<GraphDef,NodeDef,AttrValue> importOverride, OpImportFilter<GraphDef,NodeDef,AttrValue> opFilter) Map a node in to the import state covering the SameDiff instance
void	mapProperty(String name, DifferentialFunction on, NodeDef node, GraphDef graph, SameDiff sameDiff, Map<String,Map<String,PropertyMapping>> propertyMappingsForFunction)
INDArray	mapTensorProto(TensorProto tfTensor)
IfImportState	nodesForIf(NodeDef from, GraphDef graph) Returns the node for an if statement
int	numInputsFor(NodeDef nodeDef) Get the number of inputs for a node.
Set<String>	opsToIgnore() Ops to ignore for mapping
GraphDef	parseGraphFrom(byte[] inputStream) Parse a graph from an input stream
GraphDef	parseGraphFrom(InputStream inputStream) Parse a graph from an input stream
protected static void	setFloat16ValueFromInt(INDArray arr, int idx, int bytesAsPaddedInt)
boolean	shouldSkip(NodeDef opType)
String	translateToSameDiffName(String name, NodeDef node)
Map<String,NodeDef>	variablesForGraph(GraphDef graphDef) Get the variables for the given graph
String	varNameToOpName(String varName)
static int	varNameToOpOutputNumber(String varName)

Methods inherited from class org.nd4j.imports.graphmapper.BaseGraphMapper

importGraph, importGraph, importGraph, importGraph, importGraph, importGraph, initOutputVariables, mapProperties, nameIndexForGraph, nodesByName, opTypeForNode, readGraph, validateGraphStructure, validTensorDataType

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

VALUE_ATTR_KEY

public static final String VALUE_ATTR_KEY

See Also:

Constant Field Values

SHAPE_KEY

public static final String SHAPE_KEY

See Also:

Constant Field Values

Method Detail

getInstance

public static TFGraphMapper getInstance()

Singleton. Get the needed instance.

Returns:

dumpBinaryProtoAsText

public void dumpBinaryProtoAsText(InputStream inputFile,
                                  File outputFile)

Description copied from interface: GraphMapper

Dump a binary proto file representation as a plain string in to the target text file

isOpIgnoreException

public boolean isOpIgnoreException(NodeDef node)

Description copied from interface: GraphMapper

Returns true if this node is a special case (maybe because of name or other scenarios) that should override GraphMapper.opsToIgnore() in certain circumstances

Parameters:

node - the node to check

Returns:

true if this node is an exception false otherwise

getTargetMappingForOp

public String getTargetMappingForOp(DifferentialFunction function,
                                    NodeDef node)

Description copied from interface: GraphMapper

Get the target mapping key (usually based on the node name) for the given function

Parameters:

function - the function

node - the node to derive the target mapping from

Returns:

getNodeWithNameFromGraph

public NodeDef getNodeWithNameFromGraph(GraphDef graph,
                                        String name)

Description copied from interface: GraphMapper

Get the node from the graph

Parameters:

graph - the graph to get the node from

name - the name of the node to get from the graph

Returns:

mapProperty

public void mapProperty(String name,
                        DifferentialFunction on,
                        NodeDef node,
                        GraphDef graph,
                        SameDiff sameDiff,
                        Map<String,Map<String,PropertyMapping>> propertyMappingsForFunction)

isPlaceHolderNode

public boolean isPlaceHolderNode(NodeDef node)

Returns true if the given node is a place holder

Parameters:

node - the node to check

Returns:

true if the node is a place holder or not

dumpBinaryProtoAsText

public void dumpBinaryProtoAsText(File inputFile,
                                  File outputFile)

Dump a binary proto file representation as a plain string in to the target text file

getShapeFromAttr

public long[] getShapeFromAttr(AttrValue attr)

Description copied from interface: GraphMapper

Get the shape of the attribute value

Parameters:

attr - the attribute value

Returns:

the shape of the attribute if any or null

getAttrMap

public Map<String,AttrValue> getAttrMap(NodeDef nodeDef)

Description copied from interface: GraphMapper

Get the attribute map for given node

Parameters:

nodeDef - the node

Returns:

the attribute map for the attribute

getName

public String getName(NodeDef nodeDef)

Description copied from interface: GraphMapper

Get the name of the node

Parameters:

nodeDef - the node to get the name for

Returns:

alreadySeen

public boolean alreadySeen(NodeDef nodeDef)

Returns:

isVariableNode

public boolean isVariableNode(NodeDef nodeDef)

Returns:

shouldSkip

public boolean shouldSkip(NodeDef opType)

Returns:

hasShape

public boolean hasShape(NodeDef nodeDef)

Returns:

getShape

public long[] getShape(NodeDef nodeDef)

Returns:

getArrayFrom

public INDArray getArrayFrom(NodeDef nodeDef,
                             GraphDef graph)

Returns:

getOpType

public String getOpType(NodeDef nodeDef)

getNodeList

public List<NodeDef> getNodeList(GraphDef graphDef)

Parameters:

graphDef -

Returns:

getMappedOp

public DifferentialFunction getMappedOp(String name)

Description copied from interface: GraphMapper

Get the mapped op name for a given op relative to the type of node being mapped. The input name should be based on a tensorflow type or onnx type, not the nd4j name

Parameters:

name - the tensorflow or onnx name

Returns:

getNodeName

public String getNodeName(String name)

Map a tensorflow node name to the samediff equivalent for import

Parameters:

name - the name to change

Returns:

the input tensorflow name

isControlDependency

public boolean isControlDependency(String name)

variablesForGraph

public Map<String,NodeDef> variablesForGraph(GraphDef graphDef)

Description copied from interface: GraphMapper

Get the variables for the given graph

Parameters:

graphDef - the graph to get the variables for

Returns:

a map of variable name to tensor

translateToSameDiffName

public String translateToSameDiffName(String name,
                                      NodeDef node)

Returns:

varNameToOpName

public String varNameToOpName(String varName)

varNameToOpOutputNumber

public static int varNameToOpOutputNumber(String varName)

getNewGraphBuilder

public org.nd4j.shade.protobuf.Message.Builder getNewGraphBuilder()

Description copied from interface: GraphMapper

Returns a graph builder for initial definition and parsing.

Returns:

parseGraphFrom

public GraphDef parseGraphFrom(byte[] inputStream)
                        throws IOException

Description copied from interface: GraphMapper

Parse a graph from an input stream

Parameters:

inputStream - the input stream to load from

Returns:

Throws:

IOException

parseGraphFrom

public GraphDef parseGraphFrom(InputStream inputStream)
                        throws IOException

Description copied from interface: GraphMapper

Parse a graph from an input stream

Parameters:

inputStream - the input stream to load from

Returns:

Throws:

IOException

importCondition

protected void importCondition(String conditionName,
                               NodeDef tfNode,
                               ImportState<GraphDef,NodeDef> importState)

mapNodeType

public void mapNodeType(NodeDef tfNode,
                        ImportState<GraphDef,NodeDef> importState,
                        OpImportOverride<GraphDef,NodeDef,AttrValue> importOverride,
                        OpImportFilter<GraphDef,NodeDef,AttrValue> opFilter)

Description copied from interface: GraphMapper

Map a node in to the import state covering the SameDiff instance

Parameters:

tfNode - the node to map

importState - the current import state

opFilter - Optional filter for skipping operations

initFunctionFromProperties

public void initFunctionFromProperties(DifferentialFunction on,
                                       Map<String,AttrValue> attributesForNode,
                                       NodeDef node,
                                       GraphDef graph)

Calls initFunctionFromProperties(DifferentialFunction, Map, NodeDef, GraphDef) using DifferentialFunction.tensorflowName()

Parameters:

on - the function to use init on

attributesForNode - the attributes for the node

node -

graph -

initFunctionFromProperties

public void initFunctionFromProperties(String mappedTfName,
                                       DifferentialFunction on,
                                       Map<String,AttrValue> attributesForNode,
                                       NodeDef node,
                                       GraphDef graph)

Init a function's attributes

Parameters:

mappedTfName - the tensorflow name to pick (sometimes ops have multiple names

on - the function to map

attributesForNode - the attributes for the node

node -

graph -

dataTypeForTensor

public DataType dataTypeForTensor(NodeDef tensorProto,
                                  int outNum)

Returns:

convertType

public static DataType convertType(DataType tfType)

isStringType

public boolean isStringType(NodeDef tensorProto)

getAttrValueFromNode

public String getAttrValueFromNode(NodeDef nodeDef,
                                   String key)

Returns:

getShapeFromAttribute

public long[] getShapeFromAttribute(AttrValue attrValue)

Returns:

isPlaceHolder

public boolean isPlaceHolder(NodeDef nodeDef)

Description copied from interface: GraphMapper

Returns true if the given node is a place holder type (think a yet to be determined shape)_

Returns:

isConstant

public boolean isConstant(NodeDef nodeDef)

Description copied from interface: GraphMapper

Returns true if the given node is a constant

Returns:

getControlDependencies

public List<String> getControlDependencies(NodeDef node)

Description copied from interface: GraphMapper

Get the list of control dependencies for the current node (or null if none exist)

Parameters:

node - Node to get the control dependencies (if any) for

Returns:

getNDArrayFromTensor

public INDArray getNDArrayFromTensor(String tensorName,
                                     NodeDef node,
                                     GraphDef graph)

Returns:

mapTensorProto

public INDArray mapTensorProto(TensorProto tfTensor)

setFloat16ValueFromInt

protected static void setFloat16ValueFromInt(INDArray arr,
                                             int idx,
                                             int bytesAsPaddedInt)

getShapeFromTensor

public long[] getShapeFromTensor(NodeDef tensorProto)

Description copied from interface: GraphMapper

Get the shape for the given tensor type

Returns:

opsToIgnore

public Set<String> opsToIgnore()

Description copied from interface: GraphMapper

Ops to ignore for mapping

Returns:

getInputFromNode

public String getInputFromNode(NodeDef node,
                               int index)

Description copied from interface: GraphMapper

Get the input node for the given node

Parameters:

node - the node

index - hte index

Returns:

numInputsFor

public int numInputsFor(NodeDef nodeDef)

Description copied from interface: GraphMapper

Get the number of inputs for a node.

Parameters:

nodeDef - the node to get the number of inputs for

Returns:

nodesForIf

public IfImportState nodesForIf(NodeDef from,
                                GraphDef graph)

Returns the node for an if statement

Parameters:

from - the starting node (a merge node that represents a conditional)

graph - the graph to search

Returns:

an import state representing the nodes for each scope