Introduction

This document is intended for developers / designers participating in design of model-derived Java Binding for MD-SAL, or for developers interested in understanding issues with current design and initial input / reasoning for follow-up design of next version.

Important
 This is living document, which may be updated when new issues are raised, solutions proposed.

Assumed Knowledge

  • Good understanding of YANG language

  • Good understanding of Java language, language syntax, semantics and rules.

  • Good understanding of common design patterns and their application in Java.

Knowledge of other JVM-based languages such as Groovy or Scala, and understanding how these languages rendered their specific concepts is also beneficial.

What is Java Binding Specification

  • Binding specification defines mapping of YANG modeled data to respective Java Objects, structures and DTOs

  • has compile time and runtime aspects, compile time aspects are mostly visible to users of MD-SAL

  • Currently used specification is Binding Specification v1

Requirements for Binding Specification v2

Interfaces & classes provided and generated by Binding Specification v2:

  • SHOULD model YANG semantics correctly

  • MUST NOT prevent support of YANG 1.1. See Appendix 1: YANG 1.1 List of changes

  • Provide / enforce maximal possible constrains during compile-time

    • This prevents bugs at compile time, more allows for assumptions at runtime and performance optimizations

  • Isolation - Code generation MUST BE isolated. Presence of additional model should not change generation of original model.

  • SHOULD allow for reusing instances of objects at various subtrees

Known issues of Binding Specification 1

Issues preventing correct model generation

Grouping, data, typedef and identity namespaces collide

Related bugs

This bugs shared common root in Binding Specification v1 and that was caused by not accounting for identifier namespace differences between Java and YANG.

  • Java has 1 namespace, where unique identifier is is Package name with Class name

  • YANG has 6 separate namespaces and allows for same name to be used in this separate namespaces.

    Module and submodule namespace

    Each module and submodule name must be unique.

    Extension namespace

    All extensions names defined in module and its submodules must be unique.

    Typedef namespace

    All derived type names defined within a parent node or at the top level of the module or its submodules must be unique.

    Grouping namespace

    All grouping names defined within a parent node or at the top level of the module or its submodules must be unique.

    Data namespace

    All leafs, leaf-lists, lists, containers, choices, rpcs, notifications, and anyxmls defined (directly or through a uses statement) within a parent node or at the top level of the module must be unique.

    Identity namespace

    All identity names defined in a module and its submodules share the same identity identifier namespace.

Example valid model
module example {
    namespace "urn:example";

    identity example {}
    typedef example {type string;}
    grouping example {}
    container example {
        container example {
          leaf example {type example;}
        }
    }
}
Proposed solution

Use different packages names for identities, types, groupings and data tree items.

The format of package name is {gen-prefix}.{module-id}.{namespace-id}.{tree-id} where:

gen-prefix

Constant prefix for all generated code in order to not conflict with hand-written code. Value is org.opendaylight.mdsal.gen.v2

module-id

Module name translated to package identifier. It is shorter than namespace, requires less substitutions and still is unique identifier of module, which can not change over time.

namespace-id

One of YANG defined identifier namespaces:

  • ident - identity namespace, package for identities

  • type - type namespace, package for types

  • grp - grouping namespace, package for groupings

  • data - package for all instantiated data tree nodes

tree-id

Package identifier derived from schema-node-identifier in order to separate namespace on each level of data tree.

Example

  • If module name is example-network-topology unique identifier is example.network.topology

    • org.opendaylight.mdsal.gen.v2.urn.example.network.topology - module specific items

    • org.opendaylight.mdsal.gen.v2.urn.example.network.topology.type - interfaces / classes representing derived types

    • org.opendaylight.mdsal.gen.v2.urn.example.network.topology.grp - interfaces / classes representing grouping and their children

    • org.opendaylight.mdsal.gen.v2.urn.example.network.topology.data - interfaces / classes representing notifications, rpcs, data tree

Methods introduced by Binding Specification conflicts with modeled items

Related bugs

Binding Specification v1 uses getter pattern for representing nested children derived from YANG model. Name of child is converted to valid JAVA name and prepended with is or get prefix.

Unfortunately Java & Binding Specification v1 also uses get prefix for some methods.

Example conflicting model
container example {

  list property {
    key `key`;
    leaf key { (1)
        type string;
    }
  }

  leaf implemented-interface { (2)
    type string;
  }

  leaf class { (3)
    type string;
  }

}

  1. Conflicts with getKey introduced by Identifiable which is used for lists with key

  2. Conflicts with getImplementedInterface defined in DataContainer which is base interface of all generated lists, containers, cases, choices

  3. Conflicts with getClass defined in Object which is root of all Java classes

Enumeration mapping is based on incorrect assumptions

Related Bugs

Enumeration mapping was based on notion / idea that names of possibles values are identifier as defined in RFC6020, but actual name is string.

RFC 6020: Section 9.6.4 The enum Statement
It takes as an argument a string which is the assigned name.  The
string MUST NOT be zero-length and MUST NOT have any leading or
trailing whitespace characters (any Unicode character with the
"White_Space" property).  The use of Unicode control codes SHOULD be
avoided.

This mapping makes impossible to represent following model:

typedef math-operand {
  type enumeration {
    enum "+";
    enum "/";
    enum "*";
    enum "-"";
  }
}

4625: groupings should not share classes with their instantiations

Correct use of Java language

  • 2791: Java Bindings: do not generate underscores in identifiers

  • 5671: Java Binding: missing @return (binding spec. v2)

Inner class with same name as outer class is not allowed

Related bugs

Inner classes are used for generation of anonymous union, bit and enumeration types defined in model.

In Java inner class MUST NOT have same name as outer class, which causes compilation error for following model:

grouping flags {
  leaf flags {
    type bits {
      bit one;
      bit two;
    }
  }
}

grouping status {
  leaf status {
    type enumeration {
      enum open;
      enum closed;
    }
  }
}

Incompatibilities with allowed model upgrade paths

Multiple augmentations of same target should result in one interface

Mappings incompatible with YANG 1.1

Identity mapping does not allow for identities with multiple bases

YANG 1.1

Example model
module example-crypto-base {
  yang-version 1.1;
  namespace "urn:example:crypto-base";
  prefix `crypto`;

  identity crypto-alg {
   description
     "Base identity from which all crypto algorithms
      are derived.";
  }

  identity symmetric-key {
   description
     "Base identity used to identify symmetric-key crypto
      algorithms.";
   }

  identity public-key {
   description
     "Base identity used to identify public-key crypto
      algorithms.";
   }
  }

  module example-des {
  yang-version 1.1;
  namespace "urn:example:des";
  prefix `des`;

  import `example-crypto-base` {
   prefix `crypto`;
  }

  identity des {
   base "crypto:crypto-alg";
   base "crypto:symmetric-key";
   description "DES crypto algorithm";
  }

  identity des3 {
   base "crypto:crypto-alg";
   base "crypto:symmetric-key";
   description "Triple DES crypto algorithm";
  }
}

Derived enumeration could limit valid values

Derived bits could limit valid values

Missing Mappings of YANG concepts

  • 706: Missing support for anyxml // Supplier<Source>

Usability issues

  • 2872: Generated Java Enumerations should contain mapping to the string counter part

  • 1870: Binding Specification: Type empty needs better representation than Boolean or Null vs NonNul

  • 5673: Add "add"/"del" utility methods to builders.

  • 5667: Incorrect use of format strings in generated code when backing type is an array (binding spec v2)

2641: Enumeration value defined in yang model is translated without underscore

Generate Equivalency for comparison of items by key and unique

ChildOf<> does not properly work with Choice / Case

  • 1466: InstanceIdentifier is unable to represent Choice / Case, only children

  • 1644: InstanceIdentifier does not properly capture choice-case child relationship

Mapping of list and leaf-list does not properly captures modeled semantic

After analysis of specification, implementation of applications and MD-SAL, we found out that list and leaf-list keyword actually has three different behaviors based on combination of key and ordered-by statements.

In order to correctly expose this to Binding Applications, representation in parent node should be extended to facilitate this mapping should be changed.

Table 1. List mappings / behavior
Key statement Ordered-by Behaviour v2 Type

key is defined

system (default)

Unordered map

Map

key is defined

user

Ordered map

Map

key is not defined

 — 

Ordered

List
Table 2. Leaf list mapping / behavior
Ordered-by Behaviour v2 Type

system (default)

Unordered Distinct

Set

user

Ordered Distinct

Set

1097:Return an empty list and never null from list-valued parameters

Leaf, leaf-list Instance Identifiers

Instance Identifier currently are constructed using classes as path arguments

  • is fine and allows for Instance Identifier to capture target type, but works only for container and list

Instance Identifier needs to be extended to allow targeting:

  • leaves

  • choice and case statements

Proposed solution

Introduce LeafPathArgument. LeafPathArguments for leafes will be stored in interface describing parent container as constants. This will allow for use such as:

InstanceIdentifier<Boolean> activePath = InstanceIdentifier.create(Foo.class).leaf(Foo.ACTIVE);
ListenableFuture<Optional<Boolean>> active = tx.read(CONFIGURATION,activePath);

This will require changing signature of MD-SAL to allow Object in its interfaces if we want to read boolean directly. Other approach is to have special DTO which implements DataObject and encapsulates LeafValue, this will allow MD-SAL to still limit input to DataObject.

InstanceIdentifier<LeafValue<Boolean>> activePath = InstanceIdentifier.create(Foo.class).leaf(Foo.ACTIVE);
ListenableFuture<Optional<LeafValue<Boolean>>> active = tx.read(CONFIGURATION,activePath);

Note: Use of Optional is property of MD-SAL and not of Binding Specification

Collections should be really immutable in immutable transfer objects

Performance issues

  • 3642: Improve equals() implementation

  • 3147: auto generated code by YANGTOOLS could be more efficient.

  • 5669: auto generated code by YANGTOOLS could be more efficient (binding spec. v2)

Other

  • 1478: Autoboxing support

  • 1095: Simplify InstanceIdentifer creation

  • 1117: Improve RPC API error handling

  • 1459: Reorganize yang-binding

  • 2289: Binding codegen: RFC6020 defines the order of evaluation for union members

  • 5668: Binding codegen: RFC6020 defines the order of evaluation for union members (binding spec v2)

Appendix 1: YANG 1.1 List of changes

Note
 This is verbatim copy of Section 1.1 of YANG 1.1 Draft

  • Changed the YANG version from "1" to "1.1".

  • Made the yang-version statement mandatory.

  • Made noncharacters illegal in the built-in type string.

  • Defined the legal characters in YANG modules.

  • Changed the rules for the interpretation of escaped characters in double quoted strings. This is an backwards incompatible change from YANG version 1. A module that uses a character sequence that is now illegal must change the string to match the new rules.

  • An unquoted string cannot contain any single or double quote characters. This is an backwards incompatible change from YANG version 1.

  • Extended the if-feature syntax to be a boolean expression over feature names.

  • Allow if-feature in bit, enum, and identity.

  • Allow if-feature in refine.

  • Made when and if-feature illegal on list keys.

  • Allow choice as a shorthand case statement.

  • Added a new substatement modifier to pattern.

  • Allow must in input, output, and notification.

  • Allow require-instance in leafref.

  • Allow augment to add conditionally mandatory nodes.

  • Added a set of new XPath functions.

  • Clarified the XPath context’s tree.

  • Defined the string value of an identityref in XPath expressions.

  • Clarified what unprefixed names mean in leafrefs in typedefs.

  • Allow identities to be derived from multiple base identities.

  • Allow enumerations and bits to be subtyped.

  • Allow leaf-lists to have default values.

  • Allow non-unique values in non-configuration leaf-lists.

  • Use [RFC7405] syntax for case-sensitive strings in the grammar.

  • Changed the module advertisement mechanism.

  • Changed the scoping rules for definitions in submodules. A submodule can now reference all definitions in all submodules that belong to the same module, without using the include statement.

  • Added a new statement action that is used to define operations tied to data nodes.

  • Allow notifications to be tied to data nodes.

  • Added a new data definition statement anydata.

  • Allow types empty and leafref in unions.

  • Allow type empty in a key.

Appendix 2: YANG 1.1 Updating a Module

Note
 Italics text means section was added in YANG 1.1. This is verbatim copy of Section 11 of YANG 1.1 Draft

As experience is gained with a module, it may be desirable to revise that module. However, changes to published modules are not allowed if they have any potential to cause interoperability problems between a client using an original specification and a server using an updated specification.

For any published change, a new revision statement (Section 7.1.9) MUST be included in front of the existing revision statements. If there are no existing revision statements, then one MUST be added to identify the new revision. Furthermore, any necessary changes MUST be applied to any meta-data statements, including the organization and contact statements (Section 7.1.7, Section 7.1.8).

Note that definitions contained in a module are available to be imported by any other module, and are referenced in import statements via the module name. Thus, a module name MUST NOT be changed. Furthermore, the namespace statement MUST NOT be changed, since all XML elements are qualified by the namespace.

Obsolete definitions MUST NOT be removed from published modules since their identifiers may still be referenced by other modules.

A definition in a published module may be revised in any of the following ways:

  • An enumeration type may have new enums added, provided the old enums’s values do not change. Note that inserting a new enum before an existing enum or reordering existing enums will result in new values for the existing enums, unless they have explicit values assigned to them.

  • A bits type may have new bits added, provided the old bit positions do not change. Note that inserting a new bit before an existing bit or reordering existing bit will result in new positions for the existing bits, unless they have explicit positions assigned to them.

  • A range, length, or pattern statement may expand the allowed value space.

  • A default statement may be added to a leaf that does not have a default value (either directly or indirectly through its type).

  • A units statement may be added.

  • A reference statement may be added or updated.

  • A must statement may be removed or its constraint relaxed.

  • A when statement may be removed or its constraint relaxed.

  • A mandatory statement may be removed or changed from true to false.

  • A min-elements statement may be removed, or changed to require fewer elements.

  • A max-elements statement may be removed, or changed to allow more elements.

  • A description statement may be added or clarified without changing the semantics of the definition.

  • A base statement may be added to an identity statement.

  • A base statement may be removed from an identityref type, provided there is at least one base statement left.

  • New typedefs, groupings, rpcs, notifications, extensions, features, and identities may be added.

  • New data definition statements may be added if they do not add mandatory nodes (Section 3) to existing nodes or at the top level in a module or submodule, or if they are conditionally dependent on a new feature (i.e., have an if-feature statement that refers to a new feature).

  • A new case statement may be added.

  • A node that represented state data may be changed to represent configuration, provided it is not mandatory (Section 3).

  • An if-feature statement may be removed, provided its node is not mandatory (Section 3).

  • A status statement may be added, or changed from current to deprecated or obsolete, or from deprecated to obsolete.

  • A type statement may be replaced with another type statement that does not change the syntax or semantics of the type. For example, an inline type definition may be replaced with a typedef, but an int8 type cannot be replaced by an int16, since the syntax would change.

  • Any set of data definition nodes may be replaced with another set of syntactically and semantically equivalent nodes. For example, a set of leafs may be replaced by a uses of a grouping with the same leafs.

  • A module may be split into a set of submodules, or a submodule may be removed, provided the definitions in the module do not change in any other way than allowed here.

  • The prefix statement may be changed, provided all local uses of the prefix also are changed.

Otherwise, if the semantics of any previous definition are changed (i.e., if a non-editorial change is made to any definition other than those specifically allowed above), then this MUST be achieved by a new definition with a new identifier.

In statements that have any data definition statements as substatements, those data definition substatements MUST NOT be reordered. If new data definition statements are added, they can be added anywhere in the sequence of existing substatement.

Appendix 3: Refactoring YANG model example

Design of binding specification version 2 in case of refactoring initial YANG model:

Example 1a, 1b:

module foo1a {
    namespace "urn:test:foo1a";
    prefix f1a;

    revision 2016-01-01 {
        description "Initial YANG model";
    }

    container a {
        container b {
            container c {
            }
        }
    }
}

module foo1b {
    namespace "urn:test:foo1b";
    prefix f1b;

    revision 2016-01-01 {
        description "First refactor only augment";
    }

    container a {
    }

    augment "/a" {
        container b {
        }
    }

    augment "/a/b" {
        container c {
        }
    }
}

Both previous modules foo1a & foo1b generate following instantiated Java structure:

      getB        getC
data.A -> data.a.B -> data.a.b.C

as augments become "invisible" in this one module context.

Example 2a:

module foo2a {
    namespace "urn:test:foo2a";
    prefix f2b;

    revision 2016-01-01 {
        description "Second refactor one grouping";
    }

    grouping a {
        container b {
            container c {
            }
        }
    }

    container a {
        uses a;
    }
}

In module foo2a, one grouping is added:

grp.A  ->  grp.a.B -> grp.a.bC
  |   getB   |    getC   |
data.A -> data.a.B -> data.a.b.C

Example 2b:

module foo2b {
    namespace "urn:test:foo2b";
    prefix f2;

    revision 2016-01-01 {
        description "Third refactor grouping augment";
    }

    grouping a {
        container b {
        }
    }

    container a {
        uses a {
            augment b {
                container c {
                }
            }
        }
    }
}

In module foo2b, one grouping and one augment is added:

grp.A  ->  grp.a.B
  |   getB   |    getC
data.A -> data.a.B -> data.a.b.C
module foo3 {
    namespace "urn:test:foo3";
    prefix f3;

    revision 2016-01-01 {
        description "Fourth refactor groupings only";
    }

    grouping a {
        container b {
            uses b;
        }
    }

    grouping b {
        container c {
        }
    }

    container a {
        uses a;
    }
}
            grp B  -> grp b.C
      getB   |    getC   |
grp.A  ->  grp.a.B -> grp a.b.C
  |   getB   |    getC   |
data.A -> data.a.B -> data.a.b.C

  • pros vs. binding spec v1:

    • well covered relations between elements

    • classes with same name in different packages (partially solves binding spec. v1 issue)

  • cons vs. binding spec v1:

    • higher amount of classes

    • higher memory consumption

    • amount of classes with same name (will be tackled by aliases)

Appendix 4: Augmenting YANG model example

Design of binding specification version 2 in case of augment:

  • one YANG model

module foo1a {
    namespace "urn:test:foo1a";
    prefix f1a;

    revision 2016-01-01 {
        description "Default code";
    }

    container a {
        container b {
            container c {
            }
        }
        container bar {
        }
    }
}

or

module foo1b {
    namespace "urn:test:foo1b";
    prefix f1b;

    revision 2016-01-01 {
        description "First refactor, this code should look the same as the default code due
        to the fact that these augments are in the same module";
    }

    container a {
    }

    augment a {
        container b {
        }
    }

    augment a {
        container bar {
        }
    }

    augment "/a/b" {
        container c {
        }
    }
}

Both previous modules foo1a & foo1b generate following instantiated Java structure:

A -> a.B -> a.b.C
  -> a.Bar

  • multiple YANG models

module foo2 {
    namespace "urn:test:foo2";
    prefix f2;

    revision 2016-01-01 {
        description "Augments of the same element should be put together";
    }

    import foo1a {
        prefix f1;
        revision-date 2016-01-01;
    }

    augment "/f1:a" {
        container from-b {
        }
    }

    augment "/f1:a/f1:b" {
        container from-b-1 {
        }
    }

    augment "/f1:a/f1:b" {
        container from-b-2 {
        }
    }
}

Previous module foo2 (alias "b") and foo1a (alias a) generates following instantiated java structure:

A -> a.B -> a.b.C
         -> b.BB -> FromB1
                 -> FromB2
  -> a.Bar
  -> b.BA -> b.ba.FromB

Appendix 5: DTOs & builders

DTO and builders needs to be in different packages

    container foo {             class fooBuilder
    }

    container foo-builder {     interface fooBuilder
    }
    list foo {                  data.Foo
        key identifier;         key.foo.FooIdentifier
        leaf identifier {
            type union {        type.foo.identifier.IdentifierUnion
                type string;
            }
        }
    }

    container foo-identifier {  data.FooIdentifier
    }

    typedef foo-identifier {    type.FooIdentifier
    }
    grouping nodes {
        list node {     for grouping key.grp.nodes.node.nodeidentifier
            key id;
            leaf id {
                type leafref;
            }
        }
    }

    container nodes {
        uses nodes;     for instantiated key.data.nodes.node.nodeidentifier
    }

Appendix 6: Various YANG model snippets & mappings

grouping A {          ->  grp.AGrouping
    container B       ->  grp.a.BData
    container B-DATA  ->  grp.a.BDataData
}

container A {         ->  data.A extends AGrouping
    uses A;           ->  data.a.B extends BData
                          data.a.BData extends BDataData
}
grouping a {          ->  interface grp.AGrouping {
    list b;                 List<? extends grp.a.BData>();
}                         }

container a {         ->  interface data.A extends AGrouping {
    uses a;                 List<data.a.B> getB();
}                         }