<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE rfc [
<!ENTITY nbsp    "&#160;">
<!ENTITY zwsp   "&#8203;">
<!ENTITY nbhy   "&#8209;">
<!ENTITY wj     "&#8288;">
]>
<?xml-model href="rfc7991bis.rnc"?>
<rfc
    xmlns:xi="http://www.w3.org/2001/XInclude"
    docName="draft-jouqui-netmod-yang-full-include-00"
    category="std"
    ipr="trust200902"
    obsoletes=""
    updates=""
    submissionType="IETF"
    xml:lang="en"
    tocInclude="true"
    sortRefs="true"
    symRefs="true"
    version="3" >
<front>
    <title abbrev="YANG Full Include">YANG Full Include</title>
    <seriesInfo name="Internet-Draft" value="draft-jouqui-netmod-yang-full-include"/>
    <author fullname="Thomas Joubert" initials="T" surname="Joubert">
        <organization>Huawei</organization>
        <address>
            <email>thomas.joubert1@huawei-partners.com</email>
        </address>
    </author>
    <author fullname="Jean Quilbeuf" initials="J" surname="Quilbeuf">
        <organization>Huawei</organization>
        <address>
            <email>jean.quilbeuf@huawei.com</email>
        </address>
    </author>
    <author fullname="Benoit Claise" initials="B" surname="Claise">
        <organization>Huawei</organization>
        <address>
            <email>benoit.claise@huawei.com</email>
        </address>
    </author>
    <date year="2023" month="11" day="6"/>
    <area>General</area>
    <workgroup>NETMOD</workgroup>
    <abstract>
        <t>
            YANG lacks re-usability of models defined outside of the grouping
            and augmentation mechanisms. For instance, it is almost impossible
            to reuse a model defined for a device in the context of the
            network, i.e by encapsulating it in a list indexed by device IDs.
            <xref target="RFC8528"/> defines the YANG mount mechanism, partially solving the
            problem by allowing to include schemas at deploy or runtime. This
            document aims to provide the same mechanism at design time.
        </t>
    </abstract>
</front>
<middle>
    <section anchor="intro" >
        <name>Introduction</name>
        <t>
            <xref target="RFC8528"/> introduces the challenges of reusing existing YANG modules, especially when they need to be included under a specific node of another module.
            In that RFC, three different phases of data model life cycle are identified: "design time", "implementation time" and "run time".
            Only the last two are covered.
            We focus here on the first phase of the life cycle, that is inserting modules at design time.
        </t>
        <t>
            We identified some use cases that require this design time definition of which modules need to be included in the top-level module.
            The have in common the need to re-use YANG modules defined for the devices in the context of a network-level module.
            Also, they both aim to define a model that is independent of the underlying devices.
        </t>
        <ul>
            <li>
                The use case that triggered the creation of this document is <xref target="I-D.ietf-opsawg-collected-data-manifest"/>.
                In this draft, the goal is to provide a YANG model giving the context in which YANG-push <xref target="RFC8641"/> data are collected so that they can be exploited a posteriori.
                To get the full context, we need the hardware and os version of each device, but also the list of YANG modules supported by the devices and the parameters for the YANG-push subscriptions.
                For the last two items YANG Library <xref target="RFC8525"/> and YANG Push <xref target="RFC8641"/> provide good and standard modules for representing this information at the device level.
                However, the data manifests need to be considered at the network level, so that we can distinguish between the devices from which they come.
                In YANG, that means including them in a list indexed by the device id, which proves out to be difficult without copy-pasting the original modules.
            </li>
            <li>
                A similar use case is the digital map <xref target="I-D.havel-opsawg-digital-map"/>, where the goal is to build a model of the network.
                In particular, to model the devices a lot of standard modules have already been defined by the IETF and there is a need to reuse these modules to build this larger network model.
                The <eref target="https://datatracker.ietf.org/wg/ivy/about/">IVY workgroup</eref> might also rely on the pattern of re-using device level modules into a network model.
            </li>
        </ul>
        <t>
            YANG Schema Mount <xref target="RFC8528"/> and Peer Mount <xref target="I-D.clemm-netmod-peermount"/> focus on mounting a given part of a an existing data instance into another data instance.
            Although the final goal is the same: being able to reuse modules defined elsewhere in order to avoid redefining them, the approach is more focused on the runtime than the design time.
            In the first case, the mapping between the mount points and the existing modules to be included at that mount point is left to the NETCONF <xref target="RFC6241"/> server.
            Thus, to guarantee that the contents under a given mount point conforms to a predefined schema requires the proper configuration of the server.
            In the case of Peer mount, the focus is on synchronizing a given subtree of a remote server with a subtree of the local server.
            Again, the contents under the local subtree cannot be enforced from the design time.
        </t>
        <t>
            In this document, we propose a new extension, named full include.
            This extension enables reusing imported modules by rooting them at an arbitrary point of the data model.
            The concept of mount point from <xref target="RFC8528"/> is replaced by a list of "full:include" statement, each statement corresponding to the inclusion of one imported module at that location.
            In that sense, the design time solution is a pure YANG solution that does not rely on external configuration to specify the list of mounted modules, hence the term full include rather than mount.
        </t>
        <t>
            The obtained data model that we want to associate to our construct is similar to the one obtained by specifying a mount point and binding it to the same set of modules.
            Therefore, we can reuse the concepts of the YANG schema mount to define the semantics of our new extension.
        </t>
    </section>
    <section anchor="terminology">
        <name>Terminology</name>
        <t>
            The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL
            NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED",
            "MAY", and "OPTIONAL" in this document are to be interpreted as
            described in BCP 14 <xref target="RFC2119"/> <xref target="RFC8174"/>
            when, and only when, they appear in all capitals, as shown here.
        </t>
        <t>
            The following terms are defined in <xref target="RFC7950"/>:
        </t>
        <ul>
            <li>data model</li>
            <li>data node</li>
        </ul>
        <t>
            The following terms are defined in <xref target="RFC7950"/>:
        </t>
        <ul>
            <li>mount point</li>
        </ul>
    </section>
    <section anchor="yang-full-include">
        <name>Full Include</name>
        <t>
            The full include mechanism defined in this document completes <xref target="RFC8528"/>, by providing a mechanism to "mount" modules at design time.
            Supporting mounting modules at this step of the data model life cycle is left out of scope in <xref target="RFC8528"/>.
        </t>
        <t>
            The approach for supporting the full include mechanism is to keep the semantics of <xref target="RFC8528"/> for the resulting data model.
            In <xref target="RFC8528"/>, the list of modules to mount in each mount point is left to the NETCONF server.
            In this document, we propose the full include mechanism to define this mapping directly in the YANG module that includes the mounted modules.
        </t>
        <t>
            In the sequel, we use "full" as the prefix for the module 'ietf-yang-full-include' (see <xref target="yang-module-code"/>).
            Thus "full:include" refers to the extension 'include' defined in that module.
        </t>
        <section anchor="yang-full-include-def">
            <name>Definition</name>
            <t>
                The "full:include" statement can appear as a sub-statement of the following statements:
            </t>
            <ul>
                <li>container</li>
                <li>list</li>
            </ul>
            <t>
                The "full:include" statement takes as an argument a prefix, that must be the prefix associated to an imported module. Modules can contain
                multiple uses of the "full:include" statement. The "container" and "list" statements MAY contain multiple uses
                of the "full:include" statement on the same level.
                These multiple uses define the full list of modules to be included, rooted in node where the "full:include" statement is used.
            </t>
            <t>
                The "full:include" statement can be interpreted using YANG Schema Mount <xref target="RFC8528"/>, by following these steps:
            </t>
            <ol>
                <li>For each set of "full:include" statement located in the same node, replace them by a single "mount-point" with a unique label.</li>
                <li>Declare each of these "mount-points" as "shared-schema" in the data model defined in <xref target="RFC8528"/>.</li>
                <li>
                    In the instance corresponding to each "mount-point", define the ietf-yang-library <xref target="RFC8525"/> to include a module-set (at '/yang-library/module-set/) with the following.
                    The list 'module' contains an entry for every module referred to in the set of "full:include" statements corresponding to the "mount-point".
                    Additionally, the list 'module' contains an entry for "ietf-yang-library" as it is needed by YANG Schema mount.
                    As usual, the list 'imported-modules' contains the list of dependencies needed by the modules in the 'module' list.
                </li>
            </ol>
            <t>An example of module using "full:include" and its translation into a similar YANG Schema mount version is presented in <xref target="examples"/>.</t>
        </section>
        <section>
            <name>Limitations</name>
            <ul>
                <li> A module MUST NOT use the "full:include" statement with its own prefix as argument. This rule prevents any infinite recursion in the mounted schemas.</li>
                <li> Modules used as arguments for the "full:include" statement MUST be valid and compile successfully, independently of the module it is used in.</li>
            </ul>
        </section>
    </section>

    <!-- [We’re not yet there, let’s keep it simple for now]section>
        <name>Implementation strategies</name>
        <section>
            <name>Re-using YANG schema mount</name>
        </section>
        <section>
            <section anchor="full-include-of-YANG-schema-nodes">
                <name>Defining a new extension</name>
                <section>
                    <name>Included Data Nodes Path Adjustment</name>
                    <t>
                        In the case of the call to a "leafref" statement, one must be able to specify the path of a leaf, even through a module passed as an argument to a "full-include" statement.
                        The paths of the nodes contained in the module (referred in the following paragraph as "the included nodes") used in a "full-include" statement are computed as follows.
                        TODO: explain how the path are computed, which basically boils down to concatenating the main module path with the included one ?
                    </t>
                </section>
                <section anchor="rpc-operations">
                    <name>RPC operations</name>
                    <t>
                        Todo, read the schema-mount RFC part on this, and apply to our case
                    </t>
                </section>
            </section>
        </section>
    </section-->

    <section anchor="yang-module-code">
        <name>ietf-full-include YANG module</name>
        <t>
            We present in this section the YANG module defining the "full-include" extension.
            The module in itself defines solely the 'include' extension.
            A module importing this extension SHOULD the prefix 'full', so that the statement reads "full:include" when used in the code.
        </t>
        <sourcecode name="ietf-full-include@2023-11-03.yang" type="yang" markers="true" ><![CDATA[
module ietf-yang-full-include {
  yang-version 1.1;
  namespace "urn:ietf:params:xml:ns:yang:ietf-yang-full-include";
  prefix full;

  organization
    "IETF NETMOD (NETCONF Data Modeling Language) Working Group";
  contact
    "WG Web:   <https://datatracker.ietf.org/wg/netmod/>
     WG List:  <mailto:netmod@ietf.org>

     Editor:   ";
  description
    "This module defines a YANG extension statement that can be used
     to incorporate data models defined in other YANG modules in a
     module.

     The key words 'MUST', 'MUST NOT', 'REQUIRED', 'SHALL', 'SHALL
     NOT', 'SHOULD', 'SHOULD NOT', 'RECOMMENDED', 'NOT RECOMMENDED',
     'MAY', and 'OPTIONAL' in this document are to be interpreted as
     described in BCP 14 (RFC 2119) (RFC 8174) when, and only when,
     they appear in all capitals, as shown here.

     Copyright (c) 2023 IETF Trust and the persons identified as
     authors of the code.  All rights reserved.

     Redistribution and use in source and binary forms, with or
     without modification, is permitted pursuant to, and subject to
     the license terms contained in, the Revised BSD License set
     forth in Section 4.c of the IETF Trust's Legal Provisions
     Relating to IETF Documents
     (https://trustee.ietf.org/license-info).

     This version of this YANG module is part of RFC XXXX;
     see the RFC itself for full legal notices.";

  revision 2023-11-05 {
    description
      "Initial revision.";
    reference
      "RFC XXXX: YANG Full Include";
  }

  extension include {
    argument prefix;
    description
      "The argument 'prefix' MUST be the prefix of a module imported
       by the calling module.

       The 'include' statement MUST NOT be used in a YANG
       version 1 module, neither explicitly nor via a 'uses'
       statement.
       The 'include' statement MAY be present as a substatement
       of 'container' and 'list' and MUST NOT be present elsewhere.

       Whenever a sequence of 'include' statements is used, the
       schema tree defined by the set of the included modules is
       inserted in the schema tree of the calling module, at the
       place where the sequence is declared";
  }
}
            ]]>
        </sourcecode>
    </section>
    <!--section anchor="use-examples" >
        <name>Examples Of Use</name>
        <t>
            Todo: add examples of use. Illustrating both the "full-include" statement, as well as the "leafref" through a module that has been "fully-included".
        </t>
    </section-->

    <section anchor="security">
        <name>Security Considerations</name>
        <t>
            TODO
        </t>
    </section>
    <section anchor="iana">
        <name>IANA Considerations</name>
        <t>
            TODO
        </t>
    </section>
    <section>
        <name>Contributors</name>
    </section>
    <section>
        <name>Open issues</name>
        <ul>
            <li>What name should we give to this draft? Any suggestions instead of full include?</li>
            <li>Do we want to support the parent-nodes mechanism from <xref target="RFC8528"/>?</li>
            <li>Do we allow full include from within a grouping?</li>
            <li>Does this mechanism already exist?</li>
        </ul>
    </section>
    </middle>
    <back>
        <references>
            <name>References</name>
            <references>
                <name>Normative References</name>
                <xi:include href="http://xml.resource.org/public/rfc/bibxml/reference.RFC.2119.xml"/>
                <xi:include href="http://xml.resource.org/public/rfc/bibxml/reference.RFC.7950.xml"/>
                <xi:include href="http://xml.resource.org/public/rfc/bibxml/reference.RFC.8174.xml"/>
                <xi:include href="http://xml.resource.org/public/rfc/bibxml/reference.RFC.8528.xml"/>
            </references>
            <references>
                <name>Informative References</name>
                <xi:include href="http://xml.resource.org/public/rfc/bibxml/reference.RFC.6241.xml"/>
                <xi:include href="http://xml.resource.org/public/rfc/bibxml/reference.RFC.8525.xml"/>
                <xi:include href="http://xml.resource.org/public/rfc/bibxml/reference.RFC.8340.xml"/>
                <xi:include href="http://xml.resource.org/public/rfc/bibxml/reference.RFC.8641.xml"/>
                <xi:include href="https://bib.ietf.org/public/rfc/bibxml3/reference.I-D.ietf-opsawg-collected-data-manifest.xml"/>
                <xi:include href="https://bib.ietf.org/public/rfc/bibxml3/reference.I-D.havel-opsawg-digital-map.xml"/>
                <xi:include href="https://bib.ietf.org/public/rfc/bibxml3/reference.I-D.clemm-netmod-peermount.xml"/>

            </references>
        </references>
        <?rfc needLines="100"?>
        <section>
            <name>Changes between revisions</name>
            <t>
                No revisions
            </t>
        </section>
        <section anchor="examples">
            <name>Examples</name>
            <t>
                In this section we present some minimalistic examples in order to illustrate the "full:include" statement.
                For these examples, we are in a situation where we have a device-level module already defined and we want to have a network-level module that represent a list of device, each having an independent instance of the device-level module.
                This situation might arise if we want to simplify the network management by presenting a unified model for the network.
                In that case, the heterogeneity of the devices should be handled by mapping their model to the device-level module (which is clearly out of scope for this draft).
            </t>
            <t>
                In our simplistic example, the device-level module simply exposes the hostname and the cpu-usage of the device.
                Note that we cannot modify this device-level module, because in a more realistic example we would be reusing standard modules.
                The tree representation (<xref target="RFC8340"/>) of the 'device-level' module is depicted in <xref target="device-level-tree"/>.
            </t>
            <figure anchor="device-level-tree">
                <name>YANG Tree representation of the device-level module.</name>
                <artwork type="ascii-art">
                    <![CDATA[
module: device-level
  +--rw hostname     string
  +--ro cpu-usage?   int8
        ]]>
                </artwork>
            </figure>
            <t>
                For the network-level module, we have a list of devices indexed by their 'device-id'.
                The tree representation (<xref target="RFC8340"/>) of such a module is depicted in <xref target="network-level-tree-stub"/>.
            </t>
            <figure anchor="network-level-tree-stub">
                <name>YANG Tree representation of a stub for the network-level module</name>
                <artwork type="ascii-art">
                    <![CDATA[
module: network-level-stub
  +--rw devices
     +--rw device* [device-id]
        +--rw device-id    string
        ]]>
                </artwork>
            </figure>
            <t>
                The goal is now to complete this stub so that the full contents of the 'device-level' is added under the "device" list.
            </t>
            <section>
                <name>Example using YANG Full Include</name>
                <t>
                    We propose in this section a YANG module for 'network-level'.
                    The YANG code is presented in <xref target="network-level-full-include"/>.
                </t>
                <figure anchor="network-level-full-include">
                    <name>Version of the network-level module using full:include</name>
                    <artwork type="ascii-art">
                        <![CDATA[
module network-level {
  yang-version 1.1;
  namespace "urn:network-level";
  prefix "net-l";

  import "ietf-yang-full-include" {
    prefix "full";
  }

  import "device-level" {
    prefix "dev-l";
  }

  container devices {
    list device {
      key device-id;
      leaf device-id {
        type string;
      }
      full:include "dev-l";
    }
  }
}
                        ]]>
                    </artwork>
                </figure>
                <t>
                    At the moment, this code is accepted by the YANG compilers, but since the extension is not implemented, it simply ignores it.
                    Note that all the information (which modules to include, where to include them) is defined in this module.
                    More specifically, the line 'full:include "dev-l";' states that the full schema of the 'device-level' module, identified by its prefix "dev-l" must be included at that location.
                    By adding more occurrences of "full:include" there, one can define a more complex schema to be included at that location.
                </t>
            </section>
            <section>
                <name>Using YANG Schema Mount</name>
                <t>
                    In this section, we show how a similar result could be attained using YANG Schema Mount.
                    The network-level module is presented in <xref target="network-level-schema-mount"/>.
                </t>
                <figure anchor="network-level-schema-mount">
                    <name>Version of the network-level module using Schema Mount</name>
                    <artwork type="ascii-art">
                        <![CDATA[
module network-level {
  yang-version 1.1;
  namespace "urn:network-level";
  prefix "net-l";

  import ietf-yang-schema-mount {
    prefix yangmnt;
  }

  container devices {
    list device {
      key device-id;
      leaf device-id {
        type string;
      }
      yangmnt:mount-point "device-schema";
    }
  }
}
                        ]]>
                    </artwork>
                </figure>
                <t>
                    As explained in <xref target="yang-full-include-def"/>, the yang-library corresponding to the modules to include, as well as the data required by 'ietf-yang-mount' needs to be specified in some other files.
                    Using the 'yanglint' tool from libyang (<eref target="https://github.com/CESNET/libyang"/>), this module can be compiled to provide a tree representation as shown in <xref target="tree-full-sm"/>.
                </t>
                <figure anchor="tree-full-sm">
                    <name>Full tree of  both network- and device-level using Schema Mount</name>
                    <artwork type="ascii-art">
                        <![CDATA[
module: network-level
  +--rw devices
     +--mp device* [device-id]
        +--rw hostname/    string
        +--ro cpu-usage/?  int8
        +--rw device-id    string
                        ]]>
                    </artwork>
                </figure>
                <t>
                    The command for obtaining that schema is 'yanglint -f tree -p . -x extension-data.xml -Y network-level-yanglib.xml  yang/network-level.yang', assuming all the YANG modules and the two xml files are in the current folder.
                    The file 'network-level-yanglib.xml' contains the YANG Library data for the network-level module.
                    The file 'extension-data.xml' contains the YANG Library data defining the schema to use at the mount point, as well as the data required by YANG Schema Mount.
                    Both are reproduced in <xref target="files"/>.
                </t>
            </section>
            <section anchor="files">
                <name>Support Files</name>
                <t>
                    The code of the 'device-level' module is given in <xref target="device-level-yang"/>.
                    Then the data files 'network-level-yanglib.xml' and 'extension_data.xml' are provided.
                    These files are needed to compile the Schema Mount version of our example with yanglint.
                </t>
                <figure anchor="device-level-yang">
                    <name>device-level YANG module</name>
                    <artwork type="ascii-art">
                        <![CDATA[
module device-level {
  yang-version 1.1;
  namespace "urn:device-level";
  prefix mnt;

  leaf hostname {
    type string;
    mandatory true;
  }
  leaf cpu-usage {
    type int8;
    config false;
  }
}
                        ]]>
                    </artwork>
                </figure>
                <sourcecode name="network-level-yanglib.xml" type="xml" markers="true" ><![CDATA[
<yang-library xmlns="urn:ietf:params:xml:ns:yang:ietf-yang-library"
     xmlns:ds="urn:ietf:params:xml:ns:yang:ietf-datastores">
   <module-set>
     <name>main-set</name>
     <module>
       <name>ietf-datastores</name>
       <revision>2018-02-14</revision>
       <namespace>urn:ietf:params:xml:ns:yang:ietf-datastores</namespace>
     </module>
     <module>
       <name>ietf-yang-library</name>
       <revision>2019-01-04</revision>
       <namespace>urn:ietf:params:xml:ns:yang:ietf-yang-library</namespace>
     </module>
     <module>
       <name>ietf-yang-schema-mount</name>
       <revision>2019-01-14</revision>
       <namespace>urn:ietf:params:xml:ns:yang:ietf-yang-schema-mount</namespace>
     </module>
     <module>
       <name>network-level</name>
       <namespace>urn:network-level</namespace>
     </module>
     <import-only-module>
       <name>ietf-yang-types</name>
       <revision>2013-07-15</revision>
       <namespace>urn:ietf:params:xml:ns:yang:ietf-yang-types</namespace>
     </import-only-module>
     <import-only-module>
       <name>ietf-inet-types</name>
       <revision>2013-07-15</revision>
       <namespace>urn:ietf:params:xml:ns:yang:ietf-inet-types</namespace>
     </import-only-module>
   </module-set>
   <schema>
     <name>main-schema</name>
     <module-set>main-set</module-set>
   </schema>
   <datastore>
     <name>ds:running</name>
     <schema>main-schema</schema>
   </datastore>
   <datastore>
     <name>ds:operational</name>
     <schema>main-schema</schema>
   </datastore>
   <content-id>1</content-id>
 </yang-library>
 <modules-state xmlns="urn:ietf:params:xml:ns:yang:ietf-yang-library">
   <module-set-id>2</module-set-id>
 </modules-state>
                ]]>
                </sourcecode>
                <sourcecode name="extension_data.xml" type="xml" markers="true" ><![CDATA[
<yang-library xmlns="urn:ietf:params:xml:ns:yang:ietf-yang-library"
              xmlns:ds="urn:ietf:params:xml:ns:yang:ietf-datastores">
    <module-set>
        <name>mountee-set</name>
        <module>
            <name>device-level</name>
            <namespace>urn:device-level</namespace>
        </module>
        <module>
            <name>ietf-datastores</name>
            <revision>2018-02-14</revision>
            <namespace>urn:ietf:params:xml:ns:yang:ietf-datastores</namespace>
        </module>
        <module>
            <name>ietf-yang-library</name>
            <revision>2019-01-04</revision>
            <namespace>urn:ietf:params:xml:ns:yang:ietf-yang-library</namespace>
        </module>
        <import-only-module>
            <name>ietf-yang-types</name>
            <revision>2013-07-15</revision>
            <namespace>urn:ietf:params:xml:ns:yang:ietf-yang-types</namespace>
        </import-only-module>
        <import-only-module>
            <name>ietf-inet-types</name>
            <revision>2013-07-15</revision>
            <namespace>urn:ietf:params:xml:ns:yang:ietf-inet-types</namespace>
        </import-only-module>
    </module-set>
    <schema>
        <name>test-schema</name>
        <module-set>mountee-set</module-set>
    </schema>
    <datastore>
        <name>ds:running</name>
        <schema>test-schema</schema>
    </datastore>
    <datastore>
        <name>ds:operational</name>
        <schema>test-schema</schema>
    </datastore>
    <content-id>2</content-id>
</yang-library>
<modules-state xmlns="urn:ietf:params:xml:ns:yang:ietf-yang-library">
   <module-set-id>2</module-set-id>
</modules-state>
<schema-mounts xmlns="urn:ietf:params:xml:ns:yang:ietf-yang-schema-mount">
<mount-point>
    <module>network-level</module>
    <label>device-schema</label>
    <shared-schema/>
</mount-point>
</schema-mounts>
                ]]>
                </sourcecode>
            </section>
        </section>
        <section numbered="false">
            <name>Acknowledgements</name>
            <t>
                TODO: acknowledgements
            </t>
        </section>
    </back>
</rfc>
<!-- Local Variables: -->
<!-- fill-column:72 -->
<!-- End: -->