Watsen Networks

kent+ietf@watsen.net

General GitHub Integration and Tooling (git) This document describes motivations behind and solutions for tooling to automate the extraction/insertion of artwork and source code to/from `rfc2xml` documents. While much may appear to be working, the author believes that, in order to be such automation to be maximally useful, it is necessary to solicit broad input from the community (co-authors are welcomed, both on the draft and the tool).

This document describes motivations behind and solutions for tooling to automate the extraction/insertion of artwork and source code to/from `rfc2xml` v2 and v3 . For authors, adoption of the automation ensures completely up-to-date <artwork> and <sourcecode> inclusions every time the document is published. For reviewers (especially shepherds, doctors, and copy editors), use of the automation offers assurance that the <artwork> and <sourcecode> inclusions are syntactically valid, and the ability to quickly verify that they are when needed.

At the time of this writing, `rfc2xml` v3 is not yet in production, and thus the tooling support described herein is intended to apply to both v2 and v3. Whenever ambiguity may arise, this document will fully write out text such as "...source code stored in the v2 <artwork> element...".

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

This document uses the following terms (sorted by name): The term "artwork" is used throughout to represent two-dimensional imagery (e.g., ASCII art), such as would be referenced by the <artwork> element defined in Section 2.5 of . The term "source code" is used throughout to represent a structured sequence of lines, such as would be referenced by the <sourcecode> element defined in Section 2.48 of .

This section is just a placeholder for now, but it is expected that will need to be modified in order to support some of this work. At a minimum, should be updated to support attributes from other namespaces, such that the `rfc2xml` tool would neither process nor discard them.

The driving motivation for this work is twofold: To ensure the correctness of works in progress. To simplify the formal verification process. Firstly, far too often, throughout the lifecycle of a draft, is it that authors overlook updating artwork and/or source code in their drafts, leading to confusion and wasting some of the precious little time of other working group members. While repeated encouragement from chairs and others to embrace automation, it seems that the bar is too high for some authors to bother for their one and perhaps only draft. It is actually a self-defeating strategy, as it has been shown time and again that the effort invested to do necessary script-fu would be recouped by the authors themselves over the lifetime of their draft. Next, for formal verifications, the YANG Doctor , reviews are first in mind, but the automation is equally useful for any structured syntax other than YANG , such as ASN.1 and ABNF . That said, publication process experience shows that doctor reviews are often out of synch with the document submitted for publication, sometimes even by several draft revisions. Thusly, it is common for the draft shepherds themselves to verify the correctness of inclusions when doing the shepherd writeup. Further, the document may be subsequently updated by IESG and/or copy editor reviews, steps for which the automation would continue to support.

Section 3.2 of states that normative YANG modules and submodules contained within Internet-Drafts and RFCs must be bracketed by <CODE BEGINS> and <CODE ENDS> markers. Section 3.1.18 of notes support for this in `xml2rfc` through the use of a `markers` attribute in the <sourcecode> element. [PS: these markers attempt to support extraction from plain-text documents but, as this document shows, extraction from XML is superior and, besides, there are many interesting things to extract beyond YANG modules.] The `xym` and `rfcstrip` utilities have been developed to extract YANG modules from Internet-Drafts and RFCs using the <CODE BEGINS> and <CODE ENDS> markers. The RFC Submit tool has been modified to test YANG modules contained within I-Ds, and the resulting document page in Datatracker [datatracker] displays a new "Yang Validation" field containing a varying color yin-yang symbol (green if no errors, red if errors) along with counts. This tool is okay for what it is, but it neither aids authors between updates nor validates anything beyond YANG modules. The YANG Validator site provides an Internet-facing service, with a REST-based API, for validating YANG modules in drafts. Having a REST API enables its use throughout a document's lifecycle but, again, it doesn't validate anything beyond YANG modules.

When asked to build a submittable `rfc2xml` document, the automation should perform the following steps, in order: Prime artwork and source code as needed. Known priming steps include: Draft revision addition/substitution for the `docName` attribute in the <rfc> element as well as in the filename. Date substitution (e.g., replacing the string "YYYY-MM-DD" with the current date. This substitution needs to occur both within files and in filenames. Generation of derived views (e.g., YANG tree diagrams ). Technically, the derived views should be generated after the validation (discussed next) but, said generation is rightly part of the "priming" step and, besides, if there is an error, the validation step would still catch it, so there's no harm in generating the derived views first. Validate source code and artwork. This step includes: Validating data models (e.g., YANG modules) against the schema describing their syntax. Validating data instance examples (e.g., a snippet of configuration) against the governing data models (e.g., the aforementioned YANG modules). Validating the derived views. Technically, this step is not needed during the insertion process, since the derived views were just generated in the previous step but, since the same validation logic is used by automated verification (see ), it is automatically executed here as well. Validating derived views is accomplished by running the script to generate the view and then comparing the result to the view extracted in the draft. Pack the final submittable XML file. This step includes: Pasting the contents referenced by attributes in the <artwork> and <sourcecode> elements into the XML document, and storing information enabling the content to be extracted back to its original filename. Additional attributes can control if markers are added (e.g., the <CODE BEGINS> and <CODE ENDS> markers described by RFC 8407 Section 3.2). Character data (CDATA) wrappers may be added. Folding (line wrapping) may be added, per . Additional date substitutions (e.g., YYYY-MM-DD) in the body of the draft may be needed. Draft revision substitution (i.e., replacing placeholder "-latest" with, e.g., "-03").

When asked to extract/verify a submitted `rfc2xml` document, the automation should perform the following steps, in order: Extract the contents of the <artwork> and <sourcecode> elements to their original file-based forms, retaining their file names and directory paths. Extract any additional files that may be necessary to generate the derived views and/or validate the inclusions. Optionally, if requested, also save the "primed" or "unpacked" XML file (i.e., the source XML file before the content was packed into it). At this point, the local directory tree represents the "primed" state, and thus the same validation logic described above can executed again.

In order to support the auto-generation of derived views and the validation of data models and data instance examples, the automation solution must not automatically execute arbitrary scripts. A couple solutions present themselves: Allow arbitrary scripts, but don't execute them automatically when a document is extracted. This solution is appealing as it still ensures these scripts were executed on the author's computer at time of construction, and the scripts themselves can be extracted and audited on the reviewer's computer. If desired, after auditing a script, a reviewer could choose to manually execute it on their own computer. Don't allow arbitrary scripts but, instead, support parameterized files that declare all the information necessary to construct the command(s) necessary to generate derived views and/or validate inclusions.

International Telecommunication Union

This section illustrates bits working using the `xiax` tool . This entire document has been processed by `xiax`, albeit with a little manual tweaking to return some of the "YYYY-MM-DD" strings (such as this one) back to their original forms. Note that, as this is an `xml2rfc` v2 document, all examples use the `xml2rfc` v2 <artwork> element.

This example illustrates inclusion of static content with no additional processing, such as might be useful for pre-generated artwork. This is what the original `xml2rfc` XML looked like:

<preamble>START FIGURE</preamble> <artwork xiax:src="art/hello.txt"/> <postamble>END FIGURE</postamble>

Here is the rendered content:

START FIGURE END FIGURE

This example illustrates inclusion of static content along with date substitution. Date substitution is triggered by the string "YYYY-MM-DD" appearing in the name of the file being included. The date-substitution is applied to both the filename as well as to the content of the file. This is what the original `xml2rfc` XML looked like:

<preamble>START FIGURE</preamble> <artwork xiax:src="art/email-YYYY-MM-DD.txt"/> <postamble>END FIGURE</postamble>

Here is the rendered content:

START FIGURE END FIGURE

The content of the original "src" file (before date-substitute):

This example illustrates both inclusion of generated content along with date substitution. This is what the original `xml2rfc` XML looked like:

<preamble>START FIGURE</preamble> <artwork xiax:gen="xiax/gen-foo-tree-diagram@YYYY-MM-DD.xml"/> <postamble>END FIGURE</postamble>

Here is the rendered content:

START FIGURE END FIGURE

The content of the "gen" file being referenced is:

foo@YYYY-MM-DD.yang ]]>

This example illustrates inclusion of static content with date substitution and behind-the-scenes validation. Note that, in this example, the date substitution occurs on the validation input file (since it references a date-substituted file). This is what the original `xml2rfc` XML looked like:

<preamble>START FIGURE</preamble> <artwork xiax:src="examples/ex-foo.json" xiax:val="xiax/val-xml-ex-foo@YYYY-MM-DD.xml"/> <postamble>END FIGURE</postamble>

Here is the rendered content:

START FIGURE END FIGURE

The content of the "val" file being referenced is:

foo@YYYY-MM-DD.yang foo@YYYY-MM-DD.yang ]]>

This example illustrates inclusion of static content with date substitution, <BEGIN CODE> and <END CODE> markers, and behind-the-scenes validation. This is what the original `xml2rfc` XML looked like:

<preamble>START FIGURE</preamble> <artwork xiax:src="foo@YYYY-MM-DD.yang" xiax:markers="true" xiax:val="xiax/val-yang-foo@YYYY-MM-DD.xml"/> <postamble>END FIGURE</postamble>

Here is the rendered content:

START FIGURE file "foo@2019-02-25.yang" module foo { yang-version 1.1; namespace "https://example.com/foo"; prefix "f"; revision "2019-02-25" { description "Initial version"; } leaf foo { type empty; } } ]]> END FIGURE

The content of the "val" file being referenced is:

foo@YYYY-MM-DD.yang foo@YYYY-MM-DD.yang ]]>

In order to support extractions, `xiax` needs to encode metadata into the `xml2rfc` XML file. This metadata encodes, for instance, the original names of files, contents of the 'gen' and 'val' files, and and additional files that may have been used by the 'gen' and/or 'val' files. There are limited options for encoding metadata into the `xml2rfc` format in a way that doesn't generate errors or is discarded during the Internet-Draft submission process. The only option that was discovered is to encode the metadata into an XML comment. Thus, `xiax` adds a special XML comment referred to as the "xiax-block" to the end of the `xml2rfc` XML file. An example xiax-block follows:

]]>

Note that this data is the base64-encoding of the GZIP-ed string for the encoded metadata. Also note that the xiax-block is versioned. The intention is that, while a given version of `xiax` will only produce the "current" xiax-block version, it should be capable of extracting content from a draft produced by any prior version.

Before diving into the "xiax-block", and in the interest of full disclosure, it should be known that `xiax` currently has limited support for content types. Specifically: For generating content, `xiax` currently only knows how to generate YANG tree diagrams . For validating content, `xiax` currently only knows how to validate YANG modules and XML/JSON documents against YANG schema. However, the code has been developed anticipating a desire to extend it to support other content types. And, being an open source project on GitHub, it is hoped that others will take interest to add support for additional content types.

Being that `xiax` operates on XML files, it was intuitive to use XML to encode the xiax-block. In IETF fashion, the schema for the XML data model is defined using YANG .

Following is the YANG Tree Diagram for the xiax-block:

+-- gen +-- attrib? inet:uri +-- file? +-- val +-- attrib? inet:uri +-- file? yang-data generate: +-- generate +-- (generate-type)? +--:(yang-tree-diagram) +-- yang-tree-diagram +-- source? string +-- print-yang-data? empty yang-data validate: +-- validate +-- (content-type)? +--:(yang-module) | +-- yang-module | +-- additional-yang-modules | +-- additional-yang-module* [name] | +-- name string | +-- uri* inet:uri +--:(xml-document) +-- xml-document +-- (schema-type)? | +--:(using-yang) | +-- using-yang | +-- yang-modules | +-- yang-module* [name] | +-- name string | +-- uri* inet:uri +-- additional-xml-documents +-- additional-xml-document* [name] +-- name string +-- uri* inet:uri ]]>

Following is the YANG module for the xiax-block follows:

"; description "This module defines the data model for xiax data block. Copyright (c) 2019 Watsen Networks. All rights reserved."; revision "2019-02-25" { description "Initial version"; } grouping val-grouping { container val { leaf attrib { type inet:uri; description "The original 'xiax:val' attribute that was in the element ( cannot be validated)."; } anydata file { description "The content of the file per the 'xiax:val' attribute"; } } } rc:yang-data "xiax-block" { container xiax-block { description "Contains lists of inclusions that were processed by `xiax` during its 'packing' step."; list inclusion { key path; description "A list of inclusions, one for each and/or element processed by `xiax`."; leaf path { type string; description "The DOM path of the or element."; } container src { leaf attrib { type inet:uri; description "The original 'xiax:src' attribute that was in the or element."; } uses val-grouping; } container gen { leaf attrib { type inet:uri; description "The original 'xiax:gen' attribute that was in the or element."; } anydata file { description "The content of the file per the 'xiax:gen' attribute"; } uses val-grouping; } } // end list inclusion } // end container xiax-block } // end rc:yang-data xiax-block rc:yang-data "generate" { container generate { description "Contains instructions to `xiax` for how to generate content"; choice generate-type { description "The type of content to generate, and information for how to do so."; container yang-tree-diagram { leaf source { type string; description "The YANG file to generate the tree-diagram from."; } leaf print-yang-data { type empty; } } /*** add more gen-types here ***/ } // end choice gen-type } // end container xiax-block } // end rc:yang-data generate rc:yang-data "validate" { container validate { description "Contains information for how to validate content. Currently just the list of modules and "; choice content-type { description "The type of content to validate, and information for how to do so."; container yang-module { description "Provides information for how to validate the YANG module."; container additional-yang-modules { description "Additional YANG documents that may be needed in order to resolve, e.g., import statements. Do not include the YANG module being validated."; list additional-yang-module { key name; leaf name { type string; } leaf-list uri { type inet:uri; description "Location for where the YANG module is located. Multiple URIs are used to address availability concerns. A copy of files referenced using the 'file' schema is embedded into the xiax-block. A file will only be stored into the xiax-block at most once, in case it referenced by more than one validation."; } } // end list additional-yang-module } // end container additional-yang-modules } // end container yang-module container xml-document { description "Provides information for how to validate a XML document."; choice schema-type { description "Enables the schema-type to be selected."; container using-yang { description "Provides information for how to validate the XML document using YANG."; container yang-modules { list yang-module { key name; leaf name { type string; } leaf-list uri { type inet:uri; description "Location for where the YANG module is located. Multiple URIs are used to address availability concerns. A copy of files referenced using the 'file' schema is embedded into the xiax-block. A file will only be stored into the xiax-block at most once, in case it referenced by more than one validation."; } } // end list yang-module } // end container yang-modules } // end container using-yang /*** add other XML-validating schema-types here ***/ } // end choice schema-type container additional-xml-documents { description "Additional XML documents that may be needed in order to resolve, e.g., data references. Do not include the XML document being validated."; list additional-xml-document { key name; leaf name { type string; } leaf-list uri { type inet:uri; description "Location for where the XML document is located. Multiple URIs are used to address availability concerns. A copy of files referenced using the 'file' schema is embedded into the xiax-block. A file will only be stored into the xiax-block at most once, in case it referenced by more than one validation."; } } // end additional-xml-document } // end container additional-xml-documents } // end container xml-document /*** add content-types here ***/ } // end choice content-type } // end container validate } // end rc:yang-data validate } // end module xiax-structures-v1 ]]>

The following is a snippet of the xiax-block used in this draft. Thie example illustrates three inclusions: A "xiax:src" attribute. A "xiax:gen" attribute, including the gen-file itself. Both "xiax:src" and "xiax:val" attributes, including the val-file itself.

back/section[1]/section[1]/t[3]/figure/artwork hello.txt back/section[1]/section[3]/t[3]/figure/artwork xiax/gen-foo-tree-diagram@2019-02-25.xml foo@2019-02-25.yang back/section[1]/section[5]/t[4]/figure/artwork examples/ex-foo.xml xiax/val-xml-ex-foo@2019-02-25.xml foo@2019-02-25.yang foo@2019-02-25.yang ]]>

This example shows what the "gen" file for generating a YANG tree diagram looks like.

xiax-block-v1@YYYY-MM-DD.yang ]]>

This example shows what the "val" file for validating a YANG module looks like.

ietf-restconf@2017-01-26.yang https://raw.githubusercontent.com/YangModels/yang/master/standard/ietf/RFC/ietf-restconf%402017-01-26.yang ]]>

This example shows what the "val" file for validating an XML document looks like.

xiax-block-v1.yang ./xiax-block-v1.yang ietf-restconf@2017-01-26.yang https://raw.githubusercontent.com/YangModels/yang/master/standard/ietf/RFC/ietf-restconf%402017-01-26.yang ]]>