]> Universität Bremen TZI

Postfach 330440 Bremen D-28359 Germany +49-421-218-63921 cabo@tzi.org

Textuality

tbray@textuality.com

Internet-Draft This document specifies I-Regexp, a flavor of regular expressions that is limited in scope with the goal of interoperation across many different regular-expression libraries. About This Document Status information for this document may be found at . Discussion of this document takes place on the JSONpath Working Group mailing list (), which is archived at . Source for this draft and an issue tracker can be found at .

Introduction The present specification defines an interoperable regular expression flavor, I-Regexp. This document uses the abbreviation "regexp" for what are usually called regular expressions in programming. "I-Regexp" is used as a noun meaning a character string which conforms to the requirements in this specification; the plural is "I-Regexps". I-Regexp does not provide advanced regexp features such as capture groups, lookahead, or backreferences. It supports only a Boolean matching capability, i.e., testing whether a given regexp matches a given piece of text. I-Regexp is a subset of XSD regexps . This document includes rules for converting I-Regexps for use with several well-known regexp libraries.

Terminology The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Requirements I-Regexps should handle the vast majority of practical cases where a matching regexp is needed in a data model specification or a query language expression. A brief survey of published RFCs yielded the regexp patterns in Appendix A (with no attempt at completeness). With certain exceptions as discussed there, these should be covered by I-Regexps, both syntactically and with their intended semantics.

I-Regexp Syntax An I-Regexp MUST conform to the ABNF specification in .

I-Regexp Syntax in ABNF ' / %x40-5A ; '@'-'Z' / %x5E-7A ; '^'-'z' / %x7E-10FFFF ) charClass = "." / SingleCharEsc / charClassEsc / charClassExpr SingleCharEsc = "\" ( %x28-2B ; '('-'+' / %x2D-2E ; '-'-'.' / "?" / %x5B-5E ; '['-'^' / %s"n" / %s"r" / %s"t" / %x7B-7D ; '{'-'}' ) charClassEsc = catEsc / complEsc charClassExpr = "[" [ "^" ] ( "-" / CCE1 ) *CCE1 [ "-" ] "]" CCE1 = ( CCchar [ "-" CCchar ] ) / charClassEsc CCchar = ( %x00-2C / %x2E-5A ; '.'-'Z' / %x5E-10FFFF ) / SingleCharEsc catEsc = %s"\p{" charProp "}" complEsc = %s"\P{" charProp "}" charProp = IsCategory / IsBlock IsCategory = Letters / Marks / Numbers / Punctuation / Separators / Symbols / Others Letters = %s"L" [ ( %x6C-6D ; 'l'-'m' / %s"o" / %x74-75 ; 't'-'u' ) ] Marks = %s"M" [ ( %s"c" / %s"e" / %s"n" ) ] Numbers = %s"N" [ ( %s"d" / %s"l" / %s"o" ) ] Punctuation = %s"P" [ ( %x63-66 ; 'c'-'f' / %s"i" / %s"o" / %s"s" ) ] Separators = %s"Z" [ ( %s"l" / %s"p" / %s"s" ) ] Symbols = %s"S" [ ( %s"c" / %s"k" / %s"m" / %s"o" ) ] Others = %s"C" [ ( %s"c" / %s"f" / %x6E-6F ; 'n'-'o' ) ] IsBlock = %s"Is" 1*( "-" / %x30-39 ; '0'-'9' / %x41-5A ; 'A'-'Z' / %x61-7A ; 'a'-'z' ) ]]>

As an additional restriction, charClassExpr is not allowed to match [^], which according to this grammar would parse as a positive character class containing the single character ^. This is essentially XSD regexp without character class subtraction and multi-character escapes. An I-Regexp implementation MUST be a complete implementation of this limited subset. In particular, full Unicode support is REQUIRED; the implementation MUST NOT limit itself to 7- or 8-bit character sets such as ASCII and MUST support the Unicode character property set in character classes.

• Issues: The ABNF has been automatically generated and maybe could use some further polishing. The ABNF has been verified against , but a wider corpus of regular expressions will need to be examined. Note that about a third of the complexity of this ABNF grammar comes from going into details on the Unicode IsCategory classes. Additional complexity stems from the way hyphens can be used inside character classes to denote ranges; the grammar deliberately excludes questionable usage such as /[a-z-A-Z]/.

I-Regexp Semantics This syntax is a subset of that of . Implementations which interpret I-Regexps MUST yield Boolean results as specified in . (See also .)

Mapping I-Regexp to Regexp Dialects (TBD; these mappings need to be further verified in implementation work.)

XSD Regexps Any I-Regexp also is an XSD Regexp , so the mapping is an identity function. Note that a few errata for have been fixed in , which is therefore also included as a normative reference. XSD 1.1 is less widely implemented than XSD 1.0, and implementations of XSD 1.0 are likely to include these bugfixes, so for the intents and purposes of this specification an implementation of XSD 1.0 regexps is equivalent to an implementation of XSD 1.1 regexps.

ECMAScript Regexps Perform the following steps on an I-Regexp to obtain an ECMAScript regexp :

• For any dots (.) outside character classes (first alternative of charClass production): replace dot by [^\n\r].
• Envelope the result in ^ and $.

Note that where a regexp literal is required, this needs to enclose the actual regexp in /. The performance of an ECMAScript matcher can be increased by turning parenthesized regexps (last choice in production atom) into (?:...) constructions.

PCRE, RE2, Ruby Regexps Perform the same steps as in to obtain a valid regexp in PCRE , the Go programming language , and the Ruby programming language, except that the last step is:

• Envelope the result in \A and \z.

Again, the performance can be increased by turning parenthesized regexps (production atom) into (?:...) constructions.

<< Your kind of Regexp here >> (Please submit the mapping needed for your favorite kind of regexp.)

Motivation and Background Data modeling formats (YANG, CDDL) as well as query languages (jsonpath) often need a regular expression (regexp) sublanguage. There are many dialects of regular expressions in use in platforms, programming languages, and data modeling formats. While regular expressions originally were intended to describe a formal language, i.e., to provide a Boolean matching function, they have turned into parsing functions for many applications, with capture groups, greedy/lazy/possessive variants, etc. Language features such as backreferences allow specifying languages that actually are context-free (Chomsky type 2) instead of the regular languages (Chomsky type 3) that regular expressions are named for. YANG () and CDDL () have adopted the regexp language from W3C Schema . XSD regexp is a pure matching language, i.e., XSD regexps can be used to match a string against them and yield a simple true or false result. XSD regexps are not as widely implemented as programming language regexp dialects such as those of Perl, Python, Ruby, Go , or JavaScript (ECMAScript) . The latter are often in a state of continuous development; in the best case (ECMAScript) there is a complete specification which however is highly complex (Section 21.2 of comprises 62 pages) and evolves on a yearly timeline, with significant additions. Regexp dialects such as PCRE have evolved to cover a common set of functions available in parsing regexp dialects, offered in a widely available library. With continuing accretion of complex features, parsing regexp libraries have become susceptible to bugs and performance degradation, in particular those that can be exploited in Denial of Service (DoS) attacks. The library RE2 that is compatible with Go language regexps strives to be immune to DoS attacks, making it attractive to applications such as query languages where an attacker could control the input. The problem remains that other bugs in such libraries can lead to exploitable vulnerabilities; at the time of writing, the Common Vulnerabilities and Exposures (CVE) system has 131 entries that mention the word "regex" (not all, but many of which are such bugs, with 23 matches for arbitrary code execution). Implementations of YANG and CDDL often struggle with providing true XSD regexps; some instead cheat by providing one of the parsing regexp varieties, sometimes without even advertising this fact. A matching regexp that does not use the more complex XSD features () can usually be converted into a parsing regexp of many dialects by simply surrounding it with anchors of that dialect (e.g., ^ or \A and $ or \z). If the original matching regexps exceed the envelope of compatibility between dialects, this can lead to interoperability problems, or, worse, security vulnerabilities. Also, features of the target dialect such as capture groups may be triggered inadvertently, reducing performance.

Subsetting XSD Regexps XSD regexps are relatively easy to implement or map to widely implemented parsing regexp dialects, with a small number of notable exceptions:

• Character class subtraction. This is a very useful feature in many specifications, but it is unfortunately mostly absent from parsing regexp dialects. Discussion: This absence can often be addressed by translating character class subtraction into positive character classes (possibly requiring significant expansion) and/or inserting negative lookahead assertions (which are not universally supported by regexp libraries, most notably not by RE2 ). This specification therefore opts for leaving out character class subtraction.
• Multi-character escapes. \d, \w, \s and their uppercase equivalents (complement classes) exhibit a large amount of variation between regexp flavors. (E.g., predefined character classes such as \w may be meant to be ASCII only, or they may encompass all letters and digits defined in Unicode. The latter is usually of interest in the application of query languages to text in human languages, while the former is of interest to a subset of applications in data model specifications.)
• Unicode. While there is no doubt that a regexp flavor meant to last needs to be Unicode enabled, there are a number of aspects of this that need discussion. Not all regexp implementations that one might want to map I-Regexps into will support accesses to Unicode tables that enable executing on constructs such as \p{IsCoptic}, for mapping into such implementations, translation needs to be provided. Fortunately, the \p/\P feature in general is now quite widely available. Discussion: The ASCII focus can partially be addressed by adding a constraint outside the regexp that the matched text has to be ASCII in the first place. This often is all that is needed where regexps are used to define lexical elements of a computer language. This reduces the size of the Unicode tables required in such a constrained implementation considerably. (In , RFC 6643 contains a lone instance of \p{IsBasicLatin}{0,255}, which is needed to describe a transition from a legacy character set to Unicode. RFC2622 contains [[:digit:]], [[:alpha:]], [[:alnum:]], albeit in a specification for the flex tool; this is intended to be close to \d, \p{L}, \w in an ASCII subset.)

IANA Considerations This document makes no requests of IANA.

Security considerations As discussed in , more complex regexp libraries are likely to contain exploitable bugs leading to crashes and remote code execution. There is also the problem that such libraries often have hard to predict performance characteristics, leading to attack vectors that overload an implementation by matching against an expensive attacked controlled regexp. I-Regexps have been designed to allow implementation in a way that is resilient to both threats; this objective needs to be addressed throughout the implementation effort.

References Normative References In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. Informative References This document proposes a notational convention to express Concise Binary Object Representation (CBOR) data structures (RFC 7049). Its main goal is to provide an easy and unambiguous way to express structures for protocol messages and data formats that use CBOR or JSON. YANG is a data modeling language used to model configuration data, state data, Remote Procedure Calls, and notifications for network management protocols. This document describes the syntax and semantics of version 1.1 of the YANG language. YANG version 1.1 is a maintenance release of the YANG language, addressing ambiguities and defects in the original specification. There are a small number of backward incompatibilities from YANG version 1. This document also specifies the YANG mappings to the Network Configuration Protocol (NETCONF). n.d. n.d. Ecma International n.d. I-JSON (short for "Internet JSON") is a restricted profile of JSON designed to maximize interoperability and increase confidence that software can process it successfully with predictable results.

Regexps and Similar Constructs in Recent Published RFCs This appendix contains a number of regular expressions that have been extracted from some recently published RFCs based on some ad-hoc matching. Multi-line constructions were not included. With the exception of some (often surprisingly dubious) usage of multi-character escapes, all regular expressions validate against the ABNF in .

Example regular expressions extracted from RFCs

The multi-character escapes (MCE) or the character classes built around them used here can be substituted as shown in . Substitutes for multi-character escapes in examples

MCE/class	Substitute class
\S	[^ \t\n\r]
[\S ]	[^\t\n\r]
\d	[0-9]

Note that the semantics of \d in XSD regular expressions is that of \p{Nd}; however, this would include all Unicode characters that are digits in various writing systems and certainly is not actually meant in the RFCs listed.

Acknowledgements This draft has been motivated by the discussion in the IETF JSONPATH WG about whether to include a regexp mechanism into the JSONPath query expression specification, as well as by previous discussions about the YANG pattern and CDDL .regexp features. The basic approach for this draft was inspired by The I-JSON Message Format.