]> Google LLC

pmeenan@google.com

Google LLC

yoavweiss@google.com

Applications and Real-Time HTTP compression dictionary shared brotli zstandard dictionary This specification defines a mechanism for using designated responses as an external dictionary for future HTTP responses for compression schemes that support using external dictionaries (e.g. Brotli and Zstandard ). About This Document The latest revision of this draft can be found at . Status information for this document may be found at . Discussion of this document takes place on the HTTP Working Group mailing list (), which is archived at . Source for this draft and an issue tracker can be found at .

Introduction This specification defines a mechanism for using designated responses as an external dictionary for future HTTP responses for compression schemes that support using external dictionaries (e.g. Brotli and Zstandard ). This document describes the HTTP headers used for negotiating dictionary usage and registers media types for content encoding Brotli and Zstandard using a negotiated dictionary. This document uses the line folding strategies described in .

Dictionary Negotiation

Use-As-Dictionary When responding to a HTTP Request, a server can advertise that the response can be used as a dictionary for future requests for URLs that match the pattern specified in the Use-As-Dictionary response header. The Use-As-Dictionary response header is a Structured Field sf-dictionary with values for "match", "ttl", "type" and "hashes".

match The "match" value of the Use-As-Dictionary header is a sf-string value that provides an URL-matching pattern for requests where the dictionary can be used. The sf-string is parsed as a URL , and supports absolute URLs as well as relative URLs. When stored, any relative URLs MUST be expanded so that only absolute URL patterns are used for matching against requests. The match URL supports using * as a wildcard within the match string for pattern-matching multiple URLs. URLs with a natural * in them are not directly supported unless they can rely on the behavior of * matching an arbitrary string. The of the URL in the "match" pattern MUST be the same as the origin of the request that specifies the "Use-As-Dictionary" response and MUST not include a * wildcard. The "match" value is required and MUST be included in the Use-As-Dictionary sf-dictionary for the dictionary to be considered valid.

ttl The "ttl" value of the Use-As-Dictionary header is a sf-integer value that provides the time in seconds that the dictionary is valid for (time to live). The "ttl" is independent of the cache lifetime of the resource being used for the dictionary. If the underlying resource is evicted from cache then it is also removed but this allows for setting an explicit time to live for use as a dictionary independent of the underlying resource in cache. Expired resources can still be useful as dictionaries while they are in cache and can be used for fetching updates of the expired resource. It can also be useful to artificially limit the life of a dictionary in cases where the dictionary is updated frequently which can help limit the number of possible incoming dictionary variations. The "ttl" value is optional and defaults to 31536000 (1 year).

type The "type" value of the Use-As-Dictionary header is a sf-string value that describes the file format of the supplied dictionary. "raw" is the only defined dictionary format which represents an unformatted blob of bytes suitable for any compression scheme to use. If a client receives a dictionary with a type that it does not understand, it MUST NOT use the dictionary. The "type" value is optional and defaults to "raw".

hashes The "hashes" value of the Use-As-Dictionary header is a inner-list value that provides a list of supported hash algorithms in order of server preference. The dictionaries are identified by the hash of their contents and this value allows for negotiation of the algorithm to use. The "hashes" value is optional and defaults to (sha-256).

Examples

Path Prefix A response that contained a response header: Would specify matching any URL with a path prefix of /product/ on the same as the original request, expiring as a dictionary in 7 days independent of the cache lifetime of the resource, and advertise support for both sha-256 and sha-512 hash algorithms.

Versioned Directories A response that contained a response header: Would match main.js in any directory under /app/, expiring as a dictionary in one year and support using the sha-256 hash algorithm.

Sec-Available-Dictionary When a HTTP client makes a request for a resource for which it has an appropriate dictionary, it can add a "Sec-Available-Dictionary" request header to the request to indicate to the server that it has a dictionary available to use for compression. The "Sec-Available-Dictionary" request header is a lowercase Base16-encoded hash of the contents of a single available dictionary calculated using one of the algorithms advertised as being supported by the server. Its syntax is defined by the following : The client MUST only send a single "Sec-Available-Dictionary" request header with a single hash value for the best available match that it has available. For example:

Dictionary freshness requirement To be considered as a match, the dictionary must not yet be expired as a dictionary. When iterating through dictionaries looking for a match, the expiration time of the dictionary is calculated by taking the last time the dictionary was written and adding the "ttl" seconds from the "Use-As-Dictionary" response. If the current time is beyond the expiration time of the dictionary, it MUST be ignored.

Dictionary URL matching When a dictionary is stored as a result of a "Use-As-Dictionary" directive, it includes a "match" string with the URL pattern of request URLs that the dictionary can be used for. When comparing request URLs to the available dictionary match patterns, the comparison should account for the * wildcard when matching against request URLs. This can be accomplished with the following algorithm which returns TRUE for a successful match and FALSE for no-match:

1. Let MATCH represent the absolute URL pattern from the "match" value for the given dictionary.
2. LET URL represent the request URL being checked.
3. If there are no * characters in MATCH:
   • If the MATCH and URL strings are identical, return TRUE.
   • Else, return FALSE.
4. If there is a single * character in MATCH and it is at the end of the string:
   • If the MATCH string is identical to the start of the URL string, return TRUE.
   • Else, return FALSE.
5. Split the MATCH string by the * character into an array of MATCHES (excluding the * deliminator from the individual entries).
6. If there is not a * character at the end of MATCH:
   • Pop the last entry in MATCHES from the end of the array into PATTERN.
   • If PATTERN is identical to the end of the URL string, remove the end of the URL string to the beginning of the match to PATTERN.
   • Else, return FALSE.
7. Pop the first entry in MATCHES from the front of the array into PATTERN.
   • If PATTERN is not identical to the start of the URL string, return FALSE.
8. Pop each entry off of the front of the MATCHES array into PATTERN. For each PATTERN, in order:
   • Search for the first match of PATTERN in URL, starting from the position of the end of the previous match.
   • If no match is found, return FALSE.
9. Return TRUE.

Multiple matching dictionaries When there are multiple dictionaries that match a given request URL, the client MUST pick the dictionary with the longest match pattern string length.

Negotiating the compression algorithm When a compression dictionary is available for use for a given request, the algorithm to be used is negotiated through the regular mechanism for negotiating content encoding in HTTP. This document introduces two new content encoding algorithms:

Content-Encoding	Description
br-d	Brotli using an external compression dictionary
zstd-d	Zstandard using an external compression dictionary

The dictionary to use is negotiated separately and advertised in the "Sec-Available-Dictionary" request header.

Accept-Encoding The client adds the algorithms that it supports to the "Accept-Encoding" request header. e.g.:

Content-Encoding If a server supports one of the dictionary algorithms advertised by the client and chooses to compress the content of the response using the dictionary that the client has advertised then it sets the "Content-Encoding" response header to the appropriate value for the algorithm selected. e.g.: If the response is cacheable, it MUST include a "Vary" header to prevent caches serving dictionary-compressed resources to clients that don't support them or serving the response compressed with the wrong dictionary:

IANA Considerations

Content Encoding IANA is asked to update the "HTTP Content Coding Registry" registry () according to the table below:

Name	Description	Reference
br-d	A stream of bytes compressed using the Brotli protocol with an external dictionary
zstd-d	A stream of bytes compressed using the Zstandard protocol with an external dictionary

Header Field Registration IANA is asked to update the "Hypertext Transfer Protocol (HTTP) Field Name Registry" registry () according to the table below:

Field Name	Status	Reference
Use-As-Dictionary	permanent	of this document
Sec-Available-Dictionary	permanent	of this document

Compatibility Considerations To minimize the risk of middle-boxes incorrectly processing dictionary-compressed responses, compression dictionary transport MUST only be used in secure contexts (HTTPS).

Security Considerations The security considerations for Brotli and Zstandard apply to the dictionary-based versions of the respective algorithms.

Changing content The dictionary must be treated with the same security precautions as the content, because a change to the dictionary can result in a change to the decompressed content.

Reading content The CRIME attack shows that it's a bad idea to compress data from mixed (e.g. public and private) sources -- the data sources include not only the compressed data but also the dictionaries. For example, if you compress secret cookies using a public-data-only dictionary, you still leak information about the cookies. Not only can the dictionary reveal information about the compressed data, but vice versa, data compressed with the dictionary can reveal the contents of the dictionary when an adversary can control parts of data to compress and see the compressed size. On the other hand, if the adversary can control the dictionary, the adversary can learn information about the compressed data.

Security Mitigations If any of the mitigations do not pass, the client MUST drop the response and return an error.

Cross-origin protection To make sure that a dictionary can only impact content from the same origin where the dictionary was served, the "match" pattern used for matching a dictionary to requests MUST be for the same origin that the dictionary is served from.

Response readability For clients, like web browsers, that provide additional protection against the readability of the payload of a response and against user tracking, additional protections MUST be taken to make sure that the use of dictionary-based compression does not reveal information that would not otherwise be available. In these cases, dictionary compression MUST only be used when both the dictionary and the compressed response are fully readable by the client. In browser terms, that means that both are either same-origin to the context they are being fetched from or that the response is cross-origin and passes the CORS check (https://fetch.spec.whatwg.org/#cors-check).

Same-Origin On the client-side, same-origin determination is defined in the fetch spec (https://html.spec.whatwg.org/multipage/browsers.html#origin). On the server-side, a request with a "Sec-Fetch-Site:" request header with a value of "same-origin" is to be considered a same-origin request.

• For any request that is same-origin:
  • Response MAY be used as a dictionary.
  • Response MAY be compressed by an available dictionary.

Cross-Origin For requests that are not same-origin (), the "mode" of the request can be used to determine the readability of the response. For clients that conform to the fetch spec, the mode of the request is stored in the RequestMode attribute of the request (https://fetch.spec.whatwg.org/#requestmode). For servers responding to clients that expose the request mode information, the value of the mode is sent in the "Sec-Fetch-Mode" request header. If a "Sec-Fetch-Mode" request header is not present, the server SHOULD allow for the dictionary compression to be used.

1. If the mode is "navigate" or "same-origin":
   • Response MAY be used as a dictionary.
   • Response MAY be compressed by an available dictionary.
2. If the mode is "cors":
   • For clients, apply the CORS check from the fetch spec (https://fetch.spec.whatwg.org/#cors-check) which includes credentials checking restrictions that may not be possible to check on the server.
     • If the CORS check passes:
       • Response MAY be used as a dictionary.
       • Response MAY be compressed by an available dictionary.
     • Else:
       • Response MUST NOT be used as a dictionary.
       • Response MUST NOT be compressed by an available dictionary.
   • For servers:
     • If the response does not include an "Access-Control-Allow-Origin" response header:
       • Response MUST NOT be used as a dictionary.
       • Response MUST NOT be compressed by an available dictionary.
     • If the request does not include an "Origin" request header:
       • Response MUST NOT be used as a dictionary.
       • Response MUST NOT be compressed by an available dictionary.
     • If the value of the "Access-Control-Allow-Origin" response header is "*":
       • Response MAY be used as a dictionary.
       • Response MAY be compressed by an available dictionary.
     • If the value of the "Access-Control-Allow-Origin" response header matches the value of the "Origin" request header:
       • Response MAY be used as a dictionary.
       • Response MAY be compressed by an available dictionary.
3. If the mode is any other value (including "no-cors"):
   • Response MUST NOT be used as a dictionary.
   • Response MUST NOT be compressed by an available dictionary.

Privacy Considerations Since dictionaries are advertised in future requests using the hash of the content of the dictionary, it is possible to abuse the dictionary to turn it into a tracking cookie. To mitigate any additional tracking concerns, clients MUST treat dictionaries in the same way that they treat cookies. This includes partitioning the storage as cookies are partitioned as well as clearing the dictionaries whenever cookies are cleared.

References Normative References This document defines two strategies for handling long lines in width-bounded text content. One strategy, called the "single backslash" strategy, is based on the historical use of a single backslash ('\') character to indicate where line-folding has occurred, with the continuation occurring with the first character that is not a space character (' ') on the next line. The second strategy, called the "double backslash" strategy, extends the first strategy by adding a second backslash character to identify where the continuation begins and is thereby able to handle cases not supported by the first strategy. Both strategies use a self-describing header enabling automated reconstitution of the original content. The Hypertext Transfer Protocol (HTTP) is a stateless application-level protocol for distributed, collaborative, hypertext information systems. This document describes the overall architecture of HTTP, establishes common terminology, and defines aspects of the protocol that are shared by all versions. In this definition are core protocol elements, extensibility mechanisms, and the "http" and "https" Uniform Resource Identifier (URI) schemes. This document updates RFC 3864 and obsoletes RFCs 2818, 7231, 7232, 7233, 7235, 7538, 7615, 7694, and portions of 7230. Informative References The Hypertext Transfer Protocol (HTTP) is a stateless application-level protocol for distributed, collaborative, hypertext information systems. This document provides an overview of HTTP architecture and its associated terminology, defines the "http" and "https" Uniform Resource Identifier (URI) schemes, defines the HTTP/1.1 message syntax and parsing requirements, and describes related security concerns for implementations. This document defines the concept of an "origin", which is often used as the scope of authority or privilege by user agents. Typically, user agents isolate content retrieved from different origins to prevent malicious web site operators from interfering with the operation of benign web sites. In addition to outlining the principles that underlie the concept of origin, this document details how to determine the origin of a URI and how to serialize an origin into a string. It also defines an HTTP header field, named "Origin", that indicates which origins are associated with an HTTP request. [STANDARDS-TRACK] Internet technical specifications often need to define a formal syntax. Over the years, a modified version of Backus-Naur Form (BNF), called Augmented BNF (ABNF), has been popular among many Internet specifications. The current specification documents ABNF. It balances compactness and simplicity with reasonable representational power. The differences between standard BNF and ABNF involve naming rules, repetition, alternatives, order-independence, and value ranges. This specification also supplies additional rule definitions and encoding for a core lexical analyzer of the type common to several Internet specifications. [STANDARDS-TRACK] A Uniform Resource Identifier (URI) is a compact sequence of characters that identifies an abstract or physical resource. This specification defines the generic URI syntax and a process for resolving URI references that might be in relative form, along with guidelines and security considerations for the use of URIs on the Internet. The URI syntax defines a grammar that is a superset of all valid URIs, allowing an implementation to parse the common components of a URI reference without knowing the scheme-specific requirements of every possible identifier. This specification does not define a generative grammar for URIs; that task is performed by the individual specifications of each URI scheme. [STANDARDS-TRACK] This document describes a set of data types and associated algorithms that are intended to make it easier and safer to define and handle HTTP header and trailer fields, known as "Structured Fields", "Structured Headers", or "Structured Trailers". It is intended for use by specifications of new HTTP fields that wish to use a common syntax that is more restrictive than traditional HTTP field values. This document describes the commonly used base 64, base 32, and base 16 encoding schemes. It also discusses the use of line-feeds in encoded data, use of padding in encoded data, use of non-alphabet characters in encoded data, use of different encoding alphabets, and canonical encodings. [STANDARDS-TRACK] This specification defines a lossless compressed data format that compresses data using a combination of the LZ77 algorithm and Huffman coding, with efficiency comparable to the best currently available general-purpose compression methods. Zstandard, or "zstd" (pronounced "zee standard"), is a lossless data compression mechanism. This document describes the mechanism and registers a media type, content encoding, and a structured syntax suffix to be used when transporting zstd-compressed content via MIME. Despite use of the word "standard" as part of Zstandard, readers are advised that this document is not an Internet Standards Track specification; it is being published for informational purposes only. This document replaces and obsoletes RFC 8478.