Structured and Dynamic Email

Introduction Email can be considered a successful technology, based on its broad adoption and based on the fact dozens of email-related RFCs have been written over time . Despite of these various RFCs, which are dealing with a multitude of small improvements, the major inner workings of email remain widely unchanged since their inception. In particular, email is still mostly a text-based medium, which requires a lot of human attention, even though more and more so called "transactional" emails are authored in an automated fashion. Accordingly, the task of dealing with large amounts of email is still a major challenge for many users. Most existing attempts that have been made to improve this situation focus on particular use cases. The main point of this paper is to suggest that it might be worthwhile to consider if there exists a generalizable core underneath those use cases. Such a more streamlined approach might help to condense specifications and to increase adoption by email servers and clients. Furthermore, we argue that such an approach might enable novel use cases by leveraging the specific and partially unique properties of email technology, which include:

Its ubiquitous and simple approach of addressing
Its asynchronous nature
Its unique role in exchanging data betweeen the public Internet and the private information space of a user
Its established role as a wrapping or transport mechanism for other data
Its widespread support by many tools

Essentially, this entails that a consciously designed approach for the automated processing of formally structured information in emails could be considered a "private API" of email users. While this certainly raises security and trust concerns, the opportunities behind such unifying approach seem worthwhile discussing.

Problem statement

Lack of structured data A large share of today's emails is of transactional nature, i.e., sent by automated agents or processes to human users. While those emails often have relatively clear semantics (such as Invoice or Reservation), the medium of those emails is still human-readable text. Users and their email software cannot leverage knowledge inside and about those emails to make their processing a more efficient and enjoyable task.

Lack of structured interaction Using informal HTML email conventions, such emails can be considered as small websites, adapted and personalized for a certain use case. However, they are mostly a "dead end" for interaction. Due to the lack of explicit semantics in the underlying email, particular actions afforded by transactional mail (such as Approval or Rating) cannot be easily made actionable for users. Also, many interactions require to switch from email to a regular web browser, resulting in a process disruption which is coined Medienbruch in German language.

Current solutions Several attempts have been made to improve this situation. We discuss some of them in the following. Within the IETF, various RFCs have been devised to structure certain types of email interaction. Probably most notable, iMIP allows users to deal with meeting workflows in a structured manner. Further examples structure interaction with message delivery notifications , mailinglist subscriptions or specify header-based semantic indicators for certain domains or Emoji-based semantics of approval/disapproval in mailinglist discussions . In more administrative use cases, special kinds of email body parts are used for abuse reporting or administrative reporting . In a USENET context , so-called "control messages" are defined for what can be considered certain API calls, such as for creating a USENET group. Looking outside the IETF, the "Email Markup" approach by Google allows allows senders to integrate certain Schema.org annotations (such as for hotel reservations or parcel delivery) in their email, such that email clients can provide customized support, including certain easy to use response actions. Email markup is still in use, but has a number of limitations:

Being a proprietary standard owned by Google
It does not support to annotate standard email elements such as headers or attachments
It is supported by only few providers
It is limited to a few fixed use cases
It is only available for senders that went through an approval process
It is not designed for human end users to send structured emails

More recently, AMP email (also initiated by Google) and Microsoft Actionable Messages have been specified. Compared to Email Markup, their focus is less on semantically annotating email content, but on more rich interaction, e.g., allowing the submission of simple forms. In addition, both approaches can include dynamic elements, by retrieving certain data, or even replacing the complete email body at reading time. Both approaches however suffer from similar limitations as those already pointed out for Email Markup. There also seems to be a lack of consensus, since Microsoft Outlook is reported to have removed initial support for AMP email .

Goals To summarize, we see a need for a vendor neutral standard for structured and dynamic email. It should enable email senders to provide structured information about the semantics of their message and allow recipients (may it be human users or their agents) for rich and structured interaction with those emails. Such standard should also take into account the various RFCs specifying email semantics for specific use cases and ideally provide a more generalized framework for such use cases, which can be more consistently and easily implemented by vendors. For example, there exist various "informal" standards for certain types of email messages, that have not yet been standardized in dedicated RFCs, such as out-of-office (aka vacation notice) emails.

Design principles

Protocol-agnostic implementation Structured email should work with existing protocols such as IMAP , SMTP , and novel ones such as JMAP . Accordingly, most extensions would probably realized in the scope of the Internet Message Format or in areas not yet addressed by standardization (see, e.g., , , , ).

Co-existence of approaches A major advantage of email technology is the fact, that email is backwards compatible. For example, most HTML emails are accompanied by a plain text version to be used as a fallback in case an email client cannot deal with HTML. In a similar fashion, structured email could be provided as an additional option for clients able and willing to deal with it, thus avoiding a classic "chicken/egg" problem for novel technology.

Decentral semantics With respect to structured markup, an incremental and decentral approach is proposed. Instead of an authoritative fixed set of schemas, as in the case of Email Markup, users should be allows to create and extend schemas on their own.

Solution approach This draft can and will not yet provide a full specification for the problems stated. In this section, we will single out certain aspects that might be addressed by such specifications.

Email content (body) As described before, an (optional and alternate) structured description of the semantics of email messages is supposed be in the core of this proposal. While existing approaches such as Email Markup offer different options for how to include structured data, an alternative, machine-readable presentation of the message content in addition to human readable text and HTML representations, might be the most natural approach.

Email metadata (header) For the sake of efficient processing and perhaps also for reasons of data privacy, certain structured data about an email might also be captured in its headers. Header fields of certain message parts (such as attachments) might also be enriched by semantic annotation.

Email storage Email messages stored in IMAP are currently immutable. Receivers are not supposed to modify their actual content. Hence, the main means to add and store additional data on the receiving side are:

IMAP flags
IMAP mailboxes (sorting into folders, which can be considered as some sort of tagging)
Custom data storage on server or email client side

As of today, this already yields certain data portability problems - e.g., IMAP flags are often not considered when exporting raw email message files. Accordingly, we see a need for the standardized persistent storage of data which is added by algorithmic processing or by users and their email clients.

Email filtering Once emails contain structured data about their content, this might create a demand for related extensions to the Sieve email filtering language .

Email context Email lives in tight symbiosis with strongly related data such as address books, calendars, or tasks. However, besides specific use cases such as meeting organization, an integration between such data is mainly subject to non-standardized functionality of client software. Notably, email also operates in a wider context such as services which send transactional emails and also applications running on the same Desktop or Mobile device as the email client. Transactional emails are sent by services and organizations which are in a relation to the user. Further related data may be managed in outside applications or would be a candidate for being managed within email client software. From a broader architectural perspective, email can be considered a technology particularly suited to support novel trends in data sovereignty and decentralized data storage, since by design it bridges the public Internet data space with the private data spaces of mailbox users. Structured email will open new opportunities for exchanging data in all those cases. Accordingly, standards may help to enable and organize interoperability.

Discovery In order to allow users to send structured mails, they need to know if and which kind of structured emails a recipient can process. There might be multiple ways of conveying such information:

Headers in emails received from users, which a receiver can store/look up
A DNS/HTTP-based discovery service, which returns structured email capability for a domain or particular sender

An example for the latter could be:

User selects "john@doe.com" as recipient
Email client checks if discovery service is offered by "doe.com"
Email client queries "doe.com" for structured email capabilities
Discovery service at "doe.com" returns a document which states a number of structured data types or actions that are supported by the "john@doe.com" account
The email client will allow the user to optionally select from those structured interaction options and provide an appropriate user experience

HTTP services For dynamic email features, an endpoint must be available to provide updated data for an email currently read by a user. Similar, if a structured email allows for the execution of a certain action or the submission of form data, there needs to be a receiving endpoint in place. For interoperability reasons, those endpoint APIs should be standardized.

Trust Structured and automated interaction certainly raises various security concerns (see ). While not the focus of this current draft, we assume that parts of the proposed approach can also be used on a special "trust layer". E.g., certain structured message exchanges, in combination with signing or encryption could help to establish trust necessary for further structured communication.

Security considerations While it is clear that trust, security, and anti-spam considerations are core to the technology suggested in this document, this aspect is not discussed in depth at this stage in order to reduce complexity. While existing approaches such as Email Markup or Actionable Messages enforce strict control by registration processes, we think that a more open process is better suited for the decentralized character of email. Various trust mechanisms such as DKIM validation, challenge-response emails, or trusted receivers in address books/white lists might be considered as a prerequisite for structured email processing.

IANA Considerations This document has no IANA actions at this time.