Experience from implementing a new packet discard classification scheme

Experience from implementing a new packet discard classification scheme Amazon

jevanamz@amazon.co.uk

Amazon

opyl@amazon.com

Independent Stream Internet-Draft Router reported packet loss is the primary signal of when a network is not doing its job. Some packet loss is normal or intended in TCP/IP networks, however. To minimise network packet loss through automated network operations we need clear and accurate signals of all packets which are dropped and why. This document describes our experience from implementing a packet loss classification scheme to provide these signals and enable automated network mitigation of unintended packet loss.

Introduction The job of a network is to transport packets. Understanding both where and why packet loss occurs is essential for effective network operation. Router-reported packet loss is the most direct signal for network operations to identify customer impact from unintended packet loss. Accurate accounting of packet loss is not enough, however, as some level of packet loss is normal in TCP/IP networks. In automating network operations, there are only a relatively small number of automated actions that can be taken to mitigate customer impacting packet loss. Precise classification of packet loss signals is important to ensure the right action is taken as taking the wrong action can make problems worse. The existing metrics for packet loss as defined in - namely ifInDiscards, ifOutDiscards, ifInErrors, ifOutErrors - do not provide sufficient precision to be able to automatically identify the cause of the loss and mitigate the impact. From a network operators' perspective, ifindiscards can represent both intended packet loss (i.e. packets discarded due to policy) and unintended packet loss (e.g. packets dropped in error). Further, these definitions are ambiguous, in that vendors can and have implemented them differently. In some implementations, ifinerrors accounts only for errored packets which are dropped, whilst in others it accounts for all errored packets whether they are dropped or not. Many vendors support more discard metrics than these; where they do, they are inconsistently implemented due to an absence of a clearly defined classification scheme and semantics for packet loss reporting. This document describes our experience from implementing a packet loss classification scheme across multiple hardware platforms, which aims to address these issues and enable automated mitigation of unintended packet loss. Section 2 describes the problem. Section 3 defines the classification scheme and the accounting requirements with examples. Section 4 gives examples of discard signal-to-cause-to-auto-mitigation action mapping. Section 5 details our experience from implementing this scheme. The terms 'packet drop' and 'discard' are considered equivalent and are used interchangeably.

Problem statement Working backwards from the goal of auto-mitigation of unintended packet loss, there are only a relative small number of potential auto-mitigation actions, e.g.: Take a device, link or set of devices and/or links out of service Return a device, link or set of devices and/or links back into service Move traffic to another device Roll-back a recent change to a device that might have caused the problem Escalate to a human (e.g. network operators) as a last resort Precise signal of impact is important as taking the wrong action can be worse than taking no action. For example, taking a congested device out of service can make congestion worse by moving the traffic to other already congested links and/or devices. To be able to detect whether router reported packet loss is a problem and determine what actions should be taken to mitigate the impact and remediate the cause, depends on four primary features of the packet loss signal: the cause of the loss the rate and/or degree of the loss the duration of the loss the location of the loss Features 2, 3 and 4 are already addressed with passive monitoring statistics, e.g. obtained with SNMP or NETCONF . Feature 1, however, is dependent on the classification scheme used for packet loss reporting. In the next section we define a new classification scheme to address this problem.

Traffic and Discard Classification Scheme We define the classification scheme as a tree which follows the structure <component><direction><type><layer><sub-type><sub-sub-type><metric>, where:
a. component can be interface|device
b. direction can be ingress|egress
c. type can be traffic|discards, where traffic accounts for packets successfully received or transmitted, and discards account for packet drops
d. layer can be l2|l3

For additional context, Appendix A provides an example of where packets may be dropped in a device.

Discard Class Descriptions discards/error/l2/rx/
Frames dropped because they are invalid at L2, e.g. due to failing CRC or an invalid MAC address. discards/error/l3/rx/
These are drops due to errors in the received packet, i.e. which indicate an upstream problem, rather than a problem with the device that is dropping the errored packets. There are multiple potential errors that can cause a packet to be dropped on receipt, e.g. IP checksum errors, malformed packets. discards/error/l3/ttl_expired
There can also be multiple causes for TTL-exceed drops: i) trace-route; ii) TTL set too low by the end-system; iii) routing loops discards/error/l3/no_route/
Discards due to a packet not matching any route. discards/error/local/
A device may drop packets within its switching pipeline due to internal errors, e.g. parity errors. Any discards not explicitly assigned to the above classes are accounted here. discards/policy/
These are intended discards, i.e. packets dropped due to a configured policy. There are multiple sub-classes. discards/policy/l3/acl/
Discards due to packet matching an access control list (ACL). discards/policy/l3/policer/
Discards due to packet matching a configured policer. discards/policy/l3/null_route/
Discards due to a packet matching a route with discard action. discards/policy/l3/urpf/
Discards due to a packet failing unicast reverse path forwarding (RPF) check. discards/no_buffer/
Discards due to no available buffer to enqueue the packet. These can be tail-drop discards or due to an active queue management algorithm, e.g. RED , CODEL .

Discard Accounting Requirements Requirements 1-10 apply to the packets forwarded by the device, i.e. rather than packets destined to/from the device: All packet receipt, transmission and drops MUST be reported All packet receipt, transmission and drops SHOULD be attributed to the physical or logical interface where they occur. If a frame is discarded at L2, it MUST NOT be accounted for at L3 An individual packet MUST NOT account against both the L2 traffic and L2 discard classes on a single direction, i.e. ingress or egress An individual packet MUST NOT account against both the L3 traffic and L3 discard classes on a single direction, i.e. ingress or egress The aggregate L2 and L3 traffic and discard classes MUST account for all underlying packets received, transmitted and dropped across all other classes The aggregate QOS traffic and discard (no buffer) classes MUST account for all underlying packets received, transmitted and dropped across all other classes In addition to the L2 and L3 aggregate classes, an individual dropped packet MUST only account against a single error, policy or no buffer discard sub class Where there may be multiple drop reasons for a packet, the ordering of discard class reporting MUST be defined If Diffserv quality of service (QOS) is not used, no_buffer discards SHOULD be reported as class0 Traffic from the device control plane SHOULD be accounted for the same as other egress traffic

Examples Assuming all the requirements are met, a good unicast IPv4 packet received would increment:
- interface/ingress/traffic/l3/v4/unicast/packets
- interface/ingress/traffic/l3/v4/unicast/bytes
- interface/ingress/traffic/qos/class_0/packets
- interface/ingress/traffic/qos/class_0/bytes A received unicast IPv6 packet dropped due to TTL expiry would increment:
- interface/ingress/discards/l3/v6/unicast/packets
- interface/ingress/discards/l3/v6/unicast/bytes
- interface/ingress/discards/l3/rx/ttl_expired/packets An IPv4 packet dropped on egress due to no buffers would increment: - interface/egress/discards/l3/v4/unicast/packets
- interface/egress/discards/l3/v4/unicast/bytes
- interface/egress/discards/no_buffer/class_0/packets
- interface/egress/discards/no_buffer/class_0/bytes

A Possible Signal-Cause-Mitigation Mapping Example discard signal-to-cause-to-mitigation mappings are shown in the table below:

Implementation Experience The number and granularity of classes described in section 3 is a compromise between: providing sufficient detail to be able to take the appropriate automated actions whilst: a) not providing too much detail which may require deeper understanding rather than helping to surface the problem quickly; b) constraining the quantity of data produced where these metrics are produced per interface to limit data volume and device CPU impacts. While further granularity is possible, we found the scheme described to be generally sufficient. There are multiple ways that we could have defined the discard classification tree, e.g. we could have used a multi-rooted tree, rooted in each protocol. We opted instead to define a tree where protocol discards and causal discards are accounted orthogonally, as this reduces the number of classes and we found it sufficient to determine mitigation actions. NoBuffer discards can be realised differently with different memory architectures. Whether a NoBuffer discard is attributed to ingress or egress can differ accordingly. For successful auto-mitigation: where the discards are due to egress interface congestion, they should be reported on egress; where the discards are due to device-level congestion (exceeding the device forwarding rate), they should be reported on ingress. Most platforms account for the number of packets where the TTL has expired, and the CPU has returned an ICMP Time Exceeded message. In practise, however, there is often a policer applied to limit the number of packets to the CPU. Implicitly, this limits the rate of TTL discards processed by the CPU and hence it limits the number of discards reported. One method to account for all packets discards due to TTL exceeded, even those that are dropped by a policer when being forwarded to the CPU, is to use accounting of all ingress packets received with TTL=1. Where a no route discard is implemented with a default null route, separate accounting is needed for any explicit null routes configured, in order to differentiate between interface/ingress/discards/policy/null_route/packets and interface/ingress/discards/errors/no_route/packets. It is useful to account separately for transit packets dropped by transit ACLs or policers, and packets dropped by ACLs or policers which limit the number of packets to the device control packets. It is not possible to identify a configuration error - i.e. when intended discards are unintended - with device packet loss metrics alone. For example, to determine if ACL drops are intended or due to a misconfigured ACL some other method is needed, e.g. with configuration validation before deployment or in detecting a significant change in ACL drops after a change compared to before. Where traffic byte counters need to be 64-bit, packet and discard counters which increase at a lower rate may be encoded in fewer bits, e.g. 48-bit. There are multiple ways that the metrics can be exposed from the device, including SNMP and NETCONF , however, we consider this choice to be independent of the defined discard classification scheme.

Security Considerations There are no new security considerations introduced by this document.

IANA Considerations There are no new IANA considerations introduced by this document.

Terminology The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Acknowledgments The content of this draft has benefitted from feedback from JR Rivers, Ronan Waide, Chris DeBruin, Marcoz Sanz, Avinash Kadosh and Nadav Chachmon.

Key words for use in RFCs to Indicate Requirement Levels In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. Management Information Base for Network Management of TCP/IP-based internets: MIB-II This memo defines the second version of the Management Information Base (MIB-II) for use with network management protocols in TCP/IP-based internets. [STANDARDS-TRACK] Simple Network Management Protocol (SNMP) This RFC is a re-release of RFC 1098, with a changed "Status of this Memo" section plus a few minor typographical corrections. This memo defines a simple protocol by which management information for a network element may be inspected or altered by logically remote users. [STANDARDS-TRACK] Network Configuration Protocol (NETCONF) The Network Configuration Protocol (NETCONF) defined in this document provides mechanisms to install, manipulate, and delete the configuration of network devices. It uses an Extensible Markup Language (XML)-based data encoding for the configuration data as well as the protocol messages. The NETCONF protocol operations are realized as remote procedure calls (RPCs). This document obsoletes RFC 4741. [STANDARDS-TRACK] Random Early Detection gateways for Congestion Avoidance An Architecture for Differentiated Services This document defines an architecture for implementing scalable service differentiation in the Internet. This memo provides information for the Internet community. Controlled Delay Active Queue Management This document describes CoDel (Controlled Delay) -- a general framework that controls bufferbloat-generated excess delay in modern networking environments. CoDel consists of an estimator, a setpoint, and a control loop. It requires no configuration in normal Internet deployments.

Where do packets get dropped? The diagram below is an example of where and why packets may be dropped in a typical single ASIC, shared buffered type device, where packets ingress on the left and egress on the right.

Phy +--> Mac +--> Ingress +--> Buffers +--> Egresss +--> Mac +--> Phy |> Packet tx | | | | | Pipeline| | | | Pipeline| | | | | +----------+ +----------+ +----------+ +----------+ +----------+ +----------+ +----------+ Intended policy/acl policy/acl Discards: policy/policer policy/policer policy/urpf null_route Unintended error/rx/l2 error/rx/l3 no_buffer error/tx/l3 Discards: error/local no_route ttl ]]>