]> Juniper Networks

tony.li@tony.li

MPLS Working Group Equal Cost Multi-Path (ECMP) forwarding is an essential function in distributing traffic across parallel paths. Packets within a flow must be kept on a single path to avoid reordering, while different flows must be distributed across paths to achieve parallelism. Previously, MPLS has addressed this through the use of an entropy label, providing up to 20 bits of entropy that can be added to the label stack to distinguish different flows. With the interest in MPLS Network Actions, there are proposals to embedding entropy into alternate structures, so it is an appropriate time to consider how many bits should be used for entropy in the future. In this document, we examine the question of how to provide adequate entropy through a simple stochastic simulation. This is not intended to be a comprehensive and extensive treatise, but rather a simple investigation to build intuition into the issues.

Introduction Equal Cost Multi-Path (ECMP) forwarding is an essential function in distributing traffic across parallel paths. Packets within a flow must be kept on a single path to avoid reordering, while different flows must be distributed across paths to achieve parallelism. Previously, MPLS has addressed this through the use of an entropy label, providing up to 20 bits of entropy that can be added to the label stack to distinguish different flows. With the interest in MPLS Network Actions, there are proposals to embedding entropy into alternate structures, so it is an appropriate time to consider how many bits should be used for entropy in the future. In this document, we examine the question of how to provide adequate entropy through a simple stochastic simulation. This is not intended to be a comprehensive and extensive treatise, but rather a simple investigation to build intuition into the issues.

Model In a typical ECMP situation, a Label Edge Router (LER) would examine incoming traffic and use values in the incoming packet to compute an Entropy Label (EL). This would be placed in the label stack. Subsequent routers on the path could use this EL as input to a hashing function and the resulting hash value would then be used to select one of the possible output paths. If the entropy label and the hashing function are implemented correctly, packets would be uniformly distributed across the possible paths. If packet sizes are uniformly distributed, this would also imply that bandwidth would be uniformly distributed across paths. Moving forward, we expect that entropy will continue to be carried in the label stack, but may not occupy a Label field inside of a Label Stack Entry (LSE) . We generalize this to an Entropy Value (EV) and seek to understand the necessary size of this field.

Methodology To study this, we employ a pseudo-random number generator to create Entropy Values and then we apply a hashing function to them. Ideally, the results of the hash will be uniformly distributed. Error metrics are used to characterize how far results are from this ideal.

Randomness Real hardware does not use a random number generator to create entropy. Doing so would require mapping each flow to an entropy value and this would be painful to implement due to scale. Rather, implementations that we are familiar with use a hashing function on the packet header to create an entropy value. Ideally, the results of this would be uniformly distributed. However, we cannot assume that all traffic will be quite so cooperative. Conversely, we also cannot assume that the LER will generate no entropy. If an LER does nothing except place a constant in the entropy value, no downstream hashing function will be able to extract any entropy. To model this, we intentionally choose an imperfect random number generator, specifically, we choose one that has a normal distribution (i.e., Gaussian or bell curve). This should produce many different entropy values and if the hashing function is doing its job, the resulting entropy should result in a uniform distribution of hash values.

Hashing Similarly, there is no standardized way of implementing the hash function that downstream routers will use to convert an entropy value into a hash value. In cases where the entropy value has more bits than the hash value can support, it is up to the hashing function to fold the entropy from all of the entropy value bits into the hash value. Thus, we consider a number of hashing functions: Addition: High order bits are added to the low order bits to produce the hash value. Masking: High order bits are simply discarded. Mixing and Masking: Bits are shifted, added, and then masked. Xor: High order bits are XOR'ed into the low order bits. CRC-CCITT: Compute a CRC over the entropy value and XOR the resulting bits together. MD4: Compute the MD4 digest over the entropy value and XOR the resulting bits together. MD4: Compute the MD5 digest over the entropy value and XOR the resulting bits together. SHA1: Compute the SHA1 digest over the entropy value and XOR the resulting bits together. SHA256: Compute the SHA256 digest over the entropy value and XOR the resulting bits together. Some of these hash functions work better than others. Additional hash functions could easily be added. To simplify our simulation, we have primarily focused on hashes that have a range that is a power of two. However, in a practical application, that is not appropriate and the range must be an arbitrary integer, so that we can support n-way ECMP. This is discussed further below.

Future Hardware Considerations When ECMP was first implemented, 4-way ECMP was the norm. In the early days of sparse networks, that was frequently adequate. As the number of interfaces on a router has grown, ECMP has followed suit, and 128-way and 256-way ECMP is now in production. While hardware has provided an impetetus to grow the breadth of ECMP, it is also suggesting that there is an upper bound to what needs to be supported. It is fairly clear that with current networking trends, we will never need more paths from a single router than it has physical interfaces. While logically, more would be possible, the growth in bandwidth has driven the need for ECMP and that bandwidth is most easily provided through the use of non-fractional interfaces. Concurrently, we note a trend away from building multi-chassis routers with an arbitrary number of physical interfaces, so this implies a bound of around 1000 physical interface per router for the foreseable future. Hardware is also most conveniently impelemented on power of two boundaries. Thus, we investigate ECMP from 4-way to 4096-way, in powers of two.

Results For hash values from 8 to 12 bits, we consider entropy values with 0 to 5 additional bits (i.e., 8 to 17 bits). We generate enough random samples so that each hash bucket should ideally have 1000 entries. The error for each has bucket is then difference between the number of hash results and 1000. We then compute the following error metrics for each combination of algorithm, hash value bits, and additional bits: Root Mean Square Error (RMSE): The Root Mean Square Error is the square root of the sum of the squares of the errors for each bucket, divided by the number of buckets. Mean Average Error (MAE): The Mean Average Error is the arithmetic mean of the absolute value of the error for each bucket. Mean Average Percentage Error (MAPE): The Mean Average Percentage Error is the arithmetic mean of the percentage of error for each bucket. We then summarize the results based on the number of additional bits and on algorithm, computing the mean of each of the above metrics.

Conclusion The arguments in suggest that 4096-way ECMP will be sufficient for some time to come. This suggests that a base of 12 bits would be a good start. We arbitrarily select a MAPE of 5.0 as a threshold of acceptability. This represents a distribution that is within 5% of our goal of a uniform distribution. A solution to the issue would be a number of bits entropy value and a set of hashing algorithms that would provide acceptable performance over the full range of hash values, from 2 to 12 bits. We observe that for some entropy values, we get unacceptable solutions across all algorithms:

Clearly, a 2 bit entropy value is not acceptable. We also observe that some algorithms perform better than others:

Understanding why some algorithms perform poorly is not understood at this time and a subject for future research. Once we reach about 10 bits of entropy value, most algorithms work well, most of the time:

If we look at summaries by additional bits and by algorithm we see:

This leads us to propose that use at least 4 additional bits for our entropy value, resulting in total of 16 (12 + 4). Please note that this is a minimum. Most proposals are proposing higher numbers of bits, thus exceeding this minimum.

Validation To further validate that 16 bits of entropy is sufficient, we also simulated hashing with a non-power of two result to support n-way ECMP. To map the hash to a range of n possible results, we performed a 'mod n' operation after each of the above hashes. We then selected 10 random n values that are not powers of two and repeated the above experiment. In this step, we generated 16 bits of entropy, hashed that and then folded it to down to 16 bits. We then applied the 'mod n' operation to get down to a range of n results. In these circumstances, the 'add', 'mask', and 'xor' hashes are all identical and result in no changes, as they simply are the identity function on 16 bits. The 'add' and 'xor' functions were removed for efficiency. The results demonstrate that the 'mask' (identity) function performs adequately in this circumstance, again supporting the claim that 16 bits is sufficient entropy.

Security Considerations No security issues are discussed in this document.

IANA Considerations This document makes no requests of IANA.

&RFC6790; &RFC4221; &RFC3032; &I-D.bocci-mpls-miad-adi-requirements; &I-D.andersson-mpls-mna-fwk;

Source Code

> self.bits) return v & self.lowmask class MaskHash(HashFunc): def __init__(self, p): super().__init__(p, 'mask') def hashfunc(self, v): return v & self.lowmask class XorHash(HashFunc): def __init__(self, p): super().__init__(p, 'xor') def hashfunc(self, v): while self.highmask & v: v = (v & self.lowmask) ^ ((v & self.highmask) >> self.bits) return v class CrcHash(HashFunc): # CRC-CCITT def __init__(self, p): super().__init__(p, 'crc-ccitt') def hashfunc(self, v): v = binascii.crc_hqx(v.to_bytes(4, 'little'), 0) while self.highmask & v: v = (v & self.lowmask) ^ ((v & self.highmask) >> self.bits) return v class Md4Hash(HashFunc): def __init__(self, p): super().__init__(p, 'md4') self.func = hashlib.new('md4') def hashfunc(self, v): m = self.func.copy() m.update(v.to_bytes(4, 'little')) d = m.digest() v = int.from_bytes(d, 'little') while self.highmask & v: v = (v & self.lowmask) ^ ((v & self.highmask) >> self.bits) return v class Md5Hash(HashFunc): def __init__(self, p): super().__init__(p, 'md5') self.func = hashlib.md5() def hashfunc(self, v): m = self.func.copy() m.update(v.to_bytes(4, 'little')) d = m.digest() v = int.from_bytes(d, 'little') while self.highmask & v: v = (v & self.lowmask) ^ ((v & self.highmask) >> self.bits) return v class Sha1Hash(HashFunc): def __init__(self, p): super().__init__(p, 'sha1') self.func = hashlib.sha1() def hashfunc(self, v): m = self.func.copy() m.update(v.to_bytes(4, 'little')) d = m.digest() v = int.from_bytes(d, 'little') while self.highmask & v: v = (v & self.lowmask) ^ ((v & self.highmask) >> self.bits) return v class Sha256Hash(HashFunc): def __init__(self, p): super().__init__(p, 'sha256') self.func = hashlib.sha256() def hashfunc(self, v): m = self.func.copy() m.update(v.to_bytes(4, 'little')) d = m.digest() v = int.from_bytes(d, 'little') while self.highmask & v: v = (v & self.lowmask) ^ ((v & self.highmask) >> self.bits) return v def rmse( error ): # Compute the root mean square error of this run. errsq = [ err ** 2 for err in error ] rmse = math.sqrt( sum( errsq ) / len( error ) ) return rmse def mae( error ): # Compute the mean average error of this run. mae = sum( [ abs( err ) for err in error ] ) / len( error ) return mae def mape( error ): # Compute the mean average percentage error of this run. mape = 0.0 for ( ind, err ) in enumerate( error ): if err != 0.0: mape += abs( err / DEPTH) mape = mape / len( error ) return mape * 100.0 def mean(vec): return sum(vec)/len(vec) def samples(buckets, bits): # Uniform distribution #samples = random.randint(0, (2 ** bits), buckets * DEPTH) # Normal distribution # This emulates a 'poor' random number generator floats = random.normal(2 ** (bits - 1), scale=2 ** (bits - 3), size=(buckets * DEPTH)) rounded = round(floats) samples = [ int(x) for x in rounded ] samples = [ x if x > 0 else -x for x in samples ] samples = [ x if x < 2 ** bits else (2 ** bits - 1) for x in samples ] # Debugging # print (samples) # #count, bins, ignored = plt.hist(samples, bins = 2 ** bits) #plt.title('Random samples using %d bits of entropy' % # bits) #plt.show() # return samples headers = ['Hash', 'RMSE', 'MAE', 'MAPE'] summary_headers = ['Added bits', 'MRMSE', 'MMAE', 'MMAPE'] algo_summary_headers = ['Algorithm', 'MRMSE', 'MMAE', 'MMAPE'] def power_of_two(): def simulate(p, bits): total = p + bits entropy = samples(2 ** p, total) hashes = [ AddHash(p), MaskHash(p), XorHash(p), CrcHash(p), Md4Hash(p), Md5Hash(p), Sha1Hash(p), Sha256Hash(p) ] results = [] for h in hashes: hashvals = [ h.hashfunc(x) for x in entropy ] # print(hashvals) buckets = [ 0 ] * 2 ** p for x in hashvals: buckets[x] += 1 # print(h.name, buckets, sum(buckets)) error = [ DEPTH - x for x in buckets ] # print(error) # print(rmse(error), mae(error), mape(error)) results.append( [ h.name, FLT % rmse(error), FLT % mae(error), FLT % mape(error) ] ) return results summary = [] per_algo = {} for more in MORE_BITS_RANGE: per_bit = [] for p in P_RANGE: print( 'For %d-way ECMP with %d more bits of entropy:' % (2 ** p, more)) results = simulate(p, p + more) print(tabulate(results, headers=headers, tablefmt='pretty', stralign='right')) rmsevec = [ float(x[1]) for x in results ] maevec = [ float(x[2]) for x in results ] mapevec = [ float(x[3]) for x in results ] per_bit.append([FLT % mean(rmsevec), FLT % mean(maevec), FLT % mean(mapevec)]) print() for result in results: if not per_algo.get(result[0]): per_algo[result[0]] = [ [], [], [] ] for i in range(1,4): per_algo[result[0]][i-1].append(float(result[i])) rmsevec = [ float(x[0]) for x in per_bit ] maevec = [ float(x[1]) for x in per_bit ] mapevec = [ float(x[2]) for x in per_bit ] summary.append([more, FLT % mean(rmsevec), FLT % mean(maevec), FLT % mean(mapevec)]) print('Error per added bit') print(tabulate(summary, headers=summary_headers, tablefmt='pretty', stralign='right')) algo_summary = [] for algo in per_algo: algo_summary.append([algo, FLT % mean(per_algo[algo][0]), FLT % mean(per_algo[algo][1]), FLT % mean(per_algo[algo][2])]) print('\nError per algorithm') print(tabulate(algo_summary, headers=algo_summary_headers, tablefmt='pretty', stralign='right')) def multibit(): # Analyze hashes for non-power-of-two ranges. ENTROPY_BITS = 16 SAMPLES = 10 # For this case, Add, Xor, and Mask are equivalent: a NOP hashes = [ #AddHash(ENTROPY_BITS), XorHash(ENTROPY_BITS), MaskHash(ENTROPY_BITS), CrcHash(ENTROPY_BITS), Md4Hash(ENTROPY_BITS), Md5Hash(ENTROPY_BITS), Sha1Hash(ENTROPY_BITS), Sha256Hash(ENTROPY_BITS) ] def waygen(): start = 2 ** min(P_RANGE) stop = 2 ** max(P_RANGE) for x in range(start, stop): # Don't bother with powers of two again. if x & (x - 1) != 0: yield x def simulate(nways): entropy = samples(nways, ENTROPY_BITS) results = [] for h in hashes: hashvals = [ h.hashfunc(x) for x in entropy ] # print(hashvals) buckets = [ 0 ] * nways for x in hashvals: buckets[x % nways] += 1 #print(h.name, buckets, sum(buckets)) error = [ DEPTH - x for x in buckets ] # print(error) results.append( [ h.name, FLT % rmse(error), FLT % mae(error), FLT % mape(error) ] ) return results per_way = [] nways = [ x for x in waygen() ] for nway in random.choice(nways, size=SAMPLES, replace=False): results = simulate(nway) print('\nFor %d-way ECMP, %d bits of entropy:' % (nway, ENTROPY_BITS)) print(tabulate(results, headers=headers, tablefmt='pretty', stralign='right')) # print('%d-way' % nway) rmsevec = [ float(x[1]) for x in results ] maevec = [ float(x[2]) for x in results ] mapevec = [ float(x[3]) for x in results ] per_way.append([nway, FLT % mean(rmsevec), FLT % mean(maevec), FLT % mean(mapevec)]) summary_headers = ['n-ways', 'MRMSE', 'MMAE', 'MMAPE'] print('\nFor n-way ECMP, with %d bits of entropy, ' 'across hash functions' % ENTROPY_BITS) print(tabulate(per_way, headers=summary_headers, tablefmt='pretty', stralign='right')) power_of_two() multibit() ]]>

Full results