sFlow.org Dustin Byford http://www.sflow.org/ Cumulus Networks Inc. info@sflow.org Matt Peterson San Francisco Metropolitan Internet Exchange Steve Joiner Finisar Corp. Peter Phaal InMon Corp. April 2016 sFlow Optical Interface Structures Copyright Notice Copyright (C) sFlow.org (2016). All Rights Reserved. Abstract This memo describes sFlow version 5 structures for exporting statistics from optical interface modules. Table of Contents 1. Overview ...................................................... 1 2. Discussion .................................................... 2 3. sFlow Datagram Extensions ..................................... 3 4. References .................................................... 4 5. Author's Addresses ............................................ 5 1. Overview This document describes additional structures that allow an sFlow agent to export statistics gathered from pluggable optical modules through their management interfaces [1][2][3]. sFlow version 5 is an extensible protocol that allows the addition of new data structures without impacting existing collectors. This document does not change the sFlow version 5 protocol [4], it simply defines additional, optional, data structures that an sFlow agent can Draft sFlow.org [Page 1] Version 0.6 sFlow Optical Interface Structures April 2016 use to report table utilization statistics. 2. Discussion Extending sFlow to report optical module measurements supports a number of important use cases: * Power loss across link * Dirty connectors * Proactive replacement of failing components * Discover relationship between switch ports and optical underlay * Correlate optical errors with affected traffic * Debug network lane failure, locate source as Tx, Rx, or cable plant The following diagram illustrated the relationship between optical modules, lanes, fibers, and switch ports: +-----------------------------------------------------------------+ | Network Device | | +--------+ +--------+ +--------+ +--------+ +--------+ | | | Port 1 | | Port 2 | | Port 3 | | Port 4 | ... | Port M | | | +---+----+ +--+--+--+ +---+----+ ++-+--+-++ +---+----+ | | | | | | | | | | | | | +------+ | | +-----+ ++ | | ++ | | | | | | | | | | | | | | +-+--+--+--+-+ +-+--+--+--+-+ +----+-----+ | | | 1 2 3 4 | | 1 2 3 4 | | 1 | | | | Module 1 | | Module 2 | | Module N | | +------------+-++-++-++-+++--------+-++-++-++-+++----+----++----+-+ || || || || || || || || || Fibers Fibers Fibers The diagram shows a switch with M ports. Port M is connected to a single module N via a single lane. Two fibers, one for transmitting and the other for receiving data, connect the module to the optical network. This configuration is representative of an Small Form-factor Pluggable (SFP) module. Port 4 is connected to Module 2 via 4 lanes, each of which is connected to a Tx / Rx pair of fibers. This configuration is representative of a Quad Small Form-factor Pluggable (QSFP) module. The lanes in a multi-lane module can be divided among multiple ports, as shown in the case of Module 1. Ports 1, 2 and 3 all connect to the single QSFP Module, with one lane going to port 1, two lanes going to port 2, and 1 lane going to port 3. Capturing the relationship between modules, lanes, and ports is an important part of this extension. The relationship data allows optical lane and module measurements to be correlated with Ethernet Draft sFlow.org [Page 2] Version 0.6 sFlow Optical Interface Structures April 2016 measurements made at port the level. 3. sFlow Datagram Extensions An sFlow agent exporting a counter_sample for a port connected to an optical module must include an if_counters and an sfp structure. The sfp structure identifies the the module, reports module metrics, and includes information on the set of module lanes associated with the port. In the case where the lanes in a module are divided among multiple switch ports, the sfp lanes array should only include the subset of lanes associated with the specific port exporting the counter_sample. The module_id provides a method of identifying the set of ports sharing the module. If there is an ifIndex associated with the module then it should be used as the module_id, otherwise the lowest numbered ifIndex among the ports sharing the module should be used. For example, consider the sfp structure exported by Port 2 in the diagram: module_id=1 module_num_lanes=4 lanes Lane index numbers are encoded using one based indexing since the convention for encoding integer values is to reserve 0 to mean unknown [4] and the sFlow Agent may not know the specific index numbers. Zero based indexing may be used by the underlying APIs to identify lanes and retrieve values but the value must be incremented by one when encoding the index in the lane structure. The sFlow Collector can easily convert the index values to unknown, 0, 1, 2 .. lane index values for consistency with other reporting tools. Tx and Rx wavelength information is included with each lane structure to make it easier for an sFlow Collector to pair up fibers reported by Agents at each end of the connection. Optical Tx and Rx power is typically reported in dBm for human consumption since the unit makes it simpler to reason about signal loss. An sFlow Collector can easily convert the microwatt values in the lane structure to dBm using the following calculation: dBm = 10 * log10(microwatts / 1000) Examples of questions that are easily answered from dBm values include: 1. Has the Tx power dropped by 3 dB? Subtract starting power in dBm from todays power in dBm to get a number. 2. How many connectors with 0.5 dB loss per connector could I add? Read power at the receiver and compare to Ethernet specification. Draft sFlow.org [Page 3] Version 0.6 sFlow Optical Interface Structures April 2016 It should be noted that the optical modules are NOT precision optical meters and care should be taken to account for the error in optical power measurements when using the reading to measure link budget margin. However, modules are very good at measuring actual drift with time. Implementors of this specification should note that modules are typically wired to an Inter-Integrated Circuit (I2C) bus that may be bandwidth contrained, limiting the mininum feasible counter polling interval for these interfaces. In practice, the 30 second default polling interval should be easily attainable for most platforms and agents are permitted to limit the range of polling interval values that can be configured [4]. The following structure is used to report optical module statistics: struct lane { unsigned int index; /* 1-based index of lane within module, 0=unknown */ unsigned int tx_bias_current; /* microamps */ unsigned int tx_power; /* microwatts */ unsigned int tx_power_min; /* microwatts */ unsigned int tx_power_max; /* microwatts */ unsigned int tx_wavelength; /* nanometers */ unsigned int rx_power; /* microwatts */ unsigned int rx_power_min; /* microwatts */ unsigned int rx_power_max; /* microwatts */ unsigned int rx_wavelength; /* nanometers */ } /* Optical SFP / QSFP metrics */ /* opaque = counter_data; enterprise=0; format=10 */ struct sfp { unsigned int module_id; unsigned int module_num_lanes; /* total number of lanes in module */ unsigned int module_supply_voltage; /* millivolts */ int module_temperature; /* thousandths of a degree Celsius */ lane<> lanes; } 4. References [1] "SFF-8472 Diagnostic Monitoring Interface for Optical Transceivers", ftp://ftp.seagate.com/sff/SFF-8472.PDF November 21, 2014 [2] "SFF-8024 SFF Committee Cross Reference to Industry Products", ftp://ftp.seagate.com/sff/SFF-8024.PDF March 14, 2016 Draft sFlow.org [Page 4] Version 0.6 sFlow Optical Interface Structures April 2016 [3] "SFF-8436 QSFP+ 10 Gbs 4X PLUGGABLE TRANSCEIVER", ftp://ftp.sea- gate.com/sff/SFF-8436.PDF October 31, 2013 [4] Phaal, P. and Lavine, M., "sFlow Version 5", http://www.sflow.org/sflow_version_5.txt, July 2006 5. Author's Address Dustin Byford Cumulus Networks Inc. 185 E. Dana Street Mountain View, CA 94041 Matt Peterson San Francisco Metropolitan Internet Exchange Steve Joiner Finisar Corp. Peter Phaal InMon Corp. 1 Sansome Street, 35th Floor San Francisco, CA 94104 Draft sFlow.org [Page 5]