sFlow.org Ariel Almog http://www.sFlow.org/ Mellanox Technologies info@sflow.org Peter Phaal InMon Corp. June 2013 sFlow InfiniBand Structures Copyright Notice Copyright (C) sFlow.org (2013). All Rights Reserved. Abstract This document describes additional structures that allow an sFlow agent to export information about Infiniband [1] traffic. sFlow version 5 [2] is an extensible protocol that allows the addition of new data structures without impacting existing collectors. This document does not change the sFlow version 5 protocol, it simply defines additional, optional, structures that an Infiniband device can use when reporting on Infiniband traffic. Table of Contents 1. Overview ...................................................... 1 2. sFlow Datagram Extension ...................................... 1 3. References .................................................... 2 4. Author's Addresses ............................................ 2 1. Overview This document describes additional structures that allow an sFlow agent to export additional information about the Infiniband traffic. sFlow version 5 is an extensible protocol that allows the addition of new data structures without impacting existing collectors. This document does not change the sFlow version 5 protocol [1], it simply defines additional, optional, data structures that an Infiniband device [2] can use when reporting on Infiniband traffic. An sFlow collector that does not understand a particular structure should simply skip it. 2. sFlow Datagram Extension DRAFT sFlow.org [Page 1] Version 0.2 sFlow InfiniBand Structures June 2013 The /* Additional packet header type in enum header_protocol */ INFINIBAND = 18 /* InfiniBand */ /* Extended IB LRH Data - Local Routing Header definition from InfiniBand Architecture Specification */ /* opaque = ib_lrh_data; enterprise = 0; format = 1031 */ struct extended_ib_lrh { unsigned int src_vl; /* source virtual lane */ unsigned int src_sl; /* source service level */ unsigned int src_dlid; /* source destination-local-ID */ unsigned int src_slid; /* source source-local-ID */ unsigned int src_lnh; /* source link next header */ unsigned int dst_vl; /* Destination virtual lane */ unsigned int dst_sl; /* Destination service level */ unsigned int dst_dlid; /* Destination destination-local-ID */ unsigned int dst_slid; /* Destination source-local-ID */ unsigned int dst_lnh; /* Destination link next header */ } /* GID type 16 bytes long */ typedef opaque gid[16]; /* Extended IB GRH Data - Global Routing Header definition from InfiniBand Architecture Specification */ /* opaque = ib_grh_data; enterprise = 0; format = 1032 */ struct extended_ib_grh { unsigned int flow_label; /* flow label */ unsigned int tc; /* Traffic Class */ gid s_gid; /* source GID */ gid d_gid; /* destination GID */ unsigned int next_header /* next header type */ unsigned int length /* payload length */ } /* Extended IB BTH Data - Base Transport Header definition from InfiniBand Architecture Specification */ /* opaque = ib_bth_data; enterprise = 0; format = 1033 */ DRAFT sFlow.org [Page 2] Version 0.2 sFlow InfiniBand Structures June 2013 struct extended_ib_bth { unsigned int pkey; /* Partition key */ unsigned int dst_qp; /* Destination Queue Pair */ unsigned int opcode; /* IBA packet type */ } /* IB counters Note on multi manager handling IB counters IB counters are fetched by SM and are sent to sFlow collector. The SM implementation might cause the counter to be reset to zero by writing (Set) zero into a counter. If the SM resets the counter, the sFlow collector will get miss-leading counting A data source exporting ib_counters must also include the generic if_counters structure. The following notes describe InfiniBand specific values and must be followed when exporting the ib_counters structure: ifType = infiniband (199) ifInOctets = PortRcvData Total octets, received on all VLs ifInUcastPkts = PortRcvPkts Total packets: Ucast, Multicast & Broadcast, may include packets containing errors ifInMulticastPkts = 0 ifInBroadcastPkts = 0 ifInDiscards = VL15Dropped + PortRcvConstraintErrors ifInErrors = PortRcvErrors + PortRcvRemotePhysicalErrors + PortRcvSwitchRelayErrors ifInUnknownProtos = 0 ifOutOctetsVL15Dropped = PortXmitData Total octets, transmitted on all VLs ifOutUcastPkts = PortXmitPkts Total packets: Ucast, Multicast & Broadcast ifOutMulticastPkts = 0 ifOutBroadcastPkts = 0 ifOutDiscards = PortXmitDiscards ifOutErrors = PortXmitConstraintErrors */ /* IB Counters */ /* opaque = counter_data; enterprise = 0; format = 9 */ struct ib_counters { unsigned hyper PortXmitPkts; /* Total packets transmitted on all VLs */ unsigned hyper PortRcvPkts; /* Total packets (may include packets containing errors */ unsigned int SymbolErrorCounter; unsigned int LinkErrorRecoveryCounter; unsigned int LinkDownedCounter; unsigned int PortRcvErrors; unsigned int PortRcvRemotePhysicalErrors; unsigned int PortRcvSwitchRelayErrors; unsigned int PortXmitDiscards; unsigned int PortXmitConstraintErrors; unsigned int PortRcvConstraintErrors; unsigned int LocalLinkIntegrityErrors; unsigned int ExcessiveBufferOverrunErrors; unsigned int VL15Dropped; } 3. References [1] "InfiniBand Trade Association", http://www.infinibandta.org/ [2] Phaal, P. and Lavine, M., "sFlow Version 5", http://www.sflow.org/sflow_version_5.txt, July 2006 4. Author's Address Ariel Almog Mellanox Technologies 350 Oakmead Parkway, Suite 100 Sunnyvale, CA 94085 Phone: (408) 970-3400 EMail: ariela@mellanox.com Peter Phaal InMon Corp. 1 Sansome Street, 35th Floor San Francisco, CA 94104 DRAFT sFlow.org [Page 3] Version 0.2 sFlow InfiniBand Structures June 2013 Phone: (415) 946-8901 EMail: peter.phaal@inmon.com DRAFT sFlow.org [Page 4]