sFlow Datagram Extensibility

From: Marc Lavine (mlavine@foundrynet.com)
Date: 08/26/02

  • Next message: Peter Phaal: "RE: sFlow Datagram Extensibility"

    Hello all,

    I've been doing some work on an sFlow collector implementation and
    I've noticed some extensibility issues with the existing standard (as
    described in RFC 3176). The current standard seems to lack the
    ability to gracefully extend the types of data included in sFlow
    datagrams without breaking compatibility with existing collector

    As an example, with the existing datagram design, if a new type of
    extended data is defined, introducing it requires changing the
    datagram definition. Because the encoding of the extended data types
    does not include length information, a collector cannot skip over new
    extended data types that it does not recognize. As a result of this,
    the datagram version number must be changed when new extended data
    types are added in order to avoid having collectors mis-parse
    datagrams with the new extended data types. A collector which has not
    yet been updated to recognize the new datagram version must discard
    the entire datagram because it cannot be sure that it can parse any of
    it correctly.

    It seems that this could cause considerable hassles for customers when
    upgrading agents and collectors as the standard evolves. To address
    this problem, it seems to me that the datagram format should be
    changed so that extended data types are preceded by a length field.
    This would make them sufficiently self-descriptive that a collector
    could skip over any ones which are not known to it. There would be
    some additional space overhead in doing this, but I think it would be
    well worth the improved interoperability over the long term. I think
    it would be a good idea to apply this principle of making the data
    self-descriptive elsewhere as well. I think that counter samples
    should also have a length field added so that new types of counter
    samples could be added without breaking compatibility with existing
    collectors. (When I say "existing collectors" here, I mean those
    collectors which would support this new sFlow version being proposed.)
    If this length field is placed at the beginning of the counters_sample
    structure, then a corresponding length field can also be added to the
    flow_sample structure, which would then provide for the possibility of
    adding new sample types in the future while maintaining compatibility
    with existing collectors.

    Adding length fields to structures as I've described actually provides
    for two different types of extensibility. The first type of
    extensibility, which I referred to above, involves adding new
    structures, identified with new type tags (i.e. adding new branches to
    certain unions in the XDR specification of the sFlow datagram format).
    The second type of extensibility involves being able to extend the
    length of an already-existing structure in a way that need not break
    compatibility with collectors which understand an older version of
    that structure. As an example, let's say that a new field is
    identified that should belong in the extended_gateway structure.
    Rather than define a new extended data type just for this new field,
    it could simply be added to the end of the existing structure. Since
    the structure would include a length field, a collector can process
    the portion of the structure that it knows about, and skip over the

    To ensure that changes are only made in a compatible fashion, the
    sFlow datagram specification can state that the self-descriptive
    structures may be extended in future revisions of the specification by
    adding fields to the end of the structures, and that no fields my be
    removed or reordered, and that such changes would only be made to
    structures which are not included within other structures (eliminating
    the generic if_counters structure as a candidate for this type of
    extensibility). It can also state that such changes would not require
    a change in the type tag associated with the structure, nor in the
    sFlow datagram version number.

    There is a third type of extensibility that we also have a need for,
    and that is the ability to add vendor-specific extensions. While we
    certainly think it's good to standardize things that are common across
    vendor implementations, different vendors have various kinds of
    extensions, which are not standardized, and yet which could be very
    useful to include in sFlow samples (as long as someone will build a
    collector to utilize them, of course).

    To implement this, I propose that a new type tag or two be added to
    the list for the extended data types. These type tags would indicate
    vendor-specific extensions. The structures for these would include a
    length (as proposed above for all extended data type structures), and
    would have a vendor-assigned identifier which would indicate which
    type of content was present, and then actual content would follow.
    For the vendor-assigned identifiers, I propose two different options.
    For a vendor to be able to assign their own identifiers, these need to
    be constructed based on a globally-unique identifier. The two that
    come to mind are an IEEE OUI (Organizationally Unique Identifier),
    which appears as the first three bytes of a MAC address, and an SNMP
    enterprise OID. Most switch vendors are likely to already have one or
    both of these already assigned to them. The sFlow specification could
    support one or the other of these types of identifiers, or support
    both by defining two extended data types, one for use with each kind
    of identifier. As far as possible encodings, an OUI plus a single
    additional byte would fit in a single XDR unsigned int, while an SNMP
    OID would probably be reasonably compactly encoded as an XDR
    variable-length opaque field containing the ASN.1-encoded OID.

    Overall, I think these changes will allow for greater interoperability
    between different versions of sFlow agents and collectors and should
    make sFlow easier for customers to use. I look forward to any
    comments on this proposal.

    Marc Lavine
    Foundry Networks, Inc.


    This archive was generated by hypermail 2.1.4 : 08/26/02 PDT