From: Peter Phaal (peter_phaal@inmon.com)
Date: 08/29/02
Mark,
I agree that improving the flexibility of the sFlow Datagram encoding would
be a good idea.
I have incorporated some of your suggestions into a couple of drafts to try
and see what they would look like.
http://www.sflow.org/SFLOW-DATAGRAM5.txt
http://www.sflow.org/INMON-STRUCTS.txt
It made sense to split the datagram document into two parts: the first
document is a description of the overall datagram format, and the second
contains detailed structure definitions for standard flow and counter data.
As you suggested the datagram format now contains type and length
information that will allow additional structures to be easily added.
Additional comments below:
It seems that this could cause considerable hassles for customers when
upgrading agents and collectors as the standard evolves. To address
this problem, it seems to me that the datagram format should be
changed so that extended data types are preceded by a length field.
I changed the flow data so that it is now represented as a list of
flow_record structures. Each flow_record provides type, length and value
information so that fields can be skipped easily.
This would make them sufficiently self-descriptive that a collector
could skip over any ones which are not known to it. There would be
some additional space overhead in doing this, but I think it would be
well worth the improved interoperability over the long term. I think
it would be a good idea to apply this principle of making the data
self-descriptive elsewhere as well. I think that counter samples
should also have a length field added so that new types of counter
samples could be added without breaking compatibility with existing
collectors. (When I say "existing collectors" here, I mean those
collectors which would support this new sFlow version being proposed.)
If this length field is placed at the beginning of the counters_sample
structure, then a corresponding length field can also be added to the
flow_sample structure, which would then provide for the possibility of
adding new sample types in the future while maintaining compatibility
with existing collectors.
I also changed the counter sample to represent counters as a list of
counter blocks, each with a type, length and value. This would allow a
collector to ignore counter blocks that it didn't understand. It also makes
it easy for agent developers to add new counters.
This mechanism allows you to add new counters to the basic set by defining
a counter block for the additional counters. The agent would then report the
basic set + the additional counters. If you need to define a full set of
counters for a new media type then that would involve defining a larger
counter block.
Adding length fields to structures as I've described actually provides
for two different types of extensibility. The first type of
extensibility, which I referred to above, involves adding new
structures, identified with new type tags (i.e. adding new branches to
certain unions in the XDR specification of the sFlow datagram format).
The second type of extensibility involves being able to extend the
length of an already-existing structure in a way that need not break
compatibility with collectors which understand an older version of
that structure. As an example, let's say that a new field is
identified that should belong in the extended_gateway structure.
Rather than define a new extended data type just for this new field,
it could simply be added to the end of the existing structure. Since
the structure would include a length field, a collector can process
the portion of the structure that it knows about, and skip over the
rest.
To ensure that changes are only made in a compatible fashion, the
sFlow datagram specification can state that the self-descriptive
structures may be extended in future revisions of the specification by
adding fields to the end of the structures, and that no fields my be
removed or reordered, and that such changes would only be made to
structures which are not included within other structures (eliminating
the generic if_counters structure as a candidate for this type of
extensibility). It can also state that such changes would not require
a change in the type tag associated with the structure, nor in the
sFlow datagram version number.
I don't think it is a good idea to allow existing structures to be
extended. Different agent implementors might extend the structure in
different, incompatible ways and a receiver of the data would not have
enough information to determine how to decode the added fields.
It seems that this problem is better solved as part of the vendor-specific
extension capability - see below.
There is a third type of extensibility that we also have a need for,
and that is the ability to add vendor-specific extensions. While we
certainly think it's good to standardize things that are common across
vendor implementations, different vendors have various kinds of
extensions, which are not standardized, and yet which could be very
useful to include in sFlow samples (as long as someone will build a
collector to utilize them, of course).
To implement this, I propose that a new type tag or two be added to
the list for the extended data types. These type tags would indicate
vendor-specific extensions. The structures for these would include a
length (as proposed above for all extended data type structures), and
would have a vendor-assigned identifier which would indicate which
type of content was present, and then actual content would follow.
For the vendor-assigned identifiers, I propose two different options.
For a vendor to be able to assign their own identifiers, these need to
be constructed based on a globally-unique identifier. The two that
come to mind are an IEEE OUI (Organizationally Unique Identifier),
which appears as the first three bytes of a MAC address, and an SNMP
enterprise OID. Most switch vendors are likely to already have one or
both of these already assigned to them. The sFlow specification could
support one or the other of these types of identifiers, or support
both by defining two extended data types, one for use with each kind
of identifier. As far as possible encodings, an OUI plus a single
additional byte would fit in a single XDR unsigned int, while an SNMP
OID would probably be reasonably compactly encoded as an XDR
variable-length opaque field containing the ASN.1-encoded OID.
I think that the SNMP enterprise OID is the better candidate since any
network management agent writer is likely to have one already.
The new counter_block and flow_record structures can be easily added
without breaking backward compatibility. The combination of enterprise and
format values creates a global name space for the opaque structures.
Let me know what you think of the new documents. A big benefit of this new
approach is that new flow/counter structures can be developed and deployed
quickly without worrying about breaking existing applications. It also
provides a convenient way of experimenting with new functionality.
Regards,
Peter
PS I have also been looking at adding additional flow_record types. Included
in the second document are additional flow records to report on MPLS and NAT
data. Any suggestions on improving reporting in these areas are welcome.
This archive was generated by hypermail 2.1.4 : 08/29/02 PDT