RE: Encoding unknown values in the extended_switchstructure

Paulo,

Unfortunately there are no spare bits in the length field either. The length
field is specified by the XDR encoding rules that sFlow uses to
encode/decode data, see RFC 1832, http://www.ietf.org/rfc/rfc1832.txt

Going back to the data_format field, if the enterprise value is zero then
the upper 20 bits of the field are zero and there are only twelve bits to
encode the structure number and the bit field you are proposing. I wouldn't
want to further limit the number of standard structures (4096) that can be
defined as part of sFlow version 5. There are many areas where sFlow can be
applied and it is likely that there will be a number of new structures
defined.

Including a bit mask in new structures would provide a 32 bits (XDR does
everything in units of 32 bits). I don't imagine that there will be
structures defined that have this many fields, so it should provide a good
solution.

Peter

> -----Original Message-----
> From: Paolo Lucente [mailto:pl+list@pmacct.net]
> Sent: Saturday, May 23, 2009 2:56 PM
> To: Peter Phaal
> Cc: sflow@sflow.org
> Subject: Re: [sFlow] Encoding unknown values in the
> extended_switchstructure
>
> Hi Peter,
>
> I appreciate the hierarchical nature of the data_format tag. Perhaps
> the example of taking two bytes was highly unfortunate given that the
> field is logically split in 20+12 bits.
>
> Of course I share your view on handling unknowns in a future sFlow
> version and the recommendation to developers of structures. Indeed
> the '0xFFFFFFFF' approach can result the least impacting - except
> imagine somebody does sanity checks on those fields, which i guess
> it's why you dropped an email in the first place.
>
> As said, I was focusing only on the case in which enterprise value
> is 0 leaving in the hands of specific developers, enterprise value
> != 0, to find a way to meaningfully handle unknowns - given that
> likely they will be in control of both the agent and the collector.
>
> While thinking at an acceptable trick on the data_format tag, i
> stumbled upon the 32 bit length field just besides that. Do you see
> a chance of recommending carving part of that field (despite its
> name) for the bitmap - when enterprise value equals to zero? At this
> propo, I couldn't find any explicit reference to what is thought to
> be the maximum size of an sFlow datagram; did i miss such indication
> or should that be implicitely inferred?
>
> Cheers,
> Paolo
>
>
> On Sat, May 23, 2009 at 09:13:30AM -0700, Peter Phaal wrote:
> > It would be nice to have a general purpose "unknown" mechanism and the
> use
> > of a bitmap to indicate which elements in a structure fits well with XDR
> > encoding.
> >
> > However, your proposal to take the upper two bytes from the data_format
> > field has some problems. The sFlow version 5 data_format field is fully
> > defined. Your proposed change would not be backward compatible for sFlow
> > collectors since they would need to be altered to ignore the bits (or
> use
> > them to mask the structure).
> >
> > The data_format uniquely identifies the format of an opaque structure in
> > the sFlow specification. A data_format is contructed as follows:
> > - The most significant 20 bits correspond to the SMI Private
> Enterprise
> > Code of the entity responsible for the structure definition. A
> value
> > of zero is used to denote standard structures defined by sflow.org.
> > - The least significant 12 bits are a structure format number
> assigned
> > by the enterprise that should uniquely identify the the format of
> the
> > structure.
> >
> > I just checked IANA and the largest currently assigned enterprise number
> is
> > now 33572. Using the upper two bytes of the data_format would overlap
> with
> > enterprise codes, breaking vendor extensibility - an important feature
> of
> > sFlow version 5.
> >
> > I think you may have misunderstood the role of the structure number in
> the
> > data_format tag. The entire data_format element (32 bits) is treated as
> a
> > unique structure id. The allocation scheme is hierarchical, providing
> each
> > enterprise publishing structures with a full 12 bits that they can
> assign as
> > they choose (i.e. the structure number by itself does not uniquely
> > distinguish a structure, it must be combined with the assigning entity
> to
> > provide a unique id).
> >
> > Currently, sFlow, version 5 defines unknown fields as follows:
> > The following values should be used for fields that are
> > unknown (unless otherwise indicated in the structure
> > definitions).
> > - Unknown integer value. Use a value of 0 to indicate that
> > a value is unknown.
> > - Unknown counter. Use the maximum counter value to indicate
> > that the counter is not available. Within any given sFlow
> > session a particular counter must be always available, or
> > always unavailable. An available counter may temporarily
> > have the max value just before it rolls to zero. This is
> > permitted.
> > - Unknown string. Use the zero length empty string. */
> >
> > Of course there is also the option of omitting a structure all together
> to
> > indicate that none of its fields are known to the agent.
> >
> > These rules work reasonably well, except for the case of the
> extended_switch
> > structure. By "otherwise indicating" the unknown values as 0xFFFFFFFF
> the
> > proposal is the smallest possible change that should be backward
> compatible
> > for existing agents and collectors.
> >
> > If an sFlow version 6 is ever defined then your suggestion of using a
> bit
> > array to provide a general unknown value mechanism would be an
> attractive
> > extension.
> >
> > As new structures are defined for sFlow version 5, adopting a convention
> in
> > which the first member of the structure is a bit array indicating any
> > unknown elements in the structure would be something that authors of
> > structures should consider if there is any possibility that elements of
> the
> > structure might not be known.
> >
> > Peter
> >
> > > -----Original Message-----
> > > From: Paolo Lucente [mailto:pl+list@pmacct.net]
> > > Sent: Saturday, May 23, 2009 4:59 AM
> > > To: Peter Phaal
> > > Cc: sflow@sflow.org
> > > Subject: Re: [sFlow] Encoding unknown values in the
> > > extended_switchstructure
> > >
> > > Hi Peter,
> > >
> > > Not really an objection - rather a proposal for your consideration.
> > >
> > > What about bitmapping fields contained in standard elements within,
> > > say, the upper two bytes of the element tag?
> > >
> > > typedef struct _SFLFlow_sample_element {
> > > struct _SFLFlow_sample_element *nxt;
> > > u_int32_t tag; /* SFLFlow_type_tag */ <-----
> > > u_int32_t length;
> > > SFLFlow_type flowType;
> > > } SFLFlow_sample_element;
> > >
> > > ie.
> > >
> > > #define EX_SWITCH_SRC_VLAN 0x00010000
> > > #define EX_SWITCH_SRC_PRIORITY 0x00020000
> > > #define EX_SWITCH_DST_VLAN 0x00040000
> > > #define EX_SWITCH_DST_PRIORITY 0x00080000
> > >
> > > It would have the beauty of being backward compatible, meaning the
> > > collector can just ignore it and not take benefit from it, and will
> > > maintain intact the 32 bit field - which is not a requirement today,
> > > for this specific case, but you might be less lucky tomorrow trying
> > > to do the same trick elsewhere; so a more coherent approach.
> > >
> > > Companies developing closed extensions and thus making use of the
> > > upper bytes of the tag field, ie.
> > >
> > > /* enterprise = 4300 (inmon)...*/
> > > SFLFLOW_EX_PROCESS = (4300 << 12) + 3
> > >
> > > might (or will need to) use a different way to signal some fields
> > > are not populated within their elements.
> > >
> > > The only con of this approach is that it would put an upper bound
> > > to the number of fields that can be packed within an element, ie.
> > > 15 fields if using the upper 2 bytes (keeping out the '0x00000000'
> > > value), but i see you have never been a big fan of flat structures
> > > ( :-) ) so i wouldn't see this as a true biggie. Quickly scanning
> > > through the existing elements, all seem to fit this approach.
> > >
> > > Cheers,
> > > Paolo
> > >
> > >
> > > On Fri, May 22, 2009 at 11:07:55AM -0700, Peter Phaal wrote:
> > > > There currently isn't a satisfactory way to indicate that one or
> more of
> > > the
> > > > fields in the extended_switch structure is unknown to the sFlow
> > > exporter.
> > > >
> > > > /* Extended Switch Data */
> > > > /* opaque = flow_data; enterprise = 0; format = 1001 */
> > > > /* Note: For untagged ingress ports, use the assigned vlan and
> priority
> > > > of the port for the src_vlan and src_priority values.
> > > > For untagged egress ports, use the values for dst_vlan and
> > > > dst_priority that would have been placed in the 802.Q tag
> > > > had the egress port been a tagged member of the VLAN
> instead
> > > > of an untagged member. */
> > > >
> > > > struct extended_switch {
> > > > unsigned int src_vlan; /* The 802.1Q VLAN id of incoming
> frame */
> > > > unsigned int src_priority; /* The 802.1p priority of incoming
> frame
> > > */
> > > > unsigned int dst_vlan; /* The 802.1Q VLAN id of outgoing
> frame */
> > > > unsigned int dst_priority; /* The 802.1p priority of outgoing
> frame
> > > */
> > > > }
> > > >
> > > > Since 0 is a valid value for both 802.1Q VLAN 802.1p priority it
> should
> > > not
> > > > be used to encode the unknown value. Rather than omit the whole
> > > structure
> > > > because one field isn't known, it makes sense to define an
> unambiguous
> > > value
> > > > to indicate that a field is unknown.
> > > >
> > > > Fortunately these fields have relatively small maximum values, vlan
> <=
> > > 4096
> > > > and priority <= 7, so an unambiguous choice for an unknown value
> would
> > > be
> > > > 0xffffffff (Note: this same value that sFlow currently uses to
> indicate
> > > that
> > > > a 32 bit counter value is unknown). If there are no objections, I
> think
> > > it
> > > > would make sense to include a note to this effect on Specifications
> page
> > > on
> > > > sFlow.org.
> > > >
> > > > Peter
Received on Sat May 23 16:28:42 2009