---

From: Caitlin Bestler [mailto:caitlinb@...]
Sent: Thursday, April 05, 2007 5:17 PM
To: dat-discussions@yahoogroups.com
Subject: RE: [dat-discussions] Multicast RDMA proposal

DAT is probably not the correct forum to discuss this, since I believe
the implications of multicast RDMA would be neutral to the API. A
reliable multicast session with one producer and multiple consumers
looks amazingly like a reliable point-to-point connection.

The protocol implications would have to be discussed in the IETF
and IBTA. But there are some tricky ones to be considered:

1) When will Send/RDMA Write operations complete? If the consumers
are fully enumerated and unchanging, which is acceptable for MPI,
then the sender merely has to merge all of the ACKs. Tricky, but
doable, and inherent to any reliable multicast lower layper protocol.

2) But trickier than that: how would the producer get more send credits?
RDMA Send Credits are ULP activities that are granted by posting a
receive buffer (InfiniBand) or send a ULP message (iWARP).
Determining
that you have message-level send credits would be very tricky.
Because
RDMA does not rely on transport layer buffering, it is very important
that buffer availability be explicitly advertised. Any proposal for
multicast RDMA might have to come up with a mechanism to pace the
consumption of anonymous buffers. Receivers that have fallen behind
would have to explicitly NACK. Collecting explicit acks for each
DDP Segment (or IB packet) from each multicast receiver does not
sound very feasible.

3) Theoretically a tagged buffer could have multicast meaning, if the
RKey is given the same meaning on each recepient. This is very easy
to mandate, but difficult to implement using existing RDMA
interfaces.
It probably requires an interface to request a specific RKey that is
only valid on a specific multicast session/connection.

Have you evaluated using RDMA Read to have each receiver fetch the
data as needed and using multicast Unreliable Datagrams / UDP to
notify the receivers of the availability of new data?

DAT is probably not the correct forum to discuss this, since I believe
the implications of multicast RDMA would be neutral to the API. A
reliable multicast session with one producer and multiple consumers
looks amazingly like a reliable point-to-point connection.

The protocol implications would have to be discussed in the IETF
and IBTA. But there are some tricky ones to be considered:

1) When will Send/RDMA Write operations complete? If the consumers
    are fully enumerated and unchanging, which is acceptable for MPI,
    then the sender merely has to merge all of the ACKs. Tricky, but
    doable, and inherent to any reliable multicast lower layper protocol.

2) But trickier than that: how would the producer get more send credits?
    RDMA Send Credits are ULP activities that are granted by posting a
    receive buffer (InfiniBand) or send a ULP message (iWARP).
Determining
    that you have message-level send credits would be very tricky.
Because
    RDMA does not rely on transport layer buffering, it is very important
    that buffer availability be explicitly advertised. Any proposal for
    multicast RDMA might have to come up with a mechanism to pace the
    consumption of anonymous buffers. Receivers that have fallen behind
    would have to explicitly NACK. Collecting explicit acks for each
    DDP Segment (or IB packet) from each multicast receiver does not
    sound very feasible.

3) Theoretically a tagged buffer could have multicast meaning, if the
    RKey is given the same meaning on each recepient. This is very easy
    to mandate, but difficult to implement using existing RDMA
interfaces.
    It probably requires an interface to request a specific RKey that is
    only valid on a specific multicast session/connection.

Have you evaluated using RDMA Read to have each receiver fetch the
data as needed and using multicast Unreliable Datagrams / UDP to
notify the receivers of the availability of new data?

This proposal extends the RDMA semantics to include delivery under a
message-based reliable multicast protocol, such as NACK-Oriented
Reliable Multicast (NORM). It does not apply to unreliable multicast
delivery (which is the standard format) or to file-based reliable
multicast protocols (such as multicast FTP, or FLUTE).

First, I will state the problem that this proposal seeks to address:
Collective operations of any kind are slow. Painfully slow.
Transmitting the same data to multiple nodes is a common operation in
HPC and an often-heard complaint amongst research labs and
corporations is that conventional point-to-point messaging is
painfully bad at handling these cases. So much so that at SC|05,
several exhibitors were showing custom-built hardware for handling
collective operations.

My underlying thesis, then, is that multipoint mechanisms are needed
to handle these cases efficiently and in a timely manner, as it seems
evident that massively duplicating messages/data for widespread
distribution is not good enough or scalable enough in domains where
RDMA already rules the roost. Point-to-point would remain the bulk of
the communication, but multipoint would eliminate certain common forms
of congestion.

The proposed extensions are relatively trivial and attempt to avoid
extending the syntax on the basis that multicast RDMA would have
greatest value if there are no changes to anything in the existing API
and the fewest possible additions. (In fact, I believe no API
additions are required - it can be done entirely with what is already
available.)

The most fundamental change in the semantics is that RDMA is that it
is conventionally described as point-to-point. A group address is
still a single address and can therefore be regarded as a virtual
point. This changes the description to point-to-virtual-point. As
fundamental changes go, this doesn't look to be too bad.

Because the underlying multicast protocol is reliable, it is not
necessary for RDMA to be aware that a write operation is going to
multiple destinations. It is entirely the responsibility of the
multicast support to hide that level of detail from RDMA.

This begs the question of how to handle STags and offsets. Multicast
RDMA is pictured as being useful when handling collective operations,
where identical messages are being sent to a number of nodes. The
offsets within a message, then, will be completely consistent, so this
can be used without any changes. STags are a little more troublesome,
as you wouldn't want two nodes to use the same STag for completely
different purposes.

This requires either negotiation or the use of uuid-type STags to
guarantee that labeling is unique and cannot under any realistic
circumstance clash. The former has some bothersome overheads and has
more code impact than I'd like, so a uuSTag would be the least bothersome.

To sum up so far: write operations deliver data to multiple
destinations, although RDMA only sees a single target. There is no
change to the API and there is no change to the RDMA protocol.

Changes: The only change is that RDMA is then layered on a reliable
multipoint transport, but that transport should be totally transparent
to the RMDA mechanism. As RDMA partially resides on the RNIC, this
implies a hardware implementation of a reliable multicast protocol
that sits somewhere between the RDMA packet
encapsulation/decapsulation and the wire.

Now we get onto read operations. Here, again, I will limit changes to
generalizing what already exists, rather than actually adding anything
new.

A read operation in a conventional RDMA transaction sends to a single
point a request for some data, which is then returned to the
originator of the read. There are two ways this can be handled in a
multicast scenario - you can read from multiple sources and have the
results sent to a single destination, or you can read from one source
and have the results delivered to multiple destinations.

Both of these have their strengths, but I am going to argue that only
one should be implemented - otherwise we have two types of read for
multicast, which would change the API, which is something I'm trying
to avoid. I am also going to argue that reading from one source and
sending the results to many has the least impact on semantics (the
requesting node asks for a block of data and gets a block of data, not
an unknown number of blocks) and offers the best use of I/O resources
(data often takes more bandwidth than commands, so it is the data you
want to avoid duplicating, not the commands).

Since the reply to a read is, in essence, a write, and writes to a
group have already been covered, RNICs would be at liberty to discard
any read requests that are not directed at them. It would not impact
the transfer in any way, shape or form for them to do so. An RNIC that
picked up a read request for another node would be able to
pre-allocate resources, but that is an optimization issue and not a
design issue.

Summing up: Reads go to one node, the response goes to the whole
group, so RNICs get exactly what they would have expected anyway, with
no changes to the way things currently work.

Changes: Multicast reads, using these semantics, would not be any
different from a unicast read. So why use multicast at all for reads?
It keeps the multicast aspect encapsulated and transparent to all
layers above it. It also means that the read comes from the group
address and the response goes to the group address. Thus, there is no
change to the logic or routing. For this sort of proposal, no change
is a good change.

Next up, we get onto how these group addresses are used. There is
obviously no good candidate in RDMA for that, because it doesn't need it.

What is needed is the ability to join/leave groups and to have group
information propagated across the network as needed. Most fabrics
already have multicast protocols and multicast APIs, and many RNICs
have hardware support to some degree for multicast routing and
multicast protocols. All we need is to have the reliable multicast
mechanism bridge the existing hardware RDMA mechanism with the
existing hardware multicast mechanism.

By bridging, we can use what is already there. We don't need to
duplicate any logic and since the hardware multicast support is
already hooked up to the machine's userland, nothing extra is needed
to populate tables or handle routing. It's already there.

Finally, there is the matter of what reliable multicast transport to
use. As I've already indicated, there are quite a number. The benefits
of using NORM are that it's a standard, reference implementations
exist, it's not encumbered and it seems to scale fairly effectively.

The only three things that would need to be true are that:

1) A multicast RDMA would need to agree on a multicast transport, if
it is to be interoperable - which is quite a desirable feature in a
protocol.

2) Since it would need to be in hardware, to be transparent, it must
be a protocol that would be simple and cheap to implement in hardware.

3) Any overheads whatsoever, of any kind, be that cost, latency,
complexity, etc, must be exceeded by the benefits in those same
domains. (Improving the effective bandwidth by utilizing existing
fabrics more efficiently costs money, but so does improving the
effective bandwidth by adding more real bandwidth. A theoretical or
technical benefit is interesting, but is it profitable to providers
and viable to users? Is the congestion that this would ease common
enough and severe enough to be worth easing by this method?)

This is a very rough-and-ready description of the proposed extension,
how it would be implemented, where it would be implemented, what it
would require, and what it would achieve. The biggest question, of
course, is is it worth it?

The next question is - although I've tried to draw up a proposal that
is semi-transparent to RDMA, it is not invisible and never can be, so
does this succeed in the aim of not breaking anything that already
exists and already works?

Hi,

This is just an FYI, I'm sure people are already aware of this. The
reference code, when compiling under Linux, uses sysfs. However, the
API to sysfs has changed and the function sysfs_read_attribute_value()
no longer exists. In the repository version of dapl, for example,
udapl/openib_scm/dapl_ib_cm.c uses this call on line 96. Naturally,
this makes using the code a little awkward. :)

Are there any out-of-repository patches for those of us using
bleeding-edge environments?

Hello,

This email message is a notification to let you know that
a file has been uploaded to the Files area of the dat-discussions
group.

   File        : /dat20_headers.tar.gz
   Uploaded by : arkadynetappcom <arkady@...>
   Description : DAT 2.0 headers with IB and iW extensions

You can access this file at the URL:
http://groups.yahoo.com/group/dat-discussions/files/dat20_headers.tar.gz

To learn more about file sharing for your group, please visit:
http://help.yahoo.com/help/us/groups/files

Regards,

arkadynetappcom <arkady@...>

Hello,

This email message is a notification to let you know that
a file has been uploaded to the Files area of the dat-discussions
group.

   File        : /Minutes/DAT-MIN-01_05_07.pdf
   Uploaded by : arkadynetappcom <arkady@...>
   Description : minutes of DAT meeting 01/05/2007

You can access this file at the URL:
http://groups.yahoo.com/group/dat-discussions/files/Minutes/DAT-MIN-01_05_07.pdf

To learn more about file sharing for your group, please visit:
http://help.yahoo.com/help/us/groups/files

Regards,

arkadynetappcom <arkady@...>

---

From: David M. Brean [mailto:David.Brean@...]
Sent: Friday, January 05, 2007 5:02 PM
To: Kanevsky, Arkady
Subject: Re: [dat-discussions] DAT 2.0 is ratified!

I thought that the deadline is COB today, but for the record Sun votes Yes on all documents.

-David

Kanevsky, Arkady wrote:

Dear group,

I would like to congradulate on the ratitication of

uDAPL-2.0 and kDAPL-2.0,

DAT IB and iWARP extensions specs,

and transition document to DAT 2.0 from the earlier versions.

 

All these documents are on DAT reflector now and will appear on DAT web site shortly.

 

Thanks to everybody for all the hard work which DAT 2.0 culminates!

 

Arkady Kanevsky

Chair of DAT Collaborative

 

Arkady Kanevsky                       email: arkady@netapp.com

Network Appliance Inc.               phone: 781-768-5395

1601 Trapelo Rd. - Suite 16.        Fax: 781-895-1195

Waltham, MA 02451                   central phone: 781-768-5300

 

Hello,

This email message is a notification to let you know that
a file has been uploaded to the Files area of the dat-discussions
group.

   File        : /DAPL20-ERRATA_final.pdf
   Uploaded by : arkadynetappcom <arkady@...>
   Description : DAT 2.0 errata spreadsheet final

You can access this file at the URL:
http://groups.yahoo.com/group/dat-discussions/files/DAPL20-ERRATA_final.pdf

To learn more about file sharing for your group, please visit:
http://help.yahoo.com/help/us/groups/files

Regards,

arkadynetappcom <arkady@...>

Symantec DAT 2.0 ratification vote.

Arkady Kanevsky                       email: arkady@...
Network Appliance Inc.               phone: 781-768-5395
1601 Trapelo Rd. - Suite 16.        Fax: 781-895-1195
Waltham, MA 02451                   central phone: 781-768-5300


> -----Original Message-----
> From: somenath [mailto:somenath@...]
> Sent: Friday, January 05, 2007 2:11 PM
> To: somenath@...
> Cc: Kanevsky, Arkady
> Subject: Re: FW: [dat-discussions] DAT 2.0 RATIFICATION VOTE
>
> Arkady:
>
> sorry for the late reply....Symantec votes yes on 2.0 ratification.
>
> thanks, som.
>
> somenath wrote:
>
> > Arkady,
> >
> > I will take a look at this soon.
> >
> > thanks, som.
> >
> > Kanevsky, Arkady wrote:
> >
> >> Som,
> >> Can you cast Veritas/Symantec vote on DAT 2.0 ratification?
> >> Thanks,
> >> Arkady
> >>
> >>
> >> Arkady Kanevsky                       email: arkady@...
> >> <mailto:arkady@...>
> >>
> >> Network Appliance Inc.               phone: 781-768-5395
> >>
> >> 1601 Trapelo Rd. - Suite 16.        Fax: 781-895-1195
> >>
> >> Waltham, MA 02451                   central phone: 781-768-5300
> >>
> >>
> >>
> >>
> >>
> --------------------------------------------------------------
> ----------
> >>     *From:* Kanevsky, Arkady
> >>     *Sent:* Friday, December 01, 2006 2:35 PM
> >>     *To:* dat-discussions@yahoogroups.com
> >>     *Subject:* [dat-discussions] DAT 2.0 RATIFICATION VOTE
> >>
> >>     Dear group,
> >>     we are starting the clock for the DAT 2.0 ratification process.
> >>     This time we are ratifing 5 documents:
> >>     uDAPL 2.0, kDAPL 2.0, IB extension document,
> >>     iWARP extension document, and Forward Compatibilty Transition
> >>     document.
> >>          All 5 documents are on DAT reflector.
> >>     If you have problem accessing them, please, contact me.
> >>          The deadline for comments is Wednesday Dec 20.
> >>     The deadline for the ratification vote is Friday Dec 22.
> >>          Thanks,
> >>     Arkady
> >>     Chair of DAT Collaborative
> >>
> >>     Arkady Kanevsky                       email: arkady@...
> >>     <mailto:arkady@...>
> >>
> >>     Network Appliance Inc.               phone: 781-768-5395
> >>
> >>     1601 Trapelo Rd. - Suite 16.        Fax: 781-895-1195
> >>
> >>     Waltham, MA 02451                   central phone: 781-768-5300
> >>
> >>
> >
> >
> >
>

Hello,

This email message is a notification to let you know that
a file has been uploaded to the Files area of the dat-discussions
group.

   File        : /Minutes/DAT-MIN-01_05_07.pdf
   Uploaded by : arkadynetappcom <arkady@...>
   Description : minutes of DAT meeting 01/05/2007

You can access this file at the URL:
http://groups.yahoo.com/group/dat-discussions/files/Minutes/DAT-MIN-01_05_07.pdf

To learn more about file sharing for your group, please visit:
http://help.yahoo.com/help/us/groups/files

Regards,

arkadynetappcom <arkady@...>

Hello,

This email message is a notification to let you know that
a file has been uploaded to the Files area of the dat-discussions
group.

   File        : /Minutes/DAT-MIN-01_05_07.pdf
   Uploaded by : arkadynetappcom <arkady@...>
   Description : minutes of DAT meeting 01/05/2007

You can access this file at the URL:
http://groups.yahoo.com/group/dat-discussions/files/Minutes/DAT-MIN-01_05_07.pdf

To learn more about file sharing for your group, please visit:
http://help.yahoo.com/help/us/groups/files

Regards,

arkadynetappcom <arkady@...>

Hello,

This email message is a notification to let you know that
a file has been uploaded to the Files area of the dat-discussions
group.

   File        : /kDAPL_v20_final.zip
   Uploaded by : arkadynetappcom <arkady@...>
   Description : kDAPL-2.0 final spec

You can access this file at the URL:
http://groups.yahoo.com/group/dat-discussions/files/kDAPL_v20_final.zip

To learn more about file sharing for your group, please visit:
http://help.yahoo.com/help/us/groups/files

Regards,

arkadynetappcom <arkady@...>

Hello,

This email message is a notification to let you know that
a file has been uploaded to the Files area of the dat-discussions
group.

   File        : /transition_to_dat20_final.pdf
   Uploaded by : arkadynetappcom <arkady@...>
   Description : Transition to DAT 2.0 from earlier versions final doc

You can access this file at the URL:
http://groups.yahoo.com/group/dat-discussions/files/transition_to_dat20_final.pd\
f

To learn more about file sharing for your group, please visit:
http://help.yahoo.com/help/us/groups/files

Regards,

arkadynetappcom <arkady@...>

Hello,

This email message is a notification to let you know that
a file has been uploaded to the Files area of the dat-discussions
group.

   File        : /DAT_IB_Extensions_Final.pdf
   Uploaded by : arkadynetappcom <arkady@...>
   Description : DAT IB extension final spec

You can access this file at the URL:
http://groups.yahoo.com/group/dat-discussions/files/DAT_IB_Extensions_Final.pdf

To learn more about file sharing for your group, please visit:
http://help.yahoo.com/help/us/groups/files

Regards,

arkadynetappcom <arkady@...>

Hello,

This email message is a notification to let you know that
a file has been uploaded to the Files area of the dat-discussions
group.

   File        : /uDAPL_v20_final.zip
   Uploaded by : arkadynetappcom <arkady@...>
   Description : uDAPL-2.0 final spec

You can access this file at the URL:
http://groups.yahoo.com/group/dat-discussions/files/uDAPL_v20_final.zip

To learn more about file sharing for your group, please visit:
http://help.yahoo.com/help/us/groups/files

Regards,

arkadynetappcom <arkady@...>

/*
  * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved.
  *
  * This Software is licensed under one of the following licenses:
  *
  * 1) under the terms of the "Common Public License 1.0" a copy of which
is
  *    in the file LICENSE.txt in the root directory. The license is also
  *    available from the Open Source Initiative, see
  *    http://www.opensource.org/licenses/cpl.php.
  *
  * 2) under the terms of the "The BSD License" a copy of which is in the
file
  *    LICENSE2.txt in the root directory. The license is also available
from
  *    the Open Source Initiative, see
  *    http://www.opensource.org/licenses/bsd-license.php.
  *
  * 3) under the terms of the "GNU General Public License (GPL) Version
2" a
  *    copy of which is in the file LICENSE3.txt in the root directory.
The
  *    license is also available from the Open Source Initiative, see
  *    http://www.opensource.org/licenses/gpl-license.php.
  *
  * Licensee has the right to choose one of the above licenses.
  *
  * Redistributions of source code must retain the above copyright
  * notice and one of the license notices.
  *
  * Redistributions in binary form must reproduce both the above
copyright
  * notice, one of the license notices in the documentation
  * and/or other materials provided with the distribution.
  */
/**********************************************************************
  *
  * HEADER: dat_iw_extensions.h
  *
  * PURPOSE: extensions to the DAT API for iWARP transport specific
services
  *     NOTE:
  *         Applications MUST recompile with new dat.h definitions
  * 	  and include this file.
  *
  * Description: Header file for "uDAPL: User Direct Access Programming
  *  Library, Version: 2.0"
  *
  * Mapping rules:
  *      All global symbols are prepended with "DAT_" or "dat_"
  *      All DAT objects have an 'api' tag which, such as 'ep' or 'lmr'
  *      The method table is in the provider definition structure.
  *
  *
  **********************************************************************/
#ifndef _DAT_IW_EXTENSIONS_H_
#define _DAT_IW_EXTENSIONS_H_

/*
  * Provider specific attribute strings for extension support
  * returned with dat_ia_query() and
  * DAT_PROVIDER_ATTR_MASK ==
DAT_PROVIDER_FIELD_PROVIDER_SPECIFIC_ATTR
  *
  * DAT_NAMED_ATTR name == extended operations and version,
  * 	 version_value = version number of extension API
  */
#define DAT_EXTENSION_ATTR 	 "DAT_EXTENSION_INTERFACE"
#define DAT_EXTENSION_ATTR_VERSION  "DAT_EXTENSION_VERSION"
#define DAT_EXTENSION_ATTR_VERSION_VALUE "2.0.1"
#define DAT_IW_ATTR_SSP 		 "DAT_IW_ATTR_SSP"


/*
  * The iWARP extension supports Socket Service Points
  * These enable establishing a TCP connection first, then converting
  * it to RDMA usage.
  */
typedef enum dat_handle_iw_ext_type {
	 DAT_IW_HANDLE_TYPE_SSP = DAT_HANDLE_TYPE_EXTENSION_BASE,
} DAT_HANDLE_IW_EXT_TYPE;
typedef DAT_HANDLE DAT_IW_SSP_HANDLE;

typedef enum dat_iw_ssp_state
{
	 DAT_IW_SSP_STATE_OPERATIONAL,
	 DAT_IW_SSP_STATE_NON_OPERATIONAL
} DAT_IW_SSP_STATE;

/* The Socket ID is a local identifier for the socket. It is used for
Socket
   * based connection model. The format of the DAT_IW_SOCKET follows the
native
   * sockets programming practice of each target Operating System. The
Consumer must
   * follow normal sockets programming procedures provided by the host
platform.
   *
   * This include assumes that a socket handle is an int unless we are in
   * Linux Kernel code, in which case it is a struct socket pointer. This
   * distinction could be moved either to a distinct file (for iWARP
specific,
   * platform specific types) or to dat_platform_specific.h (where the
fact
   * that it was iWARP specific would be awkward). The coin flip was
heads,
   * so the awkwardness of platform specific typing is in the iWARP
specific
   * file instead.
  */
#if defined(__linux__)  &&  defined(__KERNEL__)
typyedef struct socket *DAT_IW_SOCKET;
#else
typedef int DAT_IW_SOCKET;
#endif

typedef struct dat_iw_ssp_param
{
	 DAT_IA_HANDLE ia_handle;
	 DAT_IW_SOCKET socket_id;
	 DAT_EVD_HANDLE evd_handle;
	 DAT_EP_HANDLE ep_handle;
	 DAT_IW_SSP_STATE ssp_state;
} DAT_IW_SSP_PARAM;

typedef enum dat_iw_ssp_param_mask
{
	 DAT_IW_SSP_FIELD_IA_HANDLE = 0x01,
	 DAT_IW_SSP_FIELD_SOCKET_ID = 0x02,
	 DAT_IW_SSP_FIELD_EVD_HANDLE = 0x04,
	 DAT_IW_SSP_FIELD_EP_HANDLE = 0x08,
	 DAT_IW_SSP_FIELD_SSP_STATE = 0x10,
	 DAT_IW_SSP_FIELD_ALL = 0x1F
} DAT_IW_SSP_PARAM_MASK;


/*
  * Extension operations
  */
typedef enum dat_iw_op
{
	 DAT_IW_SSP_CREATE_OP,
	 DAT_IW_SSP_FREE_OP,
	 DAT_IW_SSP_QUERY_OP,
	 DAT_IW_SOCKET_CONNECT_OP
} DAT_IW_OP;

/*
  * Extended IW transport specific APIs
  *  redirection via DAT extension function
  */

/* dat_iw_ssp_create creates an iWARP Socket Service Point given an
  * already connected Socket
  */

#define dat_iw_ssp_create (ia,socket_id,ep_handle,evd_handle,\
                            final_sm_msg,final_sm_msg_len,ssp_handle) \
		 dat_extension_op (ia,DAT_IW_SSP_CREATE_OP,ia,socket_id,
\

ep_handle,evd_handle,final_sm_msg, \
					 final_sm_msg_len,ssp_handle)

/* dat_iw_ssp_free destroys the specified instance of an SSP
  */
#define dat_iw_ssp_free (ssp_handle) \
	 dat_extension_op (ssp_handle,DAT_IW_SSP_FREE_OP)

/* dat_iw_ssq_query obtains the SSP_PARAMs for an ssp
  */
#define dat_iw_ssp_query (ssp_handle,ssp_param_mask,ssp_param) \
	 dat_extension_op (ssp_handle,DAT_IW_SSP_QUERY_OP, \
		 ssp_param_maks,ssp_param)

/* dat_iw_socket_connect initates a connection using an SSP
  */
#define dat_iw_socket_connect (ep_handle,socket_id,timeout, \
				        private_data_size,private_data) \
	 dat_extension_op (ep_handle,DAT_IW_SOCKET_CONNECT_OP,\
			       socket_id,timeout,private_data_size,\
				 private_data)

#endif /* _DAT_IB_EXTENSIONS_H_ */

Oracle vote forwarded.

Arkady Kanevsky                       email: arkady@...
Network Appliance Inc.               phone: 781-768-5395
1601 Trapelo Rd. - Suite 16.        Fax: 781-895-1195
Waltham, MA 02451                   central phone: 781-768-5300


> -----Original Message-----
> From: Richard Frank [mailto:Richard.Frank@...]
> Sent: Wednesday, January 03, 2007 3:25 PM
> To: Kanevsky, Arkady
> Cc: Sumanta Chatterjee
> Subject: Re: FW: [dat-discussions] DAT 2.0 RATIFICATION VOTE
>
> Arkady, we vote YES for all - is there more that I need to do ?
>
>
> On Wed, 2007-01-03 at 15:08 -0500, Kanevsky, Arkady wrote:
> > Rich or Sumanta,
> > Can one of you cast Oracle vote on DAT 2.0 ratification?
> > Thanks,
> > Arkady
> >
> >
> > Arkady Kanevsky                       email: arkady@...
> >
> > Network Appliance Inc.               phone: 781-768-5395
> >
> > 1601 Trapelo Rd. - Suite 16.        Fax: 781-895-1195
> >
> > Waltham, MA 02451                   central phone: 781-768-5300
> >
> >
> >
> >
> >
> >
> ______________________________________________________________
> >         From: Kanevsky, Arkady
> >         Sent: Friday, December 01, 2006 2:35 PM
> >         To: dat-discussions@yahoogroups.com
> >         Subject: [dat-discussions] DAT 2.0 RATIFICATION VOTE
> >
> >
> >
> >         Dear group,
> >         we are starting the clock for the DAT 2.0 ratification
> >         process.
> >         This time we are ratifing 5 documents:
> >         uDAPL 2.0, kDAPL 2.0, IB extension document,
> >         iWARP extension document, and Forward Compatibilty
> Transition
> >         document.
> >
> >         All 5 documents are on DAT reflector.
> >         If you have problem accessing them, please, contact me.
> >
> >         The deadline for comments is Wednesday Dec 20.
> >         The deadline for the ratification vote is Friday Dec 22.
> >
> >         Thanks,
> >         Arkady
> >         Chair of DAT Collaborative
> >
> >
> >
> >         Arkady Kanevsky                       email:
> arkady@...
> >
> >         Network Appliance Inc.               phone: 781-768-5395
> >
> >         1601 Trapelo Rd. - Suite 16.        Fax: 781-895-1195
> >
> >         Waltham, MA 02451                   central phone:
> >         781-768-5300
> >
> >
> >
> >
> >
>

I will be out of the office starting 12/26/2006 and will not return until 01/08/2007.

QLogic votes yes on all documents.

Johann

Hello,

This email message is a notification to let you know that
a file has been uploaded to the Files area of the dat-discussions
group.

   File        : /Minutes/DAT-MIN-12_22_06.pdf
   Uploaded by : arkadynetappcom <arkady@...>
   Description : minutes of DAT meeting 12/22/2006

You can access this file at the URL:
http://groups.yahoo.com/group/dat-discussions/files/Minutes/DAT-MIN-12_22_06.pdf

To learn more about file sharing for your group, please visit:
http://help.yahoo.com/help/us/groups/files

Regards,

arkadynetappcom <arkady@...>

dat-discussions@yahoogroups.com wrote:
> Please, post your votes on the reflector!

Lightfleet votes yes on all documents.

dat-discussions@yahoogroups.com wrote:
> Please, post your votes on the reflector!
>
>
Broadcom votes Yes on all documents.