From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <users-bounces@dpdk.org> Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1B8FB459DA for <public@inbox.dpdk.org>; Thu, 19 Sep 2024 18:45:55 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B8C84406FF; Thu, 19 Sep 2024 18:45:54 +0200 (CEST) Received: from mail-lj1-f182.google.com (mail-lj1-f182.google.com [209.85.208.182]) by mails.dpdk.org (Postfix) with ESMTP id 8ABC64026B for <users@dpdk.org>; Thu, 19 Sep 2024 18:45:52 +0200 (CEST) Received: by mail-lj1-f182.google.com with SMTP id 38308e7fff4ca-2f75de9a503so10052121fa.0 for <users@dpdk.org>; Thu, 19 Sep 2024 09:45:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1726764352; x=1727369152; darn=dpdk.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=MkULQYjRANrCihy4kJHoNInayxSAACqSepljrJtQAhY=; b=EmWaRkrJb44SkHDDWQ3H1g6+5rbIrWAM82VTWA1eXdXSPP3ZwbtdqqMTkeMZH8nTAA d/FphEG1vyhdbckifB669IupliNCF2DAx3wNfWr+Kyc3sWmZGHVw0DVw//8upAG4Apyo W7jY/Pbz/TeVoP3UGpax4mYRz6qwEfPdyHEx4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726764352; x=1727369152; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=MkULQYjRANrCihy4kJHoNInayxSAACqSepljrJtQAhY=; b=RF3Yek57p6yH/HfULN4mR3eJ7vCm7x+d2PUEMIdGyIn8+FHZpc4vCzaWM/rHc+ai8r BqGp/zlngHlwBKqWqREZ/CCdtajuijtEv4SsrWAnuWX1aHslXu+CsJtGh0KdQHLWDVNH wqTZTT0K31I1zgjd9lZN5Ktu0s1WqG1X6N6fb8sp0jKD3/dTHwjtjek8CUROcbaTpbi5 MfYW9ptgAhsuPOz9s/3ZYSYEn2ohswu9M1gxbIiikojzsxYqcJ7HSJpan6iq2kcVXbFL O8MZRUEItMVqbvWnzVOEiGPsLiZhj7eeB5eYZv8GpV6CUTBR/PjrtXdtdN0QftT05zBu LCjA== X-Forwarded-Encrypted: i=1; AJvYcCUnCTobpUj3iy0DJKx5Ah/WKPpYL5LmzI/qWx/+w+RI8bzAjDlNX8X3SXLRY9JrlAXJkv8z8A==@dpdk.org X-Gm-Message-State: AOJu0YyGoki/T8A6mbYTtQeTYTEpGYK6quPprQ5fScqm9fX2+F3OVkCW 0rsHCoQyPOptrlHpIrCqIt9oJLbTWdoV4IwvYb1ar4Df/ToS2x1+tZOJXWmELSB8S04a8uiHj/Z 5hSnGEk2vBeosf7u57u8DWs6QhasrSMdh9M1H+14cYwHlqTccA9sORWMOkjqs6bG9pVNkJ/xjtN dKR6fSf+E= X-Google-Smtp-Source: AGHT+IHTxVAvxq8NrH5f+8vlc5BicOtMd2bB/4RYfaWhMWNWWkRdZzhrbgbnlYtqMvO7XbIbQgHGuoHWytUgCUnzIb0= X-Received: by 2002:a2e:a543:0:b0:2f1:a509:ce66 with SMTP id 38308e7fff4ca-2f7cb2cfe55mr1687991fa.5.1726764351493; Thu, 19 Sep 2024 09:45:51 -0700 (PDT) MIME-Version: 1.0 References: <CAAkQrK-quv_w+1KG=H3yDsiN9Xki9DigucZCy7q_RwMLmgCYyg@mail.gmail.com> <20240903170350.7e663864@hermes.local> <CAAkQrK-gvVUsdyK+EiuaWZk9U7-CRZaJXHAELbV5qjMzmMv82A@mail.gmail.com> <20240904154246.1c5bbb58@hermes.local> <PH0PR21MB4456D6BCB98F6E6F75D0D25DCE9D2@PH0PR21MB4456.namprd21.prod.outlook.com> <CAAkQrK8uYciOV+-tgEvtGhE1UGTqyQ__zio1nVisXso6VfRp-g@mail.gmail.com> <20240912160948.3714f01d@hermes.local> <CAAkQrK_HjT3H-xBtYdskrJ+oykf+djxOM41idg2w+7mW9EGytw@mail.gmail.com> <PH0PR21MB4456B834D4CCF1D5F4987075CE652@PH0PR21MB4456.namprd21.prod.outlook.com> <CAAkQrK8FZ3deWyuzzKdmJX3yqmW9Bqbgqh2B1nNs1iDaPzJaNw@mail.gmail.com> <CAAkQrK-WAQHnZ0L5KO1PZdRqapoQ9voWpNbZuOFihwG3jaYVnw@mail.gmail.com> <PH0PR21MB445658CC7FE9605D627FBDB3CE612@PH0PR21MB4456.namprd21.prod.outlook.com> In-Reply-To: <PH0PR21MB445658CC7FE9605D627FBDB3CE612@PH0PR21MB4456.namprd21.prod.outlook.com> From: Nandini Rangaswamy <nandini.rangaswamy@broadcom.com> Date: Thu, 19 Sep 2024 09:45:39 -0700 Message-ID: <CAAkQrK9My23YMa+=uZu+8pdPp9ERZErEZDVVGY9sgvK3eBwrxQ@mail.gmail.com> Subject: Re: Netvsc vs Failsafe Performance To: Long Li <longli@microsoft.com> Cc: Stephen Hemminger <stephen@networkplumber.org>, "users@dpdk.org" <users@dpdk.org> Content-Type: multipart/alternative; boundary="000000000000c8ae1b06227ba79a" X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions <users.dpdk.org> List-Unsubscribe: <https://mails.dpdk.org/options/users>, <mailto:users-request@dpdk.org?subject=unsubscribe> List-Archive: <http://mails.dpdk.org/archives/users/> List-Post: <mailto:users@dpdk.org> List-Help: <mailto:users-request@dpdk.org?subject=help> List-Subscribe: <https://mails.dpdk.org/listinfo/users>, <mailto:users-request@dpdk.org?subject=subscribe> Errors-To: users-bounces@dpdk.org --000000000000c8ae1b06227ba79a Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Long Li, I shall test with UDP traffic and get back to you. Regards, Nandini On Tue, Sep 17, 2024 at 2:57=E2=80=AFPM Long Li <longli@microsoft.com> wrot= e: > Thank you! > > > > Are you seeing problems with UDP traffic on the receive side? If > everything works fine for you, I=E2=80=99m sending a patch. > > > > Long > > > > *From:* Nandini Rangaswamy <nandini.rangaswamy@broadcom.com> > *Sent:* Monday, September 16, 2024 3:58 PM > *To:* Long Li <longli@microsoft.com> > *Cc:* Stephen Hemminger <stephen@networkplumber.org>; users@dpdk.org > *Subject:* Re: Netvsc vs Failsafe Performance > > > > Hi Long, > > I tested this patch and it works as expected. The UDP IPv6 RSS offload bi= t > is set and my dpdk app is able to successfully configure the netvsc port. > > Regards, > > Nandini > > > > On Fri, Sep 13, 2024 at 2:29=E2=80=AFPM Nandini Rangaswamy < > nandini.rangaswamy@broadcom.com> wrote: > > Thanks Long Li. > > I shall try this patch and get back to you. > > > > On Fri, Sep 13, 2024 at 2:27=E2=80=AFPM Long Li <longli@microsoft.com> wr= ote: > > It=E2=80=99s a bug in netvsc for not reporting RTE_ETH_RSS_NONFRAG_IPV6_U= DP. It is > implied as in the case in IPV4. > > > > Can you try the following patch? > > > > diff --git a/drivers/net/netvsc/hn_rndis.c b/drivers/net/netvsc/hn_rndis.= c > > index 1ba75ee804..fe1f04d8d9 100644 > > --- a/drivers/net/netvsc/hn_rndis.c > > +++ b/drivers/net/netvsc/hn_rndis.c > > @@ -717,6 +717,7 @@ hn_rndis_query_rsscaps(struct hn_data *hv, > > if (caps.ndis_caps & NDIS_RSS_CAP_IPV6) > > hv->rss_offloads |=3D RTE_ETH_RSS_IPV6 > > | RTE_ETH_RSS_NONFRAG_IPV6_TCP; > > + | RTE_ETH_RSS_NONFRAG_IPV6_UDP; > > if (caps.ndis_caps & NDIS_RSS_CAP_IPV6_EX) > > hv->rss_offloads |=3D RTE_ETH_RSS_IPV6_EX > > | RTE_ETH_RSS_IPV6_TCP_EX; > > > > > > *From:* Nandini Rangaswamy <nandini.rangaswamy@broadcom.com> > *Sent:* Friday, September 13, 2024 10:56 AM > *To:* Stephen Hemminger <stephen@networkplumber.org> > *Cc:* Long Li <longli@microsoft.com>; users@dpdk.org > *Subject:* Re: Netvsc vs Failsafe Performance > > > > Thanks for clarifying the question regarding Txd size Stephen. > > I tested out the RSS for TCP UDP. > > As suggested , I set the TCP flags alone in RSS conf and configured the > netvsc port. > > > > struct rte_eth_conf conf =3D { > > .intr_conf =3D { > > .lsc =3D !dpdk.lsc_intr_disable && !dpdk_if->lsc_intr_disable && > > !!(dev->data->dev_flags & RTE_ETH_DEV_INTR_LSC), > > }, > > .rxmode =3D { > > .mq_mode =3D RTE_ETH_MQ_RX_RSS, > > .offloads =3D RTE_ETH_RX_OFFLOAD_VLAN_STRIP | RTE_ETH_RX_OFFLOAD_IPV4_CKS= UM > | > > RTE_ETH_RX_OFFLOAD_RSS_HASH | RTE_ETH_RX_OFFLOAD_UDP_CKSUM, > > }, > > .rx_adv_conf.rss_conf =3D { > > .rss_hf =3D RTE_ETH_RSS_NONFRAG_IPV4_TCP | RTE_ETH_RSS_NONFRAG_IPV6_TCP, > > .rss_key =3D conf_rss_key, > > .rss_key_len =3D rss_key_len, > > }, > > .txmode =3D { > > .offloads =3D RTE_ETH_TX_OFFLOAD_UDP_CKSUM | RTE_ETH_TX_OFFLOAD_IPV4_CKSU= M, > > }, > > }; > > *rte_eth_dev_configure*(<netvsc port>, num_rxq,num_txq, &conf); > > uint8_t rss_key_temp[64]; > > struct rte_eth_rss_conf rss_conf =3D { > > .rss_key =3D rss_key_temp, > > .rss_key_len =3D sizeof(rss_key_temp), > > }; > > ret =3D *rte_eth_dev_rss_hash_conf_get*(<VF port>, &rss_conf); > > > > > > Now the VF port RSS offloads show only TCP flags set and not UDP. I > assumed that even the UDP flags might be set. Is this expected ? > > > > Regards, > > Nandini > > > > > > On Thu, Sep 12, 2024 at 4:09=E2=80=AFPM Stephen Hemminger < > stephen@networkplumber.org> wrote: > > On Thu, 12 Sep 2024 13:47:37 -0700 > Nandini Rangaswamy <nandini.rangaswamy@broadcom.com> wrote: > > > Thanks for your response Long Li. > > I see with netvsc the maximum number of Tx descriptors is restricted to > > 4096 whereas the number of Rx descriptors is restricted to 8192. > > But, for failsafe PMD , we see that both the number of Txd and Rxd is > > restricted to 8192. > > How is netvsc PMD giving the same performance as failsafe PMD ? > > > > Regards > > I think the limits there were somewhat arbitrary chose with netvsc. > Don't remember a hard reason that would block larger sizes. > > > Having really big rings won't help performance (i.e BufferBloat) and > could a lot of memory consumption. When all heavy data traffic goes throu= gh > the VF and that ring is different. Only DoS attacks should be impacted > by rx/tx descriptor limits in the netvsc device. The linux driver actuall= y > has much smaller buffer. > > > This electronic communication and the information and any files > transmitted with it, or attached to it, are confidential and are intended > solely for the use of the individual or entity to whom it is addressed an= d > may contain information that is confidential, legally privileged, protect= ed > by privacy laws, or otherwise restricted from disclosure to anyone else. = If > you are not the intended recipient or the person responsible for deliveri= ng > the e-mail to the intended recipient, you are hereby notified that any us= e, > copying, distributing, dissemination, forwarding, printing, or copying of > this e-mail is strictly prohibited. If you received this e-mail in error, > please return the e-mail to the sender, delete it from your computer, and > destroy any printed copy of it. > > > This electronic communication and the information and any files > transmitted with it, or attached to it, are confidential and are intended > solely for the use of the individual or entity to whom it is addressed an= d > may contain information that is confidential, legally privileged, protect= ed > by privacy laws, or otherwise restricted from disclosure to anyone else. = If > you are not the intended recipient or the person responsible for deliveri= ng > the e-mail to the intended recipient, you are hereby notified that any us= e, > copying, distributing, dissemination, forwarding, printing, or copying of > this e-mail is strictly prohibited. If you received this e-mail in error, > please return the e-mail to the sender, delete it from your computer, and > destroy any printed copy of it. > --=20 This electronic communication and the information and any files transmitted= =20 with it, or attached to it, are confidential and are intended solely for=20 the use of the individual or entity to whom it is addressed and may contain= =20 information that is confidential, legally privileged, protected by privacy= =20 laws, or otherwise restricted from disclosure to anyone else. If you are=20 not the intended recipient or the person responsible for delivering the=20 e-mail to the intended recipient, you are hereby notified that any use,=20 copying, distributing, dissemination, forwarding, printing, or copying of= =20 this e-mail is strictly prohibited. If you received this e-mail in error,= =20 please return the e-mail to the sender, delete it from your computer, and= =20 destroy any printed copy of it. --000000000000c8ae1b06227ba79a Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">Hi Long Li,<div>I shall test with UDP traffic and get=C2= =A0back to you.</div><div>Regards,</div><div>Nandini</div></div><br><div cl= ass=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Tue, Sep 17, 2= 024 at 2:57=E2=80=AFPM Long Li <<a href=3D"mailto:longli@microsoft.com">= longli@microsoft.com</a>> wrote:<br></div><blockquote class=3D"gmail_quo= te" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204= );padding-left:1ex"><div class=3D"msg-2267519294106871571"> <div lang=3D"EN-US" style=3D"overflow-wrap: break-word;"> <div class=3D"m_-2267519294106871571WordSection1"> <p class=3D"MsoNormal">Thank you!<u></u><u></u></p> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <p class=3D"MsoNormal">Are you seeing problems with UDP traffic on the rece= ive side? If everything works fine for you, I=E2=80=99m sending a patch.<u>= </u><u></u></p> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <p class=3D"MsoNormal">Long<u></u><u></u></p> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <div style=3D"border-top:none;border-right:none;border-bottom:none;border-l= eft:1.5pt solid blue;padding:0in 0in 0in 4pt"> <div> <div style=3D"border-right:none;border-bottom:none;border-left:none;border-= top:1pt solid rgb(225,225,225);padding:3pt 0in 0in"> <p class=3D"MsoNormal"><b><span style=3D"font-size:11pt;font-family:Calibri= ,sans-serif">From:</span></b><span style=3D"font-size:11pt;font-family:Cali= bri,sans-serif"> Nandini Rangaswamy <<a href=3D"mailto:nandini.rangaswam= y@broadcom.com" target=3D"_blank">nandini.rangaswamy@broadcom.com</a>> <br> <b>Sent:</b> Monday, September 16, 2024 3:58 PM<br> <b>To:</b> Long Li <<a href=3D"mailto:longli@microsoft.com" target=3D"_b= lank">longli@microsoft.com</a>><br> <b>Cc:</b> Stephen Hemminger <<a href=3D"mailto:stephen@networkplumber.o= rg" target=3D"_blank">stephen@networkplumber.org</a>>; <a href=3D"mailto= :users@dpdk.org" target=3D"_blank">users@dpdk.org</a><br> <b>Subject:</b> Re: Netvsc vs Failsafe Performance<u></u><u></u></span></p> </div> </div> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <div> <p class=3D"MsoNormal">Hi=C2=A0Long,<u></u><u></u></p> <div> <p class=3D"MsoNormal">I tested this patch and it works as expected. The UD= P IPv6 RSS offload bit is set and my dpdk app is able to successfully confi= gure the netvsc port.<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">Regards,<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">Nandini<u></u><u></u></p> </div> </div> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <div> <div> <p class=3D"MsoNormal">On Fri, Sep 13, 2024 at 2:29<span style=3D"font-fami= ly:Arial,sans-serif">=E2=80=AF</span>PM Nandini Rangaswamy <<a href=3D"m= ailto:nandini.rangaswamy@broadcom.com" target=3D"_blank">nandini.rangaswamy= @broadcom.com</a>> wrote:<u></u><u></u></p> </div> <blockquote style=3D"border-top:none;border-right:none;border-bottom:none;b= order-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin-left:4= .8pt;margin-right:0in"> <div> <p class=3D"MsoNormal">Thanks Long Li.<u></u><u></u></p> <div> <p class=3D"MsoNormal">I shall try this patch and get back to you.<u></u><u= ></u></p> </div> </div> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <div> <div> <p class=3D"MsoNormal">On Fri, Sep 13, 2024 at 2:27<span style=3D"font-fami= ly:Arial,sans-serif">=E2=80=AF</span>PM Long Li <<a href=3D"mailto:longl= i@microsoft.com" target=3D"_blank">longli@microsoft.com</a>> wrote:<u></= u><u></u></p> </div> <blockquote style=3D"border-top:none;border-right:none;border-bottom:none;b= order-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin-left:4= .8pt;margin-right:0in"> <div> <div> <div> <p class=3D"MsoNormal"><span style=3D"font-size:11pt">It=E2=80=99s a bug in= netvsc for not reporting RTE_ETH_RSS_NONFRAG_IPV6_UDP. It is implied as in= the case in IPV4.</span><u></u><u></u></p> <p class=3D"MsoNormal"><span style=3D"font-size:11pt">=C2=A0</span><u></u><= u></u></p> <p class=3D"MsoNormal"><span style=3D"font-size:11pt">Can you try the follo= wing patch?</span><u></u><u></u></p> <p class=3D"MsoNormal"><span style=3D"font-size:11pt">=C2=A0</span><u></u><= u></u></p> <p class=3D"MsoNormal"><span style=3D"font-size:11pt">diff --git a/drivers/= net/netvsc/hn_rndis.c b/drivers/net/netvsc/hn_rndis.c</span><u></u><u></u><= /p> <p class=3D"MsoNormal"><span style=3D"font-size:11pt">index 1ba75ee804..fe1= f04d8d9 100644</span><u></u><u></u></p> <p class=3D"MsoNormal"><span style=3D"font-size:11pt">--- a/drivers/net/net= vsc/hn_rndis.c</span><u></u><u></u></p> <p class=3D"MsoNormal"><span style=3D"font-size:11pt">+++ b/drivers/net/net= vsc/hn_rndis.c</span><u></u><u></u></p> <p class=3D"MsoNormal"><span style=3D"font-size:11pt">@@ -717,6 +717,7 @@ h= n_rndis_query_rsscaps(struct hn_data *hv,</span><u></u><u></u></p> <p class=3D"MsoNormal"><span style=3D"font-size:11pt">=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 if (caps.ndis_caps & NDIS_RSS_CAP_IPV6)</span><u>= </u><u></u></p> <p class=3D"MsoNormal"><span style=3D"font-size:11pt">=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 hv-&g= t;rss_offloads |=3D RTE_ETH_RSS_IPV6</span><u></u><u></u></p> <p class=3D"MsoNormal"><span style=3D"font-size:11pt">=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 | RTE_ETH_RSS_NONFRAG_IPV6_TCP;<= /span><u></u><u></u></p> <p class=3D"MsoNormal"><span style=3D"font-size:11pt">+=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 | RTE_ETH_RSS_NONFRAG_IPV6_UDP;</sp= an><u></u><u></u></p> <p class=3D"MsoNormal"><span style=3D"font-size:11pt">=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 if (caps.ndis_caps & NDIS_RSS_CAP_IPV6_EX)</span>= <u></u><u></u></p> <p class=3D"MsoNormal"><span style=3D"font-size:11pt">=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 hv-&g= t;rss_offloads |=3D RTE_ETH_RSS_IPV6_EX</span><u></u><u></u></p> <p class=3D"MsoNormal"><span style=3D"font-size:11pt">=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 | RTE_ETH_RSS_IPV6_TCP_EX;</span= ><u></u><u></u></p> <p class=3D"MsoNormal"><span style=3D"font-size:11pt">=C2=A0</span><u></u><= u></u></p> <p class=3D"MsoNormal"><span style=3D"font-size:11pt">=C2=A0</span><u></u><= u></u></p> <div style=3D"border-top:none;border-right:none;border-bottom:none;border-l= eft:1.5pt solid blue;padding:0in 0in 0in 4pt"> <div> <div style=3D"border-right:none;border-bottom:none;border-left:none;border-= top:1pt solid rgb(225,225,225);padding:3pt 0in 0in"> <p class=3D"MsoNormal"><b><span style=3D"font-size:11pt;font-family:Calibri= ,sans-serif">From:</span></b><span style=3D"font-size:11pt;font-family:Cali= bri,sans-serif"> Nandini Rangaswamy <<a href=3D"mailto:nandini.rangaswam= y@broadcom.com" target=3D"_blank">nandini.rangaswamy@broadcom.com</a>> <br> <b>Sent:</b> Friday, September 13, 2024 10:56 AM<br> <b>To:</b> Stephen Hemminger <<a href=3D"mailto:stephen@networkplumber.o= rg" target=3D"_blank">stephen@networkplumber.org</a>><br> <b>Cc:</b> Long Li <<a href=3D"mailto:longli@microsoft.com" target=3D"_b= lank">longli@microsoft.com</a>>; <a href=3D"mailto:users@dpdk.org" target=3D"_blank">users@dpdk.org</a><br> <b>Subject:</b> Re: Netvsc vs Failsafe Performance</span><u></u><u></u></p> </div> </div> <p class=3D"MsoNormal">=C2=A0<u></u><u></u></p> <div> <div> <p class=3D"MsoNormal">Thanks for clarifying the question=C2=A0regarding Tx= d size Stephen.<u></u><u></u></p> <div> <p class=3D"MsoNormal">I tested out the RSS for TCP UDP.<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">As suggested , I set the TCP flags alone in RSS conf= and configured the netvsc port.<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">=C2=A0<u></u><u></u></p> </div> <div> <div> <div> <p class=3D"MsoNormal" style=3D"line-height:13.5pt;background:whitesmoke"> <span style=3D"font-size:9pt;font-family:"Courier New";color:rgb(= 122,62,157)">struct</span><span style=3D"font-size:9pt;font-family:"Co= urier New";color:rgb(51,51,51)"> rte_eth_conf conf </span><span style=3D"font-size:9pt;font-family:"Courier New";col= or:rgb(119,119,119)">=3D</span><span style=3D"font-size:9pt;font-family:&qu= ot;Courier New";color:rgb(51,51,51)"> </span><span style=3D"font-size:9pt;font-family:"Courier New";col= or:rgb(119,119,119)">{</span><u></u><u></u></p> </div> <div> <p class=3D"MsoNormal" style=3D"line-height:13.5pt;background:whitesmoke"> <span style=3D"font-size:9pt;font-family:"Courier New";color:rgb(= 51,51,51)">.intr_conf </span> <span style=3D"font-size:9pt;font-family:"Courier New";color:rgb(= 119,119,119)">=3D</span><span style=3D"font-size:9pt;font-family:"Cour= ier New";color:rgb(51,51,51)"> </span><span style=3D"font-size:9pt;font-family:"Courier New";col= or:rgb(119,119,119)">{</span><u></u><u></u></p> </div> <div> <p class=3D"MsoNormal" style=3D"line-height:13.5pt;background:whitesmoke"> <span style=3D"font-size:9pt;font-family:"Courier New";color:rgb(= 51,51,51)">.lsc </span> <span style=3D"font-size:9pt;font-family:"Courier New";color:rgb(= 119,119,119)">=3D</span><span style=3D"font-size:9pt;font-family:"Cour= ier New";color:rgb(51,51,51)"> </span><span style=3D"font-size:9pt;font-family:"Courier New";col= or:rgb(119,119,119)">!</span><span style=3D"font-size:9pt;font-family:"= ;Courier New";color:rgb(122,62,157)">dpdk</span><span style=3D"font-si= ze:9pt;font-family:"Courier New";color:rgb(119,119,119)">.</span>= <span style=3D"font-size:9pt;font-family:"Courier New";color:rgb(= 122,62,157)">lsc_intr_disable</span><span style=3D"font-size:9pt;font-famil= y:"Courier New";color:rgb(51,51,51)"> </span><span style=3D"font-size:9pt;font-family:"Courier New";col= or:rgb(119,119,119)">&&</span><span style=3D"font-size:9pt;font-fam= ily:"Courier New";color:rgb(51,51,51)"> </span><span style=3D"font-size:9pt;font-family:"Courier New";col= or:rgb(119,119,119)">!</span><span style=3D"font-size:9pt;font-family:"= ;Courier New";color:rgb(122,62,157)">dpdk_if</span><span style=3D"font= -size:9pt;font-family:"Courier New";color:rgb(119,119,119)">->= </span><span style=3D"font-size:9pt;font-family:"Courier New";col= or:rgb(122,62,157)">lsc_intr_disable</span><span style=3D"font-size:9pt;fon= t-family:"Courier New";color:rgb(51,51,51)"> </span><span style=3D"font-size:9pt;font-family:"Courier New";col= or:rgb(119,119,119)">&&</span><u></u><u></u></p> </div> <div> <p class=3D"MsoNormal" style=3D"line-height:13.5pt;background:whitesmoke"> <span style=3D"font-size:9pt;font-family:"Courier New";color:rgb(= 119,119,119)">!!(</span><span style=3D"font-size:9pt;font-family:"Cour= ier New";color:rgb(122,62,157)">dev</span><span style=3D"font-size:9pt= ;font-family:"Courier New";color:rgb(119,119,119)">-></span><s= pan style=3D"font-size:9pt;font-family:"Courier New";color:rgb(12= 2,62,157)">data</span><span style=3D"font-size:9pt;font-family:"Courie= r New";color:rgb(119,119,119)">-></span><span style=3D"font-size:9p= t;font-family:"Courier New";color:rgb(122,62,157)">dev_flags</spa= n><span style=3D"font-size:9pt;font-family:"Courier New";color:rg= b(51,51,51)"> </span><span style=3D"font-size:9pt;font-family:"Courier New";col= or:rgb(119,119,119)">&</span><span style=3D"font-size:9pt;font-family:&= quot;Courier New";color:rgb(51,51,51)"> RTE_ETH_DEV_INTR_LSC</span><sp= an style=3D"font-size:9pt;font-family:"Courier New";color:rgb(119= ,119,119)">),</span><u></u><u></u></p> </div> <div> <p class=3D"MsoNormal" style=3D"line-height:13.5pt;background:whitesmoke"> <span style=3D"font-size:9pt;font-family:"Courier New";color:rgb(= 119,119,119)">},</span><u></u><u></u></p> </div> <div> <p class=3D"MsoNormal" style=3D"line-height:13.5pt;background:whitesmoke"> <span style=3D"font-size:9pt;font-family:"Courier New";color:rgb(= 51,51,51)">.rxmode </span> <span style=3D"font-size:9pt;font-family:"Courier New";color:rgb(= 119,119,119)">=3D</span><span style=3D"font-size:9pt;font-family:"Cour= ier New";color:rgb(51,51,51)"> </span><span style=3D"font-size:9pt;font-family:"Courier New";col= or:rgb(119,119,119)">{</span><u></u><u></u></p> </div> <div> <p class=3D"MsoNormal" style=3D"line-height:13.5pt;background:whitesmoke"> <span style=3D"font-size:9pt;font-family:"Courier New";color:rgb(= 51,51,51)">.mq_mode </span> <span style=3D"font-size:9pt;font-family:"Courier New";color:rgb(= 119,119,119)">=3D</span><span style=3D"font-size:9pt;font-family:"Cour= ier New";color:rgb(51,51,51)"> RTE_ETH_MQ_RX_RSS</span><span style=3D"= font-size:9pt;font-family:"Courier New";color:rgb(119,119,119)">,= </span><u></u><u></u></p> </div> <div> <p class=3D"MsoNormal" style=3D"line-height:13.5pt;background:whitesmoke"> <span style=3D"font-size:9pt;font-family:"Courier New";color:rgb(= 51,51,51)">.offloads </span> <span style=3D"font-size:9pt;font-family:"Courier New";color:rgb(= 119,119,119)">=3D</span><span style=3D"font-size:9pt;font-family:"Cour= ier New";color:rgb(51,51,51)"> RTE_ETH_RX_OFFLOAD_VLAN_STRIP </span><span style=3D"font-size:9pt;font-family:"Courier New";col= or:rgb(119,119,119)">|</span><span style=3D"font-size:9pt;font-family:"= ;Courier New";color:rgb(51,51,51)"> RTE_ETH_RX_OFFLOAD_IPV4_CKSUM </span><span style=3D"font-size:9pt;font-family:"Courier New";col= or:rgb(119,119,119)">|</span><u></u><u></u></p> </div> <div> <p class=3D"MsoNormal" style=3D"line-height:13.5pt;background:whitesmoke"> <span style=3D"font-size:9pt;font-family:"Courier New";color:rgb(= 51,51,51)">RTE_ETH_RX_OFFLOAD_RSS_HASH </span><span style=3D"font-size:9pt;font-family:"Courier New";col= or:rgb(119,119,119)">|</span><span style=3D"font-size:9pt;font-family:"= ;Courier New";color:rgb(51,51,51)"> RTE_ETH_RX_OFFLOAD_UDP_CKSUM</span= ><span style=3D"font-size:9pt;font-family:"Courier New";color:rgb= (119,119,119)">,</span><u></u><u></u></p> </div> <div> <p class=3D"MsoNormal" style=3D"line-height:13.5pt;background:whitesmoke"> <span style=3D"font-size:9pt;font-family:"Courier New";color:rgb(= 119,119,119)">},</span><u></u><u></u></p> </div> <div> <p class=3D"MsoNormal" style=3D"line-height:13.5pt;background:whitesmoke"> <span style=3D"font-size:9pt;font-family:"Courier New";color:rgb(= 51,51,51)">.</span><span style=3D"font-size:9pt;font-family:"Courier N= ew";color:rgb(122,62,157)">rx_adv_conf</span><span style=3D"font-size:= 9pt;font-family:"Courier New";color:rgb(119,119,119)">.</span><sp= an style=3D"font-size:9pt;font-family:"Courier New";color:rgb(122= ,62,157)">rss_conf</span><span style=3D"font-size:9pt;font-family:"Cou= rier New";color:rgb(51,51,51)"> </span><span style=3D"font-size:9pt;font-family:"Courier New";col= or:rgb(119,119,119)">=3D</span><span style=3D"font-size:9pt;font-family:&qu= ot;Courier New";color:rgb(51,51,51)"> </span><span style=3D"font-size:9pt;font-family:"Courier New";col= or:rgb(119,119,119)">{</span><u></u><u></u></p> </div> <div> <p class=3D"MsoNormal" style=3D"line-height:13.5pt;background:whitesmoke"> <span style=3D"font-size:9pt;font-family:"Courier New";color:rgb(= 51,51,51)">.rss_hf </span> <span style=3D"font-size:9pt;font-family:"Courier New";color:rgb(= 119,119,119)">=3D</span><span style=3D"font-size:9pt;font-family:"Cour= ier New";color:rgb(51,51,51)"> RTE_ETH_RSS_NONFRAG_IPV4_TCP </span><span style=3D"font-size:9pt;font-family:"Courier New";col= or:rgb(119,119,119)">|</span><span style=3D"font-size:9pt;font-family:"= ;Courier New";color:rgb(51,51,51)"> RTE_ETH_RSS_NONFRAG_IPV6_TCP</span= ><span style=3D"font-size:9pt;font-family:"Courier New";color:rgb= (119,119,119)">,</span><u></u><u></u></p> </div> <div> <p class=3D"MsoNormal" style=3D"line-height:13.5pt;background:whitesmoke"> <span style=3D"font-size:9pt;font-family:"Courier New";color:rgb(= 51,51,51)">.rss_key </span> <span style=3D"font-size:9pt;font-family:"Courier New";color:rgb(= 119,119,119)">=3D</span><span style=3D"font-size:9pt;font-family:"Cour= ier New";color:rgb(51,51,51)"> conf_rss_key</span><span style=3D"font-= size:9pt;font-family:"Courier New";color:rgb(119,119,119)">,</spa= n><u></u><u></u></p> </div> <div> <p class=3D"MsoNormal" style=3D"line-height:13.5pt;background:whitesmoke"> <span style=3D"font-size:9pt;font-family:"Courier New";color:rgb(= 51,51,51)">.rss_key_len </span><span style=3D"font-size:9pt;font-family:"Courier New";col= or:rgb(119,119,119)">=3D</span><span style=3D"font-size:9pt;font-family:&qu= ot;Courier New";color:rgb(51,51,51)"> rss_key_len</span><span style=3D= "font-size:9pt;font-family:"Courier New";color:rgb(119,119,119)">= ,</span><u></u><u></u></p> </div> <div> <p class=3D"MsoNormal" style=3D"line-height:13.5pt;background:whitesmoke"> <span style=3D"font-size:9pt;font-family:"Courier New";color:rgb(= 119,119,119)">},</span><u></u><u></u></p> </div> <div> <p class=3D"MsoNormal" style=3D"line-height:13.5pt;background:whitesmoke"> <span style=3D"font-size:9pt;font-family:"Courier New";color:rgb(= 51,51,51)">.txmode </span> <span style=3D"font-size:9pt;font-family:"Courier New";color:rgb(= 119,119,119)">=3D</span><span style=3D"font-size:9pt;font-family:"Cour= ier New";color:rgb(51,51,51)"> </span><span style=3D"font-size:9pt;font-family:"Courier New";col= or:rgb(119,119,119)">{</span><u></u><u></u></p> </div> <div> <p class=3D"MsoNormal" style=3D"line-height:13.5pt;background:whitesmoke"> <span style=3D"font-size:9pt;font-family:"Courier New";color:rgb(= 51,51,51)">.offloads </span> <span style=3D"font-size:9pt;font-family:"Courier New";color:rgb(= 119,119,119)">=3D</span><span style=3D"font-size:9pt;font-family:"Cour= ier New";color:rgb(51,51,51)"> RTE_ETH_TX_OFFLOAD_UDP_CKSUM </span><span style=3D"font-size:9pt;font-family:"Courier New";col= or:rgb(119,119,119)">|</span><span style=3D"font-size:9pt;font-family:"= ;Courier New";color:rgb(51,51,51)"> RTE_ETH_TX_OFFLOAD_IPV4_CKSUM</spa= n><span style=3D"font-size:9pt;font-family:"Courier New";color:rg= b(119,119,119)">,</span><u></u><u></u></p> </div> <div> <p class=3D"MsoNormal" style=3D"line-height:13.5pt;background:whitesmoke"> <span style=3D"font-size:9pt;font-family:"Courier New";color:rgb(= 119,119,119)">},</span><u></u><u></u></p> </div> <div> <p class=3D"MsoNormal" style=3D"line-height:13.5pt;background:whitesmoke"> <span style=3D"font-size:9pt;font-family:"Courier New";color:rgb(= 119,119,119)">};</span><u></u><u></u></p> </div> </div> </div> <div> <div> <div> <p class=3D"MsoNormal" style=3D"line-height:13.5pt;background:whitesmoke"> <b><span style=3D"font-size:9pt;font-family:"Courier New";color:r= gb(170,55,49)">rte_eth_dev_configure</span></b><span style=3D"font-size:9pt= ;font-family:"Courier New";color:rgb(119,119,119)">(</span><span = style=3D"font-size:9pt;font-family:"Courier New";color:rgb(122,62= ,157)"><netvsc port></span><span style=3D"font-size:9pt;font-family:"Courier New&= quot;;color:rgb(119,119,119)">,</span><span style=3D"font-size:9pt;font-fam= ily:"Courier New";color:rgb(51,51,51)"> </span><span style=3D"font-size:9pt;font-family:"Courier New";col= or:rgb(122,62,157)">num_rxq</span><span style=3D"font-size:9pt;font-family:= "Courier New";color:rgb(119,119,119)">,</span><span style=3D"font= -size:9pt;font-family:"Courier New";color:rgb(122,62,157)">num_tx= q</span><span style=3D"font-size:9pt;font-family:"Courier New";co= lor:rgb(119,119,119)">,</span><span style=3D"font-size:9pt;font-family:&quo= t;Courier New";color:rgb(51,51,51)"> </span><span style=3D"font-size:9pt;font-family:"Courier New";col= or:rgb(119,119,119)">&</span><span style=3D"font-size:9pt;font-family:&= quot;Courier New";color:rgb(51,51,51)">conf</span><span style=3D"font-= size:9pt;font-family:"Courier New";color:rgb(119,119,119)">); </span><u></u><u></u></p> <div> <div> <p class=3D"MsoNormal" style=3D"line-height:13.5pt;background:whitesmoke"> <span style=3D"font-size:9pt;font-family:"Courier New";color:rgb(= 122,62,157)">uint8_t</span><span style=3D"font-size:9pt;font-family:"C= ourier New";color:rgb(51,51,51)"> </span><span style=3D"font-size:9pt;font-family:"Courier New";col= or:rgb(122,62,157)">rss_key_temp</span><span style=3D"font-size:9pt;font-fa= mily:"Courier New";color:rgb(119,119,119)">[</span><span style=3D= "font-size:9pt;font-family:"Courier New";color:rgb(156,93,39)">64= </span><span style=3D"font-size:9pt;font-family:"Courier New";col= or:rgb(119,119,119)">];</span><u></u><u></u></p> </div> <div> <p class=3D"MsoNormal" style=3D"line-height:13.5pt;background:whitesmoke"> <span style=3D"font-size:9pt;font-family:"Courier New";color:rgb(= 122,62,157)">struct</span><span style=3D"font-size:9pt;font-family:"Co= urier New";color:rgb(51,51,51)"> rte_eth_rss_conf rss_conf </span><span style=3D"font-size:9pt;font-family:"Courier New";col= or:rgb(119,119,119)">=3D</span><span style=3D"font-size:9pt;font-family:&qu= ot;Courier New";color:rgb(51,51,51)"> </span><span style=3D"font-size:9pt;font-family:"Courier New";col= or:rgb(119,119,119)">{</span><u></u><u></u></p> </div> <div> <p class=3D"MsoNormal" style=3D"line-height:13.5pt;background:whitesmoke"> <span style=3D"font-size:9pt;font-family:"Courier New";color:rgb(= 51,51,51)">.rss_key </span> <span style=3D"font-size:9pt;font-family:"Courier New";color:rgb(= 119,119,119)">=3D</span><span style=3D"font-size:9pt;font-family:"Cour= ier New";color:rgb(51,51,51)"> rss_key_temp</span><span style=3D"font-= size:9pt;font-family:"Courier New";color:rgb(119,119,119)">,</spa= n><u></u><u></u></p> </div> <div> <p class=3D"MsoNormal" style=3D"line-height:13.5pt;background:whitesmoke"> <span style=3D"font-size:9pt;font-family:"Courier New";color:rgb(= 51,51,51)">.rss_key_len </span><span style=3D"font-size:9pt;font-family:"Courier New";col= or:rgb(119,119,119)">=3D</span><span style=3D"font-size:9pt;font-family:&qu= ot;Courier New";color:rgb(51,51,51)"> </span><span style=3D"font-size:9pt;font-family:"Courier New";col= or:rgb(119,119,119)">sizeof(</span><span style=3D"font-size:9pt;font-family= :"Courier New";color:rgb(51,51,51)">rss_key_temp</span><span styl= e=3D"font-size:9pt;font-family:"Courier New";color:rgb(119,119,11= 9)">),</span><u></u><u></u></p> </div> <div> <p class=3D"MsoNormal" style=3D"line-height:13.5pt;background:whitesmoke"> <span style=3D"font-size:9pt;font-family:"Courier New";color:rgb(= 119,119,119)">};</span><u></u><u></u></p> </div> <div> <p class=3D"MsoNormal" style=3D"line-height:13.5pt;background:whitesmoke"> <span style=3D"font-size:9pt;font-family:"Courier New";color:rgb(= 51,51,51)">ret </span> <span style=3D"font-size:9pt;font-family:"Courier New";color:rgb(= 119,119,119)">=3D</span><span style=3D"font-size:9pt;font-family:"Cour= ier New";color:rgb(51,51,51)"> </span><b><span style=3D"font-size:9pt;font-family:"Courier New";= color:rgb(170,55,49)">rte_eth_dev_rss_hash_conf_get</span></b><span style= =3D"font-size:9pt;font-family:"Courier New";color:rgb(119,119,119= )">(</span><span style=3D"font-size:9pt;font-family:"Courier New"= ;color:rgb(122,62,157)"><VF port></span><span style=3D"font-size:9pt;font-family:"Courier New&= quot;;color:rgb(119,119,119)">,</span><span style=3D"font-size:9pt;font-fam= ily:"Courier New";color:rgb(51,51,51)"> </span><span style=3D"font-size:9pt;font-family:"Courier New";col= or:rgb(119,119,119)">&</span><span style=3D"font-size:9pt;font-family:&= quot;Courier New";color:rgb(51,51,51)">rss_conf</span><span style=3D"f= ont-size:9pt;font-family:"Courier New";color:rgb(119,119,119)">);= </span><u></u><u></u></p> </div> <div> <p class=3D"MsoNormal" style=3D"line-height:13.5pt;background:whitesmoke"> <span style=3D"font-size:9pt;font-family:"Courier New";color:rgb(= 51,51,51)">=C2=A0</span><u></u><u></u></p> </div> </div> </div> </div> </div> </div> <p class=3D"MsoNormal">=C2=A0<u></u><u></u></p> <div> <div> <p class=3D"MsoNormal">Now the VF port RSS offloads show only TCP flags set= and not UDP. I assumed that even the UDP flags might be set. Is this expec= ted ?<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">=C2=A0<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">Regards,<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">Nandini=C2=A0<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">=C2=A0<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">=C2=A0<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">On Thu, Sep 12, 2024 at 4:09<span style=3D"font-fami= ly:Arial,sans-serif">=E2=80=AF</span>PM Stephen Hemminger <<a href=3D"ma= ilto:stephen@networkplumber.org" target=3D"_blank">stephen@networkplumber.o= rg</a>> wrote:<u></u><u></u></p> </div> <blockquote style=3D"border-top:none;border-right:none;border-bottom:none;b= order-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin:5pt 0i= n 5pt 4.8pt"> <p class=3D"MsoNormal">On Thu, 12 Sep 2024 13:47:37 -0700<br> Nandini Rangaswamy <<a href=3D"mailto:nandini.rangaswamy@broadcom.com" t= arget=3D"_blank">nandini.rangaswamy@broadcom.com</a>> wrote:<br> <br> > Thanks for your response Long Li.<br> > I see with netvsc the maximum number of Tx descriptors is restricted t= o<br> > 4096 whereas the number of Rx descriptors is restricted to 8192.<br> > But, for failsafe PMD , we see that both the number of Txd and Rxd is<= br> > restricted to 8192.<br> > How is netvsc PMD giving the same performance as failsafe PMD ?<br> > <br> > Regards<br> <br> I think the limits there were somewhat arbitrary chose with netvsc.<br> Don't remember a hard reason that would block larger sizes.<br> <br> <br> Having really big rings won't help performance (i.e BufferBloat) and<br= > could a lot of memory consumption. When all heavy data traffic goes through= <br> the VF and that ring is different. Only DoS attacks should be impacted<br> by rx/tx descriptor limits in the netvsc device. The linux driver actually<= br> has much smaller buffer.<u></u><u></u></p> </blockquote> </div> </div> <p class=3D"MsoNormal"><br> <span style=3D"font-size:10pt;color:black;background:white">This electronic= communication and the information and any files transmitted with it, or at= tached to it, are confidential and are intended solely for the use of the i= ndividual or entity to whom it is addressed and may contain information that is confidential, legally privil= eged, protected by privacy laws, or otherwise restricted from disclosure to= anyone else. If you are not the intended recipient or the person responsib= le for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, dis= tributing, dissemination, forwarding, printing, or copying of this e-mail i= s strictly prohibited. If you received this e-mail in error, please return = the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.</span><u></u><u= ></u></p> </div> </div> </div> </div> </blockquote> </div> </blockquote> </div> <p class=3D"MsoNormal"><br> <span style=3D"font-size:10pt;color:black;background:white">This electronic= communication and the information and any files transmitted with it, or at= tached to it, are confidential and are intended solely for the use of the i= ndividual or entity to whom it is addressed and may contain information that is confidential, legally privil= eged, protected by privacy laws, or otherwise restricted from disclosure to= anyone else. If you are not the intended recipient or the person responsib= le for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, dis= tributing, dissemination, forwarding, printing, or copying of this e-mail i= s strictly prohibited. If you received this e-mail in error, please return = the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.</span><u></u><u= ></u></p> </div> </div> </div> </div></blockquote></div> <br> <span style=3D"background-color:rgb(255,255,255)"><font size=3D"2">This ele= ctronic communication and the information and any files transmitted with it= , or attached to it, are confidential and are intended solely for the use o= f the individual or entity to whom it is addressed and may contain informat= ion that is confidential, legally privileged, protected by privacy laws, or= otherwise restricted from disclosure to anyone else. If you are not the in= tended recipient or the person responsible for delivering the e-mail to the= intended recipient, you are hereby notified that any use, copying, distrib= uting, dissemination, forwarding, printing, or copying of this e-mail is st= rictly prohibited. If you received this e-mail in error, please return the = e-mail to the sender, delete it from your computer, and destroy any printed= copy of it.</font></span> --000000000000c8ae1b06227ba79a--