From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id D8048A0032 for ; Mon, 18 Jul 2022 12:25:07 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 5B4B84069F; Mon, 18 Jul 2022 12:25:07 +0200 (CEST) Received: from mail-ed1-f45.google.com (mail-ed1-f45.google.com [209.85.208.45]) by mails.dpdk.org (Postfix) with ESMTP id C778540041 for ; Mon, 18 Jul 2022 12:25:05 +0200 (CEST) Received: by mail-ed1-f45.google.com with SMTP id m16so14569657edb.11 for ; Mon, 18 Jul 2022 03:25:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=emumba-com.20210112.gappssmtp.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=uxmnhn/ev+gWQqc0qgBnpYctvuhbN94SbmklJV6/hKk=; b=G7gH8uiucCieVdfvfej52CmEAFL/d7SJ89xB079qBvk7rE0JJV049t96dv5e0XpXl8 W5COGIoy6rea+9GZvPGDd9uFzCOd0GwwlTSoQmqEq8bWExJBlSrDeQO3eVA41z9Fn2s0 U1rJTya73oxiFUF4uIeSYuub6F1mHtHx+yxrRDujQqssGaio1gJkhY5vS6hhlSXY36UH UCAJfoJc5RVZHrVlauTjkfm3IiRXMGAm9Ukjo8a774fZvmk9Z2Oj+G5AMQH1OE316TcN Wq6Vd+ovwv7qOZr5/Ms7tENLovVRo/lhkAQp5pszSYx66jdrwwDoJBcntPsY+hAE2fhf VhzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=uxmnhn/ev+gWQqc0qgBnpYctvuhbN94SbmklJV6/hKk=; b=cmH+LgJRlrgWLARDJvz9N224ESUfvHfrXFWJzueodF3lOVC2L29fOvTQZXDdj45cMr ucchiH+kecvbyILPNqOYg7fHKkE+UCiguRv2ku6WMof67r7iFdujXIQbx9B2A1+0lHZI MacJ0yzXX0RIi0pMpeh1nNC9obxugHsA4JSv/3DusoC3vNXFlubYzVp+AyEajH9xNVmS +r9RqVbqbC/XATYSy9rv7kVoWlPkcPm0mWMwfuucJsq7AwdPkJ9CGTJDaoRD9xoTOG0j gDR//KUvpHFUUg79edsTsHxLamUwv3zol5pbFCgdlbcddIo5hp3w9yWQx+7wJOXRRxvf Olgg== X-Gm-Message-State: AJIora99Ub2xoZGZWl8iCWb00JGuMPxotQsBQWQOR9Pxsgp1I+GvPu6w WyaaawyYSJN2gk9Uu5uqQ2RkP0T7J27ZCeOvohKx+slshLJBI+cu X-Google-Smtp-Source: AGRyM1uutGl03GJuflKmqlFRa8IVlEx3ZFAXEk53kPGNsU/YNkApfkP21i+1UeyElfjF2F0RNH2/FvACslkB6Qce3lk= X-Received: by 2002:a05:6402:2741:b0:434:fe8a:1f96 with SMTP id z1-20020a056402274100b00434fe8a1f96mr36555346edd.331.1658139905462; Mon, 18 Jul 2022 03:25:05 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Usman Tanveer Date: Mon, 18 Jul 2022 15:24:54 +0500 Message-ID: Subject: Re: Packet Accumulation in RX and TX in bnx2x (Usman Tanveer) To: "Varghese, Vipin" Cc: "users@dpdk.org" Content-Type: multipart/alternative; boundary="0000000000000da59b05e411c823" X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org --0000000000000da59b05e411c823 Content-Type: text/plain; charset="UTF-8" > > DPDK example L@FWD uses TX_BUFFER, which internally accumulates `n` > packets then send to device to amortize the cost. So my suggestion is not > to use DPDK L2fwd for latency test. Instead make use of DPDK example > SKELETON which directly uses `tx_burst`. I have tried SKELETON with BURST_SIZE = 1, it also shows the same behavior i.e. packets are being accumulated. On Sat, Jul 16, 2022 at 10:39 AM Varghese, Vipin wrote: > [AMD Official Use Only - General] > > DPDK example L@FWD uses TX_BUFFER, which internally accumulates `n` > packets then send to device to amortize the cost. So my suggestion is not > to use DPDK L2fwd for latency test. Instead make use of DPDK example > SKELETON which directly uses `tx_burst`. > > > -----Original Message----- > > From: users-request@dpdk.org > > Sent: Friday, July 15, 2022 3:30 PM > > To: users@dpdk.org > > Subject: users Digest, Vol 347, Issue 13 > > > > [CAUTION: External Email] > > > > Send users mailing list submissions to > > users@dpdk.org > > > > To subscribe or unsubscribe via the World Wide Web, visit > > > > > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmails.dp > > dk.org%2Flistinfo%2Fusers&data=05%7C01%7Cvipin.varghese%40amd.co > > m%7Cbe1da1e5c9884d626f9f08da6648ca0f%7C3dd8961fe4884e608e11a82d99 > > 4e183d%7C0%7C0%7C637934760074138757%7CUnknown%7CTWFpbGZsb3d8e > > yJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D% > > 7C3000%7C%7C%7C&sdata=mF6VDr1GkGr3L13V7aplaxY58dHXpvdxcmqjj > > a%2Ba3kQ%3D&reserved=0 > > or, via email, send a message with subject or body 'help' to > > users-request@dpdk.org > > > > You can reach the person managing the list at > > users-owner@dpdk.org > > > > When replying, please edit your Subject line so it is more specific than > "Re: > > Contents of users digest..." > > > > > > Today's Topics: > > > > 1. Packet Accumulation in RX and TX in bnx2x (Usman Tanveer) > > 2. mlx5: Keeping packets with invalid CRC/FCS (Pfau, Johannes (ITIV)) > > > > > > ---------------------------------------------------------------------- > > > > Message: 1 > > Date: Thu, 14 Jul 2022 16:39:54 +0500 > > From: Usman Tanveer > > To: users@dpdk.org > > Cc: shshaikh@marvell.com, rmody@marvell.com > > Subject: Packet Accumulation in RX and TX in bnx2x > > Message-ID: > > > 77rDV+n6crq14T=p1h31Q@mail.gmail.com> > > Content-Type: text/plain; charset="utf-8" > > > > Hi, > > > > I have a *BroadCom NetXtreme II BCM57810 10 Gigabit *with a *bnx2x* > > driver. > > I was trying to measure the performance of the network card. I'm running > > DPDK-L2FWD application to receive and send packets with MAX_PKT_BURST = > > 1. > > but I noticed that packets are being accumulated somewhere in bnx2x_rxtx. > > There is some sort of batching/buffering happening in rxtx. So, the > latency of > > the packets is increasing as they have to wait. > > The first packet of the batch is received with the minimum latency and > > incoming packets are received with the accumulated latency of the > previous > > packets. After a specific number of packets (6-7 packets), the same > pattern > > repeats i.e. a packet arrives with the minimum latency and incoming > packets > > with accumulated latency until the specific number of packets arrives. > I've > > copied the latency measured for some contiguous packets. > > > > I tried to explore bnx2x_recv_pkts() and bnx2x_xmit_pkts() in > bnx2x_rxtx.c, but > > didn't get any clue why the packets were being accumulated. > > > > Sequence Latency (microseconds) > > 4825105 9.37207 > > 4825106 10.72168 > > 4825107 15.06543 > > 4825108 19.394043 > > 4825109 22.665039 > > 4825110 26.979004 > > 4825111 8.74707 > > 4825112 11.145996 > > 4825113 14.439941 > > 4825114 18.701172 > > 4825115 23.082031 > > 4825116 27.402832 > > 4825117 9.164062 > > 4825118 11.623047 > > 4825119 15.944824 > > 4825120 19.066406 > > 4825121 23.589355 > > 4825122 27.909668 > > 4825123 9.701172 > > 4825124 11.13916 > > 4825125 15.489746 > > 4825126 19.780762 > > 4825127 23.111816 > > 4825128 27.403809 > > -------------- next part -------------- > > An HTML attachment was scrubbed... > > URL: > > < > https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmails.dp > > dk.org%2Farchives%2Fusers%2Fattachments%2F20220714%2F6887b7b4%2Fatt > > achment- > > 0001.htm&data=05%7C01%7Cvipin.varghese%40amd.com%7Cbe1da1e5c > > 9884d626f9f08da6648ca0f%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C > > 0%7C637934760074138757%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjA > > wMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C > > %7C&sdata=ICLtgULJ6%2BTmatMiSECKtf0HTESOyd25TOzkygF%2FUsA%3D > > &reserved=0> > > > > ------------------------------ > > > > Message: 2 > > Date: Thu, 14 Jul 2022 13:52:14 +0000 > > From: "Pfau, Johannes (ITIV)" > > To: "users@dpdk.org" > > Subject: mlx5: Keeping packets with invalid CRC/FCS > > Message-ID: > > Content-Type: text/plain; charset="iso-8859-1" > > > > Hi all, > > > > We're currently developing a low-cost DAQ system to record high-bandwidth > > data from FPGA systems. We're using DPDK with Mellanox Connect-X5 cards, > > the drivers and DPDK installed from the standard RHEL 8 repositories. We > > capture raw ethernet (no L3) on 1:1 links to the FPGA devices to avoid > all > > possible overhead. So far this setup works great, we can handle 100 > Gbit/s of > > traffic even on a single core, but we can also distribute packets to > multiple > > cores depending on the ether type if required. > > > > To save a few more bits in the transmission, we'd like to avoid encoding > packet > > counters into the data stream. In that case we have to make sure we never > > miss any packets in recording though, even if the FCS is invalid. > > There are two aspects involved, leading to two questions: > > > > First, we need to store the CRC as well so that we can detect the > invalid packets > > later on in offline processing. Using RTE_ETH_RX_OFFLOAD_KEEP_CRC is > > working fine, but it first confused me a lot: Both rte_pktmbuf_pkt_len > and > > rte_pktmbuf_data_len still report the length without CRC. If I just read > 4 bytes > > more than announced in these functions, I can read the correct CRC. Is it > > intentional that the 4 CRC bytes are not included in these counts? > > > > Second, and this is a larger issue: We also need to receive packets with > invalid > > FCS. We don't really have a an idea how to actually inject packets with > invalid > > FCS for testing, but from the documentation I assume the mlx5 driver in > default > > setup would drop invalid packets? There was an mailing list discussing > this for > > intel NICS > > ( > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmails.dp > > dk.org%2Farchives%2Fusers%2F2021- > > June%2F005651.html&data=05%7C01%7Cvipin.varghese%40amd.com%7C > > be1da1e5c9884d626f9f08da6648ca0f%7C3dd8961fe4884e608e11a82d994e183 > > d%7C0%7C0%7C637934760074138757%7CUnknown%7CTWFpbGZsb3d8eyJWIjo > > iMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000 > > %7C%7C%7C&sdata=cEDhuhV4g0v%2FueSMO7IL9BJBJwz8SFu0KWp5%2BT > > Iycbw%3D&reserved=0), but I couldn't find anything for mlx5. Does > > anybody know if the mlx5 driver also offers an option to keep invalid > packets? > > > > Best regards, > > Johannes > > > > -- > > Karlsruhe Institute of Technology (KIT) > > Institut f?r Technik der Informationsverarbeitung (ITIV) > > > > M.Sc. Johannes Pfau > > Research Associate > > > > Engesserstr. 5 > > Building 30.10, Room 218.1 > > 76131 Karlsruhe, Germany > > > > Phone: +49 721 608-41939 > > E-mail: johannes.pfau@kit.edu > > > > > > Registered office: > > Kaiserstra?e 12, 76131 Karlsruhe, Germany > > > > > > KIT ? The Research University in the Helmholtz Association > > > > -------------- next part -------------- > > A non-text attachment was scrubbed... > > Name: smime.p7s > > Type: application/pkcs7-signature > > Size: 6301 bytes > > Desc: not available > > URL: > > < > https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmails.dp > > dk.org%2Farchives%2Fusers%2Fattachments%2F20220714%2Fd44db3e4%2Fatt > > achment- > > 0001.bin&data=05%7C01%7Cvipin.varghese%40amd.com%7Cbe1da1e5c9 > > 884d626f9f08da6648ca0f%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0 > > %7C637934760074138757%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAw > > MDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7 > > C&sdata=ijZq%2FPvpK3lxu98pYyBLMUKberQqOz%2FPtBCZ7NJg9m8%3D& > > amp;reserved=0> > > > > End of users Digest, Vol 347, Issue 13 > > ************************************** > --0000000000000da59b05e411c823 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
DPDK exa= mple L@FWD uses TX_BUFFER, which internally accumulates `n` packets then se= nd to device to amortize the cost. So my suggestion is not to use DPDK L2fw= d for latency test. Instead make use of DPDK example SKELETON which directl= y uses `tx_burst`.

I have tried SKELETON wi= th BURST_SIZE =3D 1, it also shows the same behavior i.e. packets are being= accumulated.


On Sat, Jul 16, 2022 at 10:39 AM Varghese, Vipin <= ;Vipin.Varghese@amd.com> w= rote:
[AMD Offic= ial Use Only - General]

DPDK example L@FWD uses TX_BUFFER, which internally accumulates `n` packets= then send to device to amortize the cost. So my suggestion is not to use D= PDK L2fwd for latency test. Instead make use of DPDK example SKELETON which= directly uses `tx_burst`.

> -----Original Message-----
> From: user= s-request@dpdk.org <users-request@dpdk.org>
> Sent: Friday, July 15, 2022 3:30 PM
> To: users@dpdk.org=
> Subject: users Digest, Vol 347, Issue 13
>
> [CAUTION: External Email]
>
> Send users mailing list submissions to
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0users@dpdk.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>
> https://nam11.safel= inks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fmails.dp
> dk.org<= /a>%2Flistinfo%2Fusers&amp;data=3D05%7C01%7Cvipin.varghese%40amd.co
> m%7Cbe1da1e5c9884d626f9f08da6648ca0f%7C3dd8961fe4884e608e11a82d99
> 4e183d%7C0%7C0%7C637934760074138757%7CUnknown%7CTWFpbGZsb3d8e
> yJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%
> 7C3000%7C%7C%7C&amp;sdata=3DmF6VDr1GkGr3L13V7aplaxY58dHXpvdxcmqjj<= br> > a%2Ba3kQ%3D&amp;reserved=3D0
> or, via email, send a message with subject or body 'help' to >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0users-request@dpdk.org
>
> You can reach the person managing the list at
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0users-owner@dpdk.org
>
> When replying, please edit your Subject line so it is more specific th= an "Re:
> Contents of users digest..."
>
>
> Today's Topics:
>
>=C2=A0 =C2=A0 1. Packet Accumulation in RX and TX in bnx2x (Usman Tanve= er)
>=C2=A0 =C2=A0 2. mlx5: Keeping packets with invalid CRC/FCS (Pfau, Joha= nnes (ITIV))
>
>
> ----------------------------------------------------------------------=
>
> Message: 1
> Date: Thu, 14 Jul 2022 16:39:54 +0500
> From: Usman Tanveer <usman.tanveer@emumba.com>
> To: users@dpdk.org=
> Cc: shshaikh= @marvell.com, rm= ody@marvell.com
> Subject: Packet Accumulation in RX and TX in bnx2x
> Message-ID:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<CAH_O0bwKXL9pSq6k_dFS=3D-JsF1u_-<= br> > 77rDV+n6crq14T=3Dp1h31Q@mail.gmail.com>
> Content-Type: text/plain; charset=3D"utf-8"
>
> Hi,
>
> I have a *BroadCom NetXtreme II BCM57810 10 Gigabit *with a *bnx2x* > driver.
> I was trying to measure the performance of the network card. I'm r= unning
> DPDK-L2FWD application to receive and send packets with MAX_PKT_BURST = =3D
> 1.
> but I noticed that packets are being accumulated somewhere in bnx2x_rx= tx.
> There is some sort of batching/buffering happening in rxtx. So, the la= tency of
> the packets is increasing as they have to wait.
> The first packet of the batch is received with the minimum latency and=
> incoming packets are received with the accumulated latency of the prev= ious
> packets. After a specific number of packets (6-7 packets), the same pa= ttern
> repeats i.e. a packet arrives with the minimum latency and incoming pa= ckets
> with accumulated latency until the specific number of packets arrives.= I've
> copied the latency measured for some contiguous packets.
>
> I tried to explore bnx2x_recv_pkts() and bnx2x_xmit_pkts() in bnx2x_rx= tx.c, but
> didn't get any clue why the packets were being accumulated.
>
> Sequence Latency (microseconds)
> 4825105 9.37207
> 4825106 10.72168
> 4825107 15.06543
> 4825108 19.394043
> 4825109 22.665039
> 4825110 26.979004
> 4825111 8.74707
> 4825112 11.145996
> 4825113 14.439941
> 4825114 18.701172
> 4825115 23.082031
> 4825116 27.402832
> 4825117 9.164062
> 4825118 11.623047
> 4825119 15.944824
> 4825120 19.066406
> 4825121 23.589355
> 4825122 27.909668
> 4825123 9.701172
> 4825124 11.13916
> 4825125 15.489746
> 4825126 19.780762
> 4825127 23.111816
> 4825128 27.403809
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> <https://nam11.sa= felinks.protection.outlook.com/?url=3Dhttp%3A%2F%2Fmails.dp
> dk.org<= /a>%2Farchives%2Fusers%2Fattachments%2F20220714%2F6887b7b4%2Fatt
> achment-
> 0001.htm&amp;data=3D05%7C01%7Cvipin.varghese%40amd.com%7Cbe1da1e5c
> 9884d626f9f08da6648ca0f%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C
> 0%7C637934760074138757%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjA
> wMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C
> %7C&amp;sdata=3DICLtgULJ6%2BTmatMiSECKtf0HTESOyd25TOzkygF%2FUsA%3D=
> &amp;reserved=3D0>
>
> ------------------------------
>
> Message: 2
> Date: Thu, 14 Jul 2022 13:52:14 +0000
> From: "Pfau, Johannes (ITIV)" <johannes.pfau@kit.edu>
> To: "users@dp= dk.org" <us= ers@dpdk.org>
> Subject: mlx5: Keeping packets with invalid CRC/FCS
> Message-ID: <e0aacbdee7fa427ea2bf22dcb4f24c09@kit.edu> > Content-Type: text/plain; charset=3D"iso-8859-1"
>
> Hi all,
>
> We're currently developing a low-cost DAQ system to record high-ba= ndwidth
> data from FPGA systems. We're using DPDK with Mellanox Connect-X5 = cards,
> the drivers and DPDK installed from the standard RHEL 8 repositories. = We
> capture raw ethernet (no L3) on 1:1 links to the FPGA devices to avoid= all
> possible overhead. So far this setup works great, we can handle 100 Gb= it/s of
> traffic even on a single core, but we can also distribute packets to m= ultiple
> cores depending on the ether type if required.
>
> To save a few more bits in the transmission, we'd like to avoid en= coding packet
> counters into the data stream. In that case we have to make sure we ne= ver
> miss any packets in recording though, even if the FCS is invalid.
> There are two aspects involved, leading to two questions:
>
> First, we need to store the CRC as well so that we can detect the inva= lid packets
> later on in offline processing. Using RTE_ETH_RX_OFFLOAD_KEEP_CRC is > working fine, but it first confused me a lot: Both rte_pktmbuf_pkt_len= and
> rte_pktmbuf_data_len still report the length without CRC. If I just re= ad 4 bytes
> more than announced in these functions, I can read the correct CRC. Is= it
> intentional that the 4 CRC bytes are not included in these counts?
>
> Second, and this is a larger issue: We also need to receive packets wi= th invalid
> FCS. We don't really have a an idea how to actually inject packets= with invalid
> FCS for testing, but from the documentation I assume the mlx5 driver i= n default
> setup would drop invalid packets? There was an mailing list discussing= this for
> intel NICS
> (https://nam11.safe= links.protection.outlook.com/?url=3Dhttps%3A%2F%2Fmails.dp
> dk.org<= /a>%2Farchives%2Fusers%2F2021-
> June%2F005651.html&amp;data=3D05%7C01%7Cvipin.varghese%40amd.com%7C
> be1da1e5c9884d626f9f08da6648ca0f%7C3dd8961fe4884e608e11a82d994e183
> d%7C0%7C0%7C637934760074138757%7CUnknown%7CTWFpbGZsb3d8eyJWIjo
> iMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000
> %7C%7C%7C&amp;sdata=3DcEDhuhV4g0v%2FueSMO7IL9BJBJwz8SFu0KWp5%2BT > Iycbw%3D&amp;reserved=3D0), but I couldn't find anything for m= lx5. Does
> anybody know if the mlx5 driver also offers an option to keep invalid = packets?
>
> Best regards,
> Johannes
>
> --
> Karlsruhe Institute of Technology (KIT)
> Institut f?r Technik der Informationsverarbeitung (ITIV)
>
> M.Sc. Johannes Pfau
> Research Associate
>
> Engesserstr. 5
> Building 30.10, Room 218.1
> 76131 Karlsruhe, Germany
>
> Phone: +49 721 608-41939
> E-mail: joh= annes.pfau@kit.edu
>
>
> Registered office:
> Kaiserstra?e 12, 76131 Karlsruhe, Germany
>
>
> KIT ? The Research University in the Helmholtz Association
>
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: smime.p7s
> Type: application/pkcs7-signature
> Size: 6301 bytes
> Desc: not available
> URL:
> <https://nam11.sa= felinks.protection.outlook.com/?url=3Dhttp%3A%2F%2Fmails.dp
> dk.org<= /a>%2Farchives%2Fusers%2Fattachments%2F20220714%2Fd44db3e4%2Fatt
> achment-
> 0001.bin&amp;data=3D05%7C01%7Cvipin.varghese%40amd.com%7Cbe1da1e5c9
> 884d626f9f08da6648ca0f%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0
> %7C637934760074138757%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAw
> MDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7
> C&amp;sdata=3DijZq%2FPvpK3lxu98pYyBLMUKberQqOz%2FPtBCZ7NJg9m8%3D&a= mp;
> amp;reserved=3D0>
>
> End of users Digest, Vol 347, Issue 13
> **************************************
--0000000000000da59b05e411c823--