From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 493F6A0548 for ; Fri, 23 Apr 2021 10:32:55 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 8E5AB41DD0; Fri, 23 Apr 2021 10:32:46 +0200 (CEST) Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12olkn2081.outbound.protection.outlook.com [40.92.21.81]) by mails.dpdk.org (Postfix) with ESMTP id 7EF991623BF for ; Thu, 15 Apr 2021 18:03:50 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=NKOdoAtTe/ny7d/qLsevs51FlLjCN32TRrCxSZ230BCm+zQUcJjOl/HDWaSYKWQkIsu7nLD3Oa82W4/yE5jEwcvCCN1E2TaXORp7fyovmXBoe3pUds/k5lLHGfWhdRKv1DOsepE/PbM7rkJrs9nUQMsQI/cviUJg3lJjxeDHT/BxIpXeK4ALSJUXlsOzQjICQoN65AodU+BfzebYR3OBMEP4RMGBROX/rhndb9xFOWulwTyaxBo9tZmqXf5AsEY2le9XxNQHPpIcyNyyPgosevgWC+FJ1FJ9a/zOJLUT5XvqzqpbGJvx4F2cClAlPORZvAW57UftxXdBnKxRUsGjfQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=MGXb/Dijnoj4X1y6Zo9DiAFDa4NGFBOGi59z8PTcCZM=; b=G3H2ZwWotIODz0hgMSZ8QDeYTeI8qCdchZOaYqs2Rlo8lP50c4IaVGrOQeMifJABwzYcSPVbH7BXTqf1L0tcT+1mBUeAQXLG/NlY96FrsC5AdxVaF02jVl0R2IFz+YPBkhgzNvdV2p0PyIhg7NiijnHcvNQKJsK2PkU8tKzeq5bCTwtZbC4tTGRU1phm2TKazk53wTXyeDGb+bvd5fhjCjNFGBG1QYU2YbKFmlbeyrv9v7it5QOKRffPGUwfLI9eFNSsmAuy1XPQc7CRCXY177z54GHzyoF9efiinOHvQv54hm/ruqGt/6Uv/wElAYp/5RHh8YDCzZj1kdf2Ja8oZQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=outlook.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=MGXb/Dijnoj4X1y6Zo9DiAFDa4NGFBOGi59z8PTcCZM=; b=EdHnfaI5npFjPDQvldPTh7DxUPqrulePn74z/iQRTCudc3VncW0AQo8Zu3qNIs6EJIcbqSRsgHxB5wsFPVmyLXdkFrjXIjuNAyzHGfJsKqdjl5y2WJTbgQNAHXhws+hCrsRcm4Q0QXa3VV0CpDvhImh+VaydE44QG4moSOWU9U/FzZOuPdBD78KbcV9VcLYwqBi+1LY4XR9Ivp7p4bKwvxiEi8tXwHi2k56drhFml4AZEDJXwkLSRxRq20nZ/ZI0aM45/d2ebNgaszSHYs+BrRtxxCxnaCOF/ipb3Q5vNa14oYX42ulGnRvJh3zaB4AaTTtvcAmUu+PAa5/TUcErfg== Received: from MW2NAM12FT038.eop-nam12.prod.protection.outlook.com (2a01:111:e400:fc65::41) by MW2NAM12HT062.eop-nam12.prod.protection.outlook.com (2a01:111:e400:fc65::130) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4042.6; Thu, 15 Apr 2021 16:03:49 +0000 Received: from BYAPR04MB4167.namprd04.prod.outlook.com (2a01:111:e400:fc65::52) by MW2NAM12FT038.mail.protection.outlook.com (2a01:111:e400:fc65::168) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4065.7 via Frontend Transport; Thu, 15 Apr 2021 16:03:49 +0000 Received: from BYAPR04MB4167.namprd04.prod.outlook.com ([fe80::81de:50a5:6526:950b]) by BYAPR04MB4167.namprd04.prod.outlook.com ([fe80::81de:50a5:6526:950b%5]) with mapi id 15.20.4020.023; Thu, 15 Apr 2021 16:03:49 +0000 From: Hao Chen To: Pavel Vazharov CC: "users@dpdk.org" Thread-Topic: [dpdk-users] What is TCP read performance by using DPDK? Thread-Index: AQHXIDh4H8QknstLlk6mzOwh2+wPdKqarTUAgBqIIwiAABN/gIAAlO+X Date: Thu, 15 Apr 2021 16:03:49 +0000 Message-ID: References: , In-Reply-To: Accept-Language: en-US, zh-CN Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-incomingtopheadermarker: OriginalChecksum:C44DF948CB5DFD03AA4187FA7E30600DFA65A8909A2C7D62CB8008B24A8128DF; UpperCasedChecksum:FC922F94C1FC963ACE4CE72705D173B7A8676FA67EF19ED05BC3C97DA20F4B30; SizeAsReceived:7185; Count:45 x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [FNXAf3IqGGts2YE3NXgYqUFlzqcyTAIe] x-ms-publictraffictype: Email x-incomingheadercount: 45 x-eopattributedmessage: 0 x-ms-office365-filtering-correlation-id: 109ccf0e-d5dc-4f97-8f1d-08d900280ec9 x-ms-traffictypediagnostic: MW2NAM12HT062: x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: JybkEMfhl4frkdy+tBlCSiBIRj96wIte9apPkUf1lX4Jo6Q9CPprVbze6/dRGkpv0sL+2r9RrwC5Yv7wDIdOQjxgR+Lrucldk3EKl4ln0xEaxsDFoGvNSKpllnKNFeFKbCqPiszEvWNGWuuFs81R2on6YUUKkfb4qXpZp4eKMsWi9J4jQNN1M3qjthpiVQTKKDuEyUNi/mwJCJAP2QsRdcJyiEXjpsvcG4+JYd9mUsb1Y2zlHH/eLr+NSx+QCfEf8n6h9Ps14ptMQPjR2eE2PjSGDYGvelE9sLeP1JYWiJ9os16I3Fefp92I0moqV60P+jD9E+iJ0r3qaGw2wCgb1pZPlA41kb04TV6hRwTlxNCcd1rXJAzEdurox//fdfIWDGyDndMEYb90MBl1J4XvDUXZWJaEi33xELq7oLZOfVTWNNDu51zJUyb92mwdJI8r x-ms-exchange-antispam-messagedata: 7Wk8FHxSaOhTr8BH7zsT08oGQSOuo5t41+mEGoEID5Db3PfF3pWBIJS3qD5AceDK6b20jFGtJF2ABHJwiVngn+uBfvNqI3Hz6egqA2yDRlh3v5lg+JVeWvBd0WCtBKJu7TRDXsbYyKyHhKJi7Ftx5w== x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-AuthSource: MW2NAM12FT038.eop-nam12.prod.protection.outlook.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: 109ccf0e-d5dc-4f97-8f1d-08d900280ec9 X-MS-Exchange-CrossTenant-originalarrivaltime: 15 Apr 2021 16:03:49.5118 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Internet X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW2NAM12HT062 X-Mailman-Approved-At: Fri, 23 Apr 2021 10:32:43 +0200 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 Subject: Re: [dpdk-users] What is TCP read performance by using DPDK? X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Sender: "users" Hello Pavel, Appreciate for your detailed explanation. Please bear with my verbose/ques= tions. Based on your explanation, looks like your application (running on layer 7)= uses at least 3 threads. (1). first thread is for DPDK burst read(rte_eth_= rx_burst()). After read data from layer 2, put them into Q (2). second thr= ead (layer 2) read data from Q, and then use F-Stack to handle TCP data. Th= en put these data (layer 4) to TCP socket buffer. (3). third thread use epo= ll_wait() to read (layer 7) data from TCP socket buffer. And "forward" them= to outgoing TCP socket for rte_eth_tx_burst() Is my understanding right? Thanks ________________________________ From: Pavel Vazharov Sent: Wednesday, April 14, 2021 23:57 To: Hao Chen Cc: users@dpdk.org Subject: Re: [dpdk-users] What is TCP read performance by using DPDK? Hi, "Does it mean your code just look at IPHeader and TCPheader without handlin= g TCP payload?" The proxy works in the application layer. I mean, it works with regular BSD= sockets. As I said we use modified version of F-stack (https://github.com/= F-Stack/f-stack) for this. Basically our version is very close to the origi= nal libuinet (https://github.com/pkelsey/libuinet) but based on a newer ver= sion of the FreeBSD networking stack (FreeBSD 11). Here is a rough descript= ion how it works: 1. Every thread of our application reads packets in bursts from the single = RX queue using the DPDK API. 2. These packets are then passed/injected into the FreeBSD/F-stack networki= ng stack. We use separate networking stack per thread. 3. The networking stack processes the packets queueing them in the receive = buffers of the TCP sockets. These are regular sockets. 4. Every application thread also calls regularly an epoll_wait API provided= by the F-stack library. It's just a wrapper over the kevent API provided b= y the FreeBSD. 5. The application gets the read/write events from the epoll_wait and reads= /writes to the corresponding sockets. Again this is done exactly like in a = regular Linux application where you read/write data from/to the sockets. 6. Our test proxy application used sockets in pairs and all data read from = a given TCP socket were written to the corresponding TCP socket in the othe= r direction. 7. The written data to the given socket is put in the send buffers of this = socket and eventually sent out via the given TX queue using the DPDK API. T= his happens via callback that's provided to the F-stack. The callback is ca= lled for every single packet that needs to be send out by the F-stack and o= ur application implements this callback using the DPDK functionality. In ou= r design the F-stack/FreeBSD stack doesn't know about the DPDK it can work = with different packet processing framework. "Does it mean UDP-payload-size is NOT 1400 bytes (MTU size)? And it is as s= maller as 64 bytes for example?" My personal observation is that for the same amount of traffic the UTP traf= fic generates much more packets per second than the corresponding HTTP traf= fic running over TCP. These are the two tests that we did. However, I can't= provide you numbers about this at the moment but there are lots of packets= smaller than the MTU size usually. I think they come from things like the = internal ACK packets which seem to be send more frequently than TCP. Also t= he request, cancel, have, etc messages, from the BitTorrent protocol, are m= ost of the times sent in smaller packets. "Do you handle UTP payload, or just "relay" it like proxy?" Our proxies always work with sockets. We have application business logic bu= ilt over the socket layer. For the test case we just proxied the data betwe= en pairs of UTP sockets in the same way we did it for the TCP proxy above. We have implementation of the UTP protocol which provides a socket API simi= lar to the BSD socket API with read/write/shutdown/close/etc functions. As = you probably may have read, the UTP protocol is, kind of, a simplified vers= ion of the TCP protocol but also more suitable for the needs of the BitTorr= ent traffic. So this is a reliable protocol and this means that there is a = need for socket buffers. Our implementation is built over the UDP sockets p= rovided by the F-stack. The data are read from the UDP sockets and put into= the buffers of the corresponding UTP socket. If contiguous data are collec= ted into the buffers, the implementation fires notification to the applicat= ion layer. The write direction works in the opposite way. The data from the= application are first written to the buffers of the UTP socket and then la= ter send via the internal UDP socket from the F-stack. So to summarize the above. We handle the TCP/UDP payload using the regular = BSD socket API provided by the F-stack library and our UTP stack library. F= or the test we just relayed the data between a few thousands pairs of socke= ts. Currently we do much more complex manipulation of this data but this is= still work in progress and the final performance is still not tested. Hope the above explanations help. Pavel.