From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id AE7D1A0544; Mon, 10 Oct 2022 07:17:40 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 89A1C400D5; Mon, 10 Oct 2022 07:17:40 +0200 (CEST) Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by mails.dpdk.org (Postfix) with ESMTP id 8262540042 for ; Mon, 10 Oct 2022 07:17:38 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1665379058; x=1696915058; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-transfer-encoding:mime-version; bh=+h5XzTlKBeiML5TuVskqJXtqpv2AOAJY0SYmPFG1HNE=; b=QBjCw292Aoh+waG9TMduI3oqTrRwzNnsOf5lM4/jPPYG3bCjZvXJiGEJ h9ePDz4GFEew/pLy9rq7Y3K/ZCKpQoojn+Esg0pCpXQzbHTqzpb2nYWti z7NZOP4eIuVEtVdhFxinDuxtpZUhe7mSAPZNk224nq1ISAM8wX3iy4T+s 2xMNaeGGh1BzwHdjYC1eUkgr5KsAKojlvchIDvWAf06SQ5JeBCJ5Ref4X iJHv3E1oNXPUyOAkaE1gk8aB3IoV0VrU8UJxa68oEHQBQ5k6uQd1EOCYx QJvT2slV7Di2uBy/7YyZdqclhEd+FYvZvRECGyreyYlWPg7lntezq5y94 A==; X-IronPort-AV: E=McAfee;i="6500,9779,10495"; a="305739697" X-IronPort-AV: E=Sophos;i="5.95,172,1661842800"; d="scan'208";a="305739697" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Oct 2022 22:17:16 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10495"; a="694502540" X-IronPort-AV: E=Sophos;i="5.95,172,1661842800"; d="scan'208";a="694502540" Received: from fmsmsx601.amr.corp.intel.com ([10.18.126.81]) by fmsmga004.fm.intel.com with ESMTP; 09 Oct 2022 22:17:15 -0700 Received: from fmsmsx612.amr.corp.intel.com (10.18.126.92) by fmsmsx601.amr.corp.intel.com (10.18.126.81) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Sun, 9 Oct 2022 22:17:15 -0700 Received: from fmsmsx608.amr.corp.intel.com (10.18.126.88) by fmsmsx612.amr.corp.intel.com (10.18.126.92) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Sun, 9 Oct 2022 22:17:15 -0700 Received: from fmsedg601.ED.cps.intel.com (10.1.192.135) by fmsmsx608.amr.corp.intel.com (10.18.126.88) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31 via Frontend Transport; Sun, 9 Oct 2022 22:17:15 -0700 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (104.47.57.172) by edgegateway.intel.com (192.55.55.70) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2375.31; Sun, 9 Oct 2022 22:17:14 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=lYT7LzD5XBwG7vBbhV4pQTemSPeQ124aEkJHgL7WmKkMYYd7fnWApCDUQsi+jk0rNQyHrcDOXNO18q5Yw/HF6etBWURE8W6a3NsjfQhN0JY57J1qi1LZHMyOPbOC7/hkXLyWpJ0DLe8hkZforiPWckzP23gDwYhVuXswu2BIVIxpl7ie2daOmqgMbB21GHefYBQSBxkrYCiCZ7xXd5gNUJ/Wfu16Toc8BHdlGhWgx0ztDWcC8zOKfjj/xAs5DKTeCbBpP5wyuLuwq1H4Rw2qWhY8P/ggpJjew5LM6zMKVdSxX+DjuZf1eLxTuHmhOCtiNn9bJbogZvVRZo4zOuTbaA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Hc1377yMRZlhVoAa+XuuGE6tFCD/otDuWKYczRMiM38=; b=LizVDNM6t6w299rQvcTi/k8Y5bMFzpv0JMqQJLDex1OclA7ZHHMCZsd/fkht7w0fMO0vPNKFl7DPtbGACQS1VmQifNleGK9RL9IZjEmclD2kl6z6twptYiQ1LZw14gGos+arlQe5DB8ySeWJ/iz4O/Hmln2BNOrsc7ccxavPHO1aoi7TAvkt0+gc75i6lLJqsq1bgoIOU8M5rdW0KY3ornN9Q/7SgaJIxya6HFFFYi5VglpnAOO5oRIznxWc1/YPhVJwIapdrNoKDaEhZH1c3w0VAqpwJBg0rlEK6aRj413l6x5glGF5xTWB7tDqJaeW+wdj5XpioDudeFDjl6ufBg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Received: from CY5PR11MB6487.namprd11.prod.outlook.com (2603:10b6:930:31::17) by BN9PR11MB5532.namprd11.prod.outlook.com (2603:10b6:408:105::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5709.15; Mon, 10 Oct 2022 05:17:13 +0000 Received: from CY5PR11MB6487.namprd11.prod.outlook.com ([fe80::d550:5776:e363:748d]) by CY5PR11MB6487.namprd11.prod.outlook.com ([fe80::d550:5776:e363:748d%5]) with mapi id 15.20.5676.039; Mon, 10 Oct 2022 05:17:13 +0000 From: "Hu, Jiayu" To: "maxime.coquelin@redhat.com" , "Xia, Chenbo" CC: "He, Xingguang" , "Jiang, Cheng1" , "Ma, WenwuX" , "Wang, YuanX" , "dev@dpdk.org" Subject: RE: [PATCH v3] net/vhost: support asynchronous data path Thread-Topic: [PATCH v3] net/vhost: support asynchronous data path Thread-Index: AQHYts0MeNI+McmhZU2b6KvCaBF7NK4HOxRw Date: Mon, 10 Oct 2022 05:17:13 +0000 Message-ID: References: <20220814150636.2260317-1-jiayu.hu@intel.com> <20220823163534.947869-1-yuanx.wang@intel.com> In-Reply-To: <20220823163534.947869-1-yuanx.wang@intel.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-reaction: no-action dlp-version: 11.6.500.17 dlp-product: dlpe-windows authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: CY5PR11MB6487:EE_|BN9PR11MB5532:EE_ x-ms-office365-filtering-correlation-id: 5a2ed5b4-3f31-43fc-6e0f-08daaa7eb095 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 3CcwWt0Sg5zPcOWa2+agUt+CHWMpn4l3EUYcOZ/kluwdgRpbRNDdxWgd6Idry3lwvEGqQUOszGJ6hHs9G+JUVvtIUOSeHRFCVrjDXMMoEOgGixc2anKnaMI+2w9M13jaQV5aYKGojUxUJGSnVD4AurtZttasOvb4lSdEqKCK+YYj7XEW+pd/7bZ5CRk4UJ9gr6PfzMGfHTUsT8sf0ZGAHeF5yNOQkxpXaYMP0sy9IwtOSzi9FW1bOYozOo9hsrWxnemo+kdWjrt15y8Rpk1+OqFcvvyQKb8f0SFQd6UcJsHzO6HnZZddjuxhJ6Ua9jGLid6gkxjKdsz9dYm8KoLOPYpfwDkF4mX0j091aqx1gAtBCmGpGJYfy9XfUQN9ARdns7fbJ+U32YFSfkvvgmHyzoHMM8vkLZFb2fpv8vtlbp/OudYBZgxBz2UXoMOPEg1Wxb5pzH6ECAB0CiG2gWQD+UL3cVITGkFgez2RfIMTkWzvZY5F/5RkKpDggibMgfS+gL/G/Ue3v53JKJlrWnuLCZbVeYzhfUFL8aV2Y91vfe4ibuL4rGs/cAooUBg8Qt8kPYxovRkLk90rpI60EiMYgsEPiGJpBzvVRCKr/GaQw528RpwhgVX9L0aWgysbPsUKmkN8REQs5QJEp36iHis+NnKly+ypSoRZEi+h/8wKRCs8EsUOjHux7SupdzIak4Cs5x06uXOs6xvqTjf3fFB91Lru80q9O5ATGqUwD5sLZ/HvWL4K1deyf1kE09RS6Mh0zx+qM05nozlpZL1RfALezg== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CY5PR11MB6487.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230022)(396003)(376002)(366004)(39860400002)(346002)(136003)(451199015)(6506007)(55016003)(316002)(71200400001)(7696005)(38100700002)(478600001)(2906002)(66476007)(76116006)(4326008)(5660300002)(53546011)(66556008)(66446008)(66946007)(122000001)(8676002)(86362001)(64756008)(52536014)(41300700001)(8936002)(6636002)(82960400001)(38070700005)(33656002)(54906003)(110136005)(186003)(83380400001)(9686003)(26005); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?7BN72MVgJ6gljbn5w8WiiS24y5NWK3CKmE0vCM1Qjuu4AL83hcAEX5X0Brks?= =?us-ascii?Q?YZc8CHGg8X0XeVOcLg3sGqp5wk8y321cT1MjIm59JfdRejPEu9zeOiEiYESD?= =?us-ascii?Q?75daDHXWByFQXYVHDdwdHYH7zgVvZrY2Euk3V0g9jO384B6NscXQC1Ah9MdD?= =?us-ascii?Q?phZw/OVHiJ5O9RvREW4+SR6NQLTMcmCGKK4bL/osB7e1orbohSHod2NEmzyR?= =?us-ascii?Q?j6goqfA/oOym2fEQjXbe2c6wXif1vtZIRFR4P8FoGXp1hKOLVTUrYEibLM26?= =?us-ascii?Q?vOWpwVVYnJaeSfXyeoaEdxmYSba21zJ6a0M4DN/FWX4n4+IdRxlZvG464cG6?= =?us-ascii?Q?OD2zobfemGDCEoVQOELXXdBhCxynMGVex27k1NyqHL+qaTnVt4S6kLX1p875?= =?us-ascii?Q?+ou5d+b/9MBnQJLd+kYnLZvoeDCg/TQQ1azFqDPLNwvN28vRSOm6sQ9jHNwz?= =?us-ascii?Q?uSqQvd3+fE2MgDHO9PWjQG3ywyyHwHlSqSC4KuHhTmeWwwt5jAahlKf4cp1Q?= =?us-ascii?Q?OrzOhzSpb1eK7TNycqXMhX9jORgd634pKLEAYkWisr/b+0pdv/B+wqwHDBlI?= =?us-ascii?Q?RS4v/k8t2V5/NVwZn5i2Ar5OADR5slTFewb49GEKntFvxQePXN8Na5ZE+i3s?= =?us-ascii?Q?fEEQl+iINvrvn7kILo2lH9wgu3pZ6XwiCm0nXAPTZQHQLPPIp36pcARq+41q?= =?us-ascii?Q?YONtXRuJ4Lb6NKVYCLNEXSDOcOppebiPabWilkN2ZrqRWGZ/dGB2Bls2YQAf?= =?us-ascii?Q?L2awRc9RBIRqegxpII+lgwsPARgvfnnWRECKj3EvpTf+0mBuf1NPqo2k1NFF?= =?us-ascii?Q?glGGzQfG8qVQkRVkNOJidPMANh+CPbfmCV/rO23OPul4Bo48ZDJjclUgXYaI?= =?us-ascii?Q?QzUrk+u1o12joDfXAn0nBg5uzt6mXNU3JE7fqkzpKQhdwwfsRjfAblLRlnRq?= =?us-ascii?Q?eq+a+RAC5zEGJyPk6+ox3sm1v9eTFRscsSGQYZB7QUpLIH+A9+ro9X67hmlX?= =?us-ascii?Q?pB7uFoy1v4GgFldCEpz29Au/fDiGa9HdXiGJ2A+dTrlJItaorwtS4IgRTICf?= =?us-ascii?Q?RkE/cg4bWjIxsuMKlqeDfLLduyQSkzqFkazktRiDXIk1Qnu4gu376kK8SF2v?= =?us-ascii?Q?V2yNYqRZYyEj+ca8NYjTh5d71tXwNCh19KmU6yZm1h73Z0bOCydzSMiQUUD+?= =?us-ascii?Q?qnmjcDJTzNKRJdMc81G9oRtBm4Y61ZquneB3REfo/GUu7aEH2k4VXQuNpQyE?= =?us-ascii?Q?u238PuCu4UwyGKd4AIaIzEns9TI4cwuhbgoiDjBg5BXh4hxpSjHNyiqMpC+H?= =?us-ascii?Q?Ff9nFCkii8qwO/26PJto6cjAm8wP+UpF5wlzDehJPWRBqDo4VYBGN+a4q+e1?= =?us-ascii?Q?tvHzzwv+oE/gK6y+mbbfmf7EPnHPUA0C0FgIOrm92xej4adsSfXR0Sdkuac7?= =?us-ascii?Q?9xfbt1I6Fdt0LpR4B183i2iTV0nR+zD7vMa+oEUk2m17FA5fkaJLg1YKTuaP?= =?us-ascii?Q?nOHoNzikGyI3kJEziU56WL5+E3a184ybRvx+SbljBRpMA3gzjrRmQPGvkTEr?= =?us-ascii?Q?yg9PB5rLK7oNge3c9bKP5cz1kbrtvsX+c2jVnbmb?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: CY5PR11MB6487.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 5a2ed5b4-3f31-43fc-6e0f-08daaa7eb095 X-MS-Exchange-CrossTenant-originalarrivaltime: 10 Oct 2022 05:17:13.1218 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: Eo7pBJpATMiRztl1sBwdihBkoBF62O7OF575jcO9KRYS9a8CfZn1Y3apwy6LyTKfCZ2cCGkXYLlHZ60LOrs5Ig== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN9PR11MB5532 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Hi Chenbo and Maxime, > -----Original Message----- > From: Wang, YuanX > Sent: Wednesday, August 24, 2022 12:36 AM > To: maxime.coquelin@redhat.com; Xia, Chenbo ; > dev@dpdk.org > Cc: Hu, Jiayu ; He, Xingguang > ; Jiang, Cheng1 ; Wang, > YuanX ; Ma, WenwuX > Subject: [PATCH v3] net/vhost: support asynchronous data path >=20 > Vhost asynchronous data-path offloads packet copy from the CPU to the > DMA engine. As a result, large packet copy can be accelerated by the DMA > engine, and vhost can free CPU cycles for higher level functions. >=20 > In this patch, we enable asynchronous data-path for vhostpmd. > Asynchronous data path is enabled per tx/rx queue, and users need to > specify the DMA device used by the tx/rx queue. Each tx/rx queue only > supports to use one DMA device, but one DMA device can be shared among > multiple tx/rx queues of different vhostpmd ports. >=20 > Two PMD parameters are added: > - dmas: specify the used DMA device for a tx/rx queue. > (Default: no queues enable asynchronous data path) > - dma-ring-size: DMA ring size. > (Default: 4096). >=20 > Here is an example: > --vdev > 'eth_vhost0,iface=3D./s0,dmas=3D[txq0@0000:00.01.0;rxq0@0000:00.01.1],dma= - > ring-size=3D4096' >=20 > Signed-off-by: Jiayu Hu > Signed-off-by: Yuan Wang > Signed-off-by: Wenwu Ma > --- >=20 > static uint16_t > eth_vhost_rx(void *q, struct rte_mbuf **bufs, uint16_t nb_bufs) { @@ - > 403,7 +469,7 @@ eth_vhost_rx(void *q, struct rte_mbuf **bufs, uint16_t > nb_bufs) > uint16_t nb_receive =3D nb_bufs; >=20 > if (unlikely(rte_atomic32_read(&r->allow_queuing) =3D=3D 0)) > - return 0; > + goto tx_poll; >=20 > rte_atomic32_set(&r->while_queuing, 1); >=20 > @@ -411,19 +477,36 @@ eth_vhost_rx(void *q, struct rte_mbuf **bufs, > uint16_t nb_bufs) > goto out; >=20 > /* Dequeue packets from guest TX queue */ > - while (nb_receive) { > - uint16_t nb_pkts; > - uint16_t num =3D (uint16_t)RTE_MIN(nb_receive, > - VHOST_MAX_PKT_BURST); > - > - nb_pkts =3D rte_vhost_dequeue_burst(r->vid, r->virtqueue_id, > - r->mb_pool, &bufs[nb_rx], > - num); > - > - nb_rx +=3D nb_pkts; > - nb_receive -=3D nb_pkts; > - if (nb_pkts < num) > - break; > + if (!r->async_register) { > + while (nb_receive) { > + uint16_t nb_pkts; > + uint16_t num =3D (uint16_t)RTE_MIN(nb_receive, > + > VHOST_MAX_PKT_BURST); > + > + nb_pkts =3D rte_vhost_dequeue_burst(r->vid, r- > >virtqueue_id, > + r->mb_pool, &bufs[nb_rx], > + num); > + > + nb_rx +=3D nb_pkts; > + nb_receive -=3D nb_pkts; > + if (nb_pkts < num) > + break; > + } > + } else { > + while (nb_receive) { > + uint16_t nb_pkts; > + uint16_t num =3D (uint16_t)RTE_MIN(nb_receive, > VHOST_MAX_PKT_BURST); > + int nr_inflight; > + > + nb_pkts =3D rte_vhost_async_try_dequeue_burst(r- > >vid, r->virtqueue_id, > + r->mb_pool, &bufs[nb_rx], > num, &nr_inflight, > + r->dma_id, 0); > + > + nb_rx +=3D nb_pkts; > + nb_receive -=3D nb_pkts; > + if (nb_pkts < num) > + break; > + } > } >=20 > r->stats.pkts +=3D nb_rx; > @@ -444,6 +527,17 @@ eth_vhost_rx(void *q, struct rte_mbuf **bufs, > uint16_t nb_bufs) > out: > rte_atomic32_set(&r->while_queuing, 0); >=20 > +tx_poll: > + /** > + * Poll and free completed packets for the virtqueue of Tx queue. > + * Note that we access Tx queue's virtqueue, which is protected > + * by vring lock. > + */ > + if (!async_tx_poll_completed && r->txq->async_register) { > + vhost_tx_free_completed(r->vid, r->txq->virtqueue_id, r- > >txq->dma_id, > + r->cmpl_pkts, VHOST_MAX_PKT_BURST); > + } > + For Tx queue's rte_vhost_poll_enqueue_completed function, there are two pla= ce to call it. One is the TX path, the other is the RX path. Calling it in the= RX path can make the front-end receive the last burst of TX packets when there are no m= ore TX operations, but it also potentially causes more DMA contentions between lco= res. For example, testpmd has 2 lcores, one NIC port and one vhost PMD port with one queue. The RXQ and TXQ of vhost PMD port use dedicated DMA devices. So the traffic flow is like below: lcore0: NIC RXQ -> vhost TXQ lcore1: vhost RXQ -> NIC TXQ Calling rte_vhost_poll_enqueue_completed function in the vhost RX path will= cause lcore1 contend the DMA device used by the vhost TXQ while operating the vho= st RXQ. Performance degradation depends on different cases. In the 4-core 2-queue P= VP case, testpmd throughput degradation is ~10%. Calling it in the TX path can avoid the DMA contention case above, but the = front-end cannot receive the last burst of TX packets if there is no more TX operatio= ns for the same TXQ. In the current implementation, we select the first design, but users can en= able vhost PMD to call rte_vhost_poll_enqueue_completed in the TX path by the testpmd = command during runtime. So the DMA contention above can be avoided in testpmd. I wonder if you have any comments on the design? Thanks, Jiayu > return nb_rx; > } >=20