From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E221DA0560; Tue, 18 Oct 2022 13:59:33 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 81F1E40A8B; Tue, 18 Oct 2022 13:59:33 +0200 (CEST) Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by mails.dpdk.org (Postfix) with ESMTP id 370B24021E for ; Tue, 18 Oct 2022 13:59:30 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1666094371; x=1697630371; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-transfer-encoding:mime-version; bh=US+a/Q9bHcRuCT93lUXzeMYDHqIWoZI6ERU+a3Je8e8=; b=bKIAz12BfWb55Gv30ZQU9MaTaE+GrrpkFZyrPKFdLB+tVJHESk51NcZd 82eqAtCGSUrH/dYrgoFnb1lj7UTU2yxBE/OpEHWn8vED0IbTCMc49/Koz 1dxbjpTayAiH+5NbOZu4417j0goWd9GzXFZxxhPv4uWre1YglP52s/Frl yDviMVtDAR6s3ZSMmwH60FT5Xz0wSblgxMt/+UlRvZw0RlxrrMGpDemn7 JFgw/ubGfZOH3FyTKm7BBSZv1bC7X2rS1J5APmGRd3zmYRUE/866QUSs1 hq1Jq/1n2z931NhzE6GNuCm/3n5gJ54Cxqla+YxkNpHEvT2TqC1H6dW1w Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10503"; a="293450094" X-IronPort-AV: E=Sophos;i="5.95,193,1661842800"; d="scan'208";a="293450094" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Oct 2022 04:59:22 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10503"; a="659757721" X-IronPort-AV: E=Sophos;i="5.95,193,1661842800"; d="scan'208";a="659757721" Received: from fmsmsx603.amr.corp.intel.com ([10.18.126.83]) by orsmga008.jf.intel.com with ESMTP; 18 Oct 2022 04:59:22 -0700 Received: from fmsmsx610.amr.corp.intel.com (10.18.126.90) by fmsmsx603.amr.corp.intel.com (10.18.126.83) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Tue, 18 Oct 2022 04:59:21 -0700 Received: from fmsmsx611.amr.corp.intel.com (10.18.126.91) by fmsmsx610.amr.corp.intel.com (10.18.126.90) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Tue, 18 Oct 2022 04:59:21 -0700 Received: from fmsedg601.ED.cps.intel.com (10.1.192.135) by fmsmsx611.amr.corp.intel.com (10.18.126.91) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31 via Frontend Transport; Tue, 18 Oct 2022 04:59:21 -0700 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (104.47.58.168) by edgegateway.intel.com (192.55.55.70) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2375.31; Tue, 18 Oct 2022 04:59:17 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Ra09zn7tnexMDVI5R1nqxX3XtIazqJHgCY4Ur/Ly6SlvD5OV2UsN9btrd6fzznafOskO9NOnXfhfu9/r7y96CvZbnRD02LWKy6Y7avKikrKKwIJjsf2kgKKmUVU1zZseqlG5ApIKjuUTmyC19BqEED4ai5R67kntcvbUUNN+yI16nQ3MzMKymXMFETmryavGkItDf6I6XvBSayadyowHFSuQW42g+06CAIs0FgwtwMaC6J9QTNbMDJKIF8zZb1z5oF9XFLRZCAt27xNVUhSLRlmq1MUZC5V9DzCjNdAtMBuElHLW2O+RigQBiKLFKFLvdQZaTGmDKUb0c7mIANOPPg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=QwjgzmysVx4qr5GxQyJBZZP3AcyY9Sey3eY34sJmL+Y=; b=oWp3nJeHTrtP5mlz94Ja0ZKAKuvrAqEjix4WRSoWAyfb3mbfVYlDVU4bEa14iPFxLX570OmiVhRBGIDKpQ6g7JC59XmOtNsW1BswkkUrKWck7UfI9QVp5ImtnskNRFPNsgKpcAp4FLYV80OCIVEZF0tEsN2riE82saQn3oAT3ocorM9rjqVtkgLnruvlRSlsOuyetjGc7Vs/AOwI5YezVB1b5x5lBuwj/zF9ZHHInry+W2QphYcwvMomI4OHonudj+xDLZ7PoXSDiu7WmhLFoNI0EZb6A5Pyrrd1Vx3lVufroeJ8bSsQs/2gmYN/Ckg/CKBGgce7IcW3ZgMFStkRxA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Received: from SN6PR11MB3504.namprd11.prod.outlook.com (2603:10b6:805:d0::17) by DM4PR11MB5357.namprd11.prod.outlook.com (2603:10b6:5:394::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5723.26; Tue, 18 Oct 2022 11:59:15 +0000 Received: from SN6PR11MB3504.namprd11.prod.outlook.com ([fe80::39bf:57b1:4824:d40d]) by SN6PR11MB3504.namprd11.prod.outlook.com ([fe80::39bf:57b1:4824:d40d%6]) with mapi id 15.20.5723.033; Tue, 18 Oct 2022 11:59:15 +0000 From: "Xia, Chenbo" To: "Hu, Jiayu" , "maxime.coquelin@redhat.com" CC: "He, Xingguang" , "Jiang, Cheng1" , "Ma, WenwuX" , "Wang, YuanX" , "dev@dpdk.org" Subject: RE: [PATCH v3] net/vhost: support asynchronous data path Thread-Topic: [PATCH v3] net/vhost: support asynchronous data path Thread-Index: AQHYts0YCzM5cA8KAk63uznoAs43U64HYYGAgAz/SoA= Date: Tue, 18 Oct 2022 11:59:15 +0000 Message-ID: References: <20220814150636.2260317-1-jiayu.hu@intel.com> <20220823163534.947869-1-yuanx.wang@intel.com> In-Reply-To: Accept-Language: en-US, zh-CN Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: SN6PR11MB3504:EE_|DM4PR11MB5357:EE_ x-ms-office365-filtering-correlation-id: fd28cc9a-08af-437a-4a46-08dab1002e1e x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: mv76JY1AYMQ55nIFL9GLe+J6YKlUyxqcbjWlYW3kXwBLatL4FnUp1ccHRTTiKk41PvKUVVOAqgSUlexSjsi/Fn9/HuLicl3Fw/XZF8mJtXqx/RcOYW3cF/PrpYARYy19ne9BFsAxqt6yFYz3gEIQp4y08I8YqCntngRCg46oRUtPLbIHi7FjGxS/jDlOx8K1BSew18M468ZU8eV7063EQHjTujhSknbifL0r15/V8se+q5oFs+pTUyJU+dxSmJJg4fhnasQPpUlgjwfGLbQm4nrFL105jyBg3JJU4tEaqX4UvpWvZTlfJn3mZovXyMkr4FrdEnGcCfDZ9r6t1UBIziZ/HtgDZ9KntUuhSXYgQnbj3NsyrO525mHn8WSL+epH6/lBpRe/trnJABMJuuleujYSn2mYNHWX5P6nhMBWwL6cftOchB1D0aOciPa0xK221cK5CkGAM6k6VgWjDYcpRBSGUqFfsqYhLNe0zPZlCt8PLZGok541l9yT1alv9oThRNNsIjWuvE5VQCdsHqeMxl4mBqGX9meBy0Bst4PEqvX5F+HWIxIeWY1zHSA55Khhsx4HFpGOMQCGTm0XgfPaMXumVA1ln8Z0Xd53XPm+BFOdg+N+ZcM7wWH9GqriFSVw+MauUgm5441MOw3vpKRV8GcO5QoGeSC5irIJbWARIUVyRlEaOBfdzcc4/ZPHUxKe+4cXbe9SKtOuBnnyB+shPZpqRmmRblDUNYO+btChn3Q9zU+sbbh0XBvMSrwvlrQ39P6hen2Tov+NbpmiGFi5JA== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:SN6PR11MB3504.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230022)(39860400002)(136003)(366004)(396003)(346002)(376002)(451199015)(33656002)(52536014)(86362001)(2906002)(5660300002)(38100700002)(122000001)(186003)(83380400001)(38070700005)(82960400001)(53546011)(26005)(6506007)(316002)(7696005)(478600001)(54906003)(71200400001)(9686003)(66946007)(66476007)(66556008)(76116006)(66446008)(110136005)(55016003)(64756008)(4326008)(41300700001)(8936002)(8676002); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?WHMGYSVUjqtyMwxo5+x9s+33rMnC0XVNsuee0Cu+hF/W6o1hZCDdfG2J5i3P?= =?us-ascii?Q?97VqOOdZlw31cZHWEnsdCNmvrGQoSiyZ1l/E0aB+2YL/ZXtBL+4IzBb5uCKE?= =?us-ascii?Q?FdpXBGwgHllio+1rF48p/YjN09MbdoTJwNETnatc32iW8jVaKyL8mfJ+2NU+?= =?us-ascii?Q?apUWFsl3ybQBWpHfbhOLDkr1Ei9edcRb7LSqXtywiw6A74WLRrQ4OMMeNmBZ?= =?us-ascii?Q?UZ63NPNy9QBzzDjLWiAb6Tq0cYxD8609W4wEpjEwNEIZLmhK+eOPumU3HwHT?= =?us-ascii?Q?M33pl2Bp4fHhHfROljpdXGrYEl3A806QhoaOGFeiKo5QWP0nSB8v2wEGjPf/?= =?us-ascii?Q?K0pY90SfrB+zSlsuOqqWKX5tvfI0O8mSQ6NbUPEr7+6WtkzXNtIx4gg5E6k7?= =?us-ascii?Q?Z0YFwbT/6/zDO9ZimFSMwh3dBq9qm/lkAruJhGVtpwKSP64MKamBf5KlMsuW?= =?us-ascii?Q?N02Zc+pnmvRQWqqU2UxQo4rlKXz20rL3geDvzQJUMJQybg7Jx32TYsJz3zdA?= =?us-ascii?Q?nlN859qhzDR2M/Ia2JNmu/wgmfxChIf1KELDvAKsqo5QwvSQ9ZRU+14+YssM?= =?us-ascii?Q?dO/JF22Qxkj/ljL8tBxSavcuPYnqgRycM+oCV7SJX1Xawdyl2wobXSTt+eFs?= =?us-ascii?Q?c/RVt3AUBkuB9sYu0UJIYji0BuZkBI8k+hb287fOUW3Lj+bMWfo+nCzKZowE?= =?us-ascii?Q?hCqWokBqbkQje7UPcWlTLusgiKQVjEZSV5MoqKYOX2KudLuax53jgU2c/AGQ?= =?us-ascii?Q?lXvh3wcsTFw4w6wo84Srqf2HjKMGQ+FRNXCJX8q87v0dvFjFrc4WKAeasG9o?= =?us-ascii?Q?zLzK/+RxWjo3GRg7zK0qyuCsgFAN3FMcePWUXV88IkmNwb1J4oUl7iplvYJk?= =?us-ascii?Q?QkG+EvCvxiHwKHuUgaOJBDSRmtPiMRLqDPfVnhPPUCkb0O6lvlIxNZsLAHr1?= =?us-ascii?Q?oMIqn4FG99ZOfzGJYu8cMrx0EqP8I4RJttRy8oEzsncJWNSbzH9JPKPGHBsh?= =?us-ascii?Q?oPZ603MGLAn7VMJFR42yoU7OhIzyA/JxhyVTQyqCAAd8rukUf0sRMb4wf5f3?= =?us-ascii?Q?1gucwj3Ys/4xu5QlcOyude8MRhFZ+UV9vCdIhAwxVdeDcfn+5BSRrhSDN7Dh?= =?us-ascii?Q?xVkY/Q15wyaLIeNTtE1ANeTzs6ItrLYz7BhvUKxZAUuUe634sDDTTob7/hjN?= =?us-ascii?Q?jp1mFqILOe+VGSw2/cMN/ab+9uArhLxdm27ubqS7yx2802z2CLY/zRTVBwbe?= =?us-ascii?Q?4VNLSZ7fHeevob9gg08q6HN3auQvA2Q7aL2sdZvDD4ZEU30ocA4q/JQhHfyg?= =?us-ascii?Q?SLHWLIpxL0KBizHadQaYWP5FVmrTDQdBeo8UkbG5P5Cv+CbOTLnldRoVH5dR?= =?us-ascii?Q?YWf08wSMapUG8xwnB2TSR9XdwD8Ja4hxjZNJGdnQiNcxIjzLCVCUt7q2Q2eN?= =?us-ascii?Q?z7mJcoPtQR6yecLlzVHdy3QNDDAtAWHXnd8uLk9PMxckTx0GAVwbehnsqlT/?= =?us-ascii?Q?n7WUagyOjJAUxl8R7aJ8f269GluRR9otK1MgV+e4HTlT08XtDwrjcM9+8GdB?= =?us-ascii?Q?wC1AfJYCji+qSpGZbU8xGIHUP8sxBOE7gGxXgvbF?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SN6PR11MB3504.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: fd28cc9a-08af-437a-4a46-08dab1002e1e X-MS-Exchange-CrossTenant-originalarrivaltime: 18 Oct 2022 11:59:15.7887 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: nzbkvjojuZEzWs/jT3id6+YYYGuH+gLQQDzYMBSBkjnLmx1IUNkFy4nCTz89e1v7Ldt1iZobNXB+5hWn0ZPzNA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR11MB5357 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org > -----Original Message----- > From: Hu, Jiayu > Sent: Monday, October 10, 2022 1:17 PM > To: maxime.coquelin@redhat.com; Xia, Chenbo > Cc: He, Xingguang ; Jiang, Cheng1 > ; Ma, WenwuX ; Wang, YuanX > ; dev@dpdk.org > Subject: RE: [PATCH v3] net/vhost: support asynchronous data path >=20 > Hi Chenbo and Maxime, >=20 > > -----Original Message----- > > From: Wang, YuanX > > Sent: Wednesday, August 24, 2022 12:36 AM > > To: maxime.coquelin@redhat.com; Xia, Chenbo ; > > dev@dpdk.org > > Cc: Hu, Jiayu ; He, Xingguang > > ; Jiang, Cheng1 ; Wang, > > YuanX ; Ma, WenwuX > > Subject: [PATCH v3] net/vhost: support asynchronous data path > > > > Vhost asynchronous data-path offloads packet copy from the CPU to the > > DMA engine. As a result, large packet copy can be accelerated by the DM= A > > engine, and vhost can free CPU cycles for higher level functions. > > > > In this patch, we enable asynchronous data-path for vhostpmd. > > Asynchronous data path is enabled per tx/rx queue, and users need to > > specify the DMA device used by the tx/rx queue. Each tx/rx queue only > > supports to use one DMA device, but one DMA device can be shared among > > multiple tx/rx queues of different vhostpmd ports. > > > > Two PMD parameters are added: > > - dmas: specify the used DMA device for a tx/rx queue. > > (Default: no queues enable asynchronous data path) > > - dma-ring-size: DMA ring size. > > (Default: 4096). > > > > Here is an example: > > --vdev > > 'eth_vhost0,iface=3D./s0,dmas=3D[txq0@0000:00.01.0;rxq0@0000:00.01.1],d= ma- > > ring-size=3D4096' > > > > Signed-off-by: Jiayu Hu > > Signed-off-by: Yuan Wang > > Signed-off-by: Wenwu Ma > > --- > > > > static uint16_t > > eth_vhost_rx(void *q, struct rte_mbuf **bufs, uint16_t nb_bufs) { @@ = - > > 403,7 +469,7 @@ eth_vhost_rx(void *q, struct rte_mbuf **bufs, uint16_t > > nb_bufs) > > uint16_t nb_receive =3D nb_bufs; > > > > if (unlikely(rte_atomic32_read(&r->allow_queuing) =3D=3D 0)) > > - return 0; > > + goto tx_poll; > > > > rte_atomic32_set(&r->while_queuing, 1); > > > > @@ -411,19 +477,36 @@ eth_vhost_rx(void *q, struct rte_mbuf **bufs, > > uint16_t nb_bufs) > > goto out; > > > > /* Dequeue packets from guest TX queue */ > > - while (nb_receive) { > > - uint16_t nb_pkts; > > - uint16_t num =3D (uint16_t)RTE_MIN(nb_receive, > > - VHOST_MAX_PKT_BURST); > > - > > - nb_pkts =3D rte_vhost_dequeue_burst(r->vid, r->virtqueue_id, > > - r->mb_pool, &bufs[nb_rx], > > - num); > > - > > - nb_rx +=3D nb_pkts; > > - nb_receive -=3D nb_pkts; > > - if (nb_pkts < num) > > - break; > > + if (!r->async_register) { > > + while (nb_receive) { > > + uint16_t nb_pkts; > > + uint16_t num =3D (uint16_t)RTE_MIN(nb_receive, > > + > > VHOST_MAX_PKT_BURST); > > + > > + nb_pkts =3D rte_vhost_dequeue_burst(r->vid, r- > > >virtqueue_id, > > + r->mb_pool, &bufs[nb_rx], > > + num); > > + > > + nb_rx +=3D nb_pkts; > > + nb_receive -=3D nb_pkts; > > + if (nb_pkts < num) > > + break; > > + } > > + } else { > > + while (nb_receive) { > > + uint16_t nb_pkts; > > + uint16_t num =3D (uint16_t)RTE_MIN(nb_receive, > > VHOST_MAX_PKT_BURST); > > + int nr_inflight; > > + > > + nb_pkts =3D rte_vhost_async_try_dequeue_burst(r- > > >vid, r->virtqueue_id, > > + r->mb_pool, &bufs[nb_rx], > > num, &nr_inflight, > > + r->dma_id, 0); > > + > > + nb_rx +=3D nb_pkts; > > + nb_receive -=3D nb_pkts; > > + if (nb_pkts < num) > > + break; > > + } > > } > > > > r->stats.pkts +=3D nb_rx; > > @@ -444,6 +527,17 @@ eth_vhost_rx(void *q, struct rte_mbuf **bufs, > > uint16_t nb_bufs) > > out: > > rte_atomic32_set(&r->while_queuing, 0); > > > > +tx_poll: > > + /** > > + * Poll and free completed packets for the virtqueue of Tx queue. > > + * Note that we access Tx queue's virtqueue, which is protected > > + * by vring lock. > > + */ > > + if (!async_tx_poll_completed && r->txq->async_register) { > > + vhost_tx_free_completed(r->vid, r->txq->virtqueue_id, r- > > >txq->dma_id, > > + r->cmpl_pkts, VHOST_MAX_PKT_BURST); > > + } > > + >=20 > For Tx queue's rte_vhost_poll_enqueue_completed function, there are two > place > to call it. One is the TX path, the other is the RX path. Calling it in > the RX path can > make the front-end receive the last burst of TX packets when there are no > more TX > operations, but it also potentially causes more DMA contentions between > lcores. > For example, testpmd has 2 lcores, one NIC port and one vhost PMD port > with > one queue. The RXQ and TXQ of vhost PMD port use dedicated DMA devices. S= o > the traffic flow is like below: > lcore0: NIC RXQ -> vhost TXQ > lcore1: vhost RXQ -> NIC TXQ > Calling rte_vhost_poll_enqueue_completed function in the vhost RX path > will cause > lcore1 contend the DMA device used by the vhost TXQ while operating the > vhost RXQ. > Performance degradation depends on different cases. In the 4-core 2-queue > PVP case, > testpmd throughput degradation is ~10%. >=20 > Calling it in the TX path can avoid the DMA contention case above, but th= e > front-end > cannot receive the last burst of TX packets if there is no more TX > operations for the > same TXQ. >=20 > In the current implementation, we select the first design, but users can > enable vhost > PMD to call rte_vhost_poll_enqueue_completed in the TX path by the testpm= d > command > during runtime. So the DMA contention above can be avoided in testpmd. >=20 > I wonder if you have any comments on the design? I now will prefer the current design: 1. Make the packet finish by calling it in RX. 2. Still leave one testpmd option to avoid performance issue/tx-only issue = to get best perf and make tx-only work. As vhost PMD is targeted for mainly testing now, make sure functionality wo= rks well and also leave one way to get some reference performance number (which= will be useful for real production-level APP like OVS to refer). That's good for vhost PMD IMHO. Maxime, any different opinion? Thanks, Chenbo >=20 > Thanks, > Jiayu >=20 > > return nb_rx; > > } > >