From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id A25C8A0552; Thu, 20 Oct 2022 16:00:33 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 4D1E642D53; Thu, 20 Oct 2022 16:00:33 +0200 (CEST) Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by mails.dpdk.org (Postfix) with ESMTP id 1BEEC42C61 for ; Thu, 20 Oct 2022 16:00:30 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1666274431; x=1697810431; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-transfer-encoding:mime-version; bh=KhzIPLE+F7B1r7M2jbD/6jy2ySzyYwxneFU6N9b+U6o=; b=Vt0TcoUEmqOqIgNXYtjS2YsPGBmmdYLlybYCWmJBfK9Eu0yA9o/3GO7B emAEeu1Cl56Rl6fGCvyox78WRcAxFTAD4/8ezT/5FZXDMpgk3QygksJaM DGtO8gkFw7cnGFI4LtBUvoml5XOYTWMKBErt7j3iYkO+2Pi2w0a24eixj E9+KldlKbaoE2h2L/6+cqdUOOeBO/MYim/JsACsOkb2NaEEFwdSe3a7Xx ahj9Qn2vCD4C9YvTPerwerVkwMMChhI3JawlAnjdykIYu9qveBIu60vun LHkjCqQUo1k1oSzLFIApjQBTehDXFMBYNwBae0tiStpYcnwGGi5SSEH/T Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10506"; a="308393523" X-IronPort-AV: E=Sophos;i="5.95,198,1661842800"; d="scan'208";a="308393523" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Oct 2022 07:00:22 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10506"; a="804895869" X-IronPort-AV: E=Sophos;i="5.95,198,1661842800"; d="scan'208";a="804895869" Received: from fmsmsx602.amr.corp.intel.com ([10.18.126.82]) by orsmga005.jf.intel.com with ESMTP; 20 Oct 2022 07:00:21 -0700 Received: from fmsmsx611.amr.corp.intel.com (10.18.126.91) by fmsmsx602.amr.corp.intel.com (10.18.126.82) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Thu, 20 Oct 2022 07:00:21 -0700 Received: from fmsmsx610.amr.corp.intel.com (10.18.126.90) by fmsmsx611.amr.corp.intel.com (10.18.126.91) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Thu, 20 Oct 2022 07:00:20 -0700 Received: from fmsedg602.ED.cps.intel.com (10.1.192.136) by fmsmsx610.amr.corp.intel.com (10.18.126.90) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31 via Frontend Transport; Thu, 20 Oct 2022 07:00:20 -0700 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (104.47.66.43) by edgegateway.intel.com (192.55.55.71) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2375.31; Thu, 20 Oct 2022 07:00:20 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=RgKDtcEdw5O6k5n3IKvTJjNISTEUlAjw3uWSHBIhI7g7vJeTi5KCvUJgsd6bxTHZWa6Is+KcwzLsvnhG1ZKhqoecJaAhwmbOuj7Lzs6JTS4zIm8hvN+J4aKLfbu/btuZg4iV8M2DA4BDdkWu+476PwSFVwAOysyRzQZ8phgHyK+9QdDzSNxFg+qWNa0wTC5PmAo08QcGzt9UsTKWGgSEuNUe8AeGIptiRBEAX1jMNbVmiD34kx9I2cJyG2QuWxYexO1m5bVyI+EM3CDlxWJqmTuiy98U6KoqNAcKSLAGEF00Sb9+AAesn011B8tog4BZ205nsPNlgrJx/JjNlKzsbA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=o/Mx8jyJ3h5a6uPe2equapceQRO6GS/4wqSPY0jLMlY=; b=LH4QZFj/05BUIDsPqVNWrfX6eJfRubLZ/EIGKRqeK36n7Um8qiqtEp/mHBA+iUiJ171zR9dxDxsvUP6zLTBr9YHUDOc9df84UtEmAFSYxC4RccFx07q+tOi/5ztPFOrAik5sdOGdpVx8QP1Gmbtsmr62IZUQHuxNxvlDHdRMEeKAvXkWxT+Qr2qwPbDfvT7na9pPXvBy1jzVuDmzmTbAgyGIyBD8evQNc4w8HaARbz8XamYA6cuQmb+TXho+7BoBe81gMpOomV+SQvtZbwGuCDJM/HOGSWD/FmSo1FypfGFgSi7LcbE2+93eS5e6uMsnvpTNWM87JyAmAvKqS29Jgg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Received: from PH7PR11MB6953.namprd11.prod.outlook.com (2603:10b6:510:204::6) by CH0PR11MB5266.namprd11.prod.outlook.com (2603:10b6:610:e1::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5723.34; Thu, 20 Oct 2022 14:00:18 +0000 Received: from PH7PR11MB6953.namprd11.prod.outlook.com ([fe80::23da:cc93:ad21:d969]) by PH7PR11MB6953.namprd11.prod.outlook.com ([fe80::23da:cc93:ad21:d969%9]) with mapi id 15.20.5723.033; Thu, 20 Oct 2022 14:00:18 +0000 From: "Wang, YuanX" To: "Xia, Chenbo" , "maxime.coquelin@redhat.com" CC: "dev@dpdk.org" , "Hu, Jiayu" , "Jiang, Cheng1" , "Ma, WenwuX" , "He, Xingguang" Subject: RE: [PATCH v4] net/vhost: support asynchronous data path Thread-Topic: [PATCH v4] net/vhost: support asynchronous data path Thread-Index: AQHY0/t/GhEd5os8jE24PJNi+8u3oq4V4UEAgAFGPlA= Date: Thu, 20 Oct 2022 14:00:18 +0000 Message-ID: References: <20220814150636.2260317-1-jiayu.hu@intel.com> <20220929194705.1753793-1-yuanx.wang@intel.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-version: 11.6.500.17 dlp-reaction: no-action dlp-product: dlpe-windows authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: PH7PR11MB6953:EE_|CH0PR11MB5266:EE_ x-ms-office365-filtering-correlation-id: ef2b67a9-2a2d-4446-b4bb-08dab2a36c00 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: OOK+DyWx9vLInaJTJM7/cxn1M9Ea+SbOhxfkg8SJG44SCi77pvK1lrvqcUCzgESATHHSSZjHuAlDxNkc0NtK9cQ0cvpfwVbWudL6yV521A/NelfNAU9SoU29+17FbT8FGv2I6MnjPbYFoug2KLKua61OtL7TO5+Usw3/UDo5oUCJvKDAolP6ms1ggzsZdNkdsZzpOs5Zm0OROuSI0ziiFC/pPqZKNyqGtLsqEbtzLy6Jv5ok6G9y9f/GUBKmwrxQm81QN4mY6nFTDZ07DAueeaFBDRoGneqHfCb/2zXm9MTtpkQe6W0LY7srHHoomZptcXln01MAoa5snaZume5w3Wf3muK4mvQeL6PItGZ0xpqE0HiQPzlIJ5QoY1UoilysYcfPDP1m+nJBFoh8FXX3QD7+cuZ6x4pY/2kl1XJdlKjhL/URbamYxs/CkZL19e+/u88VMaC/EsDKw3Fx5NaCSPt6f21MtgrRvChZPF9tvxgMISFhcGTURRsfawssSca6rk+BF/4RrePwV8OOUJ3PZohRcbDhwUCByw3Xm8pWk1ov0sLfNh4ug7Fhf9/KKUa/WBl0J6zXbcdT66JKPkVDJ4v/8xBr07qML4TQBSS5KasakhLL/WTz4iPv5UySsrHe5AT1SmZcudbLHEPQK5qQ5VNCp/J2oODNKTfdw/hTQweXv8IeGfPVH3BtjeSs2H7YBbE8nwUPdnbujynGjzh10ZK5U4Xuf6rT0Glt63IQY3kC2p9R/dbTQLRAkZrwuAvzz4gBI+8/STOUD0hl3wLCfA== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:PH7PR11MB6953.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230022)(39860400002)(366004)(396003)(136003)(376002)(346002)(451199015)(186003)(107886003)(53546011)(83380400001)(2906002)(9686003)(7696005)(6506007)(5660300002)(26005)(55016003)(8676002)(41300700001)(110136005)(66446008)(66556008)(478600001)(4326008)(8936002)(54906003)(64756008)(66476007)(66946007)(52536014)(316002)(71200400001)(76116006)(86362001)(33656002)(82960400001)(38100700002)(38070700005)(122000001); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?V1Gu68Fxbf1bXZSI/MSngF+SjUsUReFfEYseL86kkwCn0jIJxh3M95QvG9dt?= =?us-ascii?Q?sCdEAXEGx6mGgnY2jhJ/QHfG+vHRP4p9+a1XVX6Paj/9eezDFh8T3AZ+kck6?= =?us-ascii?Q?44O2acfyOrnGm/C7vXFH9OT6kmqRRm31HjXyIS4B1nakGIT+YynOfPCESf2r?= =?us-ascii?Q?CgBgv3ZRsjkrQvhdXDhETfwIVdQMe1Bc8S5WkVmjQiOOFpECf7GOaS0b/Iks?= =?us-ascii?Q?fHq3fekq3IO5M3s8u0w0MNXbnOGjCUmYgwU+hKLz95pAMlE+MJx0hy+nn62T?= =?us-ascii?Q?3nfPOvk30TewkXCau8Gg78Qn52XGS2wkCuoq+w1W4FDyvKD/ydTFHNL3MczJ?= =?us-ascii?Q?V5gdh+79xq1wJuOOHrR1ruMOvm1NH8K61D9Bi8STD7oJcHOXc27KzfiUdrwQ?= =?us-ascii?Q?7NeDwPgFx1GJ3PbwXqPcb0ysNfUF4HQEvGrGPjyODXZ53hxwAusXlNeJ4d3p?= =?us-ascii?Q?4ZbGC+W8gjdIkKAWwfzotZfLoKUg5LYMmhONI5d2cn9s+3cRfl7JQXzW3gpu?= =?us-ascii?Q?b8t7u3BkL1mTnBiyQMbwYxhjCczdtvOe5BPsgYbCltGBUORFysLukTSDi9dE?= =?us-ascii?Q?9qwlzQpn7MllxsDgKgvpf66t3ksdDnCY3fJ98u6Qevwy3iNngFyDWm9TQtGv?= =?us-ascii?Q?OfEMQnAJFmG4NLqLiGcrVufNS5krdWejDi3jMqFQ8UUkXF8vCMNAScAfOZ8f?= =?us-ascii?Q?QOZw0VwuFEecGTimhs+8fxaTxoLHMB6vDbRF0oP7/2MnzK35tJcRNQn6a7jF?= =?us-ascii?Q?zV8hS+D+cxxISiYx+UeOMwIkoZnivO5V+uDdh2M8k5AoWIpaZeHUdDT0eymF?= =?us-ascii?Q?reeeYgbDr7GwJvP5EvwVyDZuS6vm79mWs53MEdlT1PmohtCg3T7kJc3/Tpy/?= =?us-ascii?Q?qaqsxpVt/uwV0N3857mTbEXM95P1X+I06HUcJ2oHzhn2Bhd97BqF/u0RYCkV?= =?us-ascii?Q?5R+231nBuR+1JllBGUPENYAd1YhnOWbUgcqXqm2qABW2Lzns7jjT2f5qQHd+?= =?us-ascii?Q?KPCXy/uXYj3J02iaZw5kqKG/VuG7MStN2jg+jj4ovrbiGYxt317BI/N5oM0j?= =?us-ascii?Q?houor0AnrLDxcDoLFlSbPq5q2mx1C12NFzOaJBoq6xiIxZo2y++g3qQfptwr?= =?us-ascii?Q?3ztDVeOb/Ntwly7H7YYzYWE6YBYimS++JDzfx6TPL1c4u+bguSpOhlvzg0lH?= =?us-ascii?Q?3tLRl33Wz48YDm2GlyJNLaXRLZqWqCcsWFEYxbx5CqiIARolZs9EhORAvYTi?= =?us-ascii?Q?nkV5UEkABYtEPsBZAF2SlSChG/bJZfnB44mzT6B9yqaWxKtzMg5dvrTMmQNr?= =?us-ascii?Q?YmSIVMD2LGriZUUGulm9E+fHx/X0ijK6Yo7NJGUyeB8nHfhr/9xMg82nmPYi?= =?us-ascii?Q?q3b5vuG4Vg9xlRG3f79hY5JXWT9KhwlIMtuWIC0y776EuMHMHSBg1IOqMHAQ?= =?us-ascii?Q?v02sLXnDZCrE2+dT1JIBuo4oI4h+P9McpNgVnCz1AqnFWk2mKLcCnQWTowPc?= =?us-ascii?Q?fYwMp5rJC1haI+lIi6VCbzX0VdD1PSM3fk8c9SjoVZRHHMiKqGHTcJSLa+a+?= =?us-ascii?Q?B8Yr7lVHcVBUKk8iYX6sR5zA/X/HB/XOBrrYfS0X?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: PH7PR11MB6953.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: ef2b67a9-2a2d-4446-b4bb-08dab2a36c00 X-MS-Exchange-CrossTenant-originalarrivaltime: 20 Oct 2022 14:00:18.7120 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: lAixeIA56Li7/fjXyZ/9bm018nOQZ8C0fuGJ7FbTz991JDPvuS8MFN1/zIWhZkjJLyDTSYOWyhmarpristjkxA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR11MB5266 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Hi Chenbo, Thanks for your review. Please see replies inline. > -----Original Message----- > From: Xia, Chenbo > Sent: Wednesday, October 19, 2022 10:11 PM > To: Wang, YuanX ; maxime.coquelin@redhat.com > Cc: dev@dpdk.org; Hu, Jiayu ; Jiang, Cheng1 > ; Ma, WenwuX ; He, > Xingguang > Subject: RE: [PATCH v4] net/vhost: support asynchronous data path >=20 > Hi Yuan, >=20 > Overall it looks good to me, two inline comments below. >=20 > > -----Original Message----- > > From: Wang, YuanX > > Sent: Friday, September 30, 2022 3:47 AM > > To: maxime.coquelin@redhat.com; Xia, Chenbo > > Cc: dev@dpdk.org; Hu, Jiayu ; Jiang, Cheng1 > > ; Ma, WenwuX ; He, > > Xingguang ; Wang, YuanX > > > Subject: [PATCH v4] net/vhost: support asynchronous data path > > > > Vhost asynchronous data-path offloads packet copy from the CPU to the > > DMA engine. As a result, large packet copy can be accelerated by the > > DMA engine, and vhost can free CPU cycles for higher level functions. > > [...] > > @@ -486,31 +581,49 @@ eth_vhost_tx(void *q, struct rte_mbuf **bufs, > > uint16_t nb_bufs) > > } > > > > /* Enqueue packets to guest RX queue */ > > - while (nb_send) { > > - uint16_t nb_pkts; > > - uint16_t num =3D (uint16_t)RTE_MIN(nb_send, > > - VHOST_MAX_PKT_BURST); > > + if (!r->async_register) { > > + while (nb_send) { > > + num =3D (uint16_t)RTE_MIN(nb_send, > VHOST_MAX_PKT_BURST); > > + nb_pkts =3D rte_vhost_enqueue_burst(r->vid, r- > > >virtqueue_id, > > + &bufs[nb_tx], num); > > + > > + nb_tx +=3D nb_pkts; > > + nb_send -=3D nb_pkts; > > + if (nb_pkts < num) > > + break; > > + } > > > > - nb_pkts =3D rte_vhost_enqueue_burst(r->vid, r->virtqueue_id, > > - &bufs[nb_tx], num); > > + for (i =3D 0; likely(i < nb_tx); i++) { > > + nb_bytes +=3D bufs[i]->pkt_len; > > + rte_pktmbuf_free(bufs[i]); > > + } > > > > - nb_tx +=3D nb_pkts; > > - nb_send -=3D nb_pkts; > > - if (nb_pkts < num) > > - break; > > - } > > + } else { > > + while (nb_send) { > > + num =3D (uint16_t)RTE_MIN(nb_send, > VHOST_MAX_PKT_BURST); > > + nb_pkts =3D rte_vhost_submit_enqueue_burst(r->vid, > r- > > >virtqueue_id, > > + &bufs[nb_tx], num, > r->dma_id, 0); > > + > > + nb_tx +=3D nb_pkts; > > + nb_send -=3D nb_pkts; > > + if (nb_pkts < num) > > + break; > > + } > > > > - for (i =3D 0; likely(i < nb_tx); i++) > > - nb_bytes +=3D bufs[i]->pkt_len; > > + for (i =3D 0; likely(i < nb_tx); i++) > > + nb_bytes +=3D bufs[i]->pkt_len; > > > > - nb_missed =3D nb_bufs - nb_tx; > > + if (unlikely(async_tx_poll_completed)) { > > + vhost_tx_free_completed(r->vid, r->virtqueue_id, r- > > >dma_id, r->cmpl_pkts, > > + VHOST_MAX_PKT_BURST); > > + } > > + } >=20 > About the stats, should we update them when packet is completed? > Anyway in vhost lib we have inflight xstats, user can know how many packe= ts > are finished and how many are in-flight. >=20 We think the way it is now makes more sense. For asynchronous path, the completed packets from rte_vhost_poll_enqueue_co= mpleted() may not be the packets of this burst, but from the previous one.= =20 Using this value for statistics does not accurately reflect the statistics = of this burst,=20 and makes the missed_pkts definition inconsistent between the synchronous a= nd asynchronous paths. After rte_vhost_submit_enqueue_burst(), most enqueue process is completed, = leaving the DMA copy and update vring index, while the probability of error in the DMA copy is extremely low (If an erro= r occurs, it can be considered a serious problem, then the error log recorded, and the data plane should be stopped). Therefore, the packages submitted to the DMA this burst can be used to repr= esent the completed packages. This statistic is also more in line with the = original meaning. > > [...] > > + > > +static void > > +cmd_tx_poll_parsed(void *parsed_result, __rte_unused struct cmdline > > +*cl, > > __rte_unused void *data) > > +{ > > + struct cmd_tx_poll_result *res =3D parsed_result; > > + > > + if (!strcmp(res->what, "on")) > > + rte_eth_vhost_async_tx_poll_completed(true); > > + else if (!strcmp(res->what, "off")) > > + rte_eth_vhost_async_tx_poll_completed(false); > > +} >=20 > Sorry I forgot to reply v3. I think it's better to do something like >=20 > fprintf(stderr, "Unknown parameter\n"); Thanks for the suggestion. Will do in v5. Thanks, Yuan >=20 > Thanks, > Chenbo >=20 > > + > > +static cmdline_parse_inst_t async_vhost_cmd_tx_poll =3D { > > + .f =3D cmd_tx_poll_parsed, > > + .data =3D NULL, > > + .help_str =3D "async-vhost tx poll completed on|off", > > + .tokens =3D { > > + (void *)&cmd_tx_async_vhost, > > + (void *)&cmd_tx_tx, > > + (void *)&cmd_tx_poll, > > + (void *)&cmd_tx_completed, > > + (void *)&cmd_tx_what, > > + NULL, > > + }, > > +}; > > + > > +static struct testpmd_driver_commands async_vhost_cmds =3D { > > + .commands =3D { > > + { > > + &async_vhost_cmd_tx_poll, > > + "async_vhost tx poll completed (on|off)\n" > > + " Poll and free DMA completed packets in Tx path.\n", > > + }, > > + { NULL, NULL }, > > + }, > > +}; > > + > > +TESTPMD_ADD_DRIVER_COMMANDS(async_vhost_cmds) > > -- > > 2.25.1