From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 262EAA09FF; Mon, 28 Dec 2020 09:22:06 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 7FA21C9D8; Mon, 28 Dec 2020 09:22:04 +0100 (CET) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by dpdk.org (Postfix) with ESMTP id 85E542C2A for ; Mon, 28 Dec 2020 09:22:03 +0100 (CET) IronPort-SDR: YemkiALmZ8vtQj7TQBBBPjYhbmZgdkLkXKG2n8I2r0qx6Sz8Xi5IByR/BcAfZCmyNrLv8wrOnj OeD1nXLHU7Yg== X-IronPort-AV: E=McAfee;i="6000,8403,9847"; a="175583593" X-IronPort-AV: E=Sophos;i="5.78,454,1599548400"; d="scan'208";a="175583593" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Dec 2020 00:22:00 -0800 IronPort-SDR: VQzCtnxHVp7yONARIth1MLLP2cHRMqdyd+dkI/GG9rLuG74r2OcwvjabI2gsYpdNOjyekG/yxG AzOUliRmP/hA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.78,454,1599548400"; d="scan'208";a="358485574" Received: from orsmsx601.amr.corp.intel.com ([10.22.229.14]) by orsmga002.jf.intel.com with ESMTP; 28 Dec 2020 00:21:59 -0800 Received: from orsmsx609.amr.corp.intel.com (10.22.229.22) by ORSMSX601.amr.corp.intel.com (10.22.229.14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Mon, 28 Dec 2020 00:21:59 -0800 Received: from orsmsx606.amr.corp.intel.com (10.22.229.19) by ORSMSX609.amr.corp.intel.com (10.22.229.22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Mon, 28 Dec 2020 00:21:59 -0800 Received: from ORSEDG602.ED.cps.intel.com (10.7.248.7) by orsmsx606.amr.corp.intel.com (10.22.229.19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5 via Frontend Transport; Mon, 28 Dec 2020 00:21:59 -0800 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (104.47.59.173) by edgegateway.intel.com (134.134.137.103) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.1713.5; Mon, 28 Dec 2020 00:21:58 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=YD30nD+iwj3vOXA8iT1BVORSY2x7ptUZ1Kqxy7chstOemVl8kMnOrGKzzbdAaLiTrLyvpgid6JcTC4uYFK+040VpMxQoq7FcWMXGy/o0TYjdIoTghvg4oHvfPqCeIAIcUiOgxMPA68D+uOANIVFhhIy6CGFfMHeYYHeC+2/u1cRR6UlsJWEojMpLwhUESjKpK0F+ZgTGRoe0Z+9t0j3XB9x93ImvzxNTgcuWHptHaAnh9Rc2vO0lPV0zJKbv6EcH3Otih9IHNXRHQU7VA+IN3APmQwNzS/FsjqS3zyf/vSqMlHcqyLYvpvPQxtV19KQj2mo/i8Lge/qX+U2PYYdfww== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=04D5szE0TWqCKi7tG3OlfTHa3NZ8VnG5wjiqQHzw5FE=; b=WuOpVttocTTZyp6yCEka3G8HHdDVWHLtmXbcVbrMYAedeG1eAnrYBouEBbXLcRX07wrk5Zr5d8LWhK8JUFXlugABCS/Ex0OPcA0nSiWUdYjGhAGQZuNGuo6Ou7/KbBvNyf+xSBmN+niLxs07n1PwJGMmRkSlJyqDtxnjO/YzHGNzkf64EwRGL/B0NgmM+EEKjQusLbmgJXGOw2aNUmdR+sJnGBSJWEjS6YYhAWftT76cGPr/iYP8TNKNwYSwWZ0ZXiPj8R3iYnwghmYuYskYEwmPGx03fO1OhghOEBuzBILprOR5c4eG1ltYXgPkQJdZ0dE+bo3doH0rWWbNGcznPw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel.onmicrosoft.com; s=selector2-intel-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=04D5szE0TWqCKi7tG3OlfTHa3NZ8VnG5wjiqQHzw5FE=; b=PAWa5q/UmyDmjllFT4vWM0TKT4LNNCFaVwDXjaGYAl59RRKsXsjOmNjGdEWlY+nOn3wBcJcBhIFwgQl9f0JLIlzEXX/LKNMMU9+btkIKW1Vxit8P1RHgzurg41263j0+KhhCg8l+wdorA8xodx4uENd+Z0bT7AhlXzAneuWS2UM= Received: from SJ0PR11MB5006.namprd11.prod.outlook.com (2603:10b6:a03:2db::22) by BYAPR11MB2728.namprd11.prod.outlook.com (2603:10b6:a02:ca::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3700.31; Mon, 28 Dec 2020 08:21:56 +0000 Received: from SJ0PR11MB5006.namprd11.prod.outlook.com ([fe80::551c:1427:7611:ee6b]) by SJ0PR11MB5006.namprd11.prod.outlook.com ([fe80::551c:1427:7611:ee6b%3]) with mapi id 15.20.3700.031; Mon, 28 Dec 2020 08:21:56 +0000 From: "Jiang, Cheng1" To: "Hu, Jiayu" , "maxime.coquelin@redhat.com" , "Xia, Chenbo" CC: "dev@dpdk.org" , "Yang, YvonneX" Thread-Topic: [PATCH v4 2/2] examples/vhost: refactor vhost data path Thread-Index: AQHW2paUTtCslbe8Skem3QJ4Qed8oaoL56+AgABEhyA= Date: Mon, 28 Dec 2020 08:21:56 +0000 Message-ID: References: <20201218113327.70528-1-Cheng1.jiang@intel.com> <20201225080712.36177-1-Cheng1.jiang@intel.com> <20201225080712.36177-3-Cheng1.jiang@intel.com> In-Reply-To: Accept-Language: zh-CN, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: intel.com; dkim=none (message not signed) header.d=none;intel.com; dmarc=none action=none header.from=intel.com; x-originating-ip: [192.55.46.46] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: dfe8b698-d8bd-4cd7-69da-08d8ab09a3d5 x-ms-traffictypediagnostic: BYAPR11MB2728: x-ld-processed: 46c98d88-e344-4ed4-8496-4ed7712e255d,ExtAddr x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:7691; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 4qAWQDR0f7uqflFeRlA6IDSMG9F7o+wD4gn4bMT2ATcTrBhR3d27C++3a56knkXAYpXbtqp1/czeJhNqJgqak98Y41NKwDJGM5eiapZeQcmnLXjOwiq3meAvPmKk+kDDicV8vzVSAmABOMltS7dQFC6I0W9akRQawT7RQxaPbTkEH3OPzYlrC552tR9wV4WBIZTvI9Poq6aqaQ34fgYsRyhsNIO5uvk4BYFLjsXH6/PK6tx9Y4O1vJvxVsIK+Ja4VEywAy+BPNChzQ5DBPKO9mUj+ktWNh4Gde7leZeBOOOnurfCHlpjTW1Z/unktRLkD06S3i+pRzx4DDtzztk1+J/UG1zjBiW3HB87v2l9+bzlJG2addPHF1AgROCcSAbp2br1UcAU6eGGSTW5E832H3BF9VLrIDq6cV7IBY2WEHwgnqZ1fLn2BWxyZZd3Ck6y x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:SJ0PR11MB5006.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(39860400002)(366004)(376002)(346002)(396003)(136003)(110136005)(71200400001)(30864003)(2906002)(86362001)(316002)(54906003)(478600001)(52536014)(6636002)(5660300002)(26005)(8676002)(7696005)(8936002)(76116006)(6506007)(33656002)(107886003)(55016002)(9686003)(186003)(66556008)(83380400001)(64756008)(66476007)(53546011)(66946007)(66446008)(4326008)(21314003); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata: =?us-ascii?Q?NtqpbDXkCC4htYJ0Yiead3CMv8f7vjCKAEGtyRoxJxQ/cu5nXDrV6q0P0z5T?= =?us-ascii?Q?peCRSeYuirdHFYj1Iko0Sslwg85N+xChu8qoQ1gcx+SvMAshYu59Tp7Elpw5?= =?us-ascii?Q?H7/MFGTzTZNYcAcRdVpMvozI4kAKRfnRVM6ZsyLOzr9T/lhIrKxgtKSvDQJt?= =?us-ascii?Q?qUEUwo7YpnRj4Ya2haUD9f7DK9JMeAnrQrIEjtDG5tCU7RGNiHihCfgPt5wm?= =?us-ascii?Q?GGBQlNrkjoJdk9/FhQx5opRaXlHRPoXemgTtVN0VtTKuzKSffZ9d+7Xf4+uC?= =?us-ascii?Q?I3jhh4UlC8Sux47NNmY8TcQKFfzO7BodvS6Mw1rwqu0SXKd08y3e20X39iBd?= =?us-ascii?Q?EhkU+QJbxXv5etypix3gQyxB0PYZZqfNEAi+MrACdKkDDqorKxL5+dvzbEWZ?= =?us-ascii?Q?uKVDTHjvA2ofWhk52ytnxWMZu9RfgaMwZICMNF9ZwX9QlcnsH205tHOSF81w?= =?us-ascii?Q?J9L1sJNcv7vmdGRQMdd7W6JZhokSD7WgysLmwNeSqIOqZW9ufnAyDbJu6nPH?= =?us-ascii?Q?YT0EvreXOqYEN1JM1m/YkulUp3kOJo3retGv63zz71D5FcsyScb6lit2d48a?= =?us-ascii?Q?O0seDa85cTmT5Qb4ZO7/vWXxB5u5u0+xQ/XNVAgcoYzpQaa7Sd6vd7cAPlRg?= =?us-ascii?Q?/rmiNQXsfHaeqWQ+hEtYcqeXemXDwm/b48vkJyoVGAQ83PVMlKAgTXBMrc4n?= =?us-ascii?Q?il8l0WRfFdIfXtY8dM3CkDa272bjR02k5XtE1tHgpcaGT/ZfmyP+01wlQTNi?= =?us-ascii?Q?VaXnI4oNkTWiWbzurJSwCTN4cpN01be2lAEDwHM3bZrbigkj+jQFNw9yR5My?= =?us-ascii?Q?93jO7FUOGIAido04zIok1/VP/4WoX3D0b2qTSVep7kHdG41Gmsl/UqdxNHnm?= =?us-ascii?Q?ZGQJCd1S6wOX+1g0RNqhzYkBoGPcfPR+Q5lzfzPm/OuC55Cshomfxe7c7+yB?= =?us-ascii?Q?urBbQtgJGbkMBL5aaZeRrmBrCUqfhJ1qHRFzkz8Rn/o=3D?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SJ0PR11MB5006.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: dfe8b698-d8bd-4cd7-69da-08d8ab09a3d5 X-MS-Exchange-CrossTenant-originalarrivaltime: 28 Dec 2020 08:21:56.3159 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: DhqdHQx9l5QWjvvYAOJrdRPgaFJErsf0E9sBhoNAztT0+XYqQfYqqSy2KOFl93cOzxuBhK9plhAx3z1GQL0d1g== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR11MB2728 X-OriginatorOrg: intel.com Subject: Re: [dpdk-dev] [PATCH v4 2/2] examples/vhost: refactor vhost data path X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Hi Jiayu, > -----Original Message----- > From: Hu, Jiayu > Sent: Monday, December 28, 2020 12:04 PM > To: Jiang, Cheng1 ; maxime.coquelin@redhat.com; > Xia, Chenbo > Cc: dev@dpdk.org; Yang, YvonneX > Subject: RE: [PATCH v4 2/2] examples/vhost: refactor vhost data path >=20 > Hi Cheng, >=20 > Some comments are inline. >=20 > Thanks, > Jiayu > > -----Original Message----- > > From: Jiang, Cheng1 > > Sent: Friday, December 25, 2020 4:07 PM > > To: maxime.coquelin@redhat.com; Xia, Chenbo > > Cc: dev@dpdk.org; Hu, Jiayu ; Yang, YvonneX > > ; Jiang, Cheng1 > > Subject: [PATCH v4 2/2] examples/vhost: refactor vhost data path > > > > Change the vm2vm data path to batch enqueue for better performance. > > Support latest async vhost API, refactor vhost async data path, > > replase rte_atomicNN_xxx to atomic_XXX and clean some codes. >=20 > Typo: replase -> replace I'll fix it in the next version. >=20 > > > > Signed-off-by: Cheng Jiang > > --- > > examples/vhost/main.c | 202 +++++++++++++++++++++++++++++++----- > ------ > > examples/vhost/main.h | 7 +- > > 2 files changed, 154 insertions(+), 55 deletions(-) > > > > diff --git a/examples/vhost/main.c b/examples/vhost/main.c index > > 8d8c3038b..3ea12a474 100644 > > --- a/examples/vhost/main.c > > +++ b/examples/vhost/main.c > > @@ -179,9 +179,18 @@ struct mbuf_table { struct rte_mbuf > > *m_table[MAX_PKT_BURST]; }; > > > > +struct vhost_bufftable { > > +uint32_t len; > > +uint64_t pre_tsc; > > +struct rte_mbuf *m_table[MAX_PKT_BURST]; }; > > + > > /* TX queue for each data core. */ > > struct mbuf_table lcore_tx_queue[RTE_MAX_LCORE]; > > > > +/* TX queue for each vhost device. */ >=20 > Every lcore maintains a TX buffer for every vhost device, which is to bat= ch > pkts to enqueue for higher performance. > I suggest you to update the description of vhost_txbuff above, as it is n= ot > very clear. Sure, will add some comments in the next version. >=20 > > +struct vhost_bufftable *vhost_txbuff[RTE_MAX_LCORE * > > MAX_VHOST_DEVICE]; > > + > > #define MBUF_TABLE_DRAIN_TSC((rte_get_tsc_hz() + US_PER_S - 1) \ > > / US_PER_S * BURST_TX_DRAIN_US) > > #define VLAN_HLEN 4 > > @@ -804,39 +813,114 @@ unlink_vmdq(struct vhost_dev *vdev) } } > > > > +static inline void > > +free_pkts(struct rte_mbuf **pkts, uint16_t n) { while (n--) > > +rte_pktmbuf_free(pkts[n]); } > > + > > static __rte_always_inline void > > -virtio_xmit(struct vhost_dev *dst_vdev, struct vhost_dev *src_vdev, > > +complete_async_pkts(struct vhost_dev *vdev) { struct rte_mbuf > > +*p_cpl[MAX_PKT_BURST]; uint16_t complete_count; > > + > > +complete_count =3D rte_vhost_poll_enqueue_completed(vdev->vid, > > +VIRTIO_RXQ, p_cpl, > > MAX_PKT_BURST); > > +if (complete_count) { > > +atomic_fetch_sub(&vdev->nr_async_pkts, complete_count); > > +free_pkts(p_cpl, complete_count); } } > > + > > +static __rte_always_inline void > > +sync_virtio_xmit(struct vhost_dev *dst_vdev, struct vhost_dev > > +*src_vdev, > > struct rte_mbuf *m) > > { > > uint16_t ret; > > -struct rte_mbuf *m_cpl[1]; > > > > if (builtin_net_driver) { > > ret =3D vs_enqueue_pkts(dst_vdev, VIRTIO_RXQ, &m, 1); -} else if > > (async_vhost_driver) { -ret =3D > > rte_vhost_submit_enqueue_burst(dst_vdev->vid, > > VIRTIO_RXQ, > > -&m, 1); > > - > > -if (likely(ret)) > > -dst_vdev->nr_async_pkts++; > > - > > -while (likely(dst_vdev->nr_async_pkts)) { -if > > (rte_vhost_poll_enqueue_completed(dst_vdev- > > >vid, > > -VIRTIO_RXQ, m_cpl, 1)) > > -dst_vdev->nr_async_pkts--; > > -} > > } else { > > ret =3D rte_vhost_enqueue_burst(dst_vdev->vid, VIRTIO_RXQ, &m, 1); } > > > > if (enable_stats) { > > -rte_atomic64_inc(&dst_vdev->stats.rx_total_atomic); > > -rte_atomic64_add(&dst_vdev->stats.rx_atomic, ret); > > +atomic_fetch_add(&dst_vdev->stats.rx_total_atomic, 1); > > +atomic_fetch_add(&dst_vdev->stats.rx_atomic, ret); > > src_vdev->stats.tx_total++; > > src_vdev->stats.tx +=3D ret; > > } > > } > > > > +static __rte_always_inline void > > +drain_vhost(struct vhost_dev *vdev) > > +{ > > +uint16_t ret; > > +uint64_t queue_id =3D rte_lcore_id() * MAX_VHOST_DEVICE + vdev- > > >vid; > > +uint16_t nr_xmit =3D vhost_txbuff[queue_id]->len; struct rte_mbuf **m = =3D > > +vhost_txbuff[queue_id]->m_table; >=20 > "queue_id" is not a very good name, as it's not the queue id of vhost dev= ice, > but a buffer index which holds pkts to enqueue. Sure, I will change it to buff_idx. >=20 > > + > > +if (builtin_net_driver) { > > +ret =3D vs_enqueue_pkts(vdev, VIRTIO_RXQ, m, nr_xmit); } else if > > +(async_vhost_driver) { uint32_t cpu_cpl_nr =3D 0; uint16_t enqueue_fai= l > > +=3D 0; struct rte_mbuf *m_cpu_cpl[nr_xmit]; > > + > > +complete_async_pkts(vdev); > > +ret =3D rte_vhost_submit_enqueue_burst(vdev->vid, > > VIRTIO_RXQ, > > +m, nr_xmit, m_cpu_cpl, &cpu_cpl_nr); > > +atomic_fetch_add(&vdev->nr_async_pkts, ret - cpu_cpl_nr); > > + > > +if (cpu_cpl_nr) > > +free_pkts(m_cpu_cpl, cpu_cpl_nr); > > + > > +enqueue_fail =3D nr_xmit - ret; > > +if (enqueue_fail) > > +free_pkts(&m[ret], nr_xmit - ret); > > +} else { > > +ret =3D rte_vhost_enqueue_burst(vdev->vid, VIRTIO_RXQ, m, nr_xmit); } > > + > > +if (enable_stats) { > > +atomic_fetch_add(&vdev->stats.rx_total_atomic, nr_xmit); > > +atomic_fetch_add(&vdev->stats.rx_atomic, ret); } > > + > > +if (!async_vhost_driver) > > +free_pkts(m, nr_xmit); > > +} > > + > > +static __rte_always_inline void > > +drain_vhost_table(void) > > +{ > > +const uint16_t lcore_id =3D rte_lcore_id(); struct vhost_bufftable > > +*vhost_txq; struct vhost_dev *vdev; uint64_t cur_tsc; > > + > > +TAILQ_FOREACH(vdev, &lcore_info[lcore_id].vdev_list, > > lcore_vdev_entry) { >=20 > A lcore may have pkts to enqueue for any vhost device, as it's decided by > mac address. So drain_vhost_table() shouldn't just process vhost ports > allocated to its lcore, but all vhost ports that have un-sent packets. Sure, I will fix it in the next version. >=20 > > +if (!vdev->remove) { > > +vhost_txq =3D vhost_txbuff[lcore_id * > > MAX_VHOST_DEVICE > > ++ vdev->vid]; > > +cur_tsc =3D rte_rdtsc(); > > + > > +if (unlikely(cur_tsc - vhost_txq->pre_tsc > > +> MBUF_TABLE_DRAIN_TSC)) { > > +RTE_LOG_DP(DEBUG, VHOST_DATA, > > +"Vhost tX queue drained after >=20 > "tX" -> "TX" I will fix it in the next version. >=20 > > timeout with burst size %u\n", > > +vhost_txq->len); > > +drain_vhost(vdev); > > +vhost_txq->len =3D 0; > > +vhost_txq->pre_tsc =3D cur_tsc; > > +} > > +} > > +} > > +} > > + > > /* > > * Check if the packet destination MAC address is for a local device. > > If so then put > > * the packet on that devices RX queue. If not then return. > > @@ -846,7 +930,8 @@ virtio_tx_local(struct vhost_dev *vdev, struct > > rte_mbuf *m) { struct rte_ether_hdr *pkt_hdr; struct vhost_dev > > *dst_vdev; > > - > > +struct vhost_bufftable *vhost_txq; > > +const uint16_t lcore_id =3D rte_lcore_id(); >=20 > Why use "const"? No particular reason, maybe I just forget to delete it. I will fix it in th= e next version. >=20 > > pkt_hdr =3D rte_pktmbuf_mtod(m, struct rte_ether_hdr *); > > > > dst_vdev =3D find_vhost_dev(&pkt_hdr->d_addr); @@ -869,7 +954,19 @@ > > virtio_tx_local(struct vhost_dev *vdev, struct rte_mbuf *m) return 0; > > } > > > > -virtio_xmit(dst_vdev, vdev, m); > > +vhost_txq =3D vhost_txbuff[lcore_id * MAX_VHOST_DEVICE + dst_vdev- > > >vid]; > > +vhost_txq->m_table[vhost_txq->len++] =3D m; > > + > > +if (enable_stats) { > > +vdev->stats.tx_total++; > > +vdev->stats.tx++; > > +} > > + > > +if (unlikely(vhost_txq->len =3D=3D MAX_PKT_BURST)) { > > +drain_vhost(dst_vdev); vhost_txq->len =3D 0; vhost_txq->pre_tsc =3D > > +rte_rdtsc(); } > > return 0; > > } > > > > @@ -940,13 +1037,6 @@ static void virtio_tx_offload(struct rte_mbuf > > *m) tcp_hdr->cksum =3D get_psd_sum(l3_hdr, m->ol_flags); } > > > > -static inline void > > -free_pkts(struct rte_mbuf **pkts, uint16_t n) -{ -while (n--) > > -rte_pktmbuf_free(pkts[n]); -} > > - > > static __rte_always_inline void > > do_drain_mbuf_table(struct mbuf_table *tx_q) { @@ -979,16 +1069,14 > > @@ virtio_tx_route(struct vhost_dev *vdev, struct rte_mbuf *m, > > uint16_t vlan_tag) > > > > TAILQ_FOREACH(vdev2, &vhost_dev_list, global_vdev_entry) { if (vdev2 > > !=3D vdev) -virtio_xmit(vdev2, vdev, m); > > +sync_virtio_xmit(vdev2, vdev, m); > > } > > goto queue2nic; > > } > > > > /*check if destination is local VM*/ > > -if ((vm2vm_mode =3D=3D VM2VM_SOFTWARE) && (virtio_tx_local(vdev, > > m) =3D=3D 0)) { > > -rte_pktmbuf_free(m); > > +if ((vm2vm_mode =3D=3D VM2VM_SOFTWARE) && (virtio_tx_local(vdev, > > m) =3D=3D 0)) > > return; > > -} > > > > if (unlikely(vm2vm_mode =3D=3D VM2VM_HARDWARE)) { if > > (unlikely(find_local_dest(vdev, m, &offset, @@ -1073,19 +1161,6 @@ > > drain_mbuf_table(struct mbuf_table *tx_q) } } > > > > -static __rte_always_inline void > > -complete_async_pkts(struct vhost_dev *vdev, uint16_t qid) -{ -struct > > rte_mbuf *p_cpl[MAX_PKT_BURST]; -uint16_t complete_count; > > - > > -complete_count =3D rte_vhost_poll_enqueue_completed(vdev->vid, > > -qid, p_cpl, MAX_PKT_BURST); > > -vdev->nr_async_pkts -=3D complete_count; > > -if (complete_count) > > -free_pkts(p_cpl, complete_count); > > -} > > - > > static __rte_always_inline void > > drain_eth_rx(struct vhost_dev *vdev) > > { > > @@ -1095,9 +1170,6 @@ drain_eth_rx(struct vhost_dev *vdev) rx_count > =3D > > rte_eth_rx_burst(ports[0], vdev->vmdq_rx_q, > > pkts, MAX_PKT_BURST); > > > > -while (likely(vdev->nr_async_pkts)) > > -complete_async_pkts(vdev, VIRTIO_RXQ); > > - > > if (!rx_count) > > return; > > > > @@ -1123,17 +1195,31 @@ drain_eth_rx(struct vhost_dev *vdev) > > enqueue_count =3D vs_enqueue_pkts(vdev, VIRTIO_RXQ, pkts, rx_count); = } > > else if (async_vhost_driver) { > > +uint32_t cpu_cpl_nr =3D 0; > > +uint16_t enqueue_fail =3D 0; > > +struct rte_mbuf *m_cpu_cpl[MAX_PKT_BURST]; > > + > > +complete_async_pkts(vdev); > > enqueue_count =3D rte_vhost_submit_enqueue_burst(vdev- > > >vid, > > -VIRTIO_RXQ, pkts, rx_count); > > -vdev->nr_async_pkts +=3D enqueue_count; > > +VIRTIO_RXQ, pkts, rx_count, > > +m_cpu_cpl, &cpu_cpl_nr); > > +atomic_fetch_add(&vdev->nr_async_pkts, > > +enqueue_count - cpu_cpl_nr); > > +if (cpu_cpl_nr) > > +free_pkts(m_cpu_cpl, cpu_cpl_nr); > > + > > +enqueue_fail =3D rx_count - enqueue_count; if (enqueue_fail) > > +free_pkts(&pkts[enqueue_count], enqueue_fail); > > + > > } else { > > enqueue_count =3D rte_vhost_enqueue_burst(vdev->vid, > > VIRTIO_RXQ, > > pkts, rx_count); > > } > > > > if (enable_stats) { > > -rte_atomic64_add(&vdev->stats.rx_total_atomic, rx_count); > > -rte_atomic64_add(&vdev->stats.rx_atomic, enqueue_count); > > +atomic_fetch_add(&vdev->stats.rx_total_atomic, rx_count); > > +atomic_fetch_add(&vdev->stats.rx_atomic, enqueue_count); > > } > > > > if (!async_vhost_driver) > > @@ -1202,7 +1288,7 @@ switch_worker(void *arg __rte_unused) > > > > while(1) { > > drain_mbuf_table(tx_q); > > - > > +drain_vhost_table(); > > /* > > * Inform the configuration core that we have exited the > > * linked list and that no devices are in use if requested. > > @@ -1298,6 +1384,7 @@ static int > > new_device(int vid) > > { > > int lcore, core_add =3D 0; > > +uint16_t i; > > uint32_t device_num_min =3D num_devices; struct vhost_dev *vdev; vde= v > > =3D rte_zmalloc("vhost device", sizeof(*vdev), RTE_CACHE_LINE_SIZE); @@ > > -1309,6 +1396,13 @@ new_device(int vid) } vdev->vid =3D vid; > > > > +for (i =3D 0; i < RTE_MAX_LCORE; i++) { vhost_txbuff[i * > > +MAX_VHOST_DEVICE + vid] =3D rte_zmalloc("vhost bufftable", > > +sizeof(struct vhost_bufftable), RTE_CACHE_LINE_SIZE); >=20 > Rte_zmalloc() may fail. Need to handle failure. Sure, I will add failure handler in the next version. Thanks. Cheng >=20 > > +} > > + > > if (builtin_net_driver) > > vs_vhost_net_setup(vdev); > > > > @@ -1343,12 +1437,15 @@ new_device(int vid) if (async_vhost_driver) { > > struct rte_vhost_async_features f; struct rte_vhost_async_channel_ops > > channel_ops; > > + > > if (strncmp(dma_type, "ioat", 4) =3D=3D 0) { channel_ops.transfer_dat= a =3D > > ioat_transfer_data_cb; channel_ops.check_completed_copies =3D > > ioat_check_completed_copies_cb; > > + > > f.async_inorder =3D 1; > > f.async_threshold =3D 256; > > + > > return rte_vhost_async_channel_register(vid, > > VIRTIO_RXQ, > > f.intval, &channel_ops); > > } > > @@ -1392,8 +1489,8 @@ print_stats(__rte_unused void *arg) > > tx =3D vdev->stats.tx; > > tx_dropped =3D tx_total - tx; > > > > -rx_total =3D rte_atomic64_read(&vdev- > > >stats.rx_total_atomic); > > -rx =3D rte_atomic64_read(&vdev->stats.rx_atomic); > > +rx_total =3D atomic_load(&vdev- > > >stats.rx_total_atomic); > > +rx =3D atomic_load(&vdev->stats.rx_atomic); > > rx_dropped =3D rx_total - rx; > > > > printf("Statistics for device %d\n" > > @@ -1592,6 +1689,7 @@ main(int argc, char *argv[]) > > /* Register vhost user driver to handle vhost messages. */ for (i =3D > > 0; i < nb_sockets; i++) { char *file =3D socket_files + i * PATH_MAX; > > + > > if (async_vhost_driver) > > flags =3D flags | RTE_VHOST_USER_ASYNC_COPY; > > > > diff --git a/examples/vhost/main.h b/examples/vhost/main.h index > > 4317b6ae8..6aa798a3e 100644 > > --- a/examples/vhost/main.h > > +++ b/examples/vhost/main.h > > @@ -8,6 +8,7 @@ > > #include > > > > #include > > +#include > > > > /* Macros for printing using RTE_LOG */ #define > > RTE_LOGTYPE_VHOST_CONFIG RTE_LOGTYPE_USER1 @@ -21,8 +22,8 @@ > enum > > {VIRTIO_RXQ, VIRTIO_TXQ, VIRTIO_QNUM}; struct device_statistics { > > uint64_ttx; uint64_ttx_total; -rte_atomic64_trx_atomic; > > -rte_atomic64_trx_total_atomic; > > +atomic_int_least64_trx_atomic; > > +atomic_int_least64_trx_total_atomic; > > }; > > > > struct vhost_queue { > > @@ -51,7 +52,7 @@ struct vhost_dev { > > uint64_t features; > > size_t hdr_len; > > uint16_t nr_vrings; > > -uint16_t nr_async_pkts; > > +atomic_int_least16_t nr_async_pkts; > > struct rte_vhost_memory *mem; > > struct device_statistics stats; > > TAILQ_ENTRY(vhost_dev) global_vdev_entry; > > -- > > 2.29.2 >=20