From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id D3018A0548; Thu, 8 Sep 2022 07:27:09 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 7B3FB40143; Thu, 8 Sep 2022 07:27:09 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0a-0016f401.pphosted.com [67.231.148.174]) by mails.dpdk.org (Postfix) with ESMTP id 2B7B7400D6 for ; Thu, 8 Sep 2022 07:27:07 +0200 (CEST) Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 2881nONp029959; Wed, 7 Sep 2022 22:27:06 -0700 Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2169.outbound.protection.outlook.com [104.47.56.169]) by mx0a-0016f401.pphosted.com (PPS) with ESMTPS id 3jen9ww6ht-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 07 Sep 2022 22:27:05 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=j1lj6maAHVIF9YUc2XpC4uiS9zl9AtYNH1YJw40hduXip3yYTDrnXMCzrSD689ADP+cQ4CTWpwMPmlKTsp4k3vIGmhEqgLxhKiutP2nfrUt24Rz6amPUF5N3GyQ911nrBUbRZ58eaWr4iOTc7A91WcJVCIbxVubhT/1HRbJSh+VpKRqPPIb0U5t5x3NVO9lDUFWSyI7RClxpHu9M01XWQEZsoaInxrJvTYnvIoGAQS9miV9iTmRBBfkuweSyVKbwXd9L3FmzV+U08HpJLQyBIBLCYydSbqgzBImhx7iU+Q63AaBwbqr+2AN+NVqpZkaj+BMO/FzvCowO9kYqkqli+Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=iwpjXO049uGMxksF6Dl41/fcyeyWa81bHqH68J3NWNY=; b=L7OE4XpeVqKgdgpCp5gopQFE2MBNHSVetbYOWh5rDsDwqRKTwwhw2Idi2z2qhCeJAKC213ADIaTYy2znv1aog/rjmNhtpAR2Uxmsk6o0JeG4Cja3yUwLsbUN69R2eCP6KVyTOLTBq6xmIhIY4TPJfrtQSfjAU/p7Lp9g+rx7zQn8ILlWm3XZoDLtZwoDYxRbZh0SvHYkUttSDPWVQXHj/XmfFt/RRIYRm/H9WxOqUfBGXPvJ7P0+CgffKMMnxWJA3QWXpgx10i18YRys9RzzgYG7MJLC5dUErTbs1UL0TDyg22DMPMz2Q4C6jkyJYiG2bYYNWaHU/50fozE7dRiCqQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=marvell.com; dmarc=pass action=none header.from=marvell.com; dkim=pass header.d=marvell.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.onmicrosoft.com; s=selector1-marvell-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=iwpjXO049uGMxksF6Dl41/fcyeyWa81bHqH68J3NWNY=; b=beYvTTnl58tByP4Fe6qCaDJK33fwoAaC6eTEYBNhhigG0Xq3Q7qmhoQLdOpQQYzLZa+kEMa1F1eRoYer5dCOnNfds5NWuCRXWeb5qZwFsIt9KvFadAEM/jz9nFKzP7G1c44WUN2cqpCqQo/bBM2KYACUNK/Yp+NQcCOPPHmmBUo= Received: from PH0PR18MB4086.namprd18.prod.outlook.com (2603:10b6:510:3::9) by MN2PR18MB3687.namprd18.prod.outlook.com (2603:10b6:208:261::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5612.14; Thu, 8 Sep 2022 05:27:03 +0000 Received: from PH0PR18MB4086.namprd18.prod.outlook.com ([fe80::3c67:e351:cf7a:c1fa]) by PH0PR18MB4086.namprd18.prod.outlook.com ([fe80::3c67:e351:cf7a:c1fa%3]) with mapi id 15.20.5612.016; Thu, 8 Sep 2022 05:27:03 +0000 From: Pavan Nikhilesh Bhagavatula To: Zhirun Yan , "dev@dpdk.org" , Jerin Jacob Kollanukkaran , Kiran Kumar Kokkilagadda CC: "cunming.liang@intel.com" , "haiyue.wang@intel.com" Subject: RE: [EXT] [RFC, v1 4/6] graph: enhance graph walk by cross-core dispatch Thread-Topic: [EXT] [RFC, v1 4/6] graph: enhance graph walk by cross-core dispatch Thread-Index: AQHYwyg5OJVQKsR5s0WeGmq04wtmEK3U/xjA Date: Thu, 8 Sep 2022 05:27:03 +0000 Message-ID: References: <20220908020959.1675953-1-zhirun.yan@intel.com> <20220908020959.1675953-5-zhirun.yan@intel.com> In-Reply-To: <20220908020959.1675953-5-zhirun.yan@intel.com> Accept-Language: en-IN, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-traffictypediagnostic: PH0PR18MB4086:EE_|MN2PR18MB3687:EE_ x-ms-office365-filtering-correlation-id: e0d7c31c-b19a-4a2a-3b16-08da915ac317 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: tdxS4fbv8K3GL/JFpx8sScSoNRnlHvkw0cR12uSjUcI2GN51eoUYeghw5w1bcySIOQgxbnjJqdHfehqaaonkNeSagr3VA4HfVtOMNjRf/lQiE0hv5/h9LXPnY57eG7vkG3MAQHqiKyREQFDQ0ZovKJ/k4u943VHXMi/ivXqYX4lj2w8HVxWRKM4UftQpoUB23UkOlLeCFAEX1EWsHDd48ydvfvBz1/S+2FeaYvA13Q0gs42ULLIikBbydw1fh+reLAQJIuvQvq0iwudryJvauDFol6jv988zmDERzokWHO07NQJir5YC6KY2CYhBbaag7DkMSqte4rnAI5cqEsGth5VlRlQYPVeuErpReq1aeEiqqRDRUPBIoC42ofrU3Aeix5YCCos7O+AiCzQOV1CQSqrlPrKr8dj2AM3sQ6HgtuHgJpymWT0YGOifPWEjXCTTTekPtSvsYRL6KD452sEYeCeWMa2dyYLqd11U/X0DsAGVzPHgPM8JxG68VXq0tO+TnQuD0wl8ADXryRd9iVGRg+sTAILlYoNG02x83XXroR82NSYmB0k1d8V+7jo/zE12CbF3j/v9BNR/RTTqsrMYQqlKKEXTUPxeNIoOtaiAzzfFWhXCdgkNl22H7uxjaYdvZHACCEaEzOgsQcv0aIImen6uhctdVo/1aQCGy7Li1RwwAxzdZbU2bUwLDRYV1Bg+/0t2xvINIItYCk+YPhh2IAjuTy5Fmq/XG75BICDZiJKqzP9YcLC6gYlCvvm1j6fCGGJPNWolbIQislOqDnhS3Q== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:PH0PR18MB4086.namprd18.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230016)(4636009)(39860400002)(376002)(396003)(136003)(346002)(366004)(41300700001)(9686003)(7696005)(6506007)(478600001)(26005)(71200400001)(8936002)(2906002)(52536014)(5660300002)(55016003)(6636002)(110136005)(186003)(54906003)(316002)(66946007)(4326008)(64756008)(66556008)(8676002)(66476007)(76116006)(66446008)(38100700002)(122000001)(38070700005)(33656002)(86362001); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?WhqqjZWDXDseuE3EMFvHtDjHv11eTB/Iw78WfqSN9Mo5oQ7kNx87/MRtPJg0?= =?us-ascii?Q?3IBHKDssEiu9Y2ZjnrMOrde46jIF0v9N4ZeX134YM3CYJGElxvoINApfWxyV?= =?us-ascii?Q?ERIyRKuKcNmB+49V056aVVO7XNzXMrSWyQ4RJbeIR38Q70cL6YWvCRv5wWOG?= =?us-ascii?Q?o+7GogWMOK6jDP637WO8g0Tqu48dDNB4uaQJPvI2d9IyP+b4tJRxSdjDOJ7C?= =?us-ascii?Q?/DpeQKNbYsZdkWn0EDuXta/mcN9XFfmiHmdS8YCbLgd/QUwihiaCGt1uNcFW?= =?us-ascii?Q?AR3iVOze+/o4SVGQ5jbuBAq6efvO7YhSFvPNLLAEJlIuHbhtVd+TxgO4SItg?= =?us-ascii?Q?i/50oJM5dd1QKI5jUL2gOTwTVhboSAtxSyeSdzd5K8YLir7Ym43Jujv/fKud?= =?us-ascii?Q?BYxAZ2X276HFeWE66nohouL2hpKLyGb/fvKfnD6KY9jnzpg3HC7S/FLzhGxv?= =?us-ascii?Q?lg0TwT9um7hch1LnDcJ3QGySoTnZ/UvMVPSSk0uXebRiigZFI859Suk7MeMC?= =?us-ascii?Q?blIgyQ/b9OUz+G8fCYHJXfAvG884m/lw4J7WFW6HHN4wu2wWeYLgSbs39uLS?= =?us-ascii?Q?YWUOcCs9KRdF0p8l1Y1wnEpdzoP/jvFojbM006saDDyRiuoMdusEm2J4JJCn?= =?us-ascii?Q?tawC8J0mShgNVdRX2XzxBjR2/2W7qsWBlpm4RV/ErDVIp+XjnWrtexCp4bHv?= =?us-ascii?Q?5pmHppPhW6+q1m5MM5NiyeM2uT2sohsWUKaq9IZDvoYmRj4ahHBl8L1OSJ33?= =?us-ascii?Q?NeVaOydFXEbyED6d+cEuoh3X4zOT/GLvl4IjBoiXUlmbW+iYY/Z94BIJs+D+?= =?us-ascii?Q?9mbaKiZNnjDLhgcrf4RO+dCNxCd+U8mo731+KmR9xTo0o/mIkKOvvkVGAw5W?= =?us-ascii?Q?foBR/fyk/jwFhmP1jSJ4ge0EsIlcGPB/yAWz3xx2gk8kRPcFfTglxaKmU+qp?= =?us-ascii?Q?PDlICZrH6x2j08i00O0u4SOeCp6G5wQglT+NCgwJsrQr16sFPxPcuJUzcwbw?= =?us-ascii?Q?BMFW72QetESo4sn5x1yAuIy3XMIMTyeYAUcrH2/mPeUHGSlg+04EUlzEkHks?= =?us-ascii?Q?c3AW7sO5luEFlE8igEaj/cmRbHtPmqSxFoK4me0LyIIHgg0NTdwDYiNWCvwD?= =?us-ascii?Q?z7xFXC0/Qq2usGw5+y7Mrv1DZqR2xvpLC8rvAClfoJJUAMiv0mFL6elnCC3a?= =?us-ascii?Q?sCoxWVJb1sHIupNfEwT12zuPYxh7L4XySLpb8JKY3rXeYkkHKV2BWPGrceHn?= =?us-ascii?Q?aLVM31KHE1kmBKphXHXzuGGbhsgFF0pdoxmi/A4idpwcBxoU0/Ap0ajPGy1T?= =?us-ascii?Q?i/3SpQiYDaKRvfGBffIBz46XPmjgG7Lo9s4ijUCeMcvKCWAl3P4yEmTakMwy?= =?us-ascii?Q?/wvNeU4LBecpoBC1VPxasXc/Bxg1hCqX07b3YTqiiVjWCgLYa0BsXUStoHVc?= =?us-ascii?Q?ARM+AZXEvdnjSEWqsjqz5Lj2wNv83zYqb7AKoCwwHG8v8m/5ZC0g2gHRPo9q?= =?us-ascii?Q?88eYqa0ZalcIAx+ueWPFZD3L3ugzMwIyuBxh6ZsdWuUSBmjNtQiqkkiLTz4d?= =?us-ascii?Q?p0nc19+f9FvClTv3JDNbT5ilD95MZORE3s2T8WvY?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: marvell.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: PH0PR18MB4086.namprd18.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: e0d7c31c-b19a-4a2a-3b16-08da915ac317 X-MS-Exchange-CrossTenant-originalarrivaltime: 08 Sep 2022 05:27:03.2041 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 70e1fb47-1155-421d-87fc-2e58f638b6e0 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: ig4EruqQxtdKhynMmCx8BXJSezwMr9LZjnnNItZyko1QMUSuKYuky5cwstxWQKi0C/1FvvgFto705W9fGdVJ1NgtxS2n9yDmIdJVTJ417TM= X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR18MB3687 X-Proofpoint-ORIG-GUID: Z1KOh-oRwbBTQ-77QzfcfKwwThAilYII X-Proofpoint-GUID: Z1KOh-oRwbBTQ-77QzfcfKwwThAilYII X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.528,FMLib:17.11.122.1 definitions=2022-09-08_04,2022-09-07_02,2022-06-22_01 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org > This patch enhance the task scheduler mechanism to enable dispatching > tasks to another worker cores. Currently, there is only a local work > queue for one graph to walk. We introduce a scheduler worker queue in > each worker core for dispatching tasks. It will perform the walk on > scheduler work queue first, then handle the local work queue. >=20 > Signed-off-by: Haiyue Wang > Signed-off-by: Cunming Liang > Signed-off-by: Zhirun Yan > --- > lib/graph/graph.c | 6 ++++++ > lib/graph/rte_graph_worker.h | 11 +++++++++++ > 2 files changed, 17 insertions(+) >=20 > diff --git a/lib/graph/graph.c b/lib/graph/graph.c > index b4eb18175a..49ea2b3fbb 100644 > --- a/lib/graph/graph.c > +++ b/lib/graph/graph.c > @@ -368,6 +368,8 @@ rte_graph_destroy(rte_graph_t id) > while (graph !=3D NULL) { > tmp =3D STAILQ_NEXT(graph, next); > if (graph->id =3D=3D id) { > + /* Destroy the schedule work queue if has */ > + graph_sched_wq_destroy(graph); > /* Call fini() of the all the nodes in the graph */ > graph_node_fini(graph); > /* Destroy graph fast path memory */ > @@ -470,6 +472,10 @@ graph_clone(struct graph *parent_graph, const char > *name, > if (graph_node_init(graph)) > goto graph_mem_destroy; >=20 > + /* Create the graph schedule work queue */ > + if (graph_sched_wq_create(graph, parent_graph)) > + goto graph_mem_destroy; > + > /* All good, Lets add the graph to the list */ > graph_id++; > STAILQ_INSERT_TAIL(&graph_list, graph, next); > diff --git a/lib/graph/rte_graph_worker.h b/lib/graph/rte_graph_worker.h > index faf3f31ddc..e98697d880 100644 > --- a/lib/graph/rte_graph_worker.h > +++ b/lib/graph/rte_graph_worker.h > @@ -177,6 +177,7 @@ static inline void > rte_graph_walk(struct rte_graph *graph) > { > const rte_graph_off_t *cir_start =3D graph->cir_start; > + const unsigned int lcore_id =3D graph->lcore_id; > const rte_node_t mask =3D graph->cir_mask; > uint32_t head =3D graph->head; > struct rte_node *node; > @@ -184,6 +185,9 @@ rte_graph_walk(struct rte_graph *graph) > uint16_t rc; > void **objs; >=20 > + if (graph->wq !=3D NULL) > + __rte_graph_sched_wq_process(graph); > + We should introduce a flags field in rte_graph_param which can be used by the application to define whether a graph should support multi-core dispatch. Then we can make `__rte_graph_sched_wq_process` as node 0 during graph=20 creation so that it will be always called at the start of graph processing = followed=20 by calling rest of the nodes. This will remove unnecessary branches in fastpath. > /* > * Walk on the source node(s) ((cir_start - head) -> cir_start) and > then > * on the pending streams (cir_start -> (cir_start + mask) -> cir_start= ) > @@ -205,6 +209,12 @@ rte_graph_walk(struct rte_graph *graph) > objs =3D node->objs; > rte_prefetch0(objs); >=20 > + /* Schedule the node until all task/objs are done */ > + if (node->lcore_id !=3D RTE_MAX_LCORE && (int32_t)head > 0 > && > + lcore_id !=3D node->lcore_id && graph->rq !=3D NULL && > + __rte_graph_sched_node_enqueue(node, graph->rq)) > + goto next; > + > if (rte_graph_has_stats_feature()) { > start =3D rte_rdtsc(); > rc =3D node->process(graph, node, objs, node->idx); > @@ -215,6 +225,7 @@ rte_graph_walk(struct rte_graph *graph) > node->process(graph, node, objs, node->idx); > } > node->idx =3D 0; > + next: > head =3D likely((int32_t)head > 0) ? head & mask : head; > } > graph->tail =3D 0; > -- > 2.25.1