From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <dev-bounces@dpdk.org>
Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124])
by inbox.dpdk.org (Postfix) with ESMTP id 449ADA0032;
Thu, 15 Sep 2022 03:52:29 +0200 (CEST)
Received: from [217.70.189.124] (localhost [127.0.0.1])
by mails.dpdk.org (Postfix) with ESMTP id C84B44021E;
Thu, 15 Sep 2022 03:52:28 +0200 (CEST)
Received: from mga05.intel.com (mga05.intel.com [192.55.52.43])
by mails.dpdk.org (Postfix) with ESMTP id 7C96F4021D
for <dev@dpdk.org>; Thu, 15 Sep 2022 03:52:25 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple;
d=intel.com; i=@intel.com; q=dns/txt; s=Intel;
t=1663206746; x=1694742746;
h=from:to:cc:subject:date:message-id:references:
in-reply-to:content-transfer-encoding:mime-version;
bh=H8nlUkxP+as6yQx3Yk3sMgdWWowffMTXJkeUejhivqc=;
b=nM9cOs25j+iHFn/rqQIJrQ4ZSDDlVy/ZI77JjxE0WHXf4WA7h22aRoOT
tcIwABWKPt2CYZZHmFZ61kOXaLI7t25/8ZGaBc9dCNhAzJK4Y5ARw5nxZ
NPPU1Q0DCZ+SrJURZs/eFn1feSUudEjkQufNjdFzjoF+l4IcGMmzero3/
7sY76uuj19bhpsDbCX5fe0S5biicvIyBd3uc3ZJbaFbS6Rs6r9bVRX2fO
M7hmi4G4jLAg4I+Ibo+r3p68hsQOS5KfeUohaIcLQd+i1sN+0YvFy1Xdz
CmMYSZIF0d9dPlg/YTptlw5p1I9WS2vSv+Q2T7Wp2rFmzIIvid9IIpqAE Q==;
X-IronPort-AV: E=McAfee;i="6500,9779,10470"; a="384880458"
X-IronPort-AV: E=Sophos;i="5.93,316,1654585200"; d="scan'208";a="384880458"
Received: from orsmga003.jf.intel.com ([10.7.209.27])
by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384;
14 Sep 2022 18:52:16 -0700
X-ExtLoop1: 1
X-IronPort-AV: E=Sophos;i="5.93,316,1654585200"; d="scan'208";a="568233974"
Received: from fmsmsx602.amr.corp.intel.com ([10.18.126.82])
by orsmga003.jf.intel.com with ESMTP; 14 Sep 2022 18:52:16 -0700
Received: from fmsmsx612.amr.corp.intel.com (10.18.126.92) by
fmsmsx602.amr.corp.intel.com (10.18.126.82) with Microsoft SMTP Server
(version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id
15.1.2375.31; Wed, 14 Sep 2022 18:52:16 -0700
Received: from fmsmsx609.amr.corp.intel.com (10.18.126.89) by
fmsmsx612.amr.corp.intel.com (10.18.126.92) with Microsoft SMTP Server
(version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id
15.1.2375.31; Wed, 14 Sep 2022 18:52:15 -0700
Received: from fmsedg601.ED.cps.intel.com (10.1.192.135) by
fmsmsx609.amr.corp.intel.com (10.18.126.89) with Microsoft SMTP Server
(version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id
15.1.2375.31 via Frontend Transport; Wed, 14 Sep 2022 18:52:15 -0700
Received: from NAM11-BN8-obe.outbound.protection.outlook.com (104.47.58.169)
by edgegateway.intel.com (192.55.55.70) with Microsoft SMTP Server
(version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id
15.1.2375.31; Wed, 14 Sep 2022 18:52:15 -0700
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
b=LrX0PrXRxhs96PgTHbqyC+jYVhxbBE26RnAFwmQYQmOttcf83n7iq/ObZ9B01Gmz1208qnQTYX1WC3LaFlMZxVLEsgvYqBt4GWI0RGYufy3E+c65MJ82FVK0dsZ73yUm6TjnjrpXpWVUL4zM7XXt+Ish4YJu0/a7RjKnrWyHKgc4N8bgaxD5A4yOSwHKZ4jlvMPsF8nD6Lm37nQcX+sqbMXuEZ5T37lbydeO/ZtLmiXrnoYCArzjR/KhmcDC745osYKuOlkf6j2bUMptDnnxXfvgwRU1U66OMfKL0+siWMsSBfGs4cy8bzNGaHgo9W9w2hl62BloAJK9a0zJbqC7MQ==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
s=arcselector9901;
h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
bh=PMzRrtG3o61SGDN0v7btfrewKVs/x/w9vXVZrZWOgbo=;
b=ApCJLdGE8s4yrAsx56IY7GBAVncqrXtoKEsNU1atlpjeiUOB7FdcjWj1n1hZmbhBfrzr+VcUWBR1B7qy0oKt/O7u+znN7fQCNUbpDeZHELIjS+XbyZwJjioqbK3RKXnCBaylSeHWtUWrhseN9KYVKszJ1PEVVeTHcVqNZ6HfP814XfEaSSU2mj4OuWsOL0CrhqfwIFXLhFaqb4NnJllZxbyJ1abOVmWYM2KtDk8z1alcXbZeiLfdmp821TNr5nKjShctZFCXTGnicajH6MLmjpuRDRx91/kCO41+JS9AkrYlk2719K5mUNs0DeFUxmknOIaf9wbfT1x1l+07YBHMxw==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com;
dkim=pass header.d=intel.com; arc=none
Received: from SJ0PR11MB5022.namprd11.prod.outlook.com (2603:10b6:a03:2d7::17)
by DM6PR11MB4515.namprd11.prod.outlook.com (2603:10b6:5:2a9::9) with
Microsoft SMTP Server (version=TLS1_2,
cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5632.15; Thu, 15 Sep
2022 01:52:13 +0000
Received: from SJ0PR11MB5022.namprd11.prod.outlook.com
([fe80::e128:9d7c:7aa9:f366]) by SJ0PR11MB5022.namprd11.prod.outlook.com
([fe80::e128:9d7c:7aa9:f366%4]) with mapi id 15.20.5612.022; Thu, 15 Sep 2022
01:52:13 +0000
From: "Yan, Zhirun" <zhirun.yan@intel.com>
To: Pavan Nikhilesh Bhagavatula <pbhagavatula@marvell.com>, "dev@dpdk.org"
<dev@dpdk.org>, Jerin Jacob Kollanukkaran <jerinj@marvell.com>, "Kiran Kumar
Kokkilagadda" <kirankumark@marvell.com>
CC: "Liang, Cunming" <cunming.liang@intel.com>, "Wang, Haiyue"
<haiyue.wang@intel.com>
Subject: RE: [EXT] [RFC, v1 4/6] graph: enhance graph walk by cross-core
dispatch
Thread-Topic: [EXT] [RFC, v1 4/6] graph: enhance graph walk by cross-core
dispatch
Thread-Index: AQHYwyg1or/PAbheDEad8fb0FDXI5a3VAPCAgAmtRyA=
Date: Thu, 15 Sep 2022 01:52:13 +0000
Message-ID: <SJ0PR11MB50227BB3775772A6EF1F09E685499@SJ0PR11MB5022.namprd11.prod.outlook.com>
References: <20220908020959.1675953-1-zhirun.yan@intel.com>
<20220908020959.1675953-5-zhirun.yan@intel.com>
<PH0PR18MB4086BAED6277AA628470E819DE409@PH0PR18MB4086.namprd18.prod.outlook.com>
In-Reply-To: <PH0PR18MB4086BAED6277AA628470E819DE409@PH0PR18MB4086.namprd18.prod.outlook.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
dlp-product: dlpe-windows
dlp-version: 11.6.500.17
dlp-reaction: no-action
authentication-results: dkim=none (message not signed)
header.d=none;dmarc=none action=none header.from=intel.com;
x-ms-publictraffictype: Email
x-ms-traffictypediagnostic: SJ0PR11MB5022:EE_|DM6PR11MB4515:EE_
x-ms-office365-filtering-correlation-id: 26a769ff-a369-441e-6188-08da96bce8ea
x-ms-exchange-senderadcheck: 1
x-ms-exchange-antispam-relay: 0
x-microsoft-antispam: BCL:0;
x-microsoft-antispam-message-info: TLVHG5vMe/lC4QUetrEPKnumD3KFocxBsZTEGOQAw0Q9zjmqsnfBRLRL3MB8+YzlwUUzc2Gl2WYCqiTVR2PMnWxfiSe7ej5ZvcBsKEQrYGMgpa0h1elwVnrVUbZJnm34Tyqu+Q1EAsZOm16D/X3JixSuJ6UuJwzx0aECL3t9KyVn+8/NGma8WXAh0S/YURHQpxpHtQBdL71eUBJeLtT1AfMhmOH3jAH13XYgAQG+bNgO5wFnoASWaC5BVZyDaXDWzaeIKHFUu6R8btIhg9ZCZ8T7UG/2pRPDBpBMEbYyTsYbBj0HRC08Se/LBCBRtPrFiARfPJtOtd9PzmXRoZAbfiJU06XTfn8HcpDCi7FyniB/bqweU+EG2S5LzFw1yRMTakyiNrEJFVB5QOSNAG6UjEdOvihSNRWcCt51PaB113CCPqrEX4F53faSq4OR/udiZ494DIvhRfvn8nWDeq9gBJj+NLC46AiJjZ1cQfmC/pUrAxAwbifb7Bj+B9EWqJzNxjRcyJ9c28pgiZ+It5FqX9GeWG8EPHpvBnbofMmrJr4Flrgru3cHpipLbHaFHpI8d8PCDC2yjvgg+2ya4ECUiNUA28hHJHhCR0sSFUbODfYTrTgHa5wgu8s1ay6FJBWZTx4AvvPo2wuYs3VJa3ovL+W61qR6/SWy0UKt1kZxlEcWjigWPGWok9MMi4mXLri/Os9l5gtQTHe/KqH6PDWF9bRTF2e4/+Q+is705w0T8PUVN50RlWdIDlpjY8YLYLt0KV078SNSubQTzbODzSCdKhE+ljqD3Vz/4Q2t1vzkAdBaZB7KKIgV7iqXv93IJvEc
x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:;
IPV:NLI; SFV:NSPM; H:SJ0PR11MB5022.namprd11.prod.outlook.com; PTR:; CAT:NONE;
SFS:(13230022)(366004)(346002)(136003)(396003)(39860400002)(376002)(451199015)(9686003)(53546011)(55016003)(316002)(26005)(41300700001)(7696005)(107886003)(33656002)(6506007)(110136005)(54906003)(478600001)(71200400001)(38100700002)(122000001)(82960400001)(38070700005)(186003)(83380400001)(86362001)(5660300002)(52536014)(8936002)(4326008)(8676002)(66476007)(66556008)(76116006)(2906002)(66446008)(64756008)(66946007);
DIR:OUT; SFP:1102;
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?R3+ihJO6RZ6h/WVt5PqIaB3K6H+hX6pCIkz1vDZRz0FT1uGRHIaxFWJIyF9C?=
=?us-ascii?Q?FDMoRVywOLY6b3lUWyKGSnWtvaQCiHTDonVuZUbg2EP7Z4rkmyZI4sHUhIrf?=
=?us-ascii?Q?vWOF0pC2taw+KRZmsrtTCfKRdlv3AtNEYzlHrEA+Sf7CEBem8ga5zcPH6iaM?=
=?us-ascii?Q?EXXJ1aidKNL9gk/hCmewqplWwjLHtWFL6dFF8DZ7MqdfKN13clOVvW6b075r?=
=?us-ascii?Q?ACc9CN/CFSmmEs9E5KmPo6f0oZHO+qlPa+4M41Dk8bmO2/iCi9ghIxBUTpjT?=
=?us-ascii?Q?SepIZ6AY/BtAzMBbliTzLWgu7P5WTQnF0jZ59Fip1T3o7k1EkF1CjtebBcWO?=
=?us-ascii?Q?aGsqNW2POoQ7638//XJFkYtUoSRuY2OdQgRy1zYUATZHgkQfNScnfFNt3fMc?=
=?us-ascii?Q?n75N0e07OpEeu97nprJTGPrZt7ZdqE7NIHdv0uicQ5LaotD//VCr01lhzxC9?=
=?us-ascii?Q?rbzJMxEJXIPtIXrTz6bon1nuIcopeFNSukTbqX0+s8RCl5qsGzy6sNeFQlZW?=
=?us-ascii?Q?wUlSX9CcPbpiPUyr23s5MRH0GWAaThwrl9tU3i2DTVZWHhmsb0upaJw5uUOy?=
=?us-ascii?Q?XOhHfR4UtW5XoBV3VVrcXYv6DlhHGiN+xiO+Tu3XPxwcBHGlGKBwMdZ2yJfh?=
=?us-ascii?Q?j3+ZwmKC0k0rKQQhnt1V8tZUIdYcPqWUrtEiQ3dBi/dcHf3a1nR2Ys9SDs/P?=
=?us-ascii?Q?hyOtyfdIFrSla4QPUoUZeND9XuLuK6pSuo7SggW617VFqgeV985/fhwSzVLb?=
=?us-ascii?Q?OIOP6TjmoubzH0a5halbnhZ0T1aSV3V1n9GOvNdNnU70y8TWOApqJ6yZYlN5?=
=?us-ascii?Q?dxye1bPDhmNyZyA5RAkJZFhDJgN/G7Nj5IhXAPf7PvRmCSSE07jfhWg2B3JI?=
=?us-ascii?Q?ZduNHopfknwerm9glTZq9+ItpXZVBz0mhj+Xiw844bwRSLfvnsOSUdf8JVc3?=
=?us-ascii?Q?hbmHizOTeibWC8wfqIPAvCHH2q6g9Uf7AksfcV4uUNxSenDUjUTc8fI1cm1n?=
=?us-ascii?Q?tOfv2AaX61FmTyDl4lhOnr4y8SrjP5I7q0Q4z3lwjwgE4xZkOj4LvD8KA7sw?=
=?us-ascii?Q?vqH4nXx1aBYdULTW8YkvBsk2C5PfePwVsrmmhdgILCEw7Z0KTuaTx+8YygNF?=
=?us-ascii?Q?rQ9zHOK8JDWEmTb82y3gr9sxVD32M8lxE5W+I+b9JJrs0mMyuxVFfyMofblf?=
=?us-ascii?Q?TXSx/wc/u1rSHzdDwBeHO5OmQs7JNAOjnqv7w7x4uGRKUU43q1g3Tf2LipCQ?=
=?us-ascii?Q?MiuDRsE0wAcFb88VS+ODRuNamtjmhJP2ULvZ4Tj3MQp4bXu2T1qhBE4ZryfL?=
=?us-ascii?Q?0zki1sjvRFAqXIT1ygkuj2vODYHc8GuqN8ywkjnd35iFiflTO1K4TpzR7TOZ?=
=?us-ascii?Q?9X/OdoCBS2EHhs63NVpOMtU3QZSfdb7QYzyX6B16oSH4rl/kdeWLOfstL7t9?=
=?us-ascii?Q?7LEfDjGz0UFSfLCUgYz2MZCSgcWv24x7XcI42PEiHsG91O4aA/hCDYEaQVUA?=
=?us-ascii?Q?JYS1Op43fxgbFl0UsEo/xUrRj+VS5zavevdIP2POnQQNNZct+tkLNzR9Dp2C?=
=?us-ascii?Q?6ZI/MUvjE52MybrlJgeDOciqPExPsAVDLwgGtQNH?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: SJ0PR11MB5022.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 26a769ff-a369-441e-6188-08da96bce8ea
X-MS-Exchange-CrossTenant-originalarrivaltime: 15 Sep 2022 01:52:13.1570 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: k/iE9ywUVYTV2iZLbq4w3PGzEMphpUEhhSg5BXhCSDA0/vJ80ZD/eSjw0/T+U6gjF92FEVklyoLZcV0j/zjFtA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR11MB4515
X-OriginatorOrg: intel.com
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
<mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
<mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org
> -----Original Message-----
> From: Pavan Nikhilesh Bhagavatula <pbhagavatula@marvell.com>
> Sent: Thursday, September 8, 2022 1:27 PM
> To: Yan, Zhirun <zhirun.yan@intel.com>; dev@dpdk.org; Jerin Jacob
> Kollanukkaran <jerinj@marvell.com>; Kiran Kumar Kokkilagadda
> <kirankumark@marvell.com>
> Cc: Liang, Cunming <cunming.liang@intel.com>; Wang, Haiyue
> <haiyue.wang@intel.com>
> Subject: RE: [EXT] [RFC, v1 4/6] graph: enhance graph walk by cross-core
> dispatch
>=20
> > This patch enhance the task scheduler mechanism to enable dispatching
> > tasks to another worker cores. Currently, there is only a local work
> > queue for one graph to walk. We introduce a scheduler worker queue in
> > each worker core for dispatching tasks. It will perform the walk on
> > scheduler work queue first, then handle the local work queue.
> >
> > Signed-off-by: Haiyue Wang <haiyue.wang@intel.com>
> > Signed-off-by: Cunming Liang <cunming.liang@intel.com>
> > Signed-off-by: Zhirun Yan <zhirun.yan@intel.com>
> > ---
> > lib/graph/graph.c | 6 ++++++
> > lib/graph/rte_graph_worker.h | 11 +++++++++++
> > 2 files changed, 17 insertions(+)
> >
> > diff --git a/lib/graph/graph.c b/lib/graph/graph.c index
> > b4eb18175a..49ea2b3fbb 100644
> > --- a/lib/graph/graph.c
> > +++ b/lib/graph/graph.c
> > @@ -368,6 +368,8 @@ rte_graph_destroy(rte_graph_t id)
> > while (graph !=3D NULL) {
> > tmp =3D STAILQ_NEXT(graph, next);
> > if (graph->id =3D=3D id) {
> > + /* Destroy the schedule work queue if has */
> > + graph_sched_wq_destroy(graph);
> > /* Call fini() of the all the nodes in the graph */
> > graph_node_fini(graph);
> > /* Destroy graph fast path memory */ @@ -470,6 +472,10
> @@
> > graph_clone(struct graph *parent_graph, const char *name,
> > if (graph_node_init(graph))
> > goto graph_mem_destroy;
> >
> > + /* Create the graph schedule work queue */
> > + if (graph_sched_wq_create(graph, parent_graph))
> > + goto graph_mem_destroy;
> > +
> > /* All good, Lets add the graph to the list */
> > graph_id++;
> > STAILQ_INSERT_TAIL(&graph_list, graph, next); diff --git
> > a/lib/graph/rte_graph_worker.h b/lib/graph/rte_graph_worker.h index
> > faf3f31ddc..e98697d880 100644
> > --- a/lib/graph/rte_graph_worker.h
> > +++ b/lib/graph/rte_graph_worker.h
> > @@ -177,6 +177,7 @@ static inline void rte_graph_walk(struct
> > rte_graph *graph) {
> > const rte_graph_off_t *cir_start =3D graph->cir_start;
> > + const unsigned int lcore_id =3D graph->lcore_id;
> > const rte_node_t mask =3D graph->cir_mask;
> > uint32_t head =3D graph->head;
> > struct rte_node *node;
> > @@ -184,6 +185,9 @@ rte_graph_walk(struct rte_graph *graph)
> > uint16_t rc;
> > void **objs;
> >
> > + if (graph->wq !=3D NULL)
> > + __rte_graph_sched_wq_process(graph);
> > +
>=20
>=20
> We should introduce a flags field in rte_graph_param which can be used by
> the application to define whether a graph should support multi-core
> dispatch.
>=20
Yes, I will add a flags field in next version.
> Then we can make `__rte_graph_sched_wq_process` as node 0 during graph
> creation so that it will be always called at the start of graph processin=
g
> followed by calling rest of the nodes.
> This will remove unnecessary branches in fastpath.
>=20
Thanks for your comments and sorry for my late reply.
Yes, we can make `__rte_graph_sched_wq_process` as node 0 with dispatch fla=
gs.
But I am not sure whether we need to register a new node here. It means we =
should change
the graph topo, and it will be an isolated node with no links.
> > /*
> > * Walk on the source node(s) ((cir_start - head) -> cir_start) and
> > then
> > * on the pending streams (cir_start -> (cir_start + mask) ->
> > cir_start) @@ -205,6 +209,12 @@ rte_graph_walk(struct rte_graph
> *graph)
> > objs =3D node->objs;
> > rte_prefetch0(objs);
> >
> > + /* Schedule the node until all task/objs are done */
> > + if (node->lcore_id !=3D RTE_MAX_LCORE && (int32_t)head > 0
> > &&
> > + lcore_id !=3D node->lcore_id && graph->rq !=3D NULL &&
> > + __rte_graph_sched_node_enqueue(node, graph->rq))
> > + goto next;
> > +
> > if (rte_graph_has_stats_feature()) {
> > start =3D rte_rdtsc();
> > rc =3D node->process(graph, node, objs, node->idx); @@ -215,6
> +225,7
> > @@ rte_graph_walk(struct rte_graph *graph)
> > node->process(graph, node, objs, node->idx);
> > }
> > node->idx =3D 0;
> > + next:
> > head =3D likely((int32_t)head > 0) ? head & mask : head;
> > }
> > graph->tail =3D 0;
> > --
> > 2.25.1