From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 625E441CE4; Mon, 20 Feb 2023 09:28:38 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 3A8B442D33; Mon, 20 Feb 2023 09:28:38 +0100 (CET) Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by mails.dpdk.org (Postfix) with ESMTP id 94C4640395 for ; Mon, 20 Feb 2023 09:28:36 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1676881716; x=1708417716; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-transfer-encoding:mime-version; bh=Zxb62BLQihs0VQLkd3TDmH9vL+dcYiRgW1TR7Ogt5MQ=; b=gdvMWjy/QaR+xvMIyxpyI8LHiR5EOgU3ZUjVVqw9uObusMxZVbHd7ROQ zMIDmGe2WM1dm7HdBbTPR2CuxEPQSi2fPrwXMpEXh8MCI3azPJqBK+cJ1 cdRLt7c2ukvCUymWRx33ASQ+zu5rI+nBLkFBbtVw+ZRFBZqlht55Tl+yZ xMjeFfakzImGSTpvqZLISfV+n4wFvjx7MvOIru189OjK/6TZFwR6Q2Obk rhFotT/X3lBPsjT08yN/mAidwfX+BIr/BsmTaLBubSAQoS30pb7Wuli1G tPmveJLp/5vW02SouomX023Wwmmk9eepUD26cKg3gw/AhYmp7GxX5C/Pm Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10626"; a="334550843" X-IronPort-AV: E=Sophos;i="5.97,311,1669104000"; d="scan'208";a="334550843" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Feb 2023 00:28:35 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10626"; a="1000186640" X-IronPort-AV: E=Sophos;i="5.97,311,1669104000"; d="scan'208";a="1000186640" Received: from fmsmsx602.amr.corp.intel.com ([10.18.126.82]) by fmsmga005.fm.intel.com with ESMTP; 20 Feb 2023 00:28:34 -0800 Received: from fmsmsx610.amr.corp.intel.com (10.18.126.90) by fmsmsx602.amr.corp.intel.com (10.18.126.82) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.16; Mon, 20 Feb 2023 00:28:34 -0800 Received: from fmsedg602.ED.cps.intel.com (10.1.192.136) by fmsmsx610.amr.corp.intel.com (10.18.126.90) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.16 via Frontend Transport; Mon, 20 Feb 2023 00:28:34 -0800 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (104.47.70.100) by edgegateway.intel.com (192.55.55.71) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.16; Mon, 20 Feb 2023 00:28:34 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=l1e4pq91KClUSzdENkcUAhy/5bjgh54ukNG9GxOc9Inr9ZVpG+lLs8B2iGMRbR8tuh7eirys8Ap7aOSk0lu8hiYlZzKN7/qJNylU91GumjUBiUpYQlWwaLo6mG0wWRFGZtXuhBiROB+km/QAhEDqL/vv5v/jB3mPEtLcqTgpyTIkVNA9S+i5p2eFHFmmRIBdplo4xYPrWhXZSlc2p/YZ0UVWLU3rLxFZ3OtKN6eA46Ud3D6dpXRljOgGTHq7halAQpFKWhDCql/ewS0WGR3NJMxCH9dk/paWxgND8SfyP5/fJwbKyFQbPcFr/QpVwIgC7kLjN5HLwyczBA+yhultaA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=/q7/Hbxrohzn/8jw471SYralIdNzSxc2G3mhs/wdNHM=; b=H3v4uzCuEvhwA+4gklziiSiUIwDoHiTYOg/k2L5BZLTVOZf/cl+nb1PzlJNmhakAtktMOqMbyk7TwVBhSzLZzDcwS3x3hL34OOLyV0LQAm/kHGFpgoSVWQpizx639JFhAowipMy5Sw5b3ABBqQufvW283j4CJofWbqPGNpCqe/ImD+uTy/A3vltUtuqf6Wp7WS0rl2Xgfq04Tbxk3D+gbxqUuD7PHoiumzYvmQdV/Ne+fW86ZnvD4G2b9i5Q3G5DnpWEYFsAFwsO1v1RDI+eQ4blDh4aeswcPNyIvxq+YFrtT8CaceHEvdoCyETB2ePnH9ECihvOMqwoQApAV7+FqA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Received: from SN7PR11MB6775.namprd11.prod.outlook.com (2603:10b6:806:264::21) by PH0PR11MB5208.namprd11.prod.outlook.com (2603:10b6:510:3b::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6111.20; Mon, 20 Feb 2023 08:28:32 +0000 Received: from SN7PR11MB6775.namprd11.prod.outlook.com ([fe80::6a5b:aa30:9a2c:28c3]) by SN7PR11MB6775.namprd11.prod.outlook.com ([fe80::6a5b:aa30:9a2c:28c3%5]) with mapi id 15.20.6111.020; Mon, 20 Feb 2023 08:28:32 +0000 From: "Yan, Zhirun" To: Thomas Monjalon , "jerinj@marvell.com" , "kirankumark@marvell.com" , "ndabilpuram@marvell.com" CC: "dev@dpdk.org" , "Liang, Cunming" , "Wang, Haiyue" Subject: RE: [PATCH v1 00/13] graph enhancement for multi-core dispatch Thread-Topic: [PATCH v1 00/13] graph enhancement for multi-core dispatch Thread-Index: AQHY+kLLuYPJzy3LwECL6ktbFntEDq7XjkqAgACF0zA= Date: Mon, 20 Feb 2023 08:28:31 +0000 Message-ID: References: <20221117050926.136974-1-zhirun.yan@intel.com> <3517338.eFTFzoEnKi@thomas> In-Reply-To: <3517338.eFTFzoEnKi@thomas> Accept-Language: en-US Content-Language: en-US X-Mentions: jerinj@marvell.com,kirankumark@marvell.com X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: SN7PR11MB6775:EE_|PH0PR11MB5208:EE_ x-ms-office365-filtering-correlation-id: 73c94302-4b59-4b31-f6c0-08db131c7376 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: Fk4N6aDUdnvO4T5DfzSYHYbT5TZfatmouLTaL0m3Ms1ecI3I576bTtkAd1DZv7gbyd8u2LeAe/5IkOjORzuLQ2akrQHJBP0rWYI+aKJKQqr3XAA7gXrWYUtn58851stHrUrY792KYdzOe7+g9yA8GSJVtHkwdFRWTL6JwEuvDIYkC9ZR2QWFdqK3WP4B91wjMnBKFUkMTcYtzZQTLqNTdIL7Yi/NrU4vTqLswc6b5SPVQMInJbJr11XTT7GL7hL+9nzU+xf8SsA9h8/UKT/DImd/1WwuBOWGTns6d+JCB/mhnoQ77fMaI8bifxA76cWYx2vJU8AYXwsqTQrVWyU65V+tEHLDZ+j5S9YAIoFgBG4J2Pjetoxx0eXGivZAfw93qSozMdp1cxOsENmtWsc0SjKs5Zr9vRyfsisHI6mTJY7UkJg/dEgO+jx/vOqmWRQG0S70MMEnEwmHEIb603duB/fLKmGDgPIWsKq8zMFP2MEB/ppEIbkJ8hq6eiZdOxIArPIagDCmUMqxeEp3tFMp1rusmJ/ddGIqIQDqOHgBsLRlwaIBFtus0cUKrs08/Z7naInSMZJaTpl00lEwe5eAjmBy5kE5sfj8ZVunK/iTlSQn5BP8JGaFccLLx8zESAt4xGn9hwpxJHM87Z0gN6T/2DPFQVzqSRNfO6Gi14Dw3UcarRgSonP29KOVdwL4KWjH x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:SN7PR11MB6775.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230025)(346002)(376002)(136003)(366004)(396003)(39860400002)(451199018)(33656002)(86362001)(7696005)(71200400001)(186003)(26005)(9686003)(2906002)(966005)(38070700005)(82960400001)(122000001)(38100700002)(83380400001)(8936002)(52536014)(41300700001)(478600001)(66946007)(66476007)(54906003)(110136005)(53546011)(6506007)(107886003)(76116006)(64756008)(66556008)(8676002)(66446008)(4326008)(5660300002)(55016003)(316002); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?F66wmS1eLzlfPuHqvvUmIdEU9o1kgjI5qGhQsGJ19xNFfR0P/S1x71D3jLoY?= =?us-ascii?Q?IeFnaqmU8YH24ok5//aDR8rjV85GB/+PLqssJDL1YsrUWmoczLRKSb7OEKT2?= =?us-ascii?Q?8AHtpxvYYz3vMR+bYjnAeuJFi7mDpdaVO7e45iRa+puHK/YYGFmfNiALDlFW?= =?us-ascii?Q?9NWnt6e473ksovSwFNRQ+7TpigcypNDs22tTpAvCtYpYUd3AZsLwCsf4+GVU?= =?us-ascii?Q?UeTXAR4pf7I24UrEiiTjXBv8cDE51MHEvEdT0Vy88kIYY6PWxVYt6SDjXRNZ?= =?us-ascii?Q?UKD3j/Ti3ufj9hfRpwX5i90fTR1u2I8Sny1f6LQ1y55i7t7HjAtvf3EKoS7H?= =?us-ascii?Q?D9nicpmK7WqWm5XkJ5AczeslvomKE012D1PAaBjwbGk0QX/dyu6FG0TGXSYr?= =?us-ascii?Q?GN7gn27NYvLCA/LOgOSV5dZv4Z4QLQrP8PGfNdVt7KolvO9tx+DeARUVZubr?= =?us-ascii?Q?mtUWG648TJlcgJIFcC18gJsORwQ7Sx/NQS1DuhoX3iNBJO16b3W+nQS012sX?= =?us-ascii?Q?s+pXaAbws2+7XEJ2jN8mFs0329Qe/c3I5iu6gTQk4XlYqdUaU90ELtkQyjjR?= =?us-ascii?Q?YTbUiGRQnroiO87ZwHYs1ks+1ulHZlavYbHW0ZVOgaLfMaSbIJsTQOwHXjUT?= =?us-ascii?Q?TJqGruuH1eG2nBQLRy1I77stj62teWSTAignxR5MUDMXgp3xnqFhLwlvD5Fz?= =?us-ascii?Q?QFDJ3fg7wPwRA7XfwQ5/YoZ882epXQB50if/eaf+PL9XsddUlAuUMXK2CjRj?= =?us-ascii?Q?gtY9xlYef2qwM+ODPd3oJZGlGUQy2+vXwRuAPLvGb/jA94DKgvkY2jDz2qLE?= =?us-ascii?Q?huR0f2rYiyBzS1rX6Vr+HihsexvJtmAq+rzxOLiLcqBLiZ/CRDgzgeYg0vO+?= =?us-ascii?Q?SOgczsHTjSH9xuFwmkuM4bD3zFeNg2YFL3Zu8RmbEhAroDi2hlihxcnOECrk?= =?us-ascii?Q?qEEPowRYmozzS5L195+22W0siQpCivuiXSpVr9N/PMviO3akaUGTxIJPPgZ8?= =?us-ascii?Q?seaGcPiHvbbkfXYx3xkH7IxcD/0ARot06BqmPqVhDosfpi18Fvl5s0Y+Ob6m?= =?us-ascii?Q?deAIXnqApDFZe+/AIgl8OQOKDUUk/HH8UL8Dz93Ws4E4odGg09+5XYd3ZEye?= =?us-ascii?Q?jo+D5+PIjqlHGv8jxhMnitm1impgku0jPoQT2QHpE11/RBS2LLlT7hMqx5BJ?= =?us-ascii?Q?wyV4g8xlEhGelu5YXiVso+CYTHo32x272sw7rhWnJ1B/cxNdsaHv9jh6r2j2?= =?us-ascii?Q?JvG2osruXRt79AZwqq0pSpMYwHg9I47w6uQyonRAozPUPo11GVkkdokhDLoX?= =?us-ascii?Q?OX+q1tNrGT7wpwKIxQx1nWQ3n3oFwf6AoTF0rMh2xC4xGcQj/4wxCO2Dyusf?= =?us-ascii?Q?C7F8RGh/FeYzl8Z0KTCJmrEC5Fbk4h+1P0Mo3GIcw3nWUK6rZjlXUDJm/9oA?= =?us-ascii?Q?M9OHpJy1fJDU5+j2aJtVs1Hhl40vXUDh4OtsLAl+3xPYH5CXNmFIeqLRJpgC?= =?us-ascii?Q?f1lxs+K9VLfOltGyHSrhkbDWbs6k0ivEetS9w9Z9CwUTa+CqGUvuWehxybi7?= =?us-ascii?Q?/eYxAjQAZnEWGhkbHxyz4rIzayrbf+Dsbx6m+b+N?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SN7PR11MB6775.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 73c94302-4b59-4b31-f6c0-08db131c7376 X-MS-Exchange-CrossTenant-originalarrivaltime: 20 Feb 2023 08:28:31.9933 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: fd7VjYDprXDh5oeJCDC7fx7Av5fV/g7wIKv6TRIjzKeMO4JY4/YxjTSrGMLl3gyba6TuKypXqnH64rnjr2EjgA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR11MB5208 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Hi Thomas, Jerin and Kiran gave some comments before. And @jerinj@marvell.com @kirankumark@marvell.com=20 could you help to review it? Thanks. > -----Original Message----- > From: Thomas Monjalon > Sent: Monday, February 20, 2023 8:22 AM > To: jerinj@marvell.com; kirankumark@marvell.com; > ndabilpuram@marvell.com > Cc: dev@dpdk.org; Liang, Cunming ; Wang, > Haiyue ; Yan, Zhirun ; Yan, > Zhirun > Subject: Re: [PATCH v1 00/13] graph enhancement for multi-core dispatch >=20 > This series doesn't look reviewed. > What is the status? >=20 > 17/11/2022 06:09, Zhirun Yan: > > Currently, rte_graph supports RTC (Run-To-Completion) model within > > each of a single core. > > RTC is one of the typical model of packet processing. Others like > > Pipeline or Hybrid are lack of support. > > > > The patch set introduces a 'generic' model selection which is a > > self-reacting scheme according to the core affinity. > > The new model enables a cross-core dispatching mechanism which > employs > > a scheduling work-queue to dispatch streams to other worker cores > > which being associated with the destination node. When core flavor of > > the destination node is a default 'current', the stream can be > > continue executed as normal. > > > > Example: > > 3-node graph targets 3-core budget > > > > Generic Model > > RTC: > > Config Graph-A: node-0->current; node-1->current; node-2->current; > > Graph-A':node-0/1/2 @0, Graph-A':node-0/1/2 @1, Graph-A':node-0/1/2 > @2 > > > > + - - - - - - - - - - - - - - - - - - - - - + > > ' Core #0/1/2 ' > > ' ' > > ' +--------+ +---------+ +--------+ ' > > ' | Node-0 | --> | Node-1 | --> | Node-2 | ' > > ' +--------+ +---------+ +--------+ ' > > ' ' > > + - - - - - - - - - - - - - - - - - - - - - + > > > > Pipeline: > > Config Graph-A: node-0->0; node-1->1; node-2->2; > > Graph-A':node-0 @0, Graph-A':node-1 @1, Graph-A':node-2 @2 > > > > + - - - - - -+ +- - - - - - + + - - - - - -+ > > ' Core #0 ' ' Core #1 ' ' Core #2 ' > > ' ' ' ' ' ' > > ' +--------+ ' ' +--------+ ' ' +--------+ ' > > ' | Node-0 | ' --> ' | Node-1 | ' --> ' | Node-2 | ' > > ' +--------+ ' ' +--------+ ' ' +--------+ ' > > ' ' ' ' ' ' > > + - - - - - -+ +- - - - - - + + - - - - - -+ > > > > Hybrid: > > Config Graph-A: node-0->current; node-1->current; node-2->2; > > Graph-A':node-0/1 @0, Graph-A':node-0/1 @1, Graph-A':node-2 @2 > > > > + - - - - - - - - - - - - - - - + + - - - - - -+ > > ' Core #0 ' ' Core #2 ' > > ' ' ' ' > > ' +--------+ +--------+ ' ' +--------+ ' > > ' | Node-0 | ------> | Node-1 | ' --> ' | Node-2 | ' > > ' +--------+ +--------+ ' ' +--------+ ' > > ' ' ' ' > > + - - - - - - - - - - - - - - - + + - - - - - -+ > > ^ > > | > > | > > + - - - - - - - - - - - - - - - + | > > ' Core #1 ' | > > ' ' | > > ' +--------+ +--------+ ' | > > ' | Node-0 | ------> | Node-1 | ' --------+ > > ' +--------+ +--------+ ' > > ' ' > > + - - - - - - - - - - - - - - - + > > > > > > The patch set has been break down as below: > > > > 1. Split graph worker into common and default model part. > > 2. Inline graph node processing and graph circular buffer walking to ma= ke > > it reusable. > > 3. Add set/get APIs to choose worker model. > > 4. Introduce core affinity API to set the node run on specific worker c= ore. > > (only use in new model) > > 5. Introduce graph affinity API to bind one graph with specific worker > > core. > > 6. Introduce graph clone API. > > 7. Introduce stream moving with scheduler work-queue in patch 8,9,10. > > 8. Add stats for new models. > > 9. Abstract default graph config process and integrate new model into > > example/l3fwd-graph. Add new parameters for model choosing. > > > > We could run with new worker model by this: > > ./dpdk-l3fwd-graph -l 8,9,10,11 -n 4 -- -p 0x1 --config=3D"(0,0,9)" -P > > --model=3D"generic" > > > > References: > > https://static.sched.com/hosted_files/dpdkuserspace22/a6/graph%20intro > > duce%20remote%20dispatch%20for%20mult-core%20scaling.pdf > > > > Zhirun Yan (13): > > graph: split graph worker into common and default model > > graph: move node process into inline function > > graph: add macro to walk on graph circular buffer > > graph: add get/set graph worker model APIs > > graph: introduce core affinity API > > graph: introduce graph affinity API > > graph: introduce graph clone API for other worker core > > graph: introduce stream moving cross cores > > graph: enable create and destroy graph scheduling workqueue > > graph: introduce graph walk by cross-core dispatch > > graph: enable graph generic scheduler model > > graph: add stats for corss-core dispatching > > examples/l3fwd-graph: introduce generic worker model > > > > examples/l3fwd-graph/main.c | 218 +++++++++-- > > lib/graph/graph.c | 179 +++++++++ > > lib/graph/graph_debug.c | 6 + > > lib/graph/graph_populate.c | 1 + > > lib/graph/graph_private.h | 44 +++ > > lib/graph/graph_stats.c | 74 +++- > > lib/graph/meson.build | 3 +- > > lib/graph/node.c | 1 + > > lib/graph/rte_graph.h | 44 +++ > > lib/graph/rte_graph_model_generic.c | 179 +++++++++ > > lib/graph/rte_graph_model_generic.h | 114 ++++++ > > lib/graph/rte_graph_model_rtc.h | 22 ++ > > lib/graph/rte_graph_worker.h | 516 ++------------------------ > > lib/graph/rte_graph_worker_common.h | 545 > ++++++++++++++++++++++++++++ > > lib/graph/version.map | 8 + > > 15 files changed, 1430 insertions(+), 524 deletions(-) create mode > > 100644 lib/graph/rte_graph_model_generic.c > > create mode 100644 lib/graph/rte_graph_model_generic.h > > create mode 100644 lib/graph/rte_graph_model_rtc.h create mode > > 100644 lib/graph/rte_graph_worker_common.h > > > > >=20 >=20 >=20 >=20