From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <dev-bounces@dpdk.org>
Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124])
by inbox.dpdk.org (Postfix) with ESMTP id B30DAA0C55;
Wed, 13 Oct 2021 19:57:08 +0200 (CEST)
Received: from [217.70.189.124] (localhost [127.0.0.1])
by mails.dpdk.org (Postfix) with ESMTP id A21274118E;
Wed, 13 Oct 2021 19:57:08 +0200 (CEST)
Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com
[67.231.156.173])
by mails.dpdk.org (Postfix) with ESMTP id 6D55A41140
for <dev@dpdk.org>; Wed, 13 Oct 2021 19:57:07 +0200 (CEST)
Received: from pps.filterd (m0045851.ppops.net [127.0.0.1])
by mx0b-0016f401.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 19DEsXT3013078;
Wed, 13 Oct 2021 10:57:05 -0700
Received: from nam12-dm6-obe.outbound.protection.outlook.com
(mail-dm6nam12lp2174.outbound.protection.outlook.com [104.47.59.174])
by mx0b-0016f401.pphosted.com with ESMTP id 3bp0h3s7mx-1
(version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT);
Wed, 13 Oct 2021 10:57:04 -0700
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
b=kZTBVxNWhwtnAVhwPtYltcszz+C3uRmNtQiTvWEx0MFMFJAMAiC/4khthGYsbS0rnrHIFIMPk8XoKV55CqJiPmDR8wQezsH48OfpuK96awFGluPZT9Ffr8F9fSICL3eGSXTeNoYsRlvK/wVM1rT9JjqY6JU0kNCxxcTO75RlyhcFDPIJgXjNkpwySD+ISfEjGHOYnIcYtfCu0eWT8JE/YPmY2tH2RW7iml9gIjTO39teKOoBFMjaKigOaHPrFVrKrlUiOEfoG/3yGQbwcbcPWmt7yzCD/DdFlUp/cwDi+L7fLBr4jzzpjV+oB3+6wsdXeqDmqQCTIbj65tznZl8/mQ==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
s=arcselector9901;
h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
bh=P5Zz0h10wlCa30+3zzSagg9zutFiipMdKZj7DziF5e8=;
b=SD5WdKb9LMw7FKi7alNzRiJtmGX9NAoRF140yHjRtZVk0fyZS7GMG7VhsMvXA4FDDSZwangKdPnjN6LwdVfwZCXS4W2BlxXlutISe66TcwL14K4vue33KfnMNr0laYJsg1qRuT/h1R6Fq/RLy4BLBUi2ow7GPxOd9JVo7g+9255j9s8UgOF0dKLNEYVDwjinh38nJBbTi8HoqBmzypNe7Uun9lDZM2yuwTcPWiHO6+2OWw5W1rOR6EJolCRyrUwkZzMyPEU9b4d9uB+zn9JWP/sP+3j4j8kWuOEJZ/oxq8uwSxrWGqNS34cFIcReovaKAR0uCZ11t1kximFZfMpn6w==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
smtp.mailfrom=marvell.com; dmarc=pass action=none header.from=marvell.com;
dkim=pass header.d=marvell.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=marvell.onmicrosoft.com; s=selector1-marvell-onmicrosoft-com;
h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
bh=P5Zz0h10wlCa30+3zzSagg9zutFiipMdKZj7DziF5e8=;
b=h5hOW8sWmQtSiN9AJODFW1OwC++4U62GWM2hnp4ixU6432kfNFBO61EGGiWl919B5rVp+LMMxohCuhUlUIIgZiIqgjhyaNW8S/n9OvASP8vsAvAb6gtV31rU2P96wOATcDgc76UVneOhRFAP1VM40UzMP+Byq3jvXFk1ywkVwLw=
Received: from BN9PR18MB4204.namprd18.prod.outlook.com (2603:10b6:408:119::18)
by BN8PR18MB2723.namprd18.prod.outlook.com (2603:10b6:408:76::29)
with Microsoft SMTP Server (version=TLS1_2,
cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4608.15; Wed, 13 Oct
2021 17:57:02 +0000
Received: from BN9PR18MB4204.namprd18.prod.outlook.com
([fe80::29f4:8e3d:264f:26b1]) by BN9PR18MB4204.namprd18.prod.outlook.com
([fe80::29f4:8e3d:264f:26b1%8]) with mapi id 15.20.4608.016; Wed, 13 Oct 2021
17:57:02 +0000
From: Harman Kalra <hkalra@marvell.com>
To: Harman Kalra <hkalra@marvell.com>, Thomas Monjalon <thomas@monjalon.net>
CC: Raslan Darawsheh <rasland@nvidia.com>, "dev@dpdk.org" <dev@dpdk.org>, Ray
Kinsella <mdr@ashroe.eu>, Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>, David
Marchand <david.marchand@redhat.com>, "viacheslavo@nvidia.com"
<viacheslavo@nvidia.com>, "matan@nvidia.com" <matan@nvidia.com>
Thread-Topic: [dpdk-dev] [EXT] Re: [PATCH v1 2/7] eal/interrupts: implement
get set APIs
Thread-Index: AQHXv3z/vwDxAXEBi0ulm3+8amQc96vRLyBggAAJB3A=
Date: Wed, 13 Oct 2021 17:57:01 +0000
Message-ID: <BN9PR18MB4204A56AE0FB98928883EAEBC5B79@BN9PR18MB4204.namprd18.prod.outlook.com>
References: <20210826145726.102081-1-hkalra@marvell.com>
<BN9PR18MB4204C9BC5E452A4EF3A9B946C5AE9@BN9PR18MB4204.namprd18.prod.outlook.com>
<CAJFAV8xMZPRHviqgxRfN36f__dUY1HW-n_MVgmEXECVP8DM7zQ@mail.gmail.com>
<14591983.bG23H4Sf1D@thomas>
<BN9PR18MB4204EF6156C920FBA23CEC3DC5B79@BN9PR18MB4204.namprd18.prod.outlook.com>
In-Reply-To: <BN9PR18MB4204EF6156C920FBA23CEC3DC5B79@BN9PR18MB4204.namprd18.prod.outlook.com>
Accept-Language: en-GB, en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-ms-publictraffictype: Email
x-ms-office365-filtering-correlation-id: 128b33cc-5797-4fbc-9969-08d98e72dc31
x-ms-traffictypediagnostic: BN8PR18MB2723:
x-ms-exchange-transport-forked: True
x-microsoft-antispam-prvs: <BN8PR18MB27235B832A7300DD85919790C5B79@BN8PR18MB2723.namprd18.prod.outlook.com>
x-ms-oob-tlc-oobclassifiers: OLM:9508;
x-ms-exchange-senderadcheck: 1
x-ms-exchange-antispam-relay: 0
x-microsoft-antispam: BCL:0;
x-microsoft-antispam-message-info: Puvme3KruILSS9w+k1im8nKcK0he7BsMMazCm7eAumYpxO3oafgb4tvEq5z4X7UQWyEsVthRh7V4nlI+XQiSM6BbCdAQAcUoeHKp+wgRmcXspAV7JL1EILv1Ftb+0V8pU3j6E9MkWD+jIqfS6bBQf07LX4V4oaZMYCbzXxbp5PhhcYr+tGOZ5IxgZCOjQ5/lzxrzrXthmAzdPhs+ZvfpxkqLJwDSQss/zr0KrSrDqWcMf/gvfrb+UZ4dRvrL/UpvAZyKbmcme9IUVHKe9mH76E7TUNHINn2kOB2ajQpLmVGr/A2fkAhMwuteqsKLIJ98oeDJV5VpMHTfQmDVcMIyqxlnskqEWIwkol4XiGnNxYaxTMIMHYGz6nL4EZPiAqnxTPnWT6Sd2gji/IT5kpPpO+iDB95cNkeOJBLaXaGMcg8I2GeuiTTNtd4DhsN4pDO+/SXRArI7+XKQAIOwaY0Hn8/rmGcUy5UR5TpIH156QOClOA3krNAripr8MHGlrQAVjWGDZo2SJlTWpRDjoy2Pdmw9f7Z7HwDfpaxsTdaEKJp4Ud/l/RDS3ltPJlxii3+FCgEhGbaqRWRk4fsYh0P4wJxrsmCUIonle4k2Pb3Ts7j3uCwTGWBeZAwHQR9Ed1QzmnKX/1PKn5CQRwN6gu5AbPieAOnRf2dVcwarBGDapDyeJ9vD+ERfhGnALODq4J8cSox+ns7w7yPvwKvkiP+dhw==
x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:;
IPV:NLI; SFV:NSPM; H:BN9PR18MB4204.namprd18.prod.outlook.com; PTR:; CAT:NONE;
SFS:(4636009)(366004)(186003)(66476007)(66556008)(64756008)(7696005)(66946007)(5660300002)(76116006)(71200400001)(26005)(38070700005)(33656002)(122000001)(316002)(4326008)(2940100002)(55016002)(38100700002)(54906003)(52536014)(53546011)(2906002)(508600001)(83380400001)(9686003)(110136005)(8936002)(6506007)(8676002)(66446008)(86362001);
DIR:OUT; SFP:1101;
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?SGiB2uETXkXVcFDg33ygoMJhuXUj0JZefTS1nlLkEJCaLNtSUxwwe5pJkNoK?=
=?us-ascii?Q?+RhwEAnUtBKOBcjdt1wSJFg74luAEDhX+Syvd8JQDWAyUBv1LHhkV+YfWFyI?=
=?us-ascii?Q?PSyUoVKjUxmmx4dHLt/9VSaynZSFeJn1WqXflm1yozzPv/3CQMdNozAn9iof?=
=?us-ascii?Q?IkNXHmazJud/U0YYHWmrZenUb0G75jXYsfhNvxoweNzgPqAQbD7aG3VEUdVQ?=
=?us-ascii?Q?vvhWP3fwXd3E15hF9G4v4/2KP8qONlDXKoO03qyGpE60VbsDXKu6OnULlyyd?=
=?us-ascii?Q?Oqut2wQgR2vfBYUhasLM7RksJf1TnnxDaZUGe70x+anUHYFnTL42e6zqfsnb?=
=?us-ascii?Q?51DX32ElhR9fWbZySiiu/m4a/Cmluz5E7kTnLJ9n8glIsY0GNE5ciW0VWoNE?=
=?us-ascii?Q?ZqhyFR4SrCPa5DmmHgfctDMzVBwXDJzxga+BdqaBopqdHh25XOaDPC6fIGbb?=
=?us-ascii?Q?ua/MF9eee1buqLzf+FRGbZA/rGnrpcy5BVs+XP1pA7R3bAMOVghOnTh4KN/M?=
=?us-ascii?Q?oGcI2f1Do2bhrzalk8OYxKY3Q9XDQkyHtuIFsQQX8cZ0yrxmkc4Ck7ZTOm4T?=
=?us-ascii?Q?+AQO3m7ThCApZjJskvZ4ROT+Sr2kM1vDPrAFYXSHWgzJ1O87gEIwB0K9hJTo?=
=?us-ascii?Q?0sIgf7ttLVkPj50Cts1I9Qodnjbb+HjShBh/w2BK0ckVMtgZ8k8ff4kQ12TS?=
=?us-ascii?Q?EtCZihZm/yfgP5T/3VBsa+1gFM+pX2k1DgUjtQUxtOlCDg08RZ052NTbVxWt?=
=?us-ascii?Q?XKpiDt/mrR1BXWk1TfLsqdVQUh1O6e6oq9xzdTwWZYr5a/XVBqkS5HDYbiqZ?=
=?us-ascii?Q?mjBmWTIBJGdNNgCpKUEroqcyvPCrwbt28yznnwFxuL7LjFHm9nZ3wI3w2RLz?=
=?us-ascii?Q?oEmj6dEIP7Nt+bQ9bNyTjRDmJaB3Zx38Wgnmxv7t3CJuUriF6Bszy5dcdxQ9?=
=?us-ascii?Q?lhAhzAU+qjhiNVDnsbwQGYrss3SAsv+x9VNgerdyKFBQLMJiQogvfaTb7cdJ?=
=?us-ascii?Q?/tkTpIVqXusR8rGJy8SAC0aAxKYktP0u16TyoJRlxpE8MJ9jlTrZeDC0YMoR?=
=?us-ascii?Q?ZxZDsk+fZXBvvSw0/MDINWqIpngsx2BRFCNk8YK1yTv0uXg41tK6jxzkJlDW?=
=?us-ascii?Q?ngRdu5bh+NZT7p95KBsqcvPEmUKch9O5xeEPbYE6nk1TVNyxfbaKobsSem8B?=
=?us-ascii?Q?C2ny0nEWz/4oWGx5tykrAi8Pr7MN9M2o18wJrXOM7NKaHIWS217vsJVbW0Up?=
=?us-ascii?Q?hRAkxTJOhuECdCQolR8Xv7jhMWxwaLPGJTAM7qQawSA5xZnFVDxPoPjLLirw?=
=?us-ascii?Q?6bPPLKg/XcPCdwObYvP5wb8o?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: marvell.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN9PR18MB4204.namprd18.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 128b33cc-5797-4fbc-9969-08d98e72dc31
X-MS-Exchange-CrossTenant-originalarrivaltime: 13 Oct 2021 17:57:02.0025 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 70e1fb47-1155-421d-87fc-2e58f638b6e0
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: SmgF4K2yjNhjuComRA0Yzp0dZgWK6kT7qLa+xZSjTcfPsgZA8gV4OzkeJqMbUentqmeRzJwXIEuAL3P9Y1xeHw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN8PR18MB2723
X-Proofpoint-GUID: LkICdYflHF8vc4wVuGm0yZ2bj11EhV1f
X-Proofpoint-ORIG-GUID: LkICdYflHF8vc4wVuGm0yZ2bj11EhV1f
X-Proofpoint-Virus-Version: vendor=baseguard
engine=ICAP:2.0.182.1,Aquarius:18.0.790,Hydra:6.0.425,FMLib:17.0.607.475
definitions=2021-10-13_06,2021-10-13_02,2020-04-07_01
Subject: Re: [dpdk-dev] [EXT] Re: [PATCH v1 2/7] eal/interrupts: implement
get set APIs
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
<mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
<mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org
Sender: "dev" <dev-bounces@dpdk.org>
> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Harman Kalra
> Sent: Wednesday, October 13, 2021 11:24 PM
> To: Thomas Monjalon <thomas@monjalon.net>
> Cc: Raslan Darawsheh <rasland@nvidia.com>; dev@dpdk.org; Ray Kinsella
> <mdr@ashroe.eu>; Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>; David
> Marchand <david.marchand@redhat.com>; viacheslavo@nvidia.com;
> matan@nvidia.com
> Subject: Re: [dpdk-dev] [EXT] Re: [PATCH v1 2/7] eal/interrupts: implemen=
t
> get set APIs
>=20
>=20
>=20
> > -----Original Message-----
> > From: Thomas Monjalon <thomas@monjalon.net>
> > Sent: Tuesday, October 12, 2021 8:53 PM
> > To: Harman Kalra <hkalra@marvell.com>
> > Cc: Raslan Darawsheh <rasland@nvidia.com>; dev@dpdk.org; Ray Kinsella
> > <mdr@ashroe.eu>; Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>; David
> > Marchand <david.marchand@redhat.com>; viacheslavo@nvidia.com;
> > matan@nvidia.com
> > Subject: Re: [dpdk-dev] [EXT] Re: [PATCH v1 2/7] eal/interrupts:
> > implement get set APIs
> >
> > 04/10/2021 11:57, David Marchand:
> > > On Mon, Oct 4, 2021 at 10:51 AM Harman Kalra <hkalra@marvell.com>
> > wrote:
> > > > > > +struct rte_intr_handle *rte_intr_handle_instance_alloc(int siz=
e,
> > > > > > + bool
> > > > > > +from_hugepage) {
> > > > > > + struct rte_intr_handle *intr_handle;
> > > > > > + int i;
> > > > > > +
> > > > > > + if (from_hugepage)
> > > > > > + intr_handle =3D rte_zmalloc(NULL,
> > > > > > + size * sizeof(struct =
rte_intr_handle),
> > > > > > + 0);
> > > > > > + else
> > > > > > + intr_handle =3D calloc(1, size * sizeof(struct
> > > > > > + rte_intr_handle));
> > > > >
> > > > > We can call DPDK allocator in all cases.
> > > > > That would avoid headaches on why multiprocess does not work in
> > > > > some rarely tested cases.
> > > > > Wdyt?
> > > > >
> > > > > Plus "from_hugepage" is misleading, you could be in --no-huge
> > > > > mode, rte_zmalloc still works.
> > > >
> > > > <HK> In mellanox 5 driver interrupt handle instance is freed in
> > > > destructor " mlx5_pmd_interrupt_handler_uninstall()" while DPDK
> > > > memory allocators are already cleaned up in "rte_eal_cleanup".
> > > > Hence I allocated interrupt instances for such cases from normal he=
ap.
> > > > There could be other such cases so I think its ok to keep this supp=
ort.
> > >
> > > This is surprising.
> > > Why would the mlx5 driver wait to release in a destructor?
> > > It should be done once no interrupt handler is necessary (like when
> > > stopping all ports), and that would be before rte_eal_cleanup().
> >
> > I agree with David.
> > I prefer a simpler API which always use rte_malloc, and make sure
> > interrupts are always handled between rte_eal_init and rte_eal_cleanup.
> > The mlx5 PMD could be reworked to match this requirement.
> > In any case we should not any memory management in
> > constructors/destructors.
>=20
>=20
> Hi Thomas, David
>=20
> There are couple of more dependencies on glibc heap APIs:
> 1. "rte_eal_alarm_init()" allocates an interrupt instance which is used f=
or
> timerfd, is called before "rte_eal_memory_init()" which does the memseg
> init.
> Not sure what all challenges we may face in moving alarm_init after
> memory_init as it might break some subsystem inits.
> Other option could be to allocate interrupt instance for timerfd on first
> alarm_setup call.
>=20
> 2. Currently interrupt handle field inside struct rte_pci_device is stati=
c which
> is changed to a pointer in this series(as struct rte_intr_handle is hidde=
n inside
> a c file and size is unknown outside).
> I am allocating the memory for this interrupt instance inside
> "rte_pci_probe_one_driver()" just before "pci_vfio_map_resource()" which
> sets up vfio resources and calls " pci_vfio_setup_interrupts()" which set=
ups
> the interrupt support. Here challenge is "rte_bus_probe()" also gets call=
ed
> before "rte_eal_memory_init()".
Sorry my bad, bus probing happens after "rte_eal_memory_init()", so second =
point
Is invalid.
>=20
> There are many other drivers which statically declares the interrupt hand=
les
> inside their respective private structures and memory for those structure
> was allocated from heap. For such drivers I allocated interrupt instances=
also
> using glibc heap APIs.
>=20
> Thanks
> Harman
>=20
>=20
>=20
> >