From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9B27342C04; Thu, 1 Jun 2023 17:10:06 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 499C940DDC; Thu, 1 Jun 2023 17:10:06 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mails.dpdk.org (Postfix) with ESMTP id B33DD406BA for ; Thu, 1 Jun 2023 17:10:04 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685632204; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7wKPRm/garyDe5BcbRRUbiFEhGxv45kFl5pvR67UOCI=; b=eo5eVBerdwUsto5iH0iIutBAI9bFn4zhL0NhZ8fiRxcnQtwLcq8T5qDtD14RSo1xWcykQY QyHnzjnkJPNpE1MXcb/rYmATGY3Ag/VY2Y2KwudaYadlmBqQqkUP1gn7lg5h7omH8IBUdg 0hWp4xw/AVh8dTMdplt7Zr5TAqijgGY= Received: from mail-pl1-f199.google.com (mail-pl1-f199.google.com [209.85.214.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-338-K5Ojui5jPjyV6mwSLFT1vQ-1; Thu, 01 Jun 2023 11:10:01 -0400 X-MC-Unique: K5Ojui5jPjyV6mwSLFT1vQ-1 Received: by mail-pl1-f199.google.com with SMTP id d9443c01a7336-1b031a8fbe5so4081475ad.2 for ; Thu, 01 Jun 2023 08:10:00 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685632200; x=1688224200; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7wKPRm/garyDe5BcbRRUbiFEhGxv45kFl5pvR67UOCI=; b=bVcv0s7pr5FN4v7ouZI9Bk5YAiTn4zjl3JU1xNVC4THaGqvhocgjoOTG50KLPdmBXI RkTnbOZmHxGexfUrSL2WJtOvZI4udiSuR+kWu9ktzTHQCG7JryI+QQP89E0cMbh1buHx BeP4L0SlHatM6orRsGiVQpD5BdBa08sWuapXIuRIcoRYv3GNZRWYUtZ+iKgPw5DLShoR DGjANY1iLJt1drXUERjU9qMx7A5QRIvkmZmwFEMxPT+8vAAAm081CkwEQxWLZSN7vKy6 ysio5iL183AL2CwmKFjU5IinyA0eAn5kawJS4ENX097uZte/fRAr95ViNHObcyuXQ/R7 FNXQ== X-Gm-Message-State: AC+VfDy9MinAkcStSKgyOz4RioXwtUJ1p1+O2tcJHoYkRmBFOYONBF9W tny8PFSMki3JQp2kOOh3uj6SnuQ+h7hA26eFRuh8SuADlSgwjKJMzcY35guM7KRx7FcCvHFlJ7O RQTw0iHpcJE0741cAGIs= X-Received: by 2002:a17:902:ceca:b0:1b0:6c10:6836 with SMTP id d10-20020a170902ceca00b001b06c106836mr4922251plg.33.1685632200029; Thu, 01 Jun 2023 08:10:00 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5yhakQw+qZz6y6sroORenppTWrQzSqRrlwwa1bj04ZeG9fflQPS4oEfojMbvax3jwn3uFn9P7FVl7sqrjmI70= X-Received: by 2002:a17:902:ceca:b0:1b0:6c10:6836 with SMTP id d10-20020a170902ceca00b001b06c106836mr4922225plg.33.1685632199703; Thu, 01 Jun 2023 08:09:59 -0700 (PDT) MIME-Version: 1.0 References: <20230124140746.594066-1-nipun.gupta@amd.com> <20230525100821.12148-1-nipun.gupta@amd.com> <20230525100821.12148-5-nipun.gupta@amd.com> In-Reply-To: <20230525100821.12148-5-nipun.gupta@amd.com> From: David Marchand Date: Thu, 1 Jun 2023 17:09:48 +0200 Message-ID: Subject: Re: [PATCH v5 4/5] bus/cdx: add support for MSI To: Nipun Gupta Cc: dev@dpdk.org, thomas@monjalon.net, hkalra@marvell.com, anatoly.burakov@intel.com, stephen@networkplumber.org, ferruh.yigit@amd.com, harpreet.anand@amd.com, nikhil.agarwal@amd.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Thu, May 25, 2023 at 12:09=E2=80=AFPM Nipun Gupta = wrote: > > MSI's are exposed to the devices using VFIO (vfio-cdx). This > patch uses the same to add support for MSI for the devices on > the cdx bus. > > A couple of API's have been introduced in the EAL interrupt > framework: > - rte_intr_irq_count_set: This API is used to set the total > interrupts on the interrupt handle. This would be provided > by VFIO (irq.count) for VFIO enabled devices. > - rte_intr_irq_count_get: This API returns the total number > interrupts which were set. > > Signed-off-by: Nipun Gupta > Acked-by: Ferruh Yigit > --- > drivers/bus/cdx/bus_cdx_driver.h | 25 +++++ > drivers/bus/cdx/cdx.c | 11 ++ > drivers/bus/cdx/cdx_vfio.c | 182 ++++++++++++++++++++++++++++++- > drivers/bus/cdx/version.map | 2 + > 4 files changed, 218 insertions(+), 2 deletions(-) > > diff --git a/drivers/bus/cdx/bus_cdx_driver.h b/drivers/bus/cdx/bus_cdx_d= river.h > index f1dce06a16..a8c54d728e 100644 > --- a/drivers/bus/cdx/bus_cdx_driver.h > +++ b/drivers/bus/cdx/bus_cdx_driver.h > @@ -67,6 +67,7 @@ struct rte_cdx_device { > struct rte_cdx_id id; /**< CDX ID. */ > struct rte_mem_resource mem_resource[CDX_MAX_RESOURCE]; > /**< CDX Memory Resource = */ > + struct rte_intr_handle *intr_handle; /**< Interrupt handle */ > }; > > /** > @@ -168,6 +169,30 @@ void rte_cdx_unmap_device(struct rte_cdx_device *dev= ); > __rte_internal > void rte_cdx_register(struct rte_cdx_driver *driver); > > +/** > + * Enables VFIO Interrupts for CDX bus devices. > + * > + * @param intr_handle > + * Pointer to the interrupt handle. > + * > + * @return > + * 0 on success, -1 on error. > + */ > +__rte_internal > +int rte_cdx_vfio_intr_enable(const struct rte_intr_handle *intr_handle); > + > +/** > + * Disable VFIO Interrupts for CDX bus devices. > + * > + * @param intr_handle > + * Pointer to the interrupt handle. > + * > + * @return > + * 0 on success, -1 on error. > + */ > +__rte_internal > +int rte_cdx_vfio_intr_disable(const struct rte_intr_handle *intr_handle)= ; > + > /** > * Helper for CDX device registration from driver (eth, crypto, raw) ins= tance > */ > diff --git a/drivers/bus/cdx/cdx.c b/drivers/bus/cdx/cdx.c > index 64ea879f3b..c691c38e04 100644 > --- a/drivers/bus/cdx/cdx.c > +++ b/drivers/bus/cdx/cdx.c > @@ -201,6 +201,15 @@ cdx_scan_one(const char *dirname, const char *dev_na= me) > goto err; > } > > + /* Allocate interrupt instance for cdx device */ > + dev->intr_handle =3D > + rte_intr_instance_alloc(RTE_INTR_INSTANCE_F_PRIVATE); > + if (dev->intr_handle =3D=3D NULL) { > + CDX_BUS_ERR("Failed to create interrupt instance for %s\n= ", CDX_BUS_ERR already appends a \n. > + dev->device.name); > + return -ENOMEM; > + } > + > /* > * Check if device is bound to 'vfio-cdx' driver, so that user-sp= ace > * can gracefully access the device. > @@ -391,6 +400,8 @@ cdx_probe_one_driver(struct rte_cdx_driver *dr, > return ret; > > error_probe: > + rte_intr_instance_free(dev->intr_handle); > + dev->intr_handle =3D NULL; > cdx_vfio_unmap_resource(dev); > error_map_device: > return ret; > diff --git a/drivers/bus/cdx/cdx_vfio.c b/drivers/bus/cdx/cdx_vfio.c > index e54432de5b..ec6512e158 100644 > --- a/drivers/bus/cdx/cdx_vfio.c > +++ b/drivers/bus/cdx/cdx_vfio.c > @@ -50,6 +50,10 @@ struct mapped_cdx_resource { > /** mapped cdx device list */ > TAILQ_HEAD(mapped_cdx_res_list, mapped_cdx_resource); > > +/* IRQ set buffer length for MSI interrupts */ > +#define MSI_IRQ_SET_BUF_LEN (sizeof(struct vfio_irq_set) + \ > + sizeof(int) * (RTE_MAX_RXTX_INTR_VEC_ID + 1= )) > + > static struct rte_tailq_elem cdx_vfio_tailq =3D { > .name =3D "VFIO_CDX_RESOURCE_LIST", > }; > @@ -94,6 +98,27 @@ cdx_vfio_unmap_resource_primary(struct rte_cdx_device = *dev) > char cdx_addr[PATH_MAX] =3D {0}; > struct mapped_cdx_resource *vfio_res =3D NULL; > struct mapped_cdx_res_list *vfio_res_list; > + int ret, vfio_dev_fd; > + > + if (rte_intr_fd_get(dev->intr_handle) < 0) > + return -1; > + > + if (close(rte_intr_fd_get(dev->intr_handle)) < 0) { > + CDX_BUS_ERR("Error when closing eventfd file descriptor f= or %s", > + dev->device.name); > + return -1; > + } > + > + vfio_dev_fd =3D rte_intr_dev_fd_get(dev->intr_handle); > + if (vfio_dev_fd < 0) > + return -1; > + > + ret =3D rte_vfio_release_device(rte_cdx_get_sysfs_path(), dev->de= vice.name, > + vfio_dev_fd); > + if (ret < 0) { > + CDX_BUS_ERR("Cannot release VFIO device"); > + return ret; > + } > > vfio_res_list =3D > RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list); > @@ -116,6 +141,18 @@ cdx_vfio_unmap_resource_secondary(struct rte_cdx_dev= ice *dev) > { > struct mapped_cdx_resource *vfio_res =3D NULL; > struct mapped_cdx_res_list *vfio_res_list; > + int ret, vfio_dev_fd; > + > + vfio_dev_fd =3D rte_intr_dev_fd_get(dev->intr_handle); > + if (vfio_dev_fd < 0) > + return -1; > + > + ret =3D rte_vfio_release_device(rte_cdx_get_sysfs_path(), dev->de= vice.name, > + vfio_dev_fd); > + if (ret < 0) { > + CDX_BUS_ERR("Cannot release VFIO device"); > + return ret; > + } > > vfio_res_list =3D > RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list); > @@ -140,9 +177,80 @@ cdx_vfio_unmap_resource(struct rte_cdx_device *dev) > return cdx_vfio_unmap_resource_secondary(dev); > } > > +/* set up interrupt support (but not enable interrupts) */ > static int > -cdx_rte_vfio_setup_device(int vfio_dev_fd) > +cdx_vfio_setup_interrupts(struct rte_cdx_device *dev, int vfio_dev_fd, Why rename this function? > + int num_irqs) > { > + int i, ret; > + > + if (num_irqs =3D=3D 0) > + return 0; > + > + /* start from MSI interrupt type */ > + for (i =3D 0; i < num_irqs; i++) { [snip] --=20 David Marchand