From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 0703DA0C47; Tue, 5 Oct 2021 14:17:24 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 48CCC413AA; Tue, 5 Oct 2021 14:16:59 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0a-0016f401.pphosted.com [67.231.148.174]) by mails.dpdk.org (Postfix) with ESMTP id E8A12413AA for ; Tue, 5 Oct 2021 14:16:57 +0200 (CEST) Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 195CEXgA028469; Tue, 5 Oct 2021 05:16:55 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=pfpt0220; bh=PkNY1SrgCAyR04tPSJI0nfULlVISdDG6LeKTdZNm9PE=; b=DrRs/YBfCSWBuEm13y9k63KIrMvAW81IEBcIpd3uJu0T7nNy5n+Sur/f4GRinWAw5ara O3ZSP5OAOj18W0D0AlcRb00RHcfZL31/YLSoOXUeyvXKZBM/w6Wn261lo1nZAtR+la5t KbITaquXGFQbb2L/kkUZxGD7lDcP+PrSeAU6iYKIAXl7C2tgzJEEf6tBJjHTlXnelF41 olxHCn5nesbopE4AI93mlqWkiBC024rf6z0j1IWHP/lHJg+xoa4HTMf7/wnyCPs5f+hK CB2vndtF9AyPF932kXG9AwKefUBrGOkDl01TBjufa2tqwyTyRroqKv3lRTQWAfno/98B dA== Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0a-0016f401.pphosted.com with ESMTP id 3bgmv5rara-19 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Tue, 05 Oct 2021 05:16:54 -0700 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Tue, 5 Oct 2021 05:16:13 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server id 15.0.1497.18 via Frontend Transport; Tue, 5 Oct 2021 05:16:13 -0700 Received: from localhost.localdomain (unknown [10.29.52.211]) by maili.marvell.com (Postfix) with ESMTP id E588F3F707C; Tue, 5 Oct 2021 05:16:11 -0700 (PDT) From: Harman Kalra To: , Anatoly Burakov , Harman Kalra CC: , , Date: Tue, 5 Oct 2021 17:45:01 +0530 Message-ID: <20211005121502.66964-6-hkalra@marvell.com> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20211005121502.66964-1-hkalra@marvell.com> References: <20210826145726.102081-1-hkalra@marvell.com> <20211005121502.66964-1-hkalra@marvell.com> MIME-Version: 1.0 Content-Type: text/plain X-Proofpoint-GUID: XdrO8zmlDwP0iZ3r2vOnYmmNji7s25hf X-Proofpoint-ORIG-GUID: XdrO8zmlDwP0iZ3r2vOnYmmNji7s25hf X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.182.1,Aquarius:18.0.790,Hydra:6.0.391,FMLib:17.0.607.475 definitions=2021-10-05_01,2021-10-04_01,2020-04-07_01 Subject: [dpdk-dev] [PATCH v2 5/6] eal/interrupts: make interrupt handle structure opaque X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Moving interrupt handle structure definition inside the c file to make its fields totally opaque to the outside world. Dynamically allocating the efds and elist array os intr_handle structure, based on size provided by user. Eg size can be MSIX interrupts supported by a PCI device. Signed-off-by: Harman Kalra --- drivers/bus/pci/linux/pci_vfio.c | 7 + lib/eal/common/eal_common_interrupts.c | 189 ++++++++++++++++++++++++- lib/eal/include/meson.build | 1 - lib/eal/include/rte_eal_interrupts.h | 72 ---------- lib/eal/include/rte_interrupts.h | 24 +++- 5 files changed, 212 insertions(+), 81 deletions(-) delete mode 100644 lib/eal/include/rte_eal_interrupts.h diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c index c8da3e2fe8..f274aa4aab 100644 --- a/drivers/bus/pci/linux/pci_vfio.c +++ b/drivers/bus/pci/linux/pci_vfio.c @@ -266,6 +266,13 @@ pci_vfio_setup_interrupts(struct rte_pci_device *dev, int vfio_dev_fd) return -1; } + /* Reallocate the efds and elist fields of intr_handle based + * on PCI device MSIX size. + */ + if (rte_intr_event_list_update(dev->intr_handle, + irq.count)) + return -1; + /* if this vector cannot be used with eventfd, fail if we explicitly * specified interrupt type, otherwise continue */ if ((irq.flags & VFIO_IRQ_INFO_EVENTFD) == 0) { diff --git a/lib/eal/common/eal_common_interrupts.c b/lib/eal/common/eal_common_interrupts.c index 9b572a805f..a5311a0299 100644 --- a/lib/eal/common/eal_common_interrupts.c +++ b/lib/eal/common/eal_common_interrupts.c @@ -11,6 +11,29 @@ #include +struct rte_intr_handle { + RTE_STD_C11 + union { + struct { + /** VFIO/UIO cfg device file descriptor */ + int dev_fd; + int fd; /**< interrupt event file descriptor */ + }; + void *handle; /**< device driver handle (Windows) */ + }; + bool mem_allocator; + enum rte_intr_handle_type type; /**< handle type */ + uint32_t max_intr; /**< max interrupt requested */ + uint32_t nb_efd; /**< number of available efd(event fd) */ + uint8_t efd_counter_size; /**< size of efd counter, used for vdev */ + uint16_t nb_intr; + /**< Max vector count, default RTE_MAX_RXTX_INTR_VEC_ID */ + int *efds; /**< intr vectors/efds mapping */ + struct rte_epoll_event *elist; /**< intr vector epoll event */ + uint16_t vec_list_size; + int *intr_vec; /**< intr vector number array */ +}; + struct rte_intr_handle *rte_intr_instance_alloc(uint32_t flags) { @@ -29,15 +52,52 @@ struct rte_intr_handle *rte_intr_instance_alloc(uint32_t flags) return NULL; } + if (mem_allocator) + intr_handle->efds = rte_zmalloc(NULL, + RTE_MAX_RXTX_INTR_VEC_ID * + sizeof(uint32_t), 0); + else + intr_handle->efds = calloc(RTE_MAX_RXTX_INTR_VEC_ID, + sizeof(uint32_t)); + if (!intr_handle->efds) { + RTE_LOG(ERR, EAL, "Fail to allocate event fd list\n"); + rte_errno = ENOMEM; + goto fail; + } + + if (mem_allocator) + intr_handle->elist = + rte_zmalloc(NULL, RTE_MAX_RXTX_INTR_VEC_ID * + sizeof(struct rte_epoll_event), 0); + else + intr_handle->elist = calloc(RTE_MAX_RXTX_INTR_VEC_ID, + sizeof(struct rte_epoll_event)); + if (!intr_handle->elist) { + RTE_LOG(ERR, EAL, "fail to allocate event fd list\n"); + rte_errno = ENOMEM; + goto fail; + } intr_handle->nb_intr = RTE_MAX_RXTX_INTR_VEC_ID; intr_handle->mem_allocator = mem_allocator; return intr_handle; +fail: + if (intr_handle->mem_allocator) { + rte_free(intr_handle->efds); + rte_free(intr_handle); + } else { + free(intr_handle->efds); + free(intr_handle); + } + return NULL; } int rte_intr_instance_copy(struct rte_intr_handle *intr_handle, const struct rte_intr_handle *src) { + struct rte_epoll_event *tmp_elist; + int *tmp_efds; + if (intr_handle == NULL) { RTE_LOG(ERR, EAL, "Interrupt instance unallocated\n"); rte_errno = ENOTSUP; @@ -51,16 +111,51 @@ int rte_intr_instance_copy(struct rte_intr_handle *intr_handle, } intr_handle->fd = src->fd; - intr_handle->vfio_dev_fd = src->vfio_dev_fd; + intr_handle->dev_fd = src->dev_fd; intr_handle->type = src->type; + intr_handle->mem_allocator = src->mem_allocator; intr_handle->max_intr = src->max_intr; intr_handle->nb_efd = src->nb_efd; intr_handle->efd_counter_size = src->efd_counter_size; + if (intr_handle->nb_intr != src->nb_intr) { + if (src->mem_allocator) + tmp_efds = rte_realloc(intr_handle->efds, src->nb_intr * + sizeof(uint32_t), 0); + else + tmp_efds = realloc(intr_handle->efds, src->nb_intr * + sizeof(uint32_t)); + if (tmp_efds == NULL) { + RTE_LOG(ERR, EAL, "Failed to realloc the efds list"); + rte_errno = ENOMEM; + goto fail; + } + + if (src->mem_allocator) + tmp_elist = rte_realloc(intr_handle->elist, + src->nb_intr * + sizeof(struct rte_epoll_event), + 0); + else + tmp_elist = realloc(intr_handle->elist, src->nb_intr * + sizeof(struct rte_epoll_event)); + if (tmp_elist == NULL) { + RTE_LOG(ERR, EAL, "Failed to realloc the event list"); + rte_errno = ENOMEM; + goto up_efds; + } + + intr_handle->efds = tmp_efds; + intr_handle->elist = tmp_elist; + intr_handle->nb_intr = src->nb_intr; + } + memcpy(intr_handle->efds, src->efds, src->nb_intr); memcpy(intr_handle->elist, src->elist, src->nb_intr); return 0; +up_efds: + intr_handle->efds = tmp_efds; fail: return -rte_errno; } @@ -76,17 +171,77 @@ int rte_intr_instance_mem_allocator_get( return intr_handle->mem_allocator; } -void rte_intr_instance_free(struct rte_intr_handle *intr_handle) +int rte_intr_event_list_update(struct rte_intr_handle *intr_handle, + int size) { + struct rte_epoll_event *tmp_elist; + int *tmp_efds; + if (intr_handle == NULL) { RTE_LOG(ERR, EAL, "Interrupt instance unallocated\n"); rte_errno = ENOTSUP; + goto fail; + } + + if (size == 0) { + RTE_LOG(ERR, EAL, "Size can't be zero\n"); + rte_errno = EINVAL; + goto fail; } if (intr_handle->mem_allocator) - rte_free(intr_handle); + tmp_efds = rte_realloc(intr_handle->efds, size * + sizeof(uint32_t), 0); + else + tmp_efds = realloc(intr_handle->efds, size * + sizeof(uint32_t)); + if (tmp_efds == NULL) { + RTE_LOG(ERR, EAL, "Failed to realloc the efds list"); + rte_errno = ENOMEM; + goto fail; + } + + if (intr_handle->mem_allocator) + tmp_elist = rte_realloc(intr_handle->elist, size * + sizeof(struct rte_epoll_event), + 0); else + tmp_elist = realloc(intr_handle->elist, size * + sizeof(struct rte_epoll_event)); + if (tmp_elist == NULL) { + RTE_LOG(ERR, EAL, "Failed to realloc the event list"); + rte_errno = ENOMEM; + goto up_efds; + } + + intr_handle->efds = tmp_efds; + intr_handle->elist = tmp_elist; + intr_handle->nb_intr = size; + + return 0; +up_efds: + intr_handle->efds = tmp_efds; +fail: + return -rte_errno; +} + + +void rte_intr_instance_free(struct rte_intr_handle *intr_handle) +{ + if (intr_handle == NULL) { + RTE_LOG(ERR, EAL, "Interrupt instance unallocated\n"); + rte_errno = ENOTSUP; + } + + if (intr_handle->mem_allocator) { + rte_free(intr_handle->efds); + rte_free(intr_handle->elist); + rte_free(intr_handle); + } else { + free(intr_handle->efds); + free(intr_handle->elist); free(intr_handle); + } } int rte_intr_fd_set(struct rte_intr_handle *intr_handle, int fd) @@ -153,7 +308,7 @@ int rte_intr_dev_fd_set(struct rte_intr_handle *intr_handle, int fd) goto fail; } - intr_handle->vfio_dev_fd = fd; + intr_handle->dev_fd = fd; return 0; fail: @@ -168,7 +323,7 @@ int rte_intr_dev_fd_get(const struct rte_intr_handle *intr_handle) goto fail; } - return intr_handle->vfio_dev_fd; + return intr_handle->dev_fd; fail: return -1; } @@ -289,6 +444,12 @@ int rte_intr_efds_index_get(const struct rte_intr_handle *intr_handle, goto fail; } + if (!intr_handle->efds) { + RTE_LOG(ERR, EAL, "Event fd list not allocated\n"); + rte_errno = EFAULT; + goto fail; + } + if (index >= intr_handle->nb_intr) { RTE_LOG(ERR, EAL, "Invalid size %d, max limit %d\n", index, intr_handle->nb_intr); @@ -310,6 +471,12 @@ int rte_intr_efds_index_set(struct rte_intr_handle *intr_handle, goto fail; } + if (!intr_handle->efds) { + RTE_LOG(ERR, EAL, "Event fd list not allocated\n"); + rte_errno = EFAULT; + goto fail; + } + if (index >= intr_handle->nb_intr) { RTE_LOG(ERR, EAL, "Invalid size %d, max limit %d\n", index, intr_handle->nb_intr); @@ -333,6 +500,12 @@ struct rte_epoll_event *rte_intr_elist_index_get( goto fail; } + if (!intr_handle->elist) { + RTE_LOG(ERR, EAL, "Event list not allocated\n"); + rte_errno = ENOTSUP; + goto fail; + } + if (index >= intr_handle->nb_intr) { RTE_LOG(ERR, EAL, "Invalid size %d, max limit %d\n", index, intr_handle->nb_intr); @@ -354,6 +527,12 @@ int rte_intr_elist_index_set(struct rte_intr_handle *intr_handle, goto fail; } + if (!intr_handle->elist) { + RTE_LOG(ERR, EAL, "Event list not allocated\n"); + rte_errno = ENOTSUP; + goto fail; + } + if (index >= intr_handle->nb_intr) { RTE_LOG(ERR, EAL, "Invalid size %d, max limit %d\n", index, intr_handle->nb_intr); diff --git a/lib/eal/include/meson.build b/lib/eal/include/meson.build index 8e258607b8..86468d1a2b 100644 --- a/lib/eal/include/meson.build +++ b/lib/eal/include/meson.build @@ -49,7 +49,6 @@ headers += files( 'rte_version.h', 'rte_vfio.h', ) -indirect_headers += files('rte_eal_interrupts.h') # special case install the generic headers, since they go in a subdir generic_headers = files( diff --git a/lib/eal/include/rte_eal_interrupts.h b/lib/eal/include/rte_eal_interrupts.h deleted file mode 100644 index b01e987898..0000000000 --- a/lib/eal/include/rte_eal_interrupts.h +++ /dev/null @@ -1,72 +0,0 @@ -/* SPDX-License-Identifier: BSD-3-Clause - * Copyright(c) 2010-2014 Intel Corporation - */ - -#ifndef _RTE_INTERRUPTS_H_ -#error "don't include this file directly, please include generic " -#endif - -/** - * @file rte_eal_interrupts.h - * @internal - * - * Contains function prototypes exposed by the EAL for interrupt handling by - * drivers and other DPDK internal consumers. - */ - -#ifndef _RTE_EAL_INTERRUPTS_H_ -#define _RTE_EAL_INTERRUPTS_H_ - -#define RTE_MAX_RXTX_INTR_VEC_ID 512 -#define RTE_INTR_VEC_ZERO_OFFSET 0 -#define RTE_INTR_VEC_RXTX_OFFSET 1 - -/** - * The interrupt source type, e.g. UIO, VFIO, ALARM etc. - */ -enum rte_intr_handle_type { - RTE_INTR_HANDLE_UNKNOWN = 0, /**< generic unknown handle */ - RTE_INTR_HANDLE_UIO, /**< uio device handle */ - RTE_INTR_HANDLE_UIO_INTX, /**< uio generic handle */ - RTE_INTR_HANDLE_VFIO_LEGACY, /**< vfio device handle (legacy) */ - RTE_INTR_HANDLE_VFIO_MSI, /**< vfio device handle (MSI) */ - RTE_INTR_HANDLE_VFIO_MSIX, /**< vfio device handle (MSIX) */ - RTE_INTR_HANDLE_ALARM, /**< alarm handle */ - RTE_INTR_HANDLE_EXT, /**< external handler */ - RTE_INTR_HANDLE_VDEV, /**< virtual device */ - RTE_INTR_HANDLE_DEV_EVENT, /**< device event handle */ - RTE_INTR_HANDLE_VFIO_REQ, /**< VFIO request handle */ - RTE_INTR_HANDLE_MAX /**< count of elements */ -}; - -/** Handle for interrupts. */ -struct rte_intr_handle { - RTE_STD_C11 - union { - struct { - RTE_STD_C11 - union { - /** VFIO device file descriptor */ - int vfio_dev_fd; - /** UIO cfg file desc for uio_pci_generic */ - int uio_cfg_fd; - }; - int fd; /**< interrupt event file descriptor */ - }; - void *handle; /**< device driver handle (Windows) */ - }; - bool mem_allocator; - enum rte_intr_handle_type type; /**< handle type */ - uint32_t max_intr; /**< max interrupt requested */ - uint32_t nb_efd; /**< number of available efd(event fd) */ - uint8_t efd_counter_size; /**< size of efd counter, used for vdev */ - uint16_t nb_intr; - /**< Max vector count, default RTE_MAX_RXTX_INTR_VEC_ID */ - int efds[RTE_MAX_RXTX_INTR_VEC_ID]; /**< intr vectors/efds mapping */ - struct rte_epoll_event elist[RTE_MAX_RXTX_INTR_VEC_ID]; - /**< intr vector epoll event */ - uint16_t vec_list_size; - int *intr_vec; /**< intr vector number array */ -}; - -#endif /* _RTE_EAL_INTERRUPTS_H_ */ diff --git a/lib/eal/include/rte_interrupts.h b/lib/eal/include/rte_interrupts.h index 442b02de8f..367e739f08 100644 --- a/lib/eal/include/rte_interrupts.h +++ b/lib/eal/include/rte_interrupts.h @@ -33,9 +33,27 @@ struct rte_intr_handle; /** Allocate interrupt instance using DPDK memory management APIs */ #define RTE_INTR_ALLOC_DPDK_ALLOCATOR 0x00000001 -#define RTE_INTR_HANDLE_DEFAULT_SIZE 1 - -#include "rte_eal_interrupts.h" +#define RTE_MAX_RXTX_INTR_VEC_ID 512 +#define RTE_INTR_VEC_ZERO_OFFSET 0 +#define RTE_INTR_VEC_RXTX_OFFSET 1 + +/** + * The interrupt source type, e.g. UIO, VFIO, ALARM etc. + */ +enum rte_intr_handle_type { + RTE_INTR_HANDLE_UNKNOWN = 0, /**< generic unknown handle */ + RTE_INTR_HANDLE_UIO, /**< uio device handle */ + RTE_INTR_HANDLE_UIO_INTX, /**< uio generic handle */ + RTE_INTR_HANDLE_VFIO_LEGACY, /**< vfio device handle (legacy) */ + RTE_INTR_HANDLE_VFIO_MSI, /**< vfio device handle (MSI) */ + RTE_INTR_HANDLE_VFIO_MSIX, /**< vfio device handle (MSIX) */ + RTE_INTR_HANDLE_ALARM, /**< alarm handle */ + RTE_INTR_HANDLE_EXT, /**< external handler */ + RTE_INTR_HANDLE_VDEV, /**< virtual device */ + RTE_INTR_HANDLE_DEV_EVENT, /**< device event handle */ + RTE_INTR_HANDLE_VFIO_REQ, /**< VFIO request handle */ + RTE_INTR_HANDLE_MAX /**< count of elements */ +}; /** Function to be registered for the specific interrupt */ typedef void (*rte_intr_callback_fn)(void *cb_arg); -- 2.18.0