From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 0D7ADA0547; Fri, 29 Oct 2021 16:49:09 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id EC28A426FE; Fri, 29 Oct 2021 16:48:45 +0200 (CEST) Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2057.outbound.protection.outlook.com [40.107.237.57]) by mails.dpdk.org (Postfix) with ESMTP id E1BC4426F0 for ; Fri, 29 Oct 2021 16:48:43 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=IavaMTy7k2rvb8FyGBGcrUOn28if/2ZVHLNpYlcVlE8k0eBcl7L5CE5wCKAlDRtwxhXxd9prx4a8OFXI5kpXVy76IO811CnsNFz2UsNQlQ0V5jCEXWWLTnfkjVB1TLMw+jqMl33nlIdElQua8Z0OQBnVnJfn/CGA6WJo4UzubfZyH0BpPeIPwKy2JC0xZl1DmDdgdTQ8g6gJBrj17RV5TvyoRK8Yw/AKiU/Sf3ARS0MhFmkuEzWA3ylw9Wmykn1L4GD0gbQI3EkGWn71Xi6TiYESSryVbGw18Q5HEdNHwe3Zehsysi+xlM6vSJWpgds8TPqj+bgArVI844v1E98NEg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=nOrQ0wW0CYERpc0fatCX92QWP0y2ZssMEfS6Mqoz15Y=; b=XwogFLKza/3XV1Yh9bGlUjH9UNX2zni8U2XRAh+OJLAxM/4P5x/ViIp2xmnlOf9JrlXEt8eK4V7nx4bkMbTdCuoOfnPv5vFRy6JMhka8M/yLlIZRPWTiVQwZgFsyBs0rtEkVSoEij4seiHeMfN0TEv0Buk4gE8Q1oAboPFslZFSrCsbZTGA03Q1KaAtXnXsqHtjqk40oZH7jtu7NdqR5cbhbgSFSLoPlAMEIJ+iGsuELNZ05z9iBXkzdFI7TZMC7rRpwtMLjHx+89cEwZxM3rqQ52bUScvTcYdZ1kYEIL+Kk4SbCGQlFPwsBg2Rg3R3pV1nvD0kFB4eXA5MwZ/QoZg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 149.199.62.198) smtp.rcpttodomain=dpdk.org smtp.mailfrom=xilinx.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=xilinx.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=xilinx.onmicrosoft.com; s=selector2-xilinx-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=nOrQ0wW0CYERpc0fatCX92QWP0y2ZssMEfS6Mqoz15Y=; b=iSn5NcqhJLgdYaTW0Ctq5SJ9Ci0jvCDHxjk4XT8uuloTowcNmesS3AcDCL2JSY2WdCy8tkEuiewQJPbcl3mUkuuTBSusKKhgZeNagcL9IRQVh1k2uUTvduAOqmg7uFIgqEgaf0IggiFMLfUOneRiceewq0GqUjV4jEh5JQEEAYQ= Received: from DM5PR05CA0021.namprd05.prod.outlook.com (2603:10b6:3:d4::31) by MN2PR02MB5903.namprd02.prod.outlook.com (2603:10b6:208:116::32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4628.18; Fri, 29 Oct 2021 14:48:41 +0000 Received: from DM3NAM02FT054.eop-nam02.prod.protection.outlook.com (2603:10b6:3:d4:cafe::39) by DM5PR05CA0021.outlook.office365.com (2603:10b6:3:d4::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4669.4 via Frontend Transport; Fri, 29 Oct 2021 14:48:41 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 149.199.62.198) smtp.mailfrom=xilinx.com; dpdk.org; dkim=none (message not signed) header.d=none;dpdk.org; dmarc=pass action=none header.from=xilinx.com; Received-SPF: Pass (protection.outlook.com: domain of xilinx.com designates 149.199.62.198 as permitted sender) receiver=protection.outlook.com; client-ip=149.199.62.198; helo=xsj-pvapexch02.xlnx.xilinx.com; Received: from xsj-pvapexch02.xlnx.xilinx.com (149.199.62.198) by DM3NAM02FT054.mail.protection.outlook.com (10.13.5.135) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.4649.14 via Frontend Transport; Fri, 29 Oct 2021 14:48:41 +0000 Received: from xsj-pvapexch02.xlnx.xilinx.com (172.19.86.41) by xsj-pvapexch02.xlnx.xilinx.com (172.19.86.41) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2176.14; Fri, 29 Oct 2021 07:48:39 -0700 Received: from smtp.xilinx.com (172.19.127.96) by xsj-pvapexch02.xlnx.xilinx.com (172.19.86.41) with Microsoft SMTP Server id 15.1.2176.14 via Frontend Transport; Fri, 29 Oct 2021 07:48:39 -0700 Envelope-to: dev@dpdk.org, maxime.coquelin@redhat.com, chenbo.xia@intel.com, andrew.rybchenko@oktetlabs.ru Received: from [10.170.66.108] (port=36706 helo=xndengvm004108.xilinx.com) by smtp.xilinx.com with esmtp (Exim 4.90) (envelope-from ) id 1mgTBP-0008YO-D4; Fri, 29 Oct 2021 07:48:39 -0700 From: Vijay Srivastava To: CC: , , , Vijay Kumar Srivastava Date: Fri, 29 Oct 2021 20:16:42 +0530 Message-ID: <20211029144645.30295-8-vsrivast@xilinx.com> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20211029144645.30295-1-vsrivast@xilinx.com> References: <20210706164418.32615-1-vsrivast@xilinx.com> <20211029144645.30295-1-vsrivast@xilinx.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 1cd1b0bb-39eb-4be1-c610-08d99aeb334f X-MS-TrafficTypeDiagnostic: MN2PR02MB5903: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:3044; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 5v9BYgTIZnhCJSK+rprnc8PnkDslkA+nWaVdFkGLizod9L1N/Z/lQsUtl/HF+ifOygpwvxN6l4p/VpfiBa5Ow/R4BFKqyTnE2/263p4dE/lUkfEapFyR6I22kMRtgcheyFVxGiT/LfalVgl1txcpOFtzPZIg9VTTj9v6bu7ad/7d3dCqfd6oAmMZyW6fztqkUheRRjxk7iKEpPbJRSMoveuvMjISr7kmnBWnJN6fU/aI02X4tDLOY6m4lwey6lthsVstFXXB5qSC+r+KjQ885CtlmR1P/8R6rVm2TUPtM/JQRWKJZuYUG+2VBKPy5ehjyn9MZq1NxfWx1tia89DeOqJQDWCs4ws3BmUmwOn4cUjCEtIxtdTFtUpfLTwJVZI4sCJeQVGvanq3X22phX26CAUkj0WSCCSyWZBEd2N8AXsgp1NYgBTYEAlLjBayWzC+vmerhwhjs+PUZDlToIDz3ODrAxBYNMzL6cBB7ztm3SQU83nMwdgg4j4EfDUaVBkBqSsFDe7y2CZJUXFCEY/y5kR+mOVC+n7t77svDfFOqCmj/yV0LpGC5MHAQHL8OEnVENOmy5W0dYbLEo4DndwpjkR0lJPMmvspxXuuA8zlYXOPxN+Zus2vePkXAoGvEcb2I8mEfqAPYua2Uo9SC08iJWTfvfFL7Lk9dARshddvsGagCO2o8D4H4XiTrO+lG4K3mVQHmGlHIjH00+sDouWttHTmuYjvbPyIuXygzVbWgJQ= X-Forefront-Antispam-Report: CIP:149.199.62.198; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:xsj-pvapexch02.xlnx.xilinx.com; PTR:unknown-62-198.xilinx.com; CAT:NONE; SFS:(36840700001)(46966006)(36860700001)(8936002)(186003)(426003)(4326008)(7696005)(8676002)(5660300002)(336012)(508600001)(316002)(47076005)(54906003)(6916009)(2616005)(36906005)(107886003)(7636003)(70586007)(70206006)(26005)(83380400001)(36756003)(82310400003)(6666004)(9786002)(1076003)(2906002)(356005)(44832011)(102446001); DIR:OUT; SFP:1101; X-OriginatorOrg: xilinx.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Oct 2021 14:48:41.7584 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 1cd1b0bb-39eb-4be1-c610-08d99aeb334f X-MS-Exchange-CrossTenant-Id: 657af505-d5df-48d0-8300-c31994686c5c X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=657af505-d5df-48d0-8300-c31994686c5c; Ip=[149.199.62.198]; Helo=[xsj-pvapexch02.xlnx.xilinx.com] X-MS-Exchange-CrossTenant-AuthSource: DM3NAM02FT054.eop-nam02.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR02MB5903 Subject: [dpdk-dev] [PATCH v3 07/10] vdpa/sfc: add support to get queue notify area info X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Vijay Kumar Srivastava Implement the vDPA ops get_notify_area to get the notify area info of the queue. Signed-off-by: Vijay Kumar Srivastava Acked-by: Andrew Rybchenko --- v2: * Added error log in sfc_vdpa_get_notify_area. drivers/vdpa/sfc/sfc_vdpa_ops.c | 168 ++++++++++++++++++++++++++++++++++++++-- drivers/vdpa/sfc/sfc_vdpa_ops.h | 2 + 2 files changed, 164 insertions(+), 6 deletions(-) diff --git a/drivers/vdpa/sfc/sfc_vdpa_ops.c b/drivers/vdpa/sfc/sfc_vdpa_ops.c index de1c81a..774d73e 100644 --- a/drivers/vdpa/sfc/sfc_vdpa_ops.c +++ b/drivers/vdpa/sfc/sfc_vdpa_ops.c @@ -3,6 +3,8 @@ * Copyright(c) 2020-2021 Xilinx, Inc. */ +#include +#include #include #include @@ -537,6 +539,67 @@ return 0; } +static void * +sfc_vdpa_notify_ctrl(void *arg) +{ + struct sfc_vdpa_ops_data *ops_data; + int vid; + + ops_data = arg; + if (ops_data == NULL) + return NULL; + + sfc_vdpa_adapter_lock(ops_data->dev_handle); + + vid = ops_data->vid; + + if (rte_vhost_host_notifier_ctrl(vid, RTE_VHOST_QUEUE_ALL, true) != 0) + sfc_vdpa_info(ops_data->dev_handle, + "vDPA (%s): Notifier could not get configured", + ops_data->vdpa_dev->device->name); + + sfc_vdpa_adapter_unlock(ops_data->dev_handle); + + return NULL; +} + +static int +sfc_vdpa_setup_notify_ctrl(int vid) +{ + int ret; + struct rte_vdpa_device *vdpa_dev; + struct sfc_vdpa_ops_data *ops_data; + + vdpa_dev = rte_vhost_get_vdpa_device(vid); + + ops_data = sfc_vdpa_get_data_by_dev(vdpa_dev); + if (ops_data == NULL) { + sfc_vdpa_err(ops_data->dev_handle, + "invalid vDPA device : %p, vid : %d", + vdpa_dev, vid); + return -1; + } + + ops_data->is_notify_thread_started = false; + + /* + * Use rte_vhost_host_notifier_ctrl in a thread to avoid + * dead lock scenario when multiple VFs are used in single vdpa + * application and multiple VFs are passed to a single VM. + */ + ret = pthread_create(&ops_data->notify_tid, NULL, + sfc_vdpa_notify_ctrl, ops_data); + if (ret != 0) { + sfc_vdpa_err(ops_data->dev_handle, + "failed to create notify_ctrl thread: %s", + rte_strerror(ret)); + return -1; + } + ops_data->is_notify_thread_started = true; + + return 0; +} + static int sfc_vdpa_dev_config(int vid) { @@ -570,18 +633,19 @@ if (rc != 0) goto fail_vdpa_start; - sfc_vdpa_adapter_unlock(ops_data->dev_handle); + rc = sfc_vdpa_setup_notify_ctrl(vid); + if (rc != 0) + goto fail_vdpa_notify; - sfc_vdpa_log_init(ops_data->dev_handle, "vhost notifier ctrl"); - if (rte_vhost_host_notifier_ctrl(vid, RTE_VHOST_QUEUE_ALL, true) != 0) - sfc_vdpa_info(ops_data->dev_handle, - "vDPA (%s): software relay for notify is used.", - vdpa_dev->device->name); + sfc_vdpa_adapter_unlock(ops_data->dev_handle); sfc_vdpa_log_init(ops_data->dev_handle, "done"); return 0; +fail_vdpa_notify: + sfc_vdpa_stop(ops_data); + fail_vdpa_start: sfc_vdpa_close(ops_data); @@ -594,6 +658,7 @@ static int sfc_vdpa_dev_close(int vid) { + int ret; struct rte_vdpa_device *vdpa_dev; struct sfc_vdpa_ops_data *ops_data; @@ -608,6 +673,23 @@ } sfc_vdpa_adapter_lock(ops_data->dev_handle); + if (ops_data->is_notify_thread_started == true) { + void *status; + ret = pthread_cancel(ops_data->notify_tid); + if (ret != 0) { + sfc_vdpa_err(ops_data->dev_handle, + "failed to cancel notify_ctrl thread: %s", + rte_strerror(ret)); + } + + ret = pthread_join(ops_data->notify_tid, &status); + if (ret != 0) { + sfc_vdpa_err(ops_data->dev_handle, + "failed to join terminated notify_ctrl thread: %s", + rte_strerror(ret)); + } + } + ops_data->is_notify_thread_started = false; sfc_vdpa_stop(ops_data); sfc_vdpa_close(ops_data); @@ -658,6 +740,79 @@ return vfio_dev_fd; } +static int +sfc_vdpa_get_notify_area(int vid, int qid, uint64_t *offset, uint64_t *size) +{ + int ret; + efx_nic_t *nic; + int vfio_dev_fd; + efx_rc_t rc; + unsigned int bar_offset; + struct rte_vdpa_device *vdpa_dev; + struct sfc_vdpa_ops_data *ops_data; + struct vfio_region_info reg = { .argsz = sizeof(reg) }; + const efx_nic_cfg_t *encp; + int max_vring_cnt; + int64_t len; + void *dev; + + vdpa_dev = rte_vhost_get_vdpa_device(vid); + + ops_data = sfc_vdpa_get_data_by_dev(vdpa_dev); + if (ops_data == NULL) + return -1; + + dev = ops_data->dev_handle; + + vfio_dev_fd = sfc_vdpa_adapter_by_dev_handle(dev)->vfio_dev_fd; + max_vring_cnt = + (sfc_vdpa_adapter_by_dev_handle(dev)->max_queue_count * 2); + + nic = sfc_vdpa_adapter_by_dev_handle(ops_data->dev_handle)->nic; + encp = efx_nic_cfg_get(nic); + + if (qid >= max_vring_cnt) { + sfc_vdpa_err(dev, "invalid qid : %d", qid); + return -1; + } + + if (ops_data->vq_cxt[qid].enable != B_TRUE) { + sfc_vdpa_err(dev, "vq is not enabled"); + return -1; + } + + rc = efx_virtio_get_doorbell_offset(ops_data->vq_cxt[qid].vq, + &bar_offset); + if (rc != 0) { + sfc_vdpa_err(dev, "failed to get doorbell offset: %s", + rte_strerror(rc)); + return rc; + } + + reg.index = sfc_vdpa_adapter_by_dev_handle(dev)->mem_bar.esb_rid; + ret = ioctl(vfio_dev_fd, VFIO_DEVICE_GET_REGION_INFO, ®); + if (ret != 0) { + sfc_vdpa_err(dev, "could not get device region info: %s", + strerror(errno)); + return ret; + } + + *offset = reg.offset + bar_offset; + + len = (1U << encp->enc_vi_window_shift) / 2; + if (len >= sysconf(_SC_PAGESIZE)) { + *size = sysconf(_SC_PAGESIZE); + } else { + sfc_vdpa_err(dev, "invalid VI window size : 0x%" PRIx64, len); + return -1; + } + + sfc_vdpa_info(dev, "vDPA ops get_notify_area :: offset : 0x%" PRIx64, + *offset); + + return 0; +} + static struct rte_vdpa_dev_ops sfc_vdpa_ops = { .get_queue_num = sfc_vdpa_get_queue_num, .get_features = sfc_vdpa_get_features, @@ -667,6 +822,7 @@ .set_vring_state = sfc_vdpa_set_vring_state, .set_features = sfc_vdpa_set_features, .get_vfio_device_fd = sfc_vdpa_get_vfio_device_fd, + .get_notify_area = sfc_vdpa_get_notify_area, }; struct sfc_vdpa_ops_data * diff --git a/drivers/vdpa/sfc/sfc_vdpa_ops.h b/drivers/vdpa/sfc/sfc_vdpa_ops.h index 8d553c5..f7523ef 100644 --- a/drivers/vdpa/sfc/sfc_vdpa_ops.h +++ b/drivers/vdpa/sfc/sfc_vdpa_ops.h @@ -50,6 +50,8 @@ struct sfc_vdpa_ops_data { struct rte_vdpa_device *vdpa_dev; enum sfc_vdpa_context vdpa_context; enum sfc_vdpa_state state; + pthread_t notify_tid; + bool is_notify_thread_started; uint64_t dev_features; uint64_t drv_features; -- 1.8.3.1