From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 7FF58A0032; Sat, 18 Jun 2022 11:04:04 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 20B5C42B73; Sat, 18 Jun 2022 11:03:40 +0200 (CEST) Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2081.outbound.protection.outlook.com [40.107.237.81]) by mails.dpdk.org (Postfix) with ESMTP id 38A8C42B79 for ; Sat, 18 Jun 2022 11:03:38 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=eCWVemkYHyMdcAAx1WBqyTfk/WlbvyqEMg2KCLQPPrnjvIVmOpKp1B80q0GRSVFdgsneXB8VlXfihbuidO5fAPs20u1/MthgI88G/M61/G+ue4+EiluUjlArQ1HBOeWLjtDv22GuL8FQd6Rq24zs50jU7gdU2YHpdhbxkcanswe4E6QXRMCfGcqJF4dbJxRyFvm0wjd9fa6U8gsdkvHJvEF2ZuuqGAfS86pQUtuVPMdL+YK3CO4uPn/Ix2zYxJty9OBmz8N2PBcQLb9SYcRW6bRP7dSLDmBTSbyQmBa0prxfA3mxhEMhbikoi5mI/l1AN9rjFJPhSMZ6UG64uGiY7g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=4UCgtUYYYSAazyjejbn9a1cuUy6Faq+X456aEYCkhX0=; b=bll03GBGJRGQ79vTH95HKUGWKayja+dFMyy/s/N7imFldrI9vdmQMUZrseBgn/pOZw4oRJkAHVLERNpPrU57BhyhMKxvlgBN6MCqmEoB22Lit8hLiSgFvmiayifZq/VMwa+uPjRpbaFBDr3bdQT/IvgFqIBnWaU4Rb5rYJ6mMbg6HllTAHxsjcP9GMDMJTttC8i8zQgpgw2f9E2AUlr6VWgGAIVs2nkrTvPX+GkUz9TOuHqINwy3b4E20gKqCscjDTjo9uSEfQcXhfq9+xb4ik0dazYbXaVfmx/3soo7Y5wWAbfsLVjnZhXOIB8ZwiNwR1EUUa7gSkdXr6qNOG/tUw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 12.22.5.235) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=4UCgtUYYYSAazyjejbn9a1cuUy6Faq+X456aEYCkhX0=; b=ITHbk97RVOb4j9k8J3f9aOBciVff7uYcCDk2cGy+FUrfEBcjTDS+CILtuModWevSoFEjdtW5i9t3wwP+2F+GWAhawZqXBGrvrxjo30zEcvHV02iWpRj36d/vYow3zevIzzQkMYnF/oGIT2lNc4I/1NfHaQ5UFpKHj+90bsxKo3YjeCyHebXgmHLaf/pftC+7ySvl2fL5HUvgPEZ88u/w1sX0jXCkf2EjMaPpYN4E9AeKXqeGN8j6HvnErYUUTbYidzrY14wbANipjkuce8bnSHpMtBkjMk7CruZw+X4p8OZPUStPJ71sVZGe2WwE36fTCPzZ4F9eGou2qkWBP8XpmQ== Received: from MW4PR04CA0245.namprd04.prod.outlook.com (2603:10b6:303:88::10) by CY4PR12MB1496.namprd12.prod.outlook.com (2603:10b6:910:10::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5353.15; Sat, 18 Jun 2022 09:03:35 +0000 Received: from CO1NAM11FT018.eop-nam11.prod.protection.outlook.com (2603:10b6:303:88:cafe::c3) by MW4PR04CA0245.outlook.office365.com (2603:10b6:303:88::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5353.18 via Frontend Transport; Sat, 18 Jun 2022 09:03:35 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 12.22.5.235) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 12.22.5.235 as permitted sender) receiver=protection.outlook.com; client-ip=12.22.5.235; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (12.22.5.235) by CO1NAM11FT018.mail.protection.outlook.com (10.13.175.16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.5353.14 via Frontend Transport; Sat, 18 Jun 2022 09:03:34 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by DRHQMAIL107.nvidia.com (10.27.9.16) with Microsoft SMTP Server (TLS) id 15.0.1497.32; Sat, 18 Jun 2022 09:03:34 +0000 Received: from nvidia.com (10.126.231.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.22; Sat, 18 Jun 2022 02:03:31 -0700 From: Li Zhang To: , , , CC: , , , , Subject: [PATCH v4 06/15] vdpa/mlx5: pre-create virtq at probe time Date: Sat, 18 Jun 2022 12:02:49 +0300 Message-ID: <20220618090258.91157-7-lizh@nvidia.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220618090258.91157-1-lizh@nvidia.com> References: <20220408075606.33056-1-lizh@nvidia.com> <20220618090258.91157-1-lizh@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [10.126.231.35] X-ClientProxiedBy: rnnvmail203.nvidia.com (10.129.68.9) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 926123bc-92e8-451b-d32d-08da51096ced X-MS-TrafficTypeDiagnostic: CY4PR12MB1496:EE_ X-LD-Processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 1IruSXZMLFY4MhojWC9EOUNCuzWb4UoZ14OIByG7mT97rem2Ss5eVoW+U0I8dp0wdP0csyaZOF1GWKFvroMYw6vFwf/a3EljxJunfWyf/OR5Ie7ojwArUjkNRka5FGn40hH0EDMCTBYAEe3p/WJc6hO/dQRu8GqbwCqUcjbBsQL2ylewMlaaDAx4xYDb6PQttCfaa/ZEDS/mXG2DVasSLXUtHKP0f01EgdnS1KDeSwLdcyrKtt9zBHFJOWcTMfnVPtllh+gKoiVUgV10ygnHuCPJ13Bp2gXFGw3ZmjCfVWGDj3e6NeoeOqEQYTPM/PIALEtCwXFr5fBTUsIuaV7jR5OfGthyy6wzXnGQecXTX3sEHYYoHwgvREkV42D1UlSenhZQ4numRDLHig7MYPTAcYSt8QQU9Zn0gMDtcuMS0fdGcFBYcOgAbffLqdms/h+rmwehgKSNTauoJFrlMbJM8x/eBOzTBSkUYyfcwlLrmhIpnr+ZxjxkxUTna7m6NZLO4l2qC2ifSTPvH2T3F7gQC3YzrrlU4XLhW4l0wWSb5z00vUkE4MSOichHk928SvCnrBadJitOnMVR4KX7RSAftMEGIp3py2VbQX+adn3XHxWCHQXWcOoA5fXjbd8EOjZ+5vE8gbQxpUlYkeWlEwpcC+6cautF8z+WkRGtyySTH2giwNTzU10GUSqvX7e5Dj7hMCbRIvgSjQOkwDaESmioYQ== X-Forefront-Antispam-Report: CIP:12.22.5.235; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:InfoNoRecords; CAT:NONE; SFS:(13230016)(4636009)(40470700004)(46966006)(36840700001)(70206006)(70586007)(356005)(47076005)(82310400005)(86362001)(1076003)(6666004)(4326008)(8676002)(498600001)(36860700001)(110136005)(336012)(426003)(316002)(2616005)(7696005)(54906003)(186003)(26005)(30864003)(16526019)(40460700003)(6636002)(6286002)(55016003)(8936002)(5660300002)(36756003)(2906002)(81166007)(83380400001)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Jun 2022 09:03:34.9021 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 926123bc-92e8-451b-d32d-08da51096ced X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[12.22.5.235]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT018.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR12MB1496 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org dev_config operation is called in LM progress. LM time is very critical because all the VM packets are dropped directly at that time. Move the virtq creation to probe time and only modify the configuration later in the dev_config stage using the new ability to modify virtq. This optimization accelerates the LM process and reduces its time by 70%. Signed-off-by: Li Zhang Acked-by: Matan Azrad Reviewed-by: Maxime Coquelin --- doc/guides/rel_notes/release_22_07.rst | 4 + drivers/vdpa/mlx5/mlx5_vdpa.h | 4 + drivers/vdpa/mlx5/mlx5_vdpa_lm.c | 19 +- drivers/vdpa/mlx5/mlx5_vdpa_virtq.c | 257 +++++++++++++++---------- 4 files changed, 176 insertions(+), 108 deletions(-) diff --git a/doc/guides/rel_notes/release_22_07.rst b/doc/guides/rel_notes/release_22_07.rst index f2cf41def9..2056cd9ee7 100644 --- a/doc/guides/rel_notes/release_22_07.rst +++ b/doc/guides/rel_notes/release_22_07.rst @@ -175,6 +175,10 @@ New Features This is a fall-back implementation for platforms that don't support vector operations. +* **Updated Nvidia mlx5 vDPA driver.** + + * Added new devargs ``queue_size`` and ``queues`` to allow prior creation of virtq resources. + Removed Items ------------- diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.h b/drivers/vdpa/mlx5/mlx5_vdpa.h index bf82026e37..e5553079fe 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa.h +++ b/drivers/vdpa/mlx5/mlx5_vdpa.h @@ -80,6 +80,7 @@ struct mlx5_vdpa_virtq { uint16_t vq_size; uint8_t notifier_state; bool stopped; + uint32_t configured:1; uint32_t version; struct mlx5_vdpa_priv *priv; struct mlx5_devx_obj *virtq; @@ -489,4 +490,7 @@ mlx5_vdpa_virtq_stats_reset(struct mlx5_vdpa_priv *priv, int qid); */ void mlx5_vdpa_drain_cq(struct mlx5_vdpa_priv *priv); + +bool +mlx5_vdpa_is_modify_virtq_supported(struct mlx5_vdpa_priv *priv); #endif /* RTE_PMD_MLX5_VDPA_H_ */ diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_lm.c b/drivers/vdpa/mlx5/mlx5_vdpa_lm.c index 43a2b98255..284758ad56 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa_lm.c +++ b/drivers/vdpa/mlx5/mlx5_vdpa_lm.c @@ -12,20 +12,21 @@ int mlx5_vdpa_logging_enable(struct mlx5_vdpa_priv *priv, int enable) { struct mlx5_devx_virtq_attr attr = { - .type = MLX5_VIRTQ_MODIFY_TYPE_DIRTY_BITMAP_DUMP_ENABLE, + .mod_fields_bitmap = + MLX5_VIRTQ_MODIFY_TYPE_DIRTY_BITMAP_DUMP_ENABLE, .dirty_bitmap_dump_enable = enable, }; + struct mlx5_vdpa_virtq *virtq; int i; for (i = 0; i < priv->nr_virtqs; ++i) { attr.queue_index = i; - if (!priv->virtqs[i].virtq) { - DRV_LOG(DEBUG, "virtq %d is invalid for dirty bitmap " - "enabling.", i); + virtq = &priv->virtqs[i]; + if (!virtq->configured) { + DRV_LOG(DEBUG, "virtq %d is invalid for dirty bitmap enabling.", i); } else if (mlx5_devx_cmd_modify_virtq(priv->virtqs[i].virtq, &attr)) { - DRV_LOG(ERR, "Failed to modify virtq %d for dirty " - "bitmap enabling.", i); + DRV_LOG(ERR, "Failed to modify virtq %d for dirty bitmap enabling.", i); return -1; } } @@ -37,10 +38,11 @@ mlx5_vdpa_dirty_bitmap_set(struct mlx5_vdpa_priv *priv, uint64_t log_base, uint64_t log_size) { struct mlx5_devx_virtq_attr attr = { - .type = MLX5_VIRTQ_MODIFY_TYPE_DIRTY_BITMAP_PARAMS, + .mod_fields_bitmap = MLX5_VIRTQ_MODIFY_TYPE_DIRTY_BITMAP_PARAMS, .dirty_bitmap_addr = log_base, .dirty_bitmap_size = log_size, }; + struct mlx5_vdpa_virtq *virtq; int i; int ret = mlx5_os_wrapped_mkey_create(priv->cdev->ctx, priv->cdev->pd, priv->cdev->pdn, @@ -54,7 +56,8 @@ mlx5_vdpa_dirty_bitmap_set(struct mlx5_vdpa_priv *priv, uint64_t log_base, attr.dirty_bitmap_mkey = priv->lm_mr.lkey; for (i = 0; i < priv->nr_virtqs; ++i) { attr.queue_index = i; - if (!priv->virtqs[i].virtq) { + virtq = &priv->virtqs[i]; + if (!virtq->configured) { DRV_LOG(DEBUG, "virtq %d is invalid for LM.", i); } else if (mlx5_devx_cmd_modify_virtq(priv->virtqs[i].virtq, &attr)) { diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c b/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c index 6637ba1503..6e08d619e4 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c +++ b/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c @@ -75,6 +75,7 @@ mlx5_vdpa_virtqs_cleanup(struct mlx5_vdpa_priv *priv) for (i = 0; i < priv->caps.max_num_virtio_queues; i++) { struct mlx5_vdpa_virtq *virtq = &priv->virtqs[i]; + virtq->configured = 0; for (j = 0; j < RTE_DIM(virtq->umems); ++j) { if (virtq->umems[j].obj) { claim_zero(mlx5_glue->devx_umem_dereg @@ -111,11 +112,12 @@ mlx5_vdpa_virtq_unset(struct mlx5_vdpa_virtq *virtq) rte_intr_fd_set(virtq->intr_handle, -1); } rte_intr_instance_free(virtq->intr_handle); - if (virtq->virtq) { + if (virtq->configured) { ret = mlx5_vdpa_virtq_stop(virtq->priv, virtq->index); if (ret) DRV_LOG(WARNING, "Failed to stop virtq %d.", virtq->index); + virtq->configured = 0; claim_zero(mlx5_devx_cmd_destroy(virtq->virtq)); } virtq->virtq = NULL; @@ -138,7 +140,7 @@ int mlx5_vdpa_virtq_modify(struct mlx5_vdpa_virtq *virtq, int state) { struct mlx5_devx_virtq_attr attr = { - .type = MLX5_VIRTQ_MODIFY_TYPE_STATE, + .mod_fields_bitmap = MLX5_VIRTQ_MODIFY_TYPE_STATE, .state = state ? MLX5_VIRTQ_STATE_RDY : MLX5_VIRTQ_STATE_SUSPEND, .queue_index = virtq->index, @@ -153,7 +155,7 @@ mlx5_vdpa_virtq_stop(struct mlx5_vdpa_priv *priv, int index) struct mlx5_vdpa_virtq *virtq = &priv->virtqs[index]; int ret; - if (virtq->stopped) + if (virtq->stopped || !virtq->configured) return 0; ret = mlx5_vdpa_virtq_modify(virtq, 0); if (ret) @@ -209,51 +211,54 @@ mlx5_vdpa_hva_to_gpa(struct rte_vhost_memory *mem, uint64_t hva) } static int -mlx5_vdpa_virtq_setup(struct mlx5_vdpa_priv *priv, int index) +mlx5_vdpa_virtq_sub_objs_prepare(struct mlx5_vdpa_priv *priv, + struct mlx5_devx_virtq_attr *attr, + struct rte_vhost_vring *vq, int index) { struct mlx5_vdpa_virtq *virtq = &priv->virtqs[index]; - struct rte_vhost_vring vq; - struct mlx5_devx_virtq_attr attr = {0}; uint64_t gpa; int ret; unsigned int i; - uint16_t last_avail_idx; - uint16_t last_used_idx; - uint16_t event_num = MLX5_EVENT_TYPE_OBJECT_CHANGE; - uint64_t cookie; - - ret = rte_vhost_get_vhost_vring(priv->vid, index, &vq); - if (ret) - return -1; - if (vq.size == 0) - return 0; - virtq->index = index; - virtq->vq_size = vq.size; - attr.tso_ipv4 = !!(priv->features & (1ULL << VIRTIO_NET_F_HOST_TSO4)); - attr.tso_ipv6 = !!(priv->features & (1ULL << VIRTIO_NET_F_HOST_TSO6)); - attr.tx_csum = !!(priv->features & (1ULL << VIRTIO_NET_F_CSUM)); - attr.rx_csum = !!(priv->features & (1ULL << VIRTIO_NET_F_GUEST_CSUM)); - attr.virtio_version_1_0 = !!(priv->features & (1ULL << - VIRTIO_F_VERSION_1)); - attr.type = (priv->features & (1ULL << VIRTIO_F_RING_PACKED)) ? + uint16_t last_avail_idx = 0; + uint16_t last_used_idx = 0; + + if (virtq->virtq) + attr->mod_fields_bitmap = MLX5_VIRTQ_MODIFY_TYPE_STATE | + MLX5_VIRTQ_MODIFY_TYPE_ADDR | + MLX5_VIRTQ_MODIFY_TYPE_HW_AVAILABLE_INDEX | + MLX5_VIRTQ_MODIFY_TYPE_HW_USED_INDEX | + MLX5_VIRTQ_MODIFY_TYPE_VERSION_1_0 | + MLX5_VIRTQ_MODIFY_TYPE_Q_TYPE | + MLX5_VIRTQ_MODIFY_TYPE_Q_MKEY | + MLX5_VIRTQ_MODIFY_TYPE_QUEUE_FEATURE_BIT_MASK | + MLX5_VIRTQ_MODIFY_TYPE_EVENT_MODE; + attr->tso_ipv4 = !!(priv->features & (1ULL << VIRTIO_NET_F_HOST_TSO4)); + attr->tso_ipv6 = !!(priv->features & (1ULL << VIRTIO_NET_F_HOST_TSO6)); + attr->tx_csum = !!(priv->features & (1ULL << VIRTIO_NET_F_CSUM)); + attr->rx_csum = !!(priv->features & (1ULL << VIRTIO_NET_F_GUEST_CSUM)); + attr->virtio_version_1_0 = + !!(priv->features & (1ULL << VIRTIO_F_VERSION_1)); + attr->q_type = + (priv->features & (1ULL << VIRTIO_F_RING_PACKED)) ? MLX5_VIRTQ_TYPE_PACKED : MLX5_VIRTQ_TYPE_SPLIT; /* * No need event QPs creation when the guest in poll mode or when the * capability allows it. */ - attr.event_mode = vq.callfd != -1 || !(priv->caps.event_mode & (1 << - MLX5_VIRTQ_EVENT_MODE_NO_MSIX)) ? - MLX5_VIRTQ_EVENT_MODE_QP : - MLX5_VIRTQ_EVENT_MODE_NO_MSIX; - if (attr.event_mode == MLX5_VIRTQ_EVENT_MODE_QP) { - ret = mlx5_vdpa_event_qp_prepare(priv, vq.size, vq.callfd, - &virtq->eqp); + attr->event_mode = vq->callfd != -1 || + !(priv->caps.event_mode & (1 << MLX5_VIRTQ_EVENT_MODE_NO_MSIX)) ? + MLX5_VIRTQ_EVENT_MODE_QP : MLX5_VIRTQ_EVENT_MODE_NO_MSIX; + if (attr->event_mode == MLX5_VIRTQ_EVENT_MODE_QP) { + ret = mlx5_vdpa_event_qp_prepare(priv, + vq->size, vq->callfd, &virtq->eqp); if (ret) { - DRV_LOG(ERR, "Failed to create event QPs for virtq %d.", + DRV_LOG(ERR, + "Failed to create event QPs for virtq %d.", index); return -1; } - attr.qp_id = virtq->eqp.fw_qp->id; + attr->mod_fields_bitmap |= MLX5_VIRTQ_MODIFY_TYPE_EVENT_MODE; + attr->qp_id = virtq->eqp.fw_qp->id; } else { DRV_LOG(INFO, "Virtq %d is, for sure, working by poll mode, no" " need event QPs and event mechanism.", index); @@ -265,77 +270,82 @@ mlx5_vdpa_virtq_setup(struct mlx5_vdpa_priv *priv, int index) if (!virtq->counters) { DRV_LOG(ERR, "Failed to create virtq couners for virtq" " %d.", index); - goto error; + return -1; } - attr.counters_obj_id = virtq->counters->id; + attr->counters_obj_id = virtq->counters->id; } /* Setup 3 UMEMs for each virtq. */ - for (i = 0; i < RTE_DIM(virtq->umems); ++i) { - uint32_t size; - void *buf; - struct mlx5dv_devx_umem *obj; - - size = priv->caps.umems[i].a * vq.size + priv->caps.umems[i].b; - if (virtq->umems[i].size == size && - virtq->umems[i].obj != NULL) { - /* Reuse registered memory. */ - memset(virtq->umems[i].buf, 0, size); - goto reuse; - } - if (virtq->umems[i].obj) - claim_zero(mlx5_glue->devx_umem_dereg + if (virtq->virtq) { + for (i = 0; i < RTE_DIM(virtq->umems); ++i) { + uint32_t size; + void *buf; + struct mlx5dv_devx_umem *obj; + + size = + priv->caps.umems[i].a * vq->size + priv->caps.umems[i].b; + if (virtq->umems[i].size == size && + virtq->umems[i].obj != NULL) { + /* Reuse registered memory. */ + memset(virtq->umems[i].buf, 0, size); + goto reuse; + } + if (virtq->umems[i].obj) + claim_zero(mlx5_glue->devx_umem_dereg (virtq->umems[i].obj)); - if (virtq->umems[i].buf) - rte_free(virtq->umems[i].buf); - virtq->umems[i].size = 0; - virtq->umems[i].obj = NULL; - virtq->umems[i].buf = NULL; - buf = rte_zmalloc(__func__, size, 4096); - if (buf == NULL) { - DRV_LOG(ERR, "Cannot allocate umem %d memory for virtq" + if (virtq->umems[i].buf) + rte_free(virtq->umems[i].buf); + virtq->umems[i].size = 0; + virtq->umems[i].obj = NULL; + virtq->umems[i].buf = NULL; + buf = rte_zmalloc(__func__, + size, 4096); + if (buf == NULL) { + DRV_LOG(ERR, "Cannot allocate umem %d memory for virtq" " %u.", i, index); - goto error; - } - obj = mlx5_glue->devx_umem_reg(priv->cdev->ctx, buf, size, - IBV_ACCESS_LOCAL_WRITE); - if (obj == NULL) { - DRV_LOG(ERR, "Failed to register umem %d for virtq %u.", + return -1; + } + obj = mlx5_glue->devx_umem_reg(priv->cdev->ctx, + buf, size, IBV_ACCESS_LOCAL_WRITE); + if (obj == NULL) { + DRV_LOG(ERR, "Failed to register umem %d for virtq %u.", i, index); - goto error; - } - virtq->umems[i].size = size; - virtq->umems[i].buf = buf; - virtq->umems[i].obj = obj; + rte_free(buf); + return -1; + } + virtq->umems[i].size = size; + virtq->umems[i].buf = buf; + virtq->umems[i].obj = obj; reuse: - attr.umems[i].id = virtq->umems[i].obj->umem_id; - attr.umems[i].offset = 0; - attr.umems[i].size = virtq->umems[i].size; + attr->umems[i].id = virtq->umems[i].obj->umem_id; + attr->umems[i].offset = 0; + attr->umems[i].size = virtq->umems[i].size; + } } - if (attr.type == MLX5_VIRTQ_TYPE_SPLIT) { + if (attr->q_type == MLX5_VIRTQ_TYPE_SPLIT) { gpa = mlx5_vdpa_hva_to_gpa(priv->vmem, - (uint64_t)(uintptr_t)vq.desc); + (uint64_t)(uintptr_t)vq->desc); if (!gpa) { DRV_LOG(ERR, "Failed to get descriptor ring GPA."); - goto error; + return -1; } - attr.desc_addr = gpa; + attr->desc_addr = gpa; gpa = mlx5_vdpa_hva_to_gpa(priv->vmem, - (uint64_t)(uintptr_t)vq.used); + (uint64_t)(uintptr_t)vq->used); if (!gpa) { DRV_LOG(ERR, "Failed to get GPA for used ring."); - goto error; + return -1; } - attr.used_addr = gpa; + attr->used_addr = gpa; gpa = mlx5_vdpa_hva_to_gpa(priv->vmem, - (uint64_t)(uintptr_t)vq.avail); + (uint64_t)(uintptr_t)vq->avail); if (!gpa) { DRV_LOG(ERR, "Failed to get GPA for available ring."); - goto error; + return -1; } - attr.available_addr = gpa; + attr->available_addr = gpa; } - ret = rte_vhost_get_vring_base(priv->vid, index, &last_avail_idx, - &last_used_idx); + ret = rte_vhost_get_vring_base(priv->vid, + index, &last_avail_idx, &last_used_idx); if (ret) { last_avail_idx = 0; last_used_idx = 0; @@ -345,24 +355,71 @@ mlx5_vdpa_virtq_setup(struct mlx5_vdpa_priv *priv, int index) "virtq %d.", priv->vid, last_avail_idx, last_used_idx, index); } - attr.hw_available_index = last_avail_idx; - attr.hw_used_index = last_used_idx; - attr.q_size = vq.size; - attr.mkey = priv->gpa_mkey_index; - attr.tis_id = priv->tiss[(index / 2) % priv->num_lag_ports]->id; - attr.queue_index = index; - attr.pd = priv->cdev->pdn; - attr.hw_latency_mode = priv->hw_latency_mode; - attr.hw_max_latency_us = priv->hw_max_latency_us; - attr.hw_max_pending_comp = priv->hw_max_pending_comp; - virtq->virtq = mlx5_devx_cmd_create_virtq(priv->cdev->ctx, &attr); + attr->hw_available_index = last_avail_idx; + attr->hw_used_index = last_used_idx; + attr->q_size = vq->size; + attr->mkey = priv->gpa_mkey_index; + attr->tis_id = priv->tiss[(index / 2) % priv->num_lag_ports]->id; + attr->queue_index = index; + attr->pd = priv->cdev->pdn; + attr->hw_latency_mode = priv->hw_latency_mode; + attr->hw_max_latency_us = priv->hw_max_latency_us; + attr->hw_max_pending_comp = priv->hw_max_pending_comp; + if (attr->hw_latency_mode || attr->hw_max_latency_us || + attr->hw_max_pending_comp) + attr->mod_fields_bitmap |= MLX5_VIRTQ_MODIFY_TYPE_QUEUE_PERIOD; + return 0; +} + +bool +mlx5_vdpa_is_modify_virtq_supported(struct mlx5_vdpa_priv *priv) +{ + return (priv->caps.vnet_modify_ext && + priv->caps.virtio_net_q_addr_modify && + priv->caps.virtio_q_index_modify) ? true : false; +} + +static int +mlx5_vdpa_virtq_setup(struct mlx5_vdpa_priv *priv, int index) +{ + struct mlx5_vdpa_virtq *virtq = &priv->virtqs[index]; + struct rte_vhost_vring vq; + struct mlx5_devx_virtq_attr attr = {0}; + int ret; + uint16_t event_num = MLX5_EVENT_TYPE_OBJECT_CHANGE; + uint64_t cookie; + + ret = rte_vhost_get_vhost_vring(priv->vid, index, &vq); + if (ret) + return -1; + if (vq.size == 0) + return 0; virtq->priv = priv; - if (!virtq->virtq) + virtq->stopped = 0; + ret = mlx5_vdpa_virtq_sub_objs_prepare(priv, &attr, + &vq, index); + if (ret) { + DRV_LOG(ERR, "Failed to setup update virtq attr %d.", + index); goto error; - claim_zero(rte_vhost_enable_guest_notification(priv->vid, index, 1)); - if (mlx5_vdpa_virtq_modify(virtq, 1)) + } + if (!virtq->virtq) { + virtq->index = index; + virtq->vq_size = vq.size; + virtq->virtq = mlx5_devx_cmd_create_virtq(priv->cdev->ctx, + &attr); + if (!virtq->virtq) + goto error; + attr.mod_fields_bitmap = MLX5_VIRTQ_MODIFY_TYPE_STATE; + } + attr.state = MLX5_VIRTQ_STATE_RDY; + ret = mlx5_devx_cmd_modify_virtq(virtq->virtq, &attr); + if (ret) { + DRV_LOG(ERR, "Failed to modify virtq %d.", index); goto error; - virtq->priv = priv; + } + claim_zero(rte_vhost_enable_guest_notification(priv->vid, index, 1)); + virtq->configured = 1; rte_write32(virtq->index, priv->virtq_db_addr); /* Setup doorbell mapping. */ virtq->intr_handle = @@ -553,7 +610,7 @@ mlx5_vdpa_virtq_enable(struct mlx5_vdpa_priv *priv, int index, int enable) return 0; DRV_LOG(INFO, "Virtq %d was modified, recreate it.", index); } - if (virtq->virtq) { + if (virtq->configured) { virtq->enable = 0; if (is_virtq_recvq(virtq->index, priv->nr_virtqs)) { ret = mlx5_vdpa_steer_update(priv); -- 2.31.1