From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id A89D9A055C for ; Thu, 9 Jun 2022 19:29:30 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A118F427E9; Thu, 9 Jun 2022 19:29:30 +0200 (CEST) Received: from mail-qk1-f179.google.com (mail-qk1-f179.google.com [209.85.222.179]) by mails.dpdk.org (Postfix) with ESMTP id 6E1C040689; Thu, 9 Jun 2022 19:29:28 +0200 (CEST) Received: by mail-qk1-f179.google.com with SMTP id p63so1039031qkd.10; Thu, 09 Jun 2022 10:29:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=xqS52hFBJOHBcSTqpXJwe3IL4JopV/W8CV1d2QTaffo=; b=quHoEsZqRHsn6X6D56wyYEQQ69iGFEZMtsMkFrHUuoujiruuXWVQI57AQH6S1PNg+6 n+8A7oZ2TnK1K/LImgWGjI/quQNKZg8xmZdDyOy/uzZvzBHuOLZn7/DtlksdCdco/gem zhUr6z3Rso84+xXygoRUCNwLMvZVS0e2dFyLHZaYuRTXdblxtza1u2RVBvBV2Lsyk5lY 42n941Qk520HoJiZcCqpl8HQi7X2gExqSoTcvY/Bf01FwZFALgG9hR8g5/0rxsC9Uyo0 pVfAju//Osz+v/dsrbYRbhZpHPbykeE6joyyGyQeDqVVUdvH7whwnSNtpxQCqjU2cOzN 9mjg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=xqS52hFBJOHBcSTqpXJwe3IL4JopV/W8CV1d2QTaffo=; b=EtYTWehh+J1VWcMy1DcNCQAJIkZpWrsg0FyJqMDdovL2RsTMnOxvzI70vq6b3htT+r eIPuxQJOuH4vYRX9Iz2VJ+br7VCVfpZM5Ibcak81O1vT7utIqI7DCKMgnxZv3Pkh1jD9 UqCkq6jT2WSluLt9IiwJYga+m0K7Pm1m4u5zpKTwwbVpXUGatdxfa7t66K2kdolqQ3UO X1d3VDKZx2PJvS2rkHGX+yESgXl8b20dtag5tgyljMmuXZb6TCpwvxKCXosOuyMnsWwA lpo+WZgNgJY+6y/5l9BmOKjA4s12AwuiXCRtpl81HvMFEQgcFMRoCD+TgSXLSCRTHxz9 8BmA== X-Gm-Message-State: AOAM533HjhAndUrHg9nECoxy9K1IXbdt00Yr5ZFz7ZIooB+MzuCNlIUo TxzF0QHiz05u9mYAsWE1JcR8G9J/+sCvzQoBM/gz6Zivgrg= X-Google-Smtp-Source: ABdhPJzbTmztdQyfBs6q2optqPAlDKN+rzsZpeONYaLU7nNjhgenfWVqVbV1aUMc7OkG15rknt1+bMonG+d4GZpKSpk= X-Received: by 2002:a05:620a:cd5:b0:6a6:8d8b:7b88 with SMTP id b21-20020a05620a0cd500b006a68d8b7b88mr24110719qkj.402.1654795767639; Thu, 09 Jun 2022 10:29:27 -0700 (PDT) MIME-Version: 1.0 References: <20220606155517.1226558-1-timothy.mcdaniel@intel.com> In-Reply-To: <20220606155517.1226558-1-timothy.mcdaniel@intel.com> From: Jerin Jacob Date: Thu, 9 Jun 2022 22:59:01 +0530 Message-ID: Subject: Re: [PATCH] event/dlb2: rework queue drain handling To: Timothy McDaniel Cc: Jerin Jacob , dpdk-dev , dpdk stable Content-Type: text/plain; charset="UTF-8" X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: stable-bounces@dpdk.org On Mon, Jun 6, 2022 at 9:36 PM Timothy McDaniel wrote: > > Previously, anything that prevented the CQs from being drained > would hang the PMD for an unacceptably long time. This commit updates > the drain logic to be resource and time based, thus eliminating > the potential for a long hang when draining the queues in preparation > for a reset. > > Fixes: 1857f1922ce2 ("event/dlb2: use new implementation of resource file") > Cc: stable@dpdk.org > > Signed-off-by: Timothy McDaniel Applied to dpdk-next-net-eventdev/for-main. Thanks > --- > drivers/event/dlb2/pf/base/dlb2_hw_types.h | 2 +- > drivers/event/dlb2/pf/base/dlb2_resource.c | 45 +++++++++++++++++----- > 2 files changed, 37 insertions(+), 10 deletions(-) > > diff --git a/drivers/event/dlb2/pf/base/dlb2_hw_types.h b/drivers/event/dlb2/pf/base/dlb2_hw_types.h > index 6b8fee3416..9511521e67 100644 > --- a/drivers/event/dlb2/pf/base/dlb2_hw_types.h > +++ b/drivers/event/dlb2/pf/base/dlb2_hw_types.h > @@ -27,7 +27,7 @@ > #define DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS 2 > #define DLB2_MAX_NUM_SEQUENCE_NUMBER_MODES 5 > #define DLB2_MAX_CQ_COMP_CHECK_LOOPS 409600 > -#define DLB2_MAX_QID_EMPTY_CHECK_LOOPS (32 * 64 * 1024 * (800 / 30)) > +#define DLB2_MAX_QID_EMPTY_CHECK_LOOPS (4 * DLB2_MAX_NUM_LDB_CREDITS) > > #define DLB2_FUNC_BAR 0 > #define DLB2_CSR_BAR 2 > diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c > index 548bebd068..4011c24aef 100644 > --- a/drivers/event/dlb2/pf/base/dlb2_resource.c > +++ b/drivers/event/dlb2/pf/base/dlb2_resource.c > @@ -1057,7 +1057,7 @@ static u32 dlb2_dir_cq_token_count(struct dlb2_hw *hw, > port->init_tkn_cnt; > } > > -static void dlb2_drain_dir_cq(struct dlb2_hw *hw, > +static int dlb2_drain_dir_cq(struct dlb2_hw *hw, > struct dlb2_dir_pq_pair *port) > { > unsigned int port_id = port->id.phys_id; > @@ -1089,6 +1089,8 @@ static void dlb2_drain_dir_cq(struct dlb2_hw *hw, > > os_unmap_producer_port(hw, pp_addr); > } > + > + return cnt; > } > > static void dlb2_dir_port_cq_enable(struct dlb2_hw *hw, > @@ -1107,6 +1109,7 @@ static int dlb2_domain_drain_dir_cqs(struct dlb2_hw *hw, > { > struct dlb2_list_entry *iter; > struct dlb2_dir_pq_pair *port; > + int drain_cnt = 0; > RTE_SET_USED(iter); > > DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter) { > @@ -1120,13 +1123,13 @@ static int dlb2_domain_drain_dir_cqs(struct dlb2_hw *hw, > if (toggle_port) > dlb2_dir_port_cq_disable(hw, port); > > - dlb2_drain_dir_cq(hw, port); > + drain_cnt = dlb2_drain_dir_cq(hw, port); > > if (toggle_port) > dlb2_dir_port_cq_enable(hw, port); > } > > - return 0; > + return drain_cnt; > } > > static u32 dlb2_dir_queue_depth(struct dlb2_hw *hw, > @@ -1170,10 +1173,20 @@ static int dlb2_domain_drain_dir_queues(struct dlb2_hw *hw, > return 0; > > for (i = 0; i < DLB2_MAX_QID_EMPTY_CHECK_LOOPS; i++) { > - dlb2_domain_drain_dir_cqs(hw, domain, true); > + int drain_cnt; > + > + drain_cnt = dlb2_domain_drain_dir_cqs(hw, domain, false); > > if (dlb2_domain_dir_queues_empty(hw, domain)) > break; > + > + /* > + * Allow time for DLB to schedule QEs before draining > + * the CQs again. > + */ > + if (!drain_cnt) > + rte_delay_us(1); > + > } > > if (i == DLB2_MAX_QID_EMPTY_CHECK_LOOPS) { > @@ -1249,7 +1262,7 @@ static u32 dlb2_ldb_cq_token_count(struct dlb2_hw *hw, > port->init_tkn_cnt; > } > > -static void dlb2_drain_ldb_cq(struct dlb2_hw *hw, struct dlb2_ldb_port *port) > +static int dlb2_drain_ldb_cq(struct dlb2_hw *hw, struct dlb2_ldb_port *port) > { > u32 infl_cnt, tkn_cnt; > unsigned int i; > @@ -1289,32 +1302,37 @@ static void dlb2_drain_ldb_cq(struct dlb2_hw *hw, struct dlb2_ldb_port *port) > > os_unmap_producer_port(hw, pp_addr); > } > + > + return tkn_cnt; > } > > -static void dlb2_domain_drain_ldb_cqs(struct dlb2_hw *hw, > +static int dlb2_domain_drain_ldb_cqs(struct dlb2_hw *hw, > struct dlb2_hw_domain *domain, > bool toggle_port) > { > struct dlb2_list_entry *iter; > struct dlb2_ldb_port *port; > + int drain_cnt = 0; > int i; > RTE_SET_USED(iter); > > /* If the domain hasn't been started, there's no traffic to drain */ > if (!domain->started) > - return; > + return 0; > > for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) { > DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) { > if (toggle_port) > dlb2_ldb_port_cq_disable(hw, port); > > - dlb2_drain_ldb_cq(hw, port); > + drain_cnt = dlb2_drain_ldb_cq(hw, port); > > if (toggle_port) > dlb2_ldb_port_cq_enable(hw, port); > } > } > + > + return drain_cnt; > } > > static u32 dlb2_ldb_queue_depth(struct dlb2_hw *hw, > @@ -1375,10 +1393,19 @@ static int dlb2_domain_drain_mapped_queues(struct dlb2_hw *hw, > } > > for (i = 0; i < DLB2_MAX_QID_EMPTY_CHECK_LOOPS; i++) { > - dlb2_domain_drain_ldb_cqs(hw, domain, true); > + int drain_cnt; > + > + drain_cnt = dlb2_domain_drain_ldb_cqs(hw, domain, false); > > if (dlb2_domain_mapped_queues_empty(hw, domain)) > break; > + > + /* > + * Allow time for DLB to schedule QEs before draining > + * the CQs again. > + */ > + if (!drain_cnt) > + rte_delay_us(1); > } > > if (i == DLB2_MAX_QID_EMPTY_CHECK_LOOPS) { > -- > 2.25.1 >