From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-f66.google.com (mail-wm0-f66.google.com [74.125.82.66]) by dpdk.org (Postfix) with ESMTP id B12E01AEF6 for ; Tue, 24 Oct 2017 16:18:18 +0200 (CEST) Received: by mail-wm0-f66.google.com with SMTP id 78so13804052wmb.1 for ; Tue, 24 Oct 2017 07:18:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:content-transfer-encoding:in-reply-to :user-agent; bh=y8SokP1nTHqYVfh4ECHKqR6H4gonMDm4Enq4BSnz6No=; b=bC9gpss2v6PVtn2zWakroyxQq2WbnFBSnUhFNxSlq2PmvHkjGK1nHFDYcmTSUAFVnv LeTb+QY19gnaV5OWjKfQfNR4fyfmn9KjGghx4e8WDZaMe/zl5ddoO8jYjrlrGlaBA/0X YvCBQxMfbXAOiwYFdDGSKMGsFYTysM4d8kNnaxH7kVCmIPtOcs8z4RPwHGiRVNVwyjQ3 uhZex+mSHsvtr96MYdM3v9o59NC1XtM5/lDXc8j4mS8MLGZfj/GKdKusbFw35kajXDOt k7WP6Zwh3L1yKfhE76vq18tQlDq7OO2jBTDdJZObF5iPxHZwEJNA7lKxf5N0I1wx++YC +E3g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:content-transfer-encoding :in-reply-to:user-agent; bh=y8SokP1nTHqYVfh4ECHKqR6H4gonMDm4Enq4BSnz6No=; b=WGSaRgvoebRXL3bY0malr4SY7xefKE0oJ50d7fz8iCbPHUf5nTj0PkpFSskh6OG4Ug Iwyzmr6qIr9BE6qZA7DC53Wxglb2or5zxvOuJe7cq60zGpMlg0RIN3MPXSSGDmWxfIe1 vfiZ30tUEBHiGemR3jzGwlNTald//7C1to1Tpbl23Px5/EcQsenDq22tdg1N15V4nGn7 45DnHlF2g0OdrpFUYGQADvqw9Kvlfu+Vwk/BaUFj7CdEJOfXV5bYNST8GwsM946/8Zar H6V0lob6EddYcek09u4sFcFJ9OUe//Muz7o1Fsmp8A1QJdV0mEU5MVoL+JdSO6dKgwsx 0Icw== X-Gm-Message-State: AMCzsaV1Nx8k9P9Go4fPBe5TA2OouDlyG18JkvTvhhw4kWIgKby9dLCn KaBwwwFCZdadl9YQDidPmTWCrgE+oA== X-Google-Smtp-Source: ABhQp+TXH9gE4yTKtMKH9BSujdegIqslcfZBcTPhK8c/n8SAbwE8vo01rOGqxMKvIzVhgTnjetaRCg== X-Received: by 10.28.74.193 with SMTP id n62mr6216818wmi.61.1508854698356; Tue, 24 Oct 2017 07:18:18 -0700 (PDT) Received: from laranjeiro-vm (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id c37sm570192wra.73.2017.10.24.07.18.17 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 24 Oct 2017 07:18:17 -0700 (PDT) Date: Tue, 24 Oct 2017 16:18:14 +0200 From: =?iso-8859-1?Q?N=E9lio?= Laranjeiro To: Moti Haimovsky Cc: adrien.mazarguil@6wind.com, dev@dpdk.org Message-ID: <20171024141814.jzzn7hqxt6usjfjd@laranjeiro-vm> References: <1508850674-181964-1-git-send-email-motih@mellanox.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <1508850674-181964-1-git-send-email-motih@mellanox.com> User-Agent: NeoMutt/20170113 (1.7.2) Subject: Re: [dpdk-dev] [PATCH v2] net/mlx4: fix no Rx interrupts X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Oct 2017 14:18:18 -0000 Hi Moti, On Tue, Oct 24, 2017 at 04:11:14PM +0300, Moti Haimovsky wrote: > This commit addresses the issue of rx interrupts support with > the new Rx datapath introduced in DPDK version 17.11. > In order to generate an Rx interrupt an event queue is armed with the > consumer index of the Rx completion queue. Since version 17.11 this > index is handled by the PMD so it is now the responsibility of the > PMD to write this value when enabling Rx interrupts. > > Fixes: 6681b845034c ("net/mlx4: add Rx bypassing Verbs") > > Signed-off-by: Moti Haimovsky > --- > V2: > * Rebased on top of ff3397e9 ("net/mlx4: relax Rx queue configuration order") > --- > drivers/net/mlx4/mlx4_intr.c | 38 +++++++++++++++++++++++++++++++++++++- > drivers/net/mlx4/mlx4_prm.h | 9 +++++++++ > drivers/net/mlx4/mlx4_rxq.c | 6 +++++- > 3 files changed, 51 insertions(+), 2 deletions(-) > > diff --git a/drivers/net/mlx4/mlx4_intr.c b/drivers/net/mlx4/mlx4_intr.c > index 3806322..217c591 100644 > --- a/drivers/net/mlx4/mlx4_intr.c > +++ b/drivers/net/mlx4/mlx4_intr.c > @@ -53,6 +53,7 @@ > #include > #include > #include > +#include > #include > > #include "mlx4.h" > @@ -239,6 +240,40 @@ > } > > /** > + * MLX4 CQ notification . > + * > + * @param rxq > + * Pointer to receive queue structure. > + * @param solicited > + * Is request solicited or not. > + * > + * @return > + * Always 0. Why not setting the return type to void in this case? > + */ > +static int > +mlx4_arm_cq(struct rxq *rxq, int solicited) > +{ > + struct mlx4_cq *cq = &rxq->mcq; > + uint64_t doorbell; > + uint32_t sn = cq->arm_sn & MLX4_CQ_DB_GEQ_N_MASK; > + uint32_t ci = cq->cons_index & MLX4_CQ_DB_CI_MASK; > + uint32_t cmd = solicited ? MLX4_CQ_DB_REQ_NOT_SOL : MLX4_CQ_DB_REQ_NOT; > + void *cq_db_reg = (char *)cq->cq_uar + MLX4_CQ_DOORBELL; use uint8_t* instead of char*. By the way, why not storing the address to the doorbell directly in the structure instead of computing it each time it is necessary? > + *cq->arm_db = rte_cpu_to_be_32(sn << 28 | cmd | ci); > + /* > + * Make sure that the doorbell record in host memory is > + * written before ringing the doorbell via PCI MMIO. > + */ > + rte_wmb(); > + doorbell = sn << 28 | cmd | cq->cqn; > + doorbell <<= 32; > + doorbell |= ci; > + rte_write64(rte_cpu_to_be_64(doorbell), cq_db_reg); > + return 0; > +} > + > +/** > * Uninstall interrupt handler. > * > * @param priv > @@ -333,6 +368,7 @@ > WARN("unable to disable interrupt on rx queue %d", > idx); > } else { > + rxq->mcq.arm_sn++; > ibv_ack_cq_events(rxq->cq, 1); > } > return -ret; > @@ -358,7 +394,7 @@ > if (!rxq || !rxq->channel) > ret = EINVAL; > else > - ret = ibv_req_notify_cq(rxq->cq, 0); > + ret = mlx4_arm_cq(rxq, 0); > if (ret) { > rte_errno = ret; > WARN("unable to arm interrupt on rx queue %d", idx); > diff --git a/drivers/net/mlx4/mlx4_prm.h b/drivers/net/mlx4/mlx4_prm.h > index 3a77502..92f2fec 100644 > --- a/drivers/net/mlx4/mlx4_prm.h > +++ b/drivers/net/mlx4/mlx4_prm.h > @@ -78,6 +78,11 @@ enum { > MLX4_CQE_L2_TUNNEL_IPOK = (int)(1u << 31), > }; > > +/* Completion queue events, numbers and masks. */ > +#define MLX4_CQ_DB_GEQ_N_MASK 0x3 > +#define MLX4_CQ_DB_CI_MASK 0xffffff You could have replaced all the places where this mask is hard coded by this new define for consistency. > +#define MLX4_CQ_DOORBELL 0x20 > + > /* Send queue information. */ > struct mlx4_sq { > uint8_t *buf; /**< SQ buffer. */ > @@ -100,6 +105,10 @@ struct mlx4_cq { > uint32_t cqe_64:1; /**< CQ entry size is 64 bytes. */ > uint32_t cons_index; /**< Last queue entry that was handled. */ > uint32_t *set_ci_db; /**< Pointer to the completion queue doorbell. */ > + uint32_t *arm_db; /**< Pointer to doorbell for arming Rx events. */ > + int arm_sn; /**< Rx event counter. */ > + void *cq_uar; /**< CQ user access region. */ > + uint32_t cqn; /**< CQ number. */ Try to re-arrange the variable order to avoid holes in the structure. > }; > > /** > diff --git a/drivers/net/mlx4/mlx4_rxq.c b/drivers/net/mlx4/mlx4_rxq.c > index 4c50077..24262cb 100644 > --- a/drivers/net/mlx4/mlx4_rxq.c > +++ b/drivers/net/mlx4/mlx4_rxq.c > @@ -510,7 +510,7 @@ void mlx4_rss_detach(struct mlx4_rss *rss) > struct rte_mbuf *(*elts)[elts_n] = rxq->elts; > struct mlx4dv_obj mlxdv; > struct mlx4dv_rwq dv_rwq; > - struct mlx4dv_cq dv_cq; > + struct mlx4dv_cq dv_cq = { .comp_mask = MLX4DV_CQ_MASK_UAR, }; > const char *msg; > struct ibv_cq *cq = NULL; > struct ibv_wq *wq = NULL; > @@ -604,6 +604,10 @@ void mlx4_rss_detach(struct mlx4_rss *rss) > rxq->mcq.cqe_cnt = dv_cq.cqe_cnt; > rxq->mcq.set_ci_db = dv_cq.set_ci_db; > rxq->mcq.cqe_64 = (dv_cq.cqe_size & 64) ? 1 : 0; > + rxq->mcq.arm_db = dv_cq.arm_db; > + rxq->mcq.arm_sn = dv_cq.arm_sn; > + rxq->mcq.cqn = dv_cq.cqn; > + rxq->mcq.cq_uar = dv_cq.cq_uar; > /* Update doorbell counter. */ > rxq->rq_ci = elts_n / sges_n; > rte_wmb(); > -- > 1.8.3.1 Thanks, -- Nélio Laranjeiro 6WIND