From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-f65.google.com (mail-wm0-f65.google.com [74.125.82.65]) by dpdk.org (Postfix) with ESMTP id 929E5397D for ; Mon, 22 Jan 2018 15:53:26 +0100 (CET) Received: by mail-wm0-f65.google.com with SMTP id v123so17361798wmd.5 for ; Mon, 22 Jan 2018 06:53:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:content-transfer-encoding:in-reply-to :user-agent; bh=mxoz69bRG0jqLAP57QCclHoPC0jhTGz2gmZLS2ch3vM=; b=w+nT/7qXzSr4ZXqb9dZPXtuJpM3+sS+Ivryu/EFvb0t5l7HkruHq0VrPk8Q+RkeiBS RqPyghYU3vE2esr3f3c1L1q8/RSa3vxZxmFTw/T+7FTMo6QGWQQ04mzA2JKGc8n4iFi7 dYj9+uYsGkEMO7YnH3w9Y0NGhUS3I4C/Fa/W9PU6ChAuhjxdTVJyPGh1B8vffej4LCtj 1s3e3nbqWJrF+g6uuiXSwD3f3L3AwG/JIMB7M3jU7kb7Zm5k2lco9Dbj977AAv/J9AuO iGiXMDNpfbM0iJfajOpp5YJPlG98BrAlh8DakPM3kifbJ+S6WEsgNLc/yDmAfPdhlRcy dHWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:content-transfer-encoding :in-reply-to:user-agent; bh=mxoz69bRG0jqLAP57QCclHoPC0jhTGz2gmZLS2ch3vM=; b=nA76LjYwxadveJ08IbrIWiiT1zmR31yAtVT2YBJ+S883NO7X14hG2y6lgAqNcq4OJD ZQ9/DGbxQURnsbLI6x6nJ1Xmfkfj5RgJqK9A1lb0JlAOebEV6kFnmg2aYJz1y6Ye8PqX 0DwM57D7m7ix+FQAEXDBl52USAIwOI4888NY38VKFqZUK3XRyO+7GvPqwuAIDw8eZyWv JapI0gsMifJjnqL2rlE7cKgHR22RtGzEjLZX+jDcfKzy7w4QeeVJOBRvZ9rvmymdqq2Z o6zCFhopAeN5sAvWq301U/Jg5XtxFl4yyuSwL2RuaeNv6riwLSoypvYPEIUq9ZnczVVn 0EZQ== X-Gm-Message-State: AKwxytfsHAK3oPcYpl4xtFZB33VYrZkfPiLZ9hPTMA54Rg/buO6fjJZb pW8QRApvpcHmotXq+PKpmS/K X-Google-Smtp-Source: AH8x224sQ4Je4EuG2fkT+4+D4GWg4QiTHEj/5WIGc7dLD02YcY5rCmVuz2RPcxzg38bOFI5HCEDpuA== X-Received: by 10.28.210.197 with SMTP id j188mr4853817wmg.74.1516632806236; Mon, 22 Jan 2018 06:53:26 -0800 (PST) Received: from laranjeiro-vm.dev.6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id u27sm12906506wrf.77.2018.01.22.06.53.25 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 22 Jan 2018 06:53:25 -0800 (PST) Date: Mon, 22 Jan 2018 15:53:21 +0100 From: =?iso-8859-1?Q?N=E9lio?= Laranjeiro To: Xueming Li Cc: Shahaf Shuler , dev@dpdk.org Message-ID: <20180122145321.jpyepyvjjlktillp@laranjeiro-vm.dev.6wind.com> References: <20180119150854.89828-1-xuemingl@mellanox.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20180119150854.89828-1-xuemingl@mellanox.com> User-Agent: NeoMutt/20170113 (1.7.2) Subject: Re: [dpdk-dev] [PATCH] net/mlx5: remmap UAR address for multiple process X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Jan 2018 14:53:26 -0000 Hi Xueming, On Fri, Jan 19, 2018 at 11:08:54PM +0800, Xueming Li wrote: > UAR(doorbell) is hw resources that have to be same address between > primary and secondary process, failed to mmap UAR will make TX packets > invisible to HW. > Today, UAR address returned from verbs api is mixed in heap and loaded > library address space, prone to be occupied in secondary process. > This patch reserves a dedicate UAR address space, both primary and > secondary process re-mmap UAR pages into this space. > Below is a brief picture of dpdk app address space allocation: > Before This patch > ------ ---------- > [stack] [stack] > [.so, uar, heap] [.so, heap] > [(empty)] [(empty)] > [hugepage] [hugepage] > [? others] [? others] > [(empty)] [(empty)] > [uar] > [(empty)] > To minimize conflicts, UAR address space comes after hugepage space with > an offset to skip potential usage from other drivers. Seems it is not the case when the memory is contiguous, according to what I see in my testpmd /proc//maps: PMD: mlx5.c:523: mlx5_uar_init_primary(): Reserved UAR address space: 0x0x7f4da5800000 And the fist huge page is at address 0x7f4fa5800000, new UAR space is before and not after. With this patch I still have the situation described as "before". > Once UAR space reserved successfully, UAR pages are re-mmapped into new > area to keep UAR address aligned between primary and secondary process. > > Signed-off-by: Xueming Li > --- > drivers/net/mlx5/mlx5.c | 107 ++++++++++++++++++++++++++++++++++++++++ > drivers/net/mlx5/mlx5.h | 1 + > drivers/net/mlx5/mlx5_defs.h | 10 ++++ > drivers/net/mlx5/mlx5_rxtx.h | 3 +- > drivers/net/mlx5/mlx5_trigger.c | 7 ++- > drivers/net/mlx5/mlx5_txq.c | 51 +++++++++++++------ > 6 files changed, 163 insertions(+), 16 deletions(-) > > diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c > index fc2d59fee..1539ef608 100644 > --- a/drivers/net/mlx5/mlx5.c > +++ b/drivers/net/mlx5/mlx5.c > @@ -39,6 +39,7 @@ > #include > #include > #include > +#include > > /* Verbs header. */ > /* ISO C doesn't support unnamed structs/unions, disabling -pedantic. */ > @@ -56,6 +57,7 @@ > #include > #include > #include > +#include > #include > > #include "mlx5.h" > @@ -466,6 +468,101 @@ mlx5_args(struct mlx5_dev_config *config, struct rte_devargs *devargs) > > static struct rte_pci_driver mlx5_driver; > > +/* > + * Reserved UAR address space for TXQ UAR(hw doorbell) mapping, process > + * local resource used by both primary and secondary to avoid duplicate > + * reservation. > + * The space has to be available on both primary and secondary process, > + * TXQ UAR maps to this area using fixed mmap w/o double check. > + */ > +static void *uar_base; > + > +/** > + * Reserve UAR address space for primary process > + * > + * @param[in] priv > + * Pointer to private structure. > + * > + * @return > + * 0 on success, negative errno value on failure. > + */ > +static int > +mlx5_uar_init_primary(struct priv *priv) > +{ > + void *addr = (void *)0; > + int i; > + const struct rte_mem_config *mcfg; > + > + if (uar_base) { /* UAR address space mapped */ > + priv->uar_base = uar_base; > + return 0; > + } > + /* find out lower bound of hugepage segments */ > + mcfg = rte_eal_get_configuration()->mem_config; > + for (i = 0; i < RTE_MAX_MEMSEG && mcfg->memseg[i].addr; i++) { > + if (addr) > + addr = RTE_MIN(addr, mcfg->memseg[i].addr); > + else > + addr = mcfg->memseg[i].addr; This if/else is useless as addr is already initialised with the smallest possible value. > + } > + /* offset down UAR area */ > + addr = RTE_PTR_SUB(addr, MLX5_UAR_OFFSET + MLX5_UAR_SIZE); Seems the error is here, the loops get the address of the memseg with the smallest address and then it subtract the UAR size, addr cannot be after the huge pages unless if this subtraction overflows. > + /* anonymous mmap, no real memory consumption */ > + addr = mmap(addr, MLX5_UAR_SIZE, > + PROT_NONE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); > + if (addr == MAP_FAILED) { > + ERROR("Failed to reserve UAR address space, please adjust " > + "MLX5_UAR_SIZE or try --base-virtaddr"); How does a user knows the UAR memory space the NIC needs to adjust the MLX5_UAR_SIZE? > + return -ENOMEM; > + } > + /* Accept either same addr or a new addr returned from mmap if target > + * range occupied. > + */ > + INFO("Reserved UAR address space: 0x%p", addr); The '%p' already prefix the address with the 0x. > + priv->uar_base = addr; /* for primary and secondary UAR re-mmap */ > + uar_base = addr; /* process local, don't reserve again */ > + return 0; > +} > + Regards, -- Nélio Laranjeiro 6WIND