From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-f65.google.com (mail-wm0-f65.google.com [74.125.82.65]) by dpdk.org (Postfix) with ESMTP id F379F160 for ; Tue, 14 Nov 2017 11:34:12 +0100 (CET) Received: by mail-wm0-f65.google.com with SMTP id n74so17366976wmi.1 for ; Tue, 14 Nov 2017 02:34:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:content-transfer-encoding:in-reply-to :user-agent; bh=PMZFdTwQsAq9MrR/R1iL2dtPqUMA5CHTjjMGf+5j57E=; b=Oa8kJ94ux5rnemMrdOOLeeNheXEPMJ0rs5jkJ38roxs7YbMu96IqsCHPdhJ8RWhbcL pUo8o1loyCoMson6Zd8ws3JFj6txPTnJcHeFiga9bykjBodYc4SWaYMo0N+4CXhlQRuP hr0YgBjeoifrqucvWuSUmaCPgResqETy7dE745BU3wrpcnE/Z09hRg+hafdhYwZBphII wl4Xn0URDxM1WLO0BRtFEvEsBV8EzlINqAegn8pa6CWiaL2DRB8AHU5qgMlFc42vzd1K uXtujrPzX2bUQWEgD/GARLxJhMRTJ4MtOVu8VsB7jNHdL6/EUZ8KWsWKTbZpO44sbw1z qHzw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:content-transfer-encoding :in-reply-to:user-agent; bh=PMZFdTwQsAq9MrR/R1iL2dtPqUMA5CHTjjMGf+5j57E=; b=ONbNlPZbQsL7ie6eT4LSskonlF6WReFDXi/zhFFbUcu1MniwUJuaqoJllGs7hQxPbc Dv8vY3nLic6bsOZOtEchMvFDIcjBfcWXQKipco41LfHUJfgvLPz37LS/T7A92qQICrv3 OSZp9iGPu9fGpkGlLTBNC8V/kxRUG/WGSBqUtfaIfe4VdVfx4IaCAms9+GQpILSKBRGL Qkb+i5g3Eiwewbq1zL0NPCBLFpYsJoNl5bJNm7rIWPwzjxptCIhUZn1RmBXpvNScGzwl DySnxP10MAAZU/hrxKPniAJm9cY9ibO4YCkl2m3NuyFblLJmgnT6fa0ZsWiX02nrv3UI zboA== X-Gm-Message-State: AJaThX4evkp51mSJ8IBs1UWE7MsQpJCyHk1OR+bHrM9d9s7ds1V6qrEv M6TXQE3H9tl6Ru+FZSoHjoN2 X-Google-Smtp-Source: AGs4zMbxmC0mxOxNLSpuxfCq6FiK5SydxCzaQguVTF8+cAJ8nV+78x1m2WSw8wDf684w7Z0GS/E4xw== X-Received: by 10.80.151.213 with SMTP id f21mr16712868edb.141.1510655652646; Tue, 14 Nov 2017 02:34:12 -0800 (PST) Received: from laranjeiro-vm.dev.6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id l4sm17928606edc.20.2017.11.14.02.34.11 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 14 Nov 2017 02:34:11 -0800 (PST) Date: Tue, 14 Nov 2017 11:34:02 +0100 From: Nelio Laranjeiro To: Olivier Gournet Cc: dev@dpdk.org, Adrien Mazarguil , Yongseok Koh Message-ID: <20171114103402.6rr26vrlbbvcjnf6@laranjeiro-vm.dev.6wind.com> References: <2377aa87-fc6c-ee1e-91b8-ae226cb79a87@corp.free.fr> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <2377aa87-fc6c-ee1e-91b8-ae226cb79a87@corp.free.fr> User-Agent: NeoMutt/20170113 (1.7.2) Subject: Re: [dpdk-dev] mlx5 and secondary processes X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Nov 2017 10:34:13 -0000 Hi Olivier, On Tue, Nov 14, 2017 at 10:58:02AM +0100, Olivier Gournet wrote: > Hi, > > I can't get TX on secondary process to works with the lastest dpdk, it was running fine with dpdk-16.11. > RX/TX is ok on primary processs, and I'm not interested in RX on secondary process. Each process has its > owns TX queues. > > I upgraded everything to: > linux 4.14 > dpdk 17.11-rc4 > rdma-core from yesterday git-master > mlx-fw 12.21.1000 > > On seconday process, TX queue gets full and is never emptied. > It seems like bf_reg is correctly re-mapped from secondary process; here's dmesg from primary process: > > [...] > [85361.895778] mlx5_1:uar_mmap:1722:(pid 9533): uar idx 0x6, pfn 0x00000000000f8020 > [85361.895781] mlx5_1:uar_mmap:1735:(pid 9533): mapped best effort WC at 0x7ffff7faa000, PA 0x00000000f8020000 > [85361.895784] mlx5_1:uar_mmap:1722:(pid 9533): uar idx 0x7, pfn 0x00000000000f8021 > [85361.895787] mlx5_1:uar_mmap:1735:(pid 9533): mapped best effort WC at 0x7ffff7fa9000, PA 0x00000000f8021000 > > Then from secondary process: > > [...] > [85408.038295] mlx5_1:mlx5_ib_mmap:1778:(pid 9551): mapped internal timer at 0x7ffff7f7f000, PA 0xf8001000 > [85408.040229] mlx5_1:uar_mmap:1722:(pid 9551): uar idx 0x6, pfn 0x00000000000f8020 > [85408.040233] mlx5_1:uar_mmap:1735:(pid 9551): mapped best effort WC at 0x7ffff7faa000, PA 0x00000000f8020000 > [85408.040239] mlx5_1:uar_mmap:1722:(pid 9551): uar idx 0x7, pfn 0x00000000000f8021 > [85408.040241] mlx5_1:uar_mmap:1735:(pid 9551): mapped best effort WC at 0x7ffff7fa9000, PA 0x00000000f8021000 > > But then from mlx5_rxtx.h:mlx5_tx_dbrec_cond_wmb() it doesn't seems like writing to txq->bf_reg has any effect. > I know it isn't supposed to be read, but when reading from primary process is always give *dst==0xe5ccdabae5ccdaba, > whereas from secondary process *dst is zero or the last written value. > > I don't really have any other clues, and don't know where to search. Can anybody give me some hints ? Since Yongseok patch [1] the UAR pages are no more configured though the Write Combining of the PCI but in your logs it seems it is still using the cache. Do you have the "MLX5_SHUT_UP_BF" set to 1 in your environment? If not does adding it solves your issue? Regards, [1] http://dpdk.org/browse/dpdk/commit/?id=fb870be5a879c6617fecabf47873ae2b576e6e69 -- Nélio Laranjeiro 6WIND