From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp1-g21.free.fr (smtp1-g21.free.fr [212.27.42.1]) by dpdk.org (Postfix) with ESMTP id 590121AEE8 for ; Tue, 14 Nov 2017 14:47:00 +0100 (CET) Received: from [192.168.110.2] (unknown [213.228.1.188]) by smtp1-g21.free.fr (Postfix) with ESMTPS id D9558B00490; Tue, 14 Nov 2017 14:46:59 +0100 (CET) To: Nelio Laranjeiro Cc: dev@dpdk.org, Adrien Mazarguil , Yongseok Koh References: <2377aa87-fc6c-ee1e-91b8-ae226cb79a87@corp.free.fr> <20171114103402.6rr26vrlbbvcjnf6@laranjeiro-vm.dev.6wind.com> From: Olivier Gournet Message-ID: <17627bee-4044-bf56-2763-512c1c66b09b@corp.free.fr> Date: Tue, 14 Nov 2017 14:47:06 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: <20171114103402.6rr26vrlbbvcjnf6@laranjeiro-vm.dev.6wind.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-dev] mlx5 and secondary processes X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Nov 2017 13:47:00 -0000 Hi Nelio, On 11/14/2017 11:34 AM, Nelio Laranjeiro wrote: > Hi Olivier, > > On Tue, Nov 14, 2017 at 10:58:02AM +0100, Olivier Gournet wrote: >> Hi, >> >> I can't get TX on secondary process to works with the lastest dpdk, it was running fine with dpdk-16.11. >> RX/TX is ok on primary processs, and I'm not interested in RX on secondary process. Each process has its >> owns TX queues. >> >> I upgraded everything to: >> linux 4.14 >> dpdk 17.11-rc4 >> rdma-core from yesterday git-master >> mlx-fw 12.21.1000 >> >> On seconday process, TX queue gets full and is never emptied. >> It seems like bf_reg is correctly re-mapped from secondary process; here's dmesg from primary process: >> >> [...] >> [85361.895778] mlx5_1:uar_mmap:1722:(pid 9533): uar idx 0x6, pfn 0x00000000000f8020 >> [85361.895781] mlx5_1:uar_mmap:1735:(pid 9533): mapped best effort WC at 0x7ffff7faa000, PA 0x00000000f8020000 >> [85361.895784] mlx5_1:uar_mmap:1722:(pid 9533): uar idx 0x7, pfn 0x00000000000f8021 >> [85361.895787] mlx5_1:uar_mmap:1735:(pid 9533): mapped best effort WC at 0x7ffff7fa9000, PA 0x00000000f8021000 >> >> Then from secondary process: >> >> [...] >> [85408.038295] mlx5_1:mlx5_ib_mmap:1778:(pid 9551): mapped internal timer at 0x7ffff7f7f000, PA 0xf8001000 >> [85408.040229] mlx5_1:uar_mmap:1722:(pid 9551): uar idx 0x6, pfn 0x00000000000f8020 >> [85408.040233] mlx5_1:uar_mmap:1735:(pid 9551): mapped best effort WC at 0x7ffff7faa000, PA 0x00000000f8020000 >> [85408.040239] mlx5_1:uar_mmap:1722:(pid 9551): uar idx 0x7, pfn 0x00000000000f8021 >> [85408.040241] mlx5_1:uar_mmap:1735:(pid 9551): mapped best effort WC at 0x7ffff7fa9000, PA 0x00000000f8021000 >> >> But then from mlx5_rxtx.h:mlx5_tx_dbrec_cond_wmb() it doesn't seems like writing to txq->bf_reg has any effect. >> I know it isn't supposed to be read, but when reading from primary process is always give *dst==0xe5ccdabae5ccdaba, >> whereas from secondary process *dst is zero or the last written value. >> >> I don't really have any other clues, and don't know where to search. Can anybody give me some hints ? > > Since Yongseok patch [1] the UAR pages are no more configured though the > Write Combining of the PCI but in your logs it seems it is still using > the cache. > > Do you have the "MLX5_SHUT_UP_BF" set to 1 in your environment? If not > does adding it solves your issue? I set it (on both primary and secondary) and it is seen by kernel driver, but unfortunately it is still not working. [98321.605791] mlx5_1:uar_mmap:1735:(pid 9706): mapped NC at 0x7ffff7faa000, PA 0x00000000f8020000 [98321.605799] mlx5_1:uar_mmap:1735:(pid 9706): mapped NC at 0x7ffff7fa9000, PA 0x00000000f8021000 Thanks,