From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-f66.google.com (mail-wm0-f66.google.com [74.125.82.66]) by dpdk.org (Postfix) with ESMTP id 406AD23C for ; Thu, 20 Jul 2017 16:06:14 +0200 (CEST) Received: by mail-wm0-f66.google.com with SMTP id m4so3683493wmi.4 for ; Thu, 20 Jul 2017 07:06:13 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=jVNRoQiF1zKrvfSVjSNUQCFUYd+X9QEM4fIX2UXu6kg=; b=Zu96y/Noy9nKPR92kMb8uCPPK+BlxB7TCpekHirNcGc/lvGy67G4kb70vLtNdQy2SM liQem0/vCpfKhQUT7esQzCB8L1wwB7dKIyNKx8YSimJUk0PMMIkW8i6TBzlK72fcKgv2 OPortkNPbsnY4dxd04Q5K863XKwwg1uamwRURj1S7JHbxkh9QlpO6x8yXiq5evAaPRAF oxT60e6VMHuXa1WNpAQicEzhUHjYj2wkJ1tVO1ZgRJCikPSLn8Sz+xUaP1Cb7wijFkQ1 1ZMJHM/tKdWt4Bkm4nx2y3HVgHUQUq4pmVKNZUJXjZEZ+F3W2thTW3RnUWjhwROcITT5 90MA== X-Gm-Message-State: AIVw110eLJpF2W71Plbtrc8IS6lRwYzI3mQJokImeb1wPIiD1/x0wX9R 5EYirhxRAI+nhw== X-Received: by 10.28.157.81 with SMTP id g78mr2411104wme.76.1500559573677; Thu, 20 Jul 2017 07:06:13 -0700 (PDT) Received: from [192.168.64.116] (bzq-82-81-101-184.red.bezeqint.net. [82.81.101.184]) by smtp.gmail.com with ESMTPSA id p136sm2285689wmd.39.2017.07.20.07.06.12 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 20 Jul 2017 07:06:13 -0700 (PDT) To: =?UTF-8?Q?N=c3=a9lio_Laranjeiro?= , Shahaf Shuler Cc: dev@dpdk.org, Yongseok Koh , Roy Shterman , Alexander Solganik , Leon Romanovsky References: <75d08202-1882-7660-924c-b6dbb4455b88@grimberg.me> <20170717210222.j4dwxiujqdlqhlp2@shalom> <85c0b1d9-bbf3-c6ab-727f-f508c5e5f584@grimberg.me> <20170720135548.qlkjnmzthd2vep5e@shalom> From: Sagi Grimberg Message-ID: <957f84fe-199a-e016-c21c-ab54a732e561@grimberg.me> Date: Thu, 20 Jul 2017 17:06:11 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.2.1 MIME-Version: 1.0 In-Reply-To: <20170720135548.qlkjnmzthd2vep5e@shalom> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-dev] Question on mlx5 PMD txq memory registration X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Jul 2017 14:06:14 -0000 >> Its worse than just a drop, without debug enabled the error completion >> is ignored, and the wqe_pi is taken from an invalid field, which leads >> to bogus mbufs free (elts_tail is not valid). > > Right A simple work-around would be to simply fill a correct tail so that error completions will still have it (although I'm not sure the which fields are reliable other than the status in error completions). >> Not sure why it needs a lock at all. it *may* need an rcu protection >> or rw_lock if at all. > > Tx queues may run on several CPU there is a need to be sure this array > cannot be modified by two threads at the same time. Anyway it is > costly. As I said, there are primitives which are designed to handle frequent reads and rare mutations. >> AFAICT, all this caching mechanism is just working around the fact >> that mlx5 allocates resources on top of the existing verbs interface. >> I think it should work like any other pmd driver, i.e. use mbuf the >> physical addresses. >> >> The mlx5 device (like all other rdma devices) has a global DMA lkey that >> spans the entire physical address space. Just about all the kernel >> drivers heavily use this lkey. IMO, the mlx5_pmd driver should be able >> to query the kernel what this lkey is and ask for the kernel to create >> the QP with privilege level to post send/recv operations with that lkey. >> >> And then, mlx5_pmd becomes like other drivers working with physical >> addresses instead of working around the memory registration sub-optimally. > > It is one possibility discussed also with Mellanox guys, the point is > this breaks the security point of view which is also an important stuff. What security aspect? The entire dpdk model builds on top of physical addresses awareness running under root permissions. I'm not saying exposing it to the application nor granting remote permissions to the physical space. mlx5_pmd is a network driver, and as a driver, it should allowed to use the device dma lkey as it sees fit. I honestly think its pretty much mandatory considering the work-arounds mlx5_pmd tries to do (which we agreed are broken). > If this is added in the future it will certainly be as an option, this > way both will be possible, the application could then choose about > security vs performance. Why should the application even be aware of that? Does any other driver expose the user how it maps pkt mbufs to the NIC? Just like the MR handling, its 100% internal to mlx5 and no reason why the user should ever be exposed to any of these details. > I don't know any planning on this from Mellanox side, maybe Shahaf have. rdma-core has a very nice vendor extension mechanism (which is important because we really don't want to pollute the verbs API just for dpdk). Its very easy to expose the dma lkey and create the TX queue-pairs with reserved lkey attributes via this mechanism. Just the kernel needs to verify root permissions before exposing it. >> And while were on the subject, what is the plan of detaching mlx5_pmd >> from its MLNX_OFED dependency? Mellanox has been doing a good job >> upstreaming the needed features (rdma-core). CC'ing Leon (who is >> co-maintaining the user-space rdma tree. > > This is also a in progress in PMD part, it should be part of the next > DPDK release. That is *very* good to hear! Can you guys share a branch? I'm willing to take it for testing.