Re: [dpdk-dev] Question on mlx5 PMD txq memory registration

[Sagi Grimberg <sagi@grimberg.me>]

>> As I said, there are primitives which are designed to handle frequent reads
>> and rare mutations.
> 
> Even with such primitives, rarely lock is more than never lock.

You do realize that the cache mutation involves ibv_dereg_mr() right?
Any locking scheme for mutation is negligible compared to that, and
rcu_read_lock is practically free.

>>> It is one possibility discussed also with Mellanox guys, the point is
>>> this breaks the security point of view which is also an important stuff.
>>
>> What security aspect? The entire dpdk model builds on top of physical
>> addresses awareness running under root permissions.
>> I'm not saying
>> exposing it to the application nor granting remote permissions to the physical
>> space.
>>
>> mlx5_pmd is a network driver, and as a driver, it should allowed to use the
>> device dma lkey as it sees fit. I honestly think its pretty much mandatory
>> considering the work-arounds mlx5_pmd tries to do (which we agreed are
>> broken).
> 
> True, There are many PMDs which can work only with physical memory.
> However Mellanox NICs have the option to work with virtual one thus provide more security.

I don't understand the security argument. Its completely private to the
driver. anything under librte is equivalent to an OS wrt networking, so
I fail to see what is the security feature your talking about.

> The fact running under root doesn't mean you have privileges to access every physical page on the server (even if you try very hard to be aware).

But dpdk core mbufs structs are built this way.

> The issue here, AFAIU, is performance.
> We are now looking into ways to provide the same performance as if it was only a single lkey, while preserving the security feature.

Hmm, What exactly do you have in mind?

I'm hoping that you are not referring to ODP. If you are, I think
that latency unpredictability would be a huge non-starter, page-faults
are way too expensive for dpdk users.

Again, personally, I don't think that using virtual memory registration
are very useful for a network driver, It's a driver, not an application.

>>> If this is added in the future it will certainly be as an option, this
>>> way both will be possible, the application could then choose about
>>> security vs performance.
>>
>> Why should the application even be aware of that? Does any other driver
>> expose the user how it maps pkt mbufs to the NIC? Just like the MR
>> handling, its 100% internal to mlx5 and no reason why the user should ever
>> be exposed to any of these details.
> 
> Other option is the reserved lkey as you suggested, but it will lose the security guarantees.

OK, obviously I'm missing something. Can you articulate what do you mean
by "security guarantees"?

> Like every performance optimization it should be the application decision.

I tend to disagree with you on this. The application must never be aware
of the nitty details of the underlying driver implementation. In fact,
propagating this information to the application sounds like a layering
violation.

> In fact, there are some discussions on the ML of exposing the option to use va instead of pa. [1]

My understanding is that the correct proposal was to hide this
knowledge from the user. Also the use-case is not security (which
I'm not even sure what it means yet).

>>> This is also a in progress in PMD part, it should be part of the next
>>> DPDK release.
>>
>> That is *very* good to hear! Can you guys share a branch? I'm willing to take
>> it for testing.
> 
> The branch is still pre-mature. It may be good enough for external testing in about two weeks.
> Contact me directly and I will provide it to you.

Awesome!

Thanks.