[dpdk-dev] virtio with 2MB hugepages - bringing back single file segments

[dpdk-dev] virtio with 2MB hugepages - bringing back single file segments

[Stojaczyk, DariuszX]

Hi,

I'm trying to make a vhost-user initiator built upon DPDK work with 2MB hugepages. In the initiator we have to share all memory with the host process, so it
can perform DMA. DPDK currently enforces having one descriptor per hugepage and there's an artificial limit of shared descriptors in DPDK vhost-user implementation (currently 8). Because of that, all DPDK vhost-user initiators are practically limited to 1GB hugepages at the moment. We can always increase the artificial descriptor limit, but then we're limited by sendmsg() itself, which on Linux accepts no more than 253 descriptors. However, could we increase the vhost-user implementation limit to - say - 128, and bring back "single file segments" [1]?

Could I send a patch series that does this? The single file segments code would go through a cleanup - at least making it available via a runtime option rather than #ifdefs.

I know there's an ongoing rework of the memory allocator in DPDK [2] and it includes a similar single file segments functionality. However, it will probably take quite some time before merged and even then, the new functionality would only be available in the *new* allocator. The old one is kept unchanged. It could use single file segments as well.

Regards,
D.

[1] http://dpdk.org/dev/patchwork/patch/16042/
[2] http://dpdk.org/ml/archives/dev/2017-December/084302.html 

Re: [dpdk-dev] virtio with 2MB hugepages - bringing back single file segments

[Tan, Jianfeng]

Hi Dariusz,

> -----Original Message-----
> From: Stojaczyk, DariuszX
> Sent: Friday, March 2, 2018 6:41 AM
> To: dev@dpdk.org; Tan, Jianfeng; Maxime Coquelin; Burakov, Anatoly;
> Yuanhan Liu
> Cc: Harris, James R; Thomas Monjalon
> Subject: virtio with 2MB hugepages - bringing back single file segments
> 
> Hi,
> 
> I'm trying to make a vhost-user initiator built upon DPDK work with 2MB
> hugepages. In the initiator we have to share all memory with the host
> process, so it
> can perform DMA. DPDK currently enforces having one descriptor per
> hugepage and there's an artificial limit of shared descriptors in DPDK vhost-
> user implementation (currently 8). Because of that, all DPDK vhost-user
> initiators are practically limited to 1GB hugepages at the moment. We can
> always increase the artificial descriptor limit, but then we're limited by
> sendmsg() itself, which on Linux accepts no more than 253 descriptors.
> However, could we increase the vhost-user implementation limit to - say -
> 128, and bring back "single file segments" [1]?

"Single file segments [1]" can help at the scenario that 2MB hugepages are not too scatter; i.e., some pages are physically contiguous.
But it cannot solve the issue completely (imagine the worst situation).

Plus, it makes the memory part a little complex.

So we are expecting it (with some other issues on memory part) being addressed completely by Anatoly's rework on memory.

Thanks,
Jianfeng

Re: [dpdk-dev] virtio with 2MB hugepages - bringing back single file segments

[Maxime Coquelin]

On 03/01/2018 11:40 PM, Stojaczyk, DariuszX wrote:
> Hi,
> 
> I'm trying to make a vhost-user initiator built upon DPDK work with 2MB hugepages. In the initiator we have to share all memory with the host process, so it
> can perform DMA. DPDK currently enforces having one descriptor per hugepage and there's an artificial limit of shared descriptors in DPDK vhost-user implementation (currently 8). Because of that, all DPDK vhost-user initiators are practically limited to 1GB hugepages at the moment. We can always increase the artificial descriptor limit, but then we're limited by sendmsg() itself, which on Linux accepts no more than 253 descriptors. However, could we increase the vhost-user implementation limit to - say - 128, and bring back "single file segments" [1]?

If you do something like this, you'll have first to update the
vhost-user spec, which should I think include a new protocol
feature bit.

Also, you will have to consider improving the translation functions
with a better search algorithm, else you'll have very poor performance.

> 
> Could I send a patch series that does this? The single file segments code would go through a cleanup - at least making it available via a runtime option rather than #ifdefs.
> 
> I know there's an ongoing rework of the memory allocator in DPDK [2] and it includes a similar single file segments functionality. However, it will probably take quite some time before merged and even then, the new functionality would only be available in the *new* allocator. The old one is kept unchanged. It could use single file segments as well.
> 
> Regards,
> D.
> 
> [1] http://dpdk.org/dev/patchwork/patch/16042/
> [2] http://dpdk.org/ml/archives/dev/2017-December/084302.html
> 

Re: [dpdk-dev] virtio with 2MB hugepages - bringing back single file segments

[Stojaczyk, DariuszX]

Hi Maxime,

> > Hi,
> >
> > I'm trying to make a vhost-user initiator built upon DPDK work with
> > 2MB hugepages. In the initiator we have to share all memory with the host
> process, so it can perform DMA. DPDK currently enforces having one descriptor
> per hugepage and there's an artificial limit of shared descriptors in DPDK
> vhost-user implementation (currently 8). Because of that, all DPDK vhost-user
> initiators are practically limited to 1GB hugepages at the moment. We can
> always increase the artificial descriptor limit, but then we're limited by
> sendmsg() itself, which on Linux accepts no more than 253 descriptors.
> However, could we increase the vhost-user implementation limit to - say - 128,
> and bring back "single file segments" [1]?
> 
> If you do something like this, you'll have first to update the vhost-user spec,
> which should I think include a new protocol feature bit. 

Do I? I can't see any memory region limitation in the vhost-user spec [1]. If a 128-region initiator tries to connect to an 8-region rte_vhost, the recvmsg() would simply fail - rte_vhost rejects `truncated` messages.

> 
> Also, you will have to consider improving the translation functions with a better
> search algorithm, else you'll have very poor performance.

That's right, I got that sorted out already.

Regards,
D.

[1] https://github.com/qemu/qemu/blob/master/docs/interop/vhost-user.txt

Re: [dpdk-dev] virtio with 2MB hugepages - bringing back single file segments

[Maxime Coquelin]

On 03/02/2018 10:03 AM, Stojaczyk, DariuszX wrote:
> Hi Maxime,
> 
>>> Hi,
>>>
>>> I'm trying to make a vhost-user initiator built upon DPDK work with
>>> 2MB hugepages. In the initiator we have to share all memory with the host
>> process, so it can perform DMA. DPDK currently enforces having one descriptor
>> per hugepage and there's an artificial limit of shared descriptors in DPDK
>> vhost-user implementation (currently 8). Because of that, all DPDK vhost-user
>> initiators are practically limited to 1GB hugepages at the moment. We can
>> always increase the artificial descriptor limit, but then we're limited by
>> sendmsg() itself, which on Linux accepts no more than 253 descriptors.
>> However, could we increase the vhost-user implementation limit to - say - 128,
>> and bring back "single file segments" [1]?
>>
>> If you do something like this, you'll have first to update the vhost-user spec,
>> which should I think include a new protocol feature bit.
> 
> Do I? I can't see any memory region limitation in the vhost-user spec [1]. If a 128-region initiator tries to connect to an 8-region rte_vhost, the recvmsg() would simply fail - rte_vhost rejects `truncated` messages.

Here it is:

* Memory regions description
    ---------------------------------------------------
    | num regions | padding | region0 | ... | region7 |
    ---------------------------------------------------

    Num regions: a 32-bit number of regions
    Padding: 32-bit


> 
>>
>> Also, you will have to consider improving the translation functions with a better
>> search algorithm, else you'll have very poor performance.
> 
> That's right, I got that sorted out already.
> 
> Regards,
> D.
> 
> [1] https://github.com/qemu/qemu/blob/master/docs/interop/vhost-user.txt
>