[dpdk-dev] [PATCH] virtio: fix allocating virtnet_rx not mem aligned

[dpdk-dev] [PATCH] virtio: fix allocating virtnet_rx not mem aligned

[Jianfeng Tan]

Compile DPDK with clang, below line in virtio_rxtx.c could be
optimized with four "VMOVAPS ymm, m256".
  memset(&rxvq->fake_mbuf, 0, sizeof(rxvq->fake_mbuf));

This instruction requires memory address is 32-byte aligned.
Or, it leads to segfault. Although only tested with Clang 3.6.0,
it can be reproduced in any compilers, which do aggressive
optimization, aka, change memset of known length to VMOVAPS.

The fact that struct rte_mbuf is cache line aligned, can only make
sure fake_mbuf is aligned compared to the start address of struct
virtnet_rx. Unfortunately, this address is not necessarily aligned
because it's allocated by:
  rxvq = (struct virtnet_rx *)RTE_PTR_ADD(vq, sz_vq);

When sz_vq is not aligned, then rxvq cannot be allocated with an
aligned address, and then rxvq->fake_mbuf (addr of rxvq + cache line
size) is not an aligned address.

The fix is very simple that making sz_vq 32-byte aligned. Here we
make it cache line aligned for future optimization.

Fixes: a900472aedef ("virtio: split virtio Rx/Tx queue")

Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
---
 drivers/net/virtio/virtio_ethdev.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/drivers/net/virtio/virtio_ethdev.c b/drivers/net/virtio/virtio_ethdev.c
index a995520..ad0f5a6 100644
--- a/drivers/net/virtio/virtio_ethdev.c
+++ b/drivers/net/virtio/virtio_ethdev.c
@@ -337,7 +337,10 @@ int virtio_dev_queue_setup(struct rte_eth_dev *dev,
 
 	snprintf(vq_name, sizeof(vq_name), "port%d_%s%d",
 		 dev->data->port_id, queue_names[queue_type], queue_idx);
-	sz_vq = sizeof(*vq) + vq_size * sizeof(struct vq_desc_extra);
+
+	sz_vq = RTE_ALIGN_CEIL(sizeof(*vq) +
+				vq_size * sizeof(struct vq_desc_extra),
+				RTE_CACHE_LINE_SIZE);
 	if (queue_type == VTNET_RQ) {
 		sz_q = sz_vq + sizeof(*rxvq);
 	} else if (queue_type == VTNET_TQ) {
-- 
2.1.4

Re: [dpdk-dev] [PATCH] virtio: fix allocating virtnet_rx not mem aligned

[Yuanhan Liu]

On Sun, Jun 12, 2016 at 02:29:42PM +0000, Jianfeng Tan wrote:
> Compile DPDK with clang, below line in virtio_rxtx.c could be
> optimized with four "VMOVAPS ymm, m256".
>   memset(&rxvq->fake_mbuf, 0, sizeof(rxvq->fake_mbuf));
> 
> This instruction requires memory address is 32-byte aligned.
> Or, it leads to segfault.

That looks like a dangerous optimization to me. If that's the case,
doesn't it mean we have to make sure the address is always aligned
properly while calling memset?

	--yliu

Re: [dpdk-dev] [PATCH] virtio: fix allocating virtnet_rx not mem aligned

[Yuanhan Liu]

On Mon, Jun 13, 2016 at 05:21:01PM +0800, Yuanhan Liu wrote:
> On Sun, Jun 12, 2016 at 02:29:42PM +0000, Jianfeng Tan wrote:
> > Compile DPDK with clang, below line in virtio_rxtx.c could be
> > optimized with four "VMOVAPS ymm, m256".
> >   memset(&rxvq->fake_mbuf, 0, sizeof(rxvq->fake_mbuf));
> > 
> > This instruction requires memory address is 32-byte aligned.
> > Or, it leads to segfault.
> 
> That looks like a dangerous optimization to me. If that's the case,
> doesn't it mean we have to make sure the address is always aligned
> properly while calling memset?

Above is just a side note. Anyway, I think making sure vq is cache
aligned is good here. So, I will apply it. BTW, do you mind if I
squash your 2 fixes into Huawei's Rx/Tx split commit? His commit is
not pushed to upstream yet, therefore I can still do rebase: I'm
thinking it's better to have one working commit other than one broken
commit followed with several fixing commits. And of course, I will
mention your contribution in the commit log.

	--yliu  

Re: [dpdk-dev] [PATCH] virtio: fix allocating virtnet_rx not mem aligned

[Tan, Jianfeng]

> -----Original Message-----
> From: Yuanhan Liu [mailto:yuanhan.liu@linux.intel.com]
> Sent: Monday, June 13, 2016 5:52 PM
> To: Tan, Jianfeng
> Cc: dev@dpdk.org; Xie, Huawei
> Subject: Re: [dpdk-dev] [PATCH] virtio: fix allocating virtnet_rx not mem
> aligned
> 
> On Mon, Jun 13, 2016 at 05:21:01PM +0800, Yuanhan Liu wrote:
> > On Sun, Jun 12, 2016 at 02:29:42PM +0000, Jianfeng Tan wrote:
> > > Compile DPDK with clang, below line in virtio_rxtx.c could be
> > > optimized with four "VMOVAPS ymm, m256".
> > >   memset(&rxvq->fake_mbuf, 0, sizeof(rxvq->fake_mbuf));
> > >
> > > This instruction requires memory address is 32-byte aligned.
> > > Or, it leads to segfault.
> >
> > That looks like a dangerous optimization to me. If that's the case,
> > doesn't it mean we have to make sure the address is always aligned
> > properly while calling memset?
> 
> Above is just a side note. Anyway, I think making sure vq is cache
> aligned is good here. So, I will apply it. BTW, do you mind if I
> squash your 2 fixes into Huawei's Rx/Tx split commit? His commit is
> not pushed to upstream yet, therefore I can still do rebase: I'm
> thinking it's better to have one working commit other than one broken
> commit followed with several fixing commits. And of course, I will
> mention your contribution in the commit log.

Not a problem from my side. But Huawei seems to have concerns on this fix. You can do that for the other fix firstly.

Thanks,
Jianfeng

> 
> 	--yliu

Re: [dpdk-dev] [PATCH] virtio: fix allocating virtnet_rx not mem aligned

[Tan, Jianfeng]

On 6/13/2016 5:21 PM, Yuanhan Liu wrote:
> On Sun, Jun 12, 2016 at 02:29:42PM +0000, Jianfeng Tan wrote:
>> Compile DPDK with clang, below line in virtio_rxtx.c could be
>> optimized with four "VMOVAPS ymm, m256".
>>    memset(&rxvq->fake_mbuf, 0, sizeof(rxvq->fake_mbuf));
>>
>> This instruction requires memory address is 32-byte aligned.
>> Or, it leads to segfault.
> That looks like a dangerous optimization to me.If that's the case,
> doesn't it mean we have to make sure the address is always aligned
> properly while calling memset?

I guess clang does such optimization when length is a 32-byte aligned 
immediate number. May need more information here.

Thanks,
Jianfeng

>
> 	--yliu

Re: [dpdk-dev] [PATCH] virtio: fix allocating virtnet_rx not mem aligned

[Yuanhan Liu]

On Mon, Jun 13, 2016 at 10:06:22AM +0000, Tan, Jianfeng wrote:
> 
> 
> > -----Original Message-----
> > From: Yuanhan Liu [mailto:yuanhan.liu@linux.intel.com]
> > Sent: Monday, June 13, 2016 5:52 PM
> > To: Tan, Jianfeng
> > Cc: dev@dpdk.org; Xie, Huawei
> > Subject: Re: [dpdk-dev] [PATCH] virtio: fix allocating virtnet_rx not mem
> > aligned
> > 
> > On Mon, Jun 13, 2016 at 05:21:01PM +0800, Yuanhan Liu wrote:
> > > On Sun, Jun 12, 2016 at 02:29:42PM +0000, Jianfeng Tan wrote:
> > > > Compile DPDK with clang, below line in virtio_rxtx.c could be
> > > > optimized with four "VMOVAPS ymm, m256".
> > > >   memset(&rxvq->fake_mbuf, 0, sizeof(rxvq->fake_mbuf));
> > > >
> > > > This instruction requires memory address is 32-byte aligned.
> > > > Or, it leads to segfault.
> > >
> > > That looks like a dangerous optimization to me. If that's the case,
> > > doesn't it mean we have to make sure the address is always aligned
> > > properly while calling memset?
> > 
> > Above is just a side note. Anyway, I think making sure vq is cache
> > aligned is good here. So, I will apply it. BTW, do you mind if I
> > squash your 2 fixes into Huawei's Rx/Tx split commit? His commit is
> > not pushed to upstream yet, therefore I can still do rebase: I'm
> > thinking it's better to have one working commit other than one broken
> > commit followed with several fixing commits. And of course, I will
> > mention your contribution in the commit log.
> 
> Not a problem from my side. But Huawei seems to have concerns on this fix. You can do that for the other fix firstly.

What's the concern? Despite the clang issue (I still have no idea why
clang would go that aggressively), I think making vq cache aligned is
generically a good idea.

	--yliu

Re: [dpdk-dev] [PATCH] virtio: fix allocating virtnet_rx not mem aligned

[Yuanhan Liu]

On Sun, Jun 12, 2016 at 02:29:42PM +0000, Jianfeng Tan wrote:
> Compile DPDK with clang, below line in virtio_rxtx.c could be
> optimized with four "VMOVAPS ymm, m256".
>   memset(&rxvq->fake_mbuf, 0, sizeof(rxvq->fake_mbuf));
> 
> This instruction requires memory address is 32-byte aligned.
> Or, it leads to segfault. Although only tested with Clang 3.6.0,
> it can be reproduced in any compilers, which do aggressive
> optimization, aka, change memset of known length to VMOVAPS.
> 
> The fact that struct rte_mbuf is cache line aligned, can only make
> sure fake_mbuf is aligned compared to the start address of struct
> virtnet_rx. Unfortunately, this address is not necessarily aligned
> because it's allocated by:
>   rxvq = (struct virtnet_rx *)RTE_PTR_ADD(vq, sz_vq);
> 
> When sz_vq is not aligned, then rxvq cannot be allocated with an
> aligned address, and then rxvq->fake_mbuf (addr of rxvq + cache line
> size) is not an aligned address.
> 
> The fix is very simple that making sz_vq 32-byte aligned. Here we
> make it cache line aligned for future optimization.
> 
> Fixes: a900472aedef ("virtio: split virtio Rx/Tx queue")

Folded this fix (and the other fix) to above commit, so that we could
have a clean working tree.

Thanks for good catch!

	--yliu