[dpdk-dev] [PATCH] vhost: Check descriptor number for vector Rx

[dpdk-dev] [PATCH] vhost: Check descriptor number for vector Rx

[Ouyang Changchun]

For zero copy, it need check whether RX descriptor num meets the 
least requirement when using vector PMD Rx function, and give user 
more hints if it fails to meet the least requirement.

Signed-off-by: Changchun Ouyang <changchun.ouyang@intel.com>
---
 examples/vhost/main.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/examples/vhost/main.c b/examples/vhost/main.c
index 291128e..87ab854 100644
--- a/examples/vhost/main.c
+++ b/examples/vhost/main.c
@@ -131,6 +131,10 @@
 #define RTE_TEST_RX_DESC_DEFAULT_ZCP 32   /* legacy: 32, DPDK virt FE: 128. */
 #define RTE_TEST_TX_DESC_DEFAULT_ZCP 64   /* legacy: 64, DPDK virt FE: 64.  */
 
+#ifdef RTE_IXGBE_INC_VECTOR
+#define VPMD_RX_BURST         32
+#endif
+
 /* Get first 4 bytes in mbuf headroom. */
 #define MBUF_HEADROOM_UINT32(mbuf) (*(uint32_t *)((uint8_t *)(mbuf) \
 		+ sizeof(struct rte_mbuf)))
@@ -792,6 +796,19 @@ us_vhost_parse_args(int argc, char **argv)
 		return -1;
 	}
 
+#ifdef RTE_IXGBE_INC_VECTOR
+	if ((zero_copy == 1) && (num_rx_descriptor <= VPMD_RX_BURST)) {
+		RTE_LOG(INFO, VHOST_PORT,
+			"The RX desc num: %d is too small for PMD to work\n"
+			"properly, please enlarge it to bigger than %d if\n"
+			"possible by the option: '--rx-desc-num <number>'\n"
+			"One alternative is disabling RTE_IXGBE_INC_VECTOR\n"
+			"in config file and rebuild the libraries.\n",
+			num_rx_descriptor, VPMD_RX_BURST);
+		return -1;
+	}
+#endif
+
 	return 0;
 }
 
-- 
1.8.4.2

Re: [dpdk-dev] [PATCH] vhost: Check descriptor number for vector Rx

[Thomas Monjalon]

Hi Changchun,

2014-10-24 16:38, Ouyang Changchun:
> For zero copy, it need check whether RX descriptor num meets the 
> least requirement when using vector PMD Rx function, and give user 
> more hints if it fails to meet the least requirement.
[...]
> --- a/examples/vhost/main.c
> +++ b/examples/vhost/main.c
> @@ -131,6 +131,10 @@
>  #define RTE_TEST_RX_DESC_DEFAULT_ZCP 32   /* legacy: 32, DPDK virt FE: 128. */
>  #define RTE_TEST_TX_DESC_DEFAULT_ZCP 64   /* legacy: 64, DPDK virt FE: 64.  */
>  
> +#ifdef RTE_IXGBE_INC_VECTOR
> +#define VPMD_RX_BURST         32
> +#endif
> +
>  /* Get first 4 bytes in mbuf headroom. */
>  #define MBUF_HEADROOM_UINT32(mbuf) (*(uint32_t *)((uint8_t *)(mbuf) \
>  		+ sizeof(struct rte_mbuf)))
> @@ -792,6 +796,19 @@ us_vhost_parse_args(int argc, char **argv)
>  		return -1;
>  	}
>  
> +#ifdef RTE_IXGBE_INC_VECTOR
> +	if ((zero_copy == 1) && (num_rx_descriptor <= VPMD_RX_BURST)) {
> +		RTE_LOG(INFO, VHOST_PORT,
> +			"The RX desc num: %d is too small for PMD to work\n"
> +			"properly, please enlarge it to bigger than %d if\n"
> +			"possible by the option: '--rx-desc-num <number>'\n"
> +			"One alternative is disabling RTE_IXGBE_INC_VECTOR\n"
> +			"in config file and rebuild the libraries.\n",
> +			num_rx_descriptor, VPMD_RX_BURST);
> +		return -1;
> +	}
> +#endif
> +
>  	return 0;
>  }

I feel there is a design problem here.
An application shouldn't have to care about the underlying driver.

-- 
Thomas

Re: [dpdk-dev] [PATCH] vhost: Check descriptor number for vector Rx

[Ouyang, Changchun]

Hi Thomas, 

Thanks for your comments  and my response as below.

> -----Original Message-----
> From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]
> Sent: Friday, October 24, 2014 5:28 PM
> To: Ouyang, Changchun
> Cc: dev@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH] vhost: Check descriptor number for vector
> Rx
> 
> Hi Changchun,
> 
> 2014-10-24 16:38, Ouyang Changchun:
> > For zero copy, it need check whether RX descriptor num meets the least
> > requirement when using vector PMD Rx function, and give user more
> > hints if it fails to meet the least requirement.
> [...]
> > --- a/examples/vhost/main.c
> > +++ b/examples/vhost/main.c
> > @@ -131,6 +131,10 @@
> >  #define RTE_TEST_RX_DESC_DEFAULT_ZCP 32   /* legacy: 32, DPDK virt FE:
> 128. */
> >  #define RTE_TEST_TX_DESC_DEFAULT_ZCP 64   /* legacy: 64, DPDK virt FE:
> 64.  */
> >
> > +#ifdef RTE_IXGBE_INC_VECTOR
> > +#define VPMD_RX_BURST         32
> > +#endif
> > +
> >  /* Get first 4 bytes in mbuf headroom. */  #define
> > MBUF_HEADROOM_UINT32(mbuf) (*(uint32_t *)((uint8_t *)(mbuf) \
> >  		+ sizeof(struct rte_mbuf)))
> > @@ -792,6 +796,19 @@ us_vhost_parse_args(int argc, char **argv)
> >  		return -1;
> >  	}
> >
> > +#ifdef RTE_IXGBE_INC_VECTOR
> > +	if ((zero_copy == 1) && (num_rx_descriptor <= VPMD_RX_BURST)) {
> > +		RTE_LOG(INFO, VHOST_PORT,
> > +			"The RX desc num: %d is too small for PMD to
> work\n"
> > +			"properly, please enlarge it to bigger than %d if\n"
> > +			"possible by the option: '--rx-desc-num
> <number>'\n"
> > +			"One alternative is disabling
> RTE_IXGBE_INC_VECTOR\n"
> > +			"in config file and rebuild the libraries.\n",
> > +			num_rx_descriptor, VPMD_RX_BURST);
> > +		return -1;
> > +	}
> > +#endif
> > +
> >  	return 0;
> >  }
> 
> I feel there is a design problem here.
> An application shouldn't have to care about the underlying driver.
> 

For most of other applications, as their descriptor numbers are set as big enough(1024 or so) ,
So there is no need to check the descriptor number at the early stage of running.

But for vhost zero copy(note vhost one copy also has 1024 descriptor number) has the default 
descriptor number of 32.
Why use 32? 
because vhost zero copy implementation (working as backend) need support dpdk based app which use pmd virtio-net driver,
And also need support linux legacy virtio-net based application.  
When it is the linux legacy virtio-net case, on one side the qemu has hard code to confine the total virtio descriptor size to 256, 
On other side, legacy virtio use half of them as virtio header, and then only another half i.e. 128 descriptors are available to use as real buffer.

In PMD mode, all HW descriptors need to be filled DMA address in the rx initial stage, otherwise there is probably exceptional in rx process.
Based on that, we need use really limited virtio buffer to fully fill all hw descriptor DMA address,
Or in other word, the available virtio descriptor size will determine the total mbuf size and hw descriptor size in the case of zero copy,

Tune and find that 32 is the suitable value for vhost zero copy to work properly when it legacy linux virtio case.
Another factor to reduce the value to 32, is that mempool use ring to accommodate the mbuf, it cost one to flag the ring head/tail,
And there are some other overheads like temporary mbufs(size as RX_BURST) when rx.
Note that number descriptor should need power 2.   

Why the change occur at this moment?
Recently the default rx function is modified into vector RX function, while it use non-vector mode (scalar mode) Rx previously,
Vector RX function need more than 32 descriptor to work properly,  but scalar mode RX hasn't this limitation.

As the RX function is changeable(you can use vector mode or non-vector), and descriptor number can also be changed.
So here in the vhost app, check if they match to make sure all things could work normally, and give some hints if they don't match.

Hope the above could make it a bit clearer. :-)
Thanks again,
Best regards,
Changchun

Re: [dpdk-dev] [PATCH] vhost: Check descriptor number for vector Rx

[Thomas Monjalon]

2014-10-25 00:48, Ouyang, Changchun:
> From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]
> > 2014-10-24 16:38, Ouyang Changchun:
> > > For zero copy, it need check whether RX descriptor num meets the least
> > > requirement when using vector PMD Rx function, and give user more
> > > hints if it fails to meet the least requirement.
> > [...]
> > > --- a/examples/vhost/main.c
> > > +++ b/examples/vhost/main.c
> > > @@ -131,6 +131,10 @@
> > >  #define RTE_TEST_RX_DESC_DEFAULT_ZCP 32   /* legacy: 32, DPDK virt FE: 128. */
> > >  #define RTE_TEST_TX_DESC_DEFAULT_ZCP 64   /* legacy: 64, DPDK virt FE: 64.  */
> > >
> > > +#ifdef RTE_IXGBE_INC_VECTOR
> > > +#define VPMD_RX_BURST         32
> > > +#endif
> > > +
> > >  /* Get first 4 bytes in mbuf headroom. */  #define
> > > MBUF_HEADROOM_UINT32(mbuf) (*(uint32_t *)((uint8_t *)(mbuf) \
> > >  		+ sizeof(struct rte_mbuf)))
> > > @@ -792,6 +796,19 @@ us_vhost_parse_args(int argc, char **argv)
> > >  		return -1;
> > >  	}
> > >
> > > +#ifdef RTE_IXGBE_INC_VECTOR
> > > +	if ((zero_copy == 1) && (num_rx_descriptor <= VPMD_RX_BURST)) {
> > > +		RTE_LOG(INFO, VHOST_PORT,
> > > +			"The RX desc num: %d is too small for PMD to work\n"
> > > +			"properly, please enlarge it to bigger than %d if\n"
> > > +			"possible by the option: '--rx-desc-num <number>'\n"
> > > +			"One alternative is disabling RTE_IXGBE_INC_VECTOR\n"
> > > +			"in config file and rebuild the libraries.\n",
> > > +			num_rx_descriptor, VPMD_RX_BURST);
> > > +		return -1;
> > > +	}
> > > +#endif
> > > +
> > >  	return 0;
> > >  }
> > 
> > I feel there is a design problem here.
> > An application shouldn't have to care about the underlying driver.
> 
> For most of other applications, as their descriptor numbers are set as big enough(1024 or so) ,
> So there is no need to check the descriptor number at the early stage of running.
> 
> But for vhost zero copy(note vhost one copy also has 1024 descriptor number) has the default 
> descriptor number of 32.
> Why use 32? 
> because vhost zero copy implementation (working as backend) need support dpdk based app which use pmd virtio-net driver,
> And also need support linux legacy virtio-net based application.  
> When it is the linux legacy virtio-net case, on one side the qemu has hard code to confine the total virtio descriptor size to 256, 
> On other side, legacy virtio use half of them as virtio header, and then only another half i.e. 128 descriptors are available to use as real buffer.
> 
> In PMD mode, all HW descriptors need to be filled DMA address in the rx initial stage, otherwise there is probably exceptional in rx process.
> Based on that, we need use really limited virtio buffer to fully fill all hw descriptor DMA address,
> Or in other word, the available virtio descriptor size will determine the total mbuf size and hw descriptor size in the case of zero copy,
> 
> Tune and find that 32 is the suitable value for vhost zero copy to work properly when it legacy linux virtio case.
> Another factor to reduce the value to 32, is that mempool use ring to accommodate the mbuf, it cost one to flag the ring head/tail,
> And there are some other overheads like temporary mbufs(size as RX_BURST) when rx.
> Note that number descriptor should need power 2.   
> 
> Why the change occur at this moment?
> Recently the default rx function is modified into vector RX function, while it use non-vector mode (scalar mode) Rx previously,
> Vector RX function need more than 32 descriptor to work properly,  but scalar mode RX hasn't this limitation.
> 
> As the RX function is changeable(you can use vector mode or non-vector), and descriptor number can also be changed.
> So here in the vhost app, check if they match to make sure all things could work normally, and give some hints if they don't match.
> 
> Hope the above could make it a bit clearer. :-)

Thank you for your explanation.
Your fix shows that driver and application are tightly linked.
It's a design issue. As I said:
"An application shouldn't have to care about the underlying driver."
I didn't dig enough in vhost to suggest a good fix but I'm sure
someone could have an idea.

-- 
Thomas

Re: [dpdk-dev] [PATCH] vhost: Check descriptor number for vector Rx

[Ouyang, Changchun]

Hi Thomas,

> -----Original Message-----
> From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]
> Sent: Monday, October 27, 2014 4:46 PM
> To: Ouyang, Changchun
> Cc: dev@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH] vhost: Check descriptor number for vector
> Rx
> 
> 2014-10-25 00:48, Ouyang, Changchun:
> > From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]
> > > 2014-10-24 16:38, Ouyang Changchun:
> > > > For zero copy, it need check whether RX descriptor num meets the
> > > > least requirement when using vector PMD Rx function, and give user
> > > > more hints if it fails to meet the least requirement.
> > > [...]
> > > > --- a/examples/vhost/main.c
> > > > +++ b/examples/vhost/main.c
> > > > @@ -131,6 +131,10 @@
> > > >  #define RTE_TEST_RX_DESC_DEFAULT_ZCP 32   /* legacy: 32, DPDK virt
> FE: 128. */
> > > >  #define RTE_TEST_TX_DESC_DEFAULT_ZCP 64   /* legacy: 64, DPDK virt
> FE: 64.  */
> > > >
> > > > +#ifdef RTE_IXGBE_INC_VECTOR
> > > > +#define VPMD_RX_BURST         32
> > > > +#endif
> > > > +
> > > >  /* Get first 4 bytes in mbuf headroom. */  #define
> > > > MBUF_HEADROOM_UINT32(mbuf) (*(uint32_t *)((uint8_t *)(mbuf) \
> > > >  		+ sizeof(struct rte_mbuf)))
> > > > @@ -792,6 +796,19 @@ us_vhost_parse_args(int argc, char **argv)
> > > >  		return -1;
> > > >  	}
> > > >
> > > > +#ifdef RTE_IXGBE_INC_VECTOR
> > > > +	if ((zero_copy == 1) && (num_rx_descriptor <= VPMD_RX_BURST)) {
> > > > +		RTE_LOG(INFO, VHOST_PORT,
> > > > +			"The RX desc num: %d is too small for PMD to
> work\n"
> > > > +			"properly, please enlarge it to bigger than %d if\n"
> > > > +			"possible by the option: '--rx-desc-num
> <number>'\n"
> > > > +			"One alternative is disabling
> RTE_IXGBE_INC_VECTOR\n"
> > > > +			"in config file and rebuild the libraries.\n",
> > > > +			num_rx_descriptor, VPMD_RX_BURST);
> > > > +		return -1;
> > > > +	}
> > > > +#endif
> > > > +
> > > >  	return 0;
> > > >  }
> > >
> > > I feel there is a design problem here.
> > > An application shouldn't have to care about the underlying driver.
> >
> > For most of other applications, as their descriptor numbers are set as
> > big enough(1024 or so) , So there is no need to check the descriptor
> number at the early stage of running.
> >
> > But for vhost zero copy(note vhost one copy also has 1024 descriptor
> > number) has the default descriptor number of 32.
> > Why use 32?
> > because vhost zero copy implementation (working as backend) need
> > support dpdk based app which use pmd virtio-net driver, And also need
> support linux legacy virtio-net based application.
> > When it is the linux legacy virtio-net case, on one side the qemu has
> > hard code to confine the total virtio descriptor size to 256, On other side,
> legacy virtio use half of them as virtio header, and then only another half i.e.
> 128 descriptors are available to use as real buffer.
> >
> > In PMD mode, all HW descriptors need to be filled DMA address in the rx
> initial stage, otherwise there is probably exceptional in rx process.
> > Based on that, we need use really limited virtio buffer to fully fill
> > all hw descriptor DMA address, Or in other word, the available virtio
> > descriptor size will determine the total mbuf size and hw descriptor
> > size in the case of zero copy,
> >
> > Tune and find that 32 is the suitable value for vhost zero copy to work
> properly when it legacy linux virtio case.
> > Another factor to reduce the value to 32, is that mempool use ring to
> > accommodate the mbuf, it cost one to flag the ring head/tail, And there are
> some other overheads like temporary mbufs(size as RX_BURST) when rx.
> > Note that number descriptor should need power 2.
> >
> > Why the change occur at this moment?
> > Recently the default rx function is modified into vector RX function,
> > while it use non-vector mode (scalar mode) Rx previously, Vector RX
> function need more than 32 descriptor to work properly,  but scalar mode RX
> hasn't this limitation.
> >
> > As the RX function is changeable(you can use vector mode or non-vector),
> and descriptor number can also be changed.
> > So here in the vhost app, check if they match to make sure all things could
> work normally, and give some hints if they don't match.
> >
> > Hope the above could make it a bit clearer. :-)
> 
> Thank you for your explanation.
> Your fix shows that driver and application are tightly linked.
> It's a design issue. As I said:
> "An application shouldn't have to care about the underlying driver."
> I didn't dig enough in vhost to suggest a good fix but I'm sure someone could
> have an idea.
>
Agree with you, there is something linked between app and driver, but that's due to a few things:
1.Qume has hard code to confine the total vring size;
2.PMD driver need fully fill the dma address of descriptor at setup stage;
3.PMD driver use vector PMD as default path which require more than 32 in the burst RX;
4.Zero copy need use external buffer directly and set to HW descriptor dma address;

Except for item 3, everything could not be removed or ignored easily.
As for item 3, I don't know why we set vector PMD as default path, while perusing performance, but at the cost of flexibility,
Here is an example, vhost zero copy need check it and descriptor number. I am not sure if the default vector pmd path will affect other app or not.

On the other hand, without this check, and hint to user,
User will suffer from this, vhost can't receive packets any more after forwarding  31 packets or so,
and user don't know what really happen behind, also can't easily get the information of 
their descriptor is not enough for vector pmd RX.
Then this is followed by painfully debugging and figuring out the issue.

While with this check and hint to user, things go smoothly.

Meanwhile we can waiting for other people's viewpoints.

Thanks
Changchun

Re: [dpdk-dev] [PATCH] vhost: Check descriptor number for vector Rx

[Xie, Huawei]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Ouyang, Changchun
> Sent: Monday, October 27, 2014 6:56 AM
> To: Thomas Monjalon
> Cc: dev@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH] vhost: Check descriptor number for vector Rx
> 
> Hi Thomas,
> 
> > -----Original Message-----
> > From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]
> > Sent: Monday, October 27, 2014 4:46 PM
> > To: Ouyang, Changchun
> > Cc: dev@dpdk.org
> > Subject: Re: [dpdk-dev] [PATCH] vhost: Check descriptor number for vector
> > Rx
> >
> > 2014-10-25 00:48, Ouyang, Changchun:
> > > From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]
> > > > 2014-10-24 16:38, Ouyang Changchun:
> > > > > For zero copy, it need check whether RX descriptor num meets the
> > > > > least requirement when using vector PMD Rx function, and give user
> > > > > more hints if it fails to meet the least requirement.
> > > > [...]
> > > > > --- a/examples/vhost/main.c
> > > > > +++ b/examples/vhost/main.c
> > > > > @@ -131,6 +131,10 @@
> > > > >  #define RTE_TEST_RX_DESC_DEFAULT_ZCP 32   /* legacy: 32, DPDK virt
> > FE: 128. */
> > > > >  #define RTE_TEST_TX_DESC_DEFAULT_ZCP 64   /* legacy: 64, DPDK virt
> > FE: 64.  */
> > > > >
> > > > > +#ifdef RTE_IXGBE_INC_VECTOR
> > > > > +#define VPMD_RX_BURST         32
> > > > > +#endif
> > > > > +
> > > > >  /* Get first 4 bytes in mbuf headroom. */  #define
> > > > > MBUF_HEADROOM_UINT32(mbuf) (*(uint32_t *)((uint8_t *)(mbuf) \
> > > > >  		+ sizeof(struct rte_mbuf)))
> > > > > @@ -792,6 +796,19 @@ us_vhost_parse_args(int argc, char **argv)
> > > > >  		return -1;
> > > > >  	}
> > > > >
> > > > > +#ifdef RTE_IXGBE_INC_VECTOR
> > > > > +	if ((zero_copy == 1) && (num_rx_descriptor <=
> VPMD_RX_BURST)) {
> > > > > +		RTE_LOG(INFO, VHOST_PORT,
> > > > > +			"The RX desc num: %d is too small for PMD to
> > work\n"
> > > > > +			"properly, please enlarge it to bigger than %d
> if\n"
> > > > > +			"possible by the option: '--rx-desc-num
> > <number>'\n"
> > > > > +			"One alternative is disabling
> > RTE_IXGBE_INC_VECTOR\n"
> > > > > +			"in config file and rebuild the libraries.\n",
> > > > > +			num_rx_descriptor, VPMD_RX_BURST);
> > > > > +		return -1;
> > > > > +	}
> > > > > +#endif
> > > > > +
> > > > >  	return 0;
> > > > >  }
> > > >
> > > > I feel there is a design problem here.
> > > > An application shouldn't have to care about the underlying driver.
> > >
> > > For most of other applications, as their descriptor numbers are set as
> > > big enough(1024 or so) , So there is no need to check the descriptor
> > number at the early stage of running.
> > >
> > > But for vhost zero copy(note vhost one copy also has 1024 descriptor
> > > number) has the default descriptor number of 32.
> > > Why use 32?
> > > because vhost zero copy implementation (working as backend) need
> > > support dpdk based app which use pmd virtio-net driver, And also need
> > support linux legacy virtio-net based application.
> > > When it is the linux legacy virtio-net case, on one side the qemu has
> > > hard code to confine the total virtio descriptor size to 256, On other side,
> > legacy virtio use half of them as virtio header, and then only another half i.e.
> > 128 descriptors are available to use as real buffer.
> > >
> > > In PMD mode, all HW descriptors need to be filled DMA address in the rx
> > initial stage, otherwise there is probably exceptional in rx process.
> > > Based on that, we need use really limited virtio buffer to fully fill
> > > all hw descriptor DMA address, Or in other word, the available virtio
> > > descriptor size will determine the total mbuf size and hw descriptor
> > > size in the case of zero copy,
> > >
> > > Tune and find that 32 is the suitable value for vhost zero copy to work
> > properly when it legacy linux virtio case.
> > > Another factor to reduce the value to 32, is that mempool use ring to
> > > accommodate the mbuf, it cost one to flag the ring head/tail, And there are
> > some other overheads like temporary mbufs(size as RX_BURST) when rx.
> > > Note that number descriptor should need power 2.
> > >
> > > Why the change occur at this moment?
> > > Recently the default rx function is modified into vector RX function,
> > > while it use non-vector mode (scalar mode) Rx previously, Vector RX
> > function need more than 32 descriptor to work properly,  but scalar mode RX
> > hasn't this limitation.
> > >
> > > As the RX function is changeable(you can use vector mode or non-vector),
> > and descriptor number can also be changed.
> > > So here in the vhost app, check if they match to make sure all things could
> > work normally, and give some hints if they don't match.
> > >
> > > Hope the above could make it a bit clearer. :-)
> >
> > Thank you for your explanation.
> > Your fix shows that driver and application are tightly linked.
> > It's a design issue. As I said:
> > "An application shouldn't have to care about the underlying driver."
> > I didn't dig enough in vhost to suggest a good fix but I'm sure someone could
> > have an idea.
> >
> Agree with you, there is something linked between app and driver, but that's due
> to a few things:
> 1.Qume has hard code to confine the total vring size;
> 2.PMD driver need fully fill the dma address of descriptor at setup stage;

Can PMD driver fill its desc ring until all has been filled or  there is no mbuf in pool?

> 3.PMD driver use vector PMD as default path which require more than 32 in the
> burst RX;
> 4.Zero copy need use external buffer directly and set to HW descriptor dma
> address;
> 
> Except for item 3, everything could not be removed or ignored easily.
> As for item 3, I don't know why we set vector PMD as default path, while
> perusing performance, but at the cost of flexibility,
> Here is an example, vhost zero copy need check it and descriptor number. I am
> not sure if the default vector pmd path will affect other app or not.
> 
> On the other hand, without this check, and hint to user,
> User will suffer from this, vhost can't receive packets any more after forwarding
> 31 packets or so,
> and user don't know what really happen behind, also can't easily get the
> information of
> their descriptor is not enough for vector pmd RX.
> Then this is followed by painfully debugging and figuring out the issue.
> 
> While with this check and hint to user, things go smoothly.
> 
> Meanwhile we can waiting for other people's viewpoints.
> 
> Thanks
> Changchun

Re: [dpdk-dev] [PATCH] vhost: Check descriptor number for vector Rx

[Ouyang, Changchun]

Hi,

> -----Original Message-----
> From: Xie, Huawei
> Sent: Thursday, October 30, 2014 7:37 AM
> To: Ouyang, Changchun; Thomas Monjalon
> Cc: dev@dpdk.org
> Subject: RE: [dpdk-dev] [PATCH] vhost: Check descriptor number for vector
> Rx
> 
> 
> 
> > -----Original Message-----
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Ouyang,
> Changchun
> > Sent: Monday, October 27, 2014 6:56 AM
> > To: Thomas Monjalon
> > Cc: dev@dpdk.org
> > Subject: Re: [dpdk-dev] [PATCH] vhost: Check descriptor number for
> > vector Rx
> >
> > Hi Thomas,
> >
> > > -----Original Message-----
> > > From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]
> > > Sent: Monday, October 27, 2014 4:46 PM
> > > To: Ouyang, Changchun
> > > Cc: dev@dpdk.org
> > > Subject: Re: [dpdk-dev] [PATCH] vhost: Check descriptor number for
> > > vector Rx
> > >
> > > 2014-10-25 00:48, Ouyang, Changchun:
> > > > From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]
> > > > > 2014-10-24 16:38, Ouyang Changchun:
> > > > > > For zero copy, it need check whether RX descriptor num meets
> > > > > > the least requirement when using vector PMD Rx function, and
> > > > > > give user more hints if it fails to meet the least requirement.
> > > > > [...]
> > > > > > --- a/examples/vhost/main.c
> > > > > > +++ b/examples/vhost/main.c
> > > > > > @@ -131,6 +131,10 @@
> > > > > >  #define RTE_TEST_RX_DESC_DEFAULT_ZCP 32   /* legacy: 32, DPDK
> virt
> > > FE: 128. */
> > > > > >  #define RTE_TEST_TX_DESC_DEFAULT_ZCP 64   /* legacy: 64, DPDK
> virt
> > > FE: 64.  */
> > > > > >
> > > > > > +#ifdef RTE_IXGBE_INC_VECTOR
> > > > > > +#define VPMD_RX_BURST         32
> > > > > > +#endif
> > > > > > +
> > > > > >  /* Get first 4 bytes in mbuf headroom. */  #define
> > > > > > MBUF_HEADROOM_UINT32(mbuf) (*(uint32_t *)((uint8_t *)(mbuf)
> \
> > > > > >  		+ sizeof(struct rte_mbuf))) @@ -792,6 +796,19 @@
> > > > > > us_vhost_parse_args(int argc, char **argv)
> > > > > >  		return -1;
> > > > > >  	}
> > > > > >
> > > > > > +#ifdef RTE_IXGBE_INC_VECTOR
> > > > > > +	if ((zero_copy == 1) && (num_rx_descriptor <=
> > VPMD_RX_BURST)) {
> > > > > > +		RTE_LOG(INFO, VHOST_PORT,
> > > > > > +			"The RX desc num: %d is too small for PMD to
> > > work\n"
> > > > > > +			"properly, please enlarge it to bigger than %d
> > if\n"
> > > > > > +			"possible by the option: '--rx-desc-num
> > > <number>'\n"
> > > > > > +			"One alternative is disabling
> > > RTE_IXGBE_INC_VECTOR\n"
> > > > > > +			"in config file and rebuild the libraries.\n",
> > > > > > +			num_rx_descriptor, VPMD_RX_BURST);
> > > > > > +		return -1;
> > > > > > +	}
> > > > > > +#endif
> > > > > > +
> > > > > >  	return 0;
> > > > > >  }
> > > > >
> > > > > I feel there is a design problem here.
> > > > > An application shouldn't have to care about the underlying driver.
> > > >
> > > > For most of other applications, as their descriptor numbers are
> > > > set as big enough(1024 or so) , So there is no need to check the
> > > > descriptor
> > > number at the early stage of running.
> > > >
> > > > But for vhost zero copy(note vhost one copy also has 1024
> > > > descriptor
> > > > number) has the default descriptor number of 32.
> > > > Why use 32?
> > > > because vhost zero copy implementation (working as backend) need
> > > > support dpdk based app which use pmd virtio-net driver, And also
> > > > need
> > > support linux legacy virtio-net based application.
> > > > When it is the linux legacy virtio-net case, on one side the qemu
> > > > has hard code to confine the total virtio descriptor size to 256,
> > > > On other side,
> > > legacy virtio use half of them as virtio header, and then only another half
> i.e.
> > > 128 descriptors are available to use as real buffer.
> > > >
> > > > In PMD mode, all HW descriptors need to be filled DMA address in
> > > > the rx
> > > initial stage, otherwise there is probably exceptional in rx process.
> > > > Based on that, we need use really limited virtio buffer to fully
> > > > fill all hw descriptor DMA address, Or in other word, the
> > > > available virtio descriptor size will determine the total mbuf
> > > > size and hw descriptor size in the case of zero copy,
> > > >
> > > > Tune and find that 32 is the suitable value for vhost zero copy to
> > > > work
> > > properly when it legacy linux virtio case.
> > > > Another factor to reduce the value to 32, is that mempool use ring
> > > > to accommodate the mbuf, it cost one to flag the ring head/tail,
> > > > And there are
> > > some other overheads like temporary mbufs(size as RX_BURST) when rx.
> > > > Note that number descriptor should need power 2.
> > > >
> > > > Why the change occur at this moment?
> > > > Recently the default rx function is modified into vector RX
> > > > function, while it use non-vector mode (scalar mode) Rx
> > > > previously, Vector RX
> > > function need more than 32 descriptor to work properly,  but scalar
> > > mode RX hasn't this limitation.
> > > >
> > > > As the RX function is changeable(you can use vector mode or
> > > > non-vector),
> > > and descriptor number can also be changed.
> > > > So here in the vhost app, check if they match to make sure all
> > > > things could
> > > work normally, and give some hints if they don't match.
> > > >
> > > > Hope the above could make it a bit clearer. :-)
> > >
> > > Thank you for your explanation.
> > > Your fix shows that driver and application are tightly linked.
> > > It's a design issue. As I said:
> > > "An application shouldn't have to care about the underlying driver."
> > > I didn't dig enough in vhost to suggest a good fix but I'm sure
> > > someone could have an idea.
> > >
> > Agree with you, there is something linked between app and driver, but
> > that's due to a few things:
> > 1.Qume has hard code to confine the total vring size; 2.PMD driver
> > need fully fill the dma address of descriptor at setup stage;
> 
> Can PMD driver fill its desc ring until all has been filled or  there is no mbuf in
> pool?
> 
Currently not,
If you mean can PMD be modified into that functionality, possibly yes.
It should be another topic about how to improve  our PMD.

> > 3.PMD driver use vector PMD as default path which require more than 32
> > in the burst RX; 4.Zero copy need use external buffer directly and set
> > to HW descriptor dma address;
> >
> > Except for item 3, everything could not be removed or ignored easily.
> > As for item 3, I don't know why we set vector PMD as default path,
> > while perusing performance, but at the cost of flexibility, Here is an
> > example, vhost zero copy need check it and descriptor number. I am not
> > sure if the default vector pmd path will affect other app or not.
> >
> > On the other hand, without this check, and hint to user, User will
> > suffer from this, vhost can't receive packets any more after
> > forwarding
> > 31 packets or so,
> > and user don't know what really happen behind, also can't easily get
> > the information of their descriptor is not enough for vector pmd RX.
> > Then this is followed by painfully debugging and figuring out the issue.
> >
> > While with this check and hint to user, things go smoothly.
> >
> > Meanwhile we can waiting for other people's viewpoints.
> >
> > Thanks
> > Changchun