[dpdk-dev] possible kni bug and proposed fix

[dpdk-dev] possible kni bug and proposed fix

[ALeX Wang]

Hi,

When using the kni module to test my application inside
debian (virtualbox) VM (kernel version 4.4), I get the

"KNI: Out of memory"

from syslog every time I `tcpreply` packets through
the kni interface.

After checking source code, I saw that when I call
'rte_kni_rx_burst()', no matter how many packets
are actually retrieved, we always call 'kni_allocate_mbufs()'
and try allocate 'MAX_MBUF_BURST_NUM' more
mbufs...  I fix the issue via using this patch below,

Could you confirm if this is an actual bug?

diff --git a/lib/librte_kni/rte_kni.c b/lib/librte_kni/rte_kni.c
index ea9baf4..5d7c1ce 100644
--- a/lib/librte_kni/rte_kni.c
+++ b/lib/librte_kni/rte_kni.c
@@ -129,6 +129,7 @@ struct rte_kni_memzone_pool {

 static void kni_free_mbufs(struct rte_kni *kni);
 static void kni_allocate_mbufs(struct rte_kni *kni);
+static void kni_allocate_n_mbufs(struct rte_kni *kni, int size);

 static volatile int kni_fd = -1;
 static struct rte_kni_memzone_pool kni_memzone_pool = {
@@ -556,7 +557,7 @@ rte_kni_rx_burst(struct rte_kni *kni, struct rte_mbuf
**mbufs, unsigned num)

        /* If buffers removed, allocate mbufs and then put them into
alloc_q */
        if (ret)
-               kni_allocate_mbufs(kni);
+               kni_allocate_n_mbufs(kni, (int)ret);

        return ret;
 }
@@ -577,6 +578,12 @@ kni_free_mbufs(struct rte_kni *kni)
 static void
 kni_allocate_mbufs(struct rte_kni *kni)
 {
+       kni_allocate_n_mbufs(kni, MAX_MBUF_BURST_NUM);
+}
+
+static void
+kni_allocate_n_mbufs(struct rte_kni *kni, int size)
+{
        int i, ret;
        struct rte_mbuf *pkts[MAX_MBUF_BURST_NUM];

@@ -595,13 +602,18 @@ kni_allocate_mbufs(struct rte_kni *kni)
        RTE_BUILD_BUG_ON(offsetof(struct rte_mbuf, ol_flags) !=
                         offsetof(struct rte_kni_mbuf, ol_flags));

+       if (size > MAX_MBUF_BURST_NUM) {
+               RTE_LOG(ERR, KNI, "Invalid mbufs size\n");
+               return;
+       }
+
        /* Check if pktmbuf pool has been configured */
        if (kni->pktmbuf_pool == NULL) {
                RTE_LOG(ERR, KNI, "No valid mempool for allocating
mbufs\n");
                return;
        }

-       for (i = 0; i < MAX_MBUF_BURST_NUM; i++) {
+       for (i = 0; i < size; i++) {
                pkts[i] = rte_pktmbuf_alloc(kni->pktmbuf_pool);
                if (unlikely(pkts[i] == NULL)) {
                        /* Out of memory */

Thanks,
-- 
Alex Wang,

Re: [dpdk-dev] possible kni bug and proposed fix

[Ferruh Yigit]

On 5/15/2016 5:48 AM, ALeX Wang wrote:
> Hi,
> 
> When using the kni module to test my application inside
> debian (virtualbox) VM (kernel version 4.4), I get the
> 
> "KNI: Out of memory"
> 
> from syslog every time I `tcpreply` packets through
> the kni interface.
> 
> After checking source code, I saw that when I call
> 'rte_kni_rx_burst()', no matter how many packets
> are actually retrieved, we always call 'kni_allocate_mbufs()'
> and try allocate 'MAX_MBUF_BURST_NUM' more
> mbufs...  I fix the issue via using this patch below,
> 
> Could you confirm if this is an actual bug?
> 

Hi Alex,

I don't think this is a bug.

kni_allocate_mbufs() will allocate MAX_MBUF_BURST_NUM mbufs as you
mentioned. And will fill alloc_q with these mubfs _until it gets full_.
And if the alloc_q is full and there are remaining mbufs, they will be
freed. So for some cases this isn't the most optimized way, but there is
no defect.

Since you are getting "KNI: Out of memory", somewhere else can be
missing freeing mbufs.

mbufs freeing done in rte_kni_tx_burst(), I can guess two cases that can
cause problem:
a) not calling rte_kni_tx_burst() frequent, so that all free mbufs consumed.
b) calling rte_kni_tx_burst() with number of mbufs bigger than
MAX_MBUF_BURST_NUM, because this function frees at most
MAX_MBUF_BURST_NUM of mbufs, but if you are calling calling
rte_kni_tx_burst() with bigger numbers, this will cause mbufs to stuck
in free_q


Regards,
ferruh

Re: [dpdk-dev] possible kni bug and proposed fix

[ALeX Wang]

Hi Ferruh,

Thx for pointing out the 'fill alloc_q with these mubfs _until it gets
full_.',

I saw the size of all queues are 'KNI_FIFO_COUNT_MAX (1024)'...
The corresponding memory required is more than what I specify as
'socket_mem' (since i'm using VM)...

Also, in my use case, I only `tcpreplay` through the kni interface, and
my application only do rx and then free the 'mbufs'.  So there is no tx at
all.

So, in my case, I still think this is a bug/defect, or somewhere i still
misunderstand,

P.S. The description here seems to be inverted,
http://dpdk.org/doc/api/rte__kni_8h.html#a0cdd727cdc227d005fef22c0189f3dfe
'rte_kni_rx_burst' does the 'It handles allocating the mbufs for KNI
interface alloc queue.'

Thanks,
Alex Wang,

On 16 May 2016 at 04:20, Ferruh Yigit <ferruh.yigit@intel.com> wrote:

> On 5/15/2016 5:48 AM, ALeX Wang wrote:
> > Hi,
> >
> > When using the kni module to test my application inside
> > debian (virtualbox) VM (kernel version 4.4), I get the
> >
> > "KNI: Out of memory"
> >
> > from syslog every time I `tcpreply` packets through
> > the kni interface.
> >
> > After checking source code, I saw that when I call
> > 'rte_kni_rx_burst()', no matter how many packets
> > are actually retrieved, we always call 'kni_allocate_mbufs()'
> > and try allocate 'MAX_MBUF_BURST_NUM' more
> > mbufs...  I fix the issue via using this patch below,
> >
> > Could you confirm if this is an actual bug?
> >
>
> Hi Alex,
>
> I don't think this is a bug.
>
> kni_allocate_mbufs() will allocate MAX_MBUF_BURST_NUM mbufs as you
> mentioned. And will fill alloc_q with these mubfs _until it gets full_.
> And if the alloc_q is full and there are remaining mbufs, they will be
> freed. So for some cases this isn't the most optimized way, but there is
> no defect.
>
> Since you are getting "KNI: Out of memory", somewhere else can be
> missing freeing mbufs.
>
> mbufs freeing done in rte_kni_tx_burst(), I can guess two cases that can
> cause problem:
> a) not calling rte_kni_tx_burst() frequent, so that all free mbufs
> consumed.
> b) calling rte_kni_tx_burst() with number of mbufs bigger than
> MAX_MBUF_BURST_NUM, because this function frees at most
> MAX_MBUF_BURST_NUM of mbufs, but if you are calling calling
> rte_kni_tx_burst() with bigger numbers, this will cause mbufs to stuck
> in free_q
>
>
> Regards,
> ferruh
>
>
>


-- 
Alex Wang,
Open vSwitch developer

Re: [dpdk-dev] possible kni bug and proposed fix

[Ferruh Yigit]

On 5/16/2016 4:31 PM, ALeX Wang wrote:
> Hi Ferruh,
> 
> Thx for pointing out the 'fill alloc_q with these mubfs _until it gets
> full_.',
> 
> I saw the size of all queues are 'KNI_FIFO_COUNT_MAX (1024)'...
> The corresponding memory required is more than what I specify as
> 'socket_mem' (since i'm using VM)...
> 

If the mempool is not big enough to fill the ring, this explains the
error log. Logic is to fill the alloc_q, but if you are missing the
required mbufs, in each rte_kni_rx_burst() it will complain about memory.

But the required memory for mbufs to fill the ring is not too much. It
should be ~2Mbytes, are you sure you are missing this much memory?

> Also, in my use case, I only `tcpreplay` through the kni interface, and
> my application only do rx and then free the 'mbufs'.  So there is no tx
> at all.
> 
> So, in my case, I still think this is a bug/defect, or somewhere i still
> misunderstand,
> 
> P.S. The description here seems to be inverted,
> http://dpdk.org/doc/api/rte__kni_8h.html#a0cdd727cdc227d005fef22c0189f3dfe
> 'rte_kni_rx_burst' does the 'It handles allocating the mbufs for KNI
> interface alloc queue.'
> 

You are right. That part of the description for rte_kni_rx_burst and
rte_kni_tx_burst needs to be replaced. Do you want to send a patch?

> Thanks,
> Alex Wang,
> 
> On 16 May 2016 at 04:20, Ferruh Yigit <ferruh.yigit@intel.com
> <mailto:ferruh.yigit@intel.com>> wrote:
> 
>     On 5/15/2016 5:48 AM, ALeX Wang wrote:
>     > Hi,
>     >
>     > When using the kni module to test my application inside
>     > debian (virtualbox) VM (kernel version 4.4), I get the
>     >
>     > "KNI: Out of memory"
>     >
>     > from syslog every time I `tcpreply` packets through
>     > the kni interface.
>     >
>     > After checking source code, I saw that when I call
>     > 'rte_kni_rx_burst()', no matter how many packets
>     > are actually retrieved, we always call 'kni_allocate_mbufs()'
>     > and try allocate 'MAX_MBUF_BURST_NUM' more
>     > mbufs...  I fix the issue via using this patch below,
>     >
>     > Could you confirm if this is an actual bug?
>     >
> 
>     Hi Alex,
> 
>     I don't think this is a bug.
> 
>     kni_allocate_mbufs() will allocate MAX_MBUF_BURST_NUM mbufs as you
>     mentioned. And will fill alloc_q with these mubfs _until it gets full_.
>     And if the alloc_q is full and there are remaining mbufs, they will be
>     freed. So for some cases this isn't the most optimized way, but there is
>     no defect.
> 
>     Since you are getting "KNI: Out of memory", somewhere else can be
>     missing freeing mbufs.
> 
>     mbufs freeing done in rte_kni_tx_burst(), I can guess two cases that can
>     cause problem:
>     a) not calling rte_kni_tx_burst() frequent, so that all free mbufs
>     consumed.
>     b) calling rte_kni_tx_burst() with number of mbufs bigger than
>     MAX_MBUF_BURST_NUM, because this function frees at most
>     MAX_MBUF_BURST_NUM of mbufs, but if you are calling calling
>     rte_kni_tx_burst() with bigger numbers, this will cause mbufs to stuck
>     in free_q
> 
> 
>     Regards,
>     ferruh
> 
> 
> 
> 
> 
> -- 
> Alex Wang,
> Open vSwitch developer

Re: [dpdk-dev] possible kni bug and proposed fix

[ALeX Wang]

On 17 May 2016 at 03:07, Ferruh Yigit <ferruh.yigit@intel.com> wrote:

> On 5/16/2016 4:31 PM, ALeX Wang wrote:
> > Hi Ferruh,
> >
> > Thx for pointing out the 'fill alloc_q with these mubfs _until it gets
> > full_.',
> >
> > I saw the size of all queues are 'KNI_FIFO_COUNT_MAX (1024)'...
> > The corresponding memory required is more than what I specify as
> > 'socket_mem' (since i'm using VM)...
> >
>
> If the mempool is not big enough to fill the ring, this explains the
> error log. Logic is to fill the alloc_q, but if you are missing the
> required mbufs, in each rte_kni_rx_burst() it will complain about memory.
>
> But the required memory for mbufs to fill the ring is not too much. It
> should be ~2Mbytes, are you sure you are missing this much memory?
>
>

Thx for reminding me about this, by default, i only allocate 2048 mbufs for
my
pool...  once i raise it to 4096, the issue is gone...  will the
`KNI_FIFO_COUNT_MAX `
ever change?  I want to try adding some documentation,...


> > Also, in my use case, I only `tcpreplay` through the kni interface, and
> > my application only do rx and then free the 'mbufs'.  So there is no tx
> > at all.
> >
> > So, in my case, I still think this is a bug/defect, or somewhere i still
> > misunderstand,
> >
> > P.S. The description here seems to be inverted,
> >
> http://dpdk.org/doc/api/rte__kni_8h.html#a0cdd727cdc227d005fef22c0189f3dfe
> > 'rte_kni_rx_burst' does the 'It handles allocating the mbufs for KNI
> > interface alloc queue.'
> >
>
> You are right. That part of the description for rte_kni_rx_burst and
> rte_kni_tx_burst needs to be replaced. Do you want to send a patch?
>
>
Sure, will do that,

Thanks,
Alex Wang,



> > Thanks,
> > Alex Wang,
> >
> > On 16 May 2016 at 04:20, Ferruh Yigit <ferruh.yigit@intel.com
> > <mailto:ferruh.yigit@intel.com>> wrote:
> >
> >     On 5/15/2016 5:48 AM, ALeX Wang wrote:
> >     > Hi,
> >     >
> >     > When using the kni module to test my application inside
> >     > debian (virtualbox) VM (kernel version 4.4), I get the
> >     >
> >     > "KNI: Out of memory"
> >     >
> >     > from syslog every time I `tcpreply` packets through
> >     > the kni interface.
> >     >
> >     > After checking source code, I saw that when I call
> >     > 'rte_kni_rx_burst()', no matter how many packets
> >     > are actually retrieved, we always call 'kni_allocate_mbufs()'
> >     > and try allocate 'MAX_MBUF_BURST_NUM' more
> >     > mbufs...  I fix the issue via using this patch below,
> >     >
> >     > Could you confirm if this is an actual bug?
> >     >
> >
> >     Hi Alex,
> >
> >     I don't think this is a bug.
> >
> >     kni_allocate_mbufs() will allocate MAX_MBUF_BURST_NUM mbufs as you
> >     mentioned. And will fill alloc_q with these mubfs _until it gets
> full_.
> >     And if the alloc_q is full and there are remaining mbufs, they will
> be
> >     freed. So for some cases this isn't the most optimized way, but
> there is
> >     no defect.
> >
> >     Since you are getting "KNI: Out of memory", somewhere else can be
> >     missing freeing mbufs.
> >
> >     mbufs freeing done in rte_kni_tx_burst(), I can guess two cases that
> can
> >     cause problem:
> >     a) not calling rte_kni_tx_burst() frequent, so that all free mbufs
> >     consumed.
> >     b) calling rte_kni_tx_burst() with number of mbufs bigger than
> >     MAX_MBUF_BURST_NUM, because this function frees at most
> >     MAX_MBUF_BURST_NUM of mbufs, but if you are calling calling
> >     rte_kni_tx_burst() with bigger numbers, this will cause mbufs to
> stuck
> >     in free_q
> >
> >
> >     Regards,
> >     ferruh
> >
> >
> >
> >
> >
> > --
> > Alex Wang,
> > Open vSwitch developer
>
>


-- 
Alex Wang,
Open vSwitch developer