From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 13F0942B39 for ; Thu, 18 May 2023 10:16:55 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id CFE0140E25; Thu, 18 May 2023 10:16:54 +0200 (CEST) Received: from mail-wr1-f50.google.com (mail-wr1-f50.google.com [209.85.221.50]) by mails.dpdk.org (Postfix) with ESMTP id 00D364014F for ; Thu, 18 May 2023 10:16:53 +0200 (CEST) Received: by mail-wr1-f50.google.com with SMTP id ffacd0b85a97d-3093a778089so1137967f8f.1 for ; Thu, 18 May 2023 01:16:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684397813; x=1686989813; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=tdcQdf+AUpumZJqziLSgJ+cltjwsCG5dHxgRPlvWuOE=; b=MXySET9QA5gMNrWodf/gYPLBjChF+Rdg5GGcJEKRb8y4wa0phpY2qaGvGH+iauVaXC r10/7gHEogeWXDJPqa7dgUNc2gJp8a4gbcoO9R5h83EX41zWRpWU3VdjpR3R96hQY79I QO8lMUcoroMbItBjh4X3oDyHPqTCQceNyHFOE7nqf+PdSWnqmLO2eifV/VdJ2OzYUWAI hLqBMUv3NsHp7zwFkliQzhTFzoIyoIfXLb6X5HeQmgiuo+1WrIL5vQ8YXP7aj3qyA8LH 1Y9v9gAItmAfXGkN/6fZX3YaOkkBwlXF3yUUyqwaCCC3EYNHpf7ANiARC/w6wgamZb+g jOXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684397813; x=1686989813; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=tdcQdf+AUpumZJqziLSgJ+cltjwsCG5dHxgRPlvWuOE=; b=e3BiJGA/04Kr1ULqmiEWpUB9dgzz02gmngjPDnWwLTZisb7YWtkTJgwx+q5JdcOkIY 8FWnIB+ddE4JSl3jmWRuTvSSUN/Rb53l5DadcqmMgzfcOgQHMJ00CKGCJmxEqgzLjXMc /fBBlbTq6LHMPflGLNoCBgkwWFPw0YcYYr1xg5BAURGC8Nw+7/JiESlLmOpLfu8W6f96 gvnCaijtu/oDYSuWeLts9mc1U3kqHUqFuTDACG0X9kdm5xUtwav1KZGzJwdc9CPRzFOI +NG9gR5I5+xJ4XsoanKXRL9/qTDLRfmVBiNr8qcRj1KiBEwr6D/xfJCwpMp7B6NvAsjJ /Rlg== X-Gm-Message-State: AC+VfDwtIkMfPRFC42VzjN0dlRFctUgZUD9zO2I1hbc1AkGU7mppEOkw qdSBzyYykuRS/u7xH1eY8iqlCVG3OkyuLLXdMPU= X-Google-Smtp-Source: ACHHUZ6yl9869wYxiR6WLPK8pErgYy76EvkLVvesUBXJGw3wqvDMsi/bQVNNaLgsig9ly8ZrozcYAym9c43yaquXxLc= X-Received: by 2002:adf:df0d:0:b0:306:2b5f:8d0a with SMTP id y13-20020adfdf0d000000b003062b5f8d0amr821791wrl.56.1684397813354; Thu, 18 May 2023 01:16:53 -0700 (PDT) MIME-Version: 1.0 References: <20230508091845.646caf17@hermes.local> In-Reply-To: From: Yasin CANER Date: Thu, 18 May 2023 11:14:58 +0300 Message-ID: Subject: Re: DPDK 22.11 - How to fix memory leak for KNI - How to debug To: Ferruh Yigit Cc: Stephen Hemminger , users@dpdk.org Content-Type: multipart/alternative; boundary="00000000000053550f05fbf36d13" X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org --00000000000053550f05fbf36d13 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hello Ferruh, Thanks for your kind response. Also thanks to Stephen. Even if 1 packet is consumed from the kernel , each time rx_kni allocates another 32 units. After a while all mempool is used in alloc_q from kni. there is not any room for it. Do you think my mistake is using one and common mempool usage both kni and eth? If it needs a separate mempool , i'd like to note in docs. Best regards. Ferruh Yigit , 17 May 2023 =C3=87ar, 20:53 tarihinde = =C5=9Funu yazd=C4=B1: > On 5/9/2023 12:13 PM, Yasin CANER wrote: > > Hello, > > > > I draw a flow via asciiflow to explain myself better. Problem is after > > transmitting packets(mbufs) , it never puts in the kni->free_q to back > > to the original pool. Each cycle, it allocates another 32 units that > > cause leaks. Or I am missing something. > > > > I already tried the rte_eth_tx_done_cleanup() function but it didn't fi= x > > anything. > > > > I am working on a patch to fix this issue but I am not sure if there > > is another way. > > > > Best regards. > > > > https://pastebin.ubuntu.com/p/s4h5psqtgZ/ > > > > > > > > unsigned > > rte_kni_rx_burst(struct rte_kni *kni, struct rte_mbuf **mbufs, unsigned > > int num) > > { > > unsigned int ret =3D kni_fifo_get(kni->tx_q, (void **)mbufs, num); > > > > /* If buffers removed, allocate mbufs and then put them into alloc_q */ > > /* Question, how to test buffers is removed or not?*/ > > if (ret) > > kni_allocate_mbufs(kni); > > > > return ret; > > } > > > > Selam Yasin, > > > You can expect 'kni->alloc_q' fifo to be full, this is not a memory leak. > > As you pointed out, number of mbufs consumed by kernel from 'alloc_q' > and number of mbufs added to 'alloc_q' is not equal and this is expected. > > Target here is to prevent buffer underflow from kernel perspective, so > it will always have available mbufs for new packets. > That is why new mbufs are added to 'alloc_q' at worst same or sometimes > higher rate than it is consumed. > > You should calculate your mbuf requirement with the assumption that > 'kni->alloc_q' will be full of mbufs. > > > 'kni->alloc_q' is freed when kni is removed. > Since 'alloc_q' holds physical address of the mbufs, it is a little > challenging to free them in the userspace, that is why first kernel > tries to move mbufs to 'kni->free_q' fifo, please check > 'kni_net_release_fifo_phy()' for it. > > If all moved to 'free_q' fifo, nothing left to in 'alloc_q', but if not, > userspace frees 'alloc_q' in 'rte_kni_release()', with following call: > `kni_free_fifo_phy(kni->pktmbuf_pool, kni->alloc_q);` > > > I can see you have submitted fixes for this issue, although as I > explained above I don't think a defect exist, I will review them > today/tomorrow. > > Regards, > Ferruh > > > > Stephen Hemminger > >, 8 May 2023 Pzt, 19:18 tarihinde > > =C5=9Funu yazd=C4=B1: > > > > On Mon, 8 May 2023 09:01:41 +0300 > > Yasin CANER > > > wrote: > > > > > Hello Stephen, > > > > > > Thank you for response, it helps me a lot. I understand problem > > better. > > > > > > After reading mbuf library ( > > > https://doc.dpdk.org/guides/prog_guide/mempool_lib.html > > ) i > > realized that > > > 31 units allocation memory slot doesn't return to pool! > > > > If receive burst returns 1 mbuf, the other 31 pointers in the array > > are not valid. They do not point to mbufs. > > > > > 1 unit mbuf can be freed via rte_pktmbuf_free so it can back to > pool. > > > > > > Main problem is that allocation doesn't return to original pool, > > act as > > > used. So, after following rte_pktmbuf_free > > > > > < > http://doc.dpdk.org/api/rte__mbuf_8h.html#a1215458932900b7cd5192326fa4a69= 02 > < > http://doc.dpdk.org/api/rte__mbuf_8h.html#a1215458932900b7cd5192326fa4a69= 02 > >> > > > function, > > > i realized that there is 2 function to helps to mbufs back to poo= l. > > > > > > These are rte_mbuf_raw_free > > > > > < > http://doc.dpdk.org/api/rte__mbuf_8h.html#a9f188d53834978aca01ea101576d74= 32 > < > http://doc.dpdk.org/api/rte__mbuf_8h.html#a9f188d53834978aca01ea101576d74= 32 > >> > > > and rte_pktmbuf_free_seg > > > > > < > http://doc.dpdk.org/api/rte__mbuf_8h.html#a006ee80357a78fbb9ada2b0432f82f= 37 > < > http://doc.dpdk.org/api/rte__mbuf_8h.html#a006ee80357a78fbb9ada2b0432f82f= 37 > >>. > > > I will focus on them. > > > > > > If there is another suggestion, I will be very pleased. > > > > > > Best regards. > > > > > > Yasin CANER > > > Ulak > > > > --00000000000053550f05fbf36d13 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hello Ferruh,

Thanks for your kind response. Also = thanks to Stephen.

Even if 1 packet is consumed fr= om the kernel , each time rx_kni allocates another 32 units. After a while = all mempool is used in alloc_q from kni. there is not any room for it.

Do you think my mistake is using one and common mempoo= l usage both kni and eth?

If it needs a separate= =C2=A0mempool , i'd like to note in docs.

Best= regards.

Ferruh Yigit <fe= rruh.yigit@amd.com>, 17 May 2023 =C3=87ar, 20:53 tarihinde =C5=9Funu= yazd=C4=B1:
On = 5/9/2023 12:13 PM, Yasin CANER wrote:
> Hello,
>
> I draw a flow via asciiflow to explain myself better. Problem is after=
> transmitting=C2=A0packets(mbufs) , it never puts in the kni->free_q= to back
> to the original pool. Each cycle, it allocates another 32 units that > cause leaks. Or I am missing something.
>
> I already tried the rte_eth_tx_done_cleanup() function but it didn'= ;t fix
> anything.
>
> I am working on a=C2=A0patch to fix this issue but I am not sure if th= ere
> is=C2=A0another way.
>
> Best regards.
>
> https://pastebin.ubuntu.com/p/s4h5psqtgZ/
> <https://pastebin.ubuntu.com/p/s4h5psqtgZ/><= br> >
>
> unsigned
> rte_kni_rx_burst(struct rte_kni *kni, struct rte_mbuf **mbufs, unsigne= d
> int num)
> {
> unsigned int ret =3D kni_fifo_get(kni->tx_q, (void **)mbufs, num);<= br> >
> /* If buffers removed, allocate mbufs and then put them into alloc_q *= /
> /* Question, how to test buffers is removed or not?*/
> if (ret)
> =C2=A0 =C2=A0 kni_allocate_mbufs(kni);
>
> return ret;
> }
>

Selam Yasin,


You can expect 'kni->alloc_q' fifo to be full, this is not a mem= ory leak.

As you pointed out, number of mbufs consumed by kernel from 'alloc_q= 9;
and number of mbufs added to 'alloc_q' is not equal and this is exp= ected.

Target here is to prevent buffer underflow from kernel perspective, so
it will always have available mbufs for new packets.
That is why new mbufs are added to 'alloc_q' at worst same or somet= imes
higher rate than it is consumed.

You should calculate your mbuf requirement with the assumption that
'kni->alloc_q' will be full of mbufs.


'kni->alloc_q' is freed when kni is removed.
Since 'alloc_q' holds physical address of the mbufs, it is a little=
challenging to free them in the userspace, that is why first kernel
tries to move mbufs to 'kni->free_q' fifo, please check
'kni_net_release_fifo_phy()' for it.

If all moved to 'free_q' fifo, nothing left to in 'alloc_q'= , but if not,
userspace frees 'alloc_q' in 'rte_kni_release()', with foll= owing call:
`kni_free_fifo_phy(kni->pktmbuf_pool, kni->alloc_q);`


I can see you have submitted fixes for this issue, although as I
explained above I don't think a defect exist, I will review them
today/tomorrow.

Regards,
Ferruh


> Stephen Hemminger <stephen@networkplumber.org
> <mailto:stephen@networkplumber.org>>, 8 May 2023 Pzt, 19:18 tarihind= e
> =C5=9Funu yazd=C4=B1:
>
>=C2=A0 =C2=A0 =C2=A0On Mon, 8 May 2023 09:01:41 +0300
>=C2=A0 =C2=A0 =C2=A0Yasin CANER <yasinncaner@gmail.com <mailto:yasinncaner@gmail.com>&= gt;
>=C2=A0 =C2=A0 =C2=A0wrote:
>
>=C2=A0 =C2=A0 =C2=A0> Hello Stephen,
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0> Thank you for response, it helps me a lot. I u= nderstand problem
>=C2=A0 =C2=A0 =C2=A0better.
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0> After reading mbuf library (
>=C2=A0 =C2=A0 =C2=A0> https://doc.dpdk= .org/guides/prog_guide/mempool_lib.html
>=C2=A0 =C2=A0 =C2=A0<https://doc.dpdk.= org/guides/prog_guide/mempool_lib.html>)=C2=A0 i
>=C2=A0 =C2=A0 =C2=A0realized that
>=C2=A0 =C2=A0 =C2=A0> 31 units allocation memory slot doesn't re= turn to pool!
>
>=C2=A0 =C2=A0 =C2=A0If receive burst returns 1 mbuf, the other 31 point= ers in the array
>=C2=A0 =C2=A0 =C2=A0are not valid. They do not point to mbufs.
>
>=C2=A0 =C2=A0 =C2=A0> 1 unit mbuf can be freed via rte_pktmbuf_free = so it can back to pool.
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0> Main problem is that allocation doesn't re= turn to original pool,
>=C2=A0 =C2=A0 =C2=A0act as
>=C2=A0 =C2=A0 =C2=A0> used. So, after following rte_pktmbuf_free
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0<http://doc.dpdk.org/api/rte__mbuf_8h.html#a1215458932900b7cd5192326fa4a6= 902 <http://doc.dp= dk.org/api/rte__mbuf_8h.html#a1215458932900b7cd5192326fa4a6902>><= br> >=C2=A0 =C2=A0 =C2=A0> function,
>=C2=A0 =C2=A0 =C2=A0> i realized that there is 2 function to helps t= o mbufs back to pool.
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0> These are rte_mbuf_raw_free
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0<http://doc.dpdk.org/api/rte__mbuf_8h.html#a9f188d53834978aca01ea101576d7= 432 <http://doc.dp= dk.org/api/rte__mbuf_8h.html#a9f188d53834978aca01ea101576d7432>><= br> >=C2=A0 =C2=A0 =C2=A0>=C2=A0 and rte_pktmbuf_free_seg
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0<http://doc.dpdk.org/api/rte__mbuf_8h.html#a006ee80357a78fbb9ada2b0432f82= f37 <http://doc.dp= dk.org/api/rte__mbuf_8h.html#a006ee80357a78fbb9ada2b0432f82f37>>.=
>=C2=A0 =C2=A0 =C2=A0> I will focus on them.
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0> If there is another suggestion, I will be very= pleased.
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0> Best regards.
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0> Yasin CANER
>=C2=A0 =C2=A0 =C2=A0> Ulak
>

--00000000000053550f05fbf36d13--