From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail.droids-corp.org (zoll.droids-corp.org [94.23.50.67]) by dpdk.org (Postfix) with ESMTP id DF1E62661 for ; Mon, 13 Aug 2018 23:54:29 +0200 (CEST) Received: from [37.168.129.239] (helo=droids-corp.org) by mail.droids-corp.org with esmtpsa (TLS1.0:RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1fpKo2-000498-13; Mon, 13 Aug 2018 23:55:19 +0200 Received: by droids-corp.org (sSMTP sendmail emulation); Mon, 13 Aug 2018 23:54:24 +0200 Date: Mon, 13 Aug 2018 23:54:24 +0200 From: Olivier Matz To: Matteo Lanzuisi Cc: dev@dpdk.org Message-ID: <20180813215424.cesdejskukrrt4zf@neon> References: MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: NeoMutt/20170113 (1.7.2) Subject: Re: [dpdk-dev] Multi-thread mempool usage X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Aug 2018 21:54:30 -0000 Hello Matteo, On Mon, Aug 13, 2018 at 03:20:44PM +0200, Matteo Lanzuisi wrote: > Any suggestion? any idea about this behaviour? > > Il 08/08/2018 11:56, Matteo Lanzuisi ha scritto: > > Hi all, > > > > recently I began using "dpdk-17.11-11.el7.x86_64" rpm (RedHat rpm) on > > RedHat 7.5 kernel 3.10.0-862.6.3.el7.x86_64 as a porting of an > > application from RH6 to RH7. On RH6 I used dpdk-2.2.0. > > > > This application is made up by one or more threads (each one on a > > different logical core) reading packets from i40e interfaces. > > > > Each thread can call the following code lines when receiving a specific > > packet: > > > > RTE_LCORE_FOREACH(lcore_id) > > { > >         result = > > rte_mempool_get(cea_main_lcore_conf[lcore_id].de_conf.cmd_pool, (VOID_P > > *) &new_work);        // mempools are created one for each logical core > >         if (((uint64_t)(new_work)) < 0x7f0000000000) > >             printf("Result %d, lcore di partenza %u, lcore di ricezione > > %u, pointer %p\n", result, rte_lcore_id(), lcore_id, new_work);    // > > debug print, on my server it should never happen but with multi-thread > > happens always on the last logical core!!!! Here, checking the value of new_work looks wrong to me, before ensuring that result == 0. At least, new_work should be set to NULL before calling rte_mempool_get(). > >         if (result == 0) > >         { > >             new_work->command = command; // usage of the memory gotten > > from the mempool... <<<<<- here is where the application crashes!!!! Do you know why it crashes? Is it that new_work is NULL? Can you check how the mempool is initialized? It should be in multi consumer and depending on your use case, single or multi producer. Another thing that could be checked: at all the places where you return your work object to the mempool, you should add a check that it is not NULL. Or just enabling RTE_LIBRTE_MEMPOOL_DEBUG could do the trick: it adds some additional checks when doing mempool operations. > >             result = > > rte_ring_enqueue(cea_main_lcore_conf[lcore_id].de_conf.cmd_ring, > > (VOID_P) new_work);    // enqueues the gotten buffer on the rings of all > > lcores > >             // check on result value ... > >         } > >         else > >         { > >             // do something if result != 0 ... > >         } > > } > > > > This code worked perfectly (never had an issue) on dpdk-2.2.0, while if > > I use more than 1 thread doing these operations on dpdk-17.11 it happens > > that after some times the "new_work" pointer is not a good one, and the > > application crashes when using that pointer. > > > > It seems that these lines cannot be used by more than one thread > > simultaneously. I also used many 2017 and 2018 dpdk versions without > > success. > > > > Is this code possible on the new dpdk versions? Or have I to change my > > application so that this code is called just by one lcore at a time? Assuming the mempool is properly initialized, I don't see any reason why it would not work. There has been a lot of changes in mempool between dpdk-2.2.0 and dpdk-17.11, but this behavior should remain the same. If the comments above do not help to solve the issue, it could be helpful to try to reproduce the issue in a minimal program, so we can help to review it. Regards, Olivier