From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 6C58B45F9B for ; Sat, 4 Jan 2025 20:17:00 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id EA83240278; Sat, 4 Jan 2025 20:16:59 +0100 (CET) Received: from mail-oo1-f42.google.com (mail-oo1-f42.google.com [209.85.161.42]) by mails.dpdk.org (Postfix) with ESMTP id 49F324014F for ; Sat, 4 Jan 2025 20:16:59 +0100 (CET) Received: by mail-oo1-f42.google.com with SMTP id 006d021491bc7-5f31b3db5ecso5578122eaf.0 for ; Sat, 04 Jan 2025 11:16:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1736018218; x=1736623018; darn=dpdk.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=TWcWkya8wC+6E3v4KbOeTboeriKScfTn3WVGx4sWm70=; b=d18SvSPsO6u/iM6wr68ZOo8b8NRPNLZasbw22LiS9tdZL9/snOfh/oV+CJH1ReIj8f 8JEix+SCTRLKRIVXgXmWTce7GrKQ1awtXlrfdPO05A0SJUQxMDGWD74Pw+G6F8Iqh8qH DxUJKDaDIom2LOcMJtbKD+ff07+9rfOScvmfggOlDIjIQ2U1iG+Xw6KrqNlzoqz7tjol xyouMbY+LG787IRZCfX9yLCWa4zPZc8ttuosAw2cGdpttBh6wtD8ExAxxhGQV2fksp3y VjhBdSLaNDQRYDDDQPk6a3I61cb18aEKh+jnDnHxEGD/WGGCe018rQzVmvKki3vwGYYC mRMg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736018218; x=1736623018; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=TWcWkya8wC+6E3v4KbOeTboeriKScfTn3WVGx4sWm70=; b=Fv50DPfgN9pIze3H8P22nvU4V+IaK4+X97+Rcl0q/+KWe534+WoNLqxfxVBQgKI7Uk VdXvFuJTUb2M3/mol9qasNWEN4TWanenJVpHVF7DKUJz0EBXuNdXORARUhlWQsJpEwwW hwGC01WXvlDLCtWvMipJtfTdlNdWH7+2DVppj1yU2K1vAAQJF3XxDRCn5nKAboKVa0GZ YpwJ3xGX5dt/fCfgPkCog0SV2QykStVEhVx2N69V7qxwJmnnsZYQCiOnJbnMMP9Qg3Yl lj6nmKQFLuRgajfyAhsx5WwpJcnFGH8fPwqkQxNd0wHcTVHk6MDmgtxO7DUnqM8MKfN6 Zvhw== X-Gm-Message-State: AOJu0Yzji9XRxFDBqPRFg+JC7KKneSaB+bvBdY0+SoHkug5pRQH1zEjF OBOTu9TXUDdr/EOR1+uVEH1uowZ+k1ypHfos1e5/WjlyOGahO0XuJPnRDwqr4IZ7djZFShbqAnt nb6LCQA8edD3CntF3k4ul2EXzWXQ= X-Gm-Gg: ASbGncs1XqygJpsOgQLIZd0lAglUpjv1DgOozizNEcRdXjd3HwxN8X8Yl8i2eMjatbn 3lr+7i/EPKFUN9CHXKXrhatFtHpryTGauUo3/ X-Google-Smtp-Source: AGHT+IGr5IS31cdt1NGOg4F1KiB/13854ZxwevvDMHhvbp/YLP1d92T1rpW+gEUhDviUXVI5gmbmVTimSkg3ElEi9u8= X-Received: by 2002:a05:6871:6204:b0:2a0:d2d:e60a with SMTP id 586e51a60fabf-2a7fb5e0216mr25321556fac.43.1736018218187; Sat, 04 Jan 2025 11:16:58 -0800 (PST) MIME-Version: 1.0 References: <20250104214032.04eb6d25@sovereign> In-Reply-To: <20250104214032.04eb6d25@sovereign> From: Alan Beadle Date: Sat, 4 Jan 2025 14:16:47 -0500 Message-ID: Subject: Re: Multiprocess App Problems with tx_burst To: Dmitry Kozlyuk Cc: users@dpdk.org Content-Type: text/plain; charset="UTF-8" X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org > Thread-unsafe are queues, not calls to rte_eth_rx/tx_burst(). > You can call rte_eth_rx/tx_burst() concurrently without synchronization > if they operate on different queues. > Typically you assign each lcore to operate on one or more queues, > but no queue to be operated by multiple lcores. > Otherwise you need to synchronize access, which obviously hurts scaling. > Does this hold in your case? > Lcore is a thread to which DPDK can dispatch work. > By default it is pinned to one physical core unless --lcores is used. > What is lcore-to-CPU mapping in your case? > > What is the design of your app regarding processes, lcores, and queues? > That is: which process runs which lcores and which queues to the latter serve? > > P.S. Please don't top-post. Hi Dmitry, On one machine I have two processes and on the other there are three processes. For simplicity we can focus on the former but the only difference is the number of secondary processes. It is also worth noting that I am developing a shared library which uses DPDK internally, rather than the app code directly using DPDK. Therefore, all of my DPDK command line args for rte_eal_init() are actually kept in char arrays inside of the library code. I am setting up one tx queue and one rx queue via the primary process init code. The code here resembles the "basic forwarding" sample application (in the skeleton/ subdir). Please let me know whether it would be possible for each process to use entirely separate queues and still pass mbuf pointers around. Before I worry about scaling though, I want correctness first. My application is more likely to be CPU-bound than network-bound but for other reasons (this is a research project) I must use user-mode networking, which is why I am using DPDK. I will explain the processes and lcores below. First of all though, my application uses several types of packets. There are handshake packets (acknacks and the like) and data packets. The data packets are addressed to specific subsets of my secondary processes (for now just 1 or 2 secondary processes exist per machine, but support for even more is in principle part of my design). Sometimes the data should also be read by other peer processes on the same machine (including the daemon/primary process) so I chose to make the mbufs readable instead of allocating a duplicate local buffer. It is important that mbuf pointers from one process will work in the others. Otherwise all of my data would need to be duplicated into non-dpdk shared buffers too. The first process is the "daemon". This is the primary process. It uses DPDK through my shared library (which uses DPDK internally, as explained above). The daemon just polls the NIC and periodically cleans up my non-DPDK data structures in shared memory. The intent is to rely on the daemon to watch for packets during periods of low activity and avoid unnecessary CPU usage. When a packet arrives it can wake the correct secondary process by finding a futex in shared memory for that process. On both machines the daemon is mapped to core 2 with the parameter "-l 2". The second process is the "server". It uses separate app code from the daemon but calls into the same library. Like the daemon, it receives and parses packets. The server can originate new data packets, and can also reply to inbound data packets with more data packets to be sent back to processes on the other machine. It sleeps on a shared futex during periods of inactivity. If there were additional secondary processes (as is the case on the other machine) it could wake them when packets arrive for those other processes, again using futexes in shared memory. On both machines this second process is mapped to core 4 with the parameter "-l 4". The other machine has another secondary process (a third process) which is on core 6 with "-l 6". For the purposes of this discussion, it behaves similarly to the server process above (sends, receives, and sometime sleeps). Thank you, -Alan