From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ot1-f41.google.com (mail-ot1-f41.google.com [209.85.210.41]) by dpdk.org (Postfix) with ESMTP id 916CF1F28 for ; Sat, 17 Nov 2018 23:05:23 +0100 (CET) Received: by mail-ot1-f41.google.com with SMTP id u16so24239810otk.8 for ; Sat, 17 Nov 2018 14:05:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=Qbizwnjs0UWWsRPZJkNXikLX4+MM+hfND/3vEDMJqLY=; b=EyC3zOexLeodOlNioW+kZ4B7vVZWZvOwJX98BcBe5ZcUA8Cus/+LtsPVuj4C6Esw6/ P3fkfIC+TKdwXvAtt6dBhZ2XXgA26SIPbIWYxnYikY1KXUgggK/v6vM2IaW/JkwAHxgi gf8pZhsOELBW654irqnkhFEDgecN3kcIRs69C4XaU6N6jbuD+p6A0rdgDOv8MbS51Awn Nr6t+zSrCgSddDHMtnmQFyrLxMAiOGwniiyIywuPvI2EY99pUQRJuFjMQlfAow1Ufaox unpOh7M+m2XinFtqsUEabrz0d0hsMPPdHQaRjSBFLuWjftlylyye8fNxETNzBLkW4gDf +RaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Qbizwnjs0UWWsRPZJkNXikLX4+MM+hfND/3vEDMJqLY=; b=lu3jx1fR3mU7NNU+Owv4A89yAm3g06Of/QTqK3XVHw7YgCN2Th0JuazEg05Uh9Bdx6 RXnPfUsIb+6hkj1DPXHYVpsyKzkOPdBUeIiC0Hb/LKgt1KGIJqm5Q40sTnJQHfkxV5Qw ot0Fy28m0A3vMLCFADfGfp1VyjcMpP3lNbTDbQfJYOaWl5OE7TjHUVHHem7KFi8L02Lj uGJVURWCfOsmvhmW1xKpefRIuTlevnji7akjLcY8UjDuzbQis+IXlAnzmmdJU+RW7n9G BTG6Ehq7IQf9VvU0eMJ8hgapRcL+SMJOxKWeVdQL29LCrKBLcDJiqXE+SGOiyjDe1+0U H4Cw== X-Gm-Message-State: AGRZ1gL4zoosJiiOhKTS8ce9aZMDRlS2rux98LGLnJjAofaxwrPOFyzv cMDnKo5X5eQzaFvUuZkUXBM+7EDSDXV0QBDD5sc= X-Google-Smtp-Source: AJdET5eXbi+uceKxBm9KpiFlRX0qnC3Y2Z3P911vfUY6BGYAF/YqmaoL/8YET6eAJfLDg5bfFOXhWsFtDa+wgCI/OMw= X-Received: by 2002:a9d:5b51:: with SMTP id e17mr10330484otj.44.1542492322778; Sat, 17 Nov 2018 14:05:22 -0800 (PST) MIME-Version: 1.0 References: <71CBA720-633D-4CFE-805C-606DAAEDD356@intel.com> <3C60E59D-36AD-4382-8CC3-89D4EEB0140D@intel.com> <76959924-D9DB-4C58-BB05-E33107AD98AC@intel.com> In-Reply-To: From: Kyle Larose Date: Sat, 17 Nov 2018 17:05:11 -0500 Message-ID: To: thadodaharsh10@gmail.com Cc: keith.wiles@intel.com, users@dpdk.org Content-Type: text/plain; charset="UTF-8" Subject: Re: [dpdk-users] Query on handling packets X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 Nov 2018 22:05:24 -0000 On Sat, Nov 17, 2018 at 5:22 AM Harsh Patel wrote: > > Hello, > Thanks a lot for going through the code and providing us with so much > information. > We removed all the memcpy/malloc from the data path as you suggested and ... > After removing this, we are able to see a performance gain but not as good > as raw socket. > You're using an unordered_map to map your buffer pointers back to the mbufs. While it may not do a memcpy all the time, It will likely end up doing a malloc arbitrarily when you insert or remove entries from the map. If it needs to resize the table, it'll be even worse. You may want to consider using librte_hash: https://doc.dpdk.org/api/rte__hash_8h.html instead. Or, even better, see if you can design the system to avoid needing to do a lookup like this. Can you return a handle with the mbuf pointer and the data together? You're also using floating point math where it's unnecessary (the timing check). Just multiply the numerator by 1000000 prior to doing the division. I doubt you'll overflow a uint64_t with that. It's not as efficient as integer math, though I'm not sure offhand it'd cause a major perf problem. One final thing: using a raw socket, the kernel will take over transmitting and receiving to the NIC itself. that means it is free to use multiple CPUs for the rx and tx. I notice that you only have one rx/tx queue, meaning at most one CPU can send and receive packets. When running your performance test with the raw socket, you may want to see how busy the system is doing packet sends and receives. Is it using more than one CPU's worth of processing? Is it using less, but when combined with your main application's usage, the overall system is still using more than one?