From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-lf1-f54.google.com (mail-lf1-f54.google.com [209.85.167.54]) by dpdk.org (Postfix) with ESMTP id B6FBB4C92 for ; Tue, 11 Sep 2018 10:15:40 +0200 (CEST) Received: by mail-lf1-f54.google.com with SMTP id c29-v6so19677621lfj.1 for ; Tue, 11 Sep 2018 01:15:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=FLSeSLuS44N5EpDzyOFagB+4C1uTmX73Pft2+BXRsiU=; b=cddC649bz9mTtbTToa/S9SugYEi1byI2fb2G9mvYegOVy4y37qmMuwrNCy6F1RSV9w 7qdAyWgA/Ge4MwAX8B8eDbUQnxnolfLDwLeycTVl73wRFYyXszCDEweLQ+6tXk4lWMwr qDwZJz6jr4N055jopDvzgMLpq2Wkp8k6LtAj/Dhm1sNMxcgy6ysuKyFi9vB2dP3RYoJk 8/VyR+NuXBfy5MlfJu5iUm4EiB3hAAgEXYH+sTw5KA8v2dPORNn88fPseemAoPjG5hGR 1n3mzI0+mEvhMAv6WM7IjuHCcYazb52dRKImAGQH2dTwUUUJoQTOYbW5nwgFIjHIaPcY bO+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=FLSeSLuS44N5EpDzyOFagB+4C1uTmX73Pft2+BXRsiU=; b=GsU8lTTLxRKOoBa+2GArpYldRR5MHgsR5GneeW237HLnQEf7Gzm07GEqs2h4YhvR0j 0jCmfIwQ/mdLxqnHeI68A741tix6tQGcFnnDOPGDRwhBuGdQEUezf94KXv8fcIeTZnZR Dc0fh8FOIbHzrseSDyzSi0HFWmVZ6XR67D1d53ddtMzGL6c4MMSQp8pdCNK6oCucIHVs /nxzDld6y7W9Sx9Ho+r/lV3yDz4+Vw9U+lxCy2p5NcYRLjBU8eFW1H99R2xiN1WNlLCL r0yC9AD68z046CO2ZT1XcG+reotUirPK5G810Fy7NXs8YqVTSUgGJfV/tff5x49kp35B 5nNw== X-Gm-Message-State: APzg51Cos/NO5is/rZPD4u1i4JH9cqiPkmrhwdifoWUDAtVZ6wM5Z+Eo ZTrS+L6i5noAmk/l2rk897iFnc2V6m1vidmRdLz4MvbC X-Google-Smtp-Source: ANB0VdY74HmrMZlNrgEuZJBoSgRfFuCGgf33hXVfIjeswtJWJ5gzVOubOTAyuEkrubdRtMoUdCPaoG7ZE+d7aIpb6oc= X-Received: by 2002:a19:7ce:: with SMTP id 197-v6mr7811096lfh.139.1536653739950; Tue, 11 Sep 2018 01:15:39 -0700 (PDT) MIME-Version: 1.0 From: Arvind Narayanan Date: Tue, 11 Sep 2018 03:15:27 -0500 Message-ID: To: users@dpdk.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: [dpdk-users] How to use software prefetching for custom structures to increase throughput on the fast path X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Sep 2018 08:15:40 -0000 Hi, I am trying to write a DPDK application and finding it difficult to achieve line rate on a 10G NIC. I feel this has something to do with CPU caches and related optimizations, and would be grateful if someone can point me to the right direction. I wrap every rte_mbuf into my own structure say, my_packet. Here is my_packet's structure declaration: ``` struct my_packet { struct rte_mbuf * m; uint16_t tag1; uint16_t tag2; } ``` During initialization, I reserve a mempool of type struct my_packet with 8192 elements. Whenever I form my_packet, I get them in bursts, similarly for freeing I put them back into pool as bursts. So there is a loop in the datapath which touches each of these my_packet's tag to make a decision. ``` for (i = 0; i < pkt_count; i++) { if (rte_hash_lookup_data(rx_table, &(my_packet[i]->tag1), (void **)&val[i]) < 0) { } } ``` Based on my tests, &(my_packet->tag1) is the cause for not letting me achieve line rate in the fast path. I say this because if I hardcode the tag1's value, I am able to achieve line rate. As a workaround, I tried to use rte_prefetch0() and rte_prefetch_non_temporal() to prefetch 2 to 8 my_packet(s) from my_packet[] array, but nothing seems to boost the throughput. I tried to play with the flags in rte_mempool_create() function call: -- MEMPOOL_F_NO_SPREAD gives me 8.4GB throughput out of 10G -- MEMPOOL_F_NO_CACHE_ALIGN initially gives ~9.4G but then gradually settles to ~8.5GB after 20 or 30 seconds. -- NO FLAG gives 7.7G I am running DPDK 18.05 on Ubuntu 16.04.3 LTS. Any help or pointers are highly appreciated. Thanks, Arvind