From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qt0-f173.google.com (mail-qt0-f173.google.com [209.85.216.173]) by dpdk.org (Postfix) with ESMTP id 40BEC377E for ; Tue, 18 Jul 2017 12:07:33 +0200 (CEST) Received: by mail-qt0-f173.google.com with SMTP id n42so11765364qtn.0 for ; Tue, 18 Jul 2017 03:07:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=aAPsNU9Qz9CoIdOPdKiqInLYiPHQCDyIzfYMllDqxXs=; b=lSOVKgag+6EeDvEncOSbnVwuyRUuIWVqTgdQ5jZFEGPgiNEfb/L+Z1rk265D171Hgi ZQNuSsQWSsRW/jU0EeEuRrL2iP0ZKIorXZgHqkoAEkCJ1SE/PLkt7t2PxrPb30LEKdgN 3JBvHKRQIT/WcA5ed4qDoUgOPdA1Ay3kgyejlWaWHZ3JF5dbM8cpT0ymuE0mNXs40+ST T/uGFMXX11jk6qXApE+QRGgkg78ac57xgr8vcagGe6llrddOMpKFt/S1k3ypWGf8VoLy YFQmhuE3LB167go22XrBxgoNklRPeg4MtB0HWDRwY+jT8Sn237QY8dtW342PgEiKtnHO bgVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=aAPsNU9Qz9CoIdOPdKiqInLYiPHQCDyIzfYMllDqxXs=; b=m5W9Z30SUL3ZV2Qoh1acE+UiyBtU6liY3liw1Eaqqdn1dinUxiSRmdWxD2PHPgAS2n bbr45s7Pr1v/cvny09XS2LxhgjKzVmWazH5Xc3s8ZMDnfVA7qIZYZ0aH+8AeBaESrrVc VwVRrpw2elE7n95uU0/JT1V3/pphoWtv5gxleHZk4nKFQut4owxBwECexjb4ULQUlUb8 iQySagBFOR2LSqpg+Jz7D+NgdaJi0WtARQNda8b9yWW6li1Kknl/gyp5yFtC8DZ4S7BX g976P4MAxXmdtL/F+dmXUJ9ok4BHcogbQkLnYzijzs9kJGW96zfDIV/ONvHNE2vRtHJY PwYw== X-Gm-Message-State: AIVw111rbMxccu8hJVrxNoOSJEKfuLVwE2T7u/a4g7bjhBrSN48/BCek F7Htc0UXtiRUA/wpl2ZOz9pQImPQNA== X-Received: by 10.237.54.73 with SMTP id e67mr890982qtb.230.1500372453300; Tue, 18 Jul 2017 03:07:33 -0700 (PDT) MIME-Version: 1.0 Received: by 10.12.149.235 with HTTP; Tue, 18 Jul 2017 03:07:32 -0700 (PDT) In-Reply-To: References: From: Shyam Shrivastav Date: Tue, 18 Jul 2017 15:37:32 +0530 Message-ID: To: Harold Demure Cc: Pavel Shirshov , users@dpdk.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-users] Strange packet loss with multi-frame payloads X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Jul 2017 10:07:34 -0000 Hi Harold I meant optimal performance w.r.t packets per second. If there is no loss without app fragmentation at target pps with say 8 RX queues, and same results in missing packets with app fragmentation then the issue might me somewhere else. What is RSS configuration, you should not take transport headers into account ETH_RSS_IPV4 is safe otherwise different app fragments of same packet can go to different RX queues. On Tue, Jul 18, 2017 at 3:06 PM, Harold Demure wrote: > Hello Shyam, > Thank you for your suggestion. I will try what you say. However, this > problem arises only with specific workloads. For example, if the clients > only send requests of 1 frame, everything runs smoothly even with 16 active > queues. My problem arises only with bigger payloads and multiple queues. > Shouldn't this suggest that the problem is not "simply" that my NIC drops > packets with > X active queues? > > Regards, > Harold > > 2017-07-18 7:50 GMT+02:00 Shyam Shrivastav : > >> As I understand the problem disappears with 1 RX queue on server. You can >> reduce number of queues on server from 8 and arrive at an optimal value >> without packet loss. >> For intel 82599 NIC packet loss is experienced with more than 4 RX >> queues, this was reported in dpdk dev or user mailing list, read in >> archives sometime back while looking for similar information with 82599. >> >> On Tue, Jul 18, 2017 at 4:54 AM, Harold Demure > > wrote: >> >>> Hello again, >>> I tried to convert my statically defined buffers into buffers allocated >>> through rte_malloc (as discussed in the previous email, see quoted text). >>> Unfortunately, the problem is still there :( >>> Regards, >>> Harold >>> >>> >>> >>> > >>> > 2. How do you know you have the packet loss? >>> > >>> > >>> > *I know it because some fragmented packets never get reassembled >>> fully. If >>> > I print the packets seen by the server I see something like "PCKT_ID >>> 10 >>> > FRAG 250, PCKT_ID 10 FRAG 252". And FRAG 251 is never printed.* >>> > >>> > *Actually, something strange that happens sometimes is that a core >>> > receives fragments of two packets and, say, receives frag 1 of >>> packet X, >>> > frag 2 of packet Y, frag 3 of packet X, frag 4 of packet Y.* >>> > *Or that, after "losing" a fragment for packet X, I only see printed >>> > fragments with EVEN frag_id for that packet X. At least for a while.* >>> > >>> > *This led me also to consider a bug in my implementation (I don't >>> > experience this problem if I run with a SINGLE client thread). However, >>> > with smaller payloads, even fragmented, everything runs smoothly.* >>> > *If you have any suggestions for tests to run to spot a possible bug >>> in my >>> > implementation, It'd be more than welcome!* >>> > >>> > *MORE ON THIS: the buffers in which I store the packets taken from RX >>> are >>> > statically defined arrays, like struct rte_mbuf* temp_mbuf[SIZE]. >>> SIZE >>> > can be pretty high (say, 10K entries), and there are 3 of those arrays >>> per >>> > core. Can it be that, somehow, they mess up the memory layout (e.g., >>> they >>> > intersect)?* >>> > >>> >> >> >