From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 3BB0AA046B for ; Mon, 22 Jul 2019 18:17:52 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id B4DA91BEB8; Mon, 22 Jul 2019 18:17:51 +0200 (CEST) Received: from mail-pf1-f178.google.com (mail-pf1-f178.google.com [209.85.210.178]) by dpdk.org (Postfix) with ESMTP id D887B1BEB4 for ; Mon, 22 Jul 2019 18:17:49 +0200 (CEST) Received: by mail-pf1-f178.google.com with SMTP id g2so17614387pfq.0 for ; Mon, 22 Jul 2019 09:17:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Kr3P4pknQs7shajShYEV90xH8E2GZNKNg3ryNwat1rc=; b=vt9fEDHGrTCkaSe+qOlxzn9BMOzFICeervgc4F0ik6xWPOux3u5dnHDna8w2tM8peh DR/mVbhD+fuo9JDVIPQfaUO3AgKyMP88PWjc684+xvo2CEl+0iyqTtw/WYGwHidZiOnZ ddl8ZmZQZkItW6ulje50wQeNlwOJGpOTEq3Ddc7bEzYTdUo6mgCHxW1AgqD1Gt7ZjiMz ARo2ks/oNen23XilKTtL8fkiUxJOKzsQeXg+XGyWuc7Sj1wISs8LH28QYIr1xGJkMHcw Kba/PgZAepLoa+R4lMZKsMwdHTabtMKXIT3cTywOwVgS9sXnCA1CbKgoJjR2i+3+lbnQ rsEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Kr3P4pknQs7shajShYEV90xH8E2GZNKNg3ryNwat1rc=; b=LTvfvHiY7GW9mn7fFX5/LJsqMHC5WWnA9N6JnlJx7jKlAnCep91/mYKvUAaNnb1OVN J1kYpnolzDgRVIOFQ3zVwGldWoaG0F33NfaAIwBLJcwophAVzJzEH/CnYu6IxT1HVG57 nuycq2BM3YCFr6d6zScD7hJdrHSaisKk2JXI2KfzLF/ePUl8rd7ZGWgp2Yf0EcjP7dSJ CAtdxXwT0BiWpHMOl73Va1BGOn2ACgqPnR8CZz1t60pD7G3hvHMJZZ5SUE00CmGKN8uM 8c8w/VLBO+tFNKgY3fMO0s+908DfymoNphhjXhluYRQuwuTopRdzcTsOseUun7UXjiVR +orA== X-Gm-Message-State: APjAAAUEqf55jX0yVlWIyX/m9aw4MvwALoTry2yZ8g8wQmKICU1m3Tnf IoTc4vseUIRW4ayW2ERcjFc= X-Google-Smtp-Source: APXvYqw++uMSwLjW8uNrnlhh1fwJLDVDIpQpFnlsqxOzq9WpllT4Oix/EazTFFebaOlUPLFxFvKeqA== X-Received: by 2002:aa7:8007:: with SMTP id j7mr1022800pfi.154.1563812269022; Mon, 22 Jul 2019 09:17:49 -0700 (PDT) Received: from xps13 ([167.220.63.53]) by smtp.gmail.com with ESMTPSA id i198sm33939395pgd.44.2019.07.22.09.17.48 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 22 Jul 2019 09:17:48 -0700 (PDT) Date: Mon, 22 Jul 2019 09:17:46 -0700 From: Stephen Hemminger To: Anupama Laxmi Cc: users Message-ID: <20190722091746.2e0f5131@xps13> In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-users] dpdk packet loss,delay, retransmission X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Sender: "users" On Sun, 21 Jul 2019 15:11:34 +0530 Anupama Laxmi wrote: > I see delay in TCP packet transfer while transferring huge files using > SSH/SCP. The delay is not seen with file sizes up to 35 MB. Beyond 35 MB > there is delay due to TCP out of order /retransmission and packet loss. For > file size 762835531 earlier it took 8 seconds for scp transfer now it takes > ~~ 4 mins after DPDK upgrade. I suspected Small RX and TX queue size on > NICs which are connected to DPDK is causing packet drops. > > 1. Increased TX queue and RX queue size as follows. > RTE_TEST_RX_DESC_DEFAULT 128 RTE_TEST_TX_DESC_DEFAULT 512 > > to > > RTE_TEST_RX_DESC_DEFAULT 4096 RTE_TEST_TX_DESC_DEFAULT 4096 > > Saw little improvement. The time taken for scp reduced to 2mins and 30 > seconds for file of size 762835531. > > 1. Tried to rate limit the TX rate using: rte_eth_set_queue_rate_limit > API > > Not seeing much improvement. > > Please suggest ways to configure the mbuf,queue size to optimize the > performance and reduce the delay and drop in packet transfer. Welcome to the world of buffer bloat. The root cause is that large receive and transmit queues give worse performance in the real world. The optimum queue size is actually smaller than you think. The DPDK examples are all badly tuned for real life, they encourage bufferbloat. The values are to allow UDP packet benchmarks to be fast. What is your packet size and data rate? You should aim for < 1ms of buffering. DPDK could do better by implementing something like Byte Queue Limits in Linux and/or CoDel. But this requires a software queueing layer on top of the hardware.