From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 85D1DA0C41 for ; Fri, 16 Apr 2021 08:00:18 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 56632141AFA; Fri, 16 Apr 2021 08:00:18 +0200 (CEST) Received: from mail-wm1-f51.google.com (mail-wm1-f51.google.com [209.85.128.51]) by mails.dpdk.org (Postfix) with ESMTP id 6965C40140 for ; Fri, 16 Apr 2021 08:00:17 +0200 (CEST) Received: by mail-wm1-f51.google.com with SMTP id k4-20020a7bc4040000b02901331d89fb83so1449595wmi.5 for ; Thu, 15 Apr 2021 23:00:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=uTlRXiFVS4fCZu0zWaYj0LV1Al6i+c0FMtditP4wgvU=; b=e5CvJsVBNH3Lcx5MHuWiGYwums2Baj6wFi0JnqFSaTOUeI1DlCayeG6nURurQPFmp9 0JkkAoewCcSc3pEBfA+khLBirgYO+DhmiUBG8LT9yS/dGjxgJdSnFp5I/39EuvyBUOYg hda/XMIMVrnh/vyjBa+CNg0/l8vrqvLA4fGjmdvdiLFzvIGdF9MoILKLLnYnpFNJs1tA BMDlhr6x6QDq9FW1ZgrfUR1+azj8Fp3/o81JDs/l6tgImHZPPzNRVSvV65Z640fxg8qW lP0Wmmwht7zi77VMinm3W1AtBzJq4fBCp9VjUZv4DO2ku5m0QE4EtAzhfeiFrPTHIbhO CnPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=uTlRXiFVS4fCZu0zWaYj0LV1Al6i+c0FMtditP4wgvU=; b=gFjV9olDEayEbs20GP07ywEqDmbrYAS3Q/1ZME+Lb192RORPxj9T0qtMspdOGVnO8D KmZ784aP3GtCJP98HgHGbguRFOTadnYJf7HYcA6csP4fxOyfNJtNEtk7qci/+TVQ8wZN AXAO9nZmmWOrZQiuVGOtTRgXKqYHuCdaj84otYqCZuaVwNTdJb4BhXbdAwDvwawsEhJq 5anluqAVPcg5IQGHpWziaPBs63Qit1Wx7ITl+Y9UUzJCvwTWbMaG6K2BQT7M+hRrlp5T 85VW4LtNbKKnS7jpfW2ibl214dsNieP9L2eF12azbk4bKKfyIwvcPqxrBVzZxyRhd+k2 x1Cg== X-Gm-Message-State: AOAM5304u6tL7z/3hEPaSLonQiB+m3j7DCYWQKbFBr4L5Xfq4m0F9LT5 OVRzgzIiQnPqTHYglnz6Awf90PQBX32hMAlDiKQ= X-Google-Smtp-Source: ABdhPJyVS0v3y5UsCRQxqb6QUxNArToafGNB1HHTaePwLpDfwIP4F7rAtOy4OmvS76rz3pADlYv2QdL0yn3swfGNbeI= X-Received: by 2002:a1c:80cd:: with SMTP id b196mr6478335wmd.30.1618552817189; Thu, 15 Apr 2021 23:00:17 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Pavel Vazharov Date: Fri, 16 Apr 2021 09:00:02 +0300 Message-ID: To: Hao Chen Cc: "users@dpdk.org" Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 Subject: Re: [dpdk-users] What is TCP read performance by using DPDK? X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Sender: "users" Hi Hao, The current design of the application very roughly is following: 1. There is one main thread which pumps out the packets from the NIC queues using rte_eth_rx_burst(), as you said. In the future we may need several main threads to be able to scale the application. Each one of them will work on separate groups of RX queues. The main thread distributes the received packets to N other threads using single producer single consumer rings provided by DPDK (rte_ring). 2. Each one of these N other threads runs a separate F-stack version. As I said we use a networking stack per thread and share nothing design. Let's call them worker threads for now. 3. Each worker thread has its own spsc_ring for the incoming packets and uses a separate NIC queue to send the outgoing packets using rte_eth_tx_burst. The main loop of such worker thread looks roughly in the following way (pseudo code): while (not stopped) { if (X_milliseconds_have_passed) call_fstack_tick_functionality(); send_queued_fstack_packets(); // using rte_eth_tx_burst dequeue_incoming_packets_from_spsc_ring(); enqueue_the_incoming_packets_to_the_fstack(); if (Y_milliseconds_have_passed) process_fstack_socket_events_using_epoll(); } You may not the following things from the above code. - The packets are sent (rte_eth_tx_burst) in the same thread where the socket events are processed. The outgoing packets are also sent if we queue enough of them while processing socket write events but this will complicate the explanation here. - The timer events and the socket events are not processed on each iteration of the loop. These milliseconds come from a config file and are measured using rte_rdtsc. - The loop is very similar to the one present in the F-stack itself - https://github.com/F-Stack/f-stack/blob/dev/lib/ff_dpdk_if.c#L1817. It's just that in our case this loop is decoupled from the F-stack because we removed the DPDK from the F-stack in order to use the latter as a separate library and use a separate networking stack per thread. 4. The number of worker threads is configurable via the application config file and the application sets up the NIC with the same number of RX/TX queues as the number of worker threads. This way the main thread pumps out packets from N RX queues and each worker thread enqueues packets to each own TX queue i.e. there is no sharing. So the application may run with single RX/TX queue and then it'll have one main thread and one worker thread. Or may run with 10 RX/TX queues and then it'll have 1 main thread and 10 worker threads. It depends on the traffic amount that we expect to handle, the NIC capabilities, etc. Regards, Pavel.