From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id F3A56A00C3 for ; Wed, 23 Feb 2022 16:58:16 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 8BA0441226; Wed, 23 Feb 2022 16:58:16 +0100 (CET) Received: from mail-pj1-f50.google.com (mail-pj1-f50.google.com [209.85.216.50]) by mails.dpdk.org (Postfix) with ESMTP id 59AD5411B2 for ; Wed, 23 Feb 2022 16:58:15 +0100 (CET) Received: by mail-pj1-f50.google.com with SMTP id m13-20020a17090aab0d00b001bbe267d4d1so3953960pjq.0 for ; Wed, 23 Feb 2022 07:58:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20210112.gappssmtp.com; s=20210112; h=date:from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=yrFAw1pmE+8FvVDgBY/uvZuGj6re5CWMjy+VYLbalDU=; b=VGOntul1q3ECtfOfdra/gu+5d9rhtrSnazd2HDGWEpJpmzsnpJBUZUykPTHsDS4Mu7 oYD4K8Q4Sg8HMhsdfSJz5o57uqJ8MOv5L4KQTBe7BVStmcndtoPpQ/ZNSPT4qdsyGBq0 ptRI4SAFSveFpmAs5Z6DZJWrFYKKlOTx4a08Sx9pHKFPXoi2J+kOJAc57IPqdO1Y2cqP o0T7Lxs0i7GEs9WMpz+oV+hDOvjFpSxy8Y+yqXOaod/PHRDoZ4cSZPmEvgBrThYrh5hi 073fSfCTZHEvG9U25FvBeeI5hcGzQO3vnLX/k/w8M1VG2E/76SR62HlseX0F9jdIdUIm 2nrg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=yrFAw1pmE+8FvVDgBY/uvZuGj6re5CWMjy+VYLbalDU=; b=zYUVxqSht5Bh2Hn/uuRsxkiibTqoEoJhteQBxRyUzz1GNa9jjW12t/1eQ+9q8LKRPe QcWNX1+kg5GTMT193k3ami7HZKvQYiHNJfsJW2Y17JxtvliDVQKci7x3bb6DV5x8ZX8u YcWLSwueZwIMZxKauAFxOIq1KZGaNprG0sGNUPiGVkOM395gWWtEr5qw8OUKTL9aMj85 6Rf6yx1BbiieaUHDaYbN2oKmxu600zkedTY5XrdR6HtpmfzPdGKT3HUbc6A50kwioKtd 4mSvopeA2RvNbDiXMr5uviOhRsarkJi/bh91dBhHoWo4uZcDqL6S5WLrjAtppBNaN1ph 5Szw== X-Gm-Message-State: AOAM532BzBkUvG6gt36mc7MFap6mx3O2aAhpbua1Q3Nz3iOTiNcTDlu6 SZiSjXC+YKPB6Hgq2n32XdxiXfXPMPhTNG3V X-Google-Smtp-Source: ABdhPJzwTdfqe6ax5bLP9I/gCuMmEn+l29EDhPwRKZEzjZRxSe3EJSB7xP2gaXWJMXX0QqdvpHhLRg== X-Received: by 2002:a17:902:a9c3:b0:14d:c5cc:d26c with SMTP id b3-20020a170902a9c300b0014dc5ccd26cmr269700plr.18.1645631894216; Wed, 23 Feb 2022 07:58:14 -0800 (PST) Received: from hermes.local (204-195-112-199.wavecable.com. [204.195.112.199]) by smtp.gmail.com with ESMTPSA id s3-20020a17090a5d0300b001bc2b469051sm3302474pji.29.2022.02.23.07.58.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Feb 2022 07:58:13 -0800 (PST) Date: Wed, 23 Feb 2022 07:58:11 -0800 From: Stephen Hemminger To: "Ramin.Taraz@gd-ms.com" Cc: "users@dpdk.org" Subject: Re: [dpdk-users] Accessing packet data from different lcores Message-ID: <20220223075811.70e65eb6@hermes.local> In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org On Wed, 23 Feb 2022 15:15:43 +0000 "Ramin.Taraz@gd-ms.com" wrote: > Back in December of 2021, I posted a question about accessing the buff_addr pointer from different cores, which oddly, wasn't working properly. > > The original question is pasted below. > > I was specifically using the packet_ordering example, as listed below and running it with --disable-reorder option. > > The problem of buff_addr having the wrong value in different cores ended up being the result of a bug in packet_ordering example. > > The example allows the user to select whether they want to stamp packets with a seq # or not to compare the performance difference. > > In the case of not stamping with a seq #, the rx_thread doesn't disable stamping > > Rx_thread () > { > .... > /* mark sequence number */ > for (i = 0; i < nb_rx_pkts; ) > { > *rte_reorder_seqn(pkts[i++]) = seqn++; > } > ..... > } > > When disordering is disabled, rte_reorder_create isn't called and the variable rte_reorder_seqn_dynfield_offset stays at default value of -1. And eventually *rte_reorder_seqn(pkts[i++]) = seqn++; corrupts buff_addr. > > > Proper fix is not stamping the packet with seq # in dpdk-21.11/examples/packet-ordering/main.c:rx_thread if the flag --disable-ordering is set. > > > -------------------------- > > I have been playing with dpdk 21.11 for a week or two and run into something that has me scratching my head a bit. > > I'm looking at packet_reorder example. I'm running this sample with 3 cores: 1 RX, 1 Worker, and 1 TX. > > dpdk-packet_ordering -l 0-3 -- -p 3 --disable-reorder > > In this example: > RX thread reads the packets from receive queue and puts them in a rx_to_workers ring Worker thread reads from the rx_to_workers ring, changes the port number and queues to workers_to_tx ring Tx thread reads from workers_to_tx ring and calls rte_eth_tx_buffer > > > What I like to do is access the packet content in the worker thread and print out a few bytes from it. > > What I'm finding is that mbuf_addr value, for the same mbuf, read from RX thread is different than if read in the Worker or TX thread. > > For example, when printing out the address of mbuf, and mbuf_addr values, in the three threads, I get: > > rx_thread > mbuf = 100e30000 > mbuf_addr = 100e30080 > > worker_thread > mbuf = 100e30000 > mbuf_addr = 100000000 > > tx_thread > mbuf = 100e30000 > mbuf_addr = 100000000 > > > So although mbuf is the same, the mbuf_addr is different. The packet content is obviously (?) different if read in RX, vs if read in Worker or TX thread. > > Is this what is supposed to happen? > > Basically: how do I get access to the actual ethernet packet, for reading or modifying, on different lcores; in this case the worker thread. If this field is going to be referenced by other cores it needs to be done inside lock or use atomic builtin primitives.