From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 20BDBA00C4; Mon, 25 Jul 2022 11:21:51 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id C0DC040684; Mon, 25 Jul 2022 11:21:50 +0200 (CEST) Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by mails.dpdk.org (Postfix) with ESMTP id 7A7E34067C for ; Mon, 25 Jul 2022 11:21:48 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1658740908; x=1690276908; h=message-id:date:subject:to:references:from:in-reply-to: content-transfer-encoding:mime-version; bh=kn5cEY7HQj9Bm2+djb10s8M6UcLI0Ee+ywa3xB9i5Ok=; b=EjyluVlHglxzWBcLjau3WH+fUchHtPVU+lW1YM7ttPJ9Wuz9BPwmd3Ee Wys6oi+wOGfk1VlYjze/3iILXR/SrIeht2QyXzkWBi8FFkdqN+QCQ7KjM P58EMYCxjGMlwqC2ItEQeeIqLpPvb4VPONHacH0N5ZfFwUDhv7OHd+JD5 eZB2NHHRTy+Biwmx2P9jei+FOaizWRBUvhDcoYxfU1Q1ERc1/PPLDDMR9 Gu7WS06ePgr85NcTeDl0Gyi2yzNzq5vGEJHlNXsoubwtkUuU6g9zDPyCT /FVBUqNhxg1C4A7xFdA/YbV20chs3pWqJHgM4nFZizGjfUJIB23YS/vz/ w==; X-IronPort-AV: E=McAfee;i="6400,9594,10418"; a="267426890" X-IronPort-AV: E=Sophos;i="5.93,192,1654585200"; d="scan'208";a="267426890" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2022 02:21:47 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.93,192,1654585200"; d="scan'208";a="574978072" Received: from orsmsx602.amr.corp.intel.com ([10.22.229.15]) by orsmga006.jf.intel.com with ESMTP; 25 Jul 2022 02:21:46 -0700 Received: from orsmsx610.amr.corp.intel.com (10.22.229.23) by ORSMSX602.amr.corp.intel.com (10.22.229.15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.28; Mon, 25 Jul 2022 02:21:46 -0700 Received: from orsedg603.ED.cps.intel.com (10.7.248.4) by orsmsx610.amr.corp.intel.com (10.22.229.23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.28 via Frontend Transport; Mon, 25 Jul 2022 02:21:46 -0700 Received: from NAM02-BN1-obe.outbound.protection.outlook.com (104.47.51.45) by edgegateway.intel.com (134.134.137.100) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2308.27; Mon, 25 Jul 2022 02:21:45 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ShVi9udh8yGg9lqLaXzrrP/fEacgbNrQfJ1JaEZbuDpo7sC0C/wgDJyBz6kCR3LltrAugmiXL7pKrK6plDXGqNCdCdmEWy2OOZ2a6DVYxcLaLZQ+yxuzqV6QmePMSA8HGXE0/KRdwHgM3Q9KK1E1lc99II9DF7FAtbwgYr2RBjR8q2S7beMJUaMhPFzHmTI5anF/0rbAU9t0oQio35F8Udl2grZL2MjryQUOC4lDLYR1ZMDs8wLKg6+ePwyIKJfEf9bwAQRpXt+3MbFKOf68+wTENUVM/z5DYrNwNWW6Q6/d3e/WzPbWyi1DmUV8I999oZPHgTnT4uxdolOZTn0Z/w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=4e+gF9uodNJUoOIyDU9VYEFftqxFFUY5qmkf5ZgB7OI=; b=kcxY0/OSHHpEvdwiYdbO6lR/7MAjekLachEP986/neKumzWrXJNcbHIbyIhhwdKYLgqdKA077RvswtW/gAAHdbRWdgbsZVs5KydcyWUCDAlIh+w0lZIxLArBIVuLNoEwnyXDvvLnbHbn3xhSdJ4W6zjx2XL7i0FNE5zO6fIdDDpS/EV5reBebhT+DydbE7iIY+9t7ihtTl2MfwTKwVEFgA9dFz6uXu8M62vpafjZibCHMcfFQs1lLe3nJ+NZIYWBJ88hMwd1lT/AEm1Qav9yNGZmAyWD9HhLWxo8EWHc79pirWnMetfgZjcPgTKrNZc8esAmOCtEP+sBZusd7MRrIA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; Received: from BN6PR11MB1251.namprd11.prod.outlook.com (2603:10b6:404:48::10) by BL0PR11MB3441.namprd11.prod.outlook.com (2603:10b6:208:6a::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5458.20; Mon, 25 Jul 2022 09:21:44 +0000 Received: from BN6PR11MB1251.namprd11.prod.outlook.com ([fe80::128:8fb:9d0:1a9f]) by BN6PR11MB1251.namprd11.prod.outlook.com ([fe80::128:8fb:9d0:1a9f%8]) with mapi id 15.20.5458.024; Mon, 25 Jul 2022 09:21:43 +0000 Message-ID: Date: Mon, 25 Jul 2022 10:21:39 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0 Thunderbird/91.11.0 Subject: Re: DPDK 19.11.3 with multi processes and external physical memory: unable to receive traffic in the secondary processes Content-Language: en-US To: Asaf Sinai , "dev@dpdk.org" References: <9350acbd-06b3-8981-1c2a-7a2b1a5daea9@intel.com> From: "Burakov, Anatoly" In-Reply-To: Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 8bit X-ClientProxiedBy: LO4P265CA0183.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:311::6) To BN6PR11MB1251.namprd11.prod.outlook.com (2603:10b6:404:48::10) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 6eca92f2-e71d-489c-002d-08da6e1f16e7 X-MS-TrafficTypeDiagnostic: BL0PR11MB3441:EE_ X-LD-Processed: 46c98d88-e344-4ed4-8496-4ed7712e255d,ExtAddr X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: vqiCohGacxbbGtpquzNtqVB9OwRepD7h2v5xHP4wLxuOVQ3pqX5ONaCkwJxc5OttFy5od5pXs3K01gIk6pgsKEGm6y1rsn4IfOfkpffkqNMdq1OiqWvgoLN/OS9DWoOkA8An4/pbV6t5wwQ6Er6To3eA3Oi7VSwkd+FH0eH8wCt3aYoJsgPejKA2W07QzhY74XfUy7hsZXMrMgN7EJvgJBatB2cNO3N1A8gtJ2oHP0N+PjAJivqriTnapzjndgbhaUBMg5M+BQp0biUhdtURujRUXMoY2dP1YQ+NGUeZDVvWX58wf/IJ30hyrNAjkbBmxpJs1zlOhi48E0CQYFPpRvFHakHEfVKUQMLITBxNvErLMBqEHR85jGcMGoFsjhvCZD+Wr1s9g+vgHvHsyCXo2KXj22oYaxcEaL05Mvis4bqZlR2y7f5jtj/eROWh+4v42FOOxRIqeD+ddArpxOvmV13/VNVQkvF+n57VebD3synnr9QG9kMYAsjwAivyLFNy4N/d8EeGSTSZgK7q+3IHnTj2c5ALKDR7HeEj+whTEvuS3AYaLHNb5K6dWSd5HmlNQw6jh5Lnqpu2RI4xkeXEf6JZusGKSK0PTdHLoaxU0Hw18NAk0/+q6+WmlgP5twD4R3zbNNwmdJRH/cZ07rEfaTpurvkDGbjLf9dD6wU2AOBRqpz8+pMU76UoLUQaGDNjRMknbDnkCdQSlFwUoZKrL1Nhsj18VCc+4/XbHkFsIlnPVxufwEy0GvC4BOrSsNeXfcptN7SdLhW3p6JrFk981SrsqWzFTjLlNv9mOHT7YkZVlS/RG77yN59kvcDB274P X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BN6PR11MB1251.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230016)(376002)(136003)(346002)(366004)(396003)(39860400002)(66556008)(66476007)(66946007)(8676002)(110136005)(6486002)(82960400001)(478600001)(38100700002)(6512007)(316002)(26005)(53546011)(31696002)(86362001)(41300700001)(2906002)(6666004)(6506007)(83380400001)(8936002)(36756003)(186003)(31686004)(2616005)(5660300002)(43740500002)(45980500001); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?MUp3ZTRqVzh0MmRmK0dsN3V6S29ES3RaSm9pbG1zbVdKMnRCY1kyZVcwaUFC?= =?utf-8?B?ak8xTms4dThTWUV5YWlBSmFMcFdjeFN5RS93enA5VXVyblU0VHhMbjU3QWlU?= =?utf-8?B?SHpVUmhsVDNsQW5mUlk4YW5NMVZSSUgxV25ZalhOTXhiMUIxL0lURFlOWmhN?= =?utf-8?B?VHdCWUoydlVnUUlsaTNRY3NGZ2JablNxY0Z3Vjd5bTJQOFdZRDZpZnN0R1I1?= =?utf-8?B?N09iWlZ1SzA3bWNSS3hZQTJhcGlDeUZ5SFV4YkkzMkppSURmNFozb3Z4eExp?= =?utf-8?B?Y2w5MmdRMjVTSmVwVGhKYVRybTNwWmlodmZZSGpMSTJBN3hqdElZbTBmcFND?= =?utf-8?B?S3pBVUJ4UHNmbVVySVV3WEpiWVYzTEU3eUh2ZUx3Nk1JWVlTaTgyWHRseUpI?= =?utf-8?B?VE5vaDcwS2RkZXpsOS8vNHlUbW51RFl2L3dRTkdabU9qMUJmTWpNZmNmSVk4?= =?utf-8?B?VTVNMHVaZ052TjZPVm1lUHE1QWthcXpueUt6UTduUlpLSWpRSHNnZVlmaWEr?= =?utf-8?B?SFlVamRGR1F5RlpVWFh1U056R1daZWlnOThpOEVuTXVGT09ZQ1JZVy9DdXJQ?= =?utf-8?B?ZlpndDVlYlVUVTBUK2kzaGFLbm4ybUxlMHZQTCtCUDJDTFhSMXV4NGZSbEhp?= =?utf-8?B?MmdYalNLYnpmZS9INm5vM1ZSTldwMGV0ejhOeENjeitlaWl0NDY4dVZ3S0NB?= =?utf-8?B?Y2xkWkt5YWtPRmE5aFc3NlR5UTUvNFRMT2hDREhXdEtnU2ZtdGtxek9sNEEv?= =?utf-8?B?ZTkraFk3T2NHOFQrcHQ5TkJ5dm5Fa3lTc3dQbUZqM3ZLQk5Hbkhwb3FLNHJo?= =?utf-8?B?VkNjMlpkdlNPR3ZDenIrSE43L3hONnZnLzJsdkNmTVRqNFA0TmRhNkNsZnJx?= =?utf-8?B?bk5oZzNzRlJaV0hXa3pLMzlaSGsrSVBwOXlEU1pEazg5NWhkbUJiUlhLSW5V?= =?utf-8?B?MzFDbDVDdFZQaGdwU0EvVVFmeGJLeXNXU2xxODY4Wm5obnFNbzYxVDh4bnR4?= =?utf-8?B?bis5c2NLR3AydGhBa3F2bTl0T2E5dXRaTjlnYStNdXJQMUdlUjc1M21pSmdC?= =?utf-8?B?ZVJYbUR1ckMvdnk4V1RZUnVJc20wTEJjc1BMR2lWMUdzQWt0a3FBTkE5Q2tM?= =?utf-8?B?RFJvbWVHd2FFNmdsZ2dUU1RkSGh0c1VxNjFZb1NEdkoyaFcvNWptb0ZaZEZ0?= =?utf-8?B?a3dvVWlLNURMN09zREpQSWpZNkZHUEwyY2dINEwwZFJyNk53U0NLd25KZHBy?= =?utf-8?B?MjFud0RRcjBQTFdMSGFWeitoR3pPZHpNc3FDMlRESDBBMVdzbGw3NlFLSjNR?= =?utf-8?B?cFk4VXc3cEd3MFNXVURiNmR6M1lsTXlwRmtISUkrT2JaVHdvRG9ndmZoT3Nk?= =?utf-8?B?RFdSR3FRT2pXcXN3ZnhUMHFGdWhYcUMvMDZTMmQ1ZUF4UUFoU1F2TFd5VHVm?= =?utf-8?B?Q2RTalI4ajU0bUYyc21qRk1QQUFVZzU1K05qUFlrazZWbzBXZWNEMmxvRUlQ?= =?utf-8?B?TUFDWitTQUpNbGRRdjhtdHlPNFZoWnRxdk55TFZVTy9vWnhhQkhkRG5YVnV6?= =?utf-8?B?M0NUSnRpNHJkRjVjLytxVENQSld1VEZ2aTZxOWJDdHJ6bzJSUmdtS2Z1T2V2?= =?utf-8?B?S0hGZU5WZ0hyV2FocURDTU51NUs0emNjeGhtM051VHlYcG11b2dUM3REUmFE?= =?utf-8?B?Q1FNRTlLa0IrN2FaamZSci9IU3o4UndaOU5hV2U2cjg4UXB3U1VxM29HYThv?= =?utf-8?B?aTRxK0YweWpuNE1pMEhlelZvRmlUbDVST3IrRG8rTml0U1htZkRReklNMWRP?= =?utf-8?B?Y1dDZkRnd0FmMFRMSTVENWZ1S3owb2xCcU5iay9LR2lrSnFPT0VWbFNEa3RS?= =?utf-8?B?bnEveStuM3JBb3VQYjZpRkdkSnlLMFNsaEhUZkwxQThUQ1hMenZpSzB1N3pQ?= =?utf-8?B?NFAyTWFGNEpUN1daSFJqMncxdFZoSEdHckFKc1hqM2dqM2czM2FzekRQdXhY?= =?utf-8?B?NkVQZnlZcnVhbHNHd28rMzFUbGpET2dmNlcrN3p2RWFOdThmR2RZVi9QSGNj?= =?utf-8?B?UUppYzVqR2E4ZTd4UmNZdkh2Z1RvaGwyVUMydWs5SG9lNEZreDFuWG8wNDAz?= =?utf-8?B?bExoV0tzY2JUMENzeVpTVzR2d0lXbVJiV0cwVUdaOWJlSC9WU3RQNk5FajY1?= =?utf-8?B?SVE9PQ==?= X-MS-Exchange-CrossTenant-Network-Message-Id: 6eca92f2-e71d-489c-002d-08da6e1f16e7 X-MS-Exchange-CrossTenant-AuthSource: BN6PR11MB1251.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Jul 2022 09:21:43.6281 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 2jZjt0nSNIAkhjQ0eD51rJQzlKMJXGfXcj2doNtyZK2YOmmSCQUY8n8HKXDI6BOoXdY08uMH24lnv5/FjSpPzjJB7Pere3WMvwJVmrg9O2Y= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR11MB3441 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On 18-Jul-22 12:58 PM, Asaf Sinai wrote: > Hi Anatoly, > > DPDK runs as root, and secondary processes have all the info. > > The problem was as follows: > > The external memory regions are not managed by the Linux OS (by using > "memmap=x" in 'grub.conf'). Therefore, the kernel cannot supply their > physical addresses. > > So, we added these addresses in code, and now it works fine! > > Thanks for your help! Happy to hear that! > > We have several additional questions: > > *_1. Usage of huge pages in "rte_eal_init":_* > > We see that "rte_eal_init" *_requires_* allocating huge pages for > configuring the drivers. > > It seems impossible to use the external memory, as > "rte_malloc_heap_memory_add" API cannot be used yet. Yes, there is currently no way to initialize anything using the external memory. This is because at the time of initialization, external memory is not yet discovered and therefore cannot be acted upon. It /could/ be possible to implement this using an EAL plugin, but i have not looked into it and know very little about EAL plugin infrastructure, so I cannot offer suggestions off the top of my head. > > So, we tried to use regular pages instead of huge pages, using the > option of "--no-huge". > > It failed with the following printouts: > > /.../ > > /EAL: Multi-process socket /var/run/dpdk/rte/mp_socket/ > > */EAL: FATAL: Cannot use IOVA as 'PA' since physical addresses are not > available/* > > */EAL: Cannot use IOVA as 'PA' since physical addresses are not available/* This happens because no-huge will not attempt to find physical addresses of the memory backing the allocated segments. > > /.../ > > 1a. Is there a way to use the external memory for “rte_eal_init”? There is currently no way to do that, no. > > 1b. Why using regular pages, causes DPDK to complain that “*/physical > addresses are not available/*”? We do not populate physical addresses in case of nohuge, as per eal_legacy_hugepage_init(). Technically it should be possible to do so using calls into rte_eal_virt2phys(), we just don't. I believe the rationale is that 1) we have no control over that memory and kernel might change its PA's at any time, 2) the init would take a long time because there's quite a few pages even in small nohuge segments (and we'd need to query pagemap for every single one of them), and 3) nohuge is really meant to be a debug option and is not intended for production use, so this path is not heavily tested by intent. > > 1c. Why is “—no-huge” option defined as one of "EAL options for DEBUG > use only" (in “eal_common_usage” routine)? That's kind of why it was created: to test DPDK without hugepages. The intended use case for DPDK is to be run using hugepages. > > *_2. Explanation for some details in "create_extmem" routine:_* > > 2a. What is the purpose of calling "mlock" before populating IOVA addresses? > > 2b. Why “munlock” is not used afterwards? We want these pages to stay pinned in memory (i.e. the kernel shouldn't be allowed to move them). -- Thanks, Anatoly