From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id CB9BCA0577; Tue, 14 Apr 2020 21:44:43 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id D5ABF1D173; Tue, 14 Apr 2020 21:44:42 +0200 (CEST) Received: from mail-lj1-f196.google.com (mail-lj1-f196.google.com [209.85.208.196]) by dpdk.org (Postfix) with ESMTP id 392201D15C for ; Tue, 14 Apr 2020 21:44:41 +0200 (CEST) Received: by mail-lj1-f196.google.com with SMTP id q22so1179927ljg.0 for ; Tue, 14 Apr 2020 12:44:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=A0Awuq44jBkjSznmOAcHiEupZPcUrs+Crv2Y4t1Xmpk=; b=MeqMVMUhFsTy0ofXlrSwL3VLXTq//jHcuXfQWmE68Ls6rwrzoZYF+2SqIedooj/yj3 9pVbSi7v0JPoK88bFf6BK1/zci4iv4NJ8yzz+ppbuJqmlotpm7FrxXkVfjbZ9jaQSFj+ 5B5vkJjEBVHGOlCckLzLdiXnF7FacyQQ6rMTQfegea6fBXhSyl/+ZjY0jIllX1wbLP8X KvptSuWgNWxK3pQQDeO6p/W6U7crzQolJKd7ubG+qumZs07AShPDuiOLWY2d4zSmT20I 8WJKijThVa1GvwQAWZoeLuj7uRmgzt3EhtJ/uACZ4keEc0RjfZXiklsFofMlfwCZM0Ut AakQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=A0Awuq44jBkjSznmOAcHiEupZPcUrs+Crv2Y4t1Xmpk=; b=hSD/Ve3wbRIqhoItYeLioNfC9vx4xPiFw75Zr36dlGKvAnsAFuesxixzqFP8o0w98h w/LqVmAHfWfR08XkMYK0BeaKtcbwJ+ziK9X10aS6ZxioVNwwnM0JOiiuSrOx97jew+8q jeh4in4SEuE6LfFL5G582kq16R7Ldj71Jj8h/MjrXaKzTks1bKLy3imVA1opon344h9a JqEaJXVl/snmA1/jwl5CcS7UqzmRrTo7xOVRx1JmNqBWV0ei2h5vujONazWoJB9I6Uh0 ymtr20ADMK4LKYMQJb3jzFiXLu0bYa0r9JepXrLBYk9QBEG1LU8d/XS5MWL/deOF2mhd bUFw== X-Gm-Message-State: AGi0PuaBj3GePERLi5WoWLWu/6oZEz9aEtIW7yNUmAYkwB55NBrbASsk WLvylot1kMMErqhlEae5wfhzvS3/h74qSQ== X-Google-Smtp-Source: APiQypLcWchV1asJkCs5TSsFgD6jEJ6bKVsMO5XcvLP4lQuzSB09unKlT99bX0G+h12Tb++45LB/9A== X-Received: by 2002:a05:651c:449:: with SMTP id g9mr1069411ljg.278.1586893480129; Tue, 14 Apr 2020 12:44:40 -0700 (PDT) Received: from localhost.localdomain (broadband-37-110-65-23.ip.moscow.rt.ru. [37.110.65.23]) by smtp.googlemail.com with ESMTPSA id h14sm11272426lfm.60.2020.04.14.12.44.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Apr 2020 12:44:39 -0700 (PDT) From: Dmitry Kozlyuk To: dev@dpdk.org Cc: "Dmitry Malloy (MESHCHANINOV)" , Narcisa Ana Maria Vasile , Fady Bader , Tal Shnaiderman , Dmitry Kozlyuk Date: Tue, 14 Apr 2020 22:44:15 +0300 Message-Id: <20200414194426.1640704-1-dmitry.kozliuk@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200410164342.1194634-1-dmitry.kozliuk@gmail.com> References: <20200410164342.1194634-1-dmitry.kozliuk@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Subject: [dpdk-dev] [PATCH v3 00/10] Windows basic memory management X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Note: no changes in cover letter since v2. This patchset implements basic MM with the following features: * Hugepages are dynamically allocated in user-mode. * Only 2MB hugepages are supported. * IOVA is always PA, obtained through kernel-mode driver. * No 32-bit support (presumably not demanded). * Ni multi-process support (it is forcefully disabled). * No-huge mode for testing with IOVA unavailable. The first commit introduces a new kernel-mode driver, virt2phys. It translates user-mode virtual addresses into physical addresses. On Windows community call 2020-04-01 it was decided this driver can be used for now, later netUIO may pick up its code/interface or not. New EAL public functions for memory mapping are introduced to mitigate OS differences in DPDK libraries and applications: * rte_mem_map * rte_mem_unmap * rte_mem_lock To support common MM routines, internal wrappers for low-level memory reservation and file management are introduced. These changes affect Linux and FreeBSD EAL. Shared code is placed unded /unix/ subdirectory (suggested by Thomas). Also, entire is imported from FreeBSD, replacing existing partial import. There is already a license exception for this file. Windows MM duplicates quite a lot of code from Linux EAL: * eal_memalloc_alloc_seg_bulk * eal_memalloc_free_seg_bulk * calc_num_pages_per_socket * rte_eal_hugepage_init Perhaps this should be left as-is until Windows MM evolves into having some specific requirements for these parts. Notes on checkpatch warnings: * No space after comma / no space before closing parent in macros--- definitely a false-positive, unclear how to suppress this. * Issues from imported BSD code---probably should be ignored? * Checkpatch is not run against dpdk-kmods (Windows drivers). --- v3: * Fix Linux build on and aarch64 and 32-bit x86 (reported by CI). * Fix logic and error handling while allocating segments. * Fix Unix rte_mem_map(): return NULL on failure. * Fix some checkpatch.sh issues: * Do not return positive errno, use DWORD for GetLastError(). * Make dpdk-kmods source files non-executable. * Improve GSG for Windows Server (suggested by Ranjit Menon). v2: * Rebase on ToT. Move all new code shared between Linux and FreeBSD to /unix/ subdirectory, also factor out some existing code there. * Improve description of Clang issue with rte_page_sizes on Windows. Restore -fstrict-enum for EAL. Check running, not target compiler. * Use EAL prefix for private facilities instead if RTE. * Improve documentation comments for new functions. * Remove co-installer for virt2phys. Add a typecast for clarity. * Document virt2phys in user guide, improve its own README. * Explicitly and forcefully disable multi-process. Dmitry Kozlyuk (9): eal/windows: do not expose private EAL facilities eal/windows: improve CPU and NUMA node detection eal/windows: initialize hugepage info eal: introduce internal wrappers for file operations eal: introduce memory management wrappers eal: extract common code for memseg list initialization eal/windows: fix rte_page_sizes with Clang on Windows eal/windows: replace sys/queue.h with a complete one from FreeBSD eal/windows: implement basic memory management config/meson.build | 12 +- doc/guides/windows_gsg/build_dpdk.rst | 20 - doc/guides/windows_gsg/index.rst | 1 + doc/guides/windows_gsg/run_apps.rst | 84 ++ lib/librte_eal/common/eal_common_fbarray.c | 57 +- lib/librte_eal/common/eal_common_memory.c | 104 +- lib/librte_eal/common/eal_private.h | 134 +- lib/librte_eal/common/malloc_heap.c | 1 + lib/librte_eal/common/meson.build | 9 + lib/librte_eal/freebsd/eal_memory.c | 55 +- lib/librte_eal/include/rte_memory.h | 74 ++ lib/librte_eal/linux/eal_memory.c | 68 +- lib/librte_eal/meson.build | 4 + lib/librte_eal/rte_eal_exports.def | 119 ++ lib/librte_eal/rte_eal_version.map | 4 + lib/librte_eal/unix/eal.c | 47 + lib/librte_eal/unix/eal_memory.c | 113 ++ lib/librte_eal/unix/meson.build | 7 + lib/librte_eal/windows/eal.c | 160 +++ lib/librte_eal/windows/eal_hugepages.c | 108 ++ lib/librte_eal/windows/eal_lcore.c | 187 ++- lib/librte_eal/windows/eal_memalloc.c | 418 ++++++ lib/librte_eal/windows/eal_memory.c | 1141 +++++++++++++++++ lib/librte_eal/windows/eal_mp.c | 103 ++ lib/librte_eal/windows/eal_thread.c | 1 + lib/librte_eal/windows/eal_windows.h | 129 ++ lib/librte_eal/windows/include/meson.build | 2 + lib/librte_eal/windows/include/pthread.h | 2 + lib/librte_eal/windows/include/rte_os.h | 48 +- .../windows/include/rte_virt2phys.h | 34 + lib/librte_eal/windows/include/rte_windows.h | 43 + lib/librte_eal/windows/include/sys/queue.h | 663 +++++++++- lib/librte_eal/windows/include/unistd.h | 3 + lib/librte_eal/windows/meson.build | 6 + 34 files changed, 3611 insertions(+), 350 deletions(-) create mode 100644 doc/guides/windows_gsg/run_apps.rst create mode 100644 lib/librte_eal/unix/eal.c create mode 100644 lib/librte_eal/unix/eal_memory.c create mode 100644 lib/librte_eal/unix/meson.build create mode 100644 lib/librte_eal/windows/eal_hugepages.c create mode 100644 lib/librte_eal/windows/eal_memalloc.c create mode 100644 lib/librte_eal/windows/eal_memory.c create mode 100644 lib/librte_eal/windows/eal_mp.c create mode 100644 lib/librte_eal/windows/eal_windows.h create mode 100644 lib/librte_eal/windows/include/rte_virt2phys.h create mode 100644 lib/librte_eal/windows/include/rte_windows.h -- 2.25.1