From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 15375A034C; Tue, 30 Aug 2022 11:49:56 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A5F3440F18; Tue, 30 Aug 2022 11:49:55 +0200 (CEST) Received: from out203-205-221-235.mail.qq.com (out203-205-221-235.mail.qq.com [203.205.221.235]) by mails.dpdk.org (Postfix) with ESMTP id B457940F17 for ; Tue, 30 Aug 2022 11:49:53 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=qq.com; s=s201512; t=1661852989; bh=Fc0bWu0SDpbJio5mCPpQW9w8Zd+3zAFCReFlubxHVoM=; h=Date:From:To:Cc:Subject:References:In-Reply-To; b=KKkW1kIBYhFr7DsAG4W2Duzp/2qTtgEkicDqU7JNOjUJEYr9g2pQK2Pqd+6J1oXFa b8plp7wF+B0LwRs9Rjl1oNgopl4zkKKr4JSHcaEkVu+iSji+CO265Pi/4eOlhEWmt2 jSMgt+iDmUffsHYpTijJo7X3iaMCZjvHK+h74sR8= Received: from vscode.7~ ([36.111.64.85]) by newxmesmtplogicsvrszc13.qq.com (NewEsmtp) with SMTP id C6A20C35; Tue, 30 Aug 2022 17:49:42 +0800 X-QQ-mid: xmsmtpt1661852982tu7jgq9a3 Message-ID: X-QQ-XMAILINFO: MmPNY57tR1XnC5BQJv4xRruNjK5hydBL7j3sq/fXsfgj+JacD+b7BrTAouxYO/ eAJdqZNudZiHGdlbkUzupJxVkoh0ftuHhzs/ofksUc/hq/jEVv+e45x9DDXGGHTnUJrOUNUywnK1 3ML/cop5nDZzD03O47B2OEulwqo+Ygj0wANmVPALcv9xF9WpmrMkhw6AjsdRxTda6nSpbOY1rvel NeO6+Huzdw/gN3FJQqMECPRjGIdJfuCoqXK9fL6HZrNKk9ojK0Q31vZK2tBPtr1J56Rx4ytVpYDf /JB8Ww0rnriE1X0/2hXaDZtaRBlQS9Laam0q4p9drSWwadwXjKfEOh+mYyTHfxXqD1aYvLpQ9J7f dr/aeTykyYhyRda3C0uLYq4B9rddX+ePUaCmO+8a5QBf4pHH58qGmX2SI9fIoeokTGHKQZ9NTGzB DuV9QBaNDOgps7FUPSarkAcdg+YHZxGIqTA8dwbLu+kcrZLos2Y7Ufdg2vZz+tiR7B2lZUeoLsxw va3xthZTdVNv8PTkqymNSPC2NGkzIzxdV3lOHsuDPuvrK8NqdksrSW36JUSgc9etmO7fLbl0NXH8 m4QW74JiyWpEI7J18IWKYYZNGrMBucV0tj4KWXL+h4czBFllz/8KnU10X8iWLQeVbzyLtwRFLb+U DtOkmE9gPl6mIfa/kHZTkm+4/dpJlLud+j+9zih7piPEPCKuh15zIsCRIuZgCJAj6XnWVhljVwh1 A1CX2uDq3SeOr6dghf42cGt2oaqAmKmYqDNYx1hfhbSLx0Bz+2In5rKAm9YYkVCRH8lYsTGSMTP2 lZ4KRz9vJlaYu+ffb50XD4DacoIPnEmwds2wc/fQkG8ycozSnaL+CPCl/x91KMCi+lhyV4RHFmSx 9rHhPrdkgIAyCDHNR66AkCJG89IBNu3VmNkY1zqBrhicTVKLR0fVlvbobIgeTdPmGYH8eih/KPaT z9ElEdeaDECYrzQ8XJBA== Date: Tue, 30 Aug 2022 09:49:42 +0000 From: lic121 To: Dmitry Kozlyuk Cc: Morten =?iso-8859-1?Q?Br=F8rup?= , David Marchand , dev Subject: Re: [PATCH] eal: zero out new added memory X-OQ-MSGID: <20220830094942.GA16941@vscode.7~> References: <20220827125750.291dd7d1@sovereign> <20220827175654.7a167eaf@sovereign> <98CBD80474FA8B44BF855DF32C47DC35D872CC@smartserver.smartshare.dk> <20220829154925.6575540a@sovereign> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Tue, Aug 30, 2022 at 01:11:25AM +0000, lic121 wrote: > On Mon, Aug 29, 2022 at 03:49:25PM +0300, Dmitry Kozlyuk wrote: > > 2022-08-29 14:37 (UTC+0200), Morten Brørup: > > > > From: David Marchand [mailto:david.marchand@redhat.com] > > > > Sent: Monday, 29 August 2022 13.58 > > > > > > > > > > > > On Sat, Aug 27, 2022 at 12:57:50PM +0300, Dmitry Kozlyuk wrote: > > > > > > > > > The kernel ensures that the newly mapped memory is zeroed, > > > > > > > > > and DPDK ensures that files in hugetlbfs are not re-mapped. > > > > > > David, are you suggesting that this invariant - guaranteeing that DPDK memory is zeroed - was violated by SELinux in the SELinux/container issue you were tracking? > > > > > > If so, the method to ensure the invariant is faulty for SELinux. Assuming DPDK supports SELinux, this bug should be fixed. > > > > +1, I'd like to know more about that case. > > > > EAL checks the unlink() result, so if it fails, the allocation should fail > > and the invariant should not be broken. > > Code from 20.11.5: > > > > if (rte_eal_process_type() == RTE_PROC_PRIMARY && > > unlink(path) == -1 && > > errno != ENOENT) { > > RTE_LOG(DEBUG, EAL, "%s(): could not remove '%s': %s\n", > > __func__, path, strerror(errno)); > > return -1; > > } > > > > Can SELinux restriction result in errno == ENOENT? > > I'd expect EPERM/EACCESS. > > Thanks for your info, the selinux is disabled on my server. Also I > checked that the selinux fix is already in my dpdk. Could any other > settings may cause dirty memory? If you can think of any thing related, > I can have a try. > > BTW, this is my nic info: > ``` > Intel Corporation Ethernet Controller E810-XXV for SFP (rev 02) > > driver: ice > version: 1.9.3 > firmware-version: 2.30 0x80005d22 1.2877.0 > expansion-rom-version: > bus-info: 0000:3b:00.1 > supports-statistics: yes > supports-test: yes > supports-eeprom-access: yes > supports-register-dump: yes > supports-priv-flags: yes > ``` update with more debugs: Preparation: 1. set hugepage size to 2 GB. ``` [root@gz15-compute-s3-55e247e16e22 huge]# grep -i huge /proc/meminfo AnonHugePages: 124928 kB ShmemHugePages: 0 kB HugePages_Total: 2 HugePages_Free: 2 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 1048576 kB Hugetlb: 2097152 kB ``` 2. make a simple programe to poison memory ```c #include #include #include static int memvcmp(void *memory, unsigned char val, size_t size) { unsigned char *mm = (unsigned char*)memory; return (*mm == val) && memcmp(mm, mm + 1, size - 1) == 0; } int main(int argc, char *argv[]){ size_t size = 2 * (1 << 30)-1; void *ptr2 = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB, -1, 0); if (! ptr2) { printf("failed to allocted mm"); return 0; } if (argc > 1) { memset(ptr2, 0xff, size); } unsigned char * ss = ptr2; printf("ss: %x\n", *ss); if (memvcmp(ptr2, 0, size)){ printf("all zero\n"); } else { printf("not all zero\n"); } } ``` 3. insert debug info to check if memory all zero ``` diff --git a/lib/librte_eal/common/malloc_heap.c b/lib/librte_eal/common/malloc_heap.c index 5a09247a6..026560333 100644 --- a/lib/librte_eal/common/malloc_heap.c +++ b/lib/librte_eal/common/malloc_heap.c @@ -91,16 +91,32 @@ malloc_socket_to_heap_id(unsigned int socket_id) /* * Expand the heap with a memory area. */ +static int memvcmp(void *memory, unsigned char val, size_t size) +{ + unsigned char *mm = (unsigned char*)memory; + return (*mm == val) && memcmp(mm, mm + 1, size - 1) == 0; +} static struct malloc_elem * malloc_heap_add_memory(struct malloc_heap *heap, struct rte_memseg_list *msl, void *start, size_t len) { struct malloc_elem *elem = start; + void *ptr; + size_t data_len; + malloc_elem_init(elem, heap, msl, len, elem, len); malloc_elem_insert(elem); + ptr = RTE_PTR_ADD(elem, MALLOC_ELEM_HEADER_LEN); + data_len = elem->size - MALLOC_ELEM_OVERHEAD; + if (memvcmp(ptr, 0, data_len)){ + RTE_LOG(ERR, EAL, "liiiiiiilog: all zero\n"); + } else { + RTE_LOG(ERR, EAL, "liiiiiiilog: not all zero\n"); + } + elem = malloc_elem_join_adjacent_free(elem); malloc_elem_free_list_insert(elem); ``` debug steps: 1. poison 2GB memory ``` [root@gz15-compute-s3-55e247e16e22 secure]# rm -rf /dev/hugepages/rtemap_* ; huge/a.out 1 ss: ff not all zero ``` 2. Run testpmd(with no nic bind vfio-pci) ``` [root@gz15-compute-s3-55e247e16e22 secure]# dpdk-testpmd -l 0-3 -n 4 -- -i --nb-cores=3 EAL: Detected 64 lcore(s) EAL: Detected 2 NUMA nodes EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: Selected IOVA mode 'VA' EAL: Probing VFIO support... EAL: VFIO support initialized EAL: liiiiiiilog: not all zero EAL: No legacy callbacks, legacy socket not created testpmd: No probed ethernet devices Interactive-mode selected testpmd: create a new mbuf pool : n=171456, size=2176, socket=0 testpmd: preferred mempool ops selected: ring_mp_mc testpmd: create a new mbuf pool : n=171456, size=2176, socket=1 testpmd: preferred mempool ops selected: ring_mp_mc EAL: liiiiiiilog: not all zero Done testpmd> ``` Dirty memory happens even no nic probe. I tried on two CPUs, the same issue. - Intel(R) Xeon(R) Gold 5218 CPU @ 2.30GHz - Intel(R) Xeon(R) Platinum 8378A CPU @ 3.00GHz