From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 7ED58A0C47; Tue, 12 Oct 2021 19:33:06 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id F28CC410E5; Tue, 12 Oct 2021 19:33:05 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by mails.dpdk.org (Postfix) with ESMTP id C5333410E1 for ; Tue, 12 Oct 2021 19:33:04 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1634059984; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QsUw6duIz6R+rWPtvAiFteJylvkIZILOJwSnM5RJjj4=; b=av1vzB3IEnb3/2Xqg7knotSXU1ws6A9Im9blzicteUE+e7t5MrO6LGpO6KocKNbrSb7W6i CypMF2awTC9B/PzOaVMRpQ9gIIYgWYC/QPjQqz2lrRXgRHXg9W97afzJo3/1FKBA6bXJJN blXElnndvr74mk+2Oz6XOuIAmgGgRII= Received: from mail-lf1-f72.google.com (mail-lf1-f72.google.com [209.85.167.72]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-603-DoDP0VrxNKiuzOMbTjzqqQ-1; Tue, 12 Oct 2021 13:33:02 -0400 X-MC-Unique: DoDP0VrxNKiuzOMbTjzqqQ-1 Received: by mail-lf1-f72.google.com with SMTP id br42-20020a056512402a00b003fd94a74905so34117lfb.7 for ; Tue, 12 Oct 2021 10:33:02 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=QsUw6duIz6R+rWPtvAiFteJylvkIZILOJwSnM5RJjj4=; b=ciQrNKwjyY9XHRYiBwYzvXP2PqHqBkGeDzN/LpNbRYxlj9HXry4ffFCTR9UjW/ztuD asR9Jc5Ny6/Tsic5L71I2eotHVnIEZkIlw1kj9/6w4smLRbLddm4jfDJNu6CvHyaLH3y gwZab0hfG9I8iEv64iwtLXDUlk43P9Y34C1FnnNNqyoDkQDfhiHxt2NG1FWB04Mz9whE uUY4xRtfqsZKj1Q7mBtHD5QseCsJbbJmPRzVJ7AzErnEe2TfwGuD7xVHbWp9HwLqXrR4 9MwaWX3CWgpIk20rhiKyqW2UJsMzTlgOavzboN3HRqG2h+NjJTjDKc8jLJ5L9NvdvQ/n q8iw== X-Gm-Message-State: AOAM5315+2ycg8Mqbzd3f46/yhy7XJQt6IRQT1dgS07o5hMQ+ifJhabn m1w1qrp80hxVnjh2CjRbDY/PYoUIipnNgqoCWd9GUwweLnhqnfoT5ysSQgpyM4jWcC0kcs18MjU /ka986bzlAeBDrdB+NZg= X-Received: by 2002:a05:6512:31c3:: with SMTP id j3mr31980381lfe.217.1634059981317; Tue, 12 Oct 2021 10:33:01 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwKghNDmV+zQOr/e2WxMigrSFlnryX2kvLBdK/yw6Pj24rF3SALudqVwiLCKLsRNSAibjh+GkpuJpoy1in9JUM= X-Received: by 2002:a05:6512:31c3:: with SMTP id j3mr31980333lfe.217.1634059980844; Tue, 12 Oct 2021 10:33:00 -0700 (PDT) MIME-Version: 1.0 References: <20210921081632.858873-1-dkozlyuk@nvidia.com> <20211011085644.2716490-1-dkozlyuk@nvidia.com> <20211011085644.2716490-3-dkozlyuk@nvidia.com> In-Reply-To: From: David Marchand Date: Tue, 12 Oct 2021 19:32:49 +0200 Message-ID: To: Dmitry Kozlyuk Cc: dev , Slava Ovsiienko , Anatoly Burakov , Thomas Monjalon Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=dmarchan@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [dpdk-dev] [PATCH v6 2/3] eal: add memory pre-allocation from existing files X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On Tue, Oct 12, 2021 at 5:55 PM Dmitry Kozlyuk wrote: > > I have some trouble figuring the need for the list of files. > > Why not use a global knob --mem-clear-on-alloc for this behavior change= ? > > Moving memset() doesn't speed anything up, it's a forced step for the rea= sons below. > Currently, memory is cleared by the kernel when a page is mapped during a= n allocation. > This cannot be turned off in stock kernels. The issue is that initial all= ocations are longer > by the time needed to clear the pages, which is >90%. For the memory inte= nded for DMA this time is just wasted. If allocations are large, applicatio= n startup and restart take long. The only way to get hugepages mapped witho= ut the kernel clearing them is to map existing files in hugetlbfs. However,= rte_zmalloc() needs to return clean memory, that's why we move memset() th= ere. Memory intended for DMA is just never cleared this way. But memory fre= ed and allocated again will be cleared again, unfortunately. Writing my limited understanding, please correct me. The --mem-file that is proposed does: - preallocate files which is something close to --socket-mem with the following differences - --mem-file lets user decide on dpdk hugepage files names, which I think conflicts with --huge-dir and --file-prefix, - --mem-file lets user device on hugepage size which I think could be achieved with some --huge-dir option, - bypasses unlink() of existing hugepage files which I had overlooked but is the main painpoint, - enforces "clear on alloc" in rte_malloc/rte_free. >From this, I see two parts in this patch: - faster restart, reusing hugepage files as is (combination of not calling unlink() and doing "clear on alloc"), This part is interesting, and I think a single knob for this would be eno= ugh. - finegrained control of hugepage files, but it has the drawback of imposing primary/secondary run with the same options. The second part seems complex to configure. I see conflicts with existing options, so it seems a good way to get caught up in the carpet (sorry if it translates badly from French :p). --=20 David Marchand