From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E7688425A6; Fri, 15 Sep 2023 17:33:49 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 7397F402B8; Fri, 15 Sep 2023 17:33:49 +0200 (CEST) Received: from mail-pf1-f169.google.com (mail-pf1-f169.google.com [209.85.210.169]) by mails.dpdk.org (Postfix) with ESMTP id 0C84A402AF for ; Fri, 15 Sep 2023 17:33:48 +0200 (CEST) Received: by mail-pf1-f169.google.com with SMTP id d2e1a72fcca58-68fb6fd2836so2071751b3a.0 for ; Fri, 15 Sep 2023 08:33:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20230601.gappssmtp.com; s=20230601; t=1694792027; x=1695396827; darn=dpdk.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=m2OxOqxrabIcoSUijaBECF7HI+UWcu5b+xS9LQ5kpiI=; b=n8BDHyx+V7D89M9W98DuqZ2jNJINgMC4GtxUKwkr8qh5Rkp47Wjb6PSFnEKez79ssG xMz3M/pn538+lYtAXiB60DGE7sFWRfT68wYSbkIcHRa0VXqMepg19fsuC8ACv8HGGPrV G4OqzCQPiev8OeqzOXju7eEjqU6DKGiiY6U7vTyEQyh5CzGKyz7kTDuwCb5i73U96VFh +BVnbiXqSm3ZPZVgIDauMYWmgcmdOfWB9zlo4XrDw/JOW0Qun8qKT37YYbvfU+k15N9T DxVfTnX5nFmd7qDrFrJ/qk8e4H089upX3hcy2rmTTQUm2OG8qwE42P84aHJkFPzWDDN6 GTFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694792027; x=1695396827; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=m2OxOqxrabIcoSUijaBECF7HI+UWcu5b+xS9LQ5kpiI=; b=rFNpTHzUivLV1rCAdswkA7GzQ0Ay+zMLvGU7+Uo6XzVvmnTjGC4/7nTT2IGelpCafj m0k/je/pnAM+kT4k/PDsVX9KEqvDtpcG94KqhluSMgV9kulWxbMdCB5EE3+411f24FMg gkicxhqfO5Envt7roiW5xZsjoexSsDpjnomQ9yDvxzA6q/zZyaC6h2AjK12mtHE/OOY+ +3Jj9nv8jO+6G+/0MwcwANDWWK/V/NuP5PC/CJ9HIT3lD3m9/f8U6qPDw+5SMmoJQn4S XHI7Dq3q8ZQVrMWAHfuuZMIvRpkv0blf0+sPYzQ/qre59029u0PL+ikTxU08fJwoahOq T+Ng== X-Gm-Message-State: AOJu0Yxt0bBBBy3MlNJKr2eo/odr2vyQovbNSDomnUlBq29icllZ8nKr RSzDNJAFQs497BF9MztrC4XNrg== X-Google-Smtp-Source: AGHT+IH7NWqhVUdS6By2xPdAdMyCO4DqQqYuEfFI6pDmllDRkVGG5Q1qKuUF6lHbWy2F3UEYi0z8qA== X-Received: by 2002:a05:6a20:6621:b0:153:560e:e001 with SMTP id n33-20020a056a20662100b00153560ee001mr1770512pzh.48.1694792027118; Fri, 15 Sep 2023 08:33:47 -0700 (PDT) Received: from hermes.local (204-195-112-131.wavecable.com. [204.195.112.131]) by smtp.gmail.com with ESMTPSA id z7-20020a633307000000b005646e6634dcsm2891011pgz.83.2023.09.15.08.33.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 15 Sep 2023 08:33:46 -0700 (PDT) Date: Fri, 15 Sep 2023 08:33:43 -0700 From: Stephen Hemminger To: Morten =?UTF-8?B?QnLDuHJ1cA==?= Cc: "Van Haaren, Harry" , "Thomas Monjalon" , , "Richardson, Bruce" , "Andrew Rybchenko" , "Chaoyong He" , "Niklas Soderlund" , "Chris Brezovec (cbrezove)" , , , "Tyler Retzlaff" Subject: Re: drivers use of service cores Message-ID: <20230915083343.0539729e@hermes.local> In-Reply-To: <98CBD80474FA8B44BF855DF32C47DC35D87BAD@smartserver.smartshare.dk> References: <6541993.tM3a2QDmDi@thomas> <98CBD80474FA8B44BF855DF32C47DC35D87BAD@smartserver.smartshare.dk> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Fri, 15 Sep 2023 15:54:02 +0200 Morten Br=C3=B8rup wrote: > > For context, Thomas and I (and a few others) had a brief discussion > > about this topic > > at userspace in Dublin earlier this week. I have a bit of better > > understanding of the > > problem-space, and we made some progress in technical solutions too. > > =20 > > > I think we can improve the developer experience for using service =20 > > cores =20 > > > from a driver, like finding or allocating a service core. > > > We may take some code and ideas from sfc and nfp drivers, > > > like in these functions: > > > nfp_map_service() > > > sfc_mae_counter_service_register() > > > sfc_get_service_lcore() > > > > > > If it is not possible to use a service core, we could default to usin= g =20 > > a control thread. =20 > > > So the driver would never fail because of a thread initialization. =20 > >=20 > > There was input from a few people that "hidden threads" that their DPDK > > application > > doesn't know about can cause issues (e.g. a driver creating a thread > > "behind the application's back"). > > I think Thomas suggested a callback function the application could hook- > > into, to either accept/decline > > the drivers "request" to create a thread. > >=20 > > The default could be "accept" if the application doesn't hook the > > callback, allowing drivers to default to > > achieving work, and allowing power-users to manually handle specific > > threading-requirements. I have > > not strong preference here, just writing down the discussions and > > feedback from Userspace. > > =20 > > > What do you think about proposing such a high level API > > > in order to get more drivers using it? =20 > > I believe service-cores was required to transparently enable certain > > use-cases of HW-acceleration, > > Initially Eventdev/SW PMD, but it is of course possible for other > > components in DPDK to use it. > >=20 > > I do recall some folks had concerns over "scope creep" when initially > > discussing service-cores upstreaming, and perhaps they're right. > > I'm not sure how much more functionality is desired here, vs better > > usability of the service-cores APIs. Perhaps a POC patch of the > > NFP, SFC, etc use-cases would help drive towards a code-level > > discussion? =20 >=20 > Since the discussion in Dublin, I have given this some thoughts. Here's w= hat I think... >=20 > My key objection is this: CPU cycles and CPU cores might be scarce resour= ces. And even though a driver has some "important" work to do, the applicat= ion might have some other work to do, which is more important and perhaps a= lso timing sensitive. It is a core design principle of DPDK that the applic= ation is in full control! Drivers should not have the ability to take away = resources from the application; resources should be explicitly allocated an= d given to the driver by the application. In the embedded world, it is common that cores and other resources are scar= ce. At my previous role, the number of available cores and memory was controlle= d by cgroups and the actual allocation decisions were done high up in the project manage= ment team. I doubt this is a unique case. Also, when a network virtual appliance is run in a VM. The number of cores in the VM impacts the cost. You pay a lot more for more cores. Bottom line, cores are not free. Many applications are not setup to use service lcores. And some are not even setup to use timers.