* Re: [dpdk-dev] [PATCH v3 0/7] vfio/pci: SR-IOV support
2020-03-11 21:58 2% [dpdk-dev] [PATCH v3 0/7] vfio/pci: SR-IOV support Alex Williamson
@ 2020-03-19 6:32 0% ` Tian, Kevin
0 siblings, 0 replies; 200+ results
From: Tian, Kevin @ 2020-03-19 6:32 UTC (permalink / raw)
To: Alex Williamson, kvm
Cc: linux-pci, linux-kernel, dev, mtosatti, thomas, bluca,
jerinjacobk, Richardson, Bruce, cohuck
> From: Alex Williamson <alex.williamson@redhat.com>
> Sent: Thursday, March 12, 2020 5:58 AM
>
> Only minor tweaks since v2, GET and SET on VFIO_DEVICE_FEATURE are
> enforced mutually exclusive except with the PROBE option as suggested
> by Connie, the modinfo text has been expanded for the opt-in to enable
> SR-IOV support in the vfio-pci driver per discussion with Kevin.
>
> I have not incorporated runtime warnings attempting to detect misuse
> of SR-IOV or imposed a session lifetime of a VF token, both of which
> were significant portions of the discussion of the v2 series. Both of
> these also seem to impose a usage model or make assumptions about VF
> resource usage or configuration requirements that don't seem necessary
> except for the sake of generating a warning or requiring an otherwise
> unnecessary and implicit token reinitialization. If there are new
> thoughts around these or other discussion points, please raise them.
>
> Series overview (same as provided with v1):
>
> The synopsis of this series is that we have an ongoing desire to drive
> PCIe SR-IOV PFs from userspace with VFIO. There's an immediate need
> for this with DPDK drivers and potentially interesting future use
> cases in virtualization. We've been reluctant to add this support
> previously due to the dependency and trust relationship between the
> VF device and PF driver. Minimally the PF driver can induce a denial
> of service to the VF, but depending on the specific implementation,
> the PF driver might also be responsible for moving data between VFs
> or have direct access to the state of the VF, including data or state
> otherwise private to the VF or VF driver.
>
> To help resolve these concerns, we introduce a VF token into the VFIO
> PCI ABI, which acts as a shared secret key between drivers. The
> userspace PF driver is required to set the VF token to a known value
> and userspace VF drivers are required to provide the token to access
> the VF device. If a PF driver is restarted with VF drivers in use, it
> must also provide the current token in order to prevent a rogue
> untrusted PF driver from replacing a known driver. The degree to
> which this new token is considered secret is left to the userspace
> drivers, the kernel intentionally provides no means to retrieve the
> current token.
>
> Note that the above token is only required for this new model where
> both the PF and VF devices are usable through vfio-pci. Existing
> models of VFIO drivers where the PF is used without SR-IOV enabled
> or the VF is bound to a userspace driver with an in-kernel, host PF
> driver are unaffected.
>
> The latter configuration above also highlights a new inverted scenario
> that is now possible, a userspace PF driver with in-kernel VF drivers.
> I believe this is a scenario that should be allowed, but should not be
> enabled by default. This series includes code to set a default
> driver_override for VFs sourced from a vfio-pci user owned PF, such
> that the VFs are also bound to vfio-pci. This model is compatible
> with tools like driverctl and allows the system administrator to
> decide if other bindings should be enabled. The VF token interface
> above exists only between vfio-pci PF and VF drivers, once a VF is
> bound to another driver, the administrator has effectively pronounced
> the device as trusted. The vfio-pci driver will note alternate
> binding in dmesg for logging and debugging purposes.
>
> Please review, comment, and test. The example QEMU implementation
> provided with the RFC is still current for this version. Thanks,
>
> Alex
The whole series looks good to me:
Reviewed-by: Kevin Tian <kevin.tian@intel.com>
and confirm one understanding here, since it is not discussed anywhere. For
VM live migration with assigned VF device, it is not necessary to migrate the
VF token itself and actually we don't allow userspace to retrieve it. Instead,
Qemu just follows whatever token requirement on the dest to open the new
VF: could be same or different token as/from src, or even no token if PF
driver runs in kernel on dest. I suppose either combination could work, correct?
Thanks
Kevin
>
> RFC:
> https://lore.kernel.org/lkml/158085337582.9445.17682266437583505502.stg
> it@gimli.home/
> v1:
> https://lore.kernel.org/lkml/158145472604.16827.15751375540102298130.st
> git@gimli.home/
> v2:
> https://lore.kernel.org/lkml/158213716959.17090.8399427017403507114.stg
> it@gimli.home/
>
> ---
>
> Alex Williamson (7):
> vfio: Include optional device match in vfio_device_ops callbacks
> vfio/pci: Implement match ops
> vfio/pci: Introduce VF token
> vfio: Introduce VFIO_DEVICE_FEATURE ioctl and first user
> vfio/pci: Add sriov_configure support
> vfio/pci: Remove dev_fmt definition
> vfio/pci: Cleanup .probe() exit paths
>
>
> drivers/vfio/pci/vfio_pci.c | 390
> +++++++++++++++++++++++++++++++++--
> drivers/vfio/pci/vfio_pci_private.h | 10 +
> drivers/vfio/vfio.c | 20 +-
> include/linux/vfio.h | 4
> include/uapi/linux/vfio.h | 37 +++
> 5 files changed, 433 insertions(+), 28 deletions(-)
^ permalink raw reply [relevance 0%]
* [dpdk-dev] [PATCH] cryptodev: version rte_cryptodev_info_get function
@ 2020-03-18 20:41 3% Arek Kusztal
0 siblings, 0 replies; 200+ results
From: Arek Kusztal @ 2020-03-18 20:41 UTC (permalink / raw)
To: dev; +Cc: akhil.goyal, fiona.trahe, thomas, Arek Kusztal
This patch adds versioned function rte_cryptodev_info_get.
Node 20.05 function works the same way it was working before.
Node 20.0 function strips capability added in 20.05 release
to prevent some issues with ABI policy. To do that new capability
array is allocated per device and returned to user instead of the
original array passed by PMD.
Signed-off-by: Arek Kusztal <arkadiuszx.kusztal@intel.com>
---
lib/librte_cryptodev/rte_cryptodev.c | 97 +++++++++++++++++++++++++-
lib/librte_cryptodev/rte_cryptodev.h | 12 +++-
lib/librte_cryptodev/rte_cryptodev_version.map | 7 ++
3 files changed, 113 insertions(+), 3 deletions(-)
diff --git a/lib/librte_cryptodev/rte_cryptodev.c b/lib/librte_cryptodev/rte_cryptodev.c
index 6d1d0e9..2d030ba 100644
--- a/lib/librte_cryptodev/rte_cryptodev.c
+++ b/lib/librte_cryptodev/rte_cryptodev.c
@@ -41,6 +41,9 @@
#include "rte_cryptodev.h"
#include "rte_cryptodev_pmd.h"
+#include <rte_compat.h>
+#include <rte_function_versioning.h>
+
static uint8_t nb_drivers;
static struct rte_cryptodev rte_crypto_devices[RTE_CRYPTO_MAX_DEVS];
@@ -56,6 +59,13 @@ static struct rte_cryptodev_global cryptodev_globals = {
/* spinlock for crypto device callbacks */
static rte_spinlock_t rte_cryptodev_cb_lock = RTE_SPINLOCK_INITIALIZER;
+static const struct rte_cryptodev_capabilities
+ cryptodev_undefined_capabilities[] = {
+ RTE_CRYPTODEV_END_OF_CAPABILITIES_LIST()
+};
+
+struct rte_cryptodev_capabilities *capability_copies[RTE_CRYPTO_MAX_DEVS];
+uint8_t is_capability_checked[RTE_CRYPTO_MAX_DEVS];
/**
* The user application callback description.
@@ -999,6 +1009,13 @@ rte_cryptodev_close(uint8_t dev_id)
RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_close, -ENOTSUP);
retval = (*dev->dev_ops->dev_close)(dev);
+
+ if (capability_copies[dev_id]) {
+ free(capability_copies[dev_id]);
+ capability_copies[dev_id] = NULL;
+ }
+ is_capability_checked[dev_id] = 0;
+
if (retval < 0)
return retval;
@@ -1111,11 +1128,12 @@ rte_cryptodev_stats_reset(uint8_t dev_id)
(*dev->dev_ops->stats_reset)(dev);
}
-
void
-rte_cryptodev_info_get(uint8_t dev_id, struct rte_cryptodev_info *dev_info)
+rte_cryptodev_info_get_v20(uint8_t dev_id, struct rte_cryptodev_info *dev_info)
{
struct rte_cryptodev *dev;
+ const struct rte_cryptodev_capabilities *capability;
+ uint8_t counter = 0;
if (!rte_cryptodev_pmd_is_valid_dev(dev_id)) {
CDEV_LOG_ERR("Invalid dev_id=%d", dev_id);
@@ -1129,10 +1147,85 @@ rte_cryptodev_info_get(uint8_t dev_id, struct rte_cryptodev_info *dev_info)
RTE_FUNC_PTR_OR_RET(*dev->dev_ops->dev_infos_get);
(*dev->dev_ops->dev_infos_get)(dev, dev_info);
+ if (capability_copies[dev_id] == NULL) {
+ if (!is_capability_checked[dev_id]) {
+ uint8_t found_invalid_capa = 0;
+
+ for (capability = dev_info->capabilities;
+ capability->op != RTE_CRYPTO_OP_TYPE_UNDEFINED;
+ ++capability, ++counter) {
+ if (capability->op == RTE_CRYPTO_OP_TYPE_SYMMETRIC &&
+ capability->sym.xform_type ==
+ RTE_CRYPTO_SYM_XFORM_AEAD
+ && capability->sym.aead.algo >=
+ RTE_CRYPTO_AEAD_CHACHA20_POLY1305) {
+ found_invalid_capa = 1;
+ counter--;
+ }
+ }
+ is_capability_checked[dev_id] = 1;
+ if (found_invalid_capa) {
+ capability_copies[dev_id] = malloc(counter *
+ sizeof(struct rte_cryptodev_capabilities));
+ if (capability_copies[dev_id] == NULL) {
+ /*
+ * error case - no memory to store the trimmed list,
+ * so have to return an empty list
+ */
+ dev_info->capabilities =
+ cryptodev_undefined_capabilities;
+ is_capability_checked[dev_id] = 0;
+ } else {
+ counter = 0;
+ for (capability = dev_info->capabilities;
+ capability->op !=
+ RTE_CRYPTO_OP_TYPE_UNDEFINED;
+ capability++) {
+ if (!(capability->op ==
+ RTE_CRYPTO_OP_TYPE_SYMMETRIC
+ && capability->sym.xform_type ==
+ RTE_CRYPTO_SYM_XFORM_AEAD
+ && capability->sym.aead.algo >=
+ RTE_CRYPTO_AEAD_CHACHA20_POLY1305)) {
+ capability_copies[dev_id][counter++] =
+ *capability;
+ }
+ }
+ dev_info->capabilities =
+ capability_copies[dev_id];
+ }
+ }
+ }
+ } else
+ dev_info->capabilities = capability_copies[dev_id];
+
dev_info->driver_name = dev->device->driver->name;
dev_info->device = dev->device;
}
+VERSION_SYMBOL(rte_cryptodev_info_get, _v20, 20.0);
+
+void
+rte_cryptodev_info_get_v2005(uint8_t dev_id, struct rte_cryptodev_info *dev_info)
+{
+ struct rte_cryptodev *dev;
+ if (!rte_cryptodev_pmd_is_valid_dev(dev_id)) {
+ CDEV_LOG_ERR("Invalid dev_id=%d", dev_id);
+ return;
+ }
+
+ dev = &rte_crypto_devices[dev_id];
+
+ memset(dev_info, 0, sizeof(struct rte_cryptodev_info));
+
+ RTE_FUNC_PTR_OR_RET(*dev->dev_ops->dev_infos_get);
+ (*dev->dev_ops->dev_infos_get)(dev, dev_info);
+
+ dev_info->driver_name = dev->device->driver->name;
+ dev_info->device = dev->device;
+}
+MAP_STATIC_SYMBOL(void rte_cryptodev_info_get(uint8_t dev_id,
+ struct rte_cryptodev_info *dev_info), rte_cryptodev_info_get_v2005);
int
rte_cryptodev_callback_register(uint8_t dev_id,
diff --git a/lib/librte_cryptodev/rte_cryptodev.h b/lib/librte_cryptodev/rte_cryptodev.h
index 437b8a9..06ce2f2 100644
--- a/lib/librte_cryptodev/rte_cryptodev.h
+++ b/lib/librte_cryptodev/rte_cryptodev.h
@@ -24,6 +24,9 @@ extern "C" {
#include <rte_common.h>
#include <rte_config.h>
+#include <rte_compat.h>
+#include <rte_function_versioning.h>
+
extern const char **rte_cyptodev_names;
/* Logging Macros */
@@ -758,7 +761,14 @@ rte_cryptodev_stats_reset(uint8_t dev_id);
* the last valid element has it's op field set to
* RTE_CRYPTO_OP_TYPE_UNDEFINED.
*/
-extern void
+void
+rte_cryptodev_info_get_v20(uint8_t dev_id, struct rte_cryptodev_info *dev_info);
+
+void
+rte_cryptodev_info_get_v2005(uint8_t dev_id, struct rte_cryptodev_info *dev_info);
+BIND_DEFAULT_SYMBOL(rte_cryptodev_info_get, _v2005, 20.05);
+
+void
rte_cryptodev_info_get(uint8_t dev_id, struct rte_cryptodev_info *dev_info);
diff --git a/lib/librte_cryptodev/rte_cryptodev_version.map b/lib/librte_cryptodev/rte_cryptodev_version.map
index 6e41b4b..d6c945a 100644
--- a/lib/librte_cryptodev/rte_cryptodev_version.map
+++ b/lib/librte_cryptodev/rte_cryptodev_version.map
@@ -58,6 +58,13 @@ DPDK_20.0 {
local: *;
};
+
+DPDK_20.05 {
+ global:
+ rte_cryptodev_info_get;
+} DPDK_20.0;
+
+
EXPERIMENTAL {
global:
--
2.1.0
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-03-17 15:10 4% ` Thomas Monjalon
@ 2020-03-17 19:10 4% ` Kusztal, ArkadiuszX
0 siblings, 0 replies; 200+ results
From: Kusztal, ArkadiuszX @ 2020-03-17 19:10 UTC (permalink / raw)
To: Thomas Monjalon, Trahe, Fiona
Cc: David Marchand, nhorman, bluca, ktraynor, Ray Kinsella, dev,
Akhil Goyal, Yigit, Ferruh, Ananyev, Konstantin, dev,
Anoob Joseph, Richardson, Bruce, Mcnamara, John, dodji,
Andrew Rybchenko, aconole, Trahe, Fiona
> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Tuesday, March 17, 2020 4:10 PM
> To: Trahe, Fiona <fiona.trahe@intel.com>; Kusztal, ArkadiuszX
> <arkadiuszx.kusztal@intel.com>
> Cc: David Marchand <david.marchand@redhat.com>;
> nhorman@tuxdriver.com; bluca@debian.org; ktraynor@redhat.com; Ray
> Kinsella <mdr@ashroe.eu>; dev@dpdk.org; Akhil Goyal
> <akhil.goyal@nxp.com>; Yigit, Ferruh <ferruh.yigit@intel.com>; Ananyev,
> Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org; Anoob Joseph
> <anoobj@marvell.com>; Richardson, Bruce <bruce.richardson@intel.com>;
> Mcnamara, John <john.mcnamara@intel.com>; dodji@seketeli.net; Andrew
> Rybchenko <arybchenko@solarflare.com>; aconole@redhat.com; Trahe,
> Fiona <fiona.trahe@intel.com>
> Subject: Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
>
> 17/03/2020 14:27, Kusztal, ArkadiuszX:
> > Hi Thomas,
> >
> > > -----Original Message-----
> > > From: Thomas Monjalon <thomas@monjalon.net>
> > > Sent: Monday, March 16, 2020 2:09 PM
> > > To: Trahe, Fiona <fiona.trahe@intel.com>
> > > Cc: Kusztal, ArkadiuszX <arkadiuszx.kusztal@intel.com>; David
> > > Marchand <david.marchand@redhat.com>; nhorman@tuxdriver.com;
> > > bluca@debian.org; ktraynor@redhat.com; Ray Kinsella
> <mdr@ashroe.eu>;
> > > dev@dpdk.org; Akhil Goyal <akhil.goyal@nxp.com>; Yigit, Ferruh
> > > <ferruh.yigit@intel.com>; Ananyev, Konstantin
> > > <konstantin.ananyev@intel.com>; dev@dpdk.org; Anoob Joseph
> > > <anoobj@marvell.com>; Richardson, Bruce
> > > <bruce.richardson@intel.com>; Mcnamara, John
> > > <john.mcnamara@intel.com>; dodji@seketeli.net; Andrew Rybchenko
> > > <arybchenko@solarflare.com>; aconole@redhat.com; Trahe, Fiona
> > > <fiona.trahe@intel.com>
> > > Subject: Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
> > >
> > > 16/03/2020 13:57, Trahe, Fiona:
> > > > From: Kusztal, ArkadiuszX <arkadiuszx.kusztal@intel.com>
> > > > > > > > The patch we're working on will provide two versions of
> > > > > > > > rte_cryptodev_info_get(), both call the same PMD function
> > > > > > > > from the
> > > > > > dev_ops info_get fn ptr.
> > > > > > > > The default version operates s as normal, the 19.11
> > > > > > > > version searches through the list returned by the PMD,
> > > > > > > > looking for sym.aead.algo = ChaChaPoly, it needs to strip
> > > > > > > > it from
> > > > > > > the list.
> > > > > > > > As PMDs just pass a ptr to their capabilities list ( it
> > > > > > > > isn't a linked list, but an array with an end marker =
> > > > > > > > RTE_CRYPTODEV_END_OF_CAPABILITIES_LIST) if the API layer
> > > > > > > > detects Chacha it must allocate some space and store a
> > > > > > > > local copy of the trimmed
> > > > > > list. This must be stored only once per device.
> > > > >
> > > > > [Arek] The problem with this solution is that we need to allocate
> memory.
> > > > > So the question is how to handle unlikely case of malloc error
> > > > > when we operate inside void function rte_cryptodev_info_get?
> > > > > And even if we would pass somehow error condition to the caller
> > > > > then
> > > what to do is another question.
> > > >
> > > > [Fiona] Quick recap: To avoid breaking ABI, we must return a set
> > > > of capabilities with/without ChaChaPoly depending on the appl
> > > > version. To resolve this, within the rte_cryptodev layer, we
> > > > propose to inspect the
> > > capabilities returned by PMD and strip ChaCha if it exists.
> > > > In that case memory for the new trimmed capabilities array has to
> > > > be
> > > malloced by the lib.
> > >
> > > What happens if the capability is removed from the original
> > > capabilities input?
> > >
> > > > All good, except how to handle a malloc fail is yet another API
> > > > breakage as
> > > rte_cryptodev_get_info() returns void.
> > > > We propose to return an empty capability list, i.e. a list with
> > > > only the END element (which can be done without malloc) in this
> > > > corner case of
> > > a corner case.
> > > > Anyone see any issue with this?
> > >
> > > How can we use the feature if it is not advertised in capabilities?
> > What Fiona meant is that empty capability would indicate error condition in
> this case. That's why she asked if you ok with this API breakage.
>
> Sorry I'm lost.
> Please could you show what would be the usage?
>
So this case looks more or less like that:
There are two versions of `rte_cryptodev_info_get`
rte_cryptodev_info_get_v20 (versioned)
rte_cryptodev_info_get_v2005 (new default symbol)
Default version works normally as it will be called only by app build with 20.05 version.
When prior to 20.05 app calls `rte_cryptodev_info_get` version v20 is called. This function will remove Chacha Poly from capabilities, but to achieve this we need to get some memory to store `new` set of capabilities per device (without Chacha). So:
new_capability[dev_id] = malloc( (num_of_capabilies - 1 (chacha)) * sizeof(struct rte_cryptodev_capabilities))
The small problem is how to handle malloc error:
If (new_capability[dev_id] == NULL) {
/* What to do now as rte_cryptodev_info_get is void function, and API does not say anything about error condition */
/*So Fiona suggestion above is to inform user of an error by doing this: */
dev_info->capabilities = cryptodev_undefined_capabilities;
/* Where
static const struct rte_cryptodev_capabilities cryptodev_undefined_capabilities[] = {
RTE_CRYPTODEV_END_OF_CAPABILITIES_LIST()
}; */
}
Sizeof rte_cryptodev_capabilities is 38 bytes, padded to 40. So properly constructed capabilities can take at most 1920 bytes. No system should ever fail doing that though iam not an expert. Other option could probably be to preallocate this memory. This is how I understand that.
Another question is can something like that be done if API comments of `rte_cryptodev_info_get` function does not say anything about any potential error?
Regards,
Arek
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-03-17 13:27 4% ` Kusztal, ArkadiuszX
@ 2020-03-17 15:10 4% ` Thomas Monjalon
2020-03-17 19:10 4% ` Kusztal, ArkadiuszX
0 siblings, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-03-17 15:10 UTC (permalink / raw)
To: Trahe, Fiona, Kusztal, ArkadiuszX
Cc: David Marchand, nhorman, bluca, ktraynor, Ray Kinsella, dev,
Akhil Goyal, Yigit, Ferruh, Ananyev, Konstantin, dev,
Anoob Joseph, Richardson, Bruce, Mcnamara, John, dodji,
Andrew Rybchenko, aconole, Trahe, Fiona
17/03/2020 14:27, Kusztal, ArkadiuszX:
> Hi Thomas,
>
> > -----Original Message-----
> > From: Thomas Monjalon <thomas@monjalon.net>
> > Sent: Monday, March 16, 2020 2:09 PM
> > To: Trahe, Fiona <fiona.trahe@intel.com>
> > Cc: Kusztal, ArkadiuszX <arkadiuszx.kusztal@intel.com>; David Marchand
> > <david.marchand@redhat.com>; nhorman@tuxdriver.com;
> > bluca@debian.org; ktraynor@redhat.com; Ray Kinsella <mdr@ashroe.eu>;
> > dev@dpdk.org; Akhil Goyal <akhil.goyal@nxp.com>; Yigit, Ferruh
> > <ferruh.yigit@intel.com>; Ananyev, Konstantin
> > <konstantin.ananyev@intel.com>; dev@dpdk.org; Anoob Joseph
> > <anoobj@marvell.com>; Richardson, Bruce <bruce.richardson@intel.com>;
> > Mcnamara, John <john.mcnamara@intel.com>; dodji@seketeli.net; Andrew
> > Rybchenko <arybchenko@solarflare.com>; aconole@redhat.com; Trahe,
> > Fiona <fiona.trahe@intel.com>
> > Subject: Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
> >
> > 16/03/2020 13:57, Trahe, Fiona:
> > > From: Kusztal, ArkadiuszX <arkadiuszx.kusztal@intel.com>
> > > > > > > The patch we're working on will provide two versions of
> > > > > > > rte_cryptodev_info_get(), both call the same PMD function from
> > > > > > > the
> > > > > dev_ops info_get fn ptr.
> > > > > > > The default version operates s as normal, the 19.11 version
> > > > > > > searches through the list returned by the PMD, looking for
> > > > > > > sym.aead.algo = ChaChaPoly, it needs to strip it from
> > > > > > the list.
> > > > > > > As PMDs just pass a ptr to their capabilities list ( it isn't
> > > > > > > a linked list, but an array with an end marker =
> > > > > > > RTE_CRYPTODEV_END_OF_CAPABILITIES_LIST) if the API layer
> > > > > > > detects Chacha it must allocate some space and store a local
> > > > > > > copy of the trimmed
> > > > > list. This must be stored only once per device.
> > > >
> > > > [Arek] The problem with this solution is that we need to allocate memory.
> > > > So the question is how to handle unlikely case of malloc error when
> > > > we operate inside void function rte_cryptodev_info_get?
> > > > And even if we would pass somehow error condition to the caller then
> > what to do is another question.
> > >
> > > [Fiona] Quick recap: To avoid breaking ABI, we must return a set of
> > > capabilities with/without ChaChaPoly depending on the appl version. To
> > > resolve this, within the rte_cryptodev layer, we propose to inspect the
> > capabilities returned by PMD and strip ChaCha if it exists.
> > > In that case memory for the new trimmed capabilities array has to be
> > malloced by the lib.
> >
> > What happens if the capability is removed from the original capabilities
> > input?
> >
> > > All good, except how to handle a malloc fail is yet another API breakage as
> > rte_cryptodev_get_info() returns void.
> > > We propose to return an empty capability list, i.e. a list with only
> > > the END element (which can be done without malloc) in this corner case of
> > a corner case.
> > > Anyone see any issue with this?
> >
> > How can we use the feature if it is not advertised in capabilities?
> What Fiona meant is that empty capability would indicate error condition in this case. That's why she asked if you ok with this API breakage.
Sorry I'm lost.
Please could you show what would be the usage?
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-03-16 13:09 4% ` Thomas Monjalon
@ 2020-03-17 13:27 4% ` Kusztal, ArkadiuszX
2020-03-17 15:10 4% ` Thomas Monjalon
0 siblings, 1 reply; 200+ results
From: Kusztal, ArkadiuszX @ 2020-03-17 13:27 UTC (permalink / raw)
To: Thomas Monjalon, Trahe, Fiona
Cc: David Marchand, nhorman, bluca, ktraynor, Ray Kinsella, dev,
Akhil Goyal, Yigit, Ferruh, Ananyev, Konstantin, dev,
Anoob Joseph, Richardson, Bruce, Mcnamara, John, dodji,
Andrew Rybchenko, aconole, Trahe, Fiona
Hi Thomas,
> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Monday, March 16, 2020 2:09 PM
> To: Trahe, Fiona <fiona.trahe@intel.com>
> Cc: Kusztal, ArkadiuszX <arkadiuszx.kusztal@intel.com>; David Marchand
> <david.marchand@redhat.com>; nhorman@tuxdriver.com;
> bluca@debian.org; ktraynor@redhat.com; Ray Kinsella <mdr@ashroe.eu>;
> dev@dpdk.org; Akhil Goyal <akhil.goyal@nxp.com>; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Ananyev, Konstantin
> <konstantin.ananyev@intel.com>; dev@dpdk.org; Anoob Joseph
> <anoobj@marvell.com>; Richardson, Bruce <bruce.richardson@intel.com>;
> Mcnamara, John <john.mcnamara@intel.com>; dodji@seketeli.net; Andrew
> Rybchenko <arybchenko@solarflare.com>; aconole@redhat.com; Trahe,
> Fiona <fiona.trahe@intel.com>
> Subject: Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
>
> 16/03/2020 13:57, Trahe, Fiona:
> > From: Kusztal, ArkadiuszX <arkadiuszx.kusztal@intel.com>
> > > > > > The patch we're working on will provide two versions of
> > > > > > rte_cryptodev_info_get(), both call the same PMD function from
> > > > > > the
> > > > dev_ops info_get fn ptr.
> > > > > > The default version operates s as normal, the 19.11 version
> > > > > > searches through the list returned by the PMD, looking for
> > > > > > sym.aead.algo = ChaChaPoly, it needs to strip it from
> > > > > the list.
> > > > > > As PMDs just pass a ptr to their capabilities list ( it isn't
> > > > > > a linked list, but an array with an end marker =
> > > > > > RTE_CRYPTODEV_END_OF_CAPABILITIES_LIST) if the API layer
> > > > > > detects Chacha it must allocate some space and store a local
> > > > > > copy of the trimmed
> > > > list. This must be stored only once per device.
> > >
> > > [Arek] The problem with this solution is that we need to allocate memory.
> > > So the question is how to handle unlikely case of malloc error when
> > > we operate inside void function rte_cryptodev_info_get?
> > > And even if we would pass somehow error condition to the caller then
> what to do is another question.
> >
> > [Fiona] Quick recap: To avoid breaking ABI, we must return a set of
> > capabilities with/without ChaChaPoly depending on the appl version. To
> > resolve this, within the rte_cryptodev layer, we propose to inspect the
> capabilities returned by PMD and strip ChaCha if it exists.
> > In that case memory for the new trimmed capabilities array has to be
> malloced by the lib.
>
> What happens if the capability is removed from the original capabilities
> input?
>
> > All good, except how to handle a malloc fail is yet another API breakage as
> rte_cryptodev_get_info() returns void.
> > We propose to return an empty capability list, i.e. a list with only
> > the END element (which can be done without malloc) in this corner case of
> a corner case.
> > Anyone see any issue with this?
>
> How can we use the feature if it is not advertised in capabilities?
What Fiona meant is that empty capability would indicate error condition in this case. That's why she asked if you ok with this API breakage.
>
>
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-03-16 12:57 8% ` Trahe, Fiona
@ 2020-03-16 13:09 4% ` Thomas Monjalon
2020-03-17 13:27 4% ` Kusztal, ArkadiuszX
0 siblings, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-03-16 13:09 UTC (permalink / raw)
To: Trahe, Fiona
Cc: Kusztal, ArkadiuszX, David Marchand, nhorman, bluca, ktraynor,
Ray Kinsella, dev, Akhil Goyal, Yigit, Ferruh, Ananyev,
Konstantin, dev, Anoob Joseph, Richardson, Bruce, Mcnamara, John,
dodji, Andrew Rybchenko, aconole, Trahe, Fiona
16/03/2020 13:57, Trahe, Fiona:
> From: Kusztal, ArkadiuszX <arkadiuszx.kusztal@intel.com>
> > > > > The patch we're working on will provide two versions of
> > > > > rte_cryptodev_info_get(), both call the same PMD function from the
> > > dev_ops info_get fn ptr.
> > > > > The default version operates s as normal, the 19.11 version searches
> > > > > through the list returned by the PMD, looking for sym.aead.algo =
> > > > > ChaChaPoly, it needs to strip it from
> > > > the list.
> > > > > As PMDs just pass a ptr to their capabilities list ( it isn't a
> > > > > linked list, but an array with an end marker =
> > > > > RTE_CRYPTODEV_END_OF_CAPABILITIES_LIST) if the API layer detects
> > > > > Chacha it must allocate some space and store a local copy of the trimmed
> > > list. This must be stored only once per device.
> >
> > [Arek] The problem with this solution is that we need to allocate memory.
> > So the question is how to handle unlikely case of malloc error when we operate inside void function
> > rte_cryptodev_info_get?
> > And even if we would pass somehow error condition to the caller then what to do is another question.
>
> [Fiona] Quick recap: To avoid breaking ABI, we must return a set of capabilities with/without ChaChaPoly
> depending on the appl version. To resolve this, within the rte_cryptodev layer, we propose to
> inspect the capabilities returned by PMD and strip ChaCha if it exists.
> In that case memory for the new trimmed capabilities array has to be malloced by the lib.
What happens if the capability is removed from the original capabilities input?
> All good, except how to handle a malloc fail is yet another API breakage as rte_cryptodev_get_info() returns void.
> We propose to return an empty capability list, i.e. a list with only the END element (which can be done without malloc)
> in this corner case of a corner case.
> Anyone see any issue with this?
How can we use the feature if it is not advertised in capabilities?
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-02-13 14:51 4% ` Kusztal, ArkadiuszX
@ 2020-03-16 12:57 8% ` Trahe, Fiona
2020-03-16 13:09 4% ` Thomas Monjalon
0 siblings, 1 reply; 200+ results
From: Trahe, Fiona @ 2020-03-16 12:57 UTC (permalink / raw)
To: Kusztal, ArkadiuszX, Thomas Monjalon
Cc: David Marchand, nhorman, bluca, ktraynor, Ray Kinsella, dev,
Akhil Goyal, Yigit, Ferruh, Ananyev, Konstantin, dev,
Anoob Joseph, Richardson, Bruce, Mcnamara, John, dodji,
Andrew Rybchenko, aconole, Trahe, Fiona
Hi,
> -----Original Message-----
> From: Kusztal, ArkadiuszX <arkadiuszx.kusztal@intel.com>
> Sent: Thursday, February 13, 2020 2:51 PM
> To: Trahe, Fiona <fiona.trahe@intel.com>; Thomas Monjalon <thomas@monjalon.net>
> Cc: David Marchand <david.marchand@redhat.com>; nhorman@tuxdriver.com; bluca@debian.org;
> ktraynor@redhat.com; Ray Kinsella <mdr@ashroe.eu>; dev@dpdk.org; Akhil Goyal
> <akhil.goyal@nxp.com>; Yigit, Ferruh <ferruh.yigit@intel.com>; Ananyev, Konstantin
> <konstantin.ananyev@intel.com>; dev@dpdk.org; Anoob Joseph <anoobj@marvell.com>;
> Richardson, Bruce <bruce.richardson@intel.com>; Mcnamara, John <john.mcnamara@intel.com>;
> dodji@seketeli.net; Andrew Rybchenko <arybchenko@solarflare.com>; aconole@redhat.com
> Subject: RE: [dpdk-dev] [PATCH v2 4/4] add ABI checks
>
> Hi,
>
> Two comments from me,
>
> > > > The patch we're working on will provide two versions of
> > > > rte_cryptodev_info_get(), both call the same PMD function from the
> > dev_ops info_get fn ptr.
> > > > The default version operates s as normal, the 19.11 version searches
> > > > through the list returned by the PMD, looking for sym.aead.algo =
> > > > ChaChaPoly, it needs to strip it from
> > > the list.
> > > > As PMDs just pass a ptr to their capabilities list ( it isn't a
> > > > linked list, but an array with an end marker =
> > > > RTE_CRYPTODEV_END_OF_CAPABILITIES_LIST) if the API layer detects
> > > > Chacha it must allocate some space and store a local copy of the trimmed
> > list. This must be stored only once per device.
> [Arek] The problem with this solution is that we need to allocate memory.
> So the question is how to handle unlikely case of malloc error when we operate inside void function
> rte_cryptodev_info_get?
> And even if we would pass somehow error condition to the caller then what to do is another question.
[Fiona] Quick recap: To avoid breaking ABI, we must return a set of capabilities with/without ChaChaPoly
depending on the appl version. To resolve this, within the rte_cryptodev layer, we propose to
inspect the capabilities returned by PMD and strip ChaCha if it exists.
In that case memory for the new trimmed capabilities array has to be malloced by the lib.
All good, except how to handle a malloc fail is yet another API breakage as rte_cryptodev_get_info() returns void.
We propose to return an empty capability list, i.e. a list with only the END element (which can be done without malloc)
in this corner case of a corner case.
Anyone see any issue with this?
>
> > >
> > > I don't understand what you have to store.
> > > Can't you just set the algo to 0 if it is ChaCha?
> > [Fiona] it returns a pointer to data in the PMD domain, which the API couldn't
> > and shouldn't overwrite, e.g.
> > static const struct rte_cryptodev_capabilities qat_gen3_sym_capabilities[]
> Should we print user some information
> >
> > >
> > > > This versioning will apply to any PMD which wants to take advantage
> > > > of the new API between now and
> > > 20.11.
> > > >
> > > > Note, I expect the ABI checker tools will still complain of ABI
> > > > breakage as the LIST_END value will still
> > > change.
> > >
> > > Right, you need to update the ignore list for the tool.
> > >
> > > > We are also reviewing all other cryptodev APIs in case there is any other
> > API which needs versioning.
> > > >
> > > > Anyone see any problem with this approach?
> > >
> > > The other issue is with all other functions accepting this enum as input.
> > > We should continue returning an error if getting Chacha as input with
> > > 19.11 version of these functions.
> > > But I would tend to consider this small ABI breakage can be ignored as
> > > it is in the error path.
> > [Fiona] The QAT PMD tests for and handles this error. I expect other PMDs
> > do too.
> [Arek] - Yes, it is error path but on the other hand we explicitly specify what value we will return when
> calling
> rte_cryptodev_sym_session_init so caller may expect EINVAL when wrong algorithm value selected
> (usually it probably will be ENOTSUP).
> In this case when setting 3 (LIST_END) on 19.11 app and linking against 20.02 (assuming with Chacha)
> shared build, caller would get success on return and fully set chacha session,
> which will probably result in undefined behavior.
> So shouldn't this function be versioned as well then?
[Fiona] I would agree with Tomas to ignore this small ABI break, as it is already an error case
if a appl is passing in a bad value for the algorithm. Even if it does return SUCCESS, instead of ENOTSUP,
what behaviour could the application be expecting with a session using LIST_END as an algo?
>
> Regards,
> Arek
>
>
^ permalink raw reply [relevance 8%]
* Re: [dpdk-dev] [PATCH dpdk-dev v3] mempool: sort the rte_mempool_ops by name
2020-03-16 7:43 0% ` Tonghao Zhang
@ 2020-03-16 7:55 0% ` Olivier Matz
0 siblings, 0 replies; 200+ results
From: Olivier Matz @ 2020-03-16 7:55 UTC (permalink / raw)
To: Tonghao Zhang
Cc: David Marchand, Andrew Rybchenko, Jerin Jacob, dpdk-dev,
Gage Eads, Artem V. Andreev, Jerin Jacob, Nithin Dabilpuram,
Vamsi Attunuru, Hemant Agrawal, Burakov, Anatoly,
Bruce Richardson
Hi Tonghao,
On Mon, Mar 16, 2020 at 03:43:40PM +0800, Tonghao Zhang wrote:
> On Mon, Mar 9, 2020 at 9:15 PM David Marchand <david.marchand@redhat.com> wrote:
> >
> > On Mon, Mar 9, 2020 at 9:56 AM Tonghao Zhang <xiangxia.m.yue@gmail.com> wrote:
> > > On Mon, Mar 9, 2020 at 4:27 PM Olivier Matz <olivier.matz@6wind.com> wrote:
> > > > The fact that the ops index changes during mempool driver lifetime is
> > > > indeed frightening, especially knowning that this is a dynamic
> > > > registration that could happen at any moment in the life of the
> > > > application. Also, breaking the ABI is not desirable.
> > > That solution is better.
> > >
> > > > Let me try to propose something else to solve your issue:
> > > >
> > > > 1/ At init, the primary process allocates a struct in shared memory
> > > > (named memzone):
> > > >
> > > > struct rte_mempool_shared_ops {
> > > > size_t num_mempool_ops;
> > > > struct {
> > > > char name[RTE_MEMPOOL_OPS_NAMESIZE];
> > > > } mempool_ops[RTE_MEMPOOL_MAX_OPS_IDX];
> > > > char *mempool_ops_name[RTE_MEMPOOL_MAX_OPS_IDX];
> > > > rte_spinlock_t mempool;
> > > > }
> > > >
> > > > 2/ When we register a mempool ops, we first get a name and id from the
> > > > shared struct: with the lock held, lookup for the registered name and
> > > > return its index, else get the last id and copy the name in the struct.
> > > >
> > > > 3/ Then do as before (in the per-process global table), except that we
> > > > reuse the registered id.
> > > >
> > > > We can remove the num_ops field from rte_mempool_ops_table.
> > > >
> > > > Thoughts?
> >
> > It seems better, just adding Anatoly and Bruce who know more about multiprocess.
> >
> > Tonghao, could you add a unit test to exhibit the issue as part of this work?
> >
> > Thanks.
> Hi Olivier
> any progress？will apply this patch or wait your patch?
Sorry if it wasn't clear, but my proposition was just an idea, I have no
time yet to work on it. Feel free to implement it by yourself, and I'll
help to review the patches.
Thanks,
Olivier
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH dpdk-dev v3] mempool: sort the rte_mempool_ops by name
2020-03-09 13:15 0% ` David Marchand
@ 2020-03-16 7:43 0% ` Tonghao Zhang
2020-03-16 7:55 0% ` Olivier Matz
0 siblings, 1 reply; 200+ results
From: Tonghao Zhang @ 2020-03-16 7:43 UTC (permalink / raw)
To: David Marchand
Cc: Olivier Matz, Andrew Rybchenko, Jerin Jacob, dpdk-dev, Gage Eads,
Artem V. Andreev, Jerin Jacob, Nithin Dabilpuram, Vamsi Attunuru,
Hemant Agrawal, Burakov, Anatoly, Bruce Richardson
On Mon, Mar 9, 2020 at 9:15 PM David Marchand <david.marchand@redhat.com> wrote:
>
> On Mon, Mar 9, 2020 at 9:56 AM Tonghao Zhang <xiangxia.m.yue@gmail.com> wrote:
> > On Mon, Mar 9, 2020 at 4:27 PM Olivier Matz <olivier.matz@6wind.com> wrote:
> > > The fact that the ops index changes during mempool driver lifetime is
> > > indeed frightening, especially knowning that this is a dynamic
> > > registration that could happen at any moment in the life of the
> > > application. Also, breaking the ABI is not desirable.
> > That solution is better.
> >
> > > Let me try to propose something else to solve your issue:
> > >
> > > 1/ At init, the primary process allocates a struct in shared memory
> > > (named memzone):
> > >
> > > struct rte_mempool_shared_ops {
> > > size_t num_mempool_ops;
> > > struct {
> > > char name[RTE_MEMPOOL_OPS_NAMESIZE];
> > > } mempool_ops[RTE_MEMPOOL_MAX_OPS_IDX];
> > > char *mempool_ops_name[RTE_MEMPOOL_MAX_OPS_IDX];
> > > rte_spinlock_t mempool;
> > > }
> > >
> > > 2/ When we register a mempool ops, we first get a name and id from the
> > > shared struct: with the lock held, lookup for the registered name and
> > > return its index, else get the last id and copy the name in the struct.
> > >
> > > 3/ Then do as before (in the per-process global table), except that we
> > > reuse the registered id.
> > >
> > > We can remove the num_ops field from rte_mempool_ops_table.
> > >
> > > Thoughts?
>
> It seems better, just adding Anatoly and Bruce who know more about multiprocess.
>
> Tonghao, could you add a unit test to exhibit the issue as part of this work?
>
> Thanks.
Hi Olivier
any progress？will apply this patch or wait your patch?
>
> --
> David Marchand
>
--
Thanks,
Tonghao
^ permalink raw reply [relevance 0%]
* [dpdk-dev] [PATCH v3 2/7] net/ice: add the DCF hardware initialization
@ 2020-03-16 5:52 1% ` Haiyue Wang
0 siblings, 0 replies; 200+ results
From: Haiyue Wang @ 2020-03-16 5:52 UTC (permalink / raw)
To: dev, xiaolong.ye, qi.z.zhang, qiming.yang, beilei.xing
Cc: wei.zhao1, Haiyue Wang
Introduce the DCF (Device Config Function) feature in the ice PMD, it
works as a standalone PMD which doesn't handle the packet Rx/Tx related
things. Its hardware entity is the VF.
Add the basic DCF hardware initialization, this is specified by devarg
'cap=dcf'.
Signed-off-by: Haiyue Wang <haiyue.wang@intel.com>
---
doc/guides/nics/ice.rst | 47 +++
doc/guides/nics/img/ice_dcf.png | Bin 0 -> 39168 bytes
doc/guides/rel_notes/release_20_05.rst | 5 +
drivers/common/Makefile | 1 +
drivers/net/ice/Makefile | 5 +
drivers/net/ice/ice_dcf.c | 474 +++++++++++++++++++++++++
drivers/net/ice/ice_dcf.h | 52 +++
drivers/net/ice/ice_dcf_ethdev.c | 317 +++++++++++++++++
drivers/net/ice/ice_dcf_ethdev.h | 24 ++
drivers/net/ice/meson.build | 7 +-
mk/rte.app.mk | 1 +
11 files changed, 931 insertions(+), 2 deletions(-)
create mode 100644 doc/guides/nics/img/ice_dcf.png
create mode 100644 drivers/net/ice/ice_dcf.c
create mode 100644 drivers/net/ice/ice_dcf.h
create mode 100644 drivers/net/ice/ice_dcf_ethdev.c
create mode 100644 drivers/net/ice/ice_dcf_ethdev.h
diff --git a/doc/guides/nics/ice.rst b/doc/guides/nics/ice.rst
index 8af32dabf..2639ae239 100644
--- a/doc/guides/nics/ice.rst
+++ b/doc/guides/nics/ice.rst
@@ -240,6 +240,53 @@ report a MDD event and drop the packets.
The APPs based on DPDK should avoid providing such packets.
+Device Config Function (DCF)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This section demonstrates ICE DCF PMD, which shares the core module with ICE
+PMD and iAVF PMD.
+
+A DCF (Device Config Function) PMD bounds to the device's trusted VF with ID 0,
+it can act as a sole controlling entity to exercise advance functionality (such
+as switch, ACL) for the rest VFs.
+
+The DCF PMD needs to advertise and acquire DCF capability which allows DCF to
+send AdminQ commands that it would like to execute over to the PF and receive
+responses for the same from PF.
+
+.. _figure_ice_dcf:
+
+.. figure:: img/ice_dcf.*
+
+ DCF Communication flow.
+
+#. Create the VFs::
+
+ echo 4 > /sys/bus/pci/devices/0000\:18\:00.0/sriov_numvfs
+
+#. Enable the VF0 trust on::
+
+ ip link set dev enp24s0f0 vf 0 trust on
+
+#. Bind the VF0, and run testpmd with 'cap=dcf' devarg::
+
+ testpmd -l 22-25 -n 4 -w 18:01.0,cap=dcf -- -i
+
+#. Monitor the VF2 interface network traffic::
+
+ tcpdump -e -nn -i enp24s1f2
+
+#. Create one flow to redirect the traffic to VF2 by DCF::
+
+ flow create 0 priority 0 ingress pattern eth / ipv4 src is 192.168.0.2 \
+ dst is 192.168.0.3 / end actions vf id 2 / end
+
+#. Send the packet, and it should be displayed on tcpdump::
+
+ sendp(Ether(src='3c:fd:fe:aa:bb:78', dst='00:00:00:01:02:03')/IP(src=' \
+ 192.168.0.2', dst="192.168.0.3")/TCP(flags='S')/Raw(load='XXXXXXXXXX'), \
+ iface="enp24s0f0", count=10)
+
Sample Application Notes
------------------------
diff --git a/doc/guides/nics/img/ice_dcf.png b/doc/guides/nics/img/ice_dcf.png
new file mode 100644
index 0000000000000000000000000000000000000000..540823c4a05e9ea9d558d233fea77016d1ae5df3
GIT binary patch
literal 39168
zcmZ6ybzD?k*FH`NDBT?jlF}e8B?Hn74BeolfOLbDz);d%Lw70NNOyNPNO%7Z_x(KY
z`+k3a@X<4KW}m&+Uh7)dy4F6yN(xdKsHCWHaBvvXpCpvw;1KwLKlqmjz$g7i%71`=
z;O&*A#NkSZ$ajH@7p7wJVsLO35oq^@NWk^0Z=W>n;oxu`p8w#NOsHJo;1Xh`CBz^u
zdIzZ}I(W`x-6a%La{^jclNOU&`_)rTS4-}zY76d1P4w1CVq)-OmrKC^5#ZY~-}pzu
zy$L?@i@zdO6jl2qp0>O_?^V0~==_TKu;0E)6AT_jK}Ys$GuU9xpl6NmUq4AAAS8rZ
z5<h>~BQL7h+1bg!^v{JkDk^GFZXt;WZK)wU@I4Lh>Nma-8Py=vX58uL5BwsNv0+TC
ztdd|ub|{f>I#qBR_r=Ecc2+(+F$m*+POs4gtgWq$;%5$Zz1W+)IolS4AR!?E!%MzF
zY^hc+(<6BKV$B>q*BptIQ5gaPfd*IJosgp=XT25!8zwm^L5M@!Q9uFabHF7#DiuCK
zt~qeo#^9j@Nyw!ThkWHBd%m>z=MtX)IqO&q%qUT-Q<|7KJNMhNs;a6pC&Kl7XSB)v
z3KtLW;^N}!aM9A*nl<f&LypeAH;UH%@~=<W?^;Mic5d!cle_a$qiaVX7S(%;iC{c>
zPEJmL@k|#dzb#-e#WW$n3TwM=5&|%zK#w5Rj~l++vL_;_xx2ZfV1g5o&-OTVud|dg
zY)p8a+CN(%s#Jb5B@9dFyIC8&!|9!kw<)+dnAUfkUON7RZe*>j5V@t@LFIjKzBWN}
zl6bSB?AXD@u-6mJ^$s$%>{fg;H**<Wy_z$R_0fFyxQl_NC8(r`t&yYbC$Wq!ZhWiR
zaQO>2)_CPI7Ur}ViP+8SHXAZ(qCX-8Pgw>@)`d8#Ub}%C2oieVUWG5$Z;wDPw0oh!
z(MS2V(Mt&eBO3Siyc1oDi4_JnN!?`_!F7~4=LJ<;7i4@c?GuTsp&oXOubJN~j*nRE
zr*(v=l`h&IUu)Y-G0<(qS?nZJwYZ$k+*tNcTf2SvjF$-h6dEDNR%>@OyFp@~wx8sm
zQ@KkHCa?M<DDa*q5JkVi|7}X6ooQlUmnxZR*#{T+GPIBvv|B=xJdNM>M<?%wx7c)l
zdn}q*;Wg|Gzyh^yRZNGE1Xt2+qw~^RE=E{lE|-=pha<-cjHj$U3eSXtS-y9ciA;Qt
zg#X2CIXogbtk+gV#pepYbkK~qZROc+2-nD3e6SS4d#n~;vgmMgi02}r<bB>}Fg(g-
zTDmx2YrO0kRzm_}D5S%J%DnQ0DdV>iV*NTt#-ch%|0xYIm~S%Q_BwvyfdQLqe%iUU
za<4uRAr<XM?#|y&3?~<aXTr-Hgaz#tT;9D;cK$2uv9~aYB8Zi$G+Wuxv-FN9+5PYR
zh4qI=trjcZ;=wL@A=W&9;v>?i^7Yn}O(rs$G1Q(|Z2z1sn<Yns)V;&#c$@TE2Q)Ru
z)6)`Bt!gQksJ*lfL=-F}Q=R&|&-+U>DkcK~7vFVi4u!dmmta<R;msWO$55HcDm$Mq
zYzuC++dJmp5`L=_vy^#UbXhfEraEBcG`ZP)uwPnpl~qe?xmmf+NiO2ax1Om+(c=}a
z!GBS0w-`sWU7FHRP;K|H)xhT98<o%yk#hLBt#8R=)!=YcP~pc5m_`)LD5eB~iYqs_
zj?tLy72jM<E{lrbr1en>*bVr_*W;KI;3E7^sdYFjoC{%b$NVtKVTT^7+2BYNw7pYj
zi4R5^w->W9pqU==f1#Eb;6C!>z+DXeb>TXZwZp;G`tJ%B5BYEu=0)FDN%x4fNM_s}
z&3d^+eysZRo^x@R*na2q(uHz@x%b$Uaf>KVBm(XH446M${l^6#mQ=@EaW_yr=YvCd
z+ccCShFQ?$1}?+o10_*^aev+*-pN^+qy-`EqoDYuWpSd~d>GvmJf>Q%JrCl>)WF6k
zmNOMMMp+hgneL-`A#pMMbJc=vS<;lS;}#5&f&14fUK<#DZv350Jd6LHkN$NrFqGJ8
zlLQ^XPt!kNF7<(R!K>C~vVL6sXh^GeVu@|f9?Xx@{W00`WXRZv(|^df><;rf5PNdx
zBpMO7vZw8Z5tyP#0>Vqx1X*<dv&0Kc;4ZffjQvQ21Vb2S)~HS*e}&q3vP0X5Ck>{p
zL|PgxQ2d3lia{mhxO*i0u!<pNN+=Dr*xkF}rL@3)H_B6SSLCd3U(-4g(^%Ggi&~71
zLC)Opd9psBf9m^tF-R2=i~Y+Wx~{DJ7&!b`+!vr6(LbBVOG$#4e;21&m~FS))*G-=
zPa}wqkl#vdk|f?wwVSv9(#)clt1G+|;%v%L28)#;+6VXk{7iUG+jNKVTH?KH*Mg`P
z4}U@{EuUAnkMX-Hs~^?DoM;li?~cv8zY`mcjZ7JLTz<$aU4RE4@Y}g{&JAw#$bdHS
zIsj~7y#89(M|b^nq}$-hTBu4^z2sUdUo4>v8BBVg%wyU%_MXf@bERkY-D?x<$;O<b
zpvBDRefW<DBArvMWHxmVUG9o}n*WowASDGwz0FLH(Dzku6^(`Vv9TH-Or_3WQek9&
zp{?wvKs0uy5PokudTJp-5ZP7duzj+zYp+MDA<51+GTg}}+XPcyOCY#AF=5lm{bEKD
zuYwz`>Y=}WkR(4(Y2H+%2SHKL8L4sdgQdlyUB8->Fe_=z_$V35@6;>_^Sx+A)!#{y
z%vNutx>hi*td+hI$B#YrfhqfD<l~3DlzhC=#`503zteBJi#)&=g+hzrVxg$DkD{I@
zg^onlA0{~L`RpH-+6gSVF_?>R2n@E#DgbDpEt*<Z@je@xlV4BeZ%k2#RC;@KHPXI=
zTv4h`bGcu3y~M9wU$HGWe-9h0vD0j^8a@f(O+_$Xo~<oWUu<$5exr+)S7gD4{3wVn
zgrav3%g?CuuSSpayqwxe{k^|Fjm)EGSivv!zjH;?(yL3=fBQ=%uWad!A{EMEQA^#0
z1YSLDC4#7>-9zS&zAcI<8lQ|mDf7d6im6Q*Pa;o%x1R@B!{cXusK5DSe-NiT0+q(3
zJAul<M~5YH9~^lh-eTUP{6+YH%@Tq2W2W5mXeE;-FoLq>=#Q(^6LkG!bA|cqcF*pi
zs1U^jfwNgMV;A~2jK#)8*61YsCi9QR)xxWa4oeYf>?DNCwZ`dlvWs4bPp=e=C;^0@
zKx^mvbur_;)>z-nUx7=5Qf6*XICBRQBO-{5^FAFbsff<KpkO!a8>m~+y7p>YE4&LA
zuaOFs0yDxhFwwq*qMEm7{5K*3xDA#58|>mq?5K=|*ax&{U5}%6AKyg@FV2-FFZMsS
zrHF6K$F#kK%8>ZN{<(`uWDrDi_jGHzJr(KJ|1yPYc)l)sxy8H2+vzKLzZc?{M!E7v
zF8j7?Yx*gy+=(y$qRI15nG2g2=Obg4BUlV=;xP}#z2<_JKj<^R-sjvH4CwW2sL>TH
zBFbS!1ZIVePcY_hP7Oo2*3+(G-We}hkFa4=pbOcJV<M%e|C+Y!zUqG{n|J=*>)qtn
zuvYEZ1ox-boipvgxq`t_VP5lxlooZlyR(J3H(<AZF?Y%7yd%9i@7`?6^1pYM<8}vA
z(fcoCgsMf!oK2VWC%GQAgevYXL7o?Eii4GU?~V&J?0;;|X}e4MzW2UeoiA;v_Kja^
zfO*44nT%$s<HJ_iHD0JS+i49q+(`zs=pvVW8i+;xuYtKK!0_&IS4RuYlSNh0LB0N0
z@7M}4b%hGL51O*{(vrSeZ6D8OHCSs;@XmhzeVhf(Slk`m&D&IY@;78t2t{dNPCng^
zl3t`DCq?X(3hcH>ORIJ6{H~eR;2X%NKo&enSZ7_?BtrJ5+S7I!1*N++VP8<HIBoPS
zs4s)hk6=@{5NU`^H+?`wcP3Bej!F{iUVeW-Vmz5k<8mL=To^D!#+Hw%%ierPJ88H_
zPv82kH0g1Ai1s0S9toBo^Lh+BR#Ex!G~`x30&(~+!!jfFHn1X}Pg@a3JmuSLTQ#Qb
zpQ{H2%o6CZ4m_;3gUII{#r0Kl7wZMxrF?pq^i$O~lx>HQg7%WjlKr9^Cy2f+@^BqY
zbx<#R&U+j$A5c+HBI@EZBVkOHH&Z}NE>JzIVq7w@Q)x#`<VnX!98`B2r8m%6Vo+<#
z6|brd_*__wX0{t1GqA?_MGCa@sMxH?;yyPPF1TB!b^Hhn>~1g2KzO8);$a8uAH*&3
zg)D9M>q1*@_xCVo#(yS{1NMP%E6V?!Zdx45c`#Zo`Hos^`dDktCFvy1sVN@{zyDdY
z>><Z7B&4vz?kkf60-^cx)}7#;WUQ;|Ust^A{Cc6ps{`J_Ho33e7$a~vXbL1#t`aT!
z2UJ5oWg&|^mMJOh+0J)(WRFNE*ofb`(PH0t*QCEMv<$k$g`L*rpQiFCT-S{2>uUSm
zw<=5E{Hpn^91&*!WTH|0=HF?*gcfcA(20pesM@XX<znYR-nMzrF*R^)xxTPuJ-mhU
z$llE1dA!{69o)n5LZtfsGN@04B+k+hUE_rgg5;a&za<srs{J-<3Qy6+aIcv(Vh0~)
zkKgAXo_BfQU6|#09=7DXY~^inHrqTNHErF7_5Idg-n+BOIH|92e|zJ_nvaQy)}E~B
z(h@~J+X?{0YlgxkKbSHfz6PX3+q1^PZ8&z^?eZ~;#lzEVFypvJwzmp*NjFhor7Ydu
z;qg_Ml8d`7>U`!J{-r<=sz9Ex<rB%wndw8n3T-!>HqoW|-MF)wvnft=dI;3KrdhMf
zMAysew~LW;pSGKW3p2yd|KC%+rqf%cE><%ubiE&Jx1%0MZ=Vj7xf*$_KBZqWak)pv
z4j1_4XAlThcpZNinfEz(+YMJ<jd{>q^T3Cm4}mOujeA_~ly2A-OCo@BUWkDw$?;>`
zNAbhK1bj9N)%iBl%U+H3#(W(f7`9(~6|64(y#)u!7m=9$ls^wV6HaD*K=}$*?`*(r
z%PAmMv7h_OPY`2waY<I!<1y{9lkvg*S2VsV;C^W>H9yql*ma~+oMt}#AG;;>^x|`8
z#R49-yL6HlpGSwi9Z64GB2a6w($eJZ>gJ#mxlbpK{v}O&29pecI1p8Qp<h1SHvO4N
z_+F)%xO0841x)wd$&8Vw&1HJ3w?MV^H)!Ha$X1O>IO*|zpZSM6>*QdWiv2AWGUOWT
zi-U3Fb<QX6QPsrlDFCm9oEyVYRmw8*Eij=nt7ZU<rce$)Tpe9zp*)({+wiPnEysrt
z%KsJKDNEN+O_ku)jJ2#;k}_W1r057%=`a*slt2m_4Ow4Dxwv;-?t_o4?>G2;iM1t1
zHTq@j=TP}0C)-I{SAz4qYL-K^qweE9?(nlNI~&!^GA8sit}QIH$3qI@ShGe{waqKL
z$gP6l!{HMFCb=eGSRp?HQ$W^k5NfTr!FhVBHe2dKhrT@y7@j_LUMO2X!U<qQcl9^w
zAMYGS1h^`W*Fliy$>29w+j$DuOrED%X?afH7nQAm*uGMru4R&AA=<D2a2dhUa`0r@
zkTJbBK6OSuwLJ5|P9q(YMcV`M$U<Wo(*mXYZ8CH?GyY$Y{x{kJ@CmLnT3u;ei(W?I
zMzAO`+bv3<e%#6GvtLFB?dSqKV^E+x%tEbya&0sX)%=q7bl~KNG#gIFFumkEV<7hU
zq)iJq1xI5@jQ(S?U1mj9Smdun_!|Kx7{jS*z}2gF{sN8ft`X>R+<Q>}*9|j&!o)35
z{$bev7y}Xb>i>Ao|77I<=gZUAg;co|EjBB;+t~N>_f;{P9o0(>CsVsQIY)8(M+{Z0
zX0uI|msFKEtT~ttnsz7Q&HG#`N0?_r9`7GV%pEN$8bm$wuu-KbwUr2qDS>DJ*c!rf
zo^{1mo!8969ZL|;%-@E-i_s)2-OJ(P&$+jIm%hVOr64opW`*}ty|pQ4f6VyiOYbe#
zc=({mBPV?e10QS2MJS$rn8*$>vA(j}V^5F$v@VUb2^BbA=Q!jU&j8~69bkn$Rr&R-
zO%6w?WLzx_VTtLj2CV!KGIodUcx>y$1FafezYvGt?k~pa`aUwZ1UZA|ee$tkNv=Cc
z{JGZu?=XZZBj@Ofc4+btE(a9d-5DgAA*&B(**VS-x*~(_)GHAM5wR(${3Wbl#jKod
z%!O|u`G9Q+|2vP8d@VU)x3xNg57CaY!$@wqG0aZHEbB+A(>?I5v+>m<Uz;jVsLJ*J
zW~+@y6zO<$tl?|lNJMu4hC=%f7vE5WJWMT(IHLy3^({EaKP?YB7ke1l@pqa|SOJ6V
zP+$pl?BYl#$__WuF_p)}{q`1!uG%lf;Sa^0L}IxMpnos@Sn`)SbPo_}$YtIzMIKdC
zPs>vd6ZxC5y|K}HBV94S<<b$`HYa;Z!TZt^q6v86x58P{wkI2UoRXG>hZ~Pm)?ev(
zXVwAnw+Kn2g}4PvdewlLVqV+5V`7&VqCwG_bUp~XJRAD!ZJ?#2yhSQ*$+5}6kL+Ty
z7n`c?qP19mF#uIzF`@8+FFFm+?x5Zv7Kf||U6sF$ie@YBg?|s!O5Yd)@?QvRXPw9W
zzAFH=P#$oWOh*kT(7*Ed?XvkR4WNH}In6(}n`hUC2FjAX9nI&<5ll~Xlj0N(M~gM9
z7W;3xm?1qmFD1O@Yps*F!w=>vJEt4sE)}`;KqRl-pB?W8e==fRQHbNR&yXn9-CPzT
zX|2+4M8JL)^24fW-lP2u?B&<ST&l#FFZ)D-uFKJC<Kab1+ygzELfogXQQ&I#70tnM
zy54n{`>H0Fw|hk<)fQE<3!TG@ud#0no9o!?oNKC!@}4eEg2_~&7N3qT-Qsh9Iv>z{
zm)0<yr)u#7_;6@3bvjJz+ggqdL`H_^?hjYdPzWpB%A2F0&w3<JB_EHSqr-ppKXP7$
zrWt>;AauX_U`)J@pMV`1Q@7$z#}5b8W-NWwOd}B|VY?aWP5mhB2UD^`MzduiO()As
zi*rJP!fC{s1^J?Ot$QRN*3s2*LB2-Q@>P<~K2Ce@@ChWHB0qvX-#@OXT6O<z6!)t8
zuHHuvKO2MQPf~XiwS5gw^y}|f;*G`8qVqEEj|<{jf0-d9z;+?Hsd8<$bJw!TNAoog
z!lOHQ@wE`4mlgY=k-F55@ok?~VYgZ9ii?83DUY{@BQPW0Kt&6>3A?52TkfiF>Svqg
z?k5mH0g<X3N|2TzLCY-?>EQvDCi4-yJ&q_ZOB?^Ly!aEOJF0~0B(D@5kKF=wuhD|l
zU+C#Q8yn1*!P{yZ%n10gV=xIA)5(n8w~GulmX=Xr!2;1WoHvT$;rkC#$)4|!vFT+X
zw1r+D=B(0|$vmpR_ND*2G9vq;Sw=C};CZDWLZX{Kby1`C&o%_Y=2S+BH<R2XLYGSE
z_=^|+ssvIdqk*Vve*U?%e-cz*yJPIOZgw1c$H2N3J9pcuYz={bj%-@o*jDZLI~QwC
zBAzvlhUpVeBJ9G0MuPyI?ZP@V^Kz#H7sx;!p1xk4&&-S`|3LqHbLRJvK%QV=Qkb{S
zvFK<?FJ<lVt5;392rQqy5=su0>1Ji8iLbg#!*Kvdbu_1a%bzZ^ESUW(q;MjVZ0*f;
z!p?7bS9101myQ#T`rwFer2+tPS}~s>qQQ4@qljMnW0?dl0Z$d)m31QSYb~j2ibvAI
z#x<niUH!DZdzYp5SBs_*horD3f0JK!rz1mnfewB4Gb?v?I8aoYjvl{!QI?LHC9;+~
zoO4O`Ss;?Z^Aldl-U@c;ZGbzXt2gmFZFi(IS+89!*Lk<F@bP}#ZyFV(^jK1WN90`~
zt=Tthn9XCnm;+Qjz8G0Z>*7Ne;3SWus}*&Z1bGmhWqu<dAv#}%Gd(=ndllitwoRIo
z=Y;_o^88@Qse8?9I6Pwi)bibdeYC$Yfnv85*zhy4D|RdlSSDYL0)*rCoha8?w<DT&
z1A2C)NtZt0&W>)}@#IQP`74&+aOIqCHv=OPy;3=g?ZXcQU_r3<9`^sxN#N(Sl-g5+
zhm%cbigdiDjZN%d1~Qg0ODQFKeyoJR`kd#k|IXA-l{D|)&PM<Niw)F*#IwjCReL1}
zYT#Hf>PQn`j?{K~c<F;FKn7kjrJhC8ZhL_tqV0FFyv`9W-^|cIJ!3w5CB}a^^%s2A
z!kZ-9CkOI{{doQP-#T1pWlE_zQTCS9g~NcX{+SG4b?_OC`xuy3j+4X;3IK1etFU|_
zwU)#L2MPfH`45d!QU<#gu>th)=$7BR;X|@wqUv5<r@H;<-%;IVy_9xGyk!FH?40`i
zh&xk1X(J4^;^f7B!0d@?mW)-O{}o#QN$MY^K{-1A05(IRQ<dU{w)@%ijcKn;(QdbU
zS>2MQW2XvUa)ii)_u`-rnYHKB?wPOvE8o>{*St4#(@XuD6t}M<Kjr}^p&@SEy-a^s
zicMrbiSmDPIzRp-EIO&}#9Z*`{%WHma(;pwaXmQj(l?ztiZhz)?1y!o?2<o%oYP^B
z6=Bgzb0(P_rjGYJ4x0Ex{g{L_v_D*Sp*kJ(f4vqey-LLa(*Ttf96(dRO5oVI*kvHS
z)Yrni5#mxk52&L1ST~|zsG-JRTfTf6_;CA&S~zwWFHy*R*chhF3+&o6)p-`N@F9?l
zoSB%vtJ+66h^$SOipE>?*&Oc(2)=DpfDruZ?u)O3FB}0j^bIi0vt%bB26EIJ=b_`j
zOA1%H`Vt?X?rOD8woVkiMxan81My~V*uaT99~-NAswewgQT>020z-bjLUXyB$=(5F
z9G~s*y0#nZkD*LO`6f}2YVh8EaNq&r+Ak|aWUE>bp(T*mSAqFG%OgQ+P>a@EZ6~h^
z>o?rr#(V;ElS_BiX$tkX(wtspX6DCW{Fg{%=6j`jPqn|>OY@nhl4Zk*<q)K-4WtE2
zT=Xw+#p}(Eb7^W#JwAPHY`QOYdJj{syXN6!1lUnlx{1dr@tqdHzW4wjf0jn517Qdg
z=!?1^@thZ2vBvf)uiJyS<5q(c)qhhM#%1R}>Uml%abT1~`U|=XaVE#c%g7PeQvGo0
z)-=6kkfAb6jV(L}fX;C^-057|Q>o|d{FD#WUvE&jnVy&A0k!CR%(MCH2e^R#JaS_N
znggoLNCg_gLo2g1SejKFNA;^kwYJ$B9v)jOBm2@Ez1!lKMa;9uhM-!^XPe#-5Da_T
z&m`v{tLN!0^5385dIR!-r_<Mz`Kp^bCju;?82igI@{uap{k-ljoM?_CwNSE!ijVV0
zz3^eE&kIb4-5Snu?08)+a0#C*>;Y-^pa!G+!0pq(t;w3<q&Bj#lW>Mmesfu0*hRc!
z9+d~O{*%Db-?Y1~yh*E<m_qESjvJr`{%e1IS9+h$W$(57K`FKllA9RK+##=SwgngC
zEztt?2qCD<Mnjl+!{OwtG*ENSHW;<0Z9W(_c)JW{tnWY8)wyTyk2BfSs+*cg7X2EQ
z=d9!RQ!e9TV3MPe5(d$xXXIzg%MsRqLI|_E<WHe8nFI9>+r4Eij&q(H_&ItXP8kY0
zhglRO!te5?*<|T1*jqku^}AWsT}=UkmNZ?_-w6}<0=A6MGLhUG6ZHH2?K<P9jEQ}8
zDC&66za{7|^II6Mi%l4p#)<PZPsg3p@u<d2FBtkyiJhz@lgzafkLJ#UM3Wkm?;XTj
zmT!p+XUhU%EmKC<2#;9CaPjnKoFz;fD2UKSp1>Vu{yM{NH3}k_`}6Qyb0P^HP783l
zsV!U7SEgrT(W^G`bA^LWUP5e($^kB;9^Du?Kna&UdyvR;Wg6I|skPDw-YJr4w#ER(
znm0{dZBp;MSFJjxQVq*<YpI?-@nYFvH8Z-zqQ4`s2@8mGHbH9ELNAkL)kb@!ZvOzL
zbaE~m)6AOw^82--vSzn$W^!$Licpx+R^5ALo10FnjySnP4)afW9Ejf9@0+$`DLTa1
zH`e)E4e_21xXsPBr%mY(v^fmgTT&&@_u<6rf17?ToV8FwGlSDB1?s>OEw3~CeL_5B
zXD#8)VY7!#1e-|^Uc=Q>)#3wU7FH2@(uewe0EqooU%ubdPph|-Zb{6lZ$-_+c=Fu$
z6$Wz0#!NBZb&#$q9@yuSlAOHCLTk97!#j{y#Pj;7dm7`?qvAWeS>%51QL`21@+god
zr-pamJ8Ep7Y5(wTr=3<(`G1$X<Jg6Mdr)LDm=W=YXEqk0J%gk~N1(175Fo?^mDiec
zM!sZqrY0JrI6@L<(R4uPysw!?#(=sMP};O-=5t2qzi1Q!!exVOpqdSv9IMm3MDf9y
zuD5@Fv(3pxMvK>!09T}tA7WAoy;OJj-=}>aRYYTSwU-{%bR=a0zf6?!E=9~z(>@6&
zK8C=U>}FrAk$gC=0rbZ&^TqERW<P)XLdr+5L|<EF)G7sjxbCO{#0Q@#47%PgmKZ=3
z{#$yRkDr5LjjD42;*<2=GR{(ean|!S`bG5m{QY(6P(ZIo*^`Hp{Z$X3(?(%!(@IPh
zS(4dY1Z)srZKK^fSh_<tsh*Vy<*<C9z#oj^8y;D|LHPEHTZ7SL?TeV#u?h0|JxM+F
z&v$a-Cm}G-mJe0>S58vXkI_a<dWy~KlD|J@-8tEqdfE`m;leCV4yK19wqxDYYM##e
zpA+IgG$D#dULW#0qAVl*HY$G&UF|!Im(?`qu;GVQWCXSgr075=y$@$)BfPSPQP`6|
zmkxU2MMw5&0g6{+u6G1DibM`6IA9M_94K+Hl3+~%XeZ%rpotU5r(X{R0jrbdfAqT9
zpnHFA+gctLP7gX6g{}tQ-p1!$-ZxzoaYY-zy(yIbD43H}_9niN20+k%wYhBhaUy~a
zogDY>^k#-ahKEh#r<;;>HWpeIB#Hg91m44lI^)OV>+6|oWTMHiMVf~DxxLd%!zj=S
zpkh(X{r2$k&`JB)d4s8sKY94&!OMIc0Wwk9$@`W)7}h?)^1ln8<=Nkg*hiLE-ASh1
z#JxLb_z8neh`oAU$Z!1k%|-|OK|tqCp=r0pn?ez<JMjL%A9Ps)>6LkUH%{PifZBg>
zCU|i5;<{*hMp}gt1~raLsrM+_UiP#R8v9x7jZ=rGn%OV;+t0WesG8>QBy9_Y_t~Zl
zI^~ok8{%#h2=bz$9n<5{@-Uw~H_y|6Le!wL4de)@>mmY=4n`dYz0}{_)>{s=YFwOM
zEl#?tC;@{231|1ZA8T%#4%kgr<NrIAVV3z{mQQR4f;-v`U0kxg^_FJ7eQ^{t-SV8y
zXm3N_iycbP9)oj|ix20U7BkXB+=cW-_0%QaOj|zyF*)}*)4tJ#-W!cbMM=t4zCCQp
zgJtNsV6X8Jtht!{zR@NyDX4FCb%BJYp|$utVtWfdqZeU2EbI1(r&uwtRXrqtYFOzB
zmXBU42b3lAKw;WVR{Az13Sfx8QU76x+49QGBSXAKJ;z{sLe}vrhwve$A28WZ?T0jZ
zl40o%_w8W`%%vOGY}C#$guce=W?{is)$3f64ji=tEC9M5H4%Kq&iJkIu;J2SIqq_p
zET#ZpEg<6wtS&1kL~O^oePwi--?(*H8i>@5m)l!C{Y+^qt%3m}fK@pR4zx&~-%Tn4
z3K&qP%<@>;9FXw4t=Pm@W5{x9VwAO)YuI~E2Q3T!s;_x!IYW#neb^pe)?OB5PfB<}
z=Cd?cb3=Ixf<5(D=sJI!eC=YhU4t3sT0_3%9obKJoO19m`$zf=$Z#dp7+{b|Z2r6y
z59t+iJEcZ|3Uo|#`Ca5`Dm8`zLebx_L_eU{g%($mR%g(AUWX>81#~y-+@2DhAxvM6
zi6W)IZMj0jhJJwfS%`LLop4suSKKl+eB<7+(BjN;fZ|r}%^t`{Md5%XK38MZ+U45B
zJ9wx4WlMja@ZtmP6zl7Lujh~V_yii*S}FWl?=Y<?z$7t%iq8ao)mYO-TKv7)>JHXS
zw*82~hH+$dcE1D^C^9^n@|R9syt#gfU6_0MUUmp_YUkc}l70g#@{)V3^x;iyMfwnm
zYA3a_=MD~U15x7*SMra9r4;bx$!CXWCye}&Pl~tvtI~34_NX0=!BN6ix`(Ce5`EOg
z)b=P~s3cD-LZ6J7-vx!vMCHfi3s-ysvEc5wa^C*XQbliv&wm#^vna_)&kp-03=U|&
z2G75Ybua2_)XsEv-FxuMG7JUa1bo}yJFg74OUMK?s9fz0IW^%ct)`lUc>!S=v%>u+
zoEABdTY;uWk*%G0?uFLRoSTD0pxP2mCI1M=hH42X)y&61S)UJ(Cw}21G$v}NTbx#9
zAWe(#@gYCzR|HL~0gxM}f@k1Sxoc7WnuEFXD+aoYlZQ~Ft472{lV*0DXxFhkiB784
zj&g*o=d0(m1tOX>K3a&hh`bvh_95};V}22-4}OCI(8X7<5f}o0$w=mJ;a#j!Hv$Py
z_7_0g_Fu-G3Az{&ygJ&?9K>{cau|F1&QBwt{)yx}1EAXeTLI{RkxWjvn?h|I+p#2&
z<Xj1j$#p-G`2F7-7Oy@#HQ>u>pqhEp8MN@z@L(k~ZHWAb?ZPr(Py<vG%Rj#bSgrG3
z(usssK#OWmk;PvA?+tE$JN8_JmBy4s^Uiz%na6R@KrwAWV?aUsk75bJS4Gg~egJ*#
zSDa_`l~#l`k$j0c`QPsY@KkZ7vT<6i-TTsE48EmAeu}duN`zwnBLgy|<gd-bfC5FE
z{3M3~nsM_cog)H<z*Sx#&h+0(z$loBLw_SttNw(-vrb`+(~6Y+5`3PnF8*UFBq4_q
z<$DKVi-OB)H;g%mf&Rb7VFr(8X7%CSX2-g6$-JU;*(}_lTe2NK`NW8{pdurrEc>yl
zpAn6^FL**iwePkySaJh__2YenuX%leBxJK4tI^K2n?C0{PWFefIDAg{YtaHx8Cbgk
zdsu>nL=N`q@rz#zQ44E!C)~(q=KmZGlNtq1e?fp_VZgyJumhMN{?$z5>6l%n3#s1?
zcONTLrjo|MCINwc=QwdFg5)TOpogInYC>Nj1AH)$2X+Dz;1hO^?L^wXs&oCS_v$gt
z0G|`eTFC1&5aNX1G9muYlVY?Wo0|w)c7;t1HL+EJf!^Odlu=z|U;lZ)B1Q^<h3o~`
zToQEu8aX%m_jYv%btD5X+cG`h!a`eYYggyE2sbAeeSH5r9ISQ&{5QY@@|HBO7CDXV
zU4XN1TjLC6`7-@|FubU*x=o+JI2r<Vfs&ns_yqAMH?O%)%+Mvnvvy#W!G?&M0bl<8
zK|y}2sJZ5>wU-aRY~+ml^&eVVBHKFg1-zihJBIpv_JqBxY#0+G3kGV$kFDS}dQ@vK
zF-`tD4=;WvHDcFrSO}_D%dtATihK+NQo&u{)PMz&O*cNVRWvYg`Rpa;<06eN`3T#A
z$qc=d-i}#wgd`NgLSaVss6q(h%;bv<T7~#hipwiK2?OLNB<XB=lzx#_5GE#to4{h_
zPJ=w#$|Pq61S!!1nompJ2c=`dGOzI^|Ip3UgNHM(Q}VC>kgMr|IpWoowe<AD>D3H2
zcw(W@!Ka^kFyj0jT0Q%I=pd2O-pi=F6Lhj1JXJTR@DN2&`MHAHKQU39LKv^I;DF6$
znN+CVjZD>G$ezHFAOWeWbzNZvm)l*!VQDtFXz;5$bArquU!@tBiV*&}zkrn1^2H{H
z3$+Y|SPU3KGCPnFv;!=`e`8`IR4#pm9h&mi_VSW#ZS4ab2P%Ldu~y2q1_~rOL9f4a
zLZP}!Jx1I1s3F&zo8p=5Om9XD_qZllQWrFD$lp*VwfA`9>q1^C6UexQ%5e6Y?<%Bu
z;fqL%n`XzKoETaYyGDqIFg1%353Q}kGi56)L8$D`W`xBi4<69M!aMFB4L3wkS(uJO
z$LJsHtP6yIDZ=4TqWo0)1R)10M7dm}0FD|gWz2p`8bBE9>ai;MvEJ)#pt2M;9Oy!f
z0k@NmPFu~hus$|gFuJqrT|T=VWdPyi{?EqLT*h46fEWQ^IEF&#+q^G)E)Zw7oQ+>{
zUi_4~?LCh8S`aor&A<}ksb9}_fx}k?CAj|GQ&h$Z3GYUkG~N&brlmBYB9dz=K$#`Z
z1P@Y;q-JGfrc0xym-g^&pFz6z{~D?zl?j^U{<Iz(#g`)=x*;ySIsX+np-F)AGT>Ej
z0Y1O9(ZS!y&H;jmE`tqD2Bt*3uAUFhSLS#h*P96by>%_y<g==G#9Yj;(9=^z9(dwV
z9Z)7{GZjplq#dVIJTnc>%+G!Sl}XXznHOY;0Uq9Nw1#%VP5{Kl2yD(K@pgoso>~3U
zJ#h~aDkGDT{|Ek6fVz7^A1|K}TRIJ8SX+-iu-1^@>ysXKM*N#G^+rChC|m|6+rIvO
zJQd>SIUXHGN5=lot!Ka2C~rB}(eUpKAq-56J}_LVx41c`n=`d8%J&jRt2=QqQIfN$
z2JEMJz?MM0VAc{|sOlBzC;yCF_!QMYbajrEdo3MxoZxT7swhMX#0bN$e7QAL@dT*n
zcT`cZ^eg;HZ$_pqHShPaHsAO{N<u|QLd7MXZTa6X*W>APbHN*LgQMg1-PEdGxRGg0
z!8%}2MSq8;hg)R3$UG5<mR1S!1bU16g~Ptcm9r@)OB>;XDg!>0sMTPD5&DiPdIllP
zK&^aDWa-~~)8DeCukYcBo<zT0<#;8eSMWTlA~5RWS>NoJq_vp#oCyM;brmBD_Kc!t
z<>jugH&wZL5x}AO^%;!PV70LNos_y^p~=|HW9MwpKkJ}pTtACeZvK|Q^!|g&^IG#5
zO7p|#b9L?eQ3qK6cR<|8Pk#oCwIO%dof@PRRIs4P4&cV6AKNVvTN0a$msX75wSxEV
zn@rUy%@@2fJ=cQ?J>(M|FGl;Jnu@<!7ra^d(e`T6T7G~S99PzrNo&$unjbar=-5Oc
zfsy67s5NO#Kc-`K%H5P8r^E_knSOa}or+QZ-GAawZ`MYMLl-1$L6e{JWAjTzMjcL1
z{aW1cb_#gYo1;>YDDAGW2ww&;?j{a9pwpVx-5}e_%HXMtr8VDVV=lO!%uUi~gbrhc
zg(8`23e=y-$QS}exmcZq#$OM2XR5x+$?veJ?r*NNZz4GWcUq&sj|&MbU_T|oj~y^)
z0PiNvVFwaJp>>q-Z=$L)BKu6X>||wrg)!6af3HuIR$~oO(KtU%Wn^U1h^Q|e%|Y3F
z%Dh}Vvo0omUdgq4_H9vtPUC~h7}uXH4&u6C@uOQbQ^tEQk&H}<FZ+A3@x#4r3@_HA
zqG(OwS2<2-bH0=|`#4WHzTKabBs_NJDL#PB*(*aB%}Ep;(ccD60zE+?Ne2AN8FB#k
zDx5BZ6h7fT$0r>sQ{hjYN1Z`FcQ=ll(je}j{KJC%7gAlEXETwuzuro^QRcM{w8f^N
ziL%8e$ET;op7o8SFbhw45s^};4{W+#t2-Dl6y~N=vrd=J1v31Q=sLdHd^WuYAZTvJ
zbtRt)b<w>7(UvC9@SAQ|wj1MdV+^D;-pls)0WsZM*BUEDYAekM(n<5l`Jnm2y-nGm
ztbx?V02efQuA;0(icKgT8U1!iER)%Thd^=&AXIL6U3vww_VTLNMV>y$II*UpFZ0TK
zX8WH`5gk3b=G$Rwp(zU}TJ;-nt%JOPHcm{+W|nPmdW*-Rt$qYsgZ7IC!)NE$HsT7<
zeBg;xz$T`=nEC@HR{5v|u_N_z<!7k_0zj2KSEi!_LVN@Dh4yimuEcoO>&Lb0OCU|?
zLz0f<qLs(15}0Yo=o28*TnY;P3|prezp0V2g}fdx_xPDF1<?c(JTfiuV}JhnT+vGz
zf=>{VdbCDiV6iO@w@%eGM&C7ojUSr=?8sK-8KYg)pQ*`rOyP9Tz$FDJ$C0j^3cWfY
z@6Xg?51`Gz6mTWVE^_yH2hayc=ZW89TpnJoGH!Gn-`~C#DlAVxL-4yG*5YSBo8jOV
zsH0rCm_A3OqPRvWeS8BQ5mTvtNgVn$D+6XBGob?EO>csR$}+PNeUDps@D&vYye86?
z+ehi;rQ5@jxScJRJCRwY5nlAgpp~wt-zggEqK98k9RpL&%a+CkL57L;$U+9(vQ{Mp
zvB+l}K>;*!*PQr~5dRpMud*Vzsj7U~q_Wb^?qk+EVms`mFOBg|sKX33=+CIA$i|vQ
zQa#5jY1f;Spc#J`Hw!zhAGtn^{6wG}e4ph@7dL2*O6--g-Q{U_GLRyfpbUi@Mx$OK
z69&2MA}f9wX)si15G&J`fZMgMcOeL~;a*91cp^7Do13j=<??Jgs?_cT^SdNeMI$D;
z`x4_7{XQT#Ga{r@Uf0yv6`syQToO=eY`_xWfQ`6|CX%h(>nnNs{D}O+7LVkp$i(RW
zsrSD#Lm}fH$pfF*h+fJaq_7n<z23KxD~=lH3=lU|uucoeBGW`sg7o3;{BD!P7l~{Q
zY-M|^6XcxN*&~my05NaGB77BqmnPwE8lz;w$nYC?a#D&-{y=xEs?yW1qfBIqyTU#^
zFVF2Z(8Eh@>7tvBg<NNxv&s`t;Ew3s4^CvI6h$OWc)o@NolQuKLXHNk08jI4&tlB0
zo(4i#2jy|h_6n<XOJR@FORTdI_Zgw5z<l5wJJ9gGz54E0QY)hHOAA^i5Il%>8Xnw9
zH&&u6#Nb8kQ=J+aN&D<y0?-*Sf1X`B%0|})Zvfc?&<Oz2T_LNcT%6V9m9noU_~hA6
zY~Vk}5P9@}@`7qSr|@!HN8lPgE}9hYQLEC467X0ta4&Gq2uK^MM3J91R;j*fUetM)
z3j5IJP<i_khZbiAb>~lb*qH^$ucqO@0G{$D54LvVa(8(^_QCB}YAQv1e}IgPOkag@
z_YdzR>1aCLdIv^kW;q6YTr@9AdU|@|I+v5R)>cv5-6v@onYk*{;nj{nLyWaC(_op5
z*?KkC5L6Vb<<PIWE@m`zyu4SPAw>OgELGLjgz@Y$RGql0q(>h=Q?_)+u8imie~DZ3
z1t@?WQ{md$8gL@Z^X{O=&CTt2D+8B^sOEET)KIFRU+Z6yuCcMxllAWM3gq;*&*tWg
z70k3he*6HA=~#~E4t9l+1*=8|L!n6=#$CQhXpN1He{zPSB2z!k-d%O|d>ziD;o2T*
zag!=pvnceJI2n<YGcnn!v)^{RIxNY`qG2_R5r)z-ndOm3*EN*}_-E#C0PVO~KnwAH
zxR^fM9uDR>M(Z`I$jI55saT&VP)Qu<6C@Ld3?}m)Ej2YHCMJGb>&r_Dg7&Z1x!lP@
zWmJa0^+Ht_-R@mCCirFPosV9Vkmrhn4TU3&f}whECtjH<x)S0gPD?1<0Ifx3aUT%q
zEgPH7Shlpt{V`%sZgFu`2k3R7T8Y-h{&bo5)8k6j-@jk&L=5F6ko=kuKC;l3=0@QY
z$O*XJp4@nx>~x_*QO{iQWg)$t)cyXjam-!$_2%6d7eujje)F=PCoFeBFrj&)3cB11
zyOnqY4o#FUy3*J2#Cnb7F!*)h!Ec#ZUC3zfWZ!v%0s-EX&6~nlwinHR9~oNMH=zSz
zcV2GxWFMQEA&$dUr$vC>ZpQ1!vFUql4J015JUt#RHV9EhB&ZhT13YegeB97Vy;R2s
z#H!!y@i3)pg=nmzO+PXguwS#qqBWMM-@;=xA5JRhb(8da-uXy&xThXi;cS&@(cWHA
zt0JKqT1RDN<x9=DxVYP!n<kGNT)oZn-SOika&mIfoIftq$jHdNHcQ!~X2Qb4$4u-4
zy}i5kV6nWH%R|2{r_V1gioD4Za!CUOB?&qRU9wM<>!|yfJ98dkP=!wj>TN{4Y0Z@m
zrx{$FjehN|8a4W6+nYqkGBV2v8oFK?3g}+9UD#bj_O2^$6Z|!i0ze9Sk3&n3<@g`o
zf8Yo>8EVITD-~Ks#!$$l@(xVF_YLS5&sAG#o0;td;n3Wk?;`4Be;C<@`KB>bQ?CK8
z0=WKsgY!}5$H0}RGa}}hUsl~J@)X}DXZxp32NI`?oR6lg=FrLbEmp?}$f^sGr6h)v
zMOuLNPk4AZE$y1S>I-D#cJ23ETr|JoACrYW1sW4$V(_sE?lT-69c#P>+Wb(Q0Vi}l
z+hWnK)@kwjz|Wtk@uJdVvhea?zC#3S2ldCna?A46R2((a3=b_WZJS%Y^ARRze{b)b
zxK0LDz-i>8e&9~h{w8uWsjdh<AK9s;r#cFg2m(?V86}?g-HoMDxnH0St6)ocegeL#
zmI;rC?c-9_zT&S?tCU37zxlO)=7bp|5uRPeVtP4-QAMlz>+b3*c=rb1KVKBFTwzDo
zU?Y-7%HShb;k6C~9T`y|jq-z=s9<bzGMW`?NT)AduKv@b;1$)_d!8qc>GYiF<Yb$S
zuKxHE?HVA;f4<m3jkB1qKG^7uPEAeCL`aS}K0XG3VK{|9nlECstIK3(Br}AFt-8AU
zN4g;XM{)6R3PB#1V}ox5b!2!F7wZusLy275K)4$4iC09yeBb*a7>a4!q<wJb<UAhj
zkRc72zu5jcV8w}b{n`v0tSRVjv6uWqqO)scXM-zW6jJAUgH!R!^@f~+M%Jv5vU$u;
zS_$lAO$ma$^;sP)ZuGFzH(CS`H!&#)RSxlpI4MCPRlujHkU!BEn?QqJP>>8bf;raN
z-j1N>OI>(+YNtWwVsyArcdYW7FtD}Uu;a&IV#{%2+|V|x^4quHwkO%6xPnf9YmGXg
zfBrzD;Z~pCsh578EF>sC0QOtSC*Cg&E9UHcSLG*mM7JNjZi5rjIQ9}XE<XN?GECXd
zt}a{xaY%Orm7=^nEfcmfq)e|#Mpm}L^Nz<u1sL}`N6Z1fq$^AV1~rOjbN?99%ci&v
zHf3bRpawRb&&)51gyg7%qE{e@DX_EJ@xtiy7Zi}h^0dqP#sQV0qVy*j!_S{pX_+jE
z6nAPE6m^P$&H<B3Y`)ji!xiw_UV}IKCy=hL=k8d}#~<jn_Vz{Ra;PsEnXXA8FCOG;
ztQR2=Nb3t692{ULY@P#O<mvuwYcM%D{xu>Shxy()%CNo{D^qLVtluCb@p}tMy09k#
zB4S%mZf>qho&v=OH*N=rp!1<WGMYe%Y&@&jI3gxuXZzFL#Xhh3*js)5<p>)Ua`wRj
zmAsYtiK!_%CjWfy_wNn4Bp|Q-^B=F*v98S*8eQ?4#YQMychIs@z^}m^E(F7}_cwB)
zMsJ)T>2R<<HijNHhR~p#58h{wYY*EJsra#BMe33~<2_v*tc<ho*eAGNXOn{4rV293
zUxH{$fI~({&3Bm97l@Bk-#QYxOvS~;3ui@Tq0k<{&E|fYBc>+i<>h7Px6+o61`;7D
z2I;Rn?atR6LEi(XT=L<7EOCvvbAN;Onja4j@A?g;pwn02ElB6Xh1qH=vPOx|pWlW_
zBIfxf23^4k-`yRxsH&>2-cd72XfRI!HqVsu?+O9VD^^xjMGMvZ#8wR<WLe3BU$O$i
z=I-jKx5JotGZY~h^*);@QDjZ*?cr?}9IVp$3aB|g4fkP&gwtTLMZ)UuMN{LV;`Ehb
z;u5p0KEIkzw~*NUdw>OV`(nnb`R_JI>w3Q*002();%M1>rrfZcX~+E&m?LpuzQ*Qb
z^68DNXvYihI&7_59S9*+y8Pza8t~SpI(A+}1Xi&<fQAN2-zGD}{D7BI;XxW)1GH+y
z<m3wgLWYs?cK!K-O+Z~&S9f)^i~<(3k%GJgM@o$b{cLzN7#z{O+?%vKUTIrhT^-?s
zN*Tnkp}sA2314x2ZF!UIzp<8U-$yl~M6aUbevX?G6C1rxM(%Y1?&Nm|+$<N>M3dir
z!KF|&HXkToogRWsggy{#342>g;Qy24KB+=}qF_-v?2dhzt^D$EdqD$2TCR(WQwzkT
z0DzR<RhK<m=k+mGSg{P`dsv96CJEJ4wb>{PNY5XOoY%WBSmV)BM}Rby8Si$nN6<)Z
zYlP(AW`Iu4_YJ@%YinzXfKSTGV?Kx|qAs_$w-PyS;N6{_V;3PZGP0fb^onWdMLCW3
zp{Ei)5=dUBGHk9CNy96d`NBTf1Rl0hy4{1k3@sTWf((D#AGV`lQRG-;RMKET2+?uN
z@Vd!F{$(_JLSR}tUw5<MrgoBYC*TJw<QyX;2<c4XHXplM_Fisu1?o<0AU1GlB<~-t
zmZMBrv0p`m(Wl7+X&dhPlh0~K6$$6(p1-qICB?-Cj4tyHP6vBSO?P{fMVWy#&R|~!
zC%_k*J@4`q)7!C98N?f#n!3X&(9-;-p!ptP2>UCM(*Ah1r{fN+5tvU=5C`#>5WwL+
zt$Ugo#UTLob1Nc?Zh8j0nsE2fV#5!ZFNQ2Xj?e4lUf<IFEiRVc6Gu3Z>X&R5ChQ*-
z6(`@}5dm&T$5`0dDCFpPvFvq+K}thJM0CCV1J=^=1SE!~<fK_|51!b}fP?DAJAwP#
z`GGn#y%sJyQ0&hE7gGbc)9*D+-cMenC2L^S<1itQ>*M{h765d)i1*h+g7C=51i$p*
zn-y|%bBoFzX9ZrOB?-BxAE!y@OiDnGcm31HQuyrw*A1KoGg|$VC2vFaZt({Du{xRR
zzdg+Y6@-njyW?hL*NyuyQ?pCH?c?|f`*oK}NfDKeh^_Pi(sk&It_~dyB|8raAUBYh
zhoB-X)9++|48)9rA$$<6`eJNcWxZH$3wRGXANDW!GIxN4f6u@!YlU_|OfFd?-s`bw
z*KG}_c|R=u-d}ms@e+*mLyC5EcSJ$V^lWeP-5RrmQV_sjXLg1~o;<D&DNdLqu06aU
zkck}G_^sh|5n#>0hJ}-GZLPHVO-)T5Fa7bepkty2u-EhU9G=F)+`O$NrxprMV)&U)
zt#SXZRMR#k*Z%CS?`?<-rWi2k>n|apq37r4pE(o%uu(>IhlGb?zI^%Jikl<%htK=#
z>uc->4BFn_-sa|J1GL^$n+!tMcrq#~tg=lYOb-t!NlBB^()2Vm@bPiv`FeZi#AK(B
z57)P+4~1(V%3*iN<U$^(s38DWiZVqJIkxq6w6_~rjg$t<K4E!Sl8}%9qbw~g9gDLg
zFX4XD61p?aIPoQrY5w?9gB{>9?CeUFuD*|Ke0;=9T#fTxEf23Hx4`&dj#}!#jWrxB
zSkl;FG|<yg7FPh#2EzM*I>(Ds-eIb!LIHhFPZ;G#Fd)y7@<OQ#uMvFIURwHVz0FyA
zAqtKh;m<iEWSe)sci{7rCd}eIJ8T-!s*=7)yDgI7i3i%(_^K>XDRGB?FM4T27~baO
z#-5zKunqkRV}POdZ|XY^>O8RqE88Gs<bN`AT_9x93Z)e3WfKm_|DIF&!9&1gZG*OG
zI*HGfj%aF)|1(3O*4y~hBRqUbKvQXp<j|qia}0@;#|J^cuA5qKTm@EdV%+fHKp^0l
z6mH)6P*uD!y}V|%NmFE&b_p{Qa54}_Pulzwgc(q&Ydsn<bpYwle1QR)9~BaiS$98k
zpX)@2^c_YWiBFKKa@*JAahu}WKUM601Q*AYh5kqlJYNEkuScSoQx_P4qEr*;!wAU*
zL`nlQkSE3`c*V#R(c?m)QUtu4&Z<$ZygXNd83%O1e#-(<LDoOo-dirt>*xA3guy>w
z847wcz^RTLp=?767~m!NVsHFtB9;s|&_vE-F<@-f#AgEkMFRskCcrwve)B@zIf3ad
zQ+3cgFJE@#@dWToFsQsQ9i>uX)>LUv9uau{o5Jb?FG>Lmhi-*bqB1SZ!>wo?rFbVm
zy`5z3)X_6@5PDwu9a05<y3Uz@8@B?!IE2!?F^k8XD7A{=A{QPC^<TKyCMU0Zx=u_3
zUdo<#?)e}T-d588a0-U}4iRMt&CAo^s|N#`^5);j)c$rqNJ3YHcu*f*g_)mNlmc-w
zuoelE@}owNdA<dRx4X8VRt$3e6l;u472T&aIh?qpDvOR>;B9juw?jKK3!X2Z<$=iw
z05YbN^%Bs7h>D79V|)5+?9;|A4onVtG{rT^c0?V#_sQyHm?_LCKF?Dv9`7v(DVh?;
zwu+GTaIEC6bKb&n20jQ@=-BL%z{D+3T7KP<*Kf=oh8KXX`n9<=tj*HR`Nq=2DLGI0
zv_C9%I{hsO((3L$ZL5-taqGOJlIpxsP6`~kaTFCdoTSV>MMXEPI6(<A{|jtVP@a2_
z6;%d9LMH)c9Tmr;G`B;3jHX@tHcW<*$@^Z$`M4YhFK?LL?i;<-)C+vo5=1ftElxi`
z^9TB4oRv=vj4a6Ge8674xuDM!Eq~M<8_s)}PeJ9|Phd=sSMX+KdQBbXx;8dx?R}Y2
zXAo&D%BT4Rcv}5i7OwfJ$>@JY3pa~;(brxX_`pz+-r(8s5Sv<4cka=UeA2MI$2xC)
ztcTm5if12%&wj0yn1N#8PvQLD{W`(4!-~j}C6Dwq(y*xhD4L+{@)e;d;h(6~1r-11
zoJL^C@d^Yu<ajMG?yi4@`rg>U8XBWx_SOh0()gsYHszJ2(+Js)i|}!hw(d}t3bPW#
z@zo~=q1^fo$dXTHo$K|ExLlUxBj{p4#JO3`L)3{eCwwAB%BwDw?S-dfe!Ft(bfBkN
zQ;us6?+G0NNHrc1SfvZh|LI|@M#m+x0ICkR%!Ur+1j$#Ma+b+Wx>x?tH8|`eY1vxW
zS#(;bhxbH$Iey#n^#8}yR|ZrSHSJ1=bfZY;p^@%7bf<JTlF}t04bmmuB_g17NH<6c
zNOwth-^KgA-@W%YoW1v|nOV=w^Nhry_ofOW4?FmR2pQ_ZVH4eEjT<Z|z;tvu;4sxq
zUcGf;f#~M_26(516DjzLIDo$)lnlF<JhUf#$NRljI`*oyJ2#VmDNBX|>^gREQSoRe
zkIbP)iB+WETB|K%iy^#OzK~`m=v~LxaH9-ibWzxP^kUxF?-EW(p(OI<xrdTR)ySyB
zLN)vFoEuH1$%;`?&1#O>@QY6<pYp}`6ztnpKGGC;S){=J%{*PUl~iW>I`sY=4;D}J
zje74kT~V?6b9$u2(bhZV8GwA~Z!GoYkUQAzDf9eTepz@T{Aufy(2nlBk@$AzC)Uc0
zYalt@waD>V`p3GG`;zwXKr&W~4qCX#^3T|jl5>q-DN344S$`%5gjg65Kz_pJ2Vn0c
zL_rC&-txSn=2~sPEkX)rj2vFy|5Obiv!yGOr8CQK<MNruoZgc@x1_aGBS*XY%QWwK
zl>G7kaynkT%q`s`br;J^OeOnoCL$>%5~T9fJ_|-c$eYhU8QijGaLT<Yd(8^_F6=9j
zyR2ee2q-o$_h|t3<v*5n#RQ^YCY24fNPOoz%ze<tlc*-W*V{}#`u0h`UBRM}5*?~}
zm6Uu^(f(y{ZeHZS85kD`zNYY#gjB@8>S)%MwFDcip1djSLfY?OwLgQ^ws55fI5QY1
zKZWdb#s>U<vWzfRxNy=-D6B~jyB>TQ94K{s1L^?nj|>+j@BoB+eed#&Fn%f1&`6_A
z4<9z=*52i2qA%wU)mFm+j`lyuyvbL=dlS=FR4$C=6D}qUOGQOuwGVy?7MvM+#rh(e
z|6RCZ4Ue?Jo_P@(W{KywTLNNHV03<}KVSAa><MS_Qu;aXgdm49IkkHPgFN`;Xcc;M
zO_0&+?Rg6kR{h_fI0mH={v-BhfUemnJb?QHw=z|4NH4ZBqayeTW@=xp+gbi~M0e*n
zGd*Qf3klfMg`|!6(@>vYoq3JfmdFzJ3we9)qc|@Y8SiROT^nGtd<F-v+bCAasU!1c
zT7tyqNEUVj<GnZ3H7sCP@xy-58S2n%dZv||B=j*~$upa-4&Z_6wUQS7xVTK+n9|JF
z2lZoHNc1^ee$$8Dgp7=gX2G6;o`d4g)9P?~ph7@>hL@KY><OYgm87B3Dk+V+Z-((F
zoX^v0yL#fOMMTpt@xI`ym|zWM^JQc?3%1|2imsOB!J6y|E(-6k2&XuxJRl`1lIOc8
z*48)Fk0g6Ud}&19v=%2droNOBo(~=!D7r$7r8bf^CG-&dGOFP@`!exKr<(`aBBW?N
zti2;)l;KSg0d~L2k~+QWf2KBCS+35-y#!3vzz}ARVu)%dd?A6uTLsRTVO(Xiy*DM-
z_cyF&R2<1O4`8*WsBtzow&pZiug0G96yMg^@S{InRz#1nvJ$Qol(Q)cABnq+&8^{U
zpc5;${|mMpkY!r92+J?``z7eExqLp>IX)u5mNd3ac;Xp7{I8!^S5AsxN3I1pVM*6H
z_JcfW%NiUq6`Nz;Ue~V$bN1P%hF>D`%e6s<f~i9IYHV<>t7ksv`fo>9{r9w}H~7T^
zebkwoP*!@T<Yv1#_DF!Cw&F<0*Em<Y9Fd7;c49m!D=4m~u2-=X@0KW#L!Pd!G0}?i
z{<GK!_^A(n0d~wv18dzV2qz9oF^;tfgFB`muzCF=KYw#lDh~)uDN82n+{2WSB9;iU
zWW9+`IwX8`u=U#dXjOg0qMgjS*7*XZV7rholD<MbuDGw6t?qS}{5CSM`VNJ7;VgWX
z*)@u>mMQ9%Twd=4tBc5U!Xb?-aB4!bH=R4@u(5GO@^rCdnHnf5qomn+&JkXlNf+cY
zlT@KJ`la|)q;Z5JIBJ#@Zd5GFyxSJ{ZZnz4#UD8;1`=tU;S7O;lr>M4E^@s$RkPP`
zt#B+c{ub<E{6PxF0v&mR+F4;k$rtC0HYqWUHGf!q??u)b1#jZ=gG;!Lf-0=u?m}$7
z?F*cD7J29PS1UyF+YsgA@9r0u5pwz4>qe>JYR8=-ju-Qi8J|p_O_XdVW+1Ue26}=-
z?ci?C$otdW(pS-G!{Se~8dmLW6=NSPb~jen$Vd}4C3WD%zCaws&!z>_{dX;r_|<qs
zm8zNYUS^eKzJWxRQ^*byN^zX}QsN#6!BJnq|Mj{?qBB=Pq>{l!e7r}J@7Y{sfiHv+
zRpk==A?WX9TC8VW7?XkdZt!o<<XcXbY#lk`P;z87+?`BJ)kfI$QgNDvU1^YkC8*Zv
z@gE=fEZp%8W)-`PX;2^Drqi=H#hYC&`8cgTDSQl9x0|5SH!-bWb>c0sZEBygS)=7x
zk(CD&J6-9qEY=znZ<*`={bmvd{w^l&>**Or`u;^i(j}f89)1uxI?DaDh`P{WyvH{j
z!$T_c9RxD{otE^y>amA+vLt)@G%(N*a+zSUZC~X64Uc9#dlG}BU!Wx8)eGIWex_pU
z*;bk5a}Gf%1RHBr=T8hI!a?Mb3I`5_ur3qG#sHh&kEJW-ApsL*hLftM0V$*}Z^5&Q
ze{z5q!%Lm`UXirC&zjJUt<yu;XCg42D_S8}Mvp&%r&+U()czhhh@QnFkC35#hAS|I
zRqbz<MmVS>pg>lm`H6#nuBO`P2@93L8NjnfsMcIAQP~?rj6@ZSgOnwgQl<s%f8!%0
zsaZP-#Y6w3=Q4nj5xvg6o77Yjm@kf1C<ih~?dLVbtfpk%+iN%e=snF&qn+9(juL$T
z?_XHTJ?r{;o19h-jz30o6tymtm_!WWwxRXwRiqN-pQkDnlw!e`C47m3Brtd$=G5!c
z#RETv##!tzKqI9ZlGKC)4rHcUyO3pvCGxMUdBvMmdVVPB{5So%#dA;3lGUdoTbA-!
z!Mryyc0>_+nuV+2o~RV~wE8oi3@dl`>BVMP!HzkfgCqB$@npe%OXyQ-`7}GE5GG;}
zAKv^pPq}C+85d7;6Wk1drcn?Y6gor*-9R?g@vJclHX+u&|IIkV5FuU0NJn2jD5e?@
z*dGYQk*JTCGias)bjd1vGK*mM7`lxbmPmWZXzU5lfq#J@L7~qAQsy)e{_S8Jh4F+L
zPC?OWm4=cedmcHMW9$L=|MxVV2uKQz24;W8r6A%hWRR)Qfk+aMq)u)I*;suKP;ARx
zR5YmRAy6@`{8M?x=Dmd0dC57)v&oO}mPvEt4gR}DNll;*u$%<kOd{n}<C04|T?S(c
zUM`0`*Go;P{C}V084U{fB-Q_v6jH^!`2Qa){t62egf14BVYZ|tH6;MJCKHrw3num}
z6_mubn~9diwtc<1>-IAH)>1sR320UM<hZI{#VKv*LmrB~Lw>J9@3I^9&=QyF9}KJ+
zz#B_Cpj`Wvj#2J!LUVj2%P^NbHp4y}JPA2vsa4_I1<Cg<bMJ6CFR9Hi4Su}^q6&EC
zlo0f%mdc|<gB|*EL~<78hMqY8C5pH7xMMl@AmHDmPbt+xI(-y!THhyFX00hux11&e
z-|_o8Ke8WN-(~o(xoB+fHVrp?n~g=^*Gac7URe^sl`9l5(ecTpe?BEK|4<_G)2TSR
zm1xyI2)!NVS*i$@3vZOqoa|j@C;EP|vCQa!zTj?||791a<cGx}Uu(*HI}z(CZKHri
zvnAa$u7jd8$P=Y$JosKpzT57#hzP;~IEkyFsy7;ruh8avQ#lfHpPDjWSM3@6uF-z`
zA?)P%sh0J<cJ_>oz&;+k@dJLV528cDK6OL+0q=G=e`EHmiR=6t#TC*YG6Ie{vSn0m
zR6z_*3L6NF4-!JcNKj2a4(#bK-%t6Df^NSqKFz32!_1GOE%`J)ZLP0lqj)S=9S<0-
zj*xnvV<=o21UMZpF+Tlk6uh4*Xp}NxY&k_d5L~tyj^0}Ko%HB3N#l|q3veUXKGg)}
zP?6m{kfJ!f(_Z17A5G{^dSgA^LY;2~WoIZn2X?iSIwmm$)$5NhKIa0e_cD)D@QJn?
ziHVL-UtEL_DXtmdiA8x9EybS;+<xokN4YFZk6ykA*Vjb1@l^#?o<>O98B*)$3|PC*
z+tLPzo}`GlXEk~5#5X5=AQic5&iRXzue+9naeLGF#Acl#Di2xhL?EaR7=HK1X6lEI
zxBuBlK|l^${hdC394=ZY%$WjJD!<jQnTGu;ttn{@;>Jt_IzybqH(Oa*UORk0BqMg2
zV%ctaQGVXnp47$LdCC#!&2t;C%8raijY?FHd{s}p6khV4H2$>bXG%l?`Tg!{Gc`ia
zFK&Rnyg`uS@X8J=S19Cjq?#lEXOgMVRs9|ouMUu6_8P4dsYR|{pDGS%eJU-sY5G9Q
zQ*oC$T|s@D=Ul)1^WSf;4FtDi^%)y^<3qIfW128IoC*NF7+1-se9^byv)#+U64(~W
zv^3#1YKl+P9FnM1o<vIyDHJ<aWSw^$b?S9hD+e$mAIIDU&t<X0KB?cWBR_=CI8d`Q
z*1#KM)*EQK-N#XRG>sX&weD+l=D8?tcQLZhsi956Gy{M~z6lfoZP}ZZr=6bfZPi)+
zOQ1V&-Zvo+Q;kO3Y(3J$v#cy6Ma&c-9yLhybBY0Q{4|1{<{0d{1e)FoPZgj(j}})o
zrY#@ANL}*hMAdh<t0libhe#-u$D=vLtGuEY=BW{9{K{!(V~0^^iUr1Kk>y&P4W)<<
z4fxq<D2mC0MLf$2f-T!^&a@RTAh=hNu~5yBwMdAtvgaW8dM@;f@~frTZ*A}vrzOHs
z)5!f%;``b#!0?z;mPcSJ%wv*_OIuXQHGCvtGZ;DuQwsP60I4~1=SC#gjXA3~h_Nt%
z9n|5r{dx2B<*z1I^7jf{7Wu-p`k5D9$F7@YrAdjrzQ1f-!X7Yl7tD<=6b%PFWvAY}
zZMZ+n)Ym>2KLJNEFZwG=3<HFa-!aPMl%|9U1T4%TVEGbge{FKhrgEeCyhYQ)ZNI6$
z8-u<Uh{>*-jnBi;S}mXMDEbZgxi|HGO`;aR4IIgr+DI;%ogonH{Kg{*pJAh`{og3~
zT|shEMick9cPVEh`6evYTcNN}sG5XoE8EooE2ONk^dsP~zk3VQ61m3=H`xE>cUr#p
zVUEWv#~U)>53WnjZ0uXy%jI5aYHTj)c^&G>3Dt%Bij={@l4+KvdO1)qijz$P84f_l
z$${%KX8DSac-&f-5G4c43R4>nL;0?~-ZnzuYx`+O4p_**l-^}->%{_PvgC)CUvac<
z2o=4}Pt@z%Z%hB_{Y&GYC$~O+v;#KW&^)JRnAvKh(W)JdNZ4uH>zxYCR}$)({E@Mz
ze{Btg;x3zZRFWI=Q@D8evRwHtC`$0E2W@&3<!8F?jW<2QTm;FKhquiKqb>U#X#MmB
zQ}McDRXe4>m5qu|BsDjhcDD}Wfkci#%TrZEZw0j#Ox=}L?9S7EW_18o;^tEM(yNyN
zca0W91k1joK`4uhIb-bs#ve+I{CR6_kQ;GiXEFAb-{05oS5=PMlXe19|LsDdd!^pj
z%ttz+&m1*sz*!g<8Fgr&_Z_;<_&%LgE&s5eHu+tOZI#1%Wgh6MbxRN}c-1nwvn%pp
z=X_E5z5k(zegEn_qw$|U1kFwawwp<^0KYSd3KKdqO(x0{$OQ~owdB&rMPE9;#2y9~
z9p}dj9$ZXT!4nQXMpg))&uJT7Qam1>wnoGa<%_yn5V^XFs&=H`rv}=$oZJ)IXs8-2
zvfs};eu10ui0QA^q2rs`JNz2p{lM|Q`>2~beDJI%h+D*;r{<|@rSO@Z1Z1b=h3Y*8
zbC#Q?Jr)HXy4@F!y`r{?+MIw!81Cc2#<{0q+oaUUByNXO6rEe@|F|%>5nX&0g|e!1
zR{gD%hvgV*N=Z9$Ic?9{)FyKlx6yZ79s2nFnm4ng9Zx1Fo7s)>25-S%m|U$o)gahU
z2RnM3Mraj5MUXx8dN?dF<g(JX9G4GE&M}>93Gj%un|-eLxog<j-3)8*n~IlhZ=DA^
zo(C=SfHK4)Vwu=>?IzA7?D#0(FZ!&|w?E$ab11Pf!RMeHXZpJ1jTyRDTo-j<ojp*)
zIPkYM@pDZ1mC>Yp)RD(-!2d({?K*4mqIX~{q@PJQz^nY`)!_GXYbR9(7J5j69F)uH
zfmuDazXkR%hDXW#r2DPBlv10A`RUZdSpV8KeSWFwyUwflkR{KJc7N*4sDCyKCRbG1
zceUROZYepirkiwH0VB=@Q1wxcv=dzS$;*AEopXM8`5&ppK4Cb(@YU_RGat-UNjLXN
z_XZVFTTKHsH535EQkH*{XAvJW-y%+r*E2pOMo8S~S9&?tUW4+n$qxDbkzZoFmSKo#
ztVL?9AbWt#Zu8BpfN2_K<esASqtZHF0li>Mz9GEjsq(zV=%4T?<CJyMDOO52Rhz8r
z22ayiAQ{&tC`7SrL&aW?0f?>@p=~;U&#wLe*(oTG4l@HaV6Ya)>~L5OlP&GNG_IL_
zU%Ot=2zmQd(;y_j!I)tX5cx7P5?)eMt@kQQiTx}b-rm<turq-KcJ139@8jK^pUquW
zvzKmHF9W?WIqft%8)@(SZBJ7OZ5vQ+X+EQ1m%@g+{mO*lrClp3yPy=5K8Da6(ZS~k
zXi}MzLlR$4Dmn}kO*B8WI4}Bg65xQDgOJZ#E?)Ndwcp$_c2zvHw+3zj>+JT*+Qaui
z1jv1{$&wV<Ua7zQ31{{RxVxZy%5)zEI9>V#z*Gi=bRmo7#B$C_i-8_lpPa9}C9SKG
zq|rtmIv(S)BPM!B4{VCQE4KHC(sr!vZ}PmK`gZC1i}n<#fO=PfeQ(t4{2?DUpQnv&
ztIRNAh()UO_S$&aD~i{u7IR+(ey%P9?F@q@VWuuy=HW^?jrOO%w(BiRVSmsl$N5Ae
zY?`x)5M6>0GI$*8KDc0}v^Jz!=G8d3SD64~)XJ6)3O%1|!Bwr{RYNRObE=2FN3trd
z-D^wQLYd@=5)70WD5t4jtKv7m#|UV)2Xa~Kh;E#1>{&-Ye?nB<9uE56$lqoyowE)!
z52?EzbSuH`voZZS-(R{+<z5C1VpMrCY%MPDUsZNi7m>dWq|hW(pEU^bfAa#Vwz5bQ
z69G*R66=Kw2R@7lec*`@(<4wJ%CRx8XV10|qV9T$zkQwmCh|hd2XG0O{U+~wB2F@6
zrCV6afi7U>!N?+2gf@(d1~67d^qikd<8HAA@JhYdwObW`gfoJBe7RUkzFfV2^Ly3h
zJU@OxevZq<<T5v0{AC}zUlmXw62%jsUyoPI#dLQ8sM<tV%5-dGd{VUeYAB3GdtZd~
zM2siQqMYl`=@$G%3?{wuG+`}f<=TV1id7SHbi2|{JkVC6_>LsBJZl-ckS%v_M?DX!
zhl3e(^l|n1``FCy4?HUFqf&ipXhq82;fE0a%}$7#LVdm~yjM-{#q%mm>R!+C-;=|U
zHpek&=;OSIbmg`nP!p=b!#DPJ?VS~S)4^WC{iw&I3bRtK`VjE`&g^CE_Dcmv&CFtF
z>x*A_lhYd?dkK@}(u~`5&ci2_eVCV*-2Sd9nOWWo`zytIK_0kC(~I`CPpofZXO6#}
zvVO_6oUmF9T&5d6;5BvGcCKw6G2{}x;Or0e^hXL2A^iH}x8@t^meOeP#OG)*WF)}q
zESFQXz1-w;^he11iQAN;g?=D+k&uS44FCBCR-T`~8MW0C?X!{f`g4iuR5E<sP9xbq
zh;yvuP@advL4ZH++;zh&ap&g9(=fx@+EiDT0Yrn!F)?Cn?$o0g1wyMb7GM4;aCQmW
zNY;#DE$_J;c;F`4<F(yuNBV|qFeUdSZe7VAn>F|CN@S&+i^4)BGw)#B|2R#gm?izT
z)1k-_29I!VJg@4N`~9O_ySU4H>YOM!8hzenrma4ffJ@y*S50!q%Vtrm>EC8osXiMC
zW_$m+7G{?MR6<Z)K6Ejb^D|4d)p~@jTOUvBFbWyJrjI!j<k!(=eo>;2@8Uh6bAOpu
z)Sz5`?;Oh5<aX-}1X}`Z&K`iLzUL&A$FMe66%*jvI6la^!ys;Xy4@2c)iJ`=5+0+@
zpFUb_9?@-_;==t>F6P7Xc&wKizik+2R=(}3T8T#g#p&pwUZgJjy5CpPnlG}P_E|-Q
zqP*L8oGbNTvZHmIjVIw>e(`M$Q3gsb$;*6LP9?>ii9F3LDD=Ch+7-|sv3Z=<{+(Z>
zS1vPZYp%-dCU?YPVX|FaTWr^;W?6|!Puj3<e|Ds+F!MnNSD(XpzAWVpKc>jVC(qJ8
z!W(@J`NbOJkmaWZlSXaQG)$-_?jA$ABi)-Dn_iT)I)8X0WnQ%8KW~+A5|7<7bm-dC
zJ<~{D`10IX12wR4y(g|5r=9J&5?SM8O!e3fIqJR{nB53SHuK-}&yQ2F_&wbZ{%$Y|
zWC@z2@xfL_GuVw-_+8?gK!DOaa_=C70m%DJH$Gsx+q%-%zS7Z9&z}21##ysAM}kq7
zq+=(3GilW5H(4`%oQMe=df%e|%;k}o&{p}dADJAn4nt0qZMMw@>qD5^E3TL{N!963
z-jTLjdE1A15!+k`{c>c<`*r@+ZK?y^g?iSe;;&q%+__v{e1EKQpsf(?W5H@WtUFJ*
z_Bb@N-fgVE&^M;pSZSg-NMxW)+q_nYY%;;t`?2BH$#<s!>=H!`-LnQ4&K0wL7`la_
z9s`d6;ccnQBKJ0*J^hC{NrIu)`kFV45Uv&-qx5YpES%#T-13!8IidOu0SEe<^OTR%
zDrBmP4UYG&%M{a5wjcM(EF9+S#0Ps9{~YnUKxQ{%%c4Txw2R1#e4i*#hsRT`#8^}G
zw*FJ&*QZK26@f}o?%4LU3hs&iGh0?cOLmvQFCj)oiYrR?Fm5jUZQpL6GD=#kW%se=
z#_64~8KuDkt#!G8tN@I^Wk9P*kp0Be9N0Nuq;KuApga<ko-8GNbLiec{j)>+o=d?!
zZ=Da%t?)#$=-rgO##J;RYd`8KeJstbf#iBdxb{Z#7<MTOA!3U8Ot}rN5mznHq*CUE
zqY{w(IMc5fT_kwd{an!oSC^GtN!IY2Hr_bh2ZOyH=#9}&NouS{5Q{lSGu-9iB){=k
zDly-rv8a(z4gArT^#Q)^(Q<9|(sitb?dl{(PRD<Xi@nW3MpnbwKF@&&$)}9}>=&cQ
zy1?!Jif@a&ch)nM<tNE?6QOw*NL5Za{hP`|+jH|l8<WA{a6s2Qk@qBUfNV0=jfBsl
zO^1^&6$CS0wDF6URsRGX2%Iq}J~ltvic^K}b@*JZT>r3h%=G@&0DF)GguC8}H)+E^
zxT<2?H(L-%4bA$ErG1@(u?dGPKiv<BUMgJh`l@(mRexr!3ui{{nWvqG0}u?jdcUk(
z3m_9I*-wJjF5RkFR$^?R^SNvTL?(Dx3(1+mHL!?<7+?_hU5*C5OP=!S2r`j!G)W>i
z5f=L*cNh!)DI_3?g%U$vE?;%P#q?nRV7f)3DRTXLxRX(?t8<1XO)I%HC37MO!~O|K
z<*OkTQj~ufdpjSk#{HVsj7P=FRHqZ%*y{>`57gwlz@FaAO$Zbits~=gxh|8g2nI}g
z;2rWbJ6YjEJxI9T#~UQbHDI+A;b<vgyhv%>MC0?l12|OQvNxR$T^!ytNBzG5%q8o^
zha;OWqZ@$frf5x~syO#_-fEZDXjLd-yufU*?Uw3RfjvFw`Zo_{_mmbAcy#F$_I8~V
z+8#y!tNHgI7y#Y2#7hT*fo0x?qo$c*O}UO2ixe}5^jb%#ztYe`IJIm$dGwk727HRL
z%3GZ`8qR*xrDuCxpDVys#dEuBLvFc~Vay&_&Xf7?5k?CaJTvrP`vj9eQ<caqIimcJ
znF3Mj9zT3L+Z9Dl(yYe;`nK^asD?p`zF>g^?wGNzx3uE?(}&)U?dkBnZdXR2jsAKK
z3Gycfkx(;z>@#_oydK&Pz<Ra3E1i`Y<}4D{lDfML%O9Viy6k*iTs3ZkNsC{9)LXUw
z?bP4we0*PmWG^&lb<ifVUG>^e#}2q1=a)qw0x(f%SL{P9%?@IJ;30eR=@~T~s*i0g
zd-*<n0~G}n49M*tAH930<pSVETEHav)uVm+je-;C*&J+V+3?)qeUOirP1n`(@77n-
z2<BOub02cL#J;GjRnnv>&#}x*bJbWJ75URG<U5hw*ZpcwDmbcq!NaF%%wqrAbg+tB
zP3Sxb)7h&NZYx^T(dKgmIb`oeA&@zG9tG?#&6i)-#Gt`f1_#ZnDo>SkQkh<IyrJ;+
z@G;|QcKpHa1|QjTE3iv)hJE)X&h>KM9;PgnMtzC0-_31Wr6c>fe|PlBM=Aj>g_i^1
zjAplXSDG+A2^Y5?^KUcL!?O1ULf+j;8cjRH+oXF*@+0;t&s}{=10|Itm@1N*U`*#_
zJ$cewgg^f=wdCjS>jpsUyIN{N3`Af<Qjj1h1nlY$-|>Cp%Y{^c3|iYuy$1Q}O+Xb=
zv&goYGLqK}crEOe#}&g-(IKYlrv#L*tVsQx*DGs7L#jfHSbeMSzjE_kUB4abDf#ht
zgilfH<Kw;qsgG@$_vDMVM~hKnX1f5KcfL#PF~Gvee6E5+XM_~r35)Ej4}aWGTDIGW
zqTYx)^kzG1cuRFGMT!m&fEZ8^keKt%jfA~uCNe9j31kKsP|<;=PtF_nV#0bs@2nNS
z)Ki}rUu^FVB<sQ#{G?g7<j<(lQNFkRxa-Ll%9SF$n+;Gw!viIC#NUt(cS|cZCkHT_
z<ZfD}$|<s$Pfe3(*UIkEf%4e%%VM5m5o=M^8@(L<f|Gx(<C=m}ny%LN@dmISsm_ok
z-#RdMO$>13d5&>&OYKKkNCQp;`~1iv%R_(jk?ALTH7lrTx5!4D=a;@B`oF-vcvrT9
z7&w6xCUct}3$C<+pWPMKKzi)r8AYXs;D9I;<bfy%s-g2S>Pw%t)+e!xo`I<Uf!6cH
zB$M}i`}_73^4CpFaiE$KIJ5c4#V>LuSpknS8Q{G@3k?;W5@J@sNN6N{@>5t*A-^>4
zFHUNgkJ;f=!0#11%qg>4kvZ(W*BNa`*Edi`jU}CCxj*edDCK|O@tAMgbR<x9a~|DW
zOLFBWi2&6UqR_ebAsx<em~-};m5+!RfcRhj*_d~17jn6@{a_it!4&IsJ9wc7x(qy=
zW`&qP00Ghe%`NP)LhL5x!?NvmUK<huS#bz`xox;rltc5jy{~ites`~33CL)~ApOr&
zf?Wo~U1C4XLL)#%$w*&b9&YvWXVC2HrEyk1i&Y718ZlKaQVV|{`=I3^g8kg9Ug6t6
zVxdXiR9#o|(C3s(59$2j(!!$kLPIrSE?!Y!ldj)W5o)VDq2~e1Q-A=d`YI1fVVywk
z5B@wDGFmXK28KO|uq>~6cHXT~%LPE6B%Sd5xP!Xo=~MlNMQ1sKCHk{Dilk#)l*sZ|
z_iqPbIj*n&nLSI<iUYOT?<%{?1&fW$60Qd`>RXhkFv}ic)#KC2S8hjk70I4Qg~Q2=
zod>aeX$Hm(B!yF|8acSx>KKuowRGh`-B>co)1JLuXg$O6%hSY`50G4bnaQJuNPddB
zcoVl*FjE=Z$lP2(H>RlBc=Y*o71^QB<;2)6o&`Z<B<#7evhTaG`*$!vKtTr?CQ>e9
zh4;N+KF7}fmJyfTpMT5k^UWd|MthAR9EkB&@NM%y&mGZQjHmzQdW^kwx8=9pDEn|K
z)wCbzhWWv8XVIkbsAyPRRTegdBbGtGG}&p9i;gW(IXw!i+tQT_y%N+{0G+YZpBd88
zS;xIoVl|IY!0in0_jujqJF?OqVDqRMQc8ZOg%-}EPjTK6tceQOwfU?3&M|;@4I}^d
z@cZ^H62Fea9fR2hympi;-INjhUzK@=7f>2mQSZ4yo*!^qHcQcJgG`@3^{(kPAw5hr
zjcgepQBA(2w$ky2A2X<N`4d#61jC+i!3wBP(sB#+yux4P=_C6wncKaboX|+Ws^Ja#
z)r<K1RL$t<feN8G`IIBHD%H>d6#qXZZ-!aTwLy&|K1$YEhhVXFJ8>6RdYHDNrZTdS
z$J!4agQg27wCD({lnK84LSK^)S|=lYUkZAam)x$+hf6%ZUQ?>!;WPFq5_w{k!RcVk
zD?5@x+^3!<tEBbLe9o@@@92ZE`92-Gm=3zFJ7WH)c3N&T&>?-<e^7?6g!-Of^GcUb
zwo`B952AHB(?am$&dCf%atzShc|1u6?J_XAE_xNxYMgpr*;TO(0=*ijDur9`TMjSK
z(8g3u)_gj&660|kC~1(m400Dt-jGkI!NWO**&(evrmV&PBDi5Yo{IPj!<<y9d>!eg
z$N#u=vG3RxI5GqJ7bqFD9pI2C6YJmtPAC#{)xzxfU4~;j%?5QkAkk#Ed|;%Ek<fvU
zXS|;n3ax<p`*ep<mTUDrq}WW-2U{Swt^_N|JW?I$w^WOBfzZ?tX*<>%i@1X+EQiQ~
zMGZZ|UQ9vDnU_k0DfvxG0%5u#R;qeJ5^FA61qwZ+5$^8>934ye^tsKmVv$JGlh=AY
zq6N0(w|6C}Q<q6k+Bv$Y*L!hSALmVI$3v1%YJx^@zN;{<rjs!y=%l_6<G@O0{Egh7
zam}ea!pIX5+*f1{gisEh(8AL>?6Th}<`hGtjslMUz8@Q(6dp$#R0|h;k?{xsxBz_&
z2mkQDO4#=<w8ml1s1^+AqC<qcA%qIPIO}rTtF#S}m8!_OSI((fwUoa%IS`;UM!!!7
zFDmr~+#dlks7m1eZ?*E17-*dFcvjV3e<<`}^-q_y?rjsnC`-o)yUulkofo`h5#i{Z
zsf<y%qTc-%?kkJ?9BqjvBMee+R@2{E$89{Fw~J^QR~*8;@|Vqj0Wx?X#W15^lV_7V
zEG~0DIb;~NfwW4aUV{i;PW}BFZ0~J{(nbH<y`n$_YuWiJhu`g8ABIW9)lW@p8)a`i
z*9jVFd83ZL4A&}e{GDvzKIs{9v$nfP0BZuGyx-{SE$Ew~LkJ1EOw-q*(iX1_@ZwZM
z52m)Mze8X6#;37H_UO?J0FLAhtRVcN+B=9qAdT=4Jw?6!{RXi|j-5n2Nhm$4)xq0u
zyjEeWbEuJU^n+|QqEzIp<Z!PUya1l_f<(~Irt7lx(qA=+pYmyz8gG8Q!Nm=3@>(!Y
zS}sBxvZR-XgEK2`&3Tp1KFVi)yS{k8fryt3YrIP;`2I3T`>4`}=^huejP*JF`F>(q
z+TX?T^`~Eyb=Ik}5MvSp<ca)Ys5wHU98CBSgHD_KWQbnu{rX#mS}ZTbS7pZ}f%<PY
z!c9<qN}O8RbT_W8KSCtl#`#S_40xgwKk~>X(1GlD$4j9}MS;uq`YRx>;ngeB`cG4(
zjal)t()b6ZwF5ReKFchnGAekn^!pViVP{$dN)eDP51zGO0&QFzgvL8@C-IR9y$7V=
zZr{X`$Qgvb>&35}ly_0+R34{;w><WnJk|+61=5oy4ZcUahXqy&nZAj$o~u!&#w3~z
zy9TFbWCTSl62^KYnGq%A{^Hi|H^g_*L!&yvHlCNCGS!Moi9*P*8Y!7*JT7jT!;}n=
zl?IE4#WxaXI{tMLpml}A${Em7YYW<nNitAYGj-@RNAHfAElImr+f!JuP||HU^`r#(
zQbL^3sg%bjCICq$F+M(Ch8ic8&4r}C8@7OrfVo~$kh%F+A5GxEnIc2?Sd6*cPza_F
zN1sUKryMk2=5MAO0`w3sWfPNCz>LBq;f>4AW??be;1!RTdGRz5NWv@F0T(;`CRCO$
z#Mc?+Y!&W)AHD>?XEP}&rJw_!7W@{|{%m1kK~#g)umvlXE$vk|H_OvUof-!$+!h8F
z6X8(S>+8Nxo~Q*cKu99$DM<Rxc6&4{J3D&~N6JuQh}nc-pe=ZSVa>?hn50%=fQ1P+
zHA%qcrYKhMZO)x}R0%sOK3=jZ;DRhPx!HlfXR4~I;Fph}ZcW4@(H6IhKzWFzXp?ez
zLp-(+_8$rP@l2PFUN54Mt^&8RekZ*K*ap~nDnz8D{(Dnpes`A^=H}rQ9CFUG>B<_X
z7d)e&8=1}rHCK>(m53dVUg%7}!P!oPSeVv&&7<KeQk`{Aa-i2MywitZDbqXuQqz7j
z$-Pe(R5uh-`aT2=EWMhV+S=+WOn#HgW@H0()Un(ROUDm<E2E=UN0)<`#Ny;9Ke+m7
z!Vk1Ze)$fE86zw>vFd}o-J3^Tc>=wz-IFykHjQjK+N101H}-4h_$eLjQR=W^tfKM^
zpdVahQfH_5%a>8Wk6T+?|7y={_}jp+nJPVA%gN;1N|l&?13j)V8fJ7wmnHdrxOHt1
zt1p4uv!29p>3lPjIRQsh^W&$H?W%nHzqk(;M-D^ql=%TT$WU^}A4%d!&>~|Kj&y#0
zegHnV`QEs=yVu?8>;L>WT=9J2cSFWeX0%Y7Eix{%pUc89)PS@YkMTOB<75_3RWe%-
zye<31%`L**t^p<)gieu!*GO;K*}>}Yx$S9qdh+J5-nkAp6y0xH8P%t|o&J3PvEfrZ
z^xxWPeM<rHr!4}8lW_*6h+iCE{JJSO1Enn~<uA1j&GtZvggD^uGLe%T|M{weghBHC
z#}B|z#YQ%7G&zlxCH}FhmD)q7xIooSvnk)GlBBAlrB6E)Ea74Bmej{`BH3sEA68EL
zgWnp$-H%eHh>T-nJP=Mp(bE{f$7yQ1Z4M@bje2}}IXXNH9~fc9V>*$Z*?JWdpy%Xp
zfshpkzuD?1#b5HT$jJSxg7U{pWoFaw^2zrLb=oj|w-k-XV-(rF^!`8*mkcZm;9Pw(
zY}Hj%9GRZxqNSzf;^JatjOsD%Xt!~$#=|U+lFXm7zvNS#UnVr#X+JD(w4$z3@H+Zq
zMy6LiKN4j~pY5=4d@XMA2iJH1>zYg3i>g<4UqE0RKRY+~9&p3@<0xFWhWP*u1-xl$
zdKyqxD=RA#6B2&u9gi)1v@-nlW$=1amiX(1_Jx1CtM|XrFoGr35d9)yr^?+_zur*n
zbeiDLsJir4ufj&uC$v!8>@2aLs;vm9+BI?)7R}X1=<<UCOeOPVb7^$n=?eK>-k+z}
z>eav3tG1oqU{DX(pR(C3*Nn{(1QZp9b?y6cS)FnWC;1yg?15HIhkLe*>lWNEfzHu5
z{ykc1>3sI_rwcp+S?W?{@~;DrB@_r8Kj}8ODfp}E^n~h`t-OS$hgi^6s)5T6s?vH2
z@~f5gn0~l~0B2vy@*fG9pj40|*5@q3`zkvihDhKsCgi<fMH)wL9W{PR2K>0sx5mej
zZYm_h+1?s?2Dqfeur>uoT~+bGIJH#n2xw>H7w0=B;#DSL3#~*tP@=80IjRzyb<D}n
zS<jR#{pKS*i$X+jG^kl@-GG}o2Rc7HnG<RDqxe7jsCuI_TAixPYNcJoh%MfYILC}-
zd#-q8=#(h!H*^OVc}~F$qO`tC8pQ8lB`RXk#NiDZVo85mK=Cj;UYTZaO%5z=ap7{b
zR8eAp6A4p{(#K17!Rcc{R#0g6y;}WLHd!U&OOTu{O{{|3SsTf+(^Q}C=RFa)UHuhj
zP`@x<cGLfO3^E-%9M%@eQDKv)#*~kby7CfH8PACP<CFX-^vesNR6$EAq&?Jcx|Fcl
z$7+aDY^Ad0kA3wiGdcAlz`*J7*2i&>$zscZx{o1Uo-rwqIX6CN=KY3}Qt0of5X+Mg
zq)H-f{aX6kxMH9`G#4?(>}}>yQDZURwr;i1X-e-}(ML&bu~Ub{TugFM|9IIAH^#8z
z-J{yeh~hJ>mW{X}wUw;CHvv*+?O)aMD^y6=cwsB!T7KTjHjk)hsWl`uJ{P}-_~vdp
zD;l)8lkquU|AQW-Q4cQIB9F0n?_=L}a~G=lCg>!c48E8(J*~BF4ZoM#FU^f|QD9lJ
z%ijA^uK(IjUU!_YFM8i$ZnR8)Q;@bqFt;=gRV`{L=FOt!6{(>6s;wk{A_rj^1rVgD
zi+}<_3V{-**9V^dqV99h?O|rAfxJfF!LP+J0a|4SYR*ndi^>S%cnEm-l(d1GUvHM|
zAL_E|b8J;CDY&RBH-fn;cYa8C7w}PKQ~gFZt(+(#?!i;B<dWTiiU-qwW(-cHkCONp
z{uR_7K4GWFdni9lCul-$xj!fA$SA&0h1L%FamelT#QY$RSx5drHJh6G*5sv9`t*5t
zKV$oLR_!$XjR78An`k3OKLap98@Rsti5NMp4If6Da{uPQhxfzl=CLWDxZlzczN~7p
zWcGa%(8D#vQ;UZw#{iNypCZ*&Hgu=o5FVBSmdgd3XTGszx`&jkT!FoW4MHa}fk<`%
zUJMk7L5K&6LPbt2oDB7aFV*CbB2n|iht}J;t4X-M?a7&AI9`^87K*P5Id19nMf2_s
z!RcF*YF`VI&t!EJ440#Nn|R4c1)3=oLuDfM#4#+s`6gVRkdkeT;l7f#FH8aj6>2E{
zCkz?x|K{ATJzXfW)il*tSRXW-BDS;Mz82FECN63oQFG5IzP3zJv-DFE_t$aa@8e|y
z5Y-}<=1<VUf<>@Gf|x4oV`Aq!->6UG>qHn91o`%0ZRKQ`33!p9kJHUn5~4&vAt}li
z<Nko`%g~WC*Q`*b<lle`2?4hr8rKT!R$8lDpn7IdbqzMWPi5n{_f{D|-|n4nQEmt|
zU8?)vwr+ftxcx+@3IszKT(K?SpdlJc8U%O8=`Y`IGl!2c7Y>UH)0N3sztUzq_lK_(
z-LqP5EOzn8C|S5N`LmJjKdx|?P4}m(PDI~h{ec;GS-mO?K<3=C(25J3l8Qc}K1h$v
zmCObB>_HQ}m%gTnk2~X&e7qb*r6-4og$>lM&}<u;7^f#s&xoX0537<XMg-l~BvY8y
z?}Cjoak@-X&ar^)6H<Hq4b6eGndB(=GdR_Sa%-+pf65MLeeN|hXfwc4;oXQ#J@ry)
zIY6O|cr8)G9&#+9Rm3f^W3poM6l7wm6q>M#$<mVgIgfA;xN0C8mL<|x4LZNNhPi5W
z{XlhEFze>%w5g@xP}VYXlt^ILXBSre`oP6TQWfmu>3Fq%GC|)BiAo-JKF=-9<Zc)E
zzj^;~>vog8yYg{+vrCAP^2^5-g8@%~wi<zPdBQ!ts3mGma%=Ta_!AJxxPb-6$Ka=b
ztuj-VnzW4XTlxA7>gCg<KRvQGG-UT2@`TnsS+2v#{pB<3cAM&8bI_0*r)snWlc_x)
zftL;N(lHsFJyShVEybvRnOfQ+KXu^hjH8ychMy=@?f$I)i6Ci_uJU0jbhUS-2ybSe
zEiE$~5<?ehnc9wF(MO0p;fYUIKu7qpI$j*$CLz9ovm~QabFQ%l4X64u$JtGPdIT^r
zavF~%rA7E(j(sovs6{^&J8X24LOV^xD8`4+W0g-hePxag8s?bB|7JmH$*5bTrU~ll
zdrKP!-~{nf>hF<nU^_X{6&;0CMSLDc0~4+8(1@lyR&pt*qyQ1Ge)7oRqmZ<6ulagd
zVNfAup7bkaRyBx74Bj_-w!5LL>{31sLVV9f!8YRzY`<pN8cP|{12btTaWFZaiUxA>
zto`TEz=0hUTyVL3xJzMfe{On`vvt(=0NNJjGlg*S=8A@yAA0pgBmSE<XL%-8I@@<R
z&vT|~*c>oGzmNjUAZJ_si-y{J>psMk)h~tEbvvWkTRlZhDOUv9>ZHiI`NO*s#^B~Y
zCbukqoJy6r%ejwD6u7EI{{O&rNG`mYoWqk+&<6H7LtKah#O`{PNVA$h;8(~}GPgbX
z-#toDoGT0heM?u=t3XWkSFypUCa0{8<}AM>YxD6{EvLhJ2{pw7f3Z|w=ezd0)!Vnf
zS`a4`1LdFK#3{HxR=?ENeLRM<m@xu=T<G^yF`Aar4z#n+e;A=^eUVrn)X%ObbuQX`
zF(q33m0FdKKo`4o;km4CCublhdUh`SrB*k!?jBuHAUdlMK9)kN5GE3}Ea7^glc9FY
z7s%U~O8X6O5Z5jVG^j~I=p4|NR5N~bajSXRYD2Uo88xQv{siZQv##jPI>vkdj$HjL
zT^Q&6F<-o#I)*N&t=w#Tx!rat^ScI6Oz<ET#E~&m0Z(!4Jga|AC($QbHiLAmZkiWo
z*dA46F5i<L$ybEmH+}p;xG)f$A0yHEsq!wzHz=xZyMG=Uzfo9xoLI|JH6=AlQ~4d|
z7Yh|%#kT8}FrqE`M*p$s*FwNDGT%`ecKpn7*df*?{PTX@5PvL>$<V8}fy@8G>sMTz
zhy<R|xe{UAyr8nZ4pZ!mV0V!ys=<B}d8LnU4Z5BIvK^gBtSY8y&-bC~0`XU<c=)Fl
zzUw>dw2C(4Yj*}=u{V7sDq9c=S!gF>J7$Onzhj2;DMq5XHli{%&&i7Y@^HewgN_55
zp?Dw`a|kCnIEO=QkqA}M!{96)Ye<v50`vl!?k`#&=4jM;@+kro^XE20RY@3iO={Li
za$)u6$D@9B%>wbR-#VQFVXPZZdHUg?^Hc)6)L7?qoj|{cw$^KAGT)6^N_05`;fV0W
z!?Pj4Lp<P?uU7o(*=;1zu@ZrU3Loi0W3Lw9&t0|?Epe;%<}Y@}4;2P|*-&z7dI<5^
zzqt9pjw<}1EV!J*2kBt7BxTrO7Y31S#BRC*^Lm{<i`1{zu~#~!bYgrKz|t85>_2lo
z-O%MPg;}iEo_auv1P38&oOmJVRnQB_*m%fgk$Zk<+h5H>RTbbk-**hxJP?%mjHn4t
z;7Mrs>l8_j5st29^y2@e#pf9<h{2{og3^Er)d|t^{L@i-XN4`Pt?B*l7S3$?Pde_u
zBdeuiFS#=n7vPm00zLLqbzJ7D?!vjc_(Cc~rN`p{K~}C8bRG!QfKHRMdE#&Pzvq;h
zi0W64bCeHEQCLU){Ej_kpE}d6-${8ruQW~yz&!tXnOr`q#c5uaZQLQ<!Ftxrn}n#%
zvmFdiyd|RQRLsXGCUTD|uX6VuI4@^uGD;wI!~uSYUPxWu=ilhj;^MpS{PI}i8$D;L
zW7cdOkEG)}QB}Zru9JmUNz84Xgs8}_Yc9aN`6XX~rSrUA^bi|zbk#uPKRYx_m%}J#
z(<d7LHcYWO&1V*Cx!X(BEe9|kb&6tr(~mK<+lm%1hl_`ySm_`Q@!ww>xDV(*hFYd-
z^x2iDHBgRmE-~Dl&#{_~tU>tkZ(y3}KU$|{Bh{p|7iD=2A`f<0NmHD{f|=>p*C?Ug
z%gJ%L=G!hN&P+c&?B;C^eo`akW7kUMxOV;g*;hiRoE^wKYwb;B-)&3P?t9jdfzQ%^
zyKU2(;kS@OOZPhp>P!;oXeq0rMC|+{+K{?CDj&-W4$LDkk}^D@*ULtflt5lHEk|+N
zlejl4GdI8~yl#UM<JIb#v*z|D4B-Iu6*0q`f^s_co6V)u4<@W1R71xzTer%Bk3w^R
z&l+VYKoiB{0}5ki!2L-|QpT&0n-&o+8d>>yHefV@#E6z?u=pkYsjINKTy;NvlkwlE
z+4VXbaI#F}zr#4f#gd}~EU33XqSQqVHQMnyUf}Q>3+K4db~08PLhUDURbBg~i=N^Q
z%1@=deEM{i$S&BZik35EtumgoUslt#VD;Pwzp^I0qxRex0{r(~z$xXBr~Qo@qV*%c
z&wq^!!`BWR!V?mhB`Jb<SFjnRlq8dr)1J1HkaPrjCGW1^8N{h4`F#1VPft(Sxv;+e
zA)@h9>8wtVR&3A!bR#kp2j6}!2jP3MU~EMH`b-@$YCmMrvEyA?#&v%zG3~xj9pZM9
z(D{JQw?sDSw}-_(+)G!;s({$bAteljI$5WU7BIUP=?kMBb4Rudj-{0{MwBKrQug5;
zUel>wwYcs%`3G7$En(Y*x)TXR8jQwWkcu=H=7UC?GYd6LIH(e77PI0MkpR-`1#DD|
z`@7qn()yOp$T68r$KH|6ElC4eN&G}s&#j#1j*l1*DsL2U;|%tam)Pp7OPPO&2MV=b
zr@epGkFo@(kw1sO?VOvBg#f@>vke^$q^D_NA=w-(%|e0oUbVOR->CER2~ie_1?6>B
z(Z@J|h(<Uo)`HW}=@=wA8Iwpb-shELtQYFFTNSk28|!5bUI-_sQ{w>aR5$bP*NO8m
z_X#dU##Pw<{>2ok+Sz6rx%9vBi{3E+eXwo3LztndgWiGc-IgRNJp7g)Mp_JB%>}>y
zO~|vPWBR$~l|>GuJ{U^Fmx%zwMwOG#FQ~u02aCH{n?3j8vkE<AE2(YvKqk9iIy_Qa
zgJ3(u<bn57&<T3G>tv5yBK`%oCPaB7zIPJUWahpxJQtM*(Xq;FkY&<*JuQeAV<(SY
z5<gA-l0V6BfX}sxZ#_YtKqq#j5BDaH_CGTlQHq)-7~uN%&aMr<*aNm@sn3FDv+FB|
zVPmQZ$($0*r_yp%f!dwyNPNC`w4gxZ@pnA0)gh;B#7NbyBK-~&|6oeC`TW<v8cqgA
zND7q<Jr)k-4<#mk+pM~2darf_3DyKEJ~Xi}Kz5dxiY`B2>Re2Pg^2-TR!neBM5k|H
z?P9$Xz*9mlsi~!w9sdiXx<LJSDz@?T=?u;SngzxGGSt4GT)QjupvarS5d~XH`AG-S
zA9bX%ArJ04LW{oVjJ8w9R~vDRJkqs~v5)Cn9IrGLdGi!^HS!YT(mArt-`rC3=*;l)
zB)#nPU!Ttg$ipw>Vx43!*gppZj2!hN?-eyGUK1tD+IKkAPR$$6P21ms{!mP9_?0^;
z(w91YXm(#TvghJ^D`pF1T4Dg~0E|H5C2_?E+|*s_7UcfQ?XwVU$!by60U|Hq^n3_V
zL@v`j$~u!{4J0kPMMU5)DS>H}|AEU*M7=BR5?=C>8?Q`X703zj)J_E&+LIcs#0`f~
z$_!E09cS}Qa+jrr&g@YRg8oU*4nj;bUv4$BQ|F$X_5z0B`$g;~^3L=UHOfB(W(;1c
zY8j^?s_jr|cZsYU^ok;U)CgRJ)%PtZW6Q>(WLMO-w^3<!4%d?OASxJmg_q0;nxvv-
z_lI=M-)H;J@lmPfTfr)qB5f@>jkzdQS{n%?;o<aQx5TdTmN<3LCpBKc{6Vy<Q-n;Q
z4V(>?H*gSsu*MpJ!S*grrN--DF*SjYm-w%T0XaTt-2upA_tEFuM_0Q0wD<hi5vA?O
z<|8Vxtvi`IyKioQDqMl}DDuz#$NEl|W^=B87q!MC6jzk|ndl@p<fnd|*>z#$AQAl^
z-*MbnnyKzpaSmYuS;s5D`}Y^zwSqFjYA>EyD*0hoY>h;lYr4$Ob&KaoIV~k^vrE69
z&{u2x0o$={W(#{R+m>DgWITLE)Om<$dtAFq#0Aey#Gmkzy29GZ*{iUnuzFzs$Dw|t
zg!u3idY5U@fgZi^E9C{;XupGP>;vDcss8BpGfRvL`OL78+TnG~WzEngij}F32xm)!
zqnr0jpa5Q%^WAHOp!_YH9@g=^H`SIq(5=1)BaQDaPVW{s!pH|f8Sa04d|_#3LL=bD
z|K!_L)<w1U@3-n|rv9xg9(aw1LS&=NDr`HFR)No>{Pr+N9kDU=OV>)oq0d_&VrgmJ
zJvtTcw=#lerTGsMJ<RA~b=)!`ACzu^f&jbbVgN5@3mSAkd&_!l9l`nCW8>hFMjON=
z9ZhG8Koq7^>)M0OMkydN0m}f{N%^x!S10nC+=gMLLSiW~8+vDzd~1=Mu$Z7JzN&cj
z6@X&}M4ugLFNQ@?^BOH2yUxaV&F$H8yX1GIweAN$9vg;i2%V1jy&2^G%>40U=2$Bb
zIY{GqjNtu>IN};o><B|k)V>Z3K^RsbCEHsyE<c50*5})zr>0@(W$bPvNh%U)fMJBI
ztzL90`6mKOcQARCR`hLJxoJNcH%Z-@5fwz9AnKpI+y&56_Vs2bH6Dz0p0V>nUKlz0
z^GKf~rh21M+<>hQ7isP#B6I6>vOuh&i%b%13fF0xM5ox-!5$;bJ{78`mJ+wv`<%56
zu=GKt3Z^eH-f{er1&GUXpHnVeus4bHN|xc;=t{`u>Fdi(H+V66U?l)L|Ek5qcz7!O
zz1P<@l6G5_ab<WoA<TXnCH|`sCclkTp9z+S7WL3Cpegm{x&eip@W)87w&b<3VX380
zLC8Teukn(j#Z+bCaQl7$<1YUJ$J+q&=CBmAN10kcF%dnV7hG^RF4`INRpowOm55>$
zo`9oC1db;8XH~JV50g|r)$aEq9$g|LYSZ^|R|q9&rXr+4<fq`%{xkZG-CY<yi!T&j
z^NI;b%XgC6Mr4;>uSZi_QgOh4fzp%z?l@$Cx2?~@{+<Te(e});-<r*nj}B-KTpmC?
zT~g`UR)Kc{Z9w>Q(>zyHk1$!e<&3-`Hyqd+xs$(o@bLlvJ=o<q_$a|wVyp=g`V1lD
z;hFz?K0%ThIbZa@bCz3*RS2|G|8sJ|#pyqgVJ~NK>F5~SIs0ej*pW%0b+->2uWKAU
zd%_6P@J&<ieM{gJglDZ2WrvheNB#Y#l1U_7oKxc0hmeU@a!HjmMiqHJPK!fvbVZ9Z
z9EKl${@Dlug*B-E<3%RZ@|!sO=tp@Zei43IL=oh^WmfeKnC1}@j>R7F21b2&!$ZZZ
zZ3tT!{a#Hh>2)v>RYZqHPNq2q+A&W7c5+nha+hrgY+kAuiL<s_>zN_xZS?^5aCcm$
z2ce%*^fV2jBMG>APXJBV0iroj*-DHlrQu+!aR^+bRY@e-nOL|SH4c8YJQ~Xl3F(Q^
zk(*VrMT5GnS2`A=iX;s^;g@Cns>^QP#fp~-qBiGjmbQ2mTgpwg^#Avv#F27-Rjz7I
z?sD+B-V6g_s)41SHpRbd$YPU`xI5R4cToK8W*S23;QOG2-TdQ6no&$CvXZj@_B|w_
zYvc9{_0+wIHahJ8&Ic8vYAgR5!FFPNhL*KEPn;<>mZn!ad15byhumr^DiaxE4p9lx
z|0(Ru<DuNU04^pm*2t2C7|cwCCX^-al)*%pY}chy=vJ0)mh9UIV^4_4nq@4>MY=+a
z?b?!Ly{<J%5tFi1gZB)#`);4l`_5m#&*xd@{FZZ`^PK1WzGr#YKBuX5T*67WE5TM-
zzbZx|d+I0B6S|1V_F#|G19?2(Z}(~~M*yjF<ovtON^(!K_NYss&xY;J$q9d>IX=l^
zhZFqtNbv}pW%XQZ84meDrSY6c&NVbP=V~Vr;}!VavRuLWdpA=cm0cgr%IVNt;T`JG
zb2vTCMSrg@1(_;e78^(M!^H75jb(f^jB=n&#Z?WZj&V}6sS+;bF)R9=ooHOWB-$!j
z6^%t)1+cw+bji*2cj>F$`nDClT<r<#`ue2(;zj<>zg|^%5R_z*B9WY&Y{ea7Y}8X+
zP0d(zH{@vRBDTiDM(?N0HH*`{$Lg2Ri8hRS9^&$DdM)yS2_lzDA?iKkaKYa*K_fkh
zbrl_~SQU|^1*=CT&3;!V&d<~A%s9w>yFsPnv#D8ApRwgZ?FJlOKHd%YY7r^YMHQyp
zjp7LjC(E;qq6@RB6OL)}2fhrq1Y1OiXln?_nSUP#<{J8(>ltuaxy*=i;+%7x$_+7f
zf?4J#0%`x=Qa4v}#@F-mKN5M<?La?7hp+stPvt^FSyf}$4w)EhllS@iPTl{J^fPH#
zlk<f3D|#PdJv`0XDKk8;rF;n&`A{-E<9?R0!U#Y%{?bMWE79L1>iO85CmtEc%dpDL
z$zeZ+&u(IDU7ONPGpT<h7^uMa*zNH(w2OQ{!T;g4oxAaw9u3^o^WebEVo8wh(*1>Y
zPM_uj)TSnPTkdGoE|Z`Z<JdtN^VQA%_XezNv2os9#nQ>w48Qj_rFBoz(%)tPOg4mL
z@b+AFS%w~qw2KdZUu#Cix8(YbRXkyL6Fxlbzvjh%BAv+++P0iP8)&<WGJ01z{OOtF
z%SKsLBo+1MlSEyB#?7fjD=y)8R>Sh;-qRDzTps3YCj1ls@KQBZndiCOi<A?dAIp!u
zsj95K<jMp>ZJ9ZU!%&>EDIG)YcC_*&F3XYuC~$4t-+dOa*i3V|C0|r7)$m-c@I#vu
zb_-+?pT*Z!gv59i=DQud<3l}Asq)Z0{oSYZ@FccfeARx7q~kK-xRNmD%IqKcExDlX
zc9<;oSB1dG2Vg_`?VQxdwl60xHigHEubX^2<HWmtE<XZ)n_bs+F!OHqa2e^r&t2A8
zj#v3SVAmwG7YMwVD3L^)&eP!Y*%<Y0fB$%3EEp5&*Upq9REmSFth9$DJ1w-R=Cia~
z8dx%1*)JoVK@&R73q&z20?(ft3@O|k`9GJKj=r3UbE%0H=QDUXY6W4tt%I;TAn4Q&
zLybbkzKs4k#zWu27S^p~EDo}iKAUMFRn%tE(XMm1UF-6G`U|!*`~P_`p;eG@xs3d7
zFNQF4XBlNg)bVi<l?lK5ds2xdA51SlBHj_^pDabO3EAq&v>1~b!67{es}cUZntiJS
zZ7N%$=#)ZO8cud)nw<Z>G!;uoviiCMj^OQii?O5#M^SF}5Lz3sZJ3StIjG_V4Vgi3
z)1jeT8p5oW1GO_Cph$y8HE$#`1RAdUp+XKcrq$8dts8j!zZ<+2Bqb%@m1g#%3o1Lb
zq~=gM{-g<QNfK*mOZm}sTYez*{SW#7U)M33tpmJq=WET_xeqy^$2G`O9k3E7y&dhV
zep1WHexUbd%%NN3=$y!-v9_ahUd&?kj+ZK`1-18UKaTarE@kB13a;jv$(!NvH$B+l
zYXRA+NGLNKrs<Wh5a6~X_P9V14VP5k{8qZ=Fs%}^a?iV@6ca%7eqN_%j9IjTxFkjs
z-o?l{0`<|g>;wYLO&g?-a^p!67j(}<Dkw~$J-8CExSJJa9sv>px?7GQ$bz>&l^hRK
z+6J>~1^RaYjs4i!1~YGOVn#Ux;h2^A;nx!rw?AK%mX^Msp8g_TLtdiNXBp7uHa0fR
z71}X1HQxZ$+WcA!4LltLf#Ch(EO_EzDUor?IvxNUZEMrl%3PeC^_m-aI45amXP0pQ
zKKD~H^2j`ZGH=|tL5~QwY8J&Q8JL?Vi4K?m(hh{^@VyIfmxTbN@XD1h>g_%2?EC>h
z-U)<R>eABE{QUgN!YCJfoS|POSzeF+@EEEWHAe{2*(xh5OUx$eulolF!*y4Hh^xdI
z;$OtU7>XcV6HwdT51eyJO)nPz72s^P<QcoIE}ZlA^*wc}>49FH#PMH#(9_dnj0T7;
zRaiE)6;OeJ%$|v`wjMdH%?SwDz;7#u88KdOyZ5KulBBGqZ5|R)u#pljBr?b9sxk?m
zq2-^Nnu>IN3YcI>w!J1>1W4tR2FU!tmHB(Wp?0qg1JfTRCs$NhNJGt@mg3JXC~))_
z9*Px=>O;9o7kSJL)CH`IRVf0q5vZvE3ZI&uMz~!Z;Ta!=L1r)*zuCCx>r>WzHv%&>
zz0VdKozo)htUvCqp58QB>-F)Z&7C$N*<MsYJPT?E-f@HcfrAGP7&5o@^R(NG;M;~h
z5grOK2DbUp(a~>TWHzB7q?Pl>lk1t))oKj0_4V}@*@%WiL#}u|f;+bqRx6V)&Et|w
zx{P+s25>faiiPo%=F54cJ#!o^ot?d6a(qZE5gz|ldb+y0Q@cH4{35SGeh#ETI5;@i
zUb;b)Zf$L)YXHEORZ<l8yU^qh$jODJSifbiw%(K+vIUEJ(a_M3fL&tBK|Z52yDNa!
z@R=?sY^nXGEnWtX#{<iDTDSIH1X8ue!os4ZrA0zQVybtOEpS!oUKf>$zI4GNL8Koj
zqj>Gm&`>oswH~2|Ru#IMcFXT!D4JHgFa7H7?oNR(S4=I5k;4w9Jguo|dUX5o<7L1K
zXPpG*{LItpYFbqox0C&o>*Gd;`h%b>6Fyc_n>RH)GX=Pu({DHfx_hqa=;+wg`6N-5
zPmu6^eSNkgCm+0=s6Z+Bb2~w~X~1mv{6w=)5D`M$LM&h~m?KI`LG`xH{r$z0ib-8v
zhO>%~F4P-AHWJe|#Ai?^2Af)1cB(3PYPXJ!WzcLJBsb`3r?_Qwb~Zsq5v%Ma7nk$J
zb#GM+1^N47CH_F7i?BF(0{NLD&A?2|&vVzi!|>)#q~%g4*aXGIko!`B`Wt0XYzaHs
zj;X2b_2F7_%j0Ik)klCV+}Y9b!^pm4E~rR4v8<w^c!-gjmXL|qBLtc7pkz>EEG0di
z2TPV?5fB&WE+sq1c=0cMEv}<Vfog=7?g_X8sW?(L9`i#-hPd+z3Iq#C{5wcV$;t9^
za_1S!)=yl7;;2^Br_JbZF4PC`@W}ezZBu3K;u+a5DJi+u&JWrO6Vy`}RsSI__~~@)
zY^sh6VP`NOi=TjP2-o=WFT|iybs3r0P@|w(l!JoQ;-o*xS}UM$_N^@O!3TT;u}`8x
zvS4H=NHLDt(-&W{1!`RXqmmN~?Do#*&qL1~b)rNZ;pb1%M`1CTU<C`@oj^3Gl`NLS
z>E=B{ImQlm#@dY>*0jqKajR1;Q2={q%ETJXQl&R_{hM85V<W{sL?uo$3i&KKD2Qdv
zjelc9W@DqLzdy8FAo5Off0Hpc2-S(+z%aCv70wAjcbS?G?|Zk^Dc4@N#sF&iz=rM^
z(597zfw?Nq8-85mo?*hD%e#UXXq}y0RXA9}N}R5&*J+nm0ma_TLcz^{4QaSO0LeWK
zztf3Bb-u^^1pXWyoqQ7Ey3D%p6$1WSkAgnD8ULE8`oV)-0W-Eyev^%%VCg=8c3+xM
zi_gav@$6Ft0CZgEjGVlDQ~vP4fUwqO&=as(B5@{SB)plq`B5A$>moUh_T|&t$)8yh
znmwL@iD#}vrua`Iu7Pz{P*5oIh8(b)9v-3N>Q@v|Px@}>W3e2{#yux#p8j8!X25z$
z#CgInw!29s>6xXeI=i~Ml2^pE*!kDY0h}E&uJv&~4o`x#(op!Ca_YTY5fl_?tP)l2
z0nHp99`^S3270-)F)OO~7U8k)stdrI*A^y@vX5L5i_wj*Tw>0<ywQCr1*)_bXOGKS
zEl3=OlI3*$Q)VC`EaP#&;6rz0H~aGJo5Fa?A|N;F7iVKav2|)RRm!@`Df`vCvK;~(
zh%z2^kN1gQt(&iI5q4U?g&GuY)qj?k<AxSy=jKd}Y)vId_=#k+mX<`o9y*Z{6O*JF
zL+02v6TSRNe)*UP<i?(!moGt;gD+?vKK$^7v5HG$4tajGM&u7nyguSN!rN7KZ=Qe<
z<G|Y2<+-V;OKYeGy{S(JXGnjKta=nu)fhO?tougmd_5uUG4Zg9JwTtKBO8oGsA&WS
z+!--63oQP@M;A8z#&OECfQJsV>M4mb@Q7gm@i_AqXl;Z1MX;=yfdZh+?3uTI8b#Oj
zet%|r2%A43Xc79aKz4zO(O*jvs_-zK1vD0i#_HS!uNJ~B4BQg9GJBcI1knI^{s-*!
z=824i%GU9+|LH&6@w{xYD^A>;y^<#yGhWm6uf7DoECk*Ez80UDD6e>-&}Bs7h33ni
zoZ7eS)7>GzCnhHxt90i<FIt_}9PG`mt*rex`^8|}yV-IpWiZ$t!EJX@@!vho)DA++
z<S*#gOg~dc?k|5cOu7g(6vlYl4OP^SM1DpI3n^P9-hkV2NJB#V_uEdf2yNe2MoUB}
txzm|FM#w=N&LOuUh@{=$)_=2E*kqAV>Wv<iEYD+sACX|BQ+m>t{13q)j6eVY
literal 0
HcmV?d00001
diff --git a/doc/guides/rel_notes/release_20_05.rst b/doc/guides/rel_notes/release_20_05.rst
index 000bbf501..f404ee7fd 100644
--- a/doc/guides/rel_notes/release_20_05.rst
+++ b/doc/guides/rel_notes/release_20_05.rst
@@ -62,6 +62,11 @@ New Features
* Added support for matching on IPv4 Time To Live and IPv6 Hop Limit.
+* **Updated the Intel ice driver.**
+
+ Updated the Intel ice driver with new features and improvements, including:
+
+ * Added support for DCF (Device Config Function) feature.
Removed Items
-------------
diff --git a/drivers/common/Makefile b/drivers/common/Makefile
index 96bd7ac6e..df2e840cf 100644
--- a/drivers/common/Makefile
+++ b/drivers/common/Makefile
@@ -31,6 +31,7 @@ DIRS-y += dpaax
endif
IAVF-y := $(CONFIG_RTE_LIBRTE_IAVF_PMD)
+IAVF-y += $(CONFIG_RTE_LIBRTE_ICE_PMD)
ifneq (,$(findstring y,$(IAVF-y)))
DIRS-y += iavf
endif
diff --git a/drivers/net/ice/Makefile b/drivers/net/ice/Makefile
index e22c34287..f493c9ed7 100644
--- a/drivers/net/ice/Makefile
+++ b/drivers/net/ice/Makefile
@@ -11,9 +11,11 @@ LIB = librte_pmd_ice.a
CFLAGS += -O3
CFLAGS += $(WERROR_FLAGS)
CFLAGS += -DALLOW_EXPERIMENTAL_API
+CFLAGS += -I$(RTE_SDK)/drivers/common/iavf
LDLIBS += -lrte_eal -lrte_mbuf -lrte_ethdev -lrte_kvargs
LDLIBS += -lrte_bus_pci -lrte_mempool -lrte_hash
+LDLIBS += -lrte_net -lrte_common_iavf
EXPORT_MAP := rte_pmd_ice_version.map
@@ -84,6 +86,9 @@ ifeq ($(CC_AVX2_SUPPORT), 1)
endif
SRCS-$(CONFIG_RTE_LIBRTE_ICE_PMD) += ice_generic_flow.c
+SRCS-$(CONFIG_RTE_LIBRTE_ICE_PMD) += ice_dcf.c
+SRCS-$(CONFIG_RTE_LIBRTE_ICE_PMD) += ice_dcf_ethdev.c
+
# install this header file
SYMLINK-$(CONFIG_RTE_LIBRTE_ICE_PMD)-include := rte_pmd_ice.h
diff --git a/drivers/net/ice/ice_dcf.c b/drivers/net/ice/ice_dcf.c
new file mode 100644
index 000000000..0a99cceb1
--- /dev/null
+++ b/drivers/net/ice/ice_dcf.c
@@ -0,0 +1,474 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#include <sys/queue.h>
+#include <stdio.h>
+#include <errno.h>
+#include <stdint.h>
+#include <string.h>
+#include <unistd.h>
+#include <stdarg.h>
+#include <inttypes.h>
+#include <rte_byteorder.h>
+#include <rte_common.h>
+
+#include <rte_pci.h>
+#include <rte_atomic.h>
+#include <rte_eal.h>
+#include <rte_ether.h>
+#include <rte_ethdev_driver.h>
+#include <rte_ethdev_pci.h>
+#include <rte_malloc.h>
+#include <rte_memzone.h>
+#include <rte_dev.h>
+
+#include "ice_dcf.h"
+
+#define ICE_DCF_AQ_LEN 32
+#define ICE_DCF_AQ_BUF_SZ 4096
+
+#define ICE_DCF_ARQ_MAX_RETRIES 200
+#define ICE_DCF_ARQ_CHECK_TIME 2 /* msecs */
+
+#define ICE_DCF_VF_RES_BUF_SZ \
+ (sizeof(struct virtchnl_vf_resource) + \
+ IAVF_MAX_VF_VSI * sizeof(struct virtchnl_vsi_resource))
+
+static __rte_always_inline int
+ice_dcf_send_cmd_req_no_irq(struct ice_dcf_hw *hw, enum virtchnl_ops op,
+ uint8_t *req_msg, uint16_t req_msglen)
+{
+ return iavf_aq_send_msg_to_pf(&hw->avf, op, IAVF_SUCCESS,
+ req_msg, req_msglen, NULL);
+}
+
+static int
+ice_dcf_recv_cmd_rsp_no_irq(struct ice_dcf_hw *hw, enum virtchnl_ops op,
+ uint8_t *rsp_msgbuf, uint16_t rsp_buflen,
+ uint16_t *rsp_msglen)
+{
+ struct iavf_arq_event_info event;
+ enum virtchnl_ops v_op;
+ int i = 0;
+ int err;
+
+ event.buf_len = rsp_buflen;
+ event.msg_buf = rsp_msgbuf;
+
+ do {
+ err = iavf_clean_arq_element(&hw->avf, &event, NULL);
+ if (err != IAVF_SUCCESS)
+ goto again;
+
+ v_op = rte_le_to_cpu_32(event.desc.cookie_high);
+ if (v_op != op)
+ goto again;
+
+ if (rsp_msglen != NULL)
+ *rsp_msglen = event.msg_len;
+ return rte_le_to_cpu_32(event.desc.cookie_low);
+
+again:
+ rte_delay_ms(ICE_DCF_ARQ_CHECK_TIME);
+ } while (i++ < ICE_DCF_ARQ_MAX_RETRIES);
+
+ return -EIO;
+}
+
+static __rte_always_inline void
+ice_dcf_aq_cmd_clear(struct ice_dcf_hw *hw, struct dcf_virtchnl_cmd *cmd)
+{
+ rte_spinlock_lock(&hw->vc_cmd_queue_lock);
+
+ TAILQ_REMOVE(&hw->vc_cmd_queue, cmd, next);
+
+ rte_spinlock_unlock(&hw->vc_cmd_queue_lock);
+}
+
+static __rte_always_inline void
+ice_dcf_vc_cmd_set(struct ice_dcf_hw *hw, struct dcf_virtchnl_cmd *cmd)
+{
+ cmd->v_ret = IAVF_ERR_NOT_READY;
+ cmd->rsp_msglen = 0;
+ cmd->pending = 1;
+
+ rte_spinlock_lock(&hw->vc_cmd_queue_lock);
+
+ TAILQ_INSERT_TAIL(&hw->vc_cmd_queue, cmd, next);
+
+ rte_spinlock_unlock(&hw->vc_cmd_queue_lock);
+}
+
+static __rte_always_inline int
+ice_dcf_vc_cmd_send(struct ice_dcf_hw *hw, struct dcf_virtchnl_cmd *cmd)
+{
+ return iavf_aq_send_msg_to_pf(&hw->avf,
+ cmd->v_op, IAVF_SUCCESS,
+ cmd->req_msg, cmd->req_msglen, NULL);
+}
+
+static __rte_always_inline void
+ice_dcf_aq_cmd_handle(struct ice_dcf_hw *hw, struct iavf_arq_event_info *info)
+{
+ struct dcf_virtchnl_cmd *cmd;
+ enum virtchnl_ops v_op;
+ enum iavf_status v_ret;
+ uint16_t aq_op;
+
+ aq_op = rte_le_to_cpu_16(info->desc.opcode);
+ if (unlikely(aq_op != iavf_aqc_opc_send_msg_to_vf)) {
+ PMD_DRV_LOG(ERR,
+ "Request %u is not supported yet", aq_op);
+ return;
+ }
+
+ v_op = rte_le_to_cpu_32(info->desc.cookie_high);
+ if (unlikely(v_op == VIRTCHNL_OP_EVENT))
+ return;
+
+ v_ret = rte_le_to_cpu_32(info->desc.cookie_low);
+
+ rte_spinlock_lock(&hw->vc_cmd_queue_lock);
+
+ TAILQ_FOREACH(cmd, &hw->vc_cmd_queue, next) {
+ if (cmd->v_op == v_op && cmd->pending) {
+ cmd->v_ret = v_ret;
+ cmd->rsp_msglen = RTE_MIN(info->msg_len,
+ cmd->rsp_buflen);
+ if (likely(cmd->rsp_msglen != 0))
+ rte_memcpy(cmd->rsp_msgbuf, info->msg_buf,
+ cmd->rsp_msglen);
+
+ /* prevent compiler reordering */
+ rte_compiler_barrier();
+ cmd->pending = 0;
+ break;
+ }
+ }
+
+ rte_spinlock_unlock(&hw->vc_cmd_queue_lock);
+}
+
+static void
+ice_dcf_handle_virtchnl_msg(struct ice_dcf_hw *hw)
+{
+ struct iavf_arq_event_info info;
+ uint16_t pending = 1;
+ int ret;
+
+ info.buf_len = ICE_DCF_AQ_BUF_SZ;
+ info.msg_buf = hw->arq_buf;
+
+ while (pending) {
+ ret = iavf_clean_arq_element(&hw->avf, &info, &pending);
+ if (ret != IAVF_SUCCESS)
+ break;
+
+ ice_dcf_aq_cmd_handle(hw, &info);
+ }
+}
+
+static int
+ice_dcf_init_check_api_version(struct ice_dcf_hw *hw)
+{
+#define ICE_CPF_VIRTCHNL_VERSION_MAJOR_START 1
+#define ICE_CPF_VIRTCHNL_VERSION_MINOR_START 1
+ struct virtchnl_version_info version, *pver;
+ int err;
+
+ version.major = VIRTCHNL_VERSION_MAJOR;
+ version.minor = VIRTCHNL_VERSION_MINOR;
+ err = ice_dcf_send_cmd_req_no_irq(hw, VIRTCHNL_OP_VERSION,
+ (uint8_t *)&version, sizeof(version));
+ if (err) {
+ PMD_INIT_LOG(ERR, "Failed to send OP_VERSION");
+ return err;
+ }
+
+ pver = &hw->virtchnl_version;
+ err = ice_dcf_recv_cmd_rsp_no_irq(hw, VIRTCHNL_OP_VERSION,
+ (uint8_t *)pver, sizeof(*pver), NULL);
+ if (err) {
+ PMD_INIT_LOG(ERR, "Failed to get response of OP_VERSION");
+ return -1;
+ }
+
+ PMD_INIT_LOG(DEBUG,
+ "Peer PF API version: %u.%u", pver->major, pver->minor);
+
+ if (pver->major < ICE_CPF_VIRTCHNL_VERSION_MAJOR_START ||
+ (pver->major == ICE_CPF_VIRTCHNL_VERSION_MAJOR_START &&
+ pver->minor < ICE_CPF_VIRTCHNL_VERSION_MINOR_START)) {
+ PMD_INIT_LOG(ERR,
+ "VIRTCHNL API version should not be lower than (%u.%u)",
+ ICE_CPF_VIRTCHNL_VERSION_MAJOR_START,
+ ICE_CPF_VIRTCHNL_VERSION_MAJOR_START);
+ return -1;
+ } else if (pver->major > VIRTCHNL_VERSION_MAJOR ||
+ (pver->major == VIRTCHNL_VERSION_MAJOR &&
+ pver->minor > VIRTCHNL_VERSION_MINOR)) {
+ PMD_INIT_LOG(ERR,
+ "PF/VF API version mismatch:(%u.%u)-(%u.%u)",
+ pver->major, pver->minor,
+ VIRTCHNL_VERSION_MAJOR, VIRTCHNL_VERSION_MINOR);
+ return -1;
+ }
+
+ PMD_INIT_LOG(DEBUG, "Peer is supported PF host");
+
+ return 0;
+}
+
+static int
+ice_dcf_get_vf_resource(struct ice_dcf_hw *hw)
+{
+ uint32_t caps;
+ int err, i;
+
+ caps = VIRTCHNL_VF_OFFLOAD_WB_ON_ITR | VIRTCHNL_VF_OFFLOAD_RX_POLLING |
+ VIRTCHNL_VF_CAP_ADV_LINK_SPEED |
+ VF_BASE_MODE_OFFLOADS;
+
+ err = ice_dcf_send_cmd_req_no_irq(hw, VIRTCHNL_OP_GET_VF_RESOURCES,
+ (uint8_t *)&caps, sizeof(caps));
+ if (err) {
+ PMD_DRV_LOG(ERR, "Failed to send msg OP_GET_VF_RESOURCE");
+ return err;
+ }
+
+ err = ice_dcf_recv_cmd_rsp_no_irq(hw, VIRTCHNL_OP_GET_VF_RESOURCES,
+ (uint8_t *)hw->vf_res,
+ ICE_DCF_VF_RES_BUF_SZ, NULL);
+ if (err) {
+ PMD_DRV_LOG(ERR, "Failed to get response of OP_GET_VF_RESOURCE");
+ return -1;
+ }
+
+ iavf_vf_parse_hw_config(&hw->avf, hw->vf_res);
+
+ hw->vsi_res = NULL;
+ for (i = 0; i < hw->vf_res->num_vsis; i++) {
+ if (hw->vf_res->vsi_res[i].vsi_type == VIRTCHNL_VSI_SRIOV)
+ hw->vsi_res = &hw->vf_res->vsi_res[i];
+ }
+
+ if (!hw->vsi_res) {
+ PMD_DRV_LOG(ERR, "no LAN VSI found");
+ return -1;
+ }
+
+ hw->vsi_id = hw->vsi_res->vsi_id;
+ PMD_DRV_LOG(DEBUG, "VSI ID is %u", hw->vsi_id);
+
+ return 0;
+}
+
+static int
+ice_dcf_check_reset_done(struct ice_dcf_hw *hw)
+{
+#define ICE_DCF_RESET_WAIT_CNT 50
+ struct iavf_hw *avf = &hw->avf;
+ int i, reset;
+
+ for (i = 0; i < ICE_DCF_RESET_WAIT_CNT; i++) {
+ reset = IAVF_READ_REG(avf, IAVF_VFGEN_RSTAT) &
+ IAVF_VFGEN_RSTAT_VFR_STATE_MASK;
+ reset = reset >> IAVF_VFGEN_RSTAT_VFR_STATE_SHIFT;
+
+ if (reset == VIRTCHNL_VFR_VFACTIVE ||
+ reset == VIRTCHNL_VFR_COMPLETED)
+ break;
+
+ rte_delay_ms(20);
+ }
+
+ if (i >= ICE_DCF_RESET_WAIT_CNT)
+ return -1;
+
+ return 0;
+}
+
+static inline void
+ice_dcf_enable_irq0(struct ice_dcf_hw *hw)
+{
+ struct iavf_hw *avf = &hw->avf;
+
+ /* Enable admin queue interrupt trigger */
+ IAVF_WRITE_REG(avf, IAVF_VFINT_ICR0_ENA1,
+ IAVF_VFINT_ICR0_ENA1_ADMINQ_MASK);
+ IAVF_WRITE_REG(avf, IAVF_VFINT_DYN_CTL01,
+ IAVF_VFINT_DYN_CTL01_INTENA_MASK |
+ IAVF_VFINT_DYN_CTL01_CLEARPBA_MASK |
+ IAVF_VFINT_DYN_CTL01_ITR_INDX_MASK);
+
+ IAVF_WRITE_FLUSH(avf);
+}
+
+static inline void
+ice_dcf_disable_irq0(struct ice_dcf_hw *hw)
+{
+ struct iavf_hw *avf = &hw->avf;
+
+ /* Disable all interrupt types */
+ IAVF_WRITE_REG(avf, IAVF_VFINT_ICR0_ENA1, 0);
+ IAVF_WRITE_REG(avf, IAVF_VFINT_DYN_CTL01,
+ IAVF_VFINT_DYN_CTL01_ITR_INDX_MASK);
+
+ IAVF_WRITE_FLUSH(avf);
+}
+
+static void
+ice_dcf_dev_interrupt_handler(void *param)
+{
+ struct ice_dcf_hw *hw = param;
+
+ ice_dcf_disable_irq0(hw);
+
+ ice_dcf_handle_virtchnl_msg(hw);
+
+ ice_dcf_enable_irq0(hw);
+}
+
+int
+ice_dcf_execute_virtchnl_cmd(struct ice_dcf_hw *hw,
+ struct dcf_virtchnl_cmd *cmd)
+{
+ int i = 0;
+ int err;
+
+ if ((cmd->req_msg && !cmd->req_msglen) ||
+ (!cmd->req_msg && cmd->req_msglen) ||
+ (cmd->rsp_msgbuf && !cmd->rsp_buflen) ||
+ (!cmd->rsp_msgbuf && cmd->rsp_buflen))
+ return -EINVAL;
+
+ rte_spinlock_lock(&hw->vc_cmd_send_lock);
+ ice_dcf_vc_cmd_set(hw, cmd);
+
+ err = ice_dcf_vc_cmd_send(hw, cmd);
+ if (err) {
+ PMD_DRV_LOG(ERR, "fail to send cmd %d", cmd->v_op);
+ goto ret;
+ }
+
+ do {
+ if (!cmd->pending)
+ break;
+
+ rte_delay_ms(ICE_DCF_ARQ_CHECK_TIME);
+ } while (i++ < ICE_DCF_ARQ_MAX_RETRIES);
+
+ if (cmd->v_ret != IAVF_SUCCESS) {
+ err = -1;
+ PMD_DRV_LOG(ERR,
+ "No response (%d times) or return failure (%d) for cmd %d",
+ i, cmd->v_ret, cmd->v_op);
+ }
+
+ret:
+ ice_dcf_aq_cmd_clear(hw, cmd);
+ rte_spinlock_unlock(&hw->vc_cmd_send_lock);
+ return err;
+}
+
+int
+ice_dcf_init_hw(struct rte_eth_dev *eth_dev, struct ice_dcf_hw *hw)
+{
+ struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ int ret;
+
+ hw->avf.hw_addr = pci_dev->mem_resource[0].addr;
+ hw->avf.back = hw;
+
+ hw->avf.bus.bus_id = pci_dev->addr.bus;
+ hw->avf.bus.device = pci_dev->addr.devid;
+ hw->avf.bus.func = pci_dev->addr.function;
+
+ hw->avf.device_id = pci_dev->id.device_id;
+ hw->avf.vendor_id = pci_dev->id.vendor_id;
+ hw->avf.subsystem_device_id = pci_dev->id.subsystem_device_id;
+ hw->avf.subsystem_vendor_id = pci_dev->id.subsystem_vendor_id;
+
+ hw->avf.aq.num_arq_entries = ICE_DCF_AQ_LEN;
+ hw->avf.aq.num_asq_entries = ICE_DCF_AQ_LEN;
+ hw->avf.aq.arq_buf_size = ICE_DCF_AQ_BUF_SZ;
+ hw->avf.aq.asq_buf_size = ICE_DCF_AQ_BUF_SZ;
+
+ rte_spinlock_init(&hw->vc_cmd_send_lock);
+ rte_spinlock_init(&hw->vc_cmd_queue_lock);
+ TAILQ_INIT(&hw->vc_cmd_queue);
+
+ hw->arq_buf = rte_zmalloc("arq_buf", ICE_DCF_AQ_BUF_SZ, 0);
+ if (hw->arq_buf == NULL) {
+ PMD_INIT_LOG(ERR, "unable to allocate AdminQ buffer memory");
+ goto err;
+ }
+
+ ret = iavf_set_mac_type(&hw->avf);
+ if (ret) {
+ PMD_INIT_LOG(ERR, "set_mac_type failed: %d", ret);
+ goto err;
+ }
+
+ ret = ice_dcf_check_reset_done(hw);
+ if (ret) {
+ PMD_INIT_LOG(ERR, "VF is still resetting");
+ goto err;
+ }
+
+ ret = iavf_init_adminq(&hw->avf);
+ if (ret) {
+ PMD_INIT_LOG(ERR, "init_adminq failed: %d", ret);
+ goto err;
+ }
+
+ if (ice_dcf_init_check_api_version(hw)) {
+ PMD_INIT_LOG(ERR, "check_api version failed");
+ goto err_api;
+ }
+
+ hw->vf_res = rte_zmalloc("vf_res", ICE_DCF_VF_RES_BUF_SZ, 0);
+ if (hw->vf_res == NULL) {
+ PMD_INIT_LOG(ERR, "unable to allocate vf_res memory");
+ goto err_api;
+ }
+
+ if (ice_dcf_get_vf_resource(hw)) {
+ PMD_INIT_LOG(ERR, "Failed to get VF resource");
+ goto err_alloc;
+ }
+
+ rte_intr_callback_register(&pci_dev->intr_handle,
+ ice_dcf_dev_interrupt_handler, hw);
+ rte_intr_enable(&pci_dev->intr_handle);
+ ice_dcf_enable_irq0(hw);
+
+ return 0;
+
+err_alloc:
+ rte_free(hw->vf_res);
+err_api:
+ iavf_shutdown_adminq(&hw->avf);
+err:
+ rte_free(hw->arq_buf);
+
+ return -1;
+}
+
+void
+ice_dcf_uninit_hw(struct rte_eth_dev *eth_dev, struct ice_dcf_hw *hw)
+{
+ struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_intr_handle *intr_handle = &pci_dev->intr_handle;
+
+ ice_dcf_disable_irq0(hw);
+ rte_intr_disable(intr_handle);
+ rte_intr_callback_unregister(intr_handle,
+ ice_dcf_dev_interrupt_handler, hw);
+
+ iavf_shutdown_adminq(&hw->avf);
+
+ rte_free(hw->arq_buf);
+ rte_free(hw->vf_res);
+}
diff --git a/drivers/net/ice/ice_dcf.h b/drivers/net/ice/ice_dcf.h
new file mode 100644
index 000000000..f44c09db2
--- /dev/null
+++ b/drivers/net/ice/ice_dcf.h
@@ -0,0 +1,52 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#ifndef _ICE_DCF_H_
+#define _ICE_DCF_H_
+
+#include <rte_ethdev_driver.h>
+
+#include <iavf_prototype.h>
+#include <iavf_adminq_cmd.h>
+#include <iavf_type.h>
+
+#include "ice_logs.h"
+
+struct dcf_virtchnl_cmd {
+ TAILQ_ENTRY(dcf_virtchnl_cmd) next;
+
+ enum virtchnl_ops v_op;
+ enum iavf_status v_ret;
+
+ uint16_t req_msglen;
+ uint8_t *req_msg;
+
+ uint16_t rsp_msglen;
+ uint16_t rsp_buflen;
+ uint8_t *rsp_msgbuf;
+
+ volatile int pending;
+};
+
+struct ice_dcf_hw {
+ struct iavf_hw avf;
+
+ rte_spinlock_t vc_cmd_send_lock;
+ rte_spinlock_t vc_cmd_queue_lock;
+ TAILQ_HEAD(, dcf_virtchnl_cmd) vc_cmd_queue;
+ uint8_t *arq_buf;
+
+ struct virtchnl_version_info virtchnl_version;
+ struct virtchnl_vf_resource *vf_res; /* VF resource */
+ struct virtchnl_vsi_resource *vsi_res; /* LAN VSI */
+ uint16_t vsi_id;
+};
+
+int ice_dcf_execute_virtchnl_cmd(struct ice_dcf_hw *hw,
+ struct dcf_virtchnl_cmd *cmd);
+
+int ice_dcf_init_hw(struct rte_eth_dev *eth_dev, struct ice_dcf_hw *hw);
+void ice_dcf_uninit_hw(struct rte_eth_dev *eth_dev, struct ice_dcf_hw *hw);
+
+#endif /* _ICE_DCF_H_ */
diff --git a/drivers/net/ice/ice_dcf_ethdev.c b/drivers/net/ice/ice_dcf_ethdev.c
new file mode 100644
index 000000000..23f82a487
--- /dev/null
+++ b/drivers/net/ice/ice_dcf_ethdev.c
@@ -0,0 +1,317 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#include <errno.h>
+#include <stdbool.h>
+#include <sys/types.h>
+#include <sys/ioctl.h>
+#include <unistd.h>
+
+#include <rte_interrupts.h>
+#include <rte_debug.h>
+#include <rte_pci.h>
+#include <rte_atomic.h>
+#include <rte_eal.h>
+#include <rte_ether.h>
+#include <rte_ethdev_pci.h>
+#include <rte_kvargs.h>
+#include <rte_malloc.h>
+#include <rte_memzone.h>
+#include <rte_dev.h>
+
+#include <iavf_devids.h>
+
+#include "ice_generic_flow.h"
+#include "ice_dcf_ethdev.h"
+
+static uint16_t
+ice_dcf_recv_pkts(__rte_unused void *rx_queue,
+ __rte_unused struct rte_mbuf **bufs,
+ __rte_unused uint16_t nb_pkts)
+{
+ return 0;
+}
+
+static uint16_t
+ice_dcf_xmit_pkts(__rte_unused void *tx_queue,
+ __rte_unused struct rte_mbuf **bufs,
+ __rte_unused uint16_t nb_pkts)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_start(struct rte_eth_dev *dev)
+{
+ dev->data->dev_link.link_status = ETH_LINK_UP;
+
+ return 0;
+}
+
+static void
+ice_dcf_dev_stop(struct rte_eth_dev *dev)
+{
+ dev->data->dev_link.link_status = ETH_LINK_DOWN;
+}
+
+static int
+ice_dcf_dev_configure(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_info_get(struct rte_eth_dev *dev,
+ struct rte_eth_dev_info *dev_info)
+{
+ struct ice_dcf_adapter *adapter = dev->data->dev_private;
+
+ dev_info->max_mac_addrs = 1;
+ dev_info->max_rx_pktlen = (uint32_t)-1;
+ dev_info->max_rx_queues = RTE_DIM(adapter->rxqs);
+ dev_info->max_tx_queues = RTE_DIM(adapter->txqs);
+
+ return 0;
+}
+
+static int
+ice_dcf_stats_get(__rte_unused struct rte_eth_dev *dev,
+ __rte_unused struct rte_eth_stats *igb_stats)
+{
+ return 0;
+}
+
+static int
+ice_dcf_stats_reset(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_promiscuous_enable(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_promiscuous_disable(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_allmulticast_enable(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_allmulticast_disable(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_filter_ctrl(struct rte_eth_dev *dev,
+ enum rte_filter_type filter_type,
+ __rte_unused enum rte_filter_op filter_op,
+ __rte_unused void *arg)
+{
+ int ret = 0;
+
+ if (!dev)
+ return -EINVAL;
+
+ switch (filter_type) {
+ default:
+ PMD_DRV_LOG(WARNING, "Filter type (%d) not supported",
+ filter_type);
+ ret = -EINVAL;
+ break;
+ }
+
+ return ret;
+}
+
+static void
+ice_dcf_dev_close(struct rte_eth_dev *dev)
+{
+ struct ice_dcf_adapter *adapter = dev->data->dev_private;
+
+ if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+ return;
+
+ dev->dev_ops = NULL;
+ dev->rx_pkt_burst = NULL;
+ dev->tx_pkt_burst = NULL;
+ dev->data->mac_addrs = NULL;
+
+ ice_dcf_uninit_hw(dev, &adapter->real_hw);
+}
+
+static void
+ice_dcf_queue_release(__rte_unused void *q)
+{
+}
+
+static int
+ice_dcf_link_update(__rte_unused struct rte_eth_dev *dev,
+ __rte_unused int wait_to_complete)
+{
+ return 0;
+}
+
+static int
+ice_dcf_rx_queue_setup(struct rte_eth_dev *dev,
+ uint16_t rx_queue_id,
+ __rte_unused uint16_t nb_rx_desc,
+ __rte_unused unsigned int socket_id,
+ __rte_unused const struct rte_eth_rxconf *rx_conf,
+ __rte_unused struct rte_mempool *mb_pool)
+{
+ struct ice_dcf_adapter *adapter = dev->data->dev_private;
+
+ dev->data->rx_queues[rx_queue_id] = &adapter->rxqs[rx_queue_id];
+
+ return 0;
+}
+
+static int
+ice_dcf_tx_queue_setup(struct rte_eth_dev *dev,
+ uint16_t tx_queue_id,
+ __rte_unused uint16_t nb_tx_desc,
+ __rte_unused unsigned int socket_id,
+ __rte_unused const struct rte_eth_txconf *tx_conf)
+{
+ struct ice_dcf_adapter *adapter = dev->data->dev_private;
+
+ dev->data->tx_queues[tx_queue_id] = &adapter->txqs[tx_queue_id];
+
+ return 0;
+}
+
+static const struct eth_dev_ops ice_dcf_eth_dev_ops = {
+ .dev_start = ice_dcf_dev_start,
+ .dev_stop = ice_dcf_dev_stop,
+ .dev_close = ice_dcf_dev_close,
+ .dev_configure = ice_dcf_dev_configure,
+ .dev_infos_get = ice_dcf_dev_info_get,
+ .rx_queue_setup = ice_dcf_rx_queue_setup,
+ .tx_queue_setup = ice_dcf_tx_queue_setup,
+ .rx_queue_release = ice_dcf_queue_release,
+ .tx_queue_release = ice_dcf_queue_release,
+ .link_update = ice_dcf_link_update,
+ .stats_get = ice_dcf_stats_get,
+ .stats_reset = ice_dcf_stats_reset,
+ .promiscuous_enable = ice_dcf_dev_promiscuous_enable,
+ .promiscuous_disable = ice_dcf_dev_promiscuous_disable,
+ .allmulticast_enable = ice_dcf_dev_allmulticast_enable,
+ .allmulticast_disable = ice_dcf_dev_allmulticast_disable,
+ .filter_ctrl = ice_dcf_dev_filter_ctrl,
+};
+
+static int
+ice_dcf_dev_init(struct rte_eth_dev *eth_dev)
+{
+ struct ice_dcf_adapter *adapter = eth_dev->data->dev_private;
+
+ eth_dev->dev_ops = &ice_dcf_eth_dev_ops;
+ eth_dev->rx_pkt_burst = ice_dcf_recv_pkts;
+ eth_dev->tx_pkt_burst = ice_dcf_xmit_pkts;
+
+ if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+ return 0;
+
+ eth_dev->data->dev_flags |= RTE_ETH_DEV_CLOSE_REMOVE;
+
+ if (ice_dcf_init_hw(eth_dev, &adapter->real_hw) != 0) {
+ PMD_INIT_LOG(ERR, "Failed to init DCF hardware");
+ return -1;
+ }
+
+ rte_eth_random_addr(adapter->mac_addr.addr_bytes);
+ eth_dev->data->mac_addrs = &adapter->mac_addr;
+
+ return 0;
+}
+
+static int
+ice_dcf_dev_uninit(struct rte_eth_dev *eth_dev)
+{
+ ice_dcf_dev_close(eth_dev);
+
+ return 0;
+}
+
+static int
+ice_dcf_cap_check_handler(__rte_unused const char *key,
+ const char *value, __rte_unused void *opaque)
+{
+ if (strcmp(value, "dcf"))
+ return -1;
+
+ return 0;
+}
+
+static int
+ice_dcf_cap_selected(struct rte_devargs *devargs)
+{
+ struct rte_kvargs *kvlist;
+ const char *key = "cap";
+ int ret = 0;
+
+ if (devargs == NULL)
+ return 0;
+
+ kvlist = rte_kvargs_parse(devargs->args, NULL);
+ if (kvlist == NULL)
+ return 0;
+
+ if (!rte_kvargs_count(kvlist, key))
+ goto exit;
+
+ /* dcf capability selected when there's a key-value pair: cap=dcf */
+ if (rte_kvargs_process(kvlist, key,
+ ice_dcf_cap_check_handler, NULL) < 0)
+ goto exit;
+
+ ret = 1;
+
+exit:
+ rte_kvargs_free(kvlist);
+ return ret;
+}
+
+static int eth_ice_dcf_pci_probe(__rte_unused struct rte_pci_driver *pci_drv,
+ struct rte_pci_device *pci_dev)
+{
+ if (!ice_dcf_cap_selected(pci_dev->device.devargs))
+ return 1;
+
+ return rte_eth_dev_pci_generic_probe(pci_dev,
+ sizeof(struct ice_dcf_adapter),
+ ice_dcf_dev_init);
+}
+
+static int eth_ice_dcf_pci_remove(struct rte_pci_device *pci_dev)
+{
+ return rte_eth_dev_pci_generic_remove(pci_dev, ice_dcf_dev_uninit);
+}
+
+static const struct rte_pci_id pci_id_ice_dcf_map[] = {
+ { RTE_PCI_DEVICE(IAVF_INTEL_VENDOR_ID, IAVF_DEV_ID_ADAPTIVE_VF) },
+ { .vendor_id = 0, /* sentinel */ },
+};
+
+static struct rte_pci_driver rte_ice_dcf_pmd = {
+ .id_table = pci_id_ice_dcf_map,
+ .drv_flags = RTE_PCI_DRV_NEED_MAPPING,
+ .probe = eth_ice_dcf_pci_probe,
+ .remove = eth_ice_dcf_pci_remove,
+};
+
+RTE_PMD_REGISTER_PCI(net_ice_dcf, rte_ice_dcf_pmd);
+RTE_PMD_REGISTER_PCI_TABLE(net_ice_dcf, pci_id_ice_dcf_map);
+RTE_PMD_REGISTER_KMOD_DEP(net_ice_dcf, "* igb_uio | vfio-pci");
+RTE_PMD_REGISTER_PARAM_STRING(net_ice_dcf, "cap=dcf");
diff --git a/drivers/net/ice/ice_dcf_ethdev.h b/drivers/net/ice/ice_dcf_ethdev.h
new file mode 100644
index 000000000..0c34a0095
--- /dev/null
+++ b/drivers/net/ice/ice_dcf_ethdev.h
@@ -0,0 +1,24 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#ifndef _ICE_DCF_ETHDEV_H_
+#define _ICE_DCF_ETHDEV_H_
+
+#include "ice_ethdev.h"
+#include "ice_dcf.h"
+
+#define ICE_DCF_MAX_RINGS 1
+
+struct ice_dcf_queue {
+ uint64_t dummy;
+};
+
+struct ice_dcf_adapter {
+ struct ice_dcf_hw real_hw;
+ struct rte_ether_addr mac_addr;
+ struct ice_dcf_queue rxqs[ICE_DCF_MAX_RINGS];
+ struct ice_dcf_queue txqs[ICE_DCF_MAX_RINGS];
+};
+
+#endif /* _ICE_DCF_ETHDEV_H_ */
diff --git a/drivers/net/ice/meson.build b/drivers/net/ice/meson.build
index f9e897bbc..0ba9668d1 100644
--- a/drivers/net/ice/meson.build
+++ b/drivers/net/ice/meson.build
@@ -15,8 +15,8 @@ sources = files(
'ice_hash.c'
)
-deps += ['hash']
-includes += include_directories('base')
+deps += ['hash', 'net', 'common_iavf']
+includes += include_directories('base', '../../common/iavf')
if arch_subdir == 'x86'
sources += files('ice_rxtx_vec_sse.c')
@@ -37,4 +37,7 @@ if arch_subdir == 'x86'
endif
endif
+sources += files('ice_dcf.c',
+ 'ice_dcf_ethdev.c')
+
install_headers('rte_pmd_ice.h')
diff --git a/mk/rte.app.mk b/mk/rte.app.mk
index d295ca0a5..f3798a09f 100644
--- a/mk/rte.app.mk
+++ b/mk/rte.app.mk
@@ -185,6 +185,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_I40E_PMD) += -lrte_pmd_i40e
_LDLIBS-$(CONFIG_RTE_LIBRTE_IAVF_PMD) += -lrte_pmd_iavf
_LDLIBS-$(CONFIG_RTE_LIBRTE_ICE_PMD) += -lrte_pmd_ice
IAVF-y := $(CONFIG_RTE_LIBRTE_IAVF_PMD)
+IAVF-y += $(CONFIG_RTE_LIBRTE_ICE_PMD)
ifeq ($(findstring y,$(IAVF-y)),y)
_LDLIBS-y += -lrte_common_iavf
endif
--
2.25.1
^ permalink raw reply [relevance 1%]
* [dpdk-dev] [PATCH v3 0/7] vfio/pci: SR-IOV support
@ 2020-03-11 21:58 2% Alex Williamson
2020-03-19 6:32 0% ` Tian, Kevin
0 siblings, 1 reply; 200+ results
From: Alex Williamson @ 2020-03-11 21:58 UTC (permalink / raw)
To: kvm
Cc: linux-pci, linux-kernel, dev, mtosatti, thomas, bluca,
jerinjacobk, bruce.richardson, cohuck, kevin.tian
Only minor tweaks since v2, GET and SET on VFIO_DEVICE_FEATURE are
enforced mutually exclusive except with the PROBE option as suggested
by Connie, the modinfo text has been expanded for the opt-in to enable
SR-IOV support in the vfio-pci driver per discussion with Kevin.
I have not incorporated runtime warnings attempting to detect misuse
of SR-IOV or imposed a session lifetime of a VF token, both of which
were significant portions of the discussion of the v2 series. Both of
these also seem to impose a usage model or make assumptions about VF
resource usage or configuration requirements that don't seem necessary
except for the sake of generating a warning or requiring an otherwise
unnecessary and implicit token reinitialization. If there are new
thoughts around these or other discussion points, please raise them.
Series overview (same as provided with v1):
The synopsis of this series is that we have an ongoing desire to drive
PCIe SR-IOV PFs from userspace with VFIO. There's an immediate need
for this with DPDK drivers and potentially interesting future use
cases in virtualization. We've been reluctant to add this support
previously due to the dependency and trust relationship between the
VF device and PF driver. Minimally the PF driver can induce a denial
of service to the VF, but depending on the specific implementation,
the PF driver might also be responsible for moving data between VFs
or have direct access to the state of the VF, including data or state
otherwise private to the VF or VF driver.
To help resolve these concerns, we introduce a VF token into the VFIO
PCI ABI, which acts as a shared secret key between drivers. The
userspace PF driver is required to set the VF token to a known value
and userspace VF drivers are required to provide the token to access
the VF device. If a PF driver is restarted with VF drivers in use, it
must also provide the current token in order to prevent a rogue
untrusted PF driver from replacing a known driver. The degree to
which this new token is considered secret is left to the userspace
drivers, the kernel intentionally provides no means to retrieve the
current token.
Note that the above token is only required for this new model where
both the PF and VF devices are usable through vfio-pci. Existing
models of VFIO drivers where the PF is used without SR-IOV enabled
or the VF is bound to a userspace driver with an in-kernel, host PF
driver are unaffected.
The latter configuration above also highlights a new inverted scenario
that is now possible, a userspace PF driver with in-kernel VF drivers.
I believe this is a scenario that should be allowed, but should not be
enabled by default. This series includes code to set a default
driver_override for VFs sourced from a vfio-pci user owned PF, such
that the VFs are also bound to vfio-pci. This model is compatible
with tools like driverctl and allows the system administrator to
decide if other bindings should be enabled. The VF token interface
above exists only between vfio-pci PF and VF drivers, once a VF is
bound to another driver, the administrator has effectively pronounced
the device as trusted. The vfio-pci driver will note alternate
binding in dmesg for logging and debugging purposes.
Please review, comment, and test. The example QEMU implementation
provided with the RFC is still current for this version. Thanks,
Alex
RFC: https://lore.kernel.org/lkml/158085337582.9445.17682266437583505502.stgit@gimli.home/
v1: https://lore.kernel.org/lkml/158145472604.16827.15751375540102298130.stgit@gimli.home/
v2: https://lore.kernel.org/lkml/158213716959.17090.8399427017403507114.stgit@gimli.home/
---
Alex Williamson (7):
vfio: Include optional device match in vfio_device_ops callbacks
vfio/pci: Implement match ops
vfio/pci: Introduce VF token
vfio: Introduce VFIO_DEVICE_FEATURE ioctl and first user
vfio/pci: Add sriov_configure support
vfio/pci: Remove dev_fmt definition
vfio/pci: Cleanup .probe() exit paths
drivers/vfio/pci/vfio_pci.c | 390 +++++++++++++++++++++++++++++++++--
drivers/vfio/pci/vfio_pci_private.h | 10 +
drivers/vfio/vfio.c | 20 +-
include/linux/vfio.h | 4
include/uapi/linux/vfio.h | 37 +++
5 files changed, 433 insertions(+), 28 deletions(-)
^ permalink raw reply [relevance 2%]
* Re: [dpdk-dev] [PATCH] lib/librte_net/rte_ether.h:changed RTE_ETHER_MAX_LEN
@ 2020-03-10 15:18 3% ` Stephen Hemminger
0 siblings, 0 replies; 200+ results
From: Stephen Hemminger @ 2020-03-10 15:18 UTC (permalink / raw)
To: Muhammad Ahmad; +Cc: olivier.matz, dev, stable
On Tue, 10 Mar 2020 11:50:36 +0500
Muhammad Ahmad <muhammad.ahmad@emumba.com> wrote:
> Change the RTE_ETHER_MAX_LEN from 1518 to 1526
>
> Bugzilla ID: 296
>
> Fixes: c5b2d13 (net: add rte prefix to ether defines)
>
> Cc: stable@dpdk.org
> Cc: Oliver Matz <olivier.matz@6wind.com>
>
> Reported-by: LAVA <lavaraj@gmail.com>
> Suggested-by: LAVA <lavaraj@gmail.com>
> Signed-off-by: Muhammad Ahmad <muhammad.ahmad@emumba.com>
> ---
> lib/librte_net/rte_ether.h | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/lib/librte_net/rte_ether.h b/lib/librte_net/rte_ether.h
> index e069dc7fe..8953107f3 100644
> --- a/lib/librte_net/rte_ether.h
> +++ b/lib/librte_net/rte_ether.h
> @@ -30,7 +30,7 @@ extern "C" {
> (RTE_ETHER_ADDR_LEN * 2 + \
> RTE_ETHER_TYPE_LEN) /**< Length of Ethernet header. */
> #define RTE_ETHER_MIN_LEN 64 /**< Minimum frame len, including CRC. */
> -#define RTE_ETHER_MAX_LEN 1518 /**< Maximum frame len, including CRC. */
> +#define RTE_ETHER_MAX_LEN 1526 /**< Maximum frame len, including CRC. */
> #define RTE_ETHER_MTU \
> (RTE_ETHER_MAX_LEN - RTE_ETHER_HDR_LEN - \
> RTE_ETHER_CRC_LEN) /**< Ethernet MTU. */
No that is abi change would impact a lot of code and drivers.
The problem is the comment is wrong, not the value.
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH 00/16] NXP DPAAx fixes and enhancements
2020-03-06 10:12 3% ` David Marchand
@ 2020-03-10 10:36 3% ` Dodji Seketeli
0 siblings, 0 replies; 200+ results
From: Dodji Seketeli @ 2020-03-10 10:36 UTC (permalink / raw)
To: David Marchand
Cc: Hemant Agrawal (OSS), Yigit, Ferruh, dev, Neil Horman, Thomas Monjalon
Hello,
David Marchand <david.marchand@redhat.com> writes:
> On Thu, Mar 5, 2020 at 10:19 AM Hemant Agrawal (OSS)
> <hemant.agrawal@oss.nxp.com> wrote:
>> > On Thu, Mar 5, 2020 at 10:06 AM Hemant Agrawal (OSS)
>> > <hemant.agrawal@oss.nxp.com> wrote:
>> > >
>> > > Hi David,
>> > > > On Mon, Mar 2, 2020 at 10:26 AM Hemant Agrawal
>> > > > <hemant.agrawal@nxp.com> wrote:
>> > > > >
>> > > > > This patch series add various patches for enhancing and fixing NXP
>> > > > > fslmc bus, dpaa bus, and dpaax.
>> > > > >
>> > > > > - the main change is support to allow thread migration across
>> > > > > lcores
>> > > > > - improving the multi-process support
>> > > >
>> > > > This series triggers an ABI warning that must be investigated.
>> > > >https://travis-ci.com/ovsrobot/dpdk/jobs/292904119#L2233
>> > >
>> > > [Hemant]
>> > > As per the logs:
>> > >
>> > > Variables changes summary: 1 Removed, 2 Changed, 0 Added variables
>> > > 1 Removed variable:
>> > > 'dpaa2_portal_dqrr per_lcore_dpaa2_held_bufs'
>> > {per_lcore_dpaa2_held_bufs@@DPDK_20.0}
>> > > 2 Changed variables:
>> > > [C]'dpaa2_io_portal_t dpaa2_io_portal[128]' was changed at
>> > dpaa2_hw_dpio.h:40:1: size of symbol changed from 5120 to 2048
>> > > [C]'dpaa2_io_portal_t per_lcore__dpaa2_io' was changed at
>> > > dpaa2_hw_dpio.h:20:1: size of symbol changed from 40 to 16
>> > >
>> > > Error: ABI issue reported for 'abidiff --suppr devtools/libabigail.abignore --
>> > no-added-syms --headers-dir1 reference/usr/local/include --headers-dir2
>> > install/usr/local/include reference/dump/librte_bus_fslmc.dump
>> > install/dump/librte_bus_fslmc.dump'
>> > >
>> > > ---------------
>> > >
>> > > These changes are w.r.t modifications in internal structures and variables.
>> > They may be ignored.
>> >
>> > The ABI check considers symbol exposed in headers available to final users.
>> > If those are internal, why are the headers public?
>> >
>>
>> [Hemant] These symbols are not part of any public header files, but
>> they are part of *.map files to share them between different driver
>> libs i.e bus_fslmc and net_dpaa2
>
> I would expect libabigail to skip those symbols, so there is something
> I have missed in how --headers-dirX work.
In libabigail speak, we make a difference between *ELF symbols* and
types.
--header-dirX is about telling the tool what the public *types* are. As
you rightfully implied, types that are defined in files that are not
found in the directories specified by --header-dirX are considered to be
private types and are thus not shown in the ABI change report.
ELF symbols however are a different matter. Header files don't usually
define ELF symbols, be they variable of function symbols. Header files
can at most declare variables or functions that would be actually
defined elsewhere in source code, leading to the definition of ELF
variable or function symbols in the final binary. At this point, we
aren't talking about types anymore, as the ELF format doesn't know what
types (in C or any other language) are. So --header-dirX don't deal with
ELF symbols.
And from what I understand from the message quoted above, the changes we
are talking about have to do with EFL variable symbols which size have
changed. So in practise, these are global arrays (exposed by at the
binary level as an ELF variable symbol of a given size) with public
visibility which size have changed.
So my guess would be that if you guys don't want these arrays to be part
the binary interface of this library, they should probably be declared
static at the C level and accessed through some accessor function or
something like that. At least, that's my humble uninformed opinion.
In the mean time, the tooling can be tought to ignore changes to these
ELF symbols, as you you guys all know already.
> Anyway, all of those symbols in dpaa are part of the driver ABI.
> We are still missing a way to mark internal symbols.
> Neil had posted a framework for this
> http://patchwork.dpdk.org/project/dpdk/list/?series=5004.
>
> In order to get this series passing the checks, I recommend NXP
> rebasing Neil scripts (I will help reviewing this part), then mark all
> those symbols as internal in its drivers.
> Other vendor will convert their drivers later, as there is no need at
> the moment.
>
> Thanks.
Cheers,
--
Dodji
^ permalink raw reply [relevance 3%]
* [dpdk-dev] [PATCH v2 2/7] net/ice: add the DCF hardware initialization
@ 2020-03-10 6:50 1% ` Haiyue Wang
0 siblings, 0 replies; 200+ results
From: Haiyue Wang @ 2020-03-10 6:50 UTC (permalink / raw)
To: dev, xiaolong.ye, qi.z.zhang, qiming.yang, beilei.xing
Cc: wei.zhao1, Haiyue Wang
Introduce the DCF (Device Config Function) feature in the ice PMD, it
works as a standalone PMD which doesn't handle the packet Rx/Tx related
things. Its hardware entity is the VF.
Add the basic DCF hardware initialization, this is specified by devarg
'cap=dcf'.
Signed-off-by: Haiyue Wang <haiyue.wang@intel.com>
---
doc/guides/nics/ice.rst | 47 +++
doc/guides/nics/img/ice_dcf.png | Bin 0 -> 39168 bytes
doc/guides/rel_notes/release_20_05.rst | 5 +
drivers/common/Makefile | 1 +
drivers/net/ice/Makefile | 5 +
drivers/net/ice/ice_dcf.c | 474 +++++++++++++++++++++++++
drivers/net/ice/ice_dcf.h | 52 +++
drivers/net/ice/ice_dcf_ethdev.c | 317 +++++++++++++++++
drivers/net/ice/ice_dcf_ethdev.h | 24 ++
drivers/net/ice/meson.build | 7 +-
mk/rte.app.mk | 1 +
11 files changed, 931 insertions(+), 2 deletions(-)
create mode 100644 doc/guides/nics/img/ice_dcf.png
create mode 100644 drivers/net/ice/ice_dcf.c
create mode 100644 drivers/net/ice/ice_dcf.h
create mode 100644 drivers/net/ice/ice_dcf_ethdev.c
create mode 100644 drivers/net/ice/ice_dcf_ethdev.h
diff --git a/doc/guides/nics/ice.rst b/doc/guides/nics/ice.rst
index 8af32dabf..2639ae239 100644
--- a/doc/guides/nics/ice.rst
+++ b/doc/guides/nics/ice.rst
@@ -240,6 +240,53 @@ report a MDD event and drop the packets.
The APPs based on DPDK should avoid providing such packets.
+Device Config Function (DCF)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This section demonstrates ICE DCF PMD, which shares the core module with ICE
+PMD and iAVF PMD.
+
+A DCF (Device Config Function) PMD bounds to the device's trusted VF with ID 0,
+it can act as a sole controlling entity to exercise advance functionality (such
+as switch, ACL) for the rest VFs.
+
+The DCF PMD needs to advertise and acquire DCF capability which allows DCF to
+send AdminQ commands that it would like to execute over to the PF and receive
+responses for the same from PF.
+
+.. _figure_ice_dcf:
+
+.. figure:: img/ice_dcf.*
+
+ DCF Communication flow.
+
+#. Create the VFs::
+
+ echo 4 > /sys/bus/pci/devices/0000\:18\:00.0/sriov_numvfs
+
+#. Enable the VF0 trust on::
+
+ ip link set dev enp24s0f0 vf 0 trust on
+
+#. Bind the VF0, and run testpmd with 'cap=dcf' devarg::
+
+ testpmd -l 22-25 -n 4 -w 18:01.0,cap=dcf -- -i
+
+#. Monitor the VF2 interface network traffic::
+
+ tcpdump -e -nn -i enp24s1f2
+
+#. Create one flow to redirect the traffic to VF2 by DCF::
+
+ flow create 0 priority 0 ingress pattern eth / ipv4 src is 192.168.0.2 \
+ dst is 192.168.0.3 / end actions vf id 2 / end
+
+#. Send the packet, and it should be displayed on tcpdump::
+
+ sendp(Ether(src='3c:fd:fe:aa:bb:78', dst='00:00:00:01:02:03')/IP(src=' \
+ 192.168.0.2', dst="192.168.0.3")/TCP(flags='S')/Raw(load='XXXXXXXXXX'), \
+ iface="enp24s0f0", count=10)
+
Sample Application Notes
------------------------
diff --git a/doc/guides/nics/img/ice_dcf.png b/doc/guides/nics/img/ice_dcf.png
new file mode 100644
index 0000000000000000000000000000000000000000..540823c4a05e9ea9d558d233fea77016d1ae5df3
GIT binary patch
literal 39168
zcmZ6ybzD?k*FH`NDBT?jlF}e8B?Hn74BeolfOLbDz);d%Lw70NNOyNPNO%7Z_x(KY
z`+k3a@X<4KW}m&+Uh7)dy4F6yN(xdKsHCWHaBvvXpCpvw;1KwLKlqmjz$g7i%71`=
z;O&*A#NkSZ$ajH@7p7wJVsLO35oq^@NWk^0Z=W>n;oxu`p8w#NOsHJo;1Xh`CBz^u
zdIzZ}I(W`x-6a%La{^jclNOU&`_)rTS4-}zY76d1P4w1CVq)-OmrKC^5#ZY~-}pzu
zy$L?@i@zdO6jl2qp0>O_?^V0~==_TKu;0E)6AT_jK}Ys$GuU9xpl6NmUq4AAAS8rZ
z5<h>~BQL7h+1bg!^v{JkDk^GFZXt;WZK)wU@I4Lh>Nma-8Py=vX58uL5BwsNv0+TC
ztdd|ub|{f>I#qBR_r=Ecc2+(+F$m*+POs4gtgWq$;%5$Zz1W+)IolS4AR!?E!%MzF
zY^hc+(<6BKV$B>q*BptIQ5gaPfd*IJosgp=XT25!8zwm^L5M@!Q9uFabHF7#DiuCK
zt~qeo#^9j@Nyw!ThkWHBd%m>z=MtX)IqO&q%qUT-Q<|7KJNMhNs;a6pC&Kl7XSB)v
z3KtLW;^N}!aM9A*nl<f&LypeAH;UH%@~=<W?^;Mic5d!cle_a$qiaVX7S(%;iC{c>
zPEJmL@k|#dzb#-e#WW$n3TwM=5&|%zK#w5Rj~l++vL_;_xx2ZfV1g5o&-OTVud|dg
zY)p8a+CN(%s#Jb5B@9dFyIC8&!|9!kw<)+dnAUfkUON7RZe*>j5V@t@LFIjKzBWN}
zl6bSB?AXD@u-6mJ^$s$%>{fg;H**<Wy_z$R_0fFyxQl_NC8(r`t&yYbC$Wq!ZhWiR
zaQO>2)_CPI7Ur}ViP+8SHXAZ(qCX-8Pgw>@)`d8#Ub}%C2oieVUWG5$Z;wDPw0oh!
z(MS2V(Mt&eBO3Siyc1oDi4_JnN!?`_!F7~4=LJ<;7i4@c?GuTsp&oXOubJN~j*nRE
zr*(v=l`h&IUu)Y-G0<(qS?nZJwYZ$k+*tNcTf2SvjF$-h6dEDNR%>@OyFp@~wx8sm
zQ@KkHCa?M<DDa*q5JkVi|7}X6ooQlUmnxZR*#{T+GPIBvv|B=xJdNM>M<?%wx7c)l
zdn}q*;Wg|Gzyh^yRZNGE1Xt2+qw~^RE=E{lE|-=pha<-cjHj$U3eSXtS-y9ciA;Qt
zg#X2CIXogbtk+gV#pepYbkK~qZROc+2-nD3e6SS4d#n~;vgmMgi02}r<bB>}Fg(g-
zTDmx2YrO0kRzm_}D5S%J%DnQ0DdV>iV*NTt#-ch%|0xYIm~S%Q_BwvyfdQLqe%iUU
za<4uRAr<XM?#|y&3?~<aXTr-Hgaz#tT;9D;cK$2uv9~aYB8Zi$G+Wuxv-FN9+5PYR
zh4qI=trjcZ;=wL@A=W&9;v>?i^7Yn}O(rs$G1Q(|Z2z1sn<Yns)V;&#c$@TE2Q)Ru
z)6)`Bt!gQksJ*lfL=-F}Q=R&|&-+U>DkcK~7vFVi4u!dmmta<R;msWO$55HcDm$Mq
zYzuC++dJmp5`L=_vy^#UbXhfEraEBcG`ZP)uwPnpl~qe?xmmf+NiO2ax1Om+(c=}a
z!GBS0w-`sWU7FHRP;K|H)xhT98<o%yk#hLBt#8R=)!=YcP~pc5m_`)LD5eB~iYqs_
zj?tLy72jM<E{lrbr1en>*bVr_*W;KI;3E7^sdYFjoC{%b$NVtKVTT^7+2BYNw7pYj
zi4R5^w->W9pqU==f1#Eb;6C!>z+DXeb>TXZwZp;G`tJ%B5BYEu=0)FDN%x4fNM_s}
z&3d^+eysZRo^x@R*na2q(uHz@x%b$Uaf>KVBm(XH446M${l^6#mQ=@EaW_yr=YvCd
z+ccCShFQ?$1}?+o10_*^aev+*-pN^+qy-`EqoDYuWpSd~d>GvmJf>Q%JrCl>)WF6k
zmNOMMMp+hgneL-`A#pMMbJc=vS<;lS;}#5&f&14fUK<#DZv350Jd6LHkN$NrFqGJ8
zlLQ^XPt!kNF7<(R!K>C~vVL6sXh^GeVu@|f9?Xx@{W00`WXRZv(|^df><;rf5PNdx
zBpMO7vZw8Z5tyP#0>Vqx1X*<dv&0Kc;4ZffjQvQ21Vb2S)~HS*e}&q3vP0X5Ck>{p
zL|PgxQ2d3lia{mhxO*i0u!<pNN+=Dr*xkF}rL@3)H_B6SSLCd3U(-4g(^%Ggi&~71
zLC)Opd9psBf9m^tF-R2=i~Y+Wx~{DJ7&!b`+!vr6(LbBVOG$#4e;21&m~FS))*G-=
zPa}wqkl#vdk|f?wwVSv9(#)clt1G+|;%v%L28)#;+6VXk{7iUG+jNKVTH?KH*Mg`P
z4}U@{EuUAnkMX-Hs~^?DoM;li?~cv8zY`mcjZ7JLTz<$aU4RE4@Y}g{&JAw#$bdHS
zIsj~7y#89(M|b^nq}$-hTBu4^z2sUdUo4>v8BBVg%wyU%_MXf@bERkY-D?x<$;O<b
zpvBDRefW<DBArvMWHxmVUG9o}n*WowASDGwz0FLH(Dzku6^(`Vv9TH-Or_3WQek9&
zp{?wvKs0uy5PokudTJp-5ZP7duzj+zYp+MDA<51+GTg}}+XPcyOCY#AF=5lm{bEKD
zuYwz`>Y=}WkR(4(Y2H+%2SHKL8L4sdgQdlyUB8->Fe_=z_$V35@6;>_^Sx+A)!#{y
z%vNutx>hi*td+hI$B#YrfhqfD<l~3DlzhC=#`503zteBJi#)&=g+hzrVxg$DkD{I@
zg^onlA0{~L`RpH-+6gSVF_?>R2n@E#DgbDpEt*<Z@je@xlV4BeZ%k2#RC;@KHPXI=
zTv4h`bGcu3y~M9wU$HGWe-9h0vD0j^8a@f(O+_$Xo~<oWUu<$5exr+)S7gD4{3wVn
zgrav3%g?CuuSSpayqwxe{k^|Fjm)EGSivv!zjH;?(yL3=fBQ=%uWad!A{EMEQA^#0
z1YSLDC4#7>-9zS&zAcI<8lQ|mDf7d6im6Q*Pa;o%x1R@B!{cXusK5DSe-NiT0+q(3
zJAul<M~5YH9~^lh-eTUP{6+YH%@Tq2W2W5mXeE;-FoLq>=#Q(^6LkG!bA|cqcF*pi
zs1U^jfwNgMV;A~2jK#)8*61YsCi9QR)xxWa4oeYf>?DNCwZ`dlvWs4bPp=e=C;^0@
zKx^mvbur_;)>z-nUx7=5Qf6*XICBRQBO-{5^FAFbsff<KpkO!a8>m~+y7p>YE4&LA
zuaOFs0yDxhFwwq*qMEm7{5K*3xDA#58|>mq?5K=|*ax&{U5}%6AKyg@FV2-FFZMsS
zrHF6K$F#kK%8>ZN{<(`uWDrDi_jGHzJr(KJ|1yPYc)l)sxy8H2+vzKLzZc?{M!E7v
zF8j7?Yx*gy+=(y$qRI15nG2g2=Obg4BUlV=;xP}#z2<_JKj<^R-sjvH4CwW2sL>TH
zBFbS!1ZIVePcY_hP7Oo2*3+(G-We}hkFa4=pbOcJV<M%e|C+Y!zUqG{n|J=*>)qtn
zuvYEZ1ox-boipvgxq`t_VP5lxlooZlyR(J3H(<AZF?Y%7yd%9i@7`?6^1pYM<8}vA
z(fcoCgsMf!oK2VWC%GQAgevYXL7o?Eii4GU?~V&J?0;;|X}e4MzW2UeoiA;v_Kja^
zfO*44nT%$s<HJ_iHD0JS+i49q+(`zs=pvVW8i+;xuYtKK!0_&IS4RuYlSNh0LB0N0
z@7M}4b%hGL51O*{(vrSeZ6D8OHCSs;@XmhzeVhf(Slk`m&D&IY@;78t2t{dNPCng^
zl3t`DCq?X(3hcH>ORIJ6{H~eR;2X%NKo&enSZ7_?BtrJ5+S7I!1*N++VP8<HIBoPS
zs4s)hk6=@{5NU`^H+?`wcP3Bej!F{iUVeW-Vmz5k<8mL=To^D!#+Hw%%ierPJ88H_
zPv82kH0g1Ai1s0S9toBo^Lh+BR#Ex!G~`x30&(~+!!jfFHn1X}Pg@a3JmuSLTQ#Qb
zpQ{H2%o6CZ4m_;3gUII{#r0Kl7wZMxrF?pq^i$O~lx>HQg7%WjlKr9^Cy2f+@^BqY
zbx<#R&U+j$A5c+HBI@EZBVkOHH&Z}NE>JzIVq7w@Q)x#`<VnX!98`B2r8m%6Vo+<#
z6|brd_*__wX0{t1GqA?_MGCa@sMxH?;yyPPF1TB!b^Hhn>~1g2KzO8);$a8uAH*&3
zg)D9M>q1*@_xCVo#(yS{1NMP%E6V?!Zdx45c`#Zo`Hos^`dDktCFvy1sVN@{zyDdY
z>><Z7B&4vz?kkf60-^cx)}7#;WUQ;|Ust^A{Cc6ps{`J_Ho33e7$a~vXbL1#t`aT!
z2UJ5oWg&|^mMJOh+0J)(WRFNE*ofb`(PH0t*QCEMv<$k$g`L*rpQiFCT-S{2>uUSm
zw<=5E{Hpn^91&*!WTH|0=HF?*gcfcA(20pesM@XX<znYR-nMzrF*R^)xxTPuJ-mhU
z$llE1dA!{69o)n5LZtfsGN@04B+k+hUE_rgg5;a&za<srs{J-<3Qy6+aIcv(Vh0~)
zkKgAXo_BfQU6|#09=7DXY~^inHrqTNHErF7_5Idg-n+BOIH|92e|zJ_nvaQy)}E~B
z(h@~J+X?{0YlgxkKbSHfz6PX3+q1^PZ8&z^?eZ~;#lzEVFypvJwzmp*NjFhor7Ydu
z;qg_Ml8d`7>U`!J{-r<=sz9Ex<rB%wndw8n3T-!>HqoW|-MF)wvnft=dI;3KrdhMf
zMAysew~LW;pSGKW3p2yd|KC%+rqf%cE><%ubiE&Jx1%0MZ=Vj7xf*$_KBZqWak)pv
z4j1_4XAlThcpZNinfEz(+YMJ<jd{>q^T3Cm4}mOujeA_~ly2A-OCo@BUWkDw$?;>`
zNAbhK1bj9N)%iBl%U+H3#(W(f7`9(~6|64(y#)u!7m=9$ls^wV6HaD*K=}$*?`*(r
z%PAmMv7h_OPY`2waY<I!<1y{9lkvg*S2VsV;C^W>H9yql*ma~+oMt}#AG;;>^x|`8
z#R49-yL6HlpGSwi9Z64GB2a6w($eJZ>gJ#mxlbpK{v}O&29pecI1p8Qp<h1SHvO4N
z_+F)%xO0841x)wd$&8Vw&1HJ3w?MV^H)!Ha$X1O>IO*|zpZSM6>*QdWiv2AWGUOWT
zi-U3Fb<QX6QPsrlDFCm9oEyVYRmw8*Eij=nt7ZU<rce$)Tpe9zp*)({+wiPnEysrt
z%KsJKDNEN+O_ku)jJ2#;k}_W1r057%=`a*slt2m_4Ow4Dxwv;-?t_o4?>G2;iM1t1
zHTq@j=TP}0C)-I{SAz4qYL-K^qweE9?(nlNI~&!^GA8sit}QIH$3qI@ShGe{waqKL
z$gP6l!{HMFCb=eGSRp?HQ$W^k5NfTr!FhVBHe2dKhrT@y7@j_LUMO2X!U<qQcl9^w
zAMYGS1h^`W*Fliy$>29w+j$DuOrED%X?afH7nQAm*uGMru4R&AA=<D2a2dhUa`0r@
zkTJbBK6OSuwLJ5|P9q(YMcV`M$U<Wo(*mXYZ8CH?GyY$Y{x{kJ@CmLnT3u;ei(W?I
zMzAO`+bv3<e%#6GvtLFB?dSqKV^E+x%tEbya&0sX)%=q7bl~KNG#gIFFumkEV<7hU
zq)iJq1xI5@jQ(S?U1mj9Smdun_!|Kx7{jS*z}2gF{sN8ft`X>R+<Q>}*9|j&!o)35
z{$bev7y}Xb>i>Ao|77I<=gZUAg;co|EjBB;+t~N>_f;{P9o0(>CsVsQIY)8(M+{Z0
zX0uI|msFKEtT~ttnsz7Q&HG#`N0?_r9`7GV%pEN$8bm$wuu-KbwUr2qDS>DJ*c!rf
zo^{1mo!8969ZL|;%-@E-i_s)2-OJ(P&$+jIm%hVOr64opW`*}ty|pQ4f6VyiOYbe#
zc=({mBPV?e10QS2MJS$rn8*$>vA(j}V^5F$v@VUb2^BbA=Q!jU&j8~69bkn$Rr&R-
zO%6w?WLzx_VTtLj2CV!KGIodUcx>y$1FafezYvGt?k~pa`aUwZ1UZA|ee$tkNv=Cc
z{JGZu?=XZZBj@Ofc4+btE(a9d-5DgAA*&B(**VS-x*~(_)GHAM5wR(${3Wbl#jKod
z%!O|u`G9Q+|2vP8d@VU)x3xNg57CaY!$@wqG0aZHEbB+A(>?I5v+>m<Uz;jVsLJ*J
zW~+@y6zO<$tl?|lNJMu4hC=%f7vE5WJWMT(IHLy3^({EaKP?YB7ke1l@pqa|SOJ6V
zP+$pl?BYl#$__WuF_p)}{q`1!uG%lf;Sa^0L}IxMpnos@Sn`)SbPo_}$YtIzMIKdC
zPs>vd6ZxC5y|K}HBV94S<<b$`HYa;Z!TZt^q6v86x58P{wkI2UoRXG>hZ~Pm)?ev(
zXVwAnw+Kn2g}4PvdewlLVqV+5V`7&VqCwG_bUp~XJRAD!ZJ?#2yhSQ*$+5}6kL+Ty
z7n`c?qP19mF#uIzF`@8+FFFm+?x5Zv7Kf||U6sF$ie@YBg?|s!O5Yd)@?QvRXPw9W
zzAFH=P#$oWOh*kT(7*Ed?XvkR4WNH}In6(}n`hUC2FjAX9nI&<5ll~Xlj0N(M~gM9
z7W;3xm?1qmFD1O@Yps*F!w=>vJEt4sE)}`;KqRl-pB?W8e==fRQHbNR&yXn9-CPzT
zX|2+4M8JL)^24fW-lP2u?B&<ST&l#FFZ)D-uFKJC<Kab1+ygzELfogXQQ&I#70tnM
zy54n{`>H0Fw|hk<)fQE<3!TG@ud#0no9o!?oNKC!@}4eEg2_~&7N3qT-Qsh9Iv>z{
zm)0<yr)u#7_;6@3bvjJz+ggqdL`H_^?hjYdPzWpB%A2F0&w3<JB_EHSqr-ppKXP7$
zrWt>;AauX_U`)J@pMV`1Q@7$z#}5b8W-NWwOd}B|VY?aWP5mhB2UD^`MzduiO()As
zi*rJP!fC{s1^J?Ot$QRN*3s2*LB2-Q@>P<~K2Ce@@ChWHB0qvX-#@OXT6O<z6!)t8
zuHHuvKO2MQPf~XiwS5gw^y}|f;*G`8qVqEEj|<{jf0-d9z;+?Hsd8<$bJw!TNAoog
z!lOHQ@wE`4mlgY=k-F55@ok?~VYgZ9ii?83DUY{@BQPW0Kt&6>3A?52TkfiF>Svqg
z?k5mH0g<X3N|2TzLCY-?>EQvDCi4-yJ&q_ZOB?^Ly!aEOJF0~0B(D@5kKF=wuhD|l
zU+C#Q8yn1*!P{yZ%n10gV=xIA)5(n8w~GulmX=Xr!2;1WoHvT$;rkC#$)4|!vFT+X
zw1r+D=B(0|$vmpR_ND*2G9vq;Sw=C};CZDWLZX{Kby1`C&o%_Y=2S+BH<R2XLYGSE
z_=^|+ssvIdqk*Vve*U?%e-cz*yJPIOZgw1c$H2N3J9pcuYz={bj%-@o*jDZLI~QwC
zBAzvlhUpVeBJ9G0MuPyI?ZP@V^Kz#H7sx;!p1xk4&&-S`|3LqHbLRJvK%QV=Qkb{S
zvFK<?FJ<lVt5;392rQqy5=su0>1Ji8iLbg#!*Kvdbu_1a%bzZ^ESUW(q;MjVZ0*f;
z!p?7bS9101myQ#T`rwFer2+tPS}~s>qQQ4@qljMnW0?dl0Z$d)m31QSYb~j2ibvAI
z#x<niUH!DZdzYp5SBs_*horD3f0JK!rz1mnfewB4Gb?v?I8aoYjvl{!QI?LHC9;+~
zoO4O`Ss;?Z^Aldl-U@c;ZGbzXt2gmFZFi(IS+89!*Lk<F@bP}#ZyFV(^jK1WN90`~
zt=Tthn9XCnm;+Qjz8G0Z>*7Ne;3SWus}*&Z1bGmhWqu<dAv#}%Gd(=ndllitwoRIo
z=Y;_o^88@Qse8?9I6Pwi)bibdeYC$Yfnv85*zhy4D|RdlSSDYL0)*rCoha8?w<DT&
z1A2C)NtZt0&W>)}@#IQP`74&+aOIqCHv=OPy;3=g?ZXcQU_r3<9`^sxN#N(Sl-g5+
zhm%cbigdiDjZN%d1~Qg0ODQFKeyoJR`kd#k|IXA-l{D|)&PM<Niw)F*#IwjCReL1}
zYT#Hf>PQn`j?{K~c<F;FKn7kjrJhC8ZhL_tqV0FFyv`9W-^|cIJ!3w5CB}a^^%s2A
z!kZ-9CkOI{{doQP-#T1pWlE_zQTCS9g~NcX{+SG4b?_OC`xuy3j+4X;3IK1etFU|_
zwU)#L2MPfH`45d!QU<#gu>th)=$7BR;X|@wqUv5<r@H;<-%;IVy_9xGyk!FH?40`i
zh&xk1X(J4^;^f7B!0d@?mW)-O{}o#QN$MY^K{-1A05(IRQ<dU{w)@%ijcKn;(QdbU
zS>2MQW2XvUa)ii)_u`-rnYHKB?wPOvE8o>{*St4#(@XuD6t}M<Kjr}^p&@SEy-a^s
zicMrbiSmDPIzRp-EIO&}#9Z*`{%WHma(;pwaXmQj(l?ztiZhz)?1y!o?2<o%oYP^B
z6=Bgzb0(P_rjGYJ4x0Ex{g{L_v_D*Sp*kJ(f4vqey-LLa(*Ttf96(dRO5oVI*kvHS
z)Yrni5#mxk52&L1ST~|zsG-JRTfTf6_;CA&S~zwWFHy*R*chhF3+&o6)p-`N@F9?l
zoSB%vtJ+66h^$SOipE>?*&Oc(2)=DpfDruZ?u)O3FB}0j^bIi0vt%bB26EIJ=b_`j
zOA1%H`Vt?X?rOD8woVkiMxan81My~V*uaT99~-NAswewgQT>020z-bjLUXyB$=(5F
z9G~s*y0#nZkD*LO`6f}2YVh8EaNq&r+Ak|aWUE>bp(T*mSAqFG%OgQ+P>a@EZ6~h^
z>o?rr#(V;ElS_BiX$tkX(wtspX6DCW{Fg{%=6j`jPqn|>OY@nhl4Zk*<q)K-4WtE2
zT=Xw+#p}(Eb7^W#JwAPHY`QOYdJj{syXN6!1lUnlx{1dr@tqdHzW4wjf0jn517Qdg
z=!?1^@thZ2vBvf)uiJyS<5q(c)qhhM#%1R}>Uml%abT1~`U|=XaVE#c%g7PeQvGo0
z)-=6kkfAb6jV(L}fX;C^-057|Q>o|d{FD#WUvE&jnVy&A0k!CR%(MCH2e^R#JaS_N
znggoLNCg_gLo2g1SejKFNA;^kwYJ$B9v)jOBm2@Ez1!lKMa;9uhM-!^XPe#-5Da_T
z&m`v{tLN!0^5385dIR!-r_<Mz`Kp^bCju;?82igI@{uap{k-ljoM?_CwNSE!ijVV0
zz3^eE&kIb4-5Snu?08)+a0#C*>;Y-^pa!G+!0pq(t;w3<q&Bj#lW>Mmesfu0*hRc!
z9+d~O{*%Db-?Y1~yh*E<m_qESjvJr`{%e1IS9+h$W$(57K`FKllA9RK+##=SwgngC
zEztt?2qCD<Mnjl+!{OwtG*ENSHW;<0Z9W(_c)JW{tnWY8)wyTyk2BfSs+*cg7X2EQ
z=d9!RQ!e9TV3MPe5(d$xXXIzg%MsRqLI|_E<WHe8nFI9>+r4Eij&q(H_&ItXP8kY0
zhglRO!te5?*<|T1*jqku^}AWsT}=UkmNZ?_-w6}<0=A6MGLhUG6ZHH2?K<P9jEQ}8
zDC&66za{7|^II6Mi%l4p#)<PZPsg3p@u<d2FBtkyiJhz@lgzafkLJ#UM3Wkm?;XTj
zmT!p+XUhU%EmKC<2#;9CaPjnKoFz;fD2UKSp1>Vu{yM{NH3}k_`}6Qyb0P^HP783l
zsV!U7SEgrT(W^G`bA^LWUP5e($^kB;9^Du?Kna&UdyvR;Wg6I|skPDw-YJr4w#ER(
znm0{dZBp;MSFJjxQVq*<YpI?-@nYFvH8Z-zqQ4`s2@8mGHbH9ELNAkL)kb@!ZvOzL
zbaE~m)6AOw^82--vSzn$W^!$Licpx+R^5ALo10FnjySnP4)afW9Ejf9@0+$`DLTa1
zH`e)E4e_21xXsPBr%mY(v^fmgTT&&@_u<6rf17?ToV8FwGlSDB1?s>OEw3~CeL_5B
zXD#8)VY7!#1e-|^Uc=Q>)#3wU7FH2@(uewe0EqooU%ubdPph|-Zb{6lZ$-_+c=Fu$
z6$Wz0#!NBZb&#$q9@yuSlAOHCLTk97!#j{y#Pj;7dm7`?qvAWeS>%51QL`21@+god
zr-pamJ8Ep7Y5(wTr=3<(`G1$X<Jg6Mdr)LDm=W=YXEqk0J%gk~N1(175Fo?^mDiec
zM!sZqrY0JrI6@L<(R4uPysw!?#(=sMP};O-=5t2qzi1Q!!exVOpqdSv9IMm3MDf9y
zuD5@Fv(3pxMvK>!09T}tA7WAoy;OJj-=}>aRYYTSwU-{%bR=a0zf6?!E=9~z(>@6&
zK8C=U>}FrAk$gC=0rbZ&^TqERW<P)XLdr+5L|<EF)G7sjxbCO{#0Q@#47%PgmKZ=3
z{#$yRkDr5LjjD42;*<2=GR{(ean|!S`bG5m{QY(6P(ZIo*^`Hp{Z$X3(?(%!(@IPh
zS(4dY1Z)srZKK^fSh_<tsh*Vy<*<C9z#oj^8y;D|LHPEHTZ7SL?TeV#u?h0|JxM+F
z&v$a-Cm}G-mJe0>S58vXkI_a<dWy~KlD|J@-8tEqdfE`m;leCV4yK19wqxDYYM##e
zpA+IgG$D#dULW#0qAVl*HY$G&UF|!Im(?`qu;GVQWCXSgr075=y$@$)BfPSPQP`6|
zmkxU2MMw5&0g6{+u6G1DibM`6IA9M_94K+Hl3+~%XeZ%rpotU5r(X{R0jrbdfAqT9
zpnHFA+gctLP7gX6g{}tQ-p1!$-ZxzoaYY-zy(yIbD43H}_9niN20+k%wYhBhaUy~a
zogDY>^k#-ahKEh#r<;;>HWpeIB#Hg91m44lI^)OV>+6|oWTMHiMVf~DxxLd%!zj=S
zpkh(X{r2$k&`JB)d4s8sKY94&!OMIc0Wwk9$@`W)7}h?)^1ln8<=Nkg*hiLE-ASh1
z#JxLb_z8neh`oAU$Z!1k%|-|OK|tqCp=r0pn?ez<JMjL%A9Ps)>6LkUH%{PifZBg>
zCU|i5;<{*hMp}gt1~raLsrM+_UiP#R8v9x7jZ=rGn%OV;+t0WesG8>QBy9_Y_t~Zl
zI^~ok8{%#h2=bz$9n<5{@-Uw~H_y|6Le!wL4de)@>mmY=4n`dYz0}{_)>{s=YFwOM
zEl#?tC;@{231|1ZA8T%#4%kgr<NrIAVV3z{mQQR4f;-v`U0kxg^_FJ7eQ^{t-SV8y
zXm3N_iycbP9)oj|ix20U7BkXB+=cW-_0%QaOj|zyF*)}*)4tJ#-W!cbMM=t4zCCQp
zgJtNsV6X8Jtht!{zR@NyDX4FCb%BJYp|$utVtWfdqZeU2EbI1(r&uwtRXrqtYFOzB
zmXBU42b3lAKw;WVR{Az13Sfx8QU76x+49QGBSXAKJ;z{sLe}vrhwve$A28WZ?T0jZ
zl40o%_w8W`%%vOGY}C#$guce=W?{is)$3f64ji=tEC9M5H4%Kq&iJkIu;J2SIqq_p
zET#ZpEg<6wtS&1kL~O^oePwi--?(*H8i>@5m)l!C{Y+^qt%3m}fK@pR4zx&~-%Tn4
z3K&qP%<@>;9FXw4t=Pm@W5{x9VwAO)YuI~E2Q3T!s;_x!IYW#neb^pe)?OB5PfB<}
z=Cd?cb3=Ixf<5(D=sJI!eC=YhU4t3sT0_3%9obKJoO19m`$zf=$Z#dp7+{b|Z2r6y
z59t+iJEcZ|3Uo|#`Ca5`Dm8`zLebx_L_eU{g%($mR%g(AUWX>81#~y-+@2DhAxvM6
zi6W)IZMj0jhJJwfS%`LLop4suSKKl+eB<7+(BjN;fZ|r}%^t`{Md5%XK38MZ+U45B
zJ9wx4WlMja@ZtmP6zl7Lujh~V_yii*S}FWl?=Y<?z$7t%iq8ao)mYO-TKv7)>JHXS
zw*82~hH+$dcE1D^C^9^n@|R9syt#gfU6_0MUUmp_YUkc}l70g#@{)V3^x;iyMfwnm
zYA3a_=MD~U15x7*SMra9r4;bx$!CXWCye}&Pl~tvtI~34_NX0=!BN6ix`(Ce5`EOg
z)b=P~s3cD-LZ6J7-vx!vMCHfi3s-ysvEc5wa^C*XQbliv&wm#^vna_)&kp-03=U|&
z2G75Ybua2_)XsEv-FxuMG7JUa1bo}yJFg74OUMK?s9fz0IW^%ct)`lUc>!S=v%>u+
zoEABdTY;uWk*%G0?uFLRoSTD0pxP2mCI1M=hH42X)y&61S)UJ(Cw}21G$v}NTbx#9
zAWe(#@gYCzR|HL~0gxM}f@k1Sxoc7WnuEFXD+aoYlZQ~Ft472{lV*0DXxFhkiB784
zj&g*o=d0(m1tOX>K3a&hh`bvh_95};V}22-4}OCI(8X7<5f}o0$w=mJ;a#j!Hv$Py
z_7_0g_Fu-G3Az{&ygJ&?9K>{cau|F1&QBwt{)yx}1EAXeTLI{RkxWjvn?h|I+p#2&
z<Xj1j$#p-G`2F7-7Oy@#HQ>u>pqhEp8MN@z@L(k~ZHWAb?ZPr(Py<vG%Rj#bSgrG3
z(usssK#OWmk;PvA?+tE$JN8_JmBy4s^Uiz%na6R@KrwAWV?aUsk75bJS4Gg~egJ*#
zSDa_`l~#l`k$j0c`QPsY@KkZ7vT<6i-TTsE48EmAeu}duN`zwnBLgy|<gd-bfC5FE
z{3M3~nsM_cog)H<z*Sx#&h+0(z$loBLw_SttNw(-vrb`+(~6Y+5`3PnF8*UFBq4_q
z<$DKVi-OB)H;g%mf&Rb7VFr(8X7%CSX2-g6$-JU;*(}_lTe2NK`NW8{pdurrEc>yl
zpAn6^FL**iwePkySaJh__2YenuX%leBxJK4tI^K2n?C0{PWFefIDAg{YtaHx8Cbgk
zdsu>nL=N`q@rz#zQ44E!C)~(q=KmZGlNtq1e?fp_VZgyJumhMN{?$z5>6l%n3#s1?
zcONTLrjo|MCINwc=QwdFg5)TOpogInYC>Nj1AH)$2X+Dz;1hO^?L^wXs&oCS_v$gt
z0G|`eTFC1&5aNX1G9muYlVY?Wo0|w)c7;t1HL+EJf!^Odlu=z|U;lZ)B1Q^<h3o~`
zToQEu8aX%m_jYv%btD5X+cG`h!a`eYYggyE2sbAeeSH5r9ISQ&{5QY@@|HBO7CDXV
zU4XN1TjLC6`7-@|FubU*x=o+JI2r<Vfs&ns_yqAMH?O%)%+Mvnvvy#W!G?&M0bl<8
zK|y}2sJZ5>wU-aRY~+ml^&eVVBHKFg1-zihJBIpv_JqBxY#0+G3kGV$kFDS}dQ@vK
zF-`tD4=;WvHDcFrSO}_D%dtATihK+NQo&u{)PMz&O*cNVRWvYg`Rpa;<06eN`3T#A
z$qc=d-i}#wgd`NgLSaVss6q(h%;bv<T7~#hipwiK2?OLNB<XB=lzx#_5GE#to4{h_
zPJ=w#$|Pq61S!!1nompJ2c=`dGOzI^|Ip3UgNHM(Q}VC>kgMr|IpWoowe<AD>D3H2
zcw(W@!Ka^kFyj0jT0Q%I=pd2O-pi=F6Lhj1JXJTR@DN2&`MHAHKQU39LKv^I;DF6$
znN+CVjZD>G$ezHFAOWeWbzNZvm)l*!VQDtFXz;5$bArquU!@tBiV*&}zkrn1^2H{H
z3$+Y|SPU3KGCPnFv;!=`e`8`IR4#pm9h&mi_VSW#ZS4ab2P%Ldu~y2q1_~rOL9f4a
zLZP}!Jx1I1s3F&zo8p=5Om9XD_qZllQWrFD$lp*VwfA`9>q1^C6UexQ%5e6Y?<%Bu
z;fqL%n`XzKoETaYyGDqIFg1%353Q}kGi56)L8$D`W`xBi4<69M!aMFB4L3wkS(uJO
z$LJsHtP6yIDZ=4TqWo0)1R)10M7dm}0FD|gWz2p`8bBE9>ai;MvEJ)#pt2M;9Oy!f
z0k@NmPFu~hus$|gFuJqrT|T=VWdPyi{?EqLT*h46fEWQ^IEF&#+q^G)E)Zw7oQ+>{
zUi_4~?LCh8S`aor&A<}ksb9}_fx}k?CAj|GQ&h$Z3GYUkG~N&brlmBYB9dz=K$#`Z
z1P@Y;q-JGfrc0xym-g^&pFz6z{~D?zl?j^U{<Iz(#g`)=x*;ySIsX+np-F)AGT>Ej
z0Y1O9(ZS!y&H;jmE`tqD2Bt*3uAUFhSLS#h*P96by>%_y<g==G#9Yj;(9=^z9(dwV
z9Z)7{GZjplq#dVIJTnc>%+G!Sl}XXznHOY;0Uq9Nw1#%VP5{Kl2yD(K@pgoso>~3U
zJ#h~aDkGDT{|Ek6fVz7^A1|K}TRIJ8SX+-iu-1^@>ysXKM*N#G^+rChC|m|6+rIvO
zJQd>SIUXHGN5=lot!Ka2C~rB}(eUpKAq-56J}_LVx41c`n=`d8%J&jRt2=QqQIfN$
z2JEMJz?MM0VAc{|sOlBzC;yCF_!QMYbajrEdo3MxoZxT7swhMX#0bN$e7QAL@dT*n
zcT`cZ^eg;HZ$_pqHShPaHsAO{N<u|QLd7MXZTa6X*W>APbHN*LgQMg1-PEdGxRGg0
z!8%}2MSq8;hg)R3$UG5<mR1S!1bU16g~Ptcm9r@)OB>;XDg!>0sMTPD5&DiPdIllP
zK&^aDWa-~~)8DeCukYcBo<zT0<#;8eSMWTlA~5RWS>NoJq_vp#oCyM;brmBD_Kc!t
z<>jugH&wZL5x}AO^%;!PV70LNos_y^p~=|HW9MwpKkJ}pTtACeZvK|Q^!|g&^IG#5
zO7p|#b9L?eQ3qK6cR<|8Pk#oCwIO%dof@PRRIs4P4&cV6AKNVvTN0a$msX75wSxEV
zn@rUy%@@2fJ=cQ?J>(M|FGl;Jnu@<!7ra^d(e`T6T7G~S99PzrNo&$unjbar=-5Oc
zfsy67s5NO#Kc-`K%H5P8r^E_knSOa}or+QZ-GAawZ`MYMLl-1$L6e{JWAjTzMjcL1
z{aW1cb_#gYo1;>YDDAGW2ww&;?j{a9pwpVx-5}e_%HXMtr8VDVV=lO!%uUi~gbrhc
zg(8`23e=y-$QS}exmcZq#$OM2XR5x+$?veJ?r*NNZz4GWcUq&sj|&MbU_T|oj~y^)
z0PiNvVFwaJp>>q-Z=$L)BKu6X>||wrg)!6af3HuIR$~oO(KtU%Wn^U1h^Q|e%|Y3F
z%Dh}Vvo0omUdgq4_H9vtPUC~h7}uXH4&u6C@uOQbQ^tEQk&H}<FZ+A3@x#4r3@_HA
zqG(OwS2<2-bH0=|`#4WHzTKabBs_NJDL#PB*(*aB%}Ep;(ccD60zE+?Ne2AN8FB#k
zDx5BZ6h7fT$0r>sQ{hjYN1Z`FcQ=ll(je}j{KJC%7gAlEXETwuzuro^QRcM{w8f^N
ziL%8e$ET;op7o8SFbhw45s^};4{W+#t2-Dl6y~N=vrd=J1v31Q=sLdHd^WuYAZTvJ
zbtRt)b<w>7(UvC9@SAQ|wj1MdV+^D;-pls)0WsZM*BUEDYAekM(n<5l`Jnm2y-nGm
ztbx?V02efQuA;0(icKgT8U1!iER)%Thd^=&AXIL6U3vww_VTLNMV>y$II*UpFZ0TK
zX8WH`5gk3b=G$Rwp(zU}TJ;-nt%JOPHcm{+W|nPmdW*-Rt$qYsgZ7IC!)NE$HsT7<
zeBg;xz$T`=nEC@HR{5v|u_N_z<!7k_0zj2KSEi!_LVN@Dh4yimuEcoO>&Lb0OCU|?
zLz0f<qLs(15}0Yo=o28*TnY;P3|prezp0V2g}fdx_xPDF1<?c(JTfiuV}JhnT+vGz
zf=>{VdbCDiV6iO@w@%eGM&C7ojUSr=?8sK-8KYg)pQ*`rOyP9Tz$FDJ$C0j^3cWfY
z@6Xg?51`Gz6mTWVE^_yH2hayc=ZW89TpnJoGH!Gn-`~C#DlAVxL-4yG*5YSBo8jOV
zsH0rCm_A3OqPRvWeS8BQ5mTvtNgVn$D+6XBGob?EO>csR$}+PNeUDps@D&vYye86?
z+ehi;rQ5@jxScJRJCRwY5nlAgpp~wt-zggEqK98k9RpL&%a+CkL57L;$U+9(vQ{Mp
zvB+l}K>;*!*PQr~5dRpMud*Vzsj7U~q_Wb^?qk+EVms`mFOBg|sKX33=+CIA$i|vQ
zQa#5jY1f;Spc#J`Hw!zhAGtn^{6wG}e4ph@7dL2*O6--g-Q{U_GLRyfpbUi@Mx$OK
z69&2MA}f9wX)si15G&J`fZMgMcOeL~;a*91cp^7Do13j=<??Jgs?_cT^SdNeMI$D;
z`x4_7{XQT#Ga{r@Uf0yv6`syQToO=eY`_xWfQ`6|CX%h(>nnNs{D}O+7LVkp$i(RW
zsrSD#Lm}fH$pfF*h+fJaq_7n<z23KxD~=lH3=lU|uucoeBGW`sg7o3;{BD!P7l~{Q
zY-M|^6XcxN*&~my05NaGB77BqmnPwE8lz;w$nYC?a#D&-{y=xEs?yW1qfBIqyTU#^
zFVF2Z(8Eh@>7tvBg<NNxv&s`t;Ew3s4^CvI6h$OWc)o@NolQuKLXHNk08jI4&tlB0
zo(4i#2jy|h_6n<XOJR@FORTdI_Zgw5z<l5wJJ9gGz54E0QY)hHOAA^i5Il%>8Xnw9
zH&&u6#Nb8kQ=J+aN&D<y0?-*Sf1X`B%0|})Zvfc?&<Oz2T_LNcT%6V9m9noU_~hA6
zY~Vk}5P9@}@`7qSr|@!HN8lPgE}9hYQLEC467X0ta4&Gq2uK^MM3J91R;j*fUetM)
z3j5IJP<i_khZbiAb>~lb*qH^$ucqO@0G{$D54LvVa(8(^_QCB}YAQv1e}IgPOkag@
z_YdzR>1aCLdIv^kW;q6YTr@9AdU|@|I+v5R)>cv5-6v@onYk*{;nj{nLyWaC(_op5
z*?KkC5L6Vb<<PIWE@m`zyu4SPAw>OgELGLjgz@Y$RGql0q(>h=Q?_)+u8imie~DZ3
z1t@?WQ{md$8gL@Z^X{O=&CTt2D+8B^sOEET)KIFRU+Z6yuCcMxllAWM3gq;*&*tWg
z70k3he*6HA=~#~E4t9l+1*=8|L!n6=#$CQhXpN1He{zPSB2z!k-d%O|d>ziD;o2T*
zag!=pvnceJI2n<YGcnn!v)^{RIxNY`qG2_R5r)z-ndOm3*EN*}_-E#C0PVO~KnwAH
zxR^fM9uDR>M(Z`I$jI55saT&VP)Qu<6C@Ld3?}m)Ej2YHCMJGb>&r_Dg7&Z1x!lP@
zWmJa0^+Ht_-R@mCCirFPosV9Vkmrhn4TU3&f}whECtjH<x)S0gPD?1<0Ifx3aUT%q
zEgPH7Shlpt{V`%sZgFu`2k3R7T8Y-h{&bo5)8k6j-@jk&L=5F6ko=kuKC;l3=0@QY
z$O*XJp4@nx>~x_*QO{iQWg)$t)cyXjam-!$_2%6d7eujje)F=PCoFeBFrj&)3cB11
zyOnqY4o#FUy3*J2#Cnb7F!*)h!Ec#ZUC3zfWZ!v%0s-EX&6~nlwinHR9~oNMH=zSz
zcV2GxWFMQEA&$dUr$vC>ZpQ1!vFUql4J015JUt#RHV9EhB&ZhT13YegeB97Vy;R2s
z#H!!y@i3)pg=nmzO+PXguwS#qqBWMM-@;=xA5JRhb(8da-uXy&xThXi;cS&@(cWHA
zt0JKqT1RDN<x9=DxVYP!n<kGNT)oZn-SOika&mIfoIftq$jHdNHcQ!~X2Qb4$4u-4
zy}i5kV6nWH%R|2{r_V1gioD4Za!CUOB?&qRU9wM<>!|yfJ98dkP=!wj>TN{4Y0Z@m
zrx{$FjehN|8a4W6+nYqkGBV2v8oFK?3g}+9UD#bj_O2^$6Z|!i0ze9Sk3&n3<@g`o
zf8Yo>8EVITD-~Ks#!$$l@(xVF_YLS5&sAG#o0;td;n3Wk?;`4Be;C<@`KB>bQ?CK8
z0=WKsgY!}5$H0}RGa}}hUsl~J@)X}DXZxp32NI`?oR6lg=FrLbEmp?}$f^sGr6h)v
zMOuLNPk4AZE$y1S>I-D#cJ23ETr|JoACrYW1sW4$V(_sE?lT-69c#P>+Wb(Q0Vi}l
z+hWnK)@kwjz|Wtk@uJdVvhea?zC#3S2ldCna?A46R2((a3=b_WZJS%Y^ARRze{b)b
zxK0LDz-i>8e&9~h{w8uWsjdh<AK9s;r#cFg2m(?V86}?g-HoMDxnH0St6)ocegeL#
zmI;rC?c-9_zT&S?tCU37zxlO)=7bp|5uRPeVtP4-QAMlz>+b3*c=rb1KVKBFTwzDo
zU?Y-7%HShb;k6C~9T`y|jq-z=s9<bzGMW`?NT)AduKv@b;1$)_d!8qc>GYiF<Yb$S
zuKxHE?HVA;f4<m3jkB1qKG^7uPEAeCL`aS}K0XG3VK{|9nlECstIK3(Br}AFt-8AU
zN4g;XM{)6R3PB#1V}ox5b!2!F7wZusLy275K)4$4iC09yeBb*a7>a4!q<wJb<UAhj
zkRc72zu5jcV8w}b{n`v0tSRVjv6uWqqO)scXM-zW6jJAUgH!R!^@f~+M%Jv5vU$u;
zS_$lAO$ma$^;sP)ZuGFzH(CS`H!&#)RSxlpI4MCPRlujHkU!BEn?QqJP>>8bf;raN
z-j1N>OI>(+YNtWwVsyArcdYW7FtD}Uu;a&IV#{%2+|V|x^4quHwkO%6xPnf9YmGXg
zfBrzD;Z~pCsh578EF>sC0QOtSC*Cg&E9UHcSLG*mM7JNjZi5rjIQ9}XE<XN?GECXd
zt}a{xaY%Orm7=^nEfcmfq)e|#Mpm}L^Nz<u1sL}`N6Z1fq$^AV1~rOjbN?99%ci&v
zHf3bRpawRb&&)51gyg7%qE{e@DX_EJ@xtiy7Zi}h^0dqP#sQV0qVy*j!_S{pX_+jE
z6nAPE6m^P$&H<B3Y`)ji!xiw_UV}IKCy=hL=k8d}#~<jn_Vz{Ra;PsEnXXA8FCOG;
ztQR2=Nb3t692{ULY@P#O<mvuwYcM%D{xu>Shxy()%CNo{D^qLVtluCb@p}tMy09k#
zB4S%mZf>qho&v=OH*N=rp!1<WGMYe%Y&@&jI3gxuXZzFL#Xhh3*js)5<p>)Ua`wRj
zmAsYtiK!_%CjWfy_wNn4Bp|Q-^B=F*v98S*8eQ?4#YQMychIs@z^}m^E(F7}_cwB)
zMsJ)T>2R<<HijNHhR~p#58h{wYY*EJsra#BMe33~<2_v*tc<ho*eAGNXOn{4rV293
zUxH{$fI~({&3Bm97l@Bk-#QYxOvS~;3ui@Tq0k<{&E|fYBc>+i<>h7Px6+o61`;7D
z2I;Rn?atR6LEi(XT=L<7EOCvvbAN;Onja4j@A?g;pwn02ElB6Xh1qH=vPOx|pWlW_
zBIfxf23^4k-`yRxsH&>2-cd72XfRI!HqVsu?+O9VD^^xjMGMvZ#8wR<WLe3BU$O$i
z=I-jKx5JotGZY~h^*);@QDjZ*?cr?}9IVp$3aB|g4fkP&gwtTLMZ)UuMN{LV;`Ehb
z;u5p0KEIkzw~*NUdw>OV`(nnb`R_JI>w3Q*002();%M1>rrfZcX~+E&m?LpuzQ*Qb
z^68DNXvYihI&7_59S9*+y8Pza8t~SpI(A+}1Xi&<fQAN2-zGD}{D7BI;XxW)1GH+y
z<m3wgLWYs?cK!K-O+Z~&S9f)^i~<(3k%GJgM@o$b{cLzN7#z{O+?%vKUTIrhT^-?s
zN*Tnkp}sA2314x2ZF!UIzp<8U-$yl~M6aUbevX?G6C1rxM(%Y1?&Nm|+$<N>M3dir
z!KF|&HXkToogRWsggy{#342>g;Qy24KB+=}qF_-v?2dhzt^D$EdqD$2TCR(WQwzkT
z0DzR<RhK<m=k+mGSg{P`dsv96CJEJ4wb>{PNY5XOoY%WBSmV)BM}Rby8Si$nN6<)Z
zYlP(AW`Iu4_YJ@%YinzXfKSTGV?Kx|qAs_$w-PyS;N6{_V;3PZGP0fb^onWdMLCW3
zp{Ei)5=dUBGHk9CNy96d`NBTf1Rl0hy4{1k3@sTWf((D#AGV`lQRG-;RMKET2+?uN
z@Vd!F{$(_JLSR}tUw5<MrgoBYC*TJw<QyX;2<c4XHXplM_Fisu1?o<0AU1GlB<~-t
zmZMBrv0p`m(Wl7+X&dhPlh0~K6$$6(p1-qICB?-Cj4tyHP6vBSO?P{fMVWy#&R|~!
zC%_k*J@4`q)7!C98N?f#n!3X&(9-;-p!ptP2>UCM(*Ah1r{fN+5tvU=5C`#>5WwL+
zt$Ugo#UTLob1Nc?Zh8j0nsE2fV#5!ZFNQ2Xj?e4lUf<IFEiRVc6Gu3Z>X&R5ChQ*-
z6(`@}5dm&T$5`0dDCFpPvFvq+K}thJM0CCV1J=^=1SE!~<fK_|51!b}fP?DAJAwP#
z`GGn#y%sJyQ0&hE7gGbc)9*D+-cMenC2L^S<1itQ>*M{h765d)i1*h+g7C=51i$p*
zn-y|%bBoFzX9ZrOB?-BxAE!y@OiDnGcm31HQuyrw*A1KoGg|$VC2vFaZt({Du{xRR
zzdg+Y6@-njyW?hL*NyuyQ?pCH?c?|f`*oK}NfDKeh^_Pi(sk&It_~dyB|8raAUBYh
zhoB-X)9++|48)9rA$$<6`eJNcWxZH$3wRGXANDW!GIxN4f6u@!YlU_|OfFd?-s`bw
z*KG}_c|R=u-d}ms@e+*mLyC5EcSJ$V^lWeP-5RrmQV_sjXLg1~o;<D&DNdLqu06aU
zkck}G_^sh|5n#>0hJ}-GZLPHVO-)T5Fa7bepkty2u-EhU9G=F)+`O$NrxprMV)&U)
zt#SXZRMR#k*Z%CS?`?<-rWi2k>n|apq37r4pE(o%uu(>IhlGb?zI^%Jikl<%htK=#
z>uc->4BFn_-sa|J1GL^$n+!tMcrq#~tg=lYOb-t!NlBB^()2Vm@bPiv`FeZi#AK(B
z57)P+4~1(V%3*iN<U$^(s38DWiZVqJIkxq6w6_~rjg$t<K4E!Sl8}%9qbw~g9gDLg
zFX4XD61p?aIPoQrY5w?9gB{>9?CeUFuD*|Ke0;=9T#fTxEf23Hx4`&dj#}!#jWrxB
zSkl;FG|<yg7FPh#2EzM*I>(Ds-eIb!LIHhFPZ;G#Fd)y7@<OQ#uMvFIURwHVz0FyA
zAqtKh;m<iEWSe)sci{7rCd}eIJ8T-!s*=7)yDgI7i3i%(_^K>XDRGB?FM4T27~baO
z#-5zKunqkRV}POdZ|XY^>O8RqE88Gs<bN`AT_9x93Z)e3WfKm_|DIF&!9&1gZG*OG
zI*HGfj%aF)|1(3O*4y~hBRqUbKvQXp<j|qia}0@;#|J^cuA5qKTm@EdV%+fHKp^0l
z6mH)6P*uD!y}V|%NmFE&b_p{Qa54}_Pulzwgc(q&Ydsn<bpYwle1QR)9~BaiS$98k
zpX)@2^c_YWiBFKKa@*JAahu}WKUM601Q*AYh5kqlJYNEkuScSoQx_P4qEr*;!wAU*
zL`nlQkSE3`c*V#R(c?m)QUtu4&Z<$ZygXNd83%O1e#-(<LDoOo-dirt>*xA3guy>w
z847wcz^RTLp=?767~m!NVsHFtB9;s|&_vE-F<@-f#AgEkMFRskCcrwve)B@zIf3ad
zQ+3cgFJE@#@dWToFsQsQ9i>uX)>LUv9uau{o5Jb?FG>Lmhi-*bqB1SZ!>wo?rFbVm
zy`5z3)X_6@5PDwu9a05<y3Uz@8@B?!IE2!?F^k8XD7A{=A{QPC^<TKyCMU0Zx=u_3
zUdo<#?)e}T-d588a0-U}4iRMt&CAo^s|N#`^5);j)c$rqNJ3YHcu*f*g_)mNlmc-w
zuoelE@}owNdA<dRx4X8VRt$3e6l;u472T&aIh?qpDvOR>;B9juw?jKK3!X2Z<$=iw
z05YbN^%Bs7h>D79V|)5+?9;|A4onVtG{rT^c0?V#_sQyHm?_LCKF?Dv9`7v(DVh?;
zwu+GTaIEC6bKb&n20jQ@=-BL%z{D+3T7KP<*Kf=oh8KXX`n9<=tj*HR`Nq=2DLGI0
zv_C9%I{hsO((3L$ZL5-taqGOJlIpxsP6`~kaTFCdoTSV>MMXEPI6(<A{|jtVP@a2_
z6;%d9LMH)c9Tmr;G`B;3jHX@tHcW<*$@^Z$`M4YhFK?LL?i;<-)C+vo5=1ftElxi`
z^9TB4oRv=vj4a6Ge8674xuDM!Eq~M<8_s)}PeJ9|Phd=sSMX+KdQBbXx;8dx?R}Y2
zXAo&D%BT4Rcv}5i7OwfJ$>@JY3pa~;(brxX_`pz+-r(8s5Sv<4cka=UeA2MI$2xC)
ztcTm5if12%&wj0yn1N#8PvQLD{W`(4!-~j}C6Dwq(y*xhD4L+{@)e;d;h(6~1r-11
zoJL^C@d^Yu<ajMG?yi4@`rg>U8XBWx_SOh0()gsYHszJ2(+Js)i|}!hw(d}t3bPW#
z@zo~=q1^fo$dXTHo$K|ExLlUxBj{p4#JO3`L)3{eCwwAB%BwDw?S-dfe!Ft(bfBkN
zQ;us6?+G0NNHrc1SfvZh|LI|@M#m+x0ICkR%!Ur+1j$#Ma+b+Wx>x?tH8|`eY1vxW
zS#(;bhxbH$Iey#n^#8}yR|ZrSHSJ1=bfZY;p^@%7bf<JTlF}t04bmmuB_g17NH<6c
zNOwth-^KgA-@W%YoW1v|nOV=w^Nhry_ofOW4?FmR2pQ_ZVH4eEjT<Z|z;tvu;4sxq
zUcGf;f#~M_26(516DjzLIDo$)lnlF<JhUf#$NRljI`*oyJ2#VmDNBX|>^gREQSoRe
zkIbP)iB+WETB|K%iy^#OzK~`m=v~LxaH9-ibWzxP^kUxF?-EW(p(OI<xrdTR)ySyB
zLN)vFoEuH1$%;`?&1#O>@QY6<pYp}`6ztnpKGGC;S){=J%{*PUl~iW>I`sY=4;D}J
zje74kT~V?6b9$u2(bhZV8GwA~Z!GoYkUQAzDf9eTepz@T{Aufy(2nlBk@$AzC)Uc0
zYalt@waD>V`p3GG`;zwXKr&W~4qCX#^3T|jl5>q-DN344S$`%5gjg65Kz_pJ2Vn0c
zL_rC&-txSn=2~sPEkX)rj2vFy|5Obiv!yGOr8CQK<MNruoZgc@x1_aGBS*XY%QWwK
zl>G7kaynkT%q`s`br;J^OeOnoCL$>%5~T9fJ_|-c$eYhU8QijGaLT<Yd(8^_F6=9j
zyR2ee2q-o$_h|t3<v*5n#RQ^YCY24fNPOoz%ze<tlc*-W*V{}#`u0h`UBRM}5*?~}
zm6Uu^(f(y{ZeHZS85kD`zNYY#gjB@8>S)%MwFDcip1djSLfY?OwLgQ^ws55fI5QY1
zKZWdb#s>U<vWzfRxNy=-D6B~jyB>TQ94K{s1L^?nj|>+j@BoB+eed#&Fn%f1&`6_A
z4<9z=*52i2qA%wU)mFm+j`lyuyvbL=dlS=FR4$C=6D}qUOGQOuwGVy?7MvM+#rh(e
z|6RCZ4Ue?Jo_P@(W{KywTLNNHV03<}KVSAa><MS_Qu;aXgdm49IkkHPgFN`;Xcc;M
zO_0&+?Rg6kR{h_fI0mH={v-BhfUemnJb?QHw=z|4NH4ZBqayeTW@=xp+gbi~M0e*n
zGd*Qf3klfMg`|!6(@>vYoq3JfmdFzJ3we9)qc|@Y8SiROT^nGtd<F-v+bCAasU!1c
zT7tyqNEUVj<GnZ3H7sCP@xy-58S2n%dZv||B=j*~$upa-4&Z_6wUQS7xVTK+n9|JF
z2lZoHNc1^ee$$8Dgp7=gX2G6;o`d4g)9P?~ph7@>hL@KY><OYgm87B3Dk+V+Z-((F
zoX^v0yL#fOMMTpt@xI`ym|zWM^JQc?3%1|2imsOB!J6y|E(-6k2&XuxJRl`1lIOc8
z*48)Fk0g6Ud}&19v=%2droNOBo(~=!D7r$7r8bf^CG-&dGOFP@`!exKr<(`aBBW?N
zti2;)l;KSg0d~L2k~+QWf2KBCS+35-y#!3vzz}ARVu)%dd?A6uTLsRTVO(Xiy*DM-
z_cyF&R2<1O4`8*WsBtzow&pZiug0G96yMg^@S{InRz#1nvJ$Qol(Q)cABnq+&8^{U
zpc5;${|mMpkY!r92+J?``z7eExqLp>IX)u5mNd3ac;Xp7{I8!^S5AsxN3I1pVM*6H
z_JcfW%NiUq6`Nz;Ue~V$bN1P%hF>D`%e6s<f~i9IYHV<>t7ksv`fo>9{r9w}H~7T^
zebkwoP*!@T<Yv1#_DF!Cw&F<0*Em<Y9Fd7;c49m!D=4m~u2-=X@0KW#L!Pd!G0}?i
z{<GK!_^A(n0d~wv18dzV2qz9oF^;tfgFB`muzCF=KYw#lDh~)uDN82n+{2WSB9;iU
zWW9+`IwX8`u=U#dXjOg0qMgjS*7*XZV7rholD<MbuDGw6t?qS}{5CSM`VNJ7;VgWX
z*)@u>mMQ9%Twd=4tBc5U!Xb?-aB4!bH=R4@u(5GO@^rCdnHnf5qomn+&JkXlNf+cY
zlT@KJ`la|)q;Z5JIBJ#@Zd5GFyxSJ{ZZnz4#UD8;1`=tU;S7O;lr>M4E^@s$RkPP`
zt#B+c{ub<E{6PxF0v&mR+F4;k$rtC0HYqWUHGf!q??u)b1#jZ=gG;!Lf-0=u?m}$7
z?F*cD7J29PS1UyF+YsgA@9r0u5pwz4>qe>JYR8=-ju-Qi8J|p_O_XdVW+1Ue26}=-
z?ci?C$otdW(pS-G!{Se~8dmLW6=NSPb~jen$Vd}4C3WD%zCaws&!z>_{dX;r_|<qs
zm8zNYUS^eKzJWxRQ^*byN^zX}QsN#6!BJnq|Mj{?qBB=Pq>{l!e7r}J@7Y{sfiHv+
zRpk==A?WX9TC8VW7?XkdZt!o<<XcXbY#lk`P;z87+?`BJ)kfI$QgNDvU1^YkC8*Zv
z@gE=fEZp%8W)-`PX;2^Drqi=H#hYC&`8cgTDSQl9x0|5SH!-bWb>c0sZEBygS)=7x
zk(CD&J6-9qEY=znZ<*`={bmvd{w^l&>**Or`u;^i(j}f89)1uxI?DaDh`P{WyvH{j
z!$T_c9RxD{otE^y>amA+vLt)@G%(N*a+zSUZC~X64Uc9#dlG}BU!Wx8)eGIWex_pU
z*;bk5a}Gf%1RHBr=T8hI!a?Mb3I`5_ur3qG#sHh&kEJW-ApsL*hLftM0V$*}Z^5&Q
ze{z5q!%Lm`UXirC&zjJUt<yu;XCg42D_S8}Mvp&%r&+U()czhhh@QnFkC35#hAS|I
zRqbz<MmVS>pg>lm`H6#nuBO`P2@93L8NjnfsMcIAQP~?rj6@ZSgOnwgQl<s%f8!%0
zsaZP-#Y6w3=Q4nj5xvg6o77Yjm@kf1C<ih~?dLVbtfpk%+iN%e=snF&qn+9(juL$T
z?_XHTJ?r{;o19h-jz30o6tymtm_!WWwxRXwRiqN-pQkDnlw!e`C47m3Brtd$=G5!c
z#RETv##!tzKqI9ZlGKC)4rHcUyO3pvCGxMUdBvMmdVVPB{5So%#dA;3lGUdoTbA-!
z!Mryyc0>_+nuV+2o~RV~wE8oi3@dl`>BVMP!HzkfgCqB$@npe%OXyQ-`7}GE5GG;}
zAKv^pPq}C+85d7;6Wk1drcn?Y6gor*-9R?g@vJclHX+u&|IIkV5FuU0NJn2jD5e?@
z*dGYQk*JTCGias)bjd1vGK*mM7`lxbmPmWZXzU5lfq#J@L7~qAQsy)e{_S8Jh4F+L
zPC?OWm4=cedmcHMW9$L=|MxVV2uKQz24;W8r6A%hWRR)Qfk+aMq)u)I*;suKP;ARx
zR5YmRAy6@`{8M?x=Dmd0dC57)v&oO}mPvEt4gR}DNll;*u$%<kOd{n}<C04|T?S(c
zUM`0`*Go;P{C}V084U{fB-Q_v6jH^!`2Qa){t62egf14BVYZ|tH6;MJCKHrw3num}
z6_mubn~9diwtc<1>-IAH)>1sR320UM<hZI{#VKv*LmrB~Lw>J9@3I^9&=QyF9}KJ+
zz#B_Cpj`Wvj#2J!LUVj2%P^NbHp4y}JPA2vsa4_I1<Cg<bMJ6CFR9Hi4Su}^q6&EC
zlo0f%mdc|<gB|*EL~<78hMqY8C5pH7xMMl@AmHDmPbt+xI(-y!THhyFX00hux11&e
z-|_o8Ke8WN-(~o(xoB+fHVrp?n~g=^*Gac7URe^sl`9l5(ecTpe?BEK|4<_G)2TSR
zm1xyI2)!NVS*i$@3vZOqoa|j@C;EP|vCQa!zTj?||791a<cGx}Uu(*HI}z(CZKHri
zvnAa$u7jd8$P=Y$JosKpzT57#hzP;~IEkyFsy7;ruh8avQ#lfHpPDjWSM3@6uF-z`
zA?)P%sh0J<cJ_>oz&;+k@dJLV528cDK6OL+0q=G=e`EHmiR=6t#TC*YG6Ie{vSn0m
zR6z_*3L6NF4-!JcNKj2a4(#bK-%t6Df^NSqKFz32!_1GOE%`J)ZLP0lqj)S=9S<0-
zj*xnvV<=o21UMZpF+Tlk6uh4*Xp}NxY&k_d5L~tyj^0}Ko%HB3N#l|q3veUXKGg)}
zP?6m{kfJ!f(_Z17A5G{^dSgA^LY;2~WoIZn2X?iSIwmm$)$5NhKIa0e_cD)D@QJn?
ziHVL-UtEL_DXtmdiA8x9EybS;+<xokN4YFZk6ykA*Vjb1@l^#?o<>O98B*)$3|PC*
z+tLPzo}`GlXEk~5#5X5=AQic5&iRXzue+9naeLGF#Acl#Di2xhL?EaR7=HK1X6lEI
zxBuBlK|l^${hdC394=ZY%$WjJD!<jQnTGu;ttn{@;>Jt_IzybqH(Oa*UORk0BqMg2
zV%ctaQGVXnp47$LdCC#!&2t;C%8raijY?FHd{s}p6khV4H2$>bXG%l?`Tg!{Gc`ia
zFK&Rnyg`uS@X8J=S19Cjq?#lEXOgMVRs9|ouMUu6_8P4dsYR|{pDGS%eJU-sY5G9Q
zQ*oC$T|s@D=Ul)1^WSf;4FtDi^%)y^<3qIfW128IoC*NF7+1-se9^byv)#+U64(~W
zv^3#1YKl+P9FnM1o<vIyDHJ<aWSw^$b?S9hD+e$mAIIDU&t<X0KB?cWBR_=CI8d`Q
z*1#KM)*EQK-N#XRG>sX&weD+l=D8?tcQLZhsi956Gy{M~z6lfoZP}ZZr=6bfZPi)+
zOQ1V&-Zvo+Q;kO3Y(3J$v#cy6Ma&c-9yLhybBY0Q{4|1{<{0d{1e)FoPZgj(j}})o
zrY#@ANL}*hMAdh<t0libhe#-u$D=vLtGuEY=BW{9{K{!(V~0^^iUr1Kk>y&P4W)<<
z4fxq<D2mC0MLf$2f-T!^&a@RTAh=hNu~5yBwMdAtvgaW8dM@;f@~frTZ*A}vrzOHs
z)5!f%;``b#!0?z;mPcSJ%wv*_OIuXQHGCvtGZ;DuQwsP60I4~1=SC#gjXA3~h_Nt%
z9n|5r{dx2B<*z1I^7jf{7Wu-p`k5D9$F7@YrAdjrzQ1f-!X7Yl7tD<=6b%PFWvAY}
zZMZ+n)Ym>2KLJNEFZwG=3<HFa-!aPMl%|9U1T4%TVEGbge{FKhrgEeCyhYQ)ZNI6$
z8-u<Uh{>*-jnBi;S}mXMDEbZgxi|HGO`;aR4IIgr+DI;%ogonH{Kg{*pJAh`{og3~
zT|shEMick9cPVEh`6evYTcNN}sG5XoE8EooE2ONk^dsP~zk3VQ61m3=H`xE>cUr#p
zVUEWv#~U)>53WnjZ0uXy%jI5aYHTj)c^&G>3Dt%Bij={@l4+KvdO1)qijz$P84f_l
z$${%KX8DSac-&f-5G4c43R4>nL;0?~-ZnzuYx`+O4p_**l-^}->%{_PvgC)CUvac<
z2o=4}Pt@z%Z%hB_{Y&GYC$~O+v;#KW&^)JRnAvKh(W)JdNZ4uH>zxYCR}$)({E@Mz
ze{Btg;x3zZRFWI=Q@D8evRwHtC`$0E2W@&3<!8F?jW<2QTm;FKhquiKqb>U#X#MmB
zQ}McDRXe4>m5qu|BsDjhcDD}Wfkci#%TrZEZw0j#Ox=}L?9S7EW_18o;^tEM(yNyN
zca0W91k1joK`4uhIb-bs#ve+I{CR6_kQ;GiXEFAb-{05oS5=PMlXe19|LsDdd!^pj
z%ttz+&m1*sz*!g<8Fgr&_Z_;<_&%LgE&s5eHu+tOZI#1%Wgh6MbxRN}c-1nwvn%pp
z=X_E5z5k(zegEn_qw$|U1kFwawwp<^0KYSd3KKdqO(x0{$OQ~owdB&rMPE9;#2y9~
z9p}dj9$ZXT!4nQXMpg))&uJT7Qam1>wnoGa<%_yn5V^XFs&=H`rv}=$oZJ)IXs8-2
zvfs};eu10ui0QA^q2rs`JNz2p{lM|Q`>2~beDJI%h+D*;r{<|@rSO@Z1Z1b=h3Y*8
zbC#Q?Jr)HXy4@F!y`r{?+MIw!81Cc2#<{0q+oaUUByNXO6rEe@|F|%>5nX&0g|e!1
zR{gD%hvgV*N=Z9$Ic?9{)FyKlx6yZ79s2nFnm4ng9Zx1Fo7s)>25-S%m|U$o)gahU
z2RnM3Mraj5MUXx8dN?dF<g(JX9G4GE&M}>93Gj%un|-eLxog<j-3)8*n~IlhZ=DA^
zo(C=SfHK4)Vwu=>?IzA7?D#0(FZ!&|w?E$ab11Pf!RMeHXZpJ1jTyRDTo-j<ojp*)
zIPkYM@pDZ1mC>Yp)RD(-!2d({?K*4mqIX~{q@PJQz^nY`)!_GXYbR9(7J5j69F)uH
zfmuDazXkR%hDXW#r2DPBlv10A`RUZdSpV8KeSWFwyUwflkR{KJc7N*4sDCyKCRbG1
zceUROZYepirkiwH0VB=@Q1wxcv=dzS$;*AEopXM8`5&ppK4Cb(@YU_RGat-UNjLXN
z_XZVFTTKHsH535EQkH*{XAvJW-y%+r*E2pOMo8S~S9&?tUW4+n$qxDbkzZoFmSKo#
ztVL?9AbWt#Zu8BpfN2_K<esASqtZHF0li>Mz9GEjsq(zV=%4T?<CJyMDOO52Rhz8r
z22ayiAQ{&tC`7SrL&aW?0f?>@p=~;U&#wLe*(oTG4l@HaV6Ya)>~L5OlP&GNG_IL_
zU%Ot=2zmQd(;y_j!I)tX5cx7P5?)eMt@kQQiTx}b-rm<turq-KcJ139@8jK^pUquW
zvzKmHF9W?WIqft%8)@(SZBJ7OZ5vQ+X+EQ1m%@g+{mO*lrClp3yPy=5K8Da6(ZS~k
zXi}MzLlR$4Dmn}kO*B8WI4}Bg65xQDgOJZ#E?)Ndwcp$_c2zvHw+3zj>+JT*+Qaui
z1jv1{$&wV<Ua7zQ31{{RxVxZy%5)zEI9>V#z*Gi=bRmo7#B$C_i-8_lpPa9}C9SKG
zq|rtmIv(S)BPM!B4{VCQE4KHC(sr!vZ}PmK`gZC1i}n<#fO=PfeQ(t4{2?DUpQnv&
ztIRNAh()UO_S$&aD~i{u7IR+(ey%P9?F@q@VWuuy=HW^?jrOO%w(BiRVSmsl$N5Ae
zY?`x)5M6>0GI$*8KDc0}v^Jz!=G8d3SD64~)XJ6)3O%1|!Bwr{RYNRObE=2FN3trd
z-D^wQLYd@=5)70WD5t4jtKv7m#|UV)2Xa~Kh;E#1>{&-Ye?nB<9uE56$lqoyowE)!
z52?EzbSuH`voZZS-(R{+<z5C1VpMrCY%MPDUsZNi7m>dWq|hW(pEU^bfAa#Vwz5bQ
z69G*R66=Kw2R@7lec*`@(<4wJ%CRx8XV10|qV9T$zkQwmCh|hd2XG0O{U+~wB2F@6
zrCV6afi7U>!N?+2gf@(d1~67d^qikd<8HAA@JhYdwObW`gfoJBe7RUkzFfV2^Ly3h
zJU@OxevZq<<T5v0{AC}zUlmXw62%jsUyoPI#dLQ8sM<tV%5-dGd{VUeYAB3GdtZd~
zM2siQqMYl`=@$G%3?{wuG+`}f<=TV1id7SHbi2|{JkVC6_>LsBJZl-ckS%v_M?DX!
zhl3e(^l|n1``FCy4?HUFqf&ipXhq82;fE0a%}$7#LVdm~yjM-{#q%mm>R!+C-;=|U
zHpek&=;OSIbmg`nP!p=b!#DPJ?VS~S)4^WC{iw&I3bRtK`VjE`&g^CE_Dcmv&CFtF
z>x*A_lhYd?dkK@}(u~`5&ci2_eVCV*-2Sd9nOWWo`zytIK_0kC(~I`CPpofZXO6#}
zvVO_6oUmF9T&5d6;5BvGcCKw6G2{}x;Or0e^hXL2A^iH}x8@t^meOeP#OG)*WF)}q
zESFQXz1-w;^he11iQAN;g?=D+k&uS44FCBCR-T`~8MW0C?X!{f`g4iuR5E<sP9xbq
zh;yvuP@advL4ZH++;zh&ap&g9(=fx@+EiDT0Yrn!F)?Cn?$o0g1wyMb7GM4;aCQmW
zNY;#DE$_J;c;F`4<F(yuNBV|qFeUdSZe7VAn>F|CN@S&+i^4)BGw)#B|2R#gm?izT
z)1k-_29I!VJg@4N`~9O_ySU4H>YOM!8hzenrma4ffJ@y*S50!q%Vtrm>EC8osXiMC
zW_$m+7G{?MR6<Z)K6Ejb^D|4d)p~@jTOUvBFbWyJrjI!j<k!(=eo>;2@8Uh6bAOpu
z)Sz5`?;Oh5<aX-}1X}`Z&K`iLzUL&A$FMe66%*jvI6la^!ys;Xy4@2c)iJ`=5+0+@
zpFUb_9?@-_;==t>F6P7Xc&wKizik+2R=(}3T8T#g#p&pwUZgJjy5CpPnlG}P_E|-Q
zqP*L8oGbNTvZHmIjVIw>e(`M$Q3gsb$;*6LP9?>ii9F3LDD=Ch+7-|sv3Z=<{+(Z>
zS1vPZYp%-dCU?YPVX|FaTWr^;W?6|!Puj3<e|Ds+F!MnNSD(XpzAWVpKc>jVC(qJ8
z!W(@J`NbOJkmaWZlSXaQG)$-_?jA$ABi)-Dn_iT)I)8X0WnQ%8KW~+A5|7<7bm-dC
zJ<~{D`10IX12wR4y(g|5r=9J&5?SM8O!e3fIqJR{nB53SHuK-}&yQ2F_&wbZ{%$Y|
zWC@z2@xfL_GuVw-_+8?gK!DOaa_=C70m%DJH$Gsx+q%-%zS7Z9&z}21##ysAM}kq7
zq+=(3GilW5H(4`%oQMe=df%e|%;k}o&{p}dADJAn4nt0qZMMw@>qD5^E3TL{N!963
z-jTLjdE1A15!+k`{c>c<`*r@+ZK?y^g?iSe;;&q%+__v{e1EKQpsf(?W5H@WtUFJ*
z_Bb@N-fgVE&^M;pSZSg-NMxW)+q_nYY%;;t`?2BH$#<s!>=H!`-LnQ4&K0wL7`la_
z9s`d6;ccnQBKJ0*J^hC{NrIu)`kFV45Uv&-qx5YpES%#T-13!8IidOu0SEe<^OTR%
zDrBmP4UYG&%M{a5wjcM(EF9+S#0Ps9{~YnUKxQ{%%c4Txw2R1#e4i*#hsRT`#8^}G
zw*FJ&*QZK26@f}o?%4LU3hs&iGh0?cOLmvQFCj)oiYrR?Fm5jUZQpL6GD=#kW%se=
z#_64~8KuDkt#!G8tN@I^Wk9P*kp0Be9N0Nuq;KuApga<ko-8GNbLiec{j)>+o=d?!
zZ=Da%t?)#$=-rgO##J;RYd`8KeJstbf#iBdxb{Z#7<MTOA!3U8Ot}rN5mznHq*CUE
zqY{w(IMc5fT_kwd{an!oSC^GtN!IY2Hr_bh2ZOyH=#9}&NouS{5Q{lSGu-9iB){=k
zDly-rv8a(z4gArT^#Q)^(Q<9|(sitb?dl{(PRD<Xi@nW3MpnbwKF@&&$)}9}>=&cQ
zy1?!Jif@a&ch)nM<tNE?6QOw*NL5Za{hP`|+jH|l8<WA{a6s2Qk@qBUfNV0=jfBsl
zO^1^&6$CS0wDF6URsRGX2%Iq}J~ltvic^K}b@*JZT>r3h%=G@&0DF)GguC8}H)+E^
zxT<2?H(L-%4bA$ErG1@(u?dGPKiv<BUMgJh`l@(mRexr!3ui{{nWvqG0}u?jdcUk(
z3m_9I*-wJjF5RkFR$^?R^SNvTL?(Dx3(1+mHL!?<7+?_hU5*C5OP=!S2r`j!G)W>i
z5f=L*cNh!)DI_3?g%U$vE?;%P#q?nRV7f)3DRTXLxRX(?t8<1XO)I%HC37MO!~O|K
z<*OkTQj~ufdpjSk#{HVsj7P=FRHqZ%*y{>`57gwlz@FaAO$Zbits~=gxh|8g2nI}g
z;2rWbJ6YjEJxI9T#~UQbHDI+A;b<vgyhv%>MC0?l12|OQvNxR$T^!ytNBzG5%q8o^
zha;OWqZ@$frf5x~syO#_-fEZDXjLd-yufU*?Uw3RfjvFw`Zo_{_mmbAcy#F$_I8~V
z+8#y!tNHgI7y#Y2#7hT*fo0x?qo$c*O}UO2ixe}5^jb%#ztYe`IJIm$dGwk727HRL
z%3GZ`8qR*xrDuCxpDVys#dEuBLvFc~Vay&_&Xf7?5k?CaJTvrP`vj9eQ<caqIimcJ
znF3Mj9zT3L+Z9Dl(yYe;`nK^asD?p`zF>g^?wGNzx3uE?(}&)U?dkBnZdXR2jsAKK
z3Gycfkx(;z>@#_oydK&Pz<Ra3E1i`Y<}4D{lDfML%O9Viy6k*iTs3ZkNsC{9)LXUw
z?bP4we0*PmWG^&lb<ifVUG>^e#}2q1=a)qw0x(f%SL{P9%?@IJ;30eR=@~T~s*i0g
zd-*<n0~G}n49M*tAH930<pSVETEHav)uVm+je-;C*&J+V+3?)qeUOirP1n`(@77n-
z2<BOub02cL#J;GjRnnv>&#}x*bJbWJ75URG<U5hw*ZpcwDmbcq!NaF%%wqrAbg+tB
zP3Sxb)7h&NZYx^T(dKgmIb`oeA&@zG9tG?#&6i)-#Gt`f1_#ZnDo>SkQkh<IyrJ;+
z@G;|QcKpHa1|QjTE3iv)hJE)X&h>KM9;PgnMtzC0-_31Wr6c>fe|PlBM=Aj>g_i^1
zjAplXSDG+A2^Y5?^KUcL!?O1ULf+j;8cjRH+oXF*@+0;t&s}{=10|Itm@1N*U`*#_
zJ$cewgg^f=wdCjS>jpsUyIN{N3`Af<Qjj1h1nlY$-|>Cp%Y{^c3|iYuy$1Q}O+Xb=
zv&goYGLqK}crEOe#}&g-(IKYlrv#L*tVsQx*DGs7L#jfHSbeMSzjE_kUB4abDf#ht
zgilfH<Kw;qsgG@$_vDMVM~hKnX1f5KcfL#PF~Gvee6E5+XM_~r35)Ej4}aWGTDIGW
zqTYx)^kzG1cuRFGMT!m&fEZ8^keKt%jfA~uCNe9j31kKsP|<;=PtF_nV#0bs@2nNS
z)Ki}rUu^FVB<sQ#{G?g7<j<(lQNFkRxa-Ll%9SF$n+;Gw!viIC#NUt(cS|cZCkHT_
z<ZfD}$|<s$Pfe3(*UIkEf%4e%%VM5m5o=M^8@(L<f|Gx(<C=m}ny%LN@dmISsm_ok
z-#RdMO$>13d5&>&OYKKkNCQp;`~1iv%R_(jk?ALTH7lrTx5!4D=a;@B`oF-vcvrT9
z7&w6xCUct}3$C<+pWPMKKzi)r8AYXs;D9I;<bfy%s-g2S>Pw%t)+e!xo`I<Uf!6cH
zB$M}i`}_73^4CpFaiE$KIJ5c4#V>LuSpknS8Q{G@3k?;W5@J@sNN6N{@>5t*A-^>4
zFHUNgkJ;f=!0#11%qg>4kvZ(W*BNa`*Edi`jU}CCxj*edDCK|O@tAMgbR<x9a~|DW
zOLFBWi2&6UqR_ebAsx<em~-};m5+!RfcRhj*_d~17jn6@{a_it!4&IsJ9wc7x(qy=
zW`&qP00Ghe%`NP)LhL5x!?NvmUK<huS#bz`xox;rltc5jy{~ites`~33CL)~ApOr&
zf?Wo~U1C4XLL)#%$w*&b9&YvWXVC2HrEyk1i&Y718ZlKaQVV|{`=I3^g8kg9Ug6t6
zVxdXiR9#o|(C3s(59$2j(!!$kLPIrSE?!Y!ldj)W5o)VDq2~e1Q-A=d`YI1fVVywk
z5B@wDGFmXK28KO|uq>~6cHXT~%LPE6B%Sd5xP!Xo=~MlNMQ1sKCHk{Dilk#)l*sZ|
z_iqPbIj*n&nLSI<iUYOT?<%{?1&fW$60Qd`>RXhkFv}ic)#KC2S8hjk70I4Qg~Q2=
zod>aeX$Hm(B!yF|8acSx>KKuowRGh`-B>co)1JLuXg$O6%hSY`50G4bnaQJuNPddB
zcoVl*FjE=Z$lP2(H>RlBc=Y*o71^QB<;2)6o&`Z<B<#7evhTaG`*$!vKtTr?CQ>e9
zh4;N+KF7}fmJyfTpMT5k^UWd|MthAR9EkB&@NM%y&mGZQjHmzQdW^kwx8=9pDEn|K
z)wCbzhWWv8XVIkbsAyPRRTegdBbGtGG}&p9i;gW(IXw!i+tQT_y%N+{0G+YZpBd88
zS;xIoVl|IY!0in0_jujqJF?OqVDqRMQc8ZOg%-}EPjTK6tceQOwfU?3&M|;@4I}^d
z@cZ^H62Fea9fR2hympi;-INjhUzK@=7f>2mQSZ4yo*!^qHcQcJgG`@3^{(kPAw5hr
zjcgepQBA(2w$ky2A2X<N`4d#61jC+i!3wBP(sB#+yux4P=_C6wncKaboX|+Ws^Ja#
z)r<K1RL$t<feN8G`IIBHD%H>d6#qXZZ-!aTwLy&|K1$YEhhVXFJ8>6RdYHDNrZTdS
z$J!4agQg27wCD({lnK84LSK^)S|=lYUkZAam)x$+hf6%ZUQ?>!;WPFq5_w{k!RcVk
zD?5@x+^3!<tEBbLe9o@@@92ZE`92-Gm=3zFJ7WH)c3N&T&>?-<e^7?6g!-Of^GcUb
zwo`B952AHB(?am$&dCf%atzShc|1u6?J_XAE_xNxYMgpr*;TO(0=*ijDur9`TMjSK
z(8g3u)_gj&660|kC~1(m400Dt-jGkI!NWO**&(evrmV&PBDi5Yo{IPj!<<y9d>!eg
z$N#u=vG3RxI5GqJ7bqFD9pI2C6YJmtPAC#{)xzxfU4~;j%?5QkAkk#Ed|;%Ek<fvU
zXS|;n3ax<p`*ep<mTUDrq}WW-2U{Swt^_N|JW?I$w^WOBfzZ?tX*<>%i@1X+EQiQ~
zMGZZ|UQ9vDnU_k0DfvxG0%5u#R;qeJ5^FA61qwZ+5$^8>934ye^tsKmVv$JGlh=AY
zq6N0(w|6C}Q<q6k+Bv$Y*L!hSALmVI$3v1%YJx^@zN;{<rjs!y=%l_6<G@O0{Egh7
zam}ea!pIX5+*f1{gisEh(8AL>?6Th}<`hGtjslMUz8@Q(6dp$#R0|h;k?{xsxBz_&
z2mkQDO4#=<w8ml1s1^+AqC<qcA%qIPIO}rTtF#S}m8!_OSI((fwUoa%IS`;UM!!!7
zFDmr~+#dlks7m1eZ?*E17-*dFcvjV3e<<`}^-q_y?rjsnC`-o)yUulkofo`h5#i{Z
zsf<y%qTc-%?kkJ?9BqjvBMee+R@2{E$89{Fw~J^QR~*8;@|Vqj0Wx?X#W15^lV_7V
zEG~0DIb;~NfwW4aUV{i;PW}BFZ0~J{(nbH<y`n$_YuWiJhu`g8ABIW9)lW@p8)a`i
z*9jVFd83ZL4A&}e{GDvzKIs{9v$nfP0BZuGyx-{SE$Ew~LkJ1EOw-q*(iX1_@ZwZM
z52m)Mze8X6#;37H_UO?J0FLAhtRVcN+B=9qAdT=4Jw?6!{RXi|j-5n2Nhm$4)xq0u
zyjEeWbEuJU^n+|QqEzIp<Z!PUya1l_f<(~Irt7lx(qA=+pYmyz8gG8Q!Nm=3@>(!Y
zS}sBxvZR-XgEK2`&3Tp1KFVi)yS{k8fryt3YrIP;`2I3T`>4`}=^huejP*JF`F>(q
z+TX?T^`~Eyb=Ik}5MvSp<ca)Ys5wHU98CBSgHD_KWQbnu{rX#mS}ZTbS7pZ}f%<PY
z!c9<qN}O8RbT_W8KSCtl#`#S_40xgwKk~>X(1GlD$4j9}MS;uq`YRx>;ngeB`cG4(
zjal)t()b6ZwF5ReKFchnGAekn^!pViVP{$dN)eDP51zGO0&QFzgvL8@C-IR9y$7V=
zZr{X`$Qgvb>&35}ly_0+R34{;w><WnJk|+61=5oy4ZcUahXqy&nZAj$o~u!&#w3~z
zy9TFbWCTSl62^KYnGq%A{^Hi|H^g_*L!&yvHlCNCGS!Moi9*P*8Y!7*JT7jT!;}n=
zl?IE4#WxaXI{tMLpml}A${Em7YYW<nNitAYGj-@RNAHfAElImr+f!JuP||HU^`r#(
zQbL^3sg%bjCICq$F+M(Ch8ic8&4r}C8@7OrfVo~$kh%F+A5GxEnIc2?Sd6*cPza_F
zN1sUKryMk2=5MAO0`w3sWfPNCz>LBq;f>4AW??be;1!RTdGRz5NWv@F0T(;`CRCO$
z#Mc?+Y!&W)AHD>?XEP}&rJw_!7W@{|{%m1kK~#g)umvlXE$vk|H_OvUof-!$+!h8F
z6X8(S>+8Nxo~Q*cKu99$DM<Rxc6&4{J3D&~N6JuQh}nc-pe=ZSVa>?hn50%=fQ1P+
zHA%qcrYKhMZO)x}R0%sOK3=jZ;DRhPx!HlfXR4~I;Fph}ZcW4@(H6IhKzWFzXp?ez
zLp-(+_8$rP@l2PFUN54Mt^&8RekZ*K*ap~nDnz8D{(Dnpes`A^=H}rQ9CFUG>B<_X
z7d)e&8=1}rHCK>(m53dVUg%7}!P!oPSeVv&&7<KeQk`{Aa-i2MywitZDbqXuQqz7j
z$-Pe(R5uh-`aT2=EWMhV+S=+WOn#HgW@H0()Un(ROUDm<E2E=UN0)<`#Ny;9Ke+m7
z!Vk1Ze)$fE86zw>vFd}o-J3^Tc>=wz-IFykHjQjK+N101H}-4h_$eLjQR=W^tfKM^
zpdVahQfH_5%a>8Wk6T+?|7y={_}jp+nJPVA%gN;1N|l&?13j)V8fJ7wmnHdrxOHt1
zt1p4uv!29p>3lPjIRQsh^W&$H?W%nHzqk(;M-D^ql=%TT$WU^}A4%d!&>~|Kj&y#0
zegHnV`QEs=yVu?8>;L>WT=9J2cSFWeX0%Y7Eix{%pUc89)PS@YkMTOB<75_3RWe%-
zye<31%`L**t^p<)gieu!*GO;K*}>}Yx$S9qdh+J5-nkAp6y0xH8P%t|o&J3PvEfrZ
z^xxWPeM<rHr!4}8lW_*6h+iCE{JJSO1Enn~<uA1j&GtZvggD^uGLe%T|M{weghBHC
z#}B|z#YQ%7G&zlxCH}FhmD)q7xIooSvnk)GlBBAlrB6E)Ea74Bmej{`BH3sEA68EL
zgWnp$-H%eHh>T-nJP=Mp(bE{f$7yQ1Z4M@bje2}}IXXNH9~fc9V>*$Z*?JWdpy%Xp
zfshpkzuD?1#b5HT$jJSxg7U{pWoFaw^2zrLb=oj|w-k-XV-(rF^!`8*mkcZm;9Pw(
zY}Hj%9GRZxqNSzf;^JatjOsD%Xt!~$#=|U+lFXm7zvNS#UnVr#X+JD(w4$z3@H+Zq
zMy6LiKN4j~pY5=4d@XMA2iJH1>zYg3i>g<4UqE0RKRY+~9&p3@<0xFWhWP*u1-xl$
zdKyqxD=RA#6B2&u9gi)1v@-nlW$=1amiX(1_Jx1CtM|XrFoGr35d9)yr^?+_zur*n
zbeiDLsJir4ufj&uC$v!8>@2aLs;vm9+BI?)7R}X1=<<UCOeOPVb7^$n=?eK>-k+z}
z>eav3tG1oqU{DX(pR(C3*Nn{(1QZp9b?y6cS)FnWC;1yg?15HIhkLe*>lWNEfzHu5
z{ykc1>3sI_rwcp+S?W?{@~;DrB@_r8Kj}8ODfp}E^n~h`t-OS$hgi^6s)5T6s?vH2
z@~f5gn0~l~0B2vy@*fG9pj40|*5@q3`zkvihDhKsCgi<fMH)wL9W{PR2K>0sx5mej
zZYm_h+1?s?2Dqfeur>uoT~+bGIJH#n2xw>H7w0=B;#DSL3#~*tP@=80IjRzyb<D}n
zS<jR#{pKS*i$X+jG^kl@-GG}o2Rc7HnG<RDqxe7jsCuI_TAixPYNcJoh%MfYILC}-
zd#-q8=#(h!H*^OVc}~F$qO`tC8pQ8lB`RXk#NiDZVo85mK=Cj;UYTZaO%5z=ap7{b
zR8eAp6A4p{(#K17!Rcc{R#0g6y;}WLHd!U&OOTu{O{{|3SsTf+(^Q}C=RFa)UHuhj
zP`@x<cGLfO3^E-%9M%@eQDKv)#*~kby7CfH8PACP<CFX-^vesNR6$EAq&?Jcx|Fcl
z$7+aDY^Ad0kA3wiGdcAlz`*J7*2i&>$zscZx{o1Uo-rwqIX6CN=KY3}Qt0of5X+Mg
zq)H-f{aX6kxMH9`G#4?(>}}>yQDZURwr;i1X-e-}(ML&bu~Ub{TugFM|9IIAH^#8z
z-J{yeh~hJ>mW{X}wUw;CHvv*+?O)aMD^y6=cwsB!T7KTjHjk)hsWl`uJ{P}-_~vdp
zD;l)8lkquU|AQW-Q4cQIB9F0n?_=L}a~G=lCg>!c48E8(J*~BF4ZoM#FU^f|QD9lJ
z%ijA^uK(IjUU!_YFM8i$ZnR8)Q;@bqFt;=gRV`{L=FOt!6{(>6s;wk{A_rj^1rVgD
zi+}<_3V{-**9V^dqV99h?O|rAfxJfF!LP+J0a|4SYR*ndi^>S%cnEm-l(d1GUvHM|
zAL_E|b8J;CDY&RBH-fn;cYa8C7w}PKQ~gFZt(+(#?!i;B<dWTiiU-qwW(-cHkCONp
z{uR_7K4GWFdni9lCul-$xj!fA$SA&0h1L%FamelT#QY$RSx5drHJh6G*5sv9`t*5t
zKV$oLR_!$XjR78An`k3OKLap98@Rsti5NMp4If6Da{uPQhxfzl=CLWDxZlzczN~7p
zWcGa%(8D#vQ;UZw#{iNypCZ*&Hgu=o5FVBSmdgd3XTGszx`&jkT!FoW4MHa}fk<`%
zUJMk7L5K&6LPbt2oDB7aFV*CbB2n|iht}J;t4X-M?a7&AI9`^87K*P5Id19nMf2_s
z!RcF*YF`VI&t!EJ440#Nn|R4c1)3=oLuDfM#4#+s`6gVRkdkeT;l7f#FH8aj6>2E{
zCkz?x|K{ATJzXfW)il*tSRXW-BDS;Mz82FECN63oQFG5IzP3zJv-DFE_t$aa@8e|y
z5Y-}<=1<VUf<>@Gf|x4oV`Aq!->6UG>qHn91o`%0ZRKQ`33!p9kJHUn5~4&vAt}li
z<Nko`%g~WC*Q`*b<lle`2?4hr8rKT!R$8lDpn7IdbqzMWPi5n{_f{D|-|n4nQEmt|
zU8?)vwr+ftxcx+@3IszKT(K?SpdlJc8U%O8=`Y`IGl!2c7Y>UH)0N3sztUzq_lK_(
z-LqP5EOzn8C|S5N`LmJjKdx|?P4}m(PDI~h{ec;GS-mO?K<3=C(25J3l8Qc}K1h$v
zmCObB>_HQ}m%gTnk2~X&e7qb*r6-4og$>lM&}<u;7^f#s&xoX0537<XMg-l~BvY8y
z?}Cjoak@-X&ar^)6H<Hq4b6eGndB(=GdR_Sa%-+pf65MLeeN|hXfwc4;oXQ#J@ry)
zIY6O|cr8)G9&#+9Rm3f^W3poM6l7wm6q>M#$<mVgIgfA;xN0C8mL<|x4LZNNhPi5W
z{XlhEFze>%w5g@xP}VYXlt^ILXBSre`oP6TQWfmu>3Fq%GC|)BiAo-JKF=-9<Zc)E
zzj^;~>vog8yYg{+vrCAP^2^5-g8@%~wi<zPdBQ!ts3mGma%=Ta_!AJxxPb-6$Ka=b
ztuj-VnzW4XTlxA7>gCg<KRvQGG-UT2@`TnsS+2v#{pB<3cAM&8bI_0*r)snWlc_x)
zftL;N(lHsFJyShVEybvRnOfQ+KXu^hjH8ychMy=@?f$I)i6Ci_uJU0jbhUS-2ybSe
zEiE$~5<?ehnc9wF(MO0p;fYUIKu7qpI$j*$CLz9ovm~QabFQ%l4X64u$JtGPdIT^r
zavF~%rA7E(j(sovs6{^&J8X24LOV^xD8`4+W0g-hePxag8s?bB|7JmH$*5bTrU~ll
zdrKP!-~{nf>hF<nU^_X{6&;0CMSLDc0~4+8(1@lyR&pt*qyQ1Ge)7oRqmZ<6ulagd
zVNfAup7bkaRyBx74Bj_-w!5LL>{31sLVV9f!8YRzY`<pN8cP|{12btTaWFZaiUxA>
zto`TEz=0hUTyVL3xJzMfe{On`vvt(=0NNJjGlg*S=8A@yAA0pgBmSE<XL%-8I@@<R
z&vT|~*c>oGzmNjUAZJ_si-y{J>psMk)h~tEbvvWkTRlZhDOUv9>ZHiI`NO*s#^B~Y
zCbukqoJy6r%ejwD6u7EI{{O&rNG`mYoWqk+&<6H7LtKah#O`{PNVA$h;8(~}GPgbX
z-#toDoGT0heM?u=t3XWkSFypUCa0{8<}AM>YxD6{EvLhJ2{pw7f3Z|w=ezd0)!Vnf
zS`a4`1LdFK#3{HxR=?ENeLRM<m@xu=T<G^yF`Aar4z#n+e;A=^eUVrn)X%ObbuQX`
zF(q33m0FdKKo`4o;km4CCublhdUh`SrB*k!?jBuHAUdlMK9)kN5GE3}Ea7^glc9FY
z7s%U~O8X6O5Z5jVG^j~I=p4|NR5N~bajSXRYD2Uo88xQv{siZQv##jPI>vkdj$HjL
zT^Q&6F<-o#I)*N&t=w#Tx!rat^ScI6Oz<ET#E~&m0Z(!4Jga|AC($QbHiLAmZkiWo
z*dA46F5i<L$ybEmH+}p;xG)f$A0yHEsq!wzHz=xZyMG=Uzfo9xoLI|JH6=AlQ~4d|
z7Yh|%#kT8}FrqE`M*p$s*FwNDGT%`ecKpn7*df*?{PTX@5PvL>$<V8}fy@8G>sMTz
zhy<R|xe{UAyr8nZ4pZ!mV0V!ys=<B}d8LnU4Z5BIvK^gBtSY8y&-bC~0`XU<c=)Fl
zzUw>dw2C(4Yj*}=u{V7sDq9c=S!gF>J7$Onzhj2;DMq5XHli{%&&i7Y@^HewgN_55
zp?Dw`a|kCnIEO=QkqA}M!{96)Ye<v50`vl!?k`#&=4jM;@+kro^XE20RY@3iO={Li
za$)u6$D@9B%>wbR-#VQFVXPZZdHUg?^Hc)6)L7?qoj|{cw$^KAGT)6^N_05`;fV0W
z!?Pj4Lp<P?uU7o(*=;1zu@ZrU3Loi0W3Lw9&t0|?Epe;%<}Y@}4;2P|*-&z7dI<5^
zzqt9pjw<}1EV!J*2kBt7BxTrO7Y31S#BRC*^Lm{<i`1{zu~#~!bYgrKz|t85>_2lo
z-O%MPg;}iEo_auv1P38&oOmJVRnQB_*m%fgk$Zk<+h5H>RTbbk-**hxJP?%mjHn4t
z;7Mrs>l8_j5st29^y2@e#pf9<h{2{og3^Er)d|t^{L@i-XN4`Pt?B*l7S3$?Pde_u
zBdeuiFS#=n7vPm00zLLqbzJ7D?!vjc_(Cc~rN`p{K~}C8bRG!QfKHRMdE#&Pzvq;h
zi0W64bCeHEQCLU){Ej_kpE}d6-${8ruQW~yz&!tXnOr`q#c5uaZQLQ<!Ftxrn}n#%
zvmFdiyd|RQRLsXGCUTD|uX6VuI4@^uGD;wI!~uSYUPxWu=ilhj;^MpS{PI}i8$D;L
zW7cdOkEG)}QB}Zru9JmUNz84Xgs8}_Yc9aN`6XX~rSrUA^bi|zbk#uPKRYx_m%}J#
z(<d7LHcYWO&1V*Cx!X(BEe9|kb&6tr(~mK<+lm%1hl_`ySm_`Q@!ww>xDV(*hFYd-
z^x2iDHBgRmE-~Dl&#{_~tU>tkZ(y3}KU$|{Bh{p|7iD=2A`f<0NmHD{f|=>p*C?Ug
z%gJ%L=G!hN&P+c&?B;C^eo`akW7kUMxOV;g*;hiRoE^wKYwb;B-)&3P?t9jdfzQ%^
zyKU2(;kS@OOZPhp>P!;oXeq0rMC|+{+K{?CDj&-W4$LDkk}^D@*ULtflt5lHEk|+N
zlejl4GdI8~yl#UM<JIb#v*z|D4B-Iu6*0q`f^s_co6V)u4<@W1R71xzTer%Bk3w^R
z&l+VYKoiB{0}5ki!2L-|QpT&0n-&o+8d>>yHefV@#E6z?u=pkYsjINKTy;NvlkwlE
z+4VXbaI#F}zr#4f#gd}~EU33XqSQqVHQMnyUf}Q>3+K4db~08PLhUDURbBg~i=N^Q
z%1@=deEM{i$S&BZik35EtumgoUslt#VD;Pwzp^I0qxRex0{r(~z$xXBr~Qo@qV*%c
z&wq^!!`BWR!V?mhB`Jb<SFjnRlq8dr)1J1HkaPrjCGW1^8N{h4`F#1VPft(Sxv;+e
zA)@h9>8wtVR&3A!bR#kp2j6}!2jP3MU~EMH`b-@$YCmMrvEyA?#&v%zG3~xj9pZM9
z(D{JQw?sDSw}-_(+)G!;s({$bAteljI$5WU7BIUP=?kMBb4Rudj-{0{MwBKrQug5;
zUel>wwYcs%`3G7$En(Y*x)TXR8jQwWkcu=H=7UC?GYd6LIH(e77PI0MkpR-`1#DD|
z`@7qn()yOp$T68r$KH|6ElC4eN&G}s&#j#1j*l1*DsL2U;|%tam)Pp7OPPO&2MV=b
zr@epGkFo@(kw1sO?VOvBg#f@>vke^$q^D_NA=w-(%|e0oUbVOR->CER2~ie_1?6>B
z(Z@J|h(<Uo)`HW}=@=wA8Iwpb-shELtQYFFTNSk28|!5bUI-_sQ{w>aR5$bP*NO8m
z_X#dU##Pw<{>2ok+Sz6rx%9vBi{3E+eXwo3LztndgWiGc-IgRNJp7g)Mp_JB%>}>y
zO~|vPWBR$~l|>GuJ{U^Fmx%zwMwOG#FQ~u02aCH{n?3j8vkE<AE2(YvKqk9iIy_Qa
zgJ3(u<bn57&<T3G>tv5yBK`%oCPaB7zIPJUWahpxJQtM*(Xq;FkY&<*JuQeAV<(SY
z5<gA-l0V6BfX}sxZ#_YtKqq#j5BDaH_CGTlQHq)-7~uN%&aMr<*aNm@sn3FDv+FB|
zVPmQZ$($0*r_yp%f!dwyNPNC`w4gxZ@pnA0)gh;B#7NbyBK-~&|6oeC`TW<v8cqgA
zND7q<Jr)k-4<#mk+pM~2darf_3DyKEJ~Xi}Kz5dxiY`B2>Re2Pg^2-TR!neBM5k|H
z?P9$Xz*9mlsi~!w9sdiXx<LJSDz@?T=?u;SngzxGGSt4GT)QjupvarS5d~XH`AG-S
zA9bX%ArJ04LW{oVjJ8w9R~vDRJkqs~v5)Cn9IrGLdGi!^HS!YT(mArt-`rC3=*;l)
zB)#nPU!Ttg$ipw>Vx43!*gppZj2!hN?-eyGUK1tD+IKkAPR$$6P21ms{!mP9_?0^;
z(w91YXm(#TvghJ^D`pF1T4Dg~0E|H5C2_?E+|*s_7UcfQ?XwVU$!by60U|Hq^n3_V
zL@v`j$~u!{4J0kPMMU5)DS>H}|AEU*M7=BR5?=C>8?Q`X703zj)J_E&+LIcs#0`f~
z$_!E09cS}Qa+jrr&g@YRg8oU*4nj;bUv4$BQ|F$X_5z0B`$g;~^3L=UHOfB(W(;1c
zY8j^?s_jr|cZsYU^ok;U)CgRJ)%PtZW6Q>(WLMO-w^3<!4%d?OASxJmg_q0;nxvv-
z_lI=M-)H;J@lmPfTfr)qB5f@>jkzdQS{n%?;o<aQx5TdTmN<3LCpBKc{6Vy<Q-n;Q
z4V(>?H*gSsu*MpJ!S*grrN--DF*SjYm-w%T0XaTt-2upA_tEFuM_0Q0wD<hi5vA?O
z<|8Vxtvi`IyKioQDqMl}DDuz#$NEl|W^=B87q!MC6jzk|ndl@p<fnd|*>z#$AQAl^
z-*MbnnyKzpaSmYuS;s5D`}Y^zwSqFjYA>EyD*0hoY>h;lYr4$Ob&KaoIV~k^vrE69
z&{u2x0o$={W(#{R+m>DgWITLE)Om<$dtAFq#0Aey#Gmkzy29GZ*{iUnuzFzs$Dw|t
zg!u3idY5U@fgZi^E9C{;XupGP>;vDcss8BpGfRvL`OL78+TnG~WzEngij}F32xm)!
zqnr0jpa5Q%^WAHOp!_YH9@g=^H`SIq(5=1)BaQDaPVW{s!pH|f8Sa04d|_#3LL=bD
z|K!_L)<w1U@3-n|rv9xg9(aw1LS&=NDr`HFR)No>{Pr+N9kDU=OV>)oq0d_&VrgmJ
zJvtTcw=#lerTGsMJ<RA~b=)!`ACzu^f&jbbVgN5@3mSAkd&_!l9l`nCW8>hFMjON=
z9ZhG8Koq7^>)M0OMkydN0m}f{N%^x!S10nC+=gMLLSiW~8+vDzd~1=Mu$Z7JzN&cj
z6@X&}M4ugLFNQ@?^BOH2yUxaV&F$H8yX1GIweAN$9vg;i2%V1jy&2^G%>40U=2$Bb
zIY{GqjNtu>IN};o><B|k)V>Z3K^RsbCEHsyE<c50*5})zr>0@(W$bPvNh%U)fMJBI
ztzL90`6mKOcQARCR`hLJxoJNcH%Z-@5fwz9AnKpI+y&56_Vs2bH6Dz0p0V>nUKlz0
z^GKf~rh21M+<>hQ7isP#B6I6>vOuh&i%b%13fF0xM5ox-!5$;bJ{78`mJ+wv`<%56
zu=GKt3Z^eH-f{er1&GUXpHnVeus4bHN|xc;=t{`u>Fdi(H+V66U?l)L|Ek5qcz7!O
zz1P<@l6G5_ab<WoA<TXnCH|`sCclkTp9z+S7WL3Cpegm{x&eip@W)87w&b<3VX380
zLC8Teukn(j#Z+bCaQl7$<1YUJ$J+q&=CBmAN10kcF%dnV7hG^RF4`INRpowOm55>$
zo`9oC1db;8XH~JV50g|r)$aEq9$g|LYSZ^|R|q9&rXr+4<fq`%{xkZG-CY<yi!T&j
z^NI;b%XgC6Mr4;>uSZi_QgOh4fzp%z?l@$Cx2?~@{+<Te(e});-<r*nj}B-KTpmC?
zT~g`UR)Kc{Z9w>Q(>zyHk1$!e<&3-`Hyqd+xs$(o@bLlvJ=o<q_$a|wVyp=g`V1lD
z;hFz?K0%ThIbZa@bCz3*RS2|G|8sJ|#pyqgVJ~NK>F5~SIs0ej*pW%0b+->2uWKAU
zd%_6P@J&<ieM{gJglDZ2WrvheNB#Y#l1U_7oKxc0hmeU@a!HjmMiqHJPK!fvbVZ9Z
z9EKl${@Dlug*B-E<3%RZ@|!sO=tp@Zei43IL=oh^WmfeKnC1}@j>R7F21b2&!$ZZZ
zZ3tT!{a#Hh>2)v>RYZqHPNq2q+A&W7c5+nha+hrgY+kAuiL<s_>zN_xZS?^5aCcm$
z2ce%*^fV2jBMG>APXJBV0iroj*-DHlrQu+!aR^+bRY@e-nOL|SH4c8YJQ~Xl3F(Q^
zk(*VrMT5GnS2`A=iX;s^;g@Cns>^QP#fp~-qBiGjmbQ2mTgpwg^#Avv#F27-Rjz7I
z?sD+B-V6g_s)41SHpRbd$YPU`xI5R4cToK8W*S23;QOG2-TdQ6no&$CvXZj@_B|w_
zYvc9{_0+wIHahJ8&Ic8vYAgR5!FFPNhL*KEPn;<>mZn!ad15byhumr^DiaxE4p9lx
z|0(Ru<DuNU04^pm*2t2C7|cwCCX^-al)*%pY}chy=vJ0)mh9UIV^4_4nq@4>MY=+a
z?b?!Ly{<J%5tFi1gZB)#`);4l`_5m#&*xd@{FZZ`^PK1WzGr#YKBuX5T*67WE5TM-
zzbZx|d+I0B6S|1V_F#|G19?2(Z}(~~M*yjF<ovtON^(!K_NYss&xY;J$q9d>IX=l^
zhZFqtNbv}pW%XQZ84meDrSY6c&NVbP=V~Vr;}!VavRuLWdpA=cm0cgr%IVNt;T`JG
zb2vTCMSrg@1(_;e78^(M!^H75jb(f^jB=n&#Z?WZj&V}6sS+;bF)R9=ooHOWB-$!j
z6^%t)1+cw+bji*2cj>F$`nDClT<r<#`ue2(;zj<>zg|^%5R_z*B9WY&Y{ea7Y}8X+
zP0d(zH{@vRBDTiDM(?N0HH*`{$Lg2Ri8hRS9^&$DdM)yS2_lzDA?iKkaKYa*K_fkh
zbrl_~SQU|^1*=CT&3;!V&d<~A%s9w>yFsPnv#D8ApRwgZ?FJlOKHd%YY7r^YMHQyp
zjp7LjC(E;qq6@RB6OL)}2fhrq1Y1OiXln?_nSUP#<{J8(>ltuaxy*=i;+%7x$_+7f
zf?4J#0%`x=Qa4v}#@F-mKN5M<?La?7hp+stPvt^FSyf}$4w)EhllS@iPTl{J^fPH#
zlk<f3D|#PdJv`0XDKk8;rF;n&`A{-E<9?R0!U#Y%{?bMWE79L1>iO85CmtEc%dpDL
z$zeZ+&u(IDU7ONPGpT<h7^uMa*zNH(w2OQ{!T;g4oxAaw9u3^o^WebEVo8wh(*1>Y
zPM_uj)TSnPTkdGoE|Z`Z<JdtN^VQA%_XezNv2os9#nQ>w48Qj_rFBoz(%)tPOg4mL
z@b+AFS%w~qw2KdZUu#Cix8(YbRXkyL6Fxlbzvjh%BAv+++P0iP8)&<WGJ01z{OOtF
z%SKsLBo+1MlSEyB#?7fjD=y)8R>Sh;-qRDzTps3YCj1ls@KQBZndiCOi<A?dAIp!u
zsj95K<jMp>ZJ9ZU!%&>EDIG)YcC_*&F3XYuC~$4t-+dOa*i3V|C0|r7)$m-c@I#vu
zb_-+?pT*Z!gv59i=DQud<3l}Asq)Z0{oSYZ@FccfeARx7q~kK-xRNmD%IqKcExDlX
zc9<;oSB1dG2Vg_`?VQxdwl60xHigHEubX^2<HWmtE<XZ)n_bs+F!OHqa2e^r&t2A8
zj#v3SVAmwG7YMwVD3L^)&eP!Y*%<Y0fB$%3EEp5&*Upq9REmSFth9$DJ1w-R=Cia~
z8dx%1*)JoVK@&R73q&z20?(ft3@O|k`9GJKj=r3UbE%0H=QDUXY6W4tt%I;TAn4Q&
zLybbkzKs4k#zWu27S^p~EDo}iKAUMFRn%tE(XMm1UF-6G`U|!*`~P_`p;eG@xs3d7
zFNQF4XBlNg)bVi<l?lK5ds2xdA51SlBHj_^pDabO3EAq&v>1~b!67{es}cUZntiJS
zZ7N%$=#)ZO8cud)nw<Z>G!;uoviiCMj^OQii?O5#M^SF}5Lz3sZJ3StIjG_V4Vgi3
z)1jeT8p5oW1GO_Cph$y8HE$#`1RAdUp+XKcrq$8dts8j!zZ<+2Bqb%@m1g#%3o1Lb
zq~=gM{-g<QNfK*mOZm}sTYez*{SW#7U)M33tpmJq=WET_xeqy^$2G`O9k3E7y&dhV
zep1WHexUbd%%NN3=$y!-v9_ahUd&?kj+ZK`1-18UKaTarE@kB13a;jv$(!NvH$B+l
zYXRA+NGLNKrs<Wh5a6~X_P9V14VP5k{8qZ=Fs%}^a?iV@6ca%7eqN_%j9IjTxFkjs
z-o?l{0`<|g>;wYLO&g?-a^p!67j(}<Dkw~$J-8CExSJJa9sv>px?7GQ$bz>&l^hRK
z+6J>~1^RaYjs4i!1~YGOVn#Ux;h2^A;nx!rw?AK%mX^Msp8g_TLtdiNXBp7uHa0fR
z71}X1HQxZ$+WcA!4LltLf#Ch(EO_EzDUor?IvxNUZEMrl%3PeC^_m-aI45amXP0pQ
zKKD~H^2j`ZGH=|tL5~QwY8J&Q8JL?Vi4K?m(hh{^@VyIfmxTbN@XD1h>g_%2?EC>h
z-U)<R>eABE{QUgN!YCJfoS|POSzeF+@EEEWHAe{2*(xh5OUx$eulolF!*y4Hh^xdI
z;$OtU7>XcV6HwdT51eyJO)nPz72s^P<QcoIE}ZlA^*wc}>49FH#PMH#(9_dnj0T7;
zRaiE)6;OeJ%$|v`wjMdH%?SwDz;7#u88KdOyZ5KulBBGqZ5|R)u#pljBr?b9sxk?m
zq2-^Nnu>IN3YcI>w!J1>1W4tR2FU!tmHB(Wp?0qg1JfTRCs$NhNJGt@mg3JXC~))_
z9*Px=>O;9o7kSJL)CH`IRVf0q5vZvE3ZI&uMz~!Z;Ta!=L1r)*zuCCx>r>WzHv%&>
zz0VdKozo)htUvCqp58QB>-F)Z&7C$N*<MsYJPT?E-f@HcfrAGP7&5o@^R(NG;M;~h
z5grOK2DbUp(a~>TWHzB7q?Pl>lk1t))oKj0_4V}@*@%WiL#}u|f;+bqRx6V)&Et|w
zx{P+s25>faiiPo%=F54cJ#!o^ot?d6a(qZE5gz|ldb+y0Q@cH4{35SGeh#ETI5;@i
zUb;b)Zf$L)YXHEORZ<l8yU^qh$jODJSifbiw%(K+vIUEJ(a_M3fL&tBK|Z52yDNa!
z@R=?sY^nXGEnWtX#{<iDTDSIH1X8ue!os4ZrA0zQVybtOEpS!oUKf>$zI4GNL8Koj
zqj>Gm&`>oswH~2|Ru#IMcFXT!D4JHgFa7H7?oNR(S4=I5k;4w9Jguo|dUX5o<7L1K
zXPpG*{LItpYFbqox0C&o>*Gd;`h%b>6Fyc_n>RH)GX=Pu({DHfx_hqa=;+wg`6N-5
zPmu6^eSNkgCm+0=s6Z+Bb2~w~X~1mv{6w=)5D`M$LM&h~m?KI`LG`xH{r$z0ib-8v
zhO>%~F4P-AHWJe|#Ai?^2Af)1cB(3PYPXJ!WzcLJBsb`3r?_Qwb~Zsq5v%Ma7nk$J
zb#GM+1^N47CH_F7i?BF(0{NLD&A?2|&vVzi!|>)#q~%g4*aXGIko!`B`Wt0XYzaHs
zj;X2b_2F7_%j0Ik)klCV+}Y9b!^pm4E~rR4v8<w^c!-gjmXL|qBLtc7pkz>EEG0di
z2TPV?5fB&WE+sq1c=0cMEv}<Vfog=7?g_X8sW?(L9`i#-hPd+z3Iq#C{5wcV$;t9^
za_1S!)=yl7;;2^Br_JbZF4PC`@W}ezZBu3K;u+a5DJi+u&JWrO6Vy`}RsSI__~~@)
zY^sh6VP`NOi=TjP2-o=WFT|iybs3r0P@|w(l!JoQ;-o*xS}UM$_N^@O!3TT;u}`8x
zvS4H=NHLDt(-&W{1!`RXqmmN~?Do#*&qL1~b)rNZ;pb1%M`1CTU<C`@oj^3Gl`NLS
z>E=B{ImQlm#@dY>*0jqKajR1;Q2={q%ETJXQl&R_{hM85V<W{sL?uo$3i&KKD2Qdv
zjelc9W@DqLzdy8FAo5Off0Hpc2-S(+z%aCv70wAjcbS?G?|Zk^Dc4@N#sF&iz=rM^
z(597zfw?Nq8-85mo?*hD%e#UXXq}y0RXA9}N}R5&*J+nm0ma_TLcz^{4QaSO0LeWK
zztf3Bb-u^^1pXWyoqQ7Ey3D%p6$1WSkAgnD8ULE8`oV)-0W-Eyev^%%VCg=8c3+xM
zi_gav@$6Ft0CZgEjGVlDQ~vP4fUwqO&=as(B5@{SB)plq`B5A$>moUh_T|&t$)8yh
znmwL@iD#}vrua`Iu7Pz{P*5oIh8(b)9v-3N>Q@v|Px@}>W3e2{#yux#p8j8!X25z$
z#CgInw!29s>6xXeI=i~Ml2^pE*!kDY0h}E&uJv&~4o`x#(op!Ca_YTY5fl_?tP)l2
z0nHp99`^S3270-)F)OO~7U8k)stdrI*A^y@vX5L5i_wj*Tw>0<ywQCr1*)_bXOGKS
zEl3=OlI3*$Q)VC`EaP#&;6rz0H~aGJo5Fa?A|N;F7iVKav2|)RRm!@`Df`vCvK;~(
zh%z2^kN1gQt(&iI5q4U?g&GuY)qj?k<AxSy=jKd}Y)vId_=#k+mX<`o9y*Z{6O*JF
zL+02v6TSRNe)*UP<i?(!moGt;gD+?vKK$^7v5HG$4tajGM&u7nyguSN!rN7KZ=Qe<
z<G|Y2<+-V;OKYeGy{S(JXGnjKta=nu)fhO?tougmd_5uUG4Zg9JwTtKBO8oGsA&WS
z+!--63oQP@M;A8z#&OECfQJsV>M4mb@Q7gm@i_AqXl;Z1MX;=yfdZh+?3uTI8b#Oj
zet%|r2%A43Xc79aKz4zO(O*jvs_-zK1vD0i#_HS!uNJ~B4BQg9GJBcI1knI^{s-*!
z=824i%GU9+|LH&6@w{xYD^A>;y^<#yGhWm6uf7DoECk*Ez80UDD6e>-&}Bs7h33ni
zoZ7eS)7>GzCnhHxt90i<FIt_}9PG`mt*rex`^8|}yV-IpWiZ$t!EJX@@!vho)DA++
z<S*#gOg~dc?k|5cOu7g(6vlYl4OP^SM1DpI3n^P9-hkV2NJB#V_uEdf2yNe2MoUB}
txzm|FM#w=N&LOuUh@{=$)_=2E*kqAV>Wv<iEYD+sACX|BQ+m>t{13q)j6eVY
literal 0
HcmV?d00001
diff --git a/doc/guides/rel_notes/release_20_05.rst b/doc/guides/rel_notes/release_20_05.rst
index 000bbf501..f404ee7fd 100644
--- a/doc/guides/rel_notes/release_20_05.rst
+++ b/doc/guides/rel_notes/release_20_05.rst
@@ -62,6 +62,11 @@ New Features
* Added support for matching on IPv4 Time To Live and IPv6 Hop Limit.
+* **Updated the Intel ice driver.**
+
+ Updated the Intel ice driver with new features and improvements, including:
+
+ * Added support for DCF (Device Config Function) feature.
Removed Items
-------------
diff --git a/drivers/common/Makefile b/drivers/common/Makefile
index 96bd7ac6e..df2e840cf 100644
--- a/drivers/common/Makefile
+++ b/drivers/common/Makefile
@@ -31,6 +31,7 @@ DIRS-y += dpaax
endif
IAVF-y := $(CONFIG_RTE_LIBRTE_IAVF_PMD)
+IAVF-y += $(CONFIG_RTE_LIBRTE_ICE_PMD)
ifneq (,$(findstring y,$(IAVF-y)))
DIRS-y += iavf
endif
diff --git a/drivers/net/ice/Makefile b/drivers/net/ice/Makefile
index e22c34287..f493c9ed7 100644
--- a/drivers/net/ice/Makefile
+++ b/drivers/net/ice/Makefile
@@ -11,9 +11,11 @@ LIB = librte_pmd_ice.a
CFLAGS += -O3
CFLAGS += $(WERROR_FLAGS)
CFLAGS += -DALLOW_EXPERIMENTAL_API
+CFLAGS += -I$(RTE_SDK)/drivers/common/iavf
LDLIBS += -lrte_eal -lrte_mbuf -lrte_ethdev -lrte_kvargs
LDLIBS += -lrte_bus_pci -lrte_mempool -lrte_hash
+LDLIBS += -lrte_net -lrte_common_iavf
EXPORT_MAP := rte_pmd_ice_version.map
@@ -84,6 +86,9 @@ ifeq ($(CC_AVX2_SUPPORT), 1)
endif
SRCS-$(CONFIG_RTE_LIBRTE_ICE_PMD) += ice_generic_flow.c
+SRCS-$(CONFIG_RTE_LIBRTE_ICE_PMD) += ice_dcf.c
+SRCS-$(CONFIG_RTE_LIBRTE_ICE_PMD) += ice_dcf_ethdev.c
+
# install this header file
SYMLINK-$(CONFIG_RTE_LIBRTE_ICE_PMD)-include := rte_pmd_ice.h
diff --git a/drivers/net/ice/ice_dcf.c b/drivers/net/ice/ice_dcf.c
new file mode 100644
index 000000000..efb258a5a
--- /dev/null
+++ b/drivers/net/ice/ice_dcf.c
@@ -0,0 +1,474 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#include <sys/queue.h>
+#include <stdio.h>
+#include <errno.h>
+#include <stdint.h>
+#include <string.h>
+#include <unistd.h>
+#include <stdarg.h>
+#include <inttypes.h>
+#include <rte_byteorder.h>
+#include <rte_common.h>
+
+#include <rte_pci.h>
+#include <rte_atomic.h>
+#include <rte_eal.h>
+#include <rte_ether.h>
+#include <rte_ethdev_driver.h>
+#include <rte_ethdev_pci.h>
+#include <rte_malloc.h>
+#include <rte_memzone.h>
+#include <rte_dev.h>
+
+#include "ice_dcf.h"
+
+#define ICE_DCF_AQ_LEN 32
+#define ICE_DCF_AQ_BUF_SZ 4096
+
+#define ICE_DCF_ARQ_MAX_RETRIES 200
+#define ICE_DCF_ARQ_CHECK_TIME 2 /* msecs */
+
+#define ICE_DCF_VF_RES_BUF_SZ \
+ (sizeof(struct virtchnl_vf_resource) + \
+ IAVF_MAX_VF_VSI * sizeof(struct virtchnl_vsi_resource))
+
+static __rte_always_inline int
+ice_dcf_send_cmd_req_no_irq(struct ice_dcf_hw *hw, enum virtchnl_ops op,
+ uint8_t *req_msg, uint16_t req_msglen)
+{
+ return iavf_aq_send_msg_to_pf(&hw->avf, op, IAVF_SUCCESS,
+ req_msg, req_msglen, NULL);
+}
+
+static int
+ice_dcf_recv_cmd_rsp_no_irq(struct ice_dcf_hw *hw, enum virtchnl_ops op,
+ uint8_t *rsp_msgbuf, uint16_t rsp_buflen,
+ uint16_t *rsp_msglen)
+{
+ struct iavf_arq_event_info event;
+ enum virtchnl_ops v_op;
+ int i = 0;
+ int err;
+
+ event.buf_len = rsp_buflen;
+ event.msg_buf = rsp_msgbuf;
+
+ do {
+ err = iavf_clean_arq_element(&hw->avf, &event, NULL);
+ if (err != IAVF_SUCCESS)
+ goto again;
+
+ v_op = rte_le_to_cpu_32(event.desc.cookie_high);
+ if (v_op != op)
+ goto again;
+
+ if (rsp_msglen != NULL)
+ *rsp_msglen = event.msg_len;
+ return rte_le_to_cpu_32(event.desc.cookie_low);
+
+again:
+ rte_delay_ms(ICE_DCF_ARQ_CHECK_TIME);
+ } while (i++ < ICE_DCF_ARQ_MAX_RETRIES);
+
+ return -EIO;
+}
+
+static __rte_always_inline void
+ice_dcf_aq_cmd_clear(struct ice_dcf_hw *hw, struct dcf_virtchnl_cmd *cmd)
+{
+ rte_spinlock_lock(&hw->vc_cmd_queue_lock);
+
+ TAILQ_REMOVE(&hw->vc_cmd_queue, cmd, next);
+
+ rte_spinlock_unlock(&hw->vc_cmd_queue_lock);
+}
+
+static __rte_always_inline void
+ice_dcf_vc_cmd_set(struct ice_dcf_hw *hw, struct dcf_virtchnl_cmd *cmd)
+{
+ cmd->v_ret = IAVF_ERR_NOT_READY;
+ cmd->rsp_msglen = 0;
+ cmd->pending = 1;
+
+ rte_spinlock_lock(&hw->vc_cmd_queue_lock);
+
+ TAILQ_INSERT_TAIL(&hw->vc_cmd_queue, cmd, next);
+
+ rte_spinlock_unlock(&hw->vc_cmd_queue_lock);
+}
+
+static __rte_always_inline int
+ice_dcf_vc_cmd_send(struct ice_dcf_hw *hw, struct dcf_virtchnl_cmd *cmd)
+{
+ return iavf_aq_send_msg_to_pf(&hw->avf,
+ cmd->v_op, IAVF_SUCCESS,
+ cmd->req_msg, cmd->req_msglen, NULL);
+}
+
+static __rte_always_inline void
+ice_dcf_aq_cmd_handle(struct ice_dcf_hw *hw, struct iavf_arq_event_info *info)
+{
+ struct dcf_virtchnl_cmd *cmd;
+ enum virtchnl_ops v_op;
+ enum iavf_status v_ret;
+ uint16_t aq_op;
+
+ aq_op = rte_le_to_cpu_16(info->desc.opcode);
+ if (unlikely(aq_op != iavf_aqc_opc_send_msg_to_vf)) {
+ PMD_DRV_LOG(ERR,
+ "Request %u is not supported yet", aq_op);
+ return;
+ }
+
+ v_op = rte_le_to_cpu_32(info->desc.cookie_high);
+ if (unlikely(v_op == VIRTCHNL_OP_EVENT))
+ return;
+
+ v_ret = rte_le_to_cpu_32(info->desc.cookie_low);
+
+ rte_spinlock_lock(&hw->vc_cmd_queue_lock);
+
+ TAILQ_FOREACH(cmd, &hw->vc_cmd_queue, next) {
+ if (cmd->v_op == v_op && cmd->pending) {
+ cmd->v_ret = v_ret;
+ cmd->rsp_msglen = RTE_MIN(info->msg_len,
+ cmd->rsp_buflen);
+ if (likely(cmd->rsp_msglen != 0))
+ rte_memcpy(cmd->rsp_msgbuf, info->msg_buf,
+ cmd->rsp_msglen);
+
+ /* prevent compiler reordering */
+ rte_compiler_barrier();
+ cmd->pending = 0;
+ break;
+ }
+ }
+
+ rte_spinlock_unlock(&hw->vc_cmd_queue_lock);
+}
+
+static void
+ice_dcf_handle_virtchnl_msg(struct ice_dcf_hw *hw)
+{
+ struct iavf_arq_event_info info;
+ uint16_t pending = 1;
+ int ret;
+
+ info.buf_len = ICE_DCF_AQ_BUF_SZ;
+ info.msg_buf = hw->arq_buf;
+
+ while (pending) {
+ ret = iavf_clean_arq_element(&hw->avf, &info, &pending);
+ if (ret != IAVF_SUCCESS)
+ break;
+
+ ice_dcf_aq_cmd_handle(hw, &info);
+ }
+}
+
+static int
+ice_dcf_init_check_api_version(struct ice_dcf_hw *hw)
+{
+#define ICE_CPF_VIRTCHNL_VERSION_MAJOR_START 1
+#define ICE_CPF_VIRTCHNL_VERSION_MINOR_START 1
+ struct virtchnl_version_info version, *pver;
+ int err;
+
+ version.major = VIRTCHNL_VERSION_MAJOR;
+ version.minor = VIRTCHNL_VERSION_MINOR;
+ err = ice_dcf_send_cmd_req_no_irq(hw, VIRTCHNL_OP_VERSION,
+ (uint8_t *)&version, sizeof(version));
+ if (err) {
+ PMD_INIT_LOG(ERR, "Fail to send OP_VERSION");
+ return err;
+ }
+
+ pver = &hw->virtchnl_version;
+ err = ice_dcf_recv_cmd_rsp_no_irq(hw, VIRTCHNL_OP_VERSION,
+ (uint8_t *)pver, sizeof(*pver), NULL);
+ if (err) {
+ PMD_INIT_LOG(ERR, "Fail to get response of OP_VERSION");
+ return -1;
+ }
+
+ PMD_INIT_LOG(DEBUG,
+ "Peer PF API version: %u.%u", pver->major, pver->minor);
+
+ if (pver->major < ICE_CPF_VIRTCHNL_VERSION_MAJOR_START ||
+ (pver->major == ICE_CPF_VIRTCHNL_VERSION_MAJOR_START &&
+ pver->minor < ICE_CPF_VIRTCHNL_VERSION_MINOR_START)) {
+ PMD_INIT_LOG(ERR,
+ "VIRTCHNL API version should not be lower than (%u.%u)",
+ ICE_CPF_VIRTCHNL_VERSION_MAJOR_START,
+ ICE_CPF_VIRTCHNL_VERSION_MAJOR_START);
+ return -1;
+ } else if (pver->major > VIRTCHNL_VERSION_MAJOR ||
+ (pver->major == VIRTCHNL_VERSION_MAJOR &&
+ pver->minor > VIRTCHNL_VERSION_MINOR)) {
+ PMD_INIT_LOG(ERR,
+ "PF/VF API version mismatch:(%u.%u)-(%u.%u)",
+ pver->major, pver->minor,
+ VIRTCHNL_VERSION_MAJOR, VIRTCHNL_VERSION_MINOR);
+ return -1;
+ }
+
+ PMD_INIT_LOG(DEBUG, "Peer is supported PF host");
+
+ return 0;
+}
+
+static int
+ice_dcf_get_vf_resource(struct ice_dcf_hw *hw)
+{
+ uint32_t caps;
+ int err, i;
+
+ caps = VIRTCHNL_VF_OFFLOAD_WB_ON_ITR | VIRTCHNL_VF_OFFLOAD_RX_POLLING |
+ VIRTCHNL_VF_CAP_ADV_LINK_SPEED |
+ VF_BASE_MODE_OFFLOADS;
+
+ err = ice_dcf_send_cmd_req_no_irq(hw, VIRTCHNL_OP_GET_VF_RESOURCES,
+ (uint8_t *)&caps, sizeof(caps));
+ if (err) {
+ PMD_DRV_LOG(ERR, "Fail to send msg OP_GET_VF_RESOURCE");
+ return err;
+ }
+
+ err = ice_dcf_recv_cmd_rsp_no_irq(hw, VIRTCHNL_OP_GET_VF_RESOURCES,
+ (uint8_t *)hw->vf_res,
+ ICE_DCF_VF_RES_BUF_SZ, NULL);
+ if (err) {
+ PMD_DRV_LOG(ERR, "Fail to get response of OP_GET_VF_RESOURCE");
+ return -1;
+ }
+
+ iavf_vf_parse_hw_config(&hw->avf, hw->vf_res);
+
+ hw->vsi_res = NULL;
+ for (i = 0; i < hw->vf_res->num_vsis; i++) {
+ if (hw->vf_res->vsi_res[i].vsi_type == VIRTCHNL_VSI_SRIOV)
+ hw->vsi_res = &hw->vf_res->vsi_res[i];
+ }
+
+ if (!hw->vsi_res) {
+ PMD_DRV_LOG(ERR, "no LAN VSI found");
+ return -1;
+ }
+
+ hw->vsi_id = hw->vsi_res->vsi_id;
+ PMD_DRV_LOG(DEBUG, "VSI ID is %u", hw->vsi_id);
+
+ return 0;
+}
+
+static int
+ice_dcf_check_reset_done(struct ice_dcf_hw *hw)
+{
+#define ICE_DCF_RESET_WAIT_CNT 50
+ struct iavf_hw *avf = &hw->avf;
+ int i, reset;
+
+ for (i = 0; i < ICE_DCF_RESET_WAIT_CNT; i++) {
+ reset = IAVF_READ_REG(avf, IAVF_VFGEN_RSTAT) &
+ IAVF_VFGEN_RSTAT_VFR_STATE_MASK;
+ reset = reset >> IAVF_VFGEN_RSTAT_VFR_STATE_SHIFT;
+
+ if (reset == VIRTCHNL_VFR_VFACTIVE ||
+ reset == VIRTCHNL_VFR_COMPLETED)
+ break;
+
+ rte_delay_ms(20);
+ }
+
+ if (i >= ICE_DCF_RESET_WAIT_CNT)
+ return -1;
+
+ return 0;
+}
+
+static inline void
+ice_dcf_enable_irq0(struct ice_dcf_hw *hw)
+{
+ struct iavf_hw *avf = &hw->avf;
+
+ /* Enable admin queue interrupt trigger */
+ IAVF_WRITE_REG(avf, IAVF_VFINT_ICR0_ENA1,
+ IAVF_VFINT_ICR0_ENA1_ADMINQ_MASK);
+ IAVF_WRITE_REG(avf, IAVF_VFINT_DYN_CTL01,
+ IAVF_VFINT_DYN_CTL01_INTENA_MASK |
+ IAVF_VFINT_DYN_CTL01_CLEARPBA_MASK |
+ IAVF_VFINT_DYN_CTL01_ITR_INDX_MASK);
+
+ IAVF_WRITE_FLUSH(avf);
+}
+
+static inline void
+ice_dcf_disable_irq0(struct ice_dcf_hw *hw)
+{
+ struct iavf_hw *avf = &hw->avf;
+
+ /* Disable all interrupt types */
+ IAVF_WRITE_REG(avf, IAVF_VFINT_ICR0_ENA1, 0);
+ IAVF_WRITE_REG(avf, IAVF_VFINT_DYN_CTL01,
+ IAVF_VFINT_DYN_CTL01_ITR_INDX_MASK);
+
+ IAVF_WRITE_FLUSH(avf);
+}
+
+static void
+ice_dcf_dev_interrupt_handler(void *param)
+{
+ struct ice_dcf_hw *hw = param;
+
+ ice_dcf_disable_irq0(hw);
+
+ ice_dcf_handle_virtchnl_msg(hw);
+
+ ice_dcf_enable_irq0(hw);
+}
+
+int
+ice_dcf_execute_virtchnl_cmd(struct ice_dcf_hw *hw,
+ struct dcf_virtchnl_cmd *cmd)
+{
+ int i = 0;
+ int err;
+
+ if ((cmd->req_msg && !cmd->req_msglen) ||
+ (!cmd->req_msg && cmd->req_msglen) ||
+ (cmd->rsp_msgbuf && !cmd->rsp_buflen) ||
+ (!cmd->rsp_msgbuf && cmd->rsp_buflen))
+ return -EINVAL;
+
+ rte_spinlock_lock(&hw->vc_cmd_send_lock);
+ ice_dcf_vc_cmd_set(hw, cmd);
+
+ err = ice_dcf_vc_cmd_send(hw, cmd);
+ if (err) {
+ PMD_DRV_LOG(ERR, "fail to send cmd %d", cmd->v_op);
+ goto ret;
+ }
+
+ do {
+ if (!cmd->pending)
+ break;
+
+ rte_delay_ms(ICE_DCF_ARQ_CHECK_TIME);
+ } while (i++ < ICE_DCF_ARQ_MAX_RETRIES);
+
+ if (cmd->v_ret != IAVF_SUCCESS) {
+ err = -1;
+ PMD_DRV_LOG(ERR,
+ "No response (%d times) or return failure (%d) for cmd %d",
+ i, cmd->v_ret, cmd->v_op);
+ }
+
+ret:
+ ice_dcf_aq_cmd_clear(hw, cmd);
+ rte_spinlock_unlock(&hw->vc_cmd_send_lock);
+ return err;
+}
+
+int
+ice_dcf_init_hw(struct rte_eth_dev *eth_dev, struct ice_dcf_hw *hw)
+{
+ struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ int ret;
+
+ hw->avf.hw_addr = pci_dev->mem_resource[0].addr;
+ hw->avf.back = hw;
+
+ hw->avf.bus.bus_id = pci_dev->addr.bus;
+ hw->avf.bus.device = pci_dev->addr.devid;
+ hw->avf.bus.func = pci_dev->addr.function;
+
+ hw->avf.device_id = pci_dev->id.device_id;
+ hw->avf.vendor_id = pci_dev->id.vendor_id;
+ hw->avf.subsystem_device_id = pci_dev->id.subsystem_device_id;
+ hw->avf.subsystem_vendor_id = pci_dev->id.subsystem_vendor_id;
+
+ hw->avf.aq.num_arq_entries = ICE_DCF_AQ_LEN;
+ hw->avf.aq.num_asq_entries = ICE_DCF_AQ_LEN;
+ hw->avf.aq.arq_buf_size = ICE_DCF_AQ_BUF_SZ;
+ hw->avf.aq.asq_buf_size = ICE_DCF_AQ_BUF_SZ;
+
+ rte_spinlock_init(&hw->vc_cmd_send_lock);
+ rte_spinlock_init(&hw->vc_cmd_queue_lock);
+ TAILQ_INIT(&hw->vc_cmd_queue);
+
+ hw->arq_buf = rte_zmalloc("arq_buf", ICE_DCF_AQ_BUF_SZ, 0);
+ if (hw->arq_buf == NULL) {
+ PMD_INIT_LOG(ERR, "unable to allocate AdminQ buffer memory");
+ goto err;
+ }
+
+ ret = iavf_set_mac_type(&hw->avf);
+ if (ret) {
+ PMD_INIT_LOG(ERR, "set_mac_type failed: %d", ret);
+ goto err;
+ }
+
+ ret = ice_dcf_check_reset_done(hw);
+ if (ret) {
+ PMD_INIT_LOG(ERR, "VF is still resetting");
+ goto err;
+ }
+
+ ret = iavf_init_adminq(&hw->avf);
+ if (ret) {
+ PMD_INIT_LOG(ERR, "init_adminq failed: %d", ret);
+ goto err;
+ }
+
+ if (ice_dcf_init_check_api_version(hw)) {
+ PMD_INIT_LOG(ERR, "check_api version failed");
+ goto err_api;
+ }
+
+ hw->vf_res = rte_zmalloc("vf_res", ICE_DCF_VF_RES_BUF_SZ, 0);
+ if (hw->vf_res == NULL) {
+ PMD_INIT_LOG(ERR, "unable to allocate vf_res memory");
+ goto err_api;
+ }
+
+ if (ice_dcf_get_vf_resource(hw)) {
+ PMD_INIT_LOG(ERR, "Failed to get VF resource");
+ goto err_alloc;
+ }
+
+ rte_intr_callback_register(&pci_dev->intr_handle,
+ ice_dcf_dev_interrupt_handler, hw);
+ rte_intr_enable(&pci_dev->intr_handle);
+ ice_dcf_enable_irq0(hw);
+
+ return 0;
+
+err_alloc:
+ rte_free(hw->vf_res);
+err_api:
+ iavf_shutdown_adminq(&hw->avf);
+err:
+ rte_free(hw->arq_buf);
+
+ return -1;
+}
+
+void
+ice_dcf_uninit_hw(struct rte_eth_dev *eth_dev, struct ice_dcf_hw *hw)
+{
+ struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_intr_handle *intr_handle = &pci_dev->intr_handle;
+
+ ice_dcf_disable_irq0(hw);
+ rte_intr_disable(intr_handle);
+ rte_intr_callback_unregister(intr_handle,
+ ice_dcf_dev_interrupt_handler, hw);
+
+ iavf_shutdown_adminq(&hw->avf);
+
+ rte_free(hw->arq_buf);
+ rte_free(hw->vf_res);
+}
diff --git a/drivers/net/ice/ice_dcf.h b/drivers/net/ice/ice_dcf.h
new file mode 100644
index 000000000..f44c09db2
--- /dev/null
+++ b/drivers/net/ice/ice_dcf.h
@@ -0,0 +1,52 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#ifndef _ICE_DCF_H_
+#define _ICE_DCF_H_
+
+#include <rte_ethdev_driver.h>
+
+#include <iavf_prototype.h>
+#include <iavf_adminq_cmd.h>
+#include <iavf_type.h>
+
+#include "ice_logs.h"
+
+struct dcf_virtchnl_cmd {
+ TAILQ_ENTRY(dcf_virtchnl_cmd) next;
+
+ enum virtchnl_ops v_op;
+ enum iavf_status v_ret;
+
+ uint16_t req_msglen;
+ uint8_t *req_msg;
+
+ uint16_t rsp_msglen;
+ uint16_t rsp_buflen;
+ uint8_t *rsp_msgbuf;
+
+ volatile int pending;
+};
+
+struct ice_dcf_hw {
+ struct iavf_hw avf;
+
+ rte_spinlock_t vc_cmd_send_lock;
+ rte_spinlock_t vc_cmd_queue_lock;
+ TAILQ_HEAD(, dcf_virtchnl_cmd) vc_cmd_queue;
+ uint8_t *arq_buf;
+
+ struct virtchnl_version_info virtchnl_version;
+ struct virtchnl_vf_resource *vf_res; /* VF resource */
+ struct virtchnl_vsi_resource *vsi_res; /* LAN VSI */
+ uint16_t vsi_id;
+};
+
+int ice_dcf_execute_virtchnl_cmd(struct ice_dcf_hw *hw,
+ struct dcf_virtchnl_cmd *cmd);
+
+int ice_dcf_init_hw(struct rte_eth_dev *eth_dev, struct ice_dcf_hw *hw);
+void ice_dcf_uninit_hw(struct rte_eth_dev *eth_dev, struct ice_dcf_hw *hw);
+
+#endif /* _ICE_DCF_H_ */
diff --git a/drivers/net/ice/ice_dcf_ethdev.c b/drivers/net/ice/ice_dcf_ethdev.c
new file mode 100644
index 000000000..23f82a487
--- /dev/null
+++ b/drivers/net/ice/ice_dcf_ethdev.c
@@ -0,0 +1,317 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#include <errno.h>
+#include <stdbool.h>
+#include <sys/types.h>
+#include <sys/ioctl.h>
+#include <unistd.h>
+
+#include <rte_interrupts.h>
+#include <rte_debug.h>
+#include <rte_pci.h>
+#include <rte_atomic.h>
+#include <rte_eal.h>
+#include <rte_ether.h>
+#include <rte_ethdev_pci.h>
+#include <rte_kvargs.h>
+#include <rte_malloc.h>
+#include <rte_memzone.h>
+#include <rte_dev.h>
+
+#include <iavf_devids.h>
+
+#include "ice_generic_flow.h"
+#include "ice_dcf_ethdev.h"
+
+static uint16_t
+ice_dcf_recv_pkts(__rte_unused void *rx_queue,
+ __rte_unused struct rte_mbuf **bufs,
+ __rte_unused uint16_t nb_pkts)
+{
+ return 0;
+}
+
+static uint16_t
+ice_dcf_xmit_pkts(__rte_unused void *tx_queue,
+ __rte_unused struct rte_mbuf **bufs,
+ __rte_unused uint16_t nb_pkts)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_start(struct rte_eth_dev *dev)
+{
+ dev->data->dev_link.link_status = ETH_LINK_UP;
+
+ return 0;
+}
+
+static void
+ice_dcf_dev_stop(struct rte_eth_dev *dev)
+{
+ dev->data->dev_link.link_status = ETH_LINK_DOWN;
+}
+
+static int
+ice_dcf_dev_configure(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_info_get(struct rte_eth_dev *dev,
+ struct rte_eth_dev_info *dev_info)
+{
+ struct ice_dcf_adapter *adapter = dev->data->dev_private;
+
+ dev_info->max_mac_addrs = 1;
+ dev_info->max_rx_pktlen = (uint32_t)-1;
+ dev_info->max_rx_queues = RTE_DIM(adapter->rxqs);
+ dev_info->max_tx_queues = RTE_DIM(adapter->txqs);
+
+ return 0;
+}
+
+static int
+ice_dcf_stats_get(__rte_unused struct rte_eth_dev *dev,
+ __rte_unused struct rte_eth_stats *igb_stats)
+{
+ return 0;
+}
+
+static int
+ice_dcf_stats_reset(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_promiscuous_enable(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_promiscuous_disable(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_allmulticast_enable(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_allmulticast_disable(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_filter_ctrl(struct rte_eth_dev *dev,
+ enum rte_filter_type filter_type,
+ __rte_unused enum rte_filter_op filter_op,
+ __rte_unused void *arg)
+{
+ int ret = 0;
+
+ if (!dev)
+ return -EINVAL;
+
+ switch (filter_type) {
+ default:
+ PMD_DRV_LOG(WARNING, "Filter type (%d) not supported",
+ filter_type);
+ ret = -EINVAL;
+ break;
+ }
+
+ return ret;
+}
+
+static void
+ice_dcf_dev_close(struct rte_eth_dev *dev)
+{
+ struct ice_dcf_adapter *adapter = dev->data->dev_private;
+
+ if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+ return;
+
+ dev->dev_ops = NULL;
+ dev->rx_pkt_burst = NULL;
+ dev->tx_pkt_burst = NULL;
+ dev->data->mac_addrs = NULL;
+
+ ice_dcf_uninit_hw(dev, &adapter->real_hw);
+}
+
+static void
+ice_dcf_queue_release(__rte_unused void *q)
+{
+}
+
+static int
+ice_dcf_link_update(__rte_unused struct rte_eth_dev *dev,
+ __rte_unused int wait_to_complete)
+{
+ return 0;
+}
+
+static int
+ice_dcf_rx_queue_setup(struct rte_eth_dev *dev,
+ uint16_t rx_queue_id,
+ __rte_unused uint16_t nb_rx_desc,
+ __rte_unused unsigned int socket_id,
+ __rte_unused const struct rte_eth_rxconf *rx_conf,
+ __rte_unused struct rte_mempool *mb_pool)
+{
+ struct ice_dcf_adapter *adapter = dev->data->dev_private;
+
+ dev->data->rx_queues[rx_queue_id] = &adapter->rxqs[rx_queue_id];
+
+ return 0;
+}
+
+static int
+ice_dcf_tx_queue_setup(struct rte_eth_dev *dev,
+ uint16_t tx_queue_id,
+ __rte_unused uint16_t nb_tx_desc,
+ __rte_unused unsigned int socket_id,
+ __rte_unused const struct rte_eth_txconf *tx_conf)
+{
+ struct ice_dcf_adapter *adapter = dev->data->dev_private;
+
+ dev->data->tx_queues[tx_queue_id] = &adapter->txqs[tx_queue_id];
+
+ return 0;
+}
+
+static const struct eth_dev_ops ice_dcf_eth_dev_ops = {
+ .dev_start = ice_dcf_dev_start,
+ .dev_stop = ice_dcf_dev_stop,
+ .dev_close = ice_dcf_dev_close,
+ .dev_configure = ice_dcf_dev_configure,
+ .dev_infos_get = ice_dcf_dev_info_get,
+ .rx_queue_setup = ice_dcf_rx_queue_setup,
+ .tx_queue_setup = ice_dcf_tx_queue_setup,
+ .rx_queue_release = ice_dcf_queue_release,
+ .tx_queue_release = ice_dcf_queue_release,
+ .link_update = ice_dcf_link_update,
+ .stats_get = ice_dcf_stats_get,
+ .stats_reset = ice_dcf_stats_reset,
+ .promiscuous_enable = ice_dcf_dev_promiscuous_enable,
+ .promiscuous_disable = ice_dcf_dev_promiscuous_disable,
+ .allmulticast_enable = ice_dcf_dev_allmulticast_enable,
+ .allmulticast_disable = ice_dcf_dev_allmulticast_disable,
+ .filter_ctrl = ice_dcf_dev_filter_ctrl,
+};
+
+static int
+ice_dcf_dev_init(struct rte_eth_dev *eth_dev)
+{
+ struct ice_dcf_adapter *adapter = eth_dev->data->dev_private;
+
+ eth_dev->dev_ops = &ice_dcf_eth_dev_ops;
+ eth_dev->rx_pkt_burst = ice_dcf_recv_pkts;
+ eth_dev->tx_pkt_burst = ice_dcf_xmit_pkts;
+
+ if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+ return 0;
+
+ eth_dev->data->dev_flags |= RTE_ETH_DEV_CLOSE_REMOVE;
+
+ if (ice_dcf_init_hw(eth_dev, &adapter->real_hw) != 0) {
+ PMD_INIT_LOG(ERR, "Failed to init DCF hardware");
+ return -1;
+ }
+
+ rte_eth_random_addr(adapter->mac_addr.addr_bytes);
+ eth_dev->data->mac_addrs = &adapter->mac_addr;
+
+ return 0;
+}
+
+static int
+ice_dcf_dev_uninit(struct rte_eth_dev *eth_dev)
+{
+ ice_dcf_dev_close(eth_dev);
+
+ return 0;
+}
+
+static int
+ice_dcf_cap_check_handler(__rte_unused const char *key,
+ const char *value, __rte_unused void *opaque)
+{
+ if (strcmp(value, "dcf"))
+ return -1;
+
+ return 0;
+}
+
+static int
+ice_dcf_cap_selected(struct rte_devargs *devargs)
+{
+ struct rte_kvargs *kvlist;
+ const char *key = "cap";
+ int ret = 0;
+
+ if (devargs == NULL)
+ return 0;
+
+ kvlist = rte_kvargs_parse(devargs->args, NULL);
+ if (kvlist == NULL)
+ return 0;
+
+ if (!rte_kvargs_count(kvlist, key))
+ goto exit;
+
+ /* dcf capability selected when there's a key-value pair: cap=dcf */
+ if (rte_kvargs_process(kvlist, key,
+ ice_dcf_cap_check_handler, NULL) < 0)
+ goto exit;
+
+ ret = 1;
+
+exit:
+ rte_kvargs_free(kvlist);
+ return ret;
+}
+
+static int eth_ice_dcf_pci_probe(__rte_unused struct rte_pci_driver *pci_drv,
+ struct rte_pci_device *pci_dev)
+{
+ if (!ice_dcf_cap_selected(pci_dev->device.devargs))
+ return 1;
+
+ return rte_eth_dev_pci_generic_probe(pci_dev,
+ sizeof(struct ice_dcf_adapter),
+ ice_dcf_dev_init);
+}
+
+static int eth_ice_dcf_pci_remove(struct rte_pci_device *pci_dev)
+{
+ return rte_eth_dev_pci_generic_remove(pci_dev, ice_dcf_dev_uninit);
+}
+
+static const struct rte_pci_id pci_id_ice_dcf_map[] = {
+ { RTE_PCI_DEVICE(IAVF_INTEL_VENDOR_ID, IAVF_DEV_ID_ADAPTIVE_VF) },
+ { .vendor_id = 0, /* sentinel */ },
+};
+
+static struct rte_pci_driver rte_ice_dcf_pmd = {
+ .id_table = pci_id_ice_dcf_map,
+ .drv_flags = RTE_PCI_DRV_NEED_MAPPING,
+ .probe = eth_ice_dcf_pci_probe,
+ .remove = eth_ice_dcf_pci_remove,
+};
+
+RTE_PMD_REGISTER_PCI(net_ice_dcf, rte_ice_dcf_pmd);
+RTE_PMD_REGISTER_PCI_TABLE(net_ice_dcf, pci_id_ice_dcf_map);
+RTE_PMD_REGISTER_KMOD_DEP(net_ice_dcf, "* igb_uio | vfio-pci");
+RTE_PMD_REGISTER_PARAM_STRING(net_ice_dcf, "cap=dcf");
diff --git a/drivers/net/ice/ice_dcf_ethdev.h b/drivers/net/ice/ice_dcf_ethdev.h
new file mode 100644
index 000000000..0c34a0095
--- /dev/null
+++ b/drivers/net/ice/ice_dcf_ethdev.h
@@ -0,0 +1,24 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#ifndef _ICE_DCF_ETHDEV_H_
+#define _ICE_DCF_ETHDEV_H_
+
+#include "ice_ethdev.h"
+#include "ice_dcf.h"
+
+#define ICE_DCF_MAX_RINGS 1
+
+struct ice_dcf_queue {
+ uint64_t dummy;
+};
+
+struct ice_dcf_adapter {
+ struct ice_dcf_hw real_hw;
+ struct rte_ether_addr mac_addr;
+ struct ice_dcf_queue rxqs[ICE_DCF_MAX_RINGS];
+ struct ice_dcf_queue txqs[ICE_DCF_MAX_RINGS];
+};
+
+#endif /* _ICE_DCF_ETHDEV_H_ */
diff --git a/drivers/net/ice/meson.build b/drivers/net/ice/meson.build
index f9e897bbc..0ba9668d1 100644
--- a/drivers/net/ice/meson.build
+++ b/drivers/net/ice/meson.build
@@ -15,8 +15,8 @@ sources = files(
'ice_hash.c'
)
-deps += ['hash']
-includes += include_directories('base')
+deps += ['hash', 'net', 'common_iavf']
+includes += include_directories('base', '../../common/iavf')
if arch_subdir == 'x86'
sources += files('ice_rxtx_vec_sse.c')
@@ -37,4 +37,7 @@ if arch_subdir == 'x86'
endif
endif
+sources += files('ice_dcf.c',
+ 'ice_dcf_ethdev.c')
+
install_headers('rte_pmd_ice.h')
diff --git a/mk/rte.app.mk b/mk/rte.app.mk
index d295ca0a5..f3798a09f 100644
--- a/mk/rte.app.mk
+++ b/mk/rte.app.mk
@@ -185,6 +185,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_I40E_PMD) += -lrte_pmd_i40e
_LDLIBS-$(CONFIG_RTE_LIBRTE_IAVF_PMD) += -lrte_pmd_iavf
_LDLIBS-$(CONFIG_RTE_LIBRTE_ICE_PMD) += -lrte_pmd_ice
IAVF-y := $(CONFIG_RTE_LIBRTE_IAVF_PMD)
+IAVF-y += $(CONFIG_RTE_LIBRTE_ICE_PMD)
ifeq ($(findstring y,$(IAVF-y)),y)
_LDLIBS-y += -lrte_common_iavf
endif
--
2.25.1
^ permalink raw reply [relevance 1%]
* Re: [dpdk-dev] dev Digest, Vol 288, Issue 27
[not found] ` <d8993cb75592406a993e3119d45e906a@intel.com>
@ 2020-03-10 2:55 0% ` Zhang, XiX
0 siblings, 0 replies; 200+ results
From: Zhang, XiX @ 2020-03-10 2:55 UTC (permalink / raw)
To: dev
[PATCH] vfio: map contiguous areas in one go
Test-by : xix.zhang@intel.com
Thanks
> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of dev-
> request@dpdk.org
> Sent: February 25, 2020 21:50
> To: dev@dpdk.org
> Subject: dev Digest, Vol 288, Issue 27
>
> Send dev mailing list submissions to
> dev@dpdk.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> https://mails.dpdk.org/listinfo/dev
> or, via email, send a message with subject or body 'help' to
> dev-request@dpdk.org
>
> You can reach the person managing the list at
> dev-owner@dpdk.org
>
> When replying, please edit your Subject line so it is more specific than "Re:
> Contents of dev digest..."
>
>
> Today's Topics:
>
> 1. Re: [PATCH] doc/mlx5: update mlx5 guide (Thomas Monjalon)
> 2. Re: [PATCH] doc: describe the pktmbuf pool with pinned
> extarnal memory (Thomas Monjalon)
> 3. [PATCH] vfio: map contiguous areas in one go (Anatoly Burakov)
> 4. Re: [RFC 0/6] New sync modes for ring (Ananyev, Konstantin)
> 5. Re: [PATCH] vfio: map contiguous areas in one go (Ray Kinsella)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Tue, 25 Feb 2020 14:19:10 +0100
> From: Thomas Monjalon <thomas@monjalon.net>
> To: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
> Cc: dev@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH] doc/mlx5: update mlx5 guide
> Message-ID: <2125420.IFkqi6BYcA@xps>
> Content-Type: text/plain; charset="us-ascii"
>
> 24/02/2020 18:57, Viacheslav Ovsiienko:
> > - metadata limitation is described
> > - no inline hint flag is described
> >
> > Signed-off-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
> > ---
>
> Split in 2 patches and applied, thanks
>
>
>
>
>
> ------------------------------
>
> Message: 2
> Date: Tue, 25 Feb 2020 14:22:13 +0100
> From: Thomas Monjalon <thomas@monjalon.net>
> To: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
> Cc: dev@dpdk.org, olivier.matz@6wind.com
> Subject: Re: [dpdk-dev] [PATCH] doc: describe the pktmbuf pool with
> pinned extarnal memory
> Message-ID: <13189717.lVVuGzaMjS@xps>
> Content-Type: text/plain; charset="us-ascii"
>
> 24/02/2020 18:58, Viacheslav Ovsiienko:
> > Document the new mbuf pools with external pinned buffers.
> >
> > Signed-off-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
> > ---
> > doc/guides/prog_guide/mbuf_lib.rst | 34
> ++++++++++++++++++++++++++++++++++
>
> Please submit this documentation separately for review by Olivier.
>
> > doc/guides/rel_notes/release_20_02.rst | 5 +++++
>
> Applying only the release notes in 20.02, thanks
>
>
>
>
>
> ------------------------------
>
> Message: 3
> Date: Tue, 25 Feb 2020 13:24:48 +0000
> From: Anatoly Burakov <anatoly.burakov@intel.com>
> To: dev@dpdk.org
> Subject: [dpdk-dev] [PATCH] vfio: map contiguous areas in one go
> Message-ID:
> <c6550c8e66dbc7a54a0b495ecda58b0eb79c07ca.1582637079.git.anat
> oly.burakov@intel.com>
>
>
> Currently, when we are creating DMA mappings for memory that's either
> external or is backed by hugepages in IOVA as PA mode, we assume that
> each page is necessarily discontiguous. This may not actually be the
> case, especially for external memory, where the user is able to create
> their own IOVA table and make it contiguous. This is a problem because
> VFIO has a limited number of DMA mappings, and it does not appear to
> concatenate them and treats each mapping as separate, even when they
> cover adjacent areas.
>
> Fix this so that we always map contiguous memory in a single chunk, as
> opposed to mapping each segment separately.
>
> Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
> ---
> lib/librte_eal/linux/eal/eal_vfio.c | 59
> +++++++++++++++++++++++++----
> 1 file changed, 51 insertions(+), 8 deletions(-)
>
> diff --git a/lib/librte_eal/linux/eal/eal_vfio.c
> b/lib/librte_eal/linux/eal/eal_vfio.c
> index 01b5ef3f42..4502aefed3 100644
> --- a/lib/librte_eal/linux/eal/eal_vfio.c
> +++ b/lib/librte_eal/linux/eal/eal_vfio.c
> @@ -514,9 +514,11 @@ static void
> vfio_mem_event_callback(enum rte_mem_event type, const void *addr,
> size_t len,
> void *arg __rte_unused)
> {
> + rte_iova_t iova_start, iova_expected;
> struct rte_memseg_list *msl;
> struct rte_memseg *ms;
> size_t cur_len = 0;
> + uint64_t va_start;
>
> msl = rte_mem_virt2memseg_list(addr);
>
> @@ -545,22 +547,63 @@ vfio_mem_event_callback(enum rte_mem_event type,
> const void *addr, size_t len, #endif
> /* memsegs are contiguous in memory */
> ms = rte_mem_virt2memseg(addr, msl);
> +
> + /*
> + * This memory is not guaranteed to be contiguous, but it still could
> + * be, or it could have some small contiguous chunks. Since the
> number
> + * of VFIO mappings is limited, and VFIO appears to not concatenate
> + * adjacent mappings, we have to do this ourselves.
> + *
> + * So, find contiguous chunks, then map them.
> + */
> + va_start = ms->addr_64;
> + iova_start = iova_expected = ms->iova;
> while (cur_len < len) {
> + bool new_contig_area = ms->iova != iova_expected;
> + bool last_seg = (len - cur_len) == ms->len;
> + bool skip_last = false;
> +
> + /* only do mappings when current contiguous area ends */
> + if (new_contig_area) {
> + if (type == RTE_MEM_EVENT_ALLOC)
> + vfio_dma_mem_map(default_vfio_cfg,
> va_start,
> + iova_start,
> + iova_expected - iova_start, 1);
> + else
> + vfio_dma_mem_map(default_vfio_cfg,
> va_start,
> + iova_start,
> + iova_expected - iova_start, 0);
> + va_start = ms->addr_64;
> + iova_start = ms->iova;
> + }
> /* some memory segments may have invalid IOVA */
> if (ms->iova == RTE_BAD_IOVA) {
> RTE_LOG(DEBUG, EAL, "Memory segment at %p has bad IOVA,
> skipping\n",
> ms->addr);
> - goto next;
> + skip_last = true;
> }
> - if (type == RTE_MEM_EVENT_ALLOC)
> - vfio_dma_mem_map(default_vfio_cfg, ms-
> >addr_64,
> - ms->iova, ms->len, 1);
> - else
> - vfio_dma_mem_map(default_vfio_cfg, ms-
> >addr_64,
> - ms->iova, ms->len, 0);
> -next:
> + iova_expected = ms->iova + ms->len;
> cur_len += ms->len;
> ++ms;
> +
> + /*
> + * don't count previous segment, and don't attempt to
> + * dereference a potentially invalid pointer.
> + */
> + if (skip_last && !last_seg) {
> + iova_expected = iova_start = ms->iova;
> + va_start = ms->addr_64;
> + } else if (!skip_last && last_seg) {
> + /* this is the last segment and we're not skipping */
> + if (type == RTE_MEM_EVENT_ALLOC)
> + vfio_dma_mem_map(default_vfio_cfg,
> va_start,
> + iova_start,
> + iova_expected - iova_start, 1);
> + else
> + vfio_dma_mem_map(default_vfio_cfg,
> va_start,
> + iova_start,
> + iova_expected - iova_start, 0);
> + }
> }
> #ifdef RTE_ARCH_PPC_64
> cur_len = 0;
> --
> 2.17.1
>
>
> ------------------------------
>
> Message: 4
> Date: Tue, 25 Feb 2020 13:41:14 +0000
> From: "Ananyev, Konstantin" <konstantin.ananyev@intel.com>
> To: Stephen Hemminger <stephen@networkplumber.org>, Jerin Jacob
> <jerinjacobk@gmail.com>
> Cc: dpdk-dev <dev@dpdk.org>, Olivier Matz <olivier.matz@6wind.com>,
> "drc@linux.vnet.ibm.com" <drc@linux.vnet.ibm.com>
> Subject: Re: [dpdk-dev] [RFC 0/6] New sync modes for ring
> Message-ID:
> <SN6PR11MB255845FB6665C2A2D5D76D229AED0@SN6PR11MB2558.
> namprd11.prod.outlook.com>
>
> Content-Type: text/plain; charset="us-ascii"
>
> > > > > Upfront note - that RFC is not a complete patch.
> > > > > It introduces an ABI breakage, plus it doesn't update
> > > > > ring_elem code properly, etc.
> > > > > I plan to deal with all these things in later versions.
> > > > > Right now I seek an initial feedback about proposed ideas.
> > > > > Would also ask people to repeat performance tests (see below)
> > > > > on their platforms to confirm the impact.
> > > > >
> > > > > More and more customers use(/try to use) DPDK based apps
> > > > > within overcommitted systems (multiple acttive threads over
> > > > > same pysical
> cores):
> > > > > VM, container deployments, etc.
> > > > > One quite common problem they hit: Lock-Holder-Preemption with
> rte_ring.
> > > > > LHP is quite a common problem for spin-based sync primitives
> > > > > (spin-locks, etc.) on overcommitted systems.
> > > > > The situation gets much worse when some sort of fair-locking
> > > > > technique is used (ticket-lock, etc.).
> > > > > As now not only lock-owner but also lock-waiters scheduling
> > > > > order matters a lot.
> > > > > This is a well-known problem for kernel within VMs:
> > > > > http://www-
> archive.xenproject.org/files/xensummitboston08/LHP.pdf
> > > > > https://www.cs.hs-
> rm.de/~kaiser/events/wamos2017/Slides/selcuk.pdf
> > > > > The problem with rte_ring is that while head accusion is sort
> > > > > of un-fair locking, waiting on tail is very similar to ticket
> > > > > lock schema - tail has to be updated in particular order.
> > > > > That makes current rte_ring implementation to perform really
> > > > > pure on some overcommited scenarios.
> > > >
> > > > Rather than reform rte_ring to fit this scenario, it would make
> > > > more sense to me to introduce another primitive.
>
> I don't see much advantages it will bring us.
> As a disadvantages, for developers and maintainers - code duplication,
> for end users - extra code churn and removed ability to mix and match
> different sync modes in one ring.
>
> > The current lockless
> > > > ring performs very well for the isolated thread model that DPDK
> > > > was built around. This looks like a case of customers violating
> > > > the usage model of the DPDK and then being surprised at the fallout.
>
> For customers using isolated thread model - nothing should change
> (both in terms of API and performance).
> Existing sync modes MP/MC,SP/SC kept untouched, set up in the same way
> (via flags and _init_), and MP/MC remains as default one.
> >From other side I don't see why we should ignore customers that want
> >to
> use
> their DPDK apps in different deployment scenarios.
>
> > >
> > > I agree with Stephen here.
> > >
> > > I think, adding more runtime check in the enqueue() and dequeue()
> > > will have a bad effect on the low-end cores too.
>
> We do have a run-time check in our current enqueue()/dequeue
> implementation.
> In fact we support both modes: we have generic
> rte_ring_enqueue(/dequeue)_bulk(/burst)
> where sync behaviour is determined at runtime by value of
> prod(/cons).single.
> Or user can call rte_ring_(mp/sp)_enqueue_* functions directly.
> This RFC follows exactly the same paradigm:
> rte_ring_enqueue(/dequeue)_bulk(/burst) kept generic and it's
> behaviour is determined at runtime, by value of prod(/cons).sync_type.
> Or user can call enqueue/dequeue with particular sync mode directly:
> rte_ring_(mp/sp/rts/hts)_enqueue_(bulk/burst)*.
> The only thing that changed:
> Format of prod/cons now could differ depending on mode selected at _init_.
> So you can't create a ring for let say SP mode and then in the middle
> of data- path change your mind and start using MP_RTS mode.
> For existing modes (SP/MP, SC/MC) format remains the same and user
> can still use them interchangeably, though of course that is an error
> prone practice.
>
> > > But I agree with the problem statement that in the virtualization
> > > use case, It may be possible to have N virtual cores runs on a
> > > physical core.
> > >
> > > IMO, The best solution would be keeping the ring API same and have
> > > a different flavor in "compile-time". Something like liburcu did
> > > for accommodating different flavors.
> > >
> > > i.e urcu-qsbr.h and urcu-bp.h will identical definition of API.
> > > The application can simply include ONE header file in a C file
> > > based on the flavor.
>
> I don't think it is a flexible enough approach.
> In one app user might need to have several rings with different sync modes.
> Or even user might need a ring with different sync modes for
> enqueue/dequeue.
>
> > > If need both at runtime. Need to have function pointer or so in
> > > the application and define the function in different c file by
> > > including the approaite flavor in C file.
>
> Big issue with function pointers here would be DPDK MP model.
> AFAIK, rte_ring is quite popular mechanism for IPC between DPDK apps.
> To support such model, we'll need to split rte_ring data into 'shared'
> and 'private' and initialize private one for every process that is going to use it.
> That sounds like a massive change, and I am not sure the required
> effort will worth it.
> BTW, if user just calls API functions without trying to access
> structure internals directly, I don't think it would be a big
> difference for him what is inside:
> indirect function call or inlined switch(...) {}.
>
> > This would also be a good time to consider the tradeoffs of the
> > heavy use of inlining that is done in rte_ring vs the impact that
> > has on API/ABI stability.
>
> Yes, hiding rte_ring implementation inside .c would help a lot in
> terms of ABI maintenance and would make our future life easier.
> The question is what is the price for it in terms of performance, and
> are we ready to pay it. Not to mention that it would cause changes in
> many other libs/apps...
> So I think it should be a subject for a separate discussion.
> But, agree it would be good at least to measure the performance impact
> of such change.
> If I'll have some spare cycles, will give it a try.
> Meanwhile, can I ask Jerin and other guys to repeat tests from this
> RFC on their HW? Before continuing discussion would probably be good
> to know does the suggested patch work as expected across different platforms.
> Thanks
> Konstantin
>
>
> ------------------------------
>
> Message: 5
> Date: Tue, 25 Feb 2020 13:49:38 +0000
> From: Ray Kinsella <mdr@ashroe.eu>
> To: Anatoly Burakov <anatoly.burakov@intel.com>, dev@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH] vfio: map contiguous areas in one go
> Message-ID: <276c85ed-ac2f-52c9-dffc-18ce41ab0f35@ashroe.eu>
> Content-Type: text/plain; charset=utf-8
>
> Hi Anatoly,
>
> On 25/02/2020 13:24, Anatoly Burakov wrote:
> > Currently, when we are creating DMA mappings for memory that's
> > either external or is backed by hugepages in IOVA as PA mode, we
> > assume that each page is necessarily discontiguous. This may not
> > actually be the case, especially for external memory, where the user
> > is able to create their own IOVA table and make it contiguous. This
> > is a problem because VFIO has a limited number of DMA mappings, and
> > it does not appear to concatenate them and treats each mapping as
> > separate, even when they cover adjacent areas.
> > > Fix this so that we always map contiguous memory in a single
> > chunk, as opposed to mapping each segment separately.
>
> Can I confirm my understanding.
>
> We are essentially correcting user errant behavior, trading off
> startup/mapping time to save IOMMU resources?
>
> >
> > Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
> > ---
> > lib/librte_eal/linux/eal/eal_vfio.c | 59
> > +++++++++++++++++++++++++----
> > 1 file changed, 51 insertions(+), 8 deletions(-)
> >
> > diff --git a/lib/librte_eal/linux/eal/eal_vfio.c
> b/lib/librte_eal/linux/eal/eal_vfio.c
> > index 01b5ef3f42..4502aefed3 100644
> > --- a/lib/librte_eal/linux/eal/eal_vfio.c
> > +++ b/lib/librte_eal/linux/eal/eal_vfio.c
> > @@ -514,9 +514,11 @@ static void
> > vfio_mem_event_callback(enum rte_mem_event type, const void *addr,
> size_t len,
> > void *arg __rte_unused)
> > {
> > + rte_iova_t iova_start, iova_expected;
> > struct rte_memseg_list *msl;
> > struct rte_memseg *ms;
> > size_t cur_len = 0;
> > + uint64_t va_start;
> >
> > msl = rte_mem_virt2memseg_list(addr);
> >
> > @@ -545,22 +547,63 @@ vfio_mem_event_callback(enum
> rte_mem_event type, const void *addr, size_t len,
> > #endif
> > /* memsegs are contiguous in memory */
> > ms = rte_mem_virt2memseg(addr, msl);
> > +
> > + /*
> > + * This memory is not guaranteed to be contiguous, but it still could
> > + * be, or it could have some small contiguous chunks. Since the
> number
> > + * of VFIO mappings is limited, and VFIO appears to not concatenate
> > + * adjacent mappings, we have to do this ourselves.
> > + *
> > + * So, find contiguous chunks, then map them.
> > + */
> > + va_start = ms->addr_64;
> > + iova_start = iova_expected = ms->iova;
> > while (cur_len < len) {
> > + bool new_contig_area = ms->iova != iova_expected;
> > + bool last_seg = (len - cur_len) == ms->len;
> > + bool skip_last = false;
> > +
> > + /* only do mappings when current contiguous area ends */
> > + if (new_contig_area) {
> > + if (type == RTE_MEM_EVENT_ALLOC)
> > + vfio_dma_mem_map(default_vfio_cfg,
> va_start,
> > + iova_start,
> > + iova_expected - iova_start, 1);
> > + else
> > + vfio_dma_mem_map(default_vfio_cfg,
> va_start,
> > + iova_start,
> > + iova_expected - iova_start, 0);
> > + va_start = ms->addr_64;
> > + iova_start = ms->iova;
> > + }
> > /* some memory segments may have invalid IOVA */
> > if (ms->iova == RTE_BAD_IOVA) {
> > RTE_LOG(DEBUG, EAL, "Memory segment at %p has
> bad IOVA, skipping\n",
> > ms->addr);
> > - goto next;
> > + skip_last = true;
> > }
> > - if (type == RTE_MEM_EVENT_ALLOC)
> > - vfio_dma_mem_map(default_vfio_cfg, ms-
> >addr_64,
> > - ms->iova, ms->len, 1);
> > - else
> > - vfio_dma_mem_map(default_vfio_cfg, ms-
> >addr_64,
> > - ms->iova, ms->len, 0);
> > -next:
> > + iova_expected = ms->iova + ms->len;
> > cur_len += ms->len;
> > ++ms;
> > +
> > + /*
> > + * don't count previous segment, and don't attempt to
> > + * dereference a potentially invalid pointer.
> > + */
> > + if (skip_last && !last_seg) {
> > + iova_expected = iova_start = ms->iova;
> > + va_start = ms->addr_64;
> > + } else if (!skip_last && last_seg) {
> > + /* this is the last segment and we're not skipping */
> > + if (type == RTE_MEM_EVENT_ALLOC)
> > + vfio_dma_mem_map(default_vfio_cfg,
> va_start,
> > + iova_start,
> > + iova_expected - iova_start, 1);
> > + else
> > + vfio_dma_mem_map(default_vfio_cfg,
> va_start,
> > + iova_start,
> > + iova_expected - iova_start, 0);
> > + }
> > }
> > #ifdef RTE_ARCH_PPC_64
> > cur_len = 0;
> >
>
>
> End of dev Digest, Vol 288, Issue 27
> ************************************
^ permalink raw reply [relevance 0%]
* [dpdk-dev] [PATCH v1 3/4] net/ice: add the DCF framework
@ 2020-03-09 14:14 1% ` Haiyue Wang
2 siblings, 0 replies; 200+ results
From: Haiyue Wang @ 2020-03-09 14:14 UTC (permalink / raw)
To: dev, xiaolong.ye, qi.z.zhang, qiming.yang, beilei.xing
Cc: wei.zhao1, Haiyue Wang
The DCF (Device Config Function) feature works as a standalone PMD, its
hardware entity is the trusted VF with ID 0, and the software function
is to control the flow setting of other VFs by the mailbox with PF.
It doesn't handle packet Rx/Tx related things. This PMD needs the iAVF
virtchnl to send the ICE PF's AdminQ command.
Also for security, it needs to acquire the DCF capability from PF. The
flow related things will be added later, it shares most of the ICE flow
function. This mainly handles hardware initialization etc.
.-------------.
.-| |
.-| | |
| | | iAVF PMD |
| | | |
| | | |
| | '-------------'
| '-------------'
'-------------'
^ Other VFs
/
/ Flow Distribution
.------------------. /
| |/ DCF cap req .-------------.
| CVL . <---------- | |
| Kernel | ----------> | |
. . DCF cap rsp | DCF PMD |
| | | |
'------------------' <----------- | |
rte_flow '-------------'
VF0
Signed-off-by: Haiyue Wang <haiyue.wang@intel.com>
---
doc/guides/nics/ice.rst | 47 +++
doc/guides/nics/img/ice_dcf.png | Bin 0 -> 39168 bytes
drivers/common/Makefile | 1 +
drivers/net/ice/Makefile | 6 +
drivers/net/ice/ice_dcf.c | 651 +++++++++++++++++++++++++++++++
drivers/net/ice/ice_dcf.h | 61 +++
drivers/net/ice/ice_dcf_ethdev.c | 319 +++++++++++++++
drivers/net/ice/ice_dcf_ethdev.h | 33 ++
drivers/net/ice/ice_dcf_parent.c | 348 +++++++++++++++++
drivers/net/ice/meson.build | 8 +-
mk/rte.app.mk | 1 +
11 files changed, 1473 insertions(+), 2 deletions(-)
create mode 100644 doc/guides/nics/img/ice_dcf.png
create mode 100644 drivers/net/ice/ice_dcf.c
create mode 100644 drivers/net/ice/ice_dcf.h
create mode 100644 drivers/net/ice/ice_dcf_ethdev.c
create mode 100644 drivers/net/ice/ice_dcf_ethdev.h
create mode 100644 drivers/net/ice/ice_dcf_parent.c
diff --git a/doc/guides/nics/ice.rst b/doc/guides/nics/ice.rst
index 8af32dabf..2639ae239 100644
--- a/doc/guides/nics/ice.rst
+++ b/doc/guides/nics/ice.rst
@@ -240,6 +240,53 @@ report a MDD event and drop the packets.
The APPs based on DPDK should avoid providing such packets.
+Device Config Function (DCF)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This section demonstrates ICE DCF PMD, which shares the core module with ICE
+PMD and iAVF PMD.
+
+A DCF (Device Config Function) PMD bounds to the device's trusted VF with ID 0,
+it can act as a sole controlling entity to exercise advance functionality (such
+as switch, ACL) for the rest VFs.
+
+The DCF PMD needs to advertise and acquire DCF capability which allows DCF to
+send AdminQ commands that it would like to execute over to the PF and receive
+responses for the same from PF.
+
+.. _figure_ice_dcf:
+
+.. figure:: img/ice_dcf.*
+
+ DCF Communication flow.
+
+#. Create the VFs::
+
+ echo 4 > /sys/bus/pci/devices/0000\:18\:00.0/sriov_numvfs
+
+#. Enable the VF0 trust on::
+
+ ip link set dev enp24s0f0 vf 0 trust on
+
+#. Bind the VF0, and run testpmd with 'cap=dcf' devarg::
+
+ testpmd -l 22-25 -n 4 -w 18:01.0,cap=dcf -- -i
+
+#. Monitor the VF2 interface network traffic::
+
+ tcpdump -e -nn -i enp24s1f2
+
+#. Create one flow to redirect the traffic to VF2 by DCF::
+
+ flow create 0 priority 0 ingress pattern eth / ipv4 src is 192.168.0.2 \
+ dst is 192.168.0.3 / end actions vf id 2 / end
+
+#. Send the packet, and it should be displayed on tcpdump::
+
+ sendp(Ether(src='3c:fd:fe:aa:bb:78', dst='00:00:00:01:02:03')/IP(src=' \
+ 192.168.0.2', dst="192.168.0.3")/TCP(flags='S')/Raw(load='XXXXXXXXXX'), \
+ iface="enp24s0f0", count=10)
+
Sample Application Notes
------------------------
diff --git a/doc/guides/nics/img/ice_dcf.png b/doc/guides/nics/img/ice_dcf.png
new file mode 100644
index 0000000000000000000000000000000000000000..540823c4a05e9ea9d558d233fea77016d1ae5df3
GIT binary patch
literal 39168
zcmZ6ybzD?k*FH`NDBT?jlF}e8B?Hn74BeolfOLbDz);d%Lw70NNOyNPNO%7Z_x(KY
z`+k3a@X<4KW}m&+Uh7)dy4F6yN(xdKsHCWHaBvvXpCpvw;1KwLKlqmjz$g7i%71`=
z;O&*A#NkSZ$ajH@7p7wJVsLO35oq^@NWk^0Z=W>n;oxu`p8w#NOsHJo;1Xh`CBz^u
zdIzZ}I(W`x-6a%La{^jclNOU&`_)rTS4-}zY76d1P4w1CVq)-OmrKC^5#ZY~-}pzu
zy$L?@i@zdO6jl2qp0>O_?^V0~==_TKu;0E)6AT_jK}Ys$GuU9xpl6NmUq4AAAS8rZ
z5<h>~BQL7h+1bg!^v{JkDk^GFZXt;WZK)wU@I4Lh>Nma-8Py=vX58uL5BwsNv0+TC
ztdd|ub|{f>I#qBR_r=Ecc2+(+F$m*+POs4gtgWq$;%5$Zz1W+)IolS4AR!?E!%MzF
zY^hc+(<6BKV$B>q*BptIQ5gaPfd*IJosgp=XT25!8zwm^L5M@!Q9uFabHF7#DiuCK
zt~qeo#^9j@Nyw!ThkWHBd%m>z=MtX)IqO&q%qUT-Q<|7KJNMhNs;a6pC&Kl7XSB)v
z3KtLW;^N}!aM9A*nl<f&LypeAH;UH%@~=<W?^;Mic5d!cle_a$qiaVX7S(%;iC{c>
zPEJmL@k|#dzb#-e#WW$n3TwM=5&|%zK#w5Rj~l++vL_;_xx2ZfV1g5o&-OTVud|dg
zY)p8a+CN(%s#Jb5B@9dFyIC8&!|9!kw<)+dnAUfkUON7RZe*>j5V@t@LFIjKzBWN}
zl6bSB?AXD@u-6mJ^$s$%>{fg;H**<Wy_z$R_0fFyxQl_NC8(r`t&yYbC$Wq!ZhWiR
zaQO>2)_CPI7Ur}ViP+8SHXAZ(qCX-8Pgw>@)`d8#Ub}%C2oieVUWG5$Z;wDPw0oh!
z(MS2V(Mt&eBO3Siyc1oDi4_JnN!?`_!F7~4=LJ<;7i4@c?GuTsp&oXOubJN~j*nRE
zr*(v=l`h&IUu)Y-G0<(qS?nZJwYZ$k+*tNcTf2SvjF$-h6dEDNR%>@OyFp@~wx8sm
zQ@KkHCa?M<DDa*q5JkVi|7}X6ooQlUmnxZR*#{T+GPIBvv|B=xJdNM>M<?%wx7c)l
zdn}q*;Wg|Gzyh^yRZNGE1Xt2+qw~^RE=E{lE|-=pha<-cjHj$U3eSXtS-y9ciA;Qt
zg#X2CIXogbtk+gV#pepYbkK~qZROc+2-nD3e6SS4d#n~;vgmMgi02}r<bB>}Fg(g-
zTDmx2YrO0kRzm_}D5S%J%DnQ0DdV>iV*NTt#-ch%|0xYIm~S%Q_BwvyfdQLqe%iUU
za<4uRAr<XM?#|y&3?~<aXTr-Hgaz#tT;9D;cK$2uv9~aYB8Zi$G+Wuxv-FN9+5PYR
zh4qI=trjcZ;=wL@A=W&9;v>?i^7Yn}O(rs$G1Q(|Z2z1sn<Yns)V;&#c$@TE2Q)Ru
z)6)`Bt!gQksJ*lfL=-F}Q=R&|&-+U>DkcK~7vFVi4u!dmmta<R;msWO$55HcDm$Mq
zYzuC++dJmp5`L=_vy^#UbXhfEraEBcG`ZP)uwPnpl~qe?xmmf+NiO2ax1Om+(c=}a
z!GBS0w-`sWU7FHRP;K|H)xhT98<o%yk#hLBt#8R=)!=YcP~pc5m_`)LD5eB~iYqs_
zj?tLy72jM<E{lrbr1en>*bVr_*W;KI;3E7^sdYFjoC{%b$NVtKVTT^7+2BYNw7pYj
zi4R5^w->W9pqU==f1#Eb;6C!>z+DXeb>TXZwZp;G`tJ%B5BYEu=0)FDN%x4fNM_s}
z&3d^+eysZRo^x@R*na2q(uHz@x%b$Uaf>KVBm(XH446M${l^6#mQ=@EaW_yr=YvCd
z+ccCShFQ?$1}?+o10_*^aev+*-pN^+qy-`EqoDYuWpSd~d>GvmJf>Q%JrCl>)WF6k
zmNOMMMp+hgneL-`A#pMMbJc=vS<;lS;}#5&f&14fUK<#DZv350Jd6LHkN$NrFqGJ8
zlLQ^XPt!kNF7<(R!K>C~vVL6sXh^GeVu@|f9?Xx@{W00`WXRZv(|^df><;rf5PNdx
zBpMO7vZw8Z5tyP#0>Vqx1X*<dv&0Kc;4ZffjQvQ21Vb2S)~HS*e}&q3vP0X5Ck>{p
zL|PgxQ2d3lia{mhxO*i0u!<pNN+=Dr*xkF}rL@3)H_B6SSLCd3U(-4g(^%Ggi&~71
zLC)Opd9psBf9m^tF-R2=i~Y+Wx~{DJ7&!b`+!vr6(LbBVOG$#4e;21&m~FS))*G-=
zPa}wqkl#vdk|f?wwVSv9(#)clt1G+|;%v%L28)#;+6VXk{7iUG+jNKVTH?KH*Mg`P
z4}U@{EuUAnkMX-Hs~^?DoM;li?~cv8zY`mcjZ7JLTz<$aU4RE4@Y}g{&JAw#$bdHS
zIsj~7y#89(M|b^nq}$-hTBu4^z2sUdUo4>v8BBVg%wyU%_MXf@bERkY-D?x<$;O<b
zpvBDRefW<DBArvMWHxmVUG9o}n*WowASDGwz0FLH(Dzku6^(`Vv9TH-Or_3WQek9&
zp{?wvKs0uy5PokudTJp-5ZP7duzj+zYp+MDA<51+GTg}}+XPcyOCY#AF=5lm{bEKD
zuYwz`>Y=}WkR(4(Y2H+%2SHKL8L4sdgQdlyUB8->Fe_=z_$V35@6;>_^Sx+A)!#{y
z%vNutx>hi*td+hI$B#YrfhqfD<l~3DlzhC=#`503zteBJi#)&=g+hzrVxg$DkD{I@
zg^onlA0{~L`RpH-+6gSVF_?>R2n@E#DgbDpEt*<Z@je@xlV4BeZ%k2#RC;@KHPXI=
zTv4h`bGcu3y~M9wU$HGWe-9h0vD0j^8a@f(O+_$Xo~<oWUu<$5exr+)S7gD4{3wVn
zgrav3%g?CuuSSpayqwxe{k^|Fjm)EGSivv!zjH;?(yL3=fBQ=%uWad!A{EMEQA^#0
z1YSLDC4#7>-9zS&zAcI<8lQ|mDf7d6im6Q*Pa;o%x1R@B!{cXusK5DSe-NiT0+q(3
zJAul<M~5YH9~^lh-eTUP{6+YH%@Tq2W2W5mXeE;-FoLq>=#Q(^6LkG!bA|cqcF*pi
zs1U^jfwNgMV;A~2jK#)8*61YsCi9QR)xxWa4oeYf>?DNCwZ`dlvWs4bPp=e=C;^0@
zKx^mvbur_;)>z-nUx7=5Qf6*XICBRQBO-{5^FAFbsff<KpkO!a8>m~+y7p>YE4&LA
zuaOFs0yDxhFwwq*qMEm7{5K*3xDA#58|>mq?5K=|*ax&{U5}%6AKyg@FV2-FFZMsS
zrHF6K$F#kK%8>ZN{<(`uWDrDi_jGHzJr(KJ|1yPYc)l)sxy8H2+vzKLzZc?{M!E7v
zF8j7?Yx*gy+=(y$qRI15nG2g2=Obg4BUlV=;xP}#z2<_JKj<^R-sjvH4CwW2sL>TH
zBFbS!1ZIVePcY_hP7Oo2*3+(G-We}hkFa4=pbOcJV<M%e|C+Y!zUqG{n|J=*>)qtn
zuvYEZ1ox-boipvgxq`t_VP5lxlooZlyR(J3H(<AZF?Y%7yd%9i@7`?6^1pYM<8}vA
z(fcoCgsMf!oK2VWC%GQAgevYXL7o?Eii4GU?~V&J?0;;|X}e4MzW2UeoiA;v_Kja^
zfO*44nT%$s<HJ_iHD0JS+i49q+(`zs=pvVW8i+;xuYtKK!0_&IS4RuYlSNh0LB0N0
z@7M}4b%hGL51O*{(vrSeZ6D8OHCSs;@XmhzeVhf(Slk`m&D&IY@;78t2t{dNPCng^
zl3t`DCq?X(3hcH>ORIJ6{H~eR;2X%NKo&enSZ7_?BtrJ5+S7I!1*N++VP8<HIBoPS
zs4s)hk6=@{5NU`^H+?`wcP3Bej!F{iUVeW-Vmz5k<8mL=To^D!#+Hw%%ierPJ88H_
zPv82kH0g1Ai1s0S9toBo^Lh+BR#Ex!G~`x30&(~+!!jfFHn1X}Pg@a3JmuSLTQ#Qb
zpQ{H2%o6CZ4m_;3gUII{#r0Kl7wZMxrF?pq^i$O~lx>HQg7%WjlKr9^Cy2f+@^BqY
zbx<#R&U+j$A5c+HBI@EZBVkOHH&Z}NE>JzIVq7w@Q)x#`<VnX!98`B2r8m%6Vo+<#
z6|brd_*__wX0{t1GqA?_MGCa@sMxH?;yyPPF1TB!b^Hhn>~1g2KzO8);$a8uAH*&3
zg)D9M>q1*@_xCVo#(yS{1NMP%E6V?!Zdx45c`#Zo`Hos^`dDktCFvy1sVN@{zyDdY
z>><Z7B&4vz?kkf60-^cx)}7#;WUQ;|Ust^A{Cc6ps{`J_Ho33e7$a~vXbL1#t`aT!
z2UJ5oWg&|^mMJOh+0J)(WRFNE*ofb`(PH0t*QCEMv<$k$g`L*rpQiFCT-S{2>uUSm
zw<=5E{Hpn^91&*!WTH|0=HF?*gcfcA(20pesM@XX<znYR-nMzrF*R^)xxTPuJ-mhU
z$llE1dA!{69o)n5LZtfsGN@04B+k+hUE_rgg5;a&za<srs{J-<3Qy6+aIcv(Vh0~)
zkKgAXo_BfQU6|#09=7DXY~^inHrqTNHErF7_5Idg-n+BOIH|92e|zJ_nvaQy)}E~B
z(h@~J+X?{0YlgxkKbSHfz6PX3+q1^PZ8&z^?eZ~;#lzEVFypvJwzmp*NjFhor7Ydu
z;qg_Ml8d`7>U`!J{-r<=sz9Ex<rB%wndw8n3T-!>HqoW|-MF)wvnft=dI;3KrdhMf
zMAysew~LW;pSGKW3p2yd|KC%+rqf%cE><%ubiE&Jx1%0MZ=Vj7xf*$_KBZqWak)pv
z4j1_4XAlThcpZNinfEz(+YMJ<jd{>q^T3Cm4}mOujeA_~ly2A-OCo@BUWkDw$?;>`
zNAbhK1bj9N)%iBl%U+H3#(W(f7`9(~6|64(y#)u!7m=9$ls^wV6HaD*K=}$*?`*(r
z%PAmMv7h_OPY`2waY<I!<1y{9lkvg*S2VsV;C^W>H9yql*ma~+oMt}#AG;;>^x|`8
z#R49-yL6HlpGSwi9Z64GB2a6w($eJZ>gJ#mxlbpK{v}O&29pecI1p8Qp<h1SHvO4N
z_+F)%xO0841x)wd$&8Vw&1HJ3w?MV^H)!Ha$X1O>IO*|zpZSM6>*QdWiv2AWGUOWT
zi-U3Fb<QX6QPsrlDFCm9oEyVYRmw8*Eij=nt7ZU<rce$)Tpe9zp*)({+wiPnEysrt
z%KsJKDNEN+O_ku)jJ2#;k}_W1r057%=`a*slt2m_4Ow4Dxwv;-?t_o4?>G2;iM1t1
zHTq@j=TP}0C)-I{SAz4qYL-K^qweE9?(nlNI~&!^GA8sit}QIH$3qI@ShGe{waqKL
z$gP6l!{HMFCb=eGSRp?HQ$W^k5NfTr!FhVBHe2dKhrT@y7@j_LUMO2X!U<qQcl9^w
zAMYGS1h^`W*Fliy$>29w+j$DuOrED%X?afH7nQAm*uGMru4R&AA=<D2a2dhUa`0r@
zkTJbBK6OSuwLJ5|P9q(YMcV`M$U<Wo(*mXYZ8CH?GyY$Y{x{kJ@CmLnT3u;ei(W?I
zMzAO`+bv3<e%#6GvtLFB?dSqKV^E+x%tEbya&0sX)%=q7bl~KNG#gIFFumkEV<7hU
zq)iJq1xI5@jQ(S?U1mj9Smdun_!|Kx7{jS*z}2gF{sN8ft`X>R+<Q>}*9|j&!o)35
z{$bev7y}Xb>i>Ao|77I<=gZUAg;co|EjBB;+t~N>_f;{P9o0(>CsVsQIY)8(M+{Z0
zX0uI|msFKEtT~ttnsz7Q&HG#`N0?_r9`7GV%pEN$8bm$wuu-KbwUr2qDS>DJ*c!rf
zo^{1mo!8969ZL|;%-@E-i_s)2-OJ(P&$+jIm%hVOr64opW`*}ty|pQ4f6VyiOYbe#
zc=({mBPV?e10QS2MJS$rn8*$>vA(j}V^5F$v@VUb2^BbA=Q!jU&j8~69bkn$Rr&R-
zO%6w?WLzx_VTtLj2CV!KGIodUcx>y$1FafezYvGt?k~pa`aUwZ1UZA|ee$tkNv=Cc
z{JGZu?=XZZBj@Ofc4+btE(a9d-5DgAA*&B(**VS-x*~(_)GHAM5wR(${3Wbl#jKod
z%!O|u`G9Q+|2vP8d@VU)x3xNg57CaY!$@wqG0aZHEbB+A(>?I5v+>m<Uz;jVsLJ*J
zW~+@y6zO<$tl?|lNJMu4hC=%f7vE5WJWMT(IHLy3^({EaKP?YB7ke1l@pqa|SOJ6V
zP+$pl?BYl#$__WuF_p)}{q`1!uG%lf;Sa^0L}IxMpnos@Sn`)SbPo_}$YtIzMIKdC
zPs>vd6ZxC5y|K}HBV94S<<b$`HYa;Z!TZt^q6v86x58P{wkI2UoRXG>hZ~Pm)?ev(
zXVwAnw+Kn2g}4PvdewlLVqV+5V`7&VqCwG_bUp~XJRAD!ZJ?#2yhSQ*$+5}6kL+Ty
z7n`c?qP19mF#uIzF`@8+FFFm+?x5Zv7Kf||U6sF$ie@YBg?|s!O5Yd)@?QvRXPw9W
zzAFH=P#$oWOh*kT(7*Ed?XvkR4WNH}In6(}n`hUC2FjAX9nI&<5ll~Xlj0N(M~gM9
z7W;3xm?1qmFD1O@Yps*F!w=>vJEt4sE)}`;KqRl-pB?W8e==fRQHbNR&yXn9-CPzT
zX|2+4M8JL)^24fW-lP2u?B&<ST&l#FFZ)D-uFKJC<Kab1+ygzELfogXQQ&I#70tnM
zy54n{`>H0Fw|hk<)fQE<3!TG@ud#0no9o!?oNKC!@}4eEg2_~&7N3qT-Qsh9Iv>z{
zm)0<yr)u#7_;6@3bvjJz+ggqdL`H_^?hjYdPzWpB%A2F0&w3<JB_EHSqr-ppKXP7$
zrWt>;AauX_U`)J@pMV`1Q@7$z#}5b8W-NWwOd}B|VY?aWP5mhB2UD^`MzduiO()As
zi*rJP!fC{s1^J?Ot$QRN*3s2*LB2-Q@>P<~K2Ce@@ChWHB0qvX-#@OXT6O<z6!)t8
zuHHuvKO2MQPf~XiwS5gw^y}|f;*G`8qVqEEj|<{jf0-d9z;+?Hsd8<$bJw!TNAoog
z!lOHQ@wE`4mlgY=k-F55@ok?~VYgZ9ii?83DUY{@BQPW0Kt&6>3A?52TkfiF>Svqg
z?k5mH0g<X3N|2TzLCY-?>EQvDCi4-yJ&q_ZOB?^Ly!aEOJF0~0B(D@5kKF=wuhD|l
zU+C#Q8yn1*!P{yZ%n10gV=xIA)5(n8w~GulmX=Xr!2;1WoHvT$;rkC#$)4|!vFT+X
zw1r+D=B(0|$vmpR_ND*2G9vq;Sw=C};CZDWLZX{Kby1`C&o%_Y=2S+BH<R2XLYGSE
z_=^|+ssvIdqk*Vve*U?%e-cz*yJPIOZgw1c$H2N3J9pcuYz={bj%-@o*jDZLI~QwC
zBAzvlhUpVeBJ9G0MuPyI?ZP@V^Kz#H7sx;!p1xk4&&-S`|3LqHbLRJvK%QV=Qkb{S
zvFK<?FJ<lVt5;392rQqy5=su0>1Ji8iLbg#!*Kvdbu_1a%bzZ^ESUW(q;MjVZ0*f;
z!p?7bS9101myQ#T`rwFer2+tPS}~s>qQQ4@qljMnW0?dl0Z$d)m31QSYb~j2ibvAI
z#x<niUH!DZdzYp5SBs_*horD3f0JK!rz1mnfewB4Gb?v?I8aoYjvl{!QI?LHC9;+~
zoO4O`Ss;?Z^Aldl-U@c;ZGbzXt2gmFZFi(IS+89!*Lk<F@bP}#ZyFV(^jK1WN90`~
zt=Tthn9XCnm;+Qjz8G0Z>*7Ne;3SWus}*&Z1bGmhWqu<dAv#}%Gd(=ndllitwoRIo
z=Y;_o^88@Qse8?9I6Pwi)bibdeYC$Yfnv85*zhy4D|RdlSSDYL0)*rCoha8?w<DT&
z1A2C)NtZt0&W>)}@#IQP`74&+aOIqCHv=OPy;3=g?ZXcQU_r3<9`^sxN#N(Sl-g5+
zhm%cbigdiDjZN%d1~Qg0ODQFKeyoJR`kd#k|IXA-l{D|)&PM<Niw)F*#IwjCReL1}
zYT#Hf>PQn`j?{K~c<F;FKn7kjrJhC8ZhL_tqV0FFyv`9W-^|cIJ!3w5CB}a^^%s2A
z!kZ-9CkOI{{doQP-#T1pWlE_zQTCS9g~NcX{+SG4b?_OC`xuy3j+4X;3IK1etFU|_
zwU)#L2MPfH`45d!QU<#gu>th)=$7BR;X|@wqUv5<r@H;<-%;IVy_9xGyk!FH?40`i
zh&xk1X(J4^;^f7B!0d@?mW)-O{}o#QN$MY^K{-1A05(IRQ<dU{w)@%ijcKn;(QdbU
zS>2MQW2XvUa)ii)_u`-rnYHKB?wPOvE8o>{*St4#(@XuD6t}M<Kjr}^p&@SEy-a^s
zicMrbiSmDPIzRp-EIO&}#9Z*`{%WHma(;pwaXmQj(l?ztiZhz)?1y!o?2<o%oYP^B
z6=Bgzb0(P_rjGYJ4x0Ex{g{L_v_D*Sp*kJ(f4vqey-LLa(*Ttf96(dRO5oVI*kvHS
z)Yrni5#mxk52&L1ST~|zsG-JRTfTf6_;CA&S~zwWFHy*R*chhF3+&o6)p-`N@F9?l
zoSB%vtJ+66h^$SOipE>?*&Oc(2)=DpfDruZ?u)O3FB}0j^bIi0vt%bB26EIJ=b_`j
zOA1%H`Vt?X?rOD8woVkiMxan81My~V*uaT99~-NAswewgQT>020z-bjLUXyB$=(5F
z9G~s*y0#nZkD*LO`6f}2YVh8EaNq&r+Ak|aWUE>bp(T*mSAqFG%OgQ+P>a@EZ6~h^
z>o?rr#(V;ElS_BiX$tkX(wtspX6DCW{Fg{%=6j`jPqn|>OY@nhl4Zk*<q)K-4WtE2
zT=Xw+#p}(Eb7^W#JwAPHY`QOYdJj{syXN6!1lUnlx{1dr@tqdHzW4wjf0jn517Qdg
z=!?1^@thZ2vBvf)uiJyS<5q(c)qhhM#%1R}>Uml%abT1~`U|=XaVE#c%g7PeQvGo0
z)-=6kkfAb6jV(L}fX;C^-057|Q>o|d{FD#WUvE&jnVy&A0k!CR%(MCH2e^R#JaS_N
znggoLNCg_gLo2g1SejKFNA;^kwYJ$B9v)jOBm2@Ez1!lKMa;9uhM-!^XPe#-5Da_T
z&m`v{tLN!0^5385dIR!-r_<Mz`Kp^bCju;?82igI@{uap{k-ljoM?_CwNSE!ijVV0
zz3^eE&kIb4-5Snu?08)+a0#C*>;Y-^pa!G+!0pq(t;w3<q&Bj#lW>Mmesfu0*hRc!
z9+d~O{*%Db-?Y1~yh*E<m_qESjvJr`{%e1IS9+h$W$(57K`FKllA9RK+##=SwgngC
zEztt?2qCD<Mnjl+!{OwtG*ENSHW;<0Z9W(_c)JW{tnWY8)wyTyk2BfSs+*cg7X2EQ
z=d9!RQ!e9TV3MPe5(d$xXXIzg%MsRqLI|_E<WHe8nFI9>+r4Eij&q(H_&ItXP8kY0
zhglRO!te5?*<|T1*jqku^}AWsT}=UkmNZ?_-w6}<0=A6MGLhUG6ZHH2?K<P9jEQ}8
zDC&66za{7|^II6Mi%l4p#)<PZPsg3p@u<d2FBtkyiJhz@lgzafkLJ#UM3Wkm?;XTj
zmT!p+XUhU%EmKC<2#;9CaPjnKoFz;fD2UKSp1>Vu{yM{NH3}k_`}6Qyb0P^HP783l
zsV!U7SEgrT(W^G`bA^LWUP5e($^kB;9^Du?Kna&UdyvR;Wg6I|skPDw-YJr4w#ER(
znm0{dZBp;MSFJjxQVq*<YpI?-@nYFvH8Z-zqQ4`s2@8mGHbH9ELNAkL)kb@!ZvOzL
zbaE~m)6AOw^82--vSzn$W^!$Licpx+R^5ALo10FnjySnP4)afW9Ejf9@0+$`DLTa1
zH`e)E4e_21xXsPBr%mY(v^fmgTT&&@_u<6rf17?ToV8FwGlSDB1?s>OEw3~CeL_5B
zXD#8)VY7!#1e-|^Uc=Q>)#3wU7FH2@(uewe0EqooU%ubdPph|-Zb{6lZ$-_+c=Fu$
z6$Wz0#!NBZb&#$q9@yuSlAOHCLTk97!#j{y#Pj;7dm7`?qvAWeS>%51QL`21@+god
zr-pamJ8Ep7Y5(wTr=3<(`G1$X<Jg6Mdr)LDm=W=YXEqk0J%gk~N1(175Fo?^mDiec
zM!sZqrY0JrI6@L<(R4uPysw!?#(=sMP};O-=5t2qzi1Q!!exVOpqdSv9IMm3MDf9y
zuD5@Fv(3pxMvK>!09T}tA7WAoy;OJj-=}>aRYYTSwU-{%bR=a0zf6?!E=9~z(>@6&
zK8C=U>}FrAk$gC=0rbZ&^TqERW<P)XLdr+5L|<EF)G7sjxbCO{#0Q@#47%PgmKZ=3
z{#$yRkDr5LjjD42;*<2=GR{(ean|!S`bG5m{QY(6P(ZIo*^`Hp{Z$X3(?(%!(@IPh
zS(4dY1Z)srZKK^fSh_<tsh*Vy<*<C9z#oj^8y;D|LHPEHTZ7SL?TeV#u?h0|JxM+F
z&v$a-Cm}G-mJe0>S58vXkI_a<dWy~KlD|J@-8tEqdfE`m;leCV4yK19wqxDYYM##e
zpA+IgG$D#dULW#0qAVl*HY$G&UF|!Im(?`qu;GVQWCXSgr075=y$@$)BfPSPQP`6|
zmkxU2MMw5&0g6{+u6G1DibM`6IA9M_94K+Hl3+~%XeZ%rpotU5r(X{R0jrbdfAqT9
zpnHFA+gctLP7gX6g{}tQ-p1!$-ZxzoaYY-zy(yIbD43H}_9niN20+k%wYhBhaUy~a
zogDY>^k#-ahKEh#r<;;>HWpeIB#Hg91m44lI^)OV>+6|oWTMHiMVf~DxxLd%!zj=S
zpkh(X{r2$k&`JB)d4s8sKY94&!OMIc0Wwk9$@`W)7}h?)^1ln8<=Nkg*hiLE-ASh1
z#JxLb_z8neh`oAU$Z!1k%|-|OK|tqCp=r0pn?ez<JMjL%A9Ps)>6LkUH%{PifZBg>
zCU|i5;<{*hMp}gt1~raLsrM+_UiP#R8v9x7jZ=rGn%OV;+t0WesG8>QBy9_Y_t~Zl
zI^~ok8{%#h2=bz$9n<5{@-Uw~H_y|6Le!wL4de)@>mmY=4n`dYz0}{_)>{s=YFwOM
zEl#?tC;@{231|1ZA8T%#4%kgr<NrIAVV3z{mQQR4f;-v`U0kxg^_FJ7eQ^{t-SV8y
zXm3N_iycbP9)oj|ix20U7BkXB+=cW-_0%QaOj|zyF*)}*)4tJ#-W!cbMM=t4zCCQp
zgJtNsV6X8Jtht!{zR@NyDX4FCb%BJYp|$utVtWfdqZeU2EbI1(r&uwtRXrqtYFOzB
zmXBU42b3lAKw;WVR{Az13Sfx8QU76x+49QGBSXAKJ;z{sLe}vrhwve$A28WZ?T0jZ
zl40o%_w8W`%%vOGY}C#$guce=W?{is)$3f64ji=tEC9M5H4%Kq&iJkIu;J2SIqq_p
zET#ZpEg<6wtS&1kL~O^oePwi--?(*H8i>@5m)l!C{Y+^qt%3m}fK@pR4zx&~-%Tn4
z3K&qP%<@>;9FXw4t=Pm@W5{x9VwAO)YuI~E2Q3T!s;_x!IYW#neb^pe)?OB5PfB<}
z=Cd?cb3=Ixf<5(D=sJI!eC=YhU4t3sT0_3%9obKJoO19m`$zf=$Z#dp7+{b|Z2r6y
z59t+iJEcZ|3Uo|#`Ca5`Dm8`zLebx_L_eU{g%($mR%g(AUWX>81#~y-+@2DhAxvM6
zi6W)IZMj0jhJJwfS%`LLop4suSKKl+eB<7+(BjN;fZ|r}%^t`{Md5%XK38MZ+U45B
zJ9wx4WlMja@ZtmP6zl7Lujh~V_yii*S}FWl?=Y<?z$7t%iq8ao)mYO-TKv7)>JHXS
zw*82~hH+$dcE1D^C^9^n@|R9syt#gfU6_0MUUmp_YUkc}l70g#@{)V3^x;iyMfwnm
zYA3a_=MD~U15x7*SMra9r4;bx$!CXWCye}&Pl~tvtI~34_NX0=!BN6ix`(Ce5`EOg
z)b=P~s3cD-LZ6J7-vx!vMCHfi3s-ysvEc5wa^C*XQbliv&wm#^vna_)&kp-03=U|&
z2G75Ybua2_)XsEv-FxuMG7JUa1bo}yJFg74OUMK?s9fz0IW^%ct)`lUc>!S=v%>u+
zoEABdTY;uWk*%G0?uFLRoSTD0pxP2mCI1M=hH42X)y&61S)UJ(Cw}21G$v}NTbx#9
zAWe(#@gYCzR|HL~0gxM}f@k1Sxoc7WnuEFXD+aoYlZQ~Ft472{lV*0DXxFhkiB784
zj&g*o=d0(m1tOX>K3a&hh`bvh_95};V}22-4}OCI(8X7<5f}o0$w=mJ;a#j!Hv$Py
z_7_0g_Fu-G3Az{&ygJ&?9K>{cau|F1&QBwt{)yx}1EAXeTLI{RkxWjvn?h|I+p#2&
z<Xj1j$#p-G`2F7-7Oy@#HQ>u>pqhEp8MN@z@L(k~ZHWAb?ZPr(Py<vG%Rj#bSgrG3
z(usssK#OWmk;PvA?+tE$JN8_JmBy4s^Uiz%na6R@KrwAWV?aUsk75bJS4Gg~egJ*#
zSDa_`l~#l`k$j0c`QPsY@KkZ7vT<6i-TTsE48EmAeu}duN`zwnBLgy|<gd-bfC5FE
z{3M3~nsM_cog)H<z*Sx#&h+0(z$loBLw_SttNw(-vrb`+(~6Y+5`3PnF8*UFBq4_q
z<$DKVi-OB)H;g%mf&Rb7VFr(8X7%CSX2-g6$-JU;*(}_lTe2NK`NW8{pdurrEc>yl
zpAn6^FL**iwePkySaJh__2YenuX%leBxJK4tI^K2n?C0{PWFefIDAg{YtaHx8Cbgk
zdsu>nL=N`q@rz#zQ44E!C)~(q=KmZGlNtq1e?fp_VZgyJumhMN{?$z5>6l%n3#s1?
zcONTLrjo|MCINwc=QwdFg5)TOpogInYC>Nj1AH)$2X+Dz;1hO^?L^wXs&oCS_v$gt
z0G|`eTFC1&5aNX1G9muYlVY?Wo0|w)c7;t1HL+EJf!^Odlu=z|U;lZ)B1Q^<h3o~`
zToQEu8aX%m_jYv%btD5X+cG`h!a`eYYggyE2sbAeeSH5r9ISQ&{5QY@@|HBO7CDXV
zU4XN1TjLC6`7-@|FubU*x=o+JI2r<Vfs&ns_yqAMH?O%)%+Mvnvvy#W!G?&M0bl<8
zK|y}2sJZ5>wU-aRY~+ml^&eVVBHKFg1-zihJBIpv_JqBxY#0+G3kGV$kFDS}dQ@vK
zF-`tD4=;WvHDcFrSO}_D%dtATihK+NQo&u{)PMz&O*cNVRWvYg`Rpa;<06eN`3T#A
z$qc=d-i}#wgd`NgLSaVss6q(h%;bv<T7~#hipwiK2?OLNB<XB=lzx#_5GE#to4{h_
zPJ=w#$|Pq61S!!1nompJ2c=`dGOzI^|Ip3UgNHM(Q}VC>kgMr|IpWoowe<AD>D3H2
zcw(W@!Ka^kFyj0jT0Q%I=pd2O-pi=F6Lhj1JXJTR@DN2&`MHAHKQU39LKv^I;DF6$
znN+CVjZD>G$ezHFAOWeWbzNZvm)l*!VQDtFXz;5$bArquU!@tBiV*&}zkrn1^2H{H
z3$+Y|SPU3KGCPnFv;!=`e`8`IR4#pm9h&mi_VSW#ZS4ab2P%Ldu~y2q1_~rOL9f4a
zLZP}!Jx1I1s3F&zo8p=5Om9XD_qZllQWrFD$lp*VwfA`9>q1^C6UexQ%5e6Y?<%Bu
z;fqL%n`XzKoETaYyGDqIFg1%353Q}kGi56)L8$D`W`xBi4<69M!aMFB4L3wkS(uJO
z$LJsHtP6yIDZ=4TqWo0)1R)10M7dm}0FD|gWz2p`8bBE9>ai;MvEJ)#pt2M;9Oy!f
z0k@NmPFu~hus$|gFuJqrT|T=VWdPyi{?EqLT*h46fEWQ^IEF&#+q^G)E)Zw7oQ+>{
zUi_4~?LCh8S`aor&A<}ksb9}_fx}k?CAj|GQ&h$Z3GYUkG~N&brlmBYB9dz=K$#`Z
z1P@Y;q-JGfrc0xym-g^&pFz6z{~D?zl?j^U{<Iz(#g`)=x*;ySIsX+np-F)AGT>Ej
z0Y1O9(ZS!y&H;jmE`tqD2Bt*3uAUFhSLS#h*P96by>%_y<g==G#9Yj;(9=^z9(dwV
z9Z)7{GZjplq#dVIJTnc>%+G!Sl}XXznHOY;0Uq9Nw1#%VP5{Kl2yD(K@pgoso>~3U
zJ#h~aDkGDT{|Ek6fVz7^A1|K}TRIJ8SX+-iu-1^@>ysXKM*N#G^+rChC|m|6+rIvO
zJQd>SIUXHGN5=lot!Ka2C~rB}(eUpKAq-56J}_LVx41c`n=`d8%J&jRt2=QqQIfN$
z2JEMJz?MM0VAc{|sOlBzC;yCF_!QMYbajrEdo3MxoZxT7swhMX#0bN$e7QAL@dT*n
zcT`cZ^eg;HZ$_pqHShPaHsAO{N<u|QLd7MXZTa6X*W>APbHN*LgQMg1-PEdGxRGg0
z!8%}2MSq8;hg)R3$UG5<mR1S!1bU16g~Ptcm9r@)OB>;XDg!>0sMTPD5&DiPdIllP
zK&^aDWa-~~)8DeCukYcBo<zT0<#;8eSMWTlA~5RWS>NoJq_vp#oCyM;brmBD_Kc!t
z<>jugH&wZL5x}AO^%;!PV70LNos_y^p~=|HW9MwpKkJ}pTtACeZvK|Q^!|g&^IG#5
zO7p|#b9L?eQ3qK6cR<|8Pk#oCwIO%dof@PRRIs4P4&cV6AKNVvTN0a$msX75wSxEV
zn@rUy%@@2fJ=cQ?J>(M|FGl;Jnu@<!7ra^d(e`T6T7G~S99PzrNo&$unjbar=-5Oc
zfsy67s5NO#Kc-`K%H5P8r^E_knSOa}or+QZ-GAawZ`MYMLl-1$L6e{JWAjTzMjcL1
z{aW1cb_#gYo1;>YDDAGW2ww&;?j{a9pwpVx-5}e_%HXMtr8VDVV=lO!%uUi~gbrhc
zg(8`23e=y-$QS}exmcZq#$OM2XR5x+$?veJ?r*NNZz4GWcUq&sj|&MbU_T|oj~y^)
z0PiNvVFwaJp>>q-Z=$L)BKu6X>||wrg)!6af3HuIR$~oO(KtU%Wn^U1h^Q|e%|Y3F
z%Dh}Vvo0omUdgq4_H9vtPUC~h7}uXH4&u6C@uOQbQ^tEQk&H}<FZ+A3@x#4r3@_HA
zqG(OwS2<2-bH0=|`#4WHzTKabBs_NJDL#PB*(*aB%}Ep;(ccD60zE+?Ne2AN8FB#k
zDx5BZ6h7fT$0r>sQ{hjYN1Z`FcQ=ll(je}j{KJC%7gAlEXETwuzuro^QRcM{w8f^N
ziL%8e$ET;op7o8SFbhw45s^};4{W+#t2-Dl6y~N=vrd=J1v31Q=sLdHd^WuYAZTvJ
zbtRt)b<w>7(UvC9@SAQ|wj1MdV+^D;-pls)0WsZM*BUEDYAekM(n<5l`Jnm2y-nGm
ztbx?V02efQuA;0(icKgT8U1!iER)%Thd^=&AXIL6U3vww_VTLNMV>y$II*UpFZ0TK
zX8WH`5gk3b=G$Rwp(zU}TJ;-nt%JOPHcm{+W|nPmdW*-Rt$qYsgZ7IC!)NE$HsT7<
zeBg;xz$T`=nEC@HR{5v|u_N_z<!7k_0zj2KSEi!_LVN@Dh4yimuEcoO>&Lb0OCU|?
zLz0f<qLs(15}0Yo=o28*TnY;P3|prezp0V2g}fdx_xPDF1<?c(JTfiuV}JhnT+vGz
zf=>{VdbCDiV6iO@w@%eGM&C7ojUSr=?8sK-8KYg)pQ*`rOyP9Tz$FDJ$C0j^3cWfY
z@6Xg?51`Gz6mTWVE^_yH2hayc=ZW89TpnJoGH!Gn-`~C#DlAVxL-4yG*5YSBo8jOV
zsH0rCm_A3OqPRvWeS8BQ5mTvtNgVn$D+6XBGob?EO>csR$}+PNeUDps@D&vYye86?
z+ehi;rQ5@jxScJRJCRwY5nlAgpp~wt-zggEqK98k9RpL&%a+CkL57L;$U+9(vQ{Mp
zvB+l}K>;*!*PQr~5dRpMud*Vzsj7U~q_Wb^?qk+EVms`mFOBg|sKX33=+CIA$i|vQ
zQa#5jY1f;Spc#J`Hw!zhAGtn^{6wG}e4ph@7dL2*O6--g-Q{U_GLRyfpbUi@Mx$OK
z69&2MA}f9wX)si15G&J`fZMgMcOeL~;a*91cp^7Do13j=<??Jgs?_cT^SdNeMI$D;
z`x4_7{XQT#Ga{r@Uf0yv6`syQToO=eY`_xWfQ`6|CX%h(>nnNs{D}O+7LVkp$i(RW
zsrSD#Lm}fH$pfF*h+fJaq_7n<z23KxD~=lH3=lU|uucoeBGW`sg7o3;{BD!P7l~{Q
zY-M|^6XcxN*&~my05NaGB77BqmnPwE8lz;w$nYC?a#D&-{y=xEs?yW1qfBIqyTU#^
zFVF2Z(8Eh@>7tvBg<NNxv&s`t;Ew3s4^CvI6h$OWc)o@NolQuKLXHNk08jI4&tlB0
zo(4i#2jy|h_6n<XOJR@FORTdI_Zgw5z<l5wJJ9gGz54E0QY)hHOAA^i5Il%>8Xnw9
zH&&u6#Nb8kQ=J+aN&D<y0?-*Sf1X`B%0|})Zvfc?&<Oz2T_LNcT%6V9m9noU_~hA6
zY~Vk}5P9@}@`7qSr|@!HN8lPgE}9hYQLEC467X0ta4&Gq2uK^MM3J91R;j*fUetM)
z3j5IJP<i_khZbiAb>~lb*qH^$ucqO@0G{$D54LvVa(8(^_QCB}YAQv1e}IgPOkag@
z_YdzR>1aCLdIv^kW;q6YTr@9AdU|@|I+v5R)>cv5-6v@onYk*{;nj{nLyWaC(_op5
z*?KkC5L6Vb<<PIWE@m`zyu4SPAw>OgELGLjgz@Y$RGql0q(>h=Q?_)+u8imie~DZ3
z1t@?WQ{md$8gL@Z^X{O=&CTt2D+8B^sOEET)KIFRU+Z6yuCcMxllAWM3gq;*&*tWg
z70k3he*6HA=~#~E4t9l+1*=8|L!n6=#$CQhXpN1He{zPSB2z!k-d%O|d>ziD;o2T*
zag!=pvnceJI2n<YGcnn!v)^{RIxNY`qG2_R5r)z-ndOm3*EN*}_-E#C0PVO~KnwAH
zxR^fM9uDR>M(Z`I$jI55saT&VP)Qu<6C@Ld3?}m)Ej2YHCMJGb>&r_Dg7&Z1x!lP@
zWmJa0^+Ht_-R@mCCirFPosV9Vkmrhn4TU3&f}whECtjH<x)S0gPD?1<0Ifx3aUT%q
zEgPH7Shlpt{V`%sZgFu`2k3R7T8Y-h{&bo5)8k6j-@jk&L=5F6ko=kuKC;l3=0@QY
z$O*XJp4@nx>~x_*QO{iQWg)$t)cyXjam-!$_2%6d7eujje)F=PCoFeBFrj&)3cB11
zyOnqY4o#FUy3*J2#Cnb7F!*)h!Ec#ZUC3zfWZ!v%0s-EX&6~nlwinHR9~oNMH=zSz
zcV2GxWFMQEA&$dUr$vC>ZpQ1!vFUql4J015JUt#RHV9EhB&ZhT13YegeB97Vy;R2s
z#H!!y@i3)pg=nmzO+PXguwS#qqBWMM-@;=xA5JRhb(8da-uXy&xThXi;cS&@(cWHA
zt0JKqT1RDN<x9=DxVYP!n<kGNT)oZn-SOika&mIfoIftq$jHdNHcQ!~X2Qb4$4u-4
zy}i5kV6nWH%R|2{r_V1gioD4Za!CUOB?&qRU9wM<>!|yfJ98dkP=!wj>TN{4Y0Z@m
zrx{$FjehN|8a4W6+nYqkGBV2v8oFK?3g}+9UD#bj_O2^$6Z|!i0ze9Sk3&n3<@g`o
zf8Yo>8EVITD-~Ks#!$$l@(xVF_YLS5&sAG#o0;td;n3Wk?;`4Be;C<@`KB>bQ?CK8
z0=WKsgY!}5$H0}RGa}}hUsl~J@)X}DXZxp32NI`?oR6lg=FrLbEmp?}$f^sGr6h)v
zMOuLNPk4AZE$y1S>I-D#cJ23ETr|JoACrYW1sW4$V(_sE?lT-69c#P>+Wb(Q0Vi}l
z+hWnK)@kwjz|Wtk@uJdVvhea?zC#3S2ldCna?A46R2((a3=b_WZJS%Y^ARRze{b)b
zxK0LDz-i>8e&9~h{w8uWsjdh<AK9s;r#cFg2m(?V86}?g-HoMDxnH0St6)ocegeL#
zmI;rC?c-9_zT&S?tCU37zxlO)=7bp|5uRPeVtP4-QAMlz>+b3*c=rb1KVKBFTwzDo
zU?Y-7%HShb;k6C~9T`y|jq-z=s9<bzGMW`?NT)AduKv@b;1$)_d!8qc>GYiF<Yb$S
zuKxHE?HVA;f4<m3jkB1qKG^7uPEAeCL`aS}K0XG3VK{|9nlECstIK3(Br}AFt-8AU
zN4g;XM{)6R3PB#1V}ox5b!2!F7wZusLy275K)4$4iC09yeBb*a7>a4!q<wJb<UAhj
zkRc72zu5jcV8w}b{n`v0tSRVjv6uWqqO)scXM-zW6jJAUgH!R!^@f~+M%Jv5vU$u;
zS_$lAO$ma$^;sP)ZuGFzH(CS`H!&#)RSxlpI4MCPRlujHkU!BEn?QqJP>>8bf;raN
z-j1N>OI>(+YNtWwVsyArcdYW7FtD}Uu;a&IV#{%2+|V|x^4quHwkO%6xPnf9YmGXg
zfBrzD;Z~pCsh578EF>sC0QOtSC*Cg&E9UHcSLG*mM7JNjZi5rjIQ9}XE<XN?GECXd
zt}a{xaY%Orm7=^nEfcmfq)e|#Mpm}L^Nz<u1sL}`N6Z1fq$^AV1~rOjbN?99%ci&v
zHf3bRpawRb&&)51gyg7%qE{e@DX_EJ@xtiy7Zi}h^0dqP#sQV0qVy*j!_S{pX_+jE
z6nAPE6m^P$&H<B3Y`)ji!xiw_UV}IKCy=hL=k8d}#~<jn_Vz{Ra;PsEnXXA8FCOG;
ztQR2=Nb3t692{ULY@P#O<mvuwYcM%D{xu>Shxy()%CNo{D^qLVtluCb@p}tMy09k#
zB4S%mZf>qho&v=OH*N=rp!1<WGMYe%Y&@&jI3gxuXZzFL#Xhh3*js)5<p>)Ua`wRj
zmAsYtiK!_%CjWfy_wNn4Bp|Q-^B=F*v98S*8eQ?4#YQMychIs@z^}m^E(F7}_cwB)
zMsJ)T>2R<<HijNHhR~p#58h{wYY*EJsra#BMe33~<2_v*tc<ho*eAGNXOn{4rV293
zUxH{$fI~({&3Bm97l@Bk-#QYxOvS~;3ui@Tq0k<{&E|fYBc>+i<>h7Px6+o61`;7D
z2I;Rn?atR6LEi(XT=L<7EOCvvbAN;Onja4j@A?g;pwn02ElB6Xh1qH=vPOx|pWlW_
zBIfxf23^4k-`yRxsH&>2-cd72XfRI!HqVsu?+O9VD^^xjMGMvZ#8wR<WLe3BU$O$i
z=I-jKx5JotGZY~h^*);@QDjZ*?cr?}9IVp$3aB|g4fkP&gwtTLMZ)UuMN{LV;`Ehb
z;u5p0KEIkzw~*NUdw>OV`(nnb`R_JI>w3Q*002();%M1>rrfZcX~+E&m?LpuzQ*Qb
z^68DNXvYihI&7_59S9*+y8Pza8t~SpI(A+}1Xi&<fQAN2-zGD}{D7BI;XxW)1GH+y
z<m3wgLWYs?cK!K-O+Z~&S9f)^i~<(3k%GJgM@o$b{cLzN7#z{O+?%vKUTIrhT^-?s
zN*Tnkp}sA2314x2ZF!UIzp<8U-$yl~M6aUbevX?G6C1rxM(%Y1?&Nm|+$<N>M3dir
z!KF|&HXkToogRWsggy{#342>g;Qy24KB+=}qF_-v?2dhzt^D$EdqD$2TCR(WQwzkT
z0DzR<RhK<m=k+mGSg{P`dsv96CJEJ4wb>{PNY5XOoY%WBSmV)BM}Rby8Si$nN6<)Z
zYlP(AW`Iu4_YJ@%YinzXfKSTGV?Kx|qAs_$w-PyS;N6{_V;3PZGP0fb^onWdMLCW3
zp{Ei)5=dUBGHk9CNy96d`NBTf1Rl0hy4{1k3@sTWf((D#AGV`lQRG-;RMKET2+?uN
z@Vd!F{$(_JLSR}tUw5<MrgoBYC*TJw<QyX;2<c4XHXplM_Fisu1?o<0AU1GlB<~-t
zmZMBrv0p`m(Wl7+X&dhPlh0~K6$$6(p1-qICB?-Cj4tyHP6vBSO?P{fMVWy#&R|~!
zC%_k*J@4`q)7!C98N?f#n!3X&(9-;-p!ptP2>UCM(*Ah1r{fN+5tvU=5C`#>5WwL+
zt$Ugo#UTLob1Nc?Zh8j0nsE2fV#5!ZFNQ2Xj?e4lUf<IFEiRVc6Gu3Z>X&R5ChQ*-
z6(`@}5dm&T$5`0dDCFpPvFvq+K}thJM0CCV1J=^=1SE!~<fK_|51!b}fP?DAJAwP#
z`GGn#y%sJyQ0&hE7gGbc)9*D+-cMenC2L^S<1itQ>*M{h765d)i1*h+g7C=51i$p*
zn-y|%bBoFzX9ZrOB?-BxAE!y@OiDnGcm31HQuyrw*A1KoGg|$VC2vFaZt({Du{xRR
zzdg+Y6@-njyW?hL*NyuyQ?pCH?c?|f`*oK}NfDKeh^_Pi(sk&It_~dyB|8raAUBYh
zhoB-X)9++|48)9rA$$<6`eJNcWxZH$3wRGXANDW!GIxN4f6u@!YlU_|OfFd?-s`bw
z*KG}_c|R=u-d}ms@e+*mLyC5EcSJ$V^lWeP-5RrmQV_sjXLg1~o;<D&DNdLqu06aU
zkck}G_^sh|5n#>0hJ}-GZLPHVO-)T5Fa7bepkty2u-EhU9G=F)+`O$NrxprMV)&U)
zt#SXZRMR#k*Z%CS?`?<-rWi2k>n|apq37r4pE(o%uu(>IhlGb?zI^%Jikl<%htK=#
z>uc->4BFn_-sa|J1GL^$n+!tMcrq#~tg=lYOb-t!NlBB^()2Vm@bPiv`FeZi#AK(B
z57)P+4~1(V%3*iN<U$^(s38DWiZVqJIkxq6w6_~rjg$t<K4E!Sl8}%9qbw~g9gDLg
zFX4XD61p?aIPoQrY5w?9gB{>9?CeUFuD*|Ke0;=9T#fTxEf23Hx4`&dj#}!#jWrxB
zSkl;FG|<yg7FPh#2EzM*I>(Ds-eIb!LIHhFPZ;G#Fd)y7@<OQ#uMvFIURwHVz0FyA
zAqtKh;m<iEWSe)sci{7rCd}eIJ8T-!s*=7)yDgI7i3i%(_^K>XDRGB?FM4T27~baO
z#-5zKunqkRV}POdZ|XY^>O8RqE88Gs<bN`AT_9x93Z)e3WfKm_|DIF&!9&1gZG*OG
zI*HGfj%aF)|1(3O*4y~hBRqUbKvQXp<j|qia}0@;#|J^cuA5qKTm@EdV%+fHKp^0l
z6mH)6P*uD!y}V|%NmFE&b_p{Qa54}_Pulzwgc(q&Ydsn<bpYwle1QR)9~BaiS$98k
zpX)@2^c_YWiBFKKa@*JAahu}WKUM601Q*AYh5kqlJYNEkuScSoQx_P4qEr*;!wAU*
zL`nlQkSE3`c*V#R(c?m)QUtu4&Z<$ZygXNd83%O1e#-(<LDoOo-dirt>*xA3guy>w
z847wcz^RTLp=?767~m!NVsHFtB9;s|&_vE-F<@-f#AgEkMFRskCcrwve)B@zIf3ad
zQ+3cgFJE@#@dWToFsQsQ9i>uX)>LUv9uau{o5Jb?FG>Lmhi-*bqB1SZ!>wo?rFbVm
zy`5z3)X_6@5PDwu9a05<y3Uz@8@B?!IE2!?F^k8XD7A{=A{QPC^<TKyCMU0Zx=u_3
zUdo<#?)e}T-d588a0-U}4iRMt&CAo^s|N#`^5);j)c$rqNJ3YHcu*f*g_)mNlmc-w
zuoelE@}owNdA<dRx4X8VRt$3e6l;u472T&aIh?qpDvOR>;B9juw?jKK3!X2Z<$=iw
z05YbN^%Bs7h>D79V|)5+?9;|A4onVtG{rT^c0?V#_sQyHm?_LCKF?Dv9`7v(DVh?;
zwu+GTaIEC6bKb&n20jQ@=-BL%z{D+3T7KP<*Kf=oh8KXX`n9<=tj*HR`Nq=2DLGI0
zv_C9%I{hsO((3L$ZL5-taqGOJlIpxsP6`~kaTFCdoTSV>MMXEPI6(<A{|jtVP@a2_
z6;%d9LMH)c9Tmr;G`B;3jHX@tHcW<*$@^Z$`M4YhFK?LL?i;<-)C+vo5=1ftElxi`
z^9TB4oRv=vj4a6Ge8674xuDM!Eq~M<8_s)}PeJ9|Phd=sSMX+KdQBbXx;8dx?R}Y2
zXAo&D%BT4Rcv}5i7OwfJ$>@JY3pa~;(brxX_`pz+-r(8s5Sv<4cka=UeA2MI$2xC)
ztcTm5if12%&wj0yn1N#8PvQLD{W`(4!-~j}C6Dwq(y*xhD4L+{@)e;d;h(6~1r-11
zoJL^C@d^Yu<ajMG?yi4@`rg>U8XBWx_SOh0()gsYHszJ2(+Js)i|}!hw(d}t3bPW#
z@zo~=q1^fo$dXTHo$K|ExLlUxBj{p4#JO3`L)3{eCwwAB%BwDw?S-dfe!Ft(bfBkN
zQ;us6?+G0NNHrc1SfvZh|LI|@M#m+x0ICkR%!Ur+1j$#Ma+b+Wx>x?tH8|`eY1vxW
zS#(;bhxbH$Iey#n^#8}yR|ZrSHSJ1=bfZY;p^@%7bf<JTlF}t04bmmuB_g17NH<6c
zNOwth-^KgA-@W%YoW1v|nOV=w^Nhry_ofOW4?FmR2pQ_ZVH4eEjT<Z|z;tvu;4sxq
zUcGf;f#~M_26(516DjzLIDo$)lnlF<JhUf#$NRljI`*oyJ2#VmDNBX|>^gREQSoRe
zkIbP)iB+WETB|K%iy^#OzK~`m=v~LxaH9-ibWzxP^kUxF?-EW(p(OI<xrdTR)ySyB
zLN)vFoEuH1$%;`?&1#O>@QY6<pYp}`6ztnpKGGC;S){=J%{*PUl~iW>I`sY=4;D}J
zje74kT~V?6b9$u2(bhZV8GwA~Z!GoYkUQAzDf9eTepz@T{Aufy(2nlBk@$AzC)Uc0
zYalt@waD>V`p3GG`;zwXKr&W~4qCX#^3T|jl5>q-DN344S$`%5gjg65Kz_pJ2Vn0c
zL_rC&-txSn=2~sPEkX)rj2vFy|5Obiv!yGOr8CQK<MNruoZgc@x1_aGBS*XY%QWwK
zl>G7kaynkT%q`s`br;J^OeOnoCL$>%5~T9fJ_|-c$eYhU8QijGaLT<Yd(8^_F6=9j
zyR2ee2q-o$_h|t3<v*5n#RQ^YCY24fNPOoz%ze<tlc*-W*V{}#`u0h`UBRM}5*?~}
zm6Uu^(f(y{ZeHZS85kD`zNYY#gjB@8>S)%MwFDcip1djSLfY?OwLgQ^ws55fI5QY1
zKZWdb#s>U<vWzfRxNy=-D6B~jyB>TQ94K{s1L^?nj|>+j@BoB+eed#&Fn%f1&`6_A
z4<9z=*52i2qA%wU)mFm+j`lyuyvbL=dlS=FR4$C=6D}qUOGQOuwGVy?7MvM+#rh(e
z|6RCZ4Ue?Jo_P@(W{KywTLNNHV03<}KVSAa><MS_Qu;aXgdm49IkkHPgFN`;Xcc;M
zO_0&+?Rg6kR{h_fI0mH={v-BhfUemnJb?QHw=z|4NH4ZBqayeTW@=xp+gbi~M0e*n
zGd*Qf3klfMg`|!6(@>vYoq3JfmdFzJ3we9)qc|@Y8SiROT^nGtd<F-v+bCAasU!1c
zT7tyqNEUVj<GnZ3H7sCP@xy-58S2n%dZv||B=j*~$upa-4&Z_6wUQS7xVTK+n9|JF
z2lZoHNc1^ee$$8Dgp7=gX2G6;o`d4g)9P?~ph7@>hL@KY><OYgm87B3Dk+V+Z-((F
zoX^v0yL#fOMMTpt@xI`ym|zWM^JQc?3%1|2imsOB!J6y|E(-6k2&XuxJRl`1lIOc8
z*48)Fk0g6Ud}&19v=%2droNOBo(~=!D7r$7r8bf^CG-&dGOFP@`!exKr<(`aBBW?N
zti2;)l;KSg0d~L2k~+QWf2KBCS+35-y#!3vzz}ARVu)%dd?A6uTLsRTVO(Xiy*DM-
z_cyF&R2<1O4`8*WsBtzow&pZiug0G96yMg^@S{InRz#1nvJ$Qol(Q)cABnq+&8^{U
zpc5;${|mMpkY!r92+J?``z7eExqLp>IX)u5mNd3ac;Xp7{I8!^S5AsxN3I1pVM*6H
z_JcfW%NiUq6`Nz;Ue~V$bN1P%hF>D`%e6s<f~i9IYHV<>t7ksv`fo>9{r9w}H~7T^
zebkwoP*!@T<Yv1#_DF!Cw&F<0*Em<Y9Fd7;c49m!D=4m~u2-=X@0KW#L!Pd!G0}?i
z{<GK!_^A(n0d~wv18dzV2qz9oF^;tfgFB`muzCF=KYw#lDh~)uDN82n+{2WSB9;iU
zWW9+`IwX8`u=U#dXjOg0qMgjS*7*XZV7rholD<MbuDGw6t?qS}{5CSM`VNJ7;VgWX
z*)@u>mMQ9%Twd=4tBc5U!Xb?-aB4!bH=R4@u(5GO@^rCdnHnf5qomn+&JkXlNf+cY
zlT@KJ`la|)q;Z5JIBJ#@Zd5GFyxSJ{ZZnz4#UD8;1`=tU;S7O;lr>M4E^@s$RkPP`
zt#B+c{ub<E{6PxF0v&mR+F4;k$rtC0HYqWUHGf!q??u)b1#jZ=gG;!Lf-0=u?m}$7
z?F*cD7J29PS1UyF+YsgA@9r0u5pwz4>qe>JYR8=-ju-Qi8J|p_O_XdVW+1Ue26}=-
z?ci?C$otdW(pS-G!{Se~8dmLW6=NSPb~jen$Vd}4C3WD%zCaws&!z>_{dX;r_|<qs
zm8zNYUS^eKzJWxRQ^*byN^zX}QsN#6!BJnq|Mj{?qBB=Pq>{l!e7r}J@7Y{sfiHv+
zRpk==A?WX9TC8VW7?XkdZt!o<<XcXbY#lk`P;z87+?`BJ)kfI$QgNDvU1^YkC8*Zv
z@gE=fEZp%8W)-`PX;2^Drqi=H#hYC&`8cgTDSQl9x0|5SH!-bWb>c0sZEBygS)=7x
zk(CD&J6-9qEY=znZ<*`={bmvd{w^l&>**Or`u;^i(j}f89)1uxI?DaDh`P{WyvH{j
z!$T_c9RxD{otE^y>amA+vLt)@G%(N*a+zSUZC~X64Uc9#dlG}BU!Wx8)eGIWex_pU
z*;bk5a}Gf%1RHBr=T8hI!a?Mb3I`5_ur3qG#sHh&kEJW-ApsL*hLftM0V$*}Z^5&Q
ze{z5q!%Lm`UXirC&zjJUt<yu;XCg42D_S8}Mvp&%r&+U()czhhh@QnFkC35#hAS|I
zRqbz<MmVS>pg>lm`H6#nuBO`P2@93L8NjnfsMcIAQP~?rj6@ZSgOnwgQl<s%f8!%0
zsaZP-#Y6w3=Q4nj5xvg6o77Yjm@kf1C<ih~?dLVbtfpk%+iN%e=snF&qn+9(juL$T
z?_XHTJ?r{;o19h-jz30o6tymtm_!WWwxRXwRiqN-pQkDnlw!e`C47m3Brtd$=G5!c
z#RETv##!tzKqI9ZlGKC)4rHcUyO3pvCGxMUdBvMmdVVPB{5So%#dA;3lGUdoTbA-!
z!Mryyc0>_+nuV+2o~RV~wE8oi3@dl`>BVMP!HzkfgCqB$@npe%OXyQ-`7}GE5GG;}
zAKv^pPq}C+85d7;6Wk1drcn?Y6gor*-9R?g@vJclHX+u&|IIkV5FuU0NJn2jD5e?@
z*dGYQk*JTCGias)bjd1vGK*mM7`lxbmPmWZXzU5lfq#J@L7~qAQsy)e{_S8Jh4F+L
zPC?OWm4=cedmcHMW9$L=|MxVV2uKQz24;W8r6A%hWRR)Qfk+aMq)u)I*;suKP;ARx
zR5YmRAy6@`{8M?x=Dmd0dC57)v&oO}mPvEt4gR}DNll;*u$%<kOd{n}<C04|T?S(c
zUM`0`*Go;P{C}V084U{fB-Q_v6jH^!`2Qa){t62egf14BVYZ|tH6;MJCKHrw3num}
z6_mubn~9diwtc<1>-IAH)>1sR320UM<hZI{#VKv*LmrB~Lw>J9@3I^9&=QyF9}KJ+
zz#B_Cpj`Wvj#2J!LUVj2%P^NbHp4y}JPA2vsa4_I1<Cg<bMJ6CFR9Hi4Su}^q6&EC
zlo0f%mdc|<gB|*EL~<78hMqY8C5pH7xMMl@AmHDmPbt+xI(-y!THhyFX00hux11&e
z-|_o8Ke8WN-(~o(xoB+fHVrp?n~g=^*Gac7URe^sl`9l5(ecTpe?BEK|4<_G)2TSR
zm1xyI2)!NVS*i$@3vZOqoa|j@C;EP|vCQa!zTj?||791a<cGx}Uu(*HI}z(CZKHri
zvnAa$u7jd8$P=Y$JosKpzT57#hzP;~IEkyFsy7;ruh8avQ#lfHpPDjWSM3@6uF-z`
zA?)P%sh0J<cJ_>oz&;+k@dJLV528cDK6OL+0q=G=e`EHmiR=6t#TC*YG6Ie{vSn0m
zR6z_*3L6NF4-!JcNKj2a4(#bK-%t6Df^NSqKFz32!_1GOE%`J)ZLP0lqj)S=9S<0-
zj*xnvV<=o21UMZpF+Tlk6uh4*Xp}NxY&k_d5L~tyj^0}Ko%HB3N#l|q3veUXKGg)}
zP?6m{kfJ!f(_Z17A5G{^dSgA^LY;2~WoIZn2X?iSIwmm$)$5NhKIa0e_cD)D@QJn?
ziHVL-UtEL_DXtmdiA8x9EybS;+<xokN4YFZk6ykA*Vjb1@l^#?o<>O98B*)$3|PC*
z+tLPzo}`GlXEk~5#5X5=AQic5&iRXzue+9naeLGF#Acl#Di2xhL?EaR7=HK1X6lEI
zxBuBlK|l^${hdC394=ZY%$WjJD!<jQnTGu;ttn{@;>Jt_IzybqH(Oa*UORk0BqMg2
zV%ctaQGVXnp47$LdCC#!&2t;C%8raijY?FHd{s}p6khV4H2$>bXG%l?`Tg!{Gc`ia
zFK&Rnyg`uS@X8J=S19Cjq?#lEXOgMVRs9|ouMUu6_8P4dsYR|{pDGS%eJU-sY5G9Q
zQ*oC$T|s@D=Ul)1^WSf;4FtDi^%)y^<3qIfW128IoC*NF7+1-se9^byv)#+U64(~W
zv^3#1YKl+P9FnM1o<vIyDHJ<aWSw^$b?S9hD+e$mAIIDU&t<X0KB?cWBR_=CI8d`Q
z*1#KM)*EQK-N#XRG>sX&weD+l=D8?tcQLZhsi956Gy{M~z6lfoZP}ZZr=6bfZPi)+
zOQ1V&-Zvo+Q;kO3Y(3J$v#cy6Ma&c-9yLhybBY0Q{4|1{<{0d{1e)FoPZgj(j}})o
zrY#@ANL}*hMAdh<t0libhe#-u$D=vLtGuEY=BW{9{K{!(V~0^^iUr1Kk>y&P4W)<<
z4fxq<D2mC0MLf$2f-T!^&a@RTAh=hNu~5yBwMdAtvgaW8dM@;f@~frTZ*A}vrzOHs
z)5!f%;``b#!0?z;mPcSJ%wv*_OIuXQHGCvtGZ;DuQwsP60I4~1=SC#gjXA3~h_Nt%
z9n|5r{dx2B<*z1I^7jf{7Wu-p`k5D9$F7@YrAdjrzQ1f-!X7Yl7tD<=6b%PFWvAY}
zZMZ+n)Ym>2KLJNEFZwG=3<HFa-!aPMl%|9U1T4%TVEGbge{FKhrgEeCyhYQ)ZNI6$
z8-u<Uh{>*-jnBi;S}mXMDEbZgxi|HGO`;aR4IIgr+DI;%ogonH{Kg{*pJAh`{og3~
zT|shEMick9cPVEh`6evYTcNN}sG5XoE8EooE2ONk^dsP~zk3VQ61m3=H`xE>cUr#p
zVUEWv#~U)>53WnjZ0uXy%jI5aYHTj)c^&G>3Dt%Bij={@l4+KvdO1)qijz$P84f_l
z$${%KX8DSac-&f-5G4c43R4>nL;0?~-ZnzuYx`+O4p_**l-^}->%{_PvgC)CUvac<
z2o=4}Pt@z%Z%hB_{Y&GYC$~O+v;#KW&^)JRnAvKh(W)JdNZ4uH>zxYCR}$)({E@Mz
ze{Btg;x3zZRFWI=Q@D8evRwHtC`$0E2W@&3<!8F?jW<2QTm;FKhquiKqb>U#X#MmB
zQ}McDRXe4>m5qu|BsDjhcDD}Wfkci#%TrZEZw0j#Ox=}L?9S7EW_18o;^tEM(yNyN
zca0W91k1joK`4uhIb-bs#ve+I{CR6_kQ;GiXEFAb-{05oS5=PMlXe19|LsDdd!^pj
z%ttz+&m1*sz*!g<8Fgr&_Z_;<_&%LgE&s5eHu+tOZI#1%Wgh6MbxRN}c-1nwvn%pp
z=X_E5z5k(zegEn_qw$|U1kFwawwp<^0KYSd3KKdqO(x0{$OQ~owdB&rMPE9;#2y9~
z9p}dj9$ZXT!4nQXMpg))&uJT7Qam1>wnoGa<%_yn5V^XFs&=H`rv}=$oZJ)IXs8-2
zvfs};eu10ui0QA^q2rs`JNz2p{lM|Q`>2~beDJI%h+D*;r{<|@rSO@Z1Z1b=h3Y*8
zbC#Q?Jr)HXy4@F!y`r{?+MIw!81Cc2#<{0q+oaUUByNXO6rEe@|F|%>5nX&0g|e!1
zR{gD%hvgV*N=Z9$Ic?9{)FyKlx6yZ79s2nFnm4ng9Zx1Fo7s)>25-S%m|U$o)gahU
z2RnM3Mraj5MUXx8dN?dF<g(JX9G4GE&M}>93Gj%un|-eLxog<j-3)8*n~IlhZ=DA^
zo(C=SfHK4)Vwu=>?IzA7?D#0(FZ!&|w?E$ab11Pf!RMeHXZpJ1jTyRDTo-j<ojp*)
zIPkYM@pDZ1mC>Yp)RD(-!2d({?K*4mqIX~{q@PJQz^nY`)!_GXYbR9(7J5j69F)uH
zfmuDazXkR%hDXW#r2DPBlv10A`RUZdSpV8KeSWFwyUwflkR{KJc7N*4sDCyKCRbG1
zceUROZYepirkiwH0VB=@Q1wxcv=dzS$;*AEopXM8`5&ppK4Cb(@YU_RGat-UNjLXN
z_XZVFTTKHsH535EQkH*{XAvJW-y%+r*E2pOMo8S~S9&?tUW4+n$qxDbkzZoFmSKo#
ztVL?9AbWt#Zu8BpfN2_K<esASqtZHF0li>Mz9GEjsq(zV=%4T?<CJyMDOO52Rhz8r
z22ayiAQ{&tC`7SrL&aW?0f?>@p=~;U&#wLe*(oTG4l@HaV6Ya)>~L5OlP&GNG_IL_
zU%Ot=2zmQd(;y_j!I)tX5cx7P5?)eMt@kQQiTx}b-rm<turq-KcJ139@8jK^pUquW
zvzKmHF9W?WIqft%8)@(SZBJ7OZ5vQ+X+EQ1m%@g+{mO*lrClp3yPy=5K8Da6(ZS~k
zXi}MzLlR$4Dmn}kO*B8WI4}Bg65xQDgOJZ#E?)Ndwcp$_c2zvHw+3zj>+JT*+Qaui
z1jv1{$&wV<Ua7zQ31{{RxVxZy%5)zEI9>V#z*Gi=bRmo7#B$C_i-8_lpPa9}C9SKG
zq|rtmIv(S)BPM!B4{VCQE4KHC(sr!vZ}PmK`gZC1i}n<#fO=PfeQ(t4{2?DUpQnv&
ztIRNAh()UO_S$&aD~i{u7IR+(ey%P9?F@q@VWuuy=HW^?jrOO%w(BiRVSmsl$N5Ae
zY?`x)5M6>0GI$*8KDc0}v^Jz!=G8d3SD64~)XJ6)3O%1|!Bwr{RYNRObE=2FN3trd
z-D^wQLYd@=5)70WD5t4jtKv7m#|UV)2Xa~Kh;E#1>{&-Ye?nB<9uE56$lqoyowE)!
z52?EzbSuH`voZZS-(R{+<z5C1VpMrCY%MPDUsZNi7m>dWq|hW(pEU^bfAa#Vwz5bQ
z69G*R66=Kw2R@7lec*`@(<4wJ%CRx8XV10|qV9T$zkQwmCh|hd2XG0O{U+~wB2F@6
zrCV6afi7U>!N?+2gf@(d1~67d^qikd<8HAA@JhYdwObW`gfoJBe7RUkzFfV2^Ly3h
zJU@OxevZq<<T5v0{AC}zUlmXw62%jsUyoPI#dLQ8sM<tV%5-dGd{VUeYAB3GdtZd~
zM2siQqMYl`=@$G%3?{wuG+`}f<=TV1id7SHbi2|{JkVC6_>LsBJZl-ckS%v_M?DX!
zhl3e(^l|n1``FCy4?HUFqf&ipXhq82;fE0a%}$7#LVdm~yjM-{#q%mm>R!+C-;=|U
zHpek&=;OSIbmg`nP!p=b!#DPJ?VS~S)4^WC{iw&I3bRtK`VjE`&g^CE_Dcmv&CFtF
z>x*A_lhYd?dkK@}(u~`5&ci2_eVCV*-2Sd9nOWWo`zytIK_0kC(~I`CPpofZXO6#}
zvVO_6oUmF9T&5d6;5BvGcCKw6G2{}x;Or0e^hXL2A^iH}x8@t^meOeP#OG)*WF)}q
zESFQXz1-w;^he11iQAN;g?=D+k&uS44FCBCR-T`~8MW0C?X!{f`g4iuR5E<sP9xbq
zh;yvuP@advL4ZH++;zh&ap&g9(=fx@+EiDT0Yrn!F)?Cn?$o0g1wyMb7GM4;aCQmW
zNY;#DE$_J;c;F`4<F(yuNBV|qFeUdSZe7VAn>F|CN@S&+i^4)BGw)#B|2R#gm?izT
z)1k-_29I!VJg@4N`~9O_ySU4H>YOM!8hzenrma4ffJ@y*S50!q%Vtrm>EC8osXiMC
zW_$m+7G{?MR6<Z)K6Ejb^D|4d)p~@jTOUvBFbWyJrjI!j<k!(=eo>;2@8Uh6bAOpu
z)Sz5`?;Oh5<aX-}1X}`Z&K`iLzUL&A$FMe66%*jvI6la^!ys;Xy4@2c)iJ`=5+0+@
zpFUb_9?@-_;==t>F6P7Xc&wKizik+2R=(}3T8T#g#p&pwUZgJjy5CpPnlG}P_E|-Q
zqP*L8oGbNTvZHmIjVIw>e(`M$Q3gsb$;*6LP9?>ii9F3LDD=Ch+7-|sv3Z=<{+(Z>
zS1vPZYp%-dCU?YPVX|FaTWr^;W?6|!Puj3<e|Ds+F!MnNSD(XpzAWVpKc>jVC(qJ8
z!W(@J`NbOJkmaWZlSXaQG)$-_?jA$ABi)-Dn_iT)I)8X0WnQ%8KW~+A5|7<7bm-dC
zJ<~{D`10IX12wR4y(g|5r=9J&5?SM8O!e3fIqJR{nB53SHuK-}&yQ2F_&wbZ{%$Y|
zWC@z2@xfL_GuVw-_+8?gK!DOaa_=C70m%DJH$Gsx+q%-%zS7Z9&z}21##ysAM}kq7
zq+=(3GilW5H(4`%oQMe=df%e|%;k}o&{p}dADJAn4nt0qZMMw@>qD5^E3TL{N!963
z-jTLjdE1A15!+k`{c>c<`*r@+ZK?y^g?iSe;;&q%+__v{e1EKQpsf(?W5H@WtUFJ*
z_Bb@N-fgVE&^M;pSZSg-NMxW)+q_nYY%;;t`?2BH$#<s!>=H!`-LnQ4&K0wL7`la_
z9s`d6;ccnQBKJ0*J^hC{NrIu)`kFV45Uv&-qx5YpES%#T-13!8IidOu0SEe<^OTR%
zDrBmP4UYG&%M{a5wjcM(EF9+S#0Ps9{~YnUKxQ{%%c4Txw2R1#e4i*#hsRT`#8^}G
zw*FJ&*QZK26@f}o?%4LU3hs&iGh0?cOLmvQFCj)oiYrR?Fm5jUZQpL6GD=#kW%se=
z#_64~8KuDkt#!G8tN@I^Wk9P*kp0Be9N0Nuq;KuApga<ko-8GNbLiec{j)>+o=d?!
zZ=Da%t?)#$=-rgO##J;RYd`8KeJstbf#iBdxb{Z#7<MTOA!3U8Ot}rN5mznHq*CUE
zqY{w(IMc5fT_kwd{an!oSC^GtN!IY2Hr_bh2ZOyH=#9}&NouS{5Q{lSGu-9iB){=k
zDly-rv8a(z4gArT^#Q)^(Q<9|(sitb?dl{(PRD<Xi@nW3MpnbwKF@&&$)}9}>=&cQ
zy1?!Jif@a&ch)nM<tNE?6QOw*NL5Za{hP`|+jH|l8<WA{a6s2Qk@qBUfNV0=jfBsl
zO^1^&6$CS0wDF6URsRGX2%Iq}J~ltvic^K}b@*JZT>r3h%=G@&0DF)GguC8}H)+E^
zxT<2?H(L-%4bA$ErG1@(u?dGPKiv<BUMgJh`l@(mRexr!3ui{{nWvqG0}u?jdcUk(
z3m_9I*-wJjF5RkFR$^?R^SNvTL?(Dx3(1+mHL!?<7+?_hU5*C5OP=!S2r`j!G)W>i
z5f=L*cNh!)DI_3?g%U$vE?;%P#q?nRV7f)3DRTXLxRX(?t8<1XO)I%HC37MO!~O|K
z<*OkTQj~ufdpjSk#{HVsj7P=FRHqZ%*y{>`57gwlz@FaAO$Zbits~=gxh|8g2nI}g
z;2rWbJ6YjEJxI9T#~UQbHDI+A;b<vgyhv%>MC0?l12|OQvNxR$T^!ytNBzG5%q8o^
zha;OWqZ@$frf5x~syO#_-fEZDXjLd-yufU*?Uw3RfjvFw`Zo_{_mmbAcy#F$_I8~V
z+8#y!tNHgI7y#Y2#7hT*fo0x?qo$c*O}UO2ixe}5^jb%#ztYe`IJIm$dGwk727HRL
z%3GZ`8qR*xrDuCxpDVys#dEuBLvFc~Vay&_&Xf7?5k?CaJTvrP`vj9eQ<caqIimcJ
znF3Mj9zT3L+Z9Dl(yYe;`nK^asD?p`zF>g^?wGNzx3uE?(}&)U?dkBnZdXR2jsAKK
z3Gycfkx(;z>@#_oydK&Pz<Ra3E1i`Y<}4D{lDfML%O9Viy6k*iTs3ZkNsC{9)LXUw
z?bP4we0*PmWG^&lb<ifVUG>^e#}2q1=a)qw0x(f%SL{P9%?@IJ;30eR=@~T~s*i0g
zd-*<n0~G}n49M*tAH930<pSVETEHav)uVm+je-;C*&J+V+3?)qeUOirP1n`(@77n-
z2<BOub02cL#J;GjRnnv>&#}x*bJbWJ75URG<U5hw*ZpcwDmbcq!NaF%%wqrAbg+tB
zP3Sxb)7h&NZYx^T(dKgmIb`oeA&@zG9tG?#&6i)-#Gt`f1_#ZnDo>SkQkh<IyrJ;+
z@G;|QcKpHa1|QjTE3iv)hJE)X&h>KM9;PgnMtzC0-_31Wr6c>fe|PlBM=Aj>g_i^1
zjAplXSDG+A2^Y5?^KUcL!?O1ULf+j;8cjRH+oXF*@+0;t&s}{=10|Itm@1N*U`*#_
zJ$cewgg^f=wdCjS>jpsUyIN{N3`Af<Qjj1h1nlY$-|>Cp%Y{^c3|iYuy$1Q}O+Xb=
zv&goYGLqK}crEOe#}&g-(IKYlrv#L*tVsQx*DGs7L#jfHSbeMSzjE_kUB4abDf#ht
zgilfH<Kw;qsgG@$_vDMVM~hKnX1f5KcfL#PF~Gvee6E5+XM_~r35)Ej4}aWGTDIGW
zqTYx)^kzG1cuRFGMT!m&fEZ8^keKt%jfA~uCNe9j31kKsP|<;=PtF_nV#0bs@2nNS
z)Ki}rUu^FVB<sQ#{G?g7<j<(lQNFkRxa-Ll%9SF$n+;Gw!viIC#NUt(cS|cZCkHT_
z<ZfD}$|<s$Pfe3(*UIkEf%4e%%VM5m5o=M^8@(L<f|Gx(<C=m}ny%LN@dmISsm_ok
z-#RdMO$>13d5&>&OYKKkNCQp;`~1iv%R_(jk?ALTH7lrTx5!4D=a;@B`oF-vcvrT9
z7&w6xCUct}3$C<+pWPMKKzi)r8AYXs;D9I;<bfy%s-g2S>Pw%t)+e!xo`I<Uf!6cH
zB$M}i`}_73^4CpFaiE$KIJ5c4#V>LuSpknS8Q{G@3k?;W5@J@sNN6N{@>5t*A-^>4
zFHUNgkJ;f=!0#11%qg>4kvZ(W*BNa`*Edi`jU}CCxj*edDCK|O@tAMgbR<x9a~|DW
zOLFBWi2&6UqR_ebAsx<em~-};m5+!RfcRhj*_d~17jn6@{a_it!4&IsJ9wc7x(qy=
zW`&qP00Ghe%`NP)LhL5x!?NvmUK<huS#bz`xox;rltc5jy{~ites`~33CL)~ApOr&
zf?Wo~U1C4XLL)#%$w*&b9&YvWXVC2HrEyk1i&Y718ZlKaQVV|{`=I3^g8kg9Ug6t6
zVxdXiR9#o|(C3s(59$2j(!!$kLPIrSE?!Y!ldj)W5o)VDq2~e1Q-A=d`YI1fVVywk
z5B@wDGFmXK28KO|uq>~6cHXT~%LPE6B%Sd5xP!Xo=~MlNMQ1sKCHk{Dilk#)l*sZ|
z_iqPbIj*n&nLSI<iUYOT?<%{?1&fW$60Qd`>RXhkFv}ic)#KC2S8hjk70I4Qg~Q2=
zod>aeX$Hm(B!yF|8acSx>KKuowRGh`-B>co)1JLuXg$O6%hSY`50G4bnaQJuNPddB
zcoVl*FjE=Z$lP2(H>RlBc=Y*o71^QB<;2)6o&`Z<B<#7evhTaG`*$!vKtTr?CQ>e9
zh4;N+KF7}fmJyfTpMT5k^UWd|MthAR9EkB&@NM%y&mGZQjHmzQdW^kwx8=9pDEn|K
z)wCbzhWWv8XVIkbsAyPRRTegdBbGtGG}&p9i;gW(IXw!i+tQT_y%N+{0G+YZpBd88
zS;xIoVl|IY!0in0_jujqJF?OqVDqRMQc8ZOg%-}EPjTK6tceQOwfU?3&M|;@4I}^d
z@cZ^H62Fea9fR2hympi;-INjhUzK@=7f>2mQSZ4yo*!^qHcQcJgG`@3^{(kPAw5hr
zjcgepQBA(2w$ky2A2X<N`4d#61jC+i!3wBP(sB#+yux4P=_C6wncKaboX|+Ws^Ja#
z)r<K1RL$t<feN8G`IIBHD%H>d6#qXZZ-!aTwLy&|K1$YEhhVXFJ8>6RdYHDNrZTdS
z$J!4agQg27wCD({lnK84LSK^)S|=lYUkZAam)x$+hf6%ZUQ?>!;WPFq5_w{k!RcVk
zD?5@x+^3!<tEBbLe9o@@@92ZE`92-Gm=3zFJ7WH)c3N&T&>?-<e^7?6g!-Of^GcUb
zwo`B952AHB(?am$&dCf%atzShc|1u6?J_XAE_xNxYMgpr*;TO(0=*ijDur9`TMjSK
z(8g3u)_gj&660|kC~1(m400Dt-jGkI!NWO**&(evrmV&PBDi5Yo{IPj!<<y9d>!eg
z$N#u=vG3RxI5GqJ7bqFD9pI2C6YJmtPAC#{)xzxfU4~;j%?5QkAkk#Ed|;%Ek<fvU
zXS|;n3ax<p`*ep<mTUDrq}WW-2U{Swt^_N|JW?I$w^WOBfzZ?tX*<>%i@1X+EQiQ~
zMGZZ|UQ9vDnU_k0DfvxG0%5u#R;qeJ5^FA61qwZ+5$^8>934ye^tsKmVv$JGlh=AY
zq6N0(w|6C}Q<q6k+Bv$Y*L!hSALmVI$3v1%YJx^@zN;{<rjs!y=%l_6<G@O0{Egh7
zam}ea!pIX5+*f1{gisEh(8AL>?6Th}<`hGtjslMUz8@Q(6dp$#R0|h;k?{xsxBz_&
z2mkQDO4#=<w8ml1s1^+AqC<qcA%qIPIO}rTtF#S}m8!_OSI((fwUoa%IS`;UM!!!7
zFDmr~+#dlks7m1eZ?*E17-*dFcvjV3e<<`}^-q_y?rjsnC`-o)yUulkofo`h5#i{Z
zsf<y%qTc-%?kkJ?9BqjvBMee+R@2{E$89{Fw~J^QR~*8;@|Vqj0Wx?X#W15^lV_7V
zEG~0DIb;~NfwW4aUV{i;PW}BFZ0~J{(nbH<y`n$_YuWiJhu`g8ABIW9)lW@p8)a`i
z*9jVFd83ZL4A&}e{GDvzKIs{9v$nfP0BZuGyx-{SE$Ew~LkJ1EOw-q*(iX1_@ZwZM
z52m)Mze8X6#;37H_UO?J0FLAhtRVcN+B=9qAdT=4Jw?6!{RXi|j-5n2Nhm$4)xq0u
zyjEeWbEuJU^n+|QqEzIp<Z!PUya1l_f<(~Irt7lx(qA=+pYmyz8gG8Q!Nm=3@>(!Y
zS}sBxvZR-XgEK2`&3Tp1KFVi)yS{k8fryt3YrIP;`2I3T`>4`}=^huejP*JF`F>(q
z+TX?T^`~Eyb=Ik}5MvSp<ca)Ys5wHU98CBSgHD_KWQbnu{rX#mS}ZTbS7pZ}f%<PY
z!c9<qN}O8RbT_W8KSCtl#`#S_40xgwKk~>X(1GlD$4j9}MS;uq`YRx>;ngeB`cG4(
zjal)t()b6ZwF5ReKFchnGAekn^!pViVP{$dN)eDP51zGO0&QFzgvL8@C-IR9y$7V=
zZr{X`$Qgvb>&35}ly_0+R34{;w><WnJk|+61=5oy4ZcUahXqy&nZAj$o~u!&#w3~z
zy9TFbWCTSl62^KYnGq%A{^Hi|H^g_*L!&yvHlCNCGS!Moi9*P*8Y!7*JT7jT!;}n=
zl?IE4#WxaXI{tMLpml}A${Em7YYW<nNitAYGj-@RNAHfAElImr+f!JuP||HU^`r#(
zQbL^3sg%bjCICq$F+M(Ch8ic8&4r}C8@7OrfVo~$kh%F+A5GxEnIc2?Sd6*cPza_F
zN1sUKryMk2=5MAO0`w3sWfPNCz>LBq;f>4AW??be;1!RTdGRz5NWv@F0T(;`CRCO$
z#Mc?+Y!&W)AHD>?XEP}&rJw_!7W@{|{%m1kK~#g)umvlXE$vk|H_OvUof-!$+!h8F
z6X8(S>+8Nxo~Q*cKu99$DM<Rxc6&4{J3D&~N6JuQh}nc-pe=ZSVa>?hn50%=fQ1P+
zHA%qcrYKhMZO)x}R0%sOK3=jZ;DRhPx!HlfXR4~I;Fph}ZcW4@(H6IhKzWFzXp?ez
zLp-(+_8$rP@l2PFUN54Mt^&8RekZ*K*ap~nDnz8D{(Dnpes`A^=H}rQ9CFUG>B<_X
z7d)e&8=1}rHCK>(m53dVUg%7}!P!oPSeVv&&7<KeQk`{Aa-i2MywitZDbqXuQqz7j
z$-Pe(R5uh-`aT2=EWMhV+S=+WOn#HgW@H0()Un(ROUDm<E2E=UN0)<`#Ny;9Ke+m7
z!Vk1Ze)$fE86zw>vFd}o-J3^Tc>=wz-IFykHjQjK+N101H}-4h_$eLjQR=W^tfKM^
zpdVahQfH_5%a>8Wk6T+?|7y={_}jp+nJPVA%gN;1N|l&?13j)V8fJ7wmnHdrxOHt1
zt1p4uv!29p>3lPjIRQsh^W&$H?W%nHzqk(;M-D^ql=%TT$WU^}A4%d!&>~|Kj&y#0
zegHnV`QEs=yVu?8>;L>WT=9J2cSFWeX0%Y7Eix{%pUc89)PS@YkMTOB<75_3RWe%-
zye<31%`L**t^p<)gieu!*GO;K*}>}Yx$S9qdh+J5-nkAp6y0xH8P%t|o&J3PvEfrZ
z^xxWPeM<rHr!4}8lW_*6h+iCE{JJSO1Enn~<uA1j&GtZvggD^uGLe%T|M{weghBHC
z#}B|z#YQ%7G&zlxCH}FhmD)q7xIooSvnk)GlBBAlrB6E)Ea74Bmej{`BH3sEA68EL
zgWnp$-H%eHh>T-nJP=Mp(bE{f$7yQ1Z4M@bje2}}IXXNH9~fc9V>*$Z*?JWdpy%Xp
zfshpkzuD?1#b5HT$jJSxg7U{pWoFaw^2zrLb=oj|w-k-XV-(rF^!`8*mkcZm;9Pw(
zY}Hj%9GRZxqNSzf;^JatjOsD%Xt!~$#=|U+lFXm7zvNS#UnVr#X+JD(w4$z3@H+Zq
zMy6LiKN4j~pY5=4d@XMA2iJH1>zYg3i>g<4UqE0RKRY+~9&p3@<0xFWhWP*u1-xl$
zdKyqxD=RA#6B2&u9gi)1v@-nlW$=1amiX(1_Jx1CtM|XrFoGr35d9)yr^?+_zur*n
zbeiDLsJir4ufj&uC$v!8>@2aLs;vm9+BI?)7R}X1=<<UCOeOPVb7^$n=?eK>-k+z}
z>eav3tG1oqU{DX(pR(C3*Nn{(1QZp9b?y6cS)FnWC;1yg?15HIhkLe*>lWNEfzHu5
z{ykc1>3sI_rwcp+S?W?{@~;DrB@_r8Kj}8ODfp}E^n~h`t-OS$hgi^6s)5T6s?vH2
z@~f5gn0~l~0B2vy@*fG9pj40|*5@q3`zkvihDhKsCgi<fMH)wL9W{PR2K>0sx5mej
zZYm_h+1?s?2Dqfeur>uoT~+bGIJH#n2xw>H7w0=B;#DSL3#~*tP@=80IjRzyb<D}n
zS<jR#{pKS*i$X+jG^kl@-GG}o2Rc7HnG<RDqxe7jsCuI_TAixPYNcJoh%MfYILC}-
zd#-q8=#(h!H*^OVc}~F$qO`tC8pQ8lB`RXk#NiDZVo85mK=Cj;UYTZaO%5z=ap7{b
zR8eAp6A4p{(#K17!Rcc{R#0g6y;}WLHd!U&OOTu{O{{|3SsTf+(^Q}C=RFa)UHuhj
zP`@x<cGLfO3^E-%9M%@eQDKv)#*~kby7CfH8PACP<CFX-^vesNR6$EAq&?Jcx|Fcl
z$7+aDY^Ad0kA3wiGdcAlz`*J7*2i&>$zscZx{o1Uo-rwqIX6CN=KY3}Qt0of5X+Mg
zq)H-f{aX6kxMH9`G#4?(>}}>yQDZURwr;i1X-e-}(ML&bu~Ub{TugFM|9IIAH^#8z
z-J{yeh~hJ>mW{X}wUw;CHvv*+?O)aMD^y6=cwsB!T7KTjHjk)hsWl`uJ{P}-_~vdp
zD;l)8lkquU|AQW-Q4cQIB9F0n?_=L}a~G=lCg>!c48E8(J*~BF4ZoM#FU^f|QD9lJ
z%ijA^uK(IjUU!_YFM8i$ZnR8)Q;@bqFt;=gRV`{L=FOt!6{(>6s;wk{A_rj^1rVgD
zi+}<_3V{-**9V^dqV99h?O|rAfxJfF!LP+J0a|4SYR*ndi^>S%cnEm-l(d1GUvHM|
zAL_E|b8J;CDY&RBH-fn;cYa8C7w}PKQ~gFZt(+(#?!i;B<dWTiiU-qwW(-cHkCONp
z{uR_7K4GWFdni9lCul-$xj!fA$SA&0h1L%FamelT#QY$RSx5drHJh6G*5sv9`t*5t
zKV$oLR_!$XjR78An`k3OKLap98@Rsti5NMp4If6Da{uPQhxfzl=CLWDxZlzczN~7p
zWcGa%(8D#vQ;UZw#{iNypCZ*&Hgu=o5FVBSmdgd3XTGszx`&jkT!FoW4MHa}fk<`%
zUJMk7L5K&6LPbt2oDB7aFV*CbB2n|iht}J;t4X-M?a7&AI9`^87K*P5Id19nMf2_s
z!RcF*YF`VI&t!EJ440#Nn|R4c1)3=oLuDfM#4#+s`6gVRkdkeT;l7f#FH8aj6>2E{
zCkz?x|K{ATJzXfW)il*tSRXW-BDS;Mz82FECN63oQFG5IzP3zJv-DFE_t$aa@8e|y
z5Y-}<=1<VUf<>@Gf|x4oV`Aq!->6UG>qHn91o`%0ZRKQ`33!p9kJHUn5~4&vAt}li
z<Nko`%g~WC*Q`*b<lle`2?4hr8rKT!R$8lDpn7IdbqzMWPi5n{_f{D|-|n4nQEmt|
zU8?)vwr+ftxcx+@3IszKT(K?SpdlJc8U%O8=`Y`IGl!2c7Y>UH)0N3sztUzq_lK_(
z-LqP5EOzn8C|S5N`LmJjKdx|?P4}m(PDI~h{ec;GS-mO?K<3=C(25J3l8Qc}K1h$v
zmCObB>_HQ}m%gTnk2~X&e7qb*r6-4og$>lM&}<u;7^f#s&xoX0537<XMg-l~BvY8y
z?}Cjoak@-X&ar^)6H<Hq4b6eGndB(=GdR_Sa%-+pf65MLeeN|hXfwc4;oXQ#J@ry)
zIY6O|cr8)G9&#+9Rm3f^W3poM6l7wm6q>M#$<mVgIgfA;xN0C8mL<|x4LZNNhPi5W
z{XlhEFze>%w5g@xP}VYXlt^ILXBSre`oP6TQWfmu>3Fq%GC|)BiAo-JKF=-9<Zc)E
zzj^;~>vog8yYg{+vrCAP^2^5-g8@%~wi<zPdBQ!ts3mGma%=Ta_!AJxxPb-6$Ka=b
ztuj-VnzW4XTlxA7>gCg<KRvQGG-UT2@`TnsS+2v#{pB<3cAM&8bI_0*r)snWlc_x)
zftL;N(lHsFJyShVEybvRnOfQ+KXu^hjH8ychMy=@?f$I)i6Ci_uJU0jbhUS-2ybSe
zEiE$~5<?ehnc9wF(MO0p;fYUIKu7qpI$j*$CLz9ovm~QabFQ%l4X64u$JtGPdIT^r
zavF~%rA7E(j(sovs6{^&J8X24LOV^xD8`4+W0g-hePxag8s?bB|7JmH$*5bTrU~ll
zdrKP!-~{nf>hF<nU^_X{6&;0CMSLDc0~4+8(1@lyR&pt*qyQ1Ge)7oRqmZ<6ulagd
zVNfAup7bkaRyBx74Bj_-w!5LL>{31sLVV9f!8YRzY`<pN8cP|{12btTaWFZaiUxA>
zto`TEz=0hUTyVL3xJzMfe{On`vvt(=0NNJjGlg*S=8A@yAA0pgBmSE<XL%-8I@@<R
z&vT|~*c>oGzmNjUAZJ_si-y{J>psMk)h~tEbvvWkTRlZhDOUv9>ZHiI`NO*s#^B~Y
zCbukqoJy6r%ejwD6u7EI{{O&rNG`mYoWqk+&<6H7LtKah#O`{PNVA$h;8(~}GPgbX
z-#toDoGT0heM?u=t3XWkSFypUCa0{8<}AM>YxD6{EvLhJ2{pw7f3Z|w=ezd0)!Vnf
zS`a4`1LdFK#3{HxR=?ENeLRM<m@xu=T<G^yF`Aar4z#n+e;A=^eUVrn)X%ObbuQX`
zF(q33m0FdKKo`4o;km4CCublhdUh`SrB*k!?jBuHAUdlMK9)kN5GE3}Ea7^glc9FY
z7s%U~O8X6O5Z5jVG^j~I=p4|NR5N~bajSXRYD2Uo88xQv{siZQv##jPI>vkdj$HjL
zT^Q&6F<-o#I)*N&t=w#Tx!rat^ScI6Oz<ET#E~&m0Z(!4Jga|AC($QbHiLAmZkiWo
z*dA46F5i<L$ybEmH+}p;xG)f$A0yHEsq!wzHz=xZyMG=Uzfo9xoLI|JH6=AlQ~4d|
z7Yh|%#kT8}FrqE`M*p$s*FwNDGT%`ecKpn7*df*?{PTX@5PvL>$<V8}fy@8G>sMTz
zhy<R|xe{UAyr8nZ4pZ!mV0V!ys=<B}d8LnU4Z5BIvK^gBtSY8y&-bC~0`XU<c=)Fl
zzUw>dw2C(4Yj*}=u{V7sDq9c=S!gF>J7$Onzhj2;DMq5XHli{%&&i7Y@^HewgN_55
zp?Dw`a|kCnIEO=QkqA}M!{96)Ye<v50`vl!?k`#&=4jM;@+kro^XE20RY@3iO={Li
za$)u6$D@9B%>wbR-#VQFVXPZZdHUg?^Hc)6)L7?qoj|{cw$^KAGT)6^N_05`;fV0W
z!?Pj4Lp<P?uU7o(*=;1zu@ZrU3Loi0W3Lw9&t0|?Epe;%<}Y@}4;2P|*-&z7dI<5^
zzqt9pjw<}1EV!J*2kBt7BxTrO7Y31S#BRC*^Lm{<i`1{zu~#~!bYgrKz|t85>_2lo
z-O%MPg;}iEo_auv1P38&oOmJVRnQB_*m%fgk$Zk<+h5H>RTbbk-**hxJP?%mjHn4t
z;7Mrs>l8_j5st29^y2@e#pf9<h{2{og3^Er)d|t^{L@i-XN4`Pt?B*l7S3$?Pde_u
zBdeuiFS#=n7vPm00zLLqbzJ7D?!vjc_(Cc~rN`p{K~}C8bRG!QfKHRMdE#&Pzvq;h
zi0W64bCeHEQCLU){Ej_kpE}d6-${8ruQW~yz&!tXnOr`q#c5uaZQLQ<!Ftxrn}n#%
zvmFdiyd|RQRLsXGCUTD|uX6VuI4@^uGD;wI!~uSYUPxWu=ilhj;^MpS{PI}i8$D;L
zW7cdOkEG)}QB}Zru9JmUNz84Xgs8}_Yc9aN`6XX~rSrUA^bi|zbk#uPKRYx_m%}J#
z(<d7LHcYWO&1V*Cx!X(BEe9|kb&6tr(~mK<+lm%1hl_`ySm_`Q@!ww>xDV(*hFYd-
z^x2iDHBgRmE-~Dl&#{_~tU>tkZ(y3}KU$|{Bh{p|7iD=2A`f<0NmHD{f|=>p*C?Ug
z%gJ%L=G!hN&P+c&?B;C^eo`akW7kUMxOV;g*;hiRoE^wKYwb;B-)&3P?t9jdfzQ%^
zyKU2(;kS@OOZPhp>P!;oXeq0rMC|+{+K{?CDj&-W4$LDkk}^D@*ULtflt5lHEk|+N
zlejl4GdI8~yl#UM<JIb#v*z|D4B-Iu6*0q`f^s_co6V)u4<@W1R71xzTer%Bk3w^R
z&l+VYKoiB{0}5ki!2L-|QpT&0n-&o+8d>>yHefV@#E6z?u=pkYsjINKTy;NvlkwlE
z+4VXbaI#F}zr#4f#gd}~EU33XqSQqVHQMnyUf}Q>3+K4db~08PLhUDURbBg~i=N^Q
z%1@=deEM{i$S&BZik35EtumgoUslt#VD;Pwzp^I0qxRex0{r(~z$xXBr~Qo@qV*%c
z&wq^!!`BWR!V?mhB`Jb<SFjnRlq8dr)1J1HkaPrjCGW1^8N{h4`F#1VPft(Sxv;+e
zA)@h9>8wtVR&3A!bR#kp2j6}!2jP3MU~EMH`b-@$YCmMrvEyA?#&v%zG3~xj9pZM9
z(D{JQw?sDSw}-_(+)G!;s({$bAteljI$5WU7BIUP=?kMBb4Rudj-{0{MwBKrQug5;
zUel>wwYcs%`3G7$En(Y*x)TXR8jQwWkcu=H=7UC?GYd6LIH(e77PI0MkpR-`1#DD|
z`@7qn()yOp$T68r$KH|6ElC4eN&G}s&#j#1j*l1*DsL2U;|%tam)Pp7OPPO&2MV=b
zr@epGkFo@(kw1sO?VOvBg#f@>vke^$q^D_NA=w-(%|e0oUbVOR->CER2~ie_1?6>B
z(Z@J|h(<Uo)`HW}=@=wA8Iwpb-shELtQYFFTNSk28|!5bUI-_sQ{w>aR5$bP*NO8m
z_X#dU##Pw<{>2ok+Sz6rx%9vBi{3E+eXwo3LztndgWiGc-IgRNJp7g)Mp_JB%>}>y
zO~|vPWBR$~l|>GuJ{U^Fmx%zwMwOG#FQ~u02aCH{n?3j8vkE<AE2(YvKqk9iIy_Qa
zgJ3(u<bn57&<T3G>tv5yBK`%oCPaB7zIPJUWahpxJQtM*(Xq;FkY&<*JuQeAV<(SY
z5<gA-l0V6BfX}sxZ#_YtKqq#j5BDaH_CGTlQHq)-7~uN%&aMr<*aNm@sn3FDv+FB|
zVPmQZ$($0*r_yp%f!dwyNPNC`w4gxZ@pnA0)gh;B#7NbyBK-~&|6oeC`TW<v8cqgA
zND7q<Jr)k-4<#mk+pM~2darf_3DyKEJ~Xi}Kz5dxiY`B2>Re2Pg^2-TR!neBM5k|H
z?P9$Xz*9mlsi~!w9sdiXx<LJSDz@?T=?u;SngzxGGSt4GT)QjupvarS5d~XH`AG-S
zA9bX%ArJ04LW{oVjJ8w9R~vDRJkqs~v5)Cn9IrGLdGi!^HS!YT(mArt-`rC3=*;l)
zB)#nPU!Ttg$ipw>Vx43!*gppZj2!hN?-eyGUK1tD+IKkAPR$$6P21ms{!mP9_?0^;
z(w91YXm(#TvghJ^D`pF1T4Dg~0E|H5C2_?E+|*s_7UcfQ?XwVU$!by60U|Hq^n3_V
zL@v`j$~u!{4J0kPMMU5)DS>H}|AEU*M7=BR5?=C>8?Q`X703zj)J_E&+LIcs#0`f~
z$_!E09cS}Qa+jrr&g@YRg8oU*4nj;bUv4$BQ|F$X_5z0B`$g;~^3L=UHOfB(W(;1c
zY8j^?s_jr|cZsYU^ok;U)CgRJ)%PtZW6Q>(WLMO-w^3<!4%d?OASxJmg_q0;nxvv-
z_lI=M-)H;J@lmPfTfr)qB5f@>jkzdQS{n%?;o<aQx5TdTmN<3LCpBKc{6Vy<Q-n;Q
z4V(>?H*gSsu*MpJ!S*grrN--DF*SjYm-w%T0XaTt-2upA_tEFuM_0Q0wD<hi5vA?O
z<|8Vxtvi`IyKioQDqMl}DDuz#$NEl|W^=B87q!MC6jzk|ndl@p<fnd|*>z#$AQAl^
z-*MbnnyKzpaSmYuS;s5D`}Y^zwSqFjYA>EyD*0hoY>h;lYr4$Ob&KaoIV~k^vrE69
z&{u2x0o$={W(#{R+m>DgWITLE)Om<$dtAFq#0Aey#Gmkzy29GZ*{iUnuzFzs$Dw|t
zg!u3idY5U@fgZi^E9C{;XupGP>;vDcss8BpGfRvL`OL78+TnG~WzEngij}F32xm)!
zqnr0jpa5Q%^WAHOp!_YH9@g=^H`SIq(5=1)BaQDaPVW{s!pH|f8Sa04d|_#3LL=bD
z|K!_L)<w1U@3-n|rv9xg9(aw1LS&=NDr`HFR)No>{Pr+N9kDU=OV>)oq0d_&VrgmJ
zJvtTcw=#lerTGsMJ<RA~b=)!`ACzu^f&jbbVgN5@3mSAkd&_!l9l`nCW8>hFMjON=
z9ZhG8Koq7^>)M0OMkydN0m}f{N%^x!S10nC+=gMLLSiW~8+vDzd~1=Mu$Z7JzN&cj
z6@X&}M4ugLFNQ@?^BOH2yUxaV&F$H8yX1GIweAN$9vg;i2%V1jy&2^G%>40U=2$Bb
zIY{GqjNtu>IN};o><B|k)V>Z3K^RsbCEHsyE<c50*5})zr>0@(W$bPvNh%U)fMJBI
ztzL90`6mKOcQARCR`hLJxoJNcH%Z-@5fwz9AnKpI+y&56_Vs2bH6Dz0p0V>nUKlz0
z^GKf~rh21M+<>hQ7isP#B6I6>vOuh&i%b%13fF0xM5ox-!5$;bJ{78`mJ+wv`<%56
zu=GKt3Z^eH-f{er1&GUXpHnVeus4bHN|xc;=t{`u>Fdi(H+V66U?l)L|Ek5qcz7!O
zz1P<@l6G5_ab<WoA<TXnCH|`sCclkTp9z+S7WL3Cpegm{x&eip@W)87w&b<3VX380
zLC8Teukn(j#Z+bCaQl7$<1YUJ$J+q&=CBmAN10kcF%dnV7hG^RF4`INRpowOm55>$
zo`9oC1db;8XH~JV50g|r)$aEq9$g|LYSZ^|R|q9&rXr+4<fq`%{xkZG-CY<yi!T&j
z^NI;b%XgC6Mr4;>uSZi_QgOh4fzp%z?l@$Cx2?~@{+<Te(e});-<r*nj}B-KTpmC?
zT~g`UR)Kc{Z9w>Q(>zyHk1$!e<&3-`Hyqd+xs$(o@bLlvJ=o<q_$a|wVyp=g`V1lD
z;hFz?K0%ThIbZa@bCz3*RS2|G|8sJ|#pyqgVJ~NK>F5~SIs0ej*pW%0b+->2uWKAU
zd%_6P@J&<ieM{gJglDZ2WrvheNB#Y#l1U_7oKxc0hmeU@a!HjmMiqHJPK!fvbVZ9Z
z9EKl${@Dlug*B-E<3%RZ@|!sO=tp@Zei43IL=oh^WmfeKnC1}@j>R7F21b2&!$ZZZ
zZ3tT!{a#Hh>2)v>RYZqHPNq2q+A&W7c5+nha+hrgY+kAuiL<s_>zN_xZS?^5aCcm$
z2ce%*^fV2jBMG>APXJBV0iroj*-DHlrQu+!aR^+bRY@e-nOL|SH4c8YJQ~Xl3F(Q^
zk(*VrMT5GnS2`A=iX;s^;g@Cns>^QP#fp~-qBiGjmbQ2mTgpwg^#Avv#F27-Rjz7I
z?sD+B-V6g_s)41SHpRbd$YPU`xI5R4cToK8W*S23;QOG2-TdQ6no&$CvXZj@_B|w_
zYvc9{_0+wIHahJ8&Ic8vYAgR5!FFPNhL*KEPn;<>mZn!ad15byhumr^DiaxE4p9lx
z|0(Ru<DuNU04^pm*2t2C7|cwCCX^-al)*%pY}chy=vJ0)mh9UIV^4_4nq@4>MY=+a
z?b?!Ly{<J%5tFi1gZB)#`);4l`_5m#&*xd@{FZZ`^PK1WzGr#YKBuX5T*67WE5TM-
zzbZx|d+I0B6S|1V_F#|G19?2(Z}(~~M*yjF<ovtON^(!K_NYss&xY;J$q9d>IX=l^
zhZFqtNbv}pW%XQZ84meDrSY6c&NVbP=V~Vr;}!VavRuLWdpA=cm0cgr%IVNt;T`JG
zb2vTCMSrg@1(_;e78^(M!^H75jb(f^jB=n&#Z?WZj&V}6sS+;bF)R9=ooHOWB-$!j
z6^%t)1+cw+bji*2cj>F$`nDClT<r<#`ue2(;zj<>zg|^%5R_z*B9WY&Y{ea7Y}8X+
zP0d(zH{@vRBDTiDM(?N0HH*`{$Lg2Ri8hRS9^&$DdM)yS2_lzDA?iKkaKYa*K_fkh
zbrl_~SQU|^1*=CT&3;!V&d<~A%s9w>yFsPnv#D8ApRwgZ?FJlOKHd%YY7r^YMHQyp
zjp7LjC(E;qq6@RB6OL)}2fhrq1Y1OiXln?_nSUP#<{J8(>ltuaxy*=i;+%7x$_+7f
zf?4J#0%`x=Qa4v}#@F-mKN5M<?La?7hp+stPvt^FSyf}$4w)EhllS@iPTl{J^fPH#
zlk<f3D|#PdJv`0XDKk8;rF;n&`A{-E<9?R0!U#Y%{?bMWE79L1>iO85CmtEc%dpDL
z$zeZ+&u(IDU7ONPGpT<h7^uMa*zNH(w2OQ{!T;g4oxAaw9u3^o^WebEVo8wh(*1>Y
zPM_uj)TSnPTkdGoE|Z`Z<JdtN^VQA%_XezNv2os9#nQ>w48Qj_rFBoz(%)tPOg4mL
z@b+AFS%w~qw2KdZUu#Cix8(YbRXkyL6Fxlbzvjh%BAv+++P0iP8)&<WGJ01z{OOtF
z%SKsLBo+1MlSEyB#?7fjD=y)8R>Sh;-qRDzTps3YCj1ls@KQBZndiCOi<A?dAIp!u
zsj95K<jMp>ZJ9ZU!%&>EDIG)YcC_*&F3XYuC~$4t-+dOa*i3V|C0|r7)$m-c@I#vu
zb_-+?pT*Z!gv59i=DQud<3l}Asq)Z0{oSYZ@FccfeARx7q~kK-xRNmD%IqKcExDlX
zc9<;oSB1dG2Vg_`?VQxdwl60xHigHEubX^2<HWmtE<XZ)n_bs+F!OHqa2e^r&t2A8
zj#v3SVAmwG7YMwVD3L^)&eP!Y*%<Y0fB$%3EEp5&*Upq9REmSFth9$DJ1w-R=Cia~
z8dx%1*)JoVK@&R73q&z20?(ft3@O|k`9GJKj=r3UbE%0H=QDUXY6W4tt%I;TAn4Q&
zLybbkzKs4k#zWu27S^p~EDo}iKAUMFRn%tE(XMm1UF-6G`U|!*`~P_`p;eG@xs3d7
zFNQF4XBlNg)bVi<l?lK5ds2xdA51SlBHj_^pDabO3EAq&v>1~b!67{es}cUZntiJS
zZ7N%$=#)ZO8cud)nw<Z>G!;uoviiCMj^OQii?O5#M^SF}5Lz3sZJ3StIjG_V4Vgi3
z)1jeT8p5oW1GO_Cph$y8HE$#`1RAdUp+XKcrq$8dts8j!zZ<+2Bqb%@m1g#%3o1Lb
zq~=gM{-g<QNfK*mOZm}sTYez*{SW#7U)M33tpmJq=WET_xeqy^$2G`O9k3E7y&dhV
zep1WHexUbd%%NN3=$y!-v9_ahUd&?kj+ZK`1-18UKaTarE@kB13a;jv$(!NvH$B+l
zYXRA+NGLNKrs<Wh5a6~X_P9V14VP5k{8qZ=Fs%}^a?iV@6ca%7eqN_%j9IjTxFkjs
z-o?l{0`<|g>;wYLO&g?-a^p!67j(}<Dkw~$J-8CExSJJa9sv>px?7GQ$bz>&l^hRK
z+6J>~1^RaYjs4i!1~YGOVn#Ux;h2^A;nx!rw?AK%mX^Msp8g_TLtdiNXBp7uHa0fR
z71}X1HQxZ$+WcA!4LltLf#Ch(EO_EzDUor?IvxNUZEMrl%3PeC^_m-aI45amXP0pQ
zKKD~H^2j`ZGH=|tL5~QwY8J&Q8JL?Vi4K?m(hh{^@VyIfmxTbN@XD1h>g_%2?EC>h
z-U)<R>eABE{QUgN!YCJfoS|POSzeF+@EEEWHAe{2*(xh5OUx$eulolF!*y4Hh^xdI
z;$OtU7>XcV6HwdT51eyJO)nPz72s^P<QcoIE}ZlA^*wc}>49FH#PMH#(9_dnj0T7;
zRaiE)6;OeJ%$|v`wjMdH%?SwDz;7#u88KdOyZ5KulBBGqZ5|R)u#pljBr?b9sxk?m
zq2-^Nnu>IN3YcI>w!J1>1W4tR2FU!tmHB(Wp?0qg1JfTRCs$NhNJGt@mg3JXC~))_
z9*Px=>O;9o7kSJL)CH`IRVf0q5vZvE3ZI&uMz~!Z;Ta!=L1r)*zuCCx>r>WzHv%&>
zz0VdKozo)htUvCqp58QB>-F)Z&7C$N*<MsYJPT?E-f@HcfrAGP7&5o@^R(NG;M;~h
z5grOK2DbUp(a~>TWHzB7q?Pl>lk1t))oKj0_4V}@*@%WiL#}u|f;+bqRx6V)&Et|w
zx{P+s25>faiiPo%=F54cJ#!o^ot?d6a(qZE5gz|ldb+y0Q@cH4{35SGeh#ETI5;@i
zUb;b)Zf$L)YXHEORZ<l8yU^qh$jODJSifbiw%(K+vIUEJ(a_M3fL&tBK|Z52yDNa!
z@R=?sY^nXGEnWtX#{<iDTDSIH1X8ue!os4ZrA0zQVybtOEpS!oUKf>$zI4GNL8Koj
zqj>Gm&`>oswH~2|Ru#IMcFXT!D4JHgFa7H7?oNR(S4=I5k;4w9Jguo|dUX5o<7L1K
zXPpG*{LItpYFbqox0C&o>*Gd;`h%b>6Fyc_n>RH)GX=Pu({DHfx_hqa=;+wg`6N-5
zPmu6^eSNkgCm+0=s6Z+Bb2~w~X~1mv{6w=)5D`M$LM&h~m?KI`LG`xH{r$z0ib-8v
zhO>%~F4P-AHWJe|#Ai?^2Af)1cB(3PYPXJ!WzcLJBsb`3r?_Qwb~Zsq5v%Ma7nk$J
zb#GM+1^N47CH_F7i?BF(0{NLD&A?2|&vVzi!|>)#q~%g4*aXGIko!`B`Wt0XYzaHs
zj;X2b_2F7_%j0Ik)klCV+}Y9b!^pm4E~rR4v8<w^c!-gjmXL|qBLtc7pkz>EEG0di
z2TPV?5fB&WE+sq1c=0cMEv}<Vfog=7?g_X8sW?(L9`i#-hPd+z3Iq#C{5wcV$;t9^
za_1S!)=yl7;;2^Br_JbZF4PC`@W}ezZBu3K;u+a5DJi+u&JWrO6Vy`}RsSI__~~@)
zY^sh6VP`NOi=TjP2-o=WFT|iybs3r0P@|w(l!JoQ;-o*xS}UM$_N^@O!3TT;u}`8x
zvS4H=NHLDt(-&W{1!`RXqmmN~?Do#*&qL1~b)rNZ;pb1%M`1CTU<C`@oj^3Gl`NLS
z>E=B{ImQlm#@dY>*0jqKajR1;Q2={q%ETJXQl&R_{hM85V<W{sL?uo$3i&KKD2Qdv
zjelc9W@DqLzdy8FAo5Off0Hpc2-S(+z%aCv70wAjcbS?G?|Zk^Dc4@N#sF&iz=rM^
z(597zfw?Nq8-85mo?*hD%e#UXXq}y0RXA9}N}R5&*J+nm0ma_TLcz^{4QaSO0LeWK
zztf3Bb-u^^1pXWyoqQ7Ey3D%p6$1WSkAgnD8ULE8`oV)-0W-Eyev^%%VCg=8c3+xM
zi_gav@$6Ft0CZgEjGVlDQ~vP4fUwqO&=as(B5@{SB)plq`B5A$>moUh_T|&t$)8yh
znmwL@iD#}vrua`Iu7Pz{P*5oIh8(b)9v-3N>Q@v|Px@}>W3e2{#yux#p8j8!X25z$
z#CgInw!29s>6xXeI=i~Ml2^pE*!kDY0h}E&uJv&~4o`x#(op!Ca_YTY5fl_?tP)l2
z0nHp99`^S3270-)F)OO~7U8k)stdrI*A^y@vX5L5i_wj*Tw>0<ywQCr1*)_bXOGKS
zEl3=OlI3*$Q)VC`EaP#&;6rz0H~aGJo5Fa?A|N;F7iVKav2|)RRm!@`Df`vCvK;~(
zh%z2^kN1gQt(&iI5q4U?g&GuY)qj?k<AxSy=jKd}Y)vId_=#k+mX<`o9y*Z{6O*JF
zL+02v6TSRNe)*UP<i?(!moGt;gD+?vKK$^7v5HG$4tajGM&u7nyguSN!rN7KZ=Qe<
z<G|Y2<+-V;OKYeGy{S(JXGnjKta=nu)fhO?tougmd_5uUG4Zg9JwTtKBO8oGsA&WS
z+!--63oQP@M;A8z#&OECfQJsV>M4mb@Q7gm@i_AqXl;Z1MX;=yfdZh+?3uTI8b#Oj
zet%|r2%A43Xc79aKz4zO(O*jvs_-zK1vD0i#_HS!uNJ~B4BQg9GJBcI1knI^{s-*!
z=824i%GU9+|LH&6@w{xYD^A>;y^<#yGhWm6uf7DoECk*Ez80UDD6e>-&}Bs7h33ni
zoZ7eS)7>GzCnhHxt90i<FIt_}9PG`mt*rex`^8|}yV-IpWiZ$t!EJX@@!vho)DA++
z<S*#gOg~dc?k|5cOu7g(6vlYl4OP^SM1DpI3n^P9-hkV2NJB#V_uEdf2yNe2MoUB}
txzm|FM#w=N&LOuUh@{=$)_=2E*kqAV>Wv<iEYD+sACX|BQ+m>t{13q)j6eVY
literal 0
HcmV?d00001
diff --git a/drivers/common/Makefile b/drivers/common/Makefile
index 96bd7ac6e..df2e840cf 100644
--- a/drivers/common/Makefile
+++ b/drivers/common/Makefile
@@ -31,6 +31,7 @@ DIRS-y += dpaax
endif
IAVF-y := $(CONFIG_RTE_LIBRTE_IAVF_PMD)
+IAVF-y += $(CONFIG_RTE_LIBRTE_ICE_PMD)
ifneq (,$(findstring y,$(IAVF-y)))
DIRS-y += iavf
endif
diff --git a/drivers/net/ice/Makefile b/drivers/net/ice/Makefile
index e22c34287..643639a50 100644
--- a/drivers/net/ice/Makefile
+++ b/drivers/net/ice/Makefile
@@ -11,9 +11,11 @@ LIB = librte_pmd_ice.a
CFLAGS += -O3
CFLAGS += $(WERROR_FLAGS)
CFLAGS += -DALLOW_EXPERIMENTAL_API
+CFLAGS += -I$(RTE_SDK)/drivers/common/iavf
LDLIBS += -lrte_eal -lrte_mbuf -lrte_ethdev -lrte_kvargs
LDLIBS += -lrte_bus_pci -lrte_mempool -lrte_hash
+LDLIBS += -lrte_net -lrte_common_iavf
EXPORT_MAP := rte_pmd_ice_version.map
@@ -84,6 +86,10 @@ ifeq ($(CC_AVX2_SUPPORT), 1)
endif
SRCS-$(CONFIG_RTE_LIBRTE_ICE_PMD) += ice_generic_flow.c
+SRCS-$(CONFIG_RTE_LIBRTE_ICE_PMD) += ice_dcf.c
+SRCS-$(CONFIG_RTE_LIBRTE_ICE_PMD) += ice_dcf_parent.c
+SRCS-$(CONFIG_RTE_LIBRTE_ICE_PMD) += ice_dcf_ethdev.c
+
# install this header file
SYMLINK-$(CONFIG_RTE_LIBRTE_ICE_PMD)-include := rte_pmd_ice.h
diff --git a/drivers/net/ice/ice_dcf.c b/drivers/net/ice/ice_dcf.c
new file mode 100644
index 000000000..669122331
--- /dev/null
+++ b/drivers/net/ice/ice_dcf.c
@@ -0,0 +1,651 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+
+#include <sys/queue.h>
+#include <stdio.h>
+#include <errno.h>
+#include <stdint.h>
+#include <string.h>
+#include <unistd.h>
+#include <stdarg.h>
+#include <inttypes.h>
+#include <rte_byteorder.h>
+#include <rte_common.h>
+
+#include <rte_pci.h>
+#include <rte_atomic.h>
+#include <rte_eal.h>
+#include <rte_ether.h>
+#include <rte_ethdev_driver.h>
+#include <rte_ethdev_pci.h>
+#include <rte_malloc.h>
+#include <rte_memzone.h>
+#include <rte_dev.h>
+
+#include "ice_dcf.h"
+
+#define ICE_DCF_AQ_LEN 32
+#define ICE_DCF_AQ_BUF_SZ 4096
+
+#define ICE_DCF_ARQ_MAX_RETRIES 200
+#define ICE_DCF_ARQ_CHECK_TIME 2 /* msecs */
+
+#define ICE_DCF_VF_RES_BUF_SZ \
+ (sizeof(struct virtchnl_vf_resource) + \
+ IAVF_MAX_VF_VSI * sizeof(struct virtchnl_vsi_resource))
+
+static __rte_always_inline int
+ice_dcf_send_cmd_req_no_irq(struct ice_dcf_hw *hw, enum virtchnl_ops op,
+ uint8_t *req_msg, uint16_t req_msglen)
+{
+ return iavf_aq_send_msg_to_pf(&hw->avf, op, IAVF_SUCCESS,
+ req_msg, req_msglen, NULL);
+}
+
+static int
+ice_dcf_recv_cmd_rsp_no_irq(struct ice_dcf_hw *hw, enum virtchnl_ops op,
+ uint8_t *rsp_msgbuf, uint16_t rsp_buflen,
+ uint16_t *rsp_msglen)
+{
+ struct iavf_arq_event_info event;
+ enum virtchnl_ops v_op;
+ int i = 0;
+ int err;
+
+ event.buf_len = rsp_buflen;
+ event.msg_buf = rsp_msgbuf;
+
+ do {
+ err = iavf_clean_arq_element(&hw->avf, &event, NULL);
+ if (err != IAVF_SUCCESS)
+ goto again;
+
+ v_op = rte_le_to_cpu_32(event.desc.cookie_high);
+ if (v_op != op)
+ goto again;
+
+ if (rsp_msglen != NULL)
+ *rsp_msglen = event.msg_len;
+ return rte_le_to_cpu_32(event.desc.cookie_low);
+
+again:
+ rte_delay_ms(ICE_DCF_ARQ_CHECK_TIME);
+ } while (i++ < ICE_DCF_ARQ_MAX_RETRIES);
+
+ return -EIO;
+}
+
+static __rte_always_inline void
+ice_dcf_aq_cmd_clear(struct ice_dcf_hw *hw, struct dcf_virtchnl_cmd *cmd)
+{
+ rte_spinlock_lock(&hw->vc_cmd_queue_lock);
+
+ TAILQ_REMOVE(&hw->vc_cmd_queue, cmd, next);
+
+ rte_spinlock_unlock(&hw->vc_cmd_queue_lock);
+}
+
+static __rte_always_inline void
+ice_dcf_vc_cmd_set(struct ice_dcf_hw *hw, struct dcf_virtchnl_cmd *cmd)
+{
+ cmd->v_ret = IAVF_ERR_NOT_READY;
+ cmd->rsp_msglen = 0;
+ cmd->pending = 1;
+
+ rte_spinlock_lock(&hw->vc_cmd_queue_lock);
+
+ TAILQ_INSERT_TAIL(&hw->vc_cmd_queue, cmd, next);
+
+ rte_spinlock_unlock(&hw->vc_cmd_queue_lock);
+}
+
+static __rte_always_inline int
+ice_dcf_vc_cmd_send(struct ice_dcf_hw *hw, struct dcf_virtchnl_cmd *cmd)
+{
+ return iavf_aq_send_msg_to_pf(&hw->avf,
+ cmd->v_op, IAVF_SUCCESS,
+ cmd->req_msg, cmd->req_msglen, NULL);
+}
+
+static __rte_always_inline void
+ice_dcf_aq_cmd_handle(struct ice_dcf_hw *hw, struct iavf_arq_event_info *info)
+{
+ struct dcf_virtchnl_cmd *cmd;
+ enum virtchnl_ops v_op;
+ enum iavf_status v_ret;
+ uint16_t aq_op;
+
+ aq_op = rte_le_to_cpu_16(info->desc.opcode);
+ if (unlikely(aq_op != iavf_aqc_opc_send_msg_to_vf)) {
+ PMD_DRV_LOG(ERR,
+ "Request %u is not supported yet", aq_op);
+ return;
+ }
+
+ v_op = rte_le_to_cpu_32(info->desc.cookie_high);
+ if (unlikely(v_op == VIRTCHNL_OP_EVENT)) {
+ if (hw->vc_event_msg_cb != NULL)
+ hw->vc_event_msg_cb(hw,
+ info->msg_buf,
+ info->msg_len);
+ return;
+ }
+
+ v_ret = rte_le_to_cpu_32(info->desc.cookie_low);
+
+ rte_spinlock_lock(&hw->vc_cmd_queue_lock);
+
+ TAILQ_FOREACH(cmd, &hw->vc_cmd_queue, next) {
+ if (cmd->v_op == v_op && cmd->pending) {
+ cmd->v_ret = v_ret;
+ cmd->rsp_msglen = RTE_MIN(info->msg_len,
+ cmd->rsp_buflen);
+ if (likely(cmd->rsp_msglen != 0))
+ rte_memcpy(cmd->rsp_msgbuf, info->msg_buf,
+ cmd->rsp_msglen);
+
+ /* prevent compiler reordering */
+ rte_compiler_barrier();
+ cmd->pending = 0;
+ break;
+ }
+ }
+
+ rte_spinlock_unlock(&hw->vc_cmd_queue_lock);
+}
+
+static void
+ice_dcf_handle_virtchnl_msg(struct ice_dcf_hw *hw)
+{
+ struct iavf_arq_event_info info;
+ uint16_t pending = 1;
+ int ret;
+
+ info.buf_len = ICE_DCF_AQ_BUF_SZ;
+ info.msg_buf = hw->arq_buf;
+
+ while (pending) {
+ ret = iavf_clean_arq_element(&hw->avf, &info, &pending);
+ if (ret != IAVF_SUCCESS)
+ break;
+
+ ice_dcf_aq_cmd_handle(hw, &info);
+ }
+}
+
+static int
+ice_dcf_init_check_api_version(struct ice_dcf_hw *hw)
+{
+#define ICE_CPF_VIRTCHNL_VERSION_MAJOR_START 1
+#define ICE_CPF_VIRTCHNL_VERSION_MINOR_START 1
+ struct virtchnl_version_info version, *pver;
+ int err;
+
+ version.major = VIRTCHNL_VERSION_MAJOR;
+ version.minor = VIRTCHNL_VERSION_MINOR;
+ err = ice_dcf_send_cmd_req_no_irq(hw, VIRTCHNL_OP_VERSION,
+ (uint8_t *)&version, sizeof(version));
+ if (err) {
+ PMD_INIT_LOG(ERR, "Fail to send OP_VERSION");
+ return err;
+ }
+
+ pver = &hw->virtchnl_version;
+ err = ice_dcf_recv_cmd_rsp_no_irq(hw, VIRTCHNL_OP_VERSION,
+ (uint8_t *)pver, sizeof(*pver), NULL);
+ if (err) {
+ PMD_INIT_LOG(ERR, "Fail to get response of OP_VERSION");
+ return -1;
+ }
+
+ PMD_DRV_LOG(DEBUG,
+ "Peer PF API version: %u.%u", pver->major, pver->minor);
+
+ if (pver->major < ICE_CPF_VIRTCHNL_VERSION_MAJOR_START ||
+ (pver->major == ICE_CPF_VIRTCHNL_VERSION_MAJOR_START &&
+ pver->minor < ICE_CPF_VIRTCHNL_VERSION_MINOR_START)) {
+ PMD_INIT_LOG(ERR,
+ "VIRTCHNL API version should not be lower than (%u.%u)",
+ ICE_CPF_VIRTCHNL_VERSION_MAJOR_START,
+ ICE_CPF_VIRTCHNL_VERSION_MAJOR_START);
+ return -1;
+ } else if (pver->major > VIRTCHNL_VERSION_MAJOR ||
+ (pver->major == VIRTCHNL_VERSION_MAJOR &&
+ pver->minor > VIRTCHNL_VERSION_MINOR)) {
+ PMD_INIT_LOG(ERR,
+ "PF/VF API version mismatch:(%u.%u)-(%u.%u)",
+ pver->major, pver->minor,
+ VIRTCHNL_VERSION_MAJOR, VIRTCHNL_VERSION_MINOR);
+ return -1;
+ }
+
+ PMD_DRV_LOG(DEBUG, "Peer is supported PF host");
+
+ return 0;
+}
+
+static int
+ice_dcf_get_vf_resource(struct ice_dcf_hw *hw)
+{
+ uint32_t caps;
+ int err, i;
+
+ caps = VIRTCHNL_VF_OFFLOAD_WB_ON_ITR | VIRTCHNL_VF_OFFLOAD_RX_POLLING |
+ VIRTCHNL_VF_CAP_ADV_LINK_SPEED | VIRTCHNL_VF_CAP_DCF |
+ VF_BASE_MODE_OFFLOADS;
+
+ err = ice_dcf_send_cmd_req_no_irq(hw, VIRTCHNL_OP_GET_VF_RESOURCES,
+ (uint8_t *)&caps, sizeof(caps));
+ if (err) {
+ PMD_DRV_LOG(ERR, "Fail to send msg OP_GET_VF_RESOURCE");
+ return err;
+ }
+
+ err = ice_dcf_recv_cmd_rsp_no_irq(hw, VIRTCHNL_OP_GET_VF_RESOURCES,
+ (uint8_t *)hw->vf_res,
+ ICE_DCF_VF_RES_BUF_SZ, NULL);
+ if (err) {
+ PMD_DRV_LOG(ERR, "Fail to get response of OP_GET_VF_RESOURCE");
+ return -1;
+ }
+
+ iavf_vf_parse_hw_config(&hw->avf, hw->vf_res);
+
+ hw->vsi_res = NULL;
+ for (i = 0; i < hw->vf_res->num_vsis; i++) {
+ if (hw->vf_res->vsi_res[i].vsi_type == VIRTCHNL_VSI_SRIOV)
+ hw->vsi_res = &hw->vf_res->vsi_res[i];
+ }
+
+ if (!hw->vsi_res) {
+ PMD_DRV_LOG(ERR, "no LAN VSI found");
+ return -1;
+ }
+
+ hw->vsi_id = hw->vsi_res->vsi_id;
+ PMD_DRV_LOG(DEBUG, "VSI ID is %u", hw->vsi_id);
+
+ return 0;
+}
+
+static int
+ice_dcf_get_vf_vsi_map(struct ice_dcf_hw *hw)
+{
+ struct virtchnl_dcf_vsi_map *vsi_map;
+ uint16_t len;
+ int err;
+
+ err = ice_dcf_send_cmd_req_no_irq(hw, VIRTCHNL_OP_DCF_GET_VSI_MAP,
+ NULL, 0);
+ if (err) {
+ PMD_DRV_LOG(ERR, "Fail to send msg OP_DCF_GET_VSI_MAP");
+ return err;
+ }
+
+ err = ice_dcf_recv_cmd_rsp_no_irq(hw, VIRTCHNL_OP_DCF_GET_VSI_MAP,
+ hw->arq_buf, ICE_DCF_AQ_BUF_SZ,
+ &len);
+ if (err) {
+ PMD_DRV_LOG(ERR, "Fail to get response of OP_DCF_GET_VSI_MAP");
+ return err;
+ }
+
+ vsi_map = (struct virtchnl_dcf_vsi_map *)hw->arq_buf;
+ if (len < sizeof(*vsi_map) || !vsi_map->num_vfs ||
+ len < sizeof(*vsi_map) +
+ (vsi_map->num_vfs - 1) * sizeof(vsi_map->vf_vsi[0])) {
+ PMD_DRV_LOG(ERR, "invalid vf vsi map response with length %u",
+ len);
+ return -EINVAL;
+ }
+
+ if (hw->num_vfs != 0 && hw->num_vfs != vsi_map->num_vfs) {
+ PMD_DRV_LOG(ERR, "The number VSI map (%u) doesn't match the number of VFs (%u)",
+ vsi_map->num_vfs, hw->num_vfs);
+ return -EINVAL;
+ }
+
+ len = vsi_map->num_vfs * sizeof(vsi_map->vf_vsi[0]);
+ if (!hw->vf_vsi_map) {
+ hw->num_vfs = vsi_map->num_vfs;
+ hw->vf_vsi_map = rte_zmalloc("vf_vsi_ctx", len, 0);
+ }
+
+ if (!hw->vf_vsi_map) {
+ PMD_DRV_LOG(ERR, "Fail to alloc memory for VSI context");
+ return -ENOMEM;
+ }
+
+ if (!memcmp(hw->vf_vsi_map, vsi_map->vf_vsi, len)) {
+ PMD_DRV_LOG(DEBUG, "VF VSI map doesn't change");
+ return 1;
+ }
+
+ rte_memcpy(hw->vf_vsi_map, vsi_map->vf_vsi, len);
+ return 0;
+}
+
+static int
+ice_dcf_mode_disable(struct ice_dcf_hw *hw)
+{
+ int err;
+
+ err = ice_dcf_send_cmd_req_no_irq(hw, VIRTCHNL_OP_DCF_DISABLE,
+ NULL, 0);
+ if (err) {
+ PMD_DRV_LOG(ERR, "Fail to send msg OP_DCF_DISABLE");
+ return err;
+ }
+
+ err = ice_dcf_recv_cmd_rsp_no_irq(hw, VIRTCHNL_OP_DCF_DISABLE,
+ (uint8_t *)hw->arq_buf,
+ ICE_DCF_AQ_BUF_SZ, NULL);
+ if (err) {
+ PMD_DRV_LOG(ERR,
+ "Fail to get response of OP_DCF_DISABLE %d",
+ err);
+ return -1;
+ }
+
+ return 0;
+}
+
+static int
+ice_dcf_check_reset_done(struct ice_dcf_hw *hw)
+{
+#define ICE_DCF_RESET_WAIT_CNT 50
+ struct iavf_hw *avf = &hw->avf;
+ int i, reset;
+
+ for (i = 0; i < ICE_DCF_RESET_WAIT_CNT; i++) {
+ reset = IAVF_READ_REG(avf, IAVF_VFGEN_RSTAT) &
+ IAVF_VFGEN_RSTAT_VFR_STATE_MASK;
+ reset = reset >> IAVF_VFGEN_RSTAT_VFR_STATE_SHIFT;
+
+ if (reset == VIRTCHNL_VFR_VFACTIVE ||
+ reset == VIRTCHNL_VFR_COMPLETED)
+ break;
+
+ rte_delay_ms(20);
+ }
+
+ if (i >= ICE_DCF_RESET_WAIT_CNT)
+ return -1;
+
+ return 0;
+}
+
+static inline void
+ice_dcf_enable_irq0(struct ice_dcf_hw *hw)
+{
+ struct iavf_hw *avf = &hw->avf;
+
+ /* Enable admin queue interrupt trigger */
+ IAVF_WRITE_REG(avf, IAVF_VFINT_ICR0_ENA1,
+ IAVF_VFINT_ICR0_ENA1_ADMINQ_MASK);
+ IAVF_WRITE_REG(avf, IAVF_VFINT_DYN_CTL01,
+ IAVF_VFINT_DYN_CTL01_INTENA_MASK |
+ IAVF_VFINT_DYN_CTL01_CLEARPBA_MASK |
+ IAVF_VFINT_DYN_CTL01_ITR_INDX_MASK);
+
+ IAVF_WRITE_FLUSH(avf);
+}
+
+static inline void
+ice_dcf_disable_irq0(struct ice_dcf_hw *hw)
+{
+ struct iavf_hw *avf = &hw->avf;
+
+ /* Disable all interrupt types */
+ IAVF_WRITE_REG(avf, IAVF_VFINT_ICR0_ENA1, 0);
+ IAVF_WRITE_REG(avf, IAVF_VFINT_DYN_CTL01,
+ IAVF_VFINT_DYN_CTL01_ITR_INDX_MASK);
+
+ IAVF_WRITE_FLUSH(avf);
+}
+
+static void
+ice_dcf_dev_interrupt_handler(void *param)
+{
+ struct ice_dcf_hw *hw = param;
+
+ ice_dcf_disable_irq0(hw);
+
+ ice_dcf_handle_virtchnl_msg(hw);
+
+ ice_dcf_enable_irq0(hw);
+}
+
+int
+ice_dcf_execute_virtchnl_cmd(struct ice_dcf_hw *hw,
+ struct dcf_virtchnl_cmd *cmd)
+{
+ int i = 0;
+ int err;
+
+ if ((cmd->req_msg && !cmd->req_msglen) ||
+ (!cmd->req_msg && cmd->req_msglen) ||
+ (cmd->rsp_msgbuf && !cmd->rsp_buflen) ||
+ (!cmd->rsp_msgbuf && cmd->rsp_buflen))
+ return -EINVAL;
+
+ rte_spinlock_lock(&hw->vc_cmd_send_lock);
+ ice_dcf_vc_cmd_set(hw, cmd);
+
+ err = ice_dcf_vc_cmd_send(hw, cmd);
+ if (err) {
+ PMD_DRV_LOG(ERR, "fail to send cmd %d", cmd->v_op);
+ goto ret;
+ }
+
+ do {
+ if (!cmd->pending)
+ break;
+
+ rte_delay_ms(ICE_DCF_ARQ_CHECK_TIME);
+ } while (i++ < ICE_DCF_ARQ_MAX_RETRIES);
+
+ if (cmd->v_ret != IAVF_SUCCESS) {
+ err = -1;
+ PMD_DRV_LOG(ERR,
+ "No response (%d times) or return failure (%d) for cmd %d",
+ i, cmd->v_ret, cmd->v_op);
+ }
+
+ret:
+ ice_dcf_aq_cmd_clear(hw, cmd);
+ rte_spinlock_unlock(&hw->vc_cmd_send_lock);
+ return err;
+}
+
+int
+ice_dcf_send_aq_cmd(void *dcf_hw, struct ice_aq_desc *desc,
+ void *buf, uint16_t buf_size)
+{
+ struct dcf_virtchnl_cmd desc_cmd, buff_cmd;
+ struct ice_dcf_hw *hw = dcf_hw;
+ int err = 0;
+ int i = 0;
+
+ if ((buf && !buf_size) || (!buf && buf_size) ||
+ buf_size > ICE_DCF_AQ_BUF_SZ)
+ return -EINVAL;
+
+ desc_cmd.v_op = VIRTCHNL_OP_DCF_CMD_DESC;
+ desc_cmd.req_msglen = sizeof(*desc);
+ desc_cmd.req_msg = (uint8_t *)desc;
+ desc_cmd.rsp_buflen = sizeof(*desc);
+ desc_cmd.rsp_msgbuf = (uint8_t *)desc;
+
+ if (buf == NULL)
+ return ice_dcf_execute_virtchnl_cmd(hw, &desc_cmd);
+
+ desc->flags |= rte_cpu_to_le_16(ICE_AQ_FLAG_BUF);
+
+ buff_cmd.v_op = VIRTCHNL_OP_DCF_CMD_BUFF;
+ buff_cmd.req_msglen = buf_size;
+ buff_cmd.req_msg = buf;
+ buff_cmd.rsp_buflen = buf_size;
+ buff_cmd.rsp_msgbuf = buf;
+
+ rte_spinlock_lock(&hw->vc_cmd_send_lock);
+ ice_dcf_vc_cmd_set(hw, &desc_cmd);
+ ice_dcf_vc_cmd_set(hw, &buff_cmd);
+
+ if (ice_dcf_vc_cmd_send(hw, &desc_cmd) ||
+ ice_dcf_vc_cmd_send(hw, &buff_cmd)) {
+ err = -1;
+ PMD_DRV_LOG(ERR, "fail to send OP_DCF_CMD_DESC/BUFF");
+ goto ret;
+ }
+
+ do {
+ if ((!desc_cmd.pending && !buff_cmd.pending) ||
+ (!desc_cmd.pending && desc_cmd.v_ret != IAVF_SUCCESS) ||
+ (!buff_cmd.pending && buff_cmd.v_ret != IAVF_SUCCESS))
+ break;
+
+ rte_delay_ms(ICE_DCF_ARQ_CHECK_TIME);
+ } while (i++ < ICE_DCF_ARQ_MAX_RETRIES);
+
+ if (desc_cmd.v_ret != IAVF_SUCCESS || buff_cmd.v_ret != IAVF_SUCCESS) {
+ err = -1;
+ PMD_DRV_LOG(ERR,
+ "No response (%d times) or return failure (desc: %d / buff: %d)",
+ i, desc_cmd.v_ret, buff_cmd.v_ret);
+ }
+
+ret:
+ ice_dcf_aq_cmd_clear(hw, &desc_cmd);
+ ice_dcf_aq_cmd_clear(hw, &buff_cmd);
+ rte_spinlock_unlock(&hw->vc_cmd_send_lock);
+
+ return err;
+}
+
+int
+ice_dcf_handle_vsi_update_event(struct ice_dcf_hw *hw)
+{
+ int err = 0;
+
+ rte_spinlock_lock(&hw->vc_cmd_send_lock);
+ ice_dcf_disable_irq0(hw);
+
+ if (ice_dcf_get_vf_resource(hw) || ice_dcf_get_vf_vsi_map(hw))
+ err = -1;
+
+ ice_dcf_enable_irq0(hw);
+ rte_spinlock_unlock(&hw->vc_cmd_send_lock);
+
+ return err;
+}
+
+int
+ice_dcf_init_hw(struct rte_eth_dev *eth_dev, struct ice_dcf_hw *hw)
+{
+ struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ int ret;
+
+ hw->avf.hw_addr = pci_dev->mem_resource[0].addr;
+ hw->avf.back = hw;
+
+ hw->avf.bus.bus_id = pci_dev->addr.bus;
+ hw->avf.bus.device = pci_dev->addr.devid;
+ hw->avf.bus.func = pci_dev->addr.function;
+
+ hw->avf.device_id = pci_dev->id.device_id;
+ hw->avf.vendor_id = pci_dev->id.vendor_id;
+ hw->avf.subsystem_device_id = pci_dev->id.subsystem_device_id;
+ hw->avf.subsystem_vendor_id = pci_dev->id.subsystem_vendor_id;
+
+ hw->avf.aq.num_arq_entries = ICE_DCF_AQ_LEN;
+ hw->avf.aq.num_asq_entries = ICE_DCF_AQ_LEN;
+ hw->avf.aq.arq_buf_size = ICE_DCF_AQ_BUF_SZ;
+ hw->avf.aq.asq_buf_size = ICE_DCF_AQ_BUF_SZ;
+
+ rte_spinlock_init(&hw->vc_cmd_send_lock);
+ rte_spinlock_init(&hw->vc_cmd_queue_lock);
+ TAILQ_INIT(&hw->vc_cmd_queue);
+
+ hw->arq_buf = rte_zmalloc("arq_buf", ICE_DCF_AQ_BUF_SZ, 0);
+ if (hw->arq_buf == NULL) {
+ PMD_INIT_LOG(ERR, "unable to allocate AdminQ buffer memory");
+ goto err;
+ }
+
+ ret = iavf_set_mac_type(&hw->avf);
+ if (ret) {
+ PMD_INIT_LOG(ERR, "set_mac_type failed: %d", ret);
+ goto err;
+ }
+
+ ret = ice_dcf_check_reset_done(hw);
+ if (ret) {
+ PMD_INIT_LOG(ERR, "VF is still resetting");
+ goto err;
+ }
+
+ ret = iavf_init_adminq(&hw->avf);
+ if (ret) {
+ PMD_INIT_LOG(ERR, "init_adminq failed: %d", ret);
+ goto err;
+ }
+
+ if (ice_dcf_init_check_api_version(hw)) {
+ PMD_INIT_LOG(ERR, "check_api version failed");
+ goto err_api;
+ }
+
+ hw->vf_res = rte_zmalloc("vf_res", ICE_DCF_VF_RES_BUF_SZ, 0);
+ if (hw->vf_res == NULL) {
+ PMD_INIT_LOG(ERR, "unable to allocate vf_res memory");
+ goto err_api;
+ }
+
+ if (ice_dcf_get_vf_resource(hw)) {
+ PMD_INIT_LOG(ERR, "Failed to get VF resource");
+ goto err_alloc;
+ }
+
+ if (ice_dcf_get_vf_vsi_map(hw) < 0) {
+ PMD_INIT_LOG(ERR, "Failed to get VF VSI map");
+ ice_dcf_mode_disable(hw);
+ goto err_alloc;
+ }
+
+ rte_intr_callback_register(&pci_dev->intr_handle,
+ ice_dcf_dev_interrupt_handler, hw);
+ rte_intr_enable(&pci_dev->intr_handle);
+ ice_dcf_enable_irq0(hw);
+
+ return 0;
+
+err_alloc:
+ rte_free(hw->vf_res);
+err_api:
+ iavf_shutdown_adminq(&hw->avf);
+err:
+ rte_free(hw->arq_buf);
+
+ return -1;
+}
+
+void
+ice_dcf_uninit_hw(struct rte_eth_dev *eth_dev, struct ice_dcf_hw *hw)
+{
+ struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_intr_handle *intr_handle = &pci_dev->intr_handle;
+
+ ice_dcf_disable_irq0(hw);
+ rte_intr_disable(intr_handle);
+ rte_intr_callback_unregister(intr_handle,
+ ice_dcf_dev_interrupt_handler, hw);
+
+ ice_dcf_mode_disable(hw);
+ iavf_shutdown_adminq(&hw->avf);
+
+ rte_free(hw->arq_buf);
+ rte_free(hw->vf_vsi_map);
+ rte_free(hw->vf_res);
+}
diff --git a/drivers/net/ice/ice_dcf.h b/drivers/net/ice/ice_dcf.h
new file mode 100644
index 000000000..58647d87f
--- /dev/null
+++ b/drivers/net/ice/ice_dcf.h
@@ -0,0 +1,61 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+
+#ifndef _ICE_DCF_H_
+#define _ICE_DCF_H_
+
+#include <rte_ethdev_driver.h>
+
+#include <iavf_prototype.h>
+#include <iavf_adminq_cmd.h>
+#include <iavf_type.h>
+
+#include "base/ice_type.h"
+#include "ice_logs.h"
+
+struct dcf_virtchnl_cmd {
+ TAILQ_ENTRY(dcf_virtchnl_cmd) next;
+
+ enum virtchnl_ops v_op;
+ enum iavf_status v_ret;
+
+ uint16_t req_msglen;
+ uint8_t *req_msg;
+
+ uint16_t rsp_msglen;
+ uint16_t rsp_buflen;
+ uint8_t *rsp_msgbuf;
+
+ volatile int pending;
+};
+
+struct ice_dcf_hw {
+ struct iavf_hw avf;
+
+ rte_spinlock_t vc_cmd_send_lock;
+ rte_spinlock_t vc_cmd_queue_lock;
+ TAILQ_HEAD(, dcf_virtchnl_cmd) vc_cmd_queue;
+ void (*vc_event_msg_cb)(struct ice_dcf_hw *dcf_hw,
+ uint8_t *msg, uint16_t msglen);
+
+ uint8_t *arq_buf;
+
+ uint16_t num_vfs;
+ uint16_t *vf_vsi_map;
+
+ struct virtchnl_version_info virtchnl_version;
+ struct virtchnl_vf_resource *vf_res; /* VF resource */
+ struct virtchnl_vsi_resource *vsi_res; /* LAN VSI */
+ uint16_t vsi_id;
+};
+
+int ice_dcf_execute_virtchnl_cmd(struct ice_dcf_hw *hw,
+ struct dcf_virtchnl_cmd *cmd);
+int ice_dcf_send_aq_cmd(void *dcf_hw, struct ice_aq_desc *desc,
+ void *buf, uint16_t buf_size);
+int ice_dcf_handle_vsi_update_event(struct ice_dcf_hw *hw);
+int ice_dcf_init_hw(struct rte_eth_dev *eth_dev, struct ice_dcf_hw *hw);
+void ice_dcf_uninit_hw(struct rte_eth_dev *eth_dev, struct ice_dcf_hw *hw);
+
+#endif /* _ICE_DCF_H_ */
diff --git a/drivers/net/ice/ice_dcf_ethdev.c b/drivers/net/ice/ice_dcf_ethdev.c
new file mode 100644
index 000000000..f65b962d4
--- /dev/null
+++ b/drivers/net/ice/ice_dcf_ethdev.c
@@ -0,0 +1,319 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+
+#include <errno.h>
+#include <stdbool.h>
+#include <sys/types.h>
+#include <sys/ioctl.h>
+#include <unistd.h>
+
+#include <rte_interrupts.h>
+#include <rte_debug.h>
+#include <rte_pci.h>
+#include <rte_atomic.h>
+#include <rte_eal.h>
+#include <rte_ether.h>
+#include <rte_ethdev_pci.h>
+#include <rte_kvargs.h>
+#include <rte_malloc.h>
+#include <rte_memzone.h>
+#include <rte_dev.h>
+
+#include <iavf_devids.h>
+
+#include "ice_generic_flow.h"
+#include "ice_dcf_ethdev.h"
+
+static uint16_t
+ice_dcf_recv_pkts(__rte_unused void *rx_queue,
+ __rte_unused struct rte_mbuf **bufs,
+ __rte_unused uint16_t nb_pkts)
+{
+ return 0;
+}
+
+static uint16_t
+ice_dcf_xmit_pkts(__rte_unused void *tx_queue,
+ __rte_unused struct rte_mbuf **bufs,
+ __rte_unused uint16_t nb_pkts)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_start(struct rte_eth_dev *dev)
+{
+ dev->data->dev_link.link_status = ETH_LINK_UP;
+
+ return 0;
+}
+
+static void
+ice_dcf_dev_stop(struct rte_eth_dev *dev)
+{
+ dev->data->dev_link.link_status = ETH_LINK_DOWN;
+}
+
+static int
+ice_dcf_dev_configure(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_info_get(struct rte_eth_dev *dev,
+ struct rte_eth_dev_info *dev_info)
+{
+ struct ice_dcf_adapter *adapter = dev->data->dev_private;
+
+ dev_info->max_mac_addrs = 1;
+ dev_info->max_rx_pktlen = (uint32_t)-1;
+ dev_info->max_rx_queues = RTE_DIM(adapter->rxqs);
+ dev_info->max_tx_queues = RTE_DIM(adapter->txqs);
+
+ return 0;
+}
+
+static int
+ice_dcf_stats_get(__rte_unused struct rte_eth_dev *dev,
+ __rte_unused struct rte_eth_stats *igb_stats)
+{
+ return 0;
+}
+
+static int
+ice_dcf_stats_reset(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_promiscuous_enable(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_promiscuous_disable(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_allmulticast_enable(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_allmulticast_disable(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_filter_ctrl(struct rte_eth_dev *dev,
+ enum rte_filter_type filter_type,
+ __rte_unused enum rte_filter_op filter_op,
+ __rte_unused void *arg)
+{
+ int ret = 0;
+
+ if (!dev)
+ return -EINVAL;
+
+ switch (filter_type) {
+ default:
+ PMD_DRV_LOG(WARNING, "Filter type (%d) not supported",
+ filter_type);
+ ret = -EINVAL;
+ break;
+ }
+
+ return ret;
+}
+
+static void
+ice_dcf_dev_close(struct rte_eth_dev *dev)
+{
+ struct ice_dcf_adapter *adapter = dev->data->dev_private;
+
+ if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+ return;
+
+ dev->dev_ops = NULL;
+ dev->rx_pkt_burst = NULL;
+ dev->tx_pkt_burst = NULL;
+
+ ice_dcf_uninit_parent_adapter(dev);
+ ice_dcf_uninit_hw(dev, &adapter->real_hw);
+}
+
+static void
+ice_dcf_queue_release(__rte_unused void *q)
+{
+}
+
+static int
+ice_dcf_link_update(__rte_unused struct rte_eth_dev *dev,
+ __rte_unused int wait_to_complete)
+{
+ return 0;
+}
+
+static int
+ice_dcf_rx_queue_setup(struct rte_eth_dev *dev,
+ uint16_t rx_queue_id,
+ __rte_unused uint16_t nb_rx_desc,
+ __rte_unused unsigned int socket_id,
+ __rte_unused const struct rte_eth_rxconf *rx_conf,
+ __rte_unused struct rte_mempool *mb_pool)
+{
+ struct ice_dcf_adapter *adapter = dev->data->dev_private;
+
+ dev->data->rx_queues[rx_queue_id] = &adapter->rxqs[rx_queue_id];
+
+ return 0;
+}
+
+static int
+ice_dcf_tx_queue_setup(struct rte_eth_dev *dev,
+ uint16_t tx_queue_id,
+ __rte_unused uint16_t nb_tx_desc,
+ __rte_unused unsigned int socket_id,
+ __rte_unused const struct rte_eth_txconf *tx_conf)
+{
+ struct ice_dcf_adapter *adapter = dev->data->dev_private;
+
+ dev->data->tx_queues[tx_queue_id] = &adapter->txqs[tx_queue_id];
+
+ return 0;
+}
+
+static const struct eth_dev_ops ice_dcf_eth_dev_ops = {
+ .dev_start = ice_dcf_dev_start,
+ .dev_stop = ice_dcf_dev_stop,
+ .dev_close = ice_dcf_dev_close,
+ .dev_configure = ice_dcf_dev_configure,
+ .dev_infos_get = ice_dcf_dev_info_get,
+ .rx_queue_setup = ice_dcf_rx_queue_setup,
+ .tx_queue_setup = ice_dcf_tx_queue_setup,
+ .rx_queue_release = ice_dcf_queue_release,
+ .tx_queue_release = ice_dcf_queue_release,
+ .link_update = ice_dcf_link_update,
+ .stats_get = ice_dcf_stats_get,
+ .stats_reset = ice_dcf_stats_reset,
+ .promiscuous_enable = ice_dcf_dev_promiscuous_enable,
+ .promiscuous_disable = ice_dcf_dev_promiscuous_disable,
+ .allmulticast_enable = ice_dcf_dev_allmulticast_enable,
+ .allmulticast_disable = ice_dcf_dev_allmulticast_disable,
+ .filter_ctrl = ice_dcf_dev_filter_ctrl,
+};
+
+static int
+ice_dcf_dev_init(struct rte_eth_dev *eth_dev)
+{
+ struct ice_dcf_adapter *adapter = eth_dev->data->dev_private;
+
+ eth_dev->dev_ops = &ice_dcf_eth_dev_ops;
+ eth_dev->rx_pkt_burst = ice_dcf_recv_pkts;
+ eth_dev->tx_pkt_burst = ice_dcf_xmit_pkts;
+
+ if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+ return 0;
+
+ eth_dev->data->dev_flags |= RTE_ETH_DEV_CLOSE_REMOVE;
+
+ adapter->real_hw.vc_event_msg_cb = ice_dcf_handle_pf_event_msg;
+ if (ice_dcf_init_hw(eth_dev, &adapter->real_hw) != 0) {
+ PMD_DRV_LOG(ERR, "Failed to init DCF hardware");
+ return -1;
+ }
+
+ if (ice_dcf_init_parent_adapter(eth_dev) != 0) {
+ PMD_DRV_LOG(ERR, "Failed to init DCF parent adapter");
+ ice_dcf_uninit_hw(eth_dev, &adapter->real_hw);
+ return -1;
+ }
+
+ return 0;
+}
+
+static int
+ice_dcf_dev_uninit(struct rte_eth_dev *eth_dev)
+{
+ ice_dcf_dev_close(eth_dev);
+
+ return 0;
+}
+
+static int
+handle_dcf_arg(__rte_unused const char *key, const char *value,
+ __rte_unused void *arg)
+{
+ bool *dcf = arg;
+
+ if (arg == NULL || value == NULL)
+ return -EINVAL;
+
+ if (strcmp(value, "dcf") == 0)
+ *dcf = true;
+ else
+ *dcf = false;
+
+ return 0;
+}
+
+static bool
+check_cap_dcf_enable(struct rte_devargs *devargs)
+{
+ struct rte_kvargs *kvlist;
+ bool enable = false;
+
+ if (devargs == NULL)
+ return false;
+
+ kvlist = rte_kvargs_parse(devargs->args, NULL);
+ if (kvlist == NULL)
+ return false;
+
+ rte_kvargs_process(kvlist, "cap", handle_dcf_arg, &enable);
+
+ rte_kvargs_free(kvlist);
+
+ return enable;
+}
+
+static int eth_ice_dcf_pci_probe(__rte_unused struct rte_pci_driver *pci_drv,
+ struct rte_pci_device *pci_dev)
+{
+ if (!check_cap_dcf_enable(pci_dev->device.devargs))
+ return 1; /* continue to probe */
+
+ return rte_eth_dev_pci_generic_probe(pci_dev,
+ sizeof(struct ice_dcf_adapter),
+ ice_dcf_dev_init);
+}
+
+static int eth_ice_dcf_pci_remove(struct rte_pci_device *pci_dev)
+{
+ return rte_eth_dev_pci_generic_remove(pci_dev, ice_dcf_dev_uninit);
+}
+
+static const struct rte_pci_id pci_id_ice_dcf_map[] = {
+ { RTE_PCI_DEVICE(IAVF_INTEL_VENDOR_ID, IAVF_DEV_ID_ADAPTIVE_VF) },
+ { .vendor_id = 0, /* sentinel */ },
+};
+
+static struct rte_pci_driver rte_ice_dcf_pmd = {
+ .id_table = pci_id_ice_dcf_map,
+ .drv_flags = RTE_PCI_DRV_NEED_MAPPING,
+ .probe = eth_ice_dcf_pci_probe,
+ .remove = eth_ice_dcf_pci_remove,
+};
+
+RTE_PMD_REGISTER_PCI(net_ice_dcf, rte_ice_dcf_pmd);
+RTE_PMD_REGISTER_PCI_TABLE(net_ice_dcf, pci_id_ice_dcf_map);
+RTE_PMD_REGISTER_KMOD_DEP(net_ice_dcf, "* igb_uio | vfio-pci");
+RTE_PMD_REGISTER_PARAM_STRING(net_ice_dcf, "cap=dcf");
diff --git a/drivers/net/ice/ice_dcf_ethdev.h b/drivers/net/ice/ice_dcf_ethdev.h
new file mode 100644
index 000000000..e95266599
--- /dev/null
+++ b/drivers/net/ice/ice_dcf_ethdev.h
@@ -0,0 +1,33 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+
+#ifndef _ICE_DCF_ETHDEV_H_
+#define _ICE_DCF_ETHDEV_H_
+
+#include "base/ice_common.h"
+#include "base/ice_adminq_cmd.h"
+
+#include "ice_ethdev.h"
+#include "ice_dcf.h"
+
+#define ICE_DCF_MAX_RINGS 1
+
+struct ice_dcf_queue {
+ uint64_t dummy;
+};
+
+struct ice_dcf_adapter {
+ struct ice_adapter parent; /* Must be first */
+
+ struct ice_dcf_hw real_hw;
+ struct ice_dcf_queue rxqs[ICE_DCF_MAX_RINGS];
+ struct ice_dcf_queue txqs[ICE_DCF_MAX_RINGS];
+};
+
+void ice_dcf_handle_pf_event_msg(struct ice_dcf_hw *dcf_hw,
+ uint8_t *msg, uint16_t msglen);
+int ice_dcf_init_parent_adapter(struct rte_eth_dev *eth_dev);
+void ice_dcf_uninit_parent_adapter(struct rte_eth_dev *eth_dev);
+
+#endif /* _ICE_DCF_ETHDEV_H_ */
diff --git a/drivers/net/ice/ice_dcf_parent.c b/drivers/net/ice/ice_dcf_parent.c
new file mode 100644
index 000000000..bca9cd34a
--- /dev/null
+++ b/drivers/net/ice/ice_dcf_parent.c
@@ -0,0 +1,348 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <unistd.h>
+
+#include <rte_alarm.h>
+
+#include "ice_dcf_ethdev.h"
+#include "ice_generic_flow.h"
+
+#define ICE_DCF_VSI_UPDATE_SERVICE_INTERVAL 100000 /* us */
+
+static __rte_always_inline void
+ice_dcf_update_vsi_ctx(struct ice_hw *hw, uint16_t vsi_handle,
+ uint16_t vsi_map)
+{
+ struct ice_vsi_ctx *vsi_ctx;
+
+ if (unlikely(vsi_handle >= ICE_MAX_VSI)) {
+ PMD_DRV_LOG(ERR, "Invalid vsi handle %u", vsi_handle);
+ return;
+ }
+
+ vsi_ctx = hw->vsi_ctx[vsi_handle];
+
+ if (vsi_map & VIRTCHNL_DCF_VF_VSI_VALID) {
+ if (!vsi_ctx)
+ vsi_ctx = ice_malloc(hw, sizeof(*vsi_ctx));
+
+ if (!vsi_ctx) {
+ PMD_DRV_LOG(ERR, "No memory for vsi context %u",
+ vsi_handle);
+ return;
+ }
+
+ vsi_ctx->vsi_num = (vsi_map & VIRTCHNL_DCF_VF_VSI_ID_M) >>
+ VIRTCHNL_DCF_VF_VSI_ID_S;
+ hw->vsi_ctx[vsi_handle] = vsi_ctx;
+
+ PMD_DRV_LOG(DEBUG, "VF%u is assigned with vsi number %u",
+ vsi_handle, vsi_ctx->vsi_num);
+ } else {
+ hw->vsi_ctx[vsi_handle] = NULL;
+
+ if (vsi_ctx)
+ ice_free(hw, vsi_ctx);
+
+ PMD_DRV_LOG(NOTICE, "VF%u is disabled", vsi_handle);
+ }
+}
+
+static void
+ice_dcf_update_vf_vsi_map(struct ice_hw *hw, uint16_t num_vfs,
+ uint16_t *vf_vsi_map)
+{
+ uint16_t vf_id;
+
+ for (vf_id = 0; vf_id < num_vfs; vf_id++)
+ ice_dcf_update_vsi_ctx(hw, vf_id, vf_vsi_map[vf_id]);
+}
+
+static void
+ice_dcf_vsi_update_service_handler(void *param)
+{
+ struct ice_dcf_hw *hw = param;
+
+ if (!ice_dcf_handle_vsi_update_event(hw)) {
+ struct ice_dcf_adapter *dcf_ad =
+ container_of(hw, struct ice_dcf_adapter, real_hw);
+
+ ice_dcf_update_vf_vsi_map(&dcf_ad->parent.hw,
+ hw->num_vfs, hw->vf_vsi_map);
+ }
+}
+
+void
+ice_dcf_handle_pf_event_msg(struct ice_dcf_hw *dcf_hw,
+ uint8_t *msg, uint16_t msglen)
+{
+ struct virtchnl_pf_event *pf_msg = (struct virtchnl_pf_event *)msg;
+
+ if (msglen < sizeof(struct virtchnl_pf_event)) {
+ PMD_DRV_LOG(DEBUG, "Invalid event message length : %u", msglen);
+ return;
+ }
+
+ switch (pf_msg->event) {
+ case VIRTCHNL_EVENT_RESET_IMPENDING:
+ PMD_DRV_LOG(DEBUG, "VIRTCHNL_EVENT_RESET_IMPENDING event");
+ rte_eal_alarm_set(ICE_DCF_VSI_UPDATE_SERVICE_INTERVAL * 2,
+ ice_dcf_vsi_update_service_handler, dcf_hw);
+ break;
+ case VIRTCHNL_EVENT_LINK_CHANGE:
+ PMD_DRV_LOG(DEBUG, "VIRTCHNL_EVENT_LINK_CHANGE event");
+ break;
+ case VIRTCHNL_EVENT_PF_DRIVER_CLOSE:
+ PMD_DRV_LOG(DEBUG, "VIRTCHNL_EVENT_PF_DRIVER_CLOSE event");
+ break;
+ case VIRTCHNL_EVENT_DCF_VSI_MAP_UPDATE:
+ PMD_DRV_LOG(DEBUG, "VIRTCHNL_EVENT_DCF_VSI_MAP_UPDATE event : VF%u with VSI num %u",
+ pf_msg->event_data.vf_vsi_map.vf_id,
+ pf_msg->event_data.vf_vsi_map.vsi_id);
+ rte_eal_alarm_set(ICE_DCF_VSI_UPDATE_SERVICE_INTERVAL,
+ ice_dcf_vsi_update_service_handler, dcf_hw);
+ break;
+ default:
+ PMD_DRV_LOG(ERR, "Unknown event received %u", pf_msg->event);
+ break;
+ }
+}
+
+static int
+ice_dcf_init_parent_hw(struct ice_hw *hw)
+{
+ struct ice_aqc_get_phy_caps_data *pcaps;
+ enum ice_status status;
+
+ status = ice_aq_get_fw_ver(hw, NULL);
+ if (status)
+ return status;
+
+ status = ice_get_caps(hw);
+ if (status)
+ return status;
+
+ hw->port_info = (struct ice_port_info *)
+ ice_malloc(hw, sizeof(*hw->port_info));
+ if (!hw->port_info)
+ return ICE_ERR_NO_MEMORY;
+
+ /* set the back pointer to HW */
+ hw->port_info->hw = hw;
+
+ /* Initialize port_info struct with switch configuration data */
+ status = ice_get_initial_sw_cfg(hw);
+ if (status)
+ goto err_unroll_alloc;
+
+ pcaps = (struct ice_aqc_get_phy_caps_data *)
+ ice_malloc(hw, sizeof(*pcaps));
+ if (!pcaps) {
+ status = ICE_ERR_NO_MEMORY;
+ goto err_unroll_alloc;
+ }
+
+ /* Initialize port_info struct with PHY capabilities */
+ status = ice_aq_get_phy_caps(hw->port_info, false,
+ ICE_AQC_REPORT_TOPO_CAP, pcaps, NULL);
+ ice_free(hw, pcaps);
+ if (status)
+ goto err_unroll_alloc;
+
+ /* Initialize port_info struct with link information */
+ status = ice_aq_get_link_info(hw->port_info, false, NULL, NULL);
+ if (status)
+ goto err_unroll_alloc;
+
+ status = ice_init_fltr_mgmt_struct(hw);
+ if (status)
+ goto err_unroll_alloc;
+
+ status = ice_init_hw_tbls(hw);
+ if (status)
+ goto err_unroll_fltr_mgmt_struct;
+
+ PMD_DRV_LOG(INFO,
+ "firmware %d.%d.%d api %d.%d.%d build 0x%08x",
+ hw->fw_maj_ver, hw->fw_min_ver, hw->fw_patch,
+ hw->api_maj_ver, hw->api_min_ver, hw->api_patch,
+ hw->fw_build);
+
+ return ICE_SUCCESS;
+
+err_unroll_fltr_mgmt_struct:
+ ice_cleanup_fltr_mgmt_struct(hw);
+err_unroll_alloc:
+ ice_free(hw, hw->port_info);
+ hw->port_info = NULL;
+
+ return status;
+}
+
+static void ice_dcf_uninit_parent_hw(struct ice_hw *hw)
+{
+ ice_cleanup_fltr_mgmt_struct(hw);
+
+ ice_free_seg(hw);
+ ice_free_hw_tbls(hw);
+
+ if (hw->port_info) {
+ ice_free(hw, hw->port_info);
+ hw->port_info = NULL;
+ }
+
+ ice_clear_all_vsi_ctx(hw);
+}
+
+static int
+ice_dcf_request_pkg_name(struct ice_hw *hw, char *pkg_name)
+{
+ struct ice_dcf_adapter *dcf_adapter =
+ container_of(hw, struct ice_dcf_adapter, parent.hw);
+
+ /* TODO: check with DSN firstly by iAVF */
+ PMD_DRV_LOG(DEBUG,
+ "DCF VSI_ID = %u",
+ dcf_adapter->real_hw.vsi_id);
+
+ snprintf(pkg_name,
+ ICE_MAX_PKG_FILENAME_SIZE, "%s", ICE_PKG_FILE_UPDATES);
+ if (!access(pkg_name, 0))
+ return 0;
+
+ snprintf(pkg_name,
+ ICE_MAX_PKG_FILENAME_SIZE, "%s", ICE_PKG_FILE_DEFAULT);
+ if (!access(pkg_name, 0))
+ return 0;
+
+ return -1;
+}
+
+static int
+ice_dcf_load_pkg(struct ice_hw *hw)
+{
+ char pkg_name[ICE_MAX_PKG_FILENAME_SIZE];
+ uint8_t *pkg_buf;
+ uint32_t buf_len;
+ struct stat st;
+ FILE *fp;
+ int err;
+
+ if (ice_dcf_request_pkg_name(hw, pkg_name)) {
+ PMD_INIT_LOG(ERR, "failed to locate the package file");
+ return -ENOENT;
+ }
+
+ PMD_INIT_LOG(DEBUG, "DDP package name: %s", pkg_name);
+
+ err = stat(pkg_name, &st);
+ if (err) {
+ PMD_INIT_LOG(ERR, "failed to get file status");
+ return err;
+ }
+
+ buf_len = st.st_size;
+ pkg_buf = rte_malloc(NULL, buf_len, 0);
+ if (!pkg_buf) {
+ PMD_INIT_LOG(ERR, "failed to allocate buffer of size %u for package",
+ buf_len);
+ return -1;
+ }
+
+ fp = fopen(pkg_name, "rb");
+ if (!fp) {
+ PMD_INIT_LOG(ERR, "failed to open file: %s", pkg_name);
+ err = -1;
+ goto ret;
+ }
+
+ err = fread(pkg_buf, buf_len, 1, fp);
+ fclose(fp);
+ if (err != 1) {
+ PMD_INIT_LOG(ERR, "failed to read package data");
+ err = -1;
+ goto ret;
+ }
+
+ err = ice_copy_and_init_pkg(hw, pkg_buf, buf_len);
+ if (err)
+ PMD_INIT_LOG(ERR, "ice_copy_and_init_hw failed: %d", err);
+
+ret:
+ rte_free(pkg_buf);
+ return err;
+}
+
+int
+ice_dcf_init_parent_adapter(struct rte_eth_dev *eth_dev)
+{
+ struct ice_dcf_adapter *adapter = eth_dev->data->dev_private;
+ struct ice_adapter *parent_adapter = &adapter->parent;
+ struct ice_hw *parent_hw = &parent_adapter->hw;
+ struct ice_dcf_hw *hw = &adapter->real_hw;
+ const struct rte_ether_addr *mac;
+ int err;
+
+ parent_adapter->eth_dev = eth_dev;
+ parent_adapter->pf.adapter = parent_adapter;
+ parent_adapter->pf.dev_data = eth_dev->data;
+ parent_hw->back = parent_adapter;
+ parent_hw->mac_type = ICE_MAC_GENERIC;
+ parent_hw->vendor_id = ICE_INTEL_VENDOR_ID;
+
+ ice_init_lock(&parent_hw->adminq.sq_lock);
+ ice_init_lock(&parent_hw->adminq.rq_lock);
+ parent_hw->aq_send_cmd_fn = ice_dcf_send_aq_cmd;
+ parent_hw->aq_send_cmd_param = &adapter->real_hw;
+ parent_hw->dcf_enabled = true;
+
+ err = ice_dcf_init_parent_hw(parent_hw);
+ if (err) {
+ PMD_DRV_LOG(ERR, "failed to init the DCF parent hardware with error %d",
+ err);
+ return err;
+ }
+
+ err = ice_dcf_load_pkg(parent_hw);
+ if (err) {
+ PMD_DRV_LOG(ERR, "failed to load package with error %d",
+ err);
+ goto uninit_hw;
+ }
+ parent_adapter->active_pkg_type = ice_load_pkg_type(parent_hw);
+
+ ice_dcf_update_vf_vsi_map(parent_hw,
+ hw->num_vfs, hw->vf_vsi_map);
+
+ mac = (const struct rte_ether_addr *)hw->avf.mac.addr;
+ if (rte_is_valid_assigned_ether_addr(mac))
+ rte_ether_addr_copy(mac, &parent_adapter->pf.dev_addr);
+ else
+ rte_eth_random_addr(parent_adapter->pf.dev_addr.addr_bytes);
+
+ eth_dev->data->mac_addrs = &parent_adapter->pf.dev_addr;
+
+ return 0;
+
+uninit_hw:
+ ice_dcf_uninit_parent_hw(parent_hw);
+ return err;
+}
+
+void
+ice_dcf_uninit_parent_adapter(struct rte_eth_dev *eth_dev)
+{
+ struct ice_dcf_adapter *adapter = eth_dev->data->dev_private;
+ struct ice_adapter *parent_adapter = &adapter->parent;
+ struct ice_hw *parent_hw = &parent_adapter->hw;
+
+ eth_dev->data->mac_addrs = NULL;
+
+ rte_eal_alarm_cancel(ice_dcf_vsi_update_service_handler,
+ &adapter->real_hw);
+
+ ice_dcf_uninit_parent_hw(parent_hw);
+}
diff --git a/drivers/net/ice/meson.build b/drivers/net/ice/meson.build
index f9e897bbc..0df8ddc0f 100644
--- a/drivers/net/ice/meson.build
+++ b/drivers/net/ice/meson.build
@@ -15,8 +15,8 @@ sources = files(
'ice_hash.c'
)
-deps += ['hash']
-includes += include_directories('base')
+deps += ['hash', 'net', 'common_iavf']
+includes += include_directories('base', '../../common/iavf')
if arch_subdir == 'x86'
sources += files('ice_rxtx_vec_sse.c')
@@ -37,4 +37,8 @@ if arch_subdir == 'x86'
endif
endif
+sources += files('ice_dcf.c',
+ 'ice_dcf_parent.c',
+ 'ice_dcf_ethdev.c')
+
install_headers('rte_pmd_ice.h')
diff --git a/mk/rte.app.mk b/mk/rte.app.mk
index d295ca0a5..f3798a09f 100644
--- a/mk/rte.app.mk
+++ b/mk/rte.app.mk
@@ -185,6 +185,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_I40E_PMD) += -lrte_pmd_i40e
_LDLIBS-$(CONFIG_RTE_LIBRTE_IAVF_PMD) += -lrte_pmd_iavf
_LDLIBS-$(CONFIG_RTE_LIBRTE_ICE_PMD) += -lrte_pmd_ice
IAVF-y := $(CONFIG_RTE_LIBRTE_IAVF_PMD)
+IAVF-y += $(CONFIG_RTE_LIBRTE_ICE_PMD)
ifeq ($(findstring y,$(IAVF-y)),y)
_LDLIBS-y += -lrte_common_iavf
endif
--
2.25.1
^ permalink raw reply [relevance 1%]
* Re: [dpdk-dev] [PATCH dpdk-dev v3] mempool: sort the rte_mempool_ops by name
2020-03-09 8:55 0% ` Tonghao Zhang
2020-03-09 9:05 0% ` Olivier Matz
@ 2020-03-09 13:15 0% ` David Marchand
2020-03-16 7:43 0% ` Tonghao Zhang
1 sibling, 1 reply; 200+ results
From: David Marchand @ 2020-03-09 13:15 UTC (permalink / raw)
To: Tonghao Zhang
Cc: Olivier Matz, Andrew Rybchenko, Jerin Jacob, dpdk-dev, Gage Eads,
Artem V. Andreev, Jerin Jacob, Nithin Dabilpuram, Vamsi Attunuru,
Hemant Agrawal, Burakov, Anatoly, Bruce Richardson
On Mon, Mar 9, 2020 at 9:56 AM Tonghao Zhang <xiangxia.m.yue@gmail.com> wrote:
> On Mon, Mar 9, 2020 at 4:27 PM Olivier Matz <olivier.matz@6wind.com> wrote:
> > The fact that the ops index changes during mempool driver lifetime is
> > indeed frightening, especially knowning that this is a dynamic
> > registration that could happen at any moment in the life of the
> > application. Also, breaking the ABI is not desirable.
> That solution is better.
>
> > Let me try to propose something else to solve your issue:
> >
> > 1/ At init, the primary process allocates a struct in shared memory
> > (named memzone):
> >
> > struct rte_mempool_shared_ops {
> > size_t num_mempool_ops;
> > struct {
> > char name[RTE_MEMPOOL_OPS_NAMESIZE];
> > } mempool_ops[RTE_MEMPOOL_MAX_OPS_IDX];
> > char *mempool_ops_name[RTE_MEMPOOL_MAX_OPS_IDX];
> > rte_spinlock_t mempool;
> > }
> >
> > 2/ When we register a mempool ops, we first get a name and id from the
> > shared struct: with the lock held, lookup for the registered name and
> > return its index, else get the last id and copy the name in the struct.
> >
> > 3/ Then do as before (in the per-process global table), except that we
> > reuse the registered id.
> >
> > We can remove the num_ops field from rte_mempool_ops_table.
> >
> > Thoughts?
It seems better, just adding Anatoly and Bruce who know more about multiprocess.
Tonghao, could you add a unit test to exhibit the issue as part of this work?
Thanks.
--
David Marchand
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH dpdk-dev v3] mempool: sort the rte_mempool_ops by name
2020-03-09 8:55 0% ` Tonghao Zhang
@ 2020-03-09 9:05 0% ` Olivier Matz
2020-03-09 13:15 0% ` David Marchand
1 sibling, 0 replies; 200+ results
From: Olivier Matz @ 2020-03-09 9:05 UTC (permalink / raw)
To: Tonghao Zhang
Cc: Andrew Rybchenko, Jerin Jacob, dpdk-dev, Gage Eads,
Artem V. Andreev, Jerin Jacob, Nithin Dabilpuram, Vamsi Attunuru,
Hemant Agrawal
On Mon, Mar 09, 2020 at 04:55:28PM +0800, Tonghao Zhang wrote:
> On Mon, Mar 9, 2020 at 4:27 PM Olivier Matz <olivier.matz@6wind.com> wrote:
> >
> > Hi,
> >
> > On Mon, Mar 09, 2020 at 11:01:25AM +0800, Tonghao Zhang wrote:
> > > On Sat, Mar 7, 2020 at 8:54 PM Andrew Rybchenko
> > > <arybchenko@solarflare.com> wrote:
> > > >
> > > > On 3/7/20 3:51 PM, Andrew Rybchenko wrote:
> > > > > On 3/6/20 4:37 PM, Jerin Jacob wrote:
> > > > >> On Fri, Mar 6, 2020 at 7:06 PM <xiangxia.m.yue@gmail.com> wrote:
> > > > >>> From: Tonghao Zhang <xiangxia.m.yue@gmail.com>
> > > > >>>
> > > > >>> The order of mempool initiation affects mempool index in the
> > > > >>> rte_mempool_ops_table. For example, when building APPs with:
> > > > >>>
> > > > >>> $ gcc -lrte_mempool_bucket -lrte_mempool_ring ...
> > > > >>>
> > > > >>> The "bucket" mempool will be registered firstly, and its index
> > > > >>> in table is 0 while the index of "ring" mempool is 1. DPDK
> > > > >>> uses the mk/rte.app.mk to build APPs, and others, for example,
> > > > >>> Open vSwitch, use the libdpdk.a or libdpdk.so to build it.
> > > > >>> The mempool lib linked in dpdk and Open vSwitch is different.
> > > > >>>
> > > > >>> The mempool can be used between primary and secondary process,
> > > > >>> such as dpdk-pdump and pdump-pmd/Open vSwitch(pdump enabled).
> > > > >>> There will be a crash because dpdk-pdump creates the "ring_mp_mc"
> > > > >>> ring which index in table is 0, but the index of "bucket" ring
> > > > >>> is 0 in Open vSwitch. If Open vSwitch use the index 0 to get
> > > > >>> mempool ops and malloc memory from mempool. The crash will occur:
> > > > >>>
> > > > >>> bucket_dequeue (access null and crash)
> > > > >>> rte_mempool_get_ops (should get "ring_mp_mc",
> > > > >>> but get "bucket" mempool)
> > > > >>> rte_mempool_ops_dequeue_bulk
> > > > >>> ...
> > > > >>> rte_pktmbuf_alloc
> > > > >>> rte_pktmbuf_copy
> > > > >>> pdump_copy
> > > > >>> pdump_rx
> > > > >>> rte_eth_rx_burst
> > > > >>>
> > > > >>> To avoid the crash, there are some solution:
> > > > >>> * constructor priority: Different mempool uses different
> > > > >>> priority in RTE_INIT, but it's not easy to maintain.
> > > > >>>
> > > > >>> * change mk/rte.app.mk: Change the order in mk/rte.app.mk to
> > > > >>> be same as libdpdk.a/libdpdk.so, but when adding a new mempool
> > > > >>> driver in future, we must make sure the order.
> > > > >>>
> > > > >>> * register mempool orderly: Sort the mempool when registering,
> > > > >>> so the lib linked will not affect the index in mempool table.
> > > > >>>
> > > > >>> Signed-off-by: Tonghao Zhang <xiangxia.m.yue@gmail.com>
> > > > >>> Acked-by: Olivier Matz <olivier.matz@6wind.com>
> > > > >> Acked-by: Jerin Jacob <jerinj@marvell.com>
> > > > >
> > > > > The patch is OK, but the fact that ops index changes during
> > > > > mempool driver lifetime is frightening. In fact it breaks
> > > > > rte_mempool_register_ops() return value semantics (read
> > > > > as API break). The return value is not used in DPDK, but it
> > > > > is a public function. If I'm not mistaken it should be taken
> > > > > into account.
> >
> > Good points.
> >
> > The fact that the ops index changes during mempool driver lifetime is
> > indeed frightening, especially knowning that this is a dynamic
> > registration that could happen at any moment in the life of the
> > application. Also, breaking the ABI is not desirable.
> That solution is better.
>
> > Let me try to propose something else to solve your issue:
> >
> > 1/ At init, the primary process allocates a struct in shared memory
> > (named memzone):
> >
> > struct rte_mempool_shared_ops {
> > size_t num_mempool_ops;
> > struct {
> > char name[RTE_MEMPOOL_OPS_NAMESIZE];
> > } mempool_ops[RTE_MEMPOOL_MAX_OPS_IDX];
> > char *mempool_ops_name[RTE_MEMPOOL_MAX_OPS_IDX];
oops I forgot to remove this line (replaced by mini-struct just above).
> > rte_spinlock_t mempool;
> > }
> >
> > 2/ When we register a mempool ops, we first get a name and id from the
> > shared struct: with the lock held, lookup for the registered name and
> > return its index, else get the last id and copy the name in the struct.
> >
> > 3/ Then do as before (in the per-process global table), except that we
> > reuse the registered id.
> >
> > We can remove the num_ops field from rte_mempool_ops_table.
> >
> > Thoughts?
> >
> >
> > > Yes, should update the doc: how about this:
> > >
> > > diff --git a/lib/librte_mempool/rte_mempool.h b/lib/librte_mempool/rte_mempool.h
> > > index c90cf31..5a9c8a7 100644
> > > --- a/lib/librte_mempool/rte_mempool.h
> > > +++ b/lib/librte_mempool/rte_mempool.h
> > > @@ -904,7 +904,9 @@ int rte_mempool_ops_get_info(const struct rte_mempool *mp,
> > > * @param ops
> > > * Pointer to an ops structure to register.
> > > * @return
> > > - * - >=0: Success; return the index of the ops struct in the table.
> > > + * - >=0: Success; return the index of the last ops struct in the table.
> > > + * The number of the ops struct registered is equal to index
> > > + * returned + 1.
> > > * - -EINVAL - some missing callbacks while registering ops struct.
> > > * - -ENOSPC - the maximum number of ops structs has been reached.
> > > */
> > > diff --git a/lib/librte_mempool/rte_mempool_ops.c
> > > b/lib/librte_mempool/rte_mempool_ops.c
> > > index b0da096..053f340 100644
> > > --- a/lib/librte_mempool/rte_mempool_ops.c
> > > +++ b/lib/librte_mempool/rte_mempool_ops.c
> > > @@ -26,7 +26,11 @@ struct rte_mempool_ops_table rte_mempool_ops_table = {
> > > return strcmp(m_a->name, m_b->name);
> > > }
> > >
> > > -/* add a new ops struct in rte_mempool_ops_table, return its index. */
> > > +/*
> > > + * add a new ops struct in rte_mempool_ops_table.
> > > + * on success, return the index of the last ops
> > > + * struct in the table.
> > > + */
> > > int
> > > rte_mempool_register_ops(const struct rte_mempool_ops *h)
> > > {
> > > > > Also I remember patches which warn about above behaviour
> > > > > in documentation. If behaviour changes, corresponding
> > > > > documentation must be updated.
> > > >
> > > > One more point. If the patch is finally accepted it definitely
> > > > deserves few lines in release notes.
> > > OK, a separate patch should be sent before DPDK 20.05 release ?
> > > >
> > >
> > >
> > > --
> > > Thanks,
> > > Tonghao
>
>
>
> --
> Thanks,
> Tonghao
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v2 0/7] vfio/pci: SR-IOV support
2020-03-05 17:33 3% ` Alex Williamson
@ 2020-03-06 9:21 0% ` Tian, Kevin
0 siblings, 0 replies; 200+ results
From: Tian, Kevin @ 2020-03-06 9:21 UTC (permalink / raw)
To: Alex Williamson
Cc: kvm, linux-pci, linux-kernel, dev, mtosatti, thomas, bluca,
jerinjacobk, Richardson, Bruce, cohuck, Jason Wang
> From: Alex Williamson
> Sent: Friday, March 6, 2020 1:34 AM
>
> Hi Kevin,
>
> Sorry for the delay, I've been out on PTO...
>
> On Tue, 25 Feb 2020 02:33:27 +0000
> "Tian, Kevin" <kevin.tian@intel.com> wrote:
>
> > > From: Alex Williamson
> > > Sent: Thursday, February 20, 2020 2:54 AM
> > >
> > > Changes since v1 are primarily to patch 3/7 where the commit log is
> > > rewritten, along with option parsing and failure logging based on
> > > upstream discussions. The primary user visible difference is that
> > > option parsing is now much more strict. If a vf_token option is
> > > provided that cannot be used, we generate an error. As a result of
> > > this, opening a PF with a vf_token option will serve as a mechanism of
> > > setting the vf_token. This seems like a more user friendly API than
> > > the alternative of sometimes requiring the option (VFs in use) and
> > > sometimes rejecting it, and upholds our desire that the option is
> > > always either used or rejected.
> > >
> > > This also means that the VFIO_DEVICE_FEATURE ioctl is not the only
> > > means of setting the VF token, which might call into question whether
> > > we absolutely need this new ioctl. Currently I'm keeping it because I
> > > can imagine use cases, for example if a hypervisor were to support
> > > SR-IOV, the PF device might be opened without consideration for a VF
> > > token and we'd require the hypservisor to close and re-open the PF in
> > > order to set a known VF token, which is impractical.
> > >
> > > Series overview (same as provided with v1):
> >
> > Thanks for doing this!
> >
> > >
> > > The synopsis of this series is that we have an ongoing desire to drive
> > > PCIe SR-IOV PFs from userspace with VFIO. There's an immediate need
> > > for this with DPDK drivers and potentially interesting future use
> >
> > Can you provide a link to the DPDK discussion?
>
> There's a thread here which proposed an out-of-tree driver that enables
> a parallel sr-iov enabling interface for a vfio-pci own device.
> Clearly I felt strongly about it ;)
>
> https://patches.dpdk.org/patch/58810/
>
> Also, documentation for making use of an Intel FPGA device with DPDK
> requires the PF bound to igb_uio to support enabling SR-IOV:
>
> https://doc.dpdk.org/guides/bbdevs/fpga_lte_fec.html
thanks. it is useful.
>
> > > cases in virtualization. We've been reluctant to add this support
> > > previously due to the dependency and trust relationship between the
> > > VF device and PF driver. Minimally the PF driver can induce a denial
> > > of service to the VF, but depending on the specific implementation,
> > > the PF driver might also be responsible for moving data between VFs
> > > or have direct access to the state of the VF, including data or state
> > > otherwise private to the VF or VF driver.
> >
> > Just a loud thinking. While the motivation of VF token sounds reasonable
> > to me, I'm curious why the same concern is not raised in other usages.
> > For example, there is no such design in virtio framework, where the
> > virtio device could also be restarted, putting in separate process (vhost-
> user),
> > and even in separate VM (virtio-vhost-user), etc. Of course the para-
> > virtualized attribute of virtio implies some degree of trust, but as you
> > mentioned many SR-IOV implementations support VF->PF communication
> > which also implies some level of trust. It's perfectly fine if VFIO just tries
> > to do better than other sub-systems, but knowing how other people
> > tackle the similar problem may make the whole picture clearer. 😊
> >
> > +Jason.
>
> We can follow the thread with Jason, but I can't really speak to
> whether virtio needs something similar or doesn't provide enough PF
> access to be concerned. If they need a similar solution, we can
> collaborate, but the extension we're defining here is specifically part
> of the vfio-pci ABI, so it might not be easily portable to virtio.
>
> > > To help resolve these concerns, we introduce a VF token into the VFIO
> > > PCI ABI, which acts as a shared secret key between drivers. The
> > > userspace PF driver is required to set the VF token to a known value
> > > and userspace VF drivers are required to provide the token to access
> > > the VF device. If a PF driver is restarted with VF drivers in use, it
> > > must also provide the current token in order to prevent a rogue
> > > untrusted PF driver from replacing a known driver. The degree to
> > > which this new token is considered secret is left to the userspace
> > > drivers, the kernel intentionally provides no means to retrieve the
> > > current token.
> >
> > I'm wondering whether the token idea can be used beyond SR-IOV, e.g.
> > (1) we may allow vfio user space to manage Scalable IOV in the future,
> > which faces the similar challenge between the PF and mdev; (2) the
> > token might be used as a canonical way to replace off-tree acs-override
> > workaround, say, allowing the admin to assign devices within the
> > same iommu group to different VMs which trust each other. I'm not
> > sure how much complexity will be further introduced, but it's greatly
> > appreciated if you can help think a bit and if feasible abstract some
> > logic in vfio core layer for such potential usages...
>
> I don't see how this can be used for ACS override. Lacking ACS, we
> must assume lack of DMA isolation, which results in our IOMMU grouping.
> If we split IOMMU groups, that implies something that doesn't exist. A
> user can already create a process that can own the vfio group and pass
> vfio devices to other tasks, with the restriction of having a single
> DMA address space. If there is DMA isolation, then an mdev solution
> might be better, but given the IOMMU integration of SIOV, I'm not sure
> why the devices wouldn't simply be placed in separate groups by the
> IOMMU driver. Thanks,
You are right. I overlooked the single DMA address space limitation.
>
> Alex
>
> > > Note that the above token is only required for this new model where
> > > both the PF and VF devices are usable through vfio-pci. Existing
> > > models of VFIO drivers where the PF is used without SR-IOV enabled
> > > or the VF is bound to a userspace driver with an in-kernel, host PF
> > > driver are unaffected.
> > >
> > > The latter configuration above also highlights a new inverted scenario
> > > that is now possible, a userspace PF driver with in-kernel VF drivers.
> > > I believe this is a scenario that should be allowed, but should not be
> > > enabled by default. This series includes code to set a default
> > > driver_override for VFs sourced from a vfio-pci user owned PF, such
> > > that the VFs are also bound to vfio-pci. This model is compatible
> > > with tools like driverctl and allows the system administrator to
> > > decide if other bindings should be enabled. The VF token interface
> > > above exists only between vfio-pci PF and VF drivers, once a VF is
> > > bound to another driver, the administrator has effectively pronounced
> > > the device as trusted. The vfio-pci driver will note alternate
> > > binding in dmesg for logging and debugging purposes.
> > >
> > > Please review, comment, and test. The example QEMU implementation
> > > provided with the RFC is still current for this version. Thanks,
> > >
> > > Alex
> > >
> > > RFC:
> > >
> https://lore.kernel.org/lkml/158085337582.9445.17682266437583505502.stg
> > > it@gimli.home/
> > > v1:
> > >
> https://lore.kernel.org/lkml/158145472604.16827.15751375540102298130.st
> > > git@gimli.home/
> > >
> > > ---
> > >
> > > Alex Williamson (7):
> > > vfio: Include optional device match in vfio_device_ops callbacks
> > > vfio/pci: Implement match ops
> > > vfio/pci: Introduce VF token
> > > vfio: Introduce VFIO_DEVICE_FEATURE ioctl and first user
> > > vfio/pci: Add sriov_configure support
> > > vfio/pci: Remove dev_fmt definition
> > > vfio/pci: Cleanup .probe() exit paths
> > >
> > >
> > > drivers/vfio/pci/vfio_pci.c | 383
> > > +++++++++++++++++++++++++++++++++--
> > > drivers/vfio/pci/vfio_pci_private.h | 10 +
> > > drivers/vfio/vfio.c | 20 +-
> > > include/linux/vfio.h | 4
> > > include/uapi/linux/vfio.h | 37 +++
> > > 5 files changed, 426 insertions(+), 28 deletions(-)
> >
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH dpdk-dev v3] mempool: sort the rte_mempool_ops by name
2020-03-09 8:27 3% ` Olivier Matz
@ 2020-03-09 8:55 0% ` Tonghao Zhang
2020-03-09 9:05 0% ` Olivier Matz
2020-03-09 13:15 0% ` David Marchand
0 siblings, 2 replies; 200+ results
From: Tonghao Zhang @ 2020-03-09 8:55 UTC (permalink / raw)
To: Olivier Matz
Cc: Andrew Rybchenko, Jerin Jacob, dpdk-dev, Gage Eads,
Artem V. Andreev, Jerin Jacob, Nithin Dabilpuram, Vamsi Attunuru,
Hemant Agrawal
On Mon, Mar 9, 2020 at 4:27 PM Olivier Matz <olivier.matz@6wind.com> wrote:
>
> Hi,
>
> On Mon, Mar 09, 2020 at 11:01:25AM +0800, Tonghao Zhang wrote:
> > On Sat, Mar 7, 2020 at 8:54 PM Andrew Rybchenko
> > <arybchenko@solarflare.com> wrote:
> > >
> > > On 3/7/20 3:51 PM, Andrew Rybchenko wrote:
> > > > On 3/6/20 4:37 PM, Jerin Jacob wrote:
> > > >> On Fri, Mar 6, 2020 at 7:06 PM <xiangxia.m.yue@gmail.com> wrote:
> > > >>> From: Tonghao Zhang <xiangxia.m.yue@gmail.com>
> > > >>>
> > > >>> The order of mempool initiation affects mempool index in the
> > > >>> rte_mempool_ops_table. For example, when building APPs with:
> > > >>>
> > > >>> $ gcc -lrte_mempool_bucket -lrte_mempool_ring ...
> > > >>>
> > > >>> The "bucket" mempool will be registered firstly, and its index
> > > >>> in table is 0 while the index of "ring" mempool is 1. DPDK
> > > >>> uses the mk/rte.app.mk to build APPs, and others, for example,
> > > >>> Open vSwitch, use the libdpdk.a or libdpdk.so to build it.
> > > >>> The mempool lib linked in dpdk and Open vSwitch is different.
> > > >>>
> > > >>> The mempool can be used between primary and secondary process,
> > > >>> such as dpdk-pdump and pdump-pmd/Open vSwitch(pdump enabled).
> > > >>> There will be a crash because dpdk-pdump creates the "ring_mp_mc"
> > > >>> ring which index in table is 0, but the index of "bucket" ring
> > > >>> is 0 in Open vSwitch. If Open vSwitch use the index 0 to get
> > > >>> mempool ops and malloc memory from mempool. The crash will occur:
> > > >>>
> > > >>> bucket_dequeue (access null and crash)
> > > >>> rte_mempool_get_ops (should get "ring_mp_mc",
> > > >>> but get "bucket" mempool)
> > > >>> rte_mempool_ops_dequeue_bulk
> > > >>> ...
> > > >>> rte_pktmbuf_alloc
> > > >>> rte_pktmbuf_copy
> > > >>> pdump_copy
> > > >>> pdump_rx
> > > >>> rte_eth_rx_burst
> > > >>>
> > > >>> To avoid the crash, there are some solution:
> > > >>> * constructor priority: Different mempool uses different
> > > >>> priority in RTE_INIT, but it's not easy to maintain.
> > > >>>
> > > >>> * change mk/rte.app.mk: Change the order in mk/rte.app.mk to
> > > >>> be same as libdpdk.a/libdpdk.so, but when adding a new mempool
> > > >>> driver in future, we must make sure the order.
> > > >>>
> > > >>> * register mempool orderly: Sort the mempool when registering,
> > > >>> so the lib linked will not affect the index in mempool table.
> > > >>>
> > > >>> Signed-off-by: Tonghao Zhang <xiangxia.m.yue@gmail.com>
> > > >>> Acked-by: Olivier Matz <olivier.matz@6wind.com>
> > > >> Acked-by: Jerin Jacob <jerinj@marvell.com>
> > > >
> > > > The patch is OK, but the fact that ops index changes during
> > > > mempool driver lifetime is frightening. In fact it breaks
> > > > rte_mempool_register_ops() return value semantics (read
> > > > as API break). The return value is not used in DPDK, but it
> > > > is a public function. If I'm not mistaken it should be taken
> > > > into account.
>
> Good points.
>
> The fact that the ops index changes during mempool driver lifetime is
> indeed frightening, especially knowning that this is a dynamic
> registration that could happen at any moment in the life of the
> application. Also, breaking the ABI is not desirable.
That solution is better.
> Let me try to propose something else to solve your issue:
>
> 1/ At init, the primary process allocates a struct in shared memory
> (named memzone):
>
> struct rte_mempool_shared_ops {
> size_t num_mempool_ops;
> struct {
> char name[RTE_MEMPOOL_OPS_NAMESIZE];
> } mempool_ops[RTE_MEMPOOL_MAX_OPS_IDX];
> char *mempool_ops_name[RTE_MEMPOOL_MAX_OPS_IDX];
> rte_spinlock_t mempool;
> }
>
> 2/ When we register a mempool ops, we first get a name and id from the
> shared struct: with the lock held, lookup for the registered name and
> return its index, else get the last id and copy the name in the struct.
>
> 3/ Then do as before (in the per-process global table), except that we
> reuse the registered id.
>
> We can remove the num_ops field from rte_mempool_ops_table.
>
> Thoughts?
>
>
> > Yes, should update the doc: how about this:
> >
> > diff --git a/lib/librte_mempool/rte_mempool.h b/lib/librte_mempool/rte_mempool.h
> > index c90cf31..5a9c8a7 100644
> > --- a/lib/librte_mempool/rte_mempool.h
> > +++ b/lib/librte_mempool/rte_mempool.h
> > @@ -904,7 +904,9 @@ int rte_mempool_ops_get_info(const struct rte_mempool *mp,
> > * @param ops
> > * Pointer to an ops structure to register.
> > * @return
> > - * - >=0: Success; return the index of the ops struct in the table.
> > + * - >=0: Success; return the index of the last ops struct in the table.
> > + * The number of the ops struct registered is equal to index
> > + * returned + 1.
> > * - -EINVAL - some missing callbacks while registering ops struct.
> > * - -ENOSPC - the maximum number of ops structs has been reached.
> > */
> > diff --git a/lib/librte_mempool/rte_mempool_ops.c
> > b/lib/librte_mempool/rte_mempool_ops.c
> > index b0da096..053f340 100644
> > --- a/lib/librte_mempool/rte_mempool_ops.c
> > +++ b/lib/librte_mempool/rte_mempool_ops.c
> > @@ -26,7 +26,11 @@ struct rte_mempool_ops_table rte_mempool_ops_table = {
> > return strcmp(m_a->name, m_b->name);
> > }
> >
> > -/* add a new ops struct in rte_mempool_ops_table, return its index. */
> > +/*
> > + * add a new ops struct in rte_mempool_ops_table.
> > + * on success, return the index of the last ops
> > + * struct in the table.
> > + */
> > int
> > rte_mempool_register_ops(const struct rte_mempool_ops *h)
> > {
> > > > Also I remember patches which warn about above behaviour
> > > > in documentation. If behaviour changes, corresponding
> > > > documentation must be updated.
> > >
> > > One more point. If the patch is finally accepted it definitely
> > > deserves few lines in release notes.
> > OK, a separate patch should be sent before DPDK 20.05 release ?
> > >
> >
> >
> > --
> > Thanks,
> > Tonghao
--
Thanks,
Tonghao
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH dpdk-dev v3] mempool: sort the rte_mempool_ops by name
@ 2020-03-09 8:27 3% ` Olivier Matz
2020-03-09 8:55 0% ` Tonghao Zhang
0 siblings, 1 reply; 200+ results
From: Olivier Matz @ 2020-03-09 8:27 UTC (permalink / raw)
To: Tonghao Zhang
Cc: Andrew Rybchenko, Jerin Jacob, dpdk-dev, Gage Eads,
Artem V. Andreev, Jerin Jacob, Nithin Dabilpuram, Vamsi Attunuru,
Hemant Agrawal
Hi,
On Mon, Mar 09, 2020 at 11:01:25AM +0800, Tonghao Zhang wrote:
> On Sat, Mar 7, 2020 at 8:54 PM Andrew Rybchenko
> <arybchenko@solarflare.com> wrote:
> >
> > On 3/7/20 3:51 PM, Andrew Rybchenko wrote:
> > > On 3/6/20 4:37 PM, Jerin Jacob wrote:
> > >> On Fri, Mar 6, 2020 at 7:06 PM <xiangxia.m.yue@gmail.com> wrote:
> > >>> From: Tonghao Zhang <xiangxia.m.yue@gmail.com>
> > >>>
> > >>> The order of mempool initiation affects mempool index in the
> > >>> rte_mempool_ops_table. For example, when building APPs with:
> > >>>
> > >>> $ gcc -lrte_mempool_bucket -lrte_mempool_ring ...
> > >>>
> > >>> The "bucket" mempool will be registered firstly, and its index
> > >>> in table is 0 while the index of "ring" mempool is 1. DPDK
> > >>> uses the mk/rte.app.mk to build APPs, and others, for example,
> > >>> Open vSwitch, use the libdpdk.a or libdpdk.so to build it.
> > >>> The mempool lib linked in dpdk and Open vSwitch is different.
> > >>>
> > >>> The mempool can be used between primary and secondary process,
> > >>> such as dpdk-pdump and pdump-pmd/Open vSwitch(pdump enabled).
> > >>> There will be a crash because dpdk-pdump creates the "ring_mp_mc"
> > >>> ring which index in table is 0, but the index of "bucket" ring
> > >>> is 0 in Open vSwitch. If Open vSwitch use the index 0 to get
> > >>> mempool ops and malloc memory from mempool. The crash will occur:
> > >>>
> > >>> bucket_dequeue (access null and crash)
> > >>> rte_mempool_get_ops (should get "ring_mp_mc",
> > >>> but get "bucket" mempool)
> > >>> rte_mempool_ops_dequeue_bulk
> > >>> ...
> > >>> rte_pktmbuf_alloc
> > >>> rte_pktmbuf_copy
> > >>> pdump_copy
> > >>> pdump_rx
> > >>> rte_eth_rx_burst
> > >>>
> > >>> To avoid the crash, there are some solution:
> > >>> * constructor priority: Different mempool uses different
> > >>> priority in RTE_INIT, but it's not easy to maintain.
> > >>>
> > >>> * change mk/rte.app.mk: Change the order in mk/rte.app.mk to
> > >>> be same as libdpdk.a/libdpdk.so, but when adding a new mempool
> > >>> driver in future, we must make sure the order.
> > >>>
> > >>> * register mempool orderly: Sort the mempool when registering,
> > >>> so the lib linked will not affect the index in mempool table.
> > >>>
> > >>> Signed-off-by: Tonghao Zhang <xiangxia.m.yue@gmail.com>
> > >>> Acked-by: Olivier Matz <olivier.matz@6wind.com>
> > >> Acked-by: Jerin Jacob <jerinj@marvell.com>
> > >
> > > The patch is OK, but the fact that ops index changes during
> > > mempool driver lifetime is frightening. In fact it breaks
> > > rte_mempool_register_ops() return value semantics (read
> > > as API break). The return value is not used in DPDK, but it
> > > is a public function. If I'm not mistaken it should be taken
> > > into account.
Good points.
The fact that the ops index changes during mempool driver lifetime is
indeed frightening, especially knowning that this is a dynamic
registration that could happen at any moment in the life of the
application. Also, breaking the ABI is not desirable.
Let me try to propose something else to solve your issue:
1/ At init, the primary process allocates a struct in shared memory
(named memzone):
struct rte_mempool_shared_ops {
size_t num_mempool_ops;
struct {
char name[RTE_MEMPOOL_OPS_NAMESIZE];
} mempool_ops[RTE_MEMPOOL_MAX_OPS_IDX];
char *mempool_ops_name[RTE_MEMPOOL_MAX_OPS_IDX];
rte_spinlock_t mempool;
}
2/ When we register a mempool ops, we first get a name and id from the
shared struct: with the lock held, lookup for the registered name and
return its index, else get the last id and copy the name in the struct.
3/ Then do as before (in the per-process global table), except that we
reuse the registered id.
We can remove the num_ops field from rte_mempool_ops_table.
Thoughts?
> Yes, should update the doc: how about this:
>
> diff --git a/lib/librte_mempool/rte_mempool.h b/lib/librte_mempool/rte_mempool.h
> index c90cf31..5a9c8a7 100644
> --- a/lib/librte_mempool/rte_mempool.h
> +++ b/lib/librte_mempool/rte_mempool.h
> @@ -904,7 +904,9 @@ int rte_mempool_ops_get_info(const struct rte_mempool *mp,
> * @param ops
> * Pointer to an ops structure to register.
> * @return
> - * - >=0: Success; return the index of the ops struct in the table.
> + * - >=0: Success; return the index of the last ops struct in the table.
> + * The number of the ops struct registered is equal to index
> + * returned + 1.
> * - -EINVAL - some missing callbacks while registering ops struct.
> * - -ENOSPC - the maximum number of ops structs has been reached.
> */
> diff --git a/lib/librte_mempool/rte_mempool_ops.c
> b/lib/librte_mempool/rte_mempool_ops.c
> index b0da096..053f340 100644
> --- a/lib/librte_mempool/rte_mempool_ops.c
> +++ b/lib/librte_mempool/rte_mempool_ops.c
> @@ -26,7 +26,11 @@ struct rte_mempool_ops_table rte_mempool_ops_table = {
> return strcmp(m_a->name, m_b->name);
> }
>
> -/* add a new ops struct in rte_mempool_ops_table, return its index. */
> +/*
> + * add a new ops struct in rte_mempool_ops_table.
> + * on success, return the index of the last ops
> + * struct in the table.
> + */
> int
> rte_mempool_register_ops(const struct rte_mempool_ops *h)
> {
> > > Also I remember patches which warn about above behaviour
> > > in documentation. If behaviour changes, corresponding
> > > documentation must be updated.
> >
> > One more point. If the patch is finally accepted it definitely
> > deserves few lines in release notes.
> OK, a separate patch should be sent before DPDK 20.05 release ?
> >
>
>
> --
> Thanks,
> Tonghao
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH 00/16] NXP DPAAx fixes and enhancements
2020-03-05 9:19 0% ` Hemant Agrawal (OSS)
@ 2020-03-06 10:12 3% ` David Marchand
2020-03-10 10:36 3% ` Dodji Seketeli
0 siblings, 1 reply; 200+ results
From: David Marchand @ 2020-03-06 10:12 UTC (permalink / raw)
To: Hemant Agrawal (OSS)
Cc: Yigit, Ferruh, dev, Neil Horman, Thomas Monjalon, Dodji Seketeli
On Thu, Mar 5, 2020 at 10:19 AM Hemant Agrawal (OSS)
<hemant.agrawal@oss.nxp.com> wrote:
> > On Thu, Mar 5, 2020 at 10:06 AM Hemant Agrawal (OSS)
> > <hemant.agrawal@oss.nxp.com> wrote:
> > >
> > > Hi David,
> > > > On Mon, Mar 2, 2020 at 10:26 AM Hemant Agrawal
> > > > <hemant.agrawal@nxp.com> wrote:
> > > > >
> > > > > This patch series add various patches for enhancing and fixing NXP
> > > > > fslmc bus, dpaa bus, and dpaax.
> > > > >
> > > > > - the main change is support to allow thread migration across
> > > > > lcores
> > > > > - improving the multi-process support
> > > >
> > > > This series triggers an ABI warning that must be investigated.
> > > >https://travis-ci.com/ovsrobot/dpdk/jobs/292904119#L2233
> > >
> > > [Hemant]
> > > As per the logs:
> > >
> > > Variables changes summary: 1 Removed, 2 Changed, 0 Added variables
> > > 1 Removed variable:
> > > 'dpaa2_portal_dqrr per_lcore_dpaa2_held_bufs'
> > {per_lcore_dpaa2_held_bufs@@DPDK_20.0}
> > > 2 Changed variables:
> > > [C]'dpaa2_io_portal_t dpaa2_io_portal[128]' was changed at
> > dpaa2_hw_dpio.h:40:1: size of symbol changed from 5120 to 2048
> > > [C]'dpaa2_io_portal_t per_lcore__dpaa2_io' was changed at
> > > dpaa2_hw_dpio.h:20:1: size of symbol changed from 40 to 16
> > >
> > > Error: ABI issue reported for 'abidiff --suppr devtools/libabigail.abignore --
> > no-added-syms --headers-dir1 reference/usr/local/include --headers-dir2
> > install/usr/local/include reference/dump/librte_bus_fslmc.dump
> > install/dump/librte_bus_fslmc.dump'
> > >
> > > ---------------
> > >
> > > These changes are w.r.t modifications in internal structures and variables.
> > They may be ignored.
> >
> > The ABI check considers symbol exposed in headers available to final users.
> > If those are internal, why are the headers public?
> >
>
> [Hemant] These symbols are not part of any public header files, but they are part of *.map files to share them between different driver libs i.e bus_fslmc and net_dpaa2
I would expect libabigail to skip those symbols, so there is something
I have missed in how --headers-dirX work.
Anyway, all of those symbols in dpaa are part of the driver ABI.
We are still missing a way to mark internal symbols.
Neil had posted a framework for this
http://patchwork.dpdk.org/project/dpdk/list/?series=5004.
In order to get this series passing the checks, I recommend NXP
rebasing Neil scripts (I will help reviewing this part), then mark all
those symbols as internal in its drivers.
Other vendor will convert their drivers later, as there is no need at
the moment.
Thanks.
--
David Marchand
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH v2 0/7] vfio/pci: SR-IOV support
2020-02-25 2:33 0% ` Tian, Kevin
@ 2020-03-05 17:33 3% ` Alex Williamson
2020-03-06 9:21 0% ` Tian, Kevin
0 siblings, 1 reply; 200+ results
From: Alex Williamson @ 2020-03-05 17:33 UTC (permalink / raw)
To: Tian, Kevin
Cc: kvm, linux-pci, linux-kernel, dev, mtosatti, thomas, bluca,
jerinjacobk, Richardson, Bruce, cohuck, Jason Wang
Hi Kevin,
Sorry for the delay, I've been out on PTO...
On Tue, 25 Feb 2020 02:33:27 +0000
"Tian, Kevin" <kevin.tian@intel.com> wrote:
> > From: Alex Williamson
> > Sent: Thursday, February 20, 2020 2:54 AM
> >
> > Changes since v1 are primarily to patch 3/7 where the commit log is
> > rewritten, along with option parsing and failure logging based on
> > upstream discussions. The primary user visible difference is that
> > option parsing is now much more strict. If a vf_token option is
> > provided that cannot be used, we generate an error. As a result of
> > this, opening a PF with a vf_token option will serve as a mechanism of
> > setting the vf_token. This seems like a more user friendly API than
> > the alternative of sometimes requiring the option (VFs in use) and
> > sometimes rejecting it, and upholds our desire that the option is
> > always either used or rejected.
> >
> > This also means that the VFIO_DEVICE_FEATURE ioctl is not the only
> > means of setting the VF token, which might call into question whether
> > we absolutely need this new ioctl. Currently I'm keeping it because I
> > can imagine use cases, for example if a hypervisor were to support
> > SR-IOV, the PF device might be opened without consideration for a VF
> > token and we'd require the hypservisor to close and re-open the PF in
> > order to set a known VF token, which is impractical.
> >
> > Series overview (same as provided with v1):
>
> Thanks for doing this!
>
> >
> > The synopsis of this series is that we have an ongoing desire to drive
> > PCIe SR-IOV PFs from userspace with VFIO. There's an immediate need
> > for this with DPDK drivers and potentially interesting future use
>
> Can you provide a link to the DPDK discussion?
There's a thread here which proposed an out-of-tree driver that enables
a parallel sr-iov enabling interface for a vfio-pci own device.
Clearly I felt strongly about it ;)
https://patches.dpdk.org/patch/58810/
Also, documentation for making use of an Intel FPGA device with DPDK
requires the PF bound to igb_uio to support enabling SR-IOV:
https://doc.dpdk.org/guides/bbdevs/fpga_lte_fec.html
> > cases in virtualization. We've been reluctant to add this support
> > previously due to the dependency and trust relationship between the
> > VF device and PF driver. Minimally the PF driver can induce a denial
> > of service to the VF, but depending on the specific implementation,
> > the PF driver might also be responsible for moving data between VFs
> > or have direct access to the state of the VF, including data or state
> > otherwise private to the VF or VF driver.
>
> Just a loud thinking. While the motivation of VF token sounds reasonable
> to me, I'm curious why the same concern is not raised in other usages.
> For example, there is no such design in virtio framework, where the
> virtio device could also be restarted, putting in separate process (vhost-user),
> and even in separate VM (virtio-vhost-user), etc. Of course the para-
> virtualized attribute of virtio implies some degree of trust, but as you
> mentioned many SR-IOV implementations support VF->PF communication
> which also implies some level of trust. It's perfectly fine if VFIO just tries
> to do better than other sub-systems, but knowing how other people
> tackle the similar problem may make the whole picture clearer. 😊
>
> +Jason.
We can follow the thread with Jason, but I can't really speak to
whether virtio needs something similar or doesn't provide enough PF
access to be concerned. If they need a similar solution, we can
collaborate, but the extension we're defining here is specifically part
of the vfio-pci ABI, so it might not be easily portable to virtio.
> > To help resolve these concerns, we introduce a VF token into the VFIO
> > PCI ABI, which acts as a shared secret key between drivers. The
> > userspace PF driver is required to set the VF token to a known value
> > and userspace VF drivers are required to provide the token to access
> > the VF device. If a PF driver is restarted with VF drivers in use, it
> > must also provide the current token in order to prevent a rogue
> > untrusted PF driver from replacing a known driver. The degree to
> > which this new token is considered secret is left to the userspace
> > drivers, the kernel intentionally provides no means to retrieve the
> > current token.
>
> I'm wondering whether the token idea can be used beyond SR-IOV, e.g.
> (1) we may allow vfio user space to manage Scalable IOV in the future,
> which faces the similar challenge between the PF and mdev; (2) the
> token might be used as a canonical way to replace off-tree acs-override
> workaround, say, allowing the admin to assign devices within the
> same iommu group to different VMs which trust each other. I'm not
> sure how much complexity will be further introduced, but it's greatly
> appreciated if you can help think a bit and if feasible abstract some
> logic in vfio core layer for such potential usages...
I don't see how this can be used for ACS override. Lacking ACS, we
must assume lack of DMA isolation, which results in our IOMMU grouping.
If we split IOMMU groups, that implies something that doesn't exist. A
user can already create a process that can own the vfio group and pass
vfio devices to other tasks, with the restriction of having a single
DMA address space. If there is DMA isolation, then an mdev solution
might be better, but given the IOMMU integration of SIOV, I'm not sure
why the devices wouldn't simply be placed in separate groups by the
IOMMU driver. Thanks,
Alex
> > Note that the above token is only required for this new model where
> > both the PF and VF devices are usable through vfio-pci. Existing
> > models of VFIO drivers where the PF is used without SR-IOV enabled
> > or the VF is bound to a userspace driver with an in-kernel, host PF
> > driver are unaffected.
> >
> > The latter configuration above also highlights a new inverted scenario
> > that is now possible, a userspace PF driver with in-kernel VF drivers.
> > I believe this is a scenario that should be allowed, but should not be
> > enabled by default. This series includes code to set a default
> > driver_override for VFs sourced from a vfio-pci user owned PF, such
> > that the VFs are also bound to vfio-pci. This model is compatible
> > with tools like driverctl and allows the system administrator to
> > decide if other bindings should be enabled. The VF token interface
> > above exists only between vfio-pci PF and VF drivers, once a VF is
> > bound to another driver, the administrator has effectively pronounced
> > the device as trusted. The vfio-pci driver will note alternate
> > binding in dmesg for logging and debugging purposes.
> >
> > Please review, comment, and test. The example QEMU implementation
> > provided with the RFC is still current for this version. Thanks,
> >
> > Alex
> >
> > RFC:
> > https://lore.kernel.org/lkml/158085337582.9445.17682266437583505502.stg
> > it@gimli.home/
> > v1:
> > https://lore.kernel.org/lkml/158145472604.16827.15751375540102298130.st
> > git@gimli.home/
> >
> > ---
> >
> > Alex Williamson (7):
> > vfio: Include optional device match in vfio_device_ops callbacks
> > vfio/pci: Implement match ops
> > vfio/pci: Introduce VF token
> > vfio: Introduce VFIO_DEVICE_FEATURE ioctl and first user
> > vfio/pci: Add sriov_configure support
> > vfio/pci: Remove dev_fmt definition
> > vfio/pci: Cleanup .probe() exit paths
> >
> >
> > drivers/vfio/pci/vfio_pci.c | 383
> > +++++++++++++++++++++++++++++++++--
> > drivers/vfio/pci/vfio_pci_private.h | 10 +
> > drivers/vfio/vfio.c | 20 +-
> > include/linux/vfio.h | 4
> > include/uapi/linux/vfio.h | 37 +++
> > 5 files changed, 426 insertions(+), 28 deletions(-)
>
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH 11/16] net/dpaa: enable Tx queue taildrop
@ 2020-03-05 14:14 3% ` Ferruh Yigit
0 siblings, 0 replies; 200+ results
From: Ferruh Yigit @ 2020-03-05 14:14 UTC (permalink / raw)
To: Gagandeep Singh, Hemant Agrawal; +Cc: dev, David Marchand, Neil Horman
On 3/5/2020 6:49 AM, Gagandeep Singh wrote:
> Hi Ferruh,
>>> diff --git a/drivers/bus/dpaa/rte_bus_dpaa_version.map
>> b/drivers/bus/dpaa/rte_bus_dpaa_version.map
>>> index e6ca4361e..86f5811b0 100644
>>> --- a/drivers/bus/dpaa/rte_bus_dpaa_version.map
>>> +++ b/drivers/bus/dpaa/rte_bus_dpaa_version.map
>>> @@ -94,3 +94,10 @@ DPDK_20.0 {
>>>
>>> local: *;
>>> };
>>> +
>>> +EXPERIMENTAL {
>>> + global:
>>> +
>>> + qman_ern_poll_free;
>>> + qman_ern_register_cb;
>>> +};
>>
>> Aren't these bus APIs internal (between net/crypto/event drivers and bus
>> libraries)? And they are *not* for applications to use.
>>
>> If they are only internal is there any point to make them experimental?
>
> Yes, these are internal only APIS. So, is it ok to add them directly into DPDK_20.0 section?
>
I think it is OK, but I don't know the what will be the affect to the ABI check
tools.
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH 00/16] NXP DPAAx fixes and enhancements
2020-03-05 9:09 3% ` David Marchand
@ 2020-03-05 9:19 0% ` Hemant Agrawal (OSS)
2020-03-06 10:12 3% ` David Marchand
0 siblings, 1 reply; 200+ results
From: Hemant Agrawal (OSS) @ 2020-03-05 9:19 UTC (permalink / raw)
To: David Marchand, Hemant Agrawal (OSS); +Cc: Yigit, Ferruh, dev
Hi David,
> On Thu, Mar 5, 2020 at 10:06 AM Hemant Agrawal (OSS)
> <hemant.agrawal@oss.nxp.com> wrote:
> >
> > Hi David,
> > > On Mon, Mar 2, 2020 at 10:26 AM Hemant Agrawal
> > > <hemant.agrawal@nxp.com> wrote:
> > > >
> > > > This patch series add various patches for enhancing and fixing NXP
> > > > fslmc bus, dpaa bus, and dpaax.
> > > >
> > > > - the main change is support to allow thread migration across
> > > > lcores
> > > > - improving the multi-process support
> > >
> > > This series triggers an ABI warning that must be investigated.
> > >https://travis-ci.com/ovsrobot/dpdk/jobs/292904119#L2233
> >
> > [Hemant]
> > As per the logs:
> >
> > Variables changes summary: 1 Removed, 2 Changed, 0 Added variables
> > 1 Removed variable:
> > 'dpaa2_portal_dqrr per_lcore_dpaa2_held_bufs'
> {per_lcore_dpaa2_held_bufs@@DPDK_20.0}
> > 2 Changed variables:
> > [C]'dpaa2_io_portal_t dpaa2_io_portal[128]' was changed at
> dpaa2_hw_dpio.h:40:1: size of symbol changed from 5120 to 2048
> > [C]'dpaa2_io_portal_t per_lcore__dpaa2_io' was changed at
> > dpaa2_hw_dpio.h:20:1: size of symbol changed from 40 to 16
> >
> > Error: ABI issue reported for 'abidiff --suppr devtools/libabigail.abignore --
> no-added-syms --headers-dir1 reference/usr/local/include --headers-dir2
> install/usr/local/include reference/dump/librte_bus_fslmc.dump
> install/dump/librte_bus_fslmc.dump'
> >
> > ---------------
> >
> > These changes are w.r.t modifications in internal structures and variables.
> They may be ignored.
>
> The ABI check considers symbol exposed in headers available to final users.
> If those are internal, why are the headers public?
>
[Hemant] These symbols are not part of any public header files, but they are part of *.map files to share them between different driver libs i.e bus_fslmc and net_dpaa2
>
> --
> David Marchand
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH 00/16] NXP DPAAx fixes and enhancements
2020-03-05 9:06 3% ` Hemant Agrawal (OSS)
@ 2020-03-05 9:09 3% ` David Marchand
2020-03-05 9:19 0% ` Hemant Agrawal (OSS)
0 siblings, 1 reply; 200+ results
From: David Marchand @ 2020-03-05 9:09 UTC (permalink / raw)
To: Hemant Agrawal (OSS); +Cc: Yigit, Ferruh, dev
On Thu, Mar 5, 2020 at 10:06 AM Hemant Agrawal (OSS)
<hemant.agrawal@oss.nxp.com> wrote:
>
> Hi David,
> > On Mon, Mar 2, 2020 at 10:26 AM Hemant Agrawal
> > <hemant.agrawal@nxp.com> wrote:
> > >
> > > This patch series add various patches for enhancing and fixing NXP
> > > fslmc bus, dpaa bus, and dpaax.
> > >
> > > - the main change is support to allow thread migration across lcores
> > > - improving the multi-process support
> >
> > This series triggers an ABI warning that must be investigated.
> >https://travis-ci.com/ovsrobot/dpdk/jobs/292904119#L2233
>
> [Hemant]
> As per the logs:
>
> Variables changes summary: 1 Removed, 2 Changed, 0 Added variables
> 1 Removed variable:
> 'dpaa2_portal_dqrr per_lcore_dpaa2_held_bufs' {per_lcore_dpaa2_held_bufs@@DPDK_20.0}
> 2 Changed variables:
> [C]'dpaa2_io_portal_t dpaa2_io_portal[128]' was changed at dpaa2_hw_dpio.h:40:1: size of symbol changed from 5120 to 2048
> [C]'dpaa2_io_portal_t per_lcore__dpaa2_io' was changed at dpaa2_hw_dpio.h:20:1: size of symbol changed from 40 to 16
>
> Error: ABI issue reported for 'abidiff --suppr devtools/libabigail.abignore --no-added-syms --headers-dir1 reference/usr/local/include --headers-dir2 install/usr/local/include reference/dump/librte_bus_fslmc.dump install/dump/librte_bus_fslmc.dump'
>
> ---------------
>
> These changes are w.r.t modifications in internal structures and variables. They may be ignored.
The ABI check considers symbol exposed in headers available to final users.
If those are internal, why are the headers public?
--
David Marchand
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH 00/16] NXP DPAAx fixes and enhancements
2020-03-02 13:01 3% ` David Marchand
@ 2020-03-05 9:06 3% ` Hemant Agrawal (OSS)
2020-03-05 9:09 3% ` David Marchand
0 siblings, 1 reply; 200+ results
From: Hemant Agrawal (OSS) @ 2020-03-05 9:06 UTC (permalink / raw)
To: David Marchand; +Cc: Yigit, Ferruh, dev
Hi David,
> On Mon, Mar 2, 2020 at 10:26 AM Hemant Agrawal
> <hemant.agrawal@nxp.com> wrote:
> >
> > This patch series add various patches for enhancing and fixing NXP
> > fslmc bus, dpaa bus, and dpaax.
> >
> > - the main change is support to allow thread migration across lcores
> > - improving the multi-process support
>
> This series triggers an ABI warning that must be investigated.
>https://travis-ci.com/ovsrobot/dpdk/jobs/292904119#L2233
[Hemant]
As per the logs:
Variables changes summary: 1 Removed, 2 Changed, 0 Added variables
1 Removed variable:
'dpaa2_portal_dqrr per_lcore_dpaa2_held_bufs' {per_lcore_dpaa2_held_bufs@@DPDK_20.0}
2 Changed variables:
[C]'dpaa2_io_portal_t dpaa2_io_portal[128]' was changed at dpaa2_hw_dpio.h:40:1: size of symbol changed from 5120 to 2048
[C]'dpaa2_io_portal_t per_lcore__dpaa2_io' was changed at dpaa2_hw_dpio.h:20:1: size of symbol changed from 40 to 16
Error: ABI issue reported for 'abidiff --suppr devtools/libabigail.abignore --no-added-syms --headers-dir1 reference/usr/local/include --headers-dir2 install/usr/local/include reference/dump/librte_bus_fslmc.dump install/dump/librte_bus_fslmc.dump'
---------------
These changes are w.r.t modifications in internal structures and variables. They may be ignored.
>
> --
> David Marchand
^ permalink raw reply [relevance 3%]
* [dpdk-dev] DPDK techboard minutes of February 26
@ 2020-03-05 8:53 4% Thomas Monjalon
0 siblings, 0 replies; 200+ results
From: Thomas Monjalon @ 2020-03-05 8:53 UTC (permalink / raw)
To: dev; +Cc: techboard
Minutes of Technical Board meeting, 2020-02-26
Members Attending: 9/11
- Ferruh Yigit
- Hemant Agrawal
- Honnappa Nagarahalli
- Jerin Jacob
- Kevin Traynor
- Konstantin Ananyev
- Olivier Matz
- Stephen Hemminger
- Thomas Monjalon (Chair)
NOTE: Technical Board meetings happen every second Wednesday
on IRC channel #dpdk-board, at 3pm UTC.
Meetings are public and DPDK community members are welcome to attend.
1/ Technical Board status for the Governing Board meeting
- ABI compatibility and tooling
- SPDX conversion almost complete
- Stepping away from Linux out-of-tree modules - new dpdk-kmods repo
- Project scope discussion
- PMDT project - new acceptance guidelines
- Tooling trend (tracing, dumps, metrics, telemetry, monitoring)
- ICC support: warning is not fatal
- Windows port is progressing
- new device classes: vDPA and regex
- IPsec integration is difficult and progressing
- Security pre-release list is not growing a lot
- not tried any mentoring program this year
2/ Doc maintenance
Not much activity on documentation:
- no content improvement
- hard to get good reviews
Volunteers are welcome.
Maintenance of the documentation becomes a recurring techboard topic.
3/ Community lab goals
The Governing Board is evaluating how to progress with DPDK CI.
The Technical Board had to provide status and goals.
The first CI priority was already agreed in previous meeting:
basic ethdev functional testing on real NICs.
Coverity is doing static analysis regularly.
Travis is testing (requires integration for per-patch report):
- compilation
- ABI
- libraries (without traffic)
OBS could be integrated for per-patch compilation testing.
The community lab is testing (for each patch):
- compilation on more OSes
- libraries (without traffic)
- micro-benchmark with traffic on real HW
- downstream projects (OVS, SPDK)
Key benefits of the community lab are:
- real HW
- vendor neutrality
- confidence
The ideal lab should meet these requirements:
- reliable
- neutral
- transparent visibility
- coverage summary
- trending graphs
- easy to add/run a test
- easy to reproduce a test locally
- email reports
- per-patch testing integrated with patchwork
- per-commit testing (after merge in master or next-*)
- per-release (including -rc) testing
and should run these tests:
- compilation
- functional
- performance
- interoperability
- portability
- downstream projects
Three roles are identified.
1) Lab management: hosting, physical setup, monitoring.
2) Framework/Infrastructure (can be remote):
test integration, database, logs, reports, graphs, web pages.
3) development of test cases:
writing and/or evaluating test code for existing features.
Future features could be gated by test requirement.
The third role require community involvement and volunteering.
4/ rte_graph + DPDK scope
The new library rte_graph is accepted to be part of dpdk.org.
During the next meeting, it will be discussed whether
adding rte_graph and similar libraries in the main repository,
or splitting in different git repositories.
The question of possibly dead/unused code in some libraries
should be debated as well.
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v2 0/7] vfio/pci: SR-IOV support
2020-02-19 18:53 2% [dpdk-dev] [PATCH v2 0/7] vfio/pci: SR-IOV support Alex Williamson
2020-02-25 2:33 0% ` Tian, Kevin
@ 2020-03-05 6:38 0% ` Vamsi Krishna Attunuru
1 sibling, 0 replies; 200+ results
From: Vamsi Krishna Attunuru @ 2020-03-05 6:38 UTC (permalink / raw)
To: Alex Williamson, kvm
Cc: linux-pci, linux-kernel, dev, mtosatti, thomas, bluca,
jerinjacobk, bruce.richardson, cohuck
> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Alex Williamson
> Sent: Thursday, February 20, 2020 12:24 AM
> To: kvm@vger.kernel.org
> Cc: linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; dev@dpdk.org;
> mtosatti@redhat.com; thomas@monjalon.net; bluca@debian.org;
> jerinjacobk@gmail.com; bruce.richardson@intel.com; cohuck@redhat.com
> Subject: [dpdk-dev] [PATCH v2 0/7] vfio/pci: SR-IOV support
>
> Changes since v1 are primarily to patch 3/7 where the commit log is
> rewritten, along with option parsing and failure logging based on upstream
> discussions. The primary user visible difference is that option parsing is now
> much more strict. If a vf_token option is provided that cannot be used, we
> generate an error. As a result of this, opening a PF with a vf_token option
> will serve as a mechanism of setting the vf_token. This seems like a more
> user friendly API than the alternative of sometimes requiring the option (VFs
> in use) and sometimes rejecting it, and upholds our desire that the option is
> always either used or rejected.
>
> This also means that the VFIO_DEVICE_FEATURE ioctl is not the only means
> of setting the VF token, which might call into question whether we absolutely
> need this new ioctl. Currently I'm keeping it because I can imagine use cases,
> for example if a hypervisor were to support SR-IOV, the PF device might be
> opened without consideration for a VF token and we'd require the
> hypservisor to close and re-open the PF in order to set a known VF token,
> which is impractical.
>
> Series overview (same as provided with v1):
>
> The synopsis of this series is that we have an ongoing desire to drive PCIe SR-
> IOV PFs from userspace with VFIO. There's an immediate need for this with
> DPDK drivers and potentially interesting future use cases in virtualization.
> We've been reluctant to add this support previously due to the dependency
> and trust relationship between the VF device and PF driver. Minimally the PF
> driver can induce a denial of service to the VF, but depending on the specific
> implementation, the PF driver might also be responsible for moving data
> between VFs or have direct access to the state of the VF, including data or
> state otherwise private to the VF or VF driver.
>
> To help resolve these concerns, we introduce a VF token into the VFIO PCI
> ABI, which acts as a shared secret key between drivers. The userspace PF
> driver is required to set the VF token to a known value and userspace VF
> drivers are required to provide the token to access the VF device. If a PF
> driver is restarted with VF drivers in use, it must also provide the current
> token in order to prevent a rogue untrusted PF driver from replacing a known
> driver. The degree to which this new token is considered secret is left to the
> userspace drivers, the kernel intentionally provides no means to retrieve the
> current token.
>
> Note that the above token is only required for this new model where both
> the PF and VF devices are usable through vfio-pci. Existing models of VFIO
> drivers where the PF is used without SR-IOV enabled or the VF is bound to a
> userspace driver with an in-kernel, host PF driver are unaffected.
>
> The latter configuration above also highlights a new inverted scenario that is
> now possible, a userspace PF driver with in-kernel VF drivers.
> I believe this is a scenario that should be allowed, but should not be enabled
> by default. This series includes code to set a default driver_override for VFs
> sourced from a vfio-pci user owned PF, such that the VFs are also bound to
> vfio-pci. This model is compatible with tools like driverctl and allows the
> system administrator to decide if other bindings should be enabled. The VF
> token interface above exists only between vfio-pci PF and VF drivers, once a
> VF is bound to another driver, the administrator has effectively pronounced
> the device as trusted. The vfio-pci driver will note alternate binding in dmesg
> for logging and debugging purposes.
>
> Please review, comment, and test. The example QEMU implementation
> provided with the RFC is still current for this version. Thanks,
>
> Alex
Hi Alex,
Thanks for enabling this feature support.
Tested-by: Vamsi Attunuru <vattunuru@marvell.com>
Tested v2 patch set with below DPDK patch.
http://patches.dpdk.org/patch/66281/
Regards
A Vamsi
>
> RFC: https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__lore.kernel.org_lkml_158085337582.9445.17682266437583505502.stgit-
> 40gimli.home_&d=DwICaQ&c=nKjWec2b6R0mOyPaz7xtfQ&r=2rpxxNF2qeP0
> 2gVZIWTVrW-6zNZz5-uKt9pRqpR_M3U&m=V-6mKmCTHPZa5jwepXU_-
> Ma1_BGF0OWJ_IRCF_p4GVo&s=YnO98PGK9ro7F6_XZTccHdYcZ-
> rMMOin0nRFhPD6Uv4&e=
> v1: https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__lore.kernel.org_lkml_158145472604.16827.15751375540102298130.stgit
> -
> 40gimli.home_&d=DwICaQ&c=nKjWec2b6R0mOyPaz7xtfQ&r=2rpxxNF2qeP0
> 2gVZIWTVrW-6zNZz5-uKt9pRqpR_M3U&m=V-6mKmCTHPZa5jwepXU_-
> Ma1_BGF0OWJ_IRCF_p4GVo&s=rvUxLCENwNk0GBYkcsBVVobsLfMb4BV5gtc
> 3VqYQTS4&e=
>
> ---
>
> Alex Williamson (7):
> vfio: Include optional device match in vfio_device_ops callbacks
> vfio/pci: Implement match ops
> vfio/pci: Introduce VF token
> vfio: Introduce VFIO_DEVICE_FEATURE ioctl and first user
> vfio/pci: Add sriov_configure support
> vfio/pci: Remove dev_fmt definition
> vfio/pci: Cleanup .probe() exit paths
>
>
> drivers/vfio/pci/vfio_pci.c | 383
> +++++++++++++++++++++++++++++++++--
> drivers/vfio/pci/vfio_pci_private.h | 10 +
> drivers/vfio/vfio.c | 20 +-
> include/linux/vfio.h | 4
> include/uapi/linux/vfio.h | 37 +++
> 5 files changed, 426 insertions(+), 28 deletions(-)
^ permalink raw reply [relevance 0%]
* [dpdk-dev] [PATCH v3] doc: plan splitting the ethdev ops struct
2020-02-25 12:44 3% ` [dpdk-dev] [PATCH v2] " Ferruh Yigit
2020-02-25 15:51 3% ` Andrew Rybchenko
2020-02-25 18:13 0% ` David Marchand
@ 2020-03-04 9:57 4% ` Ferruh Yigit
2 siblings, 0 replies; 200+ results
From: Ferruh Yigit @ 2020-03-04 9:57 UTC (permalink / raw)
To: Neil Horman, John McNamara, Marko Kovacevic
Cc: dev, Ferruh Yigit, Jerin Jacob Kollanukkaran, David Marchand,
Thomas Monjalon, Andrew Rybchenko
For the ABI compatibility it is better to hide internal data structures
from the application as much as possible. But because of some inline
functions 'struct eth_dev_ops' can't be hidden completely.
Plan is to split the 'struct eth_dev_ops' into two as ones used by
inline functions and ones not used, and hide the second part that not
used by inline functions completely to the application.
Because of ABI break the work will be done in 20.11
Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
---
Cc: David Marchand <david.marchand@redhat.com>
Cc: Thomas Monjalon <thomas@monjalon.net>
Cc: Andrew Rybchenko <arybchenko@solarflare.com>
v2:
* Add target date for the work
* Give more detail on what will be done
v3:
* Drop the interim implementation, and target 20.11 with ABI break.
---
doc/guides/rel_notes/deprecation.rst | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
index 1339f54f5..0bb252f71 100644
--- a/doc/guides/rel_notes/deprecation.rst
+++ b/doc/guides/rel_notes/deprecation.rst
@@ -93,6 +93,15 @@ Deprecation Notices
In 19.11 PMDs will still update the field even when the offload is not
enabled.
+* ethdev: Split the ``struct eth_dev_ops`` struct to hide it as much as possible
+ will be done in 20.11.
+ Currently the ``struct eth_dev_ops`` struct is accessible by the application
+ because some inline functions, like ``rte_eth_tx_descriptor_status()``,
+ access the struct directly.
+ The struct will be separate in two, the ops used by inline functions will be moved
+ next to Rx/Tx burst functions, rest of the ``struct eth_dev_ops`` struct will be
+ moved to header file for drivers to hide it from applications.
+
* cryptodev: support for using IV with all sizes is added, J0 still can
be used but only when IV length in following structs ``rte_crypto_auth_xform``,
``rte_crypto_aead_xform`` is set to zero. When IV length is greater or equal
--
2.24.1
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH 00/16] NXP DPAAx fixes and enhancements
@ 2020-03-02 13:01 3% ` David Marchand
2020-03-05 9:06 3% ` Hemant Agrawal (OSS)
1 sibling, 1 reply; 200+ results
From: David Marchand @ 2020-03-02 13:01 UTC (permalink / raw)
To: Hemant Agrawal; +Cc: Yigit, Ferruh, dev
On Mon, Mar 2, 2020 at 10:26 AM Hemant Agrawal <hemant.agrawal@nxp.com> wrote:
>
> This patch series add various patches for enhancing and fixing NXP
> fslmc bus, dpaa bus, and dpaax.
>
> - the main change is support to allow thread migration across lcores
> - improving the multi-process support
This series triggers an ABI warning that must be investigated.
https://travis-ci.com/ovsrobot/dpdk/jobs/292904119#L2233
--
David Marchand
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [dpdk v1] net/mlx5: fix possible building error
2020-03-02 7:35 0% ` Ali Alnubani
@ 2020-03-02 8:13 0% ` Tonghao Zhang
0 siblings, 0 replies; 200+ results
From: Tonghao Zhang @ 2020-03-02 8:13 UTC (permalink / raw)
To: Ali Alnubani; +Cc: stable, Matan Azrad, dev
On Mon, Mar 2, 2020 at 3:35 PM Ali Alnubani <alialnu@mellanox.com> wrote:
>
> > -----Original Message-----
> > From: Tonghao Zhang <xiangxia.m.yue@gmail.com>
> > Sent: Monday, March 2, 2020 2:23 AM
> > To: Ali Alnubani <alialnu@mellanox.com>
> > Cc: stable@dpdk.org; Matan Azrad <matan@mellanox.com>; dev@dpdk.org
> > Subject: Re: [dpdk-dev] [dpdk v1] net/mlx5: fix possible building error
> >
> > On Mon, Mar 2, 2020 at 1:09 AM Ali Alnubani <alialnu@mellanox.com>
> > wrote:
> > >
> > > Hi Tonghao,
> > >
> > > > -----Original Message-----
> > > > From: dev <dev-bounces@dpdk.org> On Behalf Of
> > > > xiangxia.m.yue@gmail.com
> > > > Sent: Saturday, February 29, 2020 12:27 PM
> > > > To: Matan Azrad <matan@mellanox.com>; dev@dpdk.org
> > > > Cc: Tonghao Zhang <xiangxia.m.yue@gmail.com>; stable@dpdk.org
> > > > Subject: [dpdk-dev] [dpdk v1] net/mlx5: fix possible building error
> > > >
> > > > From: Tonghao Zhang <xiangxia.m.yue@gmail.com>
> > > >
> > > > When setting the CONFIG_RTE_BUILD_SHARED_LIB to y, and build the
> > > > mlx5 pmd, there is a building error.
> > > > To fix it, add RTE_IBVERBS_LINK_DLOPEN to include relative codes.
> > > >
> > > > > mlx5_common.o: In function `mlx5_glue_init':
> > > > > drivers/common/mlx5/mlx5_common.c:324: undefined reference to
> > > > `dlclose'
> > > >
> > > > Fixes: 7b4f1e6bd367 ("common/mlx5: introduce common library")
> > > > Cc: stable@dpdk.org
> > > >
> > > > Signed-off-by: Tonghao Zhang <xiangxia.m.yue@gmail.com>
> > > > ---
> > > > drivers/common/mlx5/mlx5_common.c | 5 +++--
> > > > 1 file changed, 3 insertions(+), 2 deletions(-)
> > > >
> > >
> > > I test this case on multiple environments and don't see the build failure.
> > Can you please provide more information about yours, and how to
> > reproduce this?
> > Hi Ali,
> > I use the dpdk-next-net, and the last commit id is
> > d7142fbae16f185e11bfa44be061399afc40a1be.
> > The build config is show as below:
> >
> > diff --git a/config/common_base b/config/common_base index
> > 7ca2f28..4bd5126 100644
> > --- a/config/common_base
> > +++ b/config/common_base
> > @@ -57,7 +57,7 @@ CONFIG_RTE_ENABLE_LTO=n # # Compile to share
> > library # -CONFIG_RTE_BUILD_SHARED_LIB=n
> > +CONFIG_RTE_BUILD_SHARED_LIB=y
> >
> > #
> > # Use newest code breaking previous ABI @@ -363,7 +363,7 @@
> > CONFIG_RTE_LIBRTE_MLX4_DEBUG=n # Compile burst-oriented Mellanox
> > ConnectX-4, ConnectX-5, # ConnectX-6 & BlueField (MLX5) PMD # -
> > CONFIG_RTE_LIBRTE_MLX5_PMD=n
> > +CONFIG_RTE_LIBRTE_MLX5_PMD=y
> > CONFIG_RTE_LIBRTE_MLX5_DEBUG=n
> >
> > #
> >
> > The building failure log show:
> > LD librte_common_mlx5.so.20.0.1
> > mlx5_common.o: In function `mlx5_glue_init':
> > /root/zth/myshell/dpdk-next-
> > net/drivers/common/mlx5/mlx5_common.c:324:
> > undefined reference to `dlclose'
> > collect2: error: ld returned 1 exit status
> >
>
> Do you mind sharing which OS are you using, which gcc/clang version, whether you have MLNX_OFED or rdma-core installed, and which version?
1. CentOS Linux release 7.6.1810 (Core)
2. kernel 3.10.0-957.1.3.el7.x86_64
3. OFED 4.7-3.2.9
4. FW 16.26.4012
5. gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-36)
6. build the dpdk with x86_64-native-linuxapp-gcc
> Thanks,
> Ali
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [dpdk v1] net/mlx5: fix possible building error
2020-03-02 0:22 3% ` Tonghao Zhang
@ 2020-03-02 7:35 0% ` Ali Alnubani
2020-03-02 8:13 0% ` Tonghao Zhang
0 siblings, 1 reply; 200+ results
From: Ali Alnubani @ 2020-03-02 7:35 UTC (permalink / raw)
To: Tonghao Zhang; +Cc: stable, Matan Azrad, dev
> -----Original Message-----
> From: Tonghao Zhang <xiangxia.m.yue@gmail.com>
> Sent: Monday, March 2, 2020 2:23 AM
> To: Ali Alnubani <alialnu@mellanox.com>
> Cc: stable@dpdk.org; Matan Azrad <matan@mellanox.com>; dev@dpdk.org
> Subject: Re: [dpdk-dev] [dpdk v1] net/mlx5: fix possible building error
>
> On Mon, Mar 2, 2020 at 1:09 AM Ali Alnubani <alialnu@mellanox.com>
> wrote:
> >
> > Hi Tonghao,
> >
> > > -----Original Message-----
> > > From: dev <dev-bounces@dpdk.org> On Behalf Of
> > > xiangxia.m.yue@gmail.com
> > > Sent: Saturday, February 29, 2020 12:27 PM
> > > To: Matan Azrad <matan@mellanox.com>; dev@dpdk.org
> > > Cc: Tonghao Zhang <xiangxia.m.yue@gmail.com>; stable@dpdk.org
> > > Subject: [dpdk-dev] [dpdk v1] net/mlx5: fix possible building error
> > >
> > > From: Tonghao Zhang <xiangxia.m.yue@gmail.com>
> > >
> > > When setting the CONFIG_RTE_BUILD_SHARED_LIB to y, and build the
> > > mlx5 pmd, there is a building error.
> > > To fix it, add RTE_IBVERBS_LINK_DLOPEN to include relative codes.
> > >
> > > > mlx5_common.o: In function `mlx5_glue_init':
> > > > drivers/common/mlx5/mlx5_common.c:324: undefined reference to
> > > `dlclose'
> > >
> > > Fixes: 7b4f1e6bd367 ("common/mlx5: introduce common library")
> > > Cc: stable@dpdk.org
> > >
> > > Signed-off-by: Tonghao Zhang <xiangxia.m.yue@gmail.com>
> > > ---
> > > drivers/common/mlx5/mlx5_common.c | 5 +++--
> > > 1 file changed, 3 insertions(+), 2 deletions(-)
> > >
> >
> > I test this case on multiple environments and don't see the build failure.
> Can you please provide more information about yours, and how to
> reproduce this?
> Hi Ali,
> I use the dpdk-next-net, and the last commit id is
> d7142fbae16f185e11bfa44be061399afc40a1be.
> The build config is show as below:
>
> diff --git a/config/common_base b/config/common_base index
> 7ca2f28..4bd5126 100644
> --- a/config/common_base
> +++ b/config/common_base
> @@ -57,7 +57,7 @@ CONFIG_RTE_ENABLE_LTO=n # # Compile to share
> library # -CONFIG_RTE_BUILD_SHARED_LIB=n
> +CONFIG_RTE_BUILD_SHARED_LIB=y
>
> #
> # Use newest code breaking previous ABI @@ -363,7 +363,7 @@
> CONFIG_RTE_LIBRTE_MLX4_DEBUG=n # Compile burst-oriented Mellanox
> ConnectX-4, ConnectX-5, # ConnectX-6 & BlueField (MLX5) PMD # -
> CONFIG_RTE_LIBRTE_MLX5_PMD=n
> +CONFIG_RTE_LIBRTE_MLX5_PMD=y
> CONFIG_RTE_LIBRTE_MLX5_DEBUG=n
>
> #
>
> The building failure log show:
> LD librte_common_mlx5.so.20.0.1
> mlx5_common.o: In function `mlx5_glue_init':
> /root/zth/myshell/dpdk-next-
> net/drivers/common/mlx5/mlx5_common.c:324:
> undefined reference to `dlclose'
> collect2: error: ld returned 1 exit status
>
Do you mind sharing which OS are you using, which gcc/clang version, whether you have MLNX_OFED or rdma-core installed, and which version?
Thanks,
Ali
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [dpdk v1] net/mlx5: fix possible building error
@ 2020-03-02 0:22 3% ` Tonghao Zhang
2020-03-02 7:35 0% ` Ali Alnubani
0 siblings, 1 reply; 200+ results
From: Tonghao Zhang @ 2020-03-02 0:22 UTC (permalink / raw)
To: Ali Alnubani; +Cc: stable, Matan Azrad, dev
On Mon, Mar 2, 2020 at 1:09 AM Ali Alnubani <alialnu@mellanox.com> wrote:
>
> Hi Tonghao,
>
> > -----Original Message-----
> > From: dev <dev-bounces@dpdk.org> On Behalf Of
> > xiangxia.m.yue@gmail.com
> > Sent: Saturday, February 29, 2020 12:27 PM
> > To: Matan Azrad <matan@mellanox.com>; dev@dpdk.org
> > Cc: Tonghao Zhang <xiangxia.m.yue@gmail.com>; stable@dpdk.org
> > Subject: [dpdk-dev] [dpdk v1] net/mlx5: fix possible building error
> >
> > From: Tonghao Zhang <xiangxia.m.yue@gmail.com>
> >
> > When setting the CONFIG_RTE_BUILD_SHARED_LIB to y, and build the mlx5
> > pmd, there is a building error.
> > To fix it, add RTE_IBVERBS_LINK_DLOPEN to include relative codes.
> >
> > > mlx5_common.o: In function `mlx5_glue_init':
> > > drivers/common/mlx5/mlx5_common.c:324: undefined reference to
> > `dlclose'
> >
> > Fixes: 7b4f1e6bd367 ("common/mlx5: introduce common library")
> > Cc: stable@dpdk.org
> >
> > Signed-off-by: Tonghao Zhang <xiangxia.m.yue@gmail.com>
> > ---
> > drivers/common/mlx5/mlx5_common.c | 5 +++--
> > 1 file changed, 3 insertions(+), 2 deletions(-)
> >
>
> I test this case on multiple environments and don't see the build failure. Can you please provide more information about yours, and how to reproduce this?
Hi Ali,
I use the dpdk-next-net, and the last commit id is
d7142fbae16f185e11bfa44be061399afc40a1be.
The build config is show as below:
diff --git a/config/common_base b/config/common_base
index 7ca2f28..4bd5126 100644
--- a/config/common_base
+++ b/config/common_base
@@ -57,7 +57,7 @@ CONFIG_RTE_ENABLE_LTO=n
#
# Compile to share library
#
-CONFIG_RTE_BUILD_SHARED_LIB=n
+CONFIG_RTE_BUILD_SHARED_LIB=y
#
# Use newest code breaking previous ABI
@@ -363,7 +363,7 @@ CONFIG_RTE_LIBRTE_MLX4_DEBUG=n
# Compile burst-oriented Mellanox ConnectX-4, ConnectX-5,
# ConnectX-6 & BlueField (MLX5) PMD
#
-CONFIG_RTE_LIBRTE_MLX5_PMD=n
+CONFIG_RTE_LIBRTE_MLX5_PMD=y
CONFIG_RTE_LIBRTE_MLX5_DEBUG=n
#
The building failure log show:
LD librte_common_mlx5.so.20.0.1
mlx5_common.o: In function `mlx5_glue_init':
/root/zth/myshell/dpdk-next-net/drivers/common/mlx5/mlx5_common.c:324:
undefined reference to `dlclose'
collect2: error: ld returned 1 exit status
> Thanks,
> Ali
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [RFC 0/6] New sync modes for ring
2020-02-27 10:31 0% ` Jerin Jacob
@ 2020-02-28 0:17 0% ` David Christensen
0 siblings, 0 replies; 200+ results
From: David Christensen @ 2020-02-28 0:17 UTC (permalink / raw)
To: Jerin Jacob, Ananyev, Konstantin
Cc: Stephen Hemminger, dpdk-dev, Olivier Matz
> On Tue, Feb 25, 2020 at 7:11 PM Ananyev, Konstantin
> <konstantin.ananyev@intel.com> wrote:
>
>> We do have a run-time check in our current enqueue()/dequeue implementation.
>> In fact we support both modes: we have generic rte_ring_enqueue(/dequeue)_bulk(/burst)
>> where sync behaviour is determined at runtime by value of prod(/cons).single.
>> Or user can call rte_ring_(mp/sp)_enqueue_* functions directly.
>> This RFC follows exactly the same paradigm:
>> rte_ring_enqueue(/dequeue)_bulk(/burst) kept generic and it's
>> behaviour is determined at runtime, by value of prod(/cons).sync_type.
>> Or user can call enqueue/dequeue with particular sync mode directly:
>> rte_ring_(mp/sp/rts/hts)_enqueue_(bulk/burst)*.
>> The only thing that changed:
>> Format of prod/cons now could differ depending on mode selected at _init_.
>> So you can't create a ring for let say SP mode and then in the middle of data-path
>> change your mind and start using MP_RTS mode.
>> For existing modes (SP/MP, SC/MC) format remains the same and user can still
>> use them interchangeably, though of course that is an error prone practice.
>
> Makes sense.
>
>
>>
>>>> But I agree with the problem statement that in the virtualization use
>>>> case, It may be possible to have N virtual cores runs on a physical
>>>> core.
>>>>
>>>> IMO, The best solution would be keeping the ring API same and have a
>>>> different flavor in "compile-time". Something like
>>>> liburcu did for accommodating different flavors.
>>>>
>>>> i.e urcu-qsbr.h and urcu-bp.h will identical definition of API. The
>>>> application can simply include ONE header file in a C file based on
>>>> the flavor.
>>
>> I don't think it is a flexible enough approach.
>> In one app user might need to have several rings with different sync modes.
>> Or even user might need a ring with different sync modes for enqueue/dequeue.
>
> Ack.
>
>
>> Yes, hiding rte_ring implementation inside .c would help a lot
>> in terms of ABI maintenance and would make our future life easier.
>> The question is what is the price for it in terms of performance,
>> and are we ready to pay it. Not to mention that it would cause
>> changes in many other libs/apps...
>> So I think it should be a subject for a separate discussion.
>> But, agree it would be good at least to measure the performance
>> impact of such change.
>> If I'll have some spare cycles, will give it a try.
>> Meanwhile, can I ask Jerin and other guys to repeat tests from this RFC
>> on their HW? Before continuing discussion would probably be good to know
>> does the suggested patch work as expected across different platforms.
>
>
> I tested on an arm64 HW. The former section is without the
> patch(20.02) and later one with this patch.
> I agree with Konstantin that getting more platform tests will be good
> early so that we can focus on the approach
> to avoid back and forth latter.
>
>
> RTE>>ring_perf_autotest // without path
>
> ### Testing single element enq/deq ###
> legacy APIs: SP/SC: single: 289.78
> legacy APIs: MP/MC: single: 516.20
>
> ### Testing burst enq/deq ###
> legacy APIs: SP/SC: burst (size: 8): 312.88
> legacy APIs: SP/SC: burst (size: 32): 426.72
> legacy APIs: MP/MC: burst (size: 8): 510.95
> legacy APIs: MP/MC: burst (size: 32): 702.01
>
> ### Testing bulk enq/deq ###
> legacy APIs: SP/SC: bulk (size: 8): 306.74
> legacy APIs: SP/SC: bulk (size: 32): 411.56
> legacy APIs: MP/MC: bulk (size: 8): 501.32
> legacy APIs: MP/MC: bulk (size: 32): 693.07
>
> ### Testing empty bulk deq ###
> legacy APIs: SP/SC: bulk (size: 8): 7.00
> legacy APIs: MP/MC: bulk (size: 8): 7.00
>
> ### Testing using two physical cores ###
> legacy APIs: SP/SC: bulk (size: 8): 74.36
> legacy APIs: MP/MC: bulk (size: 8): 110.18
> legacy APIs: SP/SC: bulk (size: 32): 23.04
> legacy APIs: MP/MC: bulk (size: 32): 32.29
>
> ### Testing using all slave nodes ##
> Bulk enq/dequeue count on size 8
> Core [8] count = 293741
> Core [9] count = 293741
> Total count (size: 8): 587482
>
> Bulk enq/dequeue count on size 32
> Core [8] count = 244909
> Core [9] count = 244909
> Total count (size: 32): 1077300
>
> ### Testing single element enq/deq ###
> elem APIs: element size 16B: SP/SC: single: 255.37
> elem APIs: element size 16B: MP/MC: single: 456.68
>
> ### Testing burst enq/deq ###
> elem APIs: element size 16B: SP/SC: burst (size: 8): 291.99
> elem APIs: element size 16B: SP/SC: burst (size: 32): 456.25
> elem APIs: element size 16B: MP/MC: burst (size: 8): 497.77
> elem APIs: element size 16B: MP/MC: burst (size: 32): 680.87
>
> ### Testing bulk enq/deq ###
> elem APIs: element size 16B: SP/SC: bulk (size: 8): 284.40
> elem APIs: element size 16B: SP/SC: bulk (size: 32): 453.17
> elem APIs: element size 16B: MP/MC: bulk (size: 8): 485.77
> elem APIs: element size 16B: MP/MC: bulk (size: 32): 675.08
>
> ### Testing empty bulk deq ###
> elem APIs: element size 16B: SP/SC: bulk (size: 8): 8.00
> elem APIs: element size 16B: MP/MC: bulk (size: 8): 7.00
>
> ### Testing using two physical cores ###
> elem APIs: element size 16B: SP/SC: bulk (size: 8): 74.45
> elem APIs: element size 16B: MP/MC: bulk (size: 8): 105.91
> elem APIs: element size 16B: SP/SC: bulk (size: 32): 22.92
> elem APIs: element size 16B: MP/MC: bulk (size: 32): 31.55
>
> ### Testing using all slave nodes ###
>
> Bulk enq/dequeue count on size 8
> Core [8] count = 308724
> Core [9] count = 308723
> Total count (size: 8): 617447
>
> Bulk enq/dequeue count on size 32
> Core [8] count = 214269
> Core [9] count = 214269
> Total count (size: 32): 1045985
>
> RTE>>ring_perf_autotest // with patch
>
> ### Testing single element enq/deq ###
> legacy APIs: SP/SC: single: 289.78
> legacy APIs: MP/MC: single: 475.76
>
> ### Testing burst enq/deq ###
> legacy APIs: SP/SC: burst (size: 8): 323.91
> legacy APIs: SP/SC: burst (size: 32): 424.60
> legacy APIs: MP/MC: burst (size: 8): 523.00
> legacy APIs: MP/MC: burst (size: 32): 717.09
>
> ### Testing bulk enq/deq ###
> legacy APIs: SP/SC: bulk (size: 8): 317.74
> legacy APIs: SP/SC: bulk (size: 32): 413.57
> legacy APIs: MP/MC: bulk (size: 8): 512.89
> legacy APIs: MP/MC: bulk (size: 32): 712.45
>
> ### Testing empty bulk deq ###
> legacy APIs: SP/SC: bulk (size: 8): 7.00
> legacy APIs: MP/MC: bulk (size: 8): 7.00
>
> ### Testing using two physical cores ###
> legacy APIs: SP/SC: bulk (size: 8): 74.82
> legacy APIs: MP/MC: bulk (size: 8): 96.45
> legacy APIs: SP/SC: bulk (size: 32): 22.97
> legacy APIs: MP/MC: bulk (size: 32): 32.52
>
> ### Testing using all slave nodes ###
>
> Bulk enq/dequeue count on size 8
> Core [8] count = 283928
> Core [9] count = 283927
> Total count (size: 8): 567855
>
> Bulk enq/dequeue count on size 32
> Core [8] count = 223916
> Core [9] count = 223915
> Total count (size: 32): 1015686
>
> ### Testing single element enq/deq ###
> elem APIs: element size 16B: SP/SC: single: 267.65
> elem APIs: element size 16B: MP/MC: single: 439.06
>
> ### Testing burst enq/deq ###
> elem APIs: element size 16B: SP/SC: burst (size: 8): 302.44
> elem APIs: element size 16B: SP/SC: burst (size: 32): 466.31
> elem APIs: element size 16B: MP/MC: burst (size: 8): 502.51
> elem APIs: element size 16B: MP/MC: burst (size: 32): 695.81
>
> ### Testing bulk enq/deq ###
> elem APIs: element size 16B: SP/SC: bulk (size: 8): 295.15
> elem APIs: element size 16B: SP/SC: bulk (size: 32): 462.77
> elem APIs: element size 16B: MP/MC: bulk (size: 8): 496.89
> elem APIs: element size 16B: MP/MC: bulk (size: 32): 690.46
>
> ### Testing empty bulk deq ###
> elem APIs: element size 16B: SP/SC: bulk (size: 8): 7.50
> elem APIs: element size 16B: MP/MC: bulk (size: 8): 7.44
>
> ### Testing using two physical cores ###
> elem APIs: element size 16B: SP/SC: bulk (size: 8): 65.85
> elem APIs: element size 16B: MP/MC: bulk (size: 8): 103.80
> elem APIs: element size 16B: SP/SC: bulk (size: 32): 23.27
> elem APIs: element size 16B: MP/MC: bulk (size: 32): 31.17
>
> ### Testing using all slave nodes ###
>
> Bulk enq/dequeue count on size 8
> Core [8] count = 304223
> Core [9] count = 304221
> Total count (size: 8): 608444
>
> Bulk enq/dequeue count on size 32
> Core [8] count = 214856
> Core [9] count = 214855
> Total count (size: 32): 1038155
> Test OK
> RTE>>quit
>
>
Encountered a couple of different build errors with these patches on my
Power 9 system:
In file included from ../lib/librte_ring/rte_ring.h:534,
from ../drivers/mempool/ring/rte_mempool_ring.c:9:
../lib/librte_ring/rte_ring_hts_generic.h: In function
‘__rte_ring_hts_update_tail’:
../lib/librte_ring/rte_ring_hts_generic.h:61:2: warning: implicit
declaration of function ‘RTE_ASSERT’; did you mean ‘RTE_STR’?
[-Wimplicit-function-declaration]
RTE_ASSERT(n >= num);
^~~~~~~~~~
RTE_STR
Fixed by adding "#include <rte_debug.h>" to rte_ring.h.
Also encountered:
In file included from ../app/test/test_ring_hts_stress.c:5:
../app/test/test_ring_stress.h: In function ‘check_updt_elem’:
../app/test/test_ring_stress.h:162:9: error: unknown type name
‘rte_spinlock_t’
static rte_spinlock_t dump_lock;
^~~~~~~~~~~~~~
../app/test/test_ring_stress.h:166:4: warning: implicit declaration of
function ‘rte_spinlock_lock’; did you mean ‘rte_calloc_socket’?
[-Wimplicit-function-declaration]
rte_spinlock_lock(&dump_lock);
^~~~~~~~~~~~~~~~~
rte_calloc_socket
../app/test/test_ring_stress.h:166:4: warning: nested extern declaration
of ‘rte_spinlock_lock’ [-Wnested-externs]
../app/test/test_ring_stress.h:172:4: warning: implicit declaration of
function ‘rte_spinlock_unlock’; did you mean ‘pthread_rwlock_unlock’?
[-Wimplicit-function-declaration]
rte_spinlock_unlock(&dump_lock);
^~~~~~~~~~~~~~~~~~~
pthread_rwlock_unlock
../app/test/test_ring_stress.h:172:4: warning: nested extern declaration
of ‘rte_spinlock_unlock’ [-Wnested-externs]
Fixed by adding "#include <rte_spinlock.h>" to test_ring_stress.h.
Autoperf test results
---------------------
RTE>>ring_perf_autotest // DPDK 20.02, without patch
### Testing single element enq/deq ###
legacy APIs: SP/SC: single: 42.14
legacy APIs: MP/MC: single: 56.26
### Testing burst enq/deq ###
legacy APIs: SP/SC: burst (size: 8): 43.59
legacy APIs: SP/SC: burst (size: 32): 49.87
legacy APIs: MP/MC: burst (size: 8): 58.43
legacy APIs: MP/MC: burst (size: 32): 65.68
### Testing bulk enq/deq ###
legacy APIs: SP/SC: bulk (size: 8): 43.59
legacy APIs: SP/SC: bulk (size: 32): 49.85
legacy APIs: MP/MC: bulk (size: 8): 58.43
legacy APIs: MP/MC: bulk (size: 32): 65.60
### Testing empty bulk deq ###
legacy APIs: SP/SC: bulk (size: 8): 7.16
legacy APIs: MP/MC: bulk (size: 8): 7.16
### Testing using two hyperthreads ###
legacy APIs: SP/SC: bulk (size: 8): 12.46
legacy APIs: MP/MC: bulk (size: 8): 16.20
legacy APIs: SP/SC: bulk (size: 32): 3.21
legacy APIs: MP/MC: bulk (size: 32): 3.73
### Testing using two physical cores ###
legacy APIs: SP/SC: bulk (size: 8): 33.34
legacy APIs: MP/MC: bulk (size: 8): 37.99
legacy APIs: SP/SC: bulk (size: 32): 10.19
legacy APIs: MP/MC: bulk (size: 32): 11.90
### Testing using two NUMA nodes ###
legacy APIs: SP/SC: bulk (size: 8): 49.50
legacy APIs: MP/MC: bulk (size: 8): 63.65
legacy APIs: SP/SC: bulk (size: 32): 12.49
legacy APIs: MP/MC: bulk (size: 32): 23.53
### Testing using all slave nodes ###
Bulk enq/dequeue count on size 8
Core [4] count = 5604
Core [5] count = 5563
Core [6] count = 5576
Core [7] count = 5630
Core [8] count = 5643
Core [9] count = 5727
Core [10] count = 5698
Core [11] count = 5711
Core [64] count = 5259
Core [65] count = 5322
Core [66] count = 5321
Core [67] count = 5310
Core [68] count = 4350
Core [69] count = 4455
Core [70] count = 4546
Core [71] count = 4475
Total count (size: 8): 84190
Bulk enq/dequeue count on size 32
Core [4] count = 5543
Core [5] count = 5555
Core [6] count = 5596
Core [7] count = 5584
Core [8] count = 5613
Core [9] count = 5686
Core [10] count = 5689
Core [11] count = 5677
Core [64] count = 5228
Core [65] count = 5389
Core [66] count = 5406
Core [67] count = 5359
Core [68] count = 4554
Core [69] count = 4673
Core [70] count = 4675
Core [71] count = 4644
Total count (size: 32): 169061
### Testing single element enq/deq ###
elem APIs: element size 16B: SP/SC: single: 42.84
elem APIs: element size 16B: MP/MC: single: 56.77
### Testing burst enq/deq ###
elem APIs: element size 16B: SP/SC: burst (size: 8): 44.98
elem APIs: element size 16B: SP/SC: burst (size: 32): 59.16
elem APIs: element size 16B: MP/MC: burst (size: 8): 60.58
elem APIs: element size 16B: MP/MC: burst (size: 32): 74.75
### Testing bulk enq/deq ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 45.01
elem APIs: element size 16B: SP/SC: bulk (size: 32): 59.08
elem APIs: element size 16B: MP/MC: bulk (size: 8): 60.58
elem APIs: element size 16B: MP/MC: bulk (size: 32): 74.76
### Testing empty bulk deq ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 7.16
elem APIs: element size 16B: MP/MC: bulk (size: 8): 7.16
### Testing using two hyperthreads ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 12.18
elem APIs: element size 16B: MP/MC: bulk (size: 8): 15.44
elem APIs: element size 16B: SP/SC: bulk (size: 32): 3.22
elem APIs: element size 16B: MP/MC: bulk (size: 32): 3.97
### Testing using two physical cores ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 42.07
elem APIs: element size 16B: MP/MC: bulk (size: 8): 44.50
elem APIs: element size 16B: SP/SC: bulk (size: 32): 10.73
elem APIs: element size 16B: MP/MC: bulk (size: 32): 11.73
### Testing using two NUMA nodes ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 49.55
elem APIs: element size 16B: MP/MC: bulk (size: 8): 93.10
elem APIs: element size 16B: SP/SC: bulk (size: 32): 12.33
elem APIs: element size 16B: MP/MC: bulk (size: 32): 27.10
### Testing using all slave nodes ###
Bulk enq/dequeue count on size 8
Core [4] count = 5489
Core [5] count = 5559
Core [6] count = 5566
Core [7] count = 5577
Core [8] count = 5645
Core [9] count = 5699
Core [10] count = 5695
Core [11] count = 5733
Core [64] count = 5202
Core [65] count = 5284
Core [66] count = 5319
Core [67] count = 5349
Core [68] count = 4331
Core [69] count = 4465
Core [70] count = 4484
Core [71] count = 4439
Total count (size: 8): 83836
Bulk enq/dequeue count on size 32
Core [4] count = 5567
Core [5] count = 5492
Core [6] count = 5517
Core [7] count = 5515
Core [8] count = 5593
Core [9] count = 5650
Core [10] count = 5694
Core [11] count = 5665
Core [64] count = 5236
Core [65] count = 5319
Core [66] count = 5333
Core [67] count = 5304
Core [68] count = 4608
Core [69] count = 4669
Core [70] count = 4690
Core [71] count = 4654
Total count (size: 32): 168342
Test OK
---------------------
RTE>>ring_perf_autotest // DPDK 20.02, without patch
### Testing single element enq/deq ###
legacy APIs: SP/SC: single: 42.18
legacy APIs: MP/MC: single: 56.26
### Testing burst enq/deq ###
legacy APIs: SP/SC: burst (size: 8): 43.60
legacy APIs: SP/SC: burst (size: 32): 49.86
legacy APIs: MP/MC: burst (size: 8): 58.43
legacy APIs: MP/MC: burst (size: 32): 65.67
### Testing bulk enq/deq ###
legacy APIs: SP/SC: bulk (size: 8): 43.59
legacy APIs: SP/SC: bulk (size: 32): 49.86
legacy APIs: MP/MC: bulk (size: 8): 58.43
legacy APIs: MP/MC: bulk (size: 32): 65.63
### Testing empty bulk deq ###
legacy APIs: SP/SC: bulk (size: 8): 7.16
legacy APIs: MP/MC: bulk (size: 8): 7.16
### Testing using two hyperthreads ###
legacy APIs: SP/SC: bulk (size: 8): 12.07
legacy APIs: MP/MC: bulk (size: 8): 16.24
legacy APIs: SP/SC: bulk (size: 32): 3.20
legacy APIs: MP/MC: bulk (size: 32): 3.72
### Testing using two physical cores ###
legacy APIs: SP/SC: bulk (size: 8): 33.41
legacy APIs: MP/MC: bulk (size: 8): 38.01
legacy APIs: SP/SC: bulk (size: 32): 10.23
legacy APIs: MP/MC: bulk (size: 32): 11.90
### Testing using two NUMA nodes ###
legacy APIs: SP/SC: bulk (size: 8): 49.27
legacy APIs: MP/MC: bulk (size: 8): 64.80
legacy APIs: SP/SC: bulk (size: 32): 12.45
legacy APIs: MP/MC: bulk (size: 32): 23.11
### Testing using all slave nodes ###
Bulk enq/dequeue count on size 8
Core [4] count = 5637
Core [5] count = 5599
Core [6] count = 5623
Core [7] count = 5627
Core [8] count = 5723
Core [9] count = 5758
Core [10] count = 5714
Core [11] count = 5724
Core [64] count = 5310
Core [65] count = 5438
Core [66] count = 5448
Core [67] count = 5374
Core [68] count = 4441
Core [69] count = 4550
Core [70] count = 4550
Core [71] count = 4558
Total count (size: 8): 85074
Bulk enq/dequeue count on size 32
Core [4] count = 5608
Core [5] count = 5623
Core [6] count = 5590
Core [7] count = 5658
Core [8] count = 5680
Core [9] count = 5738
Core [10] count = 5692
Core [11] count = 5712
Core [64] count = 5273
Core [65] count = 5363
Core [66] count = 5341
Core [67] count = 5349
Core [68] count = 4591
Core [69] count = 4673
Core [70] count = 4698
Core [71] count = 4687
Total count (size: 32): 170350
### Testing single element enq/deq ###
elem APIs: element size 16B: SP/SC: single: 42.82
elem APIs: element size 16B: MP/MC: single: 56.79
### Testing burst enq/deq ###
elem APIs: element size 16B: SP/SC: burst (size: 8): 44.99
elem APIs: element size 16B: SP/SC: burst (size: 32): 59.00
elem APIs: element size 16B: MP/MC: burst (size: 8): 60.59
elem APIs: element size 16B: MP/MC: burst (size: 32): 74.78
### Testing bulk enq/deq ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 44.97
elem APIs: element size 16B: SP/SC: bulk (size: 32): 58.91
elem APIs: element size 16B: MP/MC: bulk (size: 8): 60.60
elem APIs: element size 16B: MP/MC: bulk (size: 32): 74.61
### Testing empty bulk deq ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 7.16
elem APIs: element size 16B: MP/MC: bulk (size: 8): 7.16
### Testing using two hyperthreads ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 12.18
elem APIs: element size 16B: MP/MC: bulk (size: 8): 15.41
elem APIs: element size 16B: SP/SC: bulk (size: 32): 3.19
elem APIs: element size 16B: MP/MC: bulk (size: 32): 4.06
### Testing using two physical cores ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 42.08
elem APIs: element size 16B: MP/MC: bulk (size: 8): 44.52
elem APIs: element size 16B: SP/SC: bulk (size: 32): 10.73
elem APIs: element size 16B: MP/MC: bulk (size: 32): 12.39
### Testing using two NUMA nodes ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 49.65
elem APIs: element size 16B: MP/MC: bulk (size: 8): 93.27
elem APIs: element size 16B: SP/SC: bulk (size: 32): 12.38
elem APIs: element size 16B: MP/MC: bulk (size: 32): 27.19
### Testing using all slave nodes ###
Bulk enq/dequeue count on size 8
Core [4] count = 5629
Core [5] count = 5585
Core [6] count = 5676
Core [7] count = 5604
Core [8] count = 5639
Core [9] count = 5731
Core [10] count = 5694
Core [11] count = 5707
Core [64] count = 5254
Core [65] count = 5331
Core [66] count = 5340
Core [67] count = 5355
Core [68] count = 4339
Core [69] count = 4481
Core [70] count = 4504
Core [71] count = 4507
Total count (size: 8): 84376
Bulk enq/dequeue count on size 32
Core [4] count = 5518
Core [5] count = 5493
Core [6] count = 5559
Core [7] count = 5484
Core [8] count = 5623
Core [9] count = 5669
Core [10] count = 5661
Core [11] count = 5658
Core [64] count = 5207
Core [65] count = 5305
Core [66] count = 5273
Core [67] count = 5303
Core [68] count = 4542
Core [69] count = 4682
Core [70] count = 4672
Core [71] count = 4609
Total count (size: 32): 168634
Test OK
Dave
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] Questions about rte_timer APIs
2020-02-26 16:57 0% ` Carrillo, Erik G
@ 2020-02-27 16:58 0% ` Honnappa Nagarahalli
0 siblings, 0 replies; 200+ results
From: Honnappa Nagarahalli @ 2020-02-27 16:58 UTC (permalink / raw)
To: Carrillo, Erik G, Robert Sanford
Cc: dev, nd, Phil Yang, Gavin Hu, david.marchand, thomas, nd, nd
Thanks Erik. We will create some patches and send them for review.
> -----Original Message-----
> From: Carrillo, Erik G <erik.g.carrillo@intel.com>
> Sent: Wednesday, February 26, 2020 10:58 AM
> To: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>; Robert
> Sanford <rsanford@akamai.com>
> Cc: dev <dev@dpdk.org>; nd <nd@arm.com>; Phil Yang
> <Phil.Yang@arm.com>; Gavin Hu <Gavin.Hu@arm.com>;
> david.marchand@redhat.com; thomas@monjalon.net; nd <nd@arm.com>
> Subject: RE: Questions about rte_timer APIs
>
> Hi Honnappa,
>
> Your description below looks correct to me. For the current implementation, I
> referenced a couple of existing DPDK libraries, but primarily the rte_service
> library. However, I agree that allocating the timer data structs only as needed
> would be a good idea.
>
> You are also correct that, since the rte_timer_alt_* APIs are still experimental,
> they can change without constraint; this would allow for the APIs to change
> as needed to support the above design. It looks like the only users of those
> APIs in DPDK currently are the event timer adapter and unit test code.
>
> Thanks,
> Erik
>
> > -----Original Message-----
> > From: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
> > Sent: Tuesday, February 25, 2020 5:56 PM
> > To: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>; Robert
> > Sanford <rsanford@akamai.com>; Carrillo, Erik G
> > <erik.g.carrillo@intel.com>
> > Cc: dev <dev@dpdk.org>; nd <nd@arm.com>; Phil Yang
> > <Phil.Yang@arm.com>; Gavin Hu <Gavin.Hu@arm.com>;
> > david.marchand@redhat.com; thomas@monjalon.net; nd <nd@arm.com>
> > Subject: RE: Questions about rte_timer APIs
> >
> > Hi Erik,
> > I see that the rte_timer_alt_xxx APIs are still marked as
> > experimental. So, we should be able to change them without any ABI
> > constraints. Please let me know what you think.
> >
> > Thank you,
> > Honnappa
> >
> > > -----Original Message-----
> > > From: dev <dev-bounces@dpdk.org> On Behalf Of Honnappa Nagarahalli
> > > Sent: Thursday, February 13, 2020 11:55 PM
> > > To: Robert Sanford <rsanford@akamai.com>; Erik Gabriel Carrillo
> > > <erik.g.carrillo@intel.com>
> > > Cc: dev <dev@dpdk.org>; nd <nd@arm.com>; Phil Yang
> > > <Phil.Yang@arm.com>; Gavin Hu <Gavin.Hu@arm.com>; nd
> > <nd@arm.com>
> > > Subject: [dpdk-dev] Questions about rte_timer APIs
> > >
> > > Hi Erik/Robert,
> > > I was looking at the rte_timer library code and have questions
> > > around the way instances of timer list are allocated.
> > >
> > > The API 'rte_timer_subsystem_init' allocates RTE_MAX_DATA_ELS (64)
> > > number of timer data structs. Each timer data struct acts as an
> > > independent instance of timer list. The user allocates an instance
> > > by calling 'rte_timer_data_alloc' which provides an index in the
> > > instance array.to s Essentially, the library is allocating the
> > > memory upfront even though there might not be a need to have 64
> > > instances. Please correct me if I am
> > wrong.
> > >
> > > Usually, creating multiple instances is handled by allocating
> > > required memory, as and when required, by an 'init' API. This API
> > > also returns a pointer to that instance which is passed to other
> > > APIs in the library. For ex: rte_hash library follows this approach.
> > > IMO, this is an
> > elegant way to handle multiple instances.
> > > This approach does not waste memory and does not put any restriction
> > > on number of instances.
> > >
> > > I wanted to understand the reasoning behind the current design to
> > > handle multiple instances. Appreciate your inputs.
> > >
> > > Thank you,
> > > Honnappa
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] version: 20.05-rc0
2020-02-27 11:01 0% ` David Marchand
@ 2020-02-27 11:31 0% ` David Marchand
0 siblings, 0 replies; 200+ results
From: David Marchand @ 2020-02-27 11:31 UTC (permalink / raw)
Cc: dev, Thomas Monjalon
On Thu, Feb 27, 2020 at 12:01 PM David Marchand
<david.marchand@redhat.com> wrote:
>
> On Wed, Feb 26, 2020 at 2:55 PM Thomas Monjalon <thomas@monjalon.net> wrote:
> >
> > 26/02/2020 14:38, David Marchand:
> > > Start a new release cycle with empty release notes.
> > > ABI must now be checked with v20.02 as a reference.
> > >
> > > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > > ---
> > > .travis.yml | 4 +
> > > ABI_VERSION | 2 +-
> > > VERSION | 2 +-
> > > doc/guides/rel_notes/index.rst | 1 +
> > > doc/guides/rel_notes/release_20_05.rst | 139 +++++++++++++++++++++++++
> > > 5 files changed, 146 insertions(+), 2 deletions(-)
> > > create mode 100644 doc/guides/rel_notes/release_20_05.rst
> >
> > Acked-by: Thomas Monjalon <thomas@monjalon.net>
>
> Applied, here we go.
Patchwork has just been cleaned: old patches archived, Deferred
patches back to state NEW.
--
David Marchand
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] version: 20.05-rc0
2020-02-27 11:23 3% ` David Marchand
@ 2020-02-27 11:26 4% ` Thomas Monjalon
0 siblings, 0 replies; 200+ results
From: Thomas Monjalon @ 2020-02-27 11:26 UTC (permalink / raw)
To: Bruce Richardson, David Marchand; +Cc: dev
27/02/2020 12:23, David Marchand:
> On Thu, Feb 27, 2020 at 12:17 PM Bruce Richardson
> <bruce.richardson@intel.com> wrote:
> >
> > On Wed, Feb 26, 2020 at 03:01:04PM +0100, David Marchand wrote:
> > > On Wed, Feb 26, 2020 at 2:55 PM Thomas Monjalon <thomas@monjalon.net> wrote:
> > > >
> > > > 26/02/2020 14:38, David Marchand:
> > > > > Start a new release cycle with empty release notes.
> > > > > ABI must now be checked with v20.02 as a reference.
> > > > >
> > > > > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > > > > ---
> > > > > .travis.yml | 4 +
> > > > > ABI_VERSION | 2 +-
> > > > > VERSION | 2 +-
> > > > > doc/guides/rel_notes/index.rst | 1 +
> > > > > doc/guides/rel_notes/release_20_05.rst | 139 +++++++++++++++++++++++++
> > > > > 5 files changed, 146 insertions(+), 2 deletions(-)
> > > > > create mode 100644 doc/guides/rel_notes/release_20_05.rst
> > > >
> > > > Acked-by: Thomas Monjalon <thomas@monjalon.net>
> > > >
> > > >
> > > > > --- a/.travis.yml
> > > > > +++ b/.travis.yml
> > > > > +env:
> > > > > + global:
> > > > > + - REF_GIT_TAG=v20.02
> > > >
> > > > One question: why REF_GIT_TAG was not needed in Travis during 20.02?
> > >
> > > The .ci/linux-build.sh script has a default value for parameters.
> > > http://git.dpdk.org/dpdk/tree/.ci/linux-build.sh#n71&h=v20.02
> > >
> > > This could be removed, so that we only have explicit values in the
> > > .travis.yml configuration.
> > > No strong opinion keeping them.
> > >
> > Why do we need to update the git ref tag at all. Since these releases are
> > all ABI compatible with each other, can we not leave the checks still to be
> > against 19.11?
>
> Checking only with 19.11 (major abi) would not detected breakage
> between minor versions.
> Like the meter symbols that entered the 20.0.1 version.
We check that new ABI, introduced in a minor ABI version, is not broken.
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH] version: 20.05-rc0
2020-02-27 11:17 3% ` Bruce Richardson
@ 2020-02-27 11:23 3% ` David Marchand
2020-02-27 11:26 4% ` Thomas Monjalon
0 siblings, 1 reply; 200+ results
From: David Marchand @ 2020-02-27 11:23 UTC (permalink / raw)
To: Bruce Richardson; +Cc: Thomas Monjalon, dev
On Thu, Feb 27, 2020 at 12:17 PM Bruce Richardson
<bruce.richardson@intel.com> wrote:
>
> On Wed, Feb 26, 2020 at 03:01:04PM +0100, David Marchand wrote:
> > On Wed, Feb 26, 2020 at 2:55 PM Thomas Monjalon <thomas@monjalon.net> wrote:
> > >
> > > 26/02/2020 14:38, David Marchand:
> > > > Start a new release cycle with empty release notes.
> > > > ABI must now be checked with v20.02 as a reference.
> > > >
> > > > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > > > ---
> > > > .travis.yml | 4 +
> > > > ABI_VERSION | 2 +-
> > > > VERSION | 2 +-
> > > > doc/guides/rel_notes/index.rst | 1 +
> > > > doc/guides/rel_notes/release_20_05.rst | 139 +++++++++++++++++++++++++
> > > > 5 files changed, 146 insertions(+), 2 deletions(-)
> > > > create mode 100644 doc/guides/rel_notes/release_20_05.rst
> > >
> > > Acked-by: Thomas Monjalon <thomas@monjalon.net>
> > >
> > >
> > > > --- a/.travis.yml
> > > > +++ b/.travis.yml
> > > > +env:
> > > > + global:
> > > > + - REF_GIT_TAG=v20.02
> > >
> > > One question: why REF_GIT_TAG was not needed in Travis during 20.02?
> >
> > The .ci/linux-build.sh script has a default value for parameters.
> > http://git.dpdk.org/dpdk/tree/.ci/linux-build.sh#n71&h=v20.02
> >
> > This could be removed, so that we only have explicit values in the
> > .travis.yml configuration.
> > No strong opinion keeping them.
> >
> Why do we need to update the git ref tag at all. Since these releases are
> all ABI compatible with each other, can we not leave the checks still to be
> against 19.11?
Checking only with 19.11 (major abi) would not detected breakage
between minor versions.
Like the meter symbols that entered the 20.0.1 version.
--
David Marchand
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH] version: 20.05-rc0
2020-02-26 14:01 0% ` David Marchand
@ 2020-02-27 11:17 3% ` Bruce Richardson
2020-02-27 11:23 3% ` David Marchand
0 siblings, 1 reply; 200+ results
From: Bruce Richardson @ 2020-02-27 11:17 UTC (permalink / raw)
To: David Marchand; +Cc: Thomas Monjalon, dev
On Wed, Feb 26, 2020 at 03:01:04PM +0100, David Marchand wrote:
> On Wed, Feb 26, 2020 at 2:55 PM Thomas Monjalon <thomas@monjalon.net> wrote:
> >
> > 26/02/2020 14:38, David Marchand:
> > > Start a new release cycle with empty release notes.
> > > ABI must now be checked with v20.02 as a reference.
> > >
> > > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > > ---
> > > .travis.yml | 4 +
> > > ABI_VERSION | 2 +-
> > > VERSION | 2 +-
> > > doc/guides/rel_notes/index.rst | 1 +
> > > doc/guides/rel_notes/release_20_05.rst | 139 +++++++++++++++++++++++++
> > > 5 files changed, 146 insertions(+), 2 deletions(-)
> > > create mode 100644 doc/guides/rel_notes/release_20_05.rst
> >
> > Acked-by: Thomas Monjalon <thomas@monjalon.net>
> >
> >
> > > --- a/.travis.yml
> > > +++ b/.travis.yml
> > > +env:
> > > + global:
> > > + - REF_GIT_TAG=v20.02
> >
> > One question: why REF_GIT_TAG was not needed in Travis during 20.02?
>
> The .ci/linux-build.sh script has a default value for parameters.
> http://git.dpdk.org/dpdk/tree/.ci/linux-build.sh#n71&h=v20.02
>
> This could be removed, so that we only have explicit values in the
> .travis.yml configuration.
> No strong opinion keeping them.
>
Why do we need to update the git ref tag at all. Since these releases are
all ABI compatible with each other, can we not leave the checks still to be
against 19.11?
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH] version: 20.05-rc0
2020-02-26 13:55 0% ` Thomas Monjalon
2020-02-26 14:01 0% ` David Marchand
@ 2020-02-27 11:01 0% ` David Marchand
2020-02-27 11:31 0% ` David Marchand
1 sibling, 1 reply; 200+ results
From: David Marchand @ 2020-02-27 11:01 UTC (permalink / raw)
To: David Marchand; +Cc: dev, Thomas Monjalon
On Wed, Feb 26, 2020 at 2:55 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> 26/02/2020 14:38, David Marchand:
> > Start a new release cycle with empty release notes.
> > ABI must now be checked with v20.02 as a reference.
> >
> > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > ---
> > .travis.yml | 4 +
> > ABI_VERSION | 2 +-
> > VERSION | 2 +-
> > doc/guides/rel_notes/index.rst | 1 +
> > doc/guides/rel_notes/release_20_05.rst | 139 +++++++++++++++++++++++++
> > 5 files changed, 146 insertions(+), 2 deletions(-)
> > create mode 100644 doc/guides/rel_notes/release_20_05.rst
>
> Acked-by: Thomas Monjalon <thomas@monjalon.net>
Applied, here we go.
--
David Marchand
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [RFC 0/6] New sync modes for ring
2020-02-25 13:41 3% ` Ananyev, Konstantin
@ 2020-02-27 10:31 0% ` Jerin Jacob
2020-02-28 0:17 0% ` David Christensen
0 siblings, 1 reply; 200+ results
From: Jerin Jacob @ 2020-02-27 10:31 UTC (permalink / raw)
To: Ananyev, Konstantin; +Cc: Stephen Hemminger, dpdk-dev, Olivier Matz, drc
On Tue, Feb 25, 2020 at 7:11 PM Ananyev, Konstantin
<konstantin.ananyev@intel.com> wrote:
> We do have a run-time check in our current enqueue()/dequeue implementation.
> In fact we support both modes: we have generic rte_ring_enqueue(/dequeue)_bulk(/burst)
> where sync behaviour is determined at runtime by value of prod(/cons).single.
> Or user can call rte_ring_(mp/sp)_enqueue_* functions directly.
> This RFC follows exactly the same paradigm:
> rte_ring_enqueue(/dequeue)_bulk(/burst) kept generic and it's
> behaviour is determined at runtime, by value of prod(/cons).sync_type.
> Or user can call enqueue/dequeue with particular sync mode directly:
> rte_ring_(mp/sp/rts/hts)_enqueue_(bulk/burst)*.
> The only thing that changed:
> Format of prod/cons now could differ depending on mode selected at _init_.
> So you can't create a ring for let say SP mode and then in the middle of data-path
> change your mind and start using MP_RTS mode.
> For existing modes (SP/MP, SC/MC) format remains the same and user can still
> use them interchangeably, though of course that is an error prone practice.
Makes sense.
>
> > > But I agree with the problem statement that in the virtualization use
> > > case, It may be possible to have N virtual cores runs on a physical
> > > core.
> > >
> > > IMO, The best solution would be keeping the ring API same and have a
> > > different flavor in "compile-time". Something like
> > > liburcu did for accommodating different flavors.
> > >
> > > i.e urcu-qsbr.h and urcu-bp.h will identical definition of API. The
> > > application can simply include ONE header file in a C file based on
> > > the flavor.
>
> I don't think it is a flexible enough approach.
> In one app user might need to have several rings with different sync modes.
> Or even user might need a ring with different sync modes for enqueue/dequeue.
Ack.
> Yes, hiding rte_ring implementation inside .c would help a lot
> in terms of ABI maintenance and would make our future life easier.
> The question is what is the price for it in terms of performance,
> and are we ready to pay it. Not to mention that it would cause
> changes in many other libs/apps...
> So I think it should be a subject for a separate discussion.
> But, agree it would be good at least to measure the performance
> impact of such change.
> If I'll have some spare cycles, will give it a try.
> Meanwhile, can I ask Jerin and other guys to repeat tests from this RFC
> on their HW? Before continuing discussion would probably be good to know
> does the suggested patch work as expected across different platforms.
I tested on an arm64 HW. The former section is without the
patch(20.02) and later one with this patch.
I agree with Konstantin that getting more platform tests will be good
early so that we can focus on the approach
to avoid back and forth latter.
RTE>>ring_perf_autotest // without path
### Testing single element enq/deq ###
legacy APIs: SP/SC: single: 289.78
legacy APIs: MP/MC: single: 516.20
### Testing burst enq/deq ###
legacy APIs: SP/SC: burst (size: 8): 312.88
legacy APIs: SP/SC: burst (size: 32): 426.72
legacy APIs: MP/MC: burst (size: 8): 510.95
legacy APIs: MP/MC: burst (size: 32): 702.01
### Testing bulk enq/deq ###
legacy APIs: SP/SC: bulk (size: 8): 306.74
legacy APIs: SP/SC: bulk (size: 32): 411.56
legacy APIs: MP/MC: bulk (size: 8): 501.32
legacy APIs: MP/MC: bulk (size: 32): 693.07
### Testing empty bulk deq ###
legacy APIs: SP/SC: bulk (size: 8): 7.00
legacy APIs: MP/MC: bulk (size: 8): 7.00
### Testing using two physical cores ###
legacy APIs: SP/SC: bulk (size: 8): 74.36
legacy APIs: MP/MC: bulk (size: 8): 110.18
legacy APIs: SP/SC: bulk (size: 32): 23.04
legacy APIs: MP/MC: bulk (size: 32): 32.29
### Testing using all slave nodes ##
Bulk enq/dequeue count on size 8
Core [8] count = 293741
Core [9] count = 293741
Total count (size: 8): 587482
Bulk enq/dequeue count on size 32
Core [8] count = 244909
Core [9] count = 244909
Total count (size: 32): 1077300
### Testing single element enq/deq ###
elem APIs: element size 16B: SP/SC: single: 255.37
elem APIs: element size 16B: MP/MC: single: 456.68
### Testing burst enq/deq ###
elem APIs: element size 16B: SP/SC: burst (size: 8): 291.99
elem APIs: element size 16B: SP/SC: burst (size: 32): 456.25
elem APIs: element size 16B: MP/MC: burst (size: 8): 497.77
elem APIs: element size 16B: MP/MC: burst (size: 32): 680.87
### Testing bulk enq/deq ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 284.40
elem APIs: element size 16B: SP/SC: bulk (size: 32): 453.17
elem APIs: element size 16B: MP/MC: bulk (size: 8): 485.77
elem APIs: element size 16B: MP/MC: bulk (size: 32): 675.08
### Testing empty bulk deq ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 8.00
elem APIs: element size 16B: MP/MC: bulk (size: 8): 7.00
### Testing using two physical cores ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 74.45
elem APIs: element size 16B: MP/MC: bulk (size: 8): 105.91
elem APIs: element size 16B: SP/SC: bulk (size: 32): 22.92
elem APIs: element size 16B: MP/MC: bulk (size: 32): 31.55
### Testing using all slave nodes ###
Bulk enq/dequeue count on size 8
Core [8] count = 308724
Core [9] count = 308723
Total count (size: 8): 617447
Bulk enq/dequeue count on size 32
Core [8] count = 214269
Core [9] count = 214269
Total count (size: 32): 1045985
RTE>>ring_perf_autotest // with patch
### Testing single element enq/deq ###
legacy APIs: SP/SC: single: 289.78
legacy APIs: MP/MC: single: 475.76
### Testing burst enq/deq ###
legacy APIs: SP/SC: burst (size: 8): 323.91
legacy APIs: SP/SC: burst (size: 32): 424.60
legacy APIs: MP/MC: burst (size: 8): 523.00
legacy APIs: MP/MC: burst (size: 32): 717.09
### Testing bulk enq/deq ###
legacy APIs: SP/SC: bulk (size: 8): 317.74
legacy APIs: SP/SC: bulk (size: 32): 413.57
legacy APIs: MP/MC: bulk (size: 8): 512.89
legacy APIs: MP/MC: bulk (size: 32): 712.45
### Testing empty bulk deq ###
legacy APIs: SP/SC: bulk (size: 8): 7.00
legacy APIs: MP/MC: bulk (size: 8): 7.00
### Testing using two physical cores ###
legacy APIs: SP/SC: bulk (size: 8): 74.82
legacy APIs: MP/MC: bulk (size: 8): 96.45
legacy APIs: SP/SC: bulk (size: 32): 22.97
legacy APIs: MP/MC: bulk (size: 32): 32.52
### Testing using all slave nodes ###
Bulk enq/dequeue count on size 8
Core [8] count = 283928
Core [9] count = 283927
Total count (size: 8): 567855
Bulk enq/dequeue count on size 32
Core [8] count = 223916
Core [9] count = 223915
Total count (size: 32): 1015686
### Testing single element enq/deq ###
elem APIs: element size 16B: SP/SC: single: 267.65
elem APIs: element size 16B: MP/MC: single: 439.06
### Testing burst enq/deq ###
elem APIs: element size 16B: SP/SC: burst (size: 8): 302.44
elem APIs: element size 16B: SP/SC: burst (size: 32): 466.31
elem APIs: element size 16B: MP/MC: burst (size: 8): 502.51
elem APIs: element size 16B: MP/MC: burst (size: 32): 695.81
### Testing bulk enq/deq ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 295.15
elem APIs: element size 16B: SP/SC: bulk (size: 32): 462.77
elem APIs: element size 16B: MP/MC: bulk (size: 8): 496.89
elem APIs: element size 16B: MP/MC: bulk (size: 32): 690.46
### Testing empty bulk deq ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 7.50
elem APIs: element size 16B: MP/MC: bulk (size: 8): 7.44
### Testing using two physical cores ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 65.85
elem APIs: element size 16B: MP/MC: bulk (size: 8): 103.80
elem APIs: element size 16B: SP/SC: bulk (size: 32): 23.27
elem APIs: element size 16B: MP/MC: bulk (size: 32): 31.17
### Testing using all slave nodes ###
Bulk enq/dequeue count on size 8
Core [8] count = 304223
Core [9] count = 304221
Total count (size: 8): 608444
Bulk enq/dequeue count on size 32
Core [8] count = 214856
Core [9] count = 214855
Total count (size: 32): 1038155
Test OK
RTE>>quit
> Thanks
> Konstantin
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] Questions about rte_timer APIs
2020-02-25 23:56 3% ` Honnappa Nagarahalli
@ 2020-02-26 16:57 0% ` Carrillo, Erik G
2020-02-27 16:58 0% ` Honnappa Nagarahalli
0 siblings, 1 reply; 200+ results
From: Carrillo, Erik G @ 2020-02-26 16:57 UTC (permalink / raw)
To: Honnappa Nagarahalli, Robert Sanford
Cc: dev, nd, Phil Yang, Gavin Hu, david.marchand, thomas, nd
Hi Honnappa,
Your description below looks correct to me. For the current implementation, I referenced a couple of existing DPDK libraries, but primarily the rte_service library. However, I agree that allocating the timer data structs only as needed would be a good idea.
You are also correct that, since the rte_timer_alt_* APIs are still experimental, they can change without constraint; this would allow for the APIs to change as needed to support the above design. It looks like the only users of those APIs in DPDK currently are the event timer adapter and unit test code.
Thanks,
Erik
> -----Original Message-----
> From: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
> Sent: Tuesday, February 25, 2020 5:56 PM
> To: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>; Robert
> Sanford <rsanford@akamai.com>; Carrillo, Erik G <erik.g.carrillo@intel.com>
> Cc: dev <dev@dpdk.org>; nd <nd@arm.com>; Phil Yang
> <Phil.Yang@arm.com>; Gavin Hu <Gavin.Hu@arm.com>;
> david.marchand@redhat.com; thomas@monjalon.net; nd <nd@arm.com>
> Subject: RE: Questions about rte_timer APIs
>
> Hi Erik,
> I see that the rte_timer_alt_xxx APIs are still marked as
> experimental. So, we should be able to change them without any ABI
> constraints. Please let me know what you think.
>
> Thank you,
> Honnappa
>
> > -----Original Message-----
> > From: dev <dev-bounces@dpdk.org> On Behalf Of Honnappa Nagarahalli
> > Sent: Thursday, February 13, 2020 11:55 PM
> > To: Robert Sanford <rsanford@akamai.com>; Erik Gabriel Carrillo
> > <erik.g.carrillo@intel.com>
> > Cc: dev <dev@dpdk.org>; nd <nd@arm.com>; Phil Yang
> > <Phil.Yang@arm.com>; Gavin Hu <Gavin.Hu@arm.com>; nd
> <nd@arm.com>
> > Subject: [dpdk-dev] Questions about rte_timer APIs
> >
> > Hi Erik/Robert,
> > I was looking at the rte_timer library code and have questions around
> > the way instances of timer list are allocated.
> >
> > The API 'rte_timer_subsystem_init' allocates RTE_MAX_DATA_ELS (64)
> > number of timer data structs. Each timer data struct acts as an
> > independent instance of timer list. The user allocates an instance by
> > calling 'rte_timer_data_alloc' which provides an index in the instance array.to s
> > Essentially, the library is allocating the memory upfront even though
> > there might not be a need to have 64 instances. Please correct me if I am
> wrong.
> >
> > Usually, creating multiple instances is handled by allocating required
> > memory, as and when required, by an 'init' API. This API also returns
> > a pointer to that instance which is passed to other APIs in the
> > library. For ex: rte_hash library follows this approach. IMO, this is an
> elegant way to handle multiple instances.
> > This approach does not waste memory and does not put any restriction
> > on number of instances.
> >
> > I wanted to understand the reasoning behind the current design to
> > handle multiple instances. Appreciate your inputs.
> >
> > Thank you,
> > Honnappa
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] version: 20.05-rc0
2020-02-26 13:55 0% ` Thomas Monjalon
@ 2020-02-26 14:01 0% ` David Marchand
2020-02-27 11:17 3% ` Bruce Richardson
2020-02-27 11:01 0% ` David Marchand
1 sibling, 1 reply; 200+ results
From: David Marchand @ 2020-02-26 14:01 UTC (permalink / raw)
To: Thomas Monjalon; +Cc: dev
On Wed, Feb 26, 2020 at 2:55 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> 26/02/2020 14:38, David Marchand:
> > Start a new release cycle with empty release notes.
> > ABI must now be checked with v20.02 as a reference.
> >
> > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > ---
> > .travis.yml | 4 +
> > ABI_VERSION | 2 +-
> > VERSION | 2 +-
> > doc/guides/rel_notes/index.rst | 1 +
> > doc/guides/rel_notes/release_20_05.rst | 139 +++++++++++++++++++++++++
> > 5 files changed, 146 insertions(+), 2 deletions(-)
> > create mode 100644 doc/guides/rel_notes/release_20_05.rst
>
> Acked-by: Thomas Monjalon <thomas@monjalon.net>
>
>
> > --- a/.travis.yml
> > +++ b/.travis.yml
> > +env:
> > + global:
> > + - REF_GIT_TAG=v20.02
>
> One question: why REF_GIT_TAG was not needed in Travis during 20.02?
The .ci/linux-build.sh script has a default value for parameters.
http://git.dpdk.org/dpdk/tree/.ci/linux-build.sh#n71&h=v20.02
This could be removed, so that we only have explicit values in the
.travis.yml configuration.
No strong opinion keeping them.
--
David Marchand
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] version: 20.05-rc0
2020-02-26 13:38 10% [dpdk-dev] [PATCH] version: 20.05-rc0 David Marchand
@ 2020-02-26 13:55 0% ` Thomas Monjalon
2020-02-26 14:01 0% ` David Marchand
2020-02-27 11:01 0% ` David Marchand
0 siblings, 2 replies; 200+ results
From: Thomas Monjalon @ 2020-02-26 13:55 UTC (permalink / raw)
To: David Marchand; +Cc: dev
26/02/2020 14:38, David Marchand:
> Start a new release cycle with empty release notes.
> ABI must now be checked with v20.02 as a reference.
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
> .travis.yml | 4 +
> ABI_VERSION | 2 +-
> VERSION | 2 +-
> doc/guides/rel_notes/index.rst | 1 +
> doc/guides/rel_notes/release_20_05.rst | 139 +++++++++++++++++++++++++
> 5 files changed, 146 insertions(+), 2 deletions(-)
> create mode 100644 doc/guides/rel_notes/release_20_05.rst
Acked-by: Thomas Monjalon <thomas@monjalon.net>
> --- a/.travis.yml
> +++ b/.travis.yml
> +env:
> + global:
> + - REF_GIT_TAG=v20.02
One question: why REF_GIT_TAG was not needed in Travis during 20.02?
^ permalink raw reply [relevance 0%]
* [dpdk-dev] [PATCH] version: 20.05-rc0
@ 2020-02-26 13:38 10% David Marchand
2020-02-26 13:55 0% ` Thomas Monjalon
0 siblings, 1 reply; 200+ results
From: David Marchand @ 2020-02-26 13:38 UTC (permalink / raw)
To: dev; +Cc: thomas
Start a new release cycle with empty release notes.
ABI must now be checked with v20.02 as a reference.
Signed-off-by: David Marchand <david.marchand@redhat.com>
---
.travis.yml | 4 +
ABI_VERSION | 2 +-
VERSION | 2 +-
doc/guides/rel_notes/index.rst | 1 +
doc/guides/rel_notes/release_20_05.rst | 139 +++++++++++++++++++++++++
5 files changed, 146 insertions(+), 2 deletions(-)
create mode 100644 doc/guides/rel_notes/release_20_05.rst
diff --git a/.travis.yml b/.travis.yml
index b64a81bd0..a57790ba8 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -34,6 +34,10 @@ _doc_packages: &doc_packages
before_install: ./.ci/${TRAVIS_OS_NAME}-setup.sh
script: ./.ci/${TRAVIS_OS_NAME}-build.sh
+env:
+ global:
+ - REF_GIT_TAG=v20.02
+
jobs:
include:
# x86_64 gcc jobs
diff --git a/ABI_VERSION b/ABI_VERSION
index fcc01369a..204da679a 100644
--- a/ABI_VERSION
+++ b/ABI_VERSION
@@ -1 +1 @@
-20.0.1
+20.0.2
diff --git a/VERSION b/VERSION
index d07c5a629..9791dd8af 100644
--- a/VERSION
+++ b/VERSION
@@ -1 +1 @@
-20.02.0
+20.05.0-rc0
diff --git a/doc/guides/rel_notes/index.rst b/doc/guides/rel_notes/index.rst
index 24d927d33..31278d2a8 100644
--- a/doc/guides/rel_notes/index.rst
+++ b/doc/guides/rel_notes/index.rst
@@ -8,6 +8,7 @@ Release Notes
:maxdepth: 1
:numbered:
+ release_20_05
release_20_02
release_19_11
release_19_08
diff --git a/doc/guides/rel_notes/release_20_05.rst b/doc/guides/rel_notes/release_20_05.rst
new file mode 100644
index 000000000..2190eaf85
--- /dev/null
+++ b/doc/guides/rel_notes/release_20_05.rst
@@ -0,0 +1,139 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+ Copyright 2020 The DPDK contributors
+
+.. include:: <isonum.txt>
+
+DPDK Release 20.05
+==================
+
+.. **Read this first.**
+
+ The text in the sections below explains how to update the release notes.
+
+ Use proper spelling, capitalization and punctuation in all sections.
+
+ Variable and config names should be quoted as fixed width text:
+ ``LIKE_THIS``.
+
+ Build the docs and view the output file to ensure the changes are correct::
+
+ make doc-guides-html
+
+ xdg-open build/doc/html/guides/rel_notes/release_20_05.html
+
+
+New Features
+------------
+
+.. This section should contain new features added in this release.
+ Sample format:
+
+ * **Add a title in the past tense with a full stop.**
+
+ Add a short 1-2 sentence description in the past tense.
+ The description should be enough to allow someone scanning
+ the release notes to understand the new feature.
+
+ If the feature adds a lot of sub-features you can use a bullet list
+ like this:
+
+ * Added feature foo to do something.
+ * Enhanced feature bar to do something else.
+
+ Refer to the previous release notes for examples.
+
+ Suggested order in release notes items:
+ * Core libs (EAL, mempool, ring, mbuf, buses)
+ * Device abstraction libs and PMDs
+ - ethdev (lib, PMDs)
+ - cryptodev (lib, PMDs)
+ - eventdev (lib, PMDs)
+ - etc
+ * Other libs
+ * Apps, Examples, Tools (if significant)
+
+ This section is a comment. Do not overwrite or remove it.
+ Also, make sure to start the actual text at the margin.
+ =========================================================
+
+
+Removed Items
+-------------
+
+.. This section should contain removed items in this release. Sample format:
+
+ * Add a short 1-2 sentence description of the removed item
+ in the past tense.
+
+ This section is a comment. Do not overwrite or remove it.
+ Also, make sure to start the actual text at the margin.
+ =========================================================
+
+
+API Changes
+-----------
+
+.. This section should contain API changes. Sample format:
+
+ * sample: Add a short 1-2 sentence description of the API change
+ which was announced in the previous releases and made in this release.
+ Start with a scope label like "ethdev:".
+ Use fixed width quotes for ``function_names`` or ``struct_names``.
+ Use the past tense.
+
+ This section is a comment. Do not overwrite or remove it.
+ Also, make sure to start the actual text at the margin.
+ =========================================================
+
+
+ABI Changes
+-----------
+
+.. This section should contain ABI changes. Sample format:
+
+ * sample: Add a short 1-2 sentence description of the ABI change
+ which was announced in the previous releases and made in this release.
+ Start with a scope label like "ethdev:".
+ Use fixed width quotes for ``function_names`` or ``struct_names``.
+ Use the past tense.
+
+ This section is a comment. Do not overwrite or remove it.
+ Also, make sure to start the actual text at the margin.
+ =========================================================
+
+* No ABI change that would break compatibility with DPDK 20.02 and 19.11.
+
+
+Known Issues
+------------
+
+.. This section should contain new known issues in this release. Sample format:
+
+ * **Add title in present tense with full stop.**
+
+ Add a short 1-2 sentence description of the known issue
+ in the present tense. Add information on any known workarounds.
+
+ This section is a comment. Do not overwrite or remove it.
+ Also, make sure to start the actual text at the margin.
+ =========================================================
+
+
+Tested Platforms
+----------------
+
+.. This section should contain a list of platforms that were tested
+ with this release.
+
+ The format is:
+
+ * <vendor> platform with <vendor> <type of devices> combinations
+
+ * List of CPU
+ * List of OS
+ * List of devices
+ * Other relevant details...
+
+ This section is a comment. Do not overwrite or remove it.
+ Also, make sure to start the actual text at the margin.
+ =========================================================
--
2.23.0
^ permalink raw reply [relevance 10%]
* Re: [dpdk-dev] Questions about rte_timer APIs
@ 2020-02-25 23:56 3% ` Honnappa Nagarahalli
2020-02-26 16:57 0% ` Carrillo, Erik G
0 siblings, 1 reply; 200+ results
From: Honnappa Nagarahalli @ 2020-02-25 23:56 UTC (permalink / raw)
To: Honnappa Nagarahalli, Robert Sanford, Erik Gabriel Carrillo
Cc: dev, nd, Phil Yang, Gavin Hu, david.marchand, thomas, nd
Hi Erik,
I see that the rte_timer_alt_xxx APIs are still marked as experimental. So, we should be able to change them without any ABI constraints. Please let me know what you think.
Thank you,
Honnappa
> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Honnappa Nagarahalli
> Sent: Thursday, February 13, 2020 11:55 PM
> To: Robert Sanford <rsanford@akamai.com>; Erik Gabriel Carrillo
> <erik.g.carrillo@intel.com>
> Cc: dev <dev@dpdk.org>; nd <nd@arm.com>; Phil Yang
> <Phil.Yang@arm.com>; Gavin Hu <Gavin.Hu@arm.com>; nd <nd@arm.com>
> Subject: [dpdk-dev] Questions about rte_timer APIs
>
> Hi Erik/Robert,
> I was looking at the rte_timer library code and have questions around
> the way instances of timer list are allocated.
>
> The API 'rte_timer_subsystem_init' allocates RTE_MAX_DATA_ELS (64)
> number of timer data structs. Each timer data struct acts as an independent
> instance of timer list. The user allocates an instance by calling
> 'rte_timer_data_alloc' which provides an index in the instance array.
> Essentially, the library is allocating the memory upfront even though there
> might not be a need to have 64 instances. Please correct me if I am wrong.
>
> Usually, creating multiple instances is handled by allocating required memory,
> as and when required, by an 'init' API. This API also returns a pointer to that
> instance which is passed to other APIs in the library. For ex: rte_hash library
> follows this approach. IMO, this is an elegant way to handle multiple instances.
> This approach does not waste memory and does not put any restriction on
> number of instances.
>
> I wanted to understand the reasoning behind the current design to handle
> multiple instances. Appreciate your inputs.
>
> Thank you,
> Honnappa
^ permalink raw reply [relevance 3%]
* [dpdk-dev] [dpdk-announce] DPDK 20.02 released
@ 2020-02-25 22:59 4% Thomas Monjalon
0 siblings, 0 replies; 200+ results
From: Thomas Monjalon @ 2020-02-25 22:59 UTC (permalink / raw)
To: announce
A new release is available:
https://fast.dpdk.org/rel/dpdk-20.02.tar.xz
The statistics looks like a post-LTS ABI-compatible release:
820 commits from 156 authors
876 files changed, 59941 insertions(+), 22565 deletions(-)
It is not planned (yet) to start a maintenance branch for 20.02.
This release and next ones (20.05, 20.08) are ABI-compatible with 19.11.
Below are some new features, grouped by category.
General:
- ABI check tooling
- FreeBSD 13 support
- Linux kernel modules disabled in default build
- API wait until equal
- use CPU lcore with ID > total max
- ring with configurable element size
- pool of mbufs with pinned external buffers
Networking:
- Pensando ionic driver
- Marvell driver for OCTEON TX2 end point
- Mellanox vDPA driver
- rte_flow API for DSCP
- rte_flow API for L2TPv3 over IP
Cryptography:
- Marvell OCTEON TX2 inline IPsec
- CPU crypto based on new libraries (intel-ipsec-mb and AArch64cryptolib)
- synchronous CPU crypto API
- more algorithms: ECDSA, ECPM
Applications:
- event mode in l3fwd example
More details in the release notes:
http://doc.dpdk.org/guides/rel_notes/release_20_02.html
There are 69 new contributors (including authors, reviewers and testers).
Welcome to Abed Kamaluddin, Abhishek Marathe, Adam Ludkiewicz,
Adrian Podlawski, Aleksandr Loktionov, Alexander Kozyrev,
Alfredo Cardigliano, Andrew Pinski, Apeksha Gupta,
Balakrishna Bhamidipati, Carolyn Wyborny, Chandu Babu N,
Cheng Jiang, Chengchang Tang, Damian Milosek, Dariusz Chaberski,
Dariusz Jagus, Dexuan Cui, Dmitry Kozlyuk, Donald Lee, Dzmitry Sautsa,
Eugenio Pérez, Fang TongHao, Gal Cohen, Gargi Sau, Girish Nandibasappa,
Ilja Van Sprundel, Itsuro Oda, Jaroslaw Gawin, Jörg Thalheim,
Kiran Patil, Krzysztof Galazka, Kumar Amber, Li Feng, Lijun Ou,
Mahipal Challa, Manish Chopra, Marcin Formela, Martyna Szapar,
Mateusz Rusinski, Michael Baum, Michal Litwicki, Michal Swiatkowski,
Narcisa Vasile, Niclas Storm, Pandi Kumar Maharajan, Piotr Azarewicz,
Piotr Kwapulinski, Piotr Pietruszewski, Prateek Agarwal, Praveen Shetty,
Rafael Ávila de Espíndola, Ricardo Roldan, Robert Konklewski,
Satananda Burla, Savinay Dharmappa, Scott Wasson, Selwin Sebastian,
Shannon Nelson, Shiri Kuzin, Sunil Pai G, Sylwia Wnuczko, Thomas Faivre,
Tomasz Konieczny, Vitaliy Mysak, Xuan Ding, Xuan Li, Xueming Zhang,
and Yisen Zhuang.
Below is the number of patches per company (with authors count):
268 Intel (54)
193 Mellanox (16)
114 Marvell (23)
49 Broadcom (7)
34 Red Hat (7)
32 Huawei (8)
30 ARM (4)
25 Microsoft (1)
18 ntop (1)
9 6WIND (5)
6 NXP (4)
5 AMD (3)
4 VA Linux Systems (1)
4 Ericsson (1)
4 Cisco (1)
Based on Reviewed-by and Acked-by tags, the top reviewers are:
76 Viacheslav Ovsiienko <viacheslavo@mellanox.com>
56 Qi Zhang <qi.z.zhang@intel.com>
47 Xiaolong Ye <xiaolong.ye@intel.com>
46 Akhil Goyal <akhil.goyal@nxp.com>
44 Ferruh Yigit <ferruh.yigit@intel.com>
43 Matan Azrad <matan@mellanox.com>
43 Jerin Jacob <jerinj@marvell.com>
38 Beilei Xing <beilei.xing@intel.com>
35 Qiming Yang <qiming.yang@intel.com>
34 Ajit Khaparde <ajit.khaparde@broadcom.com>
33 Maxime Coquelin <maxime.coquelin@redhat.com>
31 Ori Kam <orika@mellanox.com>
25 Gavin Hu <gavin.hu@arm.com>
24 David Marchand <david.marchand@redhat.com>
23 Konstantin Ananyev <konstantin.ananyev@intel.com>
The new features for 20.05 may be submitted during the next 21 days,
in order to be reviewed and integrated before mid-April.
DPDK 20.05 should be released on 20-05-20.
http://core.dpdk.org/roadmap#dates
The schedule is pushed because of the special worldwide context.
Some future release cycles will have to be shorter.
Thanks everyone, and take care.
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v2] doc: plan splitting the ethdev ops struct
2020-02-25 18:13 0% ` David Marchand
@ 2020-02-25 18:18 0% ` Ferruh Yigit
0 siblings, 0 replies; 200+ results
From: Ferruh Yigit @ 2020-02-25 18:18 UTC (permalink / raw)
To: David Marchand
Cc: Neil Horman, John McNamara, Marko Kovacevic, dev, Jerin Jacob,
Thomas Monjalon, Andrew Rybchenko
On 2/25/2020 6:13 PM, David Marchand wrote:
> On Tue, Feb 25, 2020 at 1:44 PM Ferruh Yigit <ferruh.yigit@intel.com> wrote:
>>
>> For the ABI compatibility it is better to hide internal data structures
>> from the application as much as possible. But because of some inline
>> functions 'struct eth_dev_ops' can't be hidden completely.
>>
>> Plan is to split the 'struct eth_dev_ops' into two as ones used by
>> inline functions and ones not used, and hide the second part that not
>> used by inline functions completely to the application.
>>
>> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
>> Acked-by: Jerin Jacob <jerinj@marvell.com>
>
> Sorry, this patch is missing the necessary acks for merging.
> This change can still be announced for 20.11 in the next releases and
> the details can be discussed again since a conclusion was not reached.
>
>
That is OK, we can drop for 20.02, I can send a new version to target the 20.11
change only removing interim change.
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v2] doc: plan splitting the ethdev ops struct
2020-02-25 12:44 3% ` [dpdk-dev] [PATCH v2] " Ferruh Yigit
2020-02-25 15:51 3% ` Andrew Rybchenko
@ 2020-02-25 18:13 0% ` David Marchand
2020-02-25 18:18 0% ` Ferruh Yigit
2020-03-04 9:57 4% ` [dpdk-dev] [PATCH v3] " Ferruh Yigit
2 siblings, 1 reply; 200+ results
From: David Marchand @ 2020-02-25 18:13 UTC (permalink / raw)
To: Ferruh Yigit
Cc: Neil Horman, John McNamara, Marko Kovacevic, dev, Jerin Jacob,
Thomas Monjalon, Andrew Rybchenko
On Tue, Feb 25, 2020 at 1:44 PM Ferruh Yigit <ferruh.yigit@intel.com> wrote:
>
> For the ABI compatibility it is better to hide internal data structures
> from the application as much as possible. But because of some inline
> functions 'struct eth_dev_ops' can't be hidden completely.
>
> Plan is to split the 'struct eth_dev_ops' into two as ones used by
> inline functions and ones not used, and hide the second part that not
> used by inline functions completely to the application.
>
> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
> Acked-by: Jerin Jacob <jerinj@marvell.com>
Sorry, this patch is missing the necessary acks for merging.
This change can still be announced for 20.11 in the next releases and
the details can be discussed again since a conclusion was not reached.
Thanks.
--
David Marchand
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] doc: alias to experimental tag for stable apis
2020-02-25 15:35 0% ` [dpdk-dev] [PATCH] " Neil Horman
@ 2020-02-25 17:57 0% ` David Marchand
0 siblings, 0 replies; 200+ results
From: David Marchand @ 2020-02-25 17:57 UTC (permalink / raw)
To: Ray Kinsella, Neil Horman
Cc: dev, Mcnamara, John, Yigit, Ferruh, Kovacevic, Marko
On Tue, Feb 25, 2020 at 4:36 PM Neil Horman <nhorman@tuxdriver.com> wrote:
>
> On Wed, Feb 05, 2020 at 03:17:52PM +0000, Ray Kinsella wrote:
> > When a maintainer is promoting an API to become part of the next major ABI
> > version by removing the experimental tag, possibly a few releases in advance of
> > the declaration of the next ABI version. The maintainer may choose to offer an
> > alias to the experimental tag, as removing the tag before the declaration of the
> > next major ABI version, would cause an ABI breakage for applications using the
> > API.
> >
> > Signed-off-by: Ray Kinsella <mdr@ashroe.eu>
> > ---
> > doc/guides/contributing/abi_policy.rst | 10 ++++++++++
> > 1 file changed, 10 insertions(+)
> >
> > diff --git a/doc/guides/contributing/abi_policy.rst b/doc/guides/contributing/abi_policy.rst
> > index 05ca959..9a4a102 100644
> > --- a/doc/guides/contributing/abi_policy.rst
> > +++ b/doc/guides/contributing/abi_policy.rst
> > @@ -159,6 +159,11 @@ The requirements for changing the ABI are:
> > ``experimental``, as described in the section on :ref:`Experimental APIs
> > and Libraries <experimental_apis>`.
> >
> > + - In situations where an ``experimental`` API has been stable for some time.
> > + When promoting the API to become part of the next ABI version, the
> > + maintainer may choose to provide an alias to the ``experimental`` tag, so
> > + as not to break consuming applications.
> > +
> I don't have any issue with the approach, but just to ask the question, is it
> worth providing an example here, of how exactly to do this? The use of
> VERSION_SYMBOL isn't often used, and so may be non-obvious.
>
> Actually, as I look at it, the VERSION_SYMBOL macro assume a DPDK_ prefix on the
> version string, which works for versioned symbols, but not for the EXPERIMENTAL
> symbol version (no DPDK_ prefix). We should probably create a variant of the
> VERSION_SYMBOL macro to allow aliasing to the EXPERIMENTAL version, something
> like ALIAS_TO_EXPERIMENTAL() or some such
Let's clarify this part so this can be merged in early 20.05.
Thanks.
--
David Marchand
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v2] doc: plan splitting the ethdev ops struct
2020-02-25 16:13 3% ` Ferruh Yigit
@ 2020-02-25 16:41 0% ` Andrew Rybchenko
0 siblings, 0 replies; 200+ results
From: Andrew Rybchenko @ 2020-02-25 16:41 UTC (permalink / raw)
To: Ferruh Yigit, Neil Horman, John McNamara, Marko Kovacevic
Cc: dev, Jerin Jacob, David Marchand, Thomas Monjalon
On 2/25/20 7:13 PM, Ferruh Yigit wrote:
> On 2/25/2020 3:51 PM, Andrew Rybchenko wrote:
>> On 2/25/20 3:44 PM, Ferruh Yigit wrote:
>>> For the ABI compatibility it is better to hide internal data structures
>>> from the application as much as possible. But because of some inline
>>> functions 'struct eth_dev_ops' can't be hidden completely.
>>>
>>> Plan is to split the 'struct eth_dev_ops' into two as ones used by
>>> inline functions and ones not used, and hide the second part that not
>>> used by inline functions completely to the application.
>>>
>>> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
>>> Acked-by: Jerin Jacob <jerinj@marvell.com>
>>> ---
>>> Cc: David Marchand <david.marchand@redhat.com>
>>> Cc: Thomas Monjalon <thomas@monjalon.net>
>>> Cc: Andrew Rybchenko <arybchenko@solarflare.com>
>>>
>>> v2:
>>> * Add target date for the work
>>> * Give more detail on what will be done
>>> ---
>>> doc/guides/rel_notes/deprecation.rst | 11 +++++++++++
>>> 1 file changed, 11 insertions(+)
>>>
>>> diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
>>> index 99d81564a..ff612a615 100644
>>> --- a/doc/guides/rel_notes/deprecation.rst
>>> +++ b/doc/guides/rel_notes/deprecation.rst
>>> @@ -86,6 +86,17 @@ Deprecation Notices
>>> In 19.11 PMDs will still update the field even when the offload is not
>>> enabled.
>>>
>>> +* ethdev: Split the ``struct eth_dev_ops`` struct to hide it as much as possible.
>>> + Currently the ``struct eth_dev_ops`` struct is accessible by the application
>>> + because some inline functions, like ``rte_eth_tx_descriptor_status()``,
>>> + access the struct directly. The struct will be separate in two, the ops used
>>> + by inline functions still will be accessible to user but rest will be hidden.
>>> + Initial split will be done in 20.05 with adding reserved fields for the struct
>>> + used by inline functions, and by putting new struct reference into public one
>>> + to not increase the size of ``struct rte_eth_dev``, proper split will be done
>>> + in 20.11 by moving inline dev_ops function to next to Rx/Tx burst functions and
>>> + hiding rest.
>>> +
>>> * cryptodev: support for using IV with all sizes is added, J0 still can
>>> be used but only when IV length in following structs ``rte_crypto_auth_xform``,
>>> ``rte_crypto_aead_xform`` is set to zero. When IV length is greater or equal
>>
>> I'd like to understand why do we need an intermediate solution first.
>
> We don't need it really, we may prefer to postpone the update to 20.11.
>
>> Also rereading above few times I've failed to fully understand what
>> will be done and why it does not break API/ABI.
>>
>
> Agree it wasn't very clear, but I mean something like below, I hope it clarifies:
>
> Previously:
>
> struct rte_eth_dev
>
> +--------+
> | | *dev_ops struct eth_dev_ops
> | |
> | +-------------->+--------+
> | | | |
> | | | |
> | | | |
> | | | |
> +--------+ | |
> | |
> | |
> | |
> +--------+
>
>
> Proposed:
>
> struct rte_eth_dev
>
> +--------+
> | | struct eth_dev_ops
> | |
> |*dev_ops+-------------->+--------+
> | | | |
> | | | Reserv |
> | | | |
> | | | |
> +--------+ | |
> +--------+
> | inline |
> | de^_ops| struct eth_dev_ops
> +--------+
> | *priv +------------->+--------+
> +--------+ | |
> | |
> | |
> | |
> | |
> | |
> | |
> | |
> +--------+
>
> This is only to keep ABI compatibility [1] while separating the struct.
>
> [1]
> - The offset of some functions in the dev_ops struct should be same
> - The size of the "struct rte_eth_dev" should be same
>
OK, got it. Many thanks for explanations.
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v2] doc: plan splitting the ethdev ops struct
2020-02-25 15:51 3% ` Andrew Rybchenko
@ 2020-02-25 16:13 3% ` Ferruh Yigit
2020-02-25 16:41 0% ` Andrew Rybchenko
0 siblings, 1 reply; 200+ results
From: Ferruh Yigit @ 2020-02-25 16:13 UTC (permalink / raw)
To: Andrew Rybchenko, Neil Horman, John McNamara, Marko Kovacevic
Cc: dev, Jerin Jacob, David Marchand, Thomas Monjalon
On 2/25/2020 3:51 PM, Andrew Rybchenko wrote:
> On 2/25/20 3:44 PM, Ferruh Yigit wrote:
>> For the ABI compatibility it is better to hide internal data structures
>> from the application as much as possible. But because of some inline
>> functions 'struct eth_dev_ops' can't be hidden completely.
>>
>> Plan is to split the 'struct eth_dev_ops' into two as ones used by
>> inline functions and ones not used, and hide the second part that not
>> used by inline functions completely to the application.
>>
>> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
>> Acked-by: Jerin Jacob <jerinj@marvell.com>
>> ---
>> Cc: David Marchand <david.marchand@redhat.com>
>> Cc: Thomas Monjalon <thomas@monjalon.net>
>> Cc: Andrew Rybchenko <arybchenko@solarflare.com>
>>
>> v2:
>> * Add target date for the work
>> * Give more detail on what will be done
>> ---
>> doc/guides/rel_notes/deprecation.rst | 11 +++++++++++
>> 1 file changed, 11 insertions(+)
>>
>> diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
>> index 99d81564a..ff612a615 100644
>> --- a/doc/guides/rel_notes/deprecation.rst
>> +++ b/doc/guides/rel_notes/deprecation.rst
>> @@ -86,6 +86,17 @@ Deprecation Notices
>> In 19.11 PMDs will still update the field even when the offload is not
>> enabled.
>>
>> +* ethdev: Split the ``struct eth_dev_ops`` struct to hide it as much as possible.
>> + Currently the ``struct eth_dev_ops`` struct is accessible by the application
>> + because some inline functions, like ``rte_eth_tx_descriptor_status()``,
>> + access the struct directly. The struct will be separate in two, the ops used
>> + by inline functions still will be accessible to user but rest will be hidden.
>> + Initial split will be done in 20.05 with adding reserved fields for the struct
>> + used by inline functions, and by putting new struct reference into public one
>> + to not increase the size of ``struct rte_eth_dev``, proper split will be done
>> + in 20.11 by moving inline dev_ops function to next to Rx/Tx burst functions and
>> + hiding rest.
>> +
>> * cryptodev: support for using IV with all sizes is added, J0 still can
>> be used but only when IV length in following structs ``rte_crypto_auth_xform``,
>> ``rte_crypto_aead_xform`` is set to zero. When IV length is greater or equal
>
> I'd like to understand why do we need an intermediate solution first.
We don't need it really, we may prefer to postpone the update to 20.11.
> Also rereading above few times I've failed to fully understand what
> will be done and why it does not break API/ABI.
>
Agree it wasn't very clear, but I mean something like below, I hope it clarifies:
Previously:
struct rte_eth_dev
+--------+
| | *dev_ops struct eth_dev_ops
| |
| +-------------->+--------+
| | | |
| | | |
| | | |
| | | |
+--------+ | |
| |
| |
| |
+--------+
Proposed:
struct rte_eth_dev
+--------+
| | struct eth_dev_ops
| |
|*dev_ops+-------------->+--------+
| | | |
| | | Reserv |
| | | |
| | | |
+--------+ | |
+--------+
| inline |
| de^_ops| struct eth_dev_ops
+--------+
| *priv +------------->+--------+
+--------+ | |
| |
| |
| |
| |
| |
| |
| |
+--------+
This is only to keep ABI compatibility [1] while separating the struct.
[1]
- The offset of some functions in the dev_ops struct should be same
- The size of the "struct rte_eth_dev" should be same
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH v2] doc: plan splitting the ethdev ops struct
2020-02-25 12:44 3% ` [dpdk-dev] [PATCH v2] " Ferruh Yigit
@ 2020-02-25 15:51 3% ` Andrew Rybchenko
2020-02-25 16:13 3% ` Ferruh Yigit
2020-02-25 18:13 0% ` David Marchand
2020-03-04 9:57 4% ` [dpdk-dev] [PATCH v3] " Ferruh Yigit
2 siblings, 1 reply; 200+ results
From: Andrew Rybchenko @ 2020-02-25 15:51 UTC (permalink / raw)
To: Ferruh Yigit, Neil Horman, John McNamara, Marko Kovacevic
Cc: dev, Jerin Jacob, David Marchand, Thomas Monjalon
On 2/25/20 3:44 PM, Ferruh Yigit wrote:
> For the ABI compatibility it is better to hide internal data structures
> from the application as much as possible. But because of some inline
> functions 'struct eth_dev_ops' can't be hidden completely.
>
> Plan is to split the 'struct eth_dev_ops' into two as ones used by
> inline functions and ones not used, and hide the second part that not
> used by inline functions completely to the application.
>
> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
> Acked-by: Jerin Jacob <jerinj@marvell.com>
> ---
> Cc: David Marchand <david.marchand@redhat.com>
> Cc: Thomas Monjalon <thomas@monjalon.net>
> Cc: Andrew Rybchenko <arybchenko@solarflare.com>
>
> v2:
> * Add target date for the work
> * Give more detail on what will be done
> ---
> doc/guides/rel_notes/deprecation.rst | 11 +++++++++++
> 1 file changed, 11 insertions(+)
>
> diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
> index 99d81564a..ff612a615 100644
> --- a/doc/guides/rel_notes/deprecation.rst
> +++ b/doc/guides/rel_notes/deprecation.rst
> @@ -86,6 +86,17 @@ Deprecation Notices
> In 19.11 PMDs will still update the field even when the offload is not
> enabled.
>
> +* ethdev: Split the ``struct eth_dev_ops`` struct to hide it as much as possible.
> + Currently the ``struct eth_dev_ops`` struct is accessible by the application
> + because some inline functions, like ``rte_eth_tx_descriptor_status()``,
> + access the struct directly. The struct will be separate in two, the ops used
> + by inline functions still will be accessible to user but rest will be hidden.
> + Initial split will be done in 20.05 with adding reserved fields for the struct
> + used by inline functions, and by putting new struct reference into public one
> + to not increase the size of ``struct rte_eth_dev``, proper split will be done
> + in 20.11 by moving inline dev_ops function to next to Rx/Tx burst functions and
> + hiding rest.
> +
> * cryptodev: support for using IV with all sizes is added, J0 still can
> be used but only when IV length in following structs ``rte_crypto_auth_xform``,
> ``rte_crypto_aead_xform`` is set to zero. When IV length is greater or equal
I'd like to understand why do we need an intermediate solution first.
Also rereading above few times I've failed to fully understand what
will be done and why it does not break API/ABI.
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH] doc: alias to experimental tag for stable apis
2020-02-05 15:17 17% [dpdk-dev] [PATCH] doc: alias to experimental tag for stable apis Ray Kinsella
2020-02-05 15:43 17% ` [dpdk-dev] [PATCH v3] " Ray Kinsella
2020-02-18 10:46 17% ` [dpdk-dev] [PATCH v4] " Ray Kinsella
@ 2020-02-25 15:35 0% ` Neil Horman
2020-02-25 17:57 0% ` David Marchand
2 siblings, 1 reply; 200+ results
From: Neil Horman @ 2020-02-25 15:35 UTC (permalink / raw)
To: Ray Kinsella; +Cc: dev, john.mcnamara, ferruh.yigit, marko.kovacevic
On Wed, Feb 05, 2020 at 03:17:52PM +0000, Ray Kinsella wrote:
> When a maintainer is promoting an API to become part of the next major ABI
> version by removing the experimental tag, possibly a few releases in advance of
> the declaration of the next ABI version. The maintainer may choose to offer an
> alias to the experimental tag, as removing the tag before the declaration of the
> next major ABI version, would cause an ABI breakage for applications using the
> API.
>
> Signed-off-by: Ray Kinsella <mdr@ashroe.eu>
> ---
> doc/guides/contributing/abi_policy.rst | 10 ++++++++++
> 1 file changed, 10 insertions(+)
>
> diff --git a/doc/guides/contributing/abi_policy.rst b/doc/guides/contributing/abi_policy.rst
> index 05ca959..9a4a102 100644
> --- a/doc/guides/contributing/abi_policy.rst
> +++ b/doc/guides/contributing/abi_policy.rst
> @@ -159,6 +159,11 @@ The requirements for changing the ABI are:
> ``experimental``, as described in the section on :ref:`Experimental APIs
> and Libraries <experimental_apis>`.
>
> + - In situations where an ``experimental`` API has been stable for some time.
> + When promoting the API to become part of the next ABI version, the
> + maintainer may choose to provide an alias to the ``experimental`` tag, so
> + as not to break consuming applications.
> +
I don't have any issue with the approach, but just to ask the question, is it
worth providing an example here, of how exactly to do this? The use of
VERSION_SYMBOL isn't often used, and so may be non-obvious.
Actually, as I look at it, the VERSION_SYMBOL macro assume a DPDK_ prefix on the
version string, which works for versioned symbols, but not for the EXPERIMENTAL
symbol version (no DPDK_ prefix). We should probably create a variant of the
VERSION_SYMBOL macro to allow aliasing to the EXPERIMENTAL version, something
like ALIAS_TO_EXPERIMENTAL() or some such
Neil
> #. If a newly proposed API functionally replaces an existing one, when the new
> API becomes non-experimental, then the old one is marked with
> ``__rte_deprecated``.
> @@ -317,6 +322,11 @@ not required. Though, an API should remain in experimental state for at least
> one release. Thereafter, the normal process of posting patch for review to
> mailing list can be followed.
>
> +After the experimental tag has been formally removed, a tree/sub-tree maintainer
> +may choose to offer an alias to the experimental tag so as not to break
> +applications using the API. This alias can then be dropped at the declaration of
> +next major ABI version.
> +
> Libraries
> ~~~~~~~~~
>
> --
> 2.7.4
>
>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [dpdk-announce] release candidate 20.02-rc4
2020-02-25 14:44 0% ` Thomas Monjalon
@ 2020-02-25 15:27 0% ` Raslan Darawsheh
0 siblings, 0 replies; 200+ results
From: Raslan Darawsheh @ 2020-02-25 15:27 UTC (permalink / raw)
To: Ali Alnubani, Xu, Qian Q, Thomas Monjalon
Cc: dev, Asaf Penso, Peng, Yuan, Yu, PingX
Hi,
It's Ok from our side and we can release.
Kindest regards
Raslan Darawsheh
Kindest regards
Raslan Darawsheh
________________________________
From: Thomas Monjalon <thomas@monjalon.net>
Sent: Tuesday, February 25, 2020 4:44:26 PM
To: Ali Alnubani <alialnu@mellanox.com>; Xu, Qian Q <qian.q.xu@intel.com>; Raslan Darawsheh <rasland@mellanox.com>
Cc: dev@dpdk.org <dev@dpdk.org>; Asaf Penso <asafp@mellanox.com>; Peng, Yuan <yuan.peng@intel.com>; Yu, PingX <pingx.yu@intel.com>
Subject: Re: [dpdk-announce] release candidate 20.02-rc4
Can we assume we are OK to release today?
25/02/2020 15:37, Xu, Qian Q:
> Some quick update from Intel on RC4 testing
>
> Summary
>
> 1. Totally execution percentage is around 90%, pass rate is around 98%
> 2. Created 2 new medium/low issue in RC4.
>
> Details:
> 1. Intel NIC PMD PF & VF regression: 95% executed, 0 critical issue and high issue found.
> Function cases include rx/tx/forward packets, packet processing, jumboframe, dual vlan, filter cases and so on: 2 bugs found, no critical/high bugs.
> Performance cases such as l2fwd, l3fwd, single core and so on: no new bug is found.
> 2. Cryptodev cases include ipsec_gw cryptodev, fips cryptodev, l2fwd crypto, performance cases and so on: no critical and no high bugs found.
> Function cases all passed, performance cases not fully executed.
> 3. Vhost/virtio: PVP/loopback, qemu/virtio-user, multi-queue, dequeue zero copy cases and so on: 1 new medium bug is found.
> 4. Power relate cases, no new bug is found.
> 5. ABI test includes unit test, vhost and flow related tests. no new bug is found.
> 6. 20.02 new features include ABI test, L2tpv3, crypto_dev related new features and so no. no new bug is found.
>
>
> > -----Original Message-----
> > From: dev <dev-bounces@dpdk.org> On Behalf Of Ali Alnubani
> > Sent: Tuesday, February 25, 2020 5:59 PM
> > To: Thomas Monjalon <thomas@monjalon.net>; dev@dpdk.org
> > Cc: Raslan Darawsheh <rasland@mellanox.com>; Asaf Penso
> > <asafp@mellanox.com>
> > Subject: Re: [dpdk-dev] [dpdk-announce] release candidate 20.02-rc4
> >
> > Hi,
> >
> > > -----Original Message-----
> > > From: announce <announce-bounces@dpdk.org> On Behalf Of Thomas
> > > Monjalon
> > > Sent: Saturday, February 22, 2020 11:08 PM
> > > To: announce@dpdk.org
> > > Subject: [dpdk-announce] release candidate 20.02-rc4
> > >
> > > A new DPDK release candidate is ready for testing:
> > <removed>
> > >
> > > This is the last release candidate for DPDK 20.02.
> > > Please share some release validation results by replying to this
> > > message (at dev@dpdk.org).
> > > If no critical issue is reported, 20.02 will be closed on Tuesday 25th.
> > >
> > > If you are preparing the next release cycle, please send your v1
> > > patches before the 20.05 proposal deadline, which will happen on March 18th.
> > > It is also time to build an estimated roadmap for the next cycles.
> > >
> > > Thank you everyone
> > >
> >
> > The following covers the tests that we ran on Mellanox hardware for this
> > version:
> >
> > - Basic functionality:
> > Send and receive multiple types of traffic.
> > - testpmd xstats counter test.
> > - testpmd timestamp test.
> > - Changing/checking link status through testpmd.
> > - RTE flow and flow_director tests.
> > Items: eth / vlan / ipv4 / ipv6 / tcp / udp / icmp / gre / nvgre / vxlan / geneve /
> > ip in ip / mplsoudp / mplsogre / gtp
> > Actions: drop / queue / rss / mark / flag / jump / count / port_id / raw_encap /
> > raw_decap / vxlan_encap / vxlan_decap / NAT
> > - Some RSS tests.
> > - VLAN stripping and insertion tests.
> > - Checksum and TSO tests.
> > - ptype tests.
> > - l3fwd-power example application tests.
> > - Multi-process example applications tests.
> >
> > We don't see any critical issues blocking the release.
> >
> > Thanks,
> > Ali
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [RFC 0/6] New sync modes for ring
2020-02-25 0:58 0% ` Honnappa Nagarahalli
@ 2020-02-25 15:14 0% ` Ananyev, Konstantin
0 siblings, 0 replies; 200+ results
From: Ananyev, Konstantin @ 2020-02-25 15:14 UTC (permalink / raw)
To: Honnappa Nagarahalli, Jerin Jacob, Stephen Hemminger
Cc: dpdk-dev, Olivier Matz, nd, nd, wang.yong19
> > > > Upfront note - that RFC is not a complete patch.
> > > > It introduces an ABI breakage, plus it doesn't update ring_elem code
> > > > properly, etc.
> > > > I plan to deal with all these things in later versions.
> > > > Right now I seek an initial feedback about proposed ideas.
> > > > Would also ask people to repeat performance tests (see below) on
> > > > their platforms to confirm the impact.
> > > >
> > > > More and more customers use(/try to use) DPDK based apps within
> > > > overcommitted systems (multiple acttive threads over same pysical cores):
> > > > VM, container deployments, etc.
> > > > One quite common problem they hit: Lock-Holder-Preemption with
> > rte_ring.
> > > > LHP is quite a common problem for spin-based sync primitives
> > > > (spin-locks, etc.) on overcommitted systems.
> > > > The situation gets much worse when some sort of fair-locking
> > > > technique is used (ticket-lock, etc.).
> > > > As now not only lock-owner but also lock-waiters scheduling order
> > > > matters a lot.
> > > > This is a well-known problem for kernel within VMs:
> > > > http://www-archive.xenproject.org/files/xensummitboston08/LHP.pdf
> > > > https://www.cs.hs-rm.de/~kaiser/events/wamos2017/Slides/selcuk.pdf
> These slides seem to indicate that the problems are mitigated through the Hypervisor configuration. Do we still need to address the issues?
I am not really an expert here, but AFAIK current mitigations deal mostly with guest kernel:
linux implements PV version of spinlocks (unfair and/or based on hypercall availability),
hypervisor might make decision itself based on is guest in user/kernel mode,
plus on some special cpu instructions.
We do spin in user-space mode.
Might be hypervisors became smarter these days, but so far,
I heard about few different customers that hit such problem.
As an example, NA DPDK summit presentation:
https://dpdkna2019.sched.com/event/WYBG/dpdk-containers-challenges-solutions-wang-yong-zte
page 16 (problem #4) describes same issue.
>
> > > > The problem with rte_ring is that while head accusion is sort of
> > > > un-fair locking, waiting on tail is very similar to ticket lock
> > > > schema - tail has to be updated in particular order.
> > > > That makes current rte_ring implementation to perform really pure on
> > > > some overcommited scenarios.
> > >
> > > Rather than reform rte_ring to fit this scenario, it would make more
> > > sense to me to introduce another primitive. The current lockless ring
> > > performs very well for the isolated thread model that DPDK was built
> > > around. This looks like a case of customers violating the usage model
> > > of the DPDK and then being surprised at the fallout.
> >
> > I agree with Stephen here.
> >
> > I think, adding more runtime check in the enqueue() and dequeue() will have a
> > bad effect on the low-end cores too.
> > But I agree with the problem statement that in the virtualization use case, It
> > may be possible to have N virtual cores runs on a physical core.
> It is hard to imagine that there are data plane applications deployed in such environments. Wouldn't this affect the performance terribly?
It wouldn't reach same performance as isolated threads,
but for some tasks it might be enough.
AFAIK, one quite common scenario - few isolated threads(/processes) doing
actual IO and then spread packets over dozens(/hundreds) non-isolated
consumers.
>
> >
> > IMO, The best solution would be keeping the ring API same and have a
> > different flavor in "compile-time". Something like liburcu did for
> > accommodating different flavors.
> >
> > i.e urcu-qsbr.h and urcu-bp.h will identical definition of API. The application
> > can simply include ONE header file in a C file based on the flavor.
> > If need both at runtime. Need to have function pointer or so in the application
> > and define the function in different c file by including the approaite flavor in C
> > file.
> >
> > #include <urcu-qsbr.h> /* QSBR RCU flavor */ #include <urcu-bp.h> /*
> > Bulletproof RCU flavor */
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] doc: deprecate using MAX values as array size
2020-02-21 10:25 0% ` Ferruh Yigit
2020-02-25 13:52 0% ` David Marchand
@ 2020-02-25 14:53 0% ` David Marchand
2 siblings, 0 replies; 200+ results
From: David Marchand @ 2020-02-25 14:53 UTC (permalink / raw)
To: Ferruh Yigit; +Cc: Neil Horman, John McNamara, Marko Kovacevic, dev
On Thu, Jan 30, 2020 at 3:20 PM Ferruh Yigit <ferruh.yigit@intel.com> wrote:
> Adding the deprecation notice as reminder for next ABI breakage release
> (20.11).
> This one time breakage is required to be able to extend enum/define
> without breaking ABI.
>
> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
Acked-by: Andrew Rybchenko <arybchenko@solarflare.com>
Acked-by: Thomas Monjalon <thomas@monjalon.net>
Acked-by: David Marchand <david.marchand@redhat.com>
Applied, thanks.
--
David Marchand
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [dpdk-announce] release candidate 20.02-rc4
2020-02-25 14:37 4% ` Xu, Qian Q
@ 2020-02-25 14:44 0% ` Thomas Monjalon
2020-02-25 15:27 0% ` Raslan Darawsheh
0 siblings, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-02-25 14:44 UTC (permalink / raw)
To: Ali Alnubani, Xu, Qian Q, Raslan Darawsheh
Cc: dev, Asaf Penso, Peng, Yuan, Yu, PingX
Can we assume we are OK to release today?
25/02/2020 15:37, Xu, Qian Q:
> Some quick update from Intel on RC4 testing
>
> Summary
>
> 1. Totally execution percentage is around 90%, pass rate is around 98%
> 2. Created 2 new medium/low issue in RC4.
>
> Details:
> 1. Intel NIC PMD PF & VF regression: 95% executed, 0 critical issue and high issue found.
> Function cases include rx/tx/forward packets, packet processing, jumboframe, dual vlan, filter cases and so on: 2 bugs found, no critical/high bugs.
> Performance cases such as l2fwd, l3fwd, single core and so on: no new bug is found.
> 2. Cryptodev cases include ipsec_gw cryptodev, fips cryptodev, l2fwd crypto, performance cases and so on: no critical and no high bugs found.
> Function cases all passed, performance cases not fully executed.
> 3. Vhost/virtio: PVP/loopback, qemu/virtio-user, multi-queue, dequeue zero copy cases and so on: 1 new medium bug is found.
> 4. Power relate cases, no new bug is found.
> 5. ABI test includes unit test, vhost and flow related tests. no new bug is found.
> 6. 20.02 new features include ABI test, L2tpv3, crypto_dev related new features and so no. no new bug is found.
>
>
> > -----Original Message-----
> > From: dev <dev-bounces@dpdk.org> On Behalf Of Ali Alnubani
> > Sent: Tuesday, February 25, 2020 5:59 PM
> > To: Thomas Monjalon <thomas@monjalon.net>; dev@dpdk.org
> > Cc: Raslan Darawsheh <rasland@mellanox.com>; Asaf Penso
> > <asafp@mellanox.com>
> > Subject: Re: [dpdk-dev] [dpdk-announce] release candidate 20.02-rc4
> >
> > Hi,
> >
> > > -----Original Message-----
> > > From: announce <announce-bounces@dpdk.org> On Behalf Of Thomas
> > > Monjalon
> > > Sent: Saturday, February 22, 2020 11:08 PM
> > > To: announce@dpdk.org
> > > Subject: [dpdk-announce] release candidate 20.02-rc4
> > >
> > > A new DPDK release candidate is ready for testing:
> > <removed>
> > >
> > > This is the last release candidate for DPDK 20.02.
> > > Please share some release validation results by replying to this
> > > message (at dev@dpdk.org).
> > > If no critical issue is reported, 20.02 will be closed on Tuesday 25th.
> > >
> > > If you are preparing the next release cycle, please send your v1
> > > patches before the 20.05 proposal deadline, which will happen on March 18th.
> > > It is also time to build an estimated roadmap for the next cycles.
> > >
> > > Thank you everyone
> > >
> >
> > The following covers the tests that we ran on Mellanox hardware for this
> > version:
> >
> > - Basic functionality:
> > Send and receive multiple types of traffic.
> > - testpmd xstats counter test.
> > - testpmd timestamp test.
> > - Changing/checking link status through testpmd.
> > - RTE flow and flow_director tests.
> > Items: eth / vlan / ipv4 / ipv6 / tcp / udp / icmp / gre / nvgre / vxlan / geneve /
> > ip in ip / mplsoudp / mplsogre / gtp
> > Actions: drop / queue / rss / mark / flag / jump / count / port_id / raw_encap /
> > raw_decap / vxlan_encap / vxlan_decap / NAT
> > - Some RSS tests.
> > - VLAN stripping and insertion tests.
> > - Checksum and TSO tests.
> > - ptype tests.
> > - l3fwd-power example application tests.
> > - Multi-process example applications tests.
> >
> > We don't see any critical issues blocking the release.
> >
> > Thanks,
> > Ali
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [dpdk-announce] release candidate 20.02-rc4
@ 2020-02-25 14:37 4% ` Xu, Qian Q
2020-02-25 14:44 0% ` Thomas Monjalon
0 siblings, 1 reply; 200+ results
From: Xu, Qian Q @ 2020-02-25 14:37 UTC (permalink / raw)
To: Ali Alnubani, Thomas Monjalon, dev
Cc: Raslan Darawsheh, Asaf Penso, Peng, Yuan, Yu, PingX
Some quick update from Intel on RC4 testing
Summary
1. Totally execution percentage is around 90%, pass rate is around 98%
2. Created 2 new medium/low issue in RC4.
Details:
1. Intel NIC PMD PF & VF regression: 95% executed, 0 critical issue and high issue found.
Function cases include rx/tx/forward packets, packet processing, jumboframe, dual vlan, filter cases and so on: 2 bugs found, no critical/high bugs.
Performance cases such as l2fwd, l3fwd, single core and so on: no new bug is found.
2. Cryptodev cases include ipsec_gw cryptodev, fips cryptodev, l2fwd crypto, performance cases and so on: no critical and no high bugs found.
Function cases all passed, performance cases not fully executed.
3. Vhost/virtio: PVP/loopback, qemu/virtio-user, multi-queue, dequeue zero copy cases and so on: 1 new medium bug is found.
4. Power relate cases, no new bug is found.
5. ABI test includes unit test, vhost and flow related tests. no new bug is found.
6. 20.02 new features include ABI test, L2tpv3, crypto_dev related new features and so no. no new bug is found.
> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Ali Alnubani
> Sent: Tuesday, February 25, 2020 5:59 PM
> To: Thomas Monjalon <thomas@monjalon.net>; dev@dpdk.org
> Cc: Raslan Darawsheh <rasland@mellanox.com>; Asaf Penso
> <asafp@mellanox.com>
> Subject: Re: [dpdk-dev] [dpdk-announce] release candidate 20.02-rc4
>
> Hi,
>
> > -----Original Message-----
> > From: announce <announce-bounces@dpdk.org> On Behalf Of Thomas
> > Monjalon
> > Sent: Saturday, February 22, 2020 11:08 PM
> > To: announce@dpdk.org
> > Subject: [dpdk-announce] release candidate 20.02-rc4
> >
> > A new DPDK release candidate is ready for testing:
> <removed>
> >
> > This is the last release candidate for DPDK 20.02.
> > Please share some release validation results by replying to this
> > message (at dev@dpdk.org).
> > If no critical issue is reported, 20.02 will be closed on Tuesday 25th.
> >
> > If you are preparing the next release cycle, please send your v1
> > patches before the 20.05 proposal deadline, which will happen on March 18th.
> > It is also time to build an estimated roadmap for the next cycles.
> >
> > Thank you everyone
> >
>
> The following covers the tests that we ran on Mellanox hardware for this
> version:
>
> - Basic functionality:
> Send and receive multiple types of traffic.
> - testpmd xstats counter test.
> - testpmd timestamp test.
> - Changing/checking link status through testpmd.
> - RTE flow and flow_director tests.
> Items: eth / vlan / ipv4 / ipv6 / tcp / udp / icmp / gre / nvgre / vxlan / geneve /
> ip in ip / mplsoudp / mplsogre / gtp
> Actions: drop / queue / rss / mark / flag / jump / count / port_id / raw_encap /
> raw_decap / vxlan_encap / vxlan_decap / NAT
> - Some RSS tests.
> - VLAN stripping and insertion tests.
> - Checksum and TSO tests.
> - ptype tests.
> - l3fwd-power example application tests.
> - Multi-process example applications tests.
>
> We don't see any critical issues blocking the release.
>
> Thanks,
> Ali
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v4] doc: alias to experimental tag for stable apis
2020-02-18 10:46 17% ` [dpdk-dev] [PATCH v4] " Ray Kinsella
2020-02-21 17:11 0% ` Ferruh Yigit
@ 2020-02-25 14:22 0% ` Kevin Traynor
1 sibling, 0 replies; 200+ results
From: Kevin Traynor @ 2020-02-25 14:22 UTC (permalink / raw)
To: Ray Kinsella, dev
Cc: john.mcnamara, nhorman, ferruh.yigit, marko.kovacevic, bruce.richardson
On 18/02/2020 10:46, Ray Kinsella wrote:
> When a maintainer is promoting an API to become part of the next major ABI
> version by removing the experimental tag, possibly a few releases in
> advance of the declaration of the next ABI version. The maintainer may
> choose to offer an alias to the experimental tag, as removing the tag
> before the declaration of the next major ABI version, would cause an ABI
> breakage for applications using the API.
>
> Signed-off-by: Ray Kinsella <mdr@ashroe.eu>
> ---
> doc/guides/contributing/abi_policy.rst | 10 ++++++++++
> 1 file changed, 10 insertions(+)
>
> diff --git a/doc/guides/contributing/abi_policy.rst b/doc/guides/contributing/abi_policy.rst
> index 05ca959..7ae7de7 100644
> --- a/doc/guides/contributing/abi_policy.rst
> +++ b/doc/guides/contributing/abi_policy.rst
> @@ -159,6 +159,11 @@ The requirements for changing the ABI are:
> ``experimental``, as described in the section on :ref:`Experimental APIs
> and Libraries <experimental_apis>`.
>
> + - In situations in which an ``experimental`` API has been stable for some
> + time. When promoting the API to become part of the next ABI version, the
> + maintainer may choose to provide an alias to the ``experimental`` tag, so
> + as not to break consuming applications.
> +
> #. If a newly proposed API functionally replaces an existing one, when the new
> API becomes non-experimental, then the old one is marked with
> ``__rte_deprecated``.
> @@ -317,6 +322,11 @@ not required. Though, an API should remain in experimental state for at least
> one release. Thereafter, the normal process of posting patch for review to
> mailing list can be followed.
>
> +After the experimental tag has been formally removed, a tree/sub-tree maintainer
> +may choose to offer an alias to the experimental tag so as not to break
> +applications using the API. The alias is then dropped at the declaration of next
> +major ABI version.
> +
> Libraries
> ~~~~~~~~~
>
>
Sounds good. It doesn't change the minimum contract but gives some
flexibility to deal with individual cases.
Acked-by: Kevin Traynor <ktraynor@redhat.com>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] doc: deprecate using MAX values as array size
2020-02-21 10:25 0% ` Ferruh Yigit
@ 2020-02-25 13:52 0% ` David Marchand
2020-02-25 14:53 0% ` David Marchand
2 siblings, 0 replies; 200+ results
From: David Marchand @ 2020-02-25 13:52 UTC (permalink / raw)
To: Ferruh Yigit; +Cc: Neil Horman, John McNamara, Marko Kovacevic, dev
On Thu, Jan 30, 2020 at 3:20 PM Ferruh Yigit <ferruh.yigit@intel.com> wrote:
> Adding the deprecation notice as reminder for next ABI breakage release
> (20.11).
> This one time breakage is required to be able to extend enum/define
> without breaking ABI.
>
> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
> ---
> doc/guides/rel_notes/deprecation.rst | 14 ++++++++++++++
> 1 file changed, 14 insertions(+)
>
> diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
> index dfcca87ab..99d81564a 100644
> --- a/doc/guides/rel_notes/deprecation.rst
> +++ b/doc/guides/rel_notes/deprecation.rst
> @@ -38,6 +38,20 @@ Deprecation Notices
> remove it from the externally visible ABI and allow it to be updated in the
> future.
>
> +* lib: will fix extending some enum/define breaking the ABI. There are multiple
> + samples in DPDK that enum/define terminated with a ``.*MAX.*`` value which is
> + used by iterators, and arrays holding these values are sized with this
> + ``.*MAX.*`` value. So extending this enum/define increases the ``.*MAX.*``
> + value which increases the size of the array and depending on how/where the
> + array is used this may break the ABI.
> + ``RTE_ETH_FLOW_MAX`` is one sample of the mentioned case, adding a new flow
> + type will break the ABI because of ``flex_mask[RTE_ETH_FLOW_MAX]`` array
> + usage in following public struct hierarchy:
> + ``rte_eth_fdir_flex_conf -> rte_fdir_conf -> rte_eth_conf (in the middle)``.
> + Need to identify this kind of usages and fix in 20.11, otherwise this blocks
> + us extending existing enum/define.
> + One solution can be using a fixed size array instead of ``.*MAX.*`` value.
> +
> * dpaa2: removal of ``rte_dpaa2_memsegs`` structure which has been replaced
> by a pa-va search library. This structure was earlier being used for holding
> memory segments used by dpaa2 driver for faster pa->va translation. This
> --
> 2.24.1
>
Acked-by: David Marchand <david.marchand@redhat.com>
--
David Marchand
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [RFC 0/6] New sync modes for ring
2020-02-24 19:35 3% ` Stephen Hemminger
2020-02-24 20:52 0% ` Honnappa Nagarahalli
@ 2020-02-25 13:41 3% ` Ananyev, Konstantin
2020-02-27 10:31 0% ` Jerin Jacob
1 sibling, 1 reply; 200+ results
From: Ananyev, Konstantin @ 2020-02-25 13:41 UTC (permalink / raw)
To: Stephen Hemminger, Jerin Jacob; +Cc: dpdk-dev, Olivier Matz, drc
> > > > Upfront note - that RFC is not a complete patch.
> > > > It introduces an ABI breakage, plus it doesn't update ring_elem
> > > > code properly, etc.
> > > > I plan to deal with all these things in later versions.
> > > > Right now I seek an initial feedback about proposed ideas.
> > > > Would also ask people to repeat performance tests (see below)
> > > > on their platforms to confirm the impact.
> > > >
> > > > More and more customers use(/try to use) DPDK based apps within
> > > > overcommitted systems (multiple acttive threads over same pysical cores):
> > > > VM, container deployments, etc.
> > > > One quite common problem they hit: Lock-Holder-Preemption with rte_ring.
> > > > LHP is quite a common problem for spin-based sync primitives
> > > > (spin-locks, etc.) on overcommitted systems.
> > > > The situation gets much worse when some sort of
> > > > fair-locking technique is used (ticket-lock, etc.).
> > > > As now not only lock-owner but also lock-waiters scheduling
> > > > order matters a lot.
> > > > This is a well-known problem for kernel within VMs:
> > > > http://www-archive.xenproject.org/files/xensummitboston08/LHP.pdf
> > > > https://www.cs.hs-rm.de/~kaiser/events/wamos2017/Slides/selcuk.pdf
> > > > The problem with rte_ring is that while head accusion is sort of
> > > > un-fair locking, waiting on tail is very similar to ticket lock schema -
> > > > tail has to be updated in particular order.
> > > > That makes current rte_ring implementation to perform
> > > > really pure on some overcommited scenarios.
> > >
> > > Rather than reform rte_ring to fit this scenario, it would make
> > > more sense to me to introduce another primitive.
I don't see much advantages it will bring us.
As a disadvantages, for developers and maintainers - code duplication,
for end users - extra code churn and removed ability to mix and match
different sync modes in one ring.
> The current lockless
> > > ring performs very well for the isolated thread model that DPDK
> > > was built around. This looks like a case of customers violating
> > > the usage model of the DPDK and then being surprised at the fallout.
For customers using isolated thread model - nothing should change
(both in terms of API and performance).
Existing sync modes MP/MC,SP/SC kept untouched, set up in the same
way (via flags and _init_), and MP/MC remains as default one.
From other side I don't see why we should ignore customers that want to use
their DPDK apps in different deployment scenarios.
> >
> > I agree with Stephen here.
> >
> > I think, adding more runtime check in the enqueue() and dequeue() will
> > have a bad effect on the low-end cores too.
We do have a run-time check in our current enqueue()/dequeue implementation.
In fact we support both modes: we have generic rte_ring_enqueue(/dequeue)_bulk(/burst)
where sync behaviour is determined at runtime by value of prod(/cons).single.
Or user can call rte_ring_(mp/sp)_enqueue_* functions directly.
This RFC follows exactly the same paradigm:
rte_ring_enqueue(/dequeue)_bulk(/burst) kept generic and it's
behaviour is determined at runtime, by value of prod(/cons).sync_type.
Or user can call enqueue/dequeue with particular sync mode directly:
rte_ring_(mp/sp/rts/hts)_enqueue_(bulk/burst)*.
The only thing that changed:
Format of prod/cons now could differ depending on mode selected at _init_.
So you can't create a ring for let say SP mode and then in the middle of data-path
change your mind and start using MP_RTS mode.
For existing modes (SP/MP, SC/MC) format remains the same and user can still
use them interchangeably, though of course that is an error prone practice.
> > But I agree with the problem statement that in the virtualization use
> > case, It may be possible to have N virtual cores runs on a physical
> > core.
> >
> > IMO, The best solution would be keeping the ring API same and have a
> > different flavor in "compile-time". Something like
> > liburcu did for accommodating different flavors.
> >
> > i.e urcu-qsbr.h and urcu-bp.h will identical definition of API. The
> > application can simply include ONE header file in a C file based on
> > the flavor.
I don't think it is a flexible enough approach.
In one app user might need to have several rings with different sync modes.
Or even user might need a ring with different sync modes for enqueue/dequeue.
> > If need both at runtime. Need to have function pointer or so in the
> > application and define the function in different c file by including
> > the approaite flavor in C file.
Big issue with function pointers here would be DPDK MP model.
AFAIK, rte_ring is quite popular mechanism for IPC between DPDK apps.
To support such model, we'll need to split rte_ring data into 'shared'
and 'private' and initialize private one for every process that is going to use it.
That sounds like a massive change, and I am not sure the required effort will worth it.
BTW, if user just calls API functions without trying to access structure internals directly,
I don't think it would be a big difference for him what is inside:
indirect function call or inlined switch(...) {}.
> This would also be a good time to consider the tradeoffs of the
> heavy use of inlining that is done in rte_ring vs the impact that
> has on API/ABI stability.
Yes, hiding rte_ring implementation inside .c would help a lot
in terms of ABI maintenance and would make our future life easier.
The question is what is the price for it in terms of performance,
and are we ready to pay it. Not to mention that it would cause
changes in many other libs/apps...
So I think it should be a subject for a separate discussion.
But, agree it would be good at least to measure the performance
impact of such change.
If I'll have some spare cycles, will give it a try.
Meanwhile, can I ask Jerin and other guys to repeat tests from this RFC
on their HW? Before continuing discussion would probably be good to know
does the suggested patch work as expected across different platforms.
Thanks
Konstantin
^ permalink raw reply [relevance 3%]
* [dpdk-dev] [PATCH v2] doc: plan splitting the ethdev ops struct
2020-02-17 15:38 3% [dpdk-dev] [PATCH] doc: plan splitting the ethdev ops struct Ferruh Yigit
2020-02-18 5:07 0% ` Jerin Jacob
2020-02-18 6:01 0% ` Stephen Hemminger
@ 2020-02-25 12:44 3% ` Ferruh Yigit
2020-02-25 15:51 3% ` Andrew Rybchenko
` (2 more replies)
2 siblings, 3 replies; 200+ results
From: Ferruh Yigit @ 2020-02-25 12:44 UTC (permalink / raw)
To: Neil Horman, John McNamara, Marko Kovacevic
Cc: dev, Ferruh Yigit, Jerin Jacob, David Marchand, Thomas Monjalon,
Andrew Rybchenko
For the ABI compatibility it is better to hide internal data structures
from the application as much as possible. But because of some inline
functions 'struct eth_dev_ops' can't be hidden completely.
Plan is to split the 'struct eth_dev_ops' into two as ones used by
inline functions and ones not used, and hide the second part that not
used by inline functions completely to the application.
Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
Acked-by: Jerin Jacob <jerinj@marvell.com>
---
Cc: David Marchand <david.marchand@redhat.com>
Cc: Thomas Monjalon <thomas@monjalon.net>
Cc: Andrew Rybchenko <arybchenko@solarflare.com>
v2:
* Add target date for the work
* Give more detail on what will be done
---
doc/guides/rel_notes/deprecation.rst | 11 +++++++++++
1 file changed, 11 insertions(+)
diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
index 99d81564a..ff612a615 100644
--- a/doc/guides/rel_notes/deprecation.rst
+++ b/doc/guides/rel_notes/deprecation.rst
@@ -86,6 +86,17 @@ Deprecation Notices
In 19.11 PMDs will still update the field even when the offload is not
enabled.
+* ethdev: Split the ``struct eth_dev_ops`` struct to hide it as much as possible.
+ Currently the ``struct eth_dev_ops`` struct is accessible by the application
+ because some inline functions, like ``rte_eth_tx_descriptor_status()``,
+ access the struct directly. The struct will be separate in two, the ops used
+ by inline functions still will be accessible to user but rest will be hidden.
+ Initial split will be done in 20.05 with adding reserved fields for the struct
+ used by inline functions, and by putting new struct reference into public one
+ to not increase the size of ``struct rte_eth_dev``, proper split will be done
+ in 20.11 by moving inline dev_ops function to next to Rx/Tx burst functions and
+ hiding rest.
+
* cryptodev: support for using IV with all sizes is added, J0 still can
be used but only when IV length in following structs ``rte_crypto_auth_xform``,
``rte_crypto_aead_xform`` is set to zero. When IV length is greater or equal
--
2.24.1
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH] doc: plan splitting the ethdev ops struct
2020-02-18 5:07 0% ` Jerin Jacob
@ 2020-02-25 12:42 0% ` Ferruh Yigit
0 siblings, 0 replies; 200+ results
From: Ferruh Yigit @ 2020-02-25 12:42 UTC (permalink / raw)
To: Jerin Jacob
Cc: Neil Horman, John McNamara, Marko Kovacevic, dpdk-dev,
David Marchand, Thomas Monjalon, Andrew Rybchenko
On 2/18/2020 5:07 AM, Jerin Jacob wrote:
> On Mon, Feb 17, 2020 at 9:08 PM Ferruh Yigit <ferruh.yigit@intel.com> wrote:
>>
>> For the ABI compatibility it is better to hide internal data structures
>> from the application as much as possible. But because of some inline
>> functions 'struct eth_dev_ops' can't be hidden completely.
>>
>> Plan is to split the 'struct eth_dev_ops' into two as ones used by
>> inline functions and ones not used, and hide the second part that not
>> used by inline functions completely to the application.
>
> It is a good improvement. IMO, If anything used in fast-path it
> should be in ``struct rte_eth_dev``
> and rest can completely be moved to internal. In this case, if
> `rte_eth_tx_descriptor_status`
> not used on fastpath, Maybe we don't need to maintain the inline
> status and move completely
> to .c file.
Moving fast-past dev_ops to ``struct rte_eth_dev`` makes sense, also suggested
by Andrew, and will hide the rest.
Those inline dev_ops are datapath related so may be better to keep them as
inline, but not really sure about their users and how much they care about minor
performance impact if we chose to convert them into regular functions.
>
> Those may be specifics of the work. In general, this change looks good to me.
>
> Acked-by: Jerin Jacob <jerinj@marvell.com>
>
>
>
>
>
>>
>> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
>> ---
>> Cc: David Marchand <david.marchand@redhat.com>
>> Cc: Thomas Monjalon <thomas@monjalon.net>
>> Cc: Andrew Rybchenko <arybchenko@solarflare.com>
>> ---
>> doc/guides/rel_notes/deprecation.rst | 6 ++++++
>> 1 file changed, 6 insertions(+)
>>
>> diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
>> index dfcca87ab..2aa431028 100644
>> --- a/doc/guides/rel_notes/deprecation.rst
>> +++ b/doc/guides/rel_notes/deprecation.rst
>> @@ -72,6 +72,12 @@ Deprecation Notices
>> In 19.11 PMDs will still update the field even when the offload is not
>> enabled.
>>
>> +* ethdev: Split the ``struct eth_dev_ops`` struct to hide it as much as possible.
>> + Currently the ``struct eth_dev_ops`` struct is accessible by the application
>> + because some inline functions, like ``rte_eth_tx_descriptor_status()``,
>> + access the struct directly. The struct will be separate in two, the ops used
>> + by inline functions still will be accessible to user but rest will be hidden.
>> +
>> * cryptodev: support for using IV with all sizes is added, J0 still can
>> be used but only when IV length in following structs ``rte_crypto_auth_xform``,
>> ``rte_crypto_aead_xform`` is set to zero. When IV length is greater or equal
>> --
>> 2.24.1
>>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] doc: plan splitting the ethdev ops struct
2020-02-25 10:35 0% ` Andrew Rybchenko
2020-02-25 11:07 3% ` Ananyev, Konstantin
@ 2020-02-25 12:28 3% ` Ferruh Yigit
1 sibling, 0 replies; 200+ results
From: Ferruh Yigit @ 2020-02-25 12:28 UTC (permalink / raw)
To: Andrew Rybchenko, Stephen Hemminger
Cc: Neil Horman, John McNamara, Marko Kovacevic, dev, David Marchand,
Thomas Monjalon, Kinsella, Ray, Olivier MATZ
On 2/25/2020 10:35 AM, Andrew Rybchenko wrote:
> On 2/21/20 1:40 PM, Ferruh Yigit wrote:
>> On 2/18/2020 6:01 AM, Stephen Hemminger wrote:
>>> On Mon, 17 Feb 2020 15:38:05 +0000
>>> Ferruh Yigit <ferruh.yigit@intel.com> wrote:
>>>
>>>> For the ABI compatibility it is better to hide internal data structures
>>>> from the application as much as possible. But because of some inline
>>>> functions 'struct eth_dev_ops' can't be hidden completely.
>>>>
>>>> Plan is to split the 'struct eth_dev_ops' into two as ones used by
>>>> inline functions and ones not used, and hide the second part that not
>>>> used by inline functions completely to the application.
>>>>
>>>> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
>>>> ---
>>>> Cc: David Marchand <david.marchand@redhat.com>
>>>> Cc: Thomas Monjalon <thomas@monjalon.net>
>>>> Cc: Andrew Rybchenko <arybchenko@solarflare.com>
>>>> ---
>>>> doc/guides/rel_notes/deprecation.rst | 6 ++++++
>>>> 1 file changed, 6 insertions(+)
>>>>
>>>> diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
>>>> index dfcca87ab..2aa431028 100644
>>>> --- a/doc/guides/rel_notes/deprecation.rst
>>>> +++ b/doc/guides/rel_notes/deprecation.rst
>>>> @@ -72,6 +72,12 @@ Deprecation Notices
>>>> In 19.11 PMDs will still update the field even when the offload is not
>>>> enabled.
>>>>
>>>> +* ethdev: Split the ``struct eth_dev_ops`` struct to hide it as much as possible.
>>>> + Currently the ``struct eth_dev_ops`` struct is accessible by the application
>>>> + because some inline functions, like ``rte_eth_tx_descriptor_status()``,
>>>> + access the struct directly. The struct will be separate in two, the ops used
>>>> + by inline functions still will be accessible to user but rest will be hidden.
>>>> +
>>>> * cryptodev: support for using IV with all sizes is added, J0 still can
>>>> be used but only when IV length in following structs ``rte_crypto_auth_xform``,
>>>> ``rte_crypto_aead_xform`` is set to zero. When IV length is greater or equal
>>>
>>> Good luck, truely hiding internals is hard. The mbuf structure is already split but not really
>>> hidden at all (just look at dwarf output). It doesn't make sense to do it unless
>>> you can really hide it.
>>
>> I believe this can be done, only following [1] dev_ops are used by inline
>> functions, rest can be moved into separate struct and moved into ethdev driver
>> looking header.
>>
>> [1]
>> rx_queue_count
>> rx_descriptor_done
>> rx_descriptor_status
>> tx_descriptor_status
>
> I think having 3 places (if I understand the intention
> correctly) with ethdev callbacks is too much.
Yes, this is the intention.
> So, I think
> that these ops should be simply moved to nearby Tx/Rx
> burst and Tx prepare callbacks (e.g. move into inline_ops
> structure which is located at the beginning of rte_eth_dev
> in order to preserve 3 existing callback location).
I think this is reasonable, but this can be done only in 20.11 with ABI break.
What do you think doing the initial hide in 20.05 with three places and do
proper splitting in 20.11 as suggested above.
Or it is possible to drop the interim work and do the all changes in 20.11, not
sure.
>
> Also I'd consider to deprecate and remove rx_queue_count
> and rx_descriptor_done.
+1 to deprecate the 'rx_descriptor_done', we have already 'rx_descriptor_status'
& 'tx_descriptor_status' to replace the functionality.
@Thomas, @Ray, can you please remind how deprecation was done, is it marking
API, 'rte_eth_rx_descriptor_done', as deprecated in 20.11 and removing it in
21.11? If so I guess deprecation notice should be send before 20.11? (And make
sure all PMDs did the switch before 20.11)
Not sure about 'rx_queue_count', if any application may need it or not.
>
>>>
>>> I would attack the rte_device stuff first. Make rte_device opaque to the application
>>> that would help for future versions. Then work backwards to rte_tehtdev.
>>>
>
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [RFC 0/6] New sync modes for ring
2020-02-24 20:52 0% ` Honnappa Nagarahalli
@ 2020-02-25 11:45 0% ` Ananyev, Konstantin
0 siblings, 0 replies; 200+ results
From: Ananyev, Konstantin @ 2020-02-25 11:45 UTC (permalink / raw)
To: Honnappa Nagarahalli, Stephen Hemminger, Jerin Jacob
Cc: dpdk-dev, Olivier Matz, nd, nd
> > -----Original Message-----
> > From: dev <dev-bounces@dpdk.org> On Behalf Of Stephen Hemminger
> > Sent: Monday, February 24, 2020 1:35 PM
> > To: Jerin Jacob <jerinjacobk@gmail.com>
> > Cc: Konstantin Ananyev <konstantin.ananyev@intel.com>; dpdk-dev
> > <dev@dpdk.org>; Olivier Matz <olivier.matz@6wind.com>
> > Subject: Re: [dpdk-dev] [RFC 0/6] New sync modes for ring
> >
> > On Mon, 24 Feb 2020 23:29:57 +0530
> > Jerin Jacob <jerinjacobk@gmail.com> wrote:
> >
> > > On Mon, Feb 24, 2020 at 10:29 PM Stephen Hemminger
> > > <stephen@networkplumber.org> wrote:
> > > >
> > > > On Mon, 24 Feb 2020 11:35:09 +0000
> > > > Konstantin Ananyev <konstantin.ananyev@intel.com> wrote:
> > > >
> > > > > Upfront note - that RFC is not a complete patch.
> > > > > It introduces an ABI breakage, plus it doesn't update ring_elem
> > > > > code properly, etc.
> > > > > I plan to deal with all these things in later versions.
> > > > > Right now I seek an initial feedback about proposed ideas.
> > > > > Would also ask people to repeat performance tests (see below) on
> > > > > their platforms to confirm the impact.
> > > > >
> > > > > More and more customers use(/try to use) DPDK based apps within
> > > > > overcommitted systems (multiple acttive threads over same pysical
> > cores):
> > > > > VM, container deployments, etc.
> > > > > One quite common problem they hit: Lock-Holder-Preemption with
> > rte_ring.
> > > > > LHP is quite a common problem for spin-based sync primitives
> > > > > (spin-locks, etc.) on overcommitted systems.
> > > > > The situation gets much worse when some sort of fair-locking
> > > > > technique is used (ticket-lock, etc.).
> > > > > As now not only lock-owner but also lock-waiters scheduling order
> > > > > matters a lot.
> > > > > This is a well-known problem for kernel within VMs:
> > > > > http://www-archive.xenproject.org/files/xensummitboston08/LHP.pdf
> > > > > https://www.cs.hs-rm.de/~kaiser/events/wamos2017/Slides/selcuk.pdf
> > > > > The problem with rte_ring is that while head accusion is sort of
> > > > > un-fair locking, waiting on tail is very similar to ticket lock
> > > > > schema - tail has to be updated in particular order.
> > > > > That makes current rte_ring implementation to perform really pure
> > > > > on some overcommited scenarios.
> > > >
> > > > Rather than reform rte_ring to fit this scenario, it would make more
> > > > sense to me to introduce another primitive. The current lockless
> > > > ring performs very well for the isolated thread model that DPDK was
> > > > built around. This looks like a case of customers violating the
> > > > usage model of the DPDK and then being surprised at the fallout.
> > >
> > > I agree with Stephen here.
> > >
> > > I think, adding more runtime check in the enqueue() and dequeue() will
> > > have a bad effect on the low-end cores too.
> > > But I agree with the problem statement that in the virtualization use
> > > case, It may be possible to have N virtual cores runs on a physical
> > > core.
> > >
> > > IMO, The best solution would be keeping the ring API same and have a
> > > different flavor in "compile-time". Something like liburcu did for
> > > accommodating different flavors.
> > >
> > > i.e urcu-qsbr.h and urcu-bp.h will identical definition of API. The
> > > application can simply include ONE header file in a C file based on
> > > the flavor.
> > > If need both at runtime. Need to have function pointer or so in the
> > > application and define the function in different c file by including
> > > the approaite flavor in C file.
> >
> > This would also be a good time to consider the tradeoffs of the heavy use of
> > inlining that is done in rte_ring vs the impact that has on API/ABI stability.
> >
> I was working on few requirements in rte_ring library for RCU defer APIs. RFC is at https://patchwork.dpdk.org/cover/66020/.
Yep, noticed your patch, seems we sort of collided here.
As I understand you patch aims to provide functionality similar to my HTS one.
Will try to look at yours one in next fee days, hopefully we can end-up with
some common denominator.
Konstantin
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] doc: plan splitting the ethdev ops struct
2020-02-25 11:07 3% ` Ananyev, Konstantin
@ 2020-02-25 11:19 0% ` Andrew Rybchenko
0 siblings, 0 replies; 200+ results
From: Andrew Rybchenko @ 2020-02-25 11:19 UTC (permalink / raw)
To: Ananyev, Konstantin, Yigit, Ferruh, Stephen Hemminger
Cc: Neil Horman, Mcnamara, John, Kovacevic, Marko, dev,
David Marchand, Thomas Monjalon
Hi Konstantin,
On 2/25/20 2:07 PM, Ananyev, Konstantin wrote:
> Hi Andew,
>
>> On 2/21/20 1:40 PM, Ferruh Yigit wrote:
>>> On 2/18/2020 6:01 AM, Stephen Hemminger wrote:
>>>> On Mon, 17 Feb 2020 15:38:05 +0000
>>>> Ferruh Yigit <ferruh.yigit@intel.com> wrote:
>>>>
>>>>> For the ABI compatibility it is better to hide internal data structures
>>>>> from the application as much as possible. But because of some inline
>>>>> functions 'struct eth_dev_ops' can't be hidden completely.
>>>>>
>>>>> Plan is to split the 'struct eth_dev_ops' into two as ones used by
>>>>> inline functions and ones not used, and hide the second part that not
>>>>> used by inline functions completely to the application.
>>>>>
>>>>> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
>>>>> ---
>>>>> Cc: David Marchand <david.marchand@redhat.com>
>>>>> Cc: Thomas Monjalon <thomas@monjalon.net>
>>>>> Cc: Andrew Rybchenko <arybchenko@solarflare.com>
>>>>> ---
>>>>> doc/guides/rel_notes/deprecation.rst | 6 ++++++
>>>>> 1 file changed, 6 insertions(+)
>>>>>
>>>>> diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
>>>>> index dfcca87ab..2aa431028 100644
>>>>> --- a/doc/guides/rel_notes/deprecation.rst
>>>>> +++ b/doc/guides/rel_notes/deprecation.rst
>>>>> @@ -72,6 +72,12 @@ Deprecation Notices
>>>>> In 19.11 PMDs will still update the field even when the offload is not
>>>>> enabled.
>>>>>
>>>>> +* ethdev: Split the ``struct eth_dev_ops`` struct to hide it as much as possible.
>>>>> + Currently the ``struct eth_dev_ops`` struct is accessible by the application
>>>>> + because some inline functions, like ``rte_eth_tx_descriptor_status()``,
>>>>> + access the struct directly. The struct will be separate in two, the ops used
>>>>> + by inline functions still will be accessible to user but rest will be hidden.
>>>>> +
>>>>> * cryptodev: support for using IV with all sizes is added, J0 still can
>>>>> be used but only when IV length in following structs ``rte_crypto_auth_xform``,
>>>>> ``rte_crypto_aead_xform`` is set to zero. When IV length is greater or equal
>>>> Good luck, truely hiding internals is hard. The mbuf structure is already split but not really
>>>> hidden at all (just look at dwarf output). It doesn't make sense to do it unless
>>>> you can really hide it.
>>> I believe this can be done, only following [1] dev_ops are used by inline
>>> functions, rest can be moved into separate struct and moved into ethdev driver
>>> looking header.
>>>
>>> [1]
>>> rx_queue_count
>>> rx_descriptor_done
>>> rx_descriptor_status
>>> tx_descriptor_status
>> I think having 3 places (if I understand the intention
>> correctly) with ethdev callbacks is too much. So, I think
>> that these ops should be simply moved to nearby Tx/Rx
>> burst and Tx prepare callbacks (e.g. move into inline_ops
>> structure which is located at the beginning of rte_eth_dev
>> in order to preserve 3 existing callback location).
> If you are going to change ABI anyway, would it be worth to consider
> moving rx/tx burst/prepare functions to be per queue,
> instead of per device?
I'm thinking about it from time to time. In general I like the idea.
The only question I have if there is a demand for queues with
different offloads which could result in different Rx/Tx callbacks.
Also I'm not sure that it is doable without performance degradation
at all, but hopefully it will be tiny.
>> Also I'd consider to deprecate and remove rx_queue_count
>> and rx_descriptor_done.
>>
>>>> I would attack the rte_device stuff first. Make rte_device opaque to the application
>>>> that would help for future versions. Then work backwards to rte_tehtdev.
>>>>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] doc: plan splitting the ethdev ops struct
2020-02-25 10:35 0% ` Andrew Rybchenko
@ 2020-02-25 11:07 3% ` Ananyev, Konstantin
2020-02-25 11:19 0% ` Andrew Rybchenko
2020-02-25 12:28 3% ` Ferruh Yigit
1 sibling, 1 reply; 200+ results
From: Ananyev, Konstantin @ 2020-02-25 11:07 UTC (permalink / raw)
To: Andrew Rybchenko, Yigit, Ferruh, Stephen Hemminger
Cc: Neil Horman, Mcnamara, John, Kovacevic, Marko, dev,
David Marchand, Thomas Monjalon
Hi Andew,
> On 2/21/20 1:40 PM, Ferruh Yigit wrote:
> > On 2/18/2020 6:01 AM, Stephen Hemminger wrote:
> >> On Mon, 17 Feb 2020 15:38:05 +0000
> >> Ferruh Yigit <ferruh.yigit@intel.com> wrote:
> >>
> >>> For the ABI compatibility it is better to hide internal data structures
> >>> from the application as much as possible. But because of some inline
> >>> functions 'struct eth_dev_ops' can't be hidden completely.
> >>>
> >>> Plan is to split the 'struct eth_dev_ops' into two as ones used by
> >>> inline functions and ones not used, and hide the second part that not
> >>> used by inline functions completely to the application.
> >>>
> >>> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
> >>> ---
> >>> Cc: David Marchand <david.marchand@redhat.com>
> >>> Cc: Thomas Monjalon <thomas@monjalon.net>
> >>> Cc: Andrew Rybchenko <arybchenko@solarflare.com>
> >>> ---
> >>> doc/guides/rel_notes/deprecation.rst | 6 ++++++
> >>> 1 file changed, 6 insertions(+)
> >>>
> >>> diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
> >>> index dfcca87ab..2aa431028 100644
> >>> --- a/doc/guides/rel_notes/deprecation.rst
> >>> +++ b/doc/guides/rel_notes/deprecation.rst
> >>> @@ -72,6 +72,12 @@ Deprecation Notices
> >>> In 19.11 PMDs will still update the field even when the offload is not
> >>> enabled.
> >>>
> >>> +* ethdev: Split the ``struct eth_dev_ops`` struct to hide it as much as possible.
> >>> + Currently the ``struct eth_dev_ops`` struct is accessible by the application
> >>> + because some inline functions, like ``rte_eth_tx_descriptor_status()``,
> >>> + access the struct directly. The struct will be separate in two, the ops used
> >>> + by inline functions still will be accessible to user but rest will be hidden.
> >>> +
> >>> * cryptodev: support for using IV with all sizes is added, J0 still can
> >>> be used but only when IV length in following structs ``rte_crypto_auth_xform``,
> >>> ``rte_crypto_aead_xform`` is set to zero. When IV length is greater or equal
> >>
> >> Good luck, truely hiding internals is hard. The mbuf structure is already split but not really
> >> hidden at all (just look at dwarf output). It doesn't make sense to do it unless
> >> you can really hide it.
> >
> > I believe this can be done, only following [1] dev_ops are used by inline
> > functions, rest can be moved into separate struct and moved into ethdev driver
> > looking header.
> >
> > [1]
> > rx_queue_count
> > rx_descriptor_done
> > rx_descriptor_status
> > tx_descriptor_status
>
> I think having 3 places (if I understand the intention
> correctly) with ethdev callbacks is too much. So, I think
> that these ops should be simply moved to nearby Tx/Rx
> burst and Tx prepare callbacks (e.g. move into inline_ops
> structure which is located at the beginning of rte_eth_dev
> in order to preserve 3 existing callback location).
If you are going to change ABI anyway, would it be worth to consider
moving rx/tx burst/prepare functions to be per queue,
instead of per device?
>
> Also I'd consider to deprecate and remove rx_queue_count
> and rx_descriptor_done.
>
> >>
> >> I would attack the rte_device stuff first. Make rte_device opaque to the application
> >> that would help for future versions. Then work backwards to rte_tehtdev.
> >>
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH] doc: deprecate using MAX values as array size
2020-02-24 6:18 0% ` Akhil Goyal
@ 2020-02-25 11:03 0% ` Thomas Monjalon
0 siblings, 0 replies; 200+ results
From: Thomas Monjalon @ 2020-02-25 11:03 UTC (permalink / raw)
To: Ferruh Yigit
Cc: Neil Horman, John McNamara, Marko Kovacevic, dev,
Bruce Richardson, Konstantin Ananyev, David Marchand,
Andrew Rybchenko, Ori Kam, Akhil Goyal
> > > Adding the deprecation notice as reminder for next ABI breakage release
> > > (20.11).
> > > This one time breakage is required to be able to extend enum/define
> > > without breaking ABI.
> > >
> > > Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
> > > ---
> > > +* lib: will fix extending some enum/define breaking the ABI. There are
> > multiple
> > > + samples in DPDK that enum/define terminated with a ``.*MAX.*`` value
> > which is
> > > + used by iterators, and arrays holding these values are sized with this
> > > + ``.*MAX.*`` value. So extending this enum/define increases the ``.*MAX.*``
> > > + value which increases the size of the array and depending on how/where the
> > > + array is used this may break the ABI.
> > > + ``RTE_ETH_FLOW_MAX`` is one sample of the mentioned case, adding a
> > new flow
> > > + type will break the ABI because of ``flex_mask[RTE_ETH_FLOW_MAX]``
> > array
> > > + usage in following public struct hierarchy:
> > > + ``rte_eth_fdir_flex_conf -> rte_fdir_conf -> rte_eth_conf (in the middle)``.
> > > + Need to identify this kind of usages and fix in 20.11, otherwise this blocks
> > > + us extending existing enum/define.
> > > + One solution can be using a fixed size array instead of ``.*MAX.*`` value.
> >
> We admit that the issue is there and we can discuss the possible solutions for this issue.
>
> For the deprecation notice.
> Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
I think we'll need a new coding style guideline to avoid shuch issue in future.
Anyway, about fixing the issue in 20.11,
Acked-by: Thomas Monjalon <thomas@monjalon.net>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] doc: plan splitting the ethdev ops struct
2020-02-21 10:40 0% ` Ferruh Yigit
@ 2020-02-25 10:35 0% ` Andrew Rybchenko
2020-02-25 11:07 3% ` Ananyev, Konstantin
2020-02-25 12:28 3% ` Ferruh Yigit
0 siblings, 2 replies; 200+ results
From: Andrew Rybchenko @ 2020-02-25 10:35 UTC (permalink / raw)
To: Ferruh Yigit, Stephen Hemminger
Cc: Neil Horman, John McNamara, Marko Kovacevic, dev, David Marchand,
Thomas Monjalon
On 2/21/20 1:40 PM, Ferruh Yigit wrote:
> On 2/18/2020 6:01 AM, Stephen Hemminger wrote:
>> On Mon, 17 Feb 2020 15:38:05 +0000
>> Ferruh Yigit <ferruh.yigit@intel.com> wrote:
>>
>>> For the ABI compatibility it is better to hide internal data structures
>>> from the application as much as possible. But because of some inline
>>> functions 'struct eth_dev_ops' can't be hidden completely.
>>>
>>> Plan is to split the 'struct eth_dev_ops' into two as ones used by
>>> inline functions and ones not used, and hide the second part that not
>>> used by inline functions completely to the application.
>>>
>>> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
>>> ---
>>> Cc: David Marchand <david.marchand@redhat.com>
>>> Cc: Thomas Monjalon <thomas@monjalon.net>
>>> Cc: Andrew Rybchenko <arybchenko@solarflare.com>
>>> ---
>>> doc/guides/rel_notes/deprecation.rst | 6 ++++++
>>> 1 file changed, 6 insertions(+)
>>>
>>> diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
>>> index dfcca87ab..2aa431028 100644
>>> --- a/doc/guides/rel_notes/deprecation.rst
>>> +++ b/doc/guides/rel_notes/deprecation.rst
>>> @@ -72,6 +72,12 @@ Deprecation Notices
>>> In 19.11 PMDs will still update the field even when the offload is not
>>> enabled.
>>>
>>> +* ethdev: Split the ``struct eth_dev_ops`` struct to hide it as much as possible.
>>> + Currently the ``struct eth_dev_ops`` struct is accessible by the application
>>> + because some inline functions, like ``rte_eth_tx_descriptor_status()``,
>>> + access the struct directly. The struct will be separate in two, the ops used
>>> + by inline functions still will be accessible to user but rest will be hidden.
>>> +
>>> * cryptodev: support for using IV with all sizes is added, J0 still can
>>> be used but only when IV length in following structs ``rte_crypto_auth_xform``,
>>> ``rte_crypto_aead_xform`` is set to zero. When IV length is greater or equal
>>
>> Good luck, truely hiding internals is hard. The mbuf structure is already split but not really
>> hidden at all (just look at dwarf output). It doesn't make sense to do it unless
>> you can really hide it.
>
> I believe this can be done, only following [1] dev_ops are used by inline
> functions, rest can be moved into separate struct and moved into ethdev driver
> looking header.
>
> [1]
> rx_queue_count
> rx_descriptor_done
> rx_descriptor_status
> tx_descriptor_status
I think having 3 places (if I understand the intention
correctly) with ethdev callbacks is too much. So, I think
that these ops should be simply moved to nearby Tx/Rx
burst and Tx prepare callbacks (e.g. move into inline_ops
structure which is located at the beginning of rte_eth_dev
in order to preserve 3 existing callback location).
Also I'd consider to deprecate and remove rx_queue_count
and rx_descriptor_done.
>>
>> I would attack the rte_device stuff first. Make rte_device opaque to the application
>> that would help for future versions. Then work backwards to rte_tehtdev.
>>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v2 0/7] vfio/pci: SR-IOV support
2020-02-19 18:53 2% [dpdk-dev] [PATCH v2 0/7] vfio/pci: SR-IOV support Alex Williamson
@ 2020-02-25 2:33 0% ` Tian, Kevin
2020-03-05 17:33 3% ` Alex Williamson
2020-03-05 6:38 0% ` Vamsi Krishna Attunuru
1 sibling, 1 reply; 200+ results
From: Tian, Kevin @ 2020-02-25 2:33 UTC (permalink / raw)
To: Alex Williamson, kvm
Cc: linux-pci, linux-kernel, dev, mtosatti, thomas, bluca,
jerinjacobk, Richardson, Bruce, cohuck, Jason Wang
> From: Alex Williamson
> Sent: Thursday, February 20, 2020 2:54 AM
>
> Changes since v1 are primarily to patch 3/7 where the commit log is
> rewritten, along with option parsing and failure logging based on
> upstream discussions. The primary user visible difference is that
> option parsing is now much more strict. If a vf_token option is
> provided that cannot be used, we generate an error. As a result of
> this, opening a PF with a vf_token option will serve as a mechanism of
> setting the vf_token. This seems like a more user friendly API than
> the alternative of sometimes requiring the option (VFs in use) and
> sometimes rejecting it, and upholds our desire that the option is
> always either used or rejected.
>
> This also means that the VFIO_DEVICE_FEATURE ioctl is not the only
> means of setting the VF token, which might call into question whether
> we absolutely need this new ioctl. Currently I'm keeping it because I
> can imagine use cases, for example if a hypervisor were to support
> SR-IOV, the PF device might be opened without consideration for a VF
> token and we'd require the hypservisor to close and re-open the PF in
> order to set a known VF token, which is impractical.
>
> Series overview (same as provided with v1):
Thanks for doing this!
>
> The synopsis of this series is that we have an ongoing desire to drive
> PCIe SR-IOV PFs from userspace with VFIO. There's an immediate need
> for this with DPDK drivers and potentially interesting future use
Can you provide a link to the DPDK discussion?
> cases in virtualization. We've been reluctant to add this support
> previously due to the dependency and trust relationship between the
> VF device and PF driver. Minimally the PF driver can induce a denial
> of service to the VF, but depending on the specific implementation,
> the PF driver might also be responsible for moving data between VFs
> or have direct access to the state of the VF, including data or state
> otherwise private to the VF or VF driver.
Just a loud thinking. While the motivation of VF token sounds reasonable
to me, I'm curious why the same concern is not raised in other usages.
For example, there is no such design in virtio framework, where the
virtio device could also be restarted, putting in separate process (vhost-user),
and even in separate VM (virtio-vhost-user), etc. Of course the para-
virtualized attribute of virtio implies some degree of trust, but as you
mentioned many SR-IOV implementations support VF->PF communication
which also implies some level of trust. It's perfectly fine if VFIO just tries
to do better than other sub-systems, but knowing how other people
tackle the similar problem may make the whole picture clearer. 😊
+Jason.
>
> To help resolve these concerns, we introduce a VF token into the VFIO
> PCI ABI, which acts as a shared secret key between drivers. The
> userspace PF driver is required to set the VF token to a known value
> and userspace VF drivers are required to provide the token to access
> the VF device. If a PF driver is restarted with VF drivers in use, it
> must also provide the current token in order to prevent a rogue
> untrusted PF driver from replacing a known driver. The degree to
> which this new token is considered secret is left to the userspace
> drivers, the kernel intentionally provides no means to retrieve the
> current token.
I'm wondering whether the token idea can be used beyond SR-IOV, e.g.
(1) we may allow vfio user space to manage Scalable IOV in the future,
which faces the similar challenge between the PF and mdev; (2) the
token might be used as a canonical way to replace off-tree acs-override
workaround, say, allowing the admin to assign devices within the
same iommu group to different VMs which trust each other. I'm not
sure how much complexity will be further introduced, but it's greatly
appreciated if you can help think a bit and if feasible abstract some
logic in vfio core layer for such potential usages...
>
> Note that the above token is only required for this new model where
> both the PF and VF devices are usable through vfio-pci. Existing
> models of VFIO drivers where the PF is used without SR-IOV enabled
> or the VF is bound to a userspace driver with an in-kernel, host PF
> driver are unaffected.
>
> The latter configuration above also highlights a new inverted scenario
> that is now possible, a userspace PF driver with in-kernel VF drivers.
> I believe this is a scenario that should be allowed, but should not be
> enabled by default. This series includes code to set a default
> driver_override for VFs sourced from a vfio-pci user owned PF, such
> that the VFs are also bound to vfio-pci. This model is compatible
> with tools like driverctl and allows the system administrator to
> decide if other bindings should be enabled. The VF token interface
> above exists only between vfio-pci PF and VF drivers, once a VF is
> bound to another driver, the administrator has effectively pronounced
> the device as trusted. The vfio-pci driver will note alternate
> binding in dmesg for logging and debugging purposes.
>
> Please review, comment, and test. The example QEMU implementation
> provided with the RFC is still current for this version. Thanks,
>
> Alex
>
> RFC:
> https://lore.kernel.org/lkml/158085337582.9445.17682266437583505502.stg
> it@gimli.home/
> v1:
> https://lore.kernel.org/lkml/158145472604.16827.15751375540102298130.st
> git@gimli.home/
>
> ---
>
> Alex Williamson (7):
> vfio: Include optional device match in vfio_device_ops callbacks
> vfio/pci: Implement match ops
> vfio/pci: Introduce VF token
> vfio: Introduce VFIO_DEVICE_FEATURE ioctl and first user
> vfio/pci: Add sriov_configure support
> vfio/pci: Remove dev_fmt definition
> vfio/pci: Cleanup .probe() exit paths
>
>
> drivers/vfio/pci/vfio_pci.c | 383
> +++++++++++++++++++++++++++++++++--
> drivers/vfio/pci/vfio_pci_private.h | 10 +
> drivers/vfio/vfio.c | 20 +-
> include/linux/vfio.h | 4
> include/uapi/linux/vfio.h | 37 +++
> 5 files changed, 426 insertions(+), 28 deletions(-)
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [RFC 0/6] New sync modes for ring
2020-02-24 17:59 0% ` Jerin Jacob
2020-02-24 19:35 3% ` Stephen Hemminger
@ 2020-02-25 0:58 0% ` Honnappa Nagarahalli
2020-02-25 15:14 0% ` Ananyev, Konstantin
1 sibling, 1 reply; 200+ results
From: Honnappa Nagarahalli @ 2020-02-25 0:58 UTC (permalink / raw)
To: Jerin Jacob, Stephen Hemminger
Cc: Konstantin Ananyev, dpdk-dev, Olivier Matz, Honnappa Nagarahalli, nd, nd
<snip>
>
> On Mon, Feb 24, 2020 at 10:29 PM Stephen Hemminger
> <stephen@networkplumber.org> wrote:
> >
> > On Mon, 24 Feb 2020 11:35:09 +0000
> > Konstantin Ananyev <konstantin.ananyev@intel.com> wrote:
> >
> > > Upfront note - that RFC is not a complete patch.
> > > It introduces an ABI breakage, plus it doesn't update ring_elem code
> > > properly, etc.
> > > I plan to deal with all these things in later versions.
> > > Right now I seek an initial feedback about proposed ideas.
> > > Would also ask people to repeat performance tests (see below) on
> > > their platforms to confirm the impact.
> > >
> > > More and more customers use(/try to use) DPDK based apps within
> > > overcommitted systems (multiple acttive threads over same pysical cores):
> > > VM, container deployments, etc.
> > > One quite common problem they hit: Lock-Holder-Preemption with
> rte_ring.
> > > LHP is quite a common problem for spin-based sync primitives
> > > (spin-locks, etc.) on overcommitted systems.
> > > The situation gets much worse when some sort of fair-locking
> > > technique is used (ticket-lock, etc.).
> > > As now not only lock-owner but also lock-waiters scheduling order
> > > matters a lot.
> > > This is a well-known problem for kernel within VMs:
> > > http://www-archive.xenproject.org/files/xensummitboston08/LHP.pdf
> > > https://www.cs.hs-rm.de/~kaiser/events/wamos2017/Slides/selcuk.pdf
These slides seem to indicate that the problems are mitigated through the Hypervisor configuration. Do we still need to address the issues?
> > > The problem with rte_ring is that while head accusion is sort of
> > > un-fair locking, waiting on tail is very similar to ticket lock
> > > schema - tail has to be updated in particular order.
> > > That makes current rte_ring implementation to perform really pure on
> > > some overcommited scenarios.
> >
> > Rather than reform rte_ring to fit this scenario, it would make more
> > sense to me to introduce another primitive. The current lockless ring
> > performs very well for the isolated thread model that DPDK was built
> > around. This looks like a case of customers violating the usage model
> > of the DPDK and then being surprised at the fallout.
>
> I agree with Stephen here.
>
> I think, adding more runtime check in the enqueue() and dequeue() will have a
> bad effect on the low-end cores too.
> But I agree with the problem statement that in the virtualization use case, It
> may be possible to have N virtual cores runs on a physical core.
It is hard to imagine that there are data plane applications deployed in such environments. Wouldn't this affect the performance terribly?
>
> IMO, The best solution would be keeping the ring API same and have a
> different flavor in "compile-time". Something like liburcu did for
> accommodating different flavors.
>
> i.e urcu-qsbr.h and urcu-bp.h will identical definition of API. The application
> can simply include ONE header file in a C file based on the flavor.
> If need both at runtime. Need to have function pointer or so in the application
> and define the function in different c file by including the approaite flavor in C
> file.
>
> #include <urcu-qsbr.h> /* QSBR RCU flavor */ #include <urcu-bp.h> /*
> Bulletproof RCU flavor */
>
>
>
>
>
>
>
>
>
>
>
>
>
> >
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [RFC 0/6] New sync modes for ring
2020-02-24 19:35 3% ` Stephen Hemminger
@ 2020-02-24 20:52 0% ` Honnappa Nagarahalli
2020-02-25 11:45 0% ` Ananyev, Konstantin
2020-02-25 13:41 3% ` Ananyev, Konstantin
1 sibling, 1 reply; 200+ results
From: Honnappa Nagarahalli @ 2020-02-24 20:52 UTC (permalink / raw)
To: Stephen Hemminger, Jerin Jacob
Cc: Konstantin Ananyev, dpdk-dev, Olivier Matz, Honnappa Nagarahalli, nd, nd
> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Stephen Hemminger
> Sent: Monday, February 24, 2020 1:35 PM
> To: Jerin Jacob <jerinjacobk@gmail.com>
> Cc: Konstantin Ananyev <konstantin.ananyev@intel.com>; dpdk-dev
> <dev@dpdk.org>; Olivier Matz <olivier.matz@6wind.com>
> Subject: Re: [dpdk-dev] [RFC 0/6] New sync modes for ring
>
> On Mon, 24 Feb 2020 23:29:57 +0530
> Jerin Jacob <jerinjacobk@gmail.com> wrote:
>
> > On Mon, Feb 24, 2020 at 10:29 PM Stephen Hemminger
> > <stephen@networkplumber.org> wrote:
> > >
> > > On Mon, 24 Feb 2020 11:35:09 +0000
> > > Konstantin Ananyev <konstantin.ananyev@intel.com> wrote:
> > >
> > > > Upfront note - that RFC is not a complete patch.
> > > > It introduces an ABI breakage, plus it doesn't update ring_elem
> > > > code properly, etc.
> > > > I plan to deal with all these things in later versions.
> > > > Right now I seek an initial feedback about proposed ideas.
> > > > Would also ask people to repeat performance tests (see below) on
> > > > their platforms to confirm the impact.
> > > >
> > > > More and more customers use(/try to use) DPDK based apps within
> > > > overcommitted systems (multiple acttive threads over same pysical
> cores):
> > > > VM, container deployments, etc.
> > > > One quite common problem they hit: Lock-Holder-Preemption with
> rte_ring.
> > > > LHP is quite a common problem for spin-based sync primitives
> > > > (spin-locks, etc.) on overcommitted systems.
> > > > The situation gets much worse when some sort of fair-locking
> > > > technique is used (ticket-lock, etc.).
> > > > As now not only lock-owner but also lock-waiters scheduling order
> > > > matters a lot.
> > > > This is a well-known problem for kernel within VMs:
> > > > http://www-archive.xenproject.org/files/xensummitboston08/LHP.pdf
> > > > https://www.cs.hs-rm.de/~kaiser/events/wamos2017/Slides/selcuk.pdf
> > > > The problem with rte_ring is that while head accusion is sort of
> > > > un-fair locking, waiting on tail is very similar to ticket lock
> > > > schema - tail has to be updated in particular order.
> > > > That makes current rte_ring implementation to perform really pure
> > > > on some overcommited scenarios.
> > >
> > > Rather than reform rte_ring to fit this scenario, it would make more
> > > sense to me to introduce another primitive. The current lockless
> > > ring performs very well for the isolated thread model that DPDK was
> > > built around. This looks like a case of customers violating the
> > > usage model of the DPDK and then being surprised at the fallout.
> >
> > I agree with Stephen here.
> >
> > I think, adding more runtime check in the enqueue() and dequeue() will
> > have a bad effect on the low-end cores too.
> > But I agree with the problem statement that in the virtualization use
> > case, It may be possible to have N virtual cores runs on a physical
> > core.
> >
> > IMO, The best solution would be keeping the ring API same and have a
> > different flavor in "compile-time". Something like liburcu did for
> > accommodating different flavors.
> >
> > i.e urcu-qsbr.h and urcu-bp.h will identical definition of API. The
> > application can simply include ONE header file in a C file based on
> > the flavor.
> > If need both at runtime. Need to have function pointer or so in the
> > application and define the function in different c file by including
> > the approaite flavor in C file.
>
> This would also be a good time to consider the tradeoffs of the heavy use of
> inlining that is done in rte_ring vs the impact that has on API/ABI stability.
>
I was working on few requirements in rte_ring library for RCU defer APIs. RFC is at https://patchwork.dpdk.org/cover/66020/.
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [RFC 0/6] New sync modes for ring
2020-02-24 17:59 0% ` Jerin Jacob
@ 2020-02-24 19:35 3% ` Stephen Hemminger
2020-02-24 20:52 0% ` Honnappa Nagarahalli
2020-02-25 13:41 3% ` Ananyev, Konstantin
2020-02-25 0:58 0% ` Honnappa Nagarahalli
1 sibling, 2 replies; 200+ results
From: Stephen Hemminger @ 2020-02-24 19:35 UTC (permalink / raw)
To: Jerin Jacob; +Cc: Konstantin Ananyev, dpdk-dev, Olivier Matz
On Mon, 24 Feb 2020 23:29:57 +0530
Jerin Jacob <jerinjacobk@gmail.com> wrote:
> On Mon, Feb 24, 2020 at 10:29 PM Stephen Hemminger
> <stephen@networkplumber.org> wrote:
> >
> > On Mon, 24 Feb 2020 11:35:09 +0000
> > Konstantin Ananyev <konstantin.ananyev@intel.com> wrote:
> >
> > > Upfront note - that RFC is not a complete patch.
> > > It introduces an ABI breakage, plus it doesn't update ring_elem
> > > code properly, etc.
> > > I plan to deal with all these things in later versions.
> > > Right now I seek an initial feedback about proposed ideas.
> > > Would also ask people to repeat performance tests (see below)
> > > on their platforms to confirm the impact.
> > >
> > > More and more customers use(/try to use) DPDK based apps within
> > > overcommitted systems (multiple acttive threads over same pysical cores):
> > > VM, container deployments, etc.
> > > One quite common problem they hit: Lock-Holder-Preemption with rte_ring.
> > > LHP is quite a common problem for spin-based sync primitives
> > > (spin-locks, etc.) on overcommitted systems.
> > > The situation gets much worse when some sort of
> > > fair-locking technique is used (ticket-lock, etc.).
> > > As now not only lock-owner but also lock-waiters scheduling
> > > order matters a lot.
> > > This is a well-known problem for kernel within VMs:
> > > http://www-archive.xenproject.org/files/xensummitboston08/LHP.pdf
> > > https://www.cs.hs-rm.de/~kaiser/events/wamos2017/Slides/selcuk.pdf
> > > The problem with rte_ring is that while head accusion is sort of
> > > un-fair locking, waiting on tail is very similar to ticket lock schema -
> > > tail has to be updated in particular order.
> > > That makes current rte_ring implementation to perform
> > > really pure on some overcommited scenarios.
> >
> > Rather than reform rte_ring to fit this scenario, it would make
> > more sense to me to introduce another primitive. The current lockless
> > ring performs very well for the isolated thread model that DPDK
> > was built around. This looks like a case of customers violating
> > the usage model of the DPDK and then being surprised at the fallout.
>
> I agree with Stephen here.
>
> I think, adding more runtime check in the enqueue() and dequeue() will
> have a bad effect on the low-end cores too.
> But I agree with the problem statement that in the virtualization use
> case, It may be possible to have N virtual cores runs on a physical
> core.
>
> IMO, The best solution would be keeping the ring API same and have a
> different flavor in "compile-time". Something like
> liburcu did for accommodating different flavors.
>
> i.e urcu-qsbr.h and urcu-bp.h will identical definition of API. The
> application can simply include ONE header file in a C file based on
> the flavor.
> If need both at runtime. Need to have function pointer or so in the
> application and define the function in different c file by including
> the approaite flavor in C file.
This would also be a good time to consider the tradeoffs of the
heavy use of inlining that is done in rte_ring vs the impact that
has on API/ABI stability.
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [RFC 0/6] New sync modes for ring
2020-02-24 16:59 0% ` Stephen Hemminger
@ 2020-02-24 17:59 0% ` Jerin Jacob
2020-02-24 19:35 3% ` Stephen Hemminger
2020-02-25 0:58 0% ` Honnappa Nagarahalli
0 siblings, 2 replies; 200+ results
From: Jerin Jacob @ 2020-02-24 17:59 UTC (permalink / raw)
To: Stephen Hemminger; +Cc: Konstantin Ananyev, dpdk-dev, Olivier Matz
On Mon, Feb 24, 2020 at 10:29 PM Stephen Hemminger
<stephen@networkplumber.org> wrote:
>
> On Mon, 24 Feb 2020 11:35:09 +0000
> Konstantin Ananyev <konstantin.ananyev@intel.com> wrote:
>
> > Upfront note - that RFC is not a complete patch.
> > It introduces an ABI breakage, plus it doesn't update ring_elem
> > code properly, etc.
> > I plan to deal with all these things in later versions.
> > Right now I seek an initial feedback about proposed ideas.
> > Would also ask people to repeat performance tests (see below)
> > on their platforms to confirm the impact.
> >
> > More and more customers use(/try to use) DPDK based apps within
> > overcommitted systems (multiple acttive threads over same pysical cores):
> > VM, container deployments, etc.
> > One quite common problem they hit: Lock-Holder-Preemption with rte_ring.
> > LHP is quite a common problem for spin-based sync primitives
> > (spin-locks, etc.) on overcommitted systems.
> > The situation gets much worse when some sort of
> > fair-locking technique is used (ticket-lock, etc.).
> > As now not only lock-owner but also lock-waiters scheduling
> > order matters a lot.
> > This is a well-known problem for kernel within VMs:
> > http://www-archive.xenproject.org/files/xensummitboston08/LHP.pdf
> > https://www.cs.hs-rm.de/~kaiser/events/wamos2017/Slides/selcuk.pdf
> > The problem with rte_ring is that while head accusion is sort of
> > un-fair locking, waiting on tail is very similar to ticket lock schema -
> > tail has to be updated in particular order.
> > That makes current rte_ring implementation to perform
> > really pure on some overcommited scenarios.
>
> Rather than reform rte_ring to fit this scenario, it would make
> more sense to me to introduce another primitive. The current lockless
> ring performs very well for the isolated thread model that DPDK
> was built around. This looks like a case of customers violating
> the usage model of the DPDK and then being surprised at the fallout.
I agree with Stephen here.
I think, adding more runtime check in the enqueue() and dequeue() will
have a bad effect on the low-end cores too.
But I agree with the problem statement that in the virtualization use
case, It may be possible to have N virtual cores runs on a physical
core.
IMO, The best solution would be keeping the ring API same and have a
different flavor in "compile-time". Something like
liburcu did for accommodating different flavors.
i.e urcu-qsbr.h and urcu-bp.h will identical definition of API. The
application can simply include ONE header file in a C file based on
the flavor.
If need both at runtime. Need to have function pointer or so in the
application and define the function in different c file by including
the approaite flavor in C file.
#include <urcu-qsbr.h> /* QSBR RCU flavor */
#include <urcu-bp.h> /* Bulletproof RCU flavor */
>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [RFC 0/6] New sync modes for ring
2020-02-24 11:35 2% [dpdk-dev] [RFC 0/6] New sync modes for ring Konstantin Ananyev
@ 2020-02-24 16:59 0% ` Stephen Hemminger
2020-02-24 17:59 0% ` Jerin Jacob
0 siblings, 1 reply; 200+ results
From: Stephen Hemminger @ 2020-02-24 16:59 UTC (permalink / raw)
To: Konstantin Ananyev; +Cc: dev, olivier.matz
On Mon, 24 Feb 2020 11:35:09 +0000
Konstantin Ananyev <konstantin.ananyev@intel.com> wrote:
> Upfront note - that RFC is not a complete patch.
> It introduces an ABI breakage, plus it doesn't update ring_elem
> code properly, etc.
> I plan to deal with all these things in later versions.
> Right now I seek an initial feedback about proposed ideas.
> Would also ask people to repeat performance tests (see below)
> on their platforms to confirm the impact.
>
> More and more customers use(/try to use) DPDK based apps within
> overcommitted systems (multiple acttive threads over same pysical cores):
> VM, container deployments, etc.
> One quite common problem they hit: Lock-Holder-Preemption with rte_ring.
> LHP is quite a common problem for spin-based sync primitives
> (spin-locks, etc.) on overcommitted systems.
> The situation gets much worse when some sort of
> fair-locking technique is used (ticket-lock, etc.).
> As now not only lock-owner but also lock-waiters scheduling
> order matters a lot.
> This is a well-known problem for kernel within VMs:
> http://www-archive.xenproject.org/files/xensummitboston08/LHP.pdf
> https://www.cs.hs-rm.de/~kaiser/events/wamos2017/Slides/selcuk.pdf
> The problem with rte_ring is that while head accusion is sort of
> un-fair locking, waiting on tail is very similar to ticket lock schema -
> tail has to be updated in particular order.
> That makes current rte_ring implementation to perform
> really pure on some overcommited scenarios.
Rather than reform rte_ring to fit this scenario, it would make
more sense to me to introduce another primitive. The current lockless
ring performs very well for the isolated thread model that DPDK
was built around. This looks like a case of customers violating
the usage model of the DPDK and then being surprised at the fallout.
^ permalink raw reply [relevance 0%]
* [dpdk-dev] [RFC 0/6] New sync modes for ring
@ 2020-02-24 11:35 2% Konstantin Ananyev
2020-02-24 16:59 0% ` Stephen Hemminger
0 siblings, 1 reply; 200+ results
From: Konstantin Ananyev @ 2020-02-24 11:35 UTC (permalink / raw)
To: dev; +Cc: olivier.matz, Konstantin Ananyev
Upfront note - that RFC is not a complete patch.
It introduces an ABI breakage, plus it doesn't update ring_elem
code properly, etc.
I plan to deal with all these things in later versions.
Right now I seek an initial feedback about proposed ideas.
Would also ask people to repeat performance tests (see below)
on their platforms to confirm the impact.
More and more customers use(/try to use) DPDK based apps within
overcommitted systems (multiple acttive threads over same pysical cores):
VM, container deployments, etc.
One quite common problem they hit: Lock-Holder-Preemption with rte_ring.
LHP is quite a common problem for spin-based sync primitives
(spin-locks, etc.) on overcommitted systems.
The situation gets much worse when some sort of
fair-locking technique is used (ticket-lock, etc.).
As now not only lock-owner but also lock-waiters scheduling
order matters a lot.
This is a well-known problem for kernel within VMs:
http://www-archive.xenproject.org/files/xensummitboston08/LHP.pdf
https://www.cs.hs-rm.de/~kaiser/events/wamos2017/Slides/selcuk.pdf
The problem with rte_ring is that while head accusion is sort of
un-fair locking, waiting on tail is very similar to ticket lock schema -
tail has to be updated in particular order.
That makes current rte_ring implementation to perform
really pure on some overcommited scenarios.
While it is probably not possible to completely resolve this problem in
userspace only (without some kernel communication/intervention),
removing fairness in tail update can mitigate it significantly.
So this RFC proposes two new optional ring synchronization modes:
1) Head/Tail Sync (HTS) mode
In that mode enqueue/dequeue operation is fully serialized:
only one thread at a time is allowed to perform given op.
As another enhancement provide ability to split enqueue/dequeue
operation into two phases:
- enqueue/dequeue start
- enqueue/dequeue finish
That allows user to inspect objects in the ring without removing
them from it (aka MT safe peek).
2) Relaxed Tail Sync (RTS)
The main difference from original MP/MC algorithm is that
tail value is increased not by every thread that finished enqueue/dequeue,
but only by the last one.
That allows threads to avoid spinning on ring tail value,
leaving actual tail value change to the last thread in the update queue.
Test results on IA (see below) show significant improvements
for average enqueue/dequeue op times on overcommitted systems.
For 'classic' DPDK deployments (one thread per core) original MP/MC
algorithm still shows best numbers, though for 64-bit target
RTS numbers are not that far away.
Numbers were produced by ring_stress_*autotest
(first patch in these series).
X86_64 @ Intel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz
DEQ+ENQ average cycles/obj
MP/MC HTS RTS
1thread@1core(--lcores=6-7) 8.00 8.15 8.99
2thread@2core(--lcores=6-8) 19.14 19.61 20.35
4thread@4core(--lcores=6-10) 29.43 29.79 31.82
8thread@8core(--lcores=6-14) 110.59 192.81 119.50
16thread@16core(--lcores=6-22) 461.03 813.12 495.59
32thread/@32core(--lcores='6-22,55-70') 982.90 1972.38 1160.51
2thread@1core(--lcores='6,(10-11)@7' 20140.50 23.58 25.14
4thread@2core(--lcores='6,(10-11)@7,(20-21)@8' 153680.60 76.88 80.05
8thread@2core(--lcores='6,(10-13)@7,(20-23)@8' 280314.32 294.72 318.79
16thread@2core(--lcores='6,(10-17)@7,(20-27)@8' 643176.59 1144.02 1175.14
32thread@2core(--lcores='6,(10-25)@7,(30-45)@8' 4264238.80 4627.48 4892.68
8thread@2core(--lcores='6,(10-17)@(7,8))' 321085.98 298.59 307.47
16thread@4core(--lcores='6,(20-35)@(7-10))' 1900705.61 575.35 678.29
32thread@4core(--lcores='6,(20-51)@(7-10))' 5510445.85 2164.36 2714.12
i686 @ Intel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz
DEQ+ENQ average cycles/obj
MP/MC HTS RTS
1thread@1core(--lcores=6-7) 7.85 12.13 11.31
2thread@2core(--lcores=6-8) 17.89 24.52 21.86
8thread@8core(--lcores=6-14) 32.58 354.20 54.58
32thread/@32core(--lcores='6-22,55-70') 813.77 6072.41 2169.91
2thread@1core(--lcores='6,(10-11)@7' 16095.00 36.06 34.74
8thread@2core(--lcores='6,(10-13)@7,(20-23)@8' 1140354.54 346.61 361.57
16thread@2core(--lcores='6,(10-17)@7,(20-27)@8' 1920417.86 1314.90 1416.65
8thread@2core(--lcores='6,(10-17)@(7,8))' 594358.61 332.70 357.74
32thread@4core(--lcores='6,(20-51)@(7-10))' 5319896.86 2836.44 3028.87
Konstantin Ananyev (6):
test/ring: add contention stress test
ring: rework ring layout to allow new sync schemes
ring: introduce RTS ring mode
test/ring: add contention stress test for RTS ring
ring: introduce HTS ring mode
test/ring: add contention stress test for HTS ring
app/test/Makefile | 3 +
app/test/meson.build | 3 +
app/test/test_pdump.c | 6 +-
app/test/test_ring_hts_stress.c | 28 ++
app/test/test_ring_rts_stress.c | 28 ++
app/test/test_ring_stress.c | 27 ++
app/test/test_ring_stress.h | 477 +++++++++++++++++++
lib/librte_pdump/rte_pdump.c | 2 +-
lib/librte_port/rte_port_ring.c | 12 +-
lib/librte_ring/Makefile | 4 +-
lib/librte_ring/meson.build | 4 +-
lib/librte_ring/rte_ring.c | 84 +++-
lib/librte_ring/rte_ring.h | 619 +++++++++++++++++++++++--
lib/librte_ring/rte_ring_elem.h | 8 +-
lib/librte_ring/rte_ring_hts_generic.h | 228 +++++++++
lib/librte_ring/rte_ring_rts_generic.h | 240 ++++++++++
16 files changed, 1721 insertions(+), 52 deletions(-)
create mode 100644 app/test/test_ring_hts_stress.c
create mode 100644 app/test/test_ring_rts_stress.c
create mode 100644 app/test/test_ring_stress.c
create mode 100644 app/test/test_ring_stress.h
create mode 100644 lib/librte_ring/rte_ring_hts_generic.h
create mode 100644 lib/librte_ring/rte_ring_rts_generic.h
--
2.17.1
^ permalink raw reply [relevance 2%]
* Re: [dpdk-dev] ABI version of experimental libraries
2020-02-21 16:57 4% ` Thomas Monjalon
@ 2020-02-24 9:32 4% ` Ray Kinsella
0 siblings, 0 replies; 200+ results
From: Ray Kinsella @ 2020-02-24 9:32 UTC (permalink / raw)
To: Thomas Monjalon, dev
On 21/02/2020 16:57, Thomas Monjalon wrote:
> 19/02/2020 14:50, Ray Kinsella:
>> On 19/02/2020 12:43, Thomas Monjalon wrote:
>>> 19/02/2020 12:43, Neil Horman:
>>>> On Tue, Feb 18, 2020 at 10:50:09AM +0100, Thomas Monjalon wrote:
>>>>> 18/02/2020 10:42, Bruce Richardson:
>>>>>> On Tue, Feb 18, 2020 at 12:15:56AM +0100, Thomas Monjalon wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> I would like to remind everybody our mistake when defining ABI versions.
>>>>>>> It has been "fixed" in this commit:
>>>>>>> http://git.dpdk.org/dpdk/commit/?id=f26c2b39
>>>>>>>
>>>>>>> Please let's think about the consequence for the experimental libraries.
>>>>>>>
>>>>>>> In DPDK 19.11, we use the ABI version 0.200 with soname 0.20 In DPDK
>>>>>>> 20.02, we use the ABI version 0.2001 with soname 0.201 Numbers are
>>>>>>> increasing, that's fine. When we'll switch to the new major ABI and use
>>>>>>> a normal numbering: In DPDK 20.11, we will use the ABI version 0.210 with
>>>>>>> soname 0.21 Numbers are dropping.
>>>>>>>
>>>>>>> In short, for experimental libs, ABI 20.1 > ABI 21.0
>>>>>>>
>>>>>>> Are we OK with this or do we prefer reverting to normal numbering for
>>>>>>> experimental libraries in DPDK 20.02?
>>>>>>>
>>>>>> Personally, I would not be too concerned about the verions of experimental
>>>>>> libs, so long as they don't conflict across versions and have some
>>>>>> similarity to the major ABI version for the release.
>>>>>
>>>>> You think sorting of the version numbers is not important?
>>>>> If we don't care comparing experimental version numbers,
>>>>> then OK, let's drop this patch. But please we need a small vote.
>>>>>
>>>>> Note: there would be no problem if we did not vote for having
>>>>> a special numbering for pure experimental libraries (I am still against).
>>>>>
>>>> I don't understand. Why would we change the ABI_VERSION at all in an LTS release at
>>>> all? This operation is meant to take an an experimental API and mark it as
>>>> stable by promoting its version number to the next major releases number. As
>>>> such, in the LTS release, we should keep the soname the same, as there should be
>>>> no other ABI changes in the promoted API.
>>>
>>> The library version number is updated because we add new symbols.
>>>
>>>
>>
>> So while experimental library version numbers are not "important".
>> I do agree with Thomas they should be sane, increase and should have a consistent format.
>>
>> Should we always just pad them to 4 places as the simple solution?
>> i.e.
>>
>> DPDK 19.11 ... 0.20 (needs to remain 0.20).
>> DPDK 20.02 ... 0.2001
>> DPDK 20.11 ... 0.2100
>> DPDK 21.02 ... 0.2101
>
> A patch from Ferruh got merged.
> It is adding a dot to keep versioning consistent.
>
> Marking this patch as rejected.
>
Ferruh's was a better solution.
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH] doc: deprecate using MAX values as array size
2020-02-21 10:25 0% ` Ferruh Yigit
2020-02-24 6:18 0% ` Akhil Goyal
@ 2020-02-24 8:51 0% ` Andrew Rybchenko
1 sibling, 0 replies; 200+ results
From: Andrew Rybchenko @ 2020-02-24 8:51 UTC (permalink / raw)
To: Ferruh Yigit, Neil Horman, John McNamara, Marko Kovacevic
Cc: dev, Thomas Monjalon, Bruce Richardson, Akhil Goyal,
Konstantin Ananyev, David Marchand, Jerin Jacob, Ori Kam
On 2/21/20 1:25 PM, Ferruh Yigit wrote:
> On 1/30/2020 2:20 PM, Ferruh Yigit wrote:
>> Adding the deprecation notice as reminder for next ABI breakage release
>> (20.11).
>> This one time breakage is required to be able to extend enum/define
>> without breaking ABI.
>>
>> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
>> ---
>> doc/guides/rel_notes/deprecation.rst | 14 ++++++++++++++
>> 1 file changed, 14 insertions(+)
>>
>> diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
>> index dfcca87ab..99d81564a 100644
>> --- a/doc/guides/rel_notes/deprecation.rst
>> +++ b/doc/guides/rel_notes/deprecation.rst
>> @@ -38,6 +38,20 @@ Deprecation Notices
>> remove it from the externally visible ABI and allow it to be updated in the
>> future.
>>
>> +* lib: will fix extending some enum/define breaking the ABI. There are multiple
>> + samples in DPDK that enum/define terminated with a ``.*MAX.*`` value which is
>> + used by iterators, and arrays holding these values are sized with this
>> + ``.*MAX.*`` value. So extending this enum/define increases the ``.*MAX.*``
>> + value which increases the size of the array and depending on how/where the
>> + array is used this may break the ABI.
>> + ``RTE_ETH_FLOW_MAX`` is one sample of the mentioned case, adding a new flow
>> + type will break the ABI because of ``flex_mask[RTE_ETH_FLOW_MAX]`` array
>> + usage in following public struct hierarchy:
>> + ``rte_eth_fdir_flex_conf -> rte_fdir_conf -> rte_eth_conf (in the middle)``.
>> + Need to identify this kind of usages and fix in 20.11, otherwise this blocks
>> + us extending existing enum/define.
>> + One solution can be using a fixed size array instead of ``.*MAX.*`` value.
>> +
>> * dpaa2: removal of ``rte_dpaa2_memsegs`` structure which has been replaced
>> by a pa-va search library. This structure was earlier being used for holding
>> memory segments used by dpaa2 driver for faster pa->va translation. This
>>
> Reminder of this deprecation notice.
Acked-by: Andrew Rybchenko <arybchenko@oktetlabs.ru>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [dpdk-techboard] [PATCH] doc: deprecation notice to move igb_uio to a new repo
@ 2020-02-24 7:03 0% ` Maxime Coquelin
0 siblings, 0 replies; 200+ results
From: Maxime Coquelin @ 2020-02-24 7:03 UTC (permalink / raw)
To: jerinj, dev, Neil Horman, John McNamara, Marko Kovacevic
Cc: thomas, techboard, ferruh.yigit
On 1/30/20 5:58 PM, jerinj@marvell.com wrote:
> From: Jerin Jacob <jerinj@marvell.com>
>
> Based on the tech board meeting held on 2019-11-06,
> It's been decided to move igb_uio kernel module to a new repository
> hosted by dpdk.org in v20.11 release.
>
> http://mails.dpdk.org/archives/dev/2019-November/151763.html
>
> Adding the deprecation notice for the same in advance.
>
> Signed-off-by: Jerin Jacob <jerinj@marvell.com>
> ---
> doc/guides/rel_notes/deprecation.rst | 6 ++++++
> 1 file changed, 6 insertions(+)
>
> diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
> index dfcca87ab..2b9d0113f 100644
> --- a/doc/guides/rel_notes/deprecation.rst
> +++ b/doc/guides/rel_notes/deprecation.rst
> @@ -38,6 +38,12 @@ Deprecation Notices
> remove it from the externally visible ABI and allow it to be updated in the
> future.
>
> +* igb_uio: In the view of reducing the kernel dependency from the main tree,
> + as a first step, The technical board has been decided to move ``igb_uio``
> + kernel module to a new repository hosted by dpdk.org in v20.11 release.
> + Minutes of Technical Board Meeting of `2019-11-06
> + <http://mails.dpdk.org/archives/dev/2019-November/151763.html>`_.
> +
> * dpaa2: removal of ``rte_dpaa2_memsegs`` structure which has been replaced
> by a pa-va search library. This structure was earlier being used for holding
> memory segments used by dpaa2 driver for faster pa->va translation. This
>
Acked-by: Maxime Coquelin <maxime.coquelin@redhat.com>
Thanks,
Maxime
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] doc: deprecate using MAX values as array size
2020-02-21 10:25 0% ` Ferruh Yigit
@ 2020-02-24 6:18 0% ` Akhil Goyal
2020-02-25 11:03 0% ` Thomas Monjalon
2020-02-24 8:51 0% ` Andrew Rybchenko
1 sibling, 1 reply; 200+ results
From: Akhil Goyal @ 2020-02-24 6:18 UTC (permalink / raw)
To: Ferruh Yigit, Neil Horman, John McNamara, Marko Kovacevic
Cc: dev, Thomas Monjalon, Bruce Richardson, Konstantin Ananyev,
David Marchand, Jerin Jacob, Andrew Rybchenko, Ori Kam
>
> On 1/30/2020 2:20 PM, Ferruh Yigit wrote:
> > Adding the deprecation notice as reminder for next ABI breakage release
> > (20.11).
> > This one time breakage is required to be able to extend enum/define
> > without breaking ABI.
> >
> > Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
> > ---
> > doc/guides/rel_notes/deprecation.rst | 14 ++++++++++++++
> > 1 file changed, 14 insertions(+)
> >
> > diff --git a/doc/guides/rel_notes/deprecation.rst
> b/doc/guides/rel_notes/deprecation.rst
> > index dfcca87ab..99d81564a 100644
> > --- a/doc/guides/rel_notes/deprecation.rst
> > +++ b/doc/guides/rel_notes/deprecation.rst
> > @@ -38,6 +38,20 @@ Deprecation Notices
> > remove it from the externally visible ABI and allow it to be updated in the
> > future.
> >
> > +* lib: will fix extending some enum/define breaking the ABI. There are
> multiple
> > + samples in DPDK that enum/define terminated with a ``.*MAX.*`` value
> which is
> > + used by iterators, and arrays holding these values are sized with this
> > + ``.*MAX.*`` value. So extending this enum/define increases the ``.*MAX.*``
> > + value which increases the size of the array and depending on how/where the
> > + array is used this may break the ABI.
> > + ``RTE_ETH_FLOW_MAX`` is one sample of the mentioned case, adding a
> new flow
> > + type will break the ABI because of ``flex_mask[RTE_ETH_FLOW_MAX]``
> array
> > + usage in following public struct hierarchy:
> > + ``rte_eth_fdir_flex_conf -> rte_fdir_conf -> rte_eth_conf (in the middle)``.
> > + Need to identify this kind of usages and fix in 20.11, otherwise this blocks
> > + us extending existing enum/define.
> > + One solution can be using a fixed size array instead of ``.*MAX.*`` value.
> > +
> > * dpaa2: removal of ``rte_dpaa2_memsegs`` structure which has been
> replaced
> > by a pa-va search library. This structure was earlier being used for holding
> > memory segments used by dpaa2 driver for faster pa->va translation. This
> >
>
We admit that the issue is there and we can discuss the possible solutions for this issue.
For the deprecation notice.
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
^ permalink raw reply [relevance 0%]
* [dpdk-dev] [PATCH v1] doc: update release notes for 20.02
@ 2020-02-23 10:59 8% John McNamara
0 siblings, 0 replies; 200+ results
From: John McNamara @ 2020-02-23 10:59 UTC (permalink / raw)
To: dev; +Cc: thomas, John McNamara
Fix grammar, spelling and formatting of DPDK 20.02 release notes.
Signed-off-by: John McNamara <john.mcnamara@intel.com>
---
doc/guides/rel_notes/release_20_02.rst | 70 +++++++++++++++-------------------
1 file changed, 31 insertions(+), 39 deletions(-)
diff --git a/doc/guides/rel_notes/release_20_02.rst b/doc/guides/rel_notes/release_20_02.rst
index dfebc46..8a493e4 100644
--- a/doc/guides/rel_notes/release_20_02.rst
+++ b/doc/guides/rel_notes/release_20_02.rst
@@ -74,7 +74,7 @@ New Features
Added the new ``ionic`` net driver for Pensando Ethernet Network Adapters.
See the :doc:`../nics/ionic` NIC guide for more details on this new driver.
-* **Updated Broadcom bnxt driver**
+* **Updated Broadcom bnxt driver.**
Updated Broadcom bnxt driver with new features and improvements, including:
@@ -91,10 +91,10 @@ New Features
Updated the Intel ice driver with new features and improvements, including:
- * Added support for MAC rules on specific port.
+ * Added support for MAC rules on a specific port.
* Added support for MAC/VLAN with TCP/UDP in switch rule.
* Added support for 1/10G device.
- * Added support for API rte_eth_tx_done_cleanup.
+ * Added support for API ``rte_eth_tx_done_cleanup``.
* **Updated Intel iavf driver.**
@@ -105,7 +105,9 @@ New Features
* **Updated the Intel ixgbe driver.**
- * Added support for API rte_eth_tx_done_cleanup.
+ Updated ixgbe PMD with new features and improvements, including:
+
+ * Added support for API ``rte_eth_tx_done_cleanup()``.
* Added support setting VF MAC address by PF driver.
* Added support for setting the link to specific speed.
@@ -122,7 +124,7 @@ New Features
* Extended PHY access AQ cmd.
* Added support for reading LPI counters.
* Added support for Energy Efficient Ethernet.
- * Added support for API rte_eth_tx_done_cleanup.
+ * Added support for API ``rte_eth_tx_done_cleanup()``.
* Added support for VF multiple queues interrupt.
* Added support for setting the link to specific speed.
@@ -135,7 +137,7 @@ New Features
* Removed limitation of matching on tagged/untagged packets (when using DV flow engine).
* Added BlueField-2 integrated ConnectX-6 Dx device support.
-* **Add new vDPA PMD based on Mellanox devices**
+* **Add new vDPA PMD based on Mellanox devices.**
Added a new Mellanox vDPA (``mlx5_vdpa``) PMD.
See the :doc:`../vdpadevs/mlx5` guide for more details on this driver.
@@ -155,6 +157,8 @@ New Features
* **Added algorithms to cryptodev API.**
+ Added new algorithms to the cryptodev API:
+
* ECDSA (Elliptic Curve Digital Signature Algorithm) is added to
asymmetric crypto library specifications.
* ECPM (Elliptic Curve Point Multiplication) is added to
@@ -162,8 +166,8 @@ New Features
* **Added synchronous Crypto burst API.**
- A new API is introduced in crypto library to handle synchronous cryptographic
- operations allowing to achieve performance gain for cryptodevs which use
+ A new API has been introduced in the crypto library to handle synchronous cryptographic
+ operations allowing it to achieve performance gains for cryptodevs which use
CPU based acceleration, such as Intel AES-NI. An implementation for aesni_gcm
cryptodev is provided. The IPsec example application and ipsec library itself
were changed to allow utilization of this new feature.
@@ -174,7 +178,7 @@ New Features
(generation) and cipher-hash (verification) requests (e.g. SNOW3G + ZUC or
ZUC + AES CTR) in QAT PMD possible when running on GEN3 QAT hardware.
Such algorithm combinations are not supported on GEN1/GEN2 hardware
- and executing the request returns RTE_CRYPTO_OP_STATUS_INVALID_SESSION.
+ and executing the request returns ``RTE_CRYPTO_OP_STATUS_INVALID_SESSION``.
* **Queue-pairs are now thread-safe on Intel QuickAssist Technology (QAT) PMD.**
@@ -185,34 +189,35 @@ New Features
* **Updated the ZUC PMD.**
- * Transistioned underlying library from libSSO ZUC to intel-ipsec-mb
+ * Transitioned underlying library from libSSO ZUC to intel-ipsec-mb
library (minimum version required 0.53).
* Removed dynamic library limitation, so PMD can be built as a shared
object now.
* **Updated the KASUMI PMD.**
- * Transistioned underlying library from libSSO KASUMI to intel-ipsec-mb
+ * Transitioned underlying library from libSSO KASUMI to intel-ipsec-mb
library (minimum version required 0.53).
* **Updated the SNOW3G PMD.**
- * Transistioned underlying library from libSSO SNOW3G to intel-ipsec-mb
+ * Transitioned underlying library from libSSO SNOW3G to intel-ipsec-mb
library (minimum version required 0.53).
* **Changed armv8 crypto PMD external dependency.**
- armv8 crypto PMD now depends on Arm crypto library, and Marvell's
- armv8 crypto library is not used anymore. Library name is changed
+ Changed armv8 crypto PMD external dependency. The
+ armv8 crypto PMD now depends on the Arm crypto library, and Marvell's
+ armv8 crypto library is not used anymore. The library name has been changed
from armv8_crypto to AArch64crypto.
* **Added inline IPsec support to Marvell OCTEON TX2 PMD.**
- Added inline IPsec support to Marvell OCTEON TX2 PMD. With the feature,
- applications would be able to offload entire IPsec offload to the hardware.
+ Added inline IPsec support to Marvell OCTEON TX2 PMD. With this feature,
+ applications will be able to offload entire IPsec offload to the hardware.
For the configured sessions, hardware will do the lookup and perform
- decryption and IPsec transformation. For the outbound path, application
- can submit a plain packet to the PMD, and it would be sent out on wire
+ decryption and IPsec transformation. For the outbound path, applications
+ can submit a plain packet to the PMD, and it will be sent out on the wire
after doing encryption and IPsec transformation of the packet.
* **Added Marvell OCTEON TX2 End Point rawdev PMD.**
@@ -222,7 +227,7 @@ New Features
* **Added event mode to l3fwd sample application.**
- Add event device support for ``l3fwd`` sample application. It demonstrates
+ Added event device support for the ``l3fwd`` sample application. It demonstrates
usage of poll and event mode IO mechanism under a single application.
* **Added cycle-count mode to the compression performance tool.**
@@ -232,8 +237,8 @@ New Features
* **Added OpenWrt howto guide.**
- Added document describes how to enable DPDK on OpenWrt in both virtual and
- physical machine.
+ Added document which describes how to enable DPDK on OpenWrt in both virtual and
+ physical machines.
Removed Items
@@ -250,7 +255,7 @@ Removed Items
* **Disabled building all the Linux kernel modules by default.**
- In order to remove the build time dependency with Linux kernel,
+ In order to remove the build time dependency on the Linux kernel,
the Technical Board decided to disable all the kernel modules
by default from 20.02 version.
@@ -259,8 +264,8 @@ Removed Items
The internal tail write coalescing feature was removed as not compatible with
dual-thread feature. It was replaced with a threshold feature. At busy times
if only a small number of packets can be enqueued, each enqueue causes
- an expensive MMIO write. These MMIO write occurrences can be optimised by using
- the new threshold parameter on process start. Please see qat documentation for
+ an expensive MMIO write. These MMIO write occurrences can be optimized by using
+ the new threshold parameter on process start. Please see QAT documentation for
more details.
@@ -279,6 +284,8 @@ API Changes
Also, make sure to start the actual text at the margin.
=========================================================
+* No changes in this release.
+
ABI Changes
-----------
@@ -298,21 +305,6 @@ ABI Changes
* No change, kept ABI v20. DPDK 20.02 is compatible with DPDK 19.11.
-Known Issues
-------------
-
-.. This section should contain new known issues in this release. Sample format:
-
- * **Add title in present tense with full stop.**
-
- Add a short 1-2 sentence description of the known issue
- in the present tense. Add information on any known workarounds.
-
- This section is a comment. Do not overwrite or remove it.
- Also, make sure to start the actual text at the margin.
- =========================================================
-
-
Tested Platforms
----------------
--
2.7.5
^ permalink raw reply [relevance 8%]
* Re: [dpdk-dev] [PATCH v3] cmdline: increase maximum line length
@ 2020-02-22 15:28 3% ` David Marchand
0 siblings, 0 replies; 200+ results
From: David Marchand @ 2020-02-22 15:28 UTC (permalink / raw)
To: Wisam Jaddo
Cc: dev, Raslan, Thomas Monjalon, Olivier Matz, Iremonger, Bernard,
dpdk stable
This patch is flagged as an ABI breakage:
https://travis-ci.com/ovsrobot/dpdk/jobs/289313318#L2273
On Thu, Feb 20, 2020 at 3:53 PM Wisam Jaddo <wisamm@mellanox.com> wrote:
>
> This increase due to the usage of cmdline in dpdk applications
> as config commands such as testpmd do for rte_flow rules creation.
>
> The current size of buffer is not enough to fill
> many cases of rte_flow commands validation/creation.
>
> rte_flow now can have outer items, inner items, modify
> actions, meta data actions, duplicate action, fate action and
> more in one single rte flow, thus 512 char will not be enough
> to validate such rte flow rules.
>
> Such change shouldn't affect the memory since the cmdline
> reading again using the same buffer.
I don't get your point here.
> Cc: stable@dpdk.org
This is not a fix.
--
David Marchand
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH v4] doc: alias to experimental tag for stable apis
2020-02-18 10:46 17% ` [dpdk-dev] [PATCH v4] " Ray Kinsella
@ 2020-02-21 17:11 0% ` Ferruh Yigit
2020-02-25 14:22 0% ` Kevin Traynor
1 sibling, 0 replies; 200+ results
From: Ferruh Yigit @ 2020-02-21 17:11 UTC (permalink / raw)
To: Ray Kinsella, dev
Cc: john.mcnamara, nhorman, marko.kovacevic, bruce.richardson
On 2/18/2020 10:46 AM, Ray Kinsella wrote:
> When a maintainer is promoting an API to become part of the next major ABI
> version by removing the experimental tag, possibly a few releases in
> advance of the declaration of the next ABI version. The maintainer may
> choose to offer an alias to the experimental tag, as removing the tag
> before the declaration of the next major ABI version, would cause an ABI
> breakage for applications using the API.
>
> Signed-off-by: Ray Kinsella <mdr@ashroe.eu>
> ---
> doc/guides/contributing/abi_policy.rst | 10 ++++++++++
> 1 file changed, 10 insertions(+)
>
> diff --git a/doc/guides/contributing/abi_policy.rst b/doc/guides/contributing/abi_policy.rst
> index 05ca959..7ae7de7 100644
> --- a/doc/guides/contributing/abi_policy.rst
> +++ b/doc/guides/contributing/abi_policy.rst
> @@ -159,6 +159,11 @@ The requirements for changing the ABI are:
> ``experimental``, as described in the section on :ref:`Experimental APIs
> and Libraries <experimental_apis>`.
>
> + - In situations in which an ``experimental`` API has been stable for some
> + time. When promoting the API to become part of the next ABI version, the
> + maintainer may choose to provide an alias to the ``experimental`` tag, so
> + as not to break consuming applications.
> +
> #. If a newly proposed API functionally replaces an existing one, when the new
> API becomes non-experimental, then the old one is marked with
> ``__rte_deprecated``.
> @@ -317,6 +322,11 @@ not required. Though, an API should remain in experimental state for at least
> one release. Thereafter, the normal process of posting patch for review to
> mailing list can be followed.
>
> +After the experimental tag has been formally removed, a tree/sub-tree maintainer
> +may choose to offer an alias to the experimental tag so as not to break
> +applications using the API. The alias is then dropped at the declaration of next
> +major ABI version.
> +
> Libraries
> ~~~~~~~~~
>
>
This leaves the decision to author/maintainer, I am more for making this a
requirement, but at least this enables it and better than nothing ;)
Acked-by: Ferruh Yigit <ferruh.yigit@intel.com>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] ABI version of experimental libraries
2020-02-19 13:50 4% ` Ray Kinsella
@ 2020-02-21 16:57 4% ` Thomas Monjalon
2020-02-24 9:32 4% ` Ray Kinsella
0 siblings, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-02-21 16:57 UTC (permalink / raw)
To: dev; +Cc: Ray Kinsella
19/02/2020 14:50, Ray Kinsella:
> On 19/02/2020 12:43, Thomas Monjalon wrote:
> > 19/02/2020 12:43, Neil Horman:
> >> On Tue, Feb 18, 2020 at 10:50:09AM +0100, Thomas Monjalon wrote:
> >>> 18/02/2020 10:42, Bruce Richardson:
> >>>> On Tue, Feb 18, 2020 at 12:15:56AM +0100, Thomas Monjalon wrote:
> >>>>> Hi,
> >>>>>
> >>>>> I would like to remind everybody our mistake when defining ABI versions.
> >>>>> It has been "fixed" in this commit:
> >>>>> http://git.dpdk.org/dpdk/commit/?id=f26c2b39
> >>>>>
> >>>>> Please let's think about the consequence for the experimental libraries.
> >>>>>
> >>>>> In DPDK 19.11, we use the ABI version 0.200 with soname 0.20 In DPDK
> >>>>> 20.02, we use the ABI version 0.2001 with soname 0.201 Numbers are
> >>>>> increasing, that's fine. When we'll switch to the new major ABI and use
> >>>>> a normal numbering: In DPDK 20.11, we will use the ABI version 0.210 with
> >>>>> soname 0.21 Numbers are dropping.
> >>>>>
> >>>>> In short, for experimental libs, ABI 20.1 > ABI 21.0
> >>>>>
> >>>>> Are we OK with this or do we prefer reverting to normal numbering for
> >>>>> experimental libraries in DPDK 20.02?
> >>>>>
> >>>> Personally, I would not be too concerned about the verions of experimental
> >>>> libs, so long as they don't conflict across versions and have some
> >>>> similarity to the major ABI version for the release.
> >>>
> >>> You think sorting of the version numbers is not important?
> >>> If we don't care comparing experimental version numbers,
> >>> then OK, let's drop this patch. But please we need a small vote.
> >>>
> >>> Note: there would be no problem if we did not vote for having
> >>> a special numbering for pure experimental libraries (I am still against).
> >>>
> >> I don't understand. Why would we change the ABI_VERSION at all in an LTS release at
> >> all? This operation is meant to take an an experimental API and mark it as
> >> stable by promoting its version number to the next major releases number. As
> >> such, in the LTS release, we should keep the soname the same, as there should be
> >> no other ABI changes in the promoted API.
> >
> > The library version number is updated because we add new symbols.
> >
> >
>
> So while experimental library version numbers are not "important".
> I do agree with Thomas they should be sane, increase and should have a consistent format.
>
> Should we always just pad them to 4 places as the simple solution?
> i.e.
>
> DPDK 19.11 ... 0.20 (needs to remain 0.20).
> DPDK 20.02 ... 0.2001
> DPDK 20.11 ... 0.2100
> DPDK 21.02 ... 0.2101
A patch from Ferruh got merged.
It is adding a dot to keep versioning consistent.
Marking this patch as rejected.
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [dpdk-ci] [PATCH 4/4] ci: reorganise Travis jobs
2020-02-20 16:01 0% ` David Marchand
@ 2020-02-20 19:38 0% ` Jeremy Plsek
0 siblings, 0 replies; 200+ results
From: Jeremy Plsek @ 2020-02-20 19:38 UTC (permalink / raw)
To: David Marchand
Cc: ci, Thomas Monjalon, dev, Michael Santana, Aaron Conole,
Bruce Richardson
[-- Attachment #1: Type: text/plain, Size: 3853 bytes --]
On Thu, Feb 20, 2020 at 11:02 AM David Marchand
<david.marchand@redhat.com> wrote:
>
> On Thu, Feb 20, 2020 at 3:35 PM Aaron Conole <aconole@redhat.com> wrote:
> >
> > David Marchand <david.marchand@redhat.com> writes:
> >
> > > On Thu, Feb 20, 2020 at 11:42 AM Thomas Monjalon <thomas@monjalon.net> wrote:
> > >>
> > >> 19/02/2020 22:39, Aaron Conole:
> > >> > David Marchand <david.marchand@redhat.com> writes:
> > >> >
> > >> > > Let's prune the jobs list to limit the amount of time spent by the robot
> > >> > > in Travis.
> > >> > >
> > >> > > Since meson enables automatically the relevant components, there is not
> > >> > > much gain in testing with extra_packages vs required_packages only.
> > >> > >
> > >> > > For a given arch/compiler/env combination, compilation is first tested
> > >> > > in all jobs that run tests or build the docs or run the ABI checks.
> > >> > > In the same context, for jobs that accumulates running tests, building
> > >> > > the docs etc..., those steps are independent and can be split to save
> > >> > > some cpu on Travis.
> > >> > >
> > >> > > With this, we go down from 21 to 15 jobs.
> > >> > >
> > >> > > Note: this patch requires a flush of the existing caches in Travis.
> > >> > >
> > >> > > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > >> > > ---
> > >> >
> > >> > In general, I think the idea with required vs. extra was to have a build
> > >> > that did the minimum required, and one that did all the packages (to
> > >> > allow a minimum vs. full DPDK).
> > >> >
> > >> > At least, that's from
> > >> > http://mails.dpdk.org/archives/dev/2019-January/124007.html
> > >>
> > >> I think the benefit of a minimum build is to have a quick report,
> > >> and easy to setup.
> > >
> > > Yes, Travis serves as a first gate when submitting patches.
> > > But since Travis is best effort/free, we can't have a full coverage.
> > >
> > >
> > >> > Not sure if that's still something anyone cares about.
> > >>
> > >> Given that Travis knows how to satisfy the dependencies,
> > >> and that we must wait for all jobs to finish,
> > >> I don't see any benefit of a minimal setup.
> > >
> > > This minimal setup also tests that dpdk dependencies are correct.
> > > If a change makes something rely on libX and libX is in the packages
> > > always installed in Travis, the missing dependency would not get
> > > caught.
> > >
> > > But here, this adds too many jobs.
> > >
> > > UNH, Intel and other CIs should step in and fill this kind of gap.
> >
> > Okay, makes sense to me. Are one of these CI providers offering to
> > cover this?
>
> Maybe it is already covered, the best is to ask, so sending to ci@dpdk.org.
>
> For the CI guys, which packages are installed on the systems/vms that
> do compilation tests?
For our compile tests, we compile with pretty much all dpdk
dependencies that can easily be installed from the distro's repository
(the dependencies we installed are based on the meson output). The
thought process was to test compiling as much of dpdk as possible
(with constraints mentioned before).
Here are the dependencies we install on some of the setups:
Ubuntu 18.04: make gcc ninja-build pkg-config libnuma-dev
librdmacm-dev librdmacm1 rdma-core libelf-dev libmnl-dev libpcap-dev
libcrypto++-dev libjansson-dev
(meson installed with pip)
CentOS 8 and Fedora 31: diffutils gcc make ninja-build meson
pkg-config numactl-devel librdmacm rdma-core-devel
elfutils-libelf-devel libmnl-devel libpcap-devel jansson-devel
openSuse Leap 15: gcc make ninja pkg-config libnuma-devel librdmacm1
rdma-core-devel libelf-devel libmnl-devel libpcap-devel
libjansson-devel
(meson installed with pip)
> Is it possible to have a summary of the different setups?
>
I've attached an overview/summary of what we do at the UNH-IOL DPDK
Community Lab.
>
> Thanks.
>
> --
> David Marchand
>
[-- Attachment #2: DPDK Lab Summary February 2020.pdf --]
[-- Type: application/pdf, Size: 846111 bytes --]
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] build: fix experimental library versioning
2020-02-20 19:54 10% ` [dpdk-dev] [PATCH] build: fix experimental library versioning Ferruh Yigit
` (2 preceding siblings ...)
2020-02-21 15:24 5% ` David Marchand
@ 2020-02-21 16:41 0% ` David Marchand
3 siblings, 0 replies; 200+ results
From: David Marchand @ 2020-02-21 16:41 UTC (permalink / raw)
To: Ferruh Yigit
Cc: Thomas Monjalon, Neil Horman, Bruce Richardson, dev, Kinsella,
Ray, Luca Boccassi, Kevin Traynor
On Thu, Feb 20, 2020 at 8:55 PM Ferruh Yigit <ferruh.yigit@intel.com> wrote:
>
> The problem occurred when workaround that makes soname two digits
> applied. With this change for the ABI version "20.0.1" the experimental
> library version become ".so.2001".
> After workaround removed in ABI version 21.0, the experimental library
> version will become ".so.210".
> "2001" is bigger value than "201" although it is a previous version of
> the library version, this can break the version comparisons.
>
> To fix this, introducing a temporary sub level versioning for the
> experimental libraries, so that minor version comparison will continue
> to work.
>
> After change experimental libraries will follow below versioning:
>
> DPDK version ABI version soname library name
> ------------ ----------- ------ ------------
> DPDK 19.11 20.0 .so.0.200 .so.0.200
> DPDK 20.02 20.0.1 .so.0.200.1 .so.0.200.1
> DPDK 20.05 20.0.2 .so.0.200.2 .so.0.200.2
> DPDK 20.11 21.0 .so.0.210 .so.0.210
> DPDK 21.02 21.1 .so.0.211 .so.0.211
>
> Note: After workaround removed in DPDK 20.11 and soname switch back to
> single digit this patch won't work and needs to be updated.
>
> Fixes: f26c2b39b271 ("build: fix soname info for 19.11 compatibility")
>
> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
Acked-by: Luca Boccassi <bluca@debian.org>
Acked-by: Ray Kinsella <mdr@ashroe.eu>
Applied.
--
David Marchand
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] devtools: skip experimental libraries in ABI check
2020-02-21 16:30 4% ` Thomas Monjalon
@ 2020-02-21 16:40 4% ` David Marchand
0 siblings, 0 replies; 200+ results
From: David Marchand @ 2020-02-21 16:40 UTC (permalink / raw)
To: Thomas Monjalon; +Cc: dev, Ferruh Yigit, Dodji Seketeli
On Fri, Feb 21, 2020 at 5:30 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> 21/02/2020 17:13, Ferruh Yigit:
> > On 2/21/2020 4:10 PM, David Marchand wrote:
> > > We don't provide ABI compatibility for experimental libraries.
> > > Skip those libraries by catching a soname containing a version starting
> > > with '0.'.
> >
> > I am not familiar with the script and how libabigail works, but +1 to the idea.
> >
> > >
> > > Align the special case for the glue libraries by using the soname too.
> > > Once libabigail has support for it, we will have a single type of rule.
> > >
> > > Fixes: 777014e56d07 ("devtools: add ABI checks")
> > >
> > > Signed-off-by: David Marchand <david.marchand@redhat.com>
>
> Acked-by: Thomas Monjalon <thomas@monjalon.net>
Applied.
--
David Marchand
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH 0/4] Extend --lcores to run on cores > RTE_MAX_LCORE
@ 2020-02-21 16:38 3% ` Stephen Hemminger
0 siblings, 0 replies; 200+ results
From: Stephen Hemminger @ 2020-02-21 16:38 UTC (permalink / raw)
To: Aaron Conole
Cc: David Marchand, Song, Keesang, ktraynor, bluca, Thomas Monjalon,
dev, ferruh.yigit, bruce.richardson, honnappa.nagarahalli, drc,
stable, Grimm, Jon, Hollingsworth, Brent
On Fri, 21 Feb 2020 09:48:58 -0500
Aaron Conole <aconole@redhat.com> wrote:
> David Marchand <david.marchand@redhat.com> writes:
>
> > On Fri, Feb 21, 2020 at 9:19 AM Song, Keesang <Keesang.Song@amd.com> wrote:
> >>
> >> [AMD Official Use Only - Internal Distribution Only]
> >
> > Please, get this header removed.
> > This is a public mailing list.
> >
> >
> >> Thanks Thomas for bringing this up.
> >> I consider this is not a new feature, but rather a fix to address
> >> the issue with statically assigned maximum lcore limit on
> >> high-density CPU platform such as AMD Epyc.
> >> As I see a lot of DPDK adopters are still using LTS 18.11 & 19.11,
> >> and they have 1~2 yrs of lifetime left, we like to backport this to
> >> LTS 18.11 & 19.11 at least.
> >
> > It is not a fix.
> >
> > The use of static arrays is a design choice that goes back to the
> > early days in dpdk.
> > The addition of --lcores came in after this, but it was introduced for
> > a different use case than placing lcores on any physical core.
> > And there was no claim that a core > RTE_MAX_LCORE would be usable.
> >
> >
> > When testing on a new hardware, it is normal to observe some limitations.
> > Running DPDK on those platforms should be possible: "should be"
> > because I do not have access to this hardware and saw neither tests
> > reports nor performance numbers.
> > Before this patch, the limitation is that on Epyc, cores >
> > RTE_MAX_LCORE are not usable.
> >
> >
> > Now, this change is quite constrained.
> > If we backport it, I don't expect issues in the main dpdk components
> > (based on code review and ovs tests with a RTE_MAX_LCORE set to 16 on
> > a 24 cores system).
> > There might be issues in some examples or not widely used library
> > which uses a physical core id instead of a lcore id.
> >
> >
> > This is the same recurring question "do we allow new features in a
> > stable branch?".
>
> Usually, the answer is 'no'. But we do allow some "new" things to be
> backported (pci ids, etc) that might be required to enable older
> functionality. Additionally, I'm sure if some feature were required to
> mitigate a CVE, we'd rather favor backporting it.
>
> I guess we could pose a litmus test:
>
> 1. Is the problem this feature solves so widespread that it needs to
> be addressed ASAP?
> 2. Is there a known workaround to the problem this is solving?
> 3. How intrusive is the feature?
> 4. Is it shown to be stable in the mainline (number of fixes, testing,
> etc)?
> 5. Is it constrained enough that we know we can support it with even
> higher priority than other things?
>
> Probably other questions that will need to be asked.
>
> And even in that list of question, I'm not sure I'd be able to advocate
> backporting this in the upstream branches - it hasn't had much testing.
> It's unstable. It's "difficult" to use. It is not widespread that
> people have so many cores. The workaround is much simpler than
> supporting this (recompile).
>
> >
> > --
> > David Marchand
>
RTE_MAX_LCORES is exposed in API/ABI to application.
Many applications use that to size internal data structures.
Having rte_lcore_id() potentially return a larger value would cause
out of bounds access (and crash) in that application.
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH] devtools: skip experimental libraries in ABI check
2020-02-21 16:13 4% ` Ferruh Yigit
@ 2020-02-21 16:30 4% ` Thomas Monjalon
2020-02-21 16:40 4% ` David Marchand
0 siblings, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-02-21 16:30 UTC (permalink / raw)
To: David Marchand; +Cc: dev, Ferruh Yigit, dodji
21/02/2020 17:13, Ferruh Yigit:
> On 2/21/2020 4:10 PM, David Marchand wrote:
> > We don't provide ABI compatibility for experimental libraries.
> > Skip those libraries by catching a soname containing a version starting
> > with '0.'.
>
> I am not familiar with the script and how libabigail works, but +1 to the idea.
>
> >
> > Align the special case for the glue libraries by using the soname too.
> > Once libabigail has support for it, we will have a single type of rule.
> >
> > Fixes: 777014e56d07 ("devtools: add ABI checks")
> >
> > Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Thomas Monjalon <thomas@monjalon.net>
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH] devtools: skip experimental libraries in ABI check
2020-02-21 16:10 17% [dpdk-dev] [PATCH] devtools: skip experimental libraries in ABI check David Marchand
@ 2020-02-21 16:13 4% ` Ferruh Yigit
2020-02-21 16:30 4% ` Thomas Monjalon
0 siblings, 1 reply; 200+ results
From: Ferruh Yigit @ 2020-02-21 16:13 UTC (permalink / raw)
To: David Marchand, dev; +Cc: thomas, dodji
On 2/21/2020 4:10 PM, David Marchand wrote:
> We don't provide ABI compatibility for experimental libraries.
> Skip those libraries by catching a soname containing a version starting
> with '0.'.
I am not familiar with the script and how libabigail works, but +1 to the idea.
>
> Align the special case for the glue libraries by using the soname too.
> Once libabigail has support for it, we will have a single type of rule.
>
> Fixes: 777014e56d07 ("devtools: add ABI checks")
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
> devtools/check-abi.sh | 9 +++++++--
> 1 file changed, 7 insertions(+), 2 deletions(-)
>
> diff --git a/devtools/check-abi.sh b/devtools/check-abi.sh
> index 0b4d1a37e..dd9120e69 100755
> --- a/devtools/check-abi.sh
> +++ b/devtools/check-abi.sh
> @@ -40,8 +40,13 @@ for dump in $(find $refdir -name "*.dump"); do
> # skip glue drivers, example librte_pmd_mlx5_glue.dump
> # We can't rely on a suppression rule for now:
> # https://sourceware.org/bugzilla/show_bug.cgi?id=25480
> - if [ "$name" != "${name%%_glue.dump}" ]; then
> - echo "Skipping ${dump}..."
> + if grep -qE "\<soname='[^']*_glue\.so\.[^']*'" $dump; then
> + echo "Skipped glue library $name."
> + continue
> + fi
> + # skip experimental libraries, with a sover starting with 0.
> + if grep -qE "\<soname='[^']*\.so\.0\.[^']*'" $dump; then
> + echo "Skipped experimental library $name."
> continue
> fi
> dump2=$(find $newdir -name $name)
>
^ permalink raw reply [relevance 4%]
* [dpdk-dev] [PATCH] devtools: skip experimental libraries in ABI check
@ 2020-02-21 16:10 17% David Marchand
2020-02-21 16:13 4% ` Ferruh Yigit
0 siblings, 1 reply; 200+ results
From: David Marchand @ 2020-02-21 16:10 UTC (permalink / raw)
To: dev; +Cc: thomas, ferruh.yigit, dodji
We don't provide ABI compatibility for experimental libraries.
Skip those libraries by catching a soname containing a version starting
with '0.'.
Align the special case for the glue libraries by using the soname too.
Once libabigail has support for it, we will have a single type of rule.
Fixes: 777014e56d07 ("devtools: add ABI checks")
Signed-off-by: David Marchand <david.marchand@redhat.com>
---
devtools/check-abi.sh | 9 +++++++--
1 file changed, 7 insertions(+), 2 deletions(-)
diff --git a/devtools/check-abi.sh b/devtools/check-abi.sh
index 0b4d1a37e..dd9120e69 100755
--- a/devtools/check-abi.sh
+++ b/devtools/check-abi.sh
@@ -40,8 +40,13 @@ for dump in $(find $refdir -name "*.dump"); do
# skip glue drivers, example librte_pmd_mlx5_glue.dump
# We can't rely on a suppression rule for now:
# https://sourceware.org/bugzilla/show_bug.cgi?id=25480
- if [ "$name" != "${name%%_glue.dump}" ]; then
- echo "Skipping ${dump}..."
+ if grep -qE "\<soname='[^']*_glue\.so\.[^']*'" $dump; then
+ echo "Skipped glue library $name."
+ continue
+ fi
+ # skip experimental libraries, with a sover starting with 0.
+ if grep -qE "\<soname='[^']*\.so\.0\.[^']*'" $dump; then
+ echo "Skipped experimental library $name."
continue
fi
dump2=$(find $newdir -name $name)
--
2.23.0
^ permalink raw reply [relevance 17%]
* Re: [dpdk-dev] [PATCH] build: fix experimental library versioning
2020-02-21 15:24 5% ` David Marchand
@ 2020-02-21 15:34 3% ` Ferruh Yigit
0 siblings, 0 replies; 200+ results
From: Ferruh Yigit @ 2020-02-21 15:34 UTC (permalink / raw)
To: David Marchand
Cc: Thomas Monjalon, Neil Horman, Bruce Richardson, dev, Kinsella,
Ray, Luca Boccassi, Kevin Traynor
On 2/21/2020 3:24 PM, David Marchand wrote:
> On Thu, Feb 20, 2020 at 8:55 PM Ferruh Yigit <ferruh.yigit@intel.com> wrote:
>>
>> The problem occurred when workaround that makes soname two digits
>> applied. With this change for the ABI version "20.0.1" the experimental
>> library version become ".so.2001".
>> After workaround removed in ABI version 21.0, the experimental library
>> version will become ".so.210".
>> "2001" is bigger value than "201" although it is a previous version of
>> the library version, this can break the version comparisons.
>>
>> To fix this, introducing a temporary sub level versioning for the
>> experimental libraries, so that minor version comparison will continue
>> to work.
>>
>> After change experimental libraries will follow below versioning:
>>
>> DPDK version ABI version soname library name
>> ------------ ----------- ------ ------------
>> DPDK 19.11 20.0 .so.0.200 .so.0.200
>> DPDK 20.02 20.0.1 .so.0.200.1 .so.0.200.1
>> DPDK 20.05 20.0.2 .so.0.200.2 .so.0.200.2
>> DPDK 20.11 21.0 .so.0.210 .so.0.210
>> DPDK 21.02 21.1 .so.0.211 .so.0.211
>>
>> Note: After workaround removed in DPDK 20.11 and soname switch back to
>> single digit this patch won't work and needs to be updated.
>>
>> Fixes: f26c2b39b271 ("build: fix soname info for 19.11 compatibility")
>>
>> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
>
> The soname change breaks existing applications like testpmd (linked to
> various experimental libraries).
> I have the full details if needed in a pastebin (v19.11, master, patch_applied).
>
> $ LD_LIBRARY_PATH=$HOME/builds/x86_64-native-linux-gcc+shared+kmods/install/lib
> $HOME/abi/v19.11/x86_64-native-linux-gcc+shared+kmods/bin/testpmd
> /home/dmarchan/abi/v19.11/x86_64-native-linux-gcc+shared+kmods/bin/testpmd:
> error while loading shared libraries: librte_flow_classify.so.0.200:
> cannot open shared object file: No such file or directory
>
>
> libabigail reports this change as an error, so if it is intended, we
> need to bypass the check for those libraries.
soname change was intended, we don't provide any ABI compatibility in the
experimental libraries, keeping same soname may mislead as if we do.
The impact of it is, any application linked against experimental library will
need to recompile in new release.
Can we add exception to the the libabigail for experimental libraries?
>
> Functions changes summary: 0 Removed, 0 Changed (2 filtered out), 0
> Added functions
> Variables changes summary: 0 Removed, 0 Changed, 0 Added variable
> ELF SONAME changed
> Functions changes summary: 0 Removed, 0 Changed (7 filtered out), 0
> Added functions
> Variables changes summary: 0 Removed, 0 Changed, 0 Added variable
> SONAME changed from 'librte_flow_classify.so.0.200' to
> 'librte_flow_classify.so.0.200.1'
> Error: ABI issue reported for 'abidiff --suppr
> /home/dmarchan/dpdk/devtools/libabigail.abignore --no-added-syms
> --headers-dir1 /home/dmarchan/abi/v19.11/x86_64-native-linux-gcc+shared+kmods/include
> --headers-dir2 /home/dmarchan/builds/x86_64-native-linux-gcc+shared+kmods/install/include
> /home/dmarchan/abi/v19.11/x86_64-native-linux-gcc+shared+kmods/dump/librte_flow_classify.dump
> /home/dmarchan/builds/x86_64-native-linux-gcc+shared+kmods/install/dump/librte_flow_classify.dump'
>
>
>
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH] build: fix experimental library versioning
2020-02-20 19:54 10% ` [dpdk-dev] [PATCH] build: fix experimental library versioning Ferruh Yigit
2020-02-20 22:14 0% ` Luca Boccassi
2020-02-21 12:36 0% ` Ray Kinsella
@ 2020-02-21 15:24 5% ` David Marchand
2020-02-21 15:34 3% ` Ferruh Yigit
2020-02-21 16:41 0% ` David Marchand
3 siblings, 1 reply; 200+ results
From: David Marchand @ 2020-02-21 15:24 UTC (permalink / raw)
To: Ferruh Yigit
Cc: Thomas Monjalon, Neil Horman, Bruce Richardson, dev, Kinsella,
Ray, Luca Boccassi, Kevin Traynor
On Thu, Feb 20, 2020 at 8:55 PM Ferruh Yigit <ferruh.yigit@intel.com> wrote:
>
> The problem occurred when workaround that makes soname two digits
> applied. With this change for the ABI version "20.0.1" the experimental
> library version become ".so.2001".
> After workaround removed in ABI version 21.0, the experimental library
> version will become ".so.210".
> "2001" is bigger value than "201" although it is a previous version of
> the library version, this can break the version comparisons.
>
> To fix this, introducing a temporary sub level versioning for the
> experimental libraries, so that minor version comparison will continue
> to work.
>
> After change experimental libraries will follow below versioning:
>
> DPDK version ABI version soname library name
> ------------ ----------- ------ ------------
> DPDK 19.11 20.0 .so.0.200 .so.0.200
> DPDK 20.02 20.0.1 .so.0.200.1 .so.0.200.1
> DPDK 20.05 20.0.2 .so.0.200.2 .so.0.200.2
> DPDK 20.11 21.0 .so.0.210 .so.0.210
> DPDK 21.02 21.1 .so.0.211 .so.0.211
>
> Note: After workaround removed in DPDK 20.11 and soname switch back to
> single digit this patch won't work and needs to be updated.
>
> Fixes: f26c2b39b271 ("build: fix soname info for 19.11 compatibility")
>
> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
The soname change breaks existing applications like testpmd (linked to
various experimental libraries).
I have the full details if needed in a pastebin (v19.11, master, patch_applied).
$ LD_LIBRARY_PATH=$HOME/builds/x86_64-native-linux-gcc+shared+kmods/install/lib
$HOME/abi/v19.11/x86_64-native-linux-gcc+shared+kmods/bin/testpmd
/home/dmarchan/abi/v19.11/x86_64-native-linux-gcc+shared+kmods/bin/testpmd:
error while loading shared libraries: librte_flow_classify.so.0.200:
cannot open shared object file: No such file or directory
libabigail reports this change as an error, so if it is intended, we
need to bypass the check for those libraries.
Functions changes summary: 0 Removed, 0 Changed (2 filtered out), 0
Added functions
Variables changes summary: 0 Removed, 0 Changed, 0 Added variable
ELF SONAME changed
Functions changes summary: 0 Removed, 0 Changed (7 filtered out), 0
Added functions
Variables changes summary: 0 Removed, 0 Changed, 0 Added variable
SONAME changed from 'librte_flow_classify.so.0.200' to
'librte_flow_classify.so.0.200.1'
Error: ABI issue reported for 'abidiff --suppr
/home/dmarchan/dpdk/devtools/libabigail.abignore --no-added-syms
--headers-dir1 /home/dmarchan/abi/v19.11/x86_64-native-linux-gcc+shared+kmods/include
--headers-dir2 /home/dmarchan/builds/x86_64-native-linux-gcc+shared+kmods/install/include
/home/dmarchan/abi/v19.11/x86_64-native-linux-gcc+shared+kmods/dump/librte_flow_classify.dump
/home/dmarchan/builds/x86_64-native-linux-gcc+shared+kmods/install/dump/librte_flow_classify.dump'
--
David Marchand
^ permalink raw reply [relevance 5%]
* Re: [dpdk-dev] [PATCH] build: fix experimental library versioning
2020-02-20 19:54 10% ` [dpdk-dev] [PATCH] build: fix experimental library versioning Ferruh Yigit
2020-02-20 22:14 0% ` Luca Boccassi
@ 2020-02-21 12:36 0% ` Ray Kinsella
2020-02-21 15:24 5% ` David Marchand
2020-02-21 16:41 0% ` David Marchand
3 siblings, 0 replies; 200+ results
From: Ray Kinsella @ 2020-02-21 12:36 UTC (permalink / raw)
To: Ferruh Yigit, Thomas Monjalon, Neil Horman, Bruce Richardson
Cc: dev, ray.kinsella, bluca, david.marchand, ktraynor
On 20/02/2020 19:54, Ferruh Yigit wrote:
> The problem occurred when workaround that makes soname two digits
> applied. With this change for the ABI version "20.0.1" the experimental
> library version become ".so.2001".
> After workaround removed in ABI version 21.0, the experimental library
> version will become ".so.210".
> "2001" is bigger value than "201" although it is a previous version of
> the library version, this can break the version comparisons.
>
> To fix this, introducing a temporary sub level versioning for the
> experimental libraries, so that minor version comparison will continue
> to work.
>
> After change experimental libraries will follow below versioning:
>
> DPDK version ABI version soname library name
> ------------ ----------- ------ ------------
> DPDK 19.11 20.0 .so.0.200 .so.0.200
> DPDK 20.02 20.0.1 .so.0.200.1 .so.0.200.1
> DPDK 20.05 20.0.2 .so.0.200.2 .so.0.200.2
> DPDK 20.11 21.0 .so.0.210 .so.0.210
> DPDK 21.02 21.1 .so.0.211 .so.0.211
>
> Note: After workaround removed in DPDK 20.11 and soname switch back to
> single digit this patch won't work and needs to be updated.
>
> Fixes: f26c2b39b271 ("build: fix soname info for 19.11 compatibility")
>
Acked-by: Ray Kinsella <mdr@ashroe.eu>
> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
> ---
> Cc: Thomas Monjalon <thomas@monjalon.net>
> Cc: ray.kinsella@intel.com
> Cc: nhorman@tuxdriver.com
> Cc: bruce.richardson@intel.com
> Cc: bluca@debian.org
> Cc: david.marchand@redhat.com
> Cc: ktraynor@redhat.com
> ---
> config/meson.build | 4 ++--
> mk/rte.lib.mk | 8 +++++---
> 2 files changed, 7 insertions(+), 5 deletions(-)
>
> diff --git a/config/meson.build b/config/meson.build
> index 6c46767e3..d3d2370ce 100644
> --- a/config/meson.build
> +++ b/config/meson.build
> @@ -31,8 +31,8 @@ abi_version = run_command(find_program('cat', 'more'),
> # sonames => librte_stable.so.20.0, librte_experimental.so.0.200
> abi_va = abi_version.split('.')
> stable_so_version = abi_va.length() == 2 ? abi_va[0] : abi_va[0] + '.' + abi_va[1]
> -experimental_abi_version = '0.' + ''.join(abi_va)
> -experimental_so_version = '0.' + ''.join(stable_so_version.split('.'))
> +experimental_abi_version = '0.' + abi_va[0] + abi_va[1] + '.' + abi_va[2]
> +experimental_so_version = experimental_abi_version
>
> # extract all version information into the build configuration
> dpdk_conf.set('RTE_VER_YEAR', pver.get(0).to_int())
> diff --git a/mk/rte.lib.mk b/mk/rte.lib.mk
> index b1a8372cc..682b590db 100644
> --- a/mk/rte.lib.mk
> +++ b/mk/rte.lib.mk
> @@ -15,8 +15,8 @@ LIBABIVER ?= $(shell cat $(RTE_SRCDIR)/ABI_VERSION)
> SOVER := $(basename $(LIBABIVER))
> ifeq ($(shell grep -s "^DPDK_" $(SRCDIR)/$(EXPORT_MAP)),)
> # EXPERIMENTAL ABI is versioned as 0.major+minor, e.g. 0.201 for 20.1 ABI
> -LIBABIVER := 0.$(shell echo $(LIBABIVER) | tr -d '.')
> -SOVER := 0.$(shell echo $(SOVER) | tr -d '.')
> +LIBABIVER := 0.$(shell echo $(LIBABIVER) | awk 'BEGIN { FS="." }; { print $$1$$2"."$$3 }')
> +SOVER := $(LIBABIVER)
> endif
>
> ifeq ($(CONFIG_RTE_BUILD_SHARED_LIB),y)
> @@ -135,7 +135,9 @@ $(RTE_OUTPUT)/lib/$(LIB): $(LIB)
> $(Q)cp -f $(LIB) $(RTE_OUTPUT)/lib
> ifeq ($(CONFIG_RTE_BUILD_SHARED_LIB),y)
> $(Q)ln -s -f $< $(shell echo $@ | sed 's/\.so.*/.so/')
> - $(Q)ln -s -f $< $(shell echo $@ | sed 's/\.so.*/.so.$(SOVER)/')
> + $(Q)if [ $(SOVER) != $(LIBABIVER) ]; then \
> + ln -s -f $< $(shell echo $@ | sed 's/\.so.*/.so.$(SOVER)/') ; \
> + fi
> endif
>
> #
>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] doc: plan splitting the ethdev ops struct
2020-02-18 6:01 0% ` Stephen Hemminger
@ 2020-02-21 10:40 0% ` Ferruh Yigit
2020-02-25 10:35 0% ` Andrew Rybchenko
0 siblings, 1 reply; 200+ results
From: Ferruh Yigit @ 2020-02-21 10:40 UTC (permalink / raw)
To: Stephen Hemminger
Cc: Neil Horman, John McNamara, Marko Kovacevic, dev, David Marchand,
Thomas Monjalon, Andrew Rybchenko
On 2/18/2020 6:01 AM, Stephen Hemminger wrote:
> On Mon, 17 Feb 2020 15:38:05 +0000
> Ferruh Yigit <ferruh.yigit@intel.com> wrote:
>
>> For the ABI compatibility it is better to hide internal data structures
>> from the application as much as possible. But because of some inline
>> functions 'struct eth_dev_ops' can't be hidden completely.
>>
>> Plan is to split the 'struct eth_dev_ops' into two as ones used by
>> inline functions and ones not used, and hide the second part that not
>> used by inline functions completely to the application.
>>
>> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
>> ---
>> Cc: David Marchand <david.marchand@redhat.com>
>> Cc: Thomas Monjalon <thomas@monjalon.net>
>> Cc: Andrew Rybchenko <arybchenko@solarflare.com>
>> ---
>> doc/guides/rel_notes/deprecation.rst | 6 ++++++
>> 1 file changed, 6 insertions(+)
>>
>> diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
>> index dfcca87ab..2aa431028 100644
>> --- a/doc/guides/rel_notes/deprecation.rst
>> +++ b/doc/guides/rel_notes/deprecation.rst
>> @@ -72,6 +72,12 @@ Deprecation Notices
>> In 19.11 PMDs will still update the field even when the offload is not
>> enabled.
>>
>> +* ethdev: Split the ``struct eth_dev_ops`` struct to hide it as much as possible.
>> + Currently the ``struct eth_dev_ops`` struct is accessible by the application
>> + because some inline functions, like ``rte_eth_tx_descriptor_status()``,
>> + access the struct directly. The struct will be separate in two, the ops used
>> + by inline functions still will be accessible to user but rest will be hidden.
>> +
>> * cryptodev: support for using IV with all sizes is added, J0 still can
>> be used but only when IV length in following structs ``rte_crypto_auth_xform``,
>> ``rte_crypto_aead_xform`` is set to zero. When IV length is greater or equal
>
> Good luck, truely hiding internals is hard. The mbuf structure is already split but not really
> hidden at all (just look at dwarf output). It doesn't make sense to do it unless
> you can really hide it.
I believe this can be done, only following [1] dev_ops are used by inline
functions, rest can be moved into separate struct and moved into ethdev driver
looking header.
[1]
rx_queue_count
rx_descriptor_done
rx_descriptor_status
tx_descriptor_status
>
> I would attack the rte_device stuff first. Make rte_device opaque to the application
> that would help for future versions. Then work backwards to rte_tehtdev.
>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] doc: deprecate using MAX values as array size
@ 2020-02-21 10:25 0% ` Ferruh Yigit
2020-02-24 6:18 0% ` Akhil Goyal
2020-02-24 8:51 0% ` Andrew Rybchenko
2020-02-25 13:52 0% ` David Marchand
2020-02-25 14:53 0% ` David Marchand
2 siblings, 2 replies; 200+ results
From: Ferruh Yigit @ 2020-02-21 10:25 UTC (permalink / raw)
To: Neil Horman, John McNamara, Marko Kovacevic
Cc: dev, Thomas Monjalon, Bruce Richardson, Akhil Goyal,
Konstantin Ananyev, David Marchand, Jerin Jacob,
Andrew Rybchenko, Ori Kam
On 1/30/2020 2:20 PM, Ferruh Yigit wrote:
> Adding the deprecation notice as reminder for next ABI breakage release
> (20.11).
> This one time breakage is required to be able to extend enum/define
> without breaking ABI.
>
> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
> ---
> doc/guides/rel_notes/deprecation.rst | 14 ++++++++++++++
> 1 file changed, 14 insertions(+)
>
> diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
> index dfcca87ab..99d81564a 100644
> --- a/doc/guides/rel_notes/deprecation.rst
> +++ b/doc/guides/rel_notes/deprecation.rst
> @@ -38,6 +38,20 @@ Deprecation Notices
> remove it from the externally visible ABI and allow it to be updated in the
> future.
>
> +* lib: will fix extending some enum/define breaking the ABI. There are multiple
> + samples in DPDK that enum/define terminated with a ``.*MAX.*`` value which is
> + used by iterators, and arrays holding these values are sized with this
> + ``.*MAX.*`` value. So extending this enum/define increases the ``.*MAX.*``
> + value which increases the size of the array and depending on how/where the
> + array is used this may break the ABI.
> + ``RTE_ETH_FLOW_MAX`` is one sample of the mentioned case, adding a new flow
> + type will break the ABI because of ``flex_mask[RTE_ETH_FLOW_MAX]`` array
> + usage in following public struct hierarchy:
> + ``rte_eth_fdir_flex_conf -> rte_fdir_conf -> rte_eth_conf (in the middle)``.
> + Need to identify this kind of usages and fix in 20.11, otherwise this blocks
> + us extending existing enum/define.
> + One solution can be using a fixed size array instead of ``.*MAX.*`` value.
> +
> * dpaa2: removal of ``rte_dpaa2_memsegs`` structure which has been replaced
> by a pa-va search library. This structure was earlier being used for holding
> memory segments used by dpaa2 driver for faster pa->va translation. This
>
Reminder of this deprecation notice.
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] build: fix experimental library versioning
2020-02-20 19:54 10% ` [dpdk-dev] [PATCH] build: fix experimental library versioning Ferruh Yigit
@ 2020-02-20 22:14 0% ` Luca Boccassi
2020-02-21 12:36 0% ` Ray Kinsella
` (2 subsequent siblings)
3 siblings, 0 replies; 200+ results
From: Luca Boccassi @ 2020-02-20 22:14 UTC (permalink / raw)
To: Ferruh Yigit, Thomas Monjalon, Neil Horman, Bruce Richardson
Cc: dev, ray.kinsella, david.marchand, ktraynor
On Thu, 2020-02-20 at 19:54 +0000, Ferruh Yigit wrote:
> The problem occurred when workaround that makes soname two digits
> applied. With this change for the ABI version "20.0.1" the
> experimental
> library version become ".so.2001".
> After workaround removed in ABI version 21.0, the experimental
> library
> version will become ".so.210".
> "2001" is bigger value than "201" although it is a previous version
> of
> the library version, this can break the version comparisons.
>
> To fix this, introducing a temporary sub level versioning for the
> experimental libraries, so that minor version comparison will
> continue
> to work.
>
> After change experimental libraries will follow below versioning:
>
> DPDK version ABI version soname library name
> ------------ ----------- ------ ------------
> DPDK 19.11 20.0 .so.0.200 .so.0.200
> DPDK 20.02 20.0.1 .so.0.200.1 .so.0.200.1
> DPDK 20.05 20.0.2 .so.0.200.2 .so.0.200.2
> DPDK 20.11 21.0 .so.0.210 .so.0.210
> DPDK 21.02 21.1 .so.0.211 .so.0.211
>
> Note: After workaround removed in DPDK 20.11 and soname switch back
> to
> single digit this patch won't work and needs to be updated.
>
> Fixes: f26c2b39b271 ("build: fix soname info for 19.11
> compatibility")
>
> Signed-off-by: Ferruh Yigit <
> ferruh.yigit@intel.com
Acked-by: Luca Boccassi <bluca@debian.org>
This solution looks like a good compromise to me. Thanks Ferruh.
--
Kind regards,
Luca Boccassi
^ permalink raw reply [relevance 0%]
* [dpdk-dev] [PATCH] build: fix experimental library versioning
2020-02-20 19:50 8% ` Ferruh Yigit
@ 2020-02-20 19:54 10% ` Ferruh Yigit
2020-02-20 22:14 0% ` Luca Boccassi
` (3 more replies)
0 siblings, 4 replies; 200+ results
From: Ferruh Yigit @ 2020-02-20 19:54 UTC (permalink / raw)
To: Thomas Monjalon, Neil Horman, Bruce Richardson
Cc: dev, Ferruh Yigit, ray.kinsella, bluca, david.marchand, ktraynor
The problem occurred when workaround that makes soname two digits
applied. With this change for the ABI version "20.0.1" the experimental
library version become ".so.2001".
After workaround removed in ABI version 21.0, the experimental library
version will become ".so.210".
"2001" is bigger value than "201" although it is a previous version of
the library version, this can break the version comparisons.
To fix this, introducing a temporary sub level versioning for the
experimental libraries, so that minor version comparison will continue
to work.
After change experimental libraries will follow below versioning:
DPDK version ABI version soname library name
------------ ----------- ------ ------------
DPDK 19.11 20.0 .so.0.200 .so.0.200
DPDK 20.02 20.0.1 .so.0.200.1 .so.0.200.1
DPDK 20.05 20.0.2 .so.0.200.2 .so.0.200.2
DPDK 20.11 21.0 .so.0.210 .so.0.210
DPDK 21.02 21.1 .so.0.211 .so.0.211
Note: After workaround removed in DPDK 20.11 and soname switch back to
single digit this patch won't work and needs to be updated.
Fixes: f26c2b39b271 ("build: fix soname info for 19.11 compatibility")
Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
---
Cc: Thomas Monjalon <thomas@monjalon.net>
Cc: ray.kinsella@intel.com
Cc: nhorman@tuxdriver.com
Cc: bruce.richardson@intel.com
Cc: bluca@debian.org
Cc: david.marchand@redhat.com
Cc: ktraynor@redhat.com
---
config/meson.build | 4 ++--
mk/rte.lib.mk | 8 +++++---
2 files changed, 7 insertions(+), 5 deletions(-)
diff --git a/config/meson.build b/config/meson.build
index 6c46767e3..d3d2370ce 100644
--- a/config/meson.build
+++ b/config/meson.build
@@ -31,8 +31,8 @@ abi_version = run_command(find_program('cat', 'more'),
# sonames => librte_stable.so.20.0, librte_experimental.so.0.200
abi_va = abi_version.split('.')
stable_so_version = abi_va.length() == 2 ? abi_va[0] : abi_va[0] + '.' + abi_va[1]
-experimental_abi_version = '0.' + ''.join(abi_va)
-experimental_so_version = '0.' + ''.join(stable_so_version.split('.'))
+experimental_abi_version = '0.' + abi_va[0] + abi_va[1] + '.' + abi_va[2]
+experimental_so_version = experimental_abi_version
# extract all version information into the build configuration
dpdk_conf.set('RTE_VER_YEAR', pver.get(0).to_int())
diff --git a/mk/rte.lib.mk b/mk/rte.lib.mk
index b1a8372cc..682b590db 100644
--- a/mk/rte.lib.mk
+++ b/mk/rte.lib.mk
@@ -15,8 +15,8 @@ LIBABIVER ?= $(shell cat $(RTE_SRCDIR)/ABI_VERSION)
SOVER := $(basename $(LIBABIVER))
ifeq ($(shell grep -s "^DPDK_" $(SRCDIR)/$(EXPORT_MAP)),)
# EXPERIMENTAL ABI is versioned as 0.major+minor, e.g. 0.201 for 20.1 ABI
-LIBABIVER := 0.$(shell echo $(LIBABIVER) | tr -d '.')
-SOVER := 0.$(shell echo $(SOVER) | tr -d '.')
+LIBABIVER := 0.$(shell echo $(LIBABIVER) | awk 'BEGIN { FS="." }; { print $$1$$2"."$$3 }')
+SOVER := $(LIBABIVER)
endif
ifeq ($(CONFIG_RTE_BUILD_SHARED_LIB),y)
@@ -135,7 +135,9 @@ $(RTE_OUTPUT)/lib/$(LIB): $(LIB)
$(Q)cp -f $(LIB) $(RTE_OUTPUT)/lib
ifeq ($(CONFIG_RTE_BUILD_SHARED_LIB),y)
$(Q)ln -s -f $< $(shell echo $@ | sed 's/\.so.*/.so/')
- $(Q)ln -s -f $< $(shell echo $@ | sed 's/\.so.*/.so.$(SOVER)/')
+ $(Q)if [ $(SOVER) != $(LIBABIVER) ]; then \
+ ln -s -f $< $(shell echo $@ | sed 's/\.so.*/.so.$(SOVER)/') ; \
+ fi
endif
#
--
2.24.1
^ permalink raw reply [relevance 10%]
* Re: [dpdk-dev] ABI version of experimental libraries
2020-02-18 10:36 4% ` Kinsella, Ray
@ 2020-02-20 19:50 8% ` Ferruh Yigit
2020-02-20 19:54 10% ` [dpdk-dev] [PATCH] build: fix experimental library versioning Ferruh Yigit
0 siblings, 1 reply; 200+ results
From: Ferruh Yigit @ 2020-02-20 19:50 UTC (permalink / raw)
To: Kinsella, Ray, Thomas Monjalon, Richardson, Bruce
Cc: nhorman, bluca, david.marchand, ktraynor, dev
On 2/18/2020 10:36 AM, Kinsella, Ray wrote:
>
>
>> -----Original Message-----
>> From: Thomas Monjalon <thomas@monjalon.net>
>> Sent: Tuesday 18 February 2020 09:50
>> To: Richardson, Bruce <bruce.richardson@intel.com>
>> Cc: Kinsella, Ray <ray.kinsella@intel.com>; nhorman@tuxdriver.com;
>> bluca@debian.org; david.marchand@redhat.com; ktraynor@redhat.com;
>> dev@dpdk.org
>> Subject: Re: ABI version of experimental libraries
>>
>> 18/02/2020 10:42, Bruce Richardson:
>>> On Tue, Feb 18, 2020 at 12:15:56AM +0100, Thomas Monjalon wrote:
>>>> Hi,
>>>>
>>>> I would like to remind everybody our mistake when defining ABI
>> versions.
>>>> It has been "fixed" in this commit:
>>>> http://git.dpdk.org/dpdk/commit/?id=f26c2b39
>>>>
>>>> Please let's think about the consequence for the experimental
>> libraries.
>>>>
>>>> In DPDK 19.11, we use the ABI version 0.200 with soname 0.20 In
>> DPDK
>>>> 20.02, we use the ABI version 0.2001 with soname 0.201 Numbers are
>>>> increasing, that's fine. When we'll switch to the new major ABI
>> and
>>>> use a normal numbering: In DPDK 20.11, we will use the ABI version
>>>> 0.210 with soname 0.21 Numbers are dropping.
>>>>
>>>> In short, for experimental libs, ABI 20.1 > ABI 21.0
>>>>
>>>> Are we OK with this or do we prefer reverting to normal numbering
>>>> for experimental libraries in DPDK 20.02?
>>>>
>>> Personally, I would not be too concerned about the verions of
>>> experimental libs, so long as they don't conflict across versions and
>>> have some similarity to the major ABI version for the release.
>>
>> You think sorting of the version numbers is not important?
>> If we don't care comparing experimental version numbers, then OK, let's
>> drop this patch. But please we need a small vote.
>>
>> Note: there would be no problem if we did not vote for having a special
>> numbering for pure experimental libraries (I am still against).
>>
>
> So while experimental library version numbers are not "important".
> I do agree with Thomas they should be sane, increase and should have a consistent format.
>
> Should we always pad them to 4 places?
> i.e.
>
> DPDK 19.11 ... 0.20 (needs to remain 0.20).
> DPDK 20.02 ... 0.2001
> DPDK 20.11 ... 0.2100
> DPDK 21.02 ... 0.2101
>
> Make sense?
>
What about following:
DPDK version ABI version soname library name
------------ ----------- ------ ------------
DPDK 19.11 20.0 .so.0.200 .so.0.200
DPDK 20.02 20.0.1 .so.0.200.1 .so.0.200.1
DPDK 20.05 20.0.2 .so.0.200.2 .so.0.200.2
DPDK 20.11 21.0 .so.0.210 .so.0.210
DPDK 21.02 21.1 .so.0.211 .so.0.211
This let us turn back to sane versions when soname fixed in 20.11.
Also I think for experimental libraries soname should NOT be same in minor
versions, otherwise it can create false impression that it is ABI compatible
between two minor versions.
I will reply the patch to this email.
^ permalink raw reply [relevance 8%]
* Re: [dpdk-dev] [PATCH 4/4] ci: reorganise Travis jobs
2020-02-20 14:35 0% ` Aaron Conole
@ 2020-02-20 16:01 0% ` David Marchand
2020-02-20 19:38 0% ` [dpdk-dev] [dpdk-ci] " Jeremy Plsek
0 siblings, 1 reply; 200+ results
From: David Marchand @ 2020-02-20 16:01 UTC (permalink / raw)
To: ci; +Cc: Thomas Monjalon, dev, Michael Santana, Aaron Conole, Bruce Richardson
On Thu, Feb 20, 2020 at 3:35 PM Aaron Conole <aconole@redhat.com> wrote:
>
> David Marchand <david.marchand@redhat.com> writes:
>
> > On Thu, Feb 20, 2020 at 11:42 AM Thomas Monjalon <thomas@monjalon.net> wrote:
> >>
> >> 19/02/2020 22:39, Aaron Conole:
> >> > David Marchand <david.marchand@redhat.com> writes:
> >> >
> >> > > Let's prune the jobs list to limit the amount of time spent by the robot
> >> > > in Travis.
> >> > >
> >> > > Since meson enables automatically the relevant components, there is not
> >> > > much gain in testing with extra_packages vs required_packages only.
> >> > >
> >> > > For a given arch/compiler/env combination, compilation is first tested
> >> > > in all jobs that run tests or build the docs or run the ABI checks.
> >> > > In the same context, for jobs that accumulates running tests, building
> >> > > the docs etc..., those steps are independent and can be split to save
> >> > > some cpu on Travis.
> >> > >
> >> > > With this, we go down from 21 to 15 jobs.
> >> > >
> >> > > Note: this patch requires a flush of the existing caches in Travis.
> >> > >
> >> > > Signed-off-by: David Marchand <david.marchand@redhat.com>
> >> > > ---
> >> >
> >> > In general, I think the idea with required vs. extra was to have a build
> >> > that did the minimum required, and one that did all the packages (to
> >> > allow a minimum vs. full DPDK).
> >> >
> >> > At least, that's from
> >> > http://mails.dpdk.org/archives/dev/2019-January/124007.html
> >>
> >> I think the benefit of a minimum build is to have a quick report,
> >> and easy to setup.
> >
> > Yes, Travis serves as a first gate when submitting patches.
> > But since Travis is best effort/free, we can't have a full coverage.
> >
> >
> >> > Not sure if that's still something anyone cares about.
> >>
> >> Given that Travis knows how to satisfy the dependencies,
> >> and that we must wait for all jobs to finish,
> >> I don't see any benefit of a minimal setup.
> >
> > This minimal setup also tests that dpdk dependencies are correct.
> > If a change makes something rely on libX and libX is in the packages
> > always installed in Travis, the missing dependency would not get
> > caught.
> >
> > But here, this adds too many jobs.
> >
> > UNH, Intel and other CIs should step in and fill this kind of gap.
>
> Okay, makes sense to me. Are one of these CI providers offering to
> cover this?
Maybe it is already covered, the best is to ask, so sending to ci@dpdk.org.
For the CI guys, which packages are installed on the systems/vms that
do compilation tests?
Is it possible to have a summary of the different setups?
Thanks.
--
David Marchand
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH 4/4] ci: reorganise Travis jobs
2020-02-20 12:22 0% ` David Marchand
@ 2020-02-20 14:35 0% ` Aaron Conole
2020-02-20 16:01 0% ` David Marchand
0 siblings, 1 reply; 200+ results
From: Aaron Conole @ 2020-02-20 14:35 UTC (permalink / raw)
To: David Marchand; +Cc: Thomas Monjalon, dev, Michael Santana, Bruce Richardson
David Marchand <david.marchand@redhat.com> writes:
> On Thu, Feb 20, 2020 at 11:42 AM Thomas Monjalon <thomas@monjalon.net> wrote:
>>
>> 19/02/2020 22:39, Aaron Conole:
>> > David Marchand <david.marchand@redhat.com> writes:
>> >
>> > > Let's prune the jobs list to limit the amount of time spent by the robot
>> > > in Travis.
>> > >
>> > > Since meson enables automatically the relevant components, there is not
>> > > much gain in testing with extra_packages vs required_packages only.
>> > >
>> > > For a given arch/compiler/env combination, compilation is first tested
>> > > in all jobs that run tests or build the docs or run the ABI checks.
>> > > In the same context, for jobs that accumulates running tests, building
>> > > the docs etc..., those steps are independent and can be split to save
>> > > some cpu on Travis.
>> > >
>> > > With this, we go down from 21 to 15 jobs.
>> > >
>> > > Note: this patch requires a flush of the existing caches in Travis.
>> > >
>> > > Signed-off-by: David Marchand <david.marchand@redhat.com>
>> > > ---
>> >
>> > In general, I think the idea with required vs. extra was to have a build
>> > that did the minimum required, and one that did all the packages (to
>> > allow a minimum vs. full DPDK).
>> >
>> > At least, that's from
>> > http://mails.dpdk.org/archives/dev/2019-January/124007.html
>>
>> I think the benefit of a minimum build is to have a quick report,
>> and easy to setup.
>
> Yes, Travis serves as a first gate when submitting patches.
> But since Travis is best effort/free, we can't have a full coverage.
>
>
>> > Not sure if that's still something anyone cares about.
>>
>> Given that Travis knows how to satisfy the dependencies,
>> and that we must wait for all jobs to finish,
>> I don't see any benefit of a minimal setup.
>
> This minimal setup also tests that dpdk dependencies are correct.
> If a change makes something rely on libX and libX is in the packages
> always installed in Travis, the missing dependency would not get
> caught.
>
> But here, this adds too many jobs.
>
> UNH, Intel and other CIs should step in and fill this kind of gap.
Okay, makes sense to me. Are one of these CI providers offering to
cover this?
Also,
Acked-by: Aaron Conole <aconole@redhat.com>
>
> --
> David Marchand
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH 4/4] ci: reorganise Travis jobs
2020-02-20 10:42 0% ` Thomas Monjalon
@ 2020-02-20 12:22 0% ` David Marchand
2020-02-20 14:35 0% ` Aaron Conole
0 siblings, 1 reply; 200+ results
From: David Marchand @ 2020-02-20 12:22 UTC (permalink / raw)
To: Thomas Monjalon, Aaron Conole; +Cc: dev, Michael Santana, Bruce Richardson
On Thu, Feb 20, 2020 at 11:42 AM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> 19/02/2020 22:39, Aaron Conole:
> > David Marchand <david.marchand@redhat.com> writes:
> >
> > > Let's prune the jobs list to limit the amount of time spent by the robot
> > > in Travis.
> > >
> > > Since meson enables automatically the relevant components, there is not
> > > much gain in testing with extra_packages vs required_packages only.
> > >
> > > For a given arch/compiler/env combination, compilation is first tested
> > > in all jobs that run tests or build the docs or run the ABI checks.
> > > In the same context, for jobs that accumulates running tests, building
> > > the docs etc..., those steps are independent and can be split to save
> > > some cpu on Travis.
> > >
> > > With this, we go down from 21 to 15 jobs.
> > >
> > > Note: this patch requires a flush of the existing caches in Travis.
> > >
> > > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > > ---
> >
> > In general, I think the idea with required vs. extra was to have a build
> > that did the minimum required, and one that did all the packages (to
> > allow a minimum vs. full DPDK).
> >
> > At least, that's from
> > http://mails.dpdk.org/archives/dev/2019-January/124007.html
>
> I think the benefit of a minimum build is to have a quick report,
> and easy to setup.
Yes, Travis serves as a first gate when submitting patches.
But since Travis is best effort/free, we can't have a full coverage.
> > Not sure if that's still something anyone cares about.
>
> Given that Travis knows how to satisfy the dependencies,
> and that we must wait for all jobs to finish,
> I don't see any benefit of a minimal setup.
This minimal setup also tests that dpdk dependencies are correct.
If a change makes something rely on libX and libX is in the packages
always installed in Travis, the missing dependency would not get
caught.
But here, this adds too many jobs.
UNH, Intel and other CIs should step in and fill this kind of gap.
--
David Marchand
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH 4/4] ci: reorganise Travis jobs
2020-02-19 19:41 3% ` [dpdk-dev] [PATCH 4/4] ci: reorganise " David Marchand
2020-02-19 21:39 0% ` Aaron Conole
2020-02-20 0:18 0% ` [dpdk-dev] " dwilder
@ 2020-02-20 11:07 0% ` Thomas Monjalon
2 siblings, 0 replies; 200+ results
From: Thomas Monjalon @ 2020-02-20 11:07 UTC (permalink / raw)
To: David Marchand; +Cc: aconole, dev, Michael Santana
19/02/2020 20:41, David Marchand:
> Let's prune the jobs list to limit the amount of time spent by the robot
> in Travis.
>
> Since meson enables automatically the relevant components, there is not
> much gain in testing with extra_packages vs required_packages only.
>
> For a given arch/compiler/env combination, compilation is first tested
> in all jobs that run tests or build the docs or run the ABI checks.
> In the same context, for jobs that accumulates running tests, building
> the docs etc..., those steps are independent and can be split to save
> some cpu on Travis.
>
> With this, we go down from 21 to 15 jobs.
>
> Note: this patch requires a flush of the existing caches in Travis.
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Thomas Monjalon <thomas@monjalon.net>
A small step for DPDK but a big save for the earth ;)
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH 4/4] ci: reorganise Travis jobs
2020-02-19 21:39 0% ` Aaron Conole
@ 2020-02-20 10:42 0% ` Thomas Monjalon
2020-02-20 12:22 0% ` David Marchand
0 siblings, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-02-20 10:42 UTC (permalink / raw)
To: David Marchand, Aaron Conole; +Cc: dev, Michael Santana, Bruce Richardson
19/02/2020 22:39, Aaron Conole:
> David Marchand <david.marchand@redhat.com> writes:
>
> > Let's prune the jobs list to limit the amount of time spent by the robot
> > in Travis.
> >
> > Since meson enables automatically the relevant components, there is not
> > much gain in testing with extra_packages vs required_packages only.
> >
> > For a given arch/compiler/env combination, compilation is first tested
> > in all jobs that run tests or build the docs or run the ABI checks.
> > In the same context, for jobs that accumulates running tests, building
> > the docs etc..., those steps are independent and can be split to save
> > some cpu on Travis.
> >
> > With this, we go down from 21 to 15 jobs.
> >
> > Note: this patch requires a flush of the existing caches in Travis.
> >
> > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > ---
>
> In general, I think the idea with required vs. extra was to have a build
> that did the minimum required, and one that did all the packages (to
> allow a minimum vs. full DPDK).
>
> At least, that's from
> http://mails.dpdk.org/archives/dev/2019-January/124007.html
I think the benefit of a minimum build is to have a quick report,
and easy to setup.
> Not sure if that's still something anyone cares about.
Given that Travis knows how to satisfy the dependencies,
and that we must wait for all jobs to finish,
I don't see any benefit of a minimal setup.
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [dpdk-announce] release candidate 20.02-rc3
@ 2020-02-20 9:41 4% ` Peng, Yuan
0 siblings, 0 replies; 200+ results
From: Peng, Yuan @ 2020-02-20 9:41 UTC (permalink / raw)
To: Thomas Monjalon, dev
DPDK 20.02 RC3 Test Report from intel:
Summary
• Totally 9898 test cases for 20.02 RC3, 9629 cases are executed, 9428 test cases passed, 178 test cases failed, 23 test case blocked, Execution rate is 97.28%, pass rate is 97.91%.
• Totally create ~278 new tests for DPDK20.02 new features.
• New to create 9 issues from STV in RC3 till now, not include some issues have found and reported before RC3.
• Met 1 build issue on Centos 7.4 with DPDK 20.02 RC3, now it has been fixed, and patch applied.
Details:
• Intel NIC PMD PF & VF regression: 99% executed, 0 critical issue and 1 high issue is tracked(verified).
Function cases include rx/tx/forward packets, packet processing, jumboframe, dual vlan, filter cases and so on: 5 bugs found, no critical and high level.
Performance cases such as l2fwd, l3fwd, single core and so on: no new bug is found.
• Crypto cases include ipsec_gw cryptodev, fips cryptodev, l2fwd crypto, performance cases and so on: no critical and no high bugs found.
• Vhost/virtio: PVP/loopback, qemu/virtio-user, multi-queue, dequeue zero copy cases and so on: 1 new medium bug is found.
• Power relate cases, no new bug is found.
• ABI test includes unit test, vhost and flow related tests. no new bug is found.
• 20.02 new features include ABI test, L2tpv3, crypto_dev related new features and so no. no new bug is found.
• Scecurity tests include Nic malformed to PF/VF, virtio, Compreqssion/cryptodev, power and invalid command line, all passed.
Thanks.
Yuan.
-----Original Message-----
From: dev <dev-bounces@dpdk.org> On Behalf Of Thomas Monjalon
Sent: Monday, February 17, 2020 5:59 AM
To: announce@dpdk.org
Subject: [dpdk-dev] [dpdk-announce] release candidate 20.02-rc3
A new DPDK release candidate is ready for testing:
https://git.dpdk.org/dpdk/tag/?id=v20.02-rc3
121 patches were integrated.
The release notes so far:
http://doc.dpdk.org/guides/rel_notes/release_20_02.html
Some deprecation notices might be added if reviewed on time.
The -rc4 will include only bug fixes, doc and tooling.
This next milestone is expected during next week.
Please hurry up to do the last checks and bug fixes.
You may share some release validation results by replying to this message (at dev@dpdk.org).
If you are preparing the next release cycle, please send your v1 patches before the 20.05 proposal deadline, which will happen on March 18th.
It is also time to build an estimated roadmap for the next cycles.
Thank you everyone
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH 4/4] ci: reorganise Travis jobs
2020-02-19 19:41 3% ` [dpdk-dev] [PATCH 4/4] ci: reorganise " David Marchand
2020-02-19 21:39 0% ` Aaron Conole
@ 2020-02-20 0:18 0% ` dwilder
2020-02-20 11:07 0% ` Thomas Monjalon
2 siblings, 0 replies; 200+ results
From: dwilder @ 2020-02-20 0:18 UTC (permalink / raw)
To: David Marchand; +Cc: thomas, aconole, dev, Michael Santana
On 2020-02-19 11:41, David Marchand wrote:
> Let's prune the jobs list to limit the amount of time spent by the
> robot
> in Travis.
>
> Since meson enables automatically the relevant components, there is not
> much gain in testing with extra_packages vs required_packages only.
>
> For a given arch/compiler/env combination, compilation is first tested
> in all jobs that run tests or build the docs or run the ABI checks.
> In the same context, for jobs that accumulates running tests, building
> the docs etc..., those steps are independent and can be split to save
> some cpu on Travis.
>
> With this, we go down from 21 to 15 jobs.
>
> Note: this patch requires a flush of the existing caches in Travis.
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
> .travis.yml | 51 +++++++++------------------------------------------
> 1 file changed, 9 insertions(+), 42 deletions(-)
>
> diff --git a/.travis.yml b/.travis.yml
> index c174d5f30..b64a81bd0 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -15,15 +15,12 @@ addons:
> update: true
> packages: &required_packages
> - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip,
> ninja-build]
> + - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev,
> libfdt-dev, libjansson4]
>
> _aarch64_packages: &aarch64_packages
> - *required_packages
> - [gcc-aarch64-linux-gnu, libc6-dev-arm64-cross,
> pkg-config-aarch64-linux-gnu]
>
> -_extra_packages: &extra_packages
> - - *required_packages
> - - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev,
> libfdt-dev, libjansson4]
> -
> _libabigail_build_packages: &libabigail_build_packages
> - [autoconf, automake, libtool, pkg-config, libxml2-dev, libdw-dev]
>
> @@ -43,66 +40,39 @@ jobs:
> - env: DEF_LIB="static"
> arch: amd64
> compiler: gcc
> - - env: DEF_LIB="shared"
> - arch: amd64
> - compiler: gcc
> - env: DEF_LIB="shared" RUN_TESTS=1
> arch: amd64
> compiler: gcc
> - - env: DEF_LIB="static" EXTRA_PACKAGES=1
> - arch: amd64
> - compiler: gcc
> - addons:
> - apt:
> - packages:
> - - *extra_packages
> - - env: DEF_LIB="shared" EXTRA_PACKAGES=1 RUN_TESTS=1 BUILD_DOCS=1
> + - env: DEF_LIB="shared" BUILD_DOCS=1
> arch: amd64
> compiler: gcc
> addons:
> apt:
> packages:
> - - *extra_packages
> + - *required_packages
> - *doc_packages
> - - env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1
> + - env: DEF_LIB="shared" ABI_CHECKS=1
> arch: amd64
> compiler: gcc
> addons:
> apt:
> packages:
> - - *extra_packages
> + - *required_packages
> - *libabigail_build_packages
> # x86_64 clang jobs
> - env: DEF_LIB="static"
> arch: amd64
> compiler: clang
> - - env: DEF_LIB="shared"
> - arch: amd64
> - compiler: clang
> - env: DEF_LIB="shared" RUN_TESTS=1
> arch: amd64
> compiler: clang
> - - env: DEF_LIB="static" EXTRA_PACKAGES=1
> - arch: amd64
> - compiler: clang
> - addons:
> - apt:
> - packages:
> - - *extra_packages
> - - env: DEF_LIB="shared" EXTRA_PACKAGES=1
> - arch: amd64
> - compiler: clang
> - addons:
> - apt:
> - packages:
> - - *extra_packages
> - - env: DEF_LIB="shared" EXTRA_PACKAGES=1 RUN_TESTS=1 BUILD_DOCS=1
> + - env: DEF_LIB="shared" BUILD_DOCS=1
> arch: amd64
> compiler: clang
> addons:
> apt:
> packages:
> - - *extra_packages
> + - *required_packages
> - *doc_packages
> # x86_64 cross-compiling 32-bits jobs
> - env: DEF_LIB="static" BUILD_32BIT=1
> @@ -131,9 +101,6 @@ jobs:
> - env: DEF_LIB="static"
> arch: arm64
> compiler: gcc
> - - env: DEF_LIB="shared"
> - arch: arm64
> - compiler: gcc
> - env: DEF_LIB="shared" BUILD_DOCS=1
> arch: arm64
> compiler: gcc
> @@ -142,13 +109,13 @@ jobs:
> packages:
> - *required_packages
> - *doc_packages
> - - env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1
> + - env: DEF_LIB="shared" ABI_CHECKS=1
> arch: arm64
> compiler: gcc
> addons:
> apt:
> packages:
> - - *extra_packages
> + - *required_packages
> - *libabigail_build_packages
> # aarch64 clang jobs
> - env: DEF_LIB="static"
Much easier to follow, I had no issues adding ppc64le jobs.
All my travis jobs ran fine.
LGTM (the series)
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH 4/4] ci: reorganise Travis jobs
2020-02-19 19:41 3% ` [dpdk-dev] [PATCH 4/4] ci: reorganise " David Marchand
@ 2020-02-19 21:39 0% ` Aaron Conole
2020-02-20 10:42 0% ` Thomas Monjalon
2020-02-20 0:18 0% ` [dpdk-dev] " dwilder
2020-02-20 11:07 0% ` Thomas Monjalon
2 siblings, 1 reply; 200+ results
From: Aaron Conole @ 2020-02-19 21:39 UTC (permalink / raw)
To: David Marchand; +Cc: thomas, dev, Michael Santana, Bruce Richardson
David Marchand <david.marchand@redhat.com> writes:
> Let's prune the jobs list to limit the amount of time spent by the robot
> in Travis.
>
> Since meson enables automatically the relevant components, there is not
> much gain in testing with extra_packages vs required_packages only.
>
> For a given arch/compiler/env combination, compilation is first tested
> in all jobs that run tests or build the docs or run the ABI checks.
> In the same context, for jobs that accumulates running tests, building
> the docs etc..., those steps are independent and can be split to save
> some cpu on Travis.
>
> With this, we go down from 21 to 15 jobs.
>
> Note: this patch requires a flush of the existing caches in Travis.
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
In general, I think the idea with required vs. extra was to have a build
that did the minimum required, and one that did all the packages (to
allow a minimum vs. full DPDK).
At least, that's from
http://mails.dpdk.org/archives/dev/2019-January/124007.html
Not sure if that's still something anyone cares about.
> .travis.yml | 51 +++++++++------------------------------------------
> 1 file changed, 9 insertions(+), 42 deletions(-)
>
> diff --git a/.travis.yml b/.travis.yml
> index c174d5f30..b64a81bd0 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -15,15 +15,12 @@ addons:
> update: true
> packages: &required_packages
> - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, ninja-build]
> + - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson4]
>
> _aarch64_packages: &aarch64_packages
> - *required_packages
> - [gcc-aarch64-linux-gnu, libc6-dev-arm64-cross, pkg-config-aarch64-linux-gnu]
>
> -_extra_packages: &extra_packages
> - - *required_packages
> - - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson4]
> -
> _libabigail_build_packages: &libabigail_build_packages
> - [autoconf, automake, libtool, pkg-config, libxml2-dev, libdw-dev]
>
> @@ -43,66 +40,39 @@ jobs:
> - env: DEF_LIB="static"
> arch: amd64
> compiler: gcc
> - - env: DEF_LIB="shared"
> - arch: amd64
> - compiler: gcc
> - env: DEF_LIB="shared" RUN_TESTS=1
> arch: amd64
> compiler: gcc
> - - env: DEF_LIB="static" EXTRA_PACKAGES=1
> - arch: amd64
> - compiler: gcc
> - addons:
> - apt:
> - packages:
> - - *extra_packages
> - - env: DEF_LIB="shared" EXTRA_PACKAGES=1 RUN_TESTS=1 BUILD_DOCS=1
> + - env: DEF_LIB="shared" BUILD_DOCS=1
> arch: amd64
> compiler: gcc
> addons:
> apt:
> packages:
> - - *extra_packages
> + - *required_packages
> - *doc_packages
> - - env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1
> + - env: DEF_LIB="shared" ABI_CHECKS=1
> arch: amd64
> compiler: gcc
> addons:
> apt:
> packages:
> - - *extra_packages
> + - *required_packages
> - *libabigail_build_packages
> # x86_64 clang jobs
> - env: DEF_LIB="static"
> arch: amd64
> compiler: clang
> - - env: DEF_LIB="shared"
> - arch: amd64
> - compiler: clang
> - env: DEF_LIB="shared" RUN_TESTS=1
> arch: amd64
> compiler: clang
> - - env: DEF_LIB="static" EXTRA_PACKAGES=1
> - arch: amd64
> - compiler: clang
> - addons:
> - apt:
> - packages:
> - - *extra_packages
> - - env: DEF_LIB="shared" EXTRA_PACKAGES=1
> - arch: amd64
> - compiler: clang
> - addons:
> - apt:
> - packages:
> - - *extra_packages
> - - env: DEF_LIB="shared" EXTRA_PACKAGES=1 RUN_TESTS=1 BUILD_DOCS=1
> + - env: DEF_LIB="shared" BUILD_DOCS=1
> arch: amd64
> compiler: clang
> addons:
> apt:
> packages:
> - - *extra_packages
> + - *required_packages
> - *doc_packages
> # x86_64 cross-compiling 32-bits jobs
> - env: DEF_LIB="static" BUILD_32BIT=1
> @@ -131,9 +101,6 @@ jobs:
> - env: DEF_LIB="static"
> arch: arm64
> compiler: gcc
> - - env: DEF_LIB="shared"
> - arch: arm64
> - compiler: gcc
> - env: DEF_LIB="shared" BUILD_DOCS=1
> arch: arm64
> compiler: gcc
> @@ -142,13 +109,13 @@ jobs:
> packages:
> - *required_packages
> - *doc_packages
> - - env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1
> + - env: DEF_LIB="shared" ABI_CHECKS=1
> arch: arm64
> compiler: gcc
> addons:
> apt:
> packages:
> - - *extra_packages
> + - *required_packages
> - *libabigail_build_packages
> # aarch64 clang jobs
> - env: DEF_LIB="static"
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] ABI version of experimental libraries
2020-02-18 9:50 4% ` Thomas Monjalon
@ 2020-02-18 10:36 4% ` Kinsella, Ray
2020-02-20 19:50 8% ` Ferruh Yigit
2020-02-19 11:43 7% ` [dpdk-dev] ABI version of experimental libraries Neil Horman
1 sibling, 1 reply; 200+ results
From: Kinsella, Ray @ 2020-02-18 10:36 UTC (permalink / raw)
To: Thomas Monjalon, Richardson, Bruce
Cc: nhorman, bluca, david.marchand, ktraynor, dev
> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Tuesday 18 February 2020 09:50
> To: Richardson, Bruce <bruce.richardson@intel.com>
> Cc: Kinsella, Ray <ray.kinsella@intel.com>; nhorman@tuxdriver.com;
> bluca@debian.org; david.marchand@redhat.com; ktraynor@redhat.com;
> dev@dpdk.org
> Subject: Re: ABI version of experimental libraries
>
> 18/02/2020 10:42, Bruce Richardson:
> > On Tue, Feb 18, 2020 at 12:15:56AM +0100, Thomas Monjalon wrote:
> > > Hi,
> > >
> > > I would like to remind everybody our mistake when defining ABI
> versions.
> > > It has been "fixed" in this commit:
> > > http://git.dpdk.org/dpdk/commit/?id=f26c2b39
> > >
> > > Please let's think about the consequence for the experimental
> libraries.
> > >
> > > In DPDK 19.11, we use the ABI version 0.200 with soname 0.20 In
> DPDK
> > > 20.02, we use the ABI version 0.2001 with soname 0.201 Numbers are
> > > increasing, that's fine. When we'll switch to the new major ABI
> and
> > > use a normal numbering: In DPDK 20.11, we will use the ABI version
> > > 0.210 with soname 0.21 Numbers are dropping.
> > >
> > > In short, for experimental libs, ABI 20.1 > ABI 21.0
> > >
> > > Are we OK with this or do we prefer reverting to normal numbering
> > > for experimental libraries in DPDK 20.02?
> > >
> > Personally, I would not be too concerned about the verions of
> > experimental libs, so long as they don't conflict across versions and
> > have some similarity to the major ABI version for the release.
>
> You think sorting of the version numbers is not important?
> If we don't care comparing experimental version numbers, then OK, let's
> drop this patch. But please we need a small vote.
>
> Note: there would be no problem if we did not vote for having a special
> numbering for pure experimental libraries (I am still against).
>
So while experimental library version numbers are not "important".
I do agree with Thomas they should be sane, increase and should have a consistent format.
Should we always pad them to 4 places?
i.e.
DPDK 19.11 ... 0.20 (needs to remain 0.20).
DPDK 20.02 ... 0.2001
DPDK 20.11 ... 0.2100
DPDK 21.02 ... 0.2101
Make sense?
Ray K
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] ABI version of experimental libraries
2020-02-19 12:43 4% ` Thomas Monjalon
2020-02-19 13:50 4% ` Ray Kinsella
@ 2020-02-19 21:17 4% ` Neil Horman
1 sibling, 0 replies; 200+ results
From: Neil Horman @ 2020-02-19 21:17 UTC (permalink / raw)
To: Thomas Monjalon
Cc: Bruce Richardson, ray.kinsella, bluca, david.marchand, ktraynor, dev
On Wed, Feb 19, 2020 at 01:43:01PM +0100, Thomas Monjalon wrote:
> 19/02/2020 12:43, Neil Horman:
> > On Tue, Feb 18, 2020 at 10:50:09AM +0100, Thomas Monjalon wrote:
> > > 18/02/2020 10:42, Bruce Richardson:
> > > > On Tue, Feb 18, 2020 at 12:15:56AM +0100, Thomas Monjalon wrote:
> > > > > Hi,
> > > > >
> > > > > I would like to remind everybody our mistake when defining ABI versions.
> > > > > It has been "fixed" in this commit:
> > > > > http://git.dpdk.org/dpdk/commit/?id=f26c2b39
> > > > >
> > > > > Please let's think about the consequence for the experimental libraries.
> > > > >
> > > > > In DPDK 19.11, we use the ABI version 0.200 with soname 0.20 In DPDK
> > > > > 20.02, we use the ABI version 0.2001 with soname 0.201 Numbers are
> > > > > increasing, that's fine. When we'll switch to the new major ABI and use
> > > > > a normal numbering: In DPDK 20.11, we will use the ABI version 0.210 with
> > > > > soname 0.21 Numbers are dropping.
> > > > >
> > > > > In short, for experimental libs, ABI 20.1 > ABI 21.0
> > > > >
> > > > > Are we OK with this or do we prefer reverting to normal numbering for
> > > > > experimental libraries in DPDK 20.02?
> > > > >
> > > > Personally, I would not be too concerned about the verions of experimental
> > > > libs, so long as they don't conflict across versions and have some
> > > > similarity to the major ABI version for the release.
> > >
> > > You think sorting of the version numbers is not important?
> > > If we don't care comparing experimental version numbers,
> > > then OK, let's drop this patch. But please we need a small vote.
> > >
> > > Note: there would be no problem if we did not vote for having
> > > a special numbering for pure experimental libraries (I am still against).
> > >
> > I don't understand. Why would we change the ABI_VERSION at all in an LTS release at
> > all? This operation is meant to take an an experimental API and mark it as
> > stable by promoting its version number to the next major releases number. As
> > such, in the LTS release, we should keep the soname the same, as there should be
> > no other ABI changes in the promoted API.
>
> The library version number is updated because we add new symbols.
But thats a matter of policy (and possibly build mechanics). Not saying agree
with this approach necessecarily (as I've not fully thought it through), but we
could make room in the policy for this eventuality.
Neil
>
>
>
^ permalink raw reply [relevance 4%]
* [dpdk-dev] [PATCH 4/4] ci: reorganise Travis jobs
@ 2020-02-19 19:41 3% ` David Marchand
2020-02-19 21:39 0% ` Aaron Conole
` (2 more replies)
0 siblings, 3 replies; 200+ results
From: David Marchand @ 2020-02-19 19:41 UTC (permalink / raw)
To: thomas, aconole; +Cc: dev, Michael Santana
Let's prune the jobs list to limit the amount of time spent by the robot
in Travis.
Since meson enables automatically the relevant components, there is not
much gain in testing with extra_packages vs required_packages only.
For a given arch/compiler/env combination, compilation is first tested
in all jobs that run tests or build the docs or run the ABI checks.
In the same context, for jobs that accumulates running tests, building
the docs etc..., those steps are independent and can be split to save
some cpu on Travis.
With this, we go down from 21 to 15 jobs.
Note: this patch requires a flush of the existing caches in Travis.
Signed-off-by: David Marchand <david.marchand@redhat.com>
---
.travis.yml | 51 +++++++++------------------------------------------
1 file changed, 9 insertions(+), 42 deletions(-)
diff --git a/.travis.yml b/.travis.yml
index c174d5f30..b64a81bd0 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -15,15 +15,12 @@ addons:
update: true
packages: &required_packages
- [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, ninja-build]
+ - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson4]
_aarch64_packages: &aarch64_packages
- *required_packages
- [gcc-aarch64-linux-gnu, libc6-dev-arm64-cross, pkg-config-aarch64-linux-gnu]
-_extra_packages: &extra_packages
- - *required_packages
- - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson4]
-
_libabigail_build_packages: &libabigail_build_packages
- [autoconf, automake, libtool, pkg-config, libxml2-dev, libdw-dev]
@@ -43,66 +40,39 @@ jobs:
- env: DEF_LIB="static"
arch: amd64
compiler: gcc
- - env: DEF_LIB="shared"
- arch: amd64
- compiler: gcc
- env: DEF_LIB="shared" RUN_TESTS=1
arch: amd64
compiler: gcc
- - env: DEF_LIB="static" EXTRA_PACKAGES=1
- arch: amd64
- compiler: gcc
- addons:
- apt:
- packages:
- - *extra_packages
- - env: DEF_LIB="shared" EXTRA_PACKAGES=1 RUN_TESTS=1 BUILD_DOCS=1
+ - env: DEF_LIB="shared" BUILD_DOCS=1
arch: amd64
compiler: gcc
addons:
apt:
packages:
- - *extra_packages
+ - *required_packages
- *doc_packages
- - env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1
+ - env: DEF_LIB="shared" ABI_CHECKS=1
arch: amd64
compiler: gcc
addons:
apt:
packages:
- - *extra_packages
+ - *required_packages
- *libabigail_build_packages
# x86_64 clang jobs
- env: DEF_LIB="static"
arch: amd64
compiler: clang
- - env: DEF_LIB="shared"
- arch: amd64
- compiler: clang
- env: DEF_LIB="shared" RUN_TESTS=1
arch: amd64
compiler: clang
- - env: DEF_LIB="static" EXTRA_PACKAGES=1
- arch: amd64
- compiler: clang
- addons:
- apt:
- packages:
- - *extra_packages
- - env: DEF_LIB="shared" EXTRA_PACKAGES=1
- arch: amd64
- compiler: clang
- addons:
- apt:
- packages:
- - *extra_packages
- - env: DEF_LIB="shared" EXTRA_PACKAGES=1 RUN_TESTS=1 BUILD_DOCS=1
+ - env: DEF_LIB="shared" BUILD_DOCS=1
arch: amd64
compiler: clang
addons:
apt:
packages:
- - *extra_packages
+ - *required_packages
- *doc_packages
# x86_64 cross-compiling 32-bits jobs
- env: DEF_LIB="static" BUILD_32BIT=1
@@ -131,9 +101,6 @@ jobs:
- env: DEF_LIB="static"
arch: arm64
compiler: gcc
- - env: DEF_LIB="shared"
- arch: arm64
- compiler: gcc
- env: DEF_LIB="shared" BUILD_DOCS=1
arch: arm64
compiler: gcc
@@ -142,13 +109,13 @@ jobs:
packages:
- *required_packages
- *doc_packages
- - env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1
+ - env: DEF_LIB="shared" ABI_CHECKS=1
arch: arm64
compiler: gcc
addons:
apt:
packages:
- - *extra_packages
+ - *required_packages
- *libabigail_build_packages
# aarch64 clang jobs
- env: DEF_LIB="static"
--
2.23.0
^ permalink raw reply [relevance 3%]
* [dpdk-dev] [PATCH v2 0/7] vfio/pci: SR-IOV support
@ 2020-02-19 18:53 2% Alex Williamson
2020-02-25 2:33 0% ` Tian, Kevin
2020-03-05 6:38 0% ` Vamsi Krishna Attunuru
0 siblings, 2 replies; 200+ results
From: Alex Williamson @ 2020-02-19 18:53 UTC (permalink / raw)
To: kvm
Cc: linux-pci, linux-kernel, dev, mtosatti, thomas, bluca,
jerinjacobk, bruce.richardson, cohuck
Changes since v1 are primarily to patch 3/7 where the commit log is
rewritten, along with option parsing and failure logging based on
upstream discussions. The primary user visible difference is that
option parsing is now much more strict. If a vf_token option is
provided that cannot be used, we generate an error. As a result of
this, opening a PF with a vf_token option will serve as a mechanism of
setting the vf_token. This seems like a more user friendly API than
the alternative of sometimes requiring the option (VFs in use) and
sometimes rejecting it, and upholds our desire that the option is
always either used or rejected.
This also means that the VFIO_DEVICE_FEATURE ioctl is not the only
means of setting the VF token, which might call into question whether
we absolutely need this new ioctl. Currently I'm keeping it because I
can imagine use cases, for example if a hypervisor were to support
SR-IOV, the PF device might be opened without consideration for a VF
token and we'd require the hypservisor to close and re-open the PF in
order to set a known VF token, which is impractical.
Series overview (same as provided with v1):
The synopsis of this series is that we have an ongoing desire to drive
PCIe SR-IOV PFs from userspace with VFIO. There's an immediate need
for this with DPDK drivers and potentially interesting future use
cases in virtualization. We've been reluctant to add this support
previously due to the dependency and trust relationship between the
VF device and PF driver. Minimally the PF driver can induce a denial
of service to the VF, but depending on the specific implementation,
the PF driver might also be responsible for moving data between VFs
or have direct access to the state of the VF, including data or state
otherwise private to the VF or VF driver.
To help resolve these concerns, we introduce a VF token into the VFIO
PCI ABI, which acts as a shared secret key between drivers. The
userspace PF driver is required to set the VF token to a known value
and userspace VF drivers are required to provide the token to access
the VF device. If a PF driver is restarted with VF drivers in use, it
must also provide the current token in order to prevent a rogue
untrusted PF driver from replacing a known driver. The degree to
which this new token is considered secret is left to the userspace
drivers, the kernel intentionally provides no means to retrieve the
current token.
Note that the above token is only required for this new model where
both the PF and VF devices are usable through vfio-pci. Existing
models of VFIO drivers where the PF is used without SR-IOV enabled
or the VF is bound to a userspace driver with an in-kernel, host PF
driver are unaffected.
The latter configuration above also highlights a new inverted scenario
that is now possible, a userspace PF driver with in-kernel VF drivers.
I believe this is a scenario that should be allowed, but should not be
enabled by default. This series includes code to set a default
driver_override for VFs sourced from a vfio-pci user owned PF, such
that the VFs are also bound to vfio-pci. This model is compatible
with tools like driverctl and allows the system administrator to
decide if other bindings should be enabled. The VF token interface
above exists only between vfio-pci PF and VF drivers, once a VF is
bound to another driver, the administrator has effectively pronounced
the device as trusted. The vfio-pci driver will note alternate
binding in dmesg for logging and debugging purposes.
Please review, comment, and test. The example QEMU implementation
provided with the RFC is still current for this version. Thanks,
Alex
RFC: https://lore.kernel.org/lkml/158085337582.9445.17682266437583505502.stgit@gimli.home/
v1: https://lore.kernel.org/lkml/158145472604.16827.15751375540102298130.stgit@gimli.home/
---
Alex Williamson (7):
vfio: Include optional device match in vfio_device_ops callbacks
vfio/pci: Implement match ops
vfio/pci: Introduce VF token
vfio: Introduce VFIO_DEVICE_FEATURE ioctl and first user
vfio/pci: Add sriov_configure support
vfio/pci: Remove dev_fmt definition
vfio/pci: Cleanup .probe() exit paths
drivers/vfio/pci/vfio_pci.c | 383 +++++++++++++++++++++++++++++++++--
drivers/vfio/pci/vfio_pci_private.h | 10 +
drivers/vfio/vfio.c | 20 +-
include/linux/vfio.h | 4
include/uapi/linux/vfio.h | 37 +++
5 files changed, 426 insertions(+), 28 deletions(-)
^ permalink raw reply [relevance 2%]
* Re: [dpdk-dev] ABI version of experimental libraries
2020-02-19 12:43 4% ` Thomas Monjalon
@ 2020-02-19 13:50 4% ` Ray Kinsella
2020-02-21 16:57 4% ` Thomas Monjalon
2020-02-19 21:17 4% ` Neil Horman
1 sibling, 1 reply; 200+ results
From: Ray Kinsella @ 2020-02-19 13:50 UTC (permalink / raw)
To: dev
On 19/02/2020 12:43, Thomas Monjalon wrote:
> 19/02/2020 12:43, Neil Horman:
>> On Tue, Feb 18, 2020 at 10:50:09AM +0100, Thomas Monjalon wrote:
>>> 18/02/2020 10:42, Bruce Richardson:
>>>> On Tue, Feb 18, 2020 at 12:15:56AM +0100, Thomas Monjalon wrote:
>>>>> Hi,
>>>>>
>>>>> I would like to remind everybody our mistake when defining ABI versions.
>>>>> It has been "fixed" in this commit:
>>>>> http://git.dpdk.org/dpdk/commit/?id=f26c2b39
>>>>>
>>>>> Please let's think about the consequence for the experimental libraries.
>>>>>
>>>>> In DPDK 19.11, we use the ABI version 0.200 with soname 0.20 In DPDK
>>>>> 20.02, we use the ABI version 0.2001 with soname 0.201 Numbers are
>>>>> increasing, that's fine. When we'll switch to the new major ABI and use
>>>>> a normal numbering: In DPDK 20.11, we will use the ABI version 0.210 with
>>>>> soname 0.21 Numbers are dropping.
>>>>>
>>>>> In short, for experimental libs, ABI 20.1 > ABI 21.0
>>>>>
>>>>> Are we OK with this or do we prefer reverting to normal numbering for
>>>>> experimental libraries in DPDK 20.02?
>>>>>
>>>> Personally, I would not be too concerned about the verions of experimental
>>>> libs, so long as they don't conflict across versions and have some
>>>> similarity to the major ABI version for the release.
>>>
>>> You think sorting of the version numbers is not important?
>>> If we don't care comparing experimental version numbers,
>>> then OK, let's drop this patch. But please we need a small vote.
>>>
>>> Note: there would be no problem if we did not vote for having
>>> a special numbering for pure experimental libraries (I am still against).
>>>
>> I don't understand. Why would we change the ABI_VERSION at all in an LTS release at
>> all? This operation is meant to take an an experimental API and mark it as
>> stable by promoting its version number to the next major releases number. As
>> such, in the LTS release, we should keep the soname the same, as there should be
>> no other ABI changes in the promoted API.
>
> The library version number is updated because we add new symbols.
>
>
So while experimental library version numbers are not "important".
I do agree with Thomas they should be sane, increase and should have a consistent format.
Should we always just pad them to 4 places as the simple solution?
i.e.
DPDK 19.11 ... 0.20 (needs to remain 0.20).
DPDK 20.02 ... 0.2001
DPDK 20.11 ... 0.2100
DPDK 21.02 ... 0.2101
Make sense?
Ray K
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] ABI version of experimental libraries
2020-02-19 11:43 7% ` [dpdk-dev] ABI version of experimental libraries Neil Horman
@ 2020-02-19 12:43 4% ` Thomas Monjalon
2020-02-19 13:50 4% ` Ray Kinsella
2020-02-19 21:17 4% ` Neil Horman
0 siblings, 2 replies; 200+ results
From: Thomas Monjalon @ 2020-02-19 12:43 UTC (permalink / raw)
To: Neil Horman
Cc: Bruce Richardson, ray.kinsella, bluca, david.marchand, ktraynor, dev
19/02/2020 12:43, Neil Horman:
> On Tue, Feb 18, 2020 at 10:50:09AM +0100, Thomas Monjalon wrote:
> > 18/02/2020 10:42, Bruce Richardson:
> > > On Tue, Feb 18, 2020 at 12:15:56AM +0100, Thomas Monjalon wrote:
> > > > Hi,
> > > >
> > > > I would like to remind everybody our mistake when defining ABI versions.
> > > > It has been "fixed" in this commit:
> > > > http://git.dpdk.org/dpdk/commit/?id=f26c2b39
> > > >
> > > > Please let's think about the consequence for the experimental libraries.
> > > >
> > > > In DPDK 19.11, we use the ABI version 0.200 with soname 0.20 In DPDK
> > > > 20.02, we use the ABI version 0.2001 with soname 0.201 Numbers are
> > > > increasing, that's fine. When we'll switch to the new major ABI and use
> > > > a normal numbering: In DPDK 20.11, we will use the ABI version 0.210 with
> > > > soname 0.21 Numbers are dropping.
> > > >
> > > > In short, for experimental libs, ABI 20.1 > ABI 21.0
> > > >
> > > > Are we OK with this or do we prefer reverting to normal numbering for
> > > > experimental libraries in DPDK 20.02?
> > > >
> > > Personally, I would not be too concerned about the verions of experimental
> > > libs, so long as they don't conflict across versions and have some
> > > similarity to the major ABI version for the release.
> >
> > You think sorting of the version numbers is not important?
> > If we don't care comparing experimental version numbers,
> > then OK, let's drop this patch. But please we need a small vote.
> >
> > Note: there would be no problem if we did not vote for having
> > a special numbering for pure experimental libraries (I am still against).
> >
> I don't understand. Why would we change the ABI_VERSION at all in an LTS release at
> all? This operation is meant to take an an experimental API and mark it as
> stable by promoting its version number to the next major releases number. As
> such, in the LTS release, we should keep the soname the same, as there should be
> no other ABI changes in the promoted API.
The library version number is updated because we add new symbols.
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH] net/ionic: fix ABI version
2020-02-17 16:13 7% [dpdk-dev] [PATCH] net/ionic: fix ABI version Ferruh Yigit
@ 2020-02-19 12:37 4% ` Ferruh Yigit
0 siblings, 0 replies; 200+ results
From: Ferruh Yigit @ 2020-02-19 12:37 UTC (permalink / raw)
To: Alfredo Cardigliano, Shannon Nelson; +Cc: dev, David Marchand
On 2/17/2020 4:13 PM, Ferruh Yigit wrote:
> Changed the ABI version to 20.0.1.
>
> Fixes: a72cf98cc93e ("net/ionic: add skeleton")
>
> Reported-by: David Marchand <david.marchand@redhat.com>
> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
Applied to dpdk-next-net/master, thanks.
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] ABI version of experimental libraries
2020-02-18 9:50 4% ` Thomas Monjalon
2020-02-18 10:36 4% ` Kinsella, Ray
@ 2020-02-19 11:43 7% ` Neil Horman
2020-02-19 12:43 4% ` Thomas Monjalon
1 sibling, 1 reply; 200+ results
From: Neil Horman @ 2020-02-19 11:43 UTC (permalink / raw)
To: Thomas Monjalon
Cc: Bruce Richardson, ray.kinsella, bluca, david.marchand, ktraynor, dev
On Tue, Feb 18, 2020 at 10:50:09AM +0100, Thomas Monjalon wrote:
> 18/02/2020 10:42, Bruce Richardson:
> > On Tue, Feb 18, 2020 at 12:15:56AM +0100, Thomas Monjalon wrote:
> > > Hi,
> > >
> > > I would like to remind everybody our mistake when defining ABI versions.
> > > It has been "fixed" in this commit:
> > > http://git.dpdk.org/dpdk/commit/?id=f26c2b39
> > >
> > > Please let's think about the consequence for the experimental libraries.
> > >
> > > In DPDK 19.11, we use the ABI version 0.200 with soname 0.20 In DPDK
> > > 20.02, we use the ABI version 0.2001 with soname 0.201 Numbers are
> > > increasing, that's fine. When we'll switch to the new major ABI and use
> > > a normal numbering: In DPDK 20.11, we will use the ABI version 0.210 with
> > > soname 0.21 Numbers are dropping.
> > >
> > > In short, for experimental libs, ABI 20.1 > ABI 21.0
> > >
> > > Are we OK with this or do we prefer reverting to normal numbering for
> > > experimental libraries in DPDK 20.02?
> > >
> > Personally, I would not be too concerned about the verions of experimental
> > libs, so long as they don't conflict across versions and have some
> > similarity to the major ABI version for the release.
>
> You think sorting of the version numbers is not important?
> If we don't care comparing experimental version numbers,
> then OK, let's drop this patch. But please we need a small vote.
>
> Note: there would be no problem if we did not vote for having
> a special numbering for pure experimental libraries (I am still against).
>
I don't understand. Why would we change the ABI_VERSION at all in an LTS release at
all? This operation is meant to take an an experimental API and mark it as
stable by promoting its version number to the next major releases number. As
such, in the LTS release, we should keep the soname the same, as there should be
no other ABI changes in the promoted API.
Neil
>
>
^ permalink raw reply [relevance 7%]
* Re: [dpdk-dev] [PATCH v2] ci: build and use libabigail 1.6
2020-02-18 16:38 0% ` David Marchand
@ 2020-02-19 8:13 0% ` Dodji Seketeli
0 siblings, 0 replies; 200+ results
From: Dodji Seketeli @ 2020-02-19 8:13 UTC (permalink / raw)
To: David Marchand
Cc: Thomas Monjalon, dev, Aaron Conole, Christian Ehrhardt,
Kevin Laatz, Michael Santana
Hello,
David Marchand <david.marchand@redhat.com> writes:
>> > + LIBABIGAIL_REPO=${LIBABIGAIL_REPO:-https://sourceware.org/git/libabigail.git}
>> > + LIBABIGAIL_VERSION=${LIBABIGAIL_VERSION:-libabigail-1.6}
>> > +
>> > + if [ "$(cat libabigail/VERSION 2>/dev/null)" != "$LIBABIGAIL_VERSION" ]; then
>> > + rm -rf libabigail
>> > + # if we change libabigail, invalidate existing abi cache
>> > + rm -rf reference
>> > + fi
>> > +
>> > + if [ ! -d libabigail ]; then
>> > + git clone --single-branch -b $LIBABIGAIL_VERSION $LIBABIGAIL_REPO libabigail/src
>>
>> Why not using the tarball?
>> http://mirrors.kernel.org/sourceware/libabigail/libabigail-1.6.tar.gz
>
> No good reason "now".
>
> I was first bitten by a reference to redhat-hardened-ld in some
> libtool script in the tarball (/me looks in Dodji direction).
> I then considered switching to different versions of libabigail by
> just setting the LIBABIGAIL_VERSION env variable from .travis.yml.
> I ended up with the current latest version which is also what is in
> Ubuntu latest releases.
I think either way (distro package, tarball and git) has some benefits
and drawbacks.
I think one benefit of /being able/ to use the code from git is for
cases where you guys need some new features (and we tend to continuously
add fixes/functionalities to the git repository) that is available in
git only for now. Hopefully, with time, you'll only need to use the
distro package.
FWIW, I like the fact that your setup lets you choose between the distro
package, the tarball or the git code. That's powerful.
Cheers,
--
Dodji
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v3 0/2] improve Travis coverage
@ 2020-02-18 22:36 3% ` David Marchand
0 siblings, 0 replies; 200+ results
From: David Marchand @ 2020-02-18 22:36 UTC (permalink / raw)
To: Thomas Monjalon; +Cc: dev, Aaron Conole, Bruce Richardson
On Tue, Feb 18, 2020 at 10:20 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> Remove some useless jobs and add more packages
> to build more PMDs in Travis.
>
> v3: reword commit logs
> v2: remove libbpf because not available
>
> Thomas Monjalon (2):
> ci: remove redundant configs disabling kmods
> ci: enable more drivers in Travis builds
>
> .travis.yml | 30 +++++-------------------------
> 1 file changed, 5 insertions(+), 25 deletions(-)
About enabling kmod compilation again, this can be done in a followup patch.
For the series,
Acked-by: David Marchand <david.marchand@redhat.com>
Rebased on master, cache for abi checks jobs deleted.
Series applied.
--
David Marchand
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH v3] ci: build and use libabigail 1.6
2020-02-18 20:31 0% ` Aaron Conole
@ 2020-02-18 20:59 0% ` David Marchand
0 siblings, 0 replies; 200+ results
From: David Marchand @ 2020-02-18 20:59 UTC (permalink / raw)
To: David Marchand
Cc: Thomas Monjalon, dev, Christian Ehrhardt, Dodji Seketeli,
Kevin Laatz, Michael Santana, Aaron Conole
On Tue, Feb 18, 2020 at 9:31 PM Aaron Conole <aconole@redhat.com> wrote:
> David Marchand <david.marchand@redhat.com> writes:
>
> > libabigail 1.2 (at least) reports changes in 'const' property as an ABI
> > breakage [1].
> > This was fixed upstream in libabigail 1.4 [2], and a bug has been opened
> > in launchpad [3].
> >
> > But for now, build and use the last version 1.6 so that the ABI checks
> > can be kept.
> >
> > 1: https://travis-ci.com/DPDK/dpdk/jobs/287872118#L2242
> > 2: https://sourceware.org/git/gitweb.cgi?p=libabigail.git;a=commitdiff;h=215b7eb4fe8b
> > 3: https://bugs.launchpad.net/ubuntu/+source/libabigail/+bug/1863607
> >
> > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > Acked-by: Thomas Monjalon <thomas@monjalon.net>
> > ---
>
> Acked-by: Aaron Conole <aconole@redhat.com>
Applied.
--
David Marchand
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v3] ci: build and use libabigail 1.6
2020-02-18 20:13 9% ` [dpdk-dev] [PATCH v3] " David Marchand
@ 2020-02-18 20:31 0% ` Aaron Conole
2020-02-18 20:59 0% ` David Marchand
0 siblings, 1 reply; 200+ results
From: Aaron Conole @ 2020-02-18 20:31 UTC (permalink / raw)
To: David Marchand
Cc: thomas, dev, christian.ehrhardt, dodji, kevin.laatz, Michael Santana
David Marchand <david.marchand@redhat.com> writes:
> libabigail 1.2 (at least) reports changes in 'const' property as an ABI
> breakage [1].
> This was fixed upstream in libabigail 1.4 [2], and a bug has been opened
> in launchpad [3].
>
> But for now, build and use the last version 1.6 so that the ABI checks
> can be kept.
>
> 1: https://travis-ci.com/DPDK/dpdk/jobs/287872118#L2242
> 2: https://sourceware.org/git/gitweb.cgi?p=libabigail.git;a=commitdiff;h=215b7eb4fe8b
> 3: https://bugs.launchpad.net/ubuntu/+source/libabigail/+bug/1863607
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> Acked-by: Thomas Monjalon <thomas@monjalon.net>
> ---
Acked-by: Aaron Conole <aconole@redhat.com>
^ permalink raw reply [relevance 0%]
* [dpdk-dev] [PATCH v3] ci: build and use libabigail 1.6
2020-02-17 13:59 9% [dpdk-dev] [PATCH] ci: build and use libabigail 1.6 David Marchand
` (2 preceding siblings ...)
2020-02-18 14:29 9% ` [dpdk-dev] [PATCH v2] " David Marchand
@ 2020-02-18 20:13 9% ` David Marchand
2020-02-18 20:31 0% ` Aaron Conole
3 siblings, 1 reply; 200+ results
From: David Marchand @ 2020-02-18 20:13 UTC (permalink / raw)
To: thomas
Cc: dev, aconole, christian.ehrhardt, dodji, kevin.laatz, Michael Santana
libabigail 1.2 (at least) reports changes in 'const' property as an ABI
breakage [1].
This was fixed upstream in libabigail 1.4 [2], and a bug has been opened
in launchpad [3].
But for now, build and use the last version 1.6 so that the ABI checks
can be kept.
1: https://travis-ci.com/DPDK/dpdk/jobs/287872118#L2242
2: https://sourceware.org/git/gitweb.cgi?p=libabigail.git;a=commitdiff;h=215b7eb4fe8b
3: https://bugs.launchpad.net/ubuntu/+source/libabigail/+bug/1863607
Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Thomas Monjalon <thomas@monjalon.net>
---
Changelog since v2:
- used tarball instead of git,
- moved libabigail compilation/install in a function,
Changelog since v1:
- keep extra_packages in ABI checks jobs,
- fix configure step,
---
.ci/linux-build.sh | 29 +++++++++++++++++++++++++++++
.travis.yml | 8 +++++++-
2 files changed, 36 insertions(+), 1 deletion(-)
diff --git a/.ci/linux-build.sh b/.ci/linux-build.sh
index c7c3840fc..d500c4c00 100755
--- a/.ci/linux-build.sh
+++ b/.ci/linux-build.sh
@@ -14,6 +14,20 @@ on_error() {
}
trap on_error EXIT
+install_libabigail() {
+ version=$1
+ instdir=$2
+
+ wget -q "http://mirrors.kernel.org/sourceware/libabigail/${version}.tar.gz"
+ tar -xf ${version}.tar.gz
+ cd $version && autoreconf -vfi && cd -
+ mkdir $version/build
+ cd $version/build && ../configure --prefix=$instdir && cd -
+ make -C $version/build all install
+ rm -rf $version
+ rm ${version}.tar.gz
+}
+
if [ "$AARCH64" = "1" ]; then
# convert the arch specifier
OPTS="$OPTS --cross-file config/arm/arm64_armv8_linux_gcc"
@@ -38,6 +52,21 @@ if [ "$AARCH64" != "1" ]; then
fi
if [ "$ABI_CHECKS" = "1" ]; then
+ LIBABIGAIL_VERSION=${LIBABIGAIL_VERSION:-libabigail-1.6}
+
+ if [ "$(cat libabigail/VERSION 2>/dev/null)" != "$LIBABIGAIL_VERSION" ]; then
+ rm -rf libabigail
+ # if we change libabigail, invalidate existing abi cache
+ rm -rf reference
+ fi
+
+ if [ ! -d libabigail ]; then
+ install_libabigail $LIBABIGAIL_VERSION $(pwd)/libabigail
+ echo $LIBABIGAIL_VERSION > libabigail/VERSION
+ fi
+
+ export PATH=$(pwd)/libabigail/bin:$PATH
+
REF_GIT_REPO=${REF_GIT_REPO:-https://dpdk.org/git/dpdk}
REF_GIT_TAG=${REF_GIT_TAG:-v19.11}
diff --git a/.travis.yml b/.travis.yml
index 22539d823..030064c94 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -2,6 +2,7 @@ language: c
cache:
ccache: true
directories:
+ - libabigail
- reference
compiler:
- gcc
@@ -24,7 +25,10 @@ aarch64_packages: &aarch64_packages
extra_packages: &extra_packages
- *required_packages
- - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4, abigail-tools]
+ - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
+
+libabigail_build_packages: &libabigail_build_packages
+ - [autoconf, automake, libtool, pkg-config, libxml2-dev, libdw-dev]
build_32b_packages: &build_32b_packages
- *required_packages
@@ -160,6 +164,7 @@ matrix:
apt:
packages:
- *extra_packages
+ - *libabigail_build_packages
- env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1
arch: arm64
compiler: gcc
@@ -167,5 +172,6 @@ matrix:
apt:
packages:
- *extra_packages
+ - *libabigail_build_packages
script: ./.ci/${TRAVIS_OS_NAME}-build.sh
--
2.23.0
^ permalink raw reply [relevance 9%]
* Re: [dpdk-dev] [PATCH v2] ethdev: add comment to warn of ABI breakage
2020-02-18 13:39 13% ` [dpdk-dev] [PATCH v2] " Ciara Power
@ 2020-02-18 18:40 4% ` Ferruh Yigit
0 siblings, 0 replies; 200+ results
From: Ferruh Yigit @ 2020-02-18 18:40 UTC (permalink / raw)
To: Ciara Power, thomas, arybchenko; +Cc: dev
On 2/18/2020 1:39 PM, Ciara Power wrote:
> If a function is added to the eth_dev_ops struct before
> tx_descriptor_status function, this will cause ABI breakage. This is due
> to static inline functions using this function, and some other functions
> above it in the struct, so they cannot change position. A comment is
> added to inform developers of this possible breakage.
>
> Signed-off-by: Ciara Power <ciara.power@intel.com>
> Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
> Reviewed-by: Andrew Rybchenko <arybchenko@solarflare.com>
Acked-by: Thomas Monjalon <thomas@monjalon.net>
Applied to dpdk-next-net/master, thanks.
(uppercase ABOVE/BELOW as suggested applied while merging)
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v2] ci: build and use libabigail 1.6
2020-02-18 15:46 0% ` Thomas Monjalon
@ 2020-02-18 16:38 0% ` David Marchand
2020-02-19 8:13 0% ` Dodji Seketeli
0 siblings, 1 reply; 200+ results
From: David Marchand @ 2020-02-18 16:38 UTC (permalink / raw)
To: Thomas Monjalon
Cc: dev, Aaron Conole, Christian Ehrhardt, Dodji Seketeli,
Kevin Laatz, Michael Santana
On Tue, Feb 18, 2020 at 4:46 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> 18/02/2020 15:29, David Marchand:
> > libabigail 1.2 (at least) reports changes in 'const' property as an ABI
> > breakage [1].
> > This was fixed upstream in libabigail 1.4 [2], and a bug has been opened
> > in launchpad [3].
> >
> > But for now, build and use the last version 1.6 so that the ABI checks
> > can be kept.
> >
> > 1: https://travis-ci.com/DPDK/dpdk/jobs/287872118#L2242
> > 2: https://sourceware.org/git/gitweb.cgi?p=libabigail.git;a=commitdiff;h=215b7eb4fe8b
> > 3: https://bugs.launchpad.net/ubuntu/+source/libabigail/+bug/1863607
> >
> > Signed-off-by: David Marchand <david.marchand@redhat.com>
>
> Acked-by: Thomas Monjalon <thomas@monjalon.net>
>
> I suggest few improvements below:
>
> > --- a/.ci/linux-build.sh
> > +++ b/.ci/linux-build.sh
> > if [ "$ABI_CHECKS" = "1" ]; then
>
> What do you think about moving the libabigail install in a function?
No strong opinion, we had everything inline so far.
>
> We could justify this with a comment about installing the latest version.
>
> > + LIBABIGAIL_REPO=${LIBABIGAIL_REPO:-https://sourceware.org/git/libabigail.git}
> > + LIBABIGAIL_VERSION=${LIBABIGAIL_VERSION:-libabigail-1.6}
> > +
> > + if [ "$(cat libabigail/VERSION 2>/dev/null)" != "$LIBABIGAIL_VERSION" ]; then
> > + rm -rf libabigail
> > + # if we change libabigail, invalidate existing abi cache
> > + rm -rf reference
> > + fi
> > +
> > + if [ ! -d libabigail ]; then
> > + git clone --single-branch -b $LIBABIGAIL_VERSION $LIBABIGAIL_REPO libabigail/src
>
> Why not using the tarball?
> http://mirrors.kernel.org/sourceware/libabigail/libabigail-1.6.tar.gz
No good reason "now".
I was first bitten by a reference to redhat-hardened-ld in some
libtool script in the tarball (/me looks in Dodji direction).
I then considered switching to different versions of libabigail by
just setting the LIBABIGAIL_VERSION env variable from .travis.yml.
I ended up with the current latest version which is also what is in
Ubuntu latest releases.
The tarball is smaller than a git clone, so best to use it.
v3 incoming.
--
David Marchand
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v2] ci: build and use libabigail 1.6
2020-02-18 14:29 9% ` [dpdk-dev] [PATCH v2] " David Marchand
@ 2020-02-18 15:46 0% ` Thomas Monjalon
2020-02-18 16:38 0% ` David Marchand
0 siblings, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-02-18 15:46 UTC (permalink / raw)
To: David Marchand
Cc: dev, aconole, christian.ehrhardt, dodji, kevin.laatz, Michael Santana
18/02/2020 15:29, David Marchand:
> libabigail 1.2 (at least) reports changes in 'const' property as an ABI
> breakage [1].
> This was fixed upstream in libabigail 1.4 [2], and a bug has been opened
> in launchpad [3].
>
> But for now, build and use the last version 1.6 so that the ABI checks
> can be kept.
>
> 1: https://travis-ci.com/DPDK/dpdk/jobs/287872118#L2242
> 2: https://sourceware.org/git/gitweb.cgi?p=libabigail.git;a=commitdiff;h=215b7eb4fe8b
> 3: https://bugs.launchpad.net/ubuntu/+source/libabigail/+bug/1863607
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Thomas Monjalon <thomas@monjalon.net>
I suggest few improvements below:
> --- a/.ci/linux-build.sh
> +++ b/.ci/linux-build.sh
> if [ "$ABI_CHECKS" = "1" ]; then
What do you think about moving the libabigail install in a function?
We could justify this with a comment about installing the latest version.
> + LIBABIGAIL_REPO=${LIBABIGAIL_REPO:-https://sourceware.org/git/libabigail.git}
> + LIBABIGAIL_VERSION=${LIBABIGAIL_VERSION:-libabigail-1.6}
> +
> + if [ "$(cat libabigail/VERSION 2>/dev/null)" != "$LIBABIGAIL_VERSION" ]; then
> + rm -rf libabigail
> + # if we change libabigail, invalidate existing abi cache
> + rm -rf reference
> + fi
> +
> + if [ ! -d libabigail ]; then
> + git clone --single-branch -b $LIBABIGAIL_VERSION $LIBABIGAIL_REPO libabigail/src
Why not using the tarball?
http://mirrors.kernel.org/sourceware/libabigail/libabigail-1.6.tar.gz
> + cd libabigail/src && autoreconf -i && cd -
> + instdir=$(pwd)/libabigail
> + mkdir libabigail/src/build
> + cd libabigail/src/build && ../configure --prefix=$instdir && cd -
> + make -C libabigail/src/build all install
> +
> + rm -rf libabigail/src
> + echo $LIBABIGAIL_VERSION > libabigail/VERSION
> + fi
> +
> + export PATH=$(pwd)/libabigail/bin:$PATH
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] ci: build and use libabigail 1.6
2020-02-18 14:55 0% ` Aaron Conole
@ 2020-02-18 15:07 0% ` David Marchand
0 siblings, 0 replies; 200+ results
From: David Marchand @ 2020-02-18 15:07 UTC (permalink / raw)
To: Aaron Conole
Cc: Thomas Monjalon, dev, Christian Ehrhardt, Dodji Seketeli,
Michael Santana
On Tue, Feb 18, 2020 at 3:55 PM Aaron Conole <aconole@redhat.com> wrote:
>
> David Marchand <david.marchand@redhat.com> writes:
>
> > On Tue, Feb 18, 2020 at 10:40 AM David Marchand
> > <david.marchand@redhat.com> wrote:
> >>
> >> On Mon, Feb 17, 2020 at 7:48 PM Aaron Conole <aconole@redhat.com> wrote:
> >> >
> >> > David Marchand <david.marchand@redhat.com> writes:
> >> >
> >> > > libabigail 1.2 (at least) reports changes in 'const' property as an ABI
> >> > > breakage [1].
> >> > > This was fixed upstream in libabigail 1.4 [2], and a bug has been opened
> >> > > in launchpad [3].
> >> > >
> >> > > But for now, build and use the last version 1.6 so that the ABI checks
> >> > > can be kept.
> >> > >
> >> > > 1: https://travis-ci.com/DPDK/dpdk/jobs/287872118#L2242
> >> > > 2:
> >> > > https://sourceware.org/git/gitweb.cgi?p=libabigail.git;a=commitdiff;h=215b7eb4fe8b986fe1cc87d9d8e7412998038392
> >> > > 3: https://bugs.launchpad.net/ubuntu/+source/libabigail/+bug/1863607
> >> > >
> >> > > Signed-off-by: David Marchand <david.marchand@redhat.com>
> >> > > ---
> >> >
> >> > Does it make sense to base libabigail required ontop of extra packages?
> >> > Otherwise some libraries won't get built / checked, no?
> >>
> >> The only change I see is the pcap driver being enabled.
> >> On the principle, I agree that trying to build all possible
> >> libraries/drivers is better when checking the ABI.
> >> So I'll keep extra_packages yes.
> >>
> >> I am currently testing that touching extra_packages (well, testing
> >> Thomas patches) results in Travis treating the job as a new one (i.e.
> >> with no cache).
> >
> > Travis bases each job cache on the job description:
> > https://docs.travis-ci.com/user/caching/
> >
> > I tested Thomas change on extra_packages content, and the job used the
> > old cache.
> > My idea was to try to put *extra_packages in an env variable, but it
> > does not work (my yaml-fu is lacking).
> >
> > If there is no easy way, I will invalidate the cache manually.
>
> We don't actually use the EXTRA_PACKAGES variable for anything, so I
> guess it's probably okay to change the value and that should invalidate
> the cache. Most of the variables, in fact, could be checked for
> non-zero value rather than a specific positive value, and then it's easy
> to invalidate the cache by just bumping them. It's a thought (and
> kindof a hack). Or we can just use the travis CLI tool and delete the
> caches (we'll have to do that for the ovsrobot as well, I think).
>
What I had in mind was to convert the extra_packages yaml thing into a
string to pass into EXTRA_PACKAGES.
But I did not manage.
About bumping the value, users are likely to be unaware of this step
if they submit a patch touching .travis.yml.
Deleting the caches from ovsrobot if .travis.yml has been touched seems simpler.
On master, I will stick to manual cache invalidation.
--
David Marchand
--
David Marchand
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] ci: build and use libabigail 1.6
2020-02-18 11:18 0% ` David Marchand
@ 2020-02-18 14:55 0% ` Aaron Conole
2020-02-18 15:07 0% ` David Marchand
0 siblings, 1 reply; 200+ results
From: Aaron Conole @ 2020-02-18 14:55 UTC (permalink / raw)
To: David Marchand
Cc: Thomas Monjalon, dev, Christian Ehrhardt, Dodji Seketeli,
Michael Santana
David Marchand <david.marchand@redhat.com> writes:
> On Tue, Feb 18, 2020 at 10:40 AM David Marchand
> <david.marchand@redhat.com> wrote:
>>
>> On Mon, Feb 17, 2020 at 7:48 PM Aaron Conole <aconole@redhat.com> wrote:
>> >
>> > David Marchand <david.marchand@redhat.com> writes:
>> >
>> > > libabigail 1.2 (at least) reports changes in 'const' property as an ABI
>> > > breakage [1].
>> > > This was fixed upstream in libabigail 1.4 [2], and a bug has been opened
>> > > in launchpad [3].
>> > >
>> > > But for now, build and use the last version 1.6 so that the ABI checks
>> > > can be kept.
>> > >
>> > > 1: https://travis-ci.com/DPDK/dpdk/jobs/287872118#L2242
>> > > 2:
>> > > https://sourceware.org/git/gitweb.cgi?p=libabigail.git;a=commitdiff;h=215b7eb4fe8b986fe1cc87d9d8e7412998038392
>> > > 3: https://bugs.launchpad.net/ubuntu/+source/libabigail/+bug/1863607
>> > >
>> > > Signed-off-by: David Marchand <david.marchand@redhat.com>
>> > > ---
>> >
>> > Does it make sense to base libabigail required ontop of extra packages?
>> > Otherwise some libraries won't get built / checked, no?
>>
>> The only change I see is the pcap driver being enabled.
>> On the principle, I agree that trying to build all possible
>> libraries/drivers is better when checking the ABI.
>> So I'll keep extra_packages yes.
>>
>> I am currently testing that touching extra_packages (well, testing
>> Thomas patches) results in Travis treating the job as a new one (i.e.
>> with no cache).
>
> Travis bases each job cache on the job description:
> https://docs.travis-ci.com/user/caching/
>
> I tested Thomas change on extra_packages content, and the job used the
> old cache.
> My idea was to try to put *extra_packages in an env variable, but it
> does not work (my yaml-fu is lacking).
>
> If there is no easy way, I will invalidate the cache manually.
We don't actually use the EXTRA_PACKAGES variable for anything, so I
guess it's probably okay to change the value and that should invalidate
the cache. Most of the variables, in fact, could be checked for
non-zero value rather than a specific positive value, and then it's easy
to invalidate the cache by just bumping them. It's a thought (and
kindof a hack). Or we can just use the travis CLI tool and delete the
caches (we'll have to do that for the ovsrobot as well, I think).
>
> --
> David Marchand
^ permalink raw reply [relevance 0%]
* [dpdk-dev] [PATCH v2] ci: build and use libabigail 1.6
2020-02-17 13:59 9% [dpdk-dev] [PATCH] ci: build and use libabigail 1.6 David Marchand
2020-02-17 15:15 0% ` Thomas Monjalon
2020-02-17 18:47 0% ` Aaron Conole
@ 2020-02-18 14:29 9% ` David Marchand
2020-02-18 15:46 0% ` Thomas Monjalon
2020-02-18 20:13 9% ` [dpdk-dev] [PATCH v3] " David Marchand
3 siblings, 1 reply; 200+ results
From: David Marchand @ 2020-02-18 14:29 UTC (permalink / raw)
To: thomas
Cc: dev, aconole, christian.ehrhardt, dodji, kevin.laatz, Michael Santana
libabigail 1.2 (at least) reports changes in 'const' property as an ABI
breakage [1].
This was fixed upstream in libabigail 1.4 [2], and a bug has been opened
in launchpad [3].
But for now, build and use the last version 1.6 so that the ABI checks
can be kept.
1: https://travis-ci.com/DPDK/dpdk/jobs/287872118#L2242
2: https://sourceware.org/git/gitweb.cgi?p=libabigail.git;a=commitdiff;h=215b7eb4fe8b
3: https://bugs.launchpad.net/ubuntu/+source/libabigail/+bug/1863607
Signed-off-by: David Marchand <david.marchand@redhat.com>
---
Changelog since v1:
- keep extra_packages in ABI checks jobs,
- fix configure step,
---
.ci/linux-build.sh | 23 +++++++++++++++++++++++
.travis.yml | 8 +++++++-
2 files changed, 30 insertions(+), 1 deletion(-)
diff --git a/.ci/linux-build.sh b/.ci/linux-build.sh
index c7c3840fc..cb9627bd5 100755
--- a/.ci/linux-build.sh
+++ b/.ci/linux-build.sh
@@ -38,6 +38,29 @@ if [ "$AARCH64" != "1" ]; then
fi
if [ "$ABI_CHECKS" = "1" ]; then
+ LIBABIGAIL_REPO=${LIBABIGAIL_REPO:-https://sourceware.org/git/libabigail.git}
+ LIBABIGAIL_VERSION=${LIBABIGAIL_VERSION:-libabigail-1.6}
+
+ if [ "$(cat libabigail/VERSION 2>/dev/null)" != "$LIBABIGAIL_VERSION" ]; then
+ rm -rf libabigail
+ # if we change libabigail, invalidate existing abi cache
+ rm -rf reference
+ fi
+
+ if [ ! -d libabigail ]; then
+ git clone --single-branch -b $LIBABIGAIL_VERSION $LIBABIGAIL_REPO libabigail/src
+ cd libabigail/src && autoreconf -i && cd -
+ instdir=$(pwd)/libabigail
+ mkdir libabigail/src/build
+ cd libabigail/src/build && ../configure --prefix=$instdir && cd -
+ make -C libabigail/src/build all install
+
+ rm -rf libabigail/src
+ echo $LIBABIGAIL_VERSION > libabigail/VERSION
+ fi
+
+ export PATH=$(pwd)/libabigail/bin:$PATH
+
REF_GIT_REPO=${REF_GIT_REPO:-https://dpdk.org/git/dpdk}
REF_GIT_TAG=${REF_GIT_TAG:-v19.11}
diff --git a/.travis.yml b/.travis.yml
index 22539d823..030064c94 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -2,6 +2,7 @@ language: c
cache:
ccache: true
directories:
+ - libabigail
- reference
compiler:
- gcc
@@ -24,7 +25,10 @@ aarch64_packages: &aarch64_packages
extra_packages: &extra_packages
- *required_packages
- - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4, abigail-tools]
+ - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
+
+libabigail_build_packages: &libabigail_build_packages
+ - [autoconf, automake, libtool, pkg-config, libxml2-dev, libdw-dev]
build_32b_packages: &build_32b_packages
- *required_packages
@@ -160,6 +164,7 @@ matrix:
apt:
packages:
- *extra_packages
+ - *libabigail_build_packages
- env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1
arch: arm64
compiler: gcc
@@ -167,5 +172,6 @@ matrix:
apt:
packages:
- *extra_packages
+ - *libabigail_build_packages
script: ./.ci/${TRAVIS_OS_NAME}-build.sh
--
2.23.0
^ permalink raw reply [relevance 9%]
* Re: [dpdk-dev] [PATCH] ethdev: add comment to warn of ABI breakage
2020-02-18 13:13 4% ` Andrew Rybchenko
@ 2020-02-18 14:20 4% ` Thomas Monjalon
0 siblings, 0 replies; 200+ results
From: Thomas Monjalon @ 2020-02-18 14:20 UTC (permalink / raw)
To: Ciara Power; +Cc: Ferruh Yigit, Andrew Rybchenko, dev
18/02/2020 14:13, Andrew Rybchenko:
> On 2/18/20 2:59 PM, Ferruh Yigit wrote:
> > On 2/18/2020 10:46 AM, Ciara Power wrote:
> >> If a function is added to the eth_dev_ops struct before
> >> tx_descriptor_status function, this will cause ABI breakage. This is due
> >> to static inline functions using this function, and some other functions
> >> above it in the struct, so they cannot change position. A comment is
> >> added to inform developers of this possible breakage.
> >>
> >> Signed-off-by: Ciara Power <ciara.power@intel.com>
> >> ---
> >> lib/librte_ethdev/rte_ethdev_core.h | 3 +++
> >> 1 file changed, 3 insertions(+)
> >>
> >> diff --git a/lib/librte_ethdev/rte_ethdev_core.h b/lib/librte_ethdev/rte_ethdev_core.h
> >> index 7bf97e24e..1ad04a129 100644
> >> --- a/lib/librte_ethdev/rte_ethdev_core.h
> >> +++ b/lib/librte_ethdev/rte_ethdev_core.h
> >> @@ -667,6 +667,9 @@ struct eth_dev_ops {
> >> /**< Check the status of a Rx descriptor. */
> >> eth_tx_descriptor_status_t tx_descriptor_status;
> >> /**< Check the status of a Tx descriptor. */
> >> + /* Static inline functions use functions above this comment.
> >> + * New dev_ops functions should be added below to avoid breaking ABI.
> >> + */
> >> eth_rx_enable_intr_t rx_queue_intr_enable; /**< Enable Rx queue interrupt. */
> >> eth_rx_disable_intr_t rx_queue_intr_disable; /**< Disable Rx queue interrupt. */
> >> eth_tx_queue_setup_t tx_queue_setup;/**< Set up device TX queue. */
> >>
> >
> > Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
>
> LGTM, but may I suggest to make the first line in the comment
> empty to make it easier to notice. I.e.
> /*
> * Static inline functions use functions above this comment.
> * ...
> */
>
> Or even add empty lines before and after the comment.
>
> Reviewed-by: Andrew Rybchenko <arybchenko@solarflare.com>
In the same idea as Andrew, I would suggest some uppercases,
maybe "ABOVE" and "BELOW".
Acked-by: Thomas Monjalon <thomas@monjalon.net>
^ permalink raw reply [relevance 4%]
* [dpdk-dev] [PATCH v2] ethdev: add comment to warn of ABI breakage
2020-02-18 10:46 13% [dpdk-dev] [PATCH] ethdev: add comment to warn of ABI breakage Ciara Power
2020-02-18 11:59 4% ` Ferruh Yigit
@ 2020-02-18 13:39 13% ` Ciara Power
2020-02-18 18:40 4% ` Ferruh Yigit
1 sibling, 1 reply; 200+ results
From: Ciara Power @ 2020-02-18 13:39 UTC (permalink / raw)
To: thomas, ferruh.yigit, arybchenko; +Cc: dev, Ciara Power
If a function is added to the eth_dev_ops struct before
tx_descriptor_status function, this will cause ABI breakage. This is due
to static inline functions using this function, and some other functions
above it in the struct, so they cannot change position. A comment is
added to inform developers of this possible breakage.
Signed-off-by: Ciara Power <ciara.power@intel.com>
Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
Reviewed-by: Andrew Rybchenko <arybchenko@solarflare.com>
---
V2: Added empty first line in comment
---
lib/librte_ethdev/rte_ethdev_core.h | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/lib/librte_ethdev/rte_ethdev_core.h b/lib/librte_ethdev/rte_ethdev_core.h
index 7bf97e24e..9933ffdc1 100644
--- a/lib/librte_ethdev/rte_ethdev_core.h
+++ b/lib/librte_ethdev/rte_ethdev_core.h
@@ -667,6 +667,10 @@ struct eth_dev_ops {
/**< Check the status of a Rx descriptor. */
eth_tx_descriptor_status_t tx_descriptor_status;
/**< Check the status of a Tx descriptor. */
+ /*
+ * Static inline functions use functions above this comment.
+ * New dev_ops functions should be added below to avoid breaking ABI.
+ */
eth_rx_enable_intr_t rx_queue_intr_enable; /**< Enable Rx queue interrupt. */
eth_rx_disable_intr_t rx_queue_intr_disable; /**< Disable Rx queue interrupt. */
eth_tx_queue_setup_t tx_queue_setup;/**< Set up device TX queue. */
--
2.17.1
^ permalink raw reply [relevance 13%]
* Re: [dpdk-dev] [PATCH] ethdev: add comment to warn of ABI breakage
2020-02-18 11:59 4% ` Ferruh Yigit
@ 2020-02-18 13:13 4% ` Andrew Rybchenko
2020-02-18 14:20 4% ` Thomas Monjalon
0 siblings, 1 reply; 200+ results
From: Andrew Rybchenko @ 2020-02-18 13:13 UTC (permalink / raw)
To: Ferruh Yigit, Ciara Power, thomas, arybchenko; +Cc: dev
On 2/18/20 2:59 PM, Ferruh Yigit wrote:
> On 2/18/2020 10:46 AM, Ciara Power wrote:
>> If a function is added to the eth_dev_ops struct before
>> tx_descriptor_status function, this will cause ABI breakage. This is due
>> to static inline functions using this function, and some other functions
>> above it in the struct, so they cannot change position. A comment is
>> added to inform developers of this possible breakage.
>>
>> Signed-off-by: Ciara Power <ciara.power@intel.com>
>> ---
>> lib/librte_ethdev/rte_ethdev_core.h | 3 +++
>> 1 file changed, 3 insertions(+)
>>
>> diff --git a/lib/librte_ethdev/rte_ethdev_core.h b/lib/librte_ethdev/rte_ethdev_core.h
>> index 7bf97e24e..1ad04a129 100644
>> --- a/lib/librte_ethdev/rte_ethdev_core.h
>> +++ b/lib/librte_ethdev/rte_ethdev_core.h
>> @@ -667,6 +667,9 @@ struct eth_dev_ops {
>> /**< Check the status of a Rx descriptor. */
>> eth_tx_descriptor_status_t tx_descriptor_status;
>> /**< Check the status of a Tx descriptor. */
>> + /* Static inline functions use functions above this comment.
>> + * New dev_ops functions should be added below to avoid breaking ABI.
>> + */
>> eth_rx_enable_intr_t rx_queue_intr_enable; /**< Enable Rx queue interrupt. */
>> eth_rx_disable_intr_t rx_queue_intr_disable; /**< Disable Rx queue interrupt. */
>> eth_tx_queue_setup_t tx_queue_setup;/**< Set up device TX queue. */
>>
>
> Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
LGTM, but may I suggest to make the first line in the comment
empty to make it easier to notice. I.e.
/*
* Static inline functions use functions above this comment.
* ...
*/
Or even add empty lines before and after the comment.
Reviewed-by: Andrew Rybchenko <arybchenko@solarflare.com>
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH] ethdev: add comment to warn of ABI breakage
2020-02-18 10:46 13% [dpdk-dev] [PATCH] ethdev: add comment to warn of ABI breakage Ciara Power
@ 2020-02-18 11:59 4% ` Ferruh Yigit
2020-02-18 13:13 4% ` Andrew Rybchenko
2020-02-18 13:39 13% ` [dpdk-dev] [PATCH v2] " Ciara Power
1 sibling, 1 reply; 200+ results
From: Ferruh Yigit @ 2020-02-18 11:59 UTC (permalink / raw)
To: Ciara Power, thomas, arybchenko; +Cc: dev
On 2/18/2020 10:46 AM, Ciara Power wrote:
> If a function is added to the eth_dev_ops struct before
> tx_descriptor_status function, this will cause ABI breakage. This is due
> to static inline functions using this function, and some other functions
> above it in the struct, so they cannot change position. A comment is
> added to inform developers of this possible breakage.
>
> Signed-off-by: Ciara Power <ciara.power@intel.com>
> ---
> lib/librte_ethdev/rte_ethdev_core.h | 3 +++
> 1 file changed, 3 insertions(+)
>
> diff --git a/lib/librte_ethdev/rte_ethdev_core.h b/lib/librte_ethdev/rte_ethdev_core.h
> index 7bf97e24e..1ad04a129 100644
> --- a/lib/librte_ethdev/rte_ethdev_core.h
> +++ b/lib/librte_ethdev/rte_ethdev_core.h
> @@ -667,6 +667,9 @@ struct eth_dev_ops {
> /**< Check the status of a Rx descriptor. */
> eth_tx_descriptor_status_t tx_descriptor_status;
> /**< Check the status of a Tx descriptor. */
> + /* Static inline functions use functions above this comment.
> + * New dev_ops functions should be added below to avoid breaking ABI.
> + */
> eth_rx_enable_intr_t rx_queue_intr_enable; /**< Enable Rx queue interrupt. */
> eth_rx_disable_intr_t rx_queue_intr_disable; /**< Disable Rx queue interrupt. */
> eth_tx_queue_setup_t tx_queue_setup;/**< Set up device TX queue. */
>
Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH] ci: build and use libabigail 1.6
2020-02-18 9:40 3% ` David Marchand
@ 2020-02-18 11:18 0% ` David Marchand
2020-02-18 14:55 0% ` Aaron Conole
0 siblings, 1 reply; 200+ results
From: David Marchand @ 2020-02-18 11:18 UTC (permalink / raw)
To: Aaron Conole
Cc: Thomas Monjalon, dev, Christian Ehrhardt, Dodji Seketeli,
Michael Santana
On Tue, Feb 18, 2020 at 10:40 AM David Marchand
<david.marchand@redhat.com> wrote:
>
> On Mon, Feb 17, 2020 at 7:48 PM Aaron Conole <aconole@redhat.com> wrote:
> >
> > David Marchand <david.marchand@redhat.com> writes:
> >
> > > libabigail 1.2 (at least) reports changes in 'const' property as an ABI
> > > breakage [1].
> > > This was fixed upstream in libabigail 1.4 [2], and a bug has been opened
> > > in launchpad [3].
> > >
> > > But for now, build and use the last version 1.6 so that the ABI checks
> > > can be kept.
> > >
> > > 1: https://travis-ci.com/DPDK/dpdk/jobs/287872118#L2242
> > > 2: https://sourceware.org/git/gitweb.cgi?p=libabigail.git;a=commitdiff;h=215b7eb4fe8b986fe1cc87d9d8e7412998038392
> > > 3: https://bugs.launchpad.net/ubuntu/+source/libabigail/+bug/1863607
> > >
> > > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > > ---
> >
> > Does it make sense to base libabigail required ontop of extra packages?
> > Otherwise some libraries won't get built / checked, no?
>
> The only change I see is the pcap driver being enabled.
> On the principle, I agree that trying to build all possible
> libraries/drivers is better when checking the ABI.
> So I'll keep extra_packages yes.
>
> I am currently testing that touching extra_packages (well, testing
> Thomas patches) results in Travis treating the job as a new one (i.e.
> with no cache).
Travis bases each job cache on the job description:
https://docs.travis-ci.com/user/caching/
I tested Thomas change on extra_packages content, and the job used the
old cache.
My idea was to try to put *extra_packages in an env variable, but it
does not work (my yaml-fu is lacking).
If there is no easy way, I will invalidate the cache manually.
--
David Marchand
^ permalink raw reply [relevance 0%]
* [dpdk-dev] [PATCH] ethdev: add comment to warn of ABI breakage
@ 2020-02-18 10:46 13% Ciara Power
2020-02-18 11:59 4% ` Ferruh Yigit
2020-02-18 13:39 13% ` [dpdk-dev] [PATCH v2] " Ciara Power
0 siblings, 2 replies; 200+ results
From: Ciara Power @ 2020-02-18 10:46 UTC (permalink / raw)
To: thomas, ferruh.yigit, arybchenko; +Cc: dev, Ciara Power
If a function is added to the eth_dev_ops struct before
tx_descriptor_status function, this will cause ABI breakage. This is due
to static inline functions using this function, and some other functions
above it in the struct, so they cannot change position. A comment is
added to inform developers of this possible breakage.
Signed-off-by: Ciara Power <ciara.power@intel.com>
---
lib/librte_ethdev/rte_ethdev_core.h | 3 +++
1 file changed, 3 insertions(+)
diff --git a/lib/librte_ethdev/rte_ethdev_core.h b/lib/librte_ethdev/rte_ethdev_core.h
index 7bf97e24e..1ad04a129 100644
--- a/lib/librte_ethdev/rte_ethdev_core.h
+++ b/lib/librte_ethdev/rte_ethdev_core.h
@@ -667,6 +667,9 @@ struct eth_dev_ops {
/**< Check the status of a Rx descriptor. */
eth_tx_descriptor_status_t tx_descriptor_status;
/**< Check the status of a Tx descriptor. */
+ /* Static inline functions use functions above this comment.
+ * New dev_ops functions should be added below to avoid breaking ABI.
+ */
eth_rx_enable_intr_t rx_queue_intr_enable; /**< Enable Rx queue interrupt. */
eth_rx_disable_intr_t rx_queue_intr_disable; /**< Disable Rx queue interrupt. */
eth_tx_queue_setup_t tx_queue_setup;/**< Set up device TX queue. */
--
2.17.1
^ permalink raw reply [relevance 13%]
* [dpdk-dev] [PATCH v4] doc: alias to experimental tag for stable apis
2020-02-05 15:17 17% [dpdk-dev] [PATCH] doc: alias to experimental tag for stable apis Ray Kinsella
2020-02-05 15:43 17% ` [dpdk-dev] [PATCH v3] " Ray Kinsella
@ 2020-02-18 10:46 17% ` Ray Kinsella
2020-02-21 17:11 0% ` Ferruh Yigit
2020-02-25 14:22 0% ` Kevin Traynor
2020-02-25 15:35 0% ` [dpdk-dev] [PATCH] " Neil Horman
2 siblings, 2 replies; 200+ results
From: Ray Kinsella @ 2020-02-18 10:46 UTC (permalink / raw)
To: dev
Cc: john.mcnamara, nhorman, ferruh.yigit, marko.kovacevic,
bruce.richardson, Ray Kinsella
When a maintainer is promoting an API to become part of the next major ABI
version by removing the experimental tag, possibly a few releases in
advance of the declaration of the next ABI version. The maintainer may
choose to offer an alias to the experimental tag, as removing the tag
before the declaration of the next major ABI version, would cause an ABI
breakage for applications using the API.
Signed-off-by: Ray Kinsella <mdr@ashroe.eu>
---
doc/guides/contributing/abi_policy.rst | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/doc/guides/contributing/abi_policy.rst b/doc/guides/contributing/abi_policy.rst
index 05ca959..7ae7de7 100644
--- a/doc/guides/contributing/abi_policy.rst
+++ b/doc/guides/contributing/abi_policy.rst
@@ -159,6 +159,11 @@ The requirements for changing the ABI are:
``experimental``, as described in the section on :ref:`Experimental APIs
and Libraries <experimental_apis>`.
+ - In situations in which an ``experimental`` API has been stable for some
+ time. When promoting the API to become part of the next ABI version, the
+ maintainer may choose to provide an alias to the ``experimental`` tag, so
+ as not to break consuming applications.
+
#. If a newly proposed API functionally replaces an existing one, when the new
API becomes non-experimental, then the old one is marked with
``__rte_deprecated``.
@@ -317,6 +322,11 @@ not required. Though, an API should remain in experimental state for at least
one release. Thereafter, the normal process of posting patch for review to
mailing list can be followed.
+After the experimental tag has been formally removed, a tree/sub-tree maintainer
+may choose to offer an alias to the experimental tag so as not to break
+applications using the API. The alias is then dropped at the declaration of next
+major ABI version.
+
Libraries
~~~~~~~~~
--
2.7.4
^ permalink raw reply [relevance 17%]
* Re: [dpdk-dev] ABI version of experimental libraries
2020-02-18 9:42 7% ` [dpdk-dev] ABI version of " Bruce Richardson
@ 2020-02-18 9:50 4% ` Thomas Monjalon
2020-02-18 10:36 4% ` Kinsella, Ray
2020-02-19 11:43 7% ` [dpdk-dev] ABI version of experimental libraries Neil Horman
0 siblings, 2 replies; 200+ results
From: Thomas Monjalon @ 2020-02-18 9:50 UTC (permalink / raw)
To: Bruce Richardson
Cc: ray.kinsella, nhorman, bluca, david.marchand, ktraynor, dev
18/02/2020 10:42, Bruce Richardson:
> On Tue, Feb 18, 2020 at 12:15:56AM +0100, Thomas Monjalon wrote:
> > Hi,
> >
> > I would like to remind everybody our mistake when defining ABI versions.
> > It has been "fixed" in this commit:
> > http://git.dpdk.org/dpdk/commit/?id=f26c2b39
> >
> > Please let's think about the consequence for the experimental libraries.
> >
> > In DPDK 19.11, we use the ABI version 0.200 with soname 0.20 In DPDK
> > 20.02, we use the ABI version 0.2001 with soname 0.201 Numbers are
> > increasing, that's fine. When we'll switch to the new major ABI and use
> > a normal numbering: In DPDK 20.11, we will use the ABI version 0.210 with
> > soname 0.21 Numbers are dropping.
> >
> > In short, for experimental libs, ABI 20.1 > ABI 21.0
> >
> > Are we OK with this or do we prefer reverting to normal numbering for
> > experimental libraries in DPDK 20.02?
> >
> Personally, I would not be too concerned about the verions of experimental
> libs, so long as they don't conflict across versions and have some
> similarity to the major ABI version for the release.
You think sorting of the version numbers is not important?
If we don't care comparing experimental version numbers,
then OK, let's drop this patch. But please we need a small vote.
Note: there would be no problem if we did not vote for having
a special numbering for pure experimental libraries (I am still against).
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH] build: fix soname for experimental libraries
2020-02-18 9:40 0% ` Bruce Richardson
@ 2020-02-18 9:47 0% ` Thomas Monjalon
0 siblings, 0 replies; 200+ results
From: Thomas Monjalon @ 2020-02-18 9:47 UTC (permalink / raw)
To: Bruce Richardson
Cc: dev, ray.kinsella, nhorman, bluca, david.marchand, ktraynor
18/02/2020 10:40, Bruce Richardson:
> On Tue, Feb 18, 2020 at 12:44:02AM +0100, Thomas Monjalon wrote:
> > Because of an original mistake in ABI numbering,
> > and a temporary workaround for ABI 20,
> > for experimental libs, numbering would lead to consider
> > ABI 20.1 > ABI 21.0
> >
> > Before this patch:
> >
> > DPDK 19.11: ABI version 0.200 and soname 0.20
> > DPDK 20.02: ABI version 0.2001 and soname 0.201
> > Numbers are increasing, that's fine.
> > For the next major ABI, back to normal numbering:
> > DPDK 20.11: ABI version 0.210 and soname 0.21
> > Numbers are decreasing!
> >
> > After this patch:
> >
> > DPDK 19.11: ABI version 0.200 and soname 0.20
> > DPDK 20.02: ABI version 0.201 and soname 0.20
> > DPDK 20.11: ABI version 0.210 and soname 0.21
> >
> > Fixes: f26c2b39b271 ("build: fix soname info for 19.11 compatibility")
> >
> > Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
> > ---
> > config/meson.build | 8 ++++----
> > mk/rte.lib.mk | 4 ++--
> > 2 files changed, 6 insertions(+), 6 deletions(-)
> >
> > diff --git a/config/meson.build b/config/meson.build
> > index 6c46767e3e..e7cd74e2c2 100644
> > --- a/config/meson.build
> > +++ b/config/meson.build
> > @@ -27,12 +27,12 @@ abi_version = run_command(find_program('cat', 'more'),
> > # and the filename suffix as 0.majorminor versions,
> > # e.g. v20.1 => librte_stable.so.20.1, librte_experimental.so.0.201
> > # sonames => librte_stable.so.20, librte_experimental.so.0.20
> > -# e.g. v20.0.1 => librte_stable.so.20.0.1, librte_experimental.so.0.2001
> > -# sonames => librte_stable.so.20.0, librte_experimental.so.0.200
> > +# e.g. v20.0.1 => librte_stable.so.20.0.1, librte_experimental.so.0.201
> > +# sonames => librte_stable.so.20.0, librte_experimental.so.0.20
> > abi_va = abi_version.split('.')
> > stable_so_version = abi_va.length() == 2 ? abi_va[0] : abi_va[0] + '.' + abi_va[1]
> > -experimental_abi_version = '0.' + ''.join(abi_va)
> > -experimental_so_version = '0.' + ''.join(stable_so_version.split('.'))
> > +experimental_abi_version = '0.' + ''.join([abi_va[0], abi_va[2]])
> > +experimental_so_version = '0.' + ''.join([abi_va[0]])
> >
>
> My concern about this is that it will break, or rather need to be changed
> again for the 20.11 release. While I see the numbering as not-ideal in
> terms of version numbers, the existing scheme was originally designed to
> work with either 3-digit or 2-digit version numbers.
It could be improved to work with 2-digit too.
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] ABI version of experimental libraries
2020-02-17 23:15 9% [dpdk-dev] ABI version of experimental libraries Thomas Monjalon
2020-02-17 23:44 11% ` [dpdk-dev] [PATCH] build: fix soname for " Thomas Monjalon
@ 2020-02-18 9:42 7% ` Bruce Richardson
2020-02-18 9:50 4% ` Thomas Monjalon
1 sibling, 1 reply; 200+ results
From: Bruce Richardson @ 2020-02-18 9:42 UTC (permalink / raw)
To: Thomas Monjalon
Cc: ray.kinsella, nhorman, bluca, david.marchand, ktraynor, dev
On Tue, Feb 18, 2020 at 12:15:56AM +0100, Thomas Monjalon wrote:
> Hi,
>
> I would like to remind everybody our mistake when defining ABI versions.
> It has been "fixed" in this commit:
> http://git.dpdk.org/dpdk/commit/?id=f26c2b39
>
> Please let's think about the consequence for the experimental libraries.
>
> In DPDK 19.11, we use the ABI version 0.200 with soname 0.20 In DPDK
> 20.02, we use the ABI version 0.2001 with soname 0.201 Numbers are
> increasing, that's fine. When we'll switch to the new major ABI and use
> a normal numbering: In DPDK 20.11, we will use the ABI version 0.210 with
> soname 0.21 Numbers are dropping.
>
> In short, for experimental libs, ABI 20.1 > ABI 21.0
>
> Are we OK with this or do we prefer reverting to normal numbering for
> experimental libraries in DPDK 20.02?
>
Personally, I would not be too concerned about the verions of experimental
libs, so long as they don't conflict across versions and have some
similarity to the major ABI version for the release.
/Bruce
^ permalink raw reply [relevance 7%]
* Re: [dpdk-dev] [PATCH] ci: build and use libabigail 1.6
2020-02-17 18:47 0% ` Aaron Conole
@ 2020-02-18 9:40 3% ` David Marchand
2020-02-18 11:18 0% ` David Marchand
0 siblings, 1 reply; 200+ results
From: David Marchand @ 2020-02-18 9:40 UTC (permalink / raw)
To: Aaron Conole
Cc: Thomas Monjalon, dev, Christian Ehrhardt, Dodji Seketeli,
Michael Santana
On Mon, Feb 17, 2020 at 7:48 PM Aaron Conole <aconole@redhat.com> wrote:
>
> David Marchand <david.marchand@redhat.com> writes:
>
> > libabigail 1.2 (at least) reports changes in 'const' property as an ABI
> > breakage [1].
> > This was fixed upstream in libabigail 1.4 [2], and a bug has been opened
> > in launchpad [3].
> >
> > But for now, build and use the last version 1.6 so that the ABI checks
> > can be kept.
> >
> > 1: https://travis-ci.com/DPDK/dpdk/jobs/287872118#L2242
> > 2: https://sourceware.org/git/gitweb.cgi?p=libabigail.git;a=commitdiff;h=215b7eb4fe8b986fe1cc87d9d8e7412998038392
> > 3: https://bugs.launchpad.net/ubuntu/+source/libabigail/+bug/1863607
> >
> > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > ---
>
> Does it make sense to base libabigail required ontop of extra packages?
> Otherwise some libraries won't get built / checked, no?
The only change I see is the pcap driver being enabled.
On the principle, I agree that trying to build all possible
libraries/drivers is better when checking the ABI.
So I'll keep extra_packages yes.
I am currently testing that touching extra_packages (well, testing
Thomas patches) results in Travis treating the job as a new one (i.e.
with no cache).
>
> > .ci/linux-build.sh | 22 ++++++++++++++++++++++
> > .travis.yml | 15 ++++++++++-----
> > 2 files changed, 32 insertions(+), 5 deletions(-)
> >
> > diff --git a/.ci/linux-build.sh b/.ci/linux-build.sh
> > index c7c3840fc..0d4bc9a62 100755
> > --- a/.ci/linux-build.sh
> > +++ b/.ci/linux-build.sh
> > @@ -38,6 +38,28 @@ if [ "$AARCH64" != "1" ]; then
> > fi
> >
> > if [ "$ABI_CHECKS" = "1" ]; then
> > + LIBABIGAIL_REPO=${LIBABIGAIL_REPO:-https://sourceware.org/git/libabigail.git}
> > + LIBABIGAIL_VERSION=${LIBABIGAIL_VERSION:-libabigail-1.6}
> > +
> > + if [ "$(cat libabigail/VERSION 2>/dev/null)" != "$LIBABIGAIL_VERSION" ]; then
> > + rm -rf libabigail
> > + # if we change libabigail, invalidate existing abi cache
> > + rm -rf reference
> > + fi
> > +
> > + if [ ! -d libabigail ]; then
> > + git clone --single-branch -b $LIBABIGAIL_VERSION $LIBABIGAIL_REPO libabigail/src
> > + cd libabigail/src && autoconf -vfi && cd -
And I managed to send a "oh yeah, seems better this way"/untested version...
Fixed.
--
David Marchand
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH] build: fix soname for experimental libraries
2020-02-17 23:44 11% ` [dpdk-dev] [PATCH] build: fix soname for " Thomas Monjalon
@ 2020-02-18 9:40 0% ` Bruce Richardson
2020-02-18 9:47 0% ` Thomas Monjalon
0 siblings, 1 reply; 200+ results
From: Bruce Richardson @ 2020-02-18 9:40 UTC (permalink / raw)
To: Thomas Monjalon
Cc: dev, ray.kinsella, nhorman, bluca, david.marchand, ktraynor
On Tue, Feb 18, 2020 at 12:44:02AM +0100, Thomas Monjalon wrote:
> Because of an original mistake in ABI numbering,
> and a temporary workaround for ABI 20,
> for experimental libs, numbering would lead to consider
> ABI 20.1 > ABI 21.0
>
> Before this patch:
>
> DPDK 19.11: ABI version 0.200 and soname 0.20
> DPDK 20.02: ABI version 0.2001 and soname 0.201
> Numbers are increasing, that's fine.
> For the next major ABI, back to normal numbering:
> DPDK 20.11: ABI version 0.210 and soname 0.21
> Numbers are decreasing!
>
> After this patch:
>
> DPDK 19.11: ABI version 0.200 and soname 0.20
> DPDK 20.02: ABI version 0.201 and soname 0.20
> DPDK 20.11: ABI version 0.210 and soname 0.21
>
> Fixes: f26c2b39b271 ("build: fix soname info for 19.11 compatibility")
>
> Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
> ---
> config/meson.build | 8 ++++----
> mk/rte.lib.mk | 4 ++--
> 2 files changed, 6 insertions(+), 6 deletions(-)
>
> diff --git a/config/meson.build b/config/meson.build
> index 6c46767e3e..e7cd74e2c2 100644
> --- a/config/meson.build
> +++ b/config/meson.build
> @@ -27,12 +27,12 @@ abi_version = run_command(find_program('cat', 'more'),
> # and the filename suffix as 0.majorminor versions,
> # e.g. v20.1 => librte_stable.so.20.1, librte_experimental.so.0.201
> # sonames => librte_stable.so.20, librte_experimental.so.0.20
> -# e.g. v20.0.1 => librte_stable.so.20.0.1, librte_experimental.so.0.2001
> -# sonames => librte_stable.so.20.0, librte_experimental.so.0.200
> +# e.g. v20.0.1 => librte_stable.so.20.0.1, librte_experimental.so.0.201
> +# sonames => librte_stable.so.20.0, librte_experimental.so.0.20
> abi_va = abi_version.split('.')
> stable_so_version = abi_va.length() == 2 ? abi_va[0] : abi_va[0] + '.' + abi_va[1]
> -experimental_abi_version = '0.' + ''.join(abi_va)
> -experimental_so_version = '0.' + ''.join(stable_so_version.split('.'))
> +experimental_abi_version = '0.' + ''.join([abi_va[0], abi_va[2]])
> +experimental_so_version = '0.' + ''.join([abi_va[0]])
>
My concern about this is that it will break, or rather need to be changed
again for the 20.11 release. While I see the numbering as not-ideal in
terms of version numbers, the existing scheme was originally designed to
work with either 3-digit or 2-digit version numbers.
/Bruce
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] doc: plan splitting the ethdev ops struct
2020-02-17 15:38 3% [dpdk-dev] [PATCH] doc: plan splitting the ethdev ops struct Ferruh Yigit
2020-02-18 5:07 0% ` Jerin Jacob
@ 2020-02-18 6:01 0% ` Stephen Hemminger
2020-02-21 10:40 0% ` Ferruh Yigit
2020-02-25 12:44 3% ` [dpdk-dev] [PATCH v2] " Ferruh Yigit
2 siblings, 1 reply; 200+ results
From: Stephen Hemminger @ 2020-02-18 6:01 UTC (permalink / raw)
To: Ferruh Yigit
Cc: Neil Horman, John McNamara, Marko Kovacevic, dev, David Marchand,
Thomas Monjalon, Andrew Rybchenko
On Mon, 17 Feb 2020 15:38:05 +0000
Ferruh Yigit <ferruh.yigit@intel.com> wrote:
> For the ABI compatibility it is better to hide internal data structures
> from the application as much as possible. But because of some inline
> functions 'struct eth_dev_ops' can't be hidden completely.
>
> Plan is to split the 'struct eth_dev_ops' into two as ones used by
> inline functions and ones not used, and hide the second part that not
> used by inline functions completely to the application.
>
> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
> ---
> Cc: David Marchand <david.marchand@redhat.com>
> Cc: Thomas Monjalon <thomas@monjalon.net>
> Cc: Andrew Rybchenko <arybchenko@solarflare.com>
> ---
> doc/guides/rel_notes/deprecation.rst | 6 ++++++
> 1 file changed, 6 insertions(+)
>
> diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
> index dfcca87ab..2aa431028 100644
> --- a/doc/guides/rel_notes/deprecation.rst
> +++ b/doc/guides/rel_notes/deprecation.rst
> @@ -72,6 +72,12 @@ Deprecation Notices
> In 19.11 PMDs will still update the field even when the offload is not
> enabled.
>
> +* ethdev: Split the ``struct eth_dev_ops`` struct to hide it as much as possible.
> + Currently the ``struct eth_dev_ops`` struct is accessible by the application
> + because some inline functions, like ``rte_eth_tx_descriptor_status()``,
> + access the struct directly. The struct will be separate in two, the ops used
> + by inline functions still will be accessible to user but rest will be hidden.
> +
> * cryptodev: support for using IV with all sizes is added, J0 still can
> be used but only when IV length in following structs ``rte_crypto_auth_xform``,
> ``rte_crypto_aead_xform`` is set to zero. When IV length is greater or equal
Good luck, truely hiding internals is hard. The mbuf structure is already split but not really
hidden at all (just look at dwarf output). It doesn't make sense to do it unless
you can really hide it.
I would attack the rte_device stuff first. Make rte_device opaque to the application
that would help for future versions. Then work backwards to rte_tehtdev.
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] doc: plan splitting the ethdev ops struct
2020-02-17 15:38 3% [dpdk-dev] [PATCH] doc: plan splitting the ethdev ops struct Ferruh Yigit
@ 2020-02-18 5:07 0% ` Jerin Jacob
2020-02-25 12:42 0% ` Ferruh Yigit
2020-02-18 6:01 0% ` Stephen Hemminger
2020-02-25 12:44 3% ` [dpdk-dev] [PATCH v2] " Ferruh Yigit
2 siblings, 1 reply; 200+ results
From: Jerin Jacob @ 2020-02-18 5:07 UTC (permalink / raw)
To: Ferruh Yigit
Cc: Neil Horman, John McNamara, Marko Kovacevic, dpdk-dev,
David Marchand, Thomas Monjalon, Andrew Rybchenko
On Mon, Feb 17, 2020 at 9:08 PM Ferruh Yigit <ferruh.yigit@intel.com> wrote:
>
> For the ABI compatibility it is better to hide internal data structures
> from the application as much as possible. But because of some inline
> functions 'struct eth_dev_ops' can't be hidden completely.
>
> Plan is to split the 'struct eth_dev_ops' into two as ones used by
> inline functions and ones not used, and hide the second part that not
> used by inline functions completely to the application.
It is a good improvement. IMO, If anything used in fast-path it
should be in ``struct rte_eth_dev``
and rest can completely be moved to internal. In this case, if
`rte_eth_tx_descriptor_status`
not used on fastpath, Maybe we don't need to maintain the inline
status and move completely
to .c file.
Those may be specifics of the work. In general, this change looks good to me.
Acked-by: Jerin Jacob <jerinj@marvell.com>
>
> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
> ---
> Cc: David Marchand <david.marchand@redhat.com>
> Cc: Thomas Monjalon <thomas@monjalon.net>
> Cc: Andrew Rybchenko <arybchenko@solarflare.com>
> ---
> doc/guides/rel_notes/deprecation.rst | 6 ++++++
> 1 file changed, 6 insertions(+)
>
> diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
> index dfcca87ab..2aa431028 100644
> --- a/doc/guides/rel_notes/deprecation.rst
> +++ b/doc/guides/rel_notes/deprecation.rst
> @@ -72,6 +72,12 @@ Deprecation Notices
> In 19.11 PMDs will still update the field even when the offload is not
> enabled.
>
> +* ethdev: Split the ``struct eth_dev_ops`` struct to hide it as much as possible.
> + Currently the ``struct eth_dev_ops`` struct is accessible by the application
> + because some inline functions, like ``rte_eth_tx_descriptor_status()``,
> + access the struct directly. The struct will be separate in two, the ops used
> + by inline functions still will be accessible to user but rest will be hidden.
> +
> * cryptodev: support for using IV with all sizes is added, J0 still can
> be used but only when IV length in following structs ``rte_crypto_auth_xform``,
> ``rte_crypto_aead_xform`` is set to zero. When IV length is greater or equal
> --
> 2.24.1
>
^ permalink raw reply [relevance 0%]
* [dpdk-dev] [PATCH] build: fix soname for experimental libraries
2020-02-17 23:15 9% [dpdk-dev] ABI version of experimental libraries Thomas Monjalon
@ 2020-02-17 23:44 11% ` Thomas Monjalon
2020-02-18 9:40 0% ` Bruce Richardson
2020-02-18 9:42 7% ` [dpdk-dev] ABI version of " Bruce Richardson
1 sibling, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-02-17 23:44 UTC (permalink / raw)
To: dev
Cc: ray.kinsella, nhorman, bluca, david.marchand, ktraynor, Bruce Richardson
Because of an original mistake in ABI numbering,
and a temporary workaround for ABI 20,
for experimental libs, numbering would lead to consider
ABI 20.1 > ABI 21.0
Before this patch:
DPDK 19.11: ABI version 0.200 and soname 0.20
DPDK 20.02: ABI version 0.2001 and soname 0.201
Numbers are increasing, that's fine.
For the next major ABI, back to normal numbering:
DPDK 20.11: ABI version 0.210 and soname 0.21
Numbers are decreasing!
After this patch:
DPDK 19.11: ABI version 0.200 and soname 0.20
DPDK 20.02: ABI version 0.201 and soname 0.20
DPDK 20.11: ABI version 0.210 and soname 0.21
Fixes: f26c2b39b271 ("build: fix soname info for 19.11 compatibility")
Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
---
config/meson.build | 8 ++++----
mk/rte.lib.mk | 4 ++--
2 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/config/meson.build b/config/meson.build
index 6c46767e3e..e7cd74e2c2 100644
--- a/config/meson.build
+++ b/config/meson.build
@@ -27,12 +27,12 @@ abi_version = run_command(find_program('cat', 'more'),
# and the filename suffix as 0.majorminor versions,
# e.g. v20.1 => librte_stable.so.20.1, librte_experimental.so.0.201
# sonames => librte_stable.so.20, librte_experimental.so.0.20
-# e.g. v20.0.1 => librte_stable.so.20.0.1, librte_experimental.so.0.2001
-# sonames => librte_stable.so.20.0, librte_experimental.so.0.200
+# e.g. v20.0.1 => librte_stable.so.20.0.1, librte_experimental.so.0.201
+# sonames => librte_stable.so.20.0, librte_experimental.so.0.20
abi_va = abi_version.split('.')
stable_so_version = abi_va.length() == 2 ? abi_va[0] : abi_va[0] + '.' + abi_va[1]
-experimental_abi_version = '0.' + ''.join(abi_va)
-experimental_so_version = '0.' + ''.join(stable_so_version.split('.'))
+experimental_abi_version = '0.' + ''.join([abi_va[0], abi_va[2]])
+experimental_so_version = '0.' + ''.join([abi_va[0]])
# extract all version information into the build configuration
dpdk_conf.set('RTE_VER_YEAR', pver.get(0).to_int())
diff --git a/mk/rte.lib.mk b/mk/rte.lib.mk
index b1a8372cc2..8730c084ce 100644
--- a/mk/rte.lib.mk
+++ b/mk/rte.lib.mk
@@ -15,8 +15,8 @@ LIBABIVER ?= $(shell cat $(RTE_SRCDIR)/ABI_VERSION)
SOVER := $(basename $(LIBABIVER))
ifeq ($(shell grep -s "^DPDK_" $(SRCDIR)/$(EXPORT_MAP)),)
# EXPERIMENTAL ABI is versioned as 0.major+minor, e.g. 0.201 for 20.1 ABI
-LIBABIVER := 0.$(shell echo $(LIBABIVER) | tr -d '.')
-SOVER := 0.$(shell echo $(SOVER) | tr -d '.')
+LIBABIVER := 0.$(shell echo $(LIBABIVER) | cut -d '.' -f1,3 | tr -d '.')
+SOVER := 0.$(shell echo $(SOVER) | cut -d '.' -f1)
endif
ifeq ($(CONFIG_RTE_BUILD_SHARED_LIB),y)
--
2.25.0
^ permalink raw reply [relevance 11%]
* [dpdk-dev] ABI version of experimental libraries
@ 2020-02-17 23:15 9% Thomas Monjalon
2020-02-17 23:44 11% ` [dpdk-dev] [PATCH] build: fix soname for " Thomas Monjalon
2020-02-18 9:42 7% ` [dpdk-dev] ABI version of " Bruce Richardson
0 siblings, 2 replies; 200+ results
From: Thomas Monjalon @ 2020-02-17 23:15 UTC (permalink / raw)
To: ray.kinsella, nhorman
Cc: bruce.richardson, bluca, david.marchand, ktraynor, dev
Hi,
I would like to remind everybody our mistake when defining ABI versions.
It has been "fixed" in this commit:
http://git.dpdk.org/dpdk/commit/?id=f26c2b39
Please let's think about the consequence for the experimental libraries.
In DPDK 19.11, we use the ABI version 0.200 with soname 0.20
In DPDK 20.02, we use the ABI version 0.2001 with soname 0.201
Numbers are increasing, that's fine.
When we'll switch to the new major ABI and use a normal numbering:
In DPDK 20.11, we will use the ABI version 0.210 with soname 0.21
Numbers are dropping.
In short, for experimental libs, ABI 20.1 > ABI 21.0
Are we OK with this or do we prefer reverting to normal numbering
for experimental libraries in DPDK 20.02?
^ permalink raw reply [relevance 9%]
* Re: [dpdk-dev] [PATCH] ci: build and use libabigail 1.6
2020-02-17 13:59 9% [dpdk-dev] [PATCH] ci: build and use libabigail 1.6 David Marchand
2020-02-17 15:15 0% ` Thomas Monjalon
@ 2020-02-17 18:47 0% ` Aaron Conole
2020-02-18 9:40 3% ` David Marchand
2020-02-18 14:29 9% ` [dpdk-dev] [PATCH v2] " David Marchand
2020-02-18 20:13 9% ` [dpdk-dev] [PATCH v3] " David Marchand
3 siblings, 1 reply; 200+ results
From: Aaron Conole @ 2020-02-17 18:47 UTC (permalink / raw)
To: David Marchand; +Cc: thomas, dev, christian.ehrhardt, dodji, Michael Santana
David Marchand <david.marchand@redhat.com> writes:
> libabigail 1.2 (at least) reports changes in 'const' property as an ABI
> breakage [1].
> This was fixed upstream in libabigail 1.4 [2], and a bug has been opened
> in launchpad [3].
>
> But for now, build and use the last version 1.6 so that the ABI checks
> can be kept.
>
> 1: https://travis-ci.com/DPDK/dpdk/jobs/287872118#L2242
> 2: https://sourceware.org/git/gitweb.cgi?p=libabigail.git;a=commitdiff;h=215b7eb4fe8b986fe1cc87d9d8e7412998038392
> 3: https://bugs.launchpad.net/ubuntu/+source/libabigail/+bug/1863607
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
Does it make sense to base libabigail required ontop of extra packages?
Otherwise some libraries won't get built / checked, no?
> .ci/linux-build.sh | 22 ++++++++++++++++++++++
> .travis.yml | 15 ++++++++++-----
> 2 files changed, 32 insertions(+), 5 deletions(-)
>
> diff --git a/.ci/linux-build.sh b/.ci/linux-build.sh
> index c7c3840fc..0d4bc9a62 100755
> --- a/.ci/linux-build.sh
> +++ b/.ci/linux-build.sh
> @@ -38,6 +38,28 @@ if [ "$AARCH64" != "1" ]; then
> fi
>
> if [ "$ABI_CHECKS" = "1" ]; then
> + LIBABIGAIL_REPO=${LIBABIGAIL_REPO:-https://sourceware.org/git/libabigail.git}
> + LIBABIGAIL_VERSION=${LIBABIGAIL_VERSION:-libabigail-1.6}
> +
> + if [ "$(cat libabigail/VERSION 2>/dev/null)" != "$LIBABIGAIL_VERSION" ]; then
> + rm -rf libabigail
> + # if we change libabigail, invalidate existing abi cache
> + rm -rf reference
> + fi
> +
> + if [ ! -d libabigail ]; then
> + git clone --single-branch -b $LIBABIGAIL_VERSION $LIBABIGAIL_REPO libabigail/src
> + cd libabigail/src && autoconf -vfi && cd -
> + mkdir libabigail/src/build
> + cd libabigail/src/build && ../configure --prefix=$(pwd)/libabigail && cd -
> + make -C libabigail/src/build all install
> +
> + rm -rf libabigail/src
> + echo $LIBABIGAIL_VERSION > libabigail/VERSION
> + fi
> +
> + export PATH=$(pwd)/libabigail/bin:$PATH
> +
> REF_GIT_REPO=${REF_GIT_REPO:-https://dpdk.org/git/dpdk}
> REF_GIT_TAG=${REF_GIT_TAG:-v19.11}
>
> diff --git a/.travis.yml b/.travis.yml
> index 22539d823..d8253fdd4 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -2,6 +2,7 @@ language: c
> cache:
> ccache: true
> directories:
> + - libabigail
> - reference
> compiler:
> - gcc
> @@ -24,7 +25,11 @@ aarch64_packages: &aarch64_packages
>
> extra_packages: &extra_packages
> - *required_packages
> - - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4, abigail-tools]
> + - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
> +
> +libabigail_build_packages: &libabigail_build_packages
> + - *required_packages
> + - [autoconf, automake, libtool, pkg-config, libxml2-dev, libdw-dev]
>
> build_32b_packages: &build_32b_packages
> - *required_packages
> @@ -154,18 +159,18 @@ matrix:
> packages:
> - *required_packages
> - *doc_packages
> - - env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1
> + - env: DEF_LIB="shared" ABI_CHECKS=1
> compiler: gcc
> addons:
> apt:
> packages:
> - - *extra_packages
> - - env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1
> + - *libabigail_build_packages
> + - env: DEF_LIB="shared" ABI_CHECKS=1
> arch: arm64
> compiler: gcc
> addons:
> apt:
> packages:
> - - *extra_packages
> + - *libabigail_build_packages
>
> script: ./.ci/${TRAVIS_OS_NAME}-build.sh
^ permalink raw reply [relevance 0%]
* [dpdk-dev] [PATCH] net/ionic: fix ABI version
@ 2020-02-17 16:13 7% Ferruh Yigit
2020-02-19 12:37 4% ` Ferruh Yigit
0 siblings, 1 reply; 200+ results
From: Ferruh Yigit @ 2020-02-17 16:13 UTC (permalink / raw)
To: Alfredo Cardigliano, Shannon Nelson; +Cc: dev, Ferruh Yigit, David Marchand
Changed the ABI version to 20.0.1.
Fixes: a72cf98cc93e ("net/ionic: add skeleton")
Reported-by: David Marchand <david.marchand@redhat.com>
Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
---
Cc: Alfredo Cardigliano <cardigliano@ntop.org>
---
drivers/net/ionic/rte_pmd_ionic_version.map | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/drivers/net/ionic/rte_pmd_ionic_version.map b/drivers/net/ionic/rte_pmd_ionic_version.map
index 3fb4ff7eb..bc8fd6d4d 100644
--- a/drivers/net/ionic/rte_pmd_ionic_version.map
+++ b/drivers/net/ionic/rte_pmd_ionic_version.map
@@ -1,5 +1,4 @@
-DPDK_20.0 {
+DPDK_20.0.1 {
local: *;
};
-
--
2.24.1
^ permalink raw reply [relevance 7%]
* [dpdk-dev] [PATCH] doc: plan splitting the ethdev ops struct
@ 2020-02-17 15:38 3% Ferruh Yigit
2020-02-18 5:07 0% ` Jerin Jacob
` (2 more replies)
0 siblings, 3 replies; 200+ results
From: Ferruh Yigit @ 2020-02-17 15:38 UTC (permalink / raw)
To: Neil Horman, John McNamara, Marko Kovacevic
Cc: dev, Ferruh Yigit, David Marchand, Thomas Monjalon, Andrew Rybchenko
For the ABI compatibility it is better to hide internal data structures
from the application as much as possible. But because of some inline
functions 'struct eth_dev_ops' can't be hidden completely.
Plan is to split the 'struct eth_dev_ops' into two as ones used by
inline functions and ones not used, and hide the second part that not
used by inline functions completely to the application.
Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
---
Cc: David Marchand <david.marchand@redhat.com>
Cc: Thomas Monjalon <thomas@monjalon.net>
Cc: Andrew Rybchenko <arybchenko@solarflare.com>
---
doc/guides/rel_notes/deprecation.rst | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
index dfcca87ab..2aa431028 100644
--- a/doc/guides/rel_notes/deprecation.rst
+++ b/doc/guides/rel_notes/deprecation.rst
@@ -72,6 +72,12 @@ Deprecation Notices
In 19.11 PMDs will still update the field even when the offload is not
enabled.
+* ethdev: Split the ``struct eth_dev_ops`` struct to hide it as much as possible.
+ Currently the ``struct eth_dev_ops`` struct is accessible by the application
+ because some inline functions, like ``rte_eth_tx_descriptor_status()``,
+ access the struct directly. The struct will be separate in two, the ops used
+ by inline functions still will be accessible to user but rest will be hidden.
+
* cryptodev: support for using IV with all sizes is added, J0 still can
be used but only when IV length in following structs ``rte_crypto_auth_xform``,
``rte_crypto_aead_xform`` is set to zero. When IV length is greater or equal
--
2.24.1
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [RFC] meter: fix ABI break due to experimental tag removal
2020-02-17 14:23 8% ` Ray Kinsella
@ 2020-02-17 15:37 4% ` Neil Horman
0 siblings, 0 replies; 200+ results
From: Neil Horman @ 2020-02-17 15:37 UTC (permalink / raw)
To: Ray Kinsella
Cc: Bruce Richardson, Luca Boccassi, Ferruh Yigit,
Cristian Dumitrescu, Eelco Chaudron, dev, Thomas Monjalon,
David Marchand, Ian Stokes
On Mon, Feb 17, 2020 at 02:23:15PM +0000, Ray Kinsella wrote:
>
>
> On 14/02/2020 20:48, Neil Horman wrote:
> > On Fri, Feb 14, 2020 at 11:36:34AM +0000, Bruce Richardson wrote:
> >> On Thu, Feb 13, 2020 at 09:40:40PM -0500, Neil Horman wrote:
> >>> On Thu, Feb 13, 2020 at 05:40:43PM +0000, Ray Kinsella wrote:
> >>>>
> >>>>
> >>>> On 05/02/2020 11:32, Neil Horman wrote:
> >>>>> On Wed, Feb 05, 2020 at 11:04:29AM +0100, Luca Boccassi wrote:
> >>>>>> On Tue, 2020-02-04 at 07:02 -0500, Neil Horman wrote:
> >>>>>>>> But if we can do the versioning in the master, LTS can backport it
> >>>>>>>> and can have mature version of that API in LTS without breaking
> >>>>>>>> the existing users.
> >>>>>>>>
> >>>>>>>
> >>>>>>> But why bother? The only thing you've changed is the version
> >>>>>>> tagging. Its ok to leave that alone in LTS, you just cant change
> >>>>>>> it.
> >>>>>>>
> >>>>>>> Thats part of the burden of an LTS release, it will have some drift
> >>>>>>> from the upstream head, because you have to keep things stable. So
> >>>>>>> you stabilize the upstream ABI version for this API and just never
> >>>>>>> backport it to the current LTS release.
> >>>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>> A customer (OVS) explicitly and specifically requested backporting
> >>>>>> the symbol change to 19.11, as they don't want to enable
> >>>>>> experimental APIs in their releases. I replied that it would only be
> >>>>>> acceptable with aliasing to keep compatibility, and Ferruh very
> >>>>>> kindly did the work to implement that.
> >>>>>>
> >>>>> but, thats part of the agreement, no? You can't always have new
> >>>>> features and stability at the same time.
> >>>>>
> >>>>> I get that this is an odd corner case, because strictly speaking you
> >>>>> could waive the ABI change that libabigail is reporting, and most
> >>>>> application users (like OVS) could get away with it, because their
> >>>>> application does all the right things to make it ok, but I don't
> >>>>> think you can make a decsion like this for all users based on the
> >>>>> request of a single user.
> >>>>>
> >>>>> It seems like the right thing is for OVS to augment their build time
> >>>>> configuration to allow developers to select the ability to use
> >>>>> experimental apis at compile time, and then the decision is placed in
> >>>>> the hands of end users.
> >>>>
> >>>> So this is not isolated case right ... it is common from API's to
> >>>> graduate from experimental to mature, we _want_ that to happen, we
> >>>> _want_ APIs to mature.
> >>>>
> >>> Sure, I can absolutely agree with that, though I would suggest that the
> >>> maturity of the ABI is orthogonal to its labeling as such (more on that
> >>> below)
> >>>
> >>>> I am worried what you are proposing will encourage a behavior whereby
> >>>> maintainers to wait until the declaration of the next major ABI version
> >>>> to make these symbol changes, causing these kinds of changes to queue
> >>>> up for that release.
> >>>>
> >>> I think you're probably right about that, and would make the agrument
> >>> that thats perfectly fine (again I'll clarify below)
> >>>
> >>>> And you would have to ask why? In the case of a reasonably mature API,
> >>>> there will be no functional or signature change - its mature! So what
> >>>> is the harm of providing an alias back to Experimental until the next
> >>>> ABI version is declared?
> >>>>
> >>> From a philosophical standpoint, there is absoluely no harm, and I don't
> >>> think anyone would disagree, the harm comes from the details of the
> >>> implementation, as you've noted.
> >>>
> >>>> So while yes, you are 100% right - experimental mean no guarantees.
> >>>> But if the API is baked, it is not going to change, so I don't see the
> >>>> harm.
> >>>>
> >>>> We don't want the unnecessary triumph of policy over pragmatism.
> >>>>
> >>> I would make the converse argument here. While I agree that when an API
> >>> is mature, theres no point in calling it experimental anymore, I would
> >>> also suggest that, if an API is mature, and not expected to change,
> >>> theres no harm in leaving its version tag as experimental for an LTS
> >>> release. This is what I was referring to above. Once an application
> >>> developer has done the work to integrate an API from DPDK into its
> >>> application, that work is done. If the API remains stable, then I
> >>> honestly don't know that they care that the version label is EXPERIMENTAL
> >>> versus 20.11 or 20.05 or whatever. They care about the API being stable
> >>> more so than its version label. As such, it seems....reasonable to me to
> >>> have developers queue their migration of experimental APIs to official
> >>> versioned APIs at the next major release deliniation point.
> >>>
> >>> I'd welcome counter arguments, but it seems pretty natural to me to make
> >>> these sorts of changes at the major release mark. People using
> >>> experimantal apis have already implicity agreed to manage changes to
> >>> them, so if we just hold it stable in an LTS release (and don't update
> >>> the version label), its just gravy for them.
> >>>
> >> The counter argument that I see is that while the experimental tag remains
> >> in place the user has no way to know that an API is stable or not, and so
> >> in many projects cannot use the API. If for example an API that is
> >> experimental in 19.11 is unchanged through 20.05 at which point we decide
> >> to promote it to stable. Once the change to the exp tag it is made, any
> >> users of 19.11 can then see that it is an unchanged and now-stable ABI and
> >> can start using it in their software, if they wish, without having to wait
> >> for the 20.11 release. Changing the tag early allows ABIs to be potentially
> >> used immediately.
> >>
> >
> > I would agree with this, however, when using an LTS release, in my mind at
> > least, part of the agreement there is you get stability in the fuctions that
> > were declared stable at the time of release. I'm not sure there should be an
> > expectation of additional stabilization within the lifetime of the release
> > (thats really what the 12 month LTS release cycle is for, no)? You never have
> > to wait more than a year for a new set of stable features. If you want faster
> > feature integration than that, you can choose to enable the experimental API's
> > and accept the benefits and drawbacks thereof.
> >
> > That said, if (as I understand it) the goal is to provide a mechanism to allow
> > experimental features to be promoted to stable status, perhaps we can find a
> > happy middle ground. What if we were to create a construct such as this:
> >
> > #pragma push_macro("ALLOW_EXPERIMENTAL_APIS")
> > #undef ALLOW_EXPERIMENTAL_APIS
> > void __rte_experimental func(...);
> > #pragma pop_macro("ALLOW_EXPERIMENTAL_APIS")
> >
> > Such a consruct would allow the maintainer of an API to pseudo-promote an API
> > from a experimental to stable status, at least so far as compilation would be
> > concerned. Its messy and clunky, and it wouldn't change the function version at
> > all, but the end result would be that:
> > a) such a wraped experimental function would no longer issue a warning when used
> > during compilation/linking
> > and
> > b) provide a semi-easy grepable pattern for application writers to look for when
> > considering the use of an API that was previously experimental
> >
> > such a construct would have to be used very judiciously, in that once its
> > implemented, the API has to be treated as stable, even though its 'excused' from
> > the normal checking, but it could provide something of the more rapid promotion
> > path being sought.
> >
> > Thoughts?
> > Neil
> >> Regards,
> >> /Bruce
> >>
>
>
> Then method above seems clunky, and it is unclear to me if it solves the original problem.
> It seems simpler to me to promote a symbol the next ABI version, and then provide an alias.
>
> At the declaration of the next major ABI version (v21), we would then only need to check
> for those symbols that are both EXPERIMENTAL and v21, to know which aliases need to be removed.
>
> Thanks,
>
> Ray K
>
i don't disagree, just looking for other potential options.
Neil
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH] ci: build and use libabigail 1.6
2020-02-17 13:59 9% [dpdk-dev] [PATCH] ci: build and use libabigail 1.6 David Marchand
@ 2020-02-17 15:15 0% ` Thomas Monjalon
2020-02-17 18:47 0% ` Aaron Conole
` (2 subsequent siblings)
3 siblings, 0 replies; 200+ results
From: Thomas Monjalon @ 2020-02-17 15:15 UTC (permalink / raw)
To: David Marchand, christian.ehrhardt
Cc: dev, dodji, Aaron Conole, Michael Santana, kevin.laatz
17/02/2020 14:59, David Marchand:
> libabigail 1.2 (at least) reports changes in 'const' property as an ABI
> breakage [1].
> This was fixed upstream in libabigail 1.4 [2], and a bug has been opened
> in launchpad [3].
>
> But for now, build and use the last version 1.6 so that the ABI checks
> can be kept.
>
> 1: https://travis-ci.com/DPDK/dpdk/jobs/287872118#L2242
> 2: https://sourceware.org/git/gitweb.cgi?p=libabigail.git;a=commitdiff;h=215b7eb4fe8b986fe1cc87d9d8e7412998038392
> 3: https://bugs.launchpad.net/ubuntu/+source/libabigail/+bug/1863607
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
> + LIBABIGAIL_REPO=${LIBABIGAIL_REPO:-https://sourceware.org/git/libabigail.git}
> + LIBABIGAIL_VERSION=${LIBABIGAIL_VERSION:-libabigail-1.6}
> +
> + if [ "$(cat libabigail/VERSION 2>/dev/null)" != "$LIBABIGAIL_VERSION" ]; then
> + rm -rf libabigail
> + # if we change libabigail, invalidate existing abi cache
> + rm -rf reference
> + fi
> +
> + if [ ! -d libabigail ]; then
> + git clone --single-branch -b $LIBABIGAIL_VERSION $LIBABIGAIL_REPO libabigail/src
> + cd libabigail/src && autoconf -vfi && cd -
> + mkdir libabigail/src/build
> + cd libabigail/src/build && ../configure --prefix=$(pwd)/libabigail && cd -
> + make -C libabigail/src/build all install
> +
> + rm -rf libabigail/src
> + echo $LIBABIGAIL_VERSION > libabigail/VERSION
> + fi
Can we avoid compiling libabigail ourself?
Is there an up-to-date Ubuntu package somewhere?
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [RFC] meter: fix ABI break due to experimental tag removal
2020-02-14 20:48 4% ` Neil Horman
2020-02-14 21:52 4% ` Bruce Richardson
@ 2020-02-17 14:23 8% ` Ray Kinsella
2020-02-17 15:37 4% ` Neil Horman
1 sibling, 1 reply; 200+ results
From: Ray Kinsella @ 2020-02-17 14:23 UTC (permalink / raw)
To: Neil Horman, Bruce Richardson
Cc: Luca Boccassi, Ferruh Yigit, Cristian Dumitrescu, Eelco Chaudron,
dev, Thomas Monjalon, David Marchand, Ian Stokes
On 14/02/2020 20:48, Neil Horman wrote:
> On Fri, Feb 14, 2020 at 11:36:34AM +0000, Bruce Richardson wrote:
>> On Thu, Feb 13, 2020 at 09:40:40PM -0500, Neil Horman wrote:
>>> On Thu, Feb 13, 2020 at 05:40:43PM +0000, Ray Kinsella wrote:
>>>>
>>>>
>>>> On 05/02/2020 11:32, Neil Horman wrote:
>>>>> On Wed, Feb 05, 2020 at 11:04:29AM +0100, Luca Boccassi wrote:
>>>>>> On Tue, 2020-02-04 at 07:02 -0500, Neil Horman wrote:
>>>>>>>> But if we can do the versioning in the master, LTS can backport it
>>>>>>>> and can have mature version of that API in LTS without breaking
>>>>>>>> the existing users.
>>>>>>>>
>>>>>>>
>>>>>>> But why bother? The only thing you've changed is the version
>>>>>>> tagging. Its ok to leave that alone in LTS, you just cant change
>>>>>>> it.
>>>>>>>
>>>>>>> Thats part of the burden of an LTS release, it will have some drift
>>>>>>> from the upstream head, because you have to keep things stable. So
>>>>>>> you stabilize the upstream ABI version for this API and just never
>>>>>>> backport it to the current LTS release.
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> A customer (OVS) explicitly and specifically requested backporting
>>>>>> the symbol change to 19.11, as they don't want to enable
>>>>>> experimental APIs in their releases. I replied that it would only be
>>>>>> acceptable with aliasing to keep compatibility, and Ferruh very
>>>>>> kindly did the work to implement that.
>>>>>>
>>>>> but, thats part of the agreement, no? You can't always have new
>>>>> features and stability at the same time.
>>>>>
>>>>> I get that this is an odd corner case, because strictly speaking you
>>>>> could waive the ABI change that libabigail is reporting, and most
>>>>> application users (like OVS) could get away with it, because their
>>>>> application does all the right things to make it ok, but I don't
>>>>> think you can make a decsion like this for all users based on the
>>>>> request of a single user.
>>>>>
>>>>> It seems like the right thing is for OVS to augment their build time
>>>>> configuration to allow developers to select the ability to use
>>>>> experimental apis at compile time, and then the decision is placed in
>>>>> the hands of end users.
>>>>
>>>> So this is not isolated case right ... it is common from API's to
>>>> graduate from experimental to mature, we _want_ that to happen, we
>>>> _want_ APIs to mature.
>>>>
>>> Sure, I can absolutely agree with that, though I would suggest that the
>>> maturity of the ABI is orthogonal to its labeling as such (more on that
>>> below)
>>>
>>>> I am worried what you are proposing will encourage a behavior whereby
>>>> maintainers to wait until the declaration of the next major ABI version
>>>> to make these symbol changes, causing these kinds of changes to queue
>>>> up for that release.
>>>>
>>> I think you're probably right about that, and would make the agrument
>>> that thats perfectly fine (again I'll clarify below)
>>>
>>>> And you would have to ask why? In the case of a reasonably mature API,
>>>> there will be no functional or signature change - its mature! So what
>>>> is the harm of providing an alias back to Experimental until the next
>>>> ABI version is declared?
>>>>
>>> From a philosophical standpoint, there is absoluely no harm, and I don't
>>> think anyone would disagree, the harm comes from the details of the
>>> implementation, as you've noted.
>>>
>>>> So while yes, you are 100% right - experimental mean no guarantees.
>>>> But if the API is baked, it is not going to change, so I don't see the
>>>> harm.
>>>>
>>>> We don't want the unnecessary triumph of policy over pragmatism.
>>>>
>>> I would make the converse argument here. While I agree that when an API
>>> is mature, theres no point in calling it experimental anymore, I would
>>> also suggest that, if an API is mature, and not expected to change,
>>> theres no harm in leaving its version tag as experimental for an LTS
>>> release. This is what I was referring to above. Once an application
>>> developer has done the work to integrate an API from DPDK into its
>>> application, that work is done. If the API remains stable, then I
>>> honestly don't know that they care that the version label is EXPERIMENTAL
>>> versus 20.11 or 20.05 or whatever. They care about the API being stable
>>> more so than its version label. As such, it seems....reasonable to me to
>>> have developers queue their migration of experimental APIs to official
>>> versioned APIs at the next major release deliniation point.
>>>
>>> I'd welcome counter arguments, but it seems pretty natural to me to make
>>> these sorts of changes at the major release mark. People using
>>> experimantal apis have already implicity agreed to manage changes to
>>> them, so if we just hold it stable in an LTS release (and don't update
>>> the version label), its just gravy for them.
>>>
>> The counter argument that I see is that while the experimental tag remains
>> in place the user has no way to know that an API is stable or not, and so
>> in many projects cannot use the API. If for example an API that is
>> experimental in 19.11 is unchanged through 20.05 at which point we decide
>> to promote it to stable. Once the change to the exp tag it is made, any
>> users of 19.11 can then see that it is an unchanged and now-stable ABI and
>> can start using it in their software, if they wish, without having to wait
>> for the 20.11 release. Changing the tag early allows ABIs to be potentially
>> used immediately.
>>
>
> I would agree with this, however, when using an LTS release, in my mind at
> least, part of the agreement there is you get stability in the fuctions that
> were declared stable at the time of release. I'm not sure there should be an
> expectation of additional stabilization within the lifetime of the release
> (thats really what the 12 month LTS release cycle is for, no)? You never have
> to wait more than a year for a new set of stable features. If you want faster
> feature integration than that, you can choose to enable the experimental API's
> and accept the benefits and drawbacks thereof.
>
> That said, if (as I understand it) the goal is to provide a mechanism to allow
> experimental features to be promoted to stable status, perhaps we can find a
> happy middle ground. What if we were to create a construct such as this:
>
> #pragma push_macro("ALLOW_EXPERIMENTAL_APIS")
> #undef ALLOW_EXPERIMENTAL_APIS
> void __rte_experimental func(...);
> #pragma pop_macro("ALLOW_EXPERIMENTAL_APIS")
>
> Such a consruct would allow the maintainer of an API to pseudo-promote an API
> from a experimental to stable status, at least so far as compilation would be
> concerned. Its messy and clunky, and it wouldn't change the function version at
> all, but the end result would be that:
> a) such a wraped experimental function would no longer issue a warning when used
> during compilation/linking
> and
> b) provide a semi-easy grepable pattern for application writers to look for when
> considering the use of an API that was previously experimental
>
> such a construct would have to be used very judiciously, in that once its
> implemented, the API has to be treated as stable, even though its 'excused' from
> the normal checking, but it could provide something of the more rapid promotion
> path being sought.
>
> Thoughts?
> Neil
>> Regards,
>> /Bruce
>>
Then method above seems clunky, and it is unclear to me if it solves the original problem.
It seems simpler to me to promote a symbol the next ABI version, and then provide an alias.
At the declaration of the next major ABI version (v21), we would then only need to check
for those symbols that are both EXPERIMENTAL and v21, to know which aliases need to be removed.
Thanks,
Ray K
^ permalink raw reply [relevance 8%]
* [dpdk-dev] [PATCH] ci: build and use libabigail 1.6
@ 2020-02-17 13:59 9% David Marchand
2020-02-17 15:15 0% ` Thomas Monjalon
` (3 more replies)
0 siblings, 4 replies; 200+ results
From: David Marchand @ 2020-02-17 13:59 UTC (permalink / raw)
To: thomas; +Cc: dev, christian.ehrhardt, dodji, Aaron Conole, Michael Santana
libabigail 1.2 (at least) reports changes in 'const' property as an ABI
breakage [1].
This was fixed upstream in libabigail 1.4 [2], and a bug has been opened
in launchpad [3].
But for now, build and use the last version 1.6 so that the ABI checks
can be kept.
1: https://travis-ci.com/DPDK/dpdk/jobs/287872118#L2242
2: https://sourceware.org/git/gitweb.cgi?p=libabigail.git;a=commitdiff;h=215b7eb4fe8b986fe1cc87d9d8e7412998038392
3: https://bugs.launchpad.net/ubuntu/+source/libabigail/+bug/1863607
Signed-off-by: David Marchand <david.marchand@redhat.com>
---
.ci/linux-build.sh | 22 ++++++++++++++++++++++
.travis.yml | 15 ++++++++++-----
2 files changed, 32 insertions(+), 5 deletions(-)
diff --git a/.ci/linux-build.sh b/.ci/linux-build.sh
index c7c3840fc..0d4bc9a62 100755
--- a/.ci/linux-build.sh
+++ b/.ci/linux-build.sh
@@ -38,6 +38,28 @@ if [ "$AARCH64" != "1" ]; then
fi
if [ "$ABI_CHECKS" = "1" ]; then
+ LIBABIGAIL_REPO=${LIBABIGAIL_REPO:-https://sourceware.org/git/libabigail.git}
+ LIBABIGAIL_VERSION=${LIBABIGAIL_VERSION:-libabigail-1.6}
+
+ if [ "$(cat libabigail/VERSION 2>/dev/null)" != "$LIBABIGAIL_VERSION" ]; then
+ rm -rf libabigail
+ # if we change libabigail, invalidate existing abi cache
+ rm -rf reference
+ fi
+
+ if [ ! -d libabigail ]; then
+ git clone --single-branch -b $LIBABIGAIL_VERSION $LIBABIGAIL_REPO libabigail/src
+ cd libabigail/src && autoconf -vfi && cd -
+ mkdir libabigail/src/build
+ cd libabigail/src/build && ../configure --prefix=$(pwd)/libabigail && cd -
+ make -C libabigail/src/build all install
+
+ rm -rf libabigail/src
+ echo $LIBABIGAIL_VERSION > libabigail/VERSION
+ fi
+
+ export PATH=$(pwd)/libabigail/bin:$PATH
+
REF_GIT_REPO=${REF_GIT_REPO:-https://dpdk.org/git/dpdk}
REF_GIT_TAG=${REF_GIT_TAG:-v19.11}
diff --git a/.travis.yml b/.travis.yml
index 22539d823..d8253fdd4 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -2,6 +2,7 @@ language: c
cache:
ccache: true
directories:
+ - libabigail
- reference
compiler:
- gcc
@@ -24,7 +25,11 @@ aarch64_packages: &aarch64_packages
extra_packages: &extra_packages
- *required_packages
- - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4, abigail-tools]
+ - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
+
+libabigail_build_packages: &libabigail_build_packages
+ - *required_packages
+ - [autoconf, automake, libtool, pkg-config, libxml2-dev, libdw-dev]
build_32b_packages: &build_32b_packages
- *required_packages
@@ -154,18 +159,18 @@ matrix:
packages:
- *required_packages
- *doc_packages
- - env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1
+ - env: DEF_LIB="shared" ABI_CHECKS=1
compiler: gcc
addons:
apt:
packages:
- - *extra_packages
- - env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1
+ - *libabigail_build_packages
+ - env: DEF_LIB="shared" ABI_CHECKS=1
arch: arm64
compiler: gcc
addons:
apt:
packages:
- - *extra_packages
+ - *libabigail_build_packages
script: ./.ci/${TRAVIS_OS_NAME}-build.sh
--
2.23.0
^ permalink raw reply [relevance 9%]
* Re: [dpdk-dev] [PATCH v8 1/1] fbarray: fix duplicated fbarray file in secondary
2020-02-14 15:29 3% ` Thomas Monjalon
@ 2020-02-17 12:54 0% ` Yasufumi Ogawa
0 siblings, 0 replies; 200+ results
From: Yasufumi Ogawa @ 2020-02-17 12:54 UTC (permalink / raw)
To: Thomas Monjalon, David Marchand
Cc: Burakov, Anatoly, Ananyev, Konstantin, dev
> 14/02/2020 16:08, David Marchand:
>> On Fri, Feb 14, 2020 at 8:46 AM Yasufumi Ogawa <yasufum.o@gmail.com> wrote:
>>>
>>> Hi,
>>>
>>> Could I confirm that this patch is going to be merged in 20.02?
>>
>> Sorry, but I can't take this patch in 20.02.
>> It breaks compilation on FreeBSD.
>> http://mails.dpdk.org/archives/test-report/2019-November/109435.html
Sorry. I didn't find it. I'd see it.
>>
>>
>> I am still unconvinced on the need to change the size to something so
>> huge to accommodate with this new use case (secondary processes in
>> containers).
It is not so common actually, but serious issue for some NFV usecases. I
remember, in a talk in the last DPDK summit, ZTE was also suffered from
the same problem.
>> Why can't we truncate the container hostname so that it fits in 64 bytes?
It is just a possible maximum length of format of
"fbarray_memseg-1048576k-0-0_PID_HOSTNAME", so I think it can be
truncated if dropping some information.
>>
>>
>> Thomas, opinion?
>
> If the use case is justified enough, I would prefer merging such change in
> 20.11 avoiding an ABI breakage in a core library, even if it is experimental.I understand, thanks.
Yasufumi
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [RFC] meter: fix ABI break due to experimental tag removal
2020-02-14 21:52 4% ` Bruce Richardson
@ 2020-02-15 13:43 4% ` Neil Horman
0 siblings, 0 replies; 200+ results
From: Neil Horman @ 2020-02-15 13:43 UTC (permalink / raw)
To: Bruce Richardson
Cc: Ray Kinsella, Luca Boccassi, Ferruh Yigit, Cristian Dumitrescu,
Eelco Chaudron, dev, Thomas Monjalon, David Marchand, Ian Stokes
On Fri, Feb 14, 2020 at 09:52:53PM +0000, Bruce Richardson wrote:
> On Fri, Feb 14, 2020 at 03:48:48PM -0500, Neil Horman wrote:
> > On Fri, Feb 14, 2020 at 11:36:34AM +0000, Bruce Richardson wrote:
> > > On Thu, Feb 13, 2020 at 09:40:40PM -0500, Neil Horman wrote:
> > > > On Thu, Feb 13, 2020 at 05:40:43PM +0000, Ray Kinsella wrote:
> > > > >
> > > > >
> > > > > On 05/02/2020 11:32, Neil Horman wrote:
> > > > > > On Wed, Feb 05, 2020 at 11:04:29AM +0100, Luca Boccassi wrote:
> > > > > >> On Tue, 2020-02-04 at 07:02 -0500, Neil Horman wrote:
> > > > > >>>> But if we can do the versioning in the master, LTS can backport it
> > > > > >>>> and can have mature version of that API in LTS without breaking
> > > > > >>>> the existing users.
> > > > > >>>>
> > > > > >>>
> > > > > >>> But why bother? The only thing you've changed is the version
> > > > > >>> tagging. Its ok to leave that alone in LTS, you just cant change
> > > > > >>> it.
> > > > > >>>
> > > > > >>> Thats part of the burden of an LTS release, it will have some drift
> > > > > >>> from the upstream head, because you have to keep things stable. So
> > > > > >>> you stabilize the upstream ABI version for this API and just never
> > > > > >>> backport it to the current LTS release.
> > > > > >>
> > > > > >> Hi,
> > > > > >>
> > > > > >> A customer (OVS) explicitly and specifically requested backporting
> > > > > >> the symbol change to 19.11, as they don't want to enable
> > > > > >> experimental APIs in their releases. I replied that it would only be
> > > > > >> acceptable with aliasing to keep compatibility, and Ferruh very
> > > > > >> kindly did the work to implement that.
> > > > > >>
> > > > > > but, thats part of the agreement, no? You can't always have new
> > > > > > features and stability at the same time.
> > > > > >
> > > > > > I get that this is an odd corner case, because strictly speaking you
> > > > > > could waive the ABI change that libabigail is reporting, and most
> > > > > > application users (like OVS) could get away with it, because their
> > > > > > application does all the right things to make it ok, but I don't
> > > > > > think you can make a decsion like this for all users based on the
> > > > > > request of a single user.
> > > > > >
> > > > > > It seems like the right thing is for OVS to augment their build time
> > > > > > configuration to allow developers to select the ability to use
> > > > > > experimental apis at compile time, and then the decision is placed in
> > > > > > the hands of end users.
> > > > >
> > > > > So this is not isolated case right ... it is common from API's to
> > > > > graduate from experimental to mature, we _want_ that to happen, we
> > > > > _want_ APIs to mature.
> > > > >
> > > > Sure, I can absolutely agree with that, though I would suggest that the
> > > > maturity of the ABI is orthogonal to its labeling as such (more on that
> > > > below)
> > > >
> > > > > I am worried what you are proposing will encourage a behavior whereby
> > > > > maintainers to wait until the declaration of the next major ABI version
> > > > > to make these symbol changes, causing these kinds of changes to queue
> > > > > up for that release.
> > > > >
> > > > I think you're probably right about that, and would make the agrument
> > > > that thats perfectly fine (again I'll clarify below)
> > > >
> > > > > And you would have to ask why? In the case of a reasonably mature API,
> > > > > there will be no functional or signature change - its mature! So what
> > > > > is the harm of providing an alias back to Experimental until the next
> > > > > ABI version is declared?
> > > > >
> > > > From a philosophical standpoint, there is absoluely no harm, and I don't
> > > > think anyone would disagree, the harm comes from the details of the
> > > > implementation, as you've noted.
> > > >
> > > > > So while yes, you are 100% right - experimental mean no guarantees.
> > > > > But if the API is baked, it is not going to change, so I don't see the
> > > > > harm.
> > > > >
> > > > > We don't want the unnecessary triumph of policy over pragmatism.
> > > > >
> > > > I would make the converse argument here. While I agree that when an API
> > > > is mature, theres no point in calling it experimental anymore, I would
> > > > also suggest that, if an API is mature, and not expected to change,
> > > > theres no harm in leaving its version tag as experimental for an LTS
> > > > release. This is what I was referring to above. Once an application
> > > > developer has done the work to integrate an API from DPDK into its
> > > > application, that work is done. If the API remains stable, then I
> > > > honestly don't know that they care that the version label is EXPERIMENTAL
> > > > versus 20.11 or 20.05 or whatever. They care about the API being stable
> > > > more so than its version label. As such, it seems....reasonable to me to
> > > > have developers queue their migration of experimental APIs to official
> > > > versioned APIs at the next major release deliniation point.
> > > >
> > > > I'd welcome counter arguments, but it seems pretty natural to me to make
> > > > these sorts of changes at the major release mark. People using
> > > > experimantal apis have already implicity agreed to manage changes to
> > > > them, so if we just hold it stable in an LTS release (and don't update
> > > > the version label), its just gravy for them.
> > > >
> > > The counter argument that I see is that while the experimental tag remains
> > > in place the user has no way to know that an API is stable or not, and so
> > > in many projects cannot use the API. If for example an API that is
> > > experimental in 19.11 is unchanged through 20.05 at which point we decide
> > > to promote it to stable. Once the change to the exp tag it is made, any
> > > users of 19.11 can then see that it is an unchanged and now-stable ABI and
> > > can start using it in their software, if they wish, without having to wait
> > > for the 20.11 release. Changing the tag early allows ABIs to be potentially
> > > used immediately.
> > >
> >
> > I would agree with this, however, when using an LTS release, in my mind at
> > least, part of the agreement there is you get stability in the fuctions that
> > were declared stable at the time of release. I'm not sure there should be an
> > expectation of additional stabilization within the lifetime of the release
> > (thats really what the 12 month LTS release cycle is for, no)? You never have
> > to wait more than a year for a new set of stable features. If you want faster
> > feature integration than that, you can choose to enable the experimental API's
> > and accept the benefits and drawbacks thereof.
> >
> > That said, if (as I understand it) the goal is to provide a mechanism to allow
> > experimental features to be promoted to stable status, perhaps we can find a
> > happy middle ground. What if we were to create a construct such as this:
> >
> > #pragma push_macro("ALLOW_EXPERIMENTAL_APIS")
> > #undef ALLOW_EXPERIMENTAL_APIS
> > void __rte_experimental func(...);
> > #pragma pop_macro("ALLOW_EXPERIMENTAL_APIS")
> >
> > Such a consruct would allow the maintainer of an API to pseudo-promote an API
> > from a experimental to stable status, at least so far as compilation would be
> > concerned. Its messy and clunky, and it wouldn't change the function version at
> > all, but the end result would be that:
> > a) such a wraped experimental function would no longer issue a warning when used
> > during compilation/linking
> > and
> > b) provide a semi-easy grepable pattern for application writers to look for when
> > considering the use of an API that was previously experimental
> >
> > such a construct would have to be used very judiciously, in that once its
> > implemented, the API has to be treated as stable, even though its 'excused' from
> > the normal checking, but it could provide something of the more rapid promotion
> > path being sought.
> >
> Sure it could work and does meet the end goal. I'm not really sure it's a
> better solution than just promoting the function and putting in an alias
> for it back to experimental, though.
>
It probably isn't, I was just trying to think of alternative
implementations, that dont make life harder in keeping up with LTS
maintenence.
Neil
> /Bruce
>
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [RFC] meter: fix ABI break due to experimental tag removal
2020-02-14 20:48 4% ` Neil Horman
@ 2020-02-14 21:52 4% ` Bruce Richardson
2020-02-15 13:43 4% ` Neil Horman
2020-02-17 14:23 8% ` Ray Kinsella
1 sibling, 1 reply; 200+ results
From: Bruce Richardson @ 2020-02-14 21:52 UTC (permalink / raw)
To: Neil Horman
Cc: Ray Kinsella, Luca Boccassi, Ferruh Yigit, Cristian Dumitrescu,
Eelco Chaudron, dev, Thomas Monjalon, David Marchand, Ian Stokes
On Fri, Feb 14, 2020 at 03:48:48PM -0500, Neil Horman wrote:
> On Fri, Feb 14, 2020 at 11:36:34AM +0000, Bruce Richardson wrote:
> > On Thu, Feb 13, 2020 at 09:40:40PM -0500, Neil Horman wrote:
> > > On Thu, Feb 13, 2020 at 05:40:43PM +0000, Ray Kinsella wrote:
> > > >
> > > >
> > > > On 05/02/2020 11:32, Neil Horman wrote:
> > > > > On Wed, Feb 05, 2020 at 11:04:29AM +0100, Luca Boccassi wrote:
> > > > >> On Tue, 2020-02-04 at 07:02 -0500, Neil Horman wrote:
> > > > >>>> But if we can do the versioning in the master, LTS can backport it
> > > > >>>> and can have mature version of that API in LTS without breaking
> > > > >>>> the existing users.
> > > > >>>>
> > > > >>>
> > > > >>> But why bother? The only thing you've changed is the version
> > > > >>> tagging. Its ok to leave that alone in LTS, you just cant change
> > > > >>> it.
> > > > >>>
> > > > >>> Thats part of the burden of an LTS release, it will have some drift
> > > > >>> from the upstream head, because you have to keep things stable. So
> > > > >>> you stabilize the upstream ABI version for this API and just never
> > > > >>> backport it to the current LTS release.
> > > > >>
> > > > >> Hi,
> > > > >>
> > > > >> A customer (OVS) explicitly and specifically requested backporting
> > > > >> the symbol change to 19.11, as they don't want to enable
> > > > >> experimental APIs in their releases. I replied that it would only be
> > > > >> acceptable with aliasing to keep compatibility, and Ferruh very
> > > > >> kindly did the work to implement that.
> > > > >>
> > > > > but, thats part of the agreement, no? You can't always have new
> > > > > features and stability at the same time.
> > > > >
> > > > > I get that this is an odd corner case, because strictly speaking you
> > > > > could waive the ABI change that libabigail is reporting, and most
> > > > > application users (like OVS) could get away with it, because their
> > > > > application does all the right things to make it ok, but I don't
> > > > > think you can make a decsion like this for all users based on the
> > > > > request of a single user.
> > > > >
> > > > > It seems like the right thing is for OVS to augment their build time
> > > > > configuration to allow developers to select the ability to use
> > > > > experimental apis at compile time, and then the decision is placed in
> > > > > the hands of end users.
> > > >
> > > > So this is not isolated case right ... it is common from API's to
> > > > graduate from experimental to mature, we _want_ that to happen, we
> > > > _want_ APIs to mature.
> > > >
> > > Sure, I can absolutely agree with that, though I would suggest that the
> > > maturity of the ABI is orthogonal to its labeling as such (more on that
> > > below)
> > >
> > > > I am worried what you are proposing will encourage a behavior whereby
> > > > maintainers to wait until the declaration of the next major ABI version
> > > > to make these symbol changes, causing these kinds of changes to queue
> > > > up for that release.
> > > >
> > > I think you're probably right about that, and would make the agrument
> > > that thats perfectly fine (again I'll clarify below)
> > >
> > > > And you would have to ask why? In the case of a reasonably mature API,
> > > > there will be no functional or signature change - its mature! So what
> > > > is the harm of providing an alias back to Experimental until the next
> > > > ABI version is declared?
> > > >
> > > From a philosophical standpoint, there is absoluely no harm, and I don't
> > > think anyone would disagree, the harm comes from the details of the
> > > implementation, as you've noted.
> > >
> > > > So while yes, you are 100% right - experimental mean no guarantees.
> > > > But if the API is baked, it is not going to change, so I don't see the
> > > > harm.
> > > >
> > > > We don't want the unnecessary triumph of policy over pragmatism.
> > > >
> > > I would make the converse argument here. While I agree that when an API
> > > is mature, theres no point in calling it experimental anymore, I would
> > > also suggest that, if an API is mature, and not expected to change,
> > > theres no harm in leaving its version tag as experimental for an LTS
> > > release. This is what I was referring to above. Once an application
> > > developer has done the work to integrate an API from DPDK into its
> > > application, that work is done. If the API remains stable, then I
> > > honestly don't know that they care that the version label is EXPERIMENTAL
> > > versus 20.11 or 20.05 or whatever. They care about the API being stable
> > > more so than its version label. As such, it seems....reasonable to me to
> > > have developers queue their migration of experimental APIs to official
> > > versioned APIs at the next major release deliniation point.
> > >
> > > I'd welcome counter arguments, but it seems pretty natural to me to make
> > > these sorts of changes at the major release mark. People using
> > > experimantal apis have already implicity agreed to manage changes to
> > > them, so if we just hold it stable in an LTS release (and don't update
> > > the version label), its just gravy for them.
> > >
> > The counter argument that I see is that while the experimental tag remains
> > in place the user has no way to know that an API is stable or not, and so
> > in many projects cannot use the API. If for example an API that is
> > experimental in 19.11 is unchanged through 20.05 at which point we decide
> > to promote it to stable. Once the change to the exp tag it is made, any
> > users of 19.11 can then see that it is an unchanged and now-stable ABI and
> > can start using it in their software, if they wish, without having to wait
> > for the 20.11 release. Changing the tag early allows ABIs to be potentially
> > used immediately.
> >
>
> I would agree with this, however, when using an LTS release, in my mind at
> least, part of the agreement there is you get stability in the fuctions that
> were declared stable at the time of release. I'm not sure there should be an
> expectation of additional stabilization within the lifetime of the release
> (thats really what the 12 month LTS release cycle is for, no)? You never have
> to wait more than a year for a new set of stable features. If you want faster
> feature integration than that, you can choose to enable the experimental API's
> and accept the benefits and drawbacks thereof.
>
> That said, if (as I understand it) the goal is to provide a mechanism to allow
> experimental features to be promoted to stable status, perhaps we can find a
> happy middle ground. What if we were to create a construct such as this:
>
> #pragma push_macro("ALLOW_EXPERIMENTAL_APIS")
> #undef ALLOW_EXPERIMENTAL_APIS
> void __rte_experimental func(...);
> #pragma pop_macro("ALLOW_EXPERIMENTAL_APIS")
>
> Such a consruct would allow the maintainer of an API to pseudo-promote an API
> from a experimental to stable status, at least so far as compilation would be
> concerned. Its messy and clunky, and it wouldn't change the function version at
> all, but the end result would be that:
> a) such a wraped experimental function would no longer issue a warning when used
> during compilation/linking
> and
> b) provide a semi-easy grepable pattern for application writers to look for when
> considering the use of an API that was previously experimental
>
> such a construct would have to be used very judiciously, in that once its
> implemented, the API has to be treated as stable, even though its 'excused' from
> the normal checking, but it could provide something of the more rapid promotion
> path being sought.
>
Sure it could work and does meet the end goal. I'm not really sure it's a
better solution than just promoting the function and putting in an alias
for it back to experimental, though.
/Bruce
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [RFC] meter: fix ABI break due to experimental tag removal
2020-02-14 11:36 7% ` Bruce Richardson
@ 2020-02-14 20:48 4% ` Neil Horman
2020-02-14 21:52 4% ` Bruce Richardson
2020-02-17 14:23 8% ` Ray Kinsella
0 siblings, 2 replies; 200+ results
From: Neil Horman @ 2020-02-14 20:48 UTC (permalink / raw)
To: Bruce Richardson
Cc: Ray Kinsella, Luca Boccassi, Ferruh Yigit, Cristian Dumitrescu,
Eelco Chaudron, dev, Thomas Monjalon, David Marchand, Ian Stokes
On Fri, Feb 14, 2020 at 11:36:34AM +0000, Bruce Richardson wrote:
> On Thu, Feb 13, 2020 at 09:40:40PM -0500, Neil Horman wrote:
> > On Thu, Feb 13, 2020 at 05:40:43PM +0000, Ray Kinsella wrote:
> > >
> > >
> > > On 05/02/2020 11:32, Neil Horman wrote:
> > > > On Wed, Feb 05, 2020 at 11:04:29AM +0100, Luca Boccassi wrote:
> > > >> On Tue, 2020-02-04 at 07:02 -0500, Neil Horman wrote:
> > > >>>> But if we can do the versioning in the master, LTS can backport it
> > > >>>> and can have mature version of that API in LTS without breaking
> > > >>>> the existing users.
> > > >>>>
> > > >>>
> > > >>> But why bother? The only thing you've changed is the version
> > > >>> tagging. Its ok to leave that alone in LTS, you just cant change
> > > >>> it.
> > > >>>
> > > >>> Thats part of the burden of an LTS release, it will have some drift
> > > >>> from the upstream head, because you have to keep things stable. So
> > > >>> you stabilize the upstream ABI version for this API and just never
> > > >>> backport it to the current LTS release.
> > > >>
> > > >> Hi,
> > > >>
> > > >> A customer (OVS) explicitly and specifically requested backporting
> > > >> the symbol change to 19.11, as they don't want to enable
> > > >> experimental APIs in their releases. I replied that it would only be
> > > >> acceptable with aliasing to keep compatibility, and Ferruh very
> > > >> kindly did the work to implement that.
> > > >>
> > > > but, thats part of the agreement, no? You can't always have new
> > > > features and stability at the same time.
> > > >
> > > > I get that this is an odd corner case, because strictly speaking you
> > > > could waive the ABI change that libabigail is reporting, and most
> > > > application users (like OVS) could get away with it, because their
> > > > application does all the right things to make it ok, but I don't
> > > > think you can make a decsion like this for all users based on the
> > > > request of a single user.
> > > >
> > > > It seems like the right thing is for OVS to augment their build time
> > > > configuration to allow developers to select the ability to use
> > > > experimental apis at compile time, and then the decision is placed in
> > > > the hands of end users.
> > >
> > > So this is not isolated case right ... it is common from API's to
> > > graduate from experimental to mature, we _want_ that to happen, we
> > > _want_ APIs to mature.
> > >
> > Sure, I can absolutely agree with that, though I would suggest that the
> > maturity of the ABI is orthogonal to its labeling as such (more on that
> > below)
> >
> > > I am worried what you are proposing will encourage a behavior whereby
> > > maintainers to wait until the declaration of the next major ABI version
> > > to make these symbol changes, causing these kinds of changes to queue
> > > up for that release.
> > >
> > I think you're probably right about that, and would make the agrument
> > that thats perfectly fine (again I'll clarify below)
> >
> > > And you would have to ask why? In the case of a reasonably mature API,
> > > there will be no functional or signature change - its mature! So what
> > > is the harm of providing an alias back to Experimental until the next
> > > ABI version is declared?
> > >
> > From a philosophical standpoint, there is absoluely no harm, and I don't
> > think anyone would disagree, the harm comes from the details of the
> > implementation, as you've noted.
> >
> > > So while yes, you are 100% right - experimental mean no guarantees.
> > > But if the API is baked, it is not going to change, so I don't see the
> > > harm.
> > >
> > > We don't want the unnecessary triumph of policy over pragmatism.
> > >
> > I would make the converse argument here. While I agree that when an API
> > is mature, theres no point in calling it experimental anymore, I would
> > also suggest that, if an API is mature, and not expected to change,
> > theres no harm in leaving its version tag as experimental for an LTS
> > release. This is what I was referring to above. Once an application
> > developer has done the work to integrate an API from DPDK into its
> > application, that work is done. If the API remains stable, then I
> > honestly don't know that they care that the version label is EXPERIMENTAL
> > versus 20.11 or 20.05 or whatever. They care about the API being stable
> > more so than its version label. As such, it seems....reasonable to me to
> > have developers queue their migration of experimental APIs to official
> > versioned APIs at the next major release deliniation point.
> >
> > I'd welcome counter arguments, but it seems pretty natural to me to make
> > these sorts of changes at the major release mark. People using
> > experimantal apis have already implicity agreed to manage changes to
> > them, so if we just hold it stable in an LTS release (and don't update
> > the version label), its just gravy for them.
> >
> The counter argument that I see is that while the experimental tag remains
> in place the user has no way to know that an API is stable or not, and so
> in many projects cannot use the API. If for example an API that is
> experimental in 19.11 is unchanged through 20.05 at which point we decide
> to promote it to stable. Once the change to the exp tag it is made, any
> users of 19.11 can then see that it is an unchanged and now-stable ABI and
> can start using it in their software, if they wish, without having to wait
> for the 20.11 release. Changing the tag early allows ABIs to be potentially
> used immediately.
>
I would agree with this, however, when using an LTS release, in my mind at
least, part of the agreement there is you get stability in the fuctions that
were declared stable at the time of release. I'm not sure there should be an
expectation of additional stabilization within the lifetime of the release
(thats really what the 12 month LTS release cycle is for, no)? You never have
to wait more than a year for a new set of stable features. If you want faster
feature integration than that, you can choose to enable the experimental API's
and accept the benefits and drawbacks thereof.
That said, if (as I understand it) the goal is to provide a mechanism to allow
experimental features to be promoted to stable status, perhaps we can find a
happy middle ground. What if we were to create a construct such as this:
#pragma push_macro("ALLOW_EXPERIMENTAL_APIS")
#undef ALLOW_EXPERIMENTAL_APIS
void __rte_experimental func(...);
#pragma pop_macro("ALLOW_EXPERIMENTAL_APIS")
Such a consruct would allow the maintainer of an API to pseudo-promote an API
from a experimental to stable status, at least so far as compilation would be
concerned. Its messy and clunky, and it wouldn't change the function version at
all, but the end result would be that:
a) such a wraped experimental function would no longer issue a warning when used
during compilation/linking
and
b) provide a semi-easy grepable pattern for application writers to look for when
considering the use of an API that was previously experimental
such a construct would have to be used very judiciously, in that once its
implemented, the API has to be treated as stable, even though its 'excused' from
the normal checking, but it could provide something of the more rapid promotion
path being sought.
Thoughts?
Neil
> Regards,
> /Bruce
>
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v2] service: don't walk out of bounds when checking services
2020-02-07 14:27 3% ` Aaron Conole
@ 2020-02-14 16:38 0% ` Kevin Traynor
0 siblings, 0 replies; 200+ results
From: Kevin Traynor @ 2020-02-14 16:38 UTC (permalink / raw)
To: Aaron Conole
Cc: Harry van Haaren, nikhil.rao, David Marchand, dev,
Bruce Richardson, Pavan Nikhilesh, Gage Eads, Thomas Monjalon,
dpdk stable
On 07/02/2020 14:27, Aaron Conole wrote:
> Kevin Traynor <ktraynor@redhat.com> writes:
>
>> On 20/12/2019 14:43, David Marchand wrote:
>>> On Wed, Dec 4, 2019 at 9:34 AM David Marchand <david.marchand@redhat.com> wrote:
>>>>
>>>> On Wed, Dec 4, 2019 at 9:33 AM David Marchand <david.marchand@redhat.com> wrote:
>>>>>
>>>>> On Tue, Dec 3, 2019 at 10:15 PM Aaron Conole <aconole@redhat.com> wrote:
>>>>>>
>>>>>> The service_valid call is used without properly bounds checking the
>>>>>> input parameter. Almost all instances of the service_valid call are
>>>>>> inside a for() loop that prevents excessive walks, but some of the
>>>>>> public APIs don't bounds check and will pass invalid arguments.
>>>>>>
>>>>>> Prevent this by using SERVICE_GET_OR_ERR_RET where it makes sense,
>>>>>> and adding a bounds check to one service_valid() use.
>>>>>>
>>>>>> Fixes: 8d39d3e237c2 ("service: fix race in service on app lcore function")
>>>>>> Fixes: e9139a32f6e8 ("service: add function to run on app lcore")
>>>>>> Fixes: e30dd31847d2 ("service: add mechanism for quiescing")
>>>> Cc: stable@dpdk.org
>>>>
>>
>> With the commit below, this patch will apply cleanly on 18.11.
>>
>> Seems ok to me to add below commit, wdyt?
>
> If I'm reading it correctly, the move is for an internal data structure
> in librte_eal, so I think it shouldn't be an ABI breakage.
>
> Looks safe to me as well.
>
Thanks, both patches applied.
>> commit e484ccddbe1b41886fef1e445ef2fdfa55086198
>> Author: Nikhil Rao <nikhil.rao@intel.com>
>> Date: Mon Sep 16 15:31:02 2019 +0530
>>
>> service: avoid false sharing on core state
>>
>>
>>>>>> Signed-off-by: Aaron Conole <aconole@redhat.com>
>>>>>
>>>>> Reviewed-by: David Marchand <david.marchand@redhat.com>
>>>
>>> Applied, thanks.
>>>
>>>
>>> --
>>> David Marchand
>>>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v8 1/1] fbarray: fix duplicated fbarray file in secondary
@ 2020-02-14 15:29 3% ` Thomas Monjalon
2020-02-17 12:54 0% ` Yasufumi Ogawa
0 siblings, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-02-14 15:29 UTC (permalink / raw)
To: Yasufumi Ogawa, David Marchand; +Cc: Burakov, Anatoly, Ananyev, Konstantin, dev
14/02/2020 16:08, David Marchand:
> On Fri, Feb 14, 2020 at 8:46 AM Yasufumi Ogawa <yasufum.o@gmail.com> wrote:
> >
> > Hi,
> >
> > Could I confirm that this patch is going to be merged in 20.02?
>
> Sorry, but I can't take this patch in 20.02.
> It breaks compilation on FreeBSD.
> http://mails.dpdk.org/archives/test-report/2019-November/109435.html
>
>
> I am still unconvinced on the need to change the size to something so
> huge to accommodate with this new use case (secondary processes in
> containers).
> Why can't we truncate the container hostname so that it fits in 64 bytes?
>
>
> Thomas, opinion?
If the use case is justified enough, I would prefer merging such change in
20.11 avoiding an ABI breakage in a core library, even if it is experimental.
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH 0/7] vfio/pci: SR-IOV support
2020-02-14 4:57 0% ` Alexey Kardashevskiy
@ 2020-02-14 15:27 0% ` Alex Williamson
0 siblings, 0 replies; 200+ results
From: Alex Williamson @ 2020-02-14 15:27 UTC (permalink / raw)
To: Alexey Kardashevskiy
Cc: kvm, linux-pci, linux-kernel, dev, mtosatti, thomas, bluca,
jerinjacobk, bruce.richardson, cohuck
On Fri, 14 Feb 2020 15:57:04 +1100
Alexey Kardashevskiy <aik@ozlabs.ru> wrote:
> On 12/02/2020 10:05, Alex Williamson wrote:
> > Given the mostly positive feedback from the RFC[1], here's a new
> > non-RFC revision. Changes since RFC:
> >
> > - vfio_device_ops.match semantics refined
> > - Use helpers for struct pci_dev.physfn to avoid breakage without
> > CONFIG_PCI_IOV
> > - Relax to allow SR-IOV configuration changes while PF is opened.
> > There are potentially interesting use cases here, including
> > perhaps QEMU emulating an SR-IOV capability and calling out
> > to a privileged entity to manipulate sriov_numvfs and corral
> > the resulting devices.
> > - Retest vfio_device_feature.argsz to include uuid length.
> > - Add Connie's R-b on 6/7
> >
> > I still wish we had a solution to make it less opaque to the user
> > why a VFIO_GROUP_GET_DEVICE_FD() has failed if a VF token is
> > required, but this is still the best I've been able to come up with.
> > If there are objections or better ideas, please raise them now.
> >
> > The synopsis of this series is that we have an ongoing desire to drive
> > PCIe SR-IOV PFs from userspace with VFIO. There's an immediate need
> > for this with DPDK drivers and potentially interesting future use
> > cases in virtualization. We've been reluctant to add this support
> > previously due to the dependency and trust relationship between the
> > VF device and PF driver. Minimally the PF driver can induce a denial
> > of service to the VF, but depending on the specific implementation,
> > the PF driver might also be responsible for moving data between VFs
> > or have direct access to the state of the VF, including data or state
> > otherwise private to the VF or VF driver.
> >
> > To help resolve these concerns, we introduce a VF token into the VFIO
> > PCI ABI, which acts as a shared secret key between drivers. The
> > userspace PF driver is required to set the VF token to a known value
> > and userspace VF drivers are required to provide the token to access
> > the VF device. If a PF driver is restarted with VF drivers in use, it
> > must also provide the current token in order to prevent a rogue
> > untrusted PF driver from replacing a known driver. The degree to
> > which this new token is considered secret is left to the userspace
> > drivers, the kernel intentionally provides no means to retrieve the
> > current token.
> >
> > Note that the above token is only required for this new model where
> > both the PF and VF devices are usable through vfio-pci. Existing
> > models of VFIO drivers where the PF is used without SR-IOV enabled
> > or the VF is bound to a userspace driver with an in-kernel, host PF
> > driver are unaffected.
> >
> > The latter configuration above also highlights a new inverted scenario
> > that is now possible, a userspace PF driver with in-kernel VF drivers.
> > I believe this is a scenario that should be allowed, but should not be
> > enabled by default. This series includes code to set a default
> > driver_override for VFs sourced from a vfio-pci user owned PF, such
> > that the VFs are also bound to vfio-pci. This model is compatible
> > with tools like driverctl and allows the system administrator to
> > decide if other bindings should be enabled. The VF token interface
> > above exists only between vfio-pci PF and VF drivers, once a VF is
> > bound to another driver, the administrator has effectively pronounced
> > the device as trusted. The vfio-pci driver will note alternate
> > binding in dmesg for logging and debugging purposes.
> >
> > Please review, comment, and test. The example QEMU implementation
> > provided with the RFC[2] is still current for this version. Thanks,
>
>
> It is a cool feature. One question - what device have you tested it with?
>
> Does not a PF want to control/manage VFs on a PF driver side? I am
> thinking of Mellanox CX5 or similar NIC and it acts as an managed
> ethernet switch which might want to do something to VFs and VFs may not
> work as expected without PF's native driver doing things to it, or this
> is not a concern, is it? Thanks,
TBH, I'm starting with the premise that a userspace PF driver already
works. The DPDK folks have produced some "interesting" code that
allows SR-IOV to be enabled on a PF underneath vfio-pci. There's also
a non-upstream igb-uio driver associated with DPDK that seems to be
recommended for SR-IOV PF driver use cases, particularly for an FPGA
device. The testing I've done, and what's provided by the QEMU patch I
reference, is really only unit testing the vf_token support and
DEVICE_FEATURE ioctl provided here. I used this with an Intel 82576
(igb) where the PF driver doesn't particularly like being assigned to a
VM with SR-IOV enabled. Likewise, I can prove that the interfaces here
provide the correct restrictions for the VF, but the VF doesn't work in
a VM due to the state of the PF. I'm hoping we'll have some
confirmation from the DPDK folks that this provides what they need to
abandon the non-upstream drivers and more nefarious hacks. There's a
lot more virtualization work to be done in QEMU before I'd propose
patch I reference above upstream.
To your specific question regarding CX5, I think there are very few
SR-IOV devices where the PF doesn't act as some kind of packet router
or ring management engine. The Amazon device listed in the pci-pf-stub
driver seems to be one of the few SR-IOV devices which claim the PF has
no special interfaces other than exposing the SR-IOV capability itself.
So I think we generally expect a device specific SR-IOV aware driver
running on the PF via this interface. That's certainly the case for
the DPDK code for the FPGA device above. Thanks,
Alex
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [RFC] meter: fix ABI break due to experimental tag removal
2020-02-14 2:40 7% ` Neil Horman
@ 2020-02-14 11:36 7% ` Bruce Richardson
2020-02-14 20:48 4% ` Neil Horman
0 siblings, 1 reply; 200+ results
From: Bruce Richardson @ 2020-02-14 11:36 UTC (permalink / raw)
To: Neil Horman
Cc: Ray Kinsella, Luca Boccassi, Ferruh Yigit, Cristian Dumitrescu,
Eelco Chaudron, dev, Thomas Monjalon, David Marchand, Ian Stokes
On Thu, Feb 13, 2020 at 09:40:40PM -0500, Neil Horman wrote:
> On Thu, Feb 13, 2020 at 05:40:43PM +0000, Ray Kinsella wrote:
> >
> >
> > On 05/02/2020 11:32, Neil Horman wrote:
> > > On Wed, Feb 05, 2020 at 11:04:29AM +0100, Luca Boccassi wrote:
> > >> On Tue, 2020-02-04 at 07:02 -0500, Neil Horman wrote:
> > >>>> But if we can do the versioning in the master, LTS can backport it
> > >>>> and can have mature version of that API in LTS without breaking
> > >>>> the existing users.
> > >>>>
> > >>>
> > >>> But why bother? The only thing you've changed is the version
> > >>> tagging. Its ok to leave that alone in LTS, you just cant change
> > >>> it.
> > >>>
> > >>> Thats part of the burden of an LTS release, it will have some drift
> > >>> from the upstream head, because you have to keep things stable. So
> > >>> you stabilize the upstream ABI version for this API and just never
> > >>> backport it to the current LTS release.
> > >>
> > >> Hi,
> > >>
> > >> A customer (OVS) explicitly and specifically requested backporting
> > >> the symbol change to 19.11, as they don't want to enable
> > >> experimental APIs in their releases. I replied that it would only be
> > >> acceptable with aliasing to keep compatibility, and Ferruh very
> > >> kindly did the work to implement that.
> > >>
> > > but, thats part of the agreement, no? You can't always have new
> > > features and stability at the same time.
> > >
> > > I get that this is an odd corner case, because strictly speaking you
> > > could waive the ABI change that libabigail is reporting, and most
> > > application users (like OVS) could get away with it, because their
> > > application does all the right things to make it ok, but I don't
> > > think you can make a decsion like this for all users based on the
> > > request of a single user.
> > >
> > > It seems like the right thing is for OVS to augment their build time
> > > configuration to allow developers to select the ability to use
> > > experimental apis at compile time, and then the decision is placed in
> > > the hands of end users.
> >
> > So this is not isolated case right ... it is common from API's to
> > graduate from experimental to mature, we _want_ that to happen, we
> > _want_ APIs to mature.
> >
> Sure, I can absolutely agree with that, though I would suggest that the
> maturity of the ABI is orthogonal to its labeling as such (more on that
> below)
>
> > I am worried what you are proposing will encourage a behavior whereby
> > maintainers to wait until the declaration of the next major ABI version
> > to make these symbol changes, causing these kinds of changes to queue
> > up for that release.
> >
> I think you're probably right about that, and would make the agrument
> that thats perfectly fine (again I'll clarify below)
>
> > And you would have to ask why? In the case of a reasonably mature API,
> > there will be no functional or signature change - its mature! So what
> > is the harm of providing an alias back to Experimental until the next
> > ABI version is declared?
> >
> From a philosophical standpoint, there is absoluely no harm, and I don't
> think anyone would disagree, the harm comes from the details of the
> implementation, as you've noted.
>
> > So while yes, you are 100% right - experimental mean no guarantees.
> > But if the API is baked, it is not going to change, so I don't see the
> > harm.
> >
> > We don't want the unnecessary triumph of policy over pragmatism.
> >
> I would make the converse argument here. While I agree that when an API
> is mature, theres no point in calling it experimental anymore, I would
> also suggest that, if an API is mature, and not expected to change,
> theres no harm in leaving its version tag as experimental for an LTS
> release. This is what I was referring to above. Once an application
> developer has done the work to integrate an API from DPDK into its
> application, that work is done. If the API remains stable, then I
> honestly don't know that they care that the version label is EXPERIMENTAL
> versus 20.11 or 20.05 or whatever. They care about the API being stable
> more so than its version label. As such, it seems....reasonable to me to
> have developers queue their migration of experimental APIs to official
> versioned APIs at the next major release deliniation point.
>
> I'd welcome counter arguments, but it seems pretty natural to me to make
> these sorts of changes at the major release mark. People using
> experimantal apis have already implicity agreed to manage changes to
> them, so if we just hold it stable in an LTS release (and don't update
> the version label), its just gravy for them.
>
The counter argument that I see is that while the experimental tag remains
in place the user has no way to know that an API is stable or not, and so
in many projects cannot use the API. If for example an API that is
experimental in 19.11 is unchanged through 20.05 at which point we decide
to promote it to stable. Once the change to the exp tag it is made, any
users of 19.11 can then see that it is an unchanged and now-stable ABI and
can start using it in their software, if they wish, without having to wait
for the 20.11 release. Changing the tag early allows ABIs to be potentially
used immediately.
Regards,
/Bruce
^ permalink raw reply [relevance 7%]
* Re: [dpdk-dev] [PATCH v8 1/1] fbarray: fix duplicated fbarray file in secondary
@ 2020-02-14 7:46 0% ` Yasufumi Ogawa
0 siblings, 1 reply; 200+ results
From: Yasufumi Ogawa @ 2020-02-14 7:46 UTC (permalink / raw)
To: Burakov, Anatoly, konstantin.ananyev, david.marchand, dev
Cc: yasufumi.ogawa.gy, Yasufumi Ogawa
Hi,
Could I confirm that this patch is going to be merged in 20.02?
Regards,
Yasufumi
On 2019/12/06 19:44, Burakov, Anatoly wrote:
> On 27-Nov-19 8:48 AM, Yasufumi Ogawa wrote:
>> From: Yasufumi Ogawa <ogawa.yasufumi@lab.ntt.co.jp>
>>
>> In secondary_msl_create_walk(), it creates a file for fbarrays with its
>> PID for reserving unique name among secondary processes. However, it
>> does not work if several secondaries run as app containers because each
>> of containerized secondary has PID 1, and failed to reserve unique name
>> other than first one. To reserve unique name in each of containers, use
>> hostname in addition to PID.
>>
>> Signed-off-by: Yasufumi Ogawa <ogawa.yasufumi@lab.ntt.co.jp>
>> ---
>> lib/librte_eal/common/include/rte_fbarray.h | 7 ++++++-
>> lib/librte_eal/linux/eal/eal_memalloc.c | 11 ++++++++---
>> 2 files changed, 14 insertions(+), 4 deletions(-)
>>
>> diff --git a/lib/librte_eal/common/include/rte_fbarray.h
>> b/lib/librte_eal/common/include/rte_fbarray.h
>> index 6dccdbec9..df003b8dc 100644
>> --- a/lib/librte_eal/common/include/rte_fbarray.h
>> +++ b/lib/librte_eal/common/include/rte_fbarray.h
>> @@ -39,7 +39,12 @@ extern "C" {
>> #include <rte_compat.h>
>> #include <rte_rwlock.h>
>> -#define RTE_FBARRAY_NAME_LEN 64
>> +/* Filename of fbarray is defined as a combination of several params
>> + * such as "fbarray_memseg-1048576k-0-0_PID_HOSTNAME".
>> + * The length of string before PID can be 32bytes, and the length of
>> + * PID can be 7bytes maximamly. Final 1 byte is for null terminator.
>> + */
>> +#define RTE_FBARRAY_NAME_LEN (32 + 7 + 1 + HOST_NAME_MAX + 1)
>
> This breaks ABI, but i believe this doesn't break *stable* ABI, so it is
> OK.
>
> Reviewed-by: Anatoly Burakov <anatoly.burakov@intel.com>
>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH 0/7] vfio/pci: SR-IOV support
2020-02-11 23:05 2% [dpdk-dev] [PATCH 0/7] vfio/pci: SR-IOV support Alex Williamson
@ 2020-02-14 4:57 0% ` Alexey Kardashevskiy
2020-02-14 15:27 0% ` Alex Williamson
0 siblings, 1 reply; 200+ results
From: Alexey Kardashevskiy @ 2020-02-14 4:57 UTC (permalink / raw)
To: Alex Williamson, kvm
Cc: linux-pci, linux-kernel, dev, mtosatti, thomas, bluca,
jerinjacobk, bruce.richardson, cohuck
On 12/02/2020 10:05, Alex Williamson wrote:
> Given the mostly positive feedback from the RFC[1], here's a new
> non-RFC revision. Changes since RFC:
>
> - vfio_device_ops.match semantics refined
> - Use helpers for struct pci_dev.physfn to avoid breakage without
> CONFIG_PCI_IOV
> - Relax to allow SR-IOV configuration changes while PF is opened.
> There are potentially interesting use cases here, including
> perhaps QEMU emulating an SR-IOV capability and calling out
> to a privileged entity to manipulate sriov_numvfs and corral
> the resulting devices.
> - Retest vfio_device_feature.argsz to include uuid length.
> - Add Connie's R-b on 6/7
>
> I still wish we had a solution to make it less opaque to the user
> why a VFIO_GROUP_GET_DEVICE_FD() has failed if a VF token is
> required, but this is still the best I've been able to come up with.
> If there are objections or better ideas, please raise them now.
>
> The synopsis of this series is that we have an ongoing desire to drive
> PCIe SR-IOV PFs from userspace with VFIO. There's an immediate need
> for this with DPDK drivers and potentially interesting future use
> cases in virtualization. We've been reluctant to add this support
> previously due to the dependency and trust relationship between the
> VF device and PF driver. Minimally the PF driver can induce a denial
> of service to the VF, but depending on the specific implementation,
> the PF driver might also be responsible for moving data between VFs
> or have direct access to the state of the VF, including data or state
> otherwise private to the VF or VF driver.
>
> To help resolve these concerns, we introduce a VF token into the VFIO
> PCI ABI, which acts as a shared secret key between drivers. The
> userspace PF driver is required to set the VF token to a known value
> and userspace VF drivers are required to provide the token to access
> the VF device. If a PF driver is restarted with VF drivers in use, it
> must also provide the current token in order to prevent a rogue
> untrusted PF driver from replacing a known driver. The degree to
> which this new token is considered secret is left to the userspace
> drivers, the kernel intentionally provides no means to retrieve the
> current token.
>
> Note that the above token is only required for this new model where
> both the PF and VF devices are usable through vfio-pci. Existing
> models of VFIO drivers where the PF is used without SR-IOV enabled
> or the VF is bound to a userspace driver with an in-kernel, host PF
> driver are unaffected.
>
> The latter configuration above also highlights a new inverted scenario
> that is now possible, a userspace PF driver with in-kernel VF drivers.
> I believe this is a scenario that should be allowed, but should not be
> enabled by default. This series includes code to set a default
> driver_override for VFs sourced from a vfio-pci user owned PF, such
> that the VFs are also bound to vfio-pci. This model is compatible
> with tools like driverctl and allows the system administrator to
> decide if other bindings should be enabled. The VF token interface
> above exists only between vfio-pci PF and VF drivers, once a VF is
> bound to another driver, the administrator has effectively pronounced
> the device as trusted. The vfio-pci driver will note alternate
> binding in dmesg for logging and debugging purposes.
>
> Please review, comment, and test. The example QEMU implementation
> provided with the RFC[2] is still current for this version. Thanks,
It is a cool feature. One question - what device have you tested it with?
Does not a PF want to control/manage VFs on a PF driver side? I am
thinking of Mellanox CX5 or similar NIC and it acts as an managed
ethernet switch which might want to do something to VFs and VFs may not
work as expected without PF's native driver doing things to it, or this
is not a concern, is it? Thanks,
>
> Alex
>
> [1] https://lore.kernel.org/lkml/158085337582.9445.17682266437583505502.stgit@gimli.home/
> [2] https://lore.kernel.org/lkml/20200204161737.34696b91@w520.home/
> ---
>
> Alex Williamson (7):
> vfio: Include optional device match in vfio_device_ops callbacks
> vfio/pci: Implement match ops
> vfio/pci: Introduce VF token
> vfio: Introduce VFIO_DEVICE_FEATURE ioctl and first user
> vfio/pci: Add sriov_configure support
> vfio/pci: Remove dev_fmt definition
> vfio/pci: Cleanup .probe() exit paths
>
>
> drivers/vfio/pci/vfio_pci.c | 312 ++++++++++++++++++++++++++++++++---
> drivers/vfio/pci/vfio_pci_private.h | 10 +
> drivers/vfio/vfio.c | 20 ++
> include/linux/vfio.h | 4
> include/uapi/linux/vfio.h | 37 ++++
> 5 files changed, 355 insertions(+), 28 deletions(-)
>
--
Alexey
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [RFC] meter: fix ABI break due to experimental tag removal
2020-02-13 17:40 8% ` Ray Kinsella
@ 2020-02-14 2:40 7% ` Neil Horman
2020-02-14 11:36 7% ` Bruce Richardson
0 siblings, 1 reply; 200+ results
From: Neil Horman @ 2020-02-14 2:40 UTC (permalink / raw)
To: Ray Kinsella
Cc: Luca Boccassi, Ferruh Yigit, Cristian Dumitrescu, Eelco Chaudron,
dev, Thomas Monjalon, David Marchand, Bruce Richardson,
Ian Stokes
On Thu, Feb 13, 2020 at 05:40:43PM +0000, Ray Kinsella wrote:
>
>
> On 05/02/2020 11:32, Neil Horman wrote:
> > On Wed, Feb 05, 2020 at 11:04:29AM +0100, Luca Boccassi wrote:
> >> On Tue, 2020-02-04 at 07:02 -0500, Neil Horman wrote:
> >>>> But if we can do the versioning in the master, LTS can backport it
> >>>> and can have
> >>>> mature version of that API in LTS without breaking the existing
> >>>> users.
> >>>>
> >>>
> >>> But why bother? The only thing you've changed is the version
> >>> tagging. Its ok
> >>> to leave that alone in LTS, you just cant change it.
> >>>
> >>> Thats part of the burden of an LTS release, it will have some drift
> >>> from the
> >>> upstream head, because you have to keep things stable. So you
> >>> stabilize the
> >>> upstream ABI version for this API and just never backport it to the
> >>> current LTS
> >>> release.
> >>
> >> Hi,
> >>
> >> A customer (OVS) explicitly and specifically requested backporting the
> >> symbol change to 19.11, as they don't want to enable experimental APIs
> >> in their releases. I replied that it would only be acceptable with
> >> aliasing to keep compatibility, and Ferruh very kindly did the work to
> >> implement that.
> >>
> > but, thats part of the agreement, no? You can't always have new features and
> > stability at the same time.
> >
> > I get that this is an odd corner case, because strictly speaking you could waive
> > the ABI change that libabigail is reporting, and most application users (like
> > OVS) could get away with it, because their application does all the right things
> > to make it ok, but I don't think you can make a decsion like this for all users
> > based on the request of a single user.
> >
> > It seems like the right thing is for OVS to augment their build time
> > configuration to allow developers to select the ability to use experimental apis
> > at compile time, and then the decision is placed in the hands of end users.
>
> So this is not isolated case right ... it is common from API's to graduate
> from experimental to mature, we _want_ that to happen, we _want_ APIs to mature.
>
Sure, I can absolutely agree with that, though I would suggest that
the maturity of the ABI is orthogonal to its labeling as such (more on
that below)
> I am worried what you are proposing will encourage a behavior whereby maintainers
> to wait until the declaration of the next major ABI version to make these symbol changes,
> causing these kinds of changes to queue up for that release.
>
I think you're probably right about that, and would make the agrument
that thats perfectly fine (again I'll clarify below)
> And you would have to ask why?
> In the case of a reasonably mature API, there will be no functional or signature change - its mature!
> So what is the harm of providing an alias back to Experimental until the next ABI version is declared?
>
From a philosophical standpoint, there is absoluely no harm, and I don't
think anyone would disagree, the harm comes from the details of the
implementation, as you've noted.
> So while yes, you are 100% right - experimental mean no guarantees.
> But if the API is baked, it is not going to change, so I don't see the harm.
>
> We don't want the unnecessary triumph of policy over pragmatism.
>
I would make the converse argument here. While I agree that when an API
is mature, theres no point in calling it experimental anymore, I would
also suggest that, if an API is mature, and not expected to change,
theres no harm in leaving its version tag as experimental for an LTS
release. This is what I was referring to above. Once an application
developer has done the work to integrate an API from DPDK into its
application, that work is done. If the API remains stable, then I
honestly don't know that they care that the version label is
EXPERIMENTAL versus 20.11 or 20.05 or whatever. They care about the API
being stable more so than its version label. As such, it
seems....reasonable to me to have developers queue their migration of
experimental APIs to official versioned APIs at the next major release
deliniation point.
I'd welcome counter arguments, but it seems pretty natural to me to make
these sorts of changes at the major release mark. People using
experimantal apis have already implicity agreed to manage changes to
them, so if we just hold it stable in an LTS release (and don't update
the version label), its just gravy for them.
Regards
Neil
>
>
>
>
>
>
^ permalink raw reply [relevance 7%]
* Re: [dpdk-dev] [PATCH] doc: remove major in designation of normal releases
@ 2020-02-13 20:51 0% ` Kevin Traynor
0 siblings, 0 replies; 200+ results
From: Kevin Traynor @ 2020-02-13 20:51 UTC (permalink / raw)
To: Thomas Monjalon, John McNamara, Marko Kovacevic
Cc: dev, orika, David Marchand
On 05/08/2019 13:30, Thomas Monjalon wrote:
> The word "major" was used to differentiate with release candidates
> or stable maintenance releases.
> However the word "major" can be understood as "LTS",
> so it is less confusing to avoid this word.
>
> Reported-by: Ori Kam <orika@mellanox.com>
> Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
> ---
> doc/guides/contributing/documentation.rst | 2 +-
> doc/guides/contributing/stable.rst | 2 +-
> 2 files changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/doc/guides/contributing/documentation.rst b/doc/guides/contributing/documentation.rst
> index 27e4b13be..550d8dec2 100644
> --- a/doc/guides/contributing/documentation.rst
> +++ b/doc/guides/contributing/documentation.rst
> @@ -62,7 +62,7 @@ added to by the developer.
>
> The Release Notes document which features have been added in the current and previous releases of DPDK and highlight
> any known issues.
> - The Releases Notes also contain notifications of features that will change ABI compatibility in the next major release.
> + The Releases Notes also contain notifications of features that will change ABI compatibility in the next release.
>
> Developers should include updates to the Release Notes with patch sets that relate to any of the following sections:
>
> diff --git a/doc/guides/contributing/stable.rst b/doc/guides/contributing/stable.rst
> index 6a5eee9bd..24b9e8b7d 100644
> --- a/doc/guides/contributing/stable.rst
> +++ b/doc/guides/contributing/stable.rst
> @@ -25,7 +25,7 @@ Release to indicate longer term support.
> Stable Releases
> ---------------
>
> -Any major release of DPDK can be designated as a Stable Release if a
> +Any release of DPDK can be designated as a Stable Release if a
> maintainer volunteers to maintain it and there is a commitment from major
> contributors to validate it before releases. If a release is to be designated
> as a Stable Release, it should be done by 1 month after the master release.
>
Acked-by: Kevin Traynor <ktraynor@redhat.com>
(Prompted-by: David Marchand <david.marchand@redhat.com> ;-))
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [RFC] meter: fix ABI break due to experimental tag removal
@ 2020-02-13 17:40 8% ` Ray Kinsella
2020-02-14 2:40 7% ` Neil Horman
0 siblings, 1 reply; 200+ results
From: Ray Kinsella @ 2020-02-13 17:40 UTC (permalink / raw)
To: Neil Horman, Luca Boccassi
Cc: Ferruh Yigit, Cristian Dumitrescu, Eelco Chaudron, dev,
Thomas Monjalon, David Marchand, Bruce Richardson, Ian Stokes
On 05/02/2020 11:32, Neil Horman wrote:
> On Wed, Feb 05, 2020 at 11:04:29AM +0100, Luca Boccassi wrote:
>> On Tue, 2020-02-04 at 07:02 -0500, Neil Horman wrote:
>>>> But if we can do the versioning in the master, LTS can backport it
>>>> and can have
>>>> mature version of that API in LTS without breaking the existing
>>>> users.
>>>>
>>>
>>> But why bother? The only thing you've changed is the version
>>> tagging. Its ok
>>> to leave that alone in LTS, you just cant change it.
>>>
>>> Thats part of the burden of an LTS release, it will have some drift
>>> from the
>>> upstream head, because you have to keep things stable. So you
>>> stabilize the
>>> upstream ABI version for this API and just never backport it to the
>>> current LTS
>>> release.
>>
>> Hi,
>>
>> A customer (OVS) explicitly and specifically requested backporting the
>> symbol change to 19.11, as they don't want to enable experimental APIs
>> in their releases. I replied that it would only be acceptable with
>> aliasing to keep compatibility, and Ferruh very kindly did the work to
>> implement that.
>>
> but, thats part of the agreement, no? You can't always have new features and
> stability at the same time.
>
> I get that this is an odd corner case, because strictly speaking you could waive
> the ABI change that libabigail is reporting, and most application users (like
> OVS) could get away with it, because their application does all the right things
> to make it ok, but I don't think you can make a decsion like this for all users
> based on the request of a single user.
>
> It seems like the right thing is for OVS to augment their build time
> configuration to allow developers to select the ability to use experimental apis
> at compile time, and then the decision is placed in the hands of end users.
So this is not isolated case right ... it is common from API's to graduate
from experimental to mature, we _want_ that to happen, we _want_ APIs to mature.
I am worried what you are proposing will encourage a behavior whereby maintainers
to wait until the declaration of the next major ABI version to make these symbol changes,
causing these kinds of changes to queue up for that release.
And you would have to ask why?
In the case of a reasonably mature API, there will be no functional or signature change - its mature!
So what is the harm of providing an alias back to Experimental until the next ABI version is declared?
So while yes, you are 100% right - experimental mean no guarantees.
But if the API is baked, it is not going to change, so I don't see the harm.
We don't want the unnecessary triumph of policy over pragmatism.
^ permalink raw reply [relevance 8%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
@ 2020-02-13 14:51 4% ` Kusztal, ArkadiuszX
2020-03-16 12:57 8% ` Trahe, Fiona
0 siblings, 1 reply; 200+ results
From: Kusztal, ArkadiuszX @ 2020-02-13 14:51 UTC (permalink / raw)
To: Trahe, Fiona, Thomas Monjalon
Cc: David Marchand, nhorman, bluca, ktraynor, Ray Kinsella, dev,
Akhil Goyal, Yigit, Ferruh, Ananyev, Konstantin, dev,
Anoob Joseph, Richardson, Bruce, Mcnamara, John, dodji,
Andrew Rybchenko, aconole
Hi,
Two comments from me,
> > > The patch we're working on will provide two versions of
> > > rte_cryptodev_info_get(), both call the same PMD function from the
> dev_ops info_get fn ptr.
> > > The default version operates s as normal, the 19.11 version searches
> > > through the list returned by the PMD, looking for sym.aead.algo =
> > > ChaChaPoly, it needs to strip it from
> > the list.
> > > As PMDs just pass a ptr to their capabilities list ( it isn't a
> > > linked list, but an array with an end marker =
> > > RTE_CRYPTODEV_END_OF_CAPABILITIES_LIST) if the API layer detects
> > > Chacha it must allocate some space and store a local copy of the trimmed
> list. This must be stored only once per device.
[Arek] The problem with this solution is that we need to allocate memory.
So the question is how to handle unlikely case of malloc error when we operate inside void function rte_cryptodev_info_get?
And even if we would pass somehow error condition to the caller then what to do is another question.
> >
> > I don't understand what you have to store.
> > Can't you just set the algo to 0 if it is ChaCha?
> [Fiona] it returns a pointer to data in the PMD domain, which the API couldn't
> and shouldn't overwrite, e.g.
> static const struct rte_cryptodev_capabilities qat_gen3_sym_capabilities[]
Should we print user some information
>
> >
> > > This versioning will apply to any PMD which wants to take advantage
> > > of the new API between now and
> > 20.11.
> > >
> > > Note, I expect the ABI checker tools will still complain of ABI
> > > breakage as the LIST_END value will still
> > change.
> >
> > Right, you need to update the ignore list for the tool.
> >
> > > We are also reviewing all other cryptodev APIs in case there is any other
> API which needs versioning.
> > >
> > > Anyone see any problem with this approach?
> >
> > The other issue is with all other functions accepting this enum as input.
> > We should continue returning an error if getting Chacha as input with
> > 19.11 version of these functions.
> > But I would tend to consider this small ABI breakage can be ignored as
> > it is in the error path.
> [Fiona] The QAT PMD tests for and handles this error. I expect other PMDs
> do too.
[Arek] - Yes, it is error path but on the other hand we explicitly specify what value we will return when calling
rte_cryptodev_sym_session_init so caller may expect EINVAL when wrong algorithm value selected (usually it probably will be ENOTSUP).
In this case when setting 3 (LIST_END) on 19.11 app and linking against 20.02 (assuming with Chacha) shared build, caller would get success on return and fully set chacha session,
which will probably result in undefined behavior.
So shouldn't this function be versioned as well then?
Regards,
Arek
^ permalink raw reply [relevance 4%]
* [dpdk-dev] DPDK Release Status Meeting 13/02/2020
@ 2020-02-13 11:20 3% Ferruh Yigit
0 siblings, 0 replies; 200+ results
From: Ferruh Yigit @ 2020-02-13 11:20 UTC (permalink / raw)
To: dpdk-dev; +Cc: Thomas Monjalon
Minutes 13 February 2020
------------------------
Agenda:
* Release Dates
* -rc2 status
* Subtrees
* OvS
Participants:
* Debian/Microsoft
* Intel
* Mellanox
* NXP
* Red Hat
Release Dates
-------------
* v20.02 dates:
* -rc2 is released on 6 February
* https://mails.dpdk.org/archives/dev/2020-February/157366.html
* -rc3 Friday 14 February 2020
* Release Friday 21 February 2020
* Release pushed because of extended PRC holidays
* v20.05 *updated* proposal, please comment:
* Proposal/V1: Wednesday 18 March 2020
* Integration/Merge/RC1: Friday 17 April 2020
* Release: Wednesday 20 May 2020
* The update request is based on delays in 20.02
* Release date postponed more taking into PRC holiday on 1-5 May into account
-rc2 Status
-----------
* -rc2 has been released last week, but it has not been fully tested by Intel
because of extended PRC holiday, expecting to have a test report tomorrow.
No issue reported from Mellanox testing.
* !! Can all vendors/contributors that tested the -rc1/-rc2 please share their
test results, either via email to mail list or in the release status meeting,
to increase our confidence to the release.
* Decided go on with the existing -rc3 plan, but if we find late critical issues
because of delayed testing, we may need to have interim -rc4/-rc5 releases.
* There is still some risk around release date, based on full validation output.
Subtrees
--------
* main
* Sub-trees are not ready yet, will pull them tomorrow for -rc3
* ABI fixes merged
* ACL patches under review
* Windows patches has been merged
* Lots of doc updates for -rc3 & -rc4
* Reminder of deprecation notices, some discussed but not sent yet
* Should be ready for -rc3 tomorrow
* next-net
* Some fixes merged, pulled from sub-trees, will ready for -rc3 end of today
* Will pull from mlx and virtio trees when they are ready today
* af_xdp fix series reviewed by @Xiaolong, will merge today
* Pulling from -brcm & -mrvl trees as well regularly
* next-net-crypto
* Only a couple of patches left for -rc3
* Doc patch for inline protocol
* One test crypto patch waiting reply from @Fiona
* next-net-eventdev
* No update
* next-net-virtio
* There will be a few more fixes for -rc3, waiting @Ivan
* next-net-intel
* Some fixes merged for -rc3
* next-net-mlx
* Some small fixes merged for -rc3, there will be a few more
* LTS
* 19.11.x
* Started working on 19.11.1, going smooth for now
* Will wait 20.02 release for having a -rc1 date
* Proposed to create 19.11 branch in stable tree
* 17.11.10-rc1 released, please test and report results
* https://mails.dpdk.org/archives/dev/2020-January/154915.html
* Got test results from Intel, Red Hat & Mellanox
* QAT regression reported by Intel testing, need help from developers
* mlx reported regressions issue resolved
* Waiting ETA for QAT issue until end of today
* If no update received, will proceed with the release
* 18.11.x
* Started to work on next LTS version 18.11.7
OvS
---
* 2.13 release should be around this week
* The experimental tag dropped in master may remain as experimental in LTS
This means the OvS may have to keep consuming API as experimental, DPDK
discussion thread on the issue:
* https://mails.dpdk.org/archives/dev/2020-February/157149.html
DPDK Release Status Meetings
============================
The DPDK Release Status Meeting is intended for DPDK Committers to discuss
the status of the master tree and sub-trees, and for project managers to
track progress or milestone dates.
The meeting occurs on Thursdays at 8:30 UTC. If you wish to attend just
send an email to "John McNamara <john.mcnamara@intel.com>" for the invite.
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [RFC 1/4] ring: future proof flag settings
2020-02-12 23:08 3% ` [dpdk-dev] [RFC 1/4] ring: future proof flag settings Stephen Hemminger
@ 2020-02-13 11:12 0% ` Ananyev, Konstantin
0 siblings, 0 replies; 200+ results
From: Ananyev, Konstantin @ 2020-02-13 11:12 UTC (permalink / raw)
To: Stephen Hemminger, dev
> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Stephen Hemminger
> Sent: Wednesday, February 12, 2020 11:08 PM
> To: dev@dpdk.org
> Cc: Stephen Hemminger <stephen@networkplumber.org>
> Subject: [dpdk-dev] [RFC 1/4] ring: future proof flag settings
>
> All API's should check that they support the flag values
> passed. If an application passes an invalid flag it could
> cause problems in later ABI.
>
> Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
> ---
> lib/librte_ring/rte_ring.c | 7 +++++++
> 1 file changed, 7 insertions(+)
>
> diff --git a/lib/librte_ring/rte_ring.c b/lib/librte_ring/rte_ring.c
> index 77e5de099b81..6cd8831649ea 100644
> --- a/lib/librte_ring/rte_ring.c
> +++ b/lib/librte_ring/rte_ring.c
> @@ -100,6 +100,13 @@ rte_ring_init(struct rte_ring *r, const char *name, unsigned count,
> RTE_BUILD_BUG_ON((offsetof(struct rte_ring, prod) &
> RTE_CACHE_LINE_MASK) != 0);
>
> + /* future proof flags, only allow supported values */
> + if (flags & ~(RING_F_SP_ENQ | RING_F_SC_DEQ)) {
> + RTE_LOG(ERR, RING,
> + "Unsupported flags requested %d\n", flags);
> + return -EINVAL;
> + }
> +
Good idea, I think, but seems you forgot RING_F_EXACT_SZ.
> /* init the ring structure */
> memset(r, 0, sizeof(*r));
> ret = strlcpy(r->name, name, sizeof(r->name));
> --
> 2.20.1
^ permalink raw reply [relevance 0%]
* [dpdk-dev] [RFC 4/4] cfgfile: check flags value
2020-02-12 23:08 3% [dpdk-dev] [RFC 0/4] Enforce flag checking in API's Stephen Hemminger
` (2 preceding siblings ...)
2020-02-12 23:08 3% ` [dpdk-dev] [RFC 3/4] stack: " Stephen Hemminger
@ 2020-02-12 23:08 3% ` Stephen Hemminger
3 siblings, 0 replies; 200+ results
From: Stephen Hemminger @ 2020-02-12 23:08 UTC (permalink / raw)
To: dev; +Cc: Stephen Hemminger
All API's should check that they support the flag values
passed. If an application passes an invalid flag it could
cause problems in later ABI.
Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
---
lib/librte_cfgfile/rte_cfgfile.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/lib/librte_cfgfile/rte_cfgfile.c b/lib/librte_cfgfile/rte_cfgfile.c
index 9049fd9c2319..cec82c82ee20 100644
--- a/lib/librte_cfgfile/rte_cfgfile.c
+++ b/lib/librte_cfgfile/rte_cfgfile.c
@@ -272,6 +272,10 @@ rte_cfgfile_create(int flags)
int i;
struct rte_cfgfile *cfg;
+ /* future proof flags usage */
+ if (flags & ~(CFG_FLAG_GLOBAL_SECTION | CFG_FLAG_EMPTY_VALUES))
+ return NULL;
+
cfg = malloc(sizeof(*cfg));
if (cfg == NULL)
--
2.20.1
^ permalink raw reply [relevance 3%]
* [dpdk-dev] [RFC 3/4] stack: check flags on creation
2020-02-12 23:08 3% [dpdk-dev] [RFC 0/4] Enforce flag checking in API's Stephen Hemminger
2020-02-12 23:08 3% ` [dpdk-dev] [RFC 1/4] ring: future proof flag settings Stephen Hemminger
2020-02-12 23:08 3% ` [dpdk-dev] [RFC 2/4] hash: check flags on creation Stephen Hemminger
@ 2020-02-12 23:08 3% ` Stephen Hemminger
2020-02-12 23:08 3% ` [dpdk-dev] [RFC 4/4] cfgfile: check flags value Stephen Hemminger
3 siblings, 0 replies; 200+ results
From: Stephen Hemminger @ 2020-02-12 23:08 UTC (permalink / raw)
To: dev; +Cc: Stephen Hemminger
All API's should check that they support the flag values
passed. If an application passes an invalid flag it could
cause problems in later ABI.
Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
---
lib/librte_stack/rte_stack.c | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/lib/librte_stack/rte_stack.c b/lib/librte_stack/rte_stack.c
index d19824f00439..4b09ff30d6bd 100644
--- a/lib/librte_stack/rte_stack.c
+++ b/lib/librte_stack/rte_stack.c
@@ -59,6 +59,11 @@ rte_stack_create(const char *name, unsigned int count, int socket_id,
unsigned int sz;
int ret;
+ if (flags & ~(RTE_STACK_F_LF)) {
+ STACK_LOG_ERR("Unsupported stack flags %#x\n", flags);
+ return NULL;
+ }
+
#ifdef RTE_ARCH_64
RTE_BUILD_BUG_ON(sizeof(struct rte_stack_lf_head) != 16);
#else
--
2.20.1
^ permalink raw reply [relevance 3%]
* [dpdk-dev] [RFC 2/4] hash: check flags on creation
2020-02-12 23:08 3% [dpdk-dev] [RFC 0/4] Enforce flag checking in API's Stephen Hemminger
2020-02-12 23:08 3% ` [dpdk-dev] [RFC 1/4] ring: future proof flag settings Stephen Hemminger
@ 2020-02-12 23:08 3% ` Stephen Hemminger
2020-02-12 23:08 3% ` [dpdk-dev] [RFC 3/4] stack: " Stephen Hemminger
2020-02-12 23:08 3% ` [dpdk-dev] [RFC 4/4] cfgfile: check flags value Stephen Hemminger
3 siblings, 0 replies; 200+ results
From: Stephen Hemminger @ 2020-02-12 23:08 UTC (permalink / raw)
To: dev; +Cc: Stephen Hemminger
All API's should check that they support the flag values
passed. If an application passes an invalid flag it could
cause problems in later ABI.
Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
---
lib/librte_hash/rte_cuckoo_hash.c | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/lib/librte_hash/rte_cuckoo_hash.c b/lib/librte_hash/rte_cuckoo_hash.c
index 6af8ca42e94f..ab9cd91b4955 100644
--- a/lib/librte_hash/rte_cuckoo_hash.c
+++ b/lib/librte_hash/rte_cuckoo_hash.c
@@ -32,6 +32,9 @@
#include "rte_hash.h"
#include "rte_cuckoo_hash.h"
+/* Mask of all flags supported by this version */
+#define RTE_HASH_EXTRA_FLAGS_MASK 0x2f
+
#define FOR_EACH_BUCKET(CURRENT_BKT, START_BUCKET) \
for (CURRENT_BKT = START_BUCKET; \
CURRENT_BKT != NULL; \
@@ -164,6 +167,12 @@ rte_hash_create(const struct rte_hash_parameters *params)
return NULL;
}
+ if (params->extra_flag & ~RTE_HASH_EXTRA_FLAGS_MASK) {
+ rte_errno = EINVAL;
+ RTE_LOG(ERR, HASH, "rte_hash_create: unsupported extra flags\n");
+ return NULL;
+ }
+
/* Validate correct usage of extra options */
if ((params->extra_flag & RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY) &&
(params->extra_flag & RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF)) {
--
2.20.1
^ permalink raw reply [relevance 3%]
* [dpdk-dev] [RFC 1/4] ring: future proof flag settings
2020-02-12 23:08 3% [dpdk-dev] [RFC 0/4] Enforce flag checking in API's Stephen Hemminger
@ 2020-02-12 23:08 3% ` Stephen Hemminger
2020-02-13 11:12 0% ` Ananyev, Konstantin
2020-02-12 23:08 3% ` [dpdk-dev] [RFC 2/4] hash: check flags on creation Stephen Hemminger
` (2 subsequent siblings)
3 siblings, 1 reply; 200+ results
From: Stephen Hemminger @ 2020-02-12 23:08 UTC (permalink / raw)
To: dev; +Cc: Stephen Hemminger
All API's should check that they support the flag values
passed. If an application passes an invalid flag it could
cause problems in later ABI.
Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
---
lib/librte_ring/rte_ring.c | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/lib/librte_ring/rte_ring.c b/lib/librte_ring/rte_ring.c
index 77e5de099b81..6cd8831649ea 100644
--- a/lib/librte_ring/rte_ring.c
+++ b/lib/librte_ring/rte_ring.c
@@ -100,6 +100,13 @@ rte_ring_init(struct rte_ring *r, const char *name, unsigned count,
RTE_BUILD_BUG_ON((offsetof(struct rte_ring, prod) &
RTE_CACHE_LINE_MASK) != 0);
+ /* future proof flags, only allow supported values */
+ if (flags & ~(RING_F_SP_ENQ | RING_F_SC_DEQ)) {
+ RTE_LOG(ERR, RING,
+ "Unsupported flags requested %d\n", flags);
+ return -EINVAL;
+ }
+
/* init the ring structure */
memset(r, 0, sizeof(*r));
ret = strlcpy(r->name, name, sizeof(r->name));
--
2.20.1
^ permalink raw reply [relevance 3%]
* [dpdk-dev] [RFC 0/4] Enforce flag checking in API's
@ 2020-02-12 23:08 3% Stephen Hemminger
2020-02-12 23:08 3% ` [dpdk-dev] [RFC 1/4] ring: future proof flag settings Stephen Hemminger
` (3 more replies)
0 siblings, 4 replies; 200+ results
From: Stephen Hemminger @ 2020-02-12 23:08 UTC (permalink / raw)
To: dev; +Cc: Stephen Hemminger
Since the DPDK api's are lazy about checking for flag values
it means the flags can never be extended in future releases
without causing ABI breakage. This means we end up doing unnecessary
symbol versioning just to work around applications that might
pass in naughty bits.
This is the DPDK analog of the Linux kernel openat() problem.
Openat api was added but since kernel did not check flags it
ended up that another syscall openat2() was necessary before
the flags could be used.
Stephen Hemminger (4):
ring: future proof flag settings
hash: check flags on creation
stack: check flags on creation
cfgfile: check flags value
lib/librte_cfgfile/rte_cfgfile.c | 4 ++++
lib/librte_hash/rte_cuckoo_hash.c | 9 +++++++++
lib/librte_ring/rte_ring.c | 7 +++++++
lib/librte_stack/rte_stack.c | 5 +++++
4 files changed, 25 insertions(+)
--
2.20.1
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH 1/2] common/mlx5: fix ABI version
2020-02-11 19:48 7% [dpdk-dev] [PATCH 1/2] common/mlx5: fix " Matan Azrad
2020-02-11 19:48 7% ` [dpdk-dev] [PATCH 2/2] vdpa/mlx5: " Matan Azrad
2020-02-11 20:31 4% ` [dpdk-dev] [PATCH 1/2] common/mlx5: " Slava Ovsiienko
@ 2020-02-12 8:53 4% ` David Marchand
2 siblings, 0 replies; 200+ results
From: David Marchand @ 2020-02-12 8:53 UTC (permalink / raw)
To: Matan Azrad; +Cc: dev, Viacheslav Ovsiienko, Thomas Monjalon
On Tue, Feb 11, 2020 at 8:48 PM Matan Azrad <matan@mellanox.com> wrote:
>
> Changed the ABI version to 20.0.1.
>
> Fixes: 7b4f1e6bd367 ("common/mlx5: introduce common library")
>
> Signed-off-by: Matan Azrad <matan@mellanox.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
Series applied, thanks.
--
David Marchand
^ permalink raw reply [relevance 4%]
* [dpdk-dev] [PATCH 0/7] vfio/pci: SR-IOV support
@ 2020-02-11 23:05 2% Alex Williamson
2020-02-14 4:57 0% ` Alexey Kardashevskiy
0 siblings, 1 reply; 200+ results
From: Alex Williamson @ 2020-02-11 23:05 UTC (permalink / raw)
To: kvm
Cc: linux-pci, linux-kernel, dev, mtosatti, thomas, bluca,
jerinjacobk, bruce.richardson, cohuck
Given the mostly positive feedback from the RFC[1], here's a new
non-RFC revision. Changes since RFC:
- vfio_device_ops.match semantics refined
- Use helpers for struct pci_dev.physfn to avoid breakage without
CONFIG_PCI_IOV
- Relax to allow SR-IOV configuration changes while PF is opened.
There are potentially interesting use cases here, including
perhaps QEMU emulating an SR-IOV capability and calling out
to a privileged entity to manipulate sriov_numvfs and corral
the resulting devices.
- Retest vfio_device_feature.argsz to include uuid length.
- Add Connie's R-b on 6/7
I still wish we had a solution to make it less opaque to the user
why a VFIO_GROUP_GET_DEVICE_FD() has failed if a VF token is
required, but this is still the best I've been able to come up with.
If there are objections or better ideas, please raise them now.
The synopsis of this series is that we have an ongoing desire to drive
PCIe SR-IOV PFs from userspace with VFIO. There's an immediate need
for this with DPDK drivers and potentially interesting future use
cases in virtualization. We've been reluctant to add this support
previously due to the dependency and trust relationship between the
VF device and PF driver. Minimally the PF driver can induce a denial
of service to the VF, but depending on the specific implementation,
the PF driver might also be responsible for moving data between VFs
or have direct access to the state of the VF, including data or state
otherwise private to the VF or VF driver.
To help resolve these concerns, we introduce a VF token into the VFIO
PCI ABI, which acts as a shared secret key between drivers. The
userspace PF driver is required to set the VF token to a known value
and userspace VF drivers are required to provide the token to access
the VF device. If a PF driver is restarted with VF drivers in use, it
must also provide the current token in order to prevent a rogue
untrusted PF driver from replacing a known driver. The degree to
which this new token is considered secret is left to the userspace
drivers, the kernel intentionally provides no means to retrieve the
current token.
Note that the above token is only required for this new model where
both the PF and VF devices are usable through vfio-pci. Existing
models of VFIO drivers where the PF is used without SR-IOV enabled
or the VF is bound to a userspace driver with an in-kernel, host PF
driver are unaffected.
The latter configuration above also highlights a new inverted scenario
that is now possible, a userspace PF driver with in-kernel VF drivers.
I believe this is a scenario that should be allowed, but should not be
enabled by default. This series includes code to set a default
driver_override for VFs sourced from a vfio-pci user owned PF, such
that the VFs are also bound to vfio-pci. This model is compatible
with tools like driverctl and allows the system administrator to
decide if other bindings should be enabled. The VF token interface
above exists only between vfio-pci PF and VF drivers, once a VF is
bound to another driver, the administrator has effectively pronounced
the device as trusted. The vfio-pci driver will note alternate
binding in dmesg for logging and debugging purposes.
Please review, comment, and test. The example QEMU implementation
provided with the RFC[2] is still current for this version. Thanks,
Alex
[1] https://lore.kernel.org/lkml/158085337582.9445.17682266437583505502.stgit@gimli.home/
[2] https://lore.kernel.org/lkml/20200204161737.34696b91@w520.home/
---
Alex Williamson (7):
vfio: Include optional device match in vfio_device_ops callbacks
vfio/pci: Implement match ops
vfio/pci: Introduce VF token
vfio: Introduce VFIO_DEVICE_FEATURE ioctl and first user
vfio/pci: Add sriov_configure support
vfio/pci: Remove dev_fmt definition
vfio/pci: Cleanup .probe() exit paths
drivers/vfio/pci/vfio_pci.c | 312 ++++++++++++++++++++++++++++++++---
drivers/vfio/pci/vfio_pci_private.h | 10 +
drivers/vfio/vfio.c | 20 ++
include/linux/vfio.h | 4
include/uapi/linux/vfio.h | 37 ++++
5 files changed, 355 insertions(+), 28 deletions(-)
^ permalink raw reply [relevance 2%]
* Re: [dpdk-dev] [PATCH 2/2] vdpa/mlx5: fix ABI version
2020-02-11 19:48 7% ` [dpdk-dev] [PATCH 2/2] vdpa/mlx5: " Matan Azrad
@ 2020-02-11 20:31 4% ` Slava Ovsiienko
0 siblings, 0 replies; 200+ results
From: Slava Ovsiienko @ 2020-02-11 20:31 UTC (permalink / raw)
To: Matan Azrad, dev; +Cc: Thomas Monjalon, David Marchand
> -----Original Message-----
> From: Matan Azrad <matan@mellanox.com>
> Sent: Tuesday, February 11, 2020 21:49
> To: dev@dpdk.org
> Cc: Slava Ovsiienko <viacheslavo@mellanox.com>; Thomas Monjalon
> <thomas@monjalon.net>; David Marchand <david.marchand@redhat.com>
> Subject: [PATCH 2/2] vdpa/mlx5: fix ABI version
>
> Changed the ABI version to 20.0.1.
>
> Fixes: 95276abaaf0a ("vdpa/mlx5: introduce Mellanox vDPA driver")
>
> Signed-off-by: Matan Azrad <matan@mellanox.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
> ---
> drivers/vdpa/mlx5/rte_pmd_mlx5_vdpa_version.map | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/vdpa/mlx5/rte_pmd_mlx5_vdpa_version.map
> b/drivers/vdpa/mlx5/rte_pmd_mlx5_vdpa_version.map
> index 143836e..179f7f1 100644
> --- a/drivers/vdpa/mlx5/rte_pmd_mlx5_vdpa_version.map
> +++ b/drivers/vdpa/mlx5/rte_pmd_mlx5_vdpa_version.map
> @@ -1,3 +1,3 @@
> -DPDK_20.02 {
> +DPDK_20.0.1 {
> local: *;
> };
> --
> 1.8.3.1
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH 1/2] common/mlx5: fix ABI version
2020-02-11 19:48 7% [dpdk-dev] [PATCH 1/2] common/mlx5: fix " Matan Azrad
2020-02-11 19:48 7% ` [dpdk-dev] [PATCH 2/2] vdpa/mlx5: " Matan Azrad
@ 2020-02-11 20:31 4% ` Slava Ovsiienko
2020-02-12 8:53 4% ` David Marchand
2 siblings, 0 replies; 200+ results
From: Slava Ovsiienko @ 2020-02-11 20:31 UTC (permalink / raw)
To: Matan Azrad, dev; +Cc: Thomas Monjalon, David Marchand
> -----Original Message-----
> From: Matan Azrad <matan@mellanox.com>
> Sent: Tuesday, February 11, 2020 21:49
> To: dev@dpdk.org
> Cc: Slava Ovsiienko <viacheslavo@mellanox.com>; Thomas Monjalon
> <thomas@monjalon.net>; David Marchand <david.marchand@redhat.com>
> Subject: [PATCH 1/2] common/mlx5: fix ABI version
>
> Changed the ABI version to 20.0.1.
>
> Fixes: 7b4f1e6bd367 ("common/mlx5: introduce common library")
>
> Signed-off-by: Matan Azrad <matan@mellanox.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
> ---
> drivers/common/mlx5/rte_common_mlx5_version.map | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/common/mlx5/rte_common_mlx5_version.map
> b/drivers/common/mlx5/rte_common_mlx5_version.map
> index b1a2014..aede2a0 100644
> --- a/drivers/common/mlx5/rte_common_mlx5_version.map
> +++ b/drivers/common/mlx5/rte_common_mlx5_version.map
> @@ -1,4 +1,4 @@
> -DPDK_20.02 {
> +DPDK_20.0.1 {
> global:
>
> mlx5_class_get;
> --
> 1.8.3.1
^ permalink raw reply [relevance 4%]
* [dpdk-dev] [PATCH 1/2] common/mlx5: fix ABI version
@ 2020-02-11 19:48 7% Matan Azrad
2020-02-11 19:48 7% ` [dpdk-dev] [PATCH 2/2] vdpa/mlx5: " Matan Azrad
` (2 more replies)
0 siblings, 3 replies; 200+ results
From: Matan Azrad @ 2020-02-11 19:48 UTC (permalink / raw)
To: dev; +Cc: Viacheslav Ovsiienko, Thomas Monjalon, David Marchand
Changed the ABI version to 20.0.1.
Fixes: 7b4f1e6bd367 ("common/mlx5: introduce common library")
Signed-off-by: Matan Azrad <matan@mellanox.com>
---
drivers/common/mlx5/rte_common_mlx5_version.map | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/common/mlx5/rte_common_mlx5_version.map b/drivers/common/mlx5/rte_common_mlx5_version.map
index b1a2014..aede2a0 100644
--- a/drivers/common/mlx5/rte_common_mlx5_version.map
+++ b/drivers/common/mlx5/rte_common_mlx5_version.map
@@ -1,4 +1,4 @@
-DPDK_20.02 {
+DPDK_20.0.1 {
global:
mlx5_class_get;
--
1.8.3.1
^ permalink raw reply [relevance 7%]
* [dpdk-dev] [PATCH 2/2] vdpa/mlx5: fix ABI version
2020-02-11 19:48 7% [dpdk-dev] [PATCH 1/2] common/mlx5: fix " Matan Azrad
@ 2020-02-11 19:48 7% ` Matan Azrad
2020-02-11 20:31 4% ` Slava Ovsiienko
2020-02-11 20:31 4% ` [dpdk-dev] [PATCH 1/2] common/mlx5: " Slava Ovsiienko
2020-02-12 8:53 4% ` David Marchand
2 siblings, 1 reply; 200+ results
From: Matan Azrad @ 2020-02-11 19:48 UTC (permalink / raw)
To: dev; +Cc: Viacheslav Ovsiienko, Thomas Monjalon, David Marchand
Changed the ABI version to 20.0.1.
Fixes: 95276abaaf0a ("vdpa/mlx5: introduce Mellanox vDPA driver")
Signed-off-by: Matan Azrad <matan@mellanox.com>
---
drivers/vdpa/mlx5/rte_pmd_mlx5_vdpa_version.map | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/vdpa/mlx5/rte_pmd_mlx5_vdpa_version.map b/drivers/vdpa/mlx5/rte_pmd_mlx5_vdpa_version.map
index 143836e..179f7f1 100644
--- a/drivers/vdpa/mlx5/rte_pmd_mlx5_vdpa_version.map
+++ b/drivers/vdpa/mlx5/rte_pmd_mlx5_vdpa_version.map
@@ -1,3 +1,3 @@
-DPDK_20.02 {
+DPDK_20.0.1 {
local: *;
};
--
1.8.3.1
^ permalink raw reply [relevance 7%]
* Re: [dpdk-dev] [PATCH 1/1] raw/octeontx2_ep: update the ABI version
2020-02-11 11:27 7% [dpdk-dev] [PATCH 1/1] raw/octeontx2_ep: update the ABI version Mahipal Challa
@ 2020-02-11 14:05 4% ` David Marchand
0 siblings, 0 replies; 200+ results
From: David Marchand @ 2020-02-11 14:05 UTC (permalink / raw)
To: Mahipal Challa
Cc: Thomas Monjalon, dev, Jerin Jacob Kollanukkaran, pathreya,
snilla, venkatn
On Tue, Feb 11, 2020 at 12:29 PM Mahipal Challa <mchalla@marvell.com> wrote:
>
> Changed the ABI version to 20.0.1.
>
> Fixes: 56d46d13f736 ("raw/octeontx2_ep: add build infra and device probe")
>
> Signed-off-by: Mahipal Challa <mchalla@marvell.com>
> ---
> drivers/raw/octeontx2_ep/rte_rawdev_octeontx2_ep_version.map | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/raw/octeontx2_ep/rte_rawdev_octeontx2_ep_version.map b/drivers/raw/octeontx2_ep/rte_rawdev_octeontx2_ep_version.map
> index ff357af..bc8fd6d 100644
> --- a/drivers/raw/octeontx2_ep/rte_rawdev_octeontx2_ep_version.map
> +++ b/drivers/raw/octeontx2_ep/rte_rawdev_octeontx2_ep_version.map
> @@ -1,4 +1,4 @@
> -DPDK_20.02 {
> +DPDK_20.0.1 {
>
> local: *;
> };
> --
> 1.8.3.1
>
Acked-by: David Marchand <david.marchand@redhat.com>
Applied, thanks.
--
David Marchand
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH] common/octeontx2: update version map
2020-02-10 10:27 3% [dpdk-dev] [PATCH] common/octeontx2: update version map Anoob Joseph
@ 2020-02-11 14:04 0% ` David Marchand
0 siblings, 0 replies; 200+ results
From: David Marchand @ 2020-02-11 14:04 UTC (permalink / raw)
To: Anoob Joseph
Cc: Akhil Goyal, Jerin Jacob, Thomas Monjalon, Narayana Prasad, dev
On Mon, Feb 10, 2020 at 11:28 AM Anoob Joseph <anoobj@marvell.com> wrote:
>
> Move symbols added in 20.02 release to DPDK_20.0.1 ABI.
>
> Fixes: d06551535a09 ("common/octeontx2: add security capability routine")
> Fixes: 3fe4d07d1678 ("crypto/octeontx2: enable CPT to share QP with ethdev")
> Fixes: f44e71637755 ("net/octeontx2: add security session operations")
>
> Signed-off-by: Anoob Joseph <anoobj@marvell.com>
Small fix below.
Acked-by: David Marchand <david.marchand@redhat.com>
> ---
> .../common/octeontx2/rte_common_octeontx2_version.map | 17 +++++++++++------
> 1 file changed, 11 insertions(+), 6 deletions(-)
>
> diff --git a/drivers/common/octeontx2/rte_common_octeontx2_version.map b/drivers/common/octeontx2/rte_common_octeontx2_version.map
> index 19a7b19..a25fc69 100644
> --- a/drivers/common/octeontx2/rte_common_octeontx2_version.map
> +++ b/drivers/common/octeontx2/rte_common_octeontx2_version.map
> @@ -5,7 +5,6 @@ DPDK_20.0 {
> otx2_dev_fini;
> otx2_dev_priv_init;
> otx2_disable_irqs;
> - otx2_eth_dev_is_sec_capable;
> otx2_intra_dev_get_cfg;
> otx2_logtype_base;
> otx2_logtype_dpi;
> @@ -28,11 +27,6 @@ DPDK_20.0 {
> otx2_npa_pf_func_get;
> otx2_npa_set_defaults;
> otx2_register_irq;
> - otx2_sec_idev_cfg_init;
> - otx2_sec_idev_tx_cpt_qp_add;
> - otx2_sec_idev_tx_cpt_qp_remove;
> - otx2_sec_idev_tx_cpt_qp_get;
> - otx2_sec_idev_tx_cpt_qp_put;
> otx2_sso_pf_func_get;
> otx2_sso_pf_func_set;
> otx2_unregister_irq;
> @@ -40,6 +34,17 @@ DPDK_20.0 {
> local: *;
> };
>
> +DPDK_20.0.1 {
> + global:
> +
> + otx2_eth_dev_is_sec_capable;
> + otx2_sec_idev_cfg_init;
> + otx2_sec_idev_tx_cpt_qp_add;
> + otx2_sec_idev_tx_cpt_qp_remove;
> + otx2_sec_idev_tx_cpt_qp_get;
> + otx2_sec_idev_tx_cpt_qp_put;
> +};
> +
DPDK_20.0.1 must inherit from DPDK_20.0 node.
I fixed it.
Applied, thanks.
--
David Marchand
^ permalink raw reply [relevance 0%]
* [dpdk-dev] [PATCH 1/1] raw/octeontx2_ep: update the ABI version
@ 2020-02-11 11:27 7% Mahipal Challa
2020-02-11 14:05 4% ` David Marchand
0 siblings, 1 reply; 200+ results
From: Mahipal Challa @ 2020-02-11 11:27 UTC (permalink / raw)
To: thomas, dev; +Cc: jerinj, pathreya, snilla, venkatn
Changed the ABI version to 20.0.1.
Fixes: 56d46d13f736 ("raw/octeontx2_ep: add build infra and device probe")
Signed-off-by: Mahipal Challa <mchalla@marvell.com>
---
drivers/raw/octeontx2_ep/rte_rawdev_octeontx2_ep_version.map | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/raw/octeontx2_ep/rte_rawdev_octeontx2_ep_version.map b/drivers/raw/octeontx2_ep/rte_rawdev_octeontx2_ep_version.map
index ff357af..bc8fd6d 100644
--- a/drivers/raw/octeontx2_ep/rte_rawdev_octeontx2_ep_version.map
+++ b/drivers/raw/octeontx2_ep/rte_rawdev_octeontx2_ep_version.map
@@ -1,4 +1,4 @@
-DPDK_20.02 {
+DPDK_20.0.1 {
local: *;
};
--
1.8.3.1
^ permalink raw reply [relevance 7%]
* Re: [dpdk-dev] [RFC PATCH 0/7] vfio/pci: SR-IOV support
@ 2020-02-11 11:18 0% ` Jerin Jacob
0 siblings, 0 replies; 200+ results
From: Jerin Jacob @ 2020-02-11 11:18 UTC (permalink / raw)
To: Alex Williamson
Cc: kvm, linux-pci, linux-kernel, dpdk-dev, mtosatti,
Thomas Monjalon, Luca Boccassi, Richardson, Bruce, cohuck,
Vamsi Attunuru
On Wed, Feb 5, 2020 at 4:35 AM Alex Williamson
<alex.williamson@redhat.com> wrote:
>
> There seems to be an ongoing desire to use userspace, vfio-based
> drivers for both SR-IOV PF and VF devices. The fundamental issue
> with this concept is that the VF is not fully independent of the PF
> driver. Minimally the PF driver might be able to deny service to the
> VF, VF data paths might be dependent on the state of the PF device,
> or the PF my have some degree of ability to inspect or manipulate the
> VF data. It therefore would seem irresponsible to unleash VFs onto
> the system, managed by a user owned PF.
>
> We address this in a few ways in this series. First, we can use a bus
> notifier and the driver_override facility to make sure VFs are bound
> to the vfio-pci driver by default. This should eliminate the chance
> that a VF is accidentally bound and used by host drivers. We don't
> however remove the ability for a host admin to change this override.
>
> The next issue we need to address is how we let userspace drivers
> opt-in to this participation with the PF driver. We do not want an
> admin to be able to unwittingly assign one of these VFs to a tenant
> that isn't working in collaboration with the PF driver. We could use
> IOMMU grouping, but this seems to push too far towards tightly coupled
> PF and VF drivers. This series introduces a "VF token", implemented
> as a UUID, as a shared secret between PF and VF drivers. The token
> needs to be set by the PF driver and used as part of the device
> matching by the VF driver. Provisions in the code also account for
> restarting the PF driver with active VF drivers, requiring the PF to
> use the current token to re-gain access to the PF.
Thanks Alex for the series. DPDK realizes this use-case through, an out of
tree igb_uio module, for non VFIO devices. Supporting this use case, with
VFIO, will be a great enhancement for DPDK as we are planning to
get rid of out of tree modules any focus only on userspace aspects.
From the DPDK perspective, we have following use-cases
1) VF representer or OVS/vSwitch use cases where
DPDK PF acts as an HW switch to steer traffic to VF
using the rte_flow library backed by HW CAMs.
2) Unlike, other PCI class of devices, Network class of PCIe devices
would have additional
capability on the PF devices such as promiscuous mode support etc
leverage that in DPDK
PF and VF use cases.
That would boil down to the use of the following topology.
a) PF bound to DPDK/VFIO and VF bound to Linux
b) PF bound to DPDK/VFIO and VF bound to DPDK/VFIO
Tested the use case (a) and it works this patch. Tested use case(b), it
works with patch provided both PF and VF under the same application.
Regarding the use case where PF bound to DPDK/VFIO and
VF bound to DPDK/VFIO are _two different_ processes then sharing the UUID
will be a little tricky thing in terms of usage. But if that is the
purpose of bringing
UUID to the equation then it fine.
Overall this series looks good to me. We can test the next non-RFC
series and give
Tested-by by after testing with DPDK.
>
> The above solutions introduce a bit of a modification to the VFIO ABI
> and an additional ABI extension. The modification is that the
> VFIO_GROUP_GET_DEVICE_FD ioctl is specified to require a char string
> from the user providing the device name. For this solution, we extend
> the syntax to allow the device name followed by key/value pairs. In
> this case we add "vf_token=3e7e882e-1daf-417f-ad8d-882eea5ee337", for
> example. These options are expected to be space separated. Matching
> these key/value pairs is entirely left to the vfio bus driver (ex.
> vfio-pci) and the internal ops structure is extended to allow this
> optional support. This extension should be fully backwards compatible
> to existing userspace, such code will simply fail to open these newly
> exposed devices, as intended.
>
> I've been debating whether instead of the above we should allow the
> user to get the device fd as normal, but restrict the interfaces until
> the user authenticates, but I'm afraid this would be a less backwards
> compatible solution. It would be just as unclear to the user why a
> device read/write/mmap/ioctl failed as it might be to why getting the
> device fd could fail. However in the latter case, I believe we do a
> better job of restricting how far userspace code might go before they
> ultimately fail. I'd welcome discussion in the space, and or course
> the extension of the GET_DEVICE_FD string.
>
> Finally, the user needs to be able to set a VF token. I add a
> VFIO_DEVICE_FEATURE ioctl for this that's meant to be reusable for
> getting, setting, and probing arbitrary features of a device.
>
> I'll reply to this cover letter with a very basic example of a QEMU
> update to support this interface, though I haven't found a device yet
> that behaves well with the PF running in one VM with the VF in
> another, or really even just a PF running in a VM with SR-IOV enabled.
> I know these devices exist though, and I suspect QEMU will not be the
> primary user of this support for now, but this behavior reaffirms my
> concerns to prevent mis-use.
>
> Please comment. In particular, does this approach meet the DPDK needs
> for userspace PF and VF drivers, with the hopefully minor hurdle of
> sharing a token between drivers. The token is of course left to
> userspace how to manage, and might be static (and not very secret) for
> a given set of drivers. Thanks,
>
> Alex
>
> ---
>
> Alex Williamson (7):
> vfio: Include optional device match in vfio_device_ops callbacks
> vfio/pci: Implement match ops
> vfio/pci: Introduce VF token
> vfio: Introduce VFIO_DEVICE_FEATURE ioctl and first user
> vfio/pci: Add sriov_configure support
> vfio/pci: Remove dev_fmt definition
> vfio/pci: Cleanup .probe() exit paths
>
>
> drivers/vfio/pci/vfio_pci.c | 315 ++++++++++++++++++++++++++++++++---
> drivers/vfio/pci/vfio_pci_private.h | 10 +
> drivers/vfio/vfio.c | 19 ++
> include/linux/vfio.h | 3
> include/uapi/linux/vfio.h | 37 ++++
> 5 files changed, 356 insertions(+), 28 deletions(-)
>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v1] common/iavf: update the ABI version
2020-02-06 12:04 7% [dpdk-dev] [PATCH v1] common/iavf: update the ABI version Haiyue Wang
@ 2020-02-11 2:08 4% ` Ye Xiaolong
0 siblings, 0 replies; 200+ results
From: Ye Xiaolong @ 2020-02-11 2:08 UTC (permalink / raw)
To: Haiyue Wang; +Cc: dev, thomas
On 02/06, Haiyue Wang wrote:
>The new symbols should be in ABI version 20.0.1.
>
>Fixes: 89214fe915b8 ("net/iavf/base: move to drivers common directory")
>
>Signed-off-by: Haiyue Wang <haiyue.wang@intel.com>
>---
> drivers/common/iavf/rte_common_iavf_version.map | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
>diff --git a/drivers/common/iavf/rte_common_iavf_version.map b/drivers/common/iavf/rte_common_iavf_version.map
>index 1a0839262..2f11d67c0 100644
>--- a/drivers/common/iavf/rte_common_iavf_version.map
>+++ b/drivers/common/iavf/rte_common_iavf_version.map
>@@ -1,4 +1,4 @@
>-DPDK_20.0 {
>+DPDK_20.0.1 {
> global:
>
> iavf_init_adminq;
>--
>2.17.1
>
Acked-by: Xiaolong Ye <xiaolong.ye@intel.com>
Applied to dpdk-next-net-intel, Thanks.
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v7] eal: add manual probing option
2020-02-10 15:27 0% ` Thomas Monjalon
@ 2020-02-10 16:33 0% ` Jerin Jacob
0 siblings, 0 replies; 200+ results
From: Jerin Jacob @ 2020-02-10 16:33 UTC (permalink / raw)
To: Thomas Monjalon
Cc: Jerin Jacob Kollanukkaran, Gaetan Rivet,
Pavan Nikhilesh Bhagavatula, Vamsi Krishna Attunuru,
David Marchand, dpdk-dev, Richardson, Bruce
On Mon, Feb 10, 2020 at 8:57 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> 10/02/2020 15:51, Jerin Jacob:
> > On Tue, Feb 4, 2020 at 9:32 PM Gaetan Rivet <grive@u256.net> wrote:
> > >
> > > On 04/02/2020 16:06, Thomas Monjalon wrote:
> > > > 04/02/2020 13:43, Gaetan Rivet:
> > > >> On 04/02/2020 12:07, Thomas Monjalon wrote:
> > > >>> 04/02/2020 11:03, Gaetan Rivet:
> > > >>>> On 03/02/2020 23:21, Thomas Monjalon wrote:
> > > >>>>> 03/02/2020 06:16, Pavan Nikhilesh Bhagavatula:
> > > >>>>>> @David Marchand @thomas@monjalon.net
> > > >>>>>>
> > > >>>>>> Ping?
> > > >>>>>>
> > > >>>>>> Are there any more changes required for this patch? It's been in queue since last October.
> > > >>>>>
> > > >>>>> Sorry we have not decided whether it is a good idea or not.
> > > >>>>>
> > > >>>>> All changes related to probing are very sensitive,
> > > >>>>> and we know a big refactoring would be better than stacking
> > > >>>>> more and more options and corner cases.
> > > >>>>>
> > > >>>>> As we are busy with ABI stability stuff, we did not allocate
> > > >>>>> enough time to properly think about this feature.
> > > >>>>> Please accept our apologies, and let's consider it as
> > > >>>>> a high priority for 20.05 cycle.
> > > >>>>>
> > > >>>>
> > > >>>> Hello Thomas,
> > > >>>>
> > > >>>> This is unfortunate. I pushed Pavan to accept an alternative implementation of this functionality that was less obtrusive, to make the integration smoother. I took care to alleviate those risks from the common path.
> > > >>>>
> > > >>>> The big refactoring is needed yes, but considering the current path I'm not seeing it happen in 20.05. If that means taking this patch as-is in 20.05 for Marvell users, I'm not sure much is gained from waiting 3 months, except minimal risk avoidance.
> > > >>>
> > > >>>
> > > >>> Yes, life is full of bad decisions and consequences.
> > > >>
> > > >>
> > > >> Ah, yes, but I stand by my initial opinion, the first implementation [1] was riskier and less useful.
> > > >>
> > > >>>
> > > >>> I still think there is a risk in adding new user expectations,
> > > >>> and maintaining some code to workaround unknown issues.
> > > >>>
> > > >>> The real question here is to know why this patch?
> > > >>> Is it to workaround a broken driver?
> > > >>> Or to workaround a broken design in EAL and bus drivers?
> > > >>
> > > >> Two birds - one stone here: OVS needed a way to disable automatic probing cleanly (current workaround seen in multiple deployment is to add a dummy whitelisted device, which will be ignored by the PCI bus --> it sets the bus in whitelist mode but avoid probing anything), and as a bonus this option allows using devices that depends on other devices being probed already (LAG, representors, failsafe, etc).
> > > >>
> > > >> I'm not sure having a dependent-probe by default is good, and that would be a big change.
> > > >>
> > > >> If we are doing the genesis of this patch, the initial motivation should be asked for more details from Marvell people and David for the OVS side.
> > > >>
> > > >> [1]: First proposal:
> > > >> http://mails.dpdk.org/archives/dev/2019-September/144166.html
> > > >> My arguments:
> > > >> http://mails.dpdk.org/archives/dev/2019-September/144564.html
> > > >
> > > >
> > > > OK so there are two needs:
> > > >
> > > > 1/ Better control whitelist/blacklist mode.
> > > > We already know that a rework is needed here.
> > > > Unfortunately neither you or me had time to work on it,
> > > > and others who were interested disappeared.
> > > >
> > > > 2/ Associate ports with equivalent properties in applications.
> > > > This must be done in applications.
> > > > Tweaking the probe order is a hack.
> > > >
> > > >
> > >
> > > An application that want to tightly control the port init order, currently (by doing exactly like me here, hotpluging one by one the ports), would still need the bigger hack that consist in inserting a whitelist PCI devargs with a dummy address, depending on a undocumented PCI bus feature consisting in ignoring matching errors but keeping probing policy from failed devargs processing.
> > >
> > > Instead, with this patch this app can do
> > >
> > > rte_eal_manual_probe_set(1);
> > > rte_eal_init();
> > >
> > > to have the same behavior and be able to hotplug ports as it sees fit.
> > >
> > > You are worried about creating user expectations about this behavior (being forced to replicate in some way the functionality during the rewrite, as I understand it?), but then you are currently forcing users to expect this workaround to exist in the PCI bus, blocking devs from touching it as it will thus break current app configurations. I've seen systemd unit file using this -w dummy flag, as well as the programmatic equivalent. Which is better, to have to rework it cutting short these configs, or to propose beforehand a deprecation path?.
> > >
> > > This rework won't happen in 20.05, nor in the medium term unless you decide to drive this change. This workaround serves three needs (PCI normalization, port congruence and port dependency) in a low-risk implementation.
> >
> > Thomas,
> >
> > What would be the resolution of this? What is your recommendation to
> > fix the issue as you have the concern of this patch?
> >
> > Issue:
> > 1) When l2fwd does the forwarding for simplicity and performance
> > reason it just xor the port to find the destination port to forward.
> > 2) If the adjacent ports are not symmetrical(example: one is 40G and
> > other 10G) then forwarding will drop the packets.
> >
> > So, either
> > a) We need to control the probing order
> >
> > b) Or Application need
> > 1) To track the symmetrical ports and maintain the forwarding table OR
> > 2) Have the command-line option to specify destination port like l3fwd.
> >
> > We can fix it in the application, but do we need to complicate l2fwd?
> > I am fine with that, if that is consensus.
>
> You are describing an application issue,
> that's why I believe it should be fixed in the application.
Thanks for the quick reply and I agree.
>
> Should we have a command line option to configure the forwarding rules
> in the application (2)? I think yes.
> Should we implement an application logic to automatically create
> the best forwarding rules (1)? It would be nice, but anyway,
> I think we need manual config (2) as a fallback.
>
>
> > Thoughts? If you think, there is a rework needed in eal then could you
> > enumerate the items for the rework.
>
> Sorry I don't have time to describe dive into EAL probing and
> enumerate the items to rework.
> The most important issues I remind are:
> - white/blacklist policy is a mess and should be done in a higher layer
> - devargs syntax should allow generic matching (thanks to class awareness and generic syntax)
>
> Starting from these 2 items, we could imagine a generic path to
> disable automatic probing, but I think the l2fwd logic should not
> rely on probing order anyway.
+ Bruce as l2fwd maintainer.
Since in any case, l2fwd needs to be updated, We will focus on l2fwd
change for v20.05 and leaving the fate of this patch to EAL
maintainers.
Let us know, Are we are OK with below change in l2fwd as
- Introduce an array-based port lookup table instead of hardcoding to
xor based lookup.
- if no argument specified fill dest port as xor of source
- If argument is specified override the lookup table with a
user-specified destination port.
>
>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v7] eal: add manual probing option
2020-02-10 14:51 0% ` Jerin Jacob
@ 2020-02-10 15:27 0% ` Thomas Monjalon
2020-02-10 16:33 0% ` Jerin Jacob
0 siblings, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-02-10 15:27 UTC (permalink / raw)
To: Jerin Jacob, Jerin Jacob Kollanukkaran
Cc: Gaetan Rivet, Pavan Nikhilesh Bhagavatula,
Vamsi Krishna Attunuru, David Marchand, dpdk-dev, Richardson,
Bruce
10/02/2020 15:51, Jerin Jacob:
> On Tue, Feb 4, 2020 at 9:32 PM Gaetan Rivet <grive@u256.net> wrote:
> >
> > On 04/02/2020 16:06, Thomas Monjalon wrote:
> > > 04/02/2020 13:43, Gaetan Rivet:
> > >> On 04/02/2020 12:07, Thomas Monjalon wrote:
> > >>> 04/02/2020 11:03, Gaetan Rivet:
> > >>>> On 03/02/2020 23:21, Thomas Monjalon wrote:
> > >>>>> 03/02/2020 06:16, Pavan Nikhilesh Bhagavatula:
> > >>>>>> @David Marchand @thomas@monjalon.net
> > >>>>>>
> > >>>>>> Ping?
> > >>>>>>
> > >>>>>> Are there any more changes required for this patch? It's been in queue since last October.
> > >>>>>
> > >>>>> Sorry we have not decided whether it is a good idea or not.
> > >>>>>
> > >>>>> All changes related to probing are very sensitive,
> > >>>>> and we know a big refactoring would be better than stacking
> > >>>>> more and more options and corner cases.
> > >>>>>
> > >>>>> As we are busy with ABI stability stuff, we did not allocate
> > >>>>> enough time to properly think about this feature.
> > >>>>> Please accept our apologies, and let's consider it as
> > >>>>> a high priority for 20.05 cycle.
> > >>>>>
> > >>>>
> > >>>> Hello Thomas,
> > >>>>
> > >>>> This is unfortunate. I pushed Pavan to accept an alternative implementation of this functionality that was less obtrusive, to make the integration smoother. I took care to alleviate those risks from the common path.
> > >>>>
> > >>>> The big refactoring is needed yes, but considering the current path I'm not seeing it happen in 20.05. If that means taking this patch as-is in 20.05 for Marvell users, I'm not sure much is gained from waiting 3 months, except minimal risk avoidance.
> > >>>
> > >>>
> > >>> Yes, life is full of bad decisions and consequences.
> > >>
> > >>
> > >> Ah, yes, but I stand by my initial opinion, the first implementation [1] was riskier and less useful.
> > >>
> > >>>
> > >>> I still think there is a risk in adding new user expectations,
> > >>> and maintaining some code to workaround unknown issues.
> > >>>
> > >>> The real question here is to know why this patch?
> > >>> Is it to workaround a broken driver?
> > >>> Or to workaround a broken design in EAL and bus drivers?
> > >>
> > >> Two birds - one stone here: OVS needed a way to disable automatic probing cleanly (current workaround seen in multiple deployment is to add a dummy whitelisted device, which will be ignored by the PCI bus --> it sets the bus in whitelist mode but avoid probing anything), and as a bonus this option allows using devices that depends on other devices being probed already (LAG, representors, failsafe, etc).
> > >>
> > >> I'm not sure having a dependent-probe by default is good, and that would be a big change.
> > >>
> > >> If we are doing the genesis of this patch, the initial motivation should be asked for more details from Marvell people and David for the OVS side.
> > >>
> > >> [1]: First proposal:
> > >> http://mails.dpdk.org/archives/dev/2019-September/144166.html
> > >> My arguments:
> > >> http://mails.dpdk.org/archives/dev/2019-September/144564.html
> > >
> > >
> > > OK so there are two needs:
> > >
> > > 1/ Better control whitelist/blacklist mode.
> > > We already know that a rework is needed here.
> > > Unfortunately neither you or me had time to work on it,
> > > and others who were interested disappeared.
> > >
> > > 2/ Associate ports with equivalent properties in applications.
> > > This must be done in applications.
> > > Tweaking the probe order is a hack.
> > >
> > >
> >
> > An application that want to tightly control the port init order, currently (by doing exactly like me here, hotpluging one by one the ports), would still need the bigger hack that consist in inserting a whitelist PCI devargs with a dummy address, depending on a undocumented PCI bus feature consisting in ignoring matching errors but keeping probing policy from failed devargs processing.
> >
> > Instead, with this patch this app can do
> >
> > rte_eal_manual_probe_set(1);
> > rte_eal_init();
> >
> > to have the same behavior and be able to hotplug ports as it sees fit.
> >
> > You are worried about creating user expectations about this behavior (being forced to replicate in some way the functionality during the rewrite, as I understand it?), but then you are currently forcing users to expect this workaround to exist in the PCI bus, blocking devs from touching it as it will thus break current app configurations. I've seen systemd unit file using this -w dummy flag, as well as the programmatic equivalent. Which is better, to have to rework it cutting short these configs, or to propose beforehand a deprecation path?.
> >
> > This rework won't happen in 20.05, nor in the medium term unless you decide to drive this change. This workaround serves three needs (PCI normalization, port congruence and port dependency) in a low-risk implementation.
>
> Thomas,
>
> What would be the resolution of this? What is your recommendation to
> fix the issue as you have the concern of this patch?
>
> Issue:
> 1) When l2fwd does the forwarding for simplicity and performance
> reason it just xor the port to find the destination port to forward.
> 2) If the adjacent ports are not symmetrical(example: one is 40G and
> other 10G) then forwarding will drop the packets.
>
> So, either
> a) We need to control the probing order
>
> b) Or Application need
> 1) To track the symmetrical ports and maintain the forwarding table OR
> 2) Have the command-line option to specify destination port like l3fwd.
>
> We can fix it in the application, but do we need to complicate l2fwd?
> I am fine with that, if that is consensus.
You are describing an application issue,
that's why I believe it should be fixed in the application.
Should we have a command line option to configure the forwarding rules
in the application (2)? I think yes.
Should we implement an application logic to automatically create
the best forwarding rules (1)? It would be nice, but anyway,
I think we need manual config (2) as a fallback.
> Thoughts? If you think, there is a rework needed in eal then could you
> enumerate the items for the rework.
Sorry I don't have time to describe dive into EAL probing and
enumerate the items to rework.
The most important issues I remind are:
- white/blacklist policy is a mess and should be done in a higher layer
- devargs syntax should allow generic matching (thanks to class awareness and generic syntax)
Starting from these 2 items, we could imagine a generic path to
disable automatic probing, but I think the l2fwd logic should not
rely on probing order anyway.
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v7] eal: add manual probing option
@ 2020-02-10 14:51 0% ` Jerin Jacob
2020-02-10 15:27 0% ` Thomas Monjalon
0 siblings, 1 reply; 200+ results
From: Jerin Jacob @ 2020-02-10 14:51 UTC (permalink / raw)
To: Gaetan Rivet
Cc: Thomas Monjalon, Pavan Nikhilesh Bhagavatula,
Vamsi Krishna Attunuru, David Marchand, dpdk-dev,
Jerin Jacob Kollanukkaran, Richardson, Bruce
On Tue, Feb 4, 2020 at 9:32 PM Gaetan Rivet <grive@u256.net> wrote:
>
> On 04/02/2020 16:06, Thomas Monjalon wrote:
> > 04/02/2020 13:43, Gaetan Rivet:
> >> On 04/02/2020 12:07, Thomas Monjalon wrote:
> >>> 04/02/2020 11:03, Gaetan Rivet:
> >>>> On 03/02/2020 23:21, Thomas Monjalon wrote:
> >>>>> 03/02/2020 06:16, Pavan Nikhilesh Bhagavatula:
> >>>>>> @David Marchand @thomas@monjalon.net
> >>>>>>
> >>>>>> Ping?
> >>>>>>
> >>>>>> Are there any more changes required for this patch? It's been in queue since last October.
> >>>>>
> >>>>> Sorry we have not decided whether it is a good idea or not.
> >>>>>
> >>>>> All changes related to probing are very sensitive,
> >>>>> and we know a big refactoring would be better than stacking
> >>>>> more and more options and corner cases.
> >>>>>
> >>>>> As we are busy with ABI stability stuff, we did not allocate
> >>>>> enough time to properly think about this feature.
> >>>>> Please accept our apologies, and let's consider it as
> >>>>> a high priority for 20.05 cycle.
> >>>>>
> >>>>
> >>>> Hello Thomas,
> >>>>
> >>>> This is unfortunate. I pushed Pavan to accept an alternative implementation of this functionality that was less obtrusive, to make the integration smoother. I took care to alleviate those risks from the common path.
> >>>>
> >>>> The big refactoring is needed yes, but considering the current path I'm not seeing it happen in 20.05. If that means taking this patch as-is in 20.05 for Marvell users, I'm not sure much is gained from waiting 3 months, except minimal risk avoidance.
> >>>
> >>>
> >>> Yes, life is full of bad decisions and consequences.
> >>
> >>
> >> Ah, yes, but I stand by my initial opinion, the first implementation [1] was riskier and less useful.
> >>
> >>>
> >>> I still think there is a risk in adding new user expectations,
> >>> and maintaining some code to workaround unknown issues.
> >>>
> >>> The real question here is to know why this patch?
> >>> Is it to workaround a broken driver?
> >>> Or to workaround a broken design in EAL and bus drivers?
> >>
> >> Two birds - one stone here: OVS needed a way to disable automatic probing cleanly (current workaround seen in multiple deployment is to add a dummy whitelisted device, which will be ignored by the PCI bus --> it sets the bus in whitelist mode but avoid probing anything), and as a bonus this option allows using devices that depends on other devices being probed already (LAG, representors, failsafe, etc).
> >>
> >> I'm not sure having a dependent-probe by default is good, and that would be a big change.
> >>
> >> If we are doing the genesis of this patch, the initial motivation should be asked for more details from Marvell people and David for the OVS side.
> >>
> >> [1]: First proposal:
> >> http://mails.dpdk.org/archives/dev/2019-September/144166.html
> >> My arguments:
> >> http://mails.dpdk.org/archives/dev/2019-September/144564.html
> >
> >
> > OK so there are two needs:
> >
> > 1/ Better control whitelist/blacklist mode.
> > We already know that a rework is needed here.
> > Unfortunately neither you or me had time to work on it,
> > and others who were interested disappeared.
> >
> > 2/ Associate ports with equivalent properties in applications.
> > This must be done in applications.
> > Tweaking the probe order is a hack.
> >
> >
>
> An application that want to tightly control the port init order, currently (by doing exactly like me here, hotpluging one by one the ports), would still need the bigger hack that consist in inserting a whitelist PCI devargs with a dummy address, depending on a undocumented PCI bus feature consisting in ignoring matching errors but keeping probing policy from failed devargs processing.
>
> Instead, with this patch this app can do
>
> rte_eal_manual_probe_set(1);
> rte_eal_init();
>
> to have the same behavior and be able to hotplug ports as it sees fit.
>
> You are worried about creating user expectations about this behavior (being forced to replicate in some way the functionality during the rewrite, as I understand it?), but then you are currently forcing users to expect this workaround to exist in the PCI bus, blocking devs from touching it as it will thus break current app configurations. I've seen systemd unit file using this -w dummy flag, as well as the programmatic equivalent. Which is better, to have to rework it cutting short these configs, or to propose beforehand a deprecation path?.
>
> This rework won't happen in 20.05, nor in the medium term unless you decide to drive this change. This workaround serves three needs (PCI normalization, port congruence and port dependency) in a low-risk implementation.
Thomas,
What would be the resolution of this? What is your recommendation to
fix the issue as you have the concern of this patch?
Issue:
1) When l2fwd does the forwarding for simplicity and performance
reason it just xor the port to find the destination port to forward.
2) If the adjacent ports are not symmetrical(example: one is 40G and
other 10G) then forwarding will drop the packets.
So, either
a) We need to control the probing order
b) Or Application need
1) To track the symmetrical ports and maintain the forwarding table OR
2) Have the command-line option to specify destination port like l3fwd.
We can fix it in the application, but do we need to complicate l2fwd?
I am fine with that, if that is consensus.
Thoughts? If you think, there is a rework needed in eal then could you
enumerate the items for the rework.
^ permalink raw reply [relevance 0%]
* [dpdk-dev] [PATCH] common/octeontx2: update version map
@ 2020-02-10 10:27 3% Anoob Joseph
2020-02-11 14:04 0% ` David Marchand
0 siblings, 1 reply; 200+ results
From: Anoob Joseph @ 2020-02-10 10:27 UTC (permalink / raw)
To: Akhil Goyal, Jerin Jacob, Thomas Monjalon
Cc: Anoob Joseph, Narayana Prasad, dev
Move symbols added in 20.02 release to DPDK_20.0.1 ABI.
Fixes: d06551535a09 ("common/octeontx2: add security capability routine")
Fixes: 3fe4d07d1678 ("crypto/octeontx2: enable CPT to share QP with ethdev")
Fixes: f44e71637755 ("net/octeontx2: add security session operations")
Signed-off-by: Anoob Joseph <anoobj@marvell.com>
---
.../common/octeontx2/rte_common_octeontx2_version.map | 17 +++++++++++------
1 file changed, 11 insertions(+), 6 deletions(-)
diff --git a/drivers/common/octeontx2/rte_common_octeontx2_version.map b/drivers/common/octeontx2/rte_common_octeontx2_version.map
index 19a7b19..a25fc69 100644
--- a/drivers/common/octeontx2/rte_common_octeontx2_version.map
+++ b/drivers/common/octeontx2/rte_common_octeontx2_version.map
@@ -5,7 +5,6 @@ DPDK_20.0 {
otx2_dev_fini;
otx2_dev_priv_init;
otx2_disable_irqs;
- otx2_eth_dev_is_sec_capable;
otx2_intra_dev_get_cfg;
otx2_logtype_base;
otx2_logtype_dpi;
@@ -28,11 +27,6 @@ DPDK_20.0 {
otx2_npa_pf_func_get;
otx2_npa_set_defaults;
otx2_register_irq;
- otx2_sec_idev_cfg_init;
- otx2_sec_idev_tx_cpt_qp_add;
- otx2_sec_idev_tx_cpt_qp_remove;
- otx2_sec_idev_tx_cpt_qp_get;
- otx2_sec_idev_tx_cpt_qp_put;
otx2_sso_pf_func_get;
otx2_sso_pf_func_set;
otx2_unregister_irq;
@@ -40,6 +34,17 @@ DPDK_20.0 {
local: *;
};
+DPDK_20.0.1 {
+ global:
+
+ otx2_eth_dev_is_sec_capable;
+ otx2_sec_idev_cfg_init;
+ otx2_sec_idev_tx_cpt_qp_add;
+ otx2_sec_idev_tx_cpt_qp_remove;
+ otx2_sec_idev_tx_cpt_qp_get;
+ otx2_sec_idev_tx_cpt_qp_put;
+};
+
EXPERIMENTAL {
global:
--
2.7.4
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH v2] service: don't walk out of bounds when checking services
@ 2020-02-07 14:27 3% ` Aaron Conole
2020-02-14 16:38 0% ` Kevin Traynor
0 siblings, 1 reply; 200+ results
From: Aaron Conole @ 2020-02-07 14:27 UTC (permalink / raw)
To: Kevin Traynor
Cc: Harry van Haaren, nikhil.rao, David Marchand, dev,
Bruce Richardson, Pavan Nikhilesh, Gage Eads, Thomas Monjalon,
dpdk stable
Kevin Traynor <ktraynor@redhat.com> writes:
> On 20/12/2019 14:43, David Marchand wrote:
>> On Wed, Dec 4, 2019 at 9:34 AM David Marchand <david.marchand@redhat.com> wrote:
>>>
>>> On Wed, Dec 4, 2019 at 9:33 AM David Marchand <david.marchand@redhat.com> wrote:
>>>>
>>>> On Tue, Dec 3, 2019 at 10:15 PM Aaron Conole <aconole@redhat.com> wrote:
>>>>>
>>>>> The service_valid call is used without properly bounds checking the
>>>>> input parameter. Almost all instances of the service_valid call are
>>>>> inside a for() loop that prevents excessive walks, but some of the
>>>>> public APIs don't bounds check and will pass invalid arguments.
>>>>>
>>>>> Prevent this by using SERVICE_GET_OR_ERR_RET where it makes sense,
>>>>> and adding a bounds check to one service_valid() use.
>>>>>
>>>>> Fixes: 8d39d3e237c2 ("service: fix race in service on app lcore function")
>>>>> Fixes: e9139a32f6e8 ("service: add function to run on app lcore")
>>>>> Fixes: e30dd31847d2 ("service: add mechanism for quiescing")
>>> Cc: stable@dpdk.org
>>>
>
> With the commit below, this patch will apply cleanly on 18.11.
>
> Seems ok to me to add below commit, wdyt?
If I'm reading it correctly, the move is for an internal data structure
in librte_eal, so I think it shouldn't be an ABI breakage.
Looks safe to me as well.
> commit e484ccddbe1b41886fef1e445ef2fdfa55086198
> Author: Nikhil Rao <nikhil.rao@intel.com>
> Date: Mon Sep 16 15:31:02 2019 +0530
>
> service: avoid false sharing on core state
>
>
>>>>> Signed-off-by: Aaron Conole <aconole@redhat.com>
>>>>
>>>> Reviewed-by: David Marchand <david.marchand@redhat.com>
>>
>> Applied, thanks.
>>
>>
>> --
>> David Marchand
>>
^ permalink raw reply [relevance 3%]
* [dpdk-dev] [dpdk-announce] release candidate 20.02-rc2
@ 2020-02-06 22:33 4% Thomas Monjalon
0 siblings, 0 replies; 200+ results
From: Thomas Monjalon @ 2020-02-06 22:33 UTC (permalink / raw)
To: announce
A new DPDK release candidate is ready for testing:
https://git.dpdk.org/dpdk/tag/?id=v20.02-rc2
226 patches were integrated.
The release notes so far:
http://doc.dpdk.org/guides/rel_notes/release_20_02.html
Highlights of 20.02-rc2:
- ABI check tooling
- Mellanox vDPA driver
- Marvell OCTEON TX2 inline IPsec
- synchronous CPU crypto API
- CPU crypto based on new libraries (intel-ipsec-mb and AArch64cryptolib)
- reverted Chacha20-Poly1305 algorithm (because of ABI issue)
- event mode in l3fwd example
The -rc3 should include only some bug fixes, simple cleanups, doc
and tooling. We have one week to complete this milestone.
Then one more week (allowing -rc4) should be needed before the release.
The schedule is shifting a bit to let more time to Chinese contributors.
Thank you everyone
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v7 1/2] eal: add API to check if its interrupt context
@ 2020-02-06 15:35 3% ` David Marchand
0 siblings, 0 replies; 200+ results
From: David Marchand @ 2020-02-06 15:35 UTC (permalink / raw)
To: Sunil Kumar Kori; +Cc: Bruce Richardson, dev, Harman Kalra
On Tue, Jan 14, 2020 at 10:05 AM Sunil Kumar Kori <skori@marvell.com> wrote:
>
> From: Harman Kalra <hkalra@marvell.com>
>
> Added an API to check if current execution is in interrupt
> context. This will be helpful to handle nested interrupt cases.
>
> Signed-off-by: Harman Kalra <hkalra@marvell.com>
> Signed-off-by: Sunil Kumar Kori <skori@marvell.com>
> Reviewed-by: Jerin Jacob <jerinj@marvell.com>
> ---
> v7:
> - No changes.
> v6:
> - No changes.
> v5:
> - Fix shared library compilation error
> v4:
> - No changes.
> v3:
> - API Comment is updated as per man page.
> - Scope updated within the library/driver only.
> - Remove experimental tag
> v2:
> - Rebased patch on 19.11-rc4
>
> lib/librte_eal/common/include/rte_eal_interrupts.h | 11 +++++++++++
> lib/librte_eal/freebsd/eal/eal_interrupts.c | 5 +++++
> lib/librte_eal/linux/eal/eal_interrupts.c | 5 +++++
> lib/librte_eal/rte_eal_version.map | 1 +
> 4 files changed, 22 insertions(+)
>
> diff --git a/lib/librte_eal/common/include/rte_eal_interrupts.h b/lib/librte_eal/common/include/rte_eal_interrupts.h
> index b370c0d26..19d1d45ab 100644
> --- a/lib/librte_eal/common/include/rte_eal_interrupts.h
> +++ b/lib/librte_eal/common/include/rte_eal_interrupts.h
> @@ -220,4 +220,15 @@ rte_intr_allow_others(struct rte_intr_handle *intr_handle);
> int
> rte_intr_cap_multiple(struct rte_intr_handle *intr_handle);
>
> +/**
> + * @internal
> + * Check if currently executing in interrupt context
> + *
> + * @return
> + * - non zero in case of interrupt context
> + * - zero in case of process context
> + */
> +int
> +rte_thread_is_intr(void);
> +
> #endif /* _RTE_EAL_INTERRUPTS_H_ */
> diff --git a/lib/librte_eal/freebsd/eal/eal_interrupts.c b/lib/librte_eal/freebsd/eal/eal_interrupts.c
> index f6831b790..ce2a27b4a 100644
> --- a/lib/librte_eal/freebsd/eal/eal_interrupts.c
> +++ b/lib/librte_eal/freebsd/eal/eal_interrupts.c
> @@ -671,3 +671,8 @@ rte_intr_free_epoll_fd(struct rte_intr_handle *intr_handle)
> {
> RTE_SET_USED(intr_handle);
> }
> +
> +int rte_thread_is_intr(void)
> +{
> + return pthread_equal(intr_thread, pthread_self());
> +}
> diff --git a/lib/librte_eal/linux/eal/eal_interrupts.c b/lib/librte_eal/linux/eal/eal_interrupts.c
> index 14ebb108c..cb8e10709 100644
> --- a/lib/librte_eal/linux/eal/eal_interrupts.c
> +++ b/lib/librte_eal/linux/eal/eal_interrupts.c
> @@ -1488,3 +1488,8 @@ rte_intr_cap_multiple(struct rte_intr_handle *intr_handle)
>
> return 0;
> }
> +
> +int rte_thread_is_intr(void)
> +{
> + return pthread_equal(intr_thread, pthread_self());
> +}
> diff --git a/lib/librte_eal/rte_eal_version.map b/lib/librte_eal/rte_eal_version.map
> index e38d02530..06ed2e9dc 100644
> --- a/lib/librte_eal/rte_eal_version.map
> +++ b/lib/librte_eal/rte_eal_version.map
> @@ -91,6 +91,7 @@ DPDK_20.0 {
> rte_intr_free_epoll_fd;
> rte_intr_rx_ctl;
> rte_intr_tls_epfd;
> + rte_thread_is_intr;
> rte_keepalive_create;
> rte_keepalive_dispatch_pings;
> rte_keepalive_mark_alive;
> --
> 2.17.1
We can't put this symbol in the stable ABI and just flag it as
internal in the function description.
Users won't notice this warning.
I moved this as an experimental API and enabled the use of
experimental api in octeontx2 common driver.
We can revisit once we have the rte_internal tag Neil had proposed.
Series applied, thanks.
--
David Marchand
^ permalink raw reply [relevance 3%]
* [dpdk-dev] [PATCH v1] common/iavf: update the ABI version
@ 2020-02-06 12:04 7% Haiyue Wang
2020-02-11 2:08 4% ` Ye Xiaolong
0 siblings, 1 reply; 200+ results
From: Haiyue Wang @ 2020-02-06 12:04 UTC (permalink / raw)
To: dev, xiaolong.ye; +Cc: thomas, Haiyue Wang
The new symbols should be in ABI version 20.0.1.
Fixes: 89214fe915b8 ("net/iavf/base: move to drivers common directory")
Signed-off-by: Haiyue Wang <haiyue.wang@intel.com>
---
drivers/common/iavf/rte_common_iavf_version.map | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/common/iavf/rte_common_iavf_version.map b/drivers/common/iavf/rte_common_iavf_version.map
index 1a0839262..2f11d67c0 100644
--- a/drivers/common/iavf/rte_common_iavf_version.map
+++ b/drivers/common/iavf/rte_common_iavf_version.map
@@ -1,4 +1,4 @@
-DPDK_20.0 {
+DPDK_20.0.1 {
global:
iavf_init_adminq;
--
2.17.1
^ permalink raw reply [relevance 7%]
* [dpdk-dev] DPDK Release Status Meeting 6/02/2020
@ 2020-02-06 10:32 4% Ferruh Yigit
0 siblings, 0 replies; 200+ results
From: Ferruh Yigit @ 2020-02-06 10:32 UTC (permalink / raw)
To: dpdk-dev; +Cc: Thomas Monjalon
Minutes 6 February 2020
-----------------------
Agenda:
* Release Dates
* Subtrees
* OvS
Participants:
* Intel
* Marvell
* Mellanox
* NXP
* Red Hat
Release Dates
-------------
* v20.02 dates:
* -rc2 pushed to *Thursday 6 February 2020*
* -rc3 pushed to *Friday 14 February 2020*
* Release pushed to *Friday 21 February 2020*
* Release pushed because of extended PRC holidays
* v20.05 dates:
* Proposal/V1: Friday 6 March 2020
* Integration/Merge/RC1: Friday 10 April 2020
* Release: Friday 1 May 2020
Subtrees
--------
* main
* Pulled next-net & next-net-crypto
* Dropped two patches from crypto because of ABI breakages
* ccp sha3 & chacha20-poly
* ABI check patches merged
* In travis checks enabled by default
* next-net
* pulled, nothing in backlog for -rc3, there can be fixes
* next-net-crypto
* ipsecgw eventmode can be postponed, Akhil will check
* CPU crypto patches merged
* next-net-eventdev
* There can be a few fixes for -rc3
* next-net-virtio
* Three small fixes for -rc3
* next-net-mlx
* pulled
* LTS
* 17.11.10-rc1 released, please test and report results
* https://mails.dpdk.org/archives/dev/2020-January/154915.html
* Got test results from Intel, Red Hat & Mellanox
* QAT regression reported by Intel testing, need help from developers
* mlx reported regressions, needs more details
* Will wait two more week for above issues clarified
* Rough release date is mid February
* 18.11.6 released
* https://mails.dpdk.org/archives/dev/2020-January/156760.html
OvS
---
* Release on track
* Backporting removing experimental tag under discussion in DPDK
* https://mails.dpdk.org/archives/dev/2020-January/155419.html
DPDK Release Status Meetings
============================
The DPDK Release Status Meeting is intended for DPDK Committers to discuss
the status of the master tree and sub-trees, and for project managers to
track progress or milestone dates.
The meeting occurs on Thursdays at 8:30 UTC. If you wish to attend just
send an email to "John McNamara <john.mcnamara@intel.com>" for the invite.
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v2 1/2] crypto/ccp: sha3 support enabling in ccp
2020-02-05 22:22 3% ` Thomas Monjalon
@ 2020-02-06 9:34 0% ` David Marchand
0 siblings, 0 replies; 200+ results
From: David Marchand @ 2020-02-06 9:34 UTC (permalink / raw)
To: Thomas Monjalon
Cc: Kumar, Ravi1, Sardar, Shamsher singh, dev, Akhil Goyal, dodji
On Wed, Feb 5, 2020 at 11:22 PM Thomas Monjalon <thomas@monjalon.net> wrote:
> 05/02/2020 13:24, Akhil Goyal:
> > > For series,
> > > Acked-by: Ravi Kumar <ravi1.kumar@amd.com>
> > >
> > > >
> > > >From: Sardar Shamsher Singh <Shamshersingh.Sardar@amd.com>
> > > >
> > > >sha3 support enabled in AMD-CCP crypto controller
> > > >
> > > >Signed-off-by: Sardar Shamsher Singh <Shamshersingh.Sardar@amd.com>
> > > >---
> > Change patch title and description as below
> > cryptodev: fix missing SHA3 algo strings
> >
> > SHA3 support was added earlier but algo strings were
> > missing. This patch add the missing strings.
> >
> > Fixes: 1df800f89518 ("crypto/ccp: support SHA3 family")
> > Cc: stable@dpdk.org
> >
> > Signed-off-by: Sardar Shamsher Singh <Shamshersingh.Sardar@amd.com>
> > Acked-by: Ravi Kumar <ravi1.kumar@amd.com>
> > Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
> >
> > Applied to dpdk-next-crypto
>
> Sorry I must drop this patch because it triggers an ABI warning:
> [C]'const char* rte_crypto_auth_algorithm_strings[]' was changed at rte_crypto_sym.h:320:1:
> size of symbol changed from 168 to 232
This is still not clear to me, but here is how I understand the issue.
An exposed array (and its size) ends up in both the shared library and
the final binary data section.
[dmarchan@wsfd-netdev66 dpdk]$ readelf -sW
~/builds/build-gcc-shared/app/dpdk-test-crypto-perf |grep
rte_crypto_auth_algorithm_strings
86: 00000000004141a0 168 OBJECT GLOBAL DEFAULT 24
rte_crypto_auth_algorithm_strings@DPDK_20.0 (4)
308: 00000000004141a0 168 OBJECT GLOBAL DEFAULT 24
rte_crypto_auth_algorithm_strings@@DPDK_20.0
[dmarchan@wsfd-netdev66 dpdk]$ readelf -sW
~/builds/build-gcc-shared/lib/librte_cryptodev.so |grep
rte_crypto_auth_algorithm_strings
57: 000000000000b220 168 OBJECT GLOBAL DEFAULT 23
rte_crypto_auth_algorithm_strings@@DPDK_20.0
158: 000000000000b220 168 OBJECT GLOBAL DEFAULT 23
rte_crypto_auth_algorithm_strings
At runtime, the linker chooses to rewire all access to the final
binary data section, not the shared library local representation.
Now, if we update the array size, the shared library code is built
with the assumption of the increased size.
But at runtime with an "old" binary, the shared library code runs with
a shorter array, with potential out of bound access.
Interesting article:
https://developers.redhat.com/blog/2019/05/06/how-c-array-sizes-become-part-of-the-binary-interface-of-a-library/
--
David Marchand
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v2 1/2] crypto/ccp: sha3 support enabling in ccp
@ 2020-02-05 22:22 3% ` Thomas Monjalon
2020-02-06 9:34 0% ` David Marchand
0 siblings, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-02-05 22:22 UTC (permalink / raw)
To: Kumar, Ravi1, Sardar, Shamsher singh; +Cc: dev, Akhil Goyal, david.marchand
05/02/2020 13:24, Akhil Goyal:
> > For series,
> > Acked-by: Ravi Kumar <ravi1.kumar@amd.com>
> >
> > >
> > >From: Sardar Shamsher Singh <Shamshersingh.Sardar@amd.com>
> > >
> > >sha3 support enabled in AMD-CCP crypto controller
> > >
> > >Signed-off-by: Sardar Shamsher Singh <Shamshersingh.Sardar@amd.com>
> > >---
> Change patch title and description as below
> cryptodev: fix missing SHA3 algo strings
>
> SHA3 support was added earlier but algo strings were
> missing. This patch add the missing strings.
>
> Fixes: 1df800f89518 ("crypto/ccp: support SHA3 family")
> Cc: stable@dpdk.org
>
> Signed-off-by: Sardar Shamsher Singh <Shamshersingh.Sardar@amd.com>
> Acked-by: Ravi Kumar <ravi1.kumar@amd.com>
> Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
>
> Applied to dpdk-next-crypto
Sorry I must drop this patch because it triggers an ABI warning:
[C]'const char* rte_crypto_auth_algorithm_strings[]' was changed at rte_crypto_sym.h:320:1:
size of symbol changed from 168 to 232
I don't understand how the size of this array can impact the application,
but I prefer to stay on the safe side for now.
Please let's discuss this patch in 20.02-rc3 timeframe.
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH v2] build: remove unused function versioning
@ 2020-02-05 20:44 0% ` David Marchand
0 siblings, 0 replies; 200+ results
From: David Marchand @ 2020-02-05 20:44 UTC (permalink / raw)
To: Andrzej Ostruszka
Cc: dev, David Hunt, Bruce Richardson, Vladimir Medvedkin,
Robert Sanford, Erik Gabriel Carrillo, Marcin Baran, dpdk stable
On Thu, Jan 30, 2020 at 5:11 PM David Marchand
<david.marchand@redhat.com> wrote:
>
> On Thu, Jan 30, 2020 at 10:25 AM Andrzej Ostruszka
> <aostruszka@marvell.com> wrote:
> >
> > Timer, LPM and Distributor libraries no longer use function versioning
> > and therefore do not need separate build for static and shared version
> > of libraries.
> >
> > This patch removes use_function_versioning from their meson build files
> > and corresponding include from the sources.
> >
> > Signed-off-by: Andrzej Ostruszka <aostruszka@marvell.com>
> > Acked-by: Bruce Richardson <bruce.richardson@intel.com>
> >
> > Fixes: f2fb215843a9 ("timer: remove deprecated code")
> > Fixes: 6e5b51676176 ("distributor: remove deprecated code")
> > Fixes: c381a8d554b7 ("lpm: remove deprecated code")
> > Cc: marcinx.baran@intel.com
> >
> > ---
> > V2:
> > - removed also includes (updating only meson files causes build
> > failures), and updated commit msg accordingly
>
> Thanks, I had the exact same patch in store (caught when working with
> Olivier on some ABI stuff for mempool).
>
> Cc: stable@dpdk.org
>
> Acked-by: David Marchand <david.marchand@redhat.com>
Applied, thanks.
--
David Marchand
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] buildtools: allow pedantic empty pmdinfo
@ 2020-02-05 17:29 3% ` Thomas Monjalon
0 siblings, 0 replies; 200+ results
From: Thomas Monjalon @ 2020-02-05 17:29 UTC (permalink / raw)
To: Bruce Richardson
Cc: dev, ferruh.yigit, matan, maxime.coquelin, david.marchand
05/02/2020 18:14, Bruce Richardson:
> On Wed, Feb 05, 2020 at 06:07:23PM +0100, Thomas Monjalon wrote:
> > If a driver has no info generated by pmdinfogen,
> > and if this driver is compiled as pedantic,
> > then an failure would occur:
> > drivers/rte_common_mlx5.pmd.c:1: error:
> > ISO C forbids an empty translation unit [-Werror=pedantic]
> > Such error is triggered with the new mlx5 common directory.
> >
> > In order to allow an "empty driver info" compiled in pedantic mode,
> > the script generating .pmd.c file is modified to add a static string
> > unconditionnaly.
> > The minimal generated code is:
> > static __attribute__((unused)) const char *generator =
> > "/path/to/dpdk/buildtools/gen-pmdinfo-cfile.sh";
> >
> > Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
>
> LGTM
>
> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
Applied, thanks, it will allow pulling next-net
without breaking debug compilation for the new ABI tooling.
^ permalink raw reply [relevance 3%]
* [dpdk-dev] [PATCH v3] doc: alias to experimental tag for stable apis
2020-02-05 15:17 17% [dpdk-dev] [PATCH] doc: alias to experimental tag for stable apis Ray Kinsella
@ 2020-02-05 15:43 17% ` Ray Kinsella
2020-02-18 10:46 17% ` [dpdk-dev] [PATCH v4] " Ray Kinsella
2020-02-25 15:35 0% ` [dpdk-dev] [PATCH] " Neil Horman
2 siblings, 0 replies; 200+ results
From: Ray Kinsella @ 2020-02-05 15:43 UTC (permalink / raw)
To: dev; +Cc: john.mcnamara, nhorman, ferruh.yigit, marko.kovacevic, Ray Kinsella
When a maintainer is promoting an API to become part of the next major ABI
version by removing the experimental tag, possibly a few releases in
advance of the declaration of the next ABI version. The maintainer may
choose to offer an alias to the experimental tag, as removing the tag
before the declaration of the next major ABI version, would cause an ABI
breakage for applications using the API.
Signed-off-by: Ray Kinsella <mdr@ashroe.eu>
---
doc/guides/contributing/abi_policy.rst | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/doc/guides/contributing/abi_policy.rst b/doc/guides/contributing/abi_policy.rst
index 05ca959..9a4a102 100644
--- a/doc/guides/contributing/abi_policy.rst
+++ b/doc/guides/contributing/abi_policy.rst
@@ -159,6 +159,11 @@ The requirements for changing the ABI are:
``experimental``, as described in the section on :ref:`Experimental APIs
and Libraries <experimental_apis>`.
+ - In situations where an ``experimental`` API has been stable for some time.
+ When promoting the API to become part of the next ABI version, the
+ maintainer may choose to provide an alias to the ``experimental`` tag, so
+ as not to break consuming applications.
+
#. If a newly proposed API functionally replaces an existing one, when the new
API becomes non-experimental, then the old one is marked with
``__rte_deprecated``.
@@ -317,6 +322,11 @@ not required. Though, an API should remain in experimental state for at least
one release. Thereafter, the normal process of posting patch for review to
mailing list can be followed.
+After the experimental tag has been formally removed, a tree/sub-tree maintainer
+may choose to offer an alias to the experimental tag so as not to break
+applications using the API. This alias can then be dropped at the declaration of
+next major ABI version.
+
Libraries
~~~~~~~~~
--
2.7.4
^ permalink raw reply [relevance 17%]
* [dpdk-dev] [PATCH v2] doc: alias to experimental tag for stable apis
@ 2020-02-05 15:28 17% Ray Kinsella
0 siblings, 0 replies; 200+ results
From: Ray Kinsella @ 2020-02-05 15:28 UTC (permalink / raw)
To: dev; +Cc: john.mcnamara, nhorman, ferruh.yigit, marko.kovacevic, Ray Kinsella
When a maintainer is promoting an API to become part of the next major ABI
version by removing the experimental tag, possibly a few releases in
advance of the declaration of the next ABI version. The maintainer may
choose to offer an alias to the experimental tag, as removing the tag
before the declaration of the next major ABI version, would cause an ABI
breakage for applications using the API.
Signed-off-by: Ray Kinsella <mdr@ashroe.eu>
---
doc/guides/contributing/abi_policy.rst | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/doc/guides/contributing/abi_policy.rst b/doc/guides/contributing/abi_policy.rst
index 05ca959..9a4a102 100644
--- a/doc/guides/contributing/abi_policy.rst
+++ b/doc/guides/contributing/abi_policy.rst
@@ -159,6 +159,11 @@ The requirements for changing the ABI are:
``experimental``, as described in the section on :ref:`Experimental APIs
and Libraries <experimental_apis>`.
+ - In situations where an ``experimental`` API has been stable for some time.
+ When promoting the API to become part of the next ABI version, the
+ maintainer may choose to provide an alias to the ``experimental`` tag, so
+ as not to break consuming applications.
+
#. If a newly proposed API functionally replaces an existing one, when the new
API becomes non-experimental, then the old one is marked with
``__rte_deprecated``.
@@ -317,6 +322,11 @@ not required. Though, an API should remain in experimental state for at least
one release. Thereafter, the normal process of posting patch for review to
mailing list can be followed.
+After the experimental tag has been formally removed, a tree/sub-tree maintainer
+may choose to offer an alias to the experimental tag so as not to break
+applications using the API. This alias can then be dropped at the declaration of
+next major ABI version.
+
Libraries
~~~~~~~~~
--
2.7.4
^ permalink raw reply [relevance 17%]
* Re: [dpdk-dev] [PATCH v3] cryptodev: add chacha20-poly1305 aead algorithm
@ 2020-02-05 15:21 0% ` Thomas Monjalon
0 siblings, 0 replies; 200+ results
From: Thomas Monjalon @ 2020-02-05 15:21 UTC (permalink / raw)
To: Trahe, Fiona
Cc: Anoob Joseph, Kusztal, ArkadiuszX, Akhil Goyal, dev,
Narayana Prasad Raju Athreya, Tejasree Kondoj,
Jerin Jacob Kollanukkaran, bluca, ktraynor, david.marchand,
nhorman, Kinsella, Ray, Yigit, Ferruh, Richardson, Bruce
05/02/2020 15:41, Trahe, Fiona:
> From: Thomas Monjalon <thomas@monjalon.net>
> > 15/01/2020 16:53, Akhil Goyal:
> > > > > This patch adds Chacha20-Poly1305 AEAD algorithm to Cryptodev.
> > > > >
> > > > > Signed-off-by: Arek Kusztal <arkadiuszx.kusztal@intel.com>
> > > > > Acked-by: Fiona Trahe <fiona.trahe@intel.com>
> > > >
> > > > Acked-by: Anoob Joseph <anoobj@marvell.com>
> > > >
> > > Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
> > >
> > > Applied to dpdk-next-crypto
> >
> > As discussed in other email threads,
> > this patch was causing an ABI breakage and had to be reverted:
> > http://git.dpdk.org/dpdk/commit/?id=f5862ae99
> > Sorry, this feature will miss 20.02 release I'm afraid.
> >
> > 2 actions should be taken:
> > - try to re-intoduce the feature with function versioning
> > - improve ABI stability in DPDK 20.11 for future features
>
> [Fiona] Agreed, it's underway.
Any estimation of time?
I expect a lot of discussions on the solution,
especially because it is a primer.
> The QAT ChaChaPoly patchset including the unit tests will also need to be reverted.
> https://patches.dpdk.org/patch/64729/
> https://patches.dpdk.org/patch/64730/
Yes, this is what I did in one commit.
^ permalink raw reply [relevance 0%]
* [dpdk-dev] [PATCH] doc: alias to experimental tag for stable apis
@ 2020-02-05 15:17 17% Ray Kinsella
2020-02-05 15:43 17% ` [dpdk-dev] [PATCH v3] " Ray Kinsella
` (2 more replies)
0 siblings, 3 replies; 200+ results
From: Ray Kinsella @ 2020-02-05 15:17 UTC (permalink / raw)
To: dev; +Cc: john.mcnamara, nhorman, ferruh.yigit, marko.kovacevic, Ray Kinsella
When a maintainer is promoting an API to become part of the next major ABI
version by removing the experimental tag, possibly a few releases in advance of
the declaration of the next ABI version. The maintainer may choose to offer an
alias to the experimental tag, as removing the tag before the declaration of the
next major ABI version, would cause an ABI breakage for applications using the
API.
Signed-off-by: Ray Kinsella <mdr@ashroe.eu>
---
doc/guides/contributing/abi_policy.rst | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/doc/guides/contributing/abi_policy.rst b/doc/guides/contributing/abi_policy.rst
index 05ca959..9a4a102 100644
--- a/doc/guides/contributing/abi_policy.rst
+++ b/doc/guides/contributing/abi_policy.rst
@@ -159,6 +159,11 @@ The requirements for changing the ABI are:
``experimental``, as described in the section on :ref:`Experimental APIs
and Libraries <experimental_apis>`.
+ - In situations where an ``experimental`` API has been stable for some time.
+ When promoting the API to become part of the next ABI version, the
+ maintainer may choose to provide an alias to the ``experimental`` tag, so
+ as not to break consuming applications.
+
#. If a newly proposed API functionally replaces an existing one, when the new
API becomes non-experimental, then the old one is marked with
``__rte_deprecated``.
@@ -317,6 +322,11 @@ not required. Though, an API should remain in experimental state for at least
one release. Thereafter, the normal process of posting patch for review to
mailing list can be followed.
+After the experimental tag has been formally removed, a tree/sub-tree maintainer
+may choose to offer an alias to the experimental tag so as not to break
+applications using the API. This alias can then be dropped at the declaration of
+next major ABI version.
+
Libraries
~~~~~~~~~
--
2.7.4
^ permalink raw reply [relevance 17%]
Results 6201-6400 of ~18000 next (older) | prev (newer) | reverse | sort options + mbox downloads above
-- links below jump to the message on this page --
2019-08-05 12:30 [dpdk-dev] [PATCH] doc: remove major in designation of normal releases Thomas Monjalon
2020-02-13 20:51 0% ` Kevin Traynor
2019-10-25 15:46 [dpdk-dev] [PATCH v5] eal: add manual probing option Gaetan Rivet
2020-02-04 11:07 ` [dpdk-dev] [PATCH v7] " Thomas Monjalon
2020-02-04 12:43 ` Gaetan Rivet
2020-02-04 15:06 ` Thomas Monjalon
2020-02-04 16:02 ` Gaetan Rivet
2020-02-10 14:51 0% ` Jerin Jacob
2020-02-10 15:27 0% ` Thomas Monjalon
2020-02-10 16:33 0% ` Jerin Jacob
2019-11-13 21:43 [dpdk-dev] [PATCH v7 0/1] fbarray: fix duplicated fbarray file in secondary yasufum.o
2019-11-27 8:48 ` [dpdk-dev] [PATCH v8 " Yasufumi Ogawa
2019-11-27 8:48 ` [dpdk-dev] [PATCH v8 1/1] " Yasufumi Ogawa
2019-12-06 10:44 ` Burakov, Anatoly
2020-02-14 7:46 0% ` Yasufumi Ogawa
2020-02-14 15:08 ` David Marchand
2020-02-14 15:29 3% ` Thomas Monjalon
2020-02-17 12:54 0% ` Yasufumi Ogawa
2019-11-26 14:56 [dpdk-dev] [PATCH] service: don't walk out of bounds when checking services Aaron Conole
2019-12-03 21:15 ` [dpdk-dev] [PATCH v2] " Aaron Conole
2019-12-04 8:33 ` David Marchand
2019-12-04 8:34 ` David Marchand
2019-12-20 14:43 ` David Marchand
2020-02-07 12:04 ` Kevin Traynor
2020-02-07 14:27 3% ` Aaron Conole
2020-02-14 16:38 0% ` Kevin Traynor
2019-12-02 15:35 [dpdk-dev] [PATCH 0/4] Extend --lcores to run on cores > RTE_MAX_LCORE David Marchand
2020-01-21 0:24 ` Thomas Monjalon
2020-02-21 8:04 ` Thomas Monjalon
2020-02-21 8:19 ` Song, Keesang
2020-02-21 9:40 ` David Marchand
2020-02-21 14:48 ` Aaron Conole
2020-02-21 16:38 3% ` Stephen Hemminger
2019-12-20 6:56 [dpdk-dev] [PATCH v6 2/2] common/octeontx2: add polling based response mbox message Sunil Kumar Kori
2020-01-14 9:04 ` [dpdk-dev] [PATCH v7 1/2] eal: add API to check if its interrupt context Sunil Kumar Kori
2020-02-06 15:35 3% ` David Marchand
2019-12-20 15:20 [dpdk-dev] [PATCH] add ABI checks David Marchand
2020-02-04 12:44 ` [dpdk-dev] [PATCH v2 4/4] " Trahe, Fiona
2020-02-04 15:52 ` Trahe, Fiona
2020-02-04 15:59 ` Thomas Monjalon
2020-02-04 17:46 ` Trahe, Fiona
2020-02-13 14:51 4% ` Kusztal, ArkadiuszX
2020-03-16 12:57 8% ` Trahe, Fiona
2020-03-16 13:09 4% ` Thomas Monjalon
2020-03-17 13:27 4% ` Kusztal, ArkadiuszX
2020-03-17 15:10 4% ` Thomas Monjalon
2020-03-17 19:10 4% ` Kusztal, ArkadiuszX
2020-01-13 10:43 [dpdk-dev] [PATCH v3] cryptodev: add chacha20-poly1305 aead algorithm Arek Kusztal
2020-02-05 14:28 ` Thomas Monjalon
2020-02-05 14:41 ` Trahe, Fiona
2020-02-05 15:21 0% ` Thomas Monjalon
2020-01-22 10:52 [dpdk-dev] [PATCH v2 1/2] crypto/ccp: sha3 support enabling in ccp ssardar
2020-01-22 11:11 ` Kumar, Ravi1
2020-02-05 12:24 ` Akhil Goyal
2020-02-05 22:22 3% ` Thomas Monjalon
2020-02-06 9:34 0% ` David Marchand
2020-01-29 12:29 [dpdk-dev] [RFC] meter: fix ABI break due to experimental tag removal Ferruh Yigit
2020-02-02 18:53 ` Neil Horman
2020-02-03 12:53 ` Ferruh Yigit
2020-02-04 12:02 ` Neil Horman
2020-02-05 10:04 ` Luca Boccassi
2020-02-05 11:32 ` Neil Horman
2020-02-13 17:40 8% ` Ray Kinsella
2020-02-14 2:40 7% ` Neil Horman
2020-02-14 11:36 7% ` Bruce Richardson
2020-02-14 20:48 4% ` Neil Horman
2020-02-14 21:52 4% ` Bruce Richardson
2020-02-15 13:43 4% ` Neil Horman
2020-02-17 14:23 8% ` Ray Kinsella
2020-02-17 15:37 4% ` Neil Horman
2020-01-29 16:23 [dpdk-dev] [PATCH] build: remove function versioning from meson files Andrzej Ostruszka
2020-01-30 9:25 ` [dpdk-dev] [PATCH v2] build: remove unused function versioning Andrzej Ostruszka
2020-01-30 16:11 ` David Marchand
2020-02-05 20:44 0% ` David Marchand
2020-01-30 14:20 [dpdk-dev] [PATCH] doc: deprecate using MAX values as array size Ferruh Yigit
2020-02-21 10:25 0% ` Ferruh Yigit
2020-02-24 6:18 0% ` Akhil Goyal
2020-02-25 11:03 0% ` Thomas Monjalon
2020-02-24 8:51 0% ` Andrew Rybchenko
2020-02-25 13:52 0% ` David Marchand
2020-02-25 14:53 0% ` David Marchand
2020-01-30 16:58 [dpdk-dev] [PATCH] doc: deprecation notice to move igb_uio to a new repo jerinj
2020-02-24 7:03 0% ` [dpdk-dev] [dpdk-techboard] " Maxime Coquelin
2020-02-04 23:05 [dpdk-dev] [RFC PATCH 0/7] vfio/pci: SR-IOV support Alex Williamson
2020-02-11 11:18 0% ` Jerin Jacob
2020-02-05 15:17 17% [dpdk-dev] [PATCH] doc: alias to experimental tag for stable apis Ray Kinsella
2020-02-05 15:43 17% ` [dpdk-dev] [PATCH v3] " Ray Kinsella
2020-02-18 10:46 17% ` [dpdk-dev] [PATCH v4] " Ray Kinsella
2020-02-21 17:11 0% ` Ferruh Yigit
2020-02-25 14:22 0% ` Kevin Traynor
2020-02-25 15:35 0% ` [dpdk-dev] [PATCH] " Neil Horman
2020-02-25 17:57 0% ` David Marchand
2020-02-05 15:28 17% [dpdk-dev] [PATCH v2] " Ray Kinsella
2020-02-05 17:07 [dpdk-dev] [PATCH] buildtools: allow pedantic empty pmdinfo Thomas Monjalon
2020-02-05 17:14 ` Bruce Richardson
2020-02-05 17:29 3% ` Thomas Monjalon
2020-02-06 10:32 4% [dpdk-dev] DPDK Release Status Meeting 6/02/2020 Ferruh Yigit
2020-02-06 12:04 7% [dpdk-dev] [PATCH v1] common/iavf: update the ABI version Haiyue Wang
2020-02-11 2:08 4% ` Ye Xiaolong
2020-02-06 22:33 4% [dpdk-dev] [dpdk-announce] release candidate 20.02-rc2 Thomas Monjalon
2020-02-10 10:27 3% [dpdk-dev] [PATCH] common/octeontx2: update version map Anoob Joseph
2020-02-11 14:04 0% ` David Marchand
2020-02-11 11:27 7% [dpdk-dev] [PATCH 1/1] raw/octeontx2_ep: update the ABI version Mahipal Challa
2020-02-11 14:05 4% ` David Marchand
2020-02-11 19:48 7% [dpdk-dev] [PATCH 1/2] common/mlx5: fix " Matan Azrad
2020-02-11 19:48 7% ` [dpdk-dev] [PATCH 2/2] vdpa/mlx5: " Matan Azrad
2020-02-11 20:31 4% ` Slava Ovsiienko
2020-02-11 20:31 4% ` [dpdk-dev] [PATCH 1/2] common/mlx5: " Slava Ovsiienko
2020-02-12 8:53 4% ` David Marchand
2020-02-11 23:05 2% [dpdk-dev] [PATCH 0/7] vfio/pci: SR-IOV support Alex Williamson
2020-02-14 4:57 0% ` Alexey Kardashevskiy
2020-02-14 15:27 0% ` Alex Williamson
2020-02-12 23:08 3% [dpdk-dev] [RFC 0/4] Enforce flag checking in API's Stephen Hemminger
2020-02-12 23:08 3% ` [dpdk-dev] [RFC 1/4] ring: future proof flag settings Stephen Hemminger
2020-02-13 11:12 0% ` Ananyev, Konstantin
2020-02-12 23:08 3% ` [dpdk-dev] [RFC 2/4] hash: check flags on creation Stephen Hemminger
2020-02-12 23:08 3% ` [dpdk-dev] [RFC 3/4] stack: " Stephen Hemminger
2020-02-12 23:08 3% ` [dpdk-dev] [RFC 4/4] cfgfile: check flags value Stephen Hemminger
2020-02-13 11:20 3% [dpdk-dev] DPDK Release Status Meeting 13/02/2020 Ferruh Yigit
2020-02-14 5:54 [dpdk-dev] Questions about rte_timer APIs Honnappa Nagarahalli
2020-02-25 23:56 3% ` Honnappa Nagarahalli
2020-02-26 16:57 0% ` Carrillo, Erik G
2020-02-27 16:58 0% ` Honnappa Nagarahalli
2020-02-16 21:58 [dpdk-dev] [dpdk-announce] release candidate 20.02-rc3 Thomas Monjalon
2020-02-20 9:41 4% ` Peng, Yuan
2020-02-17 11:35 [dpdk-dev] [PATCH 0/2] improve Travis coverage Thomas Monjalon
2020-02-18 21:20 ` [dpdk-dev] [PATCH v3 " Thomas Monjalon
2020-02-18 22:36 3% ` David Marchand
2020-02-17 13:59 9% [dpdk-dev] [PATCH] ci: build and use libabigail 1.6 David Marchand
2020-02-17 15:15 0% ` Thomas Monjalon
2020-02-17 18:47 0% ` Aaron Conole
2020-02-18 9:40 3% ` David Marchand
2020-02-18 11:18 0% ` David Marchand
2020-02-18 14:55 0% ` Aaron Conole
2020-02-18 15:07 0% ` David Marchand
2020-02-18 14:29 9% ` [dpdk-dev] [PATCH v2] " David Marchand
2020-02-18 15:46 0% ` Thomas Monjalon
2020-02-18 16:38 0% ` David Marchand
2020-02-19 8:13 0% ` Dodji Seketeli
2020-02-18 20:13 9% ` [dpdk-dev] [PATCH v3] " David Marchand
2020-02-18 20:31 0% ` Aaron Conole
2020-02-18 20:59 0% ` David Marchand
2020-02-17 15:38 3% [dpdk-dev] [PATCH] doc: plan splitting the ethdev ops struct Ferruh Yigit
2020-02-18 5:07 0% ` Jerin Jacob
2020-02-25 12:42 0% ` Ferruh Yigit
2020-02-18 6:01 0% ` Stephen Hemminger
2020-02-21 10:40 0% ` Ferruh Yigit
2020-02-25 10:35 0% ` Andrew Rybchenko
2020-02-25 11:07 3% ` Ananyev, Konstantin
2020-02-25 11:19 0% ` Andrew Rybchenko
2020-02-25 12:28 3% ` Ferruh Yigit
2020-02-25 12:44 3% ` [dpdk-dev] [PATCH v2] " Ferruh Yigit
2020-02-25 15:51 3% ` Andrew Rybchenko
2020-02-25 16:13 3% ` Ferruh Yigit
2020-02-25 16:41 0% ` Andrew Rybchenko
2020-02-25 18:13 0% ` David Marchand
2020-02-25 18:18 0% ` Ferruh Yigit
2020-03-04 9:57 4% ` [dpdk-dev] [PATCH v3] " Ferruh Yigit
2020-02-17 16:13 7% [dpdk-dev] [PATCH] net/ionic: fix ABI version Ferruh Yigit
2020-02-19 12:37 4% ` Ferruh Yigit
2020-02-17 23:15 9% [dpdk-dev] ABI version of experimental libraries Thomas Monjalon
2020-02-17 23:44 11% ` [dpdk-dev] [PATCH] build: fix soname for " Thomas Monjalon
2020-02-18 9:40 0% ` Bruce Richardson
2020-02-18 9:47 0% ` Thomas Monjalon
2020-02-18 9:42 7% ` [dpdk-dev] ABI version of " Bruce Richardson
2020-02-18 9:50 4% ` Thomas Monjalon
2020-02-18 10:36 4% ` Kinsella, Ray
2020-02-20 19:50 8% ` Ferruh Yigit
2020-02-20 19:54 10% ` [dpdk-dev] [PATCH] build: fix experimental library versioning Ferruh Yigit
2020-02-20 22:14 0% ` Luca Boccassi
2020-02-21 12:36 0% ` Ray Kinsella
2020-02-21 15:24 5% ` David Marchand
2020-02-21 15:34 3% ` Ferruh Yigit
2020-02-21 16:41 0% ` David Marchand
2020-02-19 11:43 7% ` [dpdk-dev] ABI version of experimental libraries Neil Horman
2020-02-19 12:43 4% ` Thomas Monjalon
2020-02-19 13:50 4% ` Ray Kinsella
2020-02-21 16:57 4% ` Thomas Monjalon
2020-02-24 9:32 4% ` Ray Kinsella
2020-02-19 21:17 4% ` Neil Horman
2020-02-18 10:46 13% [dpdk-dev] [PATCH] ethdev: add comment to warn of ABI breakage Ciara Power
2020-02-18 11:59 4% ` Ferruh Yigit
2020-02-18 13:13 4% ` Andrew Rybchenko
2020-02-18 14:20 4% ` Thomas Monjalon
2020-02-18 13:39 13% ` [dpdk-dev] [PATCH v2] " Ciara Power
2020-02-18 18:40 4% ` Ferruh Yigit
2020-02-19 18:53 2% [dpdk-dev] [PATCH v2 0/7] vfio/pci: SR-IOV support Alex Williamson
2020-02-25 2:33 0% ` Tian, Kevin
2020-03-05 17:33 3% ` Alex Williamson
2020-03-06 9:21 0% ` Tian, Kevin
2020-03-05 6:38 0% ` Vamsi Krishna Attunuru
2020-02-19 19:41 [dpdk-dev] [PATCH 0/4] Reorganise Travis jobs David Marchand
2020-02-19 19:41 3% ` [dpdk-dev] [PATCH 4/4] ci: reorganise " David Marchand
2020-02-19 21:39 0% ` Aaron Conole
2020-02-20 10:42 0% ` Thomas Monjalon
2020-02-20 12:22 0% ` David Marchand
2020-02-20 14:35 0% ` Aaron Conole
2020-02-20 16:01 0% ` David Marchand
2020-02-20 19:38 0% ` [dpdk-dev] [dpdk-ci] " Jeremy Plsek
2020-02-20 0:18 0% ` [dpdk-dev] " dwilder
2020-02-20 11:07 0% ` Thomas Monjalon
2020-02-20 13:18 [dpdk-dev] [PATCH] lib/cmdline_rdline: increase command line buf size Wisam Jaddo
2020-02-20 14:53 ` [dpdk-dev] [PATCH v3] cmdline: increase maximum line length Wisam Jaddo
2020-02-22 15:28 3% ` David Marchand
2020-02-21 16:10 17% [dpdk-dev] [PATCH] devtools: skip experimental libraries in ABI check David Marchand
2020-02-21 16:13 4% ` Ferruh Yigit
2020-02-21 16:30 4% ` Thomas Monjalon
2020-02-21 16:40 4% ` David Marchand
2020-02-22 21:08 [dpdk-dev] [dpdk-announce] release candidate 20.02-rc4 Thomas Monjalon
2020-02-25 9:58 ` Ali Alnubani
2020-02-25 14:37 4% ` Xu, Qian Q
2020-02-25 14:44 0% ` Thomas Monjalon
2020-02-25 15:27 0% ` Raslan Darawsheh
2020-02-23 10:59 8% [dpdk-dev] [PATCH v1] doc: update release notes for 20.02 John McNamara
2020-02-24 11:35 2% [dpdk-dev] [RFC 0/6] New sync modes for ring Konstantin Ananyev
2020-02-24 16:59 0% ` Stephen Hemminger
2020-02-24 17:59 0% ` Jerin Jacob
2020-02-24 19:35 3% ` Stephen Hemminger
2020-02-24 20:52 0% ` Honnappa Nagarahalli
2020-02-25 11:45 0% ` Ananyev, Konstantin
2020-02-25 13:41 3% ` Ananyev, Konstantin
2020-02-27 10:31 0% ` Jerin Jacob
2020-02-28 0:17 0% ` David Christensen
2020-02-25 0:58 0% ` Honnappa Nagarahalli
2020-02-25 15:14 0% ` Ananyev, Konstantin
2020-02-25 22:59 4% [dpdk-dev] [dpdk-announce] DPDK 20.02 released Thomas Monjalon
2020-02-26 13:38 10% [dpdk-dev] [PATCH] version: 20.05-rc0 David Marchand
2020-02-26 13:55 0% ` Thomas Monjalon
2020-02-26 14:01 0% ` David Marchand
2020-02-27 11:17 3% ` Bruce Richardson
2020-02-27 11:23 3% ` David Marchand
2020-02-27 11:26 4% ` Thomas Monjalon
2020-02-27 11:01 0% ` David Marchand
2020-02-27 11:31 0% ` David Marchand
2020-02-29 10:27 [dpdk-dev] [dpdk v1] net/mlx5: fix possible building error xiangxia.m.yue
2020-03-01 17:09 ` Ali Alnubani
2020-03-02 0:22 3% ` Tonghao Zhang
2020-03-02 7:35 0% ` Ali Alnubani
2020-03-02 8:13 0% ` Tonghao Zhang
2020-03-02 1:57 [dpdk-dev] [PATCH] mempool: sort the rte_mempool_ops by name xiangxia.m.yue
2020-03-06 13:36 ` [dpdk-dev] [PATCH dpdk-dev v3] " xiangxia.m.yue
2020-03-06 13:37 ` Jerin Jacob
2020-03-07 12:51 ` Andrew Rybchenko
2020-03-07 12:54 ` Andrew Rybchenko
2020-03-09 3:01 ` Tonghao Zhang
2020-03-09 8:27 3% ` Olivier Matz
2020-03-09 8:55 0% ` Tonghao Zhang
2020-03-09 9:05 0% ` Olivier Matz
2020-03-09 13:15 0% ` David Marchand
2020-03-16 7:43 0% ` Tonghao Zhang
2020-03-16 7:55 0% ` Olivier Matz
2020-03-02 14:58 [dpdk-dev] [PATCH 00/16] NXP DPAAx fixes and enhancements Hemant Agrawal
2020-03-02 13:01 3% ` David Marchand
2020-03-05 9:06 3% ` Hemant Agrawal (OSS)
2020-03-05 9:09 3% ` David Marchand
2020-03-05 9:19 0% ` Hemant Agrawal (OSS)
2020-03-06 10:12 3% ` David Marchand
2020-03-10 10:36 3% ` Dodji Seketeli
2020-03-02 14:58 ` [dpdk-dev] [PATCH 11/16] net/dpaa: enable Tx queue taildrop Hemant Agrawal
2020-03-03 17:02 ` Ferruh Yigit
2020-03-05 6:49 ` Gagandeep Singh
2020-03-05 14:14 3% ` Ferruh Yigit
2020-03-05 8:53 4% [dpdk-dev] DPDK techboard minutes of February 26 Thomas Monjalon
2020-03-09 14:14 [dpdk-dev] [PATCH v1 0/4] add Intel DCF PMD support Haiyue Wang
2020-03-09 14:14 1% ` [dpdk-dev] [PATCH v1 3/4] net/ice: add the DCF framework Haiyue Wang
2020-03-10 6:50 ` [dpdk-dev] [PATCH v2 0/7] add Intel DCF PMD support Haiyue Wang
2020-03-10 6:50 1% ` [dpdk-dev] [PATCH v2 2/7] net/ice: add the DCF hardware initialization Haiyue Wang
2020-03-16 5:52 ` [dpdk-dev] [PATCH v3 0/7] add Intel DCF PMD support Haiyue Wang
2020-03-16 5:52 1% ` [dpdk-dev] [PATCH v3 2/7] net/ice: add the DCF hardware initialization Haiyue Wang
[not found] <mailman.2366.1582638587.2727.dev@dpdk.org>
[not found] ` <d8993cb75592406a993e3119d45e906a@intel.com>
2020-03-10 2:55 0% ` [dpdk-dev] dev Digest, Vol 288, Issue 27 Zhang, XiX
2020-03-10 6:50 [dpdk-dev] [PATCH] lib/librte_net/rte_ether.h:changed RTE_ETHER_MAX_LEN Muhammad Ahmad
2020-03-10 15:18 3% ` Stephen Hemminger
2020-03-11 21:58 2% [dpdk-dev] [PATCH v3 0/7] vfio/pci: SR-IOV support Alex Williamson
2020-03-19 6:32 0% ` Tian, Kevin
2020-03-18 20:41 3% [dpdk-dev] [PATCH] cryptodev: version rte_cryptodev_info_get function Arek Kusztal
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).