* [dpdk-dev] DPDK Release Status Meeting 13/02/2020
@ 2020-02-13 11:20 3% Ferruh Yigit
0 siblings, 0 replies; 200+ results
From: Ferruh Yigit @ 2020-02-13 11:20 UTC (permalink / raw)
To: dpdk-dev; +Cc: Thomas Monjalon
Minutes 13 February 2020
------------------------
Agenda:
* Release Dates
* -rc2 status
* Subtrees
* OvS
Participants:
* Debian/Microsoft
* Intel
* Mellanox
* NXP
* Red Hat
Release Dates
-------------
* v20.02 dates:
* -rc2 is released on 6 February
* https://mails.dpdk.org/archives/dev/2020-February/157366.html
* -rc3 Friday 14 February 2020
* Release Friday 21 February 2020
* Release pushed because of extended PRC holidays
* v20.05 *updated* proposal, please comment:
* Proposal/V1: Wednesday 18 March 2020
* Integration/Merge/RC1: Friday 17 April 2020
* Release: Wednesday 20 May 2020
* The update request is based on delays in 20.02
* Release date postponed more taking into PRC holiday on 1-5 May into account
-rc2 Status
-----------
* -rc2 has been released last week, but it has not been fully tested by Intel
because of extended PRC holiday, expecting to have a test report tomorrow.
No issue reported from Mellanox testing.
* !! Can all vendors/contributors that tested the -rc1/-rc2 please share their
test results, either via email to mail list or in the release status meeting,
to increase our confidence to the release.
* Decided go on with the existing -rc3 plan, but if we find late critical issues
because of delayed testing, we may need to have interim -rc4/-rc5 releases.
* There is still some risk around release date, based on full validation output.
Subtrees
--------
* main
* Sub-trees are not ready yet, will pull them tomorrow for -rc3
* ABI fixes merged
* ACL patches under review
* Windows patches has been merged
* Lots of doc updates for -rc3 & -rc4
* Reminder of deprecation notices, some discussed but not sent yet
* Should be ready for -rc3 tomorrow
* next-net
* Some fixes merged, pulled from sub-trees, will ready for -rc3 end of today
* Will pull from mlx and virtio trees when they are ready today
* af_xdp fix series reviewed by @Xiaolong, will merge today
* Pulling from -brcm & -mrvl trees as well regularly
* next-net-crypto
* Only a couple of patches left for -rc3
* Doc patch for inline protocol
* One test crypto patch waiting reply from @Fiona
* next-net-eventdev
* No update
* next-net-virtio
* There will be a few more fixes for -rc3, waiting @Ivan
* next-net-intel
* Some fixes merged for -rc3
* next-net-mlx
* Some small fixes merged for -rc3, there will be a few more
* LTS
* 19.11.x
* Started working on 19.11.1, going smooth for now
* Will wait 20.02 release for having a -rc1 date
* Proposed to create 19.11 branch in stable tree
* 17.11.10-rc1 released, please test and report results
* https://mails.dpdk.org/archives/dev/2020-January/154915.html
* Got test results from Intel, Red Hat & Mellanox
* QAT regression reported by Intel testing, need help from developers
* mlx reported regressions issue resolved
* Waiting ETA for QAT issue until end of today
* If no update received, will proceed with the release
* 18.11.x
* Started to work on next LTS version 18.11.7
OvS
---
* 2.13 release should be around this week
* The experimental tag dropped in master may remain as experimental in LTS
This means the OvS may have to keep consuming API as experimental, DPDK
discussion thread on the issue:
* https://mails.dpdk.org/archives/dev/2020-February/157149.html
DPDK Release Status Meetings
============================
The DPDK Release Status Meeting is intended for DPDK Committers to discuss
the status of the master tree and sub-trees, and for project managers to
track progress or milestone dates.
The meeting occurs on Thursdays at 8:30 UTC. If you wish to attend just
send an email to "John McNamara <john.mcnamara@intel.com>" for the invite.
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
@ 2020-02-13 14:51 4% ` Kusztal, ArkadiuszX
2020-03-16 12:57 8% ` Trahe, Fiona
0 siblings, 1 reply; 200+ results
From: Kusztal, ArkadiuszX @ 2020-02-13 14:51 UTC (permalink / raw)
To: Trahe, Fiona, Thomas Monjalon
Cc: David Marchand, nhorman, bluca, ktraynor, Ray Kinsella, dev,
Akhil Goyal, Yigit, Ferruh, Ananyev, Konstantin, dev,
Anoob Joseph, Richardson, Bruce, Mcnamara, John, dodji,
Andrew Rybchenko, aconole
Hi,
Two comments from me,
> > > The patch we're working on will provide two versions of
> > > rte_cryptodev_info_get(), both call the same PMD function from the
> dev_ops info_get fn ptr.
> > > The default version operates s as normal, the 19.11 version searches
> > > through the list returned by the PMD, looking for sym.aead.algo =
> > > ChaChaPoly, it needs to strip it from
> > the list.
> > > As PMDs just pass a ptr to their capabilities list ( it isn't a
> > > linked list, but an array with an end marker =
> > > RTE_CRYPTODEV_END_OF_CAPABILITIES_LIST) if the API layer detects
> > > Chacha it must allocate some space and store a local copy of the trimmed
> list. This must be stored only once per device.
[Arek] The problem with this solution is that we need to allocate memory.
So the question is how to handle unlikely case of malloc error when we operate inside void function rte_cryptodev_info_get?
And even if we would pass somehow error condition to the caller then what to do is another question.
> >
> > I don't understand what you have to store.
> > Can't you just set the algo to 0 if it is ChaCha?
> [Fiona] it returns a pointer to data in the PMD domain, which the API couldn't
> and shouldn't overwrite, e.g.
> static const struct rte_cryptodev_capabilities qat_gen3_sym_capabilities[]
Should we print user some information
>
> >
> > > This versioning will apply to any PMD which wants to take advantage
> > > of the new API between now and
> > 20.11.
> > >
> > > Note, I expect the ABI checker tools will still complain of ABI
> > > breakage as the LIST_END value will still
> > change.
> >
> > Right, you need to update the ignore list for the tool.
> >
> > > We are also reviewing all other cryptodev APIs in case there is any other
> API which needs versioning.
> > >
> > > Anyone see any problem with this approach?
> >
> > The other issue is with all other functions accepting this enum as input.
> > We should continue returning an error if getting Chacha as input with
> > 19.11 version of these functions.
> > But I would tend to consider this small ABI breakage can be ignored as
> > it is in the error path.
> [Fiona] The QAT PMD tests for and handles this error. I expect other PMDs
> do too.
[Arek] - Yes, it is error path but on the other hand we explicitly specify what value we will return when calling
rte_cryptodev_sym_session_init so caller may expect EINVAL when wrong algorithm value selected (usually it probably will be ENOTSUP).
In this case when setting 3 (LIST_END) on 19.11 app and linking against 20.02 (assuming with Chacha) shared build, caller would get success on return and fully set chacha session,
which will probably result in undefined behavior.
So shouldn't this function be versioned as well then?
Regards,
Arek
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [RFC] meter: fix ABI break due to experimental tag removal
@ 2020-02-13 17:40 8% ` Ray Kinsella
2020-02-14 2:40 7% ` Neil Horman
0 siblings, 1 reply; 200+ results
From: Ray Kinsella @ 2020-02-13 17:40 UTC (permalink / raw)
To: Neil Horman, Luca Boccassi
Cc: Ferruh Yigit, Cristian Dumitrescu, Eelco Chaudron, dev,
Thomas Monjalon, David Marchand, Bruce Richardson, Ian Stokes
On 05/02/2020 11:32, Neil Horman wrote:
> On Wed, Feb 05, 2020 at 11:04:29AM +0100, Luca Boccassi wrote:
>> On Tue, 2020-02-04 at 07:02 -0500, Neil Horman wrote:
>>>> But if we can do the versioning in the master, LTS can backport it
>>>> and can have
>>>> mature version of that API in LTS without breaking the existing
>>>> users.
>>>>
>>>
>>> But why bother? The only thing you've changed is the version
>>> tagging. Its ok
>>> to leave that alone in LTS, you just cant change it.
>>>
>>> Thats part of the burden of an LTS release, it will have some drift
>>> from the
>>> upstream head, because you have to keep things stable. So you
>>> stabilize the
>>> upstream ABI version for this API and just never backport it to the
>>> current LTS
>>> release.
>>
>> Hi,
>>
>> A customer (OVS) explicitly and specifically requested backporting the
>> symbol change to 19.11, as they don't want to enable experimental APIs
>> in their releases. I replied that it would only be acceptable with
>> aliasing to keep compatibility, and Ferruh very kindly did the work to
>> implement that.
>>
> but, thats part of the agreement, no? You can't always have new features and
> stability at the same time.
>
> I get that this is an odd corner case, because strictly speaking you could waive
> the ABI change that libabigail is reporting, and most application users (like
> OVS) could get away with it, because their application does all the right things
> to make it ok, but I don't think you can make a decsion like this for all users
> based on the request of a single user.
>
> It seems like the right thing is for OVS to augment their build time
> configuration to allow developers to select the ability to use experimental apis
> at compile time, and then the decision is placed in the hands of end users.
So this is not isolated case right ... it is common from API's to graduate
from experimental to mature, we _want_ that to happen, we _want_ APIs to mature.
I am worried what you are proposing will encourage a behavior whereby maintainers
to wait until the declaration of the next major ABI version to make these symbol changes,
causing these kinds of changes to queue up for that release.
And you would have to ask why?
In the case of a reasonably mature API, there will be no functional or signature change - its mature!
So what is the harm of providing an alias back to Experimental until the next ABI version is declared?
So while yes, you are 100% right - experimental mean no guarantees.
But if the API is baked, it is not going to change, so I don't see the harm.
We don't want the unnecessary triumph of policy over pragmatism.
^ permalink raw reply [relevance 8%]
* Re: [dpdk-dev] [PATCH] doc: remove major in designation of normal releases
@ 2020-02-13 20:51 0% ` Kevin Traynor
0 siblings, 0 replies; 200+ results
From: Kevin Traynor @ 2020-02-13 20:51 UTC (permalink / raw)
To: Thomas Monjalon, John McNamara, Marko Kovacevic
Cc: dev, orika, David Marchand
On 05/08/2019 13:30, Thomas Monjalon wrote:
> The word "major" was used to differentiate with release candidates
> or stable maintenance releases.
> However the word "major" can be understood as "LTS",
> so it is less confusing to avoid this word.
>
> Reported-by: Ori Kam <orika@mellanox.com>
> Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
> ---
> doc/guides/contributing/documentation.rst | 2 +-
> doc/guides/contributing/stable.rst | 2 +-
> 2 files changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/doc/guides/contributing/documentation.rst b/doc/guides/contributing/documentation.rst
> index 27e4b13be..550d8dec2 100644
> --- a/doc/guides/contributing/documentation.rst
> +++ b/doc/guides/contributing/documentation.rst
> @@ -62,7 +62,7 @@ added to by the developer.
>
> The Release Notes document which features have been added in the current and previous releases of DPDK and highlight
> any known issues.
> - The Releases Notes also contain notifications of features that will change ABI compatibility in the next major release.
> + The Releases Notes also contain notifications of features that will change ABI compatibility in the next release.
>
> Developers should include updates to the Release Notes with patch sets that relate to any of the following sections:
>
> diff --git a/doc/guides/contributing/stable.rst b/doc/guides/contributing/stable.rst
> index 6a5eee9bd..24b9e8b7d 100644
> --- a/doc/guides/contributing/stable.rst
> +++ b/doc/guides/contributing/stable.rst
> @@ -25,7 +25,7 @@ Release to indicate longer term support.
> Stable Releases
> ---------------
>
> -Any major release of DPDK can be designated as a Stable Release if a
> +Any release of DPDK can be designated as a Stable Release if a
> maintainer volunteers to maintain it and there is a commitment from major
> contributors to validate it before releases. If a release is to be designated
> as a Stable Release, it should be done by 1 month after the master release.
>
Acked-by: Kevin Traynor <ktraynor@redhat.com>
(Prompted-by: David Marchand <david.marchand@redhat.com> ;-))
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [RFC] meter: fix ABI break due to experimental tag removal
2020-02-13 17:40 8% ` Ray Kinsella
@ 2020-02-14 2:40 7% ` Neil Horman
2020-02-14 11:36 7% ` Bruce Richardson
0 siblings, 1 reply; 200+ results
From: Neil Horman @ 2020-02-14 2:40 UTC (permalink / raw)
To: Ray Kinsella
Cc: Luca Boccassi, Ferruh Yigit, Cristian Dumitrescu, Eelco Chaudron,
dev, Thomas Monjalon, David Marchand, Bruce Richardson,
Ian Stokes
On Thu, Feb 13, 2020 at 05:40:43PM +0000, Ray Kinsella wrote:
>
>
> On 05/02/2020 11:32, Neil Horman wrote:
> > On Wed, Feb 05, 2020 at 11:04:29AM +0100, Luca Boccassi wrote:
> >> On Tue, 2020-02-04 at 07:02 -0500, Neil Horman wrote:
> >>>> But if we can do the versioning in the master, LTS can backport it
> >>>> and can have
> >>>> mature version of that API in LTS without breaking the existing
> >>>> users.
> >>>>
> >>>
> >>> But why bother? The only thing you've changed is the version
> >>> tagging. Its ok
> >>> to leave that alone in LTS, you just cant change it.
> >>>
> >>> Thats part of the burden of an LTS release, it will have some drift
> >>> from the
> >>> upstream head, because you have to keep things stable. So you
> >>> stabilize the
> >>> upstream ABI version for this API and just never backport it to the
> >>> current LTS
> >>> release.
> >>
> >> Hi,
> >>
> >> A customer (OVS) explicitly and specifically requested backporting the
> >> symbol change to 19.11, as they don't want to enable experimental APIs
> >> in their releases. I replied that it would only be acceptable with
> >> aliasing to keep compatibility, and Ferruh very kindly did the work to
> >> implement that.
> >>
> > but, thats part of the agreement, no? You can't always have new features and
> > stability at the same time.
> >
> > I get that this is an odd corner case, because strictly speaking you could waive
> > the ABI change that libabigail is reporting, and most application users (like
> > OVS) could get away with it, because their application does all the right things
> > to make it ok, but I don't think you can make a decsion like this for all users
> > based on the request of a single user.
> >
> > It seems like the right thing is for OVS to augment their build time
> > configuration to allow developers to select the ability to use experimental apis
> > at compile time, and then the decision is placed in the hands of end users.
>
> So this is not isolated case right ... it is common from API's to graduate
> from experimental to mature, we _want_ that to happen, we _want_ APIs to mature.
>
Sure, I can absolutely agree with that, though I would suggest that
the maturity of the ABI is orthogonal to its labeling as such (more on
that below)
> I am worried what you are proposing will encourage a behavior whereby maintainers
> to wait until the declaration of the next major ABI version to make these symbol changes,
> causing these kinds of changes to queue up for that release.
>
I think you're probably right about that, and would make the agrument
that thats perfectly fine (again I'll clarify below)
> And you would have to ask why?
> In the case of a reasonably mature API, there will be no functional or signature change - its mature!
> So what is the harm of providing an alias back to Experimental until the next ABI version is declared?
>
From a philosophical standpoint, there is absoluely no harm, and I don't
think anyone would disagree, the harm comes from the details of the
implementation, as you've noted.
> So while yes, you are 100% right - experimental mean no guarantees.
> But if the API is baked, it is not going to change, so I don't see the harm.
>
> We don't want the unnecessary triumph of policy over pragmatism.
>
I would make the converse argument here. While I agree that when an API
is mature, theres no point in calling it experimental anymore, I would
also suggest that, if an API is mature, and not expected to change,
theres no harm in leaving its version tag as experimental for an LTS
release. This is what I was referring to above. Once an application
developer has done the work to integrate an API from DPDK into its
application, that work is done. If the API remains stable, then I
honestly don't know that they care that the version label is
EXPERIMENTAL versus 20.11 or 20.05 or whatever. They care about the API
being stable more so than its version label. As such, it
seems....reasonable to me to have developers queue their migration of
experimental APIs to official versioned APIs at the next major release
deliniation point.
I'd welcome counter arguments, but it seems pretty natural to me to make
these sorts of changes at the major release mark. People using
experimantal apis have already implicity agreed to manage changes to
them, so if we just hold it stable in an LTS release (and don't update
the version label), its just gravy for them.
Regards
Neil
>
>
>
>
>
>
^ permalink raw reply [relevance 7%]
* Re: [dpdk-dev] [PATCH 0/7] vfio/pci: SR-IOV support
@ 2020-02-14 4:57 0% ` Alexey Kardashevskiy
2020-02-14 15:27 0% ` Alex Williamson
0 siblings, 1 reply; 200+ results
From: Alexey Kardashevskiy @ 2020-02-14 4:57 UTC (permalink / raw)
To: Alex Williamson, kvm
Cc: linux-pci, linux-kernel, dev, mtosatti, thomas, bluca,
jerinjacobk, bruce.richardson, cohuck
On 12/02/2020 10:05, Alex Williamson wrote:
> Given the mostly positive feedback from the RFC[1], here's a new
> non-RFC revision. Changes since RFC:
>
> - vfio_device_ops.match semantics refined
> - Use helpers for struct pci_dev.physfn to avoid breakage without
> CONFIG_PCI_IOV
> - Relax to allow SR-IOV configuration changes while PF is opened.
> There are potentially interesting use cases here, including
> perhaps QEMU emulating an SR-IOV capability and calling out
> to a privileged entity to manipulate sriov_numvfs and corral
> the resulting devices.
> - Retest vfio_device_feature.argsz to include uuid length.
> - Add Connie's R-b on 6/7
>
> I still wish we had a solution to make it less opaque to the user
> why a VFIO_GROUP_GET_DEVICE_FD() has failed if a VF token is
> required, but this is still the best I've been able to come up with.
> If there are objections or better ideas, please raise them now.
>
> The synopsis of this series is that we have an ongoing desire to drive
> PCIe SR-IOV PFs from userspace with VFIO. There's an immediate need
> for this with DPDK drivers and potentially interesting future use
> cases in virtualization. We've been reluctant to add this support
> previously due to the dependency and trust relationship between the
> VF device and PF driver. Minimally the PF driver can induce a denial
> of service to the VF, but depending on the specific implementation,
> the PF driver might also be responsible for moving data between VFs
> or have direct access to the state of the VF, including data or state
> otherwise private to the VF or VF driver.
>
> To help resolve these concerns, we introduce a VF token into the VFIO
> PCI ABI, which acts as a shared secret key between drivers. The
> userspace PF driver is required to set the VF token to a known value
> and userspace VF drivers are required to provide the token to access
> the VF device. If a PF driver is restarted with VF drivers in use, it
> must also provide the current token in order to prevent a rogue
> untrusted PF driver from replacing a known driver. The degree to
> which this new token is considered secret is left to the userspace
> drivers, the kernel intentionally provides no means to retrieve the
> current token.
>
> Note that the above token is only required for this new model where
> both the PF and VF devices are usable through vfio-pci. Existing
> models of VFIO drivers where the PF is used without SR-IOV enabled
> or the VF is bound to a userspace driver with an in-kernel, host PF
> driver are unaffected.
>
> The latter configuration above also highlights a new inverted scenario
> that is now possible, a userspace PF driver with in-kernel VF drivers.
> I believe this is a scenario that should be allowed, but should not be
> enabled by default. This series includes code to set a default
> driver_override for VFs sourced from a vfio-pci user owned PF, such
> that the VFs are also bound to vfio-pci. This model is compatible
> with tools like driverctl and allows the system administrator to
> decide if other bindings should be enabled. The VF token interface
> above exists only between vfio-pci PF and VF drivers, once a VF is
> bound to another driver, the administrator has effectively pronounced
> the device as trusted. The vfio-pci driver will note alternate
> binding in dmesg for logging and debugging purposes.
>
> Please review, comment, and test. The example QEMU implementation
> provided with the RFC[2] is still current for this version. Thanks,
It is a cool feature. One question - what device have you tested it with?
Does not a PF want to control/manage VFs on a PF driver side? I am
thinking of Mellanox CX5 or similar NIC and it acts as an managed
ethernet switch which might want to do something to VFs and VFs may not
work as expected without PF's native driver doing things to it, or this
is not a concern, is it? Thanks,
>
> Alex
>
> [1] https://lore.kernel.org/lkml/158085337582.9445.17682266437583505502.stgit@gimli.home/
> [2] https://lore.kernel.org/lkml/20200204161737.34696b91@w520.home/
> ---
>
> Alex Williamson (7):
> vfio: Include optional device match in vfio_device_ops callbacks
> vfio/pci: Implement match ops
> vfio/pci: Introduce VF token
> vfio: Introduce VFIO_DEVICE_FEATURE ioctl and first user
> vfio/pci: Add sriov_configure support
> vfio/pci: Remove dev_fmt definition
> vfio/pci: Cleanup .probe() exit paths
>
>
> drivers/vfio/pci/vfio_pci.c | 312 ++++++++++++++++++++++++++++++++---
> drivers/vfio/pci/vfio_pci_private.h | 10 +
> drivers/vfio/vfio.c | 20 ++
> include/linux/vfio.h | 4
> include/uapi/linux/vfio.h | 37 ++++
> 5 files changed, 355 insertions(+), 28 deletions(-)
>
--
Alexey
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v8 1/1] fbarray: fix duplicated fbarray file in secondary
@ 2020-02-14 7:46 0% ` Yasufumi Ogawa
0 siblings, 1 reply; 200+ results
From: Yasufumi Ogawa @ 2020-02-14 7:46 UTC (permalink / raw)
To: Burakov, Anatoly, konstantin.ananyev, david.marchand, dev
Cc: yasufumi.ogawa.gy, Yasufumi Ogawa
Hi,
Could I confirm that this patch is going to be merged in 20.02?
Regards,
Yasufumi
On 2019/12/06 19:44, Burakov, Anatoly wrote:
> On 27-Nov-19 8:48 AM, Yasufumi Ogawa wrote:
>> From: Yasufumi Ogawa <ogawa.yasufumi@lab.ntt.co.jp>
>>
>> In secondary_msl_create_walk(), it creates a file for fbarrays with its
>> PID for reserving unique name among secondary processes. However, it
>> does not work if several secondaries run as app containers because each
>> of containerized secondary has PID 1, and failed to reserve unique name
>> other than first one. To reserve unique name in each of containers, use
>> hostname in addition to PID.
>>
>> Signed-off-by: Yasufumi Ogawa <ogawa.yasufumi@lab.ntt.co.jp>
>> ---
>> lib/librte_eal/common/include/rte_fbarray.h | 7 ++++++-
>> lib/librte_eal/linux/eal/eal_memalloc.c | 11 ++++++++---
>> 2 files changed, 14 insertions(+), 4 deletions(-)
>>
>> diff --git a/lib/librte_eal/common/include/rte_fbarray.h
>> b/lib/librte_eal/common/include/rte_fbarray.h
>> index 6dccdbec9..df003b8dc 100644
>> --- a/lib/librte_eal/common/include/rte_fbarray.h
>> +++ b/lib/librte_eal/common/include/rte_fbarray.h
>> @@ -39,7 +39,12 @@ extern "C" {
>> #include <rte_compat.h>
>> #include <rte_rwlock.h>
>> -#define RTE_FBARRAY_NAME_LEN 64
>> +/* Filename of fbarray is defined as a combination of several params
>> + * such as "fbarray_memseg-1048576k-0-0_PID_HOSTNAME".
>> + * The length of string before PID can be 32bytes, and the length of
>> + * PID can be 7bytes maximamly. Final 1 byte is for null terminator.
>> + */
>> +#define RTE_FBARRAY_NAME_LEN (32 + 7 + 1 + HOST_NAME_MAX + 1)
>
> This breaks ABI, but i believe this doesn't break *stable* ABI, so it is
> OK.
>
> Reviewed-by: Anatoly Burakov <anatoly.burakov@intel.com>
>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [RFC] meter: fix ABI break due to experimental tag removal
2020-02-14 2:40 7% ` Neil Horman
@ 2020-02-14 11:36 7% ` Bruce Richardson
2020-02-14 20:48 4% ` Neil Horman
0 siblings, 1 reply; 200+ results
From: Bruce Richardson @ 2020-02-14 11:36 UTC (permalink / raw)
To: Neil Horman
Cc: Ray Kinsella, Luca Boccassi, Ferruh Yigit, Cristian Dumitrescu,
Eelco Chaudron, dev, Thomas Monjalon, David Marchand, Ian Stokes
On Thu, Feb 13, 2020 at 09:40:40PM -0500, Neil Horman wrote:
> On Thu, Feb 13, 2020 at 05:40:43PM +0000, Ray Kinsella wrote:
> >
> >
> > On 05/02/2020 11:32, Neil Horman wrote:
> > > On Wed, Feb 05, 2020 at 11:04:29AM +0100, Luca Boccassi wrote:
> > >> On Tue, 2020-02-04 at 07:02 -0500, Neil Horman wrote:
> > >>>> But if we can do the versioning in the master, LTS can backport it
> > >>>> and can have mature version of that API in LTS without breaking
> > >>>> the existing users.
> > >>>>
> > >>>
> > >>> But why bother? The only thing you've changed is the version
> > >>> tagging. Its ok to leave that alone in LTS, you just cant change
> > >>> it.
> > >>>
> > >>> Thats part of the burden of an LTS release, it will have some drift
> > >>> from the upstream head, because you have to keep things stable. So
> > >>> you stabilize the upstream ABI version for this API and just never
> > >>> backport it to the current LTS release.
> > >>
> > >> Hi,
> > >>
> > >> A customer (OVS) explicitly and specifically requested backporting
> > >> the symbol change to 19.11, as they don't want to enable
> > >> experimental APIs in their releases. I replied that it would only be
> > >> acceptable with aliasing to keep compatibility, and Ferruh very
> > >> kindly did the work to implement that.
> > >>
> > > but, thats part of the agreement, no? You can't always have new
> > > features and stability at the same time.
> > >
> > > I get that this is an odd corner case, because strictly speaking you
> > > could waive the ABI change that libabigail is reporting, and most
> > > application users (like OVS) could get away with it, because their
> > > application does all the right things to make it ok, but I don't
> > > think you can make a decsion like this for all users based on the
> > > request of a single user.
> > >
> > > It seems like the right thing is for OVS to augment their build time
> > > configuration to allow developers to select the ability to use
> > > experimental apis at compile time, and then the decision is placed in
> > > the hands of end users.
> >
> > So this is not isolated case right ... it is common from API's to
> > graduate from experimental to mature, we _want_ that to happen, we
> > _want_ APIs to mature.
> >
> Sure, I can absolutely agree with that, though I would suggest that the
> maturity of the ABI is orthogonal to its labeling as such (more on that
> below)
>
> > I am worried what you are proposing will encourage a behavior whereby
> > maintainers to wait until the declaration of the next major ABI version
> > to make these symbol changes, causing these kinds of changes to queue
> > up for that release.
> >
> I think you're probably right about that, and would make the agrument
> that thats perfectly fine (again I'll clarify below)
>
> > And you would have to ask why? In the case of a reasonably mature API,
> > there will be no functional or signature change - its mature! So what
> > is the harm of providing an alias back to Experimental until the next
> > ABI version is declared?
> >
> From a philosophical standpoint, there is absoluely no harm, and I don't
> think anyone would disagree, the harm comes from the details of the
> implementation, as you've noted.
>
> > So while yes, you are 100% right - experimental mean no guarantees.
> > But if the API is baked, it is not going to change, so I don't see the
> > harm.
> >
> > We don't want the unnecessary triumph of policy over pragmatism.
> >
> I would make the converse argument here. While I agree that when an API
> is mature, theres no point in calling it experimental anymore, I would
> also suggest that, if an API is mature, and not expected to change,
> theres no harm in leaving its version tag as experimental for an LTS
> release. This is what I was referring to above. Once an application
> developer has done the work to integrate an API from DPDK into its
> application, that work is done. If the API remains stable, then I
> honestly don't know that they care that the version label is EXPERIMENTAL
> versus 20.11 or 20.05 or whatever. They care about the API being stable
> more so than its version label. As such, it seems....reasonable to me to
> have developers queue their migration of experimental APIs to official
> versioned APIs at the next major release deliniation point.
>
> I'd welcome counter arguments, but it seems pretty natural to me to make
> these sorts of changes at the major release mark. People using
> experimantal apis have already implicity agreed to manage changes to
> them, so if we just hold it stable in an LTS release (and don't update
> the version label), its just gravy for them.
>
The counter argument that I see is that while the experimental tag remains
in place the user has no way to know that an API is stable or not, and so
in many projects cannot use the API. If for example an API that is
experimental in 19.11 is unchanged through 20.05 at which point we decide
to promote it to stable. Once the change to the exp tag it is made, any
users of 19.11 can then see that it is an unchanged and now-stable ABI and
can start using it in their software, if they wish, without having to wait
for the 20.11 release. Changing the tag early allows ABIs to be potentially
used immediately.
Regards,
/Bruce
^ permalink raw reply [relevance 7%]
* Re: [dpdk-dev] [PATCH 0/7] vfio/pci: SR-IOV support
2020-02-14 4:57 0% ` Alexey Kardashevskiy
@ 2020-02-14 15:27 0% ` Alex Williamson
0 siblings, 0 replies; 200+ results
From: Alex Williamson @ 2020-02-14 15:27 UTC (permalink / raw)
To: Alexey Kardashevskiy
Cc: kvm, linux-pci, linux-kernel, dev, mtosatti, thomas, bluca,
jerinjacobk, bruce.richardson, cohuck
On Fri, 14 Feb 2020 15:57:04 +1100
Alexey Kardashevskiy <aik@ozlabs.ru> wrote:
> On 12/02/2020 10:05, Alex Williamson wrote:
> > Given the mostly positive feedback from the RFC[1], here's a new
> > non-RFC revision. Changes since RFC:
> >
> > - vfio_device_ops.match semantics refined
> > - Use helpers for struct pci_dev.physfn to avoid breakage without
> > CONFIG_PCI_IOV
> > - Relax to allow SR-IOV configuration changes while PF is opened.
> > There are potentially interesting use cases here, including
> > perhaps QEMU emulating an SR-IOV capability and calling out
> > to a privileged entity to manipulate sriov_numvfs and corral
> > the resulting devices.
> > - Retest vfio_device_feature.argsz to include uuid length.
> > - Add Connie's R-b on 6/7
> >
> > I still wish we had a solution to make it less opaque to the user
> > why a VFIO_GROUP_GET_DEVICE_FD() has failed if a VF token is
> > required, but this is still the best I've been able to come up with.
> > If there are objections or better ideas, please raise them now.
> >
> > The synopsis of this series is that we have an ongoing desire to drive
> > PCIe SR-IOV PFs from userspace with VFIO. There's an immediate need
> > for this with DPDK drivers and potentially interesting future use
> > cases in virtualization. We've been reluctant to add this support
> > previously due to the dependency and trust relationship between the
> > VF device and PF driver. Minimally the PF driver can induce a denial
> > of service to the VF, but depending on the specific implementation,
> > the PF driver might also be responsible for moving data between VFs
> > or have direct access to the state of the VF, including data or state
> > otherwise private to the VF or VF driver.
> >
> > To help resolve these concerns, we introduce a VF token into the VFIO
> > PCI ABI, which acts as a shared secret key between drivers. The
> > userspace PF driver is required to set the VF token to a known value
> > and userspace VF drivers are required to provide the token to access
> > the VF device. If a PF driver is restarted with VF drivers in use, it
> > must also provide the current token in order to prevent a rogue
> > untrusted PF driver from replacing a known driver. The degree to
> > which this new token is considered secret is left to the userspace
> > drivers, the kernel intentionally provides no means to retrieve the
> > current token.
> >
> > Note that the above token is only required for this new model where
> > both the PF and VF devices are usable through vfio-pci. Existing
> > models of VFIO drivers where the PF is used without SR-IOV enabled
> > or the VF is bound to a userspace driver with an in-kernel, host PF
> > driver are unaffected.
> >
> > The latter configuration above also highlights a new inverted scenario
> > that is now possible, a userspace PF driver with in-kernel VF drivers.
> > I believe this is a scenario that should be allowed, but should not be
> > enabled by default. This series includes code to set a default
> > driver_override for VFs sourced from a vfio-pci user owned PF, such
> > that the VFs are also bound to vfio-pci. This model is compatible
> > with tools like driverctl and allows the system administrator to
> > decide if other bindings should be enabled. The VF token interface
> > above exists only between vfio-pci PF and VF drivers, once a VF is
> > bound to another driver, the administrator has effectively pronounced
> > the device as trusted. The vfio-pci driver will note alternate
> > binding in dmesg for logging and debugging purposes.
> >
> > Please review, comment, and test. The example QEMU implementation
> > provided with the RFC[2] is still current for this version. Thanks,
>
>
> It is a cool feature. One question - what device have you tested it with?
>
> Does not a PF want to control/manage VFs on a PF driver side? I am
> thinking of Mellanox CX5 or similar NIC and it acts as an managed
> ethernet switch which might want to do something to VFs and VFs may not
> work as expected without PF's native driver doing things to it, or this
> is not a concern, is it? Thanks,
TBH, I'm starting with the premise that a userspace PF driver already
works. The DPDK folks have produced some "interesting" code that
allows SR-IOV to be enabled on a PF underneath vfio-pci. There's also
a non-upstream igb-uio driver associated with DPDK that seems to be
recommended for SR-IOV PF driver use cases, particularly for an FPGA
device. The testing I've done, and what's provided by the QEMU patch I
reference, is really only unit testing the vf_token support and
DEVICE_FEATURE ioctl provided here. I used this with an Intel 82576
(igb) where the PF driver doesn't particularly like being assigned to a
VM with SR-IOV enabled. Likewise, I can prove that the interfaces here
provide the correct restrictions for the VF, but the VF doesn't work in
a VM due to the state of the PF. I'm hoping we'll have some
confirmation from the DPDK folks that this provides what they need to
abandon the non-upstream drivers and more nefarious hacks. There's a
lot more virtualization work to be done in QEMU before I'd propose
patch I reference above upstream.
To your specific question regarding CX5, I think there are very few
SR-IOV devices where the PF doesn't act as some kind of packet router
or ring management engine. The Amazon device listed in the pci-pf-stub
driver seems to be one of the few SR-IOV devices which claim the PF has
no special interfaces other than exposing the SR-IOV capability itself.
So I think we generally expect a device specific SR-IOV aware driver
running on the PF via this interface. That's certainly the case for
the DPDK code for the FPGA device above. Thanks,
Alex
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v8 1/1] fbarray: fix duplicated fbarray file in secondary
@ 2020-02-14 15:29 3% ` Thomas Monjalon
2020-02-17 12:54 0% ` Yasufumi Ogawa
0 siblings, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-02-14 15:29 UTC (permalink / raw)
To: Yasufumi Ogawa, David Marchand; +Cc: Burakov, Anatoly, Ananyev, Konstantin, dev
14/02/2020 16:08, David Marchand:
> On Fri, Feb 14, 2020 at 8:46 AM Yasufumi Ogawa <yasufum.o@gmail.com> wrote:
> >
> > Hi,
> >
> > Could I confirm that this patch is going to be merged in 20.02?
>
> Sorry, but I can't take this patch in 20.02.
> It breaks compilation on FreeBSD.
> http://mails.dpdk.org/archives/test-report/2019-November/109435.html
>
>
> I am still unconvinced on the need to change the size to something so
> huge to accommodate with this new use case (secondary processes in
> containers).
> Why can't we truncate the container hostname so that it fits in 64 bytes?
>
>
> Thomas, opinion?
If the use case is justified enough, I would prefer merging such change in
20.11 avoiding an ABI breakage in a core library, even if it is experimental.
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH v2] service: don't walk out of bounds when checking services
@ 2020-02-14 16:38 0% ` Kevin Traynor
0 siblings, 0 replies; 200+ results
From: Kevin Traynor @ 2020-02-14 16:38 UTC (permalink / raw)
To: Aaron Conole
Cc: Harry van Haaren, nikhil.rao, David Marchand, dev,
Bruce Richardson, Pavan Nikhilesh, Gage Eads, Thomas Monjalon,
dpdk stable
On 07/02/2020 14:27, Aaron Conole wrote:
> Kevin Traynor <ktraynor@redhat.com> writes:
>
>> On 20/12/2019 14:43, David Marchand wrote:
>>> On Wed, Dec 4, 2019 at 9:34 AM David Marchand <david.marchand@redhat.com> wrote:
>>>>
>>>> On Wed, Dec 4, 2019 at 9:33 AM David Marchand <david.marchand@redhat.com> wrote:
>>>>>
>>>>> On Tue, Dec 3, 2019 at 10:15 PM Aaron Conole <aconole@redhat.com> wrote:
>>>>>>
>>>>>> The service_valid call is used without properly bounds checking the
>>>>>> input parameter. Almost all instances of the service_valid call are
>>>>>> inside a for() loop that prevents excessive walks, but some of the
>>>>>> public APIs don't bounds check and will pass invalid arguments.
>>>>>>
>>>>>> Prevent this by using SERVICE_GET_OR_ERR_RET where it makes sense,
>>>>>> and adding a bounds check to one service_valid() use.
>>>>>>
>>>>>> Fixes: 8d39d3e237c2 ("service: fix race in service on app lcore function")
>>>>>> Fixes: e9139a32f6e8 ("service: add function to run on app lcore")
>>>>>> Fixes: e30dd31847d2 ("service: add mechanism for quiescing")
>>>> Cc: stable@dpdk.org
>>>>
>>
>> With the commit below, this patch will apply cleanly on 18.11.
>>
>> Seems ok to me to add below commit, wdyt?
>
> If I'm reading it correctly, the move is for an internal data structure
> in librte_eal, so I think it shouldn't be an ABI breakage.
>
> Looks safe to me as well.
>
Thanks, both patches applied.
>> commit e484ccddbe1b41886fef1e445ef2fdfa55086198
>> Author: Nikhil Rao <nikhil.rao@intel.com>
>> Date: Mon Sep 16 15:31:02 2019 +0530
>>
>> service: avoid false sharing on core state
>>
>>
>>>>>> Signed-off-by: Aaron Conole <aconole@redhat.com>
>>>>>
>>>>> Reviewed-by: David Marchand <david.marchand@redhat.com>
>>>
>>> Applied, thanks.
>>>
>>>
>>> --
>>> David Marchand
>>>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [RFC] meter: fix ABI break due to experimental tag removal
2020-02-14 11:36 7% ` Bruce Richardson
@ 2020-02-14 20:48 4% ` Neil Horman
2020-02-14 21:52 4% ` Bruce Richardson
2020-02-17 14:23 8% ` Ray Kinsella
0 siblings, 2 replies; 200+ results
From: Neil Horman @ 2020-02-14 20:48 UTC (permalink / raw)
To: Bruce Richardson
Cc: Ray Kinsella, Luca Boccassi, Ferruh Yigit, Cristian Dumitrescu,
Eelco Chaudron, dev, Thomas Monjalon, David Marchand, Ian Stokes
On Fri, Feb 14, 2020 at 11:36:34AM +0000, Bruce Richardson wrote:
> On Thu, Feb 13, 2020 at 09:40:40PM -0500, Neil Horman wrote:
> > On Thu, Feb 13, 2020 at 05:40:43PM +0000, Ray Kinsella wrote:
> > >
> > >
> > > On 05/02/2020 11:32, Neil Horman wrote:
> > > > On Wed, Feb 05, 2020 at 11:04:29AM +0100, Luca Boccassi wrote:
> > > >> On Tue, 2020-02-04 at 07:02 -0500, Neil Horman wrote:
> > > >>>> But if we can do the versioning in the master, LTS can backport it
> > > >>>> and can have mature version of that API in LTS without breaking
> > > >>>> the existing users.
> > > >>>>
> > > >>>
> > > >>> But why bother? The only thing you've changed is the version
> > > >>> tagging. Its ok to leave that alone in LTS, you just cant change
> > > >>> it.
> > > >>>
> > > >>> Thats part of the burden of an LTS release, it will have some drift
> > > >>> from the upstream head, because you have to keep things stable. So
> > > >>> you stabilize the upstream ABI version for this API and just never
> > > >>> backport it to the current LTS release.
> > > >>
> > > >> Hi,
> > > >>
> > > >> A customer (OVS) explicitly and specifically requested backporting
> > > >> the symbol change to 19.11, as they don't want to enable
> > > >> experimental APIs in their releases. I replied that it would only be
> > > >> acceptable with aliasing to keep compatibility, and Ferruh very
> > > >> kindly did the work to implement that.
> > > >>
> > > > but, thats part of the agreement, no? You can't always have new
> > > > features and stability at the same time.
> > > >
> > > > I get that this is an odd corner case, because strictly speaking you
> > > > could waive the ABI change that libabigail is reporting, and most
> > > > application users (like OVS) could get away with it, because their
> > > > application does all the right things to make it ok, but I don't
> > > > think you can make a decsion like this for all users based on the
> > > > request of a single user.
> > > >
> > > > It seems like the right thing is for OVS to augment their build time
> > > > configuration to allow developers to select the ability to use
> > > > experimental apis at compile time, and then the decision is placed in
> > > > the hands of end users.
> > >
> > > So this is not isolated case right ... it is common from API's to
> > > graduate from experimental to mature, we _want_ that to happen, we
> > > _want_ APIs to mature.
> > >
> > Sure, I can absolutely agree with that, though I would suggest that the
> > maturity of the ABI is orthogonal to its labeling as such (more on that
> > below)
> >
> > > I am worried what you are proposing will encourage a behavior whereby
> > > maintainers to wait until the declaration of the next major ABI version
> > > to make these symbol changes, causing these kinds of changes to queue
> > > up for that release.
> > >
> > I think you're probably right about that, and would make the agrument
> > that thats perfectly fine (again I'll clarify below)
> >
> > > And you would have to ask why? In the case of a reasonably mature API,
> > > there will be no functional or signature change - its mature! So what
> > > is the harm of providing an alias back to Experimental until the next
> > > ABI version is declared?
> > >
> > From a philosophical standpoint, there is absoluely no harm, and I don't
> > think anyone would disagree, the harm comes from the details of the
> > implementation, as you've noted.
> >
> > > So while yes, you are 100% right - experimental mean no guarantees.
> > > But if the API is baked, it is not going to change, so I don't see the
> > > harm.
> > >
> > > We don't want the unnecessary triumph of policy over pragmatism.
> > >
> > I would make the converse argument here. While I agree that when an API
> > is mature, theres no point in calling it experimental anymore, I would
> > also suggest that, if an API is mature, and not expected to change,
> > theres no harm in leaving its version tag as experimental for an LTS
> > release. This is what I was referring to above. Once an application
> > developer has done the work to integrate an API from DPDK into its
> > application, that work is done. If the API remains stable, then I
> > honestly don't know that they care that the version label is EXPERIMENTAL
> > versus 20.11 or 20.05 or whatever. They care about the API being stable
> > more so than its version label. As such, it seems....reasonable to me to
> > have developers queue their migration of experimental APIs to official
> > versioned APIs at the next major release deliniation point.
> >
> > I'd welcome counter arguments, but it seems pretty natural to me to make
> > these sorts of changes at the major release mark. People using
> > experimantal apis have already implicity agreed to manage changes to
> > them, so if we just hold it stable in an LTS release (and don't update
> > the version label), its just gravy for them.
> >
> The counter argument that I see is that while the experimental tag remains
> in place the user has no way to know that an API is stable or not, and so
> in many projects cannot use the API. If for example an API that is
> experimental in 19.11 is unchanged through 20.05 at which point we decide
> to promote it to stable. Once the change to the exp tag it is made, any
> users of 19.11 can then see that it is an unchanged and now-stable ABI and
> can start using it in their software, if they wish, without having to wait
> for the 20.11 release. Changing the tag early allows ABIs to be potentially
> used immediately.
>
I would agree with this, however, when using an LTS release, in my mind at
least, part of the agreement there is you get stability in the fuctions that
were declared stable at the time of release. I'm not sure there should be an
expectation of additional stabilization within the lifetime of the release
(thats really what the 12 month LTS release cycle is for, no)? You never have
to wait more than a year for a new set of stable features. If you want faster
feature integration than that, you can choose to enable the experimental API's
and accept the benefits and drawbacks thereof.
That said, if (as I understand it) the goal is to provide a mechanism to allow
experimental features to be promoted to stable status, perhaps we can find a
happy middle ground. What if we were to create a construct such as this:
#pragma push_macro("ALLOW_EXPERIMENTAL_APIS")
#undef ALLOW_EXPERIMENTAL_APIS
void __rte_experimental func(...);
#pragma pop_macro("ALLOW_EXPERIMENTAL_APIS")
Such a consruct would allow the maintainer of an API to pseudo-promote an API
from a experimental to stable status, at least so far as compilation would be
concerned. Its messy and clunky, and it wouldn't change the function version at
all, but the end result would be that:
a) such a wraped experimental function would no longer issue a warning when used
during compilation/linking
and
b) provide a semi-easy grepable pattern for application writers to look for when
considering the use of an API that was previously experimental
such a construct would have to be used very judiciously, in that once its
implemented, the API has to be treated as stable, even though its 'excused' from
the normal checking, but it could provide something of the more rapid promotion
path being sought.
Thoughts?
Neil
> Regards,
> /Bruce
>
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [RFC] meter: fix ABI break due to experimental tag removal
2020-02-14 20:48 4% ` Neil Horman
@ 2020-02-14 21:52 4% ` Bruce Richardson
2020-02-15 13:43 4% ` Neil Horman
2020-02-17 14:23 8% ` Ray Kinsella
1 sibling, 1 reply; 200+ results
From: Bruce Richardson @ 2020-02-14 21:52 UTC (permalink / raw)
To: Neil Horman
Cc: Ray Kinsella, Luca Boccassi, Ferruh Yigit, Cristian Dumitrescu,
Eelco Chaudron, dev, Thomas Monjalon, David Marchand, Ian Stokes
On Fri, Feb 14, 2020 at 03:48:48PM -0500, Neil Horman wrote:
> On Fri, Feb 14, 2020 at 11:36:34AM +0000, Bruce Richardson wrote:
> > On Thu, Feb 13, 2020 at 09:40:40PM -0500, Neil Horman wrote:
> > > On Thu, Feb 13, 2020 at 05:40:43PM +0000, Ray Kinsella wrote:
> > > >
> > > >
> > > > On 05/02/2020 11:32, Neil Horman wrote:
> > > > > On Wed, Feb 05, 2020 at 11:04:29AM +0100, Luca Boccassi wrote:
> > > > >> On Tue, 2020-02-04 at 07:02 -0500, Neil Horman wrote:
> > > > >>>> But if we can do the versioning in the master, LTS can backport it
> > > > >>>> and can have mature version of that API in LTS without breaking
> > > > >>>> the existing users.
> > > > >>>>
> > > > >>>
> > > > >>> But why bother? The only thing you've changed is the version
> > > > >>> tagging. Its ok to leave that alone in LTS, you just cant change
> > > > >>> it.
> > > > >>>
> > > > >>> Thats part of the burden of an LTS release, it will have some drift
> > > > >>> from the upstream head, because you have to keep things stable. So
> > > > >>> you stabilize the upstream ABI version for this API and just never
> > > > >>> backport it to the current LTS release.
> > > > >>
> > > > >> Hi,
> > > > >>
> > > > >> A customer (OVS) explicitly and specifically requested backporting
> > > > >> the symbol change to 19.11, as they don't want to enable
> > > > >> experimental APIs in their releases. I replied that it would only be
> > > > >> acceptable with aliasing to keep compatibility, and Ferruh very
> > > > >> kindly did the work to implement that.
> > > > >>
> > > > > but, thats part of the agreement, no? You can't always have new
> > > > > features and stability at the same time.
> > > > >
> > > > > I get that this is an odd corner case, because strictly speaking you
> > > > > could waive the ABI change that libabigail is reporting, and most
> > > > > application users (like OVS) could get away with it, because their
> > > > > application does all the right things to make it ok, but I don't
> > > > > think you can make a decsion like this for all users based on the
> > > > > request of a single user.
> > > > >
> > > > > It seems like the right thing is for OVS to augment their build time
> > > > > configuration to allow developers to select the ability to use
> > > > > experimental apis at compile time, and then the decision is placed in
> > > > > the hands of end users.
> > > >
> > > > So this is not isolated case right ... it is common from API's to
> > > > graduate from experimental to mature, we _want_ that to happen, we
> > > > _want_ APIs to mature.
> > > >
> > > Sure, I can absolutely agree with that, though I would suggest that the
> > > maturity of the ABI is orthogonal to its labeling as such (more on that
> > > below)
> > >
> > > > I am worried what you are proposing will encourage a behavior whereby
> > > > maintainers to wait until the declaration of the next major ABI version
> > > > to make these symbol changes, causing these kinds of changes to queue
> > > > up for that release.
> > > >
> > > I think you're probably right about that, and would make the agrument
> > > that thats perfectly fine (again I'll clarify below)
> > >
> > > > And you would have to ask why? In the case of a reasonably mature API,
> > > > there will be no functional or signature change - its mature! So what
> > > > is the harm of providing an alias back to Experimental until the next
> > > > ABI version is declared?
> > > >
> > > From a philosophical standpoint, there is absoluely no harm, and I don't
> > > think anyone would disagree, the harm comes from the details of the
> > > implementation, as you've noted.
> > >
> > > > So while yes, you are 100% right - experimental mean no guarantees.
> > > > But if the API is baked, it is not going to change, so I don't see the
> > > > harm.
> > > >
> > > > We don't want the unnecessary triumph of policy over pragmatism.
> > > >
> > > I would make the converse argument here. While I agree that when an API
> > > is mature, theres no point in calling it experimental anymore, I would
> > > also suggest that, if an API is mature, and not expected to change,
> > > theres no harm in leaving its version tag as experimental for an LTS
> > > release. This is what I was referring to above. Once an application
> > > developer has done the work to integrate an API from DPDK into its
> > > application, that work is done. If the API remains stable, then I
> > > honestly don't know that they care that the version label is EXPERIMENTAL
> > > versus 20.11 or 20.05 or whatever. They care about the API being stable
> > > more so than its version label. As such, it seems....reasonable to me to
> > > have developers queue their migration of experimental APIs to official
> > > versioned APIs at the next major release deliniation point.
> > >
> > > I'd welcome counter arguments, but it seems pretty natural to me to make
> > > these sorts of changes at the major release mark. People using
> > > experimantal apis have already implicity agreed to manage changes to
> > > them, so if we just hold it stable in an LTS release (and don't update
> > > the version label), its just gravy for them.
> > >
> > The counter argument that I see is that while the experimental tag remains
> > in place the user has no way to know that an API is stable or not, and so
> > in many projects cannot use the API. If for example an API that is
> > experimental in 19.11 is unchanged through 20.05 at which point we decide
> > to promote it to stable. Once the change to the exp tag it is made, any
> > users of 19.11 can then see that it is an unchanged and now-stable ABI and
> > can start using it in their software, if they wish, without having to wait
> > for the 20.11 release. Changing the tag early allows ABIs to be potentially
> > used immediately.
> >
>
> I would agree with this, however, when using an LTS release, in my mind at
> least, part of the agreement there is you get stability in the fuctions that
> were declared stable at the time of release. I'm not sure there should be an
> expectation of additional stabilization within the lifetime of the release
> (thats really what the 12 month LTS release cycle is for, no)? You never have
> to wait more than a year for a new set of stable features. If you want faster
> feature integration than that, you can choose to enable the experimental API's
> and accept the benefits and drawbacks thereof.
>
> That said, if (as I understand it) the goal is to provide a mechanism to allow
> experimental features to be promoted to stable status, perhaps we can find a
> happy middle ground. What if we were to create a construct such as this:
>
> #pragma push_macro("ALLOW_EXPERIMENTAL_APIS")
> #undef ALLOW_EXPERIMENTAL_APIS
> void __rte_experimental func(...);
> #pragma pop_macro("ALLOW_EXPERIMENTAL_APIS")
>
> Such a consruct would allow the maintainer of an API to pseudo-promote an API
> from a experimental to stable status, at least so far as compilation would be
> concerned. Its messy and clunky, and it wouldn't change the function version at
> all, but the end result would be that:
> a) such a wraped experimental function would no longer issue a warning when used
> during compilation/linking
> and
> b) provide a semi-easy grepable pattern for application writers to look for when
> considering the use of an API that was previously experimental
>
> such a construct would have to be used very judiciously, in that once its
> implemented, the API has to be treated as stable, even though its 'excused' from
> the normal checking, but it could provide something of the more rapid promotion
> path being sought.
>
Sure it could work and does meet the end goal. I'm not really sure it's a
better solution than just promoting the function and putting in an alias
for it back to experimental, though.
/Bruce
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [RFC] meter: fix ABI break due to experimental tag removal
2020-02-14 21:52 4% ` Bruce Richardson
@ 2020-02-15 13:43 4% ` Neil Horman
0 siblings, 0 replies; 200+ results
From: Neil Horman @ 2020-02-15 13:43 UTC (permalink / raw)
To: Bruce Richardson
Cc: Ray Kinsella, Luca Boccassi, Ferruh Yigit, Cristian Dumitrescu,
Eelco Chaudron, dev, Thomas Monjalon, David Marchand, Ian Stokes
On Fri, Feb 14, 2020 at 09:52:53PM +0000, Bruce Richardson wrote:
> On Fri, Feb 14, 2020 at 03:48:48PM -0500, Neil Horman wrote:
> > On Fri, Feb 14, 2020 at 11:36:34AM +0000, Bruce Richardson wrote:
> > > On Thu, Feb 13, 2020 at 09:40:40PM -0500, Neil Horman wrote:
> > > > On Thu, Feb 13, 2020 at 05:40:43PM +0000, Ray Kinsella wrote:
> > > > >
> > > > >
> > > > > On 05/02/2020 11:32, Neil Horman wrote:
> > > > > > On Wed, Feb 05, 2020 at 11:04:29AM +0100, Luca Boccassi wrote:
> > > > > >> On Tue, 2020-02-04 at 07:02 -0500, Neil Horman wrote:
> > > > > >>>> But if we can do the versioning in the master, LTS can backport it
> > > > > >>>> and can have mature version of that API in LTS without breaking
> > > > > >>>> the existing users.
> > > > > >>>>
> > > > > >>>
> > > > > >>> But why bother? The only thing you've changed is the version
> > > > > >>> tagging. Its ok to leave that alone in LTS, you just cant change
> > > > > >>> it.
> > > > > >>>
> > > > > >>> Thats part of the burden of an LTS release, it will have some drift
> > > > > >>> from the upstream head, because you have to keep things stable. So
> > > > > >>> you stabilize the upstream ABI version for this API and just never
> > > > > >>> backport it to the current LTS release.
> > > > > >>
> > > > > >> Hi,
> > > > > >>
> > > > > >> A customer (OVS) explicitly and specifically requested backporting
> > > > > >> the symbol change to 19.11, as they don't want to enable
> > > > > >> experimental APIs in their releases. I replied that it would only be
> > > > > >> acceptable with aliasing to keep compatibility, and Ferruh very
> > > > > >> kindly did the work to implement that.
> > > > > >>
> > > > > > but, thats part of the agreement, no? You can't always have new
> > > > > > features and stability at the same time.
> > > > > >
> > > > > > I get that this is an odd corner case, because strictly speaking you
> > > > > > could waive the ABI change that libabigail is reporting, and most
> > > > > > application users (like OVS) could get away with it, because their
> > > > > > application does all the right things to make it ok, but I don't
> > > > > > think you can make a decsion like this for all users based on the
> > > > > > request of a single user.
> > > > > >
> > > > > > It seems like the right thing is for OVS to augment their build time
> > > > > > configuration to allow developers to select the ability to use
> > > > > > experimental apis at compile time, and then the decision is placed in
> > > > > > the hands of end users.
> > > > >
> > > > > So this is not isolated case right ... it is common from API's to
> > > > > graduate from experimental to mature, we _want_ that to happen, we
> > > > > _want_ APIs to mature.
> > > > >
> > > > Sure, I can absolutely agree with that, though I would suggest that the
> > > > maturity of the ABI is orthogonal to its labeling as such (more on that
> > > > below)
> > > >
> > > > > I am worried what you are proposing will encourage a behavior whereby
> > > > > maintainers to wait until the declaration of the next major ABI version
> > > > > to make these symbol changes, causing these kinds of changes to queue
> > > > > up for that release.
> > > > >
> > > > I think you're probably right about that, and would make the agrument
> > > > that thats perfectly fine (again I'll clarify below)
> > > >
> > > > > And you would have to ask why? In the case of a reasonably mature API,
> > > > > there will be no functional or signature change - its mature! So what
> > > > > is the harm of providing an alias back to Experimental until the next
> > > > > ABI version is declared?
> > > > >
> > > > From a philosophical standpoint, there is absoluely no harm, and I don't
> > > > think anyone would disagree, the harm comes from the details of the
> > > > implementation, as you've noted.
> > > >
> > > > > So while yes, you are 100% right - experimental mean no guarantees.
> > > > > But if the API is baked, it is not going to change, so I don't see the
> > > > > harm.
> > > > >
> > > > > We don't want the unnecessary triumph of policy over pragmatism.
> > > > >
> > > > I would make the converse argument here. While I agree that when an API
> > > > is mature, theres no point in calling it experimental anymore, I would
> > > > also suggest that, if an API is mature, and not expected to change,
> > > > theres no harm in leaving its version tag as experimental for an LTS
> > > > release. This is what I was referring to above. Once an application
> > > > developer has done the work to integrate an API from DPDK into its
> > > > application, that work is done. If the API remains stable, then I
> > > > honestly don't know that they care that the version label is EXPERIMENTAL
> > > > versus 20.11 or 20.05 or whatever. They care about the API being stable
> > > > more so than its version label. As such, it seems....reasonable to me to
> > > > have developers queue their migration of experimental APIs to official
> > > > versioned APIs at the next major release deliniation point.
> > > >
> > > > I'd welcome counter arguments, but it seems pretty natural to me to make
> > > > these sorts of changes at the major release mark. People using
> > > > experimantal apis have already implicity agreed to manage changes to
> > > > them, so if we just hold it stable in an LTS release (and don't update
> > > > the version label), its just gravy for them.
> > > >
> > > The counter argument that I see is that while the experimental tag remains
> > > in place the user has no way to know that an API is stable or not, and so
> > > in many projects cannot use the API. If for example an API that is
> > > experimental in 19.11 is unchanged through 20.05 at which point we decide
> > > to promote it to stable. Once the change to the exp tag it is made, any
> > > users of 19.11 can then see that it is an unchanged and now-stable ABI and
> > > can start using it in their software, if they wish, without having to wait
> > > for the 20.11 release. Changing the tag early allows ABIs to be potentially
> > > used immediately.
> > >
> >
> > I would agree with this, however, when using an LTS release, in my mind at
> > least, part of the agreement there is you get stability in the fuctions that
> > were declared stable at the time of release. I'm not sure there should be an
> > expectation of additional stabilization within the lifetime of the release
> > (thats really what the 12 month LTS release cycle is for, no)? You never have
> > to wait more than a year for a new set of stable features. If you want faster
> > feature integration than that, you can choose to enable the experimental API's
> > and accept the benefits and drawbacks thereof.
> >
> > That said, if (as I understand it) the goal is to provide a mechanism to allow
> > experimental features to be promoted to stable status, perhaps we can find a
> > happy middle ground. What if we were to create a construct such as this:
> >
> > #pragma push_macro("ALLOW_EXPERIMENTAL_APIS")
> > #undef ALLOW_EXPERIMENTAL_APIS
> > void __rte_experimental func(...);
> > #pragma pop_macro("ALLOW_EXPERIMENTAL_APIS")
> >
> > Such a consruct would allow the maintainer of an API to pseudo-promote an API
> > from a experimental to stable status, at least so far as compilation would be
> > concerned. Its messy and clunky, and it wouldn't change the function version at
> > all, but the end result would be that:
> > a) such a wraped experimental function would no longer issue a warning when used
> > during compilation/linking
> > and
> > b) provide a semi-easy grepable pattern for application writers to look for when
> > considering the use of an API that was previously experimental
> >
> > such a construct would have to be used very judiciously, in that once its
> > implemented, the API has to be treated as stable, even though its 'excused' from
> > the normal checking, but it could provide something of the more rapid promotion
> > path being sought.
> >
> Sure it could work and does meet the end goal. I'm not really sure it's a
> better solution than just promoting the function and putting in an alias
> for it back to experimental, though.
>
It probably isn't, I was just trying to think of alternative
implementations, that dont make life harder in keeping up with LTS
maintenence.
Neil
> /Bruce
>
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v8 1/1] fbarray: fix duplicated fbarray file in secondary
2020-02-14 15:29 3% ` Thomas Monjalon
@ 2020-02-17 12:54 0% ` Yasufumi Ogawa
0 siblings, 0 replies; 200+ results
From: Yasufumi Ogawa @ 2020-02-17 12:54 UTC (permalink / raw)
To: Thomas Monjalon, David Marchand
Cc: Burakov, Anatoly, Ananyev, Konstantin, dev
> 14/02/2020 16:08, David Marchand:
>> On Fri, Feb 14, 2020 at 8:46 AM Yasufumi Ogawa <yasufum.o@gmail.com> wrote:
>>>
>>> Hi,
>>>
>>> Could I confirm that this patch is going to be merged in 20.02?
>>
>> Sorry, but I can't take this patch in 20.02.
>> It breaks compilation on FreeBSD.
>> http://mails.dpdk.org/archives/test-report/2019-November/109435.html
Sorry. I didn't find it. I'd see it.
>>
>>
>> I am still unconvinced on the need to change the size to something so
>> huge to accommodate with this new use case (secondary processes in
>> containers).
It is not so common actually, but serious issue for some NFV usecases. I
remember, in a talk in the last DPDK summit, ZTE was also suffered from
the same problem.
>> Why can't we truncate the container hostname so that it fits in 64 bytes?
It is just a possible maximum length of format of
"fbarray_memseg-1048576k-0-0_PID_HOSTNAME", so I think it can be
truncated if dropping some information.
>>
>>
>> Thomas, opinion?
>
> If the use case is justified enough, I would prefer merging such change in
> 20.11 avoiding an ABI breakage in a core library, even if it is experimental.I understand, thanks.
Yasufumi
^ permalink raw reply [relevance 0%]
* [dpdk-dev] [PATCH] ci: build and use libabigail 1.6
@ 2020-02-17 13:59 9% David Marchand
2020-02-17 15:15 0% ` Thomas Monjalon
` (3 more replies)
0 siblings, 4 replies; 200+ results
From: David Marchand @ 2020-02-17 13:59 UTC (permalink / raw)
To: thomas; +Cc: dev, christian.ehrhardt, dodji, Aaron Conole, Michael Santana
libabigail 1.2 (at least) reports changes in 'const' property as an ABI
breakage [1].
This was fixed upstream in libabigail 1.4 [2], and a bug has been opened
in launchpad [3].
But for now, build and use the last version 1.6 so that the ABI checks
can be kept.
1: https://travis-ci.com/DPDK/dpdk/jobs/287872118#L2242
2: https://sourceware.org/git/gitweb.cgi?p=libabigail.git;a=commitdiff;h=215b7eb4fe8b986fe1cc87d9d8e7412998038392
3: https://bugs.launchpad.net/ubuntu/+source/libabigail/+bug/1863607
Signed-off-by: David Marchand <david.marchand@redhat.com>
---
.ci/linux-build.sh | 22 ++++++++++++++++++++++
.travis.yml | 15 ++++++++++-----
2 files changed, 32 insertions(+), 5 deletions(-)
diff --git a/.ci/linux-build.sh b/.ci/linux-build.sh
index c7c3840fc..0d4bc9a62 100755
--- a/.ci/linux-build.sh
+++ b/.ci/linux-build.sh
@@ -38,6 +38,28 @@ if [ "$AARCH64" != "1" ]; then
fi
if [ "$ABI_CHECKS" = "1" ]; then
+ LIBABIGAIL_REPO=${LIBABIGAIL_REPO:-https://sourceware.org/git/libabigail.git}
+ LIBABIGAIL_VERSION=${LIBABIGAIL_VERSION:-libabigail-1.6}
+
+ if [ "$(cat libabigail/VERSION 2>/dev/null)" != "$LIBABIGAIL_VERSION" ]; then
+ rm -rf libabigail
+ # if we change libabigail, invalidate existing abi cache
+ rm -rf reference
+ fi
+
+ if [ ! -d libabigail ]; then
+ git clone --single-branch -b $LIBABIGAIL_VERSION $LIBABIGAIL_REPO libabigail/src
+ cd libabigail/src && autoconf -vfi && cd -
+ mkdir libabigail/src/build
+ cd libabigail/src/build && ../configure --prefix=$(pwd)/libabigail && cd -
+ make -C libabigail/src/build all install
+
+ rm -rf libabigail/src
+ echo $LIBABIGAIL_VERSION > libabigail/VERSION
+ fi
+
+ export PATH=$(pwd)/libabigail/bin:$PATH
+
REF_GIT_REPO=${REF_GIT_REPO:-https://dpdk.org/git/dpdk}
REF_GIT_TAG=${REF_GIT_TAG:-v19.11}
diff --git a/.travis.yml b/.travis.yml
index 22539d823..d8253fdd4 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -2,6 +2,7 @@ language: c
cache:
ccache: true
directories:
+ - libabigail
- reference
compiler:
- gcc
@@ -24,7 +25,11 @@ aarch64_packages: &aarch64_packages
extra_packages: &extra_packages
- *required_packages
- - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4, abigail-tools]
+ - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
+
+libabigail_build_packages: &libabigail_build_packages
+ - *required_packages
+ - [autoconf, automake, libtool, pkg-config, libxml2-dev, libdw-dev]
build_32b_packages: &build_32b_packages
- *required_packages
@@ -154,18 +159,18 @@ matrix:
packages:
- *required_packages
- *doc_packages
- - env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1
+ - env: DEF_LIB="shared" ABI_CHECKS=1
compiler: gcc
addons:
apt:
packages:
- - *extra_packages
- - env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1
+ - *libabigail_build_packages
+ - env: DEF_LIB="shared" ABI_CHECKS=1
arch: arm64
compiler: gcc
addons:
apt:
packages:
- - *extra_packages
+ - *libabigail_build_packages
script: ./.ci/${TRAVIS_OS_NAME}-build.sh
--
2.23.0
^ permalink raw reply [relevance 9%]
* Re: [dpdk-dev] [RFC] meter: fix ABI break due to experimental tag removal
2020-02-14 20:48 4% ` Neil Horman
2020-02-14 21:52 4% ` Bruce Richardson
@ 2020-02-17 14:23 8% ` Ray Kinsella
2020-02-17 15:37 4% ` Neil Horman
1 sibling, 1 reply; 200+ results
From: Ray Kinsella @ 2020-02-17 14:23 UTC (permalink / raw)
To: Neil Horman, Bruce Richardson
Cc: Luca Boccassi, Ferruh Yigit, Cristian Dumitrescu, Eelco Chaudron,
dev, Thomas Monjalon, David Marchand, Ian Stokes
On 14/02/2020 20:48, Neil Horman wrote:
> On Fri, Feb 14, 2020 at 11:36:34AM +0000, Bruce Richardson wrote:
>> On Thu, Feb 13, 2020 at 09:40:40PM -0500, Neil Horman wrote:
>>> On Thu, Feb 13, 2020 at 05:40:43PM +0000, Ray Kinsella wrote:
>>>>
>>>>
>>>> On 05/02/2020 11:32, Neil Horman wrote:
>>>>> On Wed, Feb 05, 2020 at 11:04:29AM +0100, Luca Boccassi wrote:
>>>>>> On Tue, 2020-02-04 at 07:02 -0500, Neil Horman wrote:
>>>>>>>> But if we can do the versioning in the master, LTS can backport it
>>>>>>>> and can have mature version of that API in LTS without breaking
>>>>>>>> the existing users.
>>>>>>>>
>>>>>>>
>>>>>>> But why bother? The only thing you've changed is the version
>>>>>>> tagging. Its ok to leave that alone in LTS, you just cant change
>>>>>>> it.
>>>>>>>
>>>>>>> Thats part of the burden of an LTS release, it will have some drift
>>>>>>> from the upstream head, because you have to keep things stable. So
>>>>>>> you stabilize the upstream ABI version for this API and just never
>>>>>>> backport it to the current LTS release.
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> A customer (OVS) explicitly and specifically requested backporting
>>>>>> the symbol change to 19.11, as they don't want to enable
>>>>>> experimental APIs in their releases. I replied that it would only be
>>>>>> acceptable with aliasing to keep compatibility, and Ferruh very
>>>>>> kindly did the work to implement that.
>>>>>>
>>>>> but, thats part of the agreement, no? You can't always have new
>>>>> features and stability at the same time.
>>>>>
>>>>> I get that this is an odd corner case, because strictly speaking you
>>>>> could waive the ABI change that libabigail is reporting, and most
>>>>> application users (like OVS) could get away with it, because their
>>>>> application does all the right things to make it ok, but I don't
>>>>> think you can make a decsion like this for all users based on the
>>>>> request of a single user.
>>>>>
>>>>> It seems like the right thing is for OVS to augment their build time
>>>>> configuration to allow developers to select the ability to use
>>>>> experimental apis at compile time, and then the decision is placed in
>>>>> the hands of end users.
>>>>
>>>> So this is not isolated case right ... it is common from API's to
>>>> graduate from experimental to mature, we _want_ that to happen, we
>>>> _want_ APIs to mature.
>>>>
>>> Sure, I can absolutely agree with that, though I would suggest that the
>>> maturity of the ABI is orthogonal to its labeling as such (more on that
>>> below)
>>>
>>>> I am worried what you are proposing will encourage a behavior whereby
>>>> maintainers to wait until the declaration of the next major ABI version
>>>> to make these symbol changes, causing these kinds of changes to queue
>>>> up for that release.
>>>>
>>> I think you're probably right about that, and would make the agrument
>>> that thats perfectly fine (again I'll clarify below)
>>>
>>>> And you would have to ask why? In the case of a reasonably mature API,
>>>> there will be no functional or signature change - its mature! So what
>>>> is the harm of providing an alias back to Experimental until the next
>>>> ABI version is declared?
>>>>
>>> From a philosophical standpoint, there is absoluely no harm, and I don't
>>> think anyone would disagree, the harm comes from the details of the
>>> implementation, as you've noted.
>>>
>>>> So while yes, you are 100% right - experimental mean no guarantees.
>>>> But if the API is baked, it is not going to change, so I don't see the
>>>> harm.
>>>>
>>>> We don't want the unnecessary triumph of policy over pragmatism.
>>>>
>>> I would make the converse argument here. While I agree that when an API
>>> is mature, theres no point in calling it experimental anymore, I would
>>> also suggest that, if an API is mature, and not expected to change,
>>> theres no harm in leaving its version tag as experimental for an LTS
>>> release. This is what I was referring to above. Once an application
>>> developer has done the work to integrate an API from DPDK into its
>>> application, that work is done. If the API remains stable, then I
>>> honestly don't know that they care that the version label is EXPERIMENTAL
>>> versus 20.11 or 20.05 or whatever. They care about the API being stable
>>> more so than its version label. As such, it seems....reasonable to me to
>>> have developers queue their migration of experimental APIs to official
>>> versioned APIs at the next major release deliniation point.
>>>
>>> I'd welcome counter arguments, but it seems pretty natural to me to make
>>> these sorts of changes at the major release mark. People using
>>> experimantal apis have already implicity agreed to manage changes to
>>> them, so if we just hold it stable in an LTS release (and don't update
>>> the version label), its just gravy for them.
>>>
>> The counter argument that I see is that while the experimental tag remains
>> in place the user has no way to know that an API is stable or not, and so
>> in many projects cannot use the API. If for example an API that is
>> experimental in 19.11 is unchanged through 20.05 at which point we decide
>> to promote it to stable. Once the change to the exp tag it is made, any
>> users of 19.11 can then see that it is an unchanged and now-stable ABI and
>> can start using it in their software, if they wish, without having to wait
>> for the 20.11 release. Changing the tag early allows ABIs to be potentially
>> used immediately.
>>
>
> I would agree with this, however, when using an LTS release, in my mind at
> least, part of the agreement there is you get stability in the fuctions that
> were declared stable at the time of release. I'm not sure there should be an
> expectation of additional stabilization within the lifetime of the release
> (thats really what the 12 month LTS release cycle is for, no)? You never have
> to wait more than a year for a new set of stable features. If you want faster
> feature integration than that, you can choose to enable the experimental API's
> and accept the benefits and drawbacks thereof.
>
> That said, if (as I understand it) the goal is to provide a mechanism to allow
> experimental features to be promoted to stable status, perhaps we can find a
> happy middle ground. What if we were to create a construct such as this:
>
> #pragma push_macro("ALLOW_EXPERIMENTAL_APIS")
> #undef ALLOW_EXPERIMENTAL_APIS
> void __rte_experimental func(...);
> #pragma pop_macro("ALLOW_EXPERIMENTAL_APIS")
>
> Such a consruct would allow the maintainer of an API to pseudo-promote an API
> from a experimental to stable status, at least so far as compilation would be
> concerned. Its messy and clunky, and it wouldn't change the function version at
> all, but the end result would be that:
> a) such a wraped experimental function would no longer issue a warning when used
> during compilation/linking
> and
> b) provide a semi-easy grepable pattern for application writers to look for when
> considering the use of an API that was previously experimental
>
> such a construct would have to be used very judiciously, in that once its
> implemented, the API has to be treated as stable, even though its 'excused' from
> the normal checking, but it could provide something of the more rapid promotion
> path being sought.
>
> Thoughts?
> Neil
>> Regards,
>> /Bruce
>>
Then method above seems clunky, and it is unclear to me if it solves the original problem.
It seems simpler to me to promote a symbol the next ABI version, and then provide an alias.
At the declaration of the next major ABI version (v21), we would then only need to check
for those symbols that are both EXPERIMENTAL and v21, to know which aliases need to be removed.
Thanks,
Ray K
^ permalink raw reply [relevance 8%]
* Re: [dpdk-dev] [PATCH] ci: build and use libabigail 1.6
2020-02-17 13:59 9% [dpdk-dev] [PATCH] ci: build and use libabigail 1.6 David Marchand
@ 2020-02-17 15:15 0% ` Thomas Monjalon
2020-02-17 18:47 0% ` Aaron Conole
` (2 subsequent siblings)
3 siblings, 0 replies; 200+ results
From: Thomas Monjalon @ 2020-02-17 15:15 UTC (permalink / raw)
To: David Marchand, christian.ehrhardt
Cc: dev, dodji, Aaron Conole, Michael Santana, kevin.laatz
17/02/2020 14:59, David Marchand:
> libabigail 1.2 (at least) reports changes in 'const' property as an ABI
> breakage [1].
> This was fixed upstream in libabigail 1.4 [2], and a bug has been opened
> in launchpad [3].
>
> But for now, build and use the last version 1.6 so that the ABI checks
> can be kept.
>
> 1: https://travis-ci.com/DPDK/dpdk/jobs/287872118#L2242
> 2: https://sourceware.org/git/gitweb.cgi?p=libabigail.git;a=commitdiff;h=215b7eb4fe8b986fe1cc87d9d8e7412998038392
> 3: https://bugs.launchpad.net/ubuntu/+source/libabigail/+bug/1863607
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
> + LIBABIGAIL_REPO=${LIBABIGAIL_REPO:-https://sourceware.org/git/libabigail.git}
> + LIBABIGAIL_VERSION=${LIBABIGAIL_VERSION:-libabigail-1.6}
> +
> + if [ "$(cat libabigail/VERSION 2>/dev/null)" != "$LIBABIGAIL_VERSION" ]; then
> + rm -rf libabigail
> + # if we change libabigail, invalidate existing abi cache
> + rm -rf reference
> + fi
> +
> + if [ ! -d libabigail ]; then
> + git clone --single-branch -b $LIBABIGAIL_VERSION $LIBABIGAIL_REPO libabigail/src
> + cd libabigail/src && autoconf -vfi && cd -
> + mkdir libabigail/src/build
> + cd libabigail/src/build && ../configure --prefix=$(pwd)/libabigail && cd -
> + make -C libabigail/src/build all install
> +
> + rm -rf libabigail/src
> + echo $LIBABIGAIL_VERSION > libabigail/VERSION
> + fi
Can we avoid compiling libabigail ourself?
Is there an up-to-date Ubuntu package somewhere?
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [RFC] meter: fix ABI break due to experimental tag removal
2020-02-17 14:23 8% ` Ray Kinsella
@ 2020-02-17 15:37 4% ` Neil Horman
0 siblings, 0 replies; 200+ results
From: Neil Horman @ 2020-02-17 15:37 UTC (permalink / raw)
To: Ray Kinsella
Cc: Bruce Richardson, Luca Boccassi, Ferruh Yigit,
Cristian Dumitrescu, Eelco Chaudron, dev, Thomas Monjalon,
David Marchand, Ian Stokes
On Mon, Feb 17, 2020 at 02:23:15PM +0000, Ray Kinsella wrote:
>
>
> On 14/02/2020 20:48, Neil Horman wrote:
> > On Fri, Feb 14, 2020 at 11:36:34AM +0000, Bruce Richardson wrote:
> >> On Thu, Feb 13, 2020 at 09:40:40PM -0500, Neil Horman wrote:
> >>> On Thu, Feb 13, 2020 at 05:40:43PM +0000, Ray Kinsella wrote:
> >>>>
> >>>>
> >>>> On 05/02/2020 11:32, Neil Horman wrote:
> >>>>> On Wed, Feb 05, 2020 at 11:04:29AM +0100, Luca Boccassi wrote:
> >>>>>> On Tue, 2020-02-04 at 07:02 -0500, Neil Horman wrote:
> >>>>>>>> But if we can do the versioning in the master, LTS can backport it
> >>>>>>>> and can have mature version of that API in LTS without breaking
> >>>>>>>> the existing users.
> >>>>>>>>
> >>>>>>>
> >>>>>>> But why bother? The only thing you've changed is the version
> >>>>>>> tagging. Its ok to leave that alone in LTS, you just cant change
> >>>>>>> it.
> >>>>>>>
> >>>>>>> Thats part of the burden of an LTS release, it will have some drift
> >>>>>>> from the upstream head, because you have to keep things stable. So
> >>>>>>> you stabilize the upstream ABI version for this API and just never
> >>>>>>> backport it to the current LTS release.
> >>>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>> A customer (OVS) explicitly and specifically requested backporting
> >>>>>> the symbol change to 19.11, as they don't want to enable
> >>>>>> experimental APIs in their releases. I replied that it would only be
> >>>>>> acceptable with aliasing to keep compatibility, and Ferruh very
> >>>>>> kindly did the work to implement that.
> >>>>>>
> >>>>> but, thats part of the agreement, no? You can't always have new
> >>>>> features and stability at the same time.
> >>>>>
> >>>>> I get that this is an odd corner case, because strictly speaking you
> >>>>> could waive the ABI change that libabigail is reporting, and most
> >>>>> application users (like OVS) could get away with it, because their
> >>>>> application does all the right things to make it ok, but I don't
> >>>>> think you can make a decsion like this for all users based on the
> >>>>> request of a single user.
> >>>>>
> >>>>> It seems like the right thing is for OVS to augment their build time
> >>>>> configuration to allow developers to select the ability to use
> >>>>> experimental apis at compile time, and then the decision is placed in
> >>>>> the hands of end users.
> >>>>
> >>>> So this is not isolated case right ... it is common from API's to
> >>>> graduate from experimental to mature, we _want_ that to happen, we
> >>>> _want_ APIs to mature.
> >>>>
> >>> Sure, I can absolutely agree with that, though I would suggest that the
> >>> maturity of the ABI is orthogonal to its labeling as such (more on that
> >>> below)
> >>>
> >>>> I am worried what you are proposing will encourage a behavior whereby
> >>>> maintainers to wait until the declaration of the next major ABI version
> >>>> to make these symbol changes, causing these kinds of changes to queue
> >>>> up for that release.
> >>>>
> >>> I think you're probably right about that, and would make the agrument
> >>> that thats perfectly fine (again I'll clarify below)
> >>>
> >>>> And you would have to ask why? In the case of a reasonably mature API,
> >>>> there will be no functional or signature change - its mature! So what
> >>>> is the harm of providing an alias back to Experimental until the next
> >>>> ABI version is declared?
> >>>>
> >>> From a philosophical standpoint, there is absoluely no harm, and I don't
> >>> think anyone would disagree, the harm comes from the details of the
> >>> implementation, as you've noted.
> >>>
> >>>> So while yes, you are 100% right - experimental mean no guarantees.
> >>>> But if the API is baked, it is not going to change, so I don't see the
> >>>> harm.
> >>>>
> >>>> We don't want the unnecessary triumph of policy over pragmatism.
> >>>>
> >>> I would make the converse argument here. While I agree that when an API
> >>> is mature, theres no point in calling it experimental anymore, I would
> >>> also suggest that, if an API is mature, and not expected to change,
> >>> theres no harm in leaving its version tag as experimental for an LTS
> >>> release. This is what I was referring to above. Once an application
> >>> developer has done the work to integrate an API from DPDK into its
> >>> application, that work is done. If the API remains stable, then I
> >>> honestly don't know that they care that the version label is EXPERIMENTAL
> >>> versus 20.11 or 20.05 or whatever. They care about the API being stable
> >>> more so than its version label. As such, it seems....reasonable to me to
> >>> have developers queue their migration of experimental APIs to official
> >>> versioned APIs at the next major release deliniation point.
> >>>
> >>> I'd welcome counter arguments, but it seems pretty natural to me to make
> >>> these sorts of changes at the major release mark. People using
> >>> experimantal apis have already implicity agreed to manage changes to
> >>> them, so if we just hold it stable in an LTS release (and don't update
> >>> the version label), its just gravy for them.
> >>>
> >> The counter argument that I see is that while the experimental tag remains
> >> in place the user has no way to know that an API is stable or not, and so
> >> in many projects cannot use the API. If for example an API that is
> >> experimental in 19.11 is unchanged through 20.05 at which point we decide
> >> to promote it to stable. Once the change to the exp tag it is made, any
> >> users of 19.11 can then see that it is an unchanged and now-stable ABI and
> >> can start using it in their software, if they wish, without having to wait
> >> for the 20.11 release. Changing the tag early allows ABIs to be potentially
> >> used immediately.
> >>
> >
> > I would agree with this, however, when using an LTS release, in my mind at
> > least, part of the agreement there is you get stability in the fuctions that
> > were declared stable at the time of release. I'm not sure there should be an
> > expectation of additional stabilization within the lifetime of the release
> > (thats really what the 12 month LTS release cycle is for, no)? You never have
> > to wait more than a year for a new set of stable features. If you want faster
> > feature integration than that, you can choose to enable the experimental API's
> > and accept the benefits and drawbacks thereof.
> >
> > That said, if (as I understand it) the goal is to provide a mechanism to allow
> > experimental features to be promoted to stable status, perhaps we can find a
> > happy middle ground. What if we were to create a construct such as this:
> >
> > #pragma push_macro("ALLOW_EXPERIMENTAL_APIS")
> > #undef ALLOW_EXPERIMENTAL_APIS
> > void __rte_experimental func(...);
> > #pragma pop_macro("ALLOW_EXPERIMENTAL_APIS")
> >
> > Such a consruct would allow the maintainer of an API to pseudo-promote an API
> > from a experimental to stable status, at least so far as compilation would be
> > concerned. Its messy and clunky, and it wouldn't change the function version at
> > all, but the end result would be that:
> > a) such a wraped experimental function would no longer issue a warning when used
> > during compilation/linking
> > and
> > b) provide a semi-easy grepable pattern for application writers to look for when
> > considering the use of an API that was previously experimental
> >
> > such a construct would have to be used very judiciously, in that once its
> > implemented, the API has to be treated as stable, even though its 'excused' from
> > the normal checking, but it could provide something of the more rapid promotion
> > path being sought.
> >
> > Thoughts?
> > Neil
> >> Regards,
> >> /Bruce
> >>
>
>
> Then method above seems clunky, and it is unclear to me if it solves the original problem.
> It seems simpler to me to promote a symbol the next ABI version, and then provide an alias.
>
> At the declaration of the next major ABI version (v21), we would then only need to check
> for those symbols that are both EXPERIMENTAL and v21, to know which aliases need to be removed.
>
> Thanks,
>
> Ray K
>
i don't disagree, just looking for other potential options.
Neil
^ permalink raw reply [relevance 4%]
* [dpdk-dev] [PATCH] doc: plan splitting the ethdev ops struct
@ 2020-02-17 15:38 3% Ferruh Yigit
2020-02-18 5:07 0% ` Jerin Jacob
` (2 more replies)
0 siblings, 3 replies; 200+ results
From: Ferruh Yigit @ 2020-02-17 15:38 UTC (permalink / raw)
To: Neil Horman, John McNamara, Marko Kovacevic
Cc: dev, Ferruh Yigit, David Marchand, Thomas Monjalon, Andrew Rybchenko
For the ABI compatibility it is better to hide internal data structures
from the application as much as possible. But because of some inline
functions 'struct eth_dev_ops' can't be hidden completely.
Plan is to split the 'struct eth_dev_ops' into two as ones used by
inline functions and ones not used, and hide the second part that not
used by inline functions completely to the application.
Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
---
Cc: David Marchand <david.marchand@redhat.com>
Cc: Thomas Monjalon <thomas@monjalon.net>
Cc: Andrew Rybchenko <arybchenko@solarflare.com>
---
doc/guides/rel_notes/deprecation.rst | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
index dfcca87ab..2aa431028 100644
--- a/doc/guides/rel_notes/deprecation.rst
+++ b/doc/guides/rel_notes/deprecation.rst
@@ -72,6 +72,12 @@ Deprecation Notices
In 19.11 PMDs will still update the field even when the offload is not
enabled.
+* ethdev: Split the ``struct eth_dev_ops`` struct to hide it as much as possible.
+ Currently the ``struct eth_dev_ops`` struct is accessible by the application
+ because some inline functions, like ``rte_eth_tx_descriptor_status()``,
+ access the struct directly. The struct will be separate in two, the ops used
+ by inline functions still will be accessible to user but rest will be hidden.
+
* cryptodev: support for using IV with all sizes is added, J0 still can
be used but only when IV length in following structs ``rte_crypto_auth_xform``,
``rte_crypto_aead_xform`` is set to zero. When IV length is greater or equal
--
2.24.1
^ permalink raw reply [relevance 3%]
* [dpdk-dev] [PATCH] net/ionic: fix ABI version
@ 2020-02-17 16:13 7% Ferruh Yigit
2020-02-19 12:37 4% ` Ferruh Yigit
0 siblings, 1 reply; 200+ results
From: Ferruh Yigit @ 2020-02-17 16:13 UTC (permalink / raw)
To: Alfredo Cardigliano, Shannon Nelson; +Cc: dev, Ferruh Yigit, David Marchand
Changed the ABI version to 20.0.1.
Fixes: a72cf98cc93e ("net/ionic: add skeleton")
Reported-by: David Marchand <david.marchand@redhat.com>
Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
---
Cc: Alfredo Cardigliano <cardigliano@ntop.org>
---
drivers/net/ionic/rte_pmd_ionic_version.map | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/drivers/net/ionic/rte_pmd_ionic_version.map b/drivers/net/ionic/rte_pmd_ionic_version.map
index 3fb4ff7eb..bc8fd6d4d 100644
--- a/drivers/net/ionic/rte_pmd_ionic_version.map
+++ b/drivers/net/ionic/rte_pmd_ionic_version.map
@@ -1,5 +1,4 @@
-DPDK_20.0 {
+DPDK_20.0.1 {
local: *;
};
-
--
2.24.1
^ permalink raw reply [relevance 7%]
* Re: [dpdk-dev] [PATCH] ci: build and use libabigail 1.6
2020-02-17 13:59 9% [dpdk-dev] [PATCH] ci: build and use libabigail 1.6 David Marchand
2020-02-17 15:15 0% ` Thomas Monjalon
@ 2020-02-17 18:47 0% ` Aaron Conole
2020-02-18 9:40 3% ` David Marchand
2020-02-18 14:29 9% ` [dpdk-dev] [PATCH v2] " David Marchand
2020-02-18 20:13 9% ` [dpdk-dev] [PATCH v3] " David Marchand
3 siblings, 1 reply; 200+ results
From: Aaron Conole @ 2020-02-17 18:47 UTC (permalink / raw)
To: David Marchand; +Cc: thomas, dev, christian.ehrhardt, dodji, Michael Santana
David Marchand <david.marchand@redhat.com> writes:
> libabigail 1.2 (at least) reports changes in 'const' property as an ABI
> breakage [1].
> This was fixed upstream in libabigail 1.4 [2], and a bug has been opened
> in launchpad [3].
>
> But for now, build and use the last version 1.6 so that the ABI checks
> can be kept.
>
> 1: https://travis-ci.com/DPDK/dpdk/jobs/287872118#L2242
> 2: https://sourceware.org/git/gitweb.cgi?p=libabigail.git;a=commitdiff;h=215b7eb4fe8b986fe1cc87d9d8e7412998038392
> 3: https://bugs.launchpad.net/ubuntu/+source/libabigail/+bug/1863607
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
Does it make sense to base libabigail required ontop of extra packages?
Otherwise some libraries won't get built / checked, no?
> .ci/linux-build.sh | 22 ++++++++++++++++++++++
> .travis.yml | 15 ++++++++++-----
> 2 files changed, 32 insertions(+), 5 deletions(-)
>
> diff --git a/.ci/linux-build.sh b/.ci/linux-build.sh
> index c7c3840fc..0d4bc9a62 100755
> --- a/.ci/linux-build.sh
> +++ b/.ci/linux-build.sh
> @@ -38,6 +38,28 @@ if [ "$AARCH64" != "1" ]; then
> fi
>
> if [ "$ABI_CHECKS" = "1" ]; then
> + LIBABIGAIL_REPO=${LIBABIGAIL_REPO:-https://sourceware.org/git/libabigail.git}
> + LIBABIGAIL_VERSION=${LIBABIGAIL_VERSION:-libabigail-1.6}
> +
> + if [ "$(cat libabigail/VERSION 2>/dev/null)" != "$LIBABIGAIL_VERSION" ]; then
> + rm -rf libabigail
> + # if we change libabigail, invalidate existing abi cache
> + rm -rf reference
> + fi
> +
> + if [ ! -d libabigail ]; then
> + git clone --single-branch -b $LIBABIGAIL_VERSION $LIBABIGAIL_REPO libabigail/src
> + cd libabigail/src && autoconf -vfi && cd -
> + mkdir libabigail/src/build
> + cd libabigail/src/build && ../configure --prefix=$(pwd)/libabigail && cd -
> + make -C libabigail/src/build all install
> +
> + rm -rf libabigail/src
> + echo $LIBABIGAIL_VERSION > libabigail/VERSION
> + fi
> +
> + export PATH=$(pwd)/libabigail/bin:$PATH
> +
> REF_GIT_REPO=${REF_GIT_REPO:-https://dpdk.org/git/dpdk}
> REF_GIT_TAG=${REF_GIT_TAG:-v19.11}
>
> diff --git a/.travis.yml b/.travis.yml
> index 22539d823..d8253fdd4 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -2,6 +2,7 @@ language: c
> cache:
> ccache: true
> directories:
> + - libabigail
> - reference
> compiler:
> - gcc
> @@ -24,7 +25,11 @@ aarch64_packages: &aarch64_packages
>
> extra_packages: &extra_packages
> - *required_packages
> - - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4, abigail-tools]
> + - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
> +
> +libabigail_build_packages: &libabigail_build_packages
> + - *required_packages
> + - [autoconf, automake, libtool, pkg-config, libxml2-dev, libdw-dev]
>
> build_32b_packages: &build_32b_packages
> - *required_packages
> @@ -154,18 +159,18 @@ matrix:
> packages:
> - *required_packages
> - *doc_packages
> - - env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1
> + - env: DEF_LIB="shared" ABI_CHECKS=1
> compiler: gcc
> addons:
> apt:
> packages:
> - - *extra_packages
> - - env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1
> + - *libabigail_build_packages
> + - env: DEF_LIB="shared" ABI_CHECKS=1
> arch: arm64
> compiler: gcc
> addons:
> apt:
> packages:
> - - *extra_packages
> + - *libabigail_build_packages
>
> script: ./.ci/${TRAVIS_OS_NAME}-build.sh
^ permalink raw reply [relevance 0%]
* [dpdk-dev] ABI version of experimental libraries
@ 2020-02-17 23:15 9% Thomas Monjalon
2020-02-17 23:44 11% ` [dpdk-dev] [PATCH] build: fix soname for " Thomas Monjalon
2020-02-18 9:42 7% ` [dpdk-dev] ABI version of " Bruce Richardson
0 siblings, 2 replies; 200+ results
From: Thomas Monjalon @ 2020-02-17 23:15 UTC (permalink / raw)
To: ray.kinsella, nhorman
Cc: bruce.richardson, bluca, david.marchand, ktraynor, dev
Hi,
I would like to remind everybody our mistake when defining ABI versions.
It has been "fixed" in this commit:
http://git.dpdk.org/dpdk/commit/?id=f26c2b39
Please let's think about the consequence for the experimental libraries.
In DPDK 19.11, we use the ABI version 0.200 with soname 0.20
In DPDK 20.02, we use the ABI version 0.2001 with soname 0.201
Numbers are increasing, that's fine.
When we'll switch to the new major ABI and use a normal numbering:
In DPDK 20.11, we will use the ABI version 0.210 with soname 0.21
Numbers are dropping.
In short, for experimental libs, ABI 20.1 > ABI 21.0
Are we OK with this or do we prefer reverting to normal numbering
for experimental libraries in DPDK 20.02?
^ permalink raw reply [relevance 9%]
* [dpdk-dev] [PATCH] build: fix soname for experimental libraries
2020-02-17 23:15 9% [dpdk-dev] ABI version of experimental libraries Thomas Monjalon
@ 2020-02-17 23:44 11% ` Thomas Monjalon
2020-02-18 9:40 0% ` Bruce Richardson
2020-02-18 9:42 7% ` [dpdk-dev] ABI version of " Bruce Richardson
1 sibling, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-02-17 23:44 UTC (permalink / raw)
To: dev
Cc: ray.kinsella, nhorman, bluca, david.marchand, ktraynor, Bruce Richardson
Because of an original mistake in ABI numbering,
and a temporary workaround for ABI 20,
for experimental libs, numbering would lead to consider
ABI 20.1 > ABI 21.0
Before this patch:
DPDK 19.11: ABI version 0.200 and soname 0.20
DPDK 20.02: ABI version 0.2001 and soname 0.201
Numbers are increasing, that's fine.
For the next major ABI, back to normal numbering:
DPDK 20.11: ABI version 0.210 and soname 0.21
Numbers are decreasing!
After this patch:
DPDK 19.11: ABI version 0.200 and soname 0.20
DPDK 20.02: ABI version 0.201 and soname 0.20
DPDK 20.11: ABI version 0.210 and soname 0.21
Fixes: f26c2b39b271 ("build: fix soname info for 19.11 compatibility")
Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
---
config/meson.build | 8 ++++----
mk/rte.lib.mk | 4 ++--
2 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/config/meson.build b/config/meson.build
index 6c46767e3e..e7cd74e2c2 100644
--- a/config/meson.build
+++ b/config/meson.build
@@ -27,12 +27,12 @@ abi_version = run_command(find_program('cat', 'more'),
# and the filename suffix as 0.majorminor versions,
# e.g. v20.1 => librte_stable.so.20.1, librte_experimental.so.0.201
# sonames => librte_stable.so.20, librte_experimental.so.0.20
-# e.g. v20.0.1 => librte_stable.so.20.0.1, librte_experimental.so.0.2001
-# sonames => librte_stable.so.20.0, librte_experimental.so.0.200
+# e.g. v20.0.1 => librte_stable.so.20.0.1, librte_experimental.so.0.201
+# sonames => librte_stable.so.20.0, librte_experimental.so.0.20
abi_va = abi_version.split('.')
stable_so_version = abi_va.length() == 2 ? abi_va[0] : abi_va[0] + '.' + abi_va[1]
-experimental_abi_version = '0.' + ''.join(abi_va)
-experimental_so_version = '0.' + ''.join(stable_so_version.split('.'))
+experimental_abi_version = '0.' + ''.join([abi_va[0], abi_va[2]])
+experimental_so_version = '0.' + ''.join([abi_va[0]])
# extract all version information into the build configuration
dpdk_conf.set('RTE_VER_YEAR', pver.get(0).to_int())
diff --git a/mk/rte.lib.mk b/mk/rte.lib.mk
index b1a8372cc2..8730c084ce 100644
--- a/mk/rte.lib.mk
+++ b/mk/rte.lib.mk
@@ -15,8 +15,8 @@ LIBABIVER ?= $(shell cat $(RTE_SRCDIR)/ABI_VERSION)
SOVER := $(basename $(LIBABIVER))
ifeq ($(shell grep -s "^DPDK_" $(SRCDIR)/$(EXPORT_MAP)),)
# EXPERIMENTAL ABI is versioned as 0.major+minor, e.g. 0.201 for 20.1 ABI
-LIBABIVER := 0.$(shell echo $(LIBABIVER) | tr -d '.')
-SOVER := 0.$(shell echo $(SOVER) | tr -d '.')
+LIBABIVER := 0.$(shell echo $(LIBABIVER) | cut -d '.' -f1,3 | tr -d '.')
+SOVER := 0.$(shell echo $(SOVER) | cut -d '.' -f1)
endif
ifeq ($(CONFIG_RTE_BUILD_SHARED_LIB),y)
--
2.25.0
^ permalink raw reply [relevance 11%]
* Re: [dpdk-dev] [PATCH] doc: plan splitting the ethdev ops struct
2020-02-17 15:38 3% [dpdk-dev] [PATCH] doc: plan splitting the ethdev ops struct Ferruh Yigit
@ 2020-02-18 5:07 0% ` Jerin Jacob
2020-02-25 12:42 0% ` Ferruh Yigit
2020-02-18 6:01 0% ` Stephen Hemminger
2020-02-25 12:44 3% ` [dpdk-dev] [PATCH v2] " Ferruh Yigit
2 siblings, 1 reply; 200+ results
From: Jerin Jacob @ 2020-02-18 5:07 UTC (permalink / raw)
To: Ferruh Yigit
Cc: Neil Horman, John McNamara, Marko Kovacevic, dpdk-dev,
David Marchand, Thomas Monjalon, Andrew Rybchenko
On Mon, Feb 17, 2020 at 9:08 PM Ferruh Yigit <ferruh.yigit@intel.com> wrote:
>
> For the ABI compatibility it is better to hide internal data structures
> from the application as much as possible. But because of some inline
> functions 'struct eth_dev_ops' can't be hidden completely.
>
> Plan is to split the 'struct eth_dev_ops' into two as ones used by
> inline functions and ones not used, and hide the second part that not
> used by inline functions completely to the application.
It is a good improvement. IMO, If anything used in fast-path it
should be in ``struct rte_eth_dev``
and rest can completely be moved to internal. In this case, if
`rte_eth_tx_descriptor_status`
not used on fastpath, Maybe we don't need to maintain the inline
status and move completely
to .c file.
Those may be specifics of the work. In general, this change looks good to me.
Acked-by: Jerin Jacob <jerinj@marvell.com>
>
> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
> ---
> Cc: David Marchand <david.marchand@redhat.com>
> Cc: Thomas Monjalon <thomas@monjalon.net>
> Cc: Andrew Rybchenko <arybchenko@solarflare.com>
> ---
> doc/guides/rel_notes/deprecation.rst | 6 ++++++
> 1 file changed, 6 insertions(+)
>
> diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
> index dfcca87ab..2aa431028 100644
> --- a/doc/guides/rel_notes/deprecation.rst
> +++ b/doc/guides/rel_notes/deprecation.rst
> @@ -72,6 +72,12 @@ Deprecation Notices
> In 19.11 PMDs will still update the field even when the offload is not
> enabled.
>
> +* ethdev: Split the ``struct eth_dev_ops`` struct to hide it as much as possible.
> + Currently the ``struct eth_dev_ops`` struct is accessible by the application
> + because some inline functions, like ``rte_eth_tx_descriptor_status()``,
> + access the struct directly. The struct will be separate in two, the ops used
> + by inline functions still will be accessible to user but rest will be hidden.
> +
> * cryptodev: support for using IV with all sizes is added, J0 still can
> be used but only when IV length in following structs ``rte_crypto_auth_xform``,
> ``rte_crypto_aead_xform`` is set to zero. When IV length is greater or equal
> --
> 2.24.1
>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] doc: plan splitting the ethdev ops struct
2020-02-17 15:38 3% [dpdk-dev] [PATCH] doc: plan splitting the ethdev ops struct Ferruh Yigit
2020-02-18 5:07 0% ` Jerin Jacob
@ 2020-02-18 6:01 0% ` Stephen Hemminger
2020-02-21 10:40 0% ` Ferruh Yigit
2020-02-25 12:44 3% ` [dpdk-dev] [PATCH v2] " Ferruh Yigit
2 siblings, 1 reply; 200+ results
From: Stephen Hemminger @ 2020-02-18 6:01 UTC (permalink / raw)
To: Ferruh Yigit
Cc: Neil Horman, John McNamara, Marko Kovacevic, dev, David Marchand,
Thomas Monjalon, Andrew Rybchenko
On Mon, 17 Feb 2020 15:38:05 +0000
Ferruh Yigit <ferruh.yigit@intel.com> wrote:
> For the ABI compatibility it is better to hide internal data structures
> from the application as much as possible. But because of some inline
> functions 'struct eth_dev_ops' can't be hidden completely.
>
> Plan is to split the 'struct eth_dev_ops' into two as ones used by
> inline functions and ones not used, and hide the second part that not
> used by inline functions completely to the application.
>
> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
> ---
> Cc: David Marchand <david.marchand@redhat.com>
> Cc: Thomas Monjalon <thomas@monjalon.net>
> Cc: Andrew Rybchenko <arybchenko@solarflare.com>
> ---
> doc/guides/rel_notes/deprecation.rst | 6 ++++++
> 1 file changed, 6 insertions(+)
>
> diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
> index dfcca87ab..2aa431028 100644
> --- a/doc/guides/rel_notes/deprecation.rst
> +++ b/doc/guides/rel_notes/deprecation.rst
> @@ -72,6 +72,12 @@ Deprecation Notices
> In 19.11 PMDs will still update the field even when the offload is not
> enabled.
>
> +* ethdev: Split the ``struct eth_dev_ops`` struct to hide it as much as possible.
> + Currently the ``struct eth_dev_ops`` struct is accessible by the application
> + because some inline functions, like ``rte_eth_tx_descriptor_status()``,
> + access the struct directly. The struct will be separate in two, the ops used
> + by inline functions still will be accessible to user but rest will be hidden.
> +
> * cryptodev: support for using IV with all sizes is added, J0 still can
> be used but only when IV length in following structs ``rte_crypto_auth_xform``,
> ``rte_crypto_aead_xform`` is set to zero. When IV length is greater or equal
Good luck, truely hiding internals is hard. The mbuf structure is already split but not really
hidden at all (just look at dwarf output). It doesn't make sense to do it unless
you can really hide it.
I would attack the rte_device stuff first. Make rte_device opaque to the application
that would help for future versions. Then work backwards to rte_tehtdev.
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] build: fix soname for experimental libraries
2020-02-17 23:44 11% ` [dpdk-dev] [PATCH] build: fix soname for " Thomas Monjalon
@ 2020-02-18 9:40 0% ` Bruce Richardson
2020-02-18 9:47 0% ` Thomas Monjalon
0 siblings, 1 reply; 200+ results
From: Bruce Richardson @ 2020-02-18 9:40 UTC (permalink / raw)
To: Thomas Monjalon
Cc: dev, ray.kinsella, nhorman, bluca, david.marchand, ktraynor
On Tue, Feb 18, 2020 at 12:44:02AM +0100, Thomas Monjalon wrote:
> Because of an original mistake in ABI numbering,
> and a temporary workaround for ABI 20,
> for experimental libs, numbering would lead to consider
> ABI 20.1 > ABI 21.0
>
> Before this patch:
>
> DPDK 19.11: ABI version 0.200 and soname 0.20
> DPDK 20.02: ABI version 0.2001 and soname 0.201
> Numbers are increasing, that's fine.
> For the next major ABI, back to normal numbering:
> DPDK 20.11: ABI version 0.210 and soname 0.21
> Numbers are decreasing!
>
> After this patch:
>
> DPDK 19.11: ABI version 0.200 and soname 0.20
> DPDK 20.02: ABI version 0.201 and soname 0.20
> DPDK 20.11: ABI version 0.210 and soname 0.21
>
> Fixes: f26c2b39b271 ("build: fix soname info for 19.11 compatibility")
>
> Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
> ---
> config/meson.build | 8 ++++----
> mk/rte.lib.mk | 4 ++--
> 2 files changed, 6 insertions(+), 6 deletions(-)
>
> diff --git a/config/meson.build b/config/meson.build
> index 6c46767e3e..e7cd74e2c2 100644
> --- a/config/meson.build
> +++ b/config/meson.build
> @@ -27,12 +27,12 @@ abi_version = run_command(find_program('cat', 'more'),
> # and the filename suffix as 0.majorminor versions,
> # e.g. v20.1 => librte_stable.so.20.1, librte_experimental.so.0.201
> # sonames => librte_stable.so.20, librte_experimental.so.0.20
> -# e.g. v20.0.1 => librte_stable.so.20.0.1, librte_experimental.so.0.2001
> -# sonames => librte_stable.so.20.0, librte_experimental.so.0.200
> +# e.g. v20.0.1 => librte_stable.so.20.0.1, librte_experimental.so.0.201
> +# sonames => librte_stable.so.20.0, librte_experimental.so.0.20
> abi_va = abi_version.split('.')
> stable_so_version = abi_va.length() == 2 ? abi_va[0] : abi_va[0] + '.' + abi_va[1]
> -experimental_abi_version = '0.' + ''.join(abi_va)
> -experimental_so_version = '0.' + ''.join(stable_so_version.split('.'))
> +experimental_abi_version = '0.' + ''.join([abi_va[0], abi_va[2]])
> +experimental_so_version = '0.' + ''.join([abi_va[0]])
>
My concern about this is that it will break, or rather need to be changed
again for the 20.11 release. While I see the numbering as not-ideal in
terms of version numbers, the existing scheme was originally designed to
work with either 3-digit or 2-digit version numbers.
/Bruce
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] ci: build and use libabigail 1.6
2020-02-17 18:47 0% ` Aaron Conole
@ 2020-02-18 9:40 3% ` David Marchand
2020-02-18 11:18 0% ` David Marchand
0 siblings, 1 reply; 200+ results
From: David Marchand @ 2020-02-18 9:40 UTC (permalink / raw)
To: Aaron Conole
Cc: Thomas Monjalon, dev, Christian Ehrhardt, Dodji Seketeli,
Michael Santana
On Mon, Feb 17, 2020 at 7:48 PM Aaron Conole <aconole@redhat.com> wrote:
>
> David Marchand <david.marchand@redhat.com> writes:
>
> > libabigail 1.2 (at least) reports changes in 'const' property as an ABI
> > breakage [1].
> > This was fixed upstream in libabigail 1.4 [2], and a bug has been opened
> > in launchpad [3].
> >
> > But for now, build and use the last version 1.6 so that the ABI checks
> > can be kept.
> >
> > 1: https://travis-ci.com/DPDK/dpdk/jobs/287872118#L2242
> > 2: https://sourceware.org/git/gitweb.cgi?p=libabigail.git;a=commitdiff;h=215b7eb4fe8b986fe1cc87d9d8e7412998038392
> > 3: https://bugs.launchpad.net/ubuntu/+source/libabigail/+bug/1863607
> >
> > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > ---
>
> Does it make sense to base libabigail required ontop of extra packages?
> Otherwise some libraries won't get built / checked, no?
The only change I see is the pcap driver being enabled.
On the principle, I agree that trying to build all possible
libraries/drivers is better when checking the ABI.
So I'll keep extra_packages yes.
I am currently testing that touching extra_packages (well, testing
Thomas patches) results in Travis treating the job as a new one (i.e.
with no cache).
>
> > .ci/linux-build.sh | 22 ++++++++++++++++++++++
> > .travis.yml | 15 ++++++++++-----
> > 2 files changed, 32 insertions(+), 5 deletions(-)
> >
> > diff --git a/.ci/linux-build.sh b/.ci/linux-build.sh
> > index c7c3840fc..0d4bc9a62 100755
> > --- a/.ci/linux-build.sh
> > +++ b/.ci/linux-build.sh
> > @@ -38,6 +38,28 @@ if [ "$AARCH64" != "1" ]; then
> > fi
> >
> > if [ "$ABI_CHECKS" = "1" ]; then
> > + LIBABIGAIL_REPO=${LIBABIGAIL_REPO:-https://sourceware.org/git/libabigail.git}
> > + LIBABIGAIL_VERSION=${LIBABIGAIL_VERSION:-libabigail-1.6}
> > +
> > + if [ "$(cat libabigail/VERSION 2>/dev/null)" != "$LIBABIGAIL_VERSION" ]; then
> > + rm -rf libabigail
> > + # if we change libabigail, invalidate existing abi cache
> > + rm -rf reference
> > + fi
> > +
> > + if [ ! -d libabigail ]; then
> > + git clone --single-branch -b $LIBABIGAIL_VERSION $LIBABIGAIL_REPO libabigail/src
> > + cd libabigail/src && autoconf -vfi && cd -
And I managed to send a "oh yeah, seems better this way"/untested version...
Fixed.
--
David Marchand
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] ABI version of experimental libraries
2020-02-17 23:15 9% [dpdk-dev] ABI version of experimental libraries Thomas Monjalon
2020-02-17 23:44 11% ` [dpdk-dev] [PATCH] build: fix soname for " Thomas Monjalon
@ 2020-02-18 9:42 7% ` Bruce Richardson
2020-02-18 9:50 4% ` Thomas Monjalon
1 sibling, 1 reply; 200+ results
From: Bruce Richardson @ 2020-02-18 9:42 UTC (permalink / raw)
To: Thomas Monjalon
Cc: ray.kinsella, nhorman, bluca, david.marchand, ktraynor, dev
On Tue, Feb 18, 2020 at 12:15:56AM +0100, Thomas Monjalon wrote:
> Hi,
>
> I would like to remind everybody our mistake when defining ABI versions.
> It has been "fixed" in this commit:
> http://git.dpdk.org/dpdk/commit/?id=f26c2b39
>
> Please let's think about the consequence for the experimental libraries.
>
> In DPDK 19.11, we use the ABI version 0.200 with soname 0.20 In DPDK
> 20.02, we use the ABI version 0.2001 with soname 0.201 Numbers are
> increasing, that's fine. When we'll switch to the new major ABI and use
> a normal numbering: In DPDK 20.11, we will use the ABI version 0.210 with
> soname 0.21 Numbers are dropping.
>
> In short, for experimental libs, ABI 20.1 > ABI 21.0
>
> Are we OK with this or do we prefer reverting to normal numbering for
> experimental libraries in DPDK 20.02?
>
Personally, I would not be too concerned about the verions of experimental
libs, so long as they don't conflict across versions and have some
similarity to the major ABI version for the release.
/Bruce
^ permalink raw reply [relevance 7%]
* Re: [dpdk-dev] [PATCH] build: fix soname for experimental libraries
2020-02-18 9:40 0% ` Bruce Richardson
@ 2020-02-18 9:47 0% ` Thomas Monjalon
0 siblings, 0 replies; 200+ results
From: Thomas Monjalon @ 2020-02-18 9:47 UTC (permalink / raw)
To: Bruce Richardson
Cc: dev, ray.kinsella, nhorman, bluca, david.marchand, ktraynor
18/02/2020 10:40, Bruce Richardson:
> On Tue, Feb 18, 2020 at 12:44:02AM +0100, Thomas Monjalon wrote:
> > Because of an original mistake in ABI numbering,
> > and a temporary workaround for ABI 20,
> > for experimental libs, numbering would lead to consider
> > ABI 20.1 > ABI 21.0
> >
> > Before this patch:
> >
> > DPDK 19.11: ABI version 0.200 and soname 0.20
> > DPDK 20.02: ABI version 0.2001 and soname 0.201
> > Numbers are increasing, that's fine.
> > For the next major ABI, back to normal numbering:
> > DPDK 20.11: ABI version 0.210 and soname 0.21
> > Numbers are decreasing!
> >
> > After this patch:
> >
> > DPDK 19.11: ABI version 0.200 and soname 0.20
> > DPDK 20.02: ABI version 0.201 and soname 0.20
> > DPDK 20.11: ABI version 0.210 and soname 0.21
> >
> > Fixes: f26c2b39b271 ("build: fix soname info for 19.11 compatibility")
> >
> > Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
> > ---
> > config/meson.build | 8 ++++----
> > mk/rte.lib.mk | 4 ++--
> > 2 files changed, 6 insertions(+), 6 deletions(-)
> >
> > diff --git a/config/meson.build b/config/meson.build
> > index 6c46767e3e..e7cd74e2c2 100644
> > --- a/config/meson.build
> > +++ b/config/meson.build
> > @@ -27,12 +27,12 @@ abi_version = run_command(find_program('cat', 'more'),
> > # and the filename suffix as 0.majorminor versions,
> > # e.g. v20.1 => librte_stable.so.20.1, librte_experimental.so.0.201
> > # sonames => librte_stable.so.20, librte_experimental.so.0.20
> > -# e.g. v20.0.1 => librte_stable.so.20.0.1, librte_experimental.so.0.2001
> > -# sonames => librte_stable.so.20.0, librte_experimental.so.0.200
> > +# e.g. v20.0.1 => librte_stable.so.20.0.1, librte_experimental.so.0.201
> > +# sonames => librte_stable.so.20.0, librte_experimental.so.0.20
> > abi_va = abi_version.split('.')
> > stable_so_version = abi_va.length() == 2 ? abi_va[0] : abi_va[0] + '.' + abi_va[1]
> > -experimental_abi_version = '0.' + ''.join(abi_va)
> > -experimental_so_version = '0.' + ''.join(stable_so_version.split('.'))
> > +experimental_abi_version = '0.' + ''.join([abi_va[0], abi_va[2]])
> > +experimental_so_version = '0.' + ''.join([abi_va[0]])
> >
>
> My concern about this is that it will break, or rather need to be changed
> again for the 20.11 release. While I see the numbering as not-ideal in
> terms of version numbers, the existing scheme was originally designed to
> work with either 3-digit or 2-digit version numbers.
It could be improved to work with 2-digit too.
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] ABI version of experimental libraries
2020-02-18 9:42 7% ` [dpdk-dev] ABI version of " Bruce Richardson
@ 2020-02-18 9:50 4% ` Thomas Monjalon
2020-02-18 10:36 4% ` Kinsella, Ray
2020-02-19 11:43 7% ` [dpdk-dev] ABI version of experimental libraries Neil Horman
0 siblings, 2 replies; 200+ results
From: Thomas Monjalon @ 2020-02-18 9:50 UTC (permalink / raw)
To: Bruce Richardson
Cc: ray.kinsella, nhorman, bluca, david.marchand, ktraynor, dev
18/02/2020 10:42, Bruce Richardson:
> On Tue, Feb 18, 2020 at 12:15:56AM +0100, Thomas Monjalon wrote:
> > Hi,
> >
> > I would like to remind everybody our mistake when defining ABI versions.
> > It has been "fixed" in this commit:
> > http://git.dpdk.org/dpdk/commit/?id=f26c2b39
> >
> > Please let's think about the consequence for the experimental libraries.
> >
> > In DPDK 19.11, we use the ABI version 0.200 with soname 0.20 In DPDK
> > 20.02, we use the ABI version 0.2001 with soname 0.201 Numbers are
> > increasing, that's fine. When we'll switch to the new major ABI and use
> > a normal numbering: In DPDK 20.11, we will use the ABI version 0.210 with
> > soname 0.21 Numbers are dropping.
> >
> > In short, for experimental libs, ABI 20.1 > ABI 21.0
> >
> > Are we OK with this or do we prefer reverting to normal numbering for
> > experimental libraries in DPDK 20.02?
> >
> Personally, I would not be too concerned about the verions of experimental
> libs, so long as they don't conflict across versions and have some
> similarity to the major ABI version for the release.
You think sorting of the version numbers is not important?
If we don't care comparing experimental version numbers,
then OK, let's drop this patch. But please we need a small vote.
Note: there would be no problem if we did not vote for having
a special numbering for pure experimental libraries (I am still against).
^ permalink raw reply [relevance 4%]
* [dpdk-dev] [PATCH v4] doc: alias to experimental tag for stable apis
@ 2020-02-18 10:46 17% ` Ray Kinsella
2020-02-21 17:11 0% ` Ferruh Yigit
2020-02-25 14:22 0% ` Kevin Traynor
2020-02-25 15:35 0% ` [dpdk-dev] [PATCH] " Neil Horman
1 sibling, 2 replies; 200+ results
From: Ray Kinsella @ 2020-02-18 10:46 UTC (permalink / raw)
To: dev
Cc: john.mcnamara, nhorman, ferruh.yigit, marko.kovacevic,
bruce.richardson, Ray Kinsella
When a maintainer is promoting an API to become part of the next major ABI
version by removing the experimental tag, possibly a few releases in
advance of the declaration of the next ABI version. The maintainer may
choose to offer an alias to the experimental tag, as removing the tag
before the declaration of the next major ABI version, would cause an ABI
breakage for applications using the API.
Signed-off-by: Ray Kinsella <mdr@ashroe.eu>
---
doc/guides/contributing/abi_policy.rst | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/doc/guides/contributing/abi_policy.rst b/doc/guides/contributing/abi_policy.rst
index 05ca959..7ae7de7 100644
--- a/doc/guides/contributing/abi_policy.rst
+++ b/doc/guides/contributing/abi_policy.rst
@@ -159,6 +159,11 @@ The requirements for changing the ABI are:
``experimental``, as described in the section on :ref:`Experimental APIs
and Libraries <experimental_apis>`.
+ - In situations in which an ``experimental`` API has been stable for some
+ time. When promoting the API to become part of the next ABI version, the
+ maintainer may choose to provide an alias to the ``experimental`` tag, so
+ as not to break consuming applications.
+
#. If a newly proposed API functionally replaces an existing one, when the new
API becomes non-experimental, then the old one is marked with
``__rte_deprecated``.
@@ -317,6 +322,11 @@ not required. Though, an API should remain in experimental state for at least
one release. Thereafter, the normal process of posting patch for review to
mailing list can be followed.
+After the experimental tag has been formally removed, a tree/sub-tree maintainer
+may choose to offer an alias to the experimental tag so as not to break
+applications using the API. The alias is then dropped at the declaration of next
+major ABI version.
+
Libraries
~~~~~~~~~
--
2.7.4
^ permalink raw reply [relevance 17%]
* [dpdk-dev] [PATCH] ethdev: add comment to warn of ABI breakage
@ 2020-02-18 10:46 13% Ciara Power
2020-02-18 11:59 4% ` Ferruh Yigit
2020-02-18 13:39 13% ` [dpdk-dev] [PATCH v2] " Ciara Power
0 siblings, 2 replies; 200+ results
From: Ciara Power @ 2020-02-18 10:46 UTC (permalink / raw)
To: thomas, ferruh.yigit, arybchenko; +Cc: dev, Ciara Power
If a function is added to the eth_dev_ops struct before
tx_descriptor_status function, this will cause ABI breakage. This is due
to static inline functions using this function, and some other functions
above it in the struct, so they cannot change position. A comment is
added to inform developers of this possible breakage.
Signed-off-by: Ciara Power <ciara.power@intel.com>
---
lib/librte_ethdev/rte_ethdev_core.h | 3 +++
1 file changed, 3 insertions(+)
diff --git a/lib/librte_ethdev/rte_ethdev_core.h b/lib/librte_ethdev/rte_ethdev_core.h
index 7bf97e24e..1ad04a129 100644
--- a/lib/librte_ethdev/rte_ethdev_core.h
+++ b/lib/librte_ethdev/rte_ethdev_core.h
@@ -667,6 +667,9 @@ struct eth_dev_ops {
/**< Check the status of a Rx descriptor. */
eth_tx_descriptor_status_t tx_descriptor_status;
/**< Check the status of a Tx descriptor. */
+ /* Static inline functions use functions above this comment.
+ * New dev_ops functions should be added below to avoid breaking ABI.
+ */
eth_rx_enable_intr_t rx_queue_intr_enable; /**< Enable Rx queue interrupt. */
eth_rx_disable_intr_t rx_queue_intr_disable; /**< Disable Rx queue interrupt. */
eth_tx_queue_setup_t tx_queue_setup;/**< Set up device TX queue. */
--
2.17.1
^ permalink raw reply [relevance 13%]
* Re: [dpdk-dev] [PATCH] ci: build and use libabigail 1.6
2020-02-18 9:40 3% ` David Marchand
@ 2020-02-18 11:18 0% ` David Marchand
2020-02-18 14:55 0% ` Aaron Conole
0 siblings, 1 reply; 200+ results
From: David Marchand @ 2020-02-18 11:18 UTC (permalink / raw)
To: Aaron Conole
Cc: Thomas Monjalon, dev, Christian Ehrhardt, Dodji Seketeli,
Michael Santana
On Tue, Feb 18, 2020 at 10:40 AM David Marchand
<david.marchand@redhat.com> wrote:
>
> On Mon, Feb 17, 2020 at 7:48 PM Aaron Conole <aconole@redhat.com> wrote:
> >
> > David Marchand <david.marchand@redhat.com> writes:
> >
> > > libabigail 1.2 (at least) reports changes in 'const' property as an ABI
> > > breakage [1].
> > > This was fixed upstream in libabigail 1.4 [2], and a bug has been opened
> > > in launchpad [3].
> > >
> > > But for now, build and use the last version 1.6 so that the ABI checks
> > > can be kept.
> > >
> > > 1: https://travis-ci.com/DPDK/dpdk/jobs/287872118#L2242
> > > 2: https://sourceware.org/git/gitweb.cgi?p=libabigail.git;a=commitdiff;h=215b7eb4fe8b986fe1cc87d9d8e7412998038392
> > > 3: https://bugs.launchpad.net/ubuntu/+source/libabigail/+bug/1863607
> > >
> > > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > > ---
> >
> > Does it make sense to base libabigail required ontop of extra packages?
> > Otherwise some libraries won't get built / checked, no?
>
> The only change I see is the pcap driver being enabled.
> On the principle, I agree that trying to build all possible
> libraries/drivers is better when checking the ABI.
> So I'll keep extra_packages yes.
>
> I am currently testing that touching extra_packages (well, testing
> Thomas patches) results in Travis treating the job as a new one (i.e.
> with no cache).
Travis bases each job cache on the job description:
https://docs.travis-ci.com/user/caching/
I tested Thomas change on extra_packages content, and the job used the
old cache.
My idea was to try to put *extra_packages in an env variable, but it
does not work (my yaml-fu is lacking).
If there is no easy way, I will invalidate the cache manually.
--
David Marchand
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] ethdev: add comment to warn of ABI breakage
2020-02-18 10:46 13% [dpdk-dev] [PATCH] ethdev: add comment to warn of ABI breakage Ciara Power
@ 2020-02-18 11:59 4% ` Ferruh Yigit
2020-02-18 13:13 4% ` Andrew Rybchenko
2020-02-18 13:39 13% ` [dpdk-dev] [PATCH v2] " Ciara Power
1 sibling, 1 reply; 200+ results
From: Ferruh Yigit @ 2020-02-18 11:59 UTC (permalink / raw)
To: Ciara Power, thomas, arybchenko; +Cc: dev
On 2/18/2020 10:46 AM, Ciara Power wrote:
> If a function is added to the eth_dev_ops struct before
> tx_descriptor_status function, this will cause ABI breakage. This is due
> to static inline functions using this function, and some other functions
> above it in the struct, so they cannot change position. A comment is
> added to inform developers of this possible breakage.
>
> Signed-off-by: Ciara Power <ciara.power@intel.com>
> ---
> lib/librte_ethdev/rte_ethdev_core.h | 3 +++
> 1 file changed, 3 insertions(+)
>
> diff --git a/lib/librte_ethdev/rte_ethdev_core.h b/lib/librte_ethdev/rte_ethdev_core.h
> index 7bf97e24e..1ad04a129 100644
> --- a/lib/librte_ethdev/rte_ethdev_core.h
> +++ b/lib/librte_ethdev/rte_ethdev_core.h
> @@ -667,6 +667,9 @@ struct eth_dev_ops {
> /**< Check the status of a Rx descriptor. */
> eth_tx_descriptor_status_t tx_descriptor_status;
> /**< Check the status of a Tx descriptor. */
> + /* Static inline functions use functions above this comment.
> + * New dev_ops functions should be added below to avoid breaking ABI.
> + */
> eth_rx_enable_intr_t rx_queue_intr_enable; /**< Enable Rx queue interrupt. */
> eth_rx_disable_intr_t rx_queue_intr_disable; /**< Disable Rx queue interrupt. */
> eth_tx_queue_setup_t tx_queue_setup;/**< Set up device TX queue. */
>
Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH] ethdev: add comment to warn of ABI breakage
2020-02-18 11:59 4% ` Ferruh Yigit
@ 2020-02-18 13:13 4% ` Andrew Rybchenko
2020-02-18 14:20 4% ` Thomas Monjalon
0 siblings, 1 reply; 200+ results
From: Andrew Rybchenko @ 2020-02-18 13:13 UTC (permalink / raw)
To: Ferruh Yigit, Ciara Power, thomas, arybchenko; +Cc: dev
On 2/18/20 2:59 PM, Ferruh Yigit wrote:
> On 2/18/2020 10:46 AM, Ciara Power wrote:
>> If a function is added to the eth_dev_ops struct before
>> tx_descriptor_status function, this will cause ABI breakage. This is due
>> to static inline functions using this function, and some other functions
>> above it in the struct, so they cannot change position. A comment is
>> added to inform developers of this possible breakage.
>>
>> Signed-off-by: Ciara Power <ciara.power@intel.com>
>> ---
>> lib/librte_ethdev/rte_ethdev_core.h | 3 +++
>> 1 file changed, 3 insertions(+)
>>
>> diff --git a/lib/librte_ethdev/rte_ethdev_core.h b/lib/librte_ethdev/rte_ethdev_core.h
>> index 7bf97e24e..1ad04a129 100644
>> --- a/lib/librte_ethdev/rte_ethdev_core.h
>> +++ b/lib/librte_ethdev/rte_ethdev_core.h
>> @@ -667,6 +667,9 @@ struct eth_dev_ops {
>> /**< Check the status of a Rx descriptor. */
>> eth_tx_descriptor_status_t tx_descriptor_status;
>> /**< Check the status of a Tx descriptor. */
>> + /* Static inline functions use functions above this comment.
>> + * New dev_ops functions should be added below to avoid breaking ABI.
>> + */
>> eth_rx_enable_intr_t rx_queue_intr_enable; /**< Enable Rx queue interrupt. */
>> eth_rx_disable_intr_t rx_queue_intr_disable; /**< Disable Rx queue interrupt. */
>> eth_tx_queue_setup_t tx_queue_setup;/**< Set up device TX queue. */
>>
>
> Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
LGTM, but may I suggest to make the first line in the comment
empty to make it easier to notice. I.e.
/*
* Static inline functions use functions above this comment.
* ...
*/
Or even add empty lines before and after the comment.
Reviewed-by: Andrew Rybchenko <arybchenko@solarflare.com>
^ permalink raw reply [relevance 4%]
* [dpdk-dev] [PATCH v2] ethdev: add comment to warn of ABI breakage
2020-02-18 10:46 13% [dpdk-dev] [PATCH] ethdev: add comment to warn of ABI breakage Ciara Power
2020-02-18 11:59 4% ` Ferruh Yigit
@ 2020-02-18 13:39 13% ` Ciara Power
2020-02-18 18:40 4% ` Ferruh Yigit
1 sibling, 1 reply; 200+ results
From: Ciara Power @ 2020-02-18 13:39 UTC (permalink / raw)
To: thomas, ferruh.yigit, arybchenko; +Cc: dev, Ciara Power
If a function is added to the eth_dev_ops struct before
tx_descriptor_status function, this will cause ABI breakage. This is due
to static inline functions using this function, and some other functions
above it in the struct, so they cannot change position. A comment is
added to inform developers of this possible breakage.
Signed-off-by: Ciara Power <ciara.power@intel.com>
Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
Reviewed-by: Andrew Rybchenko <arybchenko@solarflare.com>
---
V2: Added empty first line in comment
---
lib/librte_ethdev/rte_ethdev_core.h | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/lib/librte_ethdev/rte_ethdev_core.h b/lib/librte_ethdev/rte_ethdev_core.h
index 7bf97e24e..9933ffdc1 100644
--- a/lib/librte_ethdev/rte_ethdev_core.h
+++ b/lib/librte_ethdev/rte_ethdev_core.h
@@ -667,6 +667,10 @@ struct eth_dev_ops {
/**< Check the status of a Rx descriptor. */
eth_tx_descriptor_status_t tx_descriptor_status;
/**< Check the status of a Tx descriptor. */
+ /*
+ * Static inline functions use functions above this comment.
+ * New dev_ops functions should be added below to avoid breaking ABI.
+ */
eth_rx_enable_intr_t rx_queue_intr_enable; /**< Enable Rx queue interrupt. */
eth_rx_disable_intr_t rx_queue_intr_disable; /**< Disable Rx queue interrupt. */
eth_tx_queue_setup_t tx_queue_setup;/**< Set up device TX queue. */
--
2.17.1
^ permalink raw reply [relevance 13%]
* Re: [dpdk-dev] [PATCH] ethdev: add comment to warn of ABI breakage
2020-02-18 13:13 4% ` Andrew Rybchenko
@ 2020-02-18 14:20 4% ` Thomas Monjalon
0 siblings, 0 replies; 200+ results
From: Thomas Monjalon @ 2020-02-18 14:20 UTC (permalink / raw)
To: Ciara Power; +Cc: Ferruh Yigit, Andrew Rybchenko, dev
18/02/2020 14:13, Andrew Rybchenko:
> On 2/18/20 2:59 PM, Ferruh Yigit wrote:
> > On 2/18/2020 10:46 AM, Ciara Power wrote:
> >> If a function is added to the eth_dev_ops struct before
> >> tx_descriptor_status function, this will cause ABI breakage. This is due
> >> to static inline functions using this function, and some other functions
> >> above it in the struct, so they cannot change position. A comment is
> >> added to inform developers of this possible breakage.
> >>
> >> Signed-off-by: Ciara Power <ciara.power@intel.com>
> >> ---
> >> lib/librte_ethdev/rte_ethdev_core.h | 3 +++
> >> 1 file changed, 3 insertions(+)
> >>
> >> diff --git a/lib/librte_ethdev/rte_ethdev_core.h b/lib/librte_ethdev/rte_ethdev_core.h
> >> index 7bf97e24e..1ad04a129 100644
> >> --- a/lib/librte_ethdev/rte_ethdev_core.h
> >> +++ b/lib/librte_ethdev/rte_ethdev_core.h
> >> @@ -667,6 +667,9 @@ struct eth_dev_ops {
> >> /**< Check the status of a Rx descriptor. */
> >> eth_tx_descriptor_status_t tx_descriptor_status;
> >> /**< Check the status of a Tx descriptor. */
> >> + /* Static inline functions use functions above this comment.
> >> + * New dev_ops functions should be added below to avoid breaking ABI.
> >> + */
> >> eth_rx_enable_intr_t rx_queue_intr_enable; /**< Enable Rx queue interrupt. */
> >> eth_rx_disable_intr_t rx_queue_intr_disable; /**< Disable Rx queue interrupt. */
> >> eth_tx_queue_setup_t tx_queue_setup;/**< Set up device TX queue. */
> >>
> >
> > Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
>
> LGTM, but may I suggest to make the first line in the comment
> empty to make it easier to notice. I.e.
> /*
> * Static inline functions use functions above this comment.
> * ...
> */
>
> Or even add empty lines before and after the comment.
>
> Reviewed-by: Andrew Rybchenko <arybchenko@solarflare.com>
In the same idea as Andrew, I would suggest some uppercases,
maybe "ABOVE" and "BELOW".
Acked-by: Thomas Monjalon <thomas@monjalon.net>
^ permalink raw reply [relevance 4%]
* [dpdk-dev] [PATCH v2] ci: build and use libabigail 1.6
2020-02-17 13:59 9% [dpdk-dev] [PATCH] ci: build and use libabigail 1.6 David Marchand
2020-02-17 15:15 0% ` Thomas Monjalon
2020-02-17 18:47 0% ` Aaron Conole
@ 2020-02-18 14:29 9% ` David Marchand
2020-02-18 15:46 0% ` Thomas Monjalon
2020-02-18 20:13 9% ` [dpdk-dev] [PATCH v3] " David Marchand
3 siblings, 1 reply; 200+ results
From: David Marchand @ 2020-02-18 14:29 UTC (permalink / raw)
To: thomas
Cc: dev, aconole, christian.ehrhardt, dodji, kevin.laatz, Michael Santana
libabigail 1.2 (at least) reports changes in 'const' property as an ABI
breakage [1].
This was fixed upstream in libabigail 1.4 [2], and a bug has been opened
in launchpad [3].
But for now, build and use the last version 1.6 so that the ABI checks
can be kept.
1: https://travis-ci.com/DPDK/dpdk/jobs/287872118#L2242
2: https://sourceware.org/git/gitweb.cgi?p=libabigail.git;a=commitdiff;h=215b7eb4fe8b
3: https://bugs.launchpad.net/ubuntu/+source/libabigail/+bug/1863607
Signed-off-by: David Marchand <david.marchand@redhat.com>
---
Changelog since v1:
- keep extra_packages in ABI checks jobs,
- fix configure step,
---
.ci/linux-build.sh | 23 +++++++++++++++++++++++
.travis.yml | 8 +++++++-
2 files changed, 30 insertions(+), 1 deletion(-)
diff --git a/.ci/linux-build.sh b/.ci/linux-build.sh
index c7c3840fc..cb9627bd5 100755
--- a/.ci/linux-build.sh
+++ b/.ci/linux-build.sh
@@ -38,6 +38,29 @@ if [ "$AARCH64" != "1" ]; then
fi
if [ "$ABI_CHECKS" = "1" ]; then
+ LIBABIGAIL_REPO=${LIBABIGAIL_REPO:-https://sourceware.org/git/libabigail.git}
+ LIBABIGAIL_VERSION=${LIBABIGAIL_VERSION:-libabigail-1.6}
+
+ if [ "$(cat libabigail/VERSION 2>/dev/null)" != "$LIBABIGAIL_VERSION" ]; then
+ rm -rf libabigail
+ # if we change libabigail, invalidate existing abi cache
+ rm -rf reference
+ fi
+
+ if [ ! -d libabigail ]; then
+ git clone --single-branch -b $LIBABIGAIL_VERSION $LIBABIGAIL_REPO libabigail/src
+ cd libabigail/src && autoreconf -i && cd -
+ instdir=$(pwd)/libabigail
+ mkdir libabigail/src/build
+ cd libabigail/src/build && ../configure --prefix=$instdir && cd -
+ make -C libabigail/src/build all install
+
+ rm -rf libabigail/src
+ echo $LIBABIGAIL_VERSION > libabigail/VERSION
+ fi
+
+ export PATH=$(pwd)/libabigail/bin:$PATH
+
REF_GIT_REPO=${REF_GIT_REPO:-https://dpdk.org/git/dpdk}
REF_GIT_TAG=${REF_GIT_TAG:-v19.11}
diff --git a/.travis.yml b/.travis.yml
index 22539d823..030064c94 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -2,6 +2,7 @@ language: c
cache:
ccache: true
directories:
+ - libabigail
- reference
compiler:
- gcc
@@ -24,7 +25,10 @@ aarch64_packages: &aarch64_packages
extra_packages: &extra_packages
- *required_packages
- - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4, abigail-tools]
+ - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
+
+libabigail_build_packages: &libabigail_build_packages
+ - [autoconf, automake, libtool, pkg-config, libxml2-dev, libdw-dev]
build_32b_packages: &build_32b_packages
- *required_packages
@@ -160,6 +164,7 @@ matrix:
apt:
packages:
- *extra_packages
+ - *libabigail_build_packages
- env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1
arch: arm64
compiler: gcc
@@ -167,5 +172,6 @@ matrix:
apt:
packages:
- *extra_packages
+ - *libabigail_build_packages
script: ./.ci/${TRAVIS_OS_NAME}-build.sh
--
2.23.0
^ permalink raw reply [relevance 9%]
* Re: [dpdk-dev] [PATCH] ci: build and use libabigail 1.6
2020-02-18 11:18 0% ` David Marchand
@ 2020-02-18 14:55 0% ` Aaron Conole
2020-02-18 15:07 0% ` David Marchand
0 siblings, 1 reply; 200+ results
From: Aaron Conole @ 2020-02-18 14:55 UTC (permalink / raw)
To: David Marchand
Cc: Thomas Monjalon, dev, Christian Ehrhardt, Dodji Seketeli,
Michael Santana
David Marchand <david.marchand@redhat.com> writes:
> On Tue, Feb 18, 2020 at 10:40 AM David Marchand
> <david.marchand@redhat.com> wrote:
>>
>> On Mon, Feb 17, 2020 at 7:48 PM Aaron Conole <aconole@redhat.com> wrote:
>> >
>> > David Marchand <david.marchand@redhat.com> writes:
>> >
>> > > libabigail 1.2 (at least) reports changes in 'const' property as an ABI
>> > > breakage [1].
>> > > This was fixed upstream in libabigail 1.4 [2], and a bug has been opened
>> > > in launchpad [3].
>> > >
>> > > But for now, build and use the last version 1.6 so that the ABI checks
>> > > can be kept.
>> > >
>> > > 1: https://travis-ci.com/DPDK/dpdk/jobs/287872118#L2242
>> > > 2:
>> > > https://sourceware.org/git/gitweb.cgi?p=libabigail.git;a=commitdiff;h=215b7eb4fe8b986fe1cc87d9d8e7412998038392
>> > > 3: https://bugs.launchpad.net/ubuntu/+source/libabigail/+bug/1863607
>> > >
>> > > Signed-off-by: David Marchand <david.marchand@redhat.com>
>> > > ---
>> >
>> > Does it make sense to base libabigail required ontop of extra packages?
>> > Otherwise some libraries won't get built / checked, no?
>>
>> The only change I see is the pcap driver being enabled.
>> On the principle, I agree that trying to build all possible
>> libraries/drivers is better when checking the ABI.
>> So I'll keep extra_packages yes.
>>
>> I am currently testing that touching extra_packages (well, testing
>> Thomas patches) results in Travis treating the job as a new one (i.e.
>> with no cache).
>
> Travis bases each job cache on the job description:
> https://docs.travis-ci.com/user/caching/
>
> I tested Thomas change on extra_packages content, and the job used the
> old cache.
> My idea was to try to put *extra_packages in an env variable, but it
> does not work (my yaml-fu is lacking).
>
> If there is no easy way, I will invalidate the cache manually.
We don't actually use the EXTRA_PACKAGES variable for anything, so I
guess it's probably okay to change the value and that should invalidate
the cache. Most of the variables, in fact, could be checked for
non-zero value rather than a specific positive value, and then it's easy
to invalidate the cache by just bumping them. It's a thought (and
kindof a hack). Or we can just use the travis CLI tool and delete the
caches (we'll have to do that for the ovsrobot as well, I think).
>
> --
> David Marchand
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] ci: build and use libabigail 1.6
2020-02-18 14:55 0% ` Aaron Conole
@ 2020-02-18 15:07 0% ` David Marchand
0 siblings, 0 replies; 200+ results
From: David Marchand @ 2020-02-18 15:07 UTC (permalink / raw)
To: Aaron Conole
Cc: Thomas Monjalon, dev, Christian Ehrhardt, Dodji Seketeli,
Michael Santana
On Tue, Feb 18, 2020 at 3:55 PM Aaron Conole <aconole@redhat.com> wrote:
>
> David Marchand <david.marchand@redhat.com> writes:
>
> > On Tue, Feb 18, 2020 at 10:40 AM David Marchand
> > <david.marchand@redhat.com> wrote:
> >>
> >> On Mon, Feb 17, 2020 at 7:48 PM Aaron Conole <aconole@redhat.com> wrote:
> >> >
> >> > David Marchand <david.marchand@redhat.com> writes:
> >> >
> >> > > libabigail 1.2 (at least) reports changes in 'const' property as an ABI
> >> > > breakage [1].
> >> > > This was fixed upstream in libabigail 1.4 [2], and a bug has been opened
> >> > > in launchpad [3].
> >> > >
> >> > > But for now, build and use the last version 1.6 so that the ABI checks
> >> > > can be kept.
> >> > >
> >> > > 1: https://travis-ci.com/DPDK/dpdk/jobs/287872118#L2242
> >> > > 2:
> >> > > https://sourceware.org/git/gitweb.cgi?p=libabigail.git;a=commitdiff;h=215b7eb4fe8b986fe1cc87d9d8e7412998038392
> >> > > 3: https://bugs.launchpad.net/ubuntu/+source/libabigail/+bug/1863607
> >> > >
> >> > > Signed-off-by: David Marchand <david.marchand@redhat.com>
> >> > > ---
> >> >
> >> > Does it make sense to base libabigail required ontop of extra packages?
> >> > Otherwise some libraries won't get built / checked, no?
> >>
> >> The only change I see is the pcap driver being enabled.
> >> On the principle, I agree that trying to build all possible
> >> libraries/drivers is better when checking the ABI.
> >> So I'll keep extra_packages yes.
> >>
> >> I am currently testing that touching extra_packages (well, testing
> >> Thomas patches) results in Travis treating the job as a new one (i.e.
> >> with no cache).
> >
> > Travis bases each job cache on the job description:
> > https://docs.travis-ci.com/user/caching/
> >
> > I tested Thomas change on extra_packages content, and the job used the
> > old cache.
> > My idea was to try to put *extra_packages in an env variable, but it
> > does not work (my yaml-fu is lacking).
> >
> > If there is no easy way, I will invalidate the cache manually.
>
> We don't actually use the EXTRA_PACKAGES variable for anything, so I
> guess it's probably okay to change the value and that should invalidate
> the cache. Most of the variables, in fact, could be checked for
> non-zero value rather than a specific positive value, and then it's easy
> to invalidate the cache by just bumping them. It's a thought (and
> kindof a hack). Or we can just use the travis CLI tool and delete the
> caches (we'll have to do that for the ovsrobot as well, I think).
>
What I had in mind was to convert the extra_packages yaml thing into a
string to pass into EXTRA_PACKAGES.
But I did not manage.
About bumping the value, users are likely to be unaware of this step
if they submit a patch touching .travis.yml.
Deleting the caches from ovsrobot if .travis.yml has been touched seems simpler.
On master, I will stick to manual cache invalidation.
--
David Marchand
--
David Marchand
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v2] ci: build and use libabigail 1.6
2020-02-18 14:29 9% ` [dpdk-dev] [PATCH v2] " David Marchand
@ 2020-02-18 15:46 0% ` Thomas Monjalon
2020-02-18 16:38 0% ` David Marchand
0 siblings, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-02-18 15:46 UTC (permalink / raw)
To: David Marchand
Cc: dev, aconole, christian.ehrhardt, dodji, kevin.laatz, Michael Santana
18/02/2020 15:29, David Marchand:
> libabigail 1.2 (at least) reports changes in 'const' property as an ABI
> breakage [1].
> This was fixed upstream in libabigail 1.4 [2], and a bug has been opened
> in launchpad [3].
>
> But for now, build and use the last version 1.6 so that the ABI checks
> can be kept.
>
> 1: https://travis-ci.com/DPDK/dpdk/jobs/287872118#L2242
> 2: https://sourceware.org/git/gitweb.cgi?p=libabigail.git;a=commitdiff;h=215b7eb4fe8b
> 3: https://bugs.launchpad.net/ubuntu/+source/libabigail/+bug/1863607
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Thomas Monjalon <thomas@monjalon.net>
I suggest few improvements below:
> --- a/.ci/linux-build.sh
> +++ b/.ci/linux-build.sh
> if [ "$ABI_CHECKS" = "1" ]; then
What do you think about moving the libabigail install in a function?
We could justify this with a comment about installing the latest version.
> + LIBABIGAIL_REPO=${LIBABIGAIL_REPO:-https://sourceware.org/git/libabigail.git}
> + LIBABIGAIL_VERSION=${LIBABIGAIL_VERSION:-libabigail-1.6}
> +
> + if [ "$(cat libabigail/VERSION 2>/dev/null)" != "$LIBABIGAIL_VERSION" ]; then
> + rm -rf libabigail
> + # if we change libabigail, invalidate existing abi cache
> + rm -rf reference
> + fi
> +
> + if [ ! -d libabigail ]; then
> + git clone --single-branch -b $LIBABIGAIL_VERSION $LIBABIGAIL_REPO libabigail/src
Why not using the tarball?
http://mirrors.kernel.org/sourceware/libabigail/libabigail-1.6.tar.gz
> + cd libabigail/src && autoreconf -i && cd -
> + instdir=$(pwd)/libabigail
> + mkdir libabigail/src/build
> + cd libabigail/src/build && ../configure --prefix=$instdir && cd -
> + make -C libabigail/src/build all install
> +
> + rm -rf libabigail/src
> + echo $LIBABIGAIL_VERSION > libabigail/VERSION
> + fi
> +
> + export PATH=$(pwd)/libabigail/bin:$PATH
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v2] ci: build and use libabigail 1.6
2020-02-18 15:46 0% ` Thomas Monjalon
@ 2020-02-18 16:38 0% ` David Marchand
2020-02-19 8:13 0% ` Dodji Seketeli
0 siblings, 1 reply; 200+ results
From: David Marchand @ 2020-02-18 16:38 UTC (permalink / raw)
To: Thomas Monjalon
Cc: dev, Aaron Conole, Christian Ehrhardt, Dodji Seketeli,
Kevin Laatz, Michael Santana
On Tue, Feb 18, 2020 at 4:46 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> 18/02/2020 15:29, David Marchand:
> > libabigail 1.2 (at least) reports changes in 'const' property as an ABI
> > breakage [1].
> > This was fixed upstream in libabigail 1.4 [2], and a bug has been opened
> > in launchpad [3].
> >
> > But for now, build and use the last version 1.6 so that the ABI checks
> > can be kept.
> >
> > 1: https://travis-ci.com/DPDK/dpdk/jobs/287872118#L2242
> > 2: https://sourceware.org/git/gitweb.cgi?p=libabigail.git;a=commitdiff;h=215b7eb4fe8b
> > 3: https://bugs.launchpad.net/ubuntu/+source/libabigail/+bug/1863607
> >
> > Signed-off-by: David Marchand <david.marchand@redhat.com>
>
> Acked-by: Thomas Monjalon <thomas@monjalon.net>
>
> I suggest few improvements below:
>
> > --- a/.ci/linux-build.sh
> > +++ b/.ci/linux-build.sh
> > if [ "$ABI_CHECKS" = "1" ]; then
>
> What do you think about moving the libabigail install in a function?
No strong opinion, we had everything inline so far.
>
> We could justify this with a comment about installing the latest version.
>
> > + LIBABIGAIL_REPO=${LIBABIGAIL_REPO:-https://sourceware.org/git/libabigail.git}
> > + LIBABIGAIL_VERSION=${LIBABIGAIL_VERSION:-libabigail-1.6}
> > +
> > + if [ "$(cat libabigail/VERSION 2>/dev/null)" != "$LIBABIGAIL_VERSION" ]; then
> > + rm -rf libabigail
> > + # if we change libabigail, invalidate existing abi cache
> > + rm -rf reference
> > + fi
> > +
> > + if [ ! -d libabigail ]; then
> > + git clone --single-branch -b $LIBABIGAIL_VERSION $LIBABIGAIL_REPO libabigail/src
>
> Why not using the tarball?
> http://mirrors.kernel.org/sourceware/libabigail/libabigail-1.6.tar.gz
No good reason "now".
I was first bitten by a reference to redhat-hardened-ld in some
libtool script in the tarball (/me looks in Dodji direction).
I then considered switching to different versions of libabigail by
just setting the LIBABIGAIL_VERSION env variable from .travis.yml.
I ended up with the current latest version which is also what is in
Ubuntu latest releases.
The tarball is smaller than a git clone, so best to use it.
v3 incoming.
--
David Marchand
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v2] ethdev: add comment to warn of ABI breakage
2020-02-18 13:39 13% ` [dpdk-dev] [PATCH v2] " Ciara Power
@ 2020-02-18 18:40 4% ` Ferruh Yigit
0 siblings, 0 replies; 200+ results
From: Ferruh Yigit @ 2020-02-18 18:40 UTC (permalink / raw)
To: Ciara Power, thomas, arybchenko; +Cc: dev
On 2/18/2020 1:39 PM, Ciara Power wrote:
> If a function is added to the eth_dev_ops struct before
> tx_descriptor_status function, this will cause ABI breakage. This is due
> to static inline functions using this function, and some other functions
> above it in the struct, so they cannot change position. A comment is
> added to inform developers of this possible breakage.
>
> Signed-off-by: Ciara Power <ciara.power@intel.com>
> Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
> Reviewed-by: Andrew Rybchenko <arybchenko@solarflare.com>
Acked-by: Thomas Monjalon <thomas@monjalon.net>
Applied to dpdk-next-net/master, thanks.
(uppercase ABOVE/BELOW as suggested applied while merging)
^ permalink raw reply [relevance 4%]
* [dpdk-dev] [PATCH v3] ci: build and use libabigail 1.6
2020-02-17 13:59 9% [dpdk-dev] [PATCH] ci: build and use libabigail 1.6 David Marchand
` (2 preceding siblings ...)
2020-02-18 14:29 9% ` [dpdk-dev] [PATCH v2] " David Marchand
@ 2020-02-18 20:13 9% ` David Marchand
2020-02-18 20:31 0% ` Aaron Conole
3 siblings, 1 reply; 200+ results
From: David Marchand @ 2020-02-18 20:13 UTC (permalink / raw)
To: thomas
Cc: dev, aconole, christian.ehrhardt, dodji, kevin.laatz, Michael Santana
libabigail 1.2 (at least) reports changes in 'const' property as an ABI
breakage [1].
This was fixed upstream in libabigail 1.4 [2], and a bug has been opened
in launchpad [3].
But for now, build and use the last version 1.6 so that the ABI checks
can be kept.
1: https://travis-ci.com/DPDK/dpdk/jobs/287872118#L2242
2: https://sourceware.org/git/gitweb.cgi?p=libabigail.git;a=commitdiff;h=215b7eb4fe8b
3: https://bugs.launchpad.net/ubuntu/+source/libabigail/+bug/1863607
Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Thomas Monjalon <thomas@monjalon.net>
---
Changelog since v2:
- used tarball instead of git,
- moved libabigail compilation/install in a function,
Changelog since v1:
- keep extra_packages in ABI checks jobs,
- fix configure step,
---
.ci/linux-build.sh | 29 +++++++++++++++++++++++++++++
.travis.yml | 8 +++++++-
2 files changed, 36 insertions(+), 1 deletion(-)
diff --git a/.ci/linux-build.sh b/.ci/linux-build.sh
index c7c3840fc..d500c4c00 100755
--- a/.ci/linux-build.sh
+++ b/.ci/linux-build.sh
@@ -14,6 +14,20 @@ on_error() {
}
trap on_error EXIT
+install_libabigail() {
+ version=$1
+ instdir=$2
+
+ wget -q "http://mirrors.kernel.org/sourceware/libabigail/${version}.tar.gz"
+ tar -xf ${version}.tar.gz
+ cd $version && autoreconf -vfi && cd -
+ mkdir $version/build
+ cd $version/build && ../configure --prefix=$instdir && cd -
+ make -C $version/build all install
+ rm -rf $version
+ rm ${version}.tar.gz
+}
+
if [ "$AARCH64" = "1" ]; then
# convert the arch specifier
OPTS="$OPTS --cross-file config/arm/arm64_armv8_linux_gcc"
@@ -38,6 +52,21 @@ if [ "$AARCH64" != "1" ]; then
fi
if [ "$ABI_CHECKS" = "1" ]; then
+ LIBABIGAIL_VERSION=${LIBABIGAIL_VERSION:-libabigail-1.6}
+
+ if [ "$(cat libabigail/VERSION 2>/dev/null)" != "$LIBABIGAIL_VERSION" ]; then
+ rm -rf libabigail
+ # if we change libabigail, invalidate existing abi cache
+ rm -rf reference
+ fi
+
+ if [ ! -d libabigail ]; then
+ install_libabigail $LIBABIGAIL_VERSION $(pwd)/libabigail
+ echo $LIBABIGAIL_VERSION > libabigail/VERSION
+ fi
+
+ export PATH=$(pwd)/libabigail/bin:$PATH
+
REF_GIT_REPO=${REF_GIT_REPO:-https://dpdk.org/git/dpdk}
REF_GIT_TAG=${REF_GIT_TAG:-v19.11}
diff --git a/.travis.yml b/.travis.yml
index 22539d823..030064c94 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -2,6 +2,7 @@ language: c
cache:
ccache: true
directories:
+ - libabigail
- reference
compiler:
- gcc
@@ -24,7 +25,10 @@ aarch64_packages: &aarch64_packages
extra_packages: &extra_packages
- *required_packages
- - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4, abigail-tools]
+ - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
+
+libabigail_build_packages: &libabigail_build_packages
+ - [autoconf, automake, libtool, pkg-config, libxml2-dev, libdw-dev]
build_32b_packages: &build_32b_packages
- *required_packages
@@ -160,6 +164,7 @@ matrix:
apt:
packages:
- *extra_packages
+ - *libabigail_build_packages
- env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1
arch: arm64
compiler: gcc
@@ -167,5 +172,6 @@ matrix:
apt:
packages:
- *extra_packages
+ - *libabigail_build_packages
script: ./.ci/${TRAVIS_OS_NAME}-build.sh
--
2.23.0
^ permalink raw reply [relevance 9%]
* Re: [dpdk-dev] [PATCH v3] ci: build and use libabigail 1.6
2020-02-18 20:13 9% ` [dpdk-dev] [PATCH v3] " David Marchand
@ 2020-02-18 20:31 0% ` Aaron Conole
2020-02-18 20:59 0% ` David Marchand
0 siblings, 1 reply; 200+ results
From: Aaron Conole @ 2020-02-18 20:31 UTC (permalink / raw)
To: David Marchand
Cc: thomas, dev, christian.ehrhardt, dodji, kevin.laatz, Michael Santana
David Marchand <david.marchand@redhat.com> writes:
> libabigail 1.2 (at least) reports changes in 'const' property as an ABI
> breakage [1].
> This was fixed upstream in libabigail 1.4 [2], and a bug has been opened
> in launchpad [3].
>
> But for now, build and use the last version 1.6 so that the ABI checks
> can be kept.
>
> 1: https://travis-ci.com/DPDK/dpdk/jobs/287872118#L2242
> 2: https://sourceware.org/git/gitweb.cgi?p=libabigail.git;a=commitdiff;h=215b7eb4fe8b
> 3: https://bugs.launchpad.net/ubuntu/+source/libabigail/+bug/1863607
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> Acked-by: Thomas Monjalon <thomas@monjalon.net>
> ---
Acked-by: Aaron Conole <aconole@redhat.com>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v3] ci: build and use libabigail 1.6
2020-02-18 20:31 0% ` Aaron Conole
@ 2020-02-18 20:59 0% ` David Marchand
0 siblings, 0 replies; 200+ results
From: David Marchand @ 2020-02-18 20:59 UTC (permalink / raw)
To: David Marchand
Cc: Thomas Monjalon, dev, Christian Ehrhardt, Dodji Seketeli,
Kevin Laatz, Michael Santana, Aaron Conole
On Tue, Feb 18, 2020 at 9:31 PM Aaron Conole <aconole@redhat.com> wrote:
> David Marchand <david.marchand@redhat.com> writes:
>
> > libabigail 1.2 (at least) reports changes in 'const' property as an ABI
> > breakage [1].
> > This was fixed upstream in libabigail 1.4 [2], and a bug has been opened
> > in launchpad [3].
> >
> > But for now, build and use the last version 1.6 so that the ABI checks
> > can be kept.
> >
> > 1: https://travis-ci.com/DPDK/dpdk/jobs/287872118#L2242
> > 2: https://sourceware.org/git/gitweb.cgi?p=libabigail.git;a=commitdiff;h=215b7eb4fe8b
> > 3: https://bugs.launchpad.net/ubuntu/+source/libabigail/+bug/1863607
> >
> > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > Acked-by: Thomas Monjalon <thomas@monjalon.net>
> > ---
>
> Acked-by: Aaron Conole <aconole@redhat.com>
Applied.
--
David Marchand
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v3 0/2] improve Travis coverage
@ 2020-02-18 22:36 3% ` David Marchand
0 siblings, 0 replies; 200+ results
From: David Marchand @ 2020-02-18 22:36 UTC (permalink / raw)
To: Thomas Monjalon; +Cc: dev, Aaron Conole, Bruce Richardson
On Tue, Feb 18, 2020 at 10:20 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> Remove some useless jobs and add more packages
> to build more PMDs in Travis.
>
> v3: reword commit logs
> v2: remove libbpf because not available
>
> Thomas Monjalon (2):
> ci: remove redundant configs disabling kmods
> ci: enable more drivers in Travis builds
>
> .travis.yml | 30 +++++-------------------------
> 1 file changed, 5 insertions(+), 25 deletions(-)
About enabling kmod compilation again, this can be done in a followup patch.
For the series,
Acked-by: David Marchand <david.marchand@redhat.com>
Rebased on master, cache for abi checks jobs deleted.
Series applied.
--
David Marchand
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH v2] ci: build and use libabigail 1.6
2020-02-18 16:38 0% ` David Marchand
@ 2020-02-19 8:13 0% ` Dodji Seketeli
0 siblings, 0 replies; 200+ results
From: Dodji Seketeli @ 2020-02-19 8:13 UTC (permalink / raw)
To: David Marchand
Cc: Thomas Monjalon, dev, Aaron Conole, Christian Ehrhardt,
Kevin Laatz, Michael Santana
Hello,
David Marchand <david.marchand@redhat.com> writes:
>> > + LIBABIGAIL_REPO=${LIBABIGAIL_REPO:-https://sourceware.org/git/libabigail.git}
>> > + LIBABIGAIL_VERSION=${LIBABIGAIL_VERSION:-libabigail-1.6}
>> > +
>> > + if [ "$(cat libabigail/VERSION 2>/dev/null)" != "$LIBABIGAIL_VERSION" ]; then
>> > + rm -rf libabigail
>> > + # if we change libabigail, invalidate existing abi cache
>> > + rm -rf reference
>> > + fi
>> > +
>> > + if [ ! -d libabigail ]; then
>> > + git clone --single-branch -b $LIBABIGAIL_VERSION $LIBABIGAIL_REPO libabigail/src
>>
>> Why not using the tarball?
>> http://mirrors.kernel.org/sourceware/libabigail/libabigail-1.6.tar.gz
>
> No good reason "now".
>
> I was first bitten by a reference to redhat-hardened-ld in some
> libtool script in the tarball (/me looks in Dodji direction).
> I then considered switching to different versions of libabigail by
> just setting the LIBABIGAIL_VERSION env variable from .travis.yml.
> I ended up with the current latest version which is also what is in
> Ubuntu latest releases.
I think either way (distro package, tarball and git) has some benefits
and drawbacks.
I think one benefit of /being able/ to use the code from git is for
cases where you guys need some new features (and we tend to continuously
add fixes/functionalities to the git repository) that is available in
git only for now. Hopefully, with time, you'll only need to use the
distro package.
FWIW, I like the fact that your setup lets you choose between the distro
package, the tarball or the git code. That's powerful.
Cheers,
--
Dodji
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] ABI version of experimental libraries
2020-02-18 9:50 4% ` Thomas Monjalon
2020-02-18 10:36 4% ` Kinsella, Ray
@ 2020-02-19 11:43 7% ` Neil Horman
2020-02-19 12:43 4% ` Thomas Monjalon
1 sibling, 1 reply; 200+ results
From: Neil Horman @ 2020-02-19 11:43 UTC (permalink / raw)
To: Thomas Monjalon
Cc: Bruce Richardson, ray.kinsella, bluca, david.marchand, ktraynor, dev
On Tue, Feb 18, 2020 at 10:50:09AM +0100, Thomas Monjalon wrote:
> 18/02/2020 10:42, Bruce Richardson:
> > On Tue, Feb 18, 2020 at 12:15:56AM +0100, Thomas Monjalon wrote:
> > > Hi,
> > >
> > > I would like to remind everybody our mistake when defining ABI versions.
> > > It has been "fixed" in this commit:
> > > http://git.dpdk.org/dpdk/commit/?id=f26c2b39
> > >
> > > Please let's think about the consequence for the experimental libraries.
> > >
> > > In DPDK 19.11, we use the ABI version 0.200 with soname 0.20 In DPDK
> > > 20.02, we use the ABI version 0.2001 with soname 0.201 Numbers are
> > > increasing, that's fine. When we'll switch to the new major ABI and use
> > > a normal numbering: In DPDK 20.11, we will use the ABI version 0.210 with
> > > soname 0.21 Numbers are dropping.
> > >
> > > In short, for experimental libs, ABI 20.1 > ABI 21.0
> > >
> > > Are we OK with this or do we prefer reverting to normal numbering for
> > > experimental libraries in DPDK 20.02?
> > >
> > Personally, I would not be too concerned about the verions of experimental
> > libs, so long as they don't conflict across versions and have some
> > similarity to the major ABI version for the release.
>
> You think sorting of the version numbers is not important?
> If we don't care comparing experimental version numbers,
> then OK, let's drop this patch. But please we need a small vote.
>
> Note: there would be no problem if we did not vote for having
> a special numbering for pure experimental libraries (I am still against).
>
I don't understand. Why would we change the ABI_VERSION at all in an LTS release at
all? This operation is meant to take an an experimental API and mark it as
stable by promoting its version number to the next major releases number. As
such, in the LTS release, we should keep the soname the same, as there should be
no other ABI changes in the promoted API.
Neil
>
>
^ permalink raw reply [relevance 7%]
* Re: [dpdk-dev] [PATCH] net/ionic: fix ABI version
2020-02-17 16:13 7% [dpdk-dev] [PATCH] net/ionic: fix ABI version Ferruh Yigit
@ 2020-02-19 12:37 4% ` Ferruh Yigit
0 siblings, 0 replies; 200+ results
From: Ferruh Yigit @ 2020-02-19 12:37 UTC (permalink / raw)
To: Alfredo Cardigliano, Shannon Nelson; +Cc: dev, David Marchand
On 2/17/2020 4:13 PM, Ferruh Yigit wrote:
> Changed the ABI version to 20.0.1.
>
> Fixes: a72cf98cc93e ("net/ionic: add skeleton")
>
> Reported-by: David Marchand <david.marchand@redhat.com>
> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
Applied to dpdk-next-net/master, thanks.
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] ABI version of experimental libraries
2020-02-19 11:43 7% ` [dpdk-dev] ABI version of experimental libraries Neil Horman
@ 2020-02-19 12:43 4% ` Thomas Monjalon
2020-02-19 13:50 4% ` Ray Kinsella
2020-02-19 21:17 4% ` Neil Horman
0 siblings, 2 replies; 200+ results
From: Thomas Monjalon @ 2020-02-19 12:43 UTC (permalink / raw)
To: Neil Horman
Cc: Bruce Richardson, ray.kinsella, bluca, david.marchand, ktraynor, dev
19/02/2020 12:43, Neil Horman:
> On Tue, Feb 18, 2020 at 10:50:09AM +0100, Thomas Monjalon wrote:
> > 18/02/2020 10:42, Bruce Richardson:
> > > On Tue, Feb 18, 2020 at 12:15:56AM +0100, Thomas Monjalon wrote:
> > > > Hi,
> > > >
> > > > I would like to remind everybody our mistake when defining ABI versions.
> > > > It has been "fixed" in this commit:
> > > > http://git.dpdk.org/dpdk/commit/?id=f26c2b39
> > > >
> > > > Please let's think about the consequence for the experimental libraries.
> > > >
> > > > In DPDK 19.11, we use the ABI version 0.200 with soname 0.20 In DPDK
> > > > 20.02, we use the ABI version 0.2001 with soname 0.201 Numbers are
> > > > increasing, that's fine. When we'll switch to the new major ABI and use
> > > > a normal numbering: In DPDK 20.11, we will use the ABI version 0.210 with
> > > > soname 0.21 Numbers are dropping.
> > > >
> > > > In short, for experimental libs, ABI 20.1 > ABI 21.0
> > > >
> > > > Are we OK with this or do we prefer reverting to normal numbering for
> > > > experimental libraries in DPDK 20.02?
> > > >
> > > Personally, I would not be too concerned about the verions of experimental
> > > libs, so long as they don't conflict across versions and have some
> > > similarity to the major ABI version for the release.
> >
> > You think sorting of the version numbers is not important?
> > If we don't care comparing experimental version numbers,
> > then OK, let's drop this patch. But please we need a small vote.
> >
> > Note: there would be no problem if we did not vote for having
> > a special numbering for pure experimental libraries (I am still against).
> >
> I don't understand. Why would we change the ABI_VERSION at all in an LTS release at
> all? This operation is meant to take an an experimental API and mark it as
> stable by promoting its version number to the next major releases number. As
> such, in the LTS release, we should keep the soname the same, as there should be
> no other ABI changes in the promoted API.
The library version number is updated because we add new symbols.
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] ABI version of experimental libraries
2020-02-19 12:43 4% ` Thomas Monjalon
@ 2020-02-19 13:50 4% ` Ray Kinsella
2020-02-21 16:57 4% ` Thomas Monjalon
2020-02-19 21:17 4% ` Neil Horman
1 sibling, 1 reply; 200+ results
From: Ray Kinsella @ 2020-02-19 13:50 UTC (permalink / raw)
To: dev
On 19/02/2020 12:43, Thomas Monjalon wrote:
> 19/02/2020 12:43, Neil Horman:
>> On Tue, Feb 18, 2020 at 10:50:09AM +0100, Thomas Monjalon wrote:
>>> 18/02/2020 10:42, Bruce Richardson:
>>>> On Tue, Feb 18, 2020 at 12:15:56AM +0100, Thomas Monjalon wrote:
>>>>> Hi,
>>>>>
>>>>> I would like to remind everybody our mistake when defining ABI versions.
>>>>> It has been "fixed" in this commit:
>>>>> http://git.dpdk.org/dpdk/commit/?id=f26c2b39
>>>>>
>>>>> Please let's think about the consequence for the experimental libraries.
>>>>>
>>>>> In DPDK 19.11, we use the ABI version 0.200 with soname 0.20 In DPDK
>>>>> 20.02, we use the ABI version 0.2001 with soname 0.201 Numbers are
>>>>> increasing, that's fine. When we'll switch to the new major ABI and use
>>>>> a normal numbering: In DPDK 20.11, we will use the ABI version 0.210 with
>>>>> soname 0.21 Numbers are dropping.
>>>>>
>>>>> In short, for experimental libs, ABI 20.1 > ABI 21.0
>>>>>
>>>>> Are we OK with this or do we prefer reverting to normal numbering for
>>>>> experimental libraries in DPDK 20.02?
>>>>>
>>>> Personally, I would not be too concerned about the verions of experimental
>>>> libs, so long as they don't conflict across versions and have some
>>>> similarity to the major ABI version for the release.
>>>
>>> You think sorting of the version numbers is not important?
>>> If we don't care comparing experimental version numbers,
>>> then OK, let's drop this patch. But please we need a small vote.
>>>
>>> Note: there would be no problem if we did not vote for having
>>> a special numbering for pure experimental libraries (I am still against).
>>>
>> I don't understand. Why would we change the ABI_VERSION at all in an LTS release at
>> all? This operation is meant to take an an experimental API and mark it as
>> stable by promoting its version number to the next major releases number. As
>> such, in the LTS release, we should keep the soname the same, as there should be
>> no other ABI changes in the promoted API.
>
> The library version number is updated because we add new symbols.
>
>
So while experimental library version numbers are not "important".
I do agree with Thomas they should be sane, increase and should have a consistent format.
Should we always just pad them to 4 places as the simple solution?
i.e.
DPDK 19.11 ... 0.20 (needs to remain 0.20).
DPDK 20.02 ... 0.2001
DPDK 20.11 ... 0.2100
DPDK 21.02 ... 0.2101
Make sense?
Ray K
^ permalink raw reply [relevance 4%]
* [dpdk-dev] [PATCH v2 0/7] vfio/pci: SR-IOV support
@ 2020-02-19 18:53 2% Alex Williamson
2020-02-25 2:33 0% ` Tian, Kevin
2020-03-05 6:38 0% ` Vamsi Krishna Attunuru
0 siblings, 2 replies; 200+ results
From: Alex Williamson @ 2020-02-19 18:53 UTC (permalink / raw)
To: kvm
Cc: linux-pci, linux-kernel, dev, mtosatti, thomas, bluca,
jerinjacobk, bruce.richardson, cohuck
Changes since v1 are primarily to patch 3/7 where the commit log is
rewritten, along with option parsing and failure logging based on
upstream discussions. The primary user visible difference is that
option parsing is now much more strict. If a vf_token option is
provided that cannot be used, we generate an error. As a result of
this, opening a PF with a vf_token option will serve as a mechanism of
setting the vf_token. This seems like a more user friendly API than
the alternative of sometimes requiring the option (VFs in use) and
sometimes rejecting it, and upholds our desire that the option is
always either used or rejected.
This also means that the VFIO_DEVICE_FEATURE ioctl is not the only
means of setting the VF token, which might call into question whether
we absolutely need this new ioctl. Currently I'm keeping it because I
can imagine use cases, for example if a hypervisor were to support
SR-IOV, the PF device might be opened without consideration for a VF
token and we'd require the hypservisor to close and re-open the PF in
order to set a known VF token, which is impractical.
Series overview (same as provided with v1):
The synopsis of this series is that we have an ongoing desire to drive
PCIe SR-IOV PFs from userspace with VFIO. There's an immediate need
for this with DPDK drivers and potentially interesting future use
cases in virtualization. We've been reluctant to add this support
previously due to the dependency and trust relationship between the
VF device and PF driver. Minimally the PF driver can induce a denial
of service to the VF, but depending on the specific implementation,
the PF driver might also be responsible for moving data between VFs
or have direct access to the state of the VF, including data or state
otherwise private to the VF or VF driver.
To help resolve these concerns, we introduce a VF token into the VFIO
PCI ABI, which acts as a shared secret key between drivers. The
userspace PF driver is required to set the VF token to a known value
and userspace VF drivers are required to provide the token to access
the VF device. If a PF driver is restarted with VF drivers in use, it
must also provide the current token in order to prevent a rogue
untrusted PF driver from replacing a known driver. The degree to
which this new token is considered secret is left to the userspace
drivers, the kernel intentionally provides no means to retrieve the
current token.
Note that the above token is only required for this new model where
both the PF and VF devices are usable through vfio-pci. Existing
models of VFIO drivers where the PF is used without SR-IOV enabled
or the VF is bound to a userspace driver with an in-kernel, host PF
driver are unaffected.
The latter configuration above also highlights a new inverted scenario
that is now possible, a userspace PF driver with in-kernel VF drivers.
I believe this is a scenario that should be allowed, but should not be
enabled by default. This series includes code to set a default
driver_override for VFs sourced from a vfio-pci user owned PF, such
that the VFs are also bound to vfio-pci. This model is compatible
with tools like driverctl and allows the system administrator to
decide if other bindings should be enabled. The VF token interface
above exists only between vfio-pci PF and VF drivers, once a VF is
bound to another driver, the administrator has effectively pronounced
the device as trusted. The vfio-pci driver will note alternate
binding in dmesg for logging and debugging purposes.
Please review, comment, and test. The example QEMU implementation
provided with the RFC is still current for this version. Thanks,
Alex
RFC: https://lore.kernel.org/lkml/158085337582.9445.17682266437583505502.stgit@gimli.home/
v1: https://lore.kernel.org/lkml/158145472604.16827.15751375540102298130.stgit@gimli.home/
---
Alex Williamson (7):
vfio: Include optional device match in vfio_device_ops callbacks
vfio/pci: Implement match ops
vfio/pci: Introduce VF token
vfio: Introduce VFIO_DEVICE_FEATURE ioctl and first user
vfio/pci: Add sriov_configure support
vfio/pci: Remove dev_fmt definition
vfio/pci: Cleanup .probe() exit paths
drivers/vfio/pci/vfio_pci.c | 383 +++++++++++++++++++++++++++++++++--
drivers/vfio/pci/vfio_pci_private.h | 10 +
drivers/vfio/vfio.c | 20 +-
include/linux/vfio.h | 4
include/uapi/linux/vfio.h | 37 +++
5 files changed, 426 insertions(+), 28 deletions(-)
^ permalink raw reply [relevance 2%]
* [dpdk-dev] [PATCH 4/4] ci: reorganise Travis jobs
@ 2020-02-19 19:41 3% ` David Marchand
2020-02-19 21:39 0% ` Aaron Conole
` (2 more replies)
0 siblings, 3 replies; 200+ results
From: David Marchand @ 2020-02-19 19:41 UTC (permalink / raw)
To: thomas, aconole; +Cc: dev, Michael Santana
Let's prune the jobs list to limit the amount of time spent by the robot
in Travis.
Since meson enables automatically the relevant components, there is not
much gain in testing with extra_packages vs required_packages only.
For a given arch/compiler/env combination, compilation is first tested
in all jobs that run tests or build the docs or run the ABI checks.
In the same context, for jobs that accumulates running tests, building
the docs etc..., those steps are independent and can be split to save
some cpu on Travis.
With this, we go down from 21 to 15 jobs.
Note: this patch requires a flush of the existing caches in Travis.
Signed-off-by: David Marchand <david.marchand@redhat.com>
---
.travis.yml | 51 +++++++++------------------------------------------
1 file changed, 9 insertions(+), 42 deletions(-)
diff --git a/.travis.yml b/.travis.yml
index c174d5f30..b64a81bd0 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -15,15 +15,12 @@ addons:
update: true
packages: &required_packages
- [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, ninja-build]
+ - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson4]
_aarch64_packages: &aarch64_packages
- *required_packages
- [gcc-aarch64-linux-gnu, libc6-dev-arm64-cross, pkg-config-aarch64-linux-gnu]
-_extra_packages: &extra_packages
- - *required_packages
- - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson4]
-
_libabigail_build_packages: &libabigail_build_packages
- [autoconf, automake, libtool, pkg-config, libxml2-dev, libdw-dev]
@@ -43,66 +40,39 @@ jobs:
- env: DEF_LIB="static"
arch: amd64
compiler: gcc
- - env: DEF_LIB="shared"
- arch: amd64
- compiler: gcc
- env: DEF_LIB="shared" RUN_TESTS=1
arch: amd64
compiler: gcc
- - env: DEF_LIB="static" EXTRA_PACKAGES=1
- arch: amd64
- compiler: gcc
- addons:
- apt:
- packages:
- - *extra_packages
- - env: DEF_LIB="shared" EXTRA_PACKAGES=1 RUN_TESTS=1 BUILD_DOCS=1
+ - env: DEF_LIB="shared" BUILD_DOCS=1
arch: amd64
compiler: gcc
addons:
apt:
packages:
- - *extra_packages
+ - *required_packages
- *doc_packages
- - env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1
+ - env: DEF_LIB="shared" ABI_CHECKS=1
arch: amd64
compiler: gcc
addons:
apt:
packages:
- - *extra_packages
+ - *required_packages
- *libabigail_build_packages
# x86_64 clang jobs
- env: DEF_LIB="static"
arch: amd64
compiler: clang
- - env: DEF_LIB="shared"
- arch: amd64
- compiler: clang
- env: DEF_LIB="shared" RUN_TESTS=1
arch: amd64
compiler: clang
- - env: DEF_LIB="static" EXTRA_PACKAGES=1
- arch: amd64
- compiler: clang
- addons:
- apt:
- packages:
- - *extra_packages
- - env: DEF_LIB="shared" EXTRA_PACKAGES=1
- arch: amd64
- compiler: clang
- addons:
- apt:
- packages:
- - *extra_packages
- - env: DEF_LIB="shared" EXTRA_PACKAGES=1 RUN_TESTS=1 BUILD_DOCS=1
+ - env: DEF_LIB="shared" BUILD_DOCS=1
arch: amd64
compiler: clang
addons:
apt:
packages:
- - *extra_packages
+ - *required_packages
- *doc_packages
# x86_64 cross-compiling 32-bits jobs
- env: DEF_LIB="static" BUILD_32BIT=1
@@ -131,9 +101,6 @@ jobs:
- env: DEF_LIB="static"
arch: arm64
compiler: gcc
- - env: DEF_LIB="shared"
- arch: arm64
- compiler: gcc
- env: DEF_LIB="shared" BUILD_DOCS=1
arch: arm64
compiler: gcc
@@ -142,13 +109,13 @@ jobs:
packages:
- *required_packages
- *doc_packages
- - env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1
+ - env: DEF_LIB="shared" ABI_CHECKS=1
arch: arm64
compiler: gcc
addons:
apt:
packages:
- - *extra_packages
+ - *required_packages
- *libabigail_build_packages
# aarch64 clang jobs
- env: DEF_LIB="static"
--
2.23.0
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] ABI version of experimental libraries
2020-02-19 12:43 4% ` Thomas Monjalon
2020-02-19 13:50 4% ` Ray Kinsella
@ 2020-02-19 21:17 4% ` Neil Horman
1 sibling, 0 replies; 200+ results
From: Neil Horman @ 2020-02-19 21:17 UTC (permalink / raw)
To: Thomas Monjalon
Cc: Bruce Richardson, ray.kinsella, bluca, david.marchand, ktraynor, dev
On Wed, Feb 19, 2020 at 01:43:01PM +0100, Thomas Monjalon wrote:
> 19/02/2020 12:43, Neil Horman:
> > On Tue, Feb 18, 2020 at 10:50:09AM +0100, Thomas Monjalon wrote:
> > > 18/02/2020 10:42, Bruce Richardson:
> > > > On Tue, Feb 18, 2020 at 12:15:56AM +0100, Thomas Monjalon wrote:
> > > > > Hi,
> > > > >
> > > > > I would like to remind everybody our mistake when defining ABI versions.
> > > > > It has been "fixed" in this commit:
> > > > > http://git.dpdk.org/dpdk/commit/?id=f26c2b39
> > > > >
> > > > > Please let's think about the consequence for the experimental libraries.
> > > > >
> > > > > In DPDK 19.11, we use the ABI version 0.200 with soname 0.20 In DPDK
> > > > > 20.02, we use the ABI version 0.2001 with soname 0.201 Numbers are
> > > > > increasing, that's fine. When we'll switch to the new major ABI and use
> > > > > a normal numbering: In DPDK 20.11, we will use the ABI version 0.210 with
> > > > > soname 0.21 Numbers are dropping.
> > > > >
> > > > > In short, for experimental libs, ABI 20.1 > ABI 21.0
> > > > >
> > > > > Are we OK with this or do we prefer reverting to normal numbering for
> > > > > experimental libraries in DPDK 20.02?
> > > > >
> > > > Personally, I would not be too concerned about the verions of experimental
> > > > libs, so long as they don't conflict across versions and have some
> > > > similarity to the major ABI version for the release.
> > >
> > > You think sorting of the version numbers is not important?
> > > If we don't care comparing experimental version numbers,
> > > then OK, let's drop this patch. But please we need a small vote.
> > >
> > > Note: there would be no problem if we did not vote for having
> > > a special numbering for pure experimental libraries (I am still against).
> > >
> > I don't understand. Why would we change the ABI_VERSION at all in an LTS release at
> > all? This operation is meant to take an an experimental API and mark it as
> > stable by promoting its version number to the next major releases number. As
> > such, in the LTS release, we should keep the soname the same, as there should be
> > no other ABI changes in the promoted API.
>
> The library version number is updated because we add new symbols.
But thats a matter of policy (and possibly build mechanics). Not saying agree
with this approach necessecarily (as I've not fully thought it through), but we
could make room in the policy for this eventuality.
Neil
>
>
>
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] ABI version of experimental libraries
2020-02-18 9:50 4% ` Thomas Monjalon
@ 2020-02-18 10:36 4% ` Kinsella, Ray
2020-02-20 19:50 8% ` Ferruh Yigit
2020-02-19 11:43 7% ` [dpdk-dev] ABI version of experimental libraries Neil Horman
1 sibling, 1 reply; 200+ results
From: Kinsella, Ray @ 2020-02-18 10:36 UTC (permalink / raw)
To: Thomas Monjalon, Richardson, Bruce
Cc: nhorman, bluca, david.marchand, ktraynor, dev
> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Tuesday 18 February 2020 09:50
> To: Richardson, Bruce <bruce.richardson@intel.com>
> Cc: Kinsella, Ray <ray.kinsella@intel.com>; nhorman@tuxdriver.com;
> bluca@debian.org; david.marchand@redhat.com; ktraynor@redhat.com;
> dev@dpdk.org
> Subject: Re: ABI version of experimental libraries
>
> 18/02/2020 10:42, Bruce Richardson:
> > On Tue, Feb 18, 2020 at 12:15:56AM +0100, Thomas Monjalon wrote:
> > > Hi,
> > >
> > > I would like to remind everybody our mistake when defining ABI
> versions.
> > > It has been "fixed" in this commit:
> > > http://git.dpdk.org/dpdk/commit/?id=f26c2b39
> > >
> > > Please let's think about the consequence for the experimental
> libraries.
> > >
> > > In DPDK 19.11, we use the ABI version 0.200 with soname 0.20 In
> DPDK
> > > 20.02, we use the ABI version 0.2001 with soname 0.201 Numbers are
> > > increasing, that's fine. When we'll switch to the new major ABI
> and
> > > use a normal numbering: In DPDK 20.11, we will use the ABI version
> > > 0.210 with soname 0.21 Numbers are dropping.
> > >
> > > In short, for experimental libs, ABI 20.1 > ABI 21.0
> > >
> > > Are we OK with this or do we prefer reverting to normal numbering
> > > for experimental libraries in DPDK 20.02?
> > >
> > Personally, I would not be too concerned about the verions of
> > experimental libs, so long as they don't conflict across versions and
> > have some similarity to the major ABI version for the release.
>
> You think sorting of the version numbers is not important?
> If we don't care comparing experimental version numbers, then OK, let's
> drop this patch. But please we need a small vote.
>
> Note: there would be no problem if we did not vote for having a special
> numbering for pure experimental libraries (I am still against).
>
So while experimental library version numbers are not "important".
I do agree with Thomas they should be sane, increase and should have a consistent format.
Should we always pad them to 4 places?
i.e.
DPDK 19.11 ... 0.20 (needs to remain 0.20).
DPDK 20.02 ... 0.2001
DPDK 20.11 ... 0.2100
DPDK 21.02 ... 0.2101
Make sense?
Ray K
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH 4/4] ci: reorganise Travis jobs
2020-02-19 19:41 3% ` [dpdk-dev] [PATCH 4/4] ci: reorganise " David Marchand
@ 2020-02-19 21:39 0% ` Aaron Conole
2020-02-20 10:42 0% ` Thomas Monjalon
2020-02-20 0:18 0% ` [dpdk-dev] " dwilder
2020-02-20 11:07 0% ` Thomas Monjalon
2 siblings, 1 reply; 200+ results
From: Aaron Conole @ 2020-02-19 21:39 UTC (permalink / raw)
To: David Marchand; +Cc: thomas, dev, Michael Santana, Bruce Richardson
David Marchand <david.marchand@redhat.com> writes:
> Let's prune the jobs list to limit the amount of time spent by the robot
> in Travis.
>
> Since meson enables automatically the relevant components, there is not
> much gain in testing with extra_packages vs required_packages only.
>
> For a given arch/compiler/env combination, compilation is first tested
> in all jobs that run tests or build the docs or run the ABI checks.
> In the same context, for jobs that accumulates running tests, building
> the docs etc..., those steps are independent and can be split to save
> some cpu on Travis.
>
> With this, we go down from 21 to 15 jobs.
>
> Note: this patch requires a flush of the existing caches in Travis.
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
In general, I think the idea with required vs. extra was to have a build
that did the minimum required, and one that did all the packages (to
allow a minimum vs. full DPDK).
At least, that's from
http://mails.dpdk.org/archives/dev/2019-January/124007.html
Not sure if that's still something anyone cares about.
> .travis.yml | 51 +++++++++------------------------------------------
> 1 file changed, 9 insertions(+), 42 deletions(-)
>
> diff --git a/.travis.yml b/.travis.yml
> index c174d5f30..b64a81bd0 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -15,15 +15,12 @@ addons:
> update: true
> packages: &required_packages
> - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, ninja-build]
> + - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson4]
>
> _aarch64_packages: &aarch64_packages
> - *required_packages
> - [gcc-aarch64-linux-gnu, libc6-dev-arm64-cross, pkg-config-aarch64-linux-gnu]
>
> -_extra_packages: &extra_packages
> - - *required_packages
> - - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson4]
> -
> _libabigail_build_packages: &libabigail_build_packages
> - [autoconf, automake, libtool, pkg-config, libxml2-dev, libdw-dev]
>
> @@ -43,66 +40,39 @@ jobs:
> - env: DEF_LIB="static"
> arch: amd64
> compiler: gcc
> - - env: DEF_LIB="shared"
> - arch: amd64
> - compiler: gcc
> - env: DEF_LIB="shared" RUN_TESTS=1
> arch: amd64
> compiler: gcc
> - - env: DEF_LIB="static" EXTRA_PACKAGES=1
> - arch: amd64
> - compiler: gcc
> - addons:
> - apt:
> - packages:
> - - *extra_packages
> - - env: DEF_LIB="shared" EXTRA_PACKAGES=1 RUN_TESTS=1 BUILD_DOCS=1
> + - env: DEF_LIB="shared" BUILD_DOCS=1
> arch: amd64
> compiler: gcc
> addons:
> apt:
> packages:
> - - *extra_packages
> + - *required_packages
> - *doc_packages
> - - env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1
> + - env: DEF_LIB="shared" ABI_CHECKS=1
> arch: amd64
> compiler: gcc
> addons:
> apt:
> packages:
> - - *extra_packages
> + - *required_packages
> - *libabigail_build_packages
> # x86_64 clang jobs
> - env: DEF_LIB="static"
> arch: amd64
> compiler: clang
> - - env: DEF_LIB="shared"
> - arch: amd64
> - compiler: clang
> - env: DEF_LIB="shared" RUN_TESTS=1
> arch: amd64
> compiler: clang
> - - env: DEF_LIB="static" EXTRA_PACKAGES=1
> - arch: amd64
> - compiler: clang
> - addons:
> - apt:
> - packages:
> - - *extra_packages
> - - env: DEF_LIB="shared" EXTRA_PACKAGES=1
> - arch: amd64
> - compiler: clang
> - addons:
> - apt:
> - packages:
> - - *extra_packages
> - - env: DEF_LIB="shared" EXTRA_PACKAGES=1 RUN_TESTS=1 BUILD_DOCS=1
> + - env: DEF_LIB="shared" BUILD_DOCS=1
> arch: amd64
> compiler: clang
> addons:
> apt:
> packages:
> - - *extra_packages
> + - *required_packages
> - *doc_packages
> # x86_64 cross-compiling 32-bits jobs
> - env: DEF_LIB="static" BUILD_32BIT=1
> @@ -131,9 +101,6 @@ jobs:
> - env: DEF_LIB="static"
> arch: arm64
> compiler: gcc
> - - env: DEF_LIB="shared"
> - arch: arm64
> - compiler: gcc
> - env: DEF_LIB="shared" BUILD_DOCS=1
> arch: arm64
> compiler: gcc
> @@ -142,13 +109,13 @@ jobs:
> packages:
> - *required_packages
> - *doc_packages
> - - env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1
> + - env: DEF_LIB="shared" ABI_CHECKS=1
> arch: arm64
> compiler: gcc
> addons:
> apt:
> packages:
> - - *extra_packages
> + - *required_packages
> - *libabigail_build_packages
> # aarch64 clang jobs
> - env: DEF_LIB="static"
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH 4/4] ci: reorganise Travis jobs
2020-02-19 19:41 3% ` [dpdk-dev] [PATCH 4/4] ci: reorganise " David Marchand
2020-02-19 21:39 0% ` Aaron Conole
@ 2020-02-20 0:18 0% ` dwilder
2020-02-20 11:07 0% ` Thomas Monjalon
2 siblings, 0 replies; 200+ results
From: dwilder @ 2020-02-20 0:18 UTC (permalink / raw)
To: David Marchand; +Cc: thomas, aconole, dev, Michael Santana
On 2020-02-19 11:41, David Marchand wrote:
> Let's prune the jobs list to limit the amount of time spent by the
> robot
> in Travis.
>
> Since meson enables automatically the relevant components, there is not
> much gain in testing with extra_packages vs required_packages only.
>
> For a given arch/compiler/env combination, compilation is first tested
> in all jobs that run tests or build the docs or run the ABI checks.
> In the same context, for jobs that accumulates running tests, building
> the docs etc..., those steps are independent and can be split to save
> some cpu on Travis.
>
> With this, we go down from 21 to 15 jobs.
>
> Note: this patch requires a flush of the existing caches in Travis.
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
> .travis.yml | 51 +++++++++------------------------------------------
> 1 file changed, 9 insertions(+), 42 deletions(-)
>
> diff --git a/.travis.yml b/.travis.yml
> index c174d5f30..b64a81bd0 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -15,15 +15,12 @@ addons:
> update: true
> packages: &required_packages
> - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip,
> ninja-build]
> + - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev,
> libfdt-dev, libjansson4]
>
> _aarch64_packages: &aarch64_packages
> - *required_packages
> - [gcc-aarch64-linux-gnu, libc6-dev-arm64-cross,
> pkg-config-aarch64-linux-gnu]
>
> -_extra_packages: &extra_packages
> - - *required_packages
> - - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev,
> libfdt-dev, libjansson4]
> -
> _libabigail_build_packages: &libabigail_build_packages
> - [autoconf, automake, libtool, pkg-config, libxml2-dev, libdw-dev]
>
> @@ -43,66 +40,39 @@ jobs:
> - env: DEF_LIB="static"
> arch: amd64
> compiler: gcc
> - - env: DEF_LIB="shared"
> - arch: amd64
> - compiler: gcc
> - env: DEF_LIB="shared" RUN_TESTS=1
> arch: amd64
> compiler: gcc
> - - env: DEF_LIB="static" EXTRA_PACKAGES=1
> - arch: amd64
> - compiler: gcc
> - addons:
> - apt:
> - packages:
> - - *extra_packages
> - - env: DEF_LIB="shared" EXTRA_PACKAGES=1 RUN_TESTS=1 BUILD_DOCS=1
> + - env: DEF_LIB="shared" BUILD_DOCS=1
> arch: amd64
> compiler: gcc
> addons:
> apt:
> packages:
> - - *extra_packages
> + - *required_packages
> - *doc_packages
> - - env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1
> + - env: DEF_LIB="shared" ABI_CHECKS=1
> arch: amd64
> compiler: gcc
> addons:
> apt:
> packages:
> - - *extra_packages
> + - *required_packages
> - *libabigail_build_packages
> # x86_64 clang jobs
> - env: DEF_LIB="static"
> arch: amd64
> compiler: clang
> - - env: DEF_LIB="shared"
> - arch: amd64
> - compiler: clang
> - env: DEF_LIB="shared" RUN_TESTS=1
> arch: amd64
> compiler: clang
> - - env: DEF_LIB="static" EXTRA_PACKAGES=1
> - arch: amd64
> - compiler: clang
> - addons:
> - apt:
> - packages:
> - - *extra_packages
> - - env: DEF_LIB="shared" EXTRA_PACKAGES=1
> - arch: amd64
> - compiler: clang
> - addons:
> - apt:
> - packages:
> - - *extra_packages
> - - env: DEF_LIB="shared" EXTRA_PACKAGES=1 RUN_TESTS=1 BUILD_DOCS=1
> + - env: DEF_LIB="shared" BUILD_DOCS=1
> arch: amd64
> compiler: clang
> addons:
> apt:
> packages:
> - - *extra_packages
> + - *required_packages
> - *doc_packages
> # x86_64 cross-compiling 32-bits jobs
> - env: DEF_LIB="static" BUILD_32BIT=1
> @@ -131,9 +101,6 @@ jobs:
> - env: DEF_LIB="static"
> arch: arm64
> compiler: gcc
> - - env: DEF_LIB="shared"
> - arch: arm64
> - compiler: gcc
> - env: DEF_LIB="shared" BUILD_DOCS=1
> arch: arm64
> compiler: gcc
> @@ -142,13 +109,13 @@ jobs:
> packages:
> - *required_packages
> - *doc_packages
> - - env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1
> + - env: DEF_LIB="shared" ABI_CHECKS=1
> arch: arm64
> compiler: gcc
> addons:
> apt:
> packages:
> - - *extra_packages
> + - *required_packages
> - *libabigail_build_packages
> # aarch64 clang jobs
> - env: DEF_LIB="static"
Much easier to follow, I had no issues adding ppc64le jobs.
All my travis jobs ran fine.
LGTM (the series)
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [dpdk-announce] release candidate 20.02-rc3
@ 2020-02-20 9:41 4% ` Peng, Yuan
0 siblings, 0 replies; 200+ results
From: Peng, Yuan @ 2020-02-20 9:41 UTC (permalink / raw)
To: Thomas Monjalon, dev
DPDK 20.02 RC3 Test Report from intel:
Summary
• Totally 9898 test cases for 20.02 RC3, 9629 cases are executed, 9428 test cases passed, 178 test cases failed, 23 test case blocked, Execution rate is 97.28%, pass rate is 97.91%.
• Totally create ~278 new tests for DPDK20.02 new features.
• New to create 9 issues from STV in RC3 till now, not include some issues have found and reported before RC3.
• Met 1 build issue on Centos 7.4 with DPDK 20.02 RC3, now it has been fixed, and patch applied.
Details:
• Intel NIC PMD PF & VF regression: 99% executed, 0 critical issue and 1 high issue is tracked(verified).
Function cases include rx/tx/forward packets, packet processing, jumboframe, dual vlan, filter cases and so on: 5 bugs found, no critical and high level.
Performance cases such as l2fwd, l3fwd, single core and so on: no new bug is found.
• Crypto cases include ipsec_gw cryptodev, fips cryptodev, l2fwd crypto, performance cases and so on: no critical and no high bugs found.
• Vhost/virtio: PVP/loopback, qemu/virtio-user, multi-queue, dequeue zero copy cases and so on: 1 new medium bug is found.
• Power relate cases, no new bug is found.
• ABI test includes unit test, vhost and flow related tests. no new bug is found.
• 20.02 new features include ABI test, L2tpv3, crypto_dev related new features and so no. no new bug is found.
• Scecurity tests include Nic malformed to PF/VF, virtio, Compreqssion/cryptodev, power and invalid command line, all passed.
Thanks.
Yuan.
-----Original Message-----
From: dev <dev-bounces@dpdk.org> On Behalf Of Thomas Monjalon
Sent: Monday, February 17, 2020 5:59 AM
To: announce@dpdk.org
Subject: [dpdk-dev] [dpdk-announce] release candidate 20.02-rc3
A new DPDK release candidate is ready for testing:
https://git.dpdk.org/dpdk/tag/?id=v20.02-rc3
121 patches were integrated.
The release notes so far:
http://doc.dpdk.org/guides/rel_notes/release_20_02.html
Some deprecation notices might be added if reviewed on time.
The -rc4 will include only bug fixes, doc and tooling.
This next milestone is expected during next week.
Please hurry up to do the last checks and bug fixes.
You may share some release validation results by replying to this message (at dev@dpdk.org).
If you are preparing the next release cycle, please send your v1 patches before the 20.05 proposal deadline, which will happen on March 18th.
It is also time to build an estimated roadmap for the next cycles.
Thank you everyone
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH 4/4] ci: reorganise Travis jobs
2020-02-19 21:39 0% ` Aaron Conole
@ 2020-02-20 10:42 0% ` Thomas Monjalon
2020-02-20 12:22 0% ` David Marchand
0 siblings, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-02-20 10:42 UTC (permalink / raw)
To: David Marchand, Aaron Conole; +Cc: dev, Michael Santana, Bruce Richardson
19/02/2020 22:39, Aaron Conole:
> David Marchand <david.marchand@redhat.com> writes:
>
> > Let's prune the jobs list to limit the amount of time spent by the robot
> > in Travis.
> >
> > Since meson enables automatically the relevant components, there is not
> > much gain in testing with extra_packages vs required_packages only.
> >
> > For a given arch/compiler/env combination, compilation is first tested
> > in all jobs that run tests or build the docs or run the ABI checks.
> > In the same context, for jobs that accumulates running tests, building
> > the docs etc..., those steps are independent and can be split to save
> > some cpu on Travis.
> >
> > With this, we go down from 21 to 15 jobs.
> >
> > Note: this patch requires a flush of the existing caches in Travis.
> >
> > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > ---
>
> In general, I think the idea with required vs. extra was to have a build
> that did the minimum required, and one that did all the packages (to
> allow a minimum vs. full DPDK).
>
> At least, that's from
> http://mails.dpdk.org/archives/dev/2019-January/124007.html
I think the benefit of a minimum build is to have a quick report,
and easy to setup.
> Not sure if that's still something anyone cares about.
Given that Travis knows how to satisfy the dependencies,
and that we must wait for all jobs to finish,
I don't see any benefit of a minimal setup.
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH 4/4] ci: reorganise Travis jobs
2020-02-19 19:41 3% ` [dpdk-dev] [PATCH 4/4] ci: reorganise " David Marchand
2020-02-19 21:39 0% ` Aaron Conole
2020-02-20 0:18 0% ` [dpdk-dev] " dwilder
@ 2020-02-20 11:07 0% ` Thomas Monjalon
2 siblings, 0 replies; 200+ results
From: Thomas Monjalon @ 2020-02-20 11:07 UTC (permalink / raw)
To: David Marchand; +Cc: aconole, dev, Michael Santana
19/02/2020 20:41, David Marchand:
> Let's prune the jobs list to limit the amount of time spent by the robot
> in Travis.
>
> Since meson enables automatically the relevant components, there is not
> much gain in testing with extra_packages vs required_packages only.
>
> For a given arch/compiler/env combination, compilation is first tested
> in all jobs that run tests or build the docs or run the ABI checks.
> In the same context, for jobs that accumulates running tests, building
> the docs etc..., those steps are independent and can be split to save
> some cpu on Travis.
>
> With this, we go down from 21 to 15 jobs.
>
> Note: this patch requires a flush of the existing caches in Travis.
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Thomas Monjalon <thomas@monjalon.net>
A small step for DPDK but a big save for the earth ;)
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH 4/4] ci: reorganise Travis jobs
2020-02-20 10:42 0% ` Thomas Monjalon
@ 2020-02-20 12:22 0% ` David Marchand
2020-02-20 14:35 0% ` Aaron Conole
0 siblings, 1 reply; 200+ results
From: David Marchand @ 2020-02-20 12:22 UTC (permalink / raw)
To: Thomas Monjalon, Aaron Conole; +Cc: dev, Michael Santana, Bruce Richardson
On Thu, Feb 20, 2020 at 11:42 AM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> 19/02/2020 22:39, Aaron Conole:
> > David Marchand <david.marchand@redhat.com> writes:
> >
> > > Let's prune the jobs list to limit the amount of time spent by the robot
> > > in Travis.
> > >
> > > Since meson enables automatically the relevant components, there is not
> > > much gain in testing with extra_packages vs required_packages only.
> > >
> > > For a given arch/compiler/env combination, compilation is first tested
> > > in all jobs that run tests or build the docs or run the ABI checks.
> > > In the same context, for jobs that accumulates running tests, building
> > > the docs etc..., those steps are independent and can be split to save
> > > some cpu on Travis.
> > >
> > > With this, we go down from 21 to 15 jobs.
> > >
> > > Note: this patch requires a flush of the existing caches in Travis.
> > >
> > > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > > ---
> >
> > In general, I think the idea with required vs. extra was to have a build
> > that did the minimum required, and one that did all the packages (to
> > allow a minimum vs. full DPDK).
> >
> > At least, that's from
> > http://mails.dpdk.org/archives/dev/2019-January/124007.html
>
> I think the benefit of a minimum build is to have a quick report,
> and easy to setup.
Yes, Travis serves as a first gate when submitting patches.
But since Travis is best effort/free, we can't have a full coverage.
> > Not sure if that's still something anyone cares about.
>
> Given that Travis knows how to satisfy the dependencies,
> and that we must wait for all jobs to finish,
> I don't see any benefit of a minimal setup.
This minimal setup also tests that dpdk dependencies are correct.
If a change makes something rely on libX and libX is in the packages
always installed in Travis, the missing dependency would not get
caught.
But here, this adds too many jobs.
UNH, Intel and other CIs should step in and fill this kind of gap.
--
David Marchand
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH 4/4] ci: reorganise Travis jobs
2020-02-20 12:22 0% ` David Marchand
@ 2020-02-20 14:35 0% ` Aaron Conole
2020-02-20 16:01 0% ` David Marchand
0 siblings, 1 reply; 200+ results
From: Aaron Conole @ 2020-02-20 14:35 UTC (permalink / raw)
To: David Marchand; +Cc: Thomas Monjalon, dev, Michael Santana, Bruce Richardson
David Marchand <david.marchand@redhat.com> writes:
> On Thu, Feb 20, 2020 at 11:42 AM Thomas Monjalon <thomas@monjalon.net> wrote:
>>
>> 19/02/2020 22:39, Aaron Conole:
>> > David Marchand <david.marchand@redhat.com> writes:
>> >
>> > > Let's prune the jobs list to limit the amount of time spent by the robot
>> > > in Travis.
>> > >
>> > > Since meson enables automatically the relevant components, there is not
>> > > much gain in testing with extra_packages vs required_packages only.
>> > >
>> > > For a given arch/compiler/env combination, compilation is first tested
>> > > in all jobs that run tests or build the docs or run the ABI checks.
>> > > In the same context, for jobs that accumulates running tests, building
>> > > the docs etc..., those steps are independent and can be split to save
>> > > some cpu on Travis.
>> > >
>> > > With this, we go down from 21 to 15 jobs.
>> > >
>> > > Note: this patch requires a flush of the existing caches in Travis.
>> > >
>> > > Signed-off-by: David Marchand <david.marchand@redhat.com>
>> > > ---
>> >
>> > In general, I think the idea with required vs. extra was to have a build
>> > that did the minimum required, and one that did all the packages (to
>> > allow a minimum vs. full DPDK).
>> >
>> > At least, that's from
>> > http://mails.dpdk.org/archives/dev/2019-January/124007.html
>>
>> I think the benefit of a minimum build is to have a quick report,
>> and easy to setup.
>
> Yes, Travis serves as a first gate when submitting patches.
> But since Travis is best effort/free, we can't have a full coverage.
>
>
>> > Not sure if that's still something anyone cares about.
>>
>> Given that Travis knows how to satisfy the dependencies,
>> and that we must wait for all jobs to finish,
>> I don't see any benefit of a minimal setup.
>
> This minimal setup also tests that dpdk dependencies are correct.
> If a change makes something rely on libX and libX is in the packages
> always installed in Travis, the missing dependency would not get
> caught.
>
> But here, this adds too many jobs.
>
> UNH, Intel and other CIs should step in and fill this kind of gap.
Okay, makes sense to me. Are one of these CI providers offering to
cover this?
Also,
Acked-by: Aaron Conole <aconole@redhat.com>
>
> --
> David Marchand
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH 4/4] ci: reorganise Travis jobs
2020-02-20 14:35 0% ` Aaron Conole
@ 2020-02-20 16:01 0% ` David Marchand
2020-02-20 19:38 0% ` [dpdk-dev] [dpdk-ci] " Jeremy Plsek
0 siblings, 1 reply; 200+ results
From: David Marchand @ 2020-02-20 16:01 UTC (permalink / raw)
To: ci; +Cc: Thomas Monjalon, dev, Michael Santana, Aaron Conole, Bruce Richardson
On Thu, Feb 20, 2020 at 3:35 PM Aaron Conole <aconole@redhat.com> wrote:
>
> David Marchand <david.marchand@redhat.com> writes:
>
> > On Thu, Feb 20, 2020 at 11:42 AM Thomas Monjalon <thomas@monjalon.net> wrote:
> >>
> >> 19/02/2020 22:39, Aaron Conole:
> >> > David Marchand <david.marchand@redhat.com> writes:
> >> >
> >> > > Let's prune the jobs list to limit the amount of time spent by the robot
> >> > > in Travis.
> >> > >
> >> > > Since meson enables automatically the relevant components, there is not
> >> > > much gain in testing with extra_packages vs required_packages only.
> >> > >
> >> > > For a given arch/compiler/env combination, compilation is first tested
> >> > > in all jobs that run tests or build the docs or run the ABI checks.
> >> > > In the same context, for jobs that accumulates running tests, building
> >> > > the docs etc..., those steps are independent and can be split to save
> >> > > some cpu on Travis.
> >> > >
> >> > > With this, we go down from 21 to 15 jobs.
> >> > >
> >> > > Note: this patch requires a flush of the existing caches in Travis.
> >> > >
> >> > > Signed-off-by: David Marchand <david.marchand@redhat.com>
> >> > > ---
> >> >
> >> > In general, I think the idea with required vs. extra was to have a build
> >> > that did the minimum required, and one that did all the packages (to
> >> > allow a minimum vs. full DPDK).
> >> >
> >> > At least, that's from
> >> > http://mails.dpdk.org/archives/dev/2019-January/124007.html
> >>
> >> I think the benefit of a minimum build is to have a quick report,
> >> and easy to setup.
> >
> > Yes, Travis serves as a first gate when submitting patches.
> > But since Travis is best effort/free, we can't have a full coverage.
> >
> >
> >> > Not sure if that's still something anyone cares about.
> >>
> >> Given that Travis knows how to satisfy the dependencies,
> >> and that we must wait for all jobs to finish,
> >> I don't see any benefit of a minimal setup.
> >
> > This minimal setup also tests that dpdk dependencies are correct.
> > If a change makes something rely on libX and libX is in the packages
> > always installed in Travis, the missing dependency would not get
> > caught.
> >
> > But here, this adds too many jobs.
> >
> > UNH, Intel and other CIs should step in and fill this kind of gap.
>
> Okay, makes sense to me. Are one of these CI providers offering to
> cover this?
Maybe it is already covered, the best is to ask, so sending to ci@dpdk.org.
For the CI guys, which packages are installed on the systems/vms that
do compilation tests?
Is it possible to have a summary of the different setups?
Thanks.
--
David Marchand
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] ABI version of experimental libraries
2020-02-18 10:36 4% ` Kinsella, Ray
@ 2020-02-20 19:50 8% ` Ferruh Yigit
2020-02-20 19:54 10% ` [dpdk-dev] [PATCH] build: fix experimental library versioning Ferruh Yigit
0 siblings, 1 reply; 200+ results
From: Ferruh Yigit @ 2020-02-20 19:50 UTC (permalink / raw)
To: Kinsella, Ray, Thomas Monjalon, Richardson, Bruce
Cc: nhorman, bluca, david.marchand, ktraynor, dev
On 2/18/2020 10:36 AM, Kinsella, Ray wrote:
>
>
>> -----Original Message-----
>> From: Thomas Monjalon <thomas@monjalon.net>
>> Sent: Tuesday 18 February 2020 09:50
>> To: Richardson, Bruce <bruce.richardson@intel.com>
>> Cc: Kinsella, Ray <ray.kinsella@intel.com>; nhorman@tuxdriver.com;
>> bluca@debian.org; david.marchand@redhat.com; ktraynor@redhat.com;
>> dev@dpdk.org
>> Subject: Re: ABI version of experimental libraries
>>
>> 18/02/2020 10:42, Bruce Richardson:
>>> On Tue, Feb 18, 2020 at 12:15:56AM +0100, Thomas Monjalon wrote:
>>>> Hi,
>>>>
>>>> I would like to remind everybody our mistake when defining ABI
>> versions.
>>>> It has been "fixed" in this commit:
>>>> http://git.dpdk.org/dpdk/commit/?id=f26c2b39
>>>>
>>>> Please let's think about the consequence for the experimental
>> libraries.
>>>>
>>>> In DPDK 19.11, we use the ABI version 0.200 with soname 0.20 In
>> DPDK
>>>> 20.02, we use the ABI version 0.2001 with soname 0.201 Numbers are
>>>> increasing, that's fine. When we'll switch to the new major ABI
>> and
>>>> use a normal numbering: In DPDK 20.11, we will use the ABI version
>>>> 0.210 with soname 0.21 Numbers are dropping.
>>>>
>>>> In short, for experimental libs, ABI 20.1 > ABI 21.0
>>>>
>>>> Are we OK with this or do we prefer reverting to normal numbering
>>>> for experimental libraries in DPDK 20.02?
>>>>
>>> Personally, I would not be too concerned about the verions of
>>> experimental libs, so long as they don't conflict across versions and
>>> have some similarity to the major ABI version for the release.
>>
>> You think sorting of the version numbers is not important?
>> If we don't care comparing experimental version numbers, then OK, let's
>> drop this patch. But please we need a small vote.
>>
>> Note: there would be no problem if we did not vote for having a special
>> numbering for pure experimental libraries (I am still against).
>>
>
> So while experimental library version numbers are not "important".
> I do agree with Thomas they should be sane, increase and should have a consistent format.
>
> Should we always pad them to 4 places?
> i.e.
>
> DPDK 19.11 ... 0.20 (needs to remain 0.20).
> DPDK 20.02 ... 0.2001
> DPDK 20.11 ... 0.2100
> DPDK 21.02 ... 0.2101
>
> Make sense?
>
What about following:
DPDK version ABI version soname library name
------------ ----------- ------ ------------
DPDK 19.11 20.0 .so.0.200 .so.0.200
DPDK 20.02 20.0.1 .so.0.200.1 .so.0.200.1
DPDK 20.05 20.0.2 .so.0.200.2 .so.0.200.2
DPDK 20.11 21.0 .so.0.210 .so.0.210
DPDK 21.02 21.1 .so.0.211 .so.0.211
This let us turn back to sane versions when soname fixed in 20.11.
Also I think for experimental libraries soname should NOT be same in minor
versions, otherwise it can create false impression that it is ABI compatible
between two minor versions.
I will reply the patch to this email.
^ permalink raw reply [relevance 8%]
* [dpdk-dev] [PATCH] build: fix experimental library versioning
2020-02-20 19:50 8% ` Ferruh Yigit
@ 2020-02-20 19:54 10% ` Ferruh Yigit
2020-02-20 22:14 0% ` Luca Boccassi
` (3 more replies)
0 siblings, 4 replies; 200+ results
From: Ferruh Yigit @ 2020-02-20 19:54 UTC (permalink / raw)
To: Thomas Monjalon, Neil Horman, Bruce Richardson
Cc: dev, Ferruh Yigit, ray.kinsella, bluca, david.marchand, ktraynor
The problem occurred when workaround that makes soname two digits
applied. With this change for the ABI version "20.0.1" the experimental
library version become ".so.2001".
After workaround removed in ABI version 21.0, the experimental library
version will become ".so.210".
"2001" is bigger value than "201" although it is a previous version of
the library version, this can break the version comparisons.
To fix this, introducing a temporary sub level versioning for the
experimental libraries, so that minor version comparison will continue
to work.
After change experimental libraries will follow below versioning:
DPDK version ABI version soname library name
------------ ----------- ------ ------------
DPDK 19.11 20.0 .so.0.200 .so.0.200
DPDK 20.02 20.0.1 .so.0.200.1 .so.0.200.1
DPDK 20.05 20.0.2 .so.0.200.2 .so.0.200.2
DPDK 20.11 21.0 .so.0.210 .so.0.210
DPDK 21.02 21.1 .so.0.211 .so.0.211
Note: After workaround removed in DPDK 20.11 and soname switch back to
single digit this patch won't work and needs to be updated.
Fixes: f26c2b39b271 ("build: fix soname info for 19.11 compatibility")
Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
---
Cc: Thomas Monjalon <thomas@monjalon.net>
Cc: ray.kinsella@intel.com
Cc: nhorman@tuxdriver.com
Cc: bruce.richardson@intel.com
Cc: bluca@debian.org
Cc: david.marchand@redhat.com
Cc: ktraynor@redhat.com
---
config/meson.build | 4 ++--
mk/rte.lib.mk | 8 +++++---
2 files changed, 7 insertions(+), 5 deletions(-)
diff --git a/config/meson.build b/config/meson.build
index 6c46767e3..d3d2370ce 100644
--- a/config/meson.build
+++ b/config/meson.build
@@ -31,8 +31,8 @@ abi_version = run_command(find_program('cat', 'more'),
# sonames => librte_stable.so.20.0, librte_experimental.so.0.200
abi_va = abi_version.split('.')
stable_so_version = abi_va.length() == 2 ? abi_va[0] : abi_va[0] + '.' + abi_va[1]
-experimental_abi_version = '0.' + ''.join(abi_va)
-experimental_so_version = '0.' + ''.join(stable_so_version.split('.'))
+experimental_abi_version = '0.' + abi_va[0] + abi_va[1] + '.' + abi_va[2]
+experimental_so_version = experimental_abi_version
# extract all version information into the build configuration
dpdk_conf.set('RTE_VER_YEAR', pver.get(0).to_int())
diff --git a/mk/rte.lib.mk b/mk/rte.lib.mk
index b1a8372cc..682b590db 100644
--- a/mk/rte.lib.mk
+++ b/mk/rte.lib.mk
@@ -15,8 +15,8 @@ LIBABIVER ?= $(shell cat $(RTE_SRCDIR)/ABI_VERSION)
SOVER := $(basename $(LIBABIVER))
ifeq ($(shell grep -s "^DPDK_" $(SRCDIR)/$(EXPORT_MAP)),)
# EXPERIMENTAL ABI is versioned as 0.major+minor, e.g. 0.201 for 20.1 ABI
-LIBABIVER := 0.$(shell echo $(LIBABIVER) | tr -d '.')
-SOVER := 0.$(shell echo $(SOVER) | tr -d '.')
+LIBABIVER := 0.$(shell echo $(LIBABIVER) | awk 'BEGIN { FS="." }; { print $$1$$2"."$$3 }')
+SOVER := $(LIBABIVER)
endif
ifeq ($(CONFIG_RTE_BUILD_SHARED_LIB),y)
@@ -135,7 +135,9 @@ $(RTE_OUTPUT)/lib/$(LIB): $(LIB)
$(Q)cp -f $(LIB) $(RTE_OUTPUT)/lib
ifeq ($(CONFIG_RTE_BUILD_SHARED_LIB),y)
$(Q)ln -s -f $< $(shell echo $@ | sed 's/\.so.*/.so/')
- $(Q)ln -s -f $< $(shell echo $@ | sed 's/\.so.*/.so.$(SOVER)/')
+ $(Q)if [ $(SOVER) != $(LIBABIVER) ]; then \
+ ln -s -f $< $(shell echo $@ | sed 's/\.so.*/.so.$(SOVER)/') ; \
+ fi
endif
#
--
2.24.1
^ permalink raw reply [relevance 10%]
* Re: [dpdk-dev] [PATCH] build: fix experimental library versioning
2020-02-20 19:54 10% ` [dpdk-dev] [PATCH] build: fix experimental library versioning Ferruh Yigit
@ 2020-02-20 22:14 0% ` Luca Boccassi
2020-02-21 12:36 0% ` Ray Kinsella
` (2 subsequent siblings)
3 siblings, 0 replies; 200+ results
From: Luca Boccassi @ 2020-02-20 22:14 UTC (permalink / raw)
To: Ferruh Yigit, Thomas Monjalon, Neil Horman, Bruce Richardson
Cc: dev, ray.kinsella, david.marchand, ktraynor
On Thu, 2020-02-20 at 19:54 +0000, Ferruh Yigit wrote:
> The problem occurred when workaround that makes soname two digits
> applied. With this change for the ABI version "20.0.1" the
> experimental
> library version become ".so.2001".
> After workaround removed in ABI version 21.0, the experimental
> library
> version will become ".so.210".
> "2001" is bigger value than "201" although it is a previous version
> of
> the library version, this can break the version comparisons.
>
> To fix this, introducing a temporary sub level versioning for the
> experimental libraries, so that minor version comparison will
> continue
> to work.
>
> After change experimental libraries will follow below versioning:
>
> DPDK version ABI version soname library name
> ------------ ----------- ------ ------------
> DPDK 19.11 20.0 .so.0.200 .so.0.200
> DPDK 20.02 20.0.1 .so.0.200.1 .so.0.200.1
> DPDK 20.05 20.0.2 .so.0.200.2 .so.0.200.2
> DPDK 20.11 21.0 .so.0.210 .so.0.210
> DPDK 21.02 21.1 .so.0.211 .so.0.211
>
> Note: After workaround removed in DPDK 20.11 and soname switch back
> to
> single digit this patch won't work and needs to be updated.
>
> Fixes: f26c2b39b271 ("build: fix soname info for 19.11
> compatibility")
>
> Signed-off-by: Ferruh Yigit <
> ferruh.yigit@intel.com
Acked-by: Luca Boccassi <bluca@debian.org>
This solution looks like a good compromise to me. Thanks Ferruh.
--
Kind regards,
Luca Boccassi
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] doc: deprecate using MAX values as array size
@ 2020-02-21 10:25 0% ` Ferruh Yigit
2020-02-24 6:18 0% ` Akhil Goyal
2020-02-24 8:51 0% ` Andrew Rybchenko
2020-02-25 13:52 0% ` David Marchand
2020-02-25 14:53 0% ` David Marchand
2 siblings, 2 replies; 200+ results
From: Ferruh Yigit @ 2020-02-21 10:25 UTC (permalink / raw)
To: Neil Horman, John McNamara, Marko Kovacevic
Cc: dev, Thomas Monjalon, Bruce Richardson, Akhil Goyal,
Konstantin Ananyev, David Marchand, Jerin Jacob,
Andrew Rybchenko, Ori Kam
On 1/30/2020 2:20 PM, Ferruh Yigit wrote:
> Adding the deprecation notice as reminder for next ABI breakage release
> (20.11).
> This one time breakage is required to be able to extend enum/define
> without breaking ABI.
>
> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
> ---
> doc/guides/rel_notes/deprecation.rst | 14 ++++++++++++++
> 1 file changed, 14 insertions(+)
>
> diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
> index dfcca87ab..99d81564a 100644
> --- a/doc/guides/rel_notes/deprecation.rst
> +++ b/doc/guides/rel_notes/deprecation.rst
> @@ -38,6 +38,20 @@ Deprecation Notices
> remove it from the externally visible ABI and allow it to be updated in the
> future.
>
> +* lib: will fix extending some enum/define breaking the ABI. There are multiple
> + samples in DPDK that enum/define terminated with a ``.*MAX.*`` value which is
> + used by iterators, and arrays holding these values are sized with this
> + ``.*MAX.*`` value. So extending this enum/define increases the ``.*MAX.*``
> + value which increases the size of the array and depending on how/where the
> + array is used this may break the ABI.
> + ``RTE_ETH_FLOW_MAX`` is one sample of the mentioned case, adding a new flow
> + type will break the ABI because of ``flex_mask[RTE_ETH_FLOW_MAX]`` array
> + usage in following public struct hierarchy:
> + ``rte_eth_fdir_flex_conf -> rte_fdir_conf -> rte_eth_conf (in the middle)``.
> + Need to identify this kind of usages and fix in 20.11, otherwise this blocks
> + us extending existing enum/define.
> + One solution can be using a fixed size array instead of ``.*MAX.*`` value.
> +
> * dpaa2: removal of ``rte_dpaa2_memsegs`` structure which has been replaced
> by a pa-va search library. This structure was earlier being used for holding
> memory segments used by dpaa2 driver for faster pa->va translation. This
>
Reminder of this deprecation notice.
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] doc: plan splitting the ethdev ops struct
2020-02-18 6:01 0% ` Stephen Hemminger
@ 2020-02-21 10:40 0% ` Ferruh Yigit
2020-02-25 10:35 0% ` Andrew Rybchenko
0 siblings, 1 reply; 200+ results
From: Ferruh Yigit @ 2020-02-21 10:40 UTC (permalink / raw)
To: Stephen Hemminger
Cc: Neil Horman, John McNamara, Marko Kovacevic, dev, David Marchand,
Thomas Monjalon, Andrew Rybchenko
On 2/18/2020 6:01 AM, Stephen Hemminger wrote:
> On Mon, 17 Feb 2020 15:38:05 +0000
> Ferruh Yigit <ferruh.yigit@intel.com> wrote:
>
>> For the ABI compatibility it is better to hide internal data structures
>> from the application as much as possible. But because of some inline
>> functions 'struct eth_dev_ops' can't be hidden completely.
>>
>> Plan is to split the 'struct eth_dev_ops' into two as ones used by
>> inline functions and ones not used, and hide the second part that not
>> used by inline functions completely to the application.
>>
>> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
>> ---
>> Cc: David Marchand <david.marchand@redhat.com>
>> Cc: Thomas Monjalon <thomas@monjalon.net>
>> Cc: Andrew Rybchenko <arybchenko@solarflare.com>
>> ---
>> doc/guides/rel_notes/deprecation.rst | 6 ++++++
>> 1 file changed, 6 insertions(+)
>>
>> diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
>> index dfcca87ab..2aa431028 100644
>> --- a/doc/guides/rel_notes/deprecation.rst
>> +++ b/doc/guides/rel_notes/deprecation.rst
>> @@ -72,6 +72,12 @@ Deprecation Notices
>> In 19.11 PMDs will still update the field even when the offload is not
>> enabled.
>>
>> +* ethdev: Split the ``struct eth_dev_ops`` struct to hide it as much as possible.
>> + Currently the ``struct eth_dev_ops`` struct is accessible by the application
>> + because some inline functions, like ``rte_eth_tx_descriptor_status()``,
>> + access the struct directly. The struct will be separate in two, the ops used
>> + by inline functions still will be accessible to user but rest will be hidden.
>> +
>> * cryptodev: support for using IV with all sizes is added, J0 still can
>> be used but only when IV length in following structs ``rte_crypto_auth_xform``,
>> ``rte_crypto_aead_xform`` is set to zero. When IV length is greater or equal
>
> Good luck, truely hiding internals is hard. The mbuf structure is already split but not really
> hidden at all (just look at dwarf output). It doesn't make sense to do it unless
> you can really hide it.
I believe this can be done, only following [1] dev_ops are used by inline
functions, rest can be moved into separate struct and moved into ethdev driver
looking header.
[1]
rx_queue_count
rx_descriptor_done
rx_descriptor_status
tx_descriptor_status
>
> I would attack the rte_device stuff first. Make rte_device opaque to the application
> that would help for future versions. Then work backwards to rte_tehtdev.
>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] build: fix experimental library versioning
2020-02-20 19:54 10% ` [dpdk-dev] [PATCH] build: fix experimental library versioning Ferruh Yigit
2020-02-20 22:14 0% ` Luca Boccassi
@ 2020-02-21 12:36 0% ` Ray Kinsella
2020-02-21 15:24 5% ` David Marchand
2020-02-21 16:41 0% ` David Marchand
3 siblings, 0 replies; 200+ results
From: Ray Kinsella @ 2020-02-21 12:36 UTC (permalink / raw)
To: Ferruh Yigit, Thomas Monjalon, Neil Horman, Bruce Richardson
Cc: dev, ray.kinsella, bluca, david.marchand, ktraynor
On 20/02/2020 19:54, Ferruh Yigit wrote:
> The problem occurred when workaround that makes soname two digits
> applied. With this change for the ABI version "20.0.1" the experimental
> library version become ".so.2001".
> After workaround removed in ABI version 21.0, the experimental library
> version will become ".so.210".
> "2001" is bigger value than "201" although it is a previous version of
> the library version, this can break the version comparisons.
>
> To fix this, introducing a temporary sub level versioning for the
> experimental libraries, so that minor version comparison will continue
> to work.
>
> After change experimental libraries will follow below versioning:
>
> DPDK version ABI version soname library name
> ------------ ----------- ------ ------------
> DPDK 19.11 20.0 .so.0.200 .so.0.200
> DPDK 20.02 20.0.1 .so.0.200.1 .so.0.200.1
> DPDK 20.05 20.0.2 .so.0.200.2 .so.0.200.2
> DPDK 20.11 21.0 .so.0.210 .so.0.210
> DPDK 21.02 21.1 .so.0.211 .so.0.211
>
> Note: After workaround removed in DPDK 20.11 and soname switch back to
> single digit this patch won't work and needs to be updated.
>
> Fixes: f26c2b39b271 ("build: fix soname info for 19.11 compatibility")
>
Acked-by: Ray Kinsella <mdr@ashroe.eu>
> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
> ---
> Cc: Thomas Monjalon <thomas@monjalon.net>
> Cc: ray.kinsella@intel.com
> Cc: nhorman@tuxdriver.com
> Cc: bruce.richardson@intel.com
> Cc: bluca@debian.org
> Cc: david.marchand@redhat.com
> Cc: ktraynor@redhat.com
> ---
> config/meson.build | 4 ++--
> mk/rte.lib.mk | 8 +++++---
> 2 files changed, 7 insertions(+), 5 deletions(-)
>
> diff --git a/config/meson.build b/config/meson.build
> index 6c46767e3..d3d2370ce 100644
> --- a/config/meson.build
> +++ b/config/meson.build
> @@ -31,8 +31,8 @@ abi_version = run_command(find_program('cat', 'more'),
> # sonames => librte_stable.so.20.0, librte_experimental.so.0.200
> abi_va = abi_version.split('.')
> stable_so_version = abi_va.length() == 2 ? abi_va[0] : abi_va[0] + '.' + abi_va[1]
> -experimental_abi_version = '0.' + ''.join(abi_va)
> -experimental_so_version = '0.' + ''.join(stable_so_version.split('.'))
> +experimental_abi_version = '0.' + abi_va[0] + abi_va[1] + '.' + abi_va[2]
> +experimental_so_version = experimental_abi_version
>
> # extract all version information into the build configuration
> dpdk_conf.set('RTE_VER_YEAR', pver.get(0).to_int())
> diff --git a/mk/rte.lib.mk b/mk/rte.lib.mk
> index b1a8372cc..682b590db 100644
> --- a/mk/rte.lib.mk
> +++ b/mk/rte.lib.mk
> @@ -15,8 +15,8 @@ LIBABIVER ?= $(shell cat $(RTE_SRCDIR)/ABI_VERSION)
> SOVER := $(basename $(LIBABIVER))
> ifeq ($(shell grep -s "^DPDK_" $(SRCDIR)/$(EXPORT_MAP)),)
> # EXPERIMENTAL ABI is versioned as 0.major+minor, e.g. 0.201 for 20.1 ABI
> -LIBABIVER := 0.$(shell echo $(LIBABIVER) | tr -d '.')
> -SOVER := 0.$(shell echo $(SOVER) | tr -d '.')
> +LIBABIVER := 0.$(shell echo $(LIBABIVER) | awk 'BEGIN { FS="." }; { print $$1$$2"."$$3 }')
> +SOVER := $(LIBABIVER)
> endif
>
> ifeq ($(CONFIG_RTE_BUILD_SHARED_LIB),y)
> @@ -135,7 +135,9 @@ $(RTE_OUTPUT)/lib/$(LIB): $(LIB)
> $(Q)cp -f $(LIB) $(RTE_OUTPUT)/lib
> ifeq ($(CONFIG_RTE_BUILD_SHARED_LIB),y)
> $(Q)ln -s -f $< $(shell echo $@ | sed 's/\.so.*/.so/')
> - $(Q)ln -s -f $< $(shell echo $@ | sed 's/\.so.*/.so.$(SOVER)/')
> + $(Q)if [ $(SOVER) != $(LIBABIVER) ]; then \
> + ln -s -f $< $(shell echo $@ | sed 's/\.so.*/.so.$(SOVER)/') ; \
> + fi
> endif
>
> #
>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] build: fix experimental library versioning
2020-02-20 19:54 10% ` [dpdk-dev] [PATCH] build: fix experimental library versioning Ferruh Yigit
2020-02-20 22:14 0% ` Luca Boccassi
2020-02-21 12:36 0% ` Ray Kinsella
@ 2020-02-21 15:24 5% ` David Marchand
2020-02-21 15:34 3% ` Ferruh Yigit
2020-02-21 16:41 0% ` David Marchand
3 siblings, 1 reply; 200+ results
From: David Marchand @ 2020-02-21 15:24 UTC (permalink / raw)
To: Ferruh Yigit
Cc: Thomas Monjalon, Neil Horman, Bruce Richardson, dev, Kinsella,
Ray, Luca Boccassi, Kevin Traynor
On Thu, Feb 20, 2020 at 8:55 PM Ferruh Yigit <ferruh.yigit@intel.com> wrote:
>
> The problem occurred when workaround that makes soname two digits
> applied. With this change for the ABI version "20.0.1" the experimental
> library version become ".so.2001".
> After workaround removed in ABI version 21.0, the experimental library
> version will become ".so.210".
> "2001" is bigger value than "201" although it is a previous version of
> the library version, this can break the version comparisons.
>
> To fix this, introducing a temporary sub level versioning for the
> experimental libraries, so that minor version comparison will continue
> to work.
>
> After change experimental libraries will follow below versioning:
>
> DPDK version ABI version soname library name
> ------------ ----------- ------ ------------
> DPDK 19.11 20.0 .so.0.200 .so.0.200
> DPDK 20.02 20.0.1 .so.0.200.1 .so.0.200.1
> DPDK 20.05 20.0.2 .so.0.200.2 .so.0.200.2
> DPDK 20.11 21.0 .so.0.210 .so.0.210
> DPDK 21.02 21.1 .so.0.211 .so.0.211
>
> Note: After workaround removed in DPDK 20.11 and soname switch back to
> single digit this patch won't work and needs to be updated.
>
> Fixes: f26c2b39b271 ("build: fix soname info for 19.11 compatibility")
>
> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
The soname change breaks existing applications like testpmd (linked to
various experimental libraries).
I have the full details if needed in a pastebin (v19.11, master, patch_applied).
$ LD_LIBRARY_PATH=$HOME/builds/x86_64-native-linux-gcc+shared+kmods/install/lib
$HOME/abi/v19.11/x86_64-native-linux-gcc+shared+kmods/bin/testpmd
/home/dmarchan/abi/v19.11/x86_64-native-linux-gcc+shared+kmods/bin/testpmd:
error while loading shared libraries: librte_flow_classify.so.0.200:
cannot open shared object file: No such file or directory
libabigail reports this change as an error, so if it is intended, we
need to bypass the check for those libraries.
Functions changes summary: 0 Removed, 0 Changed (2 filtered out), 0
Added functions
Variables changes summary: 0 Removed, 0 Changed, 0 Added variable
ELF SONAME changed
Functions changes summary: 0 Removed, 0 Changed (7 filtered out), 0
Added functions
Variables changes summary: 0 Removed, 0 Changed, 0 Added variable
SONAME changed from 'librte_flow_classify.so.0.200' to
'librte_flow_classify.so.0.200.1'
Error: ABI issue reported for 'abidiff --suppr
/home/dmarchan/dpdk/devtools/libabigail.abignore --no-added-syms
--headers-dir1 /home/dmarchan/abi/v19.11/x86_64-native-linux-gcc+shared+kmods/include
--headers-dir2 /home/dmarchan/builds/x86_64-native-linux-gcc+shared+kmods/install/include
/home/dmarchan/abi/v19.11/x86_64-native-linux-gcc+shared+kmods/dump/librte_flow_classify.dump
/home/dmarchan/builds/x86_64-native-linux-gcc+shared+kmods/install/dump/librte_flow_classify.dump'
--
David Marchand
^ permalink raw reply [relevance 5%]
* Re: [dpdk-dev] [PATCH] build: fix experimental library versioning
2020-02-21 15:24 5% ` David Marchand
@ 2020-02-21 15:34 3% ` Ferruh Yigit
0 siblings, 0 replies; 200+ results
From: Ferruh Yigit @ 2020-02-21 15:34 UTC (permalink / raw)
To: David Marchand
Cc: Thomas Monjalon, Neil Horman, Bruce Richardson, dev, Kinsella,
Ray, Luca Boccassi, Kevin Traynor
On 2/21/2020 3:24 PM, David Marchand wrote:
> On Thu, Feb 20, 2020 at 8:55 PM Ferruh Yigit <ferruh.yigit@intel.com> wrote:
>>
>> The problem occurred when workaround that makes soname two digits
>> applied. With this change for the ABI version "20.0.1" the experimental
>> library version become ".so.2001".
>> After workaround removed in ABI version 21.0, the experimental library
>> version will become ".so.210".
>> "2001" is bigger value than "201" although it is a previous version of
>> the library version, this can break the version comparisons.
>>
>> To fix this, introducing a temporary sub level versioning for the
>> experimental libraries, so that minor version comparison will continue
>> to work.
>>
>> After change experimental libraries will follow below versioning:
>>
>> DPDK version ABI version soname library name
>> ------------ ----------- ------ ------------
>> DPDK 19.11 20.0 .so.0.200 .so.0.200
>> DPDK 20.02 20.0.1 .so.0.200.1 .so.0.200.1
>> DPDK 20.05 20.0.2 .so.0.200.2 .so.0.200.2
>> DPDK 20.11 21.0 .so.0.210 .so.0.210
>> DPDK 21.02 21.1 .so.0.211 .so.0.211
>>
>> Note: After workaround removed in DPDK 20.11 and soname switch back to
>> single digit this patch won't work and needs to be updated.
>>
>> Fixes: f26c2b39b271 ("build: fix soname info for 19.11 compatibility")
>>
>> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
>
> The soname change breaks existing applications like testpmd (linked to
> various experimental libraries).
> I have the full details if needed in a pastebin (v19.11, master, patch_applied).
>
> $ LD_LIBRARY_PATH=$HOME/builds/x86_64-native-linux-gcc+shared+kmods/install/lib
> $HOME/abi/v19.11/x86_64-native-linux-gcc+shared+kmods/bin/testpmd
> /home/dmarchan/abi/v19.11/x86_64-native-linux-gcc+shared+kmods/bin/testpmd:
> error while loading shared libraries: librte_flow_classify.so.0.200:
> cannot open shared object file: No such file or directory
>
>
> libabigail reports this change as an error, so if it is intended, we
> need to bypass the check for those libraries.
soname change was intended, we don't provide any ABI compatibility in the
experimental libraries, keeping same soname may mislead as if we do.
The impact of it is, any application linked against experimental library will
need to recompile in new release.
Can we add exception to the the libabigail for experimental libraries?
>
> Functions changes summary: 0 Removed, 0 Changed (2 filtered out), 0
> Added functions
> Variables changes summary: 0 Removed, 0 Changed, 0 Added variable
> ELF SONAME changed
> Functions changes summary: 0 Removed, 0 Changed (7 filtered out), 0
> Added functions
> Variables changes summary: 0 Removed, 0 Changed, 0 Added variable
> SONAME changed from 'librte_flow_classify.so.0.200' to
> 'librte_flow_classify.so.0.200.1'
> Error: ABI issue reported for 'abidiff --suppr
> /home/dmarchan/dpdk/devtools/libabigail.abignore --no-added-syms
> --headers-dir1 /home/dmarchan/abi/v19.11/x86_64-native-linux-gcc+shared+kmods/include
> --headers-dir2 /home/dmarchan/builds/x86_64-native-linux-gcc+shared+kmods/install/include
> /home/dmarchan/abi/v19.11/x86_64-native-linux-gcc+shared+kmods/dump/librte_flow_classify.dump
> /home/dmarchan/builds/x86_64-native-linux-gcc+shared+kmods/install/dump/librte_flow_classify.dump'
>
>
>
^ permalink raw reply [relevance 3%]
* [dpdk-dev] [PATCH] devtools: skip experimental libraries in ABI check
@ 2020-02-21 16:10 17% David Marchand
2020-02-21 16:13 4% ` Ferruh Yigit
0 siblings, 1 reply; 200+ results
From: David Marchand @ 2020-02-21 16:10 UTC (permalink / raw)
To: dev; +Cc: thomas, ferruh.yigit, dodji
We don't provide ABI compatibility for experimental libraries.
Skip those libraries by catching a soname containing a version starting
with '0.'.
Align the special case for the glue libraries by using the soname too.
Once libabigail has support for it, we will have a single type of rule.
Fixes: 777014e56d07 ("devtools: add ABI checks")
Signed-off-by: David Marchand <david.marchand@redhat.com>
---
devtools/check-abi.sh | 9 +++++++--
1 file changed, 7 insertions(+), 2 deletions(-)
diff --git a/devtools/check-abi.sh b/devtools/check-abi.sh
index 0b4d1a37e..dd9120e69 100755
--- a/devtools/check-abi.sh
+++ b/devtools/check-abi.sh
@@ -40,8 +40,13 @@ for dump in $(find $refdir -name "*.dump"); do
# skip glue drivers, example librte_pmd_mlx5_glue.dump
# We can't rely on a suppression rule for now:
# https://sourceware.org/bugzilla/show_bug.cgi?id=25480
- if [ "$name" != "${name%%_glue.dump}" ]; then
- echo "Skipping ${dump}..."
+ if grep -qE "\<soname='[^']*_glue\.so\.[^']*'" $dump; then
+ echo "Skipped glue library $name."
+ continue
+ fi
+ # skip experimental libraries, with a sover starting with 0.
+ if grep -qE "\<soname='[^']*\.so\.0\.[^']*'" $dump; then
+ echo "Skipped experimental library $name."
continue
fi
dump2=$(find $newdir -name $name)
--
2.23.0
^ permalink raw reply [relevance 17%]
* Re: [dpdk-dev] [PATCH] devtools: skip experimental libraries in ABI check
2020-02-21 16:10 17% [dpdk-dev] [PATCH] devtools: skip experimental libraries in ABI check David Marchand
@ 2020-02-21 16:13 4% ` Ferruh Yigit
2020-02-21 16:30 4% ` Thomas Monjalon
0 siblings, 1 reply; 200+ results
From: Ferruh Yigit @ 2020-02-21 16:13 UTC (permalink / raw)
To: David Marchand, dev; +Cc: thomas, dodji
On 2/21/2020 4:10 PM, David Marchand wrote:
> We don't provide ABI compatibility for experimental libraries.
> Skip those libraries by catching a soname containing a version starting
> with '0.'.
I am not familiar with the script and how libabigail works, but +1 to the idea.
>
> Align the special case for the glue libraries by using the soname too.
> Once libabigail has support for it, we will have a single type of rule.
>
> Fixes: 777014e56d07 ("devtools: add ABI checks")
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
> devtools/check-abi.sh | 9 +++++++--
> 1 file changed, 7 insertions(+), 2 deletions(-)
>
> diff --git a/devtools/check-abi.sh b/devtools/check-abi.sh
> index 0b4d1a37e..dd9120e69 100755
> --- a/devtools/check-abi.sh
> +++ b/devtools/check-abi.sh
> @@ -40,8 +40,13 @@ for dump in $(find $refdir -name "*.dump"); do
> # skip glue drivers, example librte_pmd_mlx5_glue.dump
> # We can't rely on a suppression rule for now:
> # https://sourceware.org/bugzilla/show_bug.cgi?id=25480
> - if [ "$name" != "${name%%_glue.dump}" ]; then
> - echo "Skipping ${dump}..."
> + if grep -qE "\<soname='[^']*_glue\.so\.[^']*'" $dump; then
> + echo "Skipped glue library $name."
> + continue
> + fi
> + # skip experimental libraries, with a sover starting with 0.
> + if grep -qE "\<soname='[^']*\.so\.0\.[^']*'" $dump; then
> + echo "Skipped experimental library $name."
> continue
> fi
> dump2=$(find $newdir -name $name)
>
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH] devtools: skip experimental libraries in ABI check
2020-02-21 16:13 4% ` Ferruh Yigit
@ 2020-02-21 16:30 4% ` Thomas Monjalon
2020-02-21 16:40 4% ` David Marchand
0 siblings, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-02-21 16:30 UTC (permalink / raw)
To: David Marchand; +Cc: dev, Ferruh Yigit, dodji
21/02/2020 17:13, Ferruh Yigit:
> On 2/21/2020 4:10 PM, David Marchand wrote:
> > We don't provide ABI compatibility for experimental libraries.
> > Skip those libraries by catching a soname containing a version starting
> > with '0.'.
>
> I am not familiar with the script and how libabigail works, but +1 to the idea.
>
> >
> > Align the special case for the glue libraries by using the soname too.
> > Once libabigail has support for it, we will have a single type of rule.
> >
> > Fixes: 777014e56d07 ("devtools: add ABI checks")
> >
> > Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Thomas Monjalon <thomas@monjalon.net>
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH 0/4] Extend --lcores to run on cores > RTE_MAX_LCORE
@ 2020-02-21 16:38 3% ` Stephen Hemminger
0 siblings, 0 replies; 200+ results
From: Stephen Hemminger @ 2020-02-21 16:38 UTC (permalink / raw)
To: Aaron Conole
Cc: David Marchand, Song, Keesang, ktraynor, bluca, Thomas Monjalon,
dev, ferruh.yigit, bruce.richardson, honnappa.nagarahalli, drc,
stable, Grimm, Jon, Hollingsworth, Brent
On Fri, 21 Feb 2020 09:48:58 -0500
Aaron Conole <aconole@redhat.com> wrote:
> David Marchand <david.marchand@redhat.com> writes:
>
> > On Fri, Feb 21, 2020 at 9:19 AM Song, Keesang <Keesang.Song@amd.com> wrote:
> >>
> >> [AMD Official Use Only - Internal Distribution Only]
> >
> > Please, get this header removed.
> > This is a public mailing list.
> >
> >
> >> Thanks Thomas for bringing this up.
> >> I consider this is not a new feature, but rather a fix to address
> >> the issue with statically assigned maximum lcore limit on
> >> high-density CPU platform such as AMD Epyc.
> >> As I see a lot of DPDK adopters are still using LTS 18.11 & 19.11,
> >> and they have 1~2 yrs of lifetime left, we like to backport this to
> >> LTS 18.11 & 19.11 at least.
> >
> > It is not a fix.
> >
> > The use of static arrays is a design choice that goes back to the
> > early days in dpdk.
> > The addition of --lcores came in after this, but it was introduced for
> > a different use case than placing lcores on any physical core.
> > And there was no claim that a core > RTE_MAX_LCORE would be usable.
> >
> >
> > When testing on a new hardware, it is normal to observe some limitations.
> > Running DPDK on those platforms should be possible: "should be"
> > because I do not have access to this hardware and saw neither tests
> > reports nor performance numbers.
> > Before this patch, the limitation is that on Epyc, cores >
> > RTE_MAX_LCORE are not usable.
> >
> >
> > Now, this change is quite constrained.
> > If we backport it, I don't expect issues in the main dpdk components
> > (based on code review and ovs tests with a RTE_MAX_LCORE set to 16 on
> > a 24 cores system).
> > There might be issues in some examples or not widely used library
> > which uses a physical core id instead of a lcore id.
> >
> >
> > This is the same recurring question "do we allow new features in a
> > stable branch?".
>
> Usually, the answer is 'no'. But we do allow some "new" things to be
> backported (pci ids, etc) that might be required to enable older
> functionality. Additionally, I'm sure if some feature were required to
> mitigate a CVE, we'd rather favor backporting it.
>
> I guess we could pose a litmus test:
>
> 1. Is the problem this feature solves so widespread that it needs to
> be addressed ASAP?
> 2. Is there a known workaround to the problem this is solving?
> 3. How intrusive is the feature?
> 4. Is it shown to be stable in the mainline (number of fixes, testing,
> etc)?
> 5. Is it constrained enough that we know we can support it with even
> higher priority than other things?
>
> Probably other questions that will need to be asked.
>
> And even in that list of question, I'm not sure I'd be able to advocate
> backporting this in the upstream branches - it hasn't had much testing.
> It's unstable. It's "difficult" to use. It is not widespread that
> people have so many cores. The workaround is much simpler than
> supporting this (recompile).
>
> >
> > --
> > David Marchand
>
RTE_MAX_LCORES is exposed in API/ABI to application.
Many applications use that to size internal data structures.
Having rte_lcore_id() potentially return a larger value would cause
out of bounds access (and crash) in that application.
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH] devtools: skip experimental libraries in ABI check
2020-02-21 16:30 4% ` Thomas Monjalon
@ 2020-02-21 16:40 4% ` David Marchand
0 siblings, 0 replies; 200+ results
From: David Marchand @ 2020-02-21 16:40 UTC (permalink / raw)
To: Thomas Monjalon; +Cc: dev, Ferruh Yigit, Dodji Seketeli
On Fri, Feb 21, 2020 at 5:30 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> 21/02/2020 17:13, Ferruh Yigit:
> > On 2/21/2020 4:10 PM, David Marchand wrote:
> > > We don't provide ABI compatibility for experimental libraries.
> > > Skip those libraries by catching a soname containing a version starting
> > > with '0.'.
> >
> > I am not familiar with the script and how libabigail works, but +1 to the idea.
> >
> > >
> > > Align the special case for the glue libraries by using the soname too.
> > > Once libabigail has support for it, we will have a single type of rule.
> > >
> > > Fixes: 777014e56d07 ("devtools: add ABI checks")
> > >
> > > Signed-off-by: David Marchand <david.marchand@redhat.com>
>
> Acked-by: Thomas Monjalon <thomas@monjalon.net>
Applied.
--
David Marchand
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH] build: fix experimental library versioning
2020-02-20 19:54 10% ` [dpdk-dev] [PATCH] build: fix experimental library versioning Ferruh Yigit
` (2 preceding siblings ...)
2020-02-21 15:24 5% ` David Marchand
@ 2020-02-21 16:41 0% ` David Marchand
3 siblings, 0 replies; 200+ results
From: David Marchand @ 2020-02-21 16:41 UTC (permalink / raw)
To: Ferruh Yigit
Cc: Thomas Monjalon, Neil Horman, Bruce Richardson, dev, Kinsella,
Ray, Luca Boccassi, Kevin Traynor
On Thu, Feb 20, 2020 at 8:55 PM Ferruh Yigit <ferruh.yigit@intel.com> wrote:
>
> The problem occurred when workaround that makes soname two digits
> applied. With this change for the ABI version "20.0.1" the experimental
> library version become ".so.2001".
> After workaround removed in ABI version 21.0, the experimental library
> version will become ".so.210".
> "2001" is bigger value than "201" although it is a previous version of
> the library version, this can break the version comparisons.
>
> To fix this, introducing a temporary sub level versioning for the
> experimental libraries, so that minor version comparison will continue
> to work.
>
> After change experimental libraries will follow below versioning:
>
> DPDK version ABI version soname library name
> ------------ ----------- ------ ------------
> DPDK 19.11 20.0 .so.0.200 .so.0.200
> DPDK 20.02 20.0.1 .so.0.200.1 .so.0.200.1
> DPDK 20.05 20.0.2 .so.0.200.2 .so.0.200.2
> DPDK 20.11 21.0 .so.0.210 .so.0.210
> DPDK 21.02 21.1 .so.0.211 .so.0.211
>
> Note: After workaround removed in DPDK 20.11 and soname switch back to
> single digit this patch won't work and needs to be updated.
>
> Fixes: f26c2b39b271 ("build: fix soname info for 19.11 compatibility")
>
> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
Acked-by: Luca Boccassi <bluca@debian.org>
Acked-by: Ray Kinsella <mdr@ashroe.eu>
Applied.
--
David Marchand
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [dpdk-ci] [PATCH 4/4] ci: reorganise Travis jobs
2020-02-20 16:01 0% ` David Marchand
@ 2020-02-20 19:38 0% ` Jeremy Plsek
0 siblings, 0 replies; 200+ results
From: Jeremy Plsek @ 2020-02-20 19:38 UTC (permalink / raw)
To: David Marchand
Cc: ci, Thomas Monjalon, dev, Michael Santana, Aaron Conole,
Bruce Richardson
[-- Attachment #1: Type: text/plain, Size: 3853 bytes --]
On Thu, Feb 20, 2020 at 11:02 AM David Marchand
<david.marchand@redhat.com> wrote:
>
> On Thu, Feb 20, 2020 at 3:35 PM Aaron Conole <aconole@redhat.com> wrote:
> >
> > David Marchand <david.marchand@redhat.com> writes:
> >
> > > On Thu, Feb 20, 2020 at 11:42 AM Thomas Monjalon <thomas@monjalon.net> wrote:
> > >>
> > >> 19/02/2020 22:39, Aaron Conole:
> > >> > David Marchand <david.marchand@redhat.com> writes:
> > >> >
> > >> > > Let's prune the jobs list to limit the amount of time spent by the robot
> > >> > > in Travis.
> > >> > >
> > >> > > Since meson enables automatically the relevant components, there is not
> > >> > > much gain in testing with extra_packages vs required_packages only.
> > >> > >
> > >> > > For a given arch/compiler/env combination, compilation is first tested
> > >> > > in all jobs that run tests or build the docs or run the ABI checks.
> > >> > > In the same context, for jobs that accumulates running tests, building
> > >> > > the docs etc..., those steps are independent and can be split to save
> > >> > > some cpu on Travis.
> > >> > >
> > >> > > With this, we go down from 21 to 15 jobs.
> > >> > >
> > >> > > Note: this patch requires a flush of the existing caches in Travis.
> > >> > >
> > >> > > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > >> > > ---
> > >> >
> > >> > In general, I think the idea with required vs. extra was to have a build
> > >> > that did the minimum required, and one that did all the packages (to
> > >> > allow a minimum vs. full DPDK).
> > >> >
> > >> > At least, that's from
> > >> > http://mails.dpdk.org/archives/dev/2019-January/124007.html
> > >>
> > >> I think the benefit of a minimum build is to have a quick report,
> > >> and easy to setup.
> > >
> > > Yes, Travis serves as a first gate when submitting patches.
> > > But since Travis is best effort/free, we can't have a full coverage.
> > >
> > >
> > >> > Not sure if that's still something anyone cares about.
> > >>
> > >> Given that Travis knows how to satisfy the dependencies,
> > >> and that we must wait for all jobs to finish,
> > >> I don't see any benefit of a minimal setup.
> > >
> > > This minimal setup also tests that dpdk dependencies are correct.
> > > If a change makes something rely on libX and libX is in the packages
> > > always installed in Travis, the missing dependency would not get
> > > caught.
> > >
> > > But here, this adds too many jobs.
> > >
> > > UNH, Intel and other CIs should step in and fill this kind of gap.
> >
> > Okay, makes sense to me. Are one of these CI providers offering to
> > cover this?
>
> Maybe it is already covered, the best is to ask, so sending to ci@dpdk.org.
>
> For the CI guys, which packages are installed on the systems/vms that
> do compilation tests?
For our compile tests, we compile with pretty much all dpdk
dependencies that can easily be installed from the distro's repository
(the dependencies we installed are based on the meson output). The
thought process was to test compiling as much of dpdk as possible
(with constraints mentioned before).
Here are the dependencies we install on some of the setups:
Ubuntu 18.04: make gcc ninja-build pkg-config libnuma-dev
librdmacm-dev librdmacm1 rdma-core libelf-dev libmnl-dev libpcap-dev
libcrypto++-dev libjansson-dev
(meson installed with pip)
CentOS 8 and Fedora 31: diffutils gcc make ninja-build meson
pkg-config numactl-devel librdmacm rdma-core-devel
elfutils-libelf-devel libmnl-devel libpcap-devel jansson-devel
openSuse Leap 15: gcc make ninja pkg-config libnuma-devel librdmacm1
rdma-core-devel libelf-devel libmnl-devel libpcap-devel
libjansson-devel
(meson installed with pip)
> Is it possible to have a summary of the different setups?
>
I've attached an overview/summary of what we do at the UNH-IOL DPDK
Community Lab.
>
> Thanks.
>
> --
> David Marchand
>
[-- Attachment #2: DPDK Lab Summary February 2020.pdf --]
[-- Type: application/pdf, Size: 846111 bytes --]
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] ABI version of experimental libraries
2020-02-19 13:50 4% ` Ray Kinsella
@ 2020-02-21 16:57 4% ` Thomas Monjalon
2020-02-24 9:32 4% ` Ray Kinsella
0 siblings, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-02-21 16:57 UTC (permalink / raw)
To: dev; +Cc: Ray Kinsella
19/02/2020 14:50, Ray Kinsella:
> On 19/02/2020 12:43, Thomas Monjalon wrote:
> > 19/02/2020 12:43, Neil Horman:
> >> On Tue, Feb 18, 2020 at 10:50:09AM +0100, Thomas Monjalon wrote:
> >>> 18/02/2020 10:42, Bruce Richardson:
> >>>> On Tue, Feb 18, 2020 at 12:15:56AM +0100, Thomas Monjalon wrote:
> >>>>> Hi,
> >>>>>
> >>>>> I would like to remind everybody our mistake when defining ABI versions.
> >>>>> It has been "fixed" in this commit:
> >>>>> http://git.dpdk.org/dpdk/commit/?id=f26c2b39
> >>>>>
> >>>>> Please let's think about the consequence for the experimental libraries.
> >>>>>
> >>>>> In DPDK 19.11, we use the ABI version 0.200 with soname 0.20 In DPDK
> >>>>> 20.02, we use the ABI version 0.2001 with soname 0.201 Numbers are
> >>>>> increasing, that's fine. When we'll switch to the new major ABI and use
> >>>>> a normal numbering: In DPDK 20.11, we will use the ABI version 0.210 with
> >>>>> soname 0.21 Numbers are dropping.
> >>>>>
> >>>>> In short, for experimental libs, ABI 20.1 > ABI 21.0
> >>>>>
> >>>>> Are we OK with this or do we prefer reverting to normal numbering for
> >>>>> experimental libraries in DPDK 20.02?
> >>>>>
> >>>> Personally, I would not be too concerned about the verions of experimental
> >>>> libs, so long as they don't conflict across versions and have some
> >>>> similarity to the major ABI version for the release.
> >>>
> >>> You think sorting of the version numbers is not important?
> >>> If we don't care comparing experimental version numbers,
> >>> then OK, let's drop this patch. But please we need a small vote.
> >>>
> >>> Note: there would be no problem if we did not vote for having
> >>> a special numbering for pure experimental libraries (I am still against).
> >>>
> >> I don't understand. Why would we change the ABI_VERSION at all in an LTS release at
> >> all? This operation is meant to take an an experimental API and mark it as
> >> stable by promoting its version number to the next major releases number. As
> >> such, in the LTS release, we should keep the soname the same, as there should be
> >> no other ABI changes in the promoted API.
> >
> > The library version number is updated because we add new symbols.
> >
> >
>
> So while experimental library version numbers are not "important".
> I do agree with Thomas they should be sane, increase and should have a consistent format.
>
> Should we always just pad them to 4 places as the simple solution?
> i.e.
>
> DPDK 19.11 ... 0.20 (needs to remain 0.20).
> DPDK 20.02 ... 0.2001
> DPDK 20.11 ... 0.2100
> DPDK 21.02 ... 0.2101
A patch from Ferruh got merged.
It is adding a dot to keep versioning consistent.
Marking this patch as rejected.
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v4] doc: alias to experimental tag for stable apis
2020-02-18 10:46 17% ` [dpdk-dev] [PATCH v4] " Ray Kinsella
@ 2020-02-21 17:11 0% ` Ferruh Yigit
2020-02-25 14:22 0% ` Kevin Traynor
1 sibling, 0 replies; 200+ results
From: Ferruh Yigit @ 2020-02-21 17:11 UTC (permalink / raw)
To: Ray Kinsella, dev
Cc: john.mcnamara, nhorman, marko.kovacevic, bruce.richardson
On 2/18/2020 10:46 AM, Ray Kinsella wrote:
> When a maintainer is promoting an API to become part of the next major ABI
> version by removing the experimental tag, possibly a few releases in
> advance of the declaration of the next ABI version. The maintainer may
> choose to offer an alias to the experimental tag, as removing the tag
> before the declaration of the next major ABI version, would cause an ABI
> breakage for applications using the API.
>
> Signed-off-by: Ray Kinsella <mdr@ashroe.eu>
> ---
> doc/guides/contributing/abi_policy.rst | 10 ++++++++++
> 1 file changed, 10 insertions(+)
>
> diff --git a/doc/guides/contributing/abi_policy.rst b/doc/guides/contributing/abi_policy.rst
> index 05ca959..7ae7de7 100644
> --- a/doc/guides/contributing/abi_policy.rst
> +++ b/doc/guides/contributing/abi_policy.rst
> @@ -159,6 +159,11 @@ The requirements for changing the ABI are:
> ``experimental``, as described in the section on :ref:`Experimental APIs
> and Libraries <experimental_apis>`.
>
> + - In situations in which an ``experimental`` API has been stable for some
> + time. When promoting the API to become part of the next ABI version, the
> + maintainer may choose to provide an alias to the ``experimental`` tag, so
> + as not to break consuming applications.
> +
> #. If a newly proposed API functionally replaces an existing one, when the new
> API becomes non-experimental, then the old one is marked with
> ``__rte_deprecated``.
> @@ -317,6 +322,11 @@ not required. Though, an API should remain in experimental state for at least
> one release. Thereafter, the normal process of posting patch for review to
> mailing list can be followed.
>
> +After the experimental tag has been formally removed, a tree/sub-tree maintainer
> +may choose to offer an alias to the experimental tag so as not to break
> +applications using the API. The alias is then dropped at the declaration of next
> +major ABI version.
> +
> Libraries
> ~~~~~~~~~
>
>
This leaves the decision to author/maintainer, I am more for making this a
requirement, but at least this enables it and better than nothing ;)
Acked-by: Ferruh Yigit <ferruh.yigit@intel.com>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v3] cmdline: increase maximum line length
@ 2020-02-22 15:28 3% ` David Marchand
0 siblings, 0 replies; 200+ results
From: David Marchand @ 2020-02-22 15:28 UTC (permalink / raw)
To: Wisam Jaddo
Cc: dev, Raslan, Thomas Monjalon, Olivier Matz, Iremonger, Bernard,
dpdk stable
This patch is flagged as an ABI breakage:
https://travis-ci.com/ovsrobot/dpdk/jobs/289313318#L2273
On Thu, Feb 20, 2020 at 3:53 PM Wisam Jaddo <wisamm@mellanox.com> wrote:
>
> This increase due to the usage of cmdline in dpdk applications
> as config commands such as testpmd do for rte_flow rules creation.
>
> The current size of buffer is not enough to fill
> many cases of rte_flow commands validation/creation.
>
> rte_flow now can have outer items, inner items, modify
> actions, meta data actions, duplicate action, fate action and
> more in one single rte flow, thus 512 char will not be enough
> to validate such rte flow rules.
>
> Such change shouldn't affect the memory since the cmdline
> reading again using the same buffer.
I don't get your point here.
> Cc: stable@dpdk.org
This is not a fix.
--
David Marchand
^ permalink raw reply [relevance 3%]
* [dpdk-dev] [PATCH v1] doc: update release notes for 20.02
@ 2020-02-23 10:59 8% John McNamara
0 siblings, 0 replies; 200+ results
From: John McNamara @ 2020-02-23 10:59 UTC (permalink / raw)
To: dev; +Cc: thomas, John McNamara
Fix grammar, spelling and formatting of DPDK 20.02 release notes.
Signed-off-by: John McNamara <john.mcnamara@intel.com>
---
doc/guides/rel_notes/release_20_02.rst | 70 +++++++++++++++-------------------
1 file changed, 31 insertions(+), 39 deletions(-)
diff --git a/doc/guides/rel_notes/release_20_02.rst b/doc/guides/rel_notes/release_20_02.rst
index dfebc46..8a493e4 100644
--- a/doc/guides/rel_notes/release_20_02.rst
+++ b/doc/guides/rel_notes/release_20_02.rst
@@ -74,7 +74,7 @@ New Features
Added the new ``ionic`` net driver for Pensando Ethernet Network Adapters.
See the :doc:`../nics/ionic` NIC guide for more details on this new driver.
-* **Updated Broadcom bnxt driver**
+* **Updated Broadcom bnxt driver.**
Updated Broadcom bnxt driver with new features and improvements, including:
@@ -91,10 +91,10 @@ New Features
Updated the Intel ice driver with new features and improvements, including:
- * Added support for MAC rules on specific port.
+ * Added support for MAC rules on a specific port.
* Added support for MAC/VLAN with TCP/UDP in switch rule.
* Added support for 1/10G device.
- * Added support for API rte_eth_tx_done_cleanup.
+ * Added support for API ``rte_eth_tx_done_cleanup``.
* **Updated Intel iavf driver.**
@@ -105,7 +105,9 @@ New Features
* **Updated the Intel ixgbe driver.**
- * Added support for API rte_eth_tx_done_cleanup.
+ Updated ixgbe PMD with new features and improvements, including:
+
+ * Added support for API ``rte_eth_tx_done_cleanup()``.
* Added support setting VF MAC address by PF driver.
* Added support for setting the link to specific speed.
@@ -122,7 +124,7 @@ New Features
* Extended PHY access AQ cmd.
* Added support for reading LPI counters.
* Added support for Energy Efficient Ethernet.
- * Added support for API rte_eth_tx_done_cleanup.
+ * Added support for API ``rte_eth_tx_done_cleanup()``.
* Added support for VF multiple queues interrupt.
* Added support for setting the link to specific speed.
@@ -135,7 +137,7 @@ New Features
* Removed limitation of matching on tagged/untagged packets (when using DV flow engine).
* Added BlueField-2 integrated ConnectX-6 Dx device support.
-* **Add new vDPA PMD based on Mellanox devices**
+* **Add new vDPA PMD based on Mellanox devices.**
Added a new Mellanox vDPA (``mlx5_vdpa``) PMD.
See the :doc:`../vdpadevs/mlx5` guide for more details on this driver.
@@ -155,6 +157,8 @@ New Features
* **Added algorithms to cryptodev API.**
+ Added new algorithms to the cryptodev API:
+
* ECDSA (Elliptic Curve Digital Signature Algorithm) is added to
asymmetric crypto library specifications.
* ECPM (Elliptic Curve Point Multiplication) is added to
@@ -162,8 +166,8 @@ New Features
* **Added synchronous Crypto burst API.**
- A new API is introduced in crypto library to handle synchronous cryptographic
- operations allowing to achieve performance gain for cryptodevs which use
+ A new API has been introduced in the crypto library to handle synchronous cryptographic
+ operations allowing it to achieve performance gains for cryptodevs which use
CPU based acceleration, such as Intel AES-NI. An implementation for aesni_gcm
cryptodev is provided. The IPsec example application and ipsec library itself
were changed to allow utilization of this new feature.
@@ -174,7 +178,7 @@ New Features
(generation) and cipher-hash (verification) requests (e.g. SNOW3G + ZUC or
ZUC + AES CTR) in QAT PMD possible when running on GEN3 QAT hardware.
Such algorithm combinations are not supported on GEN1/GEN2 hardware
- and executing the request returns RTE_CRYPTO_OP_STATUS_INVALID_SESSION.
+ and executing the request returns ``RTE_CRYPTO_OP_STATUS_INVALID_SESSION``.
* **Queue-pairs are now thread-safe on Intel QuickAssist Technology (QAT) PMD.**
@@ -185,34 +189,35 @@ New Features
* **Updated the ZUC PMD.**
- * Transistioned underlying library from libSSO ZUC to intel-ipsec-mb
+ * Transitioned underlying library from libSSO ZUC to intel-ipsec-mb
library (minimum version required 0.53).
* Removed dynamic library limitation, so PMD can be built as a shared
object now.
* **Updated the KASUMI PMD.**
- * Transistioned underlying library from libSSO KASUMI to intel-ipsec-mb
+ * Transitioned underlying library from libSSO KASUMI to intel-ipsec-mb
library (minimum version required 0.53).
* **Updated the SNOW3G PMD.**
- * Transistioned underlying library from libSSO SNOW3G to intel-ipsec-mb
+ * Transitioned underlying library from libSSO SNOW3G to intel-ipsec-mb
library (minimum version required 0.53).
* **Changed armv8 crypto PMD external dependency.**
- armv8 crypto PMD now depends on Arm crypto library, and Marvell's
- armv8 crypto library is not used anymore. Library name is changed
+ Changed armv8 crypto PMD external dependency. The
+ armv8 crypto PMD now depends on the Arm crypto library, and Marvell's
+ armv8 crypto library is not used anymore. The library name has been changed
from armv8_crypto to AArch64crypto.
* **Added inline IPsec support to Marvell OCTEON TX2 PMD.**
- Added inline IPsec support to Marvell OCTEON TX2 PMD. With the feature,
- applications would be able to offload entire IPsec offload to the hardware.
+ Added inline IPsec support to Marvell OCTEON TX2 PMD. With this feature,
+ applications will be able to offload entire IPsec offload to the hardware.
For the configured sessions, hardware will do the lookup and perform
- decryption and IPsec transformation. For the outbound path, application
- can submit a plain packet to the PMD, and it would be sent out on wire
+ decryption and IPsec transformation. For the outbound path, applications
+ can submit a plain packet to the PMD, and it will be sent out on the wire
after doing encryption and IPsec transformation of the packet.
* **Added Marvell OCTEON TX2 End Point rawdev PMD.**
@@ -222,7 +227,7 @@ New Features
* **Added event mode to l3fwd sample application.**
- Add event device support for ``l3fwd`` sample application. It demonstrates
+ Added event device support for the ``l3fwd`` sample application. It demonstrates
usage of poll and event mode IO mechanism under a single application.
* **Added cycle-count mode to the compression performance tool.**
@@ -232,8 +237,8 @@ New Features
* **Added OpenWrt howto guide.**
- Added document describes how to enable DPDK on OpenWrt in both virtual and
- physical machine.
+ Added document which describes how to enable DPDK on OpenWrt in both virtual and
+ physical machines.
Removed Items
@@ -250,7 +255,7 @@ Removed Items
* **Disabled building all the Linux kernel modules by default.**
- In order to remove the build time dependency with Linux kernel,
+ In order to remove the build time dependency on the Linux kernel,
the Technical Board decided to disable all the kernel modules
by default from 20.02 version.
@@ -259,8 +264,8 @@ Removed Items
The internal tail write coalescing feature was removed as not compatible with
dual-thread feature. It was replaced with a threshold feature. At busy times
if only a small number of packets can be enqueued, each enqueue causes
- an expensive MMIO write. These MMIO write occurrences can be optimised by using
- the new threshold parameter on process start. Please see qat documentation for
+ an expensive MMIO write. These MMIO write occurrences can be optimized by using
+ the new threshold parameter on process start. Please see QAT documentation for
more details.
@@ -279,6 +284,8 @@ API Changes
Also, make sure to start the actual text at the margin.
=========================================================
+* No changes in this release.
+
ABI Changes
-----------
@@ -298,21 +305,6 @@ ABI Changes
* No change, kept ABI v20. DPDK 20.02 is compatible with DPDK 19.11.
-Known Issues
-------------
-
-.. This section should contain new known issues in this release. Sample format:
-
- * **Add title in present tense with full stop.**
-
- Add a short 1-2 sentence description of the known issue
- in the present tense. Add information on any known workarounds.
-
- This section is a comment. Do not overwrite or remove it.
- Also, make sure to start the actual text at the margin.
- =========================================================
-
-
Tested Platforms
----------------
--
2.7.5
^ permalink raw reply [relevance 8%]
* Re: [dpdk-dev] [PATCH] doc: deprecate using MAX values as array size
2020-02-21 10:25 0% ` Ferruh Yigit
@ 2020-02-24 6:18 0% ` Akhil Goyal
2020-02-25 11:03 0% ` Thomas Monjalon
2020-02-24 8:51 0% ` Andrew Rybchenko
1 sibling, 1 reply; 200+ results
From: Akhil Goyal @ 2020-02-24 6:18 UTC (permalink / raw)
To: Ferruh Yigit, Neil Horman, John McNamara, Marko Kovacevic
Cc: dev, Thomas Monjalon, Bruce Richardson, Konstantin Ananyev,
David Marchand, Jerin Jacob, Andrew Rybchenko, Ori Kam
>
> On 1/30/2020 2:20 PM, Ferruh Yigit wrote:
> > Adding the deprecation notice as reminder for next ABI breakage release
> > (20.11).
> > This one time breakage is required to be able to extend enum/define
> > without breaking ABI.
> >
> > Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
> > ---
> > doc/guides/rel_notes/deprecation.rst | 14 ++++++++++++++
> > 1 file changed, 14 insertions(+)
> >
> > diff --git a/doc/guides/rel_notes/deprecation.rst
> b/doc/guides/rel_notes/deprecation.rst
> > index dfcca87ab..99d81564a 100644
> > --- a/doc/guides/rel_notes/deprecation.rst
> > +++ b/doc/guides/rel_notes/deprecation.rst
> > @@ -38,6 +38,20 @@ Deprecation Notices
> > remove it from the externally visible ABI and allow it to be updated in the
> > future.
> >
> > +* lib: will fix extending some enum/define breaking the ABI. There are
> multiple
> > + samples in DPDK that enum/define terminated with a ``.*MAX.*`` value
> which is
> > + used by iterators, and arrays holding these values are sized with this
> > + ``.*MAX.*`` value. So extending this enum/define increases the ``.*MAX.*``
> > + value which increases the size of the array and depending on how/where the
> > + array is used this may break the ABI.
> > + ``RTE_ETH_FLOW_MAX`` is one sample of the mentioned case, adding a
> new flow
> > + type will break the ABI because of ``flex_mask[RTE_ETH_FLOW_MAX]``
> array
> > + usage in following public struct hierarchy:
> > + ``rte_eth_fdir_flex_conf -> rte_fdir_conf -> rte_eth_conf (in the middle)``.
> > + Need to identify this kind of usages and fix in 20.11, otherwise this blocks
> > + us extending existing enum/define.
> > + One solution can be using a fixed size array instead of ``.*MAX.*`` value.
> > +
> > * dpaa2: removal of ``rte_dpaa2_memsegs`` structure which has been
> replaced
> > by a pa-va search library. This structure was earlier being used for holding
> > memory segments used by dpaa2 driver for faster pa->va translation. This
> >
>
We admit that the issue is there and we can discuss the possible solutions for this issue.
For the deprecation notice.
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [dpdk-techboard] [PATCH] doc: deprecation notice to move igb_uio to a new repo
@ 2020-02-24 7:03 0% ` Maxime Coquelin
0 siblings, 0 replies; 200+ results
From: Maxime Coquelin @ 2020-02-24 7:03 UTC (permalink / raw)
To: jerinj, dev, Neil Horman, John McNamara, Marko Kovacevic
Cc: thomas, techboard, ferruh.yigit
On 1/30/20 5:58 PM, jerinj@marvell.com wrote:
> From: Jerin Jacob <jerinj@marvell.com>
>
> Based on the tech board meeting held on 2019-11-06,
> It's been decided to move igb_uio kernel module to a new repository
> hosted by dpdk.org in v20.11 release.
>
> http://mails.dpdk.org/archives/dev/2019-November/151763.html
>
> Adding the deprecation notice for the same in advance.
>
> Signed-off-by: Jerin Jacob <jerinj@marvell.com>
> ---
> doc/guides/rel_notes/deprecation.rst | 6 ++++++
> 1 file changed, 6 insertions(+)
>
> diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
> index dfcca87ab..2b9d0113f 100644
> --- a/doc/guides/rel_notes/deprecation.rst
> +++ b/doc/guides/rel_notes/deprecation.rst
> @@ -38,6 +38,12 @@ Deprecation Notices
> remove it from the externally visible ABI and allow it to be updated in the
> future.
>
> +* igb_uio: In the view of reducing the kernel dependency from the main tree,
> + as a first step, The technical board has been decided to move ``igb_uio``
> + kernel module to a new repository hosted by dpdk.org in v20.11 release.
> + Minutes of Technical Board Meeting of `2019-11-06
> + <http://mails.dpdk.org/archives/dev/2019-November/151763.html>`_.
> +
> * dpaa2: removal of ``rte_dpaa2_memsegs`` structure which has been replaced
> by a pa-va search library. This structure was earlier being used for holding
> memory segments used by dpaa2 driver for faster pa->va translation. This
>
Acked-by: Maxime Coquelin <maxime.coquelin@redhat.com>
Thanks,
Maxime
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] doc: deprecate using MAX values as array size
2020-02-21 10:25 0% ` Ferruh Yigit
2020-02-24 6:18 0% ` Akhil Goyal
@ 2020-02-24 8:51 0% ` Andrew Rybchenko
1 sibling, 0 replies; 200+ results
From: Andrew Rybchenko @ 2020-02-24 8:51 UTC (permalink / raw)
To: Ferruh Yigit, Neil Horman, John McNamara, Marko Kovacevic
Cc: dev, Thomas Monjalon, Bruce Richardson, Akhil Goyal,
Konstantin Ananyev, David Marchand, Jerin Jacob, Ori Kam
On 2/21/20 1:25 PM, Ferruh Yigit wrote:
> On 1/30/2020 2:20 PM, Ferruh Yigit wrote:
>> Adding the deprecation notice as reminder for next ABI breakage release
>> (20.11).
>> This one time breakage is required to be able to extend enum/define
>> without breaking ABI.
>>
>> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
>> ---
>> doc/guides/rel_notes/deprecation.rst | 14 ++++++++++++++
>> 1 file changed, 14 insertions(+)
>>
>> diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
>> index dfcca87ab..99d81564a 100644
>> --- a/doc/guides/rel_notes/deprecation.rst
>> +++ b/doc/guides/rel_notes/deprecation.rst
>> @@ -38,6 +38,20 @@ Deprecation Notices
>> remove it from the externally visible ABI and allow it to be updated in the
>> future.
>>
>> +* lib: will fix extending some enum/define breaking the ABI. There are multiple
>> + samples in DPDK that enum/define terminated with a ``.*MAX.*`` value which is
>> + used by iterators, and arrays holding these values are sized with this
>> + ``.*MAX.*`` value. So extending this enum/define increases the ``.*MAX.*``
>> + value which increases the size of the array and depending on how/where the
>> + array is used this may break the ABI.
>> + ``RTE_ETH_FLOW_MAX`` is one sample of the mentioned case, adding a new flow
>> + type will break the ABI because of ``flex_mask[RTE_ETH_FLOW_MAX]`` array
>> + usage in following public struct hierarchy:
>> + ``rte_eth_fdir_flex_conf -> rte_fdir_conf -> rte_eth_conf (in the middle)``.
>> + Need to identify this kind of usages and fix in 20.11, otherwise this blocks
>> + us extending existing enum/define.
>> + One solution can be using a fixed size array instead of ``.*MAX.*`` value.
>> +
>> * dpaa2: removal of ``rte_dpaa2_memsegs`` structure which has been replaced
>> by a pa-va search library. This structure was earlier being used for holding
>> memory segments used by dpaa2 driver for faster pa->va translation. This
>>
> Reminder of this deprecation notice.
Acked-by: Andrew Rybchenko <arybchenko@oktetlabs.ru>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] ABI version of experimental libraries
2020-02-21 16:57 4% ` Thomas Monjalon
@ 2020-02-24 9:32 4% ` Ray Kinsella
0 siblings, 0 replies; 200+ results
From: Ray Kinsella @ 2020-02-24 9:32 UTC (permalink / raw)
To: Thomas Monjalon, dev
On 21/02/2020 16:57, Thomas Monjalon wrote:
> 19/02/2020 14:50, Ray Kinsella:
>> On 19/02/2020 12:43, Thomas Monjalon wrote:
>>> 19/02/2020 12:43, Neil Horman:
>>>> On Tue, Feb 18, 2020 at 10:50:09AM +0100, Thomas Monjalon wrote:
>>>>> 18/02/2020 10:42, Bruce Richardson:
>>>>>> On Tue, Feb 18, 2020 at 12:15:56AM +0100, Thomas Monjalon wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> I would like to remind everybody our mistake when defining ABI versions.
>>>>>>> It has been "fixed" in this commit:
>>>>>>> http://git.dpdk.org/dpdk/commit/?id=f26c2b39
>>>>>>>
>>>>>>> Please let's think about the consequence for the experimental libraries.
>>>>>>>
>>>>>>> In DPDK 19.11, we use the ABI version 0.200 with soname 0.20 In DPDK
>>>>>>> 20.02, we use the ABI version 0.2001 with soname 0.201 Numbers are
>>>>>>> increasing, that's fine. When we'll switch to the new major ABI and use
>>>>>>> a normal numbering: In DPDK 20.11, we will use the ABI version 0.210 with
>>>>>>> soname 0.21 Numbers are dropping.
>>>>>>>
>>>>>>> In short, for experimental libs, ABI 20.1 > ABI 21.0
>>>>>>>
>>>>>>> Are we OK with this or do we prefer reverting to normal numbering for
>>>>>>> experimental libraries in DPDK 20.02?
>>>>>>>
>>>>>> Personally, I would not be too concerned about the verions of experimental
>>>>>> libs, so long as they don't conflict across versions and have some
>>>>>> similarity to the major ABI version for the release.
>>>>>
>>>>> You think sorting of the version numbers is not important?
>>>>> If we don't care comparing experimental version numbers,
>>>>> then OK, let's drop this patch. But please we need a small vote.
>>>>>
>>>>> Note: there would be no problem if we did not vote for having
>>>>> a special numbering for pure experimental libraries (I am still against).
>>>>>
>>>> I don't understand. Why would we change the ABI_VERSION at all in an LTS release at
>>>> all? This operation is meant to take an an experimental API and mark it as
>>>> stable by promoting its version number to the next major releases number. As
>>>> such, in the LTS release, we should keep the soname the same, as there should be
>>>> no other ABI changes in the promoted API.
>>>
>>> The library version number is updated because we add new symbols.
>>>
>>>
>>
>> So while experimental library version numbers are not "important".
>> I do agree with Thomas they should be sane, increase and should have a consistent format.
>>
>> Should we always just pad them to 4 places as the simple solution?
>> i.e.
>>
>> DPDK 19.11 ... 0.20 (needs to remain 0.20).
>> DPDK 20.02 ... 0.2001
>> DPDK 20.11 ... 0.2100
>> DPDK 21.02 ... 0.2101
>
> A patch from Ferruh got merged.
> It is adding a dot to keep versioning consistent.
>
> Marking this patch as rejected.
>
Ferruh's was a better solution.
^ permalink raw reply [relevance 4%]
* [dpdk-dev] [RFC 0/6] New sync modes for ring
@ 2020-02-24 11:35 2% Konstantin Ananyev
2020-02-24 16:59 0% ` Stephen Hemminger
2020-03-25 20:43 0% ` Honnappa Nagarahalli
0 siblings, 2 replies; 200+ results
From: Konstantin Ananyev @ 2020-02-24 11:35 UTC (permalink / raw)
To: dev; +Cc: olivier.matz, Konstantin Ananyev
Upfront note - that RFC is not a complete patch.
It introduces an ABI breakage, plus it doesn't update ring_elem
code properly, etc.
I plan to deal with all these things in later versions.
Right now I seek an initial feedback about proposed ideas.
Would also ask people to repeat performance tests (see below)
on their platforms to confirm the impact.
More and more customers use(/try to use) DPDK based apps within
overcommitted systems (multiple acttive threads over same pysical cores):
VM, container deployments, etc.
One quite common problem they hit: Lock-Holder-Preemption with rte_ring.
LHP is quite a common problem for spin-based sync primitives
(spin-locks, etc.) on overcommitted systems.
The situation gets much worse when some sort of
fair-locking technique is used (ticket-lock, etc.).
As now not only lock-owner but also lock-waiters scheduling
order matters a lot.
This is a well-known problem for kernel within VMs:
http://www-archive.xenproject.org/files/xensummitboston08/LHP.pdf
https://www.cs.hs-rm.de/~kaiser/events/wamos2017/Slides/selcuk.pdf
The problem with rte_ring is that while head accusion is sort of
un-fair locking, waiting on tail is very similar to ticket lock schema -
tail has to be updated in particular order.
That makes current rte_ring implementation to perform
really pure on some overcommited scenarios.
While it is probably not possible to completely resolve this problem in
userspace only (without some kernel communication/intervention),
removing fairness in tail update can mitigate it significantly.
So this RFC proposes two new optional ring synchronization modes:
1) Head/Tail Sync (HTS) mode
In that mode enqueue/dequeue operation is fully serialized:
only one thread at a time is allowed to perform given op.
As another enhancement provide ability to split enqueue/dequeue
operation into two phases:
- enqueue/dequeue start
- enqueue/dequeue finish
That allows user to inspect objects in the ring without removing
them from it (aka MT safe peek).
2) Relaxed Tail Sync (RTS)
The main difference from original MP/MC algorithm is that
tail value is increased not by every thread that finished enqueue/dequeue,
but only by the last one.
That allows threads to avoid spinning on ring tail value,
leaving actual tail value change to the last thread in the update queue.
Test results on IA (see below) show significant improvements
for average enqueue/dequeue op times on overcommitted systems.
For 'classic' DPDK deployments (one thread per core) original MP/MC
algorithm still shows best numbers, though for 64-bit target
RTS numbers are not that far away.
Numbers were produced by ring_stress_*autotest
(first patch in these series).
X86_64 @ Intel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz
DEQ+ENQ average cycles/obj
MP/MC HTS RTS
1thread@1core(--lcores=6-7) 8.00 8.15 8.99
2thread@2core(--lcores=6-8) 19.14 19.61 20.35
4thread@4core(--lcores=6-10) 29.43 29.79 31.82
8thread@8core(--lcores=6-14) 110.59 192.81 119.50
16thread@16core(--lcores=6-22) 461.03 813.12 495.59
32thread/@32core(--lcores='6-22,55-70') 982.90 1972.38 1160.51
2thread@1core(--lcores='6,(10-11)@7' 20140.50 23.58 25.14
4thread@2core(--lcores='6,(10-11)@7,(20-21)@8' 153680.60 76.88 80.05
8thread@2core(--lcores='6,(10-13)@7,(20-23)@8' 280314.32 294.72 318.79
16thread@2core(--lcores='6,(10-17)@7,(20-27)@8' 643176.59 1144.02 1175.14
32thread@2core(--lcores='6,(10-25)@7,(30-45)@8' 4264238.80 4627.48 4892.68
8thread@2core(--lcores='6,(10-17)@(7,8))' 321085.98 298.59 307.47
16thread@4core(--lcores='6,(20-35)@(7-10))' 1900705.61 575.35 678.29
32thread@4core(--lcores='6,(20-51)@(7-10))' 5510445.85 2164.36 2714.12
i686 @ Intel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz
DEQ+ENQ average cycles/obj
MP/MC HTS RTS
1thread@1core(--lcores=6-7) 7.85 12.13 11.31
2thread@2core(--lcores=6-8) 17.89 24.52 21.86
8thread@8core(--lcores=6-14) 32.58 354.20 54.58
32thread/@32core(--lcores='6-22,55-70') 813.77 6072.41 2169.91
2thread@1core(--lcores='6,(10-11)@7' 16095.00 36.06 34.74
8thread@2core(--lcores='6,(10-13)@7,(20-23)@8' 1140354.54 346.61 361.57
16thread@2core(--lcores='6,(10-17)@7,(20-27)@8' 1920417.86 1314.90 1416.65
8thread@2core(--lcores='6,(10-17)@(7,8))' 594358.61 332.70 357.74
32thread@4core(--lcores='6,(20-51)@(7-10))' 5319896.86 2836.44 3028.87
Konstantin Ananyev (6):
test/ring: add contention stress test
ring: rework ring layout to allow new sync schemes
ring: introduce RTS ring mode
test/ring: add contention stress test for RTS ring
ring: introduce HTS ring mode
test/ring: add contention stress test for HTS ring
app/test/Makefile | 3 +
app/test/meson.build | 3 +
app/test/test_pdump.c | 6 +-
app/test/test_ring_hts_stress.c | 28 ++
app/test/test_ring_rts_stress.c | 28 ++
app/test/test_ring_stress.c | 27 ++
app/test/test_ring_stress.h | 477 +++++++++++++++++++
lib/librte_pdump/rte_pdump.c | 2 +-
lib/librte_port/rte_port_ring.c | 12 +-
lib/librte_ring/Makefile | 4 +-
lib/librte_ring/meson.build | 4 +-
lib/librte_ring/rte_ring.c | 84 +++-
lib/librte_ring/rte_ring.h | 619 +++++++++++++++++++++++--
lib/librte_ring/rte_ring_elem.h | 8 +-
lib/librte_ring/rte_ring_hts_generic.h | 228 +++++++++
lib/librte_ring/rte_ring_rts_generic.h | 240 ++++++++++
16 files changed, 1721 insertions(+), 52 deletions(-)
create mode 100644 app/test/test_ring_hts_stress.c
create mode 100644 app/test/test_ring_rts_stress.c
create mode 100644 app/test/test_ring_stress.c
create mode 100644 app/test/test_ring_stress.h
create mode 100644 lib/librte_ring/rte_ring_hts_generic.h
create mode 100644 lib/librte_ring/rte_ring_rts_generic.h
--
2.17.1
^ permalink raw reply [relevance 2%]
* Re: [dpdk-dev] [RFC 0/6] New sync modes for ring
2020-02-24 11:35 2% [dpdk-dev] [RFC 0/6] New sync modes for ring Konstantin Ananyev
@ 2020-02-24 16:59 0% ` Stephen Hemminger
2020-02-24 17:59 0% ` Jerin Jacob
2020-03-25 20:43 0% ` Honnappa Nagarahalli
1 sibling, 1 reply; 200+ results
From: Stephen Hemminger @ 2020-02-24 16:59 UTC (permalink / raw)
To: Konstantin Ananyev; +Cc: dev, olivier.matz
On Mon, 24 Feb 2020 11:35:09 +0000
Konstantin Ananyev <konstantin.ananyev@intel.com> wrote:
> Upfront note - that RFC is not a complete patch.
> It introduces an ABI breakage, plus it doesn't update ring_elem
> code properly, etc.
> I plan to deal with all these things in later versions.
> Right now I seek an initial feedback about proposed ideas.
> Would also ask people to repeat performance tests (see below)
> on their platforms to confirm the impact.
>
> More and more customers use(/try to use) DPDK based apps within
> overcommitted systems (multiple acttive threads over same pysical cores):
> VM, container deployments, etc.
> One quite common problem they hit: Lock-Holder-Preemption with rte_ring.
> LHP is quite a common problem for spin-based sync primitives
> (spin-locks, etc.) on overcommitted systems.
> The situation gets much worse when some sort of
> fair-locking technique is used (ticket-lock, etc.).
> As now not only lock-owner but also lock-waiters scheduling
> order matters a lot.
> This is a well-known problem for kernel within VMs:
> http://www-archive.xenproject.org/files/xensummitboston08/LHP.pdf
> https://www.cs.hs-rm.de/~kaiser/events/wamos2017/Slides/selcuk.pdf
> The problem with rte_ring is that while head accusion is sort of
> un-fair locking, waiting on tail is very similar to ticket lock schema -
> tail has to be updated in particular order.
> That makes current rte_ring implementation to perform
> really pure on some overcommited scenarios.
Rather than reform rte_ring to fit this scenario, it would make
more sense to me to introduce another primitive. The current lockless
ring performs very well for the isolated thread model that DPDK
was built around. This looks like a case of customers violating
the usage model of the DPDK and then being surprised at the fallout.
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [RFC 0/6] New sync modes for ring
2020-02-24 16:59 0% ` Stephen Hemminger
@ 2020-02-24 17:59 0% ` Jerin Jacob
2020-02-24 19:35 3% ` Stephen Hemminger
2020-02-25 0:58 0% ` Honnappa Nagarahalli
0 siblings, 2 replies; 200+ results
From: Jerin Jacob @ 2020-02-24 17:59 UTC (permalink / raw)
To: Stephen Hemminger; +Cc: Konstantin Ananyev, dpdk-dev, Olivier Matz
On Mon, Feb 24, 2020 at 10:29 PM Stephen Hemminger
<stephen@networkplumber.org> wrote:
>
> On Mon, 24 Feb 2020 11:35:09 +0000
> Konstantin Ananyev <konstantin.ananyev@intel.com> wrote:
>
> > Upfront note - that RFC is not a complete patch.
> > It introduces an ABI breakage, plus it doesn't update ring_elem
> > code properly, etc.
> > I plan to deal with all these things in later versions.
> > Right now I seek an initial feedback about proposed ideas.
> > Would also ask people to repeat performance tests (see below)
> > on their platforms to confirm the impact.
> >
> > More and more customers use(/try to use) DPDK based apps within
> > overcommitted systems (multiple acttive threads over same pysical cores):
> > VM, container deployments, etc.
> > One quite common problem they hit: Lock-Holder-Preemption with rte_ring.
> > LHP is quite a common problem for spin-based sync primitives
> > (spin-locks, etc.) on overcommitted systems.
> > The situation gets much worse when some sort of
> > fair-locking technique is used (ticket-lock, etc.).
> > As now not only lock-owner but also lock-waiters scheduling
> > order matters a lot.
> > This is a well-known problem for kernel within VMs:
> > http://www-archive.xenproject.org/files/xensummitboston08/LHP.pdf
> > https://www.cs.hs-rm.de/~kaiser/events/wamos2017/Slides/selcuk.pdf
> > The problem with rte_ring is that while head accusion is sort of
> > un-fair locking, waiting on tail is very similar to ticket lock schema -
> > tail has to be updated in particular order.
> > That makes current rte_ring implementation to perform
> > really pure on some overcommited scenarios.
>
> Rather than reform rte_ring to fit this scenario, it would make
> more sense to me to introduce another primitive. The current lockless
> ring performs very well for the isolated thread model that DPDK
> was built around. This looks like a case of customers violating
> the usage model of the DPDK and then being surprised at the fallout.
I agree with Stephen here.
I think, adding more runtime check in the enqueue() and dequeue() will
have a bad effect on the low-end cores too.
But I agree with the problem statement that in the virtualization use
case, It may be possible to have N virtual cores runs on a physical
core.
IMO, The best solution would be keeping the ring API same and have a
different flavor in "compile-time". Something like
liburcu did for accommodating different flavors.
i.e urcu-qsbr.h and urcu-bp.h will identical definition of API. The
application can simply include ONE header file in a C file based on
the flavor.
If need both at runtime. Need to have function pointer or so in the
application and define the function in different c file by including
the approaite flavor in C file.
#include <urcu-qsbr.h> /* QSBR RCU flavor */
#include <urcu-bp.h> /* Bulletproof RCU flavor */
>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [RFC 0/6] New sync modes for ring
2020-02-24 17:59 0% ` Jerin Jacob
@ 2020-02-24 19:35 3% ` Stephen Hemminger
2020-02-24 20:52 0% ` Honnappa Nagarahalli
2020-02-25 13:41 3% ` Ananyev, Konstantin
2020-02-25 0:58 0% ` Honnappa Nagarahalli
1 sibling, 2 replies; 200+ results
From: Stephen Hemminger @ 2020-02-24 19:35 UTC (permalink / raw)
To: Jerin Jacob; +Cc: Konstantin Ananyev, dpdk-dev, Olivier Matz
On Mon, 24 Feb 2020 23:29:57 +0530
Jerin Jacob <jerinjacobk@gmail.com> wrote:
> On Mon, Feb 24, 2020 at 10:29 PM Stephen Hemminger
> <stephen@networkplumber.org> wrote:
> >
> > On Mon, 24 Feb 2020 11:35:09 +0000
> > Konstantin Ananyev <konstantin.ananyev@intel.com> wrote:
> >
> > > Upfront note - that RFC is not a complete patch.
> > > It introduces an ABI breakage, plus it doesn't update ring_elem
> > > code properly, etc.
> > > I plan to deal with all these things in later versions.
> > > Right now I seek an initial feedback about proposed ideas.
> > > Would also ask people to repeat performance tests (see below)
> > > on their platforms to confirm the impact.
> > >
> > > More and more customers use(/try to use) DPDK based apps within
> > > overcommitted systems (multiple acttive threads over same pysical cores):
> > > VM, container deployments, etc.
> > > One quite common problem they hit: Lock-Holder-Preemption with rte_ring.
> > > LHP is quite a common problem for spin-based sync primitives
> > > (spin-locks, etc.) on overcommitted systems.
> > > The situation gets much worse when some sort of
> > > fair-locking technique is used (ticket-lock, etc.).
> > > As now not only lock-owner but also lock-waiters scheduling
> > > order matters a lot.
> > > This is a well-known problem for kernel within VMs:
> > > http://www-archive.xenproject.org/files/xensummitboston08/LHP.pdf
> > > https://www.cs.hs-rm.de/~kaiser/events/wamos2017/Slides/selcuk.pdf
> > > The problem with rte_ring is that while head accusion is sort of
> > > un-fair locking, waiting on tail is very similar to ticket lock schema -
> > > tail has to be updated in particular order.
> > > That makes current rte_ring implementation to perform
> > > really pure on some overcommited scenarios.
> >
> > Rather than reform rte_ring to fit this scenario, it would make
> > more sense to me to introduce another primitive. The current lockless
> > ring performs very well for the isolated thread model that DPDK
> > was built around. This looks like a case of customers violating
> > the usage model of the DPDK and then being surprised at the fallout.
>
> I agree with Stephen here.
>
> I think, adding more runtime check in the enqueue() and dequeue() will
> have a bad effect on the low-end cores too.
> But I agree with the problem statement that in the virtualization use
> case, It may be possible to have N virtual cores runs on a physical
> core.
>
> IMO, The best solution would be keeping the ring API same and have a
> different flavor in "compile-time". Something like
> liburcu did for accommodating different flavors.
>
> i.e urcu-qsbr.h and urcu-bp.h will identical definition of API. The
> application can simply include ONE header file in a C file based on
> the flavor.
> If need both at runtime. Need to have function pointer or so in the
> application and define the function in different c file by including
> the approaite flavor in C file.
This would also be a good time to consider the tradeoffs of the
heavy use of inlining that is done in rte_ring vs the impact that
has on API/ABI stability.
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [RFC 0/6] New sync modes for ring
2020-02-24 19:35 3% ` Stephen Hemminger
@ 2020-02-24 20:52 0% ` Honnappa Nagarahalli
2020-02-25 11:45 0% ` Ananyev, Konstantin
2020-02-25 13:41 3% ` Ananyev, Konstantin
1 sibling, 1 reply; 200+ results
From: Honnappa Nagarahalli @ 2020-02-24 20:52 UTC (permalink / raw)
To: Stephen Hemminger, Jerin Jacob
Cc: Konstantin Ananyev, dpdk-dev, Olivier Matz, Honnappa Nagarahalli, nd, nd
> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Stephen Hemminger
> Sent: Monday, February 24, 2020 1:35 PM
> To: Jerin Jacob <jerinjacobk@gmail.com>
> Cc: Konstantin Ananyev <konstantin.ananyev@intel.com>; dpdk-dev
> <dev@dpdk.org>; Olivier Matz <olivier.matz@6wind.com>
> Subject: Re: [dpdk-dev] [RFC 0/6] New sync modes for ring
>
> On Mon, 24 Feb 2020 23:29:57 +0530
> Jerin Jacob <jerinjacobk@gmail.com> wrote:
>
> > On Mon, Feb 24, 2020 at 10:29 PM Stephen Hemminger
> > <stephen@networkplumber.org> wrote:
> > >
> > > On Mon, 24 Feb 2020 11:35:09 +0000
> > > Konstantin Ananyev <konstantin.ananyev@intel.com> wrote:
> > >
> > > > Upfront note - that RFC is not a complete patch.
> > > > It introduces an ABI breakage, plus it doesn't update ring_elem
> > > > code properly, etc.
> > > > I plan to deal with all these things in later versions.
> > > > Right now I seek an initial feedback about proposed ideas.
> > > > Would also ask people to repeat performance tests (see below) on
> > > > their platforms to confirm the impact.
> > > >
> > > > More and more customers use(/try to use) DPDK based apps within
> > > > overcommitted systems (multiple acttive threads over same pysical
> cores):
> > > > VM, container deployments, etc.
> > > > One quite common problem they hit: Lock-Holder-Preemption with
> rte_ring.
> > > > LHP is quite a common problem for spin-based sync primitives
> > > > (spin-locks, etc.) on overcommitted systems.
> > > > The situation gets much worse when some sort of fair-locking
> > > > technique is used (ticket-lock, etc.).
> > > > As now not only lock-owner but also lock-waiters scheduling order
> > > > matters a lot.
> > > > This is a well-known problem for kernel within VMs:
> > > > http://www-archive.xenproject.org/files/xensummitboston08/LHP.pdf
> > > > https://www.cs.hs-rm.de/~kaiser/events/wamos2017/Slides/selcuk.pdf
> > > > The problem with rte_ring is that while head accusion is sort of
> > > > un-fair locking, waiting on tail is very similar to ticket lock
> > > > schema - tail has to be updated in particular order.
> > > > That makes current rte_ring implementation to perform really pure
> > > > on some overcommited scenarios.
> > >
> > > Rather than reform rte_ring to fit this scenario, it would make more
> > > sense to me to introduce another primitive. The current lockless
> > > ring performs very well for the isolated thread model that DPDK was
> > > built around. This looks like a case of customers violating the
> > > usage model of the DPDK and then being surprised at the fallout.
> >
> > I agree with Stephen here.
> >
> > I think, adding more runtime check in the enqueue() and dequeue() will
> > have a bad effect on the low-end cores too.
> > But I agree with the problem statement that in the virtualization use
> > case, It may be possible to have N virtual cores runs on a physical
> > core.
> >
> > IMO, The best solution would be keeping the ring API same and have a
> > different flavor in "compile-time". Something like liburcu did for
> > accommodating different flavors.
> >
> > i.e urcu-qsbr.h and urcu-bp.h will identical definition of API. The
> > application can simply include ONE header file in a C file based on
> > the flavor.
> > If need both at runtime. Need to have function pointer or so in the
> > application and define the function in different c file by including
> > the approaite flavor in C file.
>
> This would also be a good time to consider the tradeoffs of the heavy use of
> inlining that is done in rte_ring vs the impact that has on API/ABI stability.
>
I was working on few requirements in rte_ring library for RCU defer APIs. RFC is at https://patchwork.dpdk.org/cover/66020/.
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [RFC 0/6] New sync modes for ring
2020-02-24 17:59 0% ` Jerin Jacob
2020-02-24 19:35 3% ` Stephen Hemminger
@ 2020-02-25 0:58 0% ` Honnappa Nagarahalli
2020-02-25 15:14 0% ` Ananyev, Konstantin
1 sibling, 1 reply; 200+ results
From: Honnappa Nagarahalli @ 2020-02-25 0:58 UTC (permalink / raw)
To: Jerin Jacob, Stephen Hemminger
Cc: Konstantin Ananyev, dpdk-dev, Olivier Matz, Honnappa Nagarahalli, nd, nd
<snip>
>
> On Mon, Feb 24, 2020 at 10:29 PM Stephen Hemminger
> <stephen@networkplumber.org> wrote:
> >
> > On Mon, 24 Feb 2020 11:35:09 +0000
> > Konstantin Ananyev <konstantin.ananyev@intel.com> wrote:
> >
> > > Upfront note - that RFC is not a complete patch.
> > > It introduces an ABI breakage, plus it doesn't update ring_elem code
> > > properly, etc.
> > > I plan to deal with all these things in later versions.
> > > Right now I seek an initial feedback about proposed ideas.
> > > Would also ask people to repeat performance tests (see below) on
> > > their platforms to confirm the impact.
> > >
> > > More and more customers use(/try to use) DPDK based apps within
> > > overcommitted systems (multiple acttive threads over same pysical cores):
> > > VM, container deployments, etc.
> > > One quite common problem they hit: Lock-Holder-Preemption with
> rte_ring.
> > > LHP is quite a common problem for spin-based sync primitives
> > > (spin-locks, etc.) on overcommitted systems.
> > > The situation gets much worse when some sort of fair-locking
> > > technique is used (ticket-lock, etc.).
> > > As now not only lock-owner but also lock-waiters scheduling order
> > > matters a lot.
> > > This is a well-known problem for kernel within VMs:
> > > http://www-archive.xenproject.org/files/xensummitboston08/LHP.pdf
> > > https://www.cs.hs-rm.de/~kaiser/events/wamos2017/Slides/selcuk.pdf
These slides seem to indicate that the problems are mitigated through the Hypervisor configuration. Do we still need to address the issues?
> > > The problem with rte_ring is that while head accusion is sort of
> > > un-fair locking, waiting on tail is very similar to ticket lock
> > > schema - tail has to be updated in particular order.
> > > That makes current rte_ring implementation to perform really pure on
> > > some overcommited scenarios.
> >
> > Rather than reform rte_ring to fit this scenario, it would make more
> > sense to me to introduce another primitive. The current lockless ring
> > performs very well for the isolated thread model that DPDK was built
> > around. This looks like a case of customers violating the usage model
> > of the DPDK and then being surprised at the fallout.
>
> I agree with Stephen here.
>
> I think, adding more runtime check in the enqueue() and dequeue() will have a
> bad effect on the low-end cores too.
> But I agree with the problem statement that in the virtualization use case, It
> may be possible to have N virtual cores runs on a physical core.
It is hard to imagine that there are data plane applications deployed in such environments. Wouldn't this affect the performance terribly?
>
> IMO, The best solution would be keeping the ring API same and have a
> different flavor in "compile-time". Something like liburcu did for
> accommodating different flavors.
>
> i.e urcu-qsbr.h and urcu-bp.h will identical definition of API. The application
> can simply include ONE header file in a C file based on the flavor.
> If need both at runtime. Need to have function pointer or so in the application
> and define the function in different c file by including the approaite flavor in C
> file.
>
> #include <urcu-qsbr.h> /* QSBR RCU flavor */ #include <urcu-bp.h> /*
> Bulletproof RCU flavor */
>
>
>
>
>
>
>
>
>
>
>
>
>
> >
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v2 0/7] vfio/pci: SR-IOV support
2020-02-19 18:53 2% [dpdk-dev] [PATCH v2 0/7] vfio/pci: SR-IOV support Alex Williamson
@ 2020-02-25 2:33 0% ` Tian, Kevin
2020-03-05 17:33 3% ` Alex Williamson
2020-03-05 6:38 0% ` Vamsi Krishna Attunuru
1 sibling, 1 reply; 200+ results
From: Tian, Kevin @ 2020-02-25 2:33 UTC (permalink / raw)
To: Alex Williamson, kvm
Cc: linux-pci, linux-kernel, dev, mtosatti, thomas, bluca,
jerinjacobk, Richardson, Bruce, cohuck, Jason Wang
> From: Alex Williamson
> Sent: Thursday, February 20, 2020 2:54 AM
>
> Changes since v1 are primarily to patch 3/7 where the commit log is
> rewritten, along with option parsing and failure logging based on
> upstream discussions. The primary user visible difference is that
> option parsing is now much more strict. If a vf_token option is
> provided that cannot be used, we generate an error. As a result of
> this, opening a PF with a vf_token option will serve as a mechanism of
> setting the vf_token. This seems like a more user friendly API than
> the alternative of sometimes requiring the option (VFs in use) and
> sometimes rejecting it, and upholds our desire that the option is
> always either used or rejected.
>
> This also means that the VFIO_DEVICE_FEATURE ioctl is not the only
> means of setting the VF token, which might call into question whether
> we absolutely need this new ioctl. Currently I'm keeping it because I
> can imagine use cases, for example if a hypervisor were to support
> SR-IOV, the PF device might be opened without consideration for a VF
> token and we'd require the hypservisor to close and re-open the PF in
> order to set a known VF token, which is impractical.
>
> Series overview (same as provided with v1):
Thanks for doing this!
>
> The synopsis of this series is that we have an ongoing desire to drive
> PCIe SR-IOV PFs from userspace with VFIO. There's an immediate need
> for this with DPDK drivers and potentially interesting future use
Can you provide a link to the DPDK discussion?
> cases in virtualization. We've been reluctant to add this support
> previously due to the dependency and trust relationship between the
> VF device and PF driver. Minimally the PF driver can induce a denial
> of service to the VF, but depending on the specific implementation,
> the PF driver might also be responsible for moving data between VFs
> or have direct access to the state of the VF, including data or state
> otherwise private to the VF or VF driver.
Just a loud thinking. While the motivation of VF token sounds reasonable
to me, I'm curious why the same concern is not raised in other usages.
For example, there is no such design in virtio framework, where the
virtio device could also be restarted, putting in separate process (vhost-user),
and even in separate VM (virtio-vhost-user), etc. Of course the para-
virtualized attribute of virtio implies some degree of trust, but as you
mentioned many SR-IOV implementations support VF->PF communication
which also implies some level of trust. It's perfectly fine if VFIO just tries
to do better than other sub-systems, but knowing how other people
tackle the similar problem may make the whole picture clearer. 😊
+Jason.
>
> To help resolve these concerns, we introduce a VF token into the VFIO
> PCI ABI, which acts as a shared secret key between drivers. The
> userspace PF driver is required to set the VF token to a known value
> and userspace VF drivers are required to provide the token to access
> the VF device. If a PF driver is restarted with VF drivers in use, it
> must also provide the current token in order to prevent a rogue
> untrusted PF driver from replacing a known driver. The degree to
> which this new token is considered secret is left to the userspace
> drivers, the kernel intentionally provides no means to retrieve the
> current token.
I'm wondering whether the token idea can be used beyond SR-IOV, e.g.
(1) we may allow vfio user space to manage Scalable IOV in the future,
which faces the similar challenge between the PF and mdev; (2) the
token might be used as a canonical way to replace off-tree acs-override
workaround, say, allowing the admin to assign devices within the
same iommu group to different VMs which trust each other. I'm not
sure how much complexity will be further introduced, but it's greatly
appreciated if you can help think a bit and if feasible abstract some
logic in vfio core layer for such potential usages...
>
> Note that the above token is only required for this new model where
> both the PF and VF devices are usable through vfio-pci. Existing
> models of VFIO drivers where the PF is used without SR-IOV enabled
> or the VF is bound to a userspace driver with an in-kernel, host PF
> driver are unaffected.
>
> The latter configuration above also highlights a new inverted scenario
> that is now possible, a userspace PF driver with in-kernel VF drivers.
> I believe this is a scenario that should be allowed, but should not be
> enabled by default. This series includes code to set a default
> driver_override for VFs sourced from a vfio-pci user owned PF, such
> that the VFs are also bound to vfio-pci. This model is compatible
> with tools like driverctl and allows the system administrator to
> decide if other bindings should be enabled. The VF token interface
> above exists only between vfio-pci PF and VF drivers, once a VF is
> bound to another driver, the administrator has effectively pronounced
> the device as trusted. The vfio-pci driver will note alternate
> binding in dmesg for logging and debugging purposes.
>
> Please review, comment, and test. The example QEMU implementation
> provided with the RFC is still current for this version. Thanks,
>
> Alex
>
> RFC:
> https://lore.kernel.org/lkml/158085337582.9445.17682266437583505502.stg
> it@gimli.home/
> v1:
> https://lore.kernel.org/lkml/158145472604.16827.15751375540102298130.st
> git@gimli.home/
>
> ---
>
> Alex Williamson (7):
> vfio: Include optional device match in vfio_device_ops callbacks
> vfio/pci: Implement match ops
> vfio/pci: Introduce VF token
> vfio: Introduce VFIO_DEVICE_FEATURE ioctl and first user
> vfio/pci: Add sriov_configure support
> vfio/pci: Remove dev_fmt definition
> vfio/pci: Cleanup .probe() exit paths
>
>
> drivers/vfio/pci/vfio_pci.c | 383
> +++++++++++++++++++++++++++++++++--
> drivers/vfio/pci/vfio_pci_private.h | 10 +
> drivers/vfio/vfio.c | 20 +-
> include/linux/vfio.h | 4
> include/uapi/linux/vfio.h | 37 +++
> 5 files changed, 426 insertions(+), 28 deletions(-)
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] doc: plan splitting the ethdev ops struct
2020-02-21 10:40 0% ` Ferruh Yigit
@ 2020-02-25 10:35 0% ` Andrew Rybchenko
2020-02-25 11:07 3% ` Ananyev, Konstantin
2020-02-25 12:28 3% ` Ferruh Yigit
0 siblings, 2 replies; 200+ results
From: Andrew Rybchenko @ 2020-02-25 10:35 UTC (permalink / raw)
To: Ferruh Yigit, Stephen Hemminger
Cc: Neil Horman, John McNamara, Marko Kovacevic, dev, David Marchand,
Thomas Monjalon
On 2/21/20 1:40 PM, Ferruh Yigit wrote:
> On 2/18/2020 6:01 AM, Stephen Hemminger wrote:
>> On Mon, 17 Feb 2020 15:38:05 +0000
>> Ferruh Yigit <ferruh.yigit@intel.com> wrote:
>>
>>> For the ABI compatibility it is better to hide internal data structures
>>> from the application as much as possible. But because of some inline
>>> functions 'struct eth_dev_ops' can't be hidden completely.
>>>
>>> Plan is to split the 'struct eth_dev_ops' into two as ones used by
>>> inline functions and ones not used, and hide the second part that not
>>> used by inline functions completely to the application.
>>>
>>> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
>>> ---
>>> Cc: David Marchand <david.marchand@redhat.com>
>>> Cc: Thomas Monjalon <thomas@monjalon.net>
>>> Cc: Andrew Rybchenko <arybchenko@solarflare.com>
>>> ---
>>> doc/guides/rel_notes/deprecation.rst | 6 ++++++
>>> 1 file changed, 6 insertions(+)
>>>
>>> diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
>>> index dfcca87ab..2aa431028 100644
>>> --- a/doc/guides/rel_notes/deprecation.rst
>>> +++ b/doc/guides/rel_notes/deprecation.rst
>>> @@ -72,6 +72,12 @@ Deprecation Notices
>>> In 19.11 PMDs will still update the field even when the offload is not
>>> enabled.
>>>
>>> +* ethdev: Split the ``struct eth_dev_ops`` struct to hide it as much as possible.
>>> + Currently the ``struct eth_dev_ops`` struct is accessible by the application
>>> + because some inline functions, like ``rte_eth_tx_descriptor_status()``,
>>> + access the struct directly. The struct will be separate in two, the ops used
>>> + by inline functions still will be accessible to user but rest will be hidden.
>>> +
>>> * cryptodev: support for using IV with all sizes is added, J0 still can
>>> be used but only when IV length in following structs ``rte_crypto_auth_xform``,
>>> ``rte_crypto_aead_xform`` is set to zero. When IV length is greater or equal
>>
>> Good luck, truely hiding internals is hard. The mbuf structure is already split but not really
>> hidden at all (just look at dwarf output). It doesn't make sense to do it unless
>> you can really hide it.
>
> I believe this can be done, only following [1] dev_ops are used by inline
> functions, rest can be moved into separate struct and moved into ethdev driver
> looking header.
>
> [1]
> rx_queue_count
> rx_descriptor_done
> rx_descriptor_status
> tx_descriptor_status
I think having 3 places (if I understand the intention
correctly) with ethdev callbacks is too much. So, I think
that these ops should be simply moved to nearby Tx/Rx
burst and Tx prepare callbacks (e.g. move into inline_ops
structure which is located at the beginning of rte_eth_dev
in order to preserve 3 existing callback location).
Also I'd consider to deprecate and remove rx_queue_count
and rx_descriptor_done.
>>
>> I would attack the rte_device stuff first. Make rte_device opaque to the application
>> that would help for future versions. Then work backwards to rte_tehtdev.
>>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] doc: deprecate using MAX values as array size
2020-02-24 6:18 0% ` Akhil Goyal
@ 2020-02-25 11:03 0% ` Thomas Monjalon
0 siblings, 0 replies; 200+ results
From: Thomas Monjalon @ 2020-02-25 11:03 UTC (permalink / raw)
To: Ferruh Yigit
Cc: Neil Horman, John McNamara, Marko Kovacevic, dev,
Bruce Richardson, Konstantin Ananyev, David Marchand,
Andrew Rybchenko, Ori Kam, Akhil Goyal
> > > Adding the deprecation notice as reminder for next ABI breakage release
> > > (20.11).
> > > This one time breakage is required to be able to extend enum/define
> > > without breaking ABI.
> > >
> > > Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
> > > ---
> > > +* lib: will fix extending some enum/define breaking the ABI. There are
> > multiple
> > > + samples in DPDK that enum/define terminated with a ``.*MAX.*`` value
> > which is
> > > + used by iterators, and arrays holding these values are sized with this
> > > + ``.*MAX.*`` value. So extending this enum/define increases the ``.*MAX.*``
> > > + value which increases the size of the array and depending on how/where the
> > > + array is used this may break the ABI.
> > > + ``RTE_ETH_FLOW_MAX`` is one sample of the mentioned case, adding a
> > new flow
> > > + type will break the ABI because of ``flex_mask[RTE_ETH_FLOW_MAX]``
> > array
> > > + usage in following public struct hierarchy:
> > > + ``rte_eth_fdir_flex_conf -> rte_fdir_conf -> rte_eth_conf (in the middle)``.
> > > + Need to identify this kind of usages and fix in 20.11, otherwise this blocks
> > > + us extending existing enum/define.
> > > + One solution can be using a fixed size array instead of ``.*MAX.*`` value.
> >
> We admit that the issue is there and we can discuss the possible solutions for this issue.
>
> For the deprecation notice.
> Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
I think we'll need a new coding style guideline to avoid shuch issue in future.
Anyway, about fixing the issue in 20.11,
Acked-by: Thomas Monjalon <thomas@monjalon.net>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] doc: plan splitting the ethdev ops struct
2020-02-25 10:35 0% ` Andrew Rybchenko
@ 2020-02-25 11:07 3% ` Ananyev, Konstantin
2020-02-25 11:19 0% ` Andrew Rybchenko
2020-02-25 12:28 3% ` Ferruh Yigit
1 sibling, 1 reply; 200+ results
From: Ananyev, Konstantin @ 2020-02-25 11:07 UTC (permalink / raw)
To: Andrew Rybchenko, Yigit, Ferruh, Stephen Hemminger
Cc: Neil Horman, Mcnamara, John, Kovacevic, Marko, dev,
David Marchand, Thomas Monjalon
Hi Andew,
> On 2/21/20 1:40 PM, Ferruh Yigit wrote:
> > On 2/18/2020 6:01 AM, Stephen Hemminger wrote:
> >> On Mon, 17 Feb 2020 15:38:05 +0000
> >> Ferruh Yigit <ferruh.yigit@intel.com> wrote:
> >>
> >>> For the ABI compatibility it is better to hide internal data structures
> >>> from the application as much as possible. But because of some inline
> >>> functions 'struct eth_dev_ops' can't be hidden completely.
> >>>
> >>> Plan is to split the 'struct eth_dev_ops' into two as ones used by
> >>> inline functions and ones not used, and hide the second part that not
> >>> used by inline functions completely to the application.
> >>>
> >>> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
> >>> ---
> >>> Cc: David Marchand <david.marchand@redhat.com>
> >>> Cc: Thomas Monjalon <thomas@monjalon.net>
> >>> Cc: Andrew Rybchenko <arybchenko@solarflare.com>
> >>> ---
> >>> doc/guides/rel_notes/deprecation.rst | 6 ++++++
> >>> 1 file changed, 6 insertions(+)
> >>>
> >>> diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
> >>> index dfcca87ab..2aa431028 100644
> >>> --- a/doc/guides/rel_notes/deprecation.rst
> >>> +++ b/doc/guides/rel_notes/deprecation.rst
> >>> @@ -72,6 +72,12 @@ Deprecation Notices
> >>> In 19.11 PMDs will still update the field even when the offload is not
> >>> enabled.
> >>>
> >>> +* ethdev: Split the ``struct eth_dev_ops`` struct to hide it as much as possible.
> >>> + Currently the ``struct eth_dev_ops`` struct is accessible by the application
> >>> + because some inline functions, like ``rte_eth_tx_descriptor_status()``,
> >>> + access the struct directly. The struct will be separate in two, the ops used
> >>> + by inline functions still will be accessible to user but rest will be hidden.
> >>> +
> >>> * cryptodev: support for using IV with all sizes is added, J0 still can
> >>> be used but only when IV length in following structs ``rte_crypto_auth_xform``,
> >>> ``rte_crypto_aead_xform`` is set to zero. When IV length is greater or equal
> >>
> >> Good luck, truely hiding internals is hard. The mbuf structure is already split but not really
> >> hidden at all (just look at dwarf output). It doesn't make sense to do it unless
> >> you can really hide it.
> >
> > I believe this can be done, only following [1] dev_ops are used by inline
> > functions, rest can be moved into separate struct and moved into ethdev driver
> > looking header.
> >
> > [1]
> > rx_queue_count
> > rx_descriptor_done
> > rx_descriptor_status
> > tx_descriptor_status
>
> I think having 3 places (if I understand the intention
> correctly) with ethdev callbacks is too much. So, I think
> that these ops should be simply moved to nearby Tx/Rx
> burst and Tx prepare callbacks (e.g. move into inline_ops
> structure which is located at the beginning of rte_eth_dev
> in order to preserve 3 existing callback location).
If you are going to change ABI anyway, would it be worth to consider
moving rx/tx burst/prepare functions to be per queue,
instead of per device?
>
> Also I'd consider to deprecate and remove rx_queue_count
> and rx_descriptor_done.
>
> >>
> >> I would attack the rte_device stuff first. Make rte_device opaque to the application
> >> that would help for future versions. Then work backwards to rte_tehtdev.
> >>
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH] doc: plan splitting the ethdev ops struct
2020-02-25 11:07 3% ` Ananyev, Konstantin
@ 2020-02-25 11:19 0% ` Andrew Rybchenko
0 siblings, 0 replies; 200+ results
From: Andrew Rybchenko @ 2020-02-25 11:19 UTC (permalink / raw)
To: Ananyev, Konstantin, Yigit, Ferruh, Stephen Hemminger
Cc: Neil Horman, Mcnamara, John, Kovacevic, Marko, dev,
David Marchand, Thomas Monjalon
Hi Konstantin,
On 2/25/20 2:07 PM, Ananyev, Konstantin wrote:
> Hi Andew,
>
>> On 2/21/20 1:40 PM, Ferruh Yigit wrote:
>>> On 2/18/2020 6:01 AM, Stephen Hemminger wrote:
>>>> On Mon, 17 Feb 2020 15:38:05 +0000
>>>> Ferruh Yigit <ferruh.yigit@intel.com> wrote:
>>>>
>>>>> For the ABI compatibility it is better to hide internal data structures
>>>>> from the application as much as possible. But because of some inline
>>>>> functions 'struct eth_dev_ops' can't be hidden completely.
>>>>>
>>>>> Plan is to split the 'struct eth_dev_ops' into two as ones used by
>>>>> inline functions and ones not used, and hide the second part that not
>>>>> used by inline functions completely to the application.
>>>>>
>>>>> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
>>>>> ---
>>>>> Cc: David Marchand <david.marchand@redhat.com>
>>>>> Cc: Thomas Monjalon <thomas@monjalon.net>
>>>>> Cc: Andrew Rybchenko <arybchenko@solarflare.com>
>>>>> ---
>>>>> doc/guides/rel_notes/deprecation.rst | 6 ++++++
>>>>> 1 file changed, 6 insertions(+)
>>>>>
>>>>> diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
>>>>> index dfcca87ab..2aa431028 100644
>>>>> --- a/doc/guides/rel_notes/deprecation.rst
>>>>> +++ b/doc/guides/rel_notes/deprecation.rst
>>>>> @@ -72,6 +72,12 @@ Deprecation Notices
>>>>> In 19.11 PMDs will still update the field even when the offload is not
>>>>> enabled.
>>>>>
>>>>> +* ethdev: Split the ``struct eth_dev_ops`` struct to hide it as much as possible.
>>>>> + Currently the ``struct eth_dev_ops`` struct is accessible by the application
>>>>> + because some inline functions, like ``rte_eth_tx_descriptor_status()``,
>>>>> + access the struct directly. The struct will be separate in two, the ops used
>>>>> + by inline functions still will be accessible to user but rest will be hidden.
>>>>> +
>>>>> * cryptodev: support for using IV with all sizes is added, J0 still can
>>>>> be used but only when IV length in following structs ``rte_crypto_auth_xform``,
>>>>> ``rte_crypto_aead_xform`` is set to zero. When IV length is greater or equal
>>>> Good luck, truely hiding internals is hard. The mbuf structure is already split but not really
>>>> hidden at all (just look at dwarf output). It doesn't make sense to do it unless
>>>> you can really hide it.
>>> I believe this can be done, only following [1] dev_ops are used by inline
>>> functions, rest can be moved into separate struct and moved into ethdev driver
>>> looking header.
>>>
>>> [1]
>>> rx_queue_count
>>> rx_descriptor_done
>>> rx_descriptor_status
>>> tx_descriptor_status
>> I think having 3 places (if I understand the intention
>> correctly) with ethdev callbacks is too much. So, I think
>> that these ops should be simply moved to nearby Tx/Rx
>> burst and Tx prepare callbacks (e.g. move into inline_ops
>> structure which is located at the beginning of rte_eth_dev
>> in order to preserve 3 existing callback location).
> If you are going to change ABI anyway, would it be worth to consider
> moving rx/tx burst/prepare functions to be per queue,
> instead of per device?
I'm thinking about it from time to time. In general I like the idea.
The only question I have if there is a demand for queues with
different offloads which could result in different Rx/Tx callbacks.
Also I'm not sure that it is doable without performance degradation
at all, but hopefully it will be tiny.
>> Also I'd consider to deprecate and remove rx_queue_count
>> and rx_descriptor_done.
>>
>>>> I would attack the rte_device stuff first. Make rte_device opaque to the application
>>>> that would help for future versions. Then work backwards to rte_tehtdev.
>>>>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [RFC 0/6] New sync modes for ring
2020-02-24 20:52 0% ` Honnappa Nagarahalli
@ 2020-02-25 11:45 0% ` Ananyev, Konstantin
0 siblings, 0 replies; 200+ results
From: Ananyev, Konstantin @ 2020-02-25 11:45 UTC (permalink / raw)
To: Honnappa Nagarahalli, Stephen Hemminger, Jerin Jacob
Cc: dpdk-dev, Olivier Matz, nd, nd
> > -----Original Message-----
> > From: dev <dev-bounces@dpdk.org> On Behalf Of Stephen Hemminger
> > Sent: Monday, February 24, 2020 1:35 PM
> > To: Jerin Jacob <jerinjacobk@gmail.com>
> > Cc: Konstantin Ananyev <konstantin.ananyev@intel.com>; dpdk-dev
> > <dev@dpdk.org>; Olivier Matz <olivier.matz@6wind.com>
> > Subject: Re: [dpdk-dev] [RFC 0/6] New sync modes for ring
> >
> > On Mon, 24 Feb 2020 23:29:57 +0530
> > Jerin Jacob <jerinjacobk@gmail.com> wrote:
> >
> > > On Mon, Feb 24, 2020 at 10:29 PM Stephen Hemminger
> > > <stephen@networkplumber.org> wrote:
> > > >
> > > > On Mon, 24 Feb 2020 11:35:09 +0000
> > > > Konstantin Ananyev <konstantin.ananyev@intel.com> wrote:
> > > >
> > > > > Upfront note - that RFC is not a complete patch.
> > > > > It introduces an ABI breakage, plus it doesn't update ring_elem
> > > > > code properly, etc.
> > > > > I plan to deal with all these things in later versions.
> > > > > Right now I seek an initial feedback about proposed ideas.
> > > > > Would also ask people to repeat performance tests (see below) on
> > > > > their platforms to confirm the impact.
> > > > >
> > > > > More and more customers use(/try to use) DPDK based apps within
> > > > > overcommitted systems (multiple acttive threads over same pysical
> > cores):
> > > > > VM, container deployments, etc.
> > > > > One quite common problem they hit: Lock-Holder-Preemption with
> > rte_ring.
> > > > > LHP is quite a common problem for spin-based sync primitives
> > > > > (spin-locks, etc.) on overcommitted systems.
> > > > > The situation gets much worse when some sort of fair-locking
> > > > > technique is used (ticket-lock, etc.).
> > > > > As now not only lock-owner but also lock-waiters scheduling order
> > > > > matters a lot.
> > > > > This is a well-known problem for kernel within VMs:
> > > > > http://www-archive.xenproject.org/files/xensummitboston08/LHP.pdf
> > > > > https://www.cs.hs-rm.de/~kaiser/events/wamos2017/Slides/selcuk.pdf
> > > > > The problem with rte_ring is that while head accusion is sort of
> > > > > un-fair locking, waiting on tail is very similar to ticket lock
> > > > > schema - tail has to be updated in particular order.
> > > > > That makes current rte_ring implementation to perform really pure
> > > > > on some overcommited scenarios.
> > > >
> > > > Rather than reform rte_ring to fit this scenario, it would make more
> > > > sense to me to introduce another primitive. The current lockless
> > > > ring performs very well for the isolated thread model that DPDK was
> > > > built around. This looks like a case of customers violating the
> > > > usage model of the DPDK and then being surprised at the fallout.
> > >
> > > I agree with Stephen here.
> > >
> > > I think, adding more runtime check in the enqueue() and dequeue() will
> > > have a bad effect on the low-end cores too.
> > > But I agree with the problem statement that in the virtualization use
> > > case, It may be possible to have N virtual cores runs on a physical
> > > core.
> > >
> > > IMO, The best solution would be keeping the ring API same and have a
> > > different flavor in "compile-time". Something like liburcu did for
> > > accommodating different flavors.
> > >
> > > i.e urcu-qsbr.h and urcu-bp.h will identical definition of API. The
> > > application can simply include ONE header file in a C file based on
> > > the flavor.
> > > If need both at runtime. Need to have function pointer or so in the
> > > application and define the function in different c file by including
> > > the approaite flavor in C file.
> >
> > This would also be a good time to consider the tradeoffs of the heavy use of
> > inlining that is done in rte_ring vs the impact that has on API/ABI stability.
> >
> I was working on few requirements in rte_ring library for RCU defer APIs. RFC is at https://patchwork.dpdk.org/cover/66020/.
Yep, noticed your patch, seems we sort of collided here.
As I understand you patch aims to provide functionality similar to my HTS one.
Will try to look at yours one in next fee days, hopefully we can end-up with
some common denominator.
Konstantin
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] doc: plan splitting the ethdev ops struct
2020-02-25 10:35 0% ` Andrew Rybchenko
2020-02-25 11:07 3% ` Ananyev, Konstantin
@ 2020-02-25 12:28 3% ` Ferruh Yigit
1 sibling, 0 replies; 200+ results
From: Ferruh Yigit @ 2020-02-25 12:28 UTC (permalink / raw)
To: Andrew Rybchenko, Stephen Hemminger
Cc: Neil Horman, John McNamara, Marko Kovacevic, dev, David Marchand,
Thomas Monjalon, Kinsella, Ray, Olivier MATZ
On 2/25/2020 10:35 AM, Andrew Rybchenko wrote:
> On 2/21/20 1:40 PM, Ferruh Yigit wrote:
>> On 2/18/2020 6:01 AM, Stephen Hemminger wrote:
>>> On Mon, 17 Feb 2020 15:38:05 +0000
>>> Ferruh Yigit <ferruh.yigit@intel.com> wrote:
>>>
>>>> For the ABI compatibility it is better to hide internal data structures
>>>> from the application as much as possible. But because of some inline
>>>> functions 'struct eth_dev_ops' can't be hidden completely.
>>>>
>>>> Plan is to split the 'struct eth_dev_ops' into two as ones used by
>>>> inline functions and ones not used, and hide the second part that not
>>>> used by inline functions completely to the application.
>>>>
>>>> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
>>>> ---
>>>> Cc: David Marchand <david.marchand@redhat.com>
>>>> Cc: Thomas Monjalon <thomas@monjalon.net>
>>>> Cc: Andrew Rybchenko <arybchenko@solarflare.com>
>>>> ---
>>>> doc/guides/rel_notes/deprecation.rst | 6 ++++++
>>>> 1 file changed, 6 insertions(+)
>>>>
>>>> diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
>>>> index dfcca87ab..2aa431028 100644
>>>> --- a/doc/guides/rel_notes/deprecation.rst
>>>> +++ b/doc/guides/rel_notes/deprecation.rst
>>>> @@ -72,6 +72,12 @@ Deprecation Notices
>>>> In 19.11 PMDs will still update the field even when the offload is not
>>>> enabled.
>>>>
>>>> +* ethdev: Split the ``struct eth_dev_ops`` struct to hide it as much as possible.
>>>> + Currently the ``struct eth_dev_ops`` struct is accessible by the application
>>>> + because some inline functions, like ``rte_eth_tx_descriptor_status()``,
>>>> + access the struct directly. The struct will be separate in two, the ops used
>>>> + by inline functions still will be accessible to user but rest will be hidden.
>>>> +
>>>> * cryptodev: support for using IV with all sizes is added, J0 still can
>>>> be used but only when IV length in following structs ``rte_crypto_auth_xform``,
>>>> ``rte_crypto_aead_xform`` is set to zero. When IV length is greater or equal
>>>
>>> Good luck, truely hiding internals is hard. The mbuf structure is already split but not really
>>> hidden at all (just look at dwarf output). It doesn't make sense to do it unless
>>> you can really hide it.
>>
>> I believe this can be done, only following [1] dev_ops are used by inline
>> functions, rest can be moved into separate struct and moved into ethdev driver
>> looking header.
>>
>> [1]
>> rx_queue_count
>> rx_descriptor_done
>> rx_descriptor_status
>> tx_descriptor_status
>
> I think having 3 places (if I understand the intention
> correctly) with ethdev callbacks is too much.
Yes, this is the intention.
> So, I think
> that these ops should be simply moved to nearby Tx/Rx
> burst and Tx prepare callbacks (e.g. move into inline_ops
> structure which is located at the beginning of rte_eth_dev
> in order to preserve 3 existing callback location).
I think this is reasonable, but this can be done only in 20.11 with ABI break.
What do you think doing the initial hide in 20.05 with three places and do
proper splitting in 20.11 as suggested above.
Or it is possible to drop the interim work and do the all changes in 20.11, not
sure.
>
> Also I'd consider to deprecate and remove rx_queue_count
> and rx_descriptor_done.
+1 to deprecate the 'rx_descriptor_done', we have already 'rx_descriptor_status'
& 'tx_descriptor_status' to replace the functionality.
@Thomas, @Ray, can you please remind how deprecation was done, is it marking
API, 'rte_eth_rx_descriptor_done', as deprecated in 20.11 and removing it in
21.11? If so I guess deprecation notice should be send before 20.11? (And make
sure all PMDs did the switch before 20.11)
Not sure about 'rx_queue_count', if any application may need it or not.
>
>>>
>>> I would attack the rte_device stuff first. Make rte_device opaque to the application
>>> that would help for future versions. Then work backwards to rte_tehtdev.
>>>
>
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH] doc: plan splitting the ethdev ops struct
2020-02-18 5:07 0% ` Jerin Jacob
@ 2020-02-25 12:42 0% ` Ferruh Yigit
0 siblings, 0 replies; 200+ results
From: Ferruh Yigit @ 2020-02-25 12:42 UTC (permalink / raw)
To: Jerin Jacob
Cc: Neil Horman, John McNamara, Marko Kovacevic, dpdk-dev,
David Marchand, Thomas Monjalon, Andrew Rybchenko
On 2/18/2020 5:07 AM, Jerin Jacob wrote:
> On Mon, Feb 17, 2020 at 9:08 PM Ferruh Yigit <ferruh.yigit@intel.com> wrote:
>>
>> For the ABI compatibility it is better to hide internal data structures
>> from the application as much as possible. But because of some inline
>> functions 'struct eth_dev_ops' can't be hidden completely.
>>
>> Plan is to split the 'struct eth_dev_ops' into two as ones used by
>> inline functions and ones not used, and hide the second part that not
>> used by inline functions completely to the application.
>
> It is a good improvement. IMO, If anything used in fast-path it
> should be in ``struct rte_eth_dev``
> and rest can completely be moved to internal. In this case, if
> `rte_eth_tx_descriptor_status`
> not used on fastpath, Maybe we don't need to maintain the inline
> status and move completely
> to .c file.
Moving fast-past dev_ops to ``struct rte_eth_dev`` makes sense, also suggested
by Andrew, and will hide the rest.
Those inline dev_ops are datapath related so may be better to keep them as
inline, but not really sure about their users and how much they care about minor
performance impact if we chose to convert them into regular functions.
>
> Those may be specifics of the work. In general, this change looks good to me.
>
> Acked-by: Jerin Jacob <jerinj@marvell.com>
>
>
>
>
>
>>
>> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
>> ---
>> Cc: David Marchand <david.marchand@redhat.com>
>> Cc: Thomas Monjalon <thomas@monjalon.net>
>> Cc: Andrew Rybchenko <arybchenko@solarflare.com>
>> ---
>> doc/guides/rel_notes/deprecation.rst | 6 ++++++
>> 1 file changed, 6 insertions(+)
>>
>> diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
>> index dfcca87ab..2aa431028 100644
>> --- a/doc/guides/rel_notes/deprecation.rst
>> +++ b/doc/guides/rel_notes/deprecation.rst
>> @@ -72,6 +72,12 @@ Deprecation Notices
>> In 19.11 PMDs will still update the field even when the offload is not
>> enabled.
>>
>> +* ethdev: Split the ``struct eth_dev_ops`` struct to hide it as much as possible.
>> + Currently the ``struct eth_dev_ops`` struct is accessible by the application
>> + because some inline functions, like ``rte_eth_tx_descriptor_status()``,
>> + access the struct directly. The struct will be separate in two, the ops used
>> + by inline functions still will be accessible to user but rest will be hidden.
>> +
>> * cryptodev: support for using IV with all sizes is added, J0 still can
>> be used but only when IV length in following structs ``rte_crypto_auth_xform``,
>> ``rte_crypto_aead_xform`` is set to zero. When IV length is greater or equal
>> --
>> 2.24.1
>>
^ permalink raw reply [relevance 0%]
* [dpdk-dev] [PATCH v2] doc: plan splitting the ethdev ops struct
2020-02-17 15:38 3% [dpdk-dev] [PATCH] doc: plan splitting the ethdev ops struct Ferruh Yigit
2020-02-18 5:07 0% ` Jerin Jacob
2020-02-18 6:01 0% ` Stephen Hemminger
@ 2020-02-25 12:44 3% ` Ferruh Yigit
2020-02-25 15:51 3% ` Andrew Rybchenko
` (2 more replies)
2 siblings, 3 replies; 200+ results
From: Ferruh Yigit @ 2020-02-25 12:44 UTC (permalink / raw)
To: Neil Horman, John McNamara, Marko Kovacevic
Cc: dev, Ferruh Yigit, Jerin Jacob, David Marchand, Thomas Monjalon,
Andrew Rybchenko
For the ABI compatibility it is better to hide internal data structures
from the application as much as possible. But because of some inline
functions 'struct eth_dev_ops' can't be hidden completely.
Plan is to split the 'struct eth_dev_ops' into two as ones used by
inline functions and ones not used, and hide the second part that not
used by inline functions completely to the application.
Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
Acked-by: Jerin Jacob <jerinj@marvell.com>
---
Cc: David Marchand <david.marchand@redhat.com>
Cc: Thomas Monjalon <thomas@monjalon.net>
Cc: Andrew Rybchenko <arybchenko@solarflare.com>
v2:
* Add target date for the work
* Give more detail on what will be done
---
doc/guides/rel_notes/deprecation.rst | 11 +++++++++++
1 file changed, 11 insertions(+)
diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
index 99d81564a..ff612a615 100644
--- a/doc/guides/rel_notes/deprecation.rst
+++ b/doc/guides/rel_notes/deprecation.rst
@@ -86,6 +86,17 @@ Deprecation Notices
In 19.11 PMDs will still update the field even when the offload is not
enabled.
+* ethdev: Split the ``struct eth_dev_ops`` struct to hide it as much as possible.
+ Currently the ``struct eth_dev_ops`` struct is accessible by the application
+ because some inline functions, like ``rte_eth_tx_descriptor_status()``,
+ access the struct directly. The struct will be separate in two, the ops used
+ by inline functions still will be accessible to user but rest will be hidden.
+ Initial split will be done in 20.05 with adding reserved fields for the struct
+ used by inline functions, and by putting new struct reference into public one
+ to not increase the size of ``struct rte_eth_dev``, proper split will be done
+ in 20.11 by moving inline dev_ops function to next to Rx/Tx burst functions and
+ hiding rest.
+
* cryptodev: support for using IV with all sizes is added, J0 still can
be used but only when IV length in following structs ``rte_crypto_auth_xform``,
``rte_crypto_aead_xform`` is set to zero. When IV length is greater or equal
--
2.24.1
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [RFC 0/6] New sync modes for ring
2020-02-24 19:35 3% ` Stephen Hemminger
2020-02-24 20:52 0% ` Honnappa Nagarahalli
@ 2020-02-25 13:41 3% ` Ananyev, Konstantin
2020-02-27 10:31 0% ` Jerin Jacob
1 sibling, 1 reply; 200+ results
From: Ananyev, Konstantin @ 2020-02-25 13:41 UTC (permalink / raw)
To: Stephen Hemminger, Jerin Jacob; +Cc: dpdk-dev, Olivier Matz, drc
> > > > Upfront note - that RFC is not a complete patch.
> > > > It introduces an ABI breakage, plus it doesn't update ring_elem
> > > > code properly, etc.
> > > > I plan to deal with all these things in later versions.
> > > > Right now I seek an initial feedback about proposed ideas.
> > > > Would also ask people to repeat performance tests (see below)
> > > > on their platforms to confirm the impact.
> > > >
> > > > More and more customers use(/try to use) DPDK based apps within
> > > > overcommitted systems (multiple acttive threads over same pysical cores):
> > > > VM, container deployments, etc.
> > > > One quite common problem they hit: Lock-Holder-Preemption with rte_ring.
> > > > LHP is quite a common problem for spin-based sync primitives
> > > > (spin-locks, etc.) on overcommitted systems.
> > > > The situation gets much worse when some sort of
> > > > fair-locking technique is used (ticket-lock, etc.).
> > > > As now not only lock-owner but also lock-waiters scheduling
> > > > order matters a lot.
> > > > This is a well-known problem for kernel within VMs:
> > > > http://www-archive.xenproject.org/files/xensummitboston08/LHP.pdf
> > > > https://www.cs.hs-rm.de/~kaiser/events/wamos2017/Slides/selcuk.pdf
> > > > The problem with rte_ring is that while head accusion is sort of
> > > > un-fair locking, waiting on tail is very similar to ticket lock schema -
> > > > tail has to be updated in particular order.
> > > > That makes current rte_ring implementation to perform
> > > > really pure on some overcommited scenarios.
> > >
> > > Rather than reform rte_ring to fit this scenario, it would make
> > > more sense to me to introduce another primitive.
I don't see much advantages it will bring us.
As a disadvantages, for developers and maintainers - code duplication,
for end users - extra code churn and removed ability to mix and match
different sync modes in one ring.
> The current lockless
> > > ring performs very well for the isolated thread model that DPDK
> > > was built around. This looks like a case of customers violating
> > > the usage model of the DPDK and then being surprised at the fallout.
For customers using isolated thread model - nothing should change
(both in terms of API and performance).
Existing sync modes MP/MC,SP/SC kept untouched, set up in the same
way (via flags and _init_), and MP/MC remains as default one.
From other side I don't see why we should ignore customers that want to use
their DPDK apps in different deployment scenarios.
> >
> > I agree with Stephen here.
> >
> > I think, adding more runtime check in the enqueue() and dequeue() will
> > have a bad effect on the low-end cores too.
We do have a run-time check in our current enqueue()/dequeue implementation.
In fact we support both modes: we have generic rte_ring_enqueue(/dequeue)_bulk(/burst)
where sync behaviour is determined at runtime by value of prod(/cons).single.
Or user can call rte_ring_(mp/sp)_enqueue_* functions directly.
This RFC follows exactly the same paradigm:
rte_ring_enqueue(/dequeue)_bulk(/burst) kept generic and it's
behaviour is determined at runtime, by value of prod(/cons).sync_type.
Or user can call enqueue/dequeue with particular sync mode directly:
rte_ring_(mp/sp/rts/hts)_enqueue_(bulk/burst)*.
The only thing that changed:
Format of prod/cons now could differ depending on mode selected at _init_.
So you can't create a ring for let say SP mode and then in the middle of data-path
change your mind and start using MP_RTS mode.
For existing modes (SP/MP, SC/MC) format remains the same and user can still
use them interchangeably, though of course that is an error prone practice.
> > But I agree with the problem statement that in the virtualization use
> > case, It may be possible to have N virtual cores runs on a physical
> > core.
> >
> > IMO, The best solution would be keeping the ring API same and have a
> > different flavor in "compile-time". Something like
> > liburcu did for accommodating different flavors.
> >
> > i.e urcu-qsbr.h and urcu-bp.h will identical definition of API. The
> > application can simply include ONE header file in a C file based on
> > the flavor.
I don't think it is a flexible enough approach.
In one app user might need to have several rings with different sync modes.
Or even user might need a ring with different sync modes for enqueue/dequeue.
> > If need both at runtime. Need to have function pointer or so in the
> > application and define the function in different c file by including
> > the approaite flavor in C file.
Big issue with function pointers here would be DPDK MP model.
AFAIK, rte_ring is quite popular mechanism for IPC between DPDK apps.
To support such model, we'll need to split rte_ring data into 'shared'
and 'private' and initialize private one for every process that is going to use it.
That sounds like a massive change, and I am not sure the required effort will worth it.
BTW, if user just calls API functions without trying to access structure internals directly,
I don't think it would be a big difference for him what is inside:
indirect function call or inlined switch(...) {}.
> This would also be a good time to consider the tradeoffs of the
> heavy use of inlining that is done in rte_ring vs the impact that
> has on API/ABI stability.
Yes, hiding rte_ring implementation inside .c would help a lot
in terms of ABI maintenance and would make our future life easier.
The question is what is the price for it in terms of performance,
and are we ready to pay it. Not to mention that it would cause
changes in many other libs/apps...
So I think it should be a subject for a separate discussion.
But, agree it would be good at least to measure the performance
impact of such change.
If I'll have some spare cycles, will give it a try.
Meanwhile, can I ask Jerin and other guys to repeat tests from this RFC
on their HW? Before continuing discussion would probably be good to know
does the suggested patch work as expected across different platforms.
Thanks
Konstantin
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH] doc: deprecate using MAX values as array size
2020-02-21 10:25 0% ` Ferruh Yigit
@ 2020-02-25 13:52 0% ` David Marchand
2020-02-25 14:53 0% ` David Marchand
2 siblings, 0 replies; 200+ results
From: David Marchand @ 2020-02-25 13:52 UTC (permalink / raw)
To: Ferruh Yigit; +Cc: Neil Horman, John McNamara, Marko Kovacevic, dev
On Thu, Jan 30, 2020 at 3:20 PM Ferruh Yigit <ferruh.yigit@intel.com> wrote:
> Adding the deprecation notice as reminder for next ABI breakage release
> (20.11).
> This one time breakage is required to be able to extend enum/define
> without breaking ABI.
>
> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
> ---
> doc/guides/rel_notes/deprecation.rst | 14 ++++++++++++++
> 1 file changed, 14 insertions(+)
>
> diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
> index dfcca87ab..99d81564a 100644
> --- a/doc/guides/rel_notes/deprecation.rst
> +++ b/doc/guides/rel_notes/deprecation.rst
> @@ -38,6 +38,20 @@ Deprecation Notices
> remove it from the externally visible ABI and allow it to be updated in the
> future.
>
> +* lib: will fix extending some enum/define breaking the ABI. There are multiple
> + samples in DPDK that enum/define terminated with a ``.*MAX.*`` value which is
> + used by iterators, and arrays holding these values are sized with this
> + ``.*MAX.*`` value. So extending this enum/define increases the ``.*MAX.*``
> + value which increases the size of the array and depending on how/where the
> + array is used this may break the ABI.
> + ``RTE_ETH_FLOW_MAX`` is one sample of the mentioned case, adding a new flow
> + type will break the ABI because of ``flex_mask[RTE_ETH_FLOW_MAX]`` array
> + usage in following public struct hierarchy:
> + ``rte_eth_fdir_flex_conf -> rte_fdir_conf -> rte_eth_conf (in the middle)``.
> + Need to identify this kind of usages and fix in 20.11, otherwise this blocks
> + us extending existing enum/define.
> + One solution can be using a fixed size array instead of ``.*MAX.*`` value.
> +
> * dpaa2: removal of ``rte_dpaa2_memsegs`` structure which has been replaced
> by a pa-va search library. This structure was earlier being used for holding
> memory segments used by dpaa2 driver for faster pa->va translation. This
> --
> 2.24.1
>
Acked-by: David Marchand <david.marchand@redhat.com>
--
David Marchand
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v4] doc: alias to experimental tag for stable apis
2020-02-18 10:46 17% ` [dpdk-dev] [PATCH v4] " Ray Kinsella
2020-02-21 17:11 0% ` Ferruh Yigit
@ 2020-02-25 14:22 0% ` Kevin Traynor
1 sibling, 0 replies; 200+ results
From: Kevin Traynor @ 2020-02-25 14:22 UTC (permalink / raw)
To: Ray Kinsella, dev
Cc: john.mcnamara, nhorman, ferruh.yigit, marko.kovacevic, bruce.richardson
On 18/02/2020 10:46, Ray Kinsella wrote:
> When a maintainer is promoting an API to become part of the next major ABI
> version by removing the experimental tag, possibly a few releases in
> advance of the declaration of the next ABI version. The maintainer may
> choose to offer an alias to the experimental tag, as removing the tag
> before the declaration of the next major ABI version, would cause an ABI
> breakage for applications using the API.
>
> Signed-off-by: Ray Kinsella <mdr@ashroe.eu>
> ---
> doc/guides/contributing/abi_policy.rst | 10 ++++++++++
> 1 file changed, 10 insertions(+)
>
> diff --git a/doc/guides/contributing/abi_policy.rst b/doc/guides/contributing/abi_policy.rst
> index 05ca959..7ae7de7 100644
> --- a/doc/guides/contributing/abi_policy.rst
> +++ b/doc/guides/contributing/abi_policy.rst
> @@ -159,6 +159,11 @@ The requirements for changing the ABI are:
> ``experimental``, as described in the section on :ref:`Experimental APIs
> and Libraries <experimental_apis>`.
>
> + - In situations in which an ``experimental`` API has been stable for some
> + time. When promoting the API to become part of the next ABI version, the
> + maintainer may choose to provide an alias to the ``experimental`` tag, so
> + as not to break consuming applications.
> +
> #. If a newly proposed API functionally replaces an existing one, when the new
> API becomes non-experimental, then the old one is marked with
> ``__rte_deprecated``.
> @@ -317,6 +322,11 @@ not required. Though, an API should remain in experimental state for at least
> one release. Thereafter, the normal process of posting patch for review to
> mailing list can be followed.
>
> +After the experimental tag has been formally removed, a tree/sub-tree maintainer
> +may choose to offer an alias to the experimental tag so as not to break
> +applications using the API. The alias is then dropped at the declaration of next
> +major ABI version.
> +
> Libraries
> ~~~~~~~~~
>
>
Sounds good. It doesn't change the minimum contract but gives some
flexibility to deal with individual cases.
Acked-by: Kevin Traynor <ktraynor@redhat.com>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [dpdk-announce] release candidate 20.02-rc4
@ 2020-02-25 14:37 4% ` Xu, Qian Q
2020-02-25 14:44 0% ` Thomas Monjalon
0 siblings, 1 reply; 200+ results
From: Xu, Qian Q @ 2020-02-25 14:37 UTC (permalink / raw)
To: Ali Alnubani, Thomas Monjalon, dev
Cc: Raslan Darawsheh, Asaf Penso, Peng, Yuan, Yu, PingX
Some quick update from Intel on RC4 testing
Summary
1. Totally execution percentage is around 90%, pass rate is around 98%
2. Created 2 new medium/low issue in RC4.
Details:
1. Intel NIC PMD PF & VF regression: 95% executed, 0 critical issue and high issue found.
Function cases include rx/tx/forward packets, packet processing, jumboframe, dual vlan, filter cases and so on: 2 bugs found, no critical/high bugs.
Performance cases such as l2fwd, l3fwd, single core and so on: no new bug is found.
2. Cryptodev cases include ipsec_gw cryptodev, fips cryptodev, l2fwd crypto, performance cases and so on: no critical and no high bugs found.
Function cases all passed, performance cases not fully executed.
3. Vhost/virtio: PVP/loopback, qemu/virtio-user, multi-queue, dequeue zero copy cases and so on: 1 new medium bug is found.
4. Power relate cases, no new bug is found.
5. ABI test includes unit test, vhost and flow related tests. no new bug is found.
6. 20.02 new features include ABI test, L2tpv3, crypto_dev related new features and so no. no new bug is found.
> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Ali Alnubani
> Sent: Tuesday, February 25, 2020 5:59 PM
> To: Thomas Monjalon <thomas@monjalon.net>; dev@dpdk.org
> Cc: Raslan Darawsheh <rasland@mellanox.com>; Asaf Penso
> <asafp@mellanox.com>
> Subject: Re: [dpdk-dev] [dpdk-announce] release candidate 20.02-rc4
>
> Hi,
>
> > -----Original Message-----
> > From: announce <announce-bounces@dpdk.org> On Behalf Of Thomas
> > Monjalon
> > Sent: Saturday, February 22, 2020 11:08 PM
> > To: announce@dpdk.org
> > Subject: [dpdk-announce] release candidate 20.02-rc4
> >
> > A new DPDK release candidate is ready for testing:
> <removed>
> >
> > This is the last release candidate for DPDK 20.02.
> > Please share some release validation results by replying to this
> > message (at dev@dpdk.org).
> > If no critical issue is reported, 20.02 will be closed on Tuesday 25th.
> >
> > If you are preparing the next release cycle, please send your v1
> > patches before the 20.05 proposal deadline, which will happen on March 18th.
> > It is also time to build an estimated roadmap for the next cycles.
> >
> > Thank you everyone
> >
>
> The following covers the tests that we ran on Mellanox hardware for this
> version:
>
> - Basic functionality:
> Send and receive multiple types of traffic.
> - testpmd xstats counter test.
> - testpmd timestamp test.
> - Changing/checking link status through testpmd.
> - RTE flow and flow_director tests.
> Items: eth / vlan / ipv4 / ipv6 / tcp / udp / icmp / gre / nvgre / vxlan / geneve /
> ip in ip / mplsoudp / mplsogre / gtp
> Actions: drop / queue / rss / mark / flag / jump / count / port_id / raw_encap /
> raw_decap / vxlan_encap / vxlan_decap / NAT
> - Some RSS tests.
> - VLAN stripping and insertion tests.
> - Checksum and TSO tests.
> - ptype tests.
> - l3fwd-power example application tests.
> - Multi-process example applications tests.
>
> We don't see any critical issues blocking the release.
>
> Thanks,
> Ali
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [dpdk-announce] release candidate 20.02-rc4
2020-02-25 14:37 4% ` Xu, Qian Q
@ 2020-02-25 14:44 0% ` Thomas Monjalon
2020-02-25 15:27 0% ` Raslan Darawsheh
0 siblings, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-02-25 14:44 UTC (permalink / raw)
To: Ali Alnubani, Xu, Qian Q, Raslan Darawsheh
Cc: dev, Asaf Penso, Peng, Yuan, Yu, PingX
Can we assume we are OK to release today?
25/02/2020 15:37, Xu, Qian Q:
> Some quick update from Intel on RC4 testing
>
> Summary
>
> 1. Totally execution percentage is around 90%, pass rate is around 98%
> 2. Created 2 new medium/low issue in RC4.
>
> Details:
> 1. Intel NIC PMD PF & VF regression: 95% executed, 0 critical issue and high issue found.
> Function cases include rx/tx/forward packets, packet processing, jumboframe, dual vlan, filter cases and so on: 2 bugs found, no critical/high bugs.
> Performance cases such as l2fwd, l3fwd, single core and so on: no new bug is found.
> 2. Cryptodev cases include ipsec_gw cryptodev, fips cryptodev, l2fwd crypto, performance cases and so on: no critical and no high bugs found.
> Function cases all passed, performance cases not fully executed.
> 3. Vhost/virtio: PVP/loopback, qemu/virtio-user, multi-queue, dequeue zero copy cases and so on: 1 new medium bug is found.
> 4. Power relate cases, no new bug is found.
> 5. ABI test includes unit test, vhost and flow related tests. no new bug is found.
> 6. 20.02 new features include ABI test, L2tpv3, crypto_dev related new features and so no. no new bug is found.
>
>
> > -----Original Message-----
> > From: dev <dev-bounces@dpdk.org> On Behalf Of Ali Alnubani
> > Sent: Tuesday, February 25, 2020 5:59 PM
> > To: Thomas Monjalon <thomas@monjalon.net>; dev@dpdk.org
> > Cc: Raslan Darawsheh <rasland@mellanox.com>; Asaf Penso
> > <asafp@mellanox.com>
> > Subject: Re: [dpdk-dev] [dpdk-announce] release candidate 20.02-rc4
> >
> > Hi,
> >
> > > -----Original Message-----
> > > From: announce <announce-bounces@dpdk.org> On Behalf Of Thomas
> > > Monjalon
> > > Sent: Saturday, February 22, 2020 11:08 PM
> > > To: announce@dpdk.org
> > > Subject: [dpdk-announce] release candidate 20.02-rc4
> > >
> > > A new DPDK release candidate is ready for testing:
> > <removed>
> > >
> > > This is the last release candidate for DPDK 20.02.
> > > Please share some release validation results by replying to this
> > > message (at dev@dpdk.org).
> > > If no critical issue is reported, 20.02 will be closed on Tuesday 25th.
> > >
> > > If you are preparing the next release cycle, please send your v1
> > > patches before the 20.05 proposal deadline, which will happen on March 18th.
> > > It is also time to build an estimated roadmap for the next cycles.
> > >
> > > Thank you everyone
> > >
> >
> > The following covers the tests that we ran on Mellanox hardware for this
> > version:
> >
> > - Basic functionality:
> > Send and receive multiple types of traffic.
> > - testpmd xstats counter test.
> > - testpmd timestamp test.
> > - Changing/checking link status through testpmd.
> > - RTE flow and flow_director tests.
> > Items: eth / vlan / ipv4 / ipv6 / tcp / udp / icmp / gre / nvgre / vxlan / geneve /
> > ip in ip / mplsoudp / mplsogre / gtp
> > Actions: drop / queue / rss / mark / flag / jump / count / port_id / raw_encap /
> > raw_decap / vxlan_encap / vxlan_decap / NAT
> > - Some RSS tests.
> > - VLAN stripping and insertion tests.
> > - Checksum and TSO tests.
> > - ptype tests.
> > - l3fwd-power example application tests.
> > - Multi-process example applications tests.
> >
> > We don't see any critical issues blocking the release.
> >
> > Thanks,
> > Ali
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] doc: deprecate using MAX values as array size
2020-02-21 10:25 0% ` Ferruh Yigit
2020-02-25 13:52 0% ` David Marchand
@ 2020-02-25 14:53 0% ` David Marchand
2 siblings, 0 replies; 200+ results
From: David Marchand @ 2020-02-25 14:53 UTC (permalink / raw)
To: Ferruh Yigit; +Cc: Neil Horman, John McNamara, Marko Kovacevic, dev
On Thu, Jan 30, 2020 at 3:20 PM Ferruh Yigit <ferruh.yigit@intel.com> wrote:
> Adding the deprecation notice as reminder for next ABI breakage release
> (20.11).
> This one time breakage is required to be able to extend enum/define
> without breaking ABI.
>
> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
Acked-by: Andrew Rybchenko <arybchenko@solarflare.com>
Acked-by: Thomas Monjalon <thomas@monjalon.net>
Acked-by: David Marchand <david.marchand@redhat.com>
Applied, thanks.
--
David Marchand
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [RFC 0/6] New sync modes for ring
2020-02-25 0:58 0% ` Honnappa Nagarahalli
@ 2020-02-25 15:14 0% ` Ananyev, Konstantin
0 siblings, 0 replies; 200+ results
From: Ananyev, Konstantin @ 2020-02-25 15:14 UTC (permalink / raw)
To: Honnappa Nagarahalli, Jerin Jacob, Stephen Hemminger
Cc: dpdk-dev, Olivier Matz, nd, nd, wang.yong19
> > > > Upfront note - that RFC is not a complete patch.
> > > > It introduces an ABI breakage, plus it doesn't update ring_elem code
> > > > properly, etc.
> > > > I plan to deal with all these things in later versions.
> > > > Right now I seek an initial feedback about proposed ideas.
> > > > Would also ask people to repeat performance tests (see below) on
> > > > their platforms to confirm the impact.
> > > >
> > > > More and more customers use(/try to use) DPDK based apps within
> > > > overcommitted systems (multiple acttive threads over same pysical cores):
> > > > VM, container deployments, etc.
> > > > One quite common problem they hit: Lock-Holder-Preemption with
> > rte_ring.
> > > > LHP is quite a common problem for spin-based sync primitives
> > > > (spin-locks, etc.) on overcommitted systems.
> > > > The situation gets much worse when some sort of fair-locking
> > > > technique is used (ticket-lock, etc.).
> > > > As now not only lock-owner but also lock-waiters scheduling order
> > > > matters a lot.
> > > > This is a well-known problem for kernel within VMs:
> > > > http://www-archive.xenproject.org/files/xensummitboston08/LHP.pdf
> > > > https://www.cs.hs-rm.de/~kaiser/events/wamos2017/Slides/selcuk.pdf
> These slides seem to indicate that the problems are mitigated through the Hypervisor configuration. Do we still need to address the issues?
I am not really an expert here, but AFAIK current mitigations deal mostly with guest kernel:
linux implements PV version of spinlocks (unfair and/or based on hypercall availability),
hypervisor might make decision itself based on is guest in user/kernel mode,
plus on some special cpu instructions.
We do spin in user-space mode.
Might be hypervisors became smarter these days, but so far,
I heard about few different customers that hit such problem.
As an example, NA DPDK summit presentation:
https://dpdkna2019.sched.com/event/WYBG/dpdk-containers-challenges-solutions-wang-yong-zte
page 16 (problem #4) describes same issue.
>
> > > > The problem with rte_ring is that while head accusion is sort of
> > > > un-fair locking, waiting on tail is very similar to ticket lock
> > > > schema - tail has to be updated in particular order.
> > > > That makes current rte_ring implementation to perform really pure on
> > > > some overcommited scenarios.
> > >
> > > Rather than reform rte_ring to fit this scenario, it would make more
> > > sense to me to introduce another primitive. The current lockless ring
> > > performs very well for the isolated thread model that DPDK was built
> > > around. This looks like a case of customers violating the usage model
> > > of the DPDK and then being surprised at the fallout.
> >
> > I agree with Stephen here.
> >
> > I think, adding more runtime check in the enqueue() and dequeue() will have a
> > bad effect on the low-end cores too.
> > But I agree with the problem statement that in the virtualization use case, It
> > may be possible to have N virtual cores runs on a physical core.
> It is hard to imagine that there are data plane applications deployed in such environments. Wouldn't this affect the performance terribly?
It wouldn't reach same performance as isolated threads,
but for some tasks it might be enough.
AFAIK, one quite common scenario - few isolated threads(/processes) doing
actual IO and then spread packets over dozens(/hundreds) non-isolated
consumers.
>
> >
> > IMO, The best solution would be keeping the ring API same and have a
> > different flavor in "compile-time". Something like liburcu did for
> > accommodating different flavors.
> >
> > i.e urcu-qsbr.h and urcu-bp.h will identical definition of API. The application
> > can simply include ONE header file in a C file based on the flavor.
> > If need both at runtime. Need to have function pointer or so in the application
> > and define the function in different c file by including the approaite flavor in C
> > file.
> >
> > #include <urcu-qsbr.h> /* QSBR RCU flavor */ #include <urcu-bp.h> /*
> > Bulletproof RCU flavor */
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [dpdk-announce] release candidate 20.02-rc4
2020-02-25 14:44 0% ` Thomas Monjalon
@ 2020-02-25 15:27 0% ` Raslan Darawsheh
0 siblings, 0 replies; 200+ results
From: Raslan Darawsheh @ 2020-02-25 15:27 UTC (permalink / raw)
To: Ali Alnubani, Xu, Qian Q, Thomas Monjalon
Cc: dev, Asaf Penso, Peng, Yuan, Yu, PingX
Hi,
It's Ok from our side and we can release.
Kindest regards
Raslan Darawsheh
Kindest regards
Raslan Darawsheh
________________________________
From: Thomas Monjalon <thomas@monjalon.net>
Sent: Tuesday, February 25, 2020 4:44:26 PM
To: Ali Alnubani <alialnu@mellanox.com>; Xu, Qian Q <qian.q.xu@intel.com>; Raslan Darawsheh <rasland@mellanox.com>
Cc: dev@dpdk.org <dev@dpdk.org>; Asaf Penso <asafp@mellanox.com>; Peng, Yuan <yuan.peng@intel.com>; Yu, PingX <pingx.yu@intel.com>
Subject: Re: [dpdk-announce] release candidate 20.02-rc4
Can we assume we are OK to release today?
25/02/2020 15:37, Xu, Qian Q:
> Some quick update from Intel on RC4 testing
>
> Summary
>
> 1. Totally execution percentage is around 90%, pass rate is around 98%
> 2. Created 2 new medium/low issue in RC4.
>
> Details:
> 1. Intel NIC PMD PF & VF regression: 95% executed, 0 critical issue and high issue found.
> Function cases include rx/tx/forward packets, packet processing, jumboframe, dual vlan, filter cases and so on: 2 bugs found, no critical/high bugs.
> Performance cases such as l2fwd, l3fwd, single core and so on: no new bug is found.
> 2. Cryptodev cases include ipsec_gw cryptodev, fips cryptodev, l2fwd crypto, performance cases and so on: no critical and no high bugs found.
> Function cases all passed, performance cases not fully executed.
> 3. Vhost/virtio: PVP/loopback, qemu/virtio-user, multi-queue, dequeue zero copy cases and so on: 1 new medium bug is found.
> 4. Power relate cases, no new bug is found.
> 5. ABI test includes unit test, vhost and flow related tests. no new bug is found.
> 6. 20.02 new features include ABI test, L2tpv3, crypto_dev related new features and so no. no new bug is found.
>
>
> > -----Original Message-----
> > From: dev <dev-bounces@dpdk.org> On Behalf Of Ali Alnubani
> > Sent: Tuesday, February 25, 2020 5:59 PM
> > To: Thomas Monjalon <thomas@monjalon.net>; dev@dpdk.org
> > Cc: Raslan Darawsheh <rasland@mellanox.com>; Asaf Penso
> > <asafp@mellanox.com>
> > Subject: Re: [dpdk-dev] [dpdk-announce] release candidate 20.02-rc4
> >
> > Hi,
> >
> > > -----Original Message-----
> > > From: announce <announce-bounces@dpdk.org> On Behalf Of Thomas
> > > Monjalon
> > > Sent: Saturday, February 22, 2020 11:08 PM
> > > To: announce@dpdk.org
> > > Subject: [dpdk-announce] release candidate 20.02-rc4
> > >
> > > A new DPDK release candidate is ready for testing:
> > <removed>
> > >
> > > This is the last release candidate for DPDK 20.02.
> > > Please share some release validation results by replying to this
> > > message (at dev@dpdk.org).
> > > If no critical issue is reported, 20.02 will be closed on Tuesday 25th.
> > >
> > > If you are preparing the next release cycle, please send your v1
> > > patches before the 20.05 proposal deadline, which will happen on March 18th.
> > > It is also time to build an estimated roadmap for the next cycles.
> > >
> > > Thank you everyone
> > >
> >
> > The following covers the tests that we ran on Mellanox hardware for this
> > version:
> >
> > - Basic functionality:
> > Send and receive multiple types of traffic.
> > - testpmd xstats counter test.
> > - testpmd timestamp test.
> > - Changing/checking link status through testpmd.
> > - RTE flow and flow_director tests.
> > Items: eth / vlan / ipv4 / ipv6 / tcp / udp / icmp / gre / nvgre / vxlan / geneve /
> > ip in ip / mplsoudp / mplsogre / gtp
> > Actions: drop / queue / rss / mark / flag / jump / count / port_id / raw_encap /
> > raw_decap / vxlan_encap / vxlan_decap / NAT
> > - Some RSS tests.
> > - VLAN stripping and insertion tests.
> > - Checksum and TSO tests.
> > - ptype tests.
> > - l3fwd-power example application tests.
> > - Multi-process example applications tests.
> >
> > We don't see any critical issues blocking the release.
> >
> > Thanks,
> > Ali
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] doc: alias to experimental tag for stable apis
2020-02-18 10:46 17% ` [dpdk-dev] [PATCH v4] " Ray Kinsella
@ 2020-02-25 15:35 0% ` Neil Horman
2020-02-25 17:57 0% ` David Marchand
1 sibling, 1 reply; 200+ results
From: Neil Horman @ 2020-02-25 15:35 UTC (permalink / raw)
To: Ray Kinsella; +Cc: dev, john.mcnamara, ferruh.yigit, marko.kovacevic
On Wed, Feb 05, 2020 at 03:17:52PM +0000, Ray Kinsella wrote:
> When a maintainer is promoting an API to become part of the next major ABI
> version by removing the experimental tag, possibly a few releases in advance of
> the declaration of the next ABI version. The maintainer may choose to offer an
> alias to the experimental tag, as removing the tag before the declaration of the
> next major ABI version, would cause an ABI breakage for applications using the
> API.
>
> Signed-off-by: Ray Kinsella <mdr@ashroe.eu>
> ---
> doc/guides/contributing/abi_policy.rst | 10 ++++++++++
> 1 file changed, 10 insertions(+)
>
> diff --git a/doc/guides/contributing/abi_policy.rst b/doc/guides/contributing/abi_policy.rst
> index 05ca959..9a4a102 100644
> --- a/doc/guides/contributing/abi_policy.rst
> +++ b/doc/guides/contributing/abi_policy.rst
> @@ -159,6 +159,11 @@ The requirements for changing the ABI are:
> ``experimental``, as described in the section on :ref:`Experimental APIs
> and Libraries <experimental_apis>`.
>
> + - In situations where an ``experimental`` API has been stable for some time.
> + When promoting the API to become part of the next ABI version, the
> + maintainer may choose to provide an alias to the ``experimental`` tag, so
> + as not to break consuming applications.
> +
I don't have any issue with the approach, but just to ask the question, is it
worth providing an example here, of how exactly to do this? The use of
VERSION_SYMBOL isn't often used, and so may be non-obvious.
Actually, as I look at it, the VERSION_SYMBOL macro assume a DPDK_ prefix on the
version string, which works for versioned symbols, but not for the EXPERIMENTAL
symbol version (no DPDK_ prefix). We should probably create a variant of the
VERSION_SYMBOL macro to allow aliasing to the EXPERIMENTAL version, something
like ALIAS_TO_EXPERIMENTAL() or some such
Neil
> #. If a newly proposed API functionally replaces an existing one, when the new
> API becomes non-experimental, then the old one is marked with
> ``__rte_deprecated``.
> @@ -317,6 +322,11 @@ not required. Though, an API should remain in experimental state for at least
> one release. Thereafter, the normal process of posting patch for review to
> mailing list can be followed.
>
> +After the experimental tag has been formally removed, a tree/sub-tree maintainer
> +may choose to offer an alias to the experimental tag so as not to break
> +applications using the API. This alias can then be dropped at the declaration of
> +next major ABI version.
> +
> Libraries
> ~~~~~~~~~
>
> --
> 2.7.4
>
>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v2] doc: plan splitting the ethdev ops struct
2020-02-25 12:44 3% ` [dpdk-dev] [PATCH v2] " Ferruh Yigit
@ 2020-02-25 15:51 3% ` Andrew Rybchenko
2020-02-25 16:13 3% ` Ferruh Yigit
2020-02-25 18:13 0% ` David Marchand
2020-03-04 9:57 4% ` [dpdk-dev] [PATCH v3] " Ferruh Yigit
2 siblings, 1 reply; 200+ results
From: Andrew Rybchenko @ 2020-02-25 15:51 UTC (permalink / raw)
To: Ferruh Yigit, Neil Horman, John McNamara, Marko Kovacevic
Cc: dev, Jerin Jacob, David Marchand, Thomas Monjalon
On 2/25/20 3:44 PM, Ferruh Yigit wrote:
> For the ABI compatibility it is better to hide internal data structures
> from the application as much as possible. But because of some inline
> functions 'struct eth_dev_ops' can't be hidden completely.
>
> Plan is to split the 'struct eth_dev_ops' into two as ones used by
> inline functions and ones not used, and hide the second part that not
> used by inline functions completely to the application.
>
> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
> Acked-by: Jerin Jacob <jerinj@marvell.com>
> ---
> Cc: David Marchand <david.marchand@redhat.com>
> Cc: Thomas Monjalon <thomas@monjalon.net>
> Cc: Andrew Rybchenko <arybchenko@solarflare.com>
>
> v2:
> * Add target date for the work
> * Give more detail on what will be done
> ---
> doc/guides/rel_notes/deprecation.rst | 11 +++++++++++
> 1 file changed, 11 insertions(+)
>
> diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
> index 99d81564a..ff612a615 100644
> --- a/doc/guides/rel_notes/deprecation.rst
> +++ b/doc/guides/rel_notes/deprecation.rst
> @@ -86,6 +86,17 @@ Deprecation Notices
> In 19.11 PMDs will still update the field even when the offload is not
> enabled.
>
> +* ethdev: Split the ``struct eth_dev_ops`` struct to hide it as much as possible.
> + Currently the ``struct eth_dev_ops`` struct is accessible by the application
> + because some inline functions, like ``rte_eth_tx_descriptor_status()``,
> + access the struct directly. The struct will be separate in two, the ops used
> + by inline functions still will be accessible to user but rest will be hidden.
> + Initial split will be done in 20.05 with adding reserved fields for the struct
> + used by inline functions, and by putting new struct reference into public one
> + to not increase the size of ``struct rte_eth_dev``, proper split will be done
> + in 20.11 by moving inline dev_ops function to next to Rx/Tx burst functions and
> + hiding rest.
> +
> * cryptodev: support for using IV with all sizes is added, J0 still can
> be used but only when IV length in following structs ``rte_crypto_auth_xform``,
> ``rte_crypto_aead_xform`` is set to zero. When IV length is greater or equal
I'd like to understand why do we need an intermediate solution first.
Also rereading above few times I've failed to fully understand what
will be done and why it does not break API/ABI.
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH v2] doc: plan splitting the ethdev ops struct
2020-02-25 15:51 3% ` Andrew Rybchenko
@ 2020-02-25 16:13 3% ` Ferruh Yigit
2020-02-25 16:41 0% ` Andrew Rybchenko
0 siblings, 1 reply; 200+ results
From: Ferruh Yigit @ 2020-02-25 16:13 UTC (permalink / raw)
To: Andrew Rybchenko, Neil Horman, John McNamara, Marko Kovacevic
Cc: dev, Jerin Jacob, David Marchand, Thomas Monjalon
On 2/25/2020 3:51 PM, Andrew Rybchenko wrote:
> On 2/25/20 3:44 PM, Ferruh Yigit wrote:
>> For the ABI compatibility it is better to hide internal data structures
>> from the application as much as possible. But because of some inline
>> functions 'struct eth_dev_ops' can't be hidden completely.
>>
>> Plan is to split the 'struct eth_dev_ops' into two as ones used by
>> inline functions and ones not used, and hide the second part that not
>> used by inline functions completely to the application.
>>
>> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
>> Acked-by: Jerin Jacob <jerinj@marvell.com>
>> ---
>> Cc: David Marchand <david.marchand@redhat.com>
>> Cc: Thomas Monjalon <thomas@monjalon.net>
>> Cc: Andrew Rybchenko <arybchenko@solarflare.com>
>>
>> v2:
>> * Add target date for the work
>> * Give more detail on what will be done
>> ---
>> doc/guides/rel_notes/deprecation.rst | 11 +++++++++++
>> 1 file changed, 11 insertions(+)
>>
>> diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
>> index 99d81564a..ff612a615 100644
>> --- a/doc/guides/rel_notes/deprecation.rst
>> +++ b/doc/guides/rel_notes/deprecation.rst
>> @@ -86,6 +86,17 @@ Deprecation Notices
>> In 19.11 PMDs will still update the field even when the offload is not
>> enabled.
>>
>> +* ethdev: Split the ``struct eth_dev_ops`` struct to hide it as much as possible.
>> + Currently the ``struct eth_dev_ops`` struct is accessible by the application
>> + because some inline functions, like ``rte_eth_tx_descriptor_status()``,
>> + access the struct directly. The struct will be separate in two, the ops used
>> + by inline functions still will be accessible to user but rest will be hidden.
>> + Initial split will be done in 20.05 with adding reserved fields for the struct
>> + used by inline functions, and by putting new struct reference into public one
>> + to not increase the size of ``struct rte_eth_dev``, proper split will be done
>> + in 20.11 by moving inline dev_ops function to next to Rx/Tx burst functions and
>> + hiding rest.
>> +
>> * cryptodev: support for using IV with all sizes is added, J0 still can
>> be used but only when IV length in following structs ``rte_crypto_auth_xform``,
>> ``rte_crypto_aead_xform`` is set to zero. When IV length is greater or equal
>
> I'd like to understand why do we need an intermediate solution first.
We don't need it really, we may prefer to postpone the update to 20.11.
> Also rereading above few times I've failed to fully understand what
> will be done and why it does not break API/ABI.
>
Agree it wasn't very clear, but I mean something like below, I hope it clarifies:
Previously:
struct rte_eth_dev
+--------+
| | *dev_ops struct eth_dev_ops
| |
| +-------------->+--------+
| | | |
| | | |
| | | |
| | | |
+--------+ | |
| |
| |
| |
+--------+
Proposed:
struct rte_eth_dev
+--------+
| | struct eth_dev_ops
| |
|*dev_ops+-------------->+--------+
| | | |
| | | Reserv |
| | | |
| | | |
+--------+ | |
+--------+
| inline |
| de^_ops| struct eth_dev_ops
+--------+
| *priv +------------->+--------+
+--------+ | |
| |
| |
| |
| |
| |
| |
| |
+--------+
This is only to keep ABI compatibility [1] while separating the struct.
[1]
- The offset of some functions in the dev_ops struct should be same
- The size of the "struct rte_eth_dev" should be same
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH v2] doc: plan splitting the ethdev ops struct
2020-02-25 16:13 3% ` Ferruh Yigit
@ 2020-02-25 16:41 0% ` Andrew Rybchenko
0 siblings, 0 replies; 200+ results
From: Andrew Rybchenko @ 2020-02-25 16:41 UTC (permalink / raw)
To: Ferruh Yigit, Neil Horman, John McNamara, Marko Kovacevic
Cc: dev, Jerin Jacob, David Marchand, Thomas Monjalon
On 2/25/20 7:13 PM, Ferruh Yigit wrote:
> On 2/25/2020 3:51 PM, Andrew Rybchenko wrote:
>> On 2/25/20 3:44 PM, Ferruh Yigit wrote:
>>> For the ABI compatibility it is better to hide internal data structures
>>> from the application as much as possible. But because of some inline
>>> functions 'struct eth_dev_ops' can't be hidden completely.
>>>
>>> Plan is to split the 'struct eth_dev_ops' into two as ones used by
>>> inline functions and ones not used, and hide the second part that not
>>> used by inline functions completely to the application.
>>>
>>> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
>>> Acked-by: Jerin Jacob <jerinj@marvell.com>
>>> ---
>>> Cc: David Marchand <david.marchand@redhat.com>
>>> Cc: Thomas Monjalon <thomas@monjalon.net>
>>> Cc: Andrew Rybchenko <arybchenko@solarflare.com>
>>>
>>> v2:
>>> * Add target date for the work
>>> * Give more detail on what will be done
>>> ---
>>> doc/guides/rel_notes/deprecation.rst | 11 +++++++++++
>>> 1 file changed, 11 insertions(+)
>>>
>>> diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
>>> index 99d81564a..ff612a615 100644
>>> --- a/doc/guides/rel_notes/deprecation.rst
>>> +++ b/doc/guides/rel_notes/deprecation.rst
>>> @@ -86,6 +86,17 @@ Deprecation Notices
>>> In 19.11 PMDs will still update the field even when the offload is not
>>> enabled.
>>>
>>> +* ethdev: Split the ``struct eth_dev_ops`` struct to hide it as much as possible.
>>> + Currently the ``struct eth_dev_ops`` struct is accessible by the application
>>> + because some inline functions, like ``rte_eth_tx_descriptor_status()``,
>>> + access the struct directly. The struct will be separate in two, the ops used
>>> + by inline functions still will be accessible to user but rest will be hidden.
>>> + Initial split will be done in 20.05 with adding reserved fields for the struct
>>> + used by inline functions, and by putting new struct reference into public one
>>> + to not increase the size of ``struct rte_eth_dev``, proper split will be done
>>> + in 20.11 by moving inline dev_ops function to next to Rx/Tx burst functions and
>>> + hiding rest.
>>> +
>>> * cryptodev: support for using IV with all sizes is added, J0 still can
>>> be used but only when IV length in following structs ``rte_crypto_auth_xform``,
>>> ``rte_crypto_aead_xform`` is set to zero. When IV length is greater or equal
>>
>> I'd like to understand why do we need an intermediate solution first.
>
> We don't need it really, we may prefer to postpone the update to 20.11.
>
>> Also rereading above few times I've failed to fully understand what
>> will be done and why it does not break API/ABI.
>>
>
> Agree it wasn't very clear, but I mean something like below, I hope it clarifies:
>
> Previously:
>
> struct rte_eth_dev
>
> +--------+
> | | *dev_ops struct eth_dev_ops
> | |
> | +-------------->+--------+
> | | | |
> | | | |
> | | | |
> | | | |
> +--------+ | |
> | |
> | |
> | |
> +--------+
>
>
> Proposed:
>
> struct rte_eth_dev
>
> +--------+
> | | struct eth_dev_ops
> | |
> |*dev_ops+-------------->+--------+
> | | | |
> | | | Reserv |
> | | | |
> | | | |
> +--------+ | |
> +--------+
> | inline |
> | de^_ops| struct eth_dev_ops
> +--------+
> | *priv +------------->+--------+
> +--------+ | |
> | |
> | |
> | |
> | |
> | |
> | |
> | |
> +--------+
>
> This is only to keep ABI compatibility [1] while separating the struct.
>
> [1]
> - The offset of some functions in the dev_ops struct should be same
> - The size of the "struct rte_eth_dev" should be same
>
OK, got it. Many thanks for explanations.
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] doc: alias to experimental tag for stable apis
2020-02-25 15:35 0% ` [dpdk-dev] [PATCH] " Neil Horman
@ 2020-02-25 17:57 0% ` David Marchand
0 siblings, 0 replies; 200+ results
From: David Marchand @ 2020-02-25 17:57 UTC (permalink / raw)
To: Ray Kinsella, Neil Horman
Cc: dev, Mcnamara, John, Yigit, Ferruh, Kovacevic, Marko
On Tue, Feb 25, 2020 at 4:36 PM Neil Horman <nhorman@tuxdriver.com> wrote:
>
> On Wed, Feb 05, 2020 at 03:17:52PM +0000, Ray Kinsella wrote:
> > When a maintainer is promoting an API to become part of the next major ABI
> > version by removing the experimental tag, possibly a few releases in advance of
> > the declaration of the next ABI version. The maintainer may choose to offer an
> > alias to the experimental tag, as removing the tag before the declaration of the
> > next major ABI version, would cause an ABI breakage for applications using the
> > API.
> >
> > Signed-off-by: Ray Kinsella <mdr@ashroe.eu>
> > ---
> > doc/guides/contributing/abi_policy.rst | 10 ++++++++++
> > 1 file changed, 10 insertions(+)
> >
> > diff --git a/doc/guides/contributing/abi_policy.rst b/doc/guides/contributing/abi_policy.rst
> > index 05ca959..9a4a102 100644
> > --- a/doc/guides/contributing/abi_policy.rst
> > +++ b/doc/guides/contributing/abi_policy.rst
> > @@ -159,6 +159,11 @@ The requirements for changing the ABI are:
> > ``experimental``, as described in the section on :ref:`Experimental APIs
> > and Libraries <experimental_apis>`.
> >
> > + - In situations where an ``experimental`` API has been stable for some time.
> > + When promoting the API to become part of the next ABI version, the
> > + maintainer may choose to provide an alias to the ``experimental`` tag, so
> > + as not to break consuming applications.
> > +
> I don't have any issue with the approach, but just to ask the question, is it
> worth providing an example here, of how exactly to do this? The use of
> VERSION_SYMBOL isn't often used, and so may be non-obvious.
>
> Actually, as I look at it, the VERSION_SYMBOL macro assume a DPDK_ prefix on the
> version string, which works for versioned symbols, but not for the EXPERIMENTAL
> symbol version (no DPDK_ prefix). We should probably create a variant of the
> VERSION_SYMBOL macro to allow aliasing to the EXPERIMENTAL version, something
> like ALIAS_TO_EXPERIMENTAL() or some such
Let's clarify this part so this can be merged in early 20.05.
Thanks.
--
David Marchand
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v2] doc: plan splitting the ethdev ops struct
2020-02-25 12:44 3% ` [dpdk-dev] [PATCH v2] " Ferruh Yigit
2020-02-25 15:51 3% ` Andrew Rybchenko
@ 2020-02-25 18:13 0% ` David Marchand
2020-02-25 18:18 0% ` Ferruh Yigit
2020-03-04 9:57 4% ` [dpdk-dev] [PATCH v3] " Ferruh Yigit
2 siblings, 1 reply; 200+ results
From: David Marchand @ 2020-02-25 18:13 UTC (permalink / raw)
To: Ferruh Yigit
Cc: Neil Horman, John McNamara, Marko Kovacevic, dev, Jerin Jacob,
Thomas Monjalon, Andrew Rybchenko
On Tue, Feb 25, 2020 at 1:44 PM Ferruh Yigit <ferruh.yigit@intel.com> wrote:
>
> For the ABI compatibility it is better to hide internal data structures
> from the application as much as possible. But because of some inline
> functions 'struct eth_dev_ops' can't be hidden completely.
>
> Plan is to split the 'struct eth_dev_ops' into two as ones used by
> inline functions and ones not used, and hide the second part that not
> used by inline functions completely to the application.
>
> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
> Acked-by: Jerin Jacob <jerinj@marvell.com>
Sorry, this patch is missing the necessary acks for merging.
This change can still be announced for 20.11 in the next releases and
the details can be discussed again since a conclusion was not reached.
Thanks.
--
David Marchand
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v2] doc: plan splitting the ethdev ops struct
2020-02-25 18:13 0% ` David Marchand
@ 2020-02-25 18:18 0% ` Ferruh Yigit
0 siblings, 0 replies; 200+ results
From: Ferruh Yigit @ 2020-02-25 18:18 UTC (permalink / raw)
To: David Marchand
Cc: Neil Horman, John McNamara, Marko Kovacevic, dev, Jerin Jacob,
Thomas Monjalon, Andrew Rybchenko
On 2/25/2020 6:13 PM, David Marchand wrote:
> On Tue, Feb 25, 2020 at 1:44 PM Ferruh Yigit <ferruh.yigit@intel.com> wrote:
>>
>> For the ABI compatibility it is better to hide internal data structures
>> from the application as much as possible. But because of some inline
>> functions 'struct eth_dev_ops' can't be hidden completely.
>>
>> Plan is to split the 'struct eth_dev_ops' into two as ones used by
>> inline functions and ones not used, and hide the second part that not
>> used by inline functions completely to the application.
>>
>> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
>> Acked-by: Jerin Jacob <jerinj@marvell.com>
>
> Sorry, this patch is missing the necessary acks for merging.
> This change can still be announced for 20.11 in the next releases and
> the details can be discussed again since a conclusion was not reached.
>
>
That is OK, we can drop for 20.02, I can send a new version to target the 20.11
change only removing interim change.
^ permalink raw reply [relevance 0%]
* [dpdk-dev] [dpdk-announce] DPDK 20.02 released
@ 2020-02-25 22:59 4% Thomas Monjalon
0 siblings, 0 replies; 200+ results
From: Thomas Monjalon @ 2020-02-25 22:59 UTC (permalink / raw)
To: announce
A new release is available:
https://fast.dpdk.org/rel/dpdk-20.02.tar.xz
The statistics looks like a post-LTS ABI-compatible release:
820 commits from 156 authors
876 files changed, 59941 insertions(+), 22565 deletions(-)
It is not planned (yet) to start a maintenance branch for 20.02.
This release and next ones (20.05, 20.08) are ABI-compatible with 19.11.
Below are some new features, grouped by category.
General:
- ABI check tooling
- FreeBSD 13 support
- Linux kernel modules disabled in default build
- API wait until equal
- use CPU lcore with ID > total max
- ring with configurable element size
- pool of mbufs with pinned external buffers
Networking:
- Pensando ionic driver
- Marvell driver for OCTEON TX2 end point
- Mellanox vDPA driver
- rte_flow API for DSCP
- rte_flow API for L2TPv3 over IP
Cryptography:
- Marvell OCTEON TX2 inline IPsec
- CPU crypto based on new libraries (intel-ipsec-mb and AArch64cryptolib)
- synchronous CPU crypto API
- more algorithms: ECDSA, ECPM
Applications:
- event mode in l3fwd example
More details in the release notes:
http://doc.dpdk.org/guides/rel_notes/release_20_02.html
There are 69 new contributors (including authors, reviewers and testers).
Welcome to Abed Kamaluddin, Abhishek Marathe, Adam Ludkiewicz,
Adrian Podlawski, Aleksandr Loktionov, Alexander Kozyrev,
Alfredo Cardigliano, Andrew Pinski, Apeksha Gupta,
Balakrishna Bhamidipati, Carolyn Wyborny, Chandu Babu N,
Cheng Jiang, Chengchang Tang, Damian Milosek, Dariusz Chaberski,
Dariusz Jagus, Dexuan Cui, Dmitry Kozlyuk, Donald Lee, Dzmitry Sautsa,
Eugenio Pérez, Fang TongHao, Gal Cohen, Gargi Sau, Girish Nandibasappa,
Ilja Van Sprundel, Itsuro Oda, Jaroslaw Gawin, Jörg Thalheim,
Kiran Patil, Krzysztof Galazka, Kumar Amber, Li Feng, Lijun Ou,
Mahipal Challa, Manish Chopra, Marcin Formela, Martyna Szapar,
Mateusz Rusinski, Michael Baum, Michal Litwicki, Michal Swiatkowski,
Narcisa Vasile, Niclas Storm, Pandi Kumar Maharajan, Piotr Azarewicz,
Piotr Kwapulinski, Piotr Pietruszewski, Prateek Agarwal, Praveen Shetty,
Rafael Ávila de Espíndola, Ricardo Roldan, Robert Konklewski,
Satananda Burla, Savinay Dharmappa, Scott Wasson, Selwin Sebastian,
Shannon Nelson, Shiri Kuzin, Sunil Pai G, Sylwia Wnuczko, Thomas Faivre,
Tomasz Konieczny, Vitaliy Mysak, Xuan Ding, Xuan Li, Xueming Zhang,
and Yisen Zhuang.
Below is the number of patches per company (with authors count):
268 Intel (54)
193 Mellanox (16)
114 Marvell (23)
49 Broadcom (7)
34 Red Hat (7)
32 Huawei (8)
30 ARM (4)
25 Microsoft (1)
18 ntop (1)
9 6WIND (5)
6 NXP (4)
5 AMD (3)
4 VA Linux Systems (1)
4 Ericsson (1)
4 Cisco (1)
Based on Reviewed-by and Acked-by tags, the top reviewers are:
76 Viacheslav Ovsiienko <viacheslavo@mellanox.com>
56 Qi Zhang <qi.z.zhang@intel.com>
47 Xiaolong Ye <xiaolong.ye@intel.com>
46 Akhil Goyal <akhil.goyal@nxp.com>
44 Ferruh Yigit <ferruh.yigit@intel.com>
43 Matan Azrad <matan@mellanox.com>
43 Jerin Jacob <jerinj@marvell.com>
38 Beilei Xing <beilei.xing@intel.com>
35 Qiming Yang <qiming.yang@intel.com>
34 Ajit Khaparde <ajit.khaparde@broadcom.com>
33 Maxime Coquelin <maxime.coquelin@redhat.com>
31 Ori Kam <orika@mellanox.com>
25 Gavin Hu <gavin.hu@arm.com>
24 David Marchand <david.marchand@redhat.com>
23 Konstantin Ananyev <konstantin.ananyev@intel.com>
The new features for 20.05 may be submitted during the next 21 days,
in order to be reviewed and integrated before mid-April.
DPDK 20.05 should be released on 20-05-20.
http://core.dpdk.org/roadmap#dates
The schedule is pushed because of the special worldwide context.
Some future release cycles will have to be shorter.
Thanks everyone, and take care.
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] Questions about rte_timer APIs
@ 2020-02-25 23:56 3% ` Honnappa Nagarahalli
2020-02-26 16:57 0% ` Carrillo, Erik G
0 siblings, 1 reply; 200+ results
From: Honnappa Nagarahalli @ 2020-02-25 23:56 UTC (permalink / raw)
To: Honnappa Nagarahalli, Robert Sanford, Erik Gabriel Carrillo
Cc: dev, nd, Phil Yang, Gavin Hu, david.marchand, thomas, nd
Hi Erik,
I see that the rte_timer_alt_xxx APIs are still marked as experimental. So, we should be able to change them without any ABI constraints. Please let me know what you think.
Thank you,
Honnappa
> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Honnappa Nagarahalli
> Sent: Thursday, February 13, 2020 11:55 PM
> To: Robert Sanford <rsanford@akamai.com>; Erik Gabriel Carrillo
> <erik.g.carrillo@intel.com>
> Cc: dev <dev@dpdk.org>; nd <nd@arm.com>; Phil Yang
> <Phil.Yang@arm.com>; Gavin Hu <Gavin.Hu@arm.com>; nd <nd@arm.com>
> Subject: [dpdk-dev] Questions about rte_timer APIs
>
> Hi Erik/Robert,
> I was looking at the rte_timer library code and have questions around
> the way instances of timer list are allocated.
>
> The API 'rte_timer_subsystem_init' allocates RTE_MAX_DATA_ELS (64)
> number of timer data structs. Each timer data struct acts as an independent
> instance of timer list. The user allocates an instance by calling
> 'rte_timer_data_alloc' which provides an index in the instance array.
> Essentially, the library is allocating the memory upfront even though there
> might not be a need to have 64 instances. Please correct me if I am wrong.
>
> Usually, creating multiple instances is handled by allocating required memory,
> as and when required, by an 'init' API. This API also returns a pointer to that
> instance which is passed to other APIs in the library. For ex: rte_hash library
> follows this approach. IMO, this is an elegant way to handle multiple instances.
> This approach does not waste memory and does not put any restriction on
> number of instances.
>
> I wanted to understand the reasoning behind the current design to handle
> multiple instances. Appreciate your inputs.
>
> Thank you,
> Honnappa
^ permalink raw reply [relevance 3%]
* [dpdk-dev] [PATCH] version: 20.05-rc0
@ 2020-02-26 13:38 10% David Marchand
2020-02-26 13:55 0% ` Thomas Monjalon
0 siblings, 1 reply; 200+ results
From: David Marchand @ 2020-02-26 13:38 UTC (permalink / raw)
To: dev; +Cc: thomas
Start a new release cycle with empty release notes.
ABI must now be checked with v20.02 as a reference.
Signed-off-by: David Marchand <david.marchand@redhat.com>
---
.travis.yml | 4 +
ABI_VERSION | 2 +-
VERSION | 2 +-
doc/guides/rel_notes/index.rst | 1 +
doc/guides/rel_notes/release_20_05.rst | 139 +++++++++++++++++++++++++
5 files changed, 146 insertions(+), 2 deletions(-)
create mode 100644 doc/guides/rel_notes/release_20_05.rst
diff --git a/.travis.yml b/.travis.yml
index b64a81bd0..a57790ba8 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -34,6 +34,10 @@ _doc_packages: &doc_packages
before_install: ./.ci/${TRAVIS_OS_NAME}-setup.sh
script: ./.ci/${TRAVIS_OS_NAME}-build.sh
+env:
+ global:
+ - REF_GIT_TAG=v20.02
+
jobs:
include:
# x86_64 gcc jobs
diff --git a/ABI_VERSION b/ABI_VERSION
index fcc01369a..204da679a 100644
--- a/ABI_VERSION
+++ b/ABI_VERSION
@@ -1 +1 @@
-20.0.1
+20.0.2
diff --git a/VERSION b/VERSION
index d07c5a629..9791dd8af 100644
--- a/VERSION
+++ b/VERSION
@@ -1 +1 @@
-20.02.0
+20.05.0-rc0
diff --git a/doc/guides/rel_notes/index.rst b/doc/guides/rel_notes/index.rst
index 24d927d33..31278d2a8 100644
--- a/doc/guides/rel_notes/index.rst
+++ b/doc/guides/rel_notes/index.rst
@@ -8,6 +8,7 @@ Release Notes
:maxdepth: 1
:numbered:
+ release_20_05
release_20_02
release_19_11
release_19_08
diff --git a/doc/guides/rel_notes/release_20_05.rst b/doc/guides/rel_notes/release_20_05.rst
new file mode 100644
index 000000000..2190eaf85
--- /dev/null
+++ b/doc/guides/rel_notes/release_20_05.rst
@@ -0,0 +1,139 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+ Copyright 2020 The DPDK contributors
+
+.. include:: <isonum.txt>
+
+DPDK Release 20.05
+==================
+
+.. **Read this first.**
+
+ The text in the sections below explains how to update the release notes.
+
+ Use proper spelling, capitalization and punctuation in all sections.
+
+ Variable and config names should be quoted as fixed width text:
+ ``LIKE_THIS``.
+
+ Build the docs and view the output file to ensure the changes are correct::
+
+ make doc-guides-html
+
+ xdg-open build/doc/html/guides/rel_notes/release_20_05.html
+
+
+New Features
+------------
+
+.. This section should contain new features added in this release.
+ Sample format:
+
+ * **Add a title in the past tense with a full stop.**
+
+ Add a short 1-2 sentence description in the past tense.
+ The description should be enough to allow someone scanning
+ the release notes to understand the new feature.
+
+ If the feature adds a lot of sub-features you can use a bullet list
+ like this:
+
+ * Added feature foo to do something.
+ * Enhanced feature bar to do something else.
+
+ Refer to the previous release notes for examples.
+
+ Suggested order in release notes items:
+ * Core libs (EAL, mempool, ring, mbuf, buses)
+ * Device abstraction libs and PMDs
+ - ethdev (lib, PMDs)
+ - cryptodev (lib, PMDs)
+ - eventdev (lib, PMDs)
+ - etc
+ * Other libs
+ * Apps, Examples, Tools (if significant)
+
+ This section is a comment. Do not overwrite or remove it.
+ Also, make sure to start the actual text at the margin.
+ =========================================================
+
+
+Removed Items
+-------------
+
+.. This section should contain removed items in this release. Sample format:
+
+ * Add a short 1-2 sentence description of the removed item
+ in the past tense.
+
+ This section is a comment. Do not overwrite or remove it.
+ Also, make sure to start the actual text at the margin.
+ =========================================================
+
+
+API Changes
+-----------
+
+.. This section should contain API changes. Sample format:
+
+ * sample: Add a short 1-2 sentence description of the API change
+ which was announced in the previous releases and made in this release.
+ Start with a scope label like "ethdev:".
+ Use fixed width quotes for ``function_names`` or ``struct_names``.
+ Use the past tense.
+
+ This section is a comment. Do not overwrite or remove it.
+ Also, make sure to start the actual text at the margin.
+ =========================================================
+
+
+ABI Changes
+-----------
+
+.. This section should contain ABI changes. Sample format:
+
+ * sample: Add a short 1-2 sentence description of the ABI change
+ which was announced in the previous releases and made in this release.
+ Start with a scope label like "ethdev:".
+ Use fixed width quotes for ``function_names`` or ``struct_names``.
+ Use the past tense.
+
+ This section is a comment. Do not overwrite or remove it.
+ Also, make sure to start the actual text at the margin.
+ =========================================================
+
+* No ABI change that would break compatibility with DPDK 20.02 and 19.11.
+
+
+Known Issues
+------------
+
+.. This section should contain new known issues in this release. Sample format:
+
+ * **Add title in present tense with full stop.**
+
+ Add a short 1-2 sentence description of the known issue
+ in the present tense. Add information on any known workarounds.
+
+ This section is a comment. Do not overwrite or remove it.
+ Also, make sure to start the actual text at the margin.
+ =========================================================
+
+
+Tested Platforms
+----------------
+
+.. This section should contain a list of platforms that were tested
+ with this release.
+
+ The format is:
+
+ * <vendor> platform with <vendor> <type of devices> combinations
+
+ * List of CPU
+ * List of OS
+ * List of devices
+ * Other relevant details...
+
+ This section is a comment. Do not overwrite or remove it.
+ Also, make sure to start the actual text at the margin.
+ =========================================================
--
2.23.0
^ permalink raw reply [relevance 10%]
* Re: [dpdk-dev] [PATCH] version: 20.05-rc0
2020-02-26 13:38 10% [dpdk-dev] [PATCH] version: 20.05-rc0 David Marchand
@ 2020-02-26 13:55 0% ` Thomas Monjalon
2020-02-26 14:01 0% ` David Marchand
2020-02-27 11:01 0% ` David Marchand
0 siblings, 2 replies; 200+ results
From: Thomas Monjalon @ 2020-02-26 13:55 UTC (permalink / raw)
To: David Marchand; +Cc: dev
26/02/2020 14:38, David Marchand:
> Start a new release cycle with empty release notes.
> ABI must now be checked with v20.02 as a reference.
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
> .travis.yml | 4 +
> ABI_VERSION | 2 +-
> VERSION | 2 +-
> doc/guides/rel_notes/index.rst | 1 +
> doc/guides/rel_notes/release_20_05.rst | 139 +++++++++++++++++++++++++
> 5 files changed, 146 insertions(+), 2 deletions(-)
> create mode 100644 doc/guides/rel_notes/release_20_05.rst
Acked-by: Thomas Monjalon <thomas@monjalon.net>
> --- a/.travis.yml
> +++ b/.travis.yml
> +env:
> + global:
> + - REF_GIT_TAG=v20.02
One question: why REF_GIT_TAG was not needed in Travis during 20.02?
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] version: 20.05-rc0
2020-02-26 13:55 0% ` Thomas Monjalon
@ 2020-02-26 14:01 0% ` David Marchand
2020-02-27 11:17 3% ` Bruce Richardson
2020-02-27 11:01 0% ` David Marchand
1 sibling, 1 reply; 200+ results
From: David Marchand @ 2020-02-26 14:01 UTC (permalink / raw)
To: Thomas Monjalon; +Cc: dev
On Wed, Feb 26, 2020 at 2:55 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> 26/02/2020 14:38, David Marchand:
> > Start a new release cycle with empty release notes.
> > ABI must now be checked with v20.02 as a reference.
> >
> > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > ---
> > .travis.yml | 4 +
> > ABI_VERSION | 2 +-
> > VERSION | 2 +-
> > doc/guides/rel_notes/index.rst | 1 +
> > doc/guides/rel_notes/release_20_05.rst | 139 +++++++++++++++++++++++++
> > 5 files changed, 146 insertions(+), 2 deletions(-)
> > create mode 100644 doc/guides/rel_notes/release_20_05.rst
>
> Acked-by: Thomas Monjalon <thomas@monjalon.net>
>
>
> > --- a/.travis.yml
> > +++ b/.travis.yml
> > +env:
> > + global:
> > + - REF_GIT_TAG=v20.02
>
> One question: why REF_GIT_TAG was not needed in Travis during 20.02?
The .ci/linux-build.sh script has a default value for parameters.
http://git.dpdk.org/dpdk/tree/.ci/linux-build.sh#n71&h=v20.02
This could be removed, so that we only have explicit values in the
.travis.yml configuration.
No strong opinion keeping them.
--
David Marchand
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] Questions about rte_timer APIs
2020-02-25 23:56 3% ` Honnappa Nagarahalli
@ 2020-02-26 16:57 0% ` Carrillo, Erik G
2020-02-27 16:58 0% ` Honnappa Nagarahalli
0 siblings, 1 reply; 200+ results
From: Carrillo, Erik G @ 2020-02-26 16:57 UTC (permalink / raw)
To: Honnappa Nagarahalli, Robert Sanford
Cc: dev, nd, Phil Yang, Gavin Hu, david.marchand, thomas, nd
Hi Honnappa,
Your description below looks correct to me. For the current implementation, I referenced a couple of existing DPDK libraries, but primarily the rte_service library. However, I agree that allocating the timer data structs only as needed would be a good idea.
You are also correct that, since the rte_timer_alt_* APIs are still experimental, they can change without constraint; this would allow for the APIs to change as needed to support the above design. It looks like the only users of those APIs in DPDK currently are the event timer adapter and unit test code.
Thanks,
Erik
> -----Original Message-----
> From: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
> Sent: Tuesday, February 25, 2020 5:56 PM
> To: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>; Robert
> Sanford <rsanford@akamai.com>; Carrillo, Erik G <erik.g.carrillo@intel.com>
> Cc: dev <dev@dpdk.org>; nd <nd@arm.com>; Phil Yang
> <Phil.Yang@arm.com>; Gavin Hu <Gavin.Hu@arm.com>;
> david.marchand@redhat.com; thomas@monjalon.net; nd <nd@arm.com>
> Subject: RE: Questions about rte_timer APIs
>
> Hi Erik,
> I see that the rte_timer_alt_xxx APIs are still marked as
> experimental. So, we should be able to change them without any ABI
> constraints. Please let me know what you think.
>
> Thank you,
> Honnappa
>
> > -----Original Message-----
> > From: dev <dev-bounces@dpdk.org> On Behalf Of Honnappa Nagarahalli
> > Sent: Thursday, February 13, 2020 11:55 PM
> > To: Robert Sanford <rsanford@akamai.com>; Erik Gabriel Carrillo
> > <erik.g.carrillo@intel.com>
> > Cc: dev <dev@dpdk.org>; nd <nd@arm.com>; Phil Yang
> > <Phil.Yang@arm.com>; Gavin Hu <Gavin.Hu@arm.com>; nd
> <nd@arm.com>
> > Subject: [dpdk-dev] Questions about rte_timer APIs
> >
> > Hi Erik/Robert,
> > I was looking at the rte_timer library code and have questions around
> > the way instances of timer list are allocated.
> >
> > The API 'rte_timer_subsystem_init' allocates RTE_MAX_DATA_ELS (64)
> > number of timer data structs. Each timer data struct acts as an
> > independent instance of timer list. The user allocates an instance by
> > calling 'rte_timer_data_alloc' which provides an index in the instance array.to s
> > Essentially, the library is allocating the memory upfront even though
> > there might not be a need to have 64 instances. Please correct me if I am
> wrong.
> >
> > Usually, creating multiple instances is handled by allocating required
> > memory, as and when required, by an 'init' API. This API also returns
> > a pointer to that instance which is passed to other APIs in the
> > library. For ex: rte_hash library follows this approach. IMO, this is an
> elegant way to handle multiple instances.
> > This approach does not waste memory and does not put any restriction
> > on number of instances.
> >
> > I wanted to understand the reasoning behind the current design to
> > handle multiple instances. Appreciate your inputs.
> >
> > Thank you,
> > Honnappa
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [RFC 0/6] New sync modes for ring
2020-02-25 13:41 3% ` Ananyev, Konstantin
@ 2020-02-27 10:31 0% ` Jerin Jacob
2020-02-28 0:17 0% ` David Christensen
0 siblings, 1 reply; 200+ results
From: Jerin Jacob @ 2020-02-27 10:31 UTC (permalink / raw)
To: Ananyev, Konstantin; +Cc: Stephen Hemminger, dpdk-dev, Olivier Matz, drc
On Tue, Feb 25, 2020 at 7:11 PM Ananyev, Konstantin
<konstantin.ananyev@intel.com> wrote:
> We do have a run-time check in our current enqueue()/dequeue implementation.
> In fact we support both modes: we have generic rte_ring_enqueue(/dequeue)_bulk(/burst)
> where sync behaviour is determined at runtime by value of prod(/cons).single.
> Or user can call rte_ring_(mp/sp)_enqueue_* functions directly.
> This RFC follows exactly the same paradigm:
> rte_ring_enqueue(/dequeue)_bulk(/burst) kept generic and it's
> behaviour is determined at runtime, by value of prod(/cons).sync_type.
> Or user can call enqueue/dequeue with particular sync mode directly:
> rte_ring_(mp/sp/rts/hts)_enqueue_(bulk/burst)*.
> The only thing that changed:
> Format of prod/cons now could differ depending on mode selected at _init_.
> So you can't create a ring for let say SP mode and then in the middle of data-path
> change your mind and start using MP_RTS mode.
> For existing modes (SP/MP, SC/MC) format remains the same and user can still
> use them interchangeably, though of course that is an error prone practice.
Makes sense.
>
> > > But I agree with the problem statement that in the virtualization use
> > > case, It may be possible to have N virtual cores runs on a physical
> > > core.
> > >
> > > IMO, The best solution would be keeping the ring API same and have a
> > > different flavor in "compile-time". Something like
> > > liburcu did for accommodating different flavors.
> > >
> > > i.e urcu-qsbr.h and urcu-bp.h will identical definition of API. The
> > > application can simply include ONE header file in a C file based on
> > > the flavor.
>
> I don't think it is a flexible enough approach.
> In one app user might need to have several rings with different sync modes.
> Or even user might need a ring with different sync modes for enqueue/dequeue.
Ack.
> Yes, hiding rte_ring implementation inside .c would help a lot
> in terms of ABI maintenance and would make our future life easier.
> The question is what is the price for it in terms of performance,
> and are we ready to pay it. Not to mention that it would cause
> changes in many other libs/apps...
> So I think it should be a subject for a separate discussion.
> But, agree it would be good at least to measure the performance
> impact of such change.
> If I'll have some spare cycles, will give it a try.
> Meanwhile, can I ask Jerin and other guys to repeat tests from this RFC
> on their HW? Before continuing discussion would probably be good to know
> does the suggested patch work as expected across different platforms.
I tested on an arm64 HW. The former section is without the
patch(20.02) and later one with this patch.
I agree with Konstantin that getting more platform tests will be good
early so that we can focus on the approach
to avoid back and forth latter.
RTE>>ring_perf_autotest // without path
### Testing single element enq/deq ###
legacy APIs: SP/SC: single: 289.78
legacy APIs: MP/MC: single: 516.20
### Testing burst enq/deq ###
legacy APIs: SP/SC: burst (size: 8): 312.88
legacy APIs: SP/SC: burst (size: 32): 426.72
legacy APIs: MP/MC: burst (size: 8): 510.95
legacy APIs: MP/MC: burst (size: 32): 702.01
### Testing bulk enq/deq ###
legacy APIs: SP/SC: bulk (size: 8): 306.74
legacy APIs: SP/SC: bulk (size: 32): 411.56
legacy APIs: MP/MC: bulk (size: 8): 501.32
legacy APIs: MP/MC: bulk (size: 32): 693.07
### Testing empty bulk deq ###
legacy APIs: SP/SC: bulk (size: 8): 7.00
legacy APIs: MP/MC: bulk (size: 8): 7.00
### Testing using two physical cores ###
legacy APIs: SP/SC: bulk (size: 8): 74.36
legacy APIs: MP/MC: bulk (size: 8): 110.18
legacy APIs: SP/SC: bulk (size: 32): 23.04
legacy APIs: MP/MC: bulk (size: 32): 32.29
### Testing using all slave nodes ##
Bulk enq/dequeue count on size 8
Core [8] count = 293741
Core [9] count = 293741
Total count (size: 8): 587482
Bulk enq/dequeue count on size 32
Core [8] count = 244909
Core [9] count = 244909
Total count (size: 32): 1077300
### Testing single element enq/deq ###
elem APIs: element size 16B: SP/SC: single: 255.37
elem APIs: element size 16B: MP/MC: single: 456.68
### Testing burst enq/deq ###
elem APIs: element size 16B: SP/SC: burst (size: 8): 291.99
elem APIs: element size 16B: SP/SC: burst (size: 32): 456.25
elem APIs: element size 16B: MP/MC: burst (size: 8): 497.77
elem APIs: element size 16B: MP/MC: burst (size: 32): 680.87
### Testing bulk enq/deq ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 284.40
elem APIs: element size 16B: SP/SC: bulk (size: 32): 453.17
elem APIs: element size 16B: MP/MC: bulk (size: 8): 485.77
elem APIs: element size 16B: MP/MC: bulk (size: 32): 675.08
### Testing empty bulk deq ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 8.00
elem APIs: element size 16B: MP/MC: bulk (size: 8): 7.00
### Testing using two physical cores ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 74.45
elem APIs: element size 16B: MP/MC: bulk (size: 8): 105.91
elem APIs: element size 16B: SP/SC: bulk (size: 32): 22.92
elem APIs: element size 16B: MP/MC: bulk (size: 32): 31.55
### Testing using all slave nodes ###
Bulk enq/dequeue count on size 8
Core [8] count = 308724
Core [9] count = 308723
Total count (size: 8): 617447
Bulk enq/dequeue count on size 32
Core [8] count = 214269
Core [9] count = 214269
Total count (size: 32): 1045985
RTE>>ring_perf_autotest // with patch
### Testing single element enq/deq ###
legacy APIs: SP/SC: single: 289.78
legacy APIs: MP/MC: single: 475.76
### Testing burst enq/deq ###
legacy APIs: SP/SC: burst (size: 8): 323.91
legacy APIs: SP/SC: burst (size: 32): 424.60
legacy APIs: MP/MC: burst (size: 8): 523.00
legacy APIs: MP/MC: burst (size: 32): 717.09
### Testing bulk enq/deq ###
legacy APIs: SP/SC: bulk (size: 8): 317.74
legacy APIs: SP/SC: bulk (size: 32): 413.57
legacy APIs: MP/MC: bulk (size: 8): 512.89
legacy APIs: MP/MC: bulk (size: 32): 712.45
### Testing empty bulk deq ###
legacy APIs: SP/SC: bulk (size: 8): 7.00
legacy APIs: MP/MC: bulk (size: 8): 7.00
### Testing using two physical cores ###
legacy APIs: SP/SC: bulk (size: 8): 74.82
legacy APIs: MP/MC: bulk (size: 8): 96.45
legacy APIs: SP/SC: bulk (size: 32): 22.97
legacy APIs: MP/MC: bulk (size: 32): 32.52
### Testing using all slave nodes ###
Bulk enq/dequeue count on size 8
Core [8] count = 283928
Core [9] count = 283927
Total count (size: 8): 567855
Bulk enq/dequeue count on size 32
Core [8] count = 223916
Core [9] count = 223915
Total count (size: 32): 1015686
### Testing single element enq/deq ###
elem APIs: element size 16B: SP/SC: single: 267.65
elem APIs: element size 16B: MP/MC: single: 439.06
### Testing burst enq/deq ###
elem APIs: element size 16B: SP/SC: burst (size: 8): 302.44
elem APIs: element size 16B: SP/SC: burst (size: 32): 466.31
elem APIs: element size 16B: MP/MC: burst (size: 8): 502.51
elem APIs: element size 16B: MP/MC: burst (size: 32): 695.81
### Testing bulk enq/deq ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 295.15
elem APIs: element size 16B: SP/SC: bulk (size: 32): 462.77
elem APIs: element size 16B: MP/MC: bulk (size: 8): 496.89
elem APIs: element size 16B: MP/MC: bulk (size: 32): 690.46
### Testing empty bulk deq ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 7.50
elem APIs: element size 16B: MP/MC: bulk (size: 8): 7.44
### Testing using two physical cores ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 65.85
elem APIs: element size 16B: MP/MC: bulk (size: 8): 103.80
elem APIs: element size 16B: SP/SC: bulk (size: 32): 23.27
elem APIs: element size 16B: MP/MC: bulk (size: 32): 31.17
### Testing using all slave nodes ###
Bulk enq/dequeue count on size 8
Core [8] count = 304223
Core [9] count = 304221
Total count (size: 8): 608444
Bulk enq/dequeue count on size 32
Core [8] count = 214856
Core [9] count = 214855
Total count (size: 32): 1038155
Test OK
RTE>>quit
> Thanks
> Konstantin
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] version: 20.05-rc0
2020-02-26 13:55 0% ` Thomas Monjalon
2020-02-26 14:01 0% ` David Marchand
@ 2020-02-27 11:01 0% ` David Marchand
2020-02-27 11:31 0% ` David Marchand
1 sibling, 1 reply; 200+ results
From: David Marchand @ 2020-02-27 11:01 UTC (permalink / raw)
To: David Marchand; +Cc: dev, Thomas Monjalon
On Wed, Feb 26, 2020 at 2:55 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> 26/02/2020 14:38, David Marchand:
> > Start a new release cycle with empty release notes.
> > ABI must now be checked with v20.02 as a reference.
> >
> > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > ---
> > .travis.yml | 4 +
> > ABI_VERSION | 2 +-
> > VERSION | 2 +-
> > doc/guides/rel_notes/index.rst | 1 +
> > doc/guides/rel_notes/release_20_05.rst | 139 +++++++++++++++++++++++++
> > 5 files changed, 146 insertions(+), 2 deletions(-)
> > create mode 100644 doc/guides/rel_notes/release_20_05.rst
>
> Acked-by: Thomas Monjalon <thomas@monjalon.net>
Applied, here we go.
--
David Marchand
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] version: 20.05-rc0
2020-02-26 14:01 0% ` David Marchand
@ 2020-02-27 11:17 3% ` Bruce Richardson
2020-02-27 11:23 3% ` David Marchand
0 siblings, 1 reply; 200+ results
From: Bruce Richardson @ 2020-02-27 11:17 UTC (permalink / raw)
To: David Marchand; +Cc: Thomas Monjalon, dev
On Wed, Feb 26, 2020 at 03:01:04PM +0100, David Marchand wrote:
> On Wed, Feb 26, 2020 at 2:55 PM Thomas Monjalon <thomas@monjalon.net> wrote:
> >
> > 26/02/2020 14:38, David Marchand:
> > > Start a new release cycle with empty release notes.
> > > ABI must now be checked with v20.02 as a reference.
> > >
> > > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > > ---
> > > .travis.yml | 4 +
> > > ABI_VERSION | 2 +-
> > > VERSION | 2 +-
> > > doc/guides/rel_notes/index.rst | 1 +
> > > doc/guides/rel_notes/release_20_05.rst | 139 +++++++++++++++++++++++++
> > > 5 files changed, 146 insertions(+), 2 deletions(-)
> > > create mode 100644 doc/guides/rel_notes/release_20_05.rst
> >
> > Acked-by: Thomas Monjalon <thomas@monjalon.net>
> >
> >
> > > --- a/.travis.yml
> > > +++ b/.travis.yml
> > > +env:
> > > + global:
> > > + - REF_GIT_TAG=v20.02
> >
> > One question: why REF_GIT_TAG was not needed in Travis during 20.02?
>
> The .ci/linux-build.sh script has a default value for parameters.
> http://git.dpdk.org/dpdk/tree/.ci/linux-build.sh#n71&h=v20.02
>
> This could be removed, so that we only have explicit values in the
> .travis.yml configuration.
> No strong opinion keeping them.
>
Why do we need to update the git ref tag at all. Since these releases are
all ABI compatible with each other, can we not leave the checks still to be
against 19.11?
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH] version: 20.05-rc0
2020-02-27 11:17 3% ` Bruce Richardson
@ 2020-02-27 11:23 3% ` David Marchand
2020-02-27 11:26 4% ` Thomas Monjalon
0 siblings, 1 reply; 200+ results
From: David Marchand @ 2020-02-27 11:23 UTC (permalink / raw)
To: Bruce Richardson; +Cc: Thomas Monjalon, dev
On Thu, Feb 27, 2020 at 12:17 PM Bruce Richardson
<bruce.richardson@intel.com> wrote:
>
> On Wed, Feb 26, 2020 at 03:01:04PM +0100, David Marchand wrote:
> > On Wed, Feb 26, 2020 at 2:55 PM Thomas Monjalon <thomas@monjalon.net> wrote:
> > >
> > > 26/02/2020 14:38, David Marchand:
> > > > Start a new release cycle with empty release notes.
> > > > ABI must now be checked with v20.02 as a reference.
> > > >
> > > > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > > > ---
> > > > .travis.yml | 4 +
> > > > ABI_VERSION | 2 +-
> > > > VERSION | 2 +-
> > > > doc/guides/rel_notes/index.rst | 1 +
> > > > doc/guides/rel_notes/release_20_05.rst | 139 +++++++++++++++++++++++++
> > > > 5 files changed, 146 insertions(+), 2 deletions(-)
> > > > create mode 100644 doc/guides/rel_notes/release_20_05.rst
> > >
> > > Acked-by: Thomas Monjalon <thomas@monjalon.net>
> > >
> > >
> > > > --- a/.travis.yml
> > > > +++ b/.travis.yml
> > > > +env:
> > > > + global:
> > > > + - REF_GIT_TAG=v20.02
> > >
> > > One question: why REF_GIT_TAG was not needed in Travis during 20.02?
> >
> > The .ci/linux-build.sh script has a default value for parameters.
> > http://git.dpdk.org/dpdk/tree/.ci/linux-build.sh#n71&h=v20.02
> >
> > This could be removed, so that we only have explicit values in the
> > .travis.yml configuration.
> > No strong opinion keeping them.
> >
> Why do we need to update the git ref tag at all. Since these releases are
> all ABI compatible with each other, can we not leave the checks still to be
> against 19.11?
Checking only with 19.11 (major abi) would not detected breakage
between minor versions.
Like the meter symbols that entered the 20.0.1 version.
--
David Marchand
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH] version: 20.05-rc0
2020-02-27 11:23 3% ` David Marchand
@ 2020-02-27 11:26 4% ` Thomas Monjalon
0 siblings, 0 replies; 200+ results
From: Thomas Monjalon @ 2020-02-27 11:26 UTC (permalink / raw)
To: Bruce Richardson, David Marchand; +Cc: dev
27/02/2020 12:23, David Marchand:
> On Thu, Feb 27, 2020 at 12:17 PM Bruce Richardson
> <bruce.richardson@intel.com> wrote:
> >
> > On Wed, Feb 26, 2020 at 03:01:04PM +0100, David Marchand wrote:
> > > On Wed, Feb 26, 2020 at 2:55 PM Thomas Monjalon <thomas@monjalon.net> wrote:
> > > >
> > > > 26/02/2020 14:38, David Marchand:
> > > > > Start a new release cycle with empty release notes.
> > > > > ABI must now be checked with v20.02 as a reference.
> > > > >
> > > > > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > > > > ---
> > > > > .travis.yml | 4 +
> > > > > ABI_VERSION | 2 +-
> > > > > VERSION | 2 +-
> > > > > doc/guides/rel_notes/index.rst | 1 +
> > > > > doc/guides/rel_notes/release_20_05.rst | 139 +++++++++++++++++++++++++
> > > > > 5 files changed, 146 insertions(+), 2 deletions(-)
> > > > > create mode 100644 doc/guides/rel_notes/release_20_05.rst
> > > >
> > > > Acked-by: Thomas Monjalon <thomas@monjalon.net>
> > > >
> > > >
> > > > > --- a/.travis.yml
> > > > > +++ b/.travis.yml
> > > > > +env:
> > > > > + global:
> > > > > + - REF_GIT_TAG=v20.02
> > > >
> > > > One question: why REF_GIT_TAG was not needed in Travis during 20.02?
> > >
> > > The .ci/linux-build.sh script has a default value for parameters.
> > > http://git.dpdk.org/dpdk/tree/.ci/linux-build.sh#n71&h=v20.02
> > >
> > > This could be removed, so that we only have explicit values in the
> > > .travis.yml configuration.
> > > No strong opinion keeping them.
> > >
> > Why do we need to update the git ref tag at all. Since these releases are
> > all ABI compatible with each other, can we not leave the checks still to be
> > against 19.11?
>
> Checking only with 19.11 (major abi) would not detected breakage
> between minor versions.
> Like the meter symbols that entered the 20.0.1 version.
We check that new ABI, introduced in a minor ABI version, is not broken.
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH] version: 20.05-rc0
2020-02-27 11:01 0% ` David Marchand
@ 2020-02-27 11:31 0% ` David Marchand
0 siblings, 0 replies; 200+ results
From: David Marchand @ 2020-02-27 11:31 UTC (permalink / raw)
Cc: dev, Thomas Monjalon
On Thu, Feb 27, 2020 at 12:01 PM David Marchand
<david.marchand@redhat.com> wrote:
>
> On Wed, Feb 26, 2020 at 2:55 PM Thomas Monjalon <thomas@monjalon.net> wrote:
> >
> > 26/02/2020 14:38, David Marchand:
> > > Start a new release cycle with empty release notes.
> > > ABI must now be checked with v20.02 as a reference.
> > >
> > > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > > ---
> > > .travis.yml | 4 +
> > > ABI_VERSION | 2 +-
> > > VERSION | 2 +-
> > > doc/guides/rel_notes/index.rst | 1 +
> > > doc/guides/rel_notes/release_20_05.rst | 139 +++++++++++++++++++++++++
> > > 5 files changed, 146 insertions(+), 2 deletions(-)
> > > create mode 100644 doc/guides/rel_notes/release_20_05.rst
> >
> > Acked-by: Thomas Monjalon <thomas@monjalon.net>
>
> Applied, here we go.
Patchwork has just been cleaned: old patches archived, Deferred
patches back to state NEW.
--
David Marchand
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] Questions about rte_timer APIs
2020-02-26 16:57 0% ` Carrillo, Erik G
@ 2020-02-27 16:58 0% ` Honnappa Nagarahalli
0 siblings, 0 replies; 200+ results
From: Honnappa Nagarahalli @ 2020-02-27 16:58 UTC (permalink / raw)
To: Carrillo, Erik G, Robert Sanford
Cc: dev, nd, Phil Yang, Gavin Hu, david.marchand, thomas, nd, nd
Thanks Erik. We will create some patches and send them for review.
> -----Original Message-----
> From: Carrillo, Erik G <erik.g.carrillo@intel.com>
> Sent: Wednesday, February 26, 2020 10:58 AM
> To: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>; Robert
> Sanford <rsanford@akamai.com>
> Cc: dev <dev@dpdk.org>; nd <nd@arm.com>; Phil Yang
> <Phil.Yang@arm.com>; Gavin Hu <Gavin.Hu@arm.com>;
> david.marchand@redhat.com; thomas@monjalon.net; nd <nd@arm.com>
> Subject: RE: Questions about rte_timer APIs
>
> Hi Honnappa,
>
> Your description below looks correct to me. For the current implementation, I
> referenced a couple of existing DPDK libraries, but primarily the rte_service
> library. However, I agree that allocating the timer data structs only as needed
> would be a good idea.
>
> You are also correct that, since the rte_timer_alt_* APIs are still experimental,
> they can change without constraint; this would allow for the APIs to change
> as needed to support the above design. It looks like the only users of those
> APIs in DPDK currently are the event timer adapter and unit test code.
>
> Thanks,
> Erik
>
> > -----Original Message-----
> > From: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
> > Sent: Tuesday, February 25, 2020 5:56 PM
> > To: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>; Robert
> > Sanford <rsanford@akamai.com>; Carrillo, Erik G
> > <erik.g.carrillo@intel.com>
> > Cc: dev <dev@dpdk.org>; nd <nd@arm.com>; Phil Yang
> > <Phil.Yang@arm.com>; Gavin Hu <Gavin.Hu@arm.com>;
> > david.marchand@redhat.com; thomas@monjalon.net; nd <nd@arm.com>
> > Subject: RE: Questions about rte_timer APIs
> >
> > Hi Erik,
> > I see that the rte_timer_alt_xxx APIs are still marked as
> > experimental. So, we should be able to change them without any ABI
> > constraints. Please let me know what you think.
> >
> > Thank you,
> > Honnappa
> >
> > > -----Original Message-----
> > > From: dev <dev-bounces@dpdk.org> On Behalf Of Honnappa Nagarahalli
> > > Sent: Thursday, February 13, 2020 11:55 PM
> > > To: Robert Sanford <rsanford@akamai.com>; Erik Gabriel Carrillo
> > > <erik.g.carrillo@intel.com>
> > > Cc: dev <dev@dpdk.org>; nd <nd@arm.com>; Phil Yang
> > > <Phil.Yang@arm.com>; Gavin Hu <Gavin.Hu@arm.com>; nd
> > <nd@arm.com>
> > > Subject: [dpdk-dev] Questions about rte_timer APIs
> > >
> > > Hi Erik/Robert,
> > > I was looking at the rte_timer library code and have questions
> > > around the way instances of timer list are allocated.
> > >
> > > The API 'rte_timer_subsystem_init' allocates RTE_MAX_DATA_ELS (64)
> > > number of timer data structs. Each timer data struct acts as an
> > > independent instance of timer list. The user allocates an instance
> > > by calling 'rte_timer_data_alloc' which provides an index in the
> > > instance array.to s Essentially, the library is allocating the
> > > memory upfront even though there might not be a need to have 64
> > > instances. Please correct me if I am
> > wrong.
> > >
> > > Usually, creating multiple instances is handled by allocating
> > > required memory, as and when required, by an 'init' API. This API
> > > also returns a pointer to that instance which is passed to other
> > > APIs in the library. For ex: rte_hash library follows this approach.
> > > IMO, this is an
> > elegant way to handle multiple instances.
> > > This approach does not waste memory and does not put any restriction
> > > on number of instances.
> > >
> > > I wanted to understand the reasoning behind the current design to
> > > handle multiple instances. Appreciate your inputs.
> > >
> > > Thank you,
> > > Honnappa
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [RFC 0/6] New sync modes for ring
2020-02-27 10:31 0% ` Jerin Jacob
@ 2020-02-28 0:17 0% ` David Christensen
0 siblings, 0 replies; 200+ results
From: David Christensen @ 2020-02-28 0:17 UTC (permalink / raw)
To: Jerin Jacob, Ananyev, Konstantin
Cc: Stephen Hemminger, dpdk-dev, Olivier Matz
> On Tue, Feb 25, 2020 at 7:11 PM Ananyev, Konstantin
> <konstantin.ananyev@intel.com> wrote:
>
>> We do have a run-time check in our current enqueue()/dequeue implementation.
>> In fact we support both modes: we have generic rte_ring_enqueue(/dequeue)_bulk(/burst)
>> where sync behaviour is determined at runtime by value of prod(/cons).single.
>> Or user can call rte_ring_(mp/sp)_enqueue_* functions directly.
>> This RFC follows exactly the same paradigm:
>> rte_ring_enqueue(/dequeue)_bulk(/burst) kept generic and it's
>> behaviour is determined at runtime, by value of prod(/cons).sync_type.
>> Or user can call enqueue/dequeue with particular sync mode directly:
>> rte_ring_(mp/sp/rts/hts)_enqueue_(bulk/burst)*.
>> The only thing that changed:
>> Format of prod/cons now could differ depending on mode selected at _init_.
>> So you can't create a ring for let say SP mode and then in the middle of data-path
>> change your mind and start using MP_RTS mode.
>> For existing modes (SP/MP, SC/MC) format remains the same and user can still
>> use them interchangeably, though of course that is an error prone practice.
>
> Makes sense.
>
>
>>
>>>> But I agree with the problem statement that in the virtualization use
>>>> case, It may be possible to have N virtual cores runs on a physical
>>>> core.
>>>>
>>>> IMO, The best solution would be keeping the ring API same and have a
>>>> different flavor in "compile-time". Something like
>>>> liburcu did for accommodating different flavors.
>>>>
>>>> i.e urcu-qsbr.h and urcu-bp.h will identical definition of API. The
>>>> application can simply include ONE header file in a C file based on
>>>> the flavor.
>>
>> I don't think it is a flexible enough approach.
>> In one app user might need to have several rings with different sync modes.
>> Or even user might need a ring with different sync modes for enqueue/dequeue.
>
> Ack.
>
>
>> Yes, hiding rte_ring implementation inside .c would help a lot
>> in terms of ABI maintenance and would make our future life easier.
>> The question is what is the price for it in terms of performance,
>> and are we ready to pay it. Not to mention that it would cause
>> changes in many other libs/apps...
>> So I think it should be a subject for a separate discussion.
>> But, agree it would be good at least to measure the performance
>> impact of such change.
>> If I'll have some spare cycles, will give it a try.
>> Meanwhile, can I ask Jerin and other guys to repeat tests from this RFC
>> on their HW? Before continuing discussion would probably be good to know
>> does the suggested patch work as expected across different platforms.
>
>
> I tested on an arm64 HW. The former section is without the
> patch(20.02) and later one with this patch.
> I agree with Konstantin that getting more platform tests will be good
> early so that we can focus on the approach
> to avoid back and forth latter.
>
>
> RTE>>ring_perf_autotest // without path
>
> ### Testing single element enq/deq ###
> legacy APIs: SP/SC: single: 289.78
> legacy APIs: MP/MC: single: 516.20
>
> ### Testing burst enq/deq ###
> legacy APIs: SP/SC: burst (size: 8): 312.88
> legacy APIs: SP/SC: burst (size: 32): 426.72
> legacy APIs: MP/MC: burst (size: 8): 510.95
> legacy APIs: MP/MC: burst (size: 32): 702.01
>
> ### Testing bulk enq/deq ###
> legacy APIs: SP/SC: bulk (size: 8): 306.74
> legacy APIs: SP/SC: bulk (size: 32): 411.56
> legacy APIs: MP/MC: bulk (size: 8): 501.32
> legacy APIs: MP/MC: bulk (size: 32): 693.07
>
> ### Testing empty bulk deq ###
> legacy APIs: SP/SC: bulk (size: 8): 7.00
> legacy APIs: MP/MC: bulk (size: 8): 7.00
>
> ### Testing using two physical cores ###
> legacy APIs: SP/SC: bulk (size: 8): 74.36
> legacy APIs: MP/MC: bulk (size: 8): 110.18
> legacy APIs: SP/SC: bulk (size: 32): 23.04
> legacy APIs: MP/MC: bulk (size: 32): 32.29
>
> ### Testing using all slave nodes ##
> Bulk enq/dequeue count on size 8
> Core [8] count = 293741
> Core [9] count = 293741
> Total count (size: 8): 587482
>
> Bulk enq/dequeue count on size 32
> Core [8] count = 244909
> Core [9] count = 244909
> Total count (size: 32): 1077300
>
> ### Testing single element enq/deq ###
> elem APIs: element size 16B: SP/SC: single: 255.37
> elem APIs: element size 16B: MP/MC: single: 456.68
>
> ### Testing burst enq/deq ###
> elem APIs: element size 16B: SP/SC: burst (size: 8): 291.99
> elem APIs: element size 16B: SP/SC: burst (size: 32): 456.25
> elem APIs: element size 16B: MP/MC: burst (size: 8): 497.77
> elem APIs: element size 16B: MP/MC: burst (size: 32): 680.87
>
> ### Testing bulk enq/deq ###
> elem APIs: element size 16B: SP/SC: bulk (size: 8): 284.40
> elem APIs: element size 16B: SP/SC: bulk (size: 32): 453.17
> elem APIs: element size 16B: MP/MC: bulk (size: 8): 485.77
> elem APIs: element size 16B: MP/MC: bulk (size: 32): 675.08
>
> ### Testing empty bulk deq ###
> elem APIs: element size 16B: SP/SC: bulk (size: 8): 8.00
> elem APIs: element size 16B: MP/MC: bulk (size: 8): 7.00
>
> ### Testing using two physical cores ###
> elem APIs: element size 16B: SP/SC: bulk (size: 8): 74.45
> elem APIs: element size 16B: MP/MC: bulk (size: 8): 105.91
> elem APIs: element size 16B: SP/SC: bulk (size: 32): 22.92
> elem APIs: element size 16B: MP/MC: bulk (size: 32): 31.55
>
> ### Testing using all slave nodes ###
>
> Bulk enq/dequeue count on size 8
> Core [8] count = 308724
> Core [9] count = 308723
> Total count (size: 8): 617447
>
> Bulk enq/dequeue count on size 32
> Core [8] count = 214269
> Core [9] count = 214269
> Total count (size: 32): 1045985
>
> RTE>>ring_perf_autotest // with patch
>
> ### Testing single element enq/deq ###
> legacy APIs: SP/SC: single: 289.78
> legacy APIs: MP/MC: single: 475.76
>
> ### Testing burst enq/deq ###
> legacy APIs: SP/SC: burst (size: 8): 323.91
> legacy APIs: SP/SC: burst (size: 32): 424.60
> legacy APIs: MP/MC: burst (size: 8): 523.00
> legacy APIs: MP/MC: burst (size: 32): 717.09
>
> ### Testing bulk enq/deq ###
> legacy APIs: SP/SC: bulk (size: 8): 317.74
> legacy APIs: SP/SC: bulk (size: 32): 413.57
> legacy APIs: MP/MC: bulk (size: 8): 512.89
> legacy APIs: MP/MC: bulk (size: 32): 712.45
>
> ### Testing empty bulk deq ###
> legacy APIs: SP/SC: bulk (size: 8): 7.00
> legacy APIs: MP/MC: bulk (size: 8): 7.00
>
> ### Testing using two physical cores ###
> legacy APIs: SP/SC: bulk (size: 8): 74.82
> legacy APIs: MP/MC: bulk (size: 8): 96.45
> legacy APIs: SP/SC: bulk (size: 32): 22.97
> legacy APIs: MP/MC: bulk (size: 32): 32.52
>
> ### Testing using all slave nodes ###
>
> Bulk enq/dequeue count on size 8
> Core [8] count = 283928
> Core [9] count = 283927
> Total count (size: 8): 567855
>
> Bulk enq/dequeue count on size 32
> Core [8] count = 223916
> Core [9] count = 223915
> Total count (size: 32): 1015686
>
> ### Testing single element enq/deq ###
> elem APIs: element size 16B: SP/SC: single: 267.65
> elem APIs: element size 16B: MP/MC: single: 439.06
>
> ### Testing burst enq/deq ###
> elem APIs: element size 16B: SP/SC: burst (size: 8): 302.44
> elem APIs: element size 16B: SP/SC: burst (size: 32): 466.31
> elem APIs: element size 16B: MP/MC: burst (size: 8): 502.51
> elem APIs: element size 16B: MP/MC: burst (size: 32): 695.81
>
> ### Testing bulk enq/deq ###
> elem APIs: element size 16B: SP/SC: bulk (size: 8): 295.15
> elem APIs: element size 16B: SP/SC: bulk (size: 32): 462.77
> elem APIs: element size 16B: MP/MC: bulk (size: 8): 496.89
> elem APIs: element size 16B: MP/MC: bulk (size: 32): 690.46
>
> ### Testing empty bulk deq ###
> elem APIs: element size 16B: SP/SC: bulk (size: 8): 7.50
> elem APIs: element size 16B: MP/MC: bulk (size: 8): 7.44
>
> ### Testing using two physical cores ###
> elem APIs: element size 16B: SP/SC: bulk (size: 8): 65.85
> elem APIs: element size 16B: MP/MC: bulk (size: 8): 103.80
> elem APIs: element size 16B: SP/SC: bulk (size: 32): 23.27
> elem APIs: element size 16B: MP/MC: bulk (size: 32): 31.17
>
> ### Testing using all slave nodes ###
>
> Bulk enq/dequeue count on size 8
> Core [8] count = 304223
> Core [9] count = 304221
> Total count (size: 8): 608444
>
> Bulk enq/dequeue count on size 32
> Core [8] count = 214856
> Core [9] count = 214855
> Total count (size: 32): 1038155
> Test OK
> RTE>>quit
>
>
Encountered a couple of different build errors with these patches on my
Power 9 system:
In file included from ../lib/librte_ring/rte_ring.h:534,
from ../drivers/mempool/ring/rte_mempool_ring.c:9:
../lib/librte_ring/rte_ring_hts_generic.h: In function
‘__rte_ring_hts_update_tail’:
../lib/librte_ring/rte_ring_hts_generic.h:61:2: warning: implicit
declaration of function ‘RTE_ASSERT’; did you mean ‘RTE_STR’?
[-Wimplicit-function-declaration]
RTE_ASSERT(n >= num);
^~~~~~~~~~
RTE_STR
Fixed by adding "#include <rte_debug.h>" to rte_ring.h.
Also encountered:
In file included from ../app/test/test_ring_hts_stress.c:5:
../app/test/test_ring_stress.h: In function ‘check_updt_elem’:
../app/test/test_ring_stress.h:162:9: error: unknown type name
‘rte_spinlock_t’
static rte_spinlock_t dump_lock;
^~~~~~~~~~~~~~
../app/test/test_ring_stress.h:166:4: warning: implicit declaration of
function ‘rte_spinlock_lock’; did you mean ‘rte_calloc_socket’?
[-Wimplicit-function-declaration]
rte_spinlock_lock(&dump_lock);
^~~~~~~~~~~~~~~~~
rte_calloc_socket
../app/test/test_ring_stress.h:166:4: warning: nested extern declaration
of ‘rte_spinlock_lock’ [-Wnested-externs]
../app/test/test_ring_stress.h:172:4: warning: implicit declaration of
function ‘rte_spinlock_unlock’; did you mean ‘pthread_rwlock_unlock’?
[-Wimplicit-function-declaration]
rte_spinlock_unlock(&dump_lock);
^~~~~~~~~~~~~~~~~~~
pthread_rwlock_unlock
../app/test/test_ring_stress.h:172:4: warning: nested extern declaration
of ‘rte_spinlock_unlock’ [-Wnested-externs]
Fixed by adding "#include <rte_spinlock.h>" to test_ring_stress.h.
Autoperf test results
---------------------
RTE>>ring_perf_autotest // DPDK 20.02, without patch
### Testing single element enq/deq ###
legacy APIs: SP/SC: single: 42.14
legacy APIs: MP/MC: single: 56.26
### Testing burst enq/deq ###
legacy APIs: SP/SC: burst (size: 8): 43.59
legacy APIs: SP/SC: burst (size: 32): 49.87
legacy APIs: MP/MC: burst (size: 8): 58.43
legacy APIs: MP/MC: burst (size: 32): 65.68
### Testing bulk enq/deq ###
legacy APIs: SP/SC: bulk (size: 8): 43.59
legacy APIs: SP/SC: bulk (size: 32): 49.85
legacy APIs: MP/MC: bulk (size: 8): 58.43
legacy APIs: MP/MC: bulk (size: 32): 65.60
### Testing empty bulk deq ###
legacy APIs: SP/SC: bulk (size: 8): 7.16
legacy APIs: MP/MC: bulk (size: 8): 7.16
### Testing using two hyperthreads ###
legacy APIs: SP/SC: bulk (size: 8): 12.46
legacy APIs: MP/MC: bulk (size: 8): 16.20
legacy APIs: SP/SC: bulk (size: 32): 3.21
legacy APIs: MP/MC: bulk (size: 32): 3.73
### Testing using two physical cores ###
legacy APIs: SP/SC: bulk (size: 8): 33.34
legacy APIs: MP/MC: bulk (size: 8): 37.99
legacy APIs: SP/SC: bulk (size: 32): 10.19
legacy APIs: MP/MC: bulk (size: 32): 11.90
### Testing using two NUMA nodes ###
legacy APIs: SP/SC: bulk (size: 8): 49.50
legacy APIs: MP/MC: bulk (size: 8): 63.65
legacy APIs: SP/SC: bulk (size: 32): 12.49
legacy APIs: MP/MC: bulk (size: 32): 23.53
### Testing using all slave nodes ###
Bulk enq/dequeue count on size 8
Core [4] count = 5604
Core [5] count = 5563
Core [6] count = 5576
Core [7] count = 5630
Core [8] count = 5643
Core [9] count = 5727
Core [10] count = 5698
Core [11] count = 5711
Core [64] count = 5259
Core [65] count = 5322
Core [66] count = 5321
Core [67] count = 5310
Core [68] count = 4350
Core [69] count = 4455
Core [70] count = 4546
Core [71] count = 4475
Total count (size: 8): 84190
Bulk enq/dequeue count on size 32
Core [4] count = 5543
Core [5] count = 5555
Core [6] count = 5596
Core [7] count = 5584
Core [8] count = 5613
Core [9] count = 5686
Core [10] count = 5689
Core [11] count = 5677
Core [64] count = 5228
Core [65] count = 5389
Core [66] count = 5406
Core [67] count = 5359
Core [68] count = 4554
Core [69] count = 4673
Core [70] count = 4675
Core [71] count = 4644
Total count (size: 32): 169061
### Testing single element enq/deq ###
elem APIs: element size 16B: SP/SC: single: 42.84
elem APIs: element size 16B: MP/MC: single: 56.77
### Testing burst enq/deq ###
elem APIs: element size 16B: SP/SC: burst (size: 8): 44.98
elem APIs: element size 16B: SP/SC: burst (size: 32): 59.16
elem APIs: element size 16B: MP/MC: burst (size: 8): 60.58
elem APIs: element size 16B: MP/MC: burst (size: 32): 74.75
### Testing bulk enq/deq ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 45.01
elem APIs: element size 16B: SP/SC: bulk (size: 32): 59.08
elem APIs: element size 16B: MP/MC: bulk (size: 8): 60.58
elem APIs: element size 16B: MP/MC: bulk (size: 32): 74.76
### Testing empty bulk deq ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 7.16
elem APIs: element size 16B: MP/MC: bulk (size: 8): 7.16
### Testing using two hyperthreads ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 12.18
elem APIs: element size 16B: MP/MC: bulk (size: 8): 15.44
elem APIs: element size 16B: SP/SC: bulk (size: 32): 3.22
elem APIs: element size 16B: MP/MC: bulk (size: 32): 3.97
### Testing using two physical cores ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 42.07
elem APIs: element size 16B: MP/MC: bulk (size: 8): 44.50
elem APIs: element size 16B: SP/SC: bulk (size: 32): 10.73
elem APIs: element size 16B: MP/MC: bulk (size: 32): 11.73
### Testing using two NUMA nodes ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 49.55
elem APIs: element size 16B: MP/MC: bulk (size: 8): 93.10
elem APIs: element size 16B: SP/SC: bulk (size: 32): 12.33
elem APIs: element size 16B: MP/MC: bulk (size: 32): 27.10
### Testing using all slave nodes ###
Bulk enq/dequeue count on size 8
Core [4] count = 5489
Core [5] count = 5559
Core [6] count = 5566
Core [7] count = 5577
Core [8] count = 5645
Core [9] count = 5699
Core [10] count = 5695
Core [11] count = 5733
Core [64] count = 5202
Core [65] count = 5284
Core [66] count = 5319
Core [67] count = 5349
Core [68] count = 4331
Core [69] count = 4465
Core [70] count = 4484
Core [71] count = 4439
Total count (size: 8): 83836
Bulk enq/dequeue count on size 32
Core [4] count = 5567
Core [5] count = 5492
Core [6] count = 5517
Core [7] count = 5515
Core [8] count = 5593
Core [9] count = 5650
Core [10] count = 5694
Core [11] count = 5665
Core [64] count = 5236
Core [65] count = 5319
Core [66] count = 5333
Core [67] count = 5304
Core [68] count = 4608
Core [69] count = 4669
Core [70] count = 4690
Core [71] count = 4654
Total count (size: 32): 168342
Test OK
---------------------
RTE>>ring_perf_autotest // DPDK 20.02, without patch
### Testing single element enq/deq ###
legacy APIs: SP/SC: single: 42.18
legacy APIs: MP/MC: single: 56.26
### Testing burst enq/deq ###
legacy APIs: SP/SC: burst (size: 8): 43.60
legacy APIs: SP/SC: burst (size: 32): 49.86
legacy APIs: MP/MC: burst (size: 8): 58.43
legacy APIs: MP/MC: burst (size: 32): 65.67
### Testing bulk enq/deq ###
legacy APIs: SP/SC: bulk (size: 8): 43.59
legacy APIs: SP/SC: bulk (size: 32): 49.86
legacy APIs: MP/MC: bulk (size: 8): 58.43
legacy APIs: MP/MC: bulk (size: 32): 65.63
### Testing empty bulk deq ###
legacy APIs: SP/SC: bulk (size: 8): 7.16
legacy APIs: MP/MC: bulk (size: 8): 7.16
### Testing using two hyperthreads ###
legacy APIs: SP/SC: bulk (size: 8): 12.07
legacy APIs: MP/MC: bulk (size: 8): 16.24
legacy APIs: SP/SC: bulk (size: 32): 3.20
legacy APIs: MP/MC: bulk (size: 32): 3.72
### Testing using two physical cores ###
legacy APIs: SP/SC: bulk (size: 8): 33.41
legacy APIs: MP/MC: bulk (size: 8): 38.01
legacy APIs: SP/SC: bulk (size: 32): 10.23
legacy APIs: MP/MC: bulk (size: 32): 11.90
### Testing using two NUMA nodes ###
legacy APIs: SP/SC: bulk (size: 8): 49.27
legacy APIs: MP/MC: bulk (size: 8): 64.80
legacy APIs: SP/SC: bulk (size: 32): 12.45
legacy APIs: MP/MC: bulk (size: 32): 23.11
### Testing using all slave nodes ###
Bulk enq/dequeue count on size 8
Core [4] count = 5637
Core [5] count = 5599
Core [6] count = 5623
Core [7] count = 5627
Core [8] count = 5723
Core [9] count = 5758
Core [10] count = 5714
Core [11] count = 5724
Core [64] count = 5310
Core [65] count = 5438
Core [66] count = 5448
Core [67] count = 5374
Core [68] count = 4441
Core [69] count = 4550
Core [70] count = 4550
Core [71] count = 4558
Total count (size: 8): 85074
Bulk enq/dequeue count on size 32
Core [4] count = 5608
Core [5] count = 5623
Core [6] count = 5590
Core [7] count = 5658
Core [8] count = 5680
Core [9] count = 5738
Core [10] count = 5692
Core [11] count = 5712
Core [64] count = 5273
Core [65] count = 5363
Core [66] count = 5341
Core [67] count = 5349
Core [68] count = 4591
Core [69] count = 4673
Core [70] count = 4698
Core [71] count = 4687
Total count (size: 32): 170350
### Testing single element enq/deq ###
elem APIs: element size 16B: SP/SC: single: 42.82
elem APIs: element size 16B: MP/MC: single: 56.79
### Testing burst enq/deq ###
elem APIs: element size 16B: SP/SC: burst (size: 8): 44.99
elem APIs: element size 16B: SP/SC: burst (size: 32): 59.00
elem APIs: element size 16B: MP/MC: burst (size: 8): 60.59
elem APIs: element size 16B: MP/MC: burst (size: 32): 74.78
### Testing bulk enq/deq ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 44.97
elem APIs: element size 16B: SP/SC: bulk (size: 32): 58.91
elem APIs: element size 16B: MP/MC: bulk (size: 8): 60.60
elem APIs: element size 16B: MP/MC: bulk (size: 32): 74.61
### Testing empty bulk deq ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 7.16
elem APIs: element size 16B: MP/MC: bulk (size: 8): 7.16
### Testing using two hyperthreads ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 12.18
elem APIs: element size 16B: MP/MC: bulk (size: 8): 15.41
elem APIs: element size 16B: SP/SC: bulk (size: 32): 3.19
elem APIs: element size 16B: MP/MC: bulk (size: 32): 4.06
### Testing using two physical cores ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 42.08
elem APIs: element size 16B: MP/MC: bulk (size: 8): 44.52
elem APIs: element size 16B: SP/SC: bulk (size: 32): 10.73
elem APIs: element size 16B: MP/MC: bulk (size: 32): 12.39
### Testing using two NUMA nodes ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 49.65
elem APIs: element size 16B: MP/MC: bulk (size: 8): 93.27
elem APIs: element size 16B: SP/SC: bulk (size: 32): 12.38
elem APIs: element size 16B: MP/MC: bulk (size: 32): 27.19
### Testing using all slave nodes ###
Bulk enq/dequeue count on size 8
Core [4] count = 5629
Core [5] count = 5585
Core [6] count = 5676
Core [7] count = 5604
Core [8] count = 5639
Core [9] count = 5731
Core [10] count = 5694
Core [11] count = 5707
Core [64] count = 5254
Core [65] count = 5331
Core [66] count = 5340
Core [67] count = 5355
Core [68] count = 4339
Core [69] count = 4481
Core [70] count = 4504
Core [71] count = 4507
Total count (size: 8): 84376
Bulk enq/dequeue count on size 32
Core [4] count = 5518
Core [5] count = 5493
Core [6] count = 5559
Core [7] count = 5484
Core [8] count = 5623
Core [9] count = 5669
Core [10] count = 5661
Core [11] count = 5658
Core [64] count = 5207
Core [65] count = 5305
Core [66] count = 5273
Core [67] count = 5303
Core [68] count = 4542
Core [69] count = 4682
Core [70] count = 4672
Core [71] count = 4609
Total count (size: 32): 168634
Test OK
Dave
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [dpdk v1] net/mlx5: fix possible building error
@ 2020-03-02 0:22 3% ` Tonghao Zhang
2020-03-02 7:35 0% ` Ali Alnubani
0 siblings, 1 reply; 200+ results
From: Tonghao Zhang @ 2020-03-02 0:22 UTC (permalink / raw)
To: Ali Alnubani; +Cc: stable, Matan Azrad, dev
On Mon, Mar 2, 2020 at 1:09 AM Ali Alnubani <alialnu@mellanox.com> wrote:
>
> Hi Tonghao,
>
> > -----Original Message-----
> > From: dev <dev-bounces@dpdk.org> On Behalf Of
> > xiangxia.m.yue@gmail.com
> > Sent: Saturday, February 29, 2020 12:27 PM
> > To: Matan Azrad <matan@mellanox.com>; dev@dpdk.org
> > Cc: Tonghao Zhang <xiangxia.m.yue@gmail.com>; stable@dpdk.org
> > Subject: [dpdk-dev] [dpdk v1] net/mlx5: fix possible building error
> >
> > From: Tonghao Zhang <xiangxia.m.yue@gmail.com>
> >
> > When setting the CONFIG_RTE_BUILD_SHARED_LIB to y, and build the mlx5
> > pmd, there is a building error.
> > To fix it, add RTE_IBVERBS_LINK_DLOPEN to include relative codes.
> >
> > > mlx5_common.o: In function `mlx5_glue_init':
> > > drivers/common/mlx5/mlx5_common.c:324: undefined reference to
> > `dlclose'
> >
> > Fixes: 7b4f1e6bd367 ("common/mlx5: introduce common library")
> > Cc: stable@dpdk.org
> >
> > Signed-off-by: Tonghao Zhang <xiangxia.m.yue@gmail.com>
> > ---
> > drivers/common/mlx5/mlx5_common.c | 5 +++--
> > 1 file changed, 3 insertions(+), 2 deletions(-)
> >
>
> I test this case on multiple environments and don't see the build failure. Can you please provide more information about yours, and how to reproduce this?
Hi Ali,
I use the dpdk-next-net, and the last commit id is
d7142fbae16f185e11bfa44be061399afc40a1be.
The build config is show as below:
diff --git a/config/common_base b/config/common_base
index 7ca2f28..4bd5126 100644
--- a/config/common_base
+++ b/config/common_base
@@ -57,7 +57,7 @@ CONFIG_RTE_ENABLE_LTO=n
#
# Compile to share library
#
-CONFIG_RTE_BUILD_SHARED_LIB=n
+CONFIG_RTE_BUILD_SHARED_LIB=y
#
# Use newest code breaking previous ABI
@@ -363,7 +363,7 @@ CONFIG_RTE_LIBRTE_MLX4_DEBUG=n
# Compile burst-oriented Mellanox ConnectX-4, ConnectX-5,
# ConnectX-6 & BlueField (MLX5) PMD
#
-CONFIG_RTE_LIBRTE_MLX5_PMD=n
+CONFIG_RTE_LIBRTE_MLX5_PMD=y
CONFIG_RTE_LIBRTE_MLX5_DEBUG=n
#
The building failure log show:
LD librte_common_mlx5.so.20.0.1
mlx5_common.o: In function `mlx5_glue_init':
/root/zth/myshell/dpdk-next-net/drivers/common/mlx5/mlx5_common.c:324:
undefined reference to `dlclose'
collect2: error: ld returned 1 exit status
> Thanks,
> Ali
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [dpdk v1] net/mlx5: fix possible building error
2020-03-02 0:22 3% ` Tonghao Zhang
@ 2020-03-02 7:35 0% ` Ali Alnubani
2020-03-02 8:13 0% ` Tonghao Zhang
0 siblings, 1 reply; 200+ results
From: Ali Alnubani @ 2020-03-02 7:35 UTC (permalink / raw)
To: Tonghao Zhang; +Cc: stable, Matan Azrad, dev
> -----Original Message-----
> From: Tonghao Zhang <xiangxia.m.yue@gmail.com>
> Sent: Monday, March 2, 2020 2:23 AM
> To: Ali Alnubani <alialnu@mellanox.com>
> Cc: stable@dpdk.org; Matan Azrad <matan@mellanox.com>; dev@dpdk.org
> Subject: Re: [dpdk-dev] [dpdk v1] net/mlx5: fix possible building error
>
> On Mon, Mar 2, 2020 at 1:09 AM Ali Alnubani <alialnu@mellanox.com>
> wrote:
> >
> > Hi Tonghao,
> >
> > > -----Original Message-----
> > > From: dev <dev-bounces@dpdk.org> On Behalf Of
> > > xiangxia.m.yue@gmail.com
> > > Sent: Saturday, February 29, 2020 12:27 PM
> > > To: Matan Azrad <matan@mellanox.com>; dev@dpdk.org
> > > Cc: Tonghao Zhang <xiangxia.m.yue@gmail.com>; stable@dpdk.org
> > > Subject: [dpdk-dev] [dpdk v1] net/mlx5: fix possible building error
> > >
> > > From: Tonghao Zhang <xiangxia.m.yue@gmail.com>
> > >
> > > When setting the CONFIG_RTE_BUILD_SHARED_LIB to y, and build the
> > > mlx5 pmd, there is a building error.
> > > To fix it, add RTE_IBVERBS_LINK_DLOPEN to include relative codes.
> > >
> > > > mlx5_common.o: In function `mlx5_glue_init':
> > > > drivers/common/mlx5/mlx5_common.c:324: undefined reference to
> > > `dlclose'
> > >
> > > Fixes: 7b4f1e6bd367 ("common/mlx5: introduce common library")
> > > Cc: stable@dpdk.org
> > >
> > > Signed-off-by: Tonghao Zhang <xiangxia.m.yue@gmail.com>
> > > ---
> > > drivers/common/mlx5/mlx5_common.c | 5 +++--
> > > 1 file changed, 3 insertions(+), 2 deletions(-)
> > >
> >
> > I test this case on multiple environments and don't see the build failure.
> Can you please provide more information about yours, and how to
> reproduce this?
> Hi Ali,
> I use the dpdk-next-net, and the last commit id is
> d7142fbae16f185e11bfa44be061399afc40a1be.
> The build config is show as below:
>
> diff --git a/config/common_base b/config/common_base index
> 7ca2f28..4bd5126 100644
> --- a/config/common_base
> +++ b/config/common_base
> @@ -57,7 +57,7 @@ CONFIG_RTE_ENABLE_LTO=n # # Compile to share
> library # -CONFIG_RTE_BUILD_SHARED_LIB=n
> +CONFIG_RTE_BUILD_SHARED_LIB=y
>
> #
> # Use newest code breaking previous ABI @@ -363,7 +363,7 @@
> CONFIG_RTE_LIBRTE_MLX4_DEBUG=n # Compile burst-oriented Mellanox
> ConnectX-4, ConnectX-5, # ConnectX-6 & BlueField (MLX5) PMD # -
> CONFIG_RTE_LIBRTE_MLX5_PMD=n
> +CONFIG_RTE_LIBRTE_MLX5_PMD=y
> CONFIG_RTE_LIBRTE_MLX5_DEBUG=n
>
> #
>
> The building failure log show:
> LD librte_common_mlx5.so.20.0.1
> mlx5_common.o: In function `mlx5_glue_init':
> /root/zth/myshell/dpdk-next-
> net/drivers/common/mlx5/mlx5_common.c:324:
> undefined reference to `dlclose'
> collect2: error: ld returned 1 exit status
>
Do you mind sharing which OS are you using, which gcc/clang version, whether you have MLNX_OFED or rdma-core installed, and which version?
Thanks,
Ali
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [dpdk v1] net/mlx5: fix possible building error
2020-03-02 7:35 0% ` Ali Alnubani
@ 2020-03-02 8:13 0% ` Tonghao Zhang
0 siblings, 0 replies; 200+ results
From: Tonghao Zhang @ 2020-03-02 8:13 UTC (permalink / raw)
To: Ali Alnubani; +Cc: stable, Matan Azrad, dev
On Mon, Mar 2, 2020 at 3:35 PM Ali Alnubani <alialnu@mellanox.com> wrote:
>
> > -----Original Message-----
> > From: Tonghao Zhang <xiangxia.m.yue@gmail.com>
> > Sent: Monday, March 2, 2020 2:23 AM
> > To: Ali Alnubani <alialnu@mellanox.com>
> > Cc: stable@dpdk.org; Matan Azrad <matan@mellanox.com>; dev@dpdk.org
> > Subject: Re: [dpdk-dev] [dpdk v1] net/mlx5: fix possible building error
> >
> > On Mon, Mar 2, 2020 at 1:09 AM Ali Alnubani <alialnu@mellanox.com>
> > wrote:
> > >
> > > Hi Tonghao,
> > >
> > > > -----Original Message-----
> > > > From: dev <dev-bounces@dpdk.org> On Behalf Of
> > > > xiangxia.m.yue@gmail.com
> > > > Sent: Saturday, February 29, 2020 12:27 PM
> > > > To: Matan Azrad <matan@mellanox.com>; dev@dpdk.org
> > > > Cc: Tonghao Zhang <xiangxia.m.yue@gmail.com>; stable@dpdk.org
> > > > Subject: [dpdk-dev] [dpdk v1] net/mlx5: fix possible building error
> > > >
> > > > From: Tonghao Zhang <xiangxia.m.yue@gmail.com>
> > > >
> > > > When setting the CONFIG_RTE_BUILD_SHARED_LIB to y, and build the
> > > > mlx5 pmd, there is a building error.
> > > > To fix it, add RTE_IBVERBS_LINK_DLOPEN to include relative codes.
> > > >
> > > > > mlx5_common.o: In function `mlx5_glue_init':
> > > > > drivers/common/mlx5/mlx5_common.c:324: undefined reference to
> > > > `dlclose'
> > > >
> > > > Fixes: 7b4f1e6bd367 ("common/mlx5: introduce common library")
> > > > Cc: stable@dpdk.org
> > > >
> > > > Signed-off-by: Tonghao Zhang <xiangxia.m.yue@gmail.com>
> > > > ---
> > > > drivers/common/mlx5/mlx5_common.c | 5 +++--
> > > > 1 file changed, 3 insertions(+), 2 deletions(-)
> > > >
> > >
> > > I test this case on multiple environments and don't see the build failure.
> > Can you please provide more information about yours, and how to
> > reproduce this?
> > Hi Ali,
> > I use the dpdk-next-net, and the last commit id is
> > d7142fbae16f185e11bfa44be061399afc40a1be.
> > The build config is show as below:
> >
> > diff --git a/config/common_base b/config/common_base index
> > 7ca2f28..4bd5126 100644
> > --- a/config/common_base
> > +++ b/config/common_base
> > @@ -57,7 +57,7 @@ CONFIG_RTE_ENABLE_LTO=n # # Compile to share
> > library # -CONFIG_RTE_BUILD_SHARED_LIB=n
> > +CONFIG_RTE_BUILD_SHARED_LIB=y
> >
> > #
> > # Use newest code breaking previous ABI @@ -363,7 +363,7 @@
> > CONFIG_RTE_LIBRTE_MLX4_DEBUG=n # Compile burst-oriented Mellanox
> > ConnectX-4, ConnectX-5, # ConnectX-6 & BlueField (MLX5) PMD # -
> > CONFIG_RTE_LIBRTE_MLX5_PMD=n
> > +CONFIG_RTE_LIBRTE_MLX5_PMD=y
> > CONFIG_RTE_LIBRTE_MLX5_DEBUG=n
> >
> > #
> >
> > The building failure log show:
> > LD librte_common_mlx5.so.20.0.1
> > mlx5_common.o: In function `mlx5_glue_init':
> > /root/zth/myshell/dpdk-next-
> > net/drivers/common/mlx5/mlx5_common.c:324:
> > undefined reference to `dlclose'
> > collect2: error: ld returned 1 exit status
> >
>
> Do you mind sharing which OS are you using, which gcc/clang version, whether you have MLNX_OFED or rdma-core installed, and which version?
1. CentOS Linux release 7.6.1810 (Core)
2. kernel 3.10.0-957.1.3.el7.x86_64
3. OFED 4.7-3.2.9
4. FW 16.26.4012
5. gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-36)
6. build the dpdk with x86_64-native-linuxapp-gcc
> Thanks,
> Ali
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH 00/16] NXP DPAAx fixes and enhancements
@ 2020-03-02 13:01 3% ` David Marchand
2020-03-05 9:06 3% ` Hemant Agrawal (OSS)
1 sibling, 1 reply; 200+ results
From: David Marchand @ 2020-03-02 13:01 UTC (permalink / raw)
To: Hemant Agrawal; +Cc: Yigit, Ferruh, dev
On Mon, Mar 2, 2020 at 10:26 AM Hemant Agrawal <hemant.agrawal@nxp.com> wrote:
>
> This patch series add various patches for enhancing and fixing NXP
> fslmc bus, dpaa bus, and dpaax.
>
> - the main change is support to allow thread migration across lcores
> - improving the multi-process support
This series triggers an ABI warning that must be investigated.
https://travis-ci.com/ovsrobot/dpdk/jobs/292904119#L2233
--
David Marchand
^ permalink raw reply [relevance 3%]
* [dpdk-dev] [PATCH v3] doc: plan splitting the ethdev ops struct
2020-02-25 12:44 3% ` [dpdk-dev] [PATCH v2] " Ferruh Yigit
2020-02-25 15:51 3% ` Andrew Rybchenko
2020-02-25 18:13 0% ` David Marchand
@ 2020-03-04 9:57 4% ` Ferruh Yigit
2 siblings, 0 replies; 200+ results
From: Ferruh Yigit @ 2020-03-04 9:57 UTC (permalink / raw)
To: Neil Horman, John McNamara, Marko Kovacevic
Cc: dev, Ferruh Yigit, Jerin Jacob Kollanukkaran, David Marchand,
Thomas Monjalon, Andrew Rybchenko
For the ABI compatibility it is better to hide internal data structures
from the application as much as possible. But because of some inline
functions 'struct eth_dev_ops' can't be hidden completely.
Plan is to split the 'struct eth_dev_ops' into two as ones used by
inline functions and ones not used, and hide the second part that not
used by inline functions completely to the application.
Because of ABI break the work will be done in 20.11
Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
---
Cc: David Marchand <david.marchand@redhat.com>
Cc: Thomas Monjalon <thomas@monjalon.net>
Cc: Andrew Rybchenko <arybchenko@solarflare.com>
v2:
* Add target date for the work
* Give more detail on what will be done
v3:
* Drop the interim implementation, and target 20.11 with ABI break.
---
doc/guides/rel_notes/deprecation.rst | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
index 1339f54f5..0bb252f71 100644
--- a/doc/guides/rel_notes/deprecation.rst
+++ b/doc/guides/rel_notes/deprecation.rst
@@ -93,6 +93,15 @@ Deprecation Notices
In 19.11 PMDs will still update the field even when the offload is not
enabled.
+* ethdev: Split the ``struct eth_dev_ops`` struct to hide it as much as possible
+ will be done in 20.11.
+ Currently the ``struct eth_dev_ops`` struct is accessible by the application
+ because some inline functions, like ``rte_eth_tx_descriptor_status()``,
+ access the struct directly.
+ The struct will be separate in two, the ops used by inline functions will be moved
+ next to Rx/Tx burst functions, rest of the ``struct eth_dev_ops`` struct will be
+ moved to header file for drivers to hide it from applications.
+
* cryptodev: support for using IV with all sizes is added, J0 still can
be used but only when IV length in following structs ``rte_crypto_auth_xform``,
``rte_crypto_aead_xform`` is set to zero. When IV length is greater or equal
--
2.24.1
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v2 0/7] vfio/pci: SR-IOV support
2020-02-19 18:53 2% [dpdk-dev] [PATCH v2 0/7] vfio/pci: SR-IOV support Alex Williamson
2020-02-25 2:33 0% ` Tian, Kevin
@ 2020-03-05 6:38 0% ` Vamsi Krishna Attunuru
1 sibling, 0 replies; 200+ results
From: Vamsi Krishna Attunuru @ 2020-03-05 6:38 UTC (permalink / raw)
To: Alex Williamson, kvm
Cc: linux-pci, linux-kernel, dev, mtosatti, thomas, bluca,
jerinjacobk, bruce.richardson, cohuck
> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Alex Williamson
> Sent: Thursday, February 20, 2020 12:24 AM
> To: kvm@vger.kernel.org
> Cc: linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; dev@dpdk.org;
> mtosatti@redhat.com; thomas@monjalon.net; bluca@debian.org;
> jerinjacobk@gmail.com; bruce.richardson@intel.com; cohuck@redhat.com
> Subject: [dpdk-dev] [PATCH v2 0/7] vfio/pci: SR-IOV support
>
> Changes since v1 are primarily to patch 3/7 where the commit log is
> rewritten, along with option parsing and failure logging based on upstream
> discussions. The primary user visible difference is that option parsing is now
> much more strict. If a vf_token option is provided that cannot be used, we
> generate an error. As a result of this, opening a PF with a vf_token option
> will serve as a mechanism of setting the vf_token. This seems like a more
> user friendly API than the alternative of sometimes requiring the option (VFs
> in use) and sometimes rejecting it, and upholds our desire that the option is
> always either used or rejected.
>
> This also means that the VFIO_DEVICE_FEATURE ioctl is not the only means
> of setting the VF token, which might call into question whether we absolutely
> need this new ioctl. Currently I'm keeping it because I can imagine use cases,
> for example if a hypervisor were to support SR-IOV, the PF device might be
> opened without consideration for a VF token and we'd require the
> hypservisor to close and re-open the PF in order to set a known VF token,
> which is impractical.
>
> Series overview (same as provided with v1):
>
> The synopsis of this series is that we have an ongoing desire to drive PCIe SR-
> IOV PFs from userspace with VFIO. There's an immediate need for this with
> DPDK drivers and potentially interesting future use cases in virtualization.
> We've been reluctant to add this support previously due to the dependency
> and trust relationship between the VF device and PF driver. Minimally the PF
> driver can induce a denial of service to the VF, but depending on the specific
> implementation, the PF driver might also be responsible for moving data
> between VFs or have direct access to the state of the VF, including data or
> state otherwise private to the VF or VF driver.
>
> To help resolve these concerns, we introduce a VF token into the VFIO PCI
> ABI, which acts as a shared secret key between drivers. The userspace PF
> driver is required to set the VF token to a known value and userspace VF
> drivers are required to provide the token to access the VF device. If a PF
> driver is restarted with VF drivers in use, it must also provide the current
> token in order to prevent a rogue untrusted PF driver from replacing a known
> driver. The degree to which this new token is considered secret is left to the
> userspace drivers, the kernel intentionally provides no means to retrieve the
> current token.
>
> Note that the above token is only required for this new model where both
> the PF and VF devices are usable through vfio-pci. Existing models of VFIO
> drivers where the PF is used without SR-IOV enabled or the VF is bound to a
> userspace driver with an in-kernel, host PF driver are unaffected.
>
> The latter configuration above also highlights a new inverted scenario that is
> now possible, a userspace PF driver with in-kernel VF drivers.
> I believe this is a scenario that should be allowed, but should not be enabled
> by default. This series includes code to set a default driver_override for VFs
> sourced from a vfio-pci user owned PF, such that the VFs are also bound to
> vfio-pci. This model is compatible with tools like driverctl and allows the
> system administrator to decide if other bindings should be enabled. The VF
> token interface above exists only between vfio-pci PF and VF drivers, once a
> VF is bound to another driver, the administrator has effectively pronounced
> the device as trusted. The vfio-pci driver will note alternate binding in dmesg
> for logging and debugging purposes.
>
> Please review, comment, and test. The example QEMU implementation
> provided with the RFC is still current for this version. Thanks,
>
> Alex
Hi Alex,
Thanks for enabling this feature support.
Tested-by: Vamsi Attunuru <vattunuru@marvell.com>
Tested v2 patch set with below DPDK patch.
http://patches.dpdk.org/patch/66281/
Regards
A Vamsi
>
> RFC: https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__lore.kernel.org_lkml_158085337582.9445.17682266437583505502.stgit-
> 40gimli.home_&d=DwICaQ&c=nKjWec2b6R0mOyPaz7xtfQ&r=2rpxxNF2qeP0
> 2gVZIWTVrW-6zNZz5-uKt9pRqpR_M3U&m=V-6mKmCTHPZa5jwepXU_-
> Ma1_BGF0OWJ_IRCF_p4GVo&s=YnO98PGK9ro7F6_XZTccHdYcZ-
> rMMOin0nRFhPD6Uv4&e=
> v1: https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__lore.kernel.org_lkml_158145472604.16827.15751375540102298130.stgit
> -
> 40gimli.home_&d=DwICaQ&c=nKjWec2b6R0mOyPaz7xtfQ&r=2rpxxNF2qeP0
> 2gVZIWTVrW-6zNZz5-uKt9pRqpR_M3U&m=V-6mKmCTHPZa5jwepXU_-
> Ma1_BGF0OWJ_IRCF_p4GVo&s=rvUxLCENwNk0GBYkcsBVVobsLfMb4BV5gtc
> 3VqYQTS4&e=
>
> ---
>
> Alex Williamson (7):
> vfio: Include optional device match in vfio_device_ops callbacks
> vfio/pci: Implement match ops
> vfio/pci: Introduce VF token
> vfio: Introduce VFIO_DEVICE_FEATURE ioctl and first user
> vfio/pci: Add sriov_configure support
> vfio/pci: Remove dev_fmt definition
> vfio/pci: Cleanup .probe() exit paths
>
>
> drivers/vfio/pci/vfio_pci.c | 383
> +++++++++++++++++++++++++++++++++--
> drivers/vfio/pci/vfio_pci_private.h | 10 +
> drivers/vfio/vfio.c | 20 +-
> include/linux/vfio.h | 4
> include/uapi/linux/vfio.h | 37 +++
> 5 files changed, 426 insertions(+), 28 deletions(-)
^ permalink raw reply [relevance 0%]
* [dpdk-dev] DPDK techboard minutes of February 26
@ 2020-03-05 8:53 4% Thomas Monjalon
0 siblings, 0 replies; 200+ results
From: Thomas Monjalon @ 2020-03-05 8:53 UTC (permalink / raw)
To: dev; +Cc: techboard
Minutes of Technical Board meeting, 2020-02-26
Members Attending: 9/11
- Ferruh Yigit
- Hemant Agrawal
- Honnappa Nagarahalli
- Jerin Jacob
- Kevin Traynor
- Konstantin Ananyev
- Olivier Matz
- Stephen Hemminger
- Thomas Monjalon (Chair)
NOTE: Technical Board meetings happen every second Wednesday
on IRC channel #dpdk-board, at 3pm UTC.
Meetings are public and DPDK community members are welcome to attend.
1/ Technical Board status for the Governing Board meeting
- ABI compatibility and tooling
- SPDX conversion almost complete
- Stepping away from Linux out-of-tree modules - new dpdk-kmods repo
- Project scope discussion
- PMDT project - new acceptance guidelines
- Tooling trend (tracing, dumps, metrics, telemetry, monitoring)
- ICC support: warning is not fatal
- Windows port is progressing
- new device classes: vDPA and regex
- IPsec integration is difficult and progressing
- Security pre-release list is not growing a lot
- not tried any mentoring program this year
2/ Doc maintenance
Not much activity on documentation:
- no content improvement
- hard to get good reviews
Volunteers are welcome.
Maintenance of the documentation becomes a recurring techboard topic.
3/ Community lab goals
The Governing Board is evaluating how to progress with DPDK CI.
The Technical Board had to provide status and goals.
The first CI priority was already agreed in previous meeting:
basic ethdev functional testing on real NICs.
Coverity is doing static analysis regularly.
Travis is testing (requires integration for per-patch report):
- compilation
- ABI
- libraries (without traffic)
OBS could be integrated for per-patch compilation testing.
The community lab is testing (for each patch):
- compilation on more OSes
- libraries (without traffic)
- micro-benchmark with traffic on real HW
- downstream projects (OVS, SPDK)
Key benefits of the community lab are:
- real HW
- vendor neutrality
- confidence
The ideal lab should meet these requirements:
- reliable
- neutral
- transparent visibility
- coverage summary
- trending graphs
- easy to add/run a test
- easy to reproduce a test locally
- email reports
- per-patch testing integrated with patchwork
- per-commit testing (after merge in master or next-*)
- per-release (including -rc) testing
and should run these tests:
- compilation
- functional
- performance
- interoperability
- portability
- downstream projects
Three roles are identified.
1) Lab management: hosting, physical setup, monitoring.
2) Framework/Infrastructure (can be remote):
test integration, database, logs, reports, graphs, web pages.
3) development of test cases:
writing and/or evaluating test code for existing features.
Future features could be gated by test requirement.
The third role require community involvement and volunteering.
4/ rte_graph + DPDK scope
The new library rte_graph is accepted to be part of dpdk.org.
During the next meeting, it will be discussed whether
adding rte_graph and similar libraries in the main repository,
or splitting in different git repositories.
The question of possibly dead/unused code in some libraries
should be debated as well.
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH 00/16] NXP DPAAx fixes and enhancements
2020-03-02 13:01 3% ` David Marchand
@ 2020-03-05 9:06 3% ` Hemant Agrawal (OSS)
2020-03-05 9:09 3% ` David Marchand
0 siblings, 1 reply; 200+ results
From: Hemant Agrawal (OSS) @ 2020-03-05 9:06 UTC (permalink / raw)
To: David Marchand; +Cc: Yigit, Ferruh, dev
Hi David,
> On Mon, Mar 2, 2020 at 10:26 AM Hemant Agrawal
> <hemant.agrawal@nxp.com> wrote:
> >
> > This patch series add various patches for enhancing and fixing NXP
> > fslmc bus, dpaa bus, and dpaax.
> >
> > - the main change is support to allow thread migration across lcores
> > - improving the multi-process support
>
> This series triggers an ABI warning that must be investigated.
>https://travis-ci.com/ovsrobot/dpdk/jobs/292904119#L2233
[Hemant]
As per the logs:
Variables changes summary: 1 Removed, 2 Changed, 0 Added variables
1 Removed variable:
'dpaa2_portal_dqrr per_lcore_dpaa2_held_bufs' {per_lcore_dpaa2_held_bufs@@DPDK_20.0}
2 Changed variables:
[C]'dpaa2_io_portal_t dpaa2_io_portal[128]' was changed at dpaa2_hw_dpio.h:40:1: size of symbol changed from 5120 to 2048
[C]'dpaa2_io_portal_t per_lcore__dpaa2_io' was changed at dpaa2_hw_dpio.h:20:1: size of symbol changed from 40 to 16
Error: ABI issue reported for 'abidiff --suppr devtools/libabigail.abignore --no-added-syms --headers-dir1 reference/usr/local/include --headers-dir2 install/usr/local/include reference/dump/librte_bus_fslmc.dump install/dump/librte_bus_fslmc.dump'
---------------
These changes are w.r.t modifications in internal structures and variables. They may be ignored.
>
> --
> David Marchand
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH 00/16] NXP DPAAx fixes and enhancements
2020-03-05 9:06 3% ` Hemant Agrawal (OSS)
@ 2020-03-05 9:09 3% ` David Marchand
2020-03-05 9:19 0% ` Hemant Agrawal (OSS)
0 siblings, 1 reply; 200+ results
From: David Marchand @ 2020-03-05 9:09 UTC (permalink / raw)
To: Hemant Agrawal (OSS); +Cc: Yigit, Ferruh, dev
On Thu, Mar 5, 2020 at 10:06 AM Hemant Agrawal (OSS)
<hemant.agrawal@oss.nxp.com> wrote:
>
> Hi David,
> > On Mon, Mar 2, 2020 at 10:26 AM Hemant Agrawal
> > <hemant.agrawal@nxp.com> wrote:
> > >
> > > This patch series add various patches for enhancing and fixing NXP
> > > fslmc bus, dpaa bus, and dpaax.
> > >
> > > - the main change is support to allow thread migration across lcores
> > > - improving the multi-process support
> >
> > This series triggers an ABI warning that must be investigated.
> >https://travis-ci.com/ovsrobot/dpdk/jobs/292904119#L2233
>
> [Hemant]
> As per the logs:
>
> Variables changes summary: 1 Removed, 2 Changed, 0 Added variables
> 1 Removed variable:
> 'dpaa2_portal_dqrr per_lcore_dpaa2_held_bufs' {per_lcore_dpaa2_held_bufs@@DPDK_20.0}
> 2 Changed variables:
> [C]'dpaa2_io_portal_t dpaa2_io_portal[128]' was changed at dpaa2_hw_dpio.h:40:1: size of symbol changed from 5120 to 2048
> [C]'dpaa2_io_portal_t per_lcore__dpaa2_io' was changed at dpaa2_hw_dpio.h:20:1: size of symbol changed from 40 to 16
>
> Error: ABI issue reported for 'abidiff --suppr devtools/libabigail.abignore --no-added-syms --headers-dir1 reference/usr/local/include --headers-dir2 install/usr/local/include reference/dump/librte_bus_fslmc.dump install/dump/librte_bus_fslmc.dump'
>
> ---------------
>
> These changes are w.r.t modifications in internal structures and variables. They may be ignored.
The ABI check considers symbol exposed in headers available to final users.
If those are internal, why are the headers public?
--
David Marchand
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH 00/16] NXP DPAAx fixes and enhancements
2020-03-05 9:09 3% ` David Marchand
@ 2020-03-05 9:19 0% ` Hemant Agrawal (OSS)
2020-03-06 10:12 3% ` David Marchand
0 siblings, 1 reply; 200+ results
From: Hemant Agrawal (OSS) @ 2020-03-05 9:19 UTC (permalink / raw)
To: David Marchand, Hemant Agrawal (OSS); +Cc: Yigit, Ferruh, dev
Hi David,
> On Thu, Mar 5, 2020 at 10:06 AM Hemant Agrawal (OSS)
> <hemant.agrawal@oss.nxp.com> wrote:
> >
> > Hi David,
> > > On Mon, Mar 2, 2020 at 10:26 AM Hemant Agrawal
> > > <hemant.agrawal@nxp.com> wrote:
> > > >
> > > > This patch series add various patches for enhancing and fixing NXP
> > > > fslmc bus, dpaa bus, and dpaax.
> > > >
> > > > - the main change is support to allow thread migration across
> > > > lcores
> > > > - improving the multi-process support
> > >
> > > This series triggers an ABI warning that must be investigated.
> > >https://travis-ci.com/ovsrobot/dpdk/jobs/292904119#L2233
> >
> > [Hemant]
> > As per the logs:
> >
> > Variables changes summary: 1 Removed, 2 Changed, 0 Added variables
> > 1 Removed variable:
> > 'dpaa2_portal_dqrr per_lcore_dpaa2_held_bufs'
> {per_lcore_dpaa2_held_bufs@@DPDK_20.0}
> > 2 Changed variables:
> > [C]'dpaa2_io_portal_t dpaa2_io_portal[128]' was changed at
> dpaa2_hw_dpio.h:40:1: size of symbol changed from 5120 to 2048
> > [C]'dpaa2_io_portal_t per_lcore__dpaa2_io' was changed at
> > dpaa2_hw_dpio.h:20:1: size of symbol changed from 40 to 16
> >
> > Error: ABI issue reported for 'abidiff --suppr devtools/libabigail.abignore --
> no-added-syms --headers-dir1 reference/usr/local/include --headers-dir2
> install/usr/local/include reference/dump/librte_bus_fslmc.dump
> install/dump/librte_bus_fslmc.dump'
> >
> > ---------------
> >
> > These changes are w.r.t modifications in internal structures and variables.
> They may be ignored.
>
> The ABI check considers symbol exposed in headers available to final users.
> If those are internal, why are the headers public?
>
[Hemant] These symbols are not part of any public header files, but they are part of *.map files to share them between different driver libs i.e bus_fslmc and net_dpaa2
>
> --
> David Marchand
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH 11/16] net/dpaa: enable Tx queue taildrop
@ 2020-03-05 14:14 3% ` Ferruh Yigit
0 siblings, 0 replies; 200+ results
From: Ferruh Yigit @ 2020-03-05 14:14 UTC (permalink / raw)
To: Gagandeep Singh, Hemant Agrawal; +Cc: dev, David Marchand, Neil Horman
On 3/5/2020 6:49 AM, Gagandeep Singh wrote:
> Hi Ferruh,
>>> diff --git a/drivers/bus/dpaa/rte_bus_dpaa_version.map
>> b/drivers/bus/dpaa/rte_bus_dpaa_version.map
>>> index e6ca4361e..86f5811b0 100644
>>> --- a/drivers/bus/dpaa/rte_bus_dpaa_version.map
>>> +++ b/drivers/bus/dpaa/rte_bus_dpaa_version.map
>>> @@ -94,3 +94,10 @@ DPDK_20.0 {
>>>
>>> local: *;
>>> };
>>> +
>>> +EXPERIMENTAL {
>>> + global:
>>> +
>>> + qman_ern_poll_free;
>>> + qman_ern_register_cb;
>>> +};
>>
>> Aren't these bus APIs internal (between net/crypto/event drivers and bus
>> libraries)? And they are *not* for applications to use.
>>
>> If they are only internal is there any point to make them experimental?
>
> Yes, these are internal only APIS. So, is it ok to add them directly into DPDK_20.0 section?
>
I think it is OK, but I don't know the what will be the affect to the ABI check
tools.
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH v2 0/7] vfio/pci: SR-IOV support
2020-02-25 2:33 0% ` Tian, Kevin
@ 2020-03-05 17:33 3% ` Alex Williamson
2020-03-06 9:21 0% ` Tian, Kevin
0 siblings, 1 reply; 200+ results
From: Alex Williamson @ 2020-03-05 17:33 UTC (permalink / raw)
To: Tian, Kevin
Cc: kvm, linux-pci, linux-kernel, dev, mtosatti, thomas, bluca,
jerinjacobk, Richardson, Bruce, cohuck, Jason Wang
Hi Kevin,
Sorry for the delay, I've been out on PTO...
On Tue, 25 Feb 2020 02:33:27 +0000
"Tian, Kevin" <kevin.tian@intel.com> wrote:
> > From: Alex Williamson
> > Sent: Thursday, February 20, 2020 2:54 AM
> >
> > Changes since v1 are primarily to patch 3/7 where the commit log is
> > rewritten, along with option parsing and failure logging based on
> > upstream discussions. The primary user visible difference is that
> > option parsing is now much more strict. If a vf_token option is
> > provided that cannot be used, we generate an error. As a result of
> > this, opening a PF with a vf_token option will serve as a mechanism of
> > setting the vf_token. This seems like a more user friendly API than
> > the alternative of sometimes requiring the option (VFs in use) and
> > sometimes rejecting it, and upholds our desire that the option is
> > always either used or rejected.
> >
> > This also means that the VFIO_DEVICE_FEATURE ioctl is not the only
> > means of setting the VF token, which might call into question whether
> > we absolutely need this new ioctl. Currently I'm keeping it because I
> > can imagine use cases, for example if a hypervisor were to support
> > SR-IOV, the PF device might be opened without consideration for a VF
> > token and we'd require the hypservisor to close and re-open the PF in
> > order to set a known VF token, which is impractical.
> >
> > Series overview (same as provided with v1):
>
> Thanks for doing this!
>
> >
> > The synopsis of this series is that we have an ongoing desire to drive
> > PCIe SR-IOV PFs from userspace with VFIO. There's an immediate need
> > for this with DPDK drivers and potentially interesting future use
>
> Can you provide a link to the DPDK discussion?
There's a thread here which proposed an out-of-tree driver that enables
a parallel sr-iov enabling interface for a vfio-pci own device.
Clearly I felt strongly about it ;)
https://patches.dpdk.org/patch/58810/
Also, documentation for making use of an Intel FPGA device with DPDK
requires the PF bound to igb_uio to support enabling SR-IOV:
https://doc.dpdk.org/guides/bbdevs/fpga_lte_fec.html
> > cases in virtualization. We've been reluctant to add this support
> > previously due to the dependency and trust relationship between the
> > VF device and PF driver. Minimally the PF driver can induce a denial
> > of service to the VF, but depending on the specific implementation,
> > the PF driver might also be responsible for moving data between VFs
> > or have direct access to the state of the VF, including data or state
> > otherwise private to the VF or VF driver.
>
> Just a loud thinking. While the motivation of VF token sounds reasonable
> to me, I'm curious why the same concern is not raised in other usages.
> For example, there is no such design in virtio framework, where the
> virtio device could also be restarted, putting in separate process (vhost-user),
> and even in separate VM (virtio-vhost-user), etc. Of course the para-
> virtualized attribute of virtio implies some degree of trust, but as you
> mentioned many SR-IOV implementations support VF->PF communication
> which also implies some level of trust. It's perfectly fine if VFIO just tries
> to do better than other sub-systems, but knowing how other people
> tackle the similar problem may make the whole picture clearer. 😊
>
> +Jason.
We can follow the thread with Jason, but I can't really speak to
whether virtio needs something similar or doesn't provide enough PF
access to be concerned. If they need a similar solution, we can
collaborate, but the extension we're defining here is specifically part
of the vfio-pci ABI, so it might not be easily portable to virtio.
> > To help resolve these concerns, we introduce a VF token into the VFIO
> > PCI ABI, which acts as a shared secret key between drivers. The
> > userspace PF driver is required to set the VF token to a known value
> > and userspace VF drivers are required to provide the token to access
> > the VF device. If a PF driver is restarted with VF drivers in use, it
> > must also provide the current token in order to prevent a rogue
> > untrusted PF driver from replacing a known driver. The degree to
> > which this new token is considered secret is left to the userspace
> > drivers, the kernel intentionally provides no means to retrieve the
> > current token.
>
> I'm wondering whether the token idea can be used beyond SR-IOV, e.g.
> (1) we may allow vfio user space to manage Scalable IOV in the future,
> which faces the similar challenge between the PF and mdev; (2) the
> token might be used as a canonical way to replace off-tree acs-override
> workaround, say, allowing the admin to assign devices within the
> same iommu group to different VMs which trust each other. I'm not
> sure how much complexity will be further introduced, but it's greatly
> appreciated if you can help think a bit and if feasible abstract some
> logic in vfio core layer for such potential usages...
I don't see how this can be used for ACS override. Lacking ACS, we
must assume lack of DMA isolation, which results in our IOMMU grouping.
If we split IOMMU groups, that implies something that doesn't exist. A
user can already create a process that can own the vfio group and pass
vfio devices to other tasks, with the restriction of having a single
DMA address space. If there is DMA isolation, then an mdev solution
might be better, but given the IOMMU integration of SIOV, I'm not sure
why the devices wouldn't simply be placed in separate groups by the
IOMMU driver. Thanks,
Alex
> > Note that the above token is only required for this new model where
> > both the PF and VF devices are usable through vfio-pci. Existing
> > models of VFIO drivers where the PF is used without SR-IOV enabled
> > or the VF is bound to a userspace driver with an in-kernel, host PF
> > driver are unaffected.
> >
> > The latter configuration above also highlights a new inverted scenario
> > that is now possible, a userspace PF driver with in-kernel VF drivers.
> > I believe this is a scenario that should be allowed, but should not be
> > enabled by default. This series includes code to set a default
> > driver_override for VFs sourced from a vfio-pci user owned PF, such
> > that the VFs are also bound to vfio-pci. This model is compatible
> > with tools like driverctl and allows the system administrator to
> > decide if other bindings should be enabled. The VF token interface
> > above exists only between vfio-pci PF and VF drivers, once a VF is
> > bound to another driver, the administrator has effectively pronounced
> > the device as trusted. The vfio-pci driver will note alternate
> > binding in dmesg for logging and debugging purposes.
> >
> > Please review, comment, and test. The example QEMU implementation
> > provided with the RFC is still current for this version. Thanks,
> >
> > Alex
> >
> > RFC:
> > https://lore.kernel.org/lkml/158085337582.9445.17682266437583505502.stg
> > it@gimli.home/
> > v1:
> > https://lore.kernel.org/lkml/158145472604.16827.15751375540102298130.st
> > git@gimli.home/
> >
> > ---
> >
> > Alex Williamson (7):
> > vfio: Include optional device match in vfio_device_ops callbacks
> > vfio/pci: Implement match ops
> > vfio/pci: Introduce VF token
> > vfio: Introduce VFIO_DEVICE_FEATURE ioctl and first user
> > vfio/pci: Add sriov_configure support
> > vfio/pci: Remove dev_fmt definition
> > vfio/pci: Cleanup .probe() exit paths
> >
> >
> > drivers/vfio/pci/vfio_pci.c | 383
> > +++++++++++++++++++++++++++++++++--
> > drivers/vfio/pci/vfio_pci_private.h | 10 +
> > drivers/vfio/vfio.c | 20 +-
> > include/linux/vfio.h | 4
> > include/uapi/linux/vfio.h | 37 +++
> > 5 files changed, 426 insertions(+), 28 deletions(-)
>
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH 00/16] NXP DPAAx fixes and enhancements
2020-03-05 9:19 0% ` Hemant Agrawal (OSS)
@ 2020-03-06 10:12 3% ` David Marchand
2020-03-10 10:36 3% ` Dodji Seketeli
0 siblings, 1 reply; 200+ results
From: David Marchand @ 2020-03-06 10:12 UTC (permalink / raw)
To: Hemant Agrawal (OSS)
Cc: Yigit, Ferruh, dev, Neil Horman, Thomas Monjalon, Dodji Seketeli
On Thu, Mar 5, 2020 at 10:19 AM Hemant Agrawal (OSS)
<hemant.agrawal@oss.nxp.com> wrote:
> > On Thu, Mar 5, 2020 at 10:06 AM Hemant Agrawal (OSS)
> > <hemant.agrawal@oss.nxp.com> wrote:
> > >
> > > Hi David,
> > > > On Mon, Mar 2, 2020 at 10:26 AM Hemant Agrawal
> > > > <hemant.agrawal@nxp.com> wrote:
> > > > >
> > > > > This patch series add various patches for enhancing and fixing NXP
> > > > > fslmc bus, dpaa bus, and dpaax.
> > > > >
> > > > > - the main change is support to allow thread migration across
> > > > > lcores
> > > > > - improving the multi-process support
> > > >
> > > > This series triggers an ABI warning that must be investigated.
> > > >https://travis-ci.com/ovsrobot/dpdk/jobs/292904119#L2233
> > >
> > > [Hemant]
> > > As per the logs:
> > >
> > > Variables changes summary: 1 Removed, 2 Changed, 0 Added variables
> > > 1 Removed variable:
> > > 'dpaa2_portal_dqrr per_lcore_dpaa2_held_bufs'
> > {per_lcore_dpaa2_held_bufs@@DPDK_20.0}
> > > 2 Changed variables:
> > > [C]'dpaa2_io_portal_t dpaa2_io_portal[128]' was changed at
> > dpaa2_hw_dpio.h:40:1: size of symbol changed from 5120 to 2048
> > > [C]'dpaa2_io_portal_t per_lcore__dpaa2_io' was changed at
> > > dpaa2_hw_dpio.h:20:1: size of symbol changed from 40 to 16
> > >
> > > Error: ABI issue reported for 'abidiff --suppr devtools/libabigail.abignore --
> > no-added-syms --headers-dir1 reference/usr/local/include --headers-dir2
> > install/usr/local/include reference/dump/librte_bus_fslmc.dump
> > install/dump/librte_bus_fslmc.dump'
> > >
> > > ---------------
> > >
> > > These changes are w.r.t modifications in internal structures and variables.
> > They may be ignored.
> >
> > The ABI check considers symbol exposed in headers available to final users.
> > If those are internal, why are the headers public?
> >
>
> [Hemant] These symbols are not part of any public header files, but they are part of *.map files to share them between different driver libs i.e bus_fslmc and net_dpaa2
I would expect libabigail to skip those symbols, so there is something
I have missed in how --headers-dirX work.
Anyway, all of those symbols in dpaa are part of the driver ABI.
We are still missing a way to mark internal symbols.
Neil had posted a framework for this
http://patchwork.dpdk.org/project/dpdk/list/?series=5004.
In order to get this series passing the checks, I recommend NXP
rebasing Neil scripts (I will help reviewing this part), then mark all
those symbols as internal in its drivers.
Other vendor will convert their drivers later, as there is no need at
the moment.
Thanks.
--
David Marchand
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH dpdk-dev v3] mempool: sort the rte_mempool_ops by name
@ 2020-03-09 8:27 3% ` Olivier Matz
2020-03-09 8:55 0% ` Tonghao Zhang
2020-03-24 9:35 0% ` Andrew Rybchenko
0 siblings, 2 replies; 200+ results
From: Olivier Matz @ 2020-03-09 8:27 UTC (permalink / raw)
To: Tonghao Zhang
Cc: Andrew Rybchenko, Jerin Jacob, dpdk-dev, Gage Eads,
Artem V. Andreev, Jerin Jacob, Nithin Dabilpuram, Vamsi Attunuru,
Hemant Agrawal
Hi,
On Mon, Mar 09, 2020 at 11:01:25AM +0800, Tonghao Zhang wrote:
> On Sat, Mar 7, 2020 at 8:54 PM Andrew Rybchenko
> <arybchenko@solarflare.com> wrote:
> >
> > On 3/7/20 3:51 PM, Andrew Rybchenko wrote:
> > > On 3/6/20 4:37 PM, Jerin Jacob wrote:
> > >> On Fri, Mar 6, 2020 at 7:06 PM <xiangxia.m.yue@gmail.com> wrote:
> > >>> From: Tonghao Zhang <xiangxia.m.yue@gmail.com>
> > >>>
> > >>> The order of mempool initiation affects mempool index in the
> > >>> rte_mempool_ops_table. For example, when building APPs with:
> > >>>
> > >>> $ gcc -lrte_mempool_bucket -lrte_mempool_ring ...
> > >>>
> > >>> The "bucket" mempool will be registered firstly, and its index
> > >>> in table is 0 while the index of "ring" mempool is 1. DPDK
> > >>> uses the mk/rte.app.mk to build APPs, and others, for example,
> > >>> Open vSwitch, use the libdpdk.a or libdpdk.so to build it.
> > >>> The mempool lib linked in dpdk and Open vSwitch is different.
> > >>>
> > >>> The mempool can be used between primary and secondary process,
> > >>> such as dpdk-pdump and pdump-pmd/Open vSwitch(pdump enabled).
> > >>> There will be a crash because dpdk-pdump creates the "ring_mp_mc"
> > >>> ring which index in table is 0, but the index of "bucket" ring
> > >>> is 0 in Open vSwitch. If Open vSwitch use the index 0 to get
> > >>> mempool ops and malloc memory from mempool. The crash will occur:
> > >>>
> > >>> bucket_dequeue (access null and crash)
> > >>> rte_mempool_get_ops (should get "ring_mp_mc",
> > >>> but get "bucket" mempool)
> > >>> rte_mempool_ops_dequeue_bulk
> > >>> ...
> > >>> rte_pktmbuf_alloc
> > >>> rte_pktmbuf_copy
> > >>> pdump_copy
> > >>> pdump_rx
> > >>> rte_eth_rx_burst
> > >>>
> > >>> To avoid the crash, there are some solution:
> > >>> * constructor priority: Different mempool uses different
> > >>> priority in RTE_INIT, but it's not easy to maintain.
> > >>>
> > >>> * change mk/rte.app.mk: Change the order in mk/rte.app.mk to
> > >>> be same as libdpdk.a/libdpdk.so, but when adding a new mempool
> > >>> driver in future, we must make sure the order.
> > >>>
> > >>> * register mempool orderly: Sort the mempool when registering,
> > >>> so the lib linked will not affect the index in mempool table.
> > >>>
> > >>> Signed-off-by: Tonghao Zhang <xiangxia.m.yue@gmail.com>
> > >>> Acked-by: Olivier Matz <olivier.matz@6wind.com>
> > >> Acked-by: Jerin Jacob <jerinj@marvell.com>
> > >
> > > The patch is OK, but the fact that ops index changes during
> > > mempool driver lifetime is frightening. In fact it breaks
> > > rte_mempool_register_ops() return value semantics (read
> > > as API break). The return value is not used in DPDK, but it
> > > is a public function. If I'm not mistaken it should be taken
> > > into account.
Good points.
The fact that the ops index changes during mempool driver lifetime is
indeed frightening, especially knowning that this is a dynamic
registration that could happen at any moment in the life of the
application. Also, breaking the ABI is not desirable.
Let me try to propose something else to solve your issue:
1/ At init, the primary process allocates a struct in shared memory
(named memzone):
struct rte_mempool_shared_ops {
size_t num_mempool_ops;
struct {
char name[RTE_MEMPOOL_OPS_NAMESIZE];
} mempool_ops[RTE_MEMPOOL_MAX_OPS_IDX];
char *mempool_ops_name[RTE_MEMPOOL_MAX_OPS_IDX];
rte_spinlock_t mempool;
}
2/ When we register a mempool ops, we first get a name and id from the
shared struct: with the lock held, lookup for the registered name and
return its index, else get the last id and copy the name in the struct.
3/ Then do as before (in the per-process global table), except that we
reuse the registered id.
We can remove the num_ops field from rte_mempool_ops_table.
Thoughts?
> Yes, should update the doc: how about this:
>
> diff --git a/lib/librte_mempool/rte_mempool.h b/lib/librte_mempool/rte_mempool.h
> index c90cf31..5a9c8a7 100644
> --- a/lib/librte_mempool/rte_mempool.h
> +++ b/lib/librte_mempool/rte_mempool.h
> @@ -904,7 +904,9 @@ int rte_mempool_ops_get_info(const struct rte_mempool *mp,
> * @param ops
> * Pointer to an ops structure to register.
> * @return
> - * - >=0: Success; return the index of the ops struct in the table.
> + * - >=0: Success; return the index of the last ops struct in the table.
> + * The number of the ops struct registered is equal to index
> + * returned + 1.
> * - -EINVAL - some missing callbacks while registering ops struct.
> * - -ENOSPC - the maximum number of ops structs has been reached.
> */
> diff --git a/lib/librte_mempool/rte_mempool_ops.c
> b/lib/librte_mempool/rte_mempool_ops.c
> index b0da096..053f340 100644
> --- a/lib/librte_mempool/rte_mempool_ops.c
> +++ b/lib/librte_mempool/rte_mempool_ops.c
> @@ -26,7 +26,11 @@ struct rte_mempool_ops_table rte_mempool_ops_table = {
> return strcmp(m_a->name, m_b->name);
> }
>
> -/* add a new ops struct in rte_mempool_ops_table, return its index. */
> +/*
> + * add a new ops struct in rte_mempool_ops_table.
> + * on success, return the index of the last ops
> + * struct in the table.
> + */
> int
> rte_mempool_register_ops(const struct rte_mempool_ops *h)
> {
> > > Also I remember patches which warn about above behaviour
> > > in documentation. If behaviour changes, corresponding
> > > documentation must be updated.
> >
> > One more point. If the patch is finally accepted it definitely
> > deserves few lines in release notes.
> OK, a separate patch should be sent before DPDK 20.05 release ?
> >
>
>
> --
> Thanks,
> Tonghao
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH dpdk-dev v3] mempool: sort the rte_mempool_ops by name
2020-03-09 8:27 3% ` Olivier Matz
@ 2020-03-09 8:55 0% ` Tonghao Zhang
2020-03-09 9:05 0% ` Olivier Matz
2020-03-09 13:15 0% ` David Marchand
2020-03-24 9:35 0% ` Andrew Rybchenko
1 sibling, 2 replies; 200+ results
From: Tonghao Zhang @ 2020-03-09 8:55 UTC (permalink / raw)
To: Olivier Matz
Cc: Andrew Rybchenko, Jerin Jacob, dpdk-dev, Gage Eads,
Artem V. Andreev, Jerin Jacob, Nithin Dabilpuram, Vamsi Attunuru,
Hemant Agrawal
On Mon, Mar 9, 2020 at 4:27 PM Olivier Matz <olivier.matz@6wind.com> wrote:
>
> Hi,
>
> On Mon, Mar 09, 2020 at 11:01:25AM +0800, Tonghao Zhang wrote:
> > On Sat, Mar 7, 2020 at 8:54 PM Andrew Rybchenko
> > <arybchenko@solarflare.com> wrote:
> > >
> > > On 3/7/20 3:51 PM, Andrew Rybchenko wrote:
> > > > On 3/6/20 4:37 PM, Jerin Jacob wrote:
> > > >> On Fri, Mar 6, 2020 at 7:06 PM <xiangxia.m.yue@gmail.com> wrote:
> > > >>> From: Tonghao Zhang <xiangxia.m.yue@gmail.com>
> > > >>>
> > > >>> The order of mempool initiation affects mempool index in the
> > > >>> rte_mempool_ops_table. For example, when building APPs with:
> > > >>>
> > > >>> $ gcc -lrte_mempool_bucket -lrte_mempool_ring ...
> > > >>>
> > > >>> The "bucket" mempool will be registered firstly, and its index
> > > >>> in table is 0 while the index of "ring" mempool is 1. DPDK
> > > >>> uses the mk/rte.app.mk to build APPs, and others, for example,
> > > >>> Open vSwitch, use the libdpdk.a or libdpdk.so to build it.
> > > >>> The mempool lib linked in dpdk and Open vSwitch is different.
> > > >>>
> > > >>> The mempool can be used between primary and secondary process,
> > > >>> such as dpdk-pdump and pdump-pmd/Open vSwitch(pdump enabled).
> > > >>> There will be a crash because dpdk-pdump creates the "ring_mp_mc"
> > > >>> ring which index in table is 0, but the index of "bucket" ring
> > > >>> is 0 in Open vSwitch. If Open vSwitch use the index 0 to get
> > > >>> mempool ops and malloc memory from mempool. The crash will occur:
> > > >>>
> > > >>> bucket_dequeue (access null and crash)
> > > >>> rte_mempool_get_ops (should get "ring_mp_mc",
> > > >>> but get "bucket" mempool)
> > > >>> rte_mempool_ops_dequeue_bulk
> > > >>> ...
> > > >>> rte_pktmbuf_alloc
> > > >>> rte_pktmbuf_copy
> > > >>> pdump_copy
> > > >>> pdump_rx
> > > >>> rte_eth_rx_burst
> > > >>>
> > > >>> To avoid the crash, there are some solution:
> > > >>> * constructor priority: Different mempool uses different
> > > >>> priority in RTE_INIT, but it's not easy to maintain.
> > > >>>
> > > >>> * change mk/rte.app.mk: Change the order in mk/rte.app.mk to
> > > >>> be same as libdpdk.a/libdpdk.so, but when adding a new mempool
> > > >>> driver in future, we must make sure the order.
> > > >>>
> > > >>> * register mempool orderly: Sort the mempool when registering,
> > > >>> so the lib linked will not affect the index in mempool table.
> > > >>>
> > > >>> Signed-off-by: Tonghao Zhang <xiangxia.m.yue@gmail.com>
> > > >>> Acked-by: Olivier Matz <olivier.matz@6wind.com>
> > > >> Acked-by: Jerin Jacob <jerinj@marvell.com>
> > > >
> > > > The patch is OK, but the fact that ops index changes during
> > > > mempool driver lifetime is frightening. In fact it breaks
> > > > rte_mempool_register_ops() return value semantics (read
> > > > as API break). The return value is not used in DPDK, but it
> > > > is a public function. If I'm not mistaken it should be taken
> > > > into account.
>
> Good points.
>
> The fact that the ops index changes during mempool driver lifetime is
> indeed frightening, especially knowning that this is a dynamic
> registration that could happen at any moment in the life of the
> application. Also, breaking the ABI is not desirable.
That solution is better.
> Let me try to propose something else to solve your issue:
>
> 1/ At init, the primary process allocates a struct in shared memory
> (named memzone):
>
> struct rte_mempool_shared_ops {
> size_t num_mempool_ops;
> struct {
> char name[RTE_MEMPOOL_OPS_NAMESIZE];
> } mempool_ops[RTE_MEMPOOL_MAX_OPS_IDX];
> char *mempool_ops_name[RTE_MEMPOOL_MAX_OPS_IDX];
> rte_spinlock_t mempool;
> }
>
> 2/ When we register a mempool ops, we first get a name and id from the
> shared struct: with the lock held, lookup for the registered name and
> return its index, else get the last id and copy the name in the struct.
>
> 3/ Then do as before (in the per-process global table), except that we
> reuse the registered id.
>
> We can remove the num_ops field from rte_mempool_ops_table.
>
> Thoughts?
>
>
> > Yes, should update the doc: how about this:
> >
> > diff --git a/lib/librte_mempool/rte_mempool.h b/lib/librte_mempool/rte_mempool.h
> > index c90cf31..5a9c8a7 100644
> > --- a/lib/librte_mempool/rte_mempool.h
> > +++ b/lib/librte_mempool/rte_mempool.h
> > @@ -904,7 +904,9 @@ int rte_mempool_ops_get_info(const struct rte_mempool *mp,
> > * @param ops
> > * Pointer to an ops structure to register.
> > * @return
> > - * - >=0: Success; return the index of the ops struct in the table.
> > + * - >=0: Success; return the index of the last ops struct in the table.
> > + * The number of the ops struct registered is equal to index
> > + * returned + 1.
> > * - -EINVAL - some missing callbacks while registering ops struct.
> > * - -ENOSPC - the maximum number of ops structs has been reached.
> > */
> > diff --git a/lib/librte_mempool/rte_mempool_ops.c
> > b/lib/librte_mempool/rte_mempool_ops.c
> > index b0da096..053f340 100644
> > --- a/lib/librte_mempool/rte_mempool_ops.c
> > +++ b/lib/librte_mempool/rte_mempool_ops.c
> > @@ -26,7 +26,11 @@ struct rte_mempool_ops_table rte_mempool_ops_table = {
> > return strcmp(m_a->name, m_b->name);
> > }
> >
> > -/* add a new ops struct in rte_mempool_ops_table, return its index. */
> > +/*
> > + * add a new ops struct in rte_mempool_ops_table.
> > + * on success, return the index of the last ops
> > + * struct in the table.
> > + */
> > int
> > rte_mempool_register_ops(const struct rte_mempool_ops *h)
> > {
> > > > Also I remember patches which warn about above behaviour
> > > > in documentation. If behaviour changes, corresponding
> > > > documentation must be updated.
> > >
> > > One more point. If the patch is finally accepted it definitely
> > > deserves few lines in release notes.
> > OK, a separate patch should be sent before DPDK 20.05 release ?
> > >
> >
> >
> > --
> > Thanks,
> > Tonghao
--
Thanks,
Tonghao
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v2 0/7] vfio/pci: SR-IOV support
2020-03-05 17:33 3% ` Alex Williamson
@ 2020-03-06 9:21 0% ` Tian, Kevin
0 siblings, 0 replies; 200+ results
From: Tian, Kevin @ 2020-03-06 9:21 UTC (permalink / raw)
To: Alex Williamson
Cc: kvm, linux-pci, linux-kernel, dev, mtosatti, thomas, bluca,
jerinjacobk, Richardson, Bruce, cohuck, Jason Wang
> From: Alex Williamson
> Sent: Friday, March 6, 2020 1:34 AM
>
> Hi Kevin,
>
> Sorry for the delay, I've been out on PTO...
>
> On Tue, 25 Feb 2020 02:33:27 +0000
> "Tian, Kevin" <kevin.tian@intel.com> wrote:
>
> > > From: Alex Williamson
> > > Sent: Thursday, February 20, 2020 2:54 AM
> > >
> > > Changes since v1 are primarily to patch 3/7 where the commit log is
> > > rewritten, along with option parsing and failure logging based on
> > > upstream discussions. The primary user visible difference is that
> > > option parsing is now much more strict. If a vf_token option is
> > > provided that cannot be used, we generate an error. As a result of
> > > this, opening a PF with a vf_token option will serve as a mechanism of
> > > setting the vf_token. This seems like a more user friendly API than
> > > the alternative of sometimes requiring the option (VFs in use) and
> > > sometimes rejecting it, and upholds our desire that the option is
> > > always either used or rejected.
> > >
> > > This also means that the VFIO_DEVICE_FEATURE ioctl is not the only
> > > means of setting the VF token, which might call into question whether
> > > we absolutely need this new ioctl. Currently I'm keeping it because I
> > > can imagine use cases, for example if a hypervisor were to support
> > > SR-IOV, the PF device might be opened without consideration for a VF
> > > token and we'd require the hypservisor to close and re-open the PF in
> > > order to set a known VF token, which is impractical.
> > >
> > > Series overview (same as provided with v1):
> >
> > Thanks for doing this!
> >
> > >
> > > The synopsis of this series is that we have an ongoing desire to drive
> > > PCIe SR-IOV PFs from userspace with VFIO. There's an immediate need
> > > for this with DPDK drivers and potentially interesting future use
> >
> > Can you provide a link to the DPDK discussion?
>
> There's a thread here which proposed an out-of-tree driver that enables
> a parallel sr-iov enabling interface for a vfio-pci own device.
> Clearly I felt strongly about it ;)
>
> https://patches.dpdk.org/patch/58810/
>
> Also, documentation for making use of an Intel FPGA device with DPDK
> requires the PF bound to igb_uio to support enabling SR-IOV:
>
> https://doc.dpdk.org/guides/bbdevs/fpga_lte_fec.html
thanks. it is useful.
>
> > > cases in virtualization. We've been reluctant to add this support
> > > previously due to the dependency and trust relationship between the
> > > VF device and PF driver. Minimally the PF driver can induce a denial
> > > of service to the VF, but depending on the specific implementation,
> > > the PF driver might also be responsible for moving data between VFs
> > > or have direct access to the state of the VF, including data or state
> > > otherwise private to the VF or VF driver.
> >
> > Just a loud thinking. While the motivation of VF token sounds reasonable
> > to me, I'm curious why the same concern is not raised in other usages.
> > For example, there is no such design in virtio framework, where the
> > virtio device could also be restarted, putting in separate process (vhost-
> user),
> > and even in separate VM (virtio-vhost-user), etc. Of course the para-
> > virtualized attribute of virtio implies some degree of trust, but as you
> > mentioned many SR-IOV implementations support VF->PF communication
> > which also implies some level of trust. It's perfectly fine if VFIO just tries
> > to do better than other sub-systems, but knowing how other people
> > tackle the similar problem may make the whole picture clearer. 😊
> >
> > +Jason.
>
> We can follow the thread with Jason, but I can't really speak to
> whether virtio needs something similar or doesn't provide enough PF
> access to be concerned. If they need a similar solution, we can
> collaborate, but the extension we're defining here is specifically part
> of the vfio-pci ABI, so it might not be easily portable to virtio.
>
> > > To help resolve these concerns, we introduce a VF token into the VFIO
> > > PCI ABI, which acts as a shared secret key between drivers. The
> > > userspace PF driver is required to set the VF token to a known value
> > > and userspace VF drivers are required to provide the token to access
> > > the VF device. If a PF driver is restarted with VF drivers in use, it
> > > must also provide the current token in order to prevent a rogue
> > > untrusted PF driver from replacing a known driver. The degree to
> > > which this new token is considered secret is left to the userspace
> > > drivers, the kernel intentionally provides no means to retrieve the
> > > current token.
> >
> > I'm wondering whether the token idea can be used beyond SR-IOV, e.g.
> > (1) we may allow vfio user space to manage Scalable IOV in the future,
> > which faces the similar challenge between the PF and mdev; (2) the
> > token might be used as a canonical way to replace off-tree acs-override
> > workaround, say, allowing the admin to assign devices within the
> > same iommu group to different VMs which trust each other. I'm not
> > sure how much complexity will be further introduced, but it's greatly
> > appreciated if you can help think a bit and if feasible abstract some
> > logic in vfio core layer for such potential usages...
>
> I don't see how this can be used for ACS override. Lacking ACS, we
> must assume lack of DMA isolation, which results in our IOMMU grouping.
> If we split IOMMU groups, that implies something that doesn't exist. A
> user can already create a process that can own the vfio group and pass
> vfio devices to other tasks, with the restriction of having a single
> DMA address space. If there is DMA isolation, then an mdev solution
> might be better, but given the IOMMU integration of SIOV, I'm not sure
> why the devices wouldn't simply be placed in separate groups by the
> IOMMU driver. Thanks,
You are right. I overlooked the single DMA address space limitation.
>
> Alex
>
> > > Note that the above token is only required for this new model where
> > > both the PF and VF devices are usable through vfio-pci. Existing
> > > models of VFIO drivers where the PF is used without SR-IOV enabled
> > > or the VF is bound to a userspace driver with an in-kernel, host PF
> > > driver are unaffected.
> > >
> > > The latter configuration above also highlights a new inverted scenario
> > > that is now possible, a userspace PF driver with in-kernel VF drivers.
> > > I believe this is a scenario that should be allowed, but should not be
> > > enabled by default. This series includes code to set a default
> > > driver_override for VFs sourced from a vfio-pci user owned PF, such
> > > that the VFs are also bound to vfio-pci. This model is compatible
> > > with tools like driverctl and allows the system administrator to
> > > decide if other bindings should be enabled. The VF token interface
> > > above exists only between vfio-pci PF and VF drivers, once a VF is
> > > bound to another driver, the administrator has effectively pronounced
> > > the device as trusted. The vfio-pci driver will note alternate
> > > binding in dmesg for logging and debugging purposes.
> > >
> > > Please review, comment, and test. The example QEMU implementation
> > > provided with the RFC is still current for this version. Thanks,
> > >
> > > Alex
> > >
> > > RFC:
> > >
> https://lore.kernel.org/lkml/158085337582.9445.17682266437583505502.stg
> > > it@gimli.home/
> > > v1:
> > >
> https://lore.kernel.org/lkml/158145472604.16827.15751375540102298130.st
> > > git@gimli.home/
> > >
> > > ---
> > >
> > > Alex Williamson (7):
> > > vfio: Include optional device match in vfio_device_ops callbacks
> > > vfio/pci: Implement match ops
> > > vfio/pci: Introduce VF token
> > > vfio: Introduce VFIO_DEVICE_FEATURE ioctl and first user
> > > vfio/pci: Add sriov_configure support
> > > vfio/pci: Remove dev_fmt definition
> > > vfio/pci: Cleanup .probe() exit paths
> > >
> > >
> > > drivers/vfio/pci/vfio_pci.c | 383
> > > +++++++++++++++++++++++++++++++++--
> > > drivers/vfio/pci/vfio_pci_private.h | 10 +
> > > drivers/vfio/vfio.c | 20 +-
> > > include/linux/vfio.h | 4
> > > include/uapi/linux/vfio.h | 37 +++
> > > 5 files changed, 426 insertions(+), 28 deletions(-)
> >
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH dpdk-dev v3] mempool: sort the rte_mempool_ops by name
2020-03-09 8:55 0% ` Tonghao Zhang
@ 2020-03-09 9:05 0% ` Olivier Matz
2020-03-09 13:15 0% ` David Marchand
1 sibling, 0 replies; 200+ results
From: Olivier Matz @ 2020-03-09 9:05 UTC (permalink / raw)
To: Tonghao Zhang
Cc: Andrew Rybchenko, Jerin Jacob, dpdk-dev, Gage Eads,
Artem V. Andreev, Jerin Jacob, Nithin Dabilpuram, Vamsi Attunuru,
Hemant Agrawal
On Mon, Mar 09, 2020 at 04:55:28PM +0800, Tonghao Zhang wrote:
> On Mon, Mar 9, 2020 at 4:27 PM Olivier Matz <olivier.matz@6wind.com> wrote:
> >
> > Hi,
> >
> > On Mon, Mar 09, 2020 at 11:01:25AM +0800, Tonghao Zhang wrote:
> > > On Sat, Mar 7, 2020 at 8:54 PM Andrew Rybchenko
> > > <arybchenko@solarflare.com> wrote:
> > > >
> > > > On 3/7/20 3:51 PM, Andrew Rybchenko wrote:
> > > > > On 3/6/20 4:37 PM, Jerin Jacob wrote:
> > > > >> On Fri, Mar 6, 2020 at 7:06 PM <xiangxia.m.yue@gmail.com> wrote:
> > > > >>> From: Tonghao Zhang <xiangxia.m.yue@gmail.com>
> > > > >>>
> > > > >>> The order of mempool initiation affects mempool index in the
> > > > >>> rte_mempool_ops_table. For example, when building APPs with:
> > > > >>>
> > > > >>> $ gcc -lrte_mempool_bucket -lrte_mempool_ring ...
> > > > >>>
> > > > >>> The "bucket" mempool will be registered firstly, and its index
> > > > >>> in table is 0 while the index of "ring" mempool is 1. DPDK
> > > > >>> uses the mk/rte.app.mk to build APPs, and others, for example,
> > > > >>> Open vSwitch, use the libdpdk.a or libdpdk.so to build it.
> > > > >>> The mempool lib linked in dpdk and Open vSwitch is different.
> > > > >>>
> > > > >>> The mempool can be used between primary and secondary process,
> > > > >>> such as dpdk-pdump and pdump-pmd/Open vSwitch(pdump enabled).
> > > > >>> There will be a crash because dpdk-pdump creates the "ring_mp_mc"
> > > > >>> ring which index in table is 0, but the index of "bucket" ring
> > > > >>> is 0 in Open vSwitch. If Open vSwitch use the index 0 to get
> > > > >>> mempool ops and malloc memory from mempool. The crash will occur:
> > > > >>>
> > > > >>> bucket_dequeue (access null and crash)
> > > > >>> rte_mempool_get_ops (should get "ring_mp_mc",
> > > > >>> but get "bucket" mempool)
> > > > >>> rte_mempool_ops_dequeue_bulk
> > > > >>> ...
> > > > >>> rte_pktmbuf_alloc
> > > > >>> rte_pktmbuf_copy
> > > > >>> pdump_copy
> > > > >>> pdump_rx
> > > > >>> rte_eth_rx_burst
> > > > >>>
> > > > >>> To avoid the crash, there are some solution:
> > > > >>> * constructor priority: Different mempool uses different
> > > > >>> priority in RTE_INIT, but it's not easy to maintain.
> > > > >>>
> > > > >>> * change mk/rte.app.mk: Change the order in mk/rte.app.mk to
> > > > >>> be same as libdpdk.a/libdpdk.so, but when adding a new mempool
> > > > >>> driver in future, we must make sure the order.
> > > > >>>
> > > > >>> * register mempool orderly: Sort the mempool when registering,
> > > > >>> so the lib linked will not affect the index in mempool table.
> > > > >>>
> > > > >>> Signed-off-by: Tonghao Zhang <xiangxia.m.yue@gmail.com>
> > > > >>> Acked-by: Olivier Matz <olivier.matz@6wind.com>
> > > > >> Acked-by: Jerin Jacob <jerinj@marvell.com>
> > > > >
> > > > > The patch is OK, but the fact that ops index changes during
> > > > > mempool driver lifetime is frightening. In fact it breaks
> > > > > rte_mempool_register_ops() return value semantics (read
> > > > > as API break). The return value is not used in DPDK, but it
> > > > > is a public function. If I'm not mistaken it should be taken
> > > > > into account.
> >
> > Good points.
> >
> > The fact that the ops index changes during mempool driver lifetime is
> > indeed frightening, especially knowning that this is a dynamic
> > registration that could happen at any moment in the life of the
> > application. Also, breaking the ABI is not desirable.
> That solution is better.
>
> > Let me try to propose something else to solve your issue:
> >
> > 1/ At init, the primary process allocates a struct in shared memory
> > (named memzone):
> >
> > struct rte_mempool_shared_ops {
> > size_t num_mempool_ops;
> > struct {
> > char name[RTE_MEMPOOL_OPS_NAMESIZE];
> > } mempool_ops[RTE_MEMPOOL_MAX_OPS_IDX];
> > char *mempool_ops_name[RTE_MEMPOOL_MAX_OPS_IDX];
oops I forgot to remove this line (replaced by mini-struct just above).
> > rte_spinlock_t mempool;
> > }
> >
> > 2/ When we register a mempool ops, we first get a name and id from the
> > shared struct: with the lock held, lookup for the registered name and
> > return its index, else get the last id and copy the name in the struct.
> >
> > 3/ Then do as before (in the per-process global table), except that we
> > reuse the registered id.
> >
> > We can remove the num_ops field from rte_mempool_ops_table.
> >
> > Thoughts?
> >
> >
> > > Yes, should update the doc: how about this:
> > >
> > > diff --git a/lib/librte_mempool/rte_mempool.h b/lib/librte_mempool/rte_mempool.h
> > > index c90cf31..5a9c8a7 100644
> > > --- a/lib/librte_mempool/rte_mempool.h
> > > +++ b/lib/librte_mempool/rte_mempool.h
> > > @@ -904,7 +904,9 @@ int rte_mempool_ops_get_info(const struct rte_mempool *mp,
> > > * @param ops
> > > * Pointer to an ops structure to register.
> > > * @return
> > > - * - >=0: Success; return the index of the ops struct in the table.
> > > + * - >=0: Success; return the index of the last ops struct in the table.
> > > + * The number of the ops struct registered is equal to index
> > > + * returned + 1.
> > > * - -EINVAL - some missing callbacks while registering ops struct.
> > > * - -ENOSPC - the maximum number of ops structs has been reached.
> > > */
> > > diff --git a/lib/librte_mempool/rte_mempool_ops.c
> > > b/lib/librte_mempool/rte_mempool_ops.c
> > > index b0da096..053f340 100644
> > > --- a/lib/librte_mempool/rte_mempool_ops.c
> > > +++ b/lib/librte_mempool/rte_mempool_ops.c
> > > @@ -26,7 +26,11 @@ struct rte_mempool_ops_table rte_mempool_ops_table = {
> > > return strcmp(m_a->name, m_b->name);
> > > }
> > >
> > > -/* add a new ops struct in rte_mempool_ops_table, return its index. */
> > > +/*
> > > + * add a new ops struct in rte_mempool_ops_table.
> > > + * on success, return the index of the last ops
> > > + * struct in the table.
> > > + */
> > > int
> > > rte_mempool_register_ops(const struct rte_mempool_ops *h)
> > > {
> > > > > Also I remember patches which warn about above behaviour
> > > > > in documentation. If behaviour changes, corresponding
> > > > > documentation must be updated.
> > > >
> > > > One more point. If the patch is finally accepted it definitely
> > > > deserves few lines in release notes.
> > > OK, a separate patch should be sent before DPDK 20.05 release ?
> > > >
> > >
> > >
> > > --
> > > Thanks,
> > > Tonghao
>
>
>
> --
> Thanks,
> Tonghao
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH dpdk-dev v3] mempool: sort the rte_mempool_ops by name
2020-03-09 8:55 0% ` Tonghao Zhang
2020-03-09 9:05 0% ` Olivier Matz
@ 2020-03-09 13:15 0% ` David Marchand
2020-03-16 7:43 0% ` Tonghao Zhang
1 sibling, 1 reply; 200+ results
From: David Marchand @ 2020-03-09 13:15 UTC (permalink / raw)
To: Tonghao Zhang
Cc: Olivier Matz, Andrew Rybchenko, Jerin Jacob, dpdk-dev, Gage Eads,
Artem V. Andreev, Jerin Jacob, Nithin Dabilpuram, Vamsi Attunuru,
Hemant Agrawal, Burakov, Anatoly, Bruce Richardson
On Mon, Mar 9, 2020 at 9:56 AM Tonghao Zhang <xiangxia.m.yue@gmail.com> wrote:
> On Mon, Mar 9, 2020 at 4:27 PM Olivier Matz <olivier.matz@6wind.com> wrote:
> > The fact that the ops index changes during mempool driver lifetime is
> > indeed frightening, especially knowning that this is a dynamic
> > registration that could happen at any moment in the life of the
> > application. Also, breaking the ABI is not desirable.
> That solution is better.
>
> > Let me try to propose something else to solve your issue:
> >
> > 1/ At init, the primary process allocates a struct in shared memory
> > (named memzone):
> >
> > struct rte_mempool_shared_ops {
> > size_t num_mempool_ops;
> > struct {
> > char name[RTE_MEMPOOL_OPS_NAMESIZE];
> > } mempool_ops[RTE_MEMPOOL_MAX_OPS_IDX];
> > char *mempool_ops_name[RTE_MEMPOOL_MAX_OPS_IDX];
> > rte_spinlock_t mempool;
> > }
> >
> > 2/ When we register a mempool ops, we first get a name and id from the
> > shared struct: with the lock held, lookup for the registered name and
> > return its index, else get the last id and copy the name in the struct.
> >
> > 3/ Then do as before (in the per-process global table), except that we
> > reuse the registered id.
> >
> > We can remove the num_ops field from rte_mempool_ops_table.
> >
> > Thoughts?
It seems better, just adding Anatoly and Bruce who know more about multiprocess.
Tonghao, could you add a unit test to exhibit the issue as part of this work?
Thanks.
--
David Marchand
^ permalink raw reply [relevance 0%]
* [dpdk-dev] [PATCH v1 3/4] net/ice: add the DCF framework
@ 2020-03-09 14:14 1% ` Haiyue Wang
` (4 subsequent siblings)
5 siblings, 0 replies; 200+ results
From: Haiyue Wang @ 2020-03-09 14:14 UTC (permalink / raw)
To: dev, xiaolong.ye, qi.z.zhang, qiming.yang, beilei.xing
Cc: wei.zhao1, Haiyue Wang
The DCF (Device Config Function) feature works as a standalone PMD, its
hardware entity is the trusted VF with ID 0, and the software function
is to control the flow setting of other VFs by the mailbox with PF.
It doesn't handle packet Rx/Tx related things. This PMD needs the iAVF
virtchnl to send the ICE PF's AdminQ command.
Also for security, it needs to acquire the DCF capability from PF. The
flow related things will be added later, it shares most of the ICE flow
function. This mainly handles hardware initialization etc.
.-------------.
.-| |
.-| | |
| | | iAVF PMD |
| | | |
| | | |
| | '-------------'
| '-------------'
'-------------'
^ Other VFs
/
/ Flow Distribution
.------------------. /
| |/ DCF cap req .-------------.
| CVL . <---------- | |
| Kernel | ----------> | |
. . DCF cap rsp | DCF PMD |
| | | |
'------------------' <----------- | |
rte_flow '-------------'
VF0
Signed-off-by: Haiyue Wang <haiyue.wang@intel.com>
---
doc/guides/nics/ice.rst | 47 +++
doc/guides/nics/img/ice_dcf.png | Bin 0 -> 39168 bytes
drivers/common/Makefile | 1 +
drivers/net/ice/Makefile | 6 +
drivers/net/ice/ice_dcf.c | 651 +++++++++++++++++++++++++++++++
drivers/net/ice/ice_dcf.h | 61 +++
drivers/net/ice/ice_dcf_ethdev.c | 319 +++++++++++++++
drivers/net/ice/ice_dcf_ethdev.h | 33 ++
drivers/net/ice/ice_dcf_parent.c | 348 +++++++++++++++++
drivers/net/ice/meson.build | 8 +-
mk/rte.app.mk | 1 +
11 files changed, 1473 insertions(+), 2 deletions(-)
create mode 100644 doc/guides/nics/img/ice_dcf.png
create mode 100644 drivers/net/ice/ice_dcf.c
create mode 100644 drivers/net/ice/ice_dcf.h
create mode 100644 drivers/net/ice/ice_dcf_ethdev.c
create mode 100644 drivers/net/ice/ice_dcf_ethdev.h
create mode 100644 drivers/net/ice/ice_dcf_parent.c
diff --git a/doc/guides/nics/ice.rst b/doc/guides/nics/ice.rst
index 8af32dabf..2639ae239 100644
--- a/doc/guides/nics/ice.rst
+++ b/doc/guides/nics/ice.rst
@@ -240,6 +240,53 @@ report a MDD event and drop the packets.
The APPs based on DPDK should avoid providing such packets.
+Device Config Function (DCF)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This section demonstrates ICE DCF PMD, which shares the core module with ICE
+PMD and iAVF PMD.
+
+A DCF (Device Config Function) PMD bounds to the device's trusted VF with ID 0,
+it can act as a sole controlling entity to exercise advance functionality (such
+as switch, ACL) for the rest VFs.
+
+The DCF PMD needs to advertise and acquire DCF capability which allows DCF to
+send AdminQ commands that it would like to execute over to the PF and receive
+responses for the same from PF.
+
+.. _figure_ice_dcf:
+
+.. figure:: img/ice_dcf.*
+
+ DCF Communication flow.
+
+#. Create the VFs::
+
+ echo 4 > /sys/bus/pci/devices/0000\:18\:00.0/sriov_numvfs
+
+#. Enable the VF0 trust on::
+
+ ip link set dev enp24s0f0 vf 0 trust on
+
+#. Bind the VF0, and run testpmd with 'cap=dcf' devarg::
+
+ testpmd -l 22-25 -n 4 -w 18:01.0,cap=dcf -- -i
+
+#. Monitor the VF2 interface network traffic::
+
+ tcpdump -e -nn -i enp24s1f2
+
+#. Create one flow to redirect the traffic to VF2 by DCF::
+
+ flow create 0 priority 0 ingress pattern eth / ipv4 src is 192.168.0.2 \
+ dst is 192.168.0.3 / end actions vf id 2 / end
+
+#. Send the packet, and it should be displayed on tcpdump::
+
+ sendp(Ether(src='3c:fd:fe:aa:bb:78', dst='00:00:00:01:02:03')/IP(src=' \
+ 192.168.0.2', dst="192.168.0.3")/TCP(flags='S')/Raw(load='XXXXXXXXXX'), \
+ iface="enp24s0f0", count=10)
+
Sample Application Notes
------------------------
diff --git a/doc/guides/nics/img/ice_dcf.png b/doc/guides/nics/img/ice_dcf.png
new file mode 100644
index 0000000000000000000000000000000000000000..540823c4a05e9ea9d558d233fea77016d1ae5df3
GIT binary patch
literal 39168
zcmZ6ybzD?k*FH`NDBT?jlF}e8B?Hn74BeolfOLbDz);d%Lw70NNOyNPNO%7Z_x(KY
z`+k3a@X<4KW}m&+Uh7)dy4F6yN(xdKsHCWHaBvvXpCpvw;1KwLKlqmjz$g7i%71`=
z;O&*A#NkSZ$ajH@7p7wJVsLO35oq^@NWk^0Z=W>n;oxu`p8w#NOsHJo;1Xh`CBz^u
zdIzZ}I(W`x-6a%La{^jclNOU&`_)rTS4-}zY76d1P4w1CVq)-OmrKC^5#ZY~-}pzu
zy$L?@i@zdO6jl2qp0>O_?^V0~==_TKu;0E)6AT_jK}Ys$GuU9xpl6NmUq4AAAS8rZ
z5<h>~BQL7h+1bg!^v{JkDk^GFZXt;WZK)wU@I4Lh>Nma-8Py=vX58uL5BwsNv0+TC
ztdd|ub|{f>I#qBR_r=Ecc2+(+F$m*+POs4gtgWq$;%5$Zz1W+)IolS4AR!?E!%MzF
zY^hc+(<6BKV$B>q*BptIQ5gaPfd*IJosgp=XT25!8zwm^L5M@!Q9uFabHF7#DiuCK
zt~qeo#^9j@Nyw!ThkWHBd%m>z=MtX)IqO&q%qUT-Q<|7KJNMhNs;a6pC&Kl7XSB)v
z3KtLW;^N}!aM9A*nl<f&LypeAH;UH%@~=<W?^;Mic5d!cle_a$qiaVX7S(%;iC{c>
zPEJmL@k|#dzb#-e#WW$n3TwM=5&|%zK#w5Rj~l++vL_;_xx2ZfV1g5o&-OTVud|dg
zY)p8a+CN(%s#Jb5B@9dFyIC8&!|9!kw<)+dnAUfkUON7RZe*>j5V@t@LFIjKzBWN}
zl6bSB?AXD@u-6mJ^$s$%>{fg;H**<Wy_z$R_0fFyxQl_NC8(r`t&yYbC$Wq!ZhWiR
zaQO>2)_CPI7Ur}ViP+8SHXAZ(qCX-8Pgw>@)`d8#Ub}%C2oieVUWG5$Z;wDPw0oh!
z(MS2V(Mt&eBO3Siyc1oDi4_JnN!?`_!F7~4=LJ<;7i4@c?GuTsp&oXOubJN~j*nRE
zr*(v=l`h&IUu)Y-G0<(qS?nZJwYZ$k+*tNcTf2SvjF$-h6dEDNR%>@OyFp@~wx8sm
zQ@KkHCa?M<DDa*q5JkVi|7}X6ooQlUmnxZR*#{T+GPIBvv|B=xJdNM>M<?%wx7c)l
zdn}q*;Wg|Gzyh^yRZNGE1Xt2+qw~^RE=E{lE|-=pha<-cjHj$U3eSXtS-y9ciA;Qt
zg#X2CIXogbtk+gV#pepYbkK~qZROc+2-nD3e6SS4d#n~;vgmMgi02}r<bB>}Fg(g-
zTDmx2YrO0kRzm_}D5S%J%DnQ0DdV>iV*NTt#-ch%|0xYIm~S%Q_BwvyfdQLqe%iUU
za<4uRAr<XM?#|y&3?~<aXTr-Hgaz#tT;9D;cK$2uv9~aYB8Zi$G+Wuxv-FN9+5PYR
zh4qI=trjcZ;=wL@A=W&9;v>?i^7Yn}O(rs$G1Q(|Z2z1sn<Yns)V;&#c$@TE2Q)Ru
z)6)`Bt!gQksJ*lfL=-F}Q=R&|&-+U>DkcK~7vFVi4u!dmmta<R;msWO$55HcDm$Mq
zYzuC++dJmp5`L=_vy^#UbXhfEraEBcG`ZP)uwPnpl~qe?xmmf+NiO2ax1Om+(c=}a
z!GBS0w-`sWU7FHRP;K|H)xhT98<o%yk#hLBt#8R=)!=YcP~pc5m_`)LD5eB~iYqs_
zj?tLy72jM<E{lrbr1en>*bVr_*W;KI;3E7^sdYFjoC{%b$NVtKVTT^7+2BYNw7pYj
zi4R5^w->W9pqU==f1#Eb;6C!>z+DXeb>TXZwZp;G`tJ%B5BYEu=0)FDN%x4fNM_s}
z&3d^+eysZRo^x@R*na2q(uHz@x%b$Uaf>KVBm(XH446M${l^6#mQ=@EaW_yr=YvCd
z+ccCShFQ?$1}?+o10_*^aev+*-pN^+qy-`EqoDYuWpSd~d>GvmJf>Q%JrCl>)WF6k
zmNOMMMp+hgneL-`A#pMMbJc=vS<;lS;}#5&f&14fUK<#DZv350Jd6LHkN$NrFqGJ8
zlLQ^XPt!kNF7<(R!K>C~vVL6sXh^GeVu@|f9?Xx@{W00`WXRZv(|^df><;rf5PNdx
zBpMO7vZw8Z5tyP#0>Vqx1X*<dv&0Kc;4ZffjQvQ21Vb2S)~HS*e}&q3vP0X5Ck>{p
zL|PgxQ2d3lia{mhxO*i0u!<pNN+=Dr*xkF}rL@3)H_B6SSLCd3U(-4g(^%Ggi&~71
zLC)Opd9psBf9m^tF-R2=i~Y+Wx~{DJ7&!b`+!vr6(LbBVOG$#4e;21&m~FS))*G-=
zPa}wqkl#vdk|f?wwVSv9(#)clt1G+|;%v%L28)#;+6VXk{7iUG+jNKVTH?KH*Mg`P
z4}U@{EuUAnkMX-Hs~^?DoM;li?~cv8zY`mcjZ7JLTz<$aU4RE4@Y}g{&JAw#$bdHS
zIsj~7y#89(M|b^nq}$-hTBu4^z2sUdUo4>v8BBVg%wyU%_MXf@bERkY-D?x<$;O<b
zpvBDRefW<DBArvMWHxmVUG9o}n*WowASDGwz0FLH(Dzku6^(`Vv9TH-Or_3WQek9&
zp{?wvKs0uy5PokudTJp-5ZP7duzj+zYp+MDA<51+GTg}}+XPcyOCY#AF=5lm{bEKD
zuYwz`>Y=}WkR(4(Y2H+%2SHKL8L4sdgQdlyUB8->Fe_=z_$V35@6;>_^Sx+A)!#{y
z%vNutx>hi*td+hI$B#YrfhqfD<l~3DlzhC=#`503zteBJi#)&=g+hzrVxg$DkD{I@
zg^onlA0{~L`RpH-+6gSVF_?>R2n@E#DgbDpEt*<Z@je@xlV4BeZ%k2#RC;@KHPXI=
zTv4h`bGcu3y~M9wU$HGWe-9h0vD0j^8a@f(O+_$Xo~<oWUu<$5exr+)S7gD4{3wVn
zgrav3%g?CuuSSpayqwxe{k^|Fjm)EGSivv!zjH;?(yL3=fBQ=%uWad!A{EMEQA^#0
z1YSLDC4#7>-9zS&zAcI<8lQ|mDf7d6im6Q*Pa;o%x1R@B!{cXusK5DSe-NiT0+q(3
zJAul<M~5YH9~^lh-eTUP{6+YH%@Tq2W2W5mXeE;-FoLq>=#Q(^6LkG!bA|cqcF*pi
zs1U^jfwNgMV;A~2jK#)8*61YsCi9QR)xxWa4oeYf>?DNCwZ`dlvWs4bPp=e=C;^0@
zKx^mvbur_;)>z-nUx7=5Qf6*XICBRQBO-{5^FAFbsff<KpkO!a8>m~+y7p>YE4&LA
zuaOFs0yDxhFwwq*qMEm7{5K*3xDA#58|>mq?5K=|*ax&{U5}%6AKyg@FV2-FFZMsS
zrHF6K$F#kK%8>ZN{<(`uWDrDi_jGHzJr(KJ|1yPYc)l)sxy8H2+vzKLzZc?{M!E7v
zF8j7?Yx*gy+=(y$qRI15nG2g2=Obg4BUlV=;xP}#z2<_JKj<^R-sjvH4CwW2sL>TH
zBFbS!1ZIVePcY_hP7Oo2*3+(G-We}hkFa4=pbOcJV<M%e|C+Y!zUqG{n|J=*>)qtn
zuvYEZ1ox-boipvgxq`t_VP5lxlooZlyR(J3H(<AZF?Y%7yd%9i@7`?6^1pYM<8}vA
z(fcoCgsMf!oK2VWC%GQAgevYXL7o?Eii4GU?~V&J?0;;|X}e4MzW2UeoiA;v_Kja^
zfO*44nT%$s<HJ_iHD0JS+i49q+(`zs=pvVW8i+;xuYtKK!0_&IS4RuYlSNh0LB0N0
z@7M}4b%hGL51O*{(vrSeZ6D8OHCSs;@XmhzeVhf(Slk`m&D&IY@;78t2t{dNPCng^
zl3t`DCq?X(3hcH>ORIJ6{H~eR;2X%NKo&enSZ7_?BtrJ5+S7I!1*N++VP8<HIBoPS
zs4s)hk6=@{5NU`^H+?`wcP3Bej!F{iUVeW-Vmz5k<8mL=To^D!#+Hw%%ierPJ88H_
zPv82kH0g1Ai1s0S9toBo^Lh+BR#Ex!G~`x30&(~+!!jfFHn1X}Pg@a3JmuSLTQ#Qb
zpQ{H2%o6CZ4m_;3gUII{#r0Kl7wZMxrF?pq^i$O~lx>HQg7%WjlKr9^Cy2f+@^BqY
zbx<#R&U+j$A5c+HBI@EZBVkOHH&Z}NE>JzIVq7w@Q)x#`<VnX!98`B2r8m%6Vo+<#
z6|brd_*__wX0{t1GqA?_MGCa@sMxH?;yyPPF1TB!b^Hhn>~1g2KzO8);$a8uAH*&3
zg)D9M>q1*@_xCVo#(yS{1NMP%E6V?!Zdx45c`#Zo`Hos^`dDktCFvy1sVN@{zyDdY
z>><Z7B&4vz?kkf60-^cx)}7#;WUQ;|Ust^A{Cc6ps{`J_Ho33e7$a~vXbL1#t`aT!
z2UJ5oWg&|^mMJOh+0J)(WRFNE*ofb`(PH0t*QCEMv<$k$g`L*rpQiFCT-S{2>uUSm
zw<=5E{Hpn^91&*!WTH|0=HF?*gcfcA(20pesM@XX<znYR-nMzrF*R^)xxTPuJ-mhU
z$llE1dA!{69o)n5LZtfsGN@04B+k+hUE_rgg5;a&za<srs{J-<3Qy6+aIcv(Vh0~)
zkKgAXo_BfQU6|#09=7DXY~^inHrqTNHErF7_5Idg-n+BOIH|92e|zJ_nvaQy)}E~B
z(h@~J+X?{0YlgxkKbSHfz6PX3+q1^PZ8&z^?eZ~;#lzEVFypvJwzmp*NjFhor7Ydu
z;qg_Ml8d`7>U`!J{-r<=sz9Ex<rB%wndw8n3T-!>HqoW|-MF)wvnft=dI;3KrdhMf
zMAysew~LW;pSGKW3p2yd|KC%+rqf%cE><%ubiE&Jx1%0MZ=Vj7xf*$_KBZqWak)pv
z4j1_4XAlThcpZNinfEz(+YMJ<jd{>q^T3Cm4}mOujeA_~ly2A-OCo@BUWkDw$?;>`
zNAbhK1bj9N)%iBl%U+H3#(W(f7`9(~6|64(y#)u!7m=9$ls^wV6HaD*K=}$*?`*(r
z%PAmMv7h_OPY`2waY<I!<1y{9lkvg*S2VsV;C^W>H9yql*ma~+oMt}#AG;;>^x|`8
z#R49-yL6HlpGSwi9Z64GB2a6w($eJZ>gJ#mxlbpK{v}O&29pecI1p8Qp<h1SHvO4N
z_+F)%xO0841x)wd$&8Vw&1HJ3w?MV^H)!Ha$X1O>IO*|zpZSM6>*QdWiv2AWGUOWT
zi-U3Fb<QX6QPsrlDFCm9oEyVYRmw8*Eij=nt7ZU<rce$)Tpe9zp*)({+wiPnEysrt
z%KsJKDNEN+O_ku)jJ2#;k}_W1r057%=`a*slt2m_4Ow4Dxwv;-?t_o4?>G2;iM1t1
zHTq@j=TP}0C)-I{SAz4qYL-K^qweE9?(nlNI~&!^GA8sit}QIH$3qI@ShGe{waqKL
z$gP6l!{HMFCb=eGSRp?HQ$W^k5NfTr!FhVBHe2dKhrT@y7@j_LUMO2X!U<qQcl9^w
zAMYGS1h^`W*Fliy$>29w+j$DuOrED%X?afH7nQAm*uGMru4R&AA=<D2a2dhUa`0r@
zkTJbBK6OSuwLJ5|P9q(YMcV`M$U<Wo(*mXYZ8CH?GyY$Y{x{kJ@CmLnT3u;ei(W?I
zMzAO`+bv3<e%#6GvtLFB?dSqKV^E+x%tEbya&0sX)%=q7bl~KNG#gIFFumkEV<7hU
zq)iJq1xI5@jQ(S?U1mj9Smdun_!|Kx7{jS*z}2gF{sN8ft`X>R+<Q>}*9|j&!o)35
z{$bev7y}Xb>i>Ao|77I<=gZUAg;co|EjBB;+t~N>_f;{P9o0(>CsVsQIY)8(M+{Z0
zX0uI|msFKEtT~ttnsz7Q&HG#`N0?_r9`7GV%pEN$8bm$wuu-KbwUr2qDS>DJ*c!rf
zo^{1mo!8969ZL|;%-@E-i_s)2-OJ(P&$+jIm%hVOr64opW`*}ty|pQ4f6VyiOYbe#
zc=({mBPV?e10QS2MJS$rn8*$>vA(j}V^5F$v@VUb2^BbA=Q!jU&j8~69bkn$Rr&R-
zO%6w?WLzx_VTtLj2CV!KGIodUcx>y$1FafezYvGt?k~pa`aUwZ1UZA|ee$tkNv=Cc
z{JGZu?=XZZBj@Ofc4+btE(a9d-5DgAA*&B(**VS-x*~(_)GHAM5wR(${3Wbl#jKod
z%!O|u`G9Q+|2vP8d@VU)x3xNg57CaY!$@wqG0aZHEbB+A(>?I5v+>m<Uz;jVsLJ*J
zW~+@y6zO<$tl?|lNJMu4hC=%f7vE5WJWMT(IHLy3^({EaKP?YB7ke1l@pqa|SOJ6V
zP+$pl?BYl#$__WuF_p)}{q`1!uG%lf;Sa^0L}IxMpnos@Sn`)SbPo_}$YtIzMIKdC
zPs>vd6ZxC5y|K}HBV94S<<b$`HYa;Z!TZt^q6v86x58P{wkI2UoRXG>hZ~Pm)?ev(
zXVwAnw+Kn2g}4PvdewlLVqV+5V`7&VqCwG_bUp~XJRAD!ZJ?#2yhSQ*$+5}6kL+Ty
z7n`c?qP19mF#uIzF`@8+FFFm+?x5Zv7Kf||U6sF$ie@YBg?|s!O5Yd)@?QvRXPw9W
zzAFH=P#$oWOh*kT(7*Ed?XvkR4WNH}In6(}n`hUC2FjAX9nI&<5ll~Xlj0N(M~gM9
z7W;3xm?1qmFD1O@Yps*F!w=>vJEt4sE)}`;KqRl-pB?W8e==fRQHbNR&yXn9-CPzT
zX|2+4M8JL)^24fW-lP2u?B&<ST&l#FFZ)D-uFKJC<Kab1+ygzELfogXQQ&I#70tnM
zy54n{`>H0Fw|hk<)fQE<3!TG@ud#0no9o!?oNKC!@}4eEg2_~&7N3qT-Qsh9Iv>z{
zm)0<yr)u#7_;6@3bvjJz+ggqdL`H_^?hjYdPzWpB%A2F0&w3<JB_EHSqr-ppKXP7$
zrWt>;AauX_U`)J@pMV`1Q@7$z#}5b8W-NWwOd}B|VY?aWP5mhB2UD^`MzduiO()As
zi*rJP!fC{s1^J?Ot$QRN*3s2*LB2-Q@>P<~K2Ce@@ChWHB0qvX-#@OXT6O<z6!)t8
zuHHuvKO2MQPf~XiwS5gw^y}|f;*G`8qVqEEj|<{jf0-d9z;+?Hsd8<$bJw!TNAoog
z!lOHQ@wE`4mlgY=k-F55@ok?~VYgZ9ii?83DUY{@BQPW0Kt&6>3A?52TkfiF>Svqg
z?k5mH0g<X3N|2TzLCY-?>EQvDCi4-yJ&q_ZOB?^Ly!aEOJF0~0B(D@5kKF=wuhD|l
zU+C#Q8yn1*!P{yZ%n10gV=xIA)5(n8w~GulmX=Xr!2;1WoHvT$;rkC#$)4|!vFT+X
zw1r+D=B(0|$vmpR_ND*2G9vq;Sw=C};CZDWLZX{Kby1`C&o%_Y=2S+BH<R2XLYGSE
z_=^|+ssvIdqk*Vve*U?%e-cz*yJPIOZgw1c$H2N3J9pcuYz={bj%-@o*jDZLI~QwC
zBAzvlhUpVeBJ9G0MuPyI?ZP@V^Kz#H7sx;!p1xk4&&-S`|3LqHbLRJvK%QV=Qkb{S
zvFK<?FJ<lVt5;392rQqy5=su0>1Ji8iLbg#!*Kvdbu_1a%bzZ^ESUW(q;MjVZ0*f;
z!p?7bS9101myQ#T`rwFer2+tPS}~s>qQQ4@qljMnW0?dl0Z$d)m31QSYb~j2ibvAI
z#x<niUH!DZdzYp5SBs_*horD3f0JK!rz1mnfewB4Gb?v?I8aoYjvl{!QI?LHC9;+~
zoO4O`Ss;?Z^Aldl-U@c;ZGbzXt2gmFZFi(IS+89!*Lk<F@bP}#ZyFV(^jK1WN90`~
zt=Tthn9XCnm;+Qjz8G0Z>*7Ne;3SWus}*&Z1bGmhWqu<dAv#}%Gd(=ndllitwoRIo
z=Y;_o^88@Qse8?9I6Pwi)bibdeYC$Yfnv85*zhy4D|RdlSSDYL0)*rCoha8?w<DT&
z1A2C)NtZt0&W>)}@#IQP`74&+aOIqCHv=OPy;3=g?ZXcQU_r3<9`^sxN#N(Sl-g5+
zhm%cbigdiDjZN%d1~Qg0ODQFKeyoJR`kd#k|IXA-l{D|)&PM<Niw)F*#IwjCReL1}
zYT#Hf>PQn`j?{K~c<F;FKn7kjrJhC8ZhL_tqV0FFyv`9W-^|cIJ!3w5CB}a^^%s2A
z!kZ-9CkOI{{doQP-#T1pWlE_zQTCS9g~NcX{+SG4b?_OC`xuy3j+4X;3IK1etFU|_
zwU)#L2MPfH`45d!QU<#gu>th)=$7BR;X|@wqUv5<r@H;<-%;IVy_9xGyk!FH?40`i
zh&xk1X(J4^;^f7B!0d@?mW)-O{}o#QN$MY^K{-1A05(IRQ<dU{w)@%ijcKn;(QdbU
zS>2MQW2XvUa)ii)_u`-rnYHKB?wPOvE8o>{*St4#(@XuD6t}M<Kjr}^p&@SEy-a^s
zicMrbiSmDPIzRp-EIO&}#9Z*`{%WHma(;pwaXmQj(l?ztiZhz)?1y!o?2<o%oYP^B
z6=Bgzb0(P_rjGYJ4x0Ex{g{L_v_D*Sp*kJ(f4vqey-LLa(*Ttf96(dRO5oVI*kvHS
z)Yrni5#mxk52&L1ST~|zsG-JRTfTf6_;CA&S~zwWFHy*R*chhF3+&o6)p-`N@F9?l
zoSB%vtJ+66h^$SOipE>?*&Oc(2)=DpfDruZ?u)O3FB}0j^bIi0vt%bB26EIJ=b_`j
zOA1%H`Vt?X?rOD8woVkiMxan81My~V*uaT99~-NAswewgQT>020z-bjLUXyB$=(5F
z9G~s*y0#nZkD*LO`6f}2YVh8EaNq&r+Ak|aWUE>bp(T*mSAqFG%OgQ+P>a@EZ6~h^
z>o?rr#(V;ElS_BiX$tkX(wtspX6DCW{Fg{%=6j`jPqn|>OY@nhl4Zk*<q)K-4WtE2
zT=Xw+#p}(Eb7^W#JwAPHY`QOYdJj{syXN6!1lUnlx{1dr@tqdHzW4wjf0jn517Qdg
z=!?1^@thZ2vBvf)uiJyS<5q(c)qhhM#%1R}>Uml%abT1~`U|=XaVE#c%g7PeQvGo0
z)-=6kkfAb6jV(L}fX;C^-057|Q>o|d{FD#WUvE&jnVy&A0k!CR%(MCH2e^R#JaS_N
znggoLNCg_gLo2g1SejKFNA;^kwYJ$B9v)jOBm2@Ez1!lKMa;9uhM-!^XPe#-5Da_T
z&m`v{tLN!0^5385dIR!-r_<Mz`Kp^bCju;?82igI@{uap{k-ljoM?_CwNSE!ijVV0
zz3^eE&kIb4-5Snu?08)+a0#C*>;Y-^pa!G+!0pq(t;w3<q&Bj#lW>Mmesfu0*hRc!
z9+d~O{*%Db-?Y1~yh*E<m_qESjvJr`{%e1IS9+h$W$(57K`FKllA9RK+##=SwgngC
zEztt?2qCD<Mnjl+!{OwtG*ENSHW;<0Z9W(_c)JW{tnWY8)wyTyk2BfSs+*cg7X2EQ
z=d9!RQ!e9TV3MPe5(d$xXXIzg%MsRqLI|_E<WHe8nFI9>+r4Eij&q(H_&ItXP8kY0
zhglRO!te5?*<|T1*jqku^}AWsT}=UkmNZ?_-w6}<0=A6MGLhUG6ZHH2?K<P9jEQ}8
zDC&66za{7|^II6Mi%l4p#)<PZPsg3p@u<d2FBtkyiJhz@lgzafkLJ#UM3Wkm?;XTj
zmT!p+XUhU%EmKC<2#;9CaPjnKoFz;fD2UKSp1>Vu{yM{NH3}k_`}6Qyb0P^HP783l
zsV!U7SEgrT(W^G`bA^LWUP5e($^kB;9^Du?Kna&UdyvR;Wg6I|skPDw-YJr4w#ER(
znm0{dZBp;MSFJjxQVq*<YpI?-@nYFvH8Z-zqQ4`s2@8mGHbH9ELNAkL)kb@!ZvOzL
zbaE~m)6AOw^82--vSzn$W^!$Licpx+R^5ALo10FnjySnP4)afW9Ejf9@0+$`DLTa1
zH`e)E4e_21xXsPBr%mY(v^fmgTT&&@_u<6rf17?ToV8FwGlSDB1?s>OEw3~CeL_5B
zXD#8)VY7!#1e-|^Uc=Q>)#3wU7FH2@(uewe0EqooU%ubdPph|-Zb{6lZ$-_+c=Fu$
z6$Wz0#!NBZb&#$q9@yuSlAOHCLTk97!#j{y#Pj;7dm7`?qvAWeS>%51QL`21@+god
zr-pamJ8Ep7Y5(wTr=3<(`G1$X<Jg6Mdr)LDm=W=YXEqk0J%gk~N1(175Fo?^mDiec
zM!sZqrY0JrI6@L<(R4uPysw!?#(=sMP};O-=5t2qzi1Q!!exVOpqdSv9IMm3MDf9y
zuD5@Fv(3pxMvK>!09T}tA7WAoy;OJj-=}>aRYYTSwU-{%bR=a0zf6?!E=9~z(>@6&
zK8C=U>}FrAk$gC=0rbZ&^TqERW<P)XLdr+5L|<EF)G7sjxbCO{#0Q@#47%PgmKZ=3
z{#$yRkDr5LjjD42;*<2=GR{(ean|!S`bG5m{QY(6P(ZIo*^`Hp{Z$X3(?(%!(@IPh
zS(4dY1Z)srZKK^fSh_<tsh*Vy<*<C9z#oj^8y;D|LHPEHTZ7SL?TeV#u?h0|JxM+F
z&v$a-Cm}G-mJe0>S58vXkI_a<dWy~KlD|J@-8tEqdfE`m;leCV4yK19wqxDYYM##e
zpA+IgG$D#dULW#0qAVl*HY$G&UF|!Im(?`qu;GVQWCXSgr075=y$@$)BfPSPQP`6|
zmkxU2MMw5&0g6{+u6G1DibM`6IA9M_94K+Hl3+~%XeZ%rpotU5r(X{R0jrbdfAqT9
zpnHFA+gctLP7gX6g{}tQ-p1!$-ZxzoaYY-zy(yIbD43H}_9niN20+k%wYhBhaUy~a
zogDY>^k#-ahKEh#r<;;>HWpeIB#Hg91m44lI^)OV>+6|oWTMHiMVf~DxxLd%!zj=S
zpkh(X{r2$k&`JB)d4s8sKY94&!OMIc0Wwk9$@`W)7}h?)^1ln8<=Nkg*hiLE-ASh1
z#JxLb_z8neh`oAU$Z!1k%|-|OK|tqCp=r0pn?ez<JMjL%A9Ps)>6LkUH%{PifZBg>
zCU|i5;<{*hMp}gt1~raLsrM+_UiP#R8v9x7jZ=rGn%OV;+t0WesG8>QBy9_Y_t~Zl
zI^~ok8{%#h2=bz$9n<5{@-Uw~H_y|6Le!wL4de)@>mmY=4n`dYz0}{_)>{s=YFwOM
zEl#?tC;@{231|1ZA8T%#4%kgr<NrIAVV3z{mQQR4f;-v`U0kxg^_FJ7eQ^{t-SV8y
zXm3N_iycbP9)oj|ix20U7BkXB+=cW-_0%QaOj|zyF*)}*)4tJ#-W!cbMM=t4zCCQp
zgJtNsV6X8Jtht!{zR@NyDX4FCb%BJYp|$utVtWfdqZeU2EbI1(r&uwtRXrqtYFOzB
zmXBU42b3lAKw;WVR{Az13Sfx8QU76x+49QGBSXAKJ;z{sLe}vrhwve$A28WZ?T0jZ
zl40o%_w8W`%%vOGY}C#$guce=W?{is)$3f64ji=tEC9M5H4%Kq&iJkIu;J2SIqq_p
zET#ZpEg<6wtS&1kL~O^oePwi--?(*H8i>@5m)l!C{Y+^qt%3m}fK@pR4zx&~-%Tn4
z3K&qP%<@>;9FXw4t=Pm@W5{x9VwAO)YuI~E2Q3T!s;_x!IYW#neb^pe)?OB5PfB<}
z=Cd?cb3=Ixf<5(D=sJI!eC=YhU4t3sT0_3%9obKJoO19m`$zf=$Z#dp7+{b|Z2r6y
z59t+iJEcZ|3Uo|#`Ca5`Dm8`zLebx_L_eU{g%($mR%g(AUWX>81#~y-+@2DhAxvM6
zi6W)IZMj0jhJJwfS%`LLop4suSKKl+eB<7+(BjN;fZ|r}%^t`{Md5%XK38MZ+U45B
zJ9wx4WlMja@ZtmP6zl7Lujh~V_yii*S}FWl?=Y<?z$7t%iq8ao)mYO-TKv7)>JHXS
zw*82~hH+$dcE1D^C^9^n@|R9syt#gfU6_0MUUmp_YUkc}l70g#@{)V3^x;iyMfwnm
zYA3a_=MD~U15x7*SMra9r4;bx$!CXWCye}&Pl~tvtI~34_NX0=!BN6ix`(Ce5`EOg
z)b=P~s3cD-LZ6J7-vx!vMCHfi3s-ysvEc5wa^C*XQbliv&wm#^vna_)&kp-03=U|&
z2G75Ybua2_)XsEv-FxuMG7JUa1bo}yJFg74OUMK?s9fz0IW^%ct)`lUc>!S=v%>u+
zoEABdTY;uWk*%G0?uFLRoSTD0pxP2mCI1M=hH42X)y&61S)UJ(Cw}21G$v}NTbx#9
zAWe(#@gYCzR|HL~0gxM}f@k1Sxoc7WnuEFXD+aoYlZQ~Ft472{lV*0DXxFhkiB784
zj&g*o=d0(m1tOX>K3a&hh`bvh_95};V}22-4}OCI(8X7<5f}o0$w=mJ;a#j!Hv$Py
z_7_0g_Fu-G3Az{&ygJ&?9K>{cau|F1&QBwt{)yx}1EAXeTLI{RkxWjvn?h|I+p#2&
z<Xj1j$#p-G`2F7-7Oy@#HQ>u>pqhEp8MN@z@L(k~ZHWAb?ZPr(Py<vG%Rj#bSgrG3
z(usssK#OWmk;PvA?+tE$JN8_JmBy4s^Uiz%na6R@KrwAWV?aUsk75bJS4Gg~egJ*#
zSDa_`l~#l`k$j0c`QPsY@KkZ7vT<6i-TTsE48EmAeu}duN`zwnBLgy|<gd-bfC5FE
z{3M3~nsM_cog)H<z*Sx#&h+0(z$loBLw_SttNw(-vrb`+(~6Y+5`3PnF8*UFBq4_q
z<$DKVi-OB)H;g%mf&Rb7VFr(8X7%CSX2-g6$-JU;*(}_lTe2NK`NW8{pdurrEc>yl
zpAn6^FL**iwePkySaJh__2YenuX%leBxJK4tI^K2n?C0{PWFefIDAg{YtaHx8Cbgk
zdsu>nL=N`q@rz#zQ44E!C)~(q=KmZGlNtq1e?fp_VZgyJumhMN{?$z5>6l%n3#s1?
zcONTLrjo|MCINwc=QwdFg5)TOpogInYC>Nj1AH)$2X+Dz;1hO^?L^wXs&oCS_v$gt
z0G|`eTFC1&5aNX1G9muYlVY?Wo0|w)c7;t1HL+EJf!^Odlu=z|U;lZ)B1Q^<h3o~`
zToQEu8aX%m_jYv%btD5X+cG`h!a`eYYggyE2sbAeeSH5r9ISQ&{5QY@@|HBO7CDXV
zU4XN1TjLC6`7-@|FubU*x=o+JI2r<Vfs&ns_yqAMH?O%)%+Mvnvvy#W!G?&M0bl<8
zK|y}2sJZ5>wU-aRY~+ml^&eVVBHKFg1-zihJBIpv_JqBxY#0+G3kGV$kFDS}dQ@vK
zF-`tD4=;WvHDcFrSO}_D%dtATihK+NQo&u{)PMz&O*cNVRWvYg`Rpa;<06eN`3T#A
z$qc=d-i}#wgd`NgLSaVss6q(h%;bv<T7~#hipwiK2?OLNB<XB=lzx#_5GE#to4{h_
zPJ=w#$|Pq61S!!1nompJ2c=`dGOzI^|Ip3UgNHM(Q}VC>kgMr|IpWoowe<AD>D3H2
zcw(W@!Ka^kFyj0jT0Q%I=pd2O-pi=F6Lhj1JXJTR@DN2&`MHAHKQU39LKv^I;DF6$
znN+CVjZD>G$ezHFAOWeWbzNZvm)l*!VQDtFXz;5$bArquU!@tBiV*&}zkrn1^2H{H
z3$+Y|SPU3KGCPnFv;!=`e`8`IR4#pm9h&mi_VSW#ZS4ab2P%Ldu~y2q1_~rOL9f4a
zLZP}!Jx1I1s3F&zo8p=5Om9XD_qZllQWrFD$lp*VwfA`9>q1^C6UexQ%5e6Y?<%Bu
z;fqL%n`XzKoETaYyGDqIFg1%353Q}kGi56)L8$D`W`xBi4<69M!aMFB4L3wkS(uJO
z$LJsHtP6yIDZ=4TqWo0)1R)10M7dm}0FD|gWz2p`8bBE9>ai;MvEJ)#pt2M;9Oy!f
z0k@NmPFu~hus$|gFuJqrT|T=VWdPyi{?EqLT*h46fEWQ^IEF&#+q^G)E)Zw7oQ+>{
zUi_4~?LCh8S`aor&A<}ksb9}_fx}k?CAj|GQ&h$Z3GYUkG~N&brlmBYB9dz=K$#`Z
z1P@Y;q-JGfrc0xym-g^&pFz6z{~D?zl?j^U{<Iz(#g`)=x*;ySIsX+np-F)AGT>Ej
z0Y1O9(ZS!y&H;jmE`tqD2Bt*3uAUFhSLS#h*P96by>%_y<g==G#9Yj;(9=^z9(dwV
z9Z)7{GZjplq#dVIJTnc>%+G!Sl}XXznHOY;0Uq9Nw1#%VP5{Kl2yD(K@pgoso>~3U
zJ#h~aDkGDT{|Ek6fVz7^A1|K}TRIJ8SX+-iu-1^@>ysXKM*N#G^+rChC|m|6+rIvO
zJQd>SIUXHGN5=lot!Ka2C~rB}(eUpKAq-56J}_LVx41c`n=`d8%J&jRt2=QqQIfN$
z2JEMJz?MM0VAc{|sOlBzC;yCF_!QMYbajrEdo3MxoZxT7swhMX#0bN$e7QAL@dT*n
zcT`cZ^eg;HZ$_pqHShPaHsAO{N<u|QLd7MXZTa6X*W>APbHN*LgQMg1-PEdGxRGg0
z!8%}2MSq8;hg)R3$UG5<mR1S!1bU16g~Ptcm9r@)OB>;XDg!>0sMTPD5&DiPdIllP
zK&^aDWa-~~)8DeCukYcBo<zT0<#;8eSMWTlA~5RWS>NoJq_vp#oCyM;brmBD_Kc!t
z<>jugH&wZL5x}AO^%;!PV70LNos_y^p~=|HW9MwpKkJ}pTtACeZvK|Q^!|g&^IG#5
zO7p|#b9L?eQ3qK6cR<|8Pk#oCwIO%dof@PRRIs4P4&cV6AKNVvTN0a$msX75wSxEV
zn@rUy%@@2fJ=cQ?J>(M|FGl;Jnu@<!7ra^d(e`T6T7G~S99PzrNo&$unjbar=-5Oc
zfsy67s5NO#Kc-`K%H5P8r^E_knSOa}or+QZ-GAawZ`MYMLl-1$L6e{JWAjTzMjcL1
z{aW1cb_#gYo1;>YDDAGW2ww&;?j{a9pwpVx-5}e_%HXMtr8VDVV=lO!%uUi~gbrhc
zg(8`23e=y-$QS}exmcZq#$OM2XR5x+$?veJ?r*NNZz4GWcUq&sj|&MbU_T|oj~y^)
z0PiNvVFwaJp>>q-Z=$L)BKu6X>||wrg)!6af3HuIR$~oO(KtU%Wn^U1h^Q|e%|Y3F
z%Dh}Vvo0omUdgq4_H9vtPUC~h7}uXH4&u6C@uOQbQ^tEQk&H}<FZ+A3@x#4r3@_HA
zqG(OwS2<2-bH0=|`#4WHzTKabBs_NJDL#PB*(*aB%}Ep;(ccD60zE+?Ne2AN8FB#k
zDx5BZ6h7fT$0r>sQ{hjYN1Z`FcQ=ll(je}j{KJC%7gAlEXETwuzuro^QRcM{w8f^N
ziL%8e$ET;op7o8SFbhw45s^};4{W+#t2-Dl6y~N=vrd=J1v31Q=sLdHd^WuYAZTvJ
zbtRt)b<w>7(UvC9@SAQ|wj1MdV+^D;-pls)0WsZM*BUEDYAekM(n<5l`Jnm2y-nGm
ztbx?V02efQuA;0(icKgT8U1!iER)%Thd^=&AXIL6U3vww_VTLNMV>y$II*UpFZ0TK
zX8WH`5gk3b=G$Rwp(zU}TJ;-nt%JOPHcm{+W|nPmdW*-Rt$qYsgZ7IC!)NE$HsT7<
zeBg;xz$T`=nEC@HR{5v|u_N_z<!7k_0zj2KSEi!_LVN@Dh4yimuEcoO>&Lb0OCU|?
zLz0f<qLs(15}0Yo=o28*TnY;P3|prezp0V2g}fdx_xPDF1<?c(JTfiuV}JhnT+vGz
zf=>{VdbCDiV6iO@w@%eGM&C7ojUSr=?8sK-8KYg)pQ*`rOyP9Tz$FDJ$C0j^3cWfY
z@6Xg?51`Gz6mTWVE^_yH2hayc=ZW89TpnJoGH!Gn-`~C#DlAVxL-4yG*5YSBo8jOV
zsH0rCm_A3OqPRvWeS8BQ5mTvtNgVn$D+6XBGob?EO>csR$}+PNeUDps@D&vYye86?
z+ehi;rQ5@jxScJRJCRwY5nlAgpp~wt-zggEqK98k9RpL&%a+CkL57L;$U+9(vQ{Mp
zvB+l}K>;*!*PQr~5dRpMud*Vzsj7U~q_Wb^?qk+EVms`mFOBg|sKX33=+CIA$i|vQ
zQa#5jY1f;Spc#J`Hw!zhAGtn^{6wG}e4ph@7dL2*O6--g-Q{U_GLRyfpbUi@Mx$OK
z69&2MA}f9wX)si15G&J`fZMgMcOeL~;a*91cp^7Do13j=<??Jgs?_cT^SdNeMI$D;
z`x4_7{XQT#Ga{r@Uf0yv6`syQToO=eY`_xWfQ`6|CX%h(>nnNs{D}O+7LVkp$i(RW
zsrSD#Lm}fH$pfF*h+fJaq_7n<z23KxD~=lH3=lU|uucoeBGW`sg7o3;{BD!P7l~{Q
zY-M|^6XcxN*&~my05NaGB77BqmnPwE8lz;w$nYC?a#D&-{y=xEs?yW1qfBIqyTU#^
zFVF2Z(8Eh@>7tvBg<NNxv&s`t;Ew3s4^CvI6h$OWc)o@NolQuKLXHNk08jI4&tlB0
zo(4i#2jy|h_6n<XOJR@FORTdI_Zgw5z<l5wJJ9gGz54E0QY)hHOAA^i5Il%>8Xnw9
zH&&u6#Nb8kQ=J+aN&D<y0?-*Sf1X`B%0|})Zvfc?&<Oz2T_LNcT%6V9m9noU_~hA6
zY~Vk}5P9@}@`7qSr|@!HN8lPgE}9hYQLEC467X0ta4&Gq2uK^MM3J91R;j*fUetM)
z3j5IJP<i_khZbiAb>~lb*qH^$ucqO@0G{$D54LvVa(8(^_QCB}YAQv1e}IgPOkag@
z_YdzR>1aCLdIv^kW;q6YTr@9AdU|@|I+v5R)>cv5-6v@onYk*{;nj{nLyWaC(_op5
z*?KkC5L6Vb<<PIWE@m`zyu4SPAw>OgELGLjgz@Y$RGql0q(>h=Q?_)+u8imie~DZ3
z1t@?WQ{md$8gL@Z^X{O=&CTt2D+8B^sOEET)KIFRU+Z6yuCcMxllAWM3gq;*&*tWg
z70k3he*6HA=~#~E4t9l+1*=8|L!n6=#$CQhXpN1He{zPSB2z!k-d%O|d>ziD;o2T*
zag!=pvnceJI2n<YGcnn!v)^{RIxNY`qG2_R5r)z-ndOm3*EN*}_-E#C0PVO~KnwAH
zxR^fM9uDR>M(Z`I$jI55saT&VP)Qu<6C@Ld3?}m)Ej2YHCMJGb>&r_Dg7&Z1x!lP@
zWmJa0^+Ht_-R@mCCirFPosV9Vkmrhn4TU3&f}whECtjH<x)S0gPD?1<0Ifx3aUT%q
zEgPH7Shlpt{V`%sZgFu`2k3R7T8Y-h{&bo5)8k6j-@jk&L=5F6ko=kuKC;l3=0@QY
z$O*XJp4@nx>~x_*QO{iQWg)$t)cyXjam-!$_2%6d7eujje)F=PCoFeBFrj&)3cB11
zyOnqY4o#FUy3*J2#Cnb7F!*)h!Ec#ZUC3zfWZ!v%0s-EX&6~nlwinHR9~oNMH=zSz
zcV2GxWFMQEA&$dUr$vC>ZpQ1!vFUql4J015JUt#RHV9EhB&ZhT13YegeB97Vy;R2s
z#H!!y@i3)pg=nmzO+PXguwS#qqBWMM-@;=xA5JRhb(8da-uXy&xThXi;cS&@(cWHA
zt0JKqT1RDN<x9=DxVYP!n<kGNT)oZn-SOika&mIfoIftq$jHdNHcQ!~X2Qb4$4u-4
zy}i5kV6nWH%R|2{r_V1gioD4Za!CUOB?&qRU9wM<>!|yfJ98dkP=!wj>TN{4Y0Z@m
zrx{$FjehN|8a4W6+nYqkGBV2v8oFK?3g}+9UD#bj_O2^$6Z|!i0ze9Sk3&n3<@g`o
zf8Yo>8EVITD-~Ks#!$$l@(xVF_YLS5&sAG#o0;td;n3Wk?;`4Be;C<@`KB>bQ?CK8
z0=WKsgY!}5$H0}RGa}}hUsl~J@)X}DXZxp32NI`?oR6lg=FrLbEmp?}$f^sGr6h)v
zMOuLNPk4AZE$y1S>I-D#cJ23ETr|JoACrYW1sW4$V(_sE?lT-69c#P>+Wb(Q0Vi}l
z+hWnK)@kwjz|Wtk@uJdVvhea?zC#3S2ldCna?A46R2((a3=b_WZJS%Y^ARRze{b)b
zxK0LDz-i>8e&9~h{w8uWsjdh<AK9s;r#cFg2m(?V86}?g-HoMDxnH0St6)ocegeL#
zmI;rC?c-9_zT&S?tCU37zxlO)=7bp|5uRPeVtP4-QAMlz>+b3*c=rb1KVKBFTwzDo
zU?Y-7%HShb;k6C~9T`y|jq-z=s9<bzGMW`?NT)AduKv@b;1$)_d!8qc>GYiF<Yb$S
zuKxHE?HVA;f4<m3jkB1qKG^7uPEAeCL`aS}K0XG3VK{|9nlECstIK3(Br}AFt-8AU
zN4g;XM{)6R3PB#1V}ox5b!2!F7wZusLy275K)4$4iC09yeBb*a7>a4!q<wJb<UAhj
zkRc72zu5jcV8w}b{n`v0tSRVjv6uWqqO)scXM-zW6jJAUgH!R!^@f~+M%Jv5vU$u;
zS_$lAO$ma$^;sP)ZuGFzH(CS`H!&#)RSxlpI4MCPRlujHkU!BEn?QqJP>>8bf;raN
z-j1N>OI>(+YNtWwVsyArcdYW7FtD}Uu;a&IV#{%2+|V|x^4quHwkO%6xPnf9YmGXg
zfBrzD;Z~pCsh578EF>sC0QOtSC*Cg&E9UHcSLG*mM7JNjZi5rjIQ9}XE<XN?GECXd
zt}a{xaY%Orm7=^nEfcmfq)e|#Mpm}L^Nz<u1sL}`N6Z1fq$^AV1~rOjbN?99%ci&v
zHf3bRpawRb&&)51gyg7%qE{e@DX_EJ@xtiy7Zi}h^0dqP#sQV0qVy*j!_S{pX_+jE
z6nAPE6m^P$&H<B3Y`)ji!xiw_UV}IKCy=hL=k8d}#~<jn_Vz{Ra;PsEnXXA8FCOG;
ztQR2=Nb3t692{ULY@P#O<mvuwYcM%D{xu>Shxy()%CNo{D^qLVtluCb@p}tMy09k#
zB4S%mZf>qho&v=OH*N=rp!1<WGMYe%Y&@&jI3gxuXZzFL#Xhh3*js)5<p>)Ua`wRj
zmAsYtiK!_%CjWfy_wNn4Bp|Q-^B=F*v98S*8eQ?4#YQMychIs@z^}m^E(F7}_cwB)
zMsJ)T>2R<<HijNHhR~p#58h{wYY*EJsra#BMe33~<2_v*tc<ho*eAGNXOn{4rV293
zUxH{$fI~({&3Bm97l@Bk-#QYxOvS~;3ui@Tq0k<{&E|fYBc>+i<>h7Px6+o61`;7D
z2I;Rn?atR6LEi(XT=L<7EOCvvbAN;Onja4j@A?g;pwn02ElB6Xh1qH=vPOx|pWlW_
zBIfxf23^4k-`yRxsH&>2-cd72XfRI!HqVsu?+O9VD^^xjMGMvZ#8wR<WLe3BU$O$i
z=I-jKx5JotGZY~h^*);@QDjZ*?cr?}9IVp$3aB|g4fkP&gwtTLMZ)UuMN{LV;`Ehb
z;u5p0KEIkzw~*NUdw>OV`(nnb`R_JI>w3Q*002();%M1>rrfZcX~+E&m?LpuzQ*Qb
z^68DNXvYihI&7_59S9*+y8Pza8t~SpI(A+}1Xi&<fQAN2-zGD}{D7BI;XxW)1GH+y
z<m3wgLWYs?cK!K-O+Z~&S9f)^i~<(3k%GJgM@o$b{cLzN7#z{O+?%vKUTIrhT^-?s
zN*Tnkp}sA2314x2ZF!UIzp<8U-$yl~M6aUbevX?G6C1rxM(%Y1?&Nm|+$<N>M3dir
z!KF|&HXkToogRWsggy{#342>g;Qy24KB+=}qF_-v?2dhzt^D$EdqD$2TCR(WQwzkT
z0DzR<RhK<m=k+mGSg{P`dsv96CJEJ4wb>{PNY5XOoY%WBSmV)BM}Rby8Si$nN6<)Z
zYlP(AW`Iu4_YJ@%YinzXfKSTGV?Kx|qAs_$w-PyS;N6{_V;3PZGP0fb^onWdMLCW3
zp{Ei)5=dUBGHk9CNy96d`NBTf1Rl0hy4{1k3@sTWf((D#AGV`lQRG-;RMKET2+?uN
z@Vd!F{$(_JLSR}tUw5<MrgoBYC*TJw<QyX;2<c4XHXplM_Fisu1?o<0AU1GlB<~-t
zmZMBrv0p`m(Wl7+X&dhPlh0~K6$$6(p1-qICB?-Cj4tyHP6vBSO?P{fMVWy#&R|~!
zC%_k*J@4`q)7!C98N?f#n!3X&(9-;-p!ptP2>UCM(*Ah1r{fN+5tvU=5C`#>5WwL+
zt$Ugo#UTLob1Nc?Zh8j0nsE2fV#5!ZFNQ2Xj?e4lUf<IFEiRVc6Gu3Z>X&R5ChQ*-
z6(`@}5dm&T$5`0dDCFpPvFvq+K}thJM0CCV1J=^=1SE!~<fK_|51!b}fP?DAJAwP#
z`GGn#y%sJyQ0&hE7gGbc)9*D+-cMenC2L^S<1itQ>*M{h765d)i1*h+g7C=51i$p*
zn-y|%bBoFzX9ZrOB?-BxAE!y@OiDnGcm31HQuyrw*A1KoGg|$VC2vFaZt({Du{xRR
zzdg+Y6@-njyW?hL*NyuyQ?pCH?c?|f`*oK}NfDKeh^_Pi(sk&It_~dyB|8raAUBYh
zhoB-X)9++|48)9rA$$<6`eJNcWxZH$3wRGXANDW!GIxN4f6u@!YlU_|OfFd?-s`bw
z*KG}_c|R=u-d}ms@e+*mLyC5EcSJ$V^lWeP-5RrmQV_sjXLg1~o;<D&DNdLqu06aU
zkck}G_^sh|5n#>0hJ}-GZLPHVO-)T5Fa7bepkty2u-EhU9G=F)+`O$NrxprMV)&U)
zt#SXZRMR#k*Z%CS?`?<-rWi2k>n|apq37r4pE(o%uu(>IhlGb?zI^%Jikl<%htK=#
z>uc->4BFn_-sa|J1GL^$n+!tMcrq#~tg=lYOb-t!NlBB^()2Vm@bPiv`FeZi#AK(B
z57)P+4~1(V%3*iN<U$^(s38DWiZVqJIkxq6w6_~rjg$t<K4E!Sl8}%9qbw~g9gDLg
zFX4XD61p?aIPoQrY5w?9gB{>9?CeUFuD*|Ke0;=9T#fTxEf23Hx4`&dj#}!#jWrxB
zSkl;FG|<yg7FPh#2EzM*I>(Ds-eIb!LIHhFPZ;G#Fd)y7@<OQ#uMvFIURwHVz0FyA
zAqtKh;m<iEWSe)sci{7rCd}eIJ8T-!s*=7)yDgI7i3i%(_^K>XDRGB?FM4T27~baO
z#-5zKunqkRV}POdZ|XY^>O8RqE88Gs<bN`AT_9x93Z)e3WfKm_|DIF&!9&1gZG*OG
zI*HGfj%aF)|1(3O*4y~hBRqUbKvQXp<j|qia}0@;#|J^cuA5qKTm@EdV%+fHKp^0l
z6mH)6P*uD!y}V|%NmFE&b_p{Qa54}_Pulzwgc(q&Ydsn<bpYwle1QR)9~BaiS$98k
zpX)@2^c_YWiBFKKa@*JAahu}WKUM601Q*AYh5kqlJYNEkuScSoQx_P4qEr*;!wAU*
zL`nlQkSE3`c*V#R(c?m)QUtu4&Z<$ZygXNd83%O1e#-(<LDoOo-dirt>*xA3guy>w
z847wcz^RTLp=?767~m!NVsHFtB9;s|&_vE-F<@-f#AgEkMFRskCcrwve)B@zIf3ad
zQ+3cgFJE@#@dWToFsQsQ9i>uX)>LUv9uau{o5Jb?FG>Lmhi-*bqB1SZ!>wo?rFbVm
zy`5z3)X_6@5PDwu9a05<y3Uz@8@B?!IE2!?F^k8XD7A{=A{QPC^<TKyCMU0Zx=u_3
zUdo<#?)e}T-d588a0-U}4iRMt&CAo^s|N#`^5);j)c$rqNJ3YHcu*f*g_)mNlmc-w
zuoelE@}owNdA<dRx4X8VRt$3e6l;u472T&aIh?qpDvOR>;B9juw?jKK3!X2Z<$=iw
z05YbN^%Bs7h>D79V|)5+?9;|A4onVtG{rT^c0?V#_sQyHm?_LCKF?Dv9`7v(DVh?;
zwu+GTaIEC6bKb&n20jQ@=-BL%z{D+3T7KP<*Kf=oh8KXX`n9<=tj*HR`Nq=2DLGI0
zv_C9%I{hsO((3L$ZL5-taqGOJlIpxsP6`~kaTFCdoTSV>MMXEPI6(<A{|jtVP@a2_
z6;%d9LMH)c9Tmr;G`B;3jHX@tHcW<*$@^Z$`M4YhFK?LL?i;<-)C+vo5=1ftElxi`
z^9TB4oRv=vj4a6Ge8674xuDM!Eq~M<8_s)}PeJ9|Phd=sSMX+KdQBbXx;8dx?R}Y2
zXAo&D%BT4Rcv}5i7OwfJ$>@JY3pa~;(brxX_`pz+-r(8s5Sv<4cka=UeA2MI$2xC)
ztcTm5if12%&wj0yn1N#8PvQLD{W`(4!-~j}C6Dwq(y*xhD4L+{@)e;d;h(6~1r-11
zoJL^C@d^Yu<ajMG?yi4@`rg>U8XBWx_SOh0()gsYHszJ2(+Js)i|}!hw(d}t3bPW#
z@zo~=q1^fo$dXTHo$K|ExLlUxBj{p4#JO3`L)3{eCwwAB%BwDw?S-dfe!Ft(bfBkN
zQ;us6?+G0NNHrc1SfvZh|LI|@M#m+x0ICkR%!Ur+1j$#Ma+b+Wx>x?tH8|`eY1vxW
zS#(;bhxbH$Iey#n^#8}yR|ZrSHSJ1=bfZY;p^@%7bf<JTlF}t04bmmuB_g17NH<6c
zNOwth-^KgA-@W%YoW1v|nOV=w^Nhry_ofOW4?FmR2pQ_ZVH4eEjT<Z|z;tvu;4sxq
zUcGf;f#~M_26(516DjzLIDo$)lnlF<JhUf#$NRljI`*oyJ2#VmDNBX|>^gREQSoRe
zkIbP)iB+WETB|K%iy^#OzK~`m=v~LxaH9-ibWzxP^kUxF?-EW(p(OI<xrdTR)ySyB
zLN)vFoEuH1$%;`?&1#O>@QY6<pYp}`6ztnpKGGC;S){=J%{*PUl~iW>I`sY=4;D}J
zje74kT~V?6b9$u2(bhZV8GwA~Z!GoYkUQAzDf9eTepz@T{Aufy(2nlBk@$AzC)Uc0
zYalt@waD>V`p3GG`;zwXKr&W~4qCX#^3T|jl5>q-DN344S$`%5gjg65Kz_pJ2Vn0c
zL_rC&-txSn=2~sPEkX)rj2vFy|5Obiv!yGOr8CQK<MNruoZgc@x1_aGBS*XY%QWwK
zl>G7kaynkT%q`s`br;J^OeOnoCL$>%5~T9fJ_|-c$eYhU8QijGaLT<Yd(8^_F6=9j
zyR2ee2q-o$_h|t3<v*5n#RQ^YCY24fNPOoz%ze<tlc*-W*V{}#`u0h`UBRM}5*?~}
zm6Uu^(f(y{ZeHZS85kD`zNYY#gjB@8>S)%MwFDcip1djSLfY?OwLgQ^ws55fI5QY1
zKZWdb#s>U<vWzfRxNy=-D6B~jyB>TQ94K{s1L^?nj|>+j@BoB+eed#&Fn%f1&`6_A
z4<9z=*52i2qA%wU)mFm+j`lyuyvbL=dlS=FR4$C=6D}qUOGQOuwGVy?7MvM+#rh(e
z|6RCZ4Ue?Jo_P@(W{KywTLNNHV03<}KVSAa><MS_Qu;aXgdm49IkkHPgFN`;Xcc;M
zO_0&+?Rg6kR{h_fI0mH={v-BhfUemnJb?QHw=z|4NH4ZBqayeTW@=xp+gbi~M0e*n
zGd*Qf3klfMg`|!6(@>vYoq3JfmdFzJ3we9)qc|@Y8SiROT^nGtd<F-v+bCAasU!1c
zT7tyqNEUVj<GnZ3H7sCP@xy-58S2n%dZv||B=j*~$upa-4&Z_6wUQS7xVTK+n9|JF
z2lZoHNc1^ee$$8Dgp7=gX2G6;o`d4g)9P?~ph7@>hL@KY><OYgm87B3Dk+V+Z-((F
zoX^v0yL#fOMMTpt@xI`ym|zWM^JQc?3%1|2imsOB!J6y|E(-6k2&XuxJRl`1lIOc8
z*48)Fk0g6Ud}&19v=%2droNOBo(~=!D7r$7r8bf^CG-&dGOFP@`!exKr<(`aBBW?N
zti2;)l;KSg0d~L2k~+QWf2KBCS+35-y#!3vzz}ARVu)%dd?A6uTLsRTVO(Xiy*DM-
z_cyF&R2<1O4`8*WsBtzow&pZiug0G96yMg^@S{InRz#1nvJ$Qol(Q)cABnq+&8^{U
zpc5;${|mMpkY!r92+J?``z7eExqLp>IX)u5mNd3ac;Xp7{I8!^S5AsxN3I1pVM*6H
z_JcfW%NiUq6`Nz;Ue~V$bN1P%hF>D`%e6s<f~i9IYHV<>t7ksv`fo>9{r9w}H~7T^
zebkwoP*!@T<Yv1#_DF!Cw&F<0*Em<Y9Fd7;c49m!D=4m~u2-=X@0KW#L!Pd!G0}?i
z{<GK!_^A(n0d~wv18dzV2qz9oF^;tfgFB`muzCF=KYw#lDh~)uDN82n+{2WSB9;iU
zWW9+`IwX8`u=U#dXjOg0qMgjS*7*XZV7rholD<MbuDGw6t?qS}{5CSM`VNJ7;VgWX
z*)@u>mMQ9%Twd=4tBc5U!Xb?-aB4!bH=R4@u(5GO@^rCdnHnf5qomn+&JkXlNf+cY
zlT@KJ`la|)q;Z5JIBJ#@Zd5GFyxSJ{ZZnz4#UD8;1`=tU;S7O;lr>M4E^@s$RkPP`
zt#B+c{ub<E{6PxF0v&mR+F4;k$rtC0HYqWUHGf!q??u)b1#jZ=gG;!Lf-0=u?m}$7
z?F*cD7J29PS1UyF+YsgA@9r0u5pwz4>qe>JYR8=-ju-Qi8J|p_O_XdVW+1Ue26}=-
z?ci?C$otdW(pS-G!{Se~8dmLW6=NSPb~jen$Vd}4C3WD%zCaws&!z>_{dX;r_|<qs
zm8zNYUS^eKzJWxRQ^*byN^zX}QsN#6!BJnq|Mj{?qBB=Pq>{l!e7r}J@7Y{sfiHv+
zRpk==A?WX9TC8VW7?XkdZt!o<<XcXbY#lk`P;z87+?`BJ)kfI$QgNDvU1^YkC8*Zv
z@gE=fEZp%8W)-`PX;2^Drqi=H#hYC&`8cgTDSQl9x0|5SH!-bWb>c0sZEBygS)=7x
zk(CD&J6-9qEY=znZ<*`={bmvd{w^l&>**Or`u;^i(j}f89)1uxI?DaDh`P{WyvH{j
z!$T_c9RxD{otE^y>amA+vLt)@G%(N*a+zSUZC~X64Uc9#dlG}BU!Wx8)eGIWex_pU
z*;bk5a}Gf%1RHBr=T8hI!a?Mb3I`5_ur3qG#sHh&kEJW-ApsL*hLftM0V$*}Z^5&Q
ze{z5q!%Lm`UXirC&zjJUt<yu;XCg42D_S8}Mvp&%r&+U()czhhh@QnFkC35#hAS|I
zRqbz<MmVS>pg>lm`H6#nuBO`P2@93L8NjnfsMcIAQP~?rj6@ZSgOnwgQl<s%f8!%0
zsaZP-#Y6w3=Q4nj5xvg6o77Yjm@kf1C<ih~?dLVbtfpk%+iN%e=snF&qn+9(juL$T
z?_XHTJ?r{;o19h-jz30o6tymtm_!WWwxRXwRiqN-pQkDnlw!e`C47m3Brtd$=G5!c
z#RETv##!tzKqI9ZlGKC)4rHcUyO3pvCGxMUdBvMmdVVPB{5So%#dA;3lGUdoTbA-!
z!Mryyc0>_+nuV+2o~RV~wE8oi3@dl`>BVMP!HzkfgCqB$@npe%OXyQ-`7}GE5GG;}
zAKv^pPq}C+85d7;6Wk1drcn?Y6gor*-9R?g@vJclHX+u&|IIkV5FuU0NJn2jD5e?@
z*dGYQk*JTCGias)bjd1vGK*mM7`lxbmPmWZXzU5lfq#J@L7~qAQsy)e{_S8Jh4F+L
zPC?OWm4=cedmcHMW9$L=|MxVV2uKQz24;W8r6A%hWRR)Qfk+aMq)u)I*;suKP;ARx
zR5YmRAy6@`{8M?x=Dmd0dC57)v&oO}mPvEt4gR}DNll;*u$%<kOd{n}<C04|T?S(c
zUM`0`*Go;P{C}V084U{fB-Q_v6jH^!`2Qa){t62egf14BVYZ|tH6;MJCKHrw3num}
z6_mubn~9diwtc<1>-IAH)>1sR320UM<hZI{#VKv*LmrB~Lw>J9@3I^9&=QyF9}KJ+
zz#B_Cpj`Wvj#2J!LUVj2%P^NbHp4y}JPA2vsa4_I1<Cg<bMJ6CFR9Hi4Su}^q6&EC
zlo0f%mdc|<gB|*EL~<78hMqY8C5pH7xMMl@AmHDmPbt+xI(-y!THhyFX00hux11&e
z-|_o8Ke8WN-(~o(xoB+fHVrp?n~g=^*Gac7URe^sl`9l5(ecTpe?BEK|4<_G)2TSR
zm1xyI2)!NVS*i$@3vZOqoa|j@C;EP|vCQa!zTj?||791a<cGx}Uu(*HI}z(CZKHri
zvnAa$u7jd8$P=Y$JosKpzT57#hzP;~IEkyFsy7;ruh8avQ#lfHpPDjWSM3@6uF-z`
zA?)P%sh0J<cJ_>oz&;+k@dJLV528cDK6OL+0q=G=e`EHmiR=6t#TC*YG6Ie{vSn0m
zR6z_*3L6NF4-!JcNKj2a4(#bK-%t6Df^NSqKFz32!_1GOE%`J)ZLP0lqj)S=9S<0-
zj*xnvV<=o21UMZpF+Tlk6uh4*Xp}NxY&k_d5L~tyj^0}Ko%HB3N#l|q3veUXKGg)}
zP?6m{kfJ!f(_Z17A5G{^dSgA^LY;2~WoIZn2X?iSIwmm$)$5NhKIa0e_cD)D@QJn?
ziHVL-UtEL_DXtmdiA8x9EybS;+<xokN4YFZk6ykA*Vjb1@l^#?o<>O98B*)$3|PC*
z+tLPzo}`GlXEk~5#5X5=AQic5&iRXzue+9naeLGF#Acl#Di2xhL?EaR7=HK1X6lEI
zxBuBlK|l^${hdC394=ZY%$WjJD!<jQnTGu;ttn{@;>Jt_IzybqH(Oa*UORk0BqMg2
zV%ctaQGVXnp47$LdCC#!&2t;C%8raijY?FHd{s}p6khV4H2$>bXG%l?`Tg!{Gc`ia
zFK&Rnyg`uS@X8J=S19Cjq?#lEXOgMVRs9|ouMUu6_8P4dsYR|{pDGS%eJU-sY5G9Q
zQ*oC$T|s@D=Ul)1^WSf;4FtDi^%)y^<3qIfW128IoC*NF7+1-se9^byv)#+U64(~W
zv^3#1YKl+P9FnM1o<vIyDHJ<aWSw^$b?S9hD+e$mAIIDU&t<X0KB?cWBR_=CI8d`Q
z*1#KM)*EQK-N#XRG>sX&weD+l=D8?tcQLZhsi956Gy{M~z6lfoZP}ZZr=6bfZPi)+
zOQ1V&-Zvo+Q;kO3Y(3J$v#cy6Ma&c-9yLhybBY0Q{4|1{<{0d{1e)FoPZgj(j}})o
zrY#@ANL}*hMAdh<t0libhe#-u$D=vLtGuEY=BW{9{K{!(V~0^^iUr1Kk>y&P4W)<<
z4fxq<D2mC0MLf$2f-T!^&a@RTAh=hNu~5yBwMdAtvgaW8dM@;f@~frTZ*A}vrzOHs
z)5!f%;``b#!0?z;mPcSJ%wv*_OIuXQHGCvtGZ;DuQwsP60I4~1=SC#gjXA3~h_Nt%
z9n|5r{dx2B<*z1I^7jf{7Wu-p`k5D9$F7@YrAdjrzQ1f-!X7Yl7tD<=6b%PFWvAY}
zZMZ+n)Ym>2KLJNEFZwG=3<HFa-!aPMl%|9U1T4%TVEGbge{FKhrgEeCyhYQ)ZNI6$
z8-u<Uh{>*-jnBi;S}mXMDEbZgxi|HGO`;aR4IIgr+DI;%ogonH{Kg{*pJAh`{og3~
zT|shEMick9cPVEh`6evYTcNN}sG5XoE8EooE2ONk^dsP~zk3VQ61m3=H`xE>cUr#p
zVUEWv#~U)>53WnjZ0uXy%jI5aYHTj)c^&G>3Dt%Bij={@l4+KvdO1)qijz$P84f_l
z$${%KX8DSac-&f-5G4c43R4>nL;0?~-ZnzuYx`+O4p_**l-^}->%{_PvgC)CUvac<
z2o=4}Pt@z%Z%hB_{Y&GYC$~O+v;#KW&^)JRnAvKh(W)JdNZ4uH>zxYCR}$)({E@Mz
ze{Btg;x3zZRFWI=Q@D8evRwHtC`$0E2W@&3<!8F?jW<2QTm;FKhquiKqb>U#X#MmB
zQ}McDRXe4>m5qu|BsDjhcDD}Wfkci#%TrZEZw0j#Ox=}L?9S7EW_18o;^tEM(yNyN
zca0W91k1joK`4uhIb-bs#ve+I{CR6_kQ;GiXEFAb-{05oS5=PMlXe19|LsDdd!^pj
z%ttz+&m1*sz*!g<8Fgr&_Z_;<_&%LgE&s5eHu+tOZI#1%Wgh6MbxRN}c-1nwvn%pp
z=X_E5z5k(zegEn_qw$|U1kFwawwp<^0KYSd3KKdqO(x0{$OQ~owdB&rMPE9;#2y9~
z9p}dj9$ZXT!4nQXMpg))&uJT7Qam1>wnoGa<%_yn5V^XFs&=H`rv}=$oZJ)IXs8-2
zvfs};eu10ui0QA^q2rs`JNz2p{lM|Q`>2~beDJI%h+D*;r{<|@rSO@Z1Z1b=h3Y*8
zbC#Q?Jr)HXy4@F!y`r{?+MIw!81Cc2#<{0q+oaUUByNXO6rEe@|F|%>5nX&0g|e!1
zR{gD%hvgV*N=Z9$Ic?9{)FyKlx6yZ79s2nFnm4ng9Zx1Fo7s)>25-S%m|U$o)gahU
z2RnM3Mraj5MUXx8dN?dF<g(JX9G4GE&M}>93Gj%un|-eLxog<j-3)8*n~IlhZ=DA^
zo(C=SfHK4)Vwu=>?IzA7?D#0(FZ!&|w?E$ab11Pf!RMeHXZpJ1jTyRDTo-j<ojp*)
zIPkYM@pDZ1mC>Yp)RD(-!2d({?K*4mqIX~{q@PJQz^nY`)!_GXYbR9(7J5j69F)uH
zfmuDazXkR%hDXW#r2DPBlv10A`RUZdSpV8KeSWFwyUwflkR{KJc7N*4sDCyKCRbG1
zceUROZYepirkiwH0VB=@Q1wxcv=dzS$;*AEopXM8`5&ppK4Cb(@YU_RGat-UNjLXN
z_XZVFTTKHsH535EQkH*{XAvJW-y%+r*E2pOMo8S~S9&?tUW4+n$qxDbkzZoFmSKo#
ztVL?9AbWt#Zu8BpfN2_K<esASqtZHF0li>Mz9GEjsq(zV=%4T?<CJyMDOO52Rhz8r
z22ayiAQ{&tC`7SrL&aW?0f?>@p=~;U&#wLe*(oTG4l@HaV6Ya)>~L5OlP&GNG_IL_
zU%Ot=2zmQd(;y_j!I)tX5cx7P5?)eMt@kQQiTx}b-rm<turq-KcJ139@8jK^pUquW
zvzKmHF9W?WIqft%8)@(SZBJ7OZ5vQ+X+EQ1m%@g+{mO*lrClp3yPy=5K8Da6(ZS~k
zXi}MzLlR$4Dmn}kO*B8WI4}Bg65xQDgOJZ#E?)Ndwcp$_c2zvHw+3zj>+JT*+Qaui
z1jv1{$&wV<Ua7zQ31{{RxVxZy%5)zEI9>V#z*Gi=bRmo7#B$C_i-8_lpPa9}C9SKG
zq|rtmIv(S)BPM!B4{VCQE4KHC(sr!vZ}PmK`gZC1i}n<#fO=PfeQ(t4{2?DUpQnv&
ztIRNAh()UO_S$&aD~i{u7IR+(ey%P9?F@q@VWuuy=HW^?jrOO%w(BiRVSmsl$N5Ae
zY?`x)5M6>0GI$*8KDc0}v^Jz!=G8d3SD64~)XJ6)3O%1|!Bwr{RYNRObE=2FN3trd
z-D^wQLYd@=5)70WD5t4jtKv7m#|UV)2Xa~Kh;E#1>{&-Ye?nB<9uE56$lqoyowE)!
z52?EzbSuH`voZZS-(R{+<z5C1VpMrCY%MPDUsZNi7m>dWq|hW(pEU^bfAa#Vwz5bQ
z69G*R66=Kw2R@7lec*`@(<4wJ%CRx8XV10|qV9T$zkQwmCh|hd2XG0O{U+~wB2F@6
zrCV6afi7U>!N?+2gf@(d1~67d^qikd<8HAA@JhYdwObW`gfoJBe7RUkzFfV2^Ly3h
zJU@OxevZq<<T5v0{AC}zUlmXw62%jsUyoPI#dLQ8sM<tV%5-dGd{VUeYAB3GdtZd~
zM2siQqMYl`=@$G%3?{wuG+`}f<=TV1id7SHbi2|{JkVC6_>LsBJZl-ckS%v_M?DX!
zhl3e(^l|n1``FCy4?HUFqf&ipXhq82;fE0a%}$7#LVdm~yjM-{#q%mm>R!+C-;=|U
zHpek&=;OSIbmg`nP!p=b!#DPJ?VS~S)4^WC{iw&I3bRtK`VjE`&g^CE_Dcmv&CFtF
z>x*A_lhYd?dkK@}(u~`5&ci2_eVCV*-2Sd9nOWWo`zytIK_0kC(~I`CPpofZXO6#}
zvVO_6oUmF9T&5d6;5BvGcCKw6G2{}x;Or0e^hXL2A^iH}x8@t^meOeP#OG)*WF)}q
zESFQXz1-w;^he11iQAN;g?=D+k&uS44FCBCR-T`~8MW0C?X!{f`g4iuR5E<sP9xbq
zh;yvuP@advL4ZH++;zh&ap&g9(=fx@+EiDT0Yrn!F)?Cn?$o0g1wyMb7GM4;aCQmW
zNY;#DE$_J;c;F`4<F(yuNBV|qFeUdSZe7VAn>F|CN@S&+i^4)BGw)#B|2R#gm?izT
z)1k-_29I!VJg@4N`~9O_ySU4H>YOM!8hzenrma4ffJ@y*S50!q%Vtrm>EC8osXiMC
zW_$m+7G{?MR6<Z)K6Ejb^D|4d)p~@jTOUvBFbWyJrjI!j<k!(=eo>;2@8Uh6bAOpu
z)Sz5`?;Oh5<aX-}1X}`Z&K`iLzUL&A$FMe66%*jvI6la^!ys;Xy4@2c)iJ`=5+0+@
zpFUb_9?@-_;==t>F6P7Xc&wKizik+2R=(}3T8T#g#p&pwUZgJjy5CpPnlG}P_E|-Q
zqP*L8oGbNTvZHmIjVIw>e(`M$Q3gsb$;*6LP9?>ii9F3LDD=Ch+7-|sv3Z=<{+(Z>
zS1vPZYp%-dCU?YPVX|FaTWr^;W?6|!Puj3<e|Ds+F!MnNSD(XpzAWVpKc>jVC(qJ8
z!W(@J`NbOJkmaWZlSXaQG)$-_?jA$ABi)-Dn_iT)I)8X0WnQ%8KW~+A5|7<7bm-dC
zJ<~{D`10IX12wR4y(g|5r=9J&5?SM8O!e3fIqJR{nB53SHuK-}&yQ2F_&wbZ{%$Y|
zWC@z2@xfL_GuVw-_+8?gK!DOaa_=C70m%DJH$Gsx+q%-%zS7Z9&z}21##ysAM}kq7
zq+=(3GilW5H(4`%oQMe=df%e|%;k}o&{p}dADJAn4nt0qZMMw@>qD5^E3TL{N!963
z-jTLjdE1A15!+k`{c>c<`*r@+ZK?y^g?iSe;;&q%+__v{e1EKQpsf(?W5H@WtUFJ*
z_Bb@N-fgVE&^M;pSZSg-NMxW)+q_nYY%;;t`?2BH$#<s!>=H!`-LnQ4&K0wL7`la_
z9s`d6;ccnQBKJ0*J^hC{NrIu)`kFV45Uv&-qx5YpES%#T-13!8IidOu0SEe<^OTR%
zDrBmP4UYG&%M{a5wjcM(EF9+S#0Ps9{~YnUKxQ{%%c4Txw2R1#e4i*#hsRT`#8^}G
zw*FJ&*QZK26@f}o?%4LU3hs&iGh0?cOLmvQFCj)oiYrR?Fm5jUZQpL6GD=#kW%se=
z#_64~8KuDkt#!G8tN@I^Wk9P*kp0Be9N0Nuq;KuApga<ko-8GNbLiec{j)>+o=d?!
zZ=Da%t?)#$=-rgO##J;RYd`8KeJstbf#iBdxb{Z#7<MTOA!3U8Ot}rN5mznHq*CUE
zqY{w(IMc5fT_kwd{an!oSC^GtN!IY2Hr_bh2ZOyH=#9}&NouS{5Q{lSGu-9iB){=k
zDly-rv8a(z4gArT^#Q)^(Q<9|(sitb?dl{(PRD<Xi@nW3MpnbwKF@&&$)}9}>=&cQ
zy1?!Jif@a&ch)nM<tNE?6QOw*NL5Za{hP`|+jH|l8<WA{a6s2Qk@qBUfNV0=jfBsl
zO^1^&6$CS0wDF6URsRGX2%Iq}J~ltvic^K}b@*JZT>r3h%=G@&0DF)GguC8}H)+E^
zxT<2?H(L-%4bA$ErG1@(u?dGPKiv<BUMgJh`l@(mRexr!3ui{{nWvqG0}u?jdcUk(
z3m_9I*-wJjF5RkFR$^?R^SNvTL?(Dx3(1+mHL!?<7+?_hU5*C5OP=!S2r`j!G)W>i
z5f=L*cNh!)DI_3?g%U$vE?;%P#q?nRV7f)3DRTXLxRX(?t8<1XO)I%HC37MO!~O|K
z<*OkTQj~ufdpjSk#{HVsj7P=FRHqZ%*y{>`57gwlz@FaAO$Zbits~=gxh|8g2nI}g
z;2rWbJ6YjEJxI9T#~UQbHDI+A;b<vgyhv%>MC0?l12|OQvNxR$T^!ytNBzG5%q8o^
zha;OWqZ@$frf5x~syO#_-fEZDXjLd-yufU*?Uw3RfjvFw`Zo_{_mmbAcy#F$_I8~V
z+8#y!tNHgI7y#Y2#7hT*fo0x?qo$c*O}UO2ixe}5^jb%#ztYe`IJIm$dGwk727HRL
z%3GZ`8qR*xrDuCxpDVys#dEuBLvFc~Vay&_&Xf7?5k?CaJTvrP`vj9eQ<caqIimcJ
znF3Mj9zT3L+Z9Dl(yYe;`nK^asD?p`zF>g^?wGNzx3uE?(}&)U?dkBnZdXR2jsAKK
z3Gycfkx(;z>@#_oydK&Pz<Ra3E1i`Y<}4D{lDfML%O9Viy6k*iTs3ZkNsC{9)LXUw
z?bP4we0*PmWG^&lb<ifVUG>^e#}2q1=a)qw0x(f%SL{P9%?@IJ;30eR=@~T~s*i0g
zd-*<n0~G}n49M*tAH930<pSVETEHav)uVm+je-;C*&J+V+3?)qeUOirP1n`(@77n-
z2<BOub02cL#J;GjRnnv>&#}x*bJbWJ75URG<U5hw*ZpcwDmbcq!NaF%%wqrAbg+tB
zP3Sxb)7h&NZYx^T(dKgmIb`oeA&@zG9tG?#&6i)-#Gt`f1_#ZnDo>SkQkh<IyrJ;+
z@G;|QcKpHa1|QjTE3iv)hJE)X&h>KM9;PgnMtzC0-_31Wr6c>fe|PlBM=Aj>g_i^1
zjAplXSDG+A2^Y5?^KUcL!?O1ULf+j;8cjRH+oXF*@+0;t&s}{=10|Itm@1N*U`*#_
zJ$cewgg^f=wdCjS>jpsUyIN{N3`Af<Qjj1h1nlY$-|>Cp%Y{^c3|iYuy$1Q}O+Xb=
zv&goYGLqK}crEOe#}&g-(IKYlrv#L*tVsQx*DGs7L#jfHSbeMSzjE_kUB4abDf#ht
zgilfH<Kw;qsgG@$_vDMVM~hKnX1f5KcfL#PF~Gvee6E5+XM_~r35)Ej4}aWGTDIGW
zqTYx)^kzG1cuRFGMT!m&fEZ8^keKt%jfA~uCNe9j31kKsP|<;=PtF_nV#0bs@2nNS
z)Ki}rUu^FVB<sQ#{G?g7<j<(lQNFkRxa-Ll%9SF$n+;Gw!viIC#NUt(cS|cZCkHT_
z<ZfD}$|<s$Pfe3(*UIkEf%4e%%VM5m5o=M^8@(L<f|Gx(<C=m}ny%LN@dmISsm_ok
z-#RdMO$>13d5&>&OYKKkNCQp;`~1iv%R_(jk?ALTH7lrTx5!4D=a;@B`oF-vcvrT9
z7&w6xCUct}3$C<+pWPMKKzi)r8AYXs;D9I;<bfy%s-g2S>Pw%t)+e!xo`I<Uf!6cH
zB$M}i`}_73^4CpFaiE$KIJ5c4#V>LuSpknS8Q{G@3k?;W5@J@sNN6N{@>5t*A-^>4
zFHUNgkJ;f=!0#11%qg>4kvZ(W*BNa`*Edi`jU}CCxj*edDCK|O@tAMgbR<x9a~|DW
zOLFBWi2&6UqR_ebAsx<em~-};m5+!RfcRhj*_d~17jn6@{a_it!4&IsJ9wc7x(qy=
zW`&qP00Ghe%`NP)LhL5x!?NvmUK<huS#bz`xox;rltc5jy{~ites`~33CL)~ApOr&
zf?Wo~U1C4XLL)#%$w*&b9&YvWXVC2HrEyk1i&Y718ZlKaQVV|{`=I3^g8kg9Ug6t6
zVxdXiR9#o|(C3s(59$2j(!!$kLPIrSE?!Y!ldj)W5o)VDq2~e1Q-A=d`YI1fVVywk
z5B@wDGFmXK28KO|uq>~6cHXT~%LPE6B%Sd5xP!Xo=~MlNMQ1sKCHk{Dilk#)l*sZ|
z_iqPbIj*n&nLSI<iUYOT?<%{?1&fW$60Qd`>RXhkFv}ic)#KC2S8hjk70I4Qg~Q2=
zod>aeX$Hm(B!yF|8acSx>KKuowRGh`-B>co)1JLuXg$O6%hSY`50G4bnaQJuNPddB
zcoVl*FjE=Z$lP2(H>RlBc=Y*o71^QB<;2)6o&`Z<B<#7evhTaG`*$!vKtTr?CQ>e9
zh4;N+KF7}fmJyfTpMT5k^UWd|MthAR9EkB&@NM%y&mGZQjHmzQdW^kwx8=9pDEn|K
z)wCbzhWWv8XVIkbsAyPRRTegdBbGtGG}&p9i;gW(IXw!i+tQT_y%N+{0G+YZpBd88
zS;xIoVl|IY!0in0_jujqJF?OqVDqRMQc8ZOg%-}EPjTK6tceQOwfU?3&M|;@4I}^d
z@cZ^H62Fea9fR2hympi;-INjhUzK@=7f>2mQSZ4yo*!^qHcQcJgG`@3^{(kPAw5hr
zjcgepQBA(2w$ky2A2X<N`4d#61jC+i!3wBP(sB#+yux4P=_C6wncKaboX|+Ws^Ja#
z)r<K1RL$t<feN8G`IIBHD%H>d6#qXZZ-!aTwLy&|K1$YEhhVXFJ8>6RdYHDNrZTdS
z$J!4agQg27wCD({lnK84LSK^)S|=lYUkZAam)x$+hf6%ZUQ?>!;WPFq5_w{k!RcVk
zD?5@x+^3!<tEBbLe9o@@@92ZE`92-Gm=3zFJ7WH)c3N&T&>?-<e^7?6g!-Of^GcUb
zwo`B952AHB(?am$&dCf%atzShc|1u6?J_XAE_xNxYMgpr*;TO(0=*ijDur9`TMjSK
z(8g3u)_gj&660|kC~1(m400Dt-jGkI!NWO**&(evrmV&PBDi5Yo{IPj!<<y9d>!eg
z$N#u=vG3RxI5GqJ7bqFD9pI2C6YJmtPAC#{)xzxfU4~;j%?5QkAkk#Ed|;%Ek<fvU
zXS|;n3ax<p`*ep<mTUDrq}WW-2U{Swt^_N|JW?I$w^WOBfzZ?tX*<>%i@1X+EQiQ~
zMGZZ|UQ9vDnU_k0DfvxG0%5u#R;qeJ5^FA61qwZ+5$^8>934ye^tsKmVv$JGlh=AY
zq6N0(w|6C}Q<q6k+Bv$Y*L!hSALmVI$3v1%YJx^@zN;{<rjs!y=%l_6<G@O0{Egh7
zam}ea!pIX5+*f1{gisEh(8AL>?6Th}<`hGtjslMUz8@Q(6dp$#R0|h;k?{xsxBz_&
z2mkQDO4#=<w8ml1s1^+AqC<qcA%qIPIO}rTtF#S}m8!_OSI((fwUoa%IS`;UM!!!7
zFDmr~+#dlks7m1eZ?*E17-*dFcvjV3e<<`}^-q_y?rjsnC`-o)yUulkofo`h5#i{Z
zsf<y%qTc-%?kkJ?9BqjvBMee+R@2{E$89{Fw~J^QR~*8;@|Vqj0Wx?X#W15^lV_7V
zEG~0DIb;~NfwW4aUV{i;PW}BFZ0~J{(nbH<y`n$_YuWiJhu`g8ABIW9)lW@p8)a`i
z*9jVFd83ZL4A&}e{GDvzKIs{9v$nfP0BZuGyx-{SE$Ew~LkJ1EOw-q*(iX1_@ZwZM
z52m)Mze8X6#;37H_UO?J0FLAhtRVcN+B=9qAdT=4Jw?6!{RXi|j-5n2Nhm$4)xq0u
zyjEeWbEuJU^n+|QqEzIp<Z!PUya1l_f<(~Irt7lx(qA=+pYmyz8gG8Q!Nm=3@>(!Y
zS}sBxvZR-XgEK2`&3Tp1KFVi)yS{k8fryt3YrIP;`2I3T`>4`}=^huejP*JF`F>(q
z+TX?T^`~Eyb=Ik}5MvSp<ca)Ys5wHU98CBSgHD_KWQbnu{rX#mS}ZTbS7pZ}f%<PY
z!c9<qN}O8RbT_W8KSCtl#`#S_40xgwKk~>X(1GlD$4j9}MS;uq`YRx>;ngeB`cG4(
zjal)t()b6ZwF5ReKFchnGAekn^!pViVP{$dN)eDP51zGO0&QFzgvL8@C-IR9y$7V=
zZr{X`$Qgvb>&35}ly_0+R34{;w><WnJk|+61=5oy4ZcUahXqy&nZAj$o~u!&#w3~z
zy9TFbWCTSl62^KYnGq%A{^Hi|H^g_*L!&yvHlCNCGS!Moi9*P*8Y!7*JT7jT!;}n=
zl?IE4#WxaXI{tMLpml}A${Em7YYW<nNitAYGj-@RNAHfAElImr+f!JuP||HU^`r#(
zQbL^3sg%bjCICq$F+M(Ch8ic8&4r}C8@7OrfVo~$kh%F+A5GxEnIc2?Sd6*cPza_F
zN1sUKryMk2=5MAO0`w3sWfPNCz>LBq;f>4AW??be;1!RTdGRz5NWv@F0T(;`CRCO$
z#Mc?+Y!&W)AHD>?XEP}&rJw_!7W@{|{%m1kK~#g)umvlXE$vk|H_OvUof-!$+!h8F
z6X8(S>+8Nxo~Q*cKu99$DM<Rxc6&4{J3D&~N6JuQh}nc-pe=ZSVa>?hn50%=fQ1P+
zHA%qcrYKhMZO)x}R0%sOK3=jZ;DRhPx!HlfXR4~I;Fph}ZcW4@(H6IhKzWFzXp?ez
zLp-(+_8$rP@l2PFUN54Mt^&8RekZ*K*ap~nDnz8D{(Dnpes`A^=H}rQ9CFUG>B<_X
z7d)e&8=1}rHCK>(m53dVUg%7}!P!oPSeVv&&7<KeQk`{Aa-i2MywitZDbqXuQqz7j
z$-Pe(R5uh-`aT2=EWMhV+S=+WOn#HgW@H0()Un(ROUDm<E2E=UN0)<`#Ny;9Ke+m7
z!Vk1Ze)$fE86zw>vFd}o-J3^Tc>=wz-IFykHjQjK+N101H}-4h_$eLjQR=W^tfKM^
zpdVahQfH_5%a>8Wk6T+?|7y={_}jp+nJPVA%gN;1N|l&?13j)V8fJ7wmnHdrxOHt1
zt1p4uv!29p>3lPjIRQsh^W&$H?W%nHzqk(;M-D^ql=%TT$WU^}A4%d!&>~|Kj&y#0
zegHnV`QEs=yVu?8>;L>WT=9J2cSFWeX0%Y7Eix{%pUc89)PS@YkMTOB<75_3RWe%-
zye<31%`L**t^p<)gieu!*GO;K*}>}Yx$S9qdh+J5-nkAp6y0xH8P%t|o&J3PvEfrZ
z^xxWPeM<rHr!4}8lW_*6h+iCE{JJSO1Enn~<uA1j&GtZvggD^uGLe%T|M{weghBHC
z#}B|z#YQ%7G&zlxCH}FhmD)q7xIooSvnk)GlBBAlrB6E)Ea74Bmej{`BH3sEA68EL
zgWnp$-H%eHh>T-nJP=Mp(bE{f$7yQ1Z4M@bje2}}IXXNH9~fc9V>*$Z*?JWdpy%Xp
zfshpkzuD?1#b5HT$jJSxg7U{pWoFaw^2zrLb=oj|w-k-XV-(rF^!`8*mkcZm;9Pw(
zY}Hj%9GRZxqNSzf;^JatjOsD%Xt!~$#=|U+lFXm7zvNS#UnVr#X+JD(w4$z3@H+Zq
zMy6LiKN4j~pY5=4d@XMA2iJH1>zYg3i>g<4UqE0RKRY+~9&p3@<0xFWhWP*u1-xl$
zdKyqxD=RA#6B2&u9gi)1v@-nlW$=1amiX(1_Jx1CtM|XrFoGr35d9)yr^?+_zur*n
zbeiDLsJir4ufj&uC$v!8>@2aLs;vm9+BI?)7R}X1=<<UCOeOPVb7^$n=?eK>-k+z}
z>eav3tG1oqU{DX(pR(C3*Nn{(1QZp9b?y6cS)FnWC;1yg?15HIhkLe*>lWNEfzHu5
z{ykc1>3sI_rwcp+S?W?{@~;DrB@_r8Kj}8ODfp}E^n~h`t-OS$hgi^6s)5T6s?vH2
z@~f5gn0~l~0B2vy@*fG9pj40|*5@q3`zkvihDhKsCgi<fMH)wL9W{PR2K>0sx5mej
zZYm_h+1?s?2Dqfeur>uoT~+bGIJH#n2xw>H7w0=B;#DSL3#~*tP@=80IjRzyb<D}n
zS<jR#{pKS*i$X+jG^kl@-GG}o2Rc7HnG<RDqxe7jsCuI_TAixPYNcJoh%MfYILC}-
zd#-q8=#(h!H*^OVc}~F$qO`tC8pQ8lB`RXk#NiDZVo85mK=Cj;UYTZaO%5z=ap7{b
zR8eAp6A4p{(#K17!Rcc{R#0g6y;}WLHd!U&OOTu{O{{|3SsTf+(^Q}C=RFa)UHuhj
zP`@x<cGLfO3^E-%9M%@eQDKv)#*~kby7CfH8PACP<CFX-^vesNR6$EAq&?Jcx|Fcl
z$7+aDY^Ad0kA3wiGdcAlz`*J7*2i&>$zscZx{o1Uo-rwqIX6CN=KY3}Qt0of5X+Mg
zq)H-f{aX6kxMH9`G#4?(>}}>yQDZURwr;i1X-e-}(ML&bu~Ub{TugFM|9IIAH^#8z
z-J{yeh~hJ>mW{X}wUw;CHvv*+?O)aMD^y6=cwsB!T7KTjHjk)hsWl`uJ{P}-_~vdp
zD;l)8lkquU|AQW-Q4cQIB9F0n?_=L}a~G=lCg>!c48E8(J*~BF4ZoM#FU^f|QD9lJ
z%ijA^uK(IjUU!_YFM8i$ZnR8)Q;@bqFt;=gRV`{L=FOt!6{(>6s;wk{A_rj^1rVgD
zi+}<_3V{-**9V^dqV99h?O|rAfxJfF!LP+J0a|4SYR*ndi^>S%cnEm-l(d1GUvHM|
zAL_E|b8J;CDY&RBH-fn;cYa8C7w}PKQ~gFZt(+(#?!i;B<dWTiiU-qwW(-cHkCONp
z{uR_7K4GWFdni9lCul-$xj!fA$SA&0h1L%FamelT#QY$RSx5drHJh6G*5sv9`t*5t
zKV$oLR_!$XjR78An`k3OKLap98@Rsti5NMp4If6Da{uPQhxfzl=CLWDxZlzczN~7p
zWcGa%(8D#vQ;UZw#{iNypCZ*&Hgu=o5FVBSmdgd3XTGszx`&jkT!FoW4MHa}fk<`%
zUJMk7L5K&6LPbt2oDB7aFV*CbB2n|iht}J;t4X-M?a7&AI9`^87K*P5Id19nMf2_s
z!RcF*YF`VI&t!EJ440#Nn|R4c1)3=oLuDfM#4#+s`6gVRkdkeT;l7f#FH8aj6>2E{
zCkz?x|K{ATJzXfW)il*tSRXW-BDS;Mz82FECN63oQFG5IzP3zJv-DFE_t$aa@8e|y
z5Y-}<=1<VUf<>@Gf|x4oV`Aq!->6UG>qHn91o`%0ZRKQ`33!p9kJHUn5~4&vAt}li
z<Nko`%g~WC*Q`*b<lle`2?4hr8rKT!R$8lDpn7IdbqzMWPi5n{_f{D|-|n4nQEmt|
zU8?)vwr+ftxcx+@3IszKT(K?SpdlJc8U%O8=`Y`IGl!2c7Y>UH)0N3sztUzq_lK_(
z-LqP5EOzn8C|S5N`LmJjKdx|?P4}m(PDI~h{ec;GS-mO?K<3=C(25J3l8Qc}K1h$v
zmCObB>_HQ}m%gTnk2~X&e7qb*r6-4og$>lM&}<u;7^f#s&xoX0537<XMg-l~BvY8y
z?}Cjoak@-X&ar^)6H<Hq4b6eGndB(=GdR_Sa%-+pf65MLeeN|hXfwc4;oXQ#J@ry)
zIY6O|cr8)G9&#+9Rm3f^W3poM6l7wm6q>M#$<mVgIgfA;xN0C8mL<|x4LZNNhPi5W
z{XlhEFze>%w5g@xP}VYXlt^ILXBSre`oP6TQWfmu>3Fq%GC|)BiAo-JKF=-9<Zc)E
zzj^;~>vog8yYg{+vrCAP^2^5-g8@%~wi<zPdBQ!ts3mGma%=Ta_!AJxxPb-6$Ka=b
ztuj-VnzW4XTlxA7>gCg<KRvQGG-UT2@`TnsS+2v#{pB<3cAM&8bI_0*r)snWlc_x)
zftL;N(lHsFJyShVEybvRnOfQ+KXu^hjH8ychMy=@?f$I)i6Ci_uJU0jbhUS-2ybSe
zEiE$~5<?ehnc9wF(MO0p;fYUIKu7qpI$j*$CLz9ovm~QabFQ%l4X64u$JtGPdIT^r
zavF~%rA7E(j(sovs6{^&J8X24LOV^xD8`4+W0g-hePxag8s?bB|7JmH$*5bTrU~ll
zdrKP!-~{nf>hF<nU^_X{6&;0CMSLDc0~4+8(1@lyR&pt*qyQ1Ge)7oRqmZ<6ulagd
zVNfAup7bkaRyBx74Bj_-w!5LL>{31sLVV9f!8YRzY`<pN8cP|{12btTaWFZaiUxA>
zto`TEz=0hUTyVL3xJzMfe{On`vvt(=0NNJjGlg*S=8A@yAA0pgBmSE<XL%-8I@@<R
z&vT|~*c>oGzmNjUAZJ_si-y{J>psMk)h~tEbvvWkTRlZhDOUv9>ZHiI`NO*s#^B~Y
zCbukqoJy6r%ejwD6u7EI{{O&rNG`mYoWqk+&<6H7LtKah#O`{PNVA$h;8(~}GPgbX
z-#toDoGT0heM?u=t3XWkSFypUCa0{8<}AM>YxD6{EvLhJ2{pw7f3Z|w=ezd0)!Vnf
zS`a4`1LdFK#3{HxR=?ENeLRM<m@xu=T<G^yF`Aar4z#n+e;A=^eUVrn)X%ObbuQX`
zF(q33m0FdKKo`4o;km4CCublhdUh`SrB*k!?jBuHAUdlMK9)kN5GE3}Ea7^glc9FY
z7s%U~O8X6O5Z5jVG^j~I=p4|NR5N~bajSXRYD2Uo88xQv{siZQv##jPI>vkdj$HjL
zT^Q&6F<-o#I)*N&t=w#Tx!rat^ScI6Oz<ET#E~&m0Z(!4Jga|AC($QbHiLAmZkiWo
z*dA46F5i<L$ybEmH+}p;xG)f$A0yHEsq!wzHz=xZyMG=Uzfo9xoLI|JH6=AlQ~4d|
z7Yh|%#kT8}FrqE`M*p$s*FwNDGT%`ecKpn7*df*?{PTX@5PvL>$<V8}fy@8G>sMTz
zhy<R|xe{UAyr8nZ4pZ!mV0V!ys=<B}d8LnU4Z5BIvK^gBtSY8y&-bC~0`XU<c=)Fl
zzUw>dw2C(4Yj*}=u{V7sDq9c=S!gF>J7$Onzhj2;DMq5XHli{%&&i7Y@^HewgN_55
zp?Dw`a|kCnIEO=QkqA}M!{96)Ye<v50`vl!?k`#&=4jM;@+kro^XE20RY@3iO={Li
za$)u6$D@9B%>wbR-#VQFVXPZZdHUg?^Hc)6)L7?qoj|{cw$^KAGT)6^N_05`;fV0W
z!?Pj4Lp<P?uU7o(*=;1zu@ZrU3Loi0W3Lw9&t0|?Epe;%<}Y@}4;2P|*-&z7dI<5^
zzqt9pjw<}1EV!J*2kBt7BxTrO7Y31S#BRC*^Lm{<i`1{zu~#~!bYgrKz|t85>_2lo
z-O%MPg;}iEo_auv1P38&oOmJVRnQB_*m%fgk$Zk<+h5H>RTbbk-**hxJP?%mjHn4t
z;7Mrs>l8_j5st29^y2@e#pf9<h{2{og3^Er)d|t^{L@i-XN4`Pt?B*l7S3$?Pde_u
zBdeuiFS#=n7vPm00zLLqbzJ7D?!vjc_(Cc~rN`p{K~}C8bRG!QfKHRMdE#&Pzvq;h
zi0W64bCeHEQCLU){Ej_kpE}d6-${8ruQW~yz&!tXnOr`q#c5uaZQLQ<!Ftxrn}n#%
zvmFdiyd|RQRLsXGCUTD|uX6VuI4@^uGD;wI!~uSYUPxWu=ilhj;^MpS{PI}i8$D;L
zW7cdOkEG)}QB}Zru9JmUNz84Xgs8}_Yc9aN`6XX~rSrUA^bi|zbk#uPKRYx_m%}J#
z(<d7LHcYWO&1V*Cx!X(BEe9|kb&6tr(~mK<+lm%1hl_`ySm_`Q@!ww>xDV(*hFYd-
z^x2iDHBgRmE-~Dl&#{_~tU>tkZ(y3}KU$|{Bh{p|7iD=2A`f<0NmHD{f|=>p*C?Ug
z%gJ%L=G!hN&P+c&?B;C^eo`akW7kUMxOV;g*;hiRoE^wKYwb;B-)&3P?t9jdfzQ%^
zyKU2(;kS@OOZPhp>P!;oXeq0rMC|+{+K{?CDj&-W4$LDkk}^D@*ULtflt5lHEk|+N
zlejl4GdI8~yl#UM<JIb#v*z|D4B-Iu6*0q`f^s_co6V)u4<@W1R71xzTer%Bk3w^R
z&l+VYKoiB{0}5ki!2L-|QpT&0n-&o+8d>>yHefV@#E6z?u=pkYsjINKTy;NvlkwlE
z+4VXbaI#F}zr#4f#gd}~EU33XqSQqVHQMnyUf}Q>3+K4db~08PLhUDURbBg~i=N^Q
z%1@=deEM{i$S&BZik35EtumgoUslt#VD;Pwzp^I0qxRex0{r(~z$xXBr~Qo@qV*%c
z&wq^!!`BWR!V?mhB`Jb<SFjnRlq8dr)1J1HkaPrjCGW1^8N{h4`F#1VPft(Sxv;+e
zA)@h9>8wtVR&3A!bR#kp2j6}!2jP3MU~EMH`b-@$YCmMrvEyA?#&v%zG3~xj9pZM9
z(D{JQw?sDSw}-_(+)G!;s({$bAteljI$5WU7BIUP=?kMBb4Rudj-{0{MwBKrQug5;
zUel>wwYcs%`3G7$En(Y*x)TXR8jQwWkcu=H=7UC?GYd6LIH(e77PI0MkpR-`1#DD|
z`@7qn()yOp$T68r$KH|6ElC4eN&G}s&#j#1j*l1*DsL2U;|%tam)Pp7OPPO&2MV=b
zr@epGkFo@(kw1sO?VOvBg#f@>vke^$q^D_NA=w-(%|e0oUbVOR->CER2~ie_1?6>B
z(Z@J|h(<Uo)`HW}=@=wA8Iwpb-shELtQYFFTNSk28|!5bUI-_sQ{w>aR5$bP*NO8m
z_X#dU##Pw<{>2ok+Sz6rx%9vBi{3E+eXwo3LztndgWiGc-IgRNJp7g)Mp_JB%>}>y
zO~|vPWBR$~l|>GuJ{U^Fmx%zwMwOG#FQ~u02aCH{n?3j8vkE<AE2(YvKqk9iIy_Qa
zgJ3(u<bn57&<T3G>tv5yBK`%oCPaB7zIPJUWahpxJQtM*(Xq;FkY&<*JuQeAV<(SY
z5<gA-l0V6BfX}sxZ#_YtKqq#j5BDaH_CGTlQHq)-7~uN%&aMr<*aNm@sn3FDv+FB|
zVPmQZ$($0*r_yp%f!dwyNPNC`w4gxZ@pnA0)gh;B#7NbyBK-~&|6oeC`TW<v8cqgA
zND7q<Jr)k-4<#mk+pM~2darf_3DyKEJ~Xi}Kz5dxiY`B2>Re2Pg^2-TR!neBM5k|H
z?P9$Xz*9mlsi~!w9sdiXx<LJSDz@?T=?u;SngzxGGSt4GT)QjupvarS5d~XH`AG-S
zA9bX%ArJ04LW{oVjJ8w9R~vDRJkqs~v5)Cn9IrGLdGi!^HS!YT(mArt-`rC3=*;l)
zB)#nPU!Ttg$ipw>Vx43!*gppZj2!hN?-eyGUK1tD+IKkAPR$$6P21ms{!mP9_?0^;
z(w91YXm(#TvghJ^D`pF1T4Dg~0E|H5C2_?E+|*s_7UcfQ?XwVU$!by60U|Hq^n3_V
zL@v`j$~u!{4J0kPMMU5)DS>H}|AEU*M7=BR5?=C>8?Q`X703zj)J_E&+LIcs#0`f~
z$_!E09cS}Qa+jrr&g@YRg8oU*4nj;bUv4$BQ|F$X_5z0B`$g;~^3L=UHOfB(W(;1c
zY8j^?s_jr|cZsYU^ok;U)CgRJ)%PtZW6Q>(WLMO-w^3<!4%d?OASxJmg_q0;nxvv-
z_lI=M-)H;J@lmPfTfr)qB5f@>jkzdQS{n%?;o<aQx5TdTmN<3LCpBKc{6Vy<Q-n;Q
z4V(>?H*gSsu*MpJ!S*grrN--DF*SjYm-w%T0XaTt-2upA_tEFuM_0Q0wD<hi5vA?O
z<|8Vxtvi`IyKioQDqMl}DDuz#$NEl|W^=B87q!MC6jzk|ndl@p<fnd|*>z#$AQAl^
z-*MbnnyKzpaSmYuS;s5D`}Y^zwSqFjYA>EyD*0hoY>h;lYr4$Ob&KaoIV~k^vrE69
z&{u2x0o$={W(#{R+m>DgWITLE)Om<$dtAFq#0Aey#Gmkzy29GZ*{iUnuzFzs$Dw|t
zg!u3idY5U@fgZi^E9C{;XupGP>;vDcss8BpGfRvL`OL78+TnG~WzEngij}F32xm)!
zqnr0jpa5Q%^WAHOp!_YH9@g=^H`SIq(5=1)BaQDaPVW{s!pH|f8Sa04d|_#3LL=bD
z|K!_L)<w1U@3-n|rv9xg9(aw1LS&=NDr`HFR)No>{Pr+N9kDU=OV>)oq0d_&VrgmJ
zJvtTcw=#lerTGsMJ<RA~b=)!`ACzu^f&jbbVgN5@3mSAkd&_!l9l`nCW8>hFMjON=
z9ZhG8Koq7^>)M0OMkydN0m}f{N%^x!S10nC+=gMLLSiW~8+vDzd~1=Mu$Z7JzN&cj
z6@X&}M4ugLFNQ@?^BOH2yUxaV&F$H8yX1GIweAN$9vg;i2%V1jy&2^G%>40U=2$Bb
zIY{GqjNtu>IN};o><B|k)V>Z3K^RsbCEHsyE<c50*5})zr>0@(W$bPvNh%U)fMJBI
ztzL90`6mKOcQARCR`hLJxoJNcH%Z-@5fwz9AnKpI+y&56_Vs2bH6Dz0p0V>nUKlz0
z^GKf~rh21M+<>hQ7isP#B6I6>vOuh&i%b%13fF0xM5ox-!5$;bJ{78`mJ+wv`<%56
zu=GKt3Z^eH-f{er1&GUXpHnVeus4bHN|xc;=t{`u>Fdi(H+V66U?l)L|Ek5qcz7!O
zz1P<@l6G5_ab<WoA<TXnCH|`sCclkTp9z+S7WL3Cpegm{x&eip@W)87w&b<3VX380
zLC8Teukn(j#Z+bCaQl7$<1YUJ$J+q&=CBmAN10kcF%dnV7hG^RF4`INRpowOm55>$
zo`9oC1db;8XH~JV50g|r)$aEq9$g|LYSZ^|R|q9&rXr+4<fq`%{xkZG-CY<yi!T&j
z^NI;b%XgC6Mr4;>uSZi_QgOh4fzp%z?l@$Cx2?~@{+<Te(e});-<r*nj}B-KTpmC?
zT~g`UR)Kc{Z9w>Q(>zyHk1$!e<&3-`Hyqd+xs$(o@bLlvJ=o<q_$a|wVyp=g`V1lD
z;hFz?K0%ThIbZa@bCz3*RS2|G|8sJ|#pyqgVJ~NK>F5~SIs0ej*pW%0b+->2uWKAU
zd%_6P@J&<ieM{gJglDZ2WrvheNB#Y#l1U_7oKxc0hmeU@a!HjmMiqHJPK!fvbVZ9Z
z9EKl${@Dlug*B-E<3%RZ@|!sO=tp@Zei43IL=oh^WmfeKnC1}@j>R7F21b2&!$ZZZ
zZ3tT!{a#Hh>2)v>RYZqHPNq2q+A&W7c5+nha+hrgY+kAuiL<s_>zN_xZS?^5aCcm$
z2ce%*^fV2jBMG>APXJBV0iroj*-DHlrQu+!aR^+bRY@e-nOL|SH4c8YJQ~Xl3F(Q^
zk(*VrMT5GnS2`A=iX;s^;g@Cns>^QP#fp~-qBiGjmbQ2mTgpwg^#Avv#F27-Rjz7I
z?sD+B-V6g_s)41SHpRbd$YPU`xI5R4cToK8W*S23;QOG2-TdQ6no&$CvXZj@_B|w_
zYvc9{_0+wIHahJ8&Ic8vYAgR5!FFPNhL*KEPn;<>mZn!ad15byhumr^DiaxE4p9lx
z|0(Ru<DuNU04^pm*2t2C7|cwCCX^-al)*%pY}chy=vJ0)mh9UIV^4_4nq@4>MY=+a
z?b?!Ly{<J%5tFi1gZB)#`);4l`_5m#&*xd@{FZZ`^PK1WzGr#YKBuX5T*67WE5TM-
zzbZx|d+I0B6S|1V_F#|G19?2(Z}(~~M*yjF<ovtON^(!K_NYss&xY;J$q9d>IX=l^
zhZFqtNbv}pW%XQZ84meDrSY6c&NVbP=V~Vr;}!VavRuLWdpA=cm0cgr%IVNt;T`JG
zb2vTCMSrg@1(_;e78^(M!^H75jb(f^jB=n&#Z?WZj&V}6sS+;bF)R9=ooHOWB-$!j
z6^%t)1+cw+bji*2cj>F$`nDClT<r<#`ue2(;zj<>zg|^%5R_z*B9WY&Y{ea7Y}8X+
zP0d(zH{@vRBDTiDM(?N0HH*`{$Lg2Ri8hRS9^&$DdM)yS2_lzDA?iKkaKYa*K_fkh
zbrl_~SQU|^1*=CT&3;!V&d<~A%s9w>yFsPnv#D8ApRwgZ?FJlOKHd%YY7r^YMHQyp
zjp7LjC(E;qq6@RB6OL)}2fhrq1Y1OiXln?_nSUP#<{J8(>ltuaxy*=i;+%7x$_+7f
zf?4J#0%`x=Qa4v}#@F-mKN5M<?La?7hp+stPvt^FSyf}$4w)EhllS@iPTl{J^fPH#
zlk<f3D|#PdJv`0XDKk8;rF;n&`A{-E<9?R0!U#Y%{?bMWE79L1>iO85CmtEc%dpDL
z$zeZ+&u(IDU7ONPGpT<h7^uMa*zNH(w2OQ{!T;g4oxAaw9u3^o^WebEVo8wh(*1>Y
zPM_uj)TSnPTkdGoE|Z`Z<JdtN^VQA%_XezNv2os9#nQ>w48Qj_rFBoz(%)tPOg4mL
z@b+AFS%w~qw2KdZUu#Cix8(YbRXkyL6Fxlbzvjh%BAv+++P0iP8)&<WGJ01z{OOtF
z%SKsLBo+1MlSEyB#?7fjD=y)8R>Sh;-qRDzTps3YCj1ls@KQBZndiCOi<A?dAIp!u
zsj95K<jMp>ZJ9ZU!%&>EDIG)YcC_*&F3XYuC~$4t-+dOa*i3V|C0|r7)$m-c@I#vu
zb_-+?pT*Z!gv59i=DQud<3l}Asq)Z0{oSYZ@FccfeARx7q~kK-xRNmD%IqKcExDlX
zc9<;oSB1dG2Vg_`?VQxdwl60xHigHEubX^2<HWmtE<XZ)n_bs+F!OHqa2e^r&t2A8
zj#v3SVAmwG7YMwVD3L^)&eP!Y*%<Y0fB$%3EEp5&*Upq9REmSFth9$DJ1w-R=Cia~
z8dx%1*)JoVK@&R73q&z20?(ft3@O|k`9GJKj=r3UbE%0H=QDUXY6W4tt%I;TAn4Q&
zLybbkzKs4k#zWu27S^p~EDo}iKAUMFRn%tE(XMm1UF-6G`U|!*`~P_`p;eG@xs3d7
zFNQF4XBlNg)bVi<l?lK5ds2xdA51SlBHj_^pDabO3EAq&v>1~b!67{es}cUZntiJS
zZ7N%$=#)ZO8cud)nw<Z>G!;uoviiCMj^OQii?O5#M^SF}5Lz3sZJ3StIjG_V4Vgi3
z)1jeT8p5oW1GO_Cph$y8HE$#`1RAdUp+XKcrq$8dts8j!zZ<+2Bqb%@m1g#%3o1Lb
zq~=gM{-g<QNfK*mOZm}sTYez*{SW#7U)M33tpmJq=WET_xeqy^$2G`O9k3E7y&dhV
zep1WHexUbd%%NN3=$y!-v9_ahUd&?kj+ZK`1-18UKaTarE@kB13a;jv$(!NvH$B+l
zYXRA+NGLNKrs<Wh5a6~X_P9V14VP5k{8qZ=Fs%}^a?iV@6ca%7eqN_%j9IjTxFkjs
z-o?l{0`<|g>;wYLO&g?-a^p!67j(}<Dkw~$J-8CExSJJa9sv>px?7GQ$bz>&l^hRK
z+6J>~1^RaYjs4i!1~YGOVn#Ux;h2^A;nx!rw?AK%mX^Msp8g_TLtdiNXBp7uHa0fR
z71}X1HQxZ$+WcA!4LltLf#Ch(EO_EzDUor?IvxNUZEMrl%3PeC^_m-aI45amXP0pQ
zKKD~H^2j`ZGH=|tL5~QwY8J&Q8JL?Vi4K?m(hh{^@VyIfmxTbN@XD1h>g_%2?EC>h
z-U)<R>eABE{QUgN!YCJfoS|POSzeF+@EEEWHAe{2*(xh5OUx$eulolF!*y4Hh^xdI
z;$OtU7>XcV6HwdT51eyJO)nPz72s^P<QcoIE}ZlA^*wc}>49FH#PMH#(9_dnj0T7;
zRaiE)6;OeJ%$|v`wjMdH%?SwDz;7#u88KdOyZ5KulBBGqZ5|R)u#pljBr?b9sxk?m
zq2-^Nnu>IN3YcI>w!J1>1W4tR2FU!tmHB(Wp?0qg1JfTRCs$NhNJGt@mg3JXC~))_
z9*Px=>O;9o7kSJL)CH`IRVf0q5vZvE3ZI&uMz~!Z;Ta!=L1r)*zuCCx>r>WzHv%&>
zz0VdKozo)htUvCqp58QB>-F)Z&7C$N*<MsYJPT?E-f@HcfrAGP7&5o@^R(NG;M;~h
z5grOK2DbUp(a~>TWHzB7q?Pl>lk1t))oKj0_4V}@*@%WiL#}u|f;+bqRx6V)&Et|w
zx{P+s25>faiiPo%=F54cJ#!o^ot?d6a(qZE5gz|ldb+y0Q@cH4{35SGeh#ETI5;@i
zUb;b)Zf$L)YXHEORZ<l8yU^qh$jODJSifbiw%(K+vIUEJ(a_M3fL&tBK|Z52yDNa!
z@R=?sY^nXGEnWtX#{<iDTDSIH1X8ue!os4ZrA0zQVybtOEpS!oUKf>$zI4GNL8Koj
zqj>Gm&`>oswH~2|Ru#IMcFXT!D4JHgFa7H7?oNR(S4=I5k;4w9Jguo|dUX5o<7L1K
zXPpG*{LItpYFbqox0C&o>*Gd;`h%b>6Fyc_n>RH)GX=Pu({DHfx_hqa=;+wg`6N-5
zPmu6^eSNkgCm+0=s6Z+Bb2~w~X~1mv{6w=)5D`M$LM&h~m?KI`LG`xH{r$z0ib-8v
zhO>%~F4P-AHWJe|#Ai?^2Af)1cB(3PYPXJ!WzcLJBsb`3r?_Qwb~Zsq5v%Ma7nk$J
zb#GM+1^N47CH_F7i?BF(0{NLD&A?2|&vVzi!|>)#q~%g4*aXGIko!`B`Wt0XYzaHs
zj;X2b_2F7_%j0Ik)klCV+}Y9b!^pm4E~rR4v8<w^c!-gjmXL|qBLtc7pkz>EEG0di
z2TPV?5fB&WE+sq1c=0cMEv}<Vfog=7?g_X8sW?(L9`i#-hPd+z3Iq#C{5wcV$;t9^
za_1S!)=yl7;;2^Br_JbZF4PC`@W}ezZBu3K;u+a5DJi+u&JWrO6Vy`}RsSI__~~@)
zY^sh6VP`NOi=TjP2-o=WFT|iybs3r0P@|w(l!JoQ;-o*xS}UM$_N^@O!3TT;u}`8x
zvS4H=NHLDt(-&W{1!`RXqmmN~?Do#*&qL1~b)rNZ;pb1%M`1CTU<C`@oj^3Gl`NLS
z>E=B{ImQlm#@dY>*0jqKajR1;Q2={q%ETJXQl&R_{hM85V<W{sL?uo$3i&KKD2Qdv
zjelc9W@DqLzdy8FAo5Off0Hpc2-S(+z%aCv70wAjcbS?G?|Zk^Dc4@N#sF&iz=rM^
z(597zfw?Nq8-85mo?*hD%e#UXXq}y0RXA9}N}R5&*J+nm0ma_TLcz^{4QaSO0LeWK
zztf3Bb-u^^1pXWyoqQ7Ey3D%p6$1WSkAgnD8ULE8`oV)-0W-Eyev^%%VCg=8c3+xM
zi_gav@$6Ft0CZgEjGVlDQ~vP4fUwqO&=as(B5@{SB)plq`B5A$>moUh_T|&t$)8yh
znmwL@iD#}vrua`Iu7Pz{P*5oIh8(b)9v-3N>Q@v|Px@}>W3e2{#yux#p8j8!X25z$
z#CgInw!29s>6xXeI=i~Ml2^pE*!kDY0h}E&uJv&~4o`x#(op!Ca_YTY5fl_?tP)l2
z0nHp99`^S3270-)F)OO~7U8k)stdrI*A^y@vX5L5i_wj*Tw>0<ywQCr1*)_bXOGKS
zEl3=OlI3*$Q)VC`EaP#&;6rz0H~aGJo5Fa?A|N;F7iVKav2|)RRm!@`Df`vCvK;~(
zh%z2^kN1gQt(&iI5q4U?g&GuY)qj?k<AxSy=jKd}Y)vId_=#k+mX<`o9y*Z{6O*JF
zL+02v6TSRNe)*UP<i?(!moGt;gD+?vKK$^7v5HG$4tajGM&u7nyguSN!rN7KZ=Qe<
z<G|Y2<+-V;OKYeGy{S(JXGnjKta=nu)fhO?tougmd_5uUG4Zg9JwTtKBO8oGsA&WS
z+!--63oQP@M;A8z#&OECfQJsV>M4mb@Q7gm@i_AqXl;Z1MX;=yfdZh+?3uTI8b#Oj
zet%|r2%A43Xc79aKz4zO(O*jvs_-zK1vD0i#_HS!uNJ~B4BQg9GJBcI1knI^{s-*!
z=824i%GU9+|LH&6@w{xYD^A>;y^<#yGhWm6uf7DoECk*Ez80UDD6e>-&}Bs7h33ni
zoZ7eS)7>GzCnhHxt90i<FIt_}9PG`mt*rex`^8|}yV-IpWiZ$t!EJX@@!vho)DA++
z<S*#gOg~dc?k|5cOu7g(6vlYl4OP^SM1DpI3n^P9-hkV2NJB#V_uEdf2yNe2MoUB}
txzm|FM#w=N&LOuUh@{=$)_=2E*kqAV>Wv<iEYD+sACX|BQ+m>t{13q)j6eVY
literal 0
HcmV?d00001
diff --git a/drivers/common/Makefile b/drivers/common/Makefile
index 96bd7ac6e..df2e840cf 100644
--- a/drivers/common/Makefile
+++ b/drivers/common/Makefile
@@ -31,6 +31,7 @@ DIRS-y += dpaax
endif
IAVF-y := $(CONFIG_RTE_LIBRTE_IAVF_PMD)
+IAVF-y += $(CONFIG_RTE_LIBRTE_ICE_PMD)
ifneq (,$(findstring y,$(IAVF-y)))
DIRS-y += iavf
endif
diff --git a/drivers/net/ice/Makefile b/drivers/net/ice/Makefile
index e22c34287..643639a50 100644
--- a/drivers/net/ice/Makefile
+++ b/drivers/net/ice/Makefile
@@ -11,9 +11,11 @@ LIB = librte_pmd_ice.a
CFLAGS += -O3
CFLAGS += $(WERROR_FLAGS)
CFLAGS += -DALLOW_EXPERIMENTAL_API
+CFLAGS += -I$(RTE_SDK)/drivers/common/iavf
LDLIBS += -lrte_eal -lrte_mbuf -lrte_ethdev -lrte_kvargs
LDLIBS += -lrte_bus_pci -lrte_mempool -lrte_hash
+LDLIBS += -lrte_net -lrte_common_iavf
EXPORT_MAP := rte_pmd_ice_version.map
@@ -84,6 +86,10 @@ ifeq ($(CC_AVX2_SUPPORT), 1)
endif
SRCS-$(CONFIG_RTE_LIBRTE_ICE_PMD) += ice_generic_flow.c
+SRCS-$(CONFIG_RTE_LIBRTE_ICE_PMD) += ice_dcf.c
+SRCS-$(CONFIG_RTE_LIBRTE_ICE_PMD) += ice_dcf_parent.c
+SRCS-$(CONFIG_RTE_LIBRTE_ICE_PMD) += ice_dcf_ethdev.c
+
# install this header file
SYMLINK-$(CONFIG_RTE_LIBRTE_ICE_PMD)-include := rte_pmd_ice.h
diff --git a/drivers/net/ice/ice_dcf.c b/drivers/net/ice/ice_dcf.c
new file mode 100644
index 000000000..669122331
--- /dev/null
+++ b/drivers/net/ice/ice_dcf.c
@@ -0,0 +1,651 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+
+#include <sys/queue.h>
+#include <stdio.h>
+#include <errno.h>
+#include <stdint.h>
+#include <string.h>
+#include <unistd.h>
+#include <stdarg.h>
+#include <inttypes.h>
+#include <rte_byteorder.h>
+#include <rte_common.h>
+
+#include <rte_pci.h>
+#include <rte_atomic.h>
+#include <rte_eal.h>
+#include <rte_ether.h>
+#include <rte_ethdev_driver.h>
+#include <rte_ethdev_pci.h>
+#include <rte_malloc.h>
+#include <rte_memzone.h>
+#include <rte_dev.h>
+
+#include "ice_dcf.h"
+
+#define ICE_DCF_AQ_LEN 32
+#define ICE_DCF_AQ_BUF_SZ 4096
+
+#define ICE_DCF_ARQ_MAX_RETRIES 200
+#define ICE_DCF_ARQ_CHECK_TIME 2 /* msecs */
+
+#define ICE_DCF_VF_RES_BUF_SZ \
+ (sizeof(struct virtchnl_vf_resource) + \
+ IAVF_MAX_VF_VSI * sizeof(struct virtchnl_vsi_resource))
+
+static __rte_always_inline int
+ice_dcf_send_cmd_req_no_irq(struct ice_dcf_hw *hw, enum virtchnl_ops op,
+ uint8_t *req_msg, uint16_t req_msglen)
+{
+ return iavf_aq_send_msg_to_pf(&hw->avf, op, IAVF_SUCCESS,
+ req_msg, req_msglen, NULL);
+}
+
+static int
+ice_dcf_recv_cmd_rsp_no_irq(struct ice_dcf_hw *hw, enum virtchnl_ops op,
+ uint8_t *rsp_msgbuf, uint16_t rsp_buflen,
+ uint16_t *rsp_msglen)
+{
+ struct iavf_arq_event_info event;
+ enum virtchnl_ops v_op;
+ int i = 0;
+ int err;
+
+ event.buf_len = rsp_buflen;
+ event.msg_buf = rsp_msgbuf;
+
+ do {
+ err = iavf_clean_arq_element(&hw->avf, &event, NULL);
+ if (err != IAVF_SUCCESS)
+ goto again;
+
+ v_op = rte_le_to_cpu_32(event.desc.cookie_high);
+ if (v_op != op)
+ goto again;
+
+ if (rsp_msglen != NULL)
+ *rsp_msglen = event.msg_len;
+ return rte_le_to_cpu_32(event.desc.cookie_low);
+
+again:
+ rte_delay_ms(ICE_DCF_ARQ_CHECK_TIME);
+ } while (i++ < ICE_DCF_ARQ_MAX_RETRIES);
+
+ return -EIO;
+}
+
+static __rte_always_inline void
+ice_dcf_aq_cmd_clear(struct ice_dcf_hw *hw, struct dcf_virtchnl_cmd *cmd)
+{
+ rte_spinlock_lock(&hw->vc_cmd_queue_lock);
+
+ TAILQ_REMOVE(&hw->vc_cmd_queue, cmd, next);
+
+ rte_spinlock_unlock(&hw->vc_cmd_queue_lock);
+}
+
+static __rte_always_inline void
+ice_dcf_vc_cmd_set(struct ice_dcf_hw *hw, struct dcf_virtchnl_cmd *cmd)
+{
+ cmd->v_ret = IAVF_ERR_NOT_READY;
+ cmd->rsp_msglen = 0;
+ cmd->pending = 1;
+
+ rte_spinlock_lock(&hw->vc_cmd_queue_lock);
+
+ TAILQ_INSERT_TAIL(&hw->vc_cmd_queue, cmd, next);
+
+ rte_spinlock_unlock(&hw->vc_cmd_queue_lock);
+}
+
+static __rte_always_inline int
+ice_dcf_vc_cmd_send(struct ice_dcf_hw *hw, struct dcf_virtchnl_cmd *cmd)
+{
+ return iavf_aq_send_msg_to_pf(&hw->avf,
+ cmd->v_op, IAVF_SUCCESS,
+ cmd->req_msg, cmd->req_msglen, NULL);
+}
+
+static __rte_always_inline void
+ice_dcf_aq_cmd_handle(struct ice_dcf_hw *hw, struct iavf_arq_event_info *info)
+{
+ struct dcf_virtchnl_cmd *cmd;
+ enum virtchnl_ops v_op;
+ enum iavf_status v_ret;
+ uint16_t aq_op;
+
+ aq_op = rte_le_to_cpu_16(info->desc.opcode);
+ if (unlikely(aq_op != iavf_aqc_opc_send_msg_to_vf)) {
+ PMD_DRV_LOG(ERR,
+ "Request %u is not supported yet", aq_op);
+ return;
+ }
+
+ v_op = rte_le_to_cpu_32(info->desc.cookie_high);
+ if (unlikely(v_op == VIRTCHNL_OP_EVENT)) {
+ if (hw->vc_event_msg_cb != NULL)
+ hw->vc_event_msg_cb(hw,
+ info->msg_buf,
+ info->msg_len);
+ return;
+ }
+
+ v_ret = rte_le_to_cpu_32(info->desc.cookie_low);
+
+ rte_spinlock_lock(&hw->vc_cmd_queue_lock);
+
+ TAILQ_FOREACH(cmd, &hw->vc_cmd_queue, next) {
+ if (cmd->v_op == v_op && cmd->pending) {
+ cmd->v_ret = v_ret;
+ cmd->rsp_msglen = RTE_MIN(info->msg_len,
+ cmd->rsp_buflen);
+ if (likely(cmd->rsp_msglen != 0))
+ rte_memcpy(cmd->rsp_msgbuf, info->msg_buf,
+ cmd->rsp_msglen);
+
+ /* prevent compiler reordering */
+ rte_compiler_barrier();
+ cmd->pending = 0;
+ break;
+ }
+ }
+
+ rte_spinlock_unlock(&hw->vc_cmd_queue_lock);
+}
+
+static void
+ice_dcf_handle_virtchnl_msg(struct ice_dcf_hw *hw)
+{
+ struct iavf_arq_event_info info;
+ uint16_t pending = 1;
+ int ret;
+
+ info.buf_len = ICE_DCF_AQ_BUF_SZ;
+ info.msg_buf = hw->arq_buf;
+
+ while (pending) {
+ ret = iavf_clean_arq_element(&hw->avf, &info, &pending);
+ if (ret != IAVF_SUCCESS)
+ break;
+
+ ice_dcf_aq_cmd_handle(hw, &info);
+ }
+}
+
+static int
+ice_dcf_init_check_api_version(struct ice_dcf_hw *hw)
+{
+#define ICE_CPF_VIRTCHNL_VERSION_MAJOR_START 1
+#define ICE_CPF_VIRTCHNL_VERSION_MINOR_START 1
+ struct virtchnl_version_info version, *pver;
+ int err;
+
+ version.major = VIRTCHNL_VERSION_MAJOR;
+ version.minor = VIRTCHNL_VERSION_MINOR;
+ err = ice_dcf_send_cmd_req_no_irq(hw, VIRTCHNL_OP_VERSION,
+ (uint8_t *)&version, sizeof(version));
+ if (err) {
+ PMD_INIT_LOG(ERR, "Fail to send OP_VERSION");
+ return err;
+ }
+
+ pver = &hw->virtchnl_version;
+ err = ice_dcf_recv_cmd_rsp_no_irq(hw, VIRTCHNL_OP_VERSION,
+ (uint8_t *)pver, sizeof(*pver), NULL);
+ if (err) {
+ PMD_INIT_LOG(ERR, "Fail to get response of OP_VERSION");
+ return -1;
+ }
+
+ PMD_DRV_LOG(DEBUG,
+ "Peer PF API version: %u.%u", pver->major, pver->minor);
+
+ if (pver->major < ICE_CPF_VIRTCHNL_VERSION_MAJOR_START ||
+ (pver->major == ICE_CPF_VIRTCHNL_VERSION_MAJOR_START &&
+ pver->minor < ICE_CPF_VIRTCHNL_VERSION_MINOR_START)) {
+ PMD_INIT_LOG(ERR,
+ "VIRTCHNL API version should not be lower than (%u.%u)",
+ ICE_CPF_VIRTCHNL_VERSION_MAJOR_START,
+ ICE_CPF_VIRTCHNL_VERSION_MAJOR_START);
+ return -1;
+ } else if (pver->major > VIRTCHNL_VERSION_MAJOR ||
+ (pver->major == VIRTCHNL_VERSION_MAJOR &&
+ pver->minor > VIRTCHNL_VERSION_MINOR)) {
+ PMD_INIT_LOG(ERR,
+ "PF/VF API version mismatch:(%u.%u)-(%u.%u)",
+ pver->major, pver->minor,
+ VIRTCHNL_VERSION_MAJOR, VIRTCHNL_VERSION_MINOR);
+ return -1;
+ }
+
+ PMD_DRV_LOG(DEBUG, "Peer is supported PF host");
+
+ return 0;
+}
+
+static int
+ice_dcf_get_vf_resource(struct ice_dcf_hw *hw)
+{
+ uint32_t caps;
+ int err, i;
+
+ caps = VIRTCHNL_VF_OFFLOAD_WB_ON_ITR | VIRTCHNL_VF_OFFLOAD_RX_POLLING |
+ VIRTCHNL_VF_CAP_ADV_LINK_SPEED | VIRTCHNL_VF_CAP_DCF |
+ VF_BASE_MODE_OFFLOADS;
+
+ err = ice_dcf_send_cmd_req_no_irq(hw, VIRTCHNL_OP_GET_VF_RESOURCES,
+ (uint8_t *)&caps, sizeof(caps));
+ if (err) {
+ PMD_DRV_LOG(ERR, "Fail to send msg OP_GET_VF_RESOURCE");
+ return err;
+ }
+
+ err = ice_dcf_recv_cmd_rsp_no_irq(hw, VIRTCHNL_OP_GET_VF_RESOURCES,
+ (uint8_t *)hw->vf_res,
+ ICE_DCF_VF_RES_BUF_SZ, NULL);
+ if (err) {
+ PMD_DRV_LOG(ERR, "Fail to get response of OP_GET_VF_RESOURCE");
+ return -1;
+ }
+
+ iavf_vf_parse_hw_config(&hw->avf, hw->vf_res);
+
+ hw->vsi_res = NULL;
+ for (i = 0; i < hw->vf_res->num_vsis; i++) {
+ if (hw->vf_res->vsi_res[i].vsi_type == VIRTCHNL_VSI_SRIOV)
+ hw->vsi_res = &hw->vf_res->vsi_res[i];
+ }
+
+ if (!hw->vsi_res) {
+ PMD_DRV_LOG(ERR, "no LAN VSI found");
+ return -1;
+ }
+
+ hw->vsi_id = hw->vsi_res->vsi_id;
+ PMD_DRV_LOG(DEBUG, "VSI ID is %u", hw->vsi_id);
+
+ return 0;
+}
+
+static int
+ice_dcf_get_vf_vsi_map(struct ice_dcf_hw *hw)
+{
+ struct virtchnl_dcf_vsi_map *vsi_map;
+ uint16_t len;
+ int err;
+
+ err = ice_dcf_send_cmd_req_no_irq(hw, VIRTCHNL_OP_DCF_GET_VSI_MAP,
+ NULL, 0);
+ if (err) {
+ PMD_DRV_LOG(ERR, "Fail to send msg OP_DCF_GET_VSI_MAP");
+ return err;
+ }
+
+ err = ice_dcf_recv_cmd_rsp_no_irq(hw, VIRTCHNL_OP_DCF_GET_VSI_MAP,
+ hw->arq_buf, ICE_DCF_AQ_BUF_SZ,
+ &len);
+ if (err) {
+ PMD_DRV_LOG(ERR, "Fail to get response of OP_DCF_GET_VSI_MAP");
+ return err;
+ }
+
+ vsi_map = (struct virtchnl_dcf_vsi_map *)hw->arq_buf;
+ if (len < sizeof(*vsi_map) || !vsi_map->num_vfs ||
+ len < sizeof(*vsi_map) +
+ (vsi_map->num_vfs - 1) * sizeof(vsi_map->vf_vsi[0])) {
+ PMD_DRV_LOG(ERR, "invalid vf vsi map response with length %u",
+ len);
+ return -EINVAL;
+ }
+
+ if (hw->num_vfs != 0 && hw->num_vfs != vsi_map->num_vfs) {
+ PMD_DRV_LOG(ERR, "The number VSI map (%u) doesn't match the number of VFs (%u)",
+ vsi_map->num_vfs, hw->num_vfs);
+ return -EINVAL;
+ }
+
+ len = vsi_map->num_vfs * sizeof(vsi_map->vf_vsi[0]);
+ if (!hw->vf_vsi_map) {
+ hw->num_vfs = vsi_map->num_vfs;
+ hw->vf_vsi_map = rte_zmalloc("vf_vsi_ctx", len, 0);
+ }
+
+ if (!hw->vf_vsi_map) {
+ PMD_DRV_LOG(ERR, "Fail to alloc memory for VSI context");
+ return -ENOMEM;
+ }
+
+ if (!memcmp(hw->vf_vsi_map, vsi_map->vf_vsi, len)) {
+ PMD_DRV_LOG(DEBUG, "VF VSI map doesn't change");
+ return 1;
+ }
+
+ rte_memcpy(hw->vf_vsi_map, vsi_map->vf_vsi, len);
+ return 0;
+}
+
+static int
+ice_dcf_mode_disable(struct ice_dcf_hw *hw)
+{
+ int err;
+
+ err = ice_dcf_send_cmd_req_no_irq(hw, VIRTCHNL_OP_DCF_DISABLE,
+ NULL, 0);
+ if (err) {
+ PMD_DRV_LOG(ERR, "Fail to send msg OP_DCF_DISABLE");
+ return err;
+ }
+
+ err = ice_dcf_recv_cmd_rsp_no_irq(hw, VIRTCHNL_OP_DCF_DISABLE,
+ (uint8_t *)hw->arq_buf,
+ ICE_DCF_AQ_BUF_SZ, NULL);
+ if (err) {
+ PMD_DRV_LOG(ERR,
+ "Fail to get response of OP_DCF_DISABLE %d",
+ err);
+ return -1;
+ }
+
+ return 0;
+}
+
+static int
+ice_dcf_check_reset_done(struct ice_dcf_hw *hw)
+{
+#define ICE_DCF_RESET_WAIT_CNT 50
+ struct iavf_hw *avf = &hw->avf;
+ int i, reset;
+
+ for (i = 0; i < ICE_DCF_RESET_WAIT_CNT; i++) {
+ reset = IAVF_READ_REG(avf, IAVF_VFGEN_RSTAT) &
+ IAVF_VFGEN_RSTAT_VFR_STATE_MASK;
+ reset = reset >> IAVF_VFGEN_RSTAT_VFR_STATE_SHIFT;
+
+ if (reset == VIRTCHNL_VFR_VFACTIVE ||
+ reset == VIRTCHNL_VFR_COMPLETED)
+ break;
+
+ rte_delay_ms(20);
+ }
+
+ if (i >= ICE_DCF_RESET_WAIT_CNT)
+ return -1;
+
+ return 0;
+}
+
+static inline void
+ice_dcf_enable_irq0(struct ice_dcf_hw *hw)
+{
+ struct iavf_hw *avf = &hw->avf;
+
+ /* Enable admin queue interrupt trigger */
+ IAVF_WRITE_REG(avf, IAVF_VFINT_ICR0_ENA1,
+ IAVF_VFINT_ICR0_ENA1_ADMINQ_MASK);
+ IAVF_WRITE_REG(avf, IAVF_VFINT_DYN_CTL01,
+ IAVF_VFINT_DYN_CTL01_INTENA_MASK |
+ IAVF_VFINT_DYN_CTL01_CLEARPBA_MASK |
+ IAVF_VFINT_DYN_CTL01_ITR_INDX_MASK);
+
+ IAVF_WRITE_FLUSH(avf);
+}
+
+static inline void
+ice_dcf_disable_irq0(struct ice_dcf_hw *hw)
+{
+ struct iavf_hw *avf = &hw->avf;
+
+ /* Disable all interrupt types */
+ IAVF_WRITE_REG(avf, IAVF_VFINT_ICR0_ENA1, 0);
+ IAVF_WRITE_REG(avf, IAVF_VFINT_DYN_CTL01,
+ IAVF_VFINT_DYN_CTL01_ITR_INDX_MASK);
+
+ IAVF_WRITE_FLUSH(avf);
+}
+
+static void
+ice_dcf_dev_interrupt_handler(void *param)
+{
+ struct ice_dcf_hw *hw = param;
+
+ ice_dcf_disable_irq0(hw);
+
+ ice_dcf_handle_virtchnl_msg(hw);
+
+ ice_dcf_enable_irq0(hw);
+}
+
+int
+ice_dcf_execute_virtchnl_cmd(struct ice_dcf_hw *hw,
+ struct dcf_virtchnl_cmd *cmd)
+{
+ int i = 0;
+ int err;
+
+ if ((cmd->req_msg && !cmd->req_msglen) ||
+ (!cmd->req_msg && cmd->req_msglen) ||
+ (cmd->rsp_msgbuf && !cmd->rsp_buflen) ||
+ (!cmd->rsp_msgbuf && cmd->rsp_buflen))
+ return -EINVAL;
+
+ rte_spinlock_lock(&hw->vc_cmd_send_lock);
+ ice_dcf_vc_cmd_set(hw, cmd);
+
+ err = ice_dcf_vc_cmd_send(hw, cmd);
+ if (err) {
+ PMD_DRV_LOG(ERR, "fail to send cmd %d", cmd->v_op);
+ goto ret;
+ }
+
+ do {
+ if (!cmd->pending)
+ break;
+
+ rte_delay_ms(ICE_DCF_ARQ_CHECK_TIME);
+ } while (i++ < ICE_DCF_ARQ_MAX_RETRIES);
+
+ if (cmd->v_ret != IAVF_SUCCESS) {
+ err = -1;
+ PMD_DRV_LOG(ERR,
+ "No response (%d times) or return failure (%d) for cmd %d",
+ i, cmd->v_ret, cmd->v_op);
+ }
+
+ret:
+ ice_dcf_aq_cmd_clear(hw, cmd);
+ rte_spinlock_unlock(&hw->vc_cmd_send_lock);
+ return err;
+}
+
+int
+ice_dcf_send_aq_cmd(void *dcf_hw, struct ice_aq_desc *desc,
+ void *buf, uint16_t buf_size)
+{
+ struct dcf_virtchnl_cmd desc_cmd, buff_cmd;
+ struct ice_dcf_hw *hw = dcf_hw;
+ int err = 0;
+ int i = 0;
+
+ if ((buf && !buf_size) || (!buf && buf_size) ||
+ buf_size > ICE_DCF_AQ_BUF_SZ)
+ return -EINVAL;
+
+ desc_cmd.v_op = VIRTCHNL_OP_DCF_CMD_DESC;
+ desc_cmd.req_msglen = sizeof(*desc);
+ desc_cmd.req_msg = (uint8_t *)desc;
+ desc_cmd.rsp_buflen = sizeof(*desc);
+ desc_cmd.rsp_msgbuf = (uint8_t *)desc;
+
+ if (buf == NULL)
+ return ice_dcf_execute_virtchnl_cmd(hw, &desc_cmd);
+
+ desc->flags |= rte_cpu_to_le_16(ICE_AQ_FLAG_BUF);
+
+ buff_cmd.v_op = VIRTCHNL_OP_DCF_CMD_BUFF;
+ buff_cmd.req_msglen = buf_size;
+ buff_cmd.req_msg = buf;
+ buff_cmd.rsp_buflen = buf_size;
+ buff_cmd.rsp_msgbuf = buf;
+
+ rte_spinlock_lock(&hw->vc_cmd_send_lock);
+ ice_dcf_vc_cmd_set(hw, &desc_cmd);
+ ice_dcf_vc_cmd_set(hw, &buff_cmd);
+
+ if (ice_dcf_vc_cmd_send(hw, &desc_cmd) ||
+ ice_dcf_vc_cmd_send(hw, &buff_cmd)) {
+ err = -1;
+ PMD_DRV_LOG(ERR, "fail to send OP_DCF_CMD_DESC/BUFF");
+ goto ret;
+ }
+
+ do {
+ if ((!desc_cmd.pending && !buff_cmd.pending) ||
+ (!desc_cmd.pending && desc_cmd.v_ret != IAVF_SUCCESS) ||
+ (!buff_cmd.pending && buff_cmd.v_ret != IAVF_SUCCESS))
+ break;
+
+ rte_delay_ms(ICE_DCF_ARQ_CHECK_TIME);
+ } while (i++ < ICE_DCF_ARQ_MAX_RETRIES);
+
+ if (desc_cmd.v_ret != IAVF_SUCCESS || buff_cmd.v_ret != IAVF_SUCCESS) {
+ err = -1;
+ PMD_DRV_LOG(ERR,
+ "No response (%d times) or return failure (desc: %d / buff: %d)",
+ i, desc_cmd.v_ret, buff_cmd.v_ret);
+ }
+
+ret:
+ ice_dcf_aq_cmd_clear(hw, &desc_cmd);
+ ice_dcf_aq_cmd_clear(hw, &buff_cmd);
+ rte_spinlock_unlock(&hw->vc_cmd_send_lock);
+
+ return err;
+}
+
+int
+ice_dcf_handle_vsi_update_event(struct ice_dcf_hw *hw)
+{
+ int err = 0;
+
+ rte_spinlock_lock(&hw->vc_cmd_send_lock);
+ ice_dcf_disable_irq0(hw);
+
+ if (ice_dcf_get_vf_resource(hw) || ice_dcf_get_vf_vsi_map(hw))
+ err = -1;
+
+ ice_dcf_enable_irq0(hw);
+ rte_spinlock_unlock(&hw->vc_cmd_send_lock);
+
+ return err;
+}
+
+int
+ice_dcf_init_hw(struct rte_eth_dev *eth_dev, struct ice_dcf_hw *hw)
+{
+ struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ int ret;
+
+ hw->avf.hw_addr = pci_dev->mem_resource[0].addr;
+ hw->avf.back = hw;
+
+ hw->avf.bus.bus_id = pci_dev->addr.bus;
+ hw->avf.bus.device = pci_dev->addr.devid;
+ hw->avf.bus.func = pci_dev->addr.function;
+
+ hw->avf.device_id = pci_dev->id.device_id;
+ hw->avf.vendor_id = pci_dev->id.vendor_id;
+ hw->avf.subsystem_device_id = pci_dev->id.subsystem_device_id;
+ hw->avf.subsystem_vendor_id = pci_dev->id.subsystem_vendor_id;
+
+ hw->avf.aq.num_arq_entries = ICE_DCF_AQ_LEN;
+ hw->avf.aq.num_asq_entries = ICE_DCF_AQ_LEN;
+ hw->avf.aq.arq_buf_size = ICE_DCF_AQ_BUF_SZ;
+ hw->avf.aq.asq_buf_size = ICE_DCF_AQ_BUF_SZ;
+
+ rte_spinlock_init(&hw->vc_cmd_send_lock);
+ rte_spinlock_init(&hw->vc_cmd_queue_lock);
+ TAILQ_INIT(&hw->vc_cmd_queue);
+
+ hw->arq_buf = rte_zmalloc("arq_buf", ICE_DCF_AQ_BUF_SZ, 0);
+ if (hw->arq_buf == NULL) {
+ PMD_INIT_LOG(ERR, "unable to allocate AdminQ buffer memory");
+ goto err;
+ }
+
+ ret = iavf_set_mac_type(&hw->avf);
+ if (ret) {
+ PMD_INIT_LOG(ERR, "set_mac_type failed: %d", ret);
+ goto err;
+ }
+
+ ret = ice_dcf_check_reset_done(hw);
+ if (ret) {
+ PMD_INIT_LOG(ERR, "VF is still resetting");
+ goto err;
+ }
+
+ ret = iavf_init_adminq(&hw->avf);
+ if (ret) {
+ PMD_INIT_LOG(ERR, "init_adminq failed: %d", ret);
+ goto err;
+ }
+
+ if (ice_dcf_init_check_api_version(hw)) {
+ PMD_INIT_LOG(ERR, "check_api version failed");
+ goto err_api;
+ }
+
+ hw->vf_res = rte_zmalloc("vf_res", ICE_DCF_VF_RES_BUF_SZ, 0);
+ if (hw->vf_res == NULL) {
+ PMD_INIT_LOG(ERR, "unable to allocate vf_res memory");
+ goto err_api;
+ }
+
+ if (ice_dcf_get_vf_resource(hw)) {
+ PMD_INIT_LOG(ERR, "Failed to get VF resource");
+ goto err_alloc;
+ }
+
+ if (ice_dcf_get_vf_vsi_map(hw) < 0) {
+ PMD_INIT_LOG(ERR, "Failed to get VF VSI map");
+ ice_dcf_mode_disable(hw);
+ goto err_alloc;
+ }
+
+ rte_intr_callback_register(&pci_dev->intr_handle,
+ ice_dcf_dev_interrupt_handler, hw);
+ rte_intr_enable(&pci_dev->intr_handle);
+ ice_dcf_enable_irq0(hw);
+
+ return 0;
+
+err_alloc:
+ rte_free(hw->vf_res);
+err_api:
+ iavf_shutdown_adminq(&hw->avf);
+err:
+ rte_free(hw->arq_buf);
+
+ return -1;
+}
+
+void
+ice_dcf_uninit_hw(struct rte_eth_dev *eth_dev, struct ice_dcf_hw *hw)
+{
+ struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_intr_handle *intr_handle = &pci_dev->intr_handle;
+
+ ice_dcf_disable_irq0(hw);
+ rte_intr_disable(intr_handle);
+ rte_intr_callback_unregister(intr_handle,
+ ice_dcf_dev_interrupt_handler, hw);
+
+ ice_dcf_mode_disable(hw);
+ iavf_shutdown_adminq(&hw->avf);
+
+ rte_free(hw->arq_buf);
+ rte_free(hw->vf_vsi_map);
+ rte_free(hw->vf_res);
+}
diff --git a/drivers/net/ice/ice_dcf.h b/drivers/net/ice/ice_dcf.h
new file mode 100644
index 000000000..58647d87f
--- /dev/null
+++ b/drivers/net/ice/ice_dcf.h
@@ -0,0 +1,61 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+
+#ifndef _ICE_DCF_H_
+#define _ICE_DCF_H_
+
+#include <rte_ethdev_driver.h>
+
+#include <iavf_prototype.h>
+#include <iavf_adminq_cmd.h>
+#include <iavf_type.h>
+
+#include "base/ice_type.h"
+#include "ice_logs.h"
+
+struct dcf_virtchnl_cmd {
+ TAILQ_ENTRY(dcf_virtchnl_cmd) next;
+
+ enum virtchnl_ops v_op;
+ enum iavf_status v_ret;
+
+ uint16_t req_msglen;
+ uint8_t *req_msg;
+
+ uint16_t rsp_msglen;
+ uint16_t rsp_buflen;
+ uint8_t *rsp_msgbuf;
+
+ volatile int pending;
+};
+
+struct ice_dcf_hw {
+ struct iavf_hw avf;
+
+ rte_spinlock_t vc_cmd_send_lock;
+ rte_spinlock_t vc_cmd_queue_lock;
+ TAILQ_HEAD(, dcf_virtchnl_cmd) vc_cmd_queue;
+ void (*vc_event_msg_cb)(struct ice_dcf_hw *dcf_hw,
+ uint8_t *msg, uint16_t msglen);
+
+ uint8_t *arq_buf;
+
+ uint16_t num_vfs;
+ uint16_t *vf_vsi_map;
+
+ struct virtchnl_version_info virtchnl_version;
+ struct virtchnl_vf_resource *vf_res; /* VF resource */
+ struct virtchnl_vsi_resource *vsi_res; /* LAN VSI */
+ uint16_t vsi_id;
+};
+
+int ice_dcf_execute_virtchnl_cmd(struct ice_dcf_hw *hw,
+ struct dcf_virtchnl_cmd *cmd);
+int ice_dcf_send_aq_cmd(void *dcf_hw, struct ice_aq_desc *desc,
+ void *buf, uint16_t buf_size);
+int ice_dcf_handle_vsi_update_event(struct ice_dcf_hw *hw);
+int ice_dcf_init_hw(struct rte_eth_dev *eth_dev, struct ice_dcf_hw *hw);
+void ice_dcf_uninit_hw(struct rte_eth_dev *eth_dev, struct ice_dcf_hw *hw);
+
+#endif /* _ICE_DCF_H_ */
diff --git a/drivers/net/ice/ice_dcf_ethdev.c b/drivers/net/ice/ice_dcf_ethdev.c
new file mode 100644
index 000000000..f65b962d4
--- /dev/null
+++ b/drivers/net/ice/ice_dcf_ethdev.c
@@ -0,0 +1,319 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+
+#include <errno.h>
+#include <stdbool.h>
+#include <sys/types.h>
+#include <sys/ioctl.h>
+#include <unistd.h>
+
+#include <rte_interrupts.h>
+#include <rte_debug.h>
+#include <rte_pci.h>
+#include <rte_atomic.h>
+#include <rte_eal.h>
+#include <rte_ether.h>
+#include <rte_ethdev_pci.h>
+#include <rte_kvargs.h>
+#include <rte_malloc.h>
+#include <rte_memzone.h>
+#include <rte_dev.h>
+
+#include <iavf_devids.h>
+
+#include "ice_generic_flow.h"
+#include "ice_dcf_ethdev.h"
+
+static uint16_t
+ice_dcf_recv_pkts(__rte_unused void *rx_queue,
+ __rte_unused struct rte_mbuf **bufs,
+ __rte_unused uint16_t nb_pkts)
+{
+ return 0;
+}
+
+static uint16_t
+ice_dcf_xmit_pkts(__rte_unused void *tx_queue,
+ __rte_unused struct rte_mbuf **bufs,
+ __rte_unused uint16_t nb_pkts)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_start(struct rte_eth_dev *dev)
+{
+ dev->data->dev_link.link_status = ETH_LINK_UP;
+
+ return 0;
+}
+
+static void
+ice_dcf_dev_stop(struct rte_eth_dev *dev)
+{
+ dev->data->dev_link.link_status = ETH_LINK_DOWN;
+}
+
+static int
+ice_dcf_dev_configure(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_info_get(struct rte_eth_dev *dev,
+ struct rte_eth_dev_info *dev_info)
+{
+ struct ice_dcf_adapter *adapter = dev->data->dev_private;
+
+ dev_info->max_mac_addrs = 1;
+ dev_info->max_rx_pktlen = (uint32_t)-1;
+ dev_info->max_rx_queues = RTE_DIM(adapter->rxqs);
+ dev_info->max_tx_queues = RTE_DIM(adapter->txqs);
+
+ return 0;
+}
+
+static int
+ice_dcf_stats_get(__rte_unused struct rte_eth_dev *dev,
+ __rte_unused struct rte_eth_stats *igb_stats)
+{
+ return 0;
+}
+
+static int
+ice_dcf_stats_reset(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_promiscuous_enable(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_promiscuous_disable(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_allmulticast_enable(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_allmulticast_disable(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_filter_ctrl(struct rte_eth_dev *dev,
+ enum rte_filter_type filter_type,
+ __rte_unused enum rte_filter_op filter_op,
+ __rte_unused void *arg)
+{
+ int ret = 0;
+
+ if (!dev)
+ return -EINVAL;
+
+ switch (filter_type) {
+ default:
+ PMD_DRV_LOG(WARNING, "Filter type (%d) not supported",
+ filter_type);
+ ret = -EINVAL;
+ break;
+ }
+
+ return ret;
+}
+
+static void
+ice_dcf_dev_close(struct rte_eth_dev *dev)
+{
+ struct ice_dcf_adapter *adapter = dev->data->dev_private;
+
+ if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+ return;
+
+ dev->dev_ops = NULL;
+ dev->rx_pkt_burst = NULL;
+ dev->tx_pkt_burst = NULL;
+
+ ice_dcf_uninit_parent_adapter(dev);
+ ice_dcf_uninit_hw(dev, &adapter->real_hw);
+}
+
+static void
+ice_dcf_queue_release(__rte_unused void *q)
+{
+}
+
+static int
+ice_dcf_link_update(__rte_unused struct rte_eth_dev *dev,
+ __rte_unused int wait_to_complete)
+{
+ return 0;
+}
+
+static int
+ice_dcf_rx_queue_setup(struct rte_eth_dev *dev,
+ uint16_t rx_queue_id,
+ __rte_unused uint16_t nb_rx_desc,
+ __rte_unused unsigned int socket_id,
+ __rte_unused const struct rte_eth_rxconf *rx_conf,
+ __rte_unused struct rte_mempool *mb_pool)
+{
+ struct ice_dcf_adapter *adapter = dev->data->dev_private;
+
+ dev->data->rx_queues[rx_queue_id] = &adapter->rxqs[rx_queue_id];
+
+ return 0;
+}
+
+static int
+ice_dcf_tx_queue_setup(struct rte_eth_dev *dev,
+ uint16_t tx_queue_id,
+ __rte_unused uint16_t nb_tx_desc,
+ __rte_unused unsigned int socket_id,
+ __rte_unused const struct rte_eth_txconf *tx_conf)
+{
+ struct ice_dcf_adapter *adapter = dev->data->dev_private;
+
+ dev->data->tx_queues[tx_queue_id] = &adapter->txqs[tx_queue_id];
+
+ return 0;
+}
+
+static const struct eth_dev_ops ice_dcf_eth_dev_ops = {
+ .dev_start = ice_dcf_dev_start,
+ .dev_stop = ice_dcf_dev_stop,
+ .dev_close = ice_dcf_dev_close,
+ .dev_configure = ice_dcf_dev_configure,
+ .dev_infos_get = ice_dcf_dev_info_get,
+ .rx_queue_setup = ice_dcf_rx_queue_setup,
+ .tx_queue_setup = ice_dcf_tx_queue_setup,
+ .rx_queue_release = ice_dcf_queue_release,
+ .tx_queue_release = ice_dcf_queue_release,
+ .link_update = ice_dcf_link_update,
+ .stats_get = ice_dcf_stats_get,
+ .stats_reset = ice_dcf_stats_reset,
+ .promiscuous_enable = ice_dcf_dev_promiscuous_enable,
+ .promiscuous_disable = ice_dcf_dev_promiscuous_disable,
+ .allmulticast_enable = ice_dcf_dev_allmulticast_enable,
+ .allmulticast_disable = ice_dcf_dev_allmulticast_disable,
+ .filter_ctrl = ice_dcf_dev_filter_ctrl,
+};
+
+static int
+ice_dcf_dev_init(struct rte_eth_dev *eth_dev)
+{
+ struct ice_dcf_adapter *adapter = eth_dev->data->dev_private;
+
+ eth_dev->dev_ops = &ice_dcf_eth_dev_ops;
+ eth_dev->rx_pkt_burst = ice_dcf_recv_pkts;
+ eth_dev->tx_pkt_burst = ice_dcf_xmit_pkts;
+
+ if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+ return 0;
+
+ eth_dev->data->dev_flags |= RTE_ETH_DEV_CLOSE_REMOVE;
+
+ adapter->real_hw.vc_event_msg_cb = ice_dcf_handle_pf_event_msg;
+ if (ice_dcf_init_hw(eth_dev, &adapter->real_hw) != 0) {
+ PMD_DRV_LOG(ERR, "Failed to init DCF hardware");
+ return -1;
+ }
+
+ if (ice_dcf_init_parent_adapter(eth_dev) != 0) {
+ PMD_DRV_LOG(ERR, "Failed to init DCF parent adapter");
+ ice_dcf_uninit_hw(eth_dev, &adapter->real_hw);
+ return -1;
+ }
+
+ return 0;
+}
+
+static int
+ice_dcf_dev_uninit(struct rte_eth_dev *eth_dev)
+{
+ ice_dcf_dev_close(eth_dev);
+
+ return 0;
+}
+
+static int
+handle_dcf_arg(__rte_unused const char *key, const char *value,
+ __rte_unused void *arg)
+{
+ bool *dcf = arg;
+
+ if (arg == NULL || value == NULL)
+ return -EINVAL;
+
+ if (strcmp(value, "dcf") == 0)
+ *dcf = true;
+ else
+ *dcf = false;
+
+ return 0;
+}
+
+static bool
+check_cap_dcf_enable(struct rte_devargs *devargs)
+{
+ struct rte_kvargs *kvlist;
+ bool enable = false;
+
+ if (devargs == NULL)
+ return false;
+
+ kvlist = rte_kvargs_parse(devargs->args, NULL);
+ if (kvlist == NULL)
+ return false;
+
+ rte_kvargs_process(kvlist, "cap", handle_dcf_arg, &enable);
+
+ rte_kvargs_free(kvlist);
+
+ return enable;
+}
+
+static int eth_ice_dcf_pci_probe(__rte_unused struct rte_pci_driver *pci_drv,
+ struct rte_pci_device *pci_dev)
+{
+ if (!check_cap_dcf_enable(pci_dev->device.devargs))
+ return 1; /* continue to probe */
+
+ return rte_eth_dev_pci_generic_probe(pci_dev,
+ sizeof(struct ice_dcf_adapter),
+ ice_dcf_dev_init);
+}
+
+static int eth_ice_dcf_pci_remove(struct rte_pci_device *pci_dev)
+{
+ return rte_eth_dev_pci_generic_remove(pci_dev, ice_dcf_dev_uninit);
+}
+
+static const struct rte_pci_id pci_id_ice_dcf_map[] = {
+ { RTE_PCI_DEVICE(IAVF_INTEL_VENDOR_ID, IAVF_DEV_ID_ADAPTIVE_VF) },
+ { .vendor_id = 0, /* sentinel */ },
+};
+
+static struct rte_pci_driver rte_ice_dcf_pmd = {
+ .id_table = pci_id_ice_dcf_map,
+ .drv_flags = RTE_PCI_DRV_NEED_MAPPING,
+ .probe = eth_ice_dcf_pci_probe,
+ .remove = eth_ice_dcf_pci_remove,
+};
+
+RTE_PMD_REGISTER_PCI(net_ice_dcf, rte_ice_dcf_pmd);
+RTE_PMD_REGISTER_PCI_TABLE(net_ice_dcf, pci_id_ice_dcf_map);
+RTE_PMD_REGISTER_KMOD_DEP(net_ice_dcf, "* igb_uio | vfio-pci");
+RTE_PMD_REGISTER_PARAM_STRING(net_ice_dcf, "cap=dcf");
diff --git a/drivers/net/ice/ice_dcf_ethdev.h b/drivers/net/ice/ice_dcf_ethdev.h
new file mode 100644
index 000000000..e95266599
--- /dev/null
+++ b/drivers/net/ice/ice_dcf_ethdev.h
@@ -0,0 +1,33 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+
+#ifndef _ICE_DCF_ETHDEV_H_
+#define _ICE_DCF_ETHDEV_H_
+
+#include "base/ice_common.h"
+#include "base/ice_adminq_cmd.h"
+
+#include "ice_ethdev.h"
+#include "ice_dcf.h"
+
+#define ICE_DCF_MAX_RINGS 1
+
+struct ice_dcf_queue {
+ uint64_t dummy;
+};
+
+struct ice_dcf_adapter {
+ struct ice_adapter parent; /* Must be first */
+
+ struct ice_dcf_hw real_hw;
+ struct ice_dcf_queue rxqs[ICE_DCF_MAX_RINGS];
+ struct ice_dcf_queue txqs[ICE_DCF_MAX_RINGS];
+};
+
+void ice_dcf_handle_pf_event_msg(struct ice_dcf_hw *dcf_hw,
+ uint8_t *msg, uint16_t msglen);
+int ice_dcf_init_parent_adapter(struct rte_eth_dev *eth_dev);
+void ice_dcf_uninit_parent_adapter(struct rte_eth_dev *eth_dev);
+
+#endif /* _ICE_DCF_ETHDEV_H_ */
diff --git a/drivers/net/ice/ice_dcf_parent.c b/drivers/net/ice/ice_dcf_parent.c
new file mode 100644
index 000000000..bca9cd34a
--- /dev/null
+++ b/drivers/net/ice/ice_dcf_parent.c
@@ -0,0 +1,348 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <unistd.h>
+
+#include <rte_alarm.h>
+
+#include "ice_dcf_ethdev.h"
+#include "ice_generic_flow.h"
+
+#define ICE_DCF_VSI_UPDATE_SERVICE_INTERVAL 100000 /* us */
+
+static __rte_always_inline void
+ice_dcf_update_vsi_ctx(struct ice_hw *hw, uint16_t vsi_handle,
+ uint16_t vsi_map)
+{
+ struct ice_vsi_ctx *vsi_ctx;
+
+ if (unlikely(vsi_handle >= ICE_MAX_VSI)) {
+ PMD_DRV_LOG(ERR, "Invalid vsi handle %u", vsi_handle);
+ return;
+ }
+
+ vsi_ctx = hw->vsi_ctx[vsi_handle];
+
+ if (vsi_map & VIRTCHNL_DCF_VF_VSI_VALID) {
+ if (!vsi_ctx)
+ vsi_ctx = ice_malloc(hw, sizeof(*vsi_ctx));
+
+ if (!vsi_ctx) {
+ PMD_DRV_LOG(ERR, "No memory for vsi context %u",
+ vsi_handle);
+ return;
+ }
+
+ vsi_ctx->vsi_num = (vsi_map & VIRTCHNL_DCF_VF_VSI_ID_M) >>
+ VIRTCHNL_DCF_VF_VSI_ID_S;
+ hw->vsi_ctx[vsi_handle] = vsi_ctx;
+
+ PMD_DRV_LOG(DEBUG, "VF%u is assigned with vsi number %u",
+ vsi_handle, vsi_ctx->vsi_num);
+ } else {
+ hw->vsi_ctx[vsi_handle] = NULL;
+
+ if (vsi_ctx)
+ ice_free(hw, vsi_ctx);
+
+ PMD_DRV_LOG(NOTICE, "VF%u is disabled", vsi_handle);
+ }
+}
+
+static void
+ice_dcf_update_vf_vsi_map(struct ice_hw *hw, uint16_t num_vfs,
+ uint16_t *vf_vsi_map)
+{
+ uint16_t vf_id;
+
+ for (vf_id = 0; vf_id < num_vfs; vf_id++)
+ ice_dcf_update_vsi_ctx(hw, vf_id, vf_vsi_map[vf_id]);
+}
+
+static void
+ice_dcf_vsi_update_service_handler(void *param)
+{
+ struct ice_dcf_hw *hw = param;
+
+ if (!ice_dcf_handle_vsi_update_event(hw)) {
+ struct ice_dcf_adapter *dcf_ad =
+ container_of(hw, struct ice_dcf_adapter, real_hw);
+
+ ice_dcf_update_vf_vsi_map(&dcf_ad->parent.hw,
+ hw->num_vfs, hw->vf_vsi_map);
+ }
+}
+
+void
+ice_dcf_handle_pf_event_msg(struct ice_dcf_hw *dcf_hw,
+ uint8_t *msg, uint16_t msglen)
+{
+ struct virtchnl_pf_event *pf_msg = (struct virtchnl_pf_event *)msg;
+
+ if (msglen < sizeof(struct virtchnl_pf_event)) {
+ PMD_DRV_LOG(DEBUG, "Invalid event message length : %u", msglen);
+ return;
+ }
+
+ switch (pf_msg->event) {
+ case VIRTCHNL_EVENT_RESET_IMPENDING:
+ PMD_DRV_LOG(DEBUG, "VIRTCHNL_EVENT_RESET_IMPENDING event");
+ rte_eal_alarm_set(ICE_DCF_VSI_UPDATE_SERVICE_INTERVAL * 2,
+ ice_dcf_vsi_update_service_handler, dcf_hw);
+ break;
+ case VIRTCHNL_EVENT_LINK_CHANGE:
+ PMD_DRV_LOG(DEBUG, "VIRTCHNL_EVENT_LINK_CHANGE event");
+ break;
+ case VIRTCHNL_EVENT_PF_DRIVER_CLOSE:
+ PMD_DRV_LOG(DEBUG, "VIRTCHNL_EVENT_PF_DRIVER_CLOSE event");
+ break;
+ case VIRTCHNL_EVENT_DCF_VSI_MAP_UPDATE:
+ PMD_DRV_LOG(DEBUG, "VIRTCHNL_EVENT_DCF_VSI_MAP_UPDATE event : VF%u with VSI num %u",
+ pf_msg->event_data.vf_vsi_map.vf_id,
+ pf_msg->event_data.vf_vsi_map.vsi_id);
+ rte_eal_alarm_set(ICE_DCF_VSI_UPDATE_SERVICE_INTERVAL,
+ ice_dcf_vsi_update_service_handler, dcf_hw);
+ break;
+ default:
+ PMD_DRV_LOG(ERR, "Unknown event received %u", pf_msg->event);
+ break;
+ }
+}
+
+static int
+ice_dcf_init_parent_hw(struct ice_hw *hw)
+{
+ struct ice_aqc_get_phy_caps_data *pcaps;
+ enum ice_status status;
+
+ status = ice_aq_get_fw_ver(hw, NULL);
+ if (status)
+ return status;
+
+ status = ice_get_caps(hw);
+ if (status)
+ return status;
+
+ hw->port_info = (struct ice_port_info *)
+ ice_malloc(hw, sizeof(*hw->port_info));
+ if (!hw->port_info)
+ return ICE_ERR_NO_MEMORY;
+
+ /* set the back pointer to HW */
+ hw->port_info->hw = hw;
+
+ /* Initialize port_info struct with switch configuration data */
+ status = ice_get_initial_sw_cfg(hw);
+ if (status)
+ goto err_unroll_alloc;
+
+ pcaps = (struct ice_aqc_get_phy_caps_data *)
+ ice_malloc(hw, sizeof(*pcaps));
+ if (!pcaps) {
+ status = ICE_ERR_NO_MEMORY;
+ goto err_unroll_alloc;
+ }
+
+ /* Initialize port_info struct with PHY capabilities */
+ status = ice_aq_get_phy_caps(hw->port_info, false,
+ ICE_AQC_REPORT_TOPO_CAP, pcaps, NULL);
+ ice_free(hw, pcaps);
+ if (status)
+ goto err_unroll_alloc;
+
+ /* Initialize port_info struct with link information */
+ status = ice_aq_get_link_info(hw->port_info, false, NULL, NULL);
+ if (status)
+ goto err_unroll_alloc;
+
+ status = ice_init_fltr_mgmt_struct(hw);
+ if (status)
+ goto err_unroll_alloc;
+
+ status = ice_init_hw_tbls(hw);
+ if (status)
+ goto err_unroll_fltr_mgmt_struct;
+
+ PMD_DRV_LOG(INFO,
+ "firmware %d.%d.%d api %d.%d.%d build 0x%08x",
+ hw->fw_maj_ver, hw->fw_min_ver, hw->fw_patch,
+ hw->api_maj_ver, hw->api_min_ver, hw->api_patch,
+ hw->fw_build);
+
+ return ICE_SUCCESS;
+
+err_unroll_fltr_mgmt_struct:
+ ice_cleanup_fltr_mgmt_struct(hw);
+err_unroll_alloc:
+ ice_free(hw, hw->port_info);
+ hw->port_info = NULL;
+
+ return status;
+}
+
+static void ice_dcf_uninit_parent_hw(struct ice_hw *hw)
+{
+ ice_cleanup_fltr_mgmt_struct(hw);
+
+ ice_free_seg(hw);
+ ice_free_hw_tbls(hw);
+
+ if (hw->port_info) {
+ ice_free(hw, hw->port_info);
+ hw->port_info = NULL;
+ }
+
+ ice_clear_all_vsi_ctx(hw);
+}
+
+static int
+ice_dcf_request_pkg_name(struct ice_hw *hw, char *pkg_name)
+{
+ struct ice_dcf_adapter *dcf_adapter =
+ container_of(hw, struct ice_dcf_adapter, parent.hw);
+
+ /* TODO: check with DSN firstly by iAVF */
+ PMD_DRV_LOG(DEBUG,
+ "DCF VSI_ID = %u",
+ dcf_adapter->real_hw.vsi_id);
+
+ snprintf(pkg_name,
+ ICE_MAX_PKG_FILENAME_SIZE, "%s", ICE_PKG_FILE_UPDATES);
+ if (!access(pkg_name, 0))
+ return 0;
+
+ snprintf(pkg_name,
+ ICE_MAX_PKG_FILENAME_SIZE, "%s", ICE_PKG_FILE_DEFAULT);
+ if (!access(pkg_name, 0))
+ return 0;
+
+ return -1;
+}
+
+static int
+ice_dcf_load_pkg(struct ice_hw *hw)
+{
+ char pkg_name[ICE_MAX_PKG_FILENAME_SIZE];
+ uint8_t *pkg_buf;
+ uint32_t buf_len;
+ struct stat st;
+ FILE *fp;
+ int err;
+
+ if (ice_dcf_request_pkg_name(hw, pkg_name)) {
+ PMD_INIT_LOG(ERR, "failed to locate the package file");
+ return -ENOENT;
+ }
+
+ PMD_INIT_LOG(DEBUG, "DDP package name: %s", pkg_name);
+
+ err = stat(pkg_name, &st);
+ if (err) {
+ PMD_INIT_LOG(ERR, "failed to get file status");
+ return err;
+ }
+
+ buf_len = st.st_size;
+ pkg_buf = rte_malloc(NULL, buf_len, 0);
+ if (!pkg_buf) {
+ PMD_INIT_LOG(ERR, "failed to allocate buffer of size %u for package",
+ buf_len);
+ return -1;
+ }
+
+ fp = fopen(pkg_name, "rb");
+ if (!fp) {
+ PMD_INIT_LOG(ERR, "failed to open file: %s", pkg_name);
+ err = -1;
+ goto ret;
+ }
+
+ err = fread(pkg_buf, buf_len, 1, fp);
+ fclose(fp);
+ if (err != 1) {
+ PMD_INIT_LOG(ERR, "failed to read package data");
+ err = -1;
+ goto ret;
+ }
+
+ err = ice_copy_and_init_pkg(hw, pkg_buf, buf_len);
+ if (err)
+ PMD_INIT_LOG(ERR, "ice_copy_and_init_hw failed: %d", err);
+
+ret:
+ rte_free(pkg_buf);
+ return err;
+}
+
+int
+ice_dcf_init_parent_adapter(struct rte_eth_dev *eth_dev)
+{
+ struct ice_dcf_adapter *adapter = eth_dev->data->dev_private;
+ struct ice_adapter *parent_adapter = &adapter->parent;
+ struct ice_hw *parent_hw = &parent_adapter->hw;
+ struct ice_dcf_hw *hw = &adapter->real_hw;
+ const struct rte_ether_addr *mac;
+ int err;
+
+ parent_adapter->eth_dev = eth_dev;
+ parent_adapter->pf.adapter = parent_adapter;
+ parent_adapter->pf.dev_data = eth_dev->data;
+ parent_hw->back = parent_adapter;
+ parent_hw->mac_type = ICE_MAC_GENERIC;
+ parent_hw->vendor_id = ICE_INTEL_VENDOR_ID;
+
+ ice_init_lock(&parent_hw->adminq.sq_lock);
+ ice_init_lock(&parent_hw->adminq.rq_lock);
+ parent_hw->aq_send_cmd_fn = ice_dcf_send_aq_cmd;
+ parent_hw->aq_send_cmd_param = &adapter->real_hw;
+ parent_hw->dcf_enabled = true;
+
+ err = ice_dcf_init_parent_hw(parent_hw);
+ if (err) {
+ PMD_DRV_LOG(ERR, "failed to init the DCF parent hardware with error %d",
+ err);
+ return err;
+ }
+
+ err = ice_dcf_load_pkg(parent_hw);
+ if (err) {
+ PMD_DRV_LOG(ERR, "failed to load package with error %d",
+ err);
+ goto uninit_hw;
+ }
+ parent_adapter->active_pkg_type = ice_load_pkg_type(parent_hw);
+
+ ice_dcf_update_vf_vsi_map(parent_hw,
+ hw->num_vfs, hw->vf_vsi_map);
+
+ mac = (const struct rte_ether_addr *)hw->avf.mac.addr;
+ if (rte_is_valid_assigned_ether_addr(mac))
+ rte_ether_addr_copy(mac, &parent_adapter->pf.dev_addr);
+ else
+ rte_eth_random_addr(parent_adapter->pf.dev_addr.addr_bytes);
+
+ eth_dev->data->mac_addrs = &parent_adapter->pf.dev_addr;
+
+ return 0;
+
+uninit_hw:
+ ice_dcf_uninit_parent_hw(parent_hw);
+ return err;
+}
+
+void
+ice_dcf_uninit_parent_adapter(struct rte_eth_dev *eth_dev)
+{
+ struct ice_dcf_adapter *adapter = eth_dev->data->dev_private;
+ struct ice_adapter *parent_adapter = &adapter->parent;
+ struct ice_hw *parent_hw = &parent_adapter->hw;
+
+ eth_dev->data->mac_addrs = NULL;
+
+ rte_eal_alarm_cancel(ice_dcf_vsi_update_service_handler,
+ &adapter->real_hw);
+
+ ice_dcf_uninit_parent_hw(parent_hw);
+}
diff --git a/drivers/net/ice/meson.build b/drivers/net/ice/meson.build
index f9e897bbc..0df8ddc0f 100644
--- a/drivers/net/ice/meson.build
+++ b/drivers/net/ice/meson.build
@@ -15,8 +15,8 @@ sources = files(
'ice_hash.c'
)
-deps += ['hash']
-includes += include_directories('base')
+deps += ['hash', 'net', 'common_iavf']
+includes += include_directories('base', '../../common/iavf')
if arch_subdir == 'x86'
sources += files('ice_rxtx_vec_sse.c')
@@ -37,4 +37,8 @@ if arch_subdir == 'x86'
endif
endif
+sources += files('ice_dcf.c',
+ 'ice_dcf_parent.c',
+ 'ice_dcf_ethdev.c')
+
install_headers('rte_pmd_ice.h')
diff --git a/mk/rte.app.mk b/mk/rte.app.mk
index d295ca0a5..f3798a09f 100644
--- a/mk/rte.app.mk
+++ b/mk/rte.app.mk
@@ -185,6 +185,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_I40E_PMD) += -lrte_pmd_i40e
_LDLIBS-$(CONFIG_RTE_LIBRTE_IAVF_PMD) += -lrte_pmd_iavf
_LDLIBS-$(CONFIG_RTE_LIBRTE_ICE_PMD) += -lrte_pmd_ice
IAVF-y := $(CONFIG_RTE_LIBRTE_IAVF_PMD)
+IAVF-y += $(CONFIG_RTE_LIBRTE_ICE_PMD)
ifeq ($(findstring y,$(IAVF-y)),y)
_LDLIBS-y += -lrte_common_iavf
endif
--
2.25.1
^ permalink raw reply [relevance 1%]
* Re: [dpdk-dev] dev Digest, Vol 288, Issue 27
[not found] ` <d8993cb75592406a993e3119d45e906a@intel.com>
@ 2020-03-10 2:55 0% ` Zhang, XiX
0 siblings, 0 replies; 200+ results
From: Zhang, XiX @ 2020-03-10 2:55 UTC (permalink / raw)
To: dev
[PATCH] vfio: map contiguous areas in one go
Test-by : xix.zhang@intel.com
Thanks
> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of dev-
> request@dpdk.org
> Sent: February 25, 2020 21:50
> To: dev@dpdk.org
> Subject: dev Digest, Vol 288, Issue 27
>
> Send dev mailing list submissions to
> dev@dpdk.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> https://mails.dpdk.org/listinfo/dev
> or, via email, send a message with subject or body 'help' to
> dev-request@dpdk.org
>
> You can reach the person managing the list at
> dev-owner@dpdk.org
>
> When replying, please edit your Subject line so it is more specific than "Re:
> Contents of dev digest..."
>
>
> Today's Topics:
>
> 1. Re: [PATCH] doc/mlx5: update mlx5 guide (Thomas Monjalon)
> 2. Re: [PATCH] doc: describe the pktmbuf pool with pinned
> extarnal memory (Thomas Monjalon)
> 3. [PATCH] vfio: map contiguous areas in one go (Anatoly Burakov)
> 4. Re: [RFC 0/6] New sync modes for ring (Ananyev, Konstantin)
> 5. Re: [PATCH] vfio: map contiguous areas in one go (Ray Kinsella)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Tue, 25 Feb 2020 14:19:10 +0100
> From: Thomas Monjalon <thomas@monjalon.net>
> To: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
> Cc: dev@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH] doc/mlx5: update mlx5 guide
> Message-ID: <2125420.IFkqi6BYcA@xps>
> Content-Type: text/plain; charset="us-ascii"
>
> 24/02/2020 18:57, Viacheslav Ovsiienko:
> > - metadata limitation is described
> > - no inline hint flag is described
> >
> > Signed-off-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
> > ---
>
> Split in 2 patches and applied, thanks
>
>
>
>
>
> ------------------------------
>
> Message: 2
> Date: Tue, 25 Feb 2020 14:22:13 +0100
> From: Thomas Monjalon <thomas@monjalon.net>
> To: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
> Cc: dev@dpdk.org, olivier.matz@6wind.com
> Subject: Re: [dpdk-dev] [PATCH] doc: describe the pktmbuf pool with
> pinned extarnal memory
> Message-ID: <13189717.lVVuGzaMjS@xps>
> Content-Type: text/plain; charset="us-ascii"
>
> 24/02/2020 18:58, Viacheslav Ovsiienko:
> > Document the new mbuf pools with external pinned buffers.
> >
> > Signed-off-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
> > ---
> > doc/guides/prog_guide/mbuf_lib.rst | 34
> ++++++++++++++++++++++++++++++++++
>
> Please submit this documentation separately for review by Olivier.
>
> > doc/guides/rel_notes/release_20_02.rst | 5 +++++
>
> Applying only the release notes in 20.02, thanks
>
>
>
>
>
> ------------------------------
>
> Message: 3
> Date: Tue, 25 Feb 2020 13:24:48 +0000
> From: Anatoly Burakov <anatoly.burakov@intel.com>
> To: dev@dpdk.org
> Subject: [dpdk-dev] [PATCH] vfio: map contiguous areas in one go
> Message-ID:
> <c6550c8e66dbc7a54a0b495ecda58b0eb79c07ca.1582637079.git.anat
> oly.burakov@intel.com>
>
>
> Currently, when we are creating DMA mappings for memory that's either
> external or is backed by hugepages in IOVA as PA mode, we assume that
> each page is necessarily discontiguous. This may not actually be the
> case, especially for external memory, where the user is able to create
> their own IOVA table and make it contiguous. This is a problem because
> VFIO has a limited number of DMA mappings, and it does not appear to
> concatenate them and treats each mapping as separate, even when they
> cover adjacent areas.
>
> Fix this so that we always map contiguous memory in a single chunk, as
> opposed to mapping each segment separately.
>
> Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
> ---
> lib/librte_eal/linux/eal/eal_vfio.c | 59
> +++++++++++++++++++++++++----
> 1 file changed, 51 insertions(+), 8 deletions(-)
>
> diff --git a/lib/librte_eal/linux/eal/eal_vfio.c
> b/lib/librte_eal/linux/eal/eal_vfio.c
> index 01b5ef3f42..4502aefed3 100644
> --- a/lib/librte_eal/linux/eal/eal_vfio.c
> +++ b/lib/librte_eal/linux/eal/eal_vfio.c
> @@ -514,9 +514,11 @@ static void
> vfio_mem_event_callback(enum rte_mem_event type, const void *addr,
> size_t len,
> void *arg __rte_unused)
> {
> + rte_iova_t iova_start, iova_expected;
> struct rte_memseg_list *msl;
> struct rte_memseg *ms;
> size_t cur_len = 0;
> + uint64_t va_start;
>
> msl = rte_mem_virt2memseg_list(addr);
>
> @@ -545,22 +547,63 @@ vfio_mem_event_callback(enum rte_mem_event type,
> const void *addr, size_t len, #endif
> /* memsegs are contiguous in memory */
> ms = rte_mem_virt2memseg(addr, msl);
> +
> + /*
> + * This memory is not guaranteed to be contiguous, but it still could
> + * be, or it could have some small contiguous chunks. Since the
> number
> + * of VFIO mappings is limited, and VFIO appears to not concatenate
> + * adjacent mappings, we have to do this ourselves.
> + *
> + * So, find contiguous chunks, then map them.
> + */
> + va_start = ms->addr_64;
> + iova_start = iova_expected = ms->iova;
> while (cur_len < len) {
> + bool new_contig_area = ms->iova != iova_expected;
> + bool last_seg = (len - cur_len) == ms->len;
> + bool skip_last = false;
> +
> + /* only do mappings when current contiguous area ends */
> + if (new_contig_area) {
> + if (type == RTE_MEM_EVENT_ALLOC)
> + vfio_dma_mem_map(default_vfio_cfg,
> va_start,
> + iova_start,
> + iova_expected - iova_start, 1);
> + else
> + vfio_dma_mem_map(default_vfio_cfg,
> va_start,
> + iova_start,
> + iova_expected - iova_start, 0);
> + va_start = ms->addr_64;
> + iova_start = ms->iova;
> + }
> /* some memory segments may have invalid IOVA */
> if (ms->iova == RTE_BAD_IOVA) {
> RTE_LOG(DEBUG, EAL, "Memory segment at %p has bad IOVA,
> skipping\n",
> ms->addr);
> - goto next;
> + skip_last = true;
> }
> - if (type == RTE_MEM_EVENT_ALLOC)
> - vfio_dma_mem_map(default_vfio_cfg, ms-
> >addr_64,
> - ms->iova, ms->len, 1);
> - else
> - vfio_dma_mem_map(default_vfio_cfg, ms-
> >addr_64,
> - ms->iova, ms->len, 0);
> -next:
> + iova_expected = ms->iova + ms->len;
> cur_len += ms->len;
> ++ms;
> +
> + /*
> + * don't count previous segment, and don't attempt to
> + * dereference a potentially invalid pointer.
> + */
> + if (skip_last && !last_seg) {
> + iova_expected = iova_start = ms->iova;
> + va_start = ms->addr_64;
> + } else if (!skip_last && last_seg) {
> + /* this is the last segment and we're not skipping */
> + if (type == RTE_MEM_EVENT_ALLOC)
> + vfio_dma_mem_map(default_vfio_cfg,
> va_start,
> + iova_start,
> + iova_expected - iova_start, 1);
> + else
> + vfio_dma_mem_map(default_vfio_cfg,
> va_start,
> + iova_start,
> + iova_expected - iova_start, 0);
> + }
> }
> #ifdef RTE_ARCH_PPC_64
> cur_len = 0;
> --
> 2.17.1
>
>
> ------------------------------
>
> Message: 4
> Date: Tue, 25 Feb 2020 13:41:14 +0000
> From: "Ananyev, Konstantin" <konstantin.ananyev@intel.com>
> To: Stephen Hemminger <stephen@networkplumber.org>, Jerin Jacob
> <jerinjacobk@gmail.com>
> Cc: dpdk-dev <dev@dpdk.org>, Olivier Matz <olivier.matz@6wind.com>,
> "drc@linux.vnet.ibm.com" <drc@linux.vnet.ibm.com>
> Subject: Re: [dpdk-dev] [RFC 0/6] New sync modes for ring
> Message-ID:
> <SN6PR11MB255845FB6665C2A2D5D76D229AED0@SN6PR11MB2558.
> namprd11.prod.outlook.com>
>
> Content-Type: text/plain; charset="us-ascii"
>
> > > > > Upfront note - that RFC is not a complete patch.
> > > > > It introduces an ABI breakage, plus it doesn't update
> > > > > ring_elem code properly, etc.
> > > > > I plan to deal with all these things in later versions.
> > > > > Right now I seek an initial feedback about proposed ideas.
> > > > > Would also ask people to repeat performance tests (see below)
> > > > > on their platforms to confirm the impact.
> > > > >
> > > > > More and more customers use(/try to use) DPDK based apps
> > > > > within overcommitted systems (multiple acttive threads over
> > > > > same pysical
> cores):
> > > > > VM, container deployments, etc.
> > > > > One quite common problem they hit: Lock-Holder-Preemption with
> rte_ring.
> > > > > LHP is quite a common problem for spin-based sync primitives
> > > > > (spin-locks, etc.) on overcommitted systems.
> > > > > The situation gets much worse when some sort of fair-locking
> > > > > technique is used (ticket-lock, etc.).
> > > > > As now not only lock-owner but also lock-waiters scheduling
> > > > > order matters a lot.
> > > > > This is a well-known problem for kernel within VMs:
> > > > > http://www-
> archive.xenproject.org/files/xensummitboston08/LHP.pdf
> > > > > https://www.cs.hs-
> rm.de/~kaiser/events/wamos2017/Slides/selcuk.pdf
> > > > > The problem with rte_ring is that while head accusion is sort
> > > > > of un-fair locking, waiting on tail is very similar to ticket
> > > > > lock schema - tail has to be updated in particular order.
> > > > > That makes current rte_ring implementation to perform really
> > > > > pure on some overcommited scenarios.
> > > >
> > > > Rather than reform rte_ring to fit this scenario, it would make
> > > > more sense to me to introduce another primitive.
>
> I don't see much advantages it will bring us.
> As a disadvantages, for developers and maintainers - code duplication,
> for end users - extra code churn and removed ability to mix and match
> different sync modes in one ring.
>
> > The current lockless
> > > > ring performs very well for the isolated thread model that DPDK
> > > > was built around. This looks like a case of customers violating
> > > > the usage model of the DPDK and then being surprised at the fallout.
>
> For customers using isolated thread model - nothing should change
> (both in terms of API and performance).
> Existing sync modes MP/MC,SP/SC kept untouched, set up in the same way
> (via flags and _init_), and MP/MC remains as default one.
> >From other side I don't see why we should ignore customers that want
> >to
> use
> their DPDK apps in different deployment scenarios.
>
> > >
> > > I agree with Stephen here.
> > >
> > > I think, adding more runtime check in the enqueue() and dequeue()
> > > will have a bad effect on the low-end cores too.
>
> We do have a run-time check in our current enqueue()/dequeue
> implementation.
> In fact we support both modes: we have generic
> rte_ring_enqueue(/dequeue)_bulk(/burst)
> where sync behaviour is determined at runtime by value of
> prod(/cons).single.
> Or user can call rte_ring_(mp/sp)_enqueue_* functions directly.
> This RFC follows exactly the same paradigm:
> rte_ring_enqueue(/dequeue)_bulk(/burst) kept generic and it's
> behaviour is determined at runtime, by value of prod(/cons).sync_type.
> Or user can call enqueue/dequeue with particular sync mode directly:
> rte_ring_(mp/sp/rts/hts)_enqueue_(bulk/burst)*.
> The only thing that changed:
> Format of prod/cons now could differ depending on mode selected at _init_.
> So you can't create a ring for let say SP mode and then in the middle
> of data- path change your mind and start using MP_RTS mode.
> For existing modes (SP/MP, SC/MC) format remains the same and user
> can still use them interchangeably, though of course that is an error
> prone practice.
>
> > > But I agree with the problem statement that in the virtualization
> > > use case, It may be possible to have N virtual cores runs on a
> > > physical core.
> > >
> > > IMO, The best solution would be keeping the ring API same and have
> > > a different flavor in "compile-time". Something like liburcu did
> > > for accommodating different flavors.
> > >
> > > i.e urcu-qsbr.h and urcu-bp.h will identical definition of API.
> > > The application can simply include ONE header file in a C file
> > > based on the flavor.
>
> I don't think it is a flexible enough approach.
> In one app user might need to have several rings with different sync modes.
> Or even user might need a ring with different sync modes for
> enqueue/dequeue.
>
> > > If need both at runtime. Need to have function pointer or so in
> > > the application and define the function in different c file by
> > > including the approaite flavor in C file.
>
> Big issue with function pointers here would be DPDK MP model.
> AFAIK, rte_ring is quite popular mechanism for IPC between DPDK apps.
> To support such model, we'll need to split rte_ring data into 'shared'
> and 'private' and initialize private one for every process that is going to use it.
> That sounds like a massive change, and I am not sure the required
> effort will worth it.
> BTW, if user just calls API functions without trying to access
> structure internals directly, I don't think it would be a big
> difference for him what is inside:
> indirect function call or inlined switch(...) {}.
>
> > This would also be a good time to consider the tradeoffs of the
> > heavy use of inlining that is done in rte_ring vs the impact that
> > has on API/ABI stability.
>
> Yes, hiding rte_ring implementation inside .c would help a lot in
> terms of ABI maintenance and would make our future life easier.
> The question is what is the price for it in terms of performance, and
> are we ready to pay it. Not to mention that it would cause changes in
> many other libs/apps...
> So I think it should be a subject for a separate discussion.
> But, agree it would be good at least to measure the performance impact
> of such change.
> If I'll have some spare cycles, will give it a try.
> Meanwhile, can I ask Jerin and other guys to repeat tests from this
> RFC on their HW? Before continuing discussion would probably be good
> to know does the suggested patch work as expected across different platforms.
> Thanks
> Konstantin
>
>
> ------------------------------
>
> Message: 5
> Date: Tue, 25 Feb 2020 13:49:38 +0000
> From: Ray Kinsella <mdr@ashroe.eu>
> To: Anatoly Burakov <anatoly.burakov@intel.com>, dev@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH] vfio: map contiguous areas in one go
> Message-ID: <276c85ed-ac2f-52c9-dffc-18ce41ab0f35@ashroe.eu>
> Content-Type: text/plain; charset=utf-8
>
> Hi Anatoly,
>
> On 25/02/2020 13:24, Anatoly Burakov wrote:
> > Currently, when we are creating DMA mappings for memory that's
> > either external or is backed by hugepages in IOVA as PA mode, we
> > assume that each page is necessarily discontiguous. This may not
> > actually be the case, especially for external memory, where the user
> > is able to create their own IOVA table and make it contiguous. This
> > is a problem because VFIO has a limited number of DMA mappings, and
> > it does not appear to concatenate them and treats each mapping as
> > separate, even when they cover adjacent areas.
> > > Fix this so that we always map contiguous memory in a single
> > chunk, as opposed to mapping each segment separately.
>
> Can I confirm my understanding.
>
> We are essentially correcting user errant behavior, trading off
> startup/mapping time to save IOMMU resources?
>
> >
> > Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
> > ---
> > lib/librte_eal/linux/eal/eal_vfio.c | 59
> > +++++++++++++++++++++++++----
> > 1 file changed, 51 insertions(+), 8 deletions(-)
> >
> > diff --git a/lib/librte_eal/linux/eal/eal_vfio.c
> b/lib/librte_eal/linux/eal/eal_vfio.c
> > index 01b5ef3f42..4502aefed3 100644
> > --- a/lib/librte_eal/linux/eal/eal_vfio.c
> > +++ b/lib/librte_eal/linux/eal/eal_vfio.c
> > @@ -514,9 +514,11 @@ static void
> > vfio_mem_event_callback(enum rte_mem_event type, const void *addr,
> size_t len,
> > void *arg __rte_unused)
> > {
> > + rte_iova_t iova_start, iova_expected;
> > struct rte_memseg_list *msl;
> > struct rte_memseg *ms;
> > size_t cur_len = 0;
> > + uint64_t va_start;
> >
> > msl = rte_mem_virt2memseg_list(addr);
> >
> > @@ -545,22 +547,63 @@ vfio_mem_event_callback(enum
> rte_mem_event type, const void *addr, size_t len,
> > #endif
> > /* memsegs are contiguous in memory */
> > ms = rte_mem_virt2memseg(addr, msl);
> > +
> > + /*
> > + * This memory is not guaranteed to be contiguous, but it still could
> > + * be, or it could have some small contiguous chunks. Since the
> number
> > + * of VFIO mappings is limited, and VFIO appears to not concatenate
> > + * adjacent mappings, we have to do this ourselves.
> > + *
> > + * So, find contiguous chunks, then map them.
> > + */
> > + va_start = ms->addr_64;
> > + iova_start = iova_expected = ms->iova;
> > while (cur_len < len) {
> > + bool new_contig_area = ms->iova != iova_expected;
> > + bool last_seg = (len - cur_len) == ms->len;
> > + bool skip_last = false;
> > +
> > + /* only do mappings when current contiguous area ends */
> > + if (new_contig_area) {
> > + if (type == RTE_MEM_EVENT_ALLOC)
> > + vfio_dma_mem_map(default_vfio_cfg,
> va_start,
> > + iova_start,
> > + iova_expected - iova_start, 1);
> > + else
> > + vfio_dma_mem_map(default_vfio_cfg,
> va_start,
> > + iova_start,
> > + iova_expected - iova_start, 0);
> > + va_start = ms->addr_64;
> > + iova_start = ms->iova;
> > + }
> > /* some memory segments may have invalid IOVA */
> > if (ms->iova == RTE_BAD_IOVA) {
> > RTE_LOG(DEBUG, EAL, "Memory segment at %p has
> bad IOVA, skipping\n",
> > ms->addr);
> > - goto next;
> > + skip_last = true;
> > }
> > - if (type == RTE_MEM_EVENT_ALLOC)
> > - vfio_dma_mem_map(default_vfio_cfg, ms-
> >addr_64,
> > - ms->iova, ms->len, 1);
> > - else
> > - vfio_dma_mem_map(default_vfio_cfg, ms-
> >addr_64,
> > - ms->iova, ms->len, 0);
> > -next:
> > + iova_expected = ms->iova + ms->len;
> > cur_len += ms->len;
> > ++ms;
> > +
> > + /*
> > + * don't count previous segment, and don't attempt to
> > + * dereference a potentially invalid pointer.
> > + */
> > + if (skip_last && !last_seg) {
> > + iova_expected = iova_start = ms->iova;
> > + va_start = ms->addr_64;
> > + } else if (!skip_last && last_seg) {
> > + /* this is the last segment and we're not skipping */
> > + if (type == RTE_MEM_EVENT_ALLOC)
> > + vfio_dma_mem_map(default_vfio_cfg,
> va_start,
> > + iova_start,
> > + iova_expected - iova_start, 1);
> > + else
> > + vfio_dma_mem_map(default_vfio_cfg,
> va_start,
> > + iova_start,
> > + iova_expected - iova_start, 0);
> > + }
> > }
> > #ifdef RTE_ARCH_PPC_64
> > cur_len = 0;
> >
>
>
> End of dev Digest, Vol 288, Issue 27
> ************************************
^ permalink raw reply [relevance 0%]
* [dpdk-dev] [PATCH v2 2/7] net/ice: add the DCF hardware initialization
@ 2020-03-10 6:50 1% ` Haiyue Wang
0 siblings, 0 replies; 200+ results
From: Haiyue Wang @ 2020-03-10 6:50 UTC (permalink / raw)
To: dev, xiaolong.ye, qi.z.zhang, qiming.yang, beilei.xing
Cc: wei.zhao1, Haiyue Wang
Introduce the DCF (Device Config Function) feature in the ice PMD, it
works as a standalone PMD which doesn't handle the packet Rx/Tx related
things. Its hardware entity is the VF.
Add the basic DCF hardware initialization, this is specified by devarg
'cap=dcf'.
Signed-off-by: Haiyue Wang <haiyue.wang@intel.com>
---
doc/guides/nics/ice.rst | 47 +++
doc/guides/nics/img/ice_dcf.png | Bin 0 -> 39168 bytes
doc/guides/rel_notes/release_20_05.rst | 5 +
drivers/common/Makefile | 1 +
drivers/net/ice/Makefile | 5 +
drivers/net/ice/ice_dcf.c | 474 +++++++++++++++++++++++++
drivers/net/ice/ice_dcf.h | 52 +++
drivers/net/ice/ice_dcf_ethdev.c | 317 +++++++++++++++++
drivers/net/ice/ice_dcf_ethdev.h | 24 ++
drivers/net/ice/meson.build | 7 +-
mk/rte.app.mk | 1 +
11 files changed, 931 insertions(+), 2 deletions(-)
create mode 100644 doc/guides/nics/img/ice_dcf.png
create mode 100644 drivers/net/ice/ice_dcf.c
create mode 100644 drivers/net/ice/ice_dcf.h
create mode 100644 drivers/net/ice/ice_dcf_ethdev.c
create mode 100644 drivers/net/ice/ice_dcf_ethdev.h
diff --git a/doc/guides/nics/ice.rst b/doc/guides/nics/ice.rst
index 8af32dabf..2639ae239 100644
--- a/doc/guides/nics/ice.rst
+++ b/doc/guides/nics/ice.rst
@@ -240,6 +240,53 @@ report a MDD event and drop the packets.
The APPs based on DPDK should avoid providing such packets.
+Device Config Function (DCF)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This section demonstrates ICE DCF PMD, which shares the core module with ICE
+PMD and iAVF PMD.
+
+A DCF (Device Config Function) PMD bounds to the device's trusted VF with ID 0,
+it can act as a sole controlling entity to exercise advance functionality (such
+as switch, ACL) for the rest VFs.
+
+The DCF PMD needs to advertise and acquire DCF capability which allows DCF to
+send AdminQ commands that it would like to execute over to the PF and receive
+responses for the same from PF.
+
+.. _figure_ice_dcf:
+
+.. figure:: img/ice_dcf.*
+
+ DCF Communication flow.
+
+#. Create the VFs::
+
+ echo 4 > /sys/bus/pci/devices/0000\:18\:00.0/sriov_numvfs
+
+#. Enable the VF0 trust on::
+
+ ip link set dev enp24s0f0 vf 0 trust on
+
+#. Bind the VF0, and run testpmd with 'cap=dcf' devarg::
+
+ testpmd -l 22-25 -n 4 -w 18:01.0,cap=dcf -- -i
+
+#. Monitor the VF2 interface network traffic::
+
+ tcpdump -e -nn -i enp24s1f2
+
+#. Create one flow to redirect the traffic to VF2 by DCF::
+
+ flow create 0 priority 0 ingress pattern eth / ipv4 src is 192.168.0.2 \
+ dst is 192.168.0.3 / end actions vf id 2 / end
+
+#. Send the packet, and it should be displayed on tcpdump::
+
+ sendp(Ether(src='3c:fd:fe:aa:bb:78', dst='00:00:00:01:02:03')/IP(src=' \
+ 192.168.0.2', dst="192.168.0.3")/TCP(flags='S')/Raw(load='XXXXXXXXXX'), \
+ iface="enp24s0f0", count=10)
+
Sample Application Notes
------------------------
diff --git a/doc/guides/nics/img/ice_dcf.png b/doc/guides/nics/img/ice_dcf.png
new file mode 100644
index 0000000000000000000000000000000000000000..540823c4a05e9ea9d558d233fea77016d1ae5df3
GIT binary patch
literal 39168
zcmZ6ybzD?k*FH`NDBT?jlF}e8B?Hn74BeolfOLbDz);d%Lw70NNOyNPNO%7Z_x(KY
z`+k3a@X<4KW}m&+Uh7)dy4F6yN(xdKsHCWHaBvvXpCpvw;1KwLKlqmjz$g7i%71`=
z;O&*A#NkSZ$ajH@7p7wJVsLO35oq^@NWk^0Z=W>n;oxu`p8w#NOsHJo;1Xh`CBz^u
zdIzZ}I(W`x-6a%La{^jclNOU&`_)rTS4-}zY76d1P4w1CVq)-OmrKC^5#ZY~-}pzu
zy$L?@i@zdO6jl2qp0>O_?^V0~==_TKu;0E)6AT_jK}Ys$GuU9xpl6NmUq4AAAS8rZ
z5<h>~BQL7h+1bg!^v{JkDk^GFZXt;WZK)wU@I4Lh>Nma-8Py=vX58uL5BwsNv0+TC
ztdd|ub|{f>I#qBR_r=Ecc2+(+F$m*+POs4gtgWq$;%5$Zz1W+)IolS4AR!?E!%MzF
zY^hc+(<6BKV$B>q*BptIQ5gaPfd*IJosgp=XT25!8zwm^L5M@!Q9uFabHF7#DiuCK
zt~qeo#^9j@Nyw!ThkWHBd%m>z=MtX)IqO&q%qUT-Q<|7KJNMhNs;a6pC&Kl7XSB)v
z3KtLW;^N}!aM9A*nl<f&LypeAH;UH%@~=<W?^;Mic5d!cle_a$qiaVX7S(%;iC{c>
zPEJmL@k|#dzb#-e#WW$n3TwM=5&|%zK#w5Rj~l++vL_;_xx2ZfV1g5o&-OTVud|dg
zY)p8a+CN(%s#Jb5B@9dFyIC8&!|9!kw<)+dnAUfkUON7RZe*>j5V@t@LFIjKzBWN}
zl6bSB?AXD@u-6mJ^$s$%>{fg;H**<Wy_z$R_0fFyxQl_NC8(r`t&yYbC$Wq!ZhWiR
zaQO>2)_CPI7Ur}ViP+8SHXAZ(qCX-8Pgw>@)`d8#Ub}%C2oieVUWG5$Z;wDPw0oh!
z(MS2V(Mt&eBO3Siyc1oDi4_JnN!?`_!F7~4=LJ<;7i4@c?GuTsp&oXOubJN~j*nRE
zr*(v=l`h&IUu)Y-G0<(qS?nZJwYZ$k+*tNcTf2SvjF$-h6dEDNR%>@OyFp@~wx8sm
zQ@KkHCa?M<DDa*q5JkVi|7}X6ooQlUmnxZR*#{T+GPIBvv|B=xJdNM>M<?%wx7c)l
zdn}q*;Wg|Gzyh^yRZNGE1Xt2+qw~^RE=E{lE|-=pha<-cjHj$U3eSXtS-y9ciA;Qt
zg#X2CIXogbtk+gV#pepYbkK~qZROc+2-nD3e6SS4d#n~;vgmMgi02}r<bB>}Fg(g-
zTDmx2YrO0kRzm_}D5S%J%DnQ0DdV>iV*NTt#-ch%|0xYIm~S%Q_BwvyfdQLqe%iUU
za<4uRAr<XM?#|y&3?~<aXTr-Hgaz#tT;9D;cK$2uv9~aYB8Zi$G+Wuxv-FN9+5PYR
zh4qI=trjcZ;=wL@A=W&9;v>?i^7Yn}O(rs$G1Q(|Z2z1sn<Yns)V;&#c$@TE2Q)Ru
z)6)`Bt!gQksJ*lfL=-F}Q=R&|&-+U>DkcK~7vFVi4u!dmmta<R;msWO$55HcDm$Mq
zYzuC++dJmp5`L=_vy^#UbXhfEraEBcG`ZP)uwPnpl~qe?xmmf+NiO2ax1Om+(c=}a
z!GBS0w-`sWU7FHRP;K|H)xhT98<o%yk#hLBt#8R=)!=YcP~pc5m_`)LD5eB~iYqs_
zj?tLy72jM<E{lrbr1en>*bVr_*W;KI;3E7^sdYFjoC{%b$NVtKVTT^7+2BYNw7pYj
zi4R5^w->W9pqU==f1#Eb;6C!>z+DXeb>TXZwZp;G`tJ%B5BYEu=0)FDN%x4fNM_s}
z&3d^+eysZRo^x@R*na2q(uHz@x%b$Uaf>KVBm(XH446M${l^6#mQ=@EaW_yr=YvCd
z+ccCShFQ?$1}?+o10_*^aev+*-pN^+qy-`EqoDYuWpSd~d>GvmJf>Q%JrCl>)WF6k
zmNOMMMp+hgneL-`A#pMMbJc=vS<;lS;}#5&f&14fUK<#DZv350Jd6LHkN$NrFqGJ8
zlLQ^XPt!kNF7<(R!K>C~vVL6sXh^GeVu@|f9?Xx@{W00`WXRZv(|^df><;rf5PNdx
zBpMO7vZw8Z5tyP#0>Vqx1X*<dv&0Kc;4ZffjQvQ21Vb2S)~HS*e}&q3vP0X5Ck>{p
zL|PgxQ2d3lia{mhxO*i0u!<pNN+=Dr*xkF}rL@3)H_B6SSLCd3U(-4g(^%Ggi&~71
zLC)Opd9psBf9m^tF-R2=i~Y+Wx~{DJ7&!b`+!vr6(LbBVOG$#4e;21&m~FS))*G-=
zPa}wqkl#vdk|f?wwVSv9(#)clt1G+|;%v%L28)#;+6VXk{7iUG+jNKVTH?KH*Mg`P
z4}U@{EuUAnkMX-Hs~^?DoM;li?~cv8zY`mcjZ7JLTz<$aU4RE4@Y}g{&JAw#$bdHS
zIsj~7y#89(M|b^nq}$-hTBu4^z2sUdUo4>v8BBVg%wyU%_MXf@bERkY-D?x<$;O<b
zpvBDRefW<DBArvMWHxmVUG9o}n*WowASDGwz0FLH(Dzku6^(`Vv9TH-Or_3WQek9&
zp{?wvKs0uy5PokudTJp-5ZP7duzj+zYp+MDA<51+GTg}}+XPcyOCY#AF=5lm{bEKD
zuYwz`>Y=}WkR(4(Y2H+%2SHKL8L4sdgQdlyUB8->Fe_=z_$V35@6;>_^Sx+A)!#{y
z%vNutx>hi*td+hI$B#YrfhqfD<l~3DlzhC=#`503zteBJi#)&=g+hzrVxg$DkD{I@
zg^onlA0{~L`RpH-+6gSVF_?>R2n@E#DgbDpEt*<Z@je@xlV4BeZ%k2#RC;@KHPXI=
zTv4h`bGcu3y~M9wU$HGWe-9h0vD0j^8a@f(O+_$Xo~<oWUu<$5exr+)S7gD4{3wVn
zgrav3%g?CuuSSpayqwxe{k^|Fjm)EGSivv!zjH;?(yL3=fBQ=%uWad!A{EMEQA^#0
z1YSLDC4#7>-9zS&zAcI<8lQ|mDf7d6im6Q*Pa;o%x1R@B!{cXusK5DSe-NiT0+q(3
zJAul<M~5YH9~^lh-eTUP{6+YH%@Tq2W2W5mXeE;-FoLq>=#Q(^6LkG!bA|cqcF*pi
zs1U^jfwNgMV;A~2jK#)8*61YsCi9QR)xxWa4oeYf>?DNCwZ`dlvWs4bPp=e=C;^0@
zKx^mvbur_;)>z-nUx7=5Qf6*XICBRQBO-{5^FAFbsff<KpkO!a8>m~+y7p>YE4&LA
zuaOFs0yDxhFwwq*qMEm7{5K*3xDA#58|>mq?5K=|*ax&{U5}%6AKyg@FV2-FFZMsS
zrHF6K$F#kK%8>ZN{<(`uWDrDi_jGHzJr(KJ|1yPYc)l)sxy8H2+vzKLzZc?{M!E7v
zF8j7?Yx*gy+=(y$qRI15nG2g2=Obg4BUlV=;xP}#z2<_JKj<^R-sjvH4CwW2sL>TH
zBFbS!1ZIVePcY_hP7Oo2*3+(G-We}hkFa4=pbOcJV<M%e|C+Y!zUqG{n|J=*>)qtn
zuvYEZ1ox-boipvgxq`t_VP5lxlooZlyR(J3H(<AZF?Y%7yd%9i@7`?6^1pYM<8}vA
z(fcoCgsMf!oK2VWC%GQAgevYXL7o?Eii4GU?~V&J?0;;|X}e4MzW2UeoiA;v_Kja^
zfO*44nT%$s<HJ_iHD0JS+i49q+(`zs=pvVW8i+;xuYtKK!0_&IS4RuYlSNh0LB0N0
z@7M}4b%hGL51O*{(vrSeZ6D8OHCSs;@XmhzeVhf(Slk`m&D&IY@;78t2t{dNPCng^
zl3t`DCq?X(3hcH>ORIJ6{H~eR;2X%NKo&enSZ7_?BtrJ5+S7I!1*N++VP8<HIBoPS
zs4s)hk6=@{5NU`^H+?`wcP3Bej!F{iUVeW-Vmz5k<8mL=To^D!#+Hw%%ierPJ88H_
zPv82kH0g1Ai1s0S9toBo^Lh+BR#Ex!G~`x30&(~+!!jfFHn1X}Pg@a3JmuSLTQ#Qb
zpQ{H2%o6CZ4m_;3gUII{#r0Kl7wZMxrF?pq^i$O~lx>HQg7%WjlKr9^Cy2f+@^BqY
zbx<#R&U+j$A5c+HBI@EZBVkOHH&Z}NE>JzIVq7w@Q)x#`<VnX!98`B2r8m%6Vo+<#
z6|brd_*__wX0{t1GqA?_MGCa@sMxH?;yyPPF1TB!b^Hhn>~1g2KzO8);$a8uAH*&3
zg)D9M>q1*@_xCVo#(yS{1NMP%E6V?!Zdx45c`#Zo`Hos^`dDktCFvy1sVN@{zyDdY
z>><Z7B&4vz?kkf60-^cx)}7#;WUQ;|Ust^A{Cc6ps{`J_Ho33e7$a~vXbL1#t`aT!
z2UJ5oWg&|^mMJOh+0J)(WRFNE*ofb`(PH0t*QCEMv<$k$g`L*rpQiFCT-S{2>uUSm
zw<=5E{Hpn^91&*!WTH|0=HF?*gcfcA(20pesM@XX<znYR-nMzrF*R^)xxTPuJ-mhU
z$llE1dA!{69o)n5LZtfsGN@04B+k+hUE_rgg5;a&za<srs{J-<3Qy6+aIcv(Vh0~)
zkKgAXo_BfQU6|#09=7DXY~^inHrqTNHErF7_5Idg-n+BOIH|92e|zJ_nvaQy)}E~B
z(h@~J+X?{0YlgxkKbSHfz6PX3+q1^PZ8&z^?eZ~;#lzEVFypvJwzmp*NjFhor7Ydu
z;qg_Ml8d`7>U`!J{-r<=sz9Ex<rB%wndw8n3T-!>HqoW|-MF)wvnft=dI;3KrdhMf
zMAysew~LW;pSGKW3p2yd|KC%+rqf%cE><%ubiE&Jx1%0MZ=Vj7xf*$_KBZqWak)pv
z4j1_4XAlThcpZNinfEz(+YMJ<jd{>q^T3Cm4}mOujeA_~ly2A-OCo@BUWkDw$?;>`
zNAbhK1bj9N)%iBl%U+H3#(W(f7`9(~6|64(y#)u!7m=9$ls^wV6HaD*K=}$*?`*(r
z%PAmMv7h_OPY`2waY<I!<1y{9lkvg*S2VsV;C^W>H9yql*ma~+oMt}#AG;;>^x|`8
z#R49-yL6HlpGSwi9Z64GB2a6w($eJZ>gJ#mxlbpK{v}O&29pecI1p8Qp<h1SHvO4N
z_+F)%xO0841x)wd$&8Vw&1HJ3w?MV^H)!Ha$X1O>IO*|zpZSM6>*QdWiv2AWGUOWT
zi-U3Fb<QX6QPsrlDFCm9oEyVYRmw8*Eij=nt7ZU<rce$)Tpe9zp*)({+wiPnEysrt
z%KsJKDNEN+O_ku)jJ2#;k}_W1r057%=`a*slt2m_4Ow4Dxwv;-?t_o4?>G2;iM1t1
zHTq@j=TP}0C)-I{SAz4qYL-K^qweE9?(nlNI~&!^GA8sit}QIH$3qI@ShGe{waqKL
z$gP6l!{HMFCb=eGSRp?HQ$W^k5NfTr!FhVBHe2dKhrT@y7@j_LUMO2X!U<qQcl9^w
zAMYGS1h^`W*Fliy$>29w+j$DuOrED%X?afH7nQAm*uGMru4R&AA=<D2a2dhUa`0r@
zkTJbBK6OSuwLJ5|P9q(YMcV`M$U<Wo(*mXYZ8CH?GyY$Y{x{kJ@CmLnT3u;ei(W?I
zMzAO`+bv3<e%#6GvtLFB?dSqKV^E+x%tEbya&0sX)%=q7bl~KNG#gIFFumkEV<7hU
zq)iJq1xI5@jQ(S?U1mj9Smdun_!|Kx7{jS*z}2gF{sN8ft`X>R+<Q>}*9|j&!o)35
z{$bev7y}Xb>i>Ao|77I<=gZUAg;co|EjBB;+t~N>_f;{P9o0(>CsVsQIY)8(M+{Z0
zX0uI|msFKEtT~ttnsz7Q&HG#`N0?_r9`7GV%pEN$8bm$wuu-KbwUr2qDS>DJ*c!rf
zo^{1mo!8969ZL|;%-@E-i_s)2-OJ(P&$+jIm%hVOr64opW`*}ty|pQ4f6VyiOYbe#
zc=({mBPV?e10QS2MJS$rn8*$>vA(j}V^5F$v@VUb2^BbA=Q!jU&j8~69bkn$Rr&R-
zO%6w?WLzx_VTtLj2CV!KGIodUcx>y$1FafezYvGt?k~pa`aUwZ1UZA|ee$tkNv=Cc
z{JGZu?=XZZBj@Ofc4+btE(a9d-5DgAA*&B(**VS-x*~(_)GHAM5wR(${3Wbl#jKod
z%!O|u`G9Q+|2vP8d@VU)x3xNg57CaY!$@wqG0aZHEbB+A(>?I5v+>m<Uz;jVsLJ*J
zW~+@y6zO<$tl?|lNJMu4hC=%f7vE5WJWMT(IHLy3^({EaKP?YB7ke1l@pqa|SOJ6V
zP+$pl?BYl#$__WuF_p)}{q`1!uG%lf;Sa^0L}IxMpnos@Sn`)SbPo_}$YtIzMIKdC
zPs>vd6ZxC5y|K}HBV94S<<b$`HYa;Z!TZt^q6v86x58P{wkI2UoRXG>hZ~Pm)?ev(
zXVwAnw+Kn2g}4PvdewlLVqV+5V`7&VqCwG_bUp~XJRAD!ZJ?#2yhSQ*$+5}6kL+Ty
z7n`c?qP19mF#uIzF`@8+FFFm+?x5Zv7Kf||U6sF$ie@YBg?|s!O5Yd)@?QvRXPw9W
zzAFH=P#$oWOh*kT(7*Ed?XvkR4WNH}In6(}n`hUC2FjAX9nI&<5ll~Xlj0N(M~gM9
z7W;3xm?1qmFD1O@Yps*F!w=>vJEt4sE)}`;KqRl-pB?W8e==fRQHbNR&yXn9-CPzT
zX|2+4M8JL)^24fW-lP2u?B&<ST&l#FFZ)D-uFKJC<Kab1+ygzELfogXQQ&I#70tnM
zy54n{`>H0Fw|hk<)fQE<3!TG@ud#0no9o!?oNKC!@}4eEg2_~&7N3qT-Qsh9Iv>z{
zm)0<yr)u#7_;6@3bvjJz+ggqdL`H_^?hjYdPzWpB%A2F0&w3<JB_EHSqr-ppKXP7$
zrWt>;AauX_U`)J@pMV`1Q@7$z#}5b8W-NWwOd}B|VY?aWP5mhB2UD^`MzduiO()As
zi*rJP!fC{s1^J?Ot$QRN*3s2*LB2-Q@>P<~K2Ce@@ChWHB0qvX-#@OXT6O<z6!)t8
zuHHuvKO2MQPf~XiwS5gw^y}|f;*G`8qVqEEj|<{jf0-d9z;+?Hsd8<$bJw!TNAoog
z!lOHQ@wE`4mlgY=k-F55@ok?~VYgZ9ii?83DUY{@BQPW0Kt&6>3A?52TkfiF>Svqg
z?k5mH0g<X3N|2TzLCY-?>EQvDCi4-yJ&q_ZOB?^Ly!aEOJF0~0B(D@5kKF=wuhD|l
zU+C#Q8yn1*!P{yZ%n10gV=xIA)5(n8w~GulmX=Xr!2;1WoHvT$;rkC#$)4|!vFT+X
zw1r+D=B(0|$vmpR_ND*2G9vq;Sw=C};CZDWLZX{Kby1`C&o%_Y=2S+BH<R2XLYGSE
z_=^|+ssvIdqk*Vve*U?%e-cz*yJPIOZgw1c$H2N3J9pcuYz={bj%-@o*jDZLI~QwC
zBAzvlhUpVeBJ9G0MuPyI?ZP@V^Kz#H7sx;!p1xk4&&-S`|3LqHbLRJvK%QV=Qkb{S
zvFK<?FJ<lVt5;392rQqy5=su0>1Ji8iLbg#!*Kvdbu_1a%bzZ^ESUW(q;MjVZ0*f;
z!p?7bS9101myQ#T`rwFer2+tPS}~s>qQQ4@qljMnW0?dl0Z$d)m31QSYb~j2ibvAI
z#x<niUH!DZdzYp5SBs_*horD3f0JK!rz1mnfewB4Gb?v?I8aoYjvl{!QI?LHC9;+~
zoO4O`Ss;?Z^Aldl-U@c;ZGbzXt2gmFZFi(IS+89!*Lk<F@bP}#ZyFV(^jK1WN90`~
zt=Tthn9XCnm;+Qjz8G0Z>*7Ne;3SWus}*&Z1bGmhWqu<dAv#}%Gd(=ndllitwoRIo
z=Y;_o^88@Qse8?9I6Pwi)bibdeYC$Yfnv85*zhy4D|RdlSSDYL0)*rCoha8?w<DT&
z1A2C)NtZt0&W>)}@#IQP`74&+aOIqCHv=OPy;3=g?ZXcQU_r3<9`^sxN#N(Sl-g5+
zhm%cbigdiDjZN%d1~Qg0ODQFKeyoJR`kd#k|IXA-l{D|)&PM<Niw)F*#IwjCReL1}
zYT#Hf>PQn`j?{K~c<F;FKn7kjrJhC8ZhL_tqV0FFyv`9W-^|cIJ!3w5CB}a^^%s2A
z!kZ-9CkOI{{doQP-#T1pWlE_zQTCS9g~NcX{+SG4b?_OC`xuy3j+4X;3IK1etFU|_
zwU)#L2MPfH`45d!QU<#gu>th)=$7BR;X|@wqUv5<r@H;<-%;IVy_9xGyk!FH?40`i
zh&xk1X(J4^;^f7B!0d@?mW)-O{}o#QN$MY^K{-1A05(IRQ<dU{w)@%ijcKn;(QdbU
zS>2MQW2XvUa)ii)_u`-rnYHKB?wPOvE8o>{*St4#(@XuD6t}M<Kjr}^p&@SEy-a^s
zicMrbiSmDPIzRp-EIO&}#9Z*`{%WHma(;pwaXmQj(l?ztiZhz)?1y!o?2<o%oYP^B
z6=Bgzb0(P_rjGYJ4x0Ex{g{L_v_D*Sp*kJ(f4vqey-LLa(*Ttf96(dRO5oVI*kvHS
z)Yrni5#mxk52&L1ST~|zsG-JRTfTf6_;CA&S~zwWFHy*R*chhF3+&o6)p-`N@F9?l
zoSB%vtJ+66h^$SOipE>?*&Oc(2)=DpfDruZ?u)O3FB}0j^bIi0vt%bB26EIJ=b_`j
zOA1%H`Vt?X?rOD8woVkiMxan81My~V*uaT99~-NAswewgQT>020z-bjLUXyB$=(5F
z9G~s*y0#nZkD*LO`6f}2YVh8EaNq&r+Ak|aWUE>bp(T*mSAqFG%OgQ+P>a@EZ6~h^
z>o?rr#(V;ElS_BiX$tkX(wtspX6DCW{Fg{%=6j`jPqn|>OY@nhl4Zk*<q)K-4WtE2
zT=Xw+#p}(Eb7^W#JwAPHY`QOYdJj{syXN6!1lUnlx{1dr@tqdHzW4wjf0jn517Qdg
z=!?1^@thZ2vBvf)uiJyS<5q(c)qhhM#%1R}>Uml%abT1~`U|=XaVE#c%g7PeQvGo0
z)-=6kkfAb6jV(L}fX;C^-057|Q>o|d{FD#WUvE&jnVy&A0k!CR%(MCH2e^R#JaS_N
znggoLNCg_gLo2g1SejKFNA;^kwYJ$B9v)jOBm2@Ez1!lKMa;9uhM-!^XPe#-5Da_T
z&m`v{tLN!0^5385dIR!-r_<Mz`Kp^bCju;?82igI@{uap{k-ljoM?_CwNSE!ijVV0
zz3^eE&kIb4-5Snu?08)+a0#C*>;Y-^pa!G+!0pq(t;w3<q&Bj#lW>Mmesfu0*hRc!
z9+d~O{*%Db-?Y1~yh*E<m_qESjvJr`{%e1IS9+h$W$(57K`FKllA9RK+##=SwgngC
zEztt?2qCD<Mnjl+!{OwtG*ENSHW;<0Z9W(_c)JW{tnWY8)wyTyk2BfSs+*cg7X2EQ
z=d9!RQ!e9TV3MPe5(d$xXXIzg%MsRqLI|_E<WHe8nFI9>+r4Eij&q(H_&ItXP8kY0
zhglRO!te5?*<|T1*jqku^}AWsT}=UkmNZ?_-w6}<0=A6MGLhUG6ZHH2?K<P9jEQ}8
zDC&66za{7|^II6Mi%l4p#)<PZPsg3p@u<d2FBtkyiJhz@lgzafkLJ#UM3Wkm?;XTj
zmT!p+XUhU%EmKC<2#;9CaPjnKoFz;fD2UKSp1>Vu{yM{NH3}k_`}6Qyb0P^HP783l
zsV!U7SEgrT(W^G`bA^LWUP5e($^kB;9^Du?Kna&UdyvR;Wg6I|skPDw-YJr4w#ER(
znm0{dZBp;MSFJjxQVq*<YpI?-@nYFvH8Z-zqQ4`s2@8mGHbH9ELNAkL)kb@!ZvOzL
zbaE~m)6AOw^82--vSzn$W^!$Licpx+R^5ALo10FnjySnP4)afW9Ejf9@0+$`DLTa1
zH`e)E4e_21xXsPBr%mY(v^fmgTT&&@_u<6rf17?ToV8FwGlSDB1?s>OEw3~CeL_5B
zXD#8)VY7!#1e-|^Uc=Q>)#3wU7FH2@(uewe0EqooU%ubdPph|-Zb{6lZ$-_+c=Fu$
z6$Wz0#!NBZb&#$q9@yuSlAOHCLTk97!#j{y#Pj;7dm7`?qvAWeS>%51QL`21@+god
zr-pamJ8Ep7Y5(wTr=3<(`G1$X<Jg6Mdr)LDm=W=YXEqk0J%gk~N1(175Fo?^mDiec
zM!sZqrY0JrI6@L<(R4uPysw!?#(=sMP};O-=5t2qzi1Q!!exVOpqdSv9IMm3MDf9y
zuD5@Fv(3pxMvK>!09T}tA7WAoy;OJj-=}>aRYYTSwU-{%bR=a0zf6?!E=9~z(>@6&
zK8C=U>}FrAk$gC=0rbZ&^TqERW<P)XLdr+5L|<EF)G7sjxbCO{#0Q@#47%PgmKZ=3
z{#$yRkDr5LjjD42;*<2=GR{(ean|!S`bG5m{QY(6P(ZIo*^`Hp{Z$X3(?(%!(@IPh
zS(4dY1Z)srZKK^fSh_<tsh*Vy<*<C9z#oj^8y;D|LHPEHTZ7SL?TeV#u?h0|JxM+F
z&v$a-Cm}G-mJe0>S58vXkI_a<dWy~KlD|J@-8tEqdfE`m;leCV4yK19wqxDYYM##e
zpA+IgG$D#dULW#0qAVl*HY$G&UF|!Im(?`qu;GVQWCXSgr075=y$@$)BfPSPQP`6|
zmkxU2MMw5&0g6{+u6G1DibM`6IA9M_94K+Hl3+~%XeZ%rpotU5r(X{R0jrbdfAqT9
zpnHFA+gctLP7gX6g{}tQ-p1!$-ZxzoaYY-zy(yIbD43H}_9niN20+k%wYhBhaUy~a
zogDY>^k#-ahKEh#r<;;>HWpeIB#Hg91m44lI^)OV>+6|oWTMHiMVf~DxxLd%!zj=S
zpkh(X{r2$k&`JB)d4s8sKY94&!OMIc0Wwk9$@`W)7}h?)^1ln8<=Nkg*hiLE-ASh1
z#JxLb_z8neh`oAU$Z!1k%|-|OK|tqCp=r0pn?ez<JMjL%A9Ps)>6LkUH%{PifZBg>
zCU|i5;<{*hMp}gt1~raLsrM+_UiP#R8v9x7jZ=rGn%OV;+t0WesG8>QBy9_Y_t~Zl
zI^~ok8{%#h2=bz$9n<5{@-Uw~H_y|6Le!wL4de)@>mmY=4n`dYz0}{_)>{s=YFwOM
zEl#?tC;@{231|1ZA8T%#4%kgr<NrIAVV3z{mQQR4f;-v`U0kxg^_FJ7eQ^{t-SV8y
zXm3N_iycbP9)oj|ix20U7BkXB+=cW-_0%QaOj|zyF*)}*)4tJ#-W!cbMM=t4zCCQp
zgJtNsV6X8Jtht!{zR@NyDX4FCb%BJYp|$utVtWfdqZeU2EbI1(r&uwtRXrqtYFOzB
zmXBU42b3lAKw;WVR{Az13Sfx8QU76x+49QGBSXAKJ;z{sLe}vrhwve$A28WZ?T0jZ
zl40o%_w8W`%%vOGY}C#$guce=W?{is)$3f64ji=tEC9M5H4%Kq&iJkIu;J2SIqq_p
zET#ZpEg<6wtS&1kL~O^oePwi--?(*H8i>@5m)l!C{Y+^qt%3m}fK@pR4zx&~-%Tn4
z3K&qP%<@>;9FXw4t=Pm@W5{x9VwAO)YuI~E2Q3T!s;_x!IYW#neb^pe)?OB5PfB<}
z=Cd?cb3=Ixf<5(D=sJI!eC=YhU4t3sT0_3%9obKJoO19m`$zf=$Z#dp7+{b|Z2r6y
z59t+iJEcZ|3Uo|#`Ca5`Dm8`zLebx_L_eU{g%($mR%g(AUWX>81#~y-+@2DhAxvM6
zi6W)IZMj0jhJJwfS%`LLop4suSKKl+eB<7+(BjN;fZ|r}%^t`{Md5%XK38MZ+U45B
zJ9wx4WlMja@ZtmP6zl7Lujh~V_yii*S}FWl?=Y<?z$7t%iq8ao)mYO-TKv7)>JHXS
zw*82~hH+$dcE1D^C^9^n@|R9syt#gfU6_0MUUmp_YUkc}l70g#@{)V3^x;iyMfwnm
zYA3a_=MD~U15x7*SMra9r4;bx$!CXWCye}&Pl~tvtI~34_NX0=!BN6ix`(Ce5`EOg
z)b=P~s3cD-LZ6J7-vx!vMCHfi3s-ysvEc5wa^C*XQbliv&wm#^vna_)&kp-03=U|&
z2G75Ybua2_)XsEv-FxuMG7JUa1bo}yJFg74OUMK?s9fz0IW^%ct)`lUc>!S=v%>u+
zoEABdTY;uWk*%G0?uFLRoSTD0pxP2mCI1M=hH42X)y&61S)UJ(Cw}21G$v}NTbx#9
zAWe(#@gYCzR|HL~0gxM}f@k1Sxoc7WnuEFXD+aoYlZQ~Ft472{lV*0DXxFhkiB784
zj&g*o=d0(m1tOX>K3a&hh`bvh_95};V}22-4}OCI(8X7<5f}o0$w=mJ;a#j!Hv$Py
z_7_0g_Fu-G3Az{&ygJ&?9K>{cau|F1&QBwt{)yx}1EAXeTLI{RkxWjvn?h|I+p#2&
z<Xj1j$#p-G`2F7-7Oy@#HQ>u>pqhEp8MN@z@L(k~ZHWAb?ZPr(Py<vG%Rj#bSgrG3
z(usssK#OWmk;PvA?+tE$JN8_JmBy4s^Uiz%na6R@KrwAWV?aUsk75bJS4Gg~egJ*#
zSDa_`l~#l`k$j0c`QPsY@KkZ7vT<6i-TTsE48EmAeu}duN`zwnBLgy|<gd-bfC5FE
z{3M3~nsM_cog)H<z*Sx#&h+0(z$loBLw_SttNw(-vrb`+(~6Y+5`3PnF8*UFBq4_q
z<$DKVi-OB)H;g%mf&Rb7VFr(8X7%CSX2-g6$-JU;*(}_lTe2NK`NW8{pdurrEc>yl
zpAn6^FL**iwePkySaJh__2YenuX%leBxJK4tI^K2n?C0{PWFefIDAg{YtaHx8Cbgk
zdsu>nL=N`q@rz#zQ44E!C)~(q=KmZGlNtq1e?fp_VZgyJumhMN{?$z5>6l%n3#s1?
zcONTLrjo|MCINwc=QwdFg5)TOpogInYC>Nj1AH)$2X+Dz;1hO^?L^wXs&oCS_v$gt
z0G|`eTFC1&5aNX1G9muYlVY?Wo0|w)c7;t1HL+EJf!^Odlu=z|U;lZ)B1Q^<h3o~`
zToQEu8aX%m_jYv%btD5X+cG`h!a`eYYggyE2sbAeeSH5r9ISQ&{5QY@@|HBO7CDXV
zU4XN1TjLC6`7-@|FubU*x=o+JI2r<Vfs&ns_yqAMH?O%)%+Mvnvvy#W!G?&M0bl<8
zK|y}2sJZ5>wU-aRY~+ml^&eVVBHKFg1-zihJBIpv_JqBxY#0+G3kGV$kFDS}dQ@vK
zF-`tD4=;WvHDcFrSO}_D%dtATihK+NQo&u{)PMz&O*cNVRWvYg`Rpa;<06eN`3T#A
z$qc=d-i}#wgd`NgLSaVss6q(h%;bv<T7~#hipwiK2?OLNB<XB=lzx#_5GE#to4{h_
zPJ=w#$|Pq61S!!1nompJ2c=`dGOzI^|Ip3UgNHM(Q}VC>kgMr|IpWoowe<AD>D3H2
zcw(W@!Ka^kFyj0jT0Q%I=pd2O-pi=F6Lhj1JXJTR@DN2&`MHAHKQU39LKv^I;DF6$
znN+CVjZD>G$ezHFAOWeWbzNZvm)l*!VQDtFXz;5$bArquU!@tBiV*&}zkrn1^2H{H
z3$+Y|SPU3KGCPnFv;!=`e`8`IR4#pm9h&mi_VSW#ZS4ab2P%Ldu~y2q1_~rOL9f4a
zLZP}!Jx1I1s3F&zo8p=5Om9XD_qZllQWrFD$lp*VwfA`9>q1^C6UexQ%5e6Y?<%Bu
z;fqL%n`XzKoETaYyGDqIFg1%353Q}kGi56)L8$D`W`xBi4<69M!aMFB4L3wkS(uJO
z$LJsHtP6yIDZ=4TqWo0)1R)10M7dm}0FD|gWz2p`8bBE9>ai;MvEJ)#pt2M;9Oy!f
z0k@NmPFu~hus$|gFuJqrT|T=VWdPyi{?EqLT*h46fEWQ^IEF&#+q^G)E)Zw7oQ+>{
zUi_4~?LCh8S`aor&A<}ksb9}_fx}k?CAj|GQ&h$Z3GYUkG~N&brlmBYB9dz=K$#`Z
z1P@Y;q-JGfrc0xym-g^&pFz6z{~D?zl?j^U{<Iz(#g`)=x*;ySIsX+np-F)AGT>Ej
z0Y1O9(ZS!y&H;jmE`tqD2Bt*3uAUFhSLS#h*P96by>%_y<g==G#9Yj;(9=^z9(dwV
z9Z)7{GZjplq#dVIJTnc>%+G!Sl}XXznHOY;0Uq9Nw1#%VP5{Kl2yD(K@pgoso>~3U
zJ#h~aDkGDT{|Ek6fVz7^A1|K}TRIJ8SX+-iu-1^@>ysXKM*N#G^+rChC|m|6+rIvO
zJQd>SIUXHGN5=lot!Ka2C~rB}(eUpKAq-56J}_LVx41c`n=`d8%J&jRt2=QqQIfN$
z2JEMJz?MM0VAc{|sOlBzC;yCF_!QMYbajrEdo3MxoZxT7swhMX#0bN$e7QAL@dT*n
zcT`cZ^eg;HZ$_pqHShPaHsAO{N<u|QLd7MXZTa6X*W>APbHN*LgQMg1-PEdGxRGg0
z!8%}2MSq8;hg)R3$UG5<mR1S!1bU16g~Ptcm9r@)OB>;XDg!>0sMTPD5&DiPdIllP
zK&^aDWa-~~)8DeCukYcBo<zT0<#;8eSMWTlA~5RWS>NoJq_vp#oCyM;brmBD_Kc!t
z<>jugH&wZL5x}AO^%;!PV70LNos_y^p~=|HW9MwpKkJ}pTtACeZvK|Q^!|g&^IG#5
zO7p|#b9L?eQ3qK6cR<|8Pk#oCwIO%dof@PRRIs4P4&cV6AKNVvTN0a$msX75wSxEV
zn@rUy%@@2fJ=cQ?J>(M|FGl;Jnu@<!7ra^d(e`T6T7G~S99PzrNo&$unjbar=-5Oc
zfsy67s5NO#Kc-`K%H5P8r^E_knSOa}or+QZ-GAawZ`MYMLl-1$L6e{JWAjTzMjcL1
z{aW1cb_#gYo1;>YDDAGW2ww&;?j{a9pwpVx-5}e_%HXMtr8VDVV=lO!%uUi~gbrhc
zg(8`23e=y-$QS}exmcZq#$OM2XR5x+$?veJ?r*NNZz4GWcUq&sj|&MbU_T|oj~y^)
z0PiNvVFwaJp>>q-Z=$L)BKu6X>||wrg)!6af3HuIR$~oO(KtU%Wn^U1h^Q|e%|Y3F
z%Dh}Vvo0omUdgq4_H9vtPUC~h7}uXH4&u6C@uOQbQ^tEQk&H}<FZ+A3@x#4r3@_HA
zqG(OwS2<2-bH0=|`#4WHzTKabBs_NJDL#PB*(*aB%}Ep;(ccD60zE+?Ne2AN8FB#k
zDx5BZ6h7fT$0r>sQ{hjYN1Z`FcQ=ll(je}j{KJC%7gAlEXETwuzuro^QRcM{w8f^N
ziL%8e$ET;op7o8SFbhw45s^};4{W+#t2-Dl6y~N=vrd=J1v31Q=sLdHd^WuYAZTvJ
zbtRt)b<w>7(UvC9@SAQ|wj1MdV+^D;-pls)0WsZM*BUEDYAekM(n<5l`Jnm2y-nGm
ztbx?V02efQuA;0(icKgT8U1!iER)%Thd^=&AXIL6U3vww_VTLNMV>y$II*UpFZ0TK
zX8WH`5gk3b=G$Rwp(zU}TJ;-nt%JOPHcm{+W|nPmdW*-Rt$qYsgZ7IC!)NE$HsT7<
zeBg;xz$T`=nEC@HR{5v|u_N_z<!7k_0zj2KSEi!_LVN@Dh4yimuEcoO>&Lb0OCU|?
zLz0f<qLs(15}0Yo=o28*TnY;P3|prezp0V2g}fdx_xPDF1<?c(JTfiuV}JhnT+vGz
zf=>{VdbCDiV6iO@w@%eGM&C7ojUSr=?8sK-8KYg)pQ*`rOyP9Tz$FDJ$C0j^3cWfY
z@6Xg?51`Gz6mTWVE^_yH2hayc=ZW89TpnJoGH!Gn-`~C#DlAVxL-4yG*5YSBo8jOV
zsH0rCm_A3OqPRvWeS8BQ5mTvtNgVn$D+6XBGob?EO>csR$}+PNeUDps@D&vYye86?
z+ehi;rQ5@jxScJRJCRwY5nlAgpp~wt-zggEqK98k9RpL&%a+CkL57L;$U+9(vQ{Mp
zvB+l}K>;*!*PQr~5dRpMud*Vzsj7U~q_Wb^?qk+EVms`mFOBg|sKX33=+CIA$i|vQ
zQa#5jY1f;Spc#J`Hw!zhAGtn^{6wG}e4ph@7dL2*O6--g-Q{U_GLRyfpbUi@Mx$OK
z69&2MA}f9wX)si15G&J`fZMgMcOeL~;a*91cp^7Do13j=<??Jgs?_cT^SdNeMI$D;
z`x4_7{XQT#Ga{r@Uf0yv6`syQToO=eY`_xWfQ`6|CX%h(>nnNs{D}O+7LVkp$i(RW
zsrSD#Lm}fH$pfF*h+fJaq_7n<z23KxD~=lH3=lU|uucoeBGW`sg7o3;{BD!P7l~{Q
zY-M|^6XcxN*&~my05NaGB77BqmnPwE8lz;w$nYC?a#D&-{y=xEs?yW1qfBIqyTU#^
zFVF2Z(8Eh@>7tvBg<NNxv&s`t;Ew3s4^CvI6h$OWc)o@NolQuKLXHNk08jI4&tlB0
zo(4i#2jy|h_6n<XOJR@FORTdI_Zgw5z<l5wJJ9gGz54E0QY)hHOAA^i5Il%>8Xnw9
zH&&u6#Nb8kQ=J+aN&D<y0?-*Sf1X`B%0|})Zvfc?&<Oz2T_LNcT%6V9m9noU_~hA6
zY~Vk}5P9@}@`7qSr|@!HN8lPgE}9hYQLEC467X0ta4&Gq2uK^MM3J91R;j*fUetM)
z3j5IJP<i_khZbiAb>~lb*qH^$ucqO@0G{$D54LvVa(8(^_QCB}YAQv1e}IgPOkag@
z_YdzR>1aCLdIv^kW;q6YTr@9AdU|@|I+v5R)>cv5-6v@onYk*{;nj{nLyWaC(_op5
z*?KkC5L6Vb<<PIWE@m`zyu4SPAw>OgELGLjgz@Y$RGql0q(>h=Q?_)+u8imie~DZ3
z1t@?WQ{md$8gL@Z^X{O=&CTt2D+8B^sOEET)KIFRU+Z6yuCcMxllAWM3gq;*&*tWg
z70k3he*6HA=~#~E4t9l+1*=8|L!n6=#$CQhXpN1He{zPSB2z!k-d%O|d>ziD;o2T*
zag!=pvnceJI2n<YGcnn!v)^{RIxNY`qG2_R5r)z-ndOm3*EN*}_-E#C0PVO~KnwAH
zxR^fM9uDR>M(Z`I$jI55saT&VP)Qu<6C@Ld3?}m)Ej2YHCMJGb>&r_Dg7&Z1x!lP@
zWmJa0^+Ht_-R@mCCirFPosV9Vkmrhn4TU3&f}whECtjH<x)S0gPD?1<0Ifx3aUT%q
zEgPH7Shlpt{V`%sZgFu`2k3R7T8Y-h{&bo5)8k6j-@jk&L=5F6ko=kuKC;l3=0@QY
z$O*XJp4@nx>~x_*QO{iQWg)$t)cyXjam-!$_2%6d7eujje)F=PCoFeBFrj&)3cB11
zyOnqY4o#FUy3*J2#Cnb7F!*)h!Ec#ZUC3zfWZ!v%0s-EX&6~nlwinHR9~oNMH=zSz
zcV2GxWFMQEA&$dUr$vC>ZpQ1!vFUql4J015JUt#RHV9EhB&ZhT13YegeB97Vy;R2s
z#H!!y@i3)pg=nmzO+PXguwS#qqBWMM-@;=xA5JRhb(8da-uXy&xThXi;cS&@(cWHA
zt0JKqT1RDN<x9=DxVYP!n<kGNT)oZn-SOika&mIfoIftq$jHdNHcQ!~X2Qb4$4u-4
zy}i5kV6nWH%R|2{r_V1gioD4Za!CUOB?&qRU9wM<>!|yfJ98dkP=!wj>TN{4Y0Z@m
zrx{$FjehN|8a4W6+nYqkGBV2v8oFK?3g}+9UD#bj_O2^$6Z|!i0ze9Sk3&n3<@g`o
zf8Yo>8EVITD-~Ks#!$$l@(xVF_YLS5&sAG#o0;td;n3Wk?;`4Be;C<@`KB>bQ?CK8
z0=WKsgY!}5$H0}RGa}}hUsl~J@)X}DXZxp32NI`?oR6lg=FrLbEmp?}$f^sGr6h)v
zMOuLNPk4AZE$y1S>I-D#cJ23ETr|JoACrYW1sW4$V(_sE?lT-69c#P>+Wb(Q0Vi}l
z+hWnK)@kwjz|Wtk@uJdVvhea?zC#3S2ldCna?A46R2((a3=b_WZJS%Y^ARRze{b)b
zxK0LDz-i>8e&9~h{w8uWsjdh<AK9s;r#cFg2m(?V86}?g-HoMDxnH0St6)ocegeL#
zmI;rC?c-9_zT&S?tCU37zxlO)=7bp|5uRPeVtP4-QAMlz>+b3*c=rb1KVKBFTwzDo
zU?Y-7%HShb;k6C~9T`y|jq-z=s9<bzGMW`?NT)AduKv@b;1$)_d!8qc>GYiF<Yb$S
zuKxHE?HVA;f4<m3jkB1qKG^7uPEAeCL`aS}K0XG3VK{|9nlECstIK3(Br}AFt-8AU
zN4g;XM{)6R3PB#1V}ox5b!2!F7wZusLy275K)4$4iC09yeBb*a7>a4!q<wJb<UAhj
zkRc72zu5jcV8w}b{n`v0tSRVjv6uWqqO)scXM-zW6jJAUgH!R!^@f~+M%Jv5vU$u;
zS_$lAO$ma$^;sP)ZuGFzH(CS`H!&#)RSxlpI4MCPRlujHkU!BEn?QqJP>>8bf;raN
z-j1N>OI>(+YNtWwVsyArcdYW7FtD}Uu;a&IV#{%2+|V|x^4quHwkO%6xPnf9YmGXg
zfBrzD;Z~pCsh578EF>sC0QOtSC*Cg&E9UHcSLG*mM7JNjZi5rjIQ9}XE<XN?GECXd
zt}a{xaY%Orm7=^nEfcmfq)e|#Mpm}L^Nz<u1sL}`N6Z1fq$^AV1~rOjbN?99%ci&v
zHf3bRpawRb&&)51gyg7%qE{e@DX_EJ@xtiy7Zi}h^0dqP#sQV0qVy*j!_S{pX_+jE
z6nAPE6m^P$&H<B3Y`)ji!xiw_UV}IKCy=hL=k8d}#~<jn_Vz{Ra;PsEnXXA8FCOG;
ztQR2=Nb3t692{ULY@P#O<mvuwYcM%D{xu>Shxy()%CNo{D^qLVtluCb@p}tMy09k#
zB4S%mZf>qho&v=OH*N=rp!1<WGMYe%Y&@&jI3gxuXZzFL#Xhh3*js)5<p>)Ua`wRj
zmAsYtiK!_%CjWfy_wNn4Bp|Q-^B=F*v98S*8eQ?4#YQMychIs@z^}m^E(F7}_cwB)
zMsJ)T>2R<<HijNHhR~p#58h{wYY*EJsra#BMe33~<2_v*tc<ho*eAGNXOn{4rV293
zUxH{$fI~({&3Bm97l@Bk-#QYxOvS~;3ui@Tq0k<{&E|fYBc>+i<>h7Px6+o61`;7D
z2I;Rn?atR6LEi(XT=L<7EOCvvbAN;Onja4j@A?g;pwn02ElB6Xh1qH=vPOx|pWlW_
zBIfxf23^4k-`yRxsH&>2-cd72XfRI!HqVsu?+O9VD^^xjMGMvZ#8wR<WLe3BU$O$i
z=I-jKx5JotGZY~h^*);@QDjZ*?cr?}9IVp$3aB|g4fkP&gwtTLMZ)UuMN{LV;`Ehb
z;u5p0KEIkzw~*NUdw>OV`(nnb`R_JI>w3Q*002();%M1>rrfZcX~+E&m?LpuzQ*Qb
z^68DNXvYihI&7_59S9*+y8Pza8t~SpI(A+}1Xi&<fQAN2-zGD}{D7BI;XxW)1GH+y
z<m3wgLWYs?cK!K-O+Z~&S9f)^i~<(3k%GJgM@o$b{cLzN7#z{O+?%vKUTIrhT^-?s
zN*Tnkp}sA2314x2ZF!UIzp<8U-$yl~M6aUbevX?G6C1rxM(%Y1?&Nm|+$<N>M3dir
z!KF|&HXkToogRWsggy{#342>g;Qy24KB+=}qF_-v?2dhzt^D$EdqD$2TCR(WQwzkT
z0DzR<RhK<m=k+mGSg{P`dsv96CJEJ4wb>{PNY5XOoY%WBSmV)BM}Rby8Si$nN6<)Z
zYlP(AW`Iu4_YJ@%YinzXfKSTGV?Kx|qAs_$w-PyS;N6{_V;3PZGP0fb^onWdMLCW3
zp{Ei)5=dUBGHk9CNy96d`NBTf1Rl0hy4{1k3@sTWf((D#AGV`lQRG-;RMKET2+?uN
z@Vd!F{$(_JLSR}tUw5<MrgoBYC*TJw<QyX;2<c4XHXplM_Fisu1?o<0AU1GlB<~-t
zmZMBrv0p`m(Wl7+X&dhPlh0~K6$$6(p1-qICB?-Cj4tyHP6vBSO?P{fMVWy#&R|~!
zC%_k*J@4`q)7!C98N?f#n!3X&(9-;-p!ptP2>UCM(*Ah1r{fN+5tvU=5C`#>5WwL+
zt$Ugo#UTLob1Nc?Zh8j0nsE2fV#5!ZFNQ2Xj?e4lUf<IFEiRVc6Gu3Z>X&R5ChQ*-
z6(`@}5dm&T$5`0dDCFpPvFvq+K}thJM0CCV1J=^=1SE!~<fK_|51!b}fP?DAJAwP#
z`GGn#y%sJyQ0&hE7gGbc)9*D+-cMenC2L^S<1itQ>*M{h765d)i1*h+g7C=51i$p*
zn-y|%bBoFzX9ZrOB?-BxAE!y@OiDnGcm31HQuyrw*A1KoGg|$VC2vFaZt({Du{xRR
zzdg+Y6@-njyW?hL*NyuyQ?pCH?c?|f`*oK}NfDKeh^_Pi(sk&It_~dyB|8raAUBYh
zhoB-X)9++|48)9rA$$<6`eJNcWxZH$3wRGXANDW!GIxN4f6u@!YlU_|OfFd?-s`bw
z*KG}_c|R=u-d}ms@e+*mLyC5EcSJ$V^lWeP-5RrmQV_sjXLg1~o;<D&DNdLqu06aU
zkck}G_^sh|5n#>0hJ}-GZLPHVO-)T5Fa7bepkty2u-EhU9G=F)+`O$NrxprMV)&U)
zt#SXZRMR#k*Z%CS?`?<-rWi2k>n|apq37r4pE(o%uu(>IhlGb?zI^%Jikl<%htK=#
z>uc->4BFn_-sa|J1GL^$n+!tMcrq#~tg=lYOb-t!NlBB^()2Vm@bPiv`FeZi#AK(B
z57)P+4~1(V%3*iN<U$^(s38DWiZVqJIkxq6w6_~rjg$t<K4E!Sl8}%9qbw~g9gDLg
zFX4XD61p?aIPoQrY5w?9gB{>9?CeUFuD*|Ke0;=9T#fTxEf23Hx4`&dj#}!#jWrxB
zSkl;FG|<yg7FPh#2EzM*I>(Ds-eIb!LIHhFPZ;G#Fd)y7@<OQ#uMvFIURwHVz0FyA
zAqtKh;m<iEWSe)sci{7rCd}eIJ8T-!s*=7)yDgI7i3i%(_^K>XDRGB?FM4T27~baO
z#-5zKunqkRV}POdZ|XY^>O8RqE88Gs<bN`AT_9x93Z)e3WfKm_|DIF&!9&1gZG*OG
zI*HGfj%aF)|1(3O*4y~hBRqUbKvQXp<j|qia}0@;#|J^cuA5qKTm@EdV%+fHKp^0l
z6mH)6P*uD!y}V|%NmFE&b_p{Qa54}_Pulzwgc(q&Ydsn<bpYwle1QR)9~BaiS$98k
zpX)@2^c_YWiBFKKa@*JAahu}WKUM601Q*AYh5kqlJYNEkuScSoQx_P4qEr*;!wAU*
zL`nlQkSE3`c*V#R(c?m)QUtu4&Z<$ZygXNd83%O1e#-(<LDoOo-dirt>*xA3guy>w
z847wcz^RTLp=?767~m!NVsHFtB9;s|&_vE-F<@-f#AgEkMFRskCcrwve)B@zIf3ad
zQ+3cgFJE@#@dWToFsQsQ9i>uX)>LUv9uau{o5Jb?FG>Lmhi-*bqB1SZ!>wo?rFbVm
zy`5z3)X_6@5PDwu9a05<y3Uz@8@B?!IE2!?F^k8XD7A{=A{QPC^<TKyCMU0Zx=u_3
zUdo<#?)e}T-d588a0-U}4iRMt&CAo^s|N#`^5);j)c$rqNJ3YHcu*f*g_)mNlmc-w
zuoelE@}owNdA<dRx4X8VRt$3e6l;u472T&aIh?qpDvOR>;B9juw?jKK3!X2Z<$=iw
z05YbN^%Bs7h>D79V|)5+?9;|A4onVtG{rT^c0?V#_sQyHm?_LCKF?Dv9`7v(DVh?;
zwu+GTaIEC6bKb&n20jQ@=-BL%z{D+3T7KP<*Kf=oh8KXX`n9<=tj*HR`Nq=2DLGI0
zv_C9%I{hsO((3L$ZL5-taqGOJlIpxsP6`~kaTFCdoTSV>MMXEPI6(<A{|jtVP@a2_
z6;%d9LMH)c9Tmr;G`B;3jHX@tHcW<*$@^Z$`M4YhFK?LL?i;<-)C+vo5=1ftElxi`
z^9TB4oRv=vj4a6Ge8674xuDM!Eq~M<8_s)}PeJ9|Phd=sSMX+KdQBbXx;8dx?R}Y2
zXAo&D%BT4Rcv}5i7OwfJ$>@JY3pa~;(brxX_`pz+-r(8s5Sv<4cka=UeA2MI$2xC)
ztcTm5if12%&wj0yn1N#8PvQLD{W`(4!-~j}C6Dwq(y*xhD4L+{@)e;d;h(6~1r-11
zoJL^C@d^Yu<ajMG?yi4@`rg>U8XBWx_SOh0()gsYHszJ2(+Js)i|}!hw(d}t3bPW#
z@zo~=q1^fo$dXTHo$K|ExLlUxBj{p4#JO3`L)3{eCwwAB%BwDw?S-dfe!Ft(bfBkN
zQ;us6?+G0NNHrc1SfvZh|LI|@M#m+x0ICkR%!Ur+1j$#Ma+b+Wx>x?tH8|`eY1vxW
zS#(;bhxbH$Iey#n^#8}yR|ZrSHSJ1=bfZY;p^@%7bf<JTlF}t04bmmuB_g17NH<6c
zNOwth-^KgA-@W%YoW1v|nOV=w^Nhry_ofOW4?FmR2pQ_ZVH4eEjT<Z|z;tvu;4sxq
zUcGf;f#~M_26(516DjzLIDo$)lnlF<JhUf#$NRljI`*oyJ2#VmDNBX|>^gREQSoRe
zkIbP)iB+WETB|K%iy^#OzK~`m=v~LxaH9-ibWzxP^kUxF?-EW(p(OI<xrdTR)ySyB
zLN)vFoEuH1$%;`?&1#O>@QY6<pYp}`6ztnpKGGC;S){=J%{*PUl~iW>I`sY=4;D}J
zje74kT~V?6b9$u2(bhZV8GwA~Z!GoYkUQAzDf9eTepz@T{Aufy(2nlBk@$AzC)Uc0
zYalt@waD>V`p3GG`;zwXKr&W~4qCX#^3T|jl5>q-DN344S$`%5gjg65Kz_pJ2Vn0c
zL_rC&-txSn=2~sPEkX)rj2vFy|5Obiv!yGOr8CQK<MNruoZgc@x1_aGBS*XY%QWwK
zl>G7kaynkT%q`s`br;J^OeOnoCL$>%5~T9fJ_|-c$eYhU8QijGaLT<Yd(8^_F6=9j
zyR2ee2q-o$_h|t3<v*5n#RQ^YCY24fNPOoz%ze<tlc*-W*V{}#`u0h`UBRM}5*?~}
zm6Uu^(f(y{ZeHZS85kD`zNYY#gjB@8>S)%MwFDcip1djSLfY?OwLgQ^ws55fI5QY1
zKZWdb#s>U<vWzfRxNy=-D6B~jyB>TQ94K{s1L^?nj|>+j@BoB+eed#&Fn%f1&`6_A
z4<9z=*52i2qA%wU)mFm+j`lyuyvbL=dlS=FR4$C=6D}qUOGQOuwGVy?7MvM+#rh(e
z|6RCZ4Ue?Jo_P@(W{KywTLNNHV03<}KVSAa><MS_Qu;aXgdm49IkkHPgFN`;Xcc;M
zO_0&+?Rg6kR{h_fI0mH={v-BhfUemnJb?QHw=z|4NH4ZBqayeTW@=xp+gbi~M0e*n
zGd*Qf3klfMg`|!6(@>vYoq3JfmdFzJ3we9)qc|@Y8SiROT^nGtd<F-v+bCAasU!1c
zT7tyqNEUVj<GnZ3H7sCP@xy-58S2n%dZv||B=j*~$upa-4&Z_6wUQS7xVTK+n9|JF
z2lZoHNc1^ee$$8Dgp7=gX2G6;o`d4g)9P?~ph7@>hL@KY><OYgm87B3Dk+V+Z-((F
zoX^v0yL#fOMMTpt@xI`ym|zWM^JQc?3%1|2imsOB!J6y|E(-6k2&XuxJRl`1lIOc8
z*48)Fk0g6Ud}&19v=%2droNOBo(~=!D7r$7r8bf^CG-&dGOFP@`!exKr<(`aBBW?N
zti2;)l;KSg0d~L2k~+QWf2KBCS+35-y#!3vzz}ARVu)%dd?A6uTLsRTVO(Xiy*DM-
z_cyF&R2<1O4`8*WsBtzow&pZiug0G96yMg^@S{InRz#1nvJ$Qol(Q)cABnq+&8^{U
zpc5;${|mMpkY!r92+J?``z7eExqLp>IX)u5mNd3ac;Xp7{I8!^S5AsxN3I1pVM*6H
z_JcfW%NiUq6`Nz;Ue~V$bN1P%hF>D`%e6s<f~i9IYHV<>t7ksv`fo>9{r9w}H~7T^
zebkwoP*!@T<Yv1#_DF!Cw&F<0*Em<Y9Fd7;c49m!D=4m~u2-=X@0KW#L!Pd!G0}?i
z{<GK!_^A(n0d~wv18dzV2qz9oF^;tfgFB`muzCF=KYw#lDh~)uDN82n+{2WSB9;iU
zWW9+`IwX8`u=U#dXjOg0qMgjS*7*XZV7rholD<MbuDGw6t?qS}{5CSM`VNJ7;VgWX
z*)@u>mMQ9%Twd=4tBc5U!Xb?-aB4!bH=R4@u(5GO@^rCdnHnf5qomn+&JkXlNf+cY
zlT@KJ`la|)q;Z5JIBJ#@Zd5GFyxSJ{ZZnz4#UD8;1`=tU;S7O;lr>M4E^@s$RkPP`
zt#B+c{ub<E{6PxF0v&mR+F4;k$rtC0HYqWUHGf!q??u)b1#jZ=gG;!Lf-0=u?m}$7
z?F*cD7J29PS1UyF+YsgA@9r0u5pwz4>qe>JYR8=-ju-Qi8J|p_O_XdVW+1Ue26}=-
z?ci?C$otdW(pS-G!{Se~8dmLW6=NSPb~jen$Vd}4C3WD%zCaws&!z>_{dX;r_|<qs
zm8zNYUS^eKzJWxRQ^*byN^zX}QsN#6!BJnq|Mj{?qBB=Pq>{l!e7r}J@7Y{sfiHv+
zRpk==A?WX9TC8VW7?XkdZt!o<<XcXbY#lk`P;z87+?`BJ)kfI$QgNDvU1^YkC8*Zv
z@gE=fEZp%8W)-`PX;2^Drqi=H#hYC&`8cgTDSQl9x0|5SH!-bWb>c0sZEBygS)=7x
zk(CD&J6-9qEY=znZ<*`={bmvd{w^l&>**Or`u;^i(j}f89)1uxI?DaDh`P{WyvH{j
z!$T_c9RxD{otE^y>amA+vLt)@G%(N*a+zSUZC~X64Uc9#dlG}BU!Wx8)eGIWex_pU
z*;bk5a}Gf%1RHBr=T8hI!a?Mb3I`5_ur3qG#sHh&kEJW-ApsL*hLftM0V$*}Z^5&Q
ze{z5q!%Lm`UXirC&zjJUt<yu;XCg42D_S8}Mvp&%r&+U()czhhh@QnFkC35#hAS|I
zRqbz<MmVS>pg>lm`H6#nuBO`P2@93L8NjnfsMcIAQP~?rj6@ZSgOnwgQl<s%f8!%0
zsaZP-#Y6w3=Q4nj5xvg6o77Yjm@kf1C<ih~?dLVbtfpk%+iN%e=snF&qn+9(juL$T
z?_XHTJ?r{;o19h-jz30o6tymtm_!WWwxRXwRiqN-pQkDnlw!e`C47m3Brtd$=G5!c
z#RETv##!tzKqI9ZlGKC)4rHcUyO3pvCGxMUdBvMmdVVPB{5So%#dA;3lGUdoTbA-!
z!Mryyc0>_+nuV+2o~RV~wE8oi3@dl`>BVMP!HzkfgCqB$@npe%OXyQ-`7}GE5GG;}
zAKv^pPq}C+85d7;6Wk1drcn?Y6gor*-9R?g@vJclHX+u&|IIkV5FuU0NJn2jD5e?@
z*dGYQk*JTCGias)bjd1vGK*mM7`lxbmPmWZXzU5lfq#J@L7~qAQsy)e{_S8Jh4F+L
zPC?OWm4=cedmcHMW9$L=|MxVV2uKQz24;W8r6A%hWRR)Qfk+aMq)u)I*;suKP;ARx
zR5YmRAy6@`{8M?x=Dmd0dC57)v&oO}mPvEt4gR}DNll;*u$%<kOd{n}<C04|T?S(c
zUM`0`*Go;P{C}V084U{fB-Q_v6jH^!`2Qa){t62egf14BVYZ|tH6;MJCKHrw3num}
z6_mubn~9diwtc<1>-IAH)>1sR320UM<hZI{#VKv*LmrB~Lw>J9@3I^9&=QyF9}KJ+
zz#B_Cpj`Wvj#2J!LUVj2%P^NbHp4y}JPA2vsa4_I1<Cg<bMJ6CFR9Hi4Su}^q6&EC
zlo0f%mdc|<gB|*EL~<78hMqY8C5pH7xMMl@AmHDmPbt+xI(-y!THhyFX00hux11&e
z-|_o8Ke8WN-(~o(xoB+fHVrp?n~g=^*Gac7URe^sl`9l5(ecTpe?BEK|4<_G)2TSR
zm1xyI2)!NVS*i$@3vZOqoa|j@C;EP|vCQa!zTj?||791a<cGx}Uu(*HI}z(CZKHri
zvnAa$u7jd8$P=Y$JosKpzT57#hzP;~IEkyFsy7;ruh8avQ#lfHpPDjWSM3@6uF-z`
zA?)P%sh0J<cJ_>oz&;+k@dJLV528cDK6OL+0q=G=e`EHmiR=6t#TC*YG6Ie{vSn0m
zR6z_*3L6NF4-!JcNKj2a4(#bK-%t6Df^NSqKFz32!_1GOE%`J)ZLP0lqj)S=9S<0-
zj*xnvV<=o21UMZpF+Tlk6uh4*Xp}NxY&k_d5L~tyj^0}Ko%HB3N#l|q3veUXKGg)}
zP?6m{kfJ!f(_Z17A5G{^dSgA^LY;2~WoIZn2X?iSIwmm$)$5NhKIa0e_cD)D@QJn?
ziHVL-UtEL_DXtmdiA8x9EybS;+<xokN4YFZk6ykA*Vjb1@l^#?o<>O98B*)$3|PC*
z+tLPzo}`GlXEk~5#5X5=AQic5&iRXzue+9naeLGF#Acl#Di2xhL?EaR7=HK1X6lEI
zxBuBlK|l^${hdC394=ZY%$WjJD!<jQnTGu;ttn{@;>Jt_IzybqH(Oa*UORk0BqMg2
zV%ctaQGVXnp47$LdCC#!&2t;C%8raijY?FHd{s}p6khV4H2$>bXG%l?`Tg!{Gc`ia
zFK&Rnyg`uS@X8J=S19Cjq?#lEXOgMVRs9|ouMUu6_8P4dsYR|{pDGS%eJU-sY5G9Q
zQ*oC$T|s@D=Ul)1^WSf;4FtDi^%)y^<3qIfW128IoC*NF7+1-se9^byv)#+U64(~W
zv^3#1YKl+P9FnM1o<vIyDHJ<aWSw^$b?S9hD+e$mAIIDU&t<X0KB?cWBR_=CI8d`Q
z*1#KM)*EQK-N#XRG>sX&weD+l=D8?tcQLZhsi956Gy{M~z6lfoZP}ZZr=6bfZPi)+
zOQ1V&-Zvo+Q;kO3Y(3J$v#cy6Ma&c-9yLhybBY0Q{4|1{<{0d{1e)FoPZgj(j}})o
zrY#@ANL}*hMAdh<t0libhe#-u$D=vLtGuEY=BW{9{K{!(V~0^^iUr1Kk>y&P4W)<<
z4fxq<D2mC0MLf$2f-T!^&a@RTAh=hNu~5yBwMdAtvgaW8dM@;f@~frTZ*A}vrzOHs
z)5!f%;``b#!0?z;mPcSJ%wv*_OIuXQHGCvtGZ;DuQwsP60I4~1=SC#gjXA3~h_Nt%
z9n|5r{dx2B<*z1I^7jf{7Wu-p`k5D9$F7@YrAdjrzQ1f-!X7Yl7tD<=6b%PFWvAY}
zZMZ+n)Ym>2KLJNEFZwG=3<HFa-!aPMl%|9U1T4%TVEGbge{FKhrgEeCyhYQ)ZNI6$
z8-u<Uh{>*-jnBi;S}mXMDEbZgxi|HGO`;aR4IIgr+DI;%ogonH{Kg{*pJAh`{og3~
zT|shEMick9cPVEh`6evYTcNN}sG5XoE8EooE2ONk^dsP~zk3VQ61m3=H`xE>cUr#p
zVUEWv#~U)>53WnjZ0uXy%jI5aYHTj)c^&G>3Dt%Bij={@l4+KvdO1)qijz$P84f_l
z$${%KX8DSac-&f-5G4c43R4>nL;0?~-ZnzuYx`+O4p_**l-^}->%{_PvgC)CUvac<
z2o=4}Pt@z%Z%hB_{Y&GYC$~O+v;#KW&^)JRnAvKh(W)JdNZ4uH>zxYCR}$)({E@Mz
ze{Btg;x3zZRFWI=Q@D8evRwHtC`$0E2W@&3<!8F?jW<2QTm;FKhquiKqb>U#X#MmB
zQ}McDRXe4>m5qu|BsDjhcDD}Wfkci#%TrZEZw0j#Ox=}L?9S7EW_18o;^tEM(yNyN
zca0W91k1joK`4uhIb-bs#ve+I{CR6_kQ;GiXEFAb-{05oS5=PMlXe19|LsDdd!^pj
z%ttz+&m1*sz*!g<8Fgr&_Z_;<_&%LgE&s5eHu+tOZI#1%Wgh6MbxRN}c-1nwvn%pp
z=X_E5z5k(zegEn_qw$|U1kFwawwp<^0KYSd3KKdqO(x0{$OQ~owdB&rMPE9;#2y9~
z9p}dj9$ZXT!4nQXMpg))&uJT7Qam1>wnoGa<%_yn5V^XFs&=H`rv}=$oZJ)IXs8-2
zvfs};eu10ui0QA^q2rs`JNz2p{lM|Q`>2~beDJI%h+D*;r{<|@rSO@Z1Z1b=h3Y*8
zbC#Q?Jr)HXy4@F!y`r{?+MIw!81Cc2#<{0q+oaUUByNXO6rEe@|F|%>5nX&0g|e!1
zR{gD%hvgV*N=Z9$Ic?9{)FyKlx6yZ79s2nFnm4ng9Zx1Fo7s)>25-S%m|U$o)gahU
z2RnM3Mraj5MUXx8dN?dF<g(JX9G4GE&M}>93Gj%un|-eLxog<j-3)8*n~IlhZ=DA^
zo(C=SfHK4)Vwu=>?IzA7?D#0(FZ!&|w?E$ab11Pf!RMeHXZpJ1jTyRDTo-j<ojp*)
zIPkYM@pDZ1mC>Yp)RD(-!2d({?K*4mqIX~{q@PJQz^nY`)!_GXYbR9(7J5j69F)uH
zfmuDazXkR%hDXW#r2DPBlv10A`RUZdSpV8KeSWFwyUwflkR{KJc7N*4sDCyKCRbG1
zceUROZYepirkiwH0VB=@Q1wxcv=dzS$;*AEopXM8`5&ppK4Cb(@YU_RGat-UNjLXN
z_XZVFTTKHsH535EQkH*{XAvJW-y%+r*E2pOMo8S~S9&?tUW4+n$qxDbkzZoFmSKo#
ztVL?9AbWt#Zu8BpfN2_K<esASqtZHF0li>Mz9GEjsq(zV=%4T?<CJyMDOO52Rhz8r
z22ayiAQ{&tC`7SrL&aW?0f?>@p=~;U&#wLe*(oTG4l@HaV6Ya)>~L5OlP&GNG_IL_
zU%Ot=2zmQd(;y_j!I)tX5cx7P5?)eMt@kQQiTx}b-rm<turq-KcJ139@8jK^pUquW
zvzKmHF9W?WIqft%8)@(SZBJ7OZ5vQ+X+EQ1m%@g+{mO*lrClp3yPy=5K8Da6(ZS~k
zXi}MzLlR$4Dmn}kO*B8WI4}Bg65xQDgOJZ#E?)Ndwcp$_c2zvHw+3zj>+JT*+Qaui
z1jv1{$&wV<Ua7zQ31{{RxVxZy%5)zEI9>V#z*Gi=bRmo7#B$C_i-8_lpPa9}C9SKG
zq|rtmIv(S)BPM!B4{VCQE4KHC(sr!vZ}PmK`gZC1i}n<#fO=PfeQ(t4{2?DUpQnv&
ztIRNAh()UO_S$&aD~i{u7IR+(ey%P9?F@q@VWuuy=HW^?jrOO%w(BiRVSmsl$N5Ae
zY?`x)5M6>0GI$*8KDc0}v^Jz!=G8d3SD64~)XJ6)3O%1|!Bwr{RYNRObE=2FN3trd
z-D^wQLYd@=5)70WD5t4jtKv7m#|UV)2Xa~Kh;E#1>{&-Ye?nB<9uE56$lqoyowE)!
z52?EzbSuH`voZZS-(R{+<z5C1VpMrCY%MPDUsZNi7m>dWq|hW(pEU^bfAa#Vwz5bQ
z69G*R66=Kw2R@7lec*`@(<4wJ%CRx8XV10|qV9T$zkQwmCh|hd2XG0O{U+~wB2F@6
zrCV6afi7U>!N?+2gf@(d1~67d^qikd<8HAA@JhYdwObW`gfoJBe7RUkzFfV2^Ly3h
zJU@OxevZq<<T5v0{AC}zUlmXw62%jsUyoPI#dLQ8sM<tV%5-dGd{VUeYAB3GdtZd~
zM2siQqMYl`=@$G%3?{wuG+`}f<=TV1id7SHbi2|{JkVC6_>LsBJZl-ckS%v_M?DX!
zhl3e(^l|n1``FCy4?HUFqf&ipXhq82;fE0a%}$7#LVdm~yjM-{#q%mm>R!+C-;=|U
zHpek&=;OSIbmg`nP!p=b!#DPJ?VS~S)4^WC{iw&I3bRtK`VjE`&g^CE_Dcmv&CFtF
z>x*A_lhYd?dkK@}(u~`5&ci2_eVCV*-2Sd9nOWWo`zytIK_0kC(~I`CPpofZXO6#}
zvVO_6oUmF9T&5d6;5BvGcCKw6G2{}x;Or0e^hXL2A^iH}x8@t^meOeP#OG)*WF)}q
zESFQXz1-w;^he11iQAN;g?=D+k&uS44FCBCR-T`~8MW0C?X!{f`g4iuR5E<sP9xbq
zh;yvuP@advL4ZH++;zh&ap&g9(=fx@+EiDT0Yrn!F)?Cn?$o0g1wyMb7GM4;aCQmW
zNY;#DE$_J;c;F`4<F(yuNBV|qFeUdSZe7VAn>F|CN@S&+i^4)BGw)#B|2R#gm?izT
z)1k-_29I!VJg@4N`~9O_ySU4H>YOM!8hzenrma4ffJ@y*S50!q%Vtrm>EC8osXiMC
zW_$m+7G{?MR6<Z)K6Ejb^D|4d)p~@jTOUvBFbWyJrjI!j<k!(=eo>;2@8Uh6bAOpu
z)Sz5`?;Oh5<aX-}1X}`Z&K`iLzUL&A$FMe66%*jvI6la^!ys;Xy4@2c)iJ`=5+0+@
zpFUb_9?@-_;==t>F6P7Xc&wKizik+2R=(}3T8T#g#p&pwUZgJjy5CpPnlG}P_E|-Q
zqP*L8oGbNTvZHmIjVIw>e(`M$Q3gsb$;*6LP9?>ii9F3LDD=Ch+7-|sv3Z=<{+(Z>
zS1vPZYp%-dCU?YPVX|FaTWr^;W?6|!Puj3<e|Ds+F!MnNSD(XpzAWVpKc>jVC(qJ8
z!W(@J`NbOJkmaWZlSXaQG)$-_?jA$ABi)-Dn_iT)I)8X0WnQ%8KW~+A5|7<7bm-dC
zJ<~{D`10IX12wR4y(g|5r=9J&5?SM8O!e3fIqJR{nB53SHuK-}&yQ2F_&wbZ{%$Y|
zWC@z2@xfL_GuVw-_+8?gK!DOaa_=C70m%DJH$Gsx+q%-%zS7Z9&z}21##ysAM}kq7
zq+=(3GilW5H(4`%oQMe=df%e|%;k}o&{p}dADJAn4nt0qZMMw@>qD5^E3TL{N!963
z-jTLjdE1A15!+k`{c>c<`*r@+ZK?y^g?iSe;;&q%+__v{e1EKQpsf(?W5H@WtUFJ*
z_Bb@N-fgVE&^M;pSZSg-NMxW)+q_nYY%;;t`?2BH$#<s!>=H!`-LnQ4&K0wL7`la_
z9s`d6;ccnQBKJ0*J^hC{NrIu)`kFV45Uv&-qx5YpES%#T-13!8IidOu0SEe<^OTR%
zDrBmP4UYG&%M{a5wjcM(EF9+S#0Ps9{~YnUKxQ{%%c4Txw2R1#e4i*#hsRT`#8^}G
zw*FJ&*QZK26@f}o?%4LU3hs&iGh0?cOLmvQFCj)oiYrR?Fm5jUZQpL6GD=#kW%se=
z#_64~8KuDkt#!G8tN@I^Wk9P*kp0Be9N0Nuq;KuApga<ko-8GNbLiec{j)>+o=d?!
zZ=Da%t?)#$=-rgO##J;RYd`8KeJstbf#iBdxb{Z#7<MTOA!3U8Ot}rN5mznHq*CUE
zqY{w(IMc5fT_kwd{an!oSC^GtN!IY2Hr_bh2ZOyH=#9}&NouS{5Q{lSGu-9iB){=k
zDly-rv8a(z4gArT^#Q)^(Q<9|(sitb?dl{(PRD<Xi@nW3MpnbwKF@&&$)}9}>=&cQ
zy1?!Jif@a&ch)nM<tNE?6QOw*NL5Za{hP`|+jH|l8<WA{a6s2Qk@qBUfNV0=jfBsl
zO^1^&6$CS0wDF6URsRGX2%Iq}J~ltvic^K}b@*JZT>r3h%=G@&0DF)GguC8}H)+E^
zxT<2?H(L-%4bA$ErG1@(u?dGPKiv<BUMgJh`l@(mRexr!3ui{{nWvqG0}u?jdcUk(
z3m_9I*-wJjF5RkFR$^?R^SNvTL?(Dx3(1+mHL!?<7+?_hU5*C5OP=!S2r`j!G)W>i
z5f=L*cNh!)DI_3?g%U$vE?;%P#q?nRV7f)3DRTXLxRX(?t8<1XO)I%HC37MO!~O|K
z<*OkTQj~ufdpjSk#{HVsj7P=FRHqZ%*y{>`57gwlz@FaAO$Zbits~=gxh|8g2nI}g
z;2rWbJ6YjEJxI9T#~UQbHDI+A;b<vgyhv%>MC0?l12|OQvNxR$T^!ytNBzG5%q8o^
zha;OWqZ@$frf5x~syO#_-fEZDXjLd-yufU*?Uw3RfjvFw`Zo_{_mmbAcy#F$_I8~V
z+8#y!tNHgI7y#Y2#7hT*fo0x?qo$c*O}UO2ixe}5^jb%#ztYe`IJIm$dGwk727HRL
z%3GZ`8qR*xrDuCxpDVys#dEuBLvFc~Vay&_&Xf7?5k?CaJTvrP`vj9eQ<caqIimcJ
znF3Mj9zT3L+Z9Dl(yYe;`nK^asD?p`zF>g^?wGNzx3uE?(}&)U?dkBnZdXR2jsAKK
z3Gycfkx(;z>@#_oydK&Pz<Ra3E1i`Y<}4D{lDfML%O9Viy6k*iTs3ZkNsC{9)LXUw
z?bP4we0*PmWG^&lb<ifVUG>^e#}2q1=a)qw0x(f%SL{P9%?@IJ;30eR=@~T~s*i0g
zd-*<n0~G}n49M*tAH930<pSVETEHav)uVm+je-;C*&J+V+3?)qeUOirP1n`(@77n-
z2<BOub02cL#J;GjRnnv>&#}x*bJbWJ75URG<U5hw*ZpcwDmbcq!NaF%%wqrAbg+tB
zP3Sxb)7h&NZYx^T(dKgmIb`oeA&@zG9tG?#&6i)-#Gt`f1_#ZnDo>SkQkh<IyrJ;+
z@G;|QcKpHa1|QjTE3iv)hJE)X&h>KM9;PgnMtzC0-_31Wr6c>fe|PlBM=Aj>g_i^1
zjAplXSDG+A2^Y5?^KUcL!?O1ULf+j;8cjRH+oXF*@+0;t&s}{=10|Itm@1N*U`*#_
zJ$cewgg^f=wdCjS>jpsUyIN{N3`Af<Qjj1h1nlY$-|>Cp%Y{^c3|iYuy$1Q}O+Xb=
zv&goYGLqK}crEOe#}&g-(IKYlrv#L*tVsQx*DGs7L#jfHSbeMSzjE_kUB4abDf#ht
zgilfH<Kw;qsgG@$_vDMVM~hKnX1f5KcfL#PF~Gvee6E5+XM_~r35)Ej4}aWGTDIGW
zqTYx)^kzG1cuRFGMT!m&fEZ8^keKt%jfA~uCNe9j31kKsP|<;=PtF_nV#0bs@2nNS
z)Ki}rUu^FVB<sQ#{G?g7<j<(lQNFkRxa-Ll%9SF$n+;Gw!viIC#NUt(cS|cZCkHT_
z<ZfD}$|<s$Pfe3(*UIkEf%4e%%VM5m5o=M^8@(L<f|Gx(<C=m}ny%LN@dmISsm_ok
z-#RdMO$>13d5&>&OYKKkNCQp;`~1iv%R_(jk?ALTH7lrTx5!4D=a;@B`oF-vcvrT9
z7&w6xCUct}3$C<+pWPMKKzi)r8AYXs;D9I;<bfy%s-g2S>Pw%t)+e!xo`I<Uf!6cH
zB$M}i`}_73^4CpFaiE$KIJ5c4#V>LuSpknS8Q{G@3k?;W5@J@sNN6N{@>5t*A-^>4
zFHUNgkJ;f=!0#11%qg>4kvZ(W*BNa`*Edi`jU}CCxj*edDCK|O@tAMgbR<x9a~|DW
zOLFBWi2&6UqR_ebAsx<em~-};m5+!RfcRhj*_d~17jn6@{a_it!4&IsJ9wc7x(qy=
zW`&qP00Ghe%`NP)LhL5x!?NvmUK<huS#bz`xox;rltc5jy{~ites`~33CL)~ApOr&
zf?Wo~U1C4XLL)#%$w*&b9&YvWXVC2HrEyk1i&Y718ZlKaQVV|{`=I3^g8kg9Ug6t6
zVxdXiR9#o|(C3s(59$2j(!!$kLPIrSE?!Y!ldj)W5o)VDq2~e1Q-A=d`YI1fVVywk
z5B@wDGFmXK28KO|uq>~6cHXT~%LPE6B%Sd5xP!Xo=~MlNMQ1sKCHk{Dilk#)l*sZ|
z_iqPbIj*n&nLSI<iUYOT?<%{?1&fW$60Qd`>RXhkFv}ic)#KC2S8hjk70I4Qg~Q2=
zod>aeX$Hm(B!yF|8acSx>KKuowRGh`-B>co)1JLuXg$O6%hSY`50G4bnaQJuNPddB
zcoVl*FjE=Z$lP2(H>RlBc=Y*o71^QB<;2)6o&`Z<B<#7evhTaG`*$!vKtTr?CQ>e9
zh4;N+KF7}fmJyfTpMT5k^UWd|MthAR9EkB&@NM%y&mGZQjHmzQdW^kwx8=9pDEn|K
z)wCbzhWWv8XVIkbsAyPRRTegdBbGtGG}&p9i;gW(IXw!i+tQT_y%N+{0G+YZpBd88
zS;xIoVl|IY!0in0_jujqJF?OqVDqRMQc8ZOg%-}EPjTK6tceQOwfU?3&M|;@4I}^d
z@cZ^H62Fea9fR2hympi;-INjhUzK@=7f>2mQSZ4yo*!^qHcQcJgG`@3^{(kPAw5hr
zjcgepQBA(2w$ky2A2X<N`4d#61jC+i!3wBP(sB#+yux4P=_C6wncKaboX|+Ws^Ja#
z)r<K1RL$t<feN8G`IIBHD%H>d6#qXZZ-!aTwLy&|K1$YEhhVXFJ8>6RdYHDNrZTdS
z$J!4agQg27wCD({lnK84LSK^)S|=lYUkZAam)x$+hf6%ZUQ?>!;WPFq5_w{k!RcVk
zD?5@x+^3!<tEBbLe9o@@@92ZE`92-Gm=3zFJ7WH)c3N&T&>?-<e^7?6g!-Of^GcUb
zwo`B952AHB(?am$&dCf%atzShc|1u6?J_XAE_xNxYMgpr*;TO(0=*ijDur9`TMjSK
z(8g3u)_gj&660|kC~1(m400Dt-jGkI!NWO**&(evrmV&PBDi5Yo{IPj!<<y9d>!eg
z$N#u=vG3RxI5GqJ7bqFD9pI2C6YJmtPAC#{)xzxfU4~;j%?5QkAkk#Ed|;%Ek<fvU
zXS|;n3ax<p`*ep<mTUDrq}WW-2U{Swt^_N|JW?I$w^WOBfzZ?tX*<>%i@1X+EQiQ~
zMGZZ|UQ9vDnU_k0DfvxG0%5u#R;qeJ5^FA61qwZ+5$^8>934ye^tsKmVv$JGlh=AY
zq6N0(w|6C}Q<q6k+Bv$Y*L!hSALmVI$3v1%YJx^@zN;{<rjs!y=%l_6<G@O0{Egh7
zam}ea!pIX5+*f1{gisEh(8AL>?6Th}<`hGtjslMUz8@Q(6dp$#R0|h;k?{xsxBz_&
z2mkQDO4#=<w8ml1s1^+AqC<qcA%qIPIO}rTtF#S}m8!_OSI((fwUoa%IS`;UM!!!7
zFDmr~+#dlks7m1eZ?*E17-*dFcvjV3e<<`}^-q_y?rjsnC`-o)yUulkofo`h5#i{Z
zsf<y%qTc-%?kkJ?9BqjvBMee+R@2{E$89{Fw~J^QR~*8;@|Vqj0Wx?X#W15^lV_7V
zEG~0DIb;~NfwW4aUV{i;PW}BFZ0~J{(nbH<y`n$_YuWiJhu`g8ABIW9)lW@p8)a`i
z*9jVFd83ZL4A&}e{GDvzKIs{9v$nfP0BZuGyx-{SE$Ew~LkJ1EOw-q*(iX1_@ZwZM
z52m)Mze8X6#;37H_UO?J0FLAhtRVcN+B=9qAdT=4Jw?6!{RXi|j-5n2Nhm$4)xq0u
zyjEeWbEuJU^n+|QqEzIp<Z!PUya1l_f<(~Irt7lx(qA=+pYmyz8gG8Q!Nm=3@>(!Y
zS}sBxvZR-XgEK2`&3Tp1KFVi)yS{k8fryt3YrIP;`2I3T`>4`}=^huejP*JF`F>(q
z+TX?T^`~Eyb=Ik}5MvSp<ca)Ys5wHU98CBSgHD_KWQbnu{rX#mS}ZTbS7pZ}f%<PY
z!c9<qN}O8RbT_W8KSCtl#`#S_40xgwKk~>X(1GlD$4j9}MS;uq`YRx>;ngeB`cG4(
zjal)t()b6ZwF5ReKFchnGAekn^!pViVP{$dN)eDP51zGO0&QFzgvL8@C-IR9y$7V=
zZr{X`$Qgvb>&35}ly_0+R34{;w><WnJk|+61=5oy4ZcUahXqy&nZAj$o~u!&#w3~z
zy9TFbWCTSl62^KYnGq%A{^Hi|H^g_*L!&yvHlCNCGS!Moi9*P*8Y!7*JT7jT!;}n=
zl?IE4#WxaXI{tMLpml}A${Em7YYW<nNitAYGj-@RNAHfAElImr+f!JuP||HU^`r#(
zQbL^3sg%bjCICq$F+M(Ch8ic8&4r}C8@7OrfVo~$kh%F+A5GxEnIc2?Sd6*cPza_F
zN1sUKryMk2=5MAO0`w3sWfPNCz>LBq;f>4AW??be;1!RTdGRz5NWv@F0T(;`CRCO$
z#Mc?+Y!&W)AHD>?XEP}&rJw_!7W@{|{%m1kK~#g)umvlXE$vk|H_OvUof-!$+!h8F
z6X8(S>+8Nxo~Q*cKu99$DM<Rxc6&4{J3D&~N6JuQh}nc-pe=ZSVa>?hn50%=fQ1P+
zHA%qcrYKhMZO)x}R0%sOK3=jZ;DRhPx!HlfXR4~I;Fph}ZcW4@(H6IhKzWFzXp?ez
zLp-(+_8$rP@l2PFUN54Mt^&8RekZ*K*ap~nDnz8D{(Dnpes`A^=H}rQ9CFUG>B<_X
z7d)e&8=1}rHCK>(m53dVUg%7}!P!oPSeVv&&7<KeQk`{Aa-i2MywitZDbqXuQqz7j
z$-Pe(R5uh-`aT2=EWMhV+S=+WOn#HgW@H0()Un(ROUDm<E2E=UN0)<`#Ny;9Ke+m7
z!Vk1Ze)$fE86zw>vFd}o-J3^Tc>=wz-IFykHjQjK+N101H}-4h_$eLjQR=W^tfKM^
zpdVahQfH_5%a>8Wk6T+?|7y={_}jp+nJPVA%gN;1N|l&?13j)V8fJ7wmnHdrxOHt1
zt1p4uv!29p>3lPjIRQsh^W&$H?W%nHzqk(;M-D^ql=%TT$WU^}A4%d!&>~|Kj&y#0
zegHnV`QEs=yVu?8>;L>WT=9J2cSFWeX0%Y7Eix{%pUc89)PS@YkMTOB<75_3RWe%-
zye<31%`L**t^p<)gieu!*GO;K*}>}Yx$S9qdh+J5-nkAp6y0xH8P%t|o&J3PvEfrZ
z^xxWPeM<rHr!4}8lW_*6h+iCE{JJSO1Enn~<uA1j&GtZvggD^uGLe%T|M{weghBHC
z#}B|z#YQ%7G&zlxCH}FhmD)q7xIooSvnk)GlBBAlrB6E)Ea74Bmej{`BH3sEA68EL
zgWnp$-H%eHh>T-nJP=Mp(bE{f$7yQ1Z4M@bje2}}IXXNH9~fc9V>*$Z*?JWdpy%Xp
zfshpkzuD?1#b5HT$jJSxg7U{pWoFaw^2zrLb=oj|w-k-XV-(rF^!`8*mkcZm;9Pw(
zY}Hj%9GRZxqNSzf;^JatjOsD%Xt!~$#=|U+lFXm7zvNS#UnVr#X+JD(w4$z3@H+Zq
zMy6LiKN4j~pY5=4d@XMA2iJH1>zYg3i>g<4UqE0RKRY+~9&p3@<0xFWhWP*u1-xl$
zdKyqxD=RA#6B2&u9gi)1v@-nlW$=1amiX(1_Jx1CtM|XrFoGr35d9)yr^?+_zur*n
zbeiDLsJir4ufj&uC$v!8>@2aLs;vm9+BI?)7R}X1=<<UCOeOPVb7^$n=?eK>-k+z}
z>eav3tG1oqU{DX(pR(C3*Nn{(1QZp9b?y6cS)FnWC;1yg?15HIhkLe*>lWNEfzHu5
z{ykc1>3sI_rwcp+S?W?{@~;DrB@_r8Kj}8ODfp}E^n~h`t-OS$hgi^6s)5T6s?vH2
z@~f5gn0~l~0B2vy@*fG9pj40|*5@q3`zkvihDhKsCgi<fMH)wL9W{PR2K>0sx5mej
zZYm_h+1?s?2Dqfeur>uoT~+bGIJH#n2xw>H7w0=B;#DSL3#~*tP@=80IjRzyb<D}n
zS<jR#{pKS*i$X+jG^kl@-GG}o2Rc7HnG<RDqxe7jsCuI_TAixPYNcJoh%MfYILC}-
zd#-q8=#(h!H*^OVc}~F$qO`tC8pQ8lB`RXk#NiDZVo85mK=Cj;UYTZaO%5z=ap7{b
zR8eAp6A4p{(#K17!Rcc{R#0g6y;}WLHd!U&OOTu{O{{|3SsTf+(^Q}C=RFa)UHuhj
zP`@x<cGLfO3^E-%9M%@eQDKv)#*~kby7CfH8PACP<CFX-^vesNR6$EAq&?Jcx|Fcl
z$7+aDY^Ad0kA3wiGdcAlz`*J7*2i&>$zscZx{o1Uo-rwqIX6CN=KY3}Qt0of5X+Mg
zq)H-f{aX6kxMH9`G#4?(>}}>yQDZURwr;i1X-e-}(ML&bu~Ub{TugFM|9IIAH^#8z
z-J{yeh~hJ>mW{X}wUw;CHvv*+?O)aMD^y6=cwsB!T7KTjHjk)hsWl`uJ{P}-_~vdp
zD;l)8lkquU|AQW-Q4cQIB9F0n?_=L}a~G=lCg>!c48E8(J*~BF4ZoM#FU^f|QD9lJ
z%ijA^uK(IjUU!_YFM8i$ZnR8)Q;@bqFt;=gRV`{L=FOt!6{(>6s;wk{A_rj^1rVgD
zi+}<_3V{-**9V^dqV99h?O|rAfxJfF!LP+J0a|4SYR*ndi^>S%cnEm-l(d1GUvHM|
zAL_E|b8J;CDY&RBH-fn;cYa8C7w}PKQ~gFZt(+(#?!i;B<dWTiiU-qwW(-cHkCONp
z{uR_7K4GWFdni9lCul-$xj!fA$SA&0h1L%FamelT#QY$RSx5drHJh6G*5sv9`t*5t
zKV$oLR_!$XjR78An`k3OKLap98@Rsti5NMp4If6Da{uPQhxfzl=CLWDxZlzczN~7p
zWcGa%(8D#vQ;UZw#{iNypCZ*&Hgu=o5FVBSmdgd3XTGszx`&jkT!FoW4MHa}fk<`%
zUJMk7L5K&6LPbt2oDB7aFV*CbB2n|iht}J;t4X-M?a7&AI9`^87K*P5Id19nMf2_s
z!RcF*YF`VI&t!EJ440#Nn|R4c1)3=oLuDfM#4#+s`6gVRkdkeT;l7f#FH8aj6>2E{
zCkz?x|K{ATJzXfW)il*tSRXW-BDS;Mz82FECN63oQFG5IzP3zJv-DFE_t$aa@8e|y
z5Y-}<=1<VUf<>@Gf|x4oV`Aq!->6UG>qHn91o`%0ZRKQ`33!p9kJHUn5~4&vAt}li
z<Nko`%g~WC*Q`*b<lle`2?4hr8rKT!R$8lDpn7IdbqzMWPi5n{_f{D|-|n4nQEmt|
zU8?)vwr+ftxcx+@3IszKT(K?SpdlJc8U%O8=`Y`IGl!2c7Y>UH)0N3sztUzq_lK_(
z-LqP5EOzn8C|S5N`LmJjKdx|?P4}m(PDI~h{ec;GS-mO?K<3=C(25J3l8Qc}K1h$v
zmCObB>_HQ}m%gTnk2~X&e7qb*r6-4og$>lM&}<u;7^f#s&xoX0537<XMg-l~BvY8y
z?}Cjoak@-X&ar^)6H<Hq4b6eGndB(=GdR_Sa%-+pf65MLeeN|hXfwc4;oXQ#J@ry)
zIY6O|cr8)G9&#+9Rm3f^W3poM6l7wm6q>M#$<mVgIgfA;xN0C8mL<|x4LZNNhPi5W
z{XlhEFze>%w5g@xP}VYXlt^ILXBSre`oP6TQWfmu>3Fq%GC|)BiAo-JKF=-9<Zc)E
zzj^;~>vog8yYg{+vrCAP^2^5-g8@%~wi<zPdBQ!ts3mGma%=Ta_!AJxxPb-6$Ka=b
ztuj-VnzW4XTlxA7>gCg<KRvQGG-UT2@`TnsS+2v#{pB<3cAM&8bI_0*r)snWlc_x)
zftL;N(lHsFJyShVEybvRnOfQ+KXu^hjH8ychMy=@?f$I)i6Ci_uJU0jbhUS-2ybSe
zEiE$~5<?ehnc9wF(MO0p;fYUIKu7qpI$j*$CLz9ovm~QabFQ%l4X64u$JtGPdIT^r
zavF~%rA7E(j(sovs6{^&J8X24LOV^xD8`4+W0g-hePxag8s?bB|7JmH$*5bTrU~ll
zdrKP!-~{nf>hF<nU^_X{6&;0CMSLDc0~4+8(1@lyR&pt*qyQ1Ge)7oRqmZ<6ulagd
zVNfAup7bkaRyBx74Bj_-w!5LL>{31sLVV9f!8YRzY`<pN8cP|{12btTaWFZaiUxA>
zto`TEz=0hUTyVL3xJzMfe{On`vvt(=0NNJjGlg*S=8A@yAA0pgBmSE<XL%-8I@@<R
z&vT|~*c>oGzmNjUAZJ_si-y{J>psMk)h~tEbvvWkTRlZhDOUv9>ZHiI`NO*s#^B~Y
zCbukqoJy6r%ejwD6u7EI{{O&rNG`mYoWqk+&<6H7LtKah#O`{PNVA$h;8(~}GPgbX
z-#toDoGT0heM?u=t3XWkSFypUCa0{8<}AM>YxD6{EvLhJ2{pw7f3Z|w=ezd0)!Vnf
zS`a4`1LdFK#3{HxR=?ENeLRM<m@xu=T<G^yF`Aar4z#n+e;A=^eUVrn)X%ObbuQX`
zF(q33m0FdKKo`4o;km4CCublhdUh`SrB*k!?jBuHAUdlMK9)kN5GE3}Ea7^glc9FY
z7s%U~O8X6O5Z5jVG^j~I=p4|NR5N~bajSXRYD2Uo88xQv{siZQv##jPI>vkdj$HjL
zT^Q&6F<-o#I)*N&t=w#Tx!rat^ScI6Oz<ET#E~&m0Z(!4Jga|AC($QbHiLAmZkiWo
z*dA46F5i<L$ybEmH+}p;xG)f$A0yHEsq!wzHz=xZyMG=Uzfo9xoLI|JH6=AlQ~4d|
z7Yh|%#kT8}FrqE`M*p$s*FwNDGT%`ecKpn7*df*?{PTX@5PvL>$<V8}fy@8G>sMTz
zhy<R|xe{UAyr8nZ4pZ!mV0V!ys=<B}d8LnU4Z5BIvK^gBtSY8y&-bC~0`XU<c=)Fl
zzUw>dw2C(4Yj*}=u{V7sDq9c=S!gF>J7$Onzhj2;DMq5XHli{%&&i7Y@^HewgN_55
zp?Dw`a|kCnIEO=QkqA}M!{96)Ye<v50`vl!?k`#&=4jM;@+kro^XE20RY@3iO={Li
za$)u6$D@9B%>wbR-#VQFVXPZZdHUg?^Hc)6)L7?qoj|{cw$^KAGT)6^N_05`;fV0W
z!?Pj4Lp<P?uU7o(*=;1zu@ZrU3Loi0W3Lw9&t0|?Epe;%<}Y@}4;2P|*-&z7dI<5^
zzqt9pjw<}1EV!J*2kBt7BxTrO7Y31S#BRC*^Lm{<i`1{zu~#~!bYgrKz|t85>_2lo
z-O%MPg;}iEo_auv1P38&oOmJVRnQB_*m%fgk$Zk<+h5H>RTbbk-**hxJP?%mjHn4t
z;7Mrs>l8_j5st29^y2@e#pf9<h{2{og3^Er)d|t^{L@i-XN4`Pt?B*l7S3$?Pde_u
zBdeuiFS#=n7vPm00zLLqbzJ7D?!vjc_(Cc~rN`p{K~}C8bRG!QfKHRMdE#&Pzvq;h
zi0W64bCeHEQCLU){Ej_kpE}d6-${8ruQW~yz&!tXnOr`q#c5uaZQLQ<!Ftxrn}n#%
zvmFdiyd|RQRLsXGCUTD|uX6VuI4@^uGD;wI!~uSYUPxWu=ilhj;^MpS{PI}i8$D;L
zW7cdOkEG)}QB}Zru9JmUNz84Xgs8}_Yc9aN`6XX~rSrUA^bi|zbk#uPKRYx_m%}J#
z(<d7LHcYWO&1V*Cx!X(BEe9|kb&6tr(~mK<+lm%1hl_`ySm_`Q@!ww>xDV(*hFYd-
z^x2iDHBgRmE-~Dl&#{_~tU>tkZ(y3}KU$|{Bh{p|7iD=2A`f<0NmHD{f|=>p*C?Ug
z%gJ%L=G!hN&P+c&?B;C^eo`akW7kUMxOV;g*;hiRoE^wKYwb;B-)&3P?t9jdfzQ%^
zyKU2(;kS@OOZPhp>P!;oXeq0rMC|+{+K{?CDj&-W4$LDkk}^D@*ULtflt5lHEk|+N
zlejl4GdI8~yl#UM<JIb#v*z|D4B-Iu6*0q`f^s_co6V)u4<@W1R71xzTer%Bk3w^R
z&l+VYKoiB{0}5ki!2L-|QpT&0n-&o+8d>>yHefV@#E6z?u=pkYsjINKTy;NvlkwlE
z+4VXbaI#F}zr#4f#gd}~EU33XqSQqVHQMnyUf}Q>3+K4db~08PLhUDURbBg~i=N^Q
z%1@=deEM{i$S&BZik35EtumgoUslt#VD;Pwzp^I0qxRex0{r(~z$xXBr~Qo@qV*%c
z&wq^!!`BWR!V?mhB`Jb<SFjnRlq8dr)1J1HkaPrjCGW1^8N{h4`F#1VPft(Sxv;+e
zA)@h9>8wtVR&3A!bR#kp2j6}!2jP3MU~EMH`b-@$YCmMrvEyA?#&v%zG3~xj9pZM9
z(D{JQw?sDSw}-_(+)G!;s({$bAteljI$5WU7BIUP=?kMBb4Rudj-{0{MwBKrQug5;
zUel>wwYcs%`3G7$En(Y*x)TXR8jQwWkcu=H=7UC?GYd6LIH(e77PI0MkpR-`1#DD|
z`@7qn()yOp$T68r$KH|6ElC4eN&G}s&#j#1j*l1*DsL2U;|%tam)Pp7OPPO&2MV=b
zr@epGkFo@(kw1sO?VOvBg#f@>vke^$q^D_NA=w-(%|e0oUbVOR->CER2~ie_1?6>B
z(Z@J|h(<Uo)`HW}=@=wA8Iwpb-shELtQYFFTNSk28|!5bUI-_sQ{w>aR5$bP*NO8m
z_X#dU##Pw<{>2ok+Sz6rx%9vBi{3E+eXwo3LztndgWiGc-IgRNJp7g)Mp_JB%>}>y
zO~|vPWBR$~l|>GuJ{U^Fmx%zwMwOG#FQ~u02aCH{n?3j8vkE<AE2(YvKqk9iIy_Qa
zgJ3(u<bn57&<T3G>tv5yBK`%oCPaB7zIPJUWahpxJQtM*(Xq;FkY&<*JuQeAV<(SY
z5<gA-l0V6BfX}sxZ#_YtKqq#j5BDaH_CGTlQHq)-7~uN%&aMr<*aNm@sn3FDv+FB|
zVPmQZ$($0*r_yp%f!dwyNPNC`w4gxZ@pnA0)gh;B#7NbyBK-~&|6oeC`TW<v8cqgA
zND7q<Jr)k-4<#mk+pM~2darf_3DyKEJ~Xi}Kz5dxiY`B2>Re2Pg^2-TR!neBM5k|H
z?P9$Xz*9mlsi~!w9sdiXx<LJSDz@?T=?u;SngzxGGSt4GT)QjupvarS5d~XH`AG-S
zA9bX%ArJ04LW{oVjJ8w9R~vDRJkqs~v5)Cn9IrGLdGi!^HS!YT(mArt-`rC3=*;l)
zB)#nPU!Ttg$ipw>Vx43!*gppZj2!hN?-eyGUK1tD+IKkAPR$$6P21ms{!mP9_?0^;
z(w91YXm(#TvghJ^D`pF1T4Dg~0E|H5C2_?E+|*s_7UcfQ?XwVU$!by60U|Hq^n3_V
zL@v`j$~u!{4J0kPMMU5)DS>H}|AEU*M7=BR5?=C>8?Q`X703zj)J_E&+LIcs#0`f~
z$_!E09cS}Qa+jrr&g@YRg8oU*4nj;bUv4$BQ|F$X_5z0B`$g;~^3L=UHOfB(W(;1c
zY8j^?s_jr|cZsYU^ok;U)CgRJ)%PtZW6Q>(WLMO-w^3<!4%d?OASxJmg_q0;nxvv-
z_lI=M-)H;J@lmPfTfr)qB5f@>jkzdQS{n%?;o<aQx5TdTmN<3LCpBKc{6Vy<Q-n;Q
z4V(>?H*gSsu*MpJ!S*grrN--DF*SjYm-w%T0XaTt-2upA_tEFuM_0Q0wD<hi5vA?O
z<|8Vxtvi`IyKioQDqMl}DDuz#$NEl|W^=B87q!MC6jzk|ndl@p<fnd|*>z#$AQAl^
z-*MbnnyKzpaSmYuS;s5D`}Y^zwSqFjYA>EyD*0hoY>h;lYr4$Ob&KaoIV~k^vrE69
z&{u2x0o$={W(#{R+m>DgWITLE)Om<$dtAFq#0Aey#Gmkzy29GZ*{iUnuzFzs$Dw|t
zg!u3idY5U@fgZi^E9C{;XupGP>;vDcss8BpGfRvL`OL78+TnG~WzEngij}F32xm)!
zqnr0jpa5Q%^WAHOp!_YH9@g=^H`SIq(5=1)BaQDaPVW{s!pH|f8Sa04d|_#3LL=bD
z|K!_L)<w1U@3-n|rv9xg9(aw1LS&=NDr`HFR)No>{Pr+N9kDU=OV>)oq0d_&VrgmJ
zJvtTcw=#lerTGsMJ<RA~b=)!`ACzu^f&jbbVgN5@3mSAkd&_!l9l`nCW8>hFMjON=
z9ZhG8Koq7^>)M0OMkydN0m}f{N%^x!S10nC+=gMLLSiW~8+vDzd~1=Mu$Z7JzN&cj
z6@X&}M4ugLFNQ@?^BOH2yUxaV&F$H8yX1GIweAN$9vg;i2%V1jy&2^G%>40U=2$Bb
zIY{GqjNtu>IN};o><B|k)V>Z3K^RsbCEHsyE<c50*5})zr>0@(W$bPvNh%U)fMJBI
ztzL90`6mKOcQARCR`hLJxoJNcH%Z-@5fwz9AnKpI+y&56_Vs2bH6Dz0p0V>nUKlz0
z^GKf~rh21M+<>hQ7isP#B6I6>vOuh&i%b%13fF0xM5ox-!5$;bJ{78`mJ+wv`<%56
zu=GKt3Z^eH-f{er1&GUXpHnVeus4bHN|xc;=t{`u>Fdi(H+V66U?l)L|Ek5qcz7!O
zz1P<@l6G5_ab<WoA<TXnCH|`sCclkTp9z+S7WL3Cpegm{x&eip@W)87w&b<3VX380
zLC8Teukn(j#Z+bCaQl7$<1YUJ$J+q&=CBmAN10kcF%dnV7hG^RF4`INRpowOm55>$
zo`9oC1db;8XH~JV50g|r)$aEq9$g|LYSZ^|R|q9&rXr+4<fq`%{xkZG-CY<yi!T&j
z^NI;b%XgC6Mr4;>uSZi_QgOh4fzp%z?l@$Cx2?~@{+<Te(e});-<r*nj}B-KTpmC?
zT~g`UR)Kc{Z9w>Q(>zyHk1$!e<&3-`Hyqd+xs$(o@bLlvJ=o<q_$a|wVyp=g`V1lD
z;hFz?K0%ThIbZa@bCz3*RS2|G|8sJ|#pyqgVJ~NK>F5~SIs0ej*pW%0b+->2uWKAU
zd%_6P@J&<ieM{gJglDZ2WrvheNB#Y#l1U_7oKxc0hmeU@a!HjmMiqHJPK!fvbVZ9Z
z9EKl${@Dlug*B-E<3%RZ@|!sO=tp@Zei43IL=oh^WmfeKnC1}@j>R7F21b2&!$ZZZ
zZ3tT!{a#Hh>2)v>RYZqHPNq2q+A&W7c5+nha+hrgY+kAuiL<s_>zN_xZS?^5aCcm$
z2ce%*^fV2jBMG>APXJBV0iroj*-DHlrQu+!aR^+bRY@e-nOL|SH4c8YJQ~Xl3F(Q^
zk(*VrMT5GnS2`A=iX;s^;g@Cns>^QP#fp~-qBiGjmbQ2mTgpwg^#Avv#F27-Rjz7I
z?sD+B-V6g_s)41SHpRbd$YPU`xI5R4cToK8W*S23;QOG2-TdQ6no&$CvXZj@_B|w_
zYvc9{_0+wIHahJ8&Ic8vYAgR5!FFPNhL*KEPn;<>mZn!ad15byhumr^DiaxE4p9lx
z|0(Ru<DuNU04^pm*2t2C7|cwCCX^-al)*%pY}chy=vJ0)mh9UIV^4_4nq@4>MY=+a
z?b?!Ly{<J%5tFi1gZB)#`);4l`_5m#&*xd@{FZZ`^PK1WzGr#YKBuX5T*67WE5TM-
zzbZx|d+I0B6S|1V_F#|G19?2(Z}(~~M*yjF<ovtON^(!K_NYss&xY;J$q9d>IX=l^
zhZFqtNbv}pW%XQZ84meDrSY6c&NVbP=V~Vr;}!VavRuLWdpA=cm0cgr%IVNt;T`JG
zb2vTCMSrg@1(_;e78^(M!^H75jb(f^jB=n&#Z?WZj&V}6sS+;bF)R9=ooHOWB-$!j
z6^%t)1+cw+bji*2cj>F$`nDClT<r<#`ue2(;zj<>zg|^%5R_z*B9WY&Y{ea7Y}8X+
zP0d(zH{@vRBDTiDM(?N0HH*`{$Lg2Ri8hRS9^&$DdM)yS2_lzDA?iKkaKYa*K_fkh
zbrl_~SQU|^1*=CT&3;!V&d<~A%s9w>yFsPnv#D8ApRwgZ?FJlOKHd%YY7r^YMHQyp
zjp7LjC(E;qq6@RB6OL)}2fhrq1Y1OiXln?_nSUP#<{J8(>ltuaxy*=i;+%7x$_+7f
zf?4J#0%`x=Qa4v}#@F-mKN5M<?La?7hp+stPvt^FSyf}$4w)EhllS@iPTl{J^fPH#
zlk<f3D|#PdJv`0XDKk8;rF;n&`A{-E<9?R0!U#Y%{?bMWE79L1>iO85CmtEc%dpDL
z$zeZ+&u(IDU7ONPGpT<h7^uMa*zNH(w2OQ{!T;g4oxAaw9u3^o^WebEVo8wh(*1>Y
zPM_uj)TSnPTkdGoE|Z`Z<JdtN^VQA%_XezNv2os9#nQ>w48Qj_rFBoz(%)tPOg4mL
z@b+AFS%w~qw2KdZUu#Cix8(YbRXkyL6Fxlbzvjh%BAv+++P0iP8)&<WGJ01z{OOtF
z%SKsLBo+1MlSEyB#?7fjD=y)8R>Sh;-qRDzTps3YCj1ls@KQBZndiCOi<A?dAIp!u
zsj95K<jMp>ZJ9ZU!%&>EDIG)YcC_*&F3XYuC~$4t-+dOa*i3V|C0|r7)$m-c@I#vu
zb_-+?pT*Z!gv59i=DQud<3l}Asq)Z0{oSYZ@FccfeARx7q~kK-xRNmD%IqKcExDlX
zc9<;oSB1dG2Vg_`?VQxdwl60xHigHEubX^2<HWmtE<XZ)n_bs+F!OHqa2e^r&t2A8
zj#v3SVAmwG7YMwVD3L^)&eP!Y*%<Y0fB$%3EEp5&*Upq9REmSFth9$DJ1w-R=Cia~
z8dx%1*)JoVK@&R73q&z20?(ft3@O|k`9GJKj=r3UbE%0H=QDUXY6W4tt%I;TAn4Q&
zLybbkzKs4k#zWu27S^p~EDo}iKAUMFRn%tE(XMm1UF-6G`U|!*`~P_`p;eG@xs3d7
zFNQF4XBlNg)bVi<l?lK5ds2xdA51SlBHj_^pDabO3EAq&v>1~b!67{es}cUZntiJS
zZ7N%$=#)ZO8cud)nw<Z>G!;uoviiCMj^OQii?O5#M^SF}5Lz3sZJ3StIjG_V4Vgi3
z)1jeT8p5oW1GO_Cph$y8HE$#`1RAdUp+XKcrq$8dts8j!zZ<+2Bqb%@m1g#%3o1Lb
zq~=gM{-g<QNfK*mOZm}sTYez*{SW#7U)M33tpmJq=WET_xeqy^$2G`O9k3E7y&dhV
zep1WHexUbd%%NN3=$y!-v9_ahUd&?kj+ZK`1-18UKaTarE@kB13a;jv$(!NvH$B+l
zYXRA+NGLNKrs<Wh5a6~X_P9V14VP5k{8qZ=Fs%}^a?iV@6ca%7eqN_%j9IjTxFkjs
z-o?l{0`<|g>;wYLO&g?-a^p!67j(}<Dkw~$J-8CExSJJa9sv>px?7GQ$bz>&l^hRK
z+6J>~1^RaYjs4i!1~YGOVn#Ux;h2^A;nx!rw?AK%mX^Msp8g_TLtdiNXBp7uHa0fR
z71}X1HQxZ$+WcA!4LltLf#Ch(EO_EzDUor?IvxNUZEMrl%3PeC^_m-aI45amXP0pQ
zKKD~H^2j`ZGH=|tL5~QwY8J&Q8JL?Vi4K?m(hh{^@VyIfmxTbN@XD1h>g_%2?EC>h
z-U)<R>eABE{QUgN!YCJfoS|POSzeF+@EEEWHAe{2*(xh5OUx$eulolF!*y4Hh^xdI
z;$OtU7>XcV6HwdT51eyJO)nPz72s^P<QcoIE}ZlA^*wc}>49FH#PMH#(9_dnj0T7;
zRaiE)6;OeJ%$|v`wjMdH%?SwDz;7#u88KdOyZ5KulBBGqZ5|R)u#pljBr?b9sxk?m
zq2-^Nnu>IN3YcI>w!J1>1W4tR2FU!tmHB(Wp?0qg1JfTRCs$NhNJGt@mg3JXC~))_
z9*Px=>O;9o7kSJL)CH`IRVf0q5vZvE3ZI&uMz~!Z;Ta!=L1r)*zuCCx>r>WzHv%&>
zz0VdKozo)htUvCqp58QB>-F)Z&7C$N*<MsYJPT?E-f@HcfrAGP7&5o@^R(NG;M;~h
z5grOK2DbUp(a~>TWHzB7q?Pl>lk1t))oKj0_4V}@*@%WiL#}u|f;+bqRx6V)&Et|w
zx{P+s25>faiiPo%=F54cJ#!o^ot?d6a(qZE5gz|ldb+y0Q@cH4{35SGeh#ETI5;@i
zUb;b)Zf$L)YXHEORZ<l8yU^qh$jODJSifbiw%(K+vIUEJ(a_M3fL&tBK|Z52yDNa!
z@R=?sY^nXGEnWtX#{<iDTDSIH1X8ue!os4ZrA0zQVybtOEpS!oUKf>$zI4GNL8Koj
zqj>Gm&`>oswH~2|Ru#IMcFXT!D4JHgFa7H7?oNR(S4=I5k;4w9Jguo|dUX5o<7L1K
zXPpG*{LItpYFbqox0C&o>*Gd;`h%b>6Fyc_n>RH)GX=Pu({DHfx_hqa=;+wg`6N-5
zPmu6^eSNkgCm+0=s6Z+Bb2~w~X~1mv{6w=)5D`M$LM&h~m?KI`LG`xH{r$z0ib-8v
zhO>%~F4P-AHWJe|#Ai?^2Af)1cB(3PYPXJ!WzcLJBsb`3r?_Qwb~Zsq5v%Ma7nk$J
zb#GM+1^N47CH_F7i?BF(0{NLD&A?2|&vVzi!|>)#q~%g4*aXGIko!`B`Wt0XYzaHs
zj;X2b_2F7_%j0Ik)klCV+}Y9b!^pm4E~rR4v8<w^c!-gjmXL|qBLtc7pkz>EEG0di
z2TPV?5fB&WE+sq1c=0cMEv}<Vfog=7?g_X8sW?(L9`i#-hPd+z3Iq#C{5wcV$;t9^
za_1S!)=yl7;;2^Br_JbZF4PC`@W}ezZBu3K;u+a5DJi+u&JWrO6Vy`}RsSI__~~@)
zY^sh6VP`NOi=TjP2-o=WFT|iybs3r0P@|w(l!JoQ;-o*xS}UM$_N^@O!3TT;u}`8x
zvS4H=NHLDt(-&W{1!`RXqmmN~?Do#*&qL1~b)rNZ;pb1%M`1CTU<C`@oj^3Gl`NLS
z>E=B{ImQlm#@dY>*0jqKajR1;Q2={q%ETJXQl&R_{hM85V<W{sL?uo$3i&KKD2Qdv
zjelc9W@DqLzdy8FAo5Off0Hpc2-S(+z%aCv70wAjcbS?G?|Zk^Dc4@N#sF&iz=rM^
z(597zfw?Nq8-85mo?*hD%e#UXXq}y0RXA9}N}R5&*J+nm0ma_TLcz^{4QaSO0LeWK
zztf3Bb-u^^1pXWyoqQ7Ey3D%p6$1WSkAgnD8ULE8`oV)-0W-Eyev^%%VCg=8c3+xM
zi_gav@$6Ft0CZgEjGVlDQ~vP4fUwqO&=as(B5@{SB)plq`B5A$>moUh_T|&t$)8yh
znmwL@iD#}vrua`Iu7Pz{P*5oIh8(b)9v-3N>Q@v|Px@}>W3e2{#yux#p8j8!X25z$
z#CgInw!29s>6xXeI=i~Ml2^pE*!kDY0h}E&uJv&~4o`x#(op!Ca_YTY5fl_?tP)l2
z0nHp99`^S3270-)F)OO~7U8k)stdrI*A^y@vX5L5i_wj*Tw>0<ywQCr1*)_bXOGKS
zEl3=OlI3*$Q)VC`EaP#&;6rz0H~aGJo5Fa?A|N;F7iVKav2|)RRm!@`Df`vCvK;~(
zh%z2^kN1gQt(&iI5q4U?g&GuY)qj?k<AxSy=jKd}Y)vId_=#k+mX<`o9y*Z{6O*JF
zL+02v6TSRNe)*UP<i?(!moGt;gD+?vKK$^7v5HG$4tajGM&u7nyguSN!rN7KZ=Qe<
z<G|Y2<+-V;OKYeGy{S(JXGnjKta=nu)fhO?tougmd_5uUG4Zg9JwTtKBO8oGsA&WS
z+!--63oQP@M;A8z#&OECfQJsV>M4mb@Q7gm@i_AqXl;Z1MX;=yfdZh+?3uTI8b#Oj
zet%|r2%A43Xc79aKz4zO(O*jvs_-zK1vD0i#_HS!uNJ~B4BQg9GJBcI1knI^{s-*!
z=824i%GU9+|LH&6@w{xYD^A>;y^<#yGhWm6uf7DoECk*Ez80UDD6e>-&}Bs7h33ni
zoZ7eS)7>GzCnhHxt90i<FIt_}9PG`mt*rex`^8|}yV-IpWiZ$t!EJX@@!vho)DA++
z<S*#gOg~dc?k|5cOu7g(6vlYl4OP^SM1DpI3n^P9-hkV2NJB#V_uEdf2yNe2MoUB}
txzm|FM#w=N&LOuUh@{=$)_=2E*kqAV>Wv<iEYD+sACX|BQ+m>t{13q)j6eVY
literal 0
HcmV?d00001
diff --git a/doc/guides/rel_notes/release_20_05.rst b/doc/guides/rel_notes/release_20_05.rst
index 000bbf501..f404ee7fd 100644
--- a/doc/guides/rel_notes/release_20_05.rst
+++ b/doc/guides/rel_notes/release_20_05.rst
@@ -62,6 +62,11 @@ New Features
* Added support for matching on IPv4 Time To Live and IPv6 Hop Limit.
+* **Updated the Intel ice driver.**
+
+ Updated the Intel ice driver with new features and improvements, including:
+
+ * Added support for DCF (Device Config Function) feature.
Removed Items
-------------
diff --git a/drivers/common/Makefile b/drivers/common/Makefile
index 96bd7ac6e..df2e840cf 100644
--- a/drivers/common/Makefile
+++ b/drivers/common/Makefile
@@ -31,6 +31,7 @@ DIRS-y += dpaax
endif
IAVF-y := $(CONFIG_RTE_LIBRTE_IAVF_PMD)
+IAVF-y += $(CONFIG_RTE_LIBRTE_ICE_PMD)
ifneq (,$(findstring y,$(IAVF-y)))
DIRS-y += iavf
endif
diff --git a/drivers/net/ice/Makefile b/drivers/net/ice/Makefile
index e22c34287..f493c9ed7 100644
--- a/drivers/net/ice/Makefile
+++ b/drivers/net/ice/Makefile
@@ -11,9 +11,11 @@ LIB = librte_pmd_ice.a
CFLAGS += -O3
CFLAGS += $(WERROR_FLAGS)
CFLAGS += -DALLOW_EXPERIMENTAL_API
+CFLAGS += -I$(RTE_SDK)/drivers/common/iavf
LDLIBS += -lrte_eal -lrte_mbuf -lrte_ethdev -lrte_kvargs
LDLIBS += -lrte_bus_pci -lrte_mempool -lrte_hash
+LDLIBS += -lrte_net -lrte_common_iavf
EXPORT_MAP := rte_pmd_ice_version.map
@@ -84,6 +86,9 @@ ifeq ($(CC_AVX2_SUPPORT), 1)
endif
SRCS-$(CONFIG_RTE_LIBRTE_ICE_PMD) += ice_generic_flow.c
+SRCS-$(CONFIG_RTE_LIBRTE_ICE_PMD) += ice_dcf.c
+SRCS-$(CONFIG_RTE_LIBRTE_ICE_PMD) += ice_dcf_ethdev.c
+
# install this header file
SYMLINK-$(CONFIG_RTE_LIBRTE_ICE_PMD)-include := rte_pmd_ice.h
diff --git a/drivers/net/ice/ice_dcf.c b/drivers/net/ice/ice_dcf.c
new file mode 100644
index 000000000..efb258a5a
--- /dev/null
+++ b/drivers/net/ice/ice_dcf.c
@@ -0,0 +1,474 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#include <sys/queue.h>
+#include <stdio.h>
+#include <errno.h>
+#include <stdint.h>
+#include <string.h>
+#include <unistd.h>
+#include <stdarg.h>
+#include <inttypes.h>
+#include <rte_byteorder.h>
+#include <rte_common.h>
+
+#include <rte_pci.h>
+#include <rte_atomic.h>
+#include <rte_eal.h>
+#include <rte_ether.h>
+#include <rte_ethdev_driver.h>
+#include <rte_ethdev_pci.h>
+#include <rte_malloc.h>
+#include <rte_memzone.h>
+#include <rte_dev.h>
+
+#include "ice_dcf.h"
+
+#define ICE_DCF_AQ_LEN 32
+#define ICE_DCF_AQ_BUF_SZ 4096
+
+#define ICE_DCF_ARQ_MAX_RETRIES 200
+#define ICE_DCF_ARQ_CHECK_TIME 2 /* msecs */
+
+#define ICE_DCF_VF_RES_BUF_SZ \
+ (sizeof(struct virtchnl_vf_resource) + \
+ IAVF_MAX_VF_VSI * sizeof(struct virtchnl_vsi_resource))
+
+static __rte_always_inline int
+ice_dcf_send_cmd_req_no_irq(struct ice_dcf_hw *hw, enum virtchnl_ops op,
+ uint8_t *req_msg, uint16_t req_msglen)
+{
+ return iavf_aq_send_msg_to_pf(&hw->avf, op, IAVF_SUCCESS,
+ req_msg, req_msglen, NULL);
+}
+
+static int
+ice_dcf_recv_cmd_rsp_no_irq(struct ice_dcf_hw *hw, enum virtchnl_ops op,
+ uint8_t *rsp_msgbuf, uint16_t rsp_buflen,
+ uint16_t *rsp_msglen)
+{
+ struct iavf_arq_event_info event;
+ enum virtchnl_ops v_op;
+ int i = 0;
+ int err;
+
+ event.buf_len = rsp_buflen;
+ event.msg_buf = rsp_msgbuf;
+
+ do {
+ err = iavf_clean_arq_element(&hw->avf, &event, NULL);
+ if (err != IAVF_SUCCESS)
+ goto again;
+
+ v_op = rte_le_to_cpu_32(event.desc.cookie_high);
+ if (v_op != op)
+ goto again;
+
+ if (rsp_msglen != NULL)
+ *rsp_msglen = event.msg_len;
+ return rte_le_to_cpu_32(event.desc.cookie_low);
+
+again:
+ rte_delay_ms(ICE_DCF_ARQ_CHECK_TIME);
+ } while (i++ < ICE_DCF_ARQ_MAX_RETRIES);
+
+ return -EIO;
+}
+
+static __rte_always_inline void
+ice_dcf_aq_cmd_clear(struct ice_dcf_hw *hw, struct dcf_virtchnl_cmd *cmd)
+{
+ rte_spinlock_lock(&hw->vc_cmd_queue_lock);
+
+ TAILQ_REMOVE(&hw->vc_cmd_queue, cmd, next);
+
+ rte_spinlock_unlock(&hw->vc_cmd_queue_lock);
+}
+
+static __rte_always_inline void
+ice_dcf_vc_cmd_set(struct ice_dcf_hw *hw, struct dcf_virtchnl_cmd *cmd)
+{
+ cmd->v_ret = IAVF_ERR_NOT_READY;
+ cmd->rsp_msglen = 0;
+ cmd->pending = 1;
+
+ rte_spinlock_lock(&hw->vc_cmd_queue_lock);
+
+ TAILQ_INSERT_TAIL(&hw->vc_cmd_queue, cmd, next);
+
+ rte_spinlock_unlock(&hw->vc_cmd_queue_lock);
+}
+
+static __rte_always_inline int
+ice_dcf_vc_cmd_send(struct ice_dcf_hw *hw, struct dcf_virtchnl_cmd *cmd)
+{
+ return iavf_aq_send_msg_to_pf(&hw->avf,
+ cmd->v_op, IAVF_SUCCESS,
+ cmd->req_msg, cmd->req_msglen, NULL);
+}
+
+static __rte_always_inline void
+ice_dcf_aq_cmd_handle(struct ice_dcf_hw *hw, struct iavf_arq_event_info *info)
+{
+ struct dcf_virtchnl_cmd *cmd;
+ enum virtchnl_ops v_op;
+ enum iavf_status v_ret;
+ uint16_t aq_op;
+
+ aq_op = rte_le_to_cpu_16(info->desc.opcode);
+ if (unlikely(aq_op != iavf_aqc_opc_send_msg_to_vf)) {
+ PMD_DRV_LOG(ERR,
+ "Request %u is not supported yet", aq_op);
+ return;
+ }
+
+ v_op = rte_le_to_cpu_32(info->desc.cookie_high);
+ if (unlikely(v_op == VIRTCHNL_OP_EVENT))
+ return;
+
+ v_ret = rte_le_to_cpu_32(info->desc.cookie_low);
+
+ rte_spinlock_lock(&hw->vc_cmd_queue_lock);
+
+ TAILQ_FOREACH(cmd, &hw->vc_cmd_queue, next) {
+ if (cmd->v_op == v_op && cmd->pending) {
+ cmd->v_ret = v_ret;
+ cmd->rsp_msglen = RTE_MIN(info->msg_len,
+ cmd->rsp_buflen);
+ if (likely(cmd->rsp_msglen != 0))
+ rte_memcpy(cmd->rsp_msgbuf, info->msg_buf,
+ cmd->rsp_msglen);
+
+ /* prevent compiler reordering */
+ rte_compiler_barrier();
+ cmd->pending = 0;
+ break;
+ }
+ }
+
+ rte_spinlock_unlock(&hw->vc_cmd_queue_lock);
+}
+
+static void
+ice_dcf_handle_virtchnl_msg(struct ice_dcf_hw *hw)
+{
+ struct iavf_arq_event_info info;
+ uint16_t pending = 1;
+ int ret;
+
+ info.buf_len = ICE_DCF_AQ_BUF_SZ;
+ info.msg_buf = hw->arq_buf;
+
+ while (pending) {
+ ret = iavf_clean_arq_element(&hw->avf, &info, &pending);
+ if (ret != IAVF_SUCCESS)
+ break;
+
+ ice_dcf_aq_cmd_handle(hw, &info);
+ }
+}
+
+static int
+ice_dcf_init_check_api_version(struct ice_dcf_hw *hw)
+{
+#define ICE_CPF_VIRTCHNL_VERSION_MAJOR_START 1
+#define ICE_CPF_VIRTCHNL_VERSION_MINOR_START 1
+ struct virtchnl_version_info version, *pver;
+ int err;
+
+ version.major = VIRTCHNL_VERSION_MAJOR;
+ version.minor = VIRTCHNL_VERSION_MINOR;
+ err = ice_dcf_send_cmd_req_no_irq(hw, VIRTCHNL_OP_VERSION,
+ (uint8_t *)&version, sizeof(version));
+ if (err) {
+ PMD_INIT_LOG(ERR, "Fail to send OP_VERSION");
+ return err;
+ }
+
+ pver = &hw->virtchnl_version;
+ err = ice_dcf_recv_cmd_rsp_no_irq(hw, VIRTCHNL_OP_VERSION,
+ (uint8_t *)pver, sizeof(*pver), NULL);
+ if (err) {
+ PMD_INIT_LOG(ERR, "Fail to get response of OP_VERSION");
+ return -1;
+ }
+
+ PMD_INIT_LOG(DEBUG,
+ "Peer PF API version: %u.%u", pver->major, pver->minor);
+
+ if (pver->major < ICE_CPF_VIRTCHNL_VERSION_MAJOR_START ||
+ (pver->major == ICE_CPF_VIRTCHNL_VERSION_MAJOR_START &&
+ pver->minor < ICE_CPF_VIRTCHNL_VERSION_MINOR_START)) {
+ PMD_INIT_LOG(ERR,
+ "VIRTCHNL API version should not be lower than (%u.%u)",
+ ICE_CPF_VIRTCHNL_VERSION_MAJOR_START,
+ ICE_CPF_VIRTCHNL_VERSION_MAJOR_START);
+ return -1;
+ } else if (pver->major > VIRTCHNL_VERSION_MAJOR ||
+ (pver->major == VIRTCHNL_VERSION_MAJOR &&
+ pver->minor > VIRTCHNL_VERSION_MINOR)) {
+ PMD_INIT_LOG(ERR,
+ "PF/VF API version mismatch:(%u.%u)-(%u.%u)",
+ pver->major, pver->minor,
+ VIRTCHNL_VERSION_MAJOR, VIRTCHNL_VERSION_MINOR);
+ return -1;
+ }
+
+ PMD_INIT_LOG(DEBUG, "Peer is supported PF host");
+
+ return 0;
+}
+
+static int
+ice_dcf_get_vf_resource(struct ice_dcf_hw *hw)
+{
+ uint32_t caps;
+ int err, i;
+
+ caps = VIRTCHNL_VF_OFFLOAD_WB_ON_ITR | VIRTCHNL_VF_OFFLOAD_RX_POLLING |
+ VIRTCHNL_VF_CAP_ADV_LINK_SPEED |
+ VF_BASE_MODE_OFFLOADS;
+
+ err = ice_dcf_send_cmd_req_no_irq(hw, VIRTCHNL_OP_GET_VF_RESOURCES,
+ (uint8_t *)&caps, sizeof(caps));
+ if (err) {
+ PMD_DRV_LOG(ERR, "Fail to send msg OP_GET_VF_RESOURCE");
+ return err;
+ }
+
+ err = ice_dcf_recv_cmd_rsp_no_irq(hw, VIRTCHNL_OP_GET_VF_RESOURCES,
+ (uint8_t *)hw->vf_res,
+ ICE_DCF_VF_RES_BUF_SZ, NULL);
+ if (err) {
+ PMD_DRV_LOG(ERR, "Fail to get response of OP_GET_VF_RESOURCE");
+ return -1;
+ }
+
+ iavf_vf_parse_hw_config(&hw->avf, hw->vf_res);
+
+ hw->vsi_res = NULL;
+ for (i = 0; i < hw->vf_res->num_vsis; i++) {
+ if (hw->vf_res->vsi_res[i].vsi_type == VIRTCHNL_VSI_SRIOV)
+ hw->vsi_res = &hw->vf_res->vsi_res[i];
+ }
+
+ if (!hw->vsi_res) {
+ PMD_DRV_LOG(ERR, "no LAN VSI found");
+ return -1;
+ }
+
+ hw->vsi_id = hw->vsi_res->vsi_id;
+ PMD_DRV_LOG(DEBUG, "VSI ID is %u", hw->vsi_id);
+
+ return 0;
+}
+
+static int
+ice_dcf_check_reset_done(struct ice_dcf_hw *hw)
+{
+#define ICE_DCF_RESET_WAIT_CNT 50
+ struct iavf_hw *avf = &hw->avf;
+ int i, reset;
+
+ for (i = 0; i < ICE_DCF_RESET_WAIT_CNT; i++) {
+ reset = IAVF_READ_REG(avf, IAVF_VFGEN_RSTAT) &
+ IAVF_VFGEN_RSTAT_VFR_STATE_MASK;
+ reset = reset >> IAVF_VFGEN_RSTAT_VFR_STATE_SHIFT;
+
+ if (reset == VIRTCHNL_VFR_VFACTIVE ||
+ reset == VIRTCHNL_VFR_COMPLETED)
+ break;
+
+ rte_delay_ms(20);
+ }
+
+ if (i >= ICE_DCF_RESET_WAIT_CNT)
+ return -1;
+
+ return 0;
+}
+
+static inline void
+ice_dcf_enable_irq0(struct ice_dcf_hw *hw)
+{
+ struct iavf_hw *avf = &hw->avf;
+
+ /* Enable admin queue interrupt trigger */
+ IAVF_WRITE_REG(avf, IAVF_VFINT_ICR0_ENA1,
+ IAVF_VFINT_ICR0_ENA1_ADMINQ_MASK);
+ IAVF_WRITE_REG(avf, IAVF_VFINT_DYN_CTL01,
+ IAVF_VFINT_DYN_CTL01_INTENA_MASK |
+ IAVF_VFINT_DYN_CTL01_CLEARPBA_MASK |
+ IAVF_VFINT_DYN_CTL01_ITR_INDX_MASK);
+
+ IAVF_WRITE_FLUSH(avf);
+}
+
+static inline void
+ice_dcf_disable_irq0(struct ice_dcf_hw *hw)
+{
+ struct iavf_hw *avf = &hw->avf;
+
+ /* Disable all interrupt types */
+ IAVF_WRITE_REG(avf, IAVF_VFINT_ICR0_ENA1, 0);
+ IAVF_WRITE_REG(avf, IAVF_VFINT_DYN_CTL01,
+ IAVF_VFINT_DYN_CTL01_ITR_INDX_MASK);
+
+ IAVF_WRITE_FLUSH(avf);
+}
+
+static void
+ice_dcf_dev_interrupt_handler(void *param)
+{
+ struct ice_dcf_hw *hw = param;
+
+ ice_dcf_disable_irq0(hw);
+
+ ice_dcf_handle_virtchnl_msg(hw);
+
+ ice_dcf_enable_irq0(hw);
+}
+
+int
+ice_dcf_execute_virtchnl_cmd(struct ice_dcf_hw *hw,
+ struct dcf_virtchnl_cmd *cmd)
+{
+ int i = 0;
+ int err;
+
+ if ((cmd->req_msg && !cmd->req_msglen) ||
+ (!cmd->req_msg && cmd->req_msglen) ||
+ (cmd->rsp_msgbuf && !cmd->rsp_buflen) ||
+ (!cmd->rsp_msgbuf && cmd->rsp_buflen))
+ return -EINVAL;
+
+ rte_spinlock_lock(&hw->vc_cmd_send_lock);
+ ice_dcf_vc_cmd_set(hw, cmd);
+
+ err = ice_dcf_vc_cmd_send(hw, cmd);
+ if (err) {
+ PMD_DRV_LOG(ERR, "fail to send cmd %d", cmd->v_op);
+ goto ret;
+ }
+
+ do {
+ if (!cmd->pending)
+ break;
+
+ rte_delay_ms(ICE_DCF_ARQ_CHECK_TIME);
+ } while (i++ < ICE_DCF_ARQ_MAX_RETRIES);
+
+ if (cmd->v_ret != IAVF_SUCCESS) {
+ err = -1;
+ PMD_DRV_LOG(ERR,
+ "No response (%d times) or return failure (%d) for cmd %d",
+ i, cmd->v_ret, cmd->v_op);
+ }
+
+ret:
+ ice_dcf_aq_cmd_clear(hw, cmd);
+ rte_spinlock_unlock(&hw->vc_cmd_send_lock);
+ return err;
+}
+
+int
+ice_dcf_init_hw(struct rte_eth_dev *eth_dev, struct ice_dcf_hw *hw)
+{
+ struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ int ret;
+
+ hw->avf.hw_addr = pci_dev->mem_resource[0].addr;
+ hw->avf.back = hw;
+
+ hw->avf.bus.bus_id = pci_dev->addr.bus;
+ hw->avf.bus.device = pci_dev->addr.devid;
+ hw->avf.bus.func = pci_dev->addr.function;
+
+ hw->avf.device_id = pci_dev->id.device_id;
+ hw->avf.vendor_id = pci_dev->id.vendor_id;
+ hw->avf.subsystem_device_id = pci_dev->id.subsystem_device_id;
+ hw->avf.subsystem_vendor_id = pci_dev->id.subsystem_vendor_id;
+
+ hw->avf.aq.num_arq_entries = ICE_DCF_AQ_LEN;
+ hw->avf.aq.num_asq_entries = ICE_DCF_AQ_LEN;
+ hw->avf.aq.arq_buf_size = ICE_DCF_AQ_BUF_SZ;
+ hw->avf.aq.asq_buf_size = ICE_DCF_AQ_BUF_SZ;
+
+ rte_spinlock_init(&hw->vc_cmd_send_lock);
+ rte_spinlock_init(&hw->vc_cmd_queue_lock);
+ TAILQ_INIT(&hw->vc_cmd_queue);
+
+ hw->arq_buf = rte_zmalloc("arq_buf", ICE_DCF_AQ_BUF_SZ, 0);
+ if (hw->arq_buf == NULL) {
+ PMD_INIT_LOG(ERR, "unable to allocate AdminQ buffer memory");
+ goto err;
+ }
+
+ ret = iavf_set_mac_type(&hw->avf);
+ if (ret) {
+ PMD_INIT_LOG(ERR, "set_mac_type failed: %d", ret);
+ goto err;
+ }
+
+ ret = ice_dcf_check_reset_done(hw);
+ if (ret) {
+ PMD_INIT_LOG(ERR, "VF is still resetting");
+ goto err;
+ }
+
+ ret = iavf_init_adminq(&hw->avf);
+ if (ret) {
+ PMD_INIT_LOG(ERR, "init_adminq failed: %d", ret);
+ goto err;
+ }
+
+ if (ice_dcf_init_check_api_version(hw)) {
+ PMD_INIT_LOG(ERR, "check_api version failed");
+ goto err_api;
+ }
+
+ hw->vf_res = rte_zmalloc("vf_res", ICE_DCF_VF_RES_BUF_SZ, 0);
+ if (hw->vf_res == NULL) {
+ PMD_INIT_LOG(ERR, "unable to allocate vf_res memory");
+ goto err_api;
+ }
+
+ if (ice_dcf_get_vf_resource(hw)) {
+ PMD_INIT_LOG(ERR, "Failed to get VF resource");
+ goto err_alloc;
+ }
+
+ rte_intr_callback_register(&pci_dev->intr_handle,
+ ice_dcf_dev_interrupt_handler, hw);
+ rte_intr_enable(&pci_dev->intr_handle);
+ ice_dcf_enable_irq0(hw);
+
+ return 0;
+
+err_alloc:
+ rte_free(hw->vf_res);
+err_api:
+ iavf_shutdown_adminq(&hw->avf);
+err:
+ rte_free(hw->arq_buf);
+
+ return -1;
+}
+
+void
+ice_dcf_uninit_hw(struct rte_eth_dev *eth_dev, struct ice_dcf_hw *hw)
+{
+ struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_intr_handle *intr_handle = &pci_dev->intr_handle;
+
+ ice_dcf_disable_irq0(hw);
+ rte_intr_disable(intr_handle);
+ rte_intr_callback_unregister(intr_handle,
+ ice_dcf_dev_interrupt_handler, hw);
+
+ iavf_shutdown_adminq(&hw->avf);
+
+ rte_free(hw->arq_buf);
+ rte_free(hw->vf_res);
+}
diff --git a/drivers/net/ice/ice_dcf.h b/drivers/net/ice/ice_dcf.h
new file mode 100644
index 000000000..f44c09db2
--- /dev/null
+++ b/drivers/net/ice/ice_dcf.h
@@ -0,0 +1,52 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#ifndef _ICE_DCF_H_
+#define _ICE_DCF_H_
+
+#include <rte_ethdev_driver.h>
+
+#include <iavf_prototype.h>
+#include <iavf_adminq_cmd.h>
+#include <iavf_type.h>
+
+#include "ice_logs.h"
+
+struct dcf_virtchnl_cmd {
+ TAILQ_ENTRY(dcf_virtchnl_cmd) next;
+
+ enum virtchnl_ops v_op;
+ enum iavf_status v_ret;
+
+ uint16_t req_msglen;
+ uint8_t *req_msg;
+
+ uint16_t rsp_msglen;
+ uint16_t rsp_buflen;
+ uint8_t *rsp_msgbuf;
+
+ volatile int pending;
+};
+
+struct ice_dcf_hw {
+ struct iavf_hw avf;
+
+ rte_spinlock_t vc_cmd_send_lock;
+ rte_spinlock_t vc_cmd_queue_lock;
+ TAILQ_HEAD(, dcf_virtchnl_cmd) vc_cmd_queue;
+ uint8_t *arq_buf;
+
+ struct virtchnl_version_info virtchnl_version;
+ struct virtchnl_vf_resource *vf_res; /* VF resource */
+ struct virtchnl_vsi_resource *vsi_res; /* LAN VSI */
+ uint16_t vsi_id;
+};
+
+int ice_dcf_execute_virtchnl_cmd(struct ice_dcf_hw *hw,
+ struct dcf_virtchnl_cmd *cmd);
+
+int ice_dcf_init_hw(struct rte_eth_dev *eth_dev, struct ice_dcf_hw *hw);
+void ice_dcf_uninit_hw(struct rte_eth_dev *eth_dev, struct ice_dcf_hw *hw);
+
+#endif /* _ICE_DCF_H_ */
diff --git a/drivers/net/ice/ice_dcf_ethdev.c b/drivers/net/ice/ice_dcf_ethdev.c
new file mode 100644
index 000000000..23f82a487
--- /dev/null
+++ b/drivers/net/ice/ice_dcf_ethdev.c
@@ -0,0 +1,317 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#include <errno.h>
+#include <stdbool.h>
+#include <sys/types.h>
+#include <sys/ioctl.h>
+#include <unistd.h>
+
+#include <rte_interrupts.h>
+#include <rte_debug.h>
+#include <rte_pci.h>
+#include <rte_atomic.h>
+#include <rte_eal.h>
+#include <rte_ether.h>
+#include <rte_ethdev_pci.h>
+#include <rte_kvargs.h>
+#include <rte_malloc.h>
+#include <rte_memzone.h>
+#include <rte_dev.h>
+
+#include <iavf_devids.h>
+
+#include "ice_generic_flow.h"
+#include "ice_dcf_ethdev.h"
+
+static uint16_t
+ice_dcf_recv_pkts(__rte_unused void *rx_queue,
+ __rte_unused struct rte_mbuf **bufs,
+ __rte_unused uint16_t nb_pkts)
+{
+ return 0;
+}
+
+static uint16_t
+ice_dcf_xmit_pkts(__rte_unused void *tx_queue,
+ __rte_unused struct rte_mbuf **bufs,
+ __rte_unused uint16_t nb_pkts)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_start(struct rte_eth_dev *dev)
+{
+ dev->data->dev_link.link_status = ETH_LINK_UP;
+
+ return 0;
+}
+
+static void
+ice_dcf_dev_stop(struct rte_eth_dev *dev)
+{
+ dev->data->dev_link.link_status = ETH_LINK_DOWN;
+}
+
+static int
+ice_dcf_dev_configure(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_info_get(struct rte_eth_dev *dev,
+ struct rte_eth_dev_info *dev_info)
+{
+ struct ice_dcf_adapter *adapter = dev->data->dev_private;
+
+ dev_info->max_mac_addrs = 1;
+ dev_info->max_rx_pktlen = (uint32_t)-1;
+ dev_info->max_rx_queues = RTE_DIM(adapter->rxqs);
+ dev_info->max_tx_queues = RTE_DIM(adapter->txqs);
+
+ return 0;
+}
+
+static int
+ice_dcf_stats_get(__rte_unused struct rte_eth_dev *dev,
+ __rte_unused struct rte_eth_stats *igb_stats)
+{
+ return 0;
+}
+
+static int
+ice_dcf_stats_reset(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_promiscuous_enable(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_promiscuous_disable(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_allmulticast_enable(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_allmulticast_disable(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_filter_ctrl(struct rte_eth_dev *dev,
+ enum rte_filter_type filter_type,
+ __rte_unused enum rte_filter_op filter_op,
+ __rte_unused void *arg)
+{
+ int ret = 0;
+
+ if (!dev)
+ return -EINVAL;
+
+ switch (filter_type) {
+ default:
+ PMD_DRV_LOG(WARNING, "Filter type (%d) not supported",
+ filter_type);
+ ret = -EINVAL;
+ break;
+ }
+
+ return ret;
+}
+
+static void
+ice_dcf_dev_close(struct rte_eth_dev *dev)
+{
+ struct ice_dcf_adapter *adapter = dev->data->dev_private;
+
+ if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+ return;
+
+ dev->dev_ops = NULL;
+ dev->rx_pkt_burst = NULL;
+ dev->tx_pkt_burst = NULL;
+ dev->data->mac_addrs = NULL;
+
+ ice_dcf_uninit_hw(dev, &adapter->real_hw);
+}
+
+static void
+ice_dcf_queue_release(__rte_unused void *q)
+{
+}
+
+static int
+ice_dcf_link_update(__rte_unused struct rte_eth_dev *dev,
+ __rte_unused int wait_to_complete)
+{
+ return 0;
+}
+
+static int
+ice_dcf_rx_queue_setup(struct rte_eth_dev *dev,
+ uint16_t rx_queue_id,
+ __rte_unused uint16_t nb_rx_desc,
+ __rte_unused unsigned int socket_id,
+ __rte_unused const struct rte_eth_rxconf *rx_conf,
+ __rte_unused struct rte_mempool *mb_pool)
+{
+ struct ice_dcf_adapter *adapter = dev->data->dev_private;
+
+ dev->data->rx_queues[rx_queue_id] = &adapter->rxqs[rx_queue_id];
+
+ return 0;
+}
+
+static int
+ice_dcf_tx_queue_setup(struct rte_eth_dev *dev,
+ uint16_t tx_queue_id,
+ __rte_unused uint16_t nb_tx_desc,
+ __rte_unused unsigned int socket_id,
+ __rte_unused const struct rte_eth_txconf *tx_conf)
+{
+ struct ice_dcf_adapter *adapter = dev->data->dev_private;
+
+ dev->data->tx_queues[tx_queue_id] = &adapter->txqs[tx_queue_id];
+
+ return 0;
+}
+
+static const struct eth_dev_ops ice_dcf_eth_dev_ops = {
+ .dev_start = ice_dcf_dev_start,
+ .dev_stop = ice_dcf_dev_stop,
+ .dev_close = ice_dcf_dev_close,
+ .dev_configure = ice_dcf_dev_configure,
+ .dev_infos_get = ice_dcf_dev_info_get,
+ .rx_queue_setup = ice_dcf_rx_queue_setup,
+ .tx_queue_setup = ice_dcf_tx_queue_setup,
+ .rx_queue_release = ice_dcf_queue_release,
+ .tx_queue_release = ice_dcf_queue_release,
+ .link_update = ice_dcf_link_update,
+ .stats_get = ice_dcf_stats_get,
+ .stats_reset = ice_dcf_stats_reset,
+ .promiscuous_enable = ice_dcf_dev_promiscuous_enable,
+ .promiscuous_disable = ice_dcf_dev_promiscuous_disable,
+ .allmulticast_enable = ice_dcf_dev_allmulticast_enable,
+ .allmulticast_disable = ice_dcf_dev_allmulticast_disable,
+ .filter_ctrl = ice_dcf_dev_filter_ctrl,
+};
+
+static int
+ice_dcf_dev_init(struct rte_eth_dev *eth_dev)
+{
+ struct ice_dcf_adapter *adapter = eth_dev->data->dev_private;
+
+ eth_dev->dev_ops = &ice_dcf_eth_dev_ops;
+ eth_dev->rx_pkt_burst = ice_dcf_recv_pkts;
+ eth_dev->tx_pkt_burst = ice_dcf_xmit_pkts;
+
+ if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+ return 0;
+
+ eth_dev->data->dev_flags |= RTE_ETH_DEV_CLOSE_REMOVE;
+
+ if (ice_dcf_init_hw(eth_dev, &adapter->real_hw) != 0) {
+ PMD_INIT_LOG(ERR, "Failed to init DCF hardware");
+ return -1;
+ }
+
+ rte_eth_random_addr(adapter->mac_addr.addr_bytes);
+ eth_dev->data->mac_addrs = &adapter->mac_addr;
+
+ return 0;
+}
+
+static int
+ice_dcf_dev_uninit(struct rte_eth_dev *eth_dev)
+{
+ ice_dcf_dev_close(eth_dev);
+
+ return 0;
+}
+
+static int
+ice_dcf_cap_check_handler(__rte_unused const char *key,
+ const char *value, __rte_unused void *opaque)
+{
+ if (strcmp(value, "dcf"))
+ return -1;
+
+ return 0;
+}
+
+static int
+ice_dcf_cap_selected(struct rte_devargs *devargs)
+{
+ struct rte_kvargs *kvlist;
+ const char *key = "cap";
+ int ret = 0;
+
+ if (devargs == NULL)
+ return 0;
+
+ kvlist = rte_kvargs_parse(devargs->args, NULL);
+ if (kvlist == NULL)
+ return 0;
+
+ if (!rte_kvargs_count(kvlist, key))
+ goto exit;
+
+ /* dcf capability selected when there's a key-value pair: cap=dcf */
+ if (rte_kvargs_process(kvlist, key,
+ ice_dcf_cap_check_handler, NULL) < 0)
+ goto exit;
+
+ ret = 1;
+
+exit:
+ rte_kvargs_free(kvlist);
+ return ret;
+}
+
+static int eth_ice_dcf_pci_probe(__rte_unused struct rte_pci_driver *pci_drv,
+ struct rte_pci_device *pci_dev)
+{
+ if (!ice_dcf_cap_selected(pci_dev->device.devargs))
+ return 1;
+
+ return rte_eth_dev_pci_generic_probe(pci_dev,
+ sizeof(struct ice_dcf_adapter),
+ ice_dcf_dev_init);
+}
+
+static int eth_ice_dcf_pci_remove(struct rte_pci_device *pci_dev)
+{
+ return rte_eth_dev_pci_generic_remove(pci_dev, ice_dcf_dev_uninit);
+}
+
+static const struct rte_pci_id pci_id_ice_dcf_map[] = {
+ { RTE_PCI_DEVICE(IAVF_INTEL_VENDOR_ID, IAVF_DEV_ID_ADAPTIVE_VF) },
+ { .vendor_id = 0, /* sentinel */ },
+};
+
+static struct rte_pci_driver rte_ice_dcf_pmd = {
+ .id_table = pci_id_ice_dcf_map,
+ .drv_flags = RTE_PCI_DRV_NEED_MAPPING,
+ .probe = eth_ice_dcf_pci_probe,
+ .remove = eth_ice_dcf_pci_remove,
+};
+
+RTE_PMD_REGISTER_PCI(net_ice_dcf, rte_ice_dcf_pmd);
+RTE_PMD_REGISTER_PCI_TABLE(net_ice_dcf, pci_id_ice_dcf_map);
+RTE_PMD_REGISTER_KMOD_DEP(net_ice_dcf, "* igb_uio | vfio-pci");
+RTE_PMD_REGISTER_PARAM_STRING(net_ice_dcf, "cap=dcf");
diff --git a/drivers/net/ice/ice_dcf_ethdev.h b/drivers/net/ice/ice_dcf_ethdev.h
new file mode 100644
index 000000000..0c34a0095
--- /dev/null
+++ b/drivers/net/ice/ice_dcf_ethdev.h
@@ -0,0 +1,24 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#ifndef _ICE_DCF_ETHDEV_H_
+#define _ICE_DCF_ETHDEV_H_
+
+#include "ice_ethdev.h"
+#include "ice_dcf.h"
+
+#define ICE_DCF_MAX_RINGS 1
+
+struct ice_dcf_queue {
+ uint64_t dummy;
+};
+
+struct ice_dcf_adapter {
+ struct ice_dcf_hw real_hw;
+ struct rte_ether_addr mac_addr;
+ struct ice_dcf_queue rxqs[ICE_DCF_MAX_RINGS];
+ struct ice_dcf_queue txqs[ICE_DCF_MAX_RINGS];
+};
+
+#endif /* _ICE_DCF_ETHDEV_H_ */
diff --git a/drivers/net/ice/meson.build b/drivers/net/ice/meson.build
index f9e897bbc..0ba9668d1 100644
--- a/drivers/net/ice/meson.build
+++ b/drivers/net/ice/meson.build
@@ -15,8 +15,8 @@ sources = files(
'ice_hash.c'
)
-deps += ['hash']
-includes += include_directories('base')
+deps += ['hash', 'net', 'common_iavf']
+includes += include_directories('base', '../../common/iavf')
if arch_subdir == 'x86'
sources += files('ice_rxtx_vec_sse.c')
@@ -37,4 +37,7 @@ if arch_subdir == 'x86'
endif
endif
+sources += files('ice_dcf.c',
+ 'ice_dcf_ethdev.c')
+
install_headers('rte_pmd_ice.h')
diff --git a/mk/rte.app.mk b/mk/rte.app.mk
index d295ca0a5..f3798a09f 100644
--- a/mk/rte.app.mk
+++ b/mk/rte.app.mk
@@ -185,6 +185,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_I40E_PMD) += -lrte_pmd_i40e
_LDLIBS-$(CONFIG_RTE_LIBRTE_IAVF_PMD) += -lrte_pmd_iavf
_LDLIBS-$(CONFIG_RTE_LIBRTE_ICE_PMD) += -lrte_pmd_ice
IAVF-y := $(CONFIG_RTE_LIBRTE_IAVF_PMD)
+IAVF-y += $(CONFIG_RTE_LIBRTE_ICE_PMD)
ifeq ($(findstring y,$(IAVF-y)),y)
_LDLIBS-y += -lrte_common_iavf
endif
--
2.25.1
^ permalink raw reply [relevance 1%]
* Re: [dpdk-dev] [PATCH 00/16] NXP DPAAx fixes and enhancements
2020-03-06 10:12 3% ` David Marchand
@ 2020-03-10 10:36 3% ` Dodji Seketeli
0 siblings, 0 replies; 200+ results
From: Dodji Seketeli @ 2020-03-10 10:36 UTC (permalink / raw)
To: David Marchand
Cc: Hemant Agrawal (OSS), Yigit, Ferruh, dev, Neil Horman, Thomas Monjalon
Hello,
David Marchand <david.marchand@redhat.com> writes:
> On Thu, Mar 5, 2020 at 10:19 AM Hemant Agrawal (OSS)
> <hemant.agrawal@oss.nxp.com> wrote:
>> > On Thu, Mar 5, 2020 at 10:06 AM Hemant Agrawal (OSS)
>> > <hemant.agrawal@oss.nxp.com> wrote:
>> > >
>> > > Hi David,
>> > > > On Mon, Mar 2, 2020 at 10:26 AM Hemant Agrawal
>> > > > <hemant.agrawal@nxp.com> wrote:
>> > > > >
>> > > > > This patch series add various patches for enhancing and fixing NXP
>> > > > > fslmc bus, dpaa bus, and dpaax.
>> > > > >
>> > > > > - the main change is support to allow thread migration across
>> > > > > lcores
>> > > > > - improving the multi-process support
>> > > >
>> > > > This series triggers an ABI warning that must be investigated.
>> > > >https://travis-ci.com/ovsrobot/dpdk/jobs/292904119#L2233
>> > >
>> > > [Hemant]
>> > > As per the logs:
>> > >
>> > > Variables changes summary: 1 Removed, 2 Changed, 0 Added variables
>> > > 1 Removed variable:
>> > > 'dpaa2_portal_dqrr per_lcore_dpaa2_held_bufs'
>> > {per_lcore_dpaa2_held_bufs@@DPDK_20.0}
>> > > 2 Changed variables:
>> > > [C]'dpaa2_io_portal_t dpaa2_io_portal[128]' was changed at
>> > dpaa2_hw_dpio.h:40:1: size of symbol changed from 5120 to 2048
>> > > [C]'dpaa2_io_portal_t per_lcore__dpaa2_io' was changed at
>> > > dpaa2_hw_dpio.h:20:1: size of symbol changed from 40 to 16
>> > >
>> > > Error: ABI issue reported for 'abidiff --suppr devtools/libabigail.abignore --
>> > no-added-syms --headers-dir1 reference/usr/local/include --headers-dir2
>> > install/usr/local/include reference/dump/librte_bus_fslmc.dump
>> > install/dump/librte_bus_fslmc.dump'
>> > >
>> > > ---------------
>> > >
>> > > These changes are w.r.t modifications in internal structures and variables.
>> > They may be ignored.
>> >
>> > The ABI check considers symbol exposed in headers available to final users.
>> > If those are internal, why are the headers public?
>> >
>>
>> [Hemant] These symbols are not part of any public header files, but
>> they are part of *.map files to share them between different driver
>> libs i.e bus_fslmc and net_dpaa2
>
> I would expect libabigail to skip those symbols, so there is something
> I have missed in how --headers-dirX work.
In libabigail speak, we make a difference between *ELF symbols* and
types.
--header-dirX is about telling the tool what the public *types* are. As
you rightfully implied, types that are defined in files that are not
found in the directories specified by --header-dirX are considered to be
private types and are thus not shown in the ABI change report.
ELF symbols however are a different matter. Header files don't usually
define ELF symbols, be they variable of function symbols. Header files
can at most declare variables or functions that would be actually
defined elsewhere in source code, leading to the definition of ELF
variable or function symbols in the final binary. At this point, we
aren't talking about types anymore, as the ELF format doesn't know what
types (in C or any other language) are. So --header-dirX don't deal with
ELF symbols.
And from what I understand from the message quoted above, the changes we
are talking about have to do with EFL variable symbols which size have
changed. So in practise, these are global arrays (exposed by at the
binary level as an ELF variable symbol of a given size) with public
visibility which size have changed.
So my guess would be that if you guys don't want these arrays to be part
the binary interface of this library, they should probably be declared
static at the C level and accessed through some accessor function or
something like that. At least, that's my humble uninformed opinion.
In the mean time, the tooling can be tought to ignore changes to these
ELF symbols, as you you guys all know already.
> Anyway, all of those symbols in dpaa are part of the driver ABI.
> We are still missing a way to mark internal symbols.
> Neil had posted a framework for this
> http://patchwork.dpdk.org/project/dpdk/list/?series=5004.
>
> In order to get this series passing the checks, I recommend NXP
> rebasing Neil scripts (I will help reviewing this part), then mark all
> those symbols as internal in its drivers.
> Other vendor will convert their drivers later, as there is no need at
> the moment.
>
> Thanks.
Cheers,
--
Dodji
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH] lib/librte_net/rte_ether.h:changed RTE_ETHER_MAX_LEN
@ 2020-03-10 15:18 3% ` Stephen Hemminger
0 siblings, 0 replies; 200+ results
From: Stephen Hemminger @ 2020-03-10 15:18 UTC (permalink / raw)
To: Muhammad Ahmad; +Cc: olivier.matz, dev, stable
On Tue, 10 Mar 2020 11:50:36 +0500
Muhammad Ahmad <muhammad.ahmad@emumba.com> wrote:
> Change the RTE_ETHER_MAX_LEN from 1518 to 1526
>
> Bugzilla ID: 296
>
> Fixes: c5b2d13 (net: add rte prefix to ether defines)
>
> Cc: stable@dpdk.org
> Cc: Oliver Matz <olivier.matz@6wind.com>
>
> Reported-by: LAVA <lavaraj@gmail.com>
> Suggested-by: LAVA <lavaraj@gmail.com>
> Signed-off-by: Muhammad Ahmad <muhammad.ahmad@emumba.com>
> ---
> lib/librte_net/rte_ether.h | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/lib/librte_net/rte_ether.h b/lib/librte_net/rte_ether.h
> index e069dc7fe..8953107f3 100644
> --- a/lib/librte_net/rte_ether.h
> +++ b/lib/librte_net/rte_ether.h
> @@ -30,7 +30,7 @@ extern "C" {
> (RTE_ETHER_ADDR_LEN * 2 + \
> RTE_ETHER_TYPE_LEN) /**< Length of Ethernet header. */
> #define RTE_ETHER_MIN_LEN 64 /**< Minimum frame len, including CRC. */
> -#define RTE_ETHER_MAX_LEN 1518 /**< Maximum frame len, including CRC. */
> +#define RTE_ETHER_MAX_LEN 1526 /**< Maximum frame len, including CRC. */
> #define RTE_ETHER_MTU \
> (RTE_ETHER_MAX_LEN - RTE_ETHER_HDR_LEN - \
> RTE_ETHER_CRC_LEN) /**< Ethernet MTU. */
No that is abi change would impact a lot of code and drivers.
The problem is the comment is wrong, not the value.
^ permalink raw reply [relevance 3%]
* [dpdk-dev] [PATCH v3 0/7] vfio/pci: SR-IOV support
@ 2020-03-11 21:58 2% Alex Williamson
2020-03-19 6:32 0% ` Tian, Kevin
0 siblings, 1 reply; 200+ results
From: Alex Williamson @ 2020-03-11 21:58 UTC (permalink / raw)
To: kvm
Cc: linux-pci, linux-kernel, dev, mtosatti, thomas, bluca,
jerinjacobk, bruce.richardson, cohuck, kevin.tian
Only minor tweaks since v2, GET and SET on VFIO_DEVICE_FEATURE are
enforced mutually exclusive except with the PROBE option as suggested
by Connie, the modinfo text has been expanded for the opt-in to enable
SR-IOV support in the vfio-pci driver per discussion with Kevin.
I have not incorporated runtime warnings attempting to detect misuse
of SR-IOV or imposed a session lifetime of a VF token, both of which
were significant portions of the discussion of the v2 series. Both of
these also seem to impose a usage model or make assumptions about VF
resource usage or configuration requirements that don't seem necessary
except for the sake of generating a warning or requiring an otherwise
unnecessary and implicit token reinitialization. If there are new
thoughts around these or other discussion points, please raise them.
Series overview (same as provided with v1):
The synopsis of this series is that we have an ongoing desire to drive
PCIe SR-IOV PFs from userspace with VFIO. There's an immediate need
for this with DPDK drivers and potentially interesting future use
cases in virtualization. We've been reluctant to add this support
previously due to the dependency and trust relationship between the
VF device and PF driver. Minimally the PF driver can induce a denial
of service to the VF, but depending on the specific implementation,
the PF driver might also be responsible for moving data between VFs
or have direct access to the state of the VF, including data or state
otherwise private to the VF or VF driver.
To help resolve these concerns, we introduce a VF token into the VFIO
PCI ABI, which acts as a shared secret key between drivers. The
userspace PF driver is required to set the VF token to a known value
and userspace VF drivers are required to provide the token to access
the VF device. If a PF driver is restarted with VF drivers in use, it
must also provide the current token in order to prevent a rogue
untrusted PF driver from replacing a known driver. The degree to
which this new token is considered secret is left to the userspace
drivers, the kernel intentionally provides no means to retrieve the
current token.
Note that the above token is only required for this new model where
both the PF and VF devices are usable through vfio-pci. Existing
models of VFIO drivers where the PF is used without SR-IOV enabled
or the VF is bound to a userspace driver with an in-kernel, host PF
driver are unaffected.
The latter configuration above also highlights a new inverted scenario
that is now possible, a userspace PF driver with in-kernel VF drivers.
I believe this is a scenario that should be allowed, but should not be
enabled by default. This series includes code to set a default
driver_override for VFs sourced from a vfio-pci user owned PF, such
that the VFs are also bound to vfio-pci. This model is compatible
with tools like driverctl and allows the system administrator to
decide if other bindings should be enabled. The VF token interface
above exists only between vfio-pci PF and VF drivers, once a VF is
bound to another driver, the administrator has effectively pronounced
the device as trusted. The vfio-pci driver will note alternate
binding in dmesg for logging and debugging purposes.
Please review, comment, and test. The example QEMU implementation
provided with the RFC is still current for this version. Thanks,
Alex
RFC: https://lore.kernel.org/lkml/158085337582.9445.17682266437583505502.stgit@gimli.home/
v1: https://lore.kernel.org/lkml/158145472604.16827.15751375540102298130.stgit@gimli.home/
v2: https://lore.kernel.org/lkml/158213716959.17090.8399427017403507114.stgit@gimli.home/
---
Alex Williamson (7):
vfio: Include optional device match in vfio_device_ops callbacks
vfio/pci: Implement match ops
vfio/pci: Introduce VF token
vfio: Introduce VFIO_DEVICE_FEATURE ioctl and first user
vfio/pci: Add sriov_configure support
vfio/pci: Remove dev_fmt definition
vfio/pci: Cleanup .probe() exit paths
drivers/vfio/pci/vfio_pci.c | 390 +++++++++++++++++++++++++++++++++--
drivers/vfio/pci/vfio_pci_private.h | 10 +
drivers/vfio/vfio.c | 20 +-
include/linux/vfio.h | 4
include/uapi/linux/vfio.h | 37 +++
5 files changed, 433 insertions(+), 28 deletions(-)
^ permalink raw reply [relevance 2%]
* [dpdk-dev] [PATCH v3 2/7] net/ice: add the DCF hardware initialization
@ 2020-03-16 5:52 1% ` Haiyue Wang
0 siblings, 0 replies; 200+ results
From: Haiyue Wang @ 2020-03-16 5:52 UTC (permalink / raw)
To: dev, xiaolong.ye, qi.z.zhang, qiming.yang, beilei.xing
Cc: wei.zhao1, Haiyue Wang
Introduce the DCF (Device Config Function) feature in the ice PMD, it
works as a standalone PMD which doesn't handle the packet Rx/Tx related
things. Its hardware entity is the VF.
Add the basic DCF hardware initialization, this is specified by devarg
'cap=dcf'.
Signed-off-by: Haiyue Wang <haiyue.wang@intel.com>
---
doc/guides/nics/ice.rst | 47 +++
doc/guides/nics/img/ice_dcf.png | Bin 0 -> 39168 bytes
doc/guides/rel_notes/release_20_05.rst | 5 +
drivers/common/Makefile | 1 +
drivers/net/ice/Makefile | 5 +
drivers/net/ice/ice_dcf.c | 474 +++++++++++++++++++++++++
drivers/net/ice/ice_dcf.h | 52 +++
drivers/net/ice/ice_dcf_ethdev.c | 317 +++++++++++++++++
drivers/net/ice/ice_dcf_ethdev.h | 24 ++
drivers/net/ice/meson.build | 7 +-
mk/rte.app.mk | 1 +
11 files changed, 931 insertions(+), 2 deletions(-)
create mode 100644 doc/guides/nics/img/ice_dcf.png
create mode 100644 drivers/net/ice/ice_dcf.c
create mode 100644 drivers/net/ice/ice_dcf.h
create mode 100644 drivers/net/ice/ice_dcf_ethdev.c
create mode 100644 drivers/net/ice/ice_dcf_ethdev.h
diff --git a/doc/guides/nics/ice.rst b/doc/guides/nics/ice.rst
index 8af32dabf..2639ae239 100644
--- a/doc/guides/nics/ice.rst
+++ b/doc/guides/nics/ice.rst
@@ -240,6 +240,53 @@ report a MDD event and drop the packets.
The APPs based on DPDK should avoid providing such packets.
+Device Config Function (DCF)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This section demonstrates ICE DCF PMD, which shares the core module with ICE
+PMD and iAVF PMD.
+
+A DCF (Device Config Function) PMD bounds to the device's trusted VF with ID 0,
+it can act as a sole controlling entity to exercise advance functionality (such
+as switch, ACL) for the rest VFs.
+
+The DCF PMD needs to advertise and acquire DCF capability which allows DCF to
+send AdminQ commands that it would like to execute over to the PF and receive
+responses for the same from PF.
+
+.. _figure_ice_dcf:
+
+.. figure:: img/ice_dcf.*
+
+ DCF Communication flow.
+
+#. Create the VFs::
+
+ echo 4 > /sys/bus/pci/devices/0000\:18\:00.0/sriov_numvfs
+
+#. Enable the VF0 trust on::
+
+ ip link set dev enp24s0f0 vf 0 trust on
+
+#. Bind the VF0, and run testpmd with 'cap=dcf' devarg::
+
+ testpmd -l 22-25 -n 4 -w 18:01.0,cap=dcf -- -i
+
+#. Monitor the VF2 interface network traffic::
+
+ tcpdump -e -nn -i enp24s1f2
+
+#. Create one flow to redirect the traffic to VF2 by DCF::
+
+ flow create 0 priority 0 ingress pattern eth / ipv4 src is 192.168.0.2 \
+ dst is 192.168.0.3 / end actions vf id 2 / end
+
+#. Send the packet, and it should be displayed on tcpdump::
+
+ sendp(Ether(src='3c:fd:fe:aa:bb:78', dst='00:00:00:01:02:03')/IP(src=' \
+ 192.168.0.2', dst="192.168.0.3")/TCP(flags='S')/Raw(load='XXXXXXXXXX'), \
+ iface="enp24s0f0", count=10)
+
Sample Application Notes
------------------------
diff --git a/doc/guides/nics/img/ice_dcf.png b/doc/guides/nics/img/ice_dcf.png
new file mode 100644
index 0000000000000000000000000000000000000000..540823c4a05e9ea9d558d233fea77016d1ae5df3
GIT binary patch
literal 39168
zcmZ6ybzD?k*FH`NDBT?jlF}e8B?Hn74BeolfOLbDz);d%Lw70NNOyNPNO%7Z_x(KY
z`+k3a@X<4KW}m&+Uh7)dy4F6yN(xdKsHCWHaBvvXpCpvw;1KwLKlqmjz$g7i%71`=
z;O&*A#NkSZ$ajH@7p7wJVsLO35oq^@NWk^0Z=W>n;oxu`p8w#NOsHJo;1Xh`CBz^u
zdIzZ}I(W`x-6a%La{^jclNOU&`_)rTS4-}zY76d1P4w1CVq)-OmrKC^5#ZY~-}pzu
zy$L?@i@zdO6jl2qp0>O_?^V0~==_TKu;0E)6AT_jK}Ys$GuU9xpl6NmUq4AAAS8rZ
z5<h>~BQL7h+1bg!^v{JkDk^GFZXt;WZK)wU@I4Lh>Nma-8Py=vX58uL5BwsNv0+TC
ztdd|ub|{f>I#qBR_r=Ecc2+(+F$m*+POs4gtgWq$;%5$Zz1W+)IolS4AR!?E!%MzF
zY^hc+(<6BKV$B>q*BptIQ5gaPfd*IJosgp=XT25!8zwm^L5M@!Q9uFabHF7#DiuCK
zt~qeo#^9j@Nyw!ThkWHBd%m>z=MtX)IqO&q%qUT-Q<|7KJNMhNs;a6pC&Kl7XSB)v
z3KtLW;^N}!aM9A*nl<f&LypeAH;UH%@~=<W?^;Mic5d!cle_a$qiaVX7S(%;iC{c>
zPEJmL@k|#dzb#-e#WW$n3TwM=5&|%zK#w5Rj~l++vL_;_xx2ZfV1g5o&-OTVud|dg
zY)p8a+CN(%s#Jb5B@9dFyIC8&!|9!kw<)+dnAUfkUON7RZe*>j5V@t@LFIjKzBWN}
zl6bSB?AXD@u-6mJ^$s$%>{fg;H**<Wy_z$R_0fFyxQl_NC8(r`t&yYbC$Wq!ZhWiR
zaQO>2)_CPI7Ur}ViP+8SHXAZ(qCX-8Pgw>@)`d8#Ub}%C2oieVUWG5$Z;wDPw0oh!
z(MS2V(Mt&eBO3Siyc1oDi4_JnN!?`_!F7~4=LJ<;7i4@c?GuTsp&oXOubJN~j*nRE
zr*(v=l`h&IUu)Y-G0<(qS?nZJwYZ$k+*tNcTf2SvjF$-h6dEDNR%>@OyFp@~wx8sm
zQ@KkHCa?M<DDa*q5JkVi|7}X6ooQlUmnxZR*#{T+GPIBvv|B=xJdNM>M<?%wx7c)l
zdn}q*;Wg|Gzyh^yRZNGE1Xt2+qw~^RE=E{lE|-=pha<-cjHj$U3eSXtS-y9ciA;Qt
zg#X2CIXogbtk+gV#pepYbkK~qZROc+2-nD3e6SS4d#n~;vgmMgi02}r<bB>}Fg(g-
zTDmx2YrO0kRzm_}D5S%J%DnQ0DdV>iV*NTt#-ch%|0xYIm~S%Q_BwvyfdQLqe%iUU
za<4uRAr<XM?#|y&3?~<aXTr-Hgaz#tT;9D;cK$2uv9~aYB8Zi$G+Wuxv-FN9+5PYR
zh4qI=trjcZ;=wL@A=W&9;v>?i^7Yn}O(rs$G1Q(|Z2z1sn<Yns)V;&#c$@TE2Q)Ru
z)6)`Bt!gQksJ*lfL=-F}Q=R&|&-+U>DkcK~7vFVi4u!dmmta<R;msWO$55HcDm$Mq
zYzuC++dJmp5`L=_vy^#UbXhfEraEBcG`ZP)uwPnpl~qe?xmmf+NiO2ax1Om+(c=}a
z!GBS0w-`sWU7FHRP;K|H)xhT98<o%yk#hLBt#8R=)!=YcP~pc5m_`)LD5eB~iYqs_
zj?tLy72jM<E{lrbr1en>*bVr_*W;KI;3E7^sdYFjoC{%b$NVtKVTT^7+2BYNw7pYj
zi4R5^w->W9pqU==f1#Eb;6C!>z+DXeb>TXZwZp;G`tJ%B5BYEu=0)FDN%x4fNM_s}
z&3d^+eysZRo^x@R*na2q(uHz@x%b$Uaf>KVBm(XH446M${l^6#mQ=@EaW_yr=YvCd
z+ccCShFQ?$1}?+o10_*^aev+*-pN^+qy-`EqoDYuWpSd~d>GvmJf>Q%JrCl>)WF6k
zmNOMMMp+hgneL-`A#pMMbJc=vS<;lS;}#5&f&14fUK<#DZv350Jd6LHkN$NrFqGJ8
zlLQ^XPt!kNF7<(R!K>C~vVL6sXh^GeVu@|f9?Xx@{W00`WXRZv(|^df><;rf5PNdx
zBpMO7vZw8Z5tyP#0>Vqx1X*<dv&0Kc;4ZffjQvQ21Vb2S)~HS*e}&q3vP0X5Ck>{p
zL|PgxQ2d3lia{mhxO*i0u!<pNN+=Dr*xkF}rL@3)H_B6SSLCd3U(-4g(^%Ggi&~71
zLC)Opd9psBf9m^tF-R2=i~Y+Wx~{DJ7&!b`+!vr6(LbBVOG$#4e;21&m~FS))*G-=
zPa}wqkl#vdk|f?wwVSv9(#)clt1G+|;%v%L28)#;+6VXk{7iUG+jNKVTH?KH*Mg`P
z4}U@{EuUAnkMX-Hs~^?DoM;li?~cv8zY`mcjZ7JLTz<$aU4RE4@Y}g{&JAw#$bdHS
zIsj~7y#89(M|b^nq}$-hTBu4^z2sUdUo4>v8BBVg%wyU%_MXf@bERkY-D?x<$;O<b
zpvBDRefW<DBArvMWHxmVUG9o}n*WowASDGwz0FLH(Dzku6^(`Vv9TH-Or_3WQek9&
zp{?wvKs0uy5PokudTJp-5ZP7duzj+zYp+MDA<51+GTg}}+XPcyOCY#AF=5lm{bEKD
zuYwz`>Y=}WkR(4(Y2H+%2SHKL8L4sdgQdlyUB8->Fe_=z_$V35@6;>_^Sx+A)!#{y
z%vNutx>hi*td+hI$B#YrfhqfD<l~3DlzhC=#`503zteBJi#)&=g+hzrVxg$DkD{I@
zg^onlA0{~L`RpH-+6gSVF_?>R2n@E#DgbDpEt*<Z@je@xlV4BeZ%k2#RC;@KHPXI=
zTv4h`bGcu3y~M9wU$HGWe-9h0vD0j^8a@f(O+_$Xo~<oWUu<$5exr+)S7gD4{3wVn
zgrav3%g?CuuSSpayqwxe{k^|Fjm)EGSivv!zjH;?(yL3=fBQ=%uWad!A{EMEQA^#0
z1YSLDC4#7>-9zS&zAcI<8lQ|mDf7d6im6Q*Pa;o%x1R@B!{cXusK5DSe-NiT0+q(3
zJAul<M~5YH9~^lh-eTUP{6+YH%@Tq2W2W5mXeE;-FoLq>=#Q(^6LkG!bA|cqcF*pi
zs1U^jfwNgMV;A~2jK#)8*61YsCi9QR)xxWa4oeYf>?DNCwZ`dlvWs4bPp=e=C;^0@
zKx^mvbur_;)>z-nUx7=5Qf6*XICBRQBO-{5^FAFbsff<KpkO!a8>m~+y7p>YE4&LA
zuaOFs0yDxhFwwq*qMEm7{5K*3xDA#58|>mq?5K=|*ax&{U5}%6AKyg@FV2-FFZMsS
zrHF6K$F#kK%8>ZN{<(`uWDrDi_jGHzJr(KJ|1yPYc)l)sxy8H2+vzKLzZc?{M!E7v
zF8j7?Yx*gy+=(y$qRI15nG2g2=Obg4BUlV=;xP}#z2<_JKj<^R-sjvH4CwW2sL>TH
zBFbS!1ZIVePcY_hP7Oo2*3+(G-We}hkFa4=pbOcJV<M%e|C+Y!zUqG{n|J=*>)qtn
zuvYEZ1ox-boipvgxq`t_VP5lxlooZlyR(J3H(<AZF?Y%7yd%9i@7`?6^1pYM<8}vA
z(fcoCgsMf!oK2VWC%GQAgevYXL7o?Eii4GU?~V&J?0;;|X}e4MzW2UeoiA;v_Kja^
zfO*44nT%$s<HJ_iHD0JS+i49q+(`zs=pvVW8i+;xuYtKK!0_&IS4RuYlSNh0LB0N0
z@7M}4b%hGL51O*{(vrSeZ6D8OHCSs;@XmhzeVhf(Slk`m&D&IY@;78t2t{dNPCng^
zl3t`DCq?X(3hcH>ORIJ6{H~eR;2X%NKo&enSZ7_?BtrJ5+S7I!1*N++VP8<HIBoPS
zs4s)hk6=@{5NU`^H+?`wcP3Bej!F{iUVeW-Vmz5k<8mL=To^D!#+Hw%%ierPJ88H_
zPv82kH0g1Ai1s0S9toBo^Lh+BR#Ex!G~`x30&(~+!!jfFHn1X}Pg@a3JmuSLTQ#Qb
zpQ{H2%o6CZ4m_;3gUII{#r0Kl7wZMxrF?pq^i$O~lx>HQg7%WjlKr9^Cy2f+@^BqY
zbx<#R&U+j$A5c+HBI@EZBVkOHH&Z}NE>JzIVq7w@Q)x#`<VnX!98`B2r8m%6Vo+<#
z6|brd_*__wX0{t1GqA?_MGCa@sMxH?;yyPPF1TB!b^Hhn>~1g2KzO8);$a8uAH*&3
zg)D9M>q1*@_xCVo#(yS{1NMP%E6V?!Zdx45c`#Zo`Hos^`dDktCFvy1sVN@{zyDdY
z>><Z7B&4vz?kkf60-^cx)}7#;WUQ;|Ust^A{Cc6ps{`J_Ho33e7$a~vXbL1#t`aT!
z2UJ5oWg&|^mMJOh+0J)(WRFNE*ofb`(PH0t*QCEMv<$k$g`L*rpQiFCT-S{2>uUSm
zw<=5E{Hpn^91&*!WTH|0=HF?*gcfcA(20pesM@XX<znYR-nMzrF*R^)xxTPuJ-mhU
z$llE1dA!{69o)n5LZtfsGN@04B+k+hUE_rgg5;a&za<srs{J-<3Qy6+aIcv(Vh0~)
zkKgAXo_BfQU6|#09=7DXY~^inHrqTNHErF7_5Idg-n+BOIH|92e|zJ_nvaQy)}E~B
z(h@~J+X?{0YlgxkKbSHfz6PX3+q1^PZ8&z^?eZ~;#lzEVFypvJwzmp*NjFhor7Ydu
z;qg_Ml8d`7>U`!J{-r<=sz9Ex<rB%wndw8n3T-!>HqoW|-MF)wvnft=dI;3KrdhMf
zMAysew~LW;pSGKW3p2yd|KC%+rqf%cE><%ubiE&Jx1%0MZ=Vj7xf*$_KBZqWak)pv
z4j1_4XAlThcpZNinfEz(+YMJ<jd{>q^T3Cm4}mOujeA_~ly2A-OCo@BUWkDw$?;>`
zNAbhK1bj9N)%iBl%U+H3#(W(f7`9(~6|64(y#)u!7m=9$ls^wV6HaD*K=}$*?`*(r
z%PAmMv7h_OPY`2waY<I!<1y{9lkvg*S2VsV;C^W>H9yql*ma~+oMt}#AG;;>^x|`8
z#R49-yL6HlpGSwi9Z64GB2a6w($eJZ>gJ#mxlbpK{v}O&29pecI1p8Qp<h1SHvO4N
z_+F)%xO0841x)wd$&8Vw&1HJ3w?MV^H)!Ha$X1O>IO*|zpZSM6>*QdWiv2AWGUOWT
zi-U3Fb<QX6QPsrlDFCm9oEyVYRmw8*Eij=nt7ZU<rce$)Tpe9zp*)({+wiPnEysrt
z%KsJKDNEN+O_ku)jJ2#;k}_W1r057%=`a*slt2m_4Ow4Dxwv;-?t_o4?>G2;iM1t1
zHTq@j=TP}0C)-I{SAz4qYL-K^qweE9?(nlNI~&!^GA8sit}QIH$3qI@ShGe{waqKL
z$gP6l!{HMFCb=eGSRp?HQ$W^k5NfTr!FhVBHe2dKhrT@y7@j_LUMO2X!U<qQcl9^w
zAMYGS1h^`W*Fliy$>29w+j$DuOrED%X?afH7nQAm*uGMru4R&AA=<D2a2dhUa`0r@
zkTJbBK6OSuwLJ5|P9q(YMcV`M$U<Wo(*mXYZ8CH?GyY$Y{x{kJ@CmLnT3u;ei(W?I
zMzAO`+bv3<e%#6GvtLFB?dSqKV^E+x%tEbya&0sX)%=q7bl~KNG#gIFFumkEV<7hU
zq)iJq1xI5@jQ(S?U1mj9Smdun_!|Kx7{jS*z}2gF{sN8ft`X>R+<Q>}*9|j&!o)35
z{$bev7y}Xb>i>Ao|77I<=gZUAg;co|EjBB;+t~N>_f;{P9o0(>CsVsQIY)8(M+{Z0
zX0uI|msFKEtT~ttnsz7Q&HG#`N0?_r9`7GV%pEN$8bm$wuu-KbwUr2qDS>DJ*c!rf
zo^{1mo!8969ZL|;%-@E-i_s)2-OJ(P&$+jIm%hVOr64opW`*}ty|pQ4f6VyiOYbe#
zc=({mBPV?e10QS2MJS$rn8*$>vA(j}V^5F$v@VUb2^BbA=Q!jU&j8~69bkn$Rr&R-
zO%6w?WLzx_VTtLj2CV!KGIodUcx>y$1FafezYvGt?k~pa`aUwZ1UZA|ee$tkNv=Cc
z{JGZu?=XZZBj@Ofc4+btE(a9d-5DgAA*&B(**VS-x*~(_)GHAM5wR(${3Wbl#jKod
z%!O|u`G9Q+|2vP8d@VU)x3xNg57CaY!$@wqG0aZHEbB+A(>?I5v+>m<Uz;jVsLJ*J
zW~+@y6zO<$tl?|lNJMu4hC=%f7vE5WJWMT(IHLy3^({EaKP?YB7ke1l@pqa|SOJ6V
zP+$pl?BYl#$__WuF_p)}{q`1!uG%lf;Sa^0L}IxMpnos@Sn`)SbPo_}$YtIzMIKdC
zPs>vd6ZxC5y|K}HBV94S<<b$`HYa;Z!TZt^q6v86x58P{wkI2UoRXG>hZ~Pm)?ev(
zXVwAnw+Kn2g}4PvdewlLVqV+5V`7&VqCwG_bUp~XJRAD!ZJ?#2yhSQ*$+5}6kL+Ty
z7n`c?qP19mF#uIzF`@8+FFFm+?x5Zv7Kf||U6sF$ie@YBg?|s!O5Yd)@?QvRXPw9W
zzAFH=P#$oWOh*kT(7*Ed?XvkR4WNH}In6(}n`hUC2FjAX9nI&<5ll~Xlj0N(M~gM9
z7W;3xm?1qmFD1O@Yps*F!w=>vJEt4sE)}`;KqRl-pB?W8e==fRQHbNR&yXn9-CPzT
zX|2+4M8JL)^24fW-lP2u?B&<ST&l#FFZ)D-uFKJC<Kab1+ygzELfogXQQ&I#70tnM
zy54n{`>H0Fw|hk<)fQE<3!TG@ud#0no9o!?oNKC!@}4eEg2_~&7N3qT-Qsh9Iv>z{
zm)0<yr)u#7_;6@3bvjJz+ggqdL`H_^?hjYdPzWpB%A2F0&w3<JB_EHSqr-ppKXP7$
zrWt>;AauX_U`)J@pMV`1Q@7$z#}5b8W-NWwOd}B|VY?aWP5mhB2UD^`MzduiO()As
zi*rJP!fC{s1^J?Ot$QRN*3s2*LB2-Q@>P<~K2Ce@@ChWHB0qvX-#@OXT6O<z6!)t8
zuHHuvKO2MQPf~XiwS5gw^y}|f;*G`8qVqEEj|<{jf0-d9z;+?Hsd8<$bJw!TNAoog
z!lOHQ@wE`4mlgY=k-F55@ok?~VYgZ9ii?83DUY{@BQPW0Kt&6>3A?52TkfiF>Svqg
z?k5mH0g<X3N|2TzLCY-?>EQvDCi4-yJ&q_ZOB?^Ly!aEOJF0~0B(D@5kKF=wuhD|l
zU+C#Q8yn1*!P{yZ%n10gV=xIA)5(n8w~GulmX=Xr!2;1WoHvT$;rkC#$)4|!vFT+X
zw1r+D=B(0|$vmpR_ND*2G9vq;Sw=C};CZDWLZX{Kby1`C&o%_Y=2S+BH<R2XLYGSE
z_=^|+ssvIdqk*Vve*U?%e-cz*yJPIOZgw1c$H2N3J9pcuYz={bj%-@o*jDZLI~QwC
zBAzvlhUpVeBJ9G0MuPyI?ZP@V^Kz#H7sx;!p1xk4&&-S`|3LqHbLRJvK%QV=Qkb{S
zvFK<?FJ<lVt5;392rQqy5=su0>1Ji8iLbg#!*Kvdbu_1a%bzZ^ESUW(q;MjVZ0*f;
z!p?7bS9101myQ#T`rwFer2+tPS}~s>qQQ4@qljMnW0?dl0Z$d)m31QSYb~j2ibvAI
z#x<niUH!DZdzYp5SBs_*horD3f0JK!rz1mnfewB4Gb?v?I8aoYjvl{!QI?LHC9;+~
zoO4O`Ss;?Z^Aldl-U@c;ZGbzXt2gmFZFi(IS+89!*Lk<F@bP}#ZyFV(^jK1WN90`~
zt=Tthn9XCnm;+Qjz8G0Z>*7Ne;3SWus}*&Z1bGmhWqu<dAv#}%Gd(=ndllitwoRIo
z=Y;_o^88@Qse8?9I6Pwi)bibdeYC$Yfnv85*zhy4D|RdlSSDYL0)*rCoha8?w<DT&
z1A2C)NtZt0&W>)}@#IQP`74&+aOIqCHv=OPy;3=g?ZXcQU_r3<9`^sxN#N(Sl-g5+
zhm%cbigdiDjZN%d1~Qg0ODQFKeyoJR`kd#k|IXA-l{D|)&PM<Niw)F*#IwjCReL1}
zYT#Hf>PQn`j?{K~c<F;FKn7kjrJhC8ZhL_tqV0FFyv`9W-^|cIJ!3w5CB}a^^%s2A
z!kZ-9CkOI{{doQP-#T1pWlE_zQTCS9g~NcX{+SG4b?_OC`xuy3j+4X;3IK1etFU|_
zwU)#L2MPfH`45d!QU<#gu>th)=$7BR;X|@wqUv5<r@H;<-%;IVy_9xGyk!FH?40`i
zh&xk1X(J4^;^f7B!0d@?mW)-O{}o#QN$MY^K{-1A05(IRQ<dU{w)@%ijcKn;(QdbU
zS>2MQW2XvUa)ii)_u`-rnYHKB?wPOvE8o>{*St4#(@XuD6t}M<Kjr}^p&@SEy-a^s
zicMrbiSmDPIzRp-EIO&}#9Z*`{%WHma(;pwaXmQj(l?ztiZhz)?1y!o?2<o%oYP^B
z6=Bgzb0(P_rjGYJ4x0Ex{g{L_v_D*Sp*kJ(f4vqey-LLa(*Ttf96(dRO5oVI*kvHS
z)Yrni5#mxk52&L1ST~|zsG-JRTfTf6_;CA&S~zwWFHy*R*chhF3+&o6)p-`N@F9?l
zoSB%vtJ+66h^$SOipE>?*&Oc(2)=DpfDruZ?u)O3FB}0j^bIi0vt%bB26EIJ=b_`j
zOA1%H`Vt?X?rOD8woVkiMxan81My~V*uaT99~-NAswewgQT>020z-bjLUXyB$=(5F
z9G~s*y0#nZkD*LO`6f}2YVh8EaNq&r+Ak|aWUE>bp(T*mSAqFG%OgQ+P>a@EZ6~h^
z>o?rr#(V;ElS_BiX$tkX(wtspX6DCW{Fg{%=6j`jPqn|>OY@nhl4Zk*<q)K-4WtE2
zT=Xw+#p}(Eb7^W#JwAPHY`QOYdJj{syXN6!1lUnlx{1dr@tqdHzW4wjf0jn517Qdg
z=!?1^@thZ2vBvf)uiJyS<5q(c)qhhM#%1R}>Uml%abT1~`U|=XaVE#c%g7PeQvGo0
z)-=6kkfAb6jV(L}fX;C^-057|Q>o|d{FD#WUvE&jnVy&A0k!CR%(MCH2e^R#JaS_N
znggoLNCg_gLo2g1SejKFNA;^kwYJ$B9v)jOBm2@Ez1!lKMa;9uhM-!^XPe#-5Da_T
z&m`v{tLN!0^5385dIR!-r_<Mz`Kp^bCju;?82igI@{uap{k-ljoM?_CwNSE!ijVV0
zz3^eE&kIb4-5Snu?08)+a0#C*>;Y-^pa!G+!0pq(t;w3<q&Bj#lW>Mmesfu0*hRc!
z9+d~O{*%Db-?Y1~yh*E<m_qESjvJr`{%e1IS9+h$W$(57K`FKllA9RK+##=SwgngC
zEztt?2qCD<Mnjl+!{OwtG*ENSHW;<0Z9W(_c)JW{tnWY8)wyTyk2BfSs+*cg7X2EQ
z=d9!RQ!e9TV3MPe5(d$xXXIzg%MsRqLI|_E<WHe8nFI9>+r4Eij&q(H_&ItXP8kY0
zhglRO!te5?*<|T1*jqku^}AWsT}=UkmNZ?_-w6}<0=A6MGLhUG6ZHH2?K<P9jEQ}8
zDC&66za{7|^II6Mi%l4p#)<PZPsg3p@u<d2FBtkyiJhz@lgzafkLJ#UM3Wkm?;XTj
zmT!p+XUhU%EmKC<2#;9CaPjnKoFz;fD2UKSp1>Vu{yM{NH3}k_`}6Qyb0P^HP783l
zsV!U7SEgrT(W^G`bA^LWUP5e($^kB;9^Du?Kna&UdyvR;Wg6I|skPDw-YJr4w#ER(
znm0{dZBp;MSFJjxQVq*<YpI?-@nYFvH8Z-zqQ4`s2@8mGHbH9ELNAkL)kb@!ZvOzL
zbaE~m)6AOw^82--vSzn$W^!$Licpx+R^5ALo10FnjySnP4)afW9Ejf9@0+$`DLTa1
zH`e)E4e_21xXsPBr%mY(v^fmgTT&&@_u<6rf17?ToV8FwGlSDB1?s>OEw3~CeL_5B
zXD#8)VY7!#1e-|^Uc=Q>)#3wU7FH2@(uewe0EqooU%ubdPph|-Zb{6lZ$-_+c=Fu$
z6$Wz0#!NBZb&#$q9@yuSlAOHCLTk97!#j{y#Pj;7dm7`?qvAWeS>%51QL`21@+god
zr-pamJ8Ep7Y5(wTr=3<(`G1$X<Jg6Mdr)LDm=W=YXEqk0J%gk~N1(175Fo?^mDiec
zM!sZqrY0JrI6@L<(R4uPysw!?#(=sMP};O-=5t2qzi1Q!!exVOpqdSv9IMm3MDf9y
zuD5@Fv(3pxMvK>!09T}tA7WAoy;OJj-=}>aRYYTSwU-{%bR=a0zf6?!E=9~z(>@6&
zK8C=U>}FrAk$gC=0rbZ&^TqERW<P)XLdr+5L|<EF)G7sjxbCO{#0Q@#47%PgmKZ=3
z{#$yRkDr5LjjD42;*<2=GR{(ean|!S`bG5m{QY(6P(ZIo*^`Hp{Z$X3(?(%!(@IPh
zS(4dY1Z)srZKK^fSh_<tsh*Vy<*<C9z#oj^8y;D|LHPEHTZ7SL?TeV#u?h0|JxM+F
z&v$a-Cm}G-mJe0>S58vXkI_a<dWy~KlD|J@-8tEqdfE`m;leCV4yK19wqxDYYM##e
zpA+IgG$D#dULW#0qAVl*HY$G&UF|!Im(?`qu;GVQWCXSgr075=y$@$)BfPSPQP`6|
zmkxU2MMw5&0g6{+u6G1DibM`6IA9M_94K+Hl3+~%XeZ%rpotU5r(X{R0jrbdfAqT9
zpnHFA+gctLP7gX6g{}tQ-p1!$-ZxzoaYY-zy(yIbD43H}_9niN20+k%wYhBhaUy~a
zogDY>^k#-ahKEh#r<;;>HWpeIB#Hg91m44lI^)OV>+6|oWTMHiMVf~DxxLd%!zj=S
zpkh(X{r2$k&`JB)d4s8sKY94&!OMIc0Wwk9$@`W)7}h?)^1ln8<=Nkg*hiLE-ASh1
z#JxLb_z8neh`oAU$Z!1k%|-|OK|tqCp=r0pn?ez<JMjL%A9Ps)>6LkUH%{PifZBg>
zCU|i5;<{*hMp}gt1~raLsrM+_UiP#R8v9x7jZ=rGn%OV;+t0WesG8>QBy9_Y_t~Zl
zI^~ok8{%#h2=bz$9n<5{@-Uw~H_y|6Le!wL4de)@>mmY=4n`dYz0}{_)>{s=YFwOM
zEl#?tC;@{231|1ZA8T%#4%kgr<NrIAVV3z{mQQR4f;-v`U0kxg^_FJ7eQ^{t-SV8y
zXm3N_iycbP9)oj|ix20U7BkXB+=cW-_0%QaOj|zyF*)}*)4tJ#-W!cbMM=t4zCCQp
zgJtNsV6X8Jtht!{zR@NyDX4FCb%BJYp|$utVtWfdqZeU2EbI1(r&uwtRXrqtYFOzB
zmXBU42b3lAKw;WVR{Az13Sfx8QU76x+49QGBSXAKJ;z{sLe}vrhwve$A28WZ?T0jZ
zl40o%_w8W`%%vOGY}C#$guce=W?{is)$3f64ji=tEC9M5H4%Kq&iJkIu;J2SIqq_p
zET#ZpEg<6wtS&1kL~O^oePwi--?(*H8i>@5m)l!C{Y+^qt%3m}fK@pR4zx&~-%Tn4
z3K&qP%<@>;9FXw4t=Pm@W5{x9VwAO)YuI~E2Q3T!s;_x!IYW#neb^pe)?OB5PfB<}
z=Cd?cb3=Ixf<5(D=sJI!eC=YhU4t3sT0_3%9obKJoO19m`$zf=$Z#dp7+{b|Z2r6y
z59t+iJEcZ|3Uo|#`Ca5`Dm8`zLebx_L_eU{g%($mR%g(AUWX>81#~y-+@2DhAxvM6
zi6W)IZMj0jhJJwfS%`LLop4suSKKl+eB<7+(BjN;fZ|r}%^t`{Md5%XK38MZ+U45B
zJ9wx4WlMja@ZtmP6zl7Lujh~V_yii*S}FWl?=Y<?z$7t%iq8ao)mYO-TKv7)>JHXS
zw*82~hH+$dcE1D^C^9^n@|R9syt#gfU6_0MUUmp_YUkc}l70g#@{)V3^x;iyMfwnm
zYA3a_=MD~U15x7*SMra9r4;bx$!CXWCye}&Pl~tvtI~34_NX0=!BN6ix`(Ce5`EOg
z)b=P~s3cD-LZ6J7-vx!vMCHfi3s-ysvEc5wa^C*XQbliv&wm#^vna_)&kp-03=U|&
z2G75Ybua2_)XsEv-FxuMG7JUa1bo}yJFg74OUMK?s9fz0IW^%ct)`lUc>!S=v%>u+
zoEABdTY;uWk*%G0?uFLRoSTD0pxP2mCI1M=hH42X)y&61S)UJ(Cw}21G$v}NTbx#9
zAWe(#@gYCzR|HL~0gxM}f@k1Sxoc7WnuEFXD+aoYlZQ~Ft472{lV*0DXxFhkiB784
zj&g*o=d0(m1tOX>K3a&hh`bvh_95};V}22-4}OCI(8X7<5f}o0$w=mJ;a#j!Hv$Py
z_7_0g_Fu-G3Az{&ygJ&?9K>{cau|F1&QBwt{)yx}1EAXeTLI{RkxWjvn?h|I+p#2&
z<Xj1j$#p-G`2F7-7Oy@#HQ>u>pqhEp8MN@z@L(k~ZHWAb?ZPr(Py<vG%Rj#bSgrG3
z(usssK#OWmk;PvA?+tE$JN8_JmBy4s^Uiz%na6R@KrwAWV?aUsk75bJS4Gg~egJ*#
zSDa_`l~#l`k$j0c`QPsY@KkZ7vT<6i-TTsE48EmAeu}duN`zwnBLgy|<gd-bfC5FE
z{3M3~nsM_cog)H<z*Sx#&h+0(z$loBLw_SttNw(-vrb`+(~6Y+5`3PnF8*UFBq4_q
z<$DKVi-OB)H;g%mf&Rb7VFr(8X7%CSX2-g6$-JU;*(}_lTe2NK`NW8{pdurrEc>yl
zpAn6^FL**iwePkySaJh__2YenuX%leBxJK4tI^K2n?C0{PWFefIDAg{YtaHx8Cbgk
zdsu>nL=N`q@rz#zQ44E!C)~(q=KmZGlNtq1e?fp_VZgyJumhMN{?$z5>6l%n3#s1?
zcONTLrjo|MCINwc=QwdFg5)TOpogInYC>Nj1AH)$2X+Dz;1hO^?L^wXs&oCS_v$gt
z0G|`eTFC1&5aNX1G9muYlVY?Wo0|w)c7;t1HL+EJf!^Odlu=z|U;lZ)B1Q^<h3o~`
zToQEu8aX%m_jYv%btD5X+cG`h!a`eYYggyE2sbAeeSH5r9ISQ&{5QY@@|HBO7CDXV
zU4XN1TjLC6`7-@|FubU*x=o+JI2r<Vfs&ns_yqAMH?O%)%+Mvnvvy#W!G?&M0bl<8
zK|y}2sJZ5>wU-aRY~+ml^&eVVBHKFg1-zihJBIpv_JqBxY#0+G3kGV$kFDS}dQ@vK
zF-`tD4=;WvHDcFrSO}_D%dtATihK+NQo&u{)PMz&O*cNVRWvYg`Rpa;<06eN`3T#A
z$qc=d-i}#wgd`NgLSaVss6q(h%;bv<T7~#hipwiK2?OLNB<XB=lzx#_5GE#to4{h_
zPJ=w#$|Pq61S!!1nompJ2c=`dGOzI^|Ip3UgNHM(Q}VC>kgMr|IpWoowe<AD>D3H2
zcw(W@!Ka^kFyj0jT0Q%I=pd2O-pi=F6Lhj1JXJTR@DN2&`MHAHKQU39LKv^I;DF6$
znN+CVjZD>G$ezHFAOWeWbzNZvm)l*!VQDtFXz;5$bArquU!@tBiV*&}zkrn1^2H{H
z3$+Y|SPU3KGCPnFv;!=`e`8`IR4#pm9h&mi_VSW#ZS4ab2P%Ldu~y2q1_~rOL9f4a
zLZP}!Jx1I1s3F&zo8p=5Om9XD_qZllQWrFD$lp*VwfA`9>q1^C6UexQ%5e6Y?<%Bu
z;fqL%n`XzKoETaYyGDqIFg1%353Q}kGi56)L8$D`W`xBi4<69M!aMFB4L3wkS(uJO
z$LJsHtP6yIDZ=4TqWo0)1R)10M7dm}0FD|gWz2p`8bBE9>ai;MvEJ)#pt2M;9Oy!f
z0k@NmPFu~hus$|gFuJqrT|T=VWdPyi{?EqLT*h46fEWQ^IEF&#+q^G)E)Zw7oQ+>{
zUi_4~?LCh8S`aor&A<}ksb9}_fx}k?CAj|GQ&h$Z3GYUkG~N&brlmBYB9dz=K$#`Z
z1P@Y;q-JGfrc0xym-g^&pFz6z{~D?zl?j^U{<Iz(#g`)=x*;ySIsX+np-F)AGT>Ej
z0Y1O9(ZS!y&H;jmE`tqD2Bt*3uAUFhSLS#h*P96by>%_y<g==G#9Yj;(9=^z9(dwV
z9Z)7{GZjplq#dVIJTnc>%+G!Sl}XXznHOY;0Uq9Nw1#%VP5{Kl2yD(K@pgoso>~3U
zJ#h~aDkGDT{|Ek6fVz7^A1|K}TRIJ8SX+-iu-1^@>ysXKM*N#G^+rChC|m|6+rIvO
zJQd>SIUXHGN5=lot!Ka2C~rB}(eUpKAq-56J}_LVx41c`n=`d8%J&jRt2=QqQIfN$
z2JEMJz?MM0VAc{|sOlBzC;yCF_!QMYbajrEdo3MxoZxT7swhMX#0bN$e7QAL@dT*n
zcT`cZ^eg;HZ$_pqHShPaHsAO{N<u|QLd7MXZTa6X*W>APbHN*LgQMg1-PEdGxRGg0
z!8%}2MSq8;hg)R3$UG5<mR1S!1bU16g~Ptcm9r@)OB>;XDg!>0sMTPD5&DiPdIllP
zK&^aDWa-~~)8DeCukYcBo<zT0<#;8eSMWTlA~5RWS>NoJq_vp#oCyM;brmBD_Kc!t
z<>jugH&wZL5x}AO^%;!PV70LNos_y^p~=|HW9MwpKkJ}pTtACeZvK|Q^!|g&^IG#5
zO7p|#b9L?eQ3qK6cR<|8Pk#oCwIO%dof@PRRIs4P4&cV6AKNVvTN0a$msX75wSxEV
zn@rUy%@@2fJ=cQ?J>(M|FGl;Jnu@<!7ra^d(e`T6T7G~S99PzrNo&$unjbar=-5Oc
zfsy67s5NO#Kc-`K%H5P8r^E_knSOa}or+QZ-GAawZ`MYMLl-1$L6e{JWAjTzMjcL1
z{aW1cb_#gYo1;>YDDAGW2ww&;?j{a9pwpVx-5}e_%HXMtr8VDVV=lO!%uUi~gbrhc
zg(8`23e=y-$QS}exmcZq#$OM2XR5x+$?veJ?r*NNZz4GWcUq&sj|&MbU_T|oj~y^)
z0PiNvVFwaJp>>q-Z=$L)BKu6X>||wrg)!6af3HuIR$~oO(KtU%Wn^U1h^Q|e%|Y3F
z%Dh}Vvo0omUdgq4_H9vtPUC~h7}uXH4&u6C@uOQbQ^tEQk&H}<FZ+A3@x#4r3@_HA
zqG(OwS2<2-bH0=|`#4WHzTKabBs_NJDL#PB*(*aB%}Ep;(ccD60zE+?Ne2AN8FB#k
zDx5BZ6h7fT$0r>sQ{hjYN1Z`FcQ=ll(je}j{KJC%7gAlEXETwuzuro^QRcM{w8f^N
ziL%8e$ET;op7o8SFbhw45s^};4{W+#t2-Dl6y~N=vrd=J1v31Q=sLdHd^WuYAZTvJ
zbtRt)b<w>7(UvC9@SAQ|wj1MdV+^D;-pls)0WsZM*BUEDYAekM(n<5l`Jnm2y-nGm
ztbx?V02efQuA;0(icKgT8U1!iER)%Thd^=&AXIL6U3vww_VTLNMV>y$II*UpFZ0TK
zX8WH`5gk3b=G$Rwp(zU}TJ;-nt%JOPHcm{+W|nPmdW*-Rt$qYsgZ7IC!)NE$HsT7<
zeBg;xz$T`=nEC@HR{5v|u_N_z<!7k_0zj2KSEi!_LVN@Dh4yimuEcoO>&Lb0OCU|?
zLz0f<qLs(15}0Yo=o28*TnY;P3|prezp0V2g}fdx_xPDF1<?c(JTfiuV}JhnT+vGz
zf=>{VdbCDiV6iO@w@%eGM&C7ojUSr=?8sK-8KYg)pQ*`rOyP9Tz$FDJ$C0j^3cWfY
z@6Xg?51`Gz6mTWVE^_yH2hayc=ZW89TpnJoGH!Gn-`~C#DlAVxL-4yG*5YSBo8jOV
zsH0rCm_A3OqPRvWeS8BQ5mTvtNgVn$D+6XBGob?EO>csR$}+PNeUDps@D&vYye86?
z+ehi;rQ5@jxScJRJCRwY5nlAgpp~wt-zggEqK98k9RpL&%a+CkL57L;$U+9(vQ{Mp
zvB+l}K>;*!*PQr~5dRpMud*Vzsj7U~q_Wb^?qk+EVms`mFOBg|sKX33=+CIA$i|vQ
zQa#5jY1f;Spc#J`Hw!zhAGtn^{6wG}e4ph@7dL2*O6--g-Q{U_GLRyfpbUi@Mx$OK
z69&2MA}f9wX)si15G&J`fZMgMcOeL~;a*91cp^7Do13j=<??Jgs?_cT^SdNeMI$D;
z`x4_7{XQT#Ga{r@Uf0yv6`syQToO=eY`_xWfQ`6|CX%h(>nnNs{D}O+7LVkp$i(RW
zsrSD#Lm}fH$pfF*h+fJaq_7n<z23KxD~=lH3=lU|uucoeBGW`sg7o3;{BD!P7l~{Q
zY-M|^6XcxN*&~my05NaGB77BqmnPwE8lz;w$nYC?a#D&-{y=xEs?yW1qfBIqyTU#^
zFVF2Z(8Eh@>7tvBg<NNxv&s`t;Ew3s4^CvI6h$OWc)o@NolQuKLXHNk08jI4&tlB0
zo(4i#2jy|h_6n<XOJR@FORTdI_Zgw5z<l5wJJ9gGz54E0QY)hHOAA^i5Il%>8Xnw9
zH&&u6#Nb8kQ=J+aN&D<y0?-*Sf1X`B%0|})Zvfc?&<Oz2T_LNcT%6V9m9noU_~hA6
zY~Vk}5P9@}@`7qSr|@!HN8lPgE}9hYQLEC467X0ta4&Gq2uK^MM3J91R;j*fUetM)
z3j5IJP<i_khZbiAb>~lb*qH^$ucqO@0G{$D54LvVa(8(^_QCB}YAQv1e}IgPOkag@
z_YdzR>1aCLdIv^kW;q6YTr@9AdU|@|I+v5R)>cv5-6v@onYk*{;nj{nLyWaC(_op5
z*?KkC5L6Vb<<PIWE@m`zyu4SPAw>OgELGLjgz@Y$RGql0q(>h=Q?_)+u8imie~DZ3
z1t@?WQ{md$8gL@Z^X{O=&CTt2D+8B^sOEET)KIFRU+Z6yuCcMxllAWM3gq;*&*tWg
z70k3he*6HA=~#~E4t9l+1*=8|L!n6=#$CQhXpN1He{zPSB2z!k-d%O|d>ziD;o2T*
zag!=pvnceJI2n<YGcnn!v)^{RIxNY`qG2_R5r)z-ndOm3*EN*}_-E#C0PVO~KnwAH
zxR^fM9uDR>M(Z`I$jI55saT&VP)Qu<6C@Ld3?}m)Ej2YHCMJGb>&r_Dg7&Z1x!lP@
zWmJa0^+Ht_-R@mCCirFPosV9Vkmrhn4TU3&f}whECtjH<x)S0gPD?1<0Ifx3aUT%q
zEgPH7Shlpt{V`%sZgFu`2k3R7T8Y-h{&bo5)8k6j-@jk&L=5F6ko=kuKC;l3=0@QY
z$O*XJp4@nx>~x_*QO{iQWg)$t)cyXjam-!$_2%6d7eujje)F=PCoFeBFrj&)3cB11
zyOnqY4o#FUy3*J2#Cnb7F!*)h!Ec#ZUC3zfWZ!v%0s-EX&6~nlwinHR9~oNMH=zSz
zcV2GxWFMQEA&$dUr$vC>ZpQ1!vFUql4J015JUt#RHV9EhB&ZhT13YegeB97Vy;R2s
z#H!!y@i3)pg=nmzO+PXguwS#qqBWMM-@;=xA5JRhb(8da-uXy&xThXi;cS&@(cWHA
zt0JKqT1RDN<x9=DxVYP!n<kGNT)oZn-SOika&mIfoIftq$jHdNHcQ!~X2Qb4$4u-4
zy}i5kV6nWH%R|2{r_V1gioD4Za!CUOB?&qRU9wM<>!|yfJ98dkP=!wj>TN{4Y0Z@m
zrx{$FjehN|8a4W6+nYqkGBV2v8oFK?3g}+9UD#bj_O2^$6Z|!i0ze9Sk3&n3<@g`o
zf8Yo>8EVITD-~Ks#!$$l@(xVF_YLS5&sAG#o0;td;n3Wk?;`4Be;C<@`KB>bQ?CK8
z0=WKsgY!}5$H0}RGa}}hUsl~J@)X}DXZxp32NI`?oR6lg=FrLbEmp?}$f^sGr6h)v
zMOuLNPk4AZE$y1S>I-D#cJ23ETr|JoACrYW1sW4$V(_sE?lT-69c#P>+Wb(Q0Vi}l
z+hWnK)@kwjz|Wtk@uJdVvhea?zC#3S2ldCna?A46R2((a3=b_WZJS%Y^ARRze{b)b
zxK0LDz-i>8e&9~h{w8uWsjdh<AK9s;r#cFg2m(?V86}?g-HoMDxnH0St6)ocegeL#
zmI;rC?c-9_zT&S?tCU37zxlO)=7bp|5uRPeVtP4-QAMlz>+b3*c=rb1KVKBFTwzDo
zU?Y-7%HShb;k6C~9T`y|jq-z=s9<bzGMW`?NT)AduKv@b;1$)_d!8qc>GYiF<Yb$S
zuKxHE?HVA;f4<m3jkB1qKG^7uPEAeCL`aS}K0XG3VK{|9nlECstIK3(Br}AFt-8AU
zN4g;XM{)6R3PB#1V}ox5b!2!F7wZusLy275K)4$4iC09yeBb*a7>a4!q<wJb<UAhj
zkRc72zu5jcV8w}b{n`v0tSRVjv6uWqqO)scXM-zW6jJAUgH!R!^@f~+M%Jv5vU$u;
zS_$lAO$ma$^;sP)ZuGFzH(CS`H!&#)RSxlpI4MCPRlujHkU!BEn?QqJP>>8bf;raN
z-j1N>OI>(+YNtWwVsyArcdYW7FtD}Uu;a&IV#{%2+|V|x^4quHwkO%6xPnf9YmGXg
zfBrzD;Z~pCsh578EF>sC0QOtSC*Cg&E9UHcSLG*mM7JNjZi5rjIQ9}XE<XN?GECXd
zt}a{xaY%Orm7=^nEfcmfq)e|#Mpm}L^Nz<u1sL}`N6Z1fq$^AV1~rOjbN?99%ci&v
zHf3bRpawRb&&)51gyg7%qE{e@DX_EJ@xtiy7Zi}h^0dqP#sQV0qVy*j!_S{pX_+jE
z6nAPE6m^P$&H<B3Y`)ji!xiw_UV}IKCy=hL=k8d}#~<jn_Vz{Ra;PsEnXXA8FCOG;
ztQR2=Nb3t692{ULY@P#O<mvuwYcM%D{xu>Shxy()%CNo{D^qLVtluCb@p}tMy09k#
zB4S%mZf>qho&v=OH*N=rp!1<WGMYe%Y&@&jI3gxuXZzFL#Xhh3*js)5<p>)Ua`wRj
zmAsYtiK!_%CjWfy_wNn4Bp|Q-^B=F*v98S*8eQ?4#YQMychIs@z^}m^E(F7}_cwB)
zMsJ)T>2R<<HijNHhR~p#58h{wYY*EJsra#BMe33~<2_v*tc<ho*eAGNXOn{4rV293
zUxH{$fI~({&3Bm97l@Bk-#QYxOvS~;3ui@Tq0k<{&E|fYBc>+i<>h7Px6+o61`;7D
z2I;Rn?atR6LEi(XT=L<7EOCvvbAN;Onja4j@A?g;pwn02ElB6Xh1qH=vPOx|pWlW_
zBIfxf23^4k-`yRxsH&>2-cd72XfRI!HqVsu?+O9VD^^xjMGMvZ#8wR<WLe3BU$O$i
z=I-jKx5JotGZY~h^*);@QDjZ*?cr?}9IVp$3aB|g4fkP&gwtTLMZ)UuMN{LV;`Ehb
z;u5p0KEIkzw~*NUdw>OV`(nnb`R_JI>w3Q*002();%M1>rrfZcX~+E&m?LpuzQ*Qb
z^68DNXvYihI&7_59S9*+y8Pza8t~SpI(A+}1Xi&<fQAN2-zGD}{D7BI;XxW)1GH+y
z<m3wgLWYs?cK!K-O+Z~&S9f)^i~<(3k%GJgM@o$b{cLzN7#z{O+?%vKUTIrhT^-?s
zN*Tnkp}sA2314x2ZF!UIzp<8U-$yl~M6aUbevX?G6C1rxM(%Y1?&Nm|+$<N>M3dir
z!KF|&HXkToogRWsggy{#342>g;Qy24KB+=}qF_-v?2dhzt^D$EdqD$2TCR(WQwzkT
z0DzR<RhK<m=k+mGSg{P`dsv96CJEJ4wb>{PNY5XOoY%WBSmV)BM}Rby8Si$nN6<)Z
zYlP(AW`Iu4_YJ@%YinzXfKSTGV?Kx|qAs_$w-PyS;N6{_V;3PZGP0fb^onWdMLCW3
zp{Ei)5=dUBGHk9CNy96d`NBTf1Rl0hy4{1k3@sTWf((D#AGV`lQRG-;RMKET2+?uN
z@Vd!F{$(_JLSR}tUw5<MrgoBYC*TJw<QyX;2<c4XHXplM_Fisu1?o<0AU1GlB<~-t
zmZMBrv0p`m(Wl7+X&dhPlh0~K6$$6(p1-qICB?-Cj4tyHP6vBSO?P{fMVWy#&R|~!
zC%_k*J@4`q)7!C98N?f#n!3X&(9-;-p!ptP2>UCM(*Ah1r{fN+5tvU=5C`#>5WwL+
zt$Ugo#UTLob1Nc?Zh8j0nsE2fV#5!ZFNQ2Xj?e4lUf<IFEiRVc6Gu3Z>X&R5ChQ*-
z6(`@}5dm&T$5`0dDCFpPvFvq+K}thJM0CCV1J=^=1SE!~<fK_|51!b}fP?DAJAwP#
z`GGn#y%sJyQ0&hE7gGbc)9*D+-cMenC2L^S<1itQ>*M{h765d)i1*h+g7C=51i$p*
zn-y|%bBoFzX9ZrOB?-BxAE!y@OiDnGcm31HQuyrw*A1KoGg|$VC2vFaZt({Du{xRR
zzdg+Y6@-njyW?hL*NyuyQ?pCH?c?|f`*oK}NfDKeh^_Pi(sk&It_~dyB|8raAUBYh
zhoB-X)9++|48)9rA$$<6`eJNcWxZH$3wRGXANDW!GIxN4f6u@!YlU_|OfFd?-s`bw
z*KG}_c|R=u-d}ms@e+*mLyC5EcSJ$V^lWeP-5RrmQV_sjXLg1~o;<D&DNdLqu06aU
zkck}G_^sh|5n#>0hJ}-GZLPHVO-)T5Fa7bepkty2u-EhU9G=F)+`O$NrxprMV)&U)
zt#SXZRMR#k*Z%CS?`?<-rWi2k>n|apq37r4pE(o%uu(>IhlGb?zI^%Jikl<%htK=#
z>uc->4BFn_-sa|J1GL^$n+!tMcrq#~tg=lYOb-t!NlBB^()2Vm@bPiv`FeZi#AK(B
z57)P+4~1(V%3*iN<U$^(s38DWiZVqJIkxq6w6_~rjg$t<K4E!Sl8}%9qbw~g9gDLg
zFX4XD61p?aIPoQrY5w?9gB{>9?CeUFuD*|Ke0;=9T#fTxEf23Hx4`&dj#}!#jWrxB
zSkl;FG|<yg7FPh#2EzM*I>(Ds-eIb!LIHhFPZ;G#Fd)y7@<OQ#uMvFIURwHVz0FyA
zAqtKh;m<iEWSe)sci{7rCd}eIJ8T-!s*=7)yDgI7i3i%(_^K>XDRGB?FM4T27~baO
z#-5zKunqkRV}POdZ|XY^>O8RqE88Gs<bN`AT_9x93Z)e3WfKm_|DIF&!9&1gZG*OG
zI*HGfj%aF)|1(3O*4y~hBRqUbKvQXp<j|qia}0@;#|J^cuA5qKTm@EdV%+fHKp^0l
z6mH)6P*uD!y}V|%NmFE&b_p{Qa54}_Pulzwgc(q&Ydsn<bpYwle1QR)9~BaiS$98k
zpX)@2^c_YWiBFKKa@*JAahu}WKUM601Q*AYh5kqlJYNEkuScSoQx_P4qEr*;!wAU*
zL`nlQkSE3`c*V#R(c?m)QUtu4&Z<$ZygXNd83%O1e#-(<LDoOo-dirt>*xA3guy>w
z847wcz^RTLp=?767~m!NVsHFtB9;s|&_vE-F<@-f#AgEkMFRskCcrwve)B@zIf3ad
zQ+3cgFJE@#@dWToFsQsQ9i>uX)>LUv9uau{o5Jb?FG>Lmhi-*bqB1SZ!>wo?rFbVm
zy`5z3)X_6@5PDwu9a05<y3Uz@8@B?!IE2!?F^k8XD7A{=A{QPC^<TKyCMU0Zx=u_3
zUdo<#?)e}T-d588a0-U}4iRMt&CAo^s|N#`^5);j)c$rqNJ3YHcu*f*g_)mNlmc-w
zuoelE@}owNdA<dRx4X8VRt$3e6l;u472T&aIh?qpDvOR>;B9juw?jKK3!X2Z<$=iw
z05YbN^%Bs7h>D79V|)5+?9;|A4onVtG{rT^c0?V#_sQyHm?_LCKF?Dv9`7v(DVh?;
zwu+GTaIEC6bKb&n20jQ@=-BL%z{D+3T7KP<*Kf=oh8KXX`n9<=tj*HR`Nq=2DLGI0
zv_C9%I{hsO((3L$ZL5-taqGOJlIpxsP6`~kaTFCdoTSV>MMXEPI6(<A{|jtVP@a2_
z6;%d9LMH)c9Tmr;G`B;3jHX@tHcW<*$@^Z$`M4YhFK?LL?i;<-)C+vo5=1ftElxi`
z^9TB4oRv=vj4a6Ge8674xuDM!Eq~M<8_s)}PeJ9|Phd=sSMX+KdQBbXx;8dx?R}Y2
zXAo&D%BT4Rcv}5i7OwfJ$>@JY3pa~;(brxX_`pz+-r(8s5Sv<4cka=UeA2MI$2xC)
ztcTm5if12%&wj0yn1N#8PvQLD{W`(4!-~j}C6Dwq(y*xhD4L+{@)e;d;h(6~1r-11
zoJL^C@d^Yu<ajMG?yi4@`rg>U8XBWx_SOh0()gsYHszJ2(+Js)i|}!hw(d}t3bPW#
z@zo~=q1^fo$dXTHo$K|ExLlUxBj{p4#JO3`L)3{eCwwAB%BwDw?S-dfe!Ft(bfBkN
zQ;us6?+G0NNHrc1SfvZh|LI|@M#m+x0ICkR%!Ur+1j$#Ma+b+Wx>x?tH8|`eY1vxW
zS#(;bhxbH$Iey#n^#8}yR|ZrSHSJ1=bfZY;p^@%7bf<JTlF}t04bmmuB_g17NH<6c
zNOwth-^KgA-@W%YoW1v|nOV=w^Nhry_ofOW4?FmR2pQ_ZVH4eEjT<Z|z;tvu;4sxq
zUcGf;f#~M_26(516DjzLIDo$)lnlF<JhUf#$NRljI`*oyJ2#VmDNBX|>^gREQSoRe
zkIbP)iB+WETB|K%iy^#OzK~`m=v~LxaH9-ibWzxP^kUxF?-EW(p(OI<xrdTR)ySyB
zLN)vFoEuH1$%;`?&1#O>@QY6<pYp}`6ztnpKGGC;S){=J%{*PUl~iW>I`sY=4;D}J
zje74kT~V?6b9$u2(bhZV8GwA~Z!GoYkUQAzDf9eTepz@T{Aufy(2nlBk@$AzC)Uc0
zYalt@waD>V`p3GG`;zwXKr&W~4qCX#^3T|jl5>q-DN344S$`%5gjg65Kz_pJ2Vn0c
zL_rC&-txSn=2~sPEkX)rj2vFy|5Obiv!yGOr8CQK<MNruoZgc@x1_aGBS*XY%QWwK
zl>G7kaynkT%q`s`br;J^OeOnoCL$>%5~T9fJ_|-c$eYhU8QijGaLT<Yd(8^_F6=9j
zyR2ee2q-o$_h|t3<v*5n#RQ^YCY24fNPOoz%ze<tlc*-W*V{}#`u0h`UBRM}5*?~}
zm6Uu^(f(y{ZeHZS85kD`zNYY#gjB@8>S)%MwFDcip1djSLfY?OwLgQ^ws55fI5QY1
zKZWdb#s>U<vWzfRxNy=-D6B~jyB>TQ94K{s1L^?nj|>+j@BoB+eed#&Fn%f1&`6_A
z4<9z=*52i2qA%wU)mFm+j`lyuyvbL=dlS=FR4$C=6D}qUOGQOuwGVy?7MvM+#rh(e
z|6RCZ4Ue?Jo_P@(W{KywTLNNHV03<}KVSAa><MS_Qu;aXgdm49IkkHPgFN`;Xcc;M
zO_0&+?Rg6kR{h_fI0mH={v-BhfUemnJb?QHw=z|4NH4ZBqayeTW@=xp+gbi~M0e*n
zGd*Qf3klfMg`|!6(@>vYoq3JfmdFzJ3we9)qc|@Y8SiROT^nGtd<F-v+bCAasU!1c
zT7tyqNEUVj<GnZ3H7sCP@xy-58S2n%dZv||B=j*~$upa-4&Z_6wUQS7xVTK+n9|JF
z2lZoHNc1^ee$$8Dgp7=gX2G6;o`d4g)9P?~ph7@>hL@KY><OYgm87B3Dk+V+Z-((F
zoX^v0yL#fOMMTpt@xI`ym|zWM^JQc?3%1|2imsOB!J6y|E(-6k2&XuxJRl`1lIOc8
z*48)Fk0g6Ud}&19v=%2droNOBo(~=!D7r$7r8bf^CG-&dGOFP@`!exKr<(`aBBW?N
zti2;)l;KSg0d~L2k~+QWf2KBCS+35-y#!3vzz}ARVu)%dd?A6uTLsRTVO(Xiy*DM-
z_cyF&R2<1O4`8*WsBtzow&pZiug0G96yMg^@S{InRz#1nvJ$Qol(Q)cABnq+&8^{U
zpc5;${|mMpkY!r92+J?``z7eExqLp>IX)u5mNd3ac;Xp7{I8!^S5AsxN3I1pVM*6H
z_JcfW%NiUq6`Nz;Ue~V$bN1P%hF>D`%e6s<f~i9IYHV<>t7ksv`fo>9{r9w}H~7T^
zebkwoP*!@T<Yv1#_DF!Cw&F<0*Em<Y9Fd7;c49m!D=4m~u2-=X@0KW#L!Pd!G0}?i
z{<GK!_^A(n0d~wv18dzV2qz9oF^;tfgFB`muzCF=KYw#lDh~)uDN82n+{2WSB9;iU
zWW9+`IwX8`u=U#dXjOg0qMgjS*7*XZV7rholD<MbuDGw6t?qS}{5CSM`VNJ7;VgWX
z*)@u>mMQ9%Twd=4tBc5U!Xb?-aB4!bH=R4@u(5GO@^rCdnHnf5qomn+&JkXlNf+cY
zlT@KJ`la|)q;Z5JIBJ#@Zd5GFyxSJ{ZZnz4#UD8;1`=tU;S7O;lr>M4E^@s$RkPP`
zt#B+c{ub<E{6PxF0v&mR+F4;k$rtC0HYqWUHGf!q??u)b1#jZ=gG;!Lf-0=u?m}$7
z?F*cD7J29PS1UyF+YsgA@9r0u5pwz4>qe>JYR8=-ju-Qi8J|p_O_XdVW+1Ue26}=-
z?ci?C$otdW(pS-G!{Se~8dmLW6=NSPb~jen$Vd}4C3WD%zCaws&!z>_{dX;r_|<qs
zm8zNYUS^eKzJWxRQ^*byN^zX}QsN#6!BJnq|Mj{?qBB=Pq>{l!e7r}J@7Y{sfiHv+
zRpk==A?WX9TC8VW7?XkdZt!o<<XcXbY#lk`P;z87+?`BJ)kfI$QgNDvU1^YkC8*Zv
z@gE=fEZp%8W)-`PX;2^Drqi=H#hYC&`8cgTDSQl9x0|5SH!-bWb>c0sZEBygS)=7x
zk(CD&J6-9qEY=znZ<*`={bmvd{w^l&>**Or`u;^i(j}f89)1uxI?DaDh`P{WyvH{j
z!$T_c9RxD{otE^y>amA+vLt)@G%(N*a+zSUZC~X64Uc9#dlG}BU!Wx8)eGIWex_pU
z*;bk5a}Gf%1RHBr=T8hI!a?Mb3I`5_ur3qG#sHh&kEJW-ApsL*hLftM0V$*}Z^5&Q
ze{z5q!%Lm`UXirC&zjJUt<yu;XCg42D_S8}Mvp&%r&+U()czhhh@QnFkC35#hAS|I
zRqbz<MmVS>pg>lm`H6#nuBO`P2@93L8NjnfsMcIAQP~?rj6@ZSgOnwgQl<s%f8!%0
zsaZP-#Y6w3=Q4nj5xvg6o77Yjm@kf1C<ih~?dLVbtfpk%+iN%e=snF&qn+9(juL$T
z?_XHTJ?r{;o19h-jz30o6tymtm_!WWwxRXwRiqN-pQkDnlw!e`C47m3Brtd$=G5!c
z#RETv##!tzKqI9ZlGKC)4rHcUyO3pvCGxMUdBvMmdVVPB{5So%#dA;3lGUdoTbA-!
z!Mryyc0>_+nuV+2o~RV~wE8oi3@dl`>BVMP!HzkfgCqB$@npe%OXyQ-`7}GE5GG;}
zAKv^pPq}C+85d7;6Wk1drcn?Y6gor*-9R?g@vJclHX+u&|IIkV5FuU0NJn2jD5e?@
z*dGYQk*JTCGias)bjd1vGK*mM7`lxbmPmWZXzU5lfq#J@L7~qAQsy)e{_S8Jh4F+L
zPC?OWm4=cedmcHMW9$L=|MxVV2uKQz24;W8r6A%hWRR)Qfk+aMq)u)I*;suKP;ARx
zR5YmRAy6@`{8M?x=Dmd0dC57)v&oO}mPvEt4gR}DNll;*u$%<kOd{n}<C04|T?S(c
zUM`0`*Go;P{C}V084U{fB-Q_v6jH^!`2Qa){t62egf14BVYZ|tH6;MJCKHrw3num}
z6_mubn~9diwtc<1>-IAH)>1sR320UM<hZI{#VKv*LmrB~Lw>J9@3I^9&=QyF9}KJ+
zz#B_Cpj`Wvj#2J!LUVj2%P^NbHp4y}JPA2vsa4_I1<Cg<bMJ6CFR9Hi4Su}^q6&EC
zlo0f%mdc|<gB|*EL~<78hMqY8C5pH7xMMl@AmHDmPbt+xI(-y!THhyFX00hux11&e
z-|_o8Ke8WN-(~o(xoB+fHVrp?n~g=^*Gac7URe^sl`9l5(ecTpe?BEK|4<_G)2TSR
zm1xyI2)!NVS*i$@3vZOqoa|j@C;EP|vCQa!zTj?||791a<cGx}Uu(*HI}z(CZKHri
zvnAa$u7jd8$P=Y$JosKpzT57#hzP;~IEkyFsy7;ruh8avQ#lfHpPDjWSM3@6uF-z`
zA?)P%sh0J<cJ_>oz&;+k@dJLV528cDK6OL+0q=G=e`EHmiR=6t#TC*YG6Ie{vSn0m
zR6z_*3L6NF4-!JcNKj2a4(#bK-%t6Df^NSqKFz32!_1GOE%`J)ZLP0lqj)S=9S<0-
zj*xnvV<=o21UMZpF+Tlk6uh4*Xp}NxY&k_d5L~tyj^0}Ko%HB3N#l|q3veUXKGg)}
zP?6m{kfJ!f(_Z17A5G{^dSgA^LY;2~WoIZn2X?iSIwmm$)$5NhKIa0e_cD)D@QJn?
ziHVL-UtEL_DXtmdiA8x9EybS;+<xokN4YFZk6ykA*Vjb1@l^#?o<>O98B*)$3|PC*
z+tLPzo}`GlXEk~5#5X5=AQic5&iRXzue+9naeLGF#Acl#Di2xhL?EaR7=HK1X6lEI
zxBuBlK|l^${hdC394=ZY%$WjJD!<jQnTGu;ttn{@;>Jt_IzybqH(Oa*UORk0BqMg2
zV%ctaQGVXnp47$LdCC#!&2t;C%8raijY?FHd{s}p6khV4H2$>bXG%l?`Tg!{Gc`ia
zFK&Rnyg`uS@X8J=S19Cjq?#lEXOgMVRs9|ouMUu6_8P4dsYR|{pDGS%eJU-sY5G9Q
zQ*oC$T|s@D=Ul)1^WSf;4FtDi^%)y^<3qIfW128IoC*NF7+1-se9^byv)#+U64(~W
zv^3#1YKl+P9FnM1o<vIyDHJ<aWSw^$b?S9hD+e$mAIIDU&t<X0KB?cWBR_=CI8d`Q
z*1#KM)*EQK-N#XRG>sX&weD+l=D8?tcQLZhsi956Gy{M~z6lfoZP}ZZr=6bfZPi)+
zOQ1V&-Zvo+Q;kO3Y(3J$v#cy6Ma&c-9yLhybBY0Q{4|1{<{0d{1e)FoPZgj(j}})o
zrY#@ANL}*hMAdh<t0libhe#-u$D=vLtGuEY=BW{9{K{!(V~0^^iUr1Kk>y&P4W)<<
z4fxq<D2mC0MLf$2f-T!^&a@RTAh=hNu~5yBwMdAtvgaW8dM@;f@~frTZ*A}vrzOHs
z)5!f%;``b#!0?z;mPcSJ%wv*_OIuXQHGCvtGZ;DuQwsP60I4~1=SC#gjXA3~h_Nt%
z9n|5r{dx2B<*z1I^7jf{7Wu-p`k5D9$F7@YrAdjrzQ1f-!X7Yl7tD<=6b%PFWvAY}
zZMZ+n)Ym>2KLJNEFZwG=3<HFa-!aPMl%|9U1T4%TVEGbge{FKhrgEeCyhYQ)ZNI6$
z8-u<Uh{>*-jnBi;S}mXMDEbZgxi|HGO`;aR4IIgr+DI;%ogonH{Kg{*pJAh`{og3~
zT|shEMick9cPVEh`6evYTcNN}sG5XoE8EooE2ONk^dsP~zk3VQ61m3=H`xE>cUr#p
zVUEWv#~U)>53WnjZ0uXy%jI5aYHTj)c^&G>3Dt%Bij={@l4+KvdO1)qijz$P84f_l
z$${%KX8DSac-&f-5G4c43R4>nL;0?~-ZnzuYx`+O4p_**l-^}->%{_PvgC)CUvac<
z2o=4}Pt@z%Z%hB_{Y&GYC$~O+v;#KW&^)JRnAvKh(W)JdNZ4uH>zxYCR}$)({E@Mz
ze{Btg;x3zZRFWI=Q@D8evRwHtC`$0E2W@&3<!8F?jW<2QTm;FKhquiKqb>U#X#MmB
zQ}McDRXe4>m5qu|BsDjhcDD}Wfkci#%TrZEZw0j#Ox=}L?9S7EW_18o;^tEM(yNyN
zca0W91k1joK`4uhIb-bs#ve+I{CR6_kQ;GiXEFAb-{05oS5=PMlXe19|LsDdd!^pj
z%ttz+&m1*sz*!g<8Fgr&_Z_;<_&%LgE&s5eHu+tOZI#1%Wgh6MbxRN}c-1nwvn%pp
z=X_E5z5k(zegEn_qw$|U1kFwawwp<^0KYSd3KKdqO(x0{$OQ~owdB&rMPE9;#2y9~
z9p}dj9$ZXT!4nQXMpg))&uJT7Qam1>wnoGa<%_yn5V^XFs&=H`rv}=$oZJ)IXs8-2
zvfs};eu10ui0QA^q2rs`JNz2p{lM|Q`>2~beDJI%h+D*;r{<|@rSO@Z1Z1b=h3Y*8
zbC#Q?Jr)HXy4@F!y`r{?+MIw!81Cc2#<{0q+oaUUByNXO6rEe@|F|%>5nX&0g|e!1
zR{gD%hvgV*N=Z9$Ic?9{)FyKlx6yZ79s2nFnm4ng9Zx1Fo7s)>25-S%m|U$o)gahU
z2RnM3Mraj5MUXx8dN?dF<g(JX9G4GE&M}>93Gj%un|-eLxog<j-3)8*n~IlhZ=DA^
zo(C=SfHK4)Vwu=>?IzA7?D#0(FZ!&|w?E$ab11Pf!RMeHXZpJ1jTyRDTo-j<ojp*)
zIPkYM@pDZ1mC>Yp)RD(-!2d({?K*4mqIX~{q@PJQz^nY`)!_GXYbR9(7J5j69F)uH
zfmuDazXkR%hDXW#r2DPBlv10A`RUZdSpV8KeSWFwyUwflkR{KJc7N*4sDCyKCRbG1
zceUROZYepirkiwH0VB=@Q1wxcv=dzS$;*AEopXM8`5&ppK4Cb(@YU_RGat-UNjLXN
z_XZVFTTKHsH535EQkH*{XAvJW-y%+r*E2pOMo8S~S9&?tUW4+n$qxDbkzZoFmSKo#
ztVL?9AbWt#Zu8BpfN2_K<esASqtZHF0li>Mz9GEjsq(zV=%4T?<CJyMDOO52Rhz8r
z22ayiAQ{&tC`7SrL&aW?0f?>@p=~;U&#wLe*(oTG4l@HaV6Ya)>~L5OlP&GNG_IL_
zU%Ot=2zmQd(;y_j!I)tX5cx7P5?)eMt@kQQiTx}b-rm<turq-KcJ139@8jK^pUquW
zvzKmHF9W?WIqft%8)@(SZBJ7OZ5vQ+X+EQ1m%@g+{mO*lrClp3yPy=5K8Da6(ZS~k
zXi}MzLlR$4Dmn}kO*B8WI4}Bg65xQDgOJZ#E?)Ndwcp$_c2zvHw+3zj>+JT*+Qaui
z1jv1{$&wV<Ua7zQ31{{RxVxZy%5)zEI9>V#z*Gi=bRmo7#B$C_i-8_lpPa9}C9SKG
zq|rtmIv(S)BPM!B4{VCQE4KHC(sr!vZ}PmK`gZC1i}n<#fO=PfeQ(t4{2?DUpQnv&
ztIRNAh()UO_S$&aD~i{u7IR+(ey%P9?F@q@VWuuy=HW^?jrOO%w(BiRVSmsl$N5Ae
zY?`x)5M6>0GI$*8KDc0}v^Jz!=G8d3SD64~)XJ6)3O%1|!Bwr{RYNRObE=2FN3trd
z-D^wQLYd@=5)70WD5t4jtKv7m#|UV)2Xa~Kh;E#1>{&-Ye?nB<9uE56$lqoyowE)!
z52?EzbSuH`voZZS-(R{+<z5C1VpMrCY%MPDUsZNi7m>dWq|hW(pEU^bfAa#Vwz5bQ
z69G*R66=Kw2R@7lec*`@(<4wJ%CRx8XV10|qV9T$zkQwmCh|hd2XG0O{U+~wB2F@6
zrCV6afi7U>!N?+2gf@(d1~67d^qikd<8HAA@JhYdwObW`gfoJBe7RUkzFfV2^Ly3h
zJU@OxevZq<<T5v0{AC}zUlmXw62%jsUyoPI#dLQ8sM<tV%5-dGd{VUeYAB3GdtZd~
zM2siQqMYl`=@$G%3?{wuG+`}f<=TV1id7SHbi2|{JkVC6_>LsBJZl-ckS%v_M?DX!
zhl3e(^l|n1``FCy4?HUFqf&ipXhq82;fE0a%}$7#LVdm~yjM-{#q%mm>R!+C-;=|U
zHpek&=;OSIbmg`nP!p=b!#DPJ?VS~S)4^WC{iw&I3bRtK`VjE`&g^CE_Dcmv&CFtF
z>x*A_lhYd?dkK@}(u~`5&ci2_eVCV*-2Sd9nOWWo`zytIK_0kC(~I`CPpofZXO6#}
zvVO_6oUmF9T&5d6;5BvGcCKw6G2{}x;Or0e^hXL2A^iH}x8@t^meOeP#OG)*WF)}q
zESFQXz1-w;^he11iQAN;g?=D+k&uS44FCBCR-T`~8MW0C?X!{f`g4iuR5E<sP9xbq
zh;yvuP@advL4ZH++;zh&ap&g9(=fx@+EiDT0Yrn!F)?Cn?$o0g1wyMb7GM4;aCQmW
zNY;#DE$_J;c;F`4<F(yuNBV|qFeUdSZe7VAn>F|CN@S&+i^4)BGw)#B|2R#gm?izT
z)1k-_29I!VJg@4N`~9O_ySU4H>YOM!8hzenrma4ffJ@y*S50!q%Vtrm>EC8osXiMC
zW_$m+7G{?MR6<Z)K6Ejb^D|4d)p~@jTOUvBFbWyJrjI!j<k!(=eo>;2@8Uh6bAOpu
z)Sz5`?;Oh5<aX-}1X}`Z&K`iLzUL&A$FMe66%*jvI6la^!ys;Xy4@2c)iJ`=5+0+@
zpFUb_9?@-_;==t>F6P7Xc&wKizik+2R=(}3T8T#g#p&pwUZgJjy5CpPnlG}P_E|-Q
zqP*L8oGbNTvZHmIjVIw>e(`M$Q3gsb$;*6LP9?>ii9F3LDD=Ch+7-|sv3Z=<{+(Z>
zS1vPZYp%-dCU?YPVX|FaTWr^;W?6|!Puj3<e|Ds+F!MnNSD(XpzAWVpKc>jVC(qJ8
z!W(@J`NbOJkmaWZlSXaQG)$-_?jA$ABi)-Dn_iT)I)8X0WnQ%8KW~+A5|7<7bm-dC
zJ<~{D`10IX12wR4y(g|5r=9J&5?SM8O!e3fIqJR{nB53SHuK-}&yQ2F_&wbZ{%$Y|
zWC@z2@xfL_GuVw-_+8?gK!DOaa_=C70m%DJH$Gsx+q%-%zS7Z9&z}21##ysAM}kq7
zq+=(3GilW5H(4`%oQMe=df%e|%;k}o&{p}dADJAn4nt0qZMMw@>qD5^E3TL{N!963
z-jTLjdE1A15!+k`{c>c<`*r@+ZK?y^g?iSe;;&q%+__v{e1EKQpsf(?W5H@WtUFJ*
z_Bb@N-fgVE&^M;pSZSg-NMxW)+q_nYY%;;t`?2BH$#<s!>=H!`-LnQ4&K0wL7`la_
z9s`d6;ccnQBKJ0*J^hC{NrIu)`kFV45Uv&-qx5YpES%#T-13!8IidOu0SEe<^OTR%
zDrBmP4UYG&%M{a5wjcM(EF9+S#0Ps9{~YnUKxQ{%%c4Txw2R1#e4i*#hsRT`#8^}G
zw*FJ&*QZK26@f}o?%4LU3hs&iGh0?cOLmvQFCj)oiYrR?Fm5jUZQpL6GD=#kW%se=
z#_64~8KuDkt#!G8tN@I^Wk9P*kp0Be9N0Nuq;KuApga<ko-8GNbLiec{j)>+o=d?!
zZ=Da%t?)#$=-rgO##J;RYd`8KeJstbf#iBdxb{Z#7<MTOA!3U8Ot}rN5mznHq*CUE
zqY{w(IMc5fT_kwd{an!oSC^GtN!IY2Hr_bh2ZOyH=#9}&NouS{5Q{lSGu-9iB){=k
zDly-rv8a(z4gArT^#Q)^(Q<9|(sitb?dl{(PRD<Xi@nW3MpnbwKF@&&$)}9}>=&cQ
zy1?!Jif@a&ch)nM<tNE?6QOw*NL5Za{hP`|+jH|l8<WA{a6s2Qk@qBUfNV0=jfBsl
zO^1^&6$CS0wDF6URsRGX2%Iq}J~ltvic^K}b@*JZT>r3h%=G@&0DF)GguC8}H)+E^
zxT<2?H(L-%4bA$ErG1@(u?dGPKiv<BUMgJh`l@(mRexr!3ui{{nWvqG0}u?jdcUk(
z3m_9I*-wJjF5RkFR$^?R^SNvTL?(Dx3(1+mHL!?<7+?_hU5*C5OP=!S2r`j!G)W>i
z5f=L*cNh!)DI_3?g%U$vE?;%P#q?nRV7f)3DRTXLxRX(?t8<1XO)I%HC37MO!~O|K
z<*OkTQj~ufdpjSk#{HVsj7P=FRHqZ%*y{>`57gwlz@FaAO$Zbits~=gxh|8g2nI}g
z;2rWbJ6YjEJxI9T#~UQbHDI+A;b<vgyhv%>MC0?l12|OQvNxR$T^!ytNBzG5%q8o^
zha;OWqZ@$frf5x~syO#_-fEZDXjLd-yufU*?Uw3RfjvFw`Zo_{_mmbAcy#F$_I8~V
z+8#y!tNHgI7y#Y2#7hT*fo0x?qo$c*O}UO2ixe}5^jb%#ztYe`IJIm$dGwk727HRL
z%3GZ`8qR*xrDuCxpDVys#dEuBLvFc~Vay&_&Xf7?5k?CaJTvrP`vj9eQ<caqIimcJ
znF3Mj9zT3L+Z9Dl(yYe;`nK^asD?p`zF>g^?wGNzx3uE?(}&)U?dkBnZdXR2jsAKK
z3Gycfkx(;z>@#_oydK&Pz<Ra3E1i`Y<}4D{lDfML%O9Viy6k*iTs3ZkNsC{9)LXUw
z?bP4we0*PmWG^&lb<ifVUG>^e#}2q1=a)qw0x(f%SL{P9%?@IJ;30eR=@~T~s*i0g
zd-*<n0~G}n49M*tAH930<pSVETEHav)uVm+je-;C*&J+V+3?)qeUOirP1n`(@77n-
z2<BOub02cL#J;GjRnnv>&#}x*bJbWJ75URG<U5hw*ZpcwDmbcq!NaF%%wqrAbg+tB
zP3Sxb)7h&NZYx^T(dKgmIb`oeA&@zG9tG?#&6i)-#Gt`f1_#ZnDo>SkQkh<IyrJ;+
z@G;|QcKpHa1|QjTE3iv)hJE)X&h>KM9;PgnMtzC0-_31Wr6c>fe|PlBM=Aj>g_i^1
zjAplXSDG+A2^Y5?^KUcL!?O1ULf+j;8cjRH+oXF*@+0;t&s}{=10|Itm@1N*U`*#_
zJ$cewgg^f=wdCjS>jpsUyIN{N3`Af<Qjj1h1nlY$-|>Cp%Y{^c3|iYuy$1Q}O+Xb=
zv&goYGLqK}crEOe#}&g-(IKYlrv#L*tVsQx*DGs7L#jfHSbeMSzjE_kUB4abDf#ht
zgilfH<Kw;qsgG@$_vDMVM~hKnX1f5KcfL#PF~Gvee6E5+XM_~r35)Ej4}aWGTDIGW
zqTYx)^kzG1cuRFGMT!m&fEZ8^keKt%jfA~uCNe9j31kKsP|<;=PtF_nV#0bs@2nNS
z)Ki}rUu^FVB<sQ#{G?g7<j<(lQNFkRxa-Ll%9SF$n+;Gw!viIC#NUt(cS|cZCkHT_
z<ZfD}$|<s$Pfe3(*UIkEf%4e%%VM5m5o=M^8@(L<f|Gx(<C=m}ny%LN@dmISsm_ok
z-#RdMO$>13d5&>&OYKKkNCQp;`~1iv%R_(jk?ALTH7lrTx5!4D=a;@B`oF-vcvrT9
z7&w6xCUct}3$C<+pWPMKKzi)r8AYXs;D9I;<bfy%s-g2S>Pw%t)+e!xo`I<Uf!6cH
zB$M}i`}_73^4CpFaiE$KIJ5c4#V>LuSpknS8Q{G@3k?;W5@J@sNN6N{@>5t*A-^>4
zFHUNgkJ;f=!0#11%qg>4kvZ(W*BNa`*Edi`jU}CCxj*edDCK|O@tAMgbR<x9a~|DW
zOLFBWi2&6UqR_ebAsx<em~-};m5+!RfcRhj*_d~17jn6@{a_it!4&IsJ9wc7x(qy=
zW`&qP00Ghe%`NP)LhL5x!?NvmUK<huS#bz`xox;rltc5jy{~ites`~33CL)~ApOr&
zf?Wo~U1C4XLL)#%$w*&b9&YvWXVC2HrEyk1i&Y718ZlKaQVV|{`=I3^g8kg9Ug6t6
zVxdXiR9#o|(C3s(59$2j(!!$kLPIrSE?!Y!ldj)W5o)VDq2~e1Q-A=d`YI1fVVywk
z5B@wDGFmXK28KO|uq>~6cHXT~%LPE6B%Sd5xP!Xo=~MlNMQ1sKCHk{Dilk#)l*sZ|
z_iqPbIj*n&nLSI<iUYOT?<%{?1&fW$60Qd`>RXhkFv}ic)#KC2S8hjk70I4Qg~Q2=
zod>aeX$Hm(B!yF|8acSx>KKuowRGh`-B>co)1JLuXg$O6%hSY`50G4bnaQJuNPddB
zcoVl*FjE=Z$lP2(H>RlBc=Y*o71^QB<;2)6o&`Z<B<#7evhTaG`*$!vKtTr?CQ>e9
zh4;N+KF7}fmJyfTpMT5k^UWd|MthAR9EkB&@NM%y&mGZQjHmzQdW^kwx8=9pDEn|K
z)wCbzhWWv8XVIkbsAyPRRTegdBbGtGG}&p9i;gW(IXw!i+tQT_y%N+{0G+YZpBd88
zS;xIoVl|IY!0in0_jujqJF?OqVDqRMQc8ZOg%-}EPjTK6tceQOwfU?3&M|;@4I}^d
z@cZ^H62Fea9fR2hympi;-INjhUzK@=7f>2mQSZ4yo*!^qHcQcJgG`@3^{(kPAw5hr
zjcgepQBA(2w$ky2A2X<N`4d#61jC+i!3wBP(sB#+yux4P=_C6wncKaboX|+Ws^Ja#
z)r<K1RL$t<feN8G`IIBHD%H>d6#qXZZ-!aTwLy&|K1$YEhhVXFJ8>6RdYHDNrZTdS
z$J!4agQg27wCD({lnK84LSK^)S|=lYUkZAam)x$+hf6%ZUQ?>!;WPFq5_w{k!RcVk
zD?5@x+^3!<tEBbLe9o@@@92ZE`92-Gm=3zFJ7WH)c3N&T&>?-<e^7?6g!-Of^GcUb
zwo`B952AHB(?am$&dCf%atzShc|1u6?J_XAE_xNxYMgpr*;TO(0=*ijDur9`TMjSK
z(8g3u)_gj&660|kC~1(m400Dt-jGkI!NWO**&(evrmV&PBDi5Yo{IPj!<<y9d>!eg
z$N#u=vG3RxI5GqJ7bqFD9pI2C6YJmtPAC#{)xzxfU4~;j%?5QkAkk#Ed|;%Ek<fvU
zXS|;n3ax<p`*ep<mTUDrq}WW-2U{Swt^_N|JW?I$w^WOBfzZ?tX*<>%i@1X+EQiQ~
zMGZZ|UQ9vDnU_k0DfvxG0%5u#R;qeJ5^FA61qwZ+5$^8>934ye^tsKmVv$JGlh=AY
zq6N0(w|6C}Q<q6k+Bv$Y*L!hSALmVI$3v1%YJx^@zN;{<rjs!y=%l_6<G@O0{Egh7
zam}ea!pIX5+*f1{gisEh(8AL>?6Th}<`hGtjslMUz8@Q(6dp$#R0|h;k?{xsxBz_&
z2mkQDO4#=<w8ml1s1^+AqC<qcA%qIPIO}rTtF#S}m8!_OSI((fwUoa%IS`;UM!!!7
zFDmr~+#dlks7m1eZ?*E17-*dFcvjV3e<<`}^-q_y?rjsnC`-o)yUulkofo`h5#i{Z
zsf<y%qTc-%?kkJ?9BqjvBMee+R@2{E$89{Fw~J^QR~*8;@|Vqj0Wx?X#W15^lV_7V
zEG~0DIb;~NfwW4aUV{i;PW}BFZ0~J{(nbH<y`n$_YuWiJhu`g8ABIW9)lW@p8)a`i
z*9jVFd83ZL4A&}e{GDvzKIs{9v$nfP0BZuGyx-{SE$Ew~LkJ1EOw-q*(iX1_@ZwZM
z52m)Mze8X6#;37H_UO?J0FLAhtRVcN+B=9qAdT=4Jw?6!{RXi|j-5n2Nhm$4)xq0u
zyjEeWbEuJU^n+|QqEzIp<Z!PUya1l_f<(~Irt7lx(qA=+pYmyz8gG8Q!Nm=3@>(!Y
zS}sBxvZR-XgEK2`&3Tp1KFVi)yS{k8fryt3YrIP;`2I3T`>4`}=^huejP*JF`F>(q
z+TX?T^`~Eyb=Ik}5MvSp<ca)Ys5wHU98CBSgHD_KWQbnu{rX#mS}ZTbS7pZ}f%<PY
z!c9<qN}O8RbT_W8KSCtl#`#S_40xgwKk~>X(1GlD$4j9}MS;uq`YRx>;ngeB`cG4(
zjal)t()b6ZwF5ReKFchnGAekn^!pViVP{$dN)eDP51zGO0&QFzgvL8@C-IR9y$7V=
zZr{X`$Qgvb>&35}ly_0+R34{;w><WnJk|+61=5oy4ZcUahXqy&nZAj$o~u!&#w3~z
zy9TFbWCTSl62^KYnGq%A{^Hi|H^g_*L!&yvHlCNCGS!Moi9*P*8Y!7*JT7jT!;}n=
zl?IE4#WxaXI{tMLpml}A${Em7YYW<nNitAYGj-@RNAHfAElImr+f!JuP||HU^`r#(
zQbL^3sg%bjCICq$F+M(Ch8ic8&4r}C8@7OrfVo~$kh%F+A5GxEnIc2?Sd6*cPza_F
zN1sUKryMk2=5MAO0`w3sWfPNCz>LBq;f>4AW??be;1!RTdGRz5NWv@F0T(;`CRCO$
z#Mc?+Y!&W)AHD>?XEP}&rJw_!7W@{|{%m1kK~#g)umvlXE$vk|H_OvUof-!$+!h8F
z6X8(S>+8Nxo~Q*cKu99$DM<Rxc6&4{J3D&~N6JuQh}nc-pe=ZSVa>?hn50%=fQ1P+
zHA%qcrYKhMZO)x}R0%sOK3=jZ;DRhPx!HlfXR4~I;Fph}ZcW4@(H6IhKzWFzXp?ez
zLp-(+_8$rP@l2PFUN54Mt^&8RekZ*K*ap~nDnz8D{(Dnpes`A^=H}rQ9CFUG>B<_X
z7d)e&8=1}rHCK>(m53dVUg%7}!P!oPSeVv&&7<KeQk`{Aa-i2MywitZDbqXuQqz7j
z$-Pe(R5uh-`aT2=EWMhV+S=+WOn#HgW@H0()Un(ROUDm<E2E=UN0)<`#Ny;9Ke+m7
z!Vk1Ze)$fE86zw>vFd}o-J3^Tc>=wz-IFykHjQjK+N101H}-4h_$eLjQR=W^tfKM^
zpdVahQfH_5%a>8Wk6T+?|7y={_}jp+nJPVA%gN;1N|l&?13j)V8fJ7wmnHdrxOHt1
zt1p4uv!29p>3lPjIRQsh^W&$H?W%nHzqk(;M-D^ql=%TT$WU^}A4%d!&>~|Kj&y#0
zegHnV`QEs=yVu?8>;L>WT=9J2cSFWeX0%Y7Eix{%pUc89)PS@YkMTOB<75_3RWe%-
zye<31%`L**t^p<)gieu!*GO;K*}>}Yx$S9qdh+J5-nkAp6y0xH8P%t|o&J3PvEfrZ
z^xxWPeM<rHr!4}8lW_*6h+iCE{JJSO1Enn~<uA1j&GtZvggD^uGLe%T|M{weghBHC
z#}B|z#YQ%7G&zlxCH}FhmD)q7xIooSvnk)GlBBAlrB6E)Ea74Bmej{`BH3sEA68EL
zgWnp$-H%eHh>T-nJP=Mp(bE{f$7yQ1Z4M@bje2}}IXXNH9~fc9V>*$Z*?JWdpy%Xp
zfshpkzuD?1#b5HT$jJSxg7U{pWoFaw^2zrLb=oj|w-k-XV-(rF^!`8*mkcZm;9Pw(
zY}Hj%9GRZxqNSzf;^JatjOsD%Xt!~$#=|U+lFXm7zvNS#UnVr#X+JD(w4$z3@H+Zq
zMy6LiKN4j~pY5=4d@XMA2iJH1>zYg3i>g<4UqE0RKRY+~9&p3@<0xFWhWP*u1-xl$
zdKyqxD=RA#6B2&u9gi)1v@-nlW$=1amiX(1_Jx1CtM|XrFoGr35d9)yr^?+_zur*n
zbeiDLsJir4ufj&uC$v!8>@2aLs;vm9+BI?)7R}X1=<<UCOeOPVb7^$n=?eK>-k+z}
z>eav3tG1oqU{DX(pR(C3*Nn{(1QZp9b?y6cS)FnWC;1yg?15HIhkLe*>lWNEfzHu5
z{ykc1>3sI_rwcp+S?W?{@~;DrB@_r8Kj}8ODfp}E^n~h`t-OS$hgi^6s)5T6s?vH2
z@~f5gn0~l~0B2vy@*fG9pj40|*5@q3`zkvihDhKsCgi<fMH)wL9W{PR2K>0sx5mej
zZYm_h+1?s?2Dqfeur>uoT~+bGIJH#n2xw>H7w0=B;#DSL3#~*tP@=80IjRzyb<D}n
zS<jR#{pKS*i$X+jG^kl@-GG}o2Rc7HnG<RDqxe7jsCuI_TAixPYNcJoh%MfYILC}-
zd#-q8=#(h!H*^OVc}~F$qO`tC8pQ8lB`RXk#NiDZVo85mK=Cj;UYTZaO%5z=ap7{b
zR8eAp6A4p{(#K17!Rcc{R#0g6y;}WLHd!U&OOTu{O{{|3SsTf+(^Q}C=RFa)UHuhj
zP`@x<cGLfO3^E-%9M%@eQDKv)#*~kby7CfH8PACP<CFX-^vesNR6$EAq&?Jcx|Fcl
z$7+aDY^Ad0kA3wiGdcAlz`*J7*2i&>$zscZx{o1Uo-rwqIX6CN=KY3}Qt0of5X+Mg
zq)H-f{aX6kxMH9`G#4?(>}}>yQDZURwr;i1X-e-}(ML&bu~Ub{TugFM|9IIAH^#8z
z-J{yeh~hJ>mW{X}wUw;CHvv*+?O)aMD^y6=cwsB!T7KTjHjk)hsWl`uJ{P}-_~vdp
zD;l)8lkquU|AQW-Q4cQIB9F0n?_=L}a~G=lCg>!c48E8(J*~BF4ZoM#FU^f|QD9lJ
z%ijA^uK(IjUU!_YFM8i$ZnR8)Q;@bqFt;=gRV`{L=FOt!6{(>6s;wk{A_rj^1rVgD
zi+}<_3V{-**9V^dqV99h?O|rAfxJfF!LP+J0a|4SYR*ndi^>S%cnEm-l(d1GUvHM|
zAL_E|b8J;CDY&RBH-fn;cYa8C7w}PKQ~gFZt(+(#?!i;B<dWTiiU-qwW(-cHkCONp
z{uR_7K4GWFdni9lCul-$xj!fA$SA&0h1L%FamelT#QY$RSx5drHJh6G*5sv9`t*5t
zKV$oLR_!$XjR78An`k3OKLap98@Rsti5NMp4If6Da{uPQhxfzl=CLWDxZlzczN~7p
zWcGa%(8D#vQ;UZw#{iNypCZ*&Hgu=o5FVBSmdgd3XTGszx`&jkT!FoW4MHa}fk<`%
zUJMk7L5K&6LPbt2oDB7aFV*CbB2n|iht}J;t4X-M?a7&AI9`^87K*P5Id19nMf2_s
z!RcF*YF`VI&t!EJ440#Nn|R4c1)3=oLuDfM#4#+s`6gVRkdkeT;l7f#FH8aj6>2E{
zCkz?x|K{ATJzXfW)il*tSRXW-BDS;Mz82FECN63oQFG5IzP3zJv-DFE_t$aa@8e|y
z5Y-}<=1<VUf<>@Gf|x4oV`Aq!->6UG>qHn91o`%0ZRKQ`33!p9kJHUn5~4&vAt}li
z<Nko`%g~WC*Q`*b<lle`2?4hr8rKT!R$8lDpn7IdbqzMWPi5n{_f{D|-|n4nQEmt|
zU8?)vwr+ftxcx+@3IszKT(K?SpdlJc8U%O8=`Y`IGl!2c7Y>UH)0N3sztUzq_lK_(
z-LqP5EOzn8C|S5N`LmJjKdx|?P4}m(PDI~h{ec;GS-mO?K<3=C(25J3l8Qc}K1h$v
zmCObB>_HQ}m%gTnk2~X&e7qb*r6-4og$>lM&}<u;7^f#s&xoX0537<XMg-l~BvY8y
z?}Cjoak@-X&ar^)6H<Hq4b6eGndB(=GdR_Sa%-+pf65MLeeN|hXfwc4;oXQ#J@ry)
zIY6O|cr8)G9&#+9Rm3f^W3poM6l7wm6q>M#$<mVgIgfA;xN0C8mL<|x4LZNNhPi5W
z{XlhEFze>%w5g@xP}VYXlt^ILXBSre`oP6TQWfmu>3Fq%GC|)BiAo-JKF=-9<Zc)E
zzj^;~>vog8yYg{+vrCAP^2^5-g8@%~wi<zPdBQ!ts3mGma%=Ta_!AJxxPb-6$Ka=b
ztuj-VnzW4XTlxA7>gCg<KRvQGG-UT2@`TnsS+2v#{pB<3cAM&8bI_0*r)snWlc_x)
zftL;N(lHsFJyShVEybvRnOfQ+KXu^hjH8ychMy=@?f$I)i6Ci_uJU0jbhUS-2ybSe
zEiE$~5<?ehnc9wF(MO0p;fYUIKu7qpI$j*$CLz9ovm~QabFQ%l4X64u$JtGPdIT^r
zavF~%rA7E(j(sovs6{^&J8X24LOV^xD8`4+W0g-hePxag8s?bB|7JmH$*5bTrU~ll
zdrKP!-~{nf>hF<nU^_X{6&;0CMSLDc0~4+8(1@lyR&pt*qyQ1Ge)7oRqmZ<6ulagd
zVNfAup7bkaRyBx74Bj_-w!5LL>{31sLVV9f!8YRzY`<pN8cP|{12btTaWFZaiUxA>
zto`TEz=0hUTyVL3xJzMfe{On`vvt(=0NNJjGlg*S=8A@yAA0pgBmSE<XL%-8I@@<R
z&vT|~*c>oGzmNjUAZJ_si-y{J>psMk)h~tEbvvWkTRlZhDOUv9>ZHiI`NO*s#^B~Y
zCbukqoJy6r%ejwD6u7EI{{O&rNG`mYoWqk+&<6H7LtKah#O`{PNVA$h;8(~}GPgbX
z-#toDoGT0heM?u=t3XWkSFypUCa0{8<}AM>YxD6{EvLhJ2{pw7f3Z|w=ezd0)!Vnf
zS`a4`1LdFK#3{HxR=?ENeLRM<m@xu=T<G^yF`Aar4z#n+e;A=^eUVrn)X%ObbuQX`
zF(q33m0FdKKo`4o;km4CCublhdUh`SrB*k!?jBuHAUdlMK9)kN5GE3}Ea7^glc9FY
z7s%U~O8X6O5Z5jVG^j~I=p4|NR5N~bajSXRYD2Uo88xQv{siZQv##jPI>vkdj$HjL
zT^Q&6F<-o#I)*N&t=w#Tx!rat^ScI6Oz<ET#E~&m0Z(!4Jga|AC($QbHiLAmZkiWo
z*dA46F5i<L$ybEmH+}p;xG)f$A0yHEsq!wzHz=xZyMG=Uzfo9xoLI|JH6=AlQ~4d|
z7Yh|%#kT8}FrqE`M*p$s*FwNDGT%`ecKpn7*df*?{PTX@5PvL>$<V8}fy@8G>sMTz
zhy<R|xe{UAyr8nZ4pZ!mV0V!ys=<B}d8LnU4Z5BIvK^gBtSY8y&-bC~0`XU<c=)Fl
zzUw>dw2C(4Yj*}=u{V7sDq9c=S!gF>J7$Onzhj2;DMq5XHli{%&&i7Y@^HewgN_55
zp?Dw`a|kCnIEO=QkqA}M!{96)Ye<v50`vl!?k`#&=4jM;@+kro^XE20RY@3iO={Li
za$)u6$D@9B%>wbR-#VQFVXPZZdHUg?^Hc)6)L7?qoj|{cw$^KAGT)6^N_05`;fV0W
z!?Pj4Lp<P?uU7o(*=;1zu@ZrU3Loi0W3Lw9&t0|?Epe;%<}Y@}4;2P|*-&z7dI<5^
zzqt9pjw<}1EV!J*2kBt7BxTrO7Y31S#BRC*^Lm{<i`1{zu~#~!bYgrKz|t85>_2lo
z-O%MPg;}iEo_auv1P38&oOmJVRnQB_*m%fgk$Zk<+h5H>RTbbk-**hxJP?%mjHn4t
z;7Mrs>l8_j5st29^y2@e#pf9<h{2{og3^Er)d|t^{L@i-XN4`Pt?B*l7S3$?Pde_u
zBdeuiFS#=n7vPm00zLLqbzJ7D?!vjc_(Cc~rN`p{K~}C8bRG!QfKHRMdE#&Pzvq;h
zi0W64bCeHEQCLU){Ej_kpE}d6-${8ruQW~yz&!tXnOr`q#c5uaZQLQ<!Ftxrn}n#%
zvmFdiyd|RQRLsXGCUTD|uX6VuI4@^uGD;wI!~uSYUPxWu=ilhj;^MpS{PI}i8$D;L
zW7cdOkEG)}QB}Zru9JmUNz84Xgs8}_Yc9aN`6XX~rSrUA^bi|zbk#uPKRYx_m%}J#
z(<d7LHcYWO&1V*Cx!X(BEe9|kb&6tr(~mK<+lm%1hl_`ySm_`Q@!ww>xDV(*hFYd-
z^x2iDHBgRmE-~Dl&#{_~tU>tkZ(y3}KU$|{Bh{p|7iD=2A`f<0NmHD{f|=>p*C?Ug
z%gJ%L=G!hN&P+c&?B;C^eo`akW7kUMxOV;g*;hiRoE^wKYwb;B-)&3P?t9jdfzQ%^
zyKU2(;kS@OOZPhp>P!;oXeq0rMC|+{+K{?CDj&-W4$LDkk}^D@*ULtflt5lHEk|+N
zlejl4GdI8~yl#UM<JIb#v*z|D4B-Iu6*0q`f^s_co6V)u4<@W1R71xzTer%Bk3w^R
z&l+VYKoiB{0}5ki!2L-|QpT&0n-&o+8d>>yHefV@#E6z?u=pkYsjINKTy;NvlkwlE
z+4VXbaI#F}zr#4f#gd}~EU33XqSQqVHQMnyUf}Q>3+K4db~08PLhUDURbBg~i=N^Q
z%1@=deEM{i$S&BZik35EtumgoUslt#VD;Pwzp^I0qxRex0{r(~z$xXBr~Qo@qV*%c
z&wq^!!`BWR!V?mhB`Jb<SFjnRlq8dr)1J1HkaPrjCGW1^8N{h4`F#1VPft(Sxv;+e
zA)@h9>8wtVR&3A!bR#kp2j6}!2jP3MU~EMH`b-@$YCmMrvEyA?#&v%zG3~xj9pZM9
z(D{JQw?sDSw}-_(+)G!;s({$bAteljI$5WU7BIUP=?kMBb4Rudj-{0{MwBKrQug5;
zUel>wwYcs%`3G7$En(Y*x)TXR8jQwWkcu=H=7UC?GYd6LIH(e77PI0MkpR-`1#DD|
z`@7qn()yOp$T68r$KH|6ElC4eN&G}s&#j#1j*l1*DsL2U;|%tam)Pp7OPPO&2MV=b
zr@epGkFo@(kw1sO?VOvBg#f@>vke^$q^D_NA=w-(%|e0oUbVOR->CER2~ie_1?6>B
z(Z@J|h(<Uo)`HW}=@=wA8Iwpb-shELtQYFFTNSk28|!5bUI-_sQ{w>aR5$bP*NO8m
z_X#dU##Pw<{>2ok+Sz6rx%9vBi{3E+eXwo3LztndgWiGc-IgRNJp7g)Mp_JB%>}>y
zO~|vPWBR$~l|>GuJ{U^Fmx%zwMwOG#FQ~u02aCH{n?3j8vkE<AE2(YvKqk9iIy_Qa
zgJ3(u<bn57&<T3G>tv5yBK`%oCPaB7zIPJUWahpxJQtM*(Xq;FkY&<*JuQeAV<(SY
z5<gA-l0V6BfX}sxZ#_YtKqq#j5BDaH_CGTlQHq)-7~uN%&aMr<*aNm@sn3FDv+FB|
zVPmQZ$($0*r_yp%f!dwyNPNC`w4gxZ@pnA0)gh;B#7NbyBK-~&|6oeC`TW<v8cqgA
zND7q<Jr)k-4<#mk+pM~2darf_3DyKEJ~Xi}Kz5dxiY`B2>Re2Pg^2-TR!neBM5k|H
z?P9$Xz*9mlsi~!w9sdiXx<LJSDz@?T=?u;SngzxGGSt4GT)QjupvarS5d~XH`AG-S
zA9bX%ArJ04LW{oVjJ8w9R~vDRJkqs~v5)Cn9IrGLdGi!^HS!YT(mArt-`rC3=*;l)
zB)#nPU!Ttg$ipw>Vx43!*gppZj2!hN?-eyGUK1tD+IKkAPR$$6P21ms{!mP9_?0^;
z(w91YXm(#TvghJ^D`pF1T4Dg~0E|H5C2_?E+|*s_7UcfQ?XwVU$!by60U|Hq^n3_V
zL@v`j$~u!{4J0kPMMU5)DS>H}|AEU*M7=BR5?=C>8?Q`X703zj)J_E&+LIcs#0`f~
z$_!E09cS}Qa+jrr&g@YRg8oU*4nj;bUv4$BQ|F$X_5z0B`$g;~^3L=UHOfB(W(;1c
zY8j^?s_jr|cZsYU^ok;U)CgRJ)%PtZW6Q>(WLMO-w^3<!4%d?OASxJmg_q0;nxvv-
z_lI=M-)H;J@lmPfTfr)qB5f@>jkzdQS{n%?;o<aQx5TdTmN<3LCpBKc{6Vy<Q-n;Q
z4V(>?H*gSsu*MpJ!S*grrN--DF*SjYm-w%T0XaTt-2upA_tEFuM_0Q0wD<hi5vA?O
z<|8Vxtvi`IyKioQDqMl}DDuz#$NEl|W^=B87q!MC6jzk|ndl@p<fnd|*>z#$AQAl^
z-*MbnnyKzpaSmYuS;s5D`}Y^zwSqFjYA>EyD*0hoY>h;lYr4$Ob&KaoIV~k^vrE69
z&{u2x0o$={W(#{R+m>DgWITLE)Om<$dtAFq#0Aey#Gmkzy29GZ*{iUnuzFzs$Dw|t
zg!u3idY5U@fgZi^E9C{;XupGP>;vDcss8BpGfRvL`OL78+TnG~WzEngij}F32xm)!
zqnr0jpa5Q%^WAHOp!_YH9@g=^H`SIq(5=1)BaQDaPVW{s!pH|f8Sa04d|_#3LL=bD
z|K!_L)<w1U@3-n|rv9xg9(aw1LS&=NDr`HFR)No>{Pr+N9kDU=OV>)oq0d_&VrgmJ
zJvtTcw=#lerTGsMJ<RA~b=)!`ACzu^f&jbbVgN5@3mSAkd&_!l9l`nCW8>hFMjON=
z9ZhG8Koq7^>)M0OMkydN0m}f{N%^x!S10nC+=gMLLSiW~8+vDzd~1=Mu$Z7JzN&cj
z6@X&}M4ugLFNQ@?^BOH2yUxaV&F$H8yX1GIweAN$9vg;i2%V1jy&2^G%>40U=2$Bb
zIY{GqjNtu>IN};o><B|k)V>Z3K^RsbCEHsyE<c50*5})zr>0@(W$bPvNh%U)fMJBI
ztzL90`6mKOcQARCR`hLJxoJNcH%Z-@5fwz9AnKpI+y&56_Vs2bH6Dz0p0V>nUKlz0
z^GKf~rh21M+<>hQ7isP#B6I6>vOuh&i%b%13fF0xM5ox-!5$;bJ{78`mJ+wv`<%56
zu=GKt3Z^eH-f{er1&GUXpHnVeus4bHN|xc;=t{`u>Fdi(H+V66U?l)L|Ek5qcz7!O
zz1P<@l6G5_ab<WoA<TXnCH|`sCclkTp9z+S7WL3Cpegm{x&eip@W)87w&b<3VX380
zLC8Teukn(j#Z+bCaQl7$<1YUJ$J+q&=CBmAN10kcF%dnV7hG^RF4`INRpowOm55>$
zo`9oC1db;8XH~JV50g|r)$aEq9$g|LYSZ^|R|q9&rXr+4<fq`%{xkZG-CY<yi!T&j
z^NI;b%XgC6Mr4;>uSZi_QgOh4fzp%z?l@$Cx2?~@{+<Te(e});-<r*nj}B-KTpmC?
zT~g`UR)Kc{Z9w>Q(>zyHk1$!e<&3-`Hyqd+xs$(o@bLlvJ=o<q_$a|wVyp=g`V1lD
z;hFz?K0%ThIbZa@bCz3*RS2|G|8sJ|#pyqgVJ~NK>F5~SIs0ej*pW%0b+->2uWKAU
zd%_6P@J&<ieM{gJglDZ2WrvheNB#Y#l1U_7oKxc0hmeU@a!HjmMiqHJPK!fvbVZ9Z
z9EKl${@Dlug*B-E<3%RZ@|!sO=tp@Zei43IL=oh^WmfeKnC1}@j>R7F21b2&!$ZZZ
zZ3tT!{a#Hh>2)v>RYZqHPNq2q+A&W7c5+nha+hrgY+kAuiL<s_>zN_xZS?^5aCcm$
z2ce%*^fV2jBMG>APXJBV0iroj*-DHlrQu+!aR^+bRY@e-nOL|SH4c8YJQ~Xl3F(Q^
zk(*VrMT5GnS2`A=iX;s^;g@Cns>^QP#fp~-qBiGjmbQ2mTgpwg^#Avv#F27-Rjz7I
z?sD+B-V6g_s)41SHpRbd$YPU`xI5R4cToK8W*S23;QOG2-TdQ6no&$CvXZj@_B|w_
zYvc9{_0+wIHahJ8&Ic8vYAgR5!FFPNhL*KEPn;<>mZn!ad15byhumr^DiaxE4p9lx
z|0(Ru<DuNU04^pm*2t2C7|cwCCX^-al)*%pY}chy=vJ0)mh9UIV^4_4nq@4>MY=+a
z?b?!Ly{<J%5tFi1gZB)#`);4l`_5m#&*xd@{FZZ`^PK1WzGr#YKBuX5T*67WE5TM-
zzbZx|d+I0B6S|1V_F#|G19?2(Z}(~~M*yjF<ovtON^(!K_NYss&xY;J$q9d>IX=l^
zhZFqtNbv}pW%XQZ84meDrSY6c&NVbP=V~Vr;}!VavRuLWdpA=cm0cgr%IVNt;T`JG
zb2vTCMSrg@1(_;e78^(M!^H75jb(f^jB=n&#Z?WZj&V}6sS+;bF)R9=ooHOWB-$!j
z6^%t)1+cw+bji*2cj>F$`nDClT<r<#`ue2(;zj<>zg|^%5R_z*B9WY&Y{ea7Y}8X+
zP0d(zH{@vRBDTiDM(?N0HH*`{$Lg2Ri8hRS9^&$DdM)yS2_lzDA?iKkaKYa*K_fkh
zbrl_~SQU|^1*=CT&3;!V&d<~A%s9w>yFsPnv#D8ApRwgZ?FJlOKHd%YY7r^YMHQyp
zjp7LjC(E;qq6@RB6OL)}2fhrq1Y1OiXln?_nSUP#<{J8(>ltuaxy*=i;+%7x$_+7f
zf?4J#0%`x=Qa4v}#@F-mKN5M<?La?7hp+stPvt^FSyf}$4w)EhllS@iPTl{J^fPH#
zlk<f3D|#PdJv`0XDKk8;rF;n&`A{-E<9?R0!U#Y%{?bMWE79L1>iO85CmtEc%dpDL
z$zeZ+&u(IDU7ONPGpT<h7^uMa*zNH(w2OQ{!T;g4oxAaw9u3^o^WebEVo8wh(*1>Y
zPM_uj)TSnPTkdGoE|Z`Z<JdtN^VQA%_XezNv2os9#nQ>w48Qj_rFBoz(%)tPOg4mL
z@b+AFS%w~qw2KdZUu#Cix8(YbRXkyL6Fxlbzvjh%BAv+++P0iP8)&<WGJ01z{OOtF
z%SKsLBo+1MlSEyB#?7fjD=y)8R>Sh;-qRDzTps3YCj1ls@KQBZndiCOi<A?dAIp!u
zsj95K<jMp>ZJ9ZU!%&>EDIG)YcC_*&F3XYuC~$4t-+dOa*i3V|C0|r7)$m-c@I#vu
zb_-+?pT*Z!gv59i=DQud<3l}Asq)Z0{oSYZ@FccfeARx7q~kK-xRNmD%IqKcExDlX
zc9<;oSB1dG2Vg_`?VQxdwl60xHigHEubX^2<HWmtE<XZ)n_bs+F!OHqa2e^r&t2A8
zj#v3SVAmwG7YMwVD3L^)&eP!Y*%<Y0fB$%3EEp5&*Upq9REmSFth9$DJ1w-R=Cia~
z8dx%1*)JoVK@&R73q&z20?(ft3@O|k`9GJKj=r3UbE%0H=QDUXY6W4tt%I;TAn4Q&
zLybbkzKs4k#zWu27S^p~EDo}iKAUMFRn%tE(XMm1UF-6G`U|!*`~P_`p;eG@xs3d7
zFNQF4XBlNg)bVi<l?lK5ds2xdA51SlBHj_^pDabO3EAq&v>1~b!67{es}cUZntiJS
zZ7N%$=#)ZO8cud)nw<Z>G!;uoviiCMj^OQii?O5#M^SF}5Lz3sZJ3StIjG_V4Vgi3
z)1jeT8p5oW1GO_Cph$y8HE$#`1RAdUp+XKcrq$8dts8j!zZ<+2Bqb%@m1g#%3o1Lb
zq~=gM{-g<QNfK*mOZm}sTYez*{SW#7U)M33tpmJq=WET_xeqy^$2G`O9k3E7y&dhV
zep1WHexUbd%%NN3=$y!-v9_ahUd&?kj+ZK`1-18UKaTarE@kB13a;jv$(!NvH$B+l
zYXRA+NGLNKrs<Wh5a6~X_P9V14VP5k{8qZ=Fs%}^a?iV@6ca%7eqN_%j9IjTxFkjs
z-o?l{0`<|g>;wYLO&g?-a^p!67j(}<Dkw~$J-8CExSJJa9sv>px?7GQ$bz>&l^hRK
z+6J>~1^RaYjs4i!1~YGOVn#Ux;h2^A;nx!rw?AK%mX^Msp8g_TLtdiNXBp7uHa0fR
z71}X1HQxZ$+WcA!4LltLf#Ch(EO_EzDUor?IvxNUZEMrl%3PeC^_m-aI45amXP0pQ
zKKD~H^2j`ZGH=|tL5~QwY8J&Q8JL?Vi4K?m(hh{^@VyIfmxTbN@XD1h>g_%2?EC>h
z-U)<R>eABE{QUgN!YCJfoS|POSzeF+@EEEWHAe{2*(xh5OUx$eulolF!*y4Hh^xdI
z;$OtU7>XcV6HwdT51eyJO)nPz72s^P<QcoIE}ZlA^*wc}>49FH#PMH#(9_dnj0T7;
zRaiE)6;OeJ%$|v`wjMdH%?SwDz;7#u88KdOyZ5KulBBGqZ5|R)u#pljBr?b9sxk?m
zq2-^Nnu>IN3YcI>w!J1>1W4tR2FU!tmHB(Wp?0qg1JfTRCs$NhNJGt@mg3JXC~))_
z9*Px=>O;9o7kSJL)CH`IRVf0q5vZvE3ZI&uMz~!Z;Ta!=L1r)*zuCCx>r>WzHv%&>
zz0VdKozo)htUvCqp58QB>-F)Z&7C$N*<MsYJPT?E-f@HcfrAGP7&5o@^R(NG;M;~h
z5grOK2DbUp(a~>TWHzB7q?Pl>lk1t))oKj0_4V}@*@%WiL#}u|f;+bqRx6V)&Et|w
zx{P+s25>faiiPo%=F54cJ#!o^ot?d6a(qZE5gz|ldb+y0Q@cH4{35SGeh#ETI5;@i
zUb;b)Zf$L)YXHEORZ<l8yU^qh$jODJSifbiw%(K+vIUEJ(a_M3fL&tBK|Z52yDNa!
z@R=?sY^nXGEnWtX#{<iDTDSIH1X8ue!os4ZrA0zQVybtOEpS!oUKf>$zI4GNL8Koj
zqj>Gm&`>oswH~2|Ru#IMcFXT!D4JHgFa7H7?oNR(S4=I5k;4w9Jguo|dUX5o<7L1K
zXPpG*{LItpYFbqox0C&o>*Gd;`h%b>6Fyc_n>RH)GX=Pu({DHfx_hqa=;+wg`6N-5
zPmu6^eSNkgCm+0=s6Z+Bb2~w~X~1mv{6w=)5D`M$LM&h~m?KI`LG`xH{r$z0ib-8v
zhO>%~F4P-AHWJe|#Ai?^2Af)1cB(3PYPXJ!WzcLJBsb`3r?_Qwb~Zsq5v%Ma7nk$J
zb#GM+1^N47CH_F7i?BF(0{NLD&A?2|&vVzi!|>)#q~%g4*aXGIko!`B`Wt0XYzaHs
zj;X2b_2F7_%j0Ik)klCV+}Y9b!^pm4E~rR4v8<w^c!-gjmXL|qBLtc7pkz>EEG0di
z2TPV?5fB&WE+sq1c=0cMEv}<Vfog=7?g_X8sW?(L9`i#-hPd+z3Iq#C{5wcV$;t9^
za_1S!)=yl7;;2^Br_JbZF4PC`@W}ezZBu3K;u+a5DJi+u&JWrO6Vy`}RsSI__~~@)
zY^sh6VP`NOi=TjP2-o=WFT|iybs3r0P@|w(l!JoQ;-o*xS}UM$_N^@O!3TT;u}`8x
zvS4H=NHLDt(-&W{1!`RXqmmN~?Do#*&qL1~b)rNZ;pb1%M`1CTU<C`@oj^3Gl`NLS
z>E=B{ImQlm#@dY>*0jqKajR1;Q2={q%ETJXQl&R_{hM85V<W{sL?uo$3i&KKD2Qdv
zjelc9W@DqLzdy8FAo5Off0Hpc2-S(+z%aCv70wAjcbS?G?|Zk^Dc4@N#sF&iz=rM^
z(597zfw?Nq8-85mo?*hD%e#UXXq}y0RXA9}N}R5&*J+nm0ma_TLcz^{4QaSO0LeWK
zztf3Bb-u^^1pXWyoqQ7Ey3D%p6$1WSkAgnD8ULE8`oV)-0W-Eyev^%%VCg=8c3+xM
zi_gav@$6Ft0CZgEjGVlDQ~vP4fUwqO&=as(B5@{SB)plq`B5A$>moUh_T|&t$)8yh
znmwL@iD#}vrua`Iu7Pz{P*5oIh8(b)9v-3N>Q@v|Px@}>W3e2{#yux#p8j8!X25z$
z#CgInw!29s>6xXeI=i~Ml2^pE*!kDY0h}E&uJv&~4o`x#(op!Ca_YTY5fl_?tP)l2
z0nHp99`^S3270-)F)OO~7U8k)stdrI*A^y@vX5L5i_wj*Tw>0<ywQCr1*)_bXOGKS
zEl3=OlI3*$Q)VC`EaP#&;6rz0H~aGJo5Fa?A|N;F7iVKav2|)RRm!@`Df`vCvK;~(
zh%z2^kN1gQt(&iI5q4U?g&GuY)qj?k<AxSy=jKd}Y)vId_=#k+mX<`o9y*Z{6O*JF
zL+02v6TSRNe)*UP<i?(!moGt;gD+?vKK$^7v5HG$4tajGM&u7nyguSN!rN7KZ=Qe<
z<G|Y2<+-V;OKYeGy{S(JXGnjKta=nu)fhO?tougmd_5uUG4Zg9JwTtKBO8oGsA&WS
z+!--63oQP@M;A8z#&OECfQJsV>M4mb@Q7gm@i_AqXl;Z1MX;=yfdZh+?3uTI8b#Oj
zet%|r2%A43Xc79aKz4zO(O*jvs_-zK1vD0i#_HS!uNJ~B4BQg9GJBcI1knI^{s-*!
z=824i%GU9+|LH&6@w{xYD^A>;y^<#yGhWm6uf7DoECk*Ez80UDD6e>-&}Bs7h33ni
zoZ7eS)7>GzCnhHxt90i<FIt_}9PG`mt*rex`^8|}yV-IpWiZ$t!EJX@@!vho)DA++
z<S*#gOg~dc?k|5cOu7g(6vlYl4OP^SM1DpI3n^P9-hkV2NJB#V_uEdf2yNe2MoUB}
txzm|FM#w=N&LOuUh@{=$)_=2E*kqAV>Wv<iEYD+sACX|BQ+m>t{13q)j6eVY
literal 0
HcmV?d00001
diff --git a/doc/guides/rel_notes/release_20_05.rst b/doc/guides/rel_notes/release_20_05.rst
index 000bbf501..f404ee7fd 100644
--- a/doc/guides/rel_notes/release_20_05.rst
+++ b/doc/guides/rel_notes/release_20_05.rst
@@ -62,6 +62,11 @@ New Features
* Added support for matching on IPv4 Time To Live and IPv6 Hop Limit.
+* **Updated the Intel ice driver.**
+
+ Updated the Intel ice driver with new features and improvements, including:
+
+ * Added support for DCF (Device Config Function) feature.
Removed Items
-------------
diff --git a/drivers/common/Makefile b/drivers/common/Makefile
index 96bd7ac6e..df2e840cf 100644
--- a/drivers/common/Makefile
+++ b/drivers/common/Makefile
@@ -31,6 +31,7 @@ DIRS-y += dpaax
endif
IAVF-y := $(CONFIG_RTE_LIBRTE_IAVF_PMD)
+IAVF-y += $(CONFIG_RTE_LIBRTE_ICE_PMD)
ifneq (,$(findstring y,$(IAVF-y)))
DIRS-y += iavf
endif
diff --git a/drivers/net/ice/Makefile b/drivers/net/ice/Makefile
index e22c34287..f493c9ed7 100644
--- a/drivers/net/ice/Makefile
+++ b/drivers/net/ice/Makefile
@@ -11,9 +11,11 @@ LIB = librte_pmd_ice.a
CFLAGS += -O3
CFLAGS += $(WERROR_FLAGS)
CFLAGS += -DALLOW_EXPERIMENTAL_API
+CFLAGS += -I$(RTE_SDK)/drivers/common/iavf
LDLIBS += -lrte_eal -lrte_mbuf -lrte_ethdev -lrte_kvargs
LDLIBS += -lrte_bus_pci -lrte_mempool -lrte_hash
+LDLIBS += -lrte_net -lrte_common_iavf
EXPORT_MAP := rte_pmd_ice_version.map
@@ -84,6 +86,9 @@ ifeq ($(CC_AVX2_SUPPORT), 1)
endif
SRCS-$(CONFIG_RTE_LIBRTE_ICE_PMD) += ice_generic_flow.c
+SRCS-$(CONFIG_RTE_LIBRTE_ICE_PMD) += ice_dcf.c
+SRCS-$(CONFIG_RTE_LIBRTE_ICE_PMD) += ice_dcf_ethdev.c
+
# install this header file
SYMLINK-$(CONFIG_RTE_LIBRTE_ICE_PMD)-include := rte_pmd_ice.h
diff --git a/drivers/net/ice/ice_dcf.c b/drivers/net/ice/ice_dcf.c
new file mode 100644
index 000000000..0a99cceb1
--- /dev/null
+++ b/drivers/net/ice/ice_dcf.c
@@ -0,0 +1,474 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#include <sys/queue.h>
+#include <stdio.h>
+#include <errno.h>
+#include <stdint.h>
+#include <string.h>
+#include <unistd.h>
+#include <stdarg.h>
+#include <inttypes.h>
+#include <rte_byteorder.h>
+#include <rte_common.h>
+
+#include <rte_pci.h>
+#include <rte_atomic.h>
+#include <rte_eal.h>
+#include <rte_ether.h>
+#include <rte_ethdev_driver.h>
+#include <rte_ethdev_pci.h>
+#include <rte_malloc.h>
+#include <rte_memzone.h>
+#include <rte_dev.h>
+
+#include "ice_dcf.h"
+
+#define ICE_DCF_AQ_LEN 32
+#define ICE_DCF_AQ_BUF_SZ 4096
+
+#define ICE_DCF_ARQ_MAX_RETRIES 200
+#define ICE_DCF_ARQ_CHECK_TIME 2 /* msecs */
+
+#define ICE_DCF_VF_RES_BUF_SZ \
+ (sizeof(struct virtchnl_vf_resource) + \
+ IAVF_MAX_VF_VSI * sizeof(struct virtchnl_vsi_resource))
+
+static __rte_always_inline int
+ice_dcf_send_cmd_req_no_irq(struct ice_dcf_hw *hw, enum virtchnl_ops op,
+ uint8_t *req_msg, uint16_t req_msglen)
+{
+ return iavf_aq_send_msg_to_pf(&hw->avf, op, IAVF_SUCCESS,
+ req_msg, req_msglen, NULL);
+}
+
+static int
+ice_dcf_recv_cmd_rsp_no_irq(struct ice_dcf_hw *hw, enum virtchnl_ops op,
+ uint8_t *rsp_msgbuf, uint16_t rsp_buflen,
+ uint16_t *rsp_msglen)
+{
+ struct iavf_arq_event_info event;
+ enum virtchnl_ops v_op;
+ int i = 0;
+ int err;
+
+ event.buf_len = rsp_buflen;
+ event.msg_buf = rsp_msgbuf;
+
+ do {
+ err = iavf_clean_arq_element(&hw->avf, &event, NULL);
+ if (err != IAVF_SUCCESS)
+ goto again;
+
+ v_op = rte_le_to_cpu_32(event.desc.cookie_high);
+ if (v_op != op)
+ goto again;
+
+ if (rsp_msglen != NULL)
+ *rsp_msglen = event.msg_len;
+ return rte_le_to_cpu_32(event.desc.cookie_low);
+
+again:
+ rte_delay_ms(ICE_DCF_ARQ_CHECK_TIME);
+ } while (i++ < ICE_DCF_ARQ_MAX_RETRIES);
+
+ return -EIO;
+}
+
+static __rte_always_inline void
+ice_dcf_aq_cmd_clear(struct ice_dcf_hw *hw, struct dcf_virtchnl_cmd *cmd)
+{
+ rte_spinlock_lock(&hw->vc_cmd_queue_lock);
+
+ TAILQ_REMOVE(&hw->vc_cmd_queue, cmd, next);
+
+ rte_spinlock_unlock(&hw->vc_cmd_queue_lock);
+}
+
+static __rte_always_inline void
+ice_dcf_vc_cmd_set(struct ice_dcf_hw *hw, struct dcf_virtchnl_cmd *cmd)
+{
+ cmd->v_ret = IAVF_ERR_NOT_READY;
+ cmd->rsp_msglen = 0;
+ cmd->pending = 1;
+
+ rte_spinlock_lock(&hw->vc_cmd_queue_lock);
+
+ TAILQ_INSERT_TAIL(&hw->vc_cmd_queue, cmd, next);
+
+ rte_spinlock_unlock(&hw->vc_cmd_queue_lock);
+}
+
+static __rte_always_inline int
+ice_dcf_vc_cmd_send(struct ice_dcf_hw *hw, struct dcf_virtchnl_cmd *cmd)
+{
+ return iavf_aq_send_msg_to_pf(&hw->avf,
+ cmd->v_op, IAVF_SUCCESS,
+ cmd->req_msg, cmd->req_msglen, NULL);
+}
+
+static __rte_always_inline void
+ice_dcf_aq_cmd_handle(struct ice_dcf_hw *hw, struct iavf_arq_event_info *info)
+{
+ struct dcf_virtchnl_cmd *cmd;
+ enum virtchnl_ops v_op;
+ enum iavf_status v_ret;
+ uint16_t aq_op;
+
+ aq_op = rte_le_to_cpu_16(info->desc.opcode);
+ if (unlikely(aq_op != iavf_aqc_opc_send_msg_to_vf)) {
+ PMD_DRV_LOG(ERR,
+ "Request %u is not supported yet", aq_op);
+ return;
+ }
+
+ v_op = rte_le_to_cpu_32(info->desc.cookie_high);
+ if (unlikely(v_op == VIRTCHNL_OP_EVENT))
+ return;
+
+ v_ret = rte_le_to_cpu_32(info->desc.cookie_low);
+
+ rte_spinlock_lock(&hw->vc_cmd_queue_lock);
+
+ TAILQ_FOREACH(cmd, &hw->vc_cmd_queue, next) {
+ if (cmd->v_op == v_op && cmd->pending) {
+ cmd->v_ret = v_ret;
+ cmd->rsp_msglen = RTE_MIN(info->msg_len,
+ cmd->rsp_buflen);
+ if (likely(cmd->rsp_msglen != 0))
+ rte_memcpy(cmd->rsp_msgbuf, info->msg_buf,
+ cmd->rsp_msglen);
+
+ /* prevent compiler reordering */
+ rte_compiler_barrier();
+ cmd->pending = 0;
+ break;
+ }
+ }
+
+ rte_spinlock_unlock(&hw->vc_cmd_queue_lock);
+}
+
+static void
+ice_dcf_handle_virtchnl_msg(struct ice_dcf_hw *hw)
+{
+ struct iavf_arq_event_info info;
+ uint16_t pending = 1;
+ int ret;
+
+ info.buf_len = ICE_DCF_AQ_BUF_SZ;
+ info.msg_buf = hw->arq_buf;
+
+ while (pending) {
+ ret = iavf_clean_arq_element(&hw->avf, &info, &pending);
+ if (ret != IAVF_SUCCESS)
+ break;
+
+ ice_dcf_aq_cmd_handle(hw, &info);
+ }
+}
+
+static int
+ice_dcf_init_check_api_version(struct ice_dcf_hw *hw)
+{
+#define ICE_CPF_VIRTCHNL_VERSION_MAJOR_START 1
+#define ICE_CPF_VIRTCHNL_VERSION_MINOR_START 1
+ struct virtchnl_version_info version, *pver;
+ int err;
+
+ version.major = VIRTCHNL_VERSION_MAJOR;
+ version.minor = VIRTCHNL_VERSION_MINOR;
+ err = ice_dcf_send_cmd_req_no_irq(hw, VIRTCHNL_OP_VERSION,
+ (uint8_t *)&version, sizeof(version));
+ if (err) {
+ PMD_INIT_LOG(ERR, "Failed to send OP_VERSION");
+ return err;
+ }
+
+ pver = &hw->virtchnl_version;
+ err = ice_dcf_recv_cmd_rsp_no_irq(hw, VIRTCHNL_OP_VERSION,
+ (uint8_t *)pver, sizeof(*pver), NULL);
+ if (err) {
+ PMD_INIT_LOG(ERR, "Failed to get response of OP_VERSION");
+ return -1;
+ }
+
+ PMD_INIT_LOG(DEBUG,
+ "Peer PF API version: %u.%u", pver->major, pver->minor);
+
+ if (pver->major < ICE_CPF_VIRTCHNL_VERSION_MAJOR_START ||
+ (pver->major == ICE_CPF_VIRTCHNL_VERSION_MAJOR_START &&
+ pver->minor < ICE_CPF_VIRTCHNL_VERSION_MINOR_START)) {
+ PMD_INIT_LOG(ERR,
+ "VIRTCHNL API version should not be lower than (%u.%u)",
+ ICE_CPF_VIRTCHNL_VERSION_MAJOR_START,
+ ICE_CPF_VIRTCHNL_VERSION_MAJOR_START);
+ return -1;
+ } else if (pver->major > VIRTCHNL_VERSION_MAJOR ||
+ (pver->major == VIRTCHNL_VERSION_MAJOR &&
+ pver->minor > VIRTCHNL_VERSION_MINOR)) {
+ PMD_INIT_LOG(ERR,
+ "PF/VF API version mismatch:(%u.%u)-(%u.%u)",
+ pver->major, pver->minor,
+ VIRTCHNL_VERSION_MAJOR, VIRTCHNL_VERSION_MINOR);
+ return -1;
+ }
+
+ PMD_INIT_LOG(DEBUG, "Peer is supported PF host");
+
+ return 0;
+}
+
+static int
+ice_dcf_get_vf_resource(struct ice_dcf_hw *hw)
+{
+ uint32_t caps;
+ int err, i;
+
+ caps = VIRTCHNL_VF_OFFLOAD_WB_ON_ITR | VIRTCHNL_VF_OFFLOAD_RX_POLLING |
+ VIRTCHNL_VF_CAP_ADV_LINK_SPEED |
+ VF_BASE_MODE_OFFLOADS;
+
+ err = ice_dcf_send_cmd_req_no_irq(hw, VIRTCHNL_OP_GET_VF_RESOURCES,
+ (uint8_t *)&caps, sizeof(caps));
+ if (err) {
+ PMD_DRV_LOG(ERR, "Failed to send msg OP_GET_VF_RESOURCE");
+ return err;
+ }
+
+ err = ice_dcf_recv_cmd_rsp_no_irq(hw, VIRTCHNL_OP_GET_VF_RESOURCES,
+ (uint8_t *)hw->vf_res,
+ ICE_DCF_VF_RES_BUF_SZ, NULL);
+ if (err) {
+ PMD_DRV_LOG(ERR, "Failed to get response of OP_GET_VF_RESOURCE");
+ return -1;
+ }
+
+ iavf_vf_parse_hw_config(&hw->avf, hw->vf_res);
+
+ hw->vsi_res = NULL;
+ for (i = 0; i < hw->vf_res->num_vsis; i++) {
+ if (hw->vf_res->vsi_res[i].vsi_type == VIRTCHNL_VSI_SRIOV)
+ hw->vsi_res = &hw->vf_res->vsi_res[i];
+ }
+
+ if (!hw->vsi_res) {
+ PMD_DRV_LOG(ERR, "no LAN VSI found");
+ return -1;
+ }
+
+ hw->vsi_id = hw->vsi_res->vsi_id;
+ PMD_DRV_LOG(DEBUG, "VSI ID is %u", hw->vsi_id);
+
+ return 0;
+}
+
+static int
+ice_dcf_check_reset_done(struct ice_dcf_hw *hw)
+{
+#define ICE_DCF_RESET_WAIT_CNT 50
+ struct iavf_hw *avf = &hw->avf;
+ int i, reset;
+
+ for (i = 0; i < ICE_DCF_RESET_WAIT_CNT; i++) {
+ reset = IAVF_READ_REG(avf, IAVF_VFGEN_RSTAT) &
+ IAVF_VFGEN_RSTAT_VFR_STATE_MASK;
+ reset = reset >> IAVF_VFGEN_RSTAT_VFR_STATE_SHIFT;
+
+ if (reset == VIRTCHNL_VFR_VFACTIVE ||
+ reset == VIRTCHNL_VFR_COMPLETED)
+ break;
+
+ rte_delay_ms(20);
+ }
+
+ if (i >= ICE_DCF_RESET_WAIT_CNT)
+ return -1;
+
+ return 0;
+}
+
+static inline void
+ice_dcf_enable_irq0(struct ice_dcf_hw *hw)
+{
+ struct iavf_hw *avf = &hw->avf;
+
+ /* Enable admin queue interrupt trigger */
+ IAVF_WRITE_REG(avf, IAVF_VFINT_ICR0_ENA1,
+ IAVF_VFINT_ICR0_ENA1_ADMINQ_MASK);
+ IAVF_WRITE_REG(avf, IAVF_VFINT_DYN_CTL01,
+ IAVF_VFINT_DYN_CTL01_INTENA_MASK |
+ IAVF_VFINT_DYN_CTL01_CLEARPBA_MASK |
+ IAVF_VFINT_DYN_CTL01_ITR_INDX_MASK);
+
+ IAVF_WRITE_FLUSH(avf);
+}
+
+static inline void
+ice_dcf_disable_irq0(struct ice_dcf_hw *hw)
+{
+ struct iavf_hw *avf = &hw->avf;
+
+ /* Disable all interrupt types */
+ IAVF_WRITE_REG(avf, IAVF_VFINT_ICR0_ENA1, 0);
+ IAVF_WRITE_REG(avf, IAVF_VFINT_DYN_CTL01,
+ IAVF_VFINT_DYN_CTL01_ITR_INDX_MASK);
+
+ IAVF_WRITE_FLUSH(avf);
+}
+
+static void
+ice_dcf_dev_interrupt_handler(void *param)
+{
+ struct ice_dcf_hw *hw = param;
+
+ ice_dcf_disable_irq0(hw);
+
+ ice_dcf_handle_virtchnl_msg(hw);
+
+ ice_dcf_enable_irq0(hw);
+}
+
+int
+ice_dcf_execute_virtchnl_cmd(struct ice_dcf_hw *hw,
+ struct dcf_virtchnl_cmd *cmd)
+{
+ int i = 0;
+ int err;
+
+ if ((cmd->req_msg && !cmd->req_msglen) ||
+ (!cmd->req_msg && cmd->req_msglen) ||
+ (cmd->rsp_msgbuf && !cmd->rsp_buflen) ||
+ (!cmd->rsp_msgbuf && cmd->rsp_buflen))
+ return -EINVAL;
+
+ rte_spinlock_lock(&hw->vc_cmd_send_lock);
+ ice_dcf_vc_cmd_set(hw, cmd);
+
+ err = ice_dcf_vc_cmd_send(hw, cmd);
+ if (err) {
+ PMD_DRV_LOG(ERR, "fail to send cmd %d", cmd->v_op);
+ goto ret;
+ }
+
+ do {
+ if (!cmd->pending)
+ break;
+
+ rte_delay_ms(ICE_DCF_ARQ_CHECK_TIME);
+ } while (i++ < ICE_DCF_ARQ_MAX_RETRIES);
+
+ if (cmd->v_ret != IAVF_SUCCESS) {
+ err = -1;
+ PMD_DRV_LOG(ERR,
+ "No response (%d times) or return failure (%d) for cmd %d",
+ i, cmd->v_ret, cmd->v_op);
+ }
+
+ret:
+ ice_dcf_aq_cmd_clear(hw, cmd);
+ rte_spinlock_unlock(&hw->vc_cmd_send_lock);
+ return err;
+}
+
+int
+ice_dcf_init_hw(struct rte_eth_dev *eth_dev, struct ice_dcf_hw *hw)
+{
+ struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ int ret;
+
+ hw->avf.hw_addr = pci_dev->mem_resource[0].addr;
+ hw->avf.back = hw;
+
+ hw->avf.bus.bus_id = pci_dev->addr.bus;
+ hw->avf.bus.device = pci_dev->addr.devid;
+ hw->avf.bus.func = pci_dev->addr.function;
+
+ hw->avf.device_id = pci_dev->id.device_id;
+ hw->avf.vendor_id = pci_dev->id.vendor_id;
+ hw->avf.subsystem_device_id = pci_dev->id.subsystem_device_id;
+ hw->avf.subsystem_vendor_id = pci_dev->id.subsystem_vendor_id;
+
+ hw->avf.aq.num_arq_entries = ICE_DCF_AQ_LEN;
+ hw->avf.aq.num_asq_entries = ICE_DCF_AQ_LEN;
+ hw->avf.aq.arq_buf_size = ICE_DCF_AQ_BUF_SZ;
+ hw->avf.aq.asq_buf_size = ICE_DCF_AQ_BUF_SZ;
+
+ rte_spinlock_init(&hw->vc_cmd_send_lock);
+ rte_spinlock_init(&hw->vc_cmd_queue_lock);
+ TAILQ_INIT(&hw->vc_cmd_queue);
+
+ hw->arq_buf = rte_zmalloc("arq_buf", ICE_DCF_AQ_BUF_SZ, 0);
+ if (hw->arq_buf == NULL) {
+ PMD_INIT_LOG(ERR, "unable to allocate AdminQ buffer memory");
+ goto err;
+ }
+
+ ret = iavf_set_mac_type(&hw->avf);
+ if (ret) {
+ PMD_INIT_LOG(ERR, "set_mac_type failed: %d", ret);
+ goto err;
+ }
+
+ ret = ice_dcf_check_reset_done(hw);
+ if (ret) {
+ PMD_INIT_LOG(ERR, "VF is still resetting");
+ goto err;
+ }
+
+ ret = iavf_init_adminq(&hw->avf);
+ if (ret) {
+ PMD_INIT_LOG(ERR, "init_adminq failed: %d", ret);
+ goto err;
+ }
+
+ if (ice_dcf_init_check_api_version(hw)) {
+ PMD_INIT_LOG(ERR, "check_api version failed");
+ goto err_api;
+ }
+
+ hw->vf_res = rte_zmalloc("vf_res", ICE_DCF_VF_RES_BUF_SZ, 0);
+ if (hw->vf_res == NULL) {
+ PMD_INIT_LOG(ERR, "unable to allocate vf_res memory");
+ goto err_api;
+ }
+
+ if (ice_dcf_get_vf_resource(hw)) {
+ PMD_INIT_LOG(ERR, "Failed to get VF resource");
+ goto err_alloc;
+ }
+
+ rte_intr_callback_register(&pci_dev->intr_handle,
+ ice_dcf_dev_interrupt_handler, hw);
+ rte_intr_enable(&pci_dev->intr_handle);
+ ice_dcf_enable_irq0(hw);
+
+ return 0;
+
+err_alloc:
+ rte_free(hw->vf_res);
+err_api:
+ iavf_shutdown_adminq(&hw->avf);
+err:
+ rte_free(hw->arq_buf);
+
+ return -1;
+}
+
+void
+ice_dcf_uninit_hw(struct rte_eth_dev *eth_dev, struct ice_dcf_hw *hw)
+{
+ struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_intr_handle *intr_handle = &pci_dev->intr_handle;
+
+ ice_dcf_disable_irq0(hw);
+ rte_intr_disable(intr_handle);
+ rte_intr_callback_unregister(intr_handle,
+ ice_dcf_dev_interrupt_handler, hw);
+
+ iavf_shutdown_adminq(&hw->avf);
+
+ rte_free(hw->arq_buf);
+ rte_free(hw->vf_res);
+}
diff --git a/drivers/net/ice/ice_dcf.h b/drivers/net/ice/ice_dcf.h
new file mode 100644
index 000000000..f44c09db2
--- /dev/null
+++ b/drivers/net/ice/ice_dcf.h
@@ -0,0 +1,52 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#ifndef _ICE_DCF_H_
+#define _ICE_DCF_H_
+
+#include <rte_ethdev_driver.h>
+
+#include <iavf_prototype.h>
+#include <iavf_adminq_cmd.h>
+#include <iavf_type.h>
+
+#include "ice_logs.h"
+
+struct dcf_virtchnl_cmd {
+ TAILQ_ENTRY(dcf_virtchnl_cmd) next;
+
+ enum virtchnl_ops v_op;
+ enum iavf_status v_ret;
+
+ uint16_t req_msglen;
+ uint8_t *req_msg;
+
+ uint16_t rsp_msglen;
+ uint16_t rsp_buflen;
+ uint8_t *rsp_msgbuf;
+
+ volatile int pending;
+};
+
+struct ice_dcf_hw {
+ struct iavf_hw avf;
+
+ rte_spinlock_t vc_cmd_send_lock;
+ rte_spinlock_t vc_cmd_queue_lock;
+ TAILQ_HEAD(, dcf_virtchnl_cmd) vc_cmd_queue;
+ uint8_t *arq_buf;
+
+ struct virtchnl_version_info virtchnl_version;
+ struct virtchnl_vf_resource *vf_res; /* VF resource */
+ struct virtchnl_vsi_resource *vsi_res; /* LAN VSI */
+ uint16_t vsi_id;
+};
+
+int ice_dcf_execute_virtchnl_cmd(struct ice_dcf_hw *hw,
+ struct dcf_virtchnl_cmd *cmd);
+
+int ice_dcf_init_hw(struct rte_eth_dev *eth_dev, struct ice_dcf_hw *hw);
+void ice_dcf_uninit_hw(struct rte_eth_dev *eth_dev, struct ice_dcf_hw *hw);
+
+#endif /* _ICE_DCF_H_ */
diff --git a/drivers/net/ice/ice_dcf_ethdev.c b/drivers/net/ice/ice_dcf_ethdev.c
new file mode 100644
index 000000000..23f82a487
--- /dev/null
+++ b/drivers/net/ice/ice_dcf_ethdev.c
@@ -0,0 +1,317 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#include <errno.h>
+#include <stdbool.h>
+#include <sys/types.h>
+#include <sys/ioctl.h>
+#include <unistd.h>
+
+#include <rte_interrupts.h>
+#include <rte_debug.h>
+#include <rte_pci.h>
+#include <rte_atomic.h>
+#include <rte_eal.h>
+#include <rte_ether.h>
+#include <rte_ethdev_pci.h>
+#include <rte_kvargs.h>
+#include <rte_malloc.h>
+#include <rte_memzone.h>
+#include <rte_dev.h>
+
+#include <iavf_devids.h>
+
+#include "ice_generic_flow.h"
+#include "ice_dcf_ethdev.h"
+
+static uint16_t
+ice_dcf_recv_pkts(__rte_unused void *rx_queue,
+ __rte_unused struct rte_mbuf **bufs,
+ __rte_unused uint16_t nb_pkts)
+{
+ return 0;
+}
+
+static uint16_t
+ice_dcf_xmit_pkts(__rte_unused void *tx_queue,
+ __rte_unused struct rte_mbuf **bufs,
+ __rte_unused uint16_t nb_pkts)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_start(struct rte_eth_dev *dev)
+{
+ dev->data->dev_link.link_status = ETH_LINK_UP;
+
+ return 0;
+}
+
+static void
+ice_dcf_dev_stop(struct rte_eth_dev *dev)
+{
+ dev->data->dev_link.link_status = ETH_LINK_DOWN;
+}
+
+static int
+ice_dcf_dev_configure(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_info_get(struct rte_eth_dev *dev,
+ struct rte_eth_dev_info *dev_info)
+{
+ struct ice_dcf_adapter *adapter = dev->data->dev_private;
+
+ dev_info->max_mac_addrs = 1;
+ dev_info->max_rx_pktlen = (uint32_t)-1;
+ dev_info->max_rx_queues = RTE_DIM(adapter->rxqs);
+ dev_info->max_tx_queues = RTE_DIM(adapter->txqs);
+
+ return 0;
+}
+
+static int
+ice_dcf_stats_get(__rte_unused struct rte_eth_dev *dev,
+ __rte_unused struct rte_eth_stats *igb_stats)
+{
+ return 0;
+}
+
+static int
+ice_dcf_stats_reset(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_promiscuous_enable(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_promiscuous_disable(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_allmulticast_enable(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_allmulticast_disable(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_filter_ctrl(struct rte_eth_dev *dev,
+ enum rte_filter_type filter_type,
+ __rte_unused enum rte_filter_op filter_op,
+ __rte_unused void *arg)
+{
+ int ret = 0;
+
+ if (!dev)
+ return -EINVAL;
+
+ switch (filter_type) {
+ default:
+ PMD_DRV_LOG(WARNING, "Filter type (%d) not supported",
+ filter_type);
+ ret = -EINVAL;
+ break;
+ }
+
+ return ret;
+}
+
+static void
+ice_dcf_dev_close(struct rte_eth_dev *dev)
+{
+ struct ice_dcf_adapter *adapter = dev->data->dev_private;
+
+ if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+ return;
+
+ dev->dev_ops = NULL;
+ dev->rx_pkt_burst = NULL;
+ dev->tx_pkt_burst = NULL;
+ dev->data->mac_addrs = NULL;
+
+ ice_dcf_uninit_hw(dev, &adapter->real_hw);
+}
+
+static void
+ice_dcf_queue_release(__rte_unused void *q)
+{
+}
+
+static int
+ice_dcf_link_update(__rte_unused struct rte_eth_dev *dev,
+ __rte_unused int wait_to_complete)
+{
+ return 0;
+}
+
+static int
+ice_dcf_rx_queue_setup(struct rte_eth_dev *dev,
+ uint16_t rx_queue_id,
+ __rte_unused uint16_t nb_rx_desc,
+ __rte_unused unsigned int socket_id,
+ __rte_unused const struct rte_eth_rxconf *rx_conf,
+ __rte_unused struct rte_mempool *mb_pool)
+{
+ struct ice_dcf_adapter *adapter = dev->data->dev_private;
+
+ dev->data->rx_queues[rx_queue_id] = &adapter->rxqs[rx_queue_id];
+
+ return 0;
+}
+
+static int
+ice_dcf_tx_queue_setup(struct rte_eth_dev *dev,
+ uint16_t tx_queue_id,
+ __rte_unused uint16_t nb_tx_desc,
+ __rte_unused unsigned int socket_id,
+ __rte_unused const struct rte_eth_txconf *tx_conf)
+{
+ struct ice_dcf_adapter *adapter = dev->data->dev_private;
+
+ dev->data->tx_queues[tx_queue_id] = &adapter->txqs[tx_queue_id];
+
+ return 0;
+}
+
+static const struct eth_dev_ops ice_dcf_eth_dev_ops = {
+ .dev_start = ice_dcf_dev_start,
+ .dev_stop = ice_dcf_dev_stop,
+ .dev_close = ice_dcf_dev_close,
+ .dev_configure = ice_dcf_dev_configure,
+ .dev_infos_get = ice_dcf_dev_info_get,
+ .rx_queue_setup = ice_dcf_rx_queue_setup,
+ .tx_queue_setup = ice_dcf_tx_queue_setup,
+ .rx_queue_release = ice_dcf_queue_release,
+ .tx_queue_release = ice_dcf_queue_release,
+ .link_update = ice_dcf_link_update,
+ .stats_get = ice_dcf_stats_get,
+ .stats_reset = ice_dcf_stats_reset,
+ .promiscuous_enable = ice_dcf_dev_promiscuous_enable,
+ .promiscuous_disable = ice_dcf_dev_promiscuous_disable,
+ .allmulticast_enable = ice_dcf_dev_allmulticast_enable,
+ .allmulticast_disable = ice_dcf_dev_allmulticast_disable,
+ .filter_ctrl = ice_dcf_dev_filter_ctrl,
+};
+
+static int
+ice_dcf_dev_init(struct rte_eth_dev *eth_dev)
+{
+ struct ice_dcf_adapter *adapter = eth_dev->data->dev_private;
+
+ eth_dev->dev_ops = &ice_dcf_eth_dev_ops;
+ eth_dev->rx_pkt_burst = ice_dcf_recv_pkts;
+ eth_dev->tx_pkt_burst = ice_dcf_xmit_pkts;
+
+ if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+ return 0;
+
+ eth_dev->data->dev_flags |= RTE_ETH_DEV_CLOSE_REMOVE;
+
+ if (ice_dcf_init_hw(eth_dev, &adapter->real_hw) != 0) {
+ PMD_INIT_LOG(ERR, "Failed to init DCF hardware");
+ return -1;
+ }
+
+ rte_eth_random_addr(adapter->mac_addr.addr_bytes);
+ eth_dev->data->mac_addrs = &adapter->mac_addr;
+
+ return 0;
+}
+
+static int
+ice_dcf_dev_uninit(struct rte_eth_dev *eth_dev)
+{
+ ice_dcf_dev_close(eth_dev);
+
+ return 0;
+}
+
+static int
+ice_dcf_cap_check_handler(__rte_unused const char *key,
+ const char *value, __rte_unused void *opaque)
+{
+ if (strcmp(value, "dcf"))
+ return -1;
+
+ return 0;
+}
+
+static int
+ice_dcf_cap_selected(struct rte_devargs *devargs)
+{
+ struct rte_kvargs *kvlist;
+ const char *key = "cap";
+ int ret = 0;
+
+ if (devargs == NULL)
+ return 0;
+
+ kvlist = rte_kvargs_parse(devargs->args, NULL);
+ if (kvlist == NULL)
+ return 0;
+
+ if (!rte_kvargs_count(kvlist, key))
+ goto exit;
+
+ /* dcf capability selected when there's a key-value pair: cap=dcf */
+ if (rte_kvargs_process(kvlist, key,
+ ice_dcf_cap_check_handler, NULL) < 0)
+ goto exit;
+
+ ret = 1;
+
+exit:
+ rte_kvargs_free(kvlist);
+ return ret;
+}
+
+static int eth_ice_dcf_pci_probe(__rte_unused struct rte_pci_driver *pci_drv,
+ struct rte_pci_device *pci_dev)
+{
+ if (!ice_dcf_cap_selected(pci_dev->device.devargs))
+ return 1;
+
+ return rte_eth_dev_pci_generic_probe(pci_dev,
+ sizeof(struct ice_dcf_adapter),
+ ice_dcf_dev_init);
+}
+
+static int eth_ice_dcf_pci_remove(struct rte_pci_device *pci_dev)
+{
+ return rte_eth_dev_pci_generic_remove(pci_dev, ice_dcf_dev_uninit);
+}
+
+static const struct rte_pci_id pci_id_ice_dcf_map[] = {
+ { RTE_PCI_DEVICE(IAVF_INTEL_VENDOR_ID, IAVF_DEV_ID_ADAPTIVE_VF) },
+ { .vendor_id = 0, /* sentinel */ },
+};
+
+static struct rte_pci_driver rte_ice_dcf_pmd = {
+ .id_table = pci_id_ice_dcf_map,
+ .drv_flags = RTE_PCI_DRV_NEED_MAPPING,
+ .probe = eth_ice_dcf_pci_probe,
+ .remove = eth_ice_dcf_pci_remove,
+};
+
+RTE_PMD_REGISTER_PCI(net_ice_dcf, rte_ice_dcf_pmd);
+RTE_PMD_REGISTER_PCI_TABLE(net_ice_dcf, pci_id_ice_dcf_map);
+RTE_PMD_REGISTER_KMOD_DEP(net_ice_dcf, "* igb_uio | vfio-pci");
+RTE_PMD_REGISTER_PARAM_STRING(net_ice_dcf, "cap=dcf");
diff --git a/drivers/net/ice/ice_dcf_ethdev.h b/drivers/net/ice/ice_dcf_ethdev.h
new file mode 100644
index 000000000..0c34a0095
--- /dev/null
+++ b/drivers/net/ice/ice_dcf_ethdev.h
@@ -0,0 +1,24 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#ifndef _ICE_DCF_ETHDEV_H_
+#define _ICE_DCF_ETHDEV_H_
+
+#include "ice_ethdev.h"
+#include "ice_dcf.h"
+
+#define ICE_DCF_MAX_RINGS 1
+
+struct ice_dcf_queue {
+ uint64_t dummy;
+};
+
+struct ice_dcf_adapter {
+ struct ice_dcf_hw real_hw;
+ struct rte_ether_addr mac_addr;
+ struct ice_dcf_queue rxqs[ICE_DCF_MAX_RINGS];
+ struct ice_dcf_queue txqs[ICE_DCF_MAX_RINGS];
+};
+
+#endif /* _ICE_DCF_ETHDEV_H_ */
diff --git a/drivers/net/ice/meson.build b/drivers/net/ice/meson.build
index f9e897bbc..0ba9668d1 100644
--- a/drivers/net/ice/meson.build
+++ b/drivers/net/ice/meson.build
@@ -15,8 +15,8 @@ sources = files(
'ice_hash.c'
)
-deps += ['hash']
-includes += include_directories('base')
+deps += ['hash', 'net', 'common_iavf']
+includes += include_directories('base', '../../common/iavf')
if arch_subdir == 'x86'
sources += files('ice_rxtx_vec_sse.c')
@@ -37,4 +37,7 @@ if arch_subdir == 'x86'
endif
endif
+sources += files('ice_dcf.c',
+ 'ice_dcf_ethdev.c')
+
install_headers('rte_pmd_ice.h')
diff --git a/mk/rte.app.mk b/mk/rte.app.mk
index d295ca0a5..f3798a09f 100644
--- a/mk/rte.app.mk
+++ b/mk/rte.app.mk
@@ -185,6 +185,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_I40E_PMD) += -lrte_pmd_i40e
_LDLIBS-$(CONFIG_RTE_LIBRTE_IAVF_PMD) += -lrte_pmd_iavf
_LDLIBS-$(CONFIG_RTE_LIBRTE_ICE_PMD) += -lrte_pmd_ice
IAVF-y := $(CONFIG_RTE_LIBRTE_IAVF_PMD)
+IAVF-y += $(CONFIG_RTE_LIBRTE_ICE_PMD)
ifeq ($(findstring y,$(IAVF-y)),y)
_LDLIBS-y += -lrte_common_iavf
endif
--
2.25.1
^ permalink raw reply [relevance 1%]
* Re: [dpdk-dev] [PATCH dpdk-dev v3] mempool: sort the rte_mempool_ops by name
2020-03-09 13:15 0% ` David Marchand
@ 2020-03-16 7:43 0% ` Tonghao Zhang
2020-03-16 7:55 0% ` Olivier Matz
0 siblings, 1 reply; 200+ results
From: Tonghao Zhang @ 2020-03-16 7:43 UTC (permalink / raw)
To: David Marchand
Cc: Olivier Matz, Andrew Rybchenko, Jerin Jacob, dpdk-dev, Gage Eads,
Artem V. Andreev, Jerin Jacob, Nithin Dabilpuram, Vamsi Attunuru,
Hemant Agrawal, Burakov, Anatoly, Bruce Richardson
On Mon, Mar 9, 2020 at 9:15 PM David Marchand <david.marchand@redhat.com> wrote:
>
> On Mon, Mar 9, 2020 at 9:56 AM Tonghao Zhang <xiangxia.m.yue@gmail.com> wrote:
> > On Mon, Mar 9, 2020 at 4:27 PM Olivier Matz <olivier.matz@6wind.com> wrote:
> > > The fact that the ops index changes during mempool driver lifetime is
> > > indeed frightening, especially knowning that this is a dynamic
> > > registration that could happen at any moment in the life of the
> > > application. Also, breaking the ABI is not desirable.
> > That solution is better.
> >
> > > Let me try to propose something else to solve your issue:
> > >
> > > 1/ At init, the primary process allocates a struct in shared memory
> > > (named memzone):
> > >
> > > struct rte_mempool_shared_ops {
> > > size_t num_mempool_ops;
> > > struct {
> > > char name[RTE_MEMPOOL_OPS_NAMESIZE];
> > > } mempool_ops[RTE_MEMPOOL_MAX_OPS_IDX];
> > > char *mempool_ops_name[RTE_MEMPOOL_MAX_OPS_IDX];
> > > rte_spinlock_t mempool;
> > > }
> > >
> > > 2/ When we register a mempool ops, we first get a name and id from the
> > > shared struct: with the lock held, lookup for the registered name and
> > > return its index, else get the last id and copy the name in the struct.
> > >
> > > 3/ Then do as before (in the per-process global table), except that we
> > > reuse the registered id.
> > >
> > > We can remove the num_ops field from rte_mempool_ops_table.
> > >
> > > Thoughts?
>
> It seems better, just adding Anatoly and Bruce who know more about multiprocess.
>
> Tonghao, could you add a unit test to exhibit the issue as part of this work?
>
> Thanks.
Hi Olivier
any progress？will apply this patch or wait your patch?
>
> --
> David Marchand
>
--
Thanks,
Tonghao
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH dpdk-dev v3] mempool: sort the rte_mempool_ops by name
2020-03-16 7:43 0% ` Tonghao Zhang
@ 2020-03-16 7:55 0% ` Olivier Matz
0 siblings, 0 replies; 200+ results
From: Olivier Matz @ 2020-03-16 7:55 UTC (permalink / raw)
To: Tonghao Zhang
Cc: David Marchand, Andrew Rybchenko, Jerin Jacob, dpdk-dev,
Gage Eads, Artem V. Andreev, Jerin Jacob, Nithin Dabilpuram,
Vamsi Attunuru, Hemant Agrawal, Burakov, Anatoly,
Bruce Richardson
Hi Tonghao,
On Mon, Mar 16, 2020 at 03:43:40PM +0800, Tonghao Zhang wrote:
> On Mon, Mar 9, 2020 at 9:15 PM David Marchand <david.marchand@redhat.com> wrote:
> >
> > On Mon, Mar 9, 2020 at 9:56 AM Tonghao Zhang <xiangxia.m.yue@gmail.com> wrote:
> > > On Mon, Mar 9, 2020 at 4:27 PM Olivier Matz <olivier.matz@6wind.com> wrote:
> > > > The fact that the ops index changes during mempool driver lifetime is
> > > > indeed frightening, especially knowning that this is a dynamic
> > > > registration that could happen at any moment in the life of the
> > > > application. Also, breaking the ABI is not desirable.
> > > That solution is better.
> > >
> > > > Let me try to propose something else to solve your issue:
> > > >
> > > > 1/ At init, the primary process allocates a struct in shared memory
> > > > (named memzone):
> > > >
> > > > struct rte_mempool_shared_ops {
> > > > size_t num_mempool_ops;
> > > > struct {
> > > > char name[RTE_MEMPOOL_OPS_NAMESIZE];
> > > > } mempool_ops[RTE_MEMPOOL_MAX_OPS_IDX];
> > > > char *mempool_ops_name[RTE_MEMPOOL_MAX_OPS_IDX];
> > > > rte_spinlock_t mempool;
> > > > }
> > > >
> > > > 2/ When we register a mempool ops, we first get a name and id from the
> > > > shared struct: with the lock held, lookup for the registered name and
> > > > return its index, else get the last id and copy the name in the struct.
> > > >
> > > > 3/ Then do as before (in the per-process global table), except that we
> > > > reuse the registered id.
> > > >
> > > > We can remove the num_ops field from rte_mempool_ops_table.
> > > >
> > > > Thoughts?
> >
> > It seems better, just adding Anatoly and Bruce who know more about multiprocess.
> >
> > Tonghao, could you add a unit test to exhibit the issue as part of this work?
> >
> > Thanks.
> Hi Olivier
> any progress？will apply this patch or wait your patch?
Sorry if it wasn't clear, but my proposition was just an idea, I have no
time yet to work on it. Feel free to implement it by yourself, and I'll
help to review the patches.
Thanks,
Olivier
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-02-13 14:51 4% ` Kusztal, ArkadiuszX
@ 2020-03-16 12:57 8% ` Trahe, Fiona
2020-03-16 13:09 4% ` Thomas Monjalon
0 siblings, 1 reply; 200+ results
From: Trahe, Fiona @ 2020-03-16 12:57 UTC (permalink / raw)
To: Kusztal, ArkadiuszX, Thomas Monjalon
Cc: David Marchand, nhorman, bluca, ktraynor, Ray Kinsella, dev,
Akhil Goyal, Yigit, Ferruh, Ananyev, Konstantin, dev,
Anoob Joseph, Richardson, Bruce, Mcnamara, John, dodji,
Andrew Rybchenko, aconole, Trahe, Fiona
Hi,
> -----Original Message-----
> From: Kusztal, ArkadiuszX <arkadiuszx.kusztal@intel.com>
> Sent: Thursday, February 13, 2020 2:51 PM
> To: Trahe, Fiona <fiona.trahe@intel.com>; Thomas Monjalon <thomas@monjalon.net>
> Cc: David Marchand <david.marchand@redhat.com>; nhorman@tuxdriver.com; bluca@debian.org;
> ktraynor@redhat.com; Ray Kinsella <mdr@ashroe.eu>; dev@dpdk.org; Akhil Goyal
> <akhil.goyal@nxp.com>; Yigit, Ferruh <ferruh.yigit@intel.com>; Ananyev, Konstantin
> <konstantin.ananyev@intel.com>; dev@dpdk.org; Anoob Joseph <anoobj@marvell.com>;
> Richardson, Bruce <bruce.richardson@intel.com>; Mcnamara, John <john.mcnamara@intel.com>;
> dodji@seketeli.net; Andrew Rybchenko <arybchenko@solarflare.com>; aconole@redhat.com
> Subject: RE: [dpdk-dev] [PATCH v2 4/4] add ABI checks
>
> Hi,
>
> Two comments from me,
>
> > > > The patch we're working on will provide two versions of
> > > > rte_cryptodev_info_get(), both call the same PMD function from the
> > dev_ops info_get fn ptr.
> > > > The default version operates s as normal, the 19.11 version searches
> > > > through the list returned by the PMD, looking for sym.aead.algo =
> > > > ChaChaPoly, it needs to strip it from
> > > the list.
> > > > As PMDs just pass a ptr to their capabilities list ( it isn't a
> > > > linked list, but an array with an end marker =
> > > > RTE_CRYPTODEV_END_OF_CAPABILITIES_LIST) if the API layer detects
> > > > Chacha it must allocate some space and store a local copy of the trimmed
> > list. This must be stored only once per device.
> [Arek] The problem with this solution is that we need to allocate memory.
> So the question is how to handle unlikely case of malloc error when we operate inside void function
> rte_cryptodev_info_get?
> And even if we would pass somehow error condition to the caller then what to do is another question.
[Fiona] Quick recap: To avoid breaking ABI, we must return a set of capabilities with/without ChaChaPoly
depending on the appl version. To resolve this, within the rte_cryptodev layer, we propose to
inspect the capabilities returned by PMD and strip ChaCha if it exists.
In that case memory for the new trimmed capabilities array has to be malloced by the lib.
All good, except how to handle a malloc fail is yet another API breakage as rte_cryptodev_get_info() returns void.
We propose to return an empty capability list, i.e. a list with only the END element (which can be done without malloc)
in this corner case of a corner case.
Anyone see any issue with this?
>
> > >
> > > I don't understand what you have to store.
> > > Can't you just set the algo to 0 if it is ChaCha?
> > [Fiona] it returns a pointer to data in the PMD domain, which the API couldn't
> > and shouldn't overwrite, e.g.
> > static const struct rte_cryptodev_capabilities qat_gen3_sym_capabilities[]
> Should we print user some information
> >
> > >
> > > > This versioning will apply to any PMD which wants to take advantage
> > > > of the new API between now and
> > > 20.11.
> > > >
> > > > Note, I expect the ABI checker tools will still complain of ABI
> > > > breakage as the LIST_END value will still
> > > change.
> > >
> > > Right, you need to update the ignore list for the tool.
> > >
> > > > We are also reviewing all other cryptodev APIs in case there is any other
> > API which needs versioning.
> > > >
> > > > Anyone see any problem with this approach?
> > >
> > > The other issue is with all other functions accepting this enum as input.
> > > We should continue returning an error if getting Chacha as input with
> > > 19.11 version of these functions.
> > > But I would tend to consider this small ABI breakage can be ignored as
> > > it is in the error path.
> > [Fiona] The QAT PMD tests for and handles this error. I expect other PMDs
> > do too.
> [Arek] - Yes, it is error path but on the other hand we explicitly specify what value we will return when
> calling
> rte_cryptodev_sym_session_init so caller may expect EINVAL when wrong algorithm value selected
> (usually it probably will be ENOTSUP).
> In this case when setting 3 (LIST_END) on 19.11 app and linking against 20.02 (assuming with Chacha)
> shared build, caller would get success on return and fully set chacha session,
> which will probably result in undefined behavior.
> So shouldn't this function be versioned as well then?
[Fiona] I would agree with Tomas to ignore this small ABI break, as it is already an error case
if a appl is passing in a bad value for the algorithm. Even if it does return SUCCESS, instead of ENOTSUP,
what behaviour could the application be expecting with a session using LIST_END as an algo?
>
> Regards,
> Arek
>
>
^ permalink raw reply [relevance 8%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-03-16 12:57 8% ` Trahe, Fiona
@ 2020-03-16 13:09 4% ` Thomas Monjalon
2020-03-17 13:27 4% ` Kusztal, ArkadiuszX
0 siblings, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-03-16 13:09 UTC (permalink / raw)
To: Trahe, Fiona
Cc: Kusztal, ArkadiuszX, David Marchand, nhorman, bluca, ktraynor,
Ray Kinsella, dev, Akhil Goyal, Yigit, Ferruh, Ananyev,
Konstantin, dev, Anoob Joseph, Richardson, Bruce, Mcnamara, John,
dodji, Andrew Rybchenko, aconole, Trahe, Fiona
16/03/2020 13:57, Trahe, Fiona:
> From: Kusztal, ArkadiuszX <arkadiuszx.kusztal@intel.com>
> > > > > The patch we're working on will provide two versions of
> > > > > rte_cryptodev_info_get(), both call the same PMD function from the
> > > dev_ops info_get fn ptr.
> > > > > The default version operates s as normal, the 19.11 version searches
> > > > > through the list returned by the PMD, looking for sym.aead.algo =
> > > > > ChaChaPoly, it needs to strip it from
> > > > the list.
> > > > > As PMDs just pass a ptr to their capabilities list ( it isn't a
> > > > > linked list, but an array with an end marker =
> > > > > RTE_CRYPTODEV_END_OF_CAPABILITIES_LIST) if the API layer detects
> > > > > Chacha it must allocate some space and store a local copy of the trimmed
> > > list. This must be stored only once per device.
> >
> > [Arek] The problem with this solution is that we need to allocate memory.
> > So the question is how to handle unlikely case of malloc error when we operate inside void function
> > rte_cryptodev_info_get?
> > And even if we would pass somehow error condition to the caller then what to do is another question.
>
> [Fiona] Quick recap: To avoid breaking ABI, we must return a set of capabilities with/without ChaChaPoly
> depending on the appl version. To resolve this, within the rte_cryptodev layer, we propose to
> inspect the capabilities returned by PMD and strip ChaCha if it exists.
> In that case memory for the new trimmed capabilities array has to be malloced by the lib.
What happens if the capability is removed from the original capabilities input?
> All good, except how to handle a malloc fail is yet another API breakage as rte_cryptodev_get_info() returns void.
> We propose to return an empty capability list, i.e. a list with only the END element (which can be done without malloc)
> in this corner case of a corner case.
> Anyone see any issue with this?
How can we use the feature if it is not advertised in capabilities?
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-03-16 13:09 4% ` Thomas Monjalon
@ 2020-03-17 13:27 4% ` Kusztal, ArkadiuszX
2020-03-17 15:10 4% ` Thomas Monjalon
0 siblings, 1 reply; 200+ results
From: Kusztal, ArkadiuszX @ 2020-03-17 13:27 UTC (permalink / raw)
To: Thomas Monjalon, Trahe, Fiona
Cc: David Marchand, nhorman, bluca, ktraynor, Ray Kinsella, dev,
Akhil Goyal, Yigit, Ferruh, Ananyev, Konstantin, dev,
Anoob Joseph, Richardson, Bruce, Mcnamara, John, dodji,
Andrew Rybchenko, aconole, Trahe, Fiona
Hi Thomas,
> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Monday, March 16, 2020 2:09 PM
> To: Trahe, Fiona <fiona.trahe@intel.com>
> Cc: Kusztal, ArkadiuszX <arkadiuszx.kusztal@intel.com>; David Marchand
> <david.marchand@redhat.com>; nhorman@tuxdriver.com;
> bluca@debian.org; ktraynor@redhat.com; Ray Kinsella <mdr@ashroe.eu>;
> dev@dpdk.org; Akhil Goyal <akhil.goyal@nxp.com>; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Ananyev, Konstantin
> <konstantin.ananyev@intel.com>; dev@dpdk.org; Anoob Joseph
> <anoobj@marvell.com>; Richardson, Bruce <bruce.richardson@intel.com>;
> Mcnamara, John <john.mcnamara@intel.com>; dodji@seketeli.net; Andrew
> Rybchenko <arybchenko@solarflare.com>; aconole@redhat.com; Trahe,
> Fiona <fiona.trahe@intel.com>
> Subject: Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
>
> 16/03/2020 13:57, Trahe, Fiona:
> > From: Kusztal, ArkadiuszX <arkadiuszx.kusztal@intel.com>
> > > > > > The patch we're working on will provide two versions of
> > > > > > rte_cryptodev_info_get(), both call the same PMD function from
> > > > > > the
> > > > dev_ops info_get fn ptr.
> > > > > > The default version operates s as normal, the 19.11 version
> > > > > > searches through the list returned by the PMD, looking for
> > > > > > sym.aead.algo = ChaChaPoly, it needs to strip it from
> > > > > the list.
> > > > > > As PMDs just pass a ptr to their capabilities list ( it isn't
> > > > > > a linked list, but an array with an end marker =
> > > > > > RTE_CRYPTODEV_END_OF_CAPABILITIES_LIST) if the API layer
> > > > > > detects Chacha it must allocate some space and store a local
> > > > > > copy of the trimmed
> > > > list. This must be stored only once per device.
> > >
> > > [Arek] The problem with this solution is that we need to allocate memory.
> > > So the question is how to handle unlikely case of malloc error when
> > > we operate inside void function rte_cryptodev_info_get?
> > > And even if we would pass somehow error condition to the caller then
> what to do is another question.
> >
> > [Fiona] Quick recap: To avoid breaking ABI, we must return a set of
> > capabilities with/without ChaChaPoly depending on the appl version. To
> > resolve this, within the rte_cryptodev layer, we propose to inspect the
> capabilities returned by PMD and strip ChaCha if it exists.
> > In that case memory for the new trimmed capabilities array has to be
> malloced by the lib.
>
> What happens if the capability is removed from the original capabilities
> input?
>
> > All good, except how to handle a malloc fail is yet another API breakage as
> rte_cryptodev_get_info() returns void.
> > We propose to return an empty capability list, i.e. a list with only
> > the END element (which can be done without malloc) in this corner case of
> a corner case.
> > Anyone see any issue with this?
>
> How can we use the feature if it is not advertised in capabilities?
What Fiona meant is that empty capability would indicate error condition in this case. That's why she asked if you ok with this API breakage.
>
>
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-03-17 13:27 4% ` Kusztal, ArkadiuszX
@ 2020-03-17 15:10 4% ` Thomas Monjalon
2020-03-17 19:10 4% ` Kusztal, ArkadiuszX
0 siblings, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-03-17 15:10 UTC (permalink / raw)
To: Trahe, Fiona, Kusztal, ArkadiuszX
Cc: David Marchand, nhorman, bluca, ktraynor, Ray Kinsella, dev,
Akhil Goyal, Yigit, Ferruh, Ananyev, Konstantin, dev,
Anoob Joseph, Richardson, Bruce, Mcnamara, John, dodji,
Andrew Rybchenko, aconole, Trahe, Fiona
17/03/2020 14:27, Kusztal, ArkadiuszX:
> Hi Thomas,
>
> > -----Original Message-----
> > From: Thomas Monjalon <thomas@monjalon.net>
> > Sent: Monday, March 16, 2020 2:09 PM
> > To: Trahe, Fiona <fiona.trahe@intel.com>
> > Cc: Kusztal, ArkadiuszX <arkadiuszx.kusztal@intel.com>; David Marchand
> > <david.marchand@redhat.com>; nhorman@tuxdriver.com;
> > bluca@debian.org; ktraynor@redhat.com; Ray Kinsella <mdr@ashroe.eu>;
> > dev@dpdk.org; Akhil Goyal <akhil.goyal@nxp.com>; Yigit, Ferruh
> > <ferruh.yigit@intel.com>; Ananyev, Konstantin
> > <konstantin.ananyev@intel.com>; dev@dpdk.org; Anoob Joseph
> > <anoobj@marvell.com>; Richardson, Bruce <bruce.richardson@intel.com>;
> > Mcnamara, John <john.mcnamara@intel.com>; dodji@seketeli.net; Andrew
> > Rybchenko <arybchenko@solarflare.com>; aconole@redhat.com; Trahe,
> > Fiona <fiona.trahe@intel.com>
> > Subject: Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
> >
> > 16/03/2020 13:57, Trahe, Fiona:
> > > From: Kusztal, ArkadiuszX <arkadiuszx.kusztal@intel.com>
> > > > > > > The patch we're working on will provide two versions of
> > > > > > > rte_cryptodev_info_get(), both call the same PMD function from
> > > > > > > the
> > > > > dev_ops info_get fn ptr.
> > > > > > > The default version operates s as normal, the 19.11 version
> > > > > > > searches through the list returned by the PMD, looking for
> > > > > > > sym.aead.algo = ChaChaPoly, it needs to strip it from
> > > > > > the list.
> > > > > > > As PMDs just pass a ptr to their capabilities list ( it isn't
> > > > > > > a linked list, but an array with an end marker =
> > > > > > > RTE_CRYPTODEV_END_OF_CAPABILITIES_LIST) if the API layer
> > > > > > > detects Chacha it must allocate some space and store a local
> > > > > > > copy of the trimmed
> > > > > list. This must be stored only once per device.
> > > >
> > > > [Arek] The problem with this solution is that we need to allocate memory.
> > > > So the question is how to handle unlikely case of malloc error when
> > > > we operate inside void function rte_cryptodev_info_get?
> > > > And even if we would pass somehow error condition to the caller then
> > what to do is another question.
> > >
> > > [Fiona] Quick recap: To avoid breaking ABI, we must return a set of
> > > capabilities with/without ChaChaPoly depending on the appl version. To
> > > resolve this, within the rte_cryptodev layer, we propose to inspect the
> > capabilities returned by PMD and strip ChaCha if it exists.
> > > In that case memory for the new trimmed capabilities array has to be
> > malloced by the lib.
> >
> > What happens if the capability is removed from the original capabilities
> > input?
> >
> > > All good, except how to handle a malloc fail is yet another API breakage as
> > rte_cryptodev_get_info() returns void.
> > > We propose to return an empty capability list, i.e. a list with only
> > > the END element (which can be done without malloc) in this corner case of
> > a corner case.
> > > Anyone see any issue with this?
> >
> > How can we use the feature if it is not advertised in capabilities?
> What Fiona meant is that empty capability would indicate error condition in this case. That's why she asked if you ok with this API breakage.
Sorry I'm lost.
Please could you show what would be the usage?
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-03-17 15:10 4% ` Thomas Monjalon
@ 2020-03-17 19:10 4% ` Kusztal, ArkadiuszX
0 siblings, 0 replies; 200+ results
From: Kusztal, ArkadiuszX @ 2020-03-17 19:10 UTC (permalink / raw)
To: Thomas Monjalon, Trahe, Fiona
Cc: David Marchand, nhorman, bluca, ktraynor, Ray Kinsella, dev,
Akhil Goyal, Yigit, Ferruh, Ananyev, Konstantin, dev,
Anoob Joseph, Richardson, Bruce, Mcnamara, John, dodji,
Andrew Rybchenko, aconole, Trahe, Fiona
> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Tuesday, March 17, 2020 4:10 PM
> To: Trahe, Fiona <fiona.trahe@intel.com>; Kusztal, ArkadiuszX
> <arkadiuszx.kusztal@intel.com>
> Cc: David Marchand <david.marchand@redhat.com>;
> nhorman@tuxdriver.com; bluca@debian.org; ktraynor@redhat.com; Ray
> Kinsella <mdr@ashroe.eu>; dev@dpdk.org; Akhil Goyal
> <akhil.goyal@nxp.com>; Yigit, Ferruh <ferruh.yigit@intel.com>; Ananyev,
> Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org; Anoob Joseph
> <anoobj@marvell.com>; Richardson, Bruce <bruce.richardson@intel.com>;
> Mcnamara, John <john.mcnamara@intel.com>; dodji@seketeli.net; Andrew
> Rybchenko <arybchenko@solarflare.com>; aconole@redhat.com; Trahe,
> Fiona <fiona.trahe@intel.com>
> Subject: Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
>
> 17/03/2020 14:27, Kusztal, ArkadiuszX:
> > Hi Thomas,
> >
> > > -----Original Message-----
> > > From: Thomas Monjalon <thomas@monjalon.net>
> > > Sent: Monday, March 16, 2020 2:09 PM
> > > To: Trahe, Fiona <fiona.trahe@intel.com>
> > > Cc: Kusztal, ArkadiuszX <arkadiuszx.kusztal@intel.com>; David
> > > Marchand <david.marchand@redhat.com>; nhorman@tuxdriver.com;
> > > bluca@debian.org; ktraynor@redhat.com; Ray Kinsella
> <mdr@ashroe.eu>;
> > > dev@dpdk.org; Akhil Goyal <akhil.goyal@nxp.com>; Yigit, Ferruh
> > > <ferruh.yigit@intel.com>; Ananyev, Konstantin
> > > <konstantin.ananyev@intel.com>; dev@dpdk.org; Anoob Joseph
> > > <anoobj@marvell.com>; Richardson, Bruce
> > > <bruce.richardson@intel.com>; Mcnamara, John
> > > <john.mcnamara@intel.com>; dodji@seketeli.net; Andrew Rybchenko
> > > <arybchenko@solarflare.com>; aconole@redhat.com; Trahe, Fiona
> > > <fiona.trahe@intel.com>
> > > Subject: Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
> > >
> > > 16/03/2020 13:57, Trahe, Fiona:
> > > > From: Kusztal, ArkadiuszX <arkadiuszx.kusztal@intel.com>
> > > > > > > > The patch we're working on will provide two versions of
> > > > > > > > rte_cryptodev_info_get(), both call the same PMD function
> > > > > > > > from the
> > > > > > dev_ops info_get fn ptr.
> > > > > > > > The default version operates s as normal, the 19.11
> > > > > > > > version searches through the list returned by the PMD,
> > > > > > > > looking for sym.aead.algo = ChaChaPoly, it needs to strip
> > > > > > > > it from
> > > > > > > the list.
> > > > > > > > As PMDs just pass a ptr to their capabilities list ( it
> > > > > > > > isn't a linked list, but an array with an end marker =
> > > > > > > > RTE_CRYPTODEV_END_OF_CAPABILITIES_LIST) if the API layer
> > > > > > > > detects Chacha it must allocate some space and store a
> > > > > > > > local copy of the trimmed
> > > > > > list. This must be stored only once per device.
> > > > >
> > > > > [Arek] The problem with this solution is that we need to allocate
> memory.
> > > > > So the question is how to handle unlikely case of malloc error
> > > > > when we operate inside void function rte_cryptodev_info_get?
> > > > > And even if we would pass somehow error condition to the caller
> > > > > then
> > > what to do is another question.
> > > >
> > > > [Fiona] Quick recap: To avoid breaking ABI, we must return a set
> > > > of capabilities with/without ChaChaPoly depending on the appl
> > > > version. To resolve this, within the rte_cryptodev layer, we
> > > > propose to inspect the
> > > capabilities returned by PMD and strip ChaCha if it exists.
> > > > In that case memory for the new trimmed capabilities array has to
> > > > be
> > > malloced by the lib.
> > >
> > > What happens if the capability is removed from the original
> > > capabilities input?
> > >
> > > > All good, except how to handle a malloc fail is yet another API
> > > > breakage as
> > > rte_cryptodev_get_info() returns void.
> > > > We propose to return an empty capability list, i.e. a list with
> > > > only the END element (which can be done without malloc) in this
> > > > corner case of
> > > a corner case.
> > > > Anyone see any issue with this?
> > >
> > > How can we use the feature if it is not advertised in capabilities?
> > What Fiona meant is that empty capability would indicate error condition in
> this case. That's why she asked if you ok with this API breakage.
>
> Sorry I'm lost.
> Please could you show what would be the usage?
>
So this case looks more or less like that:
There are two versions of `rte_cryptodev_info_get`
rte_cryptodev_info_get_v20 (versioned)
rte_cryptodev_info_get_v2005 (new default symbol)
Default version works normally as it will be called only by app build with 20.05 version.
When prior to 20.05 app calls `rte_cryptodev_info_get` version v20 is called. This function will remove Chacha Poly from capabilities, but to achieve this we need to get some memory to store `new` set of capabilities per device (without Chacha). So:
new_capability[dev_id] = malloc( (num_of_capabilies - 1 (chacha)) * sizeof(struct rte_cryptodev_capabilities))
The small problem is how to handle malloc error:
If (new_capability[dev_id] == NULL) {
/* What to do now as rte_cryptodev_info_get is void function, and API does not say anything about error condition */
/*So Fiona suggestion above is to inform user of an error by doing this: */
dev_info->capabilities = cryptodev_undefined_capabilities;
/* Where
static const struct rte_cryptodev_capabilities cryptodev_undefined_capabilities[] = {
RTE_CRYPTODEV_END_OF_CAPABILITIES_LIST()
}; */
}
Sizeof rte_cryptodev_capabilities is 38 bytes, padded to 40. So properly constructed capabilities can take at most 1920 bytes. No system should ever fail doing that though iam not an expert. Other option could probably be to preallocate this memory. This is how I understand that.
Another question is can something like that be done if API comments of `rte_cryptodev_info_get` function does not say anything about any potential error?
Regards,
Arek
^ permalink raw reply [relevance 4%]
* [dpdk-dev] [PATCH] cryptodev: version rte_cryptodev_info_get function
@ 2020-03-18 20:41 3% Arek Kusztal
0 siblings, 0 replies; 200+ results
From: Arek Kusztal @ 2020-03-18 20:41 UTC (permalink / raw)
To: dev; +Cc: akhil.goyal, fiona.trahe, thomas, Arek Kusztal
This patch adds versioned function rte_cryptodev_info_get.
Node 20.05 function works the same way it was working before.
Node 20.0 function strips capability added in 20.05 release
to prevent some issues with ABI policy. To do that new capability
array is allocated per device and returned to user instead of the
original array passed by PMD.
Signed-off-by: Arek Kusztal <arkadiuszx.kusztal@intel.com>
---
lib/librte_cryptodev/rte_cryptodev.c | 97 +++++++++++++++++++++++++-
lib/librte_cryptodev/rte_cryptodev.h | 12 +++-
lib/librte_cryptodev/rte_cryptodev_version.map | 7 ++
3 files changed, 113 insertions(+), 3 deletions(-)
diff --git a/lib/librte_cryptodev/rte_cryptodev.c b/lib/librte_cryptodev/rte_cryptodev.c
index 6d1d0e9..2d030ba 100644
--- a/lib/librte_cryptodev/rte_cryptodev.c
+++ b/lib/librte_cryptodev/rte_cryptodev.c
@@ -41,6 +41,9 @@
#include "rte_cryptodev.h"
#include "rte_cryptodev_pmd.h"
+#include <rte_compat.h>
+#include <rte_function_versioning.h>
+
static uint8_t nb_drivers;
static struct rte_cryptodev rte_crypto_devices[RTE_CRYPTO_MAX_DEVS];
@@ -56,6 +59,13 @@ static struct rte_cryptodev_global cryptodev_globals = {
/* spinlock for crypto device callbacks */
static rte_spinlock_t rte_cryptodev_cb_lock = RTE_SPINLOCK_INITIALIZER;
+static const struct rte_cryptodev_capabilities
+ cryptodev_undefined_capabilities[] = {
+ RTE_CRYPTODEV_END_OF_CAPABILITIES_LIST()
+};
+
+struct rte_cryptodev_capabilities *capability_copies[RTE_CRYPTO_MAX_DEVS];
+uint8_t is_capability_checked[RTE_CRYPTO_MAX_DEVS];
/**
* The user application callback description.
@@ -999,6 +1009,13 @@ rte_cryptodev_close(uint8_t dev_id)
RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_close, -ENOTSUP);
retval = (*dev->dev_ops->dev_close)(dev);
+
+ if (capability_copies[dev_id]) {
+ free(capability_copies[dev_id]);
+ capability_copies[dev_id] = NULL;
+ }
+ is_capability_checked[dev_id] = 0;
+
if (retval < 0)
return retval;
@@ -1111,11 +1128,12 @@ rte_cryptodev_stats_reset(uint8_t dev_id)
(*dev->dev_ops->stats_reset)(dev);
}
-
void
-rte_cryptodev_info_get(uint8_t dev_id, struct rte_cryptodev_info *dev_info)
+rte_cryptodev_info_get_v20(uint8_t dev_id, struct rte_cryptodev_info *dev_info)
{
struct rte_cryptodev *dev;
+ const struct rte_cryptodev_capabilities *capability;
+ uint8_t counter = 0;
if (!rte_cryptodev_pmd_is_valid_dev(dev_id)) {
CDEV_LOG_ERR("Invalid dev_id=%d", dev_id);
@@ -1129,10 +1147,85 @@ rte_cryptodev_info_get(uint8_t dev_id, struct rte_cryptodev_info *dev_info)
RTE_FUNC_PTR_OR_RET(*dev->dev_ops->dev_infos_get);
(*dev->dev_ops->dev_infos_get)(dev, dev_info);
+ if (capability_copies[dev_id] == NULL) {
+ if (!is_capability_checked[dev_id]) {
+ uint8_t found_invalid_capa = 0;
+
+ for (capability = dev_info->capabilities;
+ capability->op != RTE_CRYPTO_OP_TYPE_UNDEFINED;
+ ++capability, ++counter) {
+ if (capability->op == RTE_CRYPTO_OP_TYPE_SYMMETRIC &&
+ capability->sym.xform_type ==
+ RTE_CRYPTO_SYM_XFORM_AEAD
+ && capability->sym.aead.algo >=
+ RTE_CRYPTO_AEAD_CHACHA20_POLY1305) {
+ found_invalid_capa = 1;
+ counter--;
+ }
+ }
+ is_capability_checked[dev_id] = 1;
+ if (found_invalid_capa) {
+ capability_copies[dev_id] = malloc(counter *
+ sizeof(struct rte_cryptodev_capabilities));
+ if (capability_copies[dev_id] == NULL) {
+ /*
+ * error case - no memory to store the trimmed list,
+ * so have to return an empty list
+ */
+ dev_info->capabilities =
+ cryptodev_undefined_capabilities;
+ is_capability_checked[dev_id] = 0;
+ } else {
+ counter = 0;
+ for (capability = dev_info->capabilities;
+ capability->op !=
+ RTE_CRYPTO_OP_TYPE_UNDEFINED;
+ capability++) {
+ if (!(capability->op ==
+ RTE_CRYPTO_OP_TYPE_SYMMETRIC
+ && capability->sym.xform_type ==
+ RTE_CRYPTO_SYM_XFORM_AEAD
+ && capability->sym.aead.algo >=
+ RTE_CRYPTO_AEAD_CHACHA20_POLY1305)) {
+ capability_copies[dev_id][counter++] =
+ *capability;
+ }
+ }
+ dev_info->capabilities =
+ capability_copies[dev_id];
+ }
+ }
+ }
+ } else
+ dev_info->capabilities = capability_copies[dev_id];
+
dev_info->driver_name = dev->device->driver->name;
dev_info->device = dev->device;
}
+VERSION_SYMBOL(rte_cryptodev_info_get, _v20, 20.0);
+
+void
+rte_cryptodev_info_get_v2005(uint8_t dev_id, struct rte_cryptodev_info *dev_info)
+{
+ struct rte_cryptodev *dev;
+ if (!rte_cryptodev_pmd_is_valid_dev(dev_id)) {
+ CDEV_LOG_ERR("Invalid dev_id=%d", dev_id);
+ return;
+ }
+
+ dev = &rte_crypto_devices[dev_id];
+
+ memset(dev_info, 0, sizeof(struct rte_cryptodev_info));
+
+ RTE_FUNC_PTR_OR_RET(*dev->dev_ops->dev_infos_get);
+ (*dev->dev_ops->dev_infos_get)(dev, dev_info);
+
+ dev_info->driver_name = dev->device->driver->name;
+ dev_info->device = dev->device;
+}
+MAP_STATIC_SYMBOL(void rte_cryptodev_info_get(uint8_t dev_id,
+ struct rte_cryptodev_info *dev_info), rte_cryptodev_info_get_v2005);
int
rte_cryptodev_callback_register(uint8_t dev_id,
diff --git a/lib/librte_cryptodev/rte_cryptodev.h b/lib/librte_cryptodev/rte_cryptodev.h
index 437b8a9..06ce2f2 100644
--- a/lib/librte_cryptodev/rte_cryptodev.h
+++ b/lib/librte_cryptodev/rte_cryptodev.h
@@ -24,6 +24,9 @@ extern "C" {
#include <rte_common.h>
#include <rte_config.h>
+#include <rte_compat.h>
+#include <rte_function_versioning.h>
+
extern const char **rte_cyptodev_names;
/* Logging Macros */
@@ -758,7 +761,14 @@ rte_cryptodev_stats_reset(uint8_t dev_id);
* the last valid element has it's op field set to
* RTE_CRYPTO_OP_TYPE_UNDEFINED.
*/
-extern void
+void
+rte_cryptodev_info_get_v20(uint8_t dev_id, struct rte_cryptodev_info *dev_info);
+
+void
+rte_cryptodev_info_get_v2005(uint8_t dev_id, struct rte_cryptodev_info *dev_info);
+BIND_DEFAULT_SYMBOL(rte_cryptodev_info_get, _v2005, 20.05);
+
+void
rte_cryptodev_info_get(uint8_t dev_id, struct rte_cryptodev_info *dev_info);
diff --git a/lib/librte_cryptodev/rte_cryptodev_version.map b/lib/librte_cryptodev/rte_cryptodev_version.map
index 6e41b4b..d6c945a 100644
--- a/lib/librte_cryptodev/rte_cryptodev_version.map
+++ b/lib/librte_cryptodev/rte_cryptodev_version.map
@@ -58,6 +58,13 @@ DPDK_20.0 {
local: *;
};
+
+DPDK_20.05 {
+ global:
+ rte_cryptodev_info_get;
+} DPDK_20.0;
+
+
EXPERIMENTAL {
global:
--
2.1.0
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH v3 0/7] vfio/pci: SR-IOV support
2020-03-11 21:58 2% [dpdk-dev] [PATCH v3 0/7] vfio/pci: SR-IOV support Alex Williamson
@ 2020-03-19 6:32 0% ` Tian, Kevin
2020-03-19 13:11 0% ` Alex Williamson
0 siblings, 1 reply; 200+ results
From: Tian, Kevin @ 2020-03-19 6:32 UTC (permalink / raw)
To: Alex Williamson, kvm
Cc: linux-pci, linux-kernel, dev, mtosatti, thomas, bluca,
jerinjacobk, Richardson, Bruce, cohuck
> From: Alex Williamson <alex.williamson@redhat.com>
> Sent: Thursday, March 12, 2020 5:58 AM
>
> Only minor tweaks since v2, GET and SET on VFIO_DEVICE_FEATURE are
> enforced mutually exclusive except with the PROBE option as suggested
> by Connie, the modinfo text has been expanded for the opt-in to enable
> SR-IOV support in the vfio-pci driver per discussion with Kevin.
>
> I have not incorporated runtime warnings attempting to detect misuse
> of SR-IOV or imposed a session lifetime of a VF token, both of which
> were significant portions of the discussion of the v2 series. Both of
> these also seem to impose a usage model or make assumptions about VF
> resource usage or configuration requirements that don't seem necessary
> except for the sake of generating a warning or requiring an otherwise
> unnecessary and implicit token reinitialization. If there are new
> thoughts around these or other discussion points, please raise them.
>
> Series overview (same as provided with v1):
>
> The synopsis of this series is that we have an ongoing desire to drive
> PCIe SR-IOV PFs from userspace with VFIO. There's an immediate need
> for this with DPDK drivers and potentially interesting future use
> cases in virtualization. We've been reluctant to add this support
> previously due to the dependency and trust relationship between the
> VF device and PF driver. Minimally the PF driver can induce a denial
> of service to the VF, but depending on the specific implementation,
> the PF driver might also be responsible for moving data between VFs
> or have direct access to the state of the VF, including data or state
> otherwise private to the VF or VF driver.
>
> To help resolve these concerns, we introduce a VF token into the VFIO
> PCI ABI, which acts as a shared secret key between drivers. The
> userspace PF driver is required to set the VF token to a known value
> and userspace VF drivers are required to provide the token to access
> the VF device. If a PF driver is restarted with VF drivers in use, it
> must also provide the current token in order to prevent a rogue
> untrusted PF driver from replacing a known driver. The degree to
> which this new token is considered secret is left to the userspace
> drivers, the kernel intentionally provides no means to retrieve the
> current token.
>
> Note that the above token is only required for this new model where
> both the PF and VF devices are usable through vfio-pci. Existing
> models of VFIO drivers where the PF is used without SR-IOV enabled
> or the VF is bound to a userspace driver with an in-kernel, host PF
> driver are unaffected.
>
> The latter configuration above also highlights a new inverted scenario
> that is now possible, a userspace PF driver with in-kernel VF drivers.
> I believe this is a scenario that should be allowed, but should not be
> enabled by default. This series includes code to set a default
> driver_override for VFs sourced from a vfio-pci user owned PF, such
> that the VFs are also bound to vfio-pci. This model is compatible
> with tools like driverctl and allows the system administrator to
> decide if other bindings should be enabled. The VF token interface
> above exists only between vfio-pci PF and VF drivers, once a VF is
> bound to another driver, the administrator has effectively pronounced
> the device as trusted. The vfio-pci driver will note alternate
> binding in dmesg for logging and debugging purposes.
>
> Please review, comment, and test. The example QEMU implementation
> provided with the RFC is still current for this version. Thanks,
>
> Alex
The whole series looks good to me:
Reviewed-by: Kevin Tian <kevin.tian@intel.com>
and confirm one understanding here, since it is not discussed anywhere. For
VM live migration with assigned VF device, it is not necessary to migrate the
VF token itself and actually we don't allow userspace to retrieve it. Instead,
Qemu just follows whatever token requirement on the dest to open the new
VF: could be same or different token as/from src, or even no token if PF
driver runs in kernel on dest. I suppose either combination could work, correct?
Thanks
Kevin
>
> RFC:
> https://lore.kernel.org/lkml/158085337582.9445.17682266437583505502.stg
> it@gimli.home/
> v1:
> https://lore.kernel.org/lkml/158145472604.16827.15751375540102298130.st
> git@gimli.home/
> v2:
> https://lore.kernel.org/lkml/158213716959.17090.8399427017403507114.stg
> it@gimli.home/
>
> ---
>
> Alex Williamson (7):
> vfio: Include optional device match in vfio_device_ops callbacks
> vfio/pci: Implement match ops
> vfio/pci: Introduce VF token
> vfio: Introduce VFIO_DEVICE_FEATURE ioctl and first user
> vfio/pci: Add sriov_configure support
> vfio/pci: Remove dev_fmt definition
> vfio/pci: Cleanup .probe() exit paths
>
>
> drivers/vfio/pci/vfio_pci.c | 390
> +++++++++++++++++++++++++++++++++--
> drivers/vfio/pci/vfio_pci_private.h | 10 +
> drivers/vfio/vfio.c | 20 +-
> include/linux/vfio.h | 4
> include/uapi/linux/vfio.h | 37 +++
> 5 files changed, 433 insertions(+), 28 deletions(-)
^ permalink raw reply [relevance 0%]
* [dpdk-dev] [PATCH] doc: prefer https when pointing to dpdk.org
@ 2020-03-19 8:28 4% David Marchand
2020-03-19 10:46 0% ` [dpdk-dev] [dpdk-stable] " Kevin Traynor
0 siblings, 1 reply; 200+ results
From: David Marchand @ 2020-03-19 8:28 UTC (permalink / raw)
To: dev; +Cc: thomas, stable
for file in $(git grep -l http://.*dpdk.org doc/); do
sed -i -e 's#http://\(.*dpdk.org\)#https://\1#g' $file;
done
Cc: stable@dpdk.org
Signed-off-by: David Marchand <david.marchand@redhat.com>
---
devtools/checkpatches.sh | 8 ++++++++
doc/guides/contributing/documentation.rst | 12 ++++++------
doc/guides/contributing/patches.rst | 16 ++++++++--------
doc/guides/contributing/stable.rst | 8 ++++----
doc/guides/contributing/vulnerability.rst | 6 +++---
doc/guides/freebsd_gsg/install_from_ports.rst | 2 +-
doc/guides/howto/flow_bifurcation.rst | 2 +-
doc/guides/linux_gsg/nic_perf_intel_platform.rst | 2 +-
doc/guides/nics/enic.rst | 2 +-
doc/guides/prog_guide/cryptodev_lib.rst | 2 +-
doc/guides/rel_notes/deprecation.rst | 2 +-
11 files changed, 35 insertions(+), 27 deletions(-)
diff --git a/devtools/checkpatches.sh b/devtools/checkpatches.sh
index 1794468376..d3cf0e4c5d 100755
--- a/devtools/checkpatches.sh
+++ b/devtools/checkpatches.sh
@@ -70,6 +70,14 @@ check_forbidden_additions() { # <patch>
-f $(dirname $(readlink -f $0))/check-forbidden-tokens.awk \
"$1" || res=1
+ # links must prefer https over http
+ awk -v FOLDERS='doc' \
+ -v EXPRESSIONS='http://.*dpdk.org' \
+ -v RET_ON_FAIL=1 \
+ -v MESSAGE='Using non https link to dpdk.org' \
+ -f $(dirname $(readlink -f $0))/check-forbidden-tokens.awk \
+ "$1" || res=1
+
return $res
}
diff --git a/doc/guides/contributing/documentation.rst b/doc/guides/contributing/documentation.rst
index 550d8dec28..375ea64ba8 100644
--- a/doc/guides/contributing/documentation.rst
+++ b/doc/guides/contributing/documentation.rst
@@ -82,7 +82,7 @@ added to by the developer.
* **API documentation**
The API documentation explains how to use the public DPDK functions.
- The `API index page <http://doc.dpdk.org/api/>`_ shows the generated API documentation with related groups of functions.
+ The `API index page <https://doc.dpdk.org/api/>`_ shows the generated API documentation with related groups of functions.
The API documentation should be updated via Doxygen comments when new functions are added.
@@ -561,14 +561,14 @@ Hyperlinks
~~~~~~~~~~
* Links to external websites can be plain URLs.
- The following is rendered as http://dpdk.org::
+ The following is rendered as https://dpdk.org::
- http://dpdk.org
+ https://dpdk.org
* They can contain alternative text.
- The following is rendered as `Check out DPDK <http://dpdk.org>`_::
+ The following is rendered as `Check out DPDK <https://dpdk.org>`_::
- `Check out DPDK <http://dpdk.org>`_
+ `Check out DPDK <https://dpdk.org>`_
* An internal link can be generated by placing labels in the document with the format ``.. _label_name``.
@@ -666,7 +666,7 @@ The following are some guidelines for use of Doxygen in the DPDK API documentati
*/
In the API documentation the functions will be rendered as links, see the
- `online section of the rte_ethdev.h docs <http://doc.dpdk.org/api/rte__ethdev_8h.html>`_ that contains the above text.
+ `online section of the rte_ethdev.h docs <https://doc.dpdk.org/api/rte__ethdev_8h.html>`_ that contains the above text.
* The ``@see`` keyword can be used to create a *see also* link to another file or library.
This directive should be placed on one line at the bottom of the documentation section.
diff --git a/doc/guides/contributing/patches.rst b/doc/guides/contributing/patches.rst
index 59442824a1..2ec044891e 100644
--- a/doc/guides/contributing/patches.rst
+++ b/doc/guides/contributing/patches.rst
@@ -28,9 +28,9 @@ The DPDK development process has the following features:
* All sub-repositories are merged into main repository for ``-rc1`` and ``-rc2`` versions of the release.
* After the ``-rc2`` release all patches should target the main repository.
-The mailing list for DPDK development is `dev@dpdk.org <http://mails.dpdk.org/archives/dev/>`_.
-Contributors will need to `register for the mailing list <http://mails.dpdk.org/listinfo/dev>`_ in order to submit patches.
-It is also worth registering for the DPDK `Patchwork <http://patches.dpdk.org/project/dpdk/list/>`_
+The mailing list for DPDK development is `dev@dpdk.org <https://mails.dpdk.org/archives/dev/>`_.
+Contributors will need to `register for the mailing list <https://mails.dpdk.org/listinfo/dev>`_ in order to submit patches.
+It is also worth registering for the DPDK `Patchwork <https://patches.dpdk.org/project/dpdk/list/>`_
If you are using the GitHub service, you can link your repository to
the ``travis-ci.org`` build service. When you push patches to your GitHub
@@ -130,12 +130,12 @@ The source code can be cloned using either of the following:
main repository::
git clone git://dpdk.org/dpdk
- git clone http://dpdk.org/git/dpdk
+ git clone https://dpdk.org/git/dpdk
-sub-repositories (`list <http://git.dpdk.org/next>`_)::
+sub-repositories (`list <https://git.dpdk.org/next>`_)::
git clone git://dpdk.org/next/dpdk-next-*
- git clone http://dpdk.org/git/next/dpdk-next-*
+ git clone https://dpdk.org/git/next/dpdk-next-*
Make your Changes
-----------------
@@ -320,7 +320,7 @@ Patch for Stable Releases
~~~~~~~~~~~~~~~~~~~~~~~~~
All fix patches to the master branch that are candidates for backporting
-should also be CCed to the `stable@dpdk.org <http://mails.dpdk.org/listinfo/stable>`_
+should also be CCed to the `stable@dpdk.org <https://mails.dpdk.org/listinfo/stable>`_
mailing list.
In the commit message body the Cc: stable@dpdk.org should be inserted as follows::
@@ -563,7 +563,7 @@ If the patch is in relation to a previous email thread you can add it to the sam
git send-email --to dev@dpdk.org --in-reply-to <1234-foo@bar.com> 000*.patch
The Message ID can be found in the raw text of emails or at the top of each Patchwork patch,
-`for example <http://patches.dpdk.org/patch/7646/>`_.
+`for example <https://patches.dpdk.org/patch/7646/>`_.
Shallow threading (``--thread --no-chain-reply-to``) is preferred for a patch series.
Once submitted your patches will appear on the mailing list and in Patchwork.
diff --git a/doc/guides/contributing/stable.rst b/doc/guides/contributing/stable.rst
index 90b3d8e44f..2947f27c6a 100644
--- a/doc/guides/contributing/stable.rst
+++ b/doc/guides/contributing/stable.rst
@@ -51,7 +51,7 @@ agreement and a commitment from a maintainer. The current policy is that each
year's November (X.11) release will be maintained as an LTS for 2 years.
After the X.11 release, an LTS branch will be created for it at
-http://git.dpdk.org/dpdk-stable where bugfixes will be backported to.
+https://git.dpdk.org/dpdk-stable where bugfixes will be backported to.
A LTS release may align with the declaration of a new major ABI version,
please read the :doc:`abi_policy` for more information.
@@ -107,7 +107,7 @@ The Stable and LTS release are coordinated on the stable@dpdk.org mailing
list.
All fix patches to the master branch that are candidates for backporting
-should also be CCed to the `stable@dpdk.org <http://mails.dpdk.org/listinfo/stable>`_
+should also be CCed to the `stable@dpdk.org <https://mails.dpdk.org/listinfo/stable>`_
mailing list.
@@ -118,7 +118,7 @@ A Stable Release will be released by:
* Tagging the release with YY.MM.n (year, month, number).
* Uploading a tarball of the release to dpdk.org.
-* Sending an announcement to the `announce@dpdk.org <http://mails.dpdk.org/listinfo/announce>`_
+* Sending an announcement to the `announce@dpdk.org <https://mails.dpdk.org/listinfo/announce>`_
list.
-Stable releases are available on the `dpdk.org download page <http://core.dpdk.org/download/>`_.
+Stable releases are available on the `dpdk.org download page <https://core.dpdk.org/download/>`_.
diff --git a/doc/guides/contributing/vulnerability.rst b/doc/guides/contributing/vulnerability.rst
index 5484119d19..da00acd4f0 100644
--- a/doc/guides/contributing/vulnerability.rst
+++ b/doc/guides/contributing/vulnerability.rst
@@ -36,11 +36,11 @@ Report
Do not use Bugzilla (unsecured).
Instead, send GPG-encrypted emails
-to `security@dpdk.org <http://core.dpdk.org/security#contact>`_.
+to `security@dpdk.org <https://core.dpdk.org/security#contact>`_.
Anyone can post to this list.
In order to reduce the disclosure of a vulnerability in the early stages,
membership of this list is intentionally limited to a `small number of people
-<http://mails.dpdk.org/roster/security>`_.
+<https://mails.dpdk.org/roster/security>`_.
It is additionally encouraged to GPG-sign one-on-one conversations
as part of the security process.
@@ -188,7 +188,7 @@ Downstream stakeholders are expected not to deploy or disclose patches
until the embargo is passed, otherwise they will be removed from the list.
Downstream stakeholders (in `security-prerelease list
-<http://mails.dpdk.org/roster/security-prerelease>`_), are:
+<https://mails.dpdk.org/roster/security-prerelease>`_), are:
* Operating system vendors known to package DPDK
* Major DPDK users, considered trustworthy by the technical board, who
diff --git a/doc/guides/freebsd_gsg/install_from_ports.rst b/doc/guides/freebsd_gsg/install_from_ports.rst
index 36dc4a417b..d946f3f3b2 100644
--- a/doc/guides/freebsd_gsg/install_from_ports.rst
+++ b/doc/guides/freebsd_gsg/install_from_ports.rst
@@ -72,7 +72,7 @@ These examples can be compiled and run as described in :ref:`compiling_sample_ap
.. note::
To install a copy of the DPDK compiled using gcc, please download the
- official DPDK package from http://core.dpdk.org/download/ and install manually using
+ official DPDK package from https://core.dpdk.org/download/ and install manually using
the instructions given in the next chapter, :ref:`building_from_source`
An example application can therefore be copied to a user's home directory and
diff --git a/doc/guides/howto/flow_bifurcation.rst b/doc/guides/howto/flow_bifurcation.rst
index a36126472c..68ac913e7d 100644
--- a/doc/guides/howto/flow_bifurcation.rst
+++ b/doc/guides/howto/flow_bifurcation.rst
@@ -294,4 +294,4 @@ The typical procedure to achieve this is as follows:
'not involved', while ``00`` or no mask means 'involved'.
* For more details of the configuration, refer to the
- `cloud filter test plan <http://git.dpdk.org/tools/dts/tree/test_plans/cloud_filter_test_plan.rst>`_
+ `cloud filter test plan <https://git.dpdk.org/tools/dts/tree/test_plans/cloud_filter_test_plan.rst>`_
diff --git a/doc/guides/linux_gsg/nic_perf_intel_platform.rst b/doc/guides/linux_gsg/nic_perf_intel_platform.rst
index c554c2159c..1dabbce244 100644
--- a/doc/guides/linux_gsg/nic_perf_intel_platform.rst
+++ b/doc/guides/linux_gsg/nic_perf_intel_platform.rst
@@ -64,7 +64,7 @@ This aligns with the previous output which showed that each channel has one memo
Network Interface Card Requirements
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Use a `DPDK supported <http://core.dpdk.org/supported/>`_ high end NIC such as the Intel XL710 40GbE.
+Use a `DPDK supported <https://core.dpdk.org/supported/>`_ high end NIC such as the Intel XL710 40GbE.
Make sure each NIC has been flashed the latest version of NVM/firmware.
diff --git a/doc/guides/nics/enic.rst b/doc/guides/nics/enic.rst
index 65e536d422..24d2b5713a 100644
--- a/doc/guides/nics/enic.rst
+++ b/doc/guides/nics/enic.rst
@@ -14,7 +14,7 @@ How to obtain ENIC PMD integrated DPDK
--------------------------------------
ENIC PMD support is integrated into the DPDK suite. dpdk-<version>.tar.gz
-should be downloaded from http://core.dpdk.org/download/
+should be downloaded from https://core.dpdk.org/download/
Configuration information
diff --git a/doc/guides/prog_guide/cryptodev_lib.rst b/doc/guides/prog_guide/cryptodev_lib.rst
index b91f7c8b7f..c14f750fa8 100644
--- a/doc/guides/prog_guide/cryptodev_lib.rst
+++ b/doc/guides/prog_guide/cryptodev_lib.rst
@@ -1128,4 +1128,4 @@ Asymmetric Crypto Device API
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The cryptodev Library API is described in the
-`DPDK API Reference <http://doc.dpdk.org/api/>`_
+`DPDK API Reference <https://doc.dpdk.org/api/>`_
diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
index 1339f54f5f..151a86460d 100644
--- a/doc/guides/rel_notes/deprecation.rst
+++ b/doc/guides/rel_notes/deprecation.rst
@@ -43,7 +43,7 @@ Deprecation Notices
kernel module to the dpdk-kmods repository in the /linux/igb_uio/ directory
in 20.11.
Minutes of Technical Board Meeting of `2019-11-06
- <http://mails.dpdk.org/archives/dev/2019-November/151763.html>`_.
+ <https://mails.dpdk.org/archives/dev/2019-November/151763.html>`_.
* lib: will fix extending some enum/define breaking the ABI. There are multiple
samples in DPDK that enum/define terminated with a ``.*MAX.*`` value which is
--
2.23.0
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [dpdk-stable] [PATCH] doc: prefer https when pointing to dpdk.org
2020-03-19 8:28 4% [dpdk-dev] [PATCH] doc: prefer https when pointing to dpdk.org David Marchand
@ 2020-03-19 10:46 0% ` Kevin Traynor
0 siblings, 0 replies; 200+ results
From: Kevin Traynor @ 2020-03-19 10:46 UTC (permalink / raw)
To: David Marchand, dev; +Cc: thomas, stable
On 19/03/2020 08:28, David Marchand wrote:
> for file in $(git grep -l http://.*dpdk.org doc/); do
> sed -i -e 's#http://\(.*dpdk.org\)#https://\1#g' $file;
> done
>
> Cc: stable@dpdk.org
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Kevin Traynor <ktraynor@redhat.com>
> ---
> devtools/checkpatches.sh | 8 ++++++++
> doc/guides/contributing/documentation.rst | 12 ++++++------
> doc/guides/contributing/patches.rst | 16 ++++++++--------
> doc/guides/contributing/stable.rst | 8 ++++----
> doc/guides/contributing/vulnerability.rst | 6 +++---
> doc/guides/freebsd_gsg/install_from_ports.rst | 2 +-
> doc/guides/howto/flow_bifurcation.rst | 2 +-
> doc/guides/linux_gsg/nic_perf_intel_platform.rst | 2 +-
> doc/guides/nics/enic.rst | 2 +-
> doc/guides/prog_guide/cryptodev_lib.rst | 2 +-
> doc/guides/rel_notes/deprecation.rst | 2 +-
> 11 files changed, 35 insertions(+), 27 deletions(-)
>
> diff --git a/devtools/checkpatches.sh b/devtools/checkpatches.sh
> index 1794468376..d3cf0e4c5d 100755
> --- a/devtools/checkpatches.sh
> +++ b/devtools/checkpatches.sh
> @@ -70,6 +70,14 @@ check_forbidden_additions() { # <patch>
> -f $(dirname $(readlink -f $0))/check-forbidden-tokens.awk \
> "$1" || res=1
>
> + # links must prefer https over http
> + awk -v FOLDERS='doc' \
> + -v EXPRESSIONS='http://.*dpdk.org' \
> + -v RET_ON_FAIL=1 \
> + -v MESSAGE='Using non https link to dpdk.org' \
> + -f $(dirname $(readlink -f $0))/check-forbidden-tokens.awk \
> + "$1" || res=1
> +
> return $res
> }
>
> diff --git a/doc/guides/contributing/documentation.rst b/doc/guides/contributing/documentation.rst
> index 550d8dec28..375ea64ba8 100644
> --- a/doc/guides/contributing/documentation.rst
> +++ b/doc/guides/contributing/documentation.rst
> @@ -82,7 +82,7 @@ added to by the developer.
> * **API documentation**
>
> The API documentation explains how to use the public DPDK functions.
> - The `API index page <http://doc.dpdk.org/api/>`_ shows the generated API documentation with related groups of functions.
> + The `API index page <https://doc.dpdk.org/api/>`_ shows the generated API documentation with related groups of functions.
>
> The API documentation should be updated via Doxygen comments when new functions are added.
>
> @@ -561,14 +561,14 @@ Hyperlinks
> ~~~~~~~~~~
>
> * Links to external websites can be plain URLs.
> - The following is rendered as http://dpdk.org::
> + The following is rendered as https://dpdk.org::
>
> - http://dpdk.org
> + https://dpdk.org
>
> * They can contain alternative text.
> - The following is rendered as `Check out DPDK <http://dpdk.org>`_::
> + The following is rendered as `Check out DPDK <https://dpdk.org>`_::
>
> - `Check out DPDK <http://dpdk.org>`_
> + `Check out DPDK <https://dpdk.org>`_
>
> * An internal link can be generated by placing labels in the document with the format ``.. _label_name``.
>
> @@ -666,7 +666,7 @@ The following are some guidelines for use of Doxygen in the DPDK API documentati
> */
>
> In the API documentation the functions will be rendered as links, see the
> - `online section of the rte_ethdev.h docs <http://doc.dpdk.org/api/rte__ethdev_8h.html>`_ that contains the above text.
> + `online section of the rte_ethdev.h docs <https://doc.dpdk.org/api/rte__ethdev_8h.html>`_ that contains the above text.
>
> * The ``@see`` keyword can be used to create a *see also* link to another file or library.
> This directive should be placed on one line at the bottom of the documentation section.
> diff --git a/doc/guides/contributing/patches.rst b/doc/guides/contributing/patches.rst
> index 59442824a1..2ec044891e 100644
> --- a/doc/guides/contributing/patches.rst
> +++ b/doc/guides/contributing/patches.rst
> @@ -28,9 +28,9 @@ The DPDK development process has the following features:
> * All sub-repositories are merged into main repository for ``-rc1`` and ``-rc2`` versions of the release.
> * After the ``-rc2`` release all patches should target the main repository.
>
> -The mailing list for DPDK development is `dev@dpdk.org <http://mails.dpdk.org/archives/dev/>`_.
> -Contributors will need to `register for the mailing list <http://mails.dpdk.org/listinfo/dev>`_ in order to submit patches.
> -It is also worth registering for the DPDK `Patchwork <http://patches.dpdk.org/project/dpdk/list/>`_
> +The mailing list for DPDK development is `dev@dpdk.org <https://mails.dpdk.org/archives/dev/>`_.
> +Contributors will need to `register for the mailing list <https://mails.dpdk.org/listinfo/dev>`_ in order to submit patches.
> +It is also worth registering for the DPDK `Patchwork <https://patches.dpdk.org/project/dpdk/list/>`_
>
> If you are using the GitHub service, you can link your repository to
> the ``travis-ci.org`` build service. When you push patches to your GitHub
> @@ -130,12 +130,12 @@ The source code can be cloned using either of the following:
> main repository::
>
> git clone git://dpdk.org/dpdk
> - git clone http://dpdk.org/git/dpdk
> + git clone https://dpdk.org/git/dpdk
>
> -sub-repositories (`list <http://git.dpdk.org/next>`_)::
> +sub-repositories (`list <https://git.dpdk.org/next>`_)::
>
> git clone git://dpdk.org/next/dpdk-next-*
> - git clone http://dpdk.org/git/next/dpdk-next-*
> + git clone https://dpdk.org/git/next/dpdk-next-*
>
> Make your Changes
> -----------------
> @@ -320,7 +320,7 @@ Patch for Stable Releases
> ~~~~~~~~~~~~~~~~~~~~~~~~~
>
> All fix patches to the master branch that are candidates for backporting
> -should also be CCed to the `stable@dpdk.org <http://mails.dpdk.org/listinfo/stable>`_
> +should also be CCed to the `stable@dpdk.org <https://mails.dpdk.org/listinfo/stable>`_
> mailing list.
> In the commit message body the Cc: stable@dpdk.org should be inserted as follows::
>
> @@ -563,7 +563,7 @@ If the patch is in relation to a previous email thread you can add it to the sam
> git send-email --to dev@dpdk.org --in-reply-to <1234-foo@bar.com> 000*.patch
>
> The Message ID can be found in the raw text of emails or at the top of each Patchwork patch,
> -`for example <http://patches.dpdk.org/patch/7646/>`_.
> +`for example <https://patches.dpdk.org/patch/7646/>`_.
> Shallow threading (``--thread --no-chain-reply-to``) is preferred for a patch series.
>
> Once submitted your patches will appear on the mailing list and in Patchwork.
> diff --git a/doc/guides/contributing/stable.rst b/doc/guides/contributing/stable.rst
> index 90b3d8e44f..2947f27c6a 100644
> --- a/doc/guides/contributing/stable.rst
> +++ b/doc/guides/contributing/stable.rst
> @@ -51,7 +51,7 @@ agreement and a commitment from a maintainer. The current policy is that each
> year's November (X.11) release will be maintained as an LTS for 2 years.
>
> After the X.11 release, an LTS branch will be created for it at
> -http://git.dpdk.org/dpdk-stable where bugfixes will be backported to.
> +https://git.dpdk.org/dpdk-stable where bugfixes will be backported to.
>
> A LTS release may align with the declaration of a new major ABI version,
> please read the :doc:`abi_policy` for more information.
> @@ -107,7 +107,7 @@ The Stable and LTS release are coordinated on the stable@dpdk.org mailing
> list.
>
> All fix patches to the master branch that are candidates for backporting
> -should also be CCed to the `stable@dpdk.org <http://mails.dpdk.org/listinfo/stable>`_
> +should also be CCed to the `stable@dpdk.org <https://mails.dpdk.org/listinfo/stable>`_
> mailing list.
>
>
> @@ -118,7 +118,7 @@ A Stable Release will be released by:
>
> * Tagging the release with YY.MM.n (year, month, number).
> * Uploading a tarball of the release to dpdk.org.
> -* Sending an announcement to the `announce@dpdk.org <http://mails.dpdk.org/listinfo/announce>`_
> +* Sending an announcement to the `announce@dpdk.org <https://mails.dpdk.org/listinfo/announce>`_
> list.
>
> -Stable releases are available on the `dpdk.org download page <http://core.dpdk.org/download/>`_.
> +Stable releases are available on the `dpdk.org download page <https://core.dpdk.org/download/>`_.
> diff --git a/doc/guides/contributing/vulnerability.rst b/doc/guides/contributing/vulnerability.rst
> index 5484119d19..da00acd4f0 100644
> --- a/doc/guides/contributing/vulnerability.rst
> +++ b/doc/guides/contributing/vulnerability.rst
> @@ -36,11 +36,11 @@ Report
>
> Do not use Bugzilla (unsecured).
> Instead, send GPG-encrypted emails
> -to `security@dpdk.org <http://core.dpdk.org/security#contact>`_.
> +to `security@dpdk.org <https://core.dpdk.org/security#contact>`_.
> Anyone can post to this list.
> In order to reduce the disclosure of a vulnerability in the early stages,
> membership of this list is intentionally limited to a `small number of people
> -<http://mails.dpdk.org/roster/security>`_.
> +<https://mails.dpdk.org/roster/security>`_.
>
> It is additionally encouraged to GPG-sign one-on-one conversations
> as part of the security process.
> @@ -188,7 +188,7 @@ Downstream stakeholders are expected not to deploy or disclose patches
> until the embargo is passed, otherwise they will be removed from the list.
>
> Downstream stakeholders (in `security-prerelease list
> -<http://mails.dpdk.org/roster/security-prerelease>`_), are:
> +<https://mails.dpdk.org/roster/security-prerelease>`_), are:
>
> * Operating system vendors known to package DPDK
> * Major DPDK users, considered trustworthy by the technical board, who
> diff --git a/doc/guides/freebsd_gsg/install_from_ports.rst b/doc/guides/freebsd_gsg/install_from_ports.rst
> index 36dc4a417b..d946f3f3b2 100644
> --- a/doc/guides/freebsd_gsg/install_from_ports.rst
> +++ b/doc/guides/freebsd_gsg/install_from_ports.rst
> @@ -72,7 +72,7 @@ These examples can be compiled and run as described in :ref:`compiling_sample_ap
> .. note::
>
> To install a copy of the DPDK compiled using gcc, please download the
> - official DPDK package from http://core.dpdk.org/download/ and install manually using
> + official DPDK package from https://core.dpdk.org/download/ and install manually using
> the instructions given in the next chapter, :ref:`building_from_source`
>
> An example application can therefore be copied to a user's home directory and
> diff --git a/doc/guides/howto/flow_bifurcation.rst b/doc/guides/howto/flow_bifurcation.rst
> index a36126472c..68ac913e7d 100644
> --- a/doc/guides/howto/flow_bifurcation.rst
> +++ b/doc/guides/howto/flow_bifurcation.rst
> @@ -294,4 +294,4 @@ The typical procedure to achieve this is as follows:
> 'not involved', while ``00`` or no mask means 'involved'.
>
> * For more details of the configuration, refer to the
> - `cloud filter test plan <http://git.dpdk.org/tools/dts/tree/test_plans/cloud_filter_test_plan.rst>`_
> + `cloud filter test plan <https://git.dpdk.org/tools/dts/tree/test_plans/cloud_filter_test_plan.rst>`_
> diff --git a/doc/guides/linux_gsg/nic_perf_intel_platform.rst b/doc/guides/linux_gsg/nic_perf_intel_platform.rst
> index c554c2159c..1dabbce244 100644
> --- a/doc/guides/linux_gsg/nic_perf_intel_platform.rst
> +++ b/doc/guides/linux_gsg/nic_perf_intel_platform.rst
> @@ -64,7 +64,7 @@ This aligns with the previous output which showed that each channel has one memo
> Network Interface Card Requirements
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> -Use a `DPDK supported <http://core.dpdk.org/supported/>`_ high end NIC such as the Intel XL710 40GbE.
> +Use a `DPDK supported <https://core.dpdk.org/supported/>`_ high end NIC such as the Intel XL710 40GbE.
>
> Make sure each NIC has been flashed the latest version of NVM/firmware.
>
> diff --git a/doc/guides/nics/enic.rst b/doc/guides/nics/enic.rst
> index 65e536d422..24d2b5713a 100644
> --- a/doc/guides/nics/enic.rst
> +++ b/doc/guides/nics/enic.rst
> @@ -14,7 +14,7 @@ How to obtain ENIC PMD integrated DPDK
> --------------------------------------
>
> ENIC PMD support is integrated into the DPDK suite. dpdk-<version>.tar.gz
> -should be downloaded from http://core.dpdk.org/download/
> +should be downloaded from https://core.dpdk.org/download/
>
>
> Configuration information
> diff --git a/doc/guides/prog_guide/cryptodev_lib.rst b/doc/guides/prog_guide/cryptodev_lib.rst
> index b91f7c8b7f..c14f750fa8 100644
> --- a/doc/guides/prog_guide/cryptodev_lib.rst
> +++ b/doc/guides/prog_guide/cryptodev_lib.rst
> @@ -1128,4 +1128,4 @@ Asymmetric Crypto Device API
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> The cryptodev Library API is described in the
> -`DPDK API Reference <http://doc.dpdk.org/api/>`_
> +`DPDK API Reference <https://doc.dpdk.org/api/>`_
> diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
> index 1339f54f5f..151a86460d 100644
> --- a/doc/guides/rel_notes/deprecation.rst
> +++ b/doc/guides/rel_notes/deprecation.rst
> @@ -43,7 +43,7 @@ Deprecation Notices
> kernel module to the dpdk-kmods repository in the /linux/igb_uio/ directory
> in 20.11.
> Minutes of Technical Board Meeting of `2019-11-06
> - <http://mails.dpdk.org/archives/dev/2019-November/151763.html>`_.
> + <https://mails.dpdk.org/archives/dev/2019-November/151763.html>`_.
>
> * lib: will fix extending some enum/define breaking the ABI. There are multiple
> samples in DPDK that enum/define terminated with a ``.*MAX.*`` value which is
>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v3 0/7] vfio/pci: SR-IOV support
2020-03-19 6:32 0% ` Tian, Kevin
@ 2020-03-19 13:11 0% ` Alex Williamson
0 siblings, 0 replies; 200+ results
From: Alex Williamson @ 2020-03-19 13:11 UTC (permalink / raw)
To: Tian, Kevin
Cc: kvm, linux-pci, linux-kernel, dev, mtosatti, thomas, bluca,
jerinjacobk, Richardson, Bruce, cohuck
On Thu, 19 Mar 2020 06:32:25 +0000
"Tian, Kevin" <kevin.tian@intel.com> wrote:
> > From: Alex Williamson <alex.williamson@redhat.com>
> > Sent: Thursday, March 12, 2020 5:58 AM
> >
> > Only minor tweaks since v2, GET and SET on VFIO_DEVICE_FEATURE are
> > enforced mutually exclusive except with the PROBE option as suggested
> > by Connie, the modinfo text has been expanded for the opt-in to enable
> > SR-IOV support in the vfio-pci driver per discussion with Kevin.
> >
> > I have not incorporated runtime warnings attempting to detect misuse
> > of SR-IOV or imposed a session lifetime of a VF token, both of which
> > were significant portions of the discussion of the v2 series. Both of
> > these also seem to impose a usage model or make assumptions about VF
> > resource usage or configuration requirements that don't seem necessary
> > except for the sake of generating a warning or requiring an otherwise
> > unnecessary and implicit token reinitialization. If there are new
> > thoughts around these or other discussion points, please raise them.
> >
> > Series overview (same as provided with v1):
> >
> > The synopsis of this series is that we have an ongoing desire to drive
> > PCIe SR-IOV PFs from userspace with VFIO. There's an immediate need
> > for this with DPDK drivers and potentially interesting future use
> > cases in virtualization. We've been reluctant to add this support
> > previously due to the dependency and trust relationship between the
> > VF device and PF driver. Minimally the PF driver can induce a denial
> > of service to the VF, but depending on the specific implementation,
> > the PF driver might also be responsible for moving data between VFs
> > or have direct access to the state of the VF, including data or state
> > otherwise private to the VF or VF driver.
> >
> > To help resolve these concerns, we introduce a VF token into the VFIO
> > PCI ABI, which acts as a shared secret key between drivers. The
> > userspace PF driver is required to set the VF token to a known value
> > and userspace VF drivers are required to provide the token to access
> > the VF device. If a PF driver is restarted with VF drivers in use, it
> > must also provide the current token in order to prevent a rogue
> > untrusted PF driver from replacing a known driver. The degree to
> > which this new token is considered secret is left to the userspace
> > drivers, the kernel intentionally provides no means to retrieve the
> > current token.
> >
> > Note that the above token is only required for this new model where
> > both the PF and VF devices are usable through vfio-pci. Existing
> > models of VFIO drivers where the PF is used without SR-IOV enabled
> > or the VF is bound to a userspace driver with an in-kernel, host PF
> > driver are unaffected.
> >
> > The latter configuration above also highlights a new inverted scenario
> > that is now possible, a userspace PF driver with in-kernel VF drivers.
> > I believe this is a scenario that should be allowed, but should not be
> > enabled by default. This series includes code to set a default
> > driver_override for VFs sourced from a vfio-pci user owned PF, such
> > that the VFs are also bound to vfio-pci. This model is compatible
> > with tools like driverctl and allows the system administrator to
> > decide if other bindings should be enabled. The VF token interface
> > above exists only between vfio-pci PF and VF drivers, once a VF is
> > bound to another driver, the administrator has effectively pronounced
> > the device as trusted. The vfio-pci driver will note alternate
> > binding in dmesg for logging and debugging purposes.
> >
> > Please review, comment, and test. The example QEMU implementation
> > provided with the RFC is still current for this version. Thanks,
> >
> > Alex
>
> The whole series looks good to me:
> Reviewed-by: Kevin Tian <kevin.tian@intel.com>
Thanks!
> and confirm one understanding here, since it is not discussed anywhere. For
> VM live migration with assigned VF device, it is not necessary to migrate the
> VF token itself and actually we don't allow userspace to retrieve it. Instead,
> Qemu just follows whatever token requirement on the dest to open the new
> VF: could be same or different token as/from src, or even no token if PF
> driver runs in kernel on dest. I suppose either combination could work, correct?
That's correct. Thanks,
Alex
> > RFC:
> > https://lore.kernel.org/lkml/158085337582.9445.17682266437583505502.stg
> > it@gimli.home/
> > v1:
> > https://lore.kernel.org/lkml/158145472604.16827.15751375540102298130.st
> > git@gimli.home/
> > v2:
> > https://lore.kernel.org/lkml/158213716959.17090.8399427017403507114.stg
> > it@gimli.home/
> >
> > ---
> >
> > Alex Williamson (7):
> > vfio: Include optional device match in vfio_device_ops callbacks
> > vfio/pci: Implement match ops
> > vfio/pci: Introduce VF token
> > vfio: Introduce VFIO_DEVICE_FEATURE ioctl and first user
> > vfio/pci: Add sriov_configure support
> > vfio/pci: Remove dev_fmt definition
> > vfio/pci: Cleanup .probe() exit paths
> >
> >
> > drivers/vfio/pci/vfio_pci.c | 390
> > +++++++++++++++++++++++++++++++++--
> > drivers/vfio/pci/vfio_pci_private.h | 10 +
> > drivers/vfio/vfio.c | 20 +-
> > include/linux/vfio.h | 4
> > include/uapi/linux/vfio.h | 37 +++
> > 5 files changed, 433 insertions(+), 28 deletions(-)
>
^ permalink raw reply [relevance 0%]
* [dpdk-dev] [PATCH] devtools: fix check symbol change script
@ 2020-03-19 14:44 3% Nithin Dabilpuram
2020-03-19 14:56 0% ` David Marchand
` (2 more replies)
0 siblings, 3 replies; 200+ results
From: Nithin Dabilpuram @ 2020-03-19 14:44 UTC (permalink / raw)
To: thomas, david.marchand, Neil Horman
Cc: dev, jerinj, Nithin Dabilpuram, stable
Fix check symbol change script to detect new diff file when
it is in between "--- /dev/null" to "b/lib/...".
Current awk line expects line to start with "a/..."
which is not always true for all diffs.
As a result if in_map was '1' earlier, it will not be changed
to '0' and we get check patch errors which are not true as the non
version.map files get interpreted as version map file.
Fixes: 4bec48184e33 ("devtools: add checks for ABI symbol addition")
Cc: nhorman@tuxdriver.com
Cc: stable@dpdk.org
Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
---
Note: We have two examples where checkpatch errors are because of this
because the version.map file change comes earlier in the diff. Because of
this bug, any new file change that comes after version.map file diff
as "/dev/null" to "b/.." gets misdetected as version.map file.
* http://patches.dpdk.org/patch/66878/
* https://patchwork.dpdk.org/patch/66900/
devtools/check-symbol-change.sh | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/devtools/check-symbol-change.sh b/devtools/check-symbol-change.sh
index c5434f3..19ce82f 100755
--- a/devtools/check-symbol-change.sh
+++ b/devtools/check-symbol-change.sh
@@ -17,13 +17,13 @@ build_map_changes()
# map files are altered, and all section/symbol names
# appearing between a triggering of this rule and the
# next trigger of this rule are associated with this file
- /[-+] a\/.*\.map/ {map=$2; in_map=1}
+ /[-+] [ab]\/.*\.map/ {map=$2; in_map=1}
# Same pattern as above, only it matches on anything that
# does not end in 'map', indicating we have left the map chunk.
# When we hit this, turn off the in_map variable, which
# supresses the subordonate rules below
- /[-+] a\/.*\.[^map]/ {in_map=0}
+ /[-+] [ab]\/.*\.[^map]/ {in_map=0}
# Triggering this rule, which starts a line and ends it
# with a { identifies a versioned section. The section name is
--
2.8.4
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH] devtools: fix check symbol change script
2020-03-19 14:44 3% [dpdk-dev] [PATCH] devtools: fix check symbol change script Nithin Dabilpuram
@ 2020-03-19 14:56 0% ` David Marchand
2020-03-19 15:40 0% ` Neil Horman
2020-03-22 14:37 0% ` Jerin Jacob
2020-03-23 11:56 3% ` [dpdk-dev] [PATCH v2] " Nithin Dabilpuram
2 siblings, 1 reply; 200+ results
From: David Marchand @ 2020-03-19 14:56 UTC (permalink / raw)
To: Neil Horman
Cc: Thomas Monjalon, dev, Jerin Jacob Kollanukkaran,
Nithin Dabilpuram, dpdk stable, bingz
On Thu, Mar 19, 2020 at 3:44 PM Nithin Dabilpuram
<ndabilpuram@marvell.com> wrote:
>
> Fix check symbol change script to detect new diff file when
> it is in between "--- /dev/null" to "b/lib/...".
> Current awk line expects line to start with "a/..."
> which is not always true for all diffs.
> As a result if in_map was '1' earlier, it will not be changed
> to '0' and we get check patch errors which are not true as the non
> version.map files get interpreted as version map file.
>
> Fixes: 4bec48184e33 ("devtools: add checks for ABI symbol addition")
> Cc: nhorman@tuxdriver.com
> Cc: stable@dpdk.org
There was a previous attempt at fixing this that did not get a review.
http://patchwork.dpdk.org/patch/56303/
I prefer the last submitted patch as it is simpler, but maybe I missed
something in Bing patch.
Neil, wdyt?
--
David Marchand
>
> Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
> ---
> Note: We have two examples where checkpatch errors are because of this
> because the version.map file change comes earlier in the diff. Because of
> this bug, any new file change that comes after version.map file diff
> as "/dev/null" to "b/.." gets misdetected as version.map file.
> * http://patches.dpdk.org/patch/66878/
> * https://patchwork.dpdk.org/patch/66900/
> devtools/check-symbol-change.sh | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/devtools/check-symbol-change.sh b/devtools/check-symbol-change.sh
> index c5434f3..19ce82f 100755
> --- a/devtools/check-symbol-change.sh
> +++ b/devtools/check-symbol-change.sh
> @@ -17,13 +17,13 @@ build_map_changes()
> # map files are altered, and all section/symbol names
> # appearing between a triggering of this rule and the
> # next trigger of this rule are associated with this file
> - /[-+] a\/.*\.map/ {map=$2; in_map=1}
> + /[-+] [ab]\/.*\.map/ {map=$2; in_map=1}
>
> # Same pattern as above, only it matches on anything that
> # does not end in 'map', indicating we have left the map chunk.
> # When we hit this, turn off the in_map variable, which
> # supresses the subordonate rules below
> - /[-+] a\/.*\.[^map]/ {in_map=0}
> + /[-+] [ab]\/.*\.[^map]/ {in_map=0}
>
> # Triggering this rule, which starts a line and ends it
> # with a { identifies a versioned section. The section name is
> --
> 2.8.4
>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] devtools: fix check symbol change script
2020-03-19 14:56 0% ` David Marchand
@ 2020-03-19 15:40 0% ` Neil Horman
2020-03-19 15:45 0% ` [dpdk-dev] [EXT] " Nithin Dabilpuram
2020-03-19 15:49 0% ` [dpdk-dev] " Bing Zhao
0 siblings, 2 replies; 200+ results
From: Neil Horman @ 2020-03-19 15:40 UTC (permalink / raw)
To: David Marchand
Cc: Thomas Monjalon, dev, Jerin Jacob Kollanukkaran,
Nithin Dabilpuram, dpdk stable, bingz
On Thu, Mar 19, 2020 at 03:56:03PM +0100, David Marchand wrote:
> On Thu, Mar 19, 2020 at 3:44 PM Nithin Dabilpuram
> <ndabilpuram@marvell.com> wrote:
> >
> > Fix check symbol change script to detect new diff file when
> > it is in between "--- /dev/null" to "b/lib/...".
> > Current awk line expects line to start with "a/..."
> > which is not always true for all diffs.
> > As a result if in_map was '1' earlier, it will not be changed
> > to '0' and we get check patch errors which are not true as the non
> > version.map files get interpreted as version map file.
> >
> > Fixes: 4bec48184e33 ("devtools: add checks for ABI symbol addition")
> > Cc: nhorman@tuxdriver.com
> > Cc: stable@dpdk.org
>
> There was a previous attempt at fixing this that did not get a review.
> http://patchwork.dpdk.org/patch/56303/
>
> I prefer the last submitted patch as it is simpler, but maybe I missed
> something in Bing patch.
>
> Neil, wdyt?
>
I'm not sure why I didn't review the previous patch you refrenced, apologies for
that.
That said, I'm not sure how this patch detects /dev/null patterns. It looks like
you still expect all lines to start with [+-] [ab]...
Neil
>
> --
> David Marchand
>
>
> >
> > Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
> > ---
> > Note: We have two examples where checkpatch errors are because of this
> > because the version.map file change comes earlier in the diff. Because of
> > this bug, any new file change that comes after version.map file diff
> > as "/dev/null" to "b/.." gets misdetected as version.map file.
> > * http://patches.dpdk.org/patch/66878/
> > * https://patchwork.dpdk.org/patch/66900/
> > devtools/check-symbol-change.sh | 4 ++--
> > 1 file changed, 2 insertions(+), 2 deletions(-)
> >
> > diff --git a/devtools/check-symbol-change.sh b/devtools/check-symbol-change.sh
> > index c5434f3..19ce82f 100755
> > --- a/devtools/check-symbol-change.sh
> > +++ b/devtools/check-symbol-change.sh
> > @@ -17,13 +17,13 @@ build_map_changes()
> > # map files are altered, and all section/symbol names
> > # appearing between a triggering of this rule and the
> > # next trigger of this rule are associated with this file
> > - /[-+] a\/.*\.map/ {map=$2; in_map=1}
> > + /[-+] [ab]\/.*\.map/ {map=$2; in_map=1}
> >
> > # Same pattern as above, only it matches on anything that
> > # does not end in 'map', indicating we have left the map chunk.
> > # When we hit this, turn off the in_map variable, which
> > # supresses the subordonate rules below
> > - /[-+] a\/.*\.[^map]/ {in_map=0}
> > + /[-+] [ab]\/.*\.[^map]/ {in_map=0}
> >
> > # Triggering this rule, which starts a line and ends it
> > # with a { identifies a versioned section. The section name is
> > --
> > 2.8.4
> >
>
>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [EXT] Re: [PATCH] devtools: fix check symbol change script
2020-03-19 15:40 0% ` Neil Horman
@ 2020-03-19 15:45 0% ` Nithin Dabilpuram
2020-03-19 18:59 0% ` Neil Horman
2020-03-19 15:49 0% ` [dpdk-dev] " Bing Zhao
1 sibling, 1 reply; 200+ results
From: Nithin Dabilpuram @ 2020-03-19 15:45 UTC (permalink / raw)
To: Neil Horman
Cc: David Marchand, Thomas Monjalon, dev, Jerin Jacob Kollanukkaran,
dpdk stable, bingz
On Thu, Mar 19, 2020 at 11:40:39AM -0400, Neil Horman wrote:
> External Email
>
> ----------------------------------------------------------------------
> On Thu, Mar 19, 2020 at 03:56:03PM +0100, David Marchand wrote:
> > On Thu, Mar 19, 2020 at 3:44 PM Nithin Dabilpuram
> > <ndabilpuram@marvell.com> wrote:
> > >
> > > Fix check symbol change script to detect new diff file when
> > > it is in between "--- /dev/null" to "b/lib/...".
> > > Current awk line expects line to start with "a/..."
> > > which is not always true for all diffs.
> > > As a result if in_map was '1' earlier, it will not be changed
> > > to '0' and we get check patch errors which are not true as the non
> > > version.map files get interpreted as version map file.
> > >
> > > Fixes: 4bec48184e33 ("devtools: add checks for ABI symbol addition")
> > > Cc: nhorman@tuxdriver.com
> > > Cc: stable@dpdk.org
> >
> > There was a previous attempt at fixing this that did not get a review.
> > https://urldefense.proofpoint.com/v2/url?u=http-3A__patchwork.dpdk.org_patch_56303_&d=DwIBAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=FZ_tPCbgFOh18zwRPO9H0yDx8VW38vuapifdDfc8SFQ&m=HnuykZkNgy_2cx1i0_QuR-C_vdVYLTJ9rSUA1dCM7WI&s=TW-ZZ_dYy8x6EcvJHtYrv9vaIHSmd0GshvClMwiWPi4&e=
> >
> > I prefer the last submitted patch as it is simpler, but maybe I missed
> > something in Bing patch.
> >
> > Neil, wdyt?
> >
> I'm not sure why I didn't review the previous patch you refrenced, apologies for
> that.
>
> That said, I'm not sure how this patch detects /dev/null patterns. It looks like
> you still expect all lines to start with [+-] [ab]...
It doesn't detect /dev/null but the line following /dev/null will start with
"b/...". So this checks for .map in that line as well.
>
> Neil
>
> >
> > --
> > David Marchand
> >
> >
> > >
> > > Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
> > > ---
> > > Note: We have two examples where checkpatch errors are because of this
> > > because the version.map file change comes earlier in the diff. Because of
> > > this bug, any new file change that comes after version.map file diff
> > > as "/dev/null" to "b/.." gets misdetected as version.map file.
> > > * https://urldefense.proofpoint.com/v2/url?u=http-3A__patches.dpdk.org_patch_66878_&d=DwIBAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=FZ_tPCbgFOh18zwRPO9H0yDx8VW38vuapifdDfc8SFQ&m=HnuykZkNgy_2cx1i0_QuR-C_vdVYLTJ9rSUA1dCM7WI&s=0szl1YpYLwfb1OF-R5H4_ci3f5XjBUHFrHVp-LdROFI&e=
> > > * https://urldefense.proofpoint.com/v2/url?u=https-3A__patchwork.dpdk.org_patch_66900_&d=DwIBAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=FZ_tPCbgFOh18zwRPO9H0yDx8VW38vuapifdDfc8SFQ&m=HnuykZkNgy_2cx1i0_QuR-C_vdVYLTJ9rSUA1dCM7WI&s=ugH7B1XKoFQEEopU0sJgqKGwjDJS9Kl9OsijSazmyRc&e=
> > > devtools/check-symbol-change.sh | 4 ++--
> > > 1 file changed, 2 insertions(+), 2 deletions(-)
> > >
> > > diff --git a/devtools/check-symbol-change.sh b/devtools/check-symbol-change.sh
> > > index c5434f3..19ce82f 100755
> > > --- a/devtools/check-symbol-change.sh
> > > +++ b/devtools/check-symbol-change.sh
> > > @@ -17,13 +17,13 @@ build_map_changes()
> > > # map files are altered, and all section/symbol names
> > > # appearing between a triggering of this rule and the
> > > # next trigger of this rule are associated with this file
> > > - /[-+] a\/.*\.map/ {map=$2; in_map=1}
> > > + /[-+] [ab]\/.*\.map/ {map=$2; in_map=1}
> > >
> > > # Same pattern as above, only it matches on anything that
> > > # does not end in 'map', indicating we have left the map chunk.
> > > # When we hit this, turn off the in_map variable, which
> > > # supresses the subordonate rules below
> > > - /[-+] a\/.*\.[^map]/ {in_map=0}
> > > + /[-+] [ab]\/.*\.[^map]/ {in_map=0}
> > >
> > > # Triggering this rule, which starts a line and ends it
> > > # with a { identifies a versioned section. The section name is
> > > --
> > > 2.8.4
> > >
> >
> >
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] devtools: fix check symbol change script
2020-03-19 15:40 0% ` Neil Horman
2020-03-19 15:45 0% ` [dpdk-dev] [EXT] " Nithin Dabilpuram
@ 2020-03-19 15:49 0% ` Bing Zhao
1 sibling, 0 replies; 200+ results
From: Bing Zhao @ 2020-03-19 15:49 UTC (permalink / raw)
To: Neil Horman, David Marchand
Cc: Thomas Monjalon, dev, Jerin Jacob Kollanukkaran,
Nithin Dabilpuram, dpdk stable
> -----Original Message-----
> From: Neil Horman <nhorman@tuxdriver.com>
> Sent: Thursday, March 19, 2020 11:41 PM
> To: David Marchand <david.marchand@redhat.com>
> Cc: Thomas Monjalon <thomas@monjalon.net>; dev <dev@dpdk.org>;
> Jerin Jacob Kollanukkaran <jerinj@marvell.com>; Nithin Dabilpuram
> <ndabilpuram@marvell.com>; dpdk stable <stable@dpdk.org>; Bing
> Zhao <bingz@mellanox.com>
> Subject: Re: [PATCH] devtools: fix check symbol change script
>
> On Thu, Mar 19, 2020 at 03:56:03PM +0100, David Marchand wrote:
> > On Thu, Mar 19, 2020 at 3:44 PM Nithin Dabilpuram
> > <ndabilpuram@marvell.com> wrote:
> > >
> > > Fix check symbol change script to detect new diff file when it is in
> > > between "--- /dev/null" to "b/lib/...".
> > > Current awk line expects line to start with "a/..."
> > > which is not always true for all diffs.
> > > As a result if in_map was '1' earlier, it will not be changed to '0'
> > > and we get check patch errors which are not true as the non
> > > version.map files get interpreted as version map file.
> > >
> > > Fixes: 4bec48184e33 ("devtools: add checks for ABI symbol
> addition")
> > > Cc: nhorman@tuxdriver.com
> > > Cc: stable@dpdk.org
> >
> > There was a previous attempt at fixing this that did not get a review.
> >
> https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2F
> patch
> >
> work.dpdk.org%2Fpatch%2F56303%2F&data=02%7C01%7Cbingz
> %40mellanox.c
> >
> om%7C19a838a81f4f4649f40f08d7cc1be78a%7Ca652971c7d2e4d9ba6
> a4d149256f46
> >
> 1b%7C0%7C0%7C637202292556475544&sdata=wgz8LssuucgrWlY
> xsf978%2ByRVg
> > 83btxZIm9aD56nekY%3D&reserved=0
> >
> > I prefer the last submitted patch as it is simpler, but maybe I missed
> > something in Bing patch.
> >
> > Neil, wdyt?
> >
> I'm not sure why I didn't review the previous patch you refrenced,
> apologies for that.
>
> That said, I'm not sure how this patch detects /dev/null patterns. It
> looks like you still expect all lines to start with [+-] [ab]...
>
> Neil
Hi David & Neil,
The cause and the fixing are almost the same for the two patches.
Nithin's is simpler.
When matching on "/dev/null", it just needs to continue.
While in my patch, I think I also fix the [^map] matching even though it works.
The behavior of RE in awk does not work as expected, .c and .h files don't match on the
"NOT map" at the end of the line, but match on the signle letter not among the "m/a/p".
(Correct me if anything wrong).
Thanks
Bing
>
> >
> > --
> > David Marchand
> >
> >
> > >
> > > Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
> > > ---
> > > Note: We have two examples where checkpatch errors are because
> of
> > > this because the version.map file change comes earlier in the diff.
> > > Because of this bug, any new file change that comes after
> > > version.map file diff as "/dev/null" to "b/.." gets misdetected as
> version.map file.
> > > *
> > >
> https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2F
> pat
> > >
> ches.dpdk.org%2Fpatch%2F66878%2F&data=02%7C01%7Cbingz%
> 40mellanox
> > > .com%7C19a838a81f4f4649f40f08d7cc1be78a%7Ca652971c7d2e4d
> 9ba6a4d14925
> > >
> 6f461b%7C0%7C0%7C637202292556475544&sdata=ElLbgB9IJ7B6
> kuNTAjKAFr
> > > WpNy8Jdq5%2BmfoRTxN72tI%3D&reserved=0
> > > *
> > >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2F
> pa
> > >
> tchwork.dpdk.org%2Fpatch%2F66900%2F&data=02%7C01%7Cbin
> gz%40mella
> > >
> nox.com%7C19a838a81f4f4649f40f08d7cc1be78a%7Ca652971c7d2e4d
> 9ba6a4d14
> > >
> 9256f461b%7C0%7C0%7C637202292556475544&sdata=iT%2Ft9os
> A4HFTQ2kh3
> > > baWClvD3B%2FFdGzIrQgTvB5SfqU%3D&reserved=0
> > > devtools/check-symbol-change.sh | 4 ++--
> > > 1 file changed, 2 insertions(+), 2 deletions(-)
> > >
> > > diff --git a/devtools/check-symbol-change.sh
> > > b/devtools/check-symbol-change.sh index c5434f3..19ce82f 100755
> > > --- a/devtools/check-symbol-change.sh
> > > +++ b/devtools/check-symbol-change.sh
> > > @@ -17,13 +17,13 @@ build_map_changes()
> > > # map files are altered, and all section/symbol names
> > > # appearing between a triggering of this rule and the
> > > # next trigger of this rule are associated with this file
> > > - /[-+] a\/.*\.map/ {map=$2; in_map=1}
> > > + /[-+] [ab]\/.*\.map/ {map=$2; in_map=1}
> > >
> > > # Same pattern as above, only it matches on anything that
> > > # does not end in 'map', indicating we have left the map
> chunk.
> > > # When we hit this, turn off the in_map variable, which
> > > # supresses the subordonate rules below
> > > - /[-+] a\/.*\.[^map]/ {in_map=0}
> > > + /[-+] [ab]\/.*\.[^map]/ {in_map=0}
> > >
> > > # Triggering this rule, which starts a line and ends it
> > > # with a { identifies a versioned section. The
> > > section name is
> > > --
> > > 2.8.4
> > >
> >
> >
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v1] usertools/dpdk-setup.sh: fix dpdk-setup's behaviour on non-alphanumeric inputs
@ 2020-03-19 18:20 3% ` Stephen Hemminger
0 siblings, 0 replies; 200+ results
From: Stephen Hemminger @ 2020-03-19 18:20 UTC (permalink / raw)
To: Sarosh Arif; +Cc: dev, stable
On Thu, 19 Mar 2020 22:44:23 +0500
Sarosh Arif <sarosh.arif@emumba.com> wrote:
> Bugzilla ID: 419
> Cc: stable@dpdk.org
> Signed-off-by: Sarosh Arif <sarosh.arif@emumba.com>
> ---
> usertools/dpdk-setup.sh | 10 ++++++++--
> 1 file changed, 8 insertions(+), 2 deletions(-)
>
> diff --git a/usertools/dpdk-setup.sh b/usertools/dpdk-setup.sh
> index e5bbe9fee..bdacf5256 100755
> --- a/usertools/dpdk-setup.sh
> +++ b/usertools/dpdk-setup.sh
> @@ -595,8 +595,14 @@ while [ "$QUIT" == "0" ]; do
> echo -n "Option: "
> read our_entry
> echo ""
> - ${OPTIONS[our_entry]} ${our_entry}
> -
> + echo $our_entry | grep "^[0-9]*$" > /dev/null
> +
> + if [ "$?" -eq 0 ] ; then
> + ${OPTIONS[our_entry]} ${our_entry}
> + else
> + echo "Wrong input format"
> + fi
> +
> if [ "$QUIT" == "0" ] ; then
> echo
> echo -n "Press enter to continue ..."; read
Why not do it with regex (abi check has similar code) and
use the nice format.
Also handle EOF
diff --git a/usertools/dpdk-setup.sh b/usertools/dpdk-setup.sh
index e5bbe9feec49..f594769a4ef1 100755
--- a/usertools/dpdk-setup.sh
+++ b/usertools/dpdk-setup.sh
@@ -592,10 +592,16 @@ while [ "$QUIT" == "0" ]; do
echo "[$OPTION_NUM] Exit Script"
OPTIONS[$OPTION_NUM]="quit"
echo ""
- echo -n "Option: "
- read our_entry
+ read -p "Option: " our_entry
+ [ $? -eq 0 ] || exit 0
+
echo ""
- ${OPTIONS[our_entry]} ${our_entry}
+ numeric="^[[:digit:]]+$"
+ if [[ "$our_entry" =~ $numeric ]]; then
+ ${OPTIONS[our_entry]} ${our_entry}
+ else
+ echo "Please enter a numeric value"
+ fi
if [ "$QUIT" == "0" ] ; then
echo
--
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [EXT] Re: [PATCH] devtools: fix check symbol change script
2020-03-19 15:45 0% ` [dpdk-dev] [EXT] " Nithin Dabilpuram
@ 2020-03-19 18:59 0% ` Neil Horman
0 siblings, 0 replies; 200+ results
From: Neil Horman @ 2020-03-19 18:59 UTC (permalink / raw)
To: Nithin Dabilpuram
Cc: David Marchand, Thomas Monjalon, dev, Jerin Jacob Kollanukkaran,
dpdk stable, bingz
On Thu, Mar 19, 2020 at 09:15:50PM +0530, Nithin Dabilpuram wrote:
> On Thu, Mar 19, 2020 at 11:40:39AM -0400, Neil Horman wrote:
> > External Email
> >
> > ----------------------------------------------------------------------
> > On Thu, Mar 19, 2020 at 03:56:03PM +0100, David Marchand wrote:
> > > On Thu, Mar 19, 2020 at 3:44 PM Nithin Dabilpuram
> > > <ndabilpuram@marvell.com> wrote:
> > > >
> > > > Fix check symbol change script to detect new diff file when
> > > > it is in between "--- /dev/null" to "b/lib/...".
> > > > Current awk line expects line to start with "a/..."
> > > > which is not always true for all diffs.
> > > > As a result if in_map was '1' earlier, it will not be changed
> > > > to '0' and we get check patch errors which are not true as the non
> > > > version.map files get interpreted as version map file.
> > > >
> > > > Fixes: 4bec48184e33 ("devtools: add checks for ABI symbol addition")
> > > > Cc: nhorman@tuxdriver.com
> > > > Cc: stable@dpdk.org
> > >
> > > There was a previous attempt at fixing this that did not get a review.
> > > https://urldefense.proofpoint.com/v2/url?u=http-3A__patchwork.dpdk.org_patch_56303_&d=DwIBAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=FZ_tPCbgFOh18zwRPO9H0yDx8VW38vuapifdDfc8SFQ&m=HnuykZkNgy_2cx1i0_QuR-C_vdVYLTJ9rSUA1dCM7WI&s=TW-ZZ_dYy8x6EcvJHtYrv9vaIHSmd0GshvClMwiWPi4&e=
> > >
> > > I prefer the last submitted patch as it is simpler, but maybe I missed
> > > something in Bing patch.
> > >
> > > Neil, wdyt?
> > >
> > I'm not sure why I didn't review the previous patch you refrenced, apologies for
> > that.
> >
> > That said, I'm not sure how this patch detects /dev/null patterns. It looks like
> > you still expect all lines to start with [+-] [ab]...
>
> It doesn't detect /dev/null but the line following /dev/null will start with
> "b/...". So this checks for .map in that line as well.
>
Ah, in that case, I'd really be fine with either fix.
Acked-by: Neil Horman <nhorman@tuxdriver.com>
> >
> > Neil
> >
> > >
> > > --
> > > David Marchand
> > >
> > >
> > > >
> > > > Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
> > > > ---
> > > > Note: We have two examples where checkpatch errors are because of this
> > > > because the version.map file change comes earlier in the diff. Because of
> > > > this bug, any new file change that comes after version.map file diff
> > > > as "/dev/null" to "b/.." gets misdetected as version.map file.
> > > > * https://urldefense.proofpoint.com/v2/url?u=http-3A__patches.dpdk.org_patch_66878_&d=DwIBAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=FZ_tPCbgFOh18zwRPO9H0yDx8VW38vuapifdDfc8SFQ&m=HnuykZkNgy_2cx1i0_QuR-C_vdVYLTJ9rSUA1dCM7WI&s=0szl1YpYLwfb1OF-R5H4_ci3f5XjBUHFrHVp-LdROFI&e=
> > > > * https://urldefense.proofpoint.com/v2/url?u=https-3A__patchwork.dpdk.org_patch_66900_&d=DwIBAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=FZ_tPCbgFOh18zwRPO9H0yDx8VW38vuapifdDfc8SFQ&m=HnuykZkNgy_2cx1i0_QuR-C_vdVYLTJ9rSUA1dCM7WI&s=ugH7B1XKoFQEEopU0sJgqKGwjDJS9Kl9OsijSazmyRc&e=
> > > > devtools/check-symbol-change.sh | 4 ++--
> > > > 1 file changed, 2 insertions(+), 2 deletions(-)
> > > >
> > > > diff --git a/devtools/check-symbol-change.sh b/devtools/check-symbol-change.sh
> > > > index c5434f3..19ce82f 100755
> > > > --- a/devtools/check-symbol-change.sh
> > > > +++ b/devtools/check-symbol-change.sh
> > > > @@ -17,13 +17,13 @@ build_map_changes()
> > > > # map files are altered, and all section/symbol names
> > > > # appearing between a triggering of this rule and the
> > > > # next trigger of this rule are associated with this file
> > > > - /[-+] a\/.*\.map/ {map=$2; in_map=1}
> > > > + /[-+] [ab]\/.*\.map/ {map=$2; in_map=1}
> > > >
> > > > # Same pattern as above, only it matches on anything that
> > > > # does not end in 'map', indicating we have left the map chunk.
> > > > # When we hit this, turn off the in_map variable, which
> > > > # supresses the subordonate rules below
> > > > - /[-+] a\/.*\.[^map]/ {in_map=0}
> > > > + /[-+] [ab]\/.*\.[^map]/ {in_map=0}
> > > >
> > > > # Triggering this rule, which starts a line and ends it
> > > > # with a { identifies a versioned section. The section name is
> > > > --
> > > > 2.8.4
> > > >
> > >
> > >
>
^ permalink raw reply [relevance 0%]
* [dpdk-dev] [PATCH] eal: reorganize directories layout
@ 2020-03-20 0:12 1% Thomas Monjalon
0 siblings, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-03-20 0:12 UTC (permalink / raw)
To: dev
Cc: david.marchand, Neil Horman, John McNamara, Marko Kovacevic,
Hemant Agrawal, Sachin Saxena, Matan Azrad, Shahaf Shuler,
Viacheslav Ovsiienko, Gagandeep Singh, Akhil Goyal, Nipun Gupta,
Maxime Coquelin, Zhihong Wang, Xiaolong Ye, Ferruh Yigit,
Cristian Dumitrescu, Jan Viktorin, Gavin Hu, Jerin Jacob,
Harry van Haaren, Bruce Richardson, Anatoly Burakov, Phil Yang,
Joyce Kong, Mattias Rönnblom, David Christensen,
Harini Ramakrishnan, Omar Cardona, Pallavi Kadam, Ranjit Menon,
Konstantin Ananyev, Olivier Matz
Some clean-up is done in EAL sub-directories.
The goal is to make organization easier to understand,
and to prepare moving some files from common to Unix-only place.
Since the kernel modules are moved to kernel/ directory,
there is no need anymore for the sub-directory eal/ in
linux/, freebsd/ and windows/.
The EAL API (with doxygen documentation) is moved from
common/include/ to include/, which makes more clear that
it is the global API for all environments and architectures.
Note that the arch-specific and os-specific include files are not
in this global include directory, but include/generic/ should
cover the doxygen documentation for them.
The arch-specific directories arm, ppc_64 and x86 in common/arch/
and in common/include/arch are moved at the same level as the
os-specific directories, adding an include/ sub-directory.
It makes more clear that EAL is covering a matrix combining OS and arch.
Note that ppc_64 is renamed to ppc.
These moves offer the opportunity to simplify the make and meson files.
Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
---
This change was proposed to the Technical Board in advance:
http://mails.dpdk.org/archives/dev/2020-January/156732.html
If required, the patch may be split:
1/ remove FreeBSD kernel Makefile
2/ move arch .c files
3/ move arch .h files
4/ move common and generic includes
5/ move os/eal/ to os/
---
---
MAINTAINERS | 53 ++++-----
buildtools/pmdinfogen/meson.build | 2 +-
config/arm/meson.build | 2 +-
config/common_armv8a_linux | 2 +-
config/meson.build | 2 +-
devtools/build-tags.sh | 14 +--
doc/api/doxy-api.conf.in | 4 +-
doc/guides/rel_notes/known_issues.rst | 2 +-
drivers/bus/dpaa/Makefile | 2 +-
drivers/common/mlx5/meson.build | 2 +-
drivers/crypto/caam_jr/Makefile | 2 +-
drivers/crypto/dpaa_sec/Makefile | 2 +-
drivers/event/dpaa/Makefile | 2 +-
drivers/net/dpaa/Makefile | 2 +-
drivers/net/virtio/meson.build | 2 +-
kernel/linux/igb_uio/meson.build | 2 +-
kernel/linux/kni/meson.build | 4 +-
lib/librte_cfgfile/Makefile | 2 +-
lib/librte_eal/Makefile | 6 +-
.../arch/arm => arm/include}/meson.build | 0
.../arch/arm => arm/include}/rte_atomic.h | 0
.../arch/arm => arm/include}/rte_atomic_32.h | 0
.../arch/arm => arm/include}/rte_atomic_64.h | 0
.../arch/arm => arm/include}/rte_byteorder.h | 0
.../arch/arm => arm/include}/rte_cpuflags.h | 0
.../arm => arm/include}/rte_cpuflags_32.h | 0
.../arm => arm/include}/rte_cpuflags_64.h | 0
.../arch/arm => arm/include}/rte_cycles.h | 0
.../arch/arm => arm/include}/rte_cycles_32.h | 0
.../arch/arm => arm/include}/rte_cycles_64.h | 0
.../include/arch/arm => arm/include}/rte_io.h | 0
.../arch/arm => arm/include}/rte_io_64.h | 0
.../arch/arm => arm/include}/rte_mcslock.h | 0
.../arch/arm => arm/include}/rte_memcpy.h | 0
.../arch/arm => arm/include}/rte_memcpy_32.h | 0
.../arch/arm => arm/include}/rte_memcpy_64.h | 0
.../arch/arm => arm/include}/rte_pause.h | 0
.../arch/arm => arm/include}/rte_pause_32.h | 0
.../arch/arm => arm/include}/rte_pause_64.h | 0
.../arch/arm => arm/include}/rte_prefetch.h | 0
.../arm => arm/include}/rte_prefetch_32.h | 0
.../arm => arm/include}/rte_prefetch_64.h | 0
.../arch/arm => arm/include}/rte_rwlock.h | 0
.../arch/arm => arm/include}/rte_spinlock.h | 0
.../arch/arm => arm/include}/rte_ticketlock.h | 0
.../arch/arm => arm/include}/rte_vect.h | 0
lib/librte_eal/arm/meson.build | 13 +++
.../{common/arch => }/arm/rte_cpuflags.c | 0
.../{common/arch => }/arm/rte_cycles.c | 0
.../{common/arch => }/arm/rte_hypervisor.c | 0
lib/librte_eal/common/Makefile | 38 -------
lib/librte_eal/common/arch/arm/meson.build | 5 -
lib/librte_eal/common/arch/ppc_64/meson.build | 5 -
lib/librte_eal/common/arch/x86/meson.build | 5 -
lib/librte_eal/common/meson.build | 92 ++++------------
lib/librte_eal/common/rte_service.c | 2 +-
lib/librte_eal/freebsd/BSDmakefile.meson | 14 ---
lib/librte_eal/freebsd/Makefile | 90 +++++++++++++++-
lib/librte_eal/freebsd/{eal => }/eal.c | 0
lib/librte_eal/freebsd/eal/Makefile | 93 ----------------
lib/librte_eal/freebsd/{eal => }/eal_alarm.c | 0
.../freebsd/{eal => }/eal_alarm_private.h | 0
.../freebsd/{eal => }/eal_cpuflags.c | 0
lib/librte_eal/freebsd/{eal => }/eal_debug.c | 0
lib/librte_eal/freebsd/{eal => }/eal_dev.c | 0
.../freebsd/{eal => }/eal_hugepage_info.c | 0
.../freebsd/{eal => }/eal_interrupts.c | 0
lib/librte_eal/freebsd/{eal => }/eal_lcore.c | 0
.../freebsd/{eal => }/eal_memalloc.c | 0
lib/librte_eal/freebsd/{eal => }/eal_memory.c | 0
lib/librte_eal/freebsd/{eal => }/eal_thread.c | 0
lib/librte_eal/freebsd/{eal => }/eal_timer.c | 0
.../freebsd/{eal => }/include/rte_os.h | 0
lib/librte_eal/freebsd/{eal => }/meson.build | 8 +-
.../{common => }/include/generic/rte_atomic.h | 0
.../include/generic/rte_byteorder.h | 0
.../include/generic/rte_cpuflags.h | 0
.../{common => }/include/generic/rte_cycles.h | 0
.../{common => }/include/generic/rte_io.h | 0
.../include/generic/rte_mcslock.h | 0
.../{common => }/include/generic/rte_memcpy.h | 0
.../{common => }/include/generic/rte_pause.h | 0
.../include/generic/rte_prefetch.h | 0
.../{common => }/include/generic/rte_rwlock.h | 0
.../include/generic/rte_spinlock.h | 0
.../include/generic/rte_ticketlock.h | 0
.../{common => }/include/generic/rte_vect.h | 0
.../{common => }/include/rte_alarm.h | 0
.../{common => }/include/rte_bitmap.h | 0
.../include/rte_branch_prediction.h | 0
lib/librte_eal/{common => }/include/rte_bus.h | 0
.../{common => }/include/rte_class.h | 0
.../{common => }/include/rte_common.h | 0
.../{common => }/include/rte_compat.h | 0
.../{common => }/include/rte_debug.h | 0
lib/librte_eal/{common => }/include/rte_dev.h | 0
.../{common => }/include/rte_devargs.h | 0
lib/librte_eal/{common => }/include/rte_eal.h | 0
.../{common => }/include/rte_eal_interrupts.h | 0
.../{common => }/include/rte_eal_memconfig.h | 0
.../{common => }/include/rte_errno.h | 0
.../{common => }/include/rte_fbarray.h | 0
.../include/rte_function_versioning.h | 0
.../{common => }/include/rte_hexdump.h | 0
.../{common => }/include/rte_hypervisor.h | 0
.../{common => }/include/rte_interrupts.h | 0
.../{common => }/include/rte_keepalive.h | 0
.../{common => }/include/rte_launch.h | 0
.../{common => }/include/rte_lcore.h | 0
lib/librte_eal/{common => }/include/rte_log.h | 0
.../{common => }/include/rte_malloc.h | 0
.../{common => }/include/rte_memory.h | 0
.../{common => }/include/rte_memzone.h | 0
.../{common => }/include/rte_option.h | 0
.../include/rte_pci_dev_feature_defs.h | 0
.../include/rte_pci_dev_features.h | 0
.../{common => }/include/rte_per_lcore.h | 0
.../{common => }/include/rte_random.h | 0
.../{common => }/include/rte_reciprocal.h | 0
.../{common => }/include/rte_service.h | 0
.../include/rte_service_component.h | 0
.../{common => }/include/rte_string_fns.h | 0
.../{common => }/include/rte_tailq.h | 0
.../{common => }/include/rte_test.h | 0
.../{common => }/include/rte_time.h | 0
.../{common => }/include/rte_uuid.h | 0
.../{common => }/include/rte_version.h | 0
.../{common => }/include/rte_vfio.h | 0
lib/librte_eal/linux/Makefile | 96 ++++++++++++++++-
lib/librte_eal/linux/{eal => }/eal.c | 0
lib/librte_eal/linux/eal/Makefile | 101 ------------------
lib/librte_eal/linux/{eal => }/eal_alarm.c | 0
lib/librte_eal/linux/{eal => }/eal_cpuflags.c | 0
lib/librte_eal/linux/{eal => }/eal_debug.c | 0
lib/librte_eal/linux/{eal => }/eal_dev.c | 0
.../linux/{eal => }/eal_hugepage_info.c | 0
.../linux/{eal => }/eal_interrupts.c | 0
lib/librte_eal/linux/{eal => }/eal_lcore.c | 0
lib/librte_eal/linux/{eal => }/eal_log.c | 0
lib/librte_eal/linux/{eal => }/eal_memalloc.c | 0
lib/librte_eal/linux/{eal => }/eal_memory.c | 0
lib/librte_eal/linux/{eal => }/eal_thread.c | 0
lib/librte_eal/linux/{eal => }/eal_timer.c | 0
lib/librte_eal/linux/{eal => }/eal_vfio.c | 0
lib/librte_eal/linux/{eal => }/eal_vfio.h | 0
.../linux/{eal => }/eal_vfio_mp_sync.c | 0
.../linux/{eal => }/include/rte_kni_common.h | 0
.../linux/{eal => }/include/rte_os.h | 0
lib/librte_eal/linux/{eal => }/meson.build | 9 +-
lib/librte_eal/meson.build | 20 ++--
.../arch/ppc_64 => ppc/include}/meson.build | 0
.../arch/ppc_64 => ppc/include}/rte_atomic.h | 0
.../ppc_64 => ppc/include}/rte_byteorder.h | 0
.../ppc_64 => ppc/include}/rte_cpuflags.h | 0
.../arch/ppc_64 => ppc/include}/rte_cycles.h | 0
.../arch/ppc_64 => ppc/include}/rte_io.h | 0
.../arch/ppc_64 => ppc/include}/rte_mcslock.h | 0
.../arch/ppc_64 => ppc/include}/rte_memcpy.h | 0
.../arch/ppc_64 => ppc/include}/rte_pause.h | 0
.../ppc_64 => ppc/include}/rte_prefetch.h | 0
.../arch/ppc_64 => ppc/include}/rte_rwlock.h | 0
.../ppc_64 => ppc/include}/rte_spinlock.h | 0
.../ppc_64 => ppc/include}/rte_ticketlock.h | 0
.../arch/ppc_64 => ppc/include}/rte_vect.h | 0
lib/librte_eal/ppc/meson.build | 13 +++
.../arch/ppc_64 => ppc}/rte_cpuflags.c | 0
.../{common/arch/ppc_64 => ppc}/rte_cycles.c | 0
.../arch/ppc_64 => ppc}/rte_hypervisor.c | 0
lib/librte_eal/windows/{eal => }/eal.c | 0
lib/librte_eal/windows/eal/meson.build | 27 -----
lib/librte_eal/windows/{eal => }/eal_debug.c | 0
lib/librte_eal/windows/{eal => }/eal_lcore.c | 0
lib/librte_eal/windows/{eal => }/eal_thread.c | 0
lib/librte_eal/windows/{eal => }/getopt.c | 0
.../windows/{eal => }/include/dirent.h | 0
.../windows/{eal => }/include/fnmatch.h | 0
.../windows/{eal => }/include/getopt.h | 0
.../windows/{eal => }/include/pthread.h | 0
.../windows/{eal => }/include/regex.h | 0
.../windows/{eal => }/include/rte_os.h | 0
.../windows/{eal => }/include/sched.h | 0
.../windows/{eal => }/include/sys/queue.h | 0
.../windows/{eal => }/include/unistd.h | 0
lib/librte_eal/windows/meson.build | 16 +++
.../arch/x86 => x86/include}/meson.build | 0
.../arch/x86 => x86/include}/rte_atomic.h | 0
.../arch/x86 => x86/include}/rte_atomic_32.h | 0
.../arch/x86 => x86/include}/rte_atomic_64.h | 0
.../arch/x86 => x86/include}/rte_byteorder.h | 0
.../x86 => x86/include}/rte_byteorder_32.h | 0
.../x86 => x86/include}/rte_byteorder_64.h | 0
.../arch/x86 => x86/include}/rte_cpuflags.h | 0
.../arch/x86 => x86/include}/rte_cycles.h | 0
.../include/arch/x86 => x86/include}/rte_io.h | 0
.../arch/x86 => x86/include}/rte_mcslock.h | 0
.../arch/x86 => x86/include}/rte_memcpy.h | 0
.../arch/x86 => x86/include}/rte_pause.h | 0
.../arch/x86 => x86/include}/rte_prefetch.h | 0
.../arch/x86 => x86/include}/rte_rtm.h | 0
.../arch/x86 => x86/include}/rte_rwlock.h | 0
.../arch/x86 => x86/include}/rte_spinlock.h | 0
.../arch/x86 => x86/include}/rte_ticketlock.h | 0
.../arch/x86 => x86/include}/rte_vect.h | 0
lib/librte_eal/x86/meson.build | 14 +++
.../{common/arch => }/x86/rte_cpuflags.c | 0
.../{common/arch => }/x86/rte_cpuid.h | 0
.../{common/arch => }/x86/rte_cycles.c | 0
.../{common/arch => }/x86/rte_hypervisor.c | 0
.../{common/arch => }/x86/rte_spinlock.c | 0
lib/librte_kvargs/Makefile | 2 +-
license/exceptions.txt | 6 +-
meson.build | 4 +-
mk/exec-env/freebsd/rte.vars.mk | 2 +-
mk/exec-env/linux/rte.vars.mk | 2 +-
214 files changed, 330 insertions(+), 456 deletions(-)
rename lib/librte_eal/{common/include/arch/arm => arm/include}/meson.build (100%)
rename lib/librte_eal/{common/include/arch/arm => arm/include}/rte_atomic.h (100%)
rename lib/librte_eal/{common/include/arch/arm => arm/include}/rte_atomic_32.h (100%)
rename lib/librte_eal/{common/include/arch/arm => arm/include}/rte_atomic_64.h (100%)
rename lib/librte_eal/{common/include/arch/arm => arm/include}/rte_byteorder.h (100%)
rename lib/librte_eal/{common/include/arch/arm => arm/include}/rte_cpuflags.h (100%)
rename lib/librte_eal/{common/include/arch/arm => arm/include}/rte_cpuflags_32.h (100%)
rename lib/librte_eal/{common/include/arch/arm => arm/include}/rte_cpuflags_64.h (100%)
rename lib/librte_eal/{common/include/arch/arm => arm/include}/rte_cycles.h (100%)
rename lib/librte_eal/{common/include/arch/arm => arm/include}/rte_cycles_32.h (100%)
rename lib/librte_eal/{common/include/arch/arm => arm/include}/rte_cycles_64.h (100%)
rename lib/librte_eal/{common/include/arch/arm => arm/include}/rte_io.h (100%)
rename lib/librte_eal/{common/include/arch/arm => arm/include}/rte_io_64.h (100%)
rename lib/librte_eal/{common/include/arch/arm => arm/include}/rte_mcslock.h (100%)
rename lib/librte_eal/{common/include/arch/arm => arm/include}/rte_memcpy.h (100%)
rename lib/librte_eal/{common/include/arch/arm => arm/include}/rte_memcpy_32.h (100%)
rename lib/librte_eal/{common/include/arch/arm => arm/include}/rte_memcpy_64.h (100%)
rename lib/librte_eal/{common/include/arch/arm => arm/include}/rte_pause.h (100%)
rename lib/librte_eal/{common/include/arch/arm => arm/include}/rte_pause_32.h (100%)
rename lib/librte_eal/{common/include/arch/arm => arm/include}/rte_pause_64.h (100%)
rename lib/librte_eal/{common/include/arch/arm => arm/include}/rte_prefetch.h (100%)
rename lib/librte_eal/{common/include/arch/arm => arm/include}/rte_prefetch_32.h (100%)
rename lib/librte_eal/{common/include/arch/arm => arm/include}/rte_prefetch_64.h (100%)
rename lib/librte_eal/{common/include/arch/arm => arm/include}/rte_rwlock.h (100%)
rename lib/librte_eal/{common/include/arch/arm => arm/include}/rte_spinlock.h (100%)
rename lib/librte_eal/{common/include/arch/arm => arm/include}/rte_ticketlock.h (100%)
rename lib/librte_eal/{common/include/arch/arm => arm/include}/rte_vect.h (100%)
create mode 100644 lib/librte_eal/arm/meson.build
rename lib/librte_eal/{common/arch => }/arm/rte_cpuflags.c (100%)
rename lib/librte_eal/{common/arch => }/arm/rte_cycles.c (100%)
rename lib/librte_eal/{common/arch => }/arm/rte_hypervisor.c (100%)
delete mode 100644 lib/librte_eal/common/Makefile
delete mode 100644 lib/librte_eal/common/arch/arm/meson.build
delete mode 100644 lib/librte_eal/common/arch/ppc_64/meson.build
delete mode 100644 lib/librte_eal/common/arch/x86/meson.build
delete mode 100644 lib/librte_eal/freebsd/BSDmakefile.meson
rename lib/librte_eal/freebsd/{eal => }/eal.c (100%)
delete mode 100644 lib/librte_eal/freebsd/eal/Makefile
rename lib/librte_eal/freebsd/{eal => }/eal_alarm.c (100%)
rename lib/librte_eal/freebsd/{eal => }/eal_alarm_private.h (100%)
rename lib/librte_eal/freebsd/{eal => }/eal_cpuflags.c (100%)
rename lib/librte_eal/freebsd/{eal => }/eal_debug.c (100%)
rename lib/librte_eal/freebsd/{eal => }/eal_dev.c (100%)
rename lib/librte_eal/freebsd/{eal => }/eal_hugepage_info.c (100%)
rename lib/librte_eal/freebsd/{eal => }/eal_interrupts.c (100%)
rename lib/librte_eal/freebsd/{eal => }/eal_lcore.c (100%)
rename lib/librte_eal/freebsd/{eal => }/eal_memalloc.c (100%)
rename lib/librte_eal/freebsd/{eal => }/eal_memory.c (100%)
rename lib/librte_eal/freebsd/{eal => }/eal_thread.c (100%)
rename lib/librte_eal/freebsd/{eal => }/eal_timer.c (100%)
rename lib/librte_eal/freebsd/{eal => }/include/rte_os.h (100%)
rename lib/librte_eal/freebsd/{eal => }/meson.build (77%)
rename lib/librte_eal/{common => }/include/generic/rte_atomic.h (100%)
rename lib/librte_eal/{common => }/include/generic/rte_byteorder.h (100%)
rename lib/librte_eal/{common => }/include/generic/rte_cpuflags.h (100%)
rename lib/librte_eal/{common => }/include/generic/rte_cycles.h (100%)
rename lib/librte_eal/{common => }/include/generic/rte_io.h (100%)
rename lib/librte_eal/{common => }/include/generic/rte_mcslock.h (100%)
rename lib/librte_eal/{common => }/include/generic/rte_memcpy.h (100%)
rename lib/librte_eal/{common => }/include/generic/rte_pause.h (100%)
rename lib/librte_eal/{common => }/include/generic/rte_prefetch.h (100%)
rename lib/librte_eal/{common => }/include/generic/rte_rwlock.h (100%)
rename lib/librte_eal/{common => }/include/generic/rte_spinlock.h (100%)
rename lib/librte_eal/{common => }/include/generic/rte_ticketlock.h (100%)
rename lib/librte_eal/{common => }/include/generic/rte_vect.h (100%)
rename lib/librte_eal/{common => }/include/rte_alarm.h (100%)
rename lib/librte_eal/{common => }/include/rte_bitmap.h (100%)
rename lib/librte_eal/{common => }/include/rte_branch_prediction.h (100%)
rename lib/librte_eal/{common => }/include/rte_bus.h (100%)
rename lib/librte_eal/{common => }/include/rte_class.h (100%)
rename lib/librte_eal/{common => }/include/rte_common.h (100%)
rename lib/librte_eal/{common => }/include/rte_compat.h (100%)
rename lib/librte_eal/{common => }/include/rte_debug.h (100%)
rename lib/librte_eal/{common => }/include/rte_dev.h (100%)
rename lib/librte_eal/{common => }/include/rte_devargs.h (100%)
rename lib/librte_eal/{common => }/include/rte_eal.h (100%)
rename lib/librte_eal/{common => }/include/rte_eal_interrupts.h (100%)
rename lib/librte_eal/{common => }/include/rte_eal_memconfig.h (100%)
rename lib/librte_eal/{common => }/include/rte_errno.h (100%)
rename lib/librte_eal/{common => }/include/rte_fbarray.h (100%)
rename lib/librte_eal/{common => }/include/rte_function_versioning.h (100%)
rename lib/librte_eal/{common => }/include/rte_hexdump.h (100%)
rename lib/librte_eal/{common => }/include/rte_hypervisor.h (100%)
rename lib/librte_eal/{common => }/include/rte_interrupts.h (100%)
rename lib/librte_eal/{common => }/include/rte_keepalive.h (100%)
rename lib/librte_eal/{common => }/include/rte_launch.h (100%)
rename lib/librte_eal/{common => }/include/rte_lcore.h (100%)
rename lib/librte_eal/{common => }/include/rte_log.h (100%)
rename lib/librte_eal/{common => }/include/rte_malloc.h (100%)
rename lib/librte_eal/{common => }/include/rte_memory.h (100%)
rename lib/librte_eal/{common => }/include/rte_memzone.h (100%)
rename lib/librte_eal/{common => }/include/rte_option.h (100%)
rename lib/librte_eal/{common => }/include/rte_pci_dev_feature_defs.h (100%)
rename lib/librte_eal/{common => }/include/rte_pci_dev_features.h (100%)
rename lib/librte_eal/{common => }/include/rte_per_lcore.h (100%)
rename lib/librte_eal/{common => }/include/rte_random.h (100%)
rename lib/librte_eal/{common => }/include/rte_reciprocal.h (100%)
rename lib/librte_eal/{common => }/include/rte_service.h (100%)
rename lib/librte_eal/{common => }/include/rte_service_component.h (100%)
rename lib/librte_eal/{common => }/include/rte_string_fns.h (100%)
rename lib/librte_eal/{common => }/include/rte_tailq.h (100%)
rename lib/librte_eal/{common => }/include/rte_test.h (100%)
rename lib/librte_eal/{common => }/include/rte_time.h (100%)
rename lib/librte_eal/{common => }/include/rte_uuid.h (100%)
rename lib/librte_eal/{common => }/include/rte_version.h (100%)
rename lib/librte_eal/{common => }/include/rte_vfio.h (100%)
rename lib/librte_eal/linux/{eal => }/eal.c (100%)
delete mode 100644 lib/librte_eal/linux/eal/Makefile
rename lib/librte_eal/linux/{eal => }/eal_alarm.c (100%)
rename lib/librte_eal/linux/{eal => }/eal_cpuflags.c (100%)
rename lib/librte_eal/linux/{eal => }/eal_debug.c (100%)
rename lib/librte_eal/linux/{eal => }/eal_dev.c (100%)
rename lib/librte_eal/linux/{eal => }/eal_hugepage_info.c (100%)
rename lib/librte_eal/linux/{eal => }/eal_interrupts.c (100%)
rename lib/librte_eal/linux/{eal => }/eal_lcore.c (100%)
rename lib/librte_eal/linux/{eal => }/eal_log.c (100%)
rename lib/librte_eal/linux/{eal => }/eal_memalloc.c (100%)
rename lib/librte_eal/linux/{eal => }/eal_memory.c (100%)
rename lib/librte_eal/linux/{eal => }/eal_thread.c (100%)
rename lib/librte_eal/linux/{eal => }/eal_timer.c (100%)
rename lib/librte_eal/linux/{eal => }/eal_vfio.c (100%)
rename lib/librte_eal/linux/{eal => }/eal_vfio.h (100%)
rename lib/librte_eal/linux/{eal => }/eal_vfio_mp_sync.c (100%)
rename lib/librte_eal/linux/{eal => }/include/rte_kni_common.h (100%)
rename lib/librte_eal/linux/{eal => }/include/rte_os.h (100%)
rename lib/librte_eal/linux/{eal => }/meson.build (81%)
rename lib/librte_eal/{common/include/arch/ppc_64 => ppc/include}/meson.build (100%)
rename lib/librte_eal/{common/include/arch/ppc_64 => ppc/include}/rte_atomic.h (100%)
rename lib/librte_eal/{common/include/arch/ppc_64 => ppc/include}/rte_byteorder.h (100%)
rename lib/librte_eal/{common/include/arch/ppc_64 => ppc/include}/rte_cpuflags.h (100%)
rename lib/librte_eal/{common/include/arch/ppc_64 => ppc/include}/rte_cycles.h (100%)
rename lib/librte_eal/{common/include/arch/ppc_64 => ppc/include}/rte_io.h (100%)
rename lib/librte_eal/{common/include/arch/ppc_64 => ppc/include}/rte_mcslock.h (100%)
rename lib/librte_eal/{common/include/arch/ppc_64 => ppc/include}/rte_memcpy.h (100%)
rename lib/librte_eal/{common/include/arch/ppc_64 => ppc/include}/rte_pause.h (100%)
rename lib/librte_eal/{common/include/arch/ppc_64 => ppc/include}/rte_prefetch.h (100%)
rename lib/librte_eal/{common/include/arch/ppc_64 => ppc/include}/rte_rwlock.h (100%)
rename lib/librte_eal/{common/include/arch/ppc_64 => ppc/include}/rte_spinlock.h (100%)
rename lib/librte_eal/{common/include/arch/ppc_64 => ppc/include}/rte_ticketlock.h (100%)
rename lib/librte_eal/{common/include/arch/ppc_64 => ppc/include}/rte_vect.h (100%)
create mode 100644 lib/librte_eal/ppc/meson.build
rename lib/librte_eal/{common/arch/ppc_64 => ppc}/rte_cpuflags.c (100%)
rename lib/librte_eal/{common/arch/ppc_64 => ppc}/rte_cycles.c (100%)
rename lib/librte_eal/{common/arch/ppc_64 => ppc}/rte_hypervisor.c (100%)
rename lib/librte_eal/windows/{eal => }/eal.c (100%)
delete mode 100644 lib/librte_eal/windows/eal/meson.build
rename lib/librte_eal/windows/{eal => }/eal_debug.c (100%)
rename lib/librte_eal/windows/{eal => }/eal_lcore.c (100%)
rename lib/librte_eal/windows/{eal => }/eal_thread.c (100%)
rename lib/librte_eal/windows/{eal => }/getopt.c (100%)
rename lib/librte_eal/windows/{eal => }/include/dirent.h (100%)
rename lib/librte_eal/windows/{eal => }/include/fnmatch.h (100%)
rename lib/librte_eal/windows/{eal => }/include/getopt.h (100%)
rename lib/librte_eal/windows/{eal => }/include/pthread.h (100%)
rename lib/librte_eal/windows/{eal => }/include/regex.h (100%)
rename lib/librte_eal/windows/{eal => }/include/rte_os.h (100%)
rename lib/librte_eal/windows/{eal => }/include/sched.h (100%)
rename lib/librte_eal/windows/{eal => }/include/sys/queue.h (100%)
rename lib/librte_eal/windows/{eal => }/include/unistd.h (100%)
create mode 100644 lib/librte_eal/windows/meson.build
rename lib/librte_eal/{common/include/arch/x86 => x86/include}/meson.build (100%)
rename lib/librte_eal/{common/include/arch/x86 => x86/include}/rte_atomic.h (100%)
rename lib/librte_eal/{common/include/arch/x86 => x86/include}/rte_atomic_32.h (100%)
rename lib/librte_eal/{common/include/arch/x86 => x86/include}/rte_atomic_64.h (100%)
rename lib/librte_eal/{common/include/arch/x86 => x86/include}/rte_byteorder.h (100%)
rename lib/librte_eal/{common/include/arch/x86 => x86/include}/rte_byteorder_32.h (100%)
rename lib/librte_eal/{common/include/arch/x86 => x86/include}/rte_byteorder_64.h (100%)
rename lib/librte_eal/{common/include/arch/x86 => x86/include}/rte_cpuflags.h (100%)
rename lib/librte_eal/{common/include/arch/x86 => x86/include}/rte_cycles.h (100%)
rename lib/librte_eal/{common/include/arch/x86 => x86/include}/rte_io.h (100%)
rename lib/librte_eal/{common/include/arch/x86 => x86/include}/rte_mcslock.h (100%)
rename lib/librte_eal/{common/include/arch/x86 => x86/include}/rte_memcpy.h (100%)
rename lib/librte_eal/{common/include/arch/x86 => x86/include}/rte_pause.h (100%)
rename lib/librte_eal/{common/include/arch/x86 => x86/include}/rte_prefetch.h (100%)
rename lib/librte_eal/{common/include/arch/x86 => x86/include}/rte_rtm.h (100%)
rename lib/librte_eal/{common/include/arch/x86 => x86/include}/rte_rwlock.h (100%)
rename lib/librte_eal/{common/include/arch/x86 => x86/include}/rte_spinlock.h (100%)
rename lib/librte_eal/{common/include/arch/x86 => x86/include}/rte_ticketlock.h (100%)
rename lib/librte_eal/{common/include/arch/x86 => x86/include}/rte_vect.h (100%)
create mode 100644 lib/librte_eal/x86/meson.build
rename lib/librte_eal/{common/arch => }/x86/rte_cpuflags.c (100%)
rename lib/librte_eal/{common/arch => }/x86/rte_cpuid.h (100%)
rename lib/librte_eal/{common/arch => }/x86/rte_cycles.c (100%)
rename lib/librte_eal/{common/arch => }/x86/rte_hypervisor.c (100%)
rename lib/librte_eal/{common/arch => }/x86/rte_spinlock.c (100%)
diff --git a/MAINTAINERS b/MAINTAINERS
index c3785554fc..6be6c77574 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -126,7 +126,6 @@ F: doc/guides/prog_guide/ext_app_lib_make_help.rst
Meson build
M: Bruce Richardson <bruce.richardson@intel.com>
F: meson.build
-F: lib/librte_eal/freebsd/BSDmakefile.meson
F: meson_options.txt
F: config/rte_config.h
F: buildtools/call-sphinx-build.py
@@ -143,8 +142,8 @@ F: .ci/
ABI versioning
M: Neil Horman <nhorman@tuxdriver.com>
-F: lib/librte_eal/common/include/rte_compat.h
-F: lib/librte_eal/common/include/rte_function_versioning.h
+F: lib/librte_eal/include/rte_compat.h
+F: lib/librte_eal/include/rte_function_versioning.h
F: doc/guides/rel_notes/deprecation.rst
F: devtools/check-abi.sh
F: devtools/check-abi-version.sh
@@ -169,9 +168,8 @@ Environment Abstraction Layer
T: git://dpdk.org/dpdk
EAL API and common code
-F: lib/librte_eal/common/*
-F: lib/librte_eal/common/include/*
-F: lib/librte_eal/common/include/generic/
+F: lib/librte_eal/common/
+F: lib/librte_eal/include/
F: lib/librte_eal/rte_eal_version.map
F: doc/guides/prog_guide/env_abstraction_layer.rst
F: app/test/test_alarm.c
@@ -198,15 +196,15 @@ F: app/test/test_version.c
Memory Allocation
M: Anatoly Burakov <anatoly.burakov@intel.com>
-F: lib/librte_eal/common/include/rte_fbarray.h
-F: lib/librte_eal/common/include/rte_mem*
-F: lib/librte_eal/common/include/rte_malloc.h
+F: lib/librte_eal/include/rte_fbarray.h
+F: lib/librte_eal/include/rte_mem*
+F: lib/librte_eal/include/rte_malloc.h
F: lib/librte_eal/common/*malloc*
F: lib/librte_eal/common/eal_common_fbarray.c
F: lib/librte_eal/common/eal_common_mem*
F: lib/librte_eal/common/eal_hugepages.h
-F: lib/librte_eal/linux/eal/eal_mem*
-F: lib/librte_eal/freebsd/eal/eal_mem*
+F: lib/librte_eal/linux/eal_mem*
+F: lib/librte_eal/freebsd/eal_mem*
F: doc/guides/prog_guide/env_abstraction_layer.rst
F: app/test/test_external_mem.c
F: app/test/test_func_reentrancy.c
@@ -216,7 +214,7 @@ F: app/test/test_memory.c
F: app/test/test_memzone.c
Keep alive
-F: lib/librte_eal/common/include/rte_keepalive.h
+F: lib/librte_eal/include/rte_keepalive.h
F: lib/librte_eal/common/rte_keepalive.c
F: examples/l2fwd-keepalive/
F: doc/guides/sample_app_ug/keep_alive.rst
@@ -232,43 +230,42 @@ F: doc/guides/sample_app_ug/multi_process.rst
Service Cores
M: Harry van Haaren <harry.van.haaren@intel.com>
-F: lib/librte_eal/common/include/rte_service.h
-F: lib/librte_eal/common/include/rte_service_component.h
+F: lib/librte_eal/include/rte_service.h
+F: lib/librte_eal/include/rte_service_component.h
F: lib/librte_eal/common/rte_service.c
F: doc/guides/prog_guide/service_cores.rst
F: app/test/test_service_cores.c
Bitmap
M: Cristian Dumitrescu <cristian.dumitrescu@intel.com>
-F: lib/librte_eal/common/include/rte_bitmap.h
+F: lib/librte_eal/include/rte_bitmap.h
F: app/test/test_bitmap.c
MCSlock - EXPERIMENTAL
M: Phil Yang <phil.yang@arm.com>
-F: lib/librte_eal/common/include/generic/rte_mcslock.h
+F: lib/librte_eal/include/generic/rte_mcslock.h
F: app/test/test_mcslock.c
Ticketlock
M: Joyce Kong <joyce.kong@arm.com>
-F: lib/librte_eal/common/include/generic/rte_ticketlock.h
+F: lib/librte_eal/include/generic/rte_ticketlock.h
F: app/test/test_ticketlock.c
Pseudo-random Number Generation
M: Mattias Rönnblom <mattias.ronnblom@ericsson.com>
-F: lib/librte_eal/common/include/rte_random.h
+F: lib/librte_eal/include/rte_random.h
F: lib/librte_eal/common/rte_random.c
F: app/test/test_rand_perf.c
ARM v7
M: Jan Viktorin <viktorin@rehivetech.com>
M: Gavin Hu <gavin.hu@arm.com>
-F: lib/librte_eal/common/arch/arm/
-F: lib/librte_eal/common/include/arch/arm/
+F: lib/librte_eal/arm/
ARM v8
M: Jerin Jacob <jerinj@marvell.com>
M: Gavin Hu <gavin.hu@arm.com>
-F: lib/librte_eal/common/include/arch/arm/*_64.h
+F: lib/librte_eal/arm/include/*_64.h
F: lib/librte_net/net_crc_neon.h
F: lib/librte_acl/acl_run_neon.*
F: lib/librte_bpf/bpf_jit_arm64.c
@@ -282,20 +279,17 @@ F: drivers/net/virtio/virtio_rxtx_simple_neon.c
IBM POWER (alpha)
M: David Christensen <drc@linux.vnet.ibm.com>
-F: lib/librte_eal/common/arch/ppc_64/
-F: lib/librte_eal/common/include/arch/ppc_64/
+F: lib/librte_eal/ppc/
F: drivers/net/i40e/i40e_rxtx_vec_altivec.c
F: examples/l3fwd/*altivec.h
Intel x86
M: Bruce Richardson <bruce.richardson@intel.com>
M: Konstantin Ananyev <konstantin.ananyev@intel.com>
-F: lib/librte_eal/common/arch/x86/
-F: lib/librte_eal/common/include/arch/x86/
+F: lib/librte_eal/x86/
Linux EAL (with overlaps)
-F: lib/librte_eal/linux/Makefile
-F: lib/librte_eal/linux/eal/
+F: lib/librte_eal/linux/
F: doc/guides/linux_gsg/
Linux UIO
@@ -305,13 +299,12 @@ F: drivers/bus/pci/linux/*uio*
Linux VFIO
M: Anatoly Burakov <anatoly.burakov@intel.com>
-F: lib/librte_eal/linux/eal/*vfio*
+F: lib/librte_eal/linux/*vfio*
F: drivers/bus/pci/linux/*vfio*
FreeBSD EAL (with overlaps)
M: Bruce Richardson <bruce.richardson@intel.com>
-F: lib/librte_eal/freebsd/Makefile
-F: lib/librte_eal/freebsd/eal/
+F: lib/librte_eal/freebsd/
F: doc/guides/freebsd_gsg/
FreeBSD contigmem
diff --git a/buildtools/pmdinfogen/meson.build b/buildtools/pmdinfogen/meson.build
index 899ba112cd..7da415b3b7 100644
--- a/buildtools/pmdinfogen/meson.build
+++ b/buildtools/pmdinfogen/meson.build
@@ -6,7 +6,7 @@ if host_machine.system() == 'windows'
endif
pmdinfogen_inc = [global_inc]
-pmdinfogen_inc += include_directories('../../lib/librte_eal/common/include')
+pmdinfogen_inc += include_directories('../../lib/librte_eal/include')
pmdinfogen_inc += include_directories('../../lib/librte_pci')
pmdinfogen = executable('pmdinfogen',
'pmdinfogen.c',
diff --git a/config/arm/meson.build b/config/arm/meson.build
index 7e22358358..6e75e6d97e 100644
--- a/config/arm/meson.build
+++ b/config/arm/meson.build
@@ -11,7 +11,7 @@ arm_force_default_march = (machine == 'default')
flags_common_default = [
# Accelarate rte_memcpy. Be sure to run unit test (memcpy_perf_autotest)
# to determine the best threshold in code. Refer to notes in source file
- # (lib/librte_eal/common/include/arch/arm/rte_memcpy_64.h) for more info.
+ # (lib/librte_eal/arm/include/rte_memcpy_64.h) for more info.
['RTE_ARCH_ARM64_MEMCPY', false],
# ['RTE_ARM64_MEMCPY_ALIGNED_THRESHOLD', 2048],
# ['RTE_ARM64_MEMCPY_UNALIGNED_THRESHOLD', 512],
diff --git a/config/common_armv8a_linux b/config/common_armv8a_linux
index 020f8f006c..e942657a0f 100644
--- a/config/common_armv8a_linux
+++ b/config/common_armv8a_linux
@@ -21,7 +21,7 @@ CONFIG_RTE_USE_C11_MEM_MODEL=y
# Accelarate rte_memcpy. Be sure to run unit test (memcpy_perf_autotest)
# to determine the best threshold in code. Refer to notes in source file
-# (lib/librte_eal/common/include/arch/arm/rte_memcpy_64.h) for more info.
+# (lib/librte_eal/arm/include/rte_memcpy_64.h) for more info.
CONFIG_RTE_ARCH_ARM64_MEMCPY=n
#CONFIG_RTE_ARM64_MEMCPY_ALIGNED_THRESHOLD=2048
#CONFIG_RTE_ARM64_MEMCPY_UNALIGNED_THRESHOLD=512
diff --git a/config/meson.build b/config/meson.build
index abedd76f23..58421342b1 100644
--- a/config/meson.build
+++ b/config/meson.build
@@ -234,7 +234,7 @@ if host_machine.cpu_family().startswith('x86')
elif host_machine.cpu_family().startswith('arm') or host_machine.cpu_family().startswith('aarch')
arch_subdir = 'arm'
elif host_machine.cpu_family().startswith('ppc')
- arch_subdir = 'ppc_64'
+ arch_subdir = 'ppc'
endif
subdir(arch_subdir)
dpdk_conf.set('RTE_COMPILE_TIME_CPUFLAGS', ','.join(compile_time_cpuflags))
diff --git a/devtools/build-tags.sh b/devtools/build-tags.sh
index dc037f92b3..276fff647b 100755
--- a/devtools/build-tags.sh
+++ b/devtools/build-tags.sh
@@ -79,29 +79,26 @@ bsd_sources()
arm_common()
{
- find_sources "lib/librte_eal/common/arch/arm" '*.[chS]'
find_sources "$source_dirs" '*neon*.[chS]'
}
arm_32_sources()
{
arm_common
- find_sources "lib/librte_eal/common/include/arch/arm" '*.[chS]' \
+ find_sources "lib/librte_eal/arm" '*.[chS]' \
"$skip_64b_files"
}
arm_64_sources()
{
arm_common
- find_sources "lib/librte_eal/common/include/arch/arm" '*.[chS]' \
+ find_sources "lib/librte_eal/arm" '*.[chS]' \
"$skip_32b_files"
find_sources "$source_dirs" '*arm64.[chS]'
}
x86_common()
{
- find_sources "lib/librte_eal/common/arch/x86" '*.[chS]'
-
find_sources "examples/performance-thread/common/arch/x86" '*.[chS]'
find_sources "$source_dirs" '*_sse*.[chS]'
find_sources "$source_dirs" '*_avx*.[chS]'
@@ -111,21 +108,20 @@ x86_common()
x86_32_sources()
{
x86_common
- find_sources "lib/librte_eal/common/include/arch/x86" '*.[chS]' \
+ find_sources "lib/librte_eal/x86" '*.[chS]' \
"$skip_64b_files"
}
x86_64_sources()
{
x86_common
- find_sources "lib/librte_eal/common/include/arch/x86" '*.[chS]' \
+ find_sources "lib/librte_eal/x86" '*.[chS]' \
"$skip_32b_files"
}
ppc_64_sources()
{
- find_sources "lib/librte_eal/common/arch/ppc_64" '*.[chS]'
- find_sources "lib/librte_eal/common/include/arch/ppc_64" '*.[chS]'
+ find_sources "lib/librte_eal/ppc" '*.[chS]'
find_sources "$source_dirs" '*altivec*.[chS]'
}
diff --git a/doc/api/doxy-api.conf.in b/doc/api/doxy-api.conf.in
index 1c4392eecc..65e8146bef 100644
--- a/doc/api/doxy-api.conf.in
+++ b/doc/api/doxy-api.conf.in
@@ -17,8 +17,8 @@ INPUT = @TOPDIR@/doc/api/doxy-api-index.md \
@TOPDIR@/drivers/net/softnic \
@TOPDIR@/drivers/raw/dpaa2_cmdif \
@TOPDIR@/drivers/raw/dpaa2_qdma \
- @TOPDIR@/lib/librte_eal/common/include \
- @TOPDIR@/lib/librte_eal/common/include/generic \
+ @TOPDIR@/lib/librte_eal/include \
+ @TOPDIR@/lib/librte_eal/include/generic \
@TOPDIR@/lib/librte_acl \
@TOPDIR@/lib/librte_bbdev \
@TOPDIR@/lib/librte_bitratestats \
diff --git a/doc/guides/rel_notes/known_issues.rst b/doc/guides/rel_notes/known_issues.rst
index 68c3d22bea..de0782136d 100644
--- a/doc/guides/rel_notes/known_issues.rst
+++ b/doc/guides/rel_notes/known_issues.rst
@@ -127,7 +127,7 @@ HPET timers do not work on the Osage customer reference platform
work correctly, provided the BIOS supports HPET.
**Driver/Module**:
- ``lib/librte_eal/common/include/rte_cycles.h``
+ ``lib/librte_eal/include/rte_cycles.h``
Not all variants of supported NIC types have been used in testing
diff --git a/drivers/bus/dpaa/Makefile b/drivers/bus/dpaa/Makefile
index cd1093f744..9e1a31bb75 100644
--- a/drivers/bus/dpaa/Makefile
+++ b/drivers/bus/dpaa/Makefile
@@ -18,7 +18,7 @@ CFLAGS += -I$(RTE_BUS_DPAA)/
CFLAGS += -I$(RTE_BUS_DPAA)/include
CFLAGS += -I$(RTE_BUS_DPAA)/base/qbman
CFLAGS += -I$(RTE_SDK)/drivers/common/dpaax
-CFLAGS += -I$(RTE_SDK)/lib/librte_eal/common/include
+CFLAGS += -I$(RTE_SDK)/lib/librte_eal/include
# versioning export map
EXPORT_MAP := rte_bus_dpaa_version.map
diff --git a/drivers/common/mlx5/meson.build b/drivers/common/mlx5/meson.build
index 141739fd6f..f671710714 100644
--- a/drivers/common/mlx5/meson.build
+++ b/drivers/common/mlx5/meson.build
@@ -203,7 +203,7 @@ if dlopen_ibverbs
dlopen_install_dir = [ eal_pmd_path + '-glue' ]
dlopen_includes = [global_inc]
dlopen_includes += include_directories(
- '../../../lib/librte_eal/common/include/generic',
+ '../../../lib/librte_eal/include/generic',
)
shared_lib = shared_library(
dlopen_lib_name,
diff --git a/drivers/crypto/caam_jr/Makefile b/drivers/crypto/caam_jr/Makefile
index 1b1f25a2a2..db17294395 100644
--- a/drivers/crypto/caam_jr/Makefile
+++ b/drivers/crypto/caam_jr/Makefile
@@ -20,7 +20,7 @@ CFLAGS += -I$(RTE_SDK)/drivers/bus/dpaa/include
CFLAGS += -I$(RTE_SDK)/drivers/common/dpaax
CFLAGS += -I$(RTE_SDK)/drivers/common/dpaax/caamflib/
CFLAGS += -I$(RTE_SDK)/drivers/crypto/caam_jr
-CFLAGS += -I$(RTE_SDK)/lib/librte_eal/common/include
+CFLAGS += -I$(RTE_SDK)/lib/librte_eal/include
# versioning export map
EXPORT_MAP := rte_pmd_caam_jr_version.map
diff --git a/drivers/crypto/dpaa_sec/Makefile b/drivers/crypto/dpaa_sec/Makefile
index fbfd775855..13a5ff20cf 100644
--- a/drivers/crypto/dpaa_sec/Makefile
+++ b/drivers/crypto/dpaa_sec/Makefile
@@ -20,7 +20,7 @@ CFLAGS += -I$(RTE_SDK)/drivers/bus/dpaa/base/qbman
CFLAGS += -I$(RTE_SDK)/drivers/common/dpaax
CFLAGS += -I$(RTE_SDK)/drivers/crypto/dpaa_sec/
CFLAGS += -I$(RTE_SDK)/drivers/common/dpaax/caamflib/
-CFLAGS += -I$(RTE_SDK)/lib/librte_eal/common/include
+CFLAGS += -I$(RTE_SDK)/lib/librte_eal/include
LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool -lrte_ring
LDLIBS += -lrte_cryptodev
diff --git a/drivers/event/dpaa/Makefile b/drivers/event/dpaa/Makefile
index 2f53efdf9e..15ffc157f8 100644
--- a/drivers/event/dpaa/Makefile
+++ b/drivers/event/dpaa/Makefile
@@ -20,7 +20,7 @@ CFLAGS += -I$(RTE_SDK)/drivers/bus/dpaa
CFLAGS += -I$(RTE_SDK)/drivers/bus/dpaa/include/
CFLAGS += -I$(RTE_SDK)/drivers/mempool/dpaa
CFLAGS += -I$(RTE_SDK)/drivers/common/dpaax
-CFLAGS += -I$(RTE_SDK)/lib/librte_eal/common/include
+CFLAGS += -I$(RTE_SDK)/lib/librte_eal/include
LDLIBS += -lrte_pmd_dpaa_sec
CFLAGS += -I$(RTE_SDK)/drivers/crypto/dpaa_sec
diff --git a/drivers/net/dpaa/Makefile b/drivers/net/dpaa/Makefile
index 8e049b2a0b..f63c9bf540 100644
--- a/drivers/net/dpaa/Makefile
+++ b/drivers/net/dpaa/Makefile
@@ -21,7 +21,7 @@ CFLAGS += -I$(RTE_SDK)/drivers/bus/dpaa/base/qbman
CFLAGS += -I$(RTE_SDK)/drivers/mempool/dpaa
CFLAGS += -I$(RTE_SDK)/drivers/common/dpaax
CFLAGS += -I$(RTE_SDK)/drivers/event/dpaa
-CFLAGS += -I$(RTE_SDK)/lib/librte_eal/common/include
+CFLAGS += -I$(RTE_SDK)/lib/librte_eal/include
EXPORT_MAP := rte_pmd_dpaa_version.map
diff --git a/drivers/net/virtio/meson.build b/drivers/net/virtio/meson.build
index 04c7fdf25d..5e7ca855c8 100644
--- a/drivers/net/virtio/meson.build
+++ b/drivers/net/virtio/meson.build
@@ -11,7 +11,7 @@ deps += ['kvargs', 'bus_pci']
if arch_subdir == 'x86'
sources += files('virtio_rxtx_simple_sse.c')
-elif arch_subdir == 'ppc_64'
+elif arch_subdir == 'ppc'
sources += files('virtio_rxtx_simple_altivec.c')
elif arch_subdir == 'arm' and host_machine.cpu_family().startswith('aarch64')
sources += files('virtio_rxtx_simple_neon.c')
diff --git a/kernel/linux/igb_uio/meson.build b/kernel/linux/igb_uio/meson.build
index fac404f078..80540aecee 100644
--- a/kernel/linux/igb_uio/meson.build
+++ b/kernel/linux/igb_uio/meson.build
@@ -12,7 +12,7 @@ custom_target('igb_uio',
'M=' + meson.current_build_dir(),
'src=' + meson.current_source_dir(),
'EXTRA_CFLAGS=-I' + meson.current_source_dir() +
- '/../../../lib/librte_eal/common/include',
+ '/../../../lib/librte_eal/include',
'modules'],
depends: mkfile,
install: true,
diff --git a/kernel/linux/kni/meson.build b/kernel/linux/kni/meson.build
index f93e97fa09..d696347f22 100644
--- a/kernel/linux/kni/meson.build
+++ b/kernel/linux/kni/meson.build
@@ -17,8 +17,8 @@ custom_target('rte_kni',
'M=' + meson.current_build_dir(),
'src=' + meson.current_source_dir(),
'MODULE_CFLAGS=-include ' + meson.source_root() + '/config/rte_config.h' +
- ' -I' + meson.source_root() + '/lib/librte_eal/common/include' +
- ' -I' + meson.source_root() + '/lib/librte_eal/linux/eal/include' +
+ ' -I' + meson.source_root() + '/lib/librte_eal/include' +
+ ' -I' + meson.source_root() + '/lib/librte_eal/linux/include' +
' -I' + meson.build_root() +
' -I' + meson.current_source_dir(),
'modules'],
diff --git a/lib/librte_cfgfile/Makefile b/lib/librte_cfgfile/Makefile
index d3b08420ff..7c10a4e56c 100644
--- a/lib/librte_cfgfile/Makefile
+++ b/lib/librte_cfgfile/Makefile
@@ -10,7 +10,7 @@ LIB = librte_cfgfile.a
CFLAGS += -O3
CFLAGS += $(WERROR_FLAGS)
-CFLAGS += -I$(SRCDIR)/../librte_eal/common/include
+CFLAGS += -I$(SRCDIR)/../librte_eal/include
LDLIBS += -lrte_eal
EXPORT_MAP := rte_cfgfile_version.map
diff --git a/lib/librte_eal/Makefile b/lib/librte_eal/Makefile
index 86434f5b31..2fda40d230 100644
--- a/lib/librte_eal/Makefile
+++ b/lib/librte_eal/Makefile
@@ -3,10 +3,10 @@
include $(RTE_SDK)/mk/rte.vars.mk
-DIRS-y += common
+DIRS-y += include
DIRS-$(CONFIG_RTE_EXEC_ENV_LINUX) += linux
-DEPDIRS-linux := common
+DEPDIRS-linux := include
DIRS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += freebsd
-DEPDIRS-freebsd := common
+DEPDIRS-freebsd := include
include $(RTE_SDK)/mk/rte.subdir.mk
diff --git a/lib/librte_eal/common/include/arch/arm/meson.build b/lib/librte_eal/arm/include/meson.build
similarity index 100%
rename from lib/librte_eal/common/include/arch/arm/meson.build
rename to lib/librte_eal/arm/include/meson.build
diff --git a/lib/librte_eal/common/include/arch/arm/rte_atomic.h b/lib/librte_eal/arm/include/rte_atomic.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/arm/rte_atomic.h
rename to lib/librte_eal/arm/include/rte_atomic.h
diff --git a/lib/librte_eal/common/include/arch/arm/rte_atomic_32.h b/lib/librte_eal/arm/include/rte_atomic_32.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/arm/rte_atomic_32.h
rename to lib/librte_eal/arm/include/rte_atomic_32.h
diff --git a/lib/librte_eal/common/include/arch/arm/rte_atomic_64.h b/lib/librte_eal/arm/include/rte_atomic_64.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/arm/rte_atomic_64.h
rename to lib/librte_eal/arm/include/rte_atomic_64.h
diff --git a/lib/librte_eal/common/include/arch/arm/rte_byteorder.h b/lib/librte_eal/arm/include/rte_byteorder.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/arm/rte_byteorder.h
rename to lib/librte_eal/arm/include/rte_byteorder.h
diff --git a/lib/librte_eal/common/include/arch/arm/rte_cpuflags.h b/lib/librte_eal/arm/include/rte_cpuflags.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/arm/rte_cpuflags.h
rename to lib/librte_eal/arm/include/rte_cpuflags.h
diff --git a/lib/librte_eal/common/include/arch/arm/rte_cpuflags_32.h b/lib/librte_eal/arm/include/rte_cpuflags_32.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/arm/rte_cpuflags_32.h
rename to lib/librte_eal/arm/include/rte_cpuflags_32.h
diff --git a/lib/librte_eal/common/include/arch/arm/rte_cpuflags_64.h b/lib/librte_eal/arm/include/rte_cpuflags_64.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/arm/rte_cpuflags_64.h
rename to lib/librte_eal/arm/include/rte_cpuflags_64.h
diff --git a/lib/librte_eal/common/include/arch/arm/rte_cycles.h b/lib/librte_eal/arm/include/rte_cycles.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/arm/rte_cycles.h
rename to lib/librte_eal/arm/include/rte_cycles.h
diff --git a/lib/librte_eal/common/include/arch/arm/rte_cycles_32.h b/lib/librte_eal/arm/include/rte_cycles_32.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/arm/rte_cycles_32.h
rename to lib/librte_eal/arm/include/rte_cycles_32.h
diff --git a/lib/librte_eal/common/include/arch/arm/rte_cycles_64.h b/lib/librte_eal/arm/include/rte_cycles_64.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/arm/rte_cycles_64.h
rename to lib/librte_eal/arm/include/rte_cycles_64.h
diff --git a/lib/librte_eal/common/include/arch/arm/rte_io.h b/lib/librte_eal/arm/include/rte_io.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/arm/rte_io.h
rename to lib/librte_eal/arm/include/rte_io.h
diff --git a/lib/librte_eal/common/include/arch/arm/rte_io_64.h b/lib/librte_eal/arm/include/rte_io_64.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/arm/rte_io_64.h
rename to lib/librte_eal/arm/include/rte_io_64.h
diff --git a/lib/librte_eal/common/include/arch/arm/rte_mcslock.h b/lib/librte_eal/arm/include/rte_mcslock.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/arm/rte_mcslock.h
rename to lib/librte_eal/arm/include/rte_mcslock.h
diff --git a/lib/librte_eal/common/include/arch/arm/rte_memcpy.h b/lib/librte_eal/arm/include/rte_memcpy.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/arm/rte_memcpy.h
rename to lib/librte_eal/arm/include/rte_memcpy.h
diff --git a/lib/librte_eal/common/include/arch/arm/rte_memcpy_32.h b/lib/librte_eal/arm/include/rte_memcpy_32.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/arm/rte_memcpy_32.h
rename to lib/librte_eal/arm/include/rte_memcpy_32.h
diff --git a/lib/librte_eal/common/include/arch/arm/rte_memcpy_64.h b/lib/librte_eal/arm/include/rte_memcpy_64.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/arm/rte_memcpy_64.h
rename to lib/librte_eal/arm/include/rte_memcpy_64.h
diff --git a/lib/librte_eal/common/include/arch/arm/rte_pause.h b/lib/librte_eal/arm/include/rte_pause.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/arm/rte_pause.h
rename to lib/librte_eal/arm/include/rte_pause.h
diff --git a/lib/librte_eal/common/include/arch/arm/rte_pause_32.h b/lib/librte_eal/arm/include/rte_pause_32.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/arm/rte_pause_32.h
rename to lib/librte_eal/arm/include/rte_pause_32.h
diff --git a/lib/librte_eal/common/include/arch/arm/rte_pause_64.h b/lib/librte_eal/arm/include/rte_pause_64.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/arm/rte_pause_64.h
rename to lib/librte_eal/arm/include/rte_pause_64.h
diff --git a/lib/librte_eal/common/include/arch/arm/rte_prefetch.h b/lib/librte_eal/arm/include/rte_prefetch.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/arm/rte_prefetch.h
rename to lib/librte_eal/arm/include/rte_prefetch.h
diff --git a/lib/librte_eal/common/include/arch/arm/rte_prefetch_32.h b/lib/librte_eal/arm/include/rte_prefetch_32.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/arm/rte_prefetch_32.h
rename to lib/librte_eal/arm/include/rte_prefetch_32.h
diff --git a/lib/librte_eal/common/include/arch/arm/rte_prefetch_64.h b/lib/librte_eal/arm/include/rte_prefetch_64.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/arm/rte_prefetch_64.h
rename to lib/librte_eal/arm/include/rte_prefetch_64.h
diff --git a/lib/librte_eal/common/include/arch/arm/rte_rwlock.h b/lib/librte_eal/arm/include/rte_rwlock.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/arm/rte_rwlock.h
rename to lib/librte_eal/arm/include/rte_rwlock.h
diff --git a/lib/librte_eal/common/include/arch/arm/rte_spinlock.h b/lib/librte_eal/arm/include/rte_spinlock.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/arm/rte_spinlock.h
rename to lib/librte_eal/arm/include/rte_spinlock.h
diff --git a/lib/librte_eal/common/include/arch/arm/rte_ticketlock.h b/lib/librte_eal/arm/include/rte_ticketlock.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/arm/rte_ticketlock.h
rename to lib/librte_eal/arm/include/rte_ticketlock.h
diff --git a/lib/librte_eal/common/include/arch/arm/rte_vect.h b/lib/librte_eal/arm/include/rte_vect.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/arm/rte_vect.h
rename to lib/librte_eal/arm/include/rte_vect.h
diff --git a/lib/librte_eal/arm/meson.build b/lib/librte_eal/arm/meson.build
new file mode 100644
index 0000000000..883cdd8060
--- /dev/null
+++ b/lib/librte_eal/arm/meson.build
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2017 Intel Corporation.
+
+includes += include_directories('include')
+
+headers += files(
+)
+
+sources += files(
+ 'rte_cpuflags.c',
+ 'rte_cycles.c',
+ 'rte_hypervisor.c',
+)
diff --git a/lib/librte_eal/common/arch/arm/rte_cpuflags.c b/lib/librte_eal/arm/rte_cpuflags.c
similarity index 100%
rename from lib/librte_eal/common/arch/arm/rte_cpuflags.c
rename to lib/librte_eal/arm/rte_cpuflags.c
diff --git a/lib/librte_eal/common/arch/arm/rte_cycles.c b/lib/librte_eal/arm/rte_cycles.c
similarity index 100%
rename from lib/librte_eal/common/arch/arm/rte_cycles.c
rename to lib/librte_eal/arm/rte_cycles.c
diff --git a/lib/librte_eal/common/arch/arm/rte_hypervisor.c b/lib/librte_eal/arm/rte_hypervisor.c
similarity index 100%
rename from lib/librte_eal/common/arch/arm/rte_hypervisor.c
rename to lib/librte_eal/arm/rte_hypervisor.c
diff --git a/lib/librte_eal/common/Makefile b/lib/librte_eal/common/Makefile
deleted file mode 100644
index c2c6d92cd3..0000000000
--- a/lib/librte_eal/common/Makefile
+++ /dev/null
@@ -1,38 +0,0 @@
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2010-2014 Intel Corporation
-
-include $(RTE_SDK)/mk/rte.vars.mk
-
-INC := rte_branch_prediction.h rte_common.h rte_compat.h
-INC += rte_function_versioning.h
-INC += rte_debug.h rte_eal.h rte_eal_interrupts.h
-INC += rte_errno.h rte_launch.h rte_lcore.h
-INC += rte_log.h rte_memory.h rte_memzone.h
-INC += rte_per_lcore.h rte_random.h
-INC += rte_tailq.h rte_interrupts.h rte_alarm.h
-INC += rte_string_fns.h rte_version.h
-INC += rte_eal_memconfig.h
-INC += rte_hexdump.h rte_devargs.h rte_bus.h rte_dev.h rte_class.h
-INC += rte_option.h
-INC += rte_pci_dev_feature_defs.h rte_pci_dev_features.h
-INC += rte_malloc.h rte_keepalive.h rte_time.h
-INC += rte_service.h rte_service_component.h
-INC += rte_bitmap.h rte_vfio.h rte_hypervisor.h rte_test.h
-INC += rte_reciprocal.h rte_fbarray.h rte_uuid.h
-
-GENERIC_INC := rte_atomic.h rte_byteorder.h rte_cycles.h rte_prefetch.h
-GENERIC_INC += rte_memcpy.h rte_cpuflags.h
-GENERIC_INC += rte_mcslock.h rte_spinlock.h rte_rwlock.h rte_ticketlock.h
-GENERIC_INC += rte_vect.h rte_pause.h rte_io.h
-
-# defined in mk/arch/$(RTE_ARCH)/rte.vars.mk
-ARCH_DIR ?= $(RTE_ARCH)
-ARCH_INC := $(sort $(notdir $(wildcard $(RTE_SDK)/lib/librte_eal/common/include/arch/$(ARCH_DIR)/*.h)))
-
-SYMLINK-$(CONFIG_RTE_LIBRTE_EAL)-include := $(addprefix include/,$(INC))
-SYMLINK-$(CONFIG_RTE_LIBRTE_EAL)-include += \
- $(addprefix include/arch/$(ARCH_DIR)/,$(ARCH_INC))
-SYMLINK-$(CONFIG_RTE_LIBRTE_EAL)-include/generic := \
- $(addprefix include/generic/,$(GENERIC_INC))
-
-include $(RTE_SDK)/mk/rte.install.mk
diff --git a/lib/librte_eal/common/arch/arm/meson.build b/lib/librte_eal/common/arch/arm/meson.build
deleted file mode 100644
index 79731e1a2c..0000000000
--- a/lib/librte_eal/common/arch/arm/meson.build
+++ /dev/null
@@ -1,5 +0,0 @@
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2017 Intel Corporation.
-
-eal_common_arch_sources = files('rte_cpuflags.c',
- 'rte_cycles.c', 'rte_hypervisor.c')
diff --git a/lib/librte_eal/common/arch/ppc_64/meson.build b/lib/librte_eal/common/arch/ppc_64/meson.build
deleted file mode 100644
index 40b3dc533a..0000000000
--- a/lib/librte_eal/common/arch/ppc_64/meson.build
+++ /dev/null
@@ -1,5 +0,0 @@
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2018 Luca Boccassi <bluca@debian.org>
-
-eal_common_arch_sources = files('rte_cpuflags.c',
- 'rte_cycles.c', 'rte_hypervisor.c')
diff --git a/lib/librte_eal/common/arch/x86/meson.build b/lib/librte_eal/common/arch/x86/meson.build
deleted file mode 100644
index 14bf204c6f..0000000000
--- a/lib/librte_eal/common/arch/x86/meson.build
+++ /dev/null
@@ -1,5 +0,0 @@
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2017 Intel Corporation
-
-eal_common_arch_sources = files('rte_spinlock.c', 'rte_cpuflags.c',
- 'rte_cycles.c', 'rte_hypervisor.c')
diff --git a/lib/librte_eal/common/meson.build b/lib/librte_eal/common/meson.build
index 2b97715a20..02d9280cc3 100644
--- a/lib/librte_eal/common/meson.build
+++ b/lib/librte_eal/common/meson.build
@@ -1,11 +1,25 @@
# SPDX-License-Identifier: BSD-3-Clause
# Copyright(c) 2017 Intel Corporation
-eal_inc += include_directories('.', 'include',
- join_paths('include/arch', arch_subdir))
+includes += include_directories('.')
-common_objs = []
-common_sources = files(
+if is_windows
+ sources += files(
+ 'eal_common_bus.c',
+ 'eal_common_class.c',
+ 'eal_common_devargs.c',
+ 'eal_common_errno.c',
+ 'eal_common_launch.c',
+ 'eal_common_lcore.c',
+ 'eal_common_log.c',
+ 'eal_common_options.c',
+ 'eal_common_thread.c',
+ 'rte_option.c',
+ )
+ subdir_done()
+endif
+
+sources += files(
'eal_common_bus.c',
'eal_common_cpuflags.c',
'eal_common_class.c',
@@ -38,73 +52,5 @@ common_sources = files(
'rte_option.c',
'rte_random.c',
'rte_reciprocal.c',
- 'rte_service.c'
+ 'rte_service.c',
)
-
-# get architecture specific sources and objs
-eal_common_arch_sources = []
-eal_common_arch_objs = []
-subdir(join_paths('arch', arch_subdir))
-common_sources += eal_common_arch_sources
-common_objs += eal_common_arch_objs
-
-common_headers = files(
- 'include/rte_alarm.h',
- 'include/rte_branch_prediction.h',
- 'include/rte_bus.h',
- 'include/rte_bitmap.h',
- 'include/rte_class.h',
- 'include/rte_common.h',
- 'include/rte_compat.h',
- 'include/rte_debug.h',
- 'include/rte_devargs.h',
- 'include/rte_dev.h',
- 'include/rte_eal.h',
- 'include/rte_eal_memconfig.h',
- 'include/rte_eal_interrupts.h',
- 'include/rte_errno.h',
- 'include/rte_fbarray.h',
- 'include/rte_hexdump.h',
- 'include/rte_hypervisor.h',
- 'include/rte_interrupts.h',
- 'include/rte_keepalive.h',
- 'include/rte_launch.h',
- 'include/rte_lcore.h',
- 'include/rte_log.h',
- 'include/rte_malloc.h',
- 'include/rte_memory.h',
- 'include/rte_memzone.h',
- 'include/rte_option.h',
- 'include/rte_pci_dev_feature_defs.h',
- 'include/rte_pci_dev_features.h',
- 'include/rte_per_lcore.h',
- 'include/rte_random.h',
- 'include/rte_reciprocal.h',
- 'include/rte_service.h',
- 'include/rte_service_component.h',
- 'include/rte_string_fns.h',
- 'include/rte_tailq.h',
- 'include/rte_time.h',
- 'include/rte_uuid.h',
- 'include/rte_version.h',
- 'include/rte_vfio.h')
-
-# special case install the generic headers, since they go in a subdir
-generic_headers = files(
- 'include/generic/rte_atomic.h',
- 'include/generic/rte_byteorder.h',
- 'include/generic/rte_cpuflags.h',
- 'include/generic/rte_cycles.h',
- 'include/generic/rte_io.h',
- 'include/generic/rte_mcslock.h',
- 'include/generic/rte_memcpy.h',
- 'include/generic/rte_pause.h',
- 'include/generic/rte_prefetch.h',
- 'include/generic/rte_rwlock.h',
- 'include/generic/rte_spinlock.h',
- 'include/generic/rte_ticketlock.h',
- 'include/generic/rte_vect.h')
-install_headers(generic_headers, subdir: 'generic')
-
-# get and install the architecture specific headers
-subdir(join_paths('include/arch', arch_subdir))
diff --git a/lib/librte_eal/common/rte_service.c b/lib/librte_eal/common/rte_service.c
index b0b78baabd..70d17a5d79 100644
--- a/lib/librte_eal/common/rte_service.c
+++ b/lib/librte_eal/common/rte_service.c
@@ -10,7 +10,7 @@
#include <rte_compat.h>
#include <rte_service.h>
-#include "include/rte_service_component.h"
+#include <rte_service_component.h>
#include <rte_eal.h>
#include <rte_lcore.h>
diff --git a/lib/librte_eal/freebsd/BSDmakefile.meson b/lib/librte_eal/freebsd/BSDmakefile.meson
deleted file mode 100644
index 53c4e79c61..0000000000
--- a/lib/librte_eal/freebsd/BSDmakefile.meson
+++ /dev/null
@@ -1,14 +0,0 @@
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2017 Intel Corporation.
-
-# makefile for building kernel modules using meson
-# takes parameters from the environment
-
-# source file is passed via KMOD_SRC as full path, we only use final
-# component of it, as VPATH is used to find actual file, so as to
-# have the .o files placed in the build, not source directory
-VPATH = ${KMOD_SRC:H}
-SRCS = ${KMOD_SRC:T} device_if.h bus_if.h pci_if.h
-CFLAGS += $(KMOD_CFLAGS)
-
-.include <bsd.kmod.mk>
diff --git a/lib/librte_eal/freebsd/Makefile b/lib/librte_eal/freebsd/Makefile
index fc42058b6b..265edb06f6 100644
--- a/lib/librte_eal/freebsd/Makefile
+++ b/lib/librte_eal/freebsd/Makefile
@@ -1,8 +1,92 @@
# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2010-2014 Intel Corporation
+# Copyright(c) 2010-2019 Intel Corporation
include $(RTE_SDK)/mk/rte.vars.mk
-DIRS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal
+LIB = librte_eal.a
-include $(RTE_SDK)/mk/rte.subdir.mk
+ARCH_DIR ?= $(RTE_ARCH)
+VPATH += $(RTE_SDK)/lib/librte_eal/$(ARCH_DIR)
+VPATH += $(RTE_SDK)/lib/librte_eal/common
+
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+CFLAGS += -I$(SRCDIR)/include
+CFLAGS += -I$(RTE_SDK)/lib/librte_eal/common
+CFLAGS += $(WERROR_FLAGS) -O3
+
+LDLIBS += -lexecinfo
+LDLIBS += -lpthread
+LDLIBS += -lgcc_s
+LDLIBS += -lrte_kvargs
+
+EXPORT_MAP := ../rte_eal_version.map
+
+# specific to freebsd exec-env
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) := eal.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_cpuflags.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_memory.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_hugepage_info.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_thread.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_debug.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_memalloc.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_lcore.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_timer.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_interrupts.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_alarm.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_dev.c
+
+# from common dir
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_common_lcore.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_common_timer.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_common_memzone.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_common_log.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_common_launch.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_common_mcfg.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_common_memalloc.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_common_memory.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_common_tailqs.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_common_errno.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_common_cpuflags.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_common_hypervisor.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_common_string_fns.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_common_hexdump.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_common_devargs.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_common_class.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_common_bus.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_common_dev.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_common_options.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_common_thread.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_common_proc.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_common_fbarray.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_common_uuid.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += rte_malloc.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += hotplug_mp.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += malloc_elem.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += malloc_heap.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += malloc_mp.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += rte_keepalive.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += rte_option.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += rte_service.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += rte_random.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += rte_reciprocal.c
+
+# from arch dir
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += rte_cpuflags.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += rte_hypervisor.c
+SRCS-$(CONFIG_RTE_ARCH_X86) += rte_spinlock.c
+SRCS-y += rte_cycles.c
+
+CFLAGS_eal_common_cpuflags.o := $(CPUFLAGS_LIST)
+
+# workaround for a gcc bug with noreturn attribute
+# http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12603
+ifeq ($(CONFIG_RTE_TOOLCHAIN_GCC),y)
+CFLAGS_eal_thread.o += -Wno-return-type
+CFLAGS_eal_hpet.o += -Wno-return-type
+endif
+
+INC := rte_os.h
+
+SYMLINK-$(CONFIG_RTE_EXEC_ENV_FREEBSD)-include := $(addprefix include/,$(INC))
+
+include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/lib/librte_eal/freebsd/eal/eal.c b/lib/librte_eal/freebsd/eal.c
similarity index 100%
rename from lib/librte_eal/freebsd/eal/eal.c
rename to lib/librte_eal/freebsd/eal.c
diff --git a/lib/librte_eal/freebsd/eal/Makefile b/lib/librte_eal/freebsd/eal/Makefile
deleted file mode 100644
index b160b57906..0000000000
--- a/lib/librte_eal/freebsd/eal/Makefile
+++ /dev/null
@@ -1,93 +0,0 @@
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2010-2019 Intel Corporation
-
-include $(RTE_SDK)/mk/rte.vars.mk
-
-LIB = librte_eal.a
-
-ARCH_DIR ?= $(RTE_ARCH)
-VPATH += $(RTE_SDK)/lib/librte_eal/common
-VPATH += $(RTE_SDK)/lib/librte_eal/common/arch/$(ARCH_DIR)
-
-CFLAGS += -DALLOW_EXPERIMENTAL_API
-CFLAGS += -I$(SRCDIR)/include
-CFLAGS += -I$(RTE_SDK)/lib/librte_eal/common
-CFLAGS += -I$(RTE_SDK)/lib/librte_eal/common/include
-CFLAGS += $(WERROR_FLAGS) -O3
-
-LDLIBS += -lexecinfo
-LDLIBS += -lpthread
-LDLIBS += -lgcc_s
-LDLIBS += -lrte_kvargs
-
-EXPORT_MAP := ../../rte_eal_version.map
-
-# specific to freebsd exec-env
-SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) := eal.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_cpuflags.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_memory.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_hugepage_info.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_thread.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_debug.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_memalloc.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_lcore.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_timer.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_interrupts.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_alarm.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_dev.c
-
-# from common dir
-SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_common_lcore.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_common_timer.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_common_memzone.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_common_log.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_common_launch.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_common_mcfg.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_common_memalloc.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_common_memory.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_common_tailqs.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_common_errno.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_common_cpuflags.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_common_hypervisor.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_common_string_fns.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_common_hexdump.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_common_devargs.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_common_class.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_common_bus.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_common_dev.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_common_options.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_common_thread.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_common_proc.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_common_fbarray.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_common_uuid.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += rte_malloc.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += hotplug_mp.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += malloc_elem.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += malloc_heap.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += malloc_mp.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += rte_keepalive.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += rte_option.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += rte_service.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += rte_random.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += rte_reciprocal.c
-
-# from arch dir
-SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += rte_cpuflags.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += rte_hypervisor.c
-SRCS-$(CONFIG_RTE_ARCH_X86) += rte_spinlock.c
-SRCS-y += rte_cycles.c
-
-CFLAGS_eal_common_cpuflags.o := $(CPUFLAGS_LIST)
-
-# workaround for a gcc bug with noreturn attribute
-# http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12603
-ifeq ($(CONFIG_RTE_TOOLCHAIN_GCC),y)
-CFLAGS_eal_thread.o += -Wno-return-type
-CFLAGS_eal_hpet.o += -Wno-return-type
-endif
-
-INC := rte_os.h
-
-SYMLINK-$(CONFIG_RTE_EXEC_ENV_FREEBSD)-include := $(addprefix include/,$(INC))
-
-include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/lib/librte_eal/freebsd/eal/eal_alarm.c b/lib/librte_eal/freebsd/eal_alarm.c
similarity index 100%
rename from lib/librte_eal/freebsd/eal/eal_alarm.c
rename to lib/librte_eal/freebsd/eal_alarm.c
diff --git a/lib/librte_eal/freebsd/eal/eal_alarm_private.h b/lib/librte_eal/freebsd/eal_alarm_private.h
similarity index 100%
rename from lib/librte_eal/freebsd/eal/eal_alarm_private.h
rename to lib/librte_eal/freebsd/eal_alarm_private.h
diff --git a/lib/librte_eal/freebsd/eal/eal_cpuflags.c b/lib/librte_eal/freebsd/eal_cpuflags.c
similarity index 100%
rename from lib/librte_eal/freebsd/eal/eal_cpuflags.c
rename to lib/librte_eal/freebsd/eal_cpuflags.c
diff --git a/lib/librte_eal/freebsd/eal/eal_debug.c b/lib/librte_eal/freebsd/eal_debug.c
similarity index 100%
rename from lib/librte_eal/freebsd/eal/eal_debug.c
rename to lib/librte_eal/freebsd/eal_debug.c
diff --git a/lib/librte_eal/freebsd/eal/eal_dev.c b/lib/librte_eal/freebsd/eal_dev.c
similarity index 100%
rename from lib/librte_eal/freebsd/eal/eal_dev.c
rename to lib/librte_eal/freebsd/eal_dev.c
diff --git a/lib/librte_eal/freebsd/eal/eal_hugepage_info.c b/lib/librte_eal/freebsd/eal_hugepage_info.c
similarity index 100%
rename from lib/librte_eal/freebsd/eal/eal_hugepage_info.c
rename to lib/librte_eal/freebsd/eal_hugepage_info.c
diff --git a/lib/librte_eal/freebsd/eal/eal_interrupts.c b/lib/librte_eal/freebsd/eal_interrupts.c
similarity index 100%
rename from lib/librte_eal/freebsd/eal/eal_interrupts.c
rename to lib/librte_eal/freebsd/eal_interrupts.c
diff --git a/lib/librte_eal/freebsd/eal/eal_lcore.c b/lib/librte_eal/freebsd/eal_lcore.c
similarity index 100%
rename from lib/librte_eal/freebsd/eal/eal_lcore.c
rename to lib/librte_eal/freebsd/eal_lcore.c
diff --git a/lib/librte_eal/freebsd/eal/eal_memalloc.c b/lib/librte_eal/freebsd/eal_memalloc.c
similarity index 100%
rename from lib/librte_eal/freebsd/eal/eal_memalloc.c
rename to lib/librte_eal/freebsd/eal_memalloc.c
diff --git a/lib/librte_eal/freebsd/eal/eal_memory.c b/lib/librte_eal/freebsd/eal_memory.c
similarity index 100%
rename from lib/librte_eal/freebsd/eal/eal_memory.c
rename to lib/librte_eal/freebsd/eal_memory.c
diff --git a/lib/librte_eal/freebsd/eal/eal_thread.c b/lib/librte_eal/freebsd/eal_thread.c
similarity index 100%
rename from lib/librte_eal/freebsd/eal/eal_thread.c
rename to lib/librte_eal/freebsd/eal_thread.c
diff --git a/lib/librte_eal/freebsd/eal/eal_timer.c b/lib/librte_eal/freebsd/eal_timer.c
similarity index 100%
rename from lib/librte_eal/freebsd/eal/eal_timer.c
rename to lib/librte_eal/freebsd/eal_timer.c
diff --git a/lib/librte_eal/freebsd/eal/include/rte_os.h b/lib/librte_eal/freebsd/include/rte_os.h
similarity index 100%
rename from lib/librte_eal/freebsd/eal/include/rte_os.h
rename to lib/librte_eal/freebsd/include/rte_os.h
diff --git a/lib/librte_eal/freebsd/eal/meson.build b/lib/librte_eal/freebsd/meson.build
similarity index 77%
rename from lib/librte_eal/freebsd/eal/meson.build
rename to lib/librte_eal/freebsd/meson.build
index 1426f7e5f1..b423d970f9 100644
--- a/lib/librte_eal/freebsd/eal/meson.build
+++ b/lib/librte_eal/freebsd/meson.build
@@ -1,11 +1,13 @@
# SPDX-License-Identifier: BSD-3-Clause
# Copyright(c) 2017 Intel Corporation
-env_objs = []
-env_headers = files(
+includes += include_directories('include')
+
+headers += files(
'include/rte_os.h',
)
-env_sources = files('eal_alarm.c',
+
+sources += files('eal_alarm.c',
'eal_cpuflags.c',
'eal_debug.c',
'eal_hugepage_info.c',
diff --git a/lib/librte_eal/common/include/generic/rte_atomic.h b/lib/librte_eal/include/generic/rte_atomic.h
similarity index 100%
rename from lib/librte_eal/common/include/generic/rte_atomic.h
rename to lib/librte_eal/include/generic/rte_atomic.h
diff --git a/lib/librte_eal/common/include/generic/rte_byteorder.h b/lib/librte_eal/include/generic/rte_byteorder.h
similarity index 100%
rename from lib/librte_eal/common/include/generic/rte_byteorder.h
rename to lib/librte_eal/include/generic/rte_byteorder.h
diff --git a/lib/librte_eal/common/include/generic/rte_cpuflags.h b/lib/librte_eal/include/generic/rte_cpuflags.h
similarity index 100%
rename from lib/librte_eal/common/include/generic/rte_cpuflags.h
rename to lib/librte_eal/include/generic/rte_cpuflags.h
diff --git a/lib/librte_eal/common/include/generic/rte_cycles.h b/lib/librte_eal/include/generic/rte_cycles.h
similarity index 100%
rename from lib/librte_eal/common/include/generic/rte_cycles.h
rename to lib/librte_eal/include/generic/rte_cycles.h
diff --git a/lib/librte_eal/common/include/generic/rte_io.h b/lib/librte_eal/include/generic/rte_io.h
similarity index 100%
rename from lib/librte_eal/common/include/generic/rte_io.h
rename to lib/librte_eal/include/generic/rte_io.h
diff --git a/lib/librte_eal/common/include/generic/rte_mcslock.h b/lib/librte_eal/include/generic/rte_mcslock.h
similarity index 100%
rename from lib/librte_eal/common/include/generic/rte_mcslock.h
rename to lib/librte_eal/include/generic/rte_mcslock.h
diff --git a/lib/librte_eal/common/include/generic/rte_memcpy.h b/lib/librte_eal/include/generic/rte_memcpy.h
similarity index 100%
rename from lib/librte_eal/common/include/generic/rte_memcpy.h
rename to lib/librte_eal/include/generic/rte_memcpy.h
diff --git a/lib/librte_eal/common/include/generic/rte_pause.h b/lib/librte_eal/include/generic/rte_pause.h
similarity index 100%
rename from lib/librte_eal/common/include/generic/rte_pause.h
rename to lib/librte_eal/include/generic/rte_pause.h
diff --git a/lib/librte_eal/common/include/generic/rte_prefetch.h b/lib/librte_eal/include/generic/rte_prefetch.h
similarity index 100%
rename from lib/librte_eal/common/include/generic/rte_prefetch.h
rename to lib/librte_eal/include/generic/rte_prefetch.h
diff --git a/lib/librte_eal/common/include/generic/rte_rwlock.h b/lib/librte_eal/include/generic/rte_rwlock.h
similarity index 100%
rename from lib/librte_eal/common/include/generic/rte_rwlock.h
rename to lib/librte_eal/include/generic/rte_rwlock.h
diff --git a/lib/librte_eal/common/include/generic/rte_spinlock.h b/lib/librte_eal/include/generic/rte_spinlock.h
similarity index 100%
rename from lib/librte_eal/common/include/generic/rte_spinlock.h
rename to lib/librte_eal/include/generic/rte_spinlock.h
diff --git a/lib/librte_eal/common/include/generic/rte_ticketlock.h b/lib/librte_eal/include/generic/rte_ticketlock.h
similarity index 100%
rename from lib/librte_eal/common/include/generic/rte_ticketlock.h
rename to lib/librte_eal/include/generic/rte_ticketlock.h
diff --git a/lib/librte_eal/common/include/generic/rte_vect.h b/lib/librte_eal/include/generic/rte_vect.h
similarity index 100%
rename from lib/librte_eal/common/include/generic/rte_vect.h
rename to lib/librte_eal/include/generic/rte_vect.h
diff --git a/lib/librte_eal/common/include/rte_alarm.h b/lib/librte_eal/include/rte_alarm.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_alarm.h
rename to lib/librte_eal/include/rte_alarm.h
diff --git a/lib/librte_eal/common/include/rte_bitmap.h b/lib/librte_eal/include/rte_bitmap.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_bitmap.h
rename to lib/librte_eal/include/rte_bitmap.h
diff --git a/lib/librte_eal/common/include/rte_branch_prediction.h b/lib/librte_eal/include/rte_branch_prediction.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_branch_prediction.h
rename to lib/librte_eal/include/rte_branch_prediction.h
diff --git a/lib/librte_eal/common/include/rte_bus.h b/lib/librte_eal/include/rte_bus.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_bus.h
rename to lib/librte_eal/include/rte_bus.h
diff --git a/lib/librte_eal/common/include/rte_class.h b/lib/librte_eal/include/rte_class.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_class.h
rename to lib/librte_eal/include/rte_class.h
diff --git a/lib/librte_eal/common/include/rte_common.h b/lib/librte_eal/include/rte_common.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_common.h
rename to lib/librte_eal/include/rte_common.h
diff --git a/lib/librte_eal/common/include/rte_compat.h b/lib/librte_eal/include/rte_compat.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_compat.h
rename to lib/librte_eal/include/rte_compat.h
diff --git a/lib/librte_eal/common/include/rte_debug.h b/lib/librte_eal/include/rte_debug.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_debug.h
rename to lib/librte_eal/include/rte_debug.h
diff --git a/lib/librte_eal/common/include/rte_dev.h b/lib/librte_eal/include/rte_dev.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_dev.h
rename to lib/librte_eal/include/rte_dev.h
diff --git a/lib/librte_eal/common/include/rte_devargs.h b/lib/librte_eal/include/rte_devargs.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_devargs.h
rename to lib/librte_eal/include/rte_devargs.h
diff --git a/lib/librte_eal/common/include/rte_eal.h b/lib/librte_eal/include/rte_eal.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_eal.h
rename to lib/librte_eal/include/rte_eal.h
diff --git a/lib/librte_eal/common/include/rte_eal_interrupts.h b/lib/librte_eal/include/rte_eal_interrupts.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_eal_interrupts.h
rename to lib/librte_eal/include/rte_eal_interrupts.h
diff --git a/lib/librte_eal/common/include/rte_eal_memconfig.h b/lib/librte_eal/include/rte_eal_memconfig.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_eal_memconfig.h
rename to lib/librte_eal/include/rte_eal_memconfig.h
diff --git a/lib/librte_eal/common/include/rte_errno.h b/lib/librte_eal/include/rte_errno.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_errno.h
rename to lib/librte_eal/include/rte_errno.h
diff --git a/lib/librte_eal/common/include/rte_fbarray.h b/lib/librte_eal/include/rte_fbarray.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_fbarray.h
rename to lib/librte_eal/include/rte_fbarray.h
diff --git a/lib/librte_eal/common/include/rte_function_versioning.h b/lib/librte_eal/include/rte_function_versioning.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_function_versioning.h
rename to lib/librte_eal/include/rte_function_versioning.h
diff --git a/lib/librte_eal/common/include/rte_hexdump.h b/lib/librte_eal/include/rte_hexdump.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_hexdump.h
rename to lib/librte_eal/include/rte_hexdump.h
diff --git a/lib/librte_eal/common/include/rte_hypervisor.h b/lib/librte_eal/include/rte_hypervisor.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_hypervisor.h
rename to lib/librte_eal/include/rte_hypervisor.h
diff --git a/lib/librte_eal/common/include/rte_interrupts.h b/lib/librte_eal/include/rte_interrupts.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_interrupts.h
rename to lib/librte_eal/include/rte_interrupts.h
diff --git a/lib/librte_eal/common/include/rte_keepalive.h b/lib/librte_eal/include/rte_keepalive.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_keepalive.h
rename to lib/librte_eal/include/rte_keepalive.h
diff --git a/lib/librte_eal/common/include/rte_launch.h b/lib/librte_eal/include/rte_launch.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_launch.h
rename to lib/librte_eal/include/rte_launch.h
diff --git a/lib/librte_eal/common/include/rte_lcore.h b/lib/librte_eal/include/rte_lcore.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_lcore.h
rename to lib/librte_eal/include/rte_lcore.h
diff --git a/lib/librte_eal/common/include/rte_log.h b/lib/librte_eal/include/rte_log.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_log.h
rename to lib/librte_eal/include/rte_log.h
diff --git a/lib/librte_eal/common/include/rte_malloc.h b/lib/librte_eal/include/rte_malloc.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_malloc.h
rename to lib/librte_eal/include/rte_malloc.h
diff --git a/lib/librte_eal/common/include/rte_memory.h b/lib/librte_eal/include/rte_memory.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_memory.h
rename to lib/librte_eal/include/rte_memory.h
diff --git a/lib/librte_eal/common/include/rte_memzone.h b/lib/librte_eal/include/rte_memzone.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_memzone.h
rename to lib/librte_eal/include/rte_memzone.h
diff --git a/lib/librte_eal/common/include/rte_option.h b/lib/librte_eal/include/rte_option.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_option.h
rename to lib/librte_eal/include/rte_option.h
diff --git a/lib/librte_eal/common/include/rte_pci_dev_feature_defs.h b/lib/librte_eal/include/rte_pci_dev_feature_defs.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_pci_dev_feature_defs.h
rename to lib/librte_eal/include/rte_pci_dev_feature_defs.h
diff --git a/lib/librte_eal/common/include/rte_pci_dev_features.h b/lib/librte_eal/include/rte_pci_dev_features.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_pci_dev_features.h
rename to lib/librte_eal/include/rte_pci_dev_features.h
diff --git a/lib/librte_eal/common/include/rte_per_lcore.h b/lib/librte_eal/include/rte_per_lcore.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_per_lcore.h
rename to lib/librte_eal/include/rte_per_lcore.h
diff --git a/lib/librte_eal/common/include/rte_random.h b/lib/librte_eal/include/rte_random.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_random.h
rename to lib/librte_eal/include/rte_random.h
diff --git a/lib/librte_eal/common/include/rte_reciprocal.h b/lib/librte_eal/include/rte_reciprocal.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_reciprocal.h
rename to lib/librte_eal/include/rte_reciprocal.h
diff --git a/lib/librte_eal/common/include/rte_service.h b/lib/librte_eal/include/rte_service.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_service.h
rename to lib/librte_eal/include/rte_service.h
diff --git a/lib/librte_eal/common/include/rte_service_component.h b/lib/librte_eal/include/rte_service_component.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_service_component.h
rename to lib/librte_eal/include/rte_service_component.h
diff --git a/lib/librte_eal/common/include/rte_string_fns.h b/lib/librte_eal/include/rte_string_fns.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_string_fns.h
rename to lib/librte_eal/include/rte_string_fns.h
diff --git a/lib/librte_eal/common/include/rte_tailq.h b/lib/librte_eal/include/rte_tailq.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_tailq.h
rename to lib/librte_eal/include/rte_tailq.h
diff --git a/lib/librte_eal/common/include/rte_test.h b/lib/librte_eal/include/rte_test.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_test.h
rename to lib/librte_eal/include/rte_test.h
diff --git a/lib/librte_eal/common/include/rte_time.h b/lib/librte_eal/include/rte_time.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_time.h
rename to lib/librte_eal/include/rte_time.h
diff --git a/lib/librte_eal/common/include/rte_uuid.h b/lib/librte_eal/include/rte_uuid.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_uuid.h
rename to lib/librte_eal/include/rte_uuid.h
diff --git a/lib/librte_eal/common/include/rte_version.h b/lib/librte_eal/include/rte_version.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_version.h
rename to lib/librte_eal/include/rte_version.h
diff --git a/lib/librte_eal/common/include/rte_vfio.h b/lib/librte_eal/include/rte_vfio.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_vfio.h
rename to lib/librte_eal/include/rte_vfio.h
diff --git a/lib/librte_eal/linux/Makefile b/lib/librte_eal/linux/Makefile
index 4c68bd61b8..0ed4dc3e10 100644
--- a/lib/librte_eal/linux/Makefile
+++ b/lib/librte_eal/linux/Makefile
@@ -1,11 +1,99 @@
# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2010-2014 Intel Corporation
+# Copyright(c) 2010-2019 Intel Corporation
include $(RTE_SDK)/mk/rte.vars.mk
-DIRS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal
-DEPDIRS-kni := eal
+LIB = librte_eal.a
+
+ARCH_DIR ?= $(RTE_ARCH)
+VPATH += $(RTE_SDK)/lib/librte_eal/$(ARCH_DIR)
+VPATH += $(RTE_SDK)/lib/librte_eal/common
CFLAGS += -DALLOW_EXPERIMENTAL_API
+CFLAGS += -I$(SRCDIR)/include
+CFLAGS += -I$(RTE_SDK)/lib/librte_eal/common
+CFLAGS += $(WERROR_FLAGS) -O3
-include $(RTE_SDK)/mk/rte.subdir.mk
+LDLIBS += -ldl
+LDLIBS += -lpthread
+LDLIBS += -lgcc_s
+LDLIBS += -lrt
+LDLIBS += -lrte_kvargs
+ifeq ($(CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES),y)
+LDLIBS += -lnuma
+endif
+
+EXPORT_MAP := ../rte_eal_version.map
+
+# specific to linux exec-env
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) := eal.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_cpuflags.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_hugepage_info.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_memory.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_thread.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_log.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_vfio.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_vfio_mp_sync.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_memalloc.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_debug.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_lcore.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_timer.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_interrupts.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_alarm.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_dev.c
+
+# from common dir
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_lcore.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_timer.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_memzone.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_log.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_launch.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_mcfg.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_memalloc.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_memory.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_tailqs.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_errno.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_cpuflags.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_hypervisor.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_string_fns.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_hexdump.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_devargs.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_class.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_bus.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_dev.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_options.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_thread.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_proc.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_fbarray.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_uuid.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += rte_malloc.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += hotplug_mp.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += malloc_elem.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += malloc_heap.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += malloc_mp.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += rte_keepalive.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += rte_option.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += rte_service.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += rte_random.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += rte_reciprocal.c
+
+# from arch dir
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += rte_cpuflags.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += rte_hypervisor.c
+SRCS-$(CONFIG_RTE_ARCH_X86) += rte_spinlock.c
+SRCS-y += rte_cycles.c
+
+CFLAGS_eal_common_cpuflags.o := $(CPUFLAGS_LIST)
+
+# workaround for a gcc bug with noreturn attribute
+# http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12603
+ifeq ($(CONFIG_RTE_TOOLCHAIN_GCC),y)
+CFLAGS_eal_thread.o += -Wno-return-type
+endif
+
+INC := rte_kni_common.h
+INC += rte_os.h
+
+SYMLINK-$(CONFIG_RTE_EXEC_ENV_LINUX)-include := $(addprefix include/,$(INC))
+
+include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/lib/librte_eal/linux/eal/eal.c b/lib/librte_eal/linux/eal.c
similarity index 100%
rename from lib/librte_eal/linux/eal/eal.c
rename to lib/librte_eal/linux/eal.c
diff --git a/lib/librte_eal/linux/eal/Makefile b/lib/librte_eal/linux/eal/Makefile
deleted file mode 100644
index e70cf104a4..0000000000
--- a/lib/librte_eal/linux/eal/Makefile
+++ /dev/null
@@ -1,101 +0,0 @@
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2010-2019 Intel Corporation
-
-include $(RTE_SDK)/mk/rte.vars.mk
-
-LIB = librte_eal.a
-
-ARCH_DIR ?= $(RTE_ARCH)
-
-EXPORT_MAP := ../../rte_eal_version.map
-VPATH += $(RTE_SDK)/lib/librte_eal/common/arch/$(ARCH_DIR)
-
-VPATH += $(RTE_SDK)/lib/librte_eal/common
-
-CFLAGS += -DALLOW_EXPERIMENTAL_API
-CFLAGS += -I$(SRCDIR)/include
-CFLAGS += -I$(RTE_SDK)/lib/librte_eal/common
-CFLAGS += -I$(RTE_SDK)/lib/librte_eal/common/include
-CFLAGS += $(WERROR_FLAGS) -O3
-
-LDLIBS += -ldl
-LDLIBS += -lpthread
-LDLIBS += -lgcc_s
-LDLIBS += -lrt
-LDLIBS += -lrte_kvargs
-ifeq ($(CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES),y)
-LDLIBS += -lnuma
-endif
-
-# specific to linux exec-env
-SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) := eal.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_cpuflags.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_hugepage_info.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_memory.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_thread.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_log.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_vfio.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_vfio_mp_sync.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_memalloc.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_debug.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_lcore.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_timer.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_interrupts.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_alarm.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_dev.c
-
-# from common dir
-SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_lcore.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_timer.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_memzone.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_log.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_launch.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_mcfg.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_memalloc.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_memory.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_tailqs.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_errno.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_cpuflags.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_hypervisor.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_string_fns.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_hexdump.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_devargs.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_class.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_bus.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_dev.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_options.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_thread.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_proc.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_fbarray.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_uuid.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += rte_malloc.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += hotplug_mp.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += malloc_elem.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += malloc_heap.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += malloc_mp.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += rte_keepalive.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += rte_option.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += rte_service.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += rte_random.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += rte_reciprocal.c
-
-# from arch dir
-SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += rte_cpuflags.c
-SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += rte_hypervisor.c
-SRCS-$(CONFIG_RTE_ARCH_X86) += rte_spinlock.c
-SRCS-y += rte_cycles.c
-
-CFLAGS_eal_common_cpuflags.o := $(CPUFLAGS_LIST)
-
-# workaround for a gcc bug with noreturn attribute
-# http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12603
-ifeq ($(CONFIG_RTE_TOOLCHAIN_GCC),y)
-CFLAGS_eal_thread.o += -Wno-return-type
-endif
-
-INC := rte_kni_common.h
-INC += rte_os.h
-
-SYMLINK-$(CONFIG_RTE_EXEC_ENV_LINUX)-include := $(addprefix include/,$(INC))
-
-include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/lib/librte_eal/linux/eal/eal_alarm.c b/lib/librte_eal/linux/eal_alarm.c
similarity index 100%
rename from lib/librte_eal/linux/eal/eal_alarm.c
rename to lib/librte_eal/linux/eal_alarm.c
diff --git a/lib/librte_eal/linux/eal/eal_cpuflags.c b/lib/librte_eal/linux/eal_cpuflags.c
similarity index 100%
rename from lib/librte_eal/linux/eal/eal_cpuflags.c
rename to lib/librte_eal/linux/eal_cpuflags.c
diff --git a/lib/librte_eal/linux/eal/eal_debug.c b/lib/librte_eal/linux/eal_debug.c
similarity index 100%
rename from lib/librte_eal/linux/eal/eal_debug.c
rename to lib/librte_eal/linux/eal_debug.c
diff --git a/lib/librte_eal/linux/eal/eal_dev.c b/lib/librte_eal/linux/eal_dev.c
similarity index 100%
rename from lib/librte_eal/linux/eal/eal_dev.c
rename to lib/librte_eal/linux/eal_dev.c
diff --git a/lib/librte_eal/linux/eal/eal_hugepage_info.c b/lib/librte_eal/linux/eal_hugepage_info.c
similarity index 100%
rename from lib/librte_eal/linux/eal/eal_hugepage_info.c
rename to lib/librte_eal/linux/eal_hugepage_info.c
diff --git a/lib/librte_eal/linux/eal/eal_interrupts.c b/lib/librte_eal/linux/eal_interrupts.c
similarity index 100%
rename from lib/librte_eal/linux/eal/eal_interrupts.c
rename to lib/librte_eal/linux/eal_interrupts.c
diff --git a/lib/librte_eal/linux/eal/eal_lcore.c b/lib/librte_eal/linux/eal_lcore.c
similarity index 100%
rename from lib/librte_eal/linux/eal/eal_lcore.c
rename to lib/librte_eal/linux/eal_lcore.c
diff --git a/lib/librte_eal/linux/eal/eal_log.c b/lib/librte_eal/linux/eal_log.c
similarity index 100%
rename from lib/librte_eal/linux/eal/eal_log.c
rename to lib/librte_eal/linux/eal_log.c
diff --git a/lib/librte_eal/linux/eal/eal_memalloc.c b/lib/librte_eal/linux/eal_memalloc.c
similarity index 100%
rename from lib/librte_eal/linux/eal/eal_memalloc.c
rename to lib/librte_eal/linux/eal_memalloc.c
diff --git a/lib/librte_eal/linux/eal/eal_memory.c b/lib/librte_eal/linux/eal_memory.c
similarity index 100%
rename from lib/librte_eal/linux/eal/eal_memory.c
rename to lib/librte_eal/linux/eal_memory.c
diff --git a/lib/librte_eal/linux/eal/eal_thread.c b/lib/librte_eal/linux/eal_thread.c
similarity index 100%
rename from lib/librte_eal/linux/eal/eal_thread.c
rename to lib/librte_eal/linux/eal_thread.c
diff --git a/lib/librte_eal/linux/eal/eal_timer.c b/lib/librte_eal/linux/eal_timer.c
similarity index 100%
rename from lib/librte_eal/linux/eal/eal_timer.c
rename to lib/librte_eal/linux/eal_timer.c
diff --git a/lib/librte_eal/linux/eal/eal_vfio.c b/lib/librte_eal/linux/eal_vfio.c
similarity index 100%
rename from lib/librte_eal/linux/eal/eal_vfio.c
rename to lib/librte_eal/linux/eal_vfio.c
diff --git a/lib/librte_eal/linux/eal/eal_vfio.h b/lib/librte_eal/linux/eal_vfio.h
similarity index 100%
rename from lib/librte_eal/linux/eal/eal_vfio.h
rename to lib/librte_eal/linux/eal_vfio.h
diff --git a/lib/librte_eal/linux/eal/eal_vfio_mp_sync.c b/lib/librte_eal/linux/eal_vfio_mp_sync.c
similarity index 100%
rename from lib/librte_eal/linux/eal/eal_vfio_mp_sync.c
rename to lib/librte_eal/linux/eal_vfio_mp_sync.c
diff --git a/lib/librte_eal/linux/eal/include/rte_kni_common.h b/lib/librte_eal/linux/include/rte_kni_common.h
similarity index 100%
rename from lib/librte_eal/linux/eal/include/rte_kni_common.h
rename to lib/librte_eal/linux/include/rte_kni_common.h
diff --git a/lib/librte_eal/linux/eal/include/rte_os.h b/lib/librte_eal/linux/include/rte_os.h
similarity index 100%
rename from lib/librte_eal/linux/eal/include/rte_os.h
rename to lib/librte_eal/linux/include/rte_os.h
diff --git a/lib/librte_eal/linux/eal/meson.build b/lib/librte_eal/linux/meson.build
similarity index 81%
rename from lib/librte_eal/linux/eal/meson.build
rename to lib/librte_eal/linux/meson.build
index b02b0695f5..209dd7c642 100644
--- a/lib/librte_eal/linux/eal/meson.build
+++ b/lib/librte_eal/linux/meson.build
@@ -1,14 +1,15 @@
# SPDX-License-Identifier: BSD-3-Clause
# Copyright(c) 2017 Intel Corporation
-eal_inc += include_directories('include')
+includes += include_directories('include')
-env_objs = []
-env_headers = files(
+headers += files(
'include/rte_kni_common.h',
'include/rte_os.h',
)
-env_sources = files('eal_alarm.c',
+
+sources += files(
+ 'eal_alarm.c',
'eal_cpuflags.c',
'eal_debug.c',
'eal_hugepage_info.c',
diff --git a/lib/librte_eal/meson.build b/lib/librte_eal/meson.build
index 1730d603fc..516ef09a49 100644
--- a/lib/librte_eal/meson.build
+++ b/lib/librte_eal/meson.build
@@ -1,16 +1,16 @@
# SPDX-License-Identifier: BSD-3-Clause
# Copyright(c) 2017-2019 Intel Corporation
-# Custom EAL processing. EAL is complicated enough that it can't just
-# have a straight list of headers and source files.
-# Initially pull in common settings
-eal_inc = [global_inc]
-subdir('common') # defines common_sources, common_objs, etc.
+includes += global_inc
+
+subdir('include')
+
+subdir('common')
-# Now do OS/exec-env specific settings, including building kernel modules
-# The <exec-env>/eal/meson.build file should define env_sources, etc.
dpdk_conf.set('RTE_EXEC_ENV_' + exec_env.to_upper(), 1)
-subdir(exec_env + '/eal')
+subdir(exec_env)
+
+subdir(arch_subdir)
allow_experimental_apis = true
deps += 'kvargs'
@@ -23,7 +23,3 @@ endif
if cc.has_header('getopt.h')
cflags += ['-DHAVE_GETOPT_H', '-DHAVE_GETOPT', '-DHAVE_GETOPT_LONG']
endif
-sources = common_sources + env_sources
-objs = common_objs + env_objs
-headers = common_headers + env_headers
-includes = eal_inc
diff --git a/lib/librte_eal/common/include/arch/ppc_64/meson.build b/lib/librte_eal/ppc/include/meson.build
similarity index 100%
rename from lib/librte_eal/common/include/arch/ppc_64/meson.build
rename to lib/librte_eal/ppc/include/meson.build
diff --git a/lib/librte_eal/common/include/arch/ppc_64/rte_atomic.h b/lib/librte_eal/ppc/include/rte_atomic.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/ppc_64/rte_atomic.h
rename to lib/librte_eal/ppc/include/rte_atomic.h
diff --git a/lib/librte_eal/common/include/arch/ppc_64/rte_byteorder.h b/lib/librte_eal/ppc/include/rte_byteorder.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/ppc_64/rte_byteorder.h
rename to lib/librte_eal/ppc/include/rte_byteorder.h
diff --git a/lib/librte_eal/common/include/arch/ppc_64/rte_cpuflags.h b/lib/librte_eal/ppc/include/rte_cpuflags.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/ppc_64/rte_cpuflags.h
rename to lib/librte_eal/ppc/include/rte_cpuflags.h
diff --git a/lib/librte_eal/common/include/arch/ppc_64/rte_cycles.h b/lib/librte_eal/ppc/include/rte_cycles.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/ppc_64/rte_cycles.h
rename to lib/librte_eal/ppc/include/rte_cycles.h
diff --git a/lib/librte_eal/common/include/arch/ppc_64/rte_io.h b/lib/librte_eal/ppc/include/rte_io.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/ppc_64/rte_io.h
rename to lib/librte_eal/ppc/include/rte_io.h
diff --git a/lib/librte_eal/common/include/arch/ppc_64/rte_mcslock.h b/lib/librte_eal/ppc/include/rte_mcslock.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/ppc_64/rte_mcslock.h
rename to lib/librte_eal/ppc/include/rte_mcslock.h
diff --git a/lib/librte_eal/common/include/arch/ppc_64/rte_memcpy.h b/lib/librte_eal/ppc/include/rte_memcpy.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/ppc_64/rte_memcpy.h
rename to lib/librte_eal/ppc/include/rte_memcpy.h
diff --git a/lib/librte_eal/common/include/arch/ppc_64/rte_pause.h b/lib/librte_eal/ppc/include/rte_pause.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/ppc_64/rte_pause.h
rename to lib/librte_eal/ppc/include/rte_pause.h
diff --git a/lib/librte_eal/common/include/arch/ppc_64/rte_prefetch.h b/lib/librte_eal/ppc/include/rte_prefetch.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/ppc_64/rte_prefetch.h
rename to lib/librte_eal/ppc/include/rte_prefetch.h
diff --git a/lib/librte_eal/common/include/arch/ppc_64/rte_rwlock.h b/lib/librte_eal/ppc/include/rte_rwlock.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/ppc_64/rte_rwlock.h
rename to lib/librte_eal/ppc/include/rte_rwlock.h
diff --git a/lib/librte_eal/common/include/arch/ppc_64/rte_spinlock.h b/lib/librte_eal/ppc/include/rte_spinlock.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/ppc_64/rte_spinlock.h
rename to lib/librte_eal/ppc/include/rte_spinlock.h
diff --git a/lib/librte_eal/common/include/arch/ppc_64/rte_ticketlock.h b/lib/librte_eal/ppc/include/rte_ticketlock.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/ppc_64/rte_ticketlock.h
rename to lib/librte_eal/ppc/include/rte_ticketlock.h
diff --git a/lib/librte_eal/common/include/arch/ppc_64/rte_vect.h b/lib/librte_eal/ppc/include/rte_vect.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/ppc_64/rte_vect.h
rename to lib/librte_eal/ppc/include/rte_vect.h
diff --git a/lib/librte_eal/ppc/meson.build b/lib/librte_eal/ppc/meson.build
new file mode 100644
index 0000000000..0f72bd3a36
--- /dev/null
+++ b/lib/librte_eal/ppc/meson.build
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2018 Luca Boccassi <bluca@debian.org>
+
+includes += include_directories('include')
+
+headers += files(
+)
+
+sources += files(
+ 'rte_cpuflags.c',
+ 'rte_cycles.c',
+ 'rte_hypervisor.c',
+)
diff --git a/lib/librte_eal/common/arch/ppc_64/rte_cpuflags.c b/lib/librte_eal/ppc/rte_cpuflags.c
similarity index 100%
rename from lib/librte_eal/common/arch/ppc_64/rte_cpuflags.c
rename to lib/librte_eal/ppc/rte_cpuflags.c
diff --git a/lib/librte_eal/common/arch/ppc_64/rte_cycles.c b/lib/librte_eal/ppc/rte_cycles.c
similarity index 100%
rename from lib/librte_eal/common/arch/ppc_64/rte_cycles.c
rename to lib/librte_eal/ppc/rte_cycles.c
diff --git a/lib/librte_eal/common/arch/ppc_64/rte_hypervisor.c b/lib/librte_eal/ppc/rte_hypervisor.c
similarity index 100%
rename from lib/librte_eal/common/arch/ppc_64/rte_hypervisor.c
rename to lib/librte_eal/ppc/rte_hypervisor.c
diff --git a/lib/librte_eal/windows/eal/eal.c b/lib/librte_eal/windows/eal.c
similarity index 100%
rename from lib/librte_eal/windows/eal/eal.c
rename to lib/librte_eal/windows/eal.c
diff --git a/lib/librte_eal/windows/eal/meson.build b/lib/librte_eal/windows/eal/meson.build
deleted file mode 100644
index 2a062c365e..0000000000
--- a/lib/librte_eal/windows/eal/meson.build
+++ /dev/null
@@ -1,27 +0,0 @@
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2019 Intel Corporation
-
-eal_inc += include_directories('include')
-
-env_objs = []
-env_headers = files(
- 'include/rte_os.h',
-)
-common_sources = files(
- '../../common/eal_common_bus.c',
- '../../common/eal_common_class.c',
- '../../common/eal_common_devargs.c',
- '../../common/eal_common_errno.c',
- '../../common/eal_common_launch.c',
- '../../common/eal_common_lcore.c',
- '../../common/eal_common_log.c',
- '../../common/eal_common_options.c',
- '../../common/eal_common_thread.c',
- '../../common/rte_option.c',
-)
-env_sources = files('eal.c',
- 'eal_debug.c',
- 'eal_lcore.c',
- 'eal_thread.c',
- 'getopt.c',
-)
diff --git a/lib/librte_eal/windows/eal/eal_debug.c b/lib/librte_eal/windows/eal_debug.c
similarity index 100%
rename from lib/librte_eal/windows/eal/eal_debug.c
rename to lib/librte_eal/windows/eal_debug.c
diff --git a/lib/librte_eal/windows/eal/eal_lcore.c b/lib/librte_eal/windows/eal_lcore.c
similarity index 100%
rename from lib/librte_eal/windows/eal/eal_lcore.c
rename to lib/librte_eal/windows/eal_lcore.c
diff --git a/lib/librte_eal/windows/eal/eal_thread.c b/lib/librte_eal/windows/eal_thread.c
similarity index 100%
rename from lib/librte_eal/windows/eal/eal_thread.c
rename to lib/librte_eal/windows/eal_thread.c
diff --git a/lib/librte_eal/windows/eal/getopt.c b/lib/librte_eal/windows/getopt.c
similarity index 100%
rename from lib/librte_eal/windows/eal/getopt.c
rename to lib/librte_eal/windows/getopt.c
diff --git a/lib/librte_eal/windows/eal/include/dirent.h b/lib/librte_eal/windows/include/dirent.h
similarity index 100%
rename from lib/librte_eal/windows/eal/include/dirent.h
rename to lib/librte_eal/windows/include/dirent.h
diff --git a/lib/librte_eal/windows/eal/include/fnmatch.h b/lib/librte_eal/windows/include/fnmatch.h
similarity index 100%
rename from lib/librte_eal/windows/eal/include/fnmatch.h
rename to lib/librte_eal/windows/include/fnmatch.h
diff --git a/lib/librte_eal/windows/eal/include/getopt.h b/lib/librte_eal/windows/include/getopt.h
similarity index 100%
rename from lib/librte_eal/windows/eal/include/getopt.h
rename to lib/librte_eal/windows/include/getopt.h
diff --git a/lib/librte_eal/windows/eal/include/pthread.h b/lib/librte_eal/windows/include/pthread.h
similarity index 100%
rename from lib/librte_eal/windows/eal/include/pthread.h
rename to lib/librte_eal/windows/include/pthread.h
diff --git a/lib/librte_eal/windows/eal/include/regex.h b/lib/librte_eal/windows/include/regex.h
similarity index 100%
rename from lib/librte_eal/windows/eal/include/regex.h
rename to lib/librte_eal/windows/include/regex.h
diff --git a/lib/librte_eal/windows/eal/include/rte_os.h b/lib/librte_eal/windows/include/rte_os.h
similarity index 100%
rename from lib/librte_eal/windows/eal/include/rte_os.h
rename to lib/librte_eal/windows/include/rte_os.h
diff --git a/lib/librte_eal/windows/eal/include/sched.h b/lib/librte_eal/windows/include/sched.h
similarity index 100%
rename from lib/librte_eal/windows/eal/include/sched.h
rename to lib/librte_eal/windows/include/sched.h
diff --git a/lib/librte_eal/windows/eal/include/sys/queue.h b/lib/librte_eal/windows/include/sys/queue.h
similarity index 100%
rename from lib/librte_eal/windows/eal/include/sys/queue.h
rename to lib/librte_eal/windows/include/sys/queue.h
diff --git a/lib/librte_eal/windows/eal/include/unistd.h b/lib/librte_eal/windows/include/unistd.h
similarity index 100%
rename from lib/librte_eal/windows/eal/include/unistd.h
rename to lib/librte_eal/windows/include/unistd.h
diff --git a/lib/librte_eal/windows/meson.build b/lib/librte_eal/windows/meson.build
new file mode 100644
index 0000000000..5fcaf4785a
--- /dev/null
+++ b/lib/librte_eal/windows/meson.build
@@ -0,0 +1,16 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2019 Intel Corporation
+
+includes += include_directories('include')
+
+headers += files(
+ 'include/rte_os.h',
+)
+
+sources += files(
+ 'eal.c',
+ 'eal_debug.c',
+ 'eal_lcore.c',
+ 'eal_thread.c',
+ 'getopt.c',
+)
diff --git a/lib/librte_eal/common/include/arch/x86/meson.build b/lib/librte_eal/x86/include/meson.build
similarity index 100%
rename from lib/librte_eal/common/include/arch/x86/meson.build
rename to lib/librte_eal/x86/include/meson.build
diff --git a/lib/librte_eal/common/include/arch/x86/rte_atomic.h b/lib/librte_eal/x86/include/rte_atomic.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/x86/rte_atomic.h
rename to lib/librte_eal/x86/include/rte_atomic.h
diff --git a/lib/librte_eal/common/include/arch/x86/rte_atomic_32.h b/lib/librte_eal/x86/include/rte_atomic_32.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/x86/rte_atomic_32.h
rename to lib/librte_eal/x86/include/rte_atomic_32.h
diff --git a/lib/librte_eal/common/include/arch/x86/rte_atomic_64.h b/lib/librte_eal/x86/include/rte_atomic_64.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/x86/rte_atomic_64.h
rename to lib/librte_eal/x86/include/rte_atomic_64.h
diff --git a/lib/librte_eal/common/include/arch/x86/rte_byteorder.h b/lib/librte_eal/x86/include/rte_byteorder.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/x86/rte_byteorder.h
rename to lib/librte_eal/x86/include/rte_byteorder.h
diff --git a/lib/librte_eal/common/include/arch/x86/rte_byteorder_32.h b/lib/librte_eal/x86/include/rte_byteorder_32.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/x86/rte_byteorder_32.h
rename to lib/librte_eal/x86/include/rte_byteorder_32.h
diff --git a/lib/librte_eal/common/include/arch/x86/rte_byteorder_64.h b/lib/librte_eal/x86/include/rte_byteorder_64.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/x86/rte_byteorder_64.h
rename to lib/librte_eal/x86/include/rte_byteorder_64.h
diff --git a/lib/librte_eal/common/include/arch/x86/rte_cpuflags.h b/lib/librte_eal/x86/include/rte_cpuflags.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/x86/rte_cpuflags.h
rename to lib/librte_eal/x86/include/rte_cpuflags.h
diff --git a/lib/librte_eal/common/include/arch/x86/rte_cycles.h b/lib/librte_eal/x86/include/rte_cycles.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/x86/rte_cycles.h
rename to lib/librte_eal/x86/include/rte_cycles.h
diff --git a/lib/librte_eal/common/include/arch/x86/rte_io.h b/lib/librte_eal/x86/include/rte_io.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/x86/rte_io.h
rename to lib/librte_eal/x86/include/rte_io.h
diff --git a/lib/librte_eal/common/include/arch/x86/rte_mcslock.h b/lib/librte_eal/x86/include/rte_mcslock.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/x86/rte_mcslock.h
rename to lib/librte_eal/x86/include/rte_mcslock.h
diff --git a/lib/librte_eal/common/include/arch/x86/rte_memcpy.h b/lib/librte_eal/x86/include/rte_memcpy.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/x86/rte_memcpy.h
rename to lib/librte_eal/x86/include/rte_memcpy.h
diff --git a/lib/librte_eal/common/include/arch/x86/rte_pause.h b/lib/librte_eal/x86/include/rte_pause.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/x86/rte_pause.h
rename to lib/librte_eal/x86/include/rte_pause.h
diff --git a/lib/librte_eal/common/include/arch/x86/rte_prefetch.h b/lib/librte_eal/x86/include/rte_prefetch.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/x86/rte_prefetch.h
rename to lib/librte_eal/x86/include/rte_prefetch.h
diff --git a/lib/librte_eal/common/include/arch/x86/rte_rtm.h b/lib/librte_eal/x86/include/rte_rtm.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/x86/rte_rtm.h
rename to lib/librte_eal/x86/include/rte_rtm.h
diff --git a/lib/librte_eal/common/include/arch/x86/rte_rwlock.h b/lib/librte_eal/x86/include/rte_rwlock.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/x86/rte_rwlock.h
rename to lib/librte_eal/x86/include/rte_rwlock.h
diff --git a/lib/librte_eal/common/include/arch/x86/rte_spinlock.h b/lib/librte_eal/x86/include/rte_spinlock.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/x86/rte_spinlock.h
rename to lib/librte_eal/x86/include/rte_spinlock.h
diff --git a/lib/librte_eal/common/include/arch/x86/rte_ticketlock.h b/lib/librte_eal/x86/include/rte_ticketlock.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/x86/rte_ticketlock.h
rename to lib/librte_eal/x86/include/rte_ticketlock.h
diff --git a/lib/librte_eal/common/include/arch/x86/rte_vect.h b/lib/librte_eal/x86/include/rte_vect.h
similarity index 100%
rename from lib/librte_eal/common/include/arch/x86/rte_vect.h
rename to lib/librte_eal/x86/include/rte_vect.h
diff --git a/lib/librte_eal/x86/meson.build b/lib/librte_eal/x86/meson.build
new file mode 100644
index 0000000000..305d16c01b
--- /dev/null
+++ b/lib/librte_eal/x86/meson.build
@@ -0,0 +1,14 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2017 Intel Corporation
+
+includes += include_directories('include')
+
+headers += files(
+)
+
+sources += files(
+ 'rte_cpuflags.c',
+ 'rte_cycles.c',
+ 'rte_hypervisor.c',
+ 'rte_spinlock.c',
+)
diff --git a/lib/librte_eal/common/arch/x86/rte_cpuflags.c b/lib/librte_eal/x86/rte_cpuflags.c
similarity index 100%
rename from lib/librte_eal/common/arch/x86/rte_cpuflags.c
rename to lib/librte_eal/x86/rte_cpuflags.c
diff --git a/lib/librte_eal/common/arch/x86/rte_cpuid.h b/lib/librte_eal/x86/rte_cpuid.h
similarity index 100%
rename from lib/librte_eal/common/arch/x86/rte_cpuid.h
rename to lib/librte_eal/x86/rte_cpuid.h
diff --git a/lib/librte_eal/common/arch/x86/rte_cycles.c b/lib/librte_eal/x86/rte_cycles.c
similarity index 100%
rename from lib/librte_eal/common/arch/x86/rte_cycles.c
rename to lib/librte_eal/x86/rte_cycles.c
diff --git a/lib/librte_eal/common/arch/x86/rte_hypervisor.c b/lib/librte_eal/x86/rte_hypervisor.c
similarity index 100%
rename from lib/librte_eal/common/arch/x86/rte_hypervisor.c
rename to lib/librte_eal/x86/rte_hypervisor.c
diff --git a/lib/librte_eal/common/arch/x86/rte_spinlock.c b/lib/librte_eal/x86/rte_spinlock.c
similarity index 100%
rename from lib/librte_eal/common/arch/x86/rte_spinlock.c
rename to lib/librte_eal/x86/rte_spinlock.c
diff --git a/lib/librte_kvargs/Makefile b/lib/librte_kvargs/Makefile
index 419be8bd7c..24b1c3c5b9 100644
--- a/lib/librte_kvargs/Makefile
+++ b/lib/librte_kvargs/Makefile
@@ -7,7 +7,7 @@ include $(RTE_SDK)/mk/rte.vars.mk
LIB = librte_kvargs.a
CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR) -O3
-CFLAGS += -I$(RTE_SDK)/lib/librte_eal/common/include
+CFLAGS += -I$(RTE_SDK)/lib/librte_eal/include
EXPORT_MAP := rte_kvargs_version.map
diff --git a/license/exceptions.txt b/license/exceptions.txt
index 1e21a863f1..c984764a02 100644
--- a/license/exceptions.txt
+++ b/license/exceptions.txt
@@ -12,9 +12,9 @@ Note that following licenses are not exceptions:-
---------------------------------------------------------------------------------------------------
SPDX Identifier TB Approval Date GB Approval Date File name
---------------------------------------------------------------------------------------------------
-1.MIT 10/23/2019 02/10/2020 lib/librte_eal/windows/eal/include/dirent.h
-2.BSD-2-Clause 10/23/2019 12/18/2019 lib/librte_eal/windows/eal/include/getopt.h
+1.MIT 10/23/2019 02/10/2020 lib/librte_eal/windows/include/dirent.h
+2.BSD-2-Clause 10/23/2019 12/18/2019 lib/librte_eal/windows/include/getopt.h
3.ISC AND
- BSD-2-Clause 10/23/2019 12/18/2019 lib/librte_eal/windows/eal/getopt.c
+ BSD-2-Clause 10/23/2019 12/18/2019 lib/librte_eal/windows/getopt.c
4.GPL-2.0 09/25/2019 12/18/2019 buildtools/pmdinfogen/pmdinfogen.*
---------------------------------------------------------------------------------------------------
diff --git a/meson.build b/meson.build
index b7ae9c8d9a..d36580438e 100644
--- a/meson.build
+++ b/meson.build
@@ -28,8 +28,8 @@ abi_version_file = files('ABI_VERSION')
# able to be included in any file. We also store a global array of include dirs
# for passing to pmdinfogen scripts
global_inc = include_directories('.', 'config',
- 'lib/librte_eal/common/include',
- 'lib/librte_eal/@0@/eal/include'.format(host_machine.system()),
+ 'lib/librte_eal/include',
+ 'lib/librte_eal/@0@/include'.format(host_machine.system()),
)
subdir('config')
diff --git a/mk/exec-env/freebsd/rte.vars.mk b/mk/exec-env/freebsd/rte.vars.mk
index 3608530d3f..630eb55f73 100644
--- a/mk/exec-env/freebsd/rte.vars.mk
+++ b/mk/exec-env/freebsd/rte.vars.mk
@@ -18,7 +18,7 @@ EXECENV_CFLAGS = -pthread
endif
# include in every library to build
-EXECENV_CFLAGS += -I$(RTE_SDK)/lib/librte_eal/freebsd/eal/include
+EXECENV_CFLAGS += -I$(RTE_SDK)/lib/librte_eal/freebsd/include
EXECENV_LDFLAGS =
EXECENV_LDLIBS = -lexecinfo
diff --git a/mk/exec-env/linux/rte.vars.mk b/mk/exec-env/linux/rte.vars.mk
index bea3f76577..41ef4195b0 100644
--- a/mk/exec-env/linux/rte.vars.mk
+++ b/mk/exec-env/linux/rte.vars.mk
@@ -18,7 +18,7 @@ EXECENV_CFLAGS = -pthread
endif
# include in every library to build
-EXECENV_CFLAGS += -I$(RTE_SDK)/lib/librte_eal/linux/eal/include
+EXECENV_CFLAGS += -I$(RTE_SDK)/lib/librte_eal/linux/include
EXECENV_LDLIBS =
EXECENV_ASFLAGS =
--
2.25.1
^ permalink raw reply [relevance 1%]
* [dpdk-dev] [RFC] ring: make ring implementation non-inlined
@ 2020-03-20 16:41 3% Konstantin Ananyev
2020-03-20 17:54 0% ` Stephen Hemminger
2020-03-25 21:09 4% ` Jerin Jacob
0 siblings, 2 replies; 200+ results
From: Konstantin Ananyev @ 2020-03-20 16:41 UTC (permalink / raw)
To: dev
Cc: olivier.matz, honnappa.nagarahalli, jerinj, drc, stephen,
Konstantin Ananyev
As was discussed here:
http://mails.dpdk.org/archives/dev/2020-February/158586.html
this RFC aimed to hide ring internals into .c and make all
ring functions non-inlined. In theory that might help to
maintain ABI stability in future.
This is just a POC to measure the impact of proposed idea,
proper implementation would definetly need some extra effort.
On IA box (SKX) ring_perf_autotest shows ~20-30 cycles extra for
enqueue+dequeue pair. On some more realistic code, I suspect
the impact it might be a bit higher.
For MP/MC bulk transfers degradation seems quite small,
though for SP/SC and/or small transfers it is more then noticable
(see exact numbers below).
From my perspective we'd probably keep it inlined for now
to avoid any non-anticipated perfomance degradations.
Though intersted to see perf results and opinions from
other interested parties.
Intel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz
ring_perf_autotest (without patch/with patch)
### Testing single element enq/deq ###
legacy APIs: SP/SC: single: 8.75/43.23
legacy APIs: MP/MC: single: 56.18/80.44
### Testing burst enq/deq ###
legacy APIs: SP/SC: burst (size: 8): 37.36/53.37
legacy APIs: SP/SC: burst (size: 32): 93.97/117.30
legacy APIs: MP/MC: burst (size: 8): 78.23/91.45
legacy APIs: MP/MC: burst (size: 32): 131.59/152.49
### Testing bulk enq/deq ###
legacy APIs: SP/SC: bulk (size: 8): 37.29/54.48
legacy APIs: SP/SC: bulk (size: 32): 92.68/113.01
legacy APIs: MP/MC: bulk (size: 8): 78.40/93.50
legacy APIs: MP/MC: bulk (size: 32): 131.49/154.25
### Testing empty bulk deq ###
legacy APIs: SP/SC: bulk (size: 8): 4.00/16.86
legacy APIs: MP/MC: bulk (size: 8): 7.01/15.55
### Testing using two hyperthreads ###
legacy APIs: SP/SC: bulk (size: 8): 10.64/17.56
legacy APIs: MP/MC: bulk (size: 8): 15.30/16.69
legacy APIs: SP/SC: bulk (size: 32): 5.84/7.09
legacy APIs: MP/MC: bulk (size: 32): 6.34/7.54
### Testing using two physical cores ###
legacy APIs: SP/SC: bulk (size: 8): 24.34/42.40
legacy APIs: MP/MC: bulk (size: 8): 70.34/71.82
legacy APIs: SP/SC: bulk (size: 32): 12.67/14.68
legacy APIs: MP/MC: bulk (size: 32): 22.41/17.93
### Testing single element enq/deq ###
elem APIs: element size 16B: SP/SC: single: 10.65/41.96
elem APIs: element size 16B: MP/MC: single: 44.33/81.36
### Testing burst enq/deq ###
elem APIs: element size 16B: SP/SC: burst (size: 8): 39.20/58.52
elem APIs: element size 16B: SP/SC: burst (size: 32): 123.19/142.79
elem APIs: element size 16B: MP/MC: burst (size: 8): 80.72/101.36
elem APIs: element size 16B: MP/MC: burst (size: 32): 169.21/185.38
### Testing bulk enq/deq ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 41.64/58.46
elem APIs: element size 16B: SP/SC: bulk (size: 32): 122.74/142.52
elem APIs: element size 16B: MP/MC: bulk (size: 8): 80.60/103.14
elem APIs: element size 16B: MP/MC: bulk (size: 32): 169.39/186.67
### Testing empty bulk deq ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 5.01/17.17
elem APIs: element size 16B: MP/MC: bulk (size: 8): 6.01/14.80
### Testing using two hyperthreads ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 12.02/17.18
elem APIs: element size 16B: MP/MC: bulk (size: 8): 16.81/21.14
elem APIs: element size 16B: SP/SC: bulk (size: 32): 7.87/9.01
elem APIs: element size 16B: MP/MC: bulk (size: 32): 8.22/10.57
### Testing using two physical cores ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 27.00/51.94
elem APIs: element size 16B: MP/MC: bulk (size: 8): 78.24/74.48
elem APIs: element size 16B: SP/SC: bulk (size: 32): 15.41/16.14
elem APIs: element size 16B: MP/MC: bulk (size: 32): 18.72/21.64
Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
app/proc-info/main.c | 20 +-
app/test-pmd/cmdline.c | 2 +-
app/test/commands.c | 2 +-
app/test/test_pdump.c | 6 +-
app/test/test_ring_stress.h | 2 +-
config/common_base | 2 +-
drivers/crypto/ccp/ccp_pmd_ops.c | 2 +-
drivers/crypto/meson.build | 1 -
drivers/net/ring/rte_eth_ring.c | 17 +-
lib/librte_eventdev/rte_event_ring.c | 6 +-
lib/librte_eventdev/rte_event_ring.h | 17 +-
lib/librte_pdump/rte_pdump.c | 3 +-
lib/librte_port/rte_port_ring.c | 29 +-
lib/librte_ring/Makefile | 6 +-
lib/librte_ring/meson.build | 6 +-
lib/librte_ring/ring_elem.c | 919 +++++++++++++++++++++++++++
lib/librte_ring/ring_impl.c | 750 ++++++++++++++++++++++
lib/librte_ring/ring_impl.h | 64 ++
lib/librte_ring/rte_ring.c | 3 +-
lib/librte_ring/rte_ring.h | 412 ++----------
lib/librte_ring/rte_ring_elem.h | 519 ++-------------
lib/librte_ring/rte_ring_version.map | 46 ++
22 files changed, 1938 insertions(+), 896 deletions(-)
create mode 100644 lib/librte_ring/ring_elem.c
create mode 100644 lib/librte_ring/ring_impl.c
create mode 100644 lib/librte_ring/ring_impl.h
diff --git a/app/proc-info/main.c b/app/proc-info/main.c
index abeca4aab..aa8f113c1 100644
--- a/app/proc-info/main.c
+++ b/app/proc-info/main.c
@@ -1137,25 +1137,7 @@ show_ring(char *name)
if (name != NULL) {
struct rte_ring *ptr = rte_ring_lookup(name);
if (ptr != NULL) {
- printf(" - Name (%s) on socket (%d)\n"
- " - flags:\n"
- "\t -- Single Producer Enqueue (%u)\n"
- "\t -- Single Consmer Dequeue (%u)\n",
- ptr->name,
- ptr->memzone->socket_id,
- ptr->flags & RING_F_SP_ENQ,
- ptr->flags & RING_F_SC_DEQ);
- printf(" - size (%u) mask (0x%x) capacity (%u)\n",
- ptr->size,
- ptr->mask,
- ptr->capacity);
- printf(" - count (%u) free count (%u)\n",
- rte_ring_count(ptr),
- rte_ring_free_count(ptr));
- printf(" - full (%d) empty (%d)\n",
- rte_ring_full(ptr),
- rte_ring_empty(ptr));
-
+ rte_ring_dump(stdout, ptr);
STATS_BDR_STR(50, "");
return;
}
diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c
index a037a55c6..000853067 100644
--- a/app/test-pmd/cmdline.c
+++ b/app/test-pmd/cmdline.c
@@ -9562,7 +9562,7 @@ dump_struct_sizes(void)
#define DUMP_SIZE(t) printf("sizeof(" #t ") = %u\n", (unsigned)sizeof(t));
DUMP_SIZE(struct rte_mbuf);
DUMP_SIZE(struct rte_mempool);
- DUMP_SIZE(struct rte_ring);
+ //DUMP_SIZE(struct rte_ring);
#undef DUMP_SIZE
}
diff --git a/app/test/commands.c b/app/test/commands.c
index 3bf767bf7..89f93cdb2 100644
--- a/app/test/commands.c
+++ b/app/test/commands.c
@@ -107,7 +107,7 @@ dump_struct_sizes(void)
#define DUMP_SIZE(t) printf("sizeof(" #t ") = %u\n", (unsigned)sizeof(t));
DUMP_SIZE(struct rte_mbuf);
DUMP_SIZE(struct rte_mempool);
- DUMP_SIZE(struct rte_ring);
+ //DUMP_SIZE(struct rte_ring);
#undef DUMP_SIZE
}
diff --git a/app/test/test_pdump.c b/app/test/test_pdump.c
index ad183184c..6a1180bcb 100644
--- a/app/test/test_pdump.c
+++ b/app/test/test_pdump.c
@@ -57,8 +57,7 @@ run_pdump_client_tests(void)
if (ret < 0)
return -1;
mp->flags = 0x0000;
- ring_client = rte_ring_create("SR0", RING_SIZE, rte_socket_id(),
- RING_F_SP_ENQ | RING_F_SC_DEQ);
+ ring_client = rte_ring_create("SR0", RING_SIZE, rte_socket_id(), 0);
if (ring_client == NULL) {
printf("rte_ring_create SR0 failed");
return -1;
@@ -71,9 +70,6 @@ run_pdump_client_tests(void)
}
rte_eth_dev_probing_finish(eth_dev);
- ring_client->prod.single = 0;
- ring_client->cons.single = 0;
-
printf("\n***** flags = RTE_PDUMP_FLAG_TX *****\n");
for (itr = 0; itr < NUM_ITR; itr++) {
diff --git a/app/test/test_ring_stress.h b/app/test/test_ring_stress.h
index c6f0bc9f1..27d4adfb2 100644
--- a/app/test/test_ring_stress.h
+++ b/app/test/test_ring_stress.h
@@ -350,7 +350,7 @@ mt1_init(struct rte_ring **rng, void **data, uint32_t num)
/* alloc ring */
nr = 2 * num;
sz = rte_ring_get_memsize(nr);
- r = rte_zmalloc(NULL, sz, alignof(*r));
+ r = rte_zmalloc(NULL, sz, RTE_CACHE_LINE_SIZE);
if (r == NULL) {
printf("%s: alloca(%zu) for FIFO with %u elems failed",
__func__, sz, nr);
diff --git a/config/common_base b/config/common_base
index 7ca2f28b1..87ab0766d 100644
--- a/config/common_base
+++ b/config/common_base
@@ -670,7 +670,7 @@ CONFIG_RTE_LIBRTE_PMD_ZUC=n
# Compile PMD for Crypto Scheduler device
#
-CONFIG_RTE_LIBRTE_PMD_CRYPTO_SCHEDULER=y
+CONFIG_RTE_LIBRTE_PMD_CRYPTO_SCHEDULER=n
#
# Compile PMD for NULL Crypto device
diff --git a/drivers/crypto/ccp/ccp_pmd_ops.c b/drivers/crypto/ccp/ccp_pmd_ops.c
index a19e85ecb..938977dc0 100644
--- a/drivers/crypto/ccp/ccp_pmd_ops.c
+++ b/drivers/crypto/ccp/ccp_pmd_ops.c
@@ -666,7 +666,7 @@ ccp_pmd_qp_create_batch_info_ring(struct ccp_qp *qp,
r = rte_ring_lookup(qp->name);
if (r) {
- if (r->size >= ring_size) {
+ if (rte_ring_get_size(r) >= ring_size) {
CCP_LOG_INFO(
"Reusing ring %s for processed packets",
qp->name);
diff --git a/drivers/crypto/meson.build b/drivers/crypto/meson.build
index 7fa1fbe26..8c8b78db3 100644
--- a/drivers/crypto/meson.build
+++ b/drivers/crypto/meson.build
@@ -16,7 +16,6 @@ drivers = ['aesni_gcm',
'octeontx2',
'openssl',
'qat',
- 'scheduler',
'snow3g',
'virtio',
'zuc']
diff --git a/drivers/net/ring/rte_eth_ring.c b/drivers/net/ring/rte_eth_ring.c
index 41acbc513..0eaac0dc3 100644
--- a/drivers/net/ring/rte_eth_ring.c
+++ b/drivers/net/ring/rte_eth_ring.c
@@ -41,6 +41,8 @@ struct ring_queue {
struct rte_ring *rng;
rte_atomic64_t rx_pkts;
rte_atomic64_t tx_pkts;
+ uint32_t rx_sync_type;
+ uint32_t tx_sync_type;
};
struct pmd_internals {
@@ -74,7 +76,7 @@ eth_ring_rx(void *q, struct rte_mbuf **bufs, uint16_t nb_bufs)
struct ring_queue *r = q;
const uint16_t nb_rx = (uint16_t)rte_ring_dequeue_burst(r->rng,
ptrs, nb_bufs, NULL);
- if (r->rng->flags & RING_F_SC_DEQ)
+ if (r->rx_sync_type == RTE_RING_SYNC_ST)
r->rx_pkts.cnt += nb_rx;
else
rte_atomic64_add(&(r->rx_pkts), nb_rx);
@@ -88,7 +90,7 @@ eth_ring_tx(void *q, struct rte_mbuf **bufs, uint16_t nb_bufs)
struct ring_queue *r = q;
const uint16_t nb_tx = (uint16_t)rte_ring_enqueue_burst(r->rng,
ptrs, nb_bufs, NULL);
- if (r->rng->flags & RING_F_SP_ENQ)
+ if (r->tx_sync_type == RTE_RING_SYNC_ST)
r->tx_pkts.cnt += nb_tx;
else
rte_atomic64_add(&(r->tx_pkts), nb_tx);
@@ -306,10 +308,14 @@ do_eth_dev_ring_create(const char *name,
internals->max_tx_queues = nb_tx_queues;
for (i = 0; i < nb_rx_queues; i++) {
internals->rx_ring_queues[i].rng = rx_queues[i];
+ internals->rx_ring_queues[i].rx_sync_type =
+ rte_ring_get_cons_sync_type(rx_queues[i]);
data->rx_queues[i] = &internals->rx_ring_queues[i];
}
for (i = 0; i < nb_tx_queues; i++) {
internals->tx_ring_queues[i].rng = tx_queues[i];
+ internals->tx_ring_queues[i].tx_sync_type =
+ rte_ring_get_prod_sync_type(rx_queues[i]);
data->tx_queues[i] = &internals->tx_ring_queues[i];
}
@@ -403,8 +409,11 @@ rte_eth_from_rings(const char *name, struct rte_ring *const rx_queues[],
int
rte_eth_from_ring(struct rte_ring *r)
{
- return rte_eth_from_rings(r->name, &r, 1, &r, 1,
- r->memzone ? r->memzone->socket_id : SOCKET_ID_ANY);
+ const struct rte_memzone *mz;
+
+ mz = rte_ring_get_memzone(r);
+ return rte_eth_from_rings(rte_ring_get_name(r), &r, 1, &r, 1,
+ mz ? mz->socket_id : SOCKET_ID_ANY);
}
static int
diff --git a/lib/librte_eventdev/rte_event_ring.c b/lib/librte_eventdev/rte_event_ring.c
index d27e23901..99700a18c 100644
--- a/lib/librte_eventdev/rte_event_ring.c
+++ b/lib/librte_eventdev/rte_event_ring.c
@@ -16,12 +16,8 @@ int
rte_event_ring_init(struct rte_event_ring *r, const char *name,
unsigned int count, unsigned int flags)
{
- /* compilation-time checks */
- RTE_BUILD_BUG_ON((sizeof(struct rte_event_ring) &
- RTE_CACHE_LINE_MASK) != 0);
-
/* init the ring structure */
- return rte_ring_init(&r->r, name, count, flags);
+ return rte_ring_init(r, name, count, flags);
}
/* create the ring */
diff --git a/lib/librte_eventdev/rte_event_ring.h b/lib/librte_eventdev/rte_event_ring.h
index c0861b0ec..b31b33b3e 100644
--- a/lib/librte_eventdev/rte_event_ring.h
+++ b/lib/librte_eventdev/rte_event_ring.h
@@ -32,9 +32,8 @@
* used inside software eventdev implementations and by applications
* directly as needed.
*/
-struct rte_event_ring {
- struct rte_ring r;
-};
+
+#define rte_event_ring rte_ring
/**
* Returns the number of events in the ring
@@ -47,7 +46,7 @@ struct rte_event_ring {
static __rte_always_inline unsigned int
rte_event_ring_count(const struct rte_event_ring *r)
{
- return rte_ring_count(&r->r);
+ return rte_ring_count(r);
}
/**
@@ -62,7 +61,7 @@ rte_event_ring_count(const struct rte_event_ring *r)
static __rte_always_inline unsigned int
rte_event_ring_free_count(const struct rte_event_ring *r)
{
- return rte_ring_free_count(&r->r);
+ return rte_ring_free_count(r);
}
/**
@@ -93,7 +92,7 @@ rte_event_ring_enqueue_burst(struct rte_event_ring *r,
unsigned int num;
uint32_t space;
- num = rte_ring_enqueue_burst_elem(&r->r, events,
+ num = rte_ring_enqueue_burst_elem(r, events,
sizeof(struct rte_event), n,
&space);
@@ -129,7 +128,7 @@ rte_event_ring_dequeue_burst(struct rte_event_ring *r,
unsigned int num;
uint32_t remaining;
- num = rte_ring_dequeue_burst_elem(&r->r, events,
+ num = rte_ring_dequeue_burst_elem(r, events,
sizeof(struct rte_event), n,
&remaining);
@@ -250,7 +249,7 @@ rte_event_ring_free(struct rte_event_ring *r);
static inline unsigned int
rte_event_ring_get_size(const struct rte_event_ring *r)
{
- return rte_ring_get_size(&r->r);
+ return rte_ring_get_size(r);
}
/**
@@ -264,6 +263,6 @@ rte_event_ring_get_size(const struct rte_event_ring *r)
static inline unsigned int
rte_event_ring_get_capacity(const struct rte_event_ring *r)
{
- return rte_ring_get_capacity(&r->r);
+ return rte_ring_get_capacity(r);
}
#endif
diff --git a/lib/librte_pdump/rte_pdump.c b/lib/librte_pdump/rte_pdump.c
index 8a01ac510..43aa07793 100644
--- a/lib/librte_pdump/rte_pdump.c
+++ b/lib/librte_pdump/rte_pdump.c
@@ -380,7 +380,8 @@ pdump_validate_ring_mp(struct rte_ring *ring, struct rte_mempool *mp)
rte_errno = EINVAL;
return -1;
}
- if (ring->prod.single || ring->cons.single) {
+ if (rte_ring_get_prod_sync_type(ring) == RTE_RING_SYNC_ST ||
+ rte_ring_get_cons_sync_type(ring) == RTE_RING_SYNC_ST) {
PDUMP_LOG(ERR, "ring with either SP or SC settings"
" is not valid for pdump, should have MP and MC settings\n");
rte_errno = EINVAL;
diff --git a/lib/librte_port/rte_port_ring.c b/lib/librte_port/rte_port_ring.c
index 47fcdd06a..16fdee0bd 100644
--- a/lib/librte_port/rte_port_ring.c
+++ b/lib/librte_port/rte_port_ring.c
@@ -40,12 +40,15 @@ rte_port_ring_reader_create_internal(void *params, int socket_id,
struct rte_port_ring_reader_params *conf =
params;
struct rte_port_ring_reader *port;
+ uint32_t ring_sync;
+
+ ring_sync = rte_ring_get_cons_sync_type(conf->ring);
/* Check input parameters */
if ((conf == NULL) ||
(conf->ring == NULL) ||
- (conf->ring->cons.single && is_multi) ||
- (!(conf->ring->cons.single) && !is_multi)) {
+ (ring_sync == RTE_RING_SYNC_ST && is_multi) ||
+ (ring_sync != RTE_RING_SYNC_ST && !is_multi)) {
RTE_LOG(ERR, PORT, "%s: Invalid Parameters\n", __func__);
return NULL;
}
@@ -167,13 +170,16 @@ rte_port_ring_writer_create_internal(void *params, int socket_id,
struct rte_port_ring_writer_params *conf =
params;
struct rte_port_ring_writer *port;
+ uint32_t ring_sync;
+
+ ring_sync = rte_ring_get_prod_sync_type(conf->ring);
/* Check input parameters */
if ((conf == NULL) ||
- (conf->ring == NULL) ||
- (conf->ring->prod.single && is_multi) ||
- (!(conf->ring->prod.single) && !is_multi) ||
- (conf->tx_burst_sz > RTE_PORT_IN_BURST_SIZE_MAX)) {
+ (conf->ring == NULL) ||
+ (ring_sync == RTE_RING_SYNC_ST && is_multi) ||
+ (ring_sync != RTE_RING_SYNC_ST && !is_multi) ||
+ (conf->tx_burst_sz > RTE_PORT_IN_BURST_SIZE_MAX)) {
RTE_LOG(ERR, PORT, "%s: Invalid Parameters\n", __func__);
return NULL;
}
@@ -436,13 +442,16 @@ rte_port_ring_writer_nodrop_create_internal(void *params, int socket_id,
struct rte_port_ring_writer_nodrop_params *conf =
params;
struct rte_port_ring_writer_nodrop *port;
+ uint32_t ring_sync;
+
+ ring_sync = rte_ring_get_prod_sync_type(conf->ring);
/* Check input parameters */
if ((conf == NULL) ||
- (conf->ring == NULL) ||
- (conf->ring->prod.single && is_multi) ||
- (!(conf->ring->prod.single) && !is_multi) ||
- (conf->tx_burst_sz > RTE_PORT_IN_BURST_SIZE_MAX)) {
+ (conf->ring == NULL) ||
+ (ring_sync == RTE_RING_SYNC_ST && is_multi) ||
+ (ring_sync != RTE_RING_SYNC_ST && !is_multi) ||
+ (conf->tx_burst_sz > RTE_PORT_IN_BURST_SIZE_MAX)) {
RTE_LOG(ERR, PORT, "%s: Invalid Parameters\n", __func__);
return NULL;
}
diff --git a/lib/librte_ring/Makefile b/lib/librte_ring/Makefile
index 917c560ad..8c3cd523d 100644
--- a/lib/librte_ring/Makefile
+++ b/lib/librte_ring/Makefile
@@ -13,11 +13,11 @@ EXPORT_MAP := rte_ring_version.map
# all source are stored in SRCS-y
SRCS-$(CONFIG_RTE_LIBRTE_RING) := rte_ring.c
+SRCS-$(CONFIG_RTE_LIBRTE_RING) += ring_impl.c
+SRCS-$(CONFIG_RTE_LIBRTE_RING) += ring_elem.c
# install includes
SYMLINK-$(CONFIG_RTE_LIBRTE_RING)-include := rte_ring.h \
- rte_ring_elem.h \
- rte_ring_generic.h \
- rte_ring_c11_mem.h
+ rte_ring_elem.h
include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/lib/librte_ring/meson.build b/lib/librte_ring/meson.build
index f2f3ccc88..8adbdfc9d 100644
--- a/lib/librte_ring/meson.build
+++ b/lib/librte_ring/meson.build
@@ -1,11 +1,9 @@
# SPDX-License-Identifier: BSD-3-Clause
# Copyright(c) 2017 Intel Corporation
-sources = files('rte_ring.c')
+sources = files('rte_ring.c', 'ring_impl.c', 'ring_elem.c')
headers = files('rte_ring.h',
- 'rte_ring_elem.h',
- 'rte_ring_c11_mem.h',
- 'rte_ring_generic.h')
+ 'rte_ring_elem.h')
# rte_ring_create_elem and rte_ring_get_memsize_elem are experimental
allow_experimental_apis = true
diff --git a/lib/librte_ring/ring_elem.c b/lib/librte_ring/ring_elem.c
new file mode 100644
index 000000000..3588725e7
--- /dev/null
+++ b/lib/librte_ring/ring_elem.c
@@ -0,0 +1,919 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ *
+ * Copyright (c) 2019 Arm Limited
+ * Copyright (c) 2010-2017 Intel Corporation
+ * Copyright (c) 2007-2009 Kip Macy kmacy@freebsd.org
+ * All rights reserved.
+ * Derived from FreeBSD's bufring.h
+ * Used as BSD-3 Licensed with permission from Kip Macy.
+ */
+
+/**
+ * @file
+ * RTE Ring with user defined element size
+ */
+
+#include <stdio.h>
+#include <stdint.h>
+#include <string.h>
+#include <sys/queue.h>
+#include <errno.h>
+#include <rte_common.h>
+#include <rte_config.h>
+#include <rte_memory.h>
+#include <rte_lcore.h>
+#include <rte_atomic.h>
+#include <rte_branch_prediction.h>
+#include <rte_memzone.h>
+#include <rte_pause.h>
+
+#include "ring_impl.h"
+
+static __rte_always_inline void
+__rte_ring_enqueue_elems_32(struct rte_ring *r, const uint32_t size,
+ uint32_t idx, const void *obj_table, uint32_t n)
+{
+ unsigned int i;
+ uint32_t *ring = (uint32_t *)&r[1];
+ const uint32_t *obj = (const uint32_t *)obj_table;
+ if (likely(idx + n < size)) {
+ for (i = 0; i < (n & ~0x7); i += 8, idx += 8) {
+ ring[idx] = obj[i];
+ ring[idx + 1] = obj[i + 1];
+ ring[idx + 2] = obj[i + 2];
+ ring[idx + 3] = obj[i + 3];
+ ring[idx + 4] = obj[i + 4];
+ ring[idx + 5] = obj[i + 5];
+ ring[idx + 6] = obj[i + 6];
+ ring[idx + 7] = obj[i + 7];
+ }
+ switch (n & 0x7) {
+ case 7:
+ ring[idx++] = obj[i++]; /* fallthrough */
+ case 6:
+ ring[idx++] = obj[i++]; /* fallthrough */
+ case 5:
+ ring[idx++] = obj[i++]; /* fallthrough */
+ case 4:
+ ring[idx++] = obj[i++]; /* fallthrough */
+ case 3:
+ ring[idx++] = obj[i++]; /* fallthrough */
+ case 2:
+ ring[idx++] = obj[i++]; /* fallthrough */
+ case 1:
+ ring[idx++] = obj[i++]; /* fallthrough */
+ }
+ } else {
+ for (i = 0; idx < size; i++, idx++)
+ ring[idx] = obj[i];
+ /* Start at the beginning */
+ for (idx = 0; i < n; i++, idx++)
+ ring[idx] = obj[i];
+ }
+}
+
+static __rte_always_inline void
+__rte_ring_enqueue_elems_64(struct rte_ring *r, uint32_t prod_head,
+ const void *obj_table, uint32_t n)
+{
+ unsigned int i;
+ const uint32_t size = r->size;
+ uint32_t idx = prod_head & r->mask;
+ uint64_t *ring = (uint64_t *)&r[1];
+ const uint64_t *obj = (const uint64_t *)obj_table;
+ if (likely(idx + n < size)) {
+ for (i = 0; i < (n & ~0x3); i += 4, idx += 4) {
+ ring[idx] = obj[i];
+ ring[idx + 1] = obj[i + 1];
+ ring[idx + 2] = obj[i + 2];
+ ring[idx + 3] = obj[i + 3];
+ }
+ switch (n & 0x3) {
+ case 3:
+ ring[idx++] = obj[i++]; /* fallthrough */
+ case 2:
+ ring[idx++] = obj[i++]; /* fallthrough */
+ case 1:
+ ring[idx++] = obj[i++];
+ }
+ } else {
+ for (i = 0; idx < size; i++, idx++)
+ ring[idx] = obj[i];
+ /* Start at the beginning */
+ for (idx = 0; i < n; i++, idx++)
+ ring[idx] = obj[i];
+ }
+}
+
+static __rte_always_inline void
+__rte_ring_enqueue_elems_128(struct rte_ring *r, uint32_t prod_head,
+ const void *obj_table, uint32_t n)
+{
+ unsigned int i;
+ const uint32_t size = r->size;
+ uint32_t idx = prod_head & r->mask;
+ rte_int128_t *ring = (rte_int128_t *)&r[1];
+ const rte_int128_t *obj = (const rte_int128_t *)obj_table;
+ if (likely(idx + n < size)) {
+ for (i = 0; i < (n & ~0x1); i += 2, idx += 2)
+ memcpy((void *)(ring + idx),
+ (const void *)(obj + i), 32);
+ switch (n & 0x1) {
+ case 1:
+ memcpy((void *)(ring + idx),
+ (const void *)(obj + i), 16);
+ }
+ } else {
+ for (i = 0; idx < size; i++, idx++)
+ memcpy((void *)(ring + idx),
+ (const void *)(obj + i), 16);
+ /* Start at the beginning */
+ for (idx = 0; i < n; i++, idx++)
+ memcpy((void *)(ring + idx),
+ (const void *)(obj + i), 16);
+ }
+}
+
+/* the actual enqueue of elements on the ring.
+ * Placed here since identical code needed in both
+ * single and multi producer enqueue functions.
+ */
+static __rte_always_inline void
+__rte_ring_enqueue_elems(struct rte_ring *r, uint32_t prod_head,
+ const void *obj_table, uint32_t esize, uint32_t num)
+{
+ /* 8B and 16B copies implemented individually to retain
+ * the current performance.
+ */
+ if (esize == 8)
+ __rte_ring_enqueue_elems_64(r, prod_head, obj_table, num);
+ else if (esize == 16)
+ __rte_ring_enqueue_elems_128(r, prod_head, obj_table, num);
+ else {
+ uint32_t idx, scale, nr_idx, nr_num, nr_size;
+
+ /* Normalize to uint32_t */
+ scale = esize / sizeof(uint32_t);
+ nr_num = num * scale;
+ idx = prod_head & r->mask;
+ nr_idx = idx * scale;
+ nr_size = r->size * scale;
+ __rte_ring_enqueue_elems_32(r, nr_size, nr_idx,
+ obj_table, nr_num);
+ }
+}
+
+static __rte_always_inline void
+__rte_ring_dequeue_elems_32(struct rte_ring *r, const uint32_t size,
+ uint32_t idx, void *obj_table, uint32_t n)
+{
+ unsigned int i;
+ uint32_t *ring = (uint32_t *)&r[1];
+ uint32_t *obj = (uint32_t *)obj_table;
+ if (likely(idx + n < size)) {
+ for (i = 0; i < (n & ~0x7); i += 8, idx += 8) {
+ obj[i] = ring[idx];
+ obj[i + 1] = ring[idx + 1];
+ obj[i + 2] = ring[idx + 2];
+ obj[i + 3] = ring[idx + 3];
+ obj[i + 4] = ring[idx + 4];
+ obj[i + 5] = ring[idx + 5];
+ obj[i + 6] = ring[idx + 6];
+ obj[i + 7] = ring[idx + 7];
+ }
+ switch (n & 0x7) {
+ case 7:
+ obj[i++] = ring[idx++]; /* fallthrough */
+ case 6:
+ obj[i++] = ring[idx++]; /* fallthrough */
+ case 5:
+ obj[i++] = ring[idx++]; /* fallthrough */
+ case 4:
+ obj[i++] = ring[idx++]; /* fallthrough */
+ case 3:
+ obj[i++] = ring[idx++]; /* fallthrough */
+ case 2:
+ obj[i++] = ring[idx++]; /* fallthrough */
+ case 1:
+ obj[i++] = ring[idx++]; /* fallthrough */
+ }
+ } else {
+ for (i = 0; idx < size; i++, idx++)
+ obj[i] = ring[idx];
+ /* Start at the beginning */
+ for (idx = 0; i < n; i++, idx++)
+ obj[i] = ring[idx];
+ }
+}
+
+static __rte_always_inline void
+__rte_ring_dequeue_elems_64(struct rte_ring *r, uint32_t prod_head,
+ void *obj_table, uint32_t n)
+{
+ unsigned int i;
+ const uint32_t size = r->size;
+ uint32_t idx = prod_head & r->mask;
+ uint64_t *ring = (uint64_t *)&r[1];
+ uint64_t *obj = (uint64_t *)obj_table;
+ if (likely(idx + n < size)) {
+ for (i = 0; i < (n & ~0x3); i += 4, idx += 4) {
+ obj[i] = ring[idx];
+ obj[i + 1] = ring[idx + 1];
+ obj[i + 2] = ring[idx + 2];
+ obj[i + 3] = ring[idx + 3];
+ }
+ switch (n & 0x3) {
+ case 3:
+ obj[i++] = ring[idx++]; /* fallthrough */
+ case 2:
+ obj[i++] = ring[idx++]; /* fallthrough */
+ case 1:
+ obj[i++] = ring[idx++]; /* fallthrough */
+ }
+ } else {
+ for (i = 0; idx < size; i++, idx++)
+ obj[i] = ring[idx];
+ /* Start at the beginning */
+ for (idx = 0; i < n; i++, idx++)
+ obj[i] = ring[idx];
+ }
+}
+
+static __rte_always_inline void
+__rte_ring_dequeue_elems_128(struct rte_ring *r, uint32_t prod_head,
+ void *obj_table, uint32_t n)
+{
+ unsigned int i;
+ const uint32_t size = r->size;
+ uint32_t idx = prod_head & r->mask;
+ rte_int128_t *ring = (rte_int128_t *)&r[1];
+ rte_int128_t *obj = (rte_int128_t *)obj_table;
+ if (likely(idx + n < size)) {
+ for (i = 0; i < (n & ~0x1); i += 2, idx += 2)
+ memcpy((void *)(obj + i), (void *)(ring + idx), 32);
+ switch (n & 0x1) {
+ case 1:
+ memcpy((void *)(obj + i), (void *)(ring + idx), 16);
+ }
+ } else {
+ for (i = 0; idx < size; i++, idx++)
+ memcpy((void *)(obj + i), (void *)(ring + idx), 16);
+ /* Start at the beginning */
+ for (idx = 0; i < n; i++, idx++)
+ memcpy((void *)(obj + i), (void *)(ring + idx), 16);
+ }
+}
+
+/* the actual dequeue of elements from the ring.
+ * Placed here since identical code needed in both
+ * single and multi producer enqueue functions.
+ */
+static __rte_always_inline void
+__rte_ring_dequeue_elems(struct rte_ring *r, uint32_t cons_head,
+ void *obj_table, uint32_t esize, uint32_t num)
+{
+ /* 8B and 16B copies implemented individually to retain
+ * the current performance.
+ */
+ if (esize == 8)
+ __rte_ring_dequeue_elems_64(r, cons_head, obj_table, num);
+ else if (esize == 16)
+ __rte_ring_dequeue_elems_128(r, cons_head, obj_table, num);
+ else {
+ uint32_t idx, scale, nr_idx, nr_num, nr_size;
+
+ /* Normalize to uint32_t */
+ scale = esize / sizeof(uint32_t);
+ nr_num = num * scale;
+ idx = cons_head & r->mask;
+ nr_idx = idx * scale;
+ nr_size = r->size * scale;
+ __rte_ring_dequeue_elems_32(r, nr_size, nr_idx,
+ obj_table, nr_num);
+ }
+}
+
+/* Between load and load. there might be cpu reorder in weak model
+ * (powerpc/arm).
+ * There are 2 choices for the users
+ * 1.use rmb() memory barrier
+ * 2.use one-direction load_acquire/store_release barrier,defined by
+ * CONFIG_RTE_USE_C11_MEM_MODEL=y
+ * It depends on performance test results.
+ * By default, move common functions to rte_ring_generic.h
+ */
+#ifdef RTE_USE_C11_MEM_MODEL
+#include "rte_ring_c11_mem.h"
+#else
+#include "rte_ring_generic.h"
+#endif
+
+/**
+ * @internal Enqueue several objects on the ring
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of objects.
+ * @param esize
+ * The size of ring element, in bytes. It must be a multiple of 4.
+ * This must be the same value used while creating the ring. Otherwise
+ * the results are undefined.
+ * @param n
+ * The number of objects to add in the ring from the obj_table.
+ * @param behavior
+ * RTE_RING_QUEUE_FIXED: Enqueue a fixed number of items from a ring
+ * RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible from ring
+ * @param is_sp
+ * Indicates whether to use single producer or multi-producer head update
+ * @param free_space
+ * returns the amount of space after the enqueue operation has finished
+ * @return
+ * Actual number of objects enqueued.
+ * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_enqueue_elem(struct rte_ring *r, const void *obj_table,
+ unsigned int esize, unsigned int n,
+ enum rte_ring_queue_behavior behavior, unsigned int is_sp,
+ unsigned int *free_space)
+{
+ uint32_t prod_head, prod_next;
+ uint32_t free_entries;
+
+ n = __rte_ring_move_prod_head(r, is_sp, n, behavior,
+ &prod_head, &prod_next, &free_entries);
+ if (n == 0)
+ goto end;
+
+ __rte_ring_enqueue_elems(r, prod_head, obj_table, esize, n);
+
+ update_tail(&r->prod, prod_head, prod_next, is_sp, 1);
+end:
+ if (free_space != NULL)
+ *free_space = free_entries - n;
+ return n;
+}
+
+/**
+ * @internal Dequeue several objects from the ring
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of objects.
+ * @param esize
+ * The size of ring element, in bytes. It must be a multiple of 4.
+ * This must be the same value used while creating the ring. Otherwise
+ * the results are undefined.
+ * @param n
+ * The number of objects to pull from the ring.
+ * @param behavior
+ * RTE_RING_QUEUE_FIXED: Dequeue a fixed number of items from a ring
+ * RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from ring
+ * @param is_sc
+ * Indicates whether to use single consumer or multi-consumer head update
+ * @param available
+ * returns the number of remaining ring entries after the dequeue has finished
+ * @return
+ * - Actual number of objects dequeued.
+ * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_dequeue_elem(struct rte_ring *r, void *obj_table,
+ unsigned int esize, unsigned int n,
+ enum rte_ring_queue_behavior behavior, unsigned int is_sc,
+ unsigned int *available)
+{
+ uint32_t cons_head, cons_next;
+ uint32_t entries;
+
+ n = __rte_ring_move_cons_head(r, (int)is_sc, n, behavior,
+ &cons_head, &cons_next, &entries);
+ if (n == 0)
+ goto end;
+
+ __rte_ring_dequeue_elems(r, cons_head, obj_table, esize, n);
+
+ update_tail(&r->cons, cons_head, cons_next, is_sc, 0);
+
+end:
+ if (available != NULL)
+ *available = entries - n;
+ return n;
+}
+
+/**
+ * Enqueue several objects on the ring (multi-producers safe).
+ *
+ * This function uses a "compare and set" instruction to move the
+ * producer index atomically.
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of objects.
+ * @param esize
+ * The size of ring element, in bytes. It must be a multiple of 4.
+ * This must be the same value used while creating the ring. Otherwise
+ * the results are undefined.
+ * @param n
+ * The number of objects to add in the ring from the obj_table.
+ * @param free_space
+ * if non-NULL, returns the amount of space in the ring after the
+ * enqueue operation has finished.
+ * @return
+ * The number of objects enqueued, either 0 or n
+ */
+unsigned int
+rte_ring_mp_enqueue_bulk_elem(struct rte_ring *r, const void *obj_table,
+ unsigned int esize, unsigned int n, unsigned int *free_space)
+{
+ return __rte_ring_do_enqueue_elem(r, obj_table, esize, n,
+ RTE_RING_QUEUE_FIXED, __IS_MP, free_space);
+}
+
+/**
+ * Enqueue several objects on a ring
+ *
+ * @warning This API is NOT multi-producers safe
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of objects.
+ * @param esize
+ * The size of ring element, in bytes. It must be a multiple of 4.
+ * This must be the same value used while creating the ring. Otherwise
+ * the results are undefined.
+ * @param n
+ * The number of objects to add in the ring from the obj_table.
+ * @param free_space
+ * if non-NULL, returns the amount of space in the ring after the
+ * enqueue operation has finished.
+ * @return
+ * The number of objects enqueued, either 0 or n
+ */
+unsigned int
+rte_ring_sp_enqueue_bulk_elem(struct rte_ring *r, const void *obj_table,
+ unsigned int esize, unsigned int n, unsigned int *free_space)
+{
+ return __rte_ring_do_enqueue_elem(r, obj_table, esize, n,
+ RTE_RING_QUEUE_FIXED, __IS_SP, free_space);
+}
+
+/**
+ * Enqueue several objects on a ring.
+ *
+ * This function calls the multi-producer or the single-producer
+ * version depending on the default behavior that was specified at
+ * ring creation time (see flags).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of objects.
+ * @param esize
+ * The size of ring element, in bytes. It must be a multiple of 4.
+ * This must be the same value used while creating the ring. Otherwise
+ * the results are undefined.
+ * @param n
+ * The number of objects to add in the ring from the obj_table.
+ * @param free_space
+ * if non-NULL, returns the amount of space in the ring after the
+ * enqueue operation has finished.
+ * @return
+ * The number of objects enqueued, either 0 or n
+ */
+unsigned int
+rte_ring_enqueue_bulk_elem(struct rte_ring *r, const void *obj_table,
+ unsigned int esize, unsigned int n, unsigned int *free_space)
+{
+ return __rte_ring_do_enqueue_elem(r, obj_table, esize, n,
+ RTE_RING_QUEUE_FIXED, r->prod.single, free_space);
+}
+
+/**
+ * Enqueue one object on a ring (multi-producers safe).
+ *
+ * This function uses a "compare and set" instruction to move the
+ * producer index atomically.
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj
+ * A pointer to the object to be added.
+ * @param esize
+ * The size of ring element, in bytes. It must be a multiple of 4.
+ * This must be the same value used while creating the ring. Otherwise
+ * the results are undefined.
+ * @return
+ * - 0: Success; objects enqueued.
+ * - -ENOBUFS: Not enough room in the ring to enqueue; no object is enqueued.
+ */
+int
+rte_ring_mp_enqueue_elem(struct rte_ring *r, void *obj, unsigned int esize)
+{
+ return rte_ring_mp_enqueue_bulk_elem(r, obj, esize, 1, NULL) ? 0 :
+ -ENOBUFS;
+}
+
+/**
+ * Enqueue one object on a ring
+ *
+ * @warning This API is NOT multi-producers safe
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj
+ * A pointer to the object to be added.
+ * @param esize
+ * The size of ring element, in bytes. It must be a multiple of 4.
+ * This must be the same value used while creating the ring. Otherwise
+ * the results are undefined.
+ * @return
+ * - 0: Success; objects enqueued.
+ * - -ENOBUFS: Not enough room in the ring to enqueue; no object is enqueued.
+ */
+int
+rte_ring_sp_enqueue_elem(struct rte_ring *r, void *obj, unsigned int esize)
+{
+ return rte_ring_sp_enqueue_bulk_elem(r, obj, esize, 1, NULL) ? 0 :
+ -ENOBUFS;
+}
+
+/**
+ * Enqueue one object on a ring.
+ *
+ * This function calls the multi-producer or the single-producer
+ * version, depending on the default behaviour that was specified at
+ * ring creation time (see flags).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj
+ * A pointer to the object to be added.
+ * @param esize
+ * The size of ring element, in bytes. It must be a multiple of 4.
+ * This must be the same value used while creating the ring. Otherwise
+ * the results are undefined.
+ * @return
+ * - 0: Success; objects enqueued.
+ * - -ENOBUFS: Not enough room in the ring to enqueue; no object is enqueued.
+ */
+int
+rte_ring_enqueue_elem(struct rte_ring *r, void *obj, unsigned int esize)
+{
+ return rte_ring_enqueue_bulk_elem(r, obj, esize, 1, NULL) ? 0 :
+ -ENOBUFS;
+}
+
+/**
+ * Dequeue several objects from a ring (multi-consumers safe).
+ *
+ * This function uses a "compare and set" instruction to move the
+ * consumer index atomically.
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of objects that will be filled.
+ * @param esize
+ * The size of ring element, in bytes. It must be a multiple of 4.
+ * This must be the same value used while creating the ring. Otherwise
+ * the results are undefined.
+ * @param n
+ * The number of objects to dequeue from the ring to the obj_table.
+ * @param available
+ * If non-NULL, returns the number of remaining ring entries after the
+ * dequeue has finished.
+ * @return
+ * The number of objects dequeued, either 0 or n
+ */
+unsigned int
+rte_ring_mc_dequeue_bulk_elem(struct rte_ring *r, void *obj_table,
+ unsigned int esize, unsigned int n, unsigned int *available)
+{
+ return __rte_ring_do_dequeue_elem(r, obj_table, esize, n,
+ RTE_RING_QUEUE_FIXED, __IS_MC, available);
+}
+
+/**
+ * Dequeue several objects from a ring (NOT multi-consumers safe).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of objects that will be filled.
+ * @param esize
+ * The size of ring element, in bytes. It must be a multiple of 4.
+ * This must be the same value used while creating the ring. Otherwise
+ * the results are undefined.
+ * @param n
+ * The number of objects to dequeue from the ring to the obj_table,
+ * must be strictly positive.
+ * @param available
+ * If non-NULL, returns the number of remaining ring entries after the
+ * dequeue has finished.
+ * @return
+ * The number of objects dequeued, either 0 or n
+ */
+unsigned int
+rte_ring_sc_dequeue_bulk_elem(struct rte_ring *r, void *obj_table,
+ unsigned int esize, unsigned int n, unsigned int *available)
+{
+ return __rte_ring_do_dequeue_elem(r, obj_table, esize, n,
+ RTE_RING_QUEUE_FIXED, __IS_SC, available);
+}
+
+/**
+ * Dequeue several objects from a ring.
+ *
+ * This function calls the multi-consumers or the single-consumer
+ * version, depending on the default behaviour that was specified at
+ * ring creation time (see flags).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of objects that will be filled.
+ * @param esize
+ * The size of ring element, in bytes. It must be a multiple of 4.
+ * This must be the same value used while creating the ring. Otherwise
+ * the results are undefined.
+ * @param n
+ * The number of objects to dequeue from the ring to the obj_table.
+ * @param available
+ * If non-NULL, returns the number of remaining ring entries after the
+ * dequeue has finished.
+ * @return
+ * The number of objects dequeued, either 0 or n
+ */
+unsigned int
+rte_ring_dequeue_bulk_elem(struct rte_ring *r, void *obj_table,
+ unsigned int esize, unsigned int n, unsigned int *available)
+{
+ return __rte_ring_do_dequeue_elem(r, obj_table, esize, n,
+ RTE_RING_QUEUE_FIXED, r->cons.single, available);
+}
+
+/**
+ * Dequeue one object from a ring (multi-consumers safe).
+ *
+ * This function uses a "compare and set" instruction to move the
+ * consumer index atomically.
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_p
+ * A pointer to the object that will be filled.
+ * @param esize
+ * The size of ring element, in bytes. It must be a multiple of 4.
+ * This must be the same value used while creating the ring. Otherwise
+ * the results are undefined.
+ * @return
+ * - 0: Success; objects dequeued.
+ * - -ENOENT: Not enough entries in the ring to dequeue; no object is
+ * dequeued.
+ */
+int
+rte_ring_mc_dequeue_elem(struct rte_ring *r, void *obj_p,
+ unsigned int esize)
+{
+ return rte_ring_mc_dequeue_bulk_elem(r, obj_p, esize, 1, NULL) ? 0 :
+ -ENOENT;
+}
+
+/**
+ * Dequeue one object from a ring (NOT multi-consumers safe).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_p
+ * A pointer to the object that will be filled.
+ * @param esize
+ * The size of ring element, in bytes. It must be a multiple of 4.
+ * This must be the same value used while creating the ring. Otherwise
+ * the results are undefined.
+ * @return
+ * - 0: Success; objects dequeued.
+ * - -ENOENT: Not enough entries in the ring to dequeue, no object is
+ * dequeued.
+ */
+int
+rte_ring_sc_dequeue_elem(struct rte_ring *r, void *obj_p,
+ unsigned int esize)
+{
+ return rte_ring_sc_dequeue_bulk_elem(r, obj_p, esize, 1, NULL) ? 0 :
+ -ENOENT;
+}
+
+/**
+ * Dequeue one object from a ring.
+ *
+ * This function calls the multi-consumers or the single-consumer
+ * version depending on the default behaviour that was specified at
+ * ring creation time (see flags).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_p
+ * A pointer to the object that will be filled.
+ * @param esize
+ * The size of ring element, in bytes. It must be a multiple of 4.
+ * This must be the same value used while creating the ring. Otherwise
+ * the results are undefined.
+ * @return
+ * - 0: Success, objects dequeued.
+ * - -ENOENT: Not enough entries in the ring to dequeue, no object is
+ * dequeued.
+ */
+int
+rte_ring_dequeue_elem(struct rte_ring *r, void *obj_p, unsigned int esize)
+{
+ return rte_ring_dequeue_bulk_elem(r, obj_p, esize, 1, NULL) ? 0 :
+ -ENOENT;
+}
+
+/**
+ * Enqueue several objects on the ring (multi-producers safe).
+ *
+ * This function uses a "compare and set" instruction to move the
+ * producer index atomically.
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of objects.
+ * @param esize
+ * The size of ring element, in bytes. It must be a multiple of 4.
+ * This must be the same value used while creating the ring. Otherwise
+ * the results are undefined.
+ * @param n
+ * The number of objects to add in the ring from the obj_table.
+ * @param free_space
+ * if non-NULL, returns the amount of space in the ring after the
+ * enqueue operation has finished.
+ * @return
+ * - n: Actual number of objects enqueued.
+ */
+unsigned
+rte_ring_mp_enqueue_burst_elem(struct rte_ring *r, const void *obj_table,
+ unsigned int esize, unsigned int n, unsigned int *free_space)
+{
+ return __rte_ring_do_enqueue_elem(r, obj_table, esize, n,
+ RTE_RING_QUEUE_VARIABLE, __IS_MP, free_space);
+}
+
+/**
+ * Enqueue several objects on a ring
+ *
+ * @warning This API is NOT multi-producers safe
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of objects.
+ * @param esize
+ * The size of ring element, in bytes. It must be a multiple of 4.
+ * This must be the same value used while creating the ring. Otherwise
+ * the results are undefined.
+ * @param n
+ * The number of objects to add in the ring from the obj_table.
+ * @param free_space
+ * if non-NULL, returns the amount of space in the ring after the
+ * enqueue operation has finished.
+ * @return
+ * - n: Actual number of objects enqueued.
+ */
+unsigned
+rte_ring_sp_enqueue_burst_elem(struct rte_ring *r, const void *obj_table,
+ unsigned int esize, unsigned int n, unsigned int *free_space)
+{
+ return __rte_ring_do_enqueue_elem(r, obj_table, esize, n,
+ RTE_RING_QUEUE_VARIABLE, __IS_SP, free_space);
+}
+
+/**
+ * Enqueue several objects on a ring.
+ *
+ * This function calls the multi-producer or the single-producer
+ * version depending on the default behavior that was specified at
+ * ring creation time (see flags).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of objects.
+ * @param esize
+ * The size of ring element, in bytes. It must be a multiple of 4.
+ * This must be the same value used while creating the ring. Otherwise
+ * the results are undefined.
+ * @param n
+ * The number of objects to add in the ring from the obj_table.
+ * @param free_space
+ * if non-NULL, returns the amount of space in the ring after the
+ * enqueue operation has finished.
+ * @return
+ * - n: Actual number of objects enqueued.
+ */
+unsigned
+rte_ring_enqueue_burst_elem(struct rte_ring *r, const void *obj_table,
+ unsigned int esize, unsigned int n, unsigned int *free_space)
+{
+ return __rte_ring_do_enqueue_elem(r, obj_table, esize, n,
+ RTE_RING_QUEUE_VARIABLE, r->prod.single, free_space);
+}
+
+/**
+ * Dequeue several objects from a ring (multi-consumers safe). When the request
+ * objects are more than the available objects, only dequeue the actual number
+ * of objects
+ *
+ * This function uses a "compare and set" instruction to move the
+ * consumer index atomically.
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of objects that will be filled.
+ * @param esize
+ * The size of ring element, in bytes. It must be a multiple of 4.
+ * This must be the same value used while creating the ring. Otherwise
+ * the results are undefined.
+ * @param n
+ * The number of objects to dequeue from the ring to the obj_table.
+ * @param available
+ * If non-NULL, returns the number of remaining ring entries after the
+ * dequeue has finished.
+ * @return
+ * - n: Actual number of objects dequeued, 0 if ring is empty
+ */
+unsigned
+rte_ring_mc_dequeue_burst_elem(struct rte_ring *r, void *obj_table,
+ unsigned int esize, unsigned int n, unsigned int *available)
+{
+ return __rte_ring_do_dequeue_elem(r, obj_table, esize, n,
+ RTE_RING_QUEUE_VARIABLE, __IS_MC, available);
+}
+
+/**
+ * Dequeue several objects from a ring (NOT multi-consumers safe).When the
+ * request objects are more than the available objects, only dequeue the
+ * actual number of objects
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of objects that will be filled.
+ * @param esize
+ * The size of ring element, in bytes. It must be a multiple of 4.
+ * This must be the same value used while creating the ring. Otherwise
+ * the results are undefined.
+ * @param n
+ * The number of objects to dequeue from the ring to the obj_table.
+ * @param available
+ * If non-NULL, returns the number of remaining ring entries after the
+ * dequeue has finished.
+ * @return
+ * - n: Actual number of objects dequeued, 0 if ring is empty
+ */
+unsigned
+rte_ring_sc_dequeue_burst_elem(struct rte_ring *r, void *obj_table,
+ unsigned int esize, unsigned int n, unsigned int *available)
+{
+ return __rte_ring_do_dequeue_elem(r, obj_table, esize, n,
+ RTE_RING_QUEUE_VARIABLE, __IS_SC, available);
+}
+
+/**
+ * Dequeue multiple objects from a ring up to a maximum number.
+ *
+ * This function calls the multi-consumers or the single-consumer
+ * version, depending on the default behaviour that was specified at
+ * ring creation time (see flags).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of objects that will be filled.
+ * @param esize
+ * The size of ring element, in bytes. It must be a multiple of 4.
+ * This must be the same value used while creating the ring. Otherwise
+ * the results are undefined.
+ * @param n
+ * The number of objects to dequeue from the ring to the obj_table.
+ * @param available
+ * If non-NULL, returns the number of remaining ring entries after the
+ * dequeue has finished.
+ * @return
+ * - Number of objects dequeued
+ */
+unsigned int
+rte_ring_dequeue_burst_elem(struct rte_ring *r, void *obj_table,
+ unsigned int esize, unsigned int n, unsigned int *available)
+{
+ return __rte_ring_do_dequeue_elem(r, obj_table, esize, n,
+ RTE_RING_QUEUE_VARIABLE,
+ r->cons.single, available);
+}
diff --git a/lib/librte_ring/ring_impl.c b/lib/librte_ring/ring_impl.c
new file mode 100644
index 000000000..ee85c515f
--- /dev/null
+++ b/lib/librte_ring/ring_impl.c
@@ -0,0 +1,750 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ *
+ * Copyright (c) 2010-2017 Intel Corporation
+ * Copyright (c) 2007-2009 Kip Macy kmacy@freebsd.org
+ * All rights reserved.
+ * Derived from FreeBSD's bufring.h
+ * Used as BSD-3 Licensed with permission from Kip Macy.
+ */
+
+#include <stdio.h>
+#include <stdint.h>
+#include <sys/queue.h>
+#include <errno.h>
+#include <rte_common.h>
+#include <rte_config.h>
+#include <rte_memory.h>
+#include <rte_lcore.h>
+#include <rte_atomic.h>
+#include <rte_branch_prediction.h>
+#include <rte_memzone.h>
+#include <rte_pause.h>
+#include "ring_impl.h"
+
+/* the actual enqueue of pointers on the ring.
+ * Placed here since identical code needed in both
+ * single and multi producer enqueue functions */
+#define ENQUEUE_PTRS(r, ring_start, prod_head, obj_table, n, obj_type) do { \
+ unsigned int i; \
+ const uint32_t size = (r)->size; \
+ uint32_t idx = prod_head & (r)->mask; \
+ obj_type *ring = (obj_type *)ring_start; \
+ if (likely(idx + n < size)) { \
+ for (i = 0; i < (n & ((~(unsigned)0x3))); i+=4, idx+=4) { \
+ ring[idx] = obj_table[i]; \
+ ring[idx+1] = obj_table[i+1]; \
+ ring[idx+2] = obj_table[i+2]; \
+ ring[idx+3] = obj_table[i+3]; \
+ } \
+ switch (n & 0x3) { \
+ case 3: \
+ ring[idx++] = obj_table[i++]; /* fallthrough */ \
+ case 2: \
+ ring[idx++] = obj_table[i++]; /* fallthrough */ \
+ case 1: \
+ ring[idx++] = obj_table[i++]; \
+ } \
+ } else { \
+ for (i = 0; idx < size; i++, idx++)\
+ ring[idx] = obj_table[i]; \
+ for (idx = 0; i < n; i++, idx++) \
+ ring[idx] = obj_table[i]; \
+ } \
+} while (0)
+
+/* the actual copy of pointers on the ring to obj_table.
+ * Placed here since identical code needed in both
+ * single and multi consumer dequeue functions */
+#define DEQUEUE_PTRS(r, ring_start, cons_head, obj_table, n, obj_type) do { \
+ unsigned int i; \
+ uint32_t idx = cons_head & (r)->mask; \
+ const uint32_t size = (r)->size; \
+ obj_type *ring = (obj_type *)ring_start; \
+ if (likely(idx + n < size)) { \
+ for (i = 0; i < (n & (~(unsigned)0x3)); i+=4, idx+=4) {\
+ obj_table[i] = ring[idx]; \
+ obj_table[i+1] = ring[idx+1]; \
+ obj_table[i+2] = ring[idx+2]; \
+ obj_table[i+3] = ring[idx+3]; \
+ } \
+ switch (n & 0x3) { \
+ case 3: \
+ obj_table[i++] = ring[idx++]; /* fallthrough */ \
+ case 2: \
+ obj_table[i++] = ring[idx++]; /* fallthrough */ \
+ case 1: \
+ obj_table[i++] = ring[idx++]; \
+ } \
+ } else { \
+ for (i = 0; idx < size; i++, idx++) \
+ obj_table[i] = ring[idx]; \
+ for (idx = 0; i < n; i++, idx++) \
+ obj_table[i] = ring[idx]; \
+ } \
+} while (0)
+
+/* Between load and load. there might be cpu reorder in weak model
+ * (powerpc/arm).
+ * There are 2 choices for the users
+ * 1.use rmb() memory barrier
+ * 2.use one-direction load_acquire/store_release barrier,defined by
+ * CONFIG_RTE_USE_C11_MEM_MODEL=y
+ * It depends on performance test results.
+ * By default, move common functions to rte_ring_generic.h
+ */
+#ifdef RTE_USE_C11_MEM_MODEL
+#include "rte_ring_c11_mem.h"
+#else
+#include "rte_ring_generic.h"
+#endif
+
+/**
+ * @internal Enqueue several objects on the ring
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of void * pointers (objects).
+ * @param n
+ * The number of objects to add in the ring from the obj_table.
+ * @param behavior
+ * RTE_RING_QUEUE_FIXED: Enqueue a fixed number of items from a ring
+ * RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible from ring
+ * @param is_sp
+ * Indicates whether to use single producer or multi-producer head update
+ * @param free_space
+ * returns the amount of space after the enqueue operation has finished
+ * @return
+ * Actual number of objects enqueued.
+ * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_enqueue(struct rte_ring *r, void * const *obj_table,
+ unsigned int n, enum rte_ring_queue_behavior behavior,
+ unsigned int is_sp, unsigned int *free_space)
+{
+ uint32_t prod_head, prod_next;
+ uint32_t free_entries;
+
+ n = __rte_ring_move_prod_head(r, is_sp, n, behavior,
+ &prod_head, &prod_next, &free_entries);
+ if (n == 0)
+ goto end;
+
+ ENQUEUE_PTRS(r, &r[1], prod_head, obj_table, n, void *);
+
+ update_tail(&r->prod, prod_head, prod_next, is_sp, 1);
+end:
+ if (free_space != NULL)
+ *free_space = free_entries - n;
+ return n;
+}
+
+/**
+ * @internal Dequeue several objects from the ring
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of void * pointers (objects).
+ * @param n
+ * The number of objects to pull from the ring.
+ * @param behavior
+ * RTE_RING_QUEUE_FIXED: Dequeue a fixed number of items from a ring
+ * RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from ring
+ * @param is_sc
+ * Indicates whether to use single consumer or multi-consumer head update
+ * @param available
+ * returns the number of remaining ring entries after the dequeue has finished
+ * @return
+ * - Actual number of objects dequeued.
+ * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_dequeue(struct rte_ring *r, void **obj_table,
+ unsigned int n, enum rte_ring_queue_behavior behavior,
+ unsigned int is_sc, unsigned int *available)
+{
+ uint32_t cons_head, cons_next;
+ uint32_t entries;
+
+ n = __rte_ring_move_cons_head(r, (int)is_sc, n, behavior,
+ &cons_head, &cons_next, &entries);
+ if (n == 0)
+ goto end;
+
+ DEQUEUE_PTRS(r, &r[1], cons_head, obj_table, n, void *);
+
+ update_tail(&r->cons, cons_head, cons_next, is_sc, 0);
+
+end:
+ if (available != NULL)
+ *available = entries - n;
+ return n;
+}
+
+/**
+ * Enqueue several objects on the ring (multi-producers safe).
+ *
+ * This function uses a "compare and set" instruction to move the
+ * producer index atomically.
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of void * pointers (objects).
+ * @param n
+ * The number of objects to add in the ring from the obj_table.
+ * @param free_space
+ * if non-NULL, returns the amount of space in the ring after the
+ * enqueue operation has finished.
+ * @return
+ * The number of objects enqueued, either 0 or n
+ */
+unsigned int
+rte_ring_mp_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
+ unsigned int n, unsigned int *free_space)
+{
+ return __rte_ring_do_enqueue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
+ __IS_MP, free_space);
+}
+
+/**
+ * Enqueue several objects on a ring (NOT multi-producers safe).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of void * pointers (objects).
+ * @param n
+ * The number of objects to add in the ring from the obj_table.
+ * @param free_space
+ * if non-NULL, returns the amount of space in the ring after the
+ * enqueue operation has finished.
+ * @return
+ * The number of objects enqueued, either 0 or n
+ */
+unsigned int
+rte_ring_sp_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
+ unsigned int n, unsigned int *free_space)
+{
+ return __rte_ring_do_enqueue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
+ __IS_SP, free_space);
+}
+
+/**
+ * Enqueue several objects on a ring.
+ *
+ * This function calls the multi-producer or the single-producer
+ * version depending on the default behavior that was specified at
+ * ring creation time (see flags).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of void * pointers (objects).
+ * @param n
+ * The number of objects to add in the ring from the obj_table.
+ * @param free_space
+ * if non-NULL, returns the amount of space in the ring after the
+ * enqueue operation has finished.
+ * @return
+ * The number of objects enqueued, either 0 or n
+ */
+unsigned int
+rte_ring_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
+ unsigned int n, unsigned int *free_space)
+{
+ return __rte_ring_do_enqueue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
+ r->prod.single, free_space);
+}
+
+/**
+ * Enqueue one object on a ring (multi-producers safe).
+ *
+ * This function uses a "compare and set" instruction to move the
+ * producer index atomically.
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj
+ * A pointer to the object to be added.
+ * @return
+ * - 0: Success; objects enqueued.
+ * - -ENOBUFS: Not enough room in the ring to enqueue; no object is enqueued.
+ */
+int
+rte_ring_mp_enqueue(struct rte_ring *r, void *obj)
+{
+ return rte_ring_mp_enqueue_bulk(r, &obj, 1, NULL) ? 0 : -ENOBUFS;
+}
+
+/**
+ * Enqueue one object on a ring (NOT multi-producers safe).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj
+ * A pointer to the object to be added.
+ * @return
+ * - 0: Success; objects enqueued.
+ * - -ENOBUFS: Not enough room in the ring to enqueue; no object is enqueued.
+ */
+int
+rte_ring_sp_enqueue(struct rte_ring *r, void *obj)
+{
+ return rte_ring_sp_enqueue_bulk(r, &obj, 1, NULL) ? 0 : -ENOBUFS;
+}
+
+/**
+ * Enqueue one object on a ring.
+ *
+ * This function calls the multi-producer or the single-producer
+ * version, depending on the default behaviour that was specified at
+ * ring creation time (see flags).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj
+ * A pointer to the object to be added.
+ * @return
+ * - 0: Success; objects enqueued.
+ * - -ENOBUFS: Not enough room in the ring to enqueue; no object is enqueued.
+ */
+int
+rte_ring_enqueue(struct rte_ring *r, void *obj)
+{
+ return rte_ring_enqueue_bulk(r, &obj, 1, NULL) ? 0 : -ENOBUFS;
+}
+
+/**
+ * Dequeue several objects from a ring (multi-consumers safe).
+ *
+ * This function uses a "compare and set" instruction to move the
+ * consumer index atomically.
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of void * pointers (objects) that will be filled.
+ * @param n
+ * The number of objects to dequeue from the ring to the obj_table.
+ * @param available
+ * If non-NULL, returns the number of remaining ring entries after the
+ * dequeue has finished.
+ * @return
+ * The number of objects dequeued, either 0 or n
+ */
+unsigned int
+rte_ring_mc_dequeue_bulk(struct rte_ring *r, void **obj_table,
+ unsigned int n, unsigned int *available)
+{
+ return __rte_ring_do_dequeue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
+ __IS_MC, available);
+}
+
+/**
+ * Dequeue several objects from a ring (NOT multi-consumers safe).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of void * pointers (objects) that will be filled.
+ * @param n
+ * The number of objects to dequeue from the ring to the obj_table,
+ * must be strictly positive.
+ * @param available
+ * If non-NULL, returns the number of remaining ring entries after the
+ * dequeue has finished.
+ * @return
+ * The number of objects dequeued, either 0 or n
+ */
+unsigned int
+rte_ring_sc_dequeue_bulk(struct rte_ring *r, void **obj_table,
+ unsigned int n, unsigned int *available)
+{
+ return __rte_ring_do_dequeue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
+ __IS_SC, available);
+}
+
+/**
+ * Dequeue several objects from a ring.
+ *
+ * This function calls the multi-consumers or the single-consumer
+ * version, depending on the default behaviour that was specified at
+ * ring creation time (see flags).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of void * pointers (objects) that will be filled.
+ * @param n
+ * The number of objects to dequeue from the ring to the obj_table.
+ * @param available
+ * If non-NULL, returns the number of remaining ring entries after the
+ * dequeue has finished.
+ * @return
+ * The number of objects dequeued, either 0 or n
+ */
+unsigned int
+rte_ring_dequeue_bulk(struct rte_ring *r, void **obj_table, unsigned int n,
+ unsigned int *available)
+{
+ return __rte_ring_do_dequeue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
+ r->cons.single, available);
+}
+
+/**
+ * Dequeue one object from a ring (multi-consumers safe).
+ *
+ * This function uses a "compare and set" instruction to move the
+ * consumer index atomically.
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_p
+ * A pointer to a void * pointer (object) that will be filled.
+ * @return
+ * - 0: Success; objects dequeued.
+ * - -ENOENT: Not enough entries in the ring to dequeue; no object is
+ * dequeued.
+ */
+int
+rte_ring_mc_dequeue(struct rte_ring *r, void **obj_p)
+{
+ return rte_ring_mc_dequeue_bulk(r, obj_p, 1, NULL) ? 0 : -ENOENT;
+}
+
+/**
+ * Dequeue one object from a ring (NOT multi-consumers safe).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_p
+ * A pointer to a void * pointer (object) that will be filled.
+ * @return
+ * - 0: Success; objects dequeued.
+ * - -ENOENT: Not enough entries in the ring to dequeue, no object is
+ * dequeued.
+ */
+int
+rte_ring_sc_dequeue(struct rte_ring *r, void **obj_p)
+{
+ return rte_ring_sc_dequeue_bulk(r, obj_p, 1, NULL) ? 0 : -ENOENT;
+}
+
+/**
+ * Dequeue one object from a ring.
+ *
+ * This function calls the multi-consumers or the single-consumer
+ * version depending on the default behaviour that was specified at
+ * ring creation time (see flags).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_p
+ * A pointer to a void * pointer (object) that will be filled.
+ * @return
+ * - 0: Success, objects dequeued.
+ * - -ENOENT: Not enough entries in the ring to dequeue, no object is
+ * dequeued.
+ */
+int
+rte_ring_dequeue(struct rte_ring *r, void **obj_p)
+{
+ return rte_ring_dequeue_bulk(r, obj_p, 1, NULL) ? 0 : -ENOENT;
+}
+
+/**
+ * Return the number of entries in a ring.
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @return
+ * The number of entries in the ring.
+ */
+unsigned
+rte_ring_count(const struct rte_ring *r)
+{
+ uint32_t prod_tail = r->prod.tail;
+ uint32_t cons_tail = r->cons.tail;
+ uint32_t count = (prod_tail - cons_tail) & r->mask;
+ return (count > r->capacity) ? r->capacity : count;
+}
+
+/**
+ * Return the number of free entries in a ring.
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @return
+ * The number of free entries in the ring.
+ */
+unsigned
+rte_ring_free_count(const struct rte_ring *r)
+{
+ return r->capacity - rte_ring_count(r);
+}
+
+/**
+ * Test if a ring is full.
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @return
+ * - 1: The ring is full.
+ * - 0: The ring is not full.
+ */
+int
+rte_ring_full(const struct rte_ring *r)
+{
+ return rte_ring_free_count(r) == 0;
+}
+
+/**
+ * Test if a ring is empty.
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @return
+ * - 1: The ring is empty.
+ * - 0: The ring is not empty.
+ */
+int
+rte_ring_empty(const struct rte_ring *r)
+{
+ return rte_ring_count(r) == 0;
+}
+
+const char *
+rte_ring_get_name(const struct rte_ring *r)
+{
+ return r->name;
+}
+
+const struct rte_memzone *
+rte_ring_get_memzone(const struct rte_ring *r)
+{
+ return r->memzone;
+}
+
+/**
+ * Return the size of the ring.
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @return
+ * The size of the data store used by the ring.
+ * NOTE: this is not the same as the usable space in the ring. To query that
+ * use ``rte_ring_get_capacity()``.
+ */
+unsigned int
+rte_ring_get_size(const struct rte_ring *r)
+{
+ return r->size;
+}
+
+/**
+ * Return the number of elements which can be stored in the ring.
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @return
+ * The usable size of the ring.
+ */
+unsigned int
+rte_ring_get_capacity(const struct rte_ring *r)
+{
+ return r->capacity;
+}
+
+uint32_t
+rte_ring_get_prod_sync_type(const struct rte_ring *r)
+{
+ return (r->prod.single == __IS_SP) ?
+ RTE_RING_SYNC_ST : RTE_RING_SYNC_MT;
+}
+
+uint32_t
+rte_ring_get_cons_sync_type(const struct rte_ring *r)
+{
+ return (r->cons.single == __IS_SC) ?
+ RTE_RING_SYNC_ST : RTE_RING_SYNC_MT;
+}
+
+/**
+ * Dump the status of all rings on the console
+ *
+ * @param f
+ * A pointer to a file for output
+ */
+void rte_ring_list_dump(FILE *f);
+
+/**
+ * Search a ring from its name
+ *
+ * @param name
+ * The name of the ring.
+ * @return
+ * The pointer to the ring matching the name, or NULL if not found,
+ * with rte_errno set appropriately. Possible rte_errno values include:
+ * - ENOENT - required entry not available to return.
+ */
+struct rte_ring *rte_ring_lookup(const char *name);
+
+/**
+ * Enqueue several objects on the ring (multi-producers safe).
+ *
+ * This function uses a "compare and set" instruction to move the
+ * producer index atomically.
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of void * pointers (objects).
+ * @param n
+ * The number of objects to add in the ring from the obj_table.
+ * @param free_space
+ * if non-NULL, returns the amount of space in the ring after the
+ * enqueue operation has finished.
+ * @return
+ * - n: Actual number of objects enqueued.
+ */
+unsigned
+rte_ring_mp_enqueue_burst(struct rte_ring *r, void * const *obj_table,
+ unsigned int n, unsigned int *free_space)
+{
+ return __rte_ring_do_enqueue(r, obj_table, n,
+ RTE_RING_QUEUE_VARIABLE, __IS_MP, free_space);
+}
+
+/**
+ * Enqueue several objects on a ring (NOT multi-producers safe).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of void * pointers (objects).
+ * @param n
+ * The number of objects to add in the ring from the obj_table.
+ * @param free_space
+ * if non-NULL, returns the amount of space in the ring after the
+ * enqueue operation has finished.
+ * @return
+ * - n: Actual number of objects enqueued.
+ */
+unsigned
+rte_ring_sp_enqueue_burst(struct rte_ring *r, void * const *obj_table,
+ unsigned int n, unsigned int *free_space)
+{
+ return __rte_ring_do_enqueue(r, obj_table, n,
+ RTE_RING_QUEUE_VARIABLE, __IS_SP, free_space);
+}
+
+/**
+ * Enqueue several objects on a ring.
+ *
+ * This function calls the multi-producer or the single-producer
+ * version depending on the default behavior that was specified at
+ * ring creation time (see flags).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of void * pointers (objects).
+ * @param n
+ * The number of objects to add in the ring from the obj_table.
+ * @param free_space
+ * if non-NULL, returns the amount of space in the ring after the
+ * enqueue operation has finished.
+ * @return
+ * - n: Actual number of objects enqueued.
+ */
+unsigned
+rte_ring_enqueue_burst(struct rte_ring *r, void * const *obj_table,
+ unsigned int n, unsigned int *free_space)
+{
+ return __rte_ring_do_enqueue(r, obj_table, n, RTE_RING_QUEUE_VARIABLE,
+ r->prod.single, free_space);
+}
+
+/**
+ * Dequeue several objects from a ring (multi-consumers safe). When the request
+ * objects are more than the available objects, only dequeue the actual number
+ * of objects
+ *
+ * This function uses a "compare and set" instruction to move the
+ * consumer index atomically.
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of void * pointers (objects) that will be filled.
+ * @param n
+ * The number of objects to dequeue from the ring to the obj_table.
+ * @param available
+ * If non-NULL, returns the number of remaining ring entries after the
+ * dequeue has finished.
+ * @return
+ * - n: Actual number of objects dequeued, 0 if ring is empty
+ */
+unsigned
+rte_ring_mc_dequeue_burst(struct rte_ring *r, void **obj_table,
+ unsigned int n, unsigned int *available)
+{
+ return __rte_ring_do_dequeue(r, obj_table, n,
+ RTE_RING_QUEUE_VARIABLE, __IS_MC, available);
+}
+
+/**
+ * Dequeue several objects from a ring (NOT multi-consumers safe).When the
+ * request objects are more than the available objects, only dequeue the
+ * actual number of objects
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of void * pointers (objects) that will be filled.
+ * @param n
+ * The number of objects to dequeue from the ring to the obj_table.
+ * @param available
+ * If non-NULL, returns the number of remaining ring entries after the
+ * dequeue has finished.
+ * @return
+ * - n: Actual number of objects dequeued, 0 if ring is empty
+ */
+unsigned
+rte_ring_sc_dequeue_burst(struct rte_ring *r, void **obj_table,
+ unsigned int n, unsigned int *available)
+{
+ return __rte_ring_do_dequeue(r, obj_table, n,
+ RTE_RING_QUEUE_VARIABLE, __IS_SC, available);
+}
+
+/**
+ * Dequeue multiple objects from a ring up to a maximum number.
+ *
+ * This function calls the multi-consumers or the single-consumer
+ * version, depending on the default behaviour that was specified at
+ * ring creation time (see flags).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of void * pointers (objects) that will be filled.
+ * @param n
+ * The number of objects to dequeue from the ring to the obj_table.
+ * @param available
+ * If non-NULL, returns the number of remaining ring entries after the
+ * dequeue has finished.
+ * @return
+ * - Number of objects dequeued
+ */
+unsigned
+rte_ring_dequeue_burst(struct rte_ring *r, void **obj_table,
+ unsigned int n, unsigned int *available)
+{
+ return __rte_ring_do_dequeue(r, obj_table, n,
+ RTE_RING_QUEUE_VARIABLE,
+ r->cons.single, available);
+}
diff --git a/lib/librte_ring/ring_impl.h b/lib/librte_ring/ring_impl.h
new file mode 100644
index 000000000..4ed375511
--- /dev/null
+++ b/lib/librte_ring/ring_impl.h
@@ -0,0 +1,64 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ *
+ * Copyright (c) 2010-2017 Intel Corporation
+ * Copyright (c) 2007-2009 Kip Macy kmacy@freebsd.org
+ * All rights reserved.
+ * Derived from FreeBSD's bufring.h
+ * Used as BSD-3 Licensed with permission from Kip Macy.
+ */
+
+#ifndef _RING_IMPL_H_
+#define _RING_IMPL_H_
+
+#include <rte_ring.h>
+#include <rte_ring_elem.h>
+
+/* @internal defines for passing to the enqueue dequeue worker functions */
+#define __IS_SP 1
+#define __IS_MP 0
+#define __IS_SC 1
+#define __IS_MC 0
+
+/* structure to hold a pair of head/tail values and other metadata */
+struct rte_ring_headtail {
+ volatile uint32_t head; /**< Prod/consumer head. */
+ volatile uint32_t tail; /**< Prod/consumer tail. */
+ uint32_t single; /**< True if single prod/cons */
+};
+
+/**
+ * An RTE ring structure.
+ *
+ * The producer and the consumer have a head and a tail index. The particularity
+ * of these index is that they are not between 0 and size(ring). These indexes
+ * are between 0 and 2^32, and we mask their value when we access the ring[]
+ * field. Thanks to this assumption, we can do subtractions between 2 index
+ * values in a modulo-32bit base: that's why the overflow of the indexes is not
+ * a problem.
+ */
+struct rte_ring {
+ /*
+ * Note: this field kept the RTE_MEMZONE_NAMESIZE size due to ABI
+ * compatibility requirements, it could be changed to RTE_RING_NAMESIZE
+ * next time the ABI changes
+ */
+ char name[RTE_MEMZONE_NAMESIZE] __rte_cache_aligned; /**< Name of the ring. */
+ int flags; /**< Flags supplied at creation. */
+ const struct rte_memzone *memzone;
+ /**< Memzone, if any, containing the rte_ring */
+ uint32_t size; /**< Size of ring. */
+ uint32_t mask; /**< Mask (size-1) of ring. */
+ uint32_t capacity; /**< Usable size of ring */
+
+ char pad0 __rte_cache_aligned; /**< empty cache line */
+
+ /** Ring producer status. */
+ struct rte_ring_headtail prod __rte_cache_aligned;
+ char pad1 __rte_cache_aligned; /**< empty cache line */
+
+ /** Ring consumer status. */
+ struct rte_ring_headtail cons __rte_cache_aligned;
+ char pad2 __rte_cache_aligned; /**< empty cache line */
+};
+
+#endif /* _RING_IMPL_H_ */
diff --git a/lib/librte_ring/rte_ring.c b/lib/librte_ring/rte_ring.c
index 77e5de099..ec9392585 100644
--- a/lib/librte_ring/rte_ring.c
+++ b/lib/librte_ring/rte_ring.c
@@ -32,8 +32,7 @@
#include <rte_spinlock.h>
#include <rte_tailq.h>
-#include "rte_ring.h"
-#include "rte_ring_elem.h"
+#include "ring_impl.h"
TAILQ_HEAD(rte_ring_list, rte_tailq_entry);
diff --git a/lib/librte_ring/rte_ring.h b/lib/librte_ring/rte_ring.h
index 18fc5d845..a1442c360 100644
--- a/lib/librte_ring/rte_ring.h
+++ b/lib/librte_ring/rte_ring.h
@@ -56,53 +56,17 @@ enum rte_ring_queue_behavior {
RTE_RING_QUEUE_VARIABLE /* Enq/Deq as many items as possible from ring */
};
+/** prod/cons sync types */
+enum {
+ RTE_RING_SYNC_MT, /**< multi-thread safe (default mode) */
+ RTE_RING_SYNC_ST, /**< single thread only */
+};
+
#define RTE_RING_MZ_PREFIX "RG_"
/** The maximum length of a ring name. */
#define RTE_RING_NAMESIZE (RTE_MEMZONE_NAMESIZE - \
sizeof(RTE_RING_MZ_PREFIX) + 1)
-/* structure to hold a pair of head/tail values and other metadata */
-struct rte_ring_headtail {
- volatile uint32_t head; /**< Prod/consumer head. */
- volatile uint32_t tail; /**< Prod/consumer tail. */
- uint32_t single; /**< True if single prod/cons */
-};
-
-/**
- * An RTE ring structure.
- *
- * The producer and the consumer have a head and a tail index. The particularity
- * of these index is that they are not between 0 and size(ring). These indexes
- * are between 0 and 2^32, and we mask their value when we access the ring[]
- * field. Thanks to this assumption, we can do subtractions between 2 index
- * values in a modulo-32bit base: that's why the overflow of the indexes is not
- * a problem.
- */
-struct rte_ring {
- /*
- * Note: this field kept the RTE_MEMZONE_NAMESIZE size due to ABI
- * compatibility requirements, it could be changed to RTE_RING_NAMESIZE
- * next time the ABI changes
- */
- char name[RTE_MEMZONE_NAMESIZE] __rte_cache_aligned; /**< Name of the ring. */
- int flags; /**< Flags supplied at creation. */
- const struct rte_memzone *memzone;
- /**< Memzone, if any, containing the rte_ring */
- uint32_t size; /**< Size of ring. */
- uint32_t mask; /**< Mask (size-1) of ring. */
- uint32_t capacity; /**< Usable size of ring */
-
- char pad0 __rte_cache_aligned; /**< empty cache line */
-
- /** Ring producer status. */
- struct rte_ring_headtail prod __rte_cache_aligned;
- char pad1 __rte_cache_aligned; /**< empty cache line */
-
- /** Ring consumer status. */
- struct rte_ring_headtail cons __rte_cache_aligned;
- char pad2 __rte_cache_aligned; /**< empty cache line */
-};
-
#define RING_F_SP_ENQ 0x0001 /**< The default enqueue is "single-producer". */
#define RING_F_SC_DEQ 0x0002 /**< The default dequeue is "single-consumer". */
/**
@@ -116,11 +80,7 @@ struct rte_ring {
#define RING_F_EXACT_SZ 0x0004
#define RTE_RING_SZ_MASK (0x7fffffffU) /**< Ring size mask */
-/* @internal defines for passing to the enqueue dequeue worker functions */
-#define __IS_SP 1
-#define __IS_MP 0
-#define __IS_SC 1
-#define __IS_MC 0
+struct rte_ring;
/**
* Calculate the memory size needed for a ring
@@ -235,168 +195,6 @@ void rte_ring_free(struct rte_ring *r);
*/
void rte_ring_dump(FILE *f, const struct rte_ring *r);
-/* the actual enqueue of pointers on the ring.
- * Placed here since identical code needed in both
- * single and multi producer enqueue functions */
-#define ENQUEUE_PTRS(r, ring_start, prod_head, obj_table, n, obj_type) do { \
- unsigned int i; \
- const uint32_t size = (r)->size; \
- uint32_t idx = prod_head & (r)->mask; \
- obj_type *ring = (obj_type *)ring_start; \
- if (likely(idx + n < size)) { \
- for (i = 0; i < (n & ((~(unsigned)0x3))); i+=4, idx+=4) { \
- ring[idx] = obj_table[i]; \
- ring[idx+1] = obj_table[i+1]; \
- ring[idx+2] = obj_table[i+2]; \
- ring[idx+3] = obj_table[i+3]; \
- } \
- switch (n & 0x3) { \
- case 3: \
- ring[idx++] = obj_table[i++]; /* fallthrough */ \
- case 2: \
- ring[idx++] = obj_table[i++]; /* fallthrough */ \
- case 1: \
- ring[idx++] = obj_table[i++]; \
- } \
- } else { \
- for (i = 0; idx < size; i++, idx++)\
- ring[idx] = obj_table[i]; \
- for (idx = 0; i < n; i++, idx++) \
- ring[idx] = obj_table[i]; \
- } \
-} while (0)
-
-/* the actual copy of pointers on the ring to obj_table.
- * Placed here since identical code needed in both
- * single and multi consumer dequeue functions */
-#define DEQUEUE_PTRS(r, ring_start, cons_head, obj_table, n, obj_type) do { \
- unsigned int i; \
- uint32_t idx = cons_head & (r)->mask; \
- const uint32_t size = (r)->size; \
- obj_type *ring = (obj_type *)ring_start; \
- if (likely(idx + n < size)) { \
- for (i = 0; i < (n & (~(unsigned)0x3)); i+=4, idx+=4) {\
- obj_table[i] = ring[idx]; \
- obj_table[i+1] = ring[idx+1]; \
- obj_table[i+2] = ring[idx+2]; \
- obj_table[i+3] = ring[idx+3]; \
- } \
- switch (n & 0x3) { \
- case 3: \
- obj_table[i++] = ring[idx++]; /* fallthrough */ \
- case 2: \
- obj_table[i++] = ring[idx++]; /* fallthrough */ \
- case 1: \
- obj_table[i++] = ring[idx++]; \
- } \
- } else { \
- for (i = 0; idx < size; i++, idx++) \
- obj_table[i] = ring[idx]; \
- for (idx = 0; i < n; i++, idx++) \
- obj_table[i] = ring[idx]; \
- } \
-} while (0)
-
-/* Between load and load. there might be cpu reorder in weak model
- * (powerpc/arm).
- * There are 2 choices for the users
- * 1.use rmb() memory barrier
- * 2.use one-direction load_acquire/store_release barrier,defined by
- * CONFIG_RTE_USE_C11_MEM_MODEL=y
- * It depends on performance test results.
- * By default, move common functions to rte_ring_generic.h
- */
-#ifdef RTE_USE_C11_MEM_MODEL
-#include "rte_ring_c11_mem.h"
-#else
-#include "rte_ring_generic.h"
-#endif
-
-/**
- * @internal Enqueue several objects on the ring
- *
- * @param r
- * A pointer to the ring structure.
- * @param obj_table
- * A pointer to a table of void * pointers (objects).
- * @param n
- * The number of objects to add in the ring from the obj_table.
- * @param behavior
- * RTE_RING_QUEUE_FIXED: Enqueue a fixed number of items from a ring
- * RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible from ring
- * @param is_sp
- * Indicates whether to use single producer or multi-producer head update
- * @param free_space
- * returns the amount of space after the enqueue operation has finished
- * @return
- * Actual number of objects enqueued.
- * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
- */
-static __rte_always_inline unsigned int
-__rte_ring_do_enqueue(struct rte_ring *r, void * const *obj_table,
- unsigned int n, enum rte_ring_queue_behavior behavior,
- unsigned int is_sp, unsigned int *free_space)
-{
- uint32_t prod_head, prod_next;
- uint32_t free_entries;
-
- n = __rte_ring_move_prod_head(r, is_sp, n, behavior,
- &prod_head, &prod_next, &free_entries);
- if (n == 0)
- goto end;
-
- ENQUEUE_PTRS(r, &r[1], prod_head, obj_table, n, void *);
-
- update_tail(&r->prod, prod_head, prod_next, is_sp, 1);
-end:
- if (free_space != NULL)
- *free_space = free_entries - n;
- return n;
-}
-
-/**
- * @internal Dequeue several objects from the ring
- *
- * @param r
- * A pointer to the ring structure.
- * @param obj_table
- * A pointer to a table of void * pointers (objects).
- * @param n
- * The number of objects to pull from the ring.
- * @param behavior
- * RTE_RING_QUEUE_FIXED: Dequeue a fixed number of items from a ring
- * RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from ring
- * @param is_sc
- * Indicates whether to use single consumer or multi-consumer head update
- * @param available
- * returns the number of remaining ring entries after the dequeue has finished
- * @return
- * - Actual number of objects dequeued.
- * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
- */
-static __rte_always_inline unsigned int
-__rte_ring_do_dequeue(struct rte_ring *r, void **obj_table,
- unsigned int n, enum rte_ring_queue_behavior behavior,
- unsigned int is_sc, unsigned int *available)
-{
- uint32_t cons_head, cons_next;
- uint32_t entries;
-
- n = __rte_ring_move_cons_head(r, (int)is_sc, n, behavior,
- &cons_head, &cons_next, &entries);
- if (n == 0)
- goto end;
-
- DEQUEUE_PTRS(r, &r[1], cons_head, obj_table, n, void *);
-
- update_tail(&r->cons, cons_head, cons_next, is_sc, 0);
-
-end:
- if (available != NULL)
- *available = entries - n;
- return n;
-}
-
/**
* Enqueue several objects on the ring (multi-producers safe).
*
@@ -415,13 +213,9 @@ __rte_ring_do_dequeue(struct rte_ring *r, void **obj_table,
* @return
* The number of objects enqueued, either 0 or n
*/
-static __rte_always_inline unsigned int
+unsigned int
rte_ring_mp_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
- unsigned int n, unsigned int *free_space)
-{
- return __rte_ring_do_enqueue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
- __IS_MP, free_space);
-}
+ unsigned int n, unsigned int *free_space);
/**
* Enqueue several objects on a ring (NOT multi-producers safe).
@@ -438,13 +232,9 @@ rte_ring_mp_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
* @return
* The number of objects enqueued, either 0 or n
*/
-static __rte_always_inline unsigned int
+unsigned int
rte_ring_sp_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
- unsigned int n, unsigned int *free_space)
-{
- return __rte_ring_do_enqueue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
- __IS_SP, free_space);
-}
+ unsigned int n, unsigned int *free_space);
/**
* Enqueue several objects on a ring.
@@ -465,13 +255,9 @@ rte_ring_sp_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
* @return
* The number of objects enqueued, either 0 or n
*/
-static __rte_always_inline unsigned int
+unsigned int
rte_ring_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
- unsigned int n, unsigned int *free_space)
-{
- return __rte_ring_do_enqueue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
- r->prod.single, free_space);
-}
+ unsigned int n, unsigned int *free_space);
/**
* Enqueue one object on a ring (multi-producers safe).
@@ -487,11 +273,8 @@ rte_ring_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
* - 0: Success; objects enqueued.
* - -ENOBUFS: Not enough room in the ring to enqueue; no object is enqueued.
*/
-static __rte_always_inline int
-rte_ring_mp_enqueue(struct rte_ring *r, void *obj)
-{
- return rte_ring_mp_enqueue_bulk(r, &obj, 1, NULL) ? 0 : -ENOBUFS;
-}
+int
+rte_ring_mp_enqueue(struct rte_ring *r, void *obj);
/**
* Enqueue one object on a ring (NOT multi-producers safe).
@@ -504,11 +287,8 @@ rte_ring_mp_enqueue(struct rte_ring *r, void *obj)
* - 0: Success; objects enqueued.
* - -ENOBUFS: Not enough room in the ring to enqueue; no object is enqueued.
*/
-static __rte_always_inline int
-rte_ring_sp_enqueue(struct rte_ring *r, void *obj)
-{
- return rte_ring_sp_enqueue_bulk(r, &obj, 1, NULL) ? 0 : -ENOBUFS;
-}
+int
+rte_ring_sp_enqueue(struct rte_ring *r, void *obj);
/**
* Enqueue one object on a ring.
@@ -525,11 +305,8 @@ rte_ring_sp_enqueue(struct rte_ring *r, void *obj)
* - 0: Success; objects enqueued.
* - -ENOBUFS: Not enough room in the ring to enqueue; no object is enqueued.
*/
-static __rte_always_inline int
-rte_ring_enqueue(struct rte_ring *r, void *obj)
-{
- return rte_ring_enqueue_bulk(r, &obj, 1, NULL) ? 0 : -ENOBUFS;
-}
+int
+rte_ring_enqueue(struct rte_ring *r, void *obj);
/**
* Dequeue several objects from a ring (multi-consumers safe).
@@ -549,13 +326,9 @@ rte_ring_enqueue(struct rte_ring *r, void *obj)
* @return
* The number of objects dequeued, either 0 or n
*/
-static __rte_always_inline unsigned int
+unsigned int
rte_ring_mc_dequeue_bulk(struct rte_ring *r, void **obj_table,
- unsigned int n, unsigned int *available)
-{
- return __rte_ring_do_dequeue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
- __IS_MC, available);
-}
+ unsigned int n, unsigned int *available);
/**
* Dequeue several objects from a ring (NOT multi-consumers safe).
@@ -573,13 +346,9 @@ rte_ring_mc_dequeue_bulk(struct rte_ring *r, void **obj_table,
* @return
* The number of objects dequeued, either 0 or n
*/
-static __rte_always_inline unsigned int
+unsigned int
rte_ring_sc_dequeue_bulk(struct rte_ring *r, void **obj_table,
- unsigned int n, unsigned int *available)
-{
- return __rte_ring_do_dequeue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
- __IS_SC, available);
-}
+ unsigned int n, unsigned int *available);
/**
* Dequeue several objects from a ring.
@@ -600,13 +369,9 @@ rte_ring_sc_dequeue_bulk(struct rte_ring *r, void **obj_table,
* @return
* The number of objects dequeued, either 0 or n
*/
-static __rte_always_inline unsigned int
+unsigned int
rte_ring_dequeue_bulk(struct rte_ring *r, void **obj_table, unsigned int n,
- unsigned int *available)
-{
- return __rte_ring_do_dequeue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
- r->cons.single, available);
-}
+ unsigned int *available);
/**
* Dequeue one object from a ring (multi-consumers safe).
@@ -623,11 +388,8 @@ rte_ring_dequeue_bulk(struct rte_ring *r, void **obj_table, unsigned int n,
* - -ENOENT: Not enough entries in the ring to dequeue; no object is
* dequeued.
*/
-static __rte_always_inline int
-rte_ring_mc_dequeue(struct rte_ring *r, void **obj_p)
-{
- return rte_ring_mc_dequeue_bulk(r, obj_p, 1, NULL) ? 0 : -ENOENT;
-}
+int
+rte_ring_mc_dequeue(struct rte_ring *r, void **obj_p);
/**
* Dequeue one object from a ring (NOT multi-consumers safe).
@@ -641,11 +403,8 @@ rte_ring_mc_dequeue(struct rte_ring *r, void **obj_p)
* - -ENOENT: Not enough entries in the ring to dequeue, no object is
* dequeued.
*/
-static __rte_always_inline int
-rte_ring_sc_dequeue(struct rte_ring *r, void **obj_p)
-{
- return rte_ring_sc_dequeue_bulk(r, obj_p, 1, NULL) ? 0 : -ENOENT;
-}
+int
+rte_ring_sc_dequeue(struct rte_ring *r, void **obj_p);
/**
* Dequeue one object from a ring.
@@ -663,11 +422,8 @@ rte_ring_sc_dequeue(struct rte_ring *r, void **obj_p)
* - -ENOENT: Not enough entries in the ring to dequeue, no object is
* dequeued.
*/
-static __rte_always_inline int
-rte_ring_dequeue(struct rte_ring *r, void **obj_p)
-{
- return rte_ring_dequeue_bulk(r, obj_p, 1, NULL) ? 0 : -ENOENT;
-}
+int
+rte_ring_dequeue(struct rte_ring *r, void **obj_p);
/**
* Flush a ring.
@@ -694,14 +450,8 @@ rte_ring_reset(struct rte_ring *r);
* @return
* The number of entries in the ring.
*/
-static inline unsigned
-rte_ring_count(const struct rte_ring *r)
-{
- uint32_t prod_tail = r->prod.tail;
- uint32_t cons_tail = r->cons.tail;
- uint32_t count = (prod_tail - cons_tail) & r->mask;
- return (count > r->capacity) ? r->capacity : count;
-}
+unsigned
+rte_ring_count(const struct rte_ring *r);
/**
* Return the number of free entries in a ring.
@@ -711,11 +461,8 @@ rte_ring_count(const struct rte_ring *r)
* @return
* The number of free entries in the ring.
*/
-static inline unsigned
-rte_ring_free_count(const struct rte_ring *r)
-{
- return r->capacity - rte_ring_count(r);
-}
+unsigned
+rte_ring_free_count(const struct rte_ring *r);
/**
* Test if a ring is full.
@@ -726,11 +473,8 @@ rte_ring_free_count(const struct rte_ring *r)
* - 1: The ring is full.
* - 0: The ring is not full.
*/
-static inline int
-rte_ring_full(const struct rte_ring *r)
-{
- return rte_ring_free_count(r) == 0;
-}
+int
+rte_ring_full(const struct rte_ring *r);
/**
* Test if a ring is empty.
@@ -741,11 +485,14 @@ rte_ring_full(const struct rte_ring *r)
* - 1: The ring is empty.
* - 0: The ring is not empty.
*/
-static inline int
-rte_ring_empty(const struct rte_ring *r)
-{
- return rte_ring_count(r) == 0;
-}
+int
+rte_ring_empty(const struct rte_ring *r);
+
+const char *
+rte_ring_get_name(const struct rte_ring *r);
+
+const struct rte_memzone *
+rte_ring_get_memzone(const struct rte_ring *r);
/**
* Return the size of the ring.
@@ -757,11 +504,8 @@ rte_ring_empty(const struct rte_ring *r)
* NOTE: this is not the same as the usable space in the ring. To query that
* use ``rte_ring_get_capacity()``.
*/
-static inline unsigned int
-rte_ring_get_size(const struct rte_ring *r)
-{
- return r->size;
-}
+unsigned int
+rte_ring_get_size(const struct rte_ring *r);
/**
* Return the number of elements which can be stored in the ring.
@@ -771,11 +515,14 @@ rte_ring_get_size(const struct rte_ring *r)
* @return
* The usable size of the ring.
*/
-static inline unsigned int
-rte_ring_get_capacity(const struct rte_ring *r)
-{
- return r->capacity;
-}
+unsigned int
+rte_ring_get_capacity(const struct rte_ring *r);
+
+uint32_t
+rte_ring_get_prod_sync_type(const struct rte_ring *r);
+
+uint32_t
+rte_ring_get_cons_sync_type(const struct rte_ring *r);
/**
* Dump the status of all rings on the console
@@ -815,13 +562,9 @@ struct rte_ring *rte_ring_lookup(const char *name);
* @return
* - n: Actual number of objects enqueued.
*/
-static __rte_always_inline unsigned
+unsigned
rte_ring_mp_enqueue_burst(struct rte_ring *r, void * const *obj_table,
- unsigned int n, unsigned int *free_space)
-{
- return __rte_ring_do_enqueue(r, obj_table, n,
- RTE_RING_QUEUE_VARIABLE, __IS_MP, free_space);
-}
+ unsigned int n, unsigned int *free_space);
/**
* Enqueue several objects on a ring (NOT multi-producers safe).
@@ -838,13 +581,9 @@ rte_ring_mp_enqueue_burst(struct rte_ring *r, void * const *obj_table,
* @return
* - n: Actual number of objects enqueued.
*/
-static __rte_always_inline unsigned
+unsigned
rte_ring_sp_enqueue_burst(struct rte_ring *r, void * const *obj_table,
- unsigned int n, unsigned int *free_space)
-{
- return __rte_ring_do_enqueue(r, obj_table, n,
- RTE_RING_QUEUE_VARIABLE, __IS_SP, free_space);
-}
+ unsigned int n, unsigned int *free_space);
/**
* Enqueue several objects on a ring.
@@ -865,13 +604,9 @@ rte_ring_sp_enqueue_burst(struct rte_ring *r, void * const *obj_table,
* @return
* - n: Actual number of objects enqueued.
*/
-static __rte_always_inline unsigned
+unsigned
rte_ring_enqueue_burst(struct rte_ring *r, void * const *obj_table,
- unsigned int n, unsigned int *free_space)
-{
- return __rte_ring_do_enqueue(r, obj_table, n, RTE_RING_QUEUE_VARIABLE,
- r->prod.single, free_space);
-}
+ unsigned int n, unsigned int *free_space);
/**
* Dequeue several objects from a ring (multi-consumers safe). When the request
@@ -893,13 +628,9 @@ rte_ring_enqueue_burst(struct rte_ring *r, void * const *obj_table,
* @return
* - n: Actual number of objects dequeued, 0 if ring is empty
*/
-static __rte_always_inline unsigned
+unsigned
rte_ring_mc_dequeue_burst(struct rte_ring *r, void **obj_table,
- unsigned int n, unsigned int *available)
-{
- return __rte_ring_do_dequeue(r, obj_table, n,
- RTE_RING_QUEUE_VARIABLE, __IS_MC, available);
-}
+ unsigned int n, unsigned int *available);
/**
* Dequeue several objects from a ring (NOT multi-consumers safe).When the
@@ -918,13 +649,9 @@ rte_ring_mc_dequeue_burst(struct rte_ring *r, void **obj_table,
* @return
* - n: Actual number of objects dequeued, 0 if ring is empty
*/
-static __rte_always_inline unsigned
+unsigned
rte_ring_sc_dequeue_burst(struct rte_ring *r, void **obj_table,
- unsigned int n, unsigned int *available)
-{
- return __rte_ring_do_dequeue(r, obj_table, n,
- RTE_RING_QUEUE_VARIABLE, __IS_SC, available);
-}
+ unsigned int n, unsigned int *available);
/**
* Dequeue multiple objects from a ring up to a maximum number.
@@ -945,14 +672,9 @@ rte_ring_sc_dequeue_burst(struct rte_ring *r, void **obj_table,
* @return
* - Number of objects dequeued
*/
-static __rte_always_inline unsigned
+unsigned
rte_ring_dequeue_burst(struct rte_ring *r, void **obj_table,
- unsigned int n, unsigned int *available)
-{
- return __rte_ring_do_dequeue(r, obj_table, n,
- RTE_RING_QUEUE_VARIABLE,
- r->cons.single, available);
-}
+ unsigned int n, unsigned int *available);
#ifdef __cplusplus
}
diff --git a/lib/librte_ring/rte_ring_elem.h b/lib/librte_ring/rte_ring_elem.h
index 3976757ed..2a6b00370 100644
--- a/lib/librte_ring/rte_ring_elem.h
+++ b/lib/librte_ring/rte_ring_elem.h
@@ -109,380 +109,6 @@ __rte_experimental
struct rte_ring *rte_ring_create_elem(const char *name, unsigned int esize,
unsigned int count, int socket_id, unsigned int flags);
-static __rte_always_inline void
-__rte_ring_enqueue_elems_32(struct rte_ring *r, const uint32_t size,
- uint32_t idx, const void *obj_table, uint32_t n)
-{
- unsigned int i;
- uint32_t *ring = (uint32_t *)&r[1];
- const uint32_t *obj = (const uint32_t *)obj_table;
- if (likely(idx + n < size)) {
- for (i = 0; i < (n & ~0x7); i += 8, idx += 8) {
- ring[idx] = obj[i];
- ring[idx + 1] = obj[i + 1];
- ring[idx + 2] = obj[i + 2];
- ring[idx + 3] = obj[i + 3];
- ring[idx + 4] = obj[i + 4];
- ring[idx + 5] = obj[i + 5];
- ring[idx + 6] = obj[i + 6];
- ring[idx + 7] = obj[i + 7];
- }
- switch (n & 0x7) {
- case 7:
- ring[idx++] = obj[i++]; /* fallthrough */
- case 6:
- ring[idx++] = obj[i++]; /* fallthrough */
- case 5:
- ring[idx++] = obj[i++]; /* fallthrough */
- case 4:
- ring[idx++] = obj[i++]; /* fallthrough */
- case 3:
- ring[idx++] = obj[i++]; /* fallthrough */
- case 2:
- ring[idx++] = obj[i++]; /* fallthrough */
- case 1:
- ring[idx++] = obj[i++]; /* fallthrough */
- }
- } else {
- for (i = 0; idx < size; i++, idx++)
- ring[idx] = obj[i];
- /* Start at the beginning */
- for (idx = 0; i < n; i++, idx++)
- ring[idx] = obj[i];
- }
-}
-
-static __rte_always_inline void
-__rte_ring_enqueue_elems_64(struct rte_ring *r, uint32_t prod_head,
- const void *obj_table, uint32_t n)
-{
- unsigned int i;
- const uint32_t size = r->size;
- uint32_t idx = prod_head & r->mask;
- uint64_t *ring = (uint64_t *)&r[1];
- const uint64_t *obj = (const uint64_t *)obj_table;
- if (likely(idx + n < size)) {
- for (i = 0; i < (n & ~0x3); i += 4, idx += 4) {
- ring[idx] = obj[i];
- ring[idx + 1] = obj[i + 1];
- ring[idx + 2] = obj[i + 2];
- ring[idx + 3] = obj[i + 3];
- }
- switch (n & 0x3) {
- case 3:
- ring[idx++] = obj[i++]; /* fallthrough */
- case 2:
- ring[idx++] = obj[i++]; /* fallthrough */
- case 1:
- ring[idx++] = obj[i++];
- }
- } else {
- for (i = 0; idx < size; i++, idx++)
- ring[idx] = obj[i];
- /* Start at the beginning */
- for (idx = 0; i < n; i++, idx++)
- ring[idx] = obj[i];
- }
-}
-
-static __rte_always_inline void
-__rte_ring_enqueue_elems_128(struct rte_ring *r, uint32_t prod_head,
- const void *obj_table, uint32_t n)
-{
- unsigned int i;
- const uint32_t size = r->size;
- uint32_t idx = prod_head & r->mask;
- rte_int128_t *ring = (rte_int128_t *)&r[1];
- const rte_int128_t *obj = (const rte_int128_t *)obj_table;
- if (likely(idx + n < size)) {
- for (i = 0; i < (n & ~0x1); i += 2, idx += 2)
- memcpy((void *)(ring + idx),
- (const void *)(obj + i), 32);
- switch (n & 0x1) {
- case 1:
- memcpy((void *)(ring + idx),
- (const void *)(obj + i), 16);
- }
- } else {
- for (i = 0; idx < size; i++, idx++)
- memcpy((void *)(ring + idx),
- (const void *)(obj + i), 16);
- /* Start at the beginning */
- for (idx = 0; i < n; i++, idx++)
- memcpy((void *)(ring + idx),
- (const void *)(obj + i), 16);
- }
-}
-
-/* the actual enqueue of elements on the ring.
- * Placed here since identical code needed in both
- * single and multi producer enqueue functions.
- */
-static __rte_always_inline void
-__rte_ring_enqueue_elems(struct rte_ring *r, uint32_t prod_head,
- const void *obj_table, uint32_t esize, uint32_t num)
-{
- /* 8B and 16B copies implemented individually to retain
- * the current performance.
- */
- if (esize == 8)
- __rte_ring_enqueue_elems_64(r, prod_head, obj_table, num);
- else if (esize == 16)
- __rte_ring_enqueue_elems_128(r, prod_head, obj_table, num);
- else {
- uint32_t idx, scale, nr_idx, nr_num, nr_size;
-
- /* Normalize to uint32_t */
- scale = esize / sizeof(uint32_t);
- nr_num = num * scale;
- idx = prod_head & r->mask;
- nr_idx = idx * scale;
- nr_size = r->size * scale;
- __rte_ring_enqueue_elems_32(r, nr_size, nr_idx,
- obj_table, nr_num);
- }
-}
-
-static __rte_always_inline void
-__rte_ring_dequeue_elems_32(struct rte_ring *r, const uint32_t size,
- uint32_t idx, void *obj_table, uint32_t n)
-{
- unsigned int i;
- uint32_t *ring = (uint32_t *)&r[1];
- uint32_t *obj = (uint32_t *)obj_table;
- if (likely(idx + n < size)) {
- for (i = 0; i < (n & ~0x7); i += 8, idx += 8) {
- obj[i] = ring[idx];
- obj[i + 1] = ring[idx + 1];
- obj[i + 2] = ring[idx + 2];
- obj[i + 3] = ring[idx + 3];
- obj[i + 4] = ring[idx + 4];
- obj[i + 5] = ring[idx + 5];
- obj[i + 6] = ring[idx + 6];
- obj[i + 7] = ring[idx + 7];
- }
- switch (n & 0x7) {
- case 7:
- obj[i++] = ring[idx++]; /* fallthrough */
- case 6:
- obj[i++] = ring[idx++]; /* fallthrough */
- case 5:
- obj[i++] = ring[idx++]; /* fallthrough */
- case 4:
- obj[i++] = ring[idx++]; /* fallthrough */
- case 3:
- obj[i++] = ring[idx++]; /* fallthrough */
- case 2:
- obj[i++] = ring[idx++]; /* fallthrough */
- case 1:
- obj[i++] = ring[idx++]; /* fallthrough */
- }
- } else {
- for (i = 0; idx < size; i++, idx++)
- obj[i] = ring[idx];
- /* Start at the beginning */
- for (idx = 0; i < n; i++, idx++)
- obj[i] = ring[idx];
- }
-}
-
-static __rte_always_inline void
-__rte_ring_dequeue_elems_64(struct rte_ring *r, uint32_t prod_head,
- void *obj_table, uint32_t n)
-{
- unsigned int i;
- const uint32_t size = r->size;
- uint32_t idx = prod_head & r->mask;
- uint64_t *ring = (uint64_t *)&r[1];
- uint64_t *obj = (uint64_t *)obj_table;
- if (likely(idx + n < size)) {
- for (i = 0; i < (n & ~0x3); i += 4, idx += 4) {
- obj[i] = ring[idx];
- obj[i + 1] = ring[idx + 1];
- obj[i + 2] = ring[idx + 2];
- obj[i + 3] = ring[idx + 3];
- }
- switch (n & 0x3) {
- case 3:
- obj[i++] = ring[idx++]; /* fallthrough */
- case 2:
- obj[i++] = ring[idx++]; /* fallthrough */
- case 1:
- obj[i++] = ring[idx++]; /* fallthrough */
- }
- } else {
- for (i = 0; idx < size; i++, idx++)
- obj[i] = ring[idx];
- /* Start at the beginning */
- for (idx = 0; i < n; i++, idx++)
- obj[i] = ring[idx];
- }
-}
-
-static __rte_always_inline void
-__rte_ring_dequeue_elems_128(struct rte_ring *r, uint32_t prod_head,
- void *obj_table, uint32_t n)
-{
- unsigned int i;
- const uint32_t size = r->size;
- uint32_t idx = prod_head & r->mask;
- rte_int128_t *ring = (rte_int128_t *)&r[1];
- rte_int128_t *obj = (rte_int128_t *)obj_table;
- if (likely(idx + n < size)) {
- for (i = 0; i < (n & ~0x1); i += 2, idx += 2)
- memcpy((void *)(obj + i), (void *)(ring + idx), 32);
- switch (n & 0x1) {
- case 1:
- memcpy((void *)(obj + i), (void *)(ring + idx), 16);
- }
- } else {
- for (i = 0; idx < size; i++, idx++)
- memcpy((void *)(obj + i), (void *)(ring + idx), 16);
- /* Start at the beginning */
- for (idx = 0; i < n; i++, idx++)
- memcpy((void *)(obj + i), (void *)(ring + idx), 16);
- }
-}
-
-/* the actual dequeue of elements from the ring.
- * Placed here since identical code needed in both
- * single and multi producer enqueue functions.
- */
-static __rte_always_inline void
-__rte_ring_dequeue_elems(struct rte_ring *r, uint32_t cons_head,
- void *obj_table, uint32_t esize, uint32_t num)
-{
- /* 8B and 16B copies implemented individually to retain
- * the current performance.
- */
- if (esize == 8)
- __rte_ring_dequeue_elems_64(r, cons_head, obj_table, num);
- else if (esize == 16)
- __rte_ring_dequeue_elems_128(r, cons_head, obj_table, num);
- else {
- uint32_t idx, scale, nr_idx, nr_num, nr_size;
-
- /* Normalize to uint32_t */
- scale = esize / sizeof(uint32_t);
- nr_num = num * scale;
- idx = cons_head & r->mask;
- nr_idx = idx * scale;
- nr_size = r->size * scale;
- __rte_ring_dequeue_elems_32(r, nr_size, nr_idx,
- obj_table, nr_num);
- }
-}
-
-/* Between load and load. there might be cpu reorder in weak model
- * (powerpc/arm).
- * There are 2 choices for the users
- * 1.use rmb() memory barrier
- * 2.use one-direction load_acquire/store_release barrier,defined by
- * CONFIG_RTE_USE_C11_MEM_MODEL=y
- * It depends on performance test results.
- * By default, move common functions to rte_ring_generic.h
- */
-#ifdef RTE_USE_C11_MEM_MODEL
-#include "rte_ring_c11_mem.h"
-#else
-#include "rte_ring_generic.h"
-#endif
-
-/**
- * @internal Enqueue several objects on the ring
- *
- * @param r
- * A pointer to the ring structure.
- * @param obj_table
- * A pointer to a table of objects.
- * @param esize
- * The size of ring element, in bytes. It must be a multiple of 4.
- * This must be the same value used while creating the ring. Otherwise
- * the results are undefined.
- * @param n
- * The number of objects to add in the ring from the obj_table.
- * @param behavior
- * RTE_RING_QUEUE_FIXED: Enqueue a fixed number of items from a ring
- * RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible from ring
- * @param is_sp
- * Indicates whether to use single producer or multi-producer head update
- * @param free_space
- * returns the amount of space after the enqueue operation has finished
- * @return
- * Actual number of objects enqueued.
- * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
- */
-static __rte_always_inline unsigned int
-__rte_ring_do_enqueue_elem(struct rte_ring *r, const void *obj_table,
- unsigned int esize, unsigned int n,
- enum rte_ring_queue_behavior behavior, unsigned int is_sp,
- unsigned int *free_space)
-{
- uint32_t prod_head, prod_next;
- uint32_t free_entries;
-
- n = __rte_ring_move_prod_head(r, is_sp, n, behavior,
- &prod_head, &prod_next, &free_entries);
- if (n == 0)
- goto end;
-
- __rte_ring_enqueue_elems(r, prod_head, obj_table, esize, n);
-
- update_tail(&r->prod, prod_head, prod_next, is_sp, 1);
-end:
- if (free_space != NULL)
- *free_space = free_entries - n;
- return n;
-}
-
-/**
- * @internal Dequeue several objects from the ring
- *
- * @param r
- * A pointer to the ring structure.
- * @param obj_table
- * A pointer to a table of objects.
- * @param esize
- * The size of ring element, in bytes. It must be a multiple of 4.
- * This must be the same value used while creating the ring. Otherwise
- * the results are undefined.
- * @param n
- * The number of objects to pull from the ring.
- * @param behavior
- * RTE_RING_QUEUE_FIXED: Dequeue a fixed number of items from a ring
- * RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from ring
- * @param is_sc
- * Indicates whether to use single consumer or multi-consumer head update
- * @param available
- * returns the number of remaining ring entries after the dequeue has finished
- * @return
- * - Actual number of objects dequeued.
- * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
- */
-static __rte_always_inline unsigned int
-__rte_ring_do_dequeue_elem(struct rte_ring *r, void *obj_table,
- unsigned int esize, unsigned int n,
- enum rte_ring_queue_behavior behavior, unsigned int is_sc,
- unsigned int *available)
-{
- uint32_t cons_head, cons_next;
- uint32_t entries;
-
- n = __rte_ring_move_cons_head(r, (int)is_sc, n, behavior,
- &cons_head, &cons_next, &entries);
- if (n == 0)
- goto end;
-
- __rte_ring_dequeue_elems(r, cons_head, obj_table, esize, n);
-
- update_tail(&r->cons, cons_head, cons_next, is_sc, 0);
-
-end:
- if (available != NULL)
- *available = entries - n;
- return n;
-}
-
/**
* Enqueue several objects on the ring (multi-producers safe).
*
@@ -505,13 +131,9 @@ __rte_ring_do_dequeue_elem(struct rte_ring *r, void *obj_table,
* @return
* The number of objects enqueued, either 0 or n
*/
-static __rte_always_inline unsigned int
+unsigned int
rte_ring_mp_enqueue_bulk_elem(struct rte_ring *r, const void *obj_table,
- unsigned int esize, unsigned int n, unsigned int *free_space)
-{
- return __rte_ring_do_enqueue_elem(r, obj_table, esize, n,
- RTE_RING_QUEUE_FIXED, __IS_MP, free_space);
-}
+ unsigned int esize, unsigned int n, unsigned int *free_space);
/**
* Enqueue several objects on a ring
@@ -534,13 +156,9 @@ rte_ring_mp_enqueue_bulk_elem(struct rte_ring *r, const void *obj_table,
* @return
* The number of objects enqueued, either 0 or n
*/
-static __rte_always_inline unsigned int
+unsigned int
rte_ring_sp_enqueue_bulk_elem(struct rte_ring *r, const void *obj_table,
- unsigned int esize, unsigned int n, unsigned int *free_space)
-{
- return __rte_ring_do_enqueue_elem(r, obj_table, esize, n,
- RTE_RING_QUEUE_FIXED, __IS_SP, free_space);
-}
+ unsigned int esize, unsigned int n, unsigned int *free_space);
/**
* Enqueue several objects on a ring.
@@ -565,13 +183,9 @@ rte_ring_sp_enqueue_bulk_elem(struct rte_ring *r, const void *obj_table,
* @return
* The number of objects enqueued, either 0 or n
*/
-static __rte_always_inline unsigned int
+unsigned int
rte_ring_enqueue_bulk_elem(struct rte_ring *r, const void *obj_table,
- unsigned int esize, unsigned int n, unsigned int *free_space)
-{
- return __rte_ring_do_enqueue_elem(r, obj_table, esize, n,
- RTE_RING_QUEUE_FIXED, r->prod.single, free_space);
-}
+ unsigned int esize, unsigned int n, unsigned int *free_space);
/**
* Enqueue one object on a ring (multi-producers safe).
@@ -591,12 +205,8 @@ rte_ring_enqueue_bulk_elem(struct rte_ring *r, const void *obj_table,
* - 0: Success; objects enqueued.
* - -ENOBUFS: Not enough room in the ring to enqueue; no object is enqueued.
*/
-static __rte_always_inline int
-rte_ring_mp_enqueue_elem(struct rte_ring *r, void *obj, unsigned int esize)
-{
- return rte_ring_mp_enqueue_bulk_elem(r, obj, esize, 1, NULL) ? 0 :
- -ENOBUFS;
-}
+int
+rte_ring_mp_enqueue_elem(struct rte_ring *r, void *obj, unsigned int esize);
/**
* Enqueue one object on a ring
@@ -615,12 +225,8 @@ rte_ring_mp_enqueue_elem(struct rte_ring *r, void *obj, unsigned int esize)
* - 0: Success; objects enqueued.
* - -ENOBUFS: Not enough room in the ring to enqueue; no object is enqueued.
*/
-static __rte_always_inline int
-rte_ring_sp_enqueue_elem(struct rte_ring *r, void *obj, unsigned int esize)
-{
- return rte_ring_sp_enqueue_bulk_elem(r, obj, esize, 1, NULL) ? 0 :
- -ENOBUFS;
-}
+int
+rte_ring_sp_enqueue_elem(struct rte_ring *r, void *obj, unsigned int esize);
/**
* Enqueue one object on a ring.
@@ -641,12 +247,8 @@ rte_ring_sp_enqueue_elem(struct rte_ring *r, void *obj, unsigned int esize)
* - 0: Success; objects enqueued.
* - -ENOBUFS: Not enough room in the ring to enqueue; no object is enqueued.
*/
-static __rte_always_inline int
-rte_ring_enqueue_elem(struct rte_ring *r, void *obj, unsigned int esize)
-{
- return rte_ring_enqueue_bulk_elem(r, obj, esize, 1, NULL) ? 0 :
- -ENOBUFS;
-}
+int
+rte_ring_enqueue_elem(struct rte_ring *r, void *obj, unsigned int esize);
/**
* Dequeue several objects from a ring (multi-consumers safe).
@@ -670,13 +272,9 @@ rte_ring_enqueue_elem(struct rte_ring *r, void *obj, unsigned int esize)
* @return
* The number of objects dequeued, either 0 or n
*/
-static __rte_always_inline unsigned int
+unsigned int
rte_ring_mc_dequeue_bulk_elem(struct rte_ring *r, void *obj_table,
- unsigned int esize, unsigned int n, unsigned int *available)
-{
- return __rte_ring_do_dequeue_elem(r, obj_table, esize, n,
- RTE_RING_QUEUE_FIXED, __IS_MC, available);
-}
+ unsigned int esize, unsigned int n, unsigned int *available);
/**
* Dequeue several objects from a ring (NOT multi-consumers safe).
@@ -698,13 +296,9 @@ rte_ring_mc_dequeue_bulk_elem(struct rte_ring *r, void *obj_table,
* @return
* The number of objects dequeued, either 0 or n
*/
-static __rte_always_inline unsigned int
+unsigned int
rte_ring_sc_dequeue_bulk_elem(struct rte_ring *r, void *obj_table,
- unsigned int esize, unsigned int n, unsigned int *available)
-{
- return __rte_ring_do_dequeue_elem(r, obj_table, esize, n,
- RTE_RING_QUEUE_FIXED, __IS_SC, available);
-}
+ unsigned int esize, unsigned int n, unsigned int *available);
/**
* Dequeue several objects from a ring.
@@ -729,13 +323,9 @@ rte_ring_sc_dequeue_bulk_elem(struct rte_ring *r, void *obj_table,
* @return
* The number of objects dequeued, either 0 or n
*/
-static __rte_always_inline unsigned int
+unsigned int
rte_ring_dequeue_bulk_elem(struct rte_ring *r, void *obj_table,
- unsigned int esize, unsigned int n, unsigned int *available)
-{
- return __rte_ring_do_dequeue_elem(r, obj_table, esize, n,
- RTE_RING_QUEUE_FIXED, r->cons.single, available);
-}
+ unsigned int esize, unsigned int n, unsigned int *available);
/**
* Dequeue one object from a ring (multi-consumers safe).
@@ -756,13 +346,9 @@ rte_ring_dequeue_bulk_elem(struct rte_ring *r, void *obj_table,
* - -ENOENT: Not enough entries in the ring to dequeue; no object is
* dequeued.
*/
-static __rte_always_inline int
+int
rte_ring_mc_dequeue_elem(struct rte_ring *r, void *obj_p,
- unsigned int esize)
-{
- return rte_ring_mc_dequeue_bulk_elem(r, obj_p, esize, 1, NULL) ? 0 :
- -ENOENT;
-}
+ unsigned int esize);
/**
* Dequeue one object from a ring (NOT multi-consumers safe).
@@ -780,13 +366,9 @@ rte_ring_mc_dequeue_elem(struct rte_ring *r, void *obj_p,
* - -ENOENT: Not enough entries in the ring to dequeue, no object is
* dequeued.
*/
-static __rte_always_inline int
+int
rte_ring_sc_dequeue_elem(struct rte_ring *r, void *obj_p,
- unsigned int esize)
-{
- return rte_ring_sc_dequeue_bulk_elem(r, obj_p, esize, 1, NULL) ? 0 :
- -ENOENT;
-}
+ unsigned int esize);
/**
* Dequeue one object from a ring.
@@ -808,12 +390,8 @@ rte_ring_sc_dequeue_elem(struct rte_ring *r, void *obj_p,
* - -ENOENT: Not enough entries in the ring to dequeue, no object is
* dequeued.
*/
-static __rte_always_inline int
-rte_ring_dequeue_elem(struct rte_ring *r, void *obj_p, unsigned int esize)
-{
- return rte_ring_dequeue_bulk_elem(r, obj_p, esize, 1, NULL) ? 0 :
- -ENOENT;
-}
+int
+rte_ring_dequeue_elem(struct rte_ring *r, void *obj_p, unsigned int esize);
/**
* Enqueue several objects on the ring (multi-producers safe).
@@ -837,13 +415,9 @@ rte_ring_dequeue_elem(struct rte_ring *r, void *obj_p, unsigned int esize)
* @return
* - n: Actual number of objects enqueued.
*/
-static __rte_always_inline unsigned
+unsigned
rte_ring_mp_enqueue_burst_elem(struct rte_ring *r, const void *obj_table,
- unsigned int esize, unsigned int n, unsigned int *free_space)
-{
- return __rte_ring_do_enqueue_elem(r, obj_table, esize, n,
- RTE_RING_QUEUE_VARIABLE, __IS_MP, free_space);
-}
+ unsigned int esize, unsigned int n, unsigned int *free_space);
/**
* Enqueue several objects on a ring
@@ -866,13 +440,9 @@ rte_ring_mp_enqueue_burst_elem(struct rte_ring *r, const void *obj_table,
* @return
* - n: Actual number of objects enqueued.
*/
-static __rte_always_inline unsigned
+unsigned
rte_ring_sp_enqueue_burst_elem(struct rte_ring *r, const void *obj_table,
- unsigned int esize, unsigned int n, unsigned int *free_space)
-{
- return __rte_ring_do_enqueue_elem(r, obj_table, esize, n,
- RTE_RING_QUEUE_VARIABLE, __IS_SP, free_space);
-}
+ unsigned int esize, unsigned int n, unsigned int *free_space);
/**
* Enqueue several objects on a ring.
@@ -897,13 +467,9 @@ rte_ring_sp_enqueue_burst_elem(struct rte_ring *r, const void *obj_table,
* @return
* - n: Actual number of objects enqueued.
*/
-static __rte_always_inline unsigned
+unsigned
rte_ring_enqueue_burst_elem(struct rte_ring *r, const void *obj_table,
- unsigned int esize, unsigned int n, unsigned int *free_space)
-{
- return __rte_ring_do_enqueue_elem(r, obj_table, esize, n,
- RTE_RING_QUEUE_VARIABLE, r->prod.single, free_space);
-}
+ unsigned int esize, unsigned int n, unsigned int *free_space);
/**
* Dequeue several objects from a ring (multi-consumers safe). When the request
@@ -929,13 +495,9 @@ rte_ring_enqueue_burst_elem(struct rte_ring *r, const void *obj_table,
* @return
* - n: Actual number of objects dequeued, 0 if ring is empty
*/
-static __rte_always_inline unsigned
+unsigned
rte_ring_mc_dequeue_burst_elem(struct rte_ring *r, void *obj_table,
- unsigned int esize, unsigned int n, unsigned int *available)
-{
- return __rte_ring_do_dequeue_elem(r, obj_table, esize, n,
- RTE_RING_QUEUE_VARIABLE, __IS_MC, available);
-}
+ unsigned int esize, unsigned int n, unsigned int *available);
/**
* Dequeue several objects from a ring (NOT multi-consumers safe).When the
@@ -958,13 +520,9 @@ rte_ring_mc_dequeue_burst_elem(struct rte_ring *r, void *obj_table,
* @return
* - n: Actual number of objects dequeued, 0 if ring is empty
*/
-static __rte_always_inline unsigned
+unsigned
rte_ring_sc_dequeue_burst_elem(struct rte_ring *r, void *obj_table,
- unsigned int esize, unsigned int n, unsigned int *available)
-{
- return __rte_ring_do_dequeue_elem(r, obj_table, esize, n,
- RTE_RING_QUEUE_VARIABLE, __IS_SC, available);
-}
+ unsigned int esize, unsigned int n, unsigned int *available);
/**
* Dequeue multiple objects from a ring up to a maximum number.
@@ -989,14 +547,9 @@ rte_ring_sc_dequeue_burst_elem(struct rte_ring *r, void *obj_table,
* @return
* - Number of objects dequeued
*/
-static __rte_always_inline unsigned int
+unsigned int
rte_ring_dequeue_burst_elem(struct rte_ring *r, void *obj_table,
- unsigned int esize, unsigned int n, unsigned int *available)
-{
- return __rte_ring_do_dequeue_elem(r, obj_table, esize, n,
- RTE_RING_QUEUE_VARIABLE,
- r->cons.single, available);
-}
+ unsigned int esize, unsigned int n, unsigned int *available);
#ifdef __cplusplus
}
diff --git a/lib/librte_ring/rte_ring_version.map b/lib/librte_ring/rte_ring_version.map
index e88c143cf..6cceacf95 100644
--- a/lib/librte_ring/rte_ring_version.map
+++ b/lib/librte_ring/rte_ring_version.map
@@ -8,6 +8,52 @@ DPDK_20.0 {
rte_ring_init;
rte_ring_list_dump;
rte_ring_lookup;
+ rte_ring_count;
+ rte_ring_dequeue;
+ rte_ring_dequeue_bulk;
+ rte_ring_dequeue_bulk_elem;
+ rte_ring_dequeue_burst;
+ rte_ring_dequeue_burst_elem;
+ rte_ring_dequeue_elem;
+ rte_ring_empty;
+ rte_ring_enqueue;
+ rte_ring_enqueue_bulk;
+ rte_ring_enqueue_bulk_elem;
+ rte_ring_enqueue_burst;
+ rte_ring_enqueue_burst_elem;
+ rte_ring_enqueue_elem;
+ rte_ring_free_count;
+ rte_ring_full;
+ rte_ring_get_capacity;
+ rte_ring_get_cons_sync_type;
+ rte_ring_get_memzone;
+ rte_ring_get_name;
+ rte_ring_get_prod_sync_type;
+ rte_ring_get_size;
+ rte_ring_mc_dequeue;
+ rte_ring_mc_dequeue_bulk;
+ rte_ring_mc_dequeue_bulk_elem;
+ rte_ring_mc_dequeue_burst;
+ rte_ring_mc_dequeue_burst_elem;
+ rte_ring_mc_dequeue_elem;
+ rte_ring_mp_enqueue;
+ rte_ring_mp_enqueue_bulk;
+ rte_ring_mp_enqueue_bulk_elem;
+ rte_ring_mp_enqueue_burst;
+ rte_ring_mp_enqueue_burst_elem;
+ rte_ring_mp_enqueue_elem;
+ rte_ring_sc_dequeue;
+ rte_ring_sc_dequeue_bulk;
+ rte_ring_sc_dequeue_bulk_elem;
+ rte_ring_sc_dequeue_burst;
+ rte_ring_sc_dequeue_burst_elem;
+ rte_ring_sc_dequeue_elem;
+ rte_ring_sp_enqueue;
+ rte_ring_sp_enqueue_bulk;
+ rte_ring_sp_enqueue_bulk_elem;
+ rte_ring_sp_enqueue_burst;
+ rte_ring_sp_enqueue_burst_elem;
+ rte_ring_sp_enqueue_elem;
local: *;
};
--
2.17.1
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [RFC] ring: make ring implementation non-inlined
2020-03-20 16:41 3% [dpdk-dev] [RFC] ring: make ring implementation non-inlined Konstantin Ananyev
@ 2020-03-20 17:54 0% ` Stephen Hemminger
2020-03-21 1:03 0% ` Ananyev, Konstantin
2020-03-25 21:09 4% ` Jerin Jacob
1 sibling, 1 reply; 200+ results
From: Stephen Hemminger @ 2020-03-20 17:54 UTC (permalink / raw)
To: Konstantin Ananyev; +Cc: dev, olivier.matz, honnappa.nagarahalli, jerinj, drc
On Fri, 20 Mar 2020 16:41:38 +0000
Konstantin Ananyev <konstantin.ananyev@intel.com> wrote:
> As was discussed here:
> http://mails.dpdk.org/archives/dev/2020-February/158586.html
> this RFC aimed to hide ring internals into .c and make all
> ring functions non-inlined. In theory that might help to
> maintain ABI stability in future.
> This is just a POC to measure the impact of proposed idea,
> proper implementation would definetly need some extra effort.
> On IA box (SKX) ring_perf_autotest shows ~20-30 cycles extra for
> enqueue+dequeue pair. On some more realistic code, I suspect
> the impact it might be a bit higher.
> For MP/MC bulk transfers degradation seems quite small,
> though for SP/SC and/or small transfers it is more then noticable
> (see exact numbers below).
> From my perspective we'd probably keep it inlined for now
> to avoid any non-anticipated perfomance degradations.
> Though intersted to see perf results and opinions from
> other interested parties.
>
> Intel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz
> ring_perf_autotest (without patch/with patch)
>
> ### Testing single element enq/deq ###
> legacy APIs: SP/SC: single: 8.75/43.23
> legacy APIs: MP/MC: single: 56.18/80.44
>
> ### Testing burst enq/deq ###
> legacy APIs: SP/SC: burst (size: 8): 37.36/53.37
> legacy APIs: SP/SC: burst (size: 32): 93.97/117.30
> legacy APIs: MP/MC: burst (size: 8): 78.23/91.45
> legacy APIs: MP/MC: burst (size: 32): 131.59/152.49
>
> ### Testing bulk enq/deq ###
> legacy APIs: SP/SC: bulk (size: 8): 37.29/54.48
> legacy APIs: SP/SC: bulk (size: 32): 92.68/113.01
> legacy APIs: MP/MC: bulk (size: 8): 78.40/93.50
> legacy APIs: MP/MC: bulk (size: 32): 131.49/154.25
>
> ### Testing empty bulk deq ###
> legacy APIs: SP/SC: bulk (size: 8): 4.00/16.86
> legacy APIs: MP/MC: bulk (size: 8): 7.01/15.55
>
> ### Testing using two hyperthreads ###
> legacy APIs: SP/SC: bulk (size: 8): 10.64/17.56
> legacy APIs: MP/MC: bulk (size: 8): 15.30/16.69
> legacy APIs: SP/SC: bulk (size: 32): 5.84/7.09
> legacy APIs: MP/MC: bulk (size: 32): 6.34/7.54
>
> ### Testing using two physical cores ###
> legacy APIs: SP/SC: bulk (size: 8): 24.34/42.40
> legacy APIs: MP/MC: bulk (size: 8): 70.34/71.82
> legacy APIs: SP/SC: bulk (size: 32): 12.67/14.68
> legacy APIs: MP/MC: bulk (size: 32): 22.41/17.93
>
> ### Testing single element enq/deq ###
> elem APIs: element size 16B: SP/SC: single: 10.65/41.96
> elem APIs: element size 16B: MP/MC: single: 44.33/81.36
>
> ### Testing burst enq/deq ###
> elem APIs: element size 16B: SP/SC: burst (size: 8): 39.20/58.52
> elem APIs: element size 16B: SP/SC: burst (size: 32): 123.19/142.79
> elem APIs: element size 16B: MP/MC: burst (size: 8): 80.72/101.36
> elem APIs: element size 16B: MP/MC: burst (size: 32): 169.21/185.38
>
> ### Testing bulk enq/deq ###
> elem APIs: element size 16B: SP/SC: bulk (size: 8): 41.64/58.46
> elem APIs: element size 16B: SP/SC: bulk (size: 32): 122.74/142.52
> elem APIs: element size 16B: MP/MC: bulk (size: 8): 80.60/103.14
> elem APIs: element size 16B: MP/MC: bulk (size: 32): 169.39/186.67
>
> ### Testing empty bulk deq ###
> elem APIs: element size 16B: SP/SC: bulk (size: 8): 5.01/17.17
> elem APIs: element size 16B: MP/MC: bulk (size: 8): 6.01/14.80
>
> ### Testing using two hyperthreads ###
> elem APIs: element size 16B: SP/SC: bulk (size: 8): 12.02/17.18
> elem APIs: element size 16B: MP/MC: bulk (size: 8): 16.81/21.14
> elem APIs: element size 16B: SP/SC: bulk (size: 32): 7.87/9.01
> elem APIs: element size 16B: MP/MC: bulk (size: 32): 8.22/10.57
>
> ### Testing using two physical cores ###
> elem APIs: element size 16B: SP/SC: bulk (size: 8): 27.00/51.94
> elem APIs: element size 16B: MP/MC: bulk (size: 8): 78.24/74.48
> elem APIs: element size 16B: SP/SC: bulk (size: 32): 15.41/16.14
> elem APIs: element size 16B: MP/MC: bulk (size: 32): 18.72/21.64
>
> Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
What is impact with LTO? I suspect compiler might have a chance to
get speed back with LTO.
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [RFC] ring: make ring implementation non-inlined
2020-03-20 17:54 0% ` Stephen Hemminger
@ 2020-03-21 1:03 0% ` Ananyev, Konstantin
0 siblings, 0 replies; 200+ results
From: Ananyev, Konstantin @ 2020-03-21 1:03 UTC (permalink / raw)
To: Stephen Hemminger; +Cc: dev, olivier.matz, honnappa.nagarahalli, jerinj, drc
> -----Original Message-----
> From: Stephen Hemminger <stephen@networkplumber.org>
> Sent: Friday, March 20, 2020 5:55 PM
> To: Ananyev, Konstantin <konstantin.ananyev@intel.com>
> Cc: dev@dpdk.org; olivier.matz@6wind.com; honnappa.nagarahalli@arm.com; jerinj@marvell.com; drc@linux.vnet.ibm.com
> Subject: Re: [RFC] ring: make ring implementation non-inlined
>
> On Fri, 20 Mar 2020 16:41:38 +0000
> Konstantin Ananyev <konstantin.ananyev@intel.com> wrote:
>
> > As was discussed here:
> > http://mails.dpdk.org/archives/dev/2020-February/158586.html
> > this RFC aimed to hide ring internals into .c and make all
> > ring functions non-inlined. In theory that might help to
> > maintain ABI stability in future.
> > This is just a POC to measure the impact of proposed idea,
> > proper implementation would definetly need some extra effort.
> > On IA box (SKX) ring_perf_autotest shows ~20-30 cycles extra for
> > enqueue+dequeue pair. On some more realistic code, I suspect
> > the impact it might be a bit higher.
> > For MP/MC bulk transfers degradation seems quite small,
> > though for SP/SC and/or small transfers it is more then noticable
> > (see exact numbers below).
> > From my perspective we'd probably keep it inlined for now
> > to avoid any non-anticipated perfomance degradations.
> > Though intersted to see perf results and opinions from
> > other interested parties.
> >
> > Intel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz
> > ring_perf_autotest (without patch/with patch)
> >
> > ### Testing single element enq/deq ###
> > legacy APIs: SP/SC: single: 8.75/43.23
> > legacy APIs: MP/MC: single: 56.18/80.44
> >
> > ### Testing burst enq/deq ###
> > legacy APIs: SP/SC: burst (size: 8): 37.36/53.37
> > legacy APIs: SP/SC: burst (size: 32): 93.97/117.30
> > legacy APIs: MP/MC: burst (size: 8): 78.23/91.45
> > legacy APIs: MP/MC: burst (size: 32): 131.59/152.49
> >
> > ### Testing bulk enq/deq ###
> > legacy APIs: SP/SC: bulk (size: 8): 37.29/54.48
> > legacy APIs: SP/SC: bulk (size: 32): 92.68/113.01
> > legacy APIs: MP/MC: bulk (size: 8): 78.40/93.50
> > legacy APIs: MP/MC: bulk (size: 32): 131.49/154.25
> >
> > ### Testing empty bulk deq ###
> > legacy APIs: SP/SC: bulk (size: 8): 4.00/16.86
> > legacy APIs: MP/MC: bulk (size: 8): 7.01/15.55
> >
> > ### Testing using two hyperthreads ###
> > legacy APIs: SP/SC: bulk (size: 8): 10.64/17.56
> > legacy APIs: MP/MC: bulk (size: 8): 15.30/16.69
> > legacy APIs: SP/SC: bulk (size: 32): 5.84/7.09
> > legacy APIs: MP/MC: bulk (size: 32): 6.34/7.54
> >
> > ### Testing using two physical cores ###
> > legacy APIs: SP/SC: bulk (size: 8): 24.34/42.40
> > legacy APIs: MP/MC: bulk (size: 8): 70.34/71.82
> > legacy APIs: SP/SC: bulk (size: 32): 12.67/14.68
> > legacy APIs: MP/MC: bulk (size: 32): 22.41/17.93
> >
> > ### Testing single element enq/deq ###
> > elem APIs: element size 16B: SP/SC: single: 10.65/41.96
> > elem APIs: element size 16B: MP/MC: single: 44.33/81.36
> >
> > ### Testing burst enq/deq ###
> > elem APIs: element size 16B: SP/SC: burst (size: 8): 39.20/58.52
> > elem APIs: element size 16B: SP/SC: burst (size: 32): 123.19/142.79
> > elem APIs: element size 16B: MP/MC: burst (size: 8): 80.72/101.36
> > elem APIs: element size 16B: MP/MC: burst (size: 32): 169.21/185.38
> >
> > ### Testing bulk enq/deq ###
> > elem APIs: element size 16B: SP/SC: bulk (size: 8): 41.64/58.46
> > elem APIs: element size 16B: SP/SC: bulk (size: 32): 122.74/142.52
> > elem APIs: element size 16B: MP/MC: bulk (size: 8): 80.60/103.14
> > elem APIs: element size 16B: MP/MC: bulk (size: 32): 169.39/186.67
> >
> > ### Testing empty bulk deq ###
> > elem APIs: element size 16B: SP/SC: bulk (size: 8): 5.01/17.17
> > elem APIs: element size 16B: MP/MC: bulk (size: 8): 6.01/14.80
> >
> > ### Testing using two hyperthreads ###
> > elem APIs: element size 16B: SP/SC: bulk (size: 8): 12.02/17.18
> > elem APIs: element size 16B: MP/MC: bulk (size: 8): 16.81/21.14
> > elem APIs: element size 16B: SP/SC: bulk (size: 32): 7.87/9.01
> > elem APIs: element size 16B: MP/MC: bulk (size: 32): 8.22/10.57
> >
> > ### Testing using two physical cores ###
> > elem APIs: element size 16B: SP/SC: bulk (size: 8): 27.00/51.94
> > elem APIs: element size 16B: MP/MC: bulk (size: 8): 78.24/74.48
> > elem APIs: element size 16B: SP/SC: bulk (size: 32): 15.41/16.14
> > elem APIs: element size 16B: MP/MC: bulk (size: 32): 18.72/21.64
> >
> > Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
>
> What is impact with LTO? I suspect compiler might have a chance to
> get speed back with LTO.
Might be, but LTO is not enabled by default.
So don't see much point in digging any further here.
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] devtools: fix check symbol change script
2020-03-19 14:44 3% [dpdk-dev] [PATCH] devtools: fix check symbol change script Nithin Dabilpuram
2020-03-19 14:56 0% ` David Marchand
@ 2020-03-22 14:37 0% ` Jerin Jacob
2020-03-23 11:56 3% ` [dpdk-dev] [PATCH v2] " Nithin Dabilpuram
2 siblings, 0 replies; 200+ results
From: Jerin Jacob @ 2020-03-22 14:37 UTC (permalink / raw)
To: Nithin Dabilpuram
Cc: Thomas Monjalon, David Marchand, Neil Horman, dpdk-dev,
Jerin Jacob, dpdk stable
On Thu, Mar 19, 2020 at 8:14 PM Nithin Dabilpuram
<ndabilpuram@marvell.com> wrote:
>
> Fix check symbol change script to detect new diff file when
> it is in between "--- /dev/null" to "b/lib/...".
> Current awk line expects line to start with "a/..."
> which is not always true for all diffs.
> As a result if in_map was '1' earlier, it will not be changed
> to '0' and we get check patch errors which are not true as the non
> version.map files get interpreted as version map file.
>
> Fixes: 4bec48184e33 ("devtools: add checks for ABI symbol addition")
> Cc: nhorman@tuxdriver.com
> Cc: stable@dpdk.org
>
> Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
Tested-by: Jerin Jacob <jerinj@marvell.com>
> ---
> Note: We have two examples where checkpatch errors are because of this
> because the version.map file change comes earlier in the diff. Because of
> this bug, any new file change that comes after version.map file diff
> as "/dev/null" to "b/.." gets misdetected as version.map file.
> * http://patches.dpdk.org/patch/66878/
> * https://patchwork.dpdk.org/patch/66900/
> devtools/check-symbol-change.sh | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/devtools/check-symbol-change.sh b/devtools/check-symbol-change.sh
> index c5434f3..19ce82f 100755
> --- a/devtools/check-symbol-change.sh
> +++ b/devtools/check-symbol-change.sh
> @@ -17,13 +17,13 @@ build_map_changes()
> # map files are altered, and all section/symbol names
> # appearing between a triggering of this rule and the
> # next trigger of this rule are associated with this file
> - /[-+] a\/.*\.map/ {map=$2; in_map=1}
> + /[-+] [ab]\/.*\.map/ {map=$2; in_map=1}
>
> # Same pattern as above, only it matches on anything that
> # does not end in 'map', indicating we have left the map chunk.
> # When we hit this, turn off the in_map variable, which
> # supresses the subordonate rules below
> - /[-+] a\/.*\.[^map]/ {in_map=0}
> + /[-+] [ab]\/.*\.[^map]/ {in_map=0}
>
> # Triggering this rule, which starts a line and ends it
> # with a { identifies a versioned section. The section name is
> --
> 2.8.4
>
^ permalink raw reply [relevance 0%]
* [dpdk-dev] [PATCH v2] devtools: fix check symbol change script
2020-03-19 14:44 3% [dpdk-dev] [PATCH] devtools: fix check symbol change script Nithin Dabilpuram
2020-03-19 14:56 0% ` David Marchand
2020-03-22 14:37 0% ` Jerin Jacob
@ 2020-03-23 11:56 3% ` Nithin Dabilpuram
2020-03-23 13:19 0% ` David Marchand
2 siblings, 1 reply; 200+ results
From: Nithin Dabilpuram @ 2020-03-23 11:56 UTC (permalink / raw)
To: thomas, david.marchand, Neil Horman
Cc: dev, jerinj, bingz, Nithin Dabilpuram, stable
Fix check symbol change script to detect new diff file when
it is in between "--- /dev/null" to "b/lib/...".
Current awk line expects line to start with "a/..."
which is not always true for all diffs.
As a result if in_map was '1' earlier, it will not be changed
to '0' and we get check patch errors which are not true.
Fixes: 4bec48184e33 ("devtools: add checks for ABI symbol addition")
Cc: nhorman@tuxdriver.com
Cc: stable@dpdk.org
Signed-off-by: David Marchand <david.marchand@redhat.com>
Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
Acked-by: Neil Horman <nhorman@tuxdriver.com>
Tested-by: Jerin Jacob <jerinj@marvell.com>
---
v2:
- Updated logic from David to fix "not map" check
devtools/check-symbol-change.sh | 10 ++++------
1 file changed, 4 insertions(+), 6 deletions(-)
diff --git a/devtools/check-symbol-change.sh b/devtools/check-symbol-change.sh
index c5434f3..ed2178e 100755
--- a/devtools/check-symbol-change.sh
+++ b/devtools/check-symbol-change.sh
@@ -17,13 +17,11 @@ build_map_changes()
# map files are altered, and all section/symbol names
# appearing between a triggering of this rule and the
# next trigger of this rule are associated with this file
- /[-+] a\/.*\.map/ {map=$2; in_map=1}
+ /[-+] [ab]\/.*\.map/ {map=$2; in_map=1; next}
- # Same pattern as above, only it matches on anything that
- # does not end in 'map', indicating we have left the map chunk.
- # When we hit this, turn off the in_map variable, which
- # supresses the subordonate rules below
- /[-+] a\/.*\.[^map]/ {in_map=0}
+ # The previous rule catches all .map files, anything else
+ # indicates we left the map chunk.
+ /[-+] [ab]\// {in_map=0}
# Triggering this rule, which starts a line and ends it
# with a { identifies a versioned section. The section name is
--
2.8.4
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH v2] devtools: fix check symbol change script
2020-03-23 11:56 3% ` [dpdk-dev] [PATCH v2] " Nithin Dabilpuram
@ 2020-03-23 13:19 0% ` David Marchand
0 siblings, 0 replies; 200+ results
From: David Marchand @ 2020-03-23 13:19 UTC (permalink / raw)
To: Nithin Dabilpuram
Cc: Thomas Monjalon, Neil Horman, dev, Jerin Jacob Kollanukkaran,
bingz, dpdk stable
On Mon, Mar 23, 2020 at 12:56 PM Nithin Dabilpuram
<ndabilpuram@marvell.com> wrote:
>
> Fix check symbol change script to detect new diff file when
> it is in between "--- /dev/null" to "b/lib/...".
> Current awk line expects line to start with "a/..."
> which is not always true for all diffs.
> As a result if in_map was '1' earlier, it will not be changed
> to '0' and we get check patch errors which are not true.
>
> Fixes: 4bec48184e33 ("devtools: add checks for ABI symbol addition")
> Cc: stable@dpdk.org
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
> Acked-by: Neil Horman <nhorman@tuxdriver.com>
> Tested-by: Jerin Jacob <jerinj@marvell.com>
Thanks Nithin, applied.
--
David Marchand
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH dpdk-dev v3] mempool: sort the rte_mempool_ops by name
2020-03-09 8:27 3% ` Olivier Matz
2020-03-09 8:55 0% ` Tonghao Zhang
@ 2020-03-24 9:35 0% ` Andrew Rybchenko
2020-03-24 12:41 0% ` Tonghao Zhang
1 sibling, 1 reply; 200+ results
From: Andrew Rybchenko @ 2020-03-24 9:35 UTC (permalink / raw)
To: Olivier Matz, Tonghao Zhang
Cc: Jerin Jacob, dpdk-dev, Gage Eads, Artem V. Andreev, Jerin Jacob,
Nithin Dabilpuram, Vamsi Attunuru, Hemant Agrawal
On 3/9/20 11:27 AM, Olivier Matz wrote:
> Hi,
>
> On Mon, Mar 09, 2020 at 11:01:25AM +0800, Tonghao Zhang wrote:
>> On Sat, Mar 7, 2020 at 8:54 PM Andrew Rybchenko
>> <arybchenko@solarflare.com> wrote:
>>>
>>> On 3/7/20 3:51 PM, Andrew Rybchenko wrote:
>>>> On 3/6/20 4:37 PM, Jerin Jacob wrote:
>>>>> On Fri, Mar 6, 2020 at 7:06 PM <xiangxia.m.yue@gmail.com> wrote:
>>>>>> From: Tonghao Zhang <xiangxia.m.yue@gmail.com>
>>>>>>
>>>>>> The order of mempool initiation affects mempool index in the
>>>>>> rte_mempool_ops_table. For example, when building APPs with:
>>>>>>
>>>>>> $ gcc -lrte_mempool_bucket -lrte_mempool_ring ...
>>>>>>
>>>>>> The "bucket" mempool will be registered firstly, and its index
>>>>>> in table is 0 while the index of "ring" mempool is 1. DPDK
>>>>>> uses the mk/rte.app.mk to build APPs, and others, for example,
>>>>>> Open vSwitch, use the libdpdk.a or libdpdk.so to build it.
>>>>>> The mempool lib linked in dpdk and Open vSwitch is different.
>>>>>>
>>>>>> The mempool can be used between primary and secondary process,
>>>>>> such as dpdk-pdump and pdump-pmd/Open vSwitch(pdump enabled).
>>>>>> There will be a crash because dpdk-pdump creates the "ring_mp_mc"
>>>>>> ring which index in table is 0, but the index of "bucket" ring
>>>>>> is 0 in Open vSwitch. If Open vSwitch use the index 0 to get
>>>>>> mempool ops and malloc memory from mempool. The crash will occur:
>>>>>>
>>>>>> bucket_dequeue (access null and crash)
>>>>>> rte_mempool_get_ops (should get "ring_mp_mc",
>>>>>> but get "bucket" mempool)
>>>>>> rte_mempool_ops_dequeue_bulk
>>>>>> ...
>>>>>> rte_pktmbuf_alloc
>>>>>> rte_pktmbuf_copy
>>>>>> pdump_copy
>>>>>> pdump_rx
>>>>>> rte_eth_rx_burst
>>>>>>
>>>>>> To avoid the crash, there are some solution:
>>>>>> * constructor priority: Different mempool uses different
>>>>>> priority in RTE_INIT, but it's not easy to maintain.
>>>>>>
>>>>>> * change mk/rte.app.mk: Change the order in mk/rte.app.mk to
>>>>>> be same as libdpdk.a/libdpdk.so, but when adding a new mempool
>>>>>> driver in future, we must make sure the order.
>>>>>>
>>>>>> * register mempool orderly: Sort the mempool when registering,
>>>>>> so the lib linked will not affect the index in mempool table.
>>>>>>
>>>>>> Signed-off-by: Tonghao Zhang <xiangxia.m.yue@gmail.com>
>>>>>> Acked-by: Olivier Matz <olivier.matz@6wind.com>
>>>>> Acked-by: Jerin Jacob <jerinj@marvell.com>
>>>>
>>>> The patch is OK, but the fact that ops index changes during
>>>> mempool driver lifetime is frightening. In fact it breaks
>>>> rte_mempool_register_ops() return value semantics (read
>>>> as API break). The return value is not used in DPDK, but it
>>>> is a public function. If I'm not mistaken it should be taken
>>>> into account.
>
> Good points.
>
> The fact that the ops index changes during mempool driver lifetime is
> indeed frightening, especially knowning that this is a dynamic
> registration that could happen at any moment in the life of the
> application. Also, breaking the ABI is not desirable.
>
> Let me try to propose something else to solve your issue:
>
> 1/ At init, the primary process allocates a struct in shared memory
> (named memzone):
>
> struct rte_mempool_shared_ops {
> size_t num_mempool_ops;
> struct {
> char name[RTE_MEMPOOL_OPS_NAMESIZE];
> } mempool_ops[RTE_MEMPOOL_MAX_OPS_IDX];
> char *mempool_ops_name[RTE_MEMPOOL_MAX_OPS_IDX];
> rte_spinlock_t mempool;
> }
>
> 2/ When we register a mempool ops, we first get a name and id from the
> shared struct: with the lock held, lookup for the registered name and
> return its index, else get the last id and copy the name in the struct.
>
> 3/ Then do as before (in the per-process global table), except that we
> reuse the registered id.
>
> We can remove the num_ops field from rte_mempool_ops_table.
>
> Thoughts?
I like the solution.
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH dpdk-dev v3] mempool: sort the rte_mempool_ops by name
2020-03-24 9:35 0% ` Andrew Rybchenko
@ 2020-03-24 12:41 0% ` Tonghao Zhang
0 siblings, 0 replies; 200+ results
From: Tonghao Zhang @ 2020-03-24 12:41 UTC (permalink / raw)
To: Andrew Rybchenko
Cc: Olivier Matz, Jerin Jacob, dpdk-dev, Gage Eads, Artem V. Andreev,
Jerin Jacob, Nithin Dabilpuram, Vamsi Attunuru, Hemant Agrawal
On Tue, Mar 24, 2020 at 5:36 PM Andrew Rybchenko
<arybchenko@solarflare.com> wrote:
>
> On 3/9/20 11:27 AM, Olivier Matz wrote:
> > Hi,
> >
> > On Mon, Mar 09, 2020 at 11:01:25AM +0800, Tonghao Zhang wrote:
> >> On Sat, Mar 7, 2020 at 8:54 PM Andrew Rybchenko
> >> <arybchenko@solarflare.com> wrote:
> >>>
> >>> On 3/7/20 3:51 PM, Andrew Rybchenko wrote:
> >>>> On 3/6/20 4:37 PM, Jerin Jacob wrote:
> >>>>> On Fri, Mar 6, 2020 at 7:06 PM <xiangxia.m.yue@gmail.com> wrote:
> >>>>>> From: Tonghao Zhang <xiangxia.m.yue@gmail.com>
> >>>>>>
> >>>>>> The order of mempool initiation affects mempool index in the
> >>>>>> rte_mempool_ops_table. For example, when building APPs with:
> >>>>>>
> >>>>>> $ gcc -lrte_mempool_bucket -lrte_mempool_ring ...
> >>>>>>
> >>>>>> The "bucket" mempool will be registered firstly, and its index
> >>>>>> in table is 0 while the index of "ring" mempool is 1. DPDK
> >>>>>> uses the mk/rte.app.mk to build APPs, and others, for example,
> >>>>>> Open vSwitch, use the libdpdk.a or libdpdk.so to build it.
> >>>>>> The mempool lib linked in dpdk and Open vSwitch is different.
> >>>>>>
> >>>>>> The mempool can be used between primary and secondary process,
> >>>>>> such as dpdk-pdump and pdump-pmd/Open vSwitch(pdump enabled).
> >>>>>> There will be a crash because dpdk-pdump creates the "ring_mp_mc"
> >>>>>> ring which index in table is 0, but the index of "bucket" ring
> >>>>>> is 0 in Open vSwitch. If Open vSwitch use the index 0 to get
> >>>>>> mempool ops and malloc memory from mempool. The crash will occur:
> >>>>>>
> >>>>>> bucket_dequeue (access null and crash)
> >>>>>> rte_mempool_get_ops (should get "ring_mp_mc",
> >>>>>> but get "bucket" mempool)
> >>>>>> rte_mempool_ops_dequeue_bulk
> >>>>>> ...
> >>>>>> rte_pktmbuf_alloc
> >>>>>> rte_pktmbuf_copy
> >>>>>> pdump_copy
> >>>>>> pdump_rx
> >>>>>> rte_eth_rx_burst
> >>>>>>
> >>>>>> To avoid the crash, there are some solution:
> >>>>>> * constructor priority: Different mempool uses different
> >>>>>> priority in RTE_INIT, but it's not easy to maintain.
> >>>>>>
> >>>>>> * change mk/rte.app.mk: Change the order in mk/rte.app.mk to
> >>>>>> be same as libdpdk.a/libdpdk.so, but when adding a new mempool
> >>>>>> driver in future, we must make sure the order.
> >>>>>>
> >>>>>> * register mempool orderly: Sort the mempool when registering,
> >>>>>> so the lib linked will not affect the index in mempool table.
> >>>>>>
> >>>>>> Signed-off-by: Tonghao Zhang <xiangxia.m.yue@gmail.com>
> >>>>>> Acked-by: Olivier Matz <olivier.matz@6wind.com>
> >>>>> Acked-by: Jerin Jacob <jerinj@marvell.com>
> >>>>
> >>>> The patch is OK, but the fact that ops index changes during
> >>>> mempool driver lifetime is frightening. In fact it breaks
> >>>> rte_mempool_register_ops() return value semantics (read
> >>>> as API break). The return value is not used in DPDK, but it
> >>>> is a public function. If I'm not mistaken it should be taken
> >>>> into account.
> >
> > Good points.
> >
> > The fact that the ops index changes during mempool driver lifetime is
> > indeed frightening, especially knowning that this is a dynamic
> > registration that could happen at any moment in the life of the
> > application. Also, breaking the ABI is not desirable.
> >
> > Let me try to propose something else to solve your issue:
> >
> > 1/ At init, the primary process allocates a struct in shared memory
> > (named memzone):
> >
> > struct rte_mempool_shared_ops {
> > size_t num_mempool_ops;
> > struct {
> > char name[RTE_MEMPOOL_OPS_NAMESIZE];
> > } mempool_ops[RTE_MEMPOOL_MAX_OPS_IDX];
> > char *mempool_ops_name[RTE_MEMPOOL_MAX_OPS_IDX];
> > rte_spinlock_t mempool;
> > }
> >
> > 2/ When we register a mempool ops, we first get a name and id from the
> > shared struct: with the lock held, lookup for the registered name and
> > return its index, else get the last id and copy the name in the struct.
> >
> > 3/ Then do as before (in the per-process global table), except that we
> > reuse the registered id.
> >
> > We can remove the num_ops field from rte_mempool_ops_table.
> >
> > Thoughts?
>
> I like the solution.
The patch will be sent, thanks.
--
Best regards, Tonghao
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v2 0/4] Introduce IF proxy library
@ 2020-03-25 8:08 3% ` David Marchand
2020-03-25 11:11 4% ` Morten Brørup
2020-03-26 12:41 3% ` Andrzej Ostruszka
0 siblings, 2 replies; 200+ results
From: David Marchand @ 2020-03-25 8:08 UTC (permalink / raw)
To: Andrzej Ostruszka; +Cc: dev
Hello Andrzej,
On Tue, Mar 10, 2020 at 12:11 PM Andrzej Ostruszka
<aostruszka@marvell.com> wrote:
>
> What is this useful for
> =======================
>
> Usually, when an ethernet port is assigned to DPDK it vanishes from the
> system and user looses ability to control it via normal configuration
> utilities (e.g. those from iproute2 package). Moreover by default DPDK
> application is not aware of the network configuration of the system.
>
> To address both of these issues application needs to:
> - add some command line interface (or other mechanism) allowing for
> control of the port and its configuration
> - query the status of network configuration and monitor its changes
>
> The purpose of this library is to help with both of these tasks (as long
> as they remain in domain of configuration available to the system). In
> other words, if DPDK application has some special needs, that cannot be
> addressed by the normal system configuration utilities, then they need
> to be solved by the application itself.
>
> The connection between DPDK and system is based on the existence of
> ports that are visible to both DPDK and system (like Tap, KNI and
> possibly some other drivers). These ports serve as an interface
> proxies.
>
> Let's visualize the action of the library by the following example:
>
> Linux | DPDK
> ==============================================================
> |
> | +-------+ +-------+
> | | Port1 | | Port2 |
> "ip link set dev tap1 mtu 1600" | +-------+ +-------+
> | | ^ ^ ^
> | +------+ | mtu_change | |
> `->| Tap1 |---' callback | |
> +------+ | |
> "ip addr add 198.51.100.14 \ | | |
> dev tap2" | | |
> | +------+ | |
> +->| Tap2 |------------------' |
> | +------+ addr_add callback |
> "ip route add 198.0.2.0/24 \ | | |
> dev tap2" | | route_add callback |
> | `---------------------'
>
> So we have two ports Port1 and Port2 that are not visible to the system.
> We create two proxy interfaces (here based on Tap driver) and bind the
> ports to their proxies. When user issues a command changing MTU for
> Tap1 interface the library notes this and calls "mtu_change" callback
> for the Port1. Similarly when user adds an IPv4 address to the Tap2
> interface "addr_add" callback is called for the Port2 and the same
> happens for configuration of routing rule pointing to Tap2. Apart from
> callbacks this library can notify about changes via adding events to
> notification queues. See below for more inforamtion about that and
> a complete list of available callbacks.
>
> Please note that nothing has been mentioned about forwarding of the
> packets between system and DPDK. Since the proxies are normal DPDK
> ports you can receive/send to them via usual RX/TX burst API. However
> since the library is not aware of the structure of packet processing
> used by the application it cannot automatically forward the packets - it
> is responsibility of the application to include proxy ports into its
> packet processing engine.
>
> As mentioned above the intention of the library is to:
> - provide information about network configuration that would allow
> application to decide what to do with the packets received on DPDK
> ports,
> - allow for control of the ports via standard configuration utilities
>
> Although the library only helps you to identify proxy for given port
> (and vice versa) and calls appropriate callbacks it does open some
> interesting possibilities. For example you can use the proxy ports to
> forward packets for protocols that you do not wish to handle in DPDK
> application to the system protocol stack and just listen to the
> configuration changes - so that way you can "offload" handling of those
> protocols to the system.
>
> How to use it
> =============
>
> Usage of this library is rather simple. You have to:
> 1. Create proxy (if you don't have port suitable for being proxy or you
> have one but do not wish to use it as a proxy).
> 2. Bind port to proxy.
> 3. Register callbacks and/or event queues.
> 4. Start listening to the network configuration.
>
> The only mandatory requirement for DPDK port to be able to act as
> a proxy is that it is visible in the system - this is checked during
> port to proxy binding by calling rte_eth_dev_info_get() on proxy port
> and inspecting 'if_index' field (it has to be non-zero).
> One can create such port in the application by calling:
>
> proxy_id = rte_ifpx_create(RTE_IFPX_DEFAULT);
>
> Upon success this returns id of DPDK proxy port created
> (RTE_MAX_ETHPORTS on failure). The argument selects type of proxy port
> to create (currently Tap/KNI only). This function actually is just
> a wrapper around:
>
> uint16_t rte_ifpx_create_by_devarg(const char *devarg);
>
> creating valid 'devarg' string for the chosen type of proxy. If you have
> other driver capable of acting as a proxy you can call
> rte_ifpx_create_by_devarg() directly passing appropriate argument.
>
> Once you have id of both port and proxy you can bind the two via:
>
> rte_ifpx_port_bind(port_id, proxy_id);
>
> This creates logical binding - as mentioned above there is no automatic
> packet forwarding. With this binding whenever user changes the state of
> proxy interface in the system (link up/down, change mac/mtu, add/remove
> IPv4/IPv6) you get appropriate notification for the bound port.
>
> So far we've mentioned several times that the library calls callbacks.
> They are grouped in 'struct rte_ifpx_callbacks' and user provides them
> to the library via:
>
> rte_ifpx_callbacks_register(&cbs);
>
> It is worth mentioning that the context (lcore/thread) in which these
> callbacks are called is implementation defined. It might differ between
> different platforms, so the application needs to assume that some kind
> of inter lcore/thread synchronization/communication is required.
>
> Apart from notification via callbacks this library also supports
> notifying about the changes via adding events to the configured
> notification queues. The queues are registered via:
>
> int rte_ifpx_queue_add(struct rte_ring *r);
>
> and the actual logic used is: if there is callback registered then it is
> called, if it returns non-zero then event is considered completed,
> otherwise event is added to each configured notification queue.
> That way application can update data structures that are safe to be
> modified by single writer from within callback or do the common
> preprocessing steps (if any needed) in callback and data that is
> replicated can be updated during handling of queued events.
>
> Once we have bindings in place and notification configured, the only
> essential part that remains is to get the current network configuration
> and start listening to its changes. This is accomplished via a call to:
>
> rte_ifpx_listen();
>
> And basically this is all one needs to understand how to use this
> library. Other less essential parts include:
> - ability to query what events are available for given platform
> - getting mapping between proxy and port
> - unbinding the ports from proxy
> - destroying proxy port
> - closing the listening service
> - getting basic information about proxy
>
>
> Currently available features and implementation
> ===============================================
>
> The library's API is system independent but it obviously needs some
> system dependent parts. We provide exemplary Linux implementation (based
> on netlink sockets). Very similar implementation is possible for
> FreeBSD (with the usage of PF_ROUTE sockets). Windows implementation
> would need to differ much (probably IP Helper library would be of some help).
>
> Here is the list of currently implemented callbacks:
>
> struct rte_ifpx_callbacks {
> int (*mac_change)(const struct rte_ifpx_mac_change *event);
> int (*mtu_change)(const struct rte_ifpx_mtu_change *event);
> int (*link_change)(const struct rte_ifpx_link_change *event);
> int (*addr_add)(const struct rte_ifpx_addr_change *event);
> int (*addr_del)(const struct rte_ifpx_addr_change *event);
> int (*addr6_add)(const struct rte_ifpx_addr6_change *event);
> int (*addr6_del)(const struct rte_ifpx_addr6_change *event);
> int (*route_add)(const struct rte_ifpx_route_change *event);
> int (*route_del)(const struct rte_ifpx_route_change *event);
> int (*route6_add)(const struct rte_ifpx_route6_change *event);
> int (*route6_del)(const struct rte_ifpx_route6_change *event);
> int (*neigh_add)(const struct rte_ifpx_neigh_change *event);
> int (*neigh_del)(const struct rte_ifpx_neigh_change *event);
> int (*neigh6_add)(const struct rte_ifpx_neigh6_change *event);
> int (*neigh6_del)(const struct rte_ifpx_neigh6_change *event);
> int (*cfg_done)(void);
> };
>
> They are all rather self-descriptive with the exception of the last one.
> When the user calls rte_ifpx_listen() the library first queries the
> system for its current configuration. That might require several
> request/reply exchanges between DPDK and system and once it is finished
> this callback is called to let application know that all info has been
> gathered.
>
> It is worth to mention also that while typical case would be a 1-to-1
> mapping between port and proxy, the 1-to-many mapping is also supported.
> In that case related callbacks will be called for each port bound to
> given proxy interface - it is application responsibility to define
> semantic of such mapping (e.g. all changes apply to all ports, or link
> changes apply to all but other are accepted in "round robin" fashion, or
> some other logic).
>
> As mentioned above Linux implementation is based on netlink socket.
> This socket is registered as file descriptor in EAL interrupts
> (similarly to how EAL alarms are implemented).
>
> What has changed since the RFC
> ==============================
>
> - Platform dependent parts has been separated into a ifpx_platform
> structure with callbacks for initialization, getting information about
> the interface, listening to the changes and closing of the library.
> That should allow easier reimplementation.
>
> - Notification scheme has been changed - instead of having just
> callbacks now event queueing is also available (or a mix of those
> two).
>
> - Filtering of events only related to the proxy ports - previously all
> network configuration changes were reported. But DPDK application
> doesn't need to know whole configuration - only just portion related
> to the proxy ports. If a packet comes that does not match rules then
> it can be forwarded via proxy to the system to decide what to do with
> it. If that is not desired and such packets should be dropped then
> null port can be created with proxy and e.g. default route installed
> on it.
>
> - Removed previous example which was just printing notification.
> Instead added a simplified (stripped vectorization and other
> performance improvements) version of l3fwd that should serve as an
> example of using this library in real applications.
>
> Changes in V2
> =============
> - Cleaned up checkpatch warnings
> - Removed dead/unused code and added gateway clearing in l3fwd-ifpx
I can see we end up exposing structures for registering callbacks.
Did you consider some ways to avoid exposure of those? (thinking of
ABI maintenance for when this library will elect to non-experimental).
I can see some canary at the end of an enum, can we do without it?
Is there a pb with merging ifpx support into the existing l3fwd
application rather than introduce a new example?
--
David Marchand
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH v2 0/4] Introduce IF proxy library
2020-03-25 8:08 3% ` David Marchand
@ 2020-03-25 11:11 4% ` Morten Brørup
2020-03-26 17:42 3% ` Andrzej Ostruszka
2020-03-26 12:41 3% ` Andrzej Ostruszka
1 sibling, 1 reply; 200+ results
From: Morten Brørup @ 2020-03-25 11:11 UTC (permalink / raw)
To: Andrzej Ostruszka, David Marchand; +Cc: dev
Andrzej,
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of David Marchand
> Sent: Wednesday, March 25, 2020 9:08 AM
>
> Hello Andrzej,
>
> On Tue, Mar 10, 2020 at 12:11 PM Andrzej Ostruszka
> <aostruszka@marvell.com> wrote:
> >
<snip>
> >
> > What has changed since the RFC
> > ==============================
> >
> > - Platform dependent parts has been separated into a ifpx_platform
> > structure with callbacks for initialization, getting information
> about
> > the interface, listening to the changes and closing of the library.
> > That should allow easier reimplementation.
> >
> > - Notification scheme has been changed - instead of having just
> > callbacks now event queueing is also available (or a mix of those
> > two).
Thank you for adding event queueing!
David mentions ABI forward compatibility below. Consider using a dynamically sized generic TLV (type, length, value) message format instead of a big union structure for the events. This would make it easier to extend the list of event types without breaking the ABI.
And I am still strongly opposed to the callback method:
The callbacks are handled as DPDK interrupts, which are running in a non-DPDK thread, i.e. a running callback may be preempted by some other Linux process. This makes it difficult to implement callbacks correctly.
The risk of someone calling a non-thread safe function from a callback is high, e.g. DPDK hash table manipulation (except lookup) is not thread safe.
Your documentation is far too vague about this:
Please note however that the context in which these callbacks are
called is most probably different from the one in which packets are
handled and it is application writer responsibility to use proper
synchronization mechanisms - if they are needed.
You need a big fat WARNING about how difficult the DPDK interrupt thread is to work with. As I described above, it is not "most probably" it is "certainly" a very different kind of context.
Did you check that the functions you use in your example callbacks are all thread safe and non-blocking, so they can safely be called from a non-DPDK thread that may be preempted by a another Linux process?
<snip>
>
> I can see we end up exposing structures for registering callbacks.
> Did you consider some ways to avoid exposure of those? (thinking of
> ABI maintenance for when this library will elect to non-experimental).
> I can see some canary at the end of an enum, can we do without it?
>
> Is there a pb with merging ifpx support into the existing l3fwd
> application rather than introduce a new example?
>
>
> --
> David Marchand
>
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [RFC 0/6] New sync modes for ring
2020-02-24 11:35 2% [dpdk-dev] [RFC 0/6] New sync modes for ring Konstantin Ananyev
2020-02-24 16:59 0% ` Stephen Hemminger
@ 2020-03-25 20:43 0% ` Honnappa Nagarahalli
2020-03-26 1:50 3% ` Ananyev, Konstantin
1 sibling, 1 reply; 200+ results
From: Honnappa Nagarahalli @ 2020-03-25 20:43 UTC (permalink / raw)
To: Konstantin Ananyev, dev; +Cc: olivier.matz, nd, Honnappa Nagarahalli, nd
<snip>
> Subject: [dpdk-dev] [RFC 0/6] New sync modes for ring
>
> Upfront note - that RFC is not a complete patch.
> It introduces an ABI breakage, plus it doesn't update ring_elem code properly,
As per the current rules, these changes (in the current form) will be accepted only for 20.11 release. How do we address this for immediate requirements like RCU defer APIs?
I suggest that we move forward with my RFC (taking into consideration your feedback) to make progress on RCU APIs.
> etc.
> I plan to deal with all these things in later versions.
> Right now I seek an initial feedback about proposed ideas.
> Would also ask people to repeat performance tests (see below) on their
> platforms to confirm the impact.
>
> More and more customers use(/try to use) DPDK based apps within
> overcommitted systems (multiple acttive threads over same pysical cores):
> VM, container deployments, etc.
> One quite common problem they hit: Lock-Holder-Preemption with rte_ring.
> LHP is quite a common problem for spin-based sync primitives (spin-locks, etc.)
> on overcommitted systems.
> The situation gets much worse when some sort of fair-locking technique is
> used (ticket-lock, etc.).
> As now not only lock-owner but also lock-waiters scheduling order matters a
> lot.
> This is a well-known problem for kernel within VMs:
> http://www-archive.xenproject.org/files/xensummitboston08/LHP.pdf
> https://www.cs.hs-rm.de/~kaiser/events/wamos2017/Slides/selcuk.pdf
> The problem with rte_ring is that while head accusion is sort of un-fair locking,
> waiting on tail is very similar to ticket lock schema - tail has to be updated in
> particular order.
> That makes current rte_ring implementation to perform really pure on some
> overcommited scenarios.
> While it is probably not possible to completely resolve this problem in
> userspace only (without some kernel communication/intervention), removing
> fairness in tail update can mitigate it significantly.
> So this RFC proposes two new optional ring synchronization modes:
> 1) Head/Tail Sync (HTS) mode
> In that mode enqueue/dequeue operation is fully serialized:
> only one thread at a time is allowed to perform given op.
> As another enhancement provide ability to split enqueue/dequeue
> operation into two phases:
> - enqueue/dequeue start
> - enqueue/dequeue finish
> That allows user to inspect objects in the ring without removing
> them from it (aka MT safe peek).
IMO, this will not address the problem described above. For ex: when a producer updates the head and gets scheduled out, other producers have to spin. The problem is probably worse as with non-HTS case moving of the head and copying of the ring elements can happen in parallel between the producers (similarly for consumers).
IMO, HTS should not be a configurable flag. In RCU requirement, a MP enqueue and HTS dequeue are required.
> 2) Relaxed Tail Sync (RTS)
> The main difference from original MP/MC algorithm is that tail value is
> increased not by every thread that finished enqueue/dequeue, but only by the
> last one.
> That allows threads to avoid spinning on ring tail value, leaving actual tail value
> change to the last thread in the update queue.
This can be a configurable flag on the ring.
I am not sure how this solves the problem you have stated above completely. Updating the count from all intermediate threads is still required to update the value of the head. But yes, it reduces the severity of the problem by not enforcing the order in which the tail is updated.
I also think it introduces the problem on the other side of the ring because the tail is not updated soon enough (the other side has to wait longer for the elements to become available). It also introduces another configuration parameter (HTD_MAX_DEF) which they have to deal with.
Users have to still implement the current hypervisor related solutions.
IMO, we should run the benchmark for this on an over committed setup to understand the benefits.
>
> Test results on IA (see below) show significant improvements for average
> enqueue/dequeue op times on overcommitted systems.
> For 'classic' DPDK deployments (one thread per core) original MP/MC
> algorithm still shows best numbers, though for 64-bit target RTS numbers are
> not that far away.
> Numbers were produced by ring_stress_*autotest (first patch in these series).
>
> X86_64 @ Intel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz
> DEQ+ENQ average cycles/obj
>
> MP/MC HTS RTS
> 1thread@1core(--lcores=6-7) 8.00 8.15 8.99
> 2thread@2core(--lcores=6-8) 19.14 19.61 20.35
> 4thread@4core(--lcores=6-10) 29.43 29.79 31.82
> 8thread@8core(--lcores=6-14) 110.59 192.81 119.50
> 16thread@16core(--lcores=6-22) 461.03 813.12 495.59
> 32thread/@32core(--lcores='6-22,55-70') 982.90 1972.38 1160.51
>
> 2thread@1core(--lcores='6,(10-11)@7' 20140.50 23.58 25.14
> 4thread@2core(--lcores='6,(10-11)@7,(20-21)@8' 153680.60 76.88 80.05
> 8thread@2core(--lcores='6,(10-13)@7,(20-23)@8' 280314.32 294.72 318.79
> 16thread@2core(--lcores='6,(10-17)@7,(20-27)@8' 643176.59 1144.02
> 1175.14 32thread@2core(--lcores='6,(10-25)@7,(30-45)@8' 4264238.80
> 4627.48 4892.68
>
> 8thread@2core(--lcores='6,(10-17)@(7,8))' 321085.98 298.59 307.47
> 16thread@4core(--lcores='6,(20-35)@(7-10))' 1900705.61 575.35 678.29
> 32thread@4core(--lcores='6,(20-51)@(7-10))' 5510445.85 2164.36 2714.12
>
> i686 @ Intel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz
> DEQ+ENQ average cycles/obj
>
> MP/MC HTS RTS
> 1thread@1core(--lcores=6-7) 7.85 12.13 11.31
> 2thread@2core(--lcores=6-8) 17.89 24.52 21.86
> 8thread@8core(--lcores=6-14) 32.58 354.20 54.58
> 32thread/@32core(--lcores='6-22,55-70') 813.77 6072.41 2169.91
>
> 2thread@1core(--lcores='6,(10-11)@7' 16095.00 36.06 34.74
> 8thread@2core(--lcores='6,(10-13)@7,(20-23)@8' 1140354.54 346.61 361.57
> 16thread@2core(--lcores='6,(10-17)@7,(20-27)@8' 1920417.86 1314.90
> 1416.65
>
> 8thread@2core(--lcores='6,(10-17)@(7,8))' 594358.61 332.70 357.74
> 32thread@4core(--lcores='6,(20-51)@(7-10))' 5319896.86 2836.44 3028.87
>
> Konstantin Ananyev (6):
> test/ring: add contention stress test
> ring: rework ring layout to allow new sync schemes
> ring: introduce RTS ring mode
> test/ring: add contention stress test for RTS ring
> ring: introduce HTS ring mode
> test/ring: add contention stress test for HTS ring
>
> app/test/Makefile | 3 +
> app/test/meson.build | 3 +
> app/test/test_pdump.c | 6 +-
> app/test/test_ring_hts_stress.c | 28 ++
> app/test/test_ring_rts_stress.c | 28 ++
> app/test/test_ring_stress.c | 27 ++
> app/test/test_ring_stress.h | 477 +++++++++++++++++++
> lib/librte_pdump/rte_pdump.c | 2 +-
> lib/librte_port/rte_port_ring.c | 12 +-
> lib/librte_ring/Makefile | 4 +-
> lib/librte_ring/meson.build | 4 +-
> lib/librte_ring/rte_ring.c | 84 +++-
> lib/librte_ring/rte_ring.h | 619 +++++++++++++++++++++++--
> lib/librte_ring/rte_ring_elem.h | 8 +-
> lib/librte_ring/rte_ring_hts_generic.h | 228 +++++++++
> lib/librte_ring/rte_ring_rts_generic.h | 240 ++++++++++
> 16 files changed, 1721 insertions(+), 52 deletions(-) create mode 100644
> app/test/test_ring_hts_stress.c create mode 100644
> app/test/test_ring_rts_stress.c create mode 100644
> app/test/test_ring_stress.c create mode 100644 app/test/test_ring_stress.h
> create mode 100644 lib/librte_ring/rte_ring_hts_generic.h
> create mode 100644 lib/librte_ring/rte_ring_rts_generic.h
>
> --
> 2.17.1
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [RFC] ring: make ring implementation non-inlined
2020-03-20 16:41 3% [dpdk-dev] [RFC] ring: make ring implementation non-inlined Konstantin Ananyev
2020-03-20 17:54 0% ` Stephen Hemminger
@ 2020-03-25 21:09 4% ` Jerin Jacob
2020-03-26 0:28 0% ` Ananyev, Konstantin
2020-03-26 8:04 4% ` Morten Brørup
1 sibling, 2 replies; 200+ results
From: Jerin Jacob @ 2020-03-25 21:09 UTC (permalink / raw)
To: Konstantin Ananyev
Cc: dpdk-dev, Olivier Matz, Honnappa Nagarahalli, Jerin Jacob,
David Christensen, Stephen Hemminger
On Fri, Mar 20, 2020 at 10:11 PM Konstantin Ananyev
<konstantin.ananyev@intel.com> wrote:
>
> As was discussed here:
> http://mails.dpdk.org/archives/dev/2020-February/158586.html
> this RFC aimed to hide ring internals into .c and make all
> ring functions non-inlined. In theory that might help to
> maintain ABI stability in future.
> This is just a POC to measure the impact of proposed idea,
> proper implementation would definetly need some extra effort.
> On IA box (SKX) ring_perf_autotest shows ~20-30 cycles extra for
> enqueue+dequeue pair. On some more realistic code, I suspect
> the impact it might be a bit higher.
> For MP/MC bulk transfers degradation seems quite small,
> though for SP/SC and/or small transfers it is more then noticable
> (see exact numbers below).
> From my perspective we'd probably keep it inlined for now
> to avoid any non-anticipated perfomance degradations.
> Though intersted to see perf results and opinions from
> other interested parties.
+1
My reasoning is a bit different, DPDK is using in embedded boxes too
where performance has
more weight than ABI stuff. I think we need to focus first on slow
path APIs ABI stuff.
I spend a few cycles to apply this patch +
http://mails.dpdk.org/archives/dev/2020-February/158586.html
on top of the tree, there are a lot of conflicts. If I get a mergeable
patch then I will test it on an arm64 box.
>
> Intel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz
> ring_perf_autotest (without patch/with patch)
^ permalink raw reply [relevance 4%]
* [dpdk-dev] [PATCH v2 00/32] DPDK Trace support
@ 2020-03-25 21:15 1% ` jerinj
0 siblings, 0 replies; 200+ results
From: jerinj @ 2020-03-25 21:15 UTC (permalink / raw)
Cc: dev, thomas, bruce.richardson, david.marchand, mattias.ronnblom,
skori, Jerin Jacob
From: Jerin Jacob <jerinj@marvell.com>
v2:
Addressed the following review comments from Mattias Rönnblom:
1) Changed
from:
typedef uint64_t* rte_trace_t;
to
typedef uint64_t rte_trace_t;
Initially thought to make the handle as
struct rte_trace {
uint64_t val;
}
but changed to uint64_t for the following reasons
a) It is opaque to the application and it will fix the compile-time type
check as well.
b) The handle has an index that will point to an internal slow-path
structure so no ABI change required in the future.
c) RTE_TRACE_POINT_DEFINE need to expose trace object. So it is better
to keep as uint64_t and avoid one more indirection for no use.
2)
Changed:
from:
enum rte_trace_mode_e {
to:
enum rte_trace_mode {
3) removed [out] "found" param from rte_trace_pattern() and
rte_trace_regexp()
4) Changed rte_trace_from_name to rte_trace_by_name
5) rte_trace_is_dp_enabled() return bool now
6) in __rte_trace_point_register() the argument fn change to register_fn
7) removed !! from rte_trace_is_enabled()
8) Remove uninitialized "rc warning" from rte_trace_pattern() and
rte_trace_regexp()
9) fixup bool return type for trace_entry_compare()
10) fixup calloc casting in trace_mkdir()
11) check fclose() return in trace_meta_save() and trace_mem_save()
12) rte_trace_ctf_* macro cleanup
13) added release notes
14) fix build issues reported by CI
http://mails.dpdk.org/archives/test-report/2020-March/121235.html
This patch set contains
~~~~~~~~~~~~~~~~~~~~~~~~
# The native implementation of common trace format(CTF)[1] based tracer
# Public API to create the trace points.
# Add tracepoints to eal, ethdev, mempool, eventdev and cryptodev
library for tracing support
# A unit test case
# Performance test case to measure the trace overhead. (See eal/trace:
# add trace performance test cases, patch)
# Programmers guide for Trace support(See doc: add trace library guide,
# patch)
# Tested OS:
~~~~~~~~~~~
- Linux
- FreeBSD
# Tested open source CTF trace viewers
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- Babeltrace
- Tracecompass
# Trace overhead comparison with LTTng
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
trace overhead data on x86:[2]
# 236 cycles with LTTng(>100ns)
# 18 cycles(7ns) with Native DPDK CTF emitter.(See eal/trace: add trace
# performance test cases patch)
trace overhead data on arm64:
# 312 cycles to 1100 cycles with LTTng based on the class of arm64
# CPU.
# 11 cycles to 13 cycles with Native DPDK CTF emitter based on the
class of arm64 CPU.
18 cycles(on x86) vs 11 cycles(on arm64) is due to rdtsc() overhead in
x86. It seems rdtsc takes around 15cycles in x86.
Items that needs to be sort it out
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# Makefile and meson.build are updated to allow experimental APIs.
As multiple EXPERIMENTAL symbols, exported by trace library, are
used in various drivers, lib, app and examples. So to fix compilation
warning/error, Makefile and meson.build are updated for all required
components to support EXPERIMENTAL APIs.
It results same code changes at multiple components as well as
increases source code line changes in patchset too.
Suggestions are welcome to resolve this issue with lesser code changes.
More details:
~~~~~~~~~~~~~
# The Native DPDK CTF trace support does not have any dependency on
third-party library.
The generated output file is compatible with LTTng as both are using
CTF trace format.
The performance gain comes from:
1) exploit dpdk worker thread usage model to avoid atomics and use per
core variables
2) use hugepage,
3) avoid a lot function pointers in fast-path etc
4) avoid unaligned store for arm64 etc
Features:
~~~~~~~~~
- APIs and Features are similar to rte_log dynamic framework
API(expect log prints on stdout vs it dumps on trace file)
- No specific limit on the events. A string-based event like rte_log
for pattern matching
- Dynamic enable/disable support.
- Instructmention overhead is ~1 cycle. i.e cost of adding the code
wth out using trace feature.
- Timestamp support for all the events using DPDK rte_rtdsc
- No dependency on another library. Clean room native implementation of
CTF.
Functional test case:
a) echo "trace_autotest" | sudo ./build/app/test/dpdk-test -c 0x3
--trace-level=8
The above command emits the following trace events
<code>
uint8_t i;
rte_trace_lib_eal_generic_void();
rte_trace_lib_eal_generic_u64(0x10000000000000);
rte_trace_lib_eal_generic_u32(0x10000000);
rte_trace_lib_eal_generic_u16(0xffee);
rte_trace_lib_eal_generic_u8(0xc);
rte_trace_lib_eal_generic_i64(-1234);
rte_trace_lib_eal_generic_i32(-1234567);
rte_trace_lib_eal_generic_i16(12);
rte_trace_lib_eal_generic_i8(-3);
rte_trace_lib_eal_generic_string("my string");
rte_trace_lib_eal_generic_function(__func__);
for (i = 0; i < 128; i++)
rte_trace_lib_eal_generic_u8(i);
</code>
Install babeltrace package in Linux and point the generated trace file
to babel trace. By default trace file created under
<user>/dpdk-traces/time_stamp/
example:
# babeltrace /root/dpdk-traces/rte-2020-02-15-PM-02-56-51 | more
[13:27:36.138468807] (+?.?????????) lib.eal.generic.void: { cpu_id =
0, name = "dpdk-test" }, { }
[13:27:36.138468851] (+0.000000044) lib.eal.generic.u64: { cpu_id = 0,
name = "dpdk-test" }, { in = 4503599627370496 }
[13:27:36.138468860] (+0.000000009) lib.eal.generic.u32: { cpu_id = 0,
name = "dpdk-test" }, { in = 268435456 }
[13:27:36.138468934] (+0.000000074) lib.eal.generic.u16: { cpu_id = 0,
name = "dpdk-test" }, { in = 65518 }
[13:27:36.138468949] (+0.000000015) lib.eal.generic.u8: { cpu_id = 0,
name = "dpdk-test" }, { in = 12 }
[13:27:36.138468956] (+0.000000007) lib.eal.generic.i64: { cpu_id = 0,
name = "dpdk-test" }, { in = -1234 }
[13:27:36.138468963] (+0.000000007) lib.eal.generic.i32: { cpu_id = 0,
name = "dpdk-test" }, { in = -1234567 }
[13:27:36.138469024] (+0.000000061) lib.eal.generic.i16: { cpu_id = 0,
name = "dpdk-test" }, { in = 12 }
[13:27:36.138469044] (+0.000000020) lib.eal.generic.i8: { cpu_id = 0,
name = "dpdk-test" }, { in = -3 }
[13:27:36.138469051] (+0.000000007) lib.eal.generic.string: { cpu_id =
0, name = "dpdk-test" }, { str = "my string" }
[13:27:36.138469203] (+0.000000152) lib.eal.generic.func: { cpu_id =
0, name = "dpdk-test" }, { func = "test_trace_points" }
[13:27:36.138469239] (+0.000000036) lib.eal.generic.u8: { cpu_id = 0,
name = "dpdk-test" }, { in = 0 }
[13:27:36.138469246] (+0.000000007) lib.eal.generic.u8: { cpu_id = 0,
name = "dpdk-test" }, { in = 1 }
[13:27:36.138469252] (+0.000000006) lib.eal.generic.u8: { cpu_id = 0,
name = "dpdk-test" }, { in = 2 }
[13:27:36.138469262] (+0.000000010) lib.eal.generic.u8: { cpu_id = 0,
name = "dpdk-test" }, { in = 3 }
[13:27:36.138469269] (+0.000000007) lib.eal.generic.u8: { cpu_id = 0,
name = "dpdk-test" }, { in = 4 }
[13:27:36.138469276] (+0.000000007) lib.eal.generic.u8: { cpu_id = 0,
name = "dpdk-test" }, { in = 5 }
# There is a GUI based trace viewer available in Windows, Linux and
# Mac.
It is called as tracecompass.(https://www.eclipse.org/tracecompass/)
The example screenshot and Histogram of above DPDK trace using
Tracecompass.
https://github.com/jerinjacobk/share/blob/master/dpdk_trace.JPG
File walk through:
~~~~~~~~~~~~~~~~~~
lib/librte_eal/common/include/rte_trace.h - Public API for Trace
provider and Trace control
lib/librte_eal/common/eal_common_trace.c - main trace implementation
lib/librte_eal/common/eal_common_trace_ctf.c - CTF metadata spec
implementation
lib/librte_eal/common/eal_common_trace_utils.c - command line utils
and filesystem operations.
lib/librte_eal/common/eal_common_trace_points.c - trace points for EAL
library
lib/librte_eal/common/include/rte_trace_eal.h - EAL tracepoint public
API.
lib/librte_eal/common/eal_trace.h - Private trace header file.
[1] https://diamon.org/ctf/
[2] The above test is ported to LTTng for finding the LTTng trace
overhead. It available at
https://github.com/jerinjacobk/lttng-overhead
https://github.com/jerinjacobk/lttng-overhead/blob/master/README
Jerin Jacob (21):
eal: introduce API for getting thread name
eal: define the public API for trace support
eal/trace: implement trace register API
eal/trace: implement trace operation APIs
eal/trace: add internal trace init and fini interface
eal/trace: get bootup timestamp for trace
eal/trace: create CTF TDSL metadata in memory
eal/trace: implement trace memory allocation
eal/trace: implement debug dump function
eal/trace: implement trace save
eal/trace: implement registration payload
eal/trace: implement provider payload
eal/trace: hook internal trace APIs to Linux
eal/trace: hook internal trace APIs to FreeBSD
eal/trace: add generic tracepoints
eal/trace: add alarm tracepoints
eal/trace: add memory tracepoints
eal/trace: add memzone tracepoints
eal/trace: add thread tracepoints
eal/trace: add interrupt tracepoints
eal/trace: add trace performance test cases
Sunil Kumar Kori (11):
eal/trace: handle CTF keyword collision
eal/trace: add trace level configuration parameter
eal/trace: add trace dir configuration parameter
eal/trace: add trace bufsize configuration parameter
eal/trace: add trace mode configuration parameter
eal/trace: add unit test cases
ethdev: add tracepoints
eventdev: add tracepoints
cryptodev: add tracepoints
mempool: add tracepoints
doc: add trace library guide
MAINTAINERS | 15 +
app/pdump/Makefile | 1 +
app/pdump/meson.build | 1 +
app/proc-info/Makefile | 1 +
app/proc-info/meson.build | 1 +
app/test-acl/Makefile | 1 +
app/test-acl/meson.build | 1 +
app/test-cmdline/Makefile | 1 +
app/test-cmdline/meson.build | 1 +
app/test-eventdev/Makefile | 1 +
app/test-eventdev/meson.build | 1 +
app/test-pipeline/Makefile | 1 +
app/test-pipeline/meson.build | 1 +
app/test/Makefile | 4 +-
app/test/meson.build | 3 +
app/test/test_trace.c | 618 ++++++++++++++++++
app/test/test_trace.h | 52 ++
app/test/test_trace_perf.c | 179 +++++
app/test/test_trace_register.c | 46 ++
config/common_base | 1 +
config/rte_config.h | 1 +
doc/api/doxy-api-index.md | 3 +-
doc/guides/linux_gsg/eal_args.include.rst | 55 ++
doc/guides/prog_guide/index.rst | 1 +
doc/guides/prog_guide/trace_lib.rst | 265 ++++++++
doc/guides/rel_notes/release_20_05.rst | 9 +
drivers/common/cpt/Makefile | 1 +
drivers/crypto/ccp/Makefile | 1 +
drivers/crypto/ccp/meson.build | 2 +
drivers/crypto/mvsam/Makefile | 1 +
drivers/crypto/mvsam/meson.build | 1 +
drivers/crypto/null/Makefile | 1 +
drivers/crypto/null/meson.build | 1 +
drivers/crypto/scheduler/Makefile | 1 +
drivers/crypto/scheduler/meson.build | 1 +
drivers/crypto/virtio/Makefile | 1 +
drivers/crypto/virtio/meson.build | 1 +
drivers/event/octeontx/Makefile | 1 +
drivers/event/octeontx/meson.build | 1 +
drivers/event/skeleton/Makefile | 1 +
drivers/event/skeleton/meson.build | 1 +
drivers/event/sw/Makefile | 1 +
drivers/event/sw/meson.build | 1 +
drivers/mempool/ring/Makefile | 1 +
drivers/mempool/ring/meson.build | 1 +
drivers/net/af_packet/Makefile | 1 +
drivers/net/af_packet/meson.build | 1 +
drivers/net/af_xdp/Makefile | 1 +
drivers/net/af_xdp/meson.build | 1 +
drivers/net/ark/Makefile | 1 +
drivers/net/ark/meson.build | 1 +
drivers/net/bnxt/Makefile | 1 +
drivers/net/bnxt/meson.build | 1 +
drivers/net/cxgbe/Makefile | 1 +
drivers/net/cxgbe/meson.build | 1 +
drivers/net/enetc/Makefile | 1 +
drivers/net/enetc/meson.build | 1 +
drivers/net/hinic/Makefile | 1 +
drivers/net/hinic/base/meson.build | 3 +-
drivers/net/hinic/meson.build | 1 +
drivers/net/ionic/ionic_dev.c | 1 +
drivers/net/ionic/ionic_mac_api.c | 1 +
drivers/net/ionic/ionic_main.c | 1 +
drivers/net/kni/Makefile | 1 +
drivers/net/kni/meson.build | 1 +
drivers/net/liquidio/Makefile | 1 +
drivers/net/liquidio/meson.build | 1 +
drivers/net/mvneta/Makefile | 1 +
drivers/net/mvneta/meson.build | 1 +
drivers/net/mvpp2/Makefile | 1 +
drivers/net/mvpp2/meson.build | 1 +
drivers/net/nfb/Makefile | 1 +
drivers/net/nfb/meson.build | 1 +
drivers/net/null/Makefile | 1 +
drivers/net/null/meson.build | 1 +
drivers/net/octeontx/Makefile | 1 +
drivers/net/octeontx2/Makefile | 1 +
drivers/net/octeontx2/meson.build | 1 +
drivers/net/pcap/Makefile | 1 +
drivers/net/pcap/meson.build | 1 +
drivers/net/ring/Makefile | 1 +
drivers/net/ring/meson.build | 1 +
drivers/net/szedata2/Makefile | 1 +
drivers/net/szedata2/meson.build | 1 +
drivers/net/thunderx/base/meson.build | 1 +
drivers/net/vhost/Makefile | 1 +
drivers/net/vhost/meson.build | 1 +
drivers/raw/ioat/Makefile | 1 +
drivers/raw/ioat/meson.build | 1 +
drivers/raw/octeontx2_dma/Makefile | 1 +
drivers/raw/octeontx2_dma/meson.build | 1 +
drivers/raw/octeontx2_ep/Makefile | 1 +
drivers/raw/octeontx2_ep/meson.build | 1 +
drivers/raw/skeleton/Makefile | 1 +
drivers/raw/skeleton/meson.build | 1 +
examples/cmdline/Makefile | 1 +
examples/cmdline/meson.build | 1 +
examples/distributor/Makefile | 1 +
examples/distributor/meson.build | 1 +
examples/ethtool/ethtool-app/Makefile | 1 +
examples/eventdev_pipeline/meson.build | 1 +
examples/flow_filtering/Makefile | 1 +
examples/flow_filtering/meson.build | 1 +
examples/helloworld/Makefile | 1 +
examples/helloworld/meson.build | 1 +
examples/ioat/Makefile | 1 +
examples/ioat/meson.build | 1 +
examples/ip_fragmentation/Makefile | 2 +
examples/ip_fragmentation/meson.build | 1 +
examples/ip_reassembly/Makefile | 1 +
examples/ip_reassembly/meson.build | 1 +
examples/ipv4_multicast/Makefile | 1 +
examples/ipv4_multicast/meson.build | 1 +
examples/l2fwd-cat/Makefile | 1 +
examples/l2fwd-cat/meson.build | 1 +
examples/l2fwd-event/Makefile | 1 +
examples/l2fwd-event/meson.build | 1 +
examples/l2fwd-jobstats/Makefile | 1 +
examples/l2fwd-jobstats/meson.build | 1 +
examples/l2fwd-keepalive/Makefile | 1 +
examples/l2fwd-keepalive/ka-agent/Makefile | 1 +
examples/l2fwd-keepalive/meson.build | 1 +
examples/l3fwd-acl/Makefile | 1 +
examples/l3fwd-acl/meson.build | 1 +
examples/l3fwd/Makefile | 1 +
examples/l3fwd/meson.build | 1 +
examples/link_status_interrupt/Makefile | 1 +
examples/link_status_interrupt/meson.build | 1 +
.../client_server_mp/mp_client/Makefile | 1 +
.../client_server_mp/mp_client/meson.build | 1 +
.../client_server_mp/mp_server/meson.build | 1 +
examples/multi_process/hotplug_mp/Makefile | 1 +
examples/multi_process/hotplug_mp/meson.build | 1 +
examples/multi_process/simple_mp/Makefile | 1 +
examples/multi_process/simple_mp/meson.build | 1 +
examples/multi_process/symmetric_mp/Makefile | 1 +
.../multi_process/symmetric_mp/meson.build | 1 +
examples/ntb/Makefile | 1 +
examples/ntb/meson.build | 1 +
examples/packet_ordering/Makefile | 1 +
examples/packet_ordering/meson.build | 1 +
.../performance-thread/l3fwd-thread/Makefile | 1 +
.../l3fwd-thread/meson.build | 1 +
.../performance-thread/pthread_shim/Makefile | 1 +
.../pthread_shim/meson.build | 1 +
examples/ptpclient/Makefile | 1 +
examples/ptpclient/meson.build | 1 +
examples/qos_meter/Makefile | 1 +
examples/qos_meter/meson.build | 1 +
examples/qos_sched/Makefile | 1 +
examples/qos_sched/meson.build | 1 +
examples/server_node_efd/node/Makefile | 1 +
examples/server_node_efd/node/meson.build | 1 +
examples/server_node_efd/server/Makefile | 1 +
examples/server_node_efd/server/meson.build | 1 +
examples/service_cores/Makefile | 1 +
examples/service_cores/meson.build | 1 +
examples/skeleton/Makefile | 1 +
examples/skeleton/meson.build | 1 +
examples/timer/Makefile | 1 +
examples/timer/meson.build | 1 +
examples/vm_power_manager/Makefile | 1 +
examples/vm_power_manager/meson.build | 1 +
examples/vmdq/Makefile | 1 +
examples/vmdq/meson.build | 1 +
examples/vmdq_dcb/Makefile | 1 +
examples/vmdq_dcb/meson.build | 1 +
lib/librte_bitratestats/Makefile | 1 +
lib/librte_bitratestats/meson.build | 1 +
lib/librte_cryptodev/Makefile | 4 +-
lib/librte_cryptodev/cryptodev_trace_points.c | 70 ++
lib/librte_cryptodev/meson.build | 7 +-
lib/librte_cryptodev/rte_cryptodev.c | 18 +
lib/librte_cryptodev/rte_cryptodev.h | 6 +
.../rte_cryptodev_version.map | 18 +
lib/librte_cryptodev/rte_trace_cryptodev.h | 133 ++++
lib/librte_cryptodev/rte_trace_cryptodev_fp.h | 34 +
lib/librte_distributor/Makefile | 1 +
lib/librte_distributor/meson.build | 1 +
lib/librte_eal/common/Makefile | 1 +
lib/librte_eal/common/eal_common_log.c | 9 +-
lib/librte_eal/common/eal_common_memzone.c | 9 +
lib/librte_eal/common/eal_common_options.c | 68 +-
lib/librte_eal/common/eal_common_thread.c | 3 +-
lib/librte_eal/common/eal_common_trace.c | 610 +++++++++++++++++
lib/librte_eal/common/eal_common_trace_ctf.c | 488 ++++++++++++++
.../common/eal_common_trace_points.c | 115 ++++
.../common/eal_common_trace_utils.c | 523 +++++++++++++++
lib/librte_eal/common/eal_options.h | 8 +
lib/librte_eal/common/eal_private.h | 11 +
lib/librte_eal/common/eal_trace.h | 122 ++++
lib/librte_eal/common/include/rte_lcore.h | 17 +
lib/librte_eal/common/include/rte_trace.h | 584 +++++++++++++++++
lib/librte_eal/common/include/rte_trace_eal.h | 247 +++++++
.../common/include/rte_trace_provider.h | 137 ++++
.../common/include/rte_trace_register.h | 53 ++
lib/librte_eal/common/meson.build | 8 +
lib/librte_eal/common/rte_malloc.c | 60 +-
lib/librte_eal/freebsd/eal/Makefile | 4 +
lib/librte_eal/freebsd/eal/eal.c | 10 +
lib/librte_eal/freebsd/eal/eal_alarm.c | 3 +
lib/librte_eal/freebsd/eal/eal_interrupts.c | 52 +-
lib/librte_eal/freebsd/eal/eal_thread.c | 21 +-
lib/librte_eal/linux/eal/Makefile | 4 +
lib/librte_eal/linux/eal/eal.c | 9 +
lib/librte_eal/linux/eal/eal_alarm.c | 4 +
lib/librte_eal/linux/eal/eal_interrupts.c | 84 ++-
lib/librte_eal/linux/eal/eal_thread.c | 27 +-
lib/librte_eal/rte_eal_version.map | 61 ++
lib/librte_ethdev/Makefile | 3 +
lib/librte_ethdev/ethdev_trace_points.c | 43 ++
lib/librte_ethdev/meson.build | 5 +-
lib/librte_ethdev/rte_ethdev.c | 12 +
lib/librte_ethdev/rte_ethdev.h | 5 +
lib/librte_ethdev/rte_ethdev_version.map | 10 +
lib/librte_ethdev/rte_trace_ethdev.h | 90 +++
lib/librte_ethdev/rte_trace_ethdev_fp.h | 40 ++
lib/librte_eventdev/Makefile | 3 +
lib/librte_eventdev/eventdev_trace_points.c | 173 +++++
lib/librte_eventdev/meson.build | 3 +
.../rte_event_crypto_adapter.c | 10 +
.../rte_event_eth_rx_adapter.c | 11 +
.../rte_event_eth_tx_adapter.c | 13 +-
.../rte_event_eth_tx_adapter.h | 2 +
lib/librte_eventdev/rte_event_timer_adapter.c | 8 +-
lib/librte_eventdev/rte_event_timer_adapter.h | 8 +
lib/librte_eventdev/rte_eventdev.c | 9 +
lib/librte_eventdev/rte_eventdev.h | 5 +-
lib/librte_eventdev/rte_eventdev_version.map | 42 ++
lib/librte_eventdev/rte_trace_eventdev.h | 278 ++++++++
lib/librte_eventdev/rte_trace_eventdev_fp.h | 75 +++
lib/librte_gro/Makefile | 1 +
lib/librte_gro/meson.build | 1 +
lib/librte_gso/Makefile | 1 +
lib/librte_gso/meson.build | 1 +
lib/librte_ip_frag/Makefile | 1 +
lib/librte_ip_frag/meson.build | 1 +
lib/librte_kni/Makefile | 1 +
lib/librte_kni/meson.build | 1 +
lib/librte_latencystats/Makefile | 1 +
lib/librte_latencystats/meson.build | 1 +
lib/librte_mempool/Makefile | 3 +
lib/librte_mempool/mempool_trace_points.c | 108 +++
lib/librte_mempool/meson.build | 7 +-
lib/librte_mempool/rte_mempool.c | 16 +
lib/librte_mempool/rte_mempool.h | 13 +
lib/librte_mempool/rte_mempool_ops.c | 7 +
lib/librte_mempool/rte_mempool_version.map | 26 +
lib/librte_mempool/rte_trace_mempool.h | 148 +++++
lib/librte_mempool/rte_trace_mempool_fp.h | 102 +++
lib/librte_port/Makefile | 1 +
lib/librte_port/meson.build | 1 +
lib/librte_reorder/Makefile | 1 +
lib/librte_reorder/meson.build | 1 +
lib/librte_sched/Makefile | 1 +
lib/librte_sched/meson.build | 1 +
lib/librte_security/Makefile | 1 +
lib/librte_security/meson.build | 1 +
lib/librte_table/Makefile | 1 +
lib/librte_table/meson.build | 1 +
260 files changed, 6271 insertions(+), 82 deletions(-)
create mode 100644 app/test/test_trace.c
create mode 100644 app/test/test_trace.h
create mode 100644 app/test/test_trace_perf.c
create mode 100644 app/test/test_trace_register.c
create mode 100644 doc/guides/prog_guide/trace_lib.rst
create mode 100644 lib/librte_cryptodev/cryptodev_trace_points.c
create mode 100644 lib/librte_cryptodev/rte_trace_cryptodev.h
create mode 100644 lib/librte_cryptodev/rte_trace_cryptodev_fp.h
create mode 100644 lib/librte_eal/common/eal_common_trace.c
create mode 100644 lib/librte_eal/common/eal_common_trace_ctf.c
create mode 100644 lib/librte_eal/common/eal_common_trace_points.c
create mode 100644 lib/librte_eal/common/eal_common_trace_utils.c
create mode 100644 lib/librte_eal/common/eal_trace.h
create mode 100644 lib/librte_eal/common/include/rte_trace.h
create mode 100644 lib/librte_eal/common/include/rte_trace_eal.h
create mode 100644 lib/librte_eal/common/include/rte_trace_provider.h
create mode 100644 lib/librte_eal/common/include/rte_trace_register.h
create mode 100644 lib/librte_ethdev/ethdev_trace_points.c
create mode 100644 lib/librte_ethdev/rte_trace_ethdev.h
create mode 100644 lib/librte_ethdev/rte_trace_ethdev_fp.h
create mode 100644 lib/librte_eventdev/eventdev_trace_points.c
create mode 100644 lib/librte_eventdev/rte_trace_eventdev.h
create mode 100644 lib/librte_eventdev/rte_trace_eventdev_fp.h
create mode 100644 lib/librte_mempool/mempool_trace_points.c
create mode 100644 lib/librte_mempool/rte_trace_mempool.h
create mode 100644 lib/librte_mempool/rte_trace_mempool_fp.h
--
2.25.1
^ permalink raw reply [relevance 1%]
* Re: [dpdk-dev] [RFC] ring: make ring implementation non-inlined
2020-03-25 21:09 4% ` Jerin Jacob
@ 2020-03-26 0:28 0% ` Ananyev, Konstantin
2020-03-26 8:04 4% ` Morten Brørup
1 sibling, 0 replies; 200+ results
From: Ananyev, Konstantin @ 2020-03-26 0:28 UTC (permalink / raw)
To: Jerin Jacob
Cc: dpdk-dev, Olivier Matz, Honnappa Nagarahalli, Jerin Jacob,
David Christensen, Stephen Hemminger
Hi Jerin,
>
> On Fri, Mar 20, 2020 at 10:11 PM Konstantin Ananyev
> <konstantin.ananyev@intel.com> wrote:
> >
> > As was discussed here:
> > http://mails.dpdk.org/archives/dev/2020-February/158586.html
> > this RFC aimed to hide ring internals into .c and make all
> > ring functions non-inlined. In theory that might help to
> > maintain ABI stability in future.
> > This is just a POC to measure the impact of proposed idea,
> > proper implementation would definetly need some extra effort.
> > On IA box (SKX) ring_perf_autotest shows ~20-30 cycles extra for
> > enqueue+dequeue pair. On some more realistic code, I suspect
> > the impact it might be a bit higher.
> > For MP/MC bulk transfers degradation seems quite small,
> > though for SP/SC and/or small transfers it is more then noticable
> > (see exact numbers below).
> > From my perspective we'd probably keep it inlined for now
> > to avoid any non-anticipated perfomance degradations.
> > Though intersted to see perf results and opinions from
> > other interested parties.
>
> +1
>
> My reasoning is a bit different, DPDK is using in embedded boxes too
> where performance has
> more weight than ABI stuff. I think we need to focus first on slow
> path APIs ABI stuff.
>
> I spend a few cycles to apply this patch +
> http://mails.dpdk.org/archives/dev/2020-February/158586.html
> on top of the tree, there are a lot of conflicts. If I get a mergeable
> patch then I will test it on an arm64 box.
You don’t need to apply previous patch series.
They are completely unrelated.
Just apply that one (http://patches.dpdk.org/patch/66982/)
on top of dpdk.org master.
It should be applied cleanly (at least it does for me).
Konstantin
>
>
>
>
>
> >
> > Intel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz
> > ring_perf_autotest (without patch/with patch)
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [RFC 0/6] New sync modes for ring
2020-03-25 20:43 0% ` Honnappa Nagarahalli
@ 2020-03-26 1:50 3% ` Ananyev, Konstantin
0 siblings, 0 replies; 200+ results
From: Ananyev, Konstantin @ 2020-03-26 1:50 UTC (permalink / raw)
To: Honnappa Nagarahalli, dev; +Cc: olivier.matz, nd, nd
>
> <snip>
>
> > Subject: [dpdk-dev] [RFC 0/6] New sync modes for ring
> >
> > Upfront note - that RFC is not a complete patch.
> > It introduces an ABI breakage, plus it doesn't update ring_elem code properly,
> As per the current rules, these changes (in the current form) will be accepted only for 20.11 release. How do we address this for immediate
> requirements like RCU defer APIs?
I think I found a way to introduce these new modes without API/ABI breakage.
Working on v1 right now. Plan to submit it by end of that week/start of next one.
> I suggest that we move forward with my RFC (taking into consideration your feedback) to make progress on RCU APIs.
>
> > etc.
> > I plan to deal with all these things in later versions.
> > Right now I seek an initial feedback about proposed ideas.
> > Would also ask people to repeat performance tests (see below) on their
> > platforms to confirm the impact.
> >
> > More and more customers use(/try to use) DPDK based apps within
> > overcommitted systems (multiple acttive threads over same pysical cores):
> > VM, container deployments, etc.
> > One quite common problem they hit: Lock-Holder-Preemption with rte_ring.
> > LHP is quite a common problem for spin-based sync primitives (spin-locks, etc.)
> > on overcommitted systems.
> > The situation gets much worse when some sort of fair-locking technique is
> > used (ticket-lock, etc.).
> > As now not only lock-owner but also lock-waiters scheduling order matters a
> > lot.
> > This is a well-known problem for kernel within VMs:
> > http://www-archive.xenproject.org/files/xensummitboston08/LHP.pdf
> > https://www.cs.hs-rm.de/~kaiser/events/wamos2017/Slides/selcuk.pdf
> > The problem with rte_ring is that while head accusion is sort of un-fair locking,
> > waiting on tail is very similar to ticket lock schema - tail has to be updated in
> > particular order.
> > That makes current rte_ring implementation to perform really pure on some
> > overcommited scenarios.
> > While it is probably not possible to completely resolve this problem in
> > userspace only (without some kernel communication/intervention), removing
> > fairness in tail update can mitigate it significantly.
> > So this RFC proposes two new optional ring synchronization modes:
> > 1) Head/Tail Sync (HTS) mode
> > In that mode enqueue/dequeue operation is fully serialized:
> > only one thread at a time is allowed to perform given op.
> > As another enhancement provide ability to split enqueue/dequeue
> > operation into two phases:
> > - enqueue/dequeue start
> > - enqueue/dequeue finish
> > That allows user to inspect objects in the ring without removing
> > them from it (aka MT safe peek).
> IMO, this will not address the problem described above.
It does, please see the results produced by ring_stress_*autotest below.
Let say for test-case: 8thread@2core(--lcores='6,(10-13)@7,(20-23)@8' it shows:
avg number of cycles per object for enqueue /dequeue:
MP/MC: 280314.32
HTS: 294.72
RTS: 318.79
Customer who tried it reported similar level of improvement.
Actually if you have time - would be very interesting to see what numbers will be on ARM boxes.
To reproduce, just:
$cat ring_tests_u4
ring_stress_autotest
ring_stress_hts_autotest
ring_stress_rts_autotest
/app/test/dpdk-test --lcores='6,(10-13)@7,(20-23)@8' -n 4 < ring_tests_u4 2>&1 | tee res1
Then look at the ' AGGREGATE' stats.
Right now it is a bit too verbose, so probably the easiest thing to extract same numbers quickly:
grep 'cycles/obj' res1 | grep 'cycles/obj' | cat -n | awk '{if ($(1)%9==0) print $(NF);}'
280314.32
1057833.55
294.72
480.10
318.79
461.52
First 2 numbers will be for MP/MC, next 2 for HTS, last 2 for RTS.
> For ex: when a producer updates the head and gets scheduled out, other producers
> have to spin.
Sure, as I wrote in original cover letter:
" While it is probably not possible to completely resolve this problem in
userspace only (without some kernel communication/intervention),
removing fairness in tail update can mitigate it significantly."
Results from the ring_stress_*_autotest confirm that.
> The problem is probably worse as with non-HTS case moving of the head and copying of the ring elements can happen in
> parallel between the producers (similarly for consumers).
Yes as we serialize the ring, we remove possibility of simultaneous copy.
That's why for 'normal' cases (one thread per core) original MP/MC is usually faster.
Though on overcommitted scenarios current MP/MC performance degrades dramatically.
The main problem with current MP/MC implementation is in that tail update
have to be done in strict order (sort of fair locking scheme).
Which means that we have much-much worse LHP manifestation,
then when we use unfair schemes.
With serialized ring (HTS) we remove that ordering completely
(same idea as switch from fair to unfair locking for PV spin-locks).
> IMO, HTS should not be a configurable flag.
Why?
> In RCU requirement, a MP enqueue and HTS dequeue are required.
This is supported, user can specify different modes for consumer and producer:
(0 | RING_F_MC_HTS_DEQ).
Then it is up to the user either to call generic rte_ring_enqueue/rte_ring_dequeue,
or specify mode manually by function name:
rte_ring_mp_enqueue_bulk/ rte_ring_hts_dequeue_bulk.
>
> > 2) Relaxed Tail Sync (RTS)
> > The main difference from original MP/MC algorithm is that tail value is
> > increased not by every thread that finished enqueue/dequeue, but only by the
> > last one.
> > That allows threads to avoid spinning on ring tail value, leaving actual tail value
> > change to the last thread in the update queue.
> This can be a configurable flag on the ring.
> I am not sure how this solves the problem you have stated above completely. Updating the count from all intermediate threads is still
> required to update the value of the head. But yes, it reduces the severity of the problem by not enforcing the order in which the tail is
> updated.
As I said above, main source of slowdown here -
that we have to update tail in particular order.
So the main objective (same as for HTS) is to remove
that ordering.
> I also think it introduces the problem on the other side of the ring because the tail is not updated soon enough (the other side has to wait
> longer for the elements to become available).
Yes, producer/consumer starvation.
That's why we need max allowed Head-Tail-Distance (htd_max) -
to limit how far head can go away from tail.
> It also introduces another configuration parameter (HTD_MAX_DEF) which they have to deal
> with.
If user doesn't provide any value, it will be set by default to ring.capacity / 8.
From my measurements works quite well.
Though there possibility for the user to set another value, if needed.
> Users have to still implement the current hypervisor related solutions.
Didn't get what you trying to say with that phrase.
> IMO, we should run the benchmark for this on an over committed setup to understand the benefits.
That's why I created ring_stress_*autotest test-cases and collected numbers provided below.
I suppose they clearly show the problem on overcommitted scenarios,
and how RTS/HTS improve that situation.
Would appreciate if you repeat these tests on your machines.
>
> >
> > Test results on IA (see below) show significant improvements for average
> > enqueue/dequeue op times on overcommitted systems.
> > For 'classic' DPDK deployments (one thread per core) original MP/MC
> > algorithm still shows best numbers, though for 64-bit target RTS numbers are
> > not that far away.
> > Numbers were produced by ring_stress_*autotest (first patch in these series).
> >
> > X86_64 @ Intel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz
> > DEQ+ENQ average cycles/obj
> >
> > MP/MC HTS RTS
> > 1thread@1core(--lcores=6-7) 8.00 8.15 8.99
> > 2thread@2core(--lcores=6-8) 19.14 19.61 20.35
> > 4thread@4core(--lcores=6-10) 29.43 29.79 31.82
> > 8thread@8core(--lcores=6-14) 110.59 192.81 119.50
> > 16thread@16core(--lcores=6-22) 461.03 813.12 495.59
> > 32thread/@32core(--lcores='6-22,55-70') 982.90 1972.38 1160.51
> >
> > 2thread@1core(--lcores='6,(10-11)@7' 20140.50 23.58 25.14
> > 4thread@2core(--lcores='6,(10-11)@7,(20-21)@8' 153680.60 76.88 80.05
> > 8thread@2core(--lcores='6,(10-13)@7,(20-23)@8' 280314.32 294.72 318.79
> > 16thread@2core(--lcores='6,(10-17)@7,(20-27)@8' 643176.59 1144.02
> > 1175.14 32thread@2core(--lcores='6,(10-25)@7,(30-45)@8' 4264238.80
> > 4627.48 4892.68
> >
> > 8thread@2core(--lcores='6,(10-17)@(7,8))' 321085.98 298.59 307.47
> > 16thread@4core(--lcores='6,(20-35)@(7-10))' 1900705.61 575.35 678.29
> > 32thread@4core(--lcores='6,(20-51)@(7-10))' 5510445.85 2164.36 2714.12
> >
> > i686 @ Intel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz
> > DEQ+ENQ average cycles/obj
> >
> > MP/MC HTS RTS
> > 1thread@1core(--lcores=6-7) 7.85 12.13 11.31
> > 2thread@2core(--lcores=6-8) 17.89 24.52 21.86
> > 8thread@8core(--lcores=6-14) 32.58 354.20 54.58
> > 32thread/@32core(--lcores='6-22,55-70') 813.77 6072.41 2169.91
> >
> > 2thread@1core(--lcores='6,(10-11)@7' 16095.00 36.06 34.74
> > 8thread@2core(--lcores='6,(10-13)@7,(20-23)@8' 1140354.54 346.61 361.57
> > 16thread@2core(--lcores='6,(10-17)@7,(20-27)@8' 1920417.86 1314.90
> > 1416.65
> >
> > 8thread@2core(--lcores='6,(10-17)@(7,8))' 594358.61 332.70 357.74
> > 32thread@4core(--lcores='6,(20-51)@(7-10))' 5319896.86 2836.44 3028.87
> >
> > Konstantin Ananyev (6):
> > test/ring: add contention stress test
> > ring: rework ring layout to allow new sync schemes
> > ring: introduce RTS ring mode
> > test/ring: add contention stress test for RTS ring
> > ring: introduce HTS ring mode
> > test/ring: add contention stress test for HTS ring
> >
> > app/test/Makefile | 3 +
> > app/test/meson.build | 3 +
> > app/test/test_pdump.c | 6 +-
> > app/test/test_ring_hts_stress.c | 28 ++
> > app/test/test_ring_rts_stress.c | 28 ++
> > app/test/test_ring_stress.c | 27 ++
> > app/test/test_ring_stress.h | 477 +++++++++++++++++++
> > lib/librte_pdump/rte_pdump.c | 2 +-
> > lib/librte_port/rte_port_ring.c | 12 +-
> > lib/librte_ring/Makefile | 4 +-
> > lib/librte_ring/meson.build | 4 +-
> > lib/librte_ring/rte_ring.c | 84 +++-
> > lib/librte_ring/rte_ring.h | 619 +++++++++++++++++++++++--
> > lib/librte_ring/rte_ring_elem.h | 8 +-
> > lib/librte_ring/rte_ring_hts_generic.h | 228 +++++++++
> > lib/librte_ring/rte_ring_rts_generic.h | 240 ++++++++++
> > 16 files changed, 1721 insertions(+), 52 deletions(-) create mode 100644
> > app/test/test_ring_hts_stress.c create mode 100644
> > app/test/test_ring_rts_stress.c create mode 100644
> > app/test/test_ring_stress.c create mode 100644 app/test/test_ring_stress.h
> > create mode 100644 lib/librte_ring/rte_ring_hts_generic.h
> > create mode 100644 lib/librte_ring/rte_ring_rts_generic.h
> >
> > --
> > 2.17.1
^ permalink raw reply [relevance 3%]
* [dpdk-dev] [PATCH v4 2/7] net/ice: add the DCF hardware initialization
@ 2020-03-26 3:03 1% ` Haiyue Wang
0 siblings, 0 replies; 200+ results
From: Haiyue Wang @ 2020-03-26 3:03 UTC (permalink / raw)
To: dev, xiaolong.ye, qi.z.zhang, qiming.yang, beilei.xing
Cc: wei.zhao1, Haiyue Wang
Introduce the DCF (Device Config Function) feature in the ice PMD, it
works as a standalone PMD which doesn't handle the packet Rx/Tx related
things. Its hardware entity is the VF.
Add the basic DCF hardware initialization, this is specified by devarg
'cap=dcf'.
Signed-off-by: Haiyue Wang <haiyue.wang@intel.com>
---
doc/guides/nics/ice.rst | 47 +++
doc/guides/nics/img/ice_dcf.png | Bin 0 -> 39168 bytes
doc/guides/rel_notes/release_20_05.rst | 5 +
drivers/common/Makefile | 1 +
drivers/net/ice/Makefile | 5 +
drivers/net/ice/ice_dcf.c | 474 +++++++++++++++++++++++++
drivers/net/ice/ice_dcf.h | 52 +++
drivers/net/ice/ice_dcf_ethdev.c | 317 +++++++++++++++++
drivers/net/ice/ice_dcf_ethdev.h | 24 ++
drivers/net/ice/meson.build | 7 +-
mk/rte.app.mk | 1 +
11 files changed, 931 insertions(+), 2 deletions(-)
create mode 100644 doc/guides/nics/img/ice_dcf.png
create mode 100644 drivers/net/ice/ice_dcf.c
create mode 100644 drivers/net/ice/ice_dcf.h
create mode 100644 drivers/net/ice/ice_dcf_ethdev.c
create mode 100644 drivers/net/ice/ice_dcf_ethdev.h
diff --git a/doc/guides/nics/ice.rst b/doc/guides/nics/ice.rst
index 8af32dabf..2639ae239 100644
--- a/doc/guides/nics/ice.rst
+++ b/doc/guides/nics/ice.rst
@@ -240,6 +240,53 @@ report a MDD event and drop the packets.
The APPs based on DPDK should avoid providing such packets.
+Device Config Function (DCF)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This section demonstrates ICE DCF PMD, which shares the core module with ICE
+PMD and iAVF PMD.
+
+A DCF (Device Config Function) PMD bounds to the device's trusted VF with ID 0,
+it can act as a sole controlling entity to exercise advance functionality (such
+as switch, ACL) for the rest VFs.
+
+The DCF PMD needs to advertise and acquire DCF capability which allows DCF to
+send AdminQ commands that it would like to execute over to the PF and receive
+responses for the same from PF.
+
+.. _figure_ice_dcf:
+
+.. figure:: img/ice_dcf.*
+
+ DCF Communication flow.
+
+#. Create the VFs::
+
+ echo 4 > /sys/bus/pci/devices/0000\:18\:00.0/sriov_numvfs
+
+#. Enable the VF0 trust on::
+
+ ip link set dev enp24s0f0 vf 0 trust on
+
+#. Bind the VF0, and run testpmd with 'cap=dcf' devarg::
+
+ testpmd -l 22-25 -n 4 -w 18:01.0,cap=dcf -- -i
+
+#. Monitor the VF2 interface network traffic::
+
+ tcpdump -e -nn -i enp24s1f2
+
+#. Create one flow to redirect the traffic to VF2 by DCF::
+
+ flow create 0 priority 0 ingress pattern eth / ipv4 src is 192.168.0.2 \
+ dst is 192.168.0.3 / end actions vf id 2 / end
+
+#. Send the packet, and it should be displayed on tcpdump::
+
+ sendp(Ether(src='3c:fd:fe:aa:bb:78', dst='00:00:00:01:02:03')/IP(src=' \
+ 192.168.0.2', dst="192.168.0.3")/TCP(flags='S')/Raw(load='XXXXXXXXXX'), \
+ iface="enp24s0f0", count=10)
+
Sample Application Notes
------------------------
diff --git a/doc/guides/nics/img/ice_dcf.png b/doc/guides/nics/img/ice_dcf.png
new file mode 100644
index 0000000000000000000000000000000000000000..540823c4a05e9ea9d558d233fea77016d1ae5df3
GIT binary patch
literal 39168
zcmZ6ybzD?k*FH`NDBT?jlF}e8B?Hn74BeolfOLbDz);d%Lw70NNOyNPNO%7Z_x(KY
z`+k3a@X<4KW}m&+Uh7)dy4F6yN(xdKsHCWHaBvvXpCpvw;1KwLKlqmjz$g7i%71`=
z;O&*A#NkSZ$ajH@7p7wJVsLO35oq^@NWk^0Z=W>n;oxu`p8w#NOsHJo;1Xh`CBz^u
zdIzZ}I(W`x-6a%La{^jclNOU&`_)rTS4-}zY76d1P4w1CVq)-OmrKC^5#ZY~-}pzu
zy$L?@i@zdO6jl2qp0>O_?^V0~==_TKu;0E)6AT_jK}Ys$GuU9xpl6NmUq4AAAS8rZ
z5<h>~BQL7h+1bg!^v{JkDk^GFZXt;WZK)wU@I4Lh>Nma-8Py=vX58uL5BwsNv0+TC
ztdd|ub|{f>I#qBR_r=Ecc2+(+F$m*+POs4gtgWq$;%5$Zz1W+)IolS4AR!?E!%MzF
zY^hc+(<6BKV$B>q*BptIQ5gaPfd*IJosgp=XT25!8zwm^L5M@!Q9uFabHF7#DiuCK
zt~qeo#^9j@Nyw!ThkWHBd%m>z=MtX)IqO&q%qUT-Q<|7KJNMhNs;a6pC&Kl7XSB)v
z3KtLW;^N}!aM9A*nl<f&LypeAH;UH%@~=<W?^;Mic5d!cle_a$qiaVX7S(%;iC{c>
zPEJmL@k|#dzb#-e#WW$n3TwM=5&|%zK#w5Rj~l++vL_;_xx2ZfV1g5o&-OTVud|dg
zY)p8a+CN(%s#Jb5B@9dFyIC8&!|9!kw<)+dnAUfkUON7RZe*>j5V@t@LFIjKzBWN}
zl6bSB?AXD@u-6mJ^$s$%>{fg;H**<Wy_z$R_0fFyxQl_NC8(r`t&yYbC$Wq!ZhWiR
zaQO>2)_CPI7Ur}ViP+8SHXAZ(qCX-8Pgw>@)`d8#Ub}%C2oieVUWG5$Z;wDPw0oh!
z(MS2V(Mt&eBO3Siyc1oDi4_JnN!?`_!F7~4=LJ<;7i4@c?GuTsp&oXOubJN~j*nRE
zr*(v=l`h&IUu)Y-G0<(qS?nZJwYZ$k+*tNcTf2SvjF$-h6dEDNR%>@OyFp@~wx8sm
zQ@KkHCa?M<DDa*q5JkVi|7}X6ooQlUmnxZR*#{T+GPIBvv|B=xJdNM>M<?%wx7c)l
zdn}q*;Wg|Gzyh^yRZNGE1Xt2+qw~^RE=E{lE|-=pha<-cjHj$U3eSXtS-y9ciA;Qt
zg#X2CIXogbtk+gV#pepYbkK~qZROc+2-nD3e6SS4d#n~;vgmMgi02}r<bB>}Fg(g-
zTDmx2YrO0kRzm_}D5S%J%DnQ0DdV>iV*NTt#-ch%|0xYIm~S%Q_BwvyfdQLqe%iUU
za<4uRAr<XM?#|y&3?~<aXTr-Hgaz#tT;9D;cK$2uv9~aYB8Zi$G+Wuxv-FN9+5PYR
zh4qI=trjcZ;=wL@A=W&9;v>?i^7Yn}O(rs$G1Q(|Z2z1sn<Yns)V;&#c$@TE2Q)Ru
z)6)`Bt!gQksJ*lfL=-F}Q=R&|&-+U>DkcK~7vFVi4u!dmmta<R;msWO$55HcDm$Mq
zYzuC++dJmp5`L=_vy^#UbXhfEraEBcG`ZP)uwPnpl~qe?xmmf+NiO2ax1Om+(c=}a
z!GBS0w-`sWU7FHRP;K|H)xhT98<o%yk#hLBt#8R=)!=YcP~pc5m_`)LD5eB~iYqs_
zj?tLy72jM<E{lrbr1en>*bVr_*W;KI;3E7^sdYFjoC{%b$NVtKVTT^7+2BYNw7pYj
zi4R5^w->W9pqU==f1#Eb;6C!>z+DXeb>TXZwZp;G`tJ%B5BYEu=0)FDN%x4fNM_s}
z&3d^+eysZRo^x@R*na2q(uHz@x%b$Uaf>KVBm(XH446M${l^6#mQ=@EaW_yr=YvCd
z+ccCShFQ?$1}?+o10_*^aev+*-pN^+qy-`EqoDYuWpSd~d>GvmJf>Q%JrCl>)WF6k
zmNOMMMp+hgneL-`A#pMMbJc=vS<;lS;}#5&f&14fUK<#DZv350Jd6LHkN$NrFqGJ8
zlLQ^XPt!kNF7<(R!K>C~vVL6sXh^GeVu@|f9?Xx@{W00`WXRZv(|^df><;rf5PNdx
zBpMO7vZw8Z5tyP#0>Vqx1X*<dv&0Kc;4ZffjQvQ21Vb2S)~HS*e}&q3vP0X5Ck>{p
zL|PgxQ2d3lia{mhxO*i0u!<pNN+=Dr*xkF}rL@3)H_B6SSLCd3U(-4g(^%Ggi&~71
zLC)Opd9psBf9m^tF-R2=i~Y+Wx~{DJ7&!b`+!vr6(LbBVOG$#4e;21&m~FS))*G-=
zPa}wqkl#vdk|f?wwVSv9(#)clt1G+|;%v%L28)#;+6VXk{7iUG+jNKVTH?KH*Mg`P
z4}U@{EuUAnkMX-Hs~^?DoM;li?~cv8zY`mcjZ7JLTz<$aU4RE4@Y}g{&JAw#$bdHS
zIsj~7y#89(M|b^nq}$-hTBu4^z2sUdUo4>v8BBVg%wyU%_MXf@bERkY-D?x<$;O<b
zpvBDRefW<DBArvMWHxmVUG9o}n*WowASDGwz0FLH(Dzku6^(`Vv9TH-Or_3WQek9&
zp{?wvKs0uy5PokudTJp-5ZP7duzj+zYp+MDA<51+GTg}}+XPcyOCY#AF=5lm{bEKD
zuYwz`>Y=}WkR(4(Y2H+%2SHKL8L4sdgQdlyUB8->Fe_=z_$V35@6;>_^Sx+A)!#{y
z%vNutx>hi*td+hI$B#YrfhqfD<l~3DlzhC=#`503zteBJi#)&=g+hzrVxg$DkD{I@
zg^onlA0{~L`RpH-+6gSVF_?>R2n@E#DgbDpEt*<Z@je@xlV4BeZ%k2#RC;@KHPXI=
zTv4h`bGcu3y~M9wU$HGWe-9h0vD0j^8a@f(O+_$Xo~<oWUu<$5exr+)S7gD4{3wVn
zgrav3%g?CuuSSpayqwxe{k^|Fjm)EGSivv!zjH;?(yL3=fBQ=%uWad!A{EMEQA^#0
z1YSLDC4#7>-9zS&zAcI<8lQ|mDf7d6im6Q*Pa;o%x1R@B!{cXusK5DSe-NiT0+q(3
zJAul<M~5YH9~^lh-eTUP{6+YH%@Tq2W2W5mXeE;-FoLq>=#Q(^6LkG!bA|cqcF*pi
zs1U^jfwNgMV;A~2jK#)8*61YsCi9QR)xxWa4oeYf>?DNCwZ`dlvWs4bPp=e=C;^0@
zKx^mvbur_;)>z-nUx7=5Qf6*XICBRQBO-{5^FAFbsff<KpkO!a8>m~+y7p>YE4&LA
zuaOFs0yDxhFwwq*qMEm7{5K*3xDA#58|>mq?5K=|*ax&{U5}%6AKyg@FV2-FFZMsS
zrHF6K$F#kK%8>ZN{<(`uWDrDi_jGHzJr(KJ|1yPYc)l)sxy8H2+vzKLzZc?{M!E7v
zF8j7?Yx*gy+=(y$qRI15nG2g2=Obg4BUlV=;xP}#z2<_JKj<^R-sjvH4CwW2sL>TH
zBFbS!1ZIVePcY_hP7Oo2*3+(G-We}hkFa4=pbOcJV<M%e|C+Y!zUqG{n|J=*>)qtn
zuvYEZ1ox-boipvgxq`t_VP5lxlooZlyR(J3H(<AZF?Y%7yd%9i@7`?6^1pYM<8}vA
z(fcoCgsMf!oK2VWC%GQAgevYXL7o?Eii4GU?~V&J?0;;|X}e4MzW2UeoiA;v_Kja^
zfO*44nT%$s<HJ_iHD0JS+i49q+(`zs=pvVW8i+;xuYtKK!0_&IS4RuYlSNh0LB0N0
z@7M}4b%hGL51O*{(vrSeZ6D8OHCSs;@XmhzeVhf(Slk`m&D&IY@;78t2t{dNPCng^
zl3t`DCq?X(3hcH>ORIJ6{H~eR;2X%NKo&enSZ7_?BtrJ5+S7I!1*N++VP8<HIBoPS
zs4s)hk6=@{5NU`^H+?`wcP3Bej!F{iUVeW-Vmz5k<8mL=To^D!#+Hw%%ierPJ88H_
zPv82kH0g1Ai1s0S9toBo^Lh+BR#Ex!G~`x30&(~+!!jfFHn1X}Pg@a3JmuSLTQ#Qb
zpQ{H2%o6CZ4m_;3gUII{#r0Kl7wZMxrF?pq^i$O~lx>HQg7%WjlKr9^Cy2f+@^BqY
zbx<#R&U+j$A5c+HBI@EZBVkOHH&Z}NE>JzIVq7w@Q)x#`<VnX!98`B2r8m%6Vo+<#
z6|brd_*__wX0{t1GqA?_MGCa@sMxH?;yyPPF1TB!b^Hhn>~1g2KzO8);$a8uAH*&3
zg)D9M>q1*@_xCVo#(yS{1NMP%E6V?!Zdx45c`#Zo`Hos^`dDktCFvy1sVN@{zyDdY
z>><Z7B&4vz?kkf60-^cx)}7#;WUQ;|Ust^A{Cc6ps{`J_Ho33e7$a~vXbL1#t`aT!
z2UJ5oWg&|^mMJOh+0J)(WRFNE*ofb`(PH0t*QCEMv<$k$g`L*rpQiFCT-S{2>uUSm
zw<=5E{Hpn^91&*!WTH|0=HF?*gcfcA(20pesM@XX<znYR-nMzrF*R^)xxTPuJ-mhU
z$llE1dA!{69o)n5LZtfsGN@04B+k+hUE_rgg5;a&za<srs{J-<3Qy6+aIcv(Vh0~)
zkKgAXo_BfQU6|#09=7DXY~^inHrqTNHErF7_5Idg-n+BOIH|92e|zJ_nvaQy)}E~B
z(h@~J+X?{0YlgxkKbSHfz6PX3+q1^PZ8&z^?eZ~;#lzEVFypvJwzmp*NjFhor7Ydu
z;qg_Ml8d`7>U`!J{-r<=sz9Ex<rB%wndw8n3T-!>HqoW|-MF)wvnft=dI;3KrdhMf
zMAysew~LW;pSGKW3p2yd|KC%+rqf%cE><%ubiE&Jx1%0MZ=Vj7xf*$_KBZqWak)pv
z4j1_4XAlThcpZNinfEz(+YMJ<jd{>q^T3Cm4}mOujeA_~ly2A-OCo@BUWkDw$?;>`
zNAbhK1bj9N)%iBl%U+H3#(W(f7`9(~6|64(y#)u!7m=9$ls^wV6HaD*K=}$*?`*(r
z%PAmMv7h_OPY`2waY<I!<1y{9lkvg*S2VsV;C^W>H9yql*ma~+oMt}#AG;;>^x|`8
z#R49-yL6HlpGSwi9Z64GB2a6w($eJZ>gJ#mxlbpK{v}O&29pecI1p8Qp<h1SHvO4N
z_+F)%xO0841x)wd$&8Vw&1HJ3w?MV^H)!Ha$X1O>IO*|zpZSM6>*QdWiv2AWGUOWT
zi-U3Fb<QX6QPsrlDFCm9oEyVYRmw8*Eij=nt7ZU<rce$)Tpe9zp*)({+wiPnEysrt
z%KsJKDNEN+O_ku)jJ2#;k}_W1r057%=`a*slt2m_4Ow4Dxwv;-?t_o4?>G2;iM1t1
zHTq@j=TP}0C)-I{SAz4qYL-K^qweE9?(nlNI~&!^GA8sit}QIH$3qI@ShGe{waqKL
z$gP6l!{HMFCb=eGSRp?HQ$W^k5NfTr!FhVBHe2dKhrT@y7@j_LUMO2X!U<qQcl9^w
zAMYGS1h^`W*Fliy$>29w+j$DuOrED%X?afH7nQAm*uGMru4R&AA=<D2a2dhUa`0r@
zkTJbBK6OSuwLJ5|P9q(YMcV`M$U<Wo(*mXYZ8CH?GyY$Y{x{kJ@CmLnT3u;ei(W?I
zMzAO`+bv3<e%#6GvtLFB?dSqKV^E+x%tEbya&0sX)%=q7bl~KNG#gIFFumkEV<7hU
zq)iJq1xI5@jQ(S?U1mj9Smdun_!|Kx7{jS*z}2gF{sN8ft`X>R+<Q>}*9|j&!o)35
z{$bev7y}Xb>i>Ao|77I<=gZUAg;co|EjBB;+t~N>_f;{P9o0(>CsVsQIY)8(M+{Z0
zX0uI|msFKEtT~ttnsz7Q&HG#`N0?_r9`7GV%pEN$8bm$wuu-KbwUr2qDS>DJ*c!rf
zo^{1mo!8969ZL|;%-@E-i_s)2-OJ(P&$+jIm%hVOr64opW`*}ty|pQ4f6VyiOYbe#
zc=({mBPV?e10QS2MJS$rn8*$>vA(j}V^5F$v@VUb2^BbA=Q!jU&j8~69bkn$Rr&R-
zO%6w?WLzx_VTtLj2CV!KGIodUcx>y$1FafezYvGt?k~pa`aUwZ1UZA|ee$tkNv=Cc
z{JGZu?=XZZBj@Ofc4+btE(a9d-5DgAA*&B(**VS-x*~(_)GHAM5wR(${3Wbl#jKod
z%!O|u`G9Q+|2vP8d@VU)x3xNg57CaY!$@wqG0aZHEbB+A(>?I5v+>m<Uz;jVsLJ*J
zW~+@y6zO<$tl?|lNJMu4hC=%f7vE5WJWMT(IHLy3^({EaKP?YB7ke1l@pqa|SOJ6V
zP+$pl?BYl#$__WuF_p)}{q`1!uG%lf;Sa^0L}IxMpnos@Sn`)SbPo_}$YtIzMIKdC
zPs>vd6ZxC5y|K}HBV94S<<b$`HYa;Z!TZt^q6v86x58P{wkI2UoRXG>hZ~Pm)?ev(
zXVwAnw+Kn2g}4PvdewlLVqV+5V`7&VqCwG_bUp~XJRAD!ZJ?#2yhSQ*$+5}6kL+Ty
z7n`c?qP19mF#uIzF`@8+FFFm+?x5Zv7Kf||U6sF$ie@YBg?|s!O5Yd)@?QvRXPw9W
zzAFH=P#$oWOh*kT(7*Ed?XvkR4WNH}In6(}n`hUC2FjAX9nI&<5ll~Xlj0N(M~gM9
z7W;3xm?1qmFD1O@Yps*F!w=>vJEt4sE)}`;KqRl-pB?W8e==fRQHbNR&yXn9-CPzT
zX|2+4M8JL)^24fW-lP2u?B&<ST&l#FFZ)D-uFKJC<Kab1+ygzELfogXQQ&I#70tnM
zy54n{`>H0Fw|hk<)fQE<3!TG@ud#0no9o!?oNKC!@}4eEg2_~&7N3qT-Qsh9Iv>z{
zm)0<yr)u#7_;6@3bvjJz+ggqdL`H_^?hjYdPzWpB%A2F0&w3<JB_EHSqr-ppKXP7$
zrWt>;AauX_U`)J@pMV`1Q@7$z#}5b8W-NWwOd}B|VY?aWP5mhB2UD^`MzduiO()As
zi*rJP!fC{s1^J?Ot$QRN*3s2*LB2-Q@>P<~K2Ce@@ChWHB0qvX-#@OXT6O<z6!)t8
zuHHuvKO2MQPf~XiwS5gw^y}|f;*G`8qVqEEj|<{jf0-d9z;+?Hsd8<$bJw!TNAoog
z!lOHQ@wE`4mlgY=k-F55@ok?~VYgZ9ii?83DUY{@BQPW0Kt&6>3A?52TkfiF>Svqg
z?k5mH0g<X3N|2TzLCY-?>EQvDCi4-yJ&q_ZOB?^Ly!aEOJF0~0B(D@5kKF=wuhD|l
zU+C#Q8yn1*!P{yZ%n10gV=xIA)5(n8w~GulmX=Xr!2;1WoHvT$;rkC#$)4|!vFT+X
zw1r+D=B(0|$vmpR_ND*2G9vq;Sw=C};CZDWLZX{Kby1`C&o%_Y=2S+BH<R2XLYGSE
z_=^|+ssvIdqk*Vve*U?%e-cz*yJPIOZgw1c$H2N3J9pcuYz={bj%-@o*jDZLI~QwC
zBAzvlhUpVeBJ9G0MuPyI?ZP@V^Kz#H7sx;!p1xk4&&-S`|3LqHbLRJvK%QV=Qkb{S
zvFK<?FJ<lVt5;392rQqy5=su0>1Ji8iLbg#!*Kvdbu_1a%bzZ^ESUW(q;MjVZ0*f;
z!p?7bS9101myQ#T`rwFer2+tPS}~s>qQQ4@qljMnW0?dl0Z$d)m31QSYb~j2ibvAI
z#x<niUH!DZdzYp5SBs_*horD3f0JK!rz1mnfewB4Gb?v?I8aoYjvl{!QI?LHC9;+~
zoO4O`Ss;?Z^Aldl-U@c;ZGbzXt2gmFZFi(IS+89!*Lk<F@bP}#ZyFV(^jK1WN90`~
zt=Tthn9XCnm;+Qjz8G0Z>*7Ne;3SWus}*&Z1bGmhWqu<dAv#}%Gd(=ndllitwoRIo
z=Y;_o^88@Qse8?9I6Pwi)bibdeYC$Yfnv85*zhy4D|RdlSSDYL0)*rCoha8?w<DT&
z1A2C)NtZt0&W>)}@#IQP`74&+aOIqCHv=OPy;3=g?ZXcQU_r3<9`^sxN#N(Sl-g5+
zhm%cbigdiDjZN%d1~Qg0ODQFKeyoJR`kd#k|IXA-l{D|)&PM<Niw)F*#IwjCReL1}
zYT#Hf>PQn`j?{K~c<F;FKn7kjrJhC8ZhL_tqV0FFyv`9W-^|cIJ!3w5CB}a^^%s2A
z!kZ-9CkOI{{doQP-#T1pWlE_zQTCS9g~NcX{+SG4b?_OC`xuy3j+4X;3IK1etFU|_
zwU)#L2MPfH`45d!QU<#gu>th)=$7BR;X|@wqUv5<r@H;<-%;IVy_9xGyk!FH?40`i
zh&xk1X(J4^;^f7B!0d@?mW)-O{}o#QN$MY^K{-1A05(IRQ<dU{w)@%ijcKn;(QdbU
zS>2MQW2XvUa)ii)_u`-rnYHKB?wPOvE8o>{*St4#(@XuD6t}M<Kjr}^p&@SEy-a^s
zicMrbiSmDPIzRp-EIO&}#9Z*`{%WHma(;pwaXmQj(l?ztiZhz)?1y!o?2<o%oYP^B
z6=Bgzb0(P_rjGYJ4x0Ex{g{L_v_D*Sp*kJ(f4vqey-LLa(*Ttf96(dRO5oVI*kvHS
z)Yrni5#mxk52&L1ST~|zsG-JRTfTf6_;CA&S~zwWFHy*R*chhF3+&o6)p-`N@F9?l
zoSB%vtJ+66h^$SOipE>?*&Oc(2)=DpfDruZ?u)O3FB}0j^bIi0vt%bB26EIJ=b_`j
zOA1%H`Vt?X?rOD8woVkiMxan81My~V*uaT99~-NAswewgQT>020z-bjLUXyB$=(5F
z9G~s*y0#nZkD*LO`6f}2YVh8EaNq&r+Ak|aWUE>bp(T*mSAqFG%OgQ+P>a@EZ6~h^
z>o?rr#(V;ElS_BiX$tkX(wtspX6DCW{Fg{%=6j`jPqn|>OY@nhl4Zk*<q)K-4WtE2
zT=Xw+#p}(Eb7^W#JwAPHY`QOYdJj{syXN6!1lUnlx{1dr@tqdHzW4wjf0jn517Qdg
z=!?1^@thZ2vBvf)uiJyS<5q(c)qhhM#%1R}>Uml%abT1~`U|=XaVE#c%g7PeQvGo0
z)-=6kkfAb6jV(L}fX;C^-057|Q>o|d{FD#WUvE&jnVy&A0k!CR%(MCH2e^R#JaS_N
znggoLNCg_gLo2g1SejKFNA;^kwYJ$B9v)jOBm2@Ez1!lKMa;9uhM-!^XPe#-5Da_T
z&m`v{tLN!0^5385dIR!-r_<Mz`Kp^bCju;?82igI@{uap{k-ljoM?_CwNSE!ijVV0
zz3^eE&kIb4-5Snu?08)+a0#C*>;Y-^pa!G+!0pq(t;w3<q&Bj#lW>Mmesfu0*hRc!
z9+d~O{*%Db-?Y1~yh*E<m_qESjvJr`{%e1IS9+h$W$(57K`FKllA9RK+##=SwgngC
zEztt?2qCD<Mnjl+!{OwtG*ENSHW;<0Z9W(_c)JW{tnWY8)wyTyk2BfSs+*cg7X2EQ
z=d9!RQ!e9TV3MPe5(d$xXXIzg%MsRqLI|_E<WHe8nFI9>+r4Eij&q(H_&ItXP8kY0
zhglRO!te5?*<|T1*jqku^}AWsT}=UkmNZ?_-w6}<0=A6MGLhUG6ZHH2?K<P9jEQ}8
zDC&66za{7|^II6Mi%l4p#)<PZPsg3p@u<d2FBtkyiJhz@lgzafkLJ#UM3Wkm?;XTj
zmT!p+XUhU%EmKC<2#;9CaPjnKoFz;fD2UKSp1>Vu{yM{NH3}k_`}6Qyb0P^HP783l
zsV!U7SEgrT(W^G`bA^LWUP5e($^kB;9^Du?Kna&UdyvR;Wg6I|skPDw-YJr4w#ER(
znm0{dZBp;MSFJjxQVq*<YpI?-@nYFvH8Z-zqQ4`s2@8mGHbH9ELNAkL)kb@!ZvOzL
zbaE~m)6AOw^82--vSzn$W^!$Licpx+R^5ALo10FnjySnP4)afW9Ejf9@0+$`DLTa1
zH`e)E4e_21xXsPBr%mY(v^fmgTT&&@_u<6rf17?ToV8FwGlSDB1?s>OEw3~CeL_5B
zXD#8)VY7!#1e-|^Uc=Q>)#3wU7FH2@(uewe0EqooU%ubdPph|-Zb{6lZ$-_+c=Fu$
z6$Wz0#!NBZb&#$q9@yuSlAOHCLTk97!#j{y#Pj;7dm7`?qvAWeS>%51QL`21@+god
zr-pamJ8Ep7Y5(wTr=3<(`G1$X<Jg6Mdr)LDm=W=YXEqk0J%gk~N1(175Fo?^mDiec
zM!sZqrY0JrI6@L<(R4uPysw!?#(=sMP};O-=5t2qzi1Q!!exVOpqdSv9IMm3MDf9y
zuD5@Fv(3pxMvK>!09T}tA7WAoy;OJj-=}>aRYYTSwU-{%bR=a0zf6?!E=9~z(>@6&
zK8C=U>}FrAk$gC=0rbZ&^TqERW<P)XLdr+5L|<EF)G7sjxbCO{#0Q@#47%PgmKZ=3
z{#$yRkDr5LjjD42;*<2=GR{(ean|!S`bG5m{QY(6P(ZIo*^`Hp{Z$X3(?(%!(@IPh
zS(4dY1Z)srZKK^fSh_<tsh*Vy<*<C9z#oj^8y;D|LHPEHTZ7SL?TeV#u?h0|JxM+F
z&v$a-Cm}G-mJe0>S58vXkI_a<dWy~KlD|J@-8tEqdfE`m;leCV4yK19wqxDYYM##e
zpA+IgG$D#dULW#0qAVl*HY$G&UF|!Im(?`qu;GVQWCXSgr075=y$@$)BfPSPQP`6|
zmkxU2MMw5&0g6{+u6G1DibM`6IA9M_94K+Hl3+~%XeZ%rpotU5r(X{R0jrbdfAqT9
zpnHFA+gctLP7gX6g{}tQ-p1!$-ZxzoaYY-zy(yIbD43H}_9niN20+k%wYhBhaUy~a
zogDY>^k#-ahKEh#r<;;>HWpeIB#Hg91m44lI^)OV>+6|oWTMHiMVf~DxxLd%!zj=S
zpkh(X{r2$k&`JB)d4s8sKY94&!OMIc0Wwk9$@`W)7}h?)^1ln8<=Nkg*hiLE-ASh1
z#JxLb_z8neh`oAU$Z!1k%|-|OK|tqCp=r0pn?ez<JMjL%A9Ps)>6LkUH%{PifZBg>
zCU|i5;<{*hMp}gt1~raLsrM+_UiP#R8v9x7jZ=rGn%OV;+t0WesG8>QBy9_Y_t~Zl
zI^~ok8{%#h2=bz$9n<5{@-Uw~H_y|6Le!wL4de)@>mmY=4n`dYz0}{_)>{s=YFwOM
zEl#?tC;@{231|1ZA8T%#4%kgr<NrIAVV3z{mQQR4f;-v`U0kxg^_FJ7eQ^{t-SV8y
zXm3N_iycbP9)oj|ix20U7BkXB+=cW-_0%QaOj|zyF*)}*)4tJ#-W!cbMM=t4zCCQp
zgJtNsV6X8Jtht!{zR@NyDX4FCb%BJYp|$utVtWfdqZeU2EbI1(r&uwtRXrqtYFOzB
zmXBU42b3lAKw;WVR{Az13Sfx8QU76x+49QGBSXAKJ;z{sLe}vrhwve$A28WZ?T0jZ
zl40o%_w8W`%%vOGY}C#$guce=W?{is)$3f64ji=tEC9M5H4%Kq&iJkIu;J2SIqq_p
zET#ZpEg<6wtS&1kL~O^oePwi--?(*H8i>@5m)l!C{Y+^qt%3m}fK@pR4zx&~-%Tn4
z3K&qP%<@>;9FXw4t=Pm@W5{x9VwAO)YuI~E2Q3T!s;_x!IYW#neb^pe)?OB5PfB<}
z=Cd?cb3=Ixf<5(D=sJI!eC=YhU4t3sT0_3%9obKJoO19m`$zf=$Z#dp7+{b|Z2r6y
z59t+iJEcZ|3Uo|#`Ca5`Dm8`zLebx_L_eU{g%($mR%g(AUWX>81#~y-+@2DhAxvM6
zi6W)IZMj0jhJJwfS%`LLop4suSKKl+eB<7+(BjN;fZ|r}%^t`{Md5%XK38MZ+U45B
zJ9wx4WlMja@ZtmP6zl7Lujh~V_yii*S}FWl?=Y<?z$7t%iq8ao)mYO-TKv7)>JHXS
zw*82~hH+$dcE1D^C^9^n@|R9syt#gfU6_0MUUmp_YUkc}l70g#@{)V3^x;iyMfwnm
zYA3a_=MD~U15x7*SMra9r4;bx$!CXWCye}&Pl~tvtI~34_NX0=!BN6ix`(Ce5`EOg
z)b=P~s3cD-LZ6J7-vx!vMCHfi3s-ysvEc5wa^C*XQbliv&wm#^vna_)&kp-03=U|&
z2G75Ybua2_)XsEv-FxuMG7JUa1bo}yJFg74OUMK?s9fz0IW^%ct)`lUc>!S=v%>u+
zoEABdTY;uWk*%G0?uFLRoSTD0pxP2mCI1M=hH42X)y&61S)UJ(Cw}21G$v}NTbx#9
zAWe(#@gYCzR|HL~0gxM}f@k1Sxoc7WnuEFXD+aoYlZQ~Ft472{lV*0DXxFhkiB784
zj&g*o=d0(m1tOX>K3a&hh`bvh_95};V}22-4}OCI(8X7<5f}o0$w=mJ;a#j!Hv$Py
z_7_0g_Fu-G3Az{&ygJ&?9K>{cau|F1&QBwt{)yx}1EAXeTLI{RkxWjvn?h|I+p#2&
z<Xj1j$#p-G`2F7-7Oy@#HQ>u>pqhEp8MN@z@L(k~ZHWAb?ZPr(Py<vG%Rj#bSgrG3
z(usssK#OWmk;PvA?+tE$JN8_JmBy4s^Uiz%na6R@KrwAWV?aUsk75bJS4Gg~egJ*#
zSDa_`l~#l`k$j0c`QPsY@KkZ7vT<6i-TTsE48EmAeu}duN`zwnBLgy|<gd-bfC5FE
z{3M3~nsM_cog)H<z*Sx#&h+0(z$loBLw_SttNw(-vrb`+(~6Y+5`3PnF8*UFBq4_q
z<$DKVi-OB)H;g%mf&Rb7VFr(8X7%CSX2-g6$-JU;*(}_lTe2NK`NW8{pdurrEc>yl
zpAn6^FL**iwePkySaJh__2YenuX%leBxJK4tI^K2n?C0{PWFefIDAg{YtaHx8Cbgk
zdsu>nL=N`q@rz#zQ44E!C)~(q=KmZGlNtq1e?fp_VZgyJumhMN{?$z5>6l%n3#s1?
zcONTLrjo|MCINwc=QwdFg5)TOpogInYC>Nj1AH)$2X+Dz;1hO^?L^wXs&oCS_v$gt
z0G|`eTFC1&5aNX1G9muYlVY?Wo0|w)c7;t1HL+EJf!^Odlu=z|U;lZ)B1Q^<h3o~`
zToQEu8aX%m_jYv%btD5X+cG`h!a`eYYggyE2sbAeeSH5r9ISQ&{5QY@@|HBO7CDXV
zU4XN1TjLC6`7-@|FubU*x=o+JI2r<Vfs&ns_yqAMH?O%)%+Mvnvvy#W!G?&M0bl<8
zK|y}2sJZ5>wU-aRY~+ml^&eVVBHKFg1-zihJBIpv_JqBxY#0+G3kGV$kFDS}dQ@vK
zF-`tD4=;WvHDcFrSO}_D%dtATihK+NQo&u{)PMz&O*cNVRWvYg`Rpa;<06eN`3T#A
z$qc=d-i}#wgd`NgLSaVss6q(h%;bv<T7~#hipwiK2?OLNB<XB=lzx#_5GE#to4{h_
zPJ=w#$|Pq61S!!1nompJ2c=`dGOzI^|Ip3UgNHM(Q}VC>kgMr|IpWoowe<AD>D3H2
zcw(W@!Ka^kFyj0jT0Q%I=pd2O-pi=F6Lhj1JXJTR@DN2&`MHAHKQU39LKv^I;DF6$
znN+CVjZD>G$ezHFAOWeWbzNZvm)l*!VQDtFXz;5$bArquU!@tBiV*&}zkrn1^2H{H
z3$+Y|SPU3KGCPnFv;!=`e`8`IR4#pm9h&mi_VSW#ZS4ab2P%Ldu~y2q1_~rOL9f4a
zLZP}!Jx1I1s3F&zo8p=5Om9XD_qZllQWrFD$lp*VwfA`9>q1^C6UexQ%5e6Y?<%Bu
z;fqL%n`XzKoETaYyGDqIFg1%353Q}kGi56)L8$D`W`xBi4<69M!aMFB4L3wkS(uJO
z$LJsHtP6yIDZ=4TqWo0)1R)10M7dm}0FD|gWz2p`8bBE9>ai;MvEJ)#pt2M;9Oy!f
z0k@NmPFu~hus$|gFuJqrT|T=VWdPyi{?EqLT*h46fEWQ^IEF&#+q^G)E)Zw7oQ+>{
zUi_4~?LCh8S`aor&A<}ksb9}_fx}k?CAj|GQ&h$Z3GYUkG~N&brlmBYB9dz=K$#`Z
z1P@Y;q-JGfrc0xym-g^&pFz6z{~D?zl?j^U{<Iz(#g`)=x*;ySIsX+np-F)AGT>Ej
z0Y1O9(ZS!y&H;jmE`tqD2Bt*3uAUFhSLS#h*P96by>%_y<g==G#9Yj;(9=^z9(dwV
z9Z)7{GZjplq#dVIJTnc>%+G!Sl}XXznHOY;0Uq9Nw1#%VP5{Kl2yD(K@pgoso>~3U
zJ#h~aDkGDT{|Ek6fVz7^A1|K}TRIJ8SX+-iu-1^@>ysXKM*N#G^+rChC|m|6+rIvO
zJQd>SIUXHGN5=lot!Ka2C~rB}(eUpKAq-56J}_LVx41c`n=`d8%J&jRt2=QqQIfN$
z2JEMJz?MM0VAc{|sOlBzC;yCF_!QMYbajrEdo3MxoZxT7swhMX#0bN$e7QAL@dT*n
zcT`cZ^eg;HZ$_pqHShPaHsAO{N<u|QLd7MXZTa6X*W>APbHN*LgQMg1-PEdGxRGg0
z!8%}2MSq8;hg)R3$UG5<mR1S!1bU16g~Ptcm9r@)OB>;XDg!>0sMTPD5&DiPdIllP
zK&^aDWa-~~)8DeCukYcBo<zT0<#;8eSMWTlA~5RWS>NoJq_vp#oCyM;brmBD_Kc!t
z<>jugH&wZL5x}AO^%;!PV70LNos_y^p~=|HW9MwpKkJ}pTtACeZvK|Q^!|g&^IG#5
zO7p|#b9L?eQ3qK6cR<|8Pk#oCwIO%dof@PRRIs4P4&cV6AKNVvTN0a$msX75wSxEV
zn@rUy%@@2fJ=cQ?J>(M|FGl;Jnu@<!7ra^d(e`T6T7G~S99PzrNo&$unjbar=-5Oc
zfsy67s5NO#Kc-`K%H5P8r^E_knSOa}or+QZ-GAawZ`MYMLl-1$L6e{JWAjTzMjcL1
z{aW1cb_#gYo1;>YDDAGW2ww&;?j{a9pwpVx-5}e_%HXMtr8VDVV=lO!%uUi~gbrhc
zg(8`23e=y-$QS}exmcZq#$OM2XR5x+$?veJ?r*NNZz4GWcUq&sj|&MbU_T|oj~y^)
z0PiNvVFwaJp>>q-Z=$L)BKu6X>||wrg)!6af3HuIR$~oO(KtU%Wn^U1h^Q|e%|Y3F
z%Dh}Vvo0omUdgq4_H9vtPUC~h7}uXH4&u6C@uOQbQ^tEQk&H}<FZ+A3@x#4r3@_HA
zqG(OwS2<2-bH0=|`#4WHzTKabBs_NJDL#PB*(*aB%}Ep;(ccD60zE+?Ne2AN8FB#k
zDx5BZ6h7fT$0r>sQ{hjYN1Z`FcQ=ll(je}j{KJC%7gAlEXETwuzuro^QRcM{w8f^N
ziL%8e$ET;op7o8SFbhw45s^};4{W+#t2-Dl6y~N=vrd=J1v31Q=sLdHd^WuYAZTvJ
zbtRt)b<w>7(UvC9@SAQ|wj1MdV+^D;-pls)0WsZM*BUEDYAekM(n<5l`Jnm2y-nGm
ztbx?V02efQuA;0(icKgT8U1!iER)%Thd^=&AXIL6U3vww_VTLNMV>y$II*UpFZ0TK
zX8WH`5gk3b=G$Rwp(zU}TJ;-nt%JOPHcm{+W|nPmdW*-Rt$qYsgZ7IC!)NE$HsT7<
zeBg;xz$T`=nEC@HR{5v|u_N_z<!7k_0zj2KSEi!_LVN@Dh4yimuEcoO>&Lb0OCU|?
zLz0f<qLs(15}0Yo=o28*TnY;P3|prezp0V2g}fdx_xPDF1<?c(JTfiuV}JhnT+vGz
zf=>{VdbCDiV6iO@w@%eGM&C7ojUSr=?8sK-8KYg)pQ*`rOyP9Tz$FDJ$C0j^3cWfY
z@6Xg?51`Gz6mTWVE^_yH2hayc=ZW89TpnJoGH!Gn-`~C#DlAVxL-4yG*5YSBo8jOV
zsH0rCm_A3OqPRvWeS8BQ5mTvtNgVn$D+6XBGob?EO>csR$}+PNeUDps@D&vYye86?
z+ehi;rQ5@jxScJRJCRwY5nlAgpp~wt-zggEqK98k9RpL&%a+CkL57L;$U+9(vQ{Mp
zvB+l}K>;*!*PQr~5dRpMud*Vzsj7U~q_Wb^?qk+EVms`mFOBg|sKX33=+CIA$i|vQ
zQa#5jY1f;Spc#J`Hw!zhAGtn^{6wG}e4ph@7dL2*O6--g-Q{U_GLRyfpbUi@Mx$OK
z69&2MA}f9wX)si15G&J`fZMgMcOeL~;a*91cp^7Do13j=<??Jgs?_cT^SdNeMI$D;
z`x4_7{XQT#Ga{r@Uf0yv6`syQToO=eY`_xWfQ`6|CX%h(>nnNs{D}O+7LVkp$i(RW
zsrSD#Lm}fH$pfF*h+fJaq_7n<z23KxD~=lH3=lU|uucoeBGW`sg7o3;{BD!P7l~{Q
zY-M|^6XcxN*&~my05NaGB77BqmnPwE8lz;w$nYC?a#D&-{y=xEs?yW1qfBIqyTU#^
zFVF2Z(8Eh@>7tvBg<NNxv&s`t;Ew3s4^CvI6h$OWc)o@NolQuKLXHNk08jI4&tlB0
zo(4i#2jy|h_6n<XOJR@FORTdI_Zgw5z<l5wJJ9gGz54E0QY)hHOAA^i5Il%>8Xnw9
zH&&u6#Nb8kQ=J+aN&D<y0?-*Sf1X`B%0|})Zvfc?&<Oz2T_LNcT%6V9m9noU_~hA6
zY~Vk}5P9@}@`7qSr|@!HN8lPgE}9hYQLEC467X0ta4&Gq2uK^MM3J91R;j*fUetM)
z3j5IJP<i_khZbiAb>~lb*qH^$ucqO@0G{$D54LvVa(8(^_QCB}YAQv1e}IgPOkag@
z_YdzR>1aCLdIv^kW;q6YTr@9AdU|@|I+v5R)>cv5-6v@onYk*{;nj{nLyWaC(_op5
z*?KkC5L6Vb<<PIWE@m`zyu4SPAw>OgELGLjgz@Y$RGql0q(>h=Q?_)+u8imie~DZ3
z1t@?WQ{md$8gL@Z^X{O=&CTt2D+8B^sOEET)KIFRU+Z6yuCcMxllAWM3gq;*&*tWg
z70k3he*6HA=~#~E4t9l+1*=8|L!n6=#$CQhXpN1He{zPSB2z!k-d%O|d>ziD;o2T*
zag!=pvnceJI2n<YGcnn!v)^{RIxNY`qG2_R5r)z-ndOm3*EN*}_-E#C0PVO~KnwAH
zxR^fM9uDR>M(Z`I$jI55saT&VP)Qu<6C@Ld3?}m)Ej2YHCMJGb>&r_Dg7&Z1x!lP@
zWmJa0^+Ht_-R@mCCirFPosV9Vkmrhn4TU3&f}whECtjH<x)S0gPD?1<0Ifx3aUT%q
zEgPH7Shlpt{V`%sZgFu`2k3R7T8Y-h{&bo5)8k6j-@jk&L=5F6ko=kuKC;l3=0@QY
z$O*XJp4@nx>~x_*QO{iQWg)$t)cyXjam-!$_2%6d7eujje)F=PCoFeBFrj&)3cB11
zyOnqY4o#FUy3*J2#Cnb7F!*)h!Ec#ZUC3zfWZ!v%0s-EX&6~nlwinHR9~oNMH=zSz
zcV2GxWFMQEA&$dUr$vC>ZpQ1!vFUql4J015JUt#RHV9EhB&ZhT13YegeB97Vy;R2s
z#H!!y@i3)pg=nmzO+PXguwS#qqBWMM-@;=xA5JRhb(8da-uXy&xThXi;cS&@(cWHA
zt0JKqT1RDN<x9=DxVYP!n<kGNT)oZn-SOika&mIfoIftq$jHdNHcQ!~X2Qb4$4u-4
zy}i5kV6nWH%R|2{r_V1gioD4Za!CUOB?&qRU9wM<>!|yfJ98dkP=!wj>TN{4Y0Z@m
zrx{$FjehN|8a4W6+nYqkGBV2v8oFK?3g}+9UD#bj_O2^$6Z|!i0ze9Sk3&n3<@g`o
zf8Yo>8EVITD-~Ks#!$$l@(xVF_YLS5&sAG#o0;td;n3Wk?;`4Be;C<@`KB>bQ?CK8
z0=WKsgY!}5$H0}RGa}}hUsl~J@)X}DXZxp32NI`?oR6lg=FrLbEmp?}$f^sGr6h)v
zMOuLNPk4AZE$y1S>I-D#cJ23ETr|JoACrYW1sW4$V(_sE?lT-69c#P>+Wb(Q0Vi}l
z+hWnK)@kwjz|Wtk@uJdVvhea?zC#3S2ldCna?A46R2((a3=b_WZJS%Y^ARRze{b)b
zxK0LDz-i>8e&9~h{w8uWsjdh<AK9s;r#cFg2m(?V86}?g-HoMDxnH0St6)ocegeL#
zmI;rC?c-9_zT&S?tCU37zxlO)=7bp|5uRPeVtP4-QAMlz>+b3*c=rb1KVKBFTwzDo
zU?Y-7%HShb;k6C~9T`y|jq-z=s9<bzGMW`?NT)AduKv@b;1$)_d!8qc>GYiF<Yb$S
zuKxHE?HVA;f4<m3jkB1qKG^7uPEAeCL`aS}K0XG3VK{|9nlECstIK3(Br}AFt-8AU
zN4g;XM{)6R3PB#1V}ox5b!2!F7wZusLy275K)4$4iC09yeBb*a7>a4!q<wJb<UAhj
zkRc72zu5jcV8w}b{n`v0tSRVjv6uWqqO)scXM-zW6jJAUgH!R!^@f~+M%Jv5vU$u;
zS_$lAO$ma$^;sP)ZuGFzH(CS`H!&#)RSxlpI4MCPRlujHkU!BEn?QqJP>>8bf;raN
z-j1N>OI>(+YNtWwVsyArcdYW7FtD}Uu;a&IV#{%2+|V|x^4quHwkO%6xPnf9YmGXg
zfBrzD;Z~pCsh578EF>sC0QOtSC*Cg&E9UHcSLG*mM7JNjZi5rjIQ9}XE<XN?GECXd
zt}a{xaY%Orm7=^nEfcmfq)e|#Mpm}L^Nz<u1sL}`N6Z1fq$^AV1~rOjbN?99%ci&v
zHf3bRpawRb&&)51gyg7%qE{e@DX_EJ@xtiy7Zi}h^0dqP#sQV0qVy*j!_S{pX_+jE
z6nAPE6m^P$&H<B3Y`)ji!xiw_UV}IKCy=hL=k8d}#~<jn_Vz{Ra;PsEnXXA8FCOG;
ztQR2=Nb3t692{ULY@P#O<mvuwYcM%D{xu>Shxy()%CNo{D^qLVtluCb@p}tMy09k#
zB4S%mZf>qho&v=OH*N=rp!1<WGMYe%Y&@&jI3gxuXZzFL#Xhh3*js)5<p>)Ua`wRj
zmAsYtiK!_%CjWfy_wNn4Bp|Q-^B=F*v98S*8eQ?4#YQMychIs@z^}m^E(F7}_cwB)
zMsJ)T>2R<<HijNHhR~p#58h{wYY*EJsra#BMe33~<2_v*tc<ho*eAGNXOn{4rV293
zUxH{$fI~({&3Bm97l@Bk-#QYxOvS~;3ui@Tq0k<{&E|fYBc>+i<>h7Px6+o61`;7D
z2I;Rn?atR6LEi(XT=L<7EOCvvbAN;Onja4j@A?g;pwn02ElB6Xh1qH=vPOx|pWlW_
zBIfxf23^4k-`yRxsH&>2-cd72XfRI!HqVsu?+O9VD^^xjMGMvZ#8wR<WLe3BU$O$i
z=I-jKx5JotGZY~h^*);@QDjZ*?cr?}9IVp$3aB|g4fkP&gwtTLMZ)UuMN{LV;`Ehb
z;u5p0KEIkzw~*NUdw>OV`(nnb`R_JI>w3Q*002();%M1>rrfZcX~+E&m?LpuzQ*Qb
z^68DNXvYihI&7_59S9*+y8Pza8t~SpI(A+}1Xi&<fQAN2-zGD}{D7BI;XxW)1GH+y
z<m3wgLWYs?cK!K-O+Z~&S9f)^i~<(3k%GJgM@o$b{cLzN7#z{O+?%vKUTIrhT^-?s
zN*Tnkp}sA2314x2ZF!UIzp<8U-$yl~M6aUbevX?G6C1rxM(%Y1?&Nm|+$<N>M3dir
z!KF|&HXkToogRWsggy{#342>g;Qy24KB+=}qF_-v?2dhzt^D$EdqD$2TCR(WQwzkT
z0DzR<RhK<m=k+mGSg{P`dsv96CJEJ4wb>{PNY5XOoY%WBSmV)BM}Rby8Si$nN6<)Z
zYlP(AW`Iu4_YJ@%YinzXfKSTGV?Kx|qAs_$w-PyS;N6{_V;3PZGP0fb^onWdMLCW3
zp{Ei)5=dUBGHk9CNy96d`NBTf1Rl0hy4{1k3@sTWf((D#AGV`lQRG-;RMKET2+?uN
z@Vd!F{$(_JLSR}tUw5<MrgoBYC*TJw<QyX;2<c4XHXplM_Fisu1?o<0AU1GlB<~-t
zmZMBrv0p`m(Wl7+X&dhPlh0~K6$$6(p1-qICB?-Cj4tyHP6vBSO?P{fMVWy#&R|~!
zC%_k*J@4`q)7!C98N?f#n!3X&(9-;-p!ptP2>UCM(*Ah1r{fN+5tvU=5C`#>5WwL+
zt$Ugo#UTLob1Nc?Zh8j0nsE2fV#5!ZFNQ2Xj?e4lUf<IFEiRVc6Gu3Z>X&R5ChQ*-
z6(`@}5dm&T$5`0dDCFpPvFvq+K}thJM0CCV1J=^=1SE!~<fK_|51!b}fP?DAJAwP#
z`GGn#y%sJyQ0&hE7gGbc)9*D+-cMenC2L^S<1itQ>*M{h765d)i1*h+g7C=51i$p*
zn-y|%bBoFzX9ZrOB?-BxAE!y@OiDnGcm31HQuyrw*A1KoGg|$VC2vFaZt({Du{xRR
zzdg+Y6@-njyW?hL*NyuyQ?pCH?c?|f`*oK}NfDKeh^_Pi(sk&It_~dyB|8raAUBYh
zhoB-X)9++|48)9rA$$<6`eJNcWxZH$3wRGXANDW!GIxN4f6u@!YlU_|OfFd?-s`bw
z*KG}_c|R=u-d}ms@e+*mLyC5EcSJ$V^lWeP-5RrmQV_sjXLg1~o;<D&DNdLqu06aU
zkck}G_^sh|5n#>0hJ}-GZLPHVO-)T5Fa7bepkty2u-EhU9G=F)+`O$NrxprMV)&U)
zt#SXZRMR#k*Z%CS?`?<-rWi2k>n|apq37r4pE(o%uu(>IhlGb?zI^%Jikl<%htK=#
z>uc->4BFn_-sa|J1GL^$n+!tMcrq#~tg=lYOb-t!NlBB^()2Vm@bPiv`FeZi#AK(B
z57)P+4~1(V%3*iN<U$^(s38DWiZVqJIkxq6w6_~rjg$t<K4E!Sl8}%9qbw~g9gDLg
zFX4XD61p?aIPoQrY5w?9gB{>9?CeUFuD*|Ke0;=9T#fTxEf23Hx4`&dj#}!#jWrxB
zSkl;FG|<yg7FPh#2EzM*I>(Ds-eIb!LIHhFPZ;G#Fd)y7@<OQ#uMvFIURwHVz0FyA
zAqtKh;m<iEWSe)sci{7rCd}eIJ8T-!s*=7)yDgI7i3i%(_^K>XDRGB?FM4T27~baO
z#-5zKunqkRV}POdZ|XY^>O8RqE88Gs<bN`AT_9x93Z)e3WfKm_|DIF&!9&1gZG*OG
zI*HGfj%aF)|1(3O*4y~hBRqUbKvQXp<j|qia}0@;#|J^cuA5qKTm@EdV%+fHKp^0l
z6mH)6P*uD!y}V|%NmFE&b_p{Qa54}_Pulzwgc(q&Ydsn<bpYwle1QR)9~BaiS$98k
zpX)@2^c_YWiBFKKa@*JAahu}WKUM601Q*AYh5kqlJYNEkuScSoQx_P4qEr*;!wAU*
zL`nlQkSE3`c*V#R(c?m)QUtu4&Z<$ZygXNd83%O1e#-(<LDoOo-dirt>*xA3guy>w
z847wcz^RTLp=?767~m!NVsHFtB9;s|&_vE-F<@-f#AgEkMFRskCcrwve)B@zIf3ad
zQ+3cgFJE@#@dWToFsQsQ9i>uX)>LUv9uau{o5Jb?FG>Lmhi-*bqB1SZ!>wo?rFbVm
zy`5z3)X_6@5PDwu9a05<y3Uz@8@B?!IE2!?F^k8XD7A{=A{QPC^<TKyCMU0Zx=u_3
zUdo<#?)e}T-d588a0-U}4iRMt&CAo^s|N#`^5);j)c$rqNJ3YHcu*f*g_)mNlmc-w
zuoelE@}owNdA<dRx4X8VRt$3e6l;u472T&aIh?qpDvOR>;B9juw?jKK3!X2Z<$=iw
z05YbN^%Bs7h>D79V|)5+?9;|A4onVtG{rT^c0?V#_sQyHm?_LCKF?Dv9`7v(DVh?;
zwu+GTaIEC6bKb&n20jQ@=-BL%z{D+3T7KP<*Kf=oh8KXX`n9<=tj*HR`Nq=2DLGI0
zv_C9%I{hsO((3L$ZL5-taqGOJlIpxsP6`~kaTFCdoTSV>MMXEPI6(<A{|jtVP@a2_
z6;%d9LMH)c9Tmr;G`B;3jHX@tHcW<*$@^Z$`M4YhFK?LL?i;<-)C+vo5=1ftElxi`
z^9TB4oRv=vj4a6Ge8674xuDM!Eq~M<8_s)}PeJ9|Phd=sSMX+KdQBbXx;8dx?R}Y2
zXAo&D%BT4Rcv}5i7OwfJ$>@JY3pa~;(brxX_`pz+-r(8s5Sv<4cka=UeA2MI$2xC)
ztcTm5if12%&wj0yn1N#8PvQLD{W`(4!-~j}C6Dwq(y*xhD4L+{@)e;d;h(6~1r-11
zoJL^C@d^Yu<ajMG?yi4@`rg>U8XBWx_SOh0()gsYHszJ2(+Js)i|}!hw(d}t3bPW#
z@zo~=q1^fo$dXTHo$K|ExLlUxBj{p4#JO3`L)3{eCwwAB%BwDw?S-dfe!Ft(bfBkN
zQ;us6?+G0NNHrc1SfvZh|LI|@M#m+x0ICkR%!Ur+1j$#Ma+b+Wx>x?tH8|`eY1vxW
zS#(;bhxbH$Iey#n^#8}yR|ZrSHSJ1=bfZY;p^@%7bf<JTlF}t04bmmuB_g17NH<6c
zNOwth-^KgA-@W%YoW1v|nOV=w^Nhry_ofOW4?FmR2pQ_ZVH4eEjT<Z|z;tvu;4sxq
zUcGf;f#~M_26(516DjzLIDo$)lnlF<JhUf#$NRljI`*oyJ2#VmDNBX|>^gREQSoRe
zkIbP)iB+WETB|K%iy^#OzK~`m=v~LxaH9-ibWzxP^kUxF?-EW(p(OI<xrdTR)ySyB
zLN)vFoEuH1$%;`?&1#O>@QY6<pYp}`6ztnpKGGC;S){=J%{*PUl~iW>I`sY=4;D}J
zje74kT~V?6b9$u2(bhZV8GwA~Z!GoYkUQAzDf9eTepz@T{Aufy(2nlBk@$AzC)Uc0
zYalt@waD>V`p3GG`;zwXKr&W~4qCX#^3T|jl5>q-DN344S$`%5gjg65Kz_pJ2Vn0c
zL_rC&-txSn=2~sPEkX)rj2vFy|5Obiv!yGOr8CQK<MNruoZgc@x1_aGBS*XY%QWwK
zl>G7kaynkT%q`s`br;J^OeOnoCL$>%5~T9fJ_|-c$eYhU8QijGaLT<Yd(8^_F6=9j
zyR2ee2q-o$_h|t3<v*5n#RQ^YCY24fNPOoz%ze<tlc*-W*V{}#`u0h`UBRM}5*?~}
zm6Uu^(f(y{ZeHZS85kD`zNYY#gjB@8>S)%MwFDcip1djSLfY?OwLgQ^ws55fI5QY1
zKZWdb#s>U<vWzfRxNy=-D6B~jyB>TQ94K{s1L^?nj|>+j@BoB+eed#&Fn%f1&`6_A
z4<9z=*52i2qA%wU)mFm+j`lyuyvbL=dlS=FR4$C=6D}qUOGQOuwGVy?7MvM+#rh(e
z|6RCZ4Ue?Jo_P@(W{KywTLNNHV03<}KVSAa><MS_Qu;aXgdm49IkkHPgFN`;Xcc;M
zO_0&+?Rg6kR{h_fI0mH={v-BhfUemnJb?QHw=z|4NH4ZBqayeTW@=xp+gbi~M0e*n
zGd*Qf3klfMg`|!6(@>vYoq3JfmdFzJ3we9)qc|@Y8SiROT^nGtd<F-v+bCAasU!1c
zT7tyqNEUVj<GnZ3H7sCP@xy-58S2n%dZv||B=j*~$upa-4&Z_6wUQS7xVTK+n9|JF
z2lZoHNc1^ee$$8Dgp7=gX2G6;o`d4g)9P?~ph7@>hL@KY><OYgm87B3Dk+V+Z-((F
zoX^v0yL#fOMMTpt@xI`ym|zWM^JQc?3%1|2imsOB!J6y|E(-6k2&XuxJRl`1lIOc8
z*48)Fk0g6Ud}&19v=%2droNOBo(~=!D7r$7r8bf^CG-&dGOFP@`!exKr<(`aBBW?N
zti2;)l;KSg0d~L2k~+QWf2KBCS+35-y#!3vzz}ARVu)%dd?A6uTLsRTVO(Xiy*DM-
z_cyF&R2<1O4`8*WsBtzow&pZiug0G96yMg^@S{InRz#1nvJ$Qol(Q)cABnq+&8^{U
zpc5;${|mMpkY!r92+J?``z7eExqLp>IX)u5mNd3ac;Xp7{I8!^S5AsxN3I1pVM*6H
z_JcfW%NiUq6`Nz;Ue~V$bN1P%hF>D`%e6s<f~i9IYHV<>t7ksv`fo>9{r9w}H~7T^
zebkwoP*!@T<Yv1#_DF!Cw&F<0*Em<Y9Fd7;c49m!D=4m~u2-=X@0KW#L!Pd!G0}?i
z{<GK!_^A(n0d~wv18dzV2qz9oF^;tfgFB`muzCF=KYw#lDh~)uDN82n+{2WSB9;iU
zWW9+`IwX8`u=U#dXjOg0qMgjS*7*XZV7rholD<MbuDGw6t?qS}{5CSM`VNJ7;VgWX
z*)@u>mMQ9%Twd=4tBc5U!Xb?-aB4!bH=R4@u(5GO@^rCdnHnf5qomn+&JkXlNf+cY
zlT@KJ`la|)q;Z5JIBJ#@Zd5GFyxSJ{ZZnz4#UD8;1`=tU;S7O;lr>M4E^@s$RkPP`
zt#B+c{ub<E{6PxF0v&mR+F4;k$rtC0HYqWUHGf!q??u)b1#jZ=gG;!Lf-0=u?m}$7
z?F*cD7J29PS1UyF+YsgA@9r0u5pwz4>qe>JYR8=-ju-Qi8J|p_O_XdVW+1Ue26}=-
z?ci?C$otdW(pS-G!{Se~8dmLW6=NSPb~jen$Vd}4C3WD%zCaws&!z>_{dX;r_|<qs
zm8zNYUS^eKzJWxRQ^*byN^zX}QsN#6!BJnq|Mj{?qBB=Pq>{l!e7r}J@7Y{sfiHv+
zRpk==A?WX9TC8VW7?XkdZt!o<<XcXbY#lk`P;z87+?`BJ)kfI$QgNDvU1^YkC8*Zv
z@gE=fEZp%8W)-`PX;2^Drqi=H#hYC&`8cgTDSQl9x0|5SH!-bWb>c0sZEBygS)=7x
zk(CD&J6-9qEY=znZ<*`={bmvd{w^l&>**Or`u;^i(j}f89)1uxI?DaDh`P{WyvH{j
z!$T_c9RxD{otE^y>amA+vLt)@G%(N*a+zSUZC~X64Uc9#dlG}BU!Wx8)eGIWex_pU
z*;bk5a}Gf%1RHBr=T8hI!a?Mb3I`5_ur3qG#sHh&kEJW-ApsL*hLftM0V$*}Z^5&Q
ze{z5q!%Lm`UXirC&zjJUt<yu;XCg42D_S8}Mvp&%r&+U()czhhh@QnFkC35#hAS|I
zRqbz<MmVS>pg>lm`H6#nuBO`P2@93L8NjnfsMcIAQP~?rj6@ZSgOnwgQl<s%f8!%0
zsaZP-#Y6w3=Q4nj5xvg6o77Yjm@kf1C<ih~?dLVbtfpk%+iN%e=snF&qn+9(juL$T
z?_XHTJ?r{;o19h-jz30o6tymtm_!WWwxRXwRiqN-pQkDnlw!e`C47m3Brtd$=G5!c
z#RETv##!tzKqI9ZlGKC)4rHcUyO3pvCGxMUdBvMmdVVPB{5So%#dA;3lGUdoTbA-!
z!Mryyc0>_+nuV+2o~RV~wE8oi3@dl`>BVMP!HzkfgCqB$@npe%OXyQ-`7}GE5GG;}
zAKv^pPq}C+85d7;6Wk1drcn?Y6gor*-9R?g@vJclHX+u&|IIkV5FuU0NJn2jD5e?@
z*dGYQk*JTCGias)bjd1vGK*mM7`lxbmPmWZXzU5lfq#J@L7~qAQsy)e{_S8Jh4F+L
zPC?OWm4=cedmcHMW9$L=|MxVV2uKQz24;W8r6A%hWRR)Qfk+aMq)u)I*;suKP;ARx
zR5YmRAy6@`{8M?x=Dmd0dC57)v&oO}mPvEt4gR}DNll;*u$%<kOd{n}<C04|T?S(c
zUM`0`*Go;P{C}V084U{fB-Q_v6jH^!`2Qa){t62egf14BVYZ|tH6;MJCKHrw3num}
z6_mubn~9diwtc<1>-IAH)>1sR320UM<hZI{#VKv*LmrB~Lw>J9@3I^9&=QyF9}KJ+
zz#B_Cpj`Wvj#2J!LUVj2%P^NbHp4y}JPA2vsa4_I1<Cg<bMJ6CFR9Hi4Su}^q6&EC
zlo0f%mdc|<gB|*EL~<78hMqY8C5pH7xMMl@AmHDmPbt+xI(-y!THhyFX00hux11&e
z-|_o8Ke8WN-(~o(xoB+fHVrp?n~g=^*Gac7URe^sl`9l5(ecTpe?BEK|4<_G)2TSR
zm1xyI2)!NVS*i$@3vZOqoa|j@C;EP|vCQa!zTj?||791a<cGx}Uu(*HI}z(CZKHri
zvnAa$u7jd8$P=Y$JosKpzT57#hzP;~IEkyFsy7;ruh8avQ#lfHpPDjWSM3@6uF-z`
zA?)P%sh0J<cJ_>oz&;+k@dJLV528cDK6OL+0q=G=e`EHmiR=6t#TC*YG6Ie{vSn0m
zR6z_*3L6NF4-!JcNKj2a4(#bK-%t6Df^NSqKFz32!_1GOE%`J)ZLP0lqj)S=9S<0-
zj*xnvV<=o21UMZpF+Tlk6uh4*Xp}NxY&k_d5L~tyj^0}Ko%HB3N#l|q3veUXKGg)}
zP?6m{kfJ!f(_Z17A5G{^dSgA^LY;2~WoIZn2X?iSIwmm$)$5NhKIa0e_cD)D@QJn?
ziHVL-UtEL_DXtmdiA8x9EybS;+<xokN4YFZk6ykA*Vjb1@l^#?o<>O98B*)$3|PC*
z+tLPzo}`GlXEk~5#5X5=AQic5&iRXzue+9naeLGF#Acl#Di2xhL?EaR7=HK1X6lEI
zxBuBlK|l^${hdC394=ZY%$WjJD!<jQnTGu;ttn{@;>Jt_IzybqH(Oa*UORk0BqMg2
zV%ctaQGVXnp47$LdCC#!&2t;C%8raijY?FHd{s}p6khV4H2$>bXG%l?`Tg!{Gc`ia
zFK&Rnyg`uS@X8J=S19Cjq?#lEXOgMVRs9|ouMUu6_8P4dsYR|{pDGS%eJU-sY5G9Q
zQ*oC$T|s@D=Ul)1^WSf;4FtDi^%)y^<3qIfW128IoC*NF7+1-se9^byv)#+U64(~W
zv^3#1YKl+P9FnM1o<vIyDHJ<aWSw^$b?S9hD+e$mAIIDU&t<X0KB?cWBR_=CI8d`Q
z*1#KM)*EQK-N#XRG>sX&weD+l=D8?tcQLZhsi956Gy{M~z6lfoZP}ZZr=6bfZPi)+
zOQ1V&-Zvo+Q;kO3Y(3J$v#cy6Ma&c-9yLhybBY0Q{4|1{<{0d{1e)FoPZgj(j}})o
zrY#@ANL}*hMAdh<t0libhe#-u$D=vLtGuEY=BW{9{K{!(V~0^^iUr1Kk>y&P4W)<<
z4fxq<D2mC0MLf$2f-T!^&a@RTAh=hNu~5yBwMdAtvgaW8dM@;f@~frTZ*A}vrzOHs
z)5!f%;``b#!0?z;mPcSJ%wv*_OIuXQHGCvtGZ;DuQwsP60I4~1=SC#gjXA3~h_Nt%
z9n|5r{dx2B<*z1I^7jf{7Wu-p`k5D9$F7@YrAdjrzQ1f-!X7Yl7tD<=6b%PFWvAY}
zZMZ+n)Ym>2KLJNEFZwG=3<HFa-!aPMl%|9U1T4%TVEGbge{FKhrgEeCyhYQ)ZNI6$
z8-u<Uh{>*-jnBi;S}mXMDEbZgxi|HGO`;aR4IIgr+DI;%ogonH{Kg{*pJAh`{og3~
zT|shEMick9cPVEh`6evYTcNN}sG5XoE8EooE2ONk^dsP~zk3VQ61m3=H`xE>cUr#p
zVUEWv#~U)>53WnjZ0uXy%jI5aYHTj)c^&G>3Dt%Bij={@l4+KvdO1)qijz$P84f_l
z$${%KX8DSac-&f-5G4c43R4>nL;0?~-ZnzuYx`+O4p_**l-^}->%{_PvgC)CUvac<
z2o=4}Pt@z%Z%hB_{Y&GYC$~O+v;#KW&^)JRnAvKh(W)JdNZ4uH>zxYCR}$)({E@Mz
ze{Btg;x3zZRFWI=Q@D8evRwHtC`$0E2W@&3<!8F?jW<2QTm;FKhquiKqb>U#X#MmB
zQ}McDRXe4>m5qu|BsDjhcDD}Wfkci#%TrZEZw0j#Ox=}L?9S7EW_18o;^tEM(yNyN
zca0W91k1joK`4uhIb-bs#ve+I{CR6_kQ;GiXEFAb-{05oS5=PMlXe19|LsDdd!^pj
z%ttz+&m1*sz*!g<8Fgr&_Z_;<_&%LgE&s5eHu+tOZI#1%Wgh6MbxRN}c-1nwvn%pp
z=X_E5z5k(zegEn_qw$|U1kFwawwp<^0KYSd3KKdqO(x0{$OQ~owdB&rMPE9;#2y9~
z9p}dj9$ZXT!4nQXMpg))&uJT7Qam1>wnoGa<%_yn5V^XFs&=H`rv}=$oZJ)IXs8-2
zvfs};eu10ui0QA^q2rs`JNz2p{lM|Q`>2~beDJI%h+D*;r{<|@rSO@Z1Z1b=h3Y*8
zbC#Q?Jr)HXy4@F!y`r{?+MIw!81Cc2#<{0q+oaUUByNXO6rEe@|F|%>5nX&0g|e!1
zR{gD%hvgV*N=Z9$Ic?9{)FyKlx6yZ79s2nFnm4ng9Zx1Fo7s)>25-S%m|U$o)gahU
z2RnM3Mraj5MUXx8dN?dF<g(JX9G4GE&M}>93Gj%un|-eLxog<j-3)8*n~IlhZ=DA^
zo(C=SfHK4)Vwu=>?IzA7?D#0(FZ!&|w?E$ab11Pf!RMeHXZpJ1jTyRDTo-j<ojp*)
zIPkYM@pDZ1mC>Yp)RD(-!2d({?K*4mqIX~{q@PJQz^nY`)!_GXYbR9(7J5j69F)uH
zfmuDazXkR%hDXW#r2DPBlv10A`RUZdSpV8KeSWFwyUwflkR{KJc7N*4sDCyKCRbG1
zceUROZYepirkiwH0VB=@Q1wxcv=dzS$;*AEopXM8`5&ppK4Cb(@YU_RGat-UNjLXN
z_XZVFTTKHsH535EQkH*{XAvJW-y%+r*E2pOMo8S~S9&?tUW4+n$qxDbkzZoFmSKo#
ztVL?9AbWt#Zu8BpfN2_K<esASqtZHF0li>Mz9GEjsq(zV=%4T?<CJyMDOO52Rhz8r
z22ayiAQ{&tC`7SrL&aW?0f?>@p=~;U&#wLe*(oTG4l@HaV6Ya)>~L5OlP&GNG_IL_
zU%Ot=2zmQd(;y_j!I)tX5cx7P5?)eMt@kQQiTx}b-rm<turq-KcJ139@8jK^pUquW
zvzKmHF9W?WIqft%8)@(SZBJ7OZ5vQ+X+EQ1m%@g+{mO*lrClp3yPy=5K8Da6(ZS~k
zXi}MzLlR$4Dmn}kO*B8WI4}Bg65xQDgOJZ#E?)Ndwcp$_c2zvHw+3zj>+JT*+Qaui
z1jv1{$&wV<Ua7zQ31{{RxVxZy%5)zEI9>V#z*Gi=bRmo7#B$C_i-8_lpPa9}C9SKG
zq|rtmIv(S)BPM!B4{VCQE4KHC(sr!vZ}PmK`gZC1i}n<#fO=PfeQ(t4{2?DUpQnv&
ztIRNAh()UO_S$&aD~i{u7IR+(ey%P9?F@q@VWuuy=HW^?jrOO%w(BiRVSmsl$N5Ae
zY?`x)5M6>0GI$*8KDc0}v^Jz!=G8d3SD64~)XJ6)3O%1|!Bwr{RYNRObE=2FN3trd
z-D^wQLYd@=5)70WD5t4jtKv7m#|UV)2Xa~Kh;E#1>{&-Ye?nB<9uE56$lqoyowE)!
z52?EzbSuH`voZZS-(R{+<z5C1VpMrCY%MPDUsZNi7m>dWq|hW(pEU^bfAa#Vwz5bQ
z69G*R66=Kw2R@7lec*`@(<4wJ%CRx8XV10|qV9T$zkQwmCh|hd2XG0O{U+~wB2F@6
zrCV6afi7U>!N?+2gf@(d1~67d^qikd<8HAA@JhYdwObW`gfoJBe7RUkzFfV2^Ly3h
zJU@OxevZq<<T5v0{AC}zUlmXw62%jsUyoPI#dLQ8sM<tV%5-dGd{VUeYAB3GdtZd~
zM2siQqMYl`=@$G%3?{wuG+`}f<=TV1id7SHbi2|{JkVC6_>LsBJZl-ckS%v_M?DX!
zhl3e(^l|n1``FCy4?HUFqf&ipXhq82;fE0a%}$7#LVdm~yjM-{#q%mm>R!+C-;=|U
zHpek&=;OSIbmg`nP!p=b!#DPJ?VS~S)4^WC{iw&I3bRtK`VjE`&g^CE_Dcmv&CFtF
z>x*A_lhYd?dkK@}(u~`5&ci2_eVCV*-2Sd9nOWWo`zytIK_0kC(~I`CPpofZXO6#}
zvVO_6oUmF9T&5d6;5BvGcCKw6G2{}x;Or0e^hXL2A^iH}x8@t^meOeP#OG)*WF)}q
zESFQXz1-w;^he11iQAN;g?=D+k&uS44FCBCR-T`~8MW0C?X!{f`g4iuR5E<sP9xbq
zh;yvuP@advL4ZH++;zh&ap&g9(=fx@+EiDT0Yrn!F)?Cn?$o0g1wyMb7GM4;aCQmW
zNY;#DE$_J;c;F`4<F(yuNBV|qFeUdSZe7VAn>F|CN@S&+i^4)BGw)#B|2R#gm?izT
z)1k-_29I!VJg@4N`~9O_ySU4H>YOM!8hzenrma4ffJ@y*S50!q%Vtrm>EC8osXiMC
zW_$m+7G{?MR6<Z)K6Ejb^D|4d)p~@jTOUvBFbWyJrjI!j<k!(=eo>;2@8Uh6bAOpu
z)Sz5`?;Oh5<aX-}1X}`Z&K`iLzUL&A$FMe66%*jvI6la^!ys;Xy4@2c)iJ`=5+0+@
zpFUb_9?@-_;==t>F6P7Xc&wKizik+2R=(}3T8T#g#p&pwUZgJjy5CpPnlG}P_E|-Q
zqP*L8oGbNTvZHmIjVIw>e(`M$Q3gsb$;*6LP9?>ii9F3LDD=Ch+7-|sv3Z=<{+(Z>
zS1vPZYp%-dCU?YPVX|FaTWr^;W?6|!Puj3<e|Ds+F!MnNSD(XpzAWVpKc>jVC(qJ8
z!W(@J`NbOJkmaWZlSXaQG)$-_?jA$ABi)-Dn_iT)I)8X0WnQ%8KW~+A5|7<7bm-dC
zJ<~{D`10IX12wR4y(g|5r=9J&5?SM8O!e3fIqJR{nB53SHuK-}&yQ2F_&wbZ{%$Y|
zWC@z2@xfL_GuVw-_+8?gK!DOaa_=C70m%DJH$Gsx+q%-%zS7Z9&z}21##ysAM}kq7
zq+=(3GilW5H(4`%oQMe=df%e|%;k}o&{p}dADJAn4nt0qZMMw@>qD5^E3TL{N!963
z-jTLjdE1A15!+k`{c>c<`*r@+ZK?y^g?iSe;;&q%+__v{e1EKQpsf(?W5H@WtUFJ*
z_Bb@N-fgVE&^M;pSZSg-NMxW)+q_nYY%;;t`?2BH$#<s!>=H!`-LnQ4&K0wL7`la_
z9s`d6;ccnQBKJ0*J^hC{NrIu)`kFV45Uv&-qx5YpES%#T-13!8IidOu0SEe<^OTR%
zDrBmP4UYG&%M{a5wjcM(EF9+S#0Ps9{~YnUKxQ{%%c4Txw2R1#e4i*#hsRT`#8^}G
zw*FJ&*QZK26@f}o?%4LU3hs&iGh0?cOLmvQFCj)oiYrR?Fm5jUZQpL6GD=#kW%se=
z#_64~8KuDkt#!G8tN@I^Wk9P*kp0Be9N0Nuq;KuApga<ko-8GNbLiec{j)>+o=d?!
zZ=Da%t?)#$=-rgO##J;RYd`8KeJstbf#iBdxb{Z#7<MTOA!3U8Ot}rN5mznHq*CUE
zqY{w(IMc5fT_kwd{an!oSC^GtN!IY2Hr_bh2ZOyH=#9}&NouS{5Q{lSGu-9iB){=k
zDly-rv8a(z4gArT^#Q)^(Q<9|(sitb?dl{(PRD<Xi@nW3MpnbwKF@&&$)}9}>=&cQ
zy1?!Jif@a&ch)nM<tNE?6QOw*NL5Za{hP`|+jH|l8<WA{a6s2Qk@qBUfNV0=jfBsl
zO^1^&6$CS0wDF6URsRGX2%Iq}J~ltvic^K}b@*JZT>r3h%=G@&0DF)GguC8}H)+E^
zxT<2?H(L-%4bA$ErG1@(u?dGPKiv<BUMgJh`l@(mRexr!3ui{{nWvqG0}u?jdcUk(
z3m_9I*-wJjF5RkFR$^?R^SNvTL?(Dx3(1+mHL!?<7+?_hU5*C5OP=!S2r`j!G)W>i
z5f=L*cNh!)DI_3?g%U$vE?;%P#q?nRV7f)3DRTXLxRX(?t8<1XO)I%HC37MO!~O|K
z<*OkTQj~ufdpjSk#{HVsj7P=FRHqZ%*y{>`57gwlz@FaAO$Zbits~=gxh|8g2nI}g
z;2rWbJ6YjEJxI9T#~UQbHDI+A;b<vgyhv%>MC0?l12|OQvNxR$T^!ytNBzG5%q8o^
zha;OWqZ@$frf5x~syO#_-fEZDXjLd-yufU*?Uw3RfjvFw`Zo_{_mmbAcy#F$_I8~V
z+8#y!tNHgI7y#Y2#7hT*fo0x?qo$c*O}UO2ixe}5^jb%#ztYe`IJIm$dGwk727HRL
z%3GZ`8qR*xrDuCxpDVys#dEuBLvFc~Vay&_&Xf7?5k?CaJTvrP`vj9eQ<caqIimcJ
znF3Mj9zT3L+Z9Dl(yYe;`nK^asD?p`zF>g^?wGNzx3uE?(}&)U?dkBnZdXR2jsAKK
z3Gycfkx(;z>@#_oydK&Pz<Ra3E1i`Y<}4D{lDfML%O9Viy6k*iTs3ZkNsC{9)LXUw
z?bP4we0*PmWG^&lb<ifVUG>^e#}2q1=a)qw0x(f%SL{P9%?@IJ;30eR=@~T~s*i0g
zd-*<n0~G}n49M*tAH930<pSVETEHav)uVm+je-;C*&J+V+3?)qeUOirP1n`(@77n-
z2<BOub02cL#J;GjRnnv>&#}x*bJbWJ75URG<U5hw*ZpcwDmbcq!NaF%%wqrAbg+tB
zP3Sxb)7h&NZYx^T(dKgmIb`oeA&@zG9tG?#&6i)-#Gt`f1_#ZnDo>SkQkh<IyrJ;+
z@G;|QcKpHa1|QjTE3iv)hJE)X&h>KM9;PgnMtzC0-_31Wr6c>fe|PlBM=Aj>g_i^1
zjAplXSDG+A2^Y5?^KUcL!?O1ULf+j;8cjRH+oXF*@+0;t&s}{=10|Itm@1N*U`*#_
zJ$cewgg^f=wdCjS>jpsUyIN{N3`Af<Qjj1h1nlY$-|>Cp%Y{^c3|iYuy$1Q}O+Xb=
zv&goYGLqK}crEOe#}&g-(IKYlrv#L*tVsQx*DGs7L#jfHSbeMSzjE_kUB4abDf#ht
zgilfH<Kw;qsgG@$_vDMVM~hKnX1f5KcfL#PF~Gvee6E5+XM_~r35)Ej4}aWGTDIGW
zqTYx)^kzG1cuRFGMT!m&fEZ8^keKt%jfA~uCNe9j31kKsP|<;=PtF_nV#0bs@2nNS
z)Ki}rUu^FVB<sQ#{G?g7<j<(lQNFkRxa-Ll%9SF$n+;Gw!viIC#NUt(cS|cZCkHT_
z<ZfD}$|<s$Pfe3(*UIkEf%4e%%VM5m5o=M^8@(L<f|Gx(<C=m}ny%LN@dmISsm_ok
z-#RdMO$>13d5&>&OYKKkNCQp;`~1iv%R_(jk?ALTH7lrTx5!4D=a;@B`oF-vcvrT9
z7&w6xCUct}3$C<+pWPMKKzi)r8AYXs;D9I;<bfy%s-g2S>Pw%t)+e!xo`I<Uf!6cH
zB$M}i`}_73^4CpFaiE$KIJ5c4#V>LuSpknS8Q{G@3k?;W5@J@sNN6N{@>5t*A-^>4
zFHUNgkJ;f=!0#11%qg>4kvZ(W*BNa`*Edi`jU}CCxj*edDCK|O@tAMgbR<x9a~|DW
zOLFBWi2&6UqR_ebAsx<em~-};m5+!RfcRhj*_d~17jn6@{a_it!4&IsJ9wc7x(qy=
zW`&qP00Ghe%`NP)LhL5x!?NvmUK<huS#bz`xox;rltc5jy{~ites`~33CL)~ApOr&
zf?Wo~U1C4XLL)#%$w*&b9&YvWXVC2HrEyk1i&Y718ZlKaQVV|{`=I3^g8kg9Ug6t6
zVxdXiR9#o|(C3s(59$2j(!!$kLPIrSE?!Y!ldj)W5o)VDq2~e1Q-A=d`YI1fVVywk
z5B@wDGFmXK28KO|uq>~6cHXT~%LPE6B%Sd5xP!Xo=~MlNMQ1sKCHk{Dilk#)l*sZ|
z_iqPbIj*n&nLSI<iUYOT?<%{?1&fW$60Qd`>RXhkFv}ic)#KC2S8hjk70I4Qg~Q2=
zod>aeX$Hm(B!yF|8acSx>KKuowRGh`-B>co)1JLuXg$O6%hSY`50G4bnaQJuNPddB
zcoVl*FjE=Z$lP2(H>RlBc=Y*o71^QB<;2)6o&`Z<B<#7evhTaG`*$!vKtTr?CQ>e9
zh4;N+KF7}fmJyfTpMT5k^UWd|MthAR9EkB&@NM%y&mGZQjHmzQdW^kwx8=9pDEn|K
z)wCbzhWWv8XVIkbsAyPRRTegdBbGtGG}&p9i;gW(IXw!i+tQT_y%N+{0G+YZpBd88
zS;xIoVl|IY!0in0_jujqJF?OqVDqRMQc8ZOg%-}EPjTK6tceQOwfU?3&M|;@4I}^d
z@cZ^H62Fea9fR2hympi;-INjhUzK@=7f>2mQSZ4yo*!^qHcQcJgG`@3^{(kPAw5hr
zjcgepQBA(2w$ky2A2X<N`4d#61jC+i!3wBP(sB#+yux4P=_C6wncKaboX|+Ws^Ja#
z)r<K1RL$t<feN8G`IIBHD%H>d6#qXZZ-!aTwLy&|K1$YEhhVXFJ8>6RdYHDNrZTdS
z$J!4agQg27wCD({lnK84LSK^)S|=lYUkZAam)x$+hf6%ZUQ?>!;WPFq5_w{k!RcVk
zD?5@x+^3!<tEBbLe9o@@@92ZE`92-Gm=3zFJ7WH)c3N&T&>?-<e^7?6g!-Of^GcUb
zwo`B952AHB(?am$&dCf%atzShc|1u6?J_XAE_xNxYMgpr*;TO(0=*ijDur9`TMjSK
z(8g3u)_gj&660|kC~1(m400Dt-jGkI!NWO**&(evrmV&PBDi5Yo{IPj!<<y9d>!eg
z$N#u=vG3RxI5GqJ7bqFD9pI2C6YJmtPAC#{)xzxfU4~;j%?5QkAkk#Ed|;%Ek<fvU
zXS|;n3ax<p`*ep<mTUDrq}WW-2U{Swt^_N|JW?I$w^WOBfzZ?tX*<>%i@1X+EQiQ~
zMGZZ|UQ9vDnU_k0DfvxG0%5u#R;qeJ5^FA61qwZ+5$^8>934ye^tsKmVv$JGlh=AY
zq6N0(w|6C}Q<q6k+Bv$Y*L!hSALmVI$3v1%YJx^@zN;{<rjs!y=%l_6<G@O0{Egh7
zam}ea!pIX5+*f1{gisEh(8AL>?6Th}<`hGtjslMUz8@Q(6dp$#R0|h;k?{xsxBz_&
z2mkQDO4#=<w8ml1s1^+AqC<qcA%qIPIO}rTtF#S}m8!_OSI((fwUoa%IS`;UM!!!7
zFDmr~+#dlks7m1eZ?*E17-*dFcvjV3e<<`}^-q_y?rjsnC`-o)yUulkofo`h5#i{Z
zsf<y%qTc-%?kkJ?9BqjvBMee+R@2{E$89{Fw~J^QR~*8;@|Vqj0Wx?X#W15^lV_7V
zEG~0DIb;~NfwW4aUV{i;PW}BFZ0~J{(nbH<y`n$_YuWiJhu`g8ABIW9)lW@p8)a`i
z*9jVFd83ZL4A&}e{GDvzKIs{9v$nfP0BZuGyx-{SE$Ew~LkJ1EOw-q*(iX1_@ZwZM
z52m)Mze8X6#;37H_UO?J0FLAhtRVcN+B=9qAdT=4Jw?6!{RXi|j-5n2Nhm$4)xq0u
zyjEeWbEuJU^n+|QqEzIp<Z!PUya1l_f<(~Irt7lx(qA=+pYmyz8gG8Q!Nm=3@>(!Y
zS}sBxvZR-XgEK2`&3Tp1KFVi)yS{k8fryt3YrIP;`2I3T`>4`}=^huejP*JF`F>(q
z+TX?T^`~Eyb=Ik}5MvSp<ca)Ys5wHU98CBSgHD_KWQbnu{rX#mS}ZTbS7pZ}f%<PY
z!c9<qN}O8RbT_W8KSCtl#`#S_40xgwKk~>X(1GlD$4j9}MS;uq`YRx>;ngeB`cG4(
zjal)t()b6ZwF5ReKFchnGAekn^!pViVP{$dN)eDP51zGO0&QFzgvL8@C-IR9y$7V=
zZr{X`$Qgvb>&35}ly_0+R34{;w><WnJk|+61=5oy4ZcUahXqy&nZAj$o~u!&#w3~z
zy9TFbWCTSl62^KYnGq%A{^Hi|H^g_*L!&yvHlCNCGS!Moi9*P*8Y!7*JT7jT!;}n=
zl?IE4#WxaXI{tMLpml}A${Em7YYW<nNitAYGj-@RNAHfAElImr+f!JuP||HU^`r#(
zQbL^3sg%bjCICq$F+M(Ch8ic8&4r}C8@7OrfVo~$kh%F+A5GxEnIc2?Sd6*cPza_F
zN1sUKryMk2=5MAO0`w3sWfPNCz>LBq;f>4AW??be;1!RTdGRz5NWv@F0T(;`CRCO$
z#Mc?+Y!&W)AHD>?XEP}&rJw_!7W@{|{%m1kK~#g)umvlXE$vk|H_OvUof-!$+!h8F
z6X8(S>+8Nxo~Q*cKu99$DM<Rxc6&4{J3D&~N6JuQh}nc-pe=ZSVa>?hn50%=fQ1P+
zHA%qcrYKhMZO)x}R0%sOK3=jZ;DRhPx!HlfXR4~I;Fph}ZcW4@(H6IhKzWFzXp?ez
zLp-(+_8$rP@l2PFUN54Mt^&8RekZ*K*ap~nDnz8D{(Dnpes`A^=H}rQ9CFUG>B<_X
z7d)e&8=1}rHCK>(m53dVUg%7}!P!oPSeVv&&7<KeQk`{Aa-i2MywitZDbqXuQqz7j
z$-Pe(R5uh-`aT2=EWMhV+S=+WOn#HgW@H0()Un(ROUDm<E2E=UN0)<`#Ny;9Ke+m7
z!Vk1Ze)$fE86zw>vFd}o-J3^Tc>=wz-IFykHjQjK+N101H}-4h_$eLjQR=W^tfKM^
zpdVahQfH_5%a>8Wk6T+?|7y={_}jp+nJPVA%gN;1N|l&?13j)V8fJ7wmnHdrxOHt1
zt1p4uv!29p>3lPjIRQsh^W&$H?W%nHzqk(;M-D^ql=%TT$WU^}A4%d!&>~|Kj&y#0
zegHnV`QEs=yVu?8>;L>WT=9J2cSFWeX0%Y7Eix{%pUc89)PS@YkMTOB<75_3RWe%-
zye<31%`L**t^p<)gieu!*GO;K*}>}Yx$S9qdh+J5-nkAp6y0xH8P%t|o&J3PvEfrZ
z^xxWPeM<rHr!4}8lW_*6h+iCE{JJSO1Enn~<uA1j&GtZvggD^uGLe%T|M{weghBHC
z#}B|z#YQ%7G&zlxCH}FhmD)q7xIooSvnk)GlBBAlrB6E)Ea74Bmej{`BH3sEA68EL
zgWnp$-H%eHh>T-nJP=Mp(bE{f$7yQ1Z4M@bje2}}IXXNH9~fc9V>*$Z*?JWdpy%Xp
zfshpkzuD?1#b5HT$jJSxg7U{pWoFaw^2zrLb=oj|w-k-XV-(rF^!`8*mkcZm;9Pw(
zY}Hj%9GRZxqNSzf;^JatjOsD%Xt!~$#=|U+lFXm7zvNS#UnVr#X+JD(w4$z3@H+Zq
zMy6LiKN4j~pY5=4d@XMA2iJH1>zYg3i>g<4UqE0RKRY+~9&p3@<0xFWhWP*u1-xl$
zdKyqxD=RA#6B2&u9gi)1v@-nlW$=1amiX(1_Jx1CtM|XrFoGr35d9)yr^?+_zur*n
zbeiDLsJir4ufj&uC$v!8>@2aLs;vm9+BI?)7R}X1=<<UCOeOPVb7^$n=?eK>-k+z}
z>eav3tG1oqU{DX(pR(C3*Nn{(1QZp9b?y6cS)FnWC;1yg?15HIhkLe*>lWNEfzHu5
z{ykc1>3sI_rwcp+S?W?{@~;DrB@_r8Kj}8ODfp}E^n~h`t-OS$hgi^6s)5T6s?vH2
z@~f5gn0~l~0B2vy@*fG9pj40|*5@q3`zkvihDhKsCgi<fMH)wL9W{PR2K>0sx5mej
zZYm_h+1?s?2Dqfeur>uoT~+bGIJH#n2xw>H7w0=B;#DSL3#~*tP@=80IjRzyb<D}n
zS<jR#{pKS*i$X+jG^kl@-GG}o2Rc7HnG<RDqxe7jsCuI_TAixPYNcJoh%MfYILC}-
zd#-q8=#(h!H*^OVc}~F$qO`tC8pQ8lB`RXk#NiDZVo85mK=Cj;UYTZaO%5z=ap7{b
zR8eAp6A4p{(#K17!Rcc{R#0g6y;}WLHd!U&OOTu{O{{|3SsTf+(^Q}C=RFa)UHuhj
zP`@x<cGLfO3^E-%9M%@eQDKv)#*~kby7CfH8PACP<CFX-^vesNR6$EAq&?Jcx|Fcl
z$7+aDY^Ad0kA3wiGdcAlz`*J7*2i&>$zscZx{o1Uo-rwqIX6CN=KY3}Qt0of5X+Mg
zq)H-f{aX6kxMH9`G#4?(>}}>yQDZURwr;i1X-e-}(ML&bu~Ub{TugFM|9IIAH^#8z
z-J{yeh~hJ>mW{X}wUw;CHvv*+?O)aMD^y6=cwsB!T7KTjHjk)hsWl`uJ{P}-_~vdp
zD;l)8lkquU|AQW-Q4cQIB9F0n?_=L}a~G=lCg>!c48E8(J*~BF4ZoM#FU^f|QD9lJ
z%ijA^uK(IjUU!_YFM8i$ZnR8)Q;@bqFt;=gRV`{L=FOt!6{(>6s;wk{A_rj^1rVgD
zi+}<_3V{-**9V^dqV99h?O|rAfxJfF!LP+J0a|4SYR*ndi^>S%cnEm-l(d1GUvHM|
zAL_E|b8J;CDY&RBH-fn;cYa8C7w}PKQ~gFZt(+(#?!i;B<dWTiiU-qwW(-cHkCONp
z{uR_7K4GWFdni9lCul-$xj!fA$SA&0h1L%FamelT#QY$RSx5drHJh6G*5sv9`t*5t
zKV$oLR_!$XjR78An`k3OKLap98@Rsti5NMp4If6Da{uPQhxfzl=CLWDxZlzczN~7p
zWcGa%(8D#vQ;UZw#{iNypCZ*&Hgu=o5FVBSmdgd3XTGszx`&jkT!FoW4MHa}fk<`%
zUJMk7L5K&6LPbt2oDB7aFV*CbB2n|iht}J;t4X-M?a7&AI9`^87K*P5Id19nMf2_s
z!RcF*YF`VI&t!EJ440#Nn|R4c1)3=oLuDfM#4#+s`6gVRkdkeT;l7f#FH8aj6>2E{
zCkz?x|K{ATJzXfW)il*tSRXW-BDS;Mz82FECN63oQFG5IzP3zJv-DFE_t$aa@8e|y
z5Y-}<=1<VUf<>@Gf|x4oV`Aq!->6UG>qHn91o`%0ZRKQ`33!p9kJHUn5~4&vAt}li
z<Nko`%g~WC*Q`*b<lle`2?4hr8rKT!R$8lDpn7IdbqzMWPi5n{_f{D|-|n4nQEmt|
zU8?)vwr+ftxcx+@3IszKT(K?SpdlJc8U%O8=`Y`IGl!2c7Y>UH)0N3sztUzq_lK_(
z-LqP5EOzn8C|S5N`LmJjKdx|?P4}m(PDI~h{ec;GS-mO?K<3=C(25J3l8Qc}K1h$v
zmCObB>_HQ}m%gTnk2~X&e7qb*r6-4og$>lM&}<u;7^f#s&xoX0537<XMg-l~BvY8y
z?}Cjoak@-X&ar^)6H<Hq4b6eGndB(=GdR_Sa%-+pf65MLeeN|hXfwc4;oXQ#J@ry)
zIY6O|cr8)G9&#+9Rm3f^W3poM6l7wm6q>M#$<mVgIgfA;xN0C8mL<|x4LZNNhPi5W
z{XlhEFze>%w5g@xP}VYXlt^ILXBSre`oP6TQWfmu>3Fq%GC|)BiAo-JKF=-9<Zc)E
zzj^;~>vog8yYg{+vrCAP^2^5-g8@%~wi<zPdBQ!ts3mGma%=Ta_!AJxxPb-6$Ka=b
ztuj-VnzW4XTlxA7>gCg<KRvQGG-UT2@`TnsS+2v#{pB<3cAM&8bI_0*r)snWlc_x)
zftL;N(lHsFJyShVEybvRnOfQ+KXu^hjH8ychMy=@?f$I)i6Ci_uJU0jbhUS-2ybSe
zEiE$~5<?ehnc9wF(MO0p;fYUIKu7qpI$j*$CLz9ovm~QabFQ%l4X64u$JtGPdIT^r
zavF~%rA7E(j(sovs6{^&J8X24LOV^xD8`4+W0g-hePxag8s?bB|7JmH$*5bTrU~ll
zdrKP!-~{nf>hF<nU^_X{6&;0CMSLDc0~4+8(1@lyR&pt*qyQ1Ge)7oRqmZ<6ulagd
zVNfAup7bkaRyBx74Bj_-w!5LL>{31sLVV9f!8YRzY`<pN8cP|{12btTaWFZaiUxA>
zto`TEz=0hUTyVL3xJzMfe{On`vvt(=0NNJjGlg*S=8A@yAA0pgBmSE<XL%-8I@@<R
z&vT|~*c>oGzmNjUAZJ_si-y{J>psMk)h~tEbvvWkTRlZhDOUv9>ZHiI`NO*s#^B~Y
zCbukqoJy6r%ejwD6u7EI{{O&rNG`mYoWqk+&<6H7LtKah#O`{PNVA$h;8(~}GPgbX
z-#toDoGT0heM?u=t3XWkSFypUCa0{8<}AM>YxD6{EvLhJ2{pw7f3Z|w=ezd0)!Vnf
zS`a4`1LdFK#3{HxR=?ENeLRM<m@xu=T<G^yF`Aar4z#n+e;A=^eUVrn)X%ObbuQX`
zF(q33m0FdKKo`4o;km4CCublhdUh`SrB*k!?jBuHAUdlMK9)kN5GE3}Ea7^glc9FY
z7s%U~O8X6O5Z5jVG^j~I=p4|NR5N~bajSXRYD2Uo88xQv{siZQv##jPI>vkdj$HjL
zT^Q&6F<-o#I)*N&t=w#Tx!rat^ScI6Oz<ET#E~&m0Z(!4Jga|AC($QbHiLAmZkiWo
z*dA46F5i<L$ybEmH+}p;xG)f$A0yHEsq!wzHz=xZyMG=Uzfo9xoLI|JH6=AlQ~4d|
z7Yh|%#kT8}FrqE`M*p$s*FwNDGT%`ecKpn7*df*?{PTX@5PvL>$<V8}fy@8G>sMTz
zhy<R|xe{UAyr8nZ4pZ!mV0V!ys=<B}d8LnU4Z5BIvK^gBtSY8y&-bC~0`XU<c=)Fl
zzUw>dw2C(4Yj*}=u{V7sDq9c=S!gF>J7$Onzhj2;DMq5XHli{%&&i7Y@^HewgN_55
zp?Dw`a|kCnIEO=QkqA}M!{96)Ye<v50`vl!?k`#&=4jM;@+kro^XE20RY@3iO={Li
za$)u6$D@9B%>wbR-#VQFVXPZZdHUg?^Hc)6)L7?qoj|{cw$^KAGT)6^N_05`;fV0W
z!?Pj4Lp<P?uU7o(*=;1zu@ZrU3Loi0W3Lw9&t0|?Epe;%<}Y@}4;2P|*-&z7dI<5^
zzqt9pjw<}1EV!J*2kBt7BxTrO7Y31S#BRC*^Lm{<i`1{zu~#~!bYgrKz|t85>_2lo
z-O%MPg;}iEo_auv1P38&oOmJVRnQB_*m%fgk$Zk<+h5H>RTbbk-**hxJP?%mjHn4t
z;7Mrs>l8_j5st29^y2@e#pf9<h{2{og3^Er)d|t^{L@i-XN4`Pt?B*l7S3$?Pde_u
zBdeuiFS#=n7vPm00zLLqbzJ7D?!vjc_(Cc~rN`p{K~}C8bRG!QfKHRMdE#&Pzvq;h
zi0W64bCeHEQCLU){Ej_kpE}d6-${8ruQW~yz&!tXnOr`q#c5uaZQLQ<!Ftxrn}n#%
zvmFdiyd|RQRLsXGCUTD|uX6VuI4@^uGD;wI!~uSYUPxWu=ilhj;^MpS{PI}i8$D;L
zW7cdOkEG)}QB}Zru9JmUNz84Xgs8}_Yc9aN`6XX~rSrUA^bi|zbk#uPKRYx_m%}J#
z(<d7LHcYWO&1V*Cx!X(BEe9|kb&6tr(~mK<+lm%1hl_`ySm_`Q@!ww>xDV(*hFYd-
z^x2iDHBgRmE-~Dl&#{_~tU>tkZ(y3}KU$|{Bh{p|7iD=2A`f<0NmHD{f|=>p*C?Ug
z%gJ%L=G!hN&P+c&?B;C^eo`akW7kUMxOV;g*;hiRoE^wKYwb;B-)&3P?t9jdfzQ%^
zyKU2(;kS@OOZPhp>P!;oXeq0rMC|+{+K{?CDj&-W4$LDkk}^D@*ULtflt5lHEk|+N
zlejl4GdI8~yl#UM<JIb#v*z|D4B-Iu6*0q`f^s_co6V)u4<@W1R71xzTer%Bk3w^R
z&l+VYKoiB{0}5ki!2L-|QpT&0n-&o+8d>>yHefV@#E6z?u=pkYsjINKTy;NvlkwlE
z+4VXbaI#F}zr#4f#gd}~EU33XqSQqVHQMnyUf}Q>3+K4db~08PLhUDURbBg~i=N^Q
z%1@=deEM{i$S&BZik35EtumgoUslt#VD;Pwzp^I0qxRex0{r(~z$xXBr~Qo@qV*%c
z&wq^!!`BWR!V?mhB`Jb<SFjnRlq8dr)1J1HkaPrjCGW1^8N{h4`F#1VPft(Sxv;+e
zA)@h9>8wtVR&3A!bR#kp2j6}!2jP3MU~EMH`b-@$YCmMrvEyA?#&v%zG3~xj9pZM9
z(D{JQw?sDSw}-_(+)G!;s({$bAteljI$5WU7BIUP=?kMBb4Rudj-{0{MwBKrQug5;
zUel>wwYcs%`3G7$En(Y*x)TXR8jQwWkcu=H=7UC?GYd6LIH(e77PI0MkpR-`1#DD|
z`@7qn()yOp$T68r$KH|6ElC4eN&G}s&#j#1j*l1*DsL2U;|%tam)Pp7OPPO&2MV=b
zr@epGkFo@(kw1sO?VOvBg#f@>vke^$q^D_NA=w-(%|e0oUbVOR->CER2~ie_1?6>B
z(Z@J|h(<Uo)`HW}=@=wA8Iwpb-shELtQYFFTNSk28|!5bUI-_sQ{w>aR5$bP*NO8m
z_X#dU##Pw<{>2ok+Sz6rx%9vBi{3E+eXwo3LztndgWiGc-IgRNJp7g)Mp_JB%>}>y
zO~|vPWBR$~l|>GuJ{U^Fmx%zwMwOG#FQ~u02aCH{n?3j8vkE<AE2(YvKqk9iIy_Qa
zgJ3(u<bn57&<T3G>tv5yBK`%oCPaB7zIPJUWahpxJQtM*(Xq;FkY&<*JuQeAV<(SY
z5<gA-l0V6BfX}sxZ#_YtKqq#j5BDaH_CGTlQHq)-7~uN%&aMr<*aNm@sn3FDv+FB|
zVPmQZ$($0*r_yp%f!dwyNPNC`w4gxZ@pnA0)gh;B#7NbyBK-~&|6oeC`TW<v8cqgA
zND7q<Jr)k-4<#mk+pM~2darf_3DyKEJ~Xi}Kz5dxiY`B2>Re2Pg^2-TR!neBM5k|H
z?P9$Xz*9mlsi~!w9sdiXx<LJSDz@?T=?u;SngzxGGSt4GT)QjupvarS5d~XH`AG-S
zA9bX%ArJ04LW{oVjJ8w9R~vDRJkqs~v5)Cn9IrGLdGi!^HS!YT(mArt-`rC3=*;l)
zB)#nPU!Ttg$ipw>Vx43!*gppZj2!hN?-eyGUK1tD+IKkAPR$$6P21ms{!mP9_?0^;
z(w91YXm(#TvghJ^D`pF1T4Dg~0E|H5C2_?E+|*s_7UcfQ?XwVU$!by60U|Hq^n3_V
zL@v`j$~u!{4J0kPMMU5)DS>H}|AEU*M7=BR5?=C>8?Q`X703zj)J_E&+LIcs#0`f~
z$_!E09cS}Qa+jrr&g@YRg8oU*4nj;bUv4$BQ|F$X_5z0B`$g;~^3L=UHOfB(W(;1c
zY8j^?s_jr|cZsYU^ok;U)CgRJ)%PtZW6Q>(WLMO-w^3<!4%d?OASxJmg_q0;nxvv-
z_lI=M-)H;J@lmPfTfr)qB5f@>jkzdQS{n%?;o<aQx5TdTmN<3LCpBKc{6Vy<Q-n;Q
z4V(>?H*gSsu*MpJ!S*grrN--DF*SjYm-w%T0XaTt-2upA_tEFuM_0Q0wD<hi5vA?O
z<|8Vxtvi`IyKioQDqMl}DDuz#$NEl|W^=B87q!MC6jzk|ndl@p<fnd|*>z#$AQAl^
z-*MbnnyKzpaSmYuS;s5D`}Y^zwSqFjYA>EyD*0hoY>h;lYr4$Ob&KaoIV~k^vrE69
z&{u2x0o$={W(#{R+m>DgWITLE)Om<$dtAFq#0Aey#Gmkzy29GZ*{iUnuzFzs$Dw|t
zg!u3idY5U@fgZi^E9C{;XupGP>;vDcss8BpGfRvL`OL78+TnG~WzEngij}F32xm)!
zqnr0jpa5Q%^WAHOp!_YH9@g=^H`SIq(5=1)BaQDaPVW{s!pH|f8Sa04d|_#3LL=bD
z|K!_L)<w1U@3-n|rv9xg9(aw1LS&=NDr`HFR)No>{Pr+N9kDU=OV>)oq0d_&VrgmJ
zJvtTcw=#lerTGsMJ<RA~b=)!`ACzu^f&jbbVgN5@3mSAkd&_!l9l`nCW8>hFMjON=
z9ZhG8Koq7^>)M0OMkydN0m}f{N%^x!S10nC+=gMLLSiW~8+vDzd~1=Mu$Z7JzN&cj
z6@X&}M4ugLFNQ@?^BOH2yUxaV&F$H8yX1GIweAN$9vg;i2%V1jy&2^G%>40U=2$Bb
zIY{GqjNtu>IN};o><B|k)V>Z3K^RsbCEHsyE<c50*5})zr>0@(W$bPvNh%U)fMJBI
ztzL90`6mKOcQARCR`hLJxoJNcH%Z-@5fwz9AnKpI+y&56_Vs2bH6Dz0p0V>nUKlz0
z^GKf~rh21M+<>hQ7isP#B6I6>vOuh&i%b%13fF0xM5ox-!5$;bJ{78`mJ+wv`<%56
zu=GKt3Z^eH-f{er1&GUXpHnVeus4bHN|xc;=t{`u>Fdi(H+V66U?l)L|Ek5qcz7!O
zz1P<@l6G5_ab<WoA<TXnCH|`sCclkTp9z+S7WL3Cpegm{x&eip@W)87w&b<3VX380
zLC8Teukn(j#Z+bCaQl7$<1YUJ$J+q&=CBmAN10kcF%dnV7hG^RF4`INRpowOm55>$
zo`9oC1db;8XH~JV50g|r)$aEq9$g|LYSZ^|R|q9&rXr+4<fq`%{xkZG-CY<yi!T&j
z^NI;b%XgC6Mr4;>uSZi_QgOh4fzp%z?l@$Cx2?~@{+<Te(e});-<r*nj}B-KTpmC?
zT~g`UR)Kc{Z9w>Q(>zyHk1$!e<&3-`Hyqd+xs$(o@bLlvJ=o<q_$a|wVyp=g`V1lD
z;hFz?K0%ThIbZa@bCz3*RS2|G|8sJ|#pyqgVJ~NK>F5~SIs0ej*pW%0b+->2uWKAU
zd%_6P@J&<ieM{gJglDZ2WrvheNB#Y#l1U_7oKxc0hmeU@a!HjmMiqHJPK!fvbVZ9Z
z9EKl${@Dlug*B-E<3%RZ@|!sO=tp@Zei43IL=oh^WmfeKnC1}@j>R7F21b2&!$ZZZ
zZ3tT!{a#Hh>2)v>RYZqHPNq2q+A&W7c5+nha+hrgY+kAuiL<s_>zN_xZS?^5aCcm$
z2ce%*^fV2jBMG>APXJBV0iroj*-DHlrQu+!aR^+bRY@e-nOL|SH4c8YJQ~Xl3F(Q^
zk(*VrMT5GnS2`A=iX;s^;g@Cns>^QP#fp~-qBiGjmbQ2mTgpwg^#Avv#F27-Rjz7I
z?sD+B-V6g_s)41SHpRbd$YPU`xI5R4cToK8W*S23;QOG2-TdQ6no&$CvXZj@_B|w_
zYvc9{_0+wIHahJ8&Ic8vYAgR5!FFPNhL*KEPn;<>mZn!ad15byhumr^DiaxE4p9lx
z|0(Ru<DuNU04^pm*2t2C7|cwCCX^-al)*%pY}chy=vJ0)mh9UIV^4_4nq@4>MY=+a
z?b?!Ly{<J%5tFi1gZB)#`);4l`_5m#&*xd@{FZZ`^PK1WzGr#YKBuX5T*67WE5TM-
zzbZx|d+I0B6S|1V_F#|G19?2(Z}(~~M*yjF<ovtON^(!K_NYss&xY;J$q9d>IX=l^
zhZFqtNbv}pW%XQZ84meDrSY6c&NVbP=V~Vr;}!VavRuLWdpA=cm0cgr%IVNt;T`JG
zb2vTCMSrg@1(_;e78^(M!^H75jb(f^jB=n&#Z?WZj&V}6sS+;bF)R9=ooHOWB-$!j
z6^%t)1+cw+bji*2cj>F$`nDClT<r<#`ue2(;zj<>zg|^%5R_z*B9WY&Y{ea7Y}8X+
zP0d(zH{@vRBDTiDM(?N0HH*`{$Lg2Ri8hRS9^&$DdM)yS2_lzDA?iKkaKYa*K_fkh
zbrl_~SQU|^1*=CT&3;!V&d<~A%s9w>yFsPnv#D8ApRwgZ?FJlOKHd%YY7r^YMHQyp
zjp7LjC(E;qq6@RB6OL)}2fhrq1Y1OiXln?_nSUP#<{J8(>ltuaxy*=i;+%7x$_+7f
zf?4J#0%`x=Qa4v}#@F-mKN5M<?La?7hp+stPvt^FSyf}$4w)EhllS@iPTl{J^fPH#
zlk<f3D|#PdJv`0XDKk8;rF;n&`A{-E<9?R0!U#Y%{?bMWE79L1>iO85CmtEc%dpDL
z$zeZ+&u(IDU7ONPGpT<h7^uMa*zNH(w2OQ{!T;g4oxAaw9u3^o^WebEVo8wh(*1>Y
zPM_uj)TSnPTkdGoE|Z`Z<JdtN^VQA%_XezNv2os9#nQ>w48Qj_rFBoz(%)tPOg4mL
z@b+AFS%w~qw2KdZUu#Cix8(YbRXkyL6Fxlbzvjh%BAv+++P0iP8)&<WGJ01z{OOtF
z%SKsLBo+1MlSEyB#?7fjD=y)8R>Sh;-qRDzTps3YCj1ls@KQBZndiCOi<A?dAIp!u
zsj95K<jMp>ZJ9ZU!%&>EDIG)YcC_*&F3XYuC~$4t-+dOa*i3V|C0|r7)$m-c@I#vu
zb_-+?pT*Z!gv59i=DQud<3l}Asq)Z0{oSYZ@FccfeARx7q~kK-xRNmD%IqKcExDlX
zc9<;oSB1dG2Vg_`?VQxdwl60xHigHEubX^2<HWmtE<XZ)n_bs+F!OHqa2e^r&t2A8
zj#v3SVAmwG7YMwVD3L^)&eP!Y*%<Y0fB$%3EEp5&*Upq9REmSFth9$DJ1w-R=Cia~
z8dx%1*)JoVK@&R73q&z20?(ft3@O|k`9GJKj=r3UbE%0H=QDUXY6W4tt%I;TAn4Q&
zLybbkzKs4k#zWu27S^p~EDo}iKAUMFRn%tE(XMm1UF-6G`U|!*`~P_`p;eG@xs3d7
zFNQF4XBlNg)bVi<l?lK5ds2xdA51SlBHj_^pDabO3EAq&v>1~b!67{es}cUZntiJS
zZ7N%$=#)ZO8cud)nw<Z>G!;uoviiCMj^OQii?O5#M^SF}5Lz3sZJ3StIjG_V4Vgi3
z)1jeT8p5oW1GO_Cph$y8HE$#`1RAdUp+XKcrq$8dts8j!zZ<+2Bqb%@m1g#%3o1Lb
zq~=gM{-g<QNfK*mOZm}sTYez*{SW#7U)M33tpmJq=WET_xeqy^$2G`O9k3E7y&dhV
zep1WHexUbd%%NN3=$y!-v9_ahUd&?kj+ZK`1-18UKaTarE@kB13a;jv$(!NvH$B+l
zYXRA+NGLNKrs<Wh5a6~X_P9V14VP5k{8qZ=Fs%}^a?iV@6ca%7eqN_%j9IjTxFkjs
z-o?l{0`<|g>;wYLO&g?-a^p!67j(}<Dkw~$J-8CExSJJa9sv>px?7GQ$bz>&l^hRK
z+6J>~1^RaYjs4i!1~YGOVn#Ux;h2^A;nx!rw?AK%mX^Msp8g_TLtdiNXBp7uHa0fR
z71}X1HQxZ$+WcA!4LltLf#Ch(EO_EzDUor?IvxNUZEMrl%3PeC^_m-aI45amXP0pQ
zKKD~H^2j`ZGH=|tL5~QwY8J&Q8JL?Vi4K?m(hh{^@VyIfmxTbN@XD1h>g_%2?EC>h
z-U)<R>eABE{QUgN!YCJfoS|POSzeF+@EEEWHAe{2*(xh5OUx$eulolF!*y4Hh^xdI
z;$OtU7>XcV6HwdT51eyJO)nPz72s^P<QcoIE}ZlA^*wc}>49FH#PMH#(9_dnj0T7;
zRaiE)6;OeJ%$|v`wjMdH%?SwDz;7#u88KdOyZ5KulBBGqZ5|R)u#pljBr?b9sxk?m
zq2-^Nnu>IN3YcI>w!J1>1W4tR2FU!tmHB(Wp?0qg1JfTRCs$NhNJGt@mg3JXC~))_
z9*Px=>O;9o7kSJL)CH`IRVf0q5vZvE3ZI&uMz~!Z;Ta!=L1r)*zuCCx>r>WzHv%&>
zz0VdKozo)htUvCqp58QB>-F)Z&7C$N*<MsYJPT?E-f@HcfrAGP7&5o@^R(NG;M;~h
z5grOK2DbUp(a~>TWHzB7q?Pl>lk1t))oKj0_4V}@*@%WiL#}u|f;+bqRx6V)&Et|w
zx{P+s25>faiiPo%=F54cJ#!o^ot?d6a(qZE5gz|ldb+y0Q@cH4{35SGeh#ETI5;@i
zUb;b)Zf$L)YXHEORZ<l8yU^qh$jODJSifbiw%(K+vIUEJ(a_M3fL&tBK|Z52yDNa!
z@R=?sY^nXGEnWtX#{<iDTDSIH1X8ue!os4ZrA0zQVybtOEpS!oUKf>$zI4GNL8Koj
zqj>Gm&`>oswH~2|Ru#IMcFXT!D4JHgFa7H7?oNR(S4=I5k;4w9Jguo|dUX5o<7L1K
zXPpG*{LItpYFbqox0C&o>*Gd;`h%b>6Fyc_n>RH)GX=Pu({DHfx_hqa=;+wg`6N-5
zPmu6^eSNkgCm+0=s6Z+Bb2~w~X~1mv{6w=)5D`M$LM&h~m?KI`LG`xH{r$z0ib-8v
zhO>%~F4P-AHWJe|#Ai?^2Af)1cB(3PYPXJ!WzcLJBsb`3r?_Qwb~Zsq5v%Ma7nk$J
zb#GM+1^N47CH_F7i?BF(0{NLD&A?2|&vVzi!|>)#q~%g4*aXGIko!`B`Wt0XYzaHs
zj;X2b_2F7_%j0Ik)klCV+}Y9b!^pm4E~rR4v8<w^c!-gjmXL|qBLtc7pkz>EEG0di
z2TPV?5fB&WE+sq1c=0cMEv}<Vfog=7?g_X8sW?(L9`i#-hPd+z3Iq#C{5wcV$;t9^
za_1S!)=yl7;;2^Br_JbZF4PC`@W}ezZBu3K;u+a5DJi+u&JWrO6Vy`}RsSI__~~@)
zY^sh6VP`NOi=TjP2-o=WFT|iybs3r0P@|w(l!JoQ;-o*xS}UM$_N^@O!3TT;u}`8x
zvS4H=NHLDt(-&W{1!`RXqmmN~?Do#*&qL1~b)rNZ;pb1%M`1CTU<C`@oj^3Gl`NLS
z>E=B{ImQlm#@dY>*0jqKajR1;Q2={q%ETJXQl&R_{hM85V<W{sL?uo$3i&KKD2Qdv
zjelc9W@DqLzdy8FAo5Off0Hpc2-S(+z%aCv70wAjcbS?G?|Zk^Dc4@N#sF&iz=rM^
z(597zfw?Nq8-85mo?*hD%e#UXXq}y0RXA9}N}R5&*J+nm0ma_TLcz^{4QaSO0LeWK
zztf3Bb-u^^1pXWyoqQ7Ey3D%p6$1WSkAgnD8ULE8`oV)-0W-Eyev^%%VCg=8c3+xM
zi_gav@$6Ft0CZgEjGVlDQ~vP4fUwqO&=as(B5@{SB)plq`B5A$>moUh_T|&t$)8yh
znmwL@iD#}vrua`Iu7Pz{P*5oIh8(b)9v-3N>Q@v|Px@}>W3e2{#yux#p8j8!X25z$
z#CgInw!29s>6xXeI=i~Ml2^pE*!kDY0h}E&uJv&~4o`x#(op!Ca_YTY5fl_?tP)l2
z0nHp99`^S3270-)F)OO~7U8k)stdrI*A^y@vX5L5i_wj*Tw>0<ywQCr1*)_bXOGKS
zEl3=OlI3*$Q)VC`EaP#&;6rz0H~aGJo5Fa?A|N;F7iVKav2|)RRm!@`Df`vCvK;~(
zh%z2^kN1gQt(&iI5q4U?g&GuY)qj?k<AxSy=jKd}Y)vId_=#k+mX<`o9y*Z{6O*JF
zL+02v6TSRNe)*UP<i?(!moGt;gD+?vKK$^7v5HG$4tajGM&u7nyguSN!rN7KZ=Qe<
z<G|Y2<+-V;OKYeGy{S(JXGnjKta=nu)fhO?tougmd_5uUG4Zg9JwTtKBO8oGsA&WS
z+!--63oQP@M;A8z#&OECfQJsV>M4mb@Q7gm@i_AqXl;Z1MX;=yfdZh+?3uTI8b#Oj
zet%|r2%A43Xc79aKz4zO(O*jvs_-zK1vD0i#_HS!uNJ~B4BQg9GJBcI1knI^{s-*!
z=824i%GU9+|LH&6@w{xYD^A>;y^<#yGhWm6uf7DoECk*Ez80UDD6e>-&}Bs7h33ni
zoZ7eS)7>GzCnhHxt90i<FIt_}9PG`mt*rex`^8|}yV-IpWiZ$t!EJX@@!vho)DA++
z<S*#gOg~dc?k|5cOu7g(6vlYl4OP^SM1DpI3n^P9-hkV2NJB#V_uEdf2yNe2MoUB}
txzm|FM#w=N&LOuUh@{=$)_=2E*kqAV>Wv<iEYD+sACX|BQ+m>t{13q)j6eVY
literal 0
HcmV?d00001
diff --git a/doc/guides/rel_notes/release_20_05.rst b/doc/guides/rel_notes/release_20_05.rst
index 000bbf501..f404ee7fd 100644
--- a/doc/guides/rel_notes/release_20_05.rst
+++ b/doc/guides/rel_notes/release_20_05.rst
@@ -62,6 +62,11 @@ New Features
* Added support for matching on IPv4 Time To Live and IPv6 Hop Limit.
+* **Updated the Intel ice driver.**
+
+ Updated the Intel ice driver with new features and improvements, including:
+
+ * Added support for DCF (Device Config Function) feature.
Removed Items
-------------
diff --git a/drivers/common/Makefile b/drivers/common/Makefile
index 96bd7ac6e..df2e840cf 100644
--- a/drivers/common/Makefile
+++ b/drivers/common/Makefile
@@ -31,6 +31,7 @@ DIRS-y += dpaax
endif
IAVF-y := $(CONFIG_RTE_LIBRTE_IAVF_PMD)
+IAVF-y += $(CONFIG_RTE_LIBRTE_ICE_PMD)
ifneq (,$(findstring y,$(IAVF-y)))
DIRS-y += iavf
endif
diff --git a/drivers/net/ice/Makefile b/drivers/net/ice/Makefile
index e22c34287..f493c9ed7 100644
--- a/drivers/net/ice/Makefile
+++ b/drivers/net/ice/Makefile
@@ -11,9 +11,11 @@ LIB = librte_pmd_ice.a
CFLAGS += -O3
CFLAGS += $(WERROR_FLAGS)
CFLAGS += -DALLOW_EXPERIMENTAL_API
+CFLAGS += -I$(RTE_SDK)/drivers/common/iavf
LDLIBS += -lrte_eal -lrte_mbuf -lrte_ethdev -lrte_kvargs
LDLIBS += -lrte_bus_pci -lrte_mempool -lrte_hash
+LDLIBS += -lrte_net -lrte_common_iavf
EXPORT_MAP := rte_pmd_ice_version.map
@@ -84,6 +86,9 @@ ifeq ($(CC_AVX2_SUPPORT), 1)
endif
SRCS-$(CONFIG_RTE_LIBRTE_ICE_PMD) += ice_generic_flow.c
+SRCS-$(CONFIG_RTE_LIBRTE_ICE_PMD) += ice_dcf.c
+SRCS-$(CONFIG_RTE_LIBRTE_ICE_PMD) += ice_dcf_ethdev.c
+
# install this header file
SYMLINK-$(CONFIG_RTE_LIBRTE_ICE_PMD)-include := rte_pmd_ice.h
diff --git a/drivers/net/ice/ice_dcf.c b/drivers/net/ice/ice_dcf.c
new file mode 100644
index 000000000..0a99cceb1
--- /dev/null
+++ b/drivers/net/ice/ice_dcf.c
@@ -0,0 +1,474 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#include <sys/queue.h>
+#include <stdio.h>
+#include <errno.h>
+#include <stdint.h>
+#include <string.h>
+#include <unistd.h>
+#include <stdarg.h>
+#include <inttypes.h>
+#include <rte_byteorder.h>
+#include <rte_common.h>
+
+#include <rte_pci.h>
+#include <rte_atomic.h>
+#include <rte_eal.h>
+#include <rte_ether.h>
+#include <rte_ethdev_driver.h>
+#include <rte_ethdev_pci.h>
+#include <rte_malloc.h>
+#include <rte_memzone.h>
+#include <rte_dev.h>
+
+#include "ice_dcf.h"
+
+#define ICE_DCF_AQ_LEN 32
+#define ICE_DCF_AQ_BUF_SZ 4096
+
+#define ICE_DCF_ARQ_MAX_RETRIES 200
+#define ICE_DCF_ARQ_CHECK_TIME 2 /* msecs */
+
+#define ICE_DCF_VF_RES_BUF_SZ \
+ (sizeof(struct virtchnl_vf_resource) + \
+ IAVF_MAX_VF_VSI * sizeof(struct virtchnl_vsi_resource))
+
+static __rte_always_inline int
+ice_dcf_send_cmd_req_no_irq(struct ice_dcf_hw *hw, enum virtchnl_ops op,
+ uint8_t *req_msg, uint16_t req_msglen)
+{
+ return iavf_aq_send_msg_to_pf(&hw->avf, op, IAVF_SUCCESS,
+ req_msg, req_msglen, NULL);
+}
+
+static int
+ice_dcf_recv_cmd_rsp_no_irq(struct ice_dcf_hw *hw, enum virtchnl_ops op,
+ uint8_t *rsp_msgbuf, uint16_t rsp_buflen,
+ uint16_t *rsp_msglen)
+{
+ struct iavf_arq_event_info event;
+ enum virtchnl_ops v_op;
+ int i = 0;
+ int err;
+
+ event.buf_len = rsp_buflen;
+ event.msg_buf = rsp_msgbuf;
+
+ do {
+ err = iavf_clean_arq_element(&hw->avf, &event, NULL);
+ if (err != IAVF_SUCCESS)
+ goto again;
+
+ v_op = rte_le_to_cpu_32(event.desc.cookie_high);
+ if (v_op != op)
+ goto again;
+
+ if (rsp_msglen != NULL)
+ *rsp_msglen = event.msg_len;
+ return rte_le_to_cpu_32(event.desc.cookie_low);
+
+again:
+ rte_delay_ms(ICE_DCF_ARQ_CHECK_TIME);
+ } while (i++ < ICE_DCF_ARQ_MAX_RETRIES);
+
+ return -EIO;
+}
+
+static __rte_always_inline void
+ice_dcf_aq_cmd_clear(struct ice_dcf_hw *hw, struct dcf_virtchnl_cmd *cmd)
+{
+ rte_spinlock_lock(&hw->vc_cmd_queue_lock);
+
+ TAILQ_REMOVE(&hw->vc_cmd_queue, cmd, next);
+
+ rte_spinlock_unlock(&hw->vc_cmd_queue_lock);
+}
+
+static __rte_always_inline void
+ice_dcf_vc_cmd_set(struct ice_dcf_hw *hw, struct dcf_virtchnl_cmd *cmd)
+{
+ cmd->v_ret = IAVF_ERR_NOT_READY;
+ cmd->rsp_msglen = 0;
+ cmd->pending = 1;
+
+ rte_spinlock_lock(&hw->vc_cmd_queue_lock);
+
+ TAILQ_INSERT_TAIL(&hw->vc_cmd_queue, cmd, next);
+
+ rte_spinlock_unlock(&hw->vc_cmd_queue_lock);
+}
+
+static __rte_always_inline int
+ice_dcf_vc_cmd_send(struct ice_dcf_hw *hw, struct dcf_virtchnl_cmd *cmd)
+{
+ return iavf_aq_send_msg_to_pf(&hw->avf,
+ cmd->v_op, IAVF_SUCCESS,
+ cmd->req_msg, cmd->req_msglen, NULL);
+}
+
+static __rte_always_inline void
+ice_dcf_aq_cmd_handle(struct ice_dcf_hw *hw, struct iavf_arq_event_info *info)
+{
+ struct dcf_virtchnl_cmd *cmd;
+ enum virtchnl_ops v_op;
+ enum iavf_status v_ret;
+ uint16_t aq_op;
+
+ aq_op = rte_le_to_cpu_16(info->desc.opcode);
+ if (unlikely(aq_op != iavf_aqc_opc_send_msg_to_vf)) {
+ PMD_DRV_LOG(ERR,
+ "Request %u is not supported yet", aq_op);
+ return;
+ }
+
+ v_op = rte_le_to_cpu_32(info->desc.cookie_high);
+ if (unlikely(v_op == VIRTCHNL_OP_EVENT))
+ return;
+
+ v_ret = rte_le_to_cpu_32(info->desc.cookie_low);
+
+ rte_spinlock_lock(&hw->vc_cmd_queue_lock);
+
+ TAILQ_FOREACH(cmd, &hw->vc_cmd_queue, next) {
+ if (cmd->v_op == v_op && cmd->pending) {
+ cmd->v_ret = v_ret;
+ cmd->rsp_msglen = RTE_MIN(info->msg_len,
+ cmd->rsp_buflen);
+ if (likely(cmd->rsp_msglen != 0))
+ rte_memcpy(cmd->rsp_msgbuf, info->msg_buf,
+ cmd->rsp_msglen);
+
+ /* prevent compiler reordering */
+ rte_compiler_barrier();
+ cmd->pending = 0;
+ break;
+ }
+ }
+
+ rte_spinlock_unlock(&hw->vc_cmd_queue_lock);
+}
+
+static void
+ice_dcf_handle_virtchnl_msg(struct ice_dcf_hw *hw)
+{
+ struct iavf_arq_event_info info;
+ uint16_t pending = 1;
+ int ret;
+
+ info.buf_len = ICE_DCF_AQ_BUF_SZ;
+ info.msg_buf = hw->arq_buf;
+
+ while (pending) {
+ ret = iavf_clean_arq_element(&hw->avf, &info, &pending);
+ if (ret != IAVF_SUCCESS)
+ break;
+
+ ice_dcf_aq_cmd_handle(hw, &info);
+ }
+}
+
+static int
+ice_dcf_init_check_api_version(struct ice_dcf_hw *hw)
+{
+#define ICE_CPF_VIRTCHNL_VERSION_MAJOR_START 1
+#define ICE_CPF_VIRTCHNL_VERSION_MINOR_START 1
+ struct virtchnl_version_info version, *pver;
+ int err;
+
+ version.major = VIRTCHNL_VERSION_MAJOR;
+ version.minor = VIRTCHNL_VERSION_MINOR;
+ err = ice_dcf_send_cmd_req_no_irq(hw, VIRTCHNL_OP_VERSION,
+ (uint8_t *)&version, sizeof(version));
+ if (err) {
+ PMD_INIT_LOG(ERR, "Failed to send OP_VERSION");
+ return err;
+ }
+
+ pver = &hw->virtchnl_version;
+ err = ice_dcf_recv_cmd_rsp_no_irq(hw, VIRTCHNL_OP_VERSION,
+ (uint8_t *)pver, sizeof(*pver), NULL);
+ if (err) {
+ PMD_INIT_LOG(ERR, "Failed to get response of OP_VERSION");
+ return -1;
+ }
+
+ PMD_INIT_LOG(DEBUG,
+ "Peer PF API version: %u.%u", pver->major, pver->minor);
+
+ if (pver->major < ICE_CPF_VIRTCHNL_VERSION_MAJOR_START ||
+ (pver->major == ICE_CPF_VIRTCHNL_VERSION_MAJOR_START &&
+ pver->minor < ICE_CPF_VIRTCHNL_VERSION_MINOR_START)) {
+ PMD_INIT_LOG(ERR,
+ "VIRTCHNL API version should not be lower than (%u.%u)",
+ ICE_CPF_VIRTCHNL_VERSION_MAJOR_START,
+ ICE_CPF_VIRTCHNL_VERSION_MAJOR_START);
+ return -1;
+ } else if (pver->major > VIRTCHNL_VERSION_MAJOR ||
+ (pver->major == VIRTCHNL_VERSION_MAJOR &&
+ pver->minor > VIRTCHNL_VERSION_MINOR)) {
+ PMD_INIT_LOG(ERR,
+ "PF/VF API version mismatch:(%u.%u)-(%u.%u)",
+ pver->major, pver->minor,
+ VIRTCHNL_VERSION_MAJOR, VIRTCHNL_VERSION_MINOR);
+ return -1;
+ }
+
+ PMD_INIT_LOG(DEBUG, "Peer is supported PF host");
+
+ return 0;
+}
+
+static int
+ice_dcf_get_vf_resource(struct ice_dcf_hw *hw)
+{
+ uint32_t caps;
+ int err, i;
+
+ caps = VIRTCHNL_VF_OFFLOAD_WB_ON_ITR | VIRTCHNL_VF_OFFLOAD_RX_POLLING |
+ VIRTCHNL_VF_CAP_ADV_LINK_SPEED |
+ VF_BASE_MODE_OFFLOADS;
+
+ err = ice_dcf_send_cmd_req_no_irq(hw, VIRTCHNL_OP_GET_VF_RESOURCES,
+ (uint8_t *)&caps, sizeof(caps));
+ if (err) {
+ PMD_DRV_LOG(ERR, "Failed to send msg OP_GET_VF_RESOURCE");
+ return err;
+ }
+
+ err = ice_dcf_recv_cmd_rsp_no_irq(hw, VIRTCHNL_OP_GET_VF_RESOURCES,
+ (uint8_t *)hw->vf_res,
+ ICE_DCF_VF_RES_BUF_SZ, NULL);
+ if (err) {
+ PMD_DRV_LOG(ERR, "Failed to get response of OP_GET_VF_RESOURCE");
+ return -1;
+ }
+
+ iavf_vf_parse_hw_config(&hw->avf, hw->vf_res);
+
+ hw->vsi_res = NULL;
+ for (i = 0; i < hw->vf_res->num_vsis; i++) {
+ if (hw->vf_res->vsi_res[i].vsi_type == VIRTCHNL_VSI_SRIOV)
+ hw->vsi_res = &hw->vf_res->vsi_res[i];
+ }
+
+ if (!hw->vsi_res) {
+ PMD_DRV_LOG(ERR, "no LAN VSI found");
+ return -1;
+ }
+
+ hw->vsi_id = hw->vsi_res->vsi_id;
+ PMD_DRV_LOG(DEBUG, "VSI ID is %u", hw->vsi_id);
+
+ return 0;
+}
+
+static int
+ice_dcf_check_reset_done(struct ice_dcf_hw *hw)
+{
+#define ICE_DCF_RESET_WAIT_CNT 50
+ struct iavf_hw *avf = &hw->avf;
+ int i, reset;
+
+ for (i = 0; i < ICE_DCF_RESET_WAIT_CNT; i++) {
+ reset = IAVF_READ_REG(avf, IAVF_VFGEN_RSTAT) &
+ IAVF_VFGEN_RSTAT_VFR_STATE_MASK;
+ reset = reset >> IAVF_VFGEN_RSTAT_VFR_STATE_SHIFT;
+
+ if (reset == VIRTCHNL_VFR_VFACTIVE ||
+ reset == VIRTCHNL_VFR_COMPLETED)
+ break;
+
+ rte_delay_ms(20);
+ }
+
+ if (i >= ICE_DCF_RESET_WAIT_CNT)
+ return -1;
+
+ return 0;
+}
+
+static inline void
+ice_dcf_enable_irq0(struct ice_dcf_hw *hw)
+{
+ struct iavf_hw *avf = &hw->avf;
+
+ /* Enable admin queue interrupt trigger */
+ IAVF_WRITE_REG(avf, IAVF_VFINT_ICR0_ENA1,
+ IAVF_VFINT_ICR0_ENA1_ADMINQ_MASK);
+ IAVF_WRITE_REG(avf, IAVF_VFINT_DYN_CTL01,
+ IAVF_VFINT_DYN_CTL01_INTENA_MASK |
+ IAVF_VFINT_DYN_CTL01_CLEARPBA_MASK |
+ IAVF_VFINT_DYN_CTL01_ITR_INDX_MASK);
+
+ IAVF_WRITE_FLUSH(avf);
+}
+
+static inline void
+ice_dcf_disable_irq0(struct ice_dcf_hw *hw)
+{
+ struct iavf_hw *avf = &hw->avf;
+
+ /* Disable all interrupt types */
+ IAVF_WRITE_REG(avf, IAVF_VFINT_ICR0_ENA1, 0);
+ IAVF_WRITE_REG(avf, IAVF_VFINT_DYN_CTL01,
+ IAVF_VFINT_DYN_CTL01_ITR_INDX_MASK);
+
+ IAVF_WRITE_FLUSH(avf);
+}
+
+static void
+ice_dcf_dev_interrupt_handler(void *param)
+{
+ struct ice_dcf_hw *hw = param;
+
+ ice_dcf_disable_irq0(hw);
+
+ ice_dcf_handle_virtchnl_msg(hw);
+
+ ice_dcf_enable_irq0(hw);
+}
+
+int
+ice_dcf_execute_virtchnl_cmd(struct ice_dcf_hw *hw,
+ struct dcf_virtchnl_cmd *cmd)
+{
+ int i = 0;
+ int err;
+
+ if ((cmd->req_msg && !cmd->req_msglen) ||
+ (!cmd->req_msg && cmd->req_msglen) ||
+ (cmd->rsp_msgbuf && !cmd->rsp_buflen) ||
+ (!cmd->rsp_msgbuf && cmd->rsp_buflen))
+ return -EINVAL;
+
+ rte_spinlock_lock(&hw->vc_cmd_send_lock);
+ ice_dcf_vc_cmd_set(hw, cmd);
+
+ err = ice_dcf_vc_cmd_send(hw, cmd);
+ if (err) {
+ PMD_DRV_LOG(ERR, "fail to send cmd %d", cmd->v_op);
+ goto ret;
+ }
+
+ do {
+ if (!cmd->pending)
+ break;
+
+ rte_delay_ms(ICE_DCF_ARQ_CHECK_TIME);
+ } while (i++ < ICE_DCF_ARQ_MAX_RETRIES);
+
+ if (cmd->v_ret != IAVF_SUCCESS) {
+ err = -1;
+ PMD_DRV_LOG(ERR,
+ "No response (%d times) or return failure (%d) for cmd %d",
+ i, cmd->v_ret, cmd->v_op);
+ }
+
+ret:
+ ice_dcf_aq_cmd_clear(hw, cmd);
+ rte_spinlock_unlock(&hw->vc_cmd_send_lock);
+ return err;
+}
+
+int
+ice_dcf_init_hw(struct rte_eth_dev *eth_dev, struct ice_dcf_hw *hw)
+{
+ struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ int ret;
+
+ hw->avf.hw_addr = pci_dev->mem_resource[0].addr;
+ hw->avf.back = hw;
+
+ hw->avf.bus.bus_id = pci_dev->addr.bus;
+ hw->avf.bus.device = pci_dev->addr.devid;
+ hw->avf.bus.func = pci_dev->addr.function;
+
+ hw->avf.device_id = pci_dev->id.device_id;
+ hw->avf.vendor_id = pci_dev->id.vendor_id;
+ hw->avf.subsystem_device_id = pci_dev->id.subsystem_device_id;
+ hw->avf.subsystem_vendor_id = pci_dev->id.subsystem_vendor_id;
+
+ hw->avf.aq.num_arq_entries = ICE_DCF_AQ_LEN;
+ hw->avf.aq.num_asq_entries = ICE_DCF_AQ_LEN;
+ hw->avf.aq.arq_buf_size = ICE_DCF_AQ_BUF_SZ;
+ hw->avf.aq.asq_buf_size = ICE_DCF_AQ_BUF_SZ;
+
+ rte_spinlock_init(&hw->vc_cmd_send_lock);
+ rte_spinlock_init(&hw->vc_cmd_queue_lock);
+ TAILQ_INIT(&hw->vc_cmd_queue);
+
+ hw->arq_buf = rte_zmalloc("arq_buf", ICE_DCF_AQ_BUF_SZ, 0);
+ if (hw->arq_buf == NULL) {
+ PMD_INIT_LOG(ERR, "unable to allocate AdminQ buffer memory");
+ goto err;
+ }
+
+ ret = iavf_set_mac_type(&hw->avf);
+ if (ret) {
+ PMD_INIT_LOG(ERR, "set_mac_type failed: %d", ret);
+ goto err;
+ }
+
+ ret = ice_dcf_check_reset_done(hw);
+ if (ret) {
+ PMD_INIT_LOG(ERR, "VF is still resetting");
+ goto err;
+ }
+
+ ret = iavf_init_adminq(&hw->avf);
+ if (ret) {
+ PMD_INIT_LOG(ERR, "init_adminq failed: %d", ret);
+ goto err;
+ }
+
+ if (ice_dcf_init_check_api_version(hw)) {
+ PMD_INIT_LOG(ERR, "check_api version failed");
+ goto err_api;
+ }
+
+ hw->vf_res = rte_zmalloc("vf_res", ICE_DCF_VF_RES_BUF_SZ, 0);
+ if (hw->vf_res == NULL) {
+ PMD_INIT_LOG(ERR, "unable to allocate vf_res memory");
+ goto err_api;
+ }
+
+ if (ice_dcf_get_vf_resource(hw)) {
+ PMD_INIT_LOG(ERR, "Failed to get VF resource");
+ goto err_alloc;
+ }
+
+ rte_intr_callback_register(&pci_dev->intr_handle,
+ ice_dcf_dev_interrupt_handler, hw);
+ rte_intr_enable(&pci_dev->intr_handle);
+ ice_dcf_enable_irq0(hw);
+
+ return 0;
+
+err_alloc:
+ rte_free(hw->vf_res);
+err_api:
+ iavf_shutdown_adminq(&hw->avf);
+err:
+ rte_free(hw->arq_buf);
+
+ return -1;
+}
+
+void
+ice_dcf_uninit_hw(struct rte_eth_dev *eth_dev, struct ice_dcf_hw *hw)
+{
+ struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_intr_handle *intr_handle = &pci_dev->intr_handle;
+
+ ice_dcf_disable_irq0(hw);
+ rte_intr_disable(intr_handle);
+ rte_intr_callback_unregister(intr_handle,
+ ice_dcf_dev_interrupt_handler, hw);
+
+ iavf_shutdown_adminq(&hw->avf);
+
+ rte_free(hw->arq_buf);
+ rte_free(hw->vf_res);
+}
diff --git a/drivers/net/ice/ice_dcf.h b/drivers/net/ice/ice_dcf.h
new file mode 100644
index 000000000..f44c09db2
--- /dev/null
+++ b/drivers/net/ice/ice_dcf.h
@@ -0,0 +1,52 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#ifndef _ICE_DCF_H_
+#define _ICE_DCF_H_
+
+#include <rte_ethdev_driver.h>
+
+#include <iavf_prototype.h>
+#include <iavf_adminq_cmd.h>
+#include <iavf_type.h>
+
+#include "ice_logs.h"
+
+struct dcf_virtchnl_cmd {
+ TAILQ_ENTRY(dcf_virtchnl_cmd) next;
+
+ enum virtchnl_ops v_op;
+ enum iavf_status v_ret;
+
+ uint16_t req_msglen;
+ uint8_t *req_msg;
+
+ uint16_t rsp_msglen;
+ uint16_t rsp_buflen;
+ uint8_t *rsp_msgbuf;
+
+ volatile int pending;
+};
+
+struct ice_dcf_hw {
+ struct iavf_hw avf;
+
+ rte_spinlock_t vc_cmd_send_lock;
+ rte_spinlock_t vc_cmd_queue_lock;
+ TAILQ_HEAD(, dcf_virtchnl_cmd) vc_cmd_queue;
+ uint8_t *arq_buf;
+
+ struct virtchnl_version_info virtchnl_version;
+ struct virtchnl_vf_resource *vf_res; /* VF resource */
+ struct virtchnl_vsi_resource *vsi_res; /* LAN VSI */
+ uint16_t vsi_id;
+};
+
+int ice_dcf_execute_virtchnl_cmd(struct ice_dcf_hw *hw,
+ struct dcf_virtchnl_cmd *cmd);
+
+int ice_dcf_init_hw(struct rte_eth_dev *eth_dev, struct ice_dcf_hw *hw);
+void ice_dcf_uninit_hw(struct rte_eth_dev *eth_dev, struct ice_dcf_hw *hw);
+
+#endif /* _ICE_DCF_H_ */
diff --git a/drivers/net/ice/ice_dcf_ethdev.c b/drivers/net/ice/ice_dcf_ethdev.c
new file mode 100644
index 000000000..23f82a487
--- /dev/null
+++ b/drivers/net/ice/ice_dcf_ethdev.c
@@ -0,0 +1,317 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#include <errno.h>
+#include <stdbool.h>
+#include <sys/types.h>
+#include <sys/ioctl.h>
+#include <unistd.h>
+
+#include <rte_interrupts.h>
+#include <rte_debug.h>
+#include <rte_pci.h>
+#include <rte_atomic.h>
+#include <rte_eal.h>
+#include <rte_ether.h>
+#include <rte_ethdev_pci.h>
+#include <rte_kvargs.h>
+#include <rte_malloc.h>
+#include <rte_memzone.h>
+#include <rte_dev.h>
+
+#include <iavf_devids.h>
+
+#include "ice_generic_flow.h"
+#include "ice_dcf_ethdev.h"
+
+static uint16_t
+ice_dcf_recv_pkts(__rte_unused void *rx_queue,
+ __rte_unused struct rte_mbuf **bufs,
+ __rte_unused uint16_t nb_pkts)
+{
+ return 0;
+}
+
+static uint16_t
+ice_dcf_xmit_pkts(__rte_unused void *tx_queue,
+ __rte_unused struct rte_mbuf **bufs,
+ __rte_unused uint16_t nb_pkts)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_start(struct rte_eth_dev *dev)
+{
+ dev->data->dev_link.link_status = ETH_LINK_UP;
+
+ return 0;
+}
+
+static void
+ice_dcf_dev_stop(struct rte_eth_dev *dev)
+{
+ dev->data->dev_link.link_status = ETH_LINK_DOWN;
+}
+
+static int
+ice_dcf_dev_configure(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_info_get(struct rte_eth_dev *dev,
+ struct rte_eth_dev_info *dev_info)
+{
+ struct ice_dcf_adapter *adapter = dev->data->dev_private;
+
+ dev_info->max_mac_addrs = 1;
+ dev_info->max_rx_pktlen = (uint32_t)-1;
+ dev_info->max_rx_queues = RTE_DIM(adapter->rxqs);
+ dev_info->max_tx_queues = RTE_DIM(adapter->txqs);
+
+ return 0;
+}
+
+static int
+ice_dcf_stats_get(__rte_unused struct rte_eth_dev *dev,
+ __rte_unused struct rte_eth_stats *igb_stats)
+{
+ return 0;
+}
+
+static int
+ice_dcf_stats_reset(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_promiscuous_enable(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_promiscuous_disable(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_allmulticast_enable(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_allmulticast_disable(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_filter_ctrl(struct rte_eth_dev *dev,
+ enum rte_filter_type filter_type,
+ __rte_unused enum rte_filter_op filter_op,
+ __rte_unused void *arg)
+{
+ int ret = 0;
+
+ if (!dev)
+ return -EINVAL;
+
+ switch (filter_type) {
+ default:
+ PMD_DRV_LOG(WARNING, "Filter type (%d) not supported",
+ filter_type);
+ ret = -EINVAL;
+ break;
+ }
+
+ return ret;
+}
+
+static void
+ice_dcf_dev_close(struct rte_eth_dev *dev)
+{
+ struct ice_dcf_adapter *adapter = dev->data->dev_private;
+
+ if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+ return;
+
+ dev->dev_ops = NULL;
+ dev->rx_pkt_burst = NULL;
+ dev->tx_pkt_burst = NULL;
+ dev->data->mac_addrs = NULL;
+
+ ice_dcf_uninit_hw(dev, &adapter->real_hw);
+}
+
+static void
+ice_dcf_queue_release(__rte_unused void *q)
+{
+}
+
+static int
+ice_dcf_link_update(__rte_unused struct rte_eth_dev *dev,
+ __rte_unused int wait_to_complete)
+{
+ return 0;
+}
+
+static int
+ice_dcf_rx_queue_setup(struct rte_eth_dev *dev,
+ uint16_t rx_queue_id,
+ __rte_unused uint16_t nb_rx_desc,
+ __rte_unused unsigned int socket_id,
+ __rte_unused const struct rte_eth_rxconf *rx_conf,
+ __rte_unused struct rte_mempool *mb_pool)
+{
+ struct ice_dcf_adapter *adapter = dev->data->dev_private;
+
+ dev->data->rx_queues[rx_queue_id] = &adapter->rxqs[rx_queue_id];
+
+ return 0;
+}
+
+static int
+ice_dcf_tx_queue_setup(struct rte_eth_dev *dev,
+ uint16_t tx_queue_id,
+ __rte_unused uint16_t nb_tx_desc,
+ __rte_unused unsigned int socket_id,
+ __rte_unused const struct rte_eth_txconf *tx_conf)
+{
+ struct ice_dcf_adapter *adapter = dev->data->dev_private;
+
+ dev->data->tx_queues[tx_queue_id] = &adapter->txqs[tx_queue_id];
+
+ return 0;
+}
+
+static const struct eth_dev_ops ice_dcf_eth_dev_ops = {
+ .dev_start = ice_dcf_dev_start,
+ .dev_stop = ice_dcf_dev_stop,
+ .dev_close = ice_dcf_dev_close,
+ .dev_configure = ice_dcf_dev_configure,
+ .dev_infos_get = ice_dcf_dev_info_get,
+ .rx_queue_setup = ice_dcf_rx_queue_setup,
+ .tx_queue_setup = ice_dcf_tx_queue_setup,
+ .rx_queue_release = ice_dcf_queue_release,
+ .tx_queue_release = ice_dcf_queue_release,
+ .link_update = ice_dcf_link_update,
+ .stats_get = ice_dcf_stats_get,
+ .stats_reset = ice_dcf_stats_reset,
+ .promiscuous_enable = ice_dcf_dev_promiscuous_enable,
+ .promiscuous_disable = ice_dcf_dev_promiscuous_disable,
+ .allmulticast_enable = ice_dcf_dev_allmulticast_enable,
+ .allmulticast_disable = ice_dcf_dev_allmulticast_disable,
+ .filter_ctrl = ice_dcf_dev_filter_ctrl,
+};
+
+static int
+ice_dcf_dev_init(struct rte_eth_dev *eth_dev)
+{
+ struct ice_dcf_adapter *adapter = eth_dev->data->dev_private;
+
+ eth_dev->dev_ops = &ice_dcf_eth_dev_ops;
+ eth_dev->rx_pkt_burst = ice_dcf_recv_pkts;
+ eth_dev->tx_pkt_burst = ice_dcf_xmit_pkts;
+
+ if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+ return 0;
+
+ eth_dev->data->dev_flags |= RTE_ETH_DEV_CLOSE_REMOVE;
+
+ if (ice_dcf_init_hw(eth_dev, &adapter->real_hw) != 0) {
+ PMD_INIT_LOG(ERR, "Failed to init DCF hardware");
+ return -1;
+ }
+
+ rte_eth_random_addr(adapter->mac_addr.addr_bytes);
+ eth_dev->data->mac_addrs = &adapter->mac_addr;
+
+ return 0;
+}
+
+static int
+ice_dcf_dev_uninit(struct rte_eth_dev *eth_dev)
+{
+ ice_dcf_dev_close(eth_dev);
+
+ return 0;
+}
+
+static int
+ice_dcf_cap_check_handler(__rte_unused const char *key,
+ const char *value, __rte_unused void *opaque)
+{
+ if (strcmp(value, "dcf"))
+ return -1;
+
+ return 0;
+}
+
+static int
+ice_dcf_cap_selected(struct rte_devargs *devargs)
+{
+ struct rte_kvargs *kvlist;
+ const char *key = "cap";
+ int ret = 0;
+
+ if (devargs == NULL)
+ return 0;
+
+ kvlist = rte_kvargs_parse(devargs->args, NULL);
+ if (kvlist == NULL)
+ return 0;
+
+ if (!rte_kvargs_count(kvlist, key))
+ goto exit;
+
+ /* dcf capability selected when there's a key-value pair: cap=dcf */
+ if (rte_kvargs_process(kvlist, key,
+ ice_dcf_cap_check_handler, NULL) < 0)
+ goto exit;
+
+ ret = 1;
+
+exit:
+ rte_kvargs_free(kvlist);
+ return ret;
+}
+
+static int eth_ice_dcf_pci_probe(__rte_unused struct rte_pci_driver *pci_drv,
+ struct rte_pci_device *pci_dev)
+{
+ if (!ice_dcf_cap_selected(pci_dev->device.devargs))
+ return 1;
+
+ return rte_eth_dev_pci_generic_probe(pci_dev,
+ sizeof(struct ice_dcf_adapter),
+ ice_dcf_dev_init);
+}
+
+static int eth_ice_dcf_pci_remove(struct rte_pci_device *pci_dev)
+{
+ return rte_eth_dev_pci_generic_remove(pci_dev, ice_dcf_dev_uninit);
+}
+
+static const struct rte_pci_id pci_id_ice_dcf_map[] = {
+ { RTE_PCI_DEVICE(IAVF_INTEL_VENDOR_ID, IAVF_DEV_ID_ADAPTIVE_VF) },
+ { .vendor_id = 0, /* sentinel */ },
+};
+
+static struct rte_pci_driver rte_ice_dcf_pmd = {
+ .id_table = pci_id_ice_dcf_map,
+ .drv_flags = RTE_PCI_DRV_NEED_MAPPING,
+ .probe = eth_ice_dcf_pci_probe,
+ .remove = eth_ice_dcf_pci_remove,
+};
+
+RTE_PMD_REGISTER_PCI(net_ice_dcf, rte_ice_dcf_pmd);
+RTE_PMD_REGISTER_PCI_TABLE(net_ice_dcf, pci_id_ice_dcf_map);
+RTE_PMD_REGISTER_KMOD_DEP(net_ice_dcf, "* igb_uio | vfio-pci");
+RTE_PMD_REGISTER_PARAM_STRING(net_ice_dcf, "cap=dcf");
diff --git a/drivers/net/ice/ice_dcf_ethdev.h b/drivers/net/ice/ice_dcf_ethdev.h
new file mode 100644
index 000000000..0c34a0095
--- /dev/null
+++ b/drivers/net/ice/ice_dcf_ethdev.h
@@ -0,0 +1,24 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#ifndef _ICE_DCF_ETHDEV_H_
+#define _ICE_DCF_ETHDEV_H_
+
+#include "ice_ethdev.h"
+#include "ice_dcf.h"
+
+#define ICE_DCF_MAX_RINGS 1
+
+struct ice_dcf_queue {
+ uint64_t dummy;
+};
+
+struct ice_dcf_adapter {
+ struct ice_dcf_hw real_hw;
+ struct rte_ether_addr mac_addr;
+ struct ice_dcf_queue rxqs[ICE_DCF_MAX_RINGS];
+ struct ice_dcf_queue txqs[ICE_DCF_MAX_RINGS];
+};
+
+#endif /* _ICE_DCF_ETHDEV_H_ */
diff --git a/drivers/net/ice/meson.build b/drivers/net/ice/meson.build
index f9e897bbc..0ba9668d1 100644
--- a/drivers/net/ice/meson.build
+++ b/drivers/net/ice/meson.build
@@ -15,8 +15,8 @@ sources = files(
'ice_hash.c'
)
-deps += ['hash']
-includes += include_directories('base')
+deps += ['hash', 'net', 'common_iavf']
+includes += include_directories('base', '../../common/iavf')
if arch_subdir == 'x86'
sources += files('ice_rxtx_vec_sse.c')
@@ -37,4 +37,7 @@ if arch_subdir == 'x86'
endif
endif
+sources += files('ice_dcf.c',
+ 'ice_dcf_ethdev.c')
+
install_headers('rte_pmd_ice.h')
diff --git a/mk/rte.app.mk b/mk/rte.app.mk
index d295ca0a5..f3798a09f 100644
--- a/mk/rte.app.mk
+++ b/mk/rte.app.mk
@@ -185,6 +185,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_I40E_PMD) += -lrte_pmd_i40e
_LDLIBS-$(CONFIG_RTE_LIBRTE_IAVF_PMD) += -lrte_pmd_iavf
_LDLIBS-$(CONFIG_RTE_LIBRTE_ICE_PMD) += -lrte_pmd_ice
IAVF-y := $(CONFIG_RTE_LIBRTE_IAVF_PMD)
+IAVF-y += $(CONFIG_RTE_LIBRTE_ICE_PMD)
ifeq ($(findstring y,$(IAVF-y)),y)
_LDLIBS-y += -lrte_common_iavf
endif
--
2.26.0
^ permalink raw reply [relevance 1%]
* [dpdk-dev] [PATCH v5 2/7] net/ice: add the DCF hardware initialization
@ 2020-03-26 7:15 1% ` Haiyue Wang
0 siblings, 0 replies; 200+ results
From: Haiyue Wang @ 2020-03-26 7:15 UTC (permalink / raw)
To: dev, xiaolong.ye, qi.z.zhang, jingjing.wu, qiming.yang, beilei.xing
Cc: wei.zhao1, Haiyue Wang
Introduce the DCF (Device Config Function) feature in the ice PMD, it
works as a standalone PMD which doesn't handle the packet Rx/Tx related
things. Its hardware entity is the VF.
Add the basic DCF hardware initialization, this is specified by devarg
'cap=dcf'.
Signed-off-by: Haiyue Wang <haiyue.wang@intel.com>
Acked-by: Qi Zhang <qi.z.zhang@intel.com>
---
doc/guides/nics/ice.rst | 47 +++
doc/guides/nics/img/ice_dcf.png | Bin 0 -> 39168 bytes
doc/guides/rel_notes/release_20_05.rst | 5 +
drivers/common/Makefile | 1 +
drivers/net/ice/Makefile | 5 +
drivers/net/ice/ice_dcf.c | 474 +++++++++++++++++++++++++
drivers/net/ice/ice_dcf.h | 52 +++
drivers/net/ice/ice_dcf_ethdev.c | 317 +++++++++++++++++
drivers/net/ice/ice_dcf_ethdev.h | 24 ++
drivers/net/ice/meson.build | 7 +-
mk/rte.app.mk | 1 +
11 files changed, 931 insertions(+), 2 deletions(-)
create mode 100644 doc/guides/nics/img/ice_dcf.png
create mode 100644 drivers/net/ice/ice_dcf.c
create mode 100644 drivers/net/ice/ice_dcf.h
create mode 100644 drivers/net/ice/ice_dcf_ethdev.c
create mode 100644 drivers/net/ice/ice_dcf_ethdev.h
diff --git a/doc/guides/nics/ice.rst b/doc/guides/nics/ice.rst
index 8af32dabf..2639ae239 100644
--- a/doc/guides/nics/ice.rst
+++ b/doc/guides/nics/ice.rst
@@ -240,6 +240,53 @@ report a MDD event and drop the packets.
The APPs based on DPDK should avoid providing such packets.
+Device Config Function (DCF)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This section demonstrates ICE DCF PMD, which shares the core module with ICE
+PMD and iAVF PMD.
+
+A DCF (Device Config Function) PMD bounds to the device's trusted VF with ID 0,
+it can act as a sole controlling entity to exercise advance functionality (such
+as switch, ACL) for the rest VFs.
+
+The DCF PMD needs to advertise and acquire DCF capability which allows DCF to
+send AdminQ commands that it would like to execute over to the PF and receive
+responses for the same from PF.
+
+.. _figure_ice_dcf:
+
+.. figure:: img/ice_dcf.*
+
+ DCF Communication flow.
+
+#. Create the VFs::
+
+ echo 4 > /sys/bus/pci/devices/0000\:18\:00.0/sriov_numvfs
+
+#. Enable the VF0 trust on::
+
+ ip link set dev enp24s0f0 vf 0 trust on
+
+#. Bind the VF0, and run testpmd with 'cap=dcf' devarg::
+
+ testpmd -l 22-25 -n 4 -w 18:01.0,cap=dcf -- -i
+
+#. Monitor the VF2 interface network traffic::
+
+ tcpdump -e -nn -i enp24s1f2
+
+#. Create one flow to redirect the traffic to VF2 by DCF::
+
+ flow create 0 priority 0 ingress pattern eth / ipv4 src is 192.168.0.2 \
+ dst is 192.168.0.3 / end actions vf id 2 / end
+
+#. Send the packet, and it should be displayed on tcpdump::
+
+ sendp(Ether(src='3c:fd:fe:aa:bb:78', dst='00:00:00:01:02:03')/IP(src=' \
+ 192.168.0.2', dst="192.168.0.3")/TCP(flags='S')/Raw(load='XXXXXXXXXX'), \
+ iface="enp24s0f0", count=10)
+
Sample Application Notes
------------------------
diff --git a/doc/guides/nics/img/ice_dcf.png b/doc/guides/nics/img/ice_dcf.png
new file mode 100644
index 0000000000000000000000000000000000000000..540823c4a05e9ea9d558d233fea77016d1ae5df3
GIT binary patch
literal 39168
zcmZ6ybzD?k*FH`NDBT?jlF}e8B?Hn74BeolfOLbDz);d%Lw70NNOyNPNO%7Z_x(KY
z`+k3a@X<4KW}m&+Uh7)dy4F6yN(xdKsHCWHaBvvXpCpvw;1KwLKlqmjz$g7i%71`=
z;O&*A#NkSZ$ajH@7p7wJVsLO35oq^@NWk^0Z=W>n;oxu`p8w#NOsHJo;1Xh`CBz^u
zdIzZ}I(W`x-6a%La{^jclNOU&`_)rTS4-}zY76d1P4w1CVq)-OmrKC^5#ZY~-}pzu
zy$L?@i@zdO6jl2qp0>O_?^V0~==_TKu;0E)6AT_jK}Ys$GuU9xpl6NmUq4AAAS8rZ
z5<h>~BQL7h+1bg!^v{JkDk^GFZXt;WZK)wU@I4Lh>Nma-8Py=vX58uL5BwsNv0+TC
ztdd|ub|{f>I#qBR_r=Ecc2+(+F$m*+POs4gtgWq$;%5$Zz1W+)IolS4AR!?E!%MzF
zY^hc+(<6BKV$B>q*BptIQ5gaPfd*IJosgp=XT25!8zwm^L5M@!Q9uFabHF7#DiuCK
zt~qeo#^9j@Nyw!ThkWHBd%m>z=MtX)IqO&q%qUT-Q<|7KJNMhNs;a6pC&Kl7XSB)v
z3KtLW;^N}!aM9A*nl<f&LypeAH;UH%@~=<W?^;Mic5d!cle_a$qiaVX7S(%;iC{c>
zPEJmL@k|#dzb#-e#WW$n3TwM=5&|%zK#w5Rj~l++vL_;_xx2ZfV1g5o&-OTVud|dg
zY)p8a+CN(%s#Jb5B@9dFyIC8&!|9!kw<)+dnAUfkUON7RZe*>j5V@t@LFIjKzBWN}
zl6bSB?AXD@u-6mJ^$s$%>{fg;H**<Wy_z$R_0fFyxQl_NC8(r`t&yYbC$Wq!ZhWiR
zaQO>2)_CPI7Ur}ViP+8SHXAZ(qCX-8Pgw>@)`d8#Ub}%C2oieVUWG5$Z;wDPw0oh!
z(MS2V(Mt&eBO3Siyc1oDi4_JnN!?`_!F7~4=LJ<;7i4@c?GuTsp&oXOubJN~j*nRE
zr*(v=l`h&IUu)Y-G0<(qS?nZJwYZ$k+*tNcTf2SvjF$-h6dEDNR%>@OyFp@~wx8sm
zQ@KkHCa?M<DDa*q5JkVi|7}X6ooQlUmnxZR*#{T+GPIBvv|B=xJdNM>M<?%wx7c)l
zdn}q*;Wg|Gzyh^yRZNGE1Xt2+qw~^RE=E{lE|-=pha<-cjHj$U3eSXtS-y9ciA;Qt
zg#X2CIXogbtk+gV#pepYbkK~qZROc+2-nD3e6SS4d#n~;vgmMgi02}r<bB>}Fg(g-
zTDmx2YrO0kRzm_}D5S%J%DnQ0DdV>iV*NTt#-ch%|0xYIm~S%Q_BwvyfdQLqe%iUU
za<4uRAr<XM?#|y&3?~<aXTr-Hgaz#tT;9D;cK$2uv9~aYB8Zi$G+Wuxv-FN9+5PYR
zh4qI=trjcZ;=wL@A=W&9;v>?i^7Yn}O(rs$G1Q(|Z2z1sn<Yns)V;&#c$@TE2Q)Ru
z)6)`Bt!gQksJ*lfL=-F}Q=R&|&-+U>DkcK~7vFVi4u!dmmta<R;msWO$55HcDm$Mq
zYzuC++dJmp5`L=_vy^#UbXhfEraEBcG`ZP)uwPnpl~qe?xmmf+NiO2ax1Om+(c=}a
z!GBS0w-`sWU7FHRP;K|H)xhT98<o%yk#hLBt#8R=)!=YcP~pc5m_`)LD5eB~iYqs_
zj?tLy72jM<E{lrbr1en>*bVr_*W;KI;3E7^sdYFjoC{%b$NVtKVTT^7+2BYNw7pYj
zi4R5^w->W9pqU==f1#Eb;6C!>z+DXeb>TXZwZp;G`tJ%B5BYEu=0)FDN%x4fNM_s}
z&3d^+eysZRo^x@R*na2q(uHz@x%b$Uaf>KVBm(XH446M${l^6#mQ=@EaW_yr=YvCd
z+ccCShFQ?$1}?+o10_*^aev+*-pN^+qy-`EqoDYuWpSd~d>GvmJf>Q%JrCl>)WF6k
zmNOMMMp+hgneL-`A#pMMbJc=vS<;lS;}#5&f&14fUK<#DZv350Jd6LHkN$NrFqGJ8
zlLQ^XPt!kNF7<(R!K>C~vVL6sXh^GeVu@|f9?Xx@{W00`WXRZv(|^df><;rf5PNdx
zBpMO7vZw8Z5tyP#0>Vqx1X*<dv&0Kc;4ZffjQvQ21Vb2S)~HS*e}&q3vP0X5Ck>{p
zL|PgxQ2d3lia{mhxO*i0u!<pNN+=Dr*xkF}rL@3)H_B6SSLCd3U(-4g(^%Ggi&~71
zLC)Opd9psBf9m^tF-R2=i~Y+Wx~{DJ7&!b`+!vr6(LbBVOG$#4e;21&m~FS))*G-=
zPa}wqkl#vdk|f?wwVSv9(#)clt1G+|;%v%L28)#;+6VXk{7iUG+jNKVTH?KH*Mg`P
z4}U@{EuUAnkMX-Hs~^?DoM;li?~cv8zY`mcjZ7JLTz<$aU4RE4@Y}g{&JAw#$bdHS
zIsj~7y#89(M|b^nq}$-hTBu4^z2sUdUo4>v8BBVg%wyU%_MXf@bERkY-D?x<$;O<b
zpvBDRefW<DBArvMWHxmVUG9o}n*WowASDGwz0FLH(Dzku6^(`Vv9TH-Or_3WQek9&
zp{?wvKs0uy5PokudTJp-5ZP7duzj+zYp+MDA<51+GTg}}+XPcyOCY#AF=5lm{bEKD
zuYwz`>Y=}WkR(4(Y2H+%2SHKL8L4sdgQdlyUB8->Fe_=z_$V35@6;>_^Sx+A)!#{y
z%vNutx>hi*td+hI$B#YrfhqfD<l~3DlzhC=#`503zteBJi#)&=g+hzrVxg$DkD{I@
zg^onlA0{~L`RpH-+6gSVF_?>R2n@E#DgbDpEt*<Z@je@xlV4BeZ%k2#RC;@KHPXI=
zTv4h`bGcu3y~M9wU$HGWe-9h0vD0j^8a@f(O+_$Xo~<oWUu<$5exr+)S7gD4{3wVn
zgrav3%g?CuuSSpayqwxe{k^|Fjm)EGSivv!zjH;?(yL3=fBQ=%uWad!A{EMEQA^#0
z1YSLDC4#7>-9zS&zAcI<8lQ|mDf7d6im6Q*Pa;o%x1R@B!{cXusK5DSe-NiT0+q(3
zJAul<M~5YH9~^lh-eTUP{6+YH%@Tq2W2W5mXeE;-FoLq>=#Q(^6LkG!bA|cqcF*pi
zs1U^jfwNgMV;A~2jK#)8*61YsCi9QR)xxWa4oeYf>?DNCwZ`dlvWs4bPp=e=C;^0@
zKx^mvbur_;)>z-nUx7=5Qf6*XICBRQBO-{5^FAFbsff<KpkO!a8>m~+y7p>YE4&LA
zuaOFs0yDxhFwwq*qMEm7{5K*3xDA#58|>mq?5K=|*ax&{U5}%6AKyg@FV2-FFZMsS
zrHF6K$F#kK%8>ZN{<(`uWDrDi_jGHzJr(KJ|1yPYc)l)sxy8H2+vzKLzZc?{M!E7v
zF8j7?Yx*gy+=(y$qRI15nG2g2=Obg4BUlV=;xP}#z2<_JKj<^R-sjvH4CwW2sL>TH
zBFbS!1ZIVePcY_hP7Oo2*3+(G-We}hkFa4=pbOcJV<M%e|C+Y!zUqG{n|J=*>)qtn
zuvYEZ1ox-boipvgxq`t_VP5lxlooZlyR(J3H(<AZF?Y%7yd%9i@7`?6^1pYM<8}vA
z(fcoCgsMf!oK2VWC%GQAgevYXL7o?Eii4GU?~V&J?0;;|X}e4MzW2UeoiA;v_Kja^
zfO*44nT%$s<HJ_iHD0JS+i49q+(`zs=pvVW8i+;xuYtKK!0_&IS4RuYlSNh0LB0N0
z@7M}4b%hGL51O*{(vrSeZ6D8OHCSs;@XmhzeVhf(Slk`m&D&IY@;78t2t{dNPCng^
zl3t`DCq?X(3hcH>ORIJ6{H~eR;2X%NKo&enSZ7_?BtrJ5+S7I!1*N++VP8<HIBoPS
zs4s)hk6=@{5NU`^H+?`wcP3Bej!F{iUVeW-Vmz5k<8mL=To^D!#+Hw%%ierPJ88H_
zPv82kH0g1Ai1s0S9toBo^Lh+BR#Ex!G~`x30&(~+!!jfFHn1X}Pg@a3JmuSLTQ#Qb
zpQ{H2%o6CZ4m_;3gUII{#r0Kl7wZMxrF?pq^i$O~lx>HQg7%WjlKr9^Cy2f+@^BqY
zbx<#R&U+j$A5c+HBI@EZBVkOHH&Z}NE>JzIVq7w@Q)x#`<VnX!98`B2r8m%6Vo+<#
z6|brd_*__wX0{t1GqA?_MGCa@sMxH?;yyPPF1TB!b^Hhn>~1g2KzO8);$a8uAH*&3
zg)D9M>q1*@_xCVo#(yS{1NMP%E6V?!Zdx45c`#Zo`Hos^`dDktCFvy1sVN@{zyDdY
z>><Z7B&4vz?kkf60-^cx)}7#;WUQ;|Ust^A{Cc6ps{`J_Ho33e7$a~vXbL1#t`aT!
z2UJ5oWg&|^mMJOh+0J)(WRFNE*ofb`(PH0t*QCEMv<$k$g`L*rpQiFCT-S{2>uUSm
zw<=5E{Hpn^91&*!WTH|0=HF?*gcfcA(20pesM@XX<znYR-nMzrF*R^)xxTPuJ-mhU
z$llE1dA!{69o)n5LZtfsGN@04B+k+hUE_rgg5;a&za<srs{J-<3Qy6+aIcv(Vh0~)
zkKgAXo_BfQU6|#09=7DXY~^inHrqTNHErF7_5Idg-n+BOIH|92e|zJ_nvaQy)}E~B
z(h@~J+X?{0YlgxkKbSHfz6PX3+q1^PZ8&z^?eZ~;#lzEVFypvJwzmp*NjFhor7Ydu
z;qg_Ml8d`7>U`!J{-r<=sz9Ex<rB%wndw8n3T-!>HqoW|-MF)wvnft=dI;3KrdhMf
zMAysew~LW;pSGKW3p2yd|KC%+rqf%cE><%ubiE&Jx1%0MZ=Vj7xf*$_KBZqWak)pv
z4j1_4XAlThcpZNinfEz(+YMJ<jd{>q^T3Cm4}mOujeA_~ly2A-OCo@BUWkDw$?;>`
zNAbhK1bj9N)%iBl%U+H3#(W(f7`9(~6|64(y#)u!7m=9$ls^wV6HaD*K=}$*?`*(r
z%PAmMv7h_OPY`2waY<I!<1y{9lkvg*S2VsV;C^W>H9yql*ma~+oMt}#AG;;>^x|`8
z#R49-yL6HlpGSwi9Z64GB2a6w($eJZ>gJ#mxlbpK{v}O&29pecI1p8Qp<h1SHvO4N
z_+F)%xO0841x)wd$&8Vw&1HJ3w?MV^H)!Ha$X1O>IO*|zpZSM6>*QdWiv2AWGUOWT
zi-U3Fb<QX6QPsrlDFCm9oEyVYRmw8*Eij=nt7ZU<rce$)Tpe9zp*)({+wiPnEysrt
z%KsJKDNEN+O_ku)jJ2#;k}_W1r057%=`a*slt2m_4Ow4Dxwv;-?t_o4?>G2;iM1t1
zHTq@j=TP}0C)-I{SAz4qYL-K^qweE9?(nlNI~&!^GA8sit}QIH$3qI@ShGe{waqKL
z$gP6l!{HMFCb=eGSRp?HQ$W^k5NfTr!FhVBHe2dKhrT@y7@j_LUMO2X!U<qQcl9^w
zAMYGS1h^`W*Fliy$>29w+j$DuOrED%X?afH7nQAm*uGMru4R&AA=<D2a2dhUa`0r@
zkTJbBK6OSuwLJ5|P9q(YMcV`M$U<Wo(*mXYZ8CH?GyY$Y{x{kJ@CmLnT3u;ei(W?I
zMzAO`+bv3<e%#6GvtLFB?dSqKV^E+x%tEbya&0sX)%=q7bl~KNG#gIFFumkEV<7hU
zq)iJq1xI5@jQ(S?U1mj9Smdun_!|Kx7{jS*z}2gF{sN8ft`X>R+<Q>}*9|j&!o)35
z{$bev7y}Xb>i>Ao|77I<=gZUAg;co|EjBB;+t~N>_f;{P9o0(>CsVsQIY)8(M+{Z0
zX0uI|msFKEtT~ttnsz7Q&HG#`N0?_r9`7GV%pEN$8bm$wuu-KbwUr2qDS>DJ*c!rf
zo^{1mo!8969ZL|;%-@E-i_s)2-OJ(P&$+jIm%hVOr64opW`*}ty|pQ4f6VyiOYbe#
zc=({mBPV?e10QS2MJS$rn8*$>vA(j}V^5F$v@VUb2^BbA=Q!jU&j8~69bkn$Rr&R-
zO%6w?WLzx_VTtLj2CV!KGIodUcx>y$1FafezYvGt?k~pa`aUwZ1UZA|ee$tkNv=Cc
z{JGZu?=XZZBj@Ofc4+btE(a9d-5DgAA*&B(**VS-x*~(_)GHAM5wR(${3Wbl#jKod
z%!O|u`G9Q+|2vP8d@VU)x3xNg57CaY!$@wqG0aZHEbB+A(>?I5v+>m<Uz;jVsLJ*J
zW~+@y6zO<$tl?|lNJMu4hC=%f7vE5WJWMT(IHLy3^({EaKP?YB7ke1l@pqa|SOJ6V
zP+$pl?BYl#$__WuF_p)}{q`1!uG%lf;Sa^0L}IxMpnos@Sn`)SbPo_}$YtIzMIKdC
zPs>vd6ZxC5y|K}HBV94S<<b$`HYa;Z!TZt^q6v86x58P{wkI2UoRXG>hZ~Pm)?ev(
zXVwAnw+Kn2g}4PvdewlLVqV+5V`7&VqCwG_bUp~XJRAD!ZJ?#2yhSQ*$+5}6kL+Ty
z7n`c?qP19mF#uIzF`@8+FFFm+?x5Zv7Kf||U6sF$ie@YBg?|s!O5Yd)@?QvRXPw9W
zzAFH=P#$oWOh*kT(7*Ed?XvkR4WNH}In6(}n`hUC2FjAX9nI&<5ll~Xlj0N(M~gM9
z7W;3xm?1qmFD1O@Yps*F!w=>vJEt4sE)}`;KqRl-pB?W8e==fRQHbNR&yXn9-CPzT
zX|2+4M8JL)^24fW-lP2u?B&<ST&l#FFZ)D-uFKJC<Kab1+ygzELfogXQQ&I#70tnM
zy54n{`>H0Fw|hk<)fQE<3!TG@ud#0no9o!?oNKC!@}4eEg2_~&7N3qT-Qsh9Iv>z{
zm)0<yr)u#7_;6@3bvjJz+ggqdL`H_^?hjYdPzWpB%A2F0&w3<JB_EHSqr-ppKXP7$
zrWt>;AauX_U`)J@pMV`1Q@7$z#}5b8W-NWwOd}B|VY?aWP5mhB2UD^`MzduiO()As
zi*rJP!fC{s1^J?Ot$QRN*3s2*LB2-Q@>P<~K2Ce@@ChWHB0qvX-#@OXT6O<z6!)t8
zuHHuvKO2MQPf~XiwS5gw^y}|f;*G`8qVqEEj|<{jf0-d9z;+?Hsd8<$bJw!TNAoog
z!lOHQ@wE`4mlgY=k-F55@ok?~VYgZ9ii?83DUY{@BQPW0Kt&6>3A?52TkfiF>Svqg
z?k5mH0g<X3N|2TzLCY-?>EQvDCi4-yJ&q_ZOB?^Ly!aEOJF0~0B(D@5kKF=wuhD|l
zU+C#Q8yn1*!P{yZ%n10gV=xIA)5(n8w~GulmX=Xr!2;1WoHvT$;rkC#$)4|!vFT+X
zw1r+D=B(0|$vmpR_ND*2G9vq;Sw=C};CZDWLZX{Kby1`C&o%_Y=2S+BH<R2XLYGSE
z_=^|+ssvIdqk*Vve*U?%e-cz*yJPIOZgw1c$H2N3J9pcuYz={bj%-@o*jDZLI~QwC
zBAzvlhUpVeBJ9G0MuPyI?ZP@V^Kz#H7sx;!p1xk4&&-S`|3LqHbLRJvK%QV=Qkb{S
zvFK<?FJ<lVt5;392rQqy5=su0>1Ji8iLbg#!*Kvdbu_1a%bzZ^ESUW(q;MjVZ0*f;
z!p?7bS9101myQ#T`rwFer2+tPS}~s>qQQ4@qljMnW0?dl0Z$d)m31QSYb~j2ibvAI
z#x<niUH!DZdzYp5SBs_*horD3f0JK!rz1mnfewB4Gb?v?I8aoYjvl{!QI?LHC9;+~
zoO4O`Ss;?Z^Aldl-U@c;ZGbzXt2gmFZFi(IS+89!*Lk<F@bP}#ZyFV(^jK1WN90`~
zt=Tthn9XCnm;+Qjz8G0Z>*7Ne;3SWus}*&Z1bGmhWqu<dAv#}%Gd(=ndllitwoRIo
z=Y;_o^88@Qse8?9I6Pwi)bibdeYC$Yfnv85*zhy4D|RdlSSDYL0)*rCoha8?w<DT&
z1A2C)NtZt0&W>)}@#IQP`74&+aOIqCHv=OPy;3=g?ZXcQU_r3<9`^sxN#N(Sl-g5+
zhm%cbigdiDjZN%d1~Qg0ODQFKeyoJR`kd#k|IXA-l{D|)&PM<Niw)F*#IwjCReL1}
zYT#Hf>PQn`j?{K~c<F;FKn7kjrJhC8ZhL_tqV0FFyv`9W-^|cIJ!3w5CB}a^^%s2A
z!kZ-9CkOI{{doQP-#T1pWlE_zQTCS9g~NcX{+SG4b?_OC`xuy3j+4X;3IK1etFU|_
zwU)#L2MPfH`45d!QU<#gu>th)=$7BR;X|@wqUv5<r@H;<-%;IVy_9xGyk!FH?40`i
zh&xk1X(J4^;^f7B!0d@?mW)-O{}o#QN$MY^K{-1A05(IRQ<dU{w)@%ijcKn;(QdbU
zS>2MQW2XvUa)ii)_u`-rnYHKB?wPOvE8o>{*St4#(@XuD6t}M<Kjr}^p&@SEy-a^s
zicMrbiSmDPIzRp-EIO&}#9Z*`{%WHma(;pwaXmQj(l?ztiZhz)?1y!o?2<o%oYP^B
z6=Bgzb0(P_rjGYJ4x0Ex{g{L_v_D*Sp*kJ(f4vqey-LLa(*Ttf96(dRO5oVI*kvHS
z)Yrni5#mxk52&L1ST~|zsG-JRTfTf6_;CA&S~zwWFHy*R*chhF3+&o6)p-`N@F9?l
zoSB%vtJ+66h^$SOipE>?*&Oc(2)=DpfDruZ?u)O3FB}0j^bIi0vt%bB26EIJ=b_`j
zOA1%H`Vt?X?rOD8woVkiMxan81My~V*uaT99~-NAswewgQT>020z-bjLUXyB$=(5F
z9G~s*y0#nZkD*LO`6f}2YVh8EaNq&r+Ak|aWUE>bp(T*mSAqFG%OgQ+P>a@EZ6~h^
z>o?rr#(V;ElS_BiX$tkX(wtspX6DCW{Fg{%=6j`jPqn|>OY@nhl4Zk*<q)K-4WtE2
zT=Xw+#p}(Eb7^W#JwAPHY`QOYdJj{syXN6!1lUnlx{1dr@tqdHzW4wjf0jn517Qdg
z=!?1^@thZ2vBvf)uiJyS<5q(c)qhhM#%1R}>Uml%abT1~`U|=XaVE#c%g7PeQvGo0
z)-=6kkfAb6jV(L}fX;C^-057|Q>o|d{FD#WUvE&jnVy&A0k!CR%(MCH2e^R#JaS_N
znggoLNCg_gLo2g1SejKFNA;^kwYJ$B9v)jOBm2@Ez1!lKMa;9uhM-!^XPe#-5Da_T
z&m`v{tLN!0^5385dIR!-r_<Mz`Kp^bCju;?82igI@{uap{k-ljoM?_CwNSE!ijVV0
zz3^eE&kIb4-5Snu?08)+a0#C*>;Y-^pa!G+!0pq(t;w3<q&Bj#lW>Mmesfu0*hRc!
z9+d~O{*%Db-?Y1~yh*E<m_qESjvJr`{%e1IS9+h$W$(57K`FKllA9RK+##=SwgngC
zEztt?2qCD<Mnjl+!{OwtG*ENSHW;<0Z9W(_c)JW{tnWY8)wyTyk2BfSs+*cg7X2EQ
z=d9!RQ!e9TV3MPe5(d$xXXIzg%MsRqLI|_E<WHe8nFI9>+r4Eij&q(H_&ItXP8kY0
zhglRO!te5?*<|T1*jqku^}AWsT}=UkmNZ?_-w6}<0=A6MGLhUG6ZHH2?K<P9jEQ}8
zDC&66za{7|^II6Mi%l4p#)<PZPsg3p@u<d2FBtkyiJhz@lgzafkLJ#UM3Wkm?;XTj
zmT!p+XUhU%EmKC<2#;9CaPjnKoFz;fD2UKSp1>Vu{yM{NH3}k_`}6Qyb0P^HP783l
zsV!U7SEgrT(W^G`bA^LWUP5e($^kB;9^Du?Kna&UdyvR;Wg6I|skPDw-YJr4w#ER(
znm0{dZBp;MSFJjxQVq*<YpI?-@nYFvH8Z-zqQ4`s2@8mGHbH9ELNAkL)kb@!ZvOzL
zbaE~m)6AOw^82--vSzn$W^!$Licpx+R^5ALo10FnjySnP4)afW9Ejf9@0+$`DLTa1
zH`e)E4e_21xXsPBr%mY(v^fmgTT&&@_u<6rf17?ToV8FwGlSDB1?s>OEw3~CeL_5B
zXD#8)VY7!#1e-|^Uc=Q>)#3wU7FH2@(uewe0EqooU%ubdPph|-Zb{6lZ$-_+c=Fu$
z6$Wz0#!NBZb&#$q9@yuSlAOHCLTk97!#j{y#Pj;7dm7`?qvAWeS>%51QL`21@+god
zr-pamJ8Ep7Y5(wTr=3<(`G1$X<Jg6Mdr)LDm=W=YXEqk0J%gk~N1(175Fo?^mDiec
zM!sZqrY0JrI6@L<(R4uPysw!?#(=sMP};O-=5t2qzi1Q!!exVOpqdSv9IMm3MDf9y
zuD5@Fv(3pxMvK>!09T}tA7WAoy;OJj-=}>aRYYTSwU-{%bR=a0zf6?!E=9~z(>@6&
zK8C=U>}FrAk$gC=0rbZ&^TqERW<P)XLdr+5L|<EF)G7sjxbCO{#0Q@#47%PgmKZ=3
z{#$yRkDr5LjjD42;*<2=GR{(ean|!S`bG5m{QY(6P(ZIo*^`Hp{Z$X3(?(%!(@IPh
zS(4dY1Z)srZKK^fSh_<tsh*Vy<*<C9z#oj^8y;D|LHPEHTZ7SL?TeV#u?h0|JxM+F
z&v$a-Cm}G-mJe0>S58vXkI_a<dWy~KlD|J@-8tEqdfE`m;leCV4yK19wqxDYYM##e
zpA+IgG$D#dULW#0qAVl*HY$G&UF|!Im(?`qu;GVQWCXSgr075=y$@$)BfPSPQP`6|
zmkxU2MMw5&0g6{+u6G1DibM`6IA9M_94K+Hl3+~%XeZ%rpotU5r(X{R0jrbdfAqT9
zpnHFA+gctLP7gX6g{}tQ-p1!$-ZxzoaYY-zy(yIbD43H}_9niN20+k%wYhBhaUy~a
zogDY>^k#-ahKEh#r<;;>HWpeIB#Hg91m44lI^)OV>+6|oWTMHiMVf~DxxLd%!zj=S
zpkh(X{r2$k&`JB)d4s8sKY94&!OMIc0Wwk9$@`W)7}h?)^1ln8<=Nkg*hiLE-ASh1
z#JxLb_z8neh`oAU$Z!1k%|-|OK|tqCp=r0pn?ez<JMjL%A9Ps)>6LkUH%{PifZBg>
zCU|i5;<{*hMp}gt1~raLsrM+_UiP#R8v9x7jZ=rGn%OV;+t0WesG8>QBy9_Y_t~Zl
zI^~ok8{%#h2=bz$9n<5{@-Uw~H_y|6Le!wL4de)@>mmY=4n`dYz0}{_)>{s=YFwOM
zEl#?tC;@{231|1ZA8T%#4%kgr<NrIAVV3z{mQQR4f;-v`U0kxg^_FJ7eQ^{t-SV8y
zXm3N_iycbP9)oj|ix20U7BkXB+=cW-_0%QaOj|zyF*)}*)4tJ#-W!cbMM=t4zCCQp
zgJtNsV6X8Jtht!{zR@NyDX4FCb%BJYp|$utVtWfdqZeU2EbI1(r&uwtRXrqtYFOzB
zmXBU42b3lAKw;WVR{Az13Sfx8QU76x+49QGBSXAKJ;z{sLe}vrhwve$A28WZ?T0jZ
zl40o%_w8W`%%vOGY}C#$guce=W?{is)$3f64ji=tEC9M5H4%Kq&iJkIu;J2SIqq_p
zET#ZpEg<6wtS&1kL~O^oePwi--?(*H8i>@5m)l!C{Y+^qt%3m}fK@pR4zx&~-%Tn4
z3K&qP%<@>;9FXw4t=Pm@W5{x9VwAO)YuI~E2Q3T!s;_x!IYW#neb^pe)?OB5PfB<}
z=Cd?cb3=Ixf<5(D=sJI!eC=YhU4t3sT0_3%9obKJoO19m`$zf=$Z#dp7+{b|Z2r6y
z59t+iJEcZ|3Uo|#`Ca5`Dm8`zLebx_L_eU{g%($mR%g(AUWX>81#~y-+@2DhAxvM6
zi6W)IZMj0jhJJwfS%`LLop4suSKKl+eB<7+(BjN;fZ|r}%^t`{Md5%XK38MZ+U45B
zJ9wx4WlMja@ZtmP6zl7Lujh~V_yii*S}FWl?=Y<?z$7t%iq8ao)mYO-TKv7)>JHXS
zw*82~hH+$dcE1D^C^9^n@|R9syt#gfU6_0MUUmp_YUkc}l70g#@{)V3^x;iyMfwnm
zYA3a_=MD~U15x7*SMra9r4;bx$!CXWCye}&Pl~tvtI~34_NX0=!BN6ix`(Ce5`EOg
z)b=P~s3cD-LZ6J7-vx!vMCHfi3s-ysvEc5wa^C*XQbliv&wm#^vna_)&kp-03=U|&
z2G75Ybua2_)XsEv-FxuMG7JUa1bo}yJFg74OUMK?s9fz0IW^%ct)`lUc>!S=v%>u+
zoEABdTY;uWk*%G0?uFLRoSTD0pxP2mCI1M=hH42X)y&61S)UJ(Cw}21G$v}NTbx#9
zAWe(#@gYCzR|HL~0gxM}f@k1Sxoc7WnuEFXD+aoYlZQ~Ft472{lV*0DXxFhkiB784
zj&g*o=d0(m1tOX>K3a&hh`bvh_95};V}22-4}OCI(8X7<5f}o0$w=mJ;a#j!Hv$Py
z_7_0g_Fu-G3Az{&ygJ&?9K>{cau|F1&QBwt{)yx}1EAXeTLI{RkxWjvn?h|I+p#2&
z<Xj1j$#p-G`2F7-7Oy@#HQ>u>pqhEp8MN@z@L(k~ZHWAb?ZPr(Py<vG%Rj#bSgrG3
z(usssK#OWmk;PvA?+tE$JN8_JmBy4s^Uiz%na6R@KrwAWV?aUsk75bJS4Gg~egJ*#
zSDa_`l~#l`k$j0c`QPsY@KkZ7vT<6i-TTsE48EmAeu}duN`zwnBLgy|<gd-bfC5FE
z{3M3~nsM_cog)H<z*Sx#&h+0(z$loBLw_SttNw(-vrb`+(~6Y+5`3PnF8*UFBq4_q
z<$DKVi-OB)H;g%mf&Rb7VFr(8X7%CSX2-g6$-JU;*(}_lTe2NK`NW8{pdurrEc>yl
zpAn6^FL**iwePkySaJh__2YenuX%leBxJK4tI^K2n?C0{PWFefIDAg{YtaHx8Cbgk
zdsu>nL=N`q@rz#zQ44E!C)~(q=KmZGlNtq1e?fp_VZgyJumhMN{?$z5>6l%n3#s1?
zcONTLrjo|MCINwc=QwdFg5)TOpogInYC>Nj1AH)$2X+Dz;1hO^?L^wXs&oCS_v$gt
z0G|`eTFC1&5aNX1G9muYlVY?Wo0|w)c7;t1HL+EJf!^Odlu=z|U;lZ)B1Q^<h3o~`
zToQEu8aX%m_jYv%btD5X+cG`h!a`eYYggyE2sbAeeSH5r9ISQ&{5QY@@|HBO7CDXV
zU4XN1TjLC6`7-@|FubU*x=o+JI2r<Vfs&ns_yqAMH?O%)%+Mvnvvy#W!G?&M0bl<8
zK|y}2sJZ5>wU-aRY~+ml^&eVVBHKFg1-zihJBIpv_JqBxY#0+G3kGV$kFDS}dQ@vK
zF-`tD4=;WvHDcFrSO}_D%dtATihK+NQo&u{)PMz&O*cNVRWvYg`Rpa;<06eN`3T#A
z$qc=d-i}#wgd`NgLSaVss6q(h%;bv<T7~#hipwiK2?OLNB<XB=lzx#_5GE#to4{h_
zPJ=w#$|Pq61S!!1nompJ2c=`dGOzI^|Ip3UgNHM(Q}VC>kgMr|IpWoowe<AD>D3H2
zcw(W@!Ka^kFyj0jT0Q%I=pd2O-pi=F6Lhj1JXJTR@DN2&`MHAHKQU39LKv^I;DF6$
znN+CVjZD>G$ezHFAOWeWbzNZvm)l*!VQDtFXz;5$bArquU!@tBiV*&}zkrn1^2H{H
z3$+Y|SPU3KGCPnFv;!=`e`8`IR4#pm9h&mi_VSW#ZS4ab2P%Ldu~y2q1_~rOL9f4a
zLZP}!Jx1I1s3F&zo8p=5Om9XD_qZllQWrFD$lp*VwfA`9>q1^C6UexQ%5e6Y?<%Bu
z;fqL%n`XzKoETaYyGDqIFg1%353Q}kGi56)L8$D`W`xBi4<69M!aMFB4L3wkS(uJO
z$LJsHtP6yIDZ=4TqWo0)1R)10M7dm}0FD|gWz2p`8bBE9>ai;MvEJ)#pt2M;9Oy!f
z0k@NmPFu~hus$|gFuJqrT|T=VWdPyi{?EqLT*h46fEWQ^IEF&#+q^G)E)Zw7oQ+>{
zUi_4~?LCh8S`aor&A<}ksb9}_fx}k?CAj|GQ&h$Z3GYUkG~N&brlmBYB9dz=K$#`Z
z1P@Y;q-JGfrc0xym-g^&pFz6z{~D?zl?j^U{<Iz(#g`)=x*;ySIsX+np-F)AGT>Ej
z0Y1O9(ZS!y&H;jmE`tqD2Bt*3uAUFhSLS#h*P96by>%_y<g==G#9Yj;(9=^z9(dwV
z9Z)7{GZjplq#dVIJTnc>%+G!Sl}XXznHOY;0Uq9Nw1#%VP5{Kl2yD(K@pgoso>~3U
zJ#h~aDkGDT{|Ek6fVz7^A1|K}TRIJ8SX+-iu-1^@>ysXKM*N#G^+rChC|m|6+rIvO
zJQd>SIUXHGN5=lot!Ka2C~rB}(eUpKAq-56J}_LVx41c`n=`d8%J&jRt2=QqQIfN$
z2JEMJz?MM0VAc{|sOlBzC;yCF_!QMYbajrEdo3MxoZxT7swhMX#0bN$e7QAL@dT*n
zcT`cZ^eg;HZ$_pqHShPaHsAO{N<u|QLd7MXZTa6X*W>APbHN*LgQMg1-PEdGxRGg0
z!8%}2MSq8;hg)R3$UG5<mR1S!1bU16g~Ptcm9r@)OB>;XDg!>0sMTPD5&DiPdIllP
zK&^aDWa-~~)8DeCukYcBo<zT0<#;8eSMWTlA~5RWS>NoJq_vp#oCyM;brmBD_Kc!t
z<>jugH&wZL5x}AO^%;!PV70LNos_y^p~=|HW9MwpKkJ}pTtACeZvK|Q^!|g&^IG#5
zO7p|#b9L?eQ3qK6cR<|8Pk#oCwIO%dof@PRRIs4P4&cV6AKNVvTN0a$msX75wSxEV
zn@rUy%@@2fJ=cQ?J>(M|FGl;Jnu@<!7ra^d(e`T6T7G~S99PzrNo&$unjbar=-5Oc
zfsy67s5NO#Kc-`K%H5P8r^E_knSOa}or+QZ-GAawZ`MYMLl-1$L6e{JWAjTzMjcL1
z{aW1cb_#gYo1;>YDDAGW2ww&;?j{a9pwpVx-5}e_%HXMtr8VDVV=lO!%uUi~gbrhc
zg(8`23e=y-$QS}exmcZq#$OM2XR5x+$?veJ?r*NNZz4GWcUq&sj|&MbU_T|oj~y^)
z0PiNvVFwaJp>>q-Z=$L)BKu6X>||wrg)!6af3HuIR$~oO(KtU%Wn^U1h^Q|e%|Y3F
z%Dh}Vvo0omUdgq4_H9vtPUC~h7}uXH4&u6C@uOQbQ^tEQk&H}<FZ+A3@x#4r3@_HA
zqG(OwS2<2-bH0=|`#4WHzTKabBs_NJDL#PB*(*aB%}Ep;(ccD60zE+?Ne2AN8FB#k
zDx5BZ6h7fT$0r>sQ{hjYN1Z`FcQ=ll(je}j{KJC%7gAlEXETwuzuro^QRcM{w8f^N
ziL%8e$ET;op7o8SFbhw45s^};4{W+#t2-Dl6y~N=vrd=J1v31Q=sLdHd^WuYAZTvJ
zbtRt)b<w>7(UvC9@SAQ|wj1MdV+^D;-pls)0WsZM*BUEDYAekM(n<5l`Jnm2y-nGm
ztbx?V02efQuA;0(icKgT8U1!iER)%Thd^=&AXIL6U3vww_VTLNMV>y$II*UpFZ0TK
zX8WH`5gk3b=G$Rwp(zU}TJ;-nt%JOPHcm{+W|nPmdW*-Rt$qYsgZ7IC!)NE$HsT7<
zeBg;xz$T`=nEC@HR{5v|u_N_z<!7k_0zj2KSEi!_LVN@Dh4yimuEcoO>&Lb0OCU|?
zLz0f<qLs(15}0Yo=o28*TnY;P3|prezp0V2g}fdx_xPDF1<?c(JTfiuV}JhnT+vGz
zf=>{VdbCDiV6iO@w@%eGM&C7ojUSr=?8sK-8KYg)pQ*`rOyP9Tz$FDJ$C0j^3cWfY
z@6Xg?51`Gz6mTWVE^_yH2hayc=ZW89TpnJoGH!Gn-`~C#DlAVxL-4yG*5YSBo8jOV
zsH0rCm_A3OqPRvWeS8BQ5mTvtNgVn$D+6XBGob?EO>csR$}+PNeUDps@D&vYye86?
z+ehi;rQ5@jxScJRJCRwY5nlAgpp~wt-zggEqK98k9RpL&%a+CkL57L;$U+9(vQ{Mp
zvB+l}K>;*!*PQr~5dRpMud*Vzsj7U~q_Wb^?qk+EVms`mFOBg|sKX33=+CIA$i|vQ
zQa#5jY1f;Spc#J`Hw!zhAGtn^{6wG}e4ph@7dL2*O6--g-Q{U_GLRyfpbUi@Mx$OK
z69&2MA}f9wX)si15G&J`fZMgMcOeL~;a*91cp^7Do13j=<??Jgs?_cT^SdNeMI$D;
z`x4_7{XQT#Ga{r@Uf0yv6`syQToO=eY`_xWfQ`6|CX%h(>nnNs{D}O+7LVkp$i(RW
zsrSD#Lm}fH$pfF*h+fJaq_7n<z23KxD~=lH3=lU|uucoeBGW`sg7o3;{BD!P7l~{Q
zY-M|^6XcxN*&~my05NaGB77BqmnPwE8lz;w$nYC?a#D&-{y=xEs?yW1qfBIqyTU#^
zFVF2Z(8Eh@>7tvBg<NNxv&s`t;Ew3s4^CvI6h$OWc)o@NolQuKLXHNk08jI4&tlB0
zo(4i#2jy|h_6n<XOJR@FORTdI_Zgw5z<l5wJJ9gGz54E0QY)hHOAA^i5Il%>8Xnw9
zH&&u6#Nb8kQ=J+aN&D<y0?-*Sf1X`B%0|})Zvfc?&<Oz2T_LNcT%6V9m9noU_~hA6
zY~Vk}5P9@}@`7qSr|@!HN8lPgE}9hYQLEC467X0ta4&Gq2uK^MM3J91R;j*fUetM)
z3j5IJP<i_khZbiAb>~lb*qH^$ucqO@0G{$D54LvVa(8(^_QCB}YAQv1e}IgPOkag@
z_YdzR>1aCLdIv^kW;q6YTr@9AdU|@|I+v5R)>cv5-6v@onYk*{;nj{nLyWaC(_op5
z*?KkC5L6Vb<<PIWE@m`zyu4SPAw>OgELGLjgz@Y$RGql0q(>h=Q?_)+u8imie~DZ3
z1t@?WQ{md$8gL@Z^X{O=&CTt2D+8B^sOEET)KIFRU+Z6yuCcMxllAWM3gq;*&*tWg
z70k3he*6HA=~#~E4t9l+1*=8|L!n6=#$CQhXpN1He{zPSB2z!k-d%O|d>ziD;o2T*
zag!=pvnceJI2n<YGcnn!v)^{RIxNY`qG2_R5r)z-ndOm3*EN*}_-E#C0PVO~KnwAH
zxR^fM9uDR>M(Z`I$jI55saT&VP)Qu<6C@Ld3?}m)Ej2YHCMJGb>&r_Dg7&Z1x!lP@
zWmJa0^+Ht_-R@mCCirFPosV9Vkmrhn4TU3&f}whECtjH<x)S0gPD?1<0Ifx3aUT%q
zEgPH7Shlpt{V`%sZgFu`2k3R7T8Y-h{&bo5)8k6j-@jk&L=5F6ko=kuKC;l3=0@QY
z$O*XJp4@nx>~x_*QO{iQWg)$t)cyXjam-!$_2%6d7eujje)F=PCoFeBFrj&)3cB11
zyOnqY4o#FUy3*J2#Cnb7F!*)h!Ec#ZUC3zfWZ!v%0s-EX&6~nlwinHR9~oNMH=zSz
zcV2GxWFMQEA&$dUr$vC>ZpQ1!vFUql4J015JUt#RHV9EhB&ZhT13YegeB97Vy;R2s
z#H!!y@i3)pg=nmzO+PXguwS#qqBWMM-@;=xA5JRhb(8da-uXy&xThXi;cS&@(cWHA
zt0JKqT1RDN<x9=DxVYP!n<kGNT)oZn-SOika&mIfoIftq$jHdNHcQ!~X2Qb4$4u-4
zy}i5kV6nWH%R|2{r_V1gioD4Za!CUOB?&qRU9wM<>!|yfJ98dkP=!wj>TN{4Y0Z@m
zrx{$FjehN|8a4W6+nYqkGBV2v8oFK?3g}+9UD#bj_O2^$6Z|!i0ze9Sk3&n3<@g`o
zf8Yo>8EVITD-~Ks#!$$l@(xVF_YLS5&sAG#o0;td;n3Wk?;`4Be;C<@`KB>bQ?CK8
z0=WKsgY!}5$H0}RGa}}hUsl~J@)X}DXZxp32NI`?oR6lg=FrLbEmp?}$f^sGr6h)v
zMOuLNPk4AZE$y1S>I-D#cJ23ETr|JoACrYW1sW4$V(_sE?lT-69c#P>+Wb(Q0Vi}l
z+hWnK)@kwjz|Wtk@uJdVvhea?zC#3S2ldCna?A46R2((a3=b_WZJS%Y^ARRze{b)b
zxK0LDz-i>8e&9~h{w8uWsjdh<AK9s;r#cFg2m(?V86}?g-HoMDxnH0St6)ocegeL#
zmI;rC?c-9_zT&S?tCU37zxlO)=7bp|5uRPeVtP4-QAMlz>+b3*c=rb1KVKBFTwzDo
zU?Y-7%HShb;k6C~9T`y|jq-z=s9<bzGMW`?NT)AduKv@b;1$)_d!8qc>GYiF<Yb$S
zuKxHE?HVA;f4<m3jkB1qKG^7uPEAeCL`aS}K0XG3VK{|9nlECstIK3(Br}AFt-8AU
zN4g;XM{)6R3PB#1V}ox5b!2!F7wZusLy275K)4$4iC09yeBb*a7>a4!q<wJb<UAhj
zkRc72zu5jcV8w}b{n`v0tSRVjv6uWqqO)scXM-zW6jJAUgH!R!^@f~+M%Jv5vU$u;
zS_$lAO$ma$^;sP)ZuGFzH(CS`H!&#)RSxlpI4MCPRlujHkU!BEn?QqJP>>8bf;raN
z-j1N>OI>(+YNtWwVsyArcdYW7FtD}Uu;a&IV#{%2+|V|x^4quHwkO%6xPnf9YmGXg
zfBrzD;Z~pCsh578EF>sC0QOtSC*Cg&E9UHcSLG*mM7JNjZi5rjIQ9}XE<XN?GECXd
zt}a{xaY%Orm7=^nEfcmfq)e|#Mpm}L^Nz<u1sL}`N6Z1fq$^AV1~rOjbN?99%ci&v
zHf3bRpawRb&&)51gyg7%qE{e@DX_EJ@xtiy7Zi}h^0dqP#sQV0qVy*j!_S{pX_+jE
z6nAPE6m^P$&H<B3Y`)ji!xiw_UV}IKCy=hL=k8d}#~<jn_Vz{Ra;PsEnXXA8FCOG;
ztQR2=Nb3t692{ULY@P#O<mvuwYcM%D{xu>Shxy()%CNo{D^qLVtluCb@p}tMy09k#
zB4S%mZf>qho&v=OH*N=rp!1<WGMYe%Y&@&jI3gxuXZzFL#Xhh3*js)5<p>)Ua`wRj
zmAsYtiK!_%CjWfy_wNn4Bp|Q-^B=F*v98S*8eQ?4#YQMychIs@z^}m^E(F7}_cwB)
zMsJ)T>2R<<HijNHhR~p#58h{wYY*EJsra#BMe33~<2_v*tc<ho*eAGNXOn{4rV293
zUxH{$fI~({&3Bm97l@Bk-#QYxOvS~;3ui@Tq0k<{&E|fYBc>+i<>h7Px6+o61`;7D
z2I;Rn?atR6LEi(XT=L<7EOCvvbAN;Onja4j@A?g;pwn02ElB6Xh1qH=vPOx|pWlW_
zBIfxf23^4k-`yRxsH&>2-cd72XfRI!HqVsu?+O9VD^^xjMGMvZ#8wR<WLe3BU$O$i
z=I-jKx5JotGZY~h^*);@QDjZ*?cr?}9IVp$3aB|g4fkP&gwtTLMZ)UuMN{LV;`Ehb
z;u5p0KEIkzw~*NUdw>OV`(nnb`R_JI>w3Q*002();%M1>rrfZcX~+E&m?LpuzQ*Qb
z^68DNXvYihI&7_59S9*+y8Pza8t~SpI(A+}1Xi&<fQAN2-zGD}{D7BI;XxW)1GH+y
z<m3wgLWYs?cK!K-O+Z~&S9f)^i~<(3k%GJgM@o$b{cLzN7#z{O+?%vKUTIrhT^-?s
zN*Tnkp}sA2314x2ZF!UIzp<8U-$yl~M6aUbevX?G6C1rxM(%Y1?&Nm|+$<N>M3dir
z!KF|&HXkToogRWsggy{#342>g;Qy24KB+=}qF_-v?2dhzt^D$EdqD$2TCR(WQwzkT
z0DzR<RhK<m=k+mGSg{P`dsv96CJEJ4wb>{PNY5XOoY%WBSmV)BM}Rby8Si$nN6<)Z
zYlP(AW`Iu4_YJ@%YinzXfKSTGV?Kx|qAs_$w-PyS;N6{_V;3PZGP0fb^onWdMLCW3
zp{Ei)5=dUBGHk9CNy96d`NBTf1Rl0hy4{1k3@sTWf((D#AGV`lQRG-;RMKET2+?uN
z@Vd!F{$(_JLSR}tUw5<MrgoBYC*TJw<QyX;2<c4XHXplM_Fisu1?o<0AU1GlB<~-t
zmZMBrv0p`m(Wl7+X&dhPlh0~K6$$6(p1-qICB?-Cj4tyHP6vBSO?P{fMVWy#&R|~!
zC%_k*J@4`q)7!C98N?f#n!3X&(9-;-p!ptP2>UCM(*Ah1r{fN+5tvU=5C`#>5WwL+
zt$Ugo#UTLob1Nc?Zh8j0nsE2fV#5!ZFNQ2Xj?e4lUf<IFEiRVc6Gu3Z>X&R5ChQ*-
z6(`@}5dm&T$5`0dDCFpPvFvq+K}thJM0CCV1J=^=1SE!~<fK_|51!b}fP?DAJAwP#
z`GGn#y%sJyQ0&hE7gGbc)9*D+-cMenC2L^S<1itQ>*M{h765d)i1*h+g7C=51i$p*
zn-y|%bBoFzX9ZrOB?-BxAE!y@OiDnGcm31HQuyrw*A1KoGg|$VC2vFaZt({Du{xRR
zzdg+Y6@-njyW?hL*NyuyQ?pCH?c?|f`*oK}NfDKeh^_Pi(sk&It_~dyB|8raAUBYh
zhoB-X)9++|48)9rA$$<6`eJNcWxZH$3wRGXANDW!GIxN4f6u@!YlU_|OfFd?-s`bw
z*KG}_c|R=u-d}ms@e+*mLyC5EcSJ$V^lWeP-5RrmQV_sjXLg1~o;<D&DNdLqu06aU
zkck}G_^sh|5n#>0hJ}-GZLPHVO-)T5Fa7bepkty2u-EhU9G=F)+`O$NrxprMV)&U)
zt#SXZRMR#k*Z%CS?`?<-rWi2k>n|apq37r4pE(o%uu(>IhlGb?zI^%Jikl<%htK=#
z>uc->4BFn_-sa|J1GL^$n+!tMcrq#~tg=lYOb-t!NlBB^()2Vm@bPiv`FeZi#AK(B
z57)P+4~1(V%3*iN<U$^(s38DWiZVqJIkxq6w6_~rjg$t<K4E!Sl8}%9qbw~g9gDLg
zFX4XD61p?aIPoQrY5w?9gB{>9?CeUFuD*|Ke0;=9T#fTxEf23Hx4`&dj#}!#jWrxB
zSkl;FG|<yg7FPh#2EzM*I>(Ds-eIb!LIHhFPZ;G#Fd)y7@<OQ#uMvFIURwHVz0FyA
zAqtKh;m<iEWSe)sci{7rCd}eIJ8T-!s*=7)yDgI7i3i%(_^K>XDRGB?FM4T27~baO
z#-5zKunqkRV}POdZ|XY^>O8RqE88Gs<bN`AT_9x93Z)e3WfKm_|DIF&!9&1gZG*OG
zI*HGfj%aF)|1(3O*4y~hBRqUbKvQXp<j|qia}0@;#|J^cuA5qKTm@EdV%+fHKp^0l
z6mH)6P*uD!y}V|%NmFE&b_p{Qa54}_Pulzwgc(q&Ydsn<bpYwle1QR)9~BaiS$98k
zpX)@2^c_YWiBFKKa@*JAahu}WKUM601Q*AYh5kqlJYNEkuScSoQx_P4qEr*;!wAU*
zL`nlQkSE3`c*V#R(c?m)QUtu4&Z<$ZygXNd83%O1e#-(<LDoOo-dirt>*xA3guy>w
z847wcz^RTLp=?767~m!NVsHFtB9;s|&_vE-F<@-f#AgEkMFRskCcrwve)B@zIf3ad
zQ+3cgFJE@#@dWToFsQsQ9i>uX)>LUv9uau{o5Jb?FG>Lmhi-*bqB1SZ!>wo?rFbVm
zy`5z3)X_6@5PDwu9a05<y3Uz@8@B?!IE2!?F^k8XD7A{=A{QPC^<TKyCMU0Zx=u_3
zUdo<#?)e}T-d588a0-U}4iRMt&CAo^s|N#`^5);j)c$rqNJ3YHcu*f*g_)mNlmc-w
zuoelE@}owNdA<dRx4X8VRt$3e6l;u472T&aIh?qpDvOR>;B9juw?jKK3!X2Z<$=iw
z05YbN^%Bs7h>D79V|)5+?9;|A4onVtG{rT^c0?V#_sQyHm?_LCKF?Dv9`7v(DVh?;
zwu+GTaIEC6bKb&n20jQ@=-BL%z{D+3T7KP<*Kf=oh8KXX`n9<=tj*HR`Nq=2DLGI0
zv_C9%I{hsO((3L$ZL5-taqGOJlIpxsP6`~kaTFCdoTSV>MMXEPI6(<A{|jtVP@a2_
z6;%d9LMH)c9Tmr;G`B;3jHX@tHcW<*$@^Z$`M4YhFK?LL?i;<-)C+vo5=1ftElxi`
z^9TB4oRv=vj4a6Ge8674xuDM!Eq~M<8_s)}PeJ9|Phd=sSMX+KdQBbXx;8dx?R}Y2
zXAo&D%BT4Rcv}5i7OwfJ$>@JY3pa~;(brxX_`pz+-r(8s5Sv<4cka=UeA2MI$2xC)
ztcTm5if12%&wj0yn1N#8PvQLD{W`(4!-~j}C6Dwq(y*xhD4L+{@)e;d;h(6~1r-11
zoJL^C@d^Yu<ajMG?yi4@`rg>U8XBWx_SOh0()gsYHszJ2(+Js)i|}!hw(d}t3bPW#
z@zo~=q1^fo$dXTHo$K|ExLlUxBj{p4#JO3`L)3{eCwwAB%BwDw?S-dfe!Ft(bfBkN
zQ;us6?+G0NNHrc1SfvZh|LI|@M#m+x0ICkR%!Ur+1j$#Ma+b+Wx>x?tH8|`eY1vxW
zS#(;bhxbH$Iey#n^#8}yR|ZrSHSJ1=bfZY;p^@%7bf<JTlF}t04bmmuB_g17NH<6c
zNOwth-^KgA-@W%YoW1v|nOV=w^Nhry_ofOW4?FmR2pQ_ZVH4eEjT<Z|z;tvu;4sxq
zUcGf;f#~M_26(516DjzLIDo$)lnlF<JhUf#$NRljI`*oyJ2#VmDNBX|>^gREQSoRe
zkIbP)iB+WETB|K%iy^#OzK~`m=v~LxaH9-ibWzxP^kUxF?-EW(p(OI<xrdTR)ySyB
zLN)vFoEuH1$%;`?&1#O>@QY6<pYp}`6ztnpKGGC;S){=J%{*PUl~iW>I`sY=4;D}J
zje74kT~V?6b9$u2(bhZV8GwA~Z!GoYkUQAzDf9eTepz@T{Aufy(2nlBk@$AzC)Uc0
zYalt@waD>V`p3GG`;zwXKr&W~4qCX#^3T|jl5>q-DN344S$`%5gjg65Kz_pJ2Vn0c
zL_rC&-txSn=2~sPEkX)rj2vFy|5Obiv!yGOr8CQK<MNruoZgc@x1_aGBS*XY%QWwK
zl>G7kaynkT%q`s`br;J^OeOnoCL$>%5~T9fJ_|-c$eYhU8QijGaLT<Yd(8^_F6=9j
zyR2ee2q-o$_h|t3<v*5n#RQ^YCY24fNPOoz%ze<tlc*-W*V{}#`u0h`UBRM}5*?~}
zm6Uu^(f(y{ZeHZS85kD`zNYY#gjB@8>S)%MwFDcip1djSLfY?OwLgQ^ws55fI5QY1
zKZWdb#s>U<vWzfRxNy=-D6B~jyB>TQ94K{s1L^?nj|>+j@BoB+eed#&Fn%f1&`6_A
z4<9z=*52i2qA%wU)mFm+j`lyuyvbL=dlS=FR4$C=6D}qUOGQOuwGVy?7MvM+#rh(e
z|6RCZ4Ue?Jo_P@(W{KywTLNNHV03<}KVSAa><MS_Qu;aXgdm49IkkHPgFN`;Xcc;M
zO_0&+?Rg6kR{h_fI0mH={v-BhfUemnJb?QHw=z|4NH4ZBqayeTW@=xp+gbi~M0e*n
zGd*Qf3klfMg`|!6(@>vYoq3JfmdFzJ3we9)qc|@Y8SiROT^nGtd<F-v+bCAasU!1c
zT7tyqNEUVj<GnZ3H7sCP@xy-58S2n%dZv||B=j*~$upa-4&Z_6wUQS7xVTK+n9|JF
z2lZoHNc1^ee$$8Dgp7=gX2G6;o`d4g)9P?~ph7@>hL@KY><OYgm87B3Dk+V+Z-((F
zoX^v0yL#fOMMTpt@xI`ym|zWM^JQc?3%1|2imsOB!J6y|E(-6k2&XuxJRl`1lIOc8
z*48)Fk0g6Ud}&19v=%2droNOBo(~=!D7r$7r8bf^CG-&dGOFP@`!exKr<(`aBBW?N
zti2;)l;KSg0d~L2k~+QWf2KBCS+35-y#!3vzz}ARVu)%dd?A6uTLsRTVO(Xiy*DM-
z_cyF&R2<1O4`8*WsBtzow&pZiug0G96yMg^@S{InRz#1nvJ$Qol(Q)cABnq+&8^{U
zpc5;${|mMpkY!r92+J?``z7eExqLp>IX)u5mNd3ac;Xp7{I8!^S5AsxN3I1pVM*6H
z_JcfW%NiUq6`Nz;Ue~V$bN1P%hF>D`%e6s<f~i9IYHV<>t7ksv`fo>9{r9w}H~7T^
zebkwoP*!@T<Yv1#_DF!Cw&F<0*Em<Y9Fd7;c49m!D=4m~u2-=X@0KW#L!Pd!G0}?i
z{<GK!_^A(n0d~wv18dzV2qz9oF^;tfgFB`muzCF=KYw#lDh~)uDN82n+{2WSB9;iU
zWW9+`IwX8`u=U#dXjOg0qMgjS*7*XZV7rholD<MbuDGw6t?qS}{5CSM`VNJ7;VgWX
z*)@u>mMQ9%Twd=4tBc5U!Xb?-aB4!bH=R4@u(5GO@^rCdnHnf5qomn+&JkXlNf+cY
zlT@KJ`la|)q;Z5JIBJ#@Zd5GFyxSJ{ZZnz4#UD8;1`=tU;S7O;lr>M4E^@s$RkPP`
zt#B+c{ub<E{6PxF0v&mR+F4;k$rtC0HYqWUHGf!q??u)b1#jZ=gG;!Lf-0=u?m}$7
z?F*cD7J29PS1UyF+YsgA@9r0u5pwz4>qe>JYR8=-ju-Qi8J|p_O_XdVW+1Ue26}=-
z?ci?C$otdW(pS-G!{Se~8dmLW6=NSPb~jen$Vd}4C3WD%zCaws&!z>_{dX;r_|<qs
zm8zNYUS^eKzJWxRQ^*byN^zX}QsN#6!BJnq|Mj{?qBB=Pq>{l!e7r}J@7Y{sfiHv+
zRpk==A?WX9TC8VW7?XkdZt!o<<XcXbY#lk`P;z87+?`BJ)kfI$QgNDvU1^YkC8*Zv
z@gE=fEZp%8W)-`PX;2^Drqi=H#hYC&`8cgTDSQl9x0|5SH!-bWb>c0sZEBygS)=7x
zk(CD&J6-9qEY=znZ<*`={bmvd{w^l&>**Or`u;^i(j}f89)1uxI?DaDh`P{WyvH{j
z!$T_c9RxD{otE^y>amA+vLt)@G%(N*a+zSUZC~X64Uc9#dlG}BU!Wx8)eGIWex_pU
z*;bk5a}Gf%1RHBr=T8hI!a?Mb3I`5_ur3qG#sHh&kEJW-ApsL*hLftM0V$*}Z^5&Q
ze{z5q!%Lm`UXirC&zjJUt<yu;XCg42D_S8}Mvp&%r&+U()czhhh@QnFkC35#hAS|I
zRqbz<MmVS>pg>lm`H6#nuBO`P2@93L8NjnfsMcIAQP~?rj6@ZSgOnwgQl<s%f8!%0
zsaZP-#Y6w3=Q4nj5xvg6o77Yjm@kf1C<ih~?dLVbtfpk%+iN%e=snF&qn+9(juL$T
z?_XHTJ?r{;o19h-jz30o6tymtm_!WWwxRXwRiqN-pQkDnlw!e`C47m3Brtd$=G5!c
z#RETv##!tzKqI9ZlGKC)4rHcUyO3pvCGxMUdBvMmdVVPB{5So%#dA;3lGUdoTbA-!
z!Mryyc0>_+nuV+2o~RV~wE8oi3@dl`>BVMP!HzkfgCqB$@npe%OXyQ-`7}GE5GG;}
zAKv^pPq}C+85d7;6Wk1drcn?Y6gor*-9R?g@vJclHX+u&|IIkV5FuU0NJn2jD5e?@
z*dGYQk*JTCGias)bjd1vGK*mM7`lxbmPmWZXzU5lfq#J@L7~qAQsy)e{_S8Jh4F+L
zPC?OWm4=cedmcHMW9$L=|MxVV2uKQz24;W8r6A%hWRR)Qfk+aMq)u)I*;suKP;ARx
zR5YmRAy6@`{8M?x=Dmd0dC57)v&oO}mPvEt4gR}DNll;*u$%<kOd{n}<C04|T?S(c
zUM`0`*Go;P{C}V084U{fB-Q_v6jH^!`2Qa){t62egf14BVYZ|tH6;MJCKHrw3num}
z6_mubn~9diwtc<1>-IAH)>1sR320UM<hZI{#VKv*LmrB~Lw>J9@3I^9&=QyF9}KJ+
zz#B_Cpj`Wvj#2J!LUVj2%P^NbHp4y}JPA2vsa4_I1<Cg<bMJ6CFR9Hi4Su}^q6&EC
zlo0f%mdc|<gB|*EL~<78hMqY8C5pH7xMMl@AmHDmPbt+xI(-y!THhyFX00hux11&e
z-|_o8Ke8WN-(~o(xoB+fHVrp?n~g=^*Gac7URe^sl`9l5(ecTpe?BEK|4<_G)2TSR
zm1xyI2)!NVS*i$@3vZOqoa|j@C;EP|vCQa!zTj?||791a<cGx}Uu(*HI}z(CZKHri
zvnAa$u7jd8$P=Y$JosKpzT57#hzP;~IEkyFsy7;ruh8avQ#lfHpPDjWSM3@6uF-z`
zA?)P%sh0J<cJ_>oz&;+k@dJLV528cDK6OL+0q=G=e`EHmiR=6t#TC*YG6Ie{vSn0m
zR6z_*3L6NF4-!JcNKj2a4(#bK-%t6Df^NSqKFz32!_1GOE%`J)ZLP0lqj)S=9S<0-
zj*xnvV<=o21UMZpF+Tlk6uh4*Xp}NxY&k_d5L~tyj^0}Ko%HB3N#l|q3veUXKGg)}
zP?6m{kfJ!f(_Z17A5G{^dSgA^LY;2~WoIZn2X?iSIwmm$)$5NhKIa0e_cD)D@QJn?
ziHVL-UtEL_DXtmdiA8x9EybS;+<xokN4YFZk6ykA*Vjb1@l^#?o<>O98B*)$3|PC*
z+tLPzo}`GlXEk~5#5X5=AQic5&iRXzue+9naeLGF#Acl#Di2xhL?EaR7=HK1X6lEI
zxBuBlK|l^${hdC394=ZY%$WjJD!<jQnTGu;ttn{@;>Jt_IzybqH(Oa*UORk0BqMg2
zV%ctaQGVXnp47$LdCC#!&2t;C%8raijY?FHd{s}p6khV4H2$>bXG%l?`Tg!{Gc`ia
zFK&Rnyg`uS@X8J=S19Cjq?#lEXOgMVRs9|ouMUu6_8P4dsYR|{pDGS%eJU-sY5G9Q
zQ*oC$T|s@D=Ul)1^WSf;4FtDi^%)y^<3qIfW128IoC*NF7+1-se9^byv)#+U64(~W
zv^3#1YKl+P9FnM1o<vIyDHJ<aWSw^$b?S9hD+e$mAIIDU&t<X0KB?cWBR_=CI8d`Q
z*1#KM)*EQK-N#XRG>sX&weD+l=D8?tcQLZhsi956Gy{M~z6lfoZP}ZZr=6bfZPi)+
zOQ1V&-Zvo+Q;kO3Y(3J$v#cy6Ma&c-9yLhybBY0Q{4|1{<{0d{1e)FoPZgj(j}})o
zrY#@ANL}*hMAdh<t0libhe#-u$D=vLtGuEY=BW{9{K{!(V~0^^iUr1Kk>y&P4W)<<
z4fxq<D2mC0MLf$2f-T!^&a@RTAh=hNu~5yBwMdAtvgaW8dM@;f@~frTZ*A}vrzOHs
z)5!f%;``b#!0?z;mPcSJ%wv*_OIuXQHGCvtGZ;DuQwsP60I4~1=SC#gjXA3~h_Nt%
z9n|5r{dx2B<*z1I^7jf{7Wu-p`k5D9$F7@YrAdjrzQ1f-!X7Yl7tD<=6b%PFWvAY}
zZMZ+n)Ym>2KLJNEFZwG=3<HFa-!aPMl%|9U1T4%TVEGbge{FKhrgEeCyhYQ)ZNI6$
z8-u<Uh{>*-jnBi;S}mXMDEbZgxi|HGO`;aR4IIgr+DI;%ogonH{Kg{*pJAh`{og3~
zT|shEMick9cPVEh`6evYTcNN}sG5XoE8EooE2ONk^dsP~zk3VQ61m3=H`xE>cUr#p
zVUEWv#~U)>53WnjZ0uXy%jI5aYHTj)c^&G>3Dt%Bij={@l4+KvdO1)qijz$P84f_l
z$${%KX8DSac-&f-5G4c43R4>nL;0?~-ZnzuYx`+O4p_**l-^}->%{_PvgC)CUvac<
z2o=4}Pt@z%Z%hB_{Y&GYC$~O+v;#KW&^)JRnAvKh(W)JdNZ4uH>zxYCR}$)({E@Mz
ze{Btg;x3zZRFWI=Q@D8evRwHtC`$0E2W@&3<!8F?jW<2QTm;FKhquiKqb>U#X#MmB
zQ}McDRXe4>m5qu|BsDjhcDD}Wfkci#%TrZEZw0j#Ox=}L?9S7EW_18o;^tEM(yNyN
zca0W91k1joK`4uhIb-bs#ve+I{CR6_kQ;GiXEFAb-{05oS5=PMlXe19|LsDdd!^pj
z%ttz+&m1*sz*!g<8Fgr&_Z_;<_&%LgE&s5eHu+tOZI#1%Wgh6MbxRN}c-1nwvn%pp
z=X_E5z5k(zegEn_qw$|U1kFwawwp<^0KYSd3KKdqO(x0{$OQ~owdB&rMPE9;#2y9~
z9p}dj9$ZXT!4nQXMpg))&uJT7Qam1>wnoGa<%_yn5V^XFs&=H`rv}=$oZJ)IXs8-2
zvfs};eu10ui0QA^q2rs`JNz2p{lM|Q`>2~beDJI%h+D*;r{<|@rSO@Z1Z1b=h3Y*8
zbC#Q?Jr)HXy4@F!y`r{?+MIw!81Cc2#<{0q+oaUUByNXO6rEe@|F|%>5nX&0g|e!1
zR{gD%hvgV*N=Z9$Ic?9{)FyKlx6yZ79s2nFnm4ng9Zx1Fo7s)>25-S%m|U$o)gahU
z2RnM3Mraj5MUXx8dN?dF<g(JX9G4GE&M}>93Gj%un|-eLxog<j-3)8*n~IlhZ=DA^
zo(C=SfHK4)Vwu=>?IzA7?D#0(FZ!&|w?E$ab11Pf!RMeHXZpJ1jTyRDTo-j<ojp*)
zIPkYM@pDZ1mC>Yp)RD(-!2d({?K*4mqIX~{q@PJQz^nY`)!_GXYbR9(7J5j69F)uH
zfmuDazXkR%hDXW#r2DPBlv10A`RUZdSpV8KeSWFwyUwflkR{KJc7N*4sDCyKCRbG1
zceUROZYepirkiwH0VB=@Q1wxcv=dzS$;*AEopXM8`5&ppK4Cb(@YU_RGat-UNjLXN
z_XZVFTTKHsH535EQkH*{XAvJW-y%+r*E2pOMo8S~S9&?tUW4+n$qxDbkzZoFmSKo#
ztVL?9AbWt#Zu8BpfN2_K<esASqtZHF0li>Mz9GEjsq(zV=%4T?<CJyMDOO52Rhz8r
z22ayiAQ{&tC`7SrL&aW?0f?>@p=~;U&#wLe*(oTG4l@HaV6Ya)>~L5OlP&GNG_IL_
zU%Ot=2zmQd(;y_j!I)tX5cx7P5?)eMt@kQQiTx}b-rm<turq-KcJ139@8jK^pUquW
zvzKmHF9W?WIqft%8)@(SZBJ7OZ5vQ+X+EQ1m%@g+{mO*lrClp3yPy=5K8Da6(ZS~k
zXi}MzLlR$4Dmn}kO*B8WI4}Bg65xQDgOJZ#E?)Ndwcp$_c2zvHw+3zj>+JT*+Qaui
z1jv1{$&wV<Ua7zQ31{{RxVxZy%5)zEI9>V#z*Gi=bRmo7#B$C_i-8_lpPa9}C9SKG
zq|rtmIv(S)BPM!B4{VCQE4KHC(sr!vZ}PmK`gZC1i}n<#fO=PfeQ(t4{2?DUpQnv&
ztIRNAh()UO_S$&aD~i{u7IR+(ey%P9?F@q@VWuuy=HW^?jrOO%w(BiRVSmsl$N5Ae
zY?`x)5M6>0GI$*8KDc0}v^Jz!=G8d3SD64~)XJ6)3O%1|!Bwr{RYNRObE=2FN3trd
z-D^wQLYd@=5)70WD5t4jtKv7m#|UV)2Xa~Kh;E#1>{&-Ye?nB<9uE56$lqoyowE)!
z52?EzbSuH`voZZS-(R{+<z5C1VpMrCY%MPDUsZNi7m>dWq|hW(pEU^bfAa#Vwz5bQ
z69G*R66=Kw2R@7lec*`@(<4wJ%CRx8XV10|qV9T$zkQwmCh|hd2XG0O{U+~wB2F@6
zrCV6afi7U>!N?+2gf@(d1~67d^qikd<8HAA@JhYdwObW`gfoJBe7RUkzFfV2^Ly3h
zJU@OxevZq<<T5v0{AC}zUlmXw62%jsUyoPI#dLQ8sM<tV%5-dGd{VUeYAB3GdtZd~
zM2siQqMYl`=@$G%3?{wuG+`}f<=TV1id7SHbi2|{JkVC6_>LsBJZl-ckS%v_M?DX!
zhl3e(^l|n1``FCy4?HUFqf&ipXhq82;fE0a%}$7#LVdm~yjM-{#q%mm>R!+C-;=|U
zHpek&=;OSIbmg`nP!p=b!#DPJ?VS~S)4^WC{iw&I3bRtK`VjE`&g^CE_Dcmv&CFtF
z>x*A_lhYd?dkK@}(u~`5&ci2_eVCV*-2Sd9nOWWo`zytIK_0kC(~I`CPpofZXO6#}
zvVO_6oUmF9T&5d6;5BvGcCKw6G2{}x;Or0e^hXL2A^iH}x8@t^meOeP#OG)*WF)}q
zESFQXz1-w;^he11iQAN;g?=D+k&uS44FCBCR-T`~8MW0C?X!{f`g4iuR5E<sP9xbq
zh;yvuP@advL4ZH++;zh&ap&g9(=fx@+EiDT0Yrn!F)?Cn?$o0g1wyMb7GM4;aCQmW
zNY;#DE$_J;c;F`4<F(yuNBV|qFeUdSZe7VAn>F|CN@S&+i^4)BGw)#B|2R#gm?izT
z)1k-_29I!VJg@4N`~9O_ySU4H>YOM!8hzenrma4ffJ@y*S50!q%Vtrm>EC8osXiMC
zW_$m+7G{?MR6<Z)K6Ejb^D|4d)p~@jTOUvBFbWyJrjI!j<k!(=eo>;2@8Uh6bAOpu
z)Sz5`?;Oh5<aX-}1X}`Z&K`iLzUL&A$FMe66%*jvI6la^!ys;Xy4@2c)iJ`=5+0+@
zpFUb_9?@-_;==t>F6P7Xc&wKizik+2R=(}3T8T#g#p&pwUZgJjy5CpPnlG}P_E|-Q
zqP*L8oGbNTvZHmIjVIw>e(`M$Q3gsb$;*6LP9?>ii9F3LDD=Ch+7-|sv3Z=<{+(Z>
zS1vPZYp%-dCU?YPVX|FaTWr^;W?6|!Puj3<e|Ds+F!MnNSD(XpzAWVpKc>jVC(qJ8
z!W(@J`NbOJkmaWZlSXaQG)$-_?jA$ABi)-Dn_iT)I)8X0WnQ%8KW~+A5|7<7bm-dC
zJ<~{D`10IX12wR4y(g|5r=9J&5?SM8O!e3fIqJR{nB53SHuK-}&yQ2F_&wbZ{%$Y|
zWC@z2@xfL_GuVw-_+8?gK!DOaa_=C70m%DJH$Gsx+q%-%zS7Z9&z}21##ysAM}kq7
zq+=(3GilW5H(4`%oQMe=df%e|%;k}o&{p}dADJAn4nt0qZMMw@>qD5^E3TL{N!963
z-jTLjdE1A15!+k`{c>c<`*r@+ZK?y^g?iSe;;&q%+__v{e1EKQpsf(?W5H@WtUFJ*
z_Bb@N-fgVE&^M;pSZSg-NMxW)+q_nYY%;;t`?2BH$#<s!>=H!`-LnQ4&K0wL7`la_
z9s`d6;ccnQBKJ0*J^hC{NrIu)`kFV45Uv&-qx5YpES%#T-13!8IidOu0SEe<^OTR%
zDrBmP4UYG&%M{a5wjcM(EF9+S#0Ps9{~YnUKxQ{%%c4Txw2R1#e4i*#hsRT`#8^}G
zw*FJ&*QZK26@f}o?%4LU3hs&iGh0?cOLmvQFCj)oiYrR?Fm5jUZQpL6GD=#kW%se=
z#_64~8KuDkt#!G8tN@I^Wk9P*kp0Be9N0Nuq;KuApga<ko-8GNbLiec{j)>+o=d?!
zZ=Da%t?)#$=-rgO##J;RYd`8KeJstbf#iBdxb{Z#7<MTOA!3U8Ot}rN5mznHq*CUE
zqY{w(IMc5fT_kwd{an!oSC^GtN!IY2Hr_bh2ZOyH=#9}&NouS{5Q{lSGu-9iB){=k
zDly-rv8a(z4gArT^#Q)^(Q<9|(sitb?dl{(PRD<Xi@nW3MpnbwKF@&&$)}9}>=&cQ
zy1?!Jif@a&ch)nM<tNE?6QOw*NL5Za{hP`|+jH|l8<WA{a6s2Qk@qBUfNV0=jfBsl
zO^1^&6$CS0wDF6URsRGX2%Iq}J~ltvic^K}b@*JZT>r3h%=G@&0DF)GguC8}H)+E^
zxT<2?H(L-%4bA$ErG1@(u?dGPKiv<BUMgJh`l@(mRexr!3ui{{nWvqG0}u?jdcUk(
z3m_9I*-wJjF5RkFR$^?R^SNvTL?(Dx3(1+mHL!?<7+?_hU5*C5OP=!S2r`j!G)W>i
z5f=L*cNh!)DI_3?g%U$vE?;%P#q?nRV7f)3DRTXLxRX(?t8<1XO)I%HC37MO!~O|K
z<*OkTQj~ufdpjSk#{HVsj7P=FRHqZ%*y{>`57gwlz@FaAO$Zbits~=gxh|8g2nI}g
z;2rWbJ6YjEJxI9T#~UQbHDI+A;b<vgyhv%>MC0?l12|OQvNxR$T^!ytNBzG5%q8o^
zha;OWqZ@$frf5x~syO#_-fEZDXjLd-yufU*?Uw3RfjvFw`Zo_{_mmbAcy#F$_I8~V
z+8#y!tNHgI7y#Y2#7hT*fo0x?qo$c*O}UO2ixe}5^jb%#ztYe`IJIm$dGwk727HRL
z%3GZ`8qR*xrDuCxpDVys#dEuBLvFc~Vay&_&Xf7?5k?CaJTvrP`vj9eQ<caqIimcJ
znF3Mj9zT3L+Z9Dl(yYe;`nK^asD?p`zF>g^?wGNzx3uE?(}&)U?dkBnZdXR2jsAKK
z3Gycfkx(;z>@#_oydK&Pz<Ra3E1i`Y<}4D{lDfML%O9Viy6k*iTs3ZkNsC{9)LXUw
z?bP4we0*PmWG^&lb<ifVUG>^e#}2q1=a)qw0x(f%SL{P9%?@IJ;30eR=@~T~s*i0g
zd-*<n0~G}n49M*tAH930<pSVETEHav)uVm+je-;C*&J+V+3?)qeUOirP1n`(@77n-
z2<BOub02cL#J;GjRnnv>&#}x*bJbWJ75URG<U5hw*ZpcwDmbcq!NaF%%wqrAbg+tB
zP3Sxb)7h&NZYx^T(dKgmIb`oeA&@zG9tG?#&6i)-#Gt`f1_#ZnDo>SkQkh<IyrJ;+
z@G;|QcKpHa1|QjTE3iv)hJE)X&h>KM9;PgnMtzC0-_31Wr6c>fe|PlBM=Aj>g_i^1
zjAplXSDG+A2^Y5?^KUcL!?O1ULf+j;8cjRH+oXF*@+0;t&s}{=10|Itm@1N*U`*#_
zJ$cewgg^f=wdCjS>jpsUyIN{N3`Af<Qjj1h1nlY$-|>Cp%Y{^c3|iYuy$1Q}O+Xb=
zv&goYGLqK}crEOe#}&g-(IKYlrv#L*tVsQx*DGs7L#jfHSbeMSzjE_kUB4abDf#ht
zgilfH<Kw;qsgG@$_vDMVM~hKnX1f5KcfL#PF~Gvee6E5+XM_~r35)Ej4}aWGTDIGW
zqTYx)^kzG1cuRFGMT!m&fEZ8^keKt%jfA~uCNe9j31kKsP|<;=PtF_nV#0bs@2nNS
z)Ki}rUu^FVB<sQ#{G?g7<j<(lQNFkRxa-Ll%9SF$n+;Gw!viIC#NUt(cS|cZCkHT_
z<ZfD}$|<s$Pfe3(*UIkEf%4e%%VM5m5o=M^8@(L<f|Gx(<C=m}ny%LN@dmISsm_ok
z-#RdMO$>13d5&>&OYKKkNCQp;`~1iv%R_(jk?ALTH7lrTx5!4D=a;@B`oF-vcvrT9
z7&w6xCUct}3$C<+pWPMKKzi)r8AYXs;D9I;<bfy%s-g2S>Pw%t)+e!xo`I<Uf!6cH
zB$M}i`}_73^4CpFaiE$KIJ5c4#V>LuSpknS8Q{G@3k?;W5@J@sNN6N{@>5t*A-^>4
zFHUNgkJ;f=!0#11%qg>4kvZ(W*BNa`*Edi`jU}CCxj*edDCK|O@tAMgbR<x9a~|DW
zOLFBWi2&6UqR_ebAsx<em~-};m5+!RfcRhj*_d~17jn6@{a_it!4&IsJ9wc7x(qy=
zW`&qP00Ghe%`NP)LhL5x!?NvmUK<huS#bz`xox;rltc5jy{~ites`~33CL)~ApOr&
zf?Wo~U1C4XLL)#%$w*&b9&YvWXVC2HrEyk1i&Y718ZlKaQVV|{`=I3^g8kg9Ug6t6
zVxdXiR9#o|(C3s(59$2j(!!$kLPIrSE?!Y!ldj)W5o)VDq2~e1Q-A=d`YI1fVVywk
z5B@wDGFmXK28KO|uq>~6cHXT~%LPE6B%Sd5xP!Xo=~MlNMQ1sKCHk{Dilk#)l*sZ|
z_iqPbIj*n&nLSI<iUYOT?<%{?1&fW$60Qd`>RXhkFv}ic)#KC2S8hjk70I4Qg~Q2=
zod>aeX$Hm(B!yF|8acSx>KKuowRGh`-B>co)1JLuXg$O6%hSY`50G4bnaQJuNPddB
zcoVl*FjE=Z$lP2(H>RlBc=Y*o71^QB<;2)6o&`Z<B<#7evhTaG`*$!vKtTr?CQ>e9
zh4;N+KF7}fmJyfTpMT5k^UWd|MthAR9EkB&@NM%y&mGZQjHmzQdW^kwx8=9pDEn|K
z)wCbzhWWv8XVIkbsAyPRRTegdBbGtGG}&p9i;gW(IXw!i+tQT_y%N+{0G+YZpBd88
zS;xIoVl|IY!0in0_jujqJF?OqVDqRMQc8ZOg%-}EPjTK6tceQOwfU?3&M|;@4I}^d
z@cZ^H62Fea9fR2hympi;-INjhUzK@=7f>2mQSZ4yo*!^qHcQcJgG`@3^{(kPAw5hr
zjcgepQBA(2w$ky2A2X<N`4d#61jC+i!3wBP(sB#+yux4P=_C6wncKaboX|+Ws^Ja#
z)r<K1RL$t<feN8G`IIBHD%H>d6#qXZZ-!aTwLy&|K1$YEhhVXFJ8>6RdYHDNrZTdS
z$J!4agQg27wCD({lnK84LSK^)S|=lYUkZAam)x$+hf6%ZUQ?>!;WPFq5_w{k!RcVk
zD?5@x+^3!<tEBbLe9o@@@92ZE`92-Gm=3zFJ7WH)c3N&T&>?-<e^7?6g!-Of^GcUb
zwo`B952AHB(?am$&dCf%atzShc|1u6?J_XAE_xNxYMgpr*;TO(0=*ijDur9`TMjSK
z(8g3u)_gj&660|kC~1(m400Dt-jGkI!NWO**&(evrmV&PBDi5Yo{IPj!<<y9d>!eg
z$N#u=vG3RxI5GqJ7bqFD9pI2C6YJmtPAC#{)xzxfU4~;j%?5QkAkk#Ed|;%Ek<fvU
zXS|;n3ax<p`*ep<mTUDrq}WW-2U{Swt^_N|JW?I$w^WOBfzZ?tX*<>%i@1X+EQiQ~
zMGZZ|UQ9vDnU_k0DfvxG0%5u#R;qeJ5^FA61qwZ+5$^8>934ye^tsKmVv$JGlh=AY
zq6N0(w|6C}Q<q6k+Bv$Y*L!hSALmVI$3v1%YJx^@zN;{<rjs!y=%l_6<G@O0{Egh7
zam}ea!pIX5+*f1{gisEh(8AL>?6Th}<`hGtjslMUz8@Q(6dp$#R0|h;k?{xsxBz_&
z2mkQDO4#=<w8ml1s1^+AqC<qcA%qIPIO}rTtF#S}m8!_OSI((fwUoa%IS`;UM!!!7
zFDmr~+#dlks7m1eZ?*E17-*dFcvjV3e<<`}^-q_y?rjsnC`-o)yUulkofo`h5#i{Z
zsf<y%qTc-%?kkJ?9BqjvBMee+R@2{E$89{Fw~J^QR~*8;@|Vqj0Wx?X#W15^lV_7V
zEG~0DIb;~NfwW4aUV{i;PW}BFZ0~J{(nbH<y`n$_YuWiJhu`g8ABIW9)lW@p8)a`i
z*9jVFd83ZL4A&}e{GDvzKIs{9v$nfP0BZuGyx-{SE$Ew~LkJ1EOw-q*(iX1_@ZwZM
z52m)Mze8X6#;37H_UO?J0FLAhtRVcN+B=9qAdT=4Jw?6!{RXi|j-5n2Nhm$4)xq0u
zyjEeWbEuJU^n+|QqEzIp<Z!PUya1l_f<(~Irt7lx(qA=+pYmyz8gG8Q!Nm=3@>(!Y
zS}sBxvZR-XgEK2`&3Tp1KFVi)yS{k8fryt3YrIP;`2I3T`>4`}=^huejP*JF`F>(q
z+TX?T^`~Eyb=Ik}5MvSp<ca)Ys5wHU98CBSgHD_KWQbnu{rX#mS}ZTbS7pZ}f%<PY
z!c9<qN}O8RbT_W8KSCtl#`#S_40xgwKk~>X(1GlD$4j9}MS;uq`YRx>;ngeB`cG4(
zjal)t()b6ZwF5ReKFchnGAekn^!pViVP{$dN)eDP51zGO0&QFzgvL8@C-IR9y$7V=
zZr{X`$Qgvb>&35}ly_0+R34{;w><WnJk|+61=5oy4ZcUahXqy&nZAj$o~u!&#w3~z
zy9TFbWCTSl62^KYnGq%A{^Hi|H^g_*L!&yvHlCNCGS!Moi9*P*8Y!7*JT7jT!;}n=
zl?IE4#WxaXI{tMLpml}A${Em7YYW<nNitAYGj-@RNAHfAElImr+f!JuP||HU^`r#(
zQbL^3sg%bjCICq$F+M(Ch8ic8&4r}C8@7OrfVo~$kh%F+A5GxEnIc2?Sd6*cPza_F
zN1sUKryMk2=5MAO0`w3sWfPNCz>LBq;f>4AW??be;1!RTdGRz5NWv@F0T(;`CRCO$
z#Mc?+Y!&W)AHD>?XEP}&rJw_!7W@{|{%m1kK~#g)umvlXE$vk|H_OvUof-!$+!h8F
z6X8(S>+8Nxo~Q*cKu99$DM<Rxc6&4{J3D&~N6JuQh}nc-pe=ZSVa>?hn50%=fQ1P+
zHA%qcrYKhMZO)x}R0%sOK3=jZ;DRhPx!HlfXR4~I;Fph}ZcW4@(H6IhKzWFzXp?ez
zLp-(+_8$rP@l2PFUN54Mt^&8RekZ*K*ap~nDnz8D{(Dnpes`A^=H}rQ9CFUG>B<_X
z7d)e&8=1}rHCK>(m53dVUg%7}!P!oPSeVv&&7<KeQk`{Aa-i2MywitZDbqXuQqz7j
z$-Pe(R5uh-`aT2=EWMhV+S=+WOn#HgW@H0()Un(ROUDm<E2E=UN0)<`#Ny;9Ke+m7
z!Vk1Ze)$fE86zw>vFd}o-J3^Tc>=wz-IFykHjQjK+N101H}-4h_$eLjQR=W^tfKM^
zpdVahQfH_5%a>8Wk6T+?|7y={_}jp+nJPVA%gN;1N|l&?13j)V8fJ7wmnHdrxOHt1
zt1p4uv!29p>3lPjIRQsh^W&$H?W%nHzqk(;M-D^ql=%TT$WU^}A4%d!&>~|Kj&y#0
zegHnV`QEs=yVu?8>;L>WT=9J2cSFWeX0%Y7Eix{%pUc89)PS@YkMTOB<75_3RWe%-
zye<31%`L**t^p<)gieu!*GO;K*}>}Yx$S9qdh+J5-nkAp6y0xH8P%t|o&J3PvEfrZ
z^xxWPeM<rHr!4}8lW_*6h+iCE{JJSO1Enn~<uA1j&GtZvggD^uGLe%T|M{weghBHC
z#}B|z#YQ%7G&zlxCH}FhmD)q7xIooSvnk)GlBBAlrB6E)Ea74Bmej{`BH3sEA68EL
zgWnp$-H%eHh>T-nJP=Mp(bE{f$7yQ1Z4M@bje2}}IXXNH9~fc9V>*$Z*?JWdpy%Xp
zfshpkzuD?1#b5HT$jJSxg7U{pWoFaw^2zrLb=oj|w-k-XV-(rF^!`8*mkcZm;9Pw(
zY}Hj%9GRZxqNSzf;^JatjOsD%Xt!~$#=|U+lFXm7zvNS#UnVr#X+JD(w4$z3@H+Zq
zMy6LiKN4j~pY5=4d@XMA2iJH1>zYg3i>g<4UqE0RKRY+~9&p3@<0xFWhWP*u1-xl$
zdKyqxD=RA#6B2&u9gi)1v@-nlW$=1amiX(1_Jx1CtM|XrFoGr35d9)yr^?+_zur*n
zbeiDLsJir4ufj&uC$v!8>@2aLs;vm9+BI?)7R}X1=<<UCOeOPVb7^$n=?eK>-k+z}
z>eav3tG1oqU{DX(pR(C3*Nn{(1QZp9b?y6cS)FnWC;1yg?15HIhkLe*>lWNEfzHu5
z{ykc1>3sI_rwcp+S?W?{@~;DrB@_r8Kj}8ODfp}E^n~h`t-OS$hgi^6s)5T6s?vH2
z@~f5gn0~l~0B2vy@*fG9pj40|*5@q3`zkvihDhKsCgi<fMH)wL9W{PR2K>0sx5mej
zZYm_h+1?s?2Dqfeur>uoT~+bGIJH#n2xw>H7w0=B;#DSL3#~*tP@=80IjRzyb<D}n
zS<jR#{pKS*i$X+jG^kl@-GG}o2Rc7HnG<RDqxe7jsCuI_TAixPYNcJoh%MfYILC}-
zd#-q8=#(h!H*^OVc}~F$qO`tC8pQ8lB`RXk#NiDZVo85mK=Cj;UYTZaO%5z=ap7{b
zR8eAp6A4p{(#K17!Rcc{R#0g6y;}WLHd!U&OOTu{O{{|3SsTf+(^Q}C=RFa)UHuhj
zP`@x<cGLfO3^E-%9M%@eQDKv)#*~kby7CfH8PACP<CFX-^vesNR6$EAq&?Jcx|Fcl
z$7+aDY^Ad0kA3wiGdcAlz`*J7*2i&>$zscZx{o1Uo-rwqIX6CN=KY3}Qt0of5X+Mg
zq)H-f{aX6kxMH9`G#4?(>}}>yQDZURwr;i1X-e-}(ML&bu~Ub{TugFM|9IIAH^#8z
z-J{yeh~hJ>mW{X}wUw;CHvv*+?O)aMD^y6=cwsB!T7KTjHjk)hsWl`uJ{P}-_~vdp
zD;l)8lkquU|AQW-Q4cQIB9F0n?_=L}a~G=lCg>!c48E8(J*~BF4ZoM#FU^f|QD9lJ
z%ijA^uK(IjUU!_YFM8i$ZnR8)Q;@bqFt;=gRV`{L=FOt!6{(>6s;wk{A_rj^1rVgD
zi+}<_3V{-**9V^dqV99h?O|rAfxJfF!LP+J0a|4SYR*ndi^>S%cnEm-l(d1GUvHM|
zAL_E|b8J;CDY&RBH-fn;cYa8C7w}PKQ~gFZt(+(#?!i;B<dWTiiU-qwW(-cHkCONp
z{uR_7K4GWFdni9lCul-$xj!fA$SA&0h1L%FamelT#QY$RSx5drHJh6G*5sv9`t*5t
zKV$oLR_!$XjR78An`k3OKLap98@Rsti5NMp4If6Da{uPQhxfzl=CLWDxZlzczN~7p
zWcGa%(8D#vQ;UZw#{iNypCZ*&Hgu=o5FVBSmdgd3XTGszx`&jkT!FoW4MHa}fk<`%
zUJMk7L5K&6LPbt2oDB7aFV*CbB2n|iht}J;t4X-M?a7&AI9`^87K*P5Id19nMf2_s
z!RcF*YF`VI&t!EJ440#Nn|R4c1)3=oLuDfM#4#+s`6gVRkdkeT;l7f#FH8aj6>2E{
zCkz?x|K{ATJzXfW)il*tSRXW-BDS;Mz82FECN63oQFG5IzP3zJv-DFE_t$aa@8e|y
z5Y-}<=1<VUf<>@Gf|x4oV`Aq!->6UG>qHn91o`%0ZRKQ`33!p9kJHUn5~4&vAt}li
z<Nko`%g~WC*Q`*b<lle`2?4hr8rKT!R$8lDpn7IdbqzMWPi5n{_f{D|-|n4nQEmt|
zU8?)vwr+ftxcx+@3IszKT(K?SpdlJc8U%O8=`Y`IGl!2c7Y>UH)0N3sztUzq_lK_(
z-LqP5EOzn8C|S5N`LmJjKdx|?P4}m(PDI~h{ec;GS-mO?K<3=C(25J3l8Qc}K1h$v
zmCObB>_HQ}m%gTnk2~X&e7qb*r6-4og$>lM&}<u;7^f#s&xoX0537<XMg-l~BvY8y
z?}Cjoak@-X&ar^)6H<Hq4b6eGndB(=GdR_Sa%-+pf65MLeeN|hXfwc4;oXQ#J@ry)
zIY6O|cr8)G9&#+9Rm3f^W3poM6l7wm6q>M#$<mVgIgfA;xN0C8mL<|x4LZNNhPi5W
z{XlhEFze>%w5g@xP}VYXlt^ILXBSre`oP6TQWfmu>3Fq%GC|)BiAo-JKF=-9<Zc)E
zzj^;~>vog8yYg{+vrCAP^2^5-g8@%~wi<zPdBQ!ts3mGma%=Ta_!AJxxPb-6$Ka=b
ztuj-VnzW4XTlxA7>gCg<KRvQGG-UT2@`TnsS+2v#{pB<3cAM&8bI_0*r)snWlc_x)
zftL;N(lHsFJyShVEybvRnOfQ+KXu^hjH8ychMy=@?f$I)i6Ci_uJU0jbhUS-2ybSe
zEiE$~5<?ehnc9wF(MO0p;fYUIKu7qpI$j*$CLz9ovm~QabFQ%l4X64u$JtGPdIT^r
zavF~%rA7E(j(sovs6{^&J8X24LOV^xD8`4+W0g-hePxag8s?bB|7JmH$*5bTrU~ll
zdrKP!-~{nf>hF<nU^_X{6&;0CMSLDc0~4+8(1@lyR&pt*qyQ1Ge)7oRqmZ<6ulagd
zVNfAup7bkaRyBx74Bj_-w!5LL>{31sLVV9f!8YRzY`<pN8cP|{12btTaWFZaiUxA>
zto`TEz=0hUTyVL3xJzMfe{On`vvt(=0NNJjGlg*S=8A@yAA0pgBmSE<XL%-8I@@<R
z&vT|~*c>oGzmNjUAZJ_si-y{J>psMk)h~tEbvvWkTRlZhDOUv9>ZHiI`NO*s#^B~Y
zCbukqoJy6r%ejwD6u7EI{{O&rNG`mYoWqk+&<6H7LtKah#O`{PNVA$h;8(~}GPgbX
z-#toDoGT0heM?u=t3XWkSFypUCa0{8<}AM>YxD6{EvLhJ2{pw7f3Z|w=ezd0)!Vnf
zS`a4`1LdFK#3{HxR=?ENeLRM<m@xu=T<G^yF`Aar4z#n+e;A=^eUVrn)X%ObbuQX`
zF(q33m0FdKKo`4o;km4CCublhdUh`SrB*k!?jBuHAUdlMK9)kN5GE3}Ea7^glc9FY
z7s%U~O8X6O5Z5jVG^j~I=p4|NR5N~bajSXRYD2Uo88xQv{siZQv##jPI>vkdj$HjL
zT^Q&6F<-o#I)*N&t=w#Tx!rat^ScI6Oz<ET#E~&m0Z(!4Jga|AC($QbHiLAmZkiWo
z*dA46F5i<L$ybEmH+}p;xG)f$A0yHEsq!wzHz=xZyMG=Uzfo9xoLI|JH6=AlQ~4d|
z7Yh|%#kT8}FrqE`M*p$s*FwNDGT%`ecKpn7*df*?{PTX@5PvL>$<V8}fy@8G>sMTz
zhy<R|xe{UAyr8nZ4pZ!mV0V!ys=<B}d8LnU4Z5BIvK^gBtSY8y&-bC~0`XU<c=)Fl
zzUw>dw2C(4Yj*}=u{V7sDq9c=S!gF>J7$Onzhj2;DMq5XHli{%&&i7Y@^HewgN_55
zp?Dw`a|kCnIEO=QkqA}M!{96)Ye<v50`vl!?k`#&=4jM;@+kro^XE20RY@3iO={Li
za$)u6$D@9B%>wbR-#VQFVXPZZdHUg?^Hc)6)L7?qoj|{cw$^KAGT)6^N_05`;fV0W
z!?Pj4Lp<P?uU7o(*=;1zu@ZrU3Loi0W3Lw9&t0|?Epe;%<}Y@}4;2P|*-&z7dI<5^
zzqt9pjw<}1EV!J*2kBt7BxTrO7Y31S#BRC*^Lm{<i`1{zu~#~!bYgrKz|t85>_2lo
z-O%MPg;}iEo_auv1P38&oOmJVRnQB_*m%fgk$Zk<+h5H>RTbbk-**hxJP?%mjHn4t
z;7Mrs>l8_j5st29^y2@e#pf9<h{2{og3^Er)d|t^{L@i-XN4`Pt?B*l7S3$?Pde_u
zBdeuiFS#=n7vPm00zLLqbzJ7D?!vjc_(Cc~rN`p{K~}C8bRG!QfKHRMdE#&Pzvq;h
zi0W64bCeHEQCLU){Ej_kpE}d6-${8ruQW~yz&!tXnOr`q#c5uaZQLQ<!Ftxrn}n#%
zvmFdiyd|RQRLsXGCUTD|uX6VuI4@^uGD;wI!~uSYUPxWu=ilhj;^MpS{PI}i8$D;L
zW7cdOkEG)}QB}Zru9JmUNz84Xgs8}_Yc9aN`6XX~rSrUA^bi|zbk#uPKRYx_m%}J#
z(<d7LHcYWO&1V*Cx!X(BEe9|kb&6tr(~mK<+lm%1hl_`ySm_`Q@!ww>xDV(*hFYd-
z^x2iDHBgRmE-~Dl&#{_~tU>tkZ(y3}KU$|{Bh{p|7iD=2A`f<0NmHD{f|=>p*C?Ug
z%gJ%L=G!hN&P+c&?B;C^eo`akW7kUMxOV;g*;hiRoE^wKYwb;B-)&3P?t9jdfzQ%^
zyKU2(;kS@OOZPhp>P!;oXeq0rMC|+{+K{?CDj&-W4$LDkk}^D@*ULtflt5lHEk|+N
zlejl4GdI8~yl#UM<JIb#v*z|D4B-Iu6*0q`f^s_co6V)u4<@W1R71xzTer%Bk3w^R
z&l+VYKoiB{0}5ki!2L-|QpT&0n-&o+8d>>yHefV@#E6z?u=pkYsjINKTy;NvlkwlE
z+4VXbaI#F}zr#4f#gd}~EU33XqSQqVHQMnyUf}Q>3+K4db~08PLhUDURbBg~i=N^Q
z%1@=deEM{i$S&BZik35EtumgoUslt#VD;Pwzp^I0qxRex0{r(~z$xXBr~Qo@qV*%c
z&wq^!!`BWR!V?mhB`Jb<SFjnRlq8dr)1J1HkaPrjCGW1^8N{h4`F#1VPft(Sxv;+e
zA)@h9>8wtVR&3A!bR#kp2j6}!2jP3MU~EMH`b-@$YCmMrvEyA?#&v%zG3~xj9pZM9
z(D{JQw?sDSw}-_(+)G!;s({$bAteljI$5WU7BIUP=?kMBb4Rudj-{0{MwBKrQug5;
zUel>wwYcs%`3G7$En(Y*x)TXR8jQwWkcu=H=7UC?GYd6LIH(e77PI0MkpR-`1#DD|
z`@7qn()yOp$T68r$KH|6ElC4eN&G}s&#j#1j*l1*DsL2U;|%tam)Pp7OPPO&2MV=b
zr@epGkFo@(kw1sO?VOvBg#f@>vke^$q^D_NA=w-(%|e0oUbVOR->CER2~ie_1?6>B
z(Z@J|h(<Uo)`HW}=@=wA8Iwpb-shELtQYFFTNSk28|!5bUI-_sQ{w>aR5$bP*NO8m
z_X#dU##Pw<{>2ok+Sz6rx%9vBi{3E+eXwo3LztndgWiGc-IgRNJp7g)Mp_JB%>}>y
zO~|vPWBR$~l|>GuJ{U^Fmx%zwMwOG#FQ~u02aCH{n?3j8vkE<AE2(YvKqk9iIy_Qa
zgJ3(u<bn57&<T3G>tv5yBK`%oCPaB7zIPJUWahpxJQtM*(Xq;FkY&<*JuQeAV<(SY
z5<gA-l0V6BfX}sxZ#_YtKqq#j5BDaH_CGTlQHq)-7~uN%&aMr<*aNm@sn3FDv+FB|
zVPmQZ$($0*r_yp%f!dwyNPNC`w4gxZ@pnA0)gh;B#7NbyBK-~&|6oeC`TW<v8cqgA
zND7q<Jr)k-4<#mk+pM~2darf_3DyKEJ~Xi}Kz5dxiY`B2>Re2Pg^2-TR!neBM5k|H
z?P9$Xz*9mlsi~!w9sdiXx<LJSDz@?T=?u;SngzxGGSt4GT)QjupvarS5d~XH`AG-S
zA9bX%ArJ04LW{oVjJ8w9R~vDRJkqs~v5)Cn9IrGLdGi!^HS!YT(mArt-`rC3=*;l)
zB)#nPU!Ttg$ipw>Vx43!*gppZj2!hN?-eyGUK1tD+IKkAPR$$6P21ms{!mP9_?0^;
z(w91YXm(#TvghJ^D`pF1T4Dg~0E|H5C2_?E+|*s_7UcfQ?XwVU$!by60U|Hq^n3_V
zL@v`j$~u!{4J0kPMMU5)DS>H}|AEU*M7=BR5?=C>8?Q`X703zj)J_E&+LIcs#0`f~
z$_!E09cS}Qa+jrr&g@YRg8oU*4nj;bUv4$BQ|F$X_5z0B`$g;~^3L=UHOfB(W(;1c
zY8j^?s_jr|cZsYU^ok;U)CgRJ)%PtZW6Q>(WLMO-w^3<!4%d?OASxJmg_q0;nxvv-
z_lI=M-)H;J@lmPfTfr)qB5f@>jkzdQS{n%?;o<aQx5TdTmN<3LCpBKc{6Vy<Q-n;Q
z4V(>?H*gSsu*MpJ!S*grrN--DF*SjYm-w%T0XaTt-2upA_tEFuM_0Q0wD<hi5vA?O
z<|8Vxtvi`IyKioQDqMl}DDuz#$NEl|W^=B87q!MC6jzk|ndl@p<fnd|*>z#$AQAl^
z-*MbnnyKzpaSmYuS;s5D`}Y^zwSqFjYA>EyD*0hoY>h;lYr4$Ob&KaoIV~k^vrE69
z&{u2x0o$={W(#{R+m>DgWITLE)Om<$dtAFq#0Aey#Gmkzy29GZ*{iUnuzFzs$Dw|t
zg!u3idY5U@fgZi^E9C{;XupGP>;vDcss8BpGfRvL`OL78+TnG~WzEngij}F32xm)!
zqnr0jpa5Q%^WAHOp!_YH9@g=^H`SIq(5=1)BaQDaPVW{s!pH|f8Sa04d|_#3LL=bD
z|K!_L)<w1U@3-n|rv9xg9(aw1LS&=NDr`HFR)No>{Pr+N9kDU=OV>)oq0d_&VrgmJ
zJvtTcw=#lerTGsMJ<RA~b=)!`ACzu^f&jbbVgN5@3mSAkd&_!l9l`nCW8>hFMjON=
z9ZhG8Koq7^>)M0OMkydN0m}f{N%^x!S10nC+=gMLLSiW~8+vDzd~1=Mu$Z7JzN&cj
z6@X&}M4ugLFNQ@?^BOH2yUxaV&F$H8yX1GIweAN$9vg;i2%V1jy&2^G%>40U=2$Bb
zIY{GqjNtu>IN};o><B|k)V>Z3K^RsbCEHsyE<c50*5})zr>0@(W$bPvNh%U)fMJBI
ztzL90`6mKOcQARCR`hLJxoJNcH%Z-@5fwz9AnKpI+y&56_Vs2bH6Dz0p0V>nUKlz0
z^GKf~rh21M+<>hQ7isP#B6I6>vOuh&i%b%13fF0xM5ox-!5$;bJ{78`mJ+wv`<%56
zu=GKt3Z^eH-f{er1&GUXpHnVeus4bHN|xc;=t{`u>Fdi(H+V66U?l)L|Ek5qcz7!O
zz1P<@l6G5_ab<WoA<TXnCH|`sCclkTp9z+S7WL3Cpegm{x&eip@W)87w&b<3VX380
zLC8Teukn(j#Z+bCaQl7$<1YUJ$J+q&=CBmAN10kcF%dnV7hG^RF4`INRpowOm55>$
zo`9oC1db;8XH~JV50g|r)$aEq9$g|LYSZ^|R|q9&rXr+4<fq`%{xkZG-CY<yi!T&j
z^NI;b%XgC6Mr4;>uSZi_QgOh4fzp%z?l@$Cx2?~@{+<Te(e});-<r*nj}B-KTpmC?
zT~g`UR)Kc{Z9w>Q(>zyHk1$!e<&3-`Hyqd+xs$(o@bLlvJ=o<q_$a|wVyp=g`V1lD
z;hFz?K0%ThIbZa@bCz3*RS2|G|8sJ|#pyqgVJ~NK>F5~SIs0ej*pW%0b+->2uWKAU
zd%_6P@J&<ieM{gJglDZ2WrvheNB#Y#l1U_7oKxc0hmeU@a!HjmMiqHJPK!fvbVZ9Z
z9EKl${@Dlug*B-E<3%RZ@|!sO=tp@Zei43IL=oh^WmfeKnC1}@j>R7F21b2&!$ZZZ
zZ3tT!{a#Hh>2)v>RYZqHPNq2q+A&W7c5+nha+hrgY+kAuiL<s_>zN_xZS?^5aCcm$
z2ce%*^fV2jBMG>APXJBV0iroj*-DHlrQu+!aR^+bRY@e-nOL|SH4c8YJQ~Xl3F(Q^
zk(*VrMT5GnS2`A=iX;s^;g@Cns>^QP#fp~-qBiGjmbQ2mTgpwg^#Avv#F27-Rjz7I
z?sD+B-V6g_s)41SHpRbd$YPU`xI5R4cToK8W*S23;QOG2-TdQ6no&$CvXZj@_B|w_
zYvc9{_0+wIHahJ8&Ic8vYAgR5!FFPNhL*KEPn;<>mZn!ad15byhumr^DiaxE4p9lx
z|0(Ru<DuNU04^pm*2t2C7|cwCCX^-al)*%pY}chy=vJ0)mh9UIV^4_4nq@4>MY=+a
z?b?!Ly{<J%5tFi1gZB)#`);4l`_5m#&*xd@{FZZ`^PK1WzGr#YKBuX5T*67WE5TM-
zzbZx|d+I0B6S|1V_F#|G19?2(Z}(~~M*yjF<ovtON^(!K_NYss&xY;J$q9d>IX=l^
zhZFqtNbv}pW%XQZ84meDrSY6c&NVbP=V~Vr;}!VavRuLWdpA=cm0cgr%IVNt;T`JG
zb2vTCMSrg@1(_;e78^(M!^H75jb(f^jB=n&#Z?WZj&V}6sS+;bF)R9=ooHOWB-$!j
z6^%t)1+cw+bji*2cj>F$`nDClT<r<#`ue2(;zj<>zg|^%5R_z*B9WY&Y{ea7Y}8X+
zP0d(zH{@vRBDTiDM(?N0HH*`{$Lg2Ri8hRS9^&$DdM)yS2_lzDA?iKkaKYa*K_fkh
zbrl_~SQU|^1*=CT&3;!V&d<~A%s9w>yFsPnv#D8ApRwgZ?FJlOKHd%YY7r^YMHQyp
zjp7LjC(E;qq6@RB6OL)}2fhrq1Y1OiXln?_nSUP#<{J8(>ltuaxy*=i;+%7x$_+7f
zf?4J#0%`x=Qa4v}#@F-mKN5M<?La?7hp+stPvt^FSyf}$4w)EhllS@iPTl{J^fPH#
zlk<f3D|#PdJv`0XDKk8;rF;n&`A{-E<9?R0!U#Y%{?bMWE79L1>iO85CmtEc%dpDL
z$zeZ+&u(IDU7ONPGpT<h7^uMa*zNH(w2OQ{!T;g4oxAaw9u3^o^WebEVo8wh(*1>Y
zPM_uj)TSnPTkdGoE|Z`Z<JdtN^VQA%_XezNv2os9#nQ>w48Qj_rFBoz(%)tPOg4mL
z@b+AFS%w~qw2KdZUu#Cix8(YbRXkyL6Fxlbzvjh%BAv+++P0iP8)&<WGJ01z{OOtF
z%SKsLBo+1MlSEyB#?7fjD=y)8R>Sh;-qRDzTps3YCj1ls@KQBZndiCOi<A?dAIp!u
zsj95K<jMp>ZJ9ZU!%&>EDIG)YcC_*&F3XYuC~$4t-+dOa*i3V|C0|r7)$m-c@I#vu
zb_-+?pT*Z!gv59i=DQud<3l}Asq)Z0{oSYZ@FccfeARx7q~kK-xRNmD%IqKcExDlX
zc9<;oSB1dG2Vg_`?VQxdwl60xHigHEubX^2<HWmtE<XZ)n_bs+F!OHqa2e^r&t2A8
zj#v3SVAmwG7YMwVD3L^)&eP!Y*%<Y0fB$%3EEp5&*Upq9REmSFth9$DJ1w-R=Cia~
z8dx%1*)JoVK@&R73q&z20?(ft3@O|k`9GJKj=r3UbE%0H=QDUXY6W4tt%I;TAn4Q&
zLybbkzKs4k#zWu27S^p~EDo}iKAUMFRn%tE(XMm1UF-6G`U|!*`~P_`p;eG@xs3d7
zFNQF4XBlNg)bVi<l?lK5ds2xdA51SlBHj_^pDabO3EAq&v>1~b!67{es}cUZntiJS
zZ7N%$=#)ZO8cud)nw<Z>G!;uoviiCMj^OQii?O5#M^SF}5Lz3sZJ3StIjG_V4Vgi3
z)1jeT8p5oW1GO_Cph$y8HE$#`1RAdUp+XKcrq$8dts8j!zZ<+2Bqb%@m1g#%3o1Lb
zq~=gM{-g<QNfK*mOZm}sTYez*{SW#7U)M33tpmJq=WET_xeqy^$2G`O9k3E7y&dhV
zep1WHexUbd%%NN3=$y!-v9_ahUd&?kj+ZK`1-18UKaTarE@kB13a;jv$(!NvH$B+l
zYXRA+NGLNKrs<Wh5a6~X_P9V14VP5k{8qZ=Fs%}^a?iV@6ca%7eqN_%j9IjTxFkjs
z-o?l{0`<|g>;wYLO&g?-a^p!67j(}<Dkw~$J-8CExSJJa9sv>px?7GQ$bz>&l^hRK
z+6J>~1^RaYjs4i!1~YGOVn#Ux;h2^A;nx!rw?AK%mX^Msp8g_TLtdiNXBp7uHa0fR
z71}X1HQxZ$+WcA!4LltLf#Ch(EO_EzDUor?IvxNUZEMrl%3PeC^_m-aI45amXP0pQ
zKKD~H^2j`ZGH=|tL5~QwY8J&Q8JL?Vi4K?m(hh{^@VyIfmxTbN@XD1h>g_%2?EC>h
z-U)<R>eABE{QUgN!YCJfoS|POSzeF+@EEEWHAe{2*(xh5OUx$eulolF!*y4Hh^xdI
z;$OtU7>XcV6HwdT51eyJO)nPz72s^P<QcoIE}ZlA^*wc}>49FH#PMH#(9_dnj0T7;
zRaiE)6;OeJ%$|v`wjMdH%?SwDz;7#u88KdOyZ5KulBBGqZ5|R)u#pljBr?b9sxk?m
zq2-^Nnu>IN3YcI>w!J1>1W4tR2FU!tmHB(Wp?0qg1JfTRCs$NhNJGt@mg3JXC~))_
z9*Px=>O;9o7kSJL)CH`IRVf0q5vZvE3ZI&uMz~!Z;Ta!=L1r)*zuCCx>r>WzHv%&>
zz0VdKozo)htUvCqp58QB>-F)Z&7C$N*<MsYJPT?E-f@HcfrAGP7&5o@^R(NG;M;~h
z5grOK2DbUp(a~>TWHzB7q?Pl>lk1t))oKj0_4V}@*@%WiL#}u|f;+bqRx6V)&Et|w
zx{P+s25>faiiPo%=F54cJ#!o^ot?d6a(qZE5gz|ldb+y0Q@cH4{35SGeh#ETI5;@i
zUb;b)Zf$L)YXHEORZ<l8yU^qh$jODJSifbiw%(K+vIUEJ(a_M3fL&tBK|Z52yDNa!
z@R=?sY^nXGEnWtX#{<iDTDSIH1X8ue!os4ZrA0zQVybtOEpS!oUKf>$zI4GNL8Koj
zqj>Gm&`>oswH~2|Ru#IMcFXT!D4JHgFa7H7?oNR(S4=I5k;4w9Jguo|dUX5o<7L1K
zXPpG*{LItpYFbqox0C&o>*Gd;`h%b>6Fyc_n>RH)GX=Pu({DHfx_hqa=;+wg`6N-5
zPmu6^eSNkgCm+0=s6Z+Bb2~w~X~1mv{6w=)5D`M$LM&h~m?KI`LG`xH{r$z0ib-8v
zhO>%~F4P-AHWJe|#Ai?^2Af)1cB(3PYPXJ!WzcLJBsb`3r?_Qwb~Zsq5v%Ma7nk$J
zb#GM+1^N47CH_F7i?BF(0{NLD&A?2|&vVzi!|>)#q~%g4*aXGIko!`B`Wt0XYzaHs
zj;X2b_2F7_%j0Ik)klCV+}Y9b!^pm4E~rR4v8<w^c!-gjmXL|qBLtc7pkz>EEG0di
z2TPV?5fB&WE+sq1c=0cMEv}<Vfog=7?g_X8sW?(L9`i#-hPd+z3Iq#C{5wcV$;t9^
za_1S!)=yl7;;2^Br_JbZF4PC`@W}ezZBu3K;u+a5DJi+u&JWrO6Vy`}RsSI__~~@)
zY^sh6VP`NOi=TjP2-o=WFT|iybs3r0P@|w(l!JoQ;-o*xS}UM$_N^@O!3TT;u}`8x
zvS4H=NHLDt(-&W{1!`RXqmmN~?Do#*&qL1~b)rNZ;pb1%M`1CTU<C`@oj^3Gl`NLS
z>E=B{ImQlm#@dY>*0jqKajR1;Q2={q%ETJXQl&R_{hM85V<W{sL?uo$3i&KKD2Qdv
zjelc9W@DqLzdy8FAo5Off0Hpc2-S(+z%aCv70wAjcbS?G?|Zk^Dc4@N#sF&iz=rM^
z(597zfw?Nq8-85mo?*hD%e#UXXq}y0RXA9}N}R5&*J+nm0ma_TLcz^{4QaSO0LeWK
zztf3Bb-u^^1pXWyoqQ7Ey3D%p6$1WSkAgnD8ULE8`oV)-0W-Eyev^%%VCg=8c3+xM
zi_gav@$6Ft0CZgEjGVlDQ~vP4fUwqO&=as(B5@{SB)plq`B5A$>moUh_T|&t$)8yh
znmwL@iD#}vrua`Iu7Pz{P*5oIh8(b)9v-3N>Q@v|Px@}>W3e2{#yux#p8j8!X25z$
z#CgInw!29s>6xXeI=i~Ml2^pE*!kDY0h}E&uJv&~4o`x#(op!Ca_YTY5fl_?tP)l2
z0nHp99`^S3270-)F)OO~7U8k)stdrI*A^y@vX5L5i_wj*Tw>0<ywQCr1*)_bXOGKS
zEl3=OlI3*$Q)VC`EaP#&;6rz0H~aGJo5Fa?A|N;F7iVKav2|)RRm!@`Df`vCvK;~(
zh%z2^kN1gQt(&iI5q4U?g&GuY)qj?k<AxSy=jKd}Y)vId_=#k+mX<`o9y*Z{6O*JF
zL+02v6TSRNe)*UP<i?(!moGt;gD+?vKK$^7v5HG$4tajGM&u7nyguSN!rN7KZ=Qe<
z<G|Y2<+-V;OKYeGy{S(JXGnjKta=nu)fhO?tougmd_5uUG4Zg9JwTtKBO8oGsA&WS
z+!--63oQP@M;A8z#&OECfQJsV>M4mb@Q7gm@i_AqXl;Z1MX;=yfdZh+?3uTI8b#Oj
zet%|r2%A43Xc79aKz4zO(O*jvs_-zK1vD0i#_HS!uNJ~B4BQg9GJBcI1knI^{s-*!
z=824i%GU9+|LH&6@w{xYD^A>;y^<#yGhWm6uf7DoECk*Ez80UDD6e>-&}Bs7h33ni
zoZ7eS)7>GzCnhHxt90i<FIt_}9PG`mt*rex`^8|}yV-IpWiZ$t!EJX@@!vho)DA++
z<S*#gOg~dc?k|5cOu7g(6vlYl4OP^SM1DpI3n^P9-hkV2NJB#V_uEdf2yNe2MoUB}
txzm|FM#w=N&LOuUh@{=$)_=2E*kqAV>Wv<iEYD+sACX|BQ+m>t{13q)j6eVY
literal 0
HcmV?d00001
diff --git a/doc/guides/rel_notes/release_20_05.rst b/doc/guides/rel_notes/release_20_05.rst
index 000bbf501..f404ee7fd 100644
--- a/doc/guides/rel_notes/release_20_05.rst
+++ b/doc/guides/rel_notes/release_20_05.rst
@@ -62,6 +62,11 @@ New Features
* Added support for matching on IPv4 Time To Live and IPv6 Hop Limit.
+* **Updated the Intel ice driver.**
+
+ Updated the Intel ice driver with new features and improvements, including:
+
+ * Added support for DCF (Device Config Function) feature.
Removed Items
-------------
diff --git a/drivers/common/Makefile b/drivers/common/Makefile
index 96bd7ac6e..df2e840cf 100644
--- a/drivers/common/Makefile
+++ b/drivers/common/Makefile
@@ -31,6 +31,7 @@ DIRS-y += dpaax
endif
IAVF-y := $(CONFIG_RTE_LIBRTE_IAVF_PMD)
+IAVF-y += $(CONFIG_RTE_LIBRTE_ICE_PMD)
ifneq (,$(findstring y,$(IAVF-y)))
DIRS-y += iavf
endif
diff --git a/drivers/net/ice/Makefile b/drivers/net/ice/Makefile
index e22c34287..f493c9ed7 100644
--- a/drivers/net/ice/Makefile
+++ b/drivers/net/ice/Makefile
@@ -11,9 +11,11 @@ LIB = librte_pmd_ice.a
CFLAGS += -O3
CFLAGS += $(WERROR_FLAGS)
CFLAGS += -DALLOW_EXPERIMENTAL_API
+CFLAGS += -I$(RTE_SDK)/drivers/common/iavf
LDLIBS += -lrte_eal -lrte_mbuf -lrte_ethdev -lrte_kvargs
LDLIBS += -lrte_bus_pci -lrte_mempool -lrte_hash
+LDLIBS += -lrte_net -lrte_common_iavf
EXPORT_MAP := rte_pmd_ice_version.map
@@ -84,6 +86,9 @@ ifeq ($(CC_AVX2_SUPPORT), 1)
endif
SRCS-$(CONFIG_RTE_LIBRTE_ICE_PMD) += ice_generic_flow.c
+SRCS-$(CONFIG_RTE_LIBRTE_ICE_PMD) += ice_dcf.c
+SRCS-$(CONFIG_RTE_LIBRTE_ICE_PMD) += ice_dcf_ethdev.c
+
# install this header file
SYMLINK-$(CONFIG_RTE_LIBRTE_ICE_PMD)-include := rte_pmd_ice.h
diff --git a/drivers/net/ice/ice_dcf.c b/drivers/net/ice/ice_dcf.c
new file mode 100644
index 000000000..0a99cceb1
--- /dev/null
+++ b/drivers/net/ice/ice_dcf.c
@@ -0,0 +1,474 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#include <sys/queue.h>
+#include <stdio.h>
+#include <errno.h>
+#include <stdint.h>
+#include <string.h>
+#include <unistd.h>
+#include <stdarg.h>
+#include <inttypes.h>
+#include <rte_byteorder.h>
+#include <rte_common.h>
+
+#include <rte_pci.h>
+#include <rte_atomic.h>
+#include <rte_eal.h>
+#include <rte_ether.h>
+#include <rte_ethdev_driver.h>
+#include <rte_ethdev_pci.h>
+#include <rte_malloc.h>
+#include <rte_memzone.h>
+#include <rte_dev.h>
+
+#include "ice_dcf.h"
+
+#define ICE_DCF_AQ_LEN 32
+#define ICE_DCF_AQ_BUF_SZ 4096
+
+#define ICE_DCF_ARQ_MAX_RETRIES 200
+#define ICE_DCF_ARQ_CHECK_TIME 2 /* msecs */
+
+#define ICE_DCF_VF_RES_BUF_SZ \
+ (sizeof(struct virtchnl_vf_resource) + \
+ IAVF_MAX_VF_VSI * sizeof(struct virtchnl_vsi_resource))
+
+static __rte_always_inline int
+ice_dcf_send_cmd_req_no_irq(struct ice_dcf_hw *hw, enum virtchnl_ops op,
+ uint8_t *req_msg, uint16_t req_msglen)
+{
+ return iavf_aq_send_msg_to_pf(&hw->avf, op, IAVF_SUCCESS,
+ req_msg, req_msglen, NULL);
+}
+
+static int
+ice_dcf_recv_cmd_rsp_no_irq(struct ice_dcf_hw *hw, enum virtchnl_ops op,
+ uint8_t *rsp_msgbuf, uint16_t rsp_buflen,
+ uint16_t *rsp_msglen)
+{
+ struct iavf_arq_event_info event;
+ enum virtchnl_ops v_op;
+ int i = 0;
+ int err;
+
+ event.buf_len = rsp_buflen;
+ event.msg_buf = rsp_msgbuf;
+
+ do {
+ err = iavf_clean_arq_element(&hw->avf, &event, NULL);
+ if (err != IAVF_SUCCESS)
+ goto again;
+
+ v_op = rte_le_to_cpu_32(event.desc.cookie_high);
+ if (v_op != op)
+ goto again;
+
+ if (rsp_msglen != NULL)
+ *rsp_msglen = event.msg_len;
+ return rte_le_to_cpu_32(event.desc.cookie_low);
+
+again:
+ rte_delay_ms(ICE_DCF_ARQ_CHECK_TIME);
+ } while (i++ < ICE_DCF_ARQ_MAX_RETRIES);
+
+ return -EIO;
+}
+
+static __rte_always_inline void
+ice_dcf_aq_cmd_clear(struct ice_dcf_hw *hw, struct dcf_virtchnl_cmd *cmd)
+{
+ rte_spinlock_lock(&hw->vc_cmd_queue_lock);
+
+ TAILQ_REMOVE(&hw->vc_cmd_queue, cmd, next);
+
+ rte_spinlock_unlock(&hw->vc_cmd_queue_lock);
+}
+
+static __rte_always_inline void
+ice_dcf_vc_cmd_set(struct ice_dcf_hw *hw, struct dcf_virtchnl_cmd *cmd)
+{
+ cmd->v_ret = IAVF_ERR_NOT_READY;
+ cmd->rsp_msglen = 0;
+ cmd->pending = 1;
+
+ rte_spinlock_lock(&hw->vc_cmd_queue_lock);
+
+ TAILQ_INSERT_TAIL(&hw->vc_cmd_queue, cmd, next);
+
+ rte_spinlock_unlock(&hw->vc_cmd_queue_lock);
+}
+
+static __rte_always_inline int
+ice_dcf_vc_cmd_send(struct ice_dcf_hw *hw, struct dcf_virtchnl_cmd *cmd)
+{
+ return iavf_aq_send_msg_to_pf(&hw->avf,
+ cmd->v_op, IAVF_SUCCESS,
+ cmd->req_msg, cmd->req_msglen, NULL);
+}
+
+static __rte_always_inline void
+ice_dcf_aq_cmd_handle(struct ice_dcf_hw *hw, struct iavf_arq_event_info *info)
+{
+ struct dcf_virtchnl_cmd *cmd;
+ enum virtchnl_ops v_op;
+ enum iavf_status v_ret;
+ uint16_t aq_op;
+
+ aq_op = rte_le_to_cpu_16(info->desc.opcode);
+ if (unlikely(aq_op != iavf_aqc_opc_send_msg_to_vf)) {
+ PMD_DRV_LOG(ERR,
+ "Request %u is not supported yet", aq_op);
+ return;
+ }
+
+ v_op = rte_le_to_cpu_32(info->desc.cookie_high);
+ if (unlikely(v_op == VIRTCHNL_OP_EVENT))
+ return;
+
+ v_ret = rte_le_to_cpu_32(info->desc.cookie_low);
+
+ rte_spinlock_lock(&hw->vc_cmd_queue_lock);
+
+ TAILQ_FOREACH(cmd, &hw->vc_cmd_queue, next) {
+ if (cmd->v_op == v_op && cmd->pending) {
+ cmd->v_ret = v_ret;
+ cmd->rsp_msglen = RTE_MIN(info->msg_len,
+ cmd->rsp_buflen);
+ if (likely(cmd->rsp_msglen != 0))
+ rte_memcpy(cmd->rsp_msgbuf, info->msg_buf,
+ cmd->rsp_msglen);
+
+ /* prevent compiler reordering */
+ rte_compiler_barrier();
+ cmd->pending = 0;
+ break;
+ }
+ }
+
+ rte_spinlock_unlock(&hw->vc_cmd_queue_lock);
+}
+
+static void
+ice_dcf_handle_virtchnl_msg(struct ice_dcf_hw *hw)
+{
+ struct iavf_arq_event_info info;
+ uint16_t pending = 1;
+ int ret;
+
+ info.buf_len = ICE_DCF_AQ_BUF_SZ;
+ info.msg_buf = hw->arq_buf;
+
+ while (pending) {
+ ret = iavf_clean_arq_element(&hw->avf, &info, &pending);
+ if (ret != IAVF_SUCCESS)
+ break;
+
+ ice_dcf_aq_cmd_handle(hw, &info);
+ }
+}
+
+static int
+ice_dcf_init_check_api_version(struct ice_dcf_hw *hw)
+{
+#define ICE_CPF_VIRTCHNL_VERSION_MAJOR_START 1
+#define ICE_CPF_VIRTCHNL_VERSION_MINOR_START 1
+ struct virtchnl_version_info version, *pver;
+ int err;
+
+ version.major = VIRTCHNL_VERSION_MAJOR;
+ version.minor = VIRTCHNL_VERSION_MINOR;
+ err = ice_dcf_send_cmd_req_no_irq(hw, VIRTCHNL_OP_VERSION,
+ (uint8_t *)&version, sizeof(version));
+ if (err) {
+ PMD_INIT_LOG(ERR, "Failed to send OP_VERSION");
+ return err;
+ }
+
+ pver = &hw->virtchnl_version;
+ err = ice_dcf_recv_cmd_rsp_no_irq(hw, VIRTCHNL_OP_VERSION,
+ (uint8_t *)pver, sizeof(*pver), NULL);
+ if (err) {
+ PMD_INIT_LOG(ERR, "Failed to get response of OP_VERSION");
+ return -1;
+ }
+
+ PMD_INIT_LOG(DEBUG,
+ "Peer PF API version: %u.%u", pver->major, pver->minor);
+
+ if (pver->major < ICE_CPF_VIRTCHNL_VERSION_MAJOR_START ||
+ (pver->major == ICE_CPF_VIRTCHNL_VERSION_MAJOR_START &&
+ pver->minor < ICE_CPF_VIRTCHNL_VERSION_MINOR_START)) {
+ PMD_INIT_LOG(ERR,
+ "VIRTCHNL API version should not be lower than (%u.%u)",
+ ICE_CPF_VIRTCHNL_VERSION_MAJOR_START,
+ ICE_CPF_VIRTCHNL_VERSION_MAJOR_START);
+ return -1;
+ } else if (pver->major > VIRTCHNL_VERSION_MAJOR ||
+ (pver->major == VIRTCHNL_VERSION_MAJOR &&
+ pver->minor > VIRTCHNL_VERSION_MINOR)) {
+ PMD_INIT_LOG(ERR,
+ "PF/VF API version mismatch:(%u.%u)-(%u.%u)",
+ pver->major, pver->minor,
+ VIRTCHNL_VERSION_MAJOR, VIRTCHNL_VERSION_MINOR);
+ return -1;
+ }
+
+ PMD_INIT_LOG(DEBUG, "Peer is supported PF host");
+
+ return 0;
+}
+
+static int
+ice_dcf_get_vf_resource(struct ice_dcf_hw *hw)
+{
+ uint32_t caps;
+ int err, i;
+
+ caps = VIRTCHNL_VF_OFFLOAD_WB_ON_ITR | VIRTCHNL_VF_OFFLOAD_RX_POLLING |
+ VIRTCHNL_VF_CAP_ADV_LINK_SPEED |
+ VF_BASE_MODE_OFFLOADS;
+
+ err = ice_dcf_send_cmd_req_no_irq(hw, VIRTCHNL_OP_GET_VF_RESOURCES,
+ (uint8_t *)&caps, sizeof(caps));
+ if (err) {
+ PMD_DRV_LOG(ERR, "Failed to send msg OP_GET_VF_RESOURCE");
+ return err;
+ }
+
+ err = ice_dcf_recv_cmd_rsp_no_irq(hw, VIRTCHNL_OP_GET_VF_RESOURCES,
+ (uint8_t *)hw->vf_res,
+ ICE_DCF_VF_RES_BUF_SZ, NULL);
+ if (err) {
+ PMD_DRV_LOG(ERR, "Failed to get response of OP_GET_VF_RESOURCE");
+ return -1;
+ }
+
+ iavf_vf_parse_hw_config(&hw->avf, hw->vf_res);
+
+ hw->vsi_res = NULL;
+ for (i = 0; i < hw->vf_res->num_vsis; i++) {
+ if (hw->vf_res->vsi_res[i].vsi_type == VIRTCHNL_VSI_SRIOV)
+ hw->vsi_res = &hw->vf_res->vsi_res[i];
+ }
+
+ if (!hw->vsi_res) {
+ PMD_DRV_LOG(ERR, "no LAN VSI found");
+ return -1;
+ }
+
+ hw->vsi_id = hw->vsi_res->vsi_id;
+ PMD_DRV_LOG(DEBUG, "VSI ID is %u", hw->vsi_id);
+
+ return 0;
+}
+
+static int
+ice_dcf_check_reset_done(struct ice_dcf_hw *hw)
+{
+#define ICE_DCF_RESET_WAIT_CNT 50
+ struct iavf_hw *avf = &hw->avf;
+ int i, reset;
+
+ for (i = 0; i < ICE_DCF_RESET_WAIT_CNT; i++) {
+ reset = IAVF_READ_REG(avf, IAVF_VFGEN_RSTAT) &
+ IAVF_VFGEN_RSTAT_VFR_STATE_MASK;
+ reset = reset >> IAVF_VFGEN_RSTAT_VFR_STATE_SHIFT;
+
+ if (reset == VIRTCHNL_VFR_VFACTIVE ||
+ reset == VIRTCHNL_VFR_COMPLETED)
+ break;
+
+ rte_delay_ms(20);
+ }
+
+ if (i >= ICE_DCF_RESET_WAIT_CNT)
+ return -1;
+
+ return 0;
+}
+
+static inline void
+ice_dcf_enable_irq0(struct ice_dcf_hw *hw)
+{
+ struct iavf_hw *avf = &hw->avf;
+
+ /* Enable admin queue interrupt trigger */
+ IAVF_WRITE_REG(avf, IAVF_VFINT_ICR0_ENA1,
+ IAVF_VFINT_ICR0_ENA1_ADMINQ_MASK);
+ IAVF_WRITE_REG(avf, IAVF_VFINT_DYN_CTL01,
+ IAVF_VFINT_DYN_CTL01_INTENA_MASK |
+ IAVF_VFINT_DYN_CTL01_CLEARPBA_MASK |
+ IAVF_VFINT_DYN_CTL01_ITR_INDX_MASK);
+
+ IAVF_WRITE_FLUSH(avf);
+}
+
+static inline void
+ice_dcf_disable_irq0(struct ice_dcf_hw *hw)
+{
+ struct iavf_hw *avf = &hw->avf;
+
+ /* Disable all interrupt types */
+ IAVF_WRITE_REG(avf, IAVF_VFINT_ICR0_ENA1, 0);
+ IAVF_WRITE_REG(avf, IAVF_VFINT_DYN_CTL01,
+ IAVF_VFINT_DYN_CTL01_ITR_INDX_MASK);
+
+ IAVF_WRITE_FLUSH(avf);
+}
+
+static void
+ice_dcf_dev_interrupt_handler(void *param)
+{
+ struct ice_dcf_hw *hw = param;
+
+ ice_dcf_disable_irq0(hw);
+
+ ice_dcf_handle_virtchnl_msg(hw);
+
+ ice_dcf_enable_irq0(hw);
+}
+
+int
+ice_dcf_execute_virtchnl_cmd(struct ice_dcf_hw *hw,
+ struct dcf_virtchnl_cmd *cmd)
+{
+ int i = 0;
+ int err;
+
+ if ((cmd->req_msg && !cmd->req_msglen) ||
+ (!cmd->req_msg && cmd->req_msglen) ||
+ (cmd->rsp_msgbuf && !cmd->rsp_buflen) ||
+ (!cmd->rsp_msgbuf && cmd->rsp_buflen))
+ return -EINVAL;
+
+ rte_spinlock_lock(&hw->vc_cmd_send_lock);
+ ice_dcf_vc_cmd_set(hw, cmd);
+
+ err = ice_dcf_vc_cmd_send(hw, cmd);
+ if (err) {
+ PMD_DRV_LOG(ERR, "fail to send cmd %d", cmd->v_op);
+ goto ret;
+ }
+
+ do {
+ if (!cmd->pending)
+ break;
+
+ rte_delay_ms(ICE_DCF_ARQ_CHECK_TIME);
+ } while (i++ < ICE_DCF_ARQ_MAX_RETRIES);
+
+ if (cmd->v_ret != IAVF_SUCCESS) {
+ err = -1;
+ PMD_DRV_LOG(ERR,
+ "No response (%d times) or return failure (%d) for cmd %d",
+ i, cmd->v_ret, cmd->v_op);
+ }
+
+ret:
+ ice_dcf_aq_cmd_clear(hw, cmd);
+ rte_spinlock_unlock(&hw->vc_cmd_send_lock);
+ return err;
+}
+
+int
+ice_dcf_init_hw(struct rte_eth_dev *eth_dev, struct ice_dcf_hw *hw)
+{
+ struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ int ret;
+
+ hw->avf.hw_addr = pci_dev->mem_resource[0].addr;
+ hw->avf.back = hw;
+
+ hw->avf.bus.bus_id = pci_dev->addr.bus;
+ hw->avf.bus.device = pci_dev->addr.devid;
+ hw->avf.bus.func = pci_dev->addr.function;
+
+ hw->avf.device_id = pci_dev->id.device_id;
+ hw->avf.vendor_id = pci_dev->id.vendor_id;
+ hw->avf.subsystem_device_id = pci_dev->id.subsystem_device_id;
+ hw->avf.subsystem_vendor_id = pci_dev->id.subsystem_vendor_id;
+
+ hw->avf.aq.num_arq_entries = ICE_DCF_AQ_LEN;
+ hw->avf.aq.num_asq_entries = ICE_DCF_AQ_LEN;
+ hw->avf.aq.arq_buf_size = ICE_DCF_AQ_BUF_SZ;
+ hw->avf.aq.asq_buf_size = ICE_DCF_AQ_BUF_SZ;
+
+ rte_spinlock_init(&hw->vc_cmd_send_lock);
+ rte_spinlock_init(&hw->vc_cmd_queue_lock);
+ TAILQ_INIT(&hw->vc_cmd_queue);
+
+ hw->arq_buf = rte_zmalloc("arq_buf", ICE_DCF_AQ_BUF_SZ, 0);
+ if (hw->arq_buf == NULL) {
+ PMD_INIT_LOG(ERR, "unable to allocate AdminQ buffer memory");
+ goto err;
+ }
+
+ ret = iavf_set_mac_type(&hw->avf);
+ if (ret) {
+ PMD_INIT_LOG(ERR, "set_mac_type failed: %d", ret);
+ goto err;
+ }
+
+ ret = ice_dcf_check_reset_done(hw);
+ if (ret) {
+ PMD_INIT_LOG(ERR, "VF is still resetting");
+ goto err;
+ }
+
+ ret = iavf_init_adminq(&hw->avf);
+ if (ret) {
+ PMD_INIT_LOG(ERR, "init_adminq failed: %d", ret);
+ goto err;
+ }
+
+ if (ice_dcf_init_check_api_version(hw)) {
+ PMD_INIT_LOG(ERR, "check_api version failed");
+ goto err_api;
+ }
+
+ hw->vf_res = rte_zmalloc("vf_res", ICE_DCF_VF_RES_BUF_SZ, 0);
+ if (hw->vf_res == NULL) {
+ PMD_INIT_LOG(ERR, "unable to allocate vf_res memory");
+ goto err_api;
+ }
+
+ if (ice_dcf_get_vf_resource(hw)) {
+ PMD_INIT_LOG(ERR, "Failed to get VF resource");
+ goto err_alloc;
+ }
+
+ rte_intr_callback_register(&pci_dev->intr_handle,
+ ice_dcf_dev_interrupt_handler, hw);
+ rte_intr_enable(&pci_dev->intr_handle);
+ ice_dcf_enable_irq0(hw);
+
+ return 0;
+
+err_alloc:
+ rte_free(hw->vf_res);
+err_api:
+ iavf_shutdown_adminq(&hw->avf);
+err:
+ rte_free(hw->arq_buf);
+
+ return -1;
+}
+
+void
+ice_dcf_uninit_hw(struct rte_eth_dev *eth_dev, struct ice_dcf_hw *hw)
+{
+ struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_intr_handle *intr_handle = &pci_dev->intr_handle;
+
+ ice_dcf_disable_irq0(hw);
+ rte_intr_disable(intr_handle);
+ rte_intr_callback_unregister(intr_handle,
+ ice_dcf_dev_interrupt_handler, hw);
+
+ iavf_shutdown_adminq(&hw->avf);
+
+ rte_free(hw->arq_buf);
+ rte_free(hw->vf_res);
+}
diff --git a/drivers/net/ice/ice_dcf.h b/drivers/net/ice/ice_dcf.h
new file mode 100644
index 000000000..f44c09db2
--- /dev/null
+++ b/drivers/net/ice/ice_dcf.h
@@ -0,0 +1,52 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#ifndef _ICE_DCF_H_
+#define _ICE_DCF_H_
+
+#include <rte_ethdev_driver.h>
+
+#include <iavf_prototype.h>
+#include <iavf_adminq_cmd.h>
+#include <iavf_type.h>
+
+#include "ice_logs.h"
+
+struct dcf_virtchnl_cmd {
+ TAILQ_ENTRY(dcf_virtchnl_cmd) next;
+
+ enum virtchnl_ops v_op;
+ enum iavf_status v_ret;
+
+ uint16_t req_msglen;
+ uint8_t *req_msg;
+
+ uint16_t rsp_msglen;
+ uint16_t rsp_buflen;
+ uint8_t *rsp_msgbuf;
+
+ volatile int pending;
+};
+
+struct ice_dcf_hw {
+ struct iavf_hw avf;
+
+ rte_spinlock_t vc_cmd_send_lock;
+ rte_spinlock_t vc_cmd_queue_lock;
+ TAILQ_HEAD(, dcf_virtchnl_cmd) vc_cmd_queue;
+ uint8_t *arq_buf;
+
+ struct virtchnl_version_info virtchnl_version;
+ struct virtchnl_vf_resource *vf_res; /* VF resource */
+ struct virtchnl_vsi_resource *vsi_res; /* LAN VSI */
+ uint16_t vsi_id;
+};
+
+int ice_dcf_execute_virtchnl_cmd(struct ice_dcf_hw *hw,
+ struct dcf_virtchnl_cmd *cmd);
+
+int ice_dcf_init_hw(struct rte_eth_dev *eth_dev, struct ice_dcf_hw *hw);
+void ice_dcf_uninit_hw(struct rte_eth_dev *eth_dev, struct ice_dcf_hw *hw);
+
+#endif /* _ICE_DCF_H_ */
diff --git a/drivers/net/ice/ice_dcf_ethdev.c b/drivers/net/ice/ice_dcf_ethdev.c
new file mode 100644
index 000000000..23f82a487
--- /dev/null
+++ b/drivers/net/ice/ice_dcf_ethdev.c
@@ -0,0 +1,317 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#include <errno.h>
+#include <stdbool.h>
+#include <sys/types.h>
+#include <sys/ioctl.h>
+#include <unistd.h>
+
+#include <rte_interrupts.h>
+#include <rte_debug.h>
+#include <rte_pci.h>
+#include <rte_atomic.h>
+#include <rte_eal.h>
+#include <rte_ether.h>
+#include <rte_ethdev_pci.h>
+#include <rte_kvargs.h>
+#include <rte_malloc.h>
+#include <rte_memzone.h>
+#include <rte_dev.h>
+
+#include <iavf_devids.h>
+
+#include "ice_generic_flow.h"
+#include "ice_dcf_ethdev.h"
+
+static uint16_t
+ice_dcf_recv_pkts(__rte_unused void *rx_queue,
+ __rte_unused struct rte_mbuf **bufs,
+ __rte_unused uint16_t nb_pkts)
+{
+ return 0;
+}
+
+static uint16_t
+ice_dcf_xmit_pkts(__rte_unused void *tx_queue,
+ __rte_unused struct rte_mbuf **bufs,
+ __rte_unused uint16_t nb_pkts)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_start(struct rte_eth_dev *dev)
+{
+ dev->data->dev_link.link_status = ETH_LINK_UP;
+
+ return 0;
+}
+
+static void
+ice_dcf_dev_stop(struct rte_eth_dev *dev)
+{
+ dev->data->dev_link.link_status = ETH_LINK_DOWN;
+}
+
+static int
+ice_dcf_dev_configure(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_info_get(struct rte_eth_dev *dev,
+ struct rte_eth_dev_info *dev_info)
+{
+ struct ice_dcf_adapter *adapter = dev->data->dev_private;
+
+ dev_info->max_mac_addrs = 1;
+ dev_info->max_rx_pktlen = (uint32_t)-1;
+ dev_info->max_rx_queues = RTE_DIM(adapter->rxqs);
+ dev_info->max_tx_queues = RTE_DIM(adapter->txqs);
+
+ return 0;
+}
+
+static int
+ice_dcf_stats_get(__rte_unused struct rte_eth_dev *dev,
+ __rte_unused struct rte_eth_stats *igb_stats)
+{
+ return 0;
+}
+
+static int
+ice_dcf_stats_reset(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_promiscuous_enable(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_promiscuous_disable(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_allmulticast_enable(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_allmulticast_disable(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_filter_ctrl(struct rte_eth_dev *dev,
+ enum rte_filter_type filter_type,
+ __rte_unused enum rte_filter_op filter_op,
+ __rte_unused void *arg)
+{
+ int ret = 0;
+
+ if (!dev)
+ return -EINVAL;
+
+ switch (filter_type) {
+ default:
+ PMD_DRV_LOG(WARNING, "Filter type (%d) not supported",
+ filter_type);
+ ret = -EINVAL;
+ break;
+ }
+
+ return ret;
+}
+
+static void
+ice_dcf_dev_close(struct rte_eth_dev *dev)
+{
+ struct ice_dcf_adapter *adapter = dev->data->dev_private;
+
+ if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+ return;
+
+ dev->dev_ops = NULL;
+ dev->rx_pkt_burst = NULL;
+ dev->tx_pkt_burst = NULL;
+ dev->data->mac_addrs = NULL;
+
+ ice_dcf_uninit_hw(dev, &adapter->real_hw);
+}
+
+static void
+ice_dcf_queue_release(__rte_unused void *q)
+{
+}
+
+static int
+ice_dcf_link_update(__rte_unused struct rte_eth_dev *dev,
+ __rte_unused int wait_to_complete)
+{
+ return 0;
+}
+
+static int
+ice_dcf_rx_queue_setup(struct rte_eth_dev *dev,
+ uint16_t rx_queue_id,
+ __rte_unused uint16_t nb_rx_desc,
+ __rte_unused unsigned int socket_id,
+ __rte_unused const struct rte_eth_rxconf *rx_conf,
+ __rte_unused struct rte_mempool *mb_pool)
+{
+ struct ice_dcf_adapter *adapter = dev->data->dev_private;
+
+ dev->data->rx_queues[rx_queue_id] = &adapter->rxqs[rx_queue_id];
+
+ return 0;
+}
+
+static int
+ice_dcf_tx_queue_setup(struct rte_eth_dev *dev,
+ uint16_t tx_queue_id,
+ __rte_unused uint16_t nb_tx_desc,
+ __rte_unused unsigned int socket_id,
+ __rte_unused const struct rte_eth_txconf *tx_conf)
+{
+ struct ice_dcf_adapter *adapter = dev->data->dev_private;
+
+ dev->data->tx_queues[tx_queue_id] = &adapter->txqs[tx_queue_id];
+
+ return 0;
+}
+
+static const struct eth_dev_ops ice_dcf_eth_dev_ops = {
+ .dev_start = ice_dcf_dev_start,
+ .dev_stop = ice_dcf_dev_stop,
+ .dev_close = ice_dcf_dev_close,
+ .dev_configure = ice_dcf_dev_configure,
+ .dev_infos_get = ice_dcf_dev_info_get,
+ .rx_queue_setup = ice_dcf_rx_queue_setup,
+ .tx_queue_setup = ice_dcf_tx_queue_setup,
+ .rx_queue_release = ice_dcf_queue_release,
+ .tx_queue_release = ice_dcf_queue_release,
+ .link_update = ice_dcf_link_update,
+ .stats_get = ice_dcf_stats_get,
+ .stats_reset = ice_dcf_stats_reset,
+ .promiscuous_enable = ice_dcf_dev_promiscuous_enable,
+ .promiscuous_disable = ice_dcf_dev_promiscuous_disable,
+ .allmulticast_enable = ice_dcf_dev_allmulticast_enable,
+ .allmulticast_disable = ice_dcf_dev_allmulticast_disable,
+ .filter_ctrl = ice_dcf_dev_filter_ctrl,
+};
+
+static int
+ice_dcf_dev_init(struct rte_eth_dev *eth_dev)
+{
+ struct ice_dcf_adapter *adapter = eth_dev->data->dev_private;
+
+ eth_dev->dev_ops = &ice_dcf_eth_dev_ops;
+ eth_dev->rx_pkt_burst = ice_dcf_recv_pkts;
+ eth_dev->tx_pkt_burst = ice_dcf_xmit_pkts;
+
+ if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+ return 0;
+
+ eth_dev->data->dev_flags |= RTE_ETH_DEV_CLOSE_REMOVE;
+
+ if (ice_dcf_init_hw(eth_dev, &adapter->real_hw) != 0) {
+ PMD_INIT_LOG(ERR, "Failed to init DCF hardware");
+ return -1;
+ }
+
+ rte_eth_random_addr(adapter->mac_addr.addr_bytes);
+ eth_dev->data->mac_addrs = &adapter->mac_addr;
+
+ return 0;
+}
+
+static int
+ice_dcf_dev_uninit(struct rte_eth_dev *eth_dev)
+{
+ ice_dcf_dev_close(eth_dev);
+
+ return 0;
+}
+
+static int
+ice_dcf_cap_check_handler(__rte_unused const char *key,
+ const char *value, __rte_unused void *opaque)
+{
+ if (strcmp(value, "dcf"))
+ return -1;
+
+ return 0;
+}
+
+static int
+ice_dcf_cap_selected(struct rte_devargs *devargs)
+{
+ struct rte_kvargs *kvlist;
+ const char *key = "cap";
+ int ret = 0;
+
+ if (devargs == NULL)
+ return 0;
+
+ kvlist = rte_kvargs_parse(devargs->args, NULL);
+ if (kvlist == NULL)
+ return 0;
+
+ if (!rte_kvargs_count(kvlist, key))
+ goto exit;
+
+ /* dcf capability selected when there's a key-value pair: cap=dcf */
+ if (rte_kvargs_process(kvlist, key,
+ ice_dcf_cap_check_handler, NULL) < 0)
+ goto exit;
+
+ ret = 1;
+
+exit:
+ rte_kvargs_free(kvlist);
+ return ret;
+}
+
+static int eth_ice_dcf_pci_probe(__rte_unused struct rte_pci_driver *pci_drv,
+ struct rte_pci_device *pci_dev)
+{
+ if (!ice_dcf_cap_selected(pci_dev->device.devargs))
+ return 1;
+
+ return rte_eth_dev_pci_generic_probe(pci_dev,
+ sizeof(struct ice_dcf_adapter),
+ ice_dcf_dev_init);
+}
+
+static int eth_ice_dcf_pci_remove(struct rte_pci_device *pci_dev)
+{
+ return rte_eth_dev_pci_generic_remove(pci_dev, ice_dcf_dev_uninit);
+}
+
+static const struct rte_pci_id pci_id_ice_dcf_map[] = {
+ { RTE_PCI_DEVICE(IAVF_INTEL_VENDOR_ID, IAVF_DEV_ID_ADAPTIVE_VF) },
+ { .vendor_id = 0, /* sentinel */ },
+};
+
+static struct rte_pci_driver rte_ice_dcf_pmd = {
+ .id_table = pci_id_ice_dcf_map,
+ .drv_flags = RTE_PCI_DRV_NEED_MAPPING,
+ .probe = eth_ice_dcf_pci_probe,
+ .remove = eth_ice_dcf_pci_remove,
+};
+
+RTE_PMD_REGISTER_PCI(net_ice_dcf, rte_ice_dcf_pmd);
+RTE_PMD_REGISTER_PCI_TABLE(net_ice_dcf, pci_id_ice_dcf_map);
+RTE_PMD_REGISTER_KMOD_DEP(net_ice_dcf, "* igb_uio | vfio-pci");
+RTE_PMD_REGISTER_PARAM_STRING(net_ice_dcf, "cap=dcf");
diff --git a/drivers/net/ice/ice_dcf_ethdev.h b/drivers/net/ice/ice_dcf_ethdev.h
new file mode 100644
index 000000000..0c34a0095
--- /dev/null
+++ b/drivers/net/ice/ice_dcf_ethdev.h
@@ -0,0 +1,24 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#ifndef _ICE_DCF_ETHDEV_H_
+#define _ICE_DCF_ETHDEV_H_
+
+#include "ice_ethdev.h"
+#include "ice_dcf.h"
+
+#define ICE_DCF_MAX_RINGS 1
+
+struct ice_dcf_queue {
+ uint64_t dummy;
+};
+
+struct ice_dcf_adapter {
+ struct ice_dcf_hw real_hw;
+ struct rte_ether_addr mac_addr;
+ struct ice_dcf_queue rxqs[ICE_DCF_MAX_RINGS];
+ struct ice_dcf_queue txqs[ICE_DCF_MAX_RINGS];
+};
+
+#endif /* _ICE_DCF_ETHDEV_H_ */
diff --git a/drivers/net/ice/meson.build b/drivers/net/ice/meson.build
index f9e897bbc..0ba9668d1 100644
--- a/drivers/net/ice/meson.build
+++ b/drivers/net/ice/meson.build
@@ -15,8 +15,8 @@ sources = files(
'ice_hash.c'
)
-deps += ['hash']
-includes += include_directories('base')
+deps += ['hash', 'net', 'common_iavf']
+includes += include_directories('base', '../../common/iavf')
if arch_subdir == 'x86'
sources += files('ice_rxtx_vec_sse.c')
@@ -37,4 +37,7 @@ if arch_subdir == 'x86'
endif
endif
+sources += files('ice_dcf.c',
+ 'ice_dcf_ethdev.c')
+
install_headers('rte_pmd_ice.h')
diff --git a/mk/rte.app.mk b/mk/rte.app.mk
index d295ca0a5..f3798a09f 100644
--- a/mk/rte.app.mk
+++ b/mk/rte.app.mk
@@ -185,6 +185,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_I40E_PMD) += -lrte_pmd_i40e
_LDLIBS-$(CONFIG_RTE_LIBRTE_IAVF_PMD) += -lrte_pmd_iavf
_LDLIBS-$(CONFIG_RTE_LIBRTE_ICE_PMD) += -lrte_pmd_ice
IAVF-y := $(CONFIG_RTE_LIBRTE_IAVF_PMD)
+IAVF-y += $(CONFIG_RTE_LIBRTE_ICE_PMD)
ifeq ($(findstring y,$(IAVF-y)),y)
_LDLIBS-y += -lrte_common_iavf
endif
--
2.26.0
^ permalink raw reply [relevance 1%]
* Re: [dpdk-dev] [RFC] ring: make ring implementation non-inlined
2020-03-25 21:09 4% ` Jerin Jacob
2020-03-26 0:28 0% ` Ananyev, Konstantin
@ 2020-03-26 8:04 4% ` Morten Brørup
1 sibling, 0 replies; 200+ results
From: Morten Brørup @ 2020-03-26 8:04 UTC (permalink / raw)
To: Jerin Jacob, Konstantin Ananyev
Cc: dpdk-dev, Olivier Matz, Honnappa Nagarahalli, Jerin Jacob,
David Christensen, Stephen Hemminger
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Jerin Jacob
> Sent: Wednesday, March 25, 2020 10:09 PM
>
> On Fri, Mar 20, 2020 at 10:11 PM Konstantin Ananyev
> <konstantin.ananyev@intel.com> wrote:
> >
> > As was discussed here:
> > http://mails.dpdk.org/archives/dev/2020-February/158586.html
> > this RFC aimed to hide ring internals into .c and make all
> > ring functions non-inlined. In theory that might help to
> > maintain ABI stability in future.
> > This is just a POC to measure the impact of proposed idea,
> > proper implementation would definetly need some extra effort.
> > On IA box (SKX) ring_perf_autotest shows ~20-30 cycles extra for
> > enqueue+dequeue pair. On some more realistic code, I suspect
> > the impact it might be a bit higher.
> > For MP/MC bulk transfers degradation seems quite small,
> > though for SP/SC and/or small transfers it is more then noticable
> > (see exact numbers below).
> > From my perspective we'd probably keep it inlined for now
> > to avoid any non-anticipated perfomance degradations.
> > Though intersted to see perf results and opinions from
> > other interested parties.
>
> +1
>
> My reasoning is a bit different, DPDK is using in embedded boxes too
> where performance has
> more weight than ABI stuff.
As a network appliance vendor I can confirm that we certainly care more about performance than ABI stability. ABI stability is irrelevant for us; and API instability is a non-recurring engineering cost each time we choose to switch to a new DPDK version, which we only do if we cannot avoid it, e.g. due to new drivers, security fixes or new features that we want to use.
For us, the trend pointed in the wrong direction when DPDK switched the preference towards runtime configurability and deprecated compile time configurability. I do understand the reasoning behind it, and the impact is minimal, so we accept it.
However, if DPDK starts sacrificing performance of the core libraries for the benefits of the GNU/Linux distributors, network appliance vendors may put more effort into sticking with old DPDK versions instead of updating.
> I think we need to focus first on slow
> path APIs ABI stuff.
>
> I spend a few cycles to apply this patch +
> http://mails.dpdk.org/archives/dev/2020-February/158586.html
> on top of the tree, there are a lot of conflicts. If I get a mergeable
> patch then I will test it on an arm64 box.
>
>
>
>
>
> >
> > Intel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz
> > ring_perf_autotest (without patch/with patch)
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v2 0/4] Introduce IF proxy library
2020-03-25 8:08 3% ` David Marchand
2020-03-25 11:11 4% ` Morten Brørup
@ 2020-03-26 12:41 3% ` Andrzej Ostruszka
1 sibling, 0 replies; 200+ results
From: Andrzej Ostruszka @ 2020-03-26 12:41 UTC (permalink / raw)
To: David Marchand, Andrzej Ostruszka; +Cc: dev
Thank you David for taking time to look at this.
On 3/25/20 9:08 AM, David Marchand wrote:
> Hello Andrzej,
>
> On Tue, Mar 10, 2020 at 12:11 PM Andrzej Ostruszka
[...]
> I can see we end up exposing structures for registering callbacks.
Right. I was thinking more in terms of user convenience so it seemed
like a good choice to gather them in one struct and call 'register'
once. The fact that the same structure is used to keep them is an
implementation choice and this can be decoupled.
> Did you consider some ways to avoid exposure of those? (thinking of
> ABI maintenance for when this library will elect to non-experimental).
I will. So far I used the union for the input since I like when things
are well typed :) and there is no need for casting. However I will
spend some time on this and will get back to you soon (if you have
already something in your head please share). Right now I'm thinking
about taking array of callbacks with each entry being ("event type",
callback) pair, however need to figure out how to have minimum amount of
type casting.
> I can see some canary at the end of an enum, can we do without it?
I followed discussion on the list about that and have thought about it
but deemed that to be not a problem. This enum value is never returned
from the library and the event type enum is never taken as an input
(only used for event notification). So this is really implementation
thing and you are right it would be better to hide it. This might be
resolved by itself when I come up with something for the above ABI
stability issue.
> Is there a pb with merging ifpx support into the existing l3fwd
> application rather than introduce a new example?
I don't see a problem with merging per se. That might be my
misunderstanding of what the examples are. I thought that each library
can have its own example to show how it is supposed to be used. So
decided to have simplified version of l3fwd - and initially I thought
about updating l3fwd but it has some non-trivial optimizations and two
modes of operations (hash/lpm) so I wanted something simple to just show
how to use the library. Don't know what is the reason for this
bi-modality of l3fwd:
- if this is just a need to show LPM/Hash in use then I can replace that
with single mode of l3fwd-ifpx where LPM is used for routing and Hash
is used to keep neighbouring info
- if this is to show that both LPM and Hash can be used for routing then
it would complicate things as these two have different update
properties.
I assume (but don't have a solid proof for that) that LPM can be updated
by a single writer while being used by multiple readers and use this
assumption to show how such structures can be updated (Morten please
cover your eyes ;-)) from a callback while other can be updated via
event queuing.
So if the community decides that it would be OK to morph l3fwd to:
- strip the bi-modality
- use LPM and Hash for different things (not both for routing)
then I'm OK with that and will happily do that. Otherwise adding IFPX
to l3fwd will end up with two modes with different routing
implementation and different update strategies - a bit like two
different apps bundled into one and chosen by the command arg.
There is also a question of not having FreeBSD and Windows support yet -
so things might get complicated.
With regards
Andrzej Ostruszka
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH v2 0/4] Introduce IF proxy library
2020-03-25 11:11 4% ` Morten Brørup
@ 2020-03-26 17:42 3% ` Andrzej Ostruszka
0 siblings, 0 replies; 200+ results
From: Andrzej Ostruszka @ 2020-03-26 17:42 UTC (permalink / raw)
To: Morten Brørup, Andrzej Ostruszka, David Marchand; +Cc: dev
On 3/25/20 12:11 PM, Morten Brørup wrote:
[...]
>>> - Notification scheme has been changed - instead of having just
>>> callbacks now event queueing is also available (or a mix of those
>>> two).
>
> Thank you for adding event queueing!
That was actually a good input from you - thank you.
> David mentions ABI forward compatibility below.
> Consider using a dynamically sized generic TLV (type, length, value)
> message format instead of a big union structure for the events. This
> would make it easier to extend the list of event types without breaking
> the ABI.
My understanding is that David was talking about registering of
callbacks and you want to extend this to event definition.
So let's focus on one example:
...
RTE_IFPX_NEIGH_ADD,
RTE_IFPX_NEIGH_DEL,
...
struct rte_ifpx_neigh_change {
uint16_t port_id;
struct rte_ether_addr mac;
uint32_t ip;
};
Right now the event is defined as:
struct rte_ifpx_event {
enum rte_ifpx_event_type type;
union {
...
struct rte_ifpx_neigh_change neigh_change;
...
};
};
So what the user does is a switch on event->type:
switch (ev->type) {
case RTE_IFPX_NEIGH_ADD:
handle_neigh_add(lconf, &ev->neigh_change);
break;
case RTE_IFPX_NEIGH_DEL:
handle_neigh_del(lconf, &ev->neigh_change);
break;
How does adding more event types to this union would break ABI? User
gets event from the queue (allocated by the lib) checks the type and
casts the pointer past the 'type' to proper event definition. And when
done with the event simply free()s it (BTW right now it is malloc() not
rte_malloc() - should I change that?). If app links against newer
version of lib then it might get type which it does not
understand/handle so it should skip (possibly with a warning). I'm not
sure how changing rte_ifpx_event to:
struct rte_ifpx_event {
enut rte_ifpx_event_type type;
int length;
uint8_t data[];
};
would help here. The user would need to cast data based on event type
whereas now it takes address of a proper union member - and the union is
there only to avoid casting. In both cases what is important is that
RTE_IFPX_NEIGH_ADD/DEL and "struct rte_ifpx_neigh_change" don't change
between versions (new values can be added - or new versions of the
previously existing events when trying to make a change).
And for the callbacks it is more or less the same - library will prepare
data and call callback with a pointer to this data. Handling of new
event types should be automatic when I implement what David wanted -
simply lib callback for the new event will be NULL nothing will be
called and application will work without problems.
> And I am still strongly opposed to the callback method:
Noted - however for now I would like to keep them. I don't have much
experience with this library so if they prove to be inadequate then we
will remove them. Right now they seem to add some flexibility that I like:
- if something should be changed globally and once (and it is safe to do
so!) then it can be done from the callback
- if something can be prepared once and consumed later by lcores then it
can be done in callback and the callback returns 0 so that event is
still queued and lcores (under assumption that queues are per lcore)
pick up what has been prepared.
> The callbacks are handled as DPDK interrupts, which are running in a non-DPDK
> thread, i.e. a running callback may be preempted by some other Linux process.
> This makes it difficult to implement callbacks correctly.
> The risk of someone calling a non-thread safe function from a callback is high,
> e.g. DPDK hash table manipulation (except lookup) is not thread safe.
>
> Your documentation is far too vague about this:
> Please note however that the context in which these callbacks are
> called is most probably different from the one in which packets are
> handled and it is application writer responsibility to use proper
> synchronization mechanisms - if they are needed.
>
> You need a big fat WARNING about how difficult the DPDK interrupt thread is to
> work with. As I described above, it is not "most probably" it is "certainly" a
> very different kind of context.
OK. Will update in next version.
> Did you check that the functions you use in your example callbacks are all
> thread safe and non-blocking, so they can safely be called from a non-DPDK thread
> that may be preempted by a another Linux process?
I believe so. However there is a big question whether my assumption
about LPM is correct. I've looked at the code and it looks like it so
but I'm not in power to authoritatively declare it. So again, to me LPM
looks like safe to be changed by a single writer while being used by
multiple readers (with an obvious transient period when rule is being
expanded and some IPs might go with an old and some with a new destination).
With regards
Andrzej Ostruszka
^ permalink raw reply [relevance 3%]
* [dpdk-dev] [PATCH v2 6/8] eal: move common header files
@ 2020-03-27 1:15 3% ` Thomas Monjalon
0 siblings, 0 replies; 200+ results
From: Thomas Monjalon @ 2020-03-27 1:15 UTC (permalink / raw)
To: dev
Cc: david.marchand, Neil Horman, John McNamara, Marko Kovacevic,
Hemant Agrawal, Sachin Saxena, Matan Azrad, Shahaf Shuler,
Viacheslav Ovsiienko, Gagandeep Singh, Akhil Goyal, Nipun Gupta,
Ferruh Yigit, Cristian Dumitrescu, Harry van Haaren,
Bruce Richardson, Phil Yang, Joyce Kong, Anatoly Burakov,
Mattias Rönnblom, Olivier Matz
The EAL API (with doxygen documentation) is moved from
common/include/ to include/, which makes more clear that
it is the global API for all environments and architectures.
Note that the arch-specific and OS-specific include files are not
in this global include directory, but include/generic/ should
cover the doxygen documentation for them.
Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
---
MAINTAINERS | 26 ++++----
buildtools/pmdinfogen/meson.build | 2 +-
doc/api/doxy-api.conf.in | 4 +-
doc/guides/rel_notes/known_issues.rst | 2 +-
drivers/bus/dpaa/Makefile | 2 +-
drivers/common/mlx5/meson.build | 2 +-
drivers/crypto/caam_jr/Makefile | 2 +-
drivers/crypto/dpaa_sec/Makefile | 2 +-
drivers/event/dpaa/Makefile | 2 +-
drivers/net/dpaa/Makefile | 2 +-
kernel/linux/igb_uio/meson.build | 2 +-
kernel/linux/kni/meson.build | 2 +-
lib/librte_cfgfile/Makefile | 2 +-
lib/librte_eal/Makefile | 6 +-
lib/librte_eal/common/Makefile | 38 -----------
lib/librte_eal/common/meson.build | 65 ++-----------------
lib/librte_eal/common/rte_service.c | 2 +-
lib/librte_eal/freebsd/eal/Makefile | 2 +-
lib/librte_eal/include/Makefile | 19 ++++++
.../{common => }/include/generic/rte_atomic.h | 0
.../include/generic/rte_byteorder.h | 0
.../include/generic/rte_cpuflags.h | 0
.../{common => }/include/generic/rte_cycles.h | 0
.../{common => }/include/generic/rte_io.h | 0
.../include/generic/rte_mcslock.h | 0
.../{common => }/include/generic/rte_memcpy.h | 0
.../{common => }/include/generic/rte_pause.h | 0
.../include/generic/rte_prefetch.h | 0
.../{common => }/include/generic/rte_rwlock.h | 0
.../include/generic/rte_spinlock.h | 0
.../include/generic/rte_ticketlock.h | 0
.../{common => }/include/generic/rte_vect.h | 0
lib/librte_eal/include/meson.build | 64 ++++++++++++++++++
.../{common => }/include/rte_alarm.h | 0
.../{common => }/include/rte_bitmap.h | 0
.../include/rte_branch_prediction.h | 0
lib/librte_eal/{common => }/include/rte_bus.h | 0
.../{common => }/include/rte_class.h | 0
.../{common => }/include/rte_common.h | 0
.../{common => }/include/rte_compat.h | 0
.../{common => }/include/rte_debug.h | 0
lib/librte_eal/{common => }/include/rte_dev.h | 0
.../{common => }/include/rte_devargs.h | 0
lib/librte_eal/{common => }/include/rte_eal.h | 0
.../{common => }/include/rte_eal_interrupts.h | 0
.../{common => }/include/rte_eal_memconfig.h | 0
.../{common => }/include/rte_errno.h | 0
.../{common => }/include/rte_fbarray.h | 0
.../include/rte_function_versioning.h | 0
.../{common => }/include/rte_hexdump.h | 0
.../{common => }/include/rte_hypervisor.h | 0
.../{common => }/include/rte_interrupts.h | 0
.../{common => }/include/rte_keepalive.h | 0
.../{common => }/include/rte_launch.h | 0
.../{common => }/include/rte_lcore.h | 0
lib/librte_eal/{common => }/include/rte_log.h | 0
.../{common => }/include/rte_malloc.h | 0
.../{common => }/include/rte_memory.h | 0
.../{common => }/include/rte_memzone.h | 0
.../{common => }/include/rte_option.h | 0
.../include/rte_pci_dev_feature_defs.h | 0
.../include/rte_pci_dev_features.h | 0
.../{common => }/include/rte_per_lcore.h | 0
.../{common => }/include/rte_random.h | 0
.../{common => }/include/rte_reciprocal.h | 0
.../{common => }/include/rte_service.h | 0
.../include/rte_service_component.h | 0
.../{common => }/include/rte_string_fns.h | 0
.../{common => }/include/rte_tailq.h | 0
.../{common => }/include/rte_test.h | 0
.../{common => }/include/rte_time.h | 0
.../{common => }/include/rte_uuid.h | 0
.../{common => }/include/rte_version.h | 0
.../{common => }/include/rte_vfio.h | 0
lib/librte_eal/linux/eal/Makefile | 2 +-
lib/librte_eal/meson.build | 4 +-
lib/librte_kvargs/Makefile | 2 +-
meson.build | 2 +-
78 files changed, 124 insertions(+), 134 deletions(-)
delete mode 100644 lib/librte_eal/common/Makefile
create mode 100644 lib/librte_eal/include/Makefile
rename lib/librte_eal/{common => }/include/generic/rte_atomic.h (100%)
rename lib/librte_eal/{common => }/include/generic/rte_byteorder.h (100%)
rename lib/librte_eal/{common => }/include/generic/rte_cpuflags.h (100%)
rename lib/librte_eal/{common => }/include/generic/rte_cycles.h (100%)
rename lib/librte_eal/{common => }/include/generic/rte_io.h (100%)
rename lib/librte_eal/{common => }/include/generic/rte_mcslock.h (100%)
rename lib/librte_eal/{common => }/include/generic/rte_memcpy.h (100%)
rename lib/librte_eal/{common => }/include/generic/rte_pause.h (100%)
rename lib/librte_eal/{common => }/include/generic/rte_prefetch.h (100%)
rename lib/librte_eal/{common => }/include/generic/rte_rwlock.h (100%)
rename lib/librte_eal/{common => }/include/generic/rte_spinlock.h (100%)
rename lib/librte_eal/{common => }/include/generic/rte_ticketlock.h (100%)
rename lib/librte_eal/{common => }/include/generic/rte_vect.h (100%)
create mode 100644 lib/librte_eal/include/meson.build
rename lib/librte_eal/{common => }/include/rte_alarm.h (100%)
rename lib/librte_eal/{common => }/include/rte_bitmap.h (100%)
rename lib/librte_eal/{common => }/include/rte_branch_prediction.h (100%)
rename lib/librte_eal/{common => }/include/rte_bus.h (100%)
rename lib/librte_eal/{common => }/include/rte_class.h (100%)
rename lib/librte_eal/{common => }/include/rte_common.h (100%)
rename lib/librte_eal/{common => }/include/rte_compat.h (100%)
rename lib/librte_eal/{common => }/include/rte_debug.h (100%)
rename lib/librte_eal/{common => }/include/rte_dev.h (100%)
rename lib/librte_eal/{common => }/include/rte_devargs.h (100%)
rename lib/librte_eal/{common => }/include/rte_eal.h (100%)
rename lib/librte_eal/{common => }/include/rte_eal_interrupts.h (100%)
rename lib/librte_eal/{common => }/include/rte_eal_memconfig.h (100%)
rename lib/librte_eal/{common => }/include/rte_errno.h (100%)
rename lib/librte_eal/{common => }/include/rte_fbarray.h (100%)
rename lib/librte_eal/{common => }/include/rte_function_versioning.h (100%)
rename lib/librte_eal/{common => }/include/rte_hexdump.h (100%)
rename lib/librte_eal/{common => }/include/rte_hypervisor.h (100%)
rename lib/librte_eal/{common => }/include/rte_interrupts.h (100%)
rename lib/librte_eal/{common => }/include/rte_keepalive.h (100%)
rename lib/librte_eal/{common => }/include/rte_launch.h (100%)
rename lib/librte_eal/{common => }/include/rte_lcore.h (100%)
rename lib/librte_eal/{common => }/include/rte_log.h (100%)
rename lib/librte_eal/{common => }/include/rte_malloc.h (100%)
rename lib/librte_eal/{common => }/include/rte_memory.h (100%)
rename lib/librte_eal/{common => }/include/rte_memzone.h (100%)
rename lib/librte_eal/{common => }/include/rte_option.h (100%)
rename lib/librte_eal/{common => }/include/rte_pci_dev_feature_defs.h (100%)
rename lib/librte_eal/{common => }/include/rte_pci_dev_features.h (100%)
rename lib/librte_eal/{common => }/include/rte_per_lcore.h (100%)
rename lib/librte_eal/{common => }/include/rte_random.h (100%)
rename lib/librte_eal/{common => }/include/rte_reciprocal.h (100%)
rename lib/librte_eal/{common => }/include/rte_service.h (100%)
rename lib/librte_eal/{common => }/include/rte_service_component.h (100%)
rename lib/librte_eal/{common => }/include/rte_string_fns.h (100%)
rename lib/librte_eal/{common => }/include/rte_tailq.h (100%)
rename lib/librte_eal/{common => }/include/rte_test.h (100%)
rename lib/librte_eal/{common => }/include/rte_time.h (100%)
rename lib/librte_eal/{common => }/include/rte_uuid.h (100%)
rename lib/librte_eal/{common => }/include/rte_version.h (100%)
rename lib/librte_eal/{common => }/include/rte_vfio.h (100%)
diff --git a/MAINTAINERS b/MAINTAINERS
index 89aff59ccf..3e842625f2 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -142,8 +142,8 @@ F: .ci/
ABI versioning
M: Neil Horman <nhorman@tuxdriver.com>
-F: lib/librte_eal/common/include/rte_compat.h
-F: lib/librte_eal/common/include/rte_function_versioning.h
+F: lib/librte_eal/include/rte_compat.h
+F: lib/librte_eal/include/rte_function_versioning.h
F: doc/guides/rel_notes/deprecation.rst
F: devtools/check-abi.sh
F: devtools/check-abi-version.sh
@@ -169,7 +169,7 @@ T: git://dpdk.org/dpdk
EAL API and common code
F: lib/librte_eal/common/
-F: lib/librte_eal/common/include/
+F: lib/librte_eal/include/
F: lib/librte_eal/rte_eal_version.map
F: doc/guides/prog_guide/env_abstraction_layer.rst
F: app/test/test_alarm.c
@@ -196,9 +196,9 @@ F: app/test/test_version.c
Memory Allocation
M: Anatoly Burakov <anatoly.burakov@intel.com>
-F: lib/librte_eal/common/include/rte_fbarray.h
-F: lib/librte_eal/common/include/rte_mem*
-F: lib/librte_eal/common/include/rte_malloc.h
+F: lib/librte_eal/include/rte_fbarray.h
+F: lib/librte_eal/include/rte_mem*
+F: lib/librte_eal/include/rte_malloc.h
F: lib/librte_eal/common/*malloc*
F: lib/librte_eal/common/eal_common_fbarray.c
F: lib/librte_eal/common/eal_common_mem*
@@ -214,7 +214,7 @@ F: app/test/test_memory.c
F: app/test/test_memzone.c
Keep alive
-F: lib/librte_eal/common/include/rte_keepalive.h
+F: lib/librte_eal/include/rte_keepalive.h
F: lib/librte_eal/common/rte_keepalive.c
F: examples/l2fwd-keepalive/
F: doc/guides/sample_app_ug/keep_alive.rst
@@ -230,30 +230,30 @@ F: doc/guides/sample_app_ug/multi_process.rst
Service Cores
M: Harry van Haaren <harry.van.haaren@intel.com>
-F: lib/librte_eal/common/include/rte_service.h
-F: lib/librte_eal/common/include/rte_service_component.h
+F: lib/librte_eal/include/rte_service.h
+F: lib/librte_eal/include/rte_service_component.h
F: lib/librte_eal/common/rte_service.c
F: doc/guides/prog_guide/service_cores.rst
F: app/test/test_service_cores.c
Bitmap
M: Cristian Dumitrescu <cristian.dumitrescu@intel.com>
-F: lib/librte_eal/common/include/rte_bitmap.h
+F: lib/librte_eal/include/rte_bitmap.h
F: app/test/test_bitmap.c
MCSlock - EXPERIMENTAL
M: Phil Yang <phil.yang@arm.com>
-F: lib/librte_eal/common/include/generic/rte_mcslock.h
+F: lib/librte_eal/include/generic/rte_mcslock.h
F: app/test/test_mcslock.c
Ticketlock
M: Joyce Kong <joyce.kong@arm.com>
-F: lib/librte_eal/common/include/generic/rte_ticketlock.h
+F: lib/librte_eal/include/generic/rte_ticketlock.h
F: app/test/test_ticketlock.c
Pseudo-random Number Generation
M: Mattias Rönnblom <mattias.ronnblom@ericsson.com>
-F: lib/librte_eal/common/include/rte_random.h
+F: lib/librte_eal/include/rte_random.h
F: lib/librte_eal/common/rte_random.c
F: app/test/test_rand_perf.c
diff --git a/buildtools/pmdinfogen/meson.build b/buildtools/pmdinfogen/meson.build
index 899ba112cd..7da415b3b7 100644
--- a/buildtools/pmdinfogen/meson.build
+++ b/buildtools/pmdinfogen/meson.build
@@ -6,7 +6,7 @@ if host_machine.system() == 'windows'
endif
pmdinfogen_inc = [global_inc]
-pmdinfogen_inc += include_directories('../../lib/librte_eal/common/include')
+pmdinfogen_inc += include_directories('../../lib/librte_eal/include')
pmdinfogen_inc += include_directories('../../lib/librte_pci')
pmdinfogen = executable('pmdinfogen',
'pmdinfogen.c',
diff --git a/doc/api/doxy-api.conf.in b/doc/api/doxy-api.conf.in
index 1c4392eecc..65e8146bef 100644
--- a/doc/api/doxy-api.conf.in
+++ b/doc/api/doxy-api.conf.in
@@ -17,8 +17,8 @@ INPUT = @TOPDIR@/doc/api/doxy-api-index.md \
@TOPDIR@/drivers/net/softnic \
@TOPDIR@/drivers/raw/dpaa2_cmdif \
@TOPDIR@/drivers/raw/dpaa2_qdma \
- @TOPDIR@/lib/librte_eal/common/include \
- @TOPDIR@/lib/librte_eal/common/include/generic \
+ @TOPDIR@/lib/librte_eal/include \
+ @TOPDIR@/lib/librte_eal/include/generic \
@TOPDIR@/lib/librte_acl \
@TOPDIR@/lib/librte_bbdev \
@TOPDIR@/lib/librte_bitratestats \
diff --git a/doc/guides/rel_notes/known_issues.rst b/doc/guides/rel_notes/known_issues.rst
index 68c3d22bea..de0782136d 100644
--- a/doc/guides/rel_notes/known_issues.rst
+++ b/doc/guides/rel_notes/known_issues.rst
@@ -127,7 +127,7 @@ HPET timers do not work on the Osage customer reference platform
work correctly, provided the BIOS supports HPET.
**Driver/Module**:
- ``lib/librte_eal/common/include/rte_cycles.h``
+ ``lib/librte_eal/include/rte_cycles.h``
Not all variants of supported NIC types have been used in testing
diff --git a/drivers/bus/dpaa/Makefile b/drivers/bus/dpaa/Makefile
index cd1093f744..9e1a31bb75 100644
--- a/drivers/bus/dpaa/Makefile
+++ b/drivers/bus/dpaa/Makefile
@@ -18,7 +18,7 @@ CFLAGS += -I$(RTE_BUS_DPAA)/
CFLAGS += -I$(RTE_BUS_DPAA)/include
CFLAGS += -I$(RTE_BUS_DPAA)/base/qbman
CFLAGS += -I$(RTE_SDK)/drivers/common/dpaax
-CFLAGS += -I$(RTE_SDK)/lib/librte_eal/common/include
+CFLAGS += -I$(RTE_SDK)/lib/librte_eal/include
# versioning export map
EXPORT_MAP := rte_bus_dpaa_version.map
diff --git a/drivers/common/mlx5/meson.build b/drivers/common/mlx5/meson.build
index 141739fd6f..f671710714 100644
--- a/drivers/common/mlx5/meson.build
+++ b/drivers/common/mlx5/meson.build
@@ -203,7 +203,7 @@ if dlopen_ibverbs
dlopen_install_dir = [ eal_pmd_path + '-glue' ]
dlopen_includes = [global_inc]
dlopen_includes += include_directories(
- '../../../lib/librte_eal/common/include/generic',
+ '../../../lib/librte_eal/include/generic',
)
shared_lib = shared_library(
dlopen_lib_name,
diff --git a/drivers/crypto/caam_jr/Makefile b/drivers/crypto/caam_jr/Makefile
index 1b1f25a2a2..db17294395 100644
--- a/drivers/crypto/caam_jr/Makefile
+++ b/drivers/crypto/caam_jr/Makefile
@@ -20,7 +20,7 @@ CFLAGS += -I$(RTE_SDK)/drivers/bus/dpaa/include
CFLAGS += -I$(RTE_SDK)/drivers/common/dpaax
CFLAGS += -I$(RTE_SDK)/drivers/common/dpaax/caamflib/
CFLAGS += -I$(RTE_SDK)/drivers/crypto/caam_jr
-CFLAGS += -I$(RTE_SDK)/lib/librte_eal/common/include
+CFLAGS += -I$(RTE_SDK)/lib/librte_eal/include
# versioning export map
EXPORT_MAP := rte_pmd_caam_jr_version.map
diff --git a/drivers/crypto/dpaa_sec/Makefile b/drivers/crypto/dpaa_sec/Makefile
index fbfd775855..13a5ff20cf 100644
--- a/drivers/crypto/dpaa_sec/Makefile
+++ b/drivers/crypto/dpaa_sec/Makefile
@@ -20,7 +20,7 @@ CFLAGS += -I$(RTE_SDK)/drivers/bus/dpaa/base/qbman
CFLAGS += -I$(RTE_SDK)/drivers/common/dpaax
CFLAGS += -I$(RTE_SDK)/drivers/crypto/dpaa_sec/
CFLAGS += -I$(RTE_SDK)/drivers/common/dpaax/caamflib/
-CFLAGS += -I$(RTE_SDK)/lib/librte_eal/common/include
+CFLAGS += -I$(RTE_SDK)/lib/librte_eal/include
LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool -lrte_ring
LDLIBS += -lrte_cryptodev
diff --git a/drivers/event/dpaa/Makefile b/drivers/event/dpaa/Makefile
index 2f53efdf9e..15ffc157f8 100644
--- a/drivers/event/dpaa/Makefile
+++ b/drivers/event/dpaa/Makefile
@@ -20,7 +20,7 @@ CFLAGS += -I$(RTE_SDK)/drivers/bus/dpaa
CFLAGS += -I$(RTE_SDK)/drivers/bus/dpaa/include/
CFLAGS += -I$(RTE_SDK)/drivers/mempool/dpaa
CFLAGS += -I$(RTE_SDK)/drivers/common/dpaax
-CFLAGS += -I$(RTE_SDK)/lib/librte_eal/common/include
+CFLAGS += -I$(RTE_SDK)/lib/librte_eal/include
LDLIBS += -lrte_pmd_dpaa_sec
CFLAGS += -I$(RTE_SDK)/drivers/crypto/dpaa_sec
diff --git a/drivers/net/dpaa/Makefile b/drivers/net/dpaa/Makefile
index 8e049b2a0b..f63c9bf540 100644
--- a/drivers/net/dpaa/Makefile
+++ b/drivers/net/dpaa/Makefile
@@ -21,7 +21,7 @@ CFLAGS += -I$(RTE_SDK)/drivers/bus/dpaa/base/qbman
CFLAGS += -I$(RTE_SDK)/drivers/mempool/dpaa
CFLAGS += -I$(RTE_SDK)/drivers/common/dpaax
CFLAGS += -I$(RTE_SDK)/drivers/event/dpaa
-CFLAGS += -I$(RTE_SDK)/lib/librte_eal/common/include
+CFLAGS += -I$(RTE_SDK)/lib/librte_eal/include
EXPORT_MAP := rte_pmd_dpaa_version.map
diff --git a/kernel/linux/igb_uio/meson.build b/kernel/linux/igb_uio/meson.build
index fac404f078..80540aecee 100644
--- a/kernel/linux/igb_uio/meson.build
+++ b/kernel/linux/igb_uio/meson.build
@@ -12,7 +12,7 @@ custom_target('igb_uio',
'M=' + meson.current_build_dir(),
'src=' + meson.current_source_dir(),
'EXTRA_CFLAGS=-I' + meson.current_source_dir() +
- '/../../../lib/librte_eal/common/include',
+ '/../../../lib/librte_eal/include',
'modules'],
depends: mkfile,
install: true,
diff --git a/kernel/linux/kni/meson.build b/kernel/linux/kni/meson.build
index f93e97fa09..706bea5b7f 100644
--- a/kernel/linux/kni/meson.build
+++ b/kernel/linux/kni/meson.build
@@ -17,7 +17,7 @@ custom_target('rte_kni',
'M=' + meson.current_build_dir(),
'src=' + meson.current_source_dir(),
'MODULE_CFLAGS=-include ' + meson.source_root() + '/config/rte_config.h' +
- ' -I' + meson.source_root() + '/lib/librte_eal/common/include' +
+ ' -I' + meson.source_root() + '/lib/librte_eal/include' +
' -I' + meson.source_root() + '/lib/librte_eal/linux/eal/include' +
' -I' + meson.build_root() +
' -I' + meson.current_source_dir(),
diff --git a/lib/librte_cfgfile/Makefile b/lib/librte_cfgfile/Makefile
index d3b08420ff..7c10a4e56c 100644
--- a/lib/librte_cfgfile/Makefile
+++ b/lib/librte_cfgfile/Makefile
@@ -10,7 +10,7 @@ LIB = librte_cfgfile.a
CFLAGS += -O3
CFLAGS += $(WERROR_FLAGS)
-CFLAGS += -I$(SRCDIR)/../librte_eal/common/include
+CFLAGS += -I$(SRCDIR)/../librte_eal/include
LDLIBS += -lrte_eal
EXPORT_MAP := rte_cfgfile_version.map
diff --git a/lib/librte_eal/Makefile b/lib/librte_eal/Makefile
index 9c383d42bd..ff74935932 100644
--- a/lib/librte_eal/Makefile
+++ b/lib/librte_eal/Makefile
@@ -3,10 +3,10 @@
include $(RTE_SDK)/mk/rte.vars.mk
-DIRS-y += common
+DIRS-y += include
DIRS-$(CONFIG_RTE_EXEC_ENV_LINUX) += linux/eal
-DEPDIRS-linux := common
+DEPDIRS-linux := include
DIRS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += freebsd/eal
-DEPDIRS-freebsd := common
+DEPDIRS-freebsd := include
include $(RTE_SDK)/mk/rte.subdir.mk
diff --git a/lib/librte_eal/common/Makefile b/lib/librte_eal/common/Makefile
deleted file mode 100644
index 6c52f50106..0000000000
--- a/lib/librte_eal/common/Makefile
+++ /dev/null
@@ -1,38 +0,0 @@
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2010-2014 Intel Corporation
-
-include $(RTE_SDK)/mk/rte.vars.mk
-
-INC := rte_branch_prediction.h rte_common.h rte_compat.h
-INC += rte_function_versioning.h
-INC += rte_debug.h rte_eal.h rte_eal_interrupts.h
-INC += rte_errno.h rte_launch.h rte_lcore.h
-INC += rte_log.h rte_memory.h rte_memzone.h
-INC += rte_per_lcore.h rte_random.h
-INC += rte_tailq.h rte_interrupts.h rte_alarm.h
-INC += rte_string_fns.h rte_version.h
-INC += rte_eal_memconfig.h
-INC += rte_hexdump.h rte_devargs.h rte_bus.h rte_dev.h rte_class.h
-INC += rte_option.h
-INC += rte_pci_dev_feature_defs.h rte_pci_dev_features.h
-INC += rte_malloc.h rte_keepalive.h rte_time.h
-INC += rte_service.h rte_service_component.h
-INC += rte_bitmap.h rte_vfio.h rte_hypervisor.h rte_test.h
-INC += rte_reciprocal.h rte_fbarray.h rte_uuid.h
-
-GENERIC_INC := rte_atomic.h rte_byteorder.h rte_cycles.h rte_prefetch.h
-GENERIC_INC += rte_memcpy.h rte_cpuflags.h
-GENERIC_INC += rte_mcslock.h rte_spinlock.h rte_rwlock.h rte_ticketlock.h
-GENERIC_INC += rte_vect.h rte_pause.h rte_io.h
-
-# defined in mk/arch/$(RTE_ARCH)/rte.vars.mk
-ARCH_DIR ?= $(RTE_ARCH)
-ARCH_INC := $(sort $(notdir $(wildcard $(RTE_SDK)/lib/librte_eal/$(ARCH_DIR)/include/*.h)))
-
-SYMLINK-$(CONFIG_RTE_LIBRTE_EAL)-include := $(addprefix include/,$(INC))
-SYMLINK-$(CONFIG_RTE_LIBRTE_EAL)-include += \
- $(addprefix ../$(ARCH_DIR)/include/,$(ARCH_INC))
-SYMLINK-$(CONFIG_RTE_LIBRTE_EAL)-include/generic := \
- $(addprefix include/generic/,$(GENERIC_INC))
-
-include $(RTE_SDK)/mk/rte.install.mk
diff --git a/lib/librte_eal/common/meson.build b/lib/librte_eal/common/meson.build
index 5885441b48..02d9280cc3 100644
--- a/lib/librte_eal/common/meson.build
+++ b/lib/librte_eal/common/meson.build
@@ -1,7 +1,7 @@
# SPDX-License-Identifier: BSD-3-Clause
# Copyright(c) 2017 Intel Corporation
-eal_inc += include_directories('.', 'include')
+includes += include_directories('.')
if is_windows
sources += files(
@@ -16,7 +16,9 @@ if is_windows
'eal_common_thread.c',
'rte_option.c',
)
-else # temporary bad indent
+ subdir_done()
+endif
+
sources += files(
'eal_common_bus.c',
'eal_common_cpuflags.c',
@@ -52,62 +54,3 @@ sources += files(
'rte_reciprocal.c',
'rte_service.c',
)
-endif
-
-common_headers = files(
- 'include/rte_alarm.h',
- 'include/rte_branch_prediction.h',
- 'include/rte_bus.h',
- 'include/rte_bitmap.h',
- 'include/rte_class.h',
- 'include/rte_common.h',
- 'include/rte_compat.h',
- 'include/rte_debug.h',
- 'include/rte_devargs.h',
- 'include/rte_dev.h',
- 'include/rte_eal.h',
- 'include/rte_eal_memconfig.h',
- 'include/rte_eal_interrupts.h',
- 'include/rte_errno.h',
- 'include/rte_fbarray.h',
- 'include/rte_hexdump.h',
- 'include/rte_hypervisor.h',
- 'include/rte_interrupts.h',
- 'include/rte_keepalive.h',
- 'include/rte_launch.h',
- 'include/rte_lcore.h',
- 'include/rte_log.h',
- 'include/rte_malloc.h',
- 'include/rte_memory.h',
- 'include/rte_memzone.h',
- 'include/rte_option.h',
- 'include/rte_pci_dev_feature_defs.h',
- 'include/rte_pci_dev_features.h',
- 'include/rte_per_lcore.h',
- 'include/rte_random.h',
- 'include/rte_reciprocal.h',
- 'include/rte_service.h',
- 'include/rte_service_component.h',
- 'include/rte_string_fns.h',
- 'include/rte_tailq.h',
- 'include/rte_time.h',
- 'include/rte_uuid.h',
- 'include/rte_version.h',
- 'include/rte_vfio.h')
-
-# special case install the generic headers, since they go in a subdir
-generic_headers = files(
- 'include/generic/rte_atomic.h',
- 'include/generic/rte_byteorder.h',
- 'include/generic/rte_cpuflags.h',
- 'include/generic/rte_cycles.h',
- 'include/generic/rte_io.h',
- 'include/generic/rte_mcslock.h',
- 'include/generic/rte_memcpy.h',
- 'include/generic/rte_pause.h',
- 'include/generic/rte_prefetch.h',
- 'include/generic/rte_rwlock.h',
- 'include/generic/rte_spinlock.h',
- 'include/generic/rte_ticketlock.h',
- 'include/generic/rte_vect.h')
-install_headers(generic_headers, subdir: 'generic')
diff --git a/lib/librte_eal/common/rte_service.c b/lib/librte_eal/common/rte_service.c
index b0b78baabd..70d17a5d79 100644
--- a/lib/librte_eal/common/rte_service.c
+++ b/lib/librte_eal/common/rte_service.c
@@ -10,7 +10,7 @@
#include <rte_compat.h>
#include <rte_service.h>
-#include "include/rte_service_component.h"
+#include <rte_service_component.h>
#include <rte_eal.h>
#include <rte_lcore.h>
diff --git a/lib/librte_eal/freebsd/eal/Makefile b/lib/librte_eal/freebsd/eal/Makefile
index e3023f24fa..0c809d9872 100644
--- a/lib/librte_eal/freebsd/eal/Makefile
+++ b/lib/librte_eal/freebsd/eal/Makefile
@@ -12,7 +12,7 @@ VPATH += $(RTE_SDK)/lib/librte_eal/$(ARCH_DIR)
CFLAGS += -DALLOW_EXPERIMENTAL_API
CFLAGS += -I$(SRCDIR)/include
CFLAGS += -I$(RTE_SDK)/lib/librte_eal/common
-CFLAGS += -I$(RTE_SDK)/lib/librte_eal/common/include
+CFLAGS += -I$(RTE_SDK)/lib/librte_eal/include
CFLAGS += $(WERROR_FLAGS) -O3
LDLIBS += -lexecinfo
diff --git a/lib/librte_eal/include/Makefile b/lib/librte_eal/include/Makefile
new file mode 100644
index 0000000000..eb99190d10
--- /dev/null
+++ b/lib/librte_eal/include/Makefile
@@ -0,0 +1,19 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2010-2014 Intel Corporation
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+SYMLINK-$(CONFIG_RTE_LIBRTE_EAL)-include := \
+ $(sort $(notdir \
+ $(wildcard $(RTE_SDK)/lib/librte_eal/include/*.h)))
+
+SYMLINK-$(CONFIG_RTE_LIBRTE_EAL)-include/generic := \
+ $(sort $(addprefix generic/, $(notdir \
+ $(wildcard $(RTE_SDK)/lib/librte_eal/include/generic/*.h))))
+
+ARCH_DIR ?= $(RTE_ARCH)
+SYMLINK-$(CONFIG_RTE_LIBRTE_EAL)-include += \
+ $(sort $(addprefix ../$(ARCH_DIR)/include/, $(notdir \
+ $(wildcard $(RTE_SDK)/lib/librte_eal/$(ARCH_DIR)/include/*.h))))
+
+include $(RTE_SDK)/mk/rte.install.mk
diff --git a/lib/librte_eal/common/include/generic/rte_atomic.h b/lib/librte_eal/include/generic/rte_atomic.h
similarity index 100%
rename from lib/librte_eal/common/include/generic/rte_atomic.h
rename to lib/librte_eal/include/generic/rte_atomic.h
diff --git a/lib/librte_eal/common/include/generic/rte_byteorder.h b/lib/librte_eal/include/generic/rte_byteorder.h
similarity index 100%
rename from lib/librte_eal/common/include/generic/rte_byteorder.h
rename to lib/librte_eal/include/generic/rte_byteorder.h
diff --git a/lib/librte_eal/common/include/generic/rte_cpuflags.h b/lib/librte_eal/include/generic/rte_cpuflags.h
similarity index 100%
rename from lib/librte_eal/common/include/generic/rte_cpuflags.h
rename to lib/librte_eal/include/generic/rte_cpuflags.h
diff --git a/lib/librte_eal/common/include/generic/rte_cycles.h b/lib/librte_eal/include/generic/rte_cycles.h
similarity index 100%
rename from lib/librte_eal/common/include/generic/rte_cycles.h
rename to lib/librte_eal/include/generic/rte_cycles.h
diff --git a/lib/librte_eal/common/include/generic/rte_io.h b/lib/librte_eal/include/generic/rte_io.h
similarity index 100%
rename from lib/librte_eal/common/include/generic/rte_io.h
rename to lib/librte_eal/include/generic/rte_io.h
diff --git a/lib/librte_eal/common/include/generic/rte_mcslock.h b/lib/librte_eal/include/generic/rte_mcslock.h
similarity index 100%
rename from lib/librte_eal/common/include/generic/rte_mcslock.h
rename to lib/librte_eal/include/generic/rte_mcslock.h
diff --git a/lib/librte_eal/common/include/generic/rte_memcpy.h b/lib/librte_eal/include/generic/rte_memcpy.h
similarity index 100%
rename from lib/librte_eal/common/include/generic/rte_memcpy.h
rename to lib/librte_eal/include/generic/rte_memcpy.h
diff --git a/lib/librte_eal/common/include/generic/rte_pause.h b/lib/librte_eal/include/generic/rte_pause.h
similarity index 100%
rename from lib/librte_eal/common/include/generic/rte_pause.h
rename to lib/librte_eal/include/generic/rte_pause.h
diff --git a/lib/librte_eal/common/include/generic/rte_prefetch.h b/lib/librte_eal/include/generic/rte_prefetch.h
similarity index 100%
rename from lib/librte_eal/common/include/generic/rte_prefetch.h
rename to lib/librte_eal/include/generic/rte_prefetch.h
diff --git a/lib/librte_eal/common/include/generic/rte_rwlock.h b/lib/librte_eal/include/generic/rte_rwlock.h
similarity index 100%
rename from lib/librte_eal/common/include/generic/rte_rwlock.h
rename to lib/librte_eal/include/generic/rte_rwlock.h
diff --git a/lib/librte_eal/common/include/generic/rte_spinlock.h b/lib/librte_eal/include/generic/rte_spinlock.h
similarity index 100%
rename from lib/librte_eal/common/include/generic/rte_spinlock.h
rename to lib/librte_eal/include/generic/rte_spinlock.h
diff --git a/lib/librte_eal/common/include/generic/rte_ticketlock.h b/lib/librte_eal/include/generic/rte_ticketlock.h
similarity index 100%
rename from lib/librte_eal/common/include/generic/rte_ticketlock.h
rename to lib/librte_eal/include/generic/rte_ticketlock.h
diff --git a/lib/librte_eal/common/include/generic/rte_vect.h b/lib/librte_eal/include/generic/rte_vect.h
similarity index 100%
rename from lib/librte_eal/common/include/generic/rte_vect.h
rename to lib/librte_eal/include/generic/rte_vect.h
diff --git a/lib/librte_eal/include/meson.build b/lib/librte_eal/include/meson.build
new file mode 100644
index 0000000000..6fd4274941
--- /dev/null
+++ b/lib/librte_eal/include/meson.build
@@ -0,0 +1,64 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2017 Intel Corporation
+
+includes += include_directories('.')
+
+headers += files(
+ 'rte_alarm.h',
+ 'rte_bitmap.h',
+ 'rte_branch_prediction.h',
+ 'rte_bus.h',
+ 'rte_class.h',
+ 'rte_common.h',
+ 'rte_compat.h',
+ 'rte_debug.h',
+ 'rte_dev.h',
+ 'rte_devargs.h',
+ 'rte_eal.h',
+ 'rte_eal_interrupts.h',
+ 'rte_eal_memconfig.h',
+ 'rte_errno.h',
+ 'rte_fbarray.h',
+ 'rte_hexdump.h',
+ 'rte_hypervisor.h',
+ 'rte_interrupts.h',
+ 'rte_keepalive.h',
+ 'rte_launch.h',
+ 'rte_lcore.h',
+ 'rte_log.h',
+ 'rte_malloc.h',
+ 'rte_memory.h',
+ 'rte_memzone.h',
+ 'rte_option.h',
+ 'rte_pci_dev_feature_defs.h',
+ 'rte_pci_dev_features.h',
+ 'rte_per_lcore.h',
+ 'rte_random.h',
+ 'rte_reciprocal.h',
+ 'rte_service.h',
+ 'rte_service_component.h',
+ 'rte_string_fns.h',
+ 'rte_tailq.h',
+ 'rte_time.h',
+ 'rte_uuid.h',
+ 'rte_version.h',
+ 'rte_vfio.h',
+)
+
+# special case install the generic headers, since they go in a subdir
+generic_headers = files(
+ 'generic/rte_atomic.h',
+ 'generic/rte_byteorder.h',
+ 'generic/rte_cpuflags.h',
+ 'generic/rte_cycles.h',
+ 'generic/rte_io.h',
+ 'generic/rte_mcslock.h',
+ 'generic/rte_memcpy.h',
+ 'generic/rte_pause.h',
+ 'generic/rte_prefetch.h',
+ 'generic/rte_rwlock.h',
+ 'generic/rte_spinlock.h',
+ 'generic/rte_ticketlock.h',
+ 'generic/rte_vect.h',
+)
+install_headers(generic_headers, subdir: 'generic')
diff --git a/lib/librte_eal/common/include/rte_alarm.h b/lib/librte_eal/include/rte_alarm.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_alarm.h
rename to lib/librte_eal/include/rte_alarm.h
diff --git a/lib/librte_eal/common/include/rte_bitmap.h b/lib/librte_eal/include/rte_bitmap.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_bitmap.h
rename to lib/librte_eal/include/rte_bitmap.h
diff --git a/lib/librte_eal/common/include/rte_branch_prediction.h b/lib/librte_eal/include/rte_branch_prediction.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_branch_prediction.h
rename to lib/librte_eal/include/rte_branch_prediction.h
diff --git a/lib/librte_eal/common/include/rte_bus.h b/lib/librte_eal/include/rte_bus.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_bus.h
rename to lib/librte_eal/include/rte_bus.h
diff --git a/lib/librte_eal/common/include/rte_class.h b/lib/librte_eal/include/rte_class.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_class.h
rename to lib/librte_eal/include/rte_class.h
diff --git a/lib/librte_eal/common/include/rte_common.h b/lib/librte_eal/include/rte_common.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_common.h
rename to lib/librte_eal/include/rte_common.h
diff --git a/lib/librte_eal/common/include/rte_compat.h b/lib/librte_eal/include/rte_compat.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_compat.h
rename to lib/librte_eal/include/rte_compat.h
diff --git a/lib/librte_eal/common/include/rte_debug.h b/lib/librte_eal/include/rte_debug.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_debug.h
rename to lib/librte_eal/include/rte_debug.h
diff --git a/lib/librte_eal/common/include/rte_dev.h b/lib/librte_eal/include/rte_dev.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_dev.h
rename to lib/librte_eal/include/rte_dev.h
diff --git a/lib/librte_eal/common/include/rte_devargs.h b/lib/librte_eal/include/rte_devargs.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_devargs.h
rename to lib/librte_eal/include/rte_devargs.h
diff --git a/lib/librte_eal/common/include/rte_eal.h b/lib/librte_eal/include/rte_eal.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_eal.h
rename to lib/librte_eal/include/rte_eal.h
diff --git a/lib/librte_eal/common/include/rte_eal_interrupts.h b/lib/librte_eal/include/rte_eal_interrupts.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_eal_interrupts.h
rename to lib/librte_eal/include/rte_eal_interrupts.h
diff --git a/lib/librte_eal/common/include/rte_eal_memconfig.h b/lib/librte_eal/include/rte_eal_memconfig.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_eal_memconfig.h
rename to lib/librte_eal/include/rte_eal_memconfig.h
diff --git a/lib/librte_eal/common/include/rte_errno.h b/lib/librte_eal/include/rte_errno.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_errno.h
rename to lib/librte_eal/include/rte_errno.h
diff --git a/lib/librte_eal/common/include/rte_fbarray.h b/lib/librte_eal/include/rte_fbarray.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_fbarray.h
rename to lib/librte_eal/include/rte_fbarray.h
diff --git a/lib/librte_eal/common/include/rte_function_versioning.h b/lib/librte_eal/include/rte_function_versioning.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_function_versioning.h
rename to lib/librte_eal/include/rte_function_versioning.h
diff --git a/lib/librte_eal/common/include/rte_hexdump.h b/lib/librte_eal/include/rte_hexdump.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_hexdump.h
rename to lib/librte_eal/include/rte_hexdump.h
diff --git a/lib/librte_eal/common/include/rte_hypervisor.h b/lib/librte_eal/include/rte_hypervisor.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_hypervisor.h
rename to lib/librte_eal/include/rte_hypervisor.h
diff --git a/lib/librte_eal/common/include/rte_interrupts.h b/lib/librte_eal/include/rte_interrupts.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_interrupts.h
rename to lib/librte_eal/include/rte_interrupts.h
diff --git a/lib/librte_eal/common/include/rte_keepalive.h b/lib/librte_eal/include/rte_keepalive.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_keepalive.h
rename to lib/librte_eal/include/rte_keepalive.h
diff --git a/lib/librte_eal/common/include/rte_launch.h b/lib/librte_eal/include/rte_launch.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_launch.h
rename to lib/librte_eal/include/rte_launch.h
diff --git a/lib/librte_eal/common/include/rte_lcore.h b/lib/librte_eal/include/rte_lcore.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_lcore.h
rename to lib/librte_eal/include/rte_lcore.h
diff --git a/lib/librte_eal/common/include/rte_log.h b/lib/librte_eal/include/rte_log.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_log.h
rename to lib/librte_eal/include/rte_log.h
diff --git a/lib/librte_eal/common/include/rte_malloc.h b/lib/librte_eal/include/rte_malloc.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_malloc.h
rename to lib/librte_eal/include/rte_malloc.h
diff --git a/lib/librte_eal/common/include/rte_memory.h b/lib/librte_eal/include/rte_memory.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_memory.h
rename to lib/librte_eal/include/rte_memory.h
diff --git a/lib/librte_eal/common/include/rte_memzone.h b/lib/librte_eal/include/rte_memzone.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_memzone.h
rename to lib/librte_eal/include/rte_memzone.h
diff --git a/lib/librte_eal/common/include/rte_option.h b/lib/librte_eal/include/rte_option.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_option.h
rename to lib/librte_eal/include/rte_option.h
diff --git a/lib/librte_eal/common/include/rte_pci_dev_feature_defs.h b/lib/librte_eal/include/rte_pci_dev_feature_defs.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_pci_dev_feature_defs.h
rename to lib/librte_eal/include/rte_pci_dev_feature_defs.h
diff --git a/lib/librte_eal/common/include/rte_pci_dev_features.h b/lib/librte_eal/include/rte_pci_dev_features.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_pci_dev_features.h
rename to lib/librte_eal/include/rte_pci_dev_features.h
diff --git a/lib/librte_eal/common/include/rte_per_lcore.h b/lib/librte_eal/include/rte_per_lcore.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_per_lcore.h
rename to lib/librte_eal/include/rte_per_lcore.h
diff --git a/lib/librte_eal/common/include/rte_random.h b/lib/librte_eal/include/rte_random.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_random.h
rename to lib/librte_eal/include/rte_random.h
diff --git a/lib/librte_eal/common/include/rte_reciprocal.h b/lib/librte_eal/include/rte_reciprocal.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_reciprocal.h
rename to lib/librte_eal/include/rte_reciprocal.h
diff --git a/lib/librte_eal/common/include/rte_service.h b/lib/librte_eal/include/rte_service.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_service.h
rename to lib/librte_eal/include/rte_service.h
diff --git a/lib/librte_eal/common/include/rte_service_component.h b/lib/librte_eal/include/rte_service_component.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_service_component.h
rename to lib/librte_eal/include/rte_service_component.h
diff --git a/lib/librte_eal/common/include/rte_string_fns.h b/lib/librte_eal/include/rte_string_fns.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_string_fns.h
rename to lib/librte_eal/include/rte_string_fns.h
diff --git a/lib/librte_eal/common/include/rte_tailq.h b/lib/librte_eal/include/rte_tailq.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_tailq.h
rename to lib/librte_eal/include/rte_tailq.h
diff --git a/lib/librte_eal/common/include/rte_test.h b/lib/librte_eal/include/rte_test.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_test.h
rename to lib/librte_eal/include/rte_test.h
diff --git a/lib/librte_eal/common/include/rte_time.h b/lib/librte_eal/include/rte_time.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_time.h
rename to lib/librte_eal/include/rte_time.h
diff --git a/lib/librte_eal/common/include/rte_uuid.h b/lib/librte_eal/include/rte_uuid.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_uuid.h
rename to lib/librte_eal/include/rte_uuid.h
diff --git a/lib/librte_eal/common/include/rte_version.h b/lib/librte_eal/include/rte_version.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_version.h
rename to lib/librte_eal/include/rte_version.h
diff --git a/lib/librte_eal/common/include/rte_vfio.h b/lib/librte_eal/include/rte_vfio.h
similarity index 100%
rename from lib/librte_eal/common/include/rte_vfio.h
rename to lib/librte_eal/include/rte_vfio.h
diff --git a/lib/librte_eal/linux/eal/Makefile b/lib/librte_eal/linux/eal/Makefile
index 177b7c45da..692fec2695 100644
--- a/lib/librte_eal/linux/eal/Makefile
+++ b/lib/librte_eal/linux/eal/Makefile
@@ -15,7 +15,7 @@ VPATH += $(RTE_SDK)/lib/librte_eal/common
CFLAGS += -DALLOW_EXPERIMENTAL_API
CFLAGS += -I$(SRCDIR)/include
CFLAGS += -I$(RTE_SDK)/lib/librte_eal/common
-CFLAGS += -I$(RTE_SDK)/lib/librte_eal/common/include
+CFLAGS += -I$(RTE_SDK)/lib/librte_eal/include
CFLAGS += $(WERROR_FLAGS) -O3
LDLIBS += -ldl
diff --git a/lib/librte_eal/meson.build b/lib/librte_eal/meson.build
index 1fc532139b..a433f46ade 100644
--- a/lib/librte_eal/meson.build
+++ b/lib/librte_eal/meson.build
@@ -5,6 +5,8 @@
# have a straight list of headers and source files.
# Initially pull in common settings
eal_inc = [global_inc]
+subdir('include')
+
subdir('common')
# Now do OS/exec-env specific settings, including building kernel modules
@@ -27,5 +29,5 @@ if cc.has_header('getopt.h')
endif
sources += env_sources
objs = env_objs
-headers = common_headers + env_headers
+headers = env_headers
includes += eal_inc
diff --git a/lib/librte_kvargs/Makefile b/lib/librte_kvargs/Makefile
index 419be8bd7c..24b1c3c5b9 100644
--- a/lib/librte_kvargs/Makefile
+++ b/lib/librte_kvargs/Makefile
@@ -7,7 +7,7 @@ include $(RTE_SDK)/mk/rte.vars.mk
LIB = librte_kvargs.a
CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR) -O3
-CFLAGS += -I$(RTE_SDK)/lib/librte_eal/common/include
+CFLAGS += -I$(RTE_SDK)/lib/librte_eal/include
EXPORT_MAP := rte_kvargs_version.map
diff --git a/meson.build b/meson.build
index b7ae9c8d9a..ace4a0b8bf 100644
--- a/meson.build
+++ b/meson.build
@@ -28,7 +28,7 @@ abi_version_file = files('ABI_VERSION')
# able to be included in any file. We also store a global array of include dirs
# for passing to pmdinfogen scripts
global_inc = include_directories('.', 'config',
- 'lib/librte_eal/common/include',
+ 'lib/librte_eal/include',
'lib/librte_eal/@0@/eal/include'.format(host_machine.system()),
)
subdir('config')
--
2.25.1
^ permalink raw reply [relevance 3%]
* [dpdk-dev] [PATCH v6 2/7] net/ice: add the DCF hardware initialization
@ 2020-03-27 2:56 1% ` Haiyue Wang
0 siblings, 0 replies; 200+ results
From: Haiyue Wang @ 2020-03-27 2:56 UTC (permalink / raw)
To: dev, xiaolong.ye, qi.z.zhang, jingjing.wu, qiming.yang, beilei.xing
Cc: wei.zhao1, Haiyue Wang
Introduce the DCF (Device Config Function) feature in the ice PMD, it
works as a standalone PMD which doesn't handle the packet Rx/Tx related
things. Its hardware entity is the VF.
Add the basic DCF hardware initialization, this is specified by devarg
'cap=dcf'.
Signed-off-by: Haiyue Wang <haiyue.wang@intel.com>
Acked-by: Qi Zhang <qi.z.zhang@intel.com>
---
doc/guides/nics/ice.rst | 47 +++
doc/guides/nics/img/ice_dcf.png | Bin 0 -> 39168 bytes
doc/guides/rel_notes/release_20_05.rst | 5 +
drivers/common/Makefile | 1 +
drivers/net/ice/Makefile | 5 +
drivers/net/ice/ice_dcf.c | 474 +++++++++++++++++++++++++
drivers/net/ice/ice_dcf.h | 52 +++
drivers/net/ice/ice_dcf_ethdev.c | 317 +++++++++++++++++
drivers/net/ice/ice_dcf_ethdev.h | 24 ++
drivers/net/ice/meson.build | 7 +-
mk/rte.app.mk | 1 +
11 files changed, 931 insertions(+), 2 deletions(-)
create mode 100644 doc/guides/nics/img/ice_dcf.png
create mode 100644 drivers/net/ice/ice_dcf.c
create mode 100644 drivers/net/ice/ice_dcf.h
create mode 100644 drivers/net/ice/ice_dcf_ethdev.c
create mode 100644 drivers/net/ice/ice_dcf_ethdev.h
diff --git a/doc/guides/nics/ice.rst b/doc/guides/nics/ice.rst
index 8af32dabf..2639ae239 100644
--- a/doc/guides/nics/ice.rst
+++ b/doc/guides/nics/ice.rst
@@ -240,6 +240,53 @@ report a MDD event and drop the packets.
The APPs based on DPDK should avoid providing such packets.
+Device Config Function (DCF)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This section demonstrates ICE DCF PMD, which shares the core module with ICE
+PMD and iAVF PMD.
+
+A DCF (Device Config Function) PMD bounds to the device's trusted VF with ID 0,
+it can act as a sole controlling entity to exercise advance functionality (such
+as switch, ACL) for the rest VFs.
+
+The DCF PMD needs to advertise and acquire DCF capability which allows DCF to
+send AdminQ commands that it would like to execute over to the PF and receive
+responses for the same from PF.
+
+.. _figure_ice_dcf:
+
+.. figure:: img/ice_dcf.*
+
+ DCF Communication flow.
+
+#. Create the VFs::
+
+ echo 4 > /sys/bus/pci/devices/0000\:18\:00.0/sriov_numvfs
+
+#. Enable the VF0 trust on::
+
+ ip link set dev enp24s0f0 vf 0 trust on
+
+#. Bind the VF0, and run testpmd with 'cap=dcf' devarg::
+
+ testpmd -l 22-25 -n 4 -w 18:01.0,cap=dcf -- -i
+
+#. Monitor the VF2 interface network traffic::
+
+ tcpdump -e -nn -i enp24s1f2
+
+#. Create one flow to redirect the traffic to VF2 by DCF::
+
+ flow create 0 priority 0 ingress pattern eth / ipv4 src is 192.168.0.2 \
+ dst is 192.168.0.3 / end actions vf id 2 / end
+
+#. Send the packet, and it should be displayed on tcpdump::
+
+ sendp(Ether(src='3c:fd:fe:aa:bb:78', dst='00:00:00:01:02:03')/IP(src=' \
+ 192.168.0.2', dst="192.168.0.3")/TCP(flags='S')/Raw(load='XXXXXXXXXX'), \
+ iface="enp24s0f0", count=10)
+
Sample Application Notes
------------------------
diff --git a/doc/guides/nics/img/ice_dcf.png b/doc/guides/nics/img/ice_dcf.png
new file mode 100644
index 0000000000000000000000000000000000000000..540823c4a05e9ea9d558d233fea77016d1ae5df3
GIT binary patch
literal 39168
zcmZ6ybzD?k*FH`NDBT?jlF}e8B?Hn74BeolfOLbDz);d%Lw70NNOyNPNO%7Z_x(KY
z`+k3a@X<4KW}m&+Uh7)dy4F6yN(xdKsHCWHaBvvXpCpvw;1KwLKlqmjz$g7i%71`=
z;O&*A#NkSZ$ajH@7p7wJVsLO35oq^@NWk^0Z=W>n;oxu`p8w#NOsHJo;1Xh`CBz^u
zdIzZ}I(W`x-6a%La{^jclNOU&`_)rTS4-}zY76d1P4w1CVq)-OmrKC^5#ZY~-}pzu
zy$L?@i@zdO6jl2qp0>O_?^V0~==_TKu;0E)6AT_jK}Ys$GuU9xpl6NmUq4AAAS8rZ
z5<h>~BQL7h+1bg!^v{JkDk^GFZXt;WZK)wU@I4Lh>Nma-8Py=vX58uL5BwsNv0+TC
ztdd|ub|{f>I#qBR_r=Ecc2+(+F$m*+POs4gtgWq$;%5$Zz1W+)IolS4AR!?E!%MzF
zY^hc+(<6BKV$B>q*BptIQ5gaPfd*IJosgp=XT25!8zwm^L5M@!Q9uFabHF7#DiuCK
zt~qeo#^9j@Nyw!ThkWHBd%m>z=MtX)IqO&q%qUT-Q<|7KJNMhNs;a6pC&Kl7XSB)v
z3KtLW;^N}!aM9A*nl<f&LypeAH;UH%@~=<W?^;Mic5d!cle_a$qiaVX7S(%;iC{c>
zPEJmL@k|#dzb#-e#WW$n3TwM=5&|%zK#w5Rj~l++vL_;_xx2ZfV1g5o&-OTVud|dg
zY)p8a+CN(%s#Jb5B@9dFyIC8&!|9!kw<)+dnAUfkUON7RZe*>j5V@t@LFIjKzBWN}
zl6bSB?AXD@u-6mJ^$s$%>{fg;H**<Wy_z$R_0fFyxQl_NC8(r`t&yYbC$Wq!ZhWiR
zaQO>2)_CPI7Ur}ViP+8SHXAZ(qCX-8Pgw>@)`d8#Ub}%C2oieVUWG5$Z;wDPw0oh!
z(MS2V(Mt&eBO3Siyc1oDi4_JnN!?`_!F7~4=LJ<;7i4@c?GuTsp&oXOubJN~j*nRE
zr*(v=l`h&IUu)Y-G0<(qS?nZJwYZ$k+*tNcTf2SvjF$-h6dEDNR%>@OyFp@~wx8sm
zQ@KkHCa?M<DDa*q5JkVi|7}X6ooQlUmnxZR*#{T+GPIBvv|B=xJdNM>M<?%wx7c)l
zdn}q*;Wg|Gzyh^yRZNGE1Xt2+qw~^RE=E{lE|-=pha<-cjHj$U3eSXtS-y9ciA;Qt
zg#X2CIXogbtk+gV#pepYbkK~qZROc+2-nD3e6SS4d#n~;vgmMgi02}r<bB>}Fg(g-
zTDmx2YrO0kRzm_}D5S%J%DnQ0DdV>iV*NTt#-ch%|0xYIm~S%Q_BwvyfdQLqe%iUU
za<4uRAr<XM?#|y&3?~<aXTr-Hgaz#tT;9D;cK$2uv9~aYB8Zi$G+Wuxv-FN9+5PYR
zh4qI=trjcZ;=wL@A=W&9;v>?i^7Yn}O(rs$G1Q(|Z2z1sn<Yns)V;&#c$@TE2Q)Ru
z)6)`Bt!gQksJ*lfL=-F}Q=R&|&-+U>DkcK~7vFVi4u!dmmta<R;msWO$55HcDm$Mq
zYzuC++dJmp5`L=_vy^#UbXhfEraEBcG`ZP)uwPnpl~qe?xmmf+NiO2ax1Om+(c=}a
z!GBS0w-`sWU7FHRP;K|H)xhT98<o%yk#hLBt#8R=)!=YcP~pc5m_`)LD5eB~iYqs_
zj?tLy72jM<E{lrbr1en>*bVr_*W;KI;3E7^sdYFjoC{%b$NVtKVTT^7+2BYNw7pYj
zi4R5^w->W9pqU==f1#Eb;6C!>z+DXeb>TXZwZp;G`tJ%B5BYEu=0)FDN%x4fNM_s}
z&3d^+eysZRo^x@R*na2q(uHz@x%b$Uaf>KVBm(XH446M${l^6#mQ=@EaW_yr=YvCd
z+ccCShFQ?$1}?+o10_*^aev+*-pN^+qy-`EqoDYuWpSd~d>GvmJf>Q%JrCl>)WF6k
zmNOMMMp+hgneL-`A#pMMbJc=vS<;lS;}#5&f&14fUK<#DZv350Jd6LHkN$NrFqGJ8
zlLQ^XPt!kNF7<(R!K>C~vVL6sXh^GeVu@|f9?Xx@{W00`WXRZv(|^df><;rf5PNdx
zBpMO7vZw8Z5tyP#0>Vqx1X*<dv&0Kc;4ZffjQvQ21Vb2S)~HS*e}&q3vP0X5Ck>{p
zL|PgxQ2d3lia{mhxO*i0u!<pNN+=Dr*xkF}rL@3)H_B6SSLCd3U(-4g(^%Ggi&~71
zLC)Opd9psBf9m^tF-R2=i~Y+Wx~{DJ7&!b`+!vr6(LbBVOG$#4e;21&m~FS))*G-=
zPa}wqkl#vdk|f?wwVSv9(#)clt1G+|;%v%L28)#;+6VXk{7iUG+jNKVTH?KH*Mg`P
z4}U@{EuUAnkMX-Hs~^?DoM;li?~cv8zY`mcjZ7JLTz<$aU4RE4@Y}g{&JAw#$bdHS
zIsj~7y#89(M|b^nq}$-hTBu4^z2sUdUo4>v8BBVg%wyU%_MXf@bERkY-D?x<$;O<b
zpvBDRefW<DBArvMWHxmVUG9o}n*WowASDGwz0FLH(Dzku6^(`Vv9TH-Or_3WQek9&
zp{?wvKs0uy5PokudTJp-5ZP7duzj+zYp+MDA<51+GTg}}+XPcyOCY#AF=5lm{bEKD
zuYwz`>Y=}WkR(4(Y2H+%2SHKL8L4sdgQdlyUB8->Fe_=z_$V35@6;>_^Sx+A)!#{y
z%vNutx>hi*td+hI$B#YrfhqfD<l~3DlzhC=#`503zteBJi#)&=g+hzrVxg$DkD{I@
zg^onlA0{~L`RpH-+6gSVF_?>R2n@E#DgbDpEt*<Z@je@xlV4BeZ%k2#RC;@KHPXI=
zTv4h`bGcu3y~M9wU$HGWe-9h0vD0j^8a@f(O+_$Xo~<oWUu<$5exr+)S7gD4{3wVn
zgrav3%g?CuuSSpayqwxe{k^|Fjm)EGSivv!zjH;?(yL3=fBQ=%uWad!A{EMEQA^#0
z1YSLDC4#7>-9zS&zAcI<8lQ|mDf7d6im6Q*Pa;o%x1R@B!{cXusK5DSe-NiT0+q(3
zJAul<M~5YH9~^lh-eTUP{6+YH%@Tq2W2W5mXeE;-FoLq>=#Q(^6LkG!bA|cqcF*pi
zs1U^jfwNgMV;A~2jK#)8*61YsCi9QR)xxWa4oeYf>?DNCwZ`dlvWs4bPp=e=C;^0@
zKx^mvbur_;)>z-nUx7=5Qf6*XICBRQBO-{5^FAFbsff<KpkO!a8>m~+y7p>YE4&LA
zuaOFs0yDxhFwwq*qMEm7{5K*3xDA#58|>mq?5K=|*ax&{U5}%6AKyg@FV2-FFZMsS
zrHF6K$F#kK%8>ZN{<(`uWDrDi_jGHzJr(KJ|1yPYc)l)sxy8H2+vzKLzZc?{M!E7v
zF8j7?Yx*gy+=(y$qRI15nG2g2=Obg4BUlV=;xP}#z2<_JKj<^R-sjvH4CwW2sL>TH
zBFbS!1ZIVePcY_hP7Oo2*3+(G-We}hkFa4=pbOcJV<M%e|C+Y!zUqG{n|J=*>)qtn
zuvYEZ1ox-boipvgxq`t_VP5lxlooZlyR(J3H(<AZF?Y%7yd%9i@7`?6^1pYM<8}vA
z(fcoCgsMf!oK2VWC%GQAgevYXL7o?Eii4GU?~V&J?0;;|X}e4MzW2UeoiA;v_Kja^
zfO*44nT%$s<HJ_iHD0JS+i49q+(`zs=pvVW8i+;xuYtKK!0_&IS4RuYlSNh0LB0N0
z@7M}4b%hGL51O*{(vrSeZ6D8OHCSs;@XmhzeVhf(Slk`m&D&IY@;78t2t{dNPCng^
zl3t`DCq?X(3hcH>ORIJ6{H~eR;2X%NKo&enSZ7_?BtrJ5+S7I!1*N++VP8<HIBoPS
zs4s)hk6=@{5NU`^H+?`wcP3Bej!F{iUVeW-Vmz5k<8mL=To^D!#+Hw%%ierPJ88H_
zPv82kH0g1Ai1s0S9toBo^Lh+BR#Ex!G~`x30&(~+!!jfFHn1X}Pg@a3JmuSLTQ#Qb
zpQ{H2%o6CZ4m_;3gUII{#r0Kl7wZMxrF?pq^i$O~lx>HQg7%WjlKr9^Cy2f+@^BqY
zbx<#R&U+j$A5c+HBI@EZBVkOHH&Z}NE>JzIVq7w@Q)x#`<VnX!98`B2r8m%6Vo+<#
z6|brd_*__wX0{t1GqA?_MGCa@sMxH?;yyPPF1TB!b^Hhn>~1g2KzO8);$a8uAH*&3
zg)D9M>q1*@_xCVo#(yS{1NMP%E6V?!Zdx45c`#Zo`Hos^`dDktCFvy1sVN@{zyDdY
z>><Z7B&4vz?kkf60-^cx)}7#;WUQ;|Ust^A{Cc6ps{`J_Ho33e7$a~vXbL1#t`aT!
z2UJ5oWg&|^mMJOh+0J)(WRFNE*ofb`(PH0t*QCEMv<$k$g`L*rpQiFCT-S{2>uUSm
zw<=5E{Hpn^91&*!WTH|0=HF?*gcfcA(20pesM@XX<znYR-nMzrF*R^)xxTPuJ-mhU
z$llE1dA!{69o)n5LZtfsGN@04B+k+hUE_rgg5;a&za<srs{J-<3Qy6+aIcv(Vh0~)
zkKgAXo_BfQU6|#09=7DXY~^inHrqTNHErF7_5Idg-n+BOIH|92e|zJ_nvaQy)}E~B
z(h@~J+X?{0YlgxkKbSHfz6PX3+q1^PZ8&z^?eZ~;#lzEVFypvJwzmp*NjFhor7Ydu
z;qg_Ml8d`7>U`!J{-r<=sz9Ex<rB%wndw8n3T-!>HqoW|-MF)wvnft=dI;3KrdhMf
zMAysew~LW;pSGKW3p2yd|KC%+rqf%cE><%ubiE&Jx1%0MZ=Vj7xf*$_KBZqWak)pv
z4j1_4XAlThcpZNinfEz(+YMJ<jd{>q^T3Cm4}mOujeA_~ly2A-OCo@BUWkDw$?;>`
zNAbhK1bj9N)%iBl%U+H3#(W(f7`9(~6|64(y#)u!7m=9$ls^wV6HaD*K=}$*?`*(r
z%PAmMv7h_OPY`2waY<I!<1y{9lkvg*S2VsV;C^W>H9yql*ma~+oMt}#AG;;>^x|`8
z#R49-yL6HlpGSwi9Z64GB2a6w($eJZ>gJ#mxlbpK{v}O&29pecI1p8Qp<h1SHvO4N
z_+F)%xO0841x)wd$&8Vw&1HJ3w?MV^H)!Ha$X1O>IO*|zpZSM6>*QdWiv2AWGUOWT
zi-U3Fb<QX6QPsrlDFCm9oEyVYRmw8*Eij=nt7ZU<rce$)Tpe9zp*)({+wiPnEysrt
z%KsJKDNEN+O_ku)jJ2#;k}_W1r057%=`a*slt2m_4Ow4Dxwv;-?t_o4?>G2;iM1t1
zHTq@j=TP}0C)-I{SAz4qYL-K^qweE9?(nlNI~&!^GA8sit}QIH$3qI@ShGe{waqKL
z$gP6l!{HMFCb=eGSRp?HQ$W^k5NfTr!FhVBHe2dKhrT@y7@j_LUMO2X!U<qQcl9^w
zAMYGS1h^`W*Fliy$>29w+j$DuOrED%X?afH7nQAm*uGMru4R&AA=<D2a2dhUa`0r@
zkTJbBK6OSuwLJ5|P9q(YMcV`M$U<Wo(*mXYZ8CH?GyY$Y{x{kJ@CmLnT3u;ei(W?I
zMzAO`+bv3<e%#6GvtLFB?dSqKV^E+x%tEbya&0sX)%=q7bl~KNG#gIFFumkEV<7hU
zq)iJq1xI5@jQ(S?U1mj9Smdun_!|Kx7{jS*z}2gF{sN8ft`X>R+<Q>}*9|j&!o)35
z{$bev7y}Xb>i>Ao|77I<=gZUAg;co|EjBB;+t~N>_f;{P9o0(>CsVsQIY)8(M+{Z0
zX0uI|msFKEtT~ttnsz7Q&HG#`N0?_r9`7GV%pEN$8bm$wuu-KbwUr2qDS>DJ*c!rf
zo^{1mo!8969ZL|;%-@E-i_s)2-OJ(P&$+jIm%hVOr64opW`*}ty|pQ4f6VyiOYbe#
zc=({mBPV?e10QS2MJS$rn8*$>vA(j}V^5F$v@VUb2^BbA=Q!jU&j8~69bkn$Rr&R-
zO%6w?WLzx_VTtLj2CV!KGIodUcx>y$1FafezYvGt?k~pa`aUwZ1UZA|ee$tkNv=Cc
z{JGZu?=XZZBj@Ofc4+btE(a9d-5DgAA*&B(**VS-x*~(_)GHAM5wR(${3Wbl#jKod
z%!O|u`G9Q+|2vP8d@VU)x3xNg57CaY!$@wqG0aZHEbB+A(>?I5v+>m<Uz;jVsLJ*J
zW~+@y6zO<$tl?|lNJMu4hC=%f7vE5WJWMT(IHLy3^({EaKP?YB7ke1l@pqa|SOJ6V
zP+$pl?BYl#$__WuF_p)}{q`1!uG%lf;Sa^0L}IxMpnos@Sn`)SbPo_}$YtIzMIKdC
zPs>vd6ZxC5y|K}HBV94S<<b$`HYa;Z!TZt^q6v86x58P{wkI2UoRXG>hZ~Pm)?ev(
zXVwAnw+Kn2g}4PvdewlLVqV+5V`7&VqCwG_bUp~XJRAD!ZJ?#2yhSQ*$+5}6kL+Ty
z7n`c?qP19mF#uIzF`@8+FFFm+?x5Zv7Kf||U6sF$ie@YBg?|s!O5Yd)@?QvRXPw9W
zzAFH=P#$oWOh*kT(7*Ed?XvkR4WNH}In6(}n`hUC2FjAX9nI&<5ll~Xlj0N(M~gM9
z7W;3xm?1qmFD1O@Yps*F!w=>vJEt4sE)}`;KqRl-pB?W8e==fRQHbNR&yXn9-CPzT
zX|2+4M8JL)^24fW-lP2u?B&<ST&l#FFZ)D-uFKJC<Kab1+ygzELfogXQQ&I#70tnM
zy54n{`>H0Fw|hk<)fQE<3!TG@ud#0no9o!?oNKC!@}4eEg2_~&7N3qT-Qsh9Iv>z{
zm)0<yr)u#7_;6@3bvjJz+ggqdL`H_^?hjYdPzWpB%A2F0&w3<JB_EHSqr-ppKXP7$
zrWt>;AauX_U`)J@pMV`1Q@7$z#}5b8W-NWwOd}B|VY?aWP5mhB2UD^`MzduiO()As
zi*rJP!fC{s1^J?Ot$QRN*3s2*LB2-Q@>P<~K2Ce@@ChWHB0qvX-#@OXT6O<z6!)t8
zuHHuvKO2MQPf~XiwS5gw^y}|f;*G`8qVqEEj|<{jf0-d9z;+?Hsd8<$bJw!TNAoog
z!lOHQ@wE`4mlgY=k-F55@ok?~VYgZ9ii?83DUY{@BQPW0Kt&6>3A?52TkfiF>Svqg
z?k5mH0g<X3N|2TzLCY-?>EQvDCi4-yJ&q_ZOB?^Ly!aEOJF0~0B(D@5kKF=wuhD|l
zU+C#Q8yn1*!P{yZ%n10gV=xIA)5(n8w~GulmX=Xr!2;1WoHvT$;rkC#$)4|!vFT+X
zw1r+D=B(0|$vmpR_ND*2G9vq;Sw=C};CZDWLZX{Kby1`C&o%_Y=2S+BH<R2XLYGSE
z_=^|+ssvIdqk*Vve*U?%e-cz*yJPIOZgw1c$H2N3J9pcuYz={bj%-@o*jDZLI~QwC
zBAzvlhUpVeBJ9G0MuPyI?ZP@V^Kz#H7sx;!p1xk4&&-S`|3LqHbLRJvK%QV=Qkb{S
zvFK<?FJ<lVt5;392rQqy5=su0>1Ji8iLbg#!*Kvdbu_1a%bzZ^ESUW(q;MjVZ0*f;
z!p?7bS9101myQ#T`rwFer2+tPS}~s>qQQ4@qljMnW0?dl0Z$d)m31QSYb~j2ibvAI
z#x<niUH!DZdzYp5SBs_*horD3f0JK!rz1mnfewB4Gb?v?I8aoYjvl{!QI?LHC9;+~
zoO4O`Ss;?Z^Aldl-U@c;ZGbzXt2gmFZFi(IS+89!*Lk<F@bP}#ZyFV(^jK1WN90`~
zt=Tthn9XCnm;+Qjz8G0Z>*7Ne;3SWus}*&Z1bGmhWqu<dAv#}%Gd(=ndllitwoRIo
z=Y;_o^88@Qse8?9I6Pwi)bibdeYC$Yfnv85*zhy4D|RdlSSDYL0)*rCoha8?w<DT&
z1A2C)NtZt0&W>)}@#IQP`74&+aOIqCHv=OPy;3=g?ZXcQU_r3<9`^sxN#N(Sl-g5+
zhm%cbigdiDjZN%d1~Qg0ODQFKeyoJR`kd#k|IXA-l{D|)&PM<Niw)F*#IwjCReL1}
zYT#Hf>PQn`j?{K~c<F;FKn7kjrJhC8ZhL_tqV0FFyv`9W-^|cIJ!3w5CB}a^^%s2A
z!kZ-9CkOI{{doQP-#T1pWlE_zQTCS9g~NcX{+SG4b?_OC`xuy3j+4X;3IK1etFU|_
zwU)#L2MPfH`45d!QU<#gu>th)=$7BR;X|@wqUv5<r@H;<-%;IVy_9xGyk!FH?40`i
zh&xk1X(J4^;^f7B!0d@?mW)-O{}o#QN$MY^K{-1A05(IRQ<dU{w)@%ijcKn;(QdbU
zS>2MQW2XvUa)ii)_u`-rnYHKB?wPOvE8o>{*St4#(@XuD6t}M<Kjr}^p&@SEy-a^s
zicMrbiSmDPIzRp-EIO&}#9Z*`{%WHma(;pwaXmQj(l?ztiZhz)?1y!o?2<o%oYP^B
z6=Bgzb0(P_rjGYJ4x0Ex{g{L_v_D*Sp*kJ(f4vqey-LLa(*Ttf96(dRO5oVI*kvHS
z)Yrni5#mxk52&L1ST~|zsG-JRTfTf6_;CA&S~zwWFHy*R*chhF3+&o6)p-`N@F9?l
zoSB%vtJ+66h^$SOipE>?*&Oc(2)=DpfDruZ?u)O3FB}0j^bIi0vt%bB26EIJ=b_`j
zOA1%H`Vt?X?rOD8woVkiMxan81My~V*uaT99~-NAswewgQT>020z-bjLUXyB$=(5F
z9G~s*y0#nZkD*LO`6f}2YVh8EaNq&r+Ak|aWUE>bp(T*mSAqFG%OgQ+P>a@EZ6~h^
z>o?rr#(V;ElS_BiX$tkX(wtspX6DCW{Fg{%=6j`jPqn|>OY@nhl4Zk*<q)K-4WtE2
zT=Xw+#p}(Eb7^W#JwAPHY`QOYdJj{syXN6!1lUnlx{1dr@tqdHzW4wjf0jn517Qdg
z=!?1^@thZ2vBvf)uiJyS<5q(c)qhhM#%1R}>Uml%abT1~`U|=XaVE#c%g7PeQvGo0
z)-=6kkfAb6jV(L}fX;C^-057|Q>o|d{FD#WUvE&jnVy&A0k!CR%(MCH2e^R#JaS_N
znggoLNCg_gLo2g1SejKFNA;^kwYJ$B9v)jOBm2@Ez1!lKMa;9uhM-!^XPe#-5Da_T
z&m`v{tLN!0^5385dIR!-r_<Mz`Kp^bCju;?82igI@{uap{k-ljoM?_CwNSE!ijVV0
zz3^eE&kIb4-5Snu?08)+a0#C*>;Y-^pa!G+!0pq(t;w3<q&Bj#lW>Mmesfu0*hRc!
z9+d~O{*%Db-?Y1~yh*E<m_qESjvJr`{%e1IS9+h$W$(57K`FKllA9RK+##=SwgngC
zEztt?2qCD<Mnjl+!{OwtG*ENSHW;<0Z9W(_c)JW{tnWY8)wyTyk2BfSs+*cg7X2EQ
z=d9!RQ!e9TV3MPe5(d$xXXIzg%MsRqLI|_E<WHe8nFI9>+r4Eij&q(H_&ItXP8kY0
zhglRO!te5?*<|T1*jqku^}AWsT}=UkmNZ?_-w6}<0=A6MGLhUG6ZHH2?K<P9jEQ}8
zDC&66za{7|^II6Mi%l4p#)<PZPsg3p@u<d2FBtkyiJhz@lgzafkLJ#UM3Wkm?;XTj
zmT!p+XUhU%EmKC<2#;9CaPjnKoFz;fD2UKSp1>Vu{yM{NH3}k_`}6Qyb0P^HP783l
zsV!U7SEgrT(W^G`bA^LWUP5e($^kB;9^Du?Kna&UdyvR;Wg6I|skPDw-YJr4w#ER(
znm0{dZBp;MSFJjxQVq*<YpI?-@nYFvH8Z-zqQ4`s2@8mGHbH9ELNAkL)kb@!ZvOzL
zbaE~m)6AOw^82--vSzn$W^!$Licpx+R^5ALo10FnjySnP4)afW9Ejf9@0+$`DLTa1
zH`e)E4e_21xXsPBr%mY(v^fmgTT&&@_u<6rf17?ToV8FwGlSDB1?s>OEw3~CeL_5B
zXD#8)VY7!#1e-|^Uc=Q>)#3wU7FH2@(uewe0EqooU%ubdPph|-Zb{6lZ$-_+c=Fu$
z6$Wz0#!NBZb&#$q9@yuSlAOHCLTk97!#j{y#Pj;7dm7`?qvAWeS>%51QL`21@+god
zr-pamJ8Ep7Y5(wTr=3<(`G1$X<Jg6Mdr)LDm=W=YXEqk0J%gk~N1(175Fo?^mDiec
zM!sZqrY0JrI6@L<(R4uPysw!?#(=sMP};O-=5t2qzi1Q!!exVOpqdSv9IMm3MDf9y
zuD5@Fv(3pxMvK>!09T}tA7WAoy;OJj-=}>aRYYTSwU-{%bR=a0zf6?!E=9~z(>@6&
zK8C=U>}FrAk$gC=0rbZ&^TqERW<P)XLdr+5L|<EF)G7sjxbCO{#0Q@#47%PgmKZ=3
z{#$yRkDr5LjjD42;*<2=GR{(ean|!S`bG5m{QY(6P(ZIo*^`Hp{Z$X3(?(%!(@IPh
zS(4dY1Z)srZKK^fSh_<tsh*Vy<*<C9z#oj^8y;D|LHPEHTZ7SL?TeV#u?h0|JxM+F
z&v$a-Cm}G-mJe0>S58vXkI_a<dWy~KlD|J@-8tEqdfE`m;leCV4yK19wqxDYYM##e
zpA+IgG$D#dULW#0qAVl*HY$G&UF|!Im(?`qu;GVQWCXSgr075=y$@$)BfPSPQP`6|
zmkxU2MMw5&0g6{+u6G1DibM`6IA9M_94K+Hl3+~%XeZ%rpotU5r(X{R0jrbdfAqT9
zpnHFA+gctLP7gX6g{}tQ-p1!$-ZxzoaYY-zy(yIbD43H}_9niN20+k%wYhBhaUy~a
zogDY>^k#-ahKEh#r<;;>HWpeIB#Hg91m44lI^)OV>+6|oWTMHiMVf~DxxLd%!zj=S
zpkh(X{r2$k&`JB)d4s8sKY94&!OMIc0Wwk9$@`W)7}h?)^1ln8<=Nkg*hiLE-ASh1
z#JxLb_z8neh`oAU$Z!1k%|-|OK|tqCp=r0pn?ez<JMjL%A9Ps)>6LkUH%{PifZBg>
zCU|i5;<{*hMp}gt1~raLsrM+_UiP#R8v9x7jZ=rGn%OV;+t0WesG8>QBy9_Y_t~Zl
zI^~ok8{%#h2=bz$9n<5{@-Uw~H_y|6Le!wL4de)@>mmY=4n`dYz0}{_)>{s=YFwOM
zEl#?tC;@{231|1ZA8T%#4%kgr<NrIAVV3z{mQQR4f;-v`U0kxg^_FJ7eQ^{t-SV8y
zXm3N_iycbP9)oj|ix20U7BkXB+=cW-_0%QaOj|zyF*)}*)4tJ#-W!cbMM=t4zCCQp
zgJtNsV6X8Jtht!{zR@NyDX4FCb%BJYp|$utVtWfdqZeU2EbI1(r&uwtRXrqtYFOzB
zmXBU42b3lAKw;WVR{Az13Sfx8QU76x+49QGBSXAKJ;z{sLe}vrhwve$A28WZ?T0jZ
zl40o%_w8W`%%vOGY}C#$guce=W?{is)$3f64ji=tEC9M5H4%Kq&iJkIu;J2SIqq_p
zET#ZpEg<6wtS&1kL~O^oePwi--?(*H8i>@5m)l!C{Y+^qt%3m}fK@pR4zx&~-%Tn4
z3K&qP%<@>;9FXw4t=Pm@W5{x9VwAO)YuI~E2Q3T!s;_x!IYW#neb^pe)?OB5PfB<}
z=Cd?cb3=Ixf<5(D=sJI!eC=YhU4t3sT0_3%9obKJoO19m`$zf=$Z#dp7+{b|Z2r6y
z59t+iJEcZ|3Uo|#`Ca5`Dm8`zLebx_L_eU{g%($mR%g(AUWX>81#~y-+@2DhAxvM6
zi6W)IZMj0jhJJwfS%`LLop4suSKKl+eB<7+(BjN;fZ|r}%^t`{Md5%XK38MZ+U45B
zJ9wx4WlMja@ZtmP6zl7Lujh~V_yii*S}FWl?=Y<?z$7t%iq8ao)mYO-TKv7)>JHXS
zw*82~hH+$dcE1D^C^9^n@|R9syt#gfU6_0MUUmp_YUkc}l70g#@{)V3^x;iyMfwnm
zYA3a_=MD~U15x7*SMra9r4;bx$!CXWCye}&Pl~tvtI~34_NX0=!BN6ix`(Ce5`EOg
z)b=P~s3cD-LZ6J7-vx!vMCHfi3s-ysvEc5wa^C*XQbliv&wm#^vna_)&kp-03=U|&
z2G75Ybua2_)XsEv-FxuMG7JUa1bo}yJFg74OUMK?s9fz0IW^%ct)`lUc>!S=v%>u+
zoEABdTY;uWk*%G0?uFLRoSTD0pxP2mCI1M=hH42X)y&61S)UJ(Cw}21G$v}NTbx#9
zAWe(#@gYCzR|HL~0gxM}f@k1Sxoc7WnuEFXD+aoYlZQ~Ft472{lV*0DXxFhkiB784
zj&g*o=d0(m1tOX>K3a&hh`bvh_95};V}22-4}OCI(8X7<5f}o0$w=mJ;a#j!Hv$Py
z_7_0g_Fu-G3Az{&ygJ&?9K>{cau|F1&QBwt{)yx}1EAXeTLI{RkxWjvn?h|I+p#2&
z<Xj1j$#p-G`2F7-7Oy@#HQ>u>pqhEp8MN@z@L(k~ZHWAb?ZPr(Py<vG%Rj#bSgrG3
z(usssK#OWmk;PvA?+tE$JN8_JmBy4s^Uiz%na6R@KrwAWV?aUsk75bJS4Gg~egJ*#
zSDa_`l~#l`k$j0c`QPsY@KkZ7vT<6i-TTsE48EmAeu}duN`zwnBLgy|<gd-bfC5FE
z{3M3~nsM_cog)H<z*Sx#&h+0(z$loBLw_SttNw(-vrb`+(~6Y+5`3PnF8*UFBq4_q
z<$DKVi-OB)H;g%mf&Rb7VFr(8X7%CSX2-g6$-JU;*(}_lTe2NK`NW8{pdurrEc>yl
zpAn6^FL**iwePkySaJh__2YenuX%leBxJK4tI^K2n?C0{PWFefIDAg{YtaHx8Cbgk
zdsu>nL=N`q@rz#zQ44E!C)~(q=KmZGlNtq1e?fp_VZgyJumhMN{?$z5>6l%n3#s1?
zcONTLrjo|MCINwc=QwdFg5)TOpogInYC>Nj1AH)$2X+Dz;1hO^?L^wXs&oCS_v$gt
z0G|`eTFC1&5aNX1G9muYlVY?Wo0|w)c7;t1HL+EJf!^Odlu=z|U;lZ)B1Q^<h3o~`
zToQEu8aX%m_jYv%btD5X+cG`h!a`eYYggyE2sbAeeSH5r9ISQ&{5QY@@|HBO7CDXV
zU4XN1TjLC6`7-@|FubU*x=o+JI2r<Vfs&ns_yqAMH?O%)%+Mvnvvy#W!G?&M0bl<8
zK|y}2sJZ5>wU-aRY~+ml^&eVVBHKFg1-zihJBIpv_JqBxY#0+G3kGV$kFDS}dQ@vK
zF-`tD4=;WvHDcFrSO}_D%dtATihK+NQo&u{)PMz&O*cNVRWvYg`Rpa;<06eN`3T#A
z$qc=d-i}#wgd`NgLSaVss6q(h%;bv<T7~#hipwiK2?OLNB<XB=lzx#_5GE#to4{h_
zPJ=w#$|Pq61S!!1nompJ2c=`dGOzI^|Ip3UgNHM(Q}VC>kgMr|IpWoowe<AD>D3H2
zcw(W@!Ka^kFyj0jT0Q%I=pd2O-pi=F6Lhj1JXJTR@DN2&`MHAHKQU39LKv^I;DF6$
znN+CVjZD>G$ezHFAOWeWbzNZvm)l*!VQDtFXz;5$bArquU!@tBiV*&}zkrn1^2H{H
z3$+Y|SPU3KGCPnFv;!=`e`8`IR4#pm9h&mi_VSW#ZS4ab2P%Ldu~y2q1_~rOL9f4a
zLZP}!Jx1I1s3F&zo8p=5Om9XD_qZllQWrFD$lp*VwfA`9>q1^C6UexQ%5e6Y?<%Bu
z;fqL%n`XzKoETaYyGDqIFg1%353Q}kGi56)L8$D`W`xBi4<69M!aMFB4L3wkS(uJO
z$LJsHtP6yIDZ=4TqWo0)1R)10M7dm}0FD|gWz2p`8bBE9>ai;MvEJ)#pt2M;9Oy!f
z0k@NmPFu~hus$|gFuJqrT|T=VWdPyi{?EqLT*h46fEWQ^IEF&#+q^G)E)Zw7oQ+>{
zUi_4~?LCh8S`aor&A<}ksb9}_fx}k?CAj|GQ&h$Z3GYUkG~N&brlmBYB9dz=K$#`Z
z1P@Y;q-JGfrc0xym-g^&pFz6z{~D?zl?j^U{<Iz(#g`)=x*;ySIsX+np-F)AGT>Ej
z0Y1O9(ZS!y&H;jmE`tqD2Bt*3uAUFhSLS#h*P96by>%_y<g==G#9Yj;(9=^z9(dwV
z9Z)7{GZjplq#dVIJTnc>%+G!Sl}XXznHOY;0Uq9Nw1#%VP5{Kl2yD(K@pgoso>~3U
zJ#h~aDkGDT{|Ek6fVz7^A1|K}TRIJ8SX+-iu-1^@>ysXKM*N#G^+rChC|m|6+rIvO
zJQd>SIUXHGN5=lot!Ka2C~rB}(eUpKAq-56J}_LVx41c`n=`d8%J&jRt2=QqQIfN$
z2JEMJz?MM0VAc{|sOlBzC;yCF_!QMYbajrEdo3MxoZxT7swhMX#0bN$e7QAL@dT*n
zcT`cZ^eg;HZ$_pqHShPaHsAO{N<u|QLd7MXZTa6X*W>APbHN*LgQMg1-PEdGxRGg0
z!8%}2MSq8;hg)R3$UG5<mR1S!1bU16g~Ptcm9r@)OB>;XDg!>0sMTPD5&DiPdIllP
zK&^aDWa-~~)8DeCukYcBo<zT0<#;8eSMWTlA~5RWS>NoJq_vp#oCyM;brmBD_Kc!t
z<>jugH&wZL5x}AO^%;!PV70LNos_y^p~=|HW9MwpKkJ}pTtACeZvK|Q^!|g&^IG#5
zO7p|#b9L?eQ3qK6cR<|8Pk#oCwIO%dof@PRRIs4P4&cV6AKNVvTN0a$msX75wSxEV
zn@rUy%@@2fJ=cQ?J>(M|FGl;Jnu@<!7ra^d(e`T6T7G~S99PzrNo&$unjbar=-5Oc
zfsy67s5NO#Kc-`K%H5P8r^E_knSOa}or+QZ-GAawZ`MYMLl-1$L6e{JWAjTzMjcL1
z{aW1cb_#gYo1;>YDDAGW2ww&;?j{a9pwpVx-5}e_%HXMtr8VDVV=lO!%uUi~gbrhc
zg(8`23e=y-$QS}exmcZq#$OM2XR5x+$?veJ?r*NNZz4GWcUq&sj|&MbU_T|oj~y^)
z0PiNvVFwaJp>>q-Z=$L)BKu6X>||wrg)!6af3HuIR$~oO(KtU%Wn^U1h^Q|e%|Y3F
z%Dh}Vvo0omUdgq4_H9vtPUC~h7}uXH4&u6C@uOQbQ^tEQk&H}<FZ+A3@x#4r3@_HA
zqG(OwS2<2-bH0=|`#4WHzTKabBs_NJDL#PB*(*aB%}Ep;(ccD60zE+?Ne2AN8FB#k
zDx5BZ6h7fT$0r>sQ{hjYN1Z`FcQ=ll(je}j{KJC%7gAlEXETwuzuro^QRcM{w8f^N
ziL%8e$ET;op7o8SFbhw45s^};4{W+#t2-Dl6y~N=vrd=J1v31Q=sLdHd^WuYAZTvJ
zbtRt)b<w>7(UvC9@SAQ|wj1MdV+^D;-pls)0WsZM*BUEDYAekM(n<5l`Jnm2y-nGm
ztbx?V02efQuA;0(icKgT8U1!iER)%Thd^=&AXIL6U3vww_VTLNMV>y$II*UpFZ0TK
zX8WH`5gk3b=G$Rwp(zU}TJ;-nt%JOPHcm{+W|nPmdW*-Rt$qYsgZ7IC!)NE$HsT7<
zeBg;xz$T`=nEC@HR{5v|u_N_z<!7k_0zj2KSEi!_LVN@Dh4yimuEcoO>&Lb0OCU|?
zLz0f<qLs(15}0Yo=o28*TnY;P3|prezp0V2g}fdx_xPDF1<?c(JTfiuV}JhnT+vGz
zf=>{VdbCDiV6iO@w@%eGM&C7ojUSr=?8sK-8KYg)pQ*`rOyP9Tz$FDJ$C0j^3cWfY
z@6Xg?51`Gz6mTWVE^_yH2hayc=ZW89TpnJoGH!Gn-`~C#DlAVxL-4yG*5YSBo8jOV
zsH0rCm_A3OqPRvWeS8BQ5mTvtNgVn$D+6XBGob?EO>csR$}+PNeUDps@D&vYye86?
z+ehi;rQ5@jxScJRJCRwY5nlAgpp~wt-zggEqK98k9RpL&%a+CkL57L;$U+9(vQ{Mp
zvB+l}K>;*!*PQr~5dRpMud*Vzsj7U~q_Wb^?qk+EVms`mFOBg|sKX33=+CIA$i|vQ
zQa#5jY1f;Spc#J`Hw!zhAGtn^{6wG}e4ph@7dL2*O6--g-Q{U_GLRyfpbUi@Mx$OK
z69&2MA}f9wX)si15G&J`fZMgMcOeL~;a*91cp^7Do13j=<??Jgs?_cT^SdNeMI$D;
z`x4_7{XQT#Ga{r@Uf0yv6`syQToO=eY`_xWfQ`6|CX%h(>nnNs{D}O+7LVkp$i(RW
zsrSD#Lm}fH$pfF*h+fJaq_7n<z23KxD~=lH3=lU|uucoeBGW`sg7o3;{BD!P7l~{Q
zY-M|^6XcxN*&~my05NaGB77BqmnPwE8lz;w$nYC?a#D&-{y=xEs?yW1qfBIqyTU#^
zFVF2Z(8Eh@>7tvBg<NNxv&s`t;Ew3s4^CvI6h$OWc)o@NolQuKLXHNk08jI4&tlB0
zo(4i#2jy|h_6n<XOJR@FORTdI_Zgw5z<l5wJJ9gGz54E0QY)hHOAA^i5Il%>8Xnw9
zH&&u6#Nb8kQ=J+aN&D<y0?-*Sf1X`B%0|})Zvfc?&<Oz2T_LNcT%6V9m9noU_~hA6
zY~Vk}5P9@}@`7qSr|@!HN8lPgE}9hYQLEC467X0ta4&Gq2uK^MM3J91R;j*fUetM)
z3j5IJP<i_khZbiAb>~lb*qH^$ucqO@0G{$D54LvVa(8(^_QCB}YAQv1e}IgPOkag@
z_YdzR>1aCLdIv^kW;q6YTr@9AdU|@|I+v5R)>cv5-6v@onYk*{;nj{nLyWaC(_op5
z*?KkC5L6Vb<<PIWE@m`zyu4SPAw>OgELGLjgz@Y$RGql0q(>h=Q?_)+u8imie~DZ3
z1t@?WQ{md$8gL@Z^X{O=&CTt2D+8B^sOEET)KIFRU+Z6yuCcMxllAWM3gq;*&*tWg
z70k3he*6HA=~#~E4t9l+1*=8|L!n6=#$CQhXpN1He{zPSB2z!k-d%O|d>ziD;o2T*
zag!=pvnceJI2n<YGcnn!v)^{RIxNY`qG2_R5r)z-ndOm3*EN*}_-E#C0PVO~KnwAH
zxR^fM9uDR>M(Z`I$jI55saT&VP)Qu<6C@Ld3?}m)Ej2YHCMJGb>&r_Dg7&Z1x!lP@
zWmJa0^+Ht_-R@mCCirFPosV9Vkmrhn4TU3&f}whECtjH<x)S0gPD?1<0Ifx3aUT%q
zEgPH7Shlpt{V`%sZgFu`2k3R7T8Y-h{&bo5)8k6j-@jk&L=5F6ko=kuKC;l3=0@QY
z$O*XJp4@nx>~x_*QO{iQWg)$t)cyXjam-!$_2%6d7eujje)F=PCoFeBFrj&)3cB11
zyOnqY4o#FUy3*J2#Cnb7F!*)h!Ec#ZUC3zfWZ!v%0s-EX&6~nlwinHR9~oNMH=zSz
zcV2GxWFMQEA&$dUr$vC>ZpQ1!vFUql4J015JUt#RHV9EhB&ZhT13YegeB97Vy;R2s
z#H!!y@i3)pg=nmzO+PXguwS#qqBWMM-@;=xA5JRhb(8da-uXy&xThXi;cS&@(cWHA
zt0JKqT1RDN<x9=DxVYP!n<kGNT)oZn-SOika&mIfoIftq$jHdNHcQ!~X2Qb4$4u-4
zy}i5kV6nWH%R|2{r_V1gioD4Za!CUOB?&qRU9wM<>!|yfJ98dkP=!wj>TN{4Y0Z@m
zrx{$FjehN|8a4W6+nYqkGBV2v8oFK?3g}+9UD#bj_O2^$6Z|!i0ze9Sk3&n3<@g`o
zf8Yo>8EVITD-~Ks#!$$l@(xVF_YLS5&sAG#o0;td;n3Wk?;`4Be;C<@`KB>bQ?CK8
z0=WKsgY!}5$H0}RGa}}hUsl~J@)X}DXZxp32NI`?oR6lg=FrLbEmp?}$f^sGr6h)v
zMOuLNPk4AZE$y1S>I-D#cJ23ETr|JoACrYW1sW4$V(_sE?lT-69c#P>+Wb(Q0Vi}l
z+hWnK)@kwjz|Wtk@uJdVvhea?zC#3S2ldCna?A46R2((a3=b_WZJS%Y^ARRze{b)b
zxK0LDz-i>8e&9~h{w8uWsjdh<AK9s;r#cFg2m(?V86}?g-HoMDxnH0St6)ocegeL#
zmI;rC?c-9_zT&S?tCU37zxlO)=7bp|5uRPeVtP4-QAMlz>+b3*c=rb1KVKBFTwzDo
zU?Y-7%HShb;k6C~9T`y|jq-z=s9<bzGMW`?NT)AduKv@b;1$)_d!8qc>GYiF<Yb$S
zuKxHE?HVA;f4<m3jkB1qKG^7uPEAeCL`aS}K0XG3VK{|9nlECstIK3(Br}AFt-8AU
zN4g;XM{)6R3PB#1V}ox5b!2!F7wZusLy275K)4$4iC09yeBb*a7>a4!q<wJb<UAhj
zkRc72zu5jcV8w}b{n`v0tSRVjv6uWqqO)scXM-zW6jJAUgH!R!^@f~+M%Jv5vU$u;
zS_$lAO$ma$^;sP)ZuGFzH(CS`H!&#)RSxlpI4MCPRlujHkU!BEn?QqJP>>8bf;raN
z-j1N>OI>(+YNtWwVsyArcdYW7FtD}Uu;a&IV#{%2+|V|x^4quHwkO%6xPnf9YmGXg
zfBrzD;Z~pCsh578EF>sC0QOtSC*Cg&E9UHcSLG*mM7JNjZi5rjIQ9}XE<XN?GECXd
zt}a{xaY%Orm7=^nEfcmfq)e|#Mpm}L^Nz<u1sL}`N6Z1fq$^AV1~rOjbN?99%ci&v
zHf3bRpawRb&&)51gyg7%qE{e@DX_EJ@xtiy7Zi}h^0dqP#sQV0qVy*j!_S{pX_+jE
z6nAPE6m^P$&H<B3Y`)ji!xiw_UV}IKCy=hL=k8d}#~<jn_Vz{Ra;PsEnXXA8FCOG;
ztQR2=Nb3t692{ULY@P#O<mvuwYcM%D{xu>Shxy()%CNo{D^qLVtluCb@p}tMy09k#
zB4S%mZf>qho&v=OH*N=rp!1<WGMYe%Y&@&jI3gxuXZzFL#Xhh3*js)5<p>)Ua`wRj
zmAsYtiK!_%CjWfy_wNn4Bp|Q-^B=F*v98S*8eQ?4#YQMychIs@z^}m^E(F7}_cwB)
zMsJ)T>2R<<HijNHhR~p#58h{wYY*EJsra#BMe33~<2_v*tc<ho*eAGNXOn{4rV293
zUxH{$fI~({&3Bm97l@Bk-#QYxOvS~;3ui@Tq0k<{&E|fYBc>+i<>h7Px6+o61`;7D
z2I;Rn?atR6LEi(XT=L<7EOCvvbAN;Onja4j@A?g;pwn02ElB6Xh1qH=vPOx|pWlW_
zBIfxf23^4k-`yRxsH&>2-cd72XfRI!HqVsu?+O9VD^^xjMGMvZ#8wR<WLe3BU$O$i
z=I-jKx5JotGZY~h^*);@QDjZ*?cr?}9IVp$3aB|g4fkP&gwtTLMZ)UuMN{LV;`Ehb
z;u5p0KEIkzw~*NUdw>OV`(nnb`R_JI>w3Q*002();%M1>rrfZcX~+E&m?LpuzQ*Qb
z^68DNXvYihI&7_59S9*+y8Pza8t~SpI(A+}1Xi&<fQAN2-zGD}{D7BI;XxW)1GH+y
z<m3wgLWYs?cK!K-O+Z~&S9f)^i~<(3k%GJgM@o$b{cLzN7#z{O+?%vKUTIrhT^-?s
zN*Tnkp}sA2314x2ZF!UIzp<8U-$yl~M6aUbevX?G6C1rxM(%Y1?&Nm|+$<N>M3dir
z!KF|&HXkToogRWsggy{#342>g;Qy24KB+=}qF_-v?2dhzt^D$EdqD$2TCR(WQwzkT
z0DzR<RhK<m=k+mGSg{P`dsv96CJEJ4wb>{PNY5XOoY%WBSmV)BM}Rby8Si$nN6<)Z
zYlP(AW`Iu4_YJ@%YinzXfKSTGV?Kx|qAs_$w-PyS;N6{_V;3PZGP0fb^onWdMLCW3
zp{Ei)5=dUBGHk9CNy96d`NBTf1Rl0hy4{1k3@sTWf((D#AGV`lQRG-;RMKET2+?uN
z@Vd!F{$(_JLSR}tUw5<MrgoBYC*TJw<QyX;2<c4XHXplM_Fisu1?o<0AU1GlB<~-t
zmZMBrv0p`m(Wl7+X&dhPlh0~K6$$6(p1-qICB?-Cj4tyHP6vBSO?P{fMVWy#&R|~!
zC%_k*J@4`q)7!C98N?f#n!3X&(9-;-p!ptP2>UCM(*Ah1r{fN+5tvU=5C`#>5WwL+
zt$Ugo#UTLob1Nc?Zh8j0nsE2fV#5!ZFNQ2Xj?e4lUf<IFEiRVc6Gu3Z>X&R5ChQ*-
z6(`@}5dm&T$5`0dDCFpPvFvq+K}thJM0CCV1J=^=1SE!~<fK_|51!b}fP?DAJAwP#
z`GGn#y%sJyQ0&hE7gGbc)9*D+-cMenC2L^S<1itQ>*M{h765d)i1*h+g7C=51i$p*
zn-y|%bBoFzX9ZrOB?-BxAE!y@OiDnGcm31HQuyrw*A1KoGg|$VC2vFaZt({Du{xRR
zzdg+Y6@-njyW?hL*NyuyQ?pCH?c?|f`*oK}NfDKeh^_Pi(sk&It_~dyB|8raAUBYh
zhoB-X)9++|48)9rA$$<6`eJNcWxZH$3wRGXANDW!GIxN4f6u@!YlU_|OfFd?-s`bw
z*KG}_c|R=u-d}ms@e+*mLyC5EcSJ$V^lWeP-5RrmQV_sjXLg1~o;<D&DNdLqu06aU
zkck}G_^sh|5n#>0hJ}-GZLPHVO-)T5Fa7bepkty2u-EhU9G=F)+`O$NrxprMV)&U)
zt#SXZRMR#k*Z%CS?`?<-rWi2k>n|apq37r4pE(o%uu(>IhlGb?zI^%Jikl<%htK=#
z>uc->4BFn_-sa|J1GL^$n+!tMcrq#~tg=lYOb-t!NlBB^()2Vm@bPiv`FeZi#AK(B
z57)P+4~1(V%3*iN<U$^(s38DWiZVqJIkxq6w6_~rjg$t<K4E!Sl8}%9qbw~g9gDLg
zFX4XD61p?aIPoQrY5w?9gB{>9?CeUFuD*|Ke0;=9T#fTxEf23Hx4`&dj#}!#jWrxB
zSkl;FG|<yg7FPh#2EzM*I>(Ds-eIb!LIHhFPZ;G#Fd)y7@<OQ#uMvFIURwHVz0FyA
zAqtKh;m<iEWSe)sci{7rCd}eIJ8T-!s*=7)yDgI7i3i%(_^K>XDRGB?FM4T27~baO
z#-5zKunqkRV}POdZ|XY^>O8RqE88Gs<bN`AT_9x93Z)e3WfKm_|DIF&!9&1gZG*OG
zI*HGfj%aF)|1(3O*4y~hBRqUbKvQXp<j|qia}0@;#|J^cuA5qKTm@EdV%+fHKp^0l
z6mH)6P*uD!y}V|%NmFE&b_p{Qa54}_Pulzwgc(q&Ydsn<bpYwle1QR)9~BaiS$98k
zpX)@2^c_YWiBFKKa@*JAahu}WKUM601Q*AYh5kqlJYNEkuScSoQx_P4qEr*;!wAU*
zL`nlQkSE3`c*V#R(c?m)QUtu4&Z<$ZygXNd83%O1e#-(<LDoOo-dirt>*xA3guy>w
z847wcz^RTLp=?767~m!NVsHFtB9;s|&_vE-F<@-f#AgEkMFRskCcrwve)B@zIf3ad
zQ+3cgFJE@#@dWToFsQsQ9i>uX)>LUv9uau{o5Jb?FG>Lmhi-*bqB1SZ!>wo?rFbVm
zy`5z3)X_6@5PDwu9a05<y3Uz@8@B?!IE2!?F^k8XD7A{=A{QPC^<TKyCMU0Zx=u_3
zUdo<#?)e}T-d588a0-U}4iRMt&CAo^s|N#`^5);j)c$rqNJ3YHcu*f*g_)mNlmc-w
zuoelE@}owNdA<dRx4X8VRt$3e6l;u472T&aIh?qpDvOR>;B9juw?jKK3!X2Z<$=iw
z05YbN^%Bs7h>D79V|)5+?9;|A4onVtG{rT^c0?V#_sQyHm?_LCKF?Dv9`7v(DVh?;
zwu+GTaIEC6bKb&n20jQ@=-BL%z{D+3T7KP<*Kf=oh8KXX`n9<=tj*HR`Nq=2DLGI0
zv_C9%I{hsO((3L$ZL5-taqGOJlIpxsP6`~kaTFCdoTSV>MMXEPI6(<A{|jtVP@a2_
z6;%d9LMH)c9Tmr;G`B;3jHX@tHcW<*$@^Z$`M4YhFK?LL?i;<-)C+vo5=1ftElxi`
z^9TB4oRv=vj4a6Ge8674xuDM!Eq~M<8_s)}PeJ9|Phd=sSMX+KdQBbXx;8dx?R}Y2
zXAo&D%BT4Rcv}5i7OwfJ$>@JY3pa~;(brxX_`pz+-r(8s5Sv<4cka=UeA2MI$2xC)
ztcTm5if12%&wj0yn1N#8PvQLD{W`(4!-~j}C6Dwq(y*xhD4L+{@)e;d;h(6~1r-11
zoJL^C@d^Yu<ajMG?yi4@`rg>U8XBWx_SOh0()gsYHszJ2(+Js)i|}!hw(d}t3bPW#
z@zo~=q1^fo$dXTHo$K|ExLlUxBj{p4#JO3`L)3{eCwwAB%BwDw?S-dfe!Ft(bfBkN
zQ;us6?+G0NNHrc1SfvZh|LI|@M#m+x0ICkR%!Ur+1j$#Ma+b+Wx>x?tH8|`eY1vxW
zS#(;bhxbH$Iey#n^#8}yR|ZrSHSJ1=bfZY;p^@%7bf<JTlF}t04bmmuB_g17NH<6c
zNOwth-^KgA-@W%YoW1v|nOV=w^Nhry_ofOW4?FmR2pQ_ZVH4eEjT<Z|z;tvu;4sxq
zUcGf;f#~M_26(516DjzLIDo$)lnlF<JhUf#$NRljI`*oyJ2#VmDNBX|>^gREQSoRe
zkIbP)iB+WETB|K%iy^#OzK~`m=v~LxaH9-ibWzxP^kUxF?-EW(p(OI<xrdTR)ySyB
zLN)vFoEuH1$%;`?&1#O>@QY6<pYp}`6ztnpKGGC;S){=J%{*PUl~iW>I`sY=4;D}J
zje74kT~V?6b9$u2(bhZV8GwA~Z!GoYkUQAzDf9eTepz@T{Aufy(2nlBk@$AzC)Uc0
zYalt@waD>V`p3GG`;zwXKr&W~4qCX#^3T|jl5>q-DN344S$`%5gjg65Kz_pJ2Vn0c
zL_rC&-txSn=2~sPEkX)rj2vFy|5Obiv!yGOr8CQK<MNruoZgc@x1_aGBS*XY%QWwK
zl>G7kaynkT%q`s`br;J^OeOnoCL$>%5~T9fJ_|-c$eYhU8QijGaLT<Yd(8^_F6=9j
zyR2ee2q-o$_h|t3<v*5n#RQ^YCY24fNPOoz%ze<tlc*-W*V{}#`u0h`UBRM}5*?~}
zm6Uu^(f(y{ZeHZS85kD`zNYY#gjB@8>S)%MwFDcip1djSLfY?OwLgQ^ws55fI5QY1
zKZWdb#s>U<vWzfRxNy=-D6B~jyB>TQ94K{s1L^?nj|>+j@BoB+eed#&Fn%f1&`6_A
z4<9z=*52i2qA%wU)mFm+j`lyuyvbL=dlS=FR4$C=6D}qUOGQOuwGVy?7MvM+#rh(e
z|6RCZ4Ue?Jo_P@(W{KywTLNNHV03<}KVSAa><MS_Qu;aXgdm49IkkHPgFN`;Xcc;M
zO_0&+?Rg6kR{h_fI0mH={v-BhfUemnJb?QHw=z|4NH4ZBqayeTW@=xp+gbi~M0e*n
zGd*Qf3klfMg`|!6(@>vYoq3JfmdFzJ3we9)qc|@Y8SiROT^nGtd<F-v+bCAasU!1c
zT7tyqNEUVj<GnZ3H7sCP@xy-58S2n%dZv||B=j*~$upa-4&Z_6wUQS7xVTK+n9|JF
z2lZoHNc1^ee$$8Dgp7=gX2G6;o`d4g)9P?~ph7@>hL@KY><OYgm87B3Dk+V+Z-((F
zoX^v0yL#fOMMTpt@xI`ym|zWM^JQc?3%1|2imsOB!J6y|E(-6k2&XuxJRl`1lIOc8
z*48)Fk0g6Ud}&19v=%2droNOBo(~=!D7r$7r8bf^CG-&dGOFP@`!exKr<(`aBBW?N
zti2;)l;KSg0d~L2k~+QWf2KBCS+35-y#!3vzz}ARVu)%dd?A6uTLsRTVO(Xiy*DM-
z_cyF&R2<1O4`8*WsBtzow&pZiug0G96yMg^@S{InRz#1nvJ$Qol(Q)cABnq+&8^{U
zpc5;${|mMpkY!r92+J?``z7eExqLp>IX)u5mNd3ac;Xp7{I8!^S5AsxN3I1pVM*6H
z_JcfW%NiUq6`Nz;Ue~V$bN1P%hF>D`%e6s<f~i9IYHV<>t7ksv`fo>9{r9w}H~7T^
zebkwoP*!@T<Yv1#_DF!Cw&F<0*Em<Y9Fd7;c49m!D=4m~u2-=X@0KW#L!Pd!G0}?i
z{<GK!_^A(n0d~wv18dzV2qz9oF^;tfgFB`muzCF=KYw#lDh~)uDN82n+{2WSB9;iU
zWW9+`IwX8`u=U#dXjOg0qMgjS*7*XZV7rholD<MbuDGw6t?qS}{5CSM`VNJ7;VgWX
z*)@u>mMQ9%Twd=4tBc5U!Xb?-aB4!bH=R4@u(5GO@^rCdnHnf5qomn+&JkXlNf+cY
zlT@KJ`la|)q;Z5JIBJ#@Zd5GFyxSJ{ZZnz4#UD8;1`=tU;S7O;lr>M4E^@s$RkPP`
zt#B+c{ub<E{6PxF0v&mR+F4;k$rtC0HYqWUHGf!q??u)b1#jZ=gG;!Lf-0=u?m}$7
z?F*cD7J29PS1UyF+YsgA@9r0u5pwz4>qe>JYR8=-ju-Qi8J|p_O_XdVW+1Ue26}=-
z?ci?C$otdW(pS-G!{Se~8dmLW6=NSPb~jen$Vd}4C3WD%zCaws&!z>_{dX;r_|<qs
zm8zNYUS^eKzJWxRQ^*byN^zX}QsN#6!BJnq|Mj{?qBB=Pq>{l!e7r}J@7Y{sfiHv+
zRpk==A?WX9TC8VW7?XkdZt!o<<XcXbY#lk`P;z87+?`BJ)kfI$QgNDvU1^YkC8*Zv
z@gE=fEZp%8W)-`PX;2^Drqi=H#hYC&`8cgTDSQl9x0|5SH!-bWb>c0sZEBygS)=7x
zk(CD&J6-9qEY=znZ<*`={bmvd{w^l&>**Or`u;^i(j}f89)1uxI?DaDh`P{WyvH{j
z!$T_c9RxD{otE^y>amA+vLt)@G%(N*a+zSUZC~X64Uc9#dlG}BU!Wx8)eGIWex_pU
z*;bk5a}Gf%1RHBr=T8hI!a?Mb3I`5_ur3qG#sHh&kEJW-ApsL*hLftM0V$*}Z^5&Q
ze{z5q!%Lm`UXirC&zjJUt<yu;XCg42D_S8}Mvp&%r&+U()czhhh@QnFkC35#hAS|I
zRqbz<MmVS>pg>lm`H6#nuBO`P2@93L8NjnfsMcIAQP~?rj6@ZSgOnwgQl<s%f8!%0
zsaZP-#Y6w3=Q4nj5xvg6o77Yjm@kf1C<ih~?dLVbtfpk%+iN%e=snF&qn+9(juL$T
z?_XHTJ?r{;o19h-jz30o6tymtm_!WWwxRXwRiqN-pQkDnlw!e`C47m3Brtd$=G5!c
z#RETv##!tzKqI9ZlGKC)4rHcUyO3pvCGxMUdBvMmdVVPB{5So%#dA;3lGUdoTbA-!
z!Mryyc0>_+nuV+2o~RV~wE8oi3@dl`>BVMP!HzkfgCqB$@npe%OXyQ-`7}GE5GG;}
zAKv^pPq}C+85d7;6Wk1drcn?Y6gor*-9R?g@vJclHX+u&|IIkV5FuU0NJn2jD5e?@
z*dGYQk*JTCGias)bjd1vGK*mM7`lxbmPmWZXzU5lfq#J@L7~qAQsy)e{_S8Jh4F+L
zPC?OWm4=cedmcHMW9$L=|MxVV2uKQz24;W8r6A%hWRR)Qfk+aMq)u)I*;suKP;ARx
zR5YmRAy6@`{8M?x=Dmd0dC57)v&oO}mPvEt4gR}DNll;*u$%<kOd{n}<C04|T?S(c
zUM`0`*Go;P{C}V084U{fB-Q_v6jH^!`2Qa){t62egf14BVYZ|tH6;MJCKHrw3num}
z6_mubn~9diwtc<1>-IAH)>1sR320UM<hZI{#VKv*LmrB~Lw>J9@3I^9&=QyF9}KJ+
zz#B_Cpj`Wvj#2J!LUVj2%P^NbHp4y}JPA2vsa4_I1<Cg<bMJ6CFR9Hi4Su}^q6&EC
zlo0f%mdc|<gB|*EL~<78hMqY8C5pH7xMMl@AmHDmPbt+xI(-y!THhyFX00hux11&e
z-|_o8Ke8WN-(~o(xoB+fHVrp?n~g=^*Gac7URe^sl`9l5(ecTpe?BEK|4<_G)2TSR
zm1xyI2)!NVS*i$@3vZOqoa|j@C;EP|vCQa!zTj?||791a<cGx}Uu(*HI}z(CZKHri
zvnAa$u7jd8$P=Y$JosKpzT57#hzP;~IEkyFsy7;ruh8avQ#lfHpPDjWSM3@6uF-z`
zA?)P%sh0J<cJ_>oz&;+k@dJLV528cDK6OL+0q=G=e`EHmiR=6t#TC*YG6Ie{vSn0m
zR6z_*3L6NF4-!JcNKj2a4(#bK-%t6Df^NSqKFz32!_1GOE%`J)ZLP0lqj)S=9S<0-
zj*xnvV<=o21UMZpF+Tlk6uh4*Xp}NxY&k_d5L~tyj^0}Ko%HB3N#l|q3veUXKGg)}
zP?6m{kfJ!f(_Z17A5G{^dSgA^LY;2~WoIZn2X?iSIwmm$)$5NhKIa0e_cD)D@QJn?
ziHVL-UtEL_DXtmdiA8x9EybS;+<xokN4YFZk6ykA*Vjb1@l^#?o<>O98B*)$3|PC*
z+tLPzo}`GlXEk~5#5X5=AQic5&iRXzue+9naeLGF#Acl#Di2xhL?EaR7=HK1X6lEI
zxBuBlK|l^${hdC394=ZY%$WjJD!<jQnTGu;ttn{@;>Jt_IzybqH(Oa*UORk0BqMg2
zV%ctaQGVXnp47$LdCC#!&2t;C%8raijY?FHd{s}p6khV4H2$>bXG%l?`Tg!{Gc`ia
zFK&Rnyg`uS@X8J=S19Cjq?#lEXOgMVRs9|ouMUu6_8P4dsYR|{pDGS%eJU-sY5G9Q
zQ*oC$T|s@D=Ul)1^WSf;4FtDi^%)y^<3qIfW128IoC*NF7+1-se9^byv)#+U64(~W
zv^3#1YKl+P9FnM1o<vIyDHJ<aWSw^$b?S9hD+e$mAIIDU&t<X0KB?cWBR_=CI8d`Q
z*1#KM)*EQK-N#XRG>sX&weD+l=D8?tcQLZhsi956Gy{M~z6lfoZP}ZZr=6bfZPi)+
zOQ1V&-Zvo+Q;kO3Y(3J$v#cy6Ma&c-9yLhybBY0Q{4|1{<{0d{1e)FoPZgj(j}})o
zrY#@ANL}*hMAdh<t0libhe#-u$D=vLtGuEY=BW{9{K{!(V~0^^iUr1Kk>y&P4W)<<
z4fxq<D2mC0MLf$2f-T!^&a@RTAh=hNu~5yBwMdAtvgaW8dM@;f@~frTZ*A}vrzOHs
z)5!f%;``b#!0?z;mPcSJ%wv*_OIuXQHGCvtGZ;DuQwsP60I4~1=SC#gjXA3~h_Nt%
z9n|5r{dx2B<*z1I^7jf{7Wu-p`k5D9$F7@YrAdjrzQ1f-!X7Yl7tD<=6b%PFWvAY}
zZMZ+n)Ym>2KLJNEFZwG=3<HFa-!aPMl%|9U1T4%TVEGbge{FKhrgEeCyhYQ)ZNI6$
z8-u<Uh{>*-jnBi;S}mXMDEbZgxi|HGO`;aR4IIgr+DI;%ogonH{Kg{*pJAh`{og3~
zT|shEMick9cPVEh`6evYTcNN}sG5XoE8EooE2ONk^dsP~zk3VQ61m3=H`xE>cUr#p
zVUEWv#~U)>53WnjZ0uXy%jI5aYHTj)c^&G>3Dt%Bij={@l4+KvdO1)qijz$P84f_l
z$${%KX8DSac-&f-5G4c43R4>nL;0?~-ZnzuYx`+O4p_**l-^}->%{_PvgC)CUvac<
z2o=4}Pt@z%Z%hB_{Y&GYC$~O+v;#KW&^)JRnAvKh(W)JdNZ4uH>zxYCR}$)({E@Mz
ze{Btg;x3zZRFWI=Q@D8evRwHtC`$0E2W@&3<!8F?jW<2QTm;FKhquiKqb>U#X#MmB
zQ}McDRXe4>m5qu|BsDjhcDD}Wfkci#%TrZEZw0j#Ox=}L?9S7EW_18o;^tEM(yNyN
zca0W91k1joK`4uhIb-bs#ve+I{CR6_kQ;GiXEFAb-{05oS5=PMlXe19|LsDdd!^pj
z%ttz+&m1*sz*!g<8Fgr&_Z_;<_&%LgE&s5eHu+tOZI#1%Wgh6MbxRN}c-1nwvn%pp
z=X_E5z5k(zegEn_qw$|U1kFwawwp<^0KYSd3KKdqO(x0{$OQ~owdB&rMPE9;#2y9~
z9p}dj9$ZXT!4nQXMpg))&uJT7Qam1>wnoGa<%_yn5V^XFs&=H`rv}=$oZJ)IXs8-2
zvfs};eu10ui0QA^q2rs`JNz2p{lM|Q`>2~beDJI%h+D*;r{<|@rSO@Z1Z1b=h3Y*8
zbC#Q?Jr)HXy4@F!y`r{?+MIw!81Cc2#<{0q+oaUUByNXO6rEe@|F|%>5nX&0g|e!1
zR{gD%hvgV*N=Z9$Ic?9{)FyKlx6yZ79s2nFnm4ng9Zx1Fo7s)>25-S%m|U$o)gahU
z2RnM3Mraj5MUXx8dN?dF<g(JX9G4GE&M}>93Gj%un|-eLxog<j-3)8*n~IlhZ=DA^
zo(C=SfHK4)Vwu=>?IzA7?D#0(FZ!&|w?E$ab11Pf!RMeHXZpJ1jTyRDTo-j<ojp*)
zIPkYM@pDZ1mC>Yp)RD(-!2d({?K*4mqIX~{q@PJQz^nY`)!_GXYbR9(7J5j69F)uH
zfmuDazXkR%hDXW#r2DPBlv10A`RUZdSpV8KeSWFwyUwflkR{KJc7N*4sDCyKCRbG1
zceUROZYepirkiwH0VB=@Q1wxcv=dzS$;*AEopXM8`5&ppK4Cb(@YU_RGat-UNjLXN
z_XZVFTTKHsH535EQkH*{XAvJW-y%+r*E2pOMo8S~S9&?tUW4+n$qxDbkzZoFmSKo#
ztVL?9AbWt#Zu8BpfN2_K<esASqtZHF0li>Mz9GEjsq(zV=%4T?<CJyMDOO52Rhz8r
z22ayiAQ{&tC`7SrL&aW?0f?>@p=~;U&#wLe*(oTG4l@HaV6Ya)>~L5OlP&GNG_IL_
zU%Ot=2zmQd(;y_j!I)tX5cx7P5?)eMt@kQQiTx}b-rm<turq-KcJ139@8jK^pUquW
zvzKmHF9W?WIqft%8)@(SZBJ7OZ5vQ+X+EQ1m%@g+{mO*lrClp3yPy=5K8Da6(ZS~k
zXi}MzLlR$4Dmn}kO*B8WI4}Bg65xQDgOJZ#E?)Ndwcp$_c2zvHw+3zj>+JT*+Qaui
z1jv1{$&wV<Ua7zQ31{{RxVxZy%5)zEI9>V#z*Gi=bRmo7#B$C_i-8_lpPa9}C9SKG
zq|rtmIv(S)BPM!B4{VCQE4KHC(sr!vZ}PmK`gZC1i}n<#fO=PfeQ(t4{2?DUpQnv&
ztIRNAh()UO_S$&aD~i{u7IR+(ey%P9?F@q@VWuuy=HW^?jrOO%w(BiRVSmsl$N5Ae
zY?`x)5M6>0GI$*8KDc0}v^Jz!=G8d3SD64~)XJ6)3O%1|!Bwr{RYNRObE=2FN3trd
z-D^wQLYd@=5)70WD5t4jtKv7m#|UV)2Xa~Kh;E#1>{&-Ye?nB<9uE56$lqoyowE)!
z52?EzbSuH`voZZS-(R{+<z5C1VpMrCY%MPDUsZNi7m>dWq|hW(pEU^bfAa#Vwz5bQ
z69G*R66=Kw2R@7lec*`@(<4wJ%CRx8XV10|qV9T$zkQwmCh|hd2XG0O{U+~wB2F@6
zrCV6afi7U>!N?+2gf@(d1~67d^qikd<8HAA@JhYdwObW`gfoJBe7RUkzFfV2^Ly3h
zJU@OxevZq<<T5v0{AC}zUlmXw62%jsUyoPI#dLQ8sM<tV%5-dGd{VUeYAB3GdtZd~
zM2siQqMYl`=@$G%3?{wuG+`}f<=TV1id7SHbi2|{JkVC6_>LsBJZl-ckS%v_M?DX!
zhl3e(^l|n1``FCy4?HUFqf&ipXhq82;fE0a%}$7#LVdm~yjM-{#q%mm>R!+C-;=|U
zHpek&=;OSIbmg`nP!p=b!#DPJ?VS~S)4^WC{iw&I3bRtK`VjE`&g^CE_Dcmv&CFtF
z>x*A_lhYd?dkK@}(u~`5&ci2_eVCV*-2Sd9nOWWo`zytIK_0kC(~I`CPpofZXO6#}
zvVO_6oUmF9T&5d6;5BvGcCKw6G2{}x;Or0e^hXL2A^iH}x8@t^meOeP#OG)*WF)}q
zESFQXz1-w;^he11iQAN;g?=D+k&uS44FCBCR-T`~8MW0C?X!{f`g4iuR5E<sP9xbq
zh;yvuP@advL4ZH++;zh&ap&g9(=fx@+EiDT0Yrn!F)?Cn?$o0g1wyMb7GM4;aCQmW
zNY;#DE$_J;c;F`4<F(yuNBV|qFeUdSZe7VAn>F|CN@S&+i^4)BGw)#B|2R#gm?izT
z)1k-_29I!VJg@4N`~9O_ySU4H>YOM!8hzenrma4ffJ@y*S50!q%Vtrm>EC8osXiMC
zW_$m+7G{?MR6<Z)K6Ejb^D|4d)p~@jTOUvBFbWyJrjI!j<k!(=eo>;2@8Uh6bAOpu
z)Sz5`?;Oh5<aX-}1X}`Z&K`iLzUL&A$FMe66%*jvI6la^!ys;Xy4@2c)iJ`=5+0+@
zpFUb_9?@-_;==t>F6P7Xc&wKizik+2R=(}3T8T#g#p&pwUZgJjy5CpPnlG}P_E|-Q
zqP*L8oGbNTvZHmIjVIw>e(`M$Q3gsb$;*6LP9?>ii9F3LDD=Ch+7-|sv3Z=<{+(Z>
zS1vPZYp%-dCU?YPVX|FaTWr^;W?6|!Puj3<e|Ds+F!MnNSD(XpzAWVpKc>jVC(qJ8
z!W(@J`NbOJkmaWZlSXaQG)$-_?jA$ABi)-Dn_iT)I)8X0WnQ%8KW~+A5|7<7bm-dC
zJ<~{D`10IX12wR4y(g|5r=9J&5?SM8O!e3fIqJR{nB53SHuK-}&yQ2F_&wbZ{%$Y|
zWC@z2@xfL_GuVw-_+8?gK!DOaa_=C70m%DJH$Gsx+q%-%zS7Z9&z}21##ysAM}kq7
zq+=(3GilW5H(4`%oQMe=df%e|%;k}o&{p}dADJAn4nt0qZMMw@>qD5^E3TL{N!963
z-jTLjdE1A15!+k`{c>c<`*r@+ZK?y^g?iSe;;&q%+__v{e1EKQpsf(?W5H@WtUFJ*
z_Bb@N-fgVE&^M;pSZSg-NMxW)+q_nYY%;;t`?2BH$#<s!>=H!`-LnQ4&K0wL7`la_
z9s`d6;ccnQBKJ0*J^hC{NrIu)`kFV45Uv&-qx5YpES%#T-13!8IidOu0SEe<^OTR%
zDrBmP4UYG&%M{a5wjcM(EF9+S#0Ps9{~YnUKxQ{%%c4Txw2R1#e4i*#hsRT`#8^}G
zw*FJ&*QZK26@f}o?%4LU3hs&iGh0?cOLmvQFCj)oiYrR?Fm5jUZQpL6GD=#kW%se=
z#_64~8KuDkt#!G8tN@I^Wk9P*kp0Be9N0Nuq;KuApga<ko-8GNbLiec{j)>+o=d?!
zZ=Da%t?)#$=-rgO##J;RYd`8KeJstbf#iBdxb{Z#7<MTOA!3U8Ot}rN5mznHq*CUE
zqY{w(IMc5fT_kwd{an!oSC^GtN!IY2Hr_bh2ZOyH=#9}&NouS{5Q{lSGu-9iB){=k
zDly-rv8a(z4gArT^#Q)^(Q<9|(sitb?dl{(PRD<Xi@nW3MpnbwKF@&&$)}9}>=&cQ
zy1?!Jif@a&ch)nM<tNE?6QOw*NL5Za{hP`|+jH|l8<WA{a6s2Qk@qBUfNV0=jfBsl
zO^1^&6$CS0wDF6URsRGX2%Iq}J~ltvic^K}b@*JZT>r3h%=G@&0DF)GguC8}H)+E^
zxT<2?H(L-%4bA$ErG1@(u?dGPKiv<BUMgJh`l@(mRexr!3ui{{nWvqG0}u?jdcUk(
z3m_9I*-wJjF5RkFR$^?R^SNvTL?(Dx3(1+mHL!?<7+?_hU5*C5OP=!S2r`j!G)W>i
z5f=L*cNh!)DI_3?g%U$vE?;%P#q?nRV7f)3DRTXLxRX(?t8<1XO)I%HC37MO!~O|K
z<*OkTQj~ufdpjSk#{HVsj7P=FRHqZ%*y{>`57gwlz@FaAO$Zbits~=gxh|8g2nI}g
z;2rWbJ6YjEJxI9T#~UQbHDI+A;b<vgyhv%>MC0?l12|OQvNxR$T^!ytNBzG5%q8o^
zha;OWqZ@$frf5x~syO#_-fEZDXjLd-yufU*?Uw3RfjvFw`Zo_{_mmbAcy#F$_I8~V
z+8#y!tNHgI7y#Y2#7hT*fo0x?qo$c*O}UO2ixe}5^jb%#ztYe`IJIm$dGwk727HRL
z%3GZ`8qR*xrDuCxpDVys#dEuBLvFc~Vay&_&Xf7?5k?CaJTvrP`vj9eQ<caqIimcJ
znF3Mj9zT3L+Z9Dl(yYe;`nK^asD?p`zF>g^?wGNzx3uE?(}&)U?dkBnZdXR2jsAKK
z3Gycfkx(;z>@#_oydK&Pz<Ra3E1i`Y<}4D{lDfML%O9Viy6k*iTs3ZkNsC{9)LXUw
z?bP4we0*PmWG^&lb<ifVUG>^e#}2q1=a)qw0x(f%SL{P9%?@IJ;30eR=@~T~s*i0g
zd-*<n0~G}n49M*tAH930<pSVETEHav)uVm+je-;C*&J+V+3?)qeUOirP1n`(@77n-
z2<BOub02cL#J;GjRnnv>&#}x*bJbWJ75URG<U5hw*ZpcwDmbcq!NaF%%wqrAbg+tB
zP3Sxb)7h&NZYx^T(dKgmIb`oeA&@zG9tG?#&6i)-#Gt`f1_#ZnDo>SkQkh<IyrJ;+
z@G;|QcKpHa1|QjTE3iv)hJE)X&h>KM9;PgnMtzC0-_31Wr6c>fe|PlBM=Aj>g_i^1
zjAplXSDG+A2^Y5?^KUcL!?O1ULf+j;8cjRH+oXF*@+0;t&s}{=10|Itm@1N*U`*#_
zJ$cewgg^f=wdCjS>jpsUyIN{N3`Af<Qjj1h1nlY$-|>Cp%Y{^c3|iYuy$1Q}O+Xb=
zv&goYGLqK}crEOe#}&g-(IKYlrv#L*tVsQx*DGs7L#jfHSbeMSzjE_kUB4abDf#ht
zgilfH<Kw;qsgG@$_vDMVM~hKnX1f5KcfL#PF~Gvee6E5+XM_~r35)Ej4}aWGTDIGW
zqTYx)^kzG1cuRFGMT!m&fEZ8^keKt%jfA~uCNe9j31kKsP|<;=PtF_nV#0bs@2nNS
z)Ki}rUu^FVB<sQ#{G?g7<j<(lQNFkRxa-Ll%9SF$n+;Gw!viIC#NUt(cS|cZCkHT_
z<ZfD}$|<s$Pfe3(*UIkEf%4e%%VM5m5o=M^8@(L<f|Gx(<C=m}ny%LN@dmISsm_ok
z-#RdMO$>13d5&>&OYKKkNCQp;`~1iv%R_(jk?ALTH7lrTx5!4D=a;@B`oF-vcvrT9
z7&w6xCUct}3$C<+pWPMKKzi)r8AYXs;D9I;<bfy%s-g2S>Pw%t)+e!xo`I<Uf!6cH
zB$M}i`}_73^4CpFaiE$KIJ5c4#V>LuSpknS8Q{G@3k?;W5@J@sNN6N{@>5t*A-^>4
zFHUNgkJ;f=!0#11%qg>4kvZ(W*BNa`*Edi`jU}CCxj*edDCK|O@tAMgbR<x9a~|DW
zOLFBWi2&6UqR_ebAsx<em~-};m5+!RfcRhj*_d~17jn6@{a_it!4&IsJ9wc7x(qy=
zW`&qP00Ghe%`NP)LhL5x!?NvmUK<huS#bz`xox;rltc5jy{~ites`~33CL)~ApOr&
zf?Wo~U1C4XLL)#%$w*&b9&YvWXVC2HrEyk1i&Y718ZlKaQVV|{`=I3^g8kg9Ug6t6
zVxdXiR9#o|(C3s(59$2j(!!$kLPIrSE?!Y!ldj)W5o)VDq2~e1Q-A=d`YI1fVVywk
z5B@wDGFmXK28KO|uq>~6cHXT~%LPE6B%Sd5xP!Xo=~MlNMQ1sKCHk{Dilk#)l*sZ|
z_iqPbIj*n&nLSI<iUYOT?<%{?1&fW$60Qd`>RXhkFv}ic)#KC2S8hjk70I4Qg~Q2=
zod>aeX$Hm(B!yF|8acSx>KKuowRGh`-B>co)1JLuXg$O6%hSY`50G4bnaQJuNPddB
zcoVl*FjE=Z$lP2(H>RlBc=Y*o71^QB<;2)6o&`Z<B<#7evhTaG`*$!vKtTr?CQ>e9
zh4;N+KF7}fmJyfTpMT5k^UWd|MthAR9EkB&@NM%y&mGZQjHmzQdW^kwx8=9pDEn|K
z)wCbzhWWv8XVIkbsAyPRRTegdBbGtGG}&p9i;gW(IXw!i+tQT_y%N+{0G+YZpBd88
zS;xIoVl|IY!0in0_jujqJF?OqVDqRMQc8ZOg%-}EPjTK6tceQOwfU?3&M|;@4I}^d
z@cZ^H62Fea9fR2hympi;-INjhUzK@=7f>2mQSZ4yo*!^qHcQcJgG`@3^{(kPAw5hr
zjcgepQBA(2w$ky2A2X<N`4d#61jC+i!3wBP(sB#+yux4P=_C6wncKaboX|+Ws^Ja#
z)r<K1RL$t<feN8G`IIBHD%H>d6#qXZZ-!aTwLy&|K1$YEhhVXFJ8>6RdYHDNrZTdS
z$J!4agQg27wCD({lnK84LSK^)S|=lYUkZAam)x$+hf6%ZUQ?>!;WPFq5_w{k!RcVk
zD?5@x+^3!<tEBbLe9o@@@92ZE`92-Gm=3zFJ7WH)c3N&T&>?-<e^7?6g!-Of^GcUb
zwo`B952AHB(?am$&dCf%atzShc|1u6?J_XAE_xNxYMgpr*;TO(0=*ijDur9`TMjSK
z(8g3u)_gj&660|kC~1(m400Dt-jGkI!NWO**&(evrmV&PBDi5Yo{IPj!<<y9d>!eg
z$N#u=vG3RxI5GqJ7bqFD9pI2C6YJmtPAC#{)xzxfU4~;j%?5QkAkk#Ed|;%Ek<fvU
zXS|;n3ax<p`*ep<mTUDrq}WW-2U{Swt^_N|JW?I$w^WOBfzZ?tX*<>%i@1X+EQiQ~
zMGZZ|UQ9vDnU_k0DfvxG0%5u#R;qeJ5^FA61qwZ+5$^8>934ye^tsKmVv$JGlh=AY
zq6N0(w|6C}Q<q6k+Bv$Y*L!hSALmVI$3v1%YJx^@zN;{<rjs!y=%l_6<G@O0{Egh7
zam}ea!pIX5+*f1{gisEh(8AL>?6Th}<`hGtjslMUz8@Q(6dp$#R0|h;k?{xsxBz_&
z2mkQDO4#=<w8ml1s1^+AqC<qcA%qIPIO}rTtF#S}m8!_OSI((fwUoa%IS`;UM!!!7
zFDmr~+#dlks7m1eZ?*E17-*dFcvjV3e<<`}^-q_y?rjsnC`-o)yUulkofo`h5#i{Z
zsf<y%qTc-%?kkJ?9BqjvBMee+R@2{E$89{Fw~J^QR~*8;@|Vqj0Wx?X#W15^lV_7V
zEG~0DIb;~NfwW4aUV{i;PW}BFZ0~J{(nbH<y`n$_YuWiJhu`g8ABIW9)lW@p8)a`i
z*9jVFd83ZL4A&}e{GDvzKIs{9v$nfP0BZuGyx-{SE$Ew~LkJ1EOw-q*(iX1_@ZwZM
z52m)Mze8X6#;37H_UO?J0FLAhtRVcN+B=9qAdT=4Jw?6!{RXi|j-5n2Nhm$4)xq0u
zyjEeWbEuJU^n+|QqEzIp<Z!PUya1l_f<(~Irt7lx(qA=+pYmyz8gG8Q!Nm=3@>(!Y
zS}sBxvZR-XgEK2`&3Tp1KFVi)yS{k8fryt3YrIP;`2I3T`>4`}=^huejP*JF`F>(q
z+TX?T^`~Eyb=Ik}5MvSp<ca)Ys5wHU98CBSgHD_KWQbnu{rX#mS}ZTbS7pZ}f%<PY
z!c9<qN}O8RbT_W8KSCtl#`#S_40xgwKk~>X(1GlD$4j9}MS;uq`YRx>;ngeB`cG4(
zjal)t()b6ZwF5ReKFchnGAekn^!pViVP{$dN)eDP51zGO0&QFzgvL8@C-IR9y$7V=
zZr{X`$Qgvb>&35}ly_0+R34{;w><WnJk|+61=5oy4ZcUahXqy&nZAj$o~u!&#w3~z
zy9TFbWCTSl62^KYnGq%A{^Hi|H^g_*L!&yvHlCNCGS!Moi9*P*8Y!7*JT7jT!;}n=
zl?IE4#WxaXI{tMLpml}A${Em7YYW<nNitAYGj-@RNAHfAElImr+f!JuP||HU^`r#(
zQbL^3sg%bjCICq$F+M(Ch8ic8&4r}C8@7OrfVo~$kh%F+A5GxEnIc2?Sd6*cPza_F
zN1sUKryMk2=5MAO0`w3sWfPNCz>LBq;f>4AW??be;1!RTdGRz5NWv@F0T(;`CRCO$
z#Mc?+Y!&W)AHD>?XEP}&rJw_!7W@{|{%m1kK~#g)umvlXE$vk|H_OvUof-!$+!h8F
z6X8(S>+8Nxo~Q*cKu99$DM<Rxc6&4{J3D&~N6JuQh}nc-pe=ZSVa>?hn50%=fQ1P+
zHA%qcrYKhMZO)x}R0%sOK3=jZ;DRhPx!HlfXR4~I;Fph}ZcW4@(H6IhKzWFzXp?ez
zLp-(+_8$rP@l2PFUN54Mt^&8RekZ*K*ap~nDnz8D{(Dnpes`A^=H}rQ9CFUG>B<_X
z7d)e&8=1}rHCK>(m53dVUg%7}!P!oPSeVv&&7<KeQk`{Aa-i2MywitZDbqXuQqz7j
z$-Pe(R5uh-`aT2=EWMhV+S=+WOn#HgW@H0()Un(ROUDm<E2E=UN0)<`#Ny;9Ke+m7
z!Vk1Ze)$fE86zw>vFd}o-J3^Tc>=wz-IFykHjQjK+N101H}-4h_$eLjQR=W^tfKM^
zpdVahQfH_5%a>8Wk6T+?|7y={_}jp+nJPVA%gN;1N|l&?13j)V8fJ7wmnHdrxOHt1
zt1p4uv!29p>3lPjIRQsh^W&$H?W%nHzqk(;M-D^ql=%TT$WU^}A4%d!&>~|Kj&y#0
zegHnV`QEs=yVu?8>;L>WT=9J2cSFWeX0%Y7Eix{%pUc89)PS@YkMTOB<75_3RWe%-
zye<31%`L**t^p<)gieu!*GO;K*}>}Yx$S9qdh+J5-nkAp6y0xH8P%t|o&J3PvEfrZ
z^xxWPeM<rHr!4}8lW_*6h+iCE{JJSO1Enn~<uA1j&GtZvggD^uGLe%T|M{weghBHC
z#}B|z#YQ%7G&zlxCH}FhmD)q7xIooSvnk)GlBBAlrB6E)Ea74Bmej{`BH3sEA68EL
zgWnp$-H%eHh>T-nJP=Mp(bE{f$7yQ1Z4M@bje2}}IXXNH9~fc9V>*$Z*?JWdpy%Xp
zfshpkzuD?1#b5HT$jJSxg7U{pWoFaw^2zrLb=oj|w-k-XV-(rF^!`8*mkcZm;9Pw(
zY}Hj%9GRZxqNSzf;^JatjOsD%Xt!~$#=|U+lFXm7zvNS#UnVr#X+JD(w4$z3@H+Zq
zMy6LiKN4j~pY5=4d@XMA2iJH1>zYg3i>g<4UqE0RKRY+~9&p3@<0xFWhWP*u1-xl$
zdKyqxD=RA#6B2&u9gi)1v@-nlW$=1amiX(1_Jx1CtM|XrFoGr35d9)yr^?+_zur*n
zbeiDLsJir4ufj&uC$v!8>@2aLs;vm9+BI?)7R}X1=<<UCOeOPVb7^$n=?eK>-k+z}
z>eav3tG1oqU{DX(pR(C3*Nn{(1QZp9b?y6cS)FnWC;1yg?15HIhkLe*>lWNEfzHu5
z{ykc1>3sI_rwcp+S?W?{@~;DrB@_r8Kj}8ODfp}E^n~h`t-OS$hgi^6s)5T6s?vH2
z@~f5gn0~l~0B2vy@*fG9pj40|*5@q3`zkvihDhKsCgi<fMH)wL9W{PR2K>0sx5mej
zZYm_h+1?s?2Dqfeur>uoT~+bGIJH#n2xw>H7w0=B;#DSL3#~*tP@=80IjRzyb<D}n
zS<jR#{pKS*i$X+jG^kl@-GG}o2Rc7HnG<RDqxe7jsCuI_TAixPYNcJoh%MfYILC}-
zd#-q8=#(h!H*^OVc}~F$qO`tC8pQ8lB`RXk#NiDZVo85mK=Cj;UYTZaO%5z=ap7{b
zR8eAp6A4p{(#K17!Rcc{R#0g6y;}WLHd!U&OOTu{O{{|3SsTf+(^Q}C=RFa)UHuhj
zP`@x<cGLfO3^E-%9M%@eQDKv)#*~kby7CfH8PACP<CFX-^vesNR6$EAq&?Jcx|Fcl
z$7+aDY^Ad0kA3wiGdcAlz`*J7*2i&>$zscZx{o1Uo-rwqIX6CN=KY3}Qt0of5X+Mg
zq)H-f{aX6kxMH9`G#4?(>}}>yQDZURwr;i1X-e-}(ML&bu~Ub{TugFM|9IIAH^#8z
z-J{yeh~hJ>mW{X}wUw;CHvv*+?O)aMD^y6=cwsB!T7KTjHjk)hsWl`uJ{P}-_~vdp
zD;l)8lkquU|AQW-Q4cQIB9F0n?_=L}a~G=lCg>!c48E8(J*~BF4ZoM#FU^f|QD9lJ
z%ijA^uK(IjUU!_YFM8i$ZnR8)Q;@bqFt;=gRV`{L=FOt!6{(>6s;wk{A_rj^1rVgD
zi+}<_3V{-**9V^dqV99h?O|rAfxJfF!LP+J0a|4SYR*ndi^>S%cnEm-l(d1GUvHM|
zAL_E|b8J;CDY&RBH-fn;cYa8C7w}PKQ~gFZt(+(#?!i;B<dWTiiU-qwW(-cHkCONp
z{uR_7K4GWFdni9lCul-$xj!fA$SA&0h1L%FamelT#QY$RSx5drHJh6G*5sv9`t*5t
zKV$oLR_!$XjR78An`k3OKLap98@Rsti5NMp4If6Da{uPQhxfzl=CLWDxZlzczN~7p
zWcGa%(8D#vQ;UZw#{iNypCZ*&Hgu=o5FVBSmdgd3XTGszx`&jkT!FoW4MHa}fk<`%
zUJMk7L5K&6LPbt2oDB7aFV*CbB2n|iht}J;t4X-M?a7&AI9`^87K*P5Id19nMf2_s
z!RcF*YF`VI&t!EJ440#Nn|R4c1)3=oLuDfM#4#+s`6gVRkdkeT;l7f#FH8aj6>2E{
zCkz?x|K{ATJzXfW)il*tSRXW-BDS;Mz82FECN63oQFG5IzP3zJv-DFE_t$aa@8e|y
z5Y-}<=1<VUf<>@Gf|x4oV`Aq!->6UG>qHn91o`%0ZRKQ`33!p9kJHUn5~4&vAt}li
z<Nko`%g~WC*Q`*b<lle`2?4hr8rKT!R$8lDpn7IdbqzMWPi5n{_f{D|-|n4nQEmt|
zU8?)vwr+ftxcx+@3IszKT(K?SpdlJc8U%O8=`Y`IGl!2c7Y>UH)0N3sztUzq_lK_(
z-LqP5EOzn8C|S5N`LmJjKdx|?P4}m(PDI~h{ec;GS-mO?K<3=C(25J3l8Qc}K1h$v
zmCObB>_HQ}m%gTnk2~X&e7qb*r6-4og$>lM&}<u;7^f#s&xoX0537<XMg-l~BvY8y
z?}Cjoak@-X&ar^)6H<Hq4b6eGndB(=GdR_Sa%-+pf65MLeeN|hXfwc4;oXQ#J@ry)
zIY6O|cr8)G9&#+9Rm3f^W3poM6l7wm6q>M#$<mVgIgfA;xN0C8mL<|x4LZNNhPi5W
z{XlhEFze>%w5g@xP}VYXlt^ILXBSre`oP6TQWfmu>3Fq%GC|)BiAo-JKF=-9<Zc)E
zzj^;~>vog8yYg{+vrCAP^2^5-g8@%~wi<zPdBQ!ts3mGma%=Ta_!AJxxPb-6$Ka=b
ztuj-VnzW4XTlxA7>gCg<KRvQGG-UT2@`TnsS+2v#{pB<3cAM&8bI_0*r)snWlc_x)
zftL;N(lHsFJyShVEybvRnOfQ+KXu^hjH8ychMy=@?f$I)i6Ci_uJU0jbhUS-2ybSe
zEiE$~5<?ehnc9wF(MO0p;fYUIKu7qpI$j*$CLz9ovm~QabFQ%l4X64u$JtGPdIT^r
zavF~%rA7E(j(sovs6{^&J8X24LOV^xD8`4+W0g-hePxag8s?bB|7JmH$*5bTrU~ll
zdrKP!-~{nf>hF<nU^_X{6&;0CMSLDc0~4+8(1@lyR&pt*qyQ1Ge)7oRqmZ<6ulagd
zVNfAup7bkaRyBx74Bj_-w!5LL>{31sLVV9f!8YRzY`<pN8cP|{12btTaWFZaiUxA>
zto`TEz=0hUTyVL3xJzMfe{On`vvt(=0NNJjGlg*S=8A@yAA0pgBmSE<XL%-8I@@<R
z&vT|~*c>oGzmNjUAZJ_si-y{J>psMk)h~tEbvvWkTRlZhDOUv9>ZHiI`NO*s#^B~Y
zCbukqoJy6r%ejwD6u7EI{{O&rNG`mYoWqk+&<6H7LtKah#O`{PNVA$h;8(~}GPgbX
z-#toDoGT0heM?u=t3XWkSFypUCa0{8<}AM>YxD6{EvLhJ2{pw7f3Z|w=ezd0)!Vnf
zS`a4`1LdFK#3{HxR=?ENeLRM<m@xu=T<G^yF`Aar4z#n+e;A=^eUVrn)X%ObbuQX`
zF(q33m0FdKKo`4o;km4CCublhdUh`SrB*k!?jBuHAUdlMK9)kN5GE3}Ea7^glc9FY
z7s%U~O8X6O5Z5jVG^j~I=p4|NR5N~bajSXRYD2Uo88xQv{siZQv##jPI>vkdj$HjL
zT^Q&6F<-o#I)*N&t=w#Tx!rat^ScI6Oz<ET#E~&m0Z(!4Jga|AC($QbHiLAmZkiWo
z*dA46F5i<L$ybEmH+}p;xG)f$A0yHEsq!wzHz=xZyMG=Uzfo9xoLI|JH6=AlQ~4d|
z7Yh|%#kT8}FrqE`M*p$s*FwNDGT%`ecKpn7*df*?{PTX@5PvL>$<V8}fy@8G>sMTz
zhy<R|xe{UAyr8nZ4pZ!mV0V!ys=<B}d8LnU4Z5BIvK^gBtSY8y&-bC~0`XU<c=)Fl
zzUw>dw2C(4Yj*}=u{V7sDq9c=S!gF>J7$Onzhj2;DMq5XHli{%&&i7Y@^HewgN_55
zp?Dw`a|kCnIEO=QkqA}M!{96)Ye<v50`vl!?k`#&=4jM;@+kro^XE20RY@3iO={Li
za$)u6$D@9B%>wbR-#VQFVXPZZdHUg?^Hc)6)L7?qoj|{cw$^KAGT)6^N_05`;fV0W
z!?Pj4Lp<P?uU7o(*=;1zu@ZrU3Loi0W3Lw9&t0|?Epe;%<}Y@}4;2P|*-&z7dI<5^
zzqt9pjw<}1EV!J*2kBt7BxTrO7Y31S#BRC*^Lm{<i`1{zu~#~!bYgrKz|t85>_2lo
z-O%MPg;}iEo_auv1P38&oOmJVRnQB_*m%fgk$Zk<+h5H>RTbbk-**hxJP?%mjHn4t
z;7Mrs>l8_j5st29^y2@e#pf9<h{2{og3^Er)d|t^{L@i-XN4`Pt?B*l7S3$?Pde_u
zBdeuiFS#=n7vPm00zLLqbzJ7D?!vjc_(Cc~rN`p{K~}C8bRG!QfKHRMdE#&Pzvq;h
zi0W64bCeHEQCLU){Ej_kpE}d6-${8ruQW~yz&!tXnOr`q#c5uaZQLQ<!Ftxrn}n#%
zvmFdiyd|RQRLsXGCUTD|uX6VuI4@^uGD;wI!~uSYUPxWu=ilhj;^MpS{PI}i8$D;L
zW7cdOkEG)}QB}Zru9JmUNz84Xgs8}_Yc9aN`6XX~rSrUA^bi|zbk#uPKRYx_m%}J#
z(<d7LHcYWO&1V*Cx!X(BEe9|kb&6tr(~mK<+lm%1hl_`ySm_`Q@!ww>xDV(*hFYd-
z^x2iDHBgRmE-~Dl&#{_~tU>tkZ(y3}KU$|{Bh{p|7iD=2A`f<0NmHD{f|=>p*C?Ug
z%gJ%L=G!hN&P+c&?B;C^eo`akW7kUMxOV;g*;hiRoE^wKYwb;B-)&3P?t9jdfzQ%^
zyKU2(;kS@OOZPhp>P!;oXeq0rMC|+{+K{?CDj&-W4$LDkk}^D@*ULtflt5lHEk|+N
zlejl4GdI8~yl#UM<JIb#v*z|D4B-Iu6*0q`f^s_co6V)u4<@W1R71xzTer%Bk3w^R
z&l+VYKoiB{0}5ki!2L-|QpT&0n-&o+8d>>yHefV@#E6z?u=pkYsjINKTy;NvlkwlE
z+4VXbaI#F}zr#4f#gd}~EU33XqSQqVHQMnyUf}Q>3+K4db~08PLhUDURbBg~i=N^Q
z%1@=deEM{i$S&BZik35EtumgoUslt#VD;Pwzp^I0qxRex0{r(~z$xXBr~Qo@qV*%c
z&wq^!!`BWR!V?mhB`Jb<SFjnRlq8dr)1J1HkaPrjCGW1^8N{h4`F#1VPft(Sxv;+e
zA)@h9>8wtVR&3A!bR#kp2j6}!2jP3MU~EMH`b-@$YCmMrvEyA?#&v%zG3~xj9pZM9
z(D{JQw?sDSw}-_(+)G!;s({$bAteljI$5WU7BIUP=?kMBb4Rudj-{0{MwBKrQug5;
zUel>wwYcs%`3G7$En(Y*x)TXR8jQwWkcu=H=7UC?GYd6LIH(e77PI0MkpR-`1#DD|
z`@7qn()yOp$T68r$KH|6ElC4eN&G}s&#j#1j*l1*DsL2U;|%tam)Pp7OPPO&2MV=b
zr@epGkFo@(kw1sO?VOvBg#f@>vke^$q^D_NA=w-(%|e0oUbVOR->CER2~ie_1?6>B
z(Z@J|h(<Uo)`HW}=@=wA8Iwpb-shELtQYFFTNSk28|!5bUI-_sQ{w>aR5$bP*NO8m
z_X#dU##Pw<{>2ok+Sz6rx%9vBi{3E+eXwo3LztndgWiGc-IgRNJp7g)Mp_JB%>}>y
zO~|vPWBR$~l|>GuJ{U^Fmx%zwMwOG#FQ~u02aCH{n?3j8vkE<AE2(YvKqk9iIy_Qa
zgJ3(u<bn57&<T3G>tv5yBK`%oCPaB7zIPJUWahpxJQtM*(Xq;FkY&<*JuQeAV<(SY
z5<gA-l0V6BfX}sxZ#_YtKqq#j5BDaH_CGTlQHq)-7~uN%&aMr<*aNm@sn3FDv+FB|
zVPmQZ$($0*r_yp%f!dwyNPNC`w4gxZ@pnA0)gh;B#7NbyBK-~&|6oeC`TW<v8cqgA
zND7q<Jr)k-4<#mk+pM~2darf_3DyKEJ~Xi}Kz5dxiY`B2>Re2Pg^2-TR!neBM5k|H
z?P9$Xz*9mlsi~!w9sdiXx<LJSDz@?T=?u;SngzxGGSt4GT)QjupvarS5d~XH`AG-S
zA9bX%ArJ04LW{oVjJ8w9R~vDRJkqs~v5)Cn9IrGLdGi!^HS!YT(mArt-`rC3=*;l)
zB)#nPU!Ttg$ipw>Vx43!*gppZj2!hN?-eyGUK1tD+IKkAPR$$6P21ms{!mP9_?0^;
z(w91YXm(#TvghJ^D`pF1T4Dg~0E|H5C2_?E+|*s_7UcfQ?XwVU$!by60U|Hq^n3_V
zL@v`j$~u!{4J0kPMMU5)DS>H}|AEU*M7=BR5?=C>8?Q`X703zj)J_E&+LIcs#0`f~
z$_!E09cS}Qa+jrr&g@YRg8oU*4nj;bUv4$BQ|F$X_5z0B`$g;~^3L=UHOfB(W(;1c
zY8j^?s_jr|cZsYU^ok;U)CgRJ)%PtZW6Q>(WLMO-w^3<!4%d?OASxJmg_q0;nxvv-
z_lI=M-)H;J@lmPfTfr)qB5f@>jkzdQS{n%?;o<aQx5TdTmN<3LCpBKc{6Vy<Q-n;Q
z4V(>?H*gSsu*MpJ!S*grrN--DF*SjYm-w%T0XaTt-2upA_tEFuM_0Q0wD<hi5vA?O
z<|8Vxtvi`IyKioQDqMl}DDuz#$NEl|W^=B87q!MC6jzk|ndl@p<fnd|*>z#$AQAl^
z-*MbnnyKzpaSmYuS;s5D`}Y^zwSqFjYA>EyD*0hoY>h;lYr4$Ob&KaoIV~k^vrE69
z&{u2x0o$={W(#{R+m>DgWITLE)Om<$dtAFq#0Aey#Gmkzy29GZ*{iUnuzFzs$Dw|t
zg!u3idY5U@fgZi^E9C{;XupGP>;vDcss8BpGfRvL`OL78+TnG~WzEngij}F32xm)!
zqnr0jpa5Q%^WAHOp!_YH9@g=^H`SIq(5=1)BaQDaPVW{s!pH|f8Sa04d|_#3LL=bD
z|K!_L)<w1U@3-n|rv9xg9(aw1LS&=NDr`HFR)No>{Pr+N9kDU=OV>)oq0d_&VrgmJ
zJvtTcw=#lerTGsMJ<RA~b=)!`ACzu^f&jbbVgN5@3mSAkd&_!l9l`nCW8>hFMjON=
z9ZhG8Koq7^>)M0OMkydN0m}f{N%^x!S10nC+=gMLLSiW~8+vDzd~1=Mu$Z7JzN&cj
z6@X&}M4ugLFNQ@?^BOH2yUxaV&F$H8yX1GIweAN$9vg;i2%V1jy&2^G%>40U=2$Bb
zIY{GqjNtu>IN};o><B|k)V>Z3K^RsbCEHsyE<c50*5})zr>0@(W$bPvNh%U)fMJBI
ztzL90`6mKOcQARCR`hLJxoJNcH%Z-@5fwz9AnKpI+y&56_Vs2bH6Dz0p0V>nUKlz0
z^GKf~rh21M+<>hQ7isP#B6I6>vOuh&i%b%13fF0xM5ox-!5$;bJ{78`mJ+wv`<%56
zu=GKt3Z^eH-f{er1&GUXpHnVeus4bHN|xc;=t{`u>Fdi(H+V66U?l)L|Ek5qcz7!O
zz1P<@l6G5_ab<WoA<TXnCH|`sCclkTp9z+S7WL3Cpegm{x&eip@W)87w&b<3VX380
zLC8Teukn(j#Z+bCaQl7$<1YUJ$J+q&=CBmAN10kcF%dnV7hG^RF4`INRpowOm55>$
zo`9oC1db;8XH~JV50g|r)$aEq9$g|LYSZ^|R|q9&rXr+4<fq`%{xkZG-CY<yi!T&j
z^NI;b%XgC6Mr4;>uSZi_QgOh4fzp%z?l@$Cx2?~@{+<Te(e});-<r*nj}B-KTpmC?
zT~g`UR)Kc{Z9w>Q(>zyHk1$!e<&3-`Hyqd+xs$(o@bLlvJ=o<q_$a|wVyp=g`V1lD
z;hFz?K0%ThIbZa@bCz3*RS2|G|8sJ|#pyqgVJ~NK>F5~SIs0ej*pW%0b+->2uWKAU
zd%_6P@J&<ieM{gJglDZ2WrvheNB#Y#l1U_7oKxc0hmeU@a!HjmMiqHJPK!fvbVZ9Z
z9EKl${@Dlug*B-E<3%RZ@|!sO=tp@Zei43IL=oh^WmfeKnC1}@j>R7F21b2&!$ZZZ
zZ3tT!{a#Hh>2)v>RYZqHPNq2q+A&W7c5+nha+hrgY+kAuiL<s_>zN_xZS?^5aCcm$
z2ce%*^fV2jBMG>APXJBV0iroj*-DHlrQu+!aR^+bRY@e-nOL|SH4c8YJQ~Xl3F(Q^
zk(*VrMT5GnS2`A=iX;s^;g@Cns>^QP#fp~-qBiGjmbQ2mTgpwg^#Avv#F27-Rjz7I
z?sD+B-V6g_s)41SHpRbd$YPU`xI5R4cToK8W*S23;QOG2-TdQ6no&$CvXZj@_B|w_
zYvc9{_0+wIHahJ8&Ic8vYAgR5!FFPNhL*KEPn;<>mZn!ad15byhumr^DiaxE4p9lx
z|0(Ru<DuNU04^pm*2t2C7|cwCCX^-al)*%pY}chy=vJ0)mh9UIV^4_4nq@4>MY=+a
z?b?!Ly{<J%5tFi1gZB)#`);4l`_5m#&*xd@{FZZ`^PK1WzGr#YKBuX5T*67WE5TM-
zzbZx|d+I0B6S|1V_F#|G19?2(Z}(~~M*yjF<ovtON^(!K_NYss&xY;J$q9d>IX=l^
zhZFqtNbv}pW%XQZ84meDrSY6c&NVbP=V~Vr;}!VavRuLWdpA=cm0cgr%IVNt;T`JG
zb2vTCMSrg@1(_;e78^(M!^H75jb(f^jB=n&#Z?WZj&V}6sS+;bF)R9=ooHOWB-$!j
z6^%t)1+cw+bji*2cj>F$`nDClT<r<#`ue2(;zj<>zg|^%5R_z*B9WY&Y{ea7Y}8X+
zP0d(zH{@vRBDTiDM(?N0HH*`{$Lg2Ri8hRS9^&$DdM)yS2_lzDA?iKkaKYa*K_fkh
zbrl_~SQU|^1*=CT&3;!V&d<~A%s9w>yFsPnv#D8ApRwgZ?FJlOKHd%YY7r^YMHQyp
zjp7LjC(E;qq6@RB6OL)}2fhrq1Y1OiXln?_nSUP#<{J8(>ltuaxy*=i;+%7x$_+7f
zf?4J#0%`x=Qa4v}#@F-mKN5M<?La?7hp+stPvt^FSyf}$4w)EhllS@iPTl{J^fPH#
zlk<f3D|#PdJv`0XDKk8;rF;n&`A{-E<9?R0!U#Y%{?bMWE79L1>iO85CmtEc%dpDL
z$zeZ+&u(IDU7ONPGpT<h7^uMa*zNH(w2OQ{!T;g4oxAaw9u3^o^WebEVo8wh(*1>Y
zPM_uj)TSnPTkdGoE|Z`Z<JdtN^VQA%_XezNv2os9#nQ>w48Qj_rFBoz(%)tPOg4mL
z@b+AFS%w~qw2KdZUu#Cix8(YbRXkyL6Fxlbzvjh%BAv+++P0iP8)&<WGJ01z{OOtF
z%SKsLBo+1MlSEyB#?7fjD=y)8R>Sh;-qRDzTps3YCj1ls@KQBZndiCOi<A?dAIp!u
zsj95K<jMp>ZJ9ZU!%&>EDIG)YcC_*&F3XYuC~$4t-+dOa*i3V|C0|r7)$m-c@I#vu
zb_-+?pT*Z!gv59i=DQud<3l}Asq)Z0{oSYZ@FccfeARx7q~kK-xRNmD%IqKcExDlX
zc9<;oSB1dG2Vg_`?VQxdwl60xHigHEubX^2<HWmtE<XZ)n_bs+F!OHqa2e^r&t2A8
zj#v3SVAmwG7YMwVD3L^)&eP!Y*%<Y0fB$%3EEp5&*Upq9REmSFth9$DJ1w-R=Cia~
z8dx%1*)JoVK@&R73q&z20?(ft3@O|k`9GJKj=r3UbE%0H=QDUXY6W4tt%I;TAn4Q&
zLybbkzKs4k#zWu27S^p~EDo}iKAUMFRn%tE(XMm1UF-6G`U|!*`~P_`p;eG@xs3d7
zFNQF4XBlNg)bVi<l?lK5ds2xdA51SlBHj_^pDabO3EAq&v>1~b!67{es}cUZntiJS
zZ7N%$=#)ZO8cud)nw<Z>G!;uoviiCMj^OQii?O5#M^SF}5Lz3sZJ3StIjG_V4Vgi3
z)1jeT8p5oW1GO_Cph$y8HE$#`1RAdUp+XKcrq$8dts8j!zZ<+2Bqb%@m1g#%3o1Lb
zq~=gM{-g<QNfK*mOZm}sTYez*{SW#7U)M33tpmJq=WET_xeqy^$2G`O9k3E7y&dhV
zep1WHexUbd%%NN3=$y!-v9_ahUd&?kj+ZK`1-18UKaTarE@kB13a;jv$(!NvH$B+l
zYXRA+NGLNKrs<Wh5a6~X_P9V14VP5k{8qZ=Fs%}^a?iV@6ca%7eqN_%j9IjTxFkjs
z-o?l{0`<|g>;wYLO&g?-a^p!67j(}<Dkw~$J-8CExSJJa9sv>px?7GQ$bz>&l^hRK
z+6J>~1^RaYjs4i!1~YGOVn#Ux;h2^A;nx!rw?AK%mX^Msp8g_TLtdiNXBp7uHa0fR
z71}X1HQxZ$+WcA!4LltLf#Ch(EO_EzDUor?IvxNUZEMrl%3PeC^_m-aI45amXP0pQ
zKKD~H^2j`ZGH=|tL5~QwY8J&Q8JL?Vi4K?m(hh{^@VyIfmxTbN@XD1h>g_%2?EC>h
z-U)<R>eABE{QUgN!YCJfoS|POSzeF+@EEEWHAe{2*(xh5OUx$eulolF!*y4Hh^xdI
z;$OtU7>XcV6HwdT51eyJO)nPz72s^P<QcoIE}ZlA^*wc}>49FH#PMH#(9_dnj0T7;
zRaiE)6;OeJ%$|v`wjMdH%?SwDz;7#u88KdOyZ5KulBBGqZ5|R)u#pljBr?b9sxk?m
zq2-^Nnu>IN3YcI>w!J1>1W4tR2FU!tmHB(Wp?0qg1JfTRCs$NhNJGt@mg3JXC~))_
z9*Px=>O;9o7kSJL)CH`IRVf0q5vZvE3ZI&uMz~!Z;Ta!=L1r)*zuCCx>r>WzHv%&>
zz0VdKozo)htUvCqp58QB>-F)Z&7C$N*<MsYJPT?E-f@HcfrAGP7&5o@^R(NG;M;~h
z5grOK2DbUp(a~>TWHzB7q?Pl>lk1t))oKj0_4V}@*@%WiL#}u|f;+bqRx6V)&Et|w
zx{P+s25>faiiPo%=F54cJ#!o^ot?d6a(qZE5gz|ldb+y0Q@cH4{35SGeh#ETI5;@i
zUb;b)Zf$L)YXHEORZ<l8yU^qh$jODJSifbiw%(K+vIUEJ(a_M3fL&tBK|Z52yDNa!
z@R=?sY^nXGEnWtX#{<iDTDSIH1X8ue!os4ZrA0zQVybtOEpS!oUKf>$zI4GNL8Koj
zqj>Gm&`>oswH~2|Ru#IMcFXT!D4JHgFa7H7?oNR(S4=I5k;4w9Jguo|dUX5o<7L1K
zXPpG*{LItpYFbqox0C&o>*Gd;`h%b>6Fyc_n>RH)GX=Pu({DHfx_hqa=;+wg`6N-5
zPmu6^eSNkgCm+0=s6Z+Bb2~w~X~1mv{6w=)5D`M$LM&h~m?KI`LG`xH{r$z0ib-8v
zhO>%~F4P-AHWJe|#Ai?^2Af)1cB(3PYPXJ!WzcLJBsb`3r?_Qwb~Zsq5v%Ma7nk$J
zb#GM+1^N47CH_F7i?BF(0{NLD&A?2|&vVzi!|>)#q~%g4*aXGIko!`B`Wt0XYzaHs
zj;X2b_2F7_%j0Ik)klCV+}Y9b!^pm4E~rR4v8<w^c!-gjmXL|qBLtc7pkz>EEG0di
z2TPV?5fB&WE+sq1c=0cMEv}<Vfog=7?g_X8sW?(L9`i#-hPd+z3Iq#C{5wcV$;t9^
za_1S!)=yl7;;2^Br_JbZF4PC`@W}ezZBu3K;u+a5DJi+u&JWrO6Vy`}RsSI__~~@)
zY^sh6VP`NOi=TjP2-o=WFT|iybs3r0P@|w(l!JoQ;-o*xS}UM$_N^@O!3TT;u}`8x
zvS4H=NHLDt(-&W{1!`RXqmmN~?Do#*&qL1~b)rNZ;pb1%M`1CTU<C`@oj^3Gl`NLS
z>E=B{ImQlm#@dY>*0jqKajR1;Q2={q%ETJXQl&R_{hM85V<W{sL?uo$3i&KKD2Qdv
zjelc9W@DqLzdy8FAo5Off0Hpc2-S(+z%aCv70wAjcbS?G?|Zk^Dc4@N#sF&iz=rM^
z(597zfw?Nq8-85mo?*hD%e#UXXq}y0RXA9}N}R5&*J+nm0ma_TLcz^{4QaSO0LeWK
zztf3Bb-u^^1pXWyoqQ7Ey3D%p6$1WSkAgnD8ULE8`oV)-0W-Eyev^%%VCg=8c3+xM
zi_gav@$6Ft0CZgEjGVlDQ~vP4fUwqO&=as(B5@{SB)plq`B5A$>moUh_T|&t$)8yh
znmwL@iD#}vrua`Iu7Pz{P*5oIh8(b)9v-3N>Q@v|Px@}>W3e2{#yux#p8j8!X25z$
z#CgInw!29s>6xXeI=i~Ml2^pE*!kDY0h}E&uJv&~4o`x#(op!Ca_YTY5fl_?tP)l2
z0nHp99`^S3270-)F)OO~7U8k)stdrI*A^y@vX5L5i_wj*Tw>0<ywQCr1*)_bXOGKS
zEl3=OlI3*$Q)VC`EaP#&;6rz0H~aGJo5Fa?A|N;F7iVKav2|)RRm!@`Df`vCvK;~(
zh%z2^kN1gQt(&iI5q4U?g&GuY)qj?k<AxSy=jKd}Y)vId_=#k+mX<`o9y*Z{6O*JF
zL+02v6TSRNe)*UP<i?(!moGt;gD+?vKK$^7v5HG$4tajGM&u7nyguSN!rN7KZ=Qe<
z<G|Y2<+-V;OKYeGy{S(JXGnjKta=nu)fhO?tougmd_5uUG4Zg9JwTtKBO8oGsA&WS
z+!--63oQP@M;A8z#&OECfQJsV>M4mb@Q7gm@i_AqXl;Z1MX;=yfdZh+?3uTI8b#Oj
zet%|r2%A43Xc79aKz4zO(O*jvs_-zK1vD0i#_HS!uNJ~B4BQg9GJBcI1knI^{s-*!
z=824i%GU9+|LH&6@w{xYD^A>;y^<#yGhWm6uf7DoECk*Ez80UDD6e>-&}Bs7h33ni
zoZ7eS)7>GzCnhHxt90i<FIt_}9PG`mt*rex`^8|}yV-IpWiZ$t!EJX@@!vho)DA++
z<S*#gOg~dc?k|5cOu7g(6vlYl4OP^SM1DpI3n^P9-hkV2NJB#V_uEdf2yNe2MoUB}
txzm|FM#w=N&LOuUh@{=$)_=2E*kqAV>Wv<iEYD+sACX|BQ+m>t{13q)j6eVY
literal 0
HcmV?d00001
diff --git a/doc/guides/rel_notes/release_20_05.rst b/doc/guides/rel_notes/release_20_05.rst
index 000bbf501..f404ee7fd 100644
--- a/doc/guides/rel_notes/release_20_05.rst
+++ b/doc/guides/rel_notes/release_20_05.rst
@@ -62,6 +62,11 @@ New Features
* Added support for matching on IPv4 Time To Live and IPv6 Hop Limit.
+* **Updated the Intel ice driver.**
+
+ Updated the Intel ice driver with new features and improvements, including:
+
+ * Added support for DCF (Device Config Function) feature.
Removed Items
-------------
diff --git a/drivers/common/Makefile b/drivers/common/Makefile
index 96bd7ac6e..df2e840cf 100644
--- a/drivers/common/Makefile
+++ b/drivers/common/Makefile
@@ -31,6 +31,7 @@ DIRS-y += dpaax
endif
IAVF-y := $(CONFIG_RTE_LIBRTE_IAVF_PMD)
+IAVF-y += $(CONFIG_RTE_LIBRTE_ICE_PMD)
ifneq (,$(findstring y,$(IAVF-y)))
DIRS-y += iavf
endif
diff --git a/drivers/net/ice/Makefile b/drivers/net/ice/Makefile
index e22c34287..f493c9ed7 100644
--- a/drivers/net/ice/Makefile
+++ b/drivers/net/ice/Makefile
@@ -11,9 +11,11 @@ LIB = librte_pmd_ice.a
CFLAGS += -O3
CFLAGS += $(WERROR_FLAGS)
CFLAGS += -DALLOW_EXPERIMENTAL_API
+CFLAGS += -I$(RTE_SDK)/drivers/common/iavf
LDLIBS += -lrte_eal -lrte_mbuf -lrte_ethdev -lrte_kvargs
LDLIBS += -lrte_bus_pci -lrte_mempool -lrte_hash
+LDLIBS += -lrte_net -lrte_common_iavf
EXPORT_MAP := rte_pmd_ice_version.map
@@ -84,6 +86,9 @@ ifeq ($(CC_AVX2_SUPPORT), 1)
endif
SRCS-$(CONFIG_RTE_LIBRTE_ICE_PMD) += ice_generic_flow.c
+SRCS-$(CONFIG_RTE_LIBRTE_ICE_PMD) += ice_dcf.c
+SRCS-$(CONFIG_RTE_LIBRTE_ICE_PMD) += ice_dcf_ethdev.c
+
# install this header file
SYMLINK-$(CONFIG_RTE_LIBRTE_ICE_PMD)-include := rte_pmd_ice.h
diff --git a/drivers/net/ice/ice_dcf.c b/drivers/net/ice/ice_dcf.c
new file mode 100644
index 000000000..0a99cceb1
--- /dev/null
+++ b/drivers/net/ice/ice_dcf.c
@@ -0,0 +1,474 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#include <sys/queue.h>
+#include <stdio.h>
+#include <errno.h>
+#include <stdint.h>
+#include <string.h>
+#include <unistd.h>
+#include <stdarg.h>
+#include <inttypes.h>
+#include <rte_byteorder.h>
+#include <rte_common.h>
+
+#include <rte_pci.h>
+#include <rte_atomic.h>
+#include <rte_eal.h>
+#include <rte_ether.h>
+#include <rte_ethdev_driver.h>
+#include <rte_ethdev_pci.h>
+#include <rte_malloc.h>
+#include <rte_memzone.h>
+#include <rte_dev.h>
+
+#include "ice_dcf.h"
+
+#define ICE_DCF_AQ_LEN 32
+#define ICE_DCF_AQ_BUF_SZ 4096
+
+#define ICE_DCF_ARQ_MAX_RETRIES 200
+#define ICE_DCF_ARQ_CHECK_TIME 2 /* msecs */
+
+#define ICE_DCF_VF_RES_BUF_SZ \
+ (sizeof(struct virtchnl_vf_resource) + \
+ IAVF_MAX_VF_VSI * sizeof(struct virtchnl_vsi_resource))
+
+static __rte_always_inline int
+ice_dcf_send_cmd_req_no_irq(struct ice_dcf_hw *hw, enum virtchnl_ops op,
+ uint8_t *req_msg, uint16_t req_msglen)
+{
+ return iavf_aq_send_msg_to_pf(&hw->avf, op, IAVF_SUCCESS,
+ req_msg, req_msglen, NULL);
+}
+
+static int
+ice_dcf_recv_cmd_rsp_no_irq(struct ice_dcf_hw *hw, enum virtchnl_ops op,
+ uint8_t *rsp_msgbuf, uint16_t rsp_buflen,
+ uint16_t *rsp_msglen)
+{
+ struct iavf_arq_event_info event;
+ enum virtchnl_ops v_op;
+ int i = 0;
+ int err;
+
+ event.buf_len = rsp_buflen;
+ event.msg_buf = rsp_msgbuf;
+
+ do {
+ err = iavf_clean_arq_element(&hw->avf, &event, NULL);
+ if (err != IAVF_SUCCESS)
+ goto again;
+
+ v_op = rte_le_to_cpu_32(event.desc.cookie_high);
+ if (v_op != op)
+ goto again;
+
+ if (rsp_msglen != NULL)
+ *rsp_msglen = event.msg_len;
+ return rte_le_to_cpu_32(event.desc.cookie_low);
+
+again:
+ rte_delay_ms(ICE_DCF_ARQ_CHECK_TIME);
+ } while (i++ < ICE_DCF_ARQ_MAX_RETRIES);
+
+ return -EIO;
+}
+
+static __rte_always_inline void
+ice_dcf_aq_cmd_clear(struct ice_dcf_hw *hw, struct dcf_virtchnl_cmd *cmd)
+{
+ rte_spinlock_lock(&hw->vc_cmd_queue_lock);
+
+ TAILQ_REMOVE(&hw->vc_cmd_queue, cmd, next);
+
+ rte_spinlock_unlock(&hw->vc_cmd_queue_lock);
+}
+
+static __rte_always_inline void
+ice_dcf_vc_cmd_set(struct ice_dcf_hw *hw, struct dcf_virtchnl_cmd *cmd)
+{
+ cmd->v_ret = IAVF_ERR_NOT_READY;
+ cmd->rsp_msglen = 0;
+ cmd->pending = 1;
+
+ rte_spinlock_lock(&hw->vc_cmd_queue_lock);
+
+ TAILQ_INSERT_TAIL(&hw->vc_cmd_queue, cmd, next);
+
+ rte_spinlock_unlock(&hw->vc_cmd_queue_lock);
+}
+
+static __rte_always_inline int
+ice_dcf_vc_cmd_send(struct ice_dcf_hw *hw, struct dcf_virtchnl_cmd *cmd)
+{
+ return iavf_aq_send_msg_to_pf(&hw->avf,
+ cmd->v_op, IAVF_SUCCESS,
+ cmd->req_msg, cmd->req_msglen, NULL);
+}
+
+static __rte_always_inline void
+ice_dcf_aq_cmd_handle(struct ice_dcf_hw *hw, struct iavf_arq_event_info *info)
+{
+ struct dcf_virtchnl_cmd *cmd;
+ enum virtchnl_ops v_op;
+ enum iavf_status v_ret;
+ uint16_t aq_op;
+
+ aq_op = rte_le_to_cpu_16(info->desc.opcode);
+ if (unlikely(aq_op != iavf_aqc_opc_send_msg_to_vf)) {
+ PMD_DRV_LOG(ERR,
+ "Request %u is not supported yet", aq_op);
+ return;
+ }
+
+ v_op = rte_le_to_cpu_32(info->desc.cookie_high);
+ if (unlikely(v_op == VIRTCHNL_OP_EVENT))
+ return;
+
+ v_ret = rte_le_to_cpu_32(info->desc.cookie_low);
+
+ rte_spinlock_lock(&hw->vc_cmd_queue_lock);
+
+ TAILQ_FOREACH(cmd, &hw->vc_cmd_queue, next) {
+ if (cmd->v_op == v_op && cmd->pending) {
+ cmd->v_ret = v_ret;
+ cmd->rsp_msglen = RTE_MIN(info->msg_len,
+ cmd->rsp_buflen);
+ if (likely(cmd->rsp_msglen != 0))
+ rte_memcpy(cmd->rsp_msgbuf, info->msg_buf,
+ cmd->rsp_msglen);
+
+ /* prevent compiler reordering */
+ rte_compiler_barrier();
+ cmd->pending = 0;
+ break;
+ }
+ }
+
+ rte_spinlock_unlock(&hw->vc_cmd_queue_lock);
+}
+
+static void
+ice_dcf_handle_virtchnl_msg(struct ice_dcf_hw *hw)
+{
+ struct iavf_arq_event_info info;
+ uint16_t pending = 1;
+ int ret;
+
+ info.buf_len = ICE_DCF_AQ_BUF_SZ;
+ info.msg_buf = hw->arq_buf;
+
+ while (pending) {
+ ret = iavf_clean_arq_element(&hw->avf, &info, &pending);
+ if (ret != IAVF_SUCCESS)
+ break;
+
+ ice_dcf_aq_cmd_handle(hw, &info);
+ }
+}
+
+static int
+ice_dcf_init_check_api_version(struct ice_dcf_hw *hw)
+{
+#define ICE_CPF_VIRTCHNL_VERSION_MAJOR_START 1
+#define ICE_CPF_VIRTCHNL_VERSION_MINOR_START 1
+ struct virtchnl_version_info version, *pver;
+ int err;
+
+ version.major = VIRTCHNL_VERSION_MAJOR;
+ version.minor = VIRTCHNL_VERSION_MINOR;
+ err = ice_dcf_send_cmd_req_no_irq(hw, VIRTCHNL_OP_VERSION,
+ (uint8_t *)&version, sizeof(version));
+ if (err) {
+ PMD_INIT_LOG(ERR, "Failed to send OP_VERSION");
+ return err;
+ }
+
+ pver = &hw->virtchnl_version;
+ err = ice_dcf_recv_cmd_rsp_no_irq(hw, VIRTCHNL_OP_VERSION,
+ (uint8_t *)pver, sizeof(*pver), NULL);
+ if (err) {
+ PMD_INIT_LOG(ERR, "Failed to get response of OP_VERSION");
+ return -1;
+ }
+
+ PMD_INIT_LOG(DEBUG,
+ "Peer PF API version: %u.%u", pver->major, pver->minor);
+
+ if (pver->major < ICE_CPF_VIRTCHNL_VERSION_MAJOR_START ||
+ (pver->major == ICE_CPF_VIRTCHNL_VERSION_MAJOR_START &&
+ pver->minor < ICE_CPF_VIRTCHNL_VERSION_MINOR_START)) {
+ PMD_INIT_LOG(ERR,
+ "VIRTCHNL API version should not be lower than (%u.%u)",
+ ICE_CPF_VIRTCHNL_VERSION_MAJOR_START,
+ ICE_CPF_VIRTCHNL_VERSION_MAJOR_START);
+ return -1;
+ } else if (pver->major > VIRTCHNL_VERSION_MAJOR ||
+ (pver->major == VIRTCHNL_VERSION_MAJOR &&
+ pver->minor > VIRTCHNL_VERSION_MINOR)) {
+ PMD_INIT_LOG(ERR,
+ "PF/VF API version mismatch:(%u.%u)-(%u.%u)",
+ pver->major, pver->minor,
+ VIRTCHNL_VERSION_MAJOR, VIRTCHNL_VERSION_MINOR);
+ return -1;
+ }
+
+ PMD_INIT_LOG(DEBUG, "Peer is supported PF host");
+
+ return 0;
+}
+
+static int
+ice_dcf_get_vf_resource(struct ice_dcf_hw *hw)
+{
+ uint32_t caps;
+ int err, i;
+
+ caps = VIRTCHNL_VF_OFFLOAD_WB_ON_ITR | VIRTCHNL_VF_OFFLOAD_RX_POLLING |
+ VIRTCHNL_VF_CAP_ADV_LINK_SPEED |
+ VF_BASE_MODE_OFFLOADS;
+
+ err = ice_dcf_send_cmd_req_no_irq(hw, VIRTCHNL_OP_GET_VF_RESOURCES,
+ (uint8_t *)&caps, sizeof(caps));
+ if (err) {
+ PMD_DRV_LOG(ERR, "Failed to send msg OP_GET_VF_RESOURCE");
+ return err;
+ }
+
+ err = ice_dcf_recv_cmd_rsp_no_irq(hw, VIRTCHNL_OP_GET_VF_RESOURCES,
+ (uint8_t *)hw->vf_res,
+ ICE_DCF_VF_RES_BUF_SZ, NULL);
+ if (err) {
+ PMD_DRV_LOG(ERR, "Failed to get response of OP_GET_VF_RESOURCE");
+ return -1;
+ }
+
+ iavf_vf_parse_hw_config(&hw->avf, hw->vf_res);
+
+ hw->vsi_res = NULL;
+ for (i = 0; i < hw->vf_res->num_vsis; i++) {
+ if (hw->vf_res->vsi_res[i].vsi_type == VIRTCHNL_VSI_SRIOV)
+ hw->vsi_res = &hw->vf_res->vsi_res[i];
+ }
+
+ if (!hw->vsi_res) {
+ PMD_DRV_LOG(ERR, "no LAN VSI found");
+ return -1;
+ }
+
+ hw->vsi_id = hw->vsi_res->vsi_id;
+ PMD_DRV_LOG(DEBUG, "VSI ID is %u", hw->vsi_id);
+
+ return 0;
+}
+
+static int
+ice_dcf_check_reset_done(struct ice_dcf_hw *hw)
+{
+#define ICE_DCF_RESET_WAIT_CNT 50
+ struct iavf_hw *avf = &hw->avf;
+ int i, reset;
+
+ for (i = 0; i < ICE_DCF_RESET_WAIT_CNT; i++) {
+ reset = IAVF_READ_REG(avf, IAVF_VFGEN_RSTAT) &
+ IAVF_VFGEN_RSTAT_VFR_STATE_MASK;
+ reset = reset >> IAVF_VFGEN_RSTAT_VFR_STATE_SHIFT;
+
+ if (reset == VIRTCHNL_VFR_VFACTIVE ||
+ reset == VIRTCHNL_VFR_COMPLETED)
+ break;
+
+ rte_delay_ms(20);
+ }
+
+ if (i >= ICE_DCF_RESET_WAIT_CNT)
+ return -1;
+
+ return 0;
+}
+
+static inline void
+ice_dcf_enable_irq0(struct ice_dcf_hw *hw)
+{
+ struct iavf_hw *avf = &hw->avf;
+
+ /* Enable admin queue interrupt trigger */
+ IAVF_WRITE_REG(avf, IAVF_VFINT_ICR0_ENA1,
+ IAVF_VFINT_ICR0_ENA1_ADMINQ_MASK);
+ IAVF_WRITE_REG(avf, IAVF_VFINT_DYN_CTL01,
+ IAVF_VFINT_DYN_CTL01_INTENA_MASK |
+ IAVF_VFINT_DYN_CTL01_CLEARPBA_MASK |
+ IAVF_VFINT_DYN_CTL01_ITR_INDX_MASK);
+
+ IAVF_WRITE_FLUSH(avf);
+}
+
+static inline void
+ice_dcf_disable_irq0(struct ice_dcf_hw *hw)
+{
+ struct iavf_hw *avf = &hw->avf;
+
+ /* Disable all interrupt types */
+ IAVF_WRITE_REG(avf, IAVF_VFINT_ICR0_ENA1, 0);
+ IAVF_WRITE_REG(avf, IAVF_VFINT_DYN_CTL01,
+ IAVF_VFINT_DYN_CTL01_ITR_INDX_MASK);
+
+ IAVF_WRITE_FLUSH(avf);
+}
+
+static void
+ice_dcf_dev_interrupt_handler(void *param)
+{
+ struct ice_dcf_hw *hw = param;
+
+ ice_dcf_disable_irq0(hw);
+
+ ice_dcf_handle_virtchnl_msg(hw);
+
+ ice_dcf_enable_irq0(hw);
+}
+
+int
+ice_dcf_execute_virtchnl_cmd(struct ice_dcf_hw *hw,
+ struct dcf_virtchnl_cmd *cmd)
+{
+ int i = 0;
+ int err;
+
+ if ((cmd->req_msg && !cmd->req_msglen) ||
+ (!cmd->req_msg && cmd->req_msglen) ||
+ (cmd->rsp_msgbuf && !cmd->rsp_buflen) ||
+ (!cmd->rsp_msgbuf && cmd->rsp_buflen))
+ return -EINVAL;
+
+ rte_spinlock_lock(&hw->vc_cmd_send_lock);
+ ice_dcf_vc_cmd_set(hw, cmd);
+
+ err = ice_dcf_vc_cmd_send(hw, cmd);
+ if (err) {
+ PMD_DRV_LOG(ERR, "fail to send cmd %d", cmd->v_op);
+ goto ret;
+ }
+
+ do {
+ if (!cmd->pending)
+ break;
+
+ rte_delay_ms(ICE_DCF_ARQ_CHECK_TIME);
+ } while (i++ < ICE_DCF_ARQ_MAX_RETRIES);
+
+ if (cmd->v_ret != IAVF_SUCCESS) {
+ err = -1;
+ PMD_DRV_LOG(ERR,
+ "No response (%d times) or return failure (%d) for cmd %d",
+ i, cmd->v_ret, cmd->v_op);
+ }
+
+ret:
+ ice_dcf_aq_cmd_clear(hw, cmd);
+ rte_spinlock_unlock(&hw->vc_cmd_send_lock);
+ return err;
+}
+
+int
+ice_dcf_init_hw(struct rte_eth_dev *eth_dev, struct ice_dcf_hw *hw)
+{
+ struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ int ret;
+
+ hw->avf.hw_addr = pci_dev->mem_resource[0].addr;
+ hw->avf.back = hw;
+
+ hw->avf.bus.bus_id = pci_dev->addr.bus;
+ hw->avf.bus.device = pci_dev->addr.devid;
+ hw->avf.bus.func = pci_dev->addr.function;
+
+ hw->avf.device_id = pci_dev->id.device_id;
+ hw->avf.vendor_id = pci_dev->id.vendor_id;
+ hw->avf.subsystem_device_id = pci_dev->id.subsystem_device_id;
+ hw->avf.subsystem_vendor_id = pci_dev->id.subsystem_vendor_id;
+
+ hw->avf.aq.num_arq_entries = ICE_DCF_AQ_LEN;
+ hw->avf.aq.num_asq_entries = ICE_DCF_AQ_LEN;
+ hw->avf.aq.arq_buf_size = ICE_DCF_AQ_BUF_SZ;
+ hw->avf.aq.asq_buf_size = ICE_DCF_AQ_BUF_SZ;
+
+ rte_spinlock_init(&hw->vc_cmd_send_lock);
+ rte_spinlock_init(&hw->vc_cmd_queue_lock);
+ TAILQ_INIT(&hw->vc_cmd_queue);
+
+ hw->arq_buf = rte_zmalloc("arq_buf", ICE_DCF_AQ_BUF_SZ, 0);
+ if (hw->arq_buf == NULL) {
+ PMD_INIT_LOG(ERR, "unable to allocate AdminQ buffer memory");
+ goto err;
+ }
+
+ ret = iavf_set_mac_type(&hw->avf);
+ if (ret) {
+ PMD_INIT_LOG(ERR, "set_mac_type failed: %d", ret);
+ goto err;
+ }
+
+ ret = ice_dcf_check_reset_done(hw);
+ if (ret) {
+ PMD_INIT_LOG(ERR, "VF is still resetting");
+ goto err;
+ }
+
+ ret = iavf_init_adminq(&hw->avf);
+ if (ret) {
+ PMD_INIT_LOG(ERR, "init_adminq failed: %d", ret);
+ goto err;
+ }
+
+ if (ice_dcf_init_check_api_version(hw)) {
+ PMD_INIT_LOG(ERR, "check_api version failed");
+ goto err_api;
+ }
+
+ hw->vf_res = rte_zmalloc("vf_res", ICE_DCF_VF_RES_BUF_SZ, 0);
+ if (hw->vf_res == NULL) {
+ PMD_INIT_LOG(ERR, "unable to allocate vf_res memory");
+ goto err_api;
+ }
+
+ if (ice_dcf_get_vf_resource(hw)) {
+ PMD_INIT_LOG(ERR, "Failed to get VF resource");
+ goto err_alloc;
+ }
+
+ rte_intr_callback_register(&pci_dev->intr_handle,
+ ice_dcf_dev_interrupt_handler, hw);
+ rte_intr_enable(&pci_dev->intr_handle);
+ ice_dcf_enable_irq0(hw);
+
+ return 0;
+
+err_alloc:
+ rte_free(hw->vf_res);
+err_api:
+ iavf_shutdown_adminq(&hw->avf);
+err:
+ rte_free(hw->arq_buf);
+
+ return -1;
+}
+
+void
+ice_dcf_uninit_hw(struct rte_eth_dev *eth_dev, struct ice_dcf_hw *hw)
+{
+ struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_intr_handle *intr_handle = &pci_dev->intr_handle;
+
+ ice_dcf_disable_irq0(hw);
+ rte_intr_disable(intr_handle);
+ rte_intr_callback_unregister(intr_handle,
+ ice_dcf_dev_interrupt_handler, hw);
+
+ iavf_shutdown_adminq(&hw->avf);
+
+ rte_free(hw->arq_buf);
+ rte_free(hw->vf_res);
+}
diff --git a/drivers/net/ice/ice_dcf.h b/drivers/net/ice/ice_dcf.h
new file mode 100644
index 000000000..f44c09db2
--- /dev/null
+++ b/drivers/net/ice/ice_dcf.h
@@ -0,0 +1,52 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#ifndef _ICE_DCF_H_
+#define _ICE_DCF_H_
+
+#include <rte_ethdev_driver.h>
+
+#include <iavf_prototype.h>
+#include <iavf_adminq_cmd.h>
+#include <iavf_type.h>
+
+#include "ice_logs.h"
+
+struct dcf_virtchnl_cmd {
+ TAILQ_ENTRY(dcf_virtchnl_cmd) next;
+
+ enum virtchnl_ops v_op;
+ enum iavf_status v_ret;
+
+ uint16_t req_msglen;
+ uint8_t *req_msg;
+
+ uint16_t rsp_msglen;
+ uint16_t rsp_buflen;
+ uint8_t *rsp_msgbuf;
+
+ volatile int pending;
+};
+
+struct ice_dcf_hw {
+ struct iavf_hw avf;
+
+ rte_spinlock_t vc_cmd_send_lock;
+ rte_spinlock_t vc_cmd_queue_lock;
+ TAILQ_HEAD(, dcf_virtchnl_cmd) vc_cmd_queue;
+ uint8_t *arq_buf;
+
+ struct virtchnl_version_info virtchnl_version;
+ struct virtchnl_vf_resource *vf_res; /* VF resource */
+ struct virtchnl_vsi_resource *vsi_res; /* LAN VSI */
+ uint16_t vsi_id;
+};
+
+int ice_dcf_execute_virtchnl_cmd(struct ice_dcf_hw *hw,
+ struct dcf_virtchnl_cmd *cmd);
+
+int ice_dcf_init_hw(struct rte_eth_dev *eth_dev, struct ice_dcf_hw *hw);
+void ice_dcf_uninit_hw(struct rte_eth_dev *eth_dev, struct ice_dcf_hw *hw);
+
+#endif /* _ICE_DCF_H_ */
diff --git a/drivers/net/ice/ice_dcf_ethdev.c b/drivers/net/ice/ice_dcf_ethdev.c
new file mode 100644
index 000000000..23f82a487
--- /dev/null
+++ b/drivers/net/ice/ice_dcf_ethdev.c
@@ -0,0 +1,317 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#include <errno.h>
+#include <stdbool.h>
+#include <sys/types.h>
+#include <sys/ioctl.h>
+#include <unistd.h>
+
+#include <rte_interrupts.h>
+#include <rte_debug.h>
+#include <rte_pci.h>
+#include <rte_atomic.h>
+#include <rte_eal.h>
+#include <rte_ether.h>
+#include <rte_ethdev_pci.h>
+#include <rte_kvargs.h>
+#include <rte_malloc.h>
+#include <rte_memzone.h>
+#include <rte_dev.h>
+
+#include <iavf_devids.h>
+
+#include "ice_generic_flow.h"
+#include "ice_dcf_ethdev.h"
+
+static uint16_t
+ice_dcf_recv_pkts(__rte_unused void *rx_queue,
+ __rte_unused struct rte_mbuf **bufs,
+ __rte_unused uint16_t nb_pkts)
+{
+ return 0;
+}
+
+static uint16_t
+ice_dcf_xmit_pkts(__rte_unused void *tx_queue,
+ __rte_unused struct rte_mbuf **bufs,
+ __rte_unused uint16_t nb_pkts)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_start(struct rte_eth_dev *dev)
+{
+ dev->data->dev_link.link_status = ETH_LINK_UP;
+
+ return 0;
+}
+
+static void
+ice_dcf_dev_stop(struct rte_eth_dev *dev)
+{
+ dev->data->dev_link.link_status = ETH_LINK_DOWN;
+}
+
+static int
+ice_dcf_dev_configure(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_info_get(struct rte_eth_dev *dev,
+ struct rte_eth_dev_info *dev_info)
+{
+ struct ice_dcf_adapter *adapter = dev->data->dev_private;
+
+ dev_info->max_mac_addrs = 1;
+ dev_info->max_rx_pktlen = (uint32_t)-1;
+ dev_info->max_rx_queues = RTE_DIM(adapter->rxqs);
+ dev_info->max_tx_queues = RTE_DIM(adapter->txqs);
+
+ return 0;
+}
+
+static int
+ice_dcf_stats_get(__rte_unused struct rte_eth_dev *dev,
+ __rte_unused struct rte_eth_stats *igb_stats)
+{
+ return 0;
+}
+
+static int
+ice_dcf_stats_reset(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_promiscuous_enable(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_promiscuous_disable(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_allmulticast_enable(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_allmulticast_disable(__rte_unused struct rte_eth_dev *dev)
+{
+ return 0;
+}
+
+static int
+ice_dcf_dev_filter_ctrl(struct rte_eth_dev *dev,
+ enum rte_filter_type filter_type,
+ __rte_unused enum rte_filter_op filter_op,
+ __rte_unused void *arg)
+{
+ int ret = 0;
+
+ if (!dev)
+ return -EINVAL;
+
+ switch (filter_type) {
+ default:
+ PMD_DRV_LOG(WARNING, "Filter type (%d) not supported",
+ filter_type);
+ ret = -EINVAL;
+ break;
+ }
+
+ return ret;
+}
+
+static void
+ice_dcf_dev_close(struct rte_eth_dev *dev)
+{
+ struct ice_dcf_adapter *adapter = dev->data->dev_private;
+
+ if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+ return;
+
+ dev->dev_ops = NULL;
+ dev->rx_pkt_burst = NULL;
+ dev->tx_pkt_burst = NULL;
+ dev->data->mac_addrs = NULL;
+
+ ice_dcf_uninit_hw(dev, &adapter->real_hw);
+}
+
+static void
+ice_dcf_queue_release(__rte_unused void *q)
+{
+}
+
+static int
+ice_dcf_link_update(__rte_unused struct rte_eth_dev *dev,
+ __rte_unused int wait_to_complete)
+{
+ return 0;
+}
+
+static int
+ice_dcf_rx_queue_setup(struct rte_eth_dev *dev,
+ uint16_t rx_queue_id,
+ __rte_unused uint16_t nb_rx_desc,
+ __rte_unused unsigned int socket_id,
+ __rte_unused const struct rte_eth_rxconf *rx_conf,
+ __rte_unused struct rte_mempool *mb_pool)
+{
+ struct ice_dcf_adapter *adapter = dev->data->dev_private;
+
+ dev->data->rx_queues[rx_queue_id] = &adapter->rxqs[rx_queue_id];
+
+ return 0;
+}
+
+static int
+ice_dcf_tx_queue_setup(struct rte_eth_dev *dev,
+ uint16_t tx_queue_id,
+ __rte_unused uint16_t nb_tx_desc,
+ __rte_unused unsigned int socket_id,
+ __rte_unused const struct rte_eth_txconf *tx_conf)
+{
+ struct ice_dcf_adapter *adapter = dev->data->dev_private;
+
+ dev->data->tx_queues[tx_queue_id] = &adapter->txqs[tx_queue_id];
+
+ return 0;
+}
+
+static const struct eth_dev_ops ice_dcf_eth_dev_ops = {
+ .dev_start = ice_dcf_dev_start,
+ .dev_stop = ice_dcf_dev_stop,
+ .dev_close = ice_dcf_dev_close,
+ .dev_configure = ice_dcf_dev_configure,
+ .dev_infos_get = ice_dcf_dev_info_get,
+ .rx_queue_setup = ice_dcf_rx_queue_setup,
+ .tx_queue_setup = ice_dcf_tx_queue_setup,
+ .rx_queue_release = ice_dcf_queue_release,
+ .tx_queue_release = ice_dcf_queue_release,
+ .link_update = ice_dcf_link_update,
+ .stats_get = ice_dcf_stats_get,
+ .stats_reset = ice_dcf_stats_reset,
+ .promiscuous_enable = ice_dcf_dev_promiscuous_enable,
+ .promiscuous_disable = ice_dcf_dev_promiscuous_disable,
+ .allmulticast_enable = ice_dcf_dev_allmulticast_enable,
+ .allmulticast_disable = ice_dcf_dev_allmulticast_disable,
+ .filter_ctrl = ice_dcf_dev_filter_ctrl,
+};
+
+static int
+ice_dcf_dev_init(struct rte_eth_dev *eth_dev)
+{
+ struct ice_dcf_adapter *adapter = eth_dev->data->dev_private;
+
+ eth_dev->dev_ops = &ice_dcf_eth_dev_ops;
+ eth_dev->rx_pkt_burst = ice_dcf_recv_pkts;
+ eth_dev->tx_pkt_burst = ice_dcf_xmit_pkts;
+
+ if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+ return 0;
+
+ eth_dev->data->dev_flags |= RTE_ETH_DEV_CLOSE_REMOVE;
+
+ if (ice_dcf_init_hw(eth_dev, &adapter->real_hw) != 0) {
+ PMD_INIT_LOG(ERR, "Failed to init DCF hardware");
+ return -1;
+ }
+
+ rte_eth_random_addr(adapter->mac_addr.addr_bytes);
+ eth_dev->data->mac_addrs = &adapter->mac_addr;
+
+ return 0;
+}
+
+static int
+ice_dcf_dev_uninit(struct rte_eth_dev *eth_dev)
+{
+ ice_dcf_dev_close(eth_dev);
+
+ return 0;
+}
+
+static int
+ice_dcf_cap_check_handler(__rte_unused const char *key,
+ const char *value, __rte_unused void *opaque)
+{
+ if (strcmp(value, "dcf"))
+ return -1;
+
+ return 0;
+}
+
+static int
+ice_dcf_cap_selected(struct rte_devargs *devargs)
+{
+ struct rte_kvargs *kvlist;
+ const char *key = "cap";
+ int ret = 0;
+
+ if (devargs == NULL)
+ return 0;
+
+ kvlist = rte_kvargs_parse(devargs->args, NULL);
+ if (kvlist == NULL)
+ return 0;
+
+ if (!rte_kvargs_count(kvlist, key))
+ goto exit;
+
+ /* dcf capability selected when there's a key-value pair: cap=dcf */
+ if (rte_kvargs_process(kvlist, key,
+ ice_dcf_cap_check_handler, NULL) < 0)
+ goto exit;
+
+ ret = 1;
+
+exit:
+ rte_kvargs_free(kvlist);
+ return ret;
+}
+
+static int eth_ice_dcf_pci_probe(__rte_unused struct rte_pci_driver *pci_drv,
+ struct rte_pci_device *pci_dev)
+{
+ if (!ice_dcf_cap_selected(pci_dev->device.devargs))
+ return 1;
+
+ return rte_eth_dev_pci_generic_probe(pci_dev,
+ sizeof(struct ice_dcf_adapter),
+ ice_dcf_dev_init);
+}
+
+static int eth_ice_dcf_pci_remove(struct rte_pci_device *pci_dev)
+{
+ return rte_eth_dev_pci_generic_remove(pci_dev, ice_dcf_dev_uninit);
+}
+
+static const struct rte_pci_id pci_id_ice_dcf_map[] = {
+ { RTE_PCI_DEVICE(IAVF_INTEL_VENDOR_ID, IAVF_DEV_ID_ADAPTIVE_VF) },
+ { .vendor_id = 0, /* sentinel */ },
+};
+
+static struct rte_pci_driver rte_ice_dcf_pmd = {
+ .id_table = pci_id_ice_dcf_map,
+ .drv_flags = RTE_PCI_DRV_NEED_MAPPING,
+ .probe = eth_ice_dcf_pci_probe,
+ .remove = eth_ice_dcf_pci_remove,
+};
+
+RTE_PMD_REGISTER_PCI(net_ice_dcf, rte_ice_dcf_pmd);
+RTE_PMD_REGISTER_PCI_TABLE(net_ice_dcf, pci_id_ice_dcf_map);
+RTE_PMD_REGISTER_KMOD_DEP(net_ice_dcf, "* igb_uio | vfio-pci");
+RTE_PMD_REGISTER_PARAM_STRING(net_ice_dcf, "cap=dcf");
diff --git a/drivers/net/ice/ice_dcf_ethdev.h b/drivers/net/ice/ice_dcf_ethdev.h
new file mode 100644
index 000000000..0c34a0095
--- /dev/null
+++ b/drivers/net/ice/ice_dcf_ethdev.h
@@ -0,0 +1,24 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#ifndef _ICE_DCF_ETHDEV_H_
+#define _ICE_DCF_ETHDEV_H_
+
+#include "ice_ethdev.h"
+#include "ice_dcf.h"
+
+#define ICE_DCF_MAX_RINGS 1
+
+struct ice_dcf_queue {
+ uint64_t dummy;
+};
+
+struct ice_dcf_adapter {
+ struct ice_dcf_hw real_hw;
+ struct rte_ether_addr mac_addr;
+ struct ice_dcf_queue rxqs[ICE_DCF_MAX_RINGS];
+ struct ice_dcf_queue txqs[ICE_DCF_MAX_RINGS];
+};
+
+#endif /* _ICE_DCF_ETHDEV_H_ */
diff --git a/drivers/net/ice/meson.build b/drivers/net/ice/meson.build
index f9e897bbc..0ba9668d1 100644
--- a/drivers/net/ice/meson.build
+++ b/drivers/net/ice/meson.build
@@ -15,8 +15,8 @@ sources = files(
'ice_hash.c'
)
-deps += ['hash']
-includes += include_directories('base')
+deps += ['hash', 'net', 'common_iavf']
+includes += include_directories('base', '../../common/iavf')
if arch_subdir == 'x86'
sources += files('ice_rxtx_vec_sse.c')
@@ -37,4 +37,7 @@ if arch_subdir == 'x86'
endif
endif
+sources += files('ice_dcf.c',
+ 'ice_dcf_ethdev.c')
+
install_headers('rte_pmd_ice.h')
diff --git a/mk/rte.app.mk b/mk/rte.app.mk
index d295ca0a5..f3798a09f 100644
--- a/mk/rte.app.mk
+++ b/mk/rte.app.mk
@@ -185,6 +185,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_I40E_PMD) += -lrte_pmd_i40e
_LDLIBS-$(CONFIG_RTE_LIBRTE_IAVF_PMD) += -lrte_pmd_iavf
_LDLIBS-$(CONFIG_RTE_LIBRTE_ICE_PMD) += -lrte_pmd_ice
IAVF-y := $(CONFIG_RTE_LIBRTE_IAVF_PMD)
+IAVF-y += $(CONFIG_RTE_LIBRTE_ICE_PMD)
ifeq ($(findstring y,$(IAVF-y)),y)
_LDLIBS-y += -lrte_common_iavf
endif
--
2.26.0
^ permalink raw reply [relevance 1%]
* Re: [dpdk-dev] [PATCH v2] eal/cpuflags: add x86 based cpu flags
@ 2020-03-27 12:24 3% ` David Marchand
0 siblings, 0 replies; 200+ results
From: David Marchand @ 2020-03-27 12:24 UTC (permalink / raw)
To: Kevin Laatz
Cc: dev, Bruce Richardson, Van Haaren Harry, Neil Horman,
Thomas Monjalon, Honnappa Nagarahalli, Dodji Seketeli
On Wed, Mar 25, 2020 at 12:11 PM Kevin Laatz <kevin.laatz@intel.com> wrote:
>
> This patch adds CPU flags which will enable the detection of ISA
> features available on more recent x86 based CPUs.
>
> The CPUID leaf information can be found in Section 1.7 of this
> document:
> https://software.intel.com/sites/default/files/managed/c5/15/architecture-instruction-set-extensions-programming-reference.pdf
>
> The following CPU flags are added in this patch:
> - AVX-512 doubleword and quadword instructions.
> - AVX-512 integer fused multiply-add instructions.
> - AVX-512 conflict detection instructions.
> - AVX-512 byte and word instructions.
> - AVX-512 vector length instructions.
> - AVX-512 vector bit manipulation instructions.
> - AVX-512 vector bit manipulation 2 instructions.
> - Galois field new instructions.
> - Vector AES instructions.
> - Vector carry-less multiply instructions.
> - AVX-512 vector neural network instructions.
> - AVX-512 for bit algorithm instructions.
> - AVX-512 vector popcount instructions.
> - Cache line demote instructions.
> - Direct store instructions.
> - Direct store 64B instructions.
> - AVX-512 two register intersection instructions.
>
> Signed-off-by: Kevin Laatz <kevin.laatz@intel.com>
> ---
> lib/librte_eal/common/arch/x86/rte_cpuflags.c | 18 ++++++++++++++++++
> .../common/include/arch/x86/rte_cpuflags.h | 18 ++++++++++++++++++
> 2 files changed, 36 insertions(+)
>
> diff --git a/lib/librte_eal/common/arch/x86/rte_cpuflags.c b/lib/librte_eal/common/arch/x86/rte_cpuflags.c
> index 6492df556..30439e795 100644
> --- a/lib/librte_eal/common/arch/x86/rte_cpuflags.c
> +++ b/lib/librte_eal/common/arch/x86/rte_cpuflags.c
> @@ -120,6 +120,24 @@ const struct feature_entry rte_cpu_feature_table[] = {
> FEAT_DEF(EM64T, 0x80000001, 0, RTE_REG_EDX, 29)
>
> FEAT_DEF(INVTSC, 0x80000007, 0, RTE_REG_EDX, 8)
> +
> + FEAT_DEF(AVX512DQ, 0x00000007, 0, RTE_REG_EBX, 17)
> + FEAT_DEF(AVX512IFMA, 0x00000007, 0, RTE_REG_EBX, 21)
> + FEAT_DEF(AVX512CD, 0x00000007, 0, RTE_REG_EBX, 28)
> + FEAT_DEF(AVX512BW, 0x00000007, 0, RTE_REG_EBX, 30)
> + FEAT_DEF(AVX512VL, 0x00000007, 0, RTE_REG_EBX, 31)
> + FEAT_DEF(AVX512VBMI, 0x00000007, 0, RTE_REG_ECX, 1)
> + FEAT_DEF(AVX512VBMI2, 0x00000007, 0, RTE_REG_ECX, 6)
> + FEAT_DEF(GFNI, 0x00000007, 0, RTE_REG_ECX, 8)
> + FEAT_DEF(VAES, 0x00000007, 0, RTE_REG_ECX, 9)
> + FEAT_DEF(VPCLMULQDQ, 0x00000007, 0, RTE_REG_ECX, 10)
> + FEAT_DEF(AVX512VNNI, 0x00000007, 0, RTE_REG_ECX, 11)
> + FEAT_DEF(AVX512BITALG, 0x00000007, 0, RTE_REG_ECX, 12)
> + FEAT_DEF(AVX512VPOPCNTDQ, 0x00000007, 0, RTE_REG_ECX, 14)
> + FEAT_DEF(CLDEMOTE, 0x00000007, 0, RTE_REG_ECX, 25)
> + FEAT_DEF(MOVDIRI, 0x00000007, 0, RTE_REG_ECX, 27)
> + FEAT_DEF(MOVDIR64B, 0x00000007, 0, RTE_REG_ECX, 28)
> + FEAT_DEF(AVX512VP2INTERSECT, 0x00000007, 0, RTE_REG_EDX, 8)
> };
>
> int
> diff --git a/lib/librte_eal/common/include/arch/x86/rte_cpuflags.h b/lib/librte_eal/common/include/arch/x86/rte_cpuflags.h
> index 25ba47b96..f8f73b19f 100644
> --- a/lib/librte_eal/common/include/arch/x86/rte_cpuflags.h
> +++ b/lib/librte_eal/common/include/arch/x86/rte_cpuflags.h
> @@ -113,6 +113,24 @@ enum rte_cpu_flag_t {
> /* (EAX 80000007h) EDX features */
> RTE_CPUFLAG_INVTSC, /**< INVTSC */
>
> + RTE_CPUFLAG_AVX512DQ, /**< AVX512 Doubleword and Quadword */
> + RTE_CPUFLAG_AVX512IFMA, /**< AVX512 Integer Fused Multiply-Add */
> + RTE_CPUFLAG_AVX512CD, /**< AVX512 Conflict Detection*/
> + RTE_CPUFLAG_AVX512BW, /**< AVX512 Byte and Word */
> + RTE_CPUFLAG_AVX512VL, /**< AVX512 Vector Length */
> + RTE_CPUFLAG_AVX512VBMI, /**< AVX512 Vector Bit Manipulation */
> + RTE_CPUFLAG_AVX512VBMI2, /**< AVX512 Vector Bit Manipulation 2 */
> + RTE_CPUFLAG_GFNI, /**< Galois Field New Instructions */
> + RTE_CPUFLAG_VAES, /**< Vector AES */
> + RTE_CPUFLAG_VPCLMULQDQ, /**< Vector Carry-less Multiply */
> + RTE_CPUFLAG_AVX512VNNI, /**< AVX512 Vector Neural Network Instructions */
> + RTE_CPUFLAG_AVX512BITALG, /**< AVX512 Bit Algorithms */
> + RTE_CPUFLAG_AVX512VPOPCNTDQ, /**< AVX512 Vector Popcount */
> + RTE_CPUFLAG_CLDEMOTE, /**< Cache Line Demote */
> + RTE_CPUFLAG_MOVDIRI, /**< Direct Store Instructions */
> + RTE_CPUFLAG_MOVDIR64B, /**< Direct Store Instructions 64B */
> + RTE_CPUFLAG_AVX512VP2INTERSECT, /**< AVX512 Two Register Intersection */
> +
> /* The last item */
> RTE_CPUFLAG_NUMFLAGS, /**< This should always be the last! */
This is seen as an ABI break because of the change on _NUMFLAGS:
https://travis-ci.com/github/ovsrobot/dpdk/jobs/302524264#L2351
--
David Marchand
^ permalink raw reply [relevance 3%]
Results 7401-7600 of ~18000 next (newer) | prev (older) | reverse | sort options + mbox downloads above
-- links below jump to the message on this page --
2019-08-05 12:30 [dpdk-dev] [PATCH] doc: remove major in designation of normal releases Thomas Monjalon
2020-02-13 20:51 0% ` Kevin Traynor
2019-11-13 21:43 [dpdk-dev] [PATCH v7 0/1] fbarray: fix duplicated fbarray file in secondary yasufum.o
2019-11-27 8:48 ` [dpdk-dev] [PATCH v8 " Yasufumi Ogawa
2019-11-27 8:48 ` [dpdk-dev] [PATCH v8 1/1] " Yasufumi Ogawa
2019-12-06 10:44 ` Burakov, Anatoly
2020-02-14 7:46 0% ` Yasufumi Ogawa
2020-02-14 15:08 ` David Marchand
2020-02-14 15:29 3% ` Thomas Monjalon
2020-02-17 12:54 0% ` Yasufumi Ogawa
2019-11-26 14:56 [dpdk-dev] [PATCH] service: don't walk out of bounds when checking services Aaron Conole
2019-12-03 21:15 ` [dpdk-dev] [PATCH v2] " Aaron Conole
2019-12-04 8:33 ` David Marchand
2019-12-04 8:34 ` David Marchand
2019-12-20 14:43 ` David Marchand
2020-02-07 12:04 ` Kevin Traynor
2020-02-07 14:27 ` Aaron Conole
2020-02-14 16:38 0% ` Kevin Traynor
2019-12-02 15:35 [dpdk-dev] [PATCH 0/4] Extend --lcores to run on cores > RTE_MAX_LCORE David Marchand
2020-01-21 0:24 ` Thomas Monjalon
2020-02-21 8:04 ` Thomas Monjalon
2020-02-21 8:19 ` Song, Keesang
2020-02-21 9:40 ` David Marchand
2020-02-21 14:48 ` Aaron Conole
2020-02-21 16:38 3% ` Stephen Hemminger
2019-12-20 15:20 [dpdk-dev] [PATCH] add ABI checks David Marchand
2020-02-04 12:44 ` [dpdk-dev] [PATCH v2 4/4] " Trahe, Fiona
2020-02-04 15:52 ` Trahe, Fiona
2020-02-04 15:59 ` Thomas Monjalon
2020-02-04 17:46 ` Trahe, Fiona
2020-02-13 14:51 4% ` Kusztal, ArkadiuszX
2020-03-16 12:57 8% ` Trahe, Fiona
2020-03-16 13:09 4% ` Thomas Monjalon
2020-03-17 13:27 4% ` Kusztal, ArkadiuszX
2020-03-17 15:10 4% ` Thomas Monjalon
2020-03-17 19:10 4% ` Kusztal, ArkadiuszX
2020-01-29 12:29 [dpdk-dev] [RFC] meter: fix ABI break due to experimental tag removal Ferruh Yigit
2020-02-02 18:53 ` Neil Horman
2020-02-03 12:53 ` Ferruh Yigit
2020-02-04 12:02 ` Neil Horman
2020-02-05 10:04 ` Luca Boccassi
2020-02-05 11:32 ` Neil Horman
2020-02-13 17:40 8% ` Ray Kinsella
2020-02-14 2:40 7% ` Neil Horman
2020-02-14 11:36 7% ` Bruce Richardson
2020-02-14 20:48 4% ` Neil Horman
2020-02-14 21:52 4% ` Bruce Richardson
2020-02-15 13:43 4% ` Neil Horman
2020-02-17 14:23 8% ` Ray Kinsella
2020-02-17 15:37 4% ` Neil Horman
2020-01-30 14:20 [dpdk-dev] [PATCH] doc: deprecate using MAX values as array size Ferruh Yigit
2020-02-21 10:25 0% ` Ferruh Yigit
2020-02-24 6:18 0% ` Akhil Goyal
2020-02-25 11:03 0% ` Thomas Monjalon
2020-02-24 8:51 0% ` Andrew Rybchenko
2020-02-25 13:52 0% ` David Marchand
2020-02-25 14:53 0% ` David Marchand
2020-01-30 16:58 [dpdk-dev] [PATCH] doc: deprecation notice to move igb_uio to a new repo jerinj
2020-02-24 7:03 0% ` [dpdk-dev] [dpdk-techboard] " Maxime Coquelin
2020-02-05 15:17 [dpdk-dev] [PATCH] doc: alias to experimental tag for stable apis Ray Kinsella
2020-02-18 10:46 17% ` [dpdk-dev] [PATCH v4] " Ray Kinsella
2020-02-21 17:11 0% ` Ferruh Yigit
2020-02-25 14:22 0% ` Kevin Traynor
2020-02-25 15:35 0% ` [dpdk-dev] [PATCH] " Neil Horman
2020-02-25 17:57 0% ` David Marchand
2020-02-11 23:05 [dpdk-dev] [PATCH 0/7] vfio/pci: SR-IOV support Alex Williamson
2020-02-14 4:57 0% ` Alexey Kardashevskiy
2020-02-14 15:27 0% ` Alex Williamson
2020-02-13 11:20 3% [dpdk-dev] DPDK Release Status Meeting 13/02/2020 Ferruh Yigit
2020-02-14 5:54 [dpdk-dev] Questions about rte_timer APIs Honnappa Nagarahalli
2020-02-25 23:56 3% ` Honnappa Nagarahalli
2020-02-26 16:57 0% ` Carrillo, Erik G
2020-02-27 16:58 0% ` Honnappa Nagarahalli
2020-02-16 21:58 [dpdk-dev] [dpdk-announce] release candidate 20.02-rc3 Thomas Monjalon
2020-02-20 9:41 4% ` Peng, Yuan
2020-02-17 11:35 [dpdk-dev] [PATCH 0/2] improve Travis coverage Thomas Monjalon
2020-02-18 21:20 ` [dpdk-dev] [PATCH v3 " Thomas Monjalon
2020-02-18 22:36 3% ` David Marchand
2020-02-17 13:59 9% [dpdk-dev] [PATCH] ci: build and use libabigail 1.6 David Marchand
2020-02-17 15:15 0% ` Thomas Monjalon
2020-02-17 18:47 0% ` Aaron Conole
2020-02-18 9:40 3% ` David Marchand
2020-02-18 11:18 0% ` David Marchand
2020-02-18 14:55 0% ` Aaron Conole
2020-02-18 15:07 0% ` David Marchand
2020-02-18 14:29 9% ` [dpdk-dev] [PATCH v2] " David Marchand
2020-02-18 15:46 0% ` Thomas Monjalon
2020-02-18 16:38 0% ` David Marchand
2020-02-19 8:13 0% ` Dodji Seketeli
2020-02-18 20:13 9% ` [dpdk-dev] [PATCH v3] " David Marchand
2020-02-18 20:31 0% ` Aaron Conole
2020-02-18 20:59 0% ` David Marchand
2020-02-17 15:38 3% [dpdk-dev] [PATCH] doc: plan splitting the ethdev ops struct Ferruh Yigit
2020-02-18 5:07 0% ` Jerin Jacob
2020-02-25 12:42 0% ` Ferruh Yigit
2020-02-18 6:01 0% ` Stephen Hemminger
2020-02-21 10:40 0% ` Ferruh Yigit
2020-02-25 10:35 0% ` Andrew Rybchenko
2020-02-25 11:07 3% ` Ananyev, Konstantin
2020-02-25 11:19 0% ` Andrew Rybchenko
2020-02-25 12:28 3% ` Ferruh Yigit
2020-02-25 12:44 3% ` [dpdk-dev] [PATCH v2] " Ferruh Yigit
2020-02-25 15:51 3% ` Andrew Rybchenko
2020-02-25 16:13 3% ` Ferruh Yigit
2020-02-25 16:41 0% ` Andrew Rybchenko
2020-02-25 18:13 0% ` David Marchand
2020-02-25 18:18 0% ` Ferruh Yigit
2020-03-04 9:57 4% ` [dpdk-dev] [PATCH v3] " Ferruh Yigit
2020-02-17 16:13 7% [dpdk-dev] [PATCH] net/ionic: fix ABI version Ferruh Yigit
2020-02-19 12:37 4% ` Ferruh Yigit
2020-02-17 23:15 9% [dpdk-dev] ABI version of experimental libraries Thomas Monjalon
2020-02-17 23:44 11% ` [dpdk-dev] [PATCH] build: fix soname for " Thomas Monjalon
2020-02-18 9:40 0% ` Bruce Richardson
2020-02-18 9:47 0% ` Thomas Monjalon
2020-02-18 9:42 7% ` [dpdk-dev] ABI version of " Bruce Richardson
2020-02-18 9:50 4% ` Thomas Monjalon
2020-02-18 10:36 4% ` Kinsella, Ray
2020-02-20 19:50 8% ` Ferruh Yigit
2020-02-20 19:54 10% ` [dpdk-dev] [PATCH] build: fix experimental library versioning Ferruh Yigit
2020-02-20 22:14 0% ` Luca Boccassi
2020-02-21 12:36 0% ` Ray Kinsella
2020-02-21 15:24 5% ` David Marchand
2020-02-21 15:34 3% ` Ferruh Yigit
2020-02-21 16:41 0% ` David Marchand
2020-02-19 11:43 7% ` [dpdk-dev] ABI version of experimental libraries Neil Horman
2020-02-19 12:43 4% ` Thomas Monjalon
2020-02-19 13:50 4% ` Ray Kinsella
2020-02-21 16:57 4% ` Thomas Monjalon
2020-02-24 9:32 4% ` Ray Kinsella
2020-02-19 21:17 4% ` Neil Horman
2020-02-18 10:46 13% [dpdk-dev] [PATCH] ethdev: add comment to warn of ABI breakage Ciara Power
2020-02-18 11:59 4% ` Ferruh Yigit
2020-02-18 13:13 4% ` Andrew Rybchenko
2020-02-18 14:20 4% ` Thomas Monjalon
2020-02-18 13:39 13% ` [dpdk-dev] [PATCH v2] " Ciara Power
2020-02-18 18:40 4% ` Ferruh Yigit
2020-02-19 18:53 2% [dpdk-dev] [PATCH v2 0/7] vfio/pci: SR-IOV support Alex Williamson
2020-02-25 2:33 0% ` Tian, Kevin
2020-03-05 17:33 3% ` Alex Williamson
2020-03-06 9:21 0% ` Tian, Kevin
2020-03-05 6:38 0% ` Vamsi Krishna Attunuru
2020-02-19 19:41 [dpdk-dev] [PATCH 0/4] Reorganise Travis jobs David Marchand
2020-02-19 19:41 3% ` [dpdk-dev] [PATCH 4/4] ci: reorganise " David Marchand
2020-02-19 21:39 0% ` Aaron Conole
2020-02-20 10:42 0% ` Thomas Monjalon
2020-02-20 12:22 0% ` David Marchand
2020-02-20 14:35 0% ` Aaron Conole
2020-02-20 16:01 0% ` David Marchand
2020-02-20 19:38 0% ` [dpdk-dev] [dpdk-ci] " Jeremy Plsek
2020-02-20 0:18 0% ` [dpdk-dev] " dwilder
2020-02-20 11:07 0% ` Thomas Monjalon
2020-02-20 13:18 [dpdk-dev] [PATCH] lib/cmdline_rdline: increase command line buf size Wisam Jaddo
2020-02-20 14:53 ` [dpdk-dev] [PATCH v3] cmdline: increase maximum line length Wisam Jaddo
2020-02-22 15:28 3% ` David Marchand
2020-02-21 16:10 17% [dpdk-dev] [PATCH] devtools: skip experimental libraries in ABI check David Marchand
2020-02-21 16:13 4% ` Ferruh Yigit
2020-02-21 16:30 4% ` Thomas Monjalon
2020-02-21 16:40 4% ` David Marchand
2020-02-22 21:08 [dpdk-dev] [dpdk-announce] release candidate 20.02-rc4 Thomas Monjalon
2020-02-25 9:58 ` Ali Alnubani
2020-02-25 14:37 4% ` Xu, Qian Q
2020-02-25 14:44 0% ` Thomas Monjalon
2020-02-25 15:27 0% ` Raslan Darawsheh
2020-02-23 10:59 8% [dpdk-dev] [PATCH v1] doc: update release notes for 20.02 John McNamara
2020-02-24 11:35 2% [dpdk-dev] [RFC 0/6] New sync modes for ring Konstantin Ananyev
2020-02-24 16:59 0% ` Stephen Hemminger
2020-02-24 17:59 0% ` Jerin Jacob
2020-02-24 19:35 3% ` Stephen Hemminger
2020-02-24 20:52 0% ` Honnappa Nagarahalli
2020-02-25 11:45 0% ` Ananyev, Konstantin
2020-02-25 13:41 3% ` Ananyev, Konstantin
2020-02-27 10:31 0% ` Jerin Jacob
2020-02-28 0:17 0% ` David Christensen
2020-02-25 0:58 0% ` Honnappa Nagarahalli
2020-02-25 15:14 0% ` Ananyev, Konstantin
2020-03-25 20:43 0% ` Honnappa Nagarahalli
2020-03-26 1:50 3% ` Ananyev, Konstantin
2020-02-25 22:59 4% [dpdk-dev] [dpdk-announce] DPDK 20.02 released Thomas Monjalon
2020-02-26 13:38 10% [dpdk-dev] [PATCH] version: 20.05-rc0 David Marchand
2020-02-26 13:55 0% ` Thomas Monjalon
2020-02-26 14:01 0% ` David Marchand
2020-02-27 11:17 3% ` Bruce Richardson
2020-02-27 11:23 3% ` David Marchand
2020-02-27 11:26 4% ` Thomas Monjalon
2020-02-27 11:01 0% ` David Marchand
2020-02-27 11:31 0% ` David Marchand
2020-02-29 10:27 [dpdk-dev] [dpdk v1] net/mlx5: fix possible building error xiangxia.m.yue
2020-03-01 17:09 ` Ali Alnubani
2020-03-02 0:22 3% ` Tonghao Zhang
2020-03-02 7:35 0% ` Ali Alnubani
2020-03-02 8:13 0% ` Tonghao Zhang
2020-03-02 1:57 [dpdk-dev] [PATCH] mempool: sort the rte_mempool_ops by name xiangxia.m.yue
2020-03-06 13:36 ` [dpdk-dev] [PATCH dpdk-dev v3] " xiangxia.m.yue
2020-03-06 13:37 ` Jerin Jacob
2020-03-07 12:51 ` Andrew Rybchenko
2020-03-07 12:54 ` Andrew Rybchenko
2020-03-09 3:01 ` Tonghao Zhang
2020-03-09 8:27 3% ` Olivier Matz
2020-03-09 8:55 0% ` Tonghao Zhang
2020-03-09 9:05 0% ` Olivier Matz
2020-03-09 13:15 0% ` David Marchand
2020-03-16 7:43 0% ` Tonghao Zhang
2020-03-16 7:55 0% ` Olivier Matz
2020-03-24 9:35 0% ` Andrew Rybchenko
2020-03-24 12:41 0% ` Tonghao Zhang
2020-03-02 14:58 [dpdk-dev] [PATCH 00/16] NXP DPAAx fixes and enhancements Hemant Agrawal
2020-03-02 13:01 3% ` David Marchand
2020-03-05 9:06 3% ` Hemant Agrawal (OSS)
2020-03-05 9:09 3% ` David Marchand
2020-03-05 9:19 0% ` Hemant Agrawal (OSS)
2020-03-06 10:12 3% ` David Marchand
2020-03-10 10:36 3% ` Dodji Seketeli
2020-03-02 14:58 ` [dpdk-dev] [PATCH 11/16] net/dpaa: enable Tx queue taildrop Hemant Agrawal
2020-03-03 17:02 ` Ferruh Yigit
2020-03-05 6:49 ` Gagandeep Singh
2020-03-05 14:14 3% ` Ferruh Yigit
2020-03-05 8:53 4% [dpdk-dev] DPDK techboard minutes of February 26 Thomas Monjalon
2020-03-06 16:41 [dpdk-dev] [PATCH 0/4] Introduce IF proxy library Andrzej Ostruszka
2020-03-10 11:10 ` [dpdk-dev] [PATCH v2 " Andrzej Ostruszka
2020-03-25 8:08 3% ` David Marchand
2020-03-25 11:11 4% ` Morten Brørup
2020-03-26 17:42 3% ` Andrzej Ostruszka
2020-03-26 12:41 3% ` Andrzej Ostruszka
2020-03-09 14:14 [dpdk-dev] [PATCH v1 0/4] add Intel DCF PMD support Haiyue Wang
2020-03-09 14:14 1% ` [dpdk-dev] [PATCH v1 3/4] net/ice: add the DCF framework Haiyue Wang
2020-03-10 6:50 ` [dpdk-dev] [PATCH v2 0/7] add Intel DCF PMD support Haiyue Wang
2020-03-10 6:50 1% ` [dpdk-dev] [PATCH v2 2/7] net/ice: add the DCF hardware initialization Haiyue Wang
2020-03-16 5:52 ` [dpdk-dev] [PATCH v3 0/7] add Intel DCF PMD support Haiyue Wang
2020-03-16 5:52 1% ` [dpdk-dev] [PATCH v3 2/7] net/ice: add the DCF hardware initialization Haiyue Wang
2020-03-26 3:03 ` [dpdk-dev] [PATCH v4 0/7] add Intel DCF PMD support Haiyue Wang
2020-03-26 3:03 1% ` [dpdk-dev] [PATCH v4 2/7] net/ice: add the DCF hardware initialization Haiyue Wang
2020-03-26 7:15 ` [dpdk-dev] [PATCH v5 0/7] add Intel DCF PMD support Haiyue Wang
2020-03-26 7:15 1% ` [dpdk-dev] [PATCH v5 2/7] net/ice: add the DCF hardware initialization Haiyue Wang
2020-03-27 2:56 ` [dpdk-dev] [PATCH v6 0/7] add Intel DCF PMD support Haiyue Wang
2020-03-27 2:56 1% ` [dpdk-dev] [PATCH v6 2/7] net/ice: add the DCF hardware initialization Haiyue Wang
[not found] <mailman.2366.1582638587.2727.dev@dpdk.org>
[not found] ` <d8993cb75592406a993e3119d45e906a@intel.com>
2020-03-10 2:55 0% ` [dpdk-dev] dev Digest, Vol 288, Issue 27 Zhang, XiX
2020-03-10 6:50 [dpdk-dev] [PATCH] lib/librte_net/rte_ether.h:changed RTE_ETHER_MAX_LEN Muhammad Ahmad
2020-03-10 15:18 3% ` Stephen Hemminger
2020-03-11 21:58 2% [dpdk-dev] [PATCH v3 0/7] vfio/pci: SR-IOV support Alex Williamson
2020-03-19 6:32 0% ` Tian, Kevin
2020-03-19 13:11 0% ` Alex Williamson
2020-03-18 19:02 [dpdk-dev] [PATCH v1 00/32] DPDK Trace support jerinj
2020-03-25 21:15 1% ` [dpdk-dev] [PATCH v2 " jerinj
2020-03-18 20:41 3% [dpdk-dev] [PATCH] cryptodev: version rte_cryptodev_info_get function Arek Kusztal
2020-03-19 8:28 4% [dpdk-dev] [PATCH] doc: prefer https when pointing to dpdk.org David Marchand
2020-03-19 10:46 0% ` [dpdk-dev] [dpdk-stable] " Kevin Traynor
2020-03-19 14:44 3% [dpdk-dev] [PATCH] devtools: fix check symbol change script Nithin Dabilpuram
2020-03-19 14:56 0% ` David Marchand
2020-03-19 15:40 0% ` Neil Horman
2020-03-19 15:45 0% ` [dpdk-dev] [EXT] " Nithin Dabilpuram
2020-03-19 18:59 0% ` Neil Horman
2020-03-19 15:49 0% ` [dpdk-dev] " Bing Zhao
2020-03-22 14:37 0% ` Jerin Jacob
2020-03-23 11:56 3% ` [dpdk-dev] [PATCH v2] " Nithin Dabilpuram
2020-03-23 13:19 0% ` David Marchand
2020-03-19 17:44 [dpdk-dev] [PATCH v1] usertools/dpdk-setup.sh: fix dpdk-setup's behaviour on non-alphanumeric inputs Sarosh Arif
2020-03-19 18:20 3% ` Stephen Hemminger
2020-03-20 0:12 1% [dpdk-dev] [PATCH] eal: reorganize directories layout Thomas Monjalon
2020-03-27 1:15 ` [dpdk-dev] [PATCH v2 0/8] " Thomas Monjalon
2020-03-27 1:15 3% ` [dpdk-dev] [PATCH v2 6/8] eal: move common header files Thomas Monjalon
2020-03-20 16:41 3% [dpdk-dev] [RFC] ring: make ring implementation non-inlined Konstantin Ananyev
2020-03-20 17:54 0% ` Stephen Hemminger
2020-03-21 1:03 0% ` Ananyev, Konstantin
2020-03-25 21:09 4% ` Jerin Jacob
2020-03-26 0:28 0% ` Ananyev, Konstantin
2020-03-26 8:04 4% ` Morten Brørup
2020-03-24 11:49 [dpdk-dev] [PATCH 00/17] Add CPU flags Kevin Laatz
2020-03-25 11:10 ` [dpdk-dev] [PATCH v2] eal/cpuflags: add x86 based cpu flags Kevin Laatz
2020-03-27 12:24 3% ` David Marchand
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).