Re: [dpdk-dev] 19.11.4 patches review and test

[Christian Ehrhardt <christian.ehrhardt@canonical.com>]

On Tue, Mar 23, 2021 at 7:51 PM Ilya Maximets <i.maximets@ovn.org> wrote:
>
> On 3/23/21 7:17 PM, Thomas Monjalon wrote:
> > 22/03/2021 15:27, Christian Ehrhardt:
> >> On Mon, Mar 22, 2021 at 1:25 PM Thomas Monjalon <thomas@monjalon.net> wrote:
> >>> 22/03/2021 12:59, Luca Boccassi:
> >>>> On Mon, 2021-03-22 at 11:41 +0000, Bruce Richardson wrote:
> >>>>> On Mon, Mar 22, 2021 at 10:49:54AM +0100, Christian Ehrhardt wrote:
> >>>>>> On Thu, Mar 18, 2021 at 7:25 PM Pai G, Sunil <sunil.pai.g@intel.com> wrote:
> >>>>>>> Hi Christian, Ilya
> >>>>>>> From: Ilya Maximets <i.maximets@ovn.org>
> >>>>>>>> On 3/18/21 2:36 PM, Pai G, Sunil wrote:
> >>>>>>>>> Hey Christian,
> >>>>>>>>>
> >>>>>>>>> <snipped>
> >>>>>>>>>
> >>>>>>>>>> back  in 19.11.4 these DPDK changes were not picked up as they have
> >>>>>>>>>> broken builds as discussed here.
> >>>>>>>>>> Later on the communication was that all this works fine now and
> >>>>>>>>>> thereby Luca has "reverted the reverts" in 19.11.6 [1].
> >>>>>>>>>>
> >>>>>>>>>> But today we were made aware that still no OVS 2.13 builds against a
> >>>>>>>>>> DPDK that has those changes.
> >>>>>>>>>> Not 2.13.1 as we have it in Ubuntu nor (if it needs some OVS changes
> >>>>>>>>>> backported) the recent 2.13.3 does build.
> >>>>>>>>>> They still fail with the very same issue I reported [2] back then.
> >>>>>>>>>>
> >>>>>>>>>> Unfortunately I have just released 19.11.7 so I can't revert them
> >>>>>>>>>> there - but OTOH reverting and counter reverting every other release
> >>>>>>>>>> seems wrong anyway.
> >>>>>>>>
> >>>>>>>> It is wrong indeed, but the main question here is why these patches was
> >>>>>>>> backported to stable release in a first place?
> >>>>>>>>
> >>>>>>>> Looking at these patches, they are not actual bug fixes but more like "nice to
> >>>>>>>> have" features that additionally breaks the way application links with DPDK.
> >>>>>>>> Stuff like that should not be acceptable to the stable release without a strong
> >>>>>>>> justification or, at least, testing with actual applications.
> >>>>>>
> >>>>>> I agree, but TBH IIRC these changes were initially by OVS people :-)
> >>>>>> One could chase down the old talks between Luca and the requesters, but I don't
> >>>>>> think that gains us that much.
> >>>>>>
> >>>>>>>> Since we already have a revert of revert, revert of revert of revert doesn't
> >>>>>>>> seem so bad.
> >>>>>>
> >>>>>> As long as we don't extend this series, yeah
> >>>>>>
> >>>>>>>>>> I wanted to ask if there is a set of patches that OVS would need to
> >>>>>>>>>> backport to 2.13.x to make this work?
> >>>>>>>>>> If they could be identified and prepared Distros could use them on
> >>>>>>>>>> 2.13.3 asap and 2.13.4 could officially release them for OVS later on.
> >>>>>>>>>>
> >>>>>>>>>> But for that we'd need a hint which OVS changes that would need to be.
> >>>>>>>>>> All I know atm is from the testing reports on DPDK it seems that OVS
> >>>>>>>>>> 2.14.3 and 2.15 are happy with the new DPDK code.
> >>>>>>>>>> Do you have pointers on what 2.13.3 would need to get backported to
> >>>>>>>>>> work again in regard to this build issue.
> >>>>>>>>>
> >>>>>>>>> You would need to use partial contents from patch :
> >>>>>>>>> http://patchwork.ozlabs.org/project/openvswitch/patch/1608142365-
> >>>>>>>> 26215
> >>>>>>>>> -1-git-send-email-ian.stokes@intel.com/
> >>>>>>>>>
> >>>>>>>>> If you'd like me to send patches which would work with 2.13, 2.14, I'm
> >>>>>>>>> ok with that too.[keeping only those parts from patch which fixes the issue
> >>>>>>>> you see.] But we must ensure it doesn’t cause problems for OVS too.
> >>>>>>>>> Your thoughts Ilya ?
> >>>>>>>>
> >>>>>>>> We had more fixes on top of this particular patch and I'd like to not cherry-
> >>>>>>>> pick and re-check all of this again.
> >>>>>>>
> >>>>>>> I agree, we had more fixes on top of this. It would be risky to cherry-pick.
> >>>>>>> So it might be a better option to revert.
> >>>>>>
> >>>>>> I agree, as far as I assessed the situation it would mean the revert
> >>>>>> of the following list.
> >>>>>> And since that is a lot of "reverts" in the string, to be clear it means that
> >>>>>> those original changes would not be present anymore in 19.11.x.
> >>>>>>
> >>>>>> f49248a990 Revert "Revert "build/pkg-config: prevent overlinking""
> >>>>>> 39586a4cf0 Revert "Revert "build/pkg-config: improve static linking flags""
> >>>>>> 906e935a1f Revert "Revert "build/pkg-config: output drivers first for
> >>>>>> static build""
> >>>>>> deebf95239 Revert "Revert "build/pkg-config: move pkg-config file creation""
> >>>>>> a3bd9a34bf Revert "Revert "build: always link whole DPDK static libraries""
> >>>>>> d4bc124438 Revert "Revert "devtools: test static linkage with pkg-config""
> >>>>>>
> >>>>>> But to avoid going back&forth I'd prefer to have a signed-off on that
> >>>>>> approach from:
> >>>>>> - Luca (for 19.11.6 which has added the changes)
> >>>>>> - Bruce (for being involved in the old&new case in general)
> >>>>>> - Thomas (for general master maintainer thoughts)
> >>>>>>
> >>>>>
> >>>>> If this is what is needed to ensure OVS can continue to use this release
> >>>>> series, then I am absolutely fine with it.
> >>>>
> >>>> This was requested by OVS, so if they don't need it anymore it's fine
> >>>> by me as well
> >>>
> >>> I am not sure to understand the full story,
> >>> but I am a bit worried that our release is dictated by
> >>> a single "user" (project using DPDK).
> >>
> >> Sure, fair to ask for more detail :-)
> >>
> >>> Please do you have links of discussion history?
> >>
> >> I ordered the events by time and added links to those occasions that I
> >> could find:
> >>
> >> July 2020            - Initial request by OVS - *1
> >> July 2020            - Initial queuing     -
> >> http://mails.dpdk.org/archives/stable/2020-July/024248.html
> >> September 2020 - Issues identified; changes reverted    -
> >> http://mails.dpdk.org/archives/stable/2020-September/024796.html
> >> October 2020      - Re-applying early in 19.11.6 cycle    - *1
> >> November 2020  - Tests didn't spot it with 19.11.6 as OVS 2.14.x (not
> >> the 2.13 LTS) was tested    -
> >> https://doc.dpdk.org/guides-19.11/rel_notes/release_19_11.html#id16
> >> March 2021         - Same issue re-found in >=19.11.6    -
> >> http://mails.dpdk.org/archives/stable/2021-March/029418.html
> >>
> >> *1 - Luca and I looked for logs, there are no links that I'd know of
> >> and Luca said it might have come up as a request during a meeting.
> >
> > First, I agree to revert the changes again if it causes a regression.
> > Second, do we know the root cause of the issue?
> > Is it a problem with the version of pkg-config?
> > Is it OK with DPDK 20.11?
> >
>
> I'd like to also ask someone to test build of both OVS 2.13 and OVS 2.14
> with these changes and with these changes reverted.

I've test built a few of those already.
- 19.11.4 (before the patches were applied)
  - OVS 2.13.1 worked
- 19.11.6/19.11.7 (patches not yet reverted)
  - OVS 2.13.1 fails
  - OVS 2.13.3 fails
- 19.11.7 patches reverted
  - OVS 2.13.3 works

I'd also be happy to hear about OVS 2.14 test builds, so yeah if you
could do so  @Sunil that would be great.
For the code, I've not yet pushed it to "real dpdk-stable" until we
are sure about it, but already to:
  https://github.com/cpaelzer/dpdk-stable-queue/tree/19.11
If you happen to build on Ubuntu there is a 19.11.7 + reverts already
available here
  https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3690/

> Sunil, could you do that?
>
> > About the process, I see multiple issues:
> >
> > 1/ Some patches were backported for OVS only,
> > but it could break other applications.

As we found it even breaks (older) OVS, but importantly the OVS LTS
which has the highest chance to be in use together with DPDK 19.11 in
many places :-/

> > 2/ It is not clear whether the patches were really needed in 19.11.
> >
> > 3/ There is no trace of backport requests in the mailing list.
> >
> > So I feel we should be stricter on the reasons for a backport.
> > Note: I am not blaming anyone. Everybody tries to do the best.
> > I believe sharing requests and discussions on the mailing list
> > could help in the decision process.

Agreed

> +1
>
> >
> > Thanks for all the work.



-- 
Christian Ehrhardt
Staff Engineer, Ubuntu Server
Canonical Ltd