Re: [dpdk-dev] RFC enabling dll/dso for dpdk on windows

[Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>]

2021-07-08 18:03 (UTC-0700), Tyler Retzlaff:
> On Thu, Jul 08, 2021 at 11:49:53PM +0300, Dmitry Kozlyuk wrote:
> > Hi Tyler,
> > 
> > 2021-07-08 12:21 (UTC-0700), Tyler Retzlaff:  
> > > hi folks,
> > > 
> > > we would like to submit a a patch series that makes dll/dso for dpdk
> > > work on windows. there are two differences in the windows platform that
> > > would need to be address through enhancements to dpdk.
> > > 
> > > (1) windows dynamic objects don't export sufficient information for
> > >     tls variables and the windows loader and runtime would need to be
> > >     enhanced in order to perform runtime linking. [1][2]  
> > 
> > When will the new loader be available?  
> 
> the solution i have prototyped does not directly export the tls variables
> and instead relies on exports of tls offsets within a module.  no loader
> change or new os is required.
> 
> > Will it be ported to Server 2019?  
> 
> not necessary (as per above)
> 
> > Will this functionality require compiler support  
> 
> the prototype was developed using windows clang, mingw code compiles but
> i did not try to run it. i suspect it is okay though haven't examine any
> side-effects when using emul tls like mingw does. anyway mingw dll's
> don't work now and it probably shouldn't block them being available with
> clang.

AFAIK it's the opposite. MinGW can handle TLS varibale export from DLL,
although with "__emutls." prefix and some performance penalty.
Clang can't at all, despite compiling and linking without an issue.

No, it is not acceptable to add a generic feature supported by only one
compiler. (FWIW, I'm displeased even by mlx5 being tied to clang.)
Particularly, I don't understand how could MinGW and clang coexist
if they export different sets of symbols. Apps will need to know
if it's MingW- or clang-compiled DPDK? Sounds messy.

> > (you mention that accessing such variables will be "non-trivial")?  
> 
> the solution involves exporting offsets that then allow explicit tls
> accesses relative to the gs segment. it's non-trivial in the sense that
> none of the normal explicit tls functions in windows are used and the
> compiler doesn't generate the code for implicit tls access. the overhead
> is relatively tolerable (one or two additional dereferences).

A thorough benchmark will be required. I'm afraid that inline assembly
(which %gs mention suggests) can impact optimization of the code nearby.
Ideally it should be a DPDK performance autotest.

> 
> >    
> > > (2) importing exported data symbols from a dll/dso on windows requires
> > >     that the symbol be decorated with dllimport. optionally loading
> > >     performance of dll/dso is also further improved by decorating
> > >     exported function symbols. [3]  
> > 
> > Does it affect ABI?  
> 
> the data symbols are already part of the abi for linux. this just allows
> them to be properly accessed when exported from dll on windows.
> surprisingly lld-link doesn't fail when building dll's now which it should
> in the absence of a __declspec(dllimport) ms link would.
> 
> on windows now the tls variables are exported but not useful with this
> change we would choose not to export them at all and each exported tls
> variable would be replaced with a new variable.
> 
> one nit (which we will get separate feedback on) is how to export
> symbols only on windows (and don't export them on linux) because similar
> to the tls variables linux has no use for my new variables.

There's already WINDOWS_NO_EXPORT mark in .map to generate .def,
likewise, .map for Linux/FreeBSD could be generated from a basic .map
with similar marks.

> > 
> > It is also a huge code change, although a mechanical one.
> > Is it required? All exported symbols are listed in .map/def, after all.  
> 
> if broad sweeping mechanical change is a sensitive issue we can limit
> the change to just the data symbols which are required. but keeping in
> mind there is a penalty on load time when the function symbols are not
> decorated. ultimately we would like them all properly decorated but we
> don't need to push it now since we're just trying to enable the
> functionality.

I was asking in connection with the previous question about ABI,
because 21.11 ABI freeze may be a two-year one. Since ABI is not affected
for Unix and for Windows we don't maintain it currently, there is no rush for
the change at least.