[dpdk-dev] [PATCH] vfio: retry creating sPAPR DMA window

[dpdk-dev] [PATCH] vfio: retry creating sPAPR DMA window

[Takeshi Yoshimura]

sPAPR allows only page_shift from VFIO_IOMMU_SPAPR_TCE_GET_INFO ioctl.
However, Linux 4.17 or before returns incorrect page_shift for Power9.
I added the code for retrying creation of sPAPR DMA window.

Signed-off-by: Takeshi Yoshimura <tyos@jp.ibm.com>
---
 lib/librte_eal/linux/eal/eal_vfio.c | 26 +++++++++++++++++++++++---
 1 file changed, 23 insertions(+), 3 deletions(-)

diff --git a/lib/librte_eal/linux/eal/eal_vfio.c b/lib/librte_eal/linux/eal/eal_vfio.c
index 6892a2c14..f16c5c3c0 100644
--- a/lib/librte_eal/linux/eal/eal_vfio.c
+++ b/lib/librte_eal/linux/eal/eal_vfio.c
@@ -1448,9 +1448,29 @@ vfio_spapr_create_new_dma_window(int vfio_container_fd,
 	/* create new DMA window */
 	ret = ioctl(vfio_container_fd, VFIO_IOMMU_SPAPR_TCE_CREATE, create);
 	if (ret) {
-		RTE_LOG(ERR, EAL, "  cannot create new DMA window, "
-				"error %i (%s)\n", errno, strerror(errno));
-		return -1;
+		/* try possible page_shift and levels for workaround */
+		uint32_t levels;
+
+		for (levels = 1; levels <= info.ddw.levels; levels++) {
+			uint32_t pgsizes = info.ddw.pgsizes;
+
+			while (pgsizes != 0) {
+				create->page_shift = 31 - __builtin_clz(pgsizes);
+				create->levels = levels;
+				ret = ioctl(vfio_container_fd,
+					VFIO_IOMMU_SPAPR_TCE_CREATE, create);
+				if (!ret)
+					break;
+				pgsizes &= ~(1 << create->page_shift);
+			}
+			if (!ret)
+				break;
+		}
+		if (ret) {
+			RTE_LOG(ERR, EAL, "  cannot create new DMA window, "
+					"error %i (%s)\n", errno, strerror(errno));
+			return -1;
+		}
 	}
 
 	if (create->start_addr != 0) {
-- 
2.17.1

Re: [dpdk-dev] [PATCH] vfio: retry creating sPAPR DMA window

[Thomas Monjalon]

Please, any review or ack for this patch?

07/06/2019 04:28, Takeshi Yoshimura:
> sPAPR allows only page_shift from VFIO_IOMMU_SPAPR_TCE_GET_INFO ioctl.
> However, Linux 4.17 or before returns incorrect page_shift for Power9.
> I added the code for retrying creation of sPAPR DMA window.
> 
> Signed-off-by: Takeshi Yoshimura <tyos@jp.ibm.com>
> ---
>  lib/librte_eal/linux/eal/eal_vfio.c | 26 +++++++++++++++++++++++---
>  1 file changed, 23 insertions(+), 3 deletions(-)

Re: [dpdk-dev] [PATCH] vfio: retry creating sPAPR DMA window

[Burakov, Anatoly]

On 07-Jun-19 3:28 AM, Takeshi Yoshimura wrote:
> sPAPR allows only page_shift from VFIO_IOMMU_SPAPR_TCE_GET_INFO ioctl.
> However, Linux 4.17 or before returns incorrect page_shift for Power9.
> I added the code for retrying creation of sPAPR DMA window.
> 
> Signed-off-by: Takeshi Yoshimura <tyos@jp.ibm.com>
> ---

This doesn't affect any code outside of sPAPR and looks sane, so

Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH] vfio: retry creating sPAPR DMA window

[Thomas Monjalon]

05/07/2019 10:15, Burakov, Anatoly:
> On 07-Jun-19 3:28 AM, Takeshi Yoshimura wrote:
> > sPAPR allows only page_shift from VFIO_IOMMU_SPAPR_TCE_GET_INFO ioctl.
> > However, Linux 4.17 or before returns incorrect page_shift for Power9.
> > I added the code for retrying creation of sPAPR DMA window.
> > 
> > Signed-off-by: Takeshi Yoshimura <tyos@jp.ibm.com>
> > ---
> 
> This doesn't affect any code outside of sPAPR and looks sane, so
> 
> Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>

Applied, thanks

Re: [dpdk-dev] [PATCH] vfio: retry creating sPAPR DMA window

[David Christensen]

> Please, any review or ack for this patch?
> 
> 07/06/2019 04:28, Takeshi Yoshimura:
>> sPAPR allows only page_shift from VFIO_IOMMU_SPAPR_TCE_GET_INFO ioctl.
>> However, Linux 4.17 or before returns incorrect page_shift for Power9.
>> I added the code for retrying creation of sPAPR DMA window.
>>
>> Signed-off-by: Takeshi Yoshimura <tyos@jp.ibm.com>
>> ---
>>   lib/librte_eal/linux/eal/eal_vfio.c | 26 +++++++++++++++++++++++---
>>   1 file changed, 23 insertions(+), 3 deletions(-)
> 

I've shared my comments with Takeshi and I don't think the patch is 
ready in it's current state.  It appears to modify x86 functionality and 
I'm not comfortable with signing off on that since it's in an anrea I'm 
not familiar with.

Dave

Re: [dpdk-dev] [PATCH] vfio: retry creating sPAPR DMA window

[Thomas Monjalon]

08/07/2019 18:47, David Christensen:
> > Please, any review or ack for this patch?
> > 
> > 07/06/2019 04:28, Takeshi Yoshimura:
> >> sPAPR allows only page_shift from VFIO_IOMMU_SPAPR_TCE_GET_INFO ioctl.
> >> However, Linux 4.17 or before returns incorrect page_shift for Power9.
> >> I added the code for retrying creation of sPAPR DMA window.
> >>
> >> Signed-off-by: Takeshi Yoshimura <tyos@jp.ibm.com>
> 
> I've shared my comments with Takeshi and I don't think the patch is 
> ready in it's current state.  It appears to modify x86 functionality and 
> I'm not comfortable with signing off on that since it's in an anrea I'm 
> not familiar with.

It is a function specific to sPAPR.
So there is no risk for x86 or Arm.
As you didn't share your comments publicly,
this patch has been merged in 19.08-rc1.

Re: [dpdk-dev] [PATCH] vfio: retry creating sPAPR DMA window

[Burakov, Anatoly]

On 08-Jul-19 6:45 PM, Thomas Monjalon wrote:
> 08/07/2019 18:47, David Christensen:
>>> Please, any review or ack for this patch?
>>>
>>> 07/06/2019 04:28, Takeshi Yoshimura:
>>>> sPAPR allows only page_shift from VFIO_IOMMU_SPAPR_TCE_GET_INFO ioctl.
>>>> However, Linux 4.17 or before returns incorrect page_shift for Power9.
>>>> I added the code for retrying creation of sPAPR DMA window.
>>>>
>>>> Signed-off-by: Takeshi Yoshimura <tyos@jp.ibm.com>
>>
>> I've shared my comments with Takeshi and I don't think the patch is
>> ready in it's current state.  It appears to modify x86 functionality and
>> I'm not comfortable with signing off on that since it's in an anrea I'm
>> not familiar with.
> 
> It is a function specific to sPAPR.
> So there is no risk for x86 or Arm.
> As you didn't share your comments publicly,
> this patch has been merged in 19.08-rc1.
> 

I have a suspicion that David has confused this patch with another one, 
which did indeed touch a lot of code (but still didn't touch x86 AFAIR).

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH] vfio: retry creating sPAPR DMA window

[Burakov, Anatoly]

On 07-Jun-19 3:28 AM, Takeshi Yoshimura wrote:
> sPAPR allows only page_shift from VFIO_IOMMU_SPAPR_TCE_GET_INFO ioctl.
> However, Linux 4.17 or before returns incorrect page_shift for Power9.
> I added the code for retrying creation of sPAPR DMA window.
> 
> Signed-off-by: Takeshi Yoshimura <tyos@jp.ibm.com>
> ---
>   lib/librte_eal/linux/eal/eal_vfio.c | 26 +++++++++++++++++++++++---
>   1 file changed, 23 insertions(+), 3 deletions(-)
> 
> diff --git a/lib/librte_eal/linux/eal/eal_vfio.c b/lib/librte_eal/linux/eal/eal_vfio.c
> index 6892a2c14..f16c5c3c0 100644
> --- a/lib/librte_eal/linux/eal/eal_vfio.c
> +++ b/lib/librte_eal/linux/eal/eal_vfio.c
> @@ -1448,9 +1448,29 @@ vfio_spapr_create_new_dma_window(int vfio_container_fd,
>   	/* create new DMA window */
>   	ret = ioctl(vfio_container_fd, VFIO_IOMMU_SPAPR_TCE_CREATE, create);
>   	if (ret) {
> -		RTE_LOG(ERR, EAL, "  cannot create new DMA window, "
> -				"error %i (%s)\n", errno, strerror(errno));
> -		return -1;
> +		/* try possible page_shift and levels for workaround */
> +		uint32_t levels;
> +
> +		for (levels = 1; levels <= info.ddw.levels; levels++) {
> +			uint32_t pgsizes = info.ddw.pgsizes;

+CC POWER maintainer from MAINTAINERS file.

This is failing compilation on some older distros (Fedora 20, Ubuntu 14.04):


/root/dpdk/lib/librte_eal/linux/eal/eal_vfio.c: In function 
‘vfio_spapr_create_new_dma_window’:
/root/dpdk/lib/librte_eal/linux/eal/eal_vfio.c:1451:34: error: ‘struct 
vfio_iommu_spapr_tce_info’ has no member named ‘ddw’
for (levels = 1; levels <= info.ddw.levels; levels++) {
^
/root/dpdk/lib/librte_eal/linux/eal/eal_vfio.c:1452:27: error: ‘struct 
vfio_iommu_spapr_tce_info’ has no member named ‘ddw’
uint32_t pgsizes = info.ddw.pgsizes;
^
/root/dpdk/lib/librte_eal/linux/eal/eal_vfio.c: At top level:
cc1: error: unrecognized command line option 
"-Wno-address-of-packed-member" [-Werror]
cc1: all warnings being treated as errors
make[7]: *** [eal_vfio.o] Error 1
make[7]: *** Waiting for unfinished jobs....
make[6]: *** [eal] Error 2
make[5]: *** [linux] Error 2
make[4]: *** [librte_eal] Error 2
make[3]: *** [lib] Error 2
make[2]: *** [all] Error 2
make[1]: *** [pre_install] Error 2
make: *** [install] Error 2


Looking at the eal_vfio.h we are handling case of missing sPAPR info/ddw 
structures, but this isn't simply a case of a missing struct; rather it 
is an issue of differing definitions between what recent kernels have, 
and what older kernels had.

@Thomas Do we still support these older distros?

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH] vfio: retry creating sPAPR DMA window

[Thomas Monjalon]

10/07/2019 12:32, Burakov, Anatoly:
> On 07-Jun-19 3:28 AM, Takeshi Yoshimura wrote:
> > sPAPR allows only page_shift from VFIO_IOMMU_SPAPR_TCE_GET_INFO ioctl.
> > However, Linux 4.17 or before returns incorrect page_shift for Power9.
> > I added the code for retrying creation of sPAPR DMA window.
> > 
> > Signed-off-by: Takeshi Yoshimura <tyos@jp.ibm.com>
> > ---
> >   lib/librte_eal/linux/eal/eal_vfio.c | 26 +++++++++++++++++++++++---
> >   1 file changed, 23 insertions(+), 3 deletions(-)
> > 
> > diff --git a/lib/librte_eal/linux/eal/eal_vfio.c b/lib/librte_eal/linux/eal/eal_vfio.c
> > index 6892a2c14..f16c5c3c0 100644
> > --- a/lib/librte_eal/linux/eal/eal_vfio.c
> > +++ b/lib/librte_eal/linux/eal/eal_vfio.c
> > @@ -1448,9 +1448,29 @@ vfio_spapr_create_new_dma_window(int vfio_container_fd,
> >   	/* create new DMA window */
> >   	ret = ioctl(vfio_container_fd, VFIO_IOMMU_SPAPR_TCE_CREATE, create);
> >   	if (ret) {
> > -		RTE_LOG(ERR, EAL, "  cannot create new DMA window, "
> > -				"error %i (%s)\n", errno, strerror(errno));
> > -		return -1;
> > +		/* try possible page_shift and levels for workaround */
> > +		uint32_t levels;
> > +
> > +		for (levels = 1; levels <= info.ddw.levels; levels++) {
> > +			uint32_t pgsizes = info.ddw.pgsizes;
> 
> +CC POWER maintainer from MAINTAINERS file.
> 
> This is failing compilation on some older distros (Fedora 20, Ubuntu 14.04):
> 
> 
> /root/dpdk/lib/librte_eal/linux/eal/eal_vfio.c: In function 
> ‘vfio_spapr_create_new_dma_window’:
> /root/dpdk/lib/librte_eal/linux/eal/eal_vfio.c:1451:34: error: ‘struct 
> vfio_iommu_spapr_tce_info’ has no member named ‘ddw’
> for (levels = 1; levels <= info.ddw.levels; levels++) {
> ^
> /root/dpdk/lib/librte_eal/linux/eal/eal_vfio.c:1452:27: error: ‘struct 
> vfio_iommu_spapr_tce_info’ has no member named ‘ddw’
> uint32_t pgsizes = info.ddw.pgsizes;
> ^
> /root/dpdk/lib/librte_eal/linux/eal/eal_vfio.c: At top level:
> cc1: error: unrecognized command line option 
> "-Wno-address-of-packed-member" [-Werror]
> cc1: all warnings being treated as errors
> make[7]: *** [eal_vfio.o] Error 1
> make[7]: *** Waiting for unfinished jobs....
> make[6]: *** [eal] Error 2
> make[5]: *** [linux] Error 2
> make[4]: *** [librte_eal] Error 2
> make[3]: *** [lib] Error 2
> make[2]: *** [all] Error 2
> make[1]: *** [pre_install] Error 2
> make: *** [install] Error 2
> 
> 
> Looking at the eal_vfio.h we are handling case of missing sPAPR info/ddw 
> structures, but this isn't simply a case of a missing struct; rather it 
> is an issue of differing definitions between what recent kernels have, 
> and what older kernels had.
> 
> @Thomas Do we still support these older distros?

If the kernel is not maintained, we don't support it.
For most distros, Linux < 3.16 is not maintained.

Re: [dpdk-dev] [PATCH] vfio: retry creating sPAPR DMA window

[Burakov, Anatoly]

On 10-Jul-19 1:17 PM, Thomas Monjalon wrote:
> 10/07/2019 12:32, Burakov, Anatoly:
>> On 07-Jun-19 3:28 AM, Takeshi Yoshimura wrote:
>>> sPAPR allows only page_shift from VFIO_IOMMU_SPAPR_TCE_GET_INFO ioctl.
>>> However, Linux 4.17 or before returns incorrect page_shift for Power9.
>>> I added the code for retrying creation of sPAPR DMA window.
>>>
>>> Signed-off-by: Takeshi Yoshimura <tyos@jp.ibm.com>
>>> ---
>>>    lib/librte_eal/linux/eal/eal_vfio.c | 26 +++++++++++++++++++++++---
>>>    1 file changed, 23 insertions(+), 3 deletions(-)
>>>
>>> diff --git a/lib/librte_eal/linux/eal/eal_vfio.c b/lib/librte_eal/linux/eal/eal_vfio.c
>>> index 6892a2c14..f16c5c3c0 100644
>>> --- a/lib/librte_eal/linux/eal/eal_vfio.c
>>> +++ b/lib/librte_eal/linux/eal/eal_vfio.c
>>> @@ -1448,9 +1448,29 @@ vfio_spapr_create_new_dma_window(int vfio_container_fd,
>>>    	/* create new DMA window */
>>>    	ret = ioctl(vfio_container_fd, VFIO_IOMMU_SPAPR_TCE_CREATE, create);
>>>    	if (ret) {
>>> -		RTE_LOG(ERR, EAL, "  cannot create new DMA window, "
>>> -				"error %i (%s)\n", errno, strerror(errno));
>>> -		return -1;
>>> +		/* try possible page_shift and levels for workaround */
>>> +		uint32_t levels;
>>> +
>>> +		for (levels = 1; levels <= info.ddw.levels; levels++) {
>>> +			uint32_t pgsizes = info.ddw.pgsizes;
>>
>> +CC POWER maintainer from MAINTAINERS file.
>>
>> This is failing compilation on some older distros (Fedora 20, Ubuntu 14.04):
>>
>>
>> /root/dpdk/lib/librte_eal/linux/eal/eal_vfio.c: In function
>> ‘vfio_spapr_create_new_dma_window’:
>> /root/dpdk/lib/librte_eal/linux/eal/eal_vfio.c:1451:34: error: ‘struct
>> vfio_iommu_spapr_tce_info’ has no member named ‘ddw’
>> for (levels = 1; levels <= info.ddw.levels; levels++) {
>> ^
>> /root/dpdk/lib/librte_eal/linux/eal/eal_vfio.c:1452:27: error: ‘struct
>> vfio_iommu_spapr_tce_info’ has no member named ‘ddw’
>> uint32_t pgsizes = info.ddw.pgsizes;
>> ^
>> /root/dpdk/lib/librte_eal/linux/eal/eal_vfio.c: At top level:
>> cc1: error: unrecognized command line option
>> "-Wno-address-of-packed-member" [-Werror]
>> cc1: all warnings being treated as errors
>> make[7]: *** [eal_vfio.o] Error 1
>> make[7]: *** Waiting for unfinished jobs....
>> make[6]: *** [eal] Error 2
>> make[5]: *** [linux] Error 2
>> make[4]: *** [librte_eal] Error 2
>> make[3]: *** [lib] Error 2
>> make[2]: *** [all] Error 2
>> make[1]: *** [pre_install] Error 2
>> make: *** [install] Error 2
>>
>>
>> Looking at the eal_vfio.h we are handling case of missing sPAPR info/ddw
>> structures, but this isn't simply a case of a missing struct; rather it
>> is an issue of differing definitions between what recent kernels have,
>> and what older kernels had.
>>
>> @Thomas Do we still support these older distros?
> 
> If the kernel is not maintained, we don't support it.
> For most distros, Linux < 3.16 is not maintained.
> 

This is on kernel 3.13 (Ubuntu 14.04 LTS which according to Canonical's 
published schedule [1] has stopped receiving even maintenance updates, 
and is only updated for select customers). It was never an LTS kernel to 
begin with, i think.

[1] https://ubuntu.com/about/release-cycle

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH] vfio: retry creating sPAPR DMA window

[Thomas Monjalon]

10/07/2019 14:35, Burakov, Anatoly:
> On 10-Jul-19 1:17 PM, Thomas Monjalon wrote:
> > 10/07/2019 12:32, Burakov, Anatoly:
> >> On 07-Jun-19 3:28 AM, Takeshi Yoshimura wrote:
> >>> sPAPR allows only page_shift from VFIO_IOMMU_SPAPR_TCE_GET_INFO ioctl.
> >>> However, Linux 4.17 or before returns incorrect page_shift for Power9.
> >>> I added the code for retrying creation of sPAPR DMA window.
> >>>
> >>> Signed-off-by: Takeshi Yoshimura <tyos@jp.ibm.com>
> >>> ---
> >>>    lib/librte_eal/linux/eal/eal_vfio.c | 26 +++++++++++++++++++++++---
> >>>    1 file changed, 23 insertions(+), 3 deletions(-)
> >>>
> >>> diff --git a/lib/librte_eal/linux/eal/eal_vfio.c b/lib/librte_eal/linux/eal/eal_vfio.c
> >>> index 6892a2c14..f16c5c3c0 100644
> >>> --- a/lib/librte_eal/linux/eal/eal_vfio.c
> >>> +++ b/lib/librte_eal/linux/eal/eal_vfio.c
> >>> @@ -1448,9 +1448,29 @@ vfio_spapr_create_new_dma_window(int vfio_container_fd,
> >>>    	/* create new DMA window */
> >>>    	ret = ioctl(vfio_container_fd, VFIO_IOMMU_SPAPR_TCE_CREATE, create);
> >>>    	if (ret) {
> >>> -		RTE_LOG(ERR, EAL, "  cannot create new DMA window, "
> >>> -				"error %i (%s)\n", errno, strerror(errno));
> >>> -		return -1;
> >>> +		/* try possible page_shift and levels for workaround */
> >>> +		uint32_t levels;
> >>> +
> >>> +		for (levels = 1; levels <= info.ddw.levels; levels++) {
> >>> +			uint32_t pgsizes = info.ddw.pgsizes;
> >>
> >> +CC POWER maintainer from MAINTAINERS file.
> >>
> >> This is failing compilation on some older distros (Fedora 20, Ubuntu 14.04):
> >>
> >>
> >> /root/dpdk/lib/librte_eal/linux/eal/eal_vfio.c: In function
> >> ‘vfio_spapr_create_new_dma_window’:
> >> /root/dpdk/lib/librte_eal/linux/eal/eal_vfio.c:1451:34: error: ‘struct
> >> vfio_iommu_spapr_tce_info’ has no member named ‘ddw’
> >> for (levels = 1; levels <= info.ddw.levels; levels++) {
> >> ^
> >> /root/dpdk/lib/librte_eal/linux/eal/eal_vfio.c:1452:27: error: ‘struct
> >> vfio_iommu_spapr_tce_info’ has no member named ‘ddw’
> >> uint32_t pgsizes = info.ddw.pgsizes;
> >> ^
> >> /root/dpdk/lib/librte_eal/linux/eal/eal_vfio.c: At top level:
> >> cc1: error: unrecognized command line option
> >> "-Wno-address-of-packed-member" [-Werror]
> >> cc1: all warnings being treated as errors
> >> make[7]: *** [eal_vfio.o] Error 1
> >> make[7]: *** Waiting for unfinished jobs....
> >> make[6]: *** [eal] Error 2
> >> make[5]: *** [linux] Error 2
> >> make[4]: *** [librte_eal] Error 2
> >> make[3]: *** [lib] Error 2
> >> make[2]: *** [all] Error 2
> >> make[1]: *** [pre_install] Error 2
> >> make: *** [install] Error 2
> >>
> >>
> >> Looking at the eal_vfio.h we are handling case of missing sPAPR info/ddw
> >> structures, but this isn't simply a case of a missing struct; rather it
> >> is an issue of differing definitions between what recent kernels have,
> >> and what older kernels had.
> >>
> >> @Thomas Do we still support these older distros?
> > 
> > If the kernel is not maintained, we don't support it.
> > For most distros, Linux < 3.16 is not maintained.
> 
> This is on kernel 3.13 (Ubuntu 14.04 LTS which according to Canonical's 
> published schedule [1] has stopped receiving even maintenance updates, 
> and is only updated for select customers). It was never an LTS kernel to 
> begin with, i think.

So we can ignore this issue.

Re: [dpdk-dev] [PATCH] vfio: retry creating sPAPR DMA window

[Burakov, Anatoly]

On 10-Jul-19 1:39 PM, Thomas Monjalon wrote:
> 10/07/2019 14:35, Burakov, Anatoly:
>> On 10-Jul-19 1:17 PM, Thomas Monjalon wrote:
>>> 10/07/2019 12:32, Burakov, Anatoly:
>>>> On 07-Jun-19 3:28 AM, Takeshi Yoshimura wrote:
>>>>> sPAPR allows only page_shift from VFIO_IOMMU_SPAPR_TCE_GET_INFO ioctl.
>>>>> However, Linux 4.17 or before returns incorrect page_shift for Power9.
>>>>> I added the code for retrying creation of sPAPR DMA window.
>>>>>
>>>>> Signed-off-by: Takeshi Yoshimura <tyos@jp.ibm.com>
>>>>> ---
>>>>>     lib/librte_eal/linux/eal/eal_vfio.c | 26 +++++++++++++++++++++++---
>>>>>     1 file changed, 23 insertions(+), 3 deletions(-)
>>>>>
>>>>> diff --git a/lib/librte_eal/linux/eal/eal_vfio.c b/lib/librte_eal/linux/eal/eal_vfio.c
>>>>> index 6892a2c14..f16c5c3c0 100644
>>>>> --- a/lib/librte_eal/linux/eal/eal_vfio.c
>>>>> +++ b/lib/librte_eal/linux/eal/eal_vfio.c
>>>>> @@ -1448,9 +1448,29 @@ vfio_spapr_create_new_dma_window(int vfio_container_fd,
>>>>>     	/* create new DMA window */
>>>>>     	ret = ioctl(vfio_container_fd, VFIO_IOMMU_SPAPR_TCE_CREATE, create);
>>>>>     	if (ret) {
>>>>> -		RTE_LOG(ERR, EAL, "  cannot create new DMA window, "
>>>>> -				"error %i (%s)\n", errno, strerror(errno));
>>>>> -		return -1;
>>>>> +		/* try possible page_shift and levels for workaround */
>>>>> +		uint32_t levels;
>>>>> +
>>>>> +		for (levels = 1; levels <= info.ddw.levels; levels++) {
>>>>> +			uint32_t pgsizes = info.ddw.pgsizes;
>>>>
>>>> +CC POWER maintainer from MAINTAINERS file.
>>>>
>>>> This is failing compilation on some older distros (Fedora 20, Ubuntu 14.04):
>>>>
>>>>
>>>> /root/dpdk/lib/librte_eal/linux/eal/eal_vfio.c: In function
>>>> ‘vfio_spapr_create_new_dma_window’:
>>>> /root/dpdk/lib/librte_eal/linux/eal/eal_vfio.c:1451:34: error: ‘struct
>>>> vfio_iommu_spapr_tce_info’ has no member named ‘ddw’
>>>> for (levels = 1; levels <= info.ddw.levels; levels++) {
>>>> ^
>>>> /root/dpdk/lib/librte_eal/linux/eal/eal_vfio.c:1452:27: error: ‘struct
>>>> vfio_iommu_spapr_tce_info’ has no member named ‘ddw’
>>>> uint32_t pgsizes = info.ddw.pgsizes;
>>>> ^
>>>> /root/dpdk/lib/librte_eal/linux/eal/eal_vfio.c: At top level:
>>>> cc1: error: unrecognized command line option
>>>> "-Wno-address-of-packed-member" [-Werror]
>>>> cc1: all warnings being treated as errors
>>>> make[7]: *** [eal_vfio.o] Error 1
>>>> make[7]: *** Waiting for unfinished jobs....
>>>> make[6]: *** [eal] Error 2
>>>> make[5]: *** [linux] Error 2
>>>> make[4]: *** [librte_eal] Error 2
>>>> make[3]: *** [lib] Error 2
>>>> make[2]: *** [all] Error 2
>>>> make[1]: *** [pre_install] Error 2
>>>> make: *** [install] Error 2
>>>>
>>>>
>>>> Looking at the eal_vfio.h we are handling case of missing sPAPR info/ddw
>>>> structures, but this isn't simply a case of a missing struct; rather it
>>>> is an issue of differing definitions between what recent kernels have,
>>>> and what older kernels had.
>>>>
>>>> @Thomas Do we still support these older distros?
>>>
>>> If the kernel is not maintained, we don't support it.
>>> For most distros, Linux < 3.16 is not maintained.
>>
>> This is on kernel 3.13 (Ubuntu 14.04 LTS which according to Canonical's
>> published schedule [1] has stopped receiving even maintenance updates,
>> and is only updated for select customers). It was never an LTS kernel to
>> begin with, i think.
> 
> So we can ignore this issue.
> 

It seems that it's also reproducible on Ubuntu 14.04 with kernel 3.19. 
Which is not an upstream LTS kernel, but it's >3.16.

It also looks like it may be an issue for us specifically, so i would 
suggest reverting the patch for rc2 if the submitter/maintainers don't 
come back and fix it.

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH] vfio: retry creating sPAPR DMA window

[David Christensen]

>>>>> @Thomas Do we still support these older distros?
>>>>
>>>> If the kernel is not maintained, we don't support it.
>>>> For most distros, Linux < 3.16 is not maintained.
>>>
>>> This is on kernel 3.13 (Ubuntu 14.04 LTS which according to Canonical's
>>> published schedule [1] has stopped receiving even maintenance updates,
>>> and is only updated for select customers). It was never an LTS kernel to
>>> begin with, i think.
>>
>> So we can ignore this issue.
>>
> 
> It seems that it's also reproducible on Ubuntu 14.04 with kernel 3.19. 
> Which is not an upstream LTS kernel, but it's >3.16.
> 
> It also looks like it may be an issue for us specifically, so i would 
> suggest reverting the patch for rc2 if the submitter/maintainers don't 
> come back and fix it.

Just to confirm, these failures are on x86 builds, is that correct? 
Power 9 systems aren't supported by these older distros, Linux kernel 
4.6 was the first kernel with Power 9 support.

Dave

Re: [dpdk-dev] [PATCH] vfio: retry creating sPAPR DMA window

[Takeshi T Yoshimura]

-----"Burakov, Anatoly" <anatoly.burakov@intel.com> wrote: -----

>To: Thomas Monjalon <thomas@monjalon.net>
>From: "Burakov, Anatoly" <anatoly.burakov@intel.com>
>Date: 07/10/2019 09:01AM
>Cc: Takeshi Yoshimura <tyos@jp.ibm.com>, dev@dpdk.org, David
>Christensen <drc@linux.vnet.ibm.com>
>Subject: [EXTERNAL] Re: [dpdk-dev] [PATCH] vfio: retry creating sPAPR
>DMA window
>
>On 10-Jul-19 1:39 PM, Thomas Monjalon wrote:
>> 10/07/2019 14:35, Burakov, Anatoly:
>>> On 10-Jul-19 1:17 PM, Thomas Monjalon wrote:
>>>> 10/07/2019 12:32, Burakov, Anatoly:
>>>>> On 07-Jun-19 3:28 AM, Takeshi Yoshimura wrote:
>>>>>> sPAPR allows only page_shift from VFIO_IOMMU_SPAPR_TCE_GET_INFO
>ioctl.
>>>>>> However, Linux 4.17 or before returns incorrect page_shift for
>Power9.
>>>>>> I added the code for retrying creation of sPAPR DMA window.
>>>>>>
>>>>>> Signed-off-by: Takeshi Yoshimura <tyos@jp.ibm.com>
>>>>>> ---
>>>>>>     lib/librte_eal/linux/eal/eal_vfio.c | 26
>+++++++++++++++++++++++---
>>>>>>     1 file changed, 23 insertions(+), 3 deletions(-)
>>>>>>
>>>>>> diff --git a/lib/librte_eal/linux/eal/eal_vfio.c
>b/lib/librte_eal/linux/eal/eal_vfio.c
>>>>>> index 6892a2c14..f16c5c3c0 100644
>>>>>> --- a/lib/librte_eal/linux/eal/eal_vfio.c
>>>>>> +++ b/lib/librte_eal/linux/eal/eal_vfio.c
>>>>>> @@ -1448,9 +1448,29 @@ vfio_spapr_create_new_dma_window(int
>vfio_container_fd,
>>>>>>     	/* create new DMA window */
>>>>>>     	ret = ioctl(vfio_container_fd,
>VFIO_IOMMU_SPAPR_TCE_CREATE, create);
>>>>>>     	if (ret) {
>>>>>> -		RTE_LOG(ERR, EAL, "  cannot create new DMA window, "
>>>>>> -				"error %i (%s)\n", errno, strerror(errno));
>>>>>> -		return -1;
>>>>>> +		/* try possible page_shift and levels for workaround */
>>>>>> +		uint32_t levels;
>>>>>> +
>>>>>> +		for (levels = 1; levels <= info.ddw.levels; levels++) {
>>>>>> +			uint32_t pgsizes = info.ddw.pgsizes;
>>>>>
>>>>> +CC POWER maintainer from MAINTAINERS file.
>>>>>
>>>>> This is failing compilation on some older distros (Fedora 20,
>Ubuntu 14.04):
>>>>>
>>>>>
>>>>> /root/dpdk/lib/librte_eal/linux/eal/eal_vfio.c: In function
>>>>> ‘vfio_spapr_create_new_dma_window’:
>>>>> /root/dpdk/lib/librte_eal/linux/eal/eal_vfio.c:1451:34: error:
>‘struct
>>>>> vfio_iommu_spapr_tce_info’ has no member named ‘ddw’
>>>>> for (levels = 1; levels <= info.ddw.levels; levels++) {
>>>>> ^
>>>>> /root/dpdk/lib/librte_eal/linux/eal/eal_vfio.c:1452:27: error:
>‘struct
>>>>> vfio_iommu_spapr_tce_info’ has no member named ‘ddw’
>>>>> uint32_t pgsizes = info.ddw.pgsizes;
>>>>> ^
>>>>> /root/dpdk/lib/librte_eal/linux/eal/eal_vfio.c: At top level:
>>>>> cc1: error: unrecognized command line option
>>>>> "-Wno-address-of-packed-member" [-Werror]
>>>>> cc1: all warnings being treated as errors
>>>>> make[7]: *** [eal_vfio.o] Error 1
>>>>> make[7]: *** Waiting for unfinished jobs....
>>>>> make[6]: *** [eal] Error 2
>>>>> make[5]: *** [linux] Error 2
>>>>> make[4]: *** [librte_eal] Error 2
>>>>> make[3]: *** [lib] Error 2
>>>>> make[2]: *** [all] Error 2
>>>>> make[1]: *** [pre_install] Error 2
>>>>> make: *** [install] Error 2
>>>>>
>>>>>
>>>>> Looking at the eal_vfio.h we are handling case of missing sPAPR
>info/ddw
>>>>> structures, but this isn't simply a case of a missing struct;
>rather it
>>>>> is an issue of differing definitions between what recent kernels
>have,
>>>>> and what older kernels had.
>>>>>
>>>>> @Thomas Do we still support these older distros?
>>>>
>>>> If the kernel is not maintained, we don't support it.
>>>> For most distros, Linux < 3.16 is not maintained.
>>>
>>> This is on kernel 3.13 (Ubuntu 14.04 LTS which according to
>Canonical's
>>> published schedule [1] has stopped receiving even maintenance
>updates,
>>> and is only updated for select customers). It was never an LTS
>kernel to
>>> begin with, i think.
>> 
>> So we can ignore this issue.
>> 
>
>It seems that it's also reproducible on Ubuntu 14.04 with kernel
>3.19. 
>Which is not an upstream LTS kernel, but it's >3.16.
>
>It also looks like it may be an issue for us specifically, so i would
>
>suggest reverting the patch for rc2 if the submitter/maintainers
>don't 
>come back and fix it.
>
>-- 
>Thanks,
>Anatoly
>
>

Hi Anatoly,
Thank you for your report and analysis. This can be avoided by adding #ifdef VFIO_IOMMU_SPAPR_INFO_DDW.
I will quickly submit the fix soon after this email.

Regards,
Takeshi