Re: [dpdk-dev] [PATCH] eal: disable IOVA mode detection by default

["Tan, Jianfeng" <jianfeng.tan@intel.com>]

On 11/1/2017 2:54 PM, Jerin Jacob wrote:
> -----Original Message-----
>> Date: Tue, 31 Oct 2017 23:21:18 -0700
>> From: Ferruh Yigit <ferruh.yigit@intel.com>
>> To: Jerin Jacob <jerin.jacob@caviumnetworks.com>
>> CC: Thomas Monjalon <thomas@monjalon.net>, Bruce Richardson
>>   <bruce.richardson@intel.com>, Sergio Gonzalez Monroy
>>   <sergio.gonzalez.monroy@intel.com>, dev@dpdk.org, Jianfeng Tan
>>   <jianfeng.tan@intel.com>, Santosh Shukla
>>   <santosh.shukla@caviumnetworks.com>
>> Subject: Re: [dpdk-dev] [PATCH] eal: disable IOVA mode detection by default
>> User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101
>>   Thunderbird/52.4.0
>>
>> On 10/31/2017 9:02 PM, Jerin Jacob wrote:
>>> -----Original Message-----
>>>> Date: Wed, 1 Nov 2017 01:07:26 +0000
>>>> From: Ferruh Yigit <ferruh.yigit@intel.com>
>>>> To: Thomas Monjalon <thomas@monjalon.net>, Bruce Richardson
>>>>   <bruce.richardson@intel.com>, Sergio Gonzalez Monroy
>>>>   <sergio.gonzalez.monroy@intel.com>
>>>> CC: dev@dpdk.org, Ferruh Yigit <ferruh.yigit@intel.com>, Jianfeng Tan
>>>>   <jianfeng.tan@intel.com>, Santosh Shukla
>>>>   <santosh.shukla@caviumnetworks.com>
>>>> Subject: [dpdk-dev] [PATCH] eal: disable IOVA mode detection by default
>>>> X-Mailer: git-send-email 2.13.6
>>>>
>>>> Fix kernel crash with KNI because KNI requires physical addresses.
>>> The actual fix would be to make KNI IOMMU aware based on the DPDK mode.
>>>
>>> ie. On slow path,
>>>
>>> /* Get iommu domain for iova to physical addr conversion */
>>> if (rte_eal_iova_mode() == RTE_IOVA_VA)
>>> 	kni->iommu_domain = iommu_get_domain_for_dev(dev);
>>> else
>>> 	kni->iommu_domain = NULL;
>>>
>>> On fast path,
>>>
>>> static inline u64 kni_iova_to_phys(struct ... *kni, dma_addr_t dma_addr)
>>> {
>>>          /* Translation is installed only when IOMMU is present */
>>>          if (kni->iommu_domain)
>>>                  return iommu_iova_to_phys(kni->iommu_domain, dma_addr);
>>>          return dma_addr;
>>> }
>>>
>>>> A config option introduced to disable IOVA mode detection and to set it
>>>> to physical address by default. Disabling config option will enable IOVA
>>>> mode detection.
>>>>
>>>> When there is no intension to use KNI, it is safe to enable detection.
>>>>
>>>> Config option disable IOVA mode detection by default to be sure only who
>>>> is aware of result enable it.
>>>>
>>>> Fixes: 72d013644bd6 ("mem: honor IOVA mode in malloc virt2phy")
>>>>
>>>> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
>>>> ---
>>>> Cc: Jianfeng Tan <jianfeng.tan@intel.com>
>>>> Cc: Santosh Shukla <santosh.shukla@caviumnetworks.com>
>>>> Cc: Thomas Monjalon <thomas@monjalon.net>
>>>> ---
>>>>   config/common_base                | 5 +++++
>>>>   lib/librte_eal/bsdapp/eal/eal.c   | 4 ++++
>>>>   lib/librte_eal/linuxapp/eal/eal.c | 4 ++++
>>>>   3 files changed, 13 insertions(+)
>>>>
>>>> diff --git a/config/common_base b/config/common_base
>>>> index 82ee75456..903e7685b 100644
>>>> --- a/config/common_base
>>>> +++ b/config/common_base
>>>> @@ -107,6 +107,11 @@ CONFIG_RTE_MALLOC_DEBUG=n
>>>>   CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES=n
>>>>   
>>>>   #
>>>> +# Disabling PHYS_IOVA may crash kernel for KNI, use with caution
>>>> +#
>>>> +CONFIG_RTE_EAL_USE_PHYS_IOVA=y
>>> Defeat the purpose of all dynamic probing scheme.
>>> Either we can fix the KNI or revert the following patch for this release.
>>>
>>> http://dpdk.org/commit/f37dfab2
>> This commit just enables IOVA VA mode for Intel drivers, that is how I can able
>> to observe the issue, but it is not the source of the problem. Reverting that
>> commit will not solve KNI crash with any other PMD that enables IOVA VA mode.
> I don't understand why a PMD needs to enable IOVA_VA if it can support IOVA_PA.
> IMO, IOVA_VA should be enabled only for those device it can WORK ONLY on
> IOVA_VA mode.

IOVA_VA is also necessary if we don't want/have root privilege to run 
DPDK app.

> Forget about KNI, If we set CONFIG_RTE_EAL_USE_PHYS_IOVA
> as y then the normal stuff wont work for if PMD can operate only in
> IOVA_VA mode(like octeontx).

Yes, agreed.

> Regarding the KNI crash, it can be avoid by first checking the exiting
> mode(rte_eal_iova_mode()). i.e since legacy driver like KNI need real
> physical address to work "now", it can grace full exit on the init time if
> mode == IOVA_VA;

For the KNI crash, do we really need to correct the case:  make those 
PMDs (requires IOVA_VA)  also work well with KNI?
- If no, we just need to report  RTE_PCI_DRV_IOVA_AS_VA smartly.
- If yes, we shall make KNI work with VA.

Thanks,
Jianfeng