Re: [dpdk-dev] [PATCH] eal: disable IOVA mode detection by default

[Ferruh Yigit <ferruh.yigit@intel.com>]

On 10/31/2017 9:02 PM, Jerin Jacob wrote:
> -----Original Message-----
>> Date: Wed, 1 Nov 2017 01:07:26 +0000
>> From: Ferruh Yigit <ferruh.yigit@intel.com>
>> To: Thomas Monjalon <thomas@monjalon.net>, Bruce Richardson
>>  <bruce.richardson@intel.com>, Sergio Gonzalez Monroy
>>  <sergio.gonzalez.monroy@intel.com>
>> CC: dev@dpdk.org, Ferruh Yigit <ferruh.yigit@intel.com>, Jianfeng Tan
>>  <jianfeng.tan@intel.com>, Santosh Shukla
>>  <santosh.shukla@caviumnetworks.com>
>> Subject: [dpdk-dev] [PATCH] eal: disable IOVA mode detection by default
>> X-Mailer: git-send-email 2.13.6
>>
>> Fix kernel crash with KNI because KNI requires physical addresses.
> 
> The actual fix would be to make KNI IOMMU aware based on the DPDK mode.
> 
> ie. On slow path,
> 
> /* Get iommu domain for iova to physical addr conversion */             
> if (rte_eal_iova_mode() == RTE_IOVA_VA)
> 	kni->iommu_domain = iommu_get_domain_for_dev(dev);
> else
> 	kni->iommu_domain = NULL;
> 
> On fast path,
> 
> static inline u64 kni_iova_to_phys(struct ... *kni, dma_addr_t dma_addr)    
> {                                                                               
>         /* Translation is installed only when IOMMU is present */               
>         if (kni->iommu_domain)                                                  
>                 return iommu_iova_to_phys(kni->iommu_domain, dma_addr);         
>         return dma_addr;                                                        
> } 
> 
>>
>> A config option introduced to disable IOVA mode detection and to set it
>> to physical address by default. Disabling config option will enable IOVA
>> mode detection.
>>
>> When there is no intension to use KNI, it is safe to enable detection.
>>
>> Config option disable IOVA mode detection by default to be sure only who
>> is aware of result enable it.
>>
>> Fixes: 72d013644bd6 ("mem: honor IOVA mode in malloc virt2phy")
>>
>> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
>> ---
>> Cc: Jianfeng Tan <jianfeng.tan@intel.com>
>> Cc: Santosh Shukla <santosh.shukla@caviumnetworks.com>
>> Cc: Thomas Monjalon <thomas@monjalon.net>
>> ---
>>  config/common_base                | 5 +++++
>>  lib/librte_eal/bsdapp/eal/eal.c   | 4 ++++
>>  lib/librte_eal/linuxapp/eal/eal.c | 4 ++++
>>  3 files changed, 13 insertions(+)
>>
>> diff --git a/config/common_base b/config/common_base
>> index 82ee75456..903e7685b 100644
>> --- a/config/common_base
>> +++ b/config/common_base
>> @@ -107,6 +107,11 @@ CONFIG_RTE_MALLOC_DEBUG=n
>>  CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES=n
>>  
>>  #
>> +# Disabling PHYS_IOVA may crash kernel for KNI, use with caution
>> +#
>> +CONFIG_RTE_EAL_USE_PHYS_IOVA=y
> 
> Defeat the purpose of all dynamic probing scheme.
> Either we can fix the KNI or revert the following patch for this release.
> 
> http://dpdk.org/commit/f37dfab2

This commit just enables IOVA VA mode for Intel drivers, that is how I can able
to observe the issue, but it is not the source of the problem. Reverting that
commit will not solve KNI crash with any other PMD that enables IOVA VA mode.

Related to the KNI, iommu is not involved at all, I am not clear with your above
suggestion, physical address is required for kernelspace - userspace communication.

KNI uses physical address for two things:

1- Creates a buffer in userspace, a memzone, and shares its physical address to
the kernel, so that both kernel and app can access same buffer.

The question is, even all devices supports IOVA VA mode, why creating a memzone
that has real physical address info is not possible anymore?
For KNI case this is not related at all with what devices supports.

2- For each mbuf that will be sent to kernel, dpdk app puts physical address of
that mbuf into shared buffer, so that kernel can access it. So that kernel can
access to mbuf data that can be coming from any mempool.
For this the physical address of the mbuf is required.


Overall KNI needs to know physical address of mbufs and memzone.

This patch provides a way to have old behavior with a config option, so that KNI
can work and anyone who needs to create memzone with physical address can a way
to have it, and all PMDs will work fine.

If there is no need for these, it is possible to disable config and dynamic
probing scheme is already there.

Thanks,
ferruh