Re: [dpdk-dev] [PATCHv3 1/2] config/armv8a: disable igb_uio

[Jerin Jacob <jerin.jacob@caviumnetworks.com>]

On Fri, May 13, 2016 at 03:37:01AM +0000, Hemant Agrawal wrote:
> 
> 
> > -----Original Message-----
> > From: Jianbo Liu [mailto:jianbo.liu@linaro.org]
> > Sent: Friday, May 13, 2016 7:13 AM
> > To: Santosh Shukla <santosh.shukla@caviumnetworks.com>
> > Cc: Stephen Hemminger <stephen@networkplumber.org>; Jerin Jacob
> > <jerin.jacob@caviumnetworks.com>; Hemant Agrawal
> > <hemant.agrawal@nxp.com>; dev@dpdk.org; Thomas Monjalon
> > <thomas.monjalon@6wind.com>
> > Subject: Re: [dpdk-dev] [PATCHv3 1/2] config/armv8a: disable igb_uio
> > 
> > On 12 May 2016 at 18:31, Santosh Shukla
> > <santosh.shukla@caviumnetworks.com> wrote:
> > > On Thu, May 12, 2016 at 05:52:54PM +0800, Jianbo Liu wrote:
> > >> On 12 May 2016 at 16:57, Santosh Shukla
> > >> <santosh.shukla@caviumnetworks.com> wrote:
> > >> > On Thu, May 12, 2016 at 01:54:13PM +0800, Jianbo Liu wrote:
> > >> >> On 12 May 2016 at 13:06, Santosh Shukla
> > >> >> <santosh.shukla@caviumnetworks.com> wrote:
> > >> >> > On Thu, May 12, 2016 at 11:42:26AM +0800, Jianbo Liu wrote:
> > >> >> >> On 12 May 2016 at 11:17, Santosh Shukla
> > >> >> >> <santosh.shukla@caviumnetworks.com> wrote:
> > >> >> >> > On Thu, May 12, 2016 at 10:01:05AM +0800, Jianbo Liu wrote:
> > >> >> >> >> On 12 May 2016 at 02:25, Stephen Hemminger
> > <stephen@networkplumber.org> wrote:
> > >> >> >> >> > On Wed, 11 May 2016 22:32:16 +0530 Jerin Jacob
> > >> >> >> >> > <jerin.jacob@caviumnetworks.com> wrote:
> > >> >> >> >> >
> > >> >> >> >> >> On Wed, May 11, 2016 at 08:22:59AM -0700, Stephen
> > Hemminger wrote:
> > >> >> >> >> >> > On Wed, 11 May 2016 19:17:58 +0530 Hemant Agrawal
> > >> >> >> >> >> > <hemant.agrawal@nxp.com> wrote:
> > >> >> >> >> >> >
> > >> >> >> >> >> > > IGB_UIO not supported for arm64 arch in kernel so disable.
> > >> >> >> >> >> > >
> > >> >> >> >> >> > > Signed-off-by: Hemant Agrawal
> > >> >> >> >> >> > > <hemant.agrawal@nxp.com>
> > >> >> >> >> >> > > Reviewed-by: Santosh Shukla
> > >> >> >> >> >> > > <santosh.shukla@caviumnetworks.com>
> > >> >> >> >> >> >
> > >> >> >> >> >> > Really, I have use IGB_UIO on ARM64
> > >> >> >> >> >>
> > >> >> >> >> >> May I know what is the technical use case for igb_uio on
> > >> >> >> >> >> arm64 which cannot be addressed through vfio or vfioionommu.
> > >> >> >> >> >
> > >> >> >> >> > I was running on older kernel which did not support vfioionommu
> > mode.
> > >> >> >> >>
> > >> >> >> >> As I said, most of DPDK developers are not kernel
> > >> >> >> >> developers. They may have their own kernel tree, and
> > >> >> >> >> couldn't like to upgrade to latest kernel.
> > >> >> >> >> They can choose to use or not use igb_uio when binding the
> > >> >> >> >> driver. But blindly disabling it in the base config seems unreasonable.
> > >> >> >> >
> > >> >> >> > if user keeping his own kernel so they could also keep
> > >> >> >> > IGB_UIO=y in their local
> > >> >> >> Most likely they don't have local dpdk tree. They write their
> > >> >> >> own applications, complie and link to dpdk lib, then done.
> > >> >> >>
> > >> >> >> > dpdk tree. Why are you imposing user-x custome depedancy on
> > >> >> >> > upstream dpdk base
> > >> >> >> Customer requiremnts is important. I want they can choose the way
> > they like.
> > >> >> >>
> > >> >> >
> > >> >> > so you choose to keep igb_uio option, provided arch doesn't support?
> > >> >> > new user did reported issues with igb_uio for arm64, refer this
> > >> >> > thread [1], as well hemanth too faced issues. we want to avoid that.
> > >> >> >
> > >> >> > If customer maintaing out-of-tree kernel then he can also switch to vfio-
> > way.
> > >> >> > isn;t it?
> > >> >> >
> > >> >> >> > config. Is it not enough for explanation that - Base config
> > >> >> >> > ie.. armv8 doesn;t support pci mmap, so igb_uio is n/a. New
> > >> >> >> > user wont able to build/run dpdk/arm64 in igb_uio-way, He'll
> > >> >> >> > prefer to use upstream stuff. I think, you are not making
> > >> >> >> You are wrong, he can build dpdk. If he like to use upstream
> > >> >> >> without patching, he can use vfio.
> > >> >> >
> > >> >> > I disagree, we want to avoid [1] for new user.
> > >> >> >
> > >> >> >> But you can't ignore the need from old user which is more
> > >> >> >> comfortable with older kernel.
> > >> >> >>
> > >> >> > arm/arm64 dpdk support recently added and I am guessing, most
> > >> >> > likely customer using near latest kernel, switching to vfio won't be so
> > difficult.
> > >> >> >
> > >> >> > Or can you take up responsibility of upstreaming pci mmap patch,
> > >> >> > then we don't need this patch.
> > >> >> >
> > >> >> > [1] http://dpdk.org/ml/archives/dev/2016-January/031313.html
> > >> >>
> > >> >> Can you read carefully about the guide at
> > >> >> http://dpdk.org/doc/guides/linux_gsg/build_dpdk.html? It says to
> > >> >> use uio_pci_generic, igb_uio or vfio-pci.
> > >> >
> > >> > *** applicable and works for x86 only, not for arm64: because pci
> > >> > mmap support not present for arm64, in that case we should update the
> > doc.
> > >> >
> > >> >> Could it be possible that the user in that thread has already read
> > >> >> and tried them all and found that he can't enable vifo with his
> > >> >> kernel, and igb_uio is the easy way for him and asked for help from
> > community?
> > >> >> If so, we have no choice but keeping igb_uio enabled.
> > >> >
> > >> > By then vfionoiommu support was wip progress in dpdk/linux. but now
> > >> > it merged and it works. So no need to retain igb_uio in base config
> > >> > for which to work - user need to use mmap patch at linux side.
> > >>
> > >> We can't decide which kernel user will use.
> > >>
> > >
> > > yes, we can't decide kernel for user but we should be explicit to user
> > > on - what works for dpdk/linux out-of-box vs what could work with use
> > > of out-of-tree patch/RFC's.. example igb_uio.
> > >
> > 
> > OK, please persuade Stephen Hemminger and the other guy to use upstream
> > kernel first!
> 
> [Hemant] It is just a matter for choice for arm64 maintainers.  You only have two choices
> 1. with the default arm64 support in DPDK, you allow it to be used with any kernel seamlessly. If they need any specific function, they can enable/disable it. 
> 2. Or, you maintain specific support in the default config, and force all others to manually disable it/or, disable it via their specific configuration.
> 
> I was not intending to change the default arm64 config in my original patch, I rather used it to disable it via NXP specific platform config.
> I still agree with Santosh that a new arm64 user should not be worry about searching  & patching the kernel. 
> 
> 
> In any case, who is going to take a call here?  

My take is to disable for arm64 to make sync with upstream kernel.
If Jianbo still feel it is better to keep for arm64 then lets disable it for
NXP and ThunderX and keep it enable for arm64 def config.

Thanks,
Jerin

> > 
> > >> >
> > >> > Or can you maintain out-of-tree pci mmap patch/ kerne source and
> > >> > make it explicit somewhere in dpdk build doc that - if user want
> > >> > igb_uio way then use kernel/mmap patch from x location.
> > >>
> > >> The patch is in the kernel maillist, and user google it.
> > >
> > > there are feature specific rfc's in plenty in lkml/qemu mailing list,
> > > and you
> > > suggest- user to hunt for all those information. Is this how we;re
> > > officially supporting igb_uio for arm64.. that let user to google?
> > >
> > 
> > Sorry I don't know you are offically support users here.
> > And you also don't know what they really want.
> > 
> > >> And isn't funny to ask someone to do something again and again (3
> > >> times) in this thread?
> > >>
> > >
> > > I am asking becasue your in favour of keeping igb_uio for arm64 but
> > > not agreeing to streamline (writing a note in dpdk doc for igb_uio for
> > > arm64 or pointing to working tree).. so that user don;t need to grep
> > > or google for known findings.
> > >
> > > I find discussion going in circle and nothing will conclude, So given up.
> > >> >
> > >> >> He use lsmod to show us the modules, most likely he know vifo-pci.
> > >> >>
> > >> >> Below are the details on modules, hugepages and device binding.
> > >> >> root at arm64:~# lsmod
> > >> >> Module                  Size  Used by
> > >> >> rte_kni               292795  0
> > >> >> igb_uio                 4338  0
> > >> >> ixgbe                 184456  0