From: Bruce Richardson <bruce.richardson@intel.com>
To: santosh <santosh.shukla@caviumnetworks.com>
Cc: thomas@monjalon.net, dev@dpdk.org,
jerin.jacob@caviumnetworks.com, hemant.agrawal@nxp.com
Subject: Re: [dpdk-dev] [RFC] eal/memory: introducing an option to set iova as va
Date: Tue, 6 Jun 2017 10:57:20 +0100 [thread overview]
Message-ID: <20170606095719.GA50888@bricha3-MOBL3.ger.corp.intel.com> (raw)
In-Reply-To: <e2aec279-2c50-135c-ba07-c21867a1bb8f@caviumnetworks.com>
On Mon, Jun 05, 2017 at 10:24:11AM +0530, santosh wrote:
> Hi Bruce,
>
>
> On Friday 02 June 2017 02:57 PM, Bruce Richardson wrote:
> > On Fri, Jun 02, 2017 at 09:54:46AM +0530, santosh wrote:
> >> Ping?
> >>
> >> On Wednesday 24 May 2017 09:41 PM, Santosh Shukla wrote:
> >>
> >>> Some NPU hardware like OCTEONTX follows push model to get
> >>> the packet from the pktio device. Where packet allocation
> >>> and freeing done by the HW. Since HW can operate only on
> >>> IOVA with help of SMMU/IOMMU, When packet receives from the
> >>> Ethernet device, It is the IOVA address(which is PA in existing scheme).
> >>>
> >>> Mapping IOVA as PA is expensive on those HW, where every
> >>> packet needs to be converted to VA from PA/IOVA.
> >>>
> >>> This patch proposes the scheme where the user can set IOVA
> >>> as VA by using an eal command line argument. That helps to
> >>> avoid costly lookup for VA in SW by leveraging the SMMU
> >>> translation feature.
> >>>
> >>> Signed-off-by: Santosh Shukla <santosh.shukla@caviumnetworks.com>
> >>> ---
> > Hi,
> >
> > I agree this is a problem that needs to be solved, but this doesn't look
> > like a particularly future-proofed solution. Given that we should
> > use the IOMMU on as many platforms as possible for protection, we
> > probably need to find an automatic way for DPDK to use IO addresses
> > correctly. Is this therefore better done as part of the VFIO and
> > UIO-specific code in EAL - as that is the part that knows how the memory
> > mapping is done, and in the VFIO case, what address ranges were
> > programmed in. The mempool driver was something else I considered but it
> > is probably too high a level to implement this.
>
> The other approach which we evaluated, Its detail:
> 0) Introduce a new bus api whose job is to detect iommu capable devices on that
> bus {/ are those devices bind to iommu capable driver or not?}. Let's call that
> api rte_bus_chk_iommu_dev();
>
> 1) The scheme is like If _all_ the devices bind to iommu kdrv then return iova=va
> 2) Otherwise switch to default mode i.e.. iova=pa.
> 3) Based on rte_bus_chk_iommu_dev() return value,
> accordingly program iova=va Or iova=pa in vfio_type1/spapr_map().
>
> 4) User from the command line can always override iova=va,
> in case if he wants to default scheme( iova=pa mode). For that purpose - Introduce eal
> option something like --iova-pa Or --override-iova Or --iova-default
> or some better name.
>
> Proposed API snap:
>
> enum iova_mode {
> iova_va;
> iova_pa;
> iova_unknown;
> };
>
> /**
> * Look for iommu devices on that Bus.
> * And find out that those devices bind to iommu
> * capable driver example vfio.
> *
> *
> * @return
> * On success return valid iova mode (iova_va or iova_pa)
> * On failure return iova_unkown.
> */
> typedef int (*rte_bus_chk_iommu_dev_t)(void);
>
>
> By this approach,
> - We can automatically detect iova is va or pa
> and then program accordingly.
> - Also, the user can always switch to default iova mode.
> - Drivers like dpaa2 can use this API to detect iova mode then
> program dma_map accordingly. Currently they are doing in ifdef-way.
>
> Comments? thoughts? Or if anyone has better proposal then, please
> suggest.
>
That sounds a more complete solution. However, it's probably a lot of
work to implement. :-)
I also wonder if we want to simplify things a little and disallow
mixed-mode operation i.e. all devices have to use UIO or all use VFIO?
Would that help to allow simplification or other options. Having a whole
new bus type seems strange for this. Can each bus just report whether
it's members require physical addresses. Then the EAL can manage a
single flag to report whether we are using VA or PA?
/Bruce
next prev parent reply other threads:[~2017-06-06 9:57 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-05-24 16:11 Santosh Shukla
2017-06-02 4:24 ` santosh
2017-06-02 9:27 ` Bruce Richardson
2017-06-05 4:54 ` santosh
2017-06-06 9:57 ` Bruce Richardson [this message]
2017-06-06 10:13 ` Gaëtan Rivet
2017-06-06 10:41 ` Jerin Jacob
2017-06-06 10:38 ` Jerin Jacob
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20170606095719.GA50888@bricha3-MOBL3.ger.corp.intel.com \
--to=bruce.richardson@intel.com \
--cc=dev@dpdk.org \
--cc=hemant.agrawal@nxp.com \
--cc=jerin.jacob@caviumnetworks.com \
--cc=santosh.shukla@caviumnetworks.com \
--cc=thomas@monjalon.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).