Re: [dpdk-dev] [PATCH v6 3/6] raw/ntb: add handshake process

["Wu, Jingjing" <jingjing.wu@intel.com>]

> -----Original Message-----
> From: Li, Xiaoyun
> Sent: Tuesday, June 18, 2019 10:11 AM
> To: Wu, Jingjing <jingjing.wu@intel.com>; Wiles, Keith <keith.wiles@intel.com>; Liang,
> Cunming <cunming.liang@intel.com>; Maslekar, Omkar <omkar.maslekar@intel.com>
> Cc: dev@dpdk.org; Li, Xiaoyun <xiaoyun.li@intel.com>
> Subject: [PATCH v6 3/6] raw/ntb: add handshake process
> 
> Add handshake process using doorbell so that two hosts can
> communicate to start and stop.
> 
> Signed-off-by: Xiaoyun Li <xiaoyun.li@intel.com>
> ---
>  drivers/raw/ntb_rawdev/ntb_rawdev.c | 336 +++++++++++++++++++++++++++-
>  1 file changed, 335 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/raw/ntb_rawdev/ntb_rawdev.c
> b/drivers/raw/ntb_rawdev/ntb_rawdev.c
> index a03decd55..d9088e825 100644
> --- a/drivers/raw/ntb_rawdev/ntb_rawdev.c
> +++ b/drivers/raw/ntb_rawdev/ntb_rawdev.c
> @@ -28,6 +28,183 @@ static const struct rte_pci_id pci_id_ntb_map[] = {
>  	{ .vendor_id = 0, /* sentinel */ },
>  };
> 
> +static int
> +ntb_set_mw(struct rte_rawdev *dev, int mw_idx, uint64_t mw_size)
> +{
> +	struct ntb_hw *hw = dev->dev_private;
> +	char mw_name[RTE_MEMZONE_NAMESIZE];
> +	const struct rte_memzone *mz;
> +	int ret = 0;
> +
> +	if (hw->ntb_ops->mw_set_trans == NULL) {
> +		NTB_LOG(ERR, "Not supported to set mw.");
> +		return -ENOTSUP;
> +	}
> +
> +	snprintf(mw_name, sizeof(mw_name), "ntb_%d_mw_%d",
> +		 dev->dev_id, mw_idx);
> +
> +	mz = rte_memzone_lookup(mw_name);
> +	if (mz)
> +		return 0;
> +
> +	/**
> +	 * Hardware requires that mapped memory base address should be
> +	 * aligned with EMBARSZ and needs continuous memzone.
> +	 */
> +	mz = rte_memzone_reserve_aligned(mw_name, mw_size, dev->socket_id,
> +				RTE_MEMZONE_IOVA_CONTIG, hw->mw_size[mw_idx]);

If the memzone is reserved inside of driver, how is the buffer be mapped without copy when enqueuer/dequeuer as the buffer might not be in the memzone?
How about to design the dev_config to set the mw to be a memzone (might address + size) which can be created by application instead of created internally?

[.....]
> +static void
> +ntb_dev_intr_handler(void *param)
> +{
> +	struct rte_rawdev *dev = (struct rte_rawdev *)param;
> +	struct ntb_hw *hw = dev->dev_private;
> +	uint32_t mw_size_h, mw_size_l;
> +	uint64_t db_bits = 0;
> +	int i = 0;
> +
> +	if (hw->ntb_ops->db_read == NULL ||
> +	    hw->ntb_ops->db_clear == NULL ||
> +	    hw->ntb_ops->peer_db_set == NULL) {
> +		NTB_LOG(ERR, "Doorbell is not supported.");
> +		return;
> +	}
> +
> +	db_bits = (*hw->ntb_ops->db_read)(dev);
> +	if (!db_bits)
> +		NTB_LOG(ERR, "No doorbells");
> +

Is the db_bits a common setting between different kind of NTB?

[......]

> 
> @@ -356,7 +608,9 @@ static int
>  ntb_init_hw(struct rte_rawdev *dev, struct rte_pci_device *pci_dev)
>  {
>  	struct ntb_hw *hw = dev->dev_private;
> -	int ret;
> +	struct rte_intr_handle *intr_handle;
> +	uint32_t val;
> +	int ret, i;
> 
>  	hw->pci_dev = pci_dev;
>  	hw->peer_dev_up = 0;
> @@ -387,6 +641,86 @@ ntb_init_hw(struct rte_rawdev *dev, struct rte_pci_device
> *pci_dev)
>  	if (ret)
>  		return ret;
> 
> +	/* Init doorbell. */
> +	hw->db_valid_mask = ((uint64_t)1 << hw->db_cnt) - 1;
Use RTE_LEN2MASK instead?

Thanks
Jingjing