From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 764E6A0543; Thu, 7 Jul 2022 23:52:36 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 674384069D; Thu, 7 Jul 2022 23:52:36 +0200 (CEST) Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) by mails.dpdk.org (Postfix) with ESMTP id 7F4AD40041 for ; Thu, 7 Jul 2022 23:52:34 +0200 (CEST) Received: by mail-pj1-f43.google.com with SMTP id t5-20020a17090a6a0500b001ef965b262eso102748pjj.5 for ; Thu, 07 Jul 2022 14:52:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20210112.gappssmtp.com; s=20210112; h=date:from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=eS0pnQKwXkP2lY+MnbvWPlKXhMbbi4PxdlSodMn8oIU=; b=lodE65LSRFZ1XHWw+NbrMG0Z7s1Dyp7pZRx6Zo9+3OrBIs0rbaizWYKTC4F059K7FS AzD/SfOYjrhJmlo8QRLSrP7bCz/rN9U4wCvckCUmrjFhZU7nBdqKoGcRGjtOvXg/qRuJ nBgTgnkW7X8uwsPWn5j41lNIGz5j66YfPD9sF7ChvwyVrRP8ixsYHAGIU6TJ6d4bWdrY s1HD96d1QwKlCfBUY6ixhIa9uL9j1LOcjCdrXHxvucgrvZjjkCy1RBikrZm8NdT+mimh MXVte14pJqAI57QAYApFKGMpiyX3B4++eAPTSwGQz6T8OX4U9wa70DVPA6pg5B9vg+NL o1Qg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=eS0pnQKwXkP2lY+MnbvWPlKXhMbbi4PxdlSodMn8oIU=; b=MAbiAZi5HPY78KUu2CXxo9w244sFxCzN4M4DWsPfyUPnQPLYQxVQa13fzBP3wDNvOh VFj/OTTw3m6SyLVsU5BEmxONEUsdFwTqIhHOavdY3PGu+rUxzEmk/G+B9ZDn+iqH+k/O bn6NtGUKeoyDS0X/z/1w7aiZhzswupTDkhhfrckAGq44gmE2/WxP15xSfnEE6DcsPif3 WXcbXtrlpuG1lxZzFNb+/fJLF9gYYUtguGF4hVWrXW4X2wn9ptEXjoA+85E4yxOsVBh4 IryrigD57FnnXJ9wCbi8snw1b/5Y76kooOybjpJrYqbiFLBXFy2mfHf0SFGn7kEoBbo2 XfoQ== X-Gm-Message-State: AJIora9B0HCveX4uca5gFREeYIC/lUKuOUbOJCR3FOl18XHy997YNrI7 in6qQiYSmcRfKY8jBpGG8fajtQ== X-Google-Smtp-Source: AGRyM1sPWgNqGUIMTN9wRi8k6PJcfEe6uh4WS1+Gq1Q1DzlQ/qul+H5FzU5UaminZFgVwZkKi3/DYA== X-Received: by 2002:a17:90b:4acb:b0:1ed:fef:5657 with SMTP id mh11-20020a17090b4acb00b001ed0fef5657mr7460633pjb.142.1657230753776; Thu, 07 Jul 2022 14:52:33 -0700 (PDT) Received: from hermes.local (204-195-112-199.wavecable.com. [204.195.112.199]) by smtp.gmail.com with ESMTPSA id t15-20020aa7946f000000b0052877d6f66bsm2892758pfq.50.2022.07.07.14.52.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Jul 2022 14:52:33 -0700 (PDT) Date: Thu, 7 Jul 2022 14:52:31 -0700 From: Stephen Hemminger To: longli@linuxonhyperv.com Cc: longli@microsoft.com, Ferruh Yigit , dev@dpdk.org, Ajay Sharma , Stephen Hemminger Subject: Re: [Patch v3 01/17] net/mana: add basic driver, build environment and doc Message-ID: <20220707145231.27b678d8@hermes.local> In-Reply-To: <1657225822-3512-2-git-send-email-longli@linuxonhyperv.com> References: <1657225822-3512-1-git-send-email-longli@linuxonhyperv.com> <1657225822-3512-2-git-send-email-longli@linuxonhyperv.com> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Thu, 7 Jul 2022 13:30:06 -0700 longli@linuxonhyperv.com wrote: > +static int mana_pci_probe_mac(struct rte_pci_driver *pci_drv __rte_unused, > + struct rte_pci_device *pci_dev, > + struct rte_ether_addr *mac_addr) > +{ > + struct ibv_device **ibv_list; > + int ibv_idx; > + struct ibv_context *ctx; > + struct ibv_device_attr_ex dev_attr; > + int num_devices; > + int ret = 0; > + uint8_t port; > + struct mana_priv *priv = NULL; > + struct rte_eth_dev *eth_dev = NULL; > + bool found_port; > + > + ibv_list = ibv_get_device_list(&num_devices); > + for (ibv_idx = 0; ibv_idx < num_devices; ibv_idx++) { > + struct ibv_device *ibdev = ibv_list[ibv_idx]; > + struct rte_pci_addr pci_addr; > + > + DRV_LOG(INFO, "Probe device name %s dev_name %s ibdev_path %s", > + ibdev->name, ibdev->dev_name, ibdev->ibdev_path); > + > + if (mana_ibv_device_to_pci_addr(ibdev, &pci_addr)) > + continue; > + > + /* Ignore if this IB device is not this PCI device */ > + if (pci_dev->addr.domain != pci_addr.domain || > + pci_dev->addr.bus != pci_addr.bus || > + pci_dev->addr.devid != pci_addr.devid || > + pci_dev->addr.function != pci_addr.function) > + continue; > + > + ctx = ibv_open_device(ibdev); > + if (!ctx) { > + DRV_LOG(ERR, "Failed to open IB device %s", > + ibdev->name); > + continue; > + } > + > + ret = ibv_query_device_ex(ctx, NULL, &dev_attr); > + DRV_LOG(INFO, "dev_attr.orig_attr.phys_port_cnt %u", > + dev_attr.orig_attr.phys_port_cnt); > + found_port = false; > + > + for (port = 1; port <= dev_attr.orig_attr.phys_port_cnt; > + port++) { > + struct ibv_parent_domain_init_attr attr = {}; > + struct rte_ether_addr addr; > + char address[64]; > + char name[RTE_ETH_NAME_MAX_LEN]; > + > + ret = get_port_mac(ibdev, port, &addr); > + if (ret) > + continue; > + > + if (mac_addr && !rte_is_same_ether_addr(&addr, mac_addr)) > + continue; > + > + rte_ether_format_addr(address, sizeof(address), &addr); > + DRV_LOG(INFO, "device located port %u address %s", > + port, address); > + found_port = true; > + > + priv = rte_zmalloc_socket(NULL, sizeof(*priv), > + RTE_CACHE_LINE_SIZE, > + SOCKET_ID_ANY); > + if (!priv) { > + ret = -ENOMEM; > + goto failed; > + } > + > + snprintf(name, sizeof(name), "%s_port%d", > + pci_dev->device.name, port); > + > + if (rte_eal_process_type() == RTE_PROC_SECONDARY) { > + int fd; > + > + eth_dev = rte_eth_dev_attach_secondary(name); > + if (!eth_dev) { > + DRV_LOG(ERR, "Can't attach to dev %s", > + name); > + ret = -ENOMEM; > + goto failed; > + } > + > + eth_dev->device = &pci_dev->device; > + eth_dev->dev_ops = &mana_dev_sec_ops; > + ret = mana_proc_priv_init(eth_dev); > + if (ret) > + goto failed; > + priv->process_priv = eth_dev->process_private; > + > + /* Get the IB FD from the primary process */ > + fd = mana_mp_req_verbs_cmd_fd(eth_dev); > + if (fd < 0) { > + DRV_LOG(ERR, "Failed to get FD %d", fd); > + ret = -ENODEV; > + goto failed; > + } > + > + ret = mana_map_doorbell_secondary(eth_dev, fd); > + if (ret) { > + DRV_LOG(ERR, "Failed secondary map %d", > + fd); > + goto failed; > + } > + > + /* fd is no not used after mapping doorbell */ > + close(fd); > + > + rte_spinlock_lock(&mana_shared_data->lock); > + mana_shared_data->secondary_cnt++; > + mana_local_data.secondary_cnt++; > + rte_spinlock_unlock(&mana_shared_data->lock); > + > + rte_eth_copy_pci_info(eth_dev, pci_dev); > + rte_eth_dev_probing_finish(eth_dev); > + > + /* Impossible to have more than one port > + * matching a MAC address > + */ > + continue; > + } > + > + eth_dev = rte_eth_dev_allocate(name); > + if (!eth_dev) { > + ret = -ENOMEM; > + goto failed; > + } > + > + eth_dev->data->mac_addrs = > + rte_calloc("mana_mac", 1, > + sizeof(struct rte_ether_addr), 0); > + if (!eth_dev->data->mac_addrs) { > + ret = -ENOMEM; > + goto failed; > + } > + > + rte_ether_addr_copy(&addr, eth_dev->data->mac_addrs); > + > + priv->ib_pd = ibv_alloc_pd(ctx); > + if (!priv->ib_pd) { > + DRV_LOG(ERR, "ibv_alloc_pd failed port %d", port); > + ret = -ENOMEM; > + goto failed; > + } > + > + /* Create a parent domain with the port number */ > + attr.pd = priv->ib_pd; > + attr.comp_mask = IBV_PARENT_DOMAIN_INIT_ATTR_PD_CONTEXT; > + attr.pd_context = (void *)(uint64_t)port; > + priv->ib_parent_pd = ibv_alloc_parent_domain(ctx, &attr); > + if (!priv->ib_parent_pd) { > + DRV_LOG(ERR, > + "ibv_alloc_parent_domain failed port %d", > + port); > + ret = -ENOMEM; > + goto failed; > + } > + > + priv->ib_ctx = ctx; > + priv->port_id = eth_dev->data->port_id; > + priv->dev_port = port; > + eth_dev->data->dev_private = priv; > + priv->dev_data = eth_dev->data; > + > + priv->max_rx_queues = dev_attr.orig_attr.max_qp; > + priv->max_tx_queues = dev_attr.orig_attr.max_qp; > + > + priv->max_rx_desc = > + RTE_MIN(dev_attr.orig_attr.max_qp_wr, > + dev_attr.orig_attr.max_cqe); > + priv->max_tx_desc = > + RTE_MIN(dev_attr.orig_attr.max_qp_wr, > + dev_attr.orig_attr.max_cqe); > + > + priv->max_send_sge = dev_attr.orig_attr.max_sge; > + priv->max_recv_sge = dev_attr.orig_attr.max_sge; > + > + priv->max_mr = dev_attr.orig_attr.max_mr; > + priv->max_mr_size = dev_attr.orig_attr.max_mr_size; > + > + DRV_LOG(INFO, "dev %s max queues %d desc %d sge %d\n", > + name, priv->max_rx_queues, priv->max_rx_desc, > + priv->max_send_sge); > + Message will end up double spaced, since DRV_LOG already adds newline. You might also want to use %u rather than %d when the values are unsigned.