From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by dpdk.org (Postfix) with ESMTP id 7D9298071 for ; Mon, 1 Dec 2014 18:11:47 +0100 (CET) Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga102.fm.intel.com with ESMTP; 01 Dec 2014 09:10:20 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.07,494,1413270000"; d="scan'208";a="630866012" Received: from irvmail001.ir.intel.com ([163.33.26.43]) by fmsmga001.fm.intel.com with ESMTP; 01 Dec 2014 09:10:16 -0800 Received: from sivswdev02.ir.intel.com (sivswdev02.ir.intel.com [10.237.217.46]) by irvmail001.ir.intel.com (8.14.3/8.13.6/MailSET/Hub) with ESMTP id sB1HAG3q022606; Mon, 1 Dec 2014 17:10:16 GMT Received: from sivswdev02.ir.intel.com (localhost [127.0.0.1]) by sivswdev02.ir.intel.com with ESMTP id sB1HAGkc014700; Mon, 1 Dec 2014 17:10:16 GMT Received: (from dwdohert@localhost) by sivswdev02.ir.intel.com with id sB1HAFSd014696; Mon, 1 Dec 2014 17:10:15 GMT From: Declan Doherty To: dev@dpdk.org Date: Mon, 1 Dec 2014 17:10:12 +0000 Message-Id: <1417453812-14599-1-git-send-email-declan.doherty@intel.com> X-Mailer: git-send-email 1.7.12.2 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Subject: [dpdk-dev] =?utf-8?q?=5BPATCH=5D_doc=3A_link_bonding_related_upda?= =?utf-8?q?tes_to_programmers_guide?= X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Dec 2014 17:11:49 -0000 Adding details for link status interrupts and link status polling. Adding details for mode 4 / mode 5 Tidying up rst document to conform to 80 character line limit Adding diagrams to explain bonding modes Signed-off-by: Declan Doherty --- doc/guides/prog_guide/img/bond-mode-0.svg | 638 +++++++++++++++++ doc/guides/prog_guide/img/bond-mode-1.svg | 724 +++++++++++++++++++ doc/guides/prog_guide/img/bond-mode-2.svg | 702 ++++++++++++++++++ doc/guides/prog_guide/img/bond-mode-3.svg | 702 ++++++++++++++++++ doc/guides/prog_guide/img/bond-mode-4.svg | 784 +++++++++++++++++++++ doc/guides/prog_guide/img/bond-mode-5.svg | 642 +++++++++++++++++ doc/guides/prog_guide/img/bond-overview.svg | 121 ++++ .../prog_guide/link_bonding_poll_mode_drv_lib.rst | 366 +++++++--- 8 files changed, 4579 insertions(+), 100 deletions(-) create mode 100644 doc/guides/prog_guide/img/bond-mode-0.svg create mode 100644 doc/guides/prog_guide/img/bond-mode-1.svg create mode 100644 doc/guides/prog_guide/img/bond-mode-2.svg create mode 100644 doc/guides/prog_guide/img/bond-mode-3.svg create mode 100644 doc/guides/prog_guide/img/bond-mode-4.svg create mode 100644 doc/guides/prog_guide/img/bond-mode-5.svg create mode 100644 doc/guides/prog_guide/img/bond-overview.svg diff --git a/doc/guides/prog_guide/img/bond-mode-0.svg b/doc/guides/prog_guide/img/bond-mode-0.svg new file mode 100644 index 0000000..eff0edb --- /dev/null +++ b/doc/guides/prog_guide/img/bond-mode-0.svg @@ -0,0 +1,638 @@ + + + +image/svg+xmlPage-4Rectangle.7User ApplicationUser Application +Sheet.2Rectangle.38DPDKDPDK +Rectangle.16bonded ethdevbonded ethdev +Rectangle.11ethdev portethdev port +Rectangle.14ethdev portethdev port +Rectangle.15ethdev portethdev port +Simple Double Arrow.14Simple ArrowSimple Arrow.37Simple Arrow.38Simple Arrow.39Square.11411 +Square.11522 +Square.11633 +Square.11744 +Square.11855 +Square.12011 +Square.12122 +Square.12233 +Square.12344 +Square.12455 + \ No newline at end of file diff --git a/doc/guides/prog_guide/img/bond-mode-1.svg b/doc/guides/prog_guide/img/bond-mode-1.svg new file mode 100644 index 0000000..c177e85 --- /dev/null +++ b/doc/guides/prog_guide/img/bond-mode-1.svg @@ -0,0 +1,724 @@ + + + +image/svg+xmlPage-4Rectangle.40User ApplicationUser Application +Sheet.40Rectangle.38DPDKDPDK +Rectangle.16bonded ethdevbonded ethdev +Rectangle.11ethdev portethdev port +Rectangle.14ethdev portethdev port +Rectangle.15ethdev portethdev port +Simple Double Arrow.47Simple Arrow.47Simple Arrow.49Square.10811 +Square.10922 +Square.11033 +Square.11111 +Square.11222 +Square.11333 + \ No newline at end of file diff --git a/doc/guides/prog_guide/img/bond-mode-2.svg b/doc/guides/prog_guide/img/bond-mode-2.svg new file mode 100644 index 0000000..3dbb598 --- /dev/null +++ b/doc/guides/prog_guide/img/bond-mode-2.svg @@ -0,0 +1,702 @@ + + + +image/svg+xmlPage-4Rectangle.151User ApplicationUser Application +Sheet.56Rectangle.38DPDKDPDK +Rectangle.16bonded ethdevbonded ethdev +Rectangle.11ethdev portethdev port +Rectangle.14ethdev portethdev port +Rectangle.15ethdev portethdev port +Simple Double Arrow.158Simple Arrow.159Simple Arrow.160Simple Arrow.161Simple Arrow.162Square.16311 +Square.16422 +Square.16533 +Square.16644 +Square.16755 +Square.16866 +Sheet.73Square.17222 +Square.17344 +Square.17566 +Sheet.77Square.16911 +Square.17033 +Square.17155 + \ No newline at end of file diff --git a/doc/guides/prog_guide/img/bond-mode-3.svg b/doc/guides/prog_guide/img/bond-mode-3.svg new file mode 100644 index 0000000..d2dbe3a --- /dev/null +++ b/doc/guides/prog_guide/img/bond-mode-3.svg @@ -0,0 +1,702 @@ + + + +image/svg+xmlPage-4Rectangle.74User ApplicationUser Application +Sheet.82Rectangle.38DPDKDPDK +Rectangle.16bonded ethdevbonded ethdev +Rectangle.11ethdev portethdev port +Rectangle.14ethdev portethdev port +Rectangle.15ethdev portethdev port +Simple Double Arrow.81Simple Arrow.82Simple Arrow.83Simple Arrow.84Simple Arrow.85Sheet.93Square.12511 +Square.12622 +Square.12733 +Sheet.97Square.12511 +Square.12622 +Square.12733 +Sheet.101Square.12511 +Square.12622 +Square.12733 +Sheet.105Square.12511 +Square.12622 +Square.12733 + \ No newline at end of file diff --git a/doc/guides/prog_guide/img/bond-mode-4.svg b/doc/guides/prog_guide/img/bond-mode-4.svg new file mode 100644 index 0000000..45749bd --- /dev/null +++ b/doc/guides/prog_guide/img/bond-mode-4.svg @@ -0,0 +1,784 @@ + + + +image/svg+xmlPage-4Rectangle.177User ApplicationUser Application +Sheet.110Rectangle.38DPDKDPDK +Rectangle.16bonded ethdevbonded ethdev +Rectangle.11ethdev portethdev port +Rectangle.14ethdev portethdev port +Rectangle.15ethdev portethdev port +Simple Double Arrow.184Simple Arrow.185Simple Arrow.186Simple Arrow.187Simple Arrow.188Square.18911 +Square.19022 +Square.19133 +Square.19244 +Square.19355 +Square.19466 +Square.17222 +Square.17344 +Square.19866 +Square.16911 +Square.17033 +Square.17155 +Square.203OO +Square.204OO +Square.205OO + \ No newline at end of file diff --git a/doc/guides/prog_guide/img/bond-mode-5.svg b/doc/guides/prog_guide/img/bond-mode-5.svg new file mode 100644 index 0000000..5ee82fc --- /dev/null +++ b/doc/guides/prog_guide/img/bond-mode-5.svg @@ -0,0 +1,642 @@ + + + +image/svg+xmlPage-4Rectangle.209User ApplicationUser Application +Sheet.137Rectangle.38DPDKDPDK +Rectangle.16bonded ethdevbonded ethdev +Rectangle.11ethdev portethdev port +Rectangle.14ethdev portethdev port +Rectangle.15ethdev portethdev port +Simple Double Arrow.216Simple Arrow.217Simple Arrow.218Simple Arrow.219Simple Arrow.220Sheet.148Rectangle50065006 +Rectangle.24250055005 +Rectangle.24300010001 +Rectangle.24400020002 +Rectangle.2461200312003 +Rectangle.24700010001 +Rectangle.24800020002 +Rectangle.24950065006 +Rectangle.25050055005 +Rectangle.2511200312003 + \ No newline at end of file diff --git a/doc/guides/prog_guide/img/bond-overview.svg b/doc/guides/prog_guide/img/bond-overview.svg new file mode 100644 index 0000000..489282a --- /dev/null +++ b/doc/guides/prog_guide/img/bond-overview.svg @@ -0,0 +1,121 @@ + + + + + + + + + + + + + + + + + + + + + + + + + Page-1 + + Rectangle.38 + DPDK + + DPDK + + Rectangle.8 + bonded ethdev + + + + + bonded ethdev + + Rectangle + User Application + + + + + User Application + + Rectangle.5 + ethdev port + + + + + ethdev port + + Rectangle.6 + ethdev port + + + + + ethdev port + + Rectangle.7 + ethdev port + + + + + ethdev port + + Rectangle.9 + ethdev port + + + + + ethdev port + + Rectangle.10 + ethdev port + + + + + ethdev port + + Simple Double Arrow + + + + Simple Double Arrow.30 + + + + Simple Double Arrow.42 + + + + diff --git a/doc/guides/prog_guide/link_bonding_poll_mode_drv_lib.rst b/doc/guides/prog_guide/link_bonding_poll_mode_drv_lib.rst index badd891..c4e0f2c 100644 --- a/doc/guides/prog_guide/link_bonding_poll_mode_drv_lib.rst +++ b/doc/guides/prog_guide/link_bonding_poll_mode_drv_lib.rst @@ -35,21 +35,25 @@ In addition to Poll Mode Drivers (PMDs) for physical and virtual hardware, Intel® DPDK also includes a pure-software library that allows physical PMD's to be bonded together to create a single logical PMD. -|link_bonding| +|bond-overview| -The Link Bonding PMD library(librte_pmd_bond) supports bonding of groups of physical ports of the same speed (1GbE, 10GbE and 40GbE) and -duplex to provide similar the capabilities to that found in Linux bonding driver to allow the aggregation of multiple (slave) NICs -into a single logical interface between a server and a switch. -The new bonded PMD will then process these interfaces based on the mode of operation specified to provide support for features -such as redundant links, fault tolerance and/or load balancing. +The Link Bonding PMD library(librte_pmd_bond) supports bonding of groups of +``rte_eth_dev`` ports of the same speed and duplex to provide +similar the capabilities to that found in Linux bonding driver to allow the +aggregation of multiple (slave) NICs into a single logical interface between a +server and a switch. The new bonded PMD will then process these interfaces +based on the mode of operation specified to provide support for features such +as redundant links, fault tolerance and/or load balancing. -The librte_pmd_bond library exports a C API which provides an API for the creation of bonded devices -as well as the configuration and management of the bonded device and its slave devices. +The librte_pmd_bond library exports a C API which provides an API for the +creation of bonded devices as well as the configuration and management of the +bonded device and its slave devices. .. note:: - The Link Bonding PMD Library is enabled by default in the build configuration files, - the library can be disabled by setting CONFIG_RTE_LIBRTE_PMD_BOND=n and recompiling the Intel® DPDK. + The Link Bonding PMD Library is enabled by default in the build + configuration files, the library can be disabled by setting + ``CONFIG_RTE_LIBRTE_PMD_BOND=n`` and recompiling the Intel® DPDK. Link Bonding Modes Overview --------------------------- @@ -57,143 +61,255 @@ Link Bonding Modes Overview Currently the Link Bonding PMD library supports 4 modes of operation: * **Round-Robin (Mode 0):** - This mode provides load balancing and fault tolerance by transmission of packets - in sequential order from the first available slave device through the last. - Packets are bulk dequeued from devices then serviced in round-robin manner. + +|bond-mode-0| + + This mode provides load balancing and fault tolerance by transmission of + packets in sequential order from the first available slave device through + the last. Packets are bulk dequeued from devices then serviced in a + round-robin manner. This mode does not guarantee in order reception of + packets and down stream should be able to handle out of order packets. * **Active Backup (Mode 1):** - In this mode only one slave in the bond is active at any time, a different slave becomes active if, - and only if, the primary active slave fails, - thereby providing fault tolerance to slave failure. - The single logical bonded interface's MAC address is externally visible on only one NIC (port) + +|bond-mode-1| + + In this mode only one slave in the bond is active at any time, a different + slave becomes active if, and only if, the primary active slave fails, + thereby providing fault tolerance to slave failure. The single logical + bonded interface's MAC address is externally visible on only one NIC (port) to avoid confusing the network switch. * **Balance XOR (Mode 2):** - This mode provides load balancing based on transmit packets based on the selected XOR transmission policy and fault tolerance. - The default policy (layer2) uses a simple XOR calculation on the packet source / destination MAC address to select the slave to transmit on. - Alternate transmission policies supported are layer 2+3, this uses the IP source and destination addresses in the calculation of the slave port and - the final supported policy is layer 3+4, this uses IP source and destination addresses as well as the UDP source and destination port. + +|bond-mode-2| + + This mode provides transmit load balancing (based on the selected + transmission policy) and fault tolerance. The default policy (layer2) uses + a simple calculation based on the packet flow source and destination MAC + addresses aswell as the number of active slaves available to the bonded + device to classify the packet to a specific slave to transmit on. Alternate + transmission policies supported are layer 2+3, this takes the IP source and + destination addresses into the calculation of the transmit slave port and + the final supported policy is layer 3+4, this uses IP source and + destination addresses as well as the TCP/UDP source and destination port. + +.. note:: + The colouring differences of the packets are used to identify different flow + classification calculated by the selected transmit policy + * **Broadcast (Mode 3):** - This mode provides fault tolerance by transmission of packets on all slave ports. + +|bond-mode-3| + + This mode provides fault tolerance by transmission of packets on all slave + ports. + +* **Link Aggregation 802.3AD (Mode 4):** + +|bond-mode-4| + + This mode provides dynamic link aggregation according to the 802.3ad + specification. It negotiates and monitors aggregation groups that share the + same speed and duplex settings using the selected balance transmit policy + for balancing outgoing traffic. + + DPDK implementation of this mode provide some additional requirements of + the application. + + 1. It needs to call ``rte_eth_tx_burst`` and ``rte_eth_rx_burst`` with + intervals period of less than 100ms. + + 2. Calls to ``rte_eth_tx_burst`` must have a buffer size of at least 2xN, + where N is the number of slaves. This is a space required for LACP + frames. Additionally LACP packets are included in the statistics, but + they are not returned to the application. + +* **Transmit Load Balancing (Mode 5):** + +|bond-mode-5| + + This mode provides an adaptive transmit load balancing. It dynamically + changes the transmitting slave, according to the computed load. Statistics + are collected in 100ms intervals and scheduled every 10ms. + Implementation Details ---------------------- -The librte_pmd_bond onded device are compatible with the Ethernet device API exported by the Ethernet PMDs described in the *Intel® DPDK API Reference*. +The librte_pmd_bond bonded device are compatible with the Ethernet device API +exported by the Ethernet PMDs described in the *Intel® DPDK API Reference*. -The Link Bonding Library supports the creation of bonded devices at application startup time during EAL initialization using the ---vdev option as well as programmatically via the C API rte_eth_bond_create function. +The Link Bonding Library supports the creation of bonded devices at application +startup time during EAL initialization using the ``--vdev`` option as well as +programmatically via the C API ``rte_eth_bond_create`` function. Bonded devices support the dynamical addition and removal of slave devices using -the rte_eth_bond_slave_add / rte_eth_bond_slave_remove APIs. +the ``rte_eth_bond_slave_add`` / ``rte_eth_bond_slave_remove`` APIs. After a slave device is added to a bonded device slave is stopped using -rte_eth_dev_stop and the slave reconfigured using rte_eth_dev_configure the RX and TX queues are also reconfigured -using rte_eth_tx_queue_setup / rte_eth_rx_queue_setup with the parameters use to configure the bonding device. +``rte_eth_dev_stop`` and then reconfigured using ``rte_eth_dev_configure`` +the RX and TX queues are also reconfigured using ``rte_eth_tx_queue_setup`` / +``rte_eth_rx_queue_setup`` with the parameters use to configure the bonding +device. + +Link Status Change Interrupts / Polling +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Link bonding devices support the registration of a link status change callback, +using the ``rte_eth_dev_callback_register`` API, this will be called when the +status of the bonding device changes. For example in the case of a bonding +device which has 3 slaves, the link status will change to up when one slave +becomes active or change to down when all slaves become inactive. There is no +callback notification when a single slave changes state and the previous +conditions are not met. If a user wishes to monitor individual slaves then they +must register callbacks with that slave directly. + +The link bonding library also supports devices which do not implement link +status change interrupts, this is achieve by polling the devices link status at +a defined period which is set using the ``rte_eth_bond_link_monitoring_set`` +API, the default polling interval is 10ms. When a device is added as a slave to +a bonding device it is determined using the ``RTE_PCI_DRV_INTR_LSC`` flag +whether the device supports interrupts or whether the link status should be +monitored by polling it. Requirements / Limitations ~~~~~~~~~~~~~~~~~~~~~~~~~~ -The current implementation only supports physical devices of the same type, speed and duplex to be added as slaves. -The bonded device inherits these values from the first active slave added to the bonded device -and then all further slaves added to the bonded device must match these parameters. +The current implementation only supports devices that support the same speed +and duplex to be added as a slaves to the same bonded device. The bonded device +inherits these attributes from the first active slave added to the bonded +device and then all further slaves added to the bonded device must support +these parameters. -A bonding device must have a minimum of one slave before the bonding device itself can be started. +A bonding device must have a minimum of one slave before the bonding device +itself can be started. -Like all other PMD, all functions exported by a PMD are lock-free functions that are assumed -not to be invoked in parallel on different logical cores to work on the same target object. +Like all other PMD, all functions exported by a PMD are lock-free functions +that are assumed not to be invoked in parallel on different logical cores to +work on the same target object. -It should also be noted that the PMD receive function should not be invoked directly on a slave devices after they have -been to a bonded device since packets read directly from the slave device will no longer be available to the bonded device to read. +It should also be noted that the PMD receive function should not be invoked +directly on a slave devices after they have been to a bonded device since +packets read directly from the slave device will no longer be available to the +bonded device to read. Configuration ~~~~~~~~~~~~~ -Link bonding devices are created using the rte_eth_bond_create API +Link bonding devices are created using the ``rte_eth_bond_create`` API which requires a unique device name, the bonding mode, and the socket Id to allocate the bonding device's resources on. -The other configurable parameters for a bonded device are its slave devices, its primary slave, -a user defined MAC address and transmission policy to use if the device is balance XOR mode. +The other configurable parameters for a bonded device are its slave devices, +its primary slave, a user defined MAC address and transmission policy to use if +the device is in balance XOR mode. Slave Devices ^^^^^^^^^^^^^ -Bonding devices support up to a maximum of RTE_MAX_ETHPORTS slave devices of the same speed and duplex. -Ethernet devices can be added as a slave to a maximum of one bonded device. -Slave devices are reconfigured with the configuration of the bonded device on being added to a bonded device. +Bonding devices support up to a maximum of ``RTE_MAX_ETHPORTS`` slave devices +of the same speed and duplex. Ethernet devices can be added as a slave to a +maximum of one bonded device. Slave devices are reconfigured with the +configuration of the bonded device on being added to a bonded device. -The bonded also guarantees to return the MAC address of the slave device to its original value of removal of a slave from it. +The bonded also guarantees to return the MAC address of the slave device to its +original value of removal of a slave from it. Primary Slave ^^^^^^^^^^^^^ -The primary slave is used to define the default port to use when a bonded device is in active backup mode. -A different port will only be used if, and only if, the current primary port goes down. -If the user does not specify a primary port it will default to being the first port added to the bonded device. +The primary slave is used to define the default port to use when a bonded +device is in active backup mode. A different port will only be used if, and +only if, the current primary port goes down. If the user does not specify a +primary port it will default to being the first port added to the bonded device. MAC Address ^^^^^^^^^^^ -The bonded device can be configured with a user specified MAC address, -this address will be inherited by the some/all slave devices depending on the operating mode. -If the device is in active backup mode then only the primary device will have the user specified MAC, -all other slaves will retain their original MAC address. -In mode 0, 2, and 3 all slaves devices are configure with the bonded devices MAC address. +The bonded device can be configured with a user specified MAC address, this +address will be inherited by the some/all slave devices depending on the +operating mode. If the device is in active backup mode then only the primary +device will have the user specified MAC, all other slaves will retain their +original MAC address. In mode 0, 2, 3, 4 all slaves devices are configure with +the bonded devices MAC address. -If a user defined MAC address is not defined then the bonded device will default to using the primary slaves MAC address. +If a user defined MAC address is not defined then the bonded device will +default to using the primary slaves MAC address. Balance XOR Transmit Policies ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -There are 3 supported transmission policies for bonded device running in Balance XOR mode. Layer 2, Layer 2+3, Layer 3+4. +There are 3 supported transmission policies for bonded device running in +Balance XOR mode. Layer 2, Layer 2+3, Layer 3+4. -* **Layer 2:** Ethernet MAC address based balancing is the default transmission policy for Balance XOR bonding mode. - It uses a simple XOR calculation on the source MAC address and destination MAC address of the packet and - then calculate the modulus of this value to calculate the slave device to transmit the packet on. +* **Layer 2:** Ethernet MAC address based balancing is the default + transmission policy for Balance XOR bonding mode. It uses a simple XOR + calculation on the source MAC address and destination MAC address of the + packet and then calculate the modulus of this value to calculate the slave + device to transmit the packet on. -* **Layer 2 + 3:** Ethernet MAC address & IP Address based balancing uses a combination of source/destination MAC addresses and - the source/destination IP addresses of the data packet to decide which slave port the packet will be transmitted on. +* **Layer 2 + 3:** Ethernet MAC address & IP Address based balancing uses a + combination of source/destination MAC addresses and the source/destination + IP addresses of the data packet to decide which slave port the packet will + be transmitted on. -* **Layer 3 + 4:** IP Address & UDP Port based balancing uses a combination of source/destination IP Address and - the source/destination UDP ports of the packet of the data packet to decide which slave port the packet will be transmitted on. +* **Layer 3 + 4:** IP Address & UDP Port based balancing uses a combination + of source/destination IP Address and the source/destination UDP ports of + the packet of the data packet to decide which slave port the packet will be + transmitted on. -All these policies support 802.1Q VLAN Ethernet packets, as well as IPv4, IPv6 and UDP protocols for load balancing. +All these policies support 802.1Q VLAN Ethernet packets, as well as IPv4, IPv6 +and UDP protocols for load balancing. Using Link Bonding Devices -------------------------- -The librte_pmd_bond library support two modes of device creation, the libraries export full C API or -using the EAL command line to statically configure link bonding devices at application startup. -Using the EAL option it is possible to use link bonding functionality transparently without specific knowledge of the libraries API, -this can be used, for example, to add bonding functionality, such as active backup, -to an existing application which has no knowledge of the link bonding C API. +The librte_pmd_bond library support two modes of device creation, the libraries +export full C API or using the EAL command line to statically configure link +bonding devices at application startup. Using the EAL option it is possible to +use link bonding functionality transparently without specific knowledge of the +libraries API, this can be used, for example, to add bonding functionality, +such as active backup, to an existing application which has no knowledge of +the link bonding C API. Using the Poll Mode Driver from an Application ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Using the librte_pmd_bond libraries API it is possible to dynamicall create and manage link bonding device from within any application. -Link bonding device are created using the rte_eth_bond_create API which requires a unqiue device name, -the link bonding mode to initial the device in and finally the socket Id which to allocate the devices resources onto. -After successful creation of a bonding device it must be configured using the generic Ethernet device configure API rte_eth_dev_configure -and then the RX and TX queues which will be used must be setup using rte_eth_tx_queue_setup / rte_eth_rx_queue_setup. - -Slave devices can be dynamically added and removed from a link bonding device using the rte_eth_bond_slave_add / rte_eth_bond_slave_remove -APIs but at least one slave device must be added to the link bonding device before it can be started using rte_eth_dev_start. - -The link status of a bonded device is dictated by that of its slaves, if all slave device link status are down or -if all slaves are removed from the link bonding device then the link status of the bonding device will go down. - -It is also possible to configure / query the configuration of the control parameters of a bonded device using the provided APIs -rte_eth_bond_mode_set/get, rte_eth_bond_primary_set/get, rte_eth_bond_mac_set/reset and rte_eth_bond_xmit_policy_set/get. +Using the librte_pmd_bond libraries API it is possible to dynamically create +and manage link bonding device from within any application. Link bonding +device are created using the ``rte_eth_bond_create`` API which requires a +unique device name, the link bonding mode to initial the device in and finally +the socket Id which to allocate the devices resources onto. After successful +creation of a bonding device it must be configured using the generic Ethernet +device configure API ``rte_eth_dev_configure`` and then the RX and TX queues +which will be used must be setup using ``rte_eth_tx_queue_setup`` / +``rte_eth_rx_queue_setup``. + +Slave devices can be dynamically added and removed from a link bonding device +using the ``rte_eth_bond_slave_add`` / ``rte_eth_bond_slave_remove`` +APIs but at least one slave device must be added to the link bonding device +before it can be started using ``rte_eth_dev_start``. + +The link status of a bonded device is dictated by that of its slaves, if all +slave device link status are down or if all slaves are removed from the link +bonding device then the link status of the bonding device will go down. + +It is also possible to configure / query the configuration of the control +parameters of a bonded device using the provided APIs +``rte_eth_bond_mode_set/ get``, ``rte_eth_bond_primary_set/get``, +``rte_eth_bond_mac_set/reset`` and ``rte_eth_bond_xmit_policy_set/get``. Using Link Bonding Devices from the EAL Command Line ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Link bonding devices can be created at application startup time using the --vdev EAL command line option. -The device name must start with the eth_bond prefix followed by numbers or letters. The name must be unique for each device. -Each device can have multiple options arranged in a comma separated list. -Multiple devices definitions can be arranged by calling the --vdev option multiple times. +Link bonding devices can be created at application startup time using the +``--vdev`` EAL command line option. The device name must start with the +eth_bond prefix followed by numbers or letters. The name must be unique for +each device. Each device can have multiple options arranged in a comma +separated list. Multiple devices definitions can be arranged by calling the +``--vdev`` option multiple times. + Device names and bonding options must be separated by commas as shown below: .. code-block:: console @@ -203,7 +319,8 @@ Device names and bonding options must be separated by commas as shown below: Link Bonding EAL Options ^^^^^^^^^^^^^^^^^^^^^^^^ -There are multiple ways of definitions that can be assessed and combined as long as the following two rules are respected: +There are multiple ways of definitions that can be assessed and combined as +long as the following two rules are respected: * A unique device name, in the format of eth_bondX is provided, where X can be any combination of numbers and/or letters, @@ -216,37 +333,79 @@ There are multiple ways of definitions that can be assessed and combined as long The different options are: * mode: Integer value defining the bonding mode of the device. - Currently supports modes 0,1,2,3 (round-robin, active backup, balance, and broadcast). + Currently supports modes 0,1,2,3,4,5 (round-robin, active backup, balance, + broadcast, link aggregation, transmit load balancing). + +.. code-block:: console mode=2 -* slave: Defines the PMD device which will be added as slave to the bonded device. - This option can be selected multiple time, for each device to be added as a slave. - Physical devices should be specified using their PCI address, in the format domain:bus:devid.function +* slave: Defines the PMD device which will be added as slave to the bonded + device. This option can be selected multiple time, for each device to be + added as a slave. Physical devices should be specified using their PCI + address, in the format domain:bus:devid.function + +.. code-block:: console slave=0000:0a:00.0,slave=0000:0a:00.1 * primary: Optional parameter which defines the primary slave port, - is used in active backup mode to select the primary slave for data TX/RX if it is available. - The primary port also is used to select the MAC address to use when it is not defined by the user. - This defaults to the first slave added to the device if it is specified. - The primary device must be a slave of the bonded device. + is used in active backup mode to select the primary slave for data TX/RX if + it is available. The primary port also is used to select the MAC address to + use when it is not defined by the user. This defaults to the first slave + added to the device if it is specified. The primary device must be a slave + of the bonded device. + +.. code-block:: console primary=0000:0a:00.0 -* socket_id: Optional parameter used to select which socket on a NUMA device the bonded devices resources will be allocated on. +* socket_id: Optional parameter used to select which socket on a NUMA device + the bonded devices resources will be allocated on. + +.. code-block:: console socket_id=0 -* mac: Optional parameter to select a MAC address for link bonding device, this overrides the value of the primary slave device. +* mac: Optional parameter to select a MAC address for link bonding device, + this overrides the value of the primary slave device. + +.. code-block:: console mac=00:1e:67:1d:fd:1d -* xmit_policy: Optional parameter which defines the transmission policy when the bonded device is in balance mode. - If not user specified this defaults to l2 (layer 2) forwarding, - the other transmission policies available are l23 (layer 2+3) and l34 (layer 3+4) +* xmit_policy: Optional parameter which defines the transmission policy when + the bonded device is in balance mode. If not user specified this defaults + to l2 (layer 2) forwarding, the other transmission policies available are + l23 (layer 2+3) and l34 (layer 3+4) + +.. code-block:: console + + xmit_policy=l23 - xmit_policy=l2 +* lsc_poll_period_ms: Optional parameter which defines the polling interval + in milli-seconds at which devices which don't support lsc interrupts are + checked for a change in the devices link status + +.. code-block:: console + + lsc_poll_period_ms=100 + +* up_delay: Optional parameter which adds a delay in milli-seconds to the + propagation of a devices link status changing to up, by default this + parameter is zero. + +.. code-block:: console + + up_delay=10 + +* down_delay: Optional parameter which adds a delay in milli-seconds to the + propagation of a devices link status changing to down, by default this + parameter is zero. + +.. code-block:: console + + down_delay=50 Examples of Usage ^^^^^^^^^^^^^^^^^ @@ -275,4 +434,11 @@ Create a bonded device in balance mode with two slaves specified by their PCI ad $RTE_TARGET/app/testpmd -c '0xf' -n 4 --vdev 'eth_bond0,mode=2, slave=0000:00a:00.01,slave=0000:004:00.00,xmit_policy=l34' -- --port-topology=chained -.. |link_bonding| image:: img/link_bonding.png +.. |bond-overview| image:: img/bond-overview.svg + +.. |bond-mode-0| image:: img/bond-mode-0.svg +.. |bond-mode-1| image:: img/bond-mode-1.svg +.. |bond-mode-2| image:: img/bond-mode-2.svg +.. |bond-mode-3| image:: img/bond-mode-3.svg +.. |bond-mode-4| image:: img/bond-mode-4.svg +.. |bond-mode-5| image:: img/bond-mode-5.svg -- 1.7.12.2