[dpdk-dev] [PATCH 0/4] eth_ring: perf test and usability improvements

[dpdk-dev] [PATCH 0/4] eth_ring: perf test and usability improvements

[Bruce Richardson]

This patchset makes it easier to create ring pmd instances from code, by
providing a simple ring->ethdev wrapper function and also ensuring that
any created rings are ready for use immediately, without having to call
configure and rx/tx queue setup.

This set also contains a set of unit tests to compare the performance of
basic ring operations against the same operations via the ring ethdev.
This shows how the perf penalty can be significant for small bursts, but
is much less so for larger bursts of 32 packets.

Bruce Richardson (4):
  ring: enhance rte_eth_from_rings
  rte_ring: store memzone pointer inside ring
  ring: add rte_eth_from_ring function
  test: Add perf test for ring pmd

 app/test/Makefile                         |   1 +
 app/test/test_pmd_ring_perf.c             | 188 ++++++++++++++++++++++++++++++
 drivers/net/ring/rte_eth_ring.c           |  55 ++++++++-
 drivers/net/ring/rte_eth_ring.h           |  14 +++
 drivers/net/ring/rte_eth_ring_version.map |   5 +
 lib/librte_ring/rte_ring.c                |   1 +
 lib/librte_ring/rte_ring.h                |   4 +
 7 files changed, 262 insertions(+), 6 deletions(-)
 create mode 100644 app/test/test_pmd_ring_perf.c

-- 
2.4.3

[dpdk-dev] [PATCH 1/4] ring: enhance rte_eth_from_rings

[Bruce Richardson]

The ring ethdev creation function creates an ethdev, but does not
actually set it up for use. Even if it's just a single ring, the user
still needs to create a mempool, call rte_eth_dev_configure, then call
rx and tx setup functions before the ethdev can be used.

This patch changes things so that the ethdev is fully set up after the
call to create the ethdev. The above-mentionned functions can still be
called - as will be the case, for instance, if the NIC is created via
commandline parameters - but they no longer are essential.

The function now also sets rte_errno appropriately on error, so the
caller can get a better indication of why a call may have failed.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 drivers/net/ring/rte_eth_ring.c | 47 +++++++++++++++++++++++++++++++++++------
 1 file changed, 41 insertions(+), 6 deletions(-)

diff --git a/drivers/net/ring/rte_eth_ring.c b/drivers/net/ring/rte_eth_ring.c
index 0ba36d5..bfd6f4e 100644
--- a/drivers/net/ring/rte_eth_ring.c
+++ b/drivers/net/ring/rte_eth_ring.c
@@ -39,6 +39,7 @@
 #include <rte_string_fns.h>
 #include <rte_dev.h>
 #include <rte_kvargs.h>
+#include <rte_errno.h>
 
 #define ETH_RING_NUMA_NODE_ACTION_ARG	"nodeaction"
 #define ETH_RING_ACTION_CREATE		"CREATE"
@@ -276,10 +277,18 @@ rte_eth_from_rings(const char *name, struct rte_ring *const rx_queues[],
 	unsigned i;
 
 	/* do some parameter checking */
-	if (rx_queues == NULL && nb_rx_queues > 0)
+	if (rx_queues == NULL && nb_rx_queues > 0) {
+		rte_errno = EINVAL;
 		goto error;
-	if (tx_queues == NULL && nb_tx_queues > 0)
+	}
+	if (tx_queues == NULL && nb_tx_queues > 0) {
+		rte_errno = EINVAL;
+		goto error;
+	}
+	if (nb_rx_queues > RTE_PMD_RING_MAX_RX_RINGS) {
+		rte_errno = EINVAL;
 		goto error;
+	}
 
 	RTE_LOG(INFO, PMD, "Creating rings-backed ethdev on numa socket %u\n",
 			numa_node);
@@ -288,21 +297,43 @@ rte_eth_from_rings(const char *name, struct rte_ring *const rx_queues[],
 	 * and internal (private) data
 	 */
 	data = rte_zmalloc_socket(name, sizeof(*data), 0, numa_node);
-	if (data == NULL)
+	if (data == NULL) {
+		rte_errno = ENOMEM;
 		goto error;
+	}
+
+	data->rx_queues = rte_zmalloc_socket(name, sizeof(void *) * nb_rx_queues,
+			0, numa_node);
+	if (data->rx_queues == NULL) {
+		rte_errno = ENOMEM;
+		goto error;
+	}
+
+	data->tx_queues = rte_zmalloc_socket(name, sizeof(void *) * nb_tx_queues,
+			0, numa_node);
+	if (data->tx_queues == NULL) {
+		rte_errno = ENOMEM;
+		goto error;
+	}
 
 	pci_dev = rte_zmalloc_socket(name, sizeof(*pci_dev), 0, numa_node);
-	if (pci_dev == NULL)
+	if (pci_dev == NULL) {
+		rte_errno = ENOMEM;
 		goto error;
+	}
 
 	internals = rte_zmalloc_socket(name, sizeof(*internals), 0, numa_node);
-	if (internals == NULL)
+	if (internals == NULL) {
+		rte_errno = ENOMEM;
 		goto error;
+	}
 
 	/* reserve an ethdev entry */
 	eth_dev = rte_eth_dev_allocate(name, RTE_ETH_DEV_VIRTUAL);
-	if (eth_dev == NULL)
+	if (eth_dev == NULL) {
+		rte_errno = ENOSPC;
 		goto error;
+	}
 
 
 	/* now put it all together
@@ -318,9 +349,11 @@ rte_eth_from_rings(const char *name, struct rte_ring *const rx_queues[],
 	internals->nb_tx_queues = nb_tx_queues;
 	for (i = 0; i < nb_rx_queues; i++) {
 		internals->rx_ring_queues[i].rng = rx_queues[i];
+		data->rx_queues[i] = &internals->rx_ring_queues[i];
 	}
 	for (i = 0; i < nb_tx_queues; i++) {
 		internals->tx_ring_queues[i].rng = tx_queues[i];
+		data->tx_queues[i] = &internals->tx_ring_queues[i];
 	}
 
 	rte_ring_pmd.pci_drv.name = ring_ethdev_driver_name;
@@ -350,6 +383,8 @@ rte_eth_from_rings(const char *name, struct rte_ring *const rx_queues[],
 	return data->port_id;
 
 error:
+	rte_free(data->rx_queues);
+	rte_free(data->tx_queues);
 	rte_free(data);
 	rte_free(pci_dev);
 	rte_free(internals);
-- 
2.4.3

[dpdk-dev] [PATCH 2/4] rte_ring: store memzone pointer inside ring

[Bruce Richardson]

Add a new field to the rte_ring structure to store the memzone pointer which
contains the ring. For rings created using rte_ring_create(), the field will
be set automatically.

This new field will allow users of the ring to query the numa node a ring is
allocated on, or to get the physical address of the ring, if so needed.

The rte_ring structure will also maintain ABI compatibility, as the
structure members, after the new one, are set to be cache line aligned,
so leaving a space.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 lib/librte_ring/rte_ring.c | 1 +
 lib/librte_ring/rte_ring.h | 4 ++++
 2 files changed, 5 insertions(+)

diff --git a/lib/librte_ring/rte_ring.c b/lib/librte_ring/rte_ring.c
index c9e59d4..4e78e14 100644
--- a/lib/librte_ring/rte_ring.c
+++ b/lib/librte_ring/rte_ring.c
@@ -196,6 +196,7 @@ rte_ring_create(const char *name, unsigned count, int socket_id,
 		rte_ring_init(r, name, count, flags);
 
 		te->data = (void *) r;
+		r->memzone = mz;
 
 		TAILQ_INSERT_TAIL(ring_list, te, next);
 	} else {
diff --git a/lib/librte_ring/rte_ring.h b/lib/librte_ring/rte_ring.h
index af68888..df45f3f 100644
--- a/lib/librte_ring/rte_ring.h
+++ b/lib/librte_ring/rte_ring.h
@@ -134,6 +134,8 @@ struct rte_ring_debug_stats {
                                     *   if RTE_RING_PAUSE_REP not defined. */
 #endif
 
+struct rte_memzone; /* forward declaration, so as not to require memzone.h */
+
 /**
  * An RTE ring structure.
  *
@@ -147,6 +149,8 @@ struct rte_ring_debug_stats {
 struct rte_ring {
 	char name[RTE_RING_NAMESIZE];    /**< Name of the ring. */
 	int flags;                       /**< Flags supplied at creation. */
+	const struct rte_memzone *memzone;
+			/**< Memzone, if any, containing the rte_ring */
 
 	/** Ring producer status. */
 	struct prod {
-- 
2.4.3

[dpdk-dev] [PATCH 3/4] ring: add rte_eth_from_ring function

[Bruce Richardson]

Add a one-parameter function to take an existing rte_ring and wrap it as
an ethdev, returning the port id of the new ethdev instance.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 drivers/net/ring/rte_eth_ring.c           |  8 ++++++++
 drivers/net/ring/rte_eth_ring.h           | 14 ++++++++++++++
 drivers/net/ring/rte_eth_ring_version.map |  5 +++++
 3 files changed, 27 insertions(+)

diff --git a/drivers/net/ring/rte_eth_ring.c b/drivers/net/ring/rte_eth_ring.c
index bfd6f4e..d851f9e 100644
--- a/drivers/net/ring/rte_eth_ring.c
+++ b/drivers/net/ring/rte_eth_ring.c
@@ -36,6 +36,7 @@
 #include <rte_ethdev.h>
 #include <rte_malloc.h>
 #include <rte_memcpy.h>
+#include <rte_memzone.h>
 #include <rte_string_fns.h>
 #include <rte_dev.h>
 #include <rte_kvargs.h>
@@ -392,6 +393,13 @@ error:
 	return -1;
 }
 
+int
+rte_eth_from_ring(struct rte_ring *r)
+{
+	return rte_eth_from_rings(r->name, &r, 1, &r, 1,
+			r->memzone ? r->memzone->socket_id : SOCKET_ID_ANY);
+}
+
 enum dev_action{
 	DEV_CREATE,
 	DEV_ATTACH
diff --git a/drivers/net/ring/rte_eth_ring.h b/drivers/net/ring/rte_eth_ring.h
index 5a69bff..4ff83ec 100644
--- a/drivers/net/ring/rte_eth_ring.h
+++ b/drivers/net/ring/rte_eth_ring.h
@@ -65,6 +65,20 @@ int rte_eth_from_rings(const char *name,
 		const unsigned nb_tx_queues,
 		const unsigned numa_node);
 
+/**
+ * Create a new ethdev port from a ring
+ *
+ * This function is a shortcut call for rte_eth_from_rings for the
+ * case where one wants to take a single rte_ring and use it as though
+ * it were an ethdev
+ *
+ * @param ring
+ *    the ring to be used as an ethdev
+ * @return
+ *    the port number of the newly created ethdev, or -1 on error
+ */
+int rte_eth_from_ring(struct rte_ring *r);
+
 #ifdef __cplusplus
 }
 #endif
diff --git a/drivers/net/ring/rte_eth_ring_version.map b/drivers/net/ring/rte_eth_ring_version.map
index 0875e25..8e113a4 100644
--- a/drivers/net/ring/rte_eth_ring_version.map
+++ b/drivers/net/ring/rte_eth_ring_version.map
@@ -5,3 +5,8 @@ DPDK_2.0 {
 
 	local: *;
 };
+
+DPDK_2.2 {
+	global:
+	rte_eth_from_ring;
+} DPDK_2.0;
-- 
2.4.3

[dpdk-dev] [PATCH 4/4] test: Add perf test for ring pmd

[Bruce Richardson]

Add a performance test for ring pmd, comparing performance of the pmd
compared to the basic rte_ring APIs.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 app/test/Makefile             |   1 +
 app/test/test_pmd_ring_perf.c | 188 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 189 insertions(+)
 create mode 100644 app/test/test_pmd_ring_perf.c

diff --git a/app/test/Makefile b/app/test/Makefile
index 294618f..14fa502 100644
--- a/app/test/Makefile
+++ b/app/test/Makefile
@@ -141,6 +141,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_PMD_BOND) += test_link_bonding_mode4.c
 endif
 
 SRCS-$(CONFIG_RTE_LIBRTE_PMD_RING) += test_pmd_ring.c
+SRCS-$(CONFIG_RTE_LIBRTE_PMD_RING) += test_pmd_ring_perf.c
 SRCS-$(CONFIG_RTE_LIBRTE_KVARGS) += test_kvargs.c
 
 CFLAGS += -O3
diff --git a/app/test/test_pmd_ring_perf.c b/app/test/test_pmd_ring_perf.c
new file mode 100644
index 0000000..3077dba
--- /dev/null
+++ b/app/test/test_pmd_ring_perf.c
@@ -0,0 +1,188 @@
+/*-
+ *   BSD LICENSE
+ *
+ *   Copyright(c) 2015 Intel Corporation. All rights reserved.
+ *   All rights reserved.
+ *
+ *   Redistribution and use in source and binary forms, with or without
+ *   modification, are permitted provided that the following conditions
+ *   are met:
+ *
+ *     * Redistributions of source code must retain the above copyright
+ *       notice, this list of conditions and the following disclaimer.
+ *     * Redistributions in binary form must reproduce the above copyright
+ *       notice, this list of conditions and the following disclaimer in
+ *       the documentation and/or other materials provided with the
+ *       distribution.
+ *     * Neither the name of Intel Corporation nor the names of its
+ *       contributors may be used to endorse or promote products derived
+ *       from this software without specific prior written permission.
+ *
+ *   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ *   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ *   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ *   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ *   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ *   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ *   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ *   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ *   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ *   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ *   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+
+#include <stdio.h>
+#include <inttypes.h>
+#include <rte_ring.h>
+#include <rte_cycles.h>
+#include <rte_launch.h>
+#include <rte_ethdev.h>
+#include <rte_eth_ring.h>
+
+#include "test.h"
+
+#define RING_NAME "RING_PERF"
+#define RING_SIZE 4096
+#define MAX_BURST 32
+
+/*
+ * the sizes to enqueue and dequeue in testing
+ * (marked volatile so they won't be seen as compile-time constants)
+ */
+static const volatile unsigned bulk_sizes[] = { 1, 8, 32 };
+
+/* The ring structure used for tests */
+static struct rte_ring *r;
+static uint8_t ring_ethdev_port;
+
+/* Get cycle counts for dequeuing from an empty ring. Should be 2 or 3 cycles */
+static void
+test_empty_dequeue(void)
+{
+	const unsigned iter_shift = 26;
+	const unsigned iterations = 1 << iter_shift;
+	unsigned i = 0;
+	void *burst[MAX_BURST];
+
+	const uint64_t sc_start = rte_rdtsc();
+	for (i = 0; i < iterations; i++)
+		rte_ring_sc_dequeue_bulk(r, burst, bulk_sizes[0]);
+	const uint64_t sc_end = rte_rdtsc();
+
+	const uint64_t eth_start = rte_rdtsc();
+	for (i = 0; i < iterations; i++)
+		rte_eth_rx_burst(ring_ethdev_port, 0, (void *)burst,
+				bulk_sizes[0]);
+	const uint64_t eth_end = rte_rdtsc();
+
+	printf("Ring empty dequeue  : %.1F\n",
+			(double)(sc_end - sc_start) / iterations);
+	printf("Ethdev empty dequeue: %.1F\n",
+			(double)(eth_end - eth_start) / iterations);
+}
+
+/*
+ * Test function that determines how long an enqueue + dequeue of a single item
+ * takes on a single lcore. Result is for comparison with the bulk enq+deq.
+ */
+static void
+test_single_enqueue_dequeue(void)
+{
+	const unsigned iter_shift = 24;
+	const unsigned iterations = 1 << iter_shift;
+	unsigned i = 0;
+	void *burst = NULL;
+	struct rte_mbuf *mburst[1] = { NULL };
+
+	const uint64_t sc_start = rte_rdtsc_precise();
+	rte_compiler_barrier();
+	for (i = 0; i < iterations; i++) {
+		rte_ring_enqueue_bulk(r, &burst, 1);
+		rte_ring_dequeue_bulk(r, &burst, 1);
+	}
+	const uint64_t sc_end = rte_rdtsc_precise();
+	rte_compiler_barrier();
+
+	const uint64_t eth_start = rte_rdtsc_precise();
+	rte_compiler_barrier();
+	for (i = 0; i < iterations; i++) {
+		rte_eth_tx_burst(ring_ethdev_port, 0, mburst, 1);
+		rte_eth_rx_burst(ring_ethdev_port, 0, mburst, 1);
+	}
+	const uint64_t eth_end = rte_rdtsc_precise();
+	rte_compiler_barrier();
+
+	printf("Ring single enq/dequeue  : %"PRIu64"\n",
+			(sc_end-sc_start) >> iter_shift);
+	printf("Ethdev single enq/dequeue: %"PRIu64"\n",
+			(eth_end-eth_start) >> iter_shift);
+}
+
+/* Times enqueue and dequeue on a single lcore */
+static void
+test_bulk_enqueue_dequeue(void)
+{
+	const unsigned iter_shift = 23;
+	const unsigned iterations = 1 << iter_shift;
+	unsigned sz, i = 0;
+	struct rte_mbuf *burst[MAX_BURST] = {0};
+
+	for (sz = 0; sz < sizeof(bulk_sizes)/sizeof(bulk_sizes[0]); sz++) {
+		const uint64_t sc_start = rte_rdtsc();
+		for (i = 0; i < iterations; i++) {
+			rte_ring_sp_enqueue_bulk(r, (void *)burst, bulk_sizes[sz]);
+			rte_ring_sc_dequeue_bulk(r, (void *)burst, bulk_sizes[sz]);
+		}
+		const uint64_t sc_end = rte_rdtsc();
+
+		const uint64_t eth_start = rte_rdtsc_precise();
+		rte_compiler_barrier();
+		for (i = 0; i < iterations; i++) {
+			rte_eth_tx_burst(ring_ethdev_port, 0, burst, bulk_sizes[sz]);
+			rte_eth_rx_burst(ring_ethdev_port, 0, burst, bulk_sizes[sz]);
+		}
+		const uint64_t eth_end = rte_rdtsc_precise();
+		rte_compiler_barrier();
+
+		double sc_avg = ((double)(sc_end-sc_start) /
+				(iterations * bulk_sizes[sz]));
+		double eth_avg = ((double)(eth_end-eth_start) /
+				(iterations * bulk_sizes[sz]));
+
+		printf("ring bulk enq/deq (size: %u) : %.1F\n", bulk_sizes[sz],
+				sc_avg);
+		printf("ethdev bulk enq/deq (size:%u): %.1F\n", bulk_sizes[sz],
+				eth_avg);
+
+		printf("\n");
+	}
+}
+
+static int
+test_ring_pmd_perf(void)
+{
+	r = rte_ring_create(RING_NAME, RING_SIZE, rte_socket_id(),
+			RING_F_SP_ENQ|RING_F_SC_DEQ);
+	if (r == NULL && (r = rte_ring_lookup(RING_NAME)) == NULL)
+		return -1;
+
+	ring_ethdev_port = rte_eth_from_ring(r);
+
+	printf("\n### Testing const single element enq/deq ###\n");
+	test_single_enqueue_dequeue();
+
+	printf("\n### Testing empty dequeue ###\n");
+	test_empty_dequeue();
+
+	printf("\n### Testing using a single lcore ###\n");
+	test_bulk_enqueue_dequeue();
+
+	return 0;
+}
+
+static struct test_command ring_pmd_perf_cmd = {
+	.command = "ring_pmd_perf_autotest",
+	.callback = test_ring_pmd_perf,
+};
+REGISTER_TEST_COMMAND(ring_pmd_perf_cmd);
-- 
2.4.3

Re: [dpdk-dev] [PATCH 2/4] rte_ring: store memzone pointer inside ring

[Olivier MATZ]

Hi Bruce,

On 09/30/2015 02:12 PM, Bruce Richardson wrote:
> Add a new field to the rte_ring structure to store the memzone pointer which
> contains the ring. For rings created using rte_ring_create(), the field will
> be set automatically.
> 
> This new field will allow users of the ring to query the numa node a ring is
> allocated on, or to get the physical address of the ring, if so needed.
> 
> The rte_ring structure will also maintain ABI compatibility, as the
> structure members, after the new one, are set to be cache line aligned,
> so leaving a space.
> 
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>

Acked-by: Olivier Matz <olivier.matz@6wind.com>

Re: [dpdk-dev] [PATCH 0/4] eth_ring: perf test and usability improvements

[De Lara Guarch, Pablo]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Bruce Richardson
> Sent: Wednesday, September 30, 2015 1:12 PM
> To: dev@dpdk.org
> Subject: [dpdk-dev] [PATCH 0/4] eth_ring: perf test and usability
> improvements
> 
> This patchset makes it easier to create ring pmd instances from code, by
> providing a simple ring->ethdev wrapper function and also ensuring that
> any created rings are ready for use immediately, without having to call
> configure and rx/tx queue setup.
> 
> This set also contains a set of unit tests to compare the performance of
> basic ring operations against the same operations via the ring ethdev.
> This shows how the perf penalty can be significant for small bursts, but
> is much less so for larger bursts of 32 packets.
> 
> Bruce Richardson (4):
>   ring: enhance rte_eth_from_rings
>   rte_ring: store memzone pointer inside ring
>   ring: add rte_eth_from_ring function
>   test: Add perf test for ring pmd
> 
>  app/test/Makefile                         |   1 +
>  app/test/test_pmd_ring_perf.c             | 188
> ++++++++++++++++++++++++++++++
>  drivers/net/ring/rte_eth_ring.c           |  55 ++++++++-
>  drivers/net/ring/rte_eth_ring.h           |  14 +++
>  drivers/net/ring/rte_eth_ring_version.map |   5 +
>  lib/librte_ring/rte_ring.c                |   1 +
>  lib/librte_ring/rte_ring.h                |   4 +
>  7 files changed, 262 insertions(+), 6 deletions(-)
>  create mode 100644 app/test/test_pmd_ring_perf.c
> 
> --
> 2.4.3

Series-acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>

Re: [dpdk-dev] [PATCH 0/4] eth_ring: perf test and usability improvements

[Thomas Monjalon]

> > This patchset makes it easier to create ring pmd instances from code, by
> > providing a simple ring->ethdev wrapper function and also ensuring that
> > any created rings are ready for use immediately, without having to call
> > configure and rx/tx queue setup.
> > 
> > This set also contains a set of unit tests to compare the performance of
> > basic ring operations against the same operations via the ring ethdev.
> > This shows how the perf penalty can be significant for small bursts, but
> > is much less so for larger bursts of 32 packets.
> > 
> > Bruce Richardson (4):
> >   ring: enhance rte_eth_from_rings
> >   rte_ring: store memzone pointer inside ring
> >   ring: add rte_eth_from_ring function
> >   test: Add perf test for ring pmd
> 
> Series-acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>

Applied, thanks