[dpdk-dev] [PATCH 1/2] test: use env variable to run tests

[dpdk-dev] [PATCH 1/2] test: use env variable to run tests

[Harry van Haaren]

With this patch the test binary checks the DPDK_TEST
environment variable and if set, the contents of the var
are inserted on the test app command line, and run.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Signed-off-by: Harry van Haaren <harry.van.haaren@intel.com>

---

Note that this patch is the combination of multiple patches
that were developed on the dpdk-next-build tree, however the
changes target code for mainline DPDK.

As such, the relevant parts of the following commits were
merged into this single patch that targets mainline:
- 32529 test: use env variable to run test if set
- 32555 test/test: fix missed failures when running meson test
- 32556 test/test: add return value to mark unit tests as skipped
---
 test/test/commands.c |  1 +
 test/test/test.c     | 18 ++++++++++++++++++
 test/test/test.h     |  1 +
 3 files changed, 20 insertions(+)

diff --git a/test/test/commands.c b/test/test/commands.c
index 4097a33..6649cfc 100644
--- a/test/test/commands.c
+++ b/test/test/commands.c
@@ -103,6 +103,7 @@ static void cmd_autotest_parsed(void *parsed_result,
 			ret = t->callback();
 	}
 
+	last_test_result = ret;
 	if (ret == 0)
 		printf("Test OK\n");
 	else
diff --git a/test/test/test.c b/test/test/test.c
index 0e6ff7c..49a61ce 100644
--- a/test/test/test.c
+++ b/test/test/test.c
@@ -102,6 +102,8 @@ do_recursive_call(void)
 	return -1;
 }
 
+int last_test_result;
+
 int
 main(int argc, char **argv)
 {
@@ -140,6 +142,20 @@ main(int argc, char **argv)
 	if (cl == NULL) {
 		return -1;
 	}
+
+	char *dpdk_test = getenv("DPDK_TEST");
+	if (dpdk_test && strlen(dpdk_test)) {
+		char buf[1024];
+		snprintf(buf, sizeof(buf), "%s\n", dpdk_test);
+		if (cmdline_in(cl, buf, strlen(buf)) < 0) {
+			printf("error on cmdline input\n");
+			return -1;
+		}
+
+		cmdline_stdin_exit(cl);
+		return last_test_result;
+	}
+	/* if no DPDK_TEST env variable, go interactive */
 	cmdline_interact(cl);
 	cmdline_stdin_exit(cl);
 #endif
@@ -231,6 +247,8 @@ unit_test_suite_runner(struct unit_test_suite *suite)
 	printf(" + Tests Failed :      %2d\n", failed);
 	printf(" + ------------------------------------------------------- +\n");
 
+	last_test_result = failed;
+
 	if (failed)
 		return -1;
 
diff --git a/test/test/test.h b/test/test/test.h
index 08ffe94..2e90184 100644
--- a/test/test/test.h
+++ b/test/test/test.h
@@ -218,6 +218,7 @@ struct unit_test_suite {
 };
 
 int unit_test_suite_runner(struct unit_test_suite *suite);
+extern int last_test_result;
 
 #define RECURSIVE_ENV_VAR "RTE_TEST_RECURSIVE"
 
-- 
2.7.4

[dpdk-dev] [PATCH 2/2] test: add skip instead of fail, update crypto test

[Harry van Haaren]

This commit adds a SKIPPED return value from the unit tests,
indicating that the test was not able to run (eg: PMD was
not enabled when DPDK was compiled).

The cryptodev tests are updated to return SKIPPED instead
of failing if the PMD is not enabled, allowing any test
infrastructure to identify that the test was not able to run.

Signed-off-by: Harry van Haaren <harry.van.haaren@intel.com>
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>

---

Note that this patch is the combination of multiple patches
that were developed on the dpdk-next-build tree, however the
changes target code for mainline DPDK.

As such, the relevant parts of the following commits were
merged into this single patch that targets mainline:
- 32529 test: use env variable to run test if set
- 32555 test/test: fix missed failures when running meson test
- 32556 test/test: add return value to mark unit tests as skipped
---
 test/test/test.h           |  5 +++--
 test/test/test_cryptodev.c | 28 ++++++++++++++--------------
 2 files changed, 17 insertions(+), 16 deletions(-)

diff --git a/test/test/test.h b/test/test/test.h
index 2e90184..df5eea0 100644
--- a/test/test/test.h
+++ b/test/test/test.h
@@ -40,8 +40,9 @@
 #include <rte_common.h>
 #include <rte_log.h>
 
-#define TEST_SUCCESS  (0)
-#define TEST_FAILED  (-1)
+#define TEST_SUCCESS EXIT_SUCCESS
+#define TEST_FAILED  -1
+#define TEST_SKIPPED  77
 
 /* Before including test.h file you can define
  * TEST_TRACE_FAILURE(_file, _line, _func) macro to better trace/debug test
diff --git a/test/test/test_cryptodev.c b/test/test/test_cryptodev.c
index 1bed65d..54e6a9e 100644
--- a/test/test/test_cryptodev.c
+++ b/test/test/test_cryptodev.c
@@ -9591,7 +9591,7 @@ test_cryptodev_qat(void /*argv __rte_unused, int argc __rte_unused*/)
 		RTE_LOG(ERR, USER1, "QAT PMD must be loaded. Check if "
 				"CONFIG_RTE_LIBRTE_PMD_QAT is enabled "
 				"in config file to run this testsuite.\n");
-		return TEST_FAILED;
+		return TEST_SKIPPED;
 	}
 
 	return unit_test_suite_runner(&cryptodev_qat_testsuite);
@@ -9607,7 +9607,7 @@ test_cryptodev_aesni_mb(void /*argv __rte_unused, int argc __rte_unused*/)
 		RTE_LOG(ERR, USER1, "AESNI MB PMD must be loaded. Check if "
 				"CONFIG_RTE_LIBRTE_PMD_AESNI_MB is enabled "
 				"in config file to run this testsuite.\n");
-		return TEST_FAILED;
+		return TEST_SKIPPED;
 	}
 
 	return unit_test_suite_runner(&cryptodev_aesni_mb_testsuite);
@@ -9623,7 +9623,7 @@ test_cryptodev_openssl(void)
 		RTE_LOG(ERR, USER1, "OPENSSL PMD must be loaded. Check if "
 				"CONFIG_RTE_LIBRTE_PMD_OPENSSL is enabled "
 				"in config file to run this testsuite.\n");
-		return TEST_FAILED;
+		return TEST_SKIPPED;
 	}
 
 	return unit_test_suite_runner(&cryptodev_openssl_testsuite);
@@ -9639,7 +9639,7 @@ test_cryptodev_aesni_gcm(void)
 		RTE_LOG(ERR, USER1, "AESNI GCM PMD must be loaded. Check if "
 				"CONFIG_RTE_LIBRTE_PMD_AESNI_GCM is enabled "
 				"in config file to run this testsuite.\n");
-		return TEST_FAILED;
+		return TEST_SKIPPED;
 	}
 
 	return unit_test_suite_runner(&cryptodev_aesni_gcm_testsuite);
@@ -9655,7 +9655,7 @@ test_cryptodev_null(void)
 		RTE_LOG(ERR, USER1, "NULL PMD must be loaded. Check if "
 				"CONFIG_RTE_LIBRTE_PMD_NULL is enabled "
 				"in config file to run this testsuite.\n");
-		return TEST_FAILED;
+		return TEST_SKIPPED;
 	}
 
 	return unit_test_suite_runner(&cryptodev_null_testsuite);
@@ -9671,7 +9671,7 @@ test_cryptodev_sw_snow3g(void /*argv __rte_unused, int argc __rte_unused*/)
 		RTE_LOG(ERR, USER1, "SNOW3G PMD must be loaded. Check if "
 				"CONFIG_RTE_LIBRTE_PMD_SNOW3G is enabled "
 				"in config file to run this testsuite.\n");
-		return TEST_FAILED;
+		return TEST_SKIPPED;
 	}
 
 	return unit_test_suite_runner(&cryptodev_sw_snow3g_testsuite);
@@ -9687,7 +9687,7 @@ test_cryptodev_sw_kasumi(void /*argv __rte_unused, int argc __rte_unused*/)
 		RTE_LOG(ERR, USER1, "ZUC PMD must be loaded. Check if "
 				"CONFIG_RTE_LIBRTE_PMD_KASUMI is enabled "
 				"in config file to run this testsuite.\n");
-		return TEST_FAILED;
+		return TEST_SKIPPED;
 	}
 
 	return unit_test_suite_runner(&cryptodev_sw_kasumi_testsuite);
@@ -9703,7 +9703,7 @@ test_cryptodev_sw_zuc(void /*argv __rte_unused, int argc __rte_unused*/)
 		RTE_LOG(ERR, USER1, "ZUC PMD must be loaded. Check if "
 				"CONFIG_RTE_LIBRTE_PMD_ZUC is enabled "
 				"in config file to run this testsuite.\n");
-		return TEST_FAILED;
+		return TEST_SKIPPED;
 	}
 
 	return unit_test_suite_runner(&cryptodev_sw_zuc_testsuite);
@@ -9719,7 +9719,7 @@ test_cryptodev_armv8(void)
 		RTE_LOG(ERR, USER1, "ARMV8 PMD must be loaded. Check if "
 				"CONFIG_RTE_LIBRTE_PMD_ARMV8 is enabled "
 				"in config file to run this testsuite.\n");
-		return TEST_FAILED;
+		return TEST_SKIPPED;
 	}
 
 	return unit_test_suite_runner(&cryptodev_armv8_testsuite);
@@ -9735,7 +9735,7 @@ test_cryptodev_mrvl(void)
 		RTE_LOG(ERR, USER1, "MRVL PMD must be loaded. Check if "
 				"CONFIG_RTE_LIBRTE_PMD_MRVL_CRYPTO is enabled "
 				"in config file to run this testsuite.\n");
-		return TEST_FAILED;
+		return TEST_SKIPPED;
 	}
 
 	return unit_test_suite_runner(&cryptodev_mrvl_testsuite);
@@ -9753,14 +9753,14 @@ test_cryptodev_scheduler(void /*argv __rte_unused, int argc __rte_unused*/)
 		RTE_LOG(ERR, USER1, "SCHEDULER PMD must be loaded. Check if "
 				"CONFIG_RTE_LIBRTE_PMD_SCHEDULER is enabled "
 				"in config file to run this testsuite.\n");
-		return TEST_FAILED;
+		return TEST_SKIPPED;
 	}
 
 	if (rte_cryptodev_driver_id_get(
 				RTE_STR(CRYPTODEV_NAME_AESNI_MB_PMD)) == -1) {
 		RTE_LOG(ERR, USER1, "CONFIG_RTE_LIBRTE_PMD_AESNI_MB must be"
 			" enabled in config file to run this testsuite.\n");
-		return TEST_FAILED;
+		return TEST_SKIPPED;
 }
 	return unit_test_suite_runner(&cryptodev_scheduler_testsuite);
 }
@@ -9779,7 +9779,7 @@ test_cryptodev_dpaa2_sec(void /*argv __rte_unused, int argc __rte_unused*/)
 		RTE_LOG(ERR, USER1, "DPAA2 SEC PMD must be loaded. Check if "
 				"CONFIG_RTE_LIBRTE_PMD_DPAA2_SEC is enabled "
 				"in config file to run this testsuite.\n");
-		return TEST_FAILED;
+		return TEST_SKIPPED;
 	}
 
 	return unit_test_suite_runner(&cryptodev_dpaa2_sec_testsuite);
@@ -9795,7 +9795,7 @@ test_cryptodev_dpaa_sec(void /*argv __rte_unused, int argc __rte_unused*/)
 		RTE_LOG(ERR, USER1, "DPAA SEC PMD must be loaded. Check if "
 				"CONFIG_RTE_LIBRTE_PMD_DPAA_SEC is enabled "
 				"in config file to run this testsuite.\n");
-		return TEST_FAILED;
+		return TEST_SKIPPED;
 	}
 
 	return unit_test_suite_runner(&cryptodev_dpaa_sec_testsuite);
-- 
2.7.4

Re: [dpdk-dev] [PATCH 1/2] test: use env variable to run tests

[De Lara Guarch, Pablo]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Harry van Haaren
> Sent: Thursday, December 21, 2017 12:07 PM
> To: dev@dpdk.org
> Cc: thomas@monjalon.net; Richardson, Bruce
> <bruce.richardson@intel.com>; Van Haaren, Harry
> <harry.van.haaren@intel.com>
> Subject: [dpdk-dev] [PATCH 1/2] test: use env variable to run tests
> 
> With this patch the test binary checks the DPDK_TEST environment variable
> and if set, the contents of the var are inserted on the test app command
> line, and run.
> 
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> Signed-off-by: Harry van Haaren <harry.van.haaren@intel.com>

Reviewed-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>

Re: [dpdk-dev] [PATCH 2/2] test: add skip instead of fail, update crypto test

[De Lara Guarch, Pablo]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Harry van Haaren
> Sent: Thursday, December 21, 2017 12:08 PM
> To: dev@dpdk.org
> Cc: thomas@monjalon.net; Richardson, Bruce
> <bruce.richardson@intel.com>; Van Haaren, Harry
> <harry.van.haaren@intel.com>
> Subject: [dpdk-dev] [PATCH 2/2] test: add skip instead of fail, update crypto
> test
> 
> This commit adds a SKIPPED return value from the unit tests, indicating that
> the test was not able to run (eg: PMD was not enabled when DPDK was
> compiled).
> 
> The cryptodev tests are updated to return SKIPPED instead of failing if the
> PMD is not enabled, allowing any test infrastructure to identify that the test
> was not able to run.

Check-git-log.sh complains about the commit title (because of the comma).
I think this patch should be split into two patches: one that adds TEST_SKIPPED
and another one that updates the crypto tests (the title suggests these two patches).

Also, when running a test that returns TEST_SKIPPED, it still prints out "Test Failed".
It looks like this is caused because cmd_autotest_parsed() hasn't been updated:

static void cmd_autotest_parsed(void *parsed_result,
                                __attribute__((unused)) struct cmdline *cl,
                                __attribute__((unused)) void *data)
{
...
        if (ret == 0)
                printf("Test OK\n");
        else
                printf("Test Failed\n");

Probably, another check should be added here.

Pablo

[dpdk-dev] [PATCH v2 1/3] test: use env variable to run tests

[Harry van Haaren]

With this patch the test binary checks the DPDK_TEST
environment variable and if set, the contents of the var
are inserted on the test app command line, and run.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Signed-off-by: Harry van Haaren <harry.van.haaren@intel.com>
Reviewed-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>

---

v2:
- Added Pablo's reviewed-by

Note that this patch is the combination of multiple patches
that were developed on the dpdk-next-build tree, however the
changes target code for mainline DPDK.

As such, the relevant parts of the following commits were
merged into this single patch that targets mainline:
- 32529 test: use env variable to run test if set
- 32555 test/test: fix missed failures when running meson test
- 32556 test/test: add return value to mark unit tests as skipped
---
 test/test/commands.c |  1 +
 test/test/test.c     | 18 ++++++++++++++++++
 test/test/test.h     |  1 +
 3 files changed, 20 insertions(+)

diff --git a/test/test/commands.c b/test/test/commands.c
index 4097a33..6649cfc 100644
--- a/test/test/commands.c
+++ b/test/test/commands.c
@@ -103,6 +103,7 @@ static void cmd_autotest_parsed(void *parsed_result,
 			ret = t->callback();
 	}
 
+	last_test_result = ret;
 	if (ret == 0)
 		printf("Test OK\n");
 	else
diff --git a/test/test/test.c b/test/test/test.c
index ffe3b92..af1b1bf 100644
--- a/test/test/test.c
+++ b/test/test/test.c
@@ -73,6 +73,8 @@ do_recursive_call(void)
 	return -1;
 }
 
+int last_test_result;
+
 int
 main(int argc, char **argv)
 {
@@ -111,6 +113,20 @@ main(int argc, char **argv)
 	if (cl == NULL) {
 		return -1;
 	}
+
+	char *dpdk_test = getenv("DPDK_TEST");
+	if (dpdk_test && strlen(dpdk_test)) {
+		char buf[1024];
+		snprintf(buf, sizeof(buf), "%s\n", dpdk_test);
+		if (cmdline_in(cl, buf, strlen(buf)) < 0) {
+			printf("error on cmdline input\n");
+			return -1;
+		}
+
+		cmdline_stdin_exit(cl);
+		return last_test_result;
+	}
+	/* if no DPDK_TEST env variable, go interactive */
 	cmdline_interact(cl);
 	cmdline_stdin_exit(cl);
 #endif
@@ -202,6 +218,8 @@ unit_test_suite_runner(struct unit_test_suite *suite)
 	printf(" + Tests Failed :      %2d\n", failed);
 	printf(" + ------------------------------------------------------- +\n");
 
+	last_test_result = failed;
+
 	if (failed)
 		return -1;
 
diff --git a/test/test/test.h b/test/test/test.h
index 21ee251..ae88a70 100644
--- a/test/test/test.h
+++ b/test/test/test.h
@@ -189,6 +189,7 @@ struct unit_test_suite {
 };
 
 int unit_test_suite_runner(struct unit_test_suite *suite);
+extern int last_test_result;
 
 #define RECURSIVE_ENV_VAR "RTE_TEST_RECURSIVE"
 
-- 
2.7.4

[dpdk-dev] [PATCH v2 2/3] test: add skipped return result

[Harry van Haaren]

This commit allows a test to return "skipped", indicating
that it cannot be run. This is useful for PMDs which have
not been compiled due to the unavailability of dependencies,
or their explicit disabling in the build configuration.

The result printing is updated to correctly indicate if a
test has been skipped.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Signed-off-by: Harry van Haaren <harry.van.haaren@intel.com>

---

v2:
- Re-split patch into two, rewording titles check-git-log (Pablo)
- Add check for "Skipped" tests to not print "Failed" (Pablo)
---
 test/test/commands.c | 2 ++
 test/test/test.h     | 5 +++--
 2 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/test/test/commands.c b/test/test/commands.c
index 6649cfc..0473dac 100644
--- a/test/test/commands.c
+++ b/test/test/commands.c
@@ -106,6 +106,8 @@ static void cmd_autotest_parsed(void *parsed_result,
 	last_test_result = ret;
 	if (ret == 0)
 		printf("Test OK\n");
+	else if (ret == TEST_SKIPPED)
+		printf("Test Skipped\n");
 	else
 		printf("Test Failed\n");
 	fflush(stdout);
diff --git a/test/test/test.h b/test/test/test.h
index ae88a70..a8f765a 100644
--- a/test/test/test.h
+++ b/test/test/test.h
@@ -11,8 +11,9 @@
 #include <rte_common.h>
 #include <rte_log.h>
 
-#define TEST_SUCCESS  (0)
-#define TEST_FAILED  (-1)
+#define TEST_SUCCESS EXIT_SUCCESS
+#define TEST_FAILED  -1
+#define TEST_SKIPPED  77
 
 /* Before including test.h file you can define
  * TEST_TRACE_FAILURE(_file, _line, _func) macro to better trace/debug test
-- 
2.7.4

[dpdk-dev] [PATCH v2 3/3] test: update cryptodev tests with skipped value

[Harry van Haaren]

The cryptodev tests are updated to return SKIPPED instead
of failing if the PMD is not enabled, allowing test
infrastructure to identify that the test was not able to run.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Signed-off-by: Harry van Haaren <harry.van.haaren@intel.com>

---

v2:
- Split cryptodev test changes into own patch (Pablo)

Note that this patch is the combination of multiple patches
that were developed on the dpdk-next-build tree, however the
changes target code for mainline DPDK.

As such, the relevant parts of the following commits were
merged into this single patch that targets mainline:
- 32529 test: use env variable to run test if set
- 32555 test/test: fix missed failures when running meson test
- 32556 test/test: add return value to mark unit tests as skipped
---
 test/test/test_cryptodev.c | 28 ++++++++++++++--------------
 1 file changed, 14 insertions(+), 14 deletions(-)

diff --git a/test/test/test_cryptodev.c b/test/test/test_cryptodev.c
index 4d7e5b1..f49f5cd 100644
--- a/test/test/test_cryptodev.c
+++ b/test/test/test_cryptodev.c
@@ -9563,7 +9563,7 @@ test_cryptodev_qat(void /*argv __rte_unused, int argc __rte_unused*/)
 		RTE_LOG(ERR, USER1, "QAT PMD must be loaded. Check if "
 				"CONFIG_RTE_LIBRTE_PMD_QAT is enabled "
 				"in config file to run this testsuite.\n");
-		return TEST_FAILED;
+		return TEST_SKIPPED;
 	}
 
 	return unit_test_suite_runner(&cryptodev_qat_testsuite);
@@ -9579,7 +9579,7 @@ test_cryptodev_aesni_mb(void /*argv __rte_unused, int argc __rte_unused*/)
 		RTE_LOG(ERR, USER1, "AESNI MB PMD must be loaded. Check if "
 				"CONFIG_RTE_LIBRTE_PMD_AESNI_MB is enabled "
 				"in config file to run this testsuite.\n");
-		return TEST_FAILED;
+		return TEST_SKIPPED;
 	}
 
 	return unit_test_suite_runner(&cryptodev_aesni_mb_testsuite);
@@ -9595,7 +9595,7 @@ test_cryptodev_openssl(void)
 		RTE_LOG(ERR, USER1, "OPENSSL PMD must be loaded. Check if "
 				"CONFIG_RTE_LIBRTE_PMD_OPENSSL is enabled "
 				"in config file to run this testsuite.\n");
-		return TEST_FAILED;
+		return TEST_SKIPPED;
 	}
 
 	return unit_test_suite_runner(&cryptodev_openssl_testsuite);
@@ -9611,7 +9611,7 @@ test_cryptodev_aesni_gcm(void)
 		RTE_LOG(ERR, USER1, "AESNI GCM PMD must be loaded. Check if "
 				"CONFIG_RTE_LIBRTE_PMD_AESNI_GCM is enabled "
 				"in config file to run this testsuite.\n");
-		return TEST_FAILED;
+		return TEST_SKIPPED;
 	}
 
 	return unit_test_suite_runner(&cryptodev_aesni_gcm_testsuite);
@@ -9627,7 +9627,7 @@ test_cryptodev_null(void)
 		RTE_LOG(ERR, USER1, "NULL PMD must be loaded. Check if "
 				"CONFIG_RTE_LIBRTE_PMD_NULL is enabled "
 				"in config file to run this testsuite.\n");
-		return TEST_FAILED;
+		return TEST_SKIPPED;
 	}
 
 	return unit_test_suite_runner(&cryptodev_null_testsuite);
@@ -9643,7 +9643,7 @@ test_cryptodev_sw_snow3g(void /*argv __rte_unused, int argc __rte_unused*/)
 		RTE_LOG(ERR, USER1, "SNOW3G PMD must be loaded. Check if "
 				"CONFIG_RTE_LIBRTE_PMD_SNOW3G is enabled "
 				"in config file to run this testsuite.\n");
-		return TEST_FAILED;
+		return TEST_SKIPPED;
 	}
 
 	return unit_test_suite_runner(&cryptodev_sw_snow3g_testsuite);
@@ -9659,7 +9659,7 @@ test_cryptodev_sw_kasumi(void /*argv __rte_unused, int argc __rte_unused*/)
 		RTE_LOG(ERR, USER1, "ZUC PMD must be loaded. Check if "
 				"CONFIG_RTE_LIBRTE_PMD_KASUMI is enabled "
 				"in config file to run this testsuite.\n");
-		return TEST_FAILED;
+		return TEST_SKIPPED;
 	}
 
 	return unit_test_suite_runner(&cryptodev_sw_kasumi_testsuite);
@@ -9675,7 +9675,7 @@ test_cryptodev_sw_zuc(void /*argv __rte_unused, int argc __rte_unused*/)
 		RTE_LOG(ERR, USER1, "ZUC PMD must be loaded. Check if "
 				"CONFIG_RTE_LIBRTE_PMD_ZUC is enabled "
 				"in config file to run this testsuite.\n");
-		return TEST_FAILED;
+		return TEST_SKIPPED;
 	}
 
 	return unit_test_suite_runner(&cryptodev_sw_zuc_testsuite);
@@ -9691,7 +9691,7 @@ test_cryptodev_armv8(void)
 		RTE_LOG(ERR, USER1, "ARMV8 PMD must be loaded. Check if "
 				"CONFIG_RTE_LIBRTE_PMD_ARMV8 is enabled "
 				"in config file to run this testsuite.\n");
-		return TEST_FAILED;
+		return TEST_SKIPPED;
 	}
 
 	return unit_test_suite_runner(&cryptodev_armv8_testsuite);
@@ -9707,7 +9707,7 @@ test_cryptodev_mrvl(void)
 		RTE_LOG(ERR, USER1, "MRVL PMD must be loaded. Check if "
 				"CONFIG_RTE_LIBRTE_PMD_MRVL_CRYPTO is enabled "
 				"in config file to run this testsuite.\n");
-		return TEST_FAILED;
+		return TEST_SKIPPED;
 	}
 
 	return unit_test_suite_runner(&cryptodev_mrvl_testsuite);
@@ -9725,14 +9725,14 @@ test_cryptodev_scheduler(void /*argv __rte_unused, int argc __rte_unused*/)
 		RTE_LOG(ERR, USER1, "SCHEDULER PMD must be loaded. Check if "
 				"CONFIG_RTE_LIBRTE_PMD_SCHEDULER is enabled "
 				"in config file to run this testsuite.\n");
-		return TEST_FAILED;
+		return TEST_SKIPPED;
 	}
 
 	if (rte_cryptodev_driver_id_get(
 				RTE_STR(CRYPTODEV_NAME_AESNI_MB_PMD)) == -1) {
 		RTE_LOG(ERR, USER1, "CONFIG_RTE_LIBRTE_PMD_AESNI_MB must be"
 			" enabled in config file to run this testsuite.\n");
-		return TEST_FAILED;
+		return TEST_SKIPPED;
 }
 	return unit_test_suite_runner(&cryptodev_scheduler_testsuite);
 }
@@ -9751,7 +9751,7 @@ test_cryptodev_dpaa2_sec(void /*argv __rte_unused, int argc __rte_unused*/)
 		RTE_LOG(ERR, USER1, "DPAA2 SEC PMD must be loaded. Check if "
 				"CONFIG_RTE_LIBRTE_PMD_DPAA2_SEC is enabled "
 				"in config file to run this testsuite.\n");
-		return TEST_FAILED;
+		return TEST_SKIPPED;
 	}
 
 	return unit_test_suite_runner(&cryptodev_dpaa2_sec_testsuite);
@@ -9767,7 +9767,7 @@ test_cryptodev_dpaa_sec(void /*argv __rte_unused, int argc __rte_unused*/)
 		RTE_LOG(ERR, USER1, "DPAA SEC PMD must be loaded. Check if "
 				"CONFIG_RTE_LIBRTE_PMD_DPAA_SEC is enabled "
 				"in config file to run this testsuite.\n");
-		return TEST_FAILED;
+		return TEST_SKIPPED;
 	}
 
 	return unit_test_suite_runner(&cryptodev_dpaa_sec_testsuite);
-- 
2.7.4

Re: [dpdk-dev] [PATCH v2 3/3] test: update cryptodev tests with skipped value

[De Lara Guarch, Pablo]

> -----Original Message-----
> From: Van Haaren, Harry
> Sent: Thursday, January 11, 2018 5:51 PM
> To: dev@dpdk.org
> Cc: De Lara Guarch, Pablo <pablo.de.lara.guarch@intel.com>; Richardson,
> Bruce <bruce.richardson@intel.com>; Van Haaren, Harry
> <harry.van.haaren@intel.com>
> Subject: [PATCH v2 3/3] test: update cryptodev tests with skipped value
> 
> The cryptodev tests are updated to return SKIPPED instead of failing if the
> PMD is not enabled, allowing test infrastructure to identify that the test
> was not able to run.
> 
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> Signed-off-by: Harry van Haaren <harry.van.haaren@intel.com>

Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>

Re: [dpdk-dev] [PATCH v2 2/3] test: add skipped return result

[De Lara Guarch, Pablo]

> -----Original Message-----
> From: Van Haaren, Harry
> Sent: Thursday, January 11, 2018 5:51 PM
> To: dev@dpdk.org
> Cc: De Lara Guarch, Pablo <pablo.de.lara.guarch@intel.com>; Richardson,
> Bruce <bruce.richardson@intel.com>; Van Haaren, Harry
> <harry.van.haaren@intel.com>
> Subject: [PATCH v2 2/3] test: add skipped return result
> 
> This commit allows a test to return "skipped", indicating that it cannot be
> run. This is useful for PMDs which have not been compiled due to the
> unavailability of dependencies, or their explicit disabling in the build
> configuration.
> 
> The result printing is updated to correctly indicate if a test has been
> skipped.
> 
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> Signed-off-by: Harry van Haaren <harry.van.haaren@intel.com>

Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>

Re: [dpdk-dev] [PATCH v2 1/3] test: use env variable to run tests

[Thomas Monjalon]

11/01/2018 18:50, Harry van Haaren:
> With this patch the test binary checks the DPDK_TEST
> environment variable and if set, the contents of the var
> are inserted on the test app command line, and run.
> 
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> Signed-off-by: Harry van Haaren <harry.van.haaren@intel.com>
> Reviewed-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>

Series applied, thanks