DPDK patches and discussions
 help / color / mirror / Atom feed
From: Aaron Conole <aconole@redhat.com>
To: Bruce Richardson <bruce.richardson@intel.com>
Cc: dev@dpdk.org,  david.marchand@redhat.com,  probb@iol.unh.edu
Subject: Re: [RFC PATCH] app/test: add support for skipping tests
Date: Tue, 29 Aug 2023 09:27:23 -0400	[thread overview]
Message-ID: <f7th6oirnb8.fsf@redhat.com> (raw)
In-Reply-To: <20230817105851.501947-1-bruce.richardson@intel.com> (Bruce Richardson's message of "Thu, 17 Aug 2023 11:58:51 +0100")

Bruce Richardson <bruce.richardson@intel.com> writes:

> When called from automated tools, like meson test, it is often useful to
> skip tests in a test suite, without having to alter the test build. To
> do so, we add support for DPDK_TEST_SKIP environment variable, where one
> can provide a comma-separated list of tests. When the test binary is
> called to run one of the tests on the list via either cmdline parameter
> or environment variable (as done with meson test), the test will not
> actually be run, but will be reported skipped.
>
> Example run:
>   $ DPDK_TEST_SKIP=dump_devargs,dump_ring meson test --suite=debug-tests
>   ...
>   1/9 DPDK:debug-tests / dump_devargs             SKIP            1.11s
>   2/9 DPDK:debug-tests / dump_log_types           OK              1.06s
>   3/9 DPDK:debug-tests / dump_malloc_heaps        OK              1.11s
>   4/9 DPDK:debug-tests / dump_malloc_stats        OK              1.07s
>   5/9 DPDK:debug-tests / dump_mempool             OK              1.11s
>   6/9 DPDK:debug-tests / dump_memzone             OK              1.06s
>   7/9 DPDK:debug-tests / dump_physmem             OK              1.13s
>   8/9 DPDK:debug-tests / dump_ring                SKIP            1.04s
>   9/9 DPDK:debug-tests / dump_struct_sizes        OK              1.10s
>
>   Ok:                 7
>   Expected Fail:      0
>   Fail:               0
>   Unexpected Pass:    0
>   Skipped:            2
>   Timeout:            0
>
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> ---

The idea looks fine to me, although I would really like it if we could
do a command like argument instead of env variable (but that's just a
suggestion to color the shed).

Just minor nit stuff below.

>  app/test/test.c | 27 +++++++++++++++++++++++++++
>  1 file changed, 27 insertions(+)
>
> diff --git a/app/test/test.c b/app/test/test.c
> index fb073ff795..3569c36049 100644
> --- a/app/test/test.c
> +++ b/app/test/test.c
> @@ -193,6 +193,20 @@ main(int argc, char **argv)
>  
>  	if (test_count > 0) {
>  		char buf[1024];
> +		char *dpdk_test_skip = getenv("DPDK_TEST_SKIP");
> +		char *skip_tests[128] = {0};
> +		size_t n_skip_tests = 0;
> +
> +		if (dpdk_test_skip != NULL && strlen(dpdk_test_skip) > 0){
> +			char *dpdk_test_skip_cp = strdup(dpdk_test_skip);
> +			if (dpdk_test_skip_cp == NULL) {
> +				ret = -1;
> +				goto out;
> +			}
> +			dpdk_test_skip = dpdk_test_skip_cp;
> +			n_skip_tests = rte_strsplit(dpdk_test_skip, strlen(dpdk_test_skip),
> +					skip_tests, RTE_DIM(skip_tests), ',');

We probably should check for n_skip_tests == -1 - although it shouldn't
ever happen for this code.  If it ever did happen, we'd walk right off
the for() loop below.  Actually, it looks like that error condition for
rte_strsplit isn't documented.

> +		}
>  
>  		cl = cmdline_new(main_ctx, "RTE>>", 0, 1);
>  		if (cl == NULL) {
> @@ -201,6 +215,15 @@ main(int argc, char **argv)
>  		}
>  
>  		for (i = 0; i < test_count; i++) {
> +			/* check if test is to be skipped */
> +			for (size_t j = 0; j < n_skip_tests; j++) {

We might want to check j < MIN(n_skip_tests, RTE_DIM(skip_tests)) to
prevent a rogue env variable that makes rte_strsplit fail in the future.

> +				if (strcmp(tests[i], skip_tests[j]) == 0) {
> +					fprintf(stderr, "Skipping %s [DPDK_TEST_SKIP]\n", tests[i]);
> +					ret = TEST_SKIPPED;
> +					goto end_of_cmd;
> +				}
> +			}
> +
>  			snprintf(buf, sizeof(buf), "%s\n", tests[i]);
>  			if (cmdline_parse_check(cl, buf) < 0) {
>  				printf("Error: invalid test command: '%s'\n", tests[i]);
> @@ -211,9 +234,13 @@ main(int argc, char **argv)
>  			} else
>  				ret = last_test_result;
>  
> +end_of_cmd:
>  			if (ret != 0)
>  				break;
>  		}
> +		if (n_skip_tests > 0)
> +			free(dpdk_test_skip);
> +
>  		cmdline_free(cl);
>  		goto out;
>  	} else {


  parent reply	other threads:[~2023-08-29 13:27 UTC|newest]

Thread overview: 15+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2023-08-17 10:58 Bruce Richardson
2023-08-29  9:34 ` Thomas Monjalon
2023-08-29 13:27 ` Aaron Conole [this message]
2023-08-29 13:39   ` Bruce Richardson
2023-08-31 10:06 ` [PATCH v2] " Bruce Richardson
2023-08-31 10:18 ` [PATCH v3] " Bruce Richardson
2023-08-31 13:58   ` Bruce Richardson
2023-09-14 15:16 ` [PATCH v4] " Bruce Richardson
2023-09-25 16:26   ` Aaron Conole
2023-10-02  9:20     ` David Marchand
2023-10-03 20:22       ` Patrick Robb
2023-10-04 15:13         ` Aaron Conole
2023-10-09 20:03           ` Patrick Robb
2023-10-20 15:02             ` Patrick Robb
2023-10-20 15:08               ` Bruce Richardson

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=f7th6oirnb8.fsf@redhat.com \
    --to=aconole@redhat.com \
    --cc=bruce.richardson@intel.com \
    --cc=david.marchand@redhat.com \
    --cc=dev@dpdk.org \
    --cc=probb@iol.unh.edu \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).