From: Jan Viktorin <viktorin@rehivetech.com>
To: dev@dpdk.org
Cc: Jan Viktorin <viktorin@rehivetech.com>,
Thomas Monjalon <thomas.monjalon@6wind.com>,
David Marchand <david.marchand@6wind.com>,
Bruce Richardson <bruce.richardson@intel.com>
Subject: [dpdk-dev] [PATCH v3 05/11] app/test: support resources archived by tar
Date: Tue, 17 May 2016 20:34:55 +0200 [thread overview]
Message-ID: <1463510101-5633-6-git-send-email-viktorin@rehivetech.com> (raw)
In-Reply-To: <1463510101-5633-1-git-send-email-viktorin@rehivetech.com>
In-Reply-To: <1461935496-20367-1-git-send-email-viktorin@rehivetech.com>
When a more complex resource (a file hierarchy) is needed, packing every single
file as a single resource would be very ineffective. For that purpose, it is
possible to pack the files into a tar archive, extract it before test from the
resource and finally clean up all the created files.
This patch introduces functions resource_untar and resource_rm_by_tar to
perform those tasks. An example of using those functions is included as a test.
A new dependency is required to build the app/test: libarchive.
Signed-off-by: Jan Viktorin <viktorin@rehivetech.com>
---
app/test/Makefile | 9 +++
app/test/resource.c | 195 +++++++++++++++++++++++++++++++++++++++++++++++
app/test/resource.h | 13 ++++
app/test/test_resource.c | 29 +++++++
4 files changed, 246 insertions(+)
diff --git a/app/test/Makefile b/app/test/Makefile
index cfe68a6..de5fa50 100644
--- a/app/test/Makefile
+++ b/app/test/Makefile
@@ -51,6 +51,13 @@ $(1).res.o: $(2)
/dev/stdin $$@ < $$<
endef
+define linked_tar_resource
+$(1).tar: $(2)
+ tar -C $$(dir $$<) -cf $$@ $$(notdir $$<)
+
+$(call linked_resource,$(1),$(1).tar)
+endef
+
#
# library name
#
@@ -64,6 +71,7 @@ SRCS-y += test.c
SRCS-y += resource.c
SRCS-y += test_resource.c
$(eval $(call linked_resource,test_resource_c,resource.c))
+$(eval $(call linked_tar_resource,test_resource_tar,test_resource.c))
SRCS-y += test_pci.c
SRCS-y += test_prefetch.c
SRCS-y += test_byteorder.c
@@ -183,6 +191,7 @@ CFLAGS += $(WERROR_FLAGS)
CFLAGS += -D_GNU_SOURCE
LDLIBS += -lm
+LDLIBS += -larchive
# Disable VTA for memcpy test
ifeq ($(CC), gcc)
diff --git a/app/test/resource.c b/app/test/resource.c
index acb63c1..668fc52 100644
--- a/app/test/resource.c
+++ b/app/test/resource.c
@@ -33,6 +33,8 @@
#include <stdio.h>
#include <string.h>
+#include <archive.h>
+#include <archive_entry.h>
#include <errno.h>
#include <sys/queue.h>
@@ -95,6 +97,199 @@ int resource_fwrite_file(const struct resource *r, const char *fname)
return ret;
}
+static int do_copy(struct archive *r, struct archive *w)
+{
+ const void *buf;
+ size_t len;
+ off_t off;
+ int ret;
+
+ while (1) {
+ ret = archive_read_data_block(r, &buf, &len, &off);
+ if (ret == ARCHIVE_RETRY)
+ continue;
+
+ if (ret == ARCHIVE_EOF)
+ return 0;
+
+ if (ret != ARCHIVE_OK)
+ return ret;
+
+ do {
+ ret = archive_write_data_block(w, buf, len, off);
+ if (ret != ARCHIVE_OK && ret != ARCHIVE_RETRY)
+ return ret;
+ } while (ret != ARCHIVE_OK);
+ }
+}
+
+int resource_untar(const struct resource *res)
+{
+ struct archive *r;
+ struct archive *w;
+ struct archive_entry *e;
+ void *p;
+ int flags = 0;
+ int ret;
+
+ p = malloc(resource_size(res));
+ if (p == NULL)
+ rte_panic("Failed to malloc %zu B\n", resource_size(res));
+
+ memcpy(p, res->begin, resource_size(res));
+
+ r = archive_read_new();
+ if (r == NULL) {
+ free(p);
+ return -1;
+ }
+
+ archive_read_support_format_all(r);
+ archive_read_support_filter_all(r);
+
+ w = archive_write_disk_new();
+ if (w == NULL) {
+ archive_read_free(r);
+ free(p);
+ return -1;
+ }
+
+ flags |= ARCHIVE_EXTRACT_PERM;
+ flags |= ARCHIVE_EXTRACT_FFLAGS;
+ archive_write_disk_set_options(w, flags);
+ archive_write_disk_set_standard_lookup(w);
+
+ ret = archive_read_open_memory(r, p, resource_size(res));
+ if (ret != ARCHIVE_OK)
+ goto fail;
+
+ while (1) {
+ ret = archive_read_next_header(r, &e);
+ if (ret == ARCHIVE_EOF)
+ break;
+ if (ret != ARCHIVE_OK)
+ goto fail;
+
+ ret = archive_write_header(w, e);
+ if (ret == ARCHIVE_EOF)
+ break;
+ if (ret != ARCHIVE_OK)
+ goto fail;
+
+ if (archive_entry_size(e) == 0)
+ continue;
+
+ ret = do_copy(r, w);
+ if (ret != ARCHIVE_OK)
+ goto fail;
+
+ ret = archive_write_finish_entry(w);
+ if (ret != ARCHIVE_OK)
+ goto fail;
+ }
+
+ archive_write_free(w);
+ archive_read_free(r);
+ free(p);
+ return 0;
+
+fail:
+ archive_write_free(w);
+ archive_read_free(r);
+ free(p);
+ rte_panic("Failed: %s\n", archive_error_string(r));
+ return -1;
+}
+
+int resource_rm_by_tar(const struct resource *res)
+{
+ struct archive *r;
+ struct archive_entry *e;
+ void *p;
+ int try_again = 1;
+ int attempts = 0;
+ int ret;
+
+ p = malloc(resource_size(res));
+ if (p == NULL)
+ rte_panic("Failed to malloc %zu B\n", resource_size(res));
+
+ memcpy(p, res->begin, resource_size(res));
+
+ /* If somebody creates a file somewhere inside the extracted TAR
+ hierarchy during a test the resource_rm_by_tar might loop
+ infinitely. We prevent this by adding the attempts counter there.
+ In normal case, max N iteration is done where N is the depth of
+ the file-hierarchy.
+ */
+ while (try_again && attempts < 10000) {
+ r = archive_read_new();
+ if (r == NULL) {
+ free(p);
+ return -1;
+ }
+
+ archive_read_support_format_all(r);
+ archive_read_support_filter_all(r);
+
+ ret = archive_read_open_memory(r, p, resource_size(res));
+ if (ret != ARCHIVE_OK) {
+ fprintf(stderr, "Failed: %s\n",
+ archive_error_string(r));
+ goto fail;
+ }
+
+ try_again = 0;
+
+ while (1) {
+ ret = archive_read_next_header(r, &e);
+ if (ret == ARCHIVE_EOF)
+ break;
+ if (ret != ARCHIVE_OK)
+ goto fail;
+
+ ret = remove(archive_entry_pathname(e));
+ if (ret < 0) {
+ switch (errno) {
+ case ENOTEMPTY:
+ case EEXIST:
+ try_again = 1;
+ break;
+
+ /* should not usually happen: */
+ case ENOENT:
+ case ENOTDIR:
+ case EROFS:
+ attempts += 1;
+ continue;
+ default:
+ perror("Failed to remove file");
+ goto fail;
+ }
+ }
+ }
+
+ archive_read_free(r);
+ attempts += 1;
+ }
+
+ if (attempts >= 10000) {
+ fprintf(stderr, "Failed to remove archive\n");
+ free(p);
+ return -1;
+ }
+
+ free(p);
+ return 0;
+
+fail:
+ archive_read_free(r);
+ free(p);
+
+ rte_panic("Failed: %s\n", archive_error_string(r));
+ return -1;
+}
+
void resource_register(struct resource *r)
{
TAILQ_INSERT_TAIL(&resource_list, r, next);
diff --git a/app/test/resource.h b/app/test/resource.h
index 849df05..5a9031a 100644
--- a/app/test/resource.h
+++ b/app/test/resource.h
@@ -89,6 +89,19 @@ int resource_fwrite(const struct resource *r, FILE *f);
int resource_fwrite_file(const struct resource *r, const char *fname);
/**
+ * Treat the given resource as a tar archive. Extract
+ * the archive to the current directory.
+ */
+int resource_untar(const struct resource *res);
+
+/**
+ * Treat the given resource as a tar archive. Remove
+ * all files (related to the current directory) listed
+ * in the tar archive.
+ */
+int resource_rm_by_tar(const struct resource *res);
+
+/**
* Register a resource in the global list of resources.
* Not intended for direct use, please check the REGISTER_RESOURCE
* macro.
diff --git a/app/test/test_resource.c b/app/test/test_resource.c
index 3d1bf00..1e85040 100644
--- a/app/test/test_resource.c
+++ b/app/test/test_resource.c
@@ -85,6 +85,32 @@ static int test_resource_c(void)
return 0;
}
+REGISTER_LINKED_RESOURCE(test_resource_tar);
+
+static int test_resource_tar(void)
+{
+ const struct resource *r;
+ FILE *f;
+
+ r = resource_find("test_resource_tar");
+ TEST_ASSERT_NOT_NULL(r, "No test_resource_tar found");
+ TEST_ASSERT(!strcmp(r->name, "test_resource_tar"),
+ "Found resource %s, expected test_resource_tar",
+ r->name);
+
+ TEST_ASSERT_SUCCESS(resource_untar(r),
+ "Failed to to untar %s", r->name);
+
+ f = fopen("test_resource.c", "r");
+ TEST_ASSERT_NOT_NULL(f,
+ "Missing extracted file test_resource.c");
+ fclose(f);
+
+ TEST_ASSERT_SUCCESS(resource_rm_by_tar(r),
+ "Failed to remove extracted contents of %s", r->name);
+ return 0;
+}
+
static int test_resource(void)
{
if (test_resource_dpdk())
@@ -93,6 +119,9 @@ static int test_resource(void)
if (test_resource_c())
return -1;
+ if (test_resource_tar())
+ return -1;
+
return 0;
}
--
2.8.0
next prev parent reply other threads:[~2016-05-17 18:36 UTC|newest]
Thread overview: 111+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-04-29 13:11 [dpdk-dev] [RFC 0/4] Include resources in tests Jan Viktorin
2016-04-29 13:11 ` [dpdk-dev] [RFC 1/4] app/test: introduce resources for tests Jan Viktorin
2016-04-29 13:11 ` [dpdk-dev] [RFC 2/4] app/test: support resources externally linked Jan Viktorin
2016-04-29 13:11 ` [dpdk-dev] [RFC 3/4] app/test: add functions to create files from resources Jan Viktorin
2016-04-29 13:11 ` [dpdk-dev] [RFC 4/4] app/test: support resources archived by tar Jan Viktorin
2016-05-05 13:33 ` Bruce Richardson
2016-05-05 17:05 ` Jan Viktorin
2016-05-06 9:02 ` Bruce Richardson
2016-04-29 14:42 ` [dpdk-dev] [RFC 0/4] Include resources in tests Bruce Richardson
2016-04-29 20:52 ` Jan Viktorin
2016-05-05 13:29 ` Bruce Richardson
2016-05-05 17:03 ` Jan Viktorin
2016-05-06 9:03 ` Bruce Richardson
2016-05-06 10:48 ` [dpdk-dev] [PATCH v1 00/10] " Jan Viktorin
2016-05-06 10:48 ` [dpdk-dev] [PATCH v1 01/10] app/test: introduce resources for tests Jan Viktorin
2016-05-06 14:01 ` Thomas Monjalon
2016-05-06 16:20 ` Jan Viktorin
2016-05-12 14:58 ` Thomas Monjalon
2016-05-12 15:01 ` Jan Viktorin
2016-05-09 15:36 ` Jan Viktorin
2016-05-06 10:48 ` [dpdk-dev] [PATCH v1 02/10] app/test: support resources externally linked Jan Viktorin
2016-05-06 14:32 ` Thomas Monjalon
2016-05-06 16:31 ` Jan Viktorin
2016-05-09 15:19 ` Jan Viktorin
2016-05-12 15:05 ` Thomas Monjalon
2016-05-06 10:48 ` [dpdk-dev] [PATCH v1 03/10] app/test: add functions to create files from resources Jan Viktorin
2016-05-06 10:48 ` [dpdk-dev] [PATCH v1 04/10] app/test: support resources archived by tar Jan Viktorin
2016-05-12 15:26 ` Thomas Monjalon
2016-05-12 15:28 ` Thomas Monjalon
2016-05-06 10:48 ` [dpdk-dev] [PATCH v1 05/10] app/test: use linked list to store PCI drivers Jan Viktorin
2016-05-06 10:48 ` [dpdk-dev] [PATCH v1 06/10] app/test: extract test_pci_setup and test_pci_cleanup Jan Viktorin
2016-05-06 10:48 ` [dpdk-dev] [PATCH v1 07/10] app/test: convert current pci_test into a single test case Jan Viktorin
2016-05-06 10:48 ` [dpdk-dev] [PATCH v1 08/10] eal/pci: replace SYSFS_PCI_DEVICES with pci_get_sysfs_path() Jan Viktorin
2016-05-06 10:48 ` [dpdk-dev] [PATCH v1 09/10] app/test: scan PCI bus using a fake sysfs Jan Viktorin
2016-05-06 10:48 ` [dpdk-dev] [PATCH v1 10/10] app/test: do not dump PCI devices in blacklist test Jan Viktorin
2016-05-10 18:13 ` [dpdk-dev] [PATCH v2 00/11] Include resources in tests Jan Viktorin
2016-05-10 18:13 ` [dpdk-dev] [PATCH v2 01/11] app/test: introduce resources for tests Jan Viktorin
2016-05-12 15:13 ` Thomas Monjalon
2016-05-12 15:19 ` Thomas Monjalon
2016-05-12 15:26 ` Jan Viktorin
2016-05-10 18:13 ` [dpdk-dev] [PATCH v2 02/11] mk: define objcopy-specific target and arch Jan Viktorin
2016-05-10 18:13 ` [dpdk-dev] [PATCH v2 03/11] app/test: support resources externally linked Jan Viktorin
2016-05-10 18:13 ` [dpdk-dev] [PATCH v2 04/11] app/test: add functions to create files from resources Jan Viktorin
2016-05-10 18:13 ` [dpdk-dev] [PATCH v2 05/11] app/test: support resources archived by tar Jan Viktorin
2016-05-10 18:13 ` [dpdk-dev] [PATCH v2 06/11] app/test: use linked list to store PCI drivers Jan Viktorin
2016-05-12 15:31 ` Thomas Monjalon
2016-05-12 15:53 ` Jan Viktorin
2016-05-12 16:08 ` Thomas Monjalon
2016-05-12 21:00 ` Jan Viktorin
2016-05-12 21:44 ` Thomas Monjalon
2016-05-10 18:13 ` [dpdk-dev] [PATCH v2 07/11] app/test: extract test_pci_setup and test_pci_cleanup Jan Viktorin
2016-05-10 18:13 ` [dpdk-dev] [PATCH v2 08/11] app/test: convert current pci_test into a single test case Jan Viktorin
2016-05-12 15:34 ` Thomas Monjalon
2016-05-13 15:19 ` Jan Viktorin
2016-05-13 15:35 ` Thomas Monjalon
2016-05-10 18:13 ` [dpdk-dev] [PATCH v2 09/11] eal/pci: replace SYSFS_PCI_DEVICES with pci_get_sysfs_path() Jan Viktorin
2016-05-12 15:41 ` Thomas Monjalon
2016-05-12 15:46 ` Jan Viktorin
2016-05-12 16:10 ` Thomas Monjalon
2016-05-13 15:26 ` Jan Viktorin
2016-05-12 15:44 ` Thomas Monjalon
2016-05-10 18:13 ` [dpdk-dev] [PATCH v2 10/11] app/test: scan PCI bus using a fake sysfs Jan Viktorin
2016-05-10 18:13 ` [dpdk-dev] [PATCH v2 11/11] app/test: do not dump PCI devices in blacklist test Jan Viktorin
2016-05-17 18:34 ` [dpdk-dev] [PATCH v3 00/11] Include resources in tests Jan Viktorin
2016-06-13 8:12 ` [dpdk-dev] [PATCH v4 00/10] " Jan Viktorin
2016-06-13 8:20 ` Jan Viktorin
2016-06-13 15:07 ` [dpdk-dev] [PATCH v5 " Jan Viktorin
2016-06-13 18:32 ` Thomas Monjalon
2016-06-14 8:23 ` Hunt, David
2016-06-14 8:37 ` Thomas Monjalon
2016-06-14 8:59 ` [dpdk-dev] [PATCH] config: make libarchive optional Thomas Monjalon
2016-06-14 9:33 ` Jan Viktorin
2016-06-14 9:50 ` Thomas Monjalon
2016-06-14 10:01 ` [dpdk-dev] [PATCH v2] " Thomas Monjalon
2016-06-14 12:53 ` Jan Viktorin
2016-06-14 13:33 ` Thomas Monjalon
2016-06-13 15:07 ` [dpdk-dev] [PATCH v5 01/10] app/test: introduce resources for tests Jan Viktorin
2016-06-13 15:07 ` [dpdk-dev] [PATCH v5 02/10] mk: define objcopy-specific target and arch Jan Viktorin
2016-06-13 15:07 ` [dpdk-dev] [PATCH v5 03/10] app/test: support resources externally linked Jan Viktorin
2016-06-13 15:07 ` [dpdk-dev] [PATCH v5 04/10] app/test: add functions to create files from resources Jan Viktorin
2016-06-13 15:07 ` [dpdk-dev] [PATCH v5 05/10] app/test: support resources archived by tar Jan Viktorin
2016-06-13 15:07 ` [dpdk-dev] [PATCH v5 06/10] app/test: use linked list to store PCI drivers Jan Viktorin
2016-06-13 15:07 ` [dpdk-dev] [PATCH v5 07/10] app/test: extract test_pci_setup and test_pci_cleanup Jan Viktorin
2016-06-13 15:07 ` [dpdk-dev] [PATCH v5 08/10] app/test: convert current pci_test into a single test case Jan Viktorin
2016-06-13 15:07 ` [dpdk-dev] [PATCH v5 09/10] eal/pci: allow to override sysfs Jan Viktorin
2016-06-13 15:07 ` [dpdk-dev] [PATCH v5 10/10] app/test: do not dump PCI devices in blacklist test Jan Viktorin
2016-06-13 8:12 ` [dpdk-dev] [PATCH v4 01/10] app/test: introduce resources for tests Jan Viktorin
2016-06-13 8:12 ` [dpdk-dev] [PATCH v4 02/10] mk: define objcopy-specific target and arch Jan Viktorin
2016-06-13 8:12 ` [dpdk-dev] [PATCH v4 03/10] app/test: support resources externally linked Jan Viktorin
2016-06-13 8:12 ` [dpdk-dev] [PATCH v4 04/10] app/test: add functions to create files from resources Jan Viktorin
2016-06-13 8:12 ` [dpdk-dev] [PATCH v4 05/10] app/test: support resources archived by tar Jan Viktorin
2016-06-13 14:40 ` Thomas Monjalon
2016-06-13 14:43 ` Jan Viktorin
2016-06-13 8:12 ` [dpdk-dev] [PATCH v4 06/10] app/test: use linked list to store PCI drivers Jan Viktorin
2016-06-13 8:12 ` [dpdk-dev] [PATCH v4 07/10] app/test: extract test_pci_setup and test_pci_cleanup Jan Viktorin
2016-06-13 8:12 ` [dpdk-dev] [PATCH v4 08/10] app/test: convert current pci_test into a single test case Jan Viktorin
2016-06-13 8:12 ` [dpdk-dev] [PATCH v4 09/10] eal/pci: allow to override sysfs Jan Viktorin
2016-06-13 8:12 ` [dpdk-dev] [PATCH v4 10/10] app/test: do not dump PCI devices in blacklist test Jan Viktorin
2016-05-17 18:34 ` [dpdk-dev] [PATCH v3 01/11] app/test: introduce resources for tests Jan Viktorin
2016-05-19 8:51 ` Jan Viktorin
2016-05-17 18:34 ` [dpdk-dev] [PATCH v3 02/11] mk: define objcopy-specific target and arch Jan Viktorin
2016-05-17 18:34 ` [dpdk-dev] [PATCH v3 03/11] app/test: support resources externally linked Jan Viktorin
2016-05-17 18:34 ` [dpdk-dev] [PATCH v3 04/11] app/test: add functions to create files from resources Jan Viktorin
2016-05-17 18:34 ` Jan Viktorin [this message]
2016-05-17 18:34 ` [dpdk-dev] [PATCH v3 06/11] app/test: use linked list to store PCI drivers Jan Viktorin
2016-05-17 18:34 ` [dpdk-dev] [PATCH v3 07/11] app/test: extract test_pci_setup and test_pci_cleanup Jan Viktorin
2016-05-17 18:34 ` [dpdk-dev] [PATCH v3 08/11] app/test: convert current pci_test into a single test case Jan Viktorin
2016-05-17 18:34 ` [dpdk-dev] [PATCH v3 09/11] eal/pci: allow to override sysfs Jan Viktorin
2016-05-19 8:51 ` Jan Viktorin
2016-05-17 18:35 ` [dpdk-dev] [PATCH v3 10/11] app/test: scan PCI bus using a fake sysfs Jan Viktorin
2016-05-17 18:35 ` [dpdk-dev] [PATCH v3 11/11] app/test: do not dump PCI devices in blacklist test Jan Viktorin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1463510101-5633-6-git-send-email-viktorin@rehivetech.com \
--to=viktorin@rehivetech.com \
--cc=bruce.richardson@intel.com \
--cc=david.marchand@6wind.com \
--cc=dev@dpdk.org \
--cc=thomas.monjalon@6wind.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).