R package that allows package users to download and cache optional data files in a local directory.
Allow Users of Your Package to Download and Manage Optional Package Data
This package allows you to give users of your package an easy way to download and manage optional data for your package. The data can be hosted on your webserver or any location accessible via HTTP(S) over the internet, and will be stored locally in a permanent directory called a package file cache. Checking of MD5 sums is supported, and re-downloading only ever occurs if MD5 sums mismatch if they are available.
The pkgfilecache package is on CRAN, so you can simply:
install.packages("pkgfilecache")
When package authors want to ship data for their package, they will quickly hit the package size limit on CRAN (which is 5 MB as of September 2019). The solution is to host the data elsewhere and download it on demand when the user requests it, then store it for future use. This is what pkgfilecache allows you to do. You can put your files onto a web server of your choice, take the MD5 sums, and have pkgfilecache download them locally. Files are automatically compared with the local package cache direcory, and only missing files or files with incorrect MD5 checksums will be downloaded. Users can then access the data in a convenient way, similar to accessing files shipped in inst/extdata via system.file. They can also erase the data if it is no longer needed.
The intended way of using pkgfilecache is to not call the download function in your package code, but have it as part of your API that the user can decide to call if they want to download the optional data. However, you are of course free to call the download function in your unit test code, which will only be run by developers or continuous integration systems.
You specify a list of optional data files, and package users can download them with a single command from within R. For each file, you provide:
Users can then access the file by the local filename. See the documentation for details.
Specifying files as three parallel vectors (local filenames, URLs, MD5 sums) is fine for a handful of files, but it gets tedious when your package has many optional data files. In that case, you can describe the files in a single declarative manifest, one row per file, either as a data.frame or as a CSV file shipped with your package.
A manifest has a path column (the file path relative to the package cache, using / as separator) and two optional columns: url (the download URL) and md5 (the checksum). If the remote directory layout mirrors the local one, you can omit the url column and derive the URLs from a base_url:
# A small manifest as a data.frame: two files, URLs derived from base_url.
manifest = data.frame(
path = c("dir1/file1.txt", "dir2/file2.txt"),
md5 = c("35261471bcd198583c3805ee2a543b1f", "85ffec2e6efb476f1ee1e3e7fddd86de")
);
cfiles = pkgfilecache::ensure_files_available_from_manifest(pkg_info, manifest, base_url = "https://your.server/yourpackage/");
You can also ship the manifest as a CSV file in your package (e.g., in inst/extdata) and pass its path instead of a data.frame; comment lines starting with # are ignored. Use read_manifest() to inspect or validate a manifest.
To avoid computing MD5 sums by hand, generate the manifest from a local directory of files:
# Put your optional data files into a directory (subdirectories are preserved), then:
pkgfilecache::write_manifest_from_dir(
dir = "~/yourpackage_data",
out = "inst/extdata/files.csv",
url_base = "https://your.server/yourpackage/"
);
# The MD5 checksums are computed for you. Add the CSV to your package.
If you prefer working in a terminal, you can generate the manifest CSV directly from your shell, without opening an interactive R session. The package provides a small command line interface for this. The recommended way is to call manifest_cli() from Rscript in a one-liner; it reads the command line arguments for you:
# One-liner, no script file needed:
Rscript -e 'pkgfilecache::manifest_cli()' --args --dir ~/yourpackage_data --out files.csv --url-base https://your.server/yourpackage/
The arguments mirror those of write_manifest_from_dir():
--dir <path>: the directory containing your data files (required)--out <path>: the CSV file to write (required)--url-base <url>: optional base URL used to derive the download URLs--help: print usage information and exitIf you prefer a standalone script file, the package also ships a thin wrapper script called make_manifest.R (in the exec directory of the installed package) that does exactly the same thing. Get its path with pkgfilecache::manifest_script() and run it with Rscript:
Rscript "$(Rscript -e 'cat(pkgfilecache::manifest_script())')" --dir ~/yourpackage_data --out files.csv --url-base https://your.server/yourpackage/
On Unix-like systems the script is executable, so you can copy it to a directory on your PATH (e.g. ~/bin) and run it like a normal command. If you have not installed the package yet, you can also download the script directly from the GitHub releases page of this package (for a release tagged <tag>, use https://raw.githubusercontent.com/dfsp-spirit/pkgfilecache/<tag>/exec/make_manifest.R). Note that the package must be installed for the script to work, as the script itself only contains a thin wrapper around the package functions.
By default, ensure_files_available() downloads the files that are missing (or that have an incorrect MD5 sum) in parallel: several files are downloaded at the same time using the curl multi interface, which can speed up downloading many small files considerably. The number of simultaneous connections defaults to 2, which is a safe choice for most servers and for CRAN checks.
You can control the number of connections in two ways:
num_connections argument of ensure_files_available(). Use num_connections=1 for strictly sequential downloads (e.g., when a server is slow or rate-limited), or a higher number to speed up downloading many files.options(pkgfilecache.num_connections = N). This changes the default for all calls that do not specify the num_connections argument themselves, for example the calls made by packages that use pkgfilecache to manage their optional data. Note that an explicit num_connections argument always takes precedence over the option.Downloads that fail (e.g., because a server closes or throttles a connection) are retried automatically up to num_retries times (default 2) using fresh connections, so temporary network hiccups do not cause files to be reported as missing. Set num_retries = 0 to disable retrying.
The package cache is a permanent directory on your system. By default it is located in the directory returned by tools::R_user_dir(packagename, "data") (for R version 4.0 or later), which is the location recommended by the CRAN repository policy for user-specific data and cache files (e.g., ~/.local/share/R/mypackage on Linux). On R versions before 4.0, the directory returned by rappdirs::user_data_dir is used. If a cache from an older version of this package already exists at the legacy location, it is reused, so you do not have to download your files again.
You can control the location with R options:
options(pkgfilecache.cachedir = "/some/dir"): Use /some/dir as the root of the package cache (the package name and optional version are appended). Useful for ramdisks or network drives.options(pkgfilecache.use_tempdir = TRUE): Use a subdirectory of the R session's temporary directory (tempdir()). Everything is cleaned up automatically when the R session ends. This is handy for unit tests and CI systems that must not write to the user's home directory.Use get_cache_dir(pkg_info) to find out where the cache is located in your current session.
See the vignette for more detailed examples!
pkg_info = pkgfilecache::get_pkg_info("yourpackage"); # to identify the cache dir
### Specify your optional data:
# 1) How the files should be called in the local package file cache
local_filenames = c("file1.txt", "file2.txt");
# 2) Where they can be downloaded
urls = c("https://your.server/yourpackage/large_file1.txt", "https://your.server/yourpackage/large_file2.txt");
# 3) Optional, but highly recommended: MD5 checksums for the files.
md5sums = c("35261471bcd198583c3805ee2a543b1f", "85ffec2e6efb476f1ee1e3e7fddd86de");
# Now use the package cache to get the files. Will only download if needed (file missing or MD5 mismatch):
cfiles = pkgfilecache::ensure_files_available(pkg_info, local_filenames, urls, md5sums=md5sums);
# Great, now let's access a file:
local_file_full_path = pkgfilecache::get_filepath(pkg_info, "file1.txt", mustWork=TRUE);
Full documentation is built-in, and can be accessed from within R in the usual ways. A vignette is also included:
library("pkgfilecache")
browseVignettes("pkgfilecache")
You can also read the pkgfilecache vignette online and read the full API docs online.
It is not allowed to store data in the user directory on CRAN servers, not even temporarily. So please do not use this package to download data into the user directory in unit tests on CRAN. In your test setup, set options(pkgfilecache.use_tempdir = TRUE) to redirect all downloads to the R session's temporary directory, which is allowed, or use testthat::skip_on_cran() at the top of test functions that require/download external data from running on CRAN. You should test on your CI provider instead, and limit CRAN unit tests to those with data that can be generated in the test code.
MIT