Cache Utilities

Dataset cache-layout helpers for resolving and structuring on-disk cache directories.

See Also

• Utilities Overview - All utility helpers
• Data Sources - Sources that use caching

---

datarax.utils.cache

Two-tier cache helpers for dataset adapters.

Datasets that download external archives (HuggingFace, TFDS, ArrayRecord, wearable physiology, etc.) all share the same cache lifecycle: download a raw archive once, extract or preprocess it into a stable layout, then re-use the processed artifacts on every subsequent run. Centralizing the layout in this module prevents each adapter from inventing its own subtly-different cache directory layout.

Layout

::

<base>/
  raw/        ← downloaded archives, large opaque files
  processed/  ← stable per-source preprocessed outputs (.npy, sharded files,
                intermediate caches)

The split allows the raw/ archive to be deleted after preprocessing without invalidating the cached training artifacts in processed/.

CacheLayout dataclass

CacheLayout(base: Path, raw: Path, processed: Path)

Two-tier cache structure: raw archives + processed artifacts.

base instance-attribute

base: Path

raw instance-attribute

raw: Path

processed instance-attribute

processed: Path

make_layout

make_layout(base: Path) -> CacheLayout

Construct a CacheLayout rooted at base.

The two subdirectories are not created eagerly; callers create them on first write so a read-only audit (e.g., processed_cache_ready) does not leave empty directories behind.

resolve_cache_dir

resolve_cache_dir(user_path: str | Path | None, *, default_name: str) -> Path

Resolve a user-provided cache dir or fall back to a default.

Default falls under $XDG_CACHE_HOME/<default_name> when XDG_CACHE_HOME is set, otherwise ~/.cache/<default_name>. This matches the convention used by Hugging Face Datasets and TFDS.

Parameters:

Name	Type	Description	Default
user_path	str | Path | None	Explicit cache directory, or None to use the default.	required
default_name	str	Subdirectory name to append to the default root.	required

Returns:

Type	Description
Path	Absolute Path to the cache directory (does not create it).

processed_cache_ready

processed_cache_ready(processed: Path, required_files: Iterable[str]) -> bool

Return True only if every required file exists under processed.

Used by dataset constructors to short-circuit re-preprocessing when a previous run has already populated the processed cache.

Parameters:

Name	Type	Description	Default
processed	Path	Path to the processed/ subdirectory.	required
required_files	Iterable[str]	Names (or relative paths) that must all exist.	required

Returns:

Type	Description
bool	True iff every file in required_files exists under processed.

ensure_zip_member_extracted

ensure_zip_member_extracted(archive: Path, extract_root: Path, member_name: str, *, target_name: str | None = None) -> Path

Idempotently extract a single zip member to extract_root.

First call extracts the member to extract_root / (target_name or Path(member_name).name). Subsequent calls are no-ops (returns the existing path without re-extraction), so this helper is safe to call on every dataset construction.

Parameters:

Name	Type	Description	Default
archive	Path	Path to the source .zip file.	required
extract_root	Path	Directory under which the member will be placed.	required
member_name	str	Path of the member inside the archive.	required
target_name	str | None	Optional override for the on-disk filename.	None

Returns:

Type	Description
Path	Path to the extracted file.