Shared Memory Manager

Cross-process shared memory for multi-worker data loading.

See Also

• Memory Overview - Memory tools
• Data Loaders - Multi-worker loading
• Performance - Optimization
• Distributed - Multi-process training

---

datarax.memory.shared_memory_manager

Shared memory manager for multi-worker data pipeline scenarios.

logger module-attribute

logger = getLogger(__name__)

SharedMemoryManager

SharedMemoryManager()

Bases: Module

Manage shared memory arrays for multi-worker scenarios.

Automatically converts large numpy arrays to shared memory to avoid duplication across workers.

shared_blocks instance-attribute

shared_blocks = Variable({})

array_metadata instance-attribute

array_metadata = Variable({})

make_shared

make_shared(name: str, array: Array, force: bool = False) -> Array

Convert array to shared memory.

Parameters:

Name	Type	Description	Default
name	str	Name for the shared memory block	required
array	Array	Array to store in shared memory	required
force	bool	If True, always use shared memory regardless of size	False

Returns:

Type	Description
Array	The original array (shared memory is accessed via get_shared)

get_shared

get_shared(name: str) -> Array | None

Get shared array by name.

cleanup

cleanup() -> None

Clean up shared memory blocks.