Skip to content

Goodput Tracking¤

Track effective training goodput — the fraction of wall-clock time spent on useful computation rather than stalls.

See Also¤

datarax.performance.goodput ¤

Goodput/badput telemetry for pipeline performance visibility.

Tracks time breakdown per batch: - Source time: data loading from source - Transform time: operator execution - Transfer time: host-to-device transfers - Overhead: control flow, Python, framework overhead

Goodput ratio = productive_time / wall_time, where productive_time = source + transform + transfer. Overhead is the remainder.

Usage::

tracker = GoodputTracker()
for batch in pipeline:
    tracker.start_batch()
    with tracker.time_source():
        data = source.get_batch()
    with tracker.time_transform():
        result = transform(data)
    with tracker.time_transfer():
        device_data = jax.device_put(result)
        block_until_ready_tree(device_data)
    tracker.end_batch()

print(tracker.summary())

logger module-attribute ¤

logger = logging.getLogger(__name__)

GoodputMetrics dataclass ¤

GoodputMetrics(total_batches: int, wall_clock_sec: float, source_sec: float, transform_sec: float, transfer_sec: float, productive_sec: float, overhead_sec: float, goodput_ratio: float)

Summary metrics from a GoodputTracker.

total_batches instance-attribute ¤

total_batches: int

wall_clock_sec instance-attribute ¤

wall_clock_sec: float

source_sec instance-attribute ¤

source_sec: float

transform_sec instance-attribute ¤

transform_sec: float

transfer_sec instance-attribute ¤

transfer_sec: float

productive_sec instance-attribute ¤

productive_sec: float

overhead_sec instance-attribute ¤

overhead_sec: float

goodput_ratio instance-attribute ¤

goodput_ratio: float

GoodputTracker ¤

GoodputTracker()

Lightweight pipeline telemetry tracker.

Records per-batch time breakdowns and computes goodput/badput ratios. Thread-safe for single-producer usage (one batch at a time).

per_batch_source instance-attribute ¤

per_batch_source: list[float] = []

per_batch_transform instance-attribute ¤

per_batch_transform: list[float] = []

per_batch_transfer instance-attribute ¤

per_batch_transfer: list[float] = []

per_batch_wall instance-attribute ¤

per_batch_wall: list[float] = []

start_batch ¤

start_batch() -> None

Mark the beginning of a new batch.

record_source ¤

record_source(duration_sec: float) -> None

Record source loading time for current batch.

record_transform ¤

record_transform(duration_sec: float) -> None

Record transform execution time for current batch.

record_transfer ¤

record_transfer(duration_sec: float) -> None

Record host-to-device transfer time for current batch.

time_source ¤

time_source() -> Generator[None, None, None]

Context manager to time source loading.

time_transform ¤

time_transform() -> Generator[None, None, None]

Context manager to time transform execution.

time_transfer ¤

time_transfer() -> Generator[None, None, None]

Context manager to time host-to-device transfer.

end_batch ¤

end_batch() -> None

Mark the end of the current batch and record times.

reset ¤

reset() -> None

Clear all accumulated metrics.

summary ¤

summary() -> GoodputMetrics

Compute summary metrics from accumulated data.