The Pytorch documentation explicitly mentions this issue with DataLoader duplicating the underlying dataset (at least on Windows and macOS as I understand).

In general, you should not eagerly load all your dataset in memory because of such issue. The dataset should be lazy loaded, i.e. samples should only be loaded when they are accessed in the __getitem__ method.

If your whole dataset in stored on disk as a monolithic tensor, you could fragment it into individual samples and save them into a folder for instance.

You could then define your dataset as:

from torch.utils.data import Dataset, DataLoader
from glob import glob
from os.path import abspath


class MyDataset(Dataset):
  def __init__(self, folder: str):
    # Retrieve all tensor file names
    folder = abspath(folder)
    self.files = glob(f"{folder}/*.pt")

  def __getitem__(self, index: int):
    # Load tensors on demand
    return torch.load(self.files[index])
 
  def __len__(self) -> int:
    return len(self.files)

Another solution is to memory-map the dataset. This is what HuggingFace does for huge datasets, take a look here. This avoids loading the whole dataset in RAM and also allows it to be shared in multiple processes without copies.

The answer depends on your OS and settings. If you are using Linux with the default process start method, you don't have to worry about duplicates or process communication, because worker processes share memory! This is efficiently implemented as Inter Process Communication (IPC) through shared memory (some more details here). For Windows, things are more complicated. From the documentation:

Since workers rely on Python multiprocessing, worker launch behavior is different on Windows compared to Unix.

On Unix, fork() is the default multiprocessing start method. Using fork(), child workers typically can access the dataset and Python argument functions directly through the cloned address space.

On Windows, spawn() is the default multiprocessing start method. Using spawn(), another interpreter is launched which runs your main script, followed by the internal worker function that receives the dataset, collate_fn and other arguments through pickle serialization.

This means that your dynamically cached Dataset members would be automatically shared between all processes on Linux. That's great! However, on Windows, processes will not have received copies of them (they only received the Dataset upon spawning), so you should use a process communication scheme, e.g. through multiprocessing Pipe, Queue or Manager (preferred for broadcasting to multiple processes, but you would have to convert tensors to lists). This is not very efficient, besides rather bothersome to implement.

Nevertheless, there is another method: memory mapping (memmaping). This means that your objects will be written to virtual memory, and again all processes will have access to it, while a respective "shadow copy" of these objects will at some point be flushed and exist on your hard drive (can be placed in a /tmp directory). You can use memmaping with the mmap module, in which case your objects will have to be serialized as a binary file, or you can use numpy.memmap. You can find more details here.