You need Copy-on-write type memory mapping for that, not write-through.

mapped_file = mmap.mmap(json_file.fileno(), 0, access=mmap.ACCESS_COPY)

From the Python's mmap module doc:

For both the Unix and Windows versions of the constructor, access may be specified as an optional keyword parameter. access accepts one of four values: ACCESS_READ, ACCESS_WRITE, or ACCESS_COPY to specify read-only, write-through or copy-on-write memory respectively, or ACCESS_DEFAULT to defer to prot. access can be used on both Unix and Windows. If access is not specified, Windows mmap returns a write-through mapping. The initial memory values for all three access types are taken from the specified file. Assignment to an ACCESS_READ memory map raises a TypeError exception. Assignment to an ACCESS_WRITE memory map affects both memory and the underlying file. Assignment to an ACCESS_COPY memory map affects memory but does not update the underlying file.

Answer from Nikhil Saxena on Stack Overflow
🌐
Python
docs.python.org › 3 › library › mmap.html
mmap — Memory-mapped file support
A memory-mapped file is created by the mmap constructor, which is different on Unix and on Windows. In either case you must provide a file descriptor for a file opened for update.
🌐
Real Python
realpython.com › python-mmap
Python mmap: Improved File I/O With Memory Mapping – Real Python
June 9, 2023 - You don’t need to understand page tables or logical-to-physical mapping to write performant I/O code in Python. However, knowing a little bit about memory gives you a better understanding of what the computer and libraries are taking care of for you. mmap uses virtual memory to make it appear that you’ve loaded a very large file into memory, even if the contents of the file are too big to fit in your physical memory.
🌐
Oxford Protein Informatics Group
blopig.com › blog › 2024 › 08 › memory-mapped-files-for-efficient-data-processing
Memory-mapped files for efficient data processing | Oxford Protein Informatics Group
August 28, 2024 - In the context of Python, a common approach to memory-mapped files is the mmap() system call. It is often used in place of malloc(). mmap() requests the operating system to map a file or device into the process’s memory.
🌐
Reddit
reddit.com › r/python › can someone explain memory mapping to me?
r/Python on Reddit: Can someone explain memory mapping to me?
January 30, 2013 -

I'm getting into an application where it seems a lot of people are using mmap. Not particularly sure what it does

I've combed through the documentation so I have a sense for what it does in practice, I'm just not quite grasping the theory behind it.

Top answer
1 of 2
18
Normally when you read data from a file, what happens is that the operating system reads the data into the filesystem cache (which belongs to the operating system, not your process), and then copies it again out of the cache and into the buffer in your process where you requested the data to be put. You can save that extra copy by instead asking the operating system to make that portion of its filesystem cache available in your process' address space. The file appears in memory as if it was read there, but there was no explicit call to read() to put it there. And that memory is not part of your regular program memory, i.e. it is not under the care of your language's memory allocator. Each memory extent in your process is a mapping -- the executables and libraries are mapped into your address space, and working memory like the stack and heap are anonymous mappings. In addition to saving a copy, mmaping a file has the advantage of being lazy -- the entire file (or a selected segment, specified by (offset, length)) appears in memory, but only those pages that are accessed need to actually be read or written from disk. Writing can be a little bit dangerous, in that just like the explicit read() step was eliminated, so is write() -- assuming you mapped the file for read/write access. Modifying the buffer in memory is the same as modifying the corresponding bytes on disk, although that will generally happen after some delay that is determined by the operating system's IO scheduler, but that effect is not user-visible, i.e. the change is committed as soon as the write happens, including to other processes that might try to read the file, even if it hasn't yet been physically written out to disk yet.
2 of 2
1
I have seen it used for inter-process communication. A program runs, maps a file and starts writing counters at given offsets. Then multiple programs map the same file and read the counters and display graphs or send messages etc. Wikipedia gives a good technical description.
🌐
YouTube
youtube.com › real python
File I/O With Memory Mapping Using Python mmap - YouTube
Python’s mmap provides memory-mapped file input and output (I/O). It allows you to take advantage of lower-level operating system functionality to read files...
Published   June 23, 2022
Views   6K
🌐
rand[om]
ricardoanderegg.com › posts › mmap-share-data-between-processes
Using mmap to share data between processes - rand[om]
March 20, 2023 - Python comes with some utilities to do this more easily. But this approach can be used regardless of the programming language that the program uses. It also lets you persist the data to disk after the programs terminate (except if the memory-mapped file is stored in a temporary location, as I did in this example). ... import tempfile import mmap fd = tempfile.NamedTemporaryFile() print(fd.name) size = 1000 fd.seek(size - 1) fd.write(b"1") fd.flush() m = mmap.mmap(fd.fileno(), size) m[:10] = b"1" * 10
Find elsewhere
🌐
Python Module of the Week
pymotw.com › 2 › mmap
mmap – Memory-map files - Python Module of the Week
The second argument to mmap() is a size in bytes for the portion of the file to map. If the value is 0, the entire file is mapped.
🌐
YouTube
youtube.com › watch
Unlock Python's mmap Module: Efficient Memory-Mapped ...
Enjoy the videos and music you love, upload original content, and share it all with friends, family, and the world on YouTube.
Top answer
1 of 1
3

Spooled temporary files

You can simplify the distinction by saying that a SpooledTemporaryFile is a piece of memory that pretends to be a file (until it starts to be a file), while a memory mapped file is a file pretending to be a piece of memory.

The point of a SpooledTemporaryFile is that you may not know a-priori whether the data is large enough to justify writing a temporary file. For example if you handle images, they might be tiny thumbnails or large HD images. If the data turns out to be tiny, the overhead of handling actual files outweighs the benefits, so the spooled file gives you a handy abstraction handling both cases.

This also means there is no point in using them compared to a regular temporary file if you want to memory-map it anyway. At that point it needs to have been written to disk.

Memory mapping

Memory mapping on the other hand works through the page cache (at least that's what it's called on Linux, I'm not sure if Windows uses the same name but they have the same mechanism). The OS uses memory that is not currently used for anything else to cache its disk content. When you read or write a file, the OS normally just copies between your read/write buffer and the page cache; and then of course it schedules the actual write to disk from the page cache or the read to fill the page cache with the requested data.

When you memory-map, the memory pages in the page cache are mapped into the address space of your application so that you can directly access them. When you try to read a page that is not currently in the cache, it will cause an access fault by the CPU. Then the OS takes over and resolves that fault by loading the page in. Conversely, when you write to a page, the OS will track that the page was "dirtied" and write it back to disk at some point. When the OS needs to make room in the page cache, it will pick some pages, make sure that their on-disk representation is consistent, and then drop the page. The next access to that page through mapped memory will then cause another access fault and the whole thing repeats.

It is interesting to note that the same way a memory-mapped file is backed by its on-disk file, regular anonymous memory is backed by the swap file. It's practically the same mechanism: If you access a piece of memory that was swapped out, it faults and the OS reloads the page from swap. And if the OS runs out of memory, it picks pages to move into swap storage.

Comparison

This means that all options will lead the OS to the same mechanism for limiting memory use: If you keep it in normal memory, the OS may swap it out, if you put it into a temporary file, the OS will move it out of page cache, and if you memory map it, it will do the same. However, there are some differences:

  • The OS prioritizes anonymous memory over page cache and is more likely to evict page cache pages than normal memory. On Linux you can control this via /proc/sys/vm/swappiness. This also means that keeping data in memory may make the system perform worse as the page cache shrinks
  • The OS may not be able to swap if none is configured. Then the OS will just start killing memory-hungry processes
  • The OS typically doesn't know that you write a temporary file. So it writes to disk very quickly and doesn't keep the data purely in memory for crash safety. Therefore a regular temporary file will often result in disk IO even if your use is brief. Also, writing to the page cache is often artificially rate-limited to match the disk performance
  • At least on Linux it is not uncommon for the /tmp directory to be a tmpfs ram-disk. So you are really just writing to an in-memory buffer. But this is something the system admin controls and presumably enabled knowing their system has enough RAM to handle it
  • Memory mapping can have worse performance in sequential accesses compared to normal IO. It can lead to excessive numbers of access faults. Part of the issue is that all the OS sees is that you wanted to access a 4 kiB page. It doesn't know how much data you want to read/write overall. I've seen it absolutely tank performance for large streaming I/O. It's much more suitable for random or repeated access
  • On the other hand, for these repeated accesses, it avoids the cost of doing system calls
  • It also moves the burden of figuring out what to keep in memory and what to store on disk to the OS which may or may not be a good thing depending on your skill for predicting this compared to the OS

Conclusion

It is hard to give good advise on what you should use. Much of it depends on your access pattern and your system. How much RAM do you have, how much do you need? How fast is your disk? Broadly speaking the order is temporary file -> memory mapping -> normal array going from most memory savings to highest effective performance, if used properly.

🌐
Medium
medium.com › @nishanth-g › mmaps-in-python-63b044c3014a
MMap's in Python
On Medium, anyone can share insightful perspectives, useful knowledge, and life wisdom with the world.
🌐
Real Python
realpython.com › videos › python-mmap-io-overview
Python mmap: Doing File I/O With Memory Mapping (Overview) (Video) – Real Python
This course is about the mmap library, a wrapper to a fairly low-level operating system call that maps the contents of a file into memory. 00:17 In this course, you will learn about how to map a file on disk into a memory block and why you might ...
Published   June 21, 2022
🌐
GitHub
gist.github.com › evandrocoan › bbd3aa06435428025f5a6c913aaa99ba
Example of shared memory usage in python using mmap! · GitHub
import mmap from multiprocessing.shared_memory import SharedMemory import subprocess import os # Create a shared memory string shared_str = "Hello from parent process!".encode("utf-8") filesize = len(shared_str) * 4 shared_mem = SharedMemory(name='MyMemory', size=1024, create=True) shared_mem.buf[0] = 1 shared_mem.buf[1] = 2 # deck_json_bin = deck_json.encode("utf-8") # filesize = len(deck_json_bin) # shared_mem = SharedMemory(name='MyMemory', size=filesize, create=True) # shared_mem.buf[:filesize] = deck_json_bin with open("hello.txt", "wb") as f: f.write(b"Hello Python!\n") with open("hello.
🌐
W3Schools
w3schools.com › python › ref_module_mmap.asp
Python mmap Module
Python Examples Python Compiler ... Python Bootcamp Python Training ... The mmap module provides memory-mapped file support, exposing file contents directly in memory....
🌐
NumPy
numpy.org › devdocs › reference › generated › numpy.memmap.html
numpy.memmap — NumPy v2.6.dev0 Manual
Memory-mapped files are used for accessing small segments of large files on disk, without reading the entire file into memory. NumPy’s memmap’s are array-like objects. This differs from Python’s mmap module, which uses file-like objects.
🌐
Pythoncomplexity
pythoncomplexity.com › stdlib › mmap
Mmap - Python Big-O: Time & Space Complexity
import mmap with open('data.bin', 'rb') as f: mm = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) idx = mm.find(b'needle') # O(n) search if idx != -1: print('Found at', idx) mm.close() Mapping large files avoids copying data into Python space for random access.
🌐
GitHub
github.com › friedelwolff › fmmap
GitHub - friedelwolff/fmmap: A fast implementation of mmap in Python · GitHub
This module is a reimplementation of Python's builtin module mmap. It aims to provide better performance while being API compatible with the builtin module. Development tracks new Python versions, therefore this module is mostly usable as a backport to older Python versions -- consult the documentation about any changes to the mmap API in Python.
Starred by 15 users
Forked by 2 users
Languages   Python 96.5% | C 3.5%
🌐
Lbl
davis.lbl.gov › Manuals › PYTHON-2.5.1 › lib › module-mmap.html
15.6 mmap -- Memory-mapped file support
April 18, 2007 - A memory-mapped file is created by the mmap() function, which is different on Unix and on Windows. In either case you must provide a file descriptor for a file opened for update. If you wish to map an existing Python file object, use its fileno() method to obtain the correct value for the fileno ...