db.clear_cache() does not re-read new db.json file

[ktdreyer]

I'm writing a transaction context manager (similar to https://github.com/eugene-eeo/tinyrecord) that can potentially replace the underlying db.json file with a newer one (from a Git repo).

The docs state:

To clear the cache and read data from the storage again you can use db.clear_cache().

In my tests, when I call db.clear_cache(), read queries will still return old data. For example:

import os
from tinydb import TinyDB, Query

# Write some data to the file:
db = TinyDB('db.json')
db.insert({'type': 'apple', 'count': 7})

# Delete the file and clear the cache:
os.unlink('db.json')
db.clear_cache()

# Queries still show results, for example:
results = db.all()
print(results)   # prints the "apple" record

To fix it, I have to reach deep into the table's storage and re-open the file handle:

# Re-initialize storage file handle to the new inode
db.storage.close()
db.storage._handle = open('db.json', mode=db.storage._mode)

[msiemens]

I think this is more or less expected behavior. On a technical level, there is an object called a file handle for a path that is neede for reading/writing (created by open(path)). Generally, when the a file is deleted, the OS will NOT remove the file from disk until all file handles are closed.¹ This means that in this case the file handle will point to the old file, as you have described. If the file is removed and replaced, one will have to close and re-open the storage either way (clear_cache only clears the query cache from memory).

A solution would be to not replace but truncate and then write the file, but this will run into trouble when TinyDB is in the process of reading/writing when this happens. So, in the end, there's no clean solution to this problem, I fear.

¹ On Windows, deletion will typically be rejected if someone has an open file handle (see here), while on Linux/macOS it will be removed from the directory structure but not removed until all file handles are closed (see here).