Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SQLite/GPKG: turn on SQLite 'PRAGMA secure_delete=1' by default #11138

Merged
merged 1 commit into from
Oct 28, 2024

Conversation

rouault
Copy link
Member

@rouault rouault commented Oct 27, 2024

unless the user specifies a value of this pragma through OGR_SQLITE_PRAGMA.

For example, Debian and Conda-Forge SQLite3 builds already turn on secure_delete. Make sure that this is done for all SQLite3 builds.

@rouault rouault added backport release/3.10 Backport to release/3.10 branch funded through GSP Work funded through the GDAL Sponsorship Program labels Oct 27, 2024
unless the user specifies a value of this pragma through OGR_SQLITE_PRAGMA.

For example, Debian and Conda-Forge SQLite3 builds already turn on secure_delete.
Make sure that this is done for all SQLite3 builds.
@jratike80
Copy link
Collaborator

jratike80 commented Oct 27, 2024

I made some tests. SECURE_DELETE=ON makes deleting 5.6 million rows from a regular, non-spatial table 6 times slower, 2.5 sec vs 15 sec.
However, deleting massive amounts of data from SQLite is always very slow when tables have triggers, and that is the case with spatial geopackage tables.

Deleting a 5.6 million table with the same data as in the previous tests but now with r-tree, triggers and everything in geopackage requires this much time:
secure_delete=no 326 seconds
secure_delete=yes 370 seconds

Considerations about similar, but more compicated use case https://gis.stackexchange.com/questions/486885/better-control-cascading-triggers-in-sqlite

Note: My test was the most optimal for the regular table, a query that makes SQLite to take the fast "truncate table" road.

delete from rakennus;

@coveralls
Copy link
Collaborator

coveralls commented Oct 27, 2024

Coverage Status

coverage: 69.468% (-4.0%) from 73.478%
when pulling 5e84b2d on rouault:sqlite_gpkg_secure_delete
into a8117d9 on OSGeo:master.

@rouault
Copy link
Member Author

rouault commented Oct 27, 2024

Deleting a 5.6 million table with the same data as in the previous tests but now with r-tree, triggers and everything in geopackage requires this much time:

Will be vastly improved with #11139

@rouault
Copy link
Member Author

rouault commented Oct 28, 2024

Conda-Forge SQLite3 builds already turn on secure_delete

On deeper review, was actually only true of the Linux and Mac builds. Submitted conda-forge/sqlite-feedstock#116 to do that on Windows too

@rouault rouault merged commit 5095aa7 into OSGeo:master Oct 28, 2024
36 of 37 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport release/3.10 Backport to release/3.10 branch funded through GSP Work funded through the GDAL Sponsorship Program
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants