Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add statique materialized view #305

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

jmaupetit
Copy link
Member

@jmaupetit jmaupetit commented Dec 18, 2024

Purpose

Having a SQL materialized view for "statique" data would ease recurrent database queries joining the same tables frequently. Moreover this should speed up database queries and response time for API endpoints using them.

Proposal

  • add sqlalchemy_utils dependency to ease materialized view management
  • add statique materialized view
  • test the materialized view
  • integrate a refresh strategy

@jmaupetit jmaupetit self-assigned this Dec 18, 2024
@jmaupetit jmaupetit marked this pull request as draft December 18, 2024 18:31
@jmaupetit jmaupetit added critical improvement WIP API Item related to the API service labels Dec 18, 2024
@jmaupetit jmaupetit force-pushed the api-add-statique-materialized-view branch from 36bcd7a to a9602f0 Compare January 16, 2025 16:15
@MTES-MCT MTES-MCT deleted a comment from github-actions bot Jan 16, 2025
@MTES-MCT MTES-MCT deleted a comment from github-actions bot Jan 16, 2025
@MTES-MCT MTES-MCT deleted a comment from github-actions bot Jan 23, 2025
@jmaupetit jmaupetit force-pushed the api-add-statique-materialized-view branch from 7fc74b5 to 0491ea7 Compare January 23, 2025 21:56
@jmaupetit jmaupetit marked this pull request as ready for review January 23, 2025 21:56
@jmaupetit jmaupetit force-pushed the api-add-statique-materialized-view branch from 0491ea7 to 35531f5 Compare January 24, 2025 10:46
Having a SQL materialized view for "statique" data will ease recurrent
database queries joining the same tables frequently. Moreover this will
speed up database queries and response time for API endpoints using
them.
@jmaupetit jmaupetit force-pushed the api-add-statique-materialized-view branch from 35531f5 to 023973c Compare January 24, 2025 17:02
Copy link

Current benchmark

Type Name Request Count Failure Count Median Response Time Average Response Time Min Response Time Max Response Time Average Content Size Requests/s Failures/s 50% 66% 75% 80% 90% 95% 98% 99% 99.9% 99.99% 100% git timestamp version
GET /auth/whoami 29 0 460.0 474.3228568620678 5.245078000001513 1222.5335109999946 160.0 1.0014214797431562 0.0 460 660 810 1000 1000 1100 1200 1200 1200 1200 1200 6776aaa 2025-01-24 17:06:35.437427+00:00 0.16.0
POST /dynamique/session/ 30 0 510.0 771.1582672333342 6.722271999990426 2177.784402000015 45.0 1.0359532549067134 0.0 820 1100 1200 1300 1700 1900 2200 2200 2200 2200 2200 6776aaa 2025-01-24 17:06:35.437427+00:00 0.16.0
POST /dynamique/session/bulk 39 0 780.0 739.305394589743 12.384120000007217 1939.8260000000107 411.0 1.3467392313787274 0.0 780 1100 1300 1300 1400 1900 1900 1900 1900 1900 1900 6776aaa 2025-01-24 17:06:35.437427+00:00 0.16.0
GET /dynamique/status/ 21 0 550.0 647.5575199047579 8.909067999979925 1677.210658000007 285.0 0.7251672784346993 0.0 550 1000 1300 1300 1400 1400 1700 1700 1700 1700 1700 6776aaa 2025-01-24 17:06:35.437427+00:00 0.16.0
POST /dynamique/status/ 31 0 810.0 785.6255065161297 12.821968999986666 2037.060359999998 45.0 1.0704850300702704 0.0 810 1100 1300 1300 1400 1500 2000 2000 2000 2000 2000 6776aaa 2025-01-24 17:06:35.437427+00:00 0.16.0
GET /dynamique/status/FRALLEGO002006P3 32 0 620.0 719.926928375001 5.429090999996333 2290.63915499998 283.0 1.1050168052338276 0.0 620 1100 1100 1100 1900 1900 2300 2300 2300 2300 2300 6776aaa 2025-01-24 17:06:35.437427+00:00 0.16.0
GET /dynamique/status/FRALLEGO002006P3/history 32 0 790.0 960.5728885 87.52567099998032 2569.276335999973 40089.375 1.1050168052338276 0.0 990 1100 1300 1400 2200 2600 2600 2600 2600 2600 2600 6776aaa 2025-01-24 17:06:35.437427+00:00 0.16.0
POST /dynamique/status/bulk 23 0 860.0 814.0468863913023 13.14913100000581 2884.4553819999987 411.0 0.7942308287618135 0.0 860 1000 1300 1300 1700 1900 2900 2900 2900 2900 2900 6776aaa 2025-01-24 17:06:35.437427+00:00 0.16.0
POST /statique/ 41 21 800.0 774.151556780486 17.7337659999921 1943.409000999992 57.609756097560975 1.4158027817058416 0.7251672784346993 800 1000 1100 1100 1400 1900 1900 1900 1900 1900 1900 6776aaa 2025-01-24 17:06:35.437427+00:00 0.16.0
GET /statique/?limit=10 38 0 790.0 870.7314088421044 22.421947999987424 2292.633448999993 13826.447368421052 1.3122074562151702 0.0 800 1200 1300 1300 2100 2300 2300 2300 2300 2300 2300 6776aaa 2025-01-24 17:06:35.437427+00:00 0.16.0
GET /statique/?limit=100 40 0 760.0 907.136470375 146.8240279999975 2301.082009999988 129006.45 1.3812710065422844 0.0 780 1100 1300 1400 1900 2200 2300 2300 2300 2300 2300 6776aaa 2025-01-24 17:06:35.437427+00:00 0.16.0
PUT /statique/FR073EP7QU5233 44 0 660.0 726.1469156590914 32.855054999998856 2294.1222420000145 1371.0 1.5193981071965128 0.0 700 910 1100 1100 1400 1900 2300 2300 2300 2300 2300 6776aaa 2025-01-24 17:06:35.437427+00:00 0.16.0
GET /statique/FRALLEGO002006P3 23 0 510.0 556.0471869565197 15.161865000010266 2187.5404589999903 1202.0 0.7942308287618135 0.0 510 640 910 930 1300 1300 2200 2200 2200 2200 2200 6776aaa 2025-01-24 17:06:35.437427+00:00 0.16.0
POST /statique/bulk 33 0 1100.0 1178.3063402121195 344.3280879999975 2297.233513999998 1123.6060606060605 1.1395485803973846 0.0 1100 1300 1300 1800 1900 2300 2300 2300 2300 2300 2300 6776aaa 2025-01-24 17:06:35.437427+00:00 0.16.0
Aggregated 456 21 760.0 791.4743802565782 5.245078000001513 2884.4553819999987 15666.320175438595 15.746489474582043 0.7251672784346993 760 1100 1200 1300 1400 1900 2300 2300 2900 2900 2900 6776aaa 2025-01-24 17:06:35.437427+00:00 0.16.0

Comparison with the latest previous benchmark

A lower (negative) value means the current version performs better than the previous one.

Type Name Request Count Failure Count Median Response Time Average Response Time Min Response Time Max Response Time Average Content Size Requests/s
GET /auth/whoami
POST /dynamique/session/
POST /dynamique/session/bulk
GET /dynamique/status/
POST /dynamique/status/
GET /dynamique/status/FRALLEGO002006P3
GET /dynamique/status/FRALLEGO002006P3/history
POST /dynamique/status/bulk
POST /statique/
GET /statique/?limit=10
GET /statique/?limit=100
PUT /statique/FR073EP7QU5233
GET /statique/FRALLEGO002006P3
POST /statique/bulk
Aggregated

@MTES-MCT MTES-MCT deleted a comment from github-actions bot Jan 24, 2025
@MTES-MCT MTES-MCT deleted a comment from github-actions bot Jan 24, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
API Item related to the API service critical improvement needs review
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant