Database scale querying

[AndyRae]

Is this the right issue type?

• Yes, I'm planning work for this project team.

Summary

Currently Bunny has only been tested on very modest OMOP dataset sizes. We need to ensure queries are resolved on much larger datasets quickly.
For example - population sizes across the SDE regions of the UK go up to 10million people, but ideally Bunny will be able to handle much larger than this.

This deliverable is not about fixing or optimising yet, but setting up the infrastructure to test Bunny at this scale, beginning to understand any issues, and setting an initial benchmark.

Benchmarks we are interested in initially:

• Time taken (T) to resolve a simple cold query (Q) across N patients

Acceptance Criteria

• Bunny has been tested on OMOP datasets of 100,000 synthetic patients
• Bunny has been tested on OMOP datasets of 1 million synthetic patients
• Bunny has been tested on OMOP datasets of 10 million synthetic patients
• Problems have been documented
• Benchmarks have been documented

Tasks

• Set up database
• Run queries
• Note benchmarks (this can just be commented on this issue for ease)

Confirm creation

• This issue is ready

[prquinlan]

Synthetic data created for 100k, 1M, 5M and 10M persons. Data available in our health informatics team. Created the 5M as well given 1M-10M is quite a jump.

[prquinlan]

Tested on 100k. All ok