Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SQLITE Contention: Add retries and enable WAL mode #355

Merged
merged 5 commits into from
Dec 11, 2024
Merged

Conversation

jlewi
Copy link
Owner

@jlewi jlewi commented Dec 11, 2024

  • Using SQLITE's busy timeout didn't fix the issue with busy errors preventing sessions from being updated and learning from occuring.
  • Enabling WAL mode seems to have made a huge difference
    • WAL uses a separate file to log edits and allows concurrent reads and writes
  • Add logic at the application layer to retry the updates.
  • Use an atomic counter to track the number of concurrent updates to try to confirm
    we don't have multiple concurrent writes.
  • To better track issues with Analyzer lag start tracking the timestamp of the lag processed log message.
  • Add a Gauge metric to report Analyzer LAG

* Using SQLITE's busy timeout didn't fix the issue with busy errors preventing
  sessions from being updated and learning from occuring.

* Add logic at the application layer to retry the updates.
* Use an atomic counter to track the number of concurrent updates to try to confirm
  we don't have multiple concurrent writes.
Copy link

netlify bot commented Dec 11, 2024

Deploy Preview for foyle canceled.

Name Link
🔨 Latest commit bcab08b
🔍 Latest deploy log https://app.netlify.com/sites/foyle/deploys/6759fab937b5740008c4c2e1

…s will help us better understand the lag in log processing.
…database. Add a Gauge to measure lag between current time and last processed log time.
@jlewi jlewi changed the title Retries to deal with SQLITE_BUSY errors and DB contention SQLITE Contention: Add retries and enable WAL mode Dec 11, 2024
@@ -86,11 +86,9 @@ type ConvertDocRequest struct {

func (x *ConvertDocRequest) Reset() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
func (x *ConvertDocRequest) Reset() {
// Reset resets the ConvertDocRequest to its zero value.
func (x *ConvertDocRequest) Reset() {

add comment to Reset method

@@ -565,11 +551,9 @@ type GetBlockLogRequest struct {

func (x *GetBlockLogRequest) Reset() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
func (x *GetBlockLogRequest) Reset() {
// Reset resets the GetBlockLogRequest to its zero value.
func (x *GetBlockLogRequest) Reset() {

add comment to Reset method

@jlewi jlewi enabled auto-merge (squash) December 11, 2024 20:51
@jlewi jlewi merged commit 4add79b into main Dec 11, 2024
5 checks passed
@jlewi jlewi deleted the jlewi/sqlretries branch December 11, 2024 20:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant