4 - Speed up migration running tasks #65
Assignees
Labels
api: spanner
Issues related to the googleapis/ruby-spanner-activerecord API.
priority: p2
Moderately-important priority. Fix may not be included in next release.
type: feature request
‘Nice-to-have’ improvement, new feature or different behavior or design.
Comments
hengfengli
added
type: feature request
product-auto-label
bot
added
the
api: spanner
|
@hengfengli Would you mind giving me write access to this repository? I'm currently not able to assign issues to myself. FYI: I'm currently looking into this. |
|
@olavloite Added to team and provided write access. |
This was referenced May 28, 2021
olavloite
added a commit
that referenced
this issue
Jun 8, 2021
* test: add mock server and tests Adds a mock server and several end-to-end tests using the mock server. Also fixes: - Invalid function calls in migrations - Unnecessary use of snapshots for INFORMATION_SCHEMA queries - Unnecessary SELECT 1 query each time a connection is checked out of the pool * chore: fix linting issues * fix: wait until server is running before proceding * test: enable acceptance tests on emulator Enables running the acceptance integration tests on the emulator. These finish in ~20 seconds instead of after several hours. All tests are green, except for one that is skipped, as Cloud Spanner does not support changing a nullable column to a not-null column for a table that already exists. Updates #65 * test: enable acceptance tests on emulator Enables running the acceptance integration tests on the emulator. These finish in ~20 seconds instead of after several hours. All tests are green, except for one that is skipped, as Cloud Spanner does not support changing a nullable column to a not-null column for a table that already exists. Updates #65 * chore: fix linting issues * fix: reset primary key prefix after change definition * test: run acceptance tests on emulator * fix: set correct task name * build: run on all supported ruby versions * fix: check instance value + document join table
olavloite
added a commit
that referenced
this issue
Jun 14, 2021
* test: add mock server and tests Adds a mock server and several end-to-end tests using the mock server. Also fixes: - Invalid function calls in migrations - Unnecessary use of snapshots for INFORMATION_SCHEMA queries - Unnecessary SELECT 1 query each time a connection is checked out of the pool * chore: fix linting issues * fix: wait until server is running before proceding * test: enable acceptance tests on emulator Enables running the acceptance integration tests on the emulator. These finish in ~20 seconds instead of after several hours. All tests are green, except for one that is skipped, as Cloud Spanner does not support changing a nullable column to a not-null column for a table that already exists. Updates #65 * test: enable acceptance tests on emulator Enables running the acceptance integration tests on the emulator. These finish in ~20 seconds instead of after several hours. All tests are green, except for one that is skipped, as Cloud Spanner does not support changing a nullable column to a not-null column for a table that already exists. Updates #65 * chore: fix linting issues * fix: reset primary key prefix after change definition * test: run acceptance tests on emulator * fix: set correct task name * build: run on all supported ruby versions * feat: support DDL batches on connection Removes the global enabling/disabling of DDL batches and moves it to the connection. The user can then specify explicitly whether a set of DDL statements should be executed as a batch or not. This is less error prone and clearer to the user, and makes it easy to define migrations that are either executed as one batch, in separate batches or not using batches at all. Fixes #71 * feat: support DDL batches on connection Removes the global enabling/disabling of DDL batches and moves it to the connection. The user can then specify explicitly whether a set of DDL statements should be executed as a batch or not. This is less error prone and clearer to the user, and makes it easy to define migrations that are either executed as one batch, in separate batches or not using batches at all. Fixes #71 * fix: linting errors * fix: linting issues * fix: remove setting unnecessary env variable
olavloite
added a commit
that referenced
this issue
Jun 14, 2021
* test: add mock server and tests Adds a mock server and several end-to-end tests using the mock server. Also fixes: - Invalid function calls in migrations - Unnecessary use of snapshots for INFORMATION_SCHEMA queries - Unnecessary SELECT 1 query each time a connection is checked out of the pool * chore: fix linting issues * fix: wait until server is running before proceding * test: enable acceptance tests on emulator Enables running the acceptance integration tests on the emulator. These finish in ~20 seconds instead of after several hours. All tests are green, except for one that is skipped, as Cloud Spanner does not support changing a nullable column to a not-null column for a table that already exists. Updates #65 * test: enable acceptance tests on emulator Enables running the acceptance integration tests on the emulator. These finish in ~20 seconds instead of after several hours. All tests are green, except for one that is skipped, as Cloud Spanner does not support changing a nullable column to a not-null column for a table that already exists. Updates #65 * chore: fix linting issues * fix: reset primary key prefix after change definition * test: run acceptance tests on emulator * fix: set correct task name * build: run on all supported ruby versions * feat: support DDL batches on connection Removes the global enabling/disabling of DDL batches and moves it to the connection. The user can then specify explicitly whether a set of DDL statements should be executed as a batch or not. This is less error prone and clearer to the user, and makes it easy to define migrations that are either executed as one batch, in separate batches or not using batches at all. Fixes #71 * feat: support DDL batches on connection Removes the global enabling/disabling of DDL batches and moves it to the connection. The user can then specify explicitly whether a set of DDL statements should be executed as a batch or not. This is less error prone and clearer to the user, and makes it easy to define migrations that are either executed as one batch, in separate batches or not using batches at all. Fixes #71 * fix: linting errors * fix: linting issues * feat: add mapping and tests for NUMERIC type * feat: add mapping and tests for NUMERIC type
olavloite
added a commit
that referenced
this issue
Jun 16, 2021
* test: add mock server and tests Adds a mock server and several end-to-end tests using the mock server. Also fixes: - Invalid function calls in migrations - Unnecessary use of snapshots for INFORMATION_SCHEMA queries - Unnecessary SELECT 1 query each time a connection is checked out of the pool * chore: fix linting issues * fix: wait until server is running before proceding * test: enable acceptance tests on emulator Enables running the acceptance integration tests on the emulator. These finish in ~20 seconds instead of after several hours. All tests are green, except for one that is skipped, as Cloud Spanner does not support changing a nullable column to a not-null column for a table that already exists. Updates #65 * test: enable acceptance tests on emulator Enables running the acceptance integration tests on the emulator. These finish in ~20 seconds instead of after several hours. All tests are green, except for one that is skipped, as Cloud Spanner does not support changing a nullable column to a not-null column for a table that already exists. Updates #65 * chore: fix linting issues * fix: reset primary key prefix after change definition * test: run acceptance tests on emulator * fix: set correct task name * build: run on all supported ruby versions * feat: support DDL batches on connection Removes the global enabling/disabling of DDL batches and moves it to the connection. The user can then specify explicitly whether a set of DDL statements should be executed as a batch or not. This is less error prone and clearer to the user, and makes it easy to define migrations that are either executed as one batch, in separate batches or not using batches at all. Fixes #71 * feat: support DDL batches on connection Removes the global enabling/disabling of DDL batches and moves it to the connection. The user can then specify explicitly whether a set of DDL statements should be executed as a batch or not. This is less error prone and clearer to the user, and makes it easy to define migrations that are either executed as one batch, in separate batches or not using batches at all. Fixes #71 * fix: linting errors * fix: linting issues * feat: add mapping and tests for NUMERIC type * feat: add mapping and tests for NUMERIC type * feat: support prepared statements and query cache Support creating prepared statements with statement parameters instead of translating values to sql literals. This both makes the queries and DML statements more efficient and less prone to SQL injection attacks. It also enables the use of the built-in query cache of ActiveRecord. Prepared statement support is enabled by default, the query cache must be enabled manually by calling ActiveRecord::Base.enable_query_cache Fixes #64 and #57 * feat: support prepared statements and query cache Support creating prepared statements with statement parameters instead of translating values to sql literals. This both makes the queries and DML statements more efficient and less prone to SQL injection attacks. It also enables the use of the built-in query cache of ActiveRecord. Prepared statement support is enabled by default, the query cache must be enabled manually by calling ActiveRecord::Base.enable_query_cache Fixes #64 and #57 * chore: fix linting issues * fix: use png instead of svg
olavloite
added a commit
that referenced
this issue
Jun 16, 2021
* test: add mock server and tests Adds a mock server and several end-to-end tests using the mock server. Also fixes: - Invalid function calls in migrations - Unnecessary use of snapshots for INFORMATION_SCHEMA queries - Unnecessary SELECT 1 query each time a connection is checked out of the pool * chore: fix linting issues * fix: wait until server is running before proceding * test: enable acceptance tests on emulator Enables running the acceptance integration tests on the emulator. These finish in ~20 seconds instead of after several hours. All tests are green, except for one that is skipped, as Cloud Spanner does not support changing a nullable column to a not-null column for a table that already exists. Updates #65 * test: enable acceptance tests on emulator Enables running the acceptance integration tests on the emulator. These finish in ~20 seconds instead of after several hours. All tests are green, except for one that is skipped, as Cloud Spanner does not support changing a nullable column to a not-null column for a table that already exists. Updates #65 * chore: fix linting issues * fix: reset primary key prefix after change definition * test: run acceptance tests on emulator * fix: set correct task name * build: run on all supported ruby versions * feat: support DDL batches on connection Removes the global enabling/disabling of DDL batches and moves it to the connection. The user can then specify explicitly whether a set of DDL statements should be executed as a batch or not. This is less error prone and clearer to the user, and makes it easy to define migrations that are either executed as one batch, in separate batches or not using batches at all. Fixes #71 * feat: support DDL batches on connection Removes the global enabling/disabling of DDL batches and moves it to the connection. The user can then specify explicitly whether a set of DDL statements should be executed as a batch or not. This is less error prone and clearer to the user, and makes it easy to define migrations that are either executed as one batch, in separate batches or not using batches at all. Fixes #71 * fix: linting errors * fix: linting issues * feat: add mapping and tests for NUMERIC type * feat: add mapping and tests for NUMERIC type * feat: support prepared statements and query cache Support creating prepared statements with statement parameters instead of translating values to sql literals. This both makes the queries and DML statements more efficient and less prone to SQL injection attacks. It also enables the use of the built-in query cache of ActiveRecord. Prepared statement support is enabled by default, the query cache must be enabled manually by calling ActiveRecord::Base.enable_query_cache Fixes #64 and #57 * feat: support prepared statements and query cache Support creating prepared statements with statement parameters instead of translating values to sql literals. This both makes the queries and DML statements more efficient and less prone to SQL injection attacks. It also enables the use of the built-in query cache of ActiveRecord. Prepared statement support is enabled by default, the query cache must be enabled manually by calling ActiveRecord::Base.enable_query_cache Fixes #64 and #57 * chore: fix linting issues * feat: retry aborted transactions Transactions in ActiveRecord are always defined as blocks. This means that a transaction that is aborted can easily be retried by retrying the entire block, instead of having to rely on registering each result that is returned by an update or a query, and comparing the original results with the new results during a retry. This PR overrides the default transaction manager of ActiveRecord so transaction blocks are automatically retried if the transaction failed because it was aborted by Cloud Spanner. Towards #63 * feat: retry aborted transactions Transactions in ActiveRecord are always defined as blocks. This means that a transaction that is aborted can easily be retried by retrying the entire block, instead of having to rely on registering each result that is returned by an update or a query, and comparing the original results with the new results during a retry. This PR overrides the default transaction manager of ActiveRecord so transaction blocks are automatically retried if the transaction failed because it was aborted by Cloud Spanner. Towards #63 * fix: cleanup and fix linting errors * test: add additional tests * feat: use mutations for implicit transactions * feat: use mutations instead of DML when possible Use mutations instead of DML for inserts/updates/deletes during implicit transactions. Implicit transactions can never be used for read-your-writes, which means that mutations can be used instead of DML. Mutations are a lot more efficient than DML, especially for multiple small inserts/updates/deletes. Fixes #69 * feat: use mutations instead of DML when possible Use mutations instead of DML for inserts/updates/deletes during implicit transactions. Implicit transactions can never be used for read-your-writes, which means that mutations can be used instead of DML. Mutations are a lot more efficient than DML, especially for multiple small inserts/updates/deletes. Fixes #69 * test: add acceptance tests for transactions * feat: support read-only transactions * chore: fix rubocop issues * chore: fix rubocop issues * fix: use png instead of svg
olavloite
added a commit
that referenced
this issue
Jun 21, 2021
* test: add mock server and tests Adds a mock server and several end-to-end tests using the mock server. Also fixes: - Invalid function calls in migrations - Unnecessary use of snapshots for INFORMATION_SCHEMA queries - Unnecessary SELECT 1 query each time a connection is checked out of the pool * chore: fix linting issues * fix: wait until server is running before proceding * test: enable acceptance tests on emulator Enables running the acceptance integration tests on the emulator. These finish in ~20 seconds instead of after several hours. All tests are green, except for one that is skipped, as Cloud Spanner does not support changing a nullable column to a not-null column for a table that already exists. Updates #65 * test: enable acceptance tests on emulator Enables running the acceptance integration tests on the emulator. These finish in ~20 seconds instead of after several hours. All tests are green, except for one that is skipped, as Cloud Spanner does not support changing a nullable column to a not-null column for a table that already exists. Updates #65 * chore: fix linting issues * fix: reset primary key prefix after change definition * test: run acceptance tests on emulator * fix: set correct task name * build: run on all supported ruby versions * feat: support DDL batches on connection Removes the global enabling/disabling of DDL batches and moves it to the connection. The user can then specify explicitly whether a set of DDL statements should be executed as a batch or not. This is less error prone and clearer to the user, and makes it easy to define migrations that are either executed as one batch, in separate batches or not using batches at all. Fixes #71 * feat: support DDL batches on connection Removes the global enabling/disabling of DDL batches and moves it to the connection. The user can then specify explicitly whether a set of DDL statements should be executed as a batch or not. This is less error prone and clearer to the user, and makes it easy to define migrations that are either executed as one batch, in separate batches or not using batches at all. Fixes #71 * fix: linting errors * fix: linting issues * feat: add mapping and tests for NUMERIC type * feat: add mapping and tests for NUMERIC type * feat: support prepared statements and query cache Support creating prepared statements with statement parameters instead of translating values to sql literals. This both makes the queries and DML statements more efficient and less prone to SQL injection attacks. It also enables the use of the built-in query cache of ActiveRecord. Prepared statement support is enabled by default, the query cache must be enabled manually by calling ActiveRecord::Base.enable_query_cache Fixes #64 and #57 * feat: support prepared statements and query cache Support creating prepared statements with statement parameters instead of translating values to sql literals. This both makes the queries and DML statements more efficient and less prone to SQL injection attacks. It also enables the use of the built-in query cache of ActiveRecord. Prepared statement support is enabled by default, the query cache must be enabled manually by calling ActiveRecord::Base.enable_query_cache Fixes #64 and #57 * chore: fix linting issues * feat: retry aborted transactions Transactions in ActiveRecord are always defined as blocks. This means that a transaction that is aborted can easily be retried by retrying the entire block, instead of having to rely on registering each result that is returned by an update or a query, and comparing the original results with the new results during a retry. This PR overrides the default transaction manager of ActiveRecord so transaction blocks are automatically retried if the transaction failed because it was aborted by Cloud Spanner. Towards #63 * feat: retry aborted transactions Transactions in ActiveRecord are always defined as blocks. This means that a transaction that is aborted can easily be retried by retrying the entire block, instead of having to rely on registering each result that is returned by an update or a query, and comparing the original results with the new results during a retry. This PR overrides the default transaction manager of ActiveRecord so transaction blocks are automatically retried if the transaction failed because it was aborted by Cloud Spanner. Towards #63 * fix: cleanup and fix linting errors * test: add additional tests * feat: use mutations for implicit transactions * feat: use mutations instead of DML when possible Use mutations instead of DML for inserts/updates/deletes during implicit transactions. Implicit transactions can never be used for read-your-writes, which means that mutations can be used instead of DML. Mutations are a lot more efficient than DML, especially for multiple small inserts/updates/deletes. Fixes #69 * feat: use mutations instead of DML when possible Use mutations instead of DML for inserts/updates/deletes during implicit transactions. Implicit transactions can never be used for read-your-writes, which means that mutations can be used instead of DML. Mutations are a lot more efficient than DML, especially for multiple small inserts/updates/deletes. Fixes #69 * test: add acceptance tests for transactions * feat: support read-only transactions * chore: fix rubocop issues * chore: fix rubocop issues * feat: retry session not found errors * feat: retry session not found errors `Session not found` errors can be returned by Cloud Spanner if the session that is used by a connection has expired or has been garbage collected by Cloud Spanner. Normally, a session should be kept-alive by the Spanner adapter automatically, as it issues a simple SELECT 1 query if the session has not been used for more than 45 minutes, but there are scenarios possible where the session will expire or otherwise has been garbage collected by Spanner. This solution uses the following retry logic: 1. If a `Session not found` occurs for a single statement outside of a transaction, the session will be re-created and the single statement will be retried on the new session. 2. If a `Session not found` occurs during a transaction, the entire transaction will be retried. This is achieved by creating a new session and then raising an AbortedError on the transaction, which will trigger the Aborted retry loop for the transaction. Fixes #76 * feat: retry session not found errors `Session not found` errors can be returned by Cloud Spanner if the session that is used by a connection has expired or has been garbage collected by Cloud Spanner. Normally, a session should be kept-alive by the Spanner adapter automatically, as it issues a simple SELECT 1 query if the session has not been used for more than 45 minutes, but there are scenarios possible where the session will expire or otherwise has been garbage collected by Spanner. This solution uses the following retry logic: 1. If a `Session not found` occurs for a single statement outside of a transaction, the session will be re-created and the single statement will be retried on the new session. 2. If a `Session not found` occurs during a transaction, the entire transaction will be retried. This is achieved by creating a new session and then raising an AbortedError on the transaction, which will trigger the Aborted retry loop for the transaction. Fixes #76 * chore: fix linting issues * fix: address review comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
No description provided.
The text was updated successfully, but these errors were encountered: