Vitess 8.0.0

This release complies with VEP-3 which removes the upgrade order requirement. Components can be upgraded in any order. It is recommended that the upgrade order should still be followed if possible, except to canary test the new version of VTGate before upgrading the rest of the components.

Incompatible Changes

The following PRs made changes to behavior that clients might be relying on. They should be reviewed carefully so that client code can be changed in concert with a Vitess release deployment.

• Change error code from 1203 to 1153 for three specific error conditions. #6630
  The three errors being changed:
  When a GRPC message coming back to the vtgate exceeds the configured size limit (configured via grpc_max_message_size)
  If a write is attempting to make a change with a payload that is larger than the configured allowable size
  If a scatter query is attempting to collect and process too many rows in memory within the vtgate to construct a response
• Zero auto-increment mode: This enables inserting a 0 value (and not just NULL) into an auto-increment column to generate sequence values. This matches the default behavior of MySQL. #6749
• Turn off schema tracker by default #6712

Known Issue(s)

• VReplication errors when a fixed-length binary column is used as the sharding key #8080

Bugs Fixed

VTGate / MySQL compatibility

• Fix where clause in information schema with correct database name #6599
• Fix DDL execution on reserved connection without in active transaction #6514
• Fix allow passed table_schema in information_schema queries to vttablet if the keyspace is not found #6653
• Fix information schema for non-existent schema #6667
• Fix reserved connection in autocommit mode on DML #6748
• Fix dbname that needs escaping that we broke in fce4cfd #6794
• MySQL CLI, mysql --ssl does not work in MySQL 8.0 #5556
• OperationalError: (_mysql_exceptions.OperationalError) (2013, 'Lost connection to MySQL server during query') #6208
• Fix support for show tables where tables_in_db to use keyspace name, not underlying db name #6446
• Common empty result set query on MySQL returns a row in Vitess #6663
• Operator precedence issue #6734
• Table names out-of-order between replica and primary #6738
• Close Idle reserved connections and Rollback Idle transactions #6552
• Minor Vindex fixes: #6526
• VTgate does not report autocommit system variable and SERVER_STATUS_IN_TRANS flag correctly #5825
• MariaDB JDBC Connector fails to connect to vtgate with default options #5851
• VTgate panics if there are no vttablets and vtctlds #6117
• Remove shard session held in vtgate for reserved connection on connection failure #6522
• Session variable does not work as expected in 7.0 #6559
• Retry should not happen when in dedicated connection #6523
• Correctly report AUTOCOMMIT status in network packets #6551
• Close Idle reserved connections and Rollback Idle transactions #6552
• Reset Session for Reserved Connection Query Failure #6673
• Errors in multi-statement queries not fatal to statement sequence #6747 #6808
• Lock wait timeout on DELETE from table using lookup index #6698
• Application couldn't connect to external db giving error tabletserver.go:1522] Code: INVALID_ARGUMENT syntax error at position 33 near ':vtg2' (CallerID: gnocchi) : Sql: "select convert(:vtg1, CHAR(:vtg2)) as anon_1 from dual", #6466
• The client got the wrong database name #6753
• DBAPIError exception wrapped from (pymysql.err.InternalError) (1105, u'vtgate: http://gnocchi-gnocchizone-vtgate:15000/: reserved connections are not supported on old gen gateway') #6577
• Provide support for “SELECT INTO OUTFILE S3” #6811
• Fix error around breaking of multi statements #6824
• Support SHOW TABLE STATUS FROM db type queries in vtgate #6354
• SHOW FULL FIELDS doesn't follow routing rules #6803
• VDiff/Tablet Picker: fix picker retry logic to handle canceled context #6954
• Healthcheck: Use isIncluded correctly to fix replica/rdonly routing bug #6922

Other

• VReplication tablet_picker should keep trying to find a tablet until context expires #6546
• VReplication improve reverse workflow: update cells/tablet_types #6556
• VDiff: fix panic for tables with a unicode_loose_md5 vindex #6640
• Vtctld UI: Add mysqld port display back to vtctld web UI #6598 #6597
• Vtctld: Fix workflow CopyState fetching (and thus printing). #6657
• Fix unsaved event regression #6710
• Change tracker to use allprivs instead of dba #6720
• Fixes long wait filter keyspace #6721
• s3 ListBackups doesn’t work from root directory #6732
• Vtctld: ExecuteFetchAsDba and ExecuteFetchAsAllPrivs do not escape db names. #6545
• Can't switch on restore_from_backup on existing shard with no tables #4896
• Refactor conn.go #6818

Functionality Added or Changed

VTGate / MySQL compatibility

• Add support for DELETE IGNORE #6722
• Only take on simple dual queries in the vtgate #6666
• Support mycli access to vtgate #4365
• Add support for unicode_loose_xxhash #6457
• Provide virtual information_schema #5394
• information_schema.tables_schema comparison with dynamic value no longer works #6827
• Mysqldump: Allow setting multiple variables in one statement #5401
• Block lock function usage other than dual table #6470
• Lock Session Support #6517
• Make sure lookup vindexes are queryable inside transaction #6499
• Add unicode_loose_xxhash Vindex type #6549
• Support vindex update or delete when destination is provided by client in dml query #6554
• Make sure to handle EXPLAIN on tablets #6579
• Rewrite SHOW TABLES #6615
• Make offset work in OLAP mode #6655
• Support sql_calc_found_rows with limit in sharded keyspace #6680
• Lock shard session heartbeat #6683
• Mysql -ssl tag is deprecated in the later versions of mysql #6682
• Use statement support in olap mode #6692
• Delete row using consistent lookup vindex in where clause #6700
• Add vtexplain support for non-evenly sharded keyspaces #6705
• Make sure dual queries set found_rows correctly #6718
• Add if not exists support to database ddl in parsing #6724
• Allow Update of same vindex using same vindex lookup on consistent lookup vindex #6736
• Update Vindex only on changes #6737
• Add support for SHOW (DATABASES|VITESS_SHARDS|VITESS_TABLETS) LIKE #6750
• 'CASCADE' support #6743
• Mask database name #6791
• Make sure to backtick schema names properly #6550
• Transaction Limiter: fix quota leak if Begin() errors or if the tx killer runs #6731
• Allow Load Data From S3 #6823

OLAP Functioanlity

• Allow switching between OLAP and OLTP #6691
• Use statement support in olap mode #6692
• Fix error handling in olap mode #6940

Set Statement Support

Set statement support is added in Vitess. There are some system variables which are disabled by default and can be enabled using flag -enable_system_settings on VTGate.These system variables are set on the backing MySQL instance, and will force the connection to be dedicated instead of part of the connection pool.

• Disabled passthrough system variables by default. #6859
• Allow switching workload between OLAP and OLTP #4086 #6691
• Support for multiple session set statement syntax #6508 #6495 #6494
• Allow enumeration in set system variables #6493
• Handle boolean settings better #6501
• Set statements refactored to use plan building #6487
• Add more system settings #6486
• Evaluate system variables #6708

VReplication

• VReplication: _vt.vreplication source column VARBINARY->BLOB, allowing more tables to participate in a workflow #6421
• VReplication and vstreamer performance and error metrics #6519, #6132
• VReplication: Find collations for char primary keys and cast comparison during catchup #6568
• Varchar primary keys with collations can cause vreplication to miss rows #6622
• VReplication: Timestamp w/ Heartbeat #6635
• VReplication: Improve few error/log messages #6669
• VReplication: handle SET statements in the binlog stream #6772
• VReplication: e2e Tests added to CI #6730
• VDiff logs: additional logging around vdiff/vstreamer for observability #6684
• VStream: experimental POC of mysql protocol #6670
• Materialize: Support rollups using a literal value as a column #6733
• Materialization: Ignore non-participating tables that are renamed/deleted #6778
• Materialize: add ability for maintaining accurate table-level aggregates #6767
• Schema Tracker: Turn off schema tracker by default #6712 (breaking change)
• Workflow: Add List-All Command #6533
• Workflow: List -> Show, and Expand Metadata #6544
• Refactor: Cleanup Noisy Workflow Logging #6744
• Schema tracker: Change tracker to use the allprivs user instead of dba #6720
• vtctl: Adding VExec and Workflow commands #6410
• vtctl Workflow cmd: don't expect all shards to have workflow streams #6576
• Reshard: Sort table definitions before comparing them #6765 #6738
• Expose a shard range calculator as a vtctl command GenerateShardRanges #6751 #6752
• VStream: Field event now has all column attributes #6525
• VStream events do not populate field "Flags" on FIELD events #5914
• RFC: VReplication based SplitClone #4604
• Experimental: Mysql Protocol support for VStream #6675
• VReplication: Add metrics around vstreamer and vreplication metrics #6787
• Fix issues with missing tables in vstreams #6789
• Add validations and logs to fix common issues faced during MoveTables/Reshard #6814
• Allow multiple blacklists for master #6816

VTtablet

• VTtablet throttling #6668
  https://vitess.io/docs/reference/features/tablet-throttler/
• TabletManager: publish displayState #6648
• TabletManager: call setReadOnly inside ChangeTabletType #6762
• TabletManager: change how SetReadOnly is called to avoid errors from externally managed tablets #6786
• TabletManager: ca...

Vitess 7.0.3 Patch Update

This update fixes several regressions that were deemed significant enough to be backported to the release branch.

Bugs Fixed

• Vtgate : Should receive healthcheck updates from all tablets in cells_to_watch #6857
• Vtgate : Fix error around breaking of multistatements #6824
• Vtgate : Fix bug in REPLICA/RDONLY routing #6904
• Vttablet : checkNoDB should not require tables to be present #6788

Known Issues

This release includes MySQL 5.7.32 and Percona Xtrabackup 2.4.20 which have a known incompatibility. Percona is working on a resolution.

The issue will manifest if a user using 5.7.32:
* Creates a new compressed table.
* Runs any ALTER TABLE  that uses the algorithm copy (table rebuild) on a compressed table.

Build Changes

• Build : Download zookeeper 3.4.14 from archive site #6868

Vitess 8.0.0-rc1

This release complies with VEP-3 which removes the upgrade order requirement. Components can be upgraded in any order. It is recommended that the upgrade order should still be followed if possible, except to canary test the new version of VTGate before upgrading the rest of the components.

Incompatible Changes

The following PRs made changes to behavior that clients might be relying on. They should be reviewed carefully so that client code can be changed in concert with a Vitess release deployment.

• Change error code from 1203 to 1153 for three specific error conditions. #6630
  The three errors being changed:
  When a GRPC message coming back to the vtgate exceeds the configured size limit (configured via grpc_max_message_size)
  If a write is attempting to make a change with a payload that is larger than the configured allowable size
  If a scatter query is attempting to collect and process too many rows in memory within the vtgate to construct a response
• Zero auto-increment mode: This enables inserting a 0 value (and not just NULL) into an auto-increment column to generate sequence values. This matches the default behavior of MySQL. #6749
• Turn off schema tracker by default #6712

Bugs Fixed

VTGate / MySQL compatibility

• Fix where clause in information schema with correct database name #6599
• Fix DDL execution on reserved connection without in active transaction #6514
• Fix allow passed table_schema in information_schema queries to vttablet if the keyspace is not found #6653
• Fix information schema for non-existent schema #6667
• Fix reserved connection in autocommit mode on DML #6748
• Fix dbname that needs escaping that we broke in fce4cfd #6794
• MySQL CLI, mysql --ssl does not work in MySQL 8.0 #5556
• OperationalError: (_mysql_exceptions.OperationalError) (2013, 'Lost connection to MySQL server during query') #6208
• Fix support for show tables where tables_in_db to use keyspace name, not underlying db name #6446
• Common empty result set query on MySQL returns a row in Vitess #6663
• Operator precedence issue #6734
• Table names out-of-order between replica and primary #6738
• Close Idle reserved connections and Rollback Idle transactions #6552
• Minor Vindex fixes: #6526
• VTgate does not report autocommit system variable and SERVER_STATUS_IN_TRANS flag correctly #5825
• MariaDB JDBC Connector fails to connect to vtgate with default options #5851
• VTgate panics if there are no vttablets and vtctlds #6117
• Remove shard session held in vtgate for reserved connection on connection failure #6522
• Session variable does not work as expected in 7.0 #6559
• Retry should not happen when in dedicated connection #6523
• Correctly report AUTOCOMMIT status in network packets #6551
• Close Idle reserved connections and Rollback Idle transactions #6552
• Reset Session for Reserved Connection Query Failure #6673
• Errors in multi-statement queries not fatal to statement sequence #6747 #6808
• Lock wait timeout on DELETE from table using lookup index #6698
• Application couldn't connect to external db giving error tabletserver.go:1522] Code: INVALID_ARGUMENT syntax error at position 33 near ':vtg2' (CallerID: gnocchi) : Sql: "select convert(:vtg1, CHAR(:vtg2)) as anon_1 from dual", #6466
• The client got the wrong database name #6753
• DBAPIError exception wrapped from (pymysql.err.InternalError) (1105, u'vtgate: http://gnocchi-gnocchizone-vtgate:15000/: reserved connections are not supported on old gen gateway') #6577
• Provide support for “SELECT INTO OUTFILE S3” #6811

Other

• VReplication tablet_picker should keep trying to find a tablet until context expires #6546
• VReplication improve reverse workflow: update cells/tablet_types #6556
• VDiff: fix panic for tables with a unicode_loose_md5 vindex #6640
• Vtctld UI: Add mysqld port display back to vtctld web UI #6598 #6597
• Vtctld: Fix workflow CopyState fetching (and thus printing). #6657
• Fix unsaved event regression #6710
• Change tracker to use allprivs instead of dba #6720
• Fixes long wait filter keyspace #6721
• s3 ListBackups doesn’t work from root directory #6732
• Vtctld: ExecuteFetchAsDba and ExecuteFetchAsAllPrivs do not escape db names. #6545
• Can't switch on restore_from_backup on existing shard with no tables #4896
• Refactor conn.go #6818

Functionality Added or Changed

VTGate / MySQL compatibility

• New functionality to set workload = 'olap'; can not repeat, but set workload = 'oltp' can; #4086

• Add support for DELETE IGNORE #6722

• Only take on simple dual queries in the vtgate #6666

• Support mycli access to vtgate #4365

• Add support for unicode_loose_xxhash #6457

• Provide virtual information_schema #5394

• Mysqldump: Allow setting multiple variables in one statement #5401

• SET planning #6487

• Allow enumeration in set system variables #6493

• Allow set to have scope for each setting #6494

• HandleSet in executor to handle multiple settings in single set statement #6495

• Passthrough system settings flag: enabled by default #6586

• Handle boolean settings better #6501

• Add more system settings #6486

• Block lock function usage other than dual table #6470

• Lock Session Support #6517

• Make sure lookup vindexes are queryable inside transaction #6499

• Add unicode_loose_xxhash Vindex type #6549

• Support vindex update or delete when destination is provided by client in dml query #6554

• Make sure to handle EXPLAIN on tablets #6579

• Rewrite SHOW TABLES #6615

• Make offset work in OLAP mode #6655

• Support sql_calc_found_rows with limit in sharded keyspace #6680

• Lock shard session heartbeat #6683

• Mysql -ssl tag is deprecated in the later versions of mysql #6682

• Allow switching between OLAP and OLTP #6691

• Use statement support in olap mode #6692

• Delete row using consistent lookup vindex in where clause #6700

• Add vtexplain support for non-evenly sharded keyspaces #6705

• Evaluate system variables #6708

• Make sure dual queries set found_rows correctly #6718

• Add if not exists support to database ddl in parsing #6724

• Allow Update of same vindex using same vindex lookup on consistent lookup vindex #6736

• Update Vindex only on changes #6737

• Add support for SHOW (DATABASES|VITESS_SHARDS|VITESS_TABLETS) LIKE #6750

• Support multiple session set statement syntax #6508

• 'CASCADE' support #6743

• Mask database name #6791

• Make sure to backtick schema names properly #6550

• Transaction Limiter: fix quota leak if Begin() errors or if the tx killer runs #6731

OLAP Functioanlity

• Allow switching between OLAP and OLTP #6691
• Use statement support in olap mode #6692

System Variables

• Allow enumeration in set system variables #6493
• Allow set to have scope for each setting #6494
• HandleSet in executor to handle multiple settings in single set statement #6495

VReplication

• VReplication: _vt.vreplication source column VARBINARY->BLOB, allowing more tables to participate in a workflow #6421
• VReplication and vstreamer performance and error metrics #6519, #6132
• VReplication: Find collations for char primary keys and cast comparison during catchup #6568
• Varchar primary keys with collations can cause vreplication to miss rows #6622
• VReplication: Timestamp w/ Heartbeat #6635
• VReplication: Improve few error/log messages #6669
• VReplication: handle SET statements in the binlog stream #6772
• VReplication: e2e Tests added to CI #6730
• VDiff logs: additional logging around vdiff/vstreamer for observability #6684
• VStream: experimental POC of mysql protocol #6670
• Materialize: Support rollups using a literal value as a column #6733
• Materialization: Ignore non-participating tables that are renamed/deleted #6778
• Materialize: add ability for maintaining accurate table-level aggregates #6767
• Schema Tracker: Turn off schema tracker by default #6712 (breaking change)
• Workflow: Add List-All Command #6533
• Workflow: List -> Show, and Expand Metadata #6544
• Refactor: Cleanup Noisy Workflow Logging #6744
• Schema tracker: Change tracker to use the allprivs user instead of dba #6720
• vtctl: Adding VExec and Workflow commands #6410
• vtctl Workflow cmd: don't expect all shards to have workflow streams #6576
• Reshard: Sort table definitions before comparing them #6765 #6738
• Expose a shard range calculator as a vtctl command GenerateShardRanges #6751 #6752
• VStream: Field event now has all column attributes #6525
• VStream events do not populate field "Flags" on FIELD events #5914
• RFC: VReplication based SplitClone #4604
• Experimental: Mysql Protocol support for VStream #6675
• VReplication: Add metrics around vstreamer and vreplication metrics #6787
• Fix issues with missing tables in vstreams #6789
• Add validations and logs to fix common issues faced during MoveTables/Reshard #6814

VTtablet

• VTtablet throttling #6668
• TabletManager: publish displayState #6648
• TabletManager: call setReadOnly inside ChangeTabletType #6762
• TabletManager: change how SetReadOnly is called to avoid errors from externally managed tablets #6786
• TabletManager: call SetReadOnly inside ChangeTabletType #6804
• vttablet: Open and Close healthStreamer #6515
• vttablet: create database if not present #6490
• Restore: do not change tablet type to RESTORE if not actually performing a restore #6679
• Restore: checkNoDB should not require tables to be present #6695
• Heartbeat: Additional metrics/stats (lag histogram) #6528 #6634
• VTtablet throttler feature flag: -enable-lag-throttler #6815

VTorc - Orchestrator

• Orchestrator: initial import of Orchestrator #6582
• Orchestrator: vitess mode #6613
• Orchestrator: more changes #6714
• Orchestrator: use contexts with timeout for remote operations #6780
• Orchestrator: fixed orchestrator govet errors #6781
• Orchestrator: Add clusters_to_watch flag to. Defaults t...

Vitess 7.0.2 Patch Update

This update fixes several regressions that were deemed significant enough to be backported to the release branch.

Bugs Fixed

• vtgate : Operator precedence must take associativity into consideration #6764
• vtgate: Fix reserved connection in autocommit mode on DML #6748
• VDiff: fix panic for tables with a unicode_loose_md5 vindex #6745
• vttablet: Turn off schema tracker by default #6746
• restore: do not change tablet type to RESTORE if not actually performing a restore #6687
• vttablet : Tablet manager init should publish displayState #6686
• vttablet: Tablet manager add more logging to checkMastership #6685
• vttablet : Fix replmanager deadlock #6634

Vitess 7.0.1 patch update

This update fixes several regressions that were deemed significant enough to be backported to the release branch.
In addition, the experimental feature to set system variables is now behind a flag that disables it by default.

Configuration Changes

• Vtgate: added flag enable_system_settings to enable passthrough system settings, defaults to false #6562

Bugs Fixed

• Vreplication: Materialize should only get schema from source tablets if target is missing tables #6623
• Vtctld UI: mysql port is not displayed in tablet list #6597 #6600
• Vtgate: transaction killer was closing reserved connection after the specified time. Only idle connections should be closed. Idle transactions should be rolled back. #6583
• Vtcompose: docker-compose example was broken by tabletmanager changes #6587
• Vreplication: tablet_picker should respect canceled context. Also, keep trying to find a tablet until context expires. #6567
• Vtgate: vindex update/delete by destination #6541 #6572
• Vtgate: some usages of EXPLAIN were broken #6574 #6581
• Vtgate: handle multiple settings in single SET statement #6504
• Vtgate: cleanup shard session for reserved connection on failure #6563
• Vtgate: if executing DDL on reserved connection, start transaction if needed #6555
• Vtgate: make sure lookup vindexes use same transaction if one is open #6505
• Healthcheck: add diagnostic logging, do not add tablet if it is shutting down #6512

Vitess 6.0.20-20200818 Patch Update

This update fixes two regressions that were deemed significant enough to be backported to the release branch.

Bugs Fixed

• Vtgate: delete by destination broken for tables with owned vindexes #6541 #6554
• Vtgate: set statements with multiple settings should not error out #6566

Vitess 7 General Availability

Incompatible Changes

This release includes the following changes which may result in incompatibilities when upgrading from a previous release. It is important that Vitess components are upgraded in the recommended order. This will change in the next release as documented in VEP-3.

• VTGate: The default gateway (-gateway_implementation flag value) is now "tabletgateway". The old default was “discoverygateway”. To use the old gateway, specify the flag in the command line. Anyone already using the flag will see no change; to use the new gateway, remove the -gateway_implementation flag. The new gateway is simpler and uses a new healthcheck module with no loss of functionality. #6155

• VTGate: VTGate incorrectly returned Warning Code field name as Type. This has now been fixed. Code or scripts that depend on the incorrect behavior might break. #6437

• VReplication: The "-cell" parameter to MoveTables has been renamed to “-cells”, since it now accepts multiple comma-separated cells or cellsAliases. The parameter specifies which cells can provide tablets as the source for vreplication streams. #6456

• VTGate: The possibility of sending OpenTracing trace spans through query comments has been changed. The old way of doing it was severely broken, so our assumption is that no-one has successfully been using this feature. Still, this is an API change, and we are reporting it as a breaking change. #6463

• VTTablet has been refactored to be generally more responsive to state changes and its environment: #6131 #6139 #6241 #6263 #6311 #6345 #6348 #6384 #6396 #6432 #6461

  • Some initialization parameters are now mandatory: init_keyspace, init_shard and init_tablet_type. The flow where you could issue a vtctlclient InitTablet and then invoke vttablet without those init parameters is no longer supported.

  • Consequently vtctl's InitTablet and UpdateTabletAddrs are deprecated.

  • demote_master_type is deprecated: Now that init_tablet_type is mandatory, that value can act as the demote_master_type. There is no need for a separate flag.

  • VTTablet will publish the discovered MySQL port on startup, and will not update it during the life of the process. There is no known use case where a MySQL port is expected to change. So, there is no need to keep polling MySQL for its port.

  • Decouple enable_replication_watcher from disable_active_reparents. Currently, vttablet attempts to fix replication only if enable_replication_watcher is true and disable_active_reparents is false. This behavior has changed to depend only on the disable_active_reparents flag.

  • Transitioning from a serving type to SPARE used to go through a brief lameduck period specified by the serving_state_grace_period (default 0). This has been removed. The transition will now happen immediately even if a grace period is specified.

  • It was possible to enable health reporting and heartbeat at the same time. Performing the two at the same time is not useful. As of this release, enabling heartbeat automatically disables the polling mode of checking for replica health.

  • The ACL rules used to be reported on the vttablet status page. The design was using an incorrect framework. This has been removed.

• Build: make targets "build_web" and “embed_static” have been removed. Instead there are new targets “web_bootstrap”, “web_build” and “web_start” #6473

Deprecations

• The following vtctl commands have been deprecated. Deprecated commands will continue to work until the next release (8.0). #6428 #6345 #6461

  • StartSlave -> Replaced with StartReplication

  • StopSlave -> Replaced with StopReplication

  • ChangeSlaveType -> Replaced with ChangeTabletType

  • InitTablet

  • UpdateTabletAddrs

• Various vtctl commands had a flag called "-wait_slave_timeout". It has been replaced with “-wait_replicas_timeout”. Scripts using the old flag will continue to work until the next major release (8.0). #6428

• Helm charts: The current helm charts have not been keeping up with the forward progress in vitess, and there has been no maintainer for it. Additionally, the operator has pulled ahead of what the helm charts could do. The operator can also be functionally more versatile than the helm charts because custom code can be written to manage complex workflows. To maintain focus and velocity in the project, we are deprecating the helm charts. #6439

• Consul: The usage of Consul has been problematic for vitess users. The main difficulty comes from the fact that it's non-trivial for a multi-cell deployment. Due to this, we want to discourage the use of consul as a topo server, and are deprecating support for it. #6440

Bugs Fixed

VTGate / MySQL compatibility

• Handle non-keyword tokens FORMAT, TREE and VITESS. (This was broken by #6110) #6445

• When using user-defined variables that have not been initialized first, a non-fatal panic occurs and the connection is dropped. #6294 #6308

• Setting system variable panics vtgate. #6162 #6163

• regression: USE does not work from MySQL client. #6153 #6157

• USE does not return an error if a keyspace doesn't exist. #3060 #6157

• Dotted table names are not escaped properly in lookup vindex flow. #6142 #6145

• Produce correct error code (1105 instead of 1062) when inserting duplicates into consistent_lookup vindex. #6144

• Rewriting expressions fail when used as function parameters. #6281

• Rollback after error clobbers updates. #6285

• Queries should not be wrapped into transactions if autocommit=1 is set. #6264

• Prepared statements with bind variables in column specification produce ‘missing bind var’ error. #6287 #6298

• Prepared statement queries returning NULL results are not parsed. #6240 #6249

• DEFAULT function not working with status column. #6221 #6224

• VTGate panic kills connection when selecting database() when using OLAP workload. #6200 #6205

• Wrong column value can be inserted due to parameterization variable confusion. #6266 #6270

• Fix owned table name formatting and duplicate error code in consistent lookup. #6145

• Support for case-insensitive user defined variables. #6225

• Fix performance regression on VTGate. #6203

• Return NULL for non-existent user-defined variables. #6308

• MySQL connection closing should not inflate query counts. #6226

• show vitess_shards intermittently returns empty list. #5038 #5189

• Protect insert lookup queries from bind variable name clashes #6328

• CAST/CONVERT should work correctly #6304

• Allow charset introducers for vindex columns #6301

• Fix panic from EXPLAIN #6204

• Handle special comments that start with ! #6186

• Fix panic with uppercase types #6140

• Allow ‘$’ in identifiers #6136

Other

• VTTablet: Etcd query timeout because auto-committable transactions are not properly rolled back. #6258 #6299

• VTTablet: Fixed race in schema engine during notification of schema changes. #6268

• VTTablet: Reloading ACL config was throwing misleading error messages.his has now been fixed and the error messages improved. #6309

• VTTablet: Panic during backup caused an error in the pgzip library has been fixed by contributing a fix to pgzip and upgrading to a new release that includes the fix. #6189

• VTTablet: Improvement of error handling by retrying while uploading to S3. #6182 #6395 #6405

• VTTablet: builtinbackupengine was uploading a manifest despite errors from S3. Error collection during backup has been fixed to address this. #6349 #6350

• VTTablet: Fix dba connection leak in LockTables. #6424

• VTTablet: Fix connection pool leak in schema engine. #6426

• VTTablet: Fix regression where a heartbeat interval flag setting of 1s would have resulted in an internal value of 1000s #6356 #6357

• VTTablet: Fix panic from binlog_streamer when ‘-watch_replication_stream’ is set., #6253

• VTTablet: Add context to MysqlDaemon schema functions. This lets us handle disconnected RPCs instead of the client hanging forever. #6243

• VTTablet: fix panic from txlogz #6238

• VTTablet: Add back support for millisecond values for flags that was dropped when yaml config was introduced #6250

• Vtctld UI: Change all external links in vitess to use https #6170 #6169

• Ensure SrvVSchema gets rebuilt for all cells #6276

• Fix MariadbGTIDSet multi-domain support #6184

• mysqlctl: Fix connection leak during killConnection() #6245

• Fix release script to install examples in the correct directory #6159

• examples/operator: fix port-forward command #6418

• VStream client: should send current position events immediately #6391 #6397

Functionality Added or Changed

VTGate/ MySQL compatibility

• Transactions on read-only replicas are now supported. Start a transaction by first choosing the desired tablet_type ("use keyspace@replica"), then issuing a “BEGIN” statement. Internally, this will issue a “START TRANSACTION READ ONLY” on the underlying database. This is only available if using the new gateway_implementation of tabletgateway (which is now the default). #6166 #6244

• Create better plans for NULL comparisons. Instead of using a SelectScatter, now vtgate will produce empty results when the predicate is a comparison against literal NULLs. #6133 #6425

• Support IN, NOT IN and equality (=) with null values #6152 #6146

• Add EXPLAIN support for vtgate plans. View the produced vtgate query plan by using explain format=vitess #6110

• Allow users to perform savepoint related constructs. #6355 #6404 #6412 #6414

• Added support for user-level advisory lock functions like get_lock and release_lock. This is considered experimental and will be improved in future releases based on user feedback...

Vitess 7.0.0-beta

Incompatible Changes

• VTGate: The default gateway ("-gateway_implementation" flag value) is now "tabletgateway". The old default was "discoverygateway”. To use the old gateway, specify the flag in the command line. Anyone already using the flag will see no change; to use the new gateway, remove the -gateway_implementation flag. The new gateway is simpler and uses a new healthcheck module with no loss of functionality. #6155

• VTGate: VTGate incorrectly returned Warning Code field name as Type. This has now been fixed. Code or scripts that depend on the incorrect behavior might break. #6437

Deprecations

• The following vtctl commands have been deprecated. Deprecated commands will continue to work until the next release (8.0). #6428

• StartSlave -> Replaced with StartReplication

• StopSlave -> Replaced with StopReplication

• ChangeSlaveType -> Replaced with ChangeTabletType

• Various vtctl commands had a flag called "-wait_slave_timeout". It has been replaced with “-wait_replicas_timeout”. Scripts using the old flag will continue to work until the next major release (8.0). #6428

• Helm chart (more details to be added)

• Consul (more details to be added)

• Replaced VTTablet ExecuteBatch api call with Execute for autocommit query. #6407 #6431

Bugs Fixed

VTGate / MySQL compatibility

• VTGate: When using user-defined variables that have not been initialized first, a non-fatal panic occurs and the connection is dropped. #6294 #6308

• VTGate: Setting system variable panics vtgate. #6162 #6163

• regression: USE does not work from mysql client. #6153 #6157

• USE does not return an error if a keyspace doesn't exist. #3060 #6157

• Dotted table names are not escaped properly in lookup vindex flow. #6142 #6145

• Error 1105 instead of 1062 when inserting duplicate into consistent_lookup vindex. #6144

• VTGate: Rewriting expressions fail when used as function parameters. #6281

• Rollback after error clobbers updates. #6285

• Queries should not be wrapped into transactions if autocommit=1 is set. #6264

• Prepared statements with bind variables in column specification produce ‘missing bind var’ error. #6287 #6298

• Prepared statement queries returning NULL results are not parsed. #6240 #6249

• DEFAULT function not working with status column. #6221 #6224

• VTGate panic kills connection when selecting database() when using OLAP workload. #6200 #6205

• Wrong column value can be inserted due to parameterization variable confusion. #6266 #6270

• Fix owned table name formatting and duplicate error code in consistent lookup. #6145

• Support for case-insensitive user defined variable. #6225

• Fix performance regression on VTGate. #6203

• Return NULL for non-existent user-defined variables. #6308

• Lock function support. #6370

• VTGate: MySQL connection closing should not inflate query counts. #6226

• VTGate: Show vitess_shards intermittently returns empty list. This has been fixed #5038 #5189

Other

• VTTablet: Etcd query timeout because auto-committable transactions are not properly rolled back. #6258 #6299

• VTTablet: Fixed race in schema engine during notification of schema changes. #6268

• VTTablet: Reloading ACL config was throwing misleading error messages.his has now been fixed and the error messages improved. #6309

• VTTablet: Panic during backup because of an error in the pgzip library has been fixed by contributing a fix to pgzip and upgrading to a new release that includes the fix. #6189

• VTTablet: Improvement of error handling by retrying while uploading to S3. #6182 #6395 #6405

• VTTablet: builtinbackupengine was uploading a manifest despite errors from S3. Error collection during backup has been fixed to address this. #6349 #6350

• VTTablet: Fix dba connection leak in LockTables. #6424

• VTTablet: Fix connection pool leak in schema engine. #6426

• VTTablet: Fix panic from binlog_streamer when ‘-watch_replication_stream’ is set., #6253

• Vtctld UI: Change all external links in vitess to use https #6170 #6169

• Ensure SrvVSchema gets rebuilt for all cells #6276

• Fix MariadbGTIDSet multi-domain support #6184

• mysqlctl: Fix connection leak during killConnection() #6245

• Fix release script to install examples in the correct directory #6159

Functionality Added or Changed

VTGate/ MySQL compatibility

• Transactions on read-only replicas are now supported. Start a transaction by first choosing the desired tablet_type ("use @replica"), then issuing a “BEGIN” statement. Internally, this will issue a “START TRANSACTION READ ONLY” on the underlying database. This is only available if using tabletgateway. #6166 #6244

• Create better plans for NULL comparisons. Instead of using a SelectScatter, now vtgate will produce empty results when the predicate is a comparison against literal NULLs. #6133 #6425

• Add EXPLAIN support for vtgate plans. View the produced vtgate query plan by using EXPLAIN FORMAT=VITESS.

• Added savepoint support. #6355 #6404 #6412 #6414

• "ignore_nulls" option to lookup vindexes #6147 #6222

• Add support for row_count() function. #6174 #6179

• XOR Operator is now supported in sql WHERE clause. #6369 #6371

• Allow 'binary' for character_set_results. It is equivalent to null. #6237

• Improve logging by adding keyspace, table, and tablet_type. #6282

• VTGate now takes in optional flags for ‘max_payload_size’ and ‘warn_payload_size’. A payload size above the warning threshold will only produce a warning. A payload size that exceeds ‘max_payload_size’ will result in an error. The default behavior is not to check payload size, which is backwards-compatible. #6143 #6375

• A new comment directive (‘/*vt+ IGNORE_MAX_PAYLOAD_SIZE=1 */) has been added to provide the ability to override the payload size checking per query. #6143

• Support ‘select * from information_schema.processlist’. #6310

• VTGate: Support batch lookup queries for integer and binary types. Other types will continue to require one query per row. #6420

• Support system settings at session level. #6149

• Prepare for future features by adding reserved connections. This is the ability for a connected client to have a dedicated connection instead of using the connection pool. #6303 #6313

• Allow charset introducers with vindex column values. #6236 #6314

• Support Union All between different shards. #6351

• VTGate: Exclude KeyspaceId(s) from plan cache key.This improves the cache hit rate. #6233

• VTGate: Reload TLS config on SIGHUP. #6215

• VTGate: Drop vindex DDL. #6151

• VTGate: Show vitess_tablets now reports master_term_start_time. We ensure that this is reported correctly after reparents. #6135 #6292 #6293

VReplication

• VStream Copy extends the VStream API to stream entire databases or tables. Earlier you specified a position to start streaming events from. #6277

• VReplication workflows now accept "-cells" as an option. This can be a comma-separated list of cells or cellsAliases. The source tablet for vreplication will be chosen randomly among the tablets that belong to the specified cells/cellsAliases. If the option is not provided, we default to the cell that is local to the vreplication target. #6442, #6456

• Schema Versioning. Tablets now have a schema tracker that can be enabled to store schema versions as DDLs are applied. The _vt.schema_version table contains the schema with the corresponding ddl and gtid position when the ddl was applied. The tracked schemas are used by the new Historian service to provide vstreams with the correct schema while streaming old events. #6300

• Foreign Keys are now supported on the target database during vreplication. In the copy phase we disable FK checks and re-enable them once the replication phase starts. #6284

• Materialize: added a drop_constraint option during table copy to strip constraints while creating target tables #6422

• Materialize: optimizations while creating target tables resulting in a significant improvement in performance when moving a large number of tables from an external MySQL instance #6207, #6251

• DropSources: Added ‘-rename_tables’ option to rename the source tables rather than dropping them entirely. #6383

• Add keyspace, shard and workflow labels to VReplicationSecondsBehindMaster metrics #6279

• Correct command line help #6187

Point-in-time Recovery

In the 4.0 release we introduced functionality to recover a keyspace into a snapshot based on a specified time. That feature only allowed you to restore a backup, but did not have the ability to replay binlogs up to the specified time. The feature has now been completed so that true point-in-time recovery is possible. This is currently only supported for MySQL compatible installations, not MariaDB. #4857 #6267 #6408

Healthcheck

• VTGate can now use a new healthcheck that is logically simpler while providing the same functionality. This is the default. The older healthcheck has been deprecated but is still available to use if desired by setting "-gateway_implementation=discoverygateway" #5750 #6155

• VTCombo has been migrated to use the new healthcheck #6302

MiniVitess

A docker container that can be used to bring up a self-contained vitess cluster, which can automatically resolve and self-configure to meet an external MySQL cluster. While this might sound very similar to VTCombo and vttestserver, it is not quite the same. MiniVitess runs in docker, brings up distinct processes for each vitess component and works with an external MySQL cluster. #6377

Examples / Tutorials

• Vitess operator example #6154

• Region sharding example #6209 #6275

• Change local example to use new command DropSources #6178

• Are-you-alive example now makes it easier to test multiple endpoints #6202

• Local docker tutorial added...

Vitess 6.0.20-20200624 Patch Update

This update fixes a regression which was significant enough that it was decided to backport to the release branch.

Bug fixed

#6250 added support for float64 durations in vttablet's yaml configuration. The change introduced a regression where the heartbeat interval unit was changed to seconds, but the conversion was not correspondingly changed. In other words, a heartbeat interval of 1s would have resulted in a value of 1000s instead. The net effect is that healthy replicas would be reported as heavily lagged.

This issue affects only those who enable heartbeats.

This regression as merged into https://github.com/vitessio/vitess/releases/tag/v6.0.20-20200617.
#6359 fixes #6356

Vitess 6.0.20-20200617 Patch Update

This update fixes a number of regressions which were significant enough that it was decided to backport to the release branch. The issue with XtraBackup not being compatible with MySQL 8.0.20 has been fixed by Percona. Anyone using MySQL 8.0.20 or higher will need to use Xtrabackup 8.0.12.

Bugs Fixed

• Vtgate: Fix panic from select database() in OLAP mode #6318 Fixes #6200
• Vtgate: Fix DEFAULT function not working with keyword column names #6319 Fixes #6221
• Vttablet: Fix regression where some options could no longer be specified as floating-point values #6320
• Vtgate: Fix regression on autocommit transactions that led to extra begin statements being issued, and updates getting rolled back because of a previous error #6321 Fixes #6264 and #6285
• Security: new versions of Netty and tcnative #6322
• Vtgate: Fix panic when selecting @variable #6323 Fixes #6294
• Vtgate: Fix problem with incorrect values being inserted due to bindvariable name collisions #6324 #6327 Fixes #6266