chore(web-console): Add DEDUP support for Copy schema to clipboard

[insmac]

Closes #184

Adding dedup configuration to table schema

If DEDUP UPSERT KEYS(key1, key2) has been used when creating a table, it should also be included in the generated DDL statement in Copy schema to clipboard.

Steps to test:

1. Create a table that has deduplication enabled

CREATE TABLE TICKS (
    ts TIMESTAMP,
    tk SYMBOL,
    p DOUBLE
) TIMESTAMP(ts) PARTITION BY DAY WAL
DEDUP UPSERT KEYS(ts, tk);

INSERT INTO TICKS VALUES ('2023-07-14', 'QQQ', 78.56);
INSERT INTO TICKS VALUES ('2023-07-14', 'AAA', 78.56);
INSERT INTO TICKS VALUES ('2023-07-14', 'QQQ', 78.56); -- this is a dupe, won't be inserted
SELECT * FROM TICKS; -- should show 2 records

2. Right-click on the table, select Copy schema to clipboard. The copied statement should include the DEDUP... statement from the first CREATE query above.

Adding dedup option to Create Table UI

This PR also adds the new "dedup" button in each column row, so that user might mark these as deduplication keys.

Disabled - incompatible column type or WAL not enabled:

Can be enabled - all good:

[javier]

I've found a weird issue. If you create a table with dedup, then disable, it might still generate a DEDUP with empty keys (invalid). But not always?

Check this out

1. create table

CREATE TABLE 'ilp_test2' (
  device_type SYMBOL capacity 256 CACHE,
  duration_ms LONG,
  lat DOUBLE,
  lon DOUBLE,
  measure1 LONG,
  timestamp TIMESTAMP,
  measure2 LONG,
  speed LONG
) timestamp (timestamp) PARTITION BY DAY WAL DEDUP UPSERT KEYS(timestamp, device_type);

2. disable DEDUP

alter table ilp_test2 dedup disable;

3. copy schema to clipboard and maybe see it fail
4. drop table and try again?

[javier]

When using the web console to create table, I can create a table using DEDUP columns even if the designated timestamp is not part of the dedup. Designated timestamp must be selected if dedup is enabled. Not sure what the right action would be here:

1. Leave it as is. User will get an error and will need to amend (maybe not too bad)
2. If designated timestamp hasn't been selected, silently add it anyway to the UPSERT KEYS on generation
3. Validate and don't allow users to submit unless designated ts has been selected as KEY if there is any other KEY already selected

[javier]

When using the web console, it properly says I cannot add a non valid type column as UPSERT key. The hover message even says that WAL must be enabled. But even when WAL is not enables, the interface still allows me to add UPSERT keys for symbols or ints and will happily generate the invalid SQL.

[insmac]

I've found a weird issue. If you create a table with dedup, then disable, it might still generate a DEDUP with empty keys (invalid). But not always?

Check this out

1. create table

CREATE TABLE 'ilp_test2' (
  device_type SYMBOL capacity 256 CACHE,
  duration_ms LONG,
  lat DOUBLE,
  lon DOUBLE,
  measure1 LONG,
  timestamp TIMESTAMP,
  measure2 LONG,
  speed LONG
) timestamp (timestamp) PARTITION BY DAY WAL DEDUP UPSERT KEYS(timestamp, device_type);

2. disable DEDUP

alter table ilp_test2 dedup disable;

3. copy schema to clipboard and maybe see it fail
4. drop table and try again?

@javier I've been able to replicate the above issue and provide a fix. Could you pull the branch and check again? Thanks! 🙂

[insmac]

When using the web console, it properly says I cannot add a non valid type column as UPSERT key. The hover message even says that WAL must be enabled. But even when WAL is not enables, the interface still allows me to add UPSERT keys for symbols or ints and will happily generate the invalid SQL.

@javier I have fixed the above issue, and added a check that even if user selects a dedup key first and disables WAL afterwards, the system will detect it and won't try to set UPSERT KEYS.

[insmac]

When using the web console to create table, I can create a table using DEDUP columns even if the designated timestamp is not part of the dedup. Designated timestamp must be selected if dedup is enabled. Not sure what the right action would be here:

Leave it as is. User will get an error and will need to amend (maybe not too bad)
If designated timestamp hasn't been selected, silently add it anyway to the UPSERT KEYS on generation
Validate and don't allow users to submit unless designated ts has been selected as KEY if there is any other KEY already selected

@javier I have decided to add the designated timestamp if user forgot to do it, instead of throwing an error.
This is because if there are other dedup keys present, I feel like he might just miss that it's required, and we can still use his settings (other keys) to proceed.

[javier]

I've found a weird issue. If you create a table with dedup, then disable, it might still generate a DEDUP with empty keys (invalid). But not always?
Check this out

1. create table

CREATE TABLE 'ilp_test2' (
  device_type SYMBOL capacity 256 CACHE,
  duration_ms LONG,
  lat DOUBLE,
  lon DOUBLE,
  measure1 LONG,
  timestamp TIMESTAMP,
  measure2 LONG,
  speed LONG
) timestamp (timestamp) PARTITION BY DAY WAL DEDUP UPSERT KEYS(timestamp, device_type);

2. disable DEDUP

alter table ilp_test2 dedup disable;

3. copy schema to clipboard and maybe see it fail
4. drop table and try again?

@javier I've been able to replicate the above issue and provide a fix. Could you pull the branch and check again? Thanks! 🙂

This works now. But I have the contrary effect. If I disable upsert on a table and then enable again, when I get the schema it comes with no DEDUP. If I alter a second time, then it is actually updated when I get the schema. Same if I change the schema on a different tab or via REST API. The get table schema gets the former version rather than the updated one.

[javier]

When using the web console, it properly says I cannot add a non valid type column as UPSERT key. The hover message even says that WAL must be enabled. But even when WAL is not enables, the interface still allows me to add UPSERT keys for symbols or ints and will happily generate the invalid SQL.

@javier I have fixed the above issue, and added a check that even if user selects a dedup key first and disables WAL afterwards, the system will detect it and won't try to set UPSERT KEYS.

tried it and seems to work fine :)

[javier]

Found a new one:

First try setting a table name and a table with both a designated timestamp and symbol (for example) and add both to the upsert keys

Now change the type from ts to SYMBOL

Invalid CREATE TABLE statement is generated

CREATE TABLE 'xxxx' (ts SYMBOL, sss SYMBOL) timestamp (ts) PARTITION BY DAY WAL DEDUP UPSERT KEYS(ts, sss);

This even works when I choose String or Binary as types. The UI will remove the UPSERT key check as it is not suported for those types, but still on create no errors and this is generated

CREATE TABLE 'xxxxxxxxx' (ts BINARY, sss SYMBOL) timestamp (ts) PARTITION BY HOUR WAL DEDUP UPSERT KEYS(ts, sss);

[javier]

When using the web console to create table, I can create a table using DEDUP columns even if the designated timestamp is not part of the dedup. Designated timestamp must be selected if dedup is enabled. Not sure what the right action would be here:

Leave it as is. User will get an error and will need to amend (maybe not too bad)
If designated timestamp hasn't been selected, silently add it anyway to the UPSERT KEYS on generation
Validate and don't allow users to submit unless designated ts has been selected as KEY if there is any other KEY already selected

@javier I have decided to add the designated timestamp if user forgot to do it, instead of throwing an error. This is because if there are other dedup keys present, I feel like he might just miss that it's required, and we can still use his settings (other keys) to proceed.

This works fine now

[insmac]

@javier Fixed the above issues:

• When the designated timestamp is set and the column type is switched to something other than TIMESTAMP, it will clear it out in the background.
• UPSERT KEYS statement should now be refreshed every time ALTER TABLE is executed.

Additionally:

• DDL Schema is no longer added to the editor, because it was indeed confusing. Now, it is executed upon submitting the form and an appropriate notification (success/error) is dispatched and appears in the Notifications bar.