Failed to create index - PAXOS Failed to update metadata source=create-index

[dioptre]

Please help! Thanks!

Elassandra version:
6.2.3.20 (apt version)

Plugins installed: []
None
JVM version (java -version):
1.8

OS version (uname -a if on a Unix-like system):

Linux cassandra1-prod 4.19.0-6-amd64 #1 SMP Debian 4.19.67-2+deb10u2 (2019-11-11) x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:

Index doesn't get created, and log complains of paxos and metadata issue? (It does work on my local dev machine but not in the test cluster)

Steps to reproduce:

curl -w "\n" -k -H 'Content-Type: application/json'  -XPUT  "http://$CASSANDRA_HOST:9200/events_recent/" -d '{
    "settings" : { "keyspace" : "sfpla" },
    "mappings": {
        "events_recent": {
            "properties" : {
                "eid": { "type": "keyword", "index": true, "cql_collection": "singleton" },
                "vid": { "type": "keyword", "index": true, "cql_collection": "singleton" },
                 "latlon" : { "type": "geo_point", "index": true, "cql_collection": "singleton" },
                "params" :  { 
                  "type": "nested", 
                  "include_in_parent": true,
                  "cql_struct" : "opaque_map",
                  "cql_collection" : "singleton"
                }
            }
        }
    }
}'

Elassandra Logs
(Initiate Machine)
2020-08-28 01:27:46,099 WARN [elasticsearch[10.0.0.2][masterService#updateTask][T#1]] ClusterService.java:1310 commitMetaData PAXOS Failed to update metadata source=create-index [events_recent], cause [api] prevMetadata=6bec5c6a-5173-4625-a0fa-75c37b83b76a/1 nextMetaData=6bec5c6a-5173-4625-a0fa-75c37b83b76a/2
2020-08-28 01:27:46,100 WARN [elasticsearch[10.0.0.2][masterService#updateTask][T#1]] CassandraDiscovery.java:1079 publishAsCoordinator PAXOS concurrent update, source=create-index [events_recent], cause [api] metadata=6bec5c6a-5173-4625-a0fa-75c37b83b76a/2, resubmit task on next metadata change

Cluster state

{
  "cluster_name" : "SFPL",
  "compressed_size_in_bytes" : 609,
  "version" : 15,
  "state_uuid" : "PcNG_k5cQpCie3fqPoLfJA",
  "master_node" : "6bec5c6a-5173-4625-a0fa-75c37b83b76a",
  "blocks" : { },
  "nodes" : {
    "13d84671-ca17-4ad0-868d-35fa5beb616f" : {
      "name" : "10.0.0.4",
      "status" : "ALIVE",
      "ephemeral_id" : "13d84671-ca17-4ad0-868d-35fa5beb616f",
      "transport_address" : "10.0.0.4:9300",
      "attributes" : {
        "rack" : "r1",
        "dc" : "DC1"
      }
    },
    "6bec5c6a-5173-4625-a0fa-75c37b83b76a" : {
      "name" : "10.0.0.2",
      "status" : "ALIVE",
      "ephemeral_id" : "6bec5c6a-5173-4625-a0fa-75c37b83b76a",
      "transport_address" : "10.0.0.2:9300",
      "attributes" : {
        "rack" : "r1",
        "dc" : "DC1"
      }
    },
    "6b4425b3-aea7-4b7d-af64-03f245c73493" : {
      "name" : "10.0.0.3",
      "status" : "ALIVE",
      "ephemeral_id" : "6b4425b3-aea7-4b7d-af64-03f245c73493",
      "transport_address" : "10.0.0.3:9300",
      "attributes" : {
        "rack" : "r1",
        "dc" : "DC1"
      }
    }
  },
  "metadata" : {
    "version" : 1,
    "cluster_uuid" : "6bec5c6a-5173-4625-a0fa-75c37b83b76a",
    "templates" : { },
    "indices" : {
      "mthreads" : {
        "state" : "open",
        "settings" : {
          "index" : {
            "keyspace" : "msgxc",
            "number_of_shards" : "3",
            "provided_name" : "mthreads",
            "creation_date" : "1585368229595",
            "number_of_replicas" : "0",
            "uuid" : "h7tXpdg8QaaoQoSZY67RJg",
            "version" : {
              "created" : "6020399"
            }
          }
        },
        "mappings" : {
          "mthreads" : {
            "properties" : {
              "subs" : {
                "type" : "keyword"
              },
              "mid" : {
                "type" : "keyword",
                "cql_collection" : "singleton"
              },
              "tid" : {
                "type" : "keyword",
                "cql_collection" : "singleton"
              }
            }
          }
        },
        "aliases" : [ ],
        "primary_terms" : {
          "0" : 0,
          "1" : 0,
          "2" : 0
        },
        "in_sync_allocations" : {
          "2" : [ ],
          "1" : [ ],
          "0" : [ ]
        }
      }
    },
    "index-graveyard" : {
      "tombstones" : [ ]
    }
  },
  "routing_table" : {
    "indices" : {
      "mthreads" : {
        "shards" : {
          "0" : [
            {
              "state" : "STARTED",
              "primary" : true,
              "node" : "6bec5c6a-5173-4625-a0fa-75c37b83b76a",
              "relocating_node" : null,
              "shard" : 0,
              "index" : "mthreads",
              "token_ranges" : [
                "(-9223372036854775808,9223372036854775807]"
              ],
              "allocation_id" : {
                "id" : "dummy_alloc_id"
              }
            }
          ]
        }
      }
    }
  },
  "routing_nodes" : {
    "unassigned" : [ ],
    "nodes" : {
      "6b4425b3-aea7-4b7d-af64-03f245c73493" : [ ],
      "13d84671-ca17-4ad0-868d-35fa5beb616f" : [ ],
      "6bec5c6a-5173-4625-a0fa-75c37b83b76a" : [
        {
          "state" : "STARTED",
          "primary" : true,
          "node" : "6bec5c6a-5173-4625-a0fa-75c37b83b76a",
          "relocating_node" : null,
          "shard" : 0,
          "index" : "mthreads",
          "token_ranges" : [
            "(-9223372036854775808,9223372036854775807]"
          ],
          "allocation_id" : {
            "id" : "dummy_alloc_id"
          }
        }
      ]
    }
  },
  "snapshots" : {
    "snapshots" : [ ]
  },
  "restore" : {
    "snapshots" : [ ]
  },
  "snapshot_deletions" : {
    "snapshot_deletions" : [ ]
  }
}

• cassandra schema (cqlsh>DESC KEYSPACE <your_keyspace>)

CREATE KEYSPACE sfpla WITH replication = {'class': 'NetworkTopologyStrategy', 'DC1': '2'}  AND durable_writes = true;

CREATE TYPE sfpla.geo_point (
    lat double,
    lon double
);

CREATE TYPE sfpla.geo_pol (
    country text,
    region text,
    county text,
    city text,
    zip text
);

CREATE TYPE sfpla.payment (
    invid timeuuid,
    invoiced timestamp,
    product text,
    pid timeuuid,
    pcat text,
    man text,
    model text,
    qty double,
    duration text,
    starts date,
    ends date,
    price double,
    discount double,
    revenue double,
    margin double,
    cost double,
    tax double,
    tax_rate double,
    commission double,
    referral double,
    fees double,
    subtotal double,
    total double,
    payment double,
    paid timestamp
);

CREATE TYPE sfpla.viewport (
    w bigint,
    h bigint
);

CREATE TABLE sfpla.redirects (
    urlfrom text PRIMARY KEY,
    hhash text,
    updated timestamp,
    updater timeuuid,
    urlto text
) WITH bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

CREATE TABLE sfpla.logs (
    id timeuuid PRIMARY KEY,
    created timestamp,
    host text,
    hostname text,
    ip inet,
    iphash text,
    ldate date,
    level int,
    ltime time,
    msg text,
    name text,
    owner timeuuid,
    params map<text, text>,
    topic text
) WITH bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

CREATE TABLE sfpla.events_recent (
    eid timeuuid PRIMARY KEY,
    app text,
    auth timeuuid,
    bhash text,
    campaign text,
    cflags bigint,
    city text,
    country text,
    created timestamp,
    duration bigint,
    ename text,
    etyp text,
    hhash text,
    ip inet,
    iphash text,
    last text,
    latlon frozen<geo_point>,
    medium text,
    nparams map<text, double>,
    params map<text, text>,
    payment frozen<payment>,
    ptyp text,
    region text,
    rel text,
    relation text,
    rid timeuuid,
    score double,
    sid timeuuid,
    sink text,
    source text,
    split text,
    targets map<text, frozen<set<text>>>,
    term text,
    uid timeuuid,
    url text,
    ver int,
    vid timeuuid,
    xid text,
    zip text
) WITH bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 2678400
    AND gc_grace_seconds = 14400
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

CREATE TABLE sfpla.dailies (
    ip inet,
    day date,
    total counter,
    PRIMARY KEY (ip, day)
) WITH CLUSTERING ORDER BY (day DESC)
    AND bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

CREATE TABLE sfpla.agreed (
    vid timeuuid,
    created timestamp,
    aff text,
    app text,
    avid timeuuid,
    bhash text,
    browser text,
    campaign text,
    cflags bigint,
    compliances map<text, frozen<set<text>>>,
    country text,
    culture text,
    device text,
    fbid text,
    gaid text,
    hhash text,
    idfa text,
    ip inet,
    iphash text,
    latlon frozen<geo_point>,
    loc frozen<geo_pol>,
    medium text,
    msid text,
    org timeuuid,
    os text,
    owner timeuuid,
    rcode text,
    ref timeuuid,
    region text,
    rel text,
    sid timeuuid,
    source text,
    term text,
    tz text,
    uid timeuuid,
    url text,
    vp frozen<viewport>,
    zip text,
    PRIMARY KEY (vid, created)
) WITH CLUSTERING ORDER BY (created DESC)
    AND bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

CREATE TABLE sfpla.locations (
    hhash text,
    vid timeuuid,
    latlon frozen<geo_point>,
    sid timeuuid,
    uid timeuuid,
    PRIMARY KEY ((hhash, vid, latlon))
) WITH bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

CREATE TABLE sfpla.actions (
    sid timeuuid,
    did timeuuid,
    completed timestamp,
    created timestamp,
    dsrc text,
    exqid timeuuid,
    meta map<text, text>,
    src text,
    started timestamp,
    PRIMARY KEY (sid, did)
) WITH CLUSTERING ORDER BY (did ASC)
    AND bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

CREATE TABLE sfpla.outcomes (
    hhash text,
    outcome text,
    sink text,
    url text,
    created date,
    total counter,
    PRIMARY KEY ((hhash, outcome), sink, url, created)
) WITH CLUSTERING ORDER BY (sink ASC, url ASC, created ASC)
    AND bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

CREATE TABLE sfpla.ips (
    hhash text,
    ip inet,
    total counter,
    PRIMARY KEY ((hhash, ip))
) WITH bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

CREATE TABLE sfpla.browsers (
    hhash text,
    bhash text,
    browser text,
    total counter,
    PRIMARY KEY ((hhash, bhash), browser)
) WITH CLUSTERING ORDER BY (browser ASC)
    AND bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

CREATE TABLE sfpla.accounts (
    uid timeuuid PRIMARY KEY,
    created timestamp,
    creds map<text, frozen<map<text, text>>>,
    expires timestamp,
    ip inet,
    msg text,
    owner timeuuid,
    pwd text
) WITH bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

CREATE TABLE sfpla.redirect_history (
    hostfrom text,
    updated timestamp,
    hostto text,
    pathto text,
    searchto text,
    slugfrom text,
    updater timeuuid,
    urlfrom text,
    urlto text,
    PRIMARY KEY (hostfrom, updated)
) WITH CLUSTERING ORDER BY (updated DESC)
    AND bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';
CREATE INDEX hostto_redirect_history_idx ON sfpla.redirect_history (hostto);

CREATE TABLE sfpla.sequences (
    name text PRIMARY KEY,
    seq int
) WITH bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

CREATE TABLE sfpla.referrers (
    hhash text,
    url text,
    total counter,
    PRIMARY KEY ((hhash, url))
) WITH bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

CREATE TABLE sfpla.aliases (
    hhash text,
    vid timeuuid,
    uid timeuuid,
    sid timeuuid,
    PRIMARY KEY ((hhash, vid), uid)
) WITH CLUSTERING ORDER BY (uid ASC)
    AND bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

CREATE TABLE sfpla.usernames (
    hhash text,
    uhash text,
    vid timeuuid,
    sid timeuuid,
    PRIMARY KEY ((hhash, uhash), vid)
) WITH CLUSTERING ORDER BY (vid ASC)
    AND bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

CREATE TABLE sfpla.jurisdictions (
    regulation text,
    compliance text,
    seq int,
    created timestamp,
    locs frozen<set<frozen<geo_pol>>>,
    rules map<text, text>,
    PRIMARY KEY (regulation, compliance, seq)
) WITH CLUSTERING ORDER BY (compliance ASC, seq DESC)
    AND bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';
CREATE INDEX locs_jurisdictions_idx ON sfpla.jurisdictions (full(locs));

CREATE TABLE sfpla.action_names (
    name text PRIMARY KEY
) WITH bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

CREATE TABLE sfpla.agreements (
    vid timeuuid PRIMARY KEY,
    aff text,
    app text,
    avid timeuuid,
    bhash text,
    browser text,
    campaign text,
    cflags bigint,
    compliances map<text, frozen<set<text>>>,
    country text,
    created timestamp,
    culture text,
    device text,
    fbid text,
    gaid text,
    hhash text,
    idfa text,
    ip inet,
    iphash text,
    latlon frozen<geo_point>,
    loc frozen<geo_pol>,
    medium text,
    msid text,
    org timeuuid,
    os text,
    owner timeuuid,
    rcode text,
    ref timeuuid,
    region text,
    rel text,
    sid timeuuid,
    source text,
    term text,
    tz text,
    uid timeuuid,
    url text,
    vp frozen<viewport>,
    zip text
) WITH bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

CREATE TABLE sfpla.actions_ext (
    sid text,
    svc text,
    created timestamp,
    iid timeuuid,
    meta map<text, text>,
    uid timeuuid,
    updated timestamp,
    PRIMARY KEY (sid, svc)
) WITH CLUSTERING ORDER BY (svc ASC)
    AND bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 1209600
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

CREATE TABLE sfpla.test (
    test int PRIMARY KEY
) WITH bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

CREATE TABLE sfpla.nodes (
    hhash text,
    vid timeuuid,
    iphash text,
    ip inet,
    sid timeuuid,
    uid timeuuid,
    PRIMARY KEY ((hhash, vid), iphash)
) WITH CLUSTERING ORDER BY (iphash ASC)
    AND bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

CREATE TABLE sfpla.events (
    eid timeuuid PRIMARY KEY,
    app text,
    auth timeuuid,
    bhash text,
    campaign text,
    cflags bigint,
    city text,
    country text,
    created timestamp,
    duration bigint,
    ename text,
    etyp text,
    hhash text,
    ip inet,
    iphash text,
    last text,
    latlon frozen<geo_point>,
    medium text,
    nparams map<text, double>,
    params map<text, text>,
    payment frozen<payment>,
    ptyp text,
    region text,
    rel text,
    relation text,
    rid timeuuid,
    score double,
    sid timeuuid,
    sink text,
    source text,
    split text,
    targets map<text, frozen<set<text>>>,
    term text,
    uid timeuuid,
    url text,
    ver int,
    vid timeuuid,
    xid text,
    zip text
) WITH bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

CREATE TABLE sfpla.counters (
    id text PRIMARY KEY,
    total counter
) WITH bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

CREATE TABLE sfpla.tlvv (
    hhash text,
    vid timeuuid,
    orid timeuuid,
    created timestamp,
    org timeuuid,
    owner timeuuid,
    paid double,
    payments set<frozen<payment>>,
    sid timeuuid,
    uid timeuuid,
    updated timestamp,
    updater timeuuid,
    PRIMARY KEY ((hhash, vid), orid)
) WITH CLUSTERING ORDER BY (orid ASC)
    AND bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

CREATE TABLE sfpla.cohorts (
    name text PRIMARY KEY,
    completed timestamp,
    created timestamp,
    imported int,
    owner timeuuid,
    started timestamp,
    uids_url text
) WITH bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

CREATE TABLE sfpla.tlvu (
    hhash text,
    uid timeuuid,
    orid timeuuid,
    created timestamp,
    org timeuuid,
    owner timeuuid,
    paid double,
    payments set<frozen<payment>>,
    sid timeuuid,
    updated timestamp,
    updater timeuuid,
    vid timeuuid,
    PRIMARY KEY ((hhash, uid), orid)
) WITH CLUSTERING ORDER BY (orid ASC)
    AND bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

CREATE TABLE sfpla.users (
    hhash text,
    uid timeuuid,
    vid timeuuid,
    sid timeuuid,
    PRIMARY KEY ((hhash, uid), vid)
) WITH CLUSTERING ORDER BY (vid ASC)
    AND bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

CREATE TABLE sfpla.sessions (
    vid timeuuid,
    sid timeuuid,
    aff text,
    app text,
    auth timeuuid,
    bhash text,
    browser text,
    campaign text,
    cflags bigint,
    city text,
    country text,
    created timestamp,
    culture text,
    device text,
    did text,
    duration bigint,
    ename text,
    etyp text,
    fbid text,
    gaid text,
    hhash text,
    idfa text,
    ip inet,
    iphash text,
    last text,
    latlon frozen<geo_point>,
    medium text,
    msid text,
    nparams map<text, double>,
    os text,
    params map<text, text>,
    ptyp text,
    rcode text,
    ref timeuuid,
    region text,
    rel text,
    score double,
    sink text,
    source text,
    split text,
    term text,
    tz text,
    uid timeuuid,
    url text,
    ver int,
    vp frozen<viewport>,
    xid text,
    zip text,
    PRIMARY KEY (vid, sid)
) WITH CLUSTERING ORDER BY (sid DESC)
    AND bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

CREATE TABLE sfpla.zips (
    country text,
    zip text,
    asian double,
    black double,
    carpool double,
    childpoverty double,
    city text,
    construction double,
    county text,
    created timestamp,
    culture text,
    drive double,
    employed int,
    familywork double,
    hispanic double,
    income double,
    incomeerr double,
    incomepercap double,
    incomepercaperr double,
    latlon frozen<geo_point>,
    loc frozen<geo_pol>,
    meancommute double,
    men int,
    native double,
    office double,
    othertransport double,
    pacific double,
    population int,
    poverty double,
    privatework double,
    production double,
    professional double,
    publicwork double,
    rcode text,
    region text,
    selfemployed double,
    service double,
    transit double,
    unemployment double,
    voters int,
    walk double,
    white double,
    women int,
    workathome double,
    PRIMARY KEY ((country, zip))
) WITH bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 60
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

CREATE TABLE sfpla.affiliates (
    hhash text,
    vid timeuuid,
    aff text,
    PRIMARY KEY ((hhash, vid))
) WITH bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

CREATE TABLE sfpla.visitors (
    vid timeuuid PRIMARY KEY,
    aff text,
    app text,
    auth timeuuid,
    bhash text,
    browser text,
    campaign text,
    cflags bigint,
    city text,
    country text,
    created timestamp,
    culture text,
    device text,
    did text,
    ename text,
    etyp text,
    fbid text,
    gaid text,
    hhash text,
    idfa text,
    ip inet,
    iphash text,
    last text,
    latlon frozen<geo_point>,
    medium text,
    msid text,
    nparams map<text, double>,
    os text,
    params map<text, text>,
    ptyp text,
    rcode text,
    ref timeuuid,
    region text,
    rel text,
    score double,
    sid timeuuid,
    sink text,
    source text,
    split text,
    term text,
    tz text,
    uid timeuuid,
    url text,
    ver int,
    vp frozen<viewport>,
    xid text,
    zip text
) WITH bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

CREATE TABLE sfpla.updates (
    id text PRIMARY KEY,
    msg text,
    updated timestamp
) WITH bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

CREATE TABLE sfpla.services (
    name text PRIMARY KEY,
    created timestamp,
    expiry date,
    owner timeuuid,
    roles set<text>,
    secret text,
    updated timestamp,
    updater timeuuid
) WITH bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

CREATE TABLE sfpla.reqs (
    hhash text,
    vid timeuuid,
    total counter,
    PRIMARY KEY ((hhash, vid))
) WITH bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

CREATE TABLE sfpla.emails (
    hhash text,
    ehash text,
    sid timeuuid,
    vid timeuuid,
    PRIMARY KEY ((hhash, ehash))
) WITH bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

CREATE TABLE sfpla.hits (
    hhash text,
    url text,
    total counter,
    PRIMARY KEY ((hhash, url))
) WITH bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

CREATE TABLE sfpla.referrals (
    hhash text,
    vid timeuuid,
    gen int,
    ref timeuuid,
    PRIMARY KEY ((hhash, vid))
) WITH bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

CREATE TABLE sfpla.tlv (
    hhash text,
    uid timeuuid,
    created timestamp,
    org timeuuid,
    owner timeuuid,
    paid double,
    payments set<frozen<payment>>,
    sid timeuuid,
    updated timestamp,
    updater timeuuid,
    vid timeuuid,
    PRIMARY KEY ((hhash, uid))
) WITH bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

CREATE TABLE sfpla.countries (
    country text PRIMARY KEY,
    continent text,
    name text
) WITH bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

CREATE TABLE sfpla.cells (
    hhash text,
    chash text,
    sid timeuuid,
    vid timeuuid,
    PRIMARY KEY ((hhash, chash))
) WITH bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

CREATE TABLE sfpla.messages (
    id timeuuid PRIMARY KEY,
    aff text,
    app text,
    auth text,
    campaign text,
    chashes set<text>,
    cohorts set<text>,
    completed timestamp,
    created timestamp,
    ehashes set<text>,
    medium text,
    owner timeuuid,
    promo text,
    ptyp text,
    ref timeuuid,
    rel text,
    repl map<text, text>,
    schedule timestamp,
    score double,
    sink text,
    source text,
    split double,
    splitn text,
    started timestamp,
    subject text,
    template text,
    term text,
    updated timestamp,
    updater timeuuid,
    ver int,
    xid text
) WITH bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

CREATE TABLE sfpla.routed (
    hhash text,
    ip inet,
    url text,
    PRIMARY KEY ((hhash, ip))
) WITH bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

CREATE TABLE sfpla.hosts (
    hhash text,
    hostname text,
    PRIMARY KEY (hhash, hostname)
) WITH CLUSTERING ORDER BY (hostname ASC)
    AND bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

CREATE TABLE sfpla.geo_ip (
    ipc text,
    ips varint,
    ipe varint,
    city text,
    country text,
    ipie inet,
    ipis inet,
    lat double,
    lon double,
    region text,
    tz text,
    zip text,
    PRIMARY KEY (ipc, ips, ipe)
) WITH CLUSTERING ORDER BY (ips ASC, ipe ASC)
    AND bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

CREATE TABLE sfpla.queues (
    id timeuuid PRIMARY KEY,
    completed timestamp,
    created timestamp,
    host text,
    ip text,
    owner timeuuid,
    schedule timestamp,
    sid timeuuid,
    skey text,
    src text,
    started timestamp,
    updated timestamp,
    updater timeuuid
) WITH bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';
CREATE INDEX queues_started_idx ON sfpla.queues (started);
CREATE INDEX queues_type_idx ON sfpla.queues (src);
CREATE INDEX queues_completed_idx ON sfpla.queues (completed);

CREATE TABLE sfpla.referred (
    hhash text,
    vid timeuuid,
    gen int,
    rcode text,
    PRIMARY KEY ((hhash, vid))
) WITH bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

• Metadata

cqlsh> select * from elastic_admin.Metadata_log ;

 cluster_name | v | version | owner                                | source                                    | ts
--------------+---+---------+--------------------------------------+-------------------------------------------+---------------------------------
         SFPL | 2 |       2 | 6bec5c6a-5173-4625-a0fa-75c37b83b76a | create-index [events_recent], cause [api] | 2020-08-27 22:11:46.052000+0000
         SFPL | 1 |       2 | 6bec5c6a-5173-4625-a0fa-75c37b83b76a |      create-index [mthreads], cause [api] | 2020-03-28 04:03:49.892000+0000
         SFPL | 0 |       2 | 13d84671-ca17-4ad0-868d-35fa5beb616f |     init table elastic_admin.metadata_log | 2020-03-25 02:34:20.047000+0000

• cassandra gossip state (run: nodetool gossipinfo)

/10.0.0.2
  generation:1585368110
  heartbeat:13640950
  STATUS:16:NORMAL,-1304836135575544344
  LOAD:13640930:6701155.0
  SCHEMA:13639170:209586f7-12dd-3e20-8a23-e891705a965a
  DC:8:DC1
  RACK:10:r1
  RELEASE_VERSION:4:3.11.4.3
  INTERNAL_IP:6:10.0.0.2
  RPC_ADDRESS:3:10.0.0.2
  NET_VERSION:1:11
  HOST_ID:2:6bec5c6a-5173-4625-a0fa-75c37b83b76a
  RPC_READY:28:true
  X1:225:{"mthreads":3}
  X2:223:6bec5c6a-5173-4625-a0fa-75c37b83b76a/1
  TOKENS:15:<hidden>
/10.0.0.3
  generation:1585138504
  heartbeat:13878297
  STATUS:16:NORMAL,-1212298838116040910
  LOAD:13878291:6739322.0
  SCHEMA:13876516:209586f7-12dd-3e20-8a23-e891705a965a
  DC:8:DC1
  RACK:10:r1
  RELEASE_VERSION:4:3.11.4.3
  INTERNAL_IP:6:10.0.0.3
  RPC_ADDRESS:3:10.0.0.3
  NET_VERSION:1:11
  HOST_ID:2:6b4425b3-aea7-4b7d-af64-03f245c73493
  RPC_READY:28:true
  X1:237426:{"mthreads":3}
  X2:237424:6bec5c6a-5173-4625-a0fa-75c37b83b76a/1
  TOKENS:15:<hidden>
/10.0.0.4
  generation:1585103620
  heartbeat:13914268
  STATUS:54:NORMAL,-1165308580040169327
  LOAD:13914241:7038248.0
  SCHEMA:13912487:209586f7-12dd-3e20-8a23-e891705a965a
  DC:8:DC1
  RACK:10:r1
  RELEASE_VERSION:4:3.11.4.3
  INTERNAL_IP:6:10.0.0.4
  RPC_ADDRESS:3:10.0.0.4
  NET_VERSION:1:11
  HOST_ID:2:13d84671-ca17-4ad0-868d-35fa5beb616f
  RPC_READY:67:true
  X1:273569:{"mthreads":3}
  X2:273566:6bec5c6a-5173-4625-a0fa-75c37b83b76a/1
  TOKENS:53:<hidden>

[vroyer]

The PAXOS update failed because it tried to write a row with version=2 in metadata_log, but there is already a row with version 2 from the day before = 2020-08-27 22:11:46.052000+0000, (while error message are 2020-08-28 01:27:46).

So, for some reason, something wrong happen around 2020-08-27 22:11:46 and the elasticsearch cluster state is not more synchronised with the metadata_log, clusterstate metadata.version = 1 and metadata_log.version = 2.

You can resynchronise manually like this:
DELETE FROM elastic_admin.metadata_log WHERE cluster_name = 'SFPL' and v = 2;
UPDATE elastic_admin.metadata_log SET version = 1 WHERE cluster_name = 'SFPL';
Then you should be able to create an index.

You probably have WARN/ERRROR messages around 2020-08-27 22:11:46 in your logs.
Check that QUORUM was available on elastic_admin keyspace.
You can also increase the cas_contention_timeout_in_ms to 5000 or more if you have a network latency issue.

[dioptre]

Thanks, there was definitely a PAXOS timeout. I wonder what caused it. Regardless what you suggested fixed it. Thanks.

Is this a bug?

[vroyer]

A high network latency could cause such error. If you have the logs of the coordinator nodes when the first PAXOS timeout occurs, it could help to investigate.

…

On 3 Sep 2020, at 20:27, Andrew Grosser ***@***.***> wrote: Thanks, there was definitely a PAXOS timeout. I wonder what caused it. Regardless what you suggested fixed it. Thanks. Is this a bug? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#367 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACOMPGKVI3CG2F25IYB7GBTSD7NYRANCNFSM4QNQ5B5Q>.

[pengpeng-fan]

Is there any way to avoid this problem

[serversteam]

@dioptre @vroyer
Can we the same for existing staging cluster. I mean does we lost data or some information if we remove rows manually ?