Skip to content

Commit

Permalink
Add a check for identifiers with maximum length (#466)
Browse files Browse the repository at this point in the history
* Add AnyObject and PgObjectType

* Add tests for AnyObject and PgObjectType

* Fix mutation tests

* Add new check and test

* Add PossibleObjectNameOverflowCheckOnCluster

* Add PossibleObjectNameOverflowCheckOnCluster to DatabaseChecks

* Extend HealthLogger and Spring Boot autoconfiguration
  • Loading branch information
mfvanek authored Oct 20, 2024
1 parent 918ec90 commit 92254d1
Show file tree
Hide file tree
Showing 63 changed files with 1,023 additions and 51 deletions.
49 changes: 25 additions & 24 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,30 +46,31 @@ All checks can be divided into 2 groups:

**pg-index-health** allows you to detect the following problems:

|| Description | Type | SQL query |
|----|------------------------------------------------------------------------------------------------------------------------------------|--------------------|----------------------------------------------------------------------------------------------------------|
| 1 | Invalid (broken) indexes | **runtime**/static | [sql](https://github.com/mfvanek/pg-index-health-sql/blob/master/sql/invalid_indexes.sql) |
| 1 | Duplicated (completely identical) indexes | static | [sql](https://github.com/mfvanek/pg-index-health-sql/blob/master/sql/duplicated_indexes.sql) |
| 3 | Intersected (partially identical) indexes | static | [sql](https://github.com/mfvanek/pg-index-health-sql/blob/master/sql/intersected_indexes.sql) |
| 4 | Unused indexes | **runtime** | [sql](https://github.com/mfvanek/pg-index-health-sql/blob/master/sql/unused_indexes.sql) |
| 5 | Foreign keys without associated indexes | static | [sql](https://github.com/mfvanek/pg-index-health-sql/blob/master/sql/foreign_keys_without_index.sql) |
| 6 | Indexes with null values | static | [sql](https://github.com/mfvanek/pg-index-health-sql/blob/master/sql/indexes_with_null_values.sql) |
| 7 | Tables with missing indexes | **runtime** | [sql](https://github.com/mfvanek/pg-index-health-sql/blob/master/sql/tables_with_missing_indexes.sql) |
| 8 | Tables without primary key | static | [sql](https://github.com/mfvanek/pg-index-health-sql/blob/master/sql/tables_without_primary_key.sql) |
| 9 | Indexes [bloat](https://www.percona.com/blog/2018/08/06/basic-understanding-bloat-vacuum-postgresql-mvcc/) | **runtime** | [sql](https://github.com/mfvanek/pg-index-health-sql/blob/master/sql/bloated_indexes.sql) |
| 10 | Tables [bloat](https://www.percona.com/blog/2018/08/06/basic-understanding-bloat-vacuum-postgresql-mvcc/) | **runtime** | [sql](https://github.com/mfvanek/pg-index-health-sql/blob/master/sql/bloated_tables.sql) |
| 11 | Tables without [description](https://www.postgresql.org/docs/current/sql-comment.html) | static | [sql](https://github.com/mfvanek/pg-index-health-sql/blob/master/sql/tables_without_description.sql) |
| 12 | Columns without [description](https://www.postgresql.org/docs/current/sql-comment.html) | static | [sql](https://github.com/mfvanek/pg-index-health-sql/blob/master/sql/columns_without_description.sql) |
| 13 | Columns with [json](https://www.postgresql.org/docs/current/datatype-json.html) type | static | [sql](https://github.com/mfvanek/pg-index-health-sql/blob/master/sql/columns_with_json_type.sql) |
| 14 | Columns of [serial types](https://www.postgresql.org/docs/current/datatype-numeric.html#DATATYPE-SERIAL) that are not primary keys | static | [sql](https://github.com/mfvanek/pg-index-health-sql/blob/master/sql/columns_with_serial_types.sql) |
| 15 | Functions without [description](https://www.postgresql.org/docs/current/sql-comment.html) | static | [sql](https://github.com/mfvanek/pg-index-health-sql/blob/master/sql/functions_without_description.sql) |
| 16 | Indexes [with boolean](https://habr.com/ru/companies/tensor/articles/488104/) | static | [sql](https://github.com/mfvanek/pg-index-health-sql/blob/master/sql/indexes_with_boolean.sql) |
| 17 | Tables with [not valid constraints](https://habr.com/ru/articles/800121/) | **runtime**/static | [sql](https://github.com/mfvanek/pg-index-health-sql/blob/master/sql/not_valid_constraints.sql) |
| 18 | B-tree indexes [on array columns](https://habr.com/ru/articles/800121/) | static | [sql](https://github.com/mfvanek/pg-index-health-sql/blob/master/sql/btree_indexes_on_array_columns.sql) |
| 19 | [Sequence overflow](https://habr.com/ru/articles/800121/) | **runtime** | [sql](https://github.com/mfvanek/pg-index-health-sql/blob/master/sql/sequence_overflow.sql) |
| 20 | Primary keys with [serial types](https://wiki.postgresql.org/wiki/Don't_Do_This#Don.27t_use_serial) | static | [sql](https://github.com/mfvanek/pg-index-health-sql/blob/master/sql/primary_keys_with_serial_types.sql) |
| 21 | Duplicated ([completely identical](https://habr.com/ru/articles/803841/)) foreign keys | static | [sql](https://github.com/mfvanek/pg-index-health-sql/blob/master/sql/duplicated_foreign_keys.sql) |
| 22 | Intersected ([partially identical](https://habr.com/ru/articles/803841/)) foreign keys | static | [sql](https://github.com/mfvanek/pg-index-health-sql/blob/master/sql/intersected_foreign_keys.sql) |
|| Description | Type | SQL query |
|----|------------------------------------------------------------------------------------------------------------------------------------|--------------------|-----------------------------------------------------------------------------------------------------------|
| 1 | Invalid (broken) indexes | **runtime**/static | [sql](https://github.com/mfvanek/pg-index-health-sql/blob/master/sql/invalid_indexes.sql) |
| 1 | Duplicated (completely identical) indexes | static | [sql](https://github.com/mfvanek/pg-index-health-sql/blob/master/sql/duplicated_indexes.sql) |
| 3 | Intersected (partially identical) indexes | static | [sql](https://github.com/mfvanek/pg-index-health-sql/blob/master/sql/intersected_indexes.sql) |
| 4 | Unused indexes | **runtime** | [sql](https://github.com/mfvanek/pg-index-health-sql/blob/master/sql/unused_indexes.sql) |
| 5 | Foreign keys without associated indexes | static | [sql](https://github.com/mfvanek/pg-index-health-sql/blob/master/sql/foreign_keys_without_index.sql) |
| 6 | Indexes with null values | static | [sql](https://github.com/mfvanek/pg-index-health-sql/blob/master/sql/indexes_with_null_values.sql) |
| 7 | Tables with missing indexes | **runtime** | [sql](https://github.com/mfvanek/pg-index-health-sql/blob/master/sql/tables_with_missing_indexes.sql) |
| 8 | Tables without primary key | static | [sql](https://github.com/mfvanek/pg-index-health-sql/blob/master/sql/tables_without_primary_key.sql) |
| 9 | Indexes [bloat](https://www.percona.com/blog/2018/08/06/basic-understanding-bloat-vacuum-postgresql-mvcc/) | **runtime** | [sql](https://github.com/mfvanek/pg-index-health-sql/blob/master/sql/bloated_indexes.sql) |
| 10 | Tables [bloat](https://www.percona.com/blog/2018/08/06/basic-understanding-bloat-vacuum-postgresql-mvcc/) | **runtime** | [sql](https://github.com/mfvanek/pg-index-health-sql/blob/master/sql/bloated_tables.sql) |
| 11 | Tables without [description](https://www.postgresql.org/docs/current/sql-comment.html) | static | [sql](https://github.com/mfvanek/pg-index-health-sql/blob/master/sql/tables_without_description.sql) |
| 12 | Columns without [description](https://www.postgresql.org/docs/current/sql-comment.html) | static | [sql](https://github.com/mfvanek/pg-index-health-sql/blob/master/sql/columns_without_description.sql) |
| 13 | Columns with [json](https://www.postgresql.org/docs/current/datatype-json.html) type | static | [sql](https://github.com/mfvanek/pg-index-health-sql/blob/master/sql/columns_with_json_type.sql) |
| 14 | Columns of [serial types](https://www.postgresql.org/docs/current/datatype-numeric.html#DATATYPE-SERIAL) that are not primary keys | static | [sql](https://github.com/mfvanek/pg-index-health-sql/blob/master/sql/columns_with_serial_types.sql) |
| 15 | Functions without [description](https://www.postgresql.org/docs/current/sql-comment.html) | static | [sql](https://github.com/mfvanek/pg-index-health-sql/blob/master/sql/functions_without_description.sql) |
| 16 | Indexes [with boolean](https://habr.com/ru/companies/tensor/articles/488104/) | static | [sql](https://github.com/mfvanek/pg-index-health-sql/blob/master/sql/indexes_with_boolean.sql) |
| 17 | Tables with [not valid constraints](https://habr.com/ru/articles/800121/) | **runtime**/static | [sql](https://github.com/mfvanek/pg-index-health-sql/blob/master/sql/not_valid_constraints.sql) |
| 18 | B-tree indexes [on array columns](https://habr.com/ru/articles/800121/) | static | [sql](https://github.com/mfvanek/pg-index-health-sql/blob/master/sql/btree_indexes_on_array_columns.sql) |
| 19 | [Sequence overflow](https://habr.com/ru/articles/800121/) | **runtime** | [sql](https://github.com/mfvanek/pg-index-health-sql/blob/master/sql/sequence_overflow.sql) |
| 20 | Primary keys with [serial types](https://wiki.postgresql.org/wiki/Don't_Do_This#Don.27t_use_serial) | static | [sql](https://github.com/mfvanek/pg-index-health-sql/blob/master/sql/primary_keys_with_serial_types.sql) |
| 21 | Duplicated ([completely identical](https://habr.com/ru/articles/803841/)) foreign keys | static | [sql](https://github.com/mfvanek/pg-index-health-sql/blob/master/sql/duplicated_foreign_keys.sql) |
| 22 | Intersected ([partially identical](https://habr.com/ru/articles/803841/)) foreign keys | static | [sql](https://github.com/mfvanek/pg-index-health-sql/blob/master/sql/intersected_foreign_keys.sql) |
| 23 | Possible object name overflow (identifiers with maximum length). | static | [sql](https://github.com/mfvanek/pg-index-health-sql/blob/master/sql/possible_object_name_overflow.sql) |

For raw sql queries see [pg-index-health-sql](https://github.com/mfvanek/pg-index-health-sql) project.

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
/*
* Copyright (c) 2019-2024. Ivan Vakhrushev and others.
* https://github.com/mfvanek/pg-index-health
*
* This file is a part of "pg-index-health" - a Java library for
* analyzing and maintaining indexes health in PostgreSQL databases.
*
* Licensed under the Apache License 2.0
*/

package io.github.mfvanek.pg.checks.host;

import io.github.mfvanek.pg.common.maintenance.Diagnostic;
import io.github.mfvanek.pg.connection.PgConnection;
import io.github.mfvanek.pg.model.PgContext;
import io.github.mfvanek.pg.model.object.AnyObject;

import java.util.List;
import javax.annotation.Nonnull;

/**
* Check for objects whose names have a length of {@code max_identifier_length} (usually it is 63) on a specific host.
* <p>
* The problem is that Postgres silently truncates such long names.
* For example, if you have a migration where you are trying to create two objects with very long names
* that start the same way (such as an index or constraint) and you use the "if not exists" statement,
* you might end up with only one object in the database instead of two.
* <p>
* If there is an object with a name of maximum length in the database, then an overflow may have occurred.
* It is advisable to avoid such situations and use shorter names.
*
* @author Ivan Vahrushev
* @see <a href="https://www.postgresql.org/docs/current/limits.html">PostgreSQL Limits</a>
* @since 0.13.2
*/
public class PossibleObjectNameOverflowCheckOnHost extends AbstractCheckOnHost<AnyObject> {

public PossibleObjectNameOverflowCheckOnHost(@Nonnull final PgConnection pgConnection) {
super(AnyObject.class, pgConnection, Diagnostic.POSSIBLE_OBJECT_NAME_OVERFLOW);
}

/**
* Returns objects whose names have a length of {@code max_identifier_length} in the specified schema.
*
* @param pgContext check's context with the specified schema; must not be null
* @return list of objects whose names have a length of {@code max_identifier_length}
*/
@Nonnull
@Override
protected List<AnyObject> doCheck(@Nonnull final PgContext pgContext) {
return executeQuery(pgContext, rs -> {
final String objectName = rs.getString("object_name");
final String objectType = rs.getString("object_type");
return AnyObject.ofRaw(objectName, objectType);
});
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,9 @@ public PrimaryKeysWithSerialTypesCheckOnHost(@Nonnull final PgConnection pgConne
super(ColumnWithSerialType.class, pgConnection, Diagnostic.PRIMARY_KEYS_WITH_SERIAL_TYPES);
}

/**
* {@inheritDoc}
*/
@Nonnull
@Override
protected List<ColumnWithSerialType> doCheck(@Nonnull final PgContext pgContext) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ public SequenceOverflowCheckOnHost(@Nonnull final PgConnection pgConnection) {
/**
* Returns sequences that are close to overflow in the specified schema.
*
* @param pgContext check's context with the specified schema
* @param pgContext check's context with the specified schema; must not be null
* @return list of sequences close to overflow
* @see SequenceState
*/
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,8 @@ public enum Diagnostic implements CheckTypeAware {
SEQUENCE_OVERFLOW(ExecutionTopology.ON_PRIMARY, "sequence_overflow.sql", QueryExecutors::executeQueryWithRemainingPercentageThreshold, true),
PRIMARY_KEYS_WITH_SERIAL_TYPES(ExecutionTopology.ON_PRIMARY, "primary_keys_with_serial_types.sql", QueryExecutors::executeQueryWithSchema, false),
DUPLICATED_FOREIGN_KEYS(ExecutionTopology.ON_PRIMARY, "duplicated_foreign_keys.sql", QueryExecutors::executeQueryWithSchema, false),
INTERSECTED_FOREIGN_KEYS(ExecutionTopology.ON_PRIMARY, "intersected_foreign_keys.sql", QueryExecutors::executeQueryWithSchema, false);
INTERSECTED_FOREIGN_KEYS(ExecutionTopology.ON_PRIMARY, "intersected_foreign_keys.sql", QueryExecutors::executeQueryWithSchema, false),
POSSIBLE_OBJECT_NAME_OVERFLOW(ExecutionTopology.ON_PRIMARY, "possible_object_name_overflow.sql", QueryExecutors::executeQueryWithSchema, false);

private final ExecutionTopology executionTopology;
private final String sqlQueryFileName;
Expand Down
2 changes: 1 addition & 1 deletion pg-index-health-core/src/main/resources
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
/*
* Copyright (c) 2019-2024. Ivan Vakhrushev and others.
* https://github.com/mfvanek/pg-index-health
*
* This file is a part of "pg-index-health" - a Java library for
* analyzing and maintaining indexes health in PostgreSQL databases.
*
* Licensed under the Apache License 2.0
*/

package io.github.mfvanek.pg.checks.host;

import io.github.mfvanek.pg.common.maintenance.DatabaseCheckOnHost;
import io.github.mfvanek.pg.common.maintenance.Diagnostic;
import io.github.mfvanek.pg.model.PgContext;
import io.github.mfvanek.pg.model.object.AnyObject;
import io.github.mfvanek.pg.model.object.PgObjectType;
import io.github.mfvanek.pg.support.DatabaseAwareTestBase;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.ValueSource;

import static io.github.mfvanek.pg.support.AbstractCheckOnHostAssert.assertThat;

class PossibleObjectNameOverflowCheckOnHostTest extends DatabaseAwareTestBase {

private final DatabaseCheckOnHost<AnyObject> check = new PossibleObjectNameOverflowCheckOnHost(getPgConnection());

@Test
void shouldSatisfyContract() {
assertThat(check)
.hasType(AnyObject.class)
.hasDiagnostic(Diagnostic.POSSIBLE_OBJECT_NAME_OVERFLOW)
.hasHost(getHost())
.isStatic();
}

@ParameterizedTest
@ValueSource(strings = {PgContext.DEFAULT_SCHEMA_NAME, "custom"})
void onDatabaseWithThem(final String schemaName) {
executeTestOnDatabase(schemaName, dbp -> dbp.withReferences().withMaterializedView().withIdentityPrimaryKey(), ctx -> {
final String matViewName = ctx.enrichWithSchema("accounts_materialized_view_with_length_63_1234567890_1234567890");
final String constraintName = ctx.enrichWithSchema("num_less_than_million_constraint_with_length_63_1234567890_1234");
assertThat(check)
.executing(ctx)
.hasSize(2)
.containsExactlyInAnyOrder(
AnyObject.ofType(matViewName, PgObjectType.MATERIALIZED_VIEW),
AnyObject.ofType(constraintName, PgObjectType.CONSTRAINT));

assertThat(check)
.executing(ctx, t -> !(t.getName().equals(matViewName) || t.getName().equals(constraintName)))
.isEmpty();
});
}
}
Loading

0 comments on commit 92254d1

Please sign in to comment.