Skip to content

Commit

Permalink
Merge with master
Browse files Browse the repository at this point in the history
  • Loading branch information
@alexanderkurash
alexanderkurash committed Oct 23, 2024
2 parents f7f9680 + ed2dd41 commit 08dc519
Show file tree
Hide file tree
Showing 68 changed files with 1,163 additions and 209 deletions.
28 changes: 26 additions & 2 deletions NEWS.md
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,26 @@
## v27.2.0 In progress
## v28.0.0 In progress
### Breaking changes
* Migrate "publicationPeriod" data to the Dates object and remove it from the Instance schema ([MODINVSTOR-1232](https://folio-org.atlassian.net/browse/MODINVSTOR-1232))

### New APIs versions
* Provides `instance-storage 11.0`
* Provides `instance-storage-batch 3.0`
* Provides `instance-storage-batch-sync 3.0`
* Provides `instance-storage-batch-sync-unsafe 3.0`
* Provides `inventory-view 3.0`
* Provides `inventory-view-instance-set 3.0`
* Provides `instance-iteration 1.0`
* Requires `holdings-storage 8.0`
* Requires `bound-with-parts-storage 2.0`
* Requires `async-migration 1.0`

### Features
* Add module descriptor validator plugin and fix the permission names ([MODINVSTOR-1247](https://folio-org.atlassian.net/browse/MODINVSTOR-1247))

### Tech Dept
* Upgrade localstack from 0.11.3 to s3-latest (=3.8.0) ([MODINVSTOR-1272](https://folio-org.atlassian.net/browse/MODINVSTOR-1272))

## v27.2.0 2024-09-24
### Breaking changes
* Required sourceId field in holdings record ([MODINVSTOR-1161](https://folio-org.atlassian.net/browse/MODINVSTOR-1161))

Expand Down Expand Up @@ -29,14 +50,16 @@
* Add new date type fields to Instance schema ([MODINVSTOR-1188](https://folio-org.atlassian.net/browse/MODINVSTOR-1188))
* Implement endpoint for bulk instances upsert from external file ([MODINVSTOR-1225](https://folio-org.atlassian.net/browse/MODINVSTOR-1225))
* Add Subject source and Subject type to schema ([MODINVSTOR-1205](https://folio-org.atlassian.net/browse/MODINVSTOR-1205))

* Add codes to Subject sources ([MODINVSTOR-1264](https://folio-org.atlassian.net/browse/MODINVSTOR-1264))

### Bug fixes
* Unintended update of instance records \_version (optimistic locking) whenever any of its holdings or items are created, updated or deleted. ([MODINVSTOR-1186](https://folio-org.atlassian.net/browse/MODINVSTOR-1186))
* Deserialization of Date from long (MODINVSTOR-1198)[https://folio-org.atlassian.net/browse/MODINVSTOR-1198]
* Do not delete Kafka topics on postTenant if collection topics is enabled ([MODINVSTOR-1192](https://folio-org.atlassian.net/browse/MODINVSTOR-1192))
* Identifier types: change Cancelled LCCN to Canceled LCCN ([MODINVSTOR-1212](https://folio-org.atlassian.net/browse/MODINVSTOR-1212))
* Add user-tenants.collection.get to all ECS APIs ([MODINVSTOR-1253](https://folio-org.atlassian.net/browse/MODINVSTOR-1253))
* Add user-tenants.collection.get to POST /\_/tenant API ([MODINVSTOR-1260](https://folio-org.atlassian.net/browse/MODINVSTOR-1260))
* Update "BC" name in GET instance-date-type to "B.C." ([MODINVSTOR-1255](https://folio-org.atlassian.net/browse/MODINVSTOR-1255))

### Tech Dept
* Kafka testcontainers: kafka.KafkaContainer, apache/kafka-native:3.8.0, KafkaTopicsExistsTest fix ([MODINVSTOR-1251](https://folio-org.atlassian.net/browse/MODINVSTOR-1251))
Expand All @@ -47,6 +70,7 @@
* Bump `holdings-storage` from `6.0` to `7.0`
* Bump `holdings-storage-batch-sync` from `1.1` to `2.0`
* Bump `holdings-storage-batch-sync-unsafe` from `1.0` to `2.0`
* Bump `folio-kafka-wrapper` from `3.1.1` to `3.2.0`
* Add `folio-s3-client` `2.2.0`
* Add `LIB_NAME` `2.7.4`
* Remove `LIB_NAME`
Expand Down
16 changes: 16 additions & 0 deletions README.MD
View file
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,20 @@ These properties can be changed by setting env variable.
* `KAFKA_REINDEX_RECORDS_TOPIC_NUM_PARTITIONS` Default value - `16`
* `KAFKA_SUBJECT_SOURCE_TOPIC_NUM_PARTITIONS` Default value - `1`

There is also possibility for customizing through properties the Kafka Topic
`message retention` (in milliseconds) and `maximum message size` (in bytes). The default values of these configurations
for any topic are 604800000 milliseconds (or 1 week) and 1048576 bytes (or 1 MB) respectively

These are the defined topic properties for `message retention` and `maximum message size` for `reindex-records` topic

* `KAFKA_REINDEX_RECORDS_TOPIC_MESSAGE_RETENTION` Default value - `86400000` (1 day)
* `KAFKA_REINDEX_RECORDS_TOPIC_MAX_MESSAGE_SIZE` Default value - `1048576` (1 MB)

in case, any of these topic properties are changed for `reindex-records` the topic needs to be recreated and module needs to be reinstalled.

Changing maximum message size for kafka producer:
* `KAFKA_REINDEX_PRODUCER_MAX_REQUEST_SIZE_BYTES` Default value - `10485760` (10 MB)

# Building

run `mvn install` from the root directory.
Expand Down Expand Up @@ -142,6 +156,8 @@ These environment variables configure Kafka topic for specific business-related
* `KAFKA_SUBJECT_TYPE_TOPIC_NUM_PARTITIONS`
* `KAFKA_REINDEX_RECORDS_TOPIC_NUM_PARTITIONS`
* `KAFKA_SUBJECT_SOURCE_TOPIC_NUM_PARTITIONS`
* `KAFKA_REINDEX_RECORDS_TOPIC_MESSAGE_RETENTION`
* `KAFKA_REINDEX_RECORDS_TOPIC_MAX_MESSAGE_SIZE`

mod-inventory-storage also supports all Raml Module Builder (RMB) environment variables,
for details see [RMB](https://github.com/folio-org/raml-module-builder#environment-variables):
Expand Down
70 changes: 52 additions & 18 deletions descriptors/ModuleDescriptor-template.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@
},
{
"id": "holdings-storage",
"version": "7.0",
"version": "8.0",
"handlers": [
{
"methods": ["GET"],
Expand All @@ -104,7 +104,7 @@
},{
"methods": ["POST"],
"pathPattern": "/holdings-storage/holdings/retrieve",
"permissionsRequired": ["inventory-storage.holdings.collection.get"]
"permissionsRequired": ["inventory-storage.holdings.retrieve.collection.post"]
}, {
"methods": ["GET"],
"pathPattern": "/holdings-storage/holdings/{id}",
Expand Down Expand Up @@ -167,7 +167,7 @@
},
{
"id": "instance-storage",
"version": "10.3",
"version": "11.0",
"handlers": [
{
"methods": ["GET"],
Expand All @@ -176,7 +176,7 @@
},{
"methods": ["POST"],
"pathPattern": "/instance-storage/instances/retrieve",
"permissionsRequired": ["inventory-storage.instances.collection.get"]
"permissionsRequired": ["inventory-storage.instances.retrieve.collection.post"]
},
{
"methods": ["GET"],
Expand Down Expand Up @@ -274,7 +274,7 @@
},
{
"id": "instance-storage-batch",
"version": "2.0",
"version": "3.0",
"handlers": [
{
"methods": ["POST"],
Expand All @@ -285,18 +285,18 @@
},
{
"id": "instance-storage-batch-sync",
"version": "2.0",
"version": "3.0",
"handlers": [
{
"methods": ["POST"],
"pathPattern": "/instance-storage/batch/synchronous",
"permissionsRequired": ["inventory-storage.instances.batch.post"]
"permissionsRequired": ["inventory-storage.instances.batch.synchronous.post"]
}
]
},
{
"id": "instance-storage-batch-sync-unsafe",
"version": "2.0",
"version": "3.0",
"handlers": [
{
"methods": ["POST"],
Expand Down Expand Up @@ -1240,7 +1240,7 @@
},
{
"id": "bound-with-parts-storage",
"version": "1.1",
"version": "2.0",
"handlers": [
{
"methods": ["GET"],
Expand All @@ -1266,9 +1266,7 @@
"methods": ["PUT"],
"pathPattern": "/inventory-storage/bound-withs",
"permissionsRequired": [
"inventory-storage.bound-with-parts.collection.get",
"inventory-storage.bound-with-parts.item.post",
"inventory-storage.bound-with-parts.item.delete"
"inventory-storage.bound-withs.collection.put"
]
}
]
Expand All @@ -1280,7 +1278,10 @@
"handlers": [
{
"methods": ["POST"],
"pathPattern": "/_/tenant"
"pathPattern": "/_/tenant",
"modulePermissions": [
"user-tenants.collection.get"
]
}, {
"methods": ["DELETE", "GET"],
"pathPattern": "/_/tenant/{id}"
Expand Down Expand Up @@ -1336,7 +1337,7 @@
},
{
"id": "inventory-view",
"version": "2.0",
"version": "3.0",
"handlers": [
{
"methods": ["GET"],
Expand All @@ -1347,7 +1348,7 @@
},
{
"id": "inventory-view-instance-set",
"version": "2.0",
"version": "3.0",
"handlers": [
{
"methods": ["GET"],
Expand Down Expand Up @@ -1384,7 +1385,7 @@
},
{
"id": "instance-iteration",
"version": "0.1",
"version": "1.0",
"handlers": [
{
"methods": ["POST"],
Expand All @@ -1405,7 +1406,7 @@
},
{
"id": "async-migration",
"version": "0.1",
"version": "1.0",
"handlers": [
{
"methods": ["POST"],
Expand All @@ -1425,7 +1426,7 @@
{
"methods": ["GET"],
"pathPattern": "/inventory-storage/migrations/jobs",
"permissionsRequired": ["inventory-storage.migration.job.item.get"]
"permissionsRequired": ["inventory-storage.migration.job.collection.get"]
},
{
"methods": ["DELETE"],
Expand Down Expand Up @@ -1522,6 +1523,11 @@
"displayName": "inventory storage - get holdings collection",
"description": "get holdings collection from storage"
},
{
"permissionName": "inventory-storage.holdings.retrieve.collection.post",
"displayName": "inventory storage - retrieve holdings collection",
"description": "retrieve holdings collection from storage"
},
{
"permissionName": "inventory-storage.holdings.collection.delete",
"displayName": "inventory storage - delete entire holdings collection",
Expand Down Expand Up @@ -1562,6 +1568,11 @@
"displayName": "inventory storage - get instance collection",
"description": "get instance collection from storage"
},
{
"permissionName": "inventory-storage.instances.retrieve.collection.post",
"displayName": "inventory storage - retrieve instance collection",
"description": "retrieve instance collection from storage"
},
{
"permissionName": "inventory-storage.instances.collection.delete",
"displayName": "inventory storage - delete entire instance collection",
Expand Down Expand Up @@ -1597,6 +1608,11 @@
"displayName": "inventory storage - create or update a number of instances",
"description": "create or update a number of instances in storage"
},
{
"permissionName": "inventory-storage.instances.batch.synchronous.post",
"displayName": "inventory storage - create or update a collection of instances in a single synchronous request",
"description": "create or update a collection of instances in a single synchronous request"
},
{
"permissionName": "inventory-storage.instances.batch-unsafe.post",
"displayName": "inventory storage - create or update a number of instances with optimistic locking disabled",
Expand Down Expand Up @@ -2538,6 +2554,11 @@
"displayName": "inventory storage - add a part to a bound-with",
"description": "add a holdings record to a bound-with by associating it with the bound-with item"
},
{
"permissionName": "inventory-storage.bound-withs.collection.put",
"displayName": "inventory storage - modify a bound-withs",
"description": "replace a holdings-records of a bound-withs or move it to a different bound-withs"
},
{
"permissionName": "inventory-storage.bound-with-parts.item.put",
"displayName": "inventory storage - modify a bound-with part",
Expand Down Expand Up @@ -2603,6 +2624,11 @@
"displayName": "inventory storage - get migration job by id",
"description": "get migration job by id"
},
{
"permissionName": "inventory-storage.migration.job.collection.get",
"displayName": "inventory storage - get migration jobs",
"description": "get migration jobs"
},
{
"permissionName": "inventory-storage.migration.item.get",
"displayName": "inventory storage - get list of available migrations",
Expand Down Expand Up @@ -2644,6 +2670,7 @@
"inventory-storage.items.batch.post",
"inventory-storage.items.batch-unsafe.post",
"inventory-storage.holdings.collection.get",
"inventory-storage.holdings.retrieve.collection.post",
"inventory-storage.holdings.item.get",
"inventory-storage.holdings.item.post",
"inventory-storage.holdings.item.put",
Expand All @@ -2652,6 +2679,7 @@
"inventory-storage.holdings.batch.post",
"inventory-storage.holdings.batch-unsafe.post",
"inventory-storage.instances.collection.get",
"inventory-storage.instances.retrieve.collection.post",
"inventory-storage.instances.item.get",
"inventory-storage.instances.item.post",
"inventory-storage.instances.item.put",
Expand All @@ -2663,6 +2691,7 @@
"inventory-storage.instances.source-record.marc-json.delete",
"inventory-storage.instances.collection.delete",
"inventory-storage.instances.batch.post",
"inventory-storage.instances.batch.synchronous.post",
"inventory-storage.instances.batch-unsafe.post",
"inventory-storage.instances.bulk.post",
"inventory-storage.loan-types.collection.get",
Expand Down Expand Up @@ -2850,6 +2879,7 @@
"inventory-storage.bound-with-parts.collection.get",
"inventory-storage.bound-with-parts.item.get",
"inventory-storage.bound-with-parts.item.post",
"inventory-storage.bound-withs.collection.put",
"inventory-storage.bound-with-parts.item.put",
"inventory-storage.bound-with-parts.item.delete",
"inventory-storage.inventory-hierarchy.updated-instances-ids.collection.get",
Expand All @@ -2867,6 +2897,7 @@
"inventory-storage.migration.job.item.delete",
"inventory-storage.migration.job.post",
"inventory-storage.migration.job.item.get",
"inventory-storage.migration.job.collection.get",
"inventory-storage.migration.item.get",
"inventory-storage.reindex-records.publish.post",
"inventory-storage.instance-date-types.collection.get",
Expand Down Expand Up @@ -2914,6 +2945,9 @@
{ "name": "KAFKA_SUBJECT_TYPE_TOPIC_NUM_PARTITIONS", "value": "1"},
{ "name": "KAFKA_SUBJECT_SOURCE_TOPIC_NUM_PARTITIONS", "value": "1"},
{ "name": "KAFKA_REINDEX_RECORDS_TOPIC_NUM_PARTITIONS", "value": "16"},
{ "name": "KAFKA_REINDEX_RECORDS_TOPIC_MESSAGE_RETENTION", "value": "86400000"},
{ "name": "KAFKA_REINDEX_RECORDS_TOPIC_MAX_MESSAGE_SIZE", "value": "1048576"},
{ "name": "KAFKA_REINDEX_PRODUCER_MAX_REQUEST_SIZE_BYTES", "value": "10485760"},
{ "name": "S3_URL", "value": "http://127.0.0.1:9000/" },
{ "name": "S3_REGION", "value": "" },
{ "name": "S3_BUCKET", "value": "marc-migrations" },
Expand Down
Loading

0 comments on commit 08dc519

Please sign in to comment.