[WIP] OGC API Features - Part 4 / Support for PostgreSQLProvider #1266

MTachon · 2023-06-15T13:39:59Z

Overview

This PR is intended to add transactions support to the PostgreSQLProvider. Implementation should follow the OGC API - Features - Part 4: Create, Replace, Update and Delete draft specification.

Additional Information

The OGC API - Features - Part 4: Create, Replace, Update and Delete draft specification is not an OGC standard and is subject to change.

As of now, the draft specification does not mention the use of schemas for data validation in POST/PUT/PATCH (create/replace/update) requests. This is a work in progress and the schema related content of both Part 3 and Part 4 will likely be moved to a Part 5 specification document. One can get more information about this process at opengeospatial/ogcapi-features#740. Once more information about schemas is available and that this part of the specification is implemented in pygeoapi, the exception handling around the database transactions lines of code will be removed, and we will let schemas validation do the work and stop requests with invalid data.

Contributions and Licensing

(as per https://github.com/geopython/pygeoapi/blob/master/CONTRIBUTING.md#contributions-and-licensing)

I'd like to contribute [feature X|bugfix Y|docs|something else] to pygeoapi. I confirm that my contributions to pygeoapi will be compatible with the pygeoapi license guidelines at the time of contribution.
I have already previously agreed to the pygeoapi Contributions and Licensing Guidelines

The new function 'crs_transform_json_geom' used in the crs_transform decorator allows the coordinates of a feature (GeoJSON-like dict) to be transformed directly from one coordinate sytem to another, without first being converted to a shapely geometrical object. This new pipeline: 'GeoJSON Feature -> coordinates transformation' can replace the current pipeline: 'GeoJSON Feature -> convertion to Shapely geometry -> coordinates transformation -> convertion back to GeoJSON Feature'.

… class

…l-transactions

…json

…ions

… PostgreSQL

…orts

MTachon · 2023-06-20T08:43:11Z

In the test_manage_collection_items_postgresql_create function, I create a new item in the database, and then I want to get back this newly created feature by using the feature's URI. I get a connection error (Failed to establish a new connection: [Errno 111] Connection refused) in the requests.get call, in the main.yml workflow. @tomkralidis , @justb4 , @webb-ben , @francbartoli , any ideas why?

…nvalid data

francbartoli · 2023-06-20T11:13:01Z

pygeoapi/provider/postgresql.py

+                session.commit()
+                session.refresh(db_obj_mapping)
+                identifier = getattr(db_obj_mapping, self.id_field)
+        except DataError as err:


Hi @MTachon, why invalid data are reaching the database session? I'm expecting a data validation layer should be implemented at the interface level since pygeoapi already should be able to serve the schema for the collection. If the schema exposed is not validated than the endpoint should just emit a 422 response code with the validation error

Hi @francbartoli, just checked for valid GeoJSON before that point. Was not sure if instantiating db_obj_mapping with wrong data passed to the table model would throw an error. No error came until the session.commit() statement. Not sure how it should be implemented. I also think it would be better with a data validation layer using schemas. As for now, I do not think pygeoapi serves collections' schemas, and in order to do so, we would need to implement a schema generation function for all providers. Right now, I am working on adding transactions support for PostgreSQLProvider, and I thought I could let PostgreSQL/PostGIS throw an error if the POST-ed data is invalid and catch the exception. ElasticsearchProvider supports transactions but is not using schemas validation.

@MTachon how does the specification handle the schema on the OpenAPI document? I'm expecting, as a developer, that the data model is exposed on the swagger page, is somehow the queryables endpoint designed to do that job?

in this issue opengeospatial/ogcapi-features#424 the mutation schema is mentioned for that purpose, @tomkralidis any clue why it is completely missing from Part-4 now?

@francbartoli , I think you will find your answer there: opengeospatial/ogcapi-features#740

Thanks @MTachon, this is a WIP indeed. Can you please put a comment that this would be abstracted as soon as this part of the specification will be released and implemented in pygeoapi?

Hi @francbartoli, just checked for valid GeoJSON before that point. Was not sure if instantiating db_obj_mapping with wrong data passed to the table model would throw an error. No error came until the session.commit() statement. Not sure how it should be implemented. I also think it would be better with a data validation layer using schemas. As for now, I do not think pygeoapi serves collections' schemas, and in order to do so, we would need to implement a schema generation function for all providers. Right now, I am working on adding transactions support for PostgreSQLProvider, and I thought I could let PostgreSQL/PostGIS throw an error if the POST-ed data is invalid and catch the exception. ElasticsearchProvider supports transactions but is not using schemas validation.

@MTachon FYI schema generation was implemented in #1022

@tomkralidis , thanks for pointing it out! I was not aware of it, I will have a look.

MTachon · 2023-06-20T11:15:17Z

In the test_manage_collection_items_postgresql_create function, I create a new item in the database, and then I want to get back this newly created feature by using the feature's URI. I get a connection error (Failed to establish a new connection: [Errno 111] Connection refused) in the requests.get call, in the main.yml workflow. @tomkralidis , @justb4 , @webb-ben , @francbartoli , any ideas why?

My mistake, no server is running here... Going to use methods from API instead, and extract the feature id from the returned URI in the Location header to send as parameter

…th APIRequest.get_request_headers

…lt in _load_and_prepare_item

…l-transactions

…rsion

…tgresql_replace, and start get_schema for PostgreSQL

…l-transactions

JakobMiksch · 2023-07-13T20:45:40Z

Thanks @MTachon for working on this.
Meanwhile QGIS 3.32.0 supports OGC API Features Part 4 as client. I just tried it with this PR, but it does not work yet (obviously, because it's still a draft :) )
I can offer to test more with QGIS and provide feedback. Just ping me if this is needed.

MTachon · 2023-08-21T09:08:39Z

Hi @JakobMiksch . I have worked on other projects lately, so did not have much time to work on this PR. But now I will have more time again. I will let you know as I advance if some things can be tested :) Thanks!

…l-transactions

JakobMiksch · 2023-08-28T06:37:13Z

@MTachon alright, I am happy to test any time :)

francbartoli · 2023-09-17T20:07:45Z

Hi @MTachon, what is the status of this?

MTachon · 2023-09-17T21:08:19Z

Hi @francbartoli, I am not really satisfied by the start of my implementation of schemas... I would like like to use schema generation to filter out invalid data, and not do exception handling around the database session, as discussed above. I felt like schema generation was too much for the scope of this PR, and wanted to dedicate new PR for that. I started PR #1349 on GeoJSON schemas generation. It is an attempt to use pydantic models to provide a standard way for data validation and schema generation in the plugin providers implementation. I see that PR #1349 would required an upgrade of the pydantic version, but that should not be too much work, as addressed in issue #1341. I also get CI issues from type annotations and features not yet available in python 3.7. Upgrade of the python version in CI were discussed in issue #1176. If we do not want to upgrade the python version, I need to do some refactoring.

github-actions · 2024-03-10T21:38:41Z

As per RFC4, this Pull Request has been inactive for 90 days. In order to manage maintenance burden, it will be automatically closed in 7 days.

github-actions · 2024-03-24T03:18:22Z

As per RFC4, this Pull Request has been closed due to there being no activity for more than 90 days.

MTachon added 5 commits April 13, 2023 13:58

refactor: make _load_and_prepare_item more compact in providers' base…

412d53f

… class

Merge remote-tracking branch 'pygeoapi/master' into ogcapif-postgresq…

fc7c853

…l-transactions

Merge remote-tracking branch 'pygeoapi/master' into crs-transform-geo…

35d046d

…json

test: database table and config for transactions with PostgreSQLProvider

033ff70

MTachon marked this pull request as draft June 15, 2023 13:40

MTachon added 15 commits June 15, 2023 15:42

Remove id/identifier from json_data

eb17840

test: function to test item creation

466dd08

Fix typo in table creation statement

bfb49f2

Check that item to create has a 'type' member with 'Feature' as value

328ec59

Add 'create' method to PostgreSQLProvider

66960a4

Merge branch 'crs-transform-geojson' into ogcapif-postgresql-transact…

395c714

…ions

Add 'replace' action, crs support, update 'create' with SRID info for…

032eefb

… PostgreSQL

Remove typing annotation not supported by python version

86713b0

crs_transform in providers implementation instead, fixes circular imp…

95d08e1

…orts

Fix Find_SRID call

0e767bb

Remove crs_transform_func parameter from _load_and_prepare_item calls

44f384e

Add 'create' action to _load_and_prepare_item call

429c3f9

Refresh database object to get updated identifier

d534b9f

Use requests instead of urllib in test function

c4e8165

Test with 'verify=False' in requests.get

1286fd7

MTachon added 4 commits June 20, 2023 11:08

Testing with get_collection_item call

8b04ba2

Fix KeyError

87f5512

Change wrong hasattr assert statement

8296f6c

Add exception handling when attempting to create a new feature from i…

8fcf509

…nvalid data

francbartoli requested changes Jun 20, 2023

View reviewed changes

MTachon added 2 commits June 20, 2023 14:08

Use only API methods to get back newly created features

4827f46

Content-Crs as header and not parameter in mock_request

e9947a8

MTachon added 9 commits June 21, 2023 11:06

Change header name Content-Crs -> HTTP_CONTENT_CRS to make it work wi…

61a6dce

…th APIRequest.get_request_headers

Fix tinydb test and don't remove id in properties.identifier by defau…

b42969e

…lt in _load_and_prepare_item

Merge remote-tracking branch 'pygeoapi/master' into ogcapif-postgresq…

66be0c7

…l-transactions

Add schemas.py and move SchemaType definition to the file

6ebe573

docs: add docstrings schemas.py and fix type annotation for python ve…

b1245b4

…rsion

Add test_schemas.py

441ad04

test: refactor test_get_geometry_schema

71dafd3

Extend test for 'create' action, add test_manage_collection_items_pos…

874915d

…tgresql_replace, and start get_schema for PostgreSQL

Merge remote-tracking branch 'pygeoapi/master' into ogcapif-postgresq…

f849a4b

…l-transactions

Merge remote-tracking branch 'pygeoapi/master' into ogcapif-postgresq…

d591812

…l-transactions

MTachon mentioned this pull request Oct 4, 2023

GeoJSON schemas generation #1349

Closed

2 tasks

github-actions bot added the stale Issue marked stale by stale-bot label Mar 10, 2024

github-actions bot closed this Mar 24, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] OGC API Features - Part 4 / Support for PostgreSQLProvider #1266

[WIP] OGC API Features - Part 4 / Support for PostgreSQLProvider #1266

MTachon commented Jun 15, 2023 •

edited

Loading

MTachon commented Jun 20, 2023

francbartoli Jun 20, 2023

MTachon Jun 20, 2023 •

edited

Loading

francbartoli Jun 20, 2023

francbartoli Jun 20, 2023

MTachon Jun 21, 2023

francbartoli Jun 21, 2023

tomkralidis Jun 23, 2023

MTachon Jun 23, 2023

MTachon commented Jun 20, 2023 •

edited

Loading

JakobMiksch commented Jul 13, 2023

MTachon commented Aug 21, 2023

JakobMiksch commented Aug 28, 2023

francbartoli commented Sep 17, 2023

MTachon commented Sep 17, 2023

github-actions bot commented Mar 10, 2024

github-actions bot commented Mar 24, 2024

[WIP] OGC API Features - Part 4 / Support for PostgreSQLProvider #1266

[WIP] OGC API Features - Part 4 / Support for PostgreSQLProvider #1266

Conversation

MTachon commented Jun 15, 2023 • edited Loading

Overview

Additional Information

Contributions and Licensing

MTachon commented Jun 20, 2023

francbartoli Jun 20, 2023

Choose a reason for hiding this comment

MTachon Jun 20, 2023 • edited Loading

Choose a reason for hiding this comment

francbartoli Jun 20, 2023

Choose a reason for hiding this comment

francbartoli Jun 20, 2023

Choose a reason for hiding this comment

MTachon Jun 21, 2023

Choose a reason for hiding this comment

francbartoli Jun 21, 2023

Choose a reason for hiding this comment

tomkralidis Jun 23, 2023

Choose a reason for hiding this comment

MTachon Jun 23, 2023

Choose a reason for hiding this comment

MTachon commented Jun 20, 2023 • edited Loading

JakobMiksch commented Jul 13, 2023

MTachon commented Aug 21, 2023

JakobMiksch commented Aug 28, 2023

francbartoli commented Sep 17, 2023

MTachon commented Sep 17, 2023

github-actions bot commented Mar 10, 2024

github-actions bot commented Mar 24, 2024

MTachon commented Jun 15, 2023 •

edited

Loading

MTachon Jun 20, 2023 •

edited

Loading

MTachon commented Jun 20, 2023 •

edited

Loading