Implement geo upload #47

mtiessen1175 · 2023-11-30T13:49:57Z

references #44

Implement the structure for the upload mechanism in the volume-edit view. Allow the upload of both - browsing layers and context layers - i.e. upload of geoTIFF and (iFDO) metadata, linkage of WMS.

Additional layers can be added in the volume edit view
An additional layer can be a georeferenced file (geoTIFF) or a link to a WMS source
- geoTIFF upload done (but should be tested more thoroughly by adding more example .tif files, e.g. a geoTIFF using the ModelTransformationTag. Corresponding code implemented here. Update: Tests fully cover this case; see commits cafb808 and bb6b2c6)
- WMS
Layers can be ordered in the volume edit view
Each layer can either be a "browsing layer" (Geospatial browsing visualisation #44) or a "context layer" or both
The filter-map shows all configured "browsing" layers
Option to upload the full image extent as metadata (also from iFDO, see Geospatial context fusion display #45)

associated PR's:

#725 in biigle/core
#761 in biigle/core

…rmat.

src/Jobs/TileSingleOverlay.php

mzur

When I upload a GeoTIFF, I get JS errors (see below). The upload seems to succeed, though.

When I upload the DEM_Rest_2014 file (you sent me?), I can't find it at the same location than QGIS tells me. Also, the tile files produced by Vips are binary (i.e. only #000 and #FFF colors).

Do you have a WMS for me to test?

14:31:59.876 [Vue warn]: $listeners is readonly.

found in

---> <Portal>
       <Tab>
         <Tabs>
           <Root> volumes.js:3924:15

14:32:01.063 [Vue warn]: $attrs is readonly.

found in

---> <Portal>
       <Tab>
         <Tabs>
           <Root> 2 volumes.js:3924:15

14:32:01.094 Uncaught (in promise) TypeError: callback.call is not a function
    finally http://localhost:8000/assets/scripts/main.js?1729685278:133407
    promise callback*./node_modules/vue-resource/dist/vue-resource.esm.js/p$1.then http://localhost:8000/assets/scripts/main.js?1729685278:133394
    finally http://localhost:8000/assets/scripts/main.js?1729685278:133406
    uploadGeoTiff http://localhost:8000/vendor/geo/scripts/volumes.js?1729686705:15985
    invokeWithErrorHandling http://localhost:8000/assets/scripts/main.js?1729685278:136516
    invoker http://localhost:8000/assets/scripts/main.js?1729685278:136841
    _wrapper http://localhost:8000/assets/scripts/main.js?1729685278:142244
    add$1 http://localhost:8000/assets/scripts/main.js?1729685278:142248
    updateListeners http://localhost:8000/assets/scripts/main.js?1729685278:136873
    updateDOMListeners http://localhost:8000/assets/scripts/main.js?1729685278:142278
    invokeCreateHooks http://localhost:8000/assets/scripts/main.js?1729685278:140759
    createElm http://localhost:8000/assets/scripts/main.js?1729685278:140646
    createChildren http://localhost:8000/assets/scripts/main.js?1729685278:140743
    createElm http://localhost:8000/assets/scripts/main.js?1729685278:140644
    createChildren http://localhost:8000/assets/scripts/main.js?1729685278:140743
    createElm http://localhost:8000/assets/scripts/main.js?1729685278:140644
    createChildren http://localhost:8000/assets/scripts/main.js?1729685278:140743
    createElm http://localhost:8000/assets/scripts/main.js?1729685278:140644
    patch http://localhost:8000/assets/scripts/main.js?1729685278:141167
    _update http://localhost:8000/assets/scripts/main.js?1729685278:138608
    updateComponent http://localhost:8000/assets/scripts/main.js?1729685278:138729
    get http://localhost:8000/assets/scripts/main.js?1729685278:139143
    Watcher http://localhost:8000/assets/scripts/main.js?1729685278:139132
    mountComponent http://localhost:8000/assets/scripts/main.js?1729685278:138736
    $mount http://localhost:8000/assets/scripts/main.js?1729685278:143742
    $mount http://localhost:8000/assets/scripts/main.js?1729685278:146656
    init http://localhost:8000/assets/scripts/main.js?1729685278:137788
    createComponent http://localhost:8000/assets/scripts/main.js?1729685278:140668
    createElm http://localhost:8000/assets/scripts/main.js?1729685278:140615
    createChildren http://localhost:8000/assets/scripts/main.js?1729685278:140743
    createElm http://localhost:8000/assets/scripts/main.js?1729685278:140644
    patch http://localhost:8000/assets/scripts/main.js?1729685278:141167
    _update http://localhost:8000/assets/scripts/main.js?1729685278:138608
    updateComponent http://localhost:8000/assets/scripts/main.js?1729685278:138729
    get http://localhost:8000/assets/scripts/main.js?1729685278:139143
    Watcher http://localhost:8000/assets/scripts/main.js?1729685278:139132
    mountComponent http://localhost:8000/assets/scripts/main.js?1729685278:138736
main.js:133407:14

mzur · 2024-10-23T12:26:03Z

src/resources/assets/js/volumes/api/geoOverlays.js

+    getGeoTiffOverlayUrlTemplate: {
+        method: 'GET',
+        url: 'api/v1/volumes{/id}/geo-overlays/geotiff/url',
+    },


Suggested change

getGeoTiffOverlayUrlTemplate: {

method: 'GET',

url: 'api/v1/volumes{/id}/geo-overlays/geotiff/url',

},

mzur · 2024-10-23T12:32:32Z

src/resources/assets/js/volumes/components/geotiffOverlayForm.vue

            let data = new FormData();
            data.append('geotiff', event.target.files[0]);
            data.append('volumeId', this.volumeId);
            this.upload(data)
                .then(this.handleSuccess, this.handleError)
-                .finally(this.finishLoading);
+                .finally(this.$emit('upload', false));


Suggested change

.finally(this.$emit('upload', false));

.finally(() => this.$emit('upload', false));

mzur · 2024-10-23T12:32:44Z

src/resources/assets/js/volumes/components/webmapOverlayForm.vue

        let data = new FormData();
        data.append('url', event.target[0].value); 
        data.append('volumeId', this.volumeId);
        geoApi.saveWebMap({id: this.volumeId}, data)
          .then(this.handleSuccess, this.handleError)
-          .finally(this.finishLoading)
+          .finally(this.$emit('upload', false))


Suggested change

.finally(this.$emit('upload', false))

.finally(() => this.$emit('upload', false))

mzur · 2024-10-23T12:33:18Z

src/resources/assets/js/volumes/components/webmapOverlayForm.vue

@@ -11,13 +11,13 @@ export default {
  },
  mixins: [
        LoaderMixin,


This doesn't need the mixin any more?

mzur · 2024-10-23T12:33:34Z

src/resources/assets/js/volumes/components/geotiffOverlayForm.vue

This doesn't need the LoaderMixin any more?

mzur · 2024-10-23T12:54:58Z

src/resources/views/volumesEditRight.blade.php

The panel would look better if you disable the margin-bottom of the <ul> or <table> at the bottom. Otherwise there is a strange gap.

mzur · 2024-10-23T12:57:50Z

src/resources/assets/js/volumes/components/overlayTable.vue

+<template>
+    <div class="table-responsive">
+        <table v-if="overlays.length !== 0" class="table table-sm" v-cloak>
+            <caption><b><slot name="title"></slot></b></caption>


No need for a caption, really. The panel in the volume edit view already has a title.

mzur · 2024-10-23T12:59:16Z

src/resources/views/volumes/edit/geotiffOverlayForm.blade.php

+    <div class="alert alert-success" v-if="success" v-cloak>
+        The geoTIFF-file was successfully uploaded.
+    </div>


IMO the highlighted element in the table below is enough of a success indication. No need for an additional message.

mzur · 2024-10-23T12:59:25Z

src/resources/views/volumes/edit/webmapOverlayForm.blade.php

+    <div class="alert alert-success" v-if="success" v-cloak>
+        The web-map-service was successfully embedded.
+    </div>


Same than above.

mzur · 2024-10-23T13:00:10Z

src/resources/assets/js/volumes/components/overlayItem.vue

+            </span>
+        </td>
+        <td>
+            <button type="button" class="close" :title="title" v-on:click="remove" v-once><span aria-hidden="true">&times;</span></button>


Can you please center this in the column?

mtiessen1175 · 2024-10-23T14:01:37Z

When I upload a GeoTIFF, I get JS errors (see below). The upload seems to succeed, though.

I get it as well, I'll have a look where it comes from.

When I upload the DEM_Rest_2014 file (you sent me?), I can't find it at the same location than QGIS tells me.

This is most certainly due to the transformation error in the proj4php package, as discussed here.

Also, the tile files produced by Vips are binary (i.e. only #000 and #FFF colors).

Yes, I created a normalization of the color-band (as geoTIFF's often contain more color-information that can be set/adjusted manually by the user). So the discussed solution for now was to normalize the color-band to 0 - 255 in case it differs from that range (see code).

mzur · 2024-10-23T14:36:11Z

This is most certainly due to the transformation error in the proj4php package, as discussed #47 (comment).

Did you find the overlay? Where do I have to look? 😃

Yes, I created a normalization of the color-band (as geoTIFF's often contain more color-information that can be set/adjusted manually by the user). So the #47 (comment) solution for now was to normalize the color-band to 0 - 255 in case it differs from that range (see code).

I saw that but I expected this to produce something else than the black/white image. There is no 0-255 range in the tiled image, just 0 and 255.

mtiessen1175 · 2024-10-23T14:45:01Z

Did you find the overlay? Where do I have to look? 😃

Ahh, yeah it is a fairly small example-TIFF! The coordinates in EPSG:3857 are: -325165.2, 7779955.1 (Loch Davan close to Dinnet, Scotland)

I saw that but I expected this to produce something else than the black/white image. There is no 0-255 range in the tiled image, just 0 and 255.

I see, so I should check what goes wrong in the normalization.

mtiessen1175 · 2024-10-28T10:15:21Z

I saw that but I expected this to produce something else than the black/white image. There is no 0-255 range in the tiled image, just 0 and 255.

I see, so I should check what goes wrong in the normalization.

The formula used for image normalization should be working. It seems that the derived min- and max-values from libvips are not providing the correct min/max colour-band values as I expected them to be (i.e. they are different to what I see in the QGIS colour settings). Moreover, I tested with two different geoTIFF's and surprisingly the min/max of the two geoTIFF's were identical (min: -3.4028230607371E+38, max: 162.15870666504).
So I need to find a different approach to get the min/max colour-value information from the image.

mzur · 2024-10-31T08:56:12Z

You should try it with the Vips CLI too. If this does not show the expected values you can ask in the Vips repo. The maintainer is extremely responsive and helpful if you provide a good description of the problem and an example file.

mtiessen1175 · 2024-11-04T15:42:20Z

You should try it with the Vips CLI too. If this does not show the expected values you can ask in the Vips repo. The maintainer is extremely responsive and helpful if you provide a good description of the problem and an example file.

I tried with the CLI and get the correct max() value but an incorrect min(). This must be due to the NoData values of the geoTIFF images (the black frame around the geoTIFF when converting it to an image). So the min() always returns this value (e.g. -99999) instead of the desired min-value (e.g. -4123.31).

mzur · 2024-11-05T09:02:05Z

(please use @mzur, otherwise I don't get notified of activity here)

If this can be fixed by creating the geoTIFF in any other way then we could add an explanation to the manual and just expect geoTIFFs with the correct format. Then there is no need for special handling of edge cases (for now).

mzur · 2024-11-06T09:43:17Z

I tested the WMS now and it works fine.

I noticed an error on the geo map of a volume (not the map in the filter modal). When I open the label trees tab there, it says Error in event handler for "sidebar.toggle": "TypeError: this.map is undefined".

…nated single arithmetic operations.

mtiessen1175 · 2024-11-12T12:02:49Z

@mzur,

If this can be fixed by creating the geoTIFF in any other way then we could add an explanation to the manual and just expect geoTIFFs with the correct format. Then there is no need for special handling of edge cases (for now).

I've implemented the color normalization such that NoData values of -99999 will get excluded in order to calculate the true min (see conversation on libvips discussion). I therefore would add an excerpt to the manual stating that color-band normalization should either be done beforehand or the NoData values of the geoTIFF should be set to -99999 in order to be handled correctly by BIIGLE.

mzur · 2024-11-12T14:22:09Z

Sounds good 👍

mtiessen1175 · 2024-11-12T14:59:42Z

@mzur,

I noticed an error on the geo map of a volume (not the map in the filter modal). When I open the label trees tab there, it says Error in event handler for "sidebar.toggle": "TypeError: this.map is undefined".

That's fixed now.

When I upload a GeoTIFF, I get JS errors (see below). The upload seems to succeed, though.
14:31:59.876 [Vue warn]: $listeners is readonly.

found in

---> <Portal>
       <Tab>
         <Tabs>
           <Root> volumes.js:3924:15 

Regarding the latter, I tried to find where the warning is coming from by debugging the code, but I couldn't figure out what causes it. It is still unclear to me...

where the readonly is allegedly violated and
to what kind of object the attributes refer to

I looked into the whole upload process of the geoTIFF files, comprised of:

and

geoTiffOverlayController.php

commenting lines step by step to see where the warning stems from, but it's still not obvious to me. Do you have any idea or instinct what could cause such warning?

mtiessen1175 added 20 commits November 22, 2023 14:17

include basic volumesEditRight mixin in geo

c0aa459

add basic geoOverlay model and controller

a71b958

Use old geo-overlay structure

0746ebe

fill geoOverlay model with functions

444cbd1

configure storage-disk for geo-overlays

d676299

fix uiv Tab(s) import

d4bcc71

rename geoOverlay to GeoOverlay

a073726

Add routes and controller structure for volume-geo-overlay

34c92bf

added some missing imports

2b4ebcd

add simple tabs for file upload

947f673

Merge branch 'browsing_vis' into geo_edit

2638389

commit production ready files

95706ba

create new geoTIFF component

d686aa2

include production files

ee70bfa

edit text and names of overlay upload-tabs

6a73c5e

Modified upload function.

fb78c78

Include geotiff.js for testing

89af5cf

added transform-coordinates package to transform any CRS to WGS 84 fo…

df0ad5c

…rmat.

Merge branch 'browsing_vis' into geo_upload

0e772f3

add production files

5761319

Base automatically changed from browsing_vis to master December 8, 2023 14:43

mtiessen1175 added 9 commits December 11, 2023 15:20

remove JS geotiff packages and code

eb718c8

pass "raw" geotiff-data to backend

dda9101

Add existing code for geo-overlay retrieval and deletion + test

9df6bec

delete the removal of geo-overlays-table

351c681

add test for VolumeGeoOverlayController

fdc0e79

Linting and fixing files

7004442

Test native php exif function

c2d9cfd

test native php function for identifying geotiff-tagnames

38f1aaf

include php exif dependency

52823b0

add vips functions for testing colour-space conversion

66f67e6

mtiessen1175 commented Oct 7, 2024

View reviewed changes

src/Jobs/TileSingleOverlay.php Outdated Show resolved Hide resolved

Added color-band normalization for geoTIFF's

644d70d

mtiessen1175 requested a review from mzur October 12, 2024 10:08

Fixed image selection description.

0c4efb7

mtiessen1175 marked this pull request as ready for review October 17, 2024 15:14

mzur reviewed Oct 23, 2024

View reviewed changes

mtiessen1175 added 3 commits October 23, 2024 16:17

fix $emit and remove unnecessary mixin.

952cd78

Fix table appearance.

99181d2

remove unnecessary success-messages and api route.

ddf61fe

Fix not existing "loading" variable

b7a13ea

re-write normalization formula

6bd7301

Commented color-band-normalization for testing

b1b0de9

reapply color-band normalization

296f2a9

mtiessen1175 added 2 commits November 12, 2024 12:33

Exclude NoData values from geoTIFF's for calculating correct min

bd3ee4e

Improve color-normalization by using vips_linear() instead of concate…

23c2ae1

…nated single arithmetic operations.

Fixed "this.map is undefined" error on sidebar-toggle

7a48b90

Remove unnecessary variable

8693f2d

Add color-band normalization note to user-manual.

19d2251

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement geo upload #47

Implement geo upload #47

mtiessen1175 commented Nov 30, 2023 •

edited

Loading

mzur left a comment

mzur Oct 23, 2024

mzur Oct 23, 2024

mzur Oct 23, 2024

mzur Oct 23, 2024

mzur Oct 23, 2024

mzur Oct 23, 2024

mzur Oct 23, 2024

mzur Oct 23, 2024

mzur Oct 23, 2024

mzur Oct 23, 2024

mtiessen1175 commented Oct 23, 2024

mzur commented Oct 23, 2024

mtiessen1175 commented Oct 23, 2024

mtiessen1175 commented Oct 28, 2024

mzur commented Oct 31, 2024

mtiessen1175 commented Nov 4, 2024

mzur commented Nov 5, 2024

mzur commented Nov 6, 2024

mtiessen1175 commented Nov 12, 2024

mzur commented Nov 12, 2024

mtiessen1175 commented Nov 12, 2024

	.finally(this.$emit('upload', false));
	.finally(() => this.$emit('upload', false));

	.finally(this.$emit('upload', false))
	.finally(() => this.$emit('upload', false))

Implement geo upload #47

Are you sure you want to change the base?

Implement geo upload #47

Conversation

mtiessen1175 commented Nov 30, 2023 • edited Loading

mzur left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mtiessen1175 commented Oct 23, 2024

mzur commented Oct 23, 2024

mtiessen1175 commented Oct 23, 2024

mtiessen1175 commented Oct 28, 2024

mzur commented Oct 31, 2024

mtiessen1175 commented Nov 4, 2024

mzur commented Nov 5, 2024

mzur commented Nov 6, 2024

mtiessen1175 commented Nov 12, 2024

mzur commented Nov 12, 2024

mtiessen1175 commented Nov 12, 2024

mtiessen1175 commented Nov 30, 2023 •

edited

Loading