diff --git a/README.md b/README.md index 6c22a507..d7a30067 100755 --- a/README.md +++ b/README.md @@ -1,15 +1,124 @@ -# Bloom Project +![banner](images/banner.png) -This project is maintained by D4G in order to gather vessels data. -A cron job is launched every 15min, does calls to API and save the data in a Postgresql database. +## What is Trawl Watch -## About directory architecture -The domain directory ... -The infra directory ... +**[Trawl Watch](https://twitter.com/TrawlWatch)** is an initiative launched by the **[Bloom Association](https://www.bloomassociation.org/en/)** to track and expose the most destructive fishing vessels. Inspired by L’[Avion de Bernard](https://www.instagram.com/laviondebernard/), which monitors the movements of private jets, **Trawl Watch** aims to make visible the impact of these massive trawlers on our oceans. These vessels, often referred to as _mégachalutiers_, deploy gigantic nets that can engulf marine life from the surface down to the ocean floor. The consequences are both ecological—as they devastate crucial nursery and breeding areas for marine animals—and social, as they deprive artisanal fishermen of a healthy marine ecosystem. The solution proposed by **Bloom** is to dismantle these industrial fishing ships and redistribute their quotas to small-scale fishers. A petition has been launched, and **Bloom** continues to track these megatrawlers while awaiting action from European institutions. +**Did you know that, in Europe, the largest fishing vessels, which represent 1% of the fleet, catch half of the fish?** These factory-vessels can measure up to 144 meters in length and catch 400,000 kilos of fish per day! This is as much as 1,000 small-scale fishing vessels in one day at sea. -More information can be found there : -1. [Database initialisation and versioning](./documentation/database_init.md) -2. [Development environment](./documentation/dev_env.md) -3. [Useful SQL examples](./documentation/sql_examples.md) -4. [Data models](#todo) \ No newline at end of file +**These veritable sea monsters are devastating Europe’s biodiversity and coastlines.** It is important to measure the scale of the damage: about 20 of these factory-vessels can obliterate hundreds of thousands of marine animals and biodiversity treasures in one day, including in the so-called ‘Marine Protected Areas’ of French territorial waters, which are not protected at all. + +## What is Bloom Association + +**BLOOM** is a non-profit organization founded in 2005 that works to preserve the marine environment and species from unnecessary destruction and to increase social benefits in the fishing sector. **BLOOM** wages awareness and advocacy campaigns in order to accelerate the adoption of concrete solutions for the ocean, humans and the climate. **BLOOM** carries out scientific research projects, independent studies and evaluations that highlight crucial and unaddressed issues such as the financing mechanisms of the fishing sector. **BLOOM**’s actions are meant for the general public as well as policy-makers and economic stakeholders. + +## Installing Trawl Watch with `poetry` + +### Prerequisites: + +1. Python (≥ `3.10`) installed on your system. +2. Ensure [Docker](https://docs.docker.com/get-docker/) is installed. +3. Ensure you have `poetry` installed. If not, you can install them using `pip`. + +```bash +pip install poetry +``` + +### Steps: + +1. **Clone the GitHub Repository:** + + Clone the GitHub repository you want to install locally using the `git clone` command. + + ```bash + git clone https://github.com/dataforgoodfr/12_bloom.git + ``` + +2. **Navigate to the Repository Directory:** + + Use the `cd` command to navigate into the repository directory. + + ```bash + cd 12_bloom/ + ``` + +3. **Configure `poetry` to create a Virtual Environment inside the project:** + + Ensure that poetry will create a `.venv` directory into the project with the command: + + ```bash + poetry config virtualenvs.in-project true + ``` + +4. **Install Project Dependencies using `poetry`:** + + Use `poetry` to install the project dependencies. + + ```bash + poetry install + ``` + + This will read the `pyproject.toml` file in the repository and install all the dependencies specified. + +5. **Make sure everything is all right using `poetry env info`:** + + ```bash + poetry env info + ``` + + It should looks something likes: + + ```bash + Virtualenv + Python: 3.11.2 + Implementation: CPython + Path: /home/guillaume/12_bloom/.venv + Executable: /home/guillaume/12_bloom/.venv/bin/python + Valid: True + + System + Platform: linux + OS: posix + Python: 3.11.2 + Path: /usr + Executable: /usr/bin/python3.11 + ``` + +6. **Activate the Virtual Environment:** + + Activate the virtual environment to work within its isolated environment. + + On Unix or MacOS: + + ```bash + poetry shell + ``` + +7. + +### Once you're done working with the project, deactivate the virtual environment. + +```bash +deactivate +``` + +## Documentation + +[Dendron](https://marketplace.visualstudio.com/items?itemName=dendron.dendron) is a powerful Visual Studio Code, or [VSCodium](https://vscodium.com/), extension designed to streamline and enhance the documentation process. With **Dendron**, documenting projects becomes intuitive and efficient, thanks to its hierarchical note-taking system. Users can organize their documentation into a tree-like structure, making it easy to navigate and manage. The extension offers robust features such as bidirectional linking, which allows for seamless navigation between related notes, and support for Markdown formatting, enabling users to create rich and visually appealing documentation. Additionally, **Dendron** provides powerful search functionality, enabling users to quickly locate specific information within their documentation vault. Overall, **Dendron** empowers developers, writers, and teams to create comprehensive and well-organized documentation, facilitating better knowledge management and collaboration. The documentation is locaed inside the `./docs/notes` directory. + +Here's some basic Dendron shortcuts: + +| Shortcut | Description | +| -------------- | ------------ | +| `Cmd/Ctrl + L` | Lookup notes | + +## More information can be found there + +1. [Database initialisation and versioning](./docs/database.initialisation.md) +2. [Development environment](./docs/development.environment.md) +3. [Useful SQL examples](./docs/sql.examples.md) +4. [Data models](#todo) + +## FAQ + +#todo diff --git a/data/_info_subset_data.docx b/data/_info_subset_data.docx new file mode 100644 index 00000000..5d89b700 Binary files /dev/null and b/data/_info_subset_data.docx differ diff --git a/data/vessels_subset_02022024.csv b/data/vessels_subset_02022024.csv new file mode 100644 index 00000000..a7451184 --- /dev/null +++ b/data/vessels_subset_02022024.csv @@ -0,0 +1,11 @@ +"id","country_iso3","cfr","IMO","registration_number","external_marking","ship_name","ircs","mmsi","loa","type","mt_activated" +621,"ESP","27678 ESP","8516043","3VI-5-1-15","3VI-5-1-15","PARADANTA PRIMERO","EAAB",224378000,"58.38",">25m",False +784,"FRA","FRA000919999","9828936","919999","CC919999","SCOMBRUS","FLVN",226347000,"81.37",">25m",False +795,"FRA","NLD198300393","8224406","716900","FC716900","PRINS BERNHARD","FKHH",227302000,"88.24",">25m",False +835,"FRA","FRA000932206","9741097","932206","CC932206","GEVRED","FIUO",228066900,"77",">25m",False +863,"FRA","FRA000914221","9352913","914221","MA914221","VENT DU NORD II","FMEU",228215800,"41.92",">25m",False +920,"GBR",NULL,"9877389","GY127",NULL,"HENDRIKA JACOBA",NULL,232031183,NULL,"Demersal seiners",False +1194,"NLD","FRA000544858","7928835","FRA000544858","SL-9","JOHANNA","PHOE",244938000,"34.3",">25m",False +1280,"NLD","NLD201001155","9599731","NLD201001155","UK-153","LUB SENIOR","PCGC",246749000,"28.5",">25m",False +1431,"POL","IRL000I13000","9204556","PL02315","GDY-151","ANNELIES ILENA","SPG4839",261084090,"144.6",">25m",False +1507,"PRT","PRT000023094","9297694","PTAVE-117345-N","PTAVE-117345-N","FRANÇA MORTE","CUFE7",263581000,"73.8",">25m",False diff --git a/docs/.gitignore b/docs/.gitignore new file mode 100644 index 00000000..77ad2f5b --- /dev/null +++ b/docs/.gitignore @@ -0,0 +1,6 @@ +node_modules +.dendron.* +build +seeds +.next +pods/service-connections \ No newline at end of file diff --git a/docs/dendron.code-workspace b/docs/dendron.code-workspace new file mode 100644 index 00000000..77e7f32e --- /dev/null +++ b/docs/dendron.code-workspace @@ -0,0 +1,37 @@ +{ + "folders": [ + { + "path": "notes", + "name": "docs" + } + ], + "settings": { + "dendron.rootDir": ".", + "files.autoSave": "onFocusChange", + "pasteImage.path": "${currentFileDir}/assets/images", + "pasteImage.prefix": "/", + "markdown-preview-enhanced.enableWikiLinkSyntax": true, + "markdown-preview-enhanced.wikiLinkFileExtension": ".md", + "editor.snippetSuggestions": "inline", + "editor.suggest.snippetsPreventQuickSuggestions": false, + "editor.suggest.showSnippets": true, + "editor.tabCompletion": "on", + "dendron.enableSelfContainedVaultWorkspace": true + }, + "extensions": { + "recommendations": [ + "dendron.dendron", + "dendron.dendron-paste-image", + "dendron.dendron-markdown-shortcuts", + "redhat.vscode-yaml" + ], + "unwantedRecommendations": [ + "dendron.dendron-markdown-links", + "dendron.dendron-markdown-notes", + "dendron.dendron-markdown-preview-enhanced", + "shd101wyy.markdown-preview-enhanced", + "kortina.vscode-markdown-notes", + "mushan.vscode-paste-image" + ] + } +} diff --git a/docs/dendron.yml b/docs/dendron.yml new file mode 100644 index 00000000..19064bf0 --- /dev/null +++ b/docs/dendron.yml @@ -0,0 +1,108 @@ +version: 5 +dev: + enablePreviewV2: true + enableSelfContainedVaults: true +commands: + lookup: + note: + selectionMode: extract + confirmVaultOnCreate: true + vaultSelectionModeOnCreate: smart + leaveTrace: false + bubbleUpCreateNew: true + fuzzThreshold: 0.2 + randomNote: {} + insertNoteLink: + aliasMode: none + enableMultiSelect: false + insertNoteIndex: + enableMarker: false + copyNoteLink: + aliasMode: title + templateHierarchy: template +workspace: + vaults: + - + fsPath: . + selfContained: true + name: docs + journal: + dailyDomain: daily + name: journal + dateFormat: y.MM.dd + addBehavior: childOfDomain + scratch: + name: scratch + dateFormat: y.MM.dd.HHmmss + addBehavior: asOwnDomain + task: + name: task + dateFormat: y.MM.dd + addBehavior: asOwnDomain + statusSymbols: + '': ' ' + wip: w + done: x + assigned: a + moved: m + blocked: b + delegated: l + dropped: d + pending: 'y' + taskCompleteStatus: + - done + - x + prioritySymbols: + H: high + M: medium + L: low + todoIntegration: false + createTaskSelectionType: selection2link + graph: + zoomSpeed: 1 + createStub: false + enableAutoCreateOnDefinition: false + enableXVaultWikiLink: false + enableRemoteVaultInit: true + enableUserTags: true + enableHashTags: true + workspaceVaultSyncMode: noCommit + enableAutoFoldFrontmatter: false + enableEditorDecorations: true + maxPreviewsCached: 10 + maxNoteLength: 204800 + enableFullHierarchyNoteTitle: false + enablePersistentHistory: false +preview: + enableFMTitle: true + enableNoteTitleForLink: true + enableFrontmatterTags: true + enableHashesForFMTags: false + enablePrettyRefs: true + enableKatex: true + automaticallyShowPreview: false +publishing: + enableFMTitle: true + enableNoteTitleForLink: true + enablePrettyRefs: true + enableKatex: true + copyAssets: true + siteHierarchies: + - root + writeStubs: false + siteRootDir: docs + seo: + title: Dendron + description: Personal Knowledge Space + github: + enableEditLink: true + editLinkText: Edit this page on GitHub + editBranch: main + editViewMode: tree + enableSiteLastModified: true + enableFrontmatterTags: true + enableHashesForFMTags: false + enableRandomlyColoredTags: true + enableTaskNotes: true + enablePrettyLinks: true + searchMode: search diff --git a/docs/notes/ais.md b/docs/notes/ais.md new file mode 100644 index 00000000..06387890 --- /dev/null +++ b/docs/notes/ais.md @@ -0,0 +1,21 @@ +--- +id: g1i0o9ctmib83n4mydtsqfo +title: Automatic identification system +desc: "" +updated: 1708461719224 +created: 1708461557185 +--- + +The automatic identification system (AIS) is an automatic tracking system that uses transceivers on ships and is used by vessel traffic services (VTS). When satellites are used to receive AIS signatures, the term Satellite-AIS (S-AIS) is used. AIS information supplements marine radar, which continues to be the primary method of collision avoidance for water transport. Although technically and operationally distinct, the ADS-B system is analogous to AIS and performs a similar function for aircraft.[^1] + +## Limits + +- out of range +- intense traffic area +- expensive + +## Data source + +https://spire.com/maritime/ + +[^1]: https://en.wikipedia.org/wiki/Automatic_identification_system diff --git a/documentation/database_init.md b/docs/notes/database.initialisation.md similarity index 64% rename from documentation/database_init.md rename to docs/notes/database.initialisation.md index 3605e284..51cb8e08 100644 --- a/documentation/database_init.md +++ b/docs/notes/database.initialisation.md @@ -1,16 +1,18 @@ -# Database initialisation and versioning - - -## Database initialisation +--- +id: 8vo1vo09y37hwnrkmcy7w77 +title: Database initialisation +desc: "" +updated: 1708410470701 +created: 1708410417004 +--- First you need to run scripts which are in alembic/init_script : + - the load_vessels_data.py script will load vessels metadata from the data/chalutier_pelagique.csv file. - the load_geometry_data.py file will load shape data from the Nonterrestrial_WDPA_Jan2023.shp file. This file is not included in this github project but you can ask for it. It's only used for the alerting part. The second step is to load the [distance-from-port-v20201104.tiff](https://globalfishingwatch.org/data-download/datasets/public-distance-from-port-v1) and [distance-from-shore.tif](https://globalfishingwatch.org/data-download/datasets/public-distance-from-shore-v1) files. They are only used for the alerting part. + - install psql and raster2pgsql. - install raster type in db with postgis-raster using `create extension postgis_raster` -- adapt this command for each file : `raster2pgsql -t auto -I -C -M /PATH_TO/distance-from-shore.tif public.distance_shore | PGPASSWORD='POSTGRES_PASSWORD' psql -h POSTGRES_HOSTNAME -d POSTGRES_DB -U POSTGRES_USER -p POSTGRES_PORT` - -## Database versioning -The command ` alembic upgrade head` can be used in the root of the project in order to update the database schema to the last version. \ No newline at end of file +- adapt this command for each file : `raster2pgsql -t auto -I -C -M /PATH_TO/distance-from-shore.tif public.distance_shore | PGPASSWORD='POSTGRES_PASSWORD' psql -h POSTGRES_HOSTNAME -d POSTGRES_DB -U POSTGRES_USER -p POSTGRES_PORT` diff --git a/docs/notes/database.versioning.md b/docs/notes/database.versioning.md new file mode 100644 index 00000000..f039cc34 --- /dev/null +++ b/docs/notes/database.versioning.md @@ -0,0 +1,9 @@ +--- +id: kofmnl4h6i8e9uacvtd2wkr +title: Database versioning +desc: "" +updated: 1708410507271 +created: 1708410490074 +--- + +The command ` alembic upgrade head` can be used in the root of the project in order to update the database schema to the last version. diff --git a/documentation/dev_env.md b/docs/notes/development.environment.md similarity index 82% rename from documentation/dev_env.md rename to docs/notes/development.environment.md index 170ff346..9d3ebcf5 100644 --- a/documentation/dev_env.md +++ b/docs/notes/development.environment.md @@ -1,24 +1,31 @@ -# Development environment +--- +id: 6gumcsjv53j1xsyandigv97 +title: Development environment +desc: "" +updated: 1708410800063 +created: 1708410548545 +--- You can develop locally using Poetry and your own database but you can also use the Makefile to : -1) launch a local postgresql dockerized -2) launch a python environment dockerized +1. launch a local postgresql dockerized +2. launch a python environment dockerized If you work with Mac m1, the containers may not work as expected -## Local Database +## Local Database You will need to source the .env.template file but before you should modify it according to your local configuration, for example using POSTGRES_HOSTNAME=localhost ## Development Database + First, you need to create an .env.test file, you can use the .env.template file : ` cp .env.template .env.test` Next, you have to set the SPIRE_TOKEN variable. You can ask for it. Launch the following command : -` make launch-dev-db ` +`make launch-dev-db` Now, a postgresql db is available in your localhost, port 5480. And a pgadmin is available port 5080 @@ -26,7 +33,7 @@ You can remove it thanks to this command: ` make rm-db` TIPS : you can use the following command to launch the psql client : -` docker exec -ti postgres_bloom psql -d bloom_db -U bloom_user ` +`docker exec -ti postgres_bloom psql -d bloom_db -U bloom_user` ## Development environment @@ -44,11 +51,9 @@ To delete the container: A second option is to launch directly the app.py command thanks to this command : (the container is automatically removed after) ` make launch-app` - ## tests & precommit hook + Please install the [precommit hook](https://pre-commit.com/) tool locally to avoid any issue with the CI/CD. You may also want to launch tests : ` tox -vv` -.. - diff --git a/docs/notes/development.poetry.md b/docs/notes/development.poetry.md new file mode 100644 index 00000000..8e8d54da --- /dev/null +++ b/docs/notes/development.poetry.md @@ -0,0 +1,9 @@ +--- +id: iuuwulp9e75lg0jd18l0wyp +title: Poetry +desc: '' +updated: 1708410825991 +created: 1708410825991 +--- + +[Poetry](https://python-poetry.org/) \ No newline at end of file diff --git a/docs/notes/mpa.md b/docs/notes/mpa.md new file mode 100644 index 00000000..1d4e3378 --- /dev/null +++ b/docs/notes/mpa.md @@ -0,0 +1,9 @@ +--- +id: t40gr1j4y9jqm2ba4yh7xuf +title: Marine Protected Areas +desc: "" +updated: 1708461758182 +created: 1708461744075 +--- + +- https://www.cell.com/one-earth/fulltext/S2590-3322(20)30150-0 diff --git a/docs/notes/root.md b/docs/notes/root.md new file mode 100644 index 00000000..ddf33b87 --- /dev/null +++ b/docs/notes/root.md @@ -0,0 +1,20 @@ +--- +id: sfuij8tbpdmv6gnsqqsb8qf +title: Root +desc: '' +updated: 1708373223323 +created: 1708373223324 +--- +# Welcome to Dendron + +This is the root of your dendron vault. If you decide to publish your entire vault, this will be your landing page. You are free to customize any part of this page except the frontmatter on top. + +## Lookup + +This section contains useful links to related resources. + +- [Getting Started Guide](https://link.dendron.so/6b25) +- [Discord](https://link.dendron.so/6b23) +- [Home Page](https://wiki.dendron.so/) +- [Github](https://link.dendron.so/6b24) +- [Developer Docs](https://docs.dendron.so/) \ No newline at end of file diff --git a/docs/notes/root.schema.yml b/docs/notes/root.schema.yml new file mode 100644 index 00000000..94335cd0 --- /dev/null +++ b/docs/notes/root.schema.yml @@ -0,0 +1,7 @@ +version: 1 +imports: [] +schemas: + - id: root + children: [] + title: root + parent: root diff --git a/docs/notes/shom.md b/docs/notes/shom.md new file mode 100644 index 00000000..cccf724b --- /dev/null +++ b/docs/notes/shom.md @@ -0,0 +1,11 @@ +--- +id: dxdppizit5jtxu6cuwl6kz6 +title: Shom +desc: "" +updated: 1708461781678 +created: 1708461768767 +--- + +- https://www.shom.fr/ + +pour visualiser les droits miles nautique et les droits historiques diff --git a/docs/notes/sql.examples.md b/docs/notes/sql.examples.md new file mode 100644 index 00000000..50821850 --- /dev/null +++ b/docs/notes/sql.examples.md @@ -0,0 +1,31 @@ +--- +id: dgw0omk9y2u92olzvdxgddp +title: Useful SQL examples +desc: "" +updated: 1708410750577 +created: 1708410619351 +--- + +## List of mmi scrapped since last update + +```sql +SELECT mmsi as tm FROM spire_vessel_positions WHERE timestamp > '2023-06-17' GROUP BY mmsi +``` + +## More recent timestamp scrapped since last update + +```sql +SELECT tm,array_agg(mmsi) FROM (SELECT mmsi,max(timestamp) as tm FROM spire_vessel_positions WHERE timestamp > '2023-06-17' GROUP BY mmsi) as foo GROUP BY tm +``` + +## mmsi not scrapped by Spire + +```sql +SELECT mmsi FROM vessels WHERE mmsi IS NOT NULL AND mmsi NOT IN (SELECT mmsi as tm FROM spire_vessel_positions WHERE timestamp > '2023-06-17' GROUP BY mmsi) +``` + +## number of distincte position per boat which has more than 1000 positions gathered + +```sql +SELECT * FROM (SELECT COUNT(*) as sum_position,mmsi FROM (SELECT DISTINCT position,mmsi FROM spire_vessel_positions) as foo GROUP BY mmsi) as bar WHERE sum_position > 1000 +``` diff --git a/documentation/sql_examples.md b/documentation/sql_examples.md deleted file mode 100644 index c0c1e083..00000000 --- a/documentation/sql_examples.md +++ /dev/null @@ -1,13 +0,0 @@ -# Useful SQL examples - -List of mmi scrapped since last update :\ -`SELECT mmsi as tm FROM spire_vessel_positions WHERE timestamp > '2023-06-17' GROUP BY mmsi` - -More recent timestamp scrapped since last update :\ -`SELECT tm,array_agg(mmsi) FROM (SELECT mmsi,max(timestamp) as tm FROM spire_vessel_positions WHERE timestamp > '2023-06-17' GROUP BY mmsi) as foo GROUP BY tm` - -mmsi not scrapped by Spire :\ -`SELECT mmsi FROM vessels WHERE mmsi IS NOT NULL AND mmsi NOT IN (SELECT mmsi as tm FROM spire_vessel_positions WHERE timestamp > '2023-06-17' GROUP BY mmsi)` - -number of distincte position per boat which has more than 1000 positions gathered :\ -`SELECT * FROM (SELECT COUNT(*) as sum_position,mmsi FROM (SELECT DISTINCT position,mmsi FROM spire_vessel_positions) as foo GROUP BY mmsi) as bar WHERE sum_position > 1000` \ No newline at end of file diff --git a/documentation/table_of_contents.md b/documentation/table_of_contents.md deleted file mode 100644 index fbd3913d..00000000 --- a/documentation/table_of_contents.md +++ /dev/null @@ -1,5 +0,0 @@ -# Table of contents -1. [Database initialisation and versioning](./database_init.md) -2. [Development environment](./dev_env.md) -3. [Useful SQL examples](./sql_examples.md) -4. [Data models](#todo) diff --git a/images/banner.png b/images/banner.png new file mode 100644 index 00000000..b49e9667 Binary files /dev/null and b/images/banner.png differ