Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[18.0][MIG] base_import_async: Migration to 18.0 #699

Open
wants to merge 77 commits into
base: 18.0
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
77 commits
Select commit Hold shift + click to select a range
18219d5
[IMP] rename base_import_connector to base_import_async
sbidoul Nov 14, 2014
18a9dc1
[FIX] silence pylint, which btw discovered a genuine bug
sbidoul Nov 14, 2014
d33fcc9
[IMP] code style using session shortcuts
sbidoul Nov 14, 2014
1d5f7e8
[IMP] slightly improve README
sbidoul Nov 14, 2014
52a4352
[IMP] add comma in README
sbidoul Nov 14, 2014
be79ba2
Add OCA as author of OCA addons
gurneyalex Mar 3, 2015
f340fcc
[IMP] OpenERP -> Odoo in comments
sbidoul Apr 20, 2015
d951f0a
[FIX] base_import_async: properly install connector module
sbidoul Apr 20, 2015
ff6b631
[IMP] base_import_async: update README
sbidoul Apr 12, 2015
ea10594
[IMP] base_import_async: further align README with OCA template
sbidoul Apr 21, 2015
d8f3586
base_import_async: allow to set the base priority
guewen May 15, 2015
1fc8669
Add bug tracker link on README.rst
yvaucher May 22, 2015
dcc7eae
Replace deprecated functions
guewen Jun 10, 2015
1470932
Remove call to install_in_connector()
guewen Jun 10, 2015
57e5312
[UPD] prefix versions with 8.0
sbidoul Oct 9, 2015
8deda99
[MIG] Make modules uninstallable
pedrobaeza Oct 14, 2015
8392c0a
[FIX] Fix new/old api compatibility
lmignon Oct 19, 2015
6564148
Update base_import_async.py
OdooMod Nov 19, 2015
8384196
Update base_import_async.py
OdooMod Nov 19, 2015
2085617
migrate base_import_async and it's test to 9.0
sbidoul Mar 3, 2016
d862023
migrate javascript
sbidoul Mar 3, 2016
54160a7
[MIG] Make modules uninstallable
pedrobaeza Oct 6, 2016
e345a85
[MIG] Rename manifest files
pedrobaeza Oct 6, 2016
6636c62
Porting to new API
danieltorres7 Nov 4, 2016
f52533f
Whitespace
danieltorres7 Nov 7, 2016
c757956
[MIG] base_import_async: replace connector by queue_job
lmignon Mar 21, 2017
62f39cc
[FIX] base_import_async: with_delay returns an instance of Job
lmignon Mar 23, 2017
007ec31
[MIG] base_import_async and test_base_import_async to 11.0
tarteo Jun 21, 2018
4b241b0
[IMP] Processed review
tarteo Jun 22, 2018
6f82da1
[IMP] Readability, speed, and added HISTORY.rst
tarteo Jun 26, 2018
f601115
[MIG] Make modules uninstallable
OCA-git-bot Sep 27, 2018
3aeb04f
Set modules as uninstallable
guewen Oct 1, 2019
01b4f4a
Run pre-commit with black, isort, ...
guewen Oct 4, 2019
9d9bd01
Migrate base_import_async to 12.0
guewen Oct 2, 2018
f195df3
Add OCA development status
guewen Oct 5, 2018
f860ba1
Exit the import screen after importing file in async
guewen Nov 15, 2019
14b0340
Set default values for encoding and separator
guewen Nov 17, 2019
44e82fa
[ADD] icon.png
OCA-git-bot Nov 17, 2019
27aafd0
[IMP] base_import_async: black, isort
guewen Dec 20, 2019
0afe0d7
[MIG] base_import_async: Migration to 13.0
guewen Dec 20, 2019
b5250ef
Apply suggestions from code review
guewen Jan 9, 2020
d0f7ed5
pre-commit update
OCA-git-bot Mar 7, 2020
75be013
[FIX] base_import_async error message
Jun 25, 2020
395372c
base_import_async 13.0.1.0.1
OCA-git-bot Jun 26, 2020
ea11b80
base_import_async: remove `@job` decorator
guewen Nov 2, 2020
f3fa0ed
Fix invalid development_status (accepted value is Production/Stable)
guewen Nov 2, 2020
bd2bcd5
base_import_async 13.0.2.0.0
OCA-git-bot Nov 12, 2020
423684f
[MIG] init V14 repo
sebastienbeau Nov 20, 2020
c2eca3c
[MIG] base_import_async: Migration to 14.0
BT-jcamacho May 27, 2021
ac6f365
base_import_async 14.0.1.0.1
OCA-git-bot Aug 11, 2021
46cd68a
Set modules as uninstallable
guewen Nov 1, 2021
e85ecf7
Added translation using Weblate (Italian)
francesco-ooops Dec 3, 2021
6bffea0
Translated using Weblate (Italian)
francesco-ooops Dec 3, 2021
dec4d0b
[IMP] update dotfiles [ci skip]
OCA-git-bot Mar 31, 2022
ad2b474
base_import_async: add a dedicated channel
sebalix Jul 29, 2022
59b7d02
base_import_async 14.0.1.0.2
OCA-git-bot Nov 22, 2022
21c9230
Added translation using Weblate (Romanian)
dhongu Nov 22, 2022
6d64018
[IMP] base_import_async: black, isort, prettier
ieski Nov 24, 2022
c61bb4e
[MIG] base_import_async: Migration to 15.0
ieski Nov 24, 2022
1fe1d9f
[16.0][MIG] base_import_async: Migration to 16.0
dzungtran89 Mar 1, 2023
4809441
Update translation files
weblate Jun 28, 2023
aeaa256
[FIX] base_import_async: restore class
danielduqma Sep 13, 2023
6dbbf65
[FIX] base_import_async: remove default values from context when crea…
marielejeune Apr 14, 2023
a986f96
oca-github-bot post-merge updates
OCA-git-bot Sep 18, 2023
666e284
Added translation using Weblate (Spanish)
asierneiradev Sep 25, 2023
be18197
Translated using Weblate (Spanish)
asierneiradev Sep 25, 2023
55f3c4c
Update translation files
weblate Oct 9, 2023
59adbb2
Added translation using Weblate (Italian)
mymage Jan 9, 2024
052d434
Translated using Weblate (Italian)
mymage Jan 11, 2024
d452c04
[16.0][FIX] base_import_async: update OPT_HAS_HEADERS option
alexbottemanne Jan 31, 2024
ea24ccf
[16.0][FIX] base_import_async, add sudo call to models without explic…
ChrisOForgeFlow Jul 10, 2024
038c1a3
[IMP] base_import_async: pre-commit stuff
Wodran14 Sep 3, 2024
4fefb21
[MIG] base_import_async: Migration to 17.0
Wodran14 Sep 3, 2024
a50da34
[IMP] base_import_async: pre-commit auto fixes
xaviedoanhduy Oct 7, 2024
0bc3ef9
[MIG] base_import_async: Migration to 18.0
xaviedoanhduy Oct 7, 2024
f643ad5
[ADD] base_import_async: add tests
xaviedoanhduy Oct 10, 2024
ca4ebd2
Fix 'document' is not defined [no-undef] lint in base_import_async
xaviedoanhduy Oct 28, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
154 changes: 154 additions & 0 deletions base_import_async/README.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,154 @@
===================
Asynchronous Import
===================

..
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! This file is generated by oca-gen-addon-readme !!
!! changes will be overwritten. !!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! source digest: sha256:e7bcaae80fd59cc8e45b2784606eb026fba7dfcac892ad4a986f51e0dc248e1e
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

.. |badge1| image:: https://img.shields.io/badge/maturity-Production%2FStable-green.png
:target: https://odoo-community.org/page/development-status
:alt: Production/Stable
.. |badge2| image:: https://img.shields.io/badge/licence-AGPL--3-blue.png
:target: http://www.gnu.org/licenses/agpl-3.0-standalone.html
:alt: License: AGPL-3
.. |badge3| image:: https://img.shields.io/badge/github-OCA%2Fqueue-lightgray.png?logo=github
:target: https://github.com/OCA/queue/tree/18.0/base_import_async
:alt: OCA/queue
.. |badge4| image:: https://img.shields.io/badge/weblate-Translate%20me-F47D42.png
:target: https://translation.odoo-community.org/projects/queue-18-0/queue-18-0-base_import_async
:alt: Translate me on Weblate
.. |badge5| image:: https://img.shields.io/badge/runboat-Try%20me-875A7B.png
:target: https://runboat.odoo-community.org/builds?repo=OCA/queue&target_branch=18.0
:alt: Try me on Runboat

|badge1| |badge2| |badge3| |badge4| |badge5|

This module extends the standard CSV import functionality to import
files in the background using the OCA/queue framework.

**Table of contents**

.. contents::
:local:

Usage
=====

The user is presented with a new checkbox in the import screen. When
selected, the import is delayed in a background job.

This job in turn splits the CSV file in chunks of minimum 100 lines (or
more to align with record boundaries). Each chunk is then imported in a
separate background job.

When an import fails, the job is marked as such and the user can read
the error in the job status. The CSV chunk being imported is stored as
an attachment to the job, making it easy to download it, fix it and run
a new import, possibly in synchronous mode since the chunks are small.

Any file that can be imported by the standard import mechanism can also
be imported in the background.

This module's scope is limited to making standard imports asynchronous.
It does not attempt to transform the data nor automate ETL flows.

Other modules may benefit from this infrastructure in the following way
(as illustrated in the test suite):

1. create an instance of base_import.import and populate its fields
(res_model, file, file_name),
2. invoke the do method with appropriate options (header, encoding,
separator, quoting, use_queue, chunk_size).

Known issues / Roadmap
======================

- There is currently no user interface to control the chunk size, which
is currently 100 by default. Should this proves to be an issue, it is
easy to add an option to extend the import screen.
- Validation cannot be run in the background.

Changelog
=========

13.0.1.0.0 (2019-12-20)
-----------------------

- [MIGRATION] from 12.0 branched at rev. a7f8031

Bug Tracker
===========

Bugs are tracked on `GitHub Issues <https://github.com/OCA/queue/issues>`_.
In case of trouble, please check there if your issue has already been reported.
If you spotted it first, help us to smash it by providing a detailed and welcomed
`feedback <https://github.com/OCA/queue/issues/new?body=module:%20base_import_async%0Aversion:%2018.0%0A%0A**Steps%20to%20reproduce**%0A-%20...%0A%0A**Current%20behavior**%0A%0A**Expected%20behavior**>`_.

Do not contact contributors directly about support or help with technical issues.

Credits
=======

Authors
-------

* Akretion
* ACSONE SA/NV

Contributors
------------

Sébastien Beau (Akretion) authored the initial prototype.

Stéphane Bidoul (ACSONE) extended it to version 1.0 to support
multi-line records, store data to import as attachments and let the user
control the asynchronous behaviour.

Other contributors include:

- Anthony Muschang (ACSONE)

- David Béal (Akretion)

- Jonathan Nemry (ACSONE)

- Laurent Mignon (ACSONE)

- Dennis Sluijk (Onestein)

- Guewen Baconnier (Camptocamp)

- `Trobz <https://trobz.com>`__:

- Dzung Tran <[email protected]>
- Do Anh Duy <[email protected]>

- Daniel Duque (FactorLibre)

Other credits
-------------

The migration of this module from 17.0 to 18.0 was financially supported
by Camptocamp

Maintainers
-----------

This module is maintained by the OCA.

.. image:: https://odoo-community.org/logo.png
:alt: Odoo Community Association
:target: https://odoo-community.org

OCA, or the Odoo Community Association, is a nonprofit organization whose
mission is to support the collaborative development of Odoo features and
promote its widespread use.

This module is part of the `OCA/queue <https://github.com/OCA/queue/tree/18.0/base_import_async>`_ project on GitHub.

You are welcome to contribute. To learn how please visit https://odoo-community.org/page/Contribute.
3 changes: 3 additions & 0 deletions base_import_async/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# License AGPL-3.0 or later (https://www.gnu.org/licenses/agpl).

from . import models
25 changes: 25 additions & 0 deletions base_import_async/__manifest__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# @author Stéphane Bidoul <[email protected]>
# @author Sébastien BEAU <[email protected]>
# License AGPL-3.0 or later (https://www.gnu.org/licenses/agpl).

{
"name": "Asynchronous Import",
"summary": "Import CSV files in the background",
"version": "18.0.1.0.0",
"author": "Akretion, ACSONE SA/NV, Odoo Community Association (OCA)",
"license": "AGPL-3",
"website": "https://github.com/OCA/queue",
"category": "Generic Modules",
"depends": ["base_import", "queue_job"],
"data": [
"data/queue_job_function_data.xml",
],
"assets": {
"web.assets_backend": [
"base_import_async/static/src/js/import_model.esm.js",
"base_import_async/static/src/xml/import_data_sidepanel.xml",
],
},
"installable": True,
"development_status": "Production/Stable",
}
21 changes: 21 additions & 0 deletions base_import_async/data/queue_job_function_data.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
<odoo noupdate="1">
<record id="job_function_base_import_import_split_file" model="queue.job.function">
<field name="model_id" ref="base_import.model_base_import_import" />
<field name="method">_split_file</field>
<field
name="related_action"
eval='{"func_name": "_related_action_attachment"}'
/>
</record>
<record
id="job_function_base_import_import_import_one_chunk"
model="queue.job.function"
>
<field name="model_id" ref="base_import.model_base_import_import" />
<field name="method">_import_one_chunk</field>
<field
name="related_action"
eval='{"func_name": "_related_action_attachment"}'
/>
</record>
</odoo>
78 changes: 78 additions & 0 deletions base_import_async/i18n/base_import_async.pot
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
# Translation of Odoo Server.
# This file contains the translation of the following modules:
# * base_import_async
#
msgid ""
msgstr ""
"Project-Id-Version: Odoo Server 16.0\n"
"Report-Msgid-Bugs-To: \n"
"Last-Translator: \n"
"Language-Team: \n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: \n"
"Plural-Forms: \n"

#. module: base_import_async
#. odoo-python
#: code:addons/base_import_async/models/queue_job.py:0
#, python-format
msgid "Attachment"
msgstr ""

#. module: base_import_async
#: model:ir.model,name:base_import_async.model_base_import_import
msgid "Base Import"
msgstr ""

#. module: base_import_async
#. odoo-python
#: code:addons/base_import_async/models/base_import_import.py:0
#, python-format
msgid ""
"Import %(model)s from file %(file_name)s - #%(chunk)s - lines %(from)s to "
"%(to)s"
msgstr ""

#. module: base_import_async
#. odoo-python
#: code:addons/base_import_async/models/base_import_import.py:0
#, python-format
msgid "Import %(model)s from file %(from_file)s"
msgstr ""

#. module: base_import_async
#. odoo-javascript
#: code:addons/base_import_async/static/src/xml/import.xml:0
#, python-format
msgid "Import in the background"
msgstr ""

#. module: base_import_async
#: model:ir.model,name:base_import_async.model_queue_job
msgid "Queue Job"
msgstr ""

#. module: base_import_async
#. odoo-javascript
#: code:addons/base_import_async/static/src/xml/import.xml:0
#, python-format
msgid ""
"When checked, the import will be executed as a background job, after "
"splitting your file in small chunks that will be processed independently. "
"Use this to import very large files."
msgstr ""

#. module: base_import_async
#. odoo-javascript
#: code:addons/base_import_async/static/src/js/import.js:0
#, python-format
msgid "You can check the status of this job in menu 'Queue / Jobs'."
msgstr ""

#. module: base_import_async
#. odoo-javascript
#: code:addons/base_import_async/static/src/js/import.js:0
#, python-format
msgid "Your request is being processed"
msgstr ""
88 changes: 88 additions & 0 deletions base_import_async/i18n/es.po
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
# Translation of Odoo Server.
# This file contains the translation of the following modules:
# * base_import_async
#
msgid ""
msgstr ""
"Project-Id-Version: Odoo Server 16.0\n"
"Report-Msgid-Bugs-To: \n"
"PO-Revision-Date: 2023-09-25 09:34+0000\n"
"Last-Translator: Asier Neira <[email protected]>\n"
"Language-Team: none\n"
"Language: es\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: \n"
"Plural-Forms: nplurals=2; plural=n != 1;\n"
"X-Generator: Weblate 4.17\n"

#. module: base_import_async
#. odoo-python
#: code:addons/base_import_async/models/queue_job.py:0
#, python-format
msgid "Attachment"
msgstr "Archivo adjunto"

#. module: base_import_async
#: model:ir.model,name:base_import_async.model_base_import_import
msgid "Base Import"
msgstr "Importación Base"

#. module: base_import_async
#. odoo-python
#: code:addons/base_import_async/models/base_import_import.py:0
#, python-format
msgid ""
"Import %(model)s from file %(file_name)s - #%(chunk)s - lines %(from)s to "
"%(to)s"
msgstr ""
"Importar %(model)s del archivo %(file_name)s - #%(chunk)s - líneas %(from)s "
"a %(to)s"

#. module: base_import_async
#. odoo-python
#: code:addons/base_import_async/models/base_import_import.py:0
#, python-format
msgid "Import %(model)s from file %(from_file)s"
msgstr "Importar %(model)s del fichero %(from_file)s"

#. module: base_import_async
#. odoo-javascript
#: code:addons/base_import_async/static/src/xml/import.xml:0
#, python-format
msgid "Import in the background"
msgstr "Importación en segundo plano"

#. module: base_import_async
#: model:ir.model,name:base_import_async.model_queue_job
msgid "Queue Job"
msgstr "Cola de trabajo"

#. module: base_import_async
#. odoo-javascript
#: code:addons/base_import_async/static/src/xml/import.xml:0
#, python-format
msgid ""
"When checked, the import will be executed as a background job, after "
"splitting your file in small chunks that will be processed independently. "
"Use this to import very large files."
msgstr ""
"Si esta opción está seleccionada, la importación se ejecutará como un "
"trabajo en segundo plano, tras dividir el archivo en pequeños trozos que se "
"procesarán de forma independiente. Utilícelo para importar archivos muy "
"grandes."

#. module: base_import_async
#. odoo-javascript
#: code:addons/base_import_async/static/src/js/import.js:0
#, python-format
msgid "You can check the status of this job in menu 'Queue / Jobs'."
msgstr ""
"Puede comprobar el estado de este trabajo en el menú 'Cola / Trabajos'."

#. module: base_import_async
#. odoo-javascript
#: code:addons/base_import_async/static/src/js/import.js:0
#, python-format
msgid "Your request is being processed"
msgstr "Su solicitud está siendo procesada"
Loading
Loading