Merge PR #146 into 18.0

Signed-off-by simahawk

connector_importer/README.rst

@@ -0,0 +1,379 @@
+==================
+Connector Importer
+==================
+
+.. 
+   !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
+   !! This file is generated by oca-gen-addon-readme !!
+   !! changes will be overwritten.                   !!
+   !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
+   !! source digest: sha256:7680b9cbb3329f4f4588d69d8487b9c03cae24d89c215adb8246c96b06f950fd
+   !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
+
+.. |badge1| image:: https://img.shields.io/badge/maturity-Beta-yellow.png
+    :target: https://odoo-community.org/page/development-status
+    :alt: Beta
+.. |badge2| image:: https://img.shields.io/badge/licence-AGPL--3-blue.png
+    :target: http://www.gnu.org/licenses/agpl-3.0-standalone.html
+    :alt: License: AGPL-3
+.. |badge3| image:: https://img.shields.io/badge/github-OCA%2Fconnector--interfaces-lightgray.png?logo=github
+    :target: https://github.com/OCA/connector-interfaces/tree/18.0/connector_importer
+    :alt: OCA/connector-interfaces
+.. |badge4| image:: https://img.shields.io/badge/weblate-Translate%20me-F47D42.png
+    :target: https://translation.odoo-community.org/projects/connector-interfaces-18-0/connector-interfaces-18-0-connector_importer
+    :alt: Translate me on Weblate
+.. |badge5| image:: https://img.shields.io/badge/runboat-Try%20me-875A7B.png
+    :target: https://runboat.odoo-community.org/builds?repo=OCA/connector-interfaces&target_branch=18.0
+    :alt: Try me on Runboat
+
+|badge1| |badge2| |badge3| |badge4| |badge5|
+
+This module allows to import / update records from files using the
+connector framework and job queue.
+
+To run an import you need at least:
+
+-  a backend, hosts the global configuration of the import.
+-  a recordset, hosts the configuration of the import for specific
+   models and source
+-  a source, provides the data to import
+-  an import type, describes which models you want to import and how to
+   import them
+
+**Table of contents**
+
+.. contents::
+   :local:
+
+Configuration
+=============
+
+Import type
+-----------
+
+Import types are the main configuration of the import. They describe
+which models you want to import and how to import them.
+
+Exaple of configuration:
+
+::
+
+   <record id="import_type_product_product_all_in_one" model="import.type">
+       <field name="name">Import Product - all in one</field>
+       <field name="key">product_product_all_in_one</field>
+       <field name="options">
+
+   - model: product.product
+       options:
+           importer:
+               odoo_unique_key: barcode
+           mapper:
+               name: product.product.mapper
+
+   - model: res.partner
+       options:
+           importer:
+               odoo_unique_key: name
+           override_existing: false
+           mapper:
+               name: importer.mapper.dynamic
+               source_key_prefix: supplier.
+               source_key_whitelist: supplier.name
+               default_keys:
+                   supplier_rank: 1
+
+   - model: product.supplierinfo
+       options:
+           importer:
+               odoo_unique_key: name
+           mapper:
+               name: product.supplierinfo.mapper
+               source_key_prefix: supplier.
+
+       </field>
+
+   </record>
+
+In this example we have 3 models to import one after the other using the
+same source file:
+
+-  product.product
+-  res.partner
+-  product.supplierinfo
+
+The import will run in the order of the configuration: first
+product.product, then res.partner and finally product.supplierinfo. For
+each model we have a configuration that describes how to import the
+data. With the ``options`` key we can define the configuration of the
+import for each component: ``importer``, ``mapper``, ``record_handler``,
+``tracking_handler``.
+
+The are 4 main components in the import configuration:
+
+-  importer
+-  mapper
+-  record_handler
+-  tracking_handler
+
+Each of them is responsible for a specific part of the import.
+
+The importer
+------------
+
+``importer`` is the main component that will import the data. It will
+use the ``mapper`` to map the data from the source to the destination
+model. If no ``name`` is defined the importer will use the default
+importer for the model which is capable of importing any model. Most of
+the time you don't need a specific importer.
+
+As the importer is the main component of the import if you want to
+customize it you'll have to declare it at an higher level, next to the
+``options`` key:
+
+::
+
+   - model: product.product
+       importer:
+           name: product.product.importer
+       options:
+           mapper:
+               name: product.product.mapper
+
+The importer accepts the following options:
+
+-  ``odoo_unique_key``: the field that will be used to find the record
+   in Odoo. If the record is found it will be updated, otherwise it will
+   be created.
+
+      NOTE: the value in the column declared as ``odoo_unique_key`` will
+      be treated as xid only if the name of the column is ``ìd`` or if
+      it starts with ``xid::``.
+
+-  ``break_on_error``: if set to True the import will stop if an error
+   occurs. Default is False.
+
+-  ``override_existing``: if set to True the existing records will be
+   updated. Default is True.
+
+-  ``translation_key_sep``: the separator used to split the translation
+   key. Default is ``:``. See below for information about translation
+   keys.
+
+-  ``translation_use_regional_lang``: if set to True the importer will
+   use the regional language, eg: fr_CH vs fr.
+
+-  ``ctx``: a dictionary of values to inject in the context of the
+   import.
+
+-  ``write_only``: if set to True the importer will not create new
+   records, it will only update existing ones. Default is False.
+
+The mapper
+----------
+
+The mapper is the component that will map the data from the source to
+the destination model.
+
+The most flexible mapper is the ``importer.mapper.dynamic`` that will
+map the data based on the model introspection and some options that you
+can define. The dynamic mapper accepts the following options:
+
+-  ``name``: the name of the mapper to use. If no name is defined the
+   default mapper for the model will be used.
+-  ``source_key_prefix``: a prefix to add to the source key. This is
+   useful when you want to map the same source key to different
+   destination fields.
+-  ``source_key_whitelist``: a list of source keys to import. If not
+   defined all the keys will be imported.
+-  ``source_key_blacklist``: a list of source keys to exclude from the
+   import.
+-  ``source_key_rename``: a dictionary of source keys to rename. The key
+   is the source key and the value is the new key.
+-  ``default_keys``: a dictionary of default values to set on the
+   destination record. The key is the field name and the value is the
+   default value.
+-  ``translation_keys``: a list of keys that will be used to translate
+   the data. See below for information about translation keys.
+-  ``required_keys``: a list of keys that are required. If one of the
+   keys is missing the record will be skipped. Please refer to the
+   documentation of the mapper to see advanced options.
+
+Considering the example above:
+
+::
+
+   - model: product.product
+       options:
+           mapper:
+               name: importer.mapper.dynamic
+               source_key_prefix: supplier.
+               source_key_whitelist: supplier.name
+               default_keys:
+                   supplier_rank: 1
+
+The mapper will:
+
+-  import only keys starting with ``supplier.`` ignoring the rest
+-  import only the key ``supplier.name``
+-  set the default value of ``supplier_rank`` to 1
+
+The record_handler
+------------------
+
+The record handler is the component that will handle the record create
+or update in Odoo. This component is responsible for:
+
+-  finding the record in Odoo
+-  creating the record if not found
+-  updating the record if found
+-  handling the translations
+
+If no ``name`` is defined the importer will use the default record
+handler for the model which is capable of handling any model. If you
+want to customize the record handler you'll have to declare it at an
+higher level, next to the ``options`` key:
+
+::
+
+   - model: product.product
+       options:
+           record_handler:
+               name: product.product.record_handler
+
+To find the record in Odoo the record handler will use the
+``odoo_unique_key`` if defined in the importer otherwise it will
+fallback to the matching domain. See below.
+
+The record handler accepts the following options:
+
+-  ``name``: the name of the record handler to use. If no name is
+   defined the default record handler for the model will be used.
+
+-  ``match_domain``: a domain to match the record in Odoo. When no
+   odoo_unique_key is provided by the importer you must provide a
+   match_domain.
+
+      This key accepts a snippet returning a domain. The snippet will be
+      evaluated in the context of the import and will receive:
+
+      -  ``orig_values``: the values from the source
+
+      -  ``values``: values computed by the mapper for the record
+
+      -  ``env``
+
+      -  ``user``
+
+      -  ``datetime``
+
+      -  ``dateutil``
+
+      -  ``time``
+
+      -  ``ref_id``: a function to get a record ID from a reference
+
+      -  ``ref``: a function to get a record from a reference
+
+            Example:
+
+            ::
+
+               match_domain: |
+                   [('name', '=', values.get('name'))]
+
+-  ``must_generate_xmlid``: if set to True the importer will generate an
+   XML ID for the record. Default is True if the unique key is an xmlid.
+
+-  ``skip_fields_unchanged``: if set to True the importer will skip the
+   fields that are unchanged. Default is False.
+
+Translations
+------------
+
+The importer can translate the data using the translation keys. The
+translation keys are a list of keys (column) that will be handled as
+translatable. Whenever a key is found in the translation keys the
+importer will look for a column with the same name suffixed by the
+language code (eg: name:fr_CH). If the column is found the importer will
+translate the data using the language code as context.
+
+Known issues / Roadmap
+======================
+
+-  with the import of standard Odoo CSV files, a concurrency error
+   occurs when updating the report_data of import_recordset table (from
+   the importer: self._do_report() -> self.recordset.set_report(...)).
+   The job is automatically retried a second time (without concurrency
+   errors). For small files it's not a big issue, but for files with a
+   huge amount of lines it takes time to process them two times.
+-  move generic functions from utils.mapper_utils to the connector
+   module
+-  unit tests for record handler and tracker
+-  add more test coverage for mapper utils and dynamic mapper
+-  consider making dynamic mapper the default one
+-  control how to generate xid (eg: from a specicic field with key
+   must_generate_xmlid_from_key)
+-  add manual control for backend_to_rel mappers
+-  refactor source to be a specific m2o to ease mgmt instead of a
+   generic relation
+
+Bug Tracker
+===========
+
+Bugs are tracked on `GitHub Issues <https://github.com/OCA/connector-interfaces/issues>`_.
+In case of trouble, please check there if your issue has already been reported.
+If you spotted it first, help us to smash it by providing a detailed and welcomed
+`feedback <https://github.com/OCA/connector-interfaces/issues/new?body=module:%20connector_importer%0Aversion:%2018.0%0A%0A**Steps%20to%20reproduce**%0A-%20...%0A%0A**Current%20behavior**%0A%0A**Expected%20behavior**>`_.
+
+Do not contact contributors directly about support or help with technical issues.
+
+Credits
+=======
+
+Authors
+-------
+
+* Camptocamp
+
+Contributors
+------------
+
+Simone Orsi (Camptocamp) for the original implementation.
+
+Other contributors include:
+
+-  Guewen Baconnier (Camptocamp)
+-  Mykhailo Panarin (Camptocamp)
+-  Sébastien Alix (Camptocamp)
+-  Thien Vo (Trobz)
+
+Other credits
+-------------
+
+The migration of this module from 16.0 to 18.0 was financially supported
+by Camptocamp.
+
+Maintainers
+-----------
+
+This module is maintained by the OCA.
+
+.. image:: https://odoo-community.org/logo.png
+   :alt: Odoo Community Association
+   :target: https://odoo-community.org
+
+OCA, or the Odoo Community Association, is a nonprofit organization whose
+mission is to support the collaborative development of Odoo features and
+promote its widespread use.
+
+.. |maintainer-simahawk| image:: https://github.com/simahawk.png?size=40px
+    :target: https://github.com/simahawk
+    :alt: simahawk
+
+Current `maintainer <https://odoo-community.org/page/maintainer-role>`__:
+
+|maintainer-simahawk| 
+
+This module is part of the `OCA/connector-interfaces <https://github.com/OCA/connector-interfaces/tree/18.0/connector_importer>`_ project on GitHub.
+
+You are welcome to contribute. To learn how please visit https://odoo-community.org/page/Contribute.

connector_importer/__init__.py

@@ -0,0 +1,3 @@
+from . import models
+from . import components
+from . import controllers