Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: datahike.migrate has a problem with schema/double (which cbor converts to float) #633

Open
awb99 opened this issue Jul 26, 2023 · 2 comments
Labels
bug Something isn't working triage

Comments

@awb99
Copy link

awb99 commented Jul 26, 2023

What version of Datahike are you using?

0.6

What version of Java are you using?

17

What operating system are you using?

guix

What database EDN configuration are you using?

irrelevant to this ticket

Describe the bug

I started my datahike database with version 0.4 with schema-on-write.
I ONLY used double in the schema definition.
Now I had to migrate the 0.4 edn dumps to import in 0.6 which uses cbor dumps.
Previously I was exporting in 0.4 edn dumps which I would import on another machine without a problem.
Now I have noted that when the edn dumps are read via the edn parser and imported into 0.6
that I had to make an adjustment to change all float values to double values.
Interestingly, this was not a problem before.

What is the expected behaviour?

put into readme that :double has a different meaning in 0.4 in relationship to 0.6.

How can the behaviour be reproduced?

(defn float->double [v]
  (if (float? v)
    (double v)
    v))

(defn db-migrate []
  (warn "migrating cbor db..")
  (let [txs-old (load-cbor crbdb/conn "data/datahike-dump/eavt-dump")
        s (stats txs-old)
        max-eid (:max-eid s)
        max-tx (:max-tx s)
        tx-old-no-schema (remove datom-schema? txs-old)
        tx-old-safe (map #(-> %
                              (update :v float->double))
                         tx-old-no-schema)
        schema-with-eids (assoc-schema-ids max-eid schema)]
    (warn "dump stats: " s)
    (warn "transacting schema with eid above: " max-eid)
    (let [result-schema (crbdb/transact schema-with-eids)]
      (print-tx-stats result-schema))
    (let [result-import (api/transact crbdb/conn (vec tx-old-safe))]
      (warn "import dump result:")
      (print-tx-stats result-import))
    ;(assoc crbdb/conn :max-tx max-tx)
    (warn "db migration finished!")))
@awb99 awb99 added bug Something isn't working triage labels Jul 26, 2023
@awb99
Copy link
Author

awb99 commented Jul 27, 2023

After more testing, this is a bug in the cbor persistence layer.
Just calling datahike.migrate/export-db and datahike.migrate.import-db results to import errors
for doubles which cbor seems to store as float:

[datahike.db.transaction:45] - Bad entity value 0.0 at [:db/add 2199133 :lineitem/price 0.0 536871102] , value does not match schema definition. Must be conform to: double? {:error :transact/schema, :value 0.0, :attribute :lineitem/price, :schema #:db{:valueType :db.type/double, :cardinality :db.cardinality/one, :ident :lineitem/price}}

@awb99 awb99 changed the title [Bug]: schema/double changed from 0.4 to 0.6 [Bug]: datahike.migrate has a problem with schema/double (which cbor converts to float) Jul 27, 2023
@TimoKramer
Copy link
Member

Hey @awb99 , thanks for reporting this issue and great find! We value contributions and are happy to help with it. In case you find some time to fix this, please don't hesitate. We are all pretty busy lately and I really hope this issue can be resolved soon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working triage
Projects
None yet
Development

No branches or pull requests

2 participants