Exclude Compound Fields when using the bulk-api #42
Comments
|
@s7clarke10 I experienced the same issue and opened a PR to fix it. Feel free to test it out on your side to see if it fixes your issue as well, as it definitely fixed mine! |
|
Thank you very much Dinis, I will definitely have a look at this. Thank you for your contribution, I hope your PR is accepted 🤞.
On Thursday, 25 January 2024 at 02:16:25 am NZDT, Dinis Louseiro ***@***.***> wrote:
@s7clarke10 I experienced the same issue and opened a PR to fix it.
Feel free to test it out on your side to see if it fixes your issue as well, as it definitely fixed mine!
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
edgarrmondragon
pushed a commit
that referenced
this issue
Jun 28, 2024
In the current implementation of the tap, fields described with `type=address` are correctly excluded when using `api_type=BULK`. Although, this is not the case for geolocation fields (described with `type=location`). I'm not a Salesforce specialist per se, but know that the changes I'm applying on this branch solved the issue described [here](#42). In my specific case I had a field like this: ``` "Geolocation__c": { "type": [ "number", "object", "null" ], "properties": { "longitude": { "type": [ "null", "number" ] }, "latitude": { "type": [ "null", "number" ] } } }, ``` As it's a compound field, split over multiple properties (`longitude` and `latitude`), the parent field `Geolocation__c` does not have much value as the relevant information is propagated in the "sub-fields" (`latitude` and `longitude`). So what I'm doing in this PR is to ensure that: - `Geolocation__c` is marked with `unsupported` in the schema metadata (snippet below) while still ensuring that the "sub-fields" are correctly queried. Snippet of the excluded schema after this PR: ``` { "breadcrumb": [ "properties", "Geolocation__c" ], "metadata": { "inclusion": "unsupported", "unsupported-description": "cannot query compound address fields with bulk API" } }, ``` Snippet of the "sub-fields" in the schema: ``` "Geolocation__Latitude__s": { "type": [ "null", "number" ] }, "Geolocation__Longitude__s": { "type": [ "null", "number" ] }, ```
nezd
pushed a commit
to dext/tap-salesforce
that referenced
this issue
Sep 26, 2024
In the current implementation of the tap, fields described with `type=address` are correctly excluded when using `api_type=BULK`. Although, this is not the case for geolocation fields (described with `type=location`). I'm not a Salesforce specialist per se, but know that the changes I'm applying on this branch solved the issue described [here](MeltanoLabs#42). In my specific case I had a field like this: ``` "Geolocation__c": { "type": [ "number", "object", "null" ], "properties": { "longitude": { "type": [ "null", "number" ] }, "latitude": { "type": [ "null", "number" ] } } }, ``` As it's a compound field, split over multiple properties (`longitude` and `latitude`), the parent field `Geolocation__c` does not have much value as the relevant information is propagated in the "sub-fields" (`latitude` and `longitude`). So what I'm doing in this PR is to ensure that: - `Geolocation__c` is marked with `unsupported` in the schema metadata (snippet below) while still ensuring that the "sub-fields" are correctly queried. Snippet of the excluded schema after this PR: ``` { "breadcrumb": [ "properties", "Geolocation__c" ], "metadata": { "inclusion": "unsupported", "unsupported-description": "cannot query compound address fields with bulk API" } }, ``` Snippet of the "sub-fields" in the schema: ``` "Geolocation__Latitude__s": { "type": [ "null", "number" ] }, "Geolocation__Longitude__s": { "type": [ "null", "number" ] }, ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
If you use the Bulk v1 API you cannot extract compound fields. You must exclude them from your list of ingested fields.
"CRITICAL Error syncing ContactPointAddress: InvalidBatch : Failed to process query: FUNCTIONALITY_NOT_ENABLED: Selecting compound data not supported in Bulk Query", "level": "info", "timestamp": "2023-08-02T10:03:30.226398Z"}I believe Compound fields could be automatically excluded by doing a look-up first against the Salesforce Data Dictionary and examining for the given table being ingested whether any of the columns are compound fields e.g.
"compoundFieldName": "Coordinates__c",. In the example provided, the compound fieldCoordinates__cshould be excluded when using a Bulk API method.To find out more about the Salesforce Data Dictionary and dumping the Data Dictionary look at this github repo https://github.com/s7clarke10/get-salesforce-data-dictionary.
The text was updated successfully, but these errors were encountered: