Unexpected parsing of Array of Objects (MQTT measure) #856

tstorek · 2025-01-27T09:26:02Z

IoT Agent JSON version the issue has been seen with

{'libVersion': '4.0.1', 'port': 4041, 'baseRoot': '/', 'version': '3.0.2'}

Bound or port used (API interaction)

Northbound (Provision API and NGSI Interactions)

NGSI version

NGSIv2

Are you running a container?

Yes, I am using a contaner (Docker, Kubernetes...)

Image type

normal

Expected behaviour you didn't see

We want to send weather forecast objects via the IoT-Agent to the orion.
Provisioning as an Array and MQTT-Traffic works as expected:

Here an example of the payload for a temperature:

[
  {"timestamp": "2025-01-27T09:00:00Z", "value": 10.67}, 
  {"timestamp": "2025-01-27T12:00:00Z", "value": 10.2}, 
  {"timestamp": "2025-01-27T15:00:00Z", "value": 9.16}, 
  {"timestamp": "2025-01-27T18:00:00Z", "value": 7.72}, 
  {"timestamp": "2025-01-27T21:00:00Z", "value": 7.39}, 
  {"timestamp": "2025-01-28T00:00:00Z", "value": 6.39}, 
  {"timestamp": "2025-01-28T03:00:00Z", "value": 5.87},
  {"timestamp": "2025-01-28T06:00:00Z", "value": 5.54}
]

As this is valid JSON we expect the payload parsed in this form.

Unexpected behaviour you saw

Unfortunetly, the IoT-Agent only parses the first item of the array and sends it the the context broker.
In the MongoDB we only find the first value:

...
        temperatureForecast: {
            creDate: 1737645987.159389,
            md: {
                description: {
                    type: 'Text',
                    value: 'Temperature forecast for the next 5 days.'
                },
                TimeInstant: {
                    type: 'DateTime',
                    value: 1737969131.6820002
                }
            },
            mdNames: [
                'description',
                'TimeInstant'
            ],
            modDate: 1737969131.696273,
            type: 'Array',
            value: {
                timestamp: '2025-01-27T09:00:00Z',
                value: 10.67
            }
        },
...

I think the issue is here:

iotagent-json/lib/commonBindings.js

Line 253 in 5cd70e4

value: parsedMessage[0]

Steps to reproduce the problem

provision device config with an Array attibute:

{
        "name": "temperatureForecast",
        "type": "Array",
        "metadata": {
          "description": {
            "type": "Text",
            "value": "Temperature forecast for the next 5 days."
          }
        },
        "expression": null,
        "entity_name": null,
        "entity_type": null,
        "reverse": null,
        "object_id": null
      }

send an payload as above
check orion

Configs

environment:
    - "IOTA_CB_HOST=orion"
    - "IOTA_CB_PORT=1026"
    - "IOTA_NORTH_PORT=4041"
    - "IOTA_REGISTRY_TYPE=mongodb"
    - "IOTA_MONGO_HOST=mongodb"
    - "IOTA_MONGO_PORT=27017"
    - "IOTA_MONGO_DB=iotagent-json"
    - "IOTA_HTTP_PORT=7896"
    - "IOTA_PROVIDER_URL=http://iot-agent:4041"

Log output

AlvaroVega · 2025-01-27T14:37:16Z

Using latest version of iotagent and sending:

curl -i -X POST 'http://localhost:7897/iot/json?i=dev&k=APIKEY' -d '[  {"timestamp": "2025-01-27T09:00:00Z", "value": 10.67}, 
  {"timestamp": "2025-01-27T12:00:00Z", "value": 10.2}, 
  {"timestamp": "2025-01-27T15:00:00Z", "value": 9.16}, 
  {"timestamp": "2025-01-27T18:00:00Z", "value": 7.72}, 
  {"timestamp": "2025-01-27T21:00:00Z", "value": 7.39}, 
  {"timestamp": "2025-01-28T00:00:00Z", "value": 6.39}, 
  {"timestamp": "2025-01-28T03:00:00Z", "value": 5.87},
  {"timestamp": "2025-01-28T06:00:00Z", "value": 5.54}
]' -H 'content-type: application/json'

I see all these measures are delivered to CB:


ime=2025-01-27T14:30:57.400Z | lvl=DEBUG | corr=8655f605-2549-43eb-94e2-28c6926fcc08 | trans=8655f605-2549-43eb-94e2-28c6926fcc08 | op=IoTAgentNGSI.Request | from=n/a | srv=smartcity | subsrv=/ | msg=Options: {
    "url": "http://iot-orion:1026/v2/op/update?options=flowControl",
    "method": "POST",
    "headers": {
        "fiware-service": "smartcity",
        "fiware-servicepath": "/"
    },
    "json": {
        "actionType": "append",
        "entities": [
            {
                "id": "thing:dev",
                "type": "thing",
                "timestamp": {
                    "type": "Text",
                    "value": "2025-01-27T09:00:00Z",
                    "metadata": {
                        "TimeInstant": {
                            "type": "DateTime",
                            "value": "2025-01-27T14:30:57.399Z"
                        }
                    }
                },
                "value": {
                    "type": "Text",
                    "value": 10.67,
                    "metadata": {
                        "TimeInstant": {
                            "type": "DateTime",
                            "value": "2025-01-27T14:30:57.399Z"
                        }
                    }
                },
                "TimeInstant": {
                    "type": "DateTime",
                    "value": "2025-01-27T14:30:57.399Z"
                }
            },
            {
                "id": "thing:dev",
                "type": "thing",
                "timestamp": {
                    "type": "Text",
                    "value": "2025-01-27T12:00:00Z",
                    "metadata": {
                        "TimeInstant": {
                            "type": "DateTime",
                            "value": "2025-01-27T14:30:57.399Z"
                        }
                    }
                },
                "value": {
                    "type": "Text",
                    "value": 10.2,
                    "metadata": {
                        "TimeInstant": {
                            "type": "DateTime",
                            "value": "2025-01-27T14:30:57.399Z"
                        }
                    }
                },
                "TimeInstant": {
                    "type": "DateTime",
                    "value": "2025-01-27T14:30:57.399Z"
                }
            },
            {
                "id": "thing:dev",
                "type": "thing",
                "timestamp": 
                    "type": "Text",
                    "value": "2025-01-27T15:00:00Z",
                    "metadata": {
                        "TimeInstant": {
                            "type": "DateTime",
                            "value": "2025-01-27T14:30:57.399Z"
                        }
                    }
                },
                "value": {
                    "type": "Text",
                    "value": 9.16,
                    "metadata": {
                        "TimeInstant": {
                            "type": "DateTime",
                            "value": "2025-01-27T14:30:57.399Z"
                        }
                    }
                },
                "TimeInstant": {
                    "type": "DateTime",
                    "value": "2025-01-27T14:30:57.399Z"
                }
            },
            {
                "id": "thing:dev",
                "type": "thing",
                "timestamp": {
                    "type": "Text",
                    "value": "2025-01-27T18:00:00Z",
                    "metadata": {
                        "TimeInstant": {
                            "type": "DateTime",
                            "value": "2025-01-27T14:30:57.399Z"
                        }
                    }
                },
                "value": {
                    "type": "Text",
                    "value": 7.72,
                    "metadata": {
                        "TimeInstant": {
                            "type": "DateTime",
                            "value": "2025-01-27T14:30:57.399Z"
                        }
                    }
                },
                "TimeInstant": {
                    "type": "DateTime",
                    "value": "2025-01-27T14:30:57.399Z"
                }
            },
            {
                "id": "thing:dev",
                "type": "thing",
                "timestamp": {
                    "type": "Text",
                    "value": "2025-01-27T21:00:00Z",
                    "metadata": {
                        "TimeInstant": {
                            "type": "DateTime",
                            "value": "2025-01-27T14:30:57.399Z"
                        }
                    }
                },
                "value": {
                    "type": "Text",
                    "value": 7.39,
                    "metadata": {
                        "TimeInstant": {
                            "type": "DateTime",
                            "value": "2025-01-27T14:30:57.399Z"
                        }
                    }
                },
                "TimeInstant": {
                    "type": "DateTime",
                    "value": "2025-01-27T14:30:57.399Z"
                }
            },
            {
                "id": "thing:dev",
                "type": "thing",
                "timestamp": {
                    "type": "Text",
                    "value": "2025-01-28T00:00:00Z",
                    "metadata": {
                        "TimeInstant": {
                            "type": "DateTime",
                            "value": "2025-01-27T14:30:57.399Z"
                        }
                    }
                },
                "value": {
                    "type": "Text",
                    "value": 6.39,
                    "metadata": {
                        "TimeInstant": {
                            "type": "DateTime",
                            "value": "2025-01-27T14:30:57.399Z"
                        }
                    }
                },
                "TimeInstant": {
                    "type": "DateTime",
                    "value": "2025-01-27T14:30:57.399Z"
                }
            },
            {
                "id": "thing:dev",
                "type": "thing",
                "timestamp": {
                    "type": "Text",
                    "value": "2025-01-28T03:00:00Z",
                    "metadata": {
                        "TimeInstant": {
                            "type": "DateTime",
                            "value": "2025-01-27T14:30:57.399Z"
                        }
                    }
                },
                "value": {
                    "type": "Text",
                    "value": 5.87,
                    "metadata": {
                        "TimeInstant": {
                            "type": "DateTime",
                            "value": "2025-01-27T14:30:57.399Z"
                        }
                    }
                },
                "TimeInstant": {
                    "type": "DateTime",
                    "value": "2025-01-27T14:30:57.399Z"
                }
            },
            {
                "id": "thing:dev",
                "type": "thing",
                "timestamp": {
                    "type": "Text",
                    "value": "2025-01-28T06:00:00Z",
                    "metadata": {
                        "TimeInstant": {
                            "type": "DateTime",
                            "value": "2025-01-27T14:30:57.399Z"
                        }
                    }
                },
                "value": {
                    "type": "Text",
                    "value": 5.54,
                    "metadata": {
                        "TimeInstant": {
                            "type": "DateTime",
                            "value": "2025-01-27T14:30:57.399Z"
                        }
                    }
                },
                "TimeInstant": {
                    "type": "DateTime",
                    "value": "2025-01-27T14:30:57.399Z"
                }
            }
        ]
    }
} | comp=IoTAgent

tstorek · 2025-01-27T15:03:12Z

@AlvaroVega hi, okay. that's exaxtly the point apparently the array is processed one by one, but I want to habe the whole array as a single attribute value, because it is a forecast for the next X-hours. Therefore, it should one single update putting the whole thing in one operation. Expected this:

NOTE: We are using the single-measure-mqtt-api. Ergo, one single value at a time

{
                "id": "thing:dev",
                "type": "thing",
                "timestamp": {
                    "type": "DateTime",
                    "value": "2025-01-28T06:00:00Z",
                    "metadata": {
                        "TimeInstant": {
                            "type": "DateTime",
                            "value": "2025-01-27T14:30:57.399Z"
                        }
                    }
                },
                "value": {
                    "type": **"Array",**
                    "value": **[
  {"timestamp": "2025-01-27T09:00:00Z", "value": 10.67}, 
  {"timestamp": "2025-01-27T12:00:00Z", "value": 10.2}, 
  {"timestamp": "2025-01-27T15:00:00Z", "value": 9.16}, 
  {"timestamp": "2025-01-27T18:00:00Z", "value": 7.72}, 
  {"timestamp": "2025-01-27T21:00:00Z", "value": 7.39}, 
  {"timestamp": "2025-01-28T00:00:00Z", "value": 6.39}, 
  {"timestamp": "2025-01-28T03:00:00Z", "value": 5.87},
  {"timestamp": "2025-01-28T06:00:00Z", "value": 5.54}
]**,
                    "metadata": {
                        "TimeInstant": {
                            "type": "DateTime",
                            "value": "2025-01-27T14:30:57.399Z"
                        }
                    }
                },
                "TimeInstant": {
                    "type": "DateTime",
                    "value": "2025-01-27T14:30:57.399Z"
                }
            }

AlvaroVega · 2025-01-27T16:03:27Z

Srroy, could you tell me exactly how you send MQTT ? i.e.:

$ mosquitto_pub -h iot-mosquitto -t /json/APIKEY/dev/attrs/ -m '{ "temperatureForecast": 10.2}'

tstorek · 2025-01-27T17:56:18Z

@AlvaroVega sure. From commandline it should be something like:

$ mosquitto_pub -h iot-mosquitto -t /json/APIKEY/dev/attrs/temperatureForecast -m '[
  {"timestamp": "2025-01-27T09:00:00Z", "value": 10.67}, 
  {"timestamp": "2025-01-27T12:00:00Z", "value": 10.2}, 
  {"timestamp": "2025-01-27T15:00:00Z", "value": 9.16}, 
  {"timestamp": "2025-01-27T18:00:00Z", "value": 7.72}, 
  {"timestamp": "2025-01-27T21:00:00Z", "value": 7.39}, 
  {"timestamp": "2025-01-28T00:00:00Z", "value": 6.39}, 
  {"timestamp": "2025-01-28T03:00:00Z", "value": 5.87},
  {"timestamp": "2025-01-28T06:00:00Z", "value": 5.54}
]'

AlvaroVega · 2025-01-28T09:31:17Z

I guess you really are sendint something like:

(HTTP):

 curl -i -X POST 'http://localhost:7897/iot/json?i=dev&k=APIKEY' -d '[  {"timestamp": "2025-01-27T09:00:00Z", "temperatureForecast": 10.67}, 
  {"timestamp": "2025-01-27T12:00:00Z", "temperatureForecast": 10.2}, 
  {"timestamp": "2025-01-27T15:00:00Z", "temperatureForecast": 9.16}, 
  {"timestamp": "2025-01-27T18:00:00Z", "temperatureForecast": 7.72}, 
  {"timestamp": "2025-01-27T21:00:00Z", "temperatureForecast": 7.39}, 
  {"timestamp": "2025-01-28T00:00:00Z", "temperatureForecast": 6.39}, 
  {"timestamp": "2025-01-28T03:00:00Z", "temperatureForecast": 5.87},
  {"timestamp": "2025-01-28T06:00:00Z", "temperatureForecast": 5.54}
]' -H 'content-type: application/json'

MQTT:

mosquitto_pub -h localhost -t /json/APIKEY/dev/attrs -m '[ {"timestamp": "2025-01-27T09:00:00Z", "temperatureForecast": 10.67}, 
  {"timestamp": "2025-01-27T12:00:00Z", "temperatureForecast": 10.2}, 
  {"timestamp": "2025-01-27T15:00:00Z", "temperatureForecast": 9.16}, 
  {"timestamp": "2025-01-27T18:00:00Z", "temperatureForecast": 7.72}, 
  {"timestamp": "2025-01-27T21:00:00Z", "temperatureForecast": 7.39}, 
  {"timestamp": "2025-01-28T00:00:00Z", "temperatureForecast": 6.39}, 
  {"timestamp": "2025-01-28T03:00:00Z", "temperatureForecast": 5.87},
  {"timestamp": "2025-01-28T06:00:00Z", "temperatureForecast": 5.54}
]'

In MQTT case CB updates are done one by one. This PR enables sent to CB in just one shot: #857

@tstorek thanks for report!

tstorek · 2025-01-28T09:57:02Z

@AlvaroVega I do not exactly understand. Just to clarify. The value of the attribute contains the whole Array in the sent message.
The format of the array item are object, but this is also not further specified in NGSIv2. I thought this was the most naiv approach.

We want to have the whole list in one single attribute in CB. Thus, the cb should show it like this:

"temperatureForecast": {
    "type": "Array",
    "value": [
        {"timestamp": "2025-01-27T09:00:00Z", "value": 10.67}, 
        {"timestamp": "2025-01-27T12:00:00Z", "value": 10.2}, 
        {"timestamp": "2025-01-27T15:00:00Z", "value": 9.16}, 
        {"timestamp": "2025-01-27T18:00:00Z", "value": 7.72}, 
        {"timestamp": "2025-01-27T21:00:00Z", "value": 7.39}, 
        {"timestamp": "2025-01-28T00:00:00Z", "value": 6.39}, 
        {"timestamp": "2025-01-28T03:00:00Z", "value": 5.87},
        {"timestamp": "2025-01-28T06:00:00Z", "value": 5.54}
   ]
}

fgalan · 2025-01-28T15:26:09Z

PR #858 with a fix for this has been merged.

@tstorek could you test again using iotagent-json:lastest image and tell us if it works as expected (or, in negative case, provided the result you get and the expectation), please?

Thanks in advance!

tstorek · 2025-01-29T07:31:26Z

@AlvaroVega @fgalan I appreciate you quick reaction. However, from my point, the solution will still lead to an unexpected behavior if the array only contains 1 element.
Automatically altering the DataType is something a user does not expect. Especially, when the attribute is clearly defined as Array.

While this is necessary for the multi-measurement api for the single one it is relatively nasty.

AlvaroVega · 2025-01-29T09:55:31Z

@AlvaroVega @fgalan I appreciate you quick reaction. However, from my point, the solution will still lead to an unexpected behavior if the array only contains 1 element. Automatically altering the DataType is something a user does not expect. Especially, when the attribute is clearly defined as Array.

While this is necessary for the multi-measurement api for the single one it is relatively nasty.

If you send just one element:

 mosquitto_pub -h localhost -t /json/APIKEY/dev1/attrs/temperatureForecast -m '[ 
  {"timestamp": "2025-01-29T09:00:00Z", "value": 44 }
]'

CB receives a single jSON:

"value": {
                "timestamp": "2025-01-29T09:00:00Z",
                "value": 44
            },

time=2025-01-29T09:48:03.796Z | lvl=DEBUG | corr=b3d4b7ea-c3eb-46b2-8567-8fc7f30a6511 | trans=b3d4b7ea-c3eb-46b2-8567-8fc7f30a6511 | op=IoTAgentNGSI.Request | from=n/a | srv=smartcity | subsrv=/ | msg=Options: {
"url": "http://iot-orion:1026/v2/entities?options=upsert,flowControl",
"method": "POST",
"headers": {
"fiware-service": "smartcity",
"fiware-servicepath": "/"
},
"json": {
"id": "thing:dev1",
"type": "thing",
"temperatureForecast": {
"type": "Array",
"value": {
"timestamp": "2025-01-29T09:00:00Z",
"value": 44
},
"metadata": {
"TimeInstant": {
"type": "DateTime",
"value": "2025-01-29T09:48:03.796Z"
}
}
},
"TimeInstant": {
"type": "DateTime",
"value": "2025-01-29T09:48:03.796Z"
}
}
} | comp=IoTAgent

The point here is that type:Array is just informative, really this does nothing in CB.

fgalan · 2025-01-29T10:09:35Z

The point here is that type:Array is just informative, really this does nothing in CB.

Exactly. As explained in the Orion API documentation

Generally speaking, user-defined attribute types are informative; they are processed by Orion in an opaque way

Only a few types (explained in that reference) have an special semantic in the NGSI API (and Array is not one of them)

tstorek · 2025-01-29T11:53:51Z

@fgalan @AlvaroVega I totally understand that type:Array is purely informative. Still, if I send an Array I also expect an Array don't I? I am not sure how to feel about this. Basically, it means that i need to check in my client code all the time if it is a single element or an Array of multiple.

Just a suggestion, restain the send type in the single measurement. Thus even an empty array [] would be allowed and indicate that there is currently nothing evailable :) Maybe, this would be a further improvement.

In the MultiMeasurement, one would do the same:

 $ mosquitto_pub -h localhost -t /json/APIKEY/dev1/attrs -m '"{
temperatureForecast": [ 
  {"timestamp": "2025-01-29T09:00:00Z", "value": 44 }
  ]
}'

As I understand it in the code (I'm not Node.js programmer) this requires another hierachical level in in message parsing.

Check if multi-measurement with containtining high level array.
Parse high-level array one by one
If the single items contain an array as value just forward the array

I think that steps 2-3 are currently lumped together with leads to the current behavior.

AlvaroVega · 2025-01-31T11:46:14Z

@tstorek maybe you can try it with latest version again.

tstorek added the bug label Jan 27, 2025

github-actions bot added the NGSIv2 label Jan 27, 2025

AlvaroVega changed the title ~~Unexpected parsing of Array of Objects~~ Unexpected parsing of Array of Objects (MQTT measure) Jan 28, 2025

AlvaroVega mentioned this issue Jan 28, 2025

send multiple MQTT measures in a batch update to CB #857

Merged

AlvaroVega mentioned this issue Jan 28, 2025

allow MQTT single array measures #858

Merged

2 tasks

AlvaroVega mentioned this issue Jan 30, 2025

Allow single array MQTT measures of 1 element handed like array #860

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unexpected parsing of Array of Objects (MQTT measure) #856

Unexpected parsing of Array of Objects (MQTT measure) #856

tstorek commented Jan 27, 2025

AlvaroVega commented Jan 27, 2025

tstorek commented Jan 27, 2025 •

edited

Loading

AlvaroVega commented Jan 27, 2025 •

edited

Loading

tstorek commented Jan 27, 2025

AlvaroVega commented Jan 28, 2025 •

edited

Loading

tstorek commented Jan 28, 2025 •

edited

Loading

fgalan commented Jan 28, 2025

tstorek commented Jan 29, 2025

AlvaroVega commented Jan 29, 2025

fgalan commented Jan 29, 2025 •

edited

Loading

tstorek commented Jan 29, 2025 •

edited

Loading

AlvaroVega commented Jan 31, 2025

Unexpected parsing of Array of Objects (MQTT measure) #856

Unexpected parsing of Array of Objects (MQTT measure) #856

Comments

tstorek commented Jan 27, 2025

IoT Agent JSON version the issue has been seen with

Bound or port used (API interaction)

NGSI version

Are you running a container?

Image type

Expected behaviour you didn't see

Unexpected behaviour you saw

Steps to reproduce the problem

Configs

Log output

AlvaroVega commented Jan 27, 2025

tstorek commented Jan 27, 2025 • edited Loading

AlvaroVega commented Jan 27, 2025 • edited Loading

tstorek commented Jan 27, 2025

AlvaroVega commented Jan 28, 2025 • edited Loading

tstorek commented Jan 28, 2025 • edited Loading

fgalan commented Jan 28, 2025

tstorek commented Jan 29, 2025

AlvaroVega commented Jan 29, 2025

fgalan commented Jan 29, 2025 • edited Loading

tstorek commented Jan 29, 2025 • edited Loading

AlvaroVega commented Jan 31, 2025

tstorek commented Jan 27, 2025 •

edited

Loading

AlvaroVega commented Jan 27, 2025 •

edited

Loading

AlvaroVega commented Jan 28, 2025 •

edited

Loading

tstorek commented Jan 28, 2025 •

edited

Loading

fgalan commented Jan 29, 2025 •

edited

Loading

tstorek commented Jan 29, 2025 •

edited

Loading