Issue with NGSI-LD and IoTAgent JSON

[ccndukwe]

Please I need urgent help.

IoT Agent JSON seems to be incompatible with NGSI-LD. I tried Scorpio broker and command was not working. I saw that there is open issue on git. I switched to Orion-LD and still command the same issues. Please help.

HTTP/1.1 404

{
"type": "https://uri.etsi.org/ngsi-ld/errors/ResourceNotFound",
"title": "Resource not found.",
"details": "Provided attribute is not present"
}

[fgalan]

Maybe @jason-fox could provide help

[jason-fox]

This issue holds insufficient information to be able to diagnose an issue. What is the set up? What provisioning have you done? Which call to the context broker is returning the error message described above.

All IoT Agents are using the same library (and it is the library that deals with North port communication into the context broker) so there is no reason why if it works on Ultralight it shouldn't do the same on JSON.

Which Docker images are you using? Which tagged versions have you pulled? A working JSON IoT Agent - NGSI-LD broker set up can be found here: https://github.com/FIWARE/tutorials.IoT-Agent-JSON/tree/NGSI-LD

[ccndukwe]

I'm using latest version of scorpio broker. I saw that other persons also reported the issue online:
https://githublab.com/profile/mohitNEC
#538

I think the issue is that NGSI-LD does not support commands. Therefore unlike NGSI-V2, the command attribute during provisioning of devices on IoT Agent JSON is not being registered as commands on the Scorpio Broker and hence the attribute is not even registered as an attribute of the entity hence the error.

I searched online and saw a number of persons making similar complain. In one of the comments someone said:

@mohitNEC wrote "Hi,
There is an NGSI-LD version issue while integrating Scorpio Broker with IoT Agent. Currently, Scorpio Broker is using NGSI-LD v1.3.3 NGSI-LD does not supports the type: "command" , as the issue regarding this has already been raised
#538
IoT Agent Integration is not part of Development branch so after IoT Agent updated then we will merge it in the Development Branch.
Thanks"

[Blobonat]

I haven't tried it myself (yet), but using bidirectional attributes (scroll down to Bidirectionality Syntax) looks like a promising way to work around this issue, since it is based on subscriptions which shouldn't be a problem with Scorpio.

[jason-fox]

@Blobonat is correct - bidirectional attributes is a workaround for this.

The issue arrises due to a fundamental disagreement between Orion (NGSI-v2) and Scorpio (NGSI-LD) on what constitutes a true federation (and how to achieve it using registrations). Commands in an IoT Agent work in Orion (and Orion-LD) because it is possible for a web service to register an attribute and for that web service to offer a limited degree of NGSI-LD endpoints. However Scorpio currently assumes that only full context brokers can register and doing so adds them to a federation.

If you look into the Scorpio logs prior to theProvided attribute is not present error, you will see Scorpio has attempted to make a call to the /ngsi-ld/v1/entities/ endpoint which the IoT Agent does not support, never has supported and never will since it isn't a full context broker. This is actually quite simple to fix - there just needs to be an agreement across the ETSI NGSI-LD committee to provide a flag to differentiate between "Scorpio-style" true federation and "Orion-style" command attributes so that Scorpio no longer assumes that all regsitration participants are full brokers.

Unfortunately the NGSI-LD specification 1.5.1 is currently ambiguous on this point. Hopefully the 1.6.1 will clarify the behaviour around how context brokers should actually cover these two options around registrations. "Orion-style" attributes are essential to be able to provide lazy attributes like device batteryLevel, and are a de-facto standard for IoT Agent commands. "Scorpio-style" all participants are full brokers assumptions are essential for federation to work.

Bidirectional attributes are different, because they are based around subscriptions, the receiver of the notification (in this case the IoT Agent) doesn't need to offer a full context broker interface - indeed all that a command does in this case is receive notifications and upsert status back to the context broker. The NGSI-LD specification is much clearer on how NGSI-LD notifications work - it is precisely the same as NGSI-v2 notifications, so Orion, Orion-LD, Scorpio and all other brokers can happily send a notification to an IoT Agent web service and then the IoT Agent can do something with it.

[ccndukwe]

The problem is not the inability of Scorpio to send commands, but IoT Agents does not register those commands as entity attributes during Device Provisioning. Hence, attempt to patch the values results to error since such attribute does not yet exist in the entity created by IoT Agent. I think the issue equally affects Orion-LD (with all NGSI-LD) enablers.

[jason-fox]

That is just a question of interpretation - the IoT Agent doesn't hold or register the command attribute or lazy attribute as you suggest because it believes that providing that attribute is the sole responsibility of the device itself. IoT Agent says "This is not my problem, I am just a limited web service". If Scorpio made the same assumptions as Orion and Orion-LD around commands, IoT Agent commands would work with Scorpio - this tutorial is a working IoT Agent for JSON working with Orion-LD and NGSI-LD and this tutorial is the same provisioning commands working with Orion classic and NGSI-v2.

The steps to fix this are as follows:

1. The ETSI NGSI-LD committee defines what should happen across brokers.
2. The Scorpio codebase is updated to accept registrants which are not full brokers
3. The Orion-LD codebase is updated to accept true federations
4. The IoT Agent Node lib is updated to provide whatever flags are necessary when making a registration to describe its implementation as a partial broker.
5. A stable release of the IoT Agent Node lib is completed and added to the IoT Agent for JSON
6. A stable release of the IoT Agent for JSON is completed.

[ccndukwe]

Thank you so much for the prompt responses and all the explanations.

I actually followed the example you mentioned (though not with PDI-IoTA-UltraLight, but JSON) and I keep having this problem. I have read all the documentation I could possibly access online. Sincerely it has actually taken so much of my time trying to fix it and yet I couldn't.

[jason-fox]

An updated NGSI-LD 1.6.1 Specification has been published, so the ETSI NGSI-LD committee has now defined what should happen across brokers. (point 1.) I have raised telefonicaid/iotagent-node-lib#1287 to deal with point 4 from the list above. Once various brokers have also made changes (2 + 3) then this issue can be progressed further.

[mapedraza]

After merging:

• Update to offer NGSI-LD 1.6.1. Registration iotagent-node-lib#1302

Should this be closed?
CC: @jason-fox

[jason-fox]

• The ETSI NGSI-LD committee defines what should happen across brokers.
• The Scorpio codebase is updated to accept registrants which are not full brokers
• The Orion-LD codebase is updated to accept true federations
• The IoT Agent Node lib is updated to provide whatever flags are necessary when making a registration to describe its implementation as a partial broker.
• A stable release of the IoT Agent Node lib is completed and added to the IoT Agent for JSON
• A stable release of the IoT Agent for JSON is completed.

Looks like this can be closed now @mapedraza