Object store publishes chunks without using domain in subject

[scottf]

Steps to recreate

CLI way of doing the file dance
add --trace to any of these for more verbose output

create the store on hub
nats --server nats://testUser:testPass@localhost:4222 --js-domain HUB object add CliStore

upload file with js domain HUB on hub
nats --server nats://testUser:testPass@localhost:4222 --js-domain HUB object put CliStore ./nats/tmp.txt

check it exists from both servers views using the js domain of HUB
nats --server nats://testUser:testPass@localhost:4222 --js-domain HUB object ls CliStore
nats --server nats://testUser:testPass@localhost:4223 --js-domain HUB object ls CliStore

this can prove the file is stored on hub and not spoke. look at mem and file in use under Jetstream
nats --server nats://admin:admin@localhost:4222 server info
nats --server nats://admin:admin@localhost:4223 server info

spoke should be able to download the file if the HUB domain is used
nats --server nats://testUser:testPass@localhost:4223 --js-domain HUB object get CliStore nats/tmp.txt -O spokeout.txt

spoke should be able to delete the file if the HUB domain is used
nats --server nats://testUser:testPass@localhost:4223 --js-domain HUB object del CliStore nats/tmp.txt -f

spoke should be able to upload the file if the HUB domain is used
nats --trace --server nats://testUser:testPass@localhost:4223 --js-domain HUB object put CliStore ./nats/tmp.txt

hub should be able to download the file that the spoke uploaded
nats --server nats://testUser:testPass@localhost:4222 --js-domain HUB object get CliStore nats/tmp.txt -O hubout.txt

hub should be able to delete the file too
nats --server nats://testUser:testPass@localhost:4222 --js-domain HUB object del CliStore nats/tmp.txt -f

Expected behavior

When a js-domain is specified, objects are published to the domain meaning the publish subject has the domain in the api.

Server and client version

Latest CLI, Server N/A

Based on

nats-io/nats.java#1157

[ripienaar]

Go code that reproduce the problem. Check with a sub on $O.> and you will see wrong subjects.

https://gist.github.com/ripienaar/554b9679983bf5cba6da6a75e7ea844f

[scottf]

@ripienaar @piotrpio The go client might be correct but I would verify.

I found a bug in the java/.net v1 libraries. On the connection to the leaf, it's correct to use the js domain information on js api calls. But for a consumer, the actual filter subject of the consumer of chunks is just the normal $O subject, not some js api subject.

My code was calling pubSubChunkSubject which gives that full domain specific subject. But it should have called my rawChunkSubject which is just normal.

https://github.com/nats-io/nats.java/pull/1160/files#diff-4fe4c868c242b0c63faee28649d7af470fcd3de4d3627929d911e3557d751663L236

[ripienaar]

Did you implement the behaviour described here @scottf ?

this seems to suggest yes - and server is doing mapping on those subjects.

I never really bought into this design but anyway seems to be what you have?

https://github.com/nats-io/nats-architecture-and-design/blob/main/adr/ADR-19.md#kv-example

[scottf]

I think I followed the KV spec
My problem was when creating a consumer, I treated the consumer filter subject as something that had to be domainified. Creating the consumer itself is done correctly with a domainified CREATE_CONSUMER