New Metadata in post-game stats

[filiptod]

There have been a few new points of data added to the post-game screen that don't seem to appear in the meta file using S2DotaMatchMetadata.CDOTAMatchMetadataFile. The new data is the player's Seasonal Rank Medal, Buffs and Pick (with a star/dice to indicate bonus or randomed hero) in the Scoreboard tab, and Win Probability in Graphs under Team XP And Net Worth. I couldn't find the data in the current build so I was wondering if they just weren't added yet. Thanks a lot!

[spheenik]

I just checked, the definition of S2DotaMatchMetadata.CDOTAMatchMetadataFile used by the current clarity-protobuf (4.9) is up to date. So I lean towards "not added yet".

If you want, you can monitor the proto definition and give me a headsup once it shows up.

[filiptod]

Hi @spheenik, thanks for the reply! I just checked the proto and it seems it was added last week, but they separated it into a different message S2DotaMatchMetadata.CDOTAMatchPrivateMetadata.

[spheenik]

have a link to the proto?

[TodorovicF]

Sure, right here.

[spheenik]

Should all be there since 4.9.

Are you really using the new protobufs?

[TodorovicF]

Hmm, it seems the private metadata is there, but I couldn't find a method that returns a CDOTAMatchPrivateMetadata object like Clarity.metadataForFile(filepath) does for CDOTAMatchMetadataFile. I tried to work around it by converting a file to byte array and passing it to S2DotaMatchMetadata.CDOTAMatchPrivateMetadata.parseFrom(byte[] data) but all I get is something like this:

1: 1
2: 3785291298
3: "\n\232\351\001\b\002\022\324*\b\215\216\3672\020\322)\020.....
5: "OV\004M<\264\227\026\026e5yw\212\317(\237\027,\261=.....

and methods like getGraphWinProbabilityCount() always return 0 or an empty list. This is the only way I'm able to get something to output, but maybe i'm doing something wrong:

        Path path = Paths.get(metaFile);
        byte[] data = Files.readAllBytes(path);
        S2DotaMatchMetadata.CDOTAMatchPrivateMetadata privateMeta = S2DotaMatchMetadata.CDOTAMatchPrivateMetadata.parseFrom(data);
        System.out.println(privateMeta.getGraphWinProbabilityCount());

I checked my versions and I am using clarity: 2.2 with clarity-protobuf: 4.9.

[spheenik]

Your way to get a CDOTAMatchPrivateMetadata from some raw data is correct, but the output you show proves that the data was not valid. If you say "I tried to work around it by converting a file to byte array...", what exactly is that file? How did you get it?

[spheenik]

also see #123.

[TodorovicF]

It's strange because I'm using the same method to download .meta files and am using current match IDs that I checked have the extra data in the Dota 2 client. These are the matches I've tried that all display the same data structure:

3785291298
3787212636
3787221092
3790963508
3790811718

I am still able to parse the regular CDOTAMatchMetadataFile data and all the data is available and valid. Also, when I output the regular CDOTAMatchMetadataFile data, at the very end I get similar strange-looking data in the key private_metadata: "\356\274\321\312>\003*\315\366\212\374\277_\021....". So I'm not sure if that's related in any way.

Here're two gists that show the two different outputs for the same match ID:

metadata: https://gist.github.com/TodorovicF/772c868d7561bec00f91547dfe2ca9e1
private metadata: https://gist.github.com/TodorovicF/521da383646f52dfc2c1f9e4a28eafd7

I'll keep trying to figure out what is going wrong in the meantime and post back if I come up with anything.

[spheenik]

Looks like you should be able to supply the ByteString of private_metadata to S2DotaMatchMetadata.CDOTAMatchPrivateMetadata.parseFrom(*byteString*) to get to it.
Did you try that already?

[spheenik]

Would be nice if you'd supply the file you got the metadata from.

[TodorovicF]

I tried supplying a ByteString with ByteString.copyFrom(data) "data" being the byte array file, and I get the exact same output. I can give you the files, would you like me to email them or how do you prefer?

I also tried supplying a ByteString by first getting the regular metadata and then using the getPrivateMetadata() function:

S2DotaMatchMetadata.CDOTAMatchMetadataFile metadata = Clarity.metadataForFile(metaFile);
S2DotaMatchMetadata.CDOTAMatchPrivateMetadata privateMeta = S2DotaMatchMetadata.CDOTAMatchPrivateMetadata.parseFrom(metadata.getPrivateMetadata());

But I get a protobuf exception when parsing:

Exception in thread "main" com.google.protobuf.InvalidProtocolBufferException: Protocol message tag had invalid wire type.
    at com.google.protobuf.InvalidProtocolBufferException.invalidWireType(InvalidProtocolBufferException.java:99)
    at com.google.protobuf.UnknownFieldSet$Builder.mergeFieldFrom(UnknownFieldSet.java:515)
    at com.google.protobuf.GeneratedMessage.parseUnknownField(GeneratedMessage.java:205)
    at skadistats.clarity.wire.s2.proto.S2DotaMatchMetadata$CDOTAMatchPrivateMetadata.<init>(S2DotaMatchMetadata.java:13871)
    at skadistats.clarity.wire.s2.proto.S2DotaMatchMetadata$CDOTAMatchPrivateMetadata.<init>(S2DotaMatchMetadata.java:13828)
    at skadistats.clarity.wire.s2.proto.S2DotaMatchMetadata$CDOTAMatchPrivateMetadata$1.parsePartialFrom(S2DotaMatchMetadata.java:13942)
    at skadistats.clarity.wire.s2.proto.S2DotaMatchMetadata$CDOTAMatchPrivateMetadata$1.parsePartialFrom(S2DotaMatchMetadata.java:13937)
    at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:104)
    at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:124)
    at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:129)
    at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
    at skadistats.clarity.wire.s2.proto.S2DotaMatchMetadata$CDOTAMatchPrivateMetadata.parseFrom(S2DotaMatchMetadata.java:21645)
    ...

[spheenik]

Well, shit :(

If you get an exception here, the binary might be packed, or be something else completely.

You can create a gist for the files, or send them to [email protected]

Cheers!

[spheenik]

I had an old meta file lying around, and I did some testing on it, and it seems that private_metadata is not decodable as a protobuf message. It's encrypted somehow :(

[TodorovicF]

I had a feeling that was the case since part of it is Dota Plus exclusive. It's a shame, but thanks a lot for looking into it!

[spheenik]

Well, the meta file was from loooong before Dota Plus, but yea, as the name says, "private".

[Dmytro4]

Hey, is that info should be extracted only from the replay file or there is a separate small file that can be downloaded somehow?
Actually, a lot of info from this class is shown in the game client when you open the match info without accessing the replay file. Is there a known way to get it not from the replay file?
UPDATE - Found the way to get this file from this comment - Arcana/node-dota2#330 (comment)

[spheenik]

Hey that's cool. Didn't know that yet.

[thedanill]

It may be a little late, but we've analyzed the encryption algorithm of private_metadata.
Currently, Python and C# implementations are available. I am not very familiar with Java, but perhaps I'll write an implementation at a later date.

[spheenik]

Only 3 years late haha!
I'm gonna look when I have the time.
Thanks!