Deserializing world only works within the same process as the serialization

[dacete]

I'm trying to serialize and deserialize a World, using Arch.Persistence. If I do it in the same process, it works fine. But if I send that byte array over to a different client and try to deserialize there, all hell breaks loose. In order to not use some pipe or networking I just write the byte array to a file and read it again after restarting the app, as a simple repro scenario.

In my FlaxEngine project, this causes the client to crash with a
the thread tried to read from or write to a virtual address for which it does not have the appropriate access
error inside the minidump.

In the repro C# console Net7 project I get

MessagePack.MessagePackSerializationException: Failed to deserialize Arch.Core.World value.
 ---> System.ArgumentNullException: Value cannot be null. (Parameter 'elementType')
   at System.Array.CreateInstance(Type elementType, Int32 length)
   at Arch.Core.Utils.ArrayRegistry.GetArray(ComponentType type, Int32 capacity)
   at Arch.Core.Chunk..ctor(Int32 capacity, Int32[] componentIdToArrayIndex, ComponentType[] types)
   at Arch.Core.Archetype..ctor(ComponentType[] types)
   at Arch.Core.Extensions.Dangerous.DangerousArchetypeExtensions.CreateArchetype(ComponentType[] types)
   at Arch.Persistence.ArchetypeFormatter.Deserialize(MessagePackReader& reader, MessagePackSerializerOptions options)
   at Arch.Persistence.WorldFormatter.Deserialize(MessagePackReader& reader, MessagePackSerializerOptions options)
   at MessagePack.MessagePackSerializer.Deserialize[T](MessagePackReader& reader, MessagePackSerializerOptions options)
   --- End of inner exception stack trace ---
   at MessagePack.MessagePackSerializer.Deserialize[T](MessagePackReader& reader, MessagePackSerializerOptions options)
   at MessagePack.MessagePackSerializer.Deserialize[T](ReadOnlyMemory`1 buffer, MessagePackSerializerOptions options, CancellationToken cancellationToken)
   at Arch.Persistence.ArchBinarySerializer.Deserialize(Byte[] world)
   at Program.Main(String[] args) in C:\Users\hdgam\source\repos\ConsoleApp5\ConsoleApp5\Program.cs:line 31

Repro steps:

1. Make new console project
2. Add Arch 1.2.7 & Arch.Persistence 1.0.3 via NuGet
3. Use this code:

using Arch.Core;
using Arch.Persistence;

internal class Program
{
    public struct Test
    {
        public int value;
    }
    static void Main(string[] args)
    {
        var input = Console.ReadLine();
        var path = AppDomain.CurrentDomain.BaseDirectory + ".data";
        ArchBinarySerializer serializer = new ArchBinarySerializer();

        if (input.Equals("Server"))
        {
            var world = World.Create(); // new world
            world.Create(new Test { value = 1 }); // only 1 basic entity
            var bytes = serializer.Serialize(world);
            Console.WriteLine(bytes.Select(b => (long)b).Sum()); // this is just to check we read the same array on the client
            File.WriteAllBytes(path, bytes);
        }
        else if (input.Equals("Client"))
        {
            try
            {
                var bytes = File.ReadAllBytes(path);
                Console.WriteLine(bytes.Select(b => (long)b).Sum()); // check we have the same array
                var world = serializer.Deserialize(bytes); // error
            }
            catch (Exception e)
            {
                Console.WriteLine(e.ToString());
            }
        }

        Console.ReadKey();
    }
}

4. Run once, writing "Server"
5. Run again, writing "Client", observe that the number printed is the same, and it should error

[genaray]

Thank you! I also think I already know why. Because the client does not know which components are registered. If you manually register all components in the correct order, it should work.

There is only one small problem. You have to make sure that the order of the components is identical. There is not really a way around that. I can't think of a more elegant way to solve this... e.g.

Server 1: Builds world, registers Transform, Velocity, Sprite
Server 2: Builds world, registers velocity, transform, sprite
Client 1: Receives world, has not registered ERROR
Client 2: Receives world, has registered other order of components, ERROR

We have to see how to fix this. Due to compile-time-statics there is not much room for maneuver here.

[dacete]

Alright, that's an acceptable workaround for my case, thanks for the quick reply!

[richdog]

Just wanted to add that I ran into this issue as well. I will follow the "register in order" approach, but I imagine the type data should actually be saved in the serialized ComponentData. Something like what I did (as a mockup) for the first one in this list:

This (or something like it) is even more necessary when saving data to be loaded back in a different process (save files). Especially when considering it will be hard to guarantee the same order across application versions (ex: if you were to remove a component type) However, much of that is past the scope of this issue. and should be part of a larger discussion on maintaining persistence across application versions.

[emelrad12]

I think a good way to solve this would be to attach a hash of the components and their order, so at least it says order is wrong, and not just crash.

[dacete]

I store the list of registered components along the save file. When loading that I clear the ComponentRegistry (had some reflection hacks because removing a type doesn't subtract some counter (I forget right now)) and load those types from the save file first, and then any new unregistered types.