Changing managed thread associated with "main" realm context

[peppy]

In our usage scenario (a game framework), we have the functionality for a user to change the game's execution mode from single-thread to multi-threaded at runtime. This stops all running threads (managed by us) and reallocates how the game's subsystems are run. An example of what happens is:

Start in multi-threaded:

Input (managed thread 0, startup thread / window loop)

runFrames()
{
    while(true) { runInputFrame(); }
}

Draw (managed thread 1)

runFrames()
{
    while(true) { runDrawFrame(); }
}

Update (managed thread 2, realm "main" context)

runFrames()
{
    while(true) {
        realm.Refresh();
        runUpdateFrame();
    }
}

Audio (managed thread 3)

runFrames()
{
    while(true) { runAudioFrame(); }
}

User switches to single-thread:

Input (managed thread 0, startup thread / window loop)

runFrames()
{
    while(true) {
        runInputFrame();

        // input thread additionally gains responsibility for other task areas

        realm.Refresh(); // exception due to managed thread changing.
        runUpdateFrame();

        runDrawFrame();
        runAudioFrame();
    }
}

Each time the mode switches, the update thread exposed to the game code changes its underlying managed thread ID, which causes realm to fall over due to the check:

realm-dotnet/Realm/Realm/Native/SynchronizationContextScheduler.cs

Line 72 in 903b4d0

internal bool IsOnContext(Scheduler other) => _context == (other?._context ?? SynchronizationContext.Current) || _threadId == (other?._threadId ?? Environment.CurrentManagedThreadId);

Note that the same applies switching in the other direction, as we spin up new threads each time and the OS generally doesn't give us matching thread IDs.

---

I'm currently working on a local solution which pins the managed thread for the update thread (where we intend to keep realm), with a proof-of-concept implementation confirmed to fix  the realm-side issue but also reducing the flexibility at our disposal (we can no longer run the update thread on the OS entry thread, which may be desirable on some platforms to get more correct window redraw timings)

From the linked realm code, there doesn't look like any room for play at our end, and I don't necessarily expect realm to be exposing this kind of low level functionality - our use case is likely a very edge case in that most applications aren't managing threads at the level of complexity we are - but I thought I would open this issue to document what we are working through.

[nirinchev]

The way I see it, you have two options:

1. Close and reopen the Realm instance when the user switches to-and-from multithreaded execution. The new realm instance will be associated with the new thread, so you'll be able to access it normally.
2. Create a synchronization context and install it on your threads. It will then take precedence over the thread id in that IsOnContext check. There are two considerations when rolling that out:
   2.1. It may affect where your async continuations are dispatched. You will probably want to do something clever and capture the original thread to make sure you dispatch continuations from the draw thread back to the draw thread.
   2.2. You need to ensure no parallel access is happening for the same Realm instance. I.e. if your sync context is managing all the threads, you need to ensure you only access the same Realm from one of them.

If you're going with option 2., you may also want to considering installing a synchronization context only on the main thread in single-thread mode and then reinstalling it on the update thread where Realm is going to be used.

[nirinchev]

Can you log Thread.CurrentThread.ManagedThreadId from both task closures? Since you're using threadpool tasks, it's possible that you're getting the same thread for both Task.Run invocations. I was testing it with .NET Core 3.1 and I got the exception on almost every run - it's possible that the .NET 5.0 runtime does more aggressive thread reuse which is why you need a lot more loops to get a new thread.

[peppy]

Something definitely still seems fishy:

using System;
using System.IO;
using System.Threading;
using System.Threading.Tasks;
using Realms;

namespace ConsoleApp2
{
    class Program
    {
        static void Main(string[] args)
        {
            int i = 1;

            while (true)
            {
                Console.WriteLine($"loop count {i++}");
                Realm realm = null;

                Task.Factory.StartNew(() =>
                {
                    Console.WriteLine($"Thread {Thread.CurrentThread.ManagedThreadId}");

                    realm = Realm.GetInstance(new RealmConfiguration(Path.GetTempFileName()));
                    realm.Write(() => realm.Add(new BeatmapInfo()));
                    realm.Refresh();
                }, TaskCreationOptions.LongRunning).Wait();

                Task.Factory.StartNew(() =>
                {
                    Console.WriteLine($"Thread {Thread.CurrentThread.ManagedThreadId}");

                    // expected one of these to crash as this context was opened on another thread?
                    realm.Write(() => realm.Add(new BeatmapInfo()));
                    realm.Refresh();
                    realm.Dispose(); // added just to be sure it wasn't some kind of delayed disposal issue.
                }, TaskCreationOptions.LongRunning).Wait();
            }
        }
    }

    class BeatmapInfo : RealmObject
    {
        public int Test { get; set; }
    }
}

❯ dotnet run
loop count 1
Thread 4
Thread 6
loop count 2
Thread 7
Thread 8
loop count 3
Thread 9
Thread 10
loop count 4
Thread 11
Thread 12
loop count 5
Thread 13
Thread 14
loop count 6
Thread 15
Thread 16
loop count 7
Thread 17
Thread 19
loop count 8
Thread 20
Thread 21
Unhandled exception. System.AggregateException: One or more errors occurred. (Realm accessed from incorrect thread.)
 ---> System.Exception: Realm accessed from incorrect thread.
   at Realms.NativeException.ThrowIfNecessary(Func`2 overrider)
   at Realms.SharedRealmHandle.BeginTransaction()
   at Realms.Transaction..ctor(Realm realm)
   at Realms.Realm.BeginWrite()
   at Realms.Realm.Write[T](Func`1 function)
   at ConsoleApp2.Program.<>c__DisplayClass0_0.<Main>b__1() in /Users/dean/Projects/ConsoleApp2/ConsoleApp2/Program.cs:line 34
   at System.Threading.Tasks.Task.InnerInvoke()
   at System.Threading.Tasks.Task.<>c.<.cctor>b__277_0(Object obj)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.Wait()
   at ConsoleApp2.Program.Main(String[] args) in /Users/dean/Projects/ConsoleApp2/ConsoleApp2/Program.cs:line 29

[nirinchev]

That is super peculiar, indeed - I don't have time to look into it more, but my best guess is that for some reason the native thread Id that we're getting in both task closures is the same, but this is the first time I've observed such behavior. If you do want to get consistent repros though, you can remove the second Task.Factory.StartNew and instead try to access the Realm on the original thread. With this modification I get the exception every loop iteration:

using System;
using System.IO;
using System.Threading;
using System.Threading.Tasks;
using Realms;

namespace ConsoleApp2
{
    class Program
    {
        static void Main(string[] args)
        {
            int i = 1;

            while (true)
            {
                try
                {
                    Console.WriteLine($"loop count {i++}");
                    Realm realm = null;

                    Task.Factory.StartNew(() =>
                    {
                        Console.WriteLine($"Thread {Thread.CurrentThread.ManagedThreadId}");

                        realm = Realm.GetInstance(new RealmConfiguration(Path.GetTempFileName()));
                        realm.Write(() => realm.Add(new BeatmapInfo()));
                        realm.Refresh();
                    }, TaskCreationOptions.LongRunning).Wait();

                    Console.WriteLine($"Thread {Thread.CurrentThread.ManagedThreadId}");

                    // expected one of these to crash as this context was opened on another thread?
                    realm.Write(() => realm.Add(new BeatmapInfo()));
                    realm.Refresh();
                    Console.WriteLine("This should not be hit!");
                }
                catch (Exception ex)
                {
                    Console.WriteLine($"An error occurred: {ex.Message}");
                }
            }
        }
    }

    class BeatmapInfo : RealmObject
    {
        public int Test { get; set; }
    }
}

[nirinchev]

Here's a slightly modified version that seems to support my theory. In this version, I'm starting the first task and looping it until we get the exception so that the runtime cannot reuse the thread when the second task is started. I believe this should be realistic in your scenario where you have multiple threads running in parallel rather than having them execute their work items sequentially.

using System;
using System.IO;
using System.Threading;
using System.Threading.Tasks;
using Realms;

namespace ConsoleApp2
{
    class Program
    {
        static void Main(string[] args)
        {
            int i = 1;

            while (true)
            {
                var cts = new CancellationTokenSource();
                try
                {
                    Console.WriteLine($"loop count {i++}");
                    Realm realm = null;


                    Task.Factory.StartNew(() =>
                    {
                        Console.WriteLine($"Thread {Thread.CurrentThread.ManagedThreadId}");

                        realm = Realm.GetInstance(new RealmConfiguration(Path.GetTempFileName()));
                        realm.Write(() => realm.Add(new BeatmapInfo()));

                        while (!cts.IsCancellationRequested)
                        {
                            Task.Delay(1).Wait();
                            realm.Refresh();
                        }
                    }, cts.Token, TaskCreationOptions.LongRunning, TaskScheduler.Default);

                    Task.Factory.StartNew(() =>
                    {
                        while (realm == null)
                        {
                            Task.Delay(1).Wait();
                        }
                        Console.WriteLine($"Thread {Thread.CurrentThread.ManagedThreadId}");

                        // expected one of these to crash as this context was opened on another thread?
                        realm.Write(() => realm.Add(new BeatmapInfo()));
                        realm.Refresh();
                        Console.WriteLine("This should not be hit!");
                    }, TaskCreationOptions.LongRunning).Wait();
                }
                catch (Exception ex)
                {
                    Console.WriteLine($"An error occurred: {ex.Message}");
                }
                finally
                {
                    cts.Cancel();
                }
            }
        }
    }

    class BeatmapInfo : RealmObject
    {
        public int Test { get; set; }
    }
}

[peppy]

Yep, that should be ample as a test setup for now, just wanted to follow through on this as it may be an actual incorrect check at realm's side.

For my own sanity, is the check in

realm-dotnet/Realm/Realm/Native/SynchronizationContextScheduler.cs

Line 72 in 903b4d0

internal bool IsOnContext(Scheduler other) => _context == (other?._context ?? SynchronizationContext.Current) || _threadId == (other?._threadId ?? Environment.CurrentManagedThreadId);

not responsible for the exception in this case? My goal here was to test and understand if the SynchronizationContext solution is going to work for us, but this discovery is pointing to realm potentially checking native thread IDs internally, even if managedThreadId and syncContext matches. Figured this out, was an issue at my end.

Thanks for your help, I have the SynchronizationContext approach working very well now.