ConnectionLostException when invoking re-assigned Remote Targets

[pattontim]

Version: 2.6.121
Caller: .NET 4.7.2

Exception:

JsonRpc Error: 10 : Exception thrown from request "68" for method syncElement: StreamJsonRpc.ConnectionLostException: The JSON-RPC connection with the remote party was lost before the request could complete. ---> System.OperationCanceledException: The operation was canceled.
   at System.Threading.CancellationToken.ThrowOperationCanceledException()
   at StreamJsonRpc.MessageHandlerBase.<WriteAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at StreamJsonRpc.JsonRpc.<TransmitAsync>d__159.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Threading.Tasks.ValueTask.ThrowIfCompletedUnsuccessfully()
   at StreamJsonRpc.JsonRpc.<InvokeCoreAsync>d__143.MoveNext()
   --- End of inner exception stack trace ---
   at StreamJsonRpc.JsonRpc.<InvokeCoreAsync>d__143.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at StreamJsonRpc.JsonRpc.<DispatchIncomingRequestAsync>d__147.MoveNext()
JsonRpc Error: 10 : StreamJsonRpc.ConnectionLostException: The JSON-RPC connection with the remote party was lost before the request could complete. ---> System.OperationCanceledException: The operation was canceled.
   at System.Threading.CancellationToken.ThrowOperationCanceledException()
   at StreamJsonRpc.MessageHandlerBase.<WriteAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at StreamJsonRpc.JsonRpc.<TransmitAsync>d__159.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Threading.Tasks.ValueTask.ThrowIfCompletedUnsuccessfully()
   at StreamJsonRpc.JsonRpc.<InvokeCoreAsync>d__143.MoveNext()
   --- End of inner exception stack trace ---
   at StreamJsonRpc.JsonRpc.<InvokeCoreAsync>d__143.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at StreamJsonRpc.JsonRpc.<DispatchIncomingRequestAsync>d__147.MoveNext(), syncElement, 68, Newtonsoft.Json.Linq.JToken[]
JsonRpc Information: 8 : {"id":68,"error":{"code":-32000,"message":"The JSON-RPC connection with the remote party was lost before the request could complete."}}
JsonRpc Verbose: 8 : Sent: {
  "jsonrpc": "2.0",
  "id": 68,
  "error": {
    "data": {
      "type": "StreamJsonRpc.ConnectionLostException",
      "message": "The JSON-RPC connection with the remote party was lost before the request could complete.",
      "stack": "   at StreamJsonRpc.JsonRpc.<InvokeCoreAsync>d__143.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at StreamJsonRpc.JsonRpc.<DispatchIncomingRequestAsync>d__147.MoveNext()",
      "code": -2146233088,
      "inner": {
        "type": "System.OperationCanceledException",
        "message": "The operation was canceled.",
        "stack": "   at System.Threading.CancellationToken.ThrowOperationCanceledException()\r\n   at StreamJsonRpc.MessageHandlerBase.<WriteAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at StreamJsonRpc.JsonRpc.<TransmitAsync>d__159.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Threading.Tasks.ValueTask.ThrowIfCompletedUnsuccessfully()\r\n   at StreamJsonRpc.JsonRpc.<InvokeCoreAsync>d__143.MoveNext()",
        "code": -2146233029,
        "inner": null
      }
    },
    "code": -32000,
    "message": "The JSON-RPC connection with the remote party was lost before the request could complete."
  }
}

Signature of the called method:

Remote 1 starts

Client 1 starts

Client 1 calls client1.Rpc.AddRemoteTarget(remote1)

Client 1 makes some calls to Remote 1.

.....

The Server closes the socket associated with Remote 1, disposes the Rpc.

Remote 2 starts, implementing the same targets.

Client 1 calls client1.Rpc.AddRemoteTarget(remote2)

Client 1 tries to make some calls to Remote 2. However, each call throws a ConnectionLostException whenever the server tries to call any remote method.

[pattontim]

Closing until I check with newer version

[pattontim]

Experience issue with 2.13.33

Methods involved:

  public static bool removeClient(int id)
   {
     if(Clients.TryGetValue(id, out var client)){
       //client.Rpc.Dispose();
       var timeout = new CancellationTokenSource(2500);
       try
       {
         LogTo.Debug("... starting close handshake");
         client.Socket.CloseAsync(WebSocketCloseStatus.NormalClosure, "Closing", timeout.Token)
         .ConfigureAwait(false).GetAwaiter().OnCompleted(() =>
         {
           LogTo.Debug("... close handshake completed");
           client.Rpc.Dispose();
           Clients.TryRemove(id, out _);
           foreach (var c in Clients)
           {
             Task task = c.Value.Rpc.NotifyAsync("removeClient", client.SocketId, client.Name);
             _ = Task.Run(() => task);
           }
         });
       }
       catch (OperationCanceledException ex)
       {
         // normal upon task/token cancellation, disregard
       }      
     }
     return true;
   }

   public static int AddRemoteTarget(string originName, string targetName)
   {
     int originId = GetClientIdByName(originName);
     int targetId = GetClientIdByName(targetName);
     ConnectedClient origin;
     ConnectedClient target;
     if (originId != 0 && Clients.TryGetValue(originId, out origin)
       && targetId != 0 && Clients.TryGetValue(targetId, out target))
     {
       origin.Rpc.AddRemoteRpcTarget(target.Rpc);
       return targetId;
     }
     return -1;
   }

C1 calls AddRemoteTarget("C1","R1"), which succeeds and calls go to R1 successfully.

C1 calls removeClient(id for R1), which succeeds and removes and disposes of R1.

C1 calls AddRemoteTarget("C1","R2") which succeeds. Calls can go to server and succeed with Local invocation, but all calls attempted remotely fail with ConnectionLostException, even though R2 is connected properly.

Info: both client and remote are using simple-jsonrpc with auto-incrementing IDs. However ConnectionLost thrown on server side.

[AArnott]

I'm not sure I can help you here. A verbose log that includes all the relevant messages leading up to disconnection may be helpful. The error message we're seeing here suggests it's the other side (simple-jsonrpc) that's hanging up, so I'd think the investigation would need to start there.