Prevent `ShellDescriptorManager` exception from bringing down the app

[themattkenney]

On one of our Tenants, that is running SQL Lite we had a System.NullReferenceException causing the whole App Pool to lockup requiring an app pool restart.

We are looking into what caused the issue however I would expect an error on one tenant to not lockup the app pool.

Issue Details from app insights:

Call Stack:

System.NullReferenceException:
   at YesSql.QueryExtensions.Query (YesSql.Abstractions, Version=0.0.0.0, Culture=neutral, PublicKeyToken=51cdb8c1938e4789)
   at OrchardCore.Environment.Shell.Data.Descriptors.ShellDescriptorManager+<GetShellDescriptorAsync>d__8.MoveNext (OrchardCore.Infrastructure, Version=1.0.0.0, Culture=neutral, PublicKeyToken=nullOrchardCore.Infrastructure, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: C:\projects\orchardcore\src\OrchardCore\OrchardCore.Infrastructure\Shell\ShellDescriptorManager.csOrchardCore.Infrastructure, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: 52)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at OrchardCore.Environment.Shell.Builders.ShellContextFactory+<OrchardCore-Environment-Shell-Builders-IShellContextFactory-CreateShellContextAsync>d__5.MoveNext (OrchardCore, Version=1.0.0.0, Culture=neutral, PublicKeyToken=nullOrchardCore, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: C:\projects\orchardcore\src\OrchardCore\OrchardCore\Shell\Builders\ShellContextFactory.csOrchardCore, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: 42)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at OrchardCore.Environment.Shell.ShellHost+<GetOrCreateShellContextAsync>d__12.MoveNext (OrchardCore, Version=1.0.0.0, Culture=neutral, PublicKeyToken=nullOrchardCore, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: C:\projects\orchardcore\src\OrchardCore\OrchardCore\Shell\ShellHost.csOrchardCore, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: 87)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at OrchardCore.Environment.Shell.ShellHost+<GetScopeAsync>d__13.MoveNext (OrchardCore, Version=1.0.0.0, Culture=neutral, PublicKeyToken=nullOrchardCore, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: C:\projects\orchardcore\src\OrchardCore\OrchardCore\Shell\ShellHost.csOrchardCore, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: 118)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at OrchardCore.Modules.ModularTenantContainerMiddleware+<Invoke>d__4.MoveNext (OrchardCore, Version=1.0.0.0, Culture=neutral, PublicKeyToken=nullOrchardCore, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: C:\projects\orchardcore\src\OrchardCore\OrchardCore\Modules\ModularTenantContainerMiddleware.csOrchardCore, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: 49)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware+<Invoke>d__9.MoveNext (Microsoft.AspNetCore.Diagnostics, Version=3.1.8.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)

[hishamco]

/cc @jtkech

[sebastienros]

It might be an issue in ShellDescriptorManager. But ideally YesSql should have thrown an ArgumentException if the inputs were not valid. That would help.

[themattkenney]

Update: We were able to figure out why we got the System.NullReferenceException. The issue was in the appsetting.json file, we removed all the unused tenants from our site but missed one tenant. This caused the 500 error when a bot called the site, we removed this tenant, and now no System.NullReferenceException.

@sebastienros if we wanted to block the error from crashing the app pool where would we start?

[Skrypt]

I think you want to wait on @jtkech answer on that one 😄

[sebastienros]

@themattkenney would be awesome to have just a little more details, to be able to repro. Like what to remove if we have two running tenants to get to this issue.

[Skrypt]

I'd guess removing the tenant folder or simply the SQLite database while keeping the tenant config could potentially cause this issue? If not, then maybe the tenant that contains all the other tenants!

[jtkech]

@themattkenney

Maybe in GetShellDescriptorAsync(), add a try catch block around

_shellDescriptor = await _session.Query<ShellDescriptor>().FirstOrDefaultAsync();

So that it returns a null _shellDescriptor and then fallback to a MinimumShellDescriptor, but without failing.

Let us know if it fixes your issue

[themattkenney]

We will give this a try! Thank you

[Piedone]

Can you share the result @themattkenney?