We have a ASP.NET application and suffered from high CPU issue occasionally - for years. It’s in production code and hard to reproduce. Fortunately, we got two dump files during application under high CPU issues.
What happened during high CPU?
Load the dump to windbg, to inspect what happened during high CPU issue:
.loadby sos clr #load the SOS
.runaway #check which thread consume most of CPU time.
~* e !ClrStack #check each call stack for all managed threads.
Most of call stack down to following calling :
System.Collections.Generic.Dictionary`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].Insert(System.__Canon, System.__Canon, Boolean)
--- or ---
System.Collections.Generic.Dictionary`2[[System.Int32, mscorlib],[System.__Canon, mscorlib]].FindEntry(System.Int32)
Then, issue the ~* e ! ClrStack -p to check each parameter of these call stack, the first parameter to above two methods was this which refer to dictionary itself. All of threads are accessing the same dictionary instance.
Got it, it was thread-safe issue to the dictionary.
How to fix
Using ConcurrentDictionary to replace the Dictionary. There are some of tricky parts on ConcurrentDictionary:
- Microsoft Docs do not guarantee the thread-safe for those interfaces implement.
- Actually most of internal interface implements are thread-safe. eg,
this[],Values.
Anyway, just do not depends on these internal implement, just do what’s the docs recommend. Depending on internal implement may lead to subtle issue, eg. using IDictionary to refer to an IDictionary implement which may or may not thread-safe.
Reference
If you know the reason you can have keyword to get bunch of references.
- https://improve.dk/debugging-in-production-part-1-analyzing-100-cpu-usage-using-windbg/
- https://blogs.msdn.microsoft.com/tess/2009/12/21/high-cpu-in-net-app-using-a-static-generic-dictionary/
Comments: