Fix high CPU issue for ASP.NET - Dictionary was not thread-safe


We have a ASP.NET application and suffered from high CPU issue occasionally - for years. It’s in production code and hard to reproduce. Fortunately, we got two dump files during application under high CPU issues.

What happened during high CPU?

Load the dump to windbg, to inspect what happened during high CPU issue:

.loadby sos clr     #load the SOS
.runaway            #check which thread consume most of CPU time.
~* e !ClrStack      #check each call stack for all managed threads.

Most of call stack down to following calling :

System.Collections.Generic.Dictionary`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].Insert(System.__Canon, System.__Canon, Boolean)
--- or ---
System.Collections.Generic.Dictionary`2[[System.Int32, mscorlib],[System.__Canon, mscorlib]].FindEntry(System.Int32)

Then, issue the ~* e ! ClrStack -p to check each parameter of these call stack, the first parameter to above two methods was this which refer to dictionary itself. All of threads are accessing the same dictionary instance.

Got it, it was thread-safe issue to the dictionary.

How to fix

Using ConcurrentDictionary to replace the Dictionary. There are some of tricky parts on ConcurrentDictionary:

Anyway, just do not depends on these internal implement, just do what’s the docs recommend. Depending on internal implement may lead to subtle issue, eg. using IDictionary to refer to an IDictionary implement which may or may not thread-safe.

Reference

If you know the reason you can have keyword to get bunch of references.

published at Oct. 28, 2019, 5:55 p.m.

Comments:

Sharing your thoughts: