I am writing several C # that will load a third-party assembly.
If a third party decided to be malicious, they could write a recursive function that would end in a StackOverflowException, citing my application.
Is it possible to detect a recursive function?
Update:
For unwanted situations like while (true) or for (;;), I already have a solution. Essentially, I run third-party code in a separate thread, and if the thread takes longer than a fixed duration, I pull out the plug. This doesn’t work very well with recursion, since stack restriction is achieved very quickly.
Update: I
may have distorted the solution I need. If I end up with a lot of intentionally malicious code, I will modify the application to run third-party code in a separate process. However, at this stage, I assume that the code will cause problems only because it is poorly written.
Accepted Answer
I decided that the best approach would be to run third-party libraries in a separate process. I can have multiple instances of running processes and even perform some load balancing of my third-party libraries in the processes. If malicious code is executed that kills one of the processes, I should be able to detect which library killed it, mark this library as malicious, and restart the process with all non-malicious libraries.
!