So, to confirm your main problem:
... and therefore can access my application and its currently running forms / methods / controls, etc.
Before embarking on the complex and complex means of loading, isolating, and limiting these extensions, you should know a few things about Windows and the CLR. First, any in-box program can use multiple Windows APIs to enter code into your process. Once the code is loaded into your process, either by you or by the operating system, accessing the CLR runtime and loading assemblies and / or running code in an existing AppDomain is quite simple.
Knowing this, you must weigh and balance the efforts that you choose to limit the "expansion". If I build something like this, I will worry more about other things besides the malicious extension code that controls the state of my application. For example, these are things you can consider:
- Downloading extensions approved by your user allows them to control what is allowed and allow them to cancel the extension later if necessary. Take a look at Office or VStudio as an example.
- Ensure that these approved extensions are not fixed using the code signing requirement (strong names or code signing certificates).
- Consider disabling the extension option for remote launch if it proves to be malicious.
- Prove the well-appointed API so that developers can easily implement the desired behavior. If it’s easy for them to use their interfaces to accomplish their task, they don’t need to “hack”.
Other than that, you really can't do much. As I said, anyone can attack your application even with the help of the aforementioned guards. Your main task is not to surprise your users. Thus, prudent care about what code your application runs in is recommended, but what these extensions do when your users give them access is not really something you can take full control of.
This does not mean that isolation of AppDomain will not give you value, it may be; however, IMHO, providing sufficient security without restricting their ability to function, will be difficult.
UPDATE
... but if you download a plugin in AppDomain that is configured with limited permissions, how can it use this vector?
Right, as I said in my final statements, you can restrict their access to unmanaged code inside the AppDomain. It also limits their ability to develop usable Windows. I expect most WinForms applications to use at least one PInvoke call or unmanaged COM control. This restriction may be acceptable, I really can’t say without additional information about what functionality they are trying to provide.
All that I tried to say is that by installing and approving the extension, your users take responsibility for allowing this extension. What the extension is and how malicious it may be, is not your responsibility, assuming, of course, that you downloaded the correct code. That's why I recommend focusing your energy on running approved code, rather than worrying about what this code can do when it is in your process.