I’ve been doing a lot of thinking about how Paint.NET needs to change or evolve before it hits version 4.0, and the big thing that keeps coming up is isolation. Right now, the control flow and error handling in Paint.NET lets one component crash or corrupt the whole program. This could be code sitting in a plugin that causes an unhandled exception in a way that my try/catch harnesses don’t handle, or it could be a COM component loaded by Windows common file Open/Save dialogs that crashes or locks up. Or maybe a piece of code doesn’t throw an exception but it does corrupt some other data, and then that corruption causes a crash later. And then that gets attributed to the wrong component. GDI+ seems to like doing that a lot.
Not only does Paint.NET need to be protected by crashing or "bad" plugins, but it needs to be protected from itself! Things like the Open/Save and Print dialogs are a good example, especially because they deal with external, "native" components. Some people have shell extensions installed that cause the Open/Save dialogs to fail, hang, or crash, and this breaks Paint.NET! This could be something as seemingly benign as a thumbnail provider for DivX videos. Yes, I get crash reports on the forum for that every so once in awhile. Why should Paint.NET have to crash if there’s a corrupted DivX video in a folder you’ve navigated to via File->Open? I’d rather crash just the Open dialog, detect the error without corruption in the main process, and then re-run the dialog in some kind of compatibility or reduced functionality mode. At least that way if it corrupts memory, then that corruption only affects the hosting process that it lives in.
So, there needs to be a really easy way for me to write a component in Paint.NET — whether it’s "built-in" or a "plug-in" — that can be isolated and not cause problems for the rest of the application. This means I want to host it in an external process and communicate with it, and then have a layer of abstraction in place for creating a "remote", isolated object.
The new System.AddIn namespace in .NET 3.5 provides a whole system for this, but honestly I find it to be way too complex, confusing, and involved. They have host views, add-in views, host-side adapter, add-in side adapters, etc. It’s a bunch of terminology that seems invented — why not just say client and server, proxy, tear-off, etc.? Every time you add a new interface or type you have to add code that spans at least 3 assemblies. Plus, they do not provide the ability for me to be running in a 64-bit process and have it communicate with an add-in that requires 32-bit hosting. This is important for certain plug-ins which are doing native interop to DLL’s that are only available in 32-bit (the VTF File Type Plugin comes to mind). They also do not provide the ability to activate an add-in that needs to run at a higher security level — for instance, if I want to require my Settings dialog to run with administrator privilege. I don’t want to have to deal with UAC at the code level, and would prefer it to be abstracted away. They also enforce a specific directory structure, and so on … anyway, it’s just too much and also too little.
.NET Remoting isn’t really an end-to-end solution for what I need, as it does not provide a hosting process. COM provides component hosting in an external process, but it seems like major overkill to register my .NET components as COM-proxied objects. Windows Communication Foundation is more for "enterprise" remoting and communication scenarios.
So it looks like I may have to roll my own solution, using System.IO.Pipes (actually I may prefer an HWND and WM_COPYDATA for marshaling) and borrowing heavily from the lessons put forth by System.AddIn (they have a very good story for versioning), and learned from my inbox full of crash logs. In any case it must be something that is easy to use by both the "host" and the "add-in" (or "built-in" :)). I’m thinking code generation will be very handy here in order to be able to declare an isolatable interface and then auto-generate the local tear-off/proxy, client stub, server stub, and registration glue.