When try/catch Isn't Enough: Isolating Native Crashes with a Worker Process

Consider a WPF CAD application with two geometry paths. The managed rendering library handles the common case, while a native geometry kernel is used as a fallback for boolean operations that need another implementation. This is useful until malformed or unusually complex geometry makes the native layer read protected memory, corrupt its heap, terminate, or stop responding.

The first instinct is to add another try/catch around the P/Invoke call. That is appropriate for ordinary operation errors, but it is not a dependable recovery boundary for corrupted process state. Microsoft's AccessViolationException documentation explicitly warns that recovery from corrupted process-state exceptions is not supported in current .NET versions.

The question therefore changes. Instead of asking how the same process can catch every failure, ask how much of the application must be allowed to fail with the native operation.

Moving the call to Task.Run or a dedicated thread can protect the UI from blocking, but both still share the same address space, native heap, loaded modules, and process lifetime. If native code corrupts that shared state, the UI thread being elsewhere does not make the process healthy.

AppDomain and AssemblyLoadContext solve different problems. They can isolate managed loading and unloading behavior, but they do not give a native DLL a separate operating-system address space. For failures that can invalidate the process itself, the useful containment boundary is another process.

The safest initial design is deliberately simple: start one x64 worker for one native operation, exchange one request and one response, then let the worker exit. If the native heap was damaged without producing an immediate crash, that state is discarded before the next request.

A long-lived worker or worker pool can reduce startup overhead, but it also reuses native global state and makes lifecycle policy more complicated. When the native path is already a fallback rather than the common path, paying process startup cost can be a favorable reliability tradeoff.

The process boundary removes direct object and pointer sharing. Inputs must be converted into a neutral representation before they cross it, and results must be reconstructed after they return. In a CAD workflow that representation might be an IGES, STEP, mesh, or application-specific byte payload; the important rule is that a native pointer never crosses the boundary.

Named Pipes work well for local duplex communication in .NET. The protocol should still be explicit: include a marker, version, operation, length prefixes, bounded payloads, bounded result counts, and a per-request authentication token. A private pipe name alone is not input validation.

A resilient host must distinguish a valid operation failure from the disappearance of the process that was supposed to report it. A structured error response means the worker remained coherent enough to follow the protocol. End-of-stream or a broken pipe before a response means the worker disappeared. A deadline means it may still be alive but is no longer making useful progress.

These outcomes should not collapse into one generic exception because they imply different diagnostics and cleanup. Timeout handling must terminate the disposable worker, while a normal rejection can include a meaningful domain error. In every case, the host converts transport behavior into an application-level result instead of allowing the native failure to decide the host's lifetime.

A timeout must cover connection, request transfer, native execution, response transfer, and process exit—not just one read call. The host owns that deadline because a hung worker cannot be trusted to cancel itself.

The public demo creates a unique pipe and random token, starts a worker, sends one bounded request, and waits under one cancellation deadline. If the deadline expires, the host kills the worker process tree and reports TimedOut. If the pipe closes before a response, it reports Crashed together with the exit code when available.

Process isolation answers how the host survives; it does not decide what the application should do next. That decision depends on the meaning of the operation. Returning the original geometry may be a safe degradation for one boolean difference, while silently doing so for a union could present incorrect geometry as a valid result.

Define fallback behavior per operation and make it visible to callers. A contained crash should not become a concealed data-integrity bug.

Putting the kernel in a worker does not excuse incorrect native integration. Pointer ownership, null checks, algorithm call order, result validation, and deterministic cleanup still reduce how often the worker fails. OCCT also provides OSD::SetSignal and thread-local signal setup for translating supported signals and Windows structured exceptions into OCCT failures when configured correctly.

Those measures improve diagnosability and reduce crash frequency, but they should not be confused with containment. If a native component can corrupt its process, the worker boundary remains the final defense even after the wrapper is hardened.

A worker architecture is incomplete if the worker executable, runtime configuration, native DLLs, or architecture-specific dependencies are missing from the release output. Build ordering, artifact copying, x64 configuration, code signing, and installer rules belong in the implementation plan from the beginning.

Diagnostics should also cross the abstraction in a controlled direction. The reusable worker client can publish a diagnostic event or structured result; the WPF application can subscribe and route it into its existing logger. Logging failure must never replace the original geometry failure or break the fallback path.

The companion repository is an independent teaching model, not extracted product code. It contains separate host, worker, and protocol projects and runs four deterministic scenarios: successful response, structured failure, abrupt worker exit, and worker hang.

The verification script asserts that the host remains alive and that each scenario is classified correctly. This validates the containment and protocol mechanics. It does not claim that every CAD kernel failure is reproducible, that every geometry format is lossless, or that a particular production integration has completed its domain-specific test matrix.

When a managed desktop application depends on native code it cannot fully trust, the following questions expose whether the architecture contains the failure or merely hopes to catch it.

The architecture is based on operating-system process isolation and documented .NET and OCCT behavior. These primary sources provide the relevant platform details.