I don't see the problem, IF you're willing to limit yourself to supporting the EFI's bit depth. Just ship 32-bit versions of Linux with a 32-bit EFI boot loader and 64-bit versions with a 64-bit EFI boot loader. Clearly specify that on EFI systems, they're intended for EFIs with the specified bit depth. I've hacked together installers that work fine on my 32-bit Mac Mini, just by adding a suitable EFI boot loader to the installer. Granted, things get uglier if you wanted to support systems with 64-bit CPUs and 32-bit EFIs, but to the best of my knowledge, only a few Apple models ship with such a weird setup. Adding 32-bit EFI support to 32-bit versions of distributions doesn't seem like a big deal to me, although I admit that the number of users who need such a thing is rather small.
So what's the problem?