I'm currently working on my own modal custom layer and I'm trying to set key maps per panel. I have it setup so that there's two different key maps, each for inserting and executing commands respectively. The problem I'm running into is that the key maps are bound to the buffers and not the panels. I could keep track of each panel myself and set the key map when switching buffers, but I can't find a way to hook EVERY time a panel switches a buffer. Is there some solution that I'm overlooking or a hook that I'm not seeing?

Yeah you've pointed out one of the serious design flaws with the current custom layer API :(

You'll have more success if you use a single command map for everything and do your own if/else or switch type dispatching on global state variables that you create yourself. is the best example of how it looks when you use this method and contains handy macros that you might be able to use as-written to setup your bindings.