The preferred way of compiling the custom layer is to use the script files in the custom/bin
If you absolutely want to use cmake, you need to compile 4coder_default_bindings.cpp
(or the file that is the entry point of your custom layer) as this file will include all the necessary files and use only a single translation unit. For the required defines and compiler flags you'll need to look into the build scripts what's necessary.
You'll also need to compile the metadata generator and run it before compiling 4coder_default_bindings.cpp
. Once again you need to look at the build scripts to see what's happening.
You'll also need to move the resulting shared library (.dll or .so) to the root directory.
Customization layer - getting started ( 4coder 4.1 )