Does 4coder works out of the box (from a new installation) ?
If you delete custom_4coder.dll from a fresh installation and call the buildscript, does it still works ?
Are there any space, parenthesis or special character in the path where you installed 4coder ?
Is the new dll (custom_4coder.dll) in the same folder as 4ed.exe ? (Not in the custom/bin folder).
Which version number are you using ?
If you're using 4.1.8, could you try running from the command line with
To produce a startup log ?
Could you share your custom layer file and how you build it ?
I haven't updated to 4.1.8 yet, so the wiki article might not be up to date (from a very quick look I think it should be OK though).