Customization layer - getting started

This is a small introduction to the 4coder customization layer. It will help you setup your custom layer, define your key bindings, set your hooks, add keywords to the syntax highlighting.

Requirements
  • the paid version of 4coder;
  • to know some C/Cpp;
  • have a compiler installed: Visual Studio on Windows, GCC or Clang on Mac and Linux.


Installation
Installing boils down to unzipping the archive where you want. There are a few things to pay attention to (on Windows):
  • If the path contains space(s):
    • The script may fail to setup Visual Studio (vcvarsall.bat), so you would need to do it yourself before launching the buildsuper.bat script;
    • you will need to be in the 4coder directory to build the custom layer, otherwise the script will not work;
  • if the path contains '(' or ')' the buildsuper.bat script may fail;
  • non ascii characters seem to be fine.

It's a good idea to put it in a folder without spaces or parenthesis in the name.

Building
To build the 4coder default custom layer, double click on the buildsuper.bat file at the root of the 4coder directory or launch it from a console. The later is preferred as you would see the eventual errors. You can pass a filename as the first argument to that script and the file will be compiled instead of the default one (4coder_default_bindings.cpp). If you launch buildsuper.bat from another folder, you need to specify the path to the custom layer code file, even if you want to use the default one.

For example, if 4coder is in D:\4coder\ but you compile form D:\, you need to compile with this command
1
D:\4coder\buildsuper.bat D:\4coder\4coder_default_bindings.cpp

On windows, buildsuper.bat will try to call vcvarsall.bat for you (if it hasn't been called yet). If you have several versions of Visual Studio installed, the version that will be used depends to the order defined in the file windows_scripts\setup_cl_generic.bat. Supported version are Visual Studio 2010 and up.

If the compile was successful you should have a custom_4coder.dll and custom_4coder.pdb in the current directory. If you didn't compile from the 4coder folder, you'll need to copy the dll and pdb in the 4coder directory.

Make it easier to update
I suggest you to make a copy of 4coder_default_bindings.cpp and to edit only the copy. Your code doesn't need to be in a specific place as long as you pass the correct file name to the build script. In a similar way I suggest you to create a build.bat file that will call buildsuper.bat with the file name you just copied as an argument. That will make it easier to update to a new version of 4coder as you will not have to worry about making backup of your files.

build.bat on windows
1
2
@echo off
buildsuper.bat custom_layer.cpp

build.sh on linux (Mac?)
1
2
#!/bin/bash
./buildsuper.sh custom_layer.cpp

I find useful to have a local svn repository with some of the files from 4coder checked in (generally those that I copied a function from to make a small modification) so that when I update 4coder I can diff those files and see if some changes need to be ported over to my functions.

Remapping keys
The "entry point" for the custom layer is the get_bindings function in your custom layer file. The default one calls default_keys (4coder_remapping_commands.cpp), which call fill_keys_default (4coder_generated\remapping.h). To make your own key bindings based on the default one, copy this function in you custom layer file, change it's name and modify the bindings.
  • Don't forget to set keys for the default_lister_ui_map, it is required to be able to move in the menus.
  • If you are using the "default_file_settings" hook, you need to define "mapid_file" and "default_code_map" (even if they are empty) otherwise 4coder will crash on startup. See those "Notes for 4coder customizers on build 4.0.29".
  • Don't forget to change the function call in the get_bindings function.
  • If you're on Mac, use the mac version of the function.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
#include "4coder_default_include.cpp"

void custom_keys(Bind_Helper *context){
    
    begin_map(context, mapid_global); {
        bind(context, 'p', MDFR_CTRL, open_panel_vsplit );
        ...
    } end_map(context);
    
    begin_map(context, mapid_file); {
        bind_vanilla_keys(context, write_character);
        ...
    } end_map(context);
    
    begin_map(context, default_code_map); {
        inherit_map(context, mapid_file);
        bind(context, key_right, MDFR_CTRL, seek_alphanumeric_or_camel_right);
        ...
    } end_map(context);

    begin_map(context, default_lister_ui_map); {
        bind_vanilla_keys(context, lister__write_character);
        bind(context, key_esc, MDFR_NONE, lister__quit);
        bind(context, '\n', MDFR_NONE, lister__activate);
        bind(context, '\t', MDFR_NONE, lister__activate);
        bind(context, key_back, MDFR_NONE, lister__backspace_text_field);
        bind(context, key_up, MDFR_NONE, lister__move_up);
        bind(context, key_page_up, MDFR_NONE, lister__move_up);
        bind(context, key_down, MDFR_NONE, lister__move_down);
        bind(context, key_page_down, MDFR_NONE, lister__move_down);
        bind(context, key_mouse_wheel, MDFR_NONE, lister__wheel_scroll);
        bind(context, key_mouse_left, MDFR_NONE, lister__mouse_press);
        bind(context, key_mouse_left_release, MDFR_NONE, lister__mouse_release);
        bind(context, key_mouse_move, MDFR_NONE, lister__repaint);
        bind(context, key_animate, MDFR_NONE, lister__repaint);
    } end_map(context);
}

extern "C" int32_t
get_bindings(void *data, int32_t size){
    Bind_Helper context_ = begin_bind_helper(data, size);
    Bind_Helper *context = &context_;
    
    set_all_default_hooks(context);
    
#if defined(__APPLE__) && defined(__MACH__)
    custom_keys_mac(context);
#else
    custom_keys(context);
#endif
    
    int32_t result = end_bind_helper(context);
    return(result);
}


Key maps
The key maps allow to have different sets of keys that can be activated/deactivated. For instance having additional bindings when you edit code instead of pure text. All your "bind" calls should be between a begin_map call and an end_map call. I like to write bindings between braces to have an additional level of indentation but it's not necessary.
1
2
3
4
begin_map( context, mapid_global ); {
    bind( ... );
    bind( ... );
} end_map( context );

You can inherit a key map using the inherit_map function. If you don't have a call to inherit_map between begin_map and end_map, you'll inherit from mapid_global. If you don't want to inherit any key, use
1
inherit_map( context, mapid_nomap );

You can define your own key maps. Map_ID is an enum defined in 4coder_API\types.h. 4coder map ids start at (1 << 24). So your map ids should not overlap that. You should start at the value of "default_maps_count" which is defined in 4coder_default_framework.h.

bind_vanilla_keys( context, function ); allow you to bind all character key to a function. bind_vanilla_keys( context, write_character); will make each character key output it's character in the active buffer. bind_vanilla_keys( context, hello ); will call the function hello each time a character key is pressed.

You can change the keymap of a buffer with buffer_set_setting. Here is an example function:
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
void set_current_keymap( Application_Links* app, int map ) {
    
    unsigned int access = AccessAll;
    View_Summary view = get_active_view( app, access );
    Buffer_Summary buffer = get_buffer( app, view.buffer_id, access );
    
    if ( buffer.exists ) {
        buffer_set_setting( app, &buffer, BufferSetting_MapID, map );
    }
}


custom function
To create a function that can be called with a key bind you need to create a function using the CUSTOM_COMMAND_SIG(name) macro (4coder_API\types.h). The actual function signature is void name(struct Application_Links* app).
1
2
3
4
5
CUSTOM_COMMAND_SIG( hello ) {
    print_message( app, "Hello Dave.", sizeof( "Hello Dave." ) - 1 );
}
...
bind( context, 'h', MDFR_CTRL | MDFR_ALT, hello );

That function can call anything you can write in CPP ( e.g. Win32 function, or other libs).

How to do something ?
The best way to find out how to do something is to look in the code how Allen does things similar to what you want. Most of the functions you use are accessible in the source files. And remember that you can attach the debugger to 4coder to debug everything in the custom layer. http://4coder.net/custom_docs.html contains the documentation for the API.

Making 4coder modal
You can use the key map system to make a modal version of 4coder. Note that the key map changes are on the buffer, not at the application level. Here is a small example.
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
void set_current_keymap( Application_Links* app, int map ) {
    
    unsigned int access = AccessAll;
    View_Summary view = get_active_view( app, access );
    Buffer_Summary buffer = get_buffer( app, view.buffer_id, access );
    
    if ( buffer.exists ) {
        buffer_set_setting( app, &buffer, BufferSetting_MapID, map );
    }
}

enum modal_mapid {
    
    mapid_insert = default_maps_count,
    mapid_shared,
    mapid_delete,
};

CUSTOM_COMMAND_SIG( modal_to_global ) {
    
    set_current_keymap( app, mapid_global );
    
    Theme_Color colors[ ] = {
        { Stag_Cursor, 0xffff5533 },
        { Stag_At_Cursor, 0xff00aacc },
        { Stag_Margin_Active, 0xffff5533 },
    };
    
    set_theme_colors( app, colors, ArrayCount( colors ) );
}

CUSTOM_COMMAND_SIG( modal_enter_insert ) {
    
    set_current_keymap( app, mapid_insert );
    
    Theme_Color colors[ ] = {
        { Stag_Cursor, 0xff80ff80 },
        { Stag_At_Cursor, 0xff293134 },
        { Stag_Margin_Active, 0xff80ff80 },
    };
    
    set_theme_colors( app, colors, ArrayCount( colors ) );
}

CUSTOM_COMMAND_SIG( modal_enter_delete ) {
    
    set_current_keymap( app, mapid_delete );
    
    Theme_Color colors[ ] = {
        { Stag_Cursor, 0xffffff00 },
        { Stag_At_Cursor, 0xff0000ff },
    };
    
    set_theme_colors( app, colors, ArrayCount( colors ) );
}

CUSTOM_COMMAND_SIG( modal_delete_word_right ) {
    exec_command( app, delete_word );
    exec_command( app, modal_to_global );
}

CUSTOM_COMMAND_SIG( modal_delete_word_left ) {
    exec_command( app, backspace_word );
    exec_command( app, modal_to_global );
}

void custom_keys(Bind_Helper* context ) {
    
    begin_map( context, mapid_shared ); {
        
        inherit_map( context, mapid_nomap );
        
        bind( context, key_left, MDFR_NONE, move_left );
        bind( context, key_right, MDFR_NONE, move_right );
        bind( context, key_up, MDFR_NONE, move_up );
        bind( context, key_down, MDFR_NONE, move_down );
        
    } end_map( context );
    
    begin_map( context, mapid_global ); {
        
        inherit_map( context, mapid_shared );
        bind( context, '\t', MDFR_NONE, modal_enter_insert );
        bind( context, 'd', MDFR_NONE, modal_enter_delete );
        
    } end_map( context );
    
    begin_map( context, mapid_insert ); {
        
        inherit_map( context, mapid_shared );
        bind( context, key_esc, MDFR_NONE, modal_to_global );
        bind_vanilla_keys( context, write_character );
        
    } end_map( context );
    
    begin_map( context, mapid_delete ); {
        
        inherit_map( context, mapid_nomap );
        bind( context, key_esc, MDFR_NONE, modal_to_global );
        
        bind( context, key_right, MDFR_NONE, modal_delete_word_right );
        bind( context, key_left, MDFR_NONE, modal_delete_word_left );
        
    } end_map( context );
}


Hooks
4coder allows you to specify functions that will be called when some events happen. get_bindings calls set_all_default_hooks function (4coder_default_hooks.cpp) to setup the default behavior. You can replace those hooks with your functions. The function signatures for those hooks can be found in 4coder_API\types.h.
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
extern "C" int32_t
get_bindings(void *data, int32_t size){
    Bind_Helper context_ = begin_bind_helper(data, size);
    Bind_Helper *context = &context_;
    
    set_hook(context, hook_exit, default_exit);
    set_hook(context, hook_view_size_change, default_view_adjust);
    
    set_start_hook(context, default_start);
    set_open_file_hook(context, default_file_settings);
    set_new_file_hook(context, default_new_file);
    set_save_file_hook(context, default_file_save);
    
    set_end_file_hook(context, end_file_close_jump_list);
    
    set_command_caller(context, default_command_caller);
    set_render_caller(context, default_render_caller);
    set_input_filter(context, default_suppress_mouse_filter);
    set_scroll_rule(context, smooth_scroll_rule);
    set_buffer_name_resolver(context, default_buffer_name_resolution);
    
#if defined(__APPLE__) && defined(__MACH__)
    custom_keys_mac(context);
#else
    custom_keys(context);
#endif
    
    int32_t result = end_bind_helper(context);
    return(result);
}


Adding key words to the syntax highlighting
You can add words that will be highlighted, but only for C/CPP files. Create a file (this needs to be in it's own file) and add one line per keyword in it, using this syntax:
1
2
3
4
{ make_stafl( "u8", CPP_TOKEN_KEY_TYPE ) },
{ make_stafl( "u16", CPP_TOKEN_KEY_TYPE ) },
{ make_stafl( "u32", CPP_TOKEN_KEY_TYPE ) },
{ make_stafl( "u64", CPP_TOKEN_KEY_TYPE ) },

The type you set doesn't have any impact.

At the top of you custom layer file, BEFORE #include "4coder_default_include.cpp", define EXTRA_KEYWORDS to the filename where your keywords are.
1
2
#defines EXTRA_KEYWORDS "custom_keywords.h"
#include "4coder_default_include.cpp"


EDIT 17th October 2018: Updated mapids to work with 4coder 4.0.29.
EDIT 27th November 2018: Updated bindings, and hooks to avoid crashes on 4.0.29.

Edited by Simon Anciaux on Reason: Formating
Thank you for this post.

What type of customization do you have in your layer?
I use a modal setup. Here are some things I added/modified:
- history for search and command;
- modify search to be more or less like in vim;
- modify execute command, execute last command, set working directory;
- save session on exit, restore session on start;
- open file from windows explorer in current session (kinda);
- find matching braces, parenthesis, brackets;
- delete end of line, delete beginning of lines, delete and comment matching brace;
- comment line, comment range;
- add/delete/find mark;
- modify seek word left/right;
- past in range;
- toggle between proportional / mono-spaced font;
- search word under the cursor on msdn;
- maximize panel, restore panel size;
- write {}, [], () and place the cursor between them;
- toggle read only flag;
- ensure I didn't add non-breakable space in source code;
- toggle treat as code;
- path auto-complete;



Thank you very much for this post, Simon! This helped improve my custom layer (and its organization) substantially.
How can I make C-n combination works like down_arrow when I'm in open\create_new file dialogue?
I just bind
1
bind(context, 'n', MDFR_CTRL, move_down);
into
1
mapid_global & mapid_file

There is no difference but combination works in standard editor.

Edited by steughar on
Right now the UI isn't very customizable. In the next build you'll be able to modify the default_lister_map and add the up down navigation commands to <ctrl n>. So take a look when 4.0.29 comes out and if you have any questions I'll be able to help then.
Great! Thanks for answer)
With build 4.0.29, setting
1
mapid_insert = 1
will cause an enum conflict with
1
default_lister_ui_map


So in the example
1
2
3
4
5
enum modal_mapid {   
    mapid_insert = 1,
    mapid_shared,
    mapid_delete,
};

Your customization layer will build but all insert commands will not work.

Changing mapid_insert to any integer other than 1 solves this issue
1
2
3
4
5
enum modal_mapid {
    mapid_insert = 42, 
    mapid_shared,
    mapid_delete,
};


Credit to FlyingSoloman of the Handmade Network for helping me fix this issue in my customization layer.
Instead of using 42 you can use "default_maps_count" (from 4coder_default_framework.h) which should give you the first unused mapid value. I updated the original post. Thanks.
I've posted an updated version of the original post in the 4coder wiki to make it easier to find. Feel free to used this thread to suggest improvements (or directly edit the wiki article) or ask questions.