4coder»Blog
Allen Webster
Intro

It looks like the next build of 4coder (4.0.29) is going to be ready sometime in the next few weeks. The new build has been in development for a couple months and is loaded to the brim with new features that have all gone through interesting architectural and algorithmic design that I believe are worth sharing for several reasons. One it will prepare 4coder users who want to start writing customizations for how to think about the new features. Two it will help anyone who likes to think about the processes of architecture and algorithm design with some examples of my own process. Three it might expose me to some criticisms or suggestions that could help me improve the specifics of the new 4coder features before I put them into the wild.

Directory of all parts:
  1. Memory Management Overview
  2. Memory Management Variables and Objects
  3. Memory Management Scopes
  4. Custom UIs and Various Layers for Lister Wrappers
  5. Custom Cursors, Markers, and Highlights, and the Render Caller


Memory Management Overview

The topic for part one is the new API for easing lifetime management of data created and used by the custom side. This becomes a problem for implementors of custom layer code for a couple of reasons. The first reason this is an issue is that, by design, it is assumed the core can create and destroy views and buffers without the custom side issuing the create/destroy operation, although right now it turns out this only happens at startup when the core creates *scratch* and *messages*, I want to design the API in such a way that, if new cases arise where the core creates or destroys objects, customizations that were written to the API still function correctly.

1
2
3
4
5
6
7
8
// We want to avoid making every module ever maintain it's own
// lookup tables and write it's own hooks like this:
END_OF_BUFFER_HOOK(cleanup_my_perbuffer_memory){
    Attachment *attachment = lookup_attachment_by_id(buffer_id);
    if (attachment != 0){
        free_attachment_by_id(buffer_id);
    }
}


The second reason is that even if the custom layer does issue the operation to create or destroy an object, the code that issues that operation should not be forced to be aware of every sub-system present in the custom layer that is interested in creation/destruction of each object. In the most extreme case, imagine you are composing your customizations and you use several openly shared modules that you were written by different authors, and imagine these modules have never before been used together and the authors never spoke to one another. Suppose one of the modules provides features that need to track information on a per-buffer basis, and so this module wants to release information that it has stored whenever a buffer is destroyed. Then suppose the other module provides features that automates opening and closing sets buffers. If the first module was written with the assumption that every other part of the custom layer would be aware of it's presence and call into it when a buffer is created or destroyed, then in order to compose these two modules you will have to manually edit the second module to do the communication the first module requires.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
CUSTOM_COMMAND_SIG(close_buffer_set)
CUSTOM_DOC("Closes all buffers in the currently active buffer set")
{
    for (int i = 0; i < current_set->count; i += 1){
        kill_buffer(app, current_set->buffer_ids[i], 0);
        // We don't want users to have to manually insert
        // stuff like this all over the place
        // (wether or not they own the code for both sides).
        notify_buffer_killed(app, current_set->buffer_ids[i]);
    }
}


I chose this as our first topic because it turns out managing this data is foundational to both of the other big things that are included in this build, and even outside of the other new features, solving this particular problem has proven to be important to almost everything in the custom layer in a foundational way, and it has the most interesting architectural and algorithmic design problems, so if someone was only going to read one part, I would want it to be this one.

API Summary

The solution presented in this series is a memory management API, but it is not like any garbage collection, smart pointer, or reference counting system, which I suspect are the sort of things that spring to mind when I say memory management. The design of this system follows directly from the kind of things I was manually doing in the custom layer for features like sticky jumps, previous word-complete state, and persistent per-view states like smooth scrolling and previous paste index, and it fits some patterns I tried to use in the core for tracking per-view-buffer cursor, mark, and scroll states.

Basically, across all these systems, I frequently find myself wanting to allocate memory and set variables tied to a particular buffer, or to a particular view, so that in the future if I have a handle to the view and I know what variable on that view I want to query I can get back whatever I set before, or so that if I allocate some memory that is only relevant to a particular view or buffer, I don't have to register a hook that hears about every buffer that ever closes and checks if there is memory that needs to be freed. This leads to three major types of things I want to talk about. Some things are "variables", they have a name that can be pulled out of "thin air", that is the person who sets the variable doesn't have to pass anything to the person who gets the variable later. Other things are "objects", they have a variable sized amount of memory, and to access them you do so through a handle that was returned when you created the object. Finally there are "scopes", which are the things to which variables and objects can be tied.

1
2
3
typedef int32_t Managed_Variable_ID;
typedef uint64_t Managed_Object;
typedef uint64_t Managed_Scope;


In the next two parts I will go into each of these types and the sort of problems they help solve, and how they interrelate to each other. Then we will look at other new APIs that benefit from the presence of this memory management system.
Allen Webster
Hello everyone! In lieu of a 4coder Friday this week I am going to explain the direction of the Lister API and how it has surprisingly affected some of my long term plans for 4coder.

The Original Lister API Plan

When I first set out to create a Lister API the idea was to just provide a short term hack that would hold over users who want to customize the file lists in 4coder, and who want to create similar lists for other commands. I assumed it would be temporary because I want 4coder's customization layer to expose more than just lists for user controlled UI. I wanted at least to support tables, and probably a number of other basic UI schemes, but I was convinced that to do that right the best option would be to integrate those features into the buffer system, so that there would just be a set of code paths for editing buffers and rendering them and no separate UI mode. Since that was the plan in my head I was convinced I was just going in to make a quick hack API that would be dumped in 4.1.0. Here is a rough outline of how I thought it might look, just based on writing the usage code first:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
// ... command signature/function signature
{
    List_Handle list = start_list();
    list_add_item(list, "first string", "little description", (void*)1);
    list_add_item(list, "second string", "***", ptr_to_a_thing);
    // ...
    list_add_item(list, "last string", "don't ever click this option!", "last string");
    void *result_ptr = execute_list(app, &view, list);
    if (result_ptr == 1){
        // ...
    }
    else if (result_ptr == ptr_to_a_thing){
        // etc
    }
}


Actually a lot of credit should go to Casey who first suggested an API direction along these lines.

This design is meant to achieve a few things. First it is very easy to create a list and the user never has to deal with the actual input processing to get back the result of the list. Inside "execute_list" all the necessary work would be done to consume future input and eventually figure out which option was selected, finally returning the user pointer paired with the item. The user pointer would just be a void* leaving it totally up to the user what that means. It could be an integer casted to (void*), a repetition of the string used for the item, a pointer to some structure managed by the user, etc. This API is made possible because 4coder commands run in a separate coroutine which can yield and wait indefinitely for input from the user, which means managing the lifetime of the user data can be as easy as using any scoped temporary storage.

In theory this could be used even for something like the file exploring list, by ending one list and starting a new one every time the user changes the path. It would look something like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
// ... command signature/function signature
{
    for (;;){
        List_Handle list = start_list();
        File_List files = get_file_list(app);
        for (int32_t i = 0; i < files.count; i += 1){
            list_add_item(list, files.info[i].name, "", &files.info[i]);
        }
        File_Info *info = (File_Info*)execute_list(app, &view, list);
        if (!info->is_folder){
            update_path(info->name);
        }
        else{
            load_buffer(info->name);
            break;
        }
    }
}


But there was another problem, which is that each mode of the list actually behaves a little differently when processing input and matching the user's input string to the listed items. For instance the file path system actually doesn't match the entire input line against the list items for filtering, just the file name found at the end after the last slash in the string. Furthermore in some modes it is not just about what item is selected, but also about the string the user has typed. Most modes use the same wildcarding rules, but maybe users would want to create more powerful matching rules, or disable wildcards. In some modes pressing enter always ends the list successfully and in other modes it might not end the list at all. I also considered the possibility that some custom lists would want input behaviors that I wasn't using anywhere in 4coder yet. All this made me think that maybe the user would want to see every input event and do something with it themselves.

More Complex Control in the API

Before I ever sat down to code up the Lister API I had already thought up an addition to the above API that would let the user inspect incoming events before the system processed them so that the user could write code to respond to any inputs and simulate inputs. I also wanted to separate the string that was used for narrowing the list and the string that would be shown in the text input field.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
// ... command signature/function signature
{
    for (;;){
        List_Handle list = start_list();
        File_List files = get_file_list(app);
        for (int32_t i = 0; i < files.count; i += 1){
            list_add_item(list, files.info[i].name, "", &files.info[i]);
        }
        List_Step_Result step = execute_list(app, &view, list, hot_directory, string_get_front_name(hot_directory));
        if (step.in.type == UserInputKey){
            // ...
        }
        if (step.user_ptr != 0){
            File_Info *info = (File_Info*)step.user_ptr;
            if (!info->is_folder){
                update_path(info->name);
            }
            else{
                load_buffer(info->name);
                break;
            }
        }
    }
}


But around this time my "fish sensor" was going off. My "fish sensor" is a double slitted sensory organ on the front of my face that detected the distinct smell of something fishy. In particular I realized that either the execute_list procedure needed to return the updated string after processing keyboard input, or processing keyboard input would need to be the responsibility of the user. I also still had no good answer for how to make the matching rule customizable, and it began to dawn on me that if the user is going to write their own filtering code anyway, AND they get notified after every input in case they want to update the list, why shouldn't they just be left to pre-filter the list they are submitting themselves? Anyone who didn't want to write their own filter will just have to use some helper, and anyone who does want to customize it will be able to have a filter rule for literally anything they have the ability to compute.

User Controlled Filtering and Input

Having already considered those things when I sat down to write this API I came to the following idea:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
// ... command signature/function signature
{
    for (;;){
        List_Handle list = start_list();
        File_List files = get_file_list(app);
        list_add_text_field(list, "Open:", hot_directory);
        for (int32_t i = 0; i < files.count; i += 1){
            if (wildcard_match(front_of_directory(hot_directory), files.info[i].name){
                list_add_item(list, files.info[i].name, "", &files.info[i]);
            }
        }
        List_Step_Result step = execute_list(app, &view, list);
        if (step.in.type == UserInputKey){
            // ...
        }
        if (step.user_ptr != 0){
            File_Info *info = (File_Info*)step.user_ptr;
            if (!info->is_folder){
                update_path(info->name);
            }
            else{
                load_buffer(info->name);
                break;
            }
        }
    }
}


In this version the user now sets explicitly how the text field will look. They submit all the items they want in their list according to the current state. Then they display that list and wait for input that will control how the state changes. After writing something like this I immediately realized another problem with it. The current listing system supports three methods of getting to the item you want. One method is to type in enough of it's name to get it near the top of the list. The second method is to click the item in the list. The third is to use the arrow keys to navigate a selection indicator to the item and then press enter. How was this going to support that behavior? If execute_list processes the arrow keys and keeps the necessary index then we end up in a nasty situation where the customizer and the API both handle some keyboard input. If the user was going to control it, which I would prefer, how would they indicate to the API which item needed the highlight indicator?

I thought, no problem I'll just specify a highlight for each item in the list:

1
list_add_item(list, files.info[i].name, "", &files.info[i], (highlighted_index == i));


Next I was thinking about how this would interact with the mouse, which also changes the highlight values of the items by hovering and clicking them. I could let one override the other, or perhaps say the highest highlight value always wins, but I thought: What if the user could just query the mouse point and the position of the item they are placing and decide for themselves whether to highlight?

User Controlled Mouse Input and Layout

This was going to be tricky though because I was thinking that I would submit these items and let the internal system compute their layout inside the "execute_list" call, so to make this work I would have to emulate the layout on the user side to get the correct rectangle for each item. The next thought to follow that was: If the user is going to end up wanting to emulate the layout anyway, why don't I just let the user control the layout? At this point a full vision for the API formed in my head and I've been stoked about it every since.

Basically the Lister API has transformed into a UI Rendering API. You as the user simply submit a set of items and those items get rendered for you. You are still responsible for 100% of input processing. You can put a view into UI mode, take it out of UI mode, and you can set the UI data for the view. Once you've placed a view into UI mode it will display it's current UI data instead of it's current buffer. In UI mode the view will not be accessible unless you use the AccessHidden or AccessAll flags, so commands that edit or navigate the buffer will not operate (assuming they were written with the correct AccessOpen and AccessProtected flags).

The layout of the UI is specified by directly setting the rectangle each item should cover. This means the API is no longer specifically tied to listing. It could be used to build tables, hierarchical data, profile data, settings editors, etc, etc.

There are two ways you can get input to your custom UI. The first is to treat the entire UI interaction as a single command and call get_user_input to block the command until there is more input from the user. The advantage to this is that you can keep all the code controlling the UI in one place. The disadvantage is that 4coder only allows one in-flight command at a time, the user will have to cancel the operation before they can switch to another view, or if they try to click on another view the command will terminate automatically. The second option is to set the view's command map to a set of commands that understand how to manipulate the state of the UI. In the next version of 4coder, I plan to have rewritten all the old built in UIs with this API using the second method.

Won't this be a lot more cumbersome for users who just want to query the user on a list of options?

Yes this API puts a lot more work on the user, but that can now be wrapped by a helper layer. My plan is to provide a wrapper for all of this that will look a lot like the original API idea. That is still the most convenient way to write interesting new list commands. Throughout all of this my philosophy was that I don't want to have to update this API in a major way for a while as I am trying to wrap up 4.0.X this year. In order to retire it I want to make sure I only add simple general options to the core API. The final design is simpler to implement, since now the core is only responsible for saving UI data and rendering it, and it is fully general to a very wide range of possible UIs.

Progress

So far I have the new API fully implemented, with most internal core details worked out. Some points of complication were transferring the old scrolling system over to this new UI system, and preserving query bars, which were rendered through the old UI system. I have a mostly working buffer switching list that uses the get_user_input method to test out the API, but I want to rewrite it and the other list based commands using the command maps in order to preserve their behavior as much as possible. Along the way I still need to develop the wrapper layer(s), and then when I have rewritten all the old list-based features I will starting adding new ones. This is probably the biggest change in the 4coder code base since adding support for Mac, and at only one or two days a week, I expect it to be a little while before it is all ready to go.

Bye!

I hope some of you have found this interesting or insightful; if I didn't think of something important along the way, or if you have any questions, I'd love to hear about it. Thanks for reading everyone!
Allen Webster
Err sorry... somehow I accidentally posted the topic twice. Please read and comment here:

https://4coder.handmade.network/b...n_of_4coders_new_lister_api#15820
Allen Webster
Hey everyone!

After yesterday's 4coder Friday stream I was trying to come up with a list of things I should do next, and I decided that the next thing to do was the listing API, but since that could be a four to six week endeavor I thought I should get all my recent tweaks out first. So I just published 4.0.28 which you can get here.

Next I'm on to exposing that listing UI to the custom API and implementing all the cool stuff I can think of with that feature.

-Allen
Allen Webster
Hey everyone!

With this build, my first step of the 4.0.X retirement schedule is complete!

The majority of the new features were already working a week ago, so I got to spend some good time using them, and I'm very happy with this build. The new project system in particular has simplified my workflow even more than the old project system. This build also moves several more options to the config file, specifically font size, and control over the scroll bar and file bar. Until now these were all controlled by command line parameters, or hard coded values in the custom layer.

You can check this build out at: https://4coder.itch.io/4coder
And you can see the accompanying project tutorial here: https://www.youtube.com/watch?v=FX8WPI2WGVI&t=2s

Next I'll be moving on to making the listing UI programmable from the custom layer, which will be a key component of a lot of edit operations and work flow features.

Thanks for following!