Register
4coder»Forums»Extending Lexer
Jens
20 posts / 1 project
Extending Lexer
5 days, 22 hours ago Edited by Jens on Sept. 13, 2020, 1:06 p.m. Reason: Further investigation
One of the features I find useful in C++14 is the number literal syntax, where you can add single quotes that are ignored by the compiler. Like 0xAABB'CCDD'EEFF or 1'234'567. 4coder doesn't support this out-of-the-box so I decided to have a stab at extending the lexer.

After adding the missing ../build folder, fixing the compilation error (conversion warning was treated as error) and fixing the read path in 4coder_cpp_lexer.cpp I managed to run the lexer test. But it crashes while reading line 414 (or later, not sure if print buffer got flushed)

1
2
3
4
#error ``` Foo bar is bad news bears! ```
#   define foo bar
#    define bar baz
#define baz bish


All output looks good until that line.
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
[ 2475,  2510)        LiteralString /       PPErrorMessage : 0x0001 / 0x0000
	: ``` Foo bar is bad news bears! ```:
[ 2510,  2511)           Whitespace /           Whitespace : 0x0000 / 0x0000
	:
:
[ 2511,  2521)         Preprocessor /             PPDefine : 0x0001 / 0x0000
	:#   define:
[ 2521,  2522)           Whitespace /           Whitespace : 0x0001 / 0x0000
	: :
[ 2522,  2525)           Idexited with code -1073741819


Should this be working or is the lexer code generated using some other method?

I'm using 4coder version 4.1.6.


Edit:

As a side-note, building the lexer, running it and rebuilding the custom layer; it does actually lex the literals properly. I guess the crash is just a false negative.
Simon Anciaux
881 posts
Extending Lexer
4 days, 18 hours ago
If you still need help with this, you might want to contact Allen directly on discord.
Jens
20 posts / 1 project
Extending Lexer
3 days, 16 hours ago
Everything seems to be working except that the lexer test crashes, but as it works I'm happy.

It was really simple to be honest, to get ' supported (the 123'4567-styled literals) all I had to do was to add ' to the places "0124.." was in 4coder_cpp_lexer_gen.cpp. Technically that's not 100% correct since 0x'aa'bb is an invalid literal but the compiler will catch that one.

For binary literals I more or less just copy-pasted the hex version and replaced x with b :D