Jens
51 posts / 1 project
Extending Lexer
Edited by Jens on Reason: Further investigation
One of the features I find useful in C++14 is the number literal syntax, where you can add single quotes that are ignored by the compiler. Like 0xAABB'CCDD'EEFF or 1'234'567. 4coder doesn't support this out-of-the-box so I decided to have a stab at extending the lexer.

After adding the missing ../build folder, fixing the compilation error (conversion warning was treated as error) and fixing the read path in 4coder_cpp_lexer.cpp I managed to run the lexer test. But it crashes while reading line 414 (or later, not sure if print buffer got flushed)

 1 2 3 4 #error  Foo bar is bad news bears!  # define foo bar # define bar baz #define baz bish 

All output looks good until that line.
  1 2 3 4 5 6 7 8 9 10 [ 2475, 2510) LiteralString / PPErrorMessage : 0x0001 / 0x0000 :  Foo bar is bad news bears! : [ 2510, 2511) Whitespace / Whitespace : 0x0000 / 0x0000 : : [ 2511, 2521) Preprocessor / PPDefine : 0x0001 / 0x0000 :# define: [ 2521, 2522) Whitespace / Whitespace : 0x0001 / 0x0000 : : [ 2522, 2525) Idexited with code -1073741819 

Should this be working or is the lexer code generated using some other method?

I'm using 4coder version 4.1.6.

Edit:

As a side-note, building the lexer, running it and rebuilding the custom layer; it does actually lex the literals properly. I guess the crash is just a false negative.
Simon Anciaux
1184 posts
Extending Lexer
If you still need help with this, you might want to contact Allen directly on discord.
Jens
51 posts / 1 project
Extending Lexer
Everything seems to be working except that the lexer test crashes, but as it works I'm happy.

It was really simple to be honest, to get ' supported (the 123'4567-styled literals) all I had to do was to add ' to the places "0124.." was in 4coder_cpp_lexer_gen.cpp. Technically that's not 100% correct since 0x'aa'bb is an invalid literal but the compiler will catch that one.

For binary literals I more or less just copy-pasted the hex version and replaced x with b :D