Wrong token pasting/concatenation when token contains spaces

[versat]

Stumbled over this issue when i added a define for the Qt macro Q_D(), see Cppcheck ticket https://trac.cppcheck.net/ticket/8479.
For this C code:

#define CONCAT(tok) tok##suffix

CONCAT(Test);
CONCAT(const Test);

gcc outputs:

$ gcc -E token_pasting_with_space.c
# 1 "token_pasting_with_space.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 1 "token_pasting_with_space.c"


Testsuffix;
const Testsuffix;

simplecpp outputs:

$ ./simplecpp token_pasting_with_space.c


Testsuffix ;
constTestsuffix ;

The space between "const" and "Test" is removed.
Not sure what the standard says about such a case but since Qt and gcc are widely used i guess it should work like Qt expects it and gcc handles it to not break things.

[danmar]

I have the feeling we merge "const Test suffix" into 1 token by intention.

You did not say if gcc preprocessor outputs 1 single token or 2.. you only showed the text output.

[versat]

I have not found a way to get gcc printing the tokens.
But clang has an option to output tokens:

$ clang test.c -Xclang -dump-tokens
identifier 'Testsuffix'  [StartOfLine]  Loc=<test.c:3:1 <Spelling=<scratch space>:2:1>>
semi ';'                Loc=<test.c:3:13>
const 'const'    [StartOfLine]  Loc=<test.c:4:1 <Spelling=test.c:4:8>>
identifier 'Testsuffix'  [LeadingSpace] Loc=<test.c:4:1 <Spelling=<scratch space>:3:1>>
semi ';'                Loc=<test.c:4:19>
eof ''          Loc=<test.c:4:20>

clang seems to see two tokens.

[danmar]

Nice! ok I agree we need to output 2 separate tokens. Well.. I still have the feeling we merge them by intention.

[danmar]

I did not see any hashhash test that required merging. Only a hash test. So maybe it won't break anything to fix this.

[orbitcowboy]

This has been tested with ad9b49d and prints now the expected result:

$ ./simplecpp test.c


Testsuffix ;
const Testsuffix ;

@danmar Do you have a test case in place already?