How do I get yacc / bison and / or lex / flex to resume scanning after doing something like swapping a token?
Is there a way to get bison and / or flex to resume scanning after replacing some token with something else?
My specific example would be with the replacement of a specific word / string. If I want a word to hello
be replaced with echo hello
, how can I get flex or bison to replace hello
and then start parsing again (instead of two words instead of two). So it would be:
- Get a WORD token (which is a string type)
- If
hello
, replace the token value withecho hello
- Restart parsing all input (which is now
echo hello
) - Get token WORD (
echo
) - Get token WORD (
hello
)
I've seen some very tempting features like yyrestart()
, but I really don't understand what this feature really does. Any help is greatly appreciated, thanks!
Update 4/23/2010
One hack-and-slash solution I have used for each word
one that goes through, I check the "alias" array. If it word
has an alias, I replace the meaning of the word (using, for example strcopy($1,aliasval)
), and check the flag aliasfound
.
After the entire input line has been parsed once, if the flag aliasfound
is true, I use yy_scan_string()
to toggle the buffer state to the extended aliased input and calls YYACCEPT
.
So it goes to the main function and I call again yyparse()
, with the buffer still pointing to my string. This continues until no aliases are found. When all my grammar activities are complete, I call yyrestart(stdin)
to return to "normal" mode.
If anyone knows how I can efficiently expand my words with their alias values, embed in stdin
(or some other method) and basically expand all aliases (even nested ones) as I go, that would be awesome. I played with yypush_buffer_state()
and yypop_buffer_state()
with yy_switch_to_buffer()
, but I couldn't get a "built-in" replacement with continued parsing ...
a source to share
Adding an "answer" based on what I was doing. Want to mark this question as an answer.
Update 4/23/2010
One of the hack-and-slash solutions I have used for every word I scan is I check the alias array. If the word has an alias, I replace the meaning of the word (using, for example, strcopy ($ 1, aliasval)) and check the aliasfound flag.
After the entire input string has been parsed once, if the aliasfound flag is true, I use yy_scan_string () to switch the buffer state to the expanded aliased input and call YYACCEPT.
So it goes to the main function and I call yyparse () again, with the buffer still pointing to my string. This continues until no aliases are found. When all my grammar steps are complete, I call yyrestart (stdin) to return to "normal" mode.
If anyone knows how I can efficiently expand my words with their alias values, inject into stdin (or some other method), and basically expand all aliases (even nested ones) as I go, that would be awesome. I played around with yypush_buffer_state () and yypop_buffer_state () along with yy_switch_to_buffer (), but I couldn't get the "inline" replacement with continued parsing ...
a source to share
It seems to me that the place to fix is the lexer. I would suggest using flex, which supports a state machine (called Start Conditions "in the flex documentation.) You change states with BEGIN
, and states must be defined in the definitions section.
So, for example, you might have a rule like
<INITIAL>hello BEGIN(in_echo); yyless(0); return (WORD_ECHO);
<in_echo>hello BEGIN(0); return (WORD_HELLO);
yyless()
truncates the value yytext
to the given value, so it returns all input back to the stream.
I haven't tried this myself, but I think this is the structure of the required solution.
a source to share