lexer-generator

I'm making use of Extras to keep track of current line information, and in my parser I also want to peek the next token.

The trouble is that if I wrap the Lexer using Peekable then I no longer have a way to access the .extras field.

lexer.token is mentioned as a solution in #82, but that doesn't seem to be available anymore.

I assume I can make my own peekable iterator that

Add an option to NimlLexer to lexNext / lexIter provide a specified token when the lexer reaches EOF only once.

Example:

# lexer setup
lexer.ignoreIf = someProc
lexer.produceEOFToken(tokenForEOF) # if it is not given, reaching EOF produces no token.

# use lexer
...

related: #70

to make the circle complete, support constructing a regular expression out of a DFA.

This is an implementation detail, and may be nice for future features.