Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Investigate accepting a stream of tokens as input #202

Open
tzlaine opened this issue Oct 24, 2024 · 6 comments
Open

Investigate accepting a stream of tokens as input #202

tzlaine opened this issue Oct 24, 2024 · 6 comments

Comments

@tzlaine
Copy link
Collaborator

tzlaine commented Oct 24, 2024

Not sure exactly how this should work -- should it take some abstract notion of a token, a lexrtl token, something I create, a CTRE-style thing, or what?

But it definitely deserves at least some investigation. Some parsers are very hard to write (or even ambiguous) without lexing first.
tzlaine added a commit that referenced this issue Nov 5, 2024
@tzlaine
Initial sketch of a lexer for Boost.Parser, based on using CTRE as an…
eaa53d5 
… external

dependency.  Lots of TODOs, somewhat thin testing, but the basics work.

See #202.
tzlaine added a commit that referenced this issue Nov 5, 2024
@tzlaine
Add a range adaptor for token_view; token_view -> tokens_view. Break
f5f77d8 
detail::make_input_subrange() out into its own header for reuse.

See #202.
tzlaine added a commit that referenced this issue Nov 5, 2024
@tzlaine
Add a range adaptor for token_view; token_view -> tokens_view. Break
4ac5244 
detail::make_input_subrange() out into its own header for reuse.

See #202.
tzlaine added a commit that referenced this issue Nov 7, 2024
@tzlaine
Make tokens_view regular by removing tokens_view::tokens_, which creates
f08f668 
internal references.

See #202.
tzlaine added a commit that referenced this issue Nov 7, 2024
@tzlaine
Add a token cache to tokens_view; grab a bunch of tokens at a time when
e09f527 
advancing past the end of the cache.  Filter out whitespace tokens entirely.
Make tokens_view noncopyable+nonmovable.

See #202.
tzlaine added a commit that referenced this issue Nov 7, 2024
@tzlaine
Add support for an externally-supplied token cache.
08b8214 
See #202.
tzlaine added a commit that referenced this issue Nov 7, 2024
@tzlaine
Change all the existing lex runs to compare their output against an e…
10aa943 
…xpected

sequence of tokens.

See #202.
tzlaine added a commit that referenced this issue Nov 8, 2024
@tzlaine
Specify the true/false and number token_specs in the adobe lexer test…
8854328 
… code to

use a bool and a double as its value, repsectively, instead of a string_view.

See #202.
tzlaine added a commit that referenced this issue Nov 8, 2024
@tzlaine
Add much longer lexing test; fix errors.
52d9cb5 
See #202.
tzlaine added a commit that referenced this issue Nov 8, 2024
@tzlaine
Switch from double to long double as token's floating point represent…
c1cc177 
…ation.

Therer's room for it, since it's in a union with a string_view anyway.

See #202.
tzlaine added a commit that referenced this issue Nov 8, 2024
@tzlaine
Use bitfields to reduce the size of token by a few bytes.
9e8da80 
See #202.
tzlaine added a commit that referenced this issue Nov 8, 2024
@tzlaine
Factor out a lot of the lexer.cpp tests into a separate file.
4ca0766 
See #202.
tzlaine added a commit that referenced this issue Nov 8, 2024
@tzlaine
Extend the lexer tests, addressing some of the testing TODOs; fix err…
6fdab1d 
…ors.

See #202.
tzlaine added a commit that referenced this issue Nov 8, 2024
@tzlaine
Extend the lexer tests, addressing some of the testing TODOs; fix err…
6a9a4b1 
…ors.

See #202.
tzlaine added a commit that referenced this issue Nov 8, 2024
@tzlaine
Address the remaining non-documentation TODOs in the lexer header and…
c1746bc 
… tests.

See #202.
tzlaine added a commit that referenced this issue Nov 9, 2024
@tzlaine
Initial, partial sketch of token_parser.
9380c75 
See #202.
tzlaine added a commit that referenced this issue Nov 9, 2024
@tzlaine
Initial, partial sketch of token_parser.
0c81549 
See #202.
tzlaine added a commit that referenced this issue Nov 9, 2024
@tzlaine
Initial, partial sketch of token_parser.
aac4dac 
See #202.
tzlaine added a commit that referenced this issue Nov 9, 2024
@tzlaine
Sketch in support for matching tokens against expected values in toke…
bd9c7a8 
…n_parser.

See #202.
tzlaine added a commit that referenced this issue Nov 9, 2024
@tzlaine
Sketch in operator() on token_spec_t.
ba2d757 
See #202.
tzlaine added a commit that referenced this issue Nov 9, 2024
@tzlaine
Completely rethink the relationship among token_parser, token_spec_t,…
c624681 
… and

parser_interface.  token_spec is now a variable template that generates a
parser_interface wrapping a token_parser, which parameterized on the
token_spec_t.  This way, a single token_spec use can be used to specify how to
lex, and how to parse.

See #202.
tzlaine added a commit that referenced this issue Nov 9, 2024
@tzlaine
Add token_parser::operator()(range) to support matchign against token…
93e0aa2 
…s that

producestring_views.

See #202.
tzlaine added a commit that referenced this issue Nov 9, 2024
@tzlaine
Add token_parser::operator()(range) to support matchign against token…
43b6a2f 
…s that

producestring_views.

See #202.
tzlaine added a commit that referenced this issue Nov 10, 2024
@tzlaine
Put detail::make_input_subrange back into parer.hpp.
364d8ae 
See #202.
tzlaine added a commit that referenced this issue Nov 10, 2024
@tzlaine
Reorganize the lexer-related header inclusion scheme slightly. lexer.…
b093b52 
…hpp is

now required to come before parser.hpp.

See #202.
tzlaine added a commit that referenced this issue Nov 10, 2024
@tzlaine
Sketch in the printing implemetnation for token_parser.
dd5ffe7 
See #202.
tzlaine added a commit that referenced this issue Nov 10, 2024
@tzlaine
Alter detail::make_input_subrange() to return its input unchanged whe…
0935656 
…n the

range's value type is a specializtion of token.

See #202.
tzlaine added a commit that referenced this issue Nov 10, 2024
@tzlaine
Change token<> to include the position of the start of the token in the
c6331b2 
underlying sequence, and change the way that the error handler is invoked, so
that it detects token iterators, and passes iterators into the underlying
range to the error handler, instead of the token iterators.

See #202.
tzlaine added a commit that referenced this issue Nov 10, 2024
@tzlaine
Add a std::string data member, and an override of what() to parser_er…
ff22ab1 
…ror, to

ensure that the messge is preserved exactly when translating exceptions in
failed lexer parsing.

See #202.
tzlaine added a commit that referenced this issue Nov 10, 2024
@tzlaine
Fix multiple thinkos in tokens_view.
a55603c 
See #202.
tzlaine added a commit that referenced this issue Nov 30, 2024
@tzlaine
Slight simplification of code trying to detect token iterators.
eab7e82 
See #202.
tzlaine added a commit that referenced this issue Dec 2, 2024
@tzlaine
Correct memory safety issue in converting token iteratos to underlying
3fd285c 
iterators.

See #202.
tzlaine added a commit that referenced this issue Dec 2, 2024
@tzlaine
Fix return statement on the wrong side of macro guard.
cafd04c 
See #202.
tzlaine added a commit that referenced this issue Dec 2, 2024
@tzlaine
Initial sketch of adapting string_view_parser for use in token parsing.
05a110c 
See #202.
tzlaine added a commit that referenced this issue Dec 2, 2024
@tzlaine
Initial sketch of adapting symbol_parser for use in token_parsing.
220eb69 
See #202.
tzlaine added a commit that referenced this issue Dec 2, 2024
@tzlaine
Initial sketch of adapting lexeme and skip directives for use in token
c66930a 
parsing.

See #202.
tzlaine added a commit that referenced this issue Dec 2, 2024
@tzlaine
Extend support for token parsing to the prefix*parse() API. Add tests…
b811e71 
… for the

full set of *parse() functions that support token parsing; fix errors.

See #202.
tzlaine added a commit that referenced this issue Dec 3, 2024
@tzlaine
Adapt symbol_parser for use in token_parsing.
8f87912 
See #202.
tzlaine added a commit that referenced this issue Dec 3, 2024
@tzlaine
Initial sketch of adapting lexeme and skip directives for use in token
06816ab 
parsing.

See #202.
tzlaine added a commit that referenced this issue Dec 3, 2024
@tzlaine
Extend support for token parsing to the prefix*parse() API. Add tests…
1995258 
… for the

full set of *parse() functions that support token parsing; fix errors.

See #202.
tzlaine added a commit that referenced this issue Dec 3, 2024
@tzlaine
Add char to the set of types that can be used as token attributes.
b1c5c4b 
See #202.
tzlaine added a commit that referenced this issue Dec 3, 2024
@tzlaine
Trim trailing null from array ranges adapted by the to_tokens range a…
9733445 
…daptor.

See #202.
tzlaine added a commit that referenced this issue Dec 5, 2024
@tzlaine
Fix behavior of tokens_view::iterator::base() in the non-common_range…
2b48b86 
… case.

See #202.
tzlaine added a commit that referenced this issue Dec 5, 2024
@tzlaine
Pull the single-character-fetching logic of the token parsing modes o…
d69bf89 
…f the

character parsers out into a standalone detail:: function; use it in all the
character parsers.

See #202.
tzlaine added a commit that referenced this issue Dec 5, 2024
@tzlaine
WIP Add lexer_and_parser_terminals.cpp, based on parser.cpp, to more …
71499c9 
…fully

exercise the token parsing code.

See #202.
tzlaine added a commit that referenced this issue Dec 6, 2024
@tzlaine
Pull the single-character-fetching logic of the token parsing modes o…
e2f0159 
…f the

character parsers out into a standalone detail:: function; use it in all the
character parsers.

See #202.
tzlaine added a commit that referenced this issue Dec 6, 2024
@tzlaine
WIP Add lexer_and_parser_terminals.cpp, based on parser.cpp, to more …
9bd772b 
…fully

exercise the token parsing code.

See #202.
tzlaine added a commit that referenced this issue Dec 6, 2024
@tzlaine
Add proper const/non-const interop to tokens_view::iterator.
c3667f5 
See #202.
tzlaine added a commit that referenced this issue Dec 6, 2024
@tzlaine
WIP Add lexer_and_parser_terminals.cpp, based on parser.cpp, to more …
df06e6e 
…fully

exercise the token parsing code.

See #202.
tzlaine added a commit that referenced this issue Dec 7, 2024
@tzlaine
WIP Add lexer_and_parser_terminals.cpp, based on parser.cpp, to more …
7e088cf 
…fully

exercise the token parsing code.

See #202.
tzlaine added a commit that referenced this issue Dec 7, 2024
@tzlaine
WIP Add lexer_and_parser_terminals.cpp, based on parser.cpp, to more …
af04be4 
…fully

exercise the token parsing code.

See #202.
tzlaine added a commit that referenced this issue Dec 7, 2024
@tzlaine
WIP Add lexer_and_parser_terminals.cpp, based on parser.cpp, to more …
280c4d2 
…fully

exercise the token parsing code.

See #202.
tzlaine added a commit that referenced this issue Dec 7, 2024
@tzlaine
Add lexer_and_parser_terminals.cpp, based on parser.cpp, to more fully
6a94e1e 
exercise the token parsing code; fix errors.

See #202.
tzlaine added a commit that referenced this issue Dec 8, 2024
@tzlaine
Add lexer_and_parser_terminals.cpp, based on parser.cpp, to more fully
5c036a7 
exercise the token parsing code; fix errors.

See #202.
tzlaine added a commit that referenced this issue Dec 8, 2024
@tzlaine
Add Doxygen comments to all normalize_iterators() overloads, and indi…
0463ecb 
…cate

which error handling function apply this function to their inputs and which do
not.

See #202.
tzlaine added a commit that referenced this issue Dec 8, 2024
@tzlaine
Complete initial pass on token parsing documentation.
ff10596 
See #202.
tzlaine added a commit that referenced this issue Dec 8, 2024
@tzlaine
Address TODOs about resolving the value used in detail::token_with_va…
c6d35e8 
…lue, and

doing transcoding for the match in detail::token_with_string_view.

See #202.
@tzlaine
Copy link
Collaborator Author

tzlaine commented Dec 8, 2024

Ok, this stuff is pretty much ready to go. I say "pretty much" only because I don't want to merge it quite yet, since I don't really have a big enough use case for it to really try it out.

@jefftrull, that's where you come in. :) Please give it a try and tell me what's under-documented, hard to use, etc. You can find all the code on the ctre branch, and the pre-built updated docs can be found on the ctre_doc branch (just point your browser at parser/doc/html/index.html).

@jefftrull
Copy link

Things are a little crazy right now but I'm very excited to try this. Thanks for taking on this big feature!

@tzlaine
Copy link
Collaborator Author

tzlaine commented Dec 10, 2024

No worries! The branch isn't going anywhere.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@tzlaine @jefftrull