Skip to content
This repository has been archived by the owner on Mar 8, 2020. It is now read-only.

uast.TokenOf returns unexpected results for position nodes #404

Open
bzz opened this issue Jun 4, 2019 · 0 comments
Open

uast.TokenOf returns unexpected results for position nodes #404

bzz opened this issue Jun 4, 2019 · 0 comments

Comments

@bzz
Copy link
Contributor

bzz commented Jun 4, 2019

As a user, I expect to be able to extract all the tokens from UAST using helpers from #363 with something like:

	var tokens []string
	iter := tools.NewIterator(root, tools.PreOrder)
	for n := range tools.Iterate(iter) {
		token := uast.TokenOf(n)
		if token != "" {
			tokens = append(tokens, token)
		}
	}

and then by printing all the tokens get a "tokenized" version of particular file.

But uast.TokenOf returns non-empty results e.g for position nodes, with is although documents, is not very intuitive.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant