Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Syntax highlighting for PTX code #275

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

Ellipse0934
Copy link

@Ellipse0934 Ellipse0934 commented Jan 18, 2022

Removes 🐍 pygments dependency and adds highlighting similar to base.

We can create a different highlighting scheme which should be easy to do but I don't have any ideas about that.

image

We have a simple lexer which processes one line at a time.

In general a PTX file has

  • pre-processor directives: We assume these will not be present
  • Comments: We assume there will never be any multiline comments
  • Statements: Which are either be a directive or an instruction
  • Directive: e.g. .reg .b32 r1, r2
  • Instructions: e.g. label: mov.b32 r1, %tid.x
  • Labels, Instruction name (mov, st, ld), data types (b32, f64),
  • Identifiers and Numbers
    (NVIDIA PTX docs)

We can use regex to distinguish between the various tokens and do a more complex theme like:
image
But I would personally not prefer that.

@codecov
Copy link

codecov bot commented Jan 18, 2022

Codecov Report

Merging #275 (64f6123) into master (09f8b5a) will decrease coverage by 0.55%.
The diff coverage is 2.43%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #275      +/-   ##
==========================================
- Coverage   86.75%   86.19%   -0.56%     
==========================================
  Files          22       22              
  Lines        2023     2246     +223     
==========================================
+ Hits         1755     1936     +181     
- Misses        268      310      +42     
Impacted Files Coverage Δ
src/reflection.jl 58.86% <2.43%> (-19.40%) ⬇️
src/execution.jl 92.00% <0.00%> (-8.00%) ⬇️
src/ptx.jl 93.67% <0.00%> (-2.65%) ⬇️
src/error.jl 28.00% <0.00%> (-2.44%) ⬇️
src/jlgen.jl 80.58% <0.00%> (-1.73%) ⬇️
src/gcn.jl 84.40% <0.00%> (-1.04%) ⬇️
src/irgen.jl 94.05% <0.00%> (-0.96%) ⬇️
src/mcgen.jl 97.36% <0.00%> (+0.07%) ⬆️
src/validation.jl 96.92% <0.00%> (+0.12%) ⬆️
src/debug.jl 97.22% <0.00%> (+0.16%) ⬆️
... and 6 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 09f8b5a...64f6123. Read the comment docs.

@maleadt
Copy link
Member

maleadt commented Jan 19, 2022

Thanks! Pushed some small changes.

I think we'll have to improve the tokenizer before this can work though, simple things like ret; or __test = {1, 2} are mismatched into resp. ret;, {1, 2} tokens. I wonder how hard it would be to just use a proper parser generator like RBNF.jl. although I couldn't find a simple BNF grammar for PTX. Here's one I found for ANTLR: https://gist.github.com/maleadt/44966c96fa389b857910ccd3a4286bf6

istok(r"^0[dD]{hexdigit}{16}") # double-precision floating point
print_tok(tok, :number)
# TODO: function names
# TODO: labels as RHS
Copy link
Author

@Ellipse0934 Ellipse0934 Jan 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: labels as RHS

What does this mean ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a distinct color for labels (foo:), but that isn't matched when the label is used as a right-hand-side value (bra foo). Generally that requires a proper parser though, but if the cases are limited we could special-case it.

@Ellipse0934
Copy link
Author

Yes, I missed the tokeniser failures. I have added some additional regex to identify the type of token. Also, some colour changes have been reverted but we will fix that before merge. I wanted to check if the current strategy is fine. It seems easy to add support for functions and other features.

I wonder how hard it would be to just use a proper parser generator

Is there any use for this besides code highlighting ? I don't think a BNF for this should be particularly difficult but then I don't have any experience in this area. There do appear to be some ANTLR to BNF converters out there. Maybe it's worth looking into one of those.

Remaining TODOs:

  • Add missing instructions/declarations (e.g call/bra.uni , %p<0-..>)
  • Tests
  • Final colour scheme

@maleadt
Copy link
Member

maleadt commented Jan 22, 2022

s there any use for this besides code highlighting ? I don't think a BNF for this should be particularly difficult but then I don't have any experience in this area. There do appear to be some ANTLR to BNF converters out there. Maybe it's worth looking into one of those.

Currently not, no. If we can get reasonable regex-based highlighting, it's probably overkill to consider a proper grammar (which would also incur some latency when getting compiled), but if we end up having to write down individual regular expressions for every token type it's a small step to a real grammer (where expressing those might be more compact even).

But I'm just spitballing here, I don't have experience with parser generators in Julia either.

@Ellipse0934
Copy link
Author

I wrote some implementations and got confused as to whether we want to be minimalist here or do something closer to an actual parser with proper tokens. The Julia replacement for Pygments is Highlights.jl and they have an issue on this. The best course of action imo is to implement this there and use that here. I am going to implement a lexer there first. If that one is unsatisfactory I can resume this PR otherwise close this PR.

@maleadt
Copy link
Member

maleadt commented Feb 8, 2022

I didn't know about Highlights.jl -- that looks to be the best place to implement PTX support, indeed. Note that you can also find some regular expression-like things in the PTX ISA manual, e.g. https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#identifiers, that may make it easier to implement a lexer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants