Full soft-wrap implementation

[adsr]

as discussed in #75

[eklausme]

First of all, thanks for this nice editor!

I have written a short introduction to mle here /1/

I have some questions regarding usage:

1. Would it be possible to catch S-home key event?
2. I would like to insert the content of a file into the currently edited file: currently I load the file into a buffer, copy it, press M-1, then uncut it. Though, once I have cut text in my 1st buffer, then the buffer from M-2 is not available. I guess, I do something wrong here.
3. Would it be possible to make mle faster during startup? It seems bline_count_chars() is a little bit slow, see /1/
4. When opening a PHP source code file then leading tabs are shown as underscore. When opening a regular Markdown file with leading tabs, they are shown as space, i.e., no underscore. I wonder why this is.
5. Line-wrap has been already mentioned.

---

/1/ Introduction to mle: Small Terminal Based Editor

[adsr]

Hi, thank you for writing up that blog post. It's better than the official help!

1. Would it be possible to catch S-home key event?

For this key press, termbox2 currently understands xterm style ("\x1b[1;2H") and rxvt style ("\x1b[7$"). It's likely your terminal emulator is not sending a distinct sequence for S-home vs home. What terminal are you using? If by chance you're using the macOS Terminal app, I believe there's a way to configure it to emit custom sequences. I helped someone do that a while back.

2. I would like to insert the content of a file into the currently edited file: currently I load the file into a buffer, copy it, press M-1, then uncut it. Though, once I have cut text in my 1st buffer, then the buffer from M-2 is not available. I guess, I do something wrong here.

Yes cut/copy buffers are local to current buffer. But there is also a global buffer that contains the last cut/copy performed in any buffer. That command (cmd_uncut_last) is bound to CM-u by default. So (with default key bindings) you can M-2, C-d q (cut all), M-1, CM-u (uncut last).

By the way, to insert the content of a file into a buffer, it may be easier to cat ... it with cmd_shell (M-e).

3. Would it be possible to make mle faster during startup? It seems bline_count_chars() is a little bit slow, see /1/

I'm very curious about this. What kind of machine are you running this on? On my machine (Linux 6.x x86_64), mle is still a bit faster than vim latest. It seems more recent versions of vim became more efficient since the last time I tested this! I do see most of the time spent in buffer_set_mmapped but bline_count_chars is near zero. I wonder if this shortcut is being skipped on your setup for some reason. If you care to, can you compile with CFLAGS="-O2 -g -pg", run the large file test, and share output of gprof -l ./mle ./gmon.out? That will do line-by-line profiling with gprof which may hint where it's spending time in that function.

4. When opening a PHP source code file then leading tabs are shown as underscore. When opening a regular Markdown file with leading tabs, they are shown as space, i.e., no underscore. I wonder why this is.

mle has 1 built-in syntax called syn_generic which applies to several common source file types including PHP. As a personal preference, I included a rule in it to underline all tabs in red. syn_generic doesn't apply to markdown (*.md) files, so you won't see red tabs nor any other styles. You can define other syntaxes to override syn_generic if you prefer. It's on my backlog to add proper syntax definitions for each language individually. In my mlerc, I do define a markdown-specifc syntax rule as well as some other rules that syn_generic doesn't cover.

5. Line-wrap has been Where is config file #75 mentioned.

Indeed, I have a branch with this that I need to revisit. It will most likely be in the next version.

[eklausme]

Thanks for the elaborate answer.

1. Indeed, the problem with S-home is unrelated to mle, but is due to st, the so called suckless terminal. I can verify that the problem does not occur in xterm.
2. Using the shell looks like the best approach -- I knew that I was missing something ;-)
3. Below you'll find the output of gprof for the smaller and larger file, see /1/ and /2/. I am using Arch Linux with kernel 6.6.2, see /3/ for all the details and mle 1.7.3-dev (mle -v). The test is with mle -NQq, so startup file should be ignored. One note: I tested this on /tmp on Arch Linux, which is essentially a RAM disk, thereby eliminating all harddisk/SSD access. Therefore any advantages, which are only due to mmap() vs. read(), will not matter much, see /4/
4. I commented line 2179 in editor.c, and can verify that it's the syntax highlighting that "confused" me.

---

/1/ "Small" file, i.e., seq 999000

Flat profile:

Each sample counts as 0.01 seconds.
  %   cumulative   self              self     total
 time   seconds   seconds    calls  Ts/call  Ts/call  name
 66.67      0.04     0.04                             bline_count_chars (buffer.c:1076 @ b328)
  8.33      0.05     0.01                             bline_count_chars (buffer.c:1053 @ b2ac)
  8.33      0.05     0.01                             bline_count_chars (buffer.c:1056 @ b2b7)
  8.33      0.06     0.01                             buffer_set_mmapped (buffer.c:370 @ e1ac)
  8.33      0.06     0.01                             buffer_set_mmapped (buffer.c:378 @ e1bb)
  0.00      0.06     0.00   999007     0.00     0.00  buffer_add_mark (buffer.c:224 @ c4f0)

/2/ "Large" file, i.e., seq 9999000

Flat profile:

Each sample counts as 0.01 seconds.
  %   cumulative   self              self     total
 time   seconds   seconds    calls  ns/call  ns/call  name
 51.79      0.29     0.29                             bline_count_chars (buffer.c:1076 @ b328)
  6.25      0.33     0.04                             buffer_set_mmapped (buffer.c:370 @ e1bf)
  5.36      0.36     0.03                             buffer_set_mmapped (buffer.c:378 @ e1bb)
  4.46      0.38     0.03                             buffer_set_mmapped (buffer.c:370 @ e1ac)
  3.57      0.40     0.02                             bline_count_chars (buffer.c:1083 @ b33c)
  3.57      0.42     0.02                             buffer_destroy (buffer.c:205 @ c610)
  3.57      0.44     0.02                             buffer_set_mmapped (buffer.c:378 @ e1a9)
  2.68      0.46     0.02  9999001     1.50     1.50  bline_count_chars (buffer.c:1044 @ b280)
  2.68      0.47     0.02                             bline_count_chars (buffer.c:1053 @ b297)
  1.79      0.48     0.01                             bline_count_chars (buffer.c:1077 @ b300)
  1.79      0.49     0.01                             bline_count_chars (buffer.c:1085 @ b32c)
  1.79      0.50     0.01                             bline_count_chars (buffer.c:1076 @ b341)
  1.79      0.51     0.01                             buffer_add_mark (buffer.c:225 @ c4fe)
  1.79      0.52     0.01                             buffer_set_mmapped (buffer.c:365 @ e166)

/3/ Uses

/4/ Performance Comparison of mmap() versus read() versus fread()

[adsr]

Interesting. I wonder why bline_count_chars is getting called for every line in the file. It's supposed to be called lazily. With no syntax definition applied I'd expect it to be called only for the lines that are rendered to the terminal. Also unsure about buffer_add_mark for the same reason.

Can you run it through a debugger or perf record -g and see where they are getting called?

[eklausme]

Results for small file seq 999000 are as below:

Samples: 275  of event 'cycles:Pu', Event count (approx.): 152924128
  Children      Self  Command  Shared Object         Symbol
+   99.16%     0.00%  mle      libc.so.6             [.] 0x00007f7b22ed2cd0                                                                           ◆
+   99.16%     0.00%  mle      mle                   [.] main                                                                                         ▒
+   77.60%     0.00%  mle      mle                   [.] editor_init                                                                                  ▒
+   77.25%     0.00%  mle      mle                   [.] editor_open_bview                                                                            ▒
+   77.25%     0.00%  mle      mle                   [.] bview_new                                                                                    ▒
+   50.43%     0.00%  mle      mle                   [.] _bview_init                                                                                  ▒
+   50.43%     0.00%  mle      mle                   [.] bview_set_syntax.part.0                                                                      ▒
+   49.42%     0.34%  mle      mle                   [.] buffer_set_tab_width                                                                         ▒
+   26.94%    25.43%  mle      mle                   [.] bline_count_chars                                                                            ▒
+   26.83%     0.00%  mle      mle                   [.] _bview_open_buffer                                                                           ▒
+   26.83%     0.00%  mle      mle                   [.] buffer_new_open                                                                              ▒
+   26.13%     0.00%  mle      mle                   [.] buffer_open                                                                                  ▒
+   21.56%     0.00%  mle      mle                   [.] editor_deinit                                                                                ▒
+   20.83%     0.00%  mle      mle                   [.] bview_destroy                                                                                ▒
+   20.83%     0.00%  mle      mle                   [.] _bview_deinit                                                                                ▒
+   20.11%     2.47%  mle      mle                   [.] buffer_destroy                                                                               ▒
+   18.51%    18.51%  mle      libc.so.6             [.] _mcount                                                                                      ▒
+    9.55%     9.29%  mle      mle                   [.] buffer_set_mmapped                                                                           ▒
+    8.15%     8.15%  mle      libc.so.6             [.] 0x0000000000113ce4                                                                           ▒
+    8.15%     0.00%  mle      libc.so.6             [.] 0x00007f7b22fbece4                                                                           ▒
+    8.04%     8.04%  mle      libc.so.6             [.] 0x00000000001577b4                                                                           ▒
+    8.04%     0.00%  mle      libc.so.6             [.] 0x00007f7b230027b4                                                                           ▒
+    3.98%     3.98%  mle      mle                   [.] _buffer_bline_free.isra.0                                                                    ▒
+    3.29%     3.29%  mle      libc.so.6             [.] 0x0000000000113e3d                                                                           ▒
+    3.29%     0.00%  mle      libc.so.6             [.] 0x00007f7b22fbee3d                                                                           ▒
+    3.26%     3.26%  mle      libc.so.6             [.] 0x00000000001577b9                                                                           ▒
+    3.26%     0.00%  mle      libc.so.6             [.] 0x00007f7b230027b9                                                                           ▒
+    2.95%     2.95%  mle      libc.so.6             [.] 0x00000000001577a8                                                                           ▒
+    2.95%     0.00%  mle      libc.so.6             [.] 0x00007f7b230027a8                                                                           ▒
+    2.53%     2.53%  mle      libc.so.6             [.] 0x0000000000113d30                                                                           ▒
+    2.53%     0.00%  mle      libc.so.6             [.] 0x00007f7b22fbed30                                                                           ▒
+    1.54%     1.54%  mle      libc.so.6             [.] 0x00000000001577bc                                                                           ▒
+    1.54%     0.00%  mle      libc.so.6             [.] 0x00007f7b230027bc                                                                           ▒
+    1.51%     1.51%  mle      libc.so.6             [.] 0x0000000000113cfe                                                                           ▒
+    1.51%     0.00%  mle      libc.so.6             [.] 0x00007f7b22fbecfe                                                                           ▒
+    1.06%     1.06%  mle      libc.so.6             [.] 0x0000000000113d23                                                                           ▒
+    1.06%     0.00%  mle      libc.so.6             [.] 0x00007f7b22fbed23                                                                           ▒
+    0.83%     0.00%  mle      [unknown]             [.] 0xffffffff914012c6                                                                           ▒
+    0.74%     0.74%  mle      libc.so.6             [.] cfree                                                                                        ▒
+    0.74%     0.00%  mle      mle                   [.] _editor_destroy_kmap                                                                         ▒
+    0.70%     0.00%  mle      [unknown]             [.] 0xffffffff914000ea                                                                           

Large file seq 9999000:

Samples: 3K of event 'cycles:Pu', Event count (approx.): 1481260200
  Children      Self  Command  Shared Object         Symbol
+   99.91%     0.00%  mle      libc.so.6             [.] 0x00007fc55e698cd0
+   99.91%     0.00%  mle      mle                   [.] main
+   80.84%     0.00%  mle      mle                   [.] editor_init
+   80.84%     0.00%  mle      mle                   [.] editor_open_bview
+   80.84%     0.00%  mle      mle                   [.] bview_new
+   53.66%     0.00%  mle      mle                   [.] _bview_init
+   53.66%     0.00%  mle      mle                   [.] bview_set_syntax.part.0
+   53.25%     1.09%  mle      mle                   [.] buffer_set_tab_width
+   33.47%    33.35%  mle      mle                   [.] bline_count_chars
+   27.18%     0.00%  mle      mle                   [.] _bview_open_buffer
+   27.18%     0.00%  mle      mle                   [.] buffer_new_open
+   26.40%     0.00%  mle      mle                   [.] buffer_open
+   19.04%     0.03%  mle      mle                   [.] editor_deinit
+   19.01%     0.00%  mle      mle                   [.] bview_destroy
+   19.01%     0.00%  mle      mle                   [.] _bview_deinit
+   18.86%     3.53%  mle      mle                   [.] buffer_destroy
+   12.00%    11.95%  mle      mle                   [.] buffer_set_mmapped
+   10.10%    10.10%  mle      libc.so.6             [.] _mcount
+    8.67%     0.00%  mle      libc.so.6             [.] 0x00007fc55e784ce4
+    8.64%     8.64%  mle      libc.so.6             [.] 0x0000000000113ce4
+    5.63%     0.00%  mle      libc.so.6             [.] 0x00007fc55e7c87b4
+    5.61%     5.61%  mle      libc.so.6             [.] 0x00000000001577b4
+    5.29%     5.29%  mle      libc.so.6             [.] 0x0000000000113d30
+    5.29%     0.00%  mle      libc.so.6             [.] 0x00007fc55e784d30
+    3.64%     0.00%  mle      libc.so.6             [.] 0x00007fc55e7c87a8
+    3.61%     3.61%  mle      libc.so.6             [.] 0x00000000001577a8
+    3.00%     3.00%  mle      mle                   [.] _buffer_bline_free.isra.0
+    2.16%     0.00%  mle      libc.so.6             [.] 0x00007fc55e7c87b9
+    2.14%     2.14%  mle      libc.so.6             [.] 0x00000000001577b9
+    2.07%     2.07%  mle      libc.so.6             [.] 0x0000000000113e3d
+    2.07%     0.00%  mle      libc.so.6             [.] 0x00007fc55e784e3d
+    1.40%     1.40%  mle      libc.so.6             [.] 0x00000000001577bc
+    1.40%     0.00%  mle      libc.so.6             [.] 0x00007fc55e7c87bc
+    1.37%     1.37%  mle      libc.so.6             [.] 0x0000000000113d23
+    1.37%     0.00%  mle      libc.so.6             [.] 0x00007fc55e784d23
+    0.97%     0.97%  mle      libc.so.6             [.] 0x0000000000157787

[adsr]

OK I see a bug in _editor_should_skip_rc. Is it faster if you run mle -N -Qq large (instead of mle -NQq large)?

[eklausme]

Yes, this change in the command line makes a difference of a factor of more than two!

File	Command	real time in s
seq 999000	time mle -NQq file	0.08
seq 999000	time mle -N -Qq file	0.04
seq 9999000	time mle -NQq file	0.82
seq 9999000	time mle -N -Qq file	0.36

Profiling mle 1.7.3-dev and the "small" files gives:

Flat profile:

Each sample counts as 0.01 seconds.
  %   cumulative   self              self     total
 time   seconds   seconds    calls  ms/call  ms/call  name
 66.67      0.02     0.02        1    20.00    20.00  buffer_set_mmapped
 33.34      0.03     0.01   999007     0.00     0.00  buffer_add_mark
  0.00      0.03     0.00     2044     0.00     0.00  bytebuf_nputs
  0.00      0.03     0.00      919     0.00     0.00  send_attr
  0.00      0.03     0.00      349     0.00     0.00  convert_num
  0.00      0.03     0.00      335     0.00     0.00  cap_trie_add
  0.00      0.03     0.00      275     0.00     0.00  utf8_char_to_unicode
  0.00      0.03     0.00      226     0.00     0.00  _editor_key_to_input
  0.00      0.03     0.00      181     0.00     0.00  _editor_init_kmap_add_binding
  0.00      0.03     0.00      181     0.00     0.00  _editor_init_kmap_add_binding_to_trie
  0.00      0.03     0.00      178     0.00     0.00  tb_set_cell
  0.00      0.03     0.00      126     0.00     0.00  tb_print
  0.00      0.03     0.00      126     0.00     0.00  tb_print_ex
  0.00      0.03     0.00      126     0.00     0.00  tb_printf_rect
  0.00      0.03     0.00      111     0.00     0.00  editor_register_cmd
  0.00      0.03     0.00       66     0.00     0.00  send_cursor_if

Results for small file with perf report:

Samples: 118  of event 'cycles:Pu', Event count (approx.): 75812993
  Children      Self  Command    Shared Object         Symbol
+   96.75%     0.00%  mle.gprof  libc.so.6             [.] 0x00007fd1ff0b8cd0
+   96.75%     0.00%  mle.gprof  mle.gprof             [.] main
+   54.64%     0.00%  mle.gprof  mle.gprof             [.] editor_init
+   54.09%     0.00%  mle.gprof  mle.gprof             [.] editor_open_bview
+   54.09%     0.00%  mle.gprof  mle.gprof             [.] bview_new
+   54.09%     0.00%  mle.gprof  mle.gprof             [.] _bview_open_buffer
+   54.09%     0.00%  mle.gprof  mle.gprof             [.] buffer_new_open
+   52.65%     0.00%  mle.gprof  mle.gprof             [.] buffer_open
+   41.02%     0.00%  mle.gprof  mle.gprof             [.] editor_deinit
+   41.02%     0.00%  mle.gprof  mle.gprof             [.] bview_destroy
+   41.02%     0.00%  mle.gprof  mle.gprof             [.] _bview_deinit
+   38.10%    10.31%  mle.gprof  mle.gprof             [.] buffer_destroy
+   18.90%    16.98%  mle.gprof  mle.gprof             [.] buffer_set_mmapped
+   11.47%    11.47%  mle.gprof  libc.so.6             [.] 0x00000000001577b4
+   11.47%     0.00%  mle.gprof  libc.so.6             [.] 0x00007fd1ff1e87b4
+   10.36%    10.36%  mle.gprof  mle.gprof             [.] _buffer_bline_free.isra.0
+    9.38%     9.38%  mle.gprof  libc.so.6             [.] 0x00000000001577a8
+    9.38%     0.00%  mle.gprof  libc.so.6             [.] 0x00007fd1ff1e87a8
+    9.27%     9.27%  mle.gprof  libc.so.6             [.] _mcount
+    6.80%     6.80%  mle.gprof  libc.so.6             [.] 0x00000000001577b9
+    6.80%     0.00%  mle.gprof  libc.so.6             [.] 0x00007fd1ff1e87b9
+    4.42%     4.42%  mle.gprof  libc.so.6             [.] 0x0000000000113ce4
+    4.42%     0.00%  mle.gprof  libc.so.6             [.] 0x00007fd1ff1a4ce4
+    3.52%     3.52%  mle.gprof  libc.so.6             [.] 0x0000000000157787
+    3.52%     0.00%  mle.gprof  libc.so.6             [.] 0x00007fd1ff1e8787
+    2.40%     2.40%  mle.gprof  libc.so.6             [.] 0x0000000000113d20
+    2.40%     0.00%  mle.gprof  libc.so.6             [.] 0x00007fd1ff1a4d20
+    1.50%     1.50%  mle.gprof  libc.so.6             [.] cfree
+    1.50%     1.50%  mle.gprof  libc.so.6             [.] 0x0000000000113d9a
+    1.50%     0.00%  mle.gprof  libc.so.6             [.] 0x00007fd1ff1a4d9a
+    1.44%     0.00%  mle.gprof  [unknown]             [.] 0xffffffffb6e000ea
+    1.44%     0.00%  mle.gprof  [unknown]             [.] 0xffffffffb6d93470
+    1.40%     1.40%  mle.gprof  libc.so.6             [.] 0x0000000000113cdb
+    1.40%     0.00%  mle.gprof  libc.so.6             [.] 0x00007fd1ff1a4cdb
+    1.39%     1.39%  mle.gprof  [unknown]             [k] 0xffffffffb6e01c60
+    1.36%     1.36%  mle.gprof  libc.so.6             [.] 0x0000000000113d30
+    1.36%     0.00%  mle.gprof  libc.so.6             [.] 0x00007fd1ff1a4d30
+    1.09%     0.00%  mle.gprof  mle.gprof             [.] editor_run
+    1.09%     0.00%  mle.gprof  mle.gprof             [.] _editor_loop
+    1.08%     1.08%  mle.gprof  [unknown]             [k] 0xffffffffb6e012a0
+    1.07%     1.07%  mle.gprof  libc.so.6             [.] 0x00000000001577b0
+    1.07%     0.00%  mle.gprof  libc.so.6             [.] 0x00007fd1ff1e87b0
+    0.97%     0.97%  mle.gprof  libc.so.6             [.] 0x00000000001577bc