fusion/pointers: add pointer arithmetic operators

[timotheecour]

• safer than using cast and inlining the code contained in the operators from this PR
• future compiler work can enable the following:

import fusion/pointers
proc fn =
  var a = @[10, 11, 12]
  let pa = a[0].addr # or from another other source, eg FFI returning directly a ptr
  doAssert pa[2] == 12 # generate a warning: must use {.cast(safe).}
  {.cast(safe).}:
    doAssert pa[2] == 12 # ok: warning is not triggered

nothing will break, the only thing is that warnings will eventually get triggered once {.cast(safe).}: is implemented, and {.cast(safe).}: can then be used to avoid triggering that warning.

this avoids using non-standard operators like +! as was suggested in nim-lang/Nim#15490 (comment)
and it gives the maximum flexibility: allows user to add --warningAsError:UnsafeBlock, or --warning:UnsafeBlock:off or localize those in code, or use {.cast(safe).}: blocks

as mentioned in nim-lang/Nim#15490 (comment)

+ has one obvious meaning for ptr T just like it does in C, C++, D, swift etc (all support pointer arithmetic as in this PR); unlike for eg for concatenation where nim rightfully departs from python and uses & to avoid ambiguities. The leibniz argument applies.

often requested, eg: Added ptrops to stdlib by awr1 · Pull Request #12101 · nim-lang/Nim

[Araq]

A single pointer is not an array.

[timotheecour]

how about this; slightly more verbose than [], []= but still short enough:

# `[]` =>
template `at`*[T](p: ptr T, off: int): T = ...
# `[]=` =>
template `at=`*[T](p: ptr T, off: int, val: T) = ...

it would still allow this:

echo p.at(1)
p.at(2) = 2
p.at(2) -= 3

(at naming precedent: http://www.cplusplus.com/reference/vector/vector/at/ or https://riptutorial.com/opencv/example/6394/access-individual-pixel-values-with-cv--mat--at-t---)

[Araq]

We already have the toUncheckedArray proc to offer array indexing.

[timotheecour]

toUncheckedArray is useful but not as a replacement for that; this module is about convenience when dealing with low-level code so you use cast less often and make the intent clearer

p[1]
vs
p.toUncheckedArray[1]

That's why it's been implemented in so many places (with more or less good implementations eg many implementations have issues eg are not safe wrt multiple template argument evaluation bugs) and requested also in many places as shown here #21 (comment)

Among other things, it makes it in particular easy to adapt C/C++ code to nim with the simplest possible syntax.

[Araq]

In my own low level code I never needed the * sizeof(T) part. It's not intuitive and probably error-prone. Why try to outsmart the programmer who chose to operate on a very low level for a reason?

[timotheecour]

see #21 (comment)

[Araq]

A single pointer is not an array.

[timotheecour]

see #21 (comment)

[timotheecour]

In my own low level code I never needed the * sizeof(T) part. It's not intuitive and probably error-prone. Why try to outsmart the programmer who chose to operate on a very low level for a reason?

I have to disagree here. Here are just a few examples:

# Nim/lib/nimhcr.nim:393:14:
      curr = cast[ptr cstring](cast[int64](curr) + sizeof(ptr cstring))
      =>
      curr += 1

# Nim/lib/pure/coro.nim
coro.stack.top = cast[pointer](cast[ByteAddress](coro) + sizeof(Coroutine))
=>
coro.stack.top = cast[pointer](coro + 1)

# Nim/lib/system/excpt.nim:109:
zeroMem(cast[pointer](cast[int](s)+%sizeof(GcFrameHeader)), s.len*sizeof(pointer))
=>
zeroMem(cast[pointer](s + 1), s.len*sizeof(pointer))

# Nim/lib/system/gc_regions.nim (with `type Chunk = ptr BaseChunk`)
r.bump = fresh +! sizeof(BaseChunk)
=>
r.bump = fresh + 1
# many other instances; system/gc* code could likely be made more maintainable with
# less `pointer` and more `ptr[T]` + standard ptr[T] arithmetics, but that's out of scope for this discussion

probably error-prone

using sizeof when computing offsets from a ptr[T] is arguably less error prone than relying on programmer to cast[ByteOffset](x) first then compute the offset themselves from a sizeof(T), especially when you have aliases and non obvious types eg type Chunk = ptr BaseChunk

There's plenty of evidence for this:

• even C has pointer arithmetics for T* p; p+=3;, interpreted as p = (T*)( (ptrdiff_t)(p) + 3*sizeof(T) ), and for good reasons;
• ditto with C++, D, swift and many languages that support pointers

Why try to outsmart the programmer who chose to operate on a very low level for a reason?

There's no need to obfuscate code just because it involves pointers/ptr[T]. Because of alignment reasons, the majority of cases you want standard C pointer arithmetic semantics on ptr[T]; in more complex cases (eg when you're combining several operations at once, eg field offset + pointer arithmetics etc), you can always use cast[ByteOffset] with arbitrary manipulation, but that's arguably more error prone.

finally, this is oft-requested and re-implemented, often poorly

• https://github.com/fowlmouth/nimlibs/blob/master/fowltek/pointer_arithm.nim
• nim-ranges/typedranges.nim at master · status-im/nim-ranges
• nim-ranges/ptr_arith.nim at master · status-im/nim-ranges
• nim-stew/ptr_arith.nim at master · status-im/nim-stew
• nim-stew/ptrops.nim at master · status-im/nim-stew
• pointer_arithmetic_arraymancer.nim: https://gist.github.com/mratsim/0bf8a8ea45b36f7bf79c43c9a98411a2#file-pointer_arithmetic_arraymancer-nim
• ptrmath.nim: https://gist.github.com/oltolm/1738289b5ac866ec6a7e4ef20095178e
• utils/binary.nim at master · nim-appkit/utils

plenty of forum posts asking about how to do it too:

• ptr arithmetics? - Nim forum
• pointer arithmetic example? - Nim forum
• Passing c-array to nim - Nim forum
• Pointer Arithmetic and constructs like castvar int += sizeof(a[0]) - Nim forum
• Nim pointer arithmetic
• Pointer arithmetic example in Nim (int16) - Nim Snippet - glot.io
• https://gitter.im/nim-lang/Nim?at=5c3653044ed1827877851fbc

I was wondering if Nim has pointer arithmetic as in C?

• Casting basics (a few questions) - Nim forum
• Help me optimize this small Nim port to the speed of the original C version - Nim forum
• For TITLE: An Attempt to Access NumPy Array Data rom Nim - Nim forum
  For pointer arithmetics, if you really needs them, you can follow this post for a safe well defined scope in which pointer arithmetics is allowed: https://forum.nim-lang.org/t/1188#7366
• An Attempt to Access NumPy Array Data rom Nim - Nim forum
• Porting an encryption algorithm from C - Nim forum
  There's a rather unusual encryption algorithm I'm trying to port to Nim from C, but it uses pointer arithmetic I'm unsure if Nim supports or not (yet?):

[Araq]

I don't see the benefit in

coro.stack.top = cast[pointer](coro + 1)

r.bump = fresh + 1

Frankly, the 1 here is pure obfuscation for me.

In fact, when we analyse the situation further:

r.bump = fresh +! sizeof(BaseChunk) # ok, we skip the header. There is 1 header, not 2 and not 3

vs

r.bump = fresh + 1 # only 1 here is valid, not 2 and not 3 because we don't have an array of headers...

Often clarity in programming is achieved by ignoring pretty much everything of what was done in C and Unix.

[timotheecour]

I don't see the benefit in

There is plenty of evidence pointing to the opposite, see the re-implementations and form posts I've linked. It is also my experience that > 90% of the time when dealing with ptr[T] variables, what I need are procs in this PR, it leads to safer code than using cast directly. cast is then only used when necessary, not when doing regular sizeof(T)-aligned arithmetics, leading to more maintainable code.