potential bug: wrap now autostarting iterators that were previously not autostarted prior to optimisation work

[jeswr]

Placeholder - will try create a minimal example or debug tommorow :)

[jeswr]

On buggy case that I encountered today is the following which blocks on the second line

const iter = wrap(wrap([1, 2, 3], { autoStart: true }))
await iter.toArray()

cc @jacoscaz

[RubenVerborgh]

Note that I slightly changed toArray behavior in 174d87c: I found a bug where iterators that had already ended would cause a hang. Probably unrelated though.

[jacoscaz]

Given the conversation around #45, should we address this before that merge or after?

[RubenVerborgh]

I think before; this seems like a bug that we still want fixed in the current vminor.

[jacoscaz]

@jeswr I'm not sure that is related to autoStart, actually. If autoStart is passed as an option to wrap(), regardless of its value, then the source is passed on to the TransformIterator constructor:

AsyncIterator/asynciterator.ts

Lines 2127 to 2128 in 915a857

	if (options && ('autoStart' in options || 'optional' in options || 'source' in options || 'maxBufferSize' in options))
	return new TransformIterator<T>(source as MaybePromise<AsyncIterator<T>>, options);

However, with the current main the following doesn't work:

const iter = new TransformIterator([1, 2, 3])
const res = await iter.toArray();

Therefore, I think what's wrong here is:

1. the fact that we're passing an array as-is to TransformIterator
2. the fact that we're not catching 1) as a mistake even though we're using TypeScript

Which is not to say that the premise of this issue is wrong. The default behavior if no autoStart option is provided might have changed, indeed.

[RubenVerborgh]

1. the fact that we're passing an array as-is to TransformIterator

Indeed; that is not supported. I would have expected the type system to catch it, and TransformIterator to throw (because it cannot attach listeners).

[RubenVerborgh]

Is it this? 25085c4 / #80

[jacoscaz]

I think the issue is at line 2128 (see above), which is in the body of wrap(). source (which might be an array) is being cast to MaybePromise<AsyncIterator<T>>, thus bypassing type checks in the TransformIterator constructor.

[RubenVerborgh]

I think the intention of the original wrap typing was to not accept autoStart options for IterableSource, and thus only for AsyncIterator. So now the question is: do we adhere to the intended typings, or do we update the function such that options are possible also for IterableSource?

In any case, it does not seem like a regression because this would not have worked before the wrap rewrite.

[jacoscaz]

IMHO and looking very long-term, the autoStart option should disappear (#25) . When working on new features I would try to limit support for it as much as possible while keeping backward compatibility when it comes to minor versions. I would adhere to the intended typings, disallowing autoStart for IterableSources.

[RubenVerborgh]

autoStart will absolutely disappear in the next vmajor.

@jeswr Are you okay with restricting the typings, so reverting the solution to #80? (Which begs the question, for which kinds of arguments do we want to allow the autoStart option?)

[jeswr]

for which kinds of arguments do we want to allow the autoStart option?

In #45 I renamed it to preBuffer (I think @RubenVerborgh s suggestion). It should only be available on iterators that extend the buffered iterator. When true the buffer will be pre-filled before the first .read() call; when false the buffer will remain empty until the first .read() call is made.

[jeswr]

In the case of wrap I think we should remove the options parameter entirely to be completely honest. By setting maxBuffer or preBuffer users can suddenly introduce buffering even if they are just wrapping synchronous iterators.

I think it is better to force users to use the TransformIterator in the limited cases where they actually want to use it (I imagine there are less than a handful across all of Comunica where we would need to do this); and include a few docs on upgrading.

[RubenVerborgh]

remove the options parameter entirely to be completely honest

But we can't for backward compatibility reasons.

It should only be available on iterators that extend the buffered iterator.

…and we probably don't want this kind of inheritance anymore, btw. I think we should do away with BufferedIterator as a base class, and only have it as a decorator where it is needed.

[RubenVerborgh]

@jacoscaz Path of least resistance seems to be accepting arrays; see #91.

[jeswr]

But we can't for backward compatibility reasons.

This suggestion was for the next vmajor

[jacoscaz]

…and we probably don't want this kind of inheritance anymore, btw. I think we should do away with BufferedIterator as a base class, and only have it as a decorator where it is needed.

Yes, buffering and transforms are orthogonal concerns and addressing them independently of one another might be a good idea.