Organization of the code for the sequence functions

[robert-strandh]

The modules for the sequence functions are a mess right now.

The reason for that is that I started implementing them by manually
specializing on all kinds of things such as vector type, whether
:START and :END where given, etc. As it turns out, there are so many
cases to consider that the code became impossible to understand, and
especially impossible to test, because of the many cases.

So then I started a different version "Small", that was not very
ambitious in terms of performance, but that would instead take up a
reasonable amount of space. Finally, I started yet another version
"Tiny" for which I tried to factor as much code as possible in order
to make the resulting executable code very small.

Recently, I figured out how to accomplish the fast version. The
technique is currently being written up in the form of a paper, and I
have implemented some of the functions. Each function now has a file
named after it and the file contains only a few functions. The main
idea for the fast version is to use macros that duplicate the body of
the code according to the different specialization criteria, and to
have a seemingly inefficient loop that will also test these criteria.
But a good compiler will then specialize the loop in each duplicated
body so that the result is basically identical to the manual
specialization I did at first.

Now, as it turns out, with those macros, the code for the sequence
functions themselves is now boiled down to something very general.
And I recently realized that I could accomplish the "Small" and "Tiny"
versions by simply providing different versions of these macros.

Each version ("Fast", "Small", "Tiny") would then have its own ASDF
system definition and its own version of those macros. All versions
would have a file containing a package definition of the package
sicl-sequence, but the definition would be different for each
version in that it would :USE a package that contains a package
containing the specific macro definition.

I think that each version should have its own sub-directory, because
there are going to be sequence functions that can not be managed by
these macros and that may require specific code. I am thinking in
particular about the sorting functions.

So I need some suggestions on how to organize all this code into
directories, files, and ASDF system definitions that makes the entire
thing maintainable and understandable. Until such an organization is
clear to me, I don't want to delete all the obsolete existing code.

[ghost]

I think one way to deal with that (without puting a burden on development freedom that is the priority for now I think) may be to add some information in system definitions. What I can think of is

:version and "0" or "-1" would mean do not bother with this system
:description, :long-description would provide information about a system that is interesting to load/use
:if-feature : I've seen that in ASDF3 (and up). It could be a way to describe and build systems that provide packages that are alternate to each other, as the sicl-sequence you describe above, by putting :sicl-sequence-small in features before building for exemple.

Also file system structure of SICL could also be a hint of the hierarchy of systems that can be loaded when necessary.

It then could the be simple to provide functionality to help someone that want to use/test some SICL code without having a good knolewdge of the structure of the code base.

From ASDF current manual :
For instance, :if-feature (:and :x86 (:or :sbcl :cmu :scl)) specifies that the given component is only to be compiled and loaded when the implementation is SBCL, CMUCL or Scieneer CL on an x86 machine. You cannot write it as :if-feature (and x86 (or sbcl cmu scl)) since the symbols would not be read as keywords.

[robert-strandh]

I think I have an idea for how to organize the code.

Let Sequences be the name of the main directory for the sequence
functions.

Create sub-directories in Sequences named Common, Fast, Small, and
Tiny.

In Sequences, create ASDF system definitions named sicl-sequence-fast,
sicl-sequence-small, and sicl-sequence-tiny.

In the Common sub-directory, create a packages.lisp file defining the
package sicl-sequence. Also in Common, put the files containing the
main code for the sequence functions that have the same definition in
all systems, in particular find.lisp, position.lisp, etc.

In Fast, Small, and Tiny, put specific file. In particular, in each
directory, there will be a utilities.lisp file that contains
definitions of the main macros.

The system definitions look like this:

(defsystem sicl-sequence-fast
  :serial t
  :components
  ((:file "Common/packages")
   (:file "Fast/utilities")
   (:file "Common/find")
   (:file "Common/position")
   ...))

with the difference being the second line, and any specific files.

Any opinions about this organization?

[ghost]

I think it's a good way to keep those files organized. That is it's clear and simple and consistent with the other parts of SICL :)