RecursionError with learn_one and progressive_val_score

[javiuni]

Hello,

I have a problem "RecursionError: maximum recursion depth exceeded while calling a Python object" with the predict_one() and evaluate.progressive_val_score() methods.

My dataset is a csv file of 84000 rows and 63 columns.

My code for evaluate.progressive_val_score() is:

dataset = stream.iter_csv('dataset.csv', target='output', **params) #**params contains 63 'converters'

model = (
    preprocessing.StandardScaler() |
    rules.AMRules(
        drift_detector=drift.ADWIN()
    )
)

metric = metrics.MAE()
evaluate.progressive_val_score(dataset, model, metric)

After this attempt, I tried an alternative that still gives the same RecursionError. My code for predict_one() is:

dataset = stream.iter_csv('dataset.csv', target='output', **params) #**params contains 63 'converters'

model = (
    preprocessing.StandardScaler() |
    rules.AMRules(
        drift_detector=drift.ADWIN()
    )
)

metric = metrics.MAE()

for x, y in dataset:
    y_pred = model.predict_one(x)
    model = model.learn_one(x, y)
    metric = metric.update(y, y_pred)
    print(metric)

What's wrong? I checked that for at least the first 5000 rows the predict_one() works correctly. Then the RecursionError appears.

Thanks

[MaxHalford]

Hello there!

@smastelini can I let you take a look? 🙏

[smastelini]

Hi @javiuni. Thanks for reporting that. Could you provide more details about your dataset?

I've seen such recursion errors in the past. They mostly appeared due to:

• Way too many distinct values in numerical features (the memory measurement originated the error)
• Too many unique categorical values

In both cases, the recursion error was generated while measuring memory. As far as I recall, we had changed the recursive memory measurement to an iterative one. I'll check that.

In the meantime, may I suggest you a quick test? Try using tree.splitter.QOSplitter() as the splitter algorithm:

# from river import tree

dataset = stream.iter_csv('dataset.csv', target='output', **params) #**params contains 63 'converters'

model = (
    preprocessing.StandardScaler() |
    rules.AMRules(
        drift_detector=drift.ADWIN(),
        splitter=tree.splitter.QOSplitter()
    )
)

metric = metrics.MAE()
evaluate.progressive_val_score(dataset, model, metric)

[javiuni]

First of all, thank you for your quick answers.

The code proposed by @smastelini has fixed RecursionError for both methods (predict_one() and progressive_val_score()).

Anyway I give you more details of my dataset in case they could help to detect some bug.

All my dataset is composed by float values in the range of -13.5 to 63060, with random steps. And these float values move in different ranges in each column. For example the ranges of some columns:

Column 1: 696 to 1584
Column 2: -13.5 to 12.9
Column 3: 1 to 589
Column 4: 10 to 3910
Column 24: 520 to 740
Column 35: 12.2 to 14
Column 46: 7880.7 to 33254.3

I hope I can be of help