Doubt of dataset and data modes

[ecastillot]

Hi creators of Spektral,

First of all, thank you for creating this amazing package.
I want to make a GNN to do node level prediction task. I can create a lot of small graphs in my dataset to train the model and feed the graph batch model. I have different dataframes with specific information to build different graphs . Also, I don't want make a big graph with all small graphs because each small graph don't have any connection with the other one. How would you advise me to generate the dataset and the data generator?

My idea is use Spektral Dataset class and after SingleLoader class. However, I don't understand if when I initialize the Dataset, I inmmediately load all small graphs. However, I don't want it because I don't have enough memory space to process all the data. Therefore, I want to load by graph batches.

Thanks for your attention,
Regards,
Emmanuel

[danielegrattarola]

Hey Emmanuel,

How would you advise me to generate the dataset and the data generator?

The implementation of the dataset does not depend on the data loader. So, just create a dataset with your graphs and then you can use any data loader that best fits your requirements.

I would advise you to use the DisjointLoader, unless you have some particular reason not to use it. Don't worry about having many disconnected graphs, there won't be any exchange of information between graphs that are not connected, the loader is designed to take care of that.

I don't have enough memory space to process all the data. Therefore, I want to load by graph batches.

Unfortunately that's a feature that is still missing in Spektral, I haven't had the time to work on it lately.
For now, you'll have to store all your graphs in a list in memory.

Cheers

[ecastillot]

Thank you very much!, DisjointLoader is what I need.
I look forward to this feature in Spektral.

[ecastillot]

Hi @danielegrattarola

Unfortunately that's a feature that is still missing in Spektral.

Although I have to store all graphs in a list in memory. Can I create a DataGenerator to use model.fit_generator for real-time data feeding in the Keras model.?
I mean, by each called in the DataGenerator, then save a list of graphs and fit the model.

[danielegrattarola]

I don't think fit_generator is supported anymore, it's just fit.
The Loaders in Spektral are generators already, maybe you can adapt them to your use case?