Issue Predicting Node Outputs

[akilgall]

Hello,

I am attempting to use DeepTrack to predict node outputs, but I'm consistently running into a bug in the framework. As a minimal working example - If I modify the cell_migration_analysis notebook to predict the mean node position of each track, I get the following error:

---

Generating 511 / 511 samples before starting training

---

ValueError Traceback (most recent call last)
Cell In[34], line 17
7 generator = GraphGenerator(
8 nodesdf=nodesdf,
9 properties=["centroid"],
(...)
13 **variables.properties()
14 )
16 with generator:
---> 17 model.fit(generator, epochs=10)

File ~/miniconda/envs/DT_TEST4/lib/python3.9/site-packages/deeptrack/models/utils.py:248, in KerasModel.fit(self, x, batch_size, generator_kwargs, *args, **kwargs)
245 # Code is not actually unreachable if fit crashes.
246 return None
--> 248 return self.model.fit(x, *args, batch_size=batch_size, **kwargs)

File ~/miniconda/envs/DT_TEST4/lib/python3.9/site-packages/keras/src/utils/traceback_utils.py:70, in filter_traceback..error_handler(*args, **kwargs)
67 filtered_tb = _process_traceback_frames(e.traceback)
68 # To get the full stack trace, call:
69 # tf.debugging.disable_traceback_filtering()
---> 70 raise e.with_traceback(filtered_tb) from None
71 finally:
72 del filtered_tb

File ~/miniconda/envs/DT_TEST4/lib/python3.9/site-packages/deeptrack/models/gnns/generators.py:153, in ContinuousGraphGenerator.getitem(self, idx)
152 def getitem(self, idx):
--> 153 batch, labels = super().getitem(idx)
155 # Extracts minimum number of nodes in the batch
156 numofnodes = list(map(lambda _batch: np.shape(_batch[0])[0], batch))

File~/miniconda/envs/DT_TEST4/lib/python3.9/site-packages/deeptrack/generators.py:366, in ContinuousGenerator.getitem(self, idx)
356 return (
357 tuple(
358 [
(...)
363 np.array(labels),
364 )
365 else:
--> 366 return np.array(data), np.array(labels)

File ~/miniconda/envs/DT_TEST4/lib/python3.9/site-packages/deeptrack/image.py:319, in Image.array(self, *args, **kwargs)
316 """Convert to numpy array."""
317 #print("self.to_numpy(): ", self.to_numpy())
318 #print("self.to_numpy()._value: ", self.to_numpy()._value)
--> 319 return np.array(self.to_numpy()._value)

ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (4,) + inhomogeneous part.

---

The notebook changes in question are:

groups = nodesdf.groupby(by='label')
tracks= []
for group in groups.groups:
    track= groups.get_group(group)
    track['solution'] = np.mean(np.sqrt(track['centroid-0']*track['centroid-0'] + track['centroid-1']*track['centroid-1']))
    tracks.append(track)
nodesdf = pd.concat(tracks).reset_index(drop=True)

_OUTPUT_TYPE = "nodes"
radius = 0.2

variables = dt.DummyFeature(
    radius=radius,
    output_type=_OUTPUT_TYPE,
    nofframes=3, # time window to associate nodes (in frames) 
)

model = dt.models.gnns.MAGIK(
    dense_layer_dimensions=(64, 96,),      # number of features in each dense encoder layer
    base_layer_dimensions=(96, 96, 96),    # Latent dimension throughout the message passing layers
    number_of_node_features=2,             # Number of node features in the graphs
    number_of_edge_features=1,             # Number of edge features in the graphs
    number_of_edge_outputs=0,              # Number of predicted features
    number_of_node_outputs=1,              # Number of predicted features
    edge_output_activation="sigmoid",      # Activation function for the output layer
    output_type=_OUTPUT_TYPE,              # Output type. Either "edges", "nodes", or "graph"
)


model.compile(
    optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),
    loss = 'mse',
    metrics=['accuracy'],
)

model.summary()

generator = GraphGenerator(
        nodesdf=nodesdf,
        properties=["centroid"],
        min_data_size=511,
        max_data_size=512,
        batch_size=8,
        **variables.properties()
)
    
with generator:
    model.fit(generator, epochs=10)

I would very much appreciate any guidance for how to handle this issue. Thank you!

[JesusPinedaC]

Hello @akilgall. Have you tried the solution you suggested in your previous issue? It seems to be a problem with Numpy!

I will have a look at it.

[akilgall]

Hello @JesusPinedaC ,
Yes, I have reverted to that previous numpy version and the issue was resolved. I also see updates to the framework which account for the numpy discrepancy - thank you for this!

Are there plans to include an example jupyter notebook? I have attempted to train a model to predict diffusion coefficients from simulation, however the model fails to learn the problem very well.

Cheers!

[JesusPinedaC]

Hello! We are currently in the process of restructuring the examples. Next week I will upload examples for both node and global estimation. Stay tuned!

One possible cause of instability in training for diffusion estimation is the output's numerical range. What range of diffusion coefficients are you attempting to estimate? Here, scaling the outputs plays an important role, but the problem is fully solvable!

[akilgall]

Thank you, I'm looking forward to seeing your examples!

I have attempted to get the model to learn the diffusion coefficient, the anomalous diffusion coefficient, and log10 of the diffusion coefficient. I haven't spent a lot of time validating this, but it appears that the model is learning the average diffusion coefficient of the 300 or so trajectories that are in the spatio-temporal slice (51.2 um x 51.2 um x 40 frames) of simulated data that I am training on with no applicability to a validation set that's a different slice of simulated data. However, since generating the graphs takes a very long time, I'm unable to feed the model more training data. Any recommendations would be very welcome!