week 03.10 09.10

8h: installing Theano, CUDA, and dependencies
3h: looking for project management software. Redmine, gitLab,... conclusion: GitHub is easiest. work tracking will be done here, with periodic updates to the faculty's hour calculator Excel sheet.
5h: setting up wiki, git (first own server, then GitHub)
5h: WildML NN from scratch, setup and tutorial

Total: 21

WildML NN from scratch

# layers: guesstimate based on problem
# output neurons: from #output classes
activity function: tanh, sigmoid, ReLU. Tanh: derivatie= 1-tanh^2 -> calculate only once, reuse
output activ. function: softmax -> probabilities

error calculation: cross-entropy loss
finding minimum of error function: gradient descent
finding the gradient vectors (input to gradient descent algorithm): backpropagation

code:

X = coordinates of dots y = class of dots
calculate_loss: probs has dimension 200x2 (200 training data, 2 output neurons), C is # output classes, N is # training data. probs[range(num_examples)] gives the 2 outputs of the net for each the training examples; take the one that should be correct as that is the main source of error (second output should be a lot smaller) Error in code? -> I think it should be corect_logprobs = -y.dot(np.log(probs[range(num_examples), y])) # -y * log (y_hoedje)
in build_model: probs = ... -> axis = 1 b/c sum over all training examples; z2 = 200x2 db1 = .... -> axis=0 b/c dim training_size x hiddenLayer_size (200x2)

Others: