-
Notifications
You must be signed in to change notification settings - Fork 0
/
index.html
1131 lines (749 loc) · 28 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8" />
<meta name="keywords" content="neural network, remark.js, slides" />
<meta name="description" content="Neural network presentation" />
<title>Neural network</title>
<style>
@import url(https://fonts.googleapis.com/css?family=Droid+Serif);
@import url(https://fonts.googleapis.com/css?family=Yanone+Kaffeesatz);
@import url(https://fonts.googleapis.com/css?family=Ubuntu+Mono:400,700,400italic);
body {
font-family: 'Droid Serif';
}
h1, h2, h3 {
font-family: 'Yanone Kaffeesatz';
font-weight: 400;
margin-bottom: 0;
opacity: 1.0;
}
.remark-slide-content h1 { font-size: 3em; }
.remark-slide-content h2 { font-size: 2em; }
.remark-slide-content h3 { font-size: 1.6em; }
.footnote {
position: absolute;
bottom: 3em;
}
li p { line-height: 1.25em; }
.red { color: #fa0000; }
.large { font-size: 2em; }
a, a > code {
color: rgb(249, 38, 114);
text-decoration: none;
}
code {
background: #e7e8e2;
border-radius: 5px;
}
.remark-code, .remark-inline-code { font-family: 'Ubuntu Mono'; }
.remark-code-line-highlighted { background-color: #373832; }
.pull-left {
float: left;
width: 47%;
}
.pull-right {
float: right;
width: 47%;
}
.pull-right ~ p {
clear: both;
}
#slideshow .slide .content code {
font-size: 0.8em;
}
#slideshow .slide .content pre code {
font-size: 0.9em;
padding: 15px;
}
.inverse {
background: #272822;
color: #777872;
text-shadow: 0 0 20px #333;
}
.inverse h1, .inverse h2 {
color: #f3f3f3;
line-height: 0.8em;
}
/* Slide-specific styling */
#slide-inverse .footnote {
bottom: 12px;
left: 20px;
}
#slide-how .slides {
font-size: 0.9em;
position: absolute;
top: 151px;
right: 140px;
}
#slide-how .slides h3 {
margin-top: 0.2em;
}
#slide-how .slides .first, #slide-how .slides .second {
padding: 1px 20px;
height: 90px;
width: 120px;
-moz-box-shadow: 0 0 10px #777;
-webkit-box-shadow: 0 0 10px #777;
box-shadow: 0 0 10px #777;
}
#slide-how .slides .first {
background: #fff;
position: absolute;
top: 20%;
left: 20%;
z-index: 1;
}
#slide-how .slides .second {
position: relative;
background: #fff;
z-index: 0;
}
/* Two-column layout */
.left-column {
color: #777;
width: 20%;
height: 92%;
float: left;
}
.left-column h2:last-of-type, .left-column h3:last-child {
color: #000;
}
.right-column {
width: 75%;
float: right;
padding-top: 1em;
}
/* Custom css classes for background images */
.neuralnet3 {
background-image: url(nnet7.jpg);
opacity:0.75;
}
.full-width {
background-size: auto 100%;
}
.full-height {
background-size: auto 100%;
}
.neuralnet4 {
background-image: url(cvbg.jpg);
opacity:0.75;
}
.neuralnet5 {
background-image: url(nlpbg.jpg);
opacity:0.75;
}
.h1 {
color: #f4fefe;
}
.neuralnet6 {
background-image: url(monalisa-gif.gif);
opacity:0.75;
}
</style>
</head>
<body>
<textarea id="source">
name: inverse
layout: true
class: center, middle, inverse
---
class: neuralnet3, full-width, full-height
# Neural Nets: from Perceptron to Deep Learning
[or mimicking brain]
.footnote[or skip it to [Tensorflow Playground](http://playground.tensorflow.org/)]
---
## What are deep artificial neural nets?
---
layout: false
.left-column[
## What is it?
]
.right-column[
Another bio-inspired, crazy-AI-maths unindentified object a.k.a:
- the *most popular ML algorithm* after linear models
- the reason for bi-weekly Breaking News from Google
- the stuff that *already* tags maps, photos, unlocks your IPhone, translate your texts, recognize your speech,...
- the stuff that *has been* reading bankchecks, detecting frauds, identifying threats...for more than 20 years
- the stuff that *will* help drive your car, diagnose a cancer, transform a panda, generate people...
<img src="panda.gif" style="width: 40%">
<img src="gen.gif" style="width: 40%">
]
---
.left-column[
## For real ?
]
.right-column[
Artificial Neural Networks are as old as Artificial Intelligence, initial goal was to create a **machine**
that would reason the same way as the humain brain, except neurons would be central computing units connected
by wires
- historical **hardware** implementation of ANN before transistors became code with transistors and high level languages
- with DL it has essentially become a piece of specific code running on GPU or TPU
- but AI chip dream still around
<img src="mark1.png" style="width: 40%">
<img src="AIchip.png" style="width: 43%">
]
---
.left-column[
## Historically
]
.right-column[
ANN started in 1943, complex timeline
- Cybernetics era, followed by a 'crazy-expectations' era
- Massively funded era (defence, IT, finance) sounded by an SVM era
- **Deep Learning era** (since 2006) is actually the third wave, most succesfull period
<img src="Google_Scholar.png" width="600">
]
---
.left-column[
## Second wave
]
.right-column[
Starting in the 80's, essentially at AT&T, MIT, Microsoft and IBM
- backpropagation, Convolutionnal Neural Networks, Long Short Term Memory
- Hinton (Univ. Toronto), LeCun (Bell Labs), Bengio (Univ. Montreal)
- training was hard but already in production for inference
<img src="asamples.gif" width="400">
> " *At some point in the late 1990s, one of these systems was reading 10 to 20% of all the checks in the US.* " Y. LeCun (2014)
]
---
.left-column[
## Second wave
]
.right-column[
(Shallow) Neural networks already in common use with crafted features engineering
- *much workload* was concerned by this *feature engineering* and *data preprocessing*
- once good features have been found, Neural Nets was just another ML technique (vs. SVM)
- more dedicated solutions (CNN, LSTM) were *hard to train*
<img src="class.png" width="600">
]
---
.left-column[
## Third wave
]
.right-column[
Deep Neural networks learn the features and make inference easier
- no *more complex preprocessing* and automatically learns the right features
- much workload consists in training made possible with GPU and very large learning set
- inference pipeline simpler but requires *computing power*
<img src="dll.png" width="600">
]
---
# How do you train deep neural nets?
<img src="mlp2.jpg" width="600">
---
.left-column[
## Jargon
]
.right-column[
As any domain, Neural Nets has its own Domain Specific Language, in addition to traditional ML issues
Comics view .red[*]
<img src="google_comics.png" width="600">
.footnote[.red[*] source : https://codelabs.developers.google.com/]
]
---
.left-column[
## Maths
]
.right-column[
An ANN output is a linear combination of nonlinear outputs taken over linear combination of the inputs
- let's call the inputs `\(x=(x_1,\ldots,x_n)\)` and the weights `\(w\)` and the bias `\(b\)` of one neuron
- let's call `\(f\)` the **activation function** of one given neuron
- the output of one neuron is
$$ f(w_1 x_1 + \ldots + w_n x_n + b) $$
<img src="formula2.png" width="400">
- now, with m neurons, the final output is
$$ g(x) = \sum_{i=1}^m \alpha_i f^i(w_1^i x_1 + \ldots + w_n^i x_n + b^i) + b_0$$
]
---
.left-column[
## MLP
]
.right-column[
What we have just described is the celebrated Multi-Layer-Perceptron architecture
- ancestors of all modern architectures (fully connected part)
- created by Rosenblatt in 1963
- part of the feedforward class of neural nets (no loop back)
<img src="mlp.gif" width="600">
]
---
.left-column[
## Activation Functions
]
.right-column[
Activation function is the unit process function that every input transformation goes through. Initially, this function purpose was to mimick ON/OFF process of axons. Before deep learning, ANN practitioners favored differentiable activation function.
But since deep learning upraisal, this situation has changed and now it can be
- a smooth saturated function e.g. sigmoid, hyperbolic tangent
- piecewise linear (Rectified Linear Unit)
- anything (cosinus, identity)
<img src="modern_activation.png" width="600">
]
---
.left-column[
## Does it work?
]
.right-column[
Hornik's universal approximation theorem (a.k.a Cybenko's theorem)
<img src="hornik.png" width="600">
]
---
.left-column[
## Training
]
.right-column[
Once architecture (number of layers, number of neurons, activation functions) is set, need to get the weights:
- Find the optimal values of weights s.t the loss function `\(L(w)\)` is minimum over the training set of `\(N\)` examples where
$$ \min L(w)=\frac{1}{N}\sum_{i=1}^N(g(x_i)-y_i)^2 $$
- Unconstrained non convex minimization problem over `\(w\)` variables
- With many optimization variables
<img src="optimization.gif" width="600">
]
---
.left-column[
## Training
]
.right-column[
The only viable option for such a training is gradient descent
- iterative algorithm where you start from an initial guess of the values of the `\(w^0\)` and update them
$$ w^{i+1}=w^{i}-\alpha \nabla_w L(w^{i}) $$
- `\(\alpha\)` is known as the gradient step (in optimization) and the learning rate (in NN)
- quite simple...except you need `\(\nabla_w L(w^{i})\)` **backpropagation algorithm**
<img src="back.gif" width="350">
]
---
.left-column[
## Training
]
.right-column[
Training an MLP (and it is even more true for Deep Neural Nets) then requires
- scientific computing skills, computing power and **craft art**
- dedicated libraries essentially for distributed backpropagation and optimization (all are in C++ but sklearn and DL4J)
- solving an **optimization problem**
<img src="cnn.png" width="350">
]
---
.left-column[
## Training
]
.right-column[
In addition to traditional ML parameters (regularization), training an DNN requires setting the right hyperparameters for the
optimization
- optimization solvers: Stochastic Gradient Descent, Momentum, Adam, RMSProp, Nesterov...
- SGD is the most used solver, it is a simple Gradient Descent where you compute `\(\nabla_w L(w^{i})\)` over a very small subset of the training basis (called *Minibatch*) and hence the Stochastic gradient
<img src="opt1.gif" width="350">
]
---
.left-column[
## Training
]
.right-column[
Regularization is ensured through a ridge penalty term a.k.a *weight decay*, but
- *learning rate*: most important term, increases model capacity indirectly through its impact on optimization. More advanced strategies update
the learning rate (typically with TensorBoard)
- *number of neurons*: too many neurons leads to overfitting
<img src="learningrates.jpeg" width="350">
]
---
## What do they work for?
<img src="googlernncnn.png" width="500">
---
.left-column[
## Automatic Processing Tasks
]
.right-column[
Deep Learning upraisal was motivated by reaching human-like errors rate for traditional challenges in Robotics, IA... by taking advantage of large amount of data collected
by american and chinese tech giants almost for free. This include
- **Computer Vision** (pictures from Mobile Phones)
- **Natural Language Processing** (Text and Comments from the World Wide Web)
- **Speech** (excerpts of talks from Mobile Phones and Collaborative Platforms)
<img src="IAtasks.png" width="500">
]
---
class: neuralnet4, full-width, full-height
# Computer Vision
---
.left-column[
## Computer Vision - Tasks
]
.right-column[
In Image Processing (or Computer Vision typically for Robotics, Detection...), traditional challenges and tasks are
- image classification (threat or not ?)
- object detection (where is the threat in the picture ?)
- segmentation (background removal), semantic (tumor detection) or instance segmentation (count people)
- scene understanding, tracking, location, mapping (SLAM), 3d reconstruction... (defence, autonomous transportation)
<img src="cvtasks.png" width="500">
]
---
.left-column[
## Computer Vision - Traditional Methods
]
.right-column[
In Image Processing, decades of research created many different methods, like
- template matching (similarity with fixed pattern)
- low-level dedicated methods (Canny edge detection, watershed or CRF for image segmentation...)
- Descriptors extractions and Machine Learning (face detection on Mobile phone, Viola-Jones)
<img src="cvtrad.png" width="500">
]
---
.left-column[
## Computer Vision - New architectures for DNN
]
.right-column[
Convolutionnal Neural Networks (created by LeCun in 1989) first apply Convolution Layers on top of inputs (pixel values) to reduce the number
of weights
- achieve state-of-the-art performance for most CV challenges since 2012
- add to the MLP architecture layers of convolution adapted from traditional CV
- add newcomers such as *pooling* (a.k.a subsampling) and *dropout* to regularize and lower the number of parameters
<img src="cnn_2.png" width="500">
]
---
.left-column[
## Computer Vision - Convolution
]
.right-column[
Convolutionnal Layers specific to pictures
- Fully Connected layers would involve too many parameters for one picture (think
of a `\(1000 \mbox{px} \times 1000 \mbox{px}\)` picture, first layer for one neuron would involve `\(1,000,000\)` weights)
- Convolutions help reduce the number of parameters
- Weights of the convolution become trainable parameters
<img src="convol.gif" width="500">
]
---
.left-column[
## Computer Vision - Dropout and Pooling
]
.right-column[
Dropout
- Some connections between neurons and layers are switched off during training. Typical amount is 20 %
of neurons randomly selected during training. Dropout helps regularization
<img src="dropout.gif" width="300">
Pooling
- Subsampling helps reduce the number of parameters
<img src="pooling.gif" width="300">
]
---
.left-column[
## Computer Vision Architectures
]
.right-column[
Many sophisticated architectures
- Combines many layers ('deep') and in general several fully connected layers at the end
- Usually ReLu-like activation functions and TanH at the end, but the devil is in the sequel of pooling, batch normalization weights sharing...
- Dropout only for training ! Not inference
<img src="comp_archi.png" width="500">
<img src="resnet.png" width="500">
]
---
.left-column[
## Computer Vision Architectures
]
.right-column[
Main challenges and architectures focus on image classification, but other CV tasks require more complex architectures
- Typically, object detection outputs one or several bounding boxes containing most probable objects
- Traditional CV uses pyramid of images and uses a classifier many times over many different regions and outputs region with gighest scores
- Object recognition recent DL architectures rely on efficient subdivision and probability map to speed up object detection
<img src="yolo.png" width="450">
]
---
.left-column[
## Computer Vision Architectures
]
.right-column[
A difficult CV task is semantic segmentation (determines which class each pixel belongs to)
- Outputs a map (same size) of the original images
- Apply convolution and then deconvolution
- Labelling is tedious
<img src="cat_segmentation.png" width="500">
]
---
.left-column[
## Computer Vision: best practices
]
.right-column[
Use a classical architecture (unless you're Google or Facebook) or even better transfer learning
- Dataset: same aspect ratio, same resolution, same colormap
- Size: as many pictures as possibles, don't overlook Labelling
- Scaling: usually mean substraction (per pixel, per channel)
- Augment: train and test (crops, shift, rotation...)
- [DL's book](https://www.deeplearningbook.org/) Rule-Of-Thumb : need at list `\5,000\` examples per category and `\10,000,000\` examples to reach human error rate
- Many practical advices [here](https://jeffmacaluso.github.io/post/DeepLearningRulesOfThumb/)
<img src="faces.png" width="500">
]
---
class: neuralnet5, h1, full-width, full-height
# NLP
---
.left-column[
## Natural Language Processing Tasks
]
.right-column[
AI initial goal was to make machine able to understand and generate texts and speech to interact with humans (Turing test). Such a goal relies on easy to very hard tasks
- **Easy** Spelling correction, Part-of-Speech tagging...
- **Medium** Sentiment Analysis, Text Classification...
- **Hard** Summarization, Translation, Question Answering...
<img src="nlptasks.png" width="500">
]
---
.left-column[
## Natural Language Processing Issues
]
.right-column[
NLP is hard because
- Hard pre-processing (segmentation) and sequential by nature
- Sparsity of examples (many words not used very often, different sizes...)
- Idioms, Languages diversity (colloquial, formal...), Domain Specific Languages
- Word and sense human ambiguity, grammar parsing not enough... **the astronomer saw the star**
- Linguistics issues (rationalist vs. empiricist)
<img src="tay.jpg" width="500">
]
---
.left-column[
## Natural Language Processing Traditional Methods
]
.right-column[
NLP has a longer history than AI, many different methods
- Rule-based (think of `\(regexp\)`)
- Language Models based on n-grams (Markov models) and dictionary
- Graph-based methods (Viterbi algorithm)
- Manual feature extraction (dictionary, bag of words, tf-idf) and ML classifier
<img src="ngrams.png" width="500">
]
---
.left-column[
## Natural Language Processing Neural Networks Approach
]
.right-column[
Similar to CV, NLP has experienced some real advances with the help of Artificial Neural Networks
- Long Short Term Memory unit in Recurrent Neural Network for Language Models, allow to retain information
- Word Embeddings : dense representation of words, sentences... **word2vec** approach
- Sequence-to-Sequence learning and attention mechanisms (Neural Machine Translation, Question Answering, chatbot...)
- Transformers (BERT, T-NLG..) and deeper architectures (17 billions parameters)
<img src="word2vec.png" width="500">
]
---
.left-column[
## Natural Language Processing LSTM
]
.right-column[
Long Short Term Memory networks are part of Recurrent Neural Networks (created in 1982 by Hopfield)
- unlike standard Feedforward they process inputs sequentially
- they retain information (*contextual*) in a memory
- works well for Language Modelling, Sequence-to-Sequence models, translation...
<img src="rnn.gif" width="500">
]
---
class: neuralnet6, h1, full-width, full-height
---
.left-column[
## Generative Adversarial Networks (GANs)
]
.right-column[
Best idea in Machine Learning over 10 years (Yann LeCun)
- Generative VS Discriminative (Classification, Regression)
- Two neural networks : the Generator and the Discriminator that fights in a minimax game
- Quite hard training... recent methods take inspiration from [Optimal Transport](https://weave.eu/le-transport-optimal-un-couteau-suisse-pour-la-data-science/) theory (Wasserstein distance)
<img src="https://camo.githubusercontent.com/7443d2adadc104b885cac75a1894567c053c987f/687474703a2f2f7777772e6b646e7567676574732e636f6d2f77702d636f6e74656e742f75706c6f6164732f67656e657261746976652d616476657273617269616c2d6e6574776f726b2e706e67" width="500">
]
---
.left-column[
## Generative Adversarial Networks (GANs)
]
.right-column[
Applications
- Generate realistic images
- Augment resolution...
- Text-to-image, image-to-image ...
<img src="https://junyanz.github.io/CycleGAN/images/teaser.jpg" width="600">
]
---
.left-column[
## Autoencoders
]
.right-column[
Autoencoders are a deep extension of MLP to automatically find the best reduction and coding/decoding at once
- part of unsupervised learning
- conceptually, it is a nonlinear dimension reduction technique decodable (unlike t-SNE)
- often used to denoise signals, also for data synthesis
<img src="enco.png" width="500">
]
---
.left-column[
## Auto-encoders
]
.right-column[
Applications :
- find the concept of cat spawning Web images and videos
- help unlock your IPhone 10
- used as a compression for speech, image...
<img src="nn_cat.jpeg" width="500">
]
---
.left-column[
## Adversarial Attacks
]
.right-column[
Basic idea :
- Find with optimzation amount of pixels to add to an image to change prediction of CNN
- DNN are very sensititive to small changes of pixel values
- Funny but very worrying for security and industrial applications of CNN
<img src="patch.gif" width="500">
]
---
## Small focus on hardware
<img src="gpu.jpg" width="500">
---
.left-column[
## Training
]
.right-column[
In addition to learning rate and algorithmic improvements, modern Neural Nets work because
- more data (millions time more), better backprop algorithm (automatic differentiation)
- progress in the secret sauce (initialization of weights, dropout, activation function...)
- **clever use of GPU** (in lieue of Moore's law): allows to distribute many easy-computations (kind of vectorization) with NVIDIA setting the standards (with CUDA language)
<img src="dl.png" width="550">
]
---
.left-column[
## Training
]
.right-column[
In addition, chip war has already started with old and new players :
- NVIDIA (with CUDA) vs. AMD (with OpenCL)
- most DL libraries support only CUDA except Caffe (Apple) but would like to get rid of NVIDIA-dependency
- Microsoft/Intel (with FPGA) vs. Amazon/Xilinx/Baidu (with XPU)
- Google (Tensor Processing Unit)
- Cloud-based services
<img src="NVIDIA.png" width="450">
]
---
.left-column[
## Training
]
.right-column[
Big AI/fonders currently deploy strategic plans where training NN is different from infering NN :
- Microsoft massively deploys Field Programmable Gate Array with Intel Nervana for Cloud Deep Learning
- TPU/XPU (mix of CPU/GPU/FPGA) would go for inference !
<img src="chips.jpg" width="600">
]
---
.left-column[
## On the future of deep learning
]
.right-column[
Future, trends and challenges :
- Much ado about **Generative Adversarial Networks** and **Autoencoders**
- Since several years ago, DNN was successfully integrated in Reinforcement Learning (Deep Q-learning)
- Safety-critical ML issues (adversarial, uncertainty...), power-efficient version...
- Prove something about DNN (data coverage with TDA, formal methods for robustness evaluation...)
- Conciliate physics-based, expert knowledge reasoning and predictive power of Deep Learning **Bayesian Deep Learning**
]
---
.left-column[
## Some applications for aerospace engineering
]
.right-column[
Every field is experiencing a Deep Learning hype :
- Aerodynamics: automatic tuning of turbulence parameters, calibration, predict flow for new geometry...
<img src="flow.gif" width="180">
- Material design: constitution-to-properties prediction, material discovery
<img src="predictive.jpg" width="180">
- Inverse design: calibration, uncertainty propagation (probabilistic programming)
<img src="inverse.png" width="180">
]
---
## Frameworks, Tools and resources
<img src="tools.jpg" width="500">
---
.left-column[
## Deep Learning at home
]
.right-column[
### Technical solutions :
- **Python** : Theano (Montreal), Tensorflow (Google), Keras (Google), Caffe and Pytorch (Facebook), CNTK (Microsoft), MXNET (Amazon), Paddle (Baidu)
- **R** : Keras (Google)
- **Javascript** : deeplearning.js, ex : [Teachable machines](https://teachablemachine.withgoogle.com/)
]
---
.left-column[
## DL Frameworks
]
.right-column[
### Deep Learning frameworks :
- Allows to build complicated architecture neural networks
- They are all optimized for performance (low-level libraries written in C/C++ wrapped to be called by a high-level API)
- Community support (continuous development)
- GPU-acceleration and massive parallelizations
- **Automatic differentation**
<img src="dlframeworks.jpg" width="500">
]
---
.left-column[
## DL Frameworks