-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathresearch.html
executable file
·669 lines (629 loc) · 49.8 KB
/
research.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
<!DOCTYPE html>
<head>
<!-- templatemo 419 black white -->
<!--
Black White
http://www.templatemo.com/preview/templatemo_419_black_white
-->
<title>Aishwarya Padmakumar</title>
<link href='images/UTLogo.png' rel='shortcut icon'>
<meta name="keywords" content="" />
<meta name="description" content="" />
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<link href="http://fonts.googleapis.com/css?family=Open+Sans:300,400,700" rel="stylesheet" type="text/css">
<link href="css/bootstrap.min.css" rel="stylesheet" type="text/css">
<link href="css/font-awesome.min.css" rel="stylesheet" type="text/css">
<link href="css/templatemo_style.css" rel="stylesheet" type="text/css">
<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.5.0/jquery.min.js" type="text/javascript"></script>
</head>
<script type="text/javascript">
$(function(){
$('#Abstract_Thesis').css('display','none');
$('#Abstract_Thesis_Link').click(function(){
$('#Abstract_Thesis').slideToggle('slow');
$(this).toggleClass('slideSign');
return false;
});
});
$(function(){
$('#Bibtex_Thesis').css('display','none');
$('#Bibtex_Thesis_Link').click(function(){
$('#Bibtex_Thesis').slideToggle('slow');
$(this).toggleClass('slideSign');
return false;
});
});
$(function(){
$('#Abstract_Arxiv_2020').css('display','none');
$('#Abstract_Arxiv_2020_Link').click(function(){
$('#Abstract_Arxiv_2020').slideToggle('slow');
$(this).toggleClass('slideSign');
return false;
});
});
$(function(){
$('#Bibtex_AAAI_2021').css('display','none');
$('#Bibtex_AAAI_2021_Link').click(function(){
$('#Bibtex_AAAI_2021').slideToggle('slow');
$(this).toggleClass('slideSign');
return false;
});
});
$(function(){
$('#Abstract_Sigdial_2020').css('display','none');
$('#Abstract_Sigdial_2020_Link').click(function(){
$('#Abstract_Sigdial_2020').slideToggle('slow');
$(this).toggleClass('slideSign');
return false;
});
});
$(function(){
$('#Bibtex_Sigdial_2020').css('display','none');
$('#Bibtex_Sigdial_2020_Link').click(function(){
$('#Bibtex_Sigdial_2020').slideToggle('slow');
$(this).toggleClass('slideSign');
return false;
});
});
$(function(){
$('#Abstract_JAIR_2020').css('display','none');
$('#Abstract_JAIR_2020_Link').click(function(){
$('#Abstract_JAIR_2020').slideToggle('slow');
$(this).toggleClass('slideSign');
return false;
});
});
$(function(){
$('#Bibtex_JAIR_2020').css('display','none');
$('#Bibtex_JAIR_2020_Link').click(function(){
$('#Bibtex_JAIR_2020').slideToggle('slow');
$(this).toggleClass('slideSign');
return false;
});
});
$(function(){
$('#Abstract_ICRA_2019').css('display','none');
$('#Abstract_ICRA_2019_Link').click(function(){
$('#Abstract_ICRA_2019').slideToggle('slow');
$(this).toggleClass('slideSign');
return false;
});
});
$(function(){
$('#Bibtex_ICRA_2019').css('display','none');
$('#Bibtex_ICRA_2019_Link').click(function(){
$('#Bibtex_ICRA_2019').slideToggle('slow');
$(this).toggleClass('slideSign');
return false;
});
});
$(function(){
$('#Abstract_Proposal').css('display','none');
$('#Abstract_Proposal_Link').click(function(){
$('#Abstract_Proposal').slideToggle('slow');
$(this).toggleClass('slideSign');
return false;
});
});
$(function(){
$('#Bibtex_Proposal').css('display','none');
$('#Bibtex_Proposal_Link').click(function(){
$('#Bibtex_Proposal').slideToggle('slow');
$(this).toggleClass('slideSign');
return false;
});
});
$(function(){
$('#Abstract_FSS_2018').css('display','none');
$('#Abstract_FSS_2018_Link').click(function(){
$('#Abstract_FSS_2018').slideToggle('slow');
$(this).toggleClass('slideSign');
return false;
});
});
$(function(){
$('#Bibtex_FSS_2018').css('display','none');
$('#Bibtex_FSS_2018_Link').click(function(){
$('#Bibtex_FSS_2018').slideToggle('slow');
$(this).toggleClass('slideSign');
return false;
});
});
$(function(){
$('#Abstract_EMNLP_2018').css('display','none');
$('#Abstract_EMNLP_2018_Link').click(function(){
$('#Abstract_EMNLP_2018').slideToggle('slow');
$(this).toggleClass('slideSign');
return false;
});
});
$(function(){
$('#Bibtex_EMNLP_2018').css('display','none');
$('#Bibtex_EMNLP_2018_Link').click(function(){
$('#Bibtex_EMNLP_2018').slideToggle('slow');
$(this).toggleClass('slideSign');
return false;
});
});
$(function(){
$('#Abstract_CoRL_2017').css('display','none');
$('#Abstract_CoRL_2017_Link').click(function(){
$('#Abstract_CoRL_2017').slideToggle('slow');
$(this).toggleClass('slideSign');
return false;
});
});
$(function(){
$('#Bibtex_CoRL_2017').css('display','none');
$('#Bibtex_CoRL_2017_Link').click(function(){
$('#Bibtex_CoRL_2017').slideToggle('slow');
$(this).toggleClass('slideSign');
return false;
});
});
$(function(){
$('#Abstract_EACL_2017').css('display','none');
$('#Abstract_EACL_2017_Link').click(function(){
$('#Abstract_EACL_2017').slideToggle('slow');
$(this).toggleClass('slideSign');
return false;
});
});
$(function(){
$('#Bibtex_EACL_2017').css('display','none');
$('#Bibtex_EACL_2017_Link').click(function(){
$('#Bibtex_EACL_2017').slideToggle('slow');
$(this).toggleClass('slideSign');
return false;
});
});
$(function(){
$('#Abstract_CICLing_2015').css('display','none');
$('#Abstract_CICLing_2015_Link').click(function(){
$('#Abstract_CICLing_2015').slideToggle('slow');
$(this).toggleClass('slideSign');
return false;
});
});
$(function(){
$('#Bibtex_CICLing_2015').css('display','none');
$('#Bibtex_CICLing_2015_Link').click(function(){
$('#Bibtex_CICLing_2015').slideToggle('slow');
$(this).toggleClass('slideSign');
return false;
});
});
$(function(){
$('#Undergraduate-Projects').css('display','none');
$('#Undergraduate-Projects-Header').click(function(){
$('#Undergraduate-Projects').slideToggle('slow');
$(this).toggleClass('slideSign');
return false;
});
});
$(function(){
$('#Graduate-Projects').css('display','none');
$('#Graduate-Projects-Header').click(function(){
$('#Graduate-Projects').slideToggle('slow');
$(this).toggleClass('slideSign');
return false;
});
});
$(function(){
$('#Senior-Thesis').css('display','none');
$('#Senior-Thesis-Header').click(function(){
$('#Senior-Thesis').slideToggle('slow');
$(this).toggleClass('slideSign');
return false;
});
});
$(function(){
$('#Undergraduate-Internships').css('display','none');
$('#Undergraduate-Internships-Header').click(function(){
$('#Undergraduate-Internships').slideToggle('slow');
$(this).toggleClass('slideSign');
return false;
});
});
</script>
<body>
<div class="templatemo-container">
<div class="col-lg-6 col-md-6 col-sm-6 col-xs-12 black-bg left-container">
<div class="tm-left-inner-container">
<ul class="nav nav-stacked templatemo-nav">
<li><a href="index.html" align="center">Homepage</a></li>
<li><a href="research.html" class="active" align="center">Research</a></li>
<li><a href="internships.html" align="center">Work Experience</a></li>
<li><a href="about.html" align="center">About Me</a></li>
</ul>
</div>
</div> <!-- left section -->
<div class="col-lg-6 col-md-6 col-sm-6 col-xs-12 white-bg right-container">
<div class="tm-right-inner-container" style="padding-top: 90px">
<div class="templatemo-subheading" style="margin-bottom:10px">Research</div>
<p> My PhD research was aimed at improving grounded language learning for robotics - that is, enabling robots to connect natural language with knowledge either stored as facts, or obtained via sensory perception, in order to meaningfully interact with humans in natural language.
The work is under the umbrella of the <a href="http://www.cs.utexas.edu/~larg/bwi_web/">Building-Wide Intelligence project</a> at UT Austin.
</p>
<!-- <div class="templatemo-subheading" style="font-size:130%;margin-bottom:10px">Preprints</div> --!>
<!-- ** START OF PUBLICATIONS **-->
<!-- *** -->
<div class="templatemo-subheading" style="font-size:130%;margin-bottom:10px">Refereed Conference and Journal Publications</div>
<!-- ** START OF PUBLICATIONS **-->
<div style="font-weight:bold;font-size:110%">Dialog Policy Learning for Joint Clarification and Active Learning Queries
</div>
<p> <i><b>Aishwarya Padmakumar</b> and Raymond J. Mooney.<br/>
To Appear in Proceedings of the 35th AAAI Conference on Artifical Intelligence (AAAI 2021).
</i><br/>
<a id="Abstract_Arxiv_2020_Link">[Abstract]</a>
<a href="https://arxiv.org/abs/2006.05456">[PDF]</a>
<a id="Bibtex_AAAI_2021_Link">[Bibtex]</a>
</p>
<div id="Abstract_Arxiv_2020">
Intelligent systems need to be able to recover from mistakes, resolve uncertainty, and adapt to novel concepts not seen during training. Dialog interaction can enable this by the use of clarifications for correction and resolving uncertainty, and active learning queries to learn new concepts encountered during operation. Prior work on dialog systems has either focused on exclusively learning how to perform clarification/ information seeking, or to perform active learning. In this work, we train a hierarchical dialog policy to jointly perform both clarification and active learning in the context of an interactive language-based image retrieval task motivated by an on-line shopping application, and demonstrate that jointly learning dialog policies for clarification and active learning is more effective than the use of static dialog policies for one or both of these functions.
<br/>
</div>
<div id="Bibtex_AAAI_2021">
@inproceedings{padmakumar:2020, <br/>
  title={Dialog Policy Learning for Joint Clarification and Active Learning Queries}, <br/>
  author={Aishwarya Padmakumar and Raymond J. Mooney}, <br/>
  booktitle={Proceedings of the 35th AAAI Conference on Artifical Intelligence (AAAI 2021)}
  year={2021} <br/>
}
</div>
<br/>
<div style="font-weight:bold;font-size:110%">Dialog as a Vehicle for Lifelong Learning
</div>
<p> <i><b>Aishwarya Padmakumar</b> and Raymond J. Mooney.<br/>
Position Paper Track at the SIGDIAL Special Session on Physically Situated Dialogue (RoboDial 2.0), July 2020.
</i><br/>
<a id="Abstract_Sigdial_2020_Link">[Abstract]</a>
<a href="https://arxiv.org/abs/2006.14767">[PDF]</a>
<a href="http://www.cs.utexas.edu/users/ml/slides/padmakumar.robodial20.slides.pdf">[Slides]</a>
<a href="https://www.youtube.com/watch?v=bApf5xbr3kw">[Talk Video]</a>
<a id="Bibtex_Sigdial_2020_Link">[Bibtex]</a>
</p>
<div id="Abstract_Sigdial_2020">
Dialog systems research has primarily been focused around two main types of applications – task-oriented dialog systems that learn to use clarification to aid in understanding a goal, and open-ended dialog systems that are expected to carry out unconstrained “chit chat” conversations. However, dialog interactions can also be used to obtain various types of knowledge that can be used to improve an underlying language understanding system, or other machine learning systems that the dialog acts over. In this position paper, we present the problem of designing dialog systems that enable lifelong learning as an important challenge problem, in particular for applications involving physically situated robots. We include examples of prior work in this direction, and discuss challenges that remain to be addressed.
<br/>
</div>
<div id="Bibtex_Sigdial_2020">
@inproceedings{padmakumar:robodial20, <br/>
  title={Dialog as a Vehicle for Lifelong Learning}, <br/>
  author={Aishwarya Padmakumar and Raymond J. Mooney}, <br/>
  booktitle={Position Paper Track at the SIGDIAL Special Session on Physically Situated Dialogue (RoboDial 2.0)}, <br/>
  month={July}, <br/>
  url="https://arxiv.org/abs/2006.14767", <br/>
  year={2020} <br/>
}
</div>
<br/>
<!-- *** -->
<div style="font-weight:bold;font-size:110%">Jointly Improving Parsing and Perception for Natural Language Commands through Human-Robot Dialog
</div>
<p> <i>Jesse Thomason, <b>Aishwarya Padmakumar</b>, Jivko Sinapov, Nick Walker, Yuqian Jiang, Harel Yedidsion, Justin Hart, Peter Stone, and Raymond J. Mooney.<br/>
To Appear in The Journal of Artificial Intelligence Research (JAIR), Vol. 67 (2020).
</i><br/>
<a id="Abstract_JAIR_2020_Link">[Abstract]</a>
<a href="http://www.cs.utexas.edu/users/ml/papers/thomason.jair19.pdf">[PDF]</a>
<a id="Bibtex_JAIR_2020_Link">[Bibtex]</a>
</p>
<div id="Abstract_JAIR_2020">
Humans use natural language to articulate their thoughts and intentions to other people, making it a natural channel for human-robot communication. Natural language understanding in robots needs to be robust to a wide-range of both human speakers and environments. In this work, we present methods for parsing natural language to underlying meanings and using robotic sensors to create multi-modal models of perceptual concepts. Through dialog, robots should learn new language constructions and perceptual concepts as they are used in context. We develop an agent for jointly improving parsing and perception in simulation through human-robot dialog, and demonstrate this agent on a robotic platform. Dialog clarification questions are used both to understand commands and to generate additional parsing training data. The agent improves its perceptual concept models through questions about how words relate to objects. We evaluate this agent on Amazon Mechanical Turk. After training on induced data from conversations, the agent can reduce the number of clarification questions asked while receiving higher usability ratings. Additionally, we demonstrate the agent on a robotic platform, where it learns new concepts on the fly while completing a real-world task.
<br/>
</div>
<div id="Bibtex_JAIR_2020">
@inproceedings{thomason:jair20, <br/>
  title={Jointly Improving Parsing and Perception for Natural Language Commands through Human-Robot Dialog}, <br/>
  author={Jesse Thomason and Aishwarya Padmakumar and Jivko Sinapov and Nick Walker and Yuqian Jiang and Harel Yedidsion and Justin Hart and Peter Stone and Raymond J. Mooney}, <br/>
  booktitle={The Journal of Artificial Intelligence Research (JAIR)}, <br/>
  month={January}, <br/>
  volume={67}, <br/>
  year={2020} <br/>
}
</div>
<br/>
<!-- *** -->
<div style="font-weight:bold;font-size:110%">Improving Grounded Natural Language Understanding through Human-Robot Dialog
</div>
<p> <i>Jesse Thomason, <b>Aishwarya Padmakumar</b>, Jivko Sinapov, Nick Walker, Yuqian Jiang, Harel Yedidsion, Justin Hart, Peter Stone, and Raymond J. Mooney.<br/>
International Conference on Robotics and Automation (ICRA), 2019. <br/>
Also presented at the SIGDIAL Special Session on Physically Situated Dialogue (RoboDIAL), 2018. <br/>
Also presented at the RSS Workshop on Models and Representations for Natural Human-Robot Communication (MRHRC), 2018.
</i><br/>
<a id="Abstract_ICRA_2019_Link">[Abstract]</a>
<a href="https://arxiv.org/abs/1903.00122">[PDF]</a>
<a id="Bibtex_ICRA_2019_Link">[Bibtex]</a>
<a href="http://www.cs.utexas.edu/users/ml/papers/thomason.robodial18.pdf">[RoboDial PDF]</a>
<a href="https://www.superlectures.com/sigdial2018/jointly-improving-parsing-and-perception-for-natural-language-commands-through-human-robot-dialog">[RoboDial Video]</a>
<a href="http://www.cs.utexas.edu/users/ml/papers/thomason.mrhrc18.pdf">[MRHRC PDF]</a>
<a href="http://www.cs.utexas.edu/users/ml/posters/thomason.mrhrc18.poster.pdf">[MRHRC Poster]</a>
<a href="https://youtu.be/PbOfteZ_CJc">[MRHRC Demo]</a>
</p>
<div id="Abstract_ICRA_2019">
Natural language understanding for robotics can require substantial domain- and platform-specific engineering. For example, for mobile robots to pick-and-place objects in an environment to satisfy human commands, we can specify the language humans use to issue such commands, and connect concept words like red can to physical object properties. One way to alleviate this engineering for a new domain is to enable robots in human environments to adapt dynamically---continually learning new language constructions and perceptual concepts. In this work, we present an end-to-end pipeline for translating natural language commands to discrete robot actions, and use clarification dialogs to jointly improve language parsing and concept grounding. We train and evaluate this agent in a virtual setting on Amazon Mechanical Turk, and we transfer the learned agent to a physical robot platform to demonstrate it in the real world.
<br/>
</div>
<div id="Bibtex_ICRA_2019">
@inproceedings{thomason:icra19, <br/>
  title={Improving Grounded Natural Language Understanding through Human-Robot Dialog}, <br/>
  author={Jesse Thomason and Aishwarya Padmakumar and Jivko Sinapov and Nick Walker and Yuqian Jiang and Harel Yedidsion and Justin Hart and Peter Stone and Raymond J. Mooney}, <br/>
  booktitle={International Conference on Robotics and Automation (ICRA)},
  month={October}, <br/>
  url="https://arxiv.org/abs/1903.00122", <br/>
  year={2019} <br/>
}
</div>
<br/>
<!-- *** -->
<div style="font-weight:bold;font-size:110%">Learning a Policy for Opportunistic Active Learning
</div>
<p> <i>
<b>Aishwarya Padmakumar</b>, Peter Stone, and Raymond J. Mooney. <br/>
In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP-18), Brussels, Belgium, November 2018.
</i><br/>
<a id="Abstract_EMNLP_2018_Link">[Abstract]</a>
<a href="http://aclweb.org/anthology/D18-1165">[PDF]</a>
<a id="Bibtex_EMNLP_2018_Link">[Bibtex]</a>
<a href="https://github.com/aishwaryap/rl_for_oal">[Code]</a>
<!-- <a href="http://www.cs.utexas.edu/users/ml/slides/padmakumar.emnlp18.slides.pdf">[Slides]</a> <br/> -->
</p>
<div id="Abstract_EMNLP_2018">
Active learning identifies data points to label that are expected to be the most useful in improving a supervised model. Opportunistic active learning incorporates active learning into interactive tasks that constrain possible queries during interactions. Prior work has shown that opportunistic active learning can be used to improve grounding of natural language descriptions in an interactive object retrieval task. In this work, we use reinforcement learning for such an object retrieval task, to learn a policy that effectively trades off task completion with model improvement that would benefit future tasks.
<br/>
</div>
<div id="Bibtex_EMNLP_2018">
@inproceedings{padmakumar:emnlp18, <br/>
  title={Learning a Policy for Opportunistic Active Learning}, <br/>
  author={Aishwarya Padmakumar and Peter Stone and Raymond J. Mooney}, <br/>
  booktitle={In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP-18)}, <br/>
  month={November}, <br/>
  address={Brussels, Belgium}, <br/>
  year={2018} <br/>
}
</div>
<br/>
<!-- *** -->
<div style="font-weight:bold;font-size:110%">Opportunistic Active Learning for Grounding Natural Language Descriptions
</div>
<p> <i>
Jesse Thomason, <b>Aishwarya Padmakumar</b>, Jivko Sinapov, Justin Hart, Peter Stone, and Raymond J. Mooney. <br/>
In Proceedings of the 1st Annual Conference on Robot Learning (CoRL-17), Mountain View, California, November 2017.
</i><br/>
<a id="Abstract_CoRL_2017_Link">[Abstract]</a>
<a href="http://www.cs.utexas.edu/users/ml/papers/thomason.corl17.pdf">[PDF]</a>
<a id="Bibtex_CoRL_2017_Link">[Bibtex]</a>
<!-- a href="http://www.cs.utexas.edu/users/ml/slides/thomason.corl17.pdf">[Slides]</a> -->
<a href="http://www.cs.utexas.edu/users/ml/posters/thomason.corl17.poster.pdf">[Poster]</a>
<a href="https://www.youtube.com/watch?v=f-CnIF92_wo&feature=youtu.be">[Demo]</a>
<a href="https://github.com/thomason-jesse/perception_classifiers/tree/active_learning">[Code]</a>
<br/>
</p>
<div id="Abstract_CoRL_2017">
Active learning identifies data points from a pool of unlabeled examples whose labels, if made available, are most likely to improve the predictions of a supervised model. Most research on active learning assumes that an agent has access to the entire pool of unlabeled data and can ask for labels of any data points during an initial training phase. However, when incorporated in a larger task, an agent may only be able to query some subset of the unlabeled pool. An agent can also opportunistically query for labels that may be useful in the future, even if they are not immediately relevant. In this paper, we demonstrate that this type of opportunistic active learning can improve performance in grounding natural language descriptions of everyday objects---an important skill for home and office robots. We find, with a real robot in an object identification setting, that inquisitive behavior---asking users important questions about the meanings of words that may be off-topic for the current dialog---leads to identifying the correct object more often over time.
<br/>
</div>
<div id="Bibtex_CoRL_2017">
@article{thomason:corl17, <br/>
  title={Opportunistic Active Learning for Grounding Natural Language Descriptions}, <br/>
  author={Jesse Thomason and Aishwarya Padmakumar and Jivko Sinapov and Justin Hart and Peter Stone and Raymond J. Mooney}, <br/>
  booktitle={In Proceedings of the 1st Annual Conference on Robot Learning (CoRL-17)}, <br/>
  month={November}, <br/>
  editor={Sergey Levine and Vincent Vanhoucke and Ken Goldberg}, <br/>
  address={Mountain View, California}, <br/>
  publisher={PMLR}, <br/>
  pages={67--76}, <br/>
  pdf = {http://proceedings.mlr.press/v78/thomason17a/thomason17a.pdf}, <br/>
  url="http://proceedings.mlr.press/v78/thomason17a.html", <br/>
  year={2017} <br/>
}
</div>
<br/>
<!-- *** -->
<div style="font-weight:bold;font-size:110%">Integrated Learning of Dialog Strategies and Semantic Parsing
</div>
<p> <i>
<b>Aishwarya Padmakumar</b> and Jesse Thomason and Raymond J. Mooney <br/>
In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2017), pp. 547--557, Valencia, Spain, April 2017.
</i><br/>
<a id="Abstract_EACL_2017_Link">[Abstract]</a>
<a href="http://www.cs.utexas.edu/users/ml/papers/padmakumar.eacl17.pdf">[PDF]</a>
<a id="Bibtex_EACL_2017_Link">[Bibtex]</a>
<!--<a href="http://www.cs.utexas.edu/users/ml/slides/padmakumar.eacl17.slides.pdf">[Slides]</a> <br/> -->
</p>
<div id="Abstract_EACL_2017">
Natural language understanding and dialog management are two integral components of interactive dialog systems. Previous research has used machine learning techniques to individually optimize these components, with different forms of direct and indirect supervision. We present an approach to integrate the learning of both a dialog strategy using reinforcement learning, and a semantic parser for robust natural language understanding, using only natural dialog interaction for supervision. Experimental results on a simulated task of robot instruction demonstrate that joint learning of both components improves dialog performance over learning either of these components alone.
<br/>
</div>
<div id="Bibtex_EACL_2017">
@inproceedings{padmakumar:eacl17, <br/>
  title={Integrated Learning of Dialog Strategies and Semantic Parsing}, <br/>
  author={Aishwarya Padmakumar and Jesse Thomason and Raymond J. Mooney}, <br/>
  booktitle={Proceedings of the 15th Conference of the European Chapter of the Association <br/>
          for Computational Linguistics (EACL 2017)}, <br/>
  month={April}, <br/>
  address={Valencia, Spain}, <br/>
  pages={547--557}, <br/>
  url="http://www.cs.utexas.edu/users/ai-lab/pub-view.php?PubID=127615", <br/>
  year={2017} <br/>
}
</div>
<br/>
<!-- *** -->
<div style="font-weight:bold;font-size:110%">Automated Linguistic Personalization of Targeted Marketing Messages Mining User-Generated Text on
Social Media
</div>
<p> <i>
Rishiraj Saha Roy, <b>Aishwarya Padmakumar</b>, Guna Prasad Jeganathan and Ponnurangam Kumaraguru <br/>
In proceedings of the 16th International Conference on Intelligent Text Processing and Computational
Linguistics (CICLing 15) (Best Paper Award)
</i><br/>
<a id="Abstract_CICLing_2015_Link">[Abstract]</a>
<a href="https://link.springer.com/chapter/10.1007%2F978-3-319-18117-2_16">[PDF]</a>
<a id="Bibtex_CICLing_2015_Link">[Bibtex]</a> <br/>
</p>
<div id="Abstract_CICLing_2015">
Personalizing marketing messages for specific audience segments is vital for increasing user engagement with advertisements, but it becomes very resource-intensive when the marketer has to deal with multiple segments, products or campaigns. In this research, we take the first steps towards automating message personalization by algorithmically inserting adjectives and adverbs that have been found to evoke positive sentiment in specific audience segments, into basic versions of ad messages. First, we build language models representative of linguistic styles from user-generated textual content on social media for each segment. Next, we mine product-specific adjectives and adverbs from content associated with positive sentiment. Finally, we insert extracted words into the basic version using the language models to enrich the message for each target segment, after statistically checking in-context readability. Decreased cross-entropy values from the basic to the transformed messages show that we are able to approach the linguistic style of the target segments. Crowdsourced experiments verify that our personalized messages are almost indistinguishable from similar human compositions. Social network data processed for this research has been made publicly available for community use.
<br/>
</div>
<div id="Bibtex_CICLing_2015">
@inproceedings {roy:cicling15, <br/>
  title={Automated Linguistic Personalization of Targeted Marketing Messages Mining User-Generated Text <br/>
      on Social Media}, <br/>
  author={Rishiraj Saha Roy and Aishwarya Padmakumar and Guna Prasad Jeganathan and Ponnurangam <br/>
      Kumaraguru}, <br/>
  booktitle={Proceedings of the 16th Conference of International Conference on Intelligent Text Processing <br/>
      and Computational Linguistics (CICLing 15)}, <br/>
  month={April}, <br/>
  address={Cairo, Egypt}, <br/>
  url="https://link.springer.com/chapter/10.1007%2F978-3-319-18117-2_16", <br/>
  year={2015} <br/>
}
</div>
<br/>
<!-- End of conference pubs -->
<div class="templatemo-subheading" style="font-size:130%;margin-bottom:10px">Workshop Publications; Thesis Work</div>
<!-- Workshop pubs, Thesis Work -->
<div style="font-weight:bold;font-size:110%">Dialog as a Vehicle for Lifelong Learning of Grounded Language Understanding Systems
</div>
<p> <i><b>Aishwarya Padmakumar</b><br/>
PhD Thesis, Department of Computer Science, The University of Texas At Austin. October 2018.
</i><br/>
<a id="Abstract_Thesis_Link">[Abstract]</a>
<a href="http://www.cs.utexas.edu/users/ml/papers/padmakumar.thesis20.pdf">[PDF]</a>
<a href="http://www.cs.utexas.edu/users/ml/slides/padmakumar.thesis20.slides.pdf">[Slides]</a>
<a id="Bibtex_Thesis_Link">[Bibtex]</a>
</p>
<div id="Abstract_Thesis">
The ability to understand and communicate in natural language can make robots much more accessible for naive users. Environments such as homes and offices contain many objects that humans describe in diverse language referencing perceptual properties. Robots operating in such environments need to be able to understand such descriptions. Different types of dialog interactions with humans can help robots clarify their understanding to reduce mistakes, and also improve their language understanding models, or adapt them to the specific domain of operation. <br/>
We present completed work on jointly learning a dialog policy that enables a robot to clarify partially understood natural language commands, while simultaneously using the dialogs to improve the underlying semantic parser for future commands. We introduce the setting of opportunistic active learning - a framework for interactive tasks that use supervised models. This framework allows a robot to ask diverse, potentially off-topic queries across interactions, requiring the robot to trade-off between task completion and knowledge acquisition for future tasks. We also attempt to learn a dialog policy in this framework using reinforcement learning. <br/>
We propose a novel distributional model for perceptual grounding, based on learning a joint space for vector representations from multiple modalities. We also propose a method for identifying more informative clarification questions that can scale well to a larger space of objects, and wish to learn a dialog policy that would make use of such clarifications.
<br/>
</div>
<div id="Bibtex_Thesis">
@misc{padmakumar:thesis18, <br/>
  title={Dialog as a Vehicle for Lifelong Learning of Grounded Language Understanding Systems}, <br/>
  author={Aishwarya Padmakumar}, <br/>
  month={August}, <br/>
  url="http://www.cs.utexas.edu/users/ml/papers/padmakumar.thesis20.pdf", <br/>
  year={2020} <br/>
}
</div>
<br/>
<!-- *** -->
<div style="font-weight:bold;font-size:110%">Improved Models and Queries for Grounded Human-Robot Dialog
</div>
<p> <i><b>Aishwarya Padmakumar</b><br/>
PhD Proposal, Department of Computer Science, The University of Texas At Austin. October 2018.
</i><br/>
<a id="Abstract_Proposal_Link">[Abstract]</a>
<a href="http://www.cs.utexas.edu/users/ml/papers/padmakumar.proposal18.pdf">[PDF]</a>
<a href="http://www.cs.utexas.edu/users/ml/slides/padmakumar.proposal18.slides.pdf">[Slides]</a>
<a id="Bibtex_Proposal_Link">[Bibtex]</a>
</p>
<div id="Abstract_Proposal">
The ability to understand and communicate in natural language can make robots much more accessible for naive users. Environments such as homes and offices contain many objects that humans describe in diverse language referencing perceptual properties. Robots operating in such environments need to be able to understand such descriptions. Different types of dialog interactions with humans can help robots clarify their understanding to reduce mistakes, and also improve their language understanding models, or adapt them to the specific domain of operation. <br/>
We present completed work on jointly learning a dialog policy that enables a robot to clarify partially understood natural language commands, while simultaneously using the dialogs to improve the underlying semantic parser for future commands. We introduce the setting of opportunistic active learning - a framework for interactive tasks that use supervised models. This framework allows a robot to ask diverse, potentially off-topic queries across interactions, requiring the robot to trade-off between task completion and knowledge acquisition for future tasks. We also attempt to learn a dialog policy in this framework using reinforcement learning. <br/>
We propose a novel distributional model for perceptual grounding, based on learning a joint space for vector representations from multiple modalities. We also propose a method for identifying more informative clarification questions that can scale well to a larger space of objects, and wish to learn a dialog policy that would make use of such clarifications.
<br/>
</div>
<div id="Bibtex_Proposal">
@misc{padmakumar:proposal18, <br/>
  title={Improved Models and Queries for Grounded Human-Robot Dialog}, <br/>
  author={Aishwarya Padmakumar}, <br/>
  month={October}, <br/>
  url="http://www.cs.utexas.edu/users/ml/papers/padmakumar.proposal18.pdf", <br/>
  year={2018} <br/>
}
</div>
<br/>
<!-- *** -->
<div style="font-weight:bold;font-size:110%">Interaction and Autonomy in RoboCup@Home and Building-Wide Intelligence
</div>
<p> <i>
Justin Hart, Harel Yedidsion, Yuqian Jiang, Nick Walker, Rishi Shah, Jesse Thomason, <b>Aishwarya Padmakumar</b>, Rolando Fernandez, Jivko Sinapov, Raymond Mooney and Peter Stone. <br/>
In Artificial Intelligence (AI) for Human-Robot Interaction (HRI) symposium, AAAI Fall Symposium Series, Arlington, Virginia, October 2018.
</i><br/>
<a id="Abstract_FSS_2018_Link">[Abstract]</a>
<a href="http://www.cs.utexas.edu/users/ml/papers/hart.fss18.pdf">[PDF]</a>
<a id="Bibtex_FSS_2018_Link">[Bibtex]</a>
</p>
<div id="Abstract_FSS_2018">
Efforts are underway at UT Austin to build autonomous robot systems that address the challenges of long-term deployments in office environments and of the more prescribed domestic service tasks of the RoboCup@Home competition. We discuss the contrasts and synergies of these efforts, highlighting how our work to build a RoboCup@Home Domestic Standard Platform League entry led us to identify an integrated software architecture that could support both projects. Further, naturalistic deployments of our office robot platform as part of the Building-Wide Intelligence project have led us to identify and research new problems in a traditional laboratory setting.
<br/>
</div>
<div id="Bibtex_FSS_2018">
@inproceedings{hari:fss18, <br/>
  title={Interaction and Autonomy in RoboCup@Home and Building-Wide Intelligence}, <br/>
  author={Justin Hart and Harel Yedidsion and Yuqian Jiang and Nick Walker and Rishi Shah and Jesse Thomason and Aishwarya Padmakumar and Rolando Fernandez and Jivko Sinapov and Raymond Mooney and Peter Stone}, <br/>
  booktitle={Artificial Intelligence (AI) for Human-Robot Interaction (HRI) symposium, AAAI Fall Symposium Series}, <br/>
  month={October}, <br/>
  address={Arlington, Virginia}, <br/>
  url="http://www.cs.utexas.edu/users/ai-lab/pub-view.php?PubID=127724", <br/>
  year={2018} <br/>
}
</div>
<br/>
<!-- *** -->
<!-- End of current research -->
<div id="Graduate-Projects-Header" style="font-weight:bold;font-size:130%;margin-bottom:2px;margin-top:10px"><a href="#" style="color:#333">Graduate Academic Projects</a> <img src="images/arrow.png" width="5" height="7" /></div>
<div id="Graduate-Projects" style="display:none">
<br/>
<div style="font-weight:bold;font-size:110%">Face to Age <a href="ut/DLProject1.pdf">[Report]</a></div>
<p> As a part of the Deep Learning Seminar course, we had to train a deep neural network to predict the year in which a yearbook photograph was taken. We fine-tuned VGGNet to do this both using standard cross-entropy loss (classification loss) and a linear combination of cross-entropy and L1 loss. We also attempted to visualize pixels most relevant for classification and the network's view of each class.
</p>
<br/>
<div style="font-weight:bold;font-size:110%">Unsupervised Text Summarization Using Sentence Embeddings <a href="ut/NLPProject.pdf">[Report]</a></div>
<p> The huge improvements in the performance of various NLP tasks using word, and more recently, sentence embeddings prompts one to attack any problem that has not yet been shown to benefit using these methods. In our class project for NLP, we attempted to perform text summarization by clustering of sentence vectors. We experimented with different choices of embeddings and different techniques for selecting a cluster representative. We outperformed a simple baseline but could not match state-of-the-art performance.
</p>
<br/>
<div style="font-weight:bold;font-size:110%">Visual Question Answering Using Natural Language Object Retrieval and Saliency Cues <a href="ut/VRProject.pdf">[Report]</a></div>
<p> Visual Question Answering is the task where a machine is given an image, and a question in natural language based on it, and is expected to answer the question in natural language. A simple neural baseline for this task uses a CNN to encode the image, an LSTM to encode the question and uses weights over these vectors to perform a classification over the 1000 most common answers. We extended this baseline to take in two additional cues - one was a bounding box retrieved when the question is passed as a query to a <a href="http://ronghanghu.com/text_obj_retrieval/">natural language object retrieval</a> pipeline. The second was a region in the image that humans would typically find salient, and hence would be likely to ask questions about. We improved performance over the baseline but could not meet the state-of-the-art.
</p>
<br/>
<div style="font-weight:bold;font-size:110%">Modelling Cooking Tutorials Using HMMs <a href="ut/RLFDProject.pdf">[Report]</a></div>
<p> If we want robots to directly read a tutorial from the web and follow the instructions, it must be able to abstract away the nuances of language in each individual tutorial and store the instructions in a common format. We wished to discover whether it was possible learn such a format in an unsupervised manner given only tutorials corresponding to the same task. To this end, we attempted to use Hidden Markov Models to model cooking recipes. We experimented with different possible language models to act as observation probabilities but could not identify a good model.
</p>
</div>
<!-- End of grad projects -->
<div id="Senior-Thesis-Header" style="font-weight:bold;font-size:130%;margin-bottom=2px;margin-top:10px"><a href="#" style="color:#333">Senior Thesis</a> <img src="images/arrow.png" width="5" height="7" /></div>
<div id="Senior-Thesis" style="display:none">
<br/>
<div style="font-weight:bold;font-size:110%">Improving Aggregate Diversity in Recommender Systems <a href="iitm/BTPThesis.pdf">[Thesis]</a></div>
<p>Over the last few years, the focus of research in Recommender Systems has shifted from simply predicting ratings accurately to more holistic metrics that examine other aspects of the recommendation process. For my senior thesis under <a href="http://www.cse.iitm.ac.in/~ravi/">Dr. B. Ravindran</a>, I attempted to improve the performance of recommender systems on their aggregate diversity, which examines the overall number of items in the inventory that the system manages to recommend and the relative number of times different itms get recommended. I first examined this problem as a course project for my data mining course, when I developed a heuristic solution to the problem that improved aggregate diversity by exploiting additional availabel context information such as user demographics. In my senior thesis, I identified the deficiencies of metrics that would intuitively be chosen to measure the aggregate diversity of a recommender system, attempted to identify a metric that overcame these defects and develop algorithms to optimize the same.
</p>
</div>
<!-- End of BTP -->
<div id="Undergraduate-Projects-Header" style="font-weight:bold;font-size:130%;margin-bottom:2px;margin-top:10px"><a href="#" style="color:#333">Undergraduate Academic Projects</a> <img src="images/arrow.png" width="5" height="7" /></div>
<div id="Undergraduate-Projects" style="display:none">
<br/>
<div style="font-weight:bold;font-size:110%">Reinforcement Learning for Coreference Resolution <a href="iitm/RLProject.pdf">[Report]</a></div>
<p> For the project in our Reinforcement Learning course, we attempted to use Reinforcement Learning techniques to exploit structural information when performing coreference resolution, as opposed to modelling it as a classification task. The coreference chain identified so far can be seen as a partial clustering. Clustering evaluation mechanisms can then be used to generate a reward which could be used by a Reinforcement Learning algorithm to learn generalizable rules for coreference resolution and can be interpreted as a metric learning task. We compared the performance of a number of standrd RL algorithms and were able to improve precision on the task over the baseline.
</p>
<br/>
<div style="font-weight:bold;font-size:110%">Diversity in Text Summarization Using LSA <a href="iitm/NLPProject.pdf">[Report]</a></div>
<p>As a part of the Natural Language Processing course, we tried to improve upon techniques that make use of Latent Semantic Analysis for text summarization. This was done by attempting to use standard relevance scores, that determine the importance of a sentence to the summary of the document obtained using LSA, discounted by the similarity of the sentence in question to already selected sentences, much like what Maximal Marginal Relevance does. We also looked at different vector spaces in which sentences could be represented such as binary word vectors, TF.IDF. word vectors and term frequency word vectors weighted by unigram probabilities of words.
</p>
<br/>
<div style="font-weight:bold;font-size:110%">Identifying Points for Active Learning in an Induced Hypergraph <a href="iitm/SocialNetworkAnalysisProject.pdf">[Report]</a></div>
<p>This was my project for the social network analysis course. We were exploring how a graphical representation of a dataset could be exploited to identify useful points for active learning. In active learning, the system identifies difficult data points, which it requests a human to label. We created an induced hypergraph for a normal relational database by grouping points into hyperedges based on attribute values. We hypothesized that influencers in this structure would make good choices for training points as label propagation techniques could then be used to learn other labels from them and experimented with different techniques of identifying influencers and propagation.
</p>
<br/>
<div style="font-weight:bold;font-size:110%">Spell Check <a href="iitm/SpellCheck.pdf">[Report]</a></div>
<p>A different solution to a classic problem - we designed a spell checker that makes use of a modified version of Jaccard similarity of character bigrams of words, rather than edit distance to rank corrections. The system can also make use of additional context when surrounding words in the text are provided.
</p>
<br/>
<div style="font-weight:bold;font-size:110%">Wait-free Binary Search Tree <a href="iitm/ConcurrentProgrammingProject.pdf">[Report]</a></div>
<p>Wait-freedom is the strongest non-blocking guarantee of progress in a concurrent system. During our concurrent programming course, we tried to design a concurrent binary search tree that provides this guarantee. The main challenge in this was to identify all possible interleavings of operations from different threads that could result in a loss of correctness and incorporate a preventive for attaining it by a mechanism that did not involve waiting. We ended up with a design that, as far as we could make out, was theoretically wait-free except for a periodic cleanup operation, but spawned a very large number of threads to create a monitoring mechanism.
</p>
<br/>
</div>
<!-- End of undergrad projects -->
<footer style="clear:left">
<p class="col-lg-6 col-md-6 col-sm-12 col-xs-12 templatemo-copyright" style="font-size:80%">Design: <a href="http://www.templatemo.com">www.templatemo.com</a></p>
</footer>
</div>
</div> <!-- right section -->
</div>
</body>
</html>