Assessor visibility: progress, logs, logic

[apatsekin]

Short summary about the issue/question: While running two different experiments and having over >12 successful trials in each, I haven't encountered any clues of assessor's work.

Brief what process you are following: I was running remote experiments with two worker machines (concurrency=2, 1 trial per 1 node). First experiment was using CurveFitting assessor: after > 10 completed trials it was still showing "not enough data for curvefitting". Second experiment was running Medianstop: 14 completed trials. No early stopping engaged. Can't pull up dispatcher.log, since I stopped/resumed experiment and the log was wiped out.

Questions/issues:

1. After stoping and resuming experiment I see wiped out dipsatcher.log, so I can't evaluate anymore any Assessor activity in the past.
2. After experiment was resumed does the Assessor pull up prior trials history for evaluation?
3. It would be great to add some indications in web UI about progress and current state of assessor. (i.e. for Medianstop what median is right now, how many trials were already stopped, how many completed trials are used for calculating params).

How to reproduce it:

nni Environment:
nni version: 1.0
nni mode(local|pai|remote): remote
OS: Ubuntu 16.04.6 LTS
python version: 3.6.8
is conda or virtualenv used?: no
is running in docker?: yes

need to update document(yes/no): no

Anything else we need to know:

[QuanluZhang]

@apatsekin thanks for your feedback and suggestions.

your question 1: yes, in the current version, logs will be wiped out after resume. We also think keeping the old log is better, so this will be supported in the next release.

your question 2: after resume, assessor's state is not resumed, only tuner's state is resumed by replaying the results of finished trials. Thanks, we will consider to support the recovery of assessor in future release.

your question 3: this is a good suggestion, we will make UX design for this requirement.

BTW, if you want to get more runtime information of assessor, its logs are stored in dispatcher.log. If you want to get more logs, the current walk around is adding more log info in the installed python code (i.e., curvefitting_assessor.py, medianstop_assessor.py).