Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FIX: refactor Network and DAG SOLVER to fix bad pruning #26

Open
wants to merge 30 commits into
base: master
Choose a base branch
from

Commits on Sep 28, 2019

  1. Configuration menu
    Copy the full SHA
    617e577 View commit details
    Browse the repository at this point in the history

Commits on Sep 29, 2019

  1. FIX(yahoo#13): BUG in plot-diagram writtin from PY2 era,

    were writing in text-mode in PY3. and failing as encoding error.
    ankostis committed Sep 29, 2019
    Configuration menu
    Copy the full SHA
    f58d148 View commit details
    Browse the repository at this point in the history

Commits on Oct 1, 2019

  1. enh(test): + x2 TC breaking UNSATISFIED operations...

    receiving partial inputs, needed for other operations.
    ankostis committed Oct 1, 2019
    Configuration menu
    Copy the full SHA
    52c0d77 View commit details
    Browse the repository at this point in the history
  2. ENH(net,yahoo#18): ignore UN-SATISFIABLE operations with partial inputs

    + The x2 TCs added just before are now passing.
    ankostis committed Oct 1, 2019
    Configuration menu
    Copy the full SHA
    bc4c221 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    b8daa07 View commit details
    Browse the repository at this point in the history
  4. ENH(core): ORDERED SETs for DETERMINISTIC results

    NOTE dict are not deterministic in <PY3.6.
    So this commit would not improve determinism
    in those pythons.
    
    + build: add `boltons` dependency for ndexedSet.
    + doc: mark all set usage if affect determinism.
    + e.g. see reproducibility problem in yahoo#14.
    ankostis committed Oct 1, 2019
    Configuration menu
    Copy the full SHA
    12bdfe4 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    b8377ca View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    489b32c View commit details
    Browse the repository at this point in the history

Commits on Oct 2, 2019

  1. Configuration menu
    Copy the full SHA
    b102d44 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    de02885 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    e1454fd View commit details
    Browse the repository at this point in the history
  4. MERGE (prune_unsatified, ordered_sets) into fix-pruning ...

    needed to refactor the new pruning algorithm.
    
    - expected 2 TCs fail for yet-to-be-solved yahoo#24 & yahoo#25 bugs.
    ankostis committed Oct 2, 2019
    Configuration menu
    Copy the full SHA
    3736738 View commit details
    Browse the repository at this point in the history
  5. DOC(net): explain DAG solution & compilation...

    the later described in yahoo#21.
    ankostis committed Oct 2, 2019
    Configuration menu
    Copy the full SHA
    c273068 View commit details
    Browse the repository at this point in the history
  6. TEST(prune): +Failing x2 TCs multi-out must run but not...

    override intermediate data.
    
    More changes only for newer pruning TCs:
    + refact(test): rename graph-->netop vars for results of compose(),
      to avoid of `graph.net.graph`.
    + Explain failure modes in v1.2.4 & this merged branch (yahoo#19 + yahoo#23).
    ankostis committed Oct 2, 2019
    Configuration menu
    Copy the full SHA
    16d42f1 View commit details
    Browse the repository at this point in the history
  7. refact(dag): call compile() before compute.compute...

    not after compose().
    
    + All TCs pass ok.
    + NOTE this is not yet what is described in yahoo#21.
    ankostis committed Oct 2, 2019
    Configuration menu
    Copy the full SHA
    b92f103 View commit details
    Browse the repository at this point in the history

Commits on Oct 3, 2019

  1. Configuration menu
    Copy the full SHA
    6d1884e View commit details
    Browse the repository at this point in the history
  2. ENH(net): move compile() after SOLVE DAG ...

    to pass +TC checking DeleteInst vary when inputs change.
    
    - x4 TCs still failing, and need revamp of dag-solution.
    ankostis committed Oct 3, 2019
    Configuration menu
    Copy the full SHA
    619cae7 View commit details
    Browse the repository at this point in the history
  3. REFACT(NET) COMPILE+COMPUTE...

    + Read the next doc-only commit to understand changes.
    + Renamed:
      + net.steps --> net.execution_plan.
      + (old)compile() --> _build_execution_plan()
      + _find_necessary_steps() --> (new)compile() + _solve_dag()
        compile() became the master function invoking _solve_dag &
        _build-execution_plan(), and do the caching.
    + refact(compute()): extract common tasks from sequential/parallel.
    + refact show_layers() to allow full-print, geting also string
      (not just printing), and using custom classes for representation.
    + Raise AssertionError when invalid class in plan.
      it's a logic error, not a language type-error.
    ankostis committed Oct 3, 2019
    Configuration menu
    Copy the full SHA
    eff351d View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    d959485 View commit details
    Browse the repository at this point in the history
  5. FIX(net): new Ops invalidate execution-plan cache...

    Probaly unreported bug in v1.2.4 for '_neccessary_steps_cache`.
    ankostis committed Oct 3, 2019
    Configuration menu
    Copy the full SHA
    17eb2fd View commit details
    Browse the repository at this point in the history
  6. ENH(DAG): NEW SOLVER

    + Pruning behaves correctly also when outputs given;
      this happens by breaking incoming provide-links
      to any given intermedediate inputs.
    + Unsatisfied detection now includes those without outputs
      due to broken links (above).
    + Remove some uneeded "glue" from unsatisfied-detection code,
      leftover from previous compile() refactoring.
    + Renamed satisfiable --> satisfied.
    + Improved unknown output requested raise-message.
    + x3 TCs PASS, x1 in yahoo#24 and the first x2 in yahoo#25.
    - 1x TCs in yahoo#25 still FAIL, and need "Pinning" of given-inputs
      (the operation MUST and MUST NOT run in these cases).
    ankostis committed Oct 3, 2019
    Configuration menu
    Copy the full SHA
    0830b7c View commit details
    Browse the repository at this point in the history

Commits on Oct 4, 2019

  1. WIP/FIX(prune,yahoo#26): PIN intermediate inputs if operation before …

    …must run
    
    - WIP: PARALLEL execution not adding PINS!
    + Insert "PinInstructions" in the execution-plan to avoid overwritting.
    + Add `_overwrite_collector` in `compose()` to collect re-calculated values.
    + FIX the last TC in yahoo#25.
    ankostis committed Oct 4, 2019
    Configuration menu
    Copy the full SHA
    0dc1293 View commit details
    Browse the repository at this point in the history
  2. REFACT(net): part 3 of new dag-solver & pin refactoring

    - STILL buggy PIN on PARALLEL, 2 DISABLED TCs FAIL:
      - test_pruning_with_given_intermediate_and_asked_out()
      - test_unsatisfied_operations_same_out()
    + move check if value in asked outputs before cache-evicting it
      in build-execution-plan time - compute methods
      don't need outputs anymore.
    + test: speed up parallel/multihtread TCs
      by reducing delays & repetitions.
    + refact: network rightfully adopted stray functions
      for parallel processing - they all worke on the net.graph,
    + upd: networkx api by indexing on `dag.nodes` views.
    + enh: add log message when deleting in parallel
      (in par with sequential code).
    + refact: var-renames, if-then-else simplifications, pythonisms.
    + doc: A lot!
    ankostis committed Oct 4, 2019
    Configuration menu
    Copy the full SHA
    06f6554 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    1cc733e View commit details
    Browse the repository at this point in the history

Commits on Oct 15, 2019

  1. Configuration menu
    Copy the full SHA
    5c3b8ef View commit details
    Browse the repository at this point in the history
  2. WIP/FIX(PIN): PARALLEL DELs decide on PRUNED-dag (not full)...

    - WIP: x4 TCs FAIL and still not discovered th bug :-(
    + BUT ALL+AUGMENTED PARALLEL TCs pass
      (yahoo#26 were failing some)
    + refact: net stores also `pruned_dag` (not only `steps`).
    + refact: _solve_dag() --> _prune_dag().
    + doc: +a lot.
    + TODO: store pruned_dag in own ExePlan class.
    ankostis committed Oct 15, 2019
    Configuration menu
    Copy the full SHA
    58977a4 View commit details
    Browse the repository at this point in the history
  3. FIX(NET): were FORGETTING PRUNED ASKED-OUTPUTs...

    ... bugged in the opening commit d403783 of this PR, and
    discovered 68(!) commits later, and all that time had to live
    with x4 broken TCs with asked-outputs.
    ankostis committed Oct 15, 2019
    Configuration menu
    Copy the full SHA
    27f222d View commit details
    Browse the repository at this point in the history
  4. FIX(<PY3.5): ORDERED DiGRAPH for old Python to fix TCs

    + Partial fix deterministic results (yahoo#22-2.4.3i).
    ankostis committed Oct 15, 2019
    Configuration menu
    Copy the full SHA
    7fe6080 View commit details
    Browse the repository at this point in the history
  5. FIX(DAG): broken_dag had PLAIN-STR instead of DataNode...

    bc subgraph was taken on plain string outputs.
    
    + minor upd err-msg terminology.
    ankostis committed Oct 15, 2019
    Configuration menu
    Copy the full SHA
    67aaadc View commit details
    Browse the repository at this point in the history

Commits on Oct 16, 2019

  1. FIX(net): bad refactor had broken PARALLEL EVICTION ...

    due to bad node check, evicting parallels it nevered kicked in.
    ankostis committed Oct 16, 2019
    Configuration menu
    Copy the full SHA
    fb1b074 View commit details
    Browse the repository at this point in the history