Updating for new EVE expansions

Note: this document is meant for developers. If you have no interest in pyfa development, then this doesn't offer you much

Each time EVE is updated, pyfa may need to be updated too. This document covers only the most basic activities. Please remember that sometimes more work is needed to update pyfa properly.

Extract data from the client

Launch EVE client, make sure it is updated. Log in, open market window, tab with market tree to cache it. Use Phobos for actual data extraction:

$ python dumpToJson.py --eve="/.wine_eve/drive_c/Program Files/CCP/EVE/" --cache="/.wine_eve/drive_c/users/"$USER"/Local Settings/Application Data/CCP/EVE/c_program_files_ccp_eve_tranquility/" --json=~/Desktop/phobos_dump_tq --tables="dgmattribs|dgmeffects|dgmtypeattribs|dgmtypeeffects|dgmunits|icons|invcategories|invgroups|invmetagroups|invmetatypes|invtypes|marketProxy()_GetMarketGroups()"

For now, additional step is needed to extract traits data:

$ python dumpTraits.py --eve="/.wine_eve/drive_c/Program Files/CCP/EVE/" --cache="/.wine_eve/drive_c/users/"$USER"/Local Settings/Application Data/CCP/EVE/c_program_files_ccp_eve_tranquility/" --json=~/Desktop/phobos_dump_tq

Convert data to sqlite database

Using eos/utils/scripts/jsonToSql.py script, convert data into suitable to pyfa format:

$ python jsonToSql.py --json=~/Desktop/phobos_dump_tq/ --db=sqlite:////home/$USER/Desktop/eve.db

Find out what has been changed

Using eos/utils/scripts/itemDiff.py script, compare existing and new databases:

$ python3 itemDiff.py --old=../../../staticdata/eve.db --new=~/Desktop/eve.db > ~/Desktop/diff

If output file is empty, then pyfa update is not needed at all. Else, use output file as your guidance for updating pyfa.

Actual pyfa update

Database

Copy eve.db into staticdata folder of pyfa.

Effects

Go through diff file. For each item with added effects, you should verify already implemented effects or implement missing ones. Let's take Rubicon 1.3's Breacher change as example:

[*] Breacher
  [-|y] shipSmallMissileEMDmgMF2
  [-|y] shipSmallMissileExpDmgMF2
  [-|y] shipSmallMissileKinDmgMF2
  [-|y] shipSmallMissileThermDmgMF2
  [+|n] shipMissileRoFMF2
  [*] agility: 3.16 => 3.18
  [*] maxVelocity: 350.0 => 360.0
  [*] powerOutput: 35.0 => 37.0
  [*] shipBonusMF2: 5.0 => -5.0

Here damage bonus was replaced with rate of fire bonus. So, what exactly does new effect do? To find it out, you can use trait description, but translating words into filter is ambiguous. Thus, i prefer to use small script for new Eos:

if __name__ == "__main__":

    import argparse
    import json

    from eos import Eos
    from eos.const.eos import *
    from eos.data.data_handler import JsonDataHandler

    parser = argparse.ArgumentParser(description="Figure out what actually effect does")
    parser.add_argument("-e", "--effect", type=str, required=True, help="effect name")
    args = parser.parse_args()


    with open('/home/dfx/src/eve/eos/data_folder/phobos/dgmattribs.json', mode='r') as file:
        dgmattribs = json.load(file)

    with open('/home/dfx/src/eve/eos/data_folder/phobos/dgmeffects.json', mode='r') as file:
        dgmeffects = json.load(file)

    with open('/home/dfx/src/eve/eos/data_folder/phobos/invtypes.json', mode='r') as file:
        invtypes = json.load(file)

    with open('/home/dfx/src/eve/eos/data_folder/phobos/invgroups.json', mode='r') as file:
        invgroups = json.load(file)

    attr_id_name = {}
    attr_id_penalized = {}
    for row in dgmattribs:
        attr_id_name[row['attributeID']] = row['attributeName']
        attr_id_penalized[row['attributeID']] = 'not penalized' if row['stackable'] else 'penalized'

    effect_id_name = {}
    for row in dgmeffects:
        effect_id_name[row['effectID']] = row['effectName']
        if row['effectName'] == args.effect:
            effect_id = row['effectID']
            break

    type_id_name = {}
    for row in invtypes:
        type_id_name[row['typeID']] = row['typeName']

    group_id_name = {}
    for row in invgroups:
        group_id_name[row['groupID']] = row['groupName']

    dataHandler = JsonDataHandler('/home/dfx/src/eve/eos/data_folder/phobos/')
    engine = Eos(dataHandler, name='eos', storage_path='~/src/eve/eos/data_folder/', make_default=True)

    effect = engine._cache_handler.get_effect(effect_id)
    modifiers = effect.modifiers
    mod_counter = 1
    indent = '  '
    print('effect {}, build status is {}'.format(args.effect, EffectBuildStatus._get_name(effect.build_status)))
    for modifier in modifiers:
        print('{}Modifier {}:'.format(indent, mod_counter))
        print('{0}{0}state: {1}'.format(indent, State._get_name(modifier.state)))
        print('{0}{0}context: {1}'.format(indent, Context._get_name(modifier.context)))
        print('{0}{0}srcattr: {1}'.format(indent, attr_id_name[modifier.source_attribute_id]))
        print('{0}{0}operator: {1}'.format(indent, Operator._get_name(modifier.operator)))
        print('{0}{0}tgtattr: {1} ({2})'.format(
            indent,
            attr_id_name[modifier.target_attribute_id],
            attr_id_penalized[modifier.target_attribute_id])
        )
        print('{0}{0}location: {1}'.format(indent, Location._get_name(modifier.location)))
        print('{0}{0}filter type: {1}'.format(indent, FilterType._get_name(modifier.filter_type)))
        if modifier.filter_type is None or modifier.filter_type in (FilterType.all_, FilterType.skill_self):
            pass
        elif modifier.filter_type == FilterType.skill:
            print('{0}{0}filter value: {1}'.format(indent, type_id_name[modifier.filter_value]))
        elif modifier.filter_type == FilterType.group:
            print('{0}{0}filter value: {1}'.format(indent, group_id_name[modifier.filter_value]))
        mod_counter += 1

Just make full phobos data dump for it and fix paths, and you should be fine. Now, let's run it:

$ python3 getmods.py -e shipMissileRoFMF2
effect shipMissileRoFMF2, build status is ok_full
  Modifier 1:
    state: offline
    context: local
    srcattr: shipBonusMF2
    operator: post_percent
    tgtattr: speed (penalized)
    location: ship
    filter type: skill
    filter value: Missile Launcher Operation

It shows that this effect is completely passive (applied regardless of item state), it takes value of shipBonusMF2 attribute and boosts (multiplies by 1 + value/100) attribute speed of all items which have skill Missile Launcher Operation in its skill requirements (as root node of skill requirement tree). Target attribute is stacking penalizable, but here it doesn't make any difference, because attribute modifications coming from ship are immune to stacking penalty. Now, let's implement an effect:

$ cat eos/effects/shipmissilerofmf2.py
type = "passive"
def handler(fit, ship, context):
    level = fit.character.getSkill("Minmatar Frigate").level
    fit.modules.filteredItemBoost(lambda mod: mod.item.requiresSkill("Missile Launcher Operation"),
                                  "speed", ship.getModifiedItemAttr("shipBonusMF2") * level)

After you've finished writing new effects, remove effects which are no longer used. You can identify them using eos/utils/scripts/effectUsedBy.py script:

$ python3 effectUsedBy.py --database=../../../staticdata/eve.db

It also goes through effects' .py files and prints by which items this effect is used, which eases maintenance and testing.

Market overrides

Pyfa does not show vanity variations of ships. This time, few were added, e.g. Incursus Aliastra Edition. Open service/market.py and add it to ITEMS_FORCEPUBLISHED dictionary as key with False value. Pyfa won't see them this way. There're multiple other settings to control what's shown and where, but they are pretty much self-explanatory.