Move materials into separate .json file

- Added MAT4 support to material_metadata.json as well
- Also, the export script no longer outputs container_info.json.

README.md

@@ -20,7 +20,7 @@ I am very thankful for TwnKey, uyjulian, Kyuuhachi, DarkStarSword, and the Kisek
 ## Usage:
 
 ### ys8_it3_export_assets.py
-Double click the python script and it will search the current folder for all .it3 files and export the meshes and textures into a folder with the same name as the it3 file.  Additionally, it will output a very incomplete JSON file with metadata, which I am using to understand this format further (there is no practical use just yet).
+Double click the python script and it will search the current folder for all .it3 files and export the meshes and textures into a folder with the same name as the it3 file.  Additionally, it will output a JSON file with all the material metadata, which is used at the time of repacking the it3 to rebuild the material data (RTY2 and MAT6 blocks).
 
 In regards to textures, for Ys 8 and 9 models (modern TEXI/TEX2 blocks) the script will output DDS files.  Older games, the script will output raw ITP files (Falcom format).  Please use [Cradle](https://github.com/Aureole-Suite/Cradle/releases/) by Kyuuhachi to convert the ITP files into useable PNG files.
 
@@ -46,18 +46,16 @@ The default behavior of the script is to output ITP files only if they cannot be
 Overwrite existing files without prompting.
 
 ### ys8_it3_import_assets.py
-Double click the python script and it will search the current folder for all .it3 files with exported folders, and import the meshes and textures in the folder back into the it3 file.  This script requires a working it3 file already be present as it does not reconstruct the entire file; only the known relevant sections.  The remaining parts of the file (the skeleton and any animation data, etc) are copied unaltered from the intact it3 file.  By default, it will apply c77 compression to the relevant blocks.
+Double click the python script and it will search the current folder for all .it3 files with exported folders, and import the meshes and textures in the folder back into the it3 file.  This script requires a working it3 file already be present as it does not reconstruct the entire file; only the known relevant sections.  The remaining parts of the file (the skeleton and any animation data, etc) are copied unaltered from the intact it3 file.  By default, it will apply c77 compression to the relevant blocks (or bz mode 2 if VPA9 blocks are detected).
 
-Meshes are attached to existing VPAX/VP11 blocks, and are found using the name of the block followed by an underscore (which matches the output from ys8_it3_export_assets.py).  For example, if the import script finds a VPAX block in the `c005_ps4_main2`, it will search for meshes with the following names: `meshes/c005_ps4_main2_*.vb`.  It will replace *all* the meshes associated with that node within the .it3 with *all* the meshes that it finds.  Therefore, if the .it3 has `c005_ps4_main2_00` through `c005_ps4_main2_02` for example, deleting `c005_ps4_main2_01.vb` will remove it from the VPAX, and a new mesh `c005_ps4_main2_03.vb` if found, will be added to the block.  A new BBOX and MAT6 (bounding box and material section, respectively) will replace the old block.
+Meshes are attached to existing VPA9/VPAX/VP11 blocks, and are found using the name of the block followed by an underscore (which matches the output from ys8_it3_export_assets.py).  For example, if the import script finds a VPAX block in the `c005_ps4_main2`, it will search for meshes with the following names: `meshes/c005_ps4_main2_*.vb`.  It will replace *all* the meshes associated with that node within the .it3 with *all* the meshes that it finds.  Therefore, if the .it3 has `c005_ps4_main2_00` through `c005_ps4_main2_02` for example, deleting `c005_ps4_main2_01.vb` will remove it from the VPAX, and a new mesh `c005_ps4_main2_03.vb` if found, will be added to the block.  A new BBOX and MAT6 (bounding box and material section, respectively) will replace the old block.
 
 If all the submeshes of a mesh node are deleted, then the script will insert a blank invisible mesh in its place so that the node remains intact.  This is required to prevent crashes, and also allow for mesh importation in the future (if the VPAX block is removed, the script will not know to look for meshes in the future under that node).
 
-*Every submesh* must have its own .material file with a valid material to be included into MAT6; submeshes without .material files will be ignored by the import script.  If two or more submeshes use identical submeshes, only one copy will be inserted into MAT6.  Note that each unique material needs a unique name; a second material with the same name will not be inserted after the first, even if settings within are different.  We do not know the values for most of the materials; however, the UV wrapping variables have been identified and are the 3rd and 4th texture flags (each texture has 4 flags underneath the texture name).  The values are 0: REPEAT, 1: CLAMP_TO_EDGE, 2: MIRRORED_REPEAT.
+*Every submesh* must have its own .material file with a valid material to be included into MAT6; submeshes without .material files will be ignored by the import script.  These files contain a *pointer* to material_metadata.json.  The actual materials (with shader parameters, textures, etc) are all in material_metadata.json.  Every .material file (and thus every submesh) *must* point to a valid entry inside material_metadata.json.  We do not know the values for most of the materials; however, the UV wrapping variables have been identified and are the 3rd and 4th texture flags (each texture has 4 flags underneath the texture name).  The values are 0: REPEAT, 1: CLAMP_TO_EDGE, 2: MIRRORED_REPEAT.
 
 If there is a .bonemap file (same format as .vgmap, should be the name of the node - so `c005_ps4_main2.bonemap`, not `c005_ps4_main2_00.bonemap`), then a new BON3 section will be written to change the bone palette of the *entire mesh node*.  (For example, if a new submesh `c005_ps4_main2_03` is added and requires a new bone map, then `c005_ps4_main2_00` and every other submesh *must* also use that new bone map).
 
-If there is a .rty2 file for the mesh node, that will be inserted into the .it3 file in place of the original (otherwise the original will be copied over).  This is mainly useful for changing material variants, which apply to the entire mesh; this is not a setting that can be applied on a submesh-level.
-
 It will make a backup of the original, then overwrite the original.  It will not overwrite backups; for example if "model.it3.bak" already exists, then it will write the backup to "model.it3.bak1", then to "model.it3.bak2", and so on.
 
 **Command line arguments:**

ys8_it3_export_assets.py

@@ -313,7 +313,7 @@ def parse_vpa78_block (f, block_type):
         except IndexError:
             print("Unable to convert local bone indices to mesh global, skipping...")
         section_info.append(mesh)
-        mesh_buffers.append({'fmt': fmt_struct, 'ib': ib, 'vb': vb})
+        mesh_buffers.append({'fmt': fmt_struct, 'ib': ib, 'vb': vb, 'material': mesh["header"]["material_id"]})
         pointer_i += mesh["header"]["num_indices"]*2
         pointer_v += mesh["header"]["num_vertices"]*40
     return(section_info, mesh_buffers)
@@ -386,7 +386,7 @@ def obtain_animation_data (f, it3_contents):
 
 def obtain_mesh_data (f, it3_contents, it3_filename, preserve_gl_order = False, trim_for_gpu = False):
     vpax_blocks = [i for i in range(len(it3_contents)) if it3_contents[i]['type'] in ['VPA7', 'VPA8', 'VPA9', 'VPAX', 'VP11']]
-    mat6_blocks = {it3_contents[i]['info_name']:it3_contents[i]['data'] for i in range(len(it3_contents)) if it3_contents[i]['type'] == 'MAT6'}
+    mat_blocks = {it3_contents[i]['info_name']:it3_contents[i]['data'] for i in range(len(it3_contents)) if it3_contents[i]['type'] in ['MAT4', 'MAT6']}
     meshes = []
     for i in range(len(vpax_blocks)):
         print("Processing mesh section {0}".format(it3_contents[vpax_blocks[i]]['info_name']))
@@ -398,10 +398,10 @@ def obtain_mesh_data (f, it3_contents, it3_filename, preserve_gl_order = False,
             it3_contents[vpax_blocks[i]]["data"], mesh_data = parse_vpax_block(f, it3_contents[vpax_blocks[i]]['type'], trim_for_gpu)
             # For some reason Ys VIII starts numbering at 1 (root is node 1, not node 0)
             node_list = [it3_filename[:-4]]
-            if it3_contents[vpax_blocks[i]]['info_name'] in mat6_blocks:
-                for j in range(len(mesh_data)):
-                    if mesh_data[j]['material'] < len(mat6_blocks[it3_contents[vpax_blocks[i]]['info_name']]):
-                        mesh_data[j]['material'] = mat6_blocks[it3_contents[vpax_blocks[i]]['info_name']][mesh_data[j]['material']]
+        if it3_contents[vpax_blocks[i]]['info_name'] in mat_blocks:
+            for j in range(len(mesh_data)):
+                if mesh_data[j]['material'] < len(mat_blocks[it3_contents[vpax_blocks[i]]['info_name']]):
+                    mesh_data[j]['material'] = mat_blocks[it3_contents[vpax_blocks[i]]['info_name']][mesh_data[j]['material']]
         if preserve_gl_order == False: # Swap triangles from OpenGL to D3D order
             for j in range(len(mesh_data)):
                 mesh_data[j]['ib'] = [[x[0],x[2],x[1]] for x in mesh_data[j]['ib']]
@@ -430,7 +430,7 @@ def write_fmt_ib_vb (mesh_buffer, filename, node_list = [], complete_maps = Fals
             f.write(json.dumps(vgmap_json, indent=4).encode("utf-8"))
     if 'material' in mesh_buffer and type(mesh_buffer['material']) == dict:
         with open(filename + '.material', 'wb') as f:
-            f.write(json.dumps(mesh_buffer['material'], indent=4).encode("utf-8"))
+            f.write(json.dumps({'material_name': mesh_buffer['material']['material_name']}, indent=4).encode("utf-8"))
     return
 
 # Currently assumes BC7 - This needs to be fixed
@@ -643,21 +643,30 @@ def process_it3 (it3_filename, complete_maps = complete_vgmaps_default, preserve
             it3_contents, textures = obtain_textures(f, it3_contents)
             if not os.path.exists(it3_filename[:-4]):
                 os.mkdir(it3_filename[:-4])
-        with open(it3_filename[:-4] + '/container_info.json', 'wb') as f:
-            f.write(json.dumps(it3_contents, indent=4).encode("utf-8"))
+        #with open(it3_filename[:-4] + '/container_info.json', 'wb') as f:
+            #f.write(json.dumps(it3_contents, indent=4).encode("utf-8"))
         if not os.path.exists(it3_filename[:-4] + '/meshes'):
             os.mkdir(it3_filename[:-4] + '/meshes')
+        material_json = {}
         for i in range(len(meshes)):
+            material_block = {}
+            rty2_block = {}
             for j in range(len(meshes[i]["meshes"])):
                 safe_filename = "".join([x if x not in "\/:*?<>|" else "_" for x in meshes[i]["name"]])
                 write_fmt_ib_vb(meshes[i]["meshes"][j], it3_filename[:-4] +\
                     '/meshes/{0}_{1:02d}'.format(safe_filename, j),\
                     node_list = meshes[i]["node_list"], complete_maps = complete_maps)
+                if "material" in meshes[i]["meshes"][j] and type(meshes[i]["meshes"][j]["material"]) == dict:
+                    material_block[meshes[i]["meshes"][j]["material"]["material_name"]] = \
+                        {key:meshes[i]["meshes"][j]["material"][key] for key \
+                        in meshes[i]["meshes"][j]["material"] if key != 'material_name'}
             rty2_blocks = [j for j in range(len(it3_contents)) if it3_contents[j]['type'] == 'RTY2'\
                 and it3_contents[j]['info_name'] == meshes[i]["name"]]
             if len(rty2_blocks) > 0:
-                with open(it3_filename[:-4] + '/meshes/{0}.rty2'.format(safe_filename), 'wb') as f2:
-                    f2.write(json.dumps(it3_contents[rty2_blocks[0]]['data'], indent = 4).encode('utf-8'))
+                rty2_block = it3_contents[rty2_blocks[0]]['data']
+            material_json[meshes[i]["name"]] = {'rty2_shader_assignment': rty2_block, 'material_parameters': material_block}
+        with open(it3_filename[:-4] + '/materials_metadata.json', 'wb') as f2:
+            f2.write(json.dumps(material_json, indent = 4).encode('utf-8'))
         print("Writing textures")
         use_alpha = {}
         for i in range(len(textures)):

ys8_it3_import_assets.py

@@ -241,7 +241,32 @@ def return_rty2_material(f, it3_section):
 def process_it3 (it3_filename, import_noskel = False):
     with open(it3_filename,"rb") as f:
         it3_contents = rapid_parse_it3 (f)
-        #Is there ever more than a single TEXI section?
+        # Will read data from JSON file, or load original data from the mdl file if JSON is missing
+        try:
+            material_struct = read_struct_from_json(it3_filename[:-4] + '/materials_metadata.json')
+        except:
+            material_struct = {}
+            print("{0}/materials_metadata.json missing or unreadable, reading data from {0}.it3 instead...".format(it3_filename[:-4]))
+            to_process_mat6 = [x for x in it3_contents if 'MAT6' in it3_contents[x]['contents']]
+            for section in to_process_mat6:
+                mat6 = []
+                rty2 = {}
+                f.seek(it3_contents[section]['offset'])
+                while f.tell() < it3_contents[section]['offset']+it3_contents[section]['length']:
+                    section_info = {}
+                    section_info["type"] = f.read(4).decode('ASCII')
+                    section_info["size"], = struct.unpack("<I",f.read(4))
+                    if (section_info["type"] == 'MAT6'):
+                        mat6 = parse_mat6_block(f)
+                    elif (section_info["type"] == 'RTY2'):
+                        rty2 = parse_rty2_block(f)
+                    else:
+                        f.seek(section_info["size"],1)
+                mats = {}
+                for i in range(len(mat6)):
+                    material_name = mat6[i].pop('material_name')
+                    mats[material_name] = mat6[i]
+                material_struct[section] = {'rty2_shader_assignment': rty2, 'material_parameters': mats}
         to_process_tex = [x for x in it3_contents if 'TEXI' in it3_contents[x]['contents']] #TEXF someday?
         to_process_vp = [x for x in it3_contents if any(y in it3_contents[x]['contents'] for y in ['VPA9','VPAX','VP11'])]
         if import_noskel == False:
@@ -272,7 +297,19 @@ def process_it3 (it3_filename, import_noskel = False):
                         ib = [[x[0],x[2],x[1]] for x in ib] # Swap DirectX triangles back to OpenGL
                         vb = read_vb(submeshfiles[j] + '.vb', fmt)
                         vgmap = read_struct_from_json(submeshfiles[j] + '.vgmap')
-                        material = read_struct_from_json(submeshfiles[j] + '.material')
+                        material_file = read_struct_from_json(submeshfiles[j] + '.material')
+                        material_name = material_file['material_name']
+                        try:
+                            material = material_struct[section]['material_parameters'][material_name]
+                            material['material_name'] = material_name
+                        except KeyError:
+                            print("KeyError: Attempted to add material {0}, but it does not exist in material_metadata.json!".format(material_name))
+                            if 'unk0' in material_file: # Backwards-compatibility with older metadata format
+                                print("Pre-v1.1.0 version metadata detected, importing...")
+                                material = material_file
+                            else:
+                                input("Press Enter to abort.")
+                                raise
                         submeshes.append({'fmt': fmt, 'ib': ib, 'vb': vb, 'vgmap': vgmap, 'material': material})
                     except FileNotFoundError:
                         print("Submesh {0} not found, skipping...".format(submeshfiles[j]))
@@ -293,9 +330,8 @@ def process_it3 (it3_filename, import_noskel = False):
                     bon3_block = create_bon3(bonemap, section, compression_type)
                     custom_bonemap = True
                 custom_rty2 = False
-                if os.path.exists(it3_filename[:-4] + '/meshes/{}.rty2'.format(section)):
-                    rty2_data = read_struct_from_json(it3_filename[:-4] + '/meshes/{}.rty2'.format(section))
-                    rty2_block = create_rty2(rty2_data)
+                if 'rty2_shader_assignment' in material_struct[section]:
+                    rty2_block = create_rty2(material_struct[section]['rty2_shader_assignment'])
                     custom_rty2 = True
                 while f.tell() < it3_contents[section]['offset']+it3_contents[section]['length']:
                     section_info = {}