Skip to content
This repository has been archived by the owner on Oct 2, 2020. It is now read-only.

Shrinking STEP Files (4.4G total repo size -> 2.6G) #223

Closed
wants to merge 2 commits into from

Conversation

sethhillbrand
Copy link

@sethhillbrand sethhillbrand commented Jan 20, 2018

STEP Files are self-referential but the STEP Files in this commit
have many identical lines. This commit removes the identical lines
and replaces them with a reference to a single unique line.

Source for shrinker available at KiCad/kicad-packages3D-source#83


Thanks for creating a pull request to contribute to the KiCad libraries! To speed up integration of your PR, please check the following items as you complete them:

  • A screenshot of the model(s) aligned with KiCad footprints is very useful
  • Was the 3D model(s) created in a mechanical modelling program?
  • Source files (or scripts) have been contributed
  • Step model submitted in 1:1 scale
  • WRL model submitted in 1:2.54 scale
  • 3D model alignment checked against KiCad footprint
  • WRL model uses correct material properties
  • Geometry check performed against mechanical models

@CLAassistant
Copy link

CLAassistant commented Jan 20, 2018

CLA assistant check
All committers have signed the CLA.

@Shackmeister
Copy link
Collaborator

@easyw can you please comment on this? :)

@easyw
Copy link
Contributor

easyw commented Jan 21, 2018

@sethhillbrand @Shackmeister
this seems a tricky solution to shrink the Step library.
At the same time I'm not sure if this wouldn't affect STEP parser compatibility with different and commercial MCAD sw. STEP parsing compatibility is still a big issue in MCAD world.
https://www.designsociety.org/publication/19812/interoperability_issues_among_cad_systems_a_benchmarking_study_of_7_commercial_mcad_software
https://www.designsociety.org/download-publication/19812/interoperability_issues_among_cad_systems_a_benchmarking_study_of_7_commercial_mcad_software

The shrink is about 40%, so the gain is good but not a big difference.
Moreover when a STEP file for the board and parts is generated, it will grown again at its previous size.

Zipping the library as it is done here:
https://kicad.github.io/packages3d/
is giving a much higher shrinking factor.

Anyway the shrinking method is intriguing :)
It would be nice to have it ported to python, so it could be used/tested without the need to compile the source.

EDIT
I tried some test and I found cases in which the shrinker fails, so I wouldn't suggest this option for the step library.

@sethhillbrand
Copy link
Author

sethhillbrand commented Jan 21, 2018

@easyw Which model failed?

I ask because I ran all files through NIST's STEP file analyzer and didn't find any invalid files. I also spot checked a number of files visually using freecad.

@Shackmeister
Copy link
Collaborator

@easyw if this is a good way to shrink the files, it could probably be embedded in stepup and the CQ scripts? :)

@sethhillbrand I believe the question is if all step parsers can handle it :)

@easyw
Copy link
Contributor

easyw commented Jan 21, 2018

@sethhillbrand a simple hierarchy step file will fail... the shrinker is flattering the structure and in some case just mess the structure up...
hstep-test-shrinker-bad-result.zip
The library has most model as a single part, but not all...
And moreover, as @Shackmeister pointed out, the question is if all step parsers can handle it...

@sethhillbrand
Copy link
Author

sethhillbrand commented Jan 21, 2018

@easyw Thanks for the example. I had only checked the files that exist in the packages3D repository, so I missed the possibility where there is a "=>". I pushed an update to the code that handles this.

-- Edit --
Oops! Sorry, I just found a couple models with structure as you mentioned. I'm re-running the parser on them now. I'll squash and re-push

@easyw
Copy link
Contributor

easyw commented Jan 21, 2018

@sethhillbrand please if you're going to change the shrinker post the update, so I can test it.

STEP Files are self-referential but the STEP Files in this commit
have many identical lines.  This commit removes the identical lines
and replaces them with a reference to a single unique line.
@sethhillbrand
Copy link
Author

@easyw - I posted the shrinker in KiCad/kicad-packages3D-source#83 Let me know if you have concerns and I'll address them.

@easyw
Copy link
Contributor

easyw commented Jan 21, 2018

@sethhillbrand
I tested the new shrinker and I found issues on complex step models ... then I wouldn't suggest to use this tricky solution... The step format is very complex and we are building the kicad library based on an Open Cascade library, which is an open source and commercial library adopted widely... the size is not the main target for a mechanical library.
Here a sample of an original step assembly and the shrinked one:
shrinker-comparison

@sethhillbrand
Copy link
Author

I'm happy to fix this issue if the tool needs to support shrinking full board exports. However, my goal with this PR was to minimize the size of the included libraries for KiCAD while maintaining the same geometry.

Definitely don't want to step on toes here so I won't push if it's not something you're interested in.

@easyw
Copy link
Contributor

easyw commented Jan 21, 2018

having found some issue in the shrink-ed models, I would not trust this method unless after having checked every generated model through most used MCAD sw STEP parser...

@sethhillbrand
Copy link
Author

Is the most used MCAD sw STEP parser FreeCAD? Or would you suggest running the checks with a different software?

@easyw
Copy link
Contributor

easyw commented Jan 22, 2018

Step library must be compatible with the wider MCAD sw available. OCC step format has been proven to be compatible to most sw MCAD (SW, NX, PTC, Catia, Inventor, SE, SpaceClaim, Fusion360, Onshape, etc).
These are the sw that users may have when dealing with kicad.

@sethhillbrand
Copy link
Author

Thanks. I was asking about your statement:

I would not trust this method unless after having checked every generated model through most used MCAD sw STEP parser

Are you saying that the only way to accept the PR is to test every model in every piece of software? That seems a rather large request. Did I misunderstand what you were saying?

@easyw
Copy link
Contributor

easyw commented Jan 22, 2018

The point is that OCC are a library tools widely used and deeply tested... for that reason the STEP exporting result can be read without issues on the other MCAD (commercial or not).
OCC has a strong reputation and its libraries are also licensed as commercial product. In fact Altium is using OCC kernel for its 3d modeling importing and exporting.
All KiCad 3D libraries are benefiting of this wide deploying and have been tested by a very large amount of users since my first SMD 3D libraries has been released on July 2015.
Till now we haven't had almost issue on STEP model libraries (I only remember a recent case regarding a Led with PT Creo sw, but the model had a geometry issue that wasn't correctly tested).
So I really consider that a so huge modification to a library that just is behaving so good should require a very strong effort in testing the compatibility with all these MCAD sw.
As I already pointed out, STEP compatibility is not so easy to reach... many times a model could just seems fine, but may have geometry issue or even malformed structure.
For that reason we made the 3D library requirements quite hard to meet.
If you would like to optimize the STEP format, I would suggest you to consider to contribute to FreeCAD code and particularly with the OCC source code. The FC developers are friendly and its forum has many resources to offer....

By splitting the PinHeaders based on the materials, we can save an
additional 10% file size.
@Shackmeister
Copy link
Collaborator

hi @sethhillbrand as this approach does not seem completely "stable" and tested I do not think it can be used for now. eventhough I find it very interesting. should we still keep the PR open or should we perhaps move it over to an issue for further discussion?

@sethhillbrand
Copy link
Author

If there's a path to showing its stability, I'm happy to do the work to show that. My takeaway from @easyw was that there is not a reasonable path.

If that's wrong, let me know and I'll happily take on what needs to be done.

@easyw
Copy link
Contributor

easyw commented Feb 13, 2018

Hi @sethhillbrand
I'm sorry but, IMO, the most important feature of KiCAD MCAD library is the reliability.
File size is not a big issue and moreover the 3D libraries have been reorganized to allow selective downloading.
https://kicad.github.io/packages3d/

Unfortunately STEP format is very complex and each MCAD parses it in its own way...
You can see that even a text comment can generate issues on some commercial CAD...
easyw/kicad-3d-models-in-freecad#151

I would close this PR.

There is an option to have STEP file reduced been compliant to STEP ISO 10303-21 format... the stpZ extension which is a gzip-ed version of STEP text files...
https://www.cax-if.org/documents/rec_prac_file_compression_v12.pdf
Unfortunately this option is not viable because only CATIA, NX and FreeCAD are able to manage this extension, and that would limit the use of the libraries to many users.

BR
Maurice

@sethhillbrand
Copy link
Author

Right, that's kind of my point. When there's an issue like easyw/kicad-3d-models-in-freecad#151 , you open a bug report, fix the issue and regenerate the model.

There's nothing that prevents us from doing that here as well so if there are things that I can do to improve the confidence in the reliability of this code, I'm happy to do them.

As it stands right now, I updated the code to address both of the issues raised (although they don't affect the library parts, only complex assemblies). I haven't found additional issues and all of the models in the library render equivalently with and without shrinking.

@easyw
Copy link
Contributor

easyw commented Feb 13, 2018

@sethhillbrand
This is exactly opposite to my point of view...
I don't want to have any issue at all for users and moreover even less on a so massive refactoring of the libraries, which gives a so little advantage on a really marginal feature.
If you have followed the developers mailing list, and you have, Wayne already covered the 3D library size dimension argument and just dropped it.

OCC are a world wide standard with very high reputation and a huge users base and experience.
Altium is using them as a core for its internal 3D MCAD core.
@Shackmeister
I would strongly suggest not to tweak OCC standard format for even a greater advantage than the one in this thread.

BR
Maurice

@sethhillbrand
Copy link
Author

Well, I certainly don't want to have this included without full buy-in from the librarians, so I'll close this for now.

@myfreescalewebpage myfreescalewebpage added Abandoned Original author has stopped working on the PR Enhancement Improves existing package 3D in the library labels May 27, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Abandoned Original author has stopped working on the PR Enhancement Improves existing package 3D in the library
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants