Collecting feedback on generalized UIDs in Godot 4.4

[akien-mga]

As we announced in UID changes coming to Godot 4.4, to be able to better handle refactoring of files within projects, both in the editor and externally, we generalized the UID system to all resources, including scenes and scripts.

That has a number of implications which are naturally tradeoffs, and we are aware some of the decisions made are unpopular with some users, notably the addition of separate .uid files for .gd, .gdshader, and a few other file types (custom resources notably).

We'd like to centralize user feedback in this discussion, which unlike PR or issue comments, has the advantage of supporting threaded replies. This makes it easier for users to make separate arguments and to discuss them in a focused manner.

Please read the blog post we published, notably the FAQ, which addresses a number of feedback points we already got before announcing this change. Of course you are welcome to bring up some of those FAQ items here too for further discussion.

We'll redirect discussions from other issues/PRs to this discussion, and evaluate that feedback while finalizing the 4.4 release.

Since this is a bit of a "hot" topic, I would like to remind everyone to be constructive and courteous, even if you disagree with points made by other users.

[scott-lin]

Can a project setting be added to disable this feature? I would assume turning UIDs on at a later time when the feature is actually needed by a project would be possible, otherwise, turning it on with 4.4 would break everyone in the first place.

What are the pros and cons of such a project setting?

[loteque]

To help newcomers to the discussion get up to speed

If you want to test the feature as implemented currently, it's present in Godot 4.4-dev5 and later dev snapshots here (current): https://godotengine.org/article/dev-snapshot-godot-4-4-dev-7/#downloads

alternate/complimentary issues and proposals:
#11223
#11571
#11565
#11464

a pull request that offered some reference implementation:
godotengine/godot#99090

the PR that implements the changes discussed in the article on 4.4-snapshots:
godotengine/godot#97352

separate but related/affected issues:
godotengine/godot#95637
godotengine/godot#86430
#10580
godotengine/godot#97916

[geowarin]

I think it's important for everyone to voice their concerns but it's equally important to say that some of us think the current implementation is a clear improvement.

uids are a pragmatic, relatively simple solution to a widespread problem with resources (they cannot be moved outside of the editor).

It's already the case that users must move .import files along with their associated files, so there's also the benefit of consistency.
Uids are relatively easy to explain to beginners.

There is value in a simple solution in term of maintenance. As everybody knows, godot is maintained mainly by volunteers so any over-engineered solution should raise some eyebrows.

Most people complain about the file system being cluttered but some users already have pointed out solutions to alleviate the problem in external editors:

• jetbrains file nesting rules
• vscode nested files
• nesting in the csproj for visual studio

[loteque]

I think the main issue is that it solves the problem for those who exclusively use external applications that support file nesting, but has recently created new problems for those who do not. So the users who do not use applications that support file nesting are left 'holding the bag' so to speak.

[daveTheOldCoder]

For newcomers: If you want to see the feature in action, it's present in Godot 4.4-dev5 and later dev snapshots.

[loteque]

The primary issue that I have with the current implementation and the other proposals is that it does not solve the fundamental issue, which is: external applications modifying the filesystem without updating a dependency graph, which ultimately Godot expects but is not well defined within Godot itself. see my comment here.

What I have gathered is these five proposals have been examined and discussed.
The 1-to-1 .uid file approach
The service approach
The monolithic .uid (or similar) file
Checksum tracking
uid as annotation

my view on each:

The 1-to-1 .uid file

• an elegant hack
• serviceable but a large pain point maintaining large codebase with many contributors
• doesn't provide a pathway to solve the root issue at some other level
• requires just as much end-user attention as the old way of doing things
• the current implementation has UX regressions IMO.

the service approach

• more 'out-of-the-way' of the end-users pipeline
• but... (O)n^2 operation every t time? (jaccard comparison)
• doesn't provide a pathway to solve the root issue

the monolithic uid file

• it just doesn't work to implement the intended effect (or at least not easily)

checksum tracking

• could work to notify the user if filepath is hashed
• it is what git does
• requires a service to be run to do the checksum (?)
• not sure how the would help Godot to automate updating filepaths
• could provide a simple path for external applications to send updates to the engine that a filepath has changed or to trigger a filepath reference update.
• I think this would require a well defined protocol that should be used internally by the engine and externally so that file path ref updates are only applied when a file is actually moved.

uid as annotation

• really most people don't like 'magic' in their source files

final thoughts

Ultimately I think that each proposal is missing the 'well defined' protocol for updating the source dependency graph. I think defining a proto will not only benefit plugin developers for external editors looking to integrate their editors with the godot engine but also be a solid foundation for needs internal to the engine. A source dependency graph exists in Godot and is a requirement of the engine whether it is defined or not, so why not define it and design its surface?

[SpockBauru]

the monolithic approach just doesn't work to implement the intended effect

Don't discard that right away! Will explain later.

really most people don't like 'magic' in their source files

Have to disagree with that. I prefer a way more a comment on my files with the "magic" number than thousands of uid files cluttering my repository. The only way to know what most people agree on would be some sort of poll.

checksum tracking

This is an interesting idea, it could checksum files that are not already annotated an centralizad file (the monolitic approach) and check if it matches with missing files. It also would checksum files modified in the editor, and store on the centralized file. The only way it wouldn't work was if the file was externally moved and changed at the same time.

Edit: Internally (in .godot folder) it could store the time each file was created, and when opening Godot only checksum the files with changed date, so it wouldn't checksum everything everytime.

[loteque]

Have to disagree with that. I prefer a way more a comment on my files with the "magic" number than thousands of uid files cluttering my repository. The only way to know what most people agree on would be some sort of poll.

you are correct. My statement is anecdotal, but I am drawing from my experience as a programmer, who has worked with large teams, for 20+ years.

I am interested in what you have on the monolithic approach! :)

[elenakrittik]

Another alternative that i think would be much better than breathing hopium that Godot manages to track and update dependencies correctly would be to let the user fix the dependencies manually via a pop-up. This would give maximum control and transparency that the current and likely most other solution(s) lack.

[JoNax97]

Without trying to argue for one solution in particular, I just want to add this to put the downsides of the current solution in perspective:

Millions of Unity users have been dealing with basically the same system (in the form of .meta files) for well over a decade and it's not been a big deal.

Most game devs nowadays know how to handle metadata files, know to commit them, move them along the source file, etc. Lots of tooling has been made around them (like git pre-commit checks to ensure the user doesn't forget a meta file), which can be adapted to work for Godot.

I have a feeling we're making a bit of a mountain over a molehill here.

Most arguments against this solution that I've read are rather cosmetic (filesystem pollution, source control noise, etc).

[Macksaur]

Forcing administrative burden upon users is not a cosmetic complaint.

[JoNax97]

Ok I Will concede that. Nevertheless my main point is that the burden is not that big or problematic, as evidenced by the most used engine in the world.

[loteque]

It is the only option for nearly a decade. And not the pinnacle of usability by its users own admissions.

[JoNax97]

I don't claim it is. I don't think Godot is trying to come up with an unprecedentedly good solution, but to solve a pressing issue in a simple and robust enough way.

I'm not saying it's perfect, I'm saying it's good enough.

[loteque]

To the point that it is a 'good enough' solution: If the point is to quickly provide a mechanism to handle external-to-godot filesystem manipulation and its effects on the dependancy environment in godot - well that book is closed, tested and merged, no? and I partially agree... I think we are tackling a new problem that is intertwined with the old problem, and a deeper one at that. The current implementation surfaced a MAJOR pain point, and is being felt by a subset of Godot users, namely those of us that use source control in our project and/or work in med to large teams and Godot's own community of plugin developers and users. (the subset is nonexclusive as dev builds are not widely used and more regression like issues can pop up as more people get this in their hands, but I really just want to talk about the effected use-cases I can see.)

[Daylily-Zeleen]

Firstly, I believe everyone agrees that UID is useful, and there is no need to discuss this further.

Like UID fallback to path, I think we can use similar way to handle different types of UID.

for example, GDScript, by handling ResourceFormatLoader::has_custom_uid_support() and ResourceFormatLoader::get_resource_uid(), we can find the comment style UID first, if not exists, find the ".uid" file.
Then add an option to ProjectSetting for deciding add comment style UID or external ".uid" file for GDScript, and configure this option in create a new project dialog.

---

All solutions are have cost, but we should try our best to avoid leaving burden on users.

I wound prefer to add a simple comment to represent the UID of GDScript.
To be honest, I have also hated the comment style UID, too.
But the new solution is more ugly and more Unity like( I'm not anti-Unity, I just think we should not use its bad solutions), it leave more burden on users, we should move 2 files when we don't use the editor's filesystem dock, we should commit 2 files to version control system, we should adapt to the ugly folder in file resource manager.

By using comment style UID, just ignore an UID line in script, it not change anything (for users).

By the way, like python, it usually have comments like #!/usr/local/bin/python3 and # -*- coding: utf-8 -*- in the front of scripts. Why we can't accept a line like # uid://xxxxx in the front of GDScript's scripts?

[loteque]

This has been discussed and not all languages that godot supports as project source/resouce support such a tagging system in their source files.

Eidit: But your suggestion here accounts for that I think.

[Daylily-Zeleen]

I know, but it is not relate. ResourceFormatLoader is case by case, we can add ResourceFormatLoaderGDScript for *.gd, ResourceFormatLoaderGDShader for *.gdshader, and so on. If we want, add ResourceFormatLoaderCSharp for .cs is possible.

Edit: We don't need every resource type to support this feature, just for some types that can pollute file system significantly.

[Daylily-Zeleen]

And add UID to all resources will bring a compatibility issue to plugins, please refer to this proposal.

[univeous]

This is actually a very worthwhile topic for discussion - most Node developers don’t include node_modules in their version control systems, but this is quite common in Godot (include addons/ folder). Due to the lack of official support, we use tools like gd-plug or godotenv to better manage plugins; however, UID disrupt this, and yet it seems not many people care. It’s quite disheartening.

I have no strong opinions about UID itself. I just hope that my plugin system can continue to function properly, or better yet, that the official team could provide some sort of package management system.