Write macOS implementation of ishidden #1

jakewilliami · 2022-08-24T06:23:44Z

macOS implementation is difficult, because it follows UNIX conventions whilst also having some other information.

As far as I can tell, the only other thing that determines whether a file is hidden in macOS is a certain attribute in finder flags. Once I finish this implementation, I should confirm with Stefan, as he may know about some other file attributes that we need to check (in his comment he mentions there are three, on top of the UNIX standard).

jakewilliami · 2022-08-24T10:35:53Z

The current problem is that, although Julia has an fstat method, the results are stuffed into a StatStruct struct that doesn't account for some of the fields that I need. As accessing a field in C is just a dereference of a pointer+offset, I think one way to go about this is using something like CStructures.jl in order to implement an equivalent struct in Julia (so that all the fields are at the same offsets as they would be in C), create an output container like Ref{MyStatStruct}(), and then pass that ref as the statbuf argument to ccall(:stat, ...). You can then do statbuf[].myfield to access the field in question.

Still need to use this output with some constant to infer whether this file is hidden or not This function (_sv_flags) could definitely be improved. One way would be to dynamically allocate required size, but I also hate that we are indexing stat buffer directly. The sv_flags field should always be in position 11 (or 21 if using 32-bit), but I still don't like it Addresses #1

After trialing many different methods for parsing the sv_flags output using UF_HIDDEN (all of which had varying resources), I think the one that works is the one I have implemented. This finalises the macOS implementation of this function. I still have improvements to add, as specified in d3ff556, but these can come later. Addresses #1. Leaving the issue open in the interest of improvements and verifying that _isinvisible is correctly defined.

jakewilliami · 2022-08-24T11:22:02Z

Now that I have the sv_flags field from lstat, I had to figure out how to parse its output to determine if a file was hidden. This proved someone controversial, as I defined the following:

Based on this StackOverflow article and the kLSItemInfoIsInvisible constant:

const KLS_ITEM_INFO_IS_INVISIBLE = 0x00000040
_isinvisible(f::AbstractString) = !iszero(_sv_flags(f) & KLS_ITEM_INFO_IS_INVISIBLE)

Based on this man page ("The flags specified are formed by or'ing the following values") and the UF_HIDDEN constant:

const UF_HIDDEN = 0x00008000
_isinvisible(f::AbstractString) = !iszero(_sv_flags(f) | UF_HIDDEN)

Based on this GitHub comment and the UF_HIDDEN constant:

const UF_HIDDEN = 0x00008000
_isinvisible(f::AbstractString) = !iszero(_sv_flags(f) & ~UF_HIDDEN)

The implementation I ended up going with (seen implemented elsewhere^{[1], [2]}), using the UF_HIDDEN constant, is:

const UF_HIDDEN = 0x00008000
_isinvisible(f::AbstractString) = (_sv_flags(f) & UF_HIDDEN) == UF_HIDDEN

Addresses #1

jakewilliami · 2022-08-24T19:25:53Z

Based on 'Apple Developer Documentation - Hidden Files and Directories: Simplifying the User Experience', I still have a couple of things to implement (some addressing separate issues as well):

Check for dot syntax
Exclude . and .. from previous check
Check UNIX-specific directories (will also be used for Research/write BSD implementation of ishidden #3)—are all subfiles of these also hidden? Leaning towards no. (Related: Identify UNIX-specific directories)
Check explicitly hidden files and directories (from Finder)
Check packages/bundles—inner files are hidden from users, should they show as hidden in this application? (Related: What C calls does mdls use?)

Addresses #1

Addresses #1. getxattr does not actually get what I want, as we need to parse whatever `mdls` uses to get file metadata, however I am commiting this now in case getxattr does come in handy Also added comments for the other cases, as per https://developer.apple.com/library/archive/documentation/FileManagement/Conceptual/FileSystemProgrammingGuide/FileSystemOverview/FileSystemOverview.html#//apple_ref/doc/uid/TP40010672-CH2-SW7

Even though . and .. directories start with ., they should be excluded from being hidden Addresses #1

Addresses #1. It occurred to me that we don't need to check . and .. if we can expand the path to ignore any relative path components

Addresses #1. In this commit I have removed any old work using getxattr, as getxattr deals with extended attributes (metadata) attached to files. They're actually unrelated to ext. attrs.: mdls involves higher level stuff that makes calls into LaunchServices and uses the LS database. I have discovered (using nm) that in particular, mdls makes calls to MDItemCopyAttributeNames and MDItemCopyAttributes. As such I have been working on porting over logic using these C calls. However, MDItem* functions require path names to be CFStrings, so a lot of work has gone into different encodings (thanks Apple). I still need to get the _mdls function working as expected, as currently I presumably have the attribute list using MDItemCreate and MDItemCopyAttributeNames, however I need to parse it correctly and extract the attribute that I want to use to determine if the path is a package or a bundle.

Addresses #1. After a _lot_ of trial and error, I figured out I was using the wrong function call! I was calling MDItemCopyAttributeNames rather than MDItemCopyAttribute (at which point I needed to _specify_ the attribute name myself). After this point, it was relatively easy to get this all working. Now I just need to clean up old functions I didn't end up using.

After all of the trial and error, there were some functions that were unused, and some variables misnamed Addresses #1

Importantly, the file or directory is hidden if one of its parents are contained in a package or bundle. Also added tests for this case. Addresses #1

jakewilliami · 2022-09-01T09:38:33Z

Just found this implementation. I wonder if Julia has any of the NSURL stuff implemented somewhere?

EDIT: The module that they import here (Foundation) is comes from calling Objective C,^{[1], [2]} therefore, perhaps out of scope for the moment. See also: #8.

Addresses #1, with #6 in mind

jakewilliami · 2022-09-10T09:30:57Z

Does checking for packages and bundles actually matter? If a file is contained within a package or bundle, is it really hidden? Should we only enable this feature with a flag? I will raise this with the others.

EDIT: see #16.

BSD-related OS's (i.e., FreeBSD and macOS) can have the st_flags check applied to them (as I implemented this for macOS), so this function was moved out into a separate branch. ZFS function stubs were also created (with errors in case people try to use them at this stage in development) so that I can (hopefully soon) start implementing them. Addresses #1, #3, #12, #13

Apparently you can get ZFS on macOS: https://github.com/openzfsonosx/zfs Addresses #1, #12, #13

Getting back to the task of checking whether a directory is a _system_ directory, I reviewed an old link I posted in #1: https://stackoverflow.com/a/1140345/12069968. While a lot of these function appear to be deprecated now, it seems to work somewhat well. For example, `_isinvisible_alt` returns `true` on all UNIX-specific directories listed in the macOS resources for this type of hidden file. We still need to: - Implement tests for all of these UNIX-specific directories; - Find non-deprecated system calls to obtain the same information; - Correctly construct LSItemInfoRecord (rather than allocating a reasonably large buffer because I don't know how large it needs to be), and find the correct offset for flags. Addresses #1 Ref: #1 (comment) ``` julia> HiddenFiles._isinvisible_alt("/bin") true julia> HiddenFiles._isinvisible_alt("/dev/") true julia> HiddenFiles._isinvisible_alt("/dev") true julia> HiddenFiles._isinvisible_alt("/etc") true julia> HiddenFiles._isinvisible_alt("/sbin/") true julia> HiddenFiles._isinvisible_alt("/sbin") true julia> HiddenFiles._isinvisible_alt("/tmp") true julia> HiddenFiles._isinvisible_alt("/usr") true julia> HiddenFiles._isinvisible_alt("/var") true ```

jakewilliami added a commit that referenced this issue Aug 24, 2022

Clean up old macOS implementation/comments

58bdb64

Addresses #1

jakewilliami added a commit that referenced this issue Aug 24, 2022

Add dynamic size constructor for statbuf array for macOS

6544777

Addresses #1

jakewilliami added a commit that referenced this issue Aug 27, 2022

Correct st_flags name

45021bc

Addresses #1

jakewilliami added a commit that referenced this issue Aug 28, 2022

Check for . and .. for UNIX

78fe51a

Even though . and .. directories start with ., they should be excluded from being hidden Addresses #1

jakewilliami added a commit that referenced this issue Aug 28, 2022

Expand path to ignore relative considerations in UNIX

663010d

Addresses #1. It occurred to me that we don't need to check . and .. if we can expand the path to ignore any relative path components

jakewilliami added a commit that referenced this issue Aug 29, 2022

Unused function clean up and bug fix

533afc6

After all of the trial and error, there were some functions that were unused, and some variables misnamed Addresses #1

jakewilliami added a commit that referenced this issue Aug 29, 2022

Finalise package/bundle case for macOS

69ad99a

Importantly, the file or directory is hidden if one of its parents are contained in a package or bundle. Also added tests for this case. Addresses #1

jakewilliami added the macOS label Aug 29, 2022

jakewilliami added a commit that referenced this issue Sep 1, 2022

Standardise string encoding with constant variable

1c1ba4b

Addresses #1, with #6 in mind

jakewilliami added a commit that referenced this issue Sep 11, 2022

Add ZFS check on macOS

77f38cb

Apparently you can get ZFS on macOS: https://github.com/openzfsonosx/zfs Addresses #1, #12, #13

jakewilliami mentioned this issue Sep 26, 2022

Draft: Identify macOS UNIX-specific system directories #18

Open

12 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Write macOS implementation of ishidden #1

Write macOS implementation of ishidden #1

jakewilliami commented Aug 24, 2022 •

edited

Loading

jakewilliami commented Aug 24, 2022

jakewilliami commented Aug 24, 2022

jakewilliami commented Aug 24, 2022 •

edited

Loading

jakewilliami commented Sep 1, 2022 •

edited

Loading

jakewilliami commented Sep 10, 2022 •

edited

Loading

Write macOS implementation of ishidden #1

Write macOS implementation of ishidden #1

Comments

jakewilliami commented Aug 24, 2022 • edited Loading

jakewilliami commented Aug 24, 2022

jakewilliami commented Aug 24, 2022

jakewilliami commented Aug 24, 2022 • edited Loading

jakewilliami commented Sep 1, 2022 • edited Loading

jakewilliami commented Sep 10, 2022 • edited Loading

jakewilliami commented Aug 24, 2022 •

edited

Loading

jakewilliami commented Aug 24, 2022 •

edited

Loading

jakewilliami commented Sep 1, 2022 •

edited

Loading

jakewilliami commented Sep 10, 2022 •

edited

Loading