Allow opting out of following directory links (reparse points / symlinks) during recursive file-system enumerations

[mklement0]

Background and Motivation

Related to #24271.

Especially on Unix-like platforms, use of (directory) symlinks (symbolic links) is common.

If, while recursively enumerating a directory (subtree) directory, symlinks to directories are encountered, these symlinks are currently invariably followed, so that the enumeration includes the content of the linked directory as well.

However, you may not want this behavior:

• because it can substantially expand the scope of the enumeration
• it can result in cycles, which:
  • in the benign case can cause re-enumeration of already enumerated directories
  • in the pathological case can result in infinite enumeration (infinite loop).

For this reason, standard Unix utilities such as find require opt-in in order for directory symlinks to be followed.

While an opt-in isn't an option anymore for reasons of backward compatibility, an opt-out should be considered.

Note:

• The opt-out should apply to all types of directory links, which additionally includes junctions and volume mount points on Windows.

• The opt-out should only apply to directory links encountered during enumeration, not to the input path (that is, a directory link as the starting point of an enumeration should always be followed).

• Using AttributesToSkip = FileAttributes.ReparsePoint | FileAttributes.Directory is not a general solution, because you may still want the link itself to be enumerated.

Proposed API

Add a new bool DoNotFollowDirectoryLinks property FollowDirectoryLinks property that defaults to true (see @Joe4evr's comment below) to the System.IO.EnumerationOptions class.

The property name is negotiable. I avoided the term "ReparsePoint" as a generalization (given that it is NTFS-specific) and chose "Link" instead (without including the word "Symbolic", given that the behavior wouldn't be limited to additional link types beside symbolic links).

Update: @carlossanlop proposes FollowSymbolicLinks instead.

Usage Examples

// E.g., on macOS, where /System/Library contains a lot of directory symlinks.
Directory.EnumerateDirectories(
  "/System/Library", 
  "*", 
  new EnumerationOptions { RecurseSubdirectories = true, FollowDirectoryLinks = false, AttributesToSkip = 0 }
);

Alternative Designs

Perhaps introduce a RecurseSubdirectoriesExceptLinks property instead, mutually exclusive with RecurseSubdirectories, which, however, requires enforcement at runtime.

Risks

None that I'm aware of.

Tagging subscribers to this area: @carlossanlop
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Background and Motivation

Related to #24271.

Especially on Unix-like platforms, use of (directory) symlinks (symbolic links) is common.

If, while recursively enumerating a directory (subtree) directory, symlinks to directories are encountered, these symlinks are currently invariably followed, so that the enumeration includes the content of the linked directory as well.

However, you may not want this behavior:

• because it can substantially expand the scope of the enumeration
• it can result in infinite enumeration due to cyclical references.

For this reason, standard Unix utilities such as find require opt-in in order for directory symlinks to be followed.

While an opt-in isn't an option anymore for reasons of backward compatibility, an opt-out should be considered.

Note:

• The opt-out should apply to all types of directory links, which additionally includes junctions and volume mount points on Windows.

• The opt-out should only apply to directory links encountered during enumeration, not to the input path (that is, a directory link as the starting point of an enumeration should always be followed).

• Using AttributesToSkip = FileAttributes.ReparsePoint | FileAttributes.Directory is not a general solution, because you may still want the link itself to be enumerated.

Proposed API

Add a new bool DoNotFollowDirectoryLinks property to the System.IO.EnumerationOptions class.

The property name is negotiable. I avoided the term "ReparsePoint" as a generalization (given that it is NTFS-specific) and chose "Link" instead.

Usage Examples

// E.g., on macOS, where /System/Library contains a lot of directory symlinks.
Directory.EnumerateDirectories(
  "/System/Library", 
  "*", 
  new EnumerationOptions { RecurseSubdirectories = true, DoNotFollowDirectoryLinks = true, AttributesToSkip = 0 }
);

Alternative Designs

Perhaps introduce a RecurseSubdirectoriesExceptLinks property instead, which, however, would have to be mutually exclusive with RecurseSubdirectories, which, however, requires enforcement at runtime.

Risks

None that I'm aware of.

Author:	mklement0
Assignees:	-
Labels:	api-suggestion, area-System.IO, untriaged
Milestone:	-

[Joe4evr]

The proposed name conflicts with the naming guidelines: https://docs.microsoft.com/en-us/dotnet/standard/design-guidelines/names-of-type-members

✔️ DO name Boolean properties with an affirmative phrase (CanSeek instead of CantSeek).

Since EnumerationOptions is a class, it is possible to default-init it to true to maintain the back-compat (and be documented as such).

public bool FollowDirectoryLinks { get; set; } = true;

[mklement0]

Thanks, @Joe4evr; I've updated the initial post accordingly.

[carlossanlop]

Thanks for creating the proposal, @mklement0 .

There are the enumeration APIs that will finish without problems when encountering cyclic symbolic links:

• FileSystemEnumerable<T> where T is anything other than a FileSystemInfo/DirectoryInfo/FileInfo (like for example, string to return the FullPath). When converting to a FileSystemInfo or derived instance, the constructor we call will force a disk hit to retrieve the detailed data of the file. If a cycle is encountered, we throw. On the other hand, if we only return a string, the enumeration can finish succesfully, because FileSystemEntry.Initialize will only do soft retrievals of the file data (won't throw on error).

• The Directory.Get* and Directory.Enumerate* methods return string enumerations of either files, directories or filesystem entries. Similar to the case above, since we do not force a conversion to a FileSystemInfo or derived instance, we don't force a disk hit.

These are the enumeration APIs that will throw when encountering a cycle:

• FileSystemEnumerable<T> where T is a FileSystemInfo or derived instance, which forces a disk hit.

• The DirectoryInfo.Get* and DirectoryInfo.Enumerate* methods return enumerations of FileInfo, DirectoryInfo or FileSystemInfo instances (depending on the method), and they force a disk hit.

I like the idea of adding the new property to EnumerationOptions, since this is a class that can be passed to all the APIs mentioned above. I don't think we need any extra methods (referring to the alternative designs).

Filtering ReparsePoint via AttributesToSkip won't solve the case of enumerating when cyclic symbolic links are found, precisely because at that point, we perform a disk hit and we throw on error. So this is a point in favor of adding this API.

Also, I think the name should be FollowSymbolicLinks, because Unix does not differentiate between file and directory symbolic links (Windows does), so cycles can be found on any symbolic link type.

BTW, I have a related PR to fix a small bug preventing the first two cases to finish successfully. I took this PR as an opportunity to add unit tests to verify enumerations when cyclic symbolic links are found (we didn't have any). #52746

[mklement0]

Thank you for the thoughtful and detailed response, @carlossanlop.

I think the name should be FollowSymbolicLinks, because Unix does not differentiate between file and directory symbolic links (Windows does), so cycles can be found on any symbolic link type.

• It's a quibble, and I'll be happy without the word "Directory" in there, but, conceptually speaking, when encountering a symlink you wouldn't initially be following the symlink from the perspective of the user's intent (enumeration); instead, you'd be inspecting the symlink's target in order to determine its type, so as to be able to make a decision about following (enumeration).

• However, the word "Symbolic" in there may be too narrow, if - which I'm hoping - this following will also be supported for other types or links (reparse points in NTFS parlance, with the exception of hardlinks, which do not point to another named entry).

  • If all these file-system-level-redirection types are supported - and whether they are would also affect the terminology in Proposed API for symbolic links #24271 - just "Link" seems like an appropriate, platform-neutral term to use, which would give us FollowLinks or FollowDirectoryLinks.

[jozkee]

Just brainstorming: would it be better to have an option that either "ignores" or "doesn't follow" links to directories that were already enumerated i.e: break the cycle?
Similar to IgnoreCycles in S.T.JSON #40099.

Of course that would be more expensive since it implies maintaining an stack throughout enumeration.

[mklement0]

@jozkee, setting Follow(Symbolic)Links to $false would implicitly prevent cycles: a link to a directory would simply never be followed and therefore never have the linked directory's contents enumerated; in other words: it seems to me what this API proposes already is the equivalent of IgnoreCycles.

However, following links - the current default behavior - should always prevent cycles, which not only prevents re-enumeration of already enumerated directories, but, more importantly, prevents infinite loops.

@carlossanlop, I haven't worked with the latest builds, but as of .NET 6.0.0-preview.3.21201.4, such infinite loops can happen, at least on Windows (on macOS, the enumeration eventually stops, after many levels of nesting, but I don't know what drives that). Has this since been fixed / will it be fixed?

There is no functional reason not to prevent cycle detection, the only conceivable reason would be to avoid the additional performance cost that it would incur.

Truthfully, the default behavior should never have been to follow links.

• Is revisiting the default - undoubtedly a breaking change - an option at all?

Then following links as an opt-in could justify the extra cost of cycle detection, as a deliberate choice.

In fact, that's how PowerShell's Get-ChildItem cmdlet works: it doesn't follow links by default, and with the -FollowSymlink opt-in you invariably also get cycle detection (when a cycle is found, a warning is issued).

[iSazonov]

There are the enumeration APIs that will finish without problems when encountering cyclic symbolic links:

@carlossanlop Do you mean only symbolic link to itself in the post?

Of course that would be more expensive since it implies maintaining an stack throughout enumeration.

For a notice. This is implemented in PowerShell 6/7. You could look the code if it helps you to implement the feature in .Net.