After some investigation, I have evidence that poor performance is related to files belonging to a user from another domain. In particular, I came to the following conclusions:
- I work in a corporate environment with several domains and thousands of users.
- Due to organizational changes, each user, probably only at the transition stage, is stored in two domains, in his main domain, as well as in the second domain. When you change ownership of objects through the Windows GUI, each user appears twice, and you need to go to the advanced user selection to identify the one that is assigned to a specific domain.
- cygwin with acl enabled displays the user of the "other domain" file as "<domain> + <username>". The primary domain is simply "<username>". Cygwin without acl only displays "<username>"; in both cases. This can be quite confusing, as permission to use and ownership, as defined by cygwin, indicates write permission, while the user does not actually have it.
- Files belonging to the "other domain" itself can be writable by "this domain" itself, so the purpose of the domain is largely transparent.
- The large source tree from our version control system (which was also reflected in the git repo) had thousands of files belonging to a "different domain." This seems to cause slow file operations. A change in ownership of the "primary domain" fixed a speed problem for both git and other file access.
I must assume that obtaining file permissions for users in other domains is slow and for some reason is not cached (he was always the same user).
The rest of the article below is what I originally published. I let it stand.
For me (working in a large company with several geographically distributed Windows domains), the culprit is that cygwin uses Windows acl by default. Consider this request for all known users in the Domain:
$ time (mkpasswd -D | wc -l) 45183 real 27m55,340s user 0m1,637s sys 0m0,123s
Fix (1) (2) was a simple matter for installing NTFS file systems using noacl , i.e. my /etc/fstab contains the line
none / cygdrive binary,posix=0,user,noacl 0 0
(while eliminating the annoying cygdrive prefix).
I can’t help but imagine that cygwin / msys (the same behavior there, except that installing Windows git is mounted by noacl by default, probably for this reason) performs a domain server request for every file it touches, and Don't cache results.
This change was introduced around 2015 using cygwin 2.4 or 2.5. From the release notes for version 2.4:
To comply with the standard Windows ACLs, the permissions of the POSIX owner and all other users in the ACL are calculated using the Windows AuthZ API. This may slow down the calculation of POSIX permissions, noticeably under certain circumstances [...] (emphasis by me).
The noacl option reduced the start time of BeyondCompare (or an echo line, for that matter) from 25 seconds to 1. It is completely incomprehensible why a simple git diff in the same file is very fast even with acl since I would naively assume that the required information and, therefore, the required actions of the FS are the same.
Now I will conduct a cygserver , which can improve the situation by caching.
Update: cygserver does not improve the situation, unfortunately.
(1) Fix for git. mkpasswd no effect.
(2) I did not understand and did not influence the access rights and ownership of git (and the ClearCase views, which we also access through cygwin). The feeling of my feeling is to stay as close as possible to the semantics of Windows (which means that noacl might run into problems).
(3) The cygwin documentation discusses scenarios in which query results are not cached. One of them consists of a sequence of cygwin processes that are not spawned from a common ancestor of cygwin (e.g. bash), but from a window program such as cmd . I must assume that Windows provides a caching mechanism for its own programs, or that Windows will not be suitable for use in this corporate environment. For some reason, cygwin is not using it.