You know, this makes me wonder.. tangentially speaking- I wonder how hard it would be to rearrange the folder structure in linux so that I have something like this:
Wow, thanks for the reply, nice find! I did some poking around on my Linux system and even re-arranging the home folder was a task of its own because the system kept trying to replace folders in their original places. I will do some digging in to Gobo and see how they're handling this. Thanks again for pointing this out.
Afaik there is an option to change it, but it is not advisable as that will break the binary cache and you are left with compiling everything yourself. This is due to a technical limitation in that different packages can contain paths everywhere and thus they are inherently part of the resulting hash, on which other packages can depend.
Since Live CDs/Flash drives were invented, I wouldn't worry about this stuff any longer. Certainly have your personal files in a centralized location and backed up first.
Probably the easiest way to experiment these days is to create a VM and make snapshot, then start knocking down walls, just to see when and where the house collapses. Then revert and try something new.
A couple of weeks after moving to UNIX from MSDOS, I thought I'd remove lots of unnecessary 'dot-directories' from the /tmp directory. I was root as I had no concept of being a 'normal user'.
At least that doesn't happen today anymore. From bash:
> When a pattern is used for pathname expansion, the character ``.'' at the start of a name or immediately following a slash must be matched explicitly, unless the shell option dotglob is set. The filenames ``.'' and ``..'' must always be matched explicitly, even if dotglob is set.
I did that on my NAS a few years ago. I had copied in a bunch of directories from a mac and they all had tons of dot files in each dir that were showing up on my windows machines. I popped open a terminal and did the exact same thing and wiped most of the NAS out =P Good thing I had it mirrored with my other synology.
GoboLinux symlinks everything into an FHS-ish structure under /System/Index/ so you still have a single place where binaries/libraries/includes/etc. live. (There are also symlinks from /usr/lib, /usr/bin, and others into /System/Index/ for compatibility with programs where those might be hardcoded.)
Want to see true craziness? POSIX file names are just a bag of bytes. They don't even have to be text, they can be anything (almost), there's no standard text encoding:
And in typical Open Source fashion, someone actually claims it's a feature: https://lwn.net/Articles/325398/ because hey, you 99.999% percenters can suffer so that I, 0.001% percenter can implement my wacky system.
I've read that a handful of times (whenever trying to figure out where to put some new random thing), and still have never come to a clear conclusion. Even better, because there are so many similar places, you might choose completely different ones depending on the day of the week and your current mood.
Too much choice for things like this is harmful IMO. Deep down I truly couldn't care less where the files end up, as long as that place is the 'right' place. There are too many 'right' places which makes it hard to find random things at a later date or when on a box you're not super familiar with. It's also a complete waste of time to think about it at all.
It’s not just you: Every distro is its own special snowflake and patches the programs they distribute to store files in a different place.
The “standard” doesn’t tell you what directory structure to use inside /etc to group related config files. The “standard” doesn’t tell you where an HTTP server should serve its files. Everyone just does their own thing which makes upstream docs incorrect and useless for newcomers.
> The “standard” doesn’t tell you what directory structure to use inside /etc to group related config files. The “standard” doesn’t tell you where an HTTP server should serve its files. Everyone just does their own thing which makes upstream docs incorrect and useless for newcomers.
The FHS, does actually answer both of of those questions. Files inside /etc/ should be grouped in subdirectories[0] andd the HTTP server should serve user-specified website files from /srv[1] and normal distro-provided files (such as the apache test page) from /var[2].
"use subdirectories" is probably the most handwavey answer possible, aside from maybe "just put it somewhere, lol". I feel like the standard could provide some sort of guidance on how to name folders or something.
> HTTP server should serve user-specified website files from /srv
I’ve never seen that in my life, but I’m sure someone does that. This is one of those cases where the people who follow the standard are increasing fragmentation
> I've read that a handful of times
(whenever trying to figure out where to
put some new random thing), and still have
never come to a clear conclusion.
So, given some data, say a file and/or
directory, maybe from saving a Web page,
that is relevant to subjects A, K, T, and
Z, where in the file system directory
trees to put that data?
My solution: Put the data in a directory
for one of the subjects A, K, T, or Z
without thinking very hard about which of
these. Then go to a file I call FACTS.DAT
(right, an old idea with an old 8.3 file
name!). I maintain that file with a few,
simple editor macros. So, sure, the file
is a catch-all for entries of random short
facts. And each entry starts with a
time-date stamp and a list of key words.
So, in the case of subjects A, K, T, or Z,
include the key words appropriate for each
of those. Then in the body of the
entry, put the tree name of the
file/directory where did store the data.
In a few seconds with my favorite text
editor I can append an entry or search for
an entry.
So far this year I have put 686 entries in
the file FACTS.DAT for about 2.1 entries
per day. For anything like current
personal computers, handling such a file
is trivial.
Yeah, except that tells me nothing useful... The question is exactly the same: So where do I install this random binary I downloaded from the internet or compiled myself? Is it /opt, /usr/bin, /usr/local/bin, or /bin? Where do I put the dependencies I compiled for this software - /usr/lib, /usr/local/lib, /lib, /opt/lib, /opt/<app name>/lib, or what?
To add: when you install software yourself you choose this, when your install software from e.g. a distribution package it is chosen by the package maintainers, and to a larger extent the maintainers of the distribution.
This is one of the big advantages of using a pre-made advantages of using a ready-made Linux distribution: beyond the convenience of having an installer or easy to install packages, you get some assurance that the system as a whole has been thoughtfully put together.
Arch Linux for example symlinks /bin and /sbin to /usr/bin and /lib to /usr/lib among other things.
Is your account the only account that's expected to run the binary? If so, then `$HOME/bin` is a perfectly acceptable (albeit not standard) place to put it.
If you expect other users to be able to execute the program, then you should put it in either `/usr/bin` or `/usr/local/bin`, depending on whether the former is already being used by a package manager. `/opt` is generally for self-contained software that doesn't play nicely with the rest of the system, but might still be installable through the default package manager.
I don’t think there’s any “official” word on that (the XDG spec that defines ~/.local/share doesn’t mention ~/.local/{bin,lib} IIRC, and the traditional per-user entry in PATH seems to be ~/bin), but a fair number of people use it this way, yes, including me.
I started out using $HOME/bin, but a fair amount of stuff assumes a /usr- or /usr/local-style folder structure when doing make install, so I've settled on using $HOME/usr/bin instead, so that programs can create $HOME/usr/include and $HOME/usr/share and whatever, without trampling on stuff in my home folder.
Can't remember the last time I had a problem arranging this. If using autotools, which covers 95+% of stuff, it's usually a question of something like "./configure --prefix=$HOME/usr".
(If I want to share stuff between users, /usr/local/ is of course a better place. macOS is a bit more restrictive, so I have a separate user for this, whose /usr folder is readable by everybody.)
On freedesktop systems there's the ~/.local directory which is supposed to be a mirror of the file system hierarchy. Seems like a good place for bin, lib, include directories.
Follow your distribution. For example Arch Linux provides PKGBUILDs for official repos and AUR. Most of the time someone has already published PKGBUILD, but if not I just patch accordingly.
And conditions that formed separation are long gone, Arch Linux symlinks most of it:
The standard is, indeed, excessively vague because it was written to let many existing implementations be conformant as is, though I’d say it’s still more helpful than many other standards with that deficiency. There’s a method to it, however:
- Things installed in /, if it’s different from /usr, are generally not to be touched;
- Things installed in /usr are under the distro’s purview or otherwise under a package manager, any modifications are on pain of confusing it;
- Things installed in /usr/local are under the admin’s purview and unmanaged one-offs, there are always some but overuse will lead to anarchy;
- Things installed in /opt are for whatever is so foreign and hopeless in not conforming to the usual factoring that you just give up and put it in its own little padded cell (hello, Mathematica);
- Everything is generally configured using files in /etc, possibly with the exception of some of the special snowflakes in /opt; the package manager will put config files meant to be edited there and expect the admin to merge any changes in manually, and sometimes put default settings meant to be overridden by them in /usr/share (see below)—both approaches can be problematic, but the difficulty is with migrating configuration in general, not the FHS as such.
There used to be additional hierarchies like /usr/X11R6, and even a /usr/etc on some (non-Linux?) systems, but AFAIU everyone agrees their existence makes no sense (anymore?), so much that even FHS doesn’t lower itself to permitting them.
The distinction between / and /usr might appear to be pointless as well, and nowadays it might be (some distros symlink them together), but previously (especially before initial ramdisks were widespread) stuff in / was whatever was needed to bring up the system enough that it could netmount a shared /usr.
Inside each of /, /usr and /usr/local there is bin for things that are supposed to be directly executable, whether binary or a script and all in a single place; share and lib for other portable and non-portable (usually but not necessarily text and binary) shared files, respectively, segregated by application or purpose; finally, due to the dominance of C ABIs and APIs on Unices, the top level of lib also hosts C and C++ library files and there’s an additional directory called include for the headers required to use them. Some people also felt that putting auxiliary executables (things like cc1, the first pass of the C compiler) inside lib was awkward so they created libexec for that purpose, but I don’t think the distinction turned out to be particularly useful so not all distros maintain it.
That’s it, basically. There are subtler but logical points (files vs subdiretories in /etc) and things people haven’t found an obviously superior solution for (multilib and cross environments), and I made no attempt to be historically accurate (the original separation of / and /usr happened for intensely silly reasons), but those are the fundamental principles of the system, and I feel it does make sense as a coherent implementation of a particular design. Other designs are possible (separation by application or package not purpose, Plan 9-ish overlays, NixOS’s isolated environments), but that’s a discussion on a different level; the point is that this one is at the very least internally consistent.
Re the unfriendly names ... I honestly don’t know. Newbie-friendliness matters, but it’s not the only thing that does; particularly in a system intended for interactive text-mode use, concise names have a quality of their own. There’s a reason I’m more willing to reach for curl and jq rather than for httpx and lxml, for regular expressions rather than for Parsec, and even for cmd.exe, as miserable as it is, rather than for PowerShell.
I feel weird that no HCI people seem to have seriously considered the tension between interactive and programmatic environments and what the text-mode user’s experience in Unix says about it, but even Tcl, which is in many ways a Bourne shell done right, loses something in casual REPL use when it eliminates (as far as idiomatic libraries are concerned) short switches. Coming up with things like rsync -avz or objdump -Ctsr is not very pleasant initially, but I certainly wouldn’t want to type out the longhand form that would be the only possible one in most programming languages (even if I find their syntax beautiful, e.g. Smalltalk/Self).
>the original separation of / and /usr happened for intensely silly reasons
As I recall, there were very good reasons for separating / and /usr (as well as /home and /var). The biggest one was that various Unix kernels would panic[0] if / was full. But that issue was almost universally fixed by 1990 or so.
And netmounts of pretty much everything other than / were pretty common for many years, due to the high cost of storage.
So no, the reasons weren't silly, they just don't apply to more modern systems.
OK, I didn’t put this completely correctly. The original separation of /usr to hold user home directories (!) and / to hold everything else was because the first RK05 disk ran out, but it makes sense in any case. The additional hierarchy under /usr was created some time later when space on the first RK05 disk ran out again, and while this can be a perfectly sensible decision for a single installation on a single site, taking it seriously decades later is silly. Neither does that mean that there weren’t good reasons the split got preserved in subsequent systems, just that they couldn’t have been the same as the original ones; there are no netmounts in V6, after all.
(I have an old Unix intro book that describes /usr as user home directories, the rest is a second-hand retelling[1].)
But why are many Windows programs under C:\Windows\System32 then, if Windows has only a single model? Why aren't all Steam-provided (for example) games in a single location? Or, if they are, does Windows really have a single model?
Yes, the Linux/POSIX model is confusing, but the split is to segregate administrative domains:
- / and /usr are the domain of the distribution. As a user, you should never install there. The administrative group is root.
- /usr/local is the domain of the machine admin. If the machine is yours to manage, you can install software there. The administrative group is staff.
- /opt/$vendor is the domain of third-party vendors. Each vendor (like Steam, Eclipse, Arduino Studio) can get its own subdirectory and its own administrative user group.
How would you achieve the same on Windows? How do you make sure the Adobe updater can only install new versions of CS, but not surreptitiously install a new (free!) spyware package under C:\Windows? How would you allow certain power users to share one Google Chrome installation, allow each of them to update it, but not let them install additional software system-wide?
Okay, but what about ProgramData? I have enough programs that put their junk in there instead of Program Files, and others that make their own directories on the root of the drive (driver installers are really bad about this).
I think the best model I've seen for consistent binary locations is the 'Applications' folder in Mac OS X, but it fails as well by retaining the /usr/bin elsewhere.
When you download a portable app (just a bare .exe), do you make a folder for it and drop it in program files? (quite possible, you'd just be unusual) If not, why does Windows get a free pass?
If I am not writing all of my installation scripts by hand, because that would be really intense, then every folder gets filled with random bits of software.
Offering too many similar choices leads to mess. There's nothing fundamentally different between using one or more of these options and using the only option, except that in the second case there isn't any opportunity to make mess.
> You can do `configure --prefix=/Program\ Files/<app>` if you want.
Thanks for the tip! Can't do that with distro repo software though :-/
Use Gnu Stow to keep the random bits contained in their own app directory that is symlinked into the /usr/local tree. Then you can manage everything without leaving orphan files behind.
I feel like it just highlights the problem of how antiquated and confusing linux terminology that so many of those reference "single-user mode", used to refer to booting into root, when the vast majority of computing devices a given user will interact with only have a single actual user, making this a confusing and almost meaningless distinction to someone not already intimate familiar with linux.
The directories that house your executables are read only to users other than root, to prevent attacks and overwriting them by non-root users.
/var stands for variable data--like log files, cache directories, spool directories, etc. You shouldn't put executables there. Ideally you should be able to set the noexec flag on it.
`/usr` actually exists because the original UNIX developers ran out of disk space and had to attach another disk. The difference between /bin and /usr/bin is not worth it and even Debian symlinks /usr/bin to bin.
But your distribution's package manager should be putting stuff in /bin or /usr/bin, not you. Anything that follows the regex "{asterisk}/local{asterisk}" is something the system owner can do whatever with. So you should be using /usr/local/bin or $HOME/local/bin. I don't know why there's no /local off of the root. (One thing I do on my own systems is make and use an /etc/local although I think you're supposed to use something like /usr/local/etc).
/opt is for third party programs that aren't installed via your distro's package manager.
If you do this, any customizations you make to a system can be easily backed up by copying all dirs with local in the name.
There's multiple decades of tradition behind these names, but they do date back to the age where actual teletypes were used.
The only real holdouts are proc/sys/dev which are the kernel and mnt/media/opt/srv which are really for the user/sysadmin and aren't really used by the OS anymore.
Genuine question: on what systems is `/tmp` persistent? Both macOS and Ubuntu 20.04 clear `/tmp` on every reboot for me, and I haven't changed the defaults at all.
Right, /var/tmp is the "Persistent Temp" directory, and /tmp is "Ephemeral Temp". The /run directory is for runtime data such as PID files, Unix sockets, named FIFOs, and generated systemd units—it has a specific internal structure and shouldn't be used as a direct alternative to the relatively unstructured /tmp directory. While both are generally ephemeral tmpfs mounts, only /tmp is writable to all users.
It's not an oxymoron to have files which are temporary but not limited in scope to a single power cycle. For example, you could have a long-running process which you want to be able to resume if it's interrupted; /var/tmp would be an appropriate place for the state. The data is temporary because it will be deleted once the process is finished, but you wouldn't want it wiped out by a system reset. Generally /tmp is cleared at every reset, and is often a tmpfs mount, while files in /var/tmp are automatically cleaned up only when they reach a certain age.
Except that the FHS says that "data stored in /var/tmp is typically deleted in a site-specific manner", and as an application vendor you have no control over that site-specific clean frequency. On all my systems, /var/tmp is a symlink to /tmp and that has never caused any issue.
The FHS is not wrong; cleaning policies are indeed site-specific and files placed in any temp directory can in principle disappear at any time. (Though, in theory, it's not supposed to happen while the files are still "in use" by running programs.) Still, historically you could count on files in /var/tmp lasting longer than files in /tmp, including across reboots.
Nothing will immediately break because you linked /var/tmp to /tmp. Whether it causes issues depends on the programs that you (or your users) run and how they make use of /var/tmp. However, if someone did have to restart a long-running process from the beginning because recent state information in /var/tmp was not preserved across a reset, I would say that is a problem with the administration of the system and not the program that stored its state there.
Basically no one uses /var/tmp for anything (and nobody should either). World writable directories are a mistake and only continue to exist because apps assume they are available.
/tmp and friends are poorly named. They really should be /shared or /dmz or /freeforall or something.
* If you need service-specific tmp space use RuntimeDirectory or PrivateTmp if your app is hardcoded to /tmp.
* If you need service-specific persistent data that goes in /var/lib/your-app.
* If you need temp space for your user it's at /var/run/user/your-uid.
* If you need more than one user/service to share files but not everyone then god have mercy on your soul because all options are bad. There sure are a lot of them but none of them are at all satisfying.
People don't reboot often. Persistent tmp basically means it will be cleared in an infrequent manner, so the likelihood of it going away 1s after you release your file handle is low.
Why not? That's how proper English text is written. Of course there are many programs that can't handle it properly (or handles it inconveniently) so in practice it might be problematic at times, but otherwise I see nothing wrong with it.
Generally just because typing it out with tab completion in zsh sucks, and I don't see a good solution (if it was solved nicely it'd be solved already)
Is it coincidence that you almost exactly replicated what macOS has? Except that /Devices is /Volumes, .../Apps is .../Applications. and /Boot is handled differently.
Of course, that's not perfect either, because a) decades of changes vs. compatibility have made it less clean in certain places, and b) pretty much all the POSIX paths still exist for unix-y compatibility, but overall it's like that.
> I wonder how hard it would be to rearrange the folder structure in linux
Restructuring the directories is the easy part. You just delete the old tree and make a new one. You can also mount procfs and sysfs wherever you want.
The hard part is modifying existing software to work with the new tree. So many programs assume you have a "standard" file system tree. So many programs assume procfs is mounted at /proc. So many programs have hardcoded paths. Shared library locationd can become part of the binaries when they're compiled. It's insane and you'd essentially be creating a new Linux distribution.
MacOS too. /usr/ and /dev/ and whatnot exist, they're just flagged as invisible in Finder. There's a command to globally unhide them for those who want to see them.
/Users/{root, user0, user1, ... }...
/System/{Logs, Apps/{opt, container, ...}, Temp, Conf ...}...
/Devices/{Mount, sda, sdb, null ...}...
/Boot/...