Part 4: Files and File Systems
About File Systems
So far all the files we've been dealing with have been in your Home directory, which is the personal directory associated with your account. All Unix systems provide you with a Home directory of your own.
Files and directories in your Home directory are part of the overall file system hierarchy – a tree of directories and their files/sub-directories.
In the file system tree, directories are branches and files are leaves.
To see the partial file system hierarchy of your Home directory, you can use the tree command. (Note that tree is not always available on Linux systems; it is an add-on tool that must be installed separately).
Just calling tree will produce output something like this:
This display shows the hierarchical structure of files and sub-directories. At top-level, there are a number of files that we pre-positioned for you (haiku.txt, mobydick.txt, etc.) along with two sub-directories: data/ and docs/, each with several files in them. And notice how tree colors the directories differently to make them stand out.
Of course there are other areas of the file system hierarchy. For example, the /stor/work/CBRS_unix/ directory.
tree /stor/work/CBRS_unixProduces output similar to this:
There are three sub-directories under the /stor/work/CBRS_unix/ directory: fastq/, test/ and unix/, each with their own sub-directories and files. Again, notice how tree colors file with the extension .gz differently. These are FASTQ files, compressed using the gzip protocol, produced by a Next Generation Sequencing (NGS) run in our Genome Sequencing and Analysis (GSAF) core facility.
Also note that on Unix, and on Macs, directories are separated by forward slash ( / ) characters, unlike Windows where the backslash ( \ ) is used.
And the root of all the file systems is that 1st forward slash ( / ).
Navigating the file system
Now that we know there are other places, how do we get there? Enter the cd (change directory) command:
cd <optional directory_name>
with no argument, always changes to your Home directory.
the special character ~ (tilde) means your Home directory.
Go Home!
cd # make sure you're in your Home directory
cd data # change into the data sub-directory
cd ~ # return to your Home directory using ~
cd data/bedfiles # change into the data sub-directory again
cd # return to HomeWhen you've changed into a directory, how do you know where you are in the file system?
On our system, we've arranged that the command prompt "follows you" around the file system.
On other systems you can use the pwd (present working directory) command to see the full pathname of the directory you're in
Full pathnames list all components of a file, starting from / (the root file system).
Exercise 4-1
What is the full pathname of your Home directory?
Special directory names
There are several "special" built-in directory names:
. (single period) means the current directory
. . (two periods) means the parent of the current directory (directory above it)
So ls .. means "list contents of the parent directory of the current directory"
- (dash) means whatever directory you were in before this one
~ (tilde) means your Home directory
Exercise 4-2
Change into the /stor/work/CBRS_unix/ directory
then back to your Home directory.
Then use the - (dash) to go back to /stor/work/CBRS_unix/ without typing it in
then go Home again using ~ (tilde)
How does the command prompt change?
Relative pathname syntax
A full pathname lists all components of the full file system hierarchy that describes a file. Full paths always start with / (the forward slash), which is the root of the file system hierarchy. For example /stor/work/CBRS_unix/unix/haiku.txt is a full path.
Rather than always typing a full pathname of where you want to go, you can specify a directory relative to where you are. This is where the special directory names . (single period = current directory) and .. (double period = parent of current directory) come in handy.
Here are some examples. Follow along if you can – and use the Tab key to help out! (See Tab key completion)
cd # make sure you're in your Home directory
pwd # where am I now, really? /stor/home/student30/
cd .. # up one level - now I'm in /stor/home/
cd ../work/CBRS_unix/ # change into /stor/work/CBRS_unix/ directory
# using relative path syntax
cd ./unix/docs/ # change into the unix/docs/ sub-directory
# relative to /stor/work/CBRS_unix/
cd ../../fastq/ # go up two levels, then into fastq/ directory
cd ../../../home/ # change to the /stor/home/ directory using
# relative path syntax
cd - # go back to /stor/work/CBRS_unix/fastq/
ls ~/ # list your Home directory contents from anywhere
cd # go HomePathname wildcards ("globbing")
Since we've seen that a goal of Unix is to type as little as possible, there are several metacharacters that serve as wildcards to represent sets of characters when typing file names. Using these metacharacters is called “globbing” (don't ask me why ) and the pattern is called a glob.
asterisk ( * ) is the most common filename wildcard. It matches any length of any characters.
brackets ( [ ] ) match any character between the brackets.
and you can use a hyphen ( - ) to specify a range of characters (e.g. [A-G])
braces ( { } ) enclose a comma-separated list of strings to match (e.g. {dog,pony})
And wildcards can be combined. Some examples:
ls *.txt # lists all files with names ending in ".txt"
ls [a-j]*.txt # does the same but only lists files starting
# with lowercase letters a-j
ls [Bhim]* # lists all filenames whose 1st letter is B, h, i or m
ls data/*.{txt,tsv} # lists filenames ending in either .txt or .tsvExercise 4-3
Design a wildcard that will match the files haiku.txt and mobydick.txt but not jabberwocky.txt in your Home directory.
Finding file systems
So if you're on a new system, how do you know what file systems are available to you? The df (disk free) command will list all the available file systems. As its name suggests, it also shows file system sizes, so it's always good to use df -h (human readable) to see sizes in more readable form. Also, there can be many many file systems available on any given system, so always pipe the output to more.
df -h | moreThis produces a rather busy display like this one:
Fortunately, you can ignore most of it. Focus on the Mounted on and Size columns.
Look for the / (forward slash), under the Mounted on column:
/ (forward slash) is the file system root (and where the operating system is installed)
Note in the output above its Size is 98G with 61G used
Look for Mounted on entries with large Size numbers:
Gigabytes (10^9 bytes), Terabytes (10^12 bytes), Petabytes (10^15 bytes)
Ignore file systems with names like /run, /dev, /snap, /sys, /boot, /tmp, /var – these are system related
Here we see several starting with /stor (/stor, /stor/home, /stor/work, etc.)
Note its large Size: 39T - a sign that it's a file system you want to know about.
Here's a similar listing from the Lonestar6 compute cluster at TACC:
Here the big important file systems are /home1 (7.0T), /scratch (8.1P) and /work (6.8P). There's also /admin (3.5T) but its name suggests that normal users won't be able to access it.
File attributes
Let's revisit a listing of our Home directory contents. Before we created any files, an ls -l listing looked something like this:
Ignoring that "total 51" line, there are 9 whitespace-separated columns in this long listing:
file permissions - a 10-character field
number of hard links point to this file - rarely important
account name of the file owner
Unix group associated with the file
file size
last modification month
last modification day
last modification year, or last modification hour:minute if within the last year
file name
Notice we call everything a file, even directories. That's because directories are just a special kind of file – one that contains information about the directory's contents.
Owner and Group
A file's owner is the Unix account that created the file (here student01). That account belongs to one or more Unix groups, and the group associated with a file is listed in field 4.
The owner will always be a member of the groups associated with a file, and other accounts may also be members of the same group. The Unix group for our class is CCBB_Workshops_1. (To see the Unix groups you belong to, just type the groups command.)
Permissions
File permissions and information about the file type are encoded in that 1st 10-character field. Permissions govern who can access a file, and what actions they are allowed.
character 1 describes the file type
d for directory
- (dash) for regular file
l for symbolic link
the remaining 9 characters are 3 sets of 3-characters
characters 2-4: what the owning user account can do
characters 5-7: what other members of the associated Unix group can do
characters 8-19: what other non-group members (everyone) can do
Each of the 3-character sets describes if read ( r ) write ( w ) and execute ( x or s ) actions are allowed or not allowed ( - ).
read ( r )
for files, read ( r ) means file contents can be read and copied
for directories, read ( r ) means directory contents can be listed (e.g. with ls)
write ( w )
for files, write ( w ) means file contents can be modified/changed
for directories, write ( w ) means directory contents can be modified (files added or deleted)
execute ( x or s )
for files, execute ( x ) means it is a program that can be called/executed
e.g. /usr/bin/ls, the file that performs the ls command
for directories, execute ( x or s ) means directory operations may be performed/executed
the directory can be opened/changed into (e.g. with cd)
Examples:
ls -l ~/haiku.txt
haiku.txt | description |
|---|---|
|
|
ls -l /usr/bin/ls
/usr/bin/ls | description |
|---|---|
|
|
ls -l -d ~/docs/ (-d says to list directory information, not directory contents)
docs | description |
|---|---|
|
|
Exercise 4-4
What are the directory permissions and ownership of /stor/work/CBRS_unix/test/? Can you list its contents? Why or why not?