/
Part 4: Files and File Systems

Part 4: Files and File Systems

About File Systems 

So far all the files we've been dealing with have been in your Home directory, which is the personal directory associated with your account. All Unix systems provide you with a Home directory of your own.

Files and directories in your Home directory are part of the overall file system hierarchy – a tree of directories and their files/sub-directories.

In the file system tree, directories are branches and files are leaves.

To see the partial file system hierarchy of your Home directory, you can use the tree command. (Note that tree is not always available on Linux systems; it is an add-on tool that must be installed separately).

Just calling tree will produce output something like this:

This display shows the hierarchical structure of files and sub-directories. At top-level, there are a number of files that we pre-positioned for you (haiku.txt, mobydick.txt, etc.) along with two sub-directories: data/ and docs/, each with several files in them. And notice how tree colors the directories differently to make them stand out.

Of course there are other areas of the file system hierarchy. For example, the /stor/work/CBRS_unix/ directory.

tree /stor/work/CBRS_unix

Produces output similar to this:

There are three sub-directories under the /stor/work/CBRS_unix/ directory: fastq/, test/ and unix/, each with their own sub-directories and files. Again, notice how tree colors file with the extension .gz differently. These are FASTQ files, compressed using the gzip protocol, produced by a Next Generation Sequencing (NGS) run in our Genome Sequencing and Analysis (GSAF) core facility.

Also note that on Unix, and on Macs, directories are separated by forward slash ( / ) characters, unlike Windows where the backslash ( \ ) is used.

And the root of all the file systems is that 1st forward slash ( / ).

Navigating the file system

Now that we know there are other places, how do we get there? Enter the cd (change directory) command:

  • cd <optional directory_name>

    • with no argument, always changes to your Home directory.

    • the special character ~ (tilde) means your Home directory.

Go Home!

cd # make sure you're in your Home directory cd data # change into the data sub-directory cd ~ # return to your Home directory using ~ cd data/bedfiles # change into the data sub-directory again cd # return to Home

When you've changed into a directory, how do you know where you are in the file system?

  • On our system, we've arranged that the command prompt "follows you" around the file system.

  • On other systems you can use the pwd (present working directory) command to see the full pathname of the directory you're in

    • Full pathnames list all components of a file, starting from / (the root file system).

Exercise 4-1

What is the full pathname of your Home directory?

pwd will display something like /stor/home/student30

Special directory names

There are several "special" built-in directory names:

  • . (single period) means the current directory

  • . . (two periods) means the parent of the current directory (directory above it)

    • So ls .. means "list contents of the parent directory of the current directory"

  • - (dash) means whatever directory you were in before this one

  • ~ (tilde) means your Home directory

Exercise 4-2

  • Change into the /stor/work/CBRS_unix/ directory

    • then back to your Home directory.

  • Then use the - (dash) to go back to /stor/work/CBRS_unix/ without typing it in

    • then go Home again using ~ (tilde)

How does the command prompt change?

cd /stor/work/CBRS_unix cd # Back Home cd - # Change to the last directory # (here /stor/work/CBRS_unix) cd ~ # And Home again

The command prompt "follows you", displaying /stor/work/CBRS_unix when you're in that directory, and ~ when you're in your Home directory.

Relative pathname syntax

A full pathname lists all components of the full file system hierarchy that describes a file. Full paths always start with  / (the forward slash), which is the root of the file system hierarchy. For example /stor/work/CBRS_unix/unix/haiku.txt is a full path.

Rather than always typing a full pathname of where you want to go, you can specify a directory relative to where you are. This is where the special directory names . (single period = current directory) and .. (double period = parent of current directory) come in handy.

Here are some examples. Follow along if you can – and use the Tab key to help out! (See Tab key completion)

cd # make sure you're in your Home directory pwd # where am I now, really? /stor/home/student30/ cd .. # up one level - now I'm in /stor/home/ cd ../work/CBRS_unix/ # change into /stor/work/CBRS_unix/ directory # using relative path syntax cd ./unix/docs/ # change into the unix/docs/ sub-directory # relative to /stor/work/CBRS_unix/ cd ../../fastq/ # go up two levels, then into fastq/ directory cd ../../../home/ # change to the /stor/home/ directory using # relative path syntax cd - # go back to /stor/work/CBRS_unix/fastq/ ls ~/ # list your Home directory contents from anywhere cd # go Home

Pathname wildcards ("globbing")

Since we've seen that a goal of Unix is to type as little as possible, there are several metacharacters that serve as wildcards to represent sets of characters when typing file names. Using these metacharacters is called “globbing” (don't ask me why ) and the pattern is called a glob.

  • asterisk ( * ) is the most common filename wildcard. It matches any length of any characters.

  • brackets ( [ ] ) match any character between the brackets.

    • and you can use a hyphen ( - ) to specify a range of characters (e.g. [A-G])

  • braces ( {  } ) enclose a comma-separated list of strings to match (e.g. {dog,pony})

And wildcards can be combined. Some examples:

ls *.txt # lists all files with names ending in ".txt" ls [a-j]*.txt # does the same but only lists files starting # with lowercase letters a-j ls [Bhim]* # lists all filenames whose 1st letter is B, h, i or m ls data/*.{txt,tsv} # lists filenames ending in either .txt or .tsv

Exercise 4-3

Design a wildcard that will match the files haiku.txt and mobydick.txt but not jabberwocky.txt in your Home directory.

There are always multiple ways of doing things in Unix. Here are two possible answers:

ls [hm]*.txt ls {haiku,jabberwocky}.txt

Finding file systems

So if you're on a new system, how do you know what file systems are available to you? The df (disk free) command will list all the available file systems. As its name suggests, it also shows file system sizes, so it's always good to use df -h (human readable) to see sizes in more readable form. Also, there can be many many file systems available on any given system, so always pipe the output to more.

df -h | more

This produces a rather busy display like this one:

Fortunately, you can ignore most of it. Focus on the Mounted on and Size columns.

  • Look for the / (forward slash), under the Mounted on column:

    • / (forward slash) is the file system root (and where the operating system is installed)

    • Note in the output above its Size is 98G with 61G used

  • Look for Mounted on entries with large Size numbers:

    • Gigabytes (10^9 bytes), Terabytes (10^12 bytes), Petabytes (10^15 bytes)

  • Ignore file systems with names like /run, /dev, /snap, /sys, /boot, /tmp, /var – these are system related

  • Here we see several starting with /stor (/stor, /stor/home, /stor/work, etc.)

    • Note its large Size: 39T - a sign that it's a file system you want to know about.

Here's a similar listing from the Lonestar6 compute cluster at TACC:

Here the big important file systems are /home1 (7.0T), /scratch (8.1P) and /work (6.8P). There's also /admin (3.5T) but its name suggests that normal users won't be able to access it.

File attributes

Let's revisit a listing of our Home directory contents. Before we created any files, an ls -l listing looked something like this:

Ignoring that "total 51" line, there are 9 whitespace-separated columns in this long listing:

  1. file permissions - a 10-character field

  2. number of hard links point to this file - rarely important

  3. account name of the file owner

  4. Unix group associated with the file

  5. file size

  6. last modification month

  7. last modification day

  8. last modification year, or last modification hour:minute if within the last year

  9. file name

Notice we call everything a file, even directories. That's because directories are just a special kind of file – one that contains information about the directory's contents.

Owner and Group

A file's owner is the Unix account that created the file (here student01). That account belongs to one or more Unix groups, and the group associated with a file is listed in field 4.

The owner will always be a member of the groups associated with a file, and other accounts may also be members of the same group. The Unix group for our class is CCBB_Workshops_1. (To see the Unix groups you belong to, just type the groups command.)

Permissions

File permissions and information about the file type are encoded in that 1st 10-character field. Permissions govern who can access a file, and what actions they are allowed.

  • character 1 describes the file type

    • d for directory

    • - (dash) for regular file

    • l for symbolic link

  • the remaining 9 characters are 3 sets of 3-characters

    • characters 2-4: what the owning user account can do

    • characters 5-7: what other members of the associated Unix group can do

    • characters 8-19: what other non-group members (everyone) can do

Each of the 3-character sets describes if read ( r ) write ( w ) and execute ( x or s ) actions are allowed or not allowed ( - ).

  • read ( r )

    • for files, read ( r ) means file contents can be read and copied

    • for directories, read ( r ) means directory contents can be listed (e.g. with ls)

  • write ( w )

    • for files, write ( w ) means file contents can be modified/changed

    • for directories, write ( w ) means directory contents can be modified (files added or deleted)

  • execute ( x or s )

    • for files, execute ( x ) means it is a program that can be called/executed

      • e.g. /usr/bin/ls, the file that performs the ls command

    • for directories, execute ( x or s ) means directory operations may be performed/executed

      • the directory can be opened/changed into (e.g. with cd)

Examples:

ls -l ~/haiku.txt

haiku.txt

description

haiku.txt

description

  • - (dash) in position one signifies this is a regular file

  • rw- for owner allows read and write access

  • r-- for group permits only read access

  • --- for everyone means no access allowed

ls -l /usr/bin/ls

/usr/bin/ls

description

/usr/bin/ls

description

  • /usr/bin/ls is the program that performs the ls command

    • root (the master admin account) is the owner, in the root group

  • - (dash) in position one signifies this is a regular file

  • rwx for owner allows read, write and execute

  • r-x for group permits read and execute

  • r-x for everyone permits read and execute (call the program!)

ls -l -d ~/docs/ (-d says to list directory information, not directory contents)

docs

description

docs

description

  • d in position one signifies this is a directory

  • rwx for owner allows read, write and "execute" (list contents, for directories)

  • r-x for group permits read and "execute" (list)

  • --- for everyone means no access allowed

Exercise 4-4

What are the directory permissions and ownership of /stor/work/CBRS_unix/test/? Can you list its contents? Why or why not?

Use ls -l -d /stor/work/CBRS_unix/test/ to see just that directory's long listing.

ls -ld /stor/work/CBRS_unix/test/ shows the permissions of that directory are drwxr-x--- and it is owned by user abattenh in the CCBB_Workshops_1 group.

Because group permissions are r-x anyone in our CCBB_Workshops_1 group can access that directory, but cannot add/delete files in it because there are no group write permissions.

So ls -l /stor/work/CBRS_unix/test/ successfully lists the directory's contents.