Executive Summary

Linux does not have one permission system. It has a stack of them, and a request only succeeds when every layer says yes. Most “Permission denied” tickets come from engineers looking at one layer (usually the nine rwx bits) while a different layer (a parent directory, an ACL mask, a mount option, or SELinux) is the one actually saying no.

This guide walks the full stack in the order the kernel evaluates it, gives you the notation and commands for each layer, and then puts it all together in a hands-on lab: a shared team directory where members can collaborate, cannot delete each other’s work, an auditor gets read-only access, and a golden config file cannot be touched even by root without a deliberate unlock.

Prerequisites and Architecture

Assumed knowledge

• Comfortable in a Linux shell and with sudo.
• Basic understanding of users and groups (/etc/passwd, /etc/group).
• Familiarity with binary to decimal conversion helps for octal notation, but the tables below cover it.

Lab requirements

• One Linux VM or container host (Ubuntu 24.04 LTS or RHEL 9 family) with sudo access.
• An ext4 or XFS filesystem for the lab path (both support ACLs and chattr attributes out of the box).
• The acl package for getfacl/setfacl and the capability tools (libcap2-bin on Ubuntu, libcap on RHEL).

Install the tooling

# Ubuntu / Debian
sudo apt update && sudo apt install -y acl libcap2-bin

# RHEL / Rocky / Alma
sudo dnf install -y acl libcap

The Linux rights stack

Think of access control as layers. The classic mode bits are the layer everyone knows, but the others can override or restrict them.

How the Kernel Decides: The Access Check

When a process opens a file, the kernel compares the process credentials (effective UID, effective GID, supplementary groups, and capabilities) against the inode. The discretionary (DAC) part of the check works like this:

1. Is the process privileged? A process with CAP_DAC_OVERRIDE (normally root) bypasses read and write checks. For execute, at least one x bit must be set somewhere on a regular file, even for root.
2. Is the process the owner? If the effective UID matches the file owner, the owner bits are used and evaluation stops.
3. Is there a matching named-user ACL entry? If so, that entry (limited by the ACL mask) is used.
4. Is the process in the group class? If the effective GID or any supplementary group matches the owning group or a named-group ACL entry, the group permissions (limited by the mask) are used.
5. Otherwise the other bits are used.
6. Then mandatory layers apply: mount options, SELinux or AppArmor policy, and file attributes such as immutable.

Demonstrating first-match-wins

$ touch trap.txt && chmod 077 trap.txt
$ cat trap.txt
cat: trap.txt: Permission denied

Reading ls -l Like a Pro

Everything starts with the long listing. The first column packs the file type, nine permission bits, the three special bits (overlaid on the execute slots), and a trailing indicator for ACLs or security contexts.

Four listings, four lessons

$ ls -ld /usr/bin/passwd /tmp /srv/netops /etc/shadow
-rwsr-xr-x  1 root root   64152 Apr  9 12:01 /usr/bin/passwd
drwxrwxrwt 18 root root    4096 Sep 29 08:14 /tmp
drwxrws--T+ 2 root netops  4096 Sep 29 08:20 /srv/netops
-rw-r-----  1 root shadow  1402 Sep 12 10:33 /etc/shadow

For scripting and audits, stat is more precise than parsing ls:

stat with a custom format

$ stat -c '%A  %a  %U:%G  %n' /usr/bin/passwd /etc/shadow
-rwsr-xr-x  4755  root:root  /usr/bin/passwd
-rw-r-----  640  root:shadow  /etc/shadow

rwx on Files vs. Directories

The same three letters mean very different things depending on the inode type. Misunderstanding directory permissions is the number one source of “but the file is 644, why can’t I read it?” tickets.

A directory with x but not r is a classic “drop box” pattern: users can reach a file if they already know its exact name, but cannot list what is there.

Octal and Symbolic Notation

Each permission triad is three bits, which maps cleanly onto one octal digit: r = 4, w = 2, x = 1. Add them up per class. An optional leading fourth digit carries the special bits: setuid = 4, setgid = 2, sticky = 1.

chmod in practice

chmod patterns worth memorizing

# Absolute (octal): set every bit explicitly
chmod 640 app.conf
chmod 2770 /srv/netops

# Symbolic: change only what you name
chmod u+x deploy.sh            # add execute for owner
chmod g-w,o= report.csv        # remove group write, clear other
chmod a+r README.md            # everyone can read
chmod u=rwx,g=rx,o= tool/      # set classes exactly

# Capital X: execute only on directories (or files already executable)
chmod -R u=rwX,g=rX,o= /srv/share

# Copy the mode from a reference file
chmod --reference=good.conf bad.conf

Ownership: Users, Groups, chown, and chgrp

Every inode has exactly one owning user and one owning group. Group membership is how you scale access without handing out ownership.

Ownership and group commands

id alice                              # uid, primary gid, supplementary groups
sudo chown alice report.csv           # change owner
sudo chown alice:netops report.csv    # change owner and group
sudo chgrp netops report.csv          # change group only
sudo chown -R svc-syslog:adm /var/log/remote
sudo chown --reference=good.conf bad.conf

# Group management
sudo groupadd netops
sudo usermod -aG netops alice         # -a is critical: append, do not replace
getent group netops

• Only root (or a process with CAP_CHOWN) can give a file away to another user.
• A regular user may chgrp their own file to any group they belong to.
• The kernel clears setuid and setgid on an executable when its owner changes. Set special bits after chown, never before.

umask: Where Default Permissions Come From

Programs request a base mode when they create things (normally 666 for files and 777 for directories). The kernel then removes any bits set in the process umask. The formula is a bitwise AND with the inverted mask, not subtraction.

Viewing and setting umask

umask            # show current mask (octal)
umask -S         # show as symbolic: u=rwx,g=rx,o=rx
umask 027        # set for this shell only

# Persistent, system-wide (applied at login via pam_umask)
grep -E '^UMASK' /etc/login.defs

# Per-service in systemd
sudo systemctl edit myservice
#   [Service]
#   UMask=0027

Special Bits: setuid, setgid, and Sticky

The fourth octal digit changes behavior rather than granting access. It displays in the execute slot of each triad: lowercase when execute is also set, uppercase when it is not (which is usually a mistake).

Setting and clearing special bits

chmod u+s /usr/local/bin/tool     # setuid     (4xxx)
chmod g+s /srv/netops             # setgid     (2xxx)
chmod +t  /srv/dropbox            # sticky     (1xxx)
chmod 3770 /srv/netops            # setgid + sticky in one shot

chmod u-s,g-s /usr/local/bin/tool # remove

• setuid is ignored on interpreted scripts. The kernel honors it only on native binaries. Never try to make a shell script setuid; use sudo with a tight rule instead.
• setuid binaries are prime privilege-escalation targets. Every one of them should be inventoried and justified (see the audit section).
• GNU chmod preserves setuid and setgid on directories when you use a 3-digit numeric mode. chmod 755 dir does not clear an existing setgid bit. Use chmod g-s dir or a 5-digit mode such as chmod 00755 dir.
• Mount with nosuid (/tmp, /home, removable media, NFS) so setuid bits on those filesystems are ignored entirely.

POSIX ACLs: Beyond One Owner and One Group

Mode bits give you exactly three principals. When an auditor, a second team, or a service account needs access without joining the owning group, use POSIX Access Control Lists. They are supported by default on ext4, XFS, and Btrfs.

ACL entry types

Everyday setfacl and getfacl

# Grant, inspect, remove
setfacl -m u:carol:rX /srv/netops          # add or modify a named user entry
setfacl -m g:auditors:rX /srv/netops       # named group entry
getfacl /srv/netops                        # view (note the #effective column)
setfacl -x u:carol /srv/netops             # remove one entry
setfacl -b /srv/netops                     # remove all extended entries

# Inheritance for new files and dirs
setfacl -d -m g:auditors:rX /srv/netops    # default ACL entry
setfacl -k /srv/netops                     # remove the default ACL

# Recursive apply to existing content
setfacl -R -m g:auditors:rX /srv/netops

# Backup and restore (run from /)
getfacl -R /srv/netops > /root/netops.acl
setfacl --restore=/root/netops.acl

• cp -a, rsync -A, and tar --acls preserve ACLs. Plain cp and many backup tools do not.
• An ACL grants access by adding entries; it cannot grant anything to the owner that the owner bits deny, because owner matching stops evaluation first.

File Attributes: Immutable and Append-Only

Filesystem attributes sit below the permission model. The two that matter most for security are immutable (i) and append-only (a). Setting or clearing them requires CAP_LINUX_IMMUTABLE, which only root has by default.

chattr and lsattr

sudo chattr +i /etc/ssh/sshd_config
lsattr /etc/ssh/sshd_config
# ----i---------e------- /etc/ssh/sshd_config

sudo rm /etc/ssh/sshd_config
# rm: cannot remove '/etc/ssh/sshd_config': Operation not permitted

sudo chattr -i /etc/ssh/sshd_config      # deliberate unlock before changes
sudo chattr +a /var/log/app/audit.log    # append-only log

Capabilities: Root Privilege in Slices

Linux splits root’s power into roughly forty capabilities. Instead of making a binary setuid-root, grant only the capability it needs. A classic example: letting a non-root web or syslog service bind to a port below 1024.

setcap, getcap, and process inspection

# Grant a binary the right to bind low ports
sudo setcap 'cap_net_bind_service=+ep' /opt/collector/bin/collector
getcap /opt/collector/bin/collector
# /opt/collector/bin/collector cap_net_bind_service=ep

# Remove file capabilities
sudo setcap -r /opt/collector/bin/collector

# Inspect a running process
grep Cap /proc/$(pidof collector)/status
capsh --decode=0000000000000400

sudo, Mount Options, and MAC in Brief

sudo: delegate commands, not shells

A least-privilege sudo rule

sudo visudo -f /etc/sudoers.d/netops
#   Cmnd_Alias NETOPS_CMDS = /usr/bin/systemctl restart frr, \
#                            /usr/bin/journalctl -u frr
#   %netops ALL=(root) NETOPS_CMDS

sudo -l -U alice           # what can alice run?

• Grant specific commands to groups, never ALL to individuals.
• Avoid rules that allow editors, pagers, or interpreters (vi, less, python) as root; they all have shell escapes.
• Always edit with visudo so a syntax error cannot lock you out.

Mount options

Check how a path is mounted

findmnt -no OPTIONS /tmp
# rw,nosuid,nodev,noexec,relatime

noexec blocks execution regardless of x bits, nosuid ignores setuid and setgid, and nodev ignores device files. A script that is 755 but lives on a noexec mount will fail with “Permission denied”.

Mandatory Access Control

SELinux (RHEL family) and AppArmor (Ubuntu) enforce policy after DAC passes, and they apply to root. A 777 file can still be denied.

Checking the MAC layer

# SELinux
getenforce
ls -Z /var/www/html/index.html
sudo ausearch -m avc -ts recent
sudo restorecon -Rv /var/www/html

# AppArmor
sudo aa-status
sudo journalctl -k | grep -i 'apparmor="DENIED"'

Step-by-Step Lab: Building a Hardened Shared Directory

Scenario: the NetOps team needs /srv/netops for configs and captures. Requirements:

• Members alice and bob can create and edit each other’s files.
• Nobody can delete a file they did not create.
• New files always land in the netops group with group read/write, regardless of each user’s umask.
• Auditor carol gets read-only access without joining the group.
• A golden baseline config cannot be modified or deleted, even by root, without a deliberate unlock.
• Everyone else gets nothing.

Step 1: Create the group and users

Goal: Establish the principals.

Action: Create the netops group, two members, and an auditor who is not a member.

sudo groupadd netops
sudo useradd -m -s /bin/bash -G netops alice
sudo useradd -m -s /bin/bash -G netops bob
sudo useradd -m -s /bin/bash carol

Verification: id alice should list netops in its groups, and id carol should not.

Step 2: Create the directory with ownership and base mode

Goal: Only root and the netops group can enter the directory; other has no access.

Action: Create the path, set ownership to root:netops, and apply mode 0770.

sudo mkdir -p /srv/netops
sudo chown root:netops /srv/netops
sudo chmod 0770 /srv/netops

Verification: ls -ld /srv/netops shows drwxrwx--- root netops.

Step 3: Add setgid and sticky

Goal: Force group inheritance and prevent cross-user deletion.

Action: Add setgid (2) and sticky (1) for a final mode of 3770.

sudo chmod 3770 /srv/netops
ls -ld /srv/netops
# drwxrws--T 2 root netops 4096 Sep 29 08:20 /srv/netops

Verification: The group slot shows lowercase s (setgid with execute). The other slot shows uppercase T because sticky is set while other has no execute. That uppercase T is expected and correct here.

Step 4: Apply a default ACL so umask cannot break collaboration

Goal: Guarantee group read/write on every new file, no matter whose umask created it.

Action: Set default ACL entries for owner, group, and other on the directory.

sudo setfacl -d -m u::rwx,g::rwx,o::--- /srv/netops

Verification: getfacl /srv/netops now shows default: lines. A user with umask 077 will still create group-writable files here.

Step 5: Grant the auditor read-only access

Goal: carol can traverse and read, but cannot create or change anything.

Action: Add a named-user access entry and a matching default entry so future files are readable too.

sudo setfacl -m u:carol:rx /srv/netops
sudo setfacl -d -m u:carol:rx /srv/netops

Verification: ls -ld /srv/netops now ends with +, signaling an ACL is present.

Step 6: Lock the golden baseline

Goal: Make the baseline config tamper-resistant.

Action: Create the file, set ownership and mode, then set the immutable attribute.

echo "hostname edge-fw-01" | sudo tee /srv/netops/baseline.conf > /dev/null
sudo chown root:netops /srv/netops/baseline.conf
sudo chmod 0640 /srv/netops/baseline.conf
sudo chattr +i /srv/netops/baseline.conf
lsattr /srv/netops/baseline.conf

Verification: lsattr shows the i flag in the attribute string.

Verification and Validation

Test as each user with sudo -u, which initializes the target user’s groups, so results match a real login.

Inspect the directory ACL

# 1. Directory state
getfacl /srv/netops

Expected success output

# file: srv/netops
# owner: root
# group: netops
# flags: -st
user::rwx
user:carol:r-x
group::rwx
mask::rwx
other::---
default:user::rwx
default:user:carol:r-x
default:group::rwx
default:mask::rwx
default:other::---

Functional tests with expected results

# 2. alice creates a file with a hostile umask; group still gets rw
sudo -u alice bash -c 'umask 077; touch /srv/netops/alice.txt'
ls -l /srv/netops/alice.txt
# -rw-rw----+ 1 alice netops 0 Sep 29 08:31 /srv/netops/alice.txt

# 3. bob can edit alice's file (group rw) ...
sudo -u bob bash -c 'echo "edit by bob" >> /srv/netops/alice.txt' && echo OK
# OK

# 4. ... but cannot delete it (sticky bit)
sudo -u bob rm /srv/netops/alice.txt
# rm: cannot remove '/srv/netops/alice.txt': Operation not permitted

# 5. carol can read ...
sudo -u carol cat /srv/netops/alice.txt
# edit by bob

# 6. ... but cannot write or create
sudo -u carol touch /srv/netops/carol.txt
# touch: cannot touch '/srv/netops/carol.txt': Permission denied

# 7. Even root cannot delete the baseline while it is immutable
sudo rm /srv/netops/baseline.conf
# rm: cannot remove '/srv/netops/baseline.conf': Operation not permitted

Auditing Permissions Across a System

These one-liners belong in every hardening checklist and baseline script. -xdev keeps find on one filesystem so it does not wander into /proc or network mounts.

Permission audit toolkit

# setuid and setgid binaries (compare against a known-good baseline)
sudo find / -xdev -type f \( -perm -4000 -o -perm -2000 \) \
  -exec stat -c '%a %U:%G %n' {} + 2>/dev/null

# World-writable files
sudo find / -xdev -type f -perm -0002 2>/dev/null

# World-writable directories missing the sticky bit
sudo find / -xdev -type d -perm -0002 ! -perm -1000 2>/dev/null

# Orphaned files (owner or group no longer exists)
sudo find / -xdev \( -nouser -o -nogroup \) 2>/dev/null

# Files carrying capabilities
sudo getcap -r / 2>/dev/null

# Files carrying ACLs under a path
sudo getfacl -R -s -p /srv 2>/dev/null | grep '^# file:'

# Immutable or append-only files under /etc
sudo lsattr -R /etc 2>/dev/null | awk '$1 ~ /^[-a-zA-Z]+$/ && $1 ~ /[ia]/'

Critical files and their expected CIS-style modes:

Troubleshooting and Gotchas

1. Permission denied, but the file is 644

Symptom: cat /srv/app/conf/app.conf fails even though the file is world-readable.

Cause: A parent directory is missing x for your class. Traversal is checked on every component of the path.

Diagnose with namei

namei -l /srv/app/conf/app.conf
# f: /srv/app/conf/app.conf
# drwxr-xr-x root root   /
# drwxr-xr-x root root   srv
# drwxr-x--- root appsvc app      <-- no x for other: stop here
# drwxr-xr-x root appsvc conf
# -rw-r--r-- root appsvc app.conf

Resolution: Grant traverse on the blocking directory (chmod o+x, or better, add the user to the group or a setfacl -m u:user:x entry).

2. The ACL is set, but the user still cannot write

Symptom: setfacl -m u:bob:rw file succeeded, yet bob gets “Permission denied”.

Cause: The mask was lowered, usually by a later chmod or by a tool that rewrote group bits.

Diagnose and fix the mask

getfacl file
# user:bob:rw-      #effective:r--
# mask::r--
setfacl -m m::rw file     # restore the mask

Resolution: Restore the mask and stop using chmod on the group class of ACL-managed files. Adjust entries with setfacl instead.

3. Mode bits and ACLs are correct, still denied

Symptom: Everything in DAC looks right, including for root.

Cause: A layer below or beside DAC: immutable attribute, noexec mount, or SELinux / AppArmor.

Check the non-DAC layers

lsattr /path/to/file                        # i or a flags?
findmnt -T /path/to/file -no OPTIONS        # noexec, nosuid, ro?
sudo ausearch -m avc -ts recent             # SELinux denials
ls -Z /path/to/file                         # SELinux context
sudo journalctl -k | grep -i denied         # AppArmor denials

Resolution: chattr -i for attribute locks, move executables off noexec mounts, and fix SELinux labels with restorecon (or semanage fcontext for custom paths) rather than disabling enforcement.

4. Files in a setgid directory have the wrong group

Symptom: Some files in /srv/netops are owned by a user’s private group instead of netops.

Cause: Group inheritance happens at creation. mv within the same filesystem keeps the file’s original group, and tar -p or cp -a restore the original group.

Normalize group ownership

sudo find /srv/netops ! -group netops -exec chgrp netops {} +

Resolution: Normalize with chgrp, and teach users to cp into shared directories instead of mv.

5. A service lost its low-port binding after an upgrade

Cause: File capabilities live in an extended attribute on the binary. Replacing the binary wipes them.

Resolution: Re-apply with setcap, or move the grant into the systemd unit using AmbientCapabilities= so upgrades cannot remove it.

Quick Reference Cheat Sheet

Linux permissions are not hard, they are layered. When access fails, walk the stack in order: path traversal, owner or group or other, ACL mask, attributes, mount options, then MAC. The layer that says no is always one of those, and now you have a command for each.