Linux & Shell Scripting | DevOps Architect

Architecture & How It Works 01.1

Linux is a monolithic kernel with modular drivers. User-space programs interact with the kernel exclusively through system calls. Understanding layers is critical for debugging containers, performance, and security.

graph TD subgraph "User Space" U1[Applications
nginx, node, bash] U2[Libraries
glibc, musl] U3[Shell & Tools] end subgraph "Kernel Space" K1[System Call Interface] K2[Process Scheduler
CFS] K3[Virtual File System
VFS] K4[Memory Manager
MMU + cgroups] K5[Network Stack
iptables/netfilter] end subgraph "Hardware" H1[CPU / Memory] H2[Storage
ext4 / xfs] H3[Network NICs] end U1 -->|syscalls| K1 U2 --> K1 K1 --> K2 & K3 & K4 & K5 K2 --> H1 K3 --> H2 K5 --> H3 style User Space fill:#161b22,stroke:#00f5ff style Kernel Space fill:#0d1117,stroke:#bc13fe

Filesystem Hierarchy

/ is root. Key dirs: /etc (config), /var (logs/var data), /usr (binaries), /home, /proc (kernel state), /sys (devices).

Everything is a file. Devices appear under /dev. Mount points allow attaching filesystems.

Process Model

Every process has PID, PPID, UID/GID, file descriptors, and namespaces. systemd (PID 1) manages services via cgroups and units.

Signals, nice values, and OOM killer are essential for production stability.

Core Components & Concepts 01.2

Domain	Tool / Command	Purpose	Key Example
File Permissions	chmod, chown, umask, getfacl	Control access. 755 vs 600 critical for keys.	`chmod 600 ~/.ssh/id_rsa`
Process Management	ps, top, htop, kill, pkill, nice, systemd	Identify leaks, restart services, control resources.	`systemctl restart nginx`
Networking	ss, ip, netstat, nmap, tcpdump, ufw	Inspect sockets, routes, firewall rules.	`ss -tuln`
Package Management	apt, yum/dnf, snap, dpkg, rpm	Install, upgrade, pin versions reproducibly.	`apt update && apt install -y nginx`
Storage & Disk	df, du, lsblk, fdisk, lsof, find	Diagnose full disks, find space hogs.	`du -sh /* \| sort -h`
Scheduling	cron, at, systemd timers	Automate backups, cleanup, monitoring.	`crontab -e`

Idempotency & Reproducibility

Always prefer declarative tools (systemd units, Ansible) over raw shell for long-lived systems. Use apt-mark hold on critical packages in production.

WSL Hands-On Lab 01.3

Complete WSL2 Production-Style Server Setup

Run these commands in Ubuntu-22.04 or 24.04 WSL2. All commands are copy-paste ready and tested.

Step 1 — Update system and install essentials

terminal — wsl

$ sudo apt update && sudo apt upgrade -y
$ sudo apt install -y curl wget git vim htop tree ufw fail2ban
$ sudo apt install -y build-essential python3-pip

Step 2 — Create a dedicated deploy user + SSH hardening

terminal — wsl

$ sudo adduser --disabled-password --gecos "" deploy
$ sudo usermod -aG sudo deploy
$ sudo mkdir -p /home/deploy/.ssh
$ sudo chmod 700 /home/deploy/.ssh
$ sudo touch /home/deploy/.ssh/authorized_keys
$ sudo chmod 600 /home/deploy/.ssh/authorized_keys
$ sudo chown -R deploy:deploy /home/deploy/.ssh

Step 3 — Create a production disk monitoring script

/usr/local/bin/disk-monitor.sh

#!/bin/bash
# Production disk + memory monitor with alerts
THRESHOLD=80
LOG_FILE="/var/log/disk-alert.log"
HOSTNAME=$(hostname)

check_disk() {
  USAGE=$(df / | awk 'NR==2 {print $5}' | tr -d '%')
  if [ "$USAGE" -gt "$THRESHOLD" ]; then
    MSG="[$HOSTNAME] CRITICAL: / is ${USAGE}% full at $(date)"
    echo "$MSG" | tee -a "$LOG_FILE"
    # Example: send to Slack or email in prod
    # curl -X POST -H 'Content-type: application/json' \
    #   --data "{\"text\":\"$MSG\"}" "$SLACK_WEBHOOK"
  fi
}

check_memory() {
  MEM_USED=$(free | awk '/Mem:/ {printf "%.0f", $3/$2 * 100}')
  if [ "$MEM_USED" -gt 85 ]; then
    echo "[$HOSTNAME] WARNING: Memory at ${MEM_USED}%" | tee -a "$LOG_FILE"
  fi
}

check_disk
check_memory

terminal — wsl

$ sudo cp disk-monitor.sh /usr/local/bin/
$ sudo chmod +x /usr/local/bin/disk-monitor.sh
$ sudo /usr/local/bin/disk-monitor.sh

Step 4 — Schedule with systemd timer (preferred over cron)

terminal — wsl

$ sudo tee /etc/systemd/system/disk-monitor.service > /dev/null << 'EOF'
[Unit]
Description=Disk and Memory Monitor

[Service]
Type=oneshot
ExecStart=/usr/local/bin/disk-monitor.sh
User=root
EOF

$ sudo tee /etc/systemd/system/disk-monitor.timer > /dev/null << 'EOF'
[Unit]
Description=Run disk monitor every 5 minutes

[Timer]
OnBootSec=1min
OnUnitActiveSec=5min
Unit=disk-monitor.service

[Install]
WantedBy=timers.target
EOF

$ sudo systemctl daemon-reload
$ sudo systemctl enable --now disk-monitor.timer
$ sudo systemctl list-timers

Step 5 — Configure UFW firewall + fail2ban basics

terminal — wsl

$ sudo ufw default deny incoming
$ sudo ufw default allow outgoing
$ sudo ufw allow 22/tcp
$ sudo ufw allow 80/tcp
$ sudo ufw allow 443/tcp
$ sudo ufw --force enable
$ sudo ufw status verbose

Lab Checklist — Track Your Progress

Update system and install base packages (curl, git, ufw, fail2ban, htop) Open Lab ↗
Create non-root deploy user with sudo and SSH key auth only
Write and test disk-monitor.sh script (threshold 80%)
Create and enable systemd timer for the monitor (every 5 min)
Harden with UFW: allow 22/80/443, deny all else
Complete SadServers "Santiago" scenario (Apache not serving) SadServers ↗
Complete Killercoda "Linux Basics" and "Taipei" (disk space) Killercoda ↗

Real-World Project — Startup → Enterprise 01.4

Secure a fresh Ubuntu server for Node.js app

Goal: Make a single server production-ready for a Node API behind nginx.

startup-hardening.sh

$ sudo apt install -y nginx nodejs npm
$ sudo ufw allow 'Nginx Full'
$ sudo ufw allow OpenSSH
$ sudo ufw enable
$ sudo systemctl enable --now nginx

# Deploy simple reverse proxy
$ sudo tee /etc/nginx/sites-available/api << 'EOT'
server {
  listen 80;
  server_name _;
  location / {
    proxy_pass http://localhost:3000;
    proxy_set_header Host $host;
  }
}
EOT
$ sudo ln -s /etc/nginx/sites-available/api /etc/nginx/sites-enabled/
$ sudo nginx -t && sudo systemctl reload nginx

Ansible-compatible hardened baseline + monitoring script

Reusable script that can be executed by Ansible later. Includes unattended upgrades and auditd.

sme-baseline.sh

$ sudo apt install -y unattended-upgrades auditd
$ sudo dpkg-reconfigure -plow unattended-upgrades
$ echo "deploy ALL=(ALL) NOPASSWD: /usr/bin/systemctl" | sudo tee /etc/sudoers.d/deploy

Compliance scanning, automated patching, SELinux/AppArmor

Enterprise Controls

Run Lynis daily, integrate with SIEM. Use apt-cacher-ng or satellite for patch management. Enable AppArmor profiles for nginx.

enterprise-compliance

$ sudo apt install -y lynis
$ sudo lynis audit system --quick
$ sudo apt install -y apparmor apparmor-profiles
$ sudo aa-enforce /etc/apparmor.d/usr.sbin.nginx

Troubleshooting Guide 01.5

diagnosis

$ df -h
$ sudo du -ah / | sort -rh | head -20
$ sudo find /var/log -type f -name "*.log" -mtime +7 -delete

Use htop (install if missing). Identify PID, renice or kill.

$ htop
$ sudo kill -9 <PID>
$ systemctl status <service>

$ sudo systemctl status nginx
$ sudo journalctl -u nginx -xe --no-pager | tail -50
$ sudo nginx -t

✓ Check: sudo systemctl status ssh
✓ Check: sudo ufw status — port 22 open?
✓ Check: tail -f /var/log/auth.log for PAM or key errors
✓ Fix: sudo chmod 600 ~/.ssh/authorized_keys

30-Day Learning Roadmap 01.6

WEEK 1

Filesystem & Permissions Mastery

Navigate and explain /proc /sys /etc
Master chmod, chown, umask, ACLs
Practice find + xargs + rsync
Complete SadServers Santiago

WEEK 2

Processes, Networking & Systemd

ps, top, lsof, ss deep dive
systemd units, timers, sockets
ufw + nftables basics
Write first monitoring script

WEEK 3

Shell Scripting & Automation

Bash: arrays, functions, traps
Write idempotent scripts
Systemd timers vs cron
Log rotation + alerting

WEEK 4

Security, Hardening & Compliance

SSH key-only + fail2ban
Lynis + CIS benchmarks
AppArmor / SELinux profiles
Enterprise patch automation

🐧Linux & Shell Scripting

Architecture & How It Works 01.1

Filesystem Hierarchy

Process Model

Core Components & Concepts 01.2

WSL Hands-On Lab 01.3

Complete WSL2 Production-Style Server Setup

Step 1 — Update system and install essentials

Step 2 — Create a dedicated deploy user + SSH hardening

Step 3 — Create a production disk monitoring script

Step 4 — Schedule with systemd timer (preferred over cron)

Step 5 — Configure UFW firewall + fail2ban basics

Lab Checklist — Track Your Progress

Real-World Project — Startup → Enterprise 01.4

Secure a fresh Ubuntu server for Node.js app

Ansible-compatible hardened baseline + monitoring script

Compliance scanning, automated patching, SELinux/AppArmor

Troubleshooting Guide 01.5

30-Day Learning Roadmap 01.6