Legacy Haven University

A compromised Linux server keeps two records of an intrusion. One lives in memory and disappears on reboot. The other sits on disk and can be altered by the same attacker. Linux forensic investigation exists to capture both records, in the right order, before containment destroys the evidence needed for root-cause analysis.

CompTIA SecurityX (CAS-005) places this work inside Security Operations. The objective expects analysts to examine host artifacts, volatile and non-volatile storage, filesystem metadata, and malware indicators, then rebuild a timeline. The same workflow applies on a bare-metal host, a virtual machine, or a container node.

What the investigation has to prove

A finished case answers four questions. How did the attacker get in? What did the process do while it ran? How did it survive reboot? What data or access did it reach?

Those answers do not come from a single log line. They come from cross-checks. Memory is compared with the process list. The process list is compared with files still open on disk. Disk timestamps are compared with authentication logs. A mismatch is often the first reliable signal.

The ultimate guide to CompTIA SecurityX (CAS-005) maps this host-analysis work to the Security Operations domain, where incident artifacts support both detection and root-cause reporting.

Order of volatility decides the first hour

Evidence decays at different speeds. RFC 3227, the standard guidance on evidence collection order, ranks sources from most fragile to most stable. CPU registers and cache vanish first. Random-access memory (RAM) — the working store of running processes — vanishes on shutdown. Network connection tables follow. Disk files last longer, but an active attacker can still change them.

On a live Linux server, the practical order is fixed.

1. Record system time and the offset against a trusted clock.
2. Capture memory to external media.
3. Snapshot network connections, logged-in users, and loaded kernel modules.
4. Copy logs and persistence files that an attacker may delete next.
5. Isolate the host from the network.
6. Image the disk, or preserve the cloud volume snapshot, for offline analysis.

Pulling the cable first looks decisive. It also drops the connection table. That table often holds the only clear link to a command-and-control address. Shutdown first looks clean. It wipes RAM, and many rootkits never write a durable file.

Volatile collection on a host that may be lying

A rootkit is software that hides processes, files, network sockets, or its own kernel code. Once it runs with root, ordinary tools can lie. The ps command reads process data through libraries the attacker may have replaced. ls can skip a directory the kernel has been told to hide.

Investigators treat live command output as a lead, not as proof. They save it, then repeat the question from a path the attacker does not control.

Memory capture is the stronger path. Linux Memory Extractor (LiME) loads a small kernel module and writes RAM to a file. AVML, Microsoft’s Acquire Volatile Memory for Linux, does a similar capture without a custom module build. The output goes to a mounted evidence share, not to the suspect disk.

Offline analysis uses a framework such as Volatility 3. It reads kernel structures from the image rather than from the live system. Useful comparisons include:

• linux.pslist walks the kernel’s active process list.
• linux.psscan carves process objects that were unlinked from that list. A hit in the scan and a miss in the list is a classic hiding technique.
• Syscall-hook checks look for a rootkit that redirected kernel calls so normal tools return filtered results.
• Bash plugins recover command history that the attacker deleted from ~/.bash_history.

A running process whose executable path ends in (deleted) is another high-value artifact. Linux keeps the file contents available through /proc/<pid>/exe until the process exits. Copying that link preserves the binary even after the attacker unlinked it from the directory tree.

Host artifacts that survive a careful attacker

Host analysis under CAS-005 means reading the operating system’s own records, not only the malware file. On Linux, the useful set is small and repeated across intrusions.

Authentication and session logs

/var/log/auth.log or /var/log/secure records SSH logins, sudo use, and failed passwords. wtmp and btmp store login success and failure in binary form. The last and lastb commands decode them. Journald, the systemd logging service, keeps a structured copy that journalctl can export even when plaintext logs were rotated or truncated.

An attacker who passes a stolen key leaves a different trace than one who guesses a password. Key logins show the key fingerprint and source address. Password attacks show bursts of failures, then one success. Both belong in the timeline.

Shell history and staging paths

Bash history is easy to delete and easy to overlook. Attackers clear ~/.bash_history, unset HISTFILE, or write history to /dev/null. Copies still appear in memory, in shell session files under /home, and in audit logs if auditd was recording execve calls. Staging often lands in /tmp, /var/tmp, /dev/shm, or a hidden directory inside a web root. /dev/shm is RAM-backed. Its files vanish on reboot, which is why memory capture cannot wait.

Persistence that does not look like malware

Most Linux intrusions persist with ordinary admin tools. The forensic task is to find the odd entry among legitimate ones.

• System crontab in /etc/crontab and drop-ins under /etc/cron.d.
• Per-user crontabs in /var/spool/cron or /var/spool/cron/crontabs. Checking only /etc misses these.
• Systemd service and timer units in /etc/systemd/system and user units under ~/.config/systemd/user. A unit with Restart=always relaunches a payload after it crashes.
• SSH authorized keys in ~/.ssh/authorized_keys and extra key options that force a command.
• /etc/ld.so.preload, which lists shared libraries the dynamic linker loads into every new process.
• PAM modules — the pluggable authentication modules that run at login — under /etc/pam.d and the security library path.
• Unexpected setuid binaries, flagged in a search for mode 4000, that run as root regardless of the caller.

A cron line that curls a remote script every minute is loud. A systemd unit named like a vendor agent is quieter. An added SSH key is quietest of all, until the source address in auth.log fails to match staff locations.

How Linux rootkits hide, and how analysis sees them

Rootkits fall into three layers. Each layer breaks a different trust assumption.

Userland hooks

A userland rootkit stays outside the kernel. The common method is LD_PRELOAD. The dynamic linker, which loads shared libraries for a program, reads /etc/ld.so.preload and maps the listed file into every dynamically linked process. A malicious library then intercepts libc calls such as readdir and stat. Directory listings skip the malware path. Process tools skip the malware PID.

This design is fragile under examination. The preload file is a single plain-text path. A statically linked binary does not use the dynamic linker, so it does not load the hook. Comparing a static ls with the system ls exposes the filter. Renaming the preload file for a controlled test removes the hook from new processes. That rename is a containment step, and it must be recorded. It changes the system under investigation.

Real campaigns use this pattern. Cryptominers have dropped a library, pointed ld.so.preload at it, and added a cron job plus a systemd unit so the miner returns after reboot. The preload file is the switch. The cron entry is the backup.

Kernel modules

A loadable kernel module (LKM) runs inside kernel space. From there it can unlink a process from the task list, hide a file from directory reads, or remove itself from the module list. lsmod then shows nothing. The module may still appear as a directory under /sys/module, or as a taint flag in /proc/sys/kernel/tainted. A taint value records that the kernel loaded out-of-tree or unsigned code.

Cross-view is the working method. One view comes from the possibly hooked tool. The second view comes from raw kernel structures in a memory image, or from a known-good baseline of /lib/modules. A module present in memory and absent from lsmod is a rootkit candidate. Volatility’s syscall checks look for the same gap at the call table.

Signature scanners such as rkhunter and chkrootkit still have a role. They catch known strings and suspicious preload files. They do not clear a host. An LKM built for that kernel can skip their checks.

Newer hook points

Extended Berkeley Packet Filter (eBPF) programs attach to kernel trace points for legitimate monitoring. The same attach points can hide activity or load a payload. Investigation looks for unexpected eBPF programs, pins under /sys/fs/bpf, and recent package installs that do not match the server role. Firmware and hypervisor hooks are rarer on typical enterprise servers. They matter when disk and memory both look clean and the hardware chain of trust is in doubt.

Non-volatile images and filesystem metadata

After volatile capture, the disk becomes the stable record. A forensic image is a bit-for-bit copy, hashed with SHA-256 at the start and again at the end. Matching hashes show the copy did not change in transit. Tools such as dc3dd or ewfacquire write the image and the hash log together. On a cloud host, a storage snapshot taken before shutdown is the equivalent artifact. Analysis happens on the copy. The original stays untouched.

Filesystem metadata is the clock inside that image. Ext4 stores modified, accessed, and changed times, often called MAC times, plus a creation time on newer layouts. Attackers timestomp files by setting those stamps backward. The inode change time — updated when permissions or ownership change — is harder to fake in a consistent way. Journal records can still show the real write.

The Sleuth Kit’s fls and mactime turn those stamps into a bodyfile, a sortable timeline of file activity. Plaso, also called log2timeline, merges that file timeline with logs, browser artifacts, and shell history into one super-timeline. CAS-005 treats timeline reconstruction as a core incident-response task, not as a reporting extra.

Deleted files are not always gone. Unallocated space can still hold the bytes until a later write reuses the blocks. Carving recovers them without directory entries. A web shell removed from the document root may still sit in free space next to an access-log hit that requested it.

From artifacts to indicators and root cause

Malware analysis in this objective starts with safe detonation and indicator extraction. The recovered binary goes to a sandbox — an isolated system that runs the sample and records its behavior. The run produces indicators of compromise (IoCs): file hashes, dropped paths, contacted domains, mutex names, and command lines.

Those IoCs feed the wider case. A hash hit on a second server shows scope. A domain in the sandbox report that also appears in the captured connection table ties the sample to this host. YARA rules — pattern rules that match bytes or strings — then hunt the same family across images.

Root-cause analysis sits on top of the timeline, not beside it. A typical Linux server chain looks like this.

1. Initial access: exposed application, reused credential, or stolen SSH key.
2. Execution: a shell or dropped binary, often from /tmp or the web root.
3. Persistence: cron, systemd, authorized keys, or ld.so.preload.
4. Defense evasion: history cleared, logs truncated, process unlinked, timestamps edited.
5. Objective: mining, proxying, data theft, or a pivot to the next host.

The report names the first reliable event, the persistence that would have survived reboot, and the gap that allowed the entry. Reimaging is the usual remediation. Cleaning a rooted host in place leaves any missed hook behind. Images and logs remain, because they are what prove the cause.

What CAS-005 expects an analyst to do with this

Security Operations is 22 percent of the SecurityX exam. The incident-response slice does not ask for kernel exploit development. It asks for judgment about artifacts.

• Choose volatile capture before shutdown when a rootkit or a live connection matters.
• Treat host tools as untrusted once kernel or preload tampering is plausible.
• Separate userland hooks, kernel modules, and ordinary persistence. They leave different files.
• Use filesystem metadata and log time to rebuild the sequence.
• Extract IoCs from a detonated sample and apply them to scope the incident.
• Close with root cause, not with a list of deleted files.

Enterprise incident response training builds the same sequence on a lab server: memory image, persistence hunt, preload and module cross-check, then a timeline. The production version adds legal hold, hash custody, and a decision on whether the host returns to service or stays as evidence.

A Linux forensic investigation is finished when the timeline, the memory findings, and the disk metadata agree. Where they disagree, the hidden object is usually the point.