Linux7 min readPublished: 3 Jul 2024Updated: 9 Nov 2026

Linux System Logs and Where to Find Them

Where every Linux log lives, how to read journalctl and the files in /var/log, and how to find the entry that explains an outage instead of endless noise.

G7Cloud Engineering
Platform team
Share:
Abstract illustration representing linux system logs and where to find them.
  • systemd services log to the journal, which you read with journalctl. Web servers and databases also write their own files in /var/log.
  • journalctl -u nginx -p err --since '1 hour ago' is the shape of nearly every useful log query: service, severity, time.
  • journalctl -b shows the current boot and -b -1 the previous one, which is how you read a crash.
  • Nginx and Apache write two logs: access for every request, error for what went wrong. You usually want the error log.
  • If the journal is not saved to disk, it is lost on reboot. Create /var/log/journal to keep it.

Where are Linux logs stored?

In two places. Services managed by systemd (the program that starts and runs services on most Linux systems) log to the journal, which you read with journalctl. Nginx, Apache and MySQL also write plain text files under /var/log. Knowing which service writes where saves most of the searching.

WhatWhereRead with
systemd services and the kernelThe journal (binary)journalctl
General system messages/var/log/syslog or /var/log/messagesless, tail, grep
Logins and sudo/var/log/auth.log or /var/log/secureless, grep
Nginx requests/var/log/nginx/access.logawk, grep
Nginx errors/var/log/nginx/error.logtail -f
Apache/var/log/apache2/ or /var/log/httpd/tail -f
MySQL and MariaDB/var/log/mysql/ or /var/log/mariadb/less, mysqldumpslow
PHP-FPM/var/log/php8.3-fpm.logtail -f
WordPress debugwp-content/debug.logtail -f

Debian and Ubuntu use syslog, auth.log and apache2. The RHEL family uses messages, secure and httpd.

Find what is actually there
# Largest log files, which is where the interesting things usually are
sudo du -sh /var/log/* | sort -rh | head -20

# Which logs were written to most recently?
sudo ls -lt /var/log/ | head -20

# Which log file does a running process have open?
sudo lsof -p $(pgrep -f 'nginx: master') | grep -i log
The last command settles it when a service writes somewhere unusual and you cannot find its log.

How do I use journalctl?

journalctl is the most useful troubleshooting command on a modern Linux server. Nearly every query combines three filters: which service, how severe, and what time range.

The queries you will actually use
# One service, follow live
sudo journalctl -u nginx -f

# Errors only, last hour
sudo journalctl -u nginx -p err --since '1 hour ago'

# Everything from this boot
sudo journalctl -b

# The PREVIOUS boot: how you read a crash after a reboot
sudo journalctl -b -1 -p err

# Kernel messages only
sudo journalctl -k

# A time window
sudo journalctl --since '2026-07-29 09:00' --until '2026-07-29 10:00'

# Full detail on one entry, including the fields not normally shown
sudo journalctl -u nginx -o verbose -n 5

# Everything a single process wrote
sudo journalctl _PID=1234
Remember -b -1. After an unexplained reboot, it is the only way to see what happened just before.
Priority-p valueMeans
emerg0System is unusable
alert1Action required immediately
crit2Critical condition
err3Error: the usual starting point
warning4Warning
notice5Normal but notable
info6Informational
debug7Debug detail

-p err also includes everything more severe: crit, alert and emerg.

Your journal may not survive a reboot

On some distributions the journal is kept in /run by default, which is memory. Restart the server and every entry is gone, including whatever explained the crash. Check with journalctl --disk-usage: if it reports a few megabytes in /run, the journal is not being saved.

Make the journal persistent
# Create the directory and restart
sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald

# Confirm
journalctl --disk-usage

# Cap the size in /etc/systemd/journald.conf
#   SystemMaxUse=1G
#   MaxRetentionSec=1month
Do this on every server, before you need it. The entries you want are the ones already lost.

How do I read web server logs?

Nginx and Apache write two separate logs. The access log records every request; use it to study traffic. The error log records what went wrong; read it when something is broken. People often search the wrong one.

The error log
# Follow it while you reproduce the problem
sudo tail -f /var/log/nginx/error.log

# Recent errors only
sudo grep -i 'error' /var/log/nginx/error.log | tail -50

# PHP errors surfacing through the proxy
sudo grep -iE 'PHP (Fatal|Parse|Warning)' /var/log/nginx/error.log | tail -30

# Upstream problems: 502 and 504 originate here
sudo grep -iE 'upstream|connect\(\) failed|timed out' /var/log/nginx/error.log | tail -30
For a 502, the Nginx error log names the upstream (the program behind Nginx) that failed. That is usually PHP-FPM, and its own log has the reason.
The access log
LOG=/var/log/nginx/access.log

# Status code distribution
awk '{print $9}' $LOG | sort | uniq -c | sort -rn

# Every 5xx, most recent last
awk '$9 ~ /^5/' $LOG | tail -30

# Top client IPs
awk '{print $1}' $LOG | sort | uniq -c | sort -rn | head -20

# Top user agents: this is where bots become visible
awk -F'"' '{print $6}' $LOG | sort | uniq -c | sort -rn | head -20

# Requests per hour
awk '{print substr($4,2,14)}' $LOG | sort | uniq -c | tail -24

# URLs consuming the most bandwidth, not just the most hits
awk '{b[$7]+=$10} END {for (u in b) print b[u], u}' $LOG | sort -rn | head -20
The field numbers assume the standard combined log format. If yours is customised, adjust them.
Pro Tip

Add $request_time to your Nginx log format and you can find slow requests straight from the access log. It is one of the most useful one-line changes you can make to a server.

How do I find the cause of a specific problem?

Start from the symptom and go to the log that owns it. This table covers most of what goes wrong on a web server.

SymptomLook hereSearch for
502 Bad Gatewaynginx error.log, then php-fpm logupstream, connect() failed
504 Gateway Timeoutnginx error.logtimed out
White screen of deathphp-fpm log, wp-content/debug.logFatal error
Service will not startjournalctl -u <service> -n 50Read the last lines before it stopped
Server rebooted on its ownjournalctl -b -1 -p errpanic, oom, hardware
Out of memorydmesg -TKilled process
Database connection errorsmysql error log, dmesgcrashed, OOM
Someone logged inauth.log or secureAccepted, sudo

Rows one and three are the most common, and each takes two commands to answer.

Chasing a 502 to its actual cause
# 1. When did it happen?
sudo grep '502' /var/log/nginx/access.log | tail -5

# 2. What did Nginx say at that moment?
sudo grep -A2 -B2 'upstream' /var/log/nginx/error.log | tail -20

# 3. What was PHP-FPM doing? This is where the real answer lives.
sudo grep -iE 'max_children|WARNING|ERROR' /var/log/php8.3-fpm.log | tail -20

# 4. Did the kernel kill something?
sudo dmesg -T | grep -i 'killed process' | tail
Step 3 is usually where it ends: "server reached pm.max_children setting". See tuning PHP-FPM.
Correlating across logs by time
# Everything from every source in one five-minute window
WHEN='2026-07-29 14:30'
sudo journalctl --since "$WHEN" --until "$WHEN:05" --no-pager

grep '29/Jul/2026:14:3' /var/log/nginx/error.log
grep '29/Jul/2026:14:3' /var/log/nginx/access.log | awk '$9 ~ /^5/'
Reading several logs for the same few minutes is what turns a symptom into a cause. One log alone rarely tells the whole story.

How do I stop logs filling the disk?

Logs are the most common cause of a full disk on a web server. A full disk stops the database writing, which takes the site down. logrotate trims the text files and journald caps its own size, but check both rather than assuming.

Rotation and limits
# Is logrotate configured for this service?
ls /etc/logrotate.d/
cat /etc/logrotate.d/nginx

# Test what it would do, without doing it
sudo logrotate -d /etc/logrotate.conf

# Force a rotation now
sudo logrotate -f /etc/logrotate.d/nginx

# Journal size
journalctl --disk-usage
sudo journalctl --vacuum-size=500M
sudo journalctl --vacuum-time=30d
logrotate -d is a dry run. Read it before you trust that rotation works the way you assume.

Truncate a live log, never delete it

Deleting a log file that a process still has open does not free the space: the kernel keeps it in use until that process restarts. sudo truncate -s 0 /var/log/nginx/access.log empties the file and leaves it usable. Find files stuck in that state with sudo lsof +L1.

Logs tell you what happened afterwards. Knowing something is wrong in the first place is a different job: checking CPU, memory and disk usage covers the live view, and monitoring covers the hours nobody is watching. On G7Cloud, uptime is checked every minute and the dashboard shows each site's CPU and memory over the last 7 days. See platform monitoring.

Frequently asked questions

Where are Linux system logs stored?

systemd services and the kernel log to the journal, read with journalctl. Text logs live under /var/log: syslog or messages for general system output, auth.log or secure for logins, and a folder each for Nginx, Apache, MySQL and PHP-FPM.

How do I view logs for a specific service?

Run sudo journalctl -u servicename. Add -f to follow it live, -p err for errors only and --since '1 hour ago' for a time window. For services that write their own files, such as Nginx, read /var/log/nginx/error.log directly.

How do I see logs from before a reboot?

sudo journalctl -b -1 shows the previous boot, and -b -2 the one before that. This only works if the journal is saved to disk. If it lives in /run, it is held in memory and cleared on every restart. Create /var/log/journal to keep it.

What is the difference between the access log and the error log?

The access log records every request with its status code, size and user agent; use it to study traffic and bots. The error log records what went wrong: PHP fatal errors, upstream failures, permission problems. When something is broken, read the error log.

Why did my disk stay full after I deleted a log file?

A process still has the file open, so the kernel keeps the space in use until that process closes it. Use sudo lsof +L1 to find deleted files that are still open, then restart the process. Next time, use truncate -s 0 on live logs instead of rm.

How long should I keep server logs?

30 days covers most troubleshooting. Keep login logs longer, often 90 days, because a break-in is often found weeks after it happened. Set limits yourself in logrotate and journald rather than relying on the defaults.

Put this into practice on G7Cloud

Every site runs in its own dedicated container behind ScaleShield, with daily backups that are restore-tested every night. Start on the free plan, no card needed.

About G7Cloud Engineering

Platform team

Articles written by the engineers who build and run G7Cloud: UK managed hosting and the AI Website Builder. We write about what we operate every day: containers, backups, databases, and the small-business websites that run on them.

More about G7Cloud →