- systemd services log to the journal, read with journalctl. Web servers and databases still write their own files in /var/log.
journalctl -u nginx -p err --since '1 hour ago'is the shape of almost every useful log query: service, priority, time.journalctl -blimits output to the current boot, and-b -1to the previous one, which is how you read a crash.- Nginx and Apache write two logs: access for every request, error for what went wrong. You usually want the error log.
- If the journal is not persistent, everything is lost on reboot. Create /var/log/journal to keep it.
Where are Linux logs stored?
In two places. Anything managed by systemd goes to the binary journal, which you read with journalctl. Applications that predate systemd, including Nginx, Apache and MySQL, still write plain text files under /var/log. You need both, and knowing which is which saves most of the searching.
| What | Where | Read with |
|---|---|---|
| systemd services and the kernel | The journal (binary) | journalctl |
| General system messages | /var/log/syslog or /var/log/messages | less, tail, grep |
| Logins and sudo | /var/log/auth.log or /var/log/secure | less, grep |
| Nginx requests | /var/log/nginx/access.log | awk, grep |
| Nginx errors | /var/log/nginx/error.log | tail -f |
| Apache | /var/log/apache2/ or /var/log/httpd/ | tail -f |
| MySQL and MariaDB | /var/log/mysql/ or /var/log/mariadb/ | less, mysqldumpslow |
| PHP-FPM | /var/log/php8.3-fpm.log | tail -f |
| WordPress debug | wp-content/debug.log | tail -f |
Debian and Ubuntu use syslog, auth.log and apache2. The RHEL family uses messages, secure and httpd.
# Largest log files, which is where the interesting things usually are
sudo du -sh /var/log/* | sort -rh | head -20
# Which logs were written to most recently?
sudo ls -lt /var/log/ | head -20
# Which log file does a running process have open?
sudo lsof -p $(pgrep -f 'nginx: master') | grep -i logHow do I use journalctl?
journalctl is the single most useful troubleshooting command on a modern Linux server. Almost every useful query is a combination of three filters: which service, what severity, and what time range.
# One service, follow live
sudo journalctl -u nginx -f
# Errors only, last hour
sudo journalctl -u nginx -p err --since '1 hour ago'
# Everything from this boot
sudo journalctl -b
# The PREVIOUS boot: how you read a crash after a reboot
sudo journalctl -b -1 -p err
# Kernel messages only
sudo journalctl -k
# A time window
sudo journalctl --since '2026-07-29 09:00' --until '2026-07-29 10:00'
# Full detail on one entry, including the fields not normally shown
sudo journalctl -u nginx -o verbose -n 5
# Everything a single process wrote
sudo journalctl _PID=1234-b -1 is the one worth remembering. After an unexplained reboot it is the only way to see what happened before it.| Priority | -p value | Means |
|---|---|---|
| emerg | 0 | System is unusable |
| alert | 1 | Action required immediately |
| crit | 2 | Critical condition |
| err | 3 | Error: the usual starting point |
| warning | 4 | Warning |
| notice | 5 | Normal but notable |
| info | 6 | Informational |
| debug | 7 | Debug detail |
-p err includes everything more severe as well, so it captures crit, alert and emerg too.
Your journal may not survive a reboot
By default on some distributions the journal lives in /run, which is memory. Restart the server and every log entry is gone, including whatever explained the crash. Check with journalctl --disk-usage: if it reports a few megabytes in /run, it is volatile.
# Create the directory and restart
sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald
# Confirm
journalctl --disk-usage
# Cap the size in /etc/systemd/journald.conf
# SystemMaxUse=1G
# MaxRetentionSec=1monthHow do I read web server logs?
Nginx and Apache write two separate logs. The access log records every request, and is what you analyse for traffic patterns. The error log records what went wrong, and is where you look when something is broken. People often search the wrong one.
# Follow it while you reproduce the problem
sudo tail -f /var/log/nginx/error.log
# Recent errors only
sudo grep -i 'error' /var/log/nginx/error.log | tail -50
# PHP errors surfacing through the proxy
sudo grep -iE 'PHP (Fatal|Parse|Warning)' /var/log/nginx/error.log | tail -30
# Upstream problems: 502 and 504 originate here
sudo grep -iE 'upstream|connect\(\) failed|timed out' /var/log/nginx/error.log | tail -30LOG=/var/log/nginx/access.log
# Status code distribution
awk '{print $9}' $LOG | sort | uniq -c | sort -rn
# Every 5xx, most recent last
awk '$9 ~ /^5/' $LOG | tail -30
# Top client IPs
awk '{print $1}' $LOG | sort | uniq -c | sort -rn | head -20
# Top user agents: this is where bots become visible
awk -F'"' '{print $6}' $LOG | sort | uniq -c | sort -rn | head -20
# Requests per hour
awk '{print substr($4,2,14)}' $LOG | sort | uniq -c | tail -24
# URLs consuming the most bandwidth, not just the most hits
awk '{b[$7]+=$10} END {for (u in b) print b[u], u}' $LOG | sort -rn | head -20Add $request_time to your Nginx log format and you can find slow requests directly from the access log rather than guessing. It is one of the highest-value one-line changes you can make to a server's observability.
How do I find the cause of a specific problem?
Start from the symptom and go to the log that owns it. This table covers most of what actually happens on a web server.
| Symptom | Look here | Search for |
|---|---|---|
| 502 Bad Gateway | nginx error.log, then php-fpm log | upstream, connect() failed |
| 504 Gateway Timeout | nginx error.log | timed out |
| White screen of death | php-fpm log, wp-content/debug.log | Fatal error |
| Service will not start | journalctl -u <service> -n 50 | Read the last lines before it stopped |
| Server rebooted on its own | journalctl -b -1 -p err | panic, oom, hardware |
| Out of memory | dmesg -T | Killed process |
| Database connection errors | mysql error log, dmesg | crashed, OOM |
| Someone logged in | auth.log or secure | Accepted, sudo |
Rows one and three are the two most common and are answered in two commands each.
# 1. When did it happen?
sudo grep '502' /var/log/nginx/access.log | tail -5
# 2. What did Nginx say at that moment?
sudo grep -A2 -B2 'upstream' /var/log/nginx/error.log | tail -20
# 3. What was PHP-FPM doing? This is where the real answer lives.
sudo grep -iE 'max_children|WARNING|ERROR' /var/log/php8.3-fpm.log | tail -20
# 4. Did the kernel kill something?
sudo dmesg -T | grep -i 'killed process' | tail# Everything from every source in one five-minute window
WHEN='2026-07-29 14:30'
sudo journalctl --since "$WHEN" --until "$WHEN:05" --no-pager
grep '29/Jul/2026:14:3' /var/log/nginx/error.log
grep '29/Jul/2026:14:3' /var/log/nginx/access.log | awk '$9 ~ /^5/'How do I stop logs filling the disk?
Logs are the most common cause of a full disk on a web server, and a full disk stops the database writing, which takes the site down. logrotate handles the text files and journald caps itself, but both need checking rather than assuming.
# Is logrotate configured for this service?
ls /etc/logrotate.d/
cat /etc/logrotate.d/nginx
# Test what it would do, without doing it
sudo logrotate -d /etc/logrotate.conf
# Force a rotation now
sudo logrotate -f /etc/logrotate.d/nginx
# Journal size
journalctl --disk-usage
sudo journalctl --vacuum-size=500M
sudo journalctl --vacuum-time=30dlogrotate -d is a dry run. Worth reading before you trust that rotation is configured the way you assume.Truncate a live log, never delete it
Deleting a log file that a process still has open does not free the space: the kernel keeps it allocated until that process restarts. sudo truncate -s 0 /var/log/nginx/access.log empties it while leaving the handle valid. Find files stuck in that state with sudo lsof +L1.
Logs answer what happened after the fact. Knowing that something is wrong in the first place is a different job: checking CPU, memory and disk usage covers the live view, and continuous monitoring covers the part nobody is watching. On G7Cloud, per-site metrics are kept for 90 days and uptime is checked every minute from outside the network, so you can look at the moment something broke rather than reconstructing it. See platform monitoring.
Frequently asked questions
Where are Linux system logs stored?
systemd services and the kernel log to the binary journal, read with journalctl. Text logs live under /var/log: syslog or messages for general system output, auth.log or secure for logins, and per-service directories for Nginx, Apache, MySQL and PHP-FPM.
How do I view logs for a specific service?
sudo journalctl -u servicename, adding -f to follow live, -p err for errors only and --since '1 hour ago' for a time window. For services that write their own files, such as Nginx, read /var/log/nginx/error.log directly.
How do I see logs from before a reboot?
sudo journalctl -b -1 shows the previous boot, -b -2 the one before that. This only works if the journal is persistent: if it lives in /run it is memory-backed and cleared on every restart. Create /var/log/journal to make it survive.
What is the difference between the access log and the error log?
The access log records every request with its status code, size and user agent, and is what you analyse for traffic and bots. The error log records what went wrong: PHP fatals, upstream failures, permission problems. When something is broken, read the error log.
Why did my disk stay full after I deleted a log file?
Because a process still has the file open, so the kernel keeps the space allocated until that process closes it. Use sudo lsof +L1 to find deleted files still held open, then restart the process. In future use truncate -s 0 on live logs instead of rm.
How long should I keep server logs?
Thirty days covers most troubleshooting. Authentication logs are worth keeping longer, often ninety days, since a compromise is frequently discovered weeks after it happened. Set limits explicitly in logrotate and journald rather than relying on the defaults.
About G7Cloud Engineering
Articles written by the engineers who build and run G7Cloud: UK managed hosting and the AI Website Builder. We write about what we operate every day: containers, backups, databases, and the small-business websites that run on them.
More about G7Cloud →
