Overview

Intermittent Linux Server Hang & Crash Diagnostics targets the hardest class of production faults: a server that occasionally freezes or becomes unreachable, a reboot command that does nothing, or an application that stalls for a few minutes and then recovers on its own — often with a gap in the logs and no recurrence for days or weeks. We do not "just reboot it". Our method is protect the scene, preserve evidence, reconstruct the timeline, analyse at kernel level, reproduce and verify, then fix the root cause, turning a random event into an explanation that can be validated and prevented. We cover RHEL / CentOS / Rocky / AlmaLinux, Ubuntu / Debian, Fedora and openSUSE / SLES, plus Chinese domestic platforms including Kylin (Galaxy Kylin V10 and NeoKylin), UnionTech UOS and openEuler, on Kunpeng, Phytium, Hygon, Zhaoxin and Loongson CPUs, across bare metal, virtual machines, private clouds and container hosts.

Symptom Patterns

These patterns frequently disguise one another: an "unexplained reboot" may be a kernel panic, and a "total freeze" may be nothing more than a stuck NFS mount. We first place the event on a timeline, then decide which evidence to capture.

Why Intermittent Faults Are Hard

• They do not reproduce: once every few days or weeks, so manual observation is impractical and the fault is gone by the time someone logs in.
• The scene gets destroyed: the instinct is to reboot, which clears the kernel ring buffer, in-memory task stacks and unflushed logs — the only evidence.
• Log gaps: the system cannot write logs to disk at the instant of a crash, leaving a blank between boots in journalctl.
• No capture capability by default: most production hosts have kdump, pstore, serial logging or NMI watchdog disabled, so a rare crash leaves nothing behind.
• Metrics are too coarse: a five-minute CPU average hides ten seconds of saturation, so the real event is invisible.
• Domestic platforms add an adaptation layer: CPU microcode, proprietary drivers, vendor-customised kernels and display / desktop services mean generic conclusions cannot be copied over.

Our first rule is therefore: preserve the scene before touching the system. Until evidence is captured, no reboot, no upgrade, no parameter change and no log cleanup.

Diagnostic Method: Preserve, Capture, Verify

The conclusion must close the loop: after identifying a candidate root cause, reproduce it in a test environment with the same workload or a SysRq trigger, verify the fix by changing one variable at a time, then observe a sufficiently long production window to confirm it does not recur.

Root Cause Map

Supported Distributions and Domestic Platforms

The method is the same, but tool paths, symbol package sources and vendor support channels differ per distribution. We deliver a runnable, environment-specific evidence-collection checklist rather than a generic tutorial.

Engagement Process

• Step 1 · Scene protection and intake: establish the incident timeline, symptom class, topology and change history; freeze reboots and configuration changes until evidence is captured; record kernel, driver, firmware and hardware details.
• Step 2 · Harden evidence capture: configure and validate kdump with crashkernel, enable pstore / ramoops and persistent journal, set up NMI watchdog and hung-task / soft-lockup panic, enable serial or IPMI SOL / netconsole, and deploy second-level metrics and kernel sampling.
• Step 3 · Preserve evidence and rebuild the timeline: export previous-boot logs, the kernel ring buffer, BMC / SEL events, monitoring curves and release records into a minute- or second-level incident timeline.
• Step 4 · Kernel-level analysis: decode vmcore with crash (log, bt, ps, per-CPU stacks, locks and modules), correlate perf / eBPF samples with metrics and changes, eliminate hypotheses and isolate the root cause.
• Step 5 · Reproduce and fix: reproduce in a test environment with the same workload or a SysRq trigger, validate each fix by changing one variable at a time (kernel, driver, parameter, hardware, configuration, quota, mount policy), then roll out gradually.
• Step 6 · Handover and prevention: deliver the root cause report, evidence package and remediation plan; complete monitoring, watchdog and dump coverage; recommend change-regression checks and periodic drills, with knowledge transfer.

Representative Diagnostic Scenarios

The following are representative patterns grouped by real failure mode, illustrating the evidence path and remediation logic. They are not descriptions of specific customer engagements.

Scenario 1 · A financial host freezing every few weeks

• Symptom: A core business host becomes unreachable at random intervals and only a remote hard reset recovers it; everything looks normal afterwards and the logs show a gap at the failure time.
• Evidence: Prohibit cold boots, enable kdump with reserved crashkernel memory, configure NMI watchdog and panic-triggered dumps; capture vmcore on the next occurrence, inspect log and bt with crash, and use serial / IPMI SOL capture.
• Root cause: The backtrace points to a third-party NIC driver triggering a soft lockup in interrupt handling — a known regression of that driver version on the installed kernel.
• Outcome: Upgrade the driver and pin a vendor-validated kernel, add watchdog fallback and alerting; no recurrence across two full business cycles.

Scenario 2 · reboot hangs and only a power cut works

• Symptom: After a reboot command the system stays in "stopping services" for a long time, SSH drops, and a hard power cycle is needed — usually after off-peak batch jobs.
• Evidence: Use Magic SysRq during the hang to dump task stacks, review D-state tasks, mount points and systemd stop job timeouts, and correlate shared-storage and multipath logs.
• Root cause: A hard NFS mount becomes unreachable when the server misbehaves; tasks enter uninterruptible sleep and cannot be killed even with kill -9, so shutdown never converges.
• Outcome: Evaluate and apply mount timeouts / soft mount policy, decouple critical dependencies, set a sane TimeoutStopSec and add a hardware watchdog fallback; reboots complete normally.

Scenario 3 · Intermittent stalls on domestic platforms

• Symptom: A mixed cluster of Galaxy Kylin V10, UnionTech UOS and openEuler shows several hundred-millisecond stalls per day; the issue cannot be reproduced on generic x86 hosts.
• Evidence: Review kernel and proprietary driver versions, UEFI firmware and display service logs per platform; collect PSI, cgroup cpu.stat throttling counters and off-CPU samples; compare with vendor compatibility lists.
• Root cause: Container CPU quotas sized on average load cause periodic throttling stalls, compounded by a proprietary driver adaptation issue on that domestic CPU platform.
• Outcome: Reset quotas and tune thread pools, apply the vendor driver patch and OS upgrade; stalls disappear and a unified cross-distribution metric baseline is established.

Scenario 4 · Unexplained reboots with a log gap

• Symptom: A cloud host occasionally restarts without warning; journalctl shows two SYSTEM_BOOT entries with no SYSTEM_SHUTDOWN between them, and monitoring records only a brief loss of connectivity.
• Evidence: Enable pstore / ramoops to retain pre-crash kernel messages; export previous-boot logs, IPMI SEL and cloud serial console records; check kernel.panic, watchdog and thermal events.
• Root cause: MCE hardware errors with a rising count of correctable memory ECC errors, ending in an uncorrectable error that triggers a firmware reset.
• Outcome: Identify and replace the failing DIMM, upgrade BMC and BIOS firmware, add ECC / MCE alerting and trend monitoring; no further unplanned reboots.

Deliverables

• Root cause analysis report: symptom classification, evidence chain, elimination process and final conclusion.
• Incident timeline: an aligned view of system logs, metrics, hardware events and business changes within the incident window.
• Evidence package: vmcore / crash analysis notes, log exports, performance samples and flame graphs, plus the command checklist used.
• Remediation and hardening plan: concrete recommendations and change steps for kernel, drivers, parameters, hardware, mount policy and resource quotas.
• Evidence-capture configuration: kdump, pstore, NMI watchdog, serial / netconsole, second-level monitoring and alert rules.
• Prevention guidance and knowledge transfer: change-regression checklists, periodic drills and an internal troubleshooting runbook.

Why Work With Us

• Nearly 20 years of Linux technical services, supporting tens of thousands of enterprises, with deep kernel-level and hardware-level troubleshooting experience.
• Coverage of RHEL, Debian and SUSE families plus Galaxy Kylin, UnionTech UOS and openEuler, with hands-on experience on domestic CPU platforms.
• Evidence first: preserve the scene before changes, and validate conclusions through reproduction and variable isolation rather than guesswork or repeated reboots.
• Senior engineers with 24/7 technical support and fast response to sudden freezes.
• Standardised processes, transparent pricing, disciplined delivery, and reusable in-house troubleshooting capability.