System time freezes??

Hi,

On one particular server, I've been experiencing strange system lockups
for months now, with all kinds of different linux versions. Applications
lock up one by one, until after maybe an hour or two (depending on
usage) the entire system is deadlocked. This happens maybe once in a
month, and I found no way to trigger it.
Now after activating network logging, I found that, before the latest
lockup, the system clock simply stopped after netdate set it 2 seconds back.
The system then kept on running for a while but by the time users wanted
to access data it was completely locked up.

Now what can this be? Can it be hardware? How does the kernel maintain
the system time? How would you go about debugging this? I'm out of ideas.

--
Victor Mataré
Server- & Netzwerk-Administration

Lehrstuhl für Ingenieurgeologie und Hydrogeologie der RWTH-Aachen
Lochnerstraße 4-20, 52064 Aachen

Tel: 0241 80 96778
Fax: 0241 80 92280

Victor Matare wrote:

> Hi,
>
> On one particular server, I've been experiencing strange system lockups
> for months now, with all kinds of different linux versions. Applications
> lock up one by one, until after maybe an hour or two (depending on
> usage) the entire system is deadlocked. This happens maybe once in a
> month, and I found no way to trigger it.
> Now after activating network logging, I found that, before the latest
> lockup, the system clock simply stopped after netdate set it 2 seconds
> back. The system then kept on running for a while but by the time users
> wanted to access data it was completely locked up.
>
> Now what can this be? Can it be hardware? How does the kernel maintain
> the system time? How would you go about debugging this? I'm out of ideas.
>

I recommend that you run a memtest program on the machine for about 4 or 5
hours, if you can spare that time.

It sounds as though you will need a kernel debugger if the machine passes a
long memory test. There have been 2 major kernel debuggers, and I believe
that one was merged with the official Linux kernel tree. From what I
understand you may need a null modem cable, or an ethernet connection
between the computer that is having problems, and another computer that you
can use the debugger with. You will probably also need to build a custom
kernel. I don't have experience with the Linux kernel debuggers, because my
usage of kernel debuggers was primarily with OpenBSD and NetBSD (which
include ddb by default), but I am aware of kgdb and kdb for Linux from
articles I have read.

If the problem is in the software for the kernel, it's most likely a driver.
The driver may have incorrect locking, which may lead to the deadlock, or a
race condition. You may find that the dmesg indicates something goes wrong.

The lock usage in a kernel can be difficult to manage, because there may be
a pattern of locks to use when acquiring a locked area or critical region,
and there are often return or error paths between the acquiring of the lock
area, and the unlocking of the lock. This is further complicated by some
kernel functions potentially needing to acquire some of the same locks,
which can lead to deadlock. It gets more complex, because if a driver
originally written to use a lock in code path X works fine one day, it may
be that the path for X later on changes the lock usage, so that X now should
be called with some lock in an unlocked state. This means that to fully
understand the proper locking behavior, you may need to understand every
piece of code that you call, or potentially call. This makes the cost of
maintaining and changing code increase.

-George