da Planet Debian Administration

If
the root filesystem has errors Linux will remount it read-only to
prevent further corruption of the filesystem (it is an option in
/etc/fstab to the mount of the root filesystem, but probably an option
worth keeping!).

In some cases you can’t read anything useful
from the root disk when this happens. i.e. confused RAID hardware. This
prevent you using tools like; ls, fsck, mount, shutdown.

In
these cases “ssh hostname” fails “Connection closed by foreign host”
because important files can’t be read when logging in. You can use “ssh
-v” to see that it authenticates, and then fails running the login
shell.

Get a root shell somehow, this worked for me:

“ssh hostname sh”

….Fumbles around in the dark….

init 6
sh: line 4: /sbin/init: cannot execute binary file

echo “1” >/proc/sys/kernel/sysrq

echo “b” > /proc/sysrq-trigger

Since
echo is a shell built in, and “/proc” isn’t an on disk filesystem, this
worked. You might want to use “s” for sync before “b” depending if you
think this will improve the state of your disks.

Read these first:

http://en.wikipedia.org/wiki/Magic_SysRq_key
http://wiki.openvz.org/Magic_SysRq_Key

The
trick probably only works if the Bash sh (or other shell) is cached in
the file system cache. Since I can’t see why “sh” should be any easier
to read than “shutdown” or “init”, but for whatever reason I could get
a root shell if not a root login shell.

Thanks to Robert for
mentioning the proc interface to sysrq when I was still drinking my
first cup of coffee (decaf alas), and pondering the drive to the server
room. I’d buy him a beer but Canada is an even longer drive than the
server room.

If this is useful to you, then you owe me a tip on persuading the boss to buy IP based KVM, and a new server.

Remote
sysrq can be invoked by the sysrqd – which is probably more reliable,
but well it hadn’t come up before so wasn’t running.

Advertisements