r2 - 04 Aug 2008 - 08:29:04 - Main.yfangYou are here: YWiki > 
MyTroubleShooting Web > ReadonlyFilesystem
MyTroubleShooting Web > ReadonlyFilesystemRead Only FileSystem
问题描述
- 环境 ：CentOS4.4
- 时间 ：2008.8.4
- 症状 ：/dev/sda3(/var) 分区突然readonly，导致重要的服务不能进行下去，因为是真服，又刚好赶上奥运不能进机房，不想贸然重启服务器，希望能够远程解决。
问题原因
- 根据以往的经验，一般因为物理原因导致文件系统突然readonly，如果是第一次出现，重启系统恢复的概率大概在90%以上（事实上以前碰到这种情况的都是这么好的）
- 这次检查以后，发现数个分区中只有/var分区readonly，于是先将数据备份到其他分区，之后尝试umount，失败，后面是详细的过程。
处理步骤
- 以下是真实的操作步骤，不过失败了，请不要贸然效仿
[root@server ~]# fuser -k /var /var: No automatic removal. Please use umount /var [root@server ~]# umount /var umount: /var: device is busy umount: /var: device is busy [root@server ~]# umount -l /var umount: l: not found umount: /var: device is busy umount: /var: device is busy [root@server ~]# man umount [root@server ~]# umount -l /var [root@server ~]# df Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda1 5036284 227248 4553204 5% / none 2075648 0 2075648 0% /dev/shm /dev/sda6 10080488 88696 9479724 1% /home /dev/sda2 2016044 253180 1660452 14% /tmp /dev/sda5 10080488 1616424 7951996 17% /usr df: `/var/lib/nfs/rpc_pipefs': No such file or directory [root@server ~]# df Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda1 5036284 227248 4553204 5% / none 2075648 0 2075648 0% /dev/shm /dev/sda6 10080488 88696 9479724 1% /home /dev/sda2 2016044 253180 1660452 14% /tmp /dev/sda5 10080488 1616424 7951996 17% /usr df: `/var/lib/nfs/rpc_pipefs': No such file or directory [root@server ~]# df Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda1 5036284 227248 4553204 5% / none 2075648 0 2075648 0% /dev/shm /dev/sda6 10080488 88696 9479724 1% /home /dev/sda2 2016044 253180 1660452 14% /tmp /dev/sda5 10080488 1616424 7951996 17% /usr df: `/var/lib/nfs/rpc_pipefs': No such file or directory [root@server ~]# df Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda1 5036284 227248 4553204 5% / none 2075648 0 2075648 0% /dev/shm /dev/sda6 10080488 88696 9479724 1% /home /dev/sda2 2016044 253180 1660452 14% /tmp /dev/sda5 10080488 1616424 7951996 17% /usr df: `/var/lib/nfs/rpc_pipefs': No such file or directory [root@server ~]# mount /dev/sda3 /var mount: /dev/sda3 already mounted or /var busy [root@server ~]# mount -t ext3 -o remount,ro /dev/sda3 /var mount: /var not mounted already, or bad option [root@server ~]# mount -t ext3 -o remount,rw /dev/sda3 /var mount: /var not mounted already, or bad option [root@server ~]# mount -o remount,rw /dev/sda3 /var mount: you must specify the filesystem type [root@server ~]# mount -t ext3 -o remount,rw /dev/sda3 /var mount: /var not mounted already, or bad option [root@server ~]# df Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda1 5036284 227248 4553204 5% / none 2075648 0 2075648 0% /dev/shm /dev/sda6 10080488 88696 9479724 1% /home /dev/sda2 2016044 253180 1660452 14% /tmp /dev/sda5 10080488 1616424 7951996 17% /usr df: `/var/lib/nfs/rpc_pipefs': No such file or directory [root@server ~]# fdisk -l Disk /dev/sda: 73.4 GB, 73407820800 bytes 255 heads, 63 sectors/track, 8924 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sda1 * 1 637 5116671 83 Linux /dev/sda2 638 892 2048287+ 83 Linux /dev/sda3 893 6119 41985877+ 83 Linux /dev/sda4 6120 8924 22531162+ 5 Extended /dev/sda5 6120 7394 10241406 83 Linux /dev/sda6 7395 8669 10241406 83 Linux /dev/sda7 8670 8924 2048256 82 Linux swap [root@server ~]# mount /dev/sda1 on / type ext3 (rw) none on /proc type proc (rw) none on /sys type sysfs (rw) none on /dev/pts type devpts (rw,gid=5,mode=620) usbfs on /proc/bus/usb type usbfs (rw) none on /dev/shm type tmpfs (rw) /dev/sda6 on /home type ext3 (rw) /dev/sda2 on /tmp type ext3 (rw) /dev/sda5 on /usr type ext3 (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) [root@server ~]# mkdir /mnt/var [root@server ~]# mount /dev/sda3 /mnt/var/ mount: /dev/sda3 already mounted or /mnt/var/ busy [root@server ~]# mount -f /dev/sda3 /mnt/var/ [root@server ~]# umount /dev/sda3 umount: /dev/sda3: not mounted umount: /dev/sda3: not mounted [root@server ~]# mount /dev/sda3 /var mount: /dev/sda3 already mounted or /var busy [root@server ~]# mount -f -t ext3 -o remount,rw /dev/sda3 /var [root@server ~]# df Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda1 5036284 227252 4553200 5% / none 2075648 0 2075648 0% /dev/shm /dev/sda6 10080488 88696 9479724 1% /home /dev/sda2 2016044 253180 1660452 14% /tmp /dev/sda5 10080488 1616424 7951996 17% /usr df: `/var/lib/nfs/rpc_pipefs': No such file or directory [root@server ~]# fuser -mk /var /var: 1r 1c 1e 2r 2c 3r 3c 4r 4c 5r 5c 6r 6c 7r 7c 8r 8c 9r 9c 10r 10c 11r 11c 12r 12c 13r 13c 14r 14c 15r 15c 36r 36c 37r 37c 38r 38c 39r 39c 40r 40c 59r 59c 60r 60c 61r 61c 62r 62c 63r 63c 207r 207c 301r 301c 301m 303r 303c 303m 304r 304e 320r 320c 321r 321c 332r 332e 333r 333c 334r 334c 335r 335e 347r 347c 479r 479e 480r 480e 506r 506c 506m 508r 508c 508m 509r 509e 535r 535e 536r 536e 657r 657e 683r 683e 684r 684e 831r 831e 832r 832e 1416r 1416c 1416e 1721r 1721c 1807r 1807c 1868r 1868c 1869r 1869c 1870r 1870c 1871r 1871c 2607r 2607c 2607e 2611r 2611c 2611e 2621r 2621c 2621m 2639r 2639c 2639e 2658r 2658e 2688r 2688c 2688m 2771r 2771c 2771m 2780r 2780c 2780m 2854r 2854c 2854m 2873r 2873c 2873m 2882r 2882c 2882m 2891r 2891m 2908r 2908m 2917r 2917c 2917m 2927 2927r 2927c 2927m 3011r 3011c 3011e 3013r 3013c 3013e 3014r 3014c 3014e 3015r 3015c 3015e 3016r 3016c 3016e 3017r 3017c 3017e 3018r 3018c 3018e 3266r 3266c 3266m 3282r 3282c 3282m 3284r 3284c 3284m 3285r 3285e 3314r 3314e 3317r 3317c 3317e 3469r 3469m 3470r 3470m 3560r 3560c 3560m 3561r 3561c 3561m 3562r 3562m 3563r 3563m 3564r 3564m 3565r 3565m 3566 3566r 3566c 3566m 3567r 3567m 3568r 3568m 3570r 3570m 3571r 3571m 3573r 3573m 3574r 3574m 3575r 3575m 3581r 3581m 3582r 3582m 3583r 3583m 3584r 3584m 3587r 3587m 3657r 3657e 3696r 3696c 3696m 5324r 5324c 5324m 6766r 6766c 6766m 7231r 7231c 7231e 7778r 7778c 7778m 8909r 8909m 8921r 8921m 8924r 8924m 8927r 8927m 8931r 8931m 8935r 8935m 8939r 8939m 8947r 8947m 9112r 9112c 9112m 10233r 10233c 10233m 11228r 11228e 13874r 13874c 13900r 13900c 16840r 16840c 16840m 18454r 18454c 18454m 18987r 18987c 18987m 19985r 19985c 19985m 21724r 21724c 21724m 22476r 22476c 22476m 23260r 23260c 23260m 23500r 23500c 23500m 24725r 24725c 24725m 26007r 26007c 26007m 26329r 26329c 26329m 27082r 27082c 27082e 28898r 28898c 28898m 29419r 29419c 29419m 30524r 30524c 30524m 31489r 31489m 31502r 31502m 31516r 31516m 31734r 31734c 31734m 31736r 31736c 31736m 31738r 31738e 31764r 31764e 31765r 31765e 31925r 31925c 31925m 31934r 31934e 31935r 31935e 31975r 31975m 31988r 31988m 31992r 31992m 32443r 32443c 32443m 32445r 32445c 32445m 32446r 32446e 32503r 32503c 32503m 32539r 32539c 32539m 32541r 32541c 32541m 32542r 32542e # 到这里就断了，想必sshd的服务也受牵连了，此时可以ping通，却再不能远程连接了 # 说到这里真是抱歉，对不起兄弟，最后还是联系机房重启了服务器。好了
写在最后
- 整个过程应该说没搞好，害了兄弟。如果以后再遇到类似的情形，建议考虑下面的步骤
- 备份重要数据，如果本地所有分区都readonly了，那就远程备份
- 尝试umount有问题的分区
- 如果失败，尝试mount -o remount,ro /dev/sda3 /var 这样的remount
- 如果还是失败，而且对应分区中不含有系统重要服务的数据，比如/var，/ 这样的尽量不要动，如果是自己分的专门放静态文件的地方，那就fuser强行杀掉使用该分区中文件的进程
- 否则，最安全的做法还是reboot，一般会活过来的，但是因为中间可能会有fsck的过程，所以如果慢一点不要担心，能活过来就好
- 最后一条，上面的做法是远程操作推荐的，如果能够直接摸到机器，reboot也无妨了!
MyTroubleShooting Web
Create New Topic
Index
Search
Changes
Notifications
Statistics
Preferences
MyTech |
|
Copyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors. Ideas, requests, problems regarding YWiki? Send feedback
