Created attachment 4369 [details]
patch for mountall.sh (lvm fix)

awesome thanks! handing to seth as he tends to take care of the init system.

how safe is --ignorelockingfailure?

Other scripts we have are overzealous about protecting the system, such 
as that we make sure fsck passes before mounting partitions rw.  The 
locking, I figure is similar to fsck's dirty partition check, I would 
think and maybe we want to have a kernel command-line option instead 
like: forcemountonerror=yes that we can use to add the 
--ignorelockingfailure switch and to bypass the fsck process in general.

All I remember about it was that if you don't use it, the script fails with 
another exit code than 0 (not sure which though). It was in the semi-howto I 
followed to get this to work. 
 
Without it you just get a warning message every time, but due to the 
accompanying exit code, it fails. I get an unbootable system as /var /home are 
LVM's here on my production box. 
 
PS. This means I will not be testing any sort of solutions you provide outside 
of the one I have here that works. I don't reboot this box unless absolutely 
necessary! ;-) 
 

man 8 lvm common option excerpt:

       --ignorelockingfailure
              This lets you proceed with read-only metadata  operations  such
              as  lvchange  -ay  and  vgchange -ay even if the locking module
              fails.  One use for this is in a system init script if the lock
              directory is mounted read-only when the script runs.

I take it our /var/lock is read-only when the script runs.  Seems 
perfectly sensible to me.  I'll get this in soon.

testing a patch in devel grimoire

Created attachment 4432 [details]
updated patch for lvm in mountall.sh

the current mountall.sh based on the previous patch doesn't actually work with
LVM.  the attached patch to the mountall.sh in test does the trick and is what
the new sourcemage box is using.  I'll apply this in a few days if no one
complains here.

I don't understand your comments that the previous fix does not work with 
LVM??? I use it and it does work. Further, your patch is just changes outside 
of the actual LVM part of mountall.sh: 
 
<snip> 
if optional_executable /sbin/vgscan && optional_executable /sbin/vgchange ; 
then 
+    echo "Mounting proc file system..." 
+    mount /proc 
+    evaluate_retval || exit 1 
     echo -n "Scanning for and initializing all available LVM volume 
groups..." 
     /sbin/vgscan       --ignorelockingfailure  --mknodes  && 
     /sbin/vgchange -ay --ignorelockingfailure 
<snip> 
 
So I have no problems with it if you do not bork out the LVM stuff! ;-) 
 
P.S. Thanks for checking with the users of this part of mountall.sh though, I 
run production stuff on this box so it does not reboot very often. 

The problem is the order of operations, not the LVM section itself.  The current
order used is:
1. mount root read-only
2. activate lvm
3. check all file systems
4. remount root rw
5. activate swap
6. mount -a

This fails here for a couple reasons:
a. lvm needs to write to /etc/lvm/archive, and can't when root is ro
b. lvm wants /proc mounted first to check /proc/partitions

This patch changes the order to:
1. mount root read-only
2. check root
3. remount root rw
4. mount /proc
5. activate lvm
6. check all non-root fs
7. mount swap
8. mount -a

I'm not sure how yours would be working with what's there unless there's ways to
make lvm not need a writeable /etc or use of /proc (quite possible, I'm a newbie
to lvm), but in any case these changes should be harmless.

Ok after a bit more digging there is one place this could fail, if the system
has no procfs support.  It looks like it might be possible to build LVM without
needing /proc, though I haven't exactly found how yet.  We could probably just
do mount -a -t proc instead of mount /proc to fix this, or at least avoid an
error regardless of how they decided to use procfs.

Well, I have a procfs and it works, hope your fix is sufficiently tested 
before hitting stable. I would really hate to have to go through rescues again 
if this borkes on my production box...  

This stuff has hit my production box months ago... might wanna mark it done?
Over 7 months old, nice to get it outta my list! ;-)

Ok, I will mark it fixed for me! 

And verified by me! 

It is not fixed, as my previous comments noted.  I haven't yet gone back to it
because there were some corner cases to work out and I wasn't going to apply
something half-done to something this critical.  LVM is still a primary concern
for the 0.9.7 ISO release series coming later this year, so it will get
addressed at that time if not before.

Final fix for this is in devel at 75279, anyone interested who has a test system
PLEASE try this out and give feedback so we know nothing was missed when we move
it to test.

Marking FIXED to note the fix is in the tree, shouldn't get VERIFIED until the
fix is in at least test.

Could you mail met the devel version? I would like to see this before it 
migrates to my boxes. 

Created attachment 4657 [details]
initial devel commit of init.d 2.2.0

I'll just attach it here in case others have the same request.	Don't try to
install it anywhere without the module-init-tools tarball attached to bug
10146.

This has been integrated to test.

Sounds like these changes don't necessarily work on initrd / lvm root systems. 
Looking into it...

/proc mounting can go to mountroot.sh before the root fs is mounted ro.

The /etc rw error is that if /etc/lvm doesn't exist, and /etc isn't writeable
when vgscan runs, you get:

"Failed to create LVM2 system dir for metadata backups, config files and
internal cache.  Set environment variable LVM_SYSTEM_DIR to alternative location
or empty string."

I'm looking into the way around this.

(In reply to comment #23)
> 
> I'm looking into the way around this.

Random useless thought: mount a tmpfs and then synchronize it back up and
destroy the tmpfs in the normal mount?
Probably not a useful solution...

It looks like we can turn off the need for /etc on a per-command basis, but I
don't want to do that since it's the default configuration, and the docs
strongly recommend against turning it off.

However, even without that there are issues for things like static /dev. 
Basically we need to do some rather different ordering depending on what the
user has setup:

normal routine:

devices  : (udev|static)
mountroot: (proc)
           (raid)
           root ro
           fsck root
           root rw
modutils : (modules)
mountall : fsck
           mountall

lvm:
- needs /proc
- needs writeable /dev
- needs writeable /etc (in default config)
- may need modules

if using root lvm w/initrd you need:
- udev or /dev on initrd
- /etc on initrd
- no modules or modules on initrd

So basically if they're using lvm for / we can assume they have most of these
things on the initrd and just scan and activate the volumes before we try to
access root.  If they *aren't* using an initrd we should wait to do anything
with lvm until root is rw and modutils has run.

Given that running the lvm commands "early" doesn't actually break anything, I'm
considering spinning lvm off to its own init script and making both mountroot
and mountall try to run it (via WANTS), with some sanity checks to avoid running
it needlessly or twice.  I'm not sure if this is the best approach but we
definitely have some cases where we want it run from mountroot and some where we
want it run from mountall.

Anyone have comments?

As I said on the list, I never observed LVM commands that I moved, to touch
/etc. I only used (successfully) LVM 2. With the mountroot.sh the way it is at
this moment (with LVM calls), my (initrd-booted) system starts without root FS
mounted. You can take a look at relevant scripts in linux-initrd spell, profile
mk_initrd.lvm.

The /etc behaviour is documented in the lvm.conf man page, see the 'archive'
section and the 'archive_dir' and 'backup_dir' options.  I'm assuming they
default to writing this to /etc because it'll be on the root fs and available at
the LVM init stage more often than /var would be.

I think perhaps it only really errors if it can't create the dir at all and not
if the dir is there but currently unwriteable, but I haven't been able to
completely confirm that yet since my test boxes have static dev as well.

This bug existed in the stable-0.3 release

The issues are now in stable-rc and a primary part of the 0.9.7 iso target, so
they're mustfix.  Note again, the current version of init.d in test works for
people with lvm, not not lvm initrds.  The version in devel works for people
with lvm initrds, but not people with lvm without initrds.  The stated target
for 0.9.7 is installer support for lvm not including initrds, but since the
existing stable works with an initrd we shouldn't regress the regular grimoire
support.

Okay I just setup root lvm and the only thing I had to add to mountroot.sh was this:

=================
--- mountroot.sh.old    2007-01-07 15:50:08.000000000 -0800
+++ mountroot.sh        2007-01-07 15:49:54.000000000 -0800
@@ -51,6 +51,12 @@
     raidstart  --all
   fi
 
+  if optional_executable /sbin/dmsetup
+  then
+    echo "Making device mapper nodes..."
+    /sbin/dmsetup mknodes
+  fi
+
   echo "Mounting root file system read only..."
   mount   -n  -o  remount,ro  /
   evaluate_retval || exit 1
==================

as long as the device entries for root in /etc/fstab are in the /dev/mapper dir
then this will create the nodes needed if you want the /dev/<volgroup>/<volume>
syntax we'll have to add the other two commands needed

I wouldn't expect running these commands twice would create ill effects.

I think that starts to work, it needs a few more things:

- It works for udev because you get /proc from init.d/*/devices udev_start(). 
It won't work for static because it won't have /proc.  We should have /proc get
mounted in devices start() so everything has it (and it can come out of
mountall.sh in that case).

- We should add the vgscan and vgchange calls in there, yeah.

- On my local test setups I only try to run the lvm commands in mountroot if I
see /dev is writeable (if [ -w /dev ]), and then have all of them in a
startlvm() function that also sets an LVM_DONE variable.  This means it only
runs once, don't know if that's really necessary or not.

This would still leave us potentially running these commands both in mountroot
and in mountall, since non-initrd setups would possibly need modutils before
being able to run these commands.  This isn't ideal but probably gets us past
"must fix" for the release.

There's a fix based on this in test now, git commit
d799c9f7d0863829c5fb7c701be785216314090c.

Sergey and erics, if you're still reading and want to try this, I'd appreciate
it.  This last bug we've been trying to deal with is one that would affect your
installs so I'd like absolute confirmation it works for you this way.

jeremy,

Thanks for asking, I will be away from the box in question the coming week, will
try to take a look next week and get back to you.

erics

It will probably be in stable by then, but we only know of three systems in the
wild that have LVM roots, and one of those reports success so far.  Just be
cautious upgrading and let us know of any issues.

Do not mark this fixed after integrating, there's more to do on the bug.  But
this much will let us clear 'must fix'.

I've integrated this fix and a few it depended on to stable-rc-0.6.

Jermey,

> Sergey and erics, if you're still reading and want to try this, I'd appreciate
> it.  This last bug we've been trying to deal with is one that would affect your
> installs so I'd like absolute confirmation it works for you this way.
>
I have upgraded with the last stable 0.6 and 0.7, but a reboot of the machine is
not possible for awhile anyway. You will hear from me when it happens either way.

reassign to sm-grimoire-bugs

For me this is fixed... you can close it and mark varified. Thanks.

What's left on this bug now that the original bug reported by erics is verified
as fixed?

This now applies to stable-rc (0.22).