Bugzilla – Bug 16011
After ISO Installation, Casting Fails For Downloaded Source Packages Compressed w/ xz
Last modified: 2016-08-02 11:42:12 UTC
Issue arose after ISO installation using 0.10.0 ISO. Following post-installation instructions (http://sourcemage.org/Install/ISO), I found that some spells were failing early in the casting process. After troubleshooting on my own, and with the assistance of several individuals from the #sourcemage channel, it was determined that 'guess_compressor' functionality within Sorcery's mechanics was not able to determine the correct compressor for a *.xz file. A copy of the sorcery debug log file of the failed cast will be forthcoming for reference. I am unable to easy pull the log file off of the VM until tomorrow morning. This will help confirm the guess_compressor behavior in the prior paragraph. More specifically, the cause for the guess not working as expected is beleived to be from an old version of the 'file' spell being present on the ISO. Information provided by a 'gaze version file' command provides the following: Installed version: 4.26 Grimoire version: 5.23 I am currently unable to confirm if the newer 'file' version works as suggested in #sourcemage, due to 'file' spell having its own problems with casting not related to this issue. Once I have a newer version of 'file' installed, I will be able to confirm whether an updated version of 'file' will definitively resolve teh issue or not. Until then, if there is anyone that would care to downgrade their version of file to test whether xz is not recognizable with the older version, please feel free. On a separate, but related note, the bug severity was set at Major due to the fact that this does appear to severely hamper the abilities of casting spells that are compressed into *.xz files. however, I did not set the severity higher as there is likely a workaround. While I have not confirmed this, disabling the sorcery option to clean up the source directory upon failure, and manually decompressing the files may provide a work around for this issue as an interim solution.
Created attachment 7583 [details] Sorcery debug log file of 'cast patch'. Line 727 shows <$COMPRESSOR> is set to <data>, expected value should be <xz>. Determined that 'file' is providing the wrong output ("data"). Will attempt to update 'file' spell to a more current version, >= 5.x.
Confirmed that updating 'file' resolves the issue. Version of 'file' used was 5.23, the current version as of 5/12/2016 in the stable grimoire. Once 'file' is updated, its output for a *.xz file is no longer "data" but instead displays "XZ" as expected. This situation will require the appropriate version of file be installed in any future released (e.g. - ISOs) to insure that XZ-compressed source files are handled appropriately.
Since we must write code to parse the output of the `file` utility, for each new format, the easiest way to fix this issue once and for all is to parse the file headers directly instead. All the formats we support can be detected by matching a short string at the beginning of the files, and both AWK implementations available in Source Mage (gawk and mawk) support parsing binary data, so we can write a viable implementation that adds no further requirements upon sorcery, and which is actually more efficient and simpler.
Created attachment 7584 [details] Prototype implementation of compressor-guessing script
Created attachment 7585 [details] Bash version of guess-compressor.sh Wrote a version in bash using od(1) from coreutils to get the file header in hex
We talked about this on sm-discuss. I don't see any benefit in externalising or obfuscating the checker. It would be hard to deliver an update in any case, so I consider this an ISO bug. However, we can make the function more robust. If we detect unknown binary data, we can fallback to guessing from the filename. Not perfectly secure, but I'm not sure it matters.