NAME
Convert::UUlib - decode uu/xx/b64/mime/yenc/etc-encoded data from a
massive number of files
SYNOPSIS
use Convert::UUlib ':all';
# read all the files named on the commandline and decode them
# into the CURRENT directory. See below for a longer example.
LoadFile $_ for @ARGV;
for my $uu (GetFileList) {
if ($uu->state & FILE_OK) {
$uu->decode;
print $uu->filename, "\n";
}
}
DESCRIPTION
This module started as an interface to the uulib/uudeview library by
Frank Pilhofer that can be used to decode all kinds of usenet (and
other) binary messages.
After upstream abondoned the project, th library was continuously
bugfixed and improved in this module, with major focuses on security
fixes, correctness and speed (that does not mean that this library is
considered safe with untrusted data, but it surely is safer than the
poriginal uudeview).
Read the file doc/library.pdf from the distribution for in-depth
information about the C-library used in this interface, and the rest of
this document and especially the non-trivial decoder program at the end.
EXPORTED CONSTANTS
Action code constants
ACT_IDLE we don't do anything
ACT_SCANNING scanning an input file
ACT_DECODING decoding into a temp file
ACT_COPYING copying temp to target
ACT_ENCODING encoding a file
Message severity levels
MSG_MESSAGE just a message, nothing important
MSG_NOTE something that should be noticed
MSG_WARNING important msg, processing continues
MSG_ERROR processing has been terminated
MSG_FATAL decoder cannot process further requests
MSG_PANIC recovery impossible, app must terminate
Options
OPT_VERSION version number MAJOR.MINORplPATCH (ro)
OPT_FAST assumes only one part per file
OPT_DUMBNESS switch off the program's intelligence
OPT_BRACKPOL give numbers in [] higher precendence
OPT_VERBOSE generate informative messages
OPT_DESPERATE try to decode incomplete files
OPT_IGNREPLY ignore RE:plies (off by default)
OPT_OVERWRITE whether it's OK to overwrite ex. files
OPT_SAVEPATH prefix to save-files on disk
OPT_IGNMODE ignore the original file mode
OPT_DEBUG print messages with FILE/LINE info
OPT_ERRNO get last error code for RET_IOERR (ro)
OPT_PROGRESS retrieve progress information
OPT_USETEXT handle text messages
OPT_PREAMB handle Mime preambles/epilogues
OPT_TINYB64 detect short B64 outside of Mime
OPT_ENCEXT extension for single-part encoded files
OPT_REMOVE remove input files after decoding (dangerous)
OPT_MOREMIME strict MIME adherence
OPT_DOTDOT ".."-unescaping has not yet been done on input files
OPT_RBUF set default read I/O buffer size in bytes
OPT_WBUF set default write I/O buffer size in bytes
OPT_AUTOCHECK automatically check file list after every loadfile
Result/Error codes
RET_OK everything went fine
RET_IOERR I/O Error - examine errno
RET_NOMEM not enough memory
RET_ILLVAL illegal value for operation
RET_NODATA decoder didn't find any data
RET_NOEND encoded data wasn't ended properly
RET_UNSUP unsupported function (encoding)
RET_EXISTS file exists (decoding)
RET_CONT continue -- special from ScanPart
RET_CANCEL operation canceled
File States
This code is zero, i.e. "false":
UUFILE_READ Read in, but not further processed
The following state codes are or'ed together:
FILE_MISPART Missing Part(s) detected
FILE_NOBEGIN No 'begin' found
FILE_NOEND No 'end' found
FILE_NODATA File does not contain valid uudata
FILE_OK All Parts found, ready to decode
FILE_ERROR Error while decoding
FILE_DECODED Successfully decoded
FILE_TMPFILE Temporary decoded file exists
Encoding types
UU_ENCODED UUencoded data
B64_ENCODED Mime-Base64 data
XX_ENCODED XXencoded data
BH_ENCODED Binhex encoded
PT_ENCODED Plain-Text encoded (MIME)
QP_ENCODED Quoted-Printable (MIME)
YENC_ENCODED yEnc encoded (non-MIME)
EXPORTED FUNCTIONS
Initializing and cleanup
Initialize is automatically called when the module is loaded and
allocates quite a small amount of memory for todays machines ;) CleanUp
releases that again.
On my machine, a fairly complete decode with DBI backend needs about
10MB RSS to decode 20000 files.
CleanUp
Release memory, file items and clean up files. Should be called
after a decoidng run, if you want to start a new one.
Setting and querying options
$option = GetOption OPT_xxx
SetOption OPT_xxx, opt-value
See the "OPT_xxx" constants above to see which options exist.
Setting various callbacks
SetMsgCallback [callback-function]
SetBusyCallback [callback-function]
SetFileCallback [callback-function]
SetFNameFilter [callback-function]
Call the currently selected FNameFilter
$file = FNameFilter $file
Loading sourcefiles, optionally fuzzy merge and start decoding
($retval, $count) = LoadFile $fname, [$id, [$delflag, [$partno]]]
Load the given file and scan it for encoded contents. Optionally tag
it with the given id, and if $delflag is true, delete the file after
it is no longer necessary. If you are certain of the part number,
you can specify it as the last argument.
A better (usually faster) way of doing this is using the
"SetFNameFilter" functionality.
$retval = Smerge $pass
If you are desperate, try to call "Smerge" with increasing $pass
values, beginning at 0, to try to merge parts that usually would not
have been merged.
Most probably this will result in garbled files, so never do this by
default, except:
If the "OPT_AUTOCHECK" option has been disabled (by default it is
enabled) to speed up file loading, then you *have* to call "Smerge
-1" after loading all files as an additional pre-pass (which is
normally done by "LoadFile").
$item = GetFileListItem $item_number
Return the $item structure for the $item_number'th found file, or
"undef" of no file with that number exists.
The first file has number 0, and the series has no holes, so you can
iterate over all files by starting with zero and incrementing until
you hit "undef".
This function has to walk the linear list of fils on each access, so
if you want to iterate over all items, it is usually faster to use
"GetFileList".
@items = GetFileList
Similar to "GetFileListItem", but returns all files in one go, which
is very much faster for large number of items, and has no drawbacks
when used for a small number of items.
Decoding files
$retval = $item->rename ($newname)
Change the ondisk filename where the decoded file will be saved.
$retval = $item->decode_temp
Decode the file into a temporary location, use "$item->infile" to
retrieve the temporary filename.
$retval = $item->remove_temp
Remove the temporarily decoded file again.
$retval = $item->decode ([$target_path])
Decode the file to its destination, or the given target path.
$retval = $item->info (callback-function)
Querying (and setting) item attributes
$state = $item->state
$mode = $item->mode ([newmode])
$uudet = $item->uudet
$size = $item->size
$filename = $item->filename ([newfilename})
$subfname = $item->subfname
$mimeid = $item->mimeid
$mimetype = $item->mimetype
$binfile = $item->binfile
Information about source parts
$parts = $item->parts
Return information about all parts (source files) used to decode the
file as a list of hashrefs with the following structure:
{
partno => <integer describing the part number, starting with 1>,
# the following member sonly exist when they contain useful information
sfname => <local pathname of the file where this part is from>,
filename => <the ondisk filename of the decoded file>,
subfname => <used to cluster postings, possibly the posting filename>,
subject => <the subject of the posting/mail>,
origin => <the possible source (From) address>,
mimetype => <the possible mimetype of the decoded file>,
mimeid => <the id part of the Content-Type>,
}
Usually you are interested mostly the "sfname" and possibly the
"partno" and "filename" members.
Functions below are not documented and not very well tested - feedback welcome
QuickDecode
EncodeMulti
EncodePartial
EncodeToStream
EncodeToFile
E_PrepSingle
E_PrepPartial
EXTENSION FUNCTIONS
Functions found in this module but not documented in the uulib
documentation:
$msg = straction ACT_xxx
Return a human readable string representing the given action code.
$msg = strerror RET_xxx
Return a human readable string representing the given error code.
$str = strencoding xxx_ENCODED
Return the name of the encoding type as a string.
$str = strmsglevel MSG_xxx
Returns the message level as a string.
SetFileNameCallback $cb
Sets (or queries) the FileNameCallback, which is called whenever the
decoding library can't find a filename and wants to extract a
filename from the subject line of a posting. The callback will be
called with two arguments, the subject line and the current
candidate for the filename. The latter argument can be "undef",
which means that no filename could be found (and likely no one
exists, so it is safe to also return "undef" in this case). If it
doesn't return anything (not even "undef"!), then nothing happens,
so this is a no-op callback:
sub cb {
return ();
}
If it returns "undef", then this indicates that no filename could be
found. In all other cases, the return value is taken to be the
filename.
This is a slightly more useful callback:
sub cb {
return unless $_[1]; # skip "Re:"-plies et al.
my ($subject, $filename) = @_;
# if we find some *.rar, take it
return $1 if $subject =~ /(\w+\.rar)/;
# otherwise just pass what we have
return ();
}
LARGE EXAMPLE DECODER
The general workflow for decoding is like this:
1. Configure options with "SetOption" or "SetXXXCallback".
2. Load all source files with "LoadFile".
3. Optionally "Smerge".
4. Iterate over all "GetFileList" items (i.e. result files).
5. "CleanUp" to delete files and free items.
What follows is the file "example-decoder" from the distribution that
illustrates the above worklfow in a non-trivial example.
#!/usr/bin/perl
# decode all the files in the directory uusrc/ and copy
# the resulting files to uudst/
use Convert::UUlib ':all';
sub namefilter {
my ($path) = @_;
$path=~s/^.*[\/\\]//;
$path
}
sub busycb {
my ($action, $curfile, $partno, $numparts, $percent, $fsize) = @_;
$_[0]=straction($action);
print "busy_callback(", (join ",",@_), ")\n";
0
}
SetOption OPT_RBUF, 128*1024;
SetOption OPT_WBUF, 1024*1024;
SetOption OPT_IGNMODE, 1;
SetOption OPT_IGNMODE, 1;
SetOption OPT_VERBOSE, 1;
SetOption OPT_AUTOCHK, 0;
# show the three ways you can set callback functions. I normally
# prefer the one with the sub inplace.
SetFNameFilter \&namefilter;
SetBusyCallback "busycb", 333;
SetMsgCallback sub {
my ($msg, $level) = @_;
print uc strmsglevel $_[1], ": $msg\n";
};
# the following non-trivial FileNameCallback takes care
# of some subject lines not detected properly by uulib:
SetFileNameCallback sub {
return unless $_[1]; # skip "Re:"-plies et al.
local $_ = $_[0];
# the following rules are rather effective on some newsgroups,
# like alt.binaries.games.anime, where non-mime, uuencoded data
# is very common
# if we find some *.rar, take it as the filename
return $1 if /(\S{3,}\.(?:[rstuvwxyz]\d\d|rar))\s/i;
# one common subject format
return $1 if /- "(.{2,}?\..+?)" (?:yenc )?\(\d+\/\d+\)/i;
# - filename.par (04/55)
return $1 if /- "?(\S{3,}\.\S+?)"? (?:yenc )?\(\d+\/\d+\)/i;
# - (xxx) No. 1 sayuri81.jpg 756565 bytes
# - (20 files) No.17 Roseanne.jpg [2/2]
return $1 if /No\.[ 0-9]+ (\S+\....) (?:\d+ bytes )?\[/;
# try to detect some common forms of filenames
return $1 if /([a-z0-9_\-+.]{3,}\.[a-z]{3,4}(?:.\d+))/i;
# otherwise just pass what we have
()
};
# now read all files in the directory uusrc/*
for (<uusrc/*>) {
my ($retval, $count) = LoadFile ($_, $_, 1);
print "file($_), status(", strerror $retval, ") parts($count)\n";
}
Smerge -1;
SetOption OPT_SAVEPATH, "uudst/";
# now wade through all files and their source parts
for my $uu (GetFileList) {
print "file ", $uu->filename, "\n";
print " state ", $uu->state, "\n";
print " mode ", $uu->mode, "\n";
print " uudet ", strencoding $uu->uudet, "\n";
print " size ", $uu->size, "\n";
print " subfname ", $uu->subfname, "\n";
print " mimeid ", $uu->mimeid, "\n";
print " mimetype ", $uu->mimetype, "\n";
# print additional info about all parts
print " parts";
for ($uu->parts) {
for my $k (sort keys %$_) {
print " $k=$_->{$k}";
}
print "\n";
}
$uu->remove_temp;
if (my $err = $uu->decode) {
print " ERROR ", strerror $err, "\n";
} else {
print " successfully saved as uudst/", $uu->filename, "\n";
}
}
print "cleanup...\n";
CleanUp;
PERLMULTICORE SUPPORT
This module supports the perlmulticore standard (see
<http://perlmulticore.schmorp.de/> for more info) for the following
functions - generally these are functions accessing the disk and/or
using considerable CPU time:
LoadFile
$item->decode
$item->decode_temp
$item->remove_temp
$item->info
The perl interpreter will be reacquired/released on every callback
invocation, so for performance reasons, callbacks should be avoided if
that is costly.
Future versions might enable multicore support for more functions.
BUGS AND LIMITATIONS
The original uulib library this module uses was written at a time where
main memory of measured in megabytes and buffer overflows as a security
thign didn't exist. While a lot of security fixes have been applied over
the years (includign some defense in depth mechanism that can shield
against a lot of as-of-yet undetected bugs), using this library for
security purposes requires care.
Likewise, file sizes when the uulib library was written were tiny
compared to today, so do not expect this library to handle files larger
than 2GB.
Lastly, this module uses a very "C-like" interface, which means it
doesn't protect you from invalid points as you might expect from "more
perlish" modules - for example, accessing a file item object after
callinbg "CleanUp" will likely result in crashes, memory corruption, or
worse.
AUTHOR
Marc Lehmann <schmorp@schmorp.de>, the original uulib library was
written by Frank Pilhofer <fp@informatik.uni-frankfurt.de>, and later
heavily bugfixed by Marc Lehmann.
SEE ALSO
perl(1), uudeview homepage at <http://www.fpx.de/fp/Software/UUDeview/>.