gutenberg.org part zwei, a dissection

Now that we, and by "we" I mean "I", have a full copy of the Project Gutenberg archive, the first thing I can do before publishing it is to have a look at what's actually there, keeping in mind the objective of separating the wheat from the chaff. However, before finding out what's there, there's the question of what "finding out what's there" even means: ideally, one'd catalogue everything, which is, I guess, not such a difficult task, given that the archive already contains some metadata; also, it's conceivable that one would want to verify the authenticity of (some of) the writings, which is hard, or maybe downright sisyphean if we're to consider every single text on gutenberg.org¹.

For now, we're going to narrow down the scope of this "finding out what's there" to discovering what file types there are, how many of them and how much they weigh in bytes. To this end, I'm publishing a script called dissect-guten.bash, that, for each file in the gutenberg.org archive:

1. looks at its extension, and
2. if the extension exists, i.e. if the file is of the form $name.$ext, then
3. it increments a counter num[$ext], and
4. it adds the size of the file to a byte-counter siz[$ext]

The full code:

#!/bin/bash

declare -A num
declare -A siz

print_data() {
    echo "--------------"
    for key in "${!num[@]}"; do
        echo ${key}:${num[${key}]}:${siz[${key}]}
    done

    exit 1
}

# Print data if we press ctrl+c
trap print_data SIGINT

# Can't use find guten | while read line ... because this will spawn a
# sub-shell where all our updates will be lost.
while read line; do
    fn=$(basename -- $line)
    ext="${fn##*.}"
    if [ ! \( \( "${fn}" = "${ext}" \) -o \( -z "${ext}" \) \) ]; then
        fsize=$(stat -c "%s" $line)
        if [ -z ${num["${ext}"]} ]; then
            num+=(["${ext}"]=1)
            siz+=(["${ext}"]="${fsize}")
        else
            num+=(["${ext}"]=$(echo ${num["${ext}"]} + 1 \
                | BC_LINE_LENGTH=0 bc))
            siz+=(["${ext}"]=$(echo ${siz["${ext}"]} + ${fsize} \
                | BC_LINE_LENGTH=0 bc))
        fi
        >&2 echo "${ext}:${num["${ext}"]}:${siz["${ext}"]}"
    fi
done < <(find guten/)
print_data

The script is unfortunately a set of shitty bash-isms, but otherwise it is fairly short and readable. Running this in a directory where guten/ is present and redirecting standard output to, say, file-types.txt, yields a result that I won't recount in full in the post, but that, in short, consists of a set of lines of the form:

ext:n:siz

where ext is a file extension, n is the number of files bearing said extension and siz is the total size in bytes. We can then e.g.:

$ sort -t: -nk3 file-types.txt > ~/file-types-sorted.txt
$ awk -F':' '{print $1,$2,$3,($3/(1024*1024*1024)),$3/$2/(1024*1024)}' \
    ~/file-types-sorted.txt

and further massage the data set to obtain a fancy graph, such as:

which reveals to us that the biggest offenders are zip archives² (~300GB), mp3 and ogg files (~230GB total), images (~140GB total) and video files (a bit over 20GB), with the rest being pdf, epub, iso etc. and, finally, ~40GB representing actual text files.

Of course, the text files can be further cut by removing e.g. copyright headers³, the illustrations can be given a thorough review to establish what's actually worth keeping, but we'll leave these for the next episodes of this saga.