[WBEL-devel] Improving availability and guaranteeing integrity in WBEL ISO downloads

[Anthony Bryan]

Hi,

I was hoping you could check out Metalinks for your ISO downloads.
Metalink is an XML format supported by download managers that lists
all the mirrors & p2p sources you can get a file from, along with
checksums and other stuff. The idea is to make the whole download
process easier, automated, and less error prone. If you include chunk
checksums, ftp/http downloads can be repaired instead of starting over
from the beginning.

I've put metalinks up at
http://www.metalinker.org/samples.html#whitebox - one containing all 4
recent ISOs & checksum file, just like the torrent, and individual
metalinks for each ISO.

The chunk checksums are very important, because w/o them I got errors in the
FTP/HTTP download. There was a lot of corruption downloading the 2nd
ISO, but 3 & 4 also had minor corruption. I ended up using bittorrent
to correct the errors until I could make metalinks w/ the chunk
checksums, but some people can't use p2p where they are. It took me
around 10 hours today to download these 4 ISOs over bittorrent. Once I
created the metalinks, it took me less than an hour to download them.
The corruption of the 2nd ISO was so bad that you might have a mirror
serving up corrupted files - you could probably tell by using aria2 to
download with the metalink, and logging it to see details.

Around 12 download managers, FTP & p2p clients support Metalink. If
you're interested in using Metalink, you can include download tools
like aria2 ( http://aria2.sourceforge.net/ ) in your distribution.
openSUSE, OpenOffice.org, and a bunch of other Linux/BSD distributions
use it.

More info at http://en.wikipedia.org/wiki/Metalink

Here's part of what a .metalink looks like (minus the chunk checksums,
so its reads easier):

<?xml version="1.0" encoding="utf-8"?>
<metalink version="3.0" generator="Metalink Editor version 1.0.0"
xmlns="http://www.metalinker.org/">
  <publisher>
    <name>White Box Linux</name>
    <url>http://www.whiteboxlinux.org/</url>
  </publisher>
  <identity>White Box Enterprise Linux</identity>
  <version>4respin2</version>
  <description>This product is derived from the Free/Open Source
Software made available by Red Hat, Inc but IS NOT produced,
maintained or supported by Red Hat.</description>
  <files>
    <file name="manifestdestiny-respin2-binary-i386-1.iso">
      <size>687329280</size>
      <os>Linux-x86</os>
      <verification>
        <hash type="md5">06aed0c7116a27d36d5997f7a6c7c978</hash>
        <hash type="sha1">9d1b2e897c526e6c642bdc579edd90c5670f88f1</hash>
      </verification>
      <resources>
        <url type="ftp">ftp://ftp.esat.net/mirrors/whiteboxlinux.org//4/en/iso/i386/manifestdestiny-respin2-binary-i386-1.iso</url>
        <url type="ftp"
location="us">ftp://ftp.gtlib.gatech.edu/pub/whitebox/4/en/iso/i386/manifestdestiny-respin2-binary-i386-1.iso</url>
        <url type="ftp">ftp://ftp.kspei.com/mirrors/whiteboxlinux.org/whitebox/4/en/iso/i386/manifestdestiny-respin2-binary-i386-1.iso</url>
        <url type="ftp"
location="us">ftp://ftp.jach.hawaii.edu/pub/linux/whitebox/4/en/iso/i386/manifestdestiny-respin2-binary-i386-1.iso</url>
        <url type="ftp"
location="at">ftp://gd.tuwien.ac.at/opsys/linux/whiteboxlinux//4/en/iso/i386/manifestdestiny-respin2-binary-i386-1.iso</url>
        <url type="ftp"
location="be">ftp://ftp.belnet.be/packages/whiteboxlinux/4/en/iso/i386/manifestdestiny-respin2-binary-i386-1.iso</url>
        <url type="ftp"
location="ca">ftp://mirror.cpsc.ucalgary.ca/mirror/whiteboxlinux.org/4/en/iso/i386/manifestdestiny-respin2-binary-i386-1.iso</url>
        <url type="ftp">ftp://ftp.ale.org/mirrors/whitebox//4/en/iso/i386/manifestdestiny-respin2-binary-i386-1.iso</url>
        <url type="ftp"
location="jp">ftp://ftp.riken.jp/Linux/whitebox/4/en/iso/i386/manifestdestiny-respin2-binary-i386-1.iso</url>
        <url type="ftp"
location="us">ftp://mirror3.cs.wisc.edu/pub/mirrors/linux/whiteboxlinux.org/4/en/iso/i386/manifestdestiny-respin2-binary-i386-1.iso</url>
        <url type="http">http://www.opensourcemirrors.org/whitebox/4/en/iso/i386/manifestdestiny-respin2-binary-i386-1.iso</url>
        <url type="ftp">ftp://mirror.netseoul.com/pub/whitebox/4/en/iso/i386/manifestdestiny-respin2-binary-i386-1.iso</url>
      </resources>
    </file>
  </files>
</metalink>

-- 
(( Anthony Bryan ... Metalink [ http://www.metalinker.org ]
  )) Easier, More Reliable, Self Healing Downloads