[WBEL-devel] Re: Ideas for improving availability

[Jon Lewis]

On Fri, 3 Dec 2004, John Morris wrote:

> Well since I suspect anyone who is reading -devel could figure it out,
> http://www.whiteboxlinux.org/pub will get you to it via http.  But
> remember that it is at the end of a single T-1 so don't expect a lot of
> throughput.

Funny thing is, when I got this message, I was already in the process of
setting up our mirror (whitebox.atlantic.net).  When I finished and
started comparing things to the NCSU one, I noticed some minor issues.

Both the 'home site' (www.whiteboxlinux.org) and NCSU seem to have their
3.0/en/os/i386 directories based on the original release.  Before noticing
this, I chose to base mine on the liberation-respin1 iso files.  I don't
know if thats going to cause problems for anyone.  I don't expect it
would, but AFAIK, all my installs were done from respin1, so I don't have
a good/easy way to test this.

A related problem is the system I used for the mirror is running an older
Red Hat release that didn't have yum and apparently can't have the latest
yum without some major work being done (upgrading rpm and a bunch of
related apps and libs).  Using yum-1.0.3, I was able to generate the
headers directory (since the isos don't include those files), but that
version of yum won't generate headers for srpms, so my
/3.0/en/os/i386/headers dir lacks srpm headers, which I assume only breaks
installation of srpms via yum.  Probably not a huge deal.

For some reason both the home site and NCSU lack pretty much all the files
from the first iso which would normally be in 3.0/en/os/i386/dosutils.  I
copied those files from the iso into my mirror.

At the moment, I have it setup to rsync updates from NCSU every 2 hours.
I see no reason to run rsync against the distribution files, since it
appears they never change.

It would probably make sense to locate a handful of high-bandwidth (for
some definition of HB) mirrors that would be willing to pull directly from
the home site and make anyone else mirror from one of them, thus creating
two tiers of mirrors.  There's been talk of this a few times on the
redhat-mirror list, but Red Hat just keeps buying more bandwidth and
servers.  That's not really an option for the library.

> Idea #2 of course is to randomize who pulls from who with the obvious
> advantages and disadvantages.

If by randomize, you mean using something like multiple A record DNS to
make mirrors rsync against a different site each time, that can get messy
since not all servers are guaranteed to be in sync at all times or even
use the same paths, so it's generally best to pick a mirror and stick with
it.

> Is there a consensus as to which way would be best?

Does consensus matter?  Isn't this your show? :)

Is this sort of mirror talk more appropriate on the devel list?  Should
there be a mirror list?

BTW, my plan for whitebox.atlantic.net is to use it internally for our
growing number of WBEL servers, and to make it publicly available,
probably rate-limited to some small multiple of 10mbit/s, which may vary
according to how full our transit pipes are at any given time.

----------------------------------------------------------------------
 Jon Lewis                   |  I route
 Senior Network Engineer     |  therefore you are
 Atlantic Net                |
_________ http://www.lewis.org/~jlewis/pgp for PGP public key_________