[nSLUG] traffic shaping and archive networks

George N. White III gnwiii at gmail.com
Sun Nov 23 16:45:18 AST 2008


On Sun, Nov 23, 2008 at 1:06 PM, Ian Campbell <ian at slu.ms> wrote:
> On Sun, Nov 23, 2008 at 12:56:23PM -0400, George N. White III wrote:
>> It seems that traffic shaping is here to stay:
>
> *snipped*
>
> It looks like you (and pc magazine and the globe and mail) only got
> half the story.
>
> http://www.crtc.gc.ca/eng/NEWS/RELEASES/2008/r081120.htm
>
> The CRTC said Bell can keep doing what it's doing because it's not
> shafting its resellers any harder than it's shafting its own
> customers.
>
> ... while I'm not sure I agree with that (what bandwidth is congested
> for ISPs like TekSavvy who aren't going through Bell for anything
> other than the last mile?) ... the CRTC is still going to look into
> traffic shaping in general in July.
>
> So, uh, there's hope yet.
>
>> How do you tell the difference between problems caused by conjestion and the
>> effects of shaping?
>
> You don't. Traffic shaping is just specific congestion, so you can
> make a guess based on whether different traffic types have different
> response times, I guess.
>
>> Have NSLUG users encountered problems that can be attributed to shaping?
>
> I can't say for certain whether it's shaping or if Eastlink just has a
> crappy network, but jitter is a real problem with them.
>
>> Is shaping causing problems for other archive networks?
>>
>> The Comprehensive TeX Archive Network is used with a new
>> package manager, tlmgr, which first downloads a master
>> database that gives the versions of the packages on the
>> server,  If you don't get the database and packages from the
>> same server you can run into difficulties if the two servers
>> have different versions of some packages.  Thsi can  create
>> problems for users with unreliable network access.
>
> I don't see how this would be specifically affected by shaping. That's
> just saying "don't pick two mirrors that are out of sync"

In the past, you could pick a reliable mirror "close" to you in network
distance.  Now, however, many people report that wget's for a list of
packages to be updated are need more retries, but that retries to the
same server generally work.  This is a bit surprising, as my experience
over the years has been that when you can reach a server there is
usually a real problem that takes some time to resolve, so (neglecting
the version issue), it is better to do retries against a list of servers
rather than hitting the same server that just had a problem.   It is
not hard to write a script to scan a list of servers and figure out which
have the latest database and good transfer rates.   For some users,
however, transfer rates and even reachability appear to be highly
variable.

At many large organizations shaping is applied to subnets with Windows
PC's: highest priority to core services like Outlook and file servers, and low
priority to ftp downloads from external sites.  It is faster and more
reliable to download a file from of the servers and then copy it to
the PC, which
avoids shaping.  This suggests a local mirror that can be populated in off
hours, so the PC's never go to the internet.   For  CTAN there are two
directories that warrant local mirroring: TeX Live and MiKTeX, totalling a
few GB.    If you have more than one system using one of these TeX distros,
the mirror reduces overall demand on the server, and if the mirror is
populated outside peak hours, the workload will be "nicer" than if the
individual systems
are updating during working hours.  Some sites find they have to do multiple
restarts of the ftp transfer, and there are concerns that this may be due to
shaping imposed externally.

-- 
George N. White III <aa056 at chebucto.ns.ca>
Head of St. Margarets Bay, Nova Scotia



More information about the nSLUG mailing list