[NNTP] NNTP Compression

[Sabahattin Gucukoglu]

On 10 Jan 2010, at 10:03, Ade Lovett wrote:
> On Jan 09, 2010, at 13:52 , Julien ÉLIE wrote:
>> As it was noted in the previous discussion, clients may want to
>> use compression only for a few commands.  Wouldn't it be wise
>> to have an NNTP command that can both activate and deactivate
>> the compression?
> 
> From a server perspective, absolutely not.  Switching in and out of compression state is simply insane.  Those clients likely to be wanting to use both compressed and uncompressed streams simultaneously will almost certainly already be opening up multiple connections to the server endpoint, and thus can coordinate accordingly.

From where I'm sitting, which is at this very moment trying to find a portable way to implement a version of the GigaNews Accelerator (I'll call it "GigaNoise"), it would be most welcome!  As it stands, I have to track what state the client and server are in, filter out capabilities, and whatnot, and all because the compression isn't clearly and obviously negotiated, and at what point I'll be getting compressed versus regular output (ARTICLE, for example, is still sent uncompressed).  And I'm not sure precisely what it is about toggling compression that's so terrible, because you haven't explained it.

> Quite frankly, compression for NNTP in this day an age is a solution looking for a problem.  The heavy downloading is coming off of already compressed data, and we're really not sitting at the end of 300baud teletypes for reading text.  Implement a 'ok, I want the session to be compressed/encrypted/whatever until it dies' option if you will, but switching back and forth?  Madness.  Not to mention, pointless.

Look, I just spent half an hour downloading the headers of alt.binaries.cd.image.  I just reassured myself that the 400+ MB of raw XOVER information could, in fact, be reduced to the about 55MB that zlib deflate offers, and that it cuts the download time to about a tenth.  I'm sorry, mate, but you aren't telling me, now, that I can't find a use for this feature.  I will not wait another second for my newsreader to download uncompressed, easily-compressed article headers when my CPU can probably burn the entire compressed stream while it's being held in memory!!!

Ironically, the only time I don't consider compression is when I'm on my 400MHz Pentium-II box; then the overhead of decompression fails utterly to overtake the raw 20mbit connection I have here (of which probably only a fraction is used, both because of my ISP's utter failure to deliver as promised and because of basic properties of TCP).

I will paste the Tcl script that did the tests.  Try it for yourself.  It writes the XOVER output to stdout.  Modify to fit your GigaNews account.  Uncomment the lines that send and get the compression feature negotiation to enable it.  Run with and without compression, use time.  Compare the data sizes and times.  They will speak for themselves.

Cheers,
Sabahattin

#!/usr/bin/tclsh
set sock [socket news-europe.giganews.com 80]
# Remove "-europe" above if in US
fconfigure $sock -encoding binary -translation crlf -buffering none
gets $sock
puts $sock "authinfo user gn453599"
gets $sock
puts $sock "authinfo pass xxxxxxxx"
gets $sock
# Uncomment to enable compression
#puts $sock "xfeature compress gzip"
#gets $sock
puts $sock "group alt.binaries.cd.image"
gets $sock
puts $sock "xover 1-"
puts stderr [gets $sock]
# This means there's a very small trailer, ignored for stats for simplicity's sake to avoid nonblocking complexity
puts $sock quit
fconfigure stdout -buffering none -encoding binary -translation crlf
puts -nonewline [read $sock]