[NNTP] NNTP Compression

[Ade Lovett]

On Jan 10, 2010, at 02:41 , Sabahattin Gucukoglu wrote:
> From where I'm sitting, which is at this very moment trying to find a portable way to implement a version of the GigaNews Accelerator (I'll call it "GigaNoise"), it would be most welcome!  As it stands, I have to track what state the client and server are in, filter out capabilities, and whatnot, and all because the compression isn't clearly and obviously negotiated, and at what point I'll be getting compressed versus regular output (ARTICLE, for example, is still sent uncompressed).

That would be an issue you would need to take up with your NSP of choice, and not relevant to protocol discussion.

> And I'm not sure precisely what it is about toggling compression that's so terrible, because you haven't explained it.

Simplicity.  Either a stream is negotiated compressed, or it isn't.  You mention having to track state as being an issue, now view that from the point of view of a server farm, dealing with tens of thousands of simultaneous connections, potentially flipping in and out of compression.  Why?  You have, essentially, unlimited connections, so use a couple for compression, and the rest normal.  Trivial state tracking on both ends.  If it is needed ...

> Look, I just spent half an hour downloading the headers of alt.binaries.cd.image.  I just reassured myself that the 400+ MB of raw XOVER information could, in fact, be reduced to the about 55MB that zlib deflate offers, and that it cuts the download time to about a tenth.  I'm sorry, mate, but you aren't telling me, now, that I can't find a use for this feature.

Sure you can.  In the degenerate case of exceptionally large binary groups, use one of the many indexer sites out there, pull the NZB file, and pull down the articles via message ID that you actually want.  Or autogenerate the DMCA takedown notice from the NZB file (code available on request for the right price).

> Ironically, the only time I don't consider compression is when I'm on my 400MHz Pentium-II box; then the overhead of decompression fails utterly to overtake the raw 20mbit connection I have here (of which probably only a fraction is used, both because of my ISP's utter failure to deliver as promised and because of basic properties of TCP).

And this is where we go back to the age old "server vs client author wars" :)  Whilst I'm sure your home system has more than enough capability to hold terabytes of data, with on-the-fly compression, for _home_ needs, scale that up to what you're talking to on the other end, and it becomes non-trivial very very quickly.

> I will paste the Tcl script that did the tests.

Sadly, irrelevant.

1.  Show a _real_world_ case, of relevance to the protocol in its entirety, and not a subset, where compression is useful at all, given the major burden it will put on the server side.

2.  Show a case why compression (in whatever form) should be able to be toggled within a single stream.

-aDe