[NNTP] Compressed LIST (and other commands) answers

Wed Dec 2 11:48:00 PST 2009

Hi Clive,

>> Although I wonder if, at that point, whether we want to just provide some
>> facility negotiate compression of all subsequent traffic on the NNTP
>> connection.  In other words, rather than treating this on a
>> command-by-command basis, what if we model it after STARTTLS?
>
> I have a vague memory that we discussed this during the development of
> 3977, and decided that this was the direction to go. In additon, TLS
> already does compression, so we don't even need to invent anything.

I see RFC 3943 "Transport Layer Security (TLS) Protocol Compression Using
Lempel-Ziv-Stac (LZS)" but OpenSSL 1.0.0 does not implement it.
It seems that the only compression that OpenSSL currently recognizes
is DEFLATE.

I have just tested with INN:

% openssl s_client -connect localhost:563 -ssl3
---
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA
Server public key is 1024 bit
Compression: zlib compression
Expansion: zlib compression
SSL-Session:
    Protocol  : SSLv3
    Cipher    : DHE-RSA-AES256-SHA
    Session-ID: 94688B87D8D12DCE21C33B050F194E949A78875D7098DF05FD391DC89839D7BA
    Session-ID-ctx:
    Master-Key: 89B44D4B8C09B70ACFF109E8C77BFD9BA2ADE7F07D28DD89990E53CCA4DAD0B227A5A6BEEACD5BE02A697302583AF98F
    Key-Arg   : None
   Compression: 1 (zlib compression)
    Start Time: 1259781851
    Timeout   : 7200 (sec)
    Verify return code: 18 (self signed certificate)
---

It seems to use "zlib compression".  Shouldn't it return DEFLATE instead?
Well, never mind.

It is strange that I have to force "-ssl3"; otherwise, no compression is used
(the default is "-tls1"):

---
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA
Server public key is 1024 bit
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : DHE-RSA-AES256-SHA
    Session-ID: 1D8D6322CD08A4244C911898CA438C7D7170B70358E76943CFAA5F27013061B9
    Session-ID-ctx:
    Master-Key: F3393B2457EC130F8433285B36466BCB7344A719FEC28A2D32C8A24F6CB64EB3A4B42267DEABCF0EBB5EB33B0179D167
    Key-Arg   : None
    Start Time: 1259781874
    Timeout   : 300 (sec)
    Verify return code: 18 (self signed certificate)
---

The problem I currently see in INN is that we do not use buffered answers
for all commands...  For instance:

CAPABILITIES
write to 0x80bef18 [0x80c9b88] (90 bytes => 90 (0x5A))
0000 - 17 03 00 00 20 40 42 78-a4 c1 b1 44 ec 62 0a 93   .... @Bx...D.b..
0010 - ef c0 df 67 70 3e aa 70-cf e0 ed 6f ef 64 3b 00   ...gp>.p...o.d;.
0020 - 9a c5 11 c0 d3 17 03 00-00 30 46 42 a6 85 9f 2d   .........0FB...-
0030 - 3c bb 4a 47 15 b4 ea 29-76 fd 2e 29 1f 17 96 c8   <.JG...)v..)....
0040 - 6f d8 5f ff a6 96 24 79-46 a4 23 6d 5f 02 88 4b   o._...$yF.#m_..K
0050 - 6f c7 9f bc 4d f5 55 1a-3e 6f                     o...M.U.>o
read from 0x80bef18 [0x80c5378] (5 bytes => 5 (0x5))
0000 - 17 03 00 00 40                                    ....@
read from 0x80bef18 [0x80c537d] (64 bytes => 64 (0x40))
0000 - c4 75 e5 9d a9 06 48 39-fe 7f 05 d1 f3 39 4f 25   .u....H9.....9O%
0010 - d6 60 d7 e8 e2 53 0e cd-6c 64 c9 f9 93 e5 ac ca   .`...S..ld......
0020 - 2e 66 62 4b 7e 9d 24 47-da c4 4f c0 d3 35 20 63   .fbK~.$G..O..5 c
0030 - 01 d2 97 f3 b0 60 36 fe-de 3d 2c d0 d0 9f 64 e8   .....`6..=,...d.
101 Capability list:
read from 0x80bef18 [0x80c5378] (5 bytes => 5 (0x5))
0000 - 17 03 00 00 30                                    ....0
read from 0x80bef18 [0x80c537d] (48 bytes => 48 (0x30))
0000 - 12 3c 94 cc a6 98 91 3a-9c f6 4f 01 0e f8 56 21   .<.....:..O...V!
0010 - af be 42 34 6c 41 a1 06-86 29 32 85 39 86 90 12   ..B4lA...)2.9...
0020 - fd 70 41 ef 4c 6f 44 e2-82 79 83 95 f1 6a 27 61   .pA.LoD..y...j'a
VERSION 2

90 bytes for "CAPABILITIES\r\n".
5+48 bytes for "VERSION 2\r\n".

That's pretty bad!

However, an OVER of about 13200 bytes is transmitted in about 1600 bytes.
That's a huge gain!

Is it the best approach we should take?
News clients and news servers bufferize their output and send it compressed,
using STARTTLS?

Note that there will still be a lot of overhead for commands like:
BODY
CHECK <mid>
GROUP group (and its answer)
...

-- 
Julien ÉLIE

« Le sel de l'existence est essentiellement dans le poivre qu'on y met. » 