[NNTP] LIST EXTENSIONS and an NNTPv2 capability

Clive D.W. Feather clive at demon.net
Fri Oct 15 01:52:18 PDT 2004


Russ Allbery said:
> The way that a proper client of a protocol that supports an extension
> mechanism should behave is that immediately after connecting, it inquires
> as to the capabilities of the server.  Subsequently, it assumes those
> listed capabilities are accurate and does not attempt to use any
> capabilities that were not advertised.  I realize that this is not how
> NNTP clients have ever worked in the past -- this is something that we're
> seeking to change by adding a proper extension mechanism.  This is how
> everyone else does it (IMAP, POP, SMTP, etc.), and we're trying to bring
> NNTP in line with other Internet protocols.

Okay, agreement so far.

> Given this new approach, and given the way a client should be implemented
> when complying with our draft, there are two problems with the LIST
> EXTENSIONS mechanism as currently present in our base draft.
[...]
> The first problem is that, since LIST EXTENSIONS has been around as an
> idea for quite some time, there is at least one known badly bogus
> implementation.

Right.

> The second problem is that, based on our drafts, at least one
> implementation (INN) and possibly others have started adding LIST
> EXTENSIONS.  However, the extension list was not finalized, and in
> particular AUTHINFO was not advertised by the early INN implementation
> even though AUTHINFO USER was supported.  A client that uses the extension
> mechanism properly would therefore think that those servers didn't support
> authentication when they do.

This is *always* going to be a problem in the stage between an extension
being invented and the specification being finalised. Agreed it's rather
worse for the stuff that has been around for a while, but it's not unique
to them.

> The proposed solution is this:  Specify in our base draft that any server
> that fully complies with the new specification advertise the NNTPv2
> capability as part of LIST EXTENSIONS.

Right, stop here. I don't like this idea. It's a kludge, and an unstructured
one at that. However, it does identify a genuine problem.

Having read this message and the following thread, I did a lot of thinking.
I also harked back to some of the bitter experiences we had in standardising
the C language.

Firstly, let's recognise that, by doing this, *no* existing server is going
to comply with the new specification without being changed. This is *NOT* a
problem, it's an opportunity. What we are doing at this point is to draw a
line in the sand (or in history, if you like). So let's look at the whole
problem and fix it properly.

======== Line in the sand :-) ========

Rather than saying that LIST EXTENSIONS is unreliable unless the server
sets the "I got it right, honest" flag, I think we're better off having a
new command. There's little downside because server implementers are going
to have to make changes anyway. This can then also fix other problems.

The two big issues I can see that are going to bite us one day (if indeed
they haven't already) are:

(1) There's no good way to indicate conformance to particular versions of
extension protocols. Adding a versioning facility usually turns out to be
a good idea in the long run.

(2) It's better to indicate what capabilities are available altogether,
rather than just what are available right now. It's better for a client to
know "you can do TLS once you've done MODE READER" than for it to have
to guess that doing MODE READER might make STARTTLS magically appear.

So let's fix these now, while we have the chance.

Let me present my idea first as a bit of protocol:

    [C] STATUS
    [S] 101 1.0     List of available extensions follows
    [S] OVER 1.0 + MSGID
    [S] HDR 1.0-RAA +
    [S] LISTGROUP 1.0 -MODE-READER
    [S] STARTTLS 0.0-02 -502
    [S] AUTHINFO 0.0-05-x -502 USER SASL:PLAIN,EXTERNAL
    [S] STREAMING 0.0-perhaps +
    [S] XSECRET -CDWF -483
    [S] XDATA -CDWF +
    [.]

The specification is roughly as follows.

The command cannot fail, apart from the generic 400 and 501 responses. The
argument to the 101 response code is a version-code; see below.

Each line in the extended response describes one extension, as with LIST
EXTENSIONS. However, the line gives more detail; there are always at least
three words:
* The extension name. This becomes case-insensitive just like the rest of
  NNTP, but we continue to require letters only.
* A version-code. This indicates the conformance status of the extension;
  I discuss it further below.
* A status code. This is either "+" to show that the extension is
  *currently* available, or "-" followed by a reason to show that it isn't.
  For reasons, I've used "502", "480", or "483" if use of the extension
  would generate that response, or an extension name (or "MODE-READER")
  if use would generate a 401 response.
Any further words are parameters to the extension, as with LIST EXTENSIONS.

The number of lines in the output does not change during a session; as
extensions become available or unavailable, the status changes instead.
(However, I'd make this a SHOULD rather than a MUST so that servers can
still have secret extensions if they want.) This allows a client to
determine what *can* be available as well as what is available right now.
I think there's significant benefit in this.

Note that we can also indicate "some commands need MODE READER" by a line
in the capabilities, or even by defining a second argument to the 101 code.

The main benefit of this proposal is the version codes. Here's how I see
them working:

* The format of a version code is:

    version-code = standard-version / private-version
    standard-version = major-version "." minor-version [version-detail]
    private-version = version-detail
    major-version = 1*DIGIT
    minor-version = 1*DIGIT
    version-detail = "-" arbitrary-string

* For a standard extension, the version is of the form "1.3". Each
published specification states the version number for that specification.
Changes in the minor version number are upwards compatible; that is, a
server that advertises 1.3 is *by definition* conformant with the 1.0, 1.1,
and 1.2 specifications as well [1.10 comes after 1.9.] Changes in the
major version are generally assumed to be incompatible.

* To allow for further development work on a standard extension, a
version-detail can be added to show conformance to drafts or other
experiments. The minor version remains that of the last real specification
that the server conforms to. The contents of the version-detail are under
the control of the new specification author.

So if work is going on to replace version 1.2, the eventual spec will be
1.3 and, in the meantime, servers can advertise "1.2-d05" to indicate they
conform to draft 05 of the new specification. Clients would normally ignore
what's after the dash.

* Version 0.0 is used for an extension that's being developed but hasn't
yet been standardised. Again, the version-detail is available.

* For a private extension, the version begins with a dash; anything beyond
that is entirely up to the implementer.

* Version 0.1 could be used to indicate conformance to RFCs that predate
our specification. But a better approach might be to treat them as private
with a code of "-RFC2980" (or whatever number is involved). Or how do
people feel about 0.2980?

The ability to flag a private extension also means that we can say that the
server *MUST* list all commands it supports instead of the present
weasel-wording.

Okay, repeating the above example so I can explain it further:

    [C] STATUS
    [S] 101 1.0     List of available extensions follows
    [S] OVER 1.0 + MSGID
    [S] HDR 1.0-RAA +
    [S] LISTGROUP 1.0 -MODE-READER
    [S] STARTTLS 0.0-02 -502
    [S] AUTHINFO 0.0-05-x -502 USER SASL:PLAIN,EXTERNAL
    [S] STREAMING 0.0-perhaps +
    [S] SECRET -CDWF -483
    [S] XDATA -CDWF +
    [.]

the meaning is:
* This server conforms to specification 1.0.
* The server supports 8 extensions.
* OVER and LISTGROUP conform to specification 1.0. So does HDR, but there's
  some development work going on with it (tagged RAA). STARTTLS, AUTHINFO,
  and STREAMING conform to draft specifications. SECRET and XDATA are
  private.
* Right now, OVER MSGID, HDR, STREAMING, and XDATA are all available while
  the other four aren't.
* LISTGROUP will work once the client has done MODE READER.
* STARTTLS and AUTHINFO aren't available for the rest of this session.
* SECRET is available if appropriate privacy is negotiated.

Note: 601 would be a better response code; we've said it should be used
for future multi-line responses. But I don't want this to divert the main
discussion.

Further thought: I'd like to find a way to provide further granularity in
the status information, showing that some bits are available and some
aren't. For example, a parameter ":" could mean "here's another status
code":

    [S] AUTHINFO 0.6-d02 + SASL:CRAM-MD5 : -483 USER SASL:PLAIN,EXTERNAL

meaning "I'll do CRAM-MD5 now, but if you want USER, PLAIN, or EXTERNAL
you'll need to sort out privacy". Or this may be getting too complex;
perhaps it should be left to individual extensions to sort out.

But I'd certainly like to encourage extensions to indicate both what is
available and what is currently-unavailable, rather than just the former.
And this is something that I'd like to see even if we don't adopt this
STATUS proposal.

> We also need to somehow specify that any server that implements AUTHINFO
> USER needs to advertise it if it advertises the NNTPv2 capability.  I'm
> not completely sure of the best way to do that; one way would be to
> specifically mention the three legacy extensions that we're standardizing
> (AUTHINFO, STARTTLS, and CHECK/TAKETHIS or STREAMING), and another would
> be to require that the server advertise an extension for any commands that
> don't begin with X.  A third approach would be to recognize that this
> blessing of widely implemented commands that predated the extension
> mechanism is a one-time transition.  This probably requires some more
> discussion to find the best approach from a transition standpoint.

Or we can bite the bullet and say that servers must advertise every command
that they implement.

However, I'm not 100% convinced. If a server does AUTHINFO SASL to spec,
and AUTHINFO GENERIC, what should it advertise?

> However, the goal is to make LIST EXTENSIONS reliable, including for
> everything that we're blessing under its existing command names, so that
> clients can just read the output and then use that information without
> having to second-guess it.  Without doing this, LIST EXTENSIONS is both
> badly different from the way that other protocols use it and not actually
> useful for clients.

Agreed.

> I think this approach has a lot of merit, and that it's easy enough to add
> the NNTPv2 requirement to the existing draft.  I also think that this is a
> minor enough change that it's worth doing this late in the process,
> although I'd appreciate it if Scott would check me on this.

I think that my above proposal is worth doing, even if we have to repeat
Last Call.

-- 
Clive D.W. Feather  | Work:  <clive at demon.net>   | Tel:    +44 20 8495 6138
Internet Expert     | Home:  <clive at davros.org>  | Fax:    +44 870 051 9937
Demon Internet      | WWW: http://www.davros.org | Mobile: +44 7973 377646
Thus plc            |                            |



More information about the ietf-nntp mailing list