ietf-nntp OVER extension

[Russ Allbery]

Clive D W Feather <clive at demon.net> writes:

> This is a significant issue: if X-Wombat is in the overview database,
> and article N does not have an X-Wombat header, what does OVER (that is,
> our new command, not anything historical) return as that field ? Is it:

> (A) "X-Wombat:"
> (B) ""             (that is, an empty string).

> I've assumed (B), which makes LIST OVERVIEW.FMT useful. If the answer is
> (A) then we need to rewrite my rewrite (or the original; take your pick).

As near as I can tell in the existing INN implementation the answer is
(B), and furthermore if you have an overview.fmt like:

    Subject:
    [...]
    Lines:
    Supersedes:full
    Xref:full

and a given article doesn't have a Supersedes header, there will be only
one TAB between the line count and Xref.

>> I've uploaded an HTML-formatted copy of the document to
>> <URL:http://members.verizon.net/~vze35rzk/nntpext/newsoverview.5.html>
>> for reference.

> (1) It's less than clear to me whether "if they are absent entirely" refers
> to the possibility of optional headers or their specific appearance in
> specific articles. So I don't think that page answers our question.

I believe it's referring to the possibility of optional headers.

>> I will again note that overview.fmt _does_ _not_ tell the client what
>> headers have already been collected in the overview.  It describes what
>> fields will be added in the immediate future, and it may or may not
>> correspond to the contents of records stored before today.

> That might be history. That's not what we're saying. It might be want we
> *want* to say, but, if so, we need to agree it and make more changes.

Implementing LIST OVERVIEW.FMT would be substantially harder if it's
required to return the headers as existant in the overview database and
the return of OVER were required to always be consistent in order.
Basically, this means requiring a complete overview rebuild every time the
list of headers added to overview changes since any change would
invalidate the entire existing overview database.

That seems really unnecessary to me when every extension header is already
clearly tagged.  Who would we actually be benefitting by requiring that?
All I can think of is a really lazy client author who doesn't want to
actually parse the overview return.

At present, LIST OVERVIEW.FMT is advisory only and not really that useful.
I'm content with not standardizing it or standardizing it in its current
informational form, but let's please not make life harder for server
authors and administrators in an attempt to make it more useful.

I don't really like the idea of requiring that the extra headers provided
be consistent across the entire overview database, even though I'm aware
that it looks much cleaner from a protocol standpoint.  This is actually
really hard to do on a running news server, since news server spools have
lifetimes frequently measured in years and numerous server policy changes
may be needed over that period of time.  A full overview rebuild is a
non-trivial and extremely time-consuming operation, and for most purposes
it's fine if those extra headers aren't included.  Clients already have to
be prepared for that and fall back on XHDR or HEAD if they're missing
something they need.

-- 
Russ Allbery (rra at stanford.edu)             <http://www.eyrie.org/~eagle/>