ietf-nntp Backfill

[Clive D.W. Feather]

Here's my attempt to explain the backfill issue.

Firstly, "backfill" is not related to a server reset, where the server
starts counting messages from article 1 all over again. I ignore reset
in this discussion.

Secondly, thinking about this carefully exposes a number of questions.
These I have labelled as [Q1] [Q2] etc. in this email; each of them
needs to be addressed by 977bis.


Suppose that I am in an NNTP session and have the following dialogue:

  -> GROUP news.misc
  <- 211 101 1200 1300 news.misc group selected

Everyone is agreed, I think, that the following now apply:
* news.misc is the current group
* article 1200 and article 1300 both exist
* article 1200 is the current article
* all, some, or none of articles 1201 to 1299 might exist.

If I now issue NEXT commands, the article numbers I will be sent will be
monotonically increasing until I reach the end. Let us suppose that
there are only 4 articles in the group. Therefore I should get:

  -> NEXT
  <- 223 1234 <article-id> article selected
  -> NEXT
  <- 223 1266 <article-id> article selected
  -> NEXT
  <- 223 1300 <article-id> article selected
  -> NEXT
  <- 421

Or will it ? Can new articles appear while the group is current ?

[Q1] When a group becomes the current group, is the set of articles made
available to the client fixed at that moment ?

An answer of "yes" to [Q1] leads to:

[Q2] When will new articles become available and expired/cancelled ones
become unavailable ? Is it:
  - not until a new NNTP session
  - next time the group is made the current group after another group
    has been the current group
  - next time the group is made the current group, whether or not it
    was the current group
?

Note that an answer of "yes" to Q1 means that NEXT and LAST move each
way consistently along a fixed list, *but* it also requires that expired
or cancelled messages remain available while the group is current. In
other words, the server must somehow defer the deleting of these
articles.

If the answer to Q1 is "no", then this implies that articles can appear
or disappear during a single NNTP session.


Whether or not articles appear or disappear during sessions, they can
certainly appear and disappear between sessions. Taking the above
example scenario, where the first session had articles 1200, 1234, 1266,
and 1300, the following questions about the addition and removal of
articles, no matter when it happens, need to be explicitly answered (I
know some seem obvious, but bear with me):

[Q3] Can an article appear with a number higher than the previous
maximum, such as 1357 ? Obviously the new maximum would be higher.

[Q4] Can an article appear with a number less than the previous minimum,
such as 1111 ? Obviously the new minimum would be lower.

[Q5] Can an article appear with a number between the previous limits,
such as 1250 ?

[Q6] Can the lowest numbered article disappear ? Obviously the minimum
would increase if necessary.

[Q7] Can the highest numbered article disappear even though a lower
numbered article (such as 1266) remains ? Obviously the maximum would
decrease if necessary.

[Q8] Can any other article disappear even though a lower numbered
article remains (e.g. 1266 disappearing with 1234 remaining) ?

[Q9] If an article disappears, can it ever reappear again ? The same
article, this is, not a new article given the same number by the server,
which we all assume is not permitted.

I believe that the answers to Q6, Q7, and Q8 are all "yes", based on the
properties of cancels and expiry. Q9 might at first instance seem to be
"no", but "yes" allows cancels to be cancelled and mistakenly expired
articles to be restored.


There are three common scenarios posited for NNTP clients and servers. I
call these "monotonic", "backfilling", and "wind-back". They answer the
first three questions:

                   Q3  Q4  Q5
Monotonic          Y   N   N
Backfilling        Y   N   Y
Wind-back          Y   Y   Y

Servers appear to exist of all three kinds, but all clients I am aware
of that use article numbers at all are monotonic.

A sensible client needs to allow for the user wanting to mark articles
as unread. Therefore it must be able to cope with the backfilling
strategy, by (for example) maintaining a list of ranges. Given this,
there seems little reason to require that servers be monotonic.

On the other hand, there are advantages to the answer to Q4 being "no".
If it is, then it is possible to forget everything about articles with
numbers below the current minimum. This ensures that historical
information with no current use can be eventually thrown away.

-- 
Clive D.W. Feather    | Associate Director  | Director
Tel: +44 181 371 1138 | Demon Internet Ltd. | CityScape Internet Services Ltd.
Fax: +44 181 371 1150 | <clive at demon.net>   | <cdwf at cityscape.co.uk>
Written on my laptop - please reply to the Reply-To address <clive at demon.net>