From netnews@DB.Stanford.EDU Sun Apr 30 10:09 PDT 1995
Return-path: 
Received: from Woodstock.Stanford.EDU by nizkor.almanac.bc.ca ; 30 APR 95 10:09:31 PDT
Received: by Woodstock.Stanford.EDU (5.57/25-DB-eef) id AA27209; Sun, 30 Apr 95 10:10:09 -0700
Date: Sun, 30 Apr 95 10:10:09 -0700
From: Net News Filter 
Content-Length: 16926
Content-Type: text/plain
Message-ID: <9504301710.AA27209@Woodstock.Stanford.EDU>
To: kmcvay@nizkor.almanac.bc.ca
Subject: Netnews: NetNews Filter Response
Status: RO

	       Stanford Netnews Filtering Service Info

		  Tak W. Yan (tyan@cs.stanford.edu)
                    Department of Computer Science
                         Stanford University
                   
			    September 1994


( A )  W H O   A R E   W E

As part of the Electronic Library project at Stanford. we are providing a 
filtering service for USENET News (Netnews) articles. A user sends his 
profiles to the service, and will receive news articles relevant to his 
interests periodically. Communication to and from the service is via email 
messages. (A WWW interface for accessing the server is available at 
http://sift.stanford.edu.) It is an experiment on large-scale information 
filtering/dissemination. Please feel free to send suggestions, comments, or 
bug reports to tyan@cs.stanford.edu. 


( B )  W H A T   I S   N E T N E W S

Netnews, or USENET News, is a bulletin board system on the Internet. It is
organized into discussion groups (called newsgroups) covering a wide variety of
topics, e.g., from robotics to video game tips, from food recipes to politics.
Its total readership is in millions and daily traffic in tens of MBs. One 
problem with Netnews is the volume and diversity of information. Our filtering
service allows the user to express her interests in finer granularity (using 
profiles) than newsgroups, and hopefully can provide a better match of 
interests. 


( C )  A   S I M P L E   E X A M P L E

First we describe a simple example to show how the service works. We then talk
about how to pick a good profile. 

( C . 1 )  E x a m p l e

Suppose a user subscribes to the service with these settings via the email 
interface (a WWW interface is also available at http://sift.stanford.edu): 

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
mail netnews@db.stanford.edu
Subject: you may leave this blank
subscribe online information services
period 5
end
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

She will first receive an acknowledgement message from netnews, specifying
the email address the subscription is associated with. The user should keep
a record of this information, as she may need it in the future to remove
herself from the service.
 
After the subscription is successfully submitted, the user will receive email 
messages like this every 5 days: 

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
From netnews@db.stanford.edu Mon Jan 24 10:40:35 1994
To: joe@cs.oceanview.edu
Subject: Netnews - online information services

 Subscription 1: online information services

 Article: misc.activism.progressive.11965
 From: hn0003@handsnet.org
 Subject: HandsNet WEEKLY DIGEST 1/15-21
 Score: 100
 First 20 lines:
  HANDSNET WEEKLY DIGEST  January 15 - 21, 1994
  News from HandsNet's Information Forums
  HandsNet is a national, nonprofit network connecting organizations working
  on social and economic justice issues.  Members use HandsNet to make new
  contacts, work collaboratively and to find and publish information, news
  ....

 Article: ca.politics.38420
 From: rlm@helen.surfcty.com (Robert L. McMillin)
 Subject: GOV-ACCESS #5:Cal.Emergency Svcs.online + Net-fax + MINN Pub Info Net
 Score: 100
 First 20 lines:
  Jan. 22, 1994
  CALIFORNIA OFFICE OF EMERGENCY SERVICES INFO AVAILABLE ONLINE
    
  The state Emergency Digitial Information Service is working fine
  Telnet to telnet oes1.oes.ca.gov 5501
  ....
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

In case an article does not fit in its entirety, the user can request the 
whole article be sent to her. She does this by sending an email message to 
netnews@db.stanford.edu with the word get followed by one or more article 
names in the message body. For example, 

mail netnews@db.stanford.edu
Subject: you can leave this blank
get misc.activism.progressive.11965 ca.politics.38420
end

To remove individual profiles, the user may use the CANCEL command discussed
below. To remove all profiles, the user may use the UNSUBSCRIBE command, as
follows:

mail netnews@db.stanford.edu
Subject: you can leave this blank
user joe@cs.oceanview.edu
unsubscribe
end

Here the user needs to specify what email address the profiles are associated
with, using the USER command. This email address is the one from which the 
user submits her subscribe requests.

( C . 2 )  H o w   t o   P i c k   a   G o o d   P r o f i l e

As mentioned already, Netnews is a very diverse source of information. To 
effectively filter the information, the user must be careful in choosing the 
profile. Say a user is interested in traveling and she submits a profile 
consisting of just one word "travel." This is too broad a topic and the 
profile is likely to match many irrelevant articles that just happen to have 
an occurrence of the word. If she describes her interests in more details, 
such as travel hawaii, the results will be far better. 

The service supports two kinds of profile: boolean and weighted. The default 
mode is boolean. In this mode, you specify words that must appear in articles 
received. For example, the profile "travel hawaii" in the boolean mode matches
articles that contain both words "travel" and "hawaii." The number of 
occurrences of the words is not considered. In the boolean mode, you can also
use the "not" operator to screen out articles. For example, say a user is 
interested in underwater fishing but does not want to receive anything from 
the alt.* newsgroups. She can send the profile "underwater fishing not alt."

The weighted mode provides some kind of "similarity" matching. You specify a 
number of words; for each article, a score is computed based on the number of 
occurrences of these words in the article. An article with a score higher than
a certain threshold is returned.

In the weighted mode, it is important to set the threshold appropriately. A 
good way to pick the appropriate threshold is to use the test run (for WWW) or
search (for email) function. This will perform a search over the previous 
day's news articles, and by looking at the scores of the articles returned, 
you can decide what is the minimum score for an article to be relevant. A 
basic rule of thumb is that if you set a high threshold (say 90), then only 
articles that contain most of the words in the profile will be returned. And 
if set a low threshold (say 10), then articles that match any of the words in 
the profile will be returned. The default threshold is set to 60.


( D )  U S E R  M E S S A G E  F O R M A T

User messages should be sent to netnews@db.stanford.edu. The subject field of
the message is ignored. Each message is a request to the service. Each
request consists of a number of commands. Each command must start with a new
line with no leading spaces. Continuation lines begin with a space or a tab.
All commands are case-insensitive. 

Requests are associated with the return address of the user message. Service
replies and deliveries will be sent to that address. 

The usages of the commands are as follows.

( D . 1 )  S u b s c r i b i n g

To subscribe for articles, use these commands:

SUBSCRIBE word word ...        Subscribe for articles relevant to the profile
                               specified by 's. Two types of profiles 
                               are supported: weighted and boolean. For 
                               boolean profiles, you first specify words that
                               must all appear in a matching article, and then
                               words that must not be in, separated by the word
                               "not." For example,

                                   subscribe food recipe not fish

                               You may also skip the "not" portion, e.g.,

                                   subscribe world cup soccer
  
                               For weighted profiles, just type in some plain 
                               text that describes your interest, such as:

                                   subscribe nba golden state warriors

                               News articles will be given score based on the
                               number of occurrences of the profile words and
                               articles whose scores are higher than the
                               threshold (see below) will be returned to you.
                               
                               The SUBSCRIBE command may be optionally 
                               followed by TYPE, LINES, PERIOD, EXPIRE, and 
                               THRESHOLD commands.

TYPE type                      (Optional - default boolean) Specify 
                                as the type of the profile.  must be
                                either the string "weighted" or "boolean." 

LINES lines                    (Optional - default 20 lines) Specify the 
                               number of lines from the beginning of an 
                               article to be included in the news articles 
                               abstract sent to you.  must be an 
                               integer from 1 to 60.
 
PERIOD period                  (Optional - default 1 day) Specify  
                               as the period between notifications (in days).

EXPIRE days                    (Optional - default 9999 days) Specify 
                                as the length (in days) for which
                               the subscription is valid.

THRESHOLD score                (Optional - default 60) Only applicable for
                               weighted profiles. Specify  as the
                               minimum score for an article to be relevant. 
                               The most relevant article is given a score of 
                               100.  must an integer between 1 to 100.

For example, to request a subscription for articles related to "food recipe" 
but not "fish," that is valid for 200 days and gives 10 lines in the news 
article excerpts, send this: 

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
mail netnews@db.stanford.edu
Subject: you may leave this blank
subscribe food recipe not fish
lines 10
expire 200
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

The service will acknowledge your subscription, returning a subscription
identifier (sid). 

( D . 2 )  G e t t i n g   A r t i c l e s

After receiving notifications of articles that may be relevant to your
interests, you may decide to see an article in its entirety. You can get the
whole article with the GET command:

GET article article ...        Get the articles specified (by their article 
                               ids).

For example, 

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
mail netnews@db.stanford.edu
Subject:
get news.announce.conferences.3670
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

( D . 3 )  R e l e v a n c e   F e e d b a c k

After reading the articles, you may find some that you like. You can provide
feedback using these commands:

FEEDBACK sid                   Provide feedback to subscription .

LIKE article article ...       Specify relevant article(s) by their ids.

For example, this message says that article news.announce.conferences.3670 is
relevant to subscription 1:

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
mail netnews@db.stanford.edu
Subject: you may leave this blank
feedback 1
like news.announce.conferences.3670
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

With feedback information, the service may be able to better match future
articles against your subscriptions. 

( D . 4 )  M a n a g i n g   S u b s c r i p t i o n s

You can manage your subscriptions with these commands: 

UPDATE sid                     Update subscription with id . Must be 
                               followed by one or more of PERIOD, EXPIRE, 
                               THRESHOLD, LINES, TYPE (see D.1.), or PROFILE 
                               commands to specify the parameter(s) to be 
                               updated.

PROFILE word word ...          Specify the new profile for the UPDATE command.

CANCEL sid                     Cancel subscription .

LIST                           List all your subscriptions.

For example, to update the period and the threshold of a subscription: 
      
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
mail netnews@db.stanford.edu
Subject: you may leave this blank
update 3
period 1
threshold 60
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

( D . 5 )  S e a r c h   F o r   P a s t   A r t i c l e s
  
Besides providing the subscription service, the service also allows you to
search for recent articles that are already in the database:

SEARCH word word ...           Do a search of the database with the given
                               query. Maybe optionally followed by a THRESHOLD
                               command to specify the minimum score for an 
                               article to be retrieved.

For example, to search for articles related to "information filtering":

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
mail netnews@db.stanford.edu
Subject: you may leave this blank
search information filtering
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

( D . 6 )  U n s u b s c r i b i n g

To remove all profiles, the user may use the USER and UNSUBSCRIBE command:

USER address                   Specify the email address with which the 
                               profiles are associated. 

UNSUBSCRIBE                    Delete all profiles for the specified email
                               address.

For example,

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
mail netnews@db.stanford.edu
Subject: you can leave this blank
user joe@cs.oceanview.edu
unsubscribe
end
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

( D . 6 )  O t h e r   C o m m a n d s

END                            End the request message. Useful for
                               preventing the processing of signatures. 
 
HELP                           Get help information on server.


( E )  F r e q u e n t l y   A s k e d   Q u e s t i o n s

(1) Is the service free?
    Yes, it's absolutely FREE!

(2) How do I access the service?
    Email access: send message to netnews@db.stanford.edu, with word "help"
                  in message body
    WWW access: URL http://sift.stanford.edu

(3) Is there any similar service you know of?
    There is a companion service at elib@cs.stanford.edu for filtering
    computer science technical reports. A search server is also available
    at URL http://elib.stanford.edu.

(4) Can I tell my friends about the service?
    Yes, please do. The server should still be able to handle more 
    subscriptions :-) The current number of subscriptions is 9800+.

(5) How to prevent the processing of "signatures" at the end of 
    email requests? 
    I have added a new command "end." You can include the word "end"
    on a line by itself at the end of the request message (before
    the signature). (Also see D.6 above.)

(9) Any papers written on the server?
    A paper is available by anonymous ftp at ftp://db.stanford.edu/sift/sift.ps
    Some related papers on information filtering available at 
    ftp://db.stanford.edu/pub/yan
    
(10)What newsgroups are covered?
                    calstate        gnu             out.going       tx
    adass           can             ieee            phl             u3b
                    chi             in.coming       rec             ucb
    alt                             info            sci             uiuc
    atl             comp            junk            scruz           uk
    aus             control         misc            soc             vmsnet
    ba                              ne                              za
    bionet          ee              news            talk
    bit             fj              no              test
    biz             fl              nz              trial
    ca              general         ont             triangle



 =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
 For help information, send email                     NetNews Filtering Server
 with word 'help' in message body                      netnews@db.stanford.edu
 =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

Home ·  Site Map ·  What's New? ·  Search Nizkor