1 ;;; nnir.el --- search mail with various search engines -*- coding: iso-8859-1 -*-
3 ;; Copyright (C) 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006,
4 ;; 2007, 2008, 2009, 2010 Free Software Foundation, Inc.
6 ;; Author: Kai Großjohann <grossjohann@ls6.cs.uni-dortmund.de>
7 ;; Swish-e and Swish++ backends by:
8 ;; Christoph Conrad <christoph.conrad@gmx.de>.
9 ;; IMAP backend by: Simon Josefsson <jas@pdc.kth.se>.
10 ;; IMAP search by: Torsten Hilbrich <torsten.hilbrich <at> gmx.net>
11 ;; IMAP search improved by Daniel Pittman <daniel@rimspace.net>.
12 ;; nnmaildir support for Swish++ and Namazu backends by:
13 ;; Justus Piater <Justus <at> Piater.name>
14 ;; Keywords: news mail searching ir
16 ;; This file is part of GNU Emacs.
18 ;; GNU Emacs is free software: you can redistribute it and/or modify
19 ;; it under the terms of the GNU General Public License as published by
20 ;; the Free Software Foundation, either version 3 of the License, or
21 ;; (at your option) any later version.
23 ;; GNU Emacs is distributed in the hope that it will be useful,
24 ;; but WITHOUT ANY WARRANTY; without even the implied warranty of
25 ;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
26 ;; GNU General Public License for more details.
28 ;; You should have received a copy of the GNU General Public License
29 ;; along with GNU Emacs. If not, see <http://www.gnu.org/licenses/>.
33 ;; TODO: Documentation in the Gnus manual
36 ;; Subject: Re: Including nnir.el
37 ;; Newsgroups: gmane.emacs.gnus.general
38 ;; Message-ID: <v9d5dnp6aq.fsf@marauder.physik.uni-ulm.de>
39 ;; Date: 2006-06-05 22:49:01 GMT
41 ;; On Sun, Jun 04 2006, Sascha Wilde wrote:
43 ;; > The one thing most hackers like to forget: Documentation. By now the
44 ;; > documentation is only in the comments at the head of the source, I
45 ;; > would use it as basis to cook up some minimal texinfo docs.
47 ;; > Where in the existing gnus manual would this fit best?
49 ;; Maybe (info "(gnus)Combined Groups") for a general description.
50 ;; `gnus-group-make-nnir-group' might be described in (info
51 ;; "(gnus)Foreign Groups") as well.
54 ;; The most recent version of this can always be fetched from the Gnus
55 ;; repository. See http://www.gnus.org/ for more information.
57 ;; This code is still in the development stage but I'd like other
58 ;; people to have a look at it. Please do not hesitate to contact me
61 ;; What does it do? Well, it allows you to index your mail using some
62 ;; search engine (freeWAIS-sf, swish-e and others -- see later),
63 ;; then type `G G' in the Group buffer and issue a query to the search
64 ;; engine. You will then get a buffer which shows all articles
65 ;; matching the query, sorted by Retrieval Status Value (score).
67 ;; When looking at the retrieval result (in the Summary buffer) you
68 ;; can type `G T' (aka M-x gnus-summary-nnir-goto-thread RET) on an
69 ;; article. You will be teleported into the group this article came
70 ;; from, showing the thread this article is part of. (See below for
73 ;; The Lisp installation is simple: just put this file on your
74 ;; load-path, byte-compile it, and load it from ~/.gnus or something.
75 ;; This will install a new command `G G' in your Group buffer for
76 ;; searching your mail. Note that you also need to configure a number
77 ;; of variables, as described below.
81 ;; * If you don't use HyREX as your search engine, this expects that
82 ;; you use nnml or another one-file-per-message backend, because the
83 ;; others doesn't support nnfolder.
84 ;; * It can only search the mail backend's which are supported by one
85 ;; search engine, because of different query languages.
86 ;; * There are restrictions to the Wais setup.
87 ;; * There are restrictions to the imap setup.
88 ;; * gnus-summary-nnir-goto-thread: Fetches whole group first, before
89 ;; limiting to the right articles. This is much too slow, of
90 ;; course. May issue a query for number of articles to fetch; you
91 ;; must accept the default of all articles at this point or things
94 ;; The Lisp setup involves setting a few variables and setting up the
95 ;; search engine. You can define the variables in the server definition
97 ;; (setq gnus-secondary-select-methods '(
98 ;; (nnimap "" (nnimap-address "localhost")
99 ;; (nnir-search-engine hyrex)
100 ;; (nnir-hyrex-additional-switches ("-d" "ddl-nnimap.xml"))
102 ;; Or you can define the global ones. The variables set in the mailer-
103 ;; definition will be used first.
104 ;; The variable to set is `nnir-search-engine'. Choose one of the engines
105 ;; listed in `nnir-engines'. (Actually `nnir-engines' is an alist,
106 ;; type `C-h v nnir-engines RET' for more information; this includes
107 ;; examples for setting `nnir-search-engine', too.)
109 ;; The variable nnir-mail-backend isn't used anymore.
112 ;; You must also set up a search engine. I'll tell you about the two
113 ;; search engines currently supported:
117 ;; As always with freeWAIS-sf, you need a so-called `format file'. I
118 ;; use the following file:
121 ;; | # Kai's format file for freeWAIS-sf for indexing mails.
122 ;; | # Each mail is in a file, much like the MH format.
124 ;; | # Document separator should never match -- each file is a document.
125 ;; | record-sep: /^@this regex should never match@$/
127 ;; | # Searchable fields specification.
129 ;; | region: /^[sS]ubject:/ /^[sS]ubject: */
130 ;; | subject "Subject header" stemming TEXT BOTH
133 ;; | region: /^([tT][oO]|[cC][cC]):/ /^([tT][oO]|[cC][cC]): */
134 ;; | to "To and Cc headers" SOUNDEX BOTH
137 ;; | region: /^[fF][rR][oO][mM]:/ /^[fF][rR][oO][mM]: */
138 ;; | from "From header" SOUNDEX BOTH
142 ;; | stemming TEXT GLOBAL
143 ;; | end: /^@this regex should never match@$/
146 ;; 1998-07-22: waisindex would dump core on me for large articles with
147 ;; the above settings. I used /^$/ as the end regex for the global
148 ;; field. That seemed to work okay.
150 ;; There is a Perl module called `WAIS.pm' which is available from
151 ;; CPAN as well as ls6-ftp.cs.uni-dortmund.de:/pub/wais/Perl. This
152 ;; module comes with a nifty tool called `makedb', which I use for
153 ;; indexing. Here's my `makedb.conf':
156 ;; | # Config file for makedb
158 ;; | # Global options
159 ;; | waisindex = /usr/local/bin/waisindex
160 ;; | wais_opt = -stem -t fields
161 ;; | # `-stem' option necessary when `stemming' is specified for the
162 ;; | # global field in the *.fmt file
165 ;; | homedir = /home/kai
167 ;; | # The mail database.
169 ;; | files = `find $homedir/Mail -name \*[0-9] -print`
170 ;; | dbdir = $homedir/.wais
174 ;; The Lisp setup involves the `nnir-wais-*' variables. The most
175 ;; difficult to understand variable is probably
176 ;; `nnir-wais-remove-prefix'. Here's what it does: the output of
177 ;; `waissearch' basically contains the file name and the (full)
178 ;; directory name. As Gnus works with group names rather than
179 ;; directory names, the directory name is transformed into a group
180 ;; name as follows: first, a prefix is removed from the (full)
181 ;; directory name, then all `/' are replaced with `.'. The variable
182 ;; `nnir-wais-remove-prefix' should contain a regex matching exactly
183 ;; this prefix. It defaults to `$HOME/Mail/' (note the trailing
188 ;; The Namazu backend requires you to have one directory containing all
189 ;; index files, this is controlled by the `nnir-namazu-index-directory'
190 ;; variable. To function the `nnir-namazu-remove-prefix' variable must
191 ;; also be correct, see the documentation for `nnir-wais-remove-prefix'
194 ;; It is particularly important not to pass any any switches to namazu
195 ;; that will change the output format. Good switches to use include
196 ;; `--sort', `--ascending', `--early' and `--late'. Refer to the Namazu
197 ;; documentation for further information on valid switches.
199 ;; To index my mail with the `mknmz' program I use the following
200 ;; configuration file:
203 ;; | package conf; # Don't remove this line!
205 ;; | # Paths which will not be indexed. Don't use `^' or `$' anchors.
206 ;; | $EXCLUDE_PATH = "spam|sent";
208 ;; | # Header fields which should be searchable. case-insensitive
209 ;; | $REMAIN_HEADER = "from|date|message-id|subject";
211 ;; | # Searchable fields. case-insensitive
212 ;; | $SEARCH_FIELD = "from|date|message-id|subject";
214 ;; | # The max length of a word.
215 ;; | $WORD_LENG_MAX = 128;
217 ;; | # The max length of a field.