[freenet-cvs] r14988 - trunk/plugins/XMLSpider

[toad at freenetproject.org]

Author: toad
Date: 2007-09-06 16:40:36 +0000 (Thu, 06 Sep 2007)
New Revision: 14988

Modified:
   trunk/plugins/XMLSpider/XMLSpider.java
Log:
Make sub-indexes much bigger.
We can't rely on grouping them together in containers, because:
- mostly words which are near to each other in the index are not closely related
- we'd need multiple container support and we don't have it
- the containers would be big chunks to fetch and often wouldn't be reused on a big index
So it makes sense to just use huge sub-indexes.
Long term we want sub-indexes to be split by size rather than number of entries.

Modified: trunk/plugins/XMLSpider/XMLSpider.java
===================================================================
--- trunk/plugins/XMLSpider/XMLSpider.java	2007-09-06 16:38:01 UTC (rev 14987)
+++ trunk/plugins/XMLSpider/XMLSpider.java	2007-09-06 16:40:36 UTC (rev 14988)
@@ -138,7 +138,7 @@
 	 * Lists the allowed mime types of the fetched page. 
 	 */
 	public Set allowedMIMETypes;
-	private static final int MAX_ENTRIES = 20;
+	private static final int MAX_ENTRIES = 200;
 	private static int version = 7;
 	private static final String pluginName = "XML spider "+version;
 	/**