Lucene 需要索引的文本文件太大,怎么解决?

发布网友发布时间：2022-04-12 10:37

共2个回答

热心网友时间：2022-04-12 12:06

就报错来看，还没有用到Lucene就出错了，意思是只到第一行就虚拟机内存溢出了，可以考虑把源文件进行切割，如把10M的文本切成5个1M的，建议你试一下

给一个可以切分文件的程序，可把它作为预处理的一部分

public static void splitToSmallFiles(File file, String outputpath) throws IOException {

int filePointer = 0;

int MAX_SIZE = 10240000;

BufferedWriter writer = null;
BufferedReader reader = new BufferedReader(new FileReader(file));
StringBuffer buffer = new StringBuffer();
String line = reader.readLine();

while (line != null) {
buffer.append(line).append("\r\n");
if (buffer.toString().getBytes().length >= MAX_SIZE)
{
writer = new BufferedWriter(new FileWriter(outputpath + "output" + filePointer + ".txt"));
writer.write(buffer.toString());
writer.close();
filePointer++;

buffer = new StringBuffer();
}
line = reader.readLine();
}

writer = new BufferedWriter(new FileWriter(outputpath + "output" + filePointer + ".txt"));
writer.write(buffer.toString());
writer.close();

}

热心网友时间：2022-04-12 13:24

建索引的过程代码？你如何读取那个大文本文件的？