Hama测试问题记录

1.DiskVerticesInfo类的使用

生成数据: hama jar hama-examples-0.6.4.jar gen symmetric 100 20 /hamaInput 20

使用DiskVerticesInfo类，0.6.3报错:

测试:hama jar hama-examples-0.6.4.jar pagerank /hamaInput /hamaOutputPK6

结果:

14/06/26 21:32:37 INFO ipc.Server: Stopping IPC Server listener on 52257
14/06/26 21:32:37 DEBUG ipc.Server: Checking for old call responses.
14/06/26 21:32:37 INFO ipc.Server: Stopping IPC Server Responder
14/06/26 21:32:37 ERROR bsp.BSPTask: Shutting down ping service.
14/06/26 21:32:37 FATAL bsp.GroomServer: Error running child
java.lang.IllegalArgumentException: Messages must never be behind the vertex in ID! Current Message ID: 100016 vs. 20
at org.apache.hama.graph.GraphJobRunner.iterate(GraphJobRunner.java:306)
at org.apache.hama.graph.GraphJobRunner.doSuperstep(GraphJobRunner.java:254)
at org.apache.hama.graph.GraphJobRunner.bsp(GraphJobRunner.java:145)
at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:191)
at org.apache.hama.bsp.BSPTask.run(BSPTask.java:146)
at org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1249)
java.lang.IllegalArgumentException: Messages must never be behind the vertex in ID! Current Message ID: 100016 vs. 20
at org.apache.hama.graph.GraphJobRunner.iterate(GraphJobRunner.java:306)
at org.apache.hama.graph.GraphJobRunner.doSuperstep(GraphJobRunner.java:254)
at org.apache.hama.graph.GraphJobRunner.bsp(GraphJobRunner.java:145)
at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:191)
at org.apache.hama.bsp.BSPTask.run(BSPTask.java:146)
at org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1249)

改成0.6.4版本后就好了

2. PartitioningRunner导致的多文件资源读写不足

tmp/hama-parts/job_201406301636_0001/part-19/file-19 could only be replicated to 0 nodes, instead of 1
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1920)
at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:783)
at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
查到datanode的日志里，显示:
java.io.FileNotFoundException: /hadoop-1.2.0/data/blocksBeingWritten/blk_10093574377987397_21088.meta (Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:241)
at org.apache.hadoop.hdfs.server.datanode.FSDataset.createBlockWriteStreams(FSDataset.java:1156)

原因:比如Pagerank作业，有20个任务，由于有PartitioningRunner的分区，导致每个任务都要对获取的20个分区分别建立文件进行归并排序，也就是说HDFS同时要容纳20x20个文件写入(考虑到对应的块，读写压力会更大)，我这里测试的时候就因为这个原因导致HDFS报错了

即使将linux的ulimit调到10240也不行

最后没招，只能将文件块大小改大了，最大任务数限制在10，这样才好.

3. 数据问题?
14/07/02 10:38:41 ERROR bsp.BSPTask: Error running bsp setup and bsp function.
java.lang.IllegalArgumentException: A message has recieved with a destination ID: 100003880356367481247 that does not exist! (Vertex iterator is at100003935752877898316 ID) 100001890836869132179
at org.apache.hama.graph.GraphJobRunner.iterate(GraphJobRunner.java:298)
at org.apache.hama.graph.GraphJobRunner.doSuperstep(GraphJobRunner.java:250)
at org.apache.hama.graph.GraphJobRunner.bsp(GraphJobRunner.java:145)
at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:171)
at org.apache.hama.bsp.BSPTask.run(BSPTask.java:144)
at org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1243)
14/07/02 10:38:41 DEBUG bsp.Counters: Adding TASK_OUTPUT_RECORDS

原因: 前期排序时由于最终调用Text的比较类时，原数据类型不匹配，导致比较的结果和实际不符。

原理:

1. PartionRunner首先调用SequenceFile自带的Sorter对各个文件内部排序，

SequenceFile调用MergeSort对单个文件进行快排(?)，直交换待比较数组的ID坐标，实际比较时为:

class SeqFileComparator implements Comparator<IntWritable> {
public int compare(IntWritable I, IntWritable J) {

return comparator.compare(rawBuffer, keyOffsets[I.get()],
keyLengths[I.get()], rawBuffer,
keyOffsets[J.get()], keyLengths[J.get()]);
}
}

//rawBuffer是rawKey产生的字节数组，keyOffsets是key的初始值, I即为顶点在数组的ID, 非常聪明的方法！

//即DataOutputBuffer rawKey; XXX ; byte[]rawBuffer = rawKeys.getData();

实际调用Text类，即测试用例的顶点类的比较方法，我修改了hadoop1.2-core的Text中比较方法以适应google的较长的边数据。同时，为了注意字节流和字符串的转换，有下面的方法，可理解rawData中的内容:

byte[] key1 = new byte[40] ;

System.arraycopy(b1,s1, key1 , 0, l1);

String k1 = new String(key1).trim();

BigDecimal bd1 = new BigDecimal(k1);

OK,这样就能看到该字节数组的内容了!

调试hama子进程时，类似hadoop. 不过hadoop中childjvm经常由于时间较短而来不及远程断点调试，hama则比较顺利。

<property>
<name>bsp.child.java.opts</name>
<value>-Xmx1024m -Xdebug -Xrunjdwp:transport=dt_socket,address=8792,server=y,suspend=y</value>//注意！ 这里参数添加时不能有多余的空格(如-Xmx1024m -Xdebug之间只能有一个空格!)，否则会报错！！！
<description>Java opts for the groom server child processes.
The following symbol, if present, will be interpolated: @taskid@ is replaced
by current TaskID. Any other occurrences of '@' will go unchanged.
For example, to enable verbose gc logging to a file named for the taskid in
/tmp and to set the heap maximum to be a gigabyte, pass a 'value' of:
-Xmx1024m -verbose:gc -Xloggc:/tmp/@taskid@.gc
The configuration variable bsp.child.ulimit can be used to control the
maximum virtual memory of the child processes.
</description>
</property>

注：这里测试最后发现，即使是对于Long型无法表示的类型数值变量，如需要BIgInteger的表示的，Text类型也可以正确比较大小，前面的基本上没用。。。。。。 教训了

从文件中删除固定行:

sed -i '/Love/d' 1.txt