yarn historyserver 使用解析

启动服务:

sbin/mr-jobhistory-daemon.sh start historyserver

查询历史页面:

http://202.117.10.25:19888/jobhistory

1. job完成后，历史信息存入hdfs下的mr-history/done/中，

执行命令分析历史信息:

hadoop job -historyall hdfs://202.117.10.26:9000/mr-history/done/2013/01/31/000000/job_1359637210567_0001-1359637233266-liuqiang-word+count-1359637716126-16-1-SUCCEEDED-default.jhist>result.txt

即可。

结果如下:

Hadoop job: job_1361498589930_0003
=====================================
User: liuqiang
JobName: word count
JobConf: hdfs://202.117.10.26:9000/tmp/hadoop-yarn/staging/liuqiang/.staging/job_1361498589930_0003/job.xml
Submitted At: 22-Feb-2013 10:23:51
Launched At: 22-Feb-2013 10:27:42 (3mins, 51sec)
Finished At: 22-Feb-2013 10:32:00 (4mins, 17sec)
Status: SUCCEEDED
Counters:

|Group Name |Counter name |Map Value |Reduce Value|Total Value|
---------------------------------------------------------------------------------------
|File System Counters |FILE: Number of bytes read |66,490,968|18,230,892|84,721,860
|File System Counters |FILE: Number of bytes written |85,744,950|18,294,425|104,039,375
|File System Counters |FILE: Number of read operations|0 |0 |0
|File System Counters |FILE: Number of large read operations|0 |0 |0
|File System Counters |FILE: Number of write operations|0 |0 |0
|File System Counters |HDFS: Number of bytes read |1,035,668,512|0 |1,035,668,512
|File System Counters |HDFS: Number of bytes written |0 |947,869 |947,869
|File System Counters |HDFS: Number of read operations|48 |3 |51
|File System Counters |HDFS: Number of large read operations|0 |0 |0
|File System Counters |HDFS: Number of write operations|0 |2 |2
|Job Counters |Killed map tasks |0 |0 |10
|Job Counters |Launched map tasks |0 |0 |26
|Job Counters |Launched reduce tasks |0 |0 |1
|Job Counters |Data-local map tasks |0 |0 |20
|Job Counters |Rack-local map tasks |0 |0 |6
|Job Counters |Total time spent by all maps in occupied slots (ms)|0 |0 |4,317,327
|Job Counters |Total time spent by all reduces in occupied slots (ms)|0 |0 |196,704
|Map-Reduce Framework |Map input records |9,383,921 |0 |9,383,921
|Map-Reduce Framework |Map output records |177,316,480|0 |177,316,480
|Map-Reduce Framework |Map output bytes |1,753,046,400|0 |1,753,046,400
|Map-Reduce Framework |Map output materialized bytes |18,230,862|0 |18,230,862
|Map-Reduce Framework |Input split bytes |1,952 |0 |1,952
|Map-Reduce Framework |Combine input records |181,455,220|0 |181,455,220
|Map-Reduce Framework |Combine output records |5,311,383 |0 |5,311,383
|Map-Reduce Framework |Reduce input groups |0 |68,979 |68,979
|Map-Reduce Framework |Reduce shuffle bytes |0 |18,230,862|18,230,862
|Map-Reduce Framework |Reduce input records |0 |1,172,643 |1,172,643
|Map-Reduce Framework |Reduce output records |0 |68,979 |68,979
|Map-Reduce Framework |Spilled Records |5,449,341 |1,172,643 |6,621,984
|Map-Reduce Framework |Shuffled Maps |0 |16 |16
|Map-Reduce Framework |Failed Shuffles |0 |0 |0
|Map-Reduce Framework |Merged Map outputs |0 |16 |16
|Map-Reduce Framework |GC time elapsed (ms) |260,716 |364 |261,080
|Map-Reduce Framework |CPU time spent (ms) |1,808,840 |7,850 |1,816,690
|Map-Reduce Framework |Physical memory (bytes) snapshot|3,380,711,424|76,144,640|3,456,856,064
|Map-Reduce Framework |Virtual memory (bytes) snapshot|6,272,446,464|352,477,184|6,624,923,648
|Map-Reduce Framework |Total committed heap usage (bytes)|2,952,212,480|40,366,080|2,992,578,560
|Shuffle Errors |BAD_ID |0 |0 |0
|Shuffle Errors |CONNECTION |0 |0 |0
|Shuffle Errors |IO_ERROR |0 |0 |0
|Shuffle Errors |WRONG_LENGTH |0 |0 |0
|Shuffle Errors |WRONG_MAP |0 |0 |0
|Shuffle Errors |WRONG_REDUCE |0 |0 |0
|File Input Format Counters |Bytes Read |1,035,666,560|0 |1,035,666,560
|File Output Format Counters |Bytes Written |0 |947,869 |947,869
|Job Counters |Killed map tasks |0 |0 |10
|Job Counters |Launched map tasks |0 |0 |26
|Job Counters |Launched reduce tasks |0 |0 |1
|Job Counters |Data-local map tasks |0 |0 |20
|Job Counters |Rack-local map tasks |0 |0 |6
|Job Counters |Total time spent by all maps in occupied slots (ms)|0 |0 |4,317,327
|Job Counters |Total time spent by all reduces in occupied slots (ms)|0 |0 |196,704
|File System Counters |FILE: Number of bytes read |66,490,968|18,230,892|84,721,860
|File System Counters |FILE: Number of bytes written |85,744,950|18,294,425|104,039,375
|File System Counters |FILE: Number of read operations|0 |0 |0
|File System Counters |FILE: Number of large read operations|0 |0 |0
|File System Counters |FILE: Number of write operations|0 |0 |0
|File System Counters |HDFS: Number of bytes read |1,035,668,512|0 |1,035,668,512
|File System Counters |HDFS: Number of bytes written |0 |947,869 |947,869
|File System Counters |HDFS: Number of read operations|48 |3 |51
|File System Counters |HDFS: Number of large read operations|0 |0 |0
|File System Counters |HDFS: Number of write operations|0 |2 |2
|Map-Reduce Framework |Map input records |9,383,921 |0 |9,383,921
|Map-Reduce Framework |Map output records |177,316,480|0 |177,316,480
|Map-Reduce Framework |Map output bytes |1,753,046,400|0 |1,753,046,400
|Map-Reduce Framework |Map output materialized bytes |18,230,862|0 |18,230,862
|Map-Reduce Framework |Input split bytes |1,952 |0 |1,952
|Map-Reduce Framework |Combine input records |181,455,220|0 |181,455,220
|Map-Reduce Framework |Combine output records |5,311,383 |0 |5,311,383
|Map-Reduce Framework |Reduce input groups |0 |68,979 |68,979
|Map-Reduce Framework |Reduce shuffle bytes |0 |18,230,862|18,230,862
|Map-Reduce Framework |Reduce input records |0 |1,172,643 |1,172,643
|Map-Reduce Framework |Reduce output records |0 |68,979 |68,979
|Map-Reduce Framework |Spilled Records |5,449,341 |1,172,643 |6,621,984
|Map-Reduce Framework |Shuffled Maps |0 |16 |16
|Map-Reduce Framework |Failed Shuffles |0 |0 |0
|Map-Reduce Framework |Merged Map outputs |0 |16 |16
|Map-Reduce Framework |GC time elapsed (ms) |260,716 |364 |261,080
|Map-Reduce Framework |CPU time spent (ms) |1,808,840 |7,850 |1,816,690
|Map-Reduce Framework |Physical memory (bytes) snapshot|3,380,711,424|76,144,640|3,456,856,064
|Map-Reduce Framework |Virtual memory (bytes) snapshot|6,272,446,464|352,477,184|6,624,923,648
|Map-Reduce Framework |Total committed heap usage (bytes)|2,952,212,480|40,366,080|2,992,578,560

=====================================

Task Summary
============================
KindTotalSuccessfulFailedKilledStartTimeFinishTime

Setup0000
Map261601022-Feb-2013 10:27:4522-Feb-2013 10:31:53 (4mins, 8sec)
Reduce110022-Feb-2013 10:28:4322-Feb-2013 10:32:00 (3mins, 16sec)
Cleanup0000
============================

Analysis
=========

Time taken by best performing map task task_1361498589930_0003_m_000015: 55sec
Average time taken by map tasks: 2mins, 58sec
Worse performing map tasks:
TaskIdTimetaken
task_1361498589930_0003_m_000004 4mins, 8sec
task_1361498589930_0003_m_000007 4mins, 7sec
task_1361498589930_0003_m_000013 4mins, 4sec
task_1361498589930_0003_m_000003 4mins, 3sec
task_1361498589930_0003_m_000006 3mins, 24sec
task_1361498589930_0003_m_000012 3mins, 23sec
task_1361498589930_0003_m_000009 3mins, 23sec
task_1361498589930_0003_m_000008 3mins, 21sec
task_1361498589930_0003_m_000005 3mins, 20sec
task_1361498589930_0003_m_000001 3mins, 20sec
The last map task task_1361498589930_0003_m_000004 finished at (relative to the Job launch time): 22-Feb-2013 10:31:53 (4mins, 11sec)

Time taken by best performing shuffle task task_1361498589930_0003_r_000000: 3mins, 10sec
Average time taken by shuffle tasks: 3mins, 10sec
Worse performing shuffle tasks:
TaskIdTimetaken
task_1361498589930_0003_r_000000 3mins, 10sec
The last shuffle task task_1361498589930_0003_r_000000 finished at (relative to the Job launch time): 22-Feb-2013 10:31:54 (4mins, 12sec)

Time taken by best performing reduce task task_1361498589930_0003_r_000000: 5sec
Average time taken by reduce tasks: 5sec
Worse performing reduce tasks:
TaskIdTimetaken
task_1361498589930_0003_r_000000 5sec
The last reduce task task_1361498589930_0003_r_000000 finished at (relative to the Job launch time): 22-Feb-2013 10:32:00 (4mins, 17sec)
=========

KILLED task attempts by nodes
HostnameFailedTasks
===============================
node13task_1361498589930_0003_m_000007, task_1361498589930_0003_m_000012, task_1361498589930_0003_m_000013,
node14task_1361498589930_0003_m_000003, task_1361498589930_0003_m_000004,
node21task_1361498589930_0003_m_000000, task_1361498589930_0003_m_000002, task_1361498589930_0003_m_000010, task_1361498589930_0003_m_000011,
node12task_1361498589930_0003_m_000006,