转自;http://www.quora.com/How-does-YARN-compare-to-Mesos
Both systems have the same goal: allowing you to share a large cluster of machines between different frameworks.
For those who don't know, NextGen MapReduce is a project to factor the existing MapReduce into a generic layer that handles distributed process execution and resource scheduling (this system is called YARN) and
then implement MapReduce as an "application" on top of this.
Mesos was originally an academic research project with a very similar goal. They created a system which could run a patched version of Hadoop, MPI and other things. This has grown into an Apache Incubator project
in its own right.
I have been looking into these two a bit because we would love something like this at LinkedIn, and the nature of these things is that you really only want one (since you want to run everything on it). So at
the moment we don't have any real experience running stuff on top of either of these, but here is what I have pieced together (may be wrong in places):
- Nextgen MapReduce (aka YARN) is primarily written in java with bits of native code. Mesos is primarily written in C++.
- YARN only handles memory scheduling (e.g. you request x containers of y MB each), but with plans to extend it to other resources. I believe Mesos handles both memory and CPU scheduling, but I don't know the details. In practice I think the OS handles CPU
scheduling pretty well so I am not sure that would help our use cases. Supporting some kind of disk space and disk I/O scheduling and enforcement would be super cool, but I don't think either do that (yet).
- Mesos uses Linux container groups (http://lxc.sourceforge.n<wbr>et</wbr>), and YARN uses simple unix processes. Linux container
groups are a stronger isolation but may have some additional overhead.
- The resource request model is weirdly backwards in Mesos. In YARN you (the framework) request containers with a given specification and give locality preferences. In Mesos you get resource "offers" and choose to accept or reject those based on your own
scheduling policy. The Mesos model is a arguably more flexible, but seemingly more work for the person implementing the framework.
- YARN is a pretty epic chunk of code, including all kinds of things right down to its own web framework. It is about 3x as much code as Mesos.
- YARN integrates something similar to the pluggable schedulers everyone knows and loves/hates in Hadoop. So if you are used to the capacity scheduler, hierarchical queues, and all that, you can get something similar. I don't think the Mesos scheduling capabilities
are quite as robust (they list hierarchical scheduling on their roadmap).
- YARN integrates with Kerberos and essentially inherits the Hadoop security architecture. I don't think Mesos attempts to deal with security.
- YARN directly handles rack and machine locality in your requests, which is convenient. In Mesos you can implement this, but it is less out of the box.
- Mesos is much more mature as a project at this point. It is a standalone thing, with great documentation, and good starter examples. YARN exists only on hadoop trunk (and some feature branches) in the mapreduce directory, and the docs are super sparse.
That said, the Hadoop guys have been really awesome at helping us get started with YARN (thanksArun!) and they seem really committed to making sure it works
as a general purpose framework, not just for Hadoop. There seems to be a lot of momentum, it is just early.
- YARN is going to be the basis for Hadoop MapReduce going forward, so if you have a big Hadoop cluster and want to be able to run other stuff on it, that is likely appealing and will probably work more transparently than Mesos.
- YARN was written by the Yahoo/HortonWorks Hadoop team which has should know a thing or two about multi-tenancy and very large-scale cluster computing. YARN is not yet in a stable Hadoop release so I am not sure how much actual testing it has had or the
extent of deployment internally at Yahoo. Regardless, if/when the YARN team is able to get the majority of the worlds Hadoop clusters successfully running on top of YARN, that will likely get the project to a level of hardening that will be hard to compete
with.
- Mesos ships with a number of out-of-the-box frameworks ported to it. This somewhat helps to validate the generality of their framework, but i don't know how much of a hack the various ports of things to it are.
Here are a few pointers for folks trying to find out more about Mesos:
- Docs:http://www.mesosproject.o<wbr>rg/docu...</wbr>
- Papers:http://www.mesosproject.o<wbr>rg/rese...</wbr>
- Sample framework implementations:https://github.com/mesos/<wbr>mesos/t...</wbr>
Here are some pointers on YARN:
- Master JIRA:https://issues.apache.org<wbr>/jira/b...</wbr>
- Article on the new resource scheduler:http://developer.yahoo.co<wbr>m/blogs...</wbr>
- Design document for YARN. This is really essential for understanding their terminology of application masters, resource manager, etc. Before we found this, just looking at code, we were lost.https://issues.apache.org<wbr>/jira/s...</wbr>
- Spark, an iterative machine learning framework, has been ported to YARN, and serves as a great example of how to do this:https://github.com/mesos/<wbr>spark-yarn</wbr>
There is a thread on the Mesos mailing list that discusses differences further:http://mail-archives.apac<wbr>he.org/...</wbr>
分享到:
相关推荐
为此,7月2日晚,在CSDNSpark高端微信群中,一场基于YARN和Mesos的讨论被拉开,主要参与分享的嘉宾包括TalkingData研发副总裁阎志涛,GrowingIO田毅,AdMaster技术副总裁卢亿雷,SparkCommitter、Mesos/...
Yarn essential resource management, hands on cookbook on how to use yarn configurations to optimize resources
赠送jar包:hadoop-yarn-api-2.5.1.jar; 赠送原API文档:hadoop-yarn-api-2.5.1-javadoc.jar; 赠送源代码:hadoop-yarn-api-2.5.1-sources.jar; 赠送Maven依赖信息文件:hadoop-yarn-api-2.5.1.pom; 包含翻译后...
npm install -g yarn yarn install 安装失败,使用官方下载的yarn.lock文件
Work through steps to scale your machine learning models and deploy them into a standalone cluster, EC2, YARN, and Mesos. Finally dip into the powerful options presented by Spark Streaming, and ...
现在官网下载缓慢,这里可以直接下载安装,所以分享给大家,以供直接下载,版本号 yarn-1.0.2.msi
yarn 前端打包工具指定配置文件
赠送jar包:hadoop-yarn-client-2.6.5.jar; 赠送原API文档:hadoop-yarn-client-2.6.5-javadoc.jar; 赠送源代码:hadoop-yarn-client-2.6.5-sources.jar; 赠送Maven依赖信息文件:hadoop-yarn-client-2.6.5.pom;...
YARN Essentials
详细介绍了Hadoop2.x的资源管理框架yarn,内容丰富,很有帮助。
YARN配置、启动与验证 YARN配置、启动与验证 序号 任务名称 任务一 YARN组件参数配置 任务二 MapReduce组件参数配置 任务三 配置SSH无密钥登录(slave1为主节点) 任务四 分发YARN与MapReduce配置文件 任务五 启动...
现在官网下载缓慢,这里可以直接下载最新版本安装,版本号 yarn-1.22.0.msi yarn是facebook发布的一款取代npm的包管理工具。 快速、可靠、安全的依赖管理工具
Yarn是 Facebook 提供的替代 npm 的工具具=
Yarn 对你的代码来说是一个包管理器, 你可以通过它使用全世界开发者的代码,或者分享自己的代码。 Yarn 做这些快捷、安全、可靠,所以你不用担心什么。 通过Yarn你可以使用其他开发者针对不同问题的解决方案,使...
yarn架构详解,关于大数据yarn资源调度的详解
yarn安装包给不能下载的用户提供下载,欢迎下载。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。
09丨为什么我们管Yarn叫作资源调度框架?.html
yarn编码,实现Application列表查询,Application的kill操作。以及Job的查询,map和reduce个数查询
yarn安装.docx