Imagine if instead of individual physical servers, we could aggregate all the resources in a data center into a single large virtual pool, exposing not virtual machines but primitives such as CPU, RAM, and I/O? In conjunction, imagine if we could break applications into small isolated units of tasks that could be dynamically assigned resources from our virtual data center pool, based on the needs of the applications in our data center? The analogy here would be a PC with an operating system that is pooling the PC’s processors and RAM and coordinating the allocation and deallocation of those resources for use by different processes. Now extend that analogy to make the data center the PC with Mesos as the operating system kernel. That, in a nutshell, is how Mesos is transforming the data center and making true SDDC a reality.
The modified diagram above from the Apache Mesos website shows how Mesos implements it’s two-level scheduling architecture for managing multiple types of applications. The first level is the master daemon which manages slave daemons running on each node in the Mesos cluster. The cluster consists of all servers, physical or virtual, that will be running applications tasks, such as Hadoop and MPI jobs. The second level consists of a component called a framework. A framework includes a scheduler and an executor process, the latter of which also runs on each node. Mesos is able to communicate with different types of frameworks with each one managing a different clustered application. The diagram above shows Hadoop and MPI but other frameworks have been written as well for other types of applications.
A framework running on top of Mesos consists of two components: a scheduler that registers with the master to be offered resources, and an executor process that is launched on slave nodes to run the framework’s tasks (see the App/Framework development guide for more details about application schedulers and executors). While the master determines how many resources are offered to each framework, the frameworks’ schedulers select which of the offered resources to use. When a frameworks accepts offered resources, it passes to Mesos a description of the tasks it wants to run on them. In turn, Mesos launches the tasks on the corresponding slaves.