Title An Architecture for Fast and General Data Processing on Large Clusters
Author Matei Zaharia
ISBN 9781970001563
List price USD 54.95
Price outside India Available on Request
Original price
Binding Paperback
No of pages 142
Book size 191 X 235 mm
Publishing year 2016
Original publisher Eurospan Publishers
Published in India by Eurospan Publishers
Exclusive distributors Viva Books Private Limited
Sales territory India, Sri Lanka, Bangladesh, Pakistan, Nepal, .
Status New Arrival
About the book Send Enquiry


The past few years have seen a major change in computing systems, as growing data volumes and stalling processor speeds require more and more applications to scale out to clusters. Today, a myriad data sources, from the Internet to business operations to scientific instruments, produce large and valuable data streams. However, the processing capabilities of single machines have not kept up with the size of data. As a result, organizations increasingly need to scale out their computations over clusters.

At the same time, the speed and sophistication required of data processing have grown. In addition to simple queries, complex algorithms like machine learning and graph analysis are becoming common. And in addition to batch processing, streaming analysis of real-time data is required to let organizations take timely action. Future computing platforms will need to not only scale out traditional workloads, but support these new applications too.

This book, a revised version of the 2014 ACM Dissertation Award winning dissertation, proposes an architecture for cluster computing systems that can tackle emerging data processing workloads at scale. Whereas early cluster computing systems, like MapReduce, handled batch processing, our architecture also enables streaming and interactive queries, while keeping MapReduce’s scalability and fault tolerance. And whereas most deployed systems only support simple one-pass computations (e.g., SQL queries), ours also extends to the multi-pass algorithms required for complex analytics like machine learning. Finally, unlike the specialized systems proposed for some of these workloads, our architecture allows these computations to be combined, enabling rich new applications that intermix, for example, streaming and batch processing.

We achieve these results through a simple extension to MapReduce that adds primitives for data sharing, called Resilient Distributed Datasets (RDDs). We show that this is enough to capture a wide range of workloads. We implement RDDs in the open source Spark system, which we evaluate using synthetic and real workloads. Spark matches or exceeds the performance of specialized systems in many domains, while offering stronger fault tolerance properties and allowing these workloads to be combined. Finally, we examine the generality of RDDs from both a theoretical modeling perspective and a systems perspective.

This version of the dissertation makes corrections throughout the text and adds a new section on the evolution of Apache Spark in industry since 2014. In addition, editing, formatting, and links for the references have been added.



Chapter 1: Introduction • Problems with Specialized Systems • Resilient Distributed Datasets (RDDs) • Models Implemented over RDDs • Summary of Results • Book Overview

Chapter 2: Resilient Distributed Datasets • Introduction • RDD Abstraction • Spark Programming Interface • Representing RDDs • Implementation • Evaluation • Discussion • Related Work • Summary

Chapter 3: Models Built over RDDs • Introduction • Techniques for Implementing Other Models on RDDs • Shark: SQL on RDDs • Implementation • Performance • Combining SQL with Complex Analytics • Summary

Chapter 4: Discretized Streams • Introduction • Goals and Background • Discretized Streams (D-Streams) • System Architecture • Fault and Straggler Recovery • Evaluation • Discussion • Related Work • Summary

Chapter 5: Generality of RDDs • Introduction • Expressiveness Perspective • Systems Perspective • Limitations and Extensions • Related Work • Summary

Chapter 6: Conclusion • Lessons Learned • Evolution of Spark in Industry • Future Work


Author’s Biography

About the Author:

Matei Zaharia received his Bachelor’s degree from the University of Waterloo in 2007 and his PhD from UC Berkeley in 2013. At Berkeley, he worked with Scott Shenker and Ion Stoica on topics in cloud computing, networking, and largescale data processing. Throughout his research, he has contributed to a variety of open source projects including Apache Hadoop, Mesos, and Spark. Matei is currently an assistant professor at MIT and CTO at Databricks, the company founded by the team that started Apache Spark.

Target Audience:

Students and Academicians of architecture for cluster computing system.

Special prices are applicable to the authorised sales territory only.
Prices are subject to change without prior notice.