Preface  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  ix

1. Introduction  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  1

2. HDFS  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  7

Goals and Motivation 7

Design 8

Daemons 9

Reading and Writing Data 11

The Read Path 12

The Write Path 13

Managing Filesystem Metadata 14

Namenode High Availability 16

Namenode Federation 18

Access and Integration 20

Command-Line Tools 20

FUSE 23

REST Support 23

3. MapReduce  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  25

The Stages of MapReduce 26

Introducing Hadoop MapReduce 33

Daemons 34

When It All Goes Wrong 36

YARN 37

4. Planning a Hadoop Cluster  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  41

Picking a Distribution and Version of Hadoop 41

Apache Hadoop 41

Cloudera’s Distribution Including Apache Hadoop 42

Versions and Features 42