Jay Jacobs, Bob Rudis-Data-Driven Security  Analysis, Visualization and Dashboards-Wiley (2014).pdf
Contents
Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv
􀀤􀁉􀁂􀁑􀁕􀁆􀁓􀀁􀀒􀀁 􀁴􀀁 􀀵􀁉􀁆􀀁􀀫􀁐􀁖􀁓􀁏􀁆􀁚􀀁􀁕􀁐􀀁􀀥􀁂􀁕􀁂􀀎􀀥􀁓􀁊􀁗􀁆􀁏􀀁􀀴􀁆􀁄􀁖􀁓􀁊􀁕􀁚 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1
A Brief History of Learning from Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2
Nineteenth Century Data Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Twentieth Century Data Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Twenty-First Century Data Analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Gathering Data Analysis Skills. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5
Domain Expertise. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Programming Skills . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Data Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Visualization (a.k.a. Communication). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Combining the Skills . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Centering on a Question. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Creating a Good Research Question . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
Exploratory Data Analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Recommended Reading. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
􀀤􀁉􀁂􀁑􀁕􀁆􀁓􀀁􀀓􀀁 􀁴􀀁 􀀣􀁖􀁊􀁍􀁅􀁊􀁏􀁈􀀁􀀺􀁐􀁖􀁓􀀁􀀢􀁏􀁂􀁍􀁚􀁕􀁊􀁄􀁔􀀁􀀵􀁐􀁐􀁍􀁃􀁐􀁙􀀛􀀁􀀢􀀁􀀱􀁓􀁊􀁎􀁆􀁓􀀁􀁐􀁏􀀁
􀀶􀁔􀁊􀁏􀁈􀀁􀀳􀀁􀁂􀁏􀁅􀀁􀀱􀁚􀁕􀁉􀁐􀁏􀀁􀁇􀁐􀁓􀀁􀀴􀁆􀁄􀁖􀁓􀁊􀁕􀁚􀀁􀀢􀁏􀁂􀁍􀁚􀁔􀁊􀁔 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
Why Python? Why R? And Why Both?. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
Why Python? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
Why R? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
Why Both?. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
Jumpstarting Your Python Analytics with Canopy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
Understanding the Python Data Analysis and Visualization Ecosystem. . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Setting Up Your R Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
Introducing Data Frames . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
Organizing Analyses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
Recommended Reading. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
􀀤􀁉􀁂􀁑􀁕􀁆􀁓􀀁􀀔􀀁 􀁴􀀁 􀀭􀁆􀁂􀁓􀁏􀁊􀁏􀁈􀀁􀁕􀁉􀁆􀀁􀁩􀀩􀁆􀁍􀁍􀁐􀀁􀀸􀁐􀁓􀁍􀁅􀁷􀀁􀁐􀁇􀀁􀀴􀁆􀁄􀁖􀁓􀁊􀁕􀁚􀀁􀀥􀁂􀁕􀁂􀀁􀀢􀁏􀁂􀁍􀁚􀁔􀁊􀁔. . . . . . . . . . . . . . . . . . . . . 39
Solving a Problem. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
Getting Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Reading In Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
Exploring Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
Rather than have you gorge at an all-you-can-eat buffet, the chapters are more like tapas—each with
their own distinct flavor profiles and textures. Like the word tapas itself suggests, each chapter covers a
different foundational topic within security data science and provides plenty of pointers for further study.
Chapter 1 lays the foundation for the journey and provides examples of how other disciplines have
evolved into data-driven practices. It also provides an overview of the skills a security data scientist needs.
Chapters 2, 3, and 4 dive right into the tools, technologies, and basic techniques that should be
part of every security data scientists’ toolbox. You’ll work with AlienVault’s IP Reputation database (one of
the most thorough sources of malicious nodes publicly available) and take a macro look at the ZeuS and
ZeroAccess botnets. We introduce the analytical side of Python in Chapters 2 and 3. Then we thrust you
into the world of statistical analysis software with a major focus on the R language in the remainder of the
book. Unlike traditional introductory texts in R (or statistics in general), we use security data throughout the
book to help make the concepts as real and practical as possible for the information security professional.
Chapter 5 introduces some techniques for creating maps and introduces some core statistical concepts,
along with a lesson or two about extraterrestrial visitors.
Chapter 6 delves into the biological and cognitive science foundations of visual communication (data
visualization) and even shows you how to animate your security data.
This lays a foundation for learning how to analyze and visualize security breaches in Chapter 7, where
you’ll also have an opportunity to work with real incident data.
Chapter 8 covers modern database concepts with new tricks for traditional database deployments and
new tools with a range of NoSQL solutions discussed. You’ll also get tips on how to answer the question,
“Have we seen this IP address on our network?”
Chapter 9 introduces you to the exciting and relatively new world of machine learning. You’ll learn
about the core concepts and explore a handful of machine-learning techniques and develop a new
appreciation for how algorithms can pick up patterns that your intuition might never recognize.
Chapters 10 and 11 give you practical advice and techniques for building effective visualizations
that will both communicate and (hopefully) impress your consumers. You’ll use everything from Microsoft
Excel to state of the art tools and libraries, and be able to translate what you’ve learned outside of security.
Visualization concepts are made even more tangible through “makeovers” of security dashboards that
many of you may be familiar with.
Finally, we show you how to apply what you’ve learned at both a personal and organizational level in
Chapter 12.