Designing for Scalability with Erlang/OTP: Implement Robust, Fault-Tolerant Systems

by Francesco Cesarini, Steve Vinoski

Overview

If you need to build a scalable, fault tolerant system with requirements for high availability, discover why the Erlang/OTP platform stands out for the breadth, depth, and consistency of its features. This hands-on guide demonstrates how to use the Erlang programming language and its OTP framework of reusable libraries, tools, and design principles to develop complex commercial-grade systems that simply cannot fail.

In the first part of the book, you’ll learn how to design and implement process behaviors and supervision trees with Erlang/OTP, and bundle them into standalone nodes. The second part addresses reliability, scalability, and high availability in your overall system design. If you’re familiar with Erlang, this book will help you understand the design choices and trade-offs necessary to keep your system running.

• Explore OTP’s building blocks: the Erlang language, tools and libraries collection, and its abstract principles and design rules
• Dive into the fundamentals of OTP reusable frameworks: the Erlang process structures OTP uses for behaviors
• Understand how OTP behaviors support client-server structures, finite state machine patterns, event handling, and runtime/code integration
• Write your own behaviors and special processes
• Use OTP’s tools, techniques, and architectures to handle deployment, monitoring, and operations

Product Details

ISBN-13:	9781449320737
Publisher:	O'Reilly Media, Incorporated
Publication date:	06/03/2016
Pages:	480
Product dimensions:	7.00(w) x 9.00(h) x 1.00(d)

About the Author

Francesco Cesarini has taught Erlang/OTP for fourteen years to all parties involved in the software cycle, including students, developers, support and testers, as well as project and technical managers. He has helped set up Erlang development centres in Ireland, the US and the UK. In 1999, the year after Erlang was released as open source, Francesco moved from Sweden to London and founded Erlang Training and Consulting, a company which he is a majority shareholder of today.

Steve Vinoski has spent most of his software development career, spanning over more than 30 years, working in the areas of middleware and distributed computing systems. He discovered Erlang in 2006 after nearly 20 years of developing middleware systems primarily in C++ and Java, and he's used Erlang as his primary development language ever since. Steve has contributed to a variety of Erlang projects, including the Riak database, developed and maintained by his employer, Basho Technologies, and the Yaws web server. He's also contributed dozens of bug-fix and feature patches to the Erlang/OTP codebase.



Steve is also a long-time author, having written or co-authored over 100 published articles and papers covering middleware, distributed systems, and web development, as well as a couple books. He wrote "The Functional Web" column for IEEE Internet Computing (IC) magazine from 2008 through 2012, and prior to that, from 2002-2008, wrote the "Toward Integration" column for IC as well. He also serves on the magazine’s editorial board. From 1995-2005, Steve co-authored the popular “Object Interconnections” column on distributed object computing for the C++ Report and later the C/C++ Users Journal. Over the years Steve has also given hundreds of conference and workshop presentations and tutorials on middleware, distributed systems, web development, and programming languages, and has served as chair or program committee member for many dozens of conferences and workshops.

Table of Contents

Preface xiii

1 Introduction 1

Defining the Problem 2

OTP 4

Erlang 6

Tools and Libraries 7

System Design Principles 10

Erlang Nodes 11

Distribution, Infrastructure, and Multicore 12

Summing Up 13

What You'll Learn in This Book 14

2 Introducing Erlang 21

Recursion and Pattern Matching 21

Functional Influence 25

Fun with Anonymous Functions 25

List Comprehensions: Generate and Test 27

Processes and Message Passing 29

Fail Safe! 33

Links and Monitors for Supervision 34

Links 35

Monitors 37

Records 38

Maps 41

Macros 42

Upgrading Modules 43

ETS: Erlang Term Storage 45

Distributed Erlang 48

Naming and Communication 48

Node Connections and Visibility 49

Summing Up 51

What's Next? 51

3 Behaviors 53

Process Skeletons 53

Design Patterns 56

Callback Modules 57

Extracting Generic Behaviors 60

Starting the Server 62

The Client Functions 64

The Server Loop 66

Functions Internal to the Server 68

The Generic Server 69

Message Passing: Under the Hood 72

Summing Up 76

What's Next? 76

4 Generic Servers 77

Generic Servers 77

Behavior Directives 78

Starting a Server 80

Message Passing 82

Synchronous Message Passing 82

Asynchronous Message Passing 84

Other Messages 85

Unhandled Messages 86

Synchronizing Clients 88

Termination 89

Call Timeouts 91

Deadlocks 94

Generic Server Timeouts 95

Hibernating Behaviors 97

Going Global 97

Linking Behaviors 99

Summing Up 99

What's Next? 100

5 Controlling OTP Behaviors 101

The sys Module 101

Tracing and Logging 101

System Messages 103

Your Own Trace Functions 103

Statistics, Status, and State 105

The sys Module Recap 107

Spawn Options 108

Memory Management and Garbage Collection 109

Spawn Options to Avoid 113

Timeouts 114

Summing Up 114

What's Next? 114

6 Finite State Machines 117

Finite State Machines the Erlang Way 118

Coffee FSM 119

The Hardware Stub 121

The Erlang Coffee Machine 122

Generic FSMs 125

A Behavior Example 127

Starting the FSM 127

Sending Events 131

Termination 139

Summing Up 141

Get Your Hands Dirty 141

The Phone Controllers 142

Let's Test It 143

What's Next? 145

7 Event Handlers 147

Events 147

Generic Event Managers and Handlers 149

Starting and Stopping Event Managers 149

Adding Event Handlers 150

Deleting an Event Handler 152

Sending Synchronous and Asynchronous Events 153

Retrieving Data 156

Handling Errors and Invalid Return Values 158

Swapping Event Handlers 160

Wrapping It All Up 162

The SASL Alarm Handler 165

Summing Up 166

What's Next? 167

8 Supervisors 169

Supervision Trees 170

OTP Supervisors 174

The Supervisor Behavior 175

Starting the Supervisor 176

The Supervisor Specification 179

Dynamic Children 186

Non-OTP-Compliant Processes 194

Scalability and Short-Lived Processes 195

Synchronous Starts for Determinism 197

Testing Your Supervision Strategy 199

How Does This Compare? 200

Summing Up 201

What's Next? 201

9 Applications 203

How Applications Run 204

The Application Structure 206

The Callback Module 209

Starting and Stopping Applications 210

Application Resource Files 213

The Base Station Controller Application File 215

Starting an Application 216

Environment Variables 219

Application Types and Termination Strategies 221

Distributed Applications 222

Start Phases 226

Included Applications 228

Start Phases in Included Applications 228

Combining Supervisors and Applications 230

The SASL Application 231

Progress Reports 236

Error Reports 236

Crash Reports 237

Supervisor Reports 238

Summing Up 239

What's Next? 240

10 Special Processes and Your Own Behaviors 241

Special Processes 242

The Mutex 242

Starting Special Processes 244

The Mutex States 247

Handling Exits 247

System Messages 249

Trace and Log Events 250

Putting It Together 251

Dynamic Modules and Hibernating 255

Your Own Behaviors 255

Rules for Creating Behaviors 256

An Example Handling TCP Streams 256

Summing Up 260

What's Next? 261

11 System Principles and Release Handling 263

System Principles 264

Release Directory Structure 265

Release Resource Files 269

Creating a Release 273

Creating the Boot File 274

Creating a Release Package 283

Start Scripts and Configuring on the Target 287

Arguments and Flags 290

The init Module 302

Rebar3 303

Generating a Rebar3 Release Project 304

Creating a Release with Rebar3 308

Rebar3 Releases with Project Dependencies 310

Wrapping Up 312

What's Next? 316

12 Release Upgrades 317

Software Upgrades 318

The First Version of the Coffee FSM 320

Adding a State 323

Creating a Release Upgrade 326

The Code to Upgrade 330

Application Upgrade Files 333

High-level Instructions 337

Release Upgrade Files 339

Low-Level Instructions 342

Installing an Upgrade 343

The Release Handler 346

Upgrading Environment Variables 350

Upgrading Special Processes 350

Upgrading in Distributed Environments 351

Upgrading the Emulator and Core Applications 352

Upgrades with Rebar3 353

Summing Up 356

What's Next? 358

13 Distributed Architectures 359

Node Types and Families 360

Networking 363

Distributed Erlang 366

Sockets and SSL 373

Service Orientation and Microservices 375

Peer to Peer 376

Interfaces 378

Summing Up 380

What's Next? 381

14 Systems That Never Stop 383

Availability 383

Fault Tolerance 384

Resilience 385

Reliability 387

Sharing Data 392

Tradeoffs Between Consistency and Availability 400

Summing Up 401

What's Next? 403

15 Scaling Out 405

Horizontal and Vertical Scaling 405

Capacity Planning 409

Capacity Testing 412

Balancing Your System 414

Finding Bottlenecks 416

System Blueprints 419

Load Regulation and Backpressure 419

Summing Up 422

What's Next? 424

16 Monitoring and Preemptive Support 425

Monitoring 426

Logs 428

Metrics 433

Alarms 436

Preemptive Support 439

Summing Up 441

What's Next? 443

Index 445