AWS System Administration: Best Practices for Sysadmins in the Amazon Cloud / Edition 1

by Mike Ryan, Federico Lucifredi

Overview

With platforms designed for rapid adaptation and failure recovery such as Amazon Web Services, cloud computing is more like programming than traditional system administration. Tools for automatic scaling and instance replacement allow even small DevOps teams to manage massively scalable application infrastructures—if team members drop their old views of development and operations and start mastering automation.

This comprehensive guide shows developers and system administrators how to configure and manage AWS services including EC2, CloudFormation, Elastic Load Balancing, S3, and Route 53. Sysadms will learn will learn to automate their favorite tools and processes; developers will pick up enough ops knowledge to build a robust and resilient AWS application infrastructure.

• Launch instances with EC2 or CloudFormation
• Securely deploy and manage your applications with AWS tools
• Learn to automate AWS configuration management with Python and Puppet
• Deploy applications with Auto Scaling and Elastic Load Balancing
• Explore approaches for deploying application and infrastructure updates
• Save time on development and operations with reusable components
• Learn strategies for managing log files in AWS environments
• Configure a cloud-aware DNS service with Route 53
• Use AWS CloudWatch to monitor your infrastructure and applications

Product Details

ISBN-13:	9781449342579
Publisher:	O'Reilly Media, Incorporated
Publication date:	09/01/2018
Pages:	381
Product dimensions:	6.90(w) x 9.10(h) x 0.90(d)

About the Author

Mike has worked as a system administrator since the early 2000s in a variety of roles, including an ISP, at Greenpeace, and most recently a social media startup. In recent years, he have spent increasing amounts of time working with cloud services such as AWS, which has dovetailed perfectly with his love of automation and scalability.



As well as doing system administration as a day job, he is also building his own AWS-based startup. Cloudfrag provides on-demand game servers and relies heavily on a wide range of AWS services.

Federico Lucifredi was the lead Product Manager for Ubuntu Server, Amazon Web Services' most popular operating system. While at Canonical, Federico led the Certified Public Cloud program, ensuring the seamless integration of Ubuntu into AWS and other public clouds. He is currently the Ceph Product Management Director at Red Hat, and can be reached on Twitter as @0xF2.



Federico is a graduate of Boston College and Harvard University. He is a frequent speaker at user group and conference events.

Table of Contents

Foreword vii

Preface ix

1 Setting Up AWS Tools 1

Getting Started 2

Preparing Your Tools 3

Installing the AWS Command Line Interface 5

Parsing JSON Output with jq 9

Legacy AWS Command-Line Tools 10

Managing Your Costs 13

2 First Steps with EC2 and Cloud Formation 15

What Is an Instance? 16

Instance Types 18

Processing Power 18

Storage 21

Networking 23

Launching Instances 25

Launching from the Management Console 25

Launching with Command-Line Tools 34

Launching from Your Own Programs and Scripts 41

Introducing CloudFormation 45

Working with CloudFormation Stacks 47

Creating the Stack 47

Updating the Stack 50

Looking Before You Leap 53

Deleting the Stack 54

Which Method Should I Use? 55

Amazon Machine Images 56

Building Your Own AMI 59

Deregistering AMIs 63

Pets versus Cattle 67

3 Access Management and Security Groups 69

The AWS Security Model 69

Account Security Checklist 71

Multi-Factor Authentication 72

Identity and Access Management 72

Amazon Resource Names 73

IAM Policies 74

IAM Users and Groups 84

IAM Roles 98

Using IAM Roles from Other AWS Accounts 106

Using IAM in CloudFormation Stacks 107

Security Groups 112

Protecting Instances with SSH Whitelists 116

Virtual Private Networks and Security Groups 118

A Security State of Mind 126

4 Configuration Management 129

Why Use Configuration Management? 129

Ops Works 130

Choosing a Configuration Management Package 132

Puppet on AWS 133

A Quick Introduction to Puppet 133

Puppet and CloudFormation 141

User Data and Tags 155

Executing Tasks with Fabric 158

Masterless Puppet 161

Building AMIs with Packer 166

Automate All the Things 170

5 An Example Application Stack 171

Overview of Application Components 171

The Web Application 172

Database and Caching 172

Background Task Processing 172

Installing the Web Application 173

Preparing Puppet and CloudFormation 179

Puppet Files 179

CloudFormation Files 186

Creating an RDS Database 188

RDS: Updating Puppet and CloudFormation 194

Creating an ElastiCache Node 201

ElastiCache: Updating Puppet and CloudFormation 207

Installing Celery with Simple Queuing Service 209

Celery: Updating Puppet and CloudFormation 219

Building the AMIs 225

Creating the Stack with CloudFormation 227

Application Factory 228

6 Auto Scaling and Elastic Load Balancing 229

Static Auto Scaling Groups 231

Notifications of Scaling Activities 236

Scaling Policies 238

Scaling on CloudWatch Metrics 239

Elastic Load Balancing 245

Elastic Load Balancer and Auto Scaling Groups 246

ELB Health Checks 248

Managing Outages 250

Mastering Scale 253

7 Deployment Strategies 255

Instance-Based Deployments 255

Executing Code on Running Instances with Fabric 257

Updating Instances at Launch Time 262

AMI-Based Deployments 263

Deploying AMIs with CloudFormation 264

Deploying AMIs with the EC2 API 264

Web scale Thinking 265

Application Immutability 266

Takeaways 266

8 Building Reusable Components 269

The Importance of Being Reusable 269

Role-Based AMIs 270

Mapping Instances to Roles 272

Patterns for Configuration Management Tools 274

Modular CloudFormation Stacks 279

9 Log Management 283

Central Logging 283

Logstash Configuration 285

Logging to S3 295

AWS Service Logs 298

S3 Lifecycle Management 300

10 DNS with Route 53 303

Why Use Route 53? 304

Failure Is an Option: Service Failover with Route 53 305

Ramping Up Traffic 310

Surviving ELB and Application Outages with Route 53 311

Takeaways 317

11 Monitoring 319

Why Are You Monitoring? 319

Cloud Watch 320

CloudWatch Basics 321

Auto Scaling and Custom Metrics 323

Old Tools, New Tricks 329

12 Backups 335

Backing Up Static Files from EC2 Instances to S3 337

Rolling Backups with S3 and Glacier 339

PostgreSQL and Other Databases 344

pg_dump 345

Snapshots and Continuous Archiving 346

Off-Site Backups 351

Index 353