Table of Contents
About the Authors; Preface; Conventions Used in This Book; Using Code Examples; Safari® Books Online; How to Contact Us; Acknowledgments; Chapter 1: Setting the Pace: What Is Bad Data?; Chapter 2: Is It Just Me, or Does This Data Smell Funny?; 2.1 Understand the Data Structure; 2.2 Field Validation; 2.3 Value Validation; 2.4 Physical Interpretation of Simple Statistics; 2.5 Visualization; 2.6 Keyword PPC Example; 2.7 Search Referral Example; 2.8 Recommendation Analysis; 2.9 Time Series Data; 2.10 Conclusion; Chapter 3: Data Intended for Human Consumption, Not Machine Consumption; 3.1 The Data; 3.2 The Problem: Data Formatted for Human Consumption; 3.3 The Solution: Writing Code; 3.4 Postscript; 3.5 Other Formats; 3.6 Summary; Chapter 4: Bad Data Lurking in Plain Text; 4.1 Which Plain Text Encoding?; 4.2 Guessing Text Encoding; 4.3 Normalizing Text; 4.4 Problem: Application-Specific Characters Leaking into Plain Text; 4.5 Text Processing with Python; 4.6 Exercises; Chapter 5: (Re)Organizing the Web’s Data; 5.1 Can You Get That?; 5.2 General Workflow Example; 5.3 The Real Difficulties; 5.4 The Dark Side; 5.5 Conclusion; Chapter 6: Detecting Liars and the Confused in Contradictory Online Reviews; 6.1 Weotta; 6.2 Getting Reviews; 6.3 Sentiment Classification; 6.4 Polarized Language; 6.5 Corpus Creation; 6.6 Training a Classifier; 6.7 Validating the Classifier; 6.8 Designing with Data; 6.9 Lessons Learned; 6.10 Summary; 6.11 Resources; Chapter 7: Will the Bad Data Please Stand Up?; 7.1 Example 1: Defect Reduction in Manufacturing; 7.2 Example 2: Who’s Calling?; 7.3 Example 3: When “Typical” Does Not Mean “Average”; 7.4 Lessons Learned; 7.5 Will This Be on the Test?; Chapter 8: Blood, Sweat, and Urine; 8.1 A Very Nerdy Body Swap Comedy; 8.2 How Chemists Make Up Numbers; 8.3 All Your Database Are Belong to Us; 8.4 Check, Please; 8.5 Live Fast, Die Young, and Leave a Good-Looking Corpse Code Repository; 8.6 Rehab for Chemists (and Other Spreadsheet Abusers); 8.7 tl;dr; Chapter 9: When Data and Reality Don’t Match; 9.1 Whose Ticker Is It Anyway?; 9.2 Splits, Dividends, and Rescaling; 9.3 Bad Reality; 9.4 Conclusion; Chapter 10: Subtle Sources of Bias and Error; 10.1 Imputation Bias: General Issues; 10.2 Reporting Errors: General Issues; 10.3 Other Sources of Bias; 10.4 Conclusions; 10.5 References ; Chapter 11: Don’t Let the Perfect Be the Enemy of the Good: Is Bad Data Really Bad?; 11.1 But First, Let’s Reflect on Graduate School …; 11.2 Moving On to the Professional World; 11.3 Moving into Government Work; 11.4 Government Data Is Very Real; 11.5 Service Call Data as an Applied Example; 11.6 Moving Forward; 11.7 Lessons Learned and Looking Ahead; Chapter 12: When Databases Attack: A Guide for When to Stick to Files; 12.1 History; 12.2 Consider Files as Your Datastore; 12.3 File Concepts; 12.4 A Web Framework Backed by Files; 12.5 Reflections; Chapter 13: Crouching Table, Hidden Network; 13.1 A Relational Cost Allocations Model; 13.2 The Delicate Sound of a Combinatorial Explosion…; 13.3 The Hidden Network Emerges; 13.4 Storing the Graph; 13.5 Navigating the Graph with Gremlin; 13.6 Finding Value in Network Properties; 13.7 Think in Terms of Multiple Data Models and Use the Right Tool for the Job; 13.8 Acknowledgments; Chapter 14: Myths of Cloud Computing; 14.1 Introduction to the Cloud; 14.2 What Is “The Cloud”?; 14.3 The Cloud and Big Data; 14.4 Introducing Fred; 14.5 At First Everything Is Great; 14.6 They Put 100% of Their Infrastructure in the Cloud; 14.7 As Things Grow, They Scale Easily at First; 14.8 Then Things Start Having Trouble; 14.9 They Need to Improve Performance; 14.10 Higher IO Becomes Critical; 14.11 A Major Regional Outage Causes Massive Downtime; 14.12 Higher IO Comes with a Cost; 14.13 Data Sizes Increase; 14.14 Geo Redundancy Becomes a Priority; 14.15 Horizontal Scale Isn’t as Easy as They Hoped; 14.16 Costs Increase Dramatically; 14.17 Fred’s Follies; 14.18 Myth 1: Cloud Is a Great Solution for All Infrastructure Components; 14.19 Myth 2: Cloud Will Save Us Money; 14.20 Myth 3: Cloud IO Performance Can Be Improved to Acceptable Levels Through Software RAID; 14.21 Myth 4: Cloud Computing Makes Horizontal Scaling Easy; 14.22 Conclusion and Recommendations; Chapter 15: The Dark Side of Data Science; 15.1 Avoid These Pitfalls; 15.2 Know Nothing About Thy Data; 15.3 Thou Shalt Provide Your Data Scientists with a Single Tool for All Tasks; 15.4 Thou Shalt Analyze for Analysis’ Sake Only; 15.5 Thou Shalt Compartmentalize Learnings; 15.6 Thou Shalt Expect Omnipotence from Data Scientists; 15.7 Final Thoughts; Chapter 16: How to Feed and Care for Your Machine-Learning Experts; 16.1 Define the Problem; 16.2 Fake It Before You Make It; 16.3 Create a Training Set; 16.4 Pick the Features; 16.5 Encode the Data; 16.6 Split Into Training, Test, and Solution Sets; 16.7 Describe the Problem; 16.8 Respond to Questions; 16.9 Integrate the Solutions; 16.10 Conclusion; Chapter 17: Data Traceability; 17.1 Why?; 17.2 Personal Experience; 17.3 Immutability: Borrowing an Idea from Functional Programming; 17.4 An Example; 17.5 Conclusion; Chapter 18: Social Media: Erasable Ink?; 18.1 Social Media: Whose Data Is This Anyway?; 18.2 Control; 18.3 Commercial Resyndication; 18.4 Expectations Around Communication and Expression; 18.5 Technical Implications of New End User Expectations; 18.6 What Does the Industry Do?; 18.7 What Should End Users Do?; 18.8 How Do We Work Together?; Chapter 19: Data Quality Analysis Demystified: Knowing When Your Data Is Good Enough; 19.1 Framework Introduction: The Four Cs of Data Quality Analysis; 19.2 Complete; 19.3 Coherent; 19.4 Correct; 19.5 aCcountable; 19.6 Conclusion; Colophon;
