Practical Statistics for Data Scientists

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Book description

Statistical methods are a key part of of data science, yet very few data scientists have any formal statistics training. Courses and books on basic statistics rarely cover the topic from a data science perspective. This practical guide explains how to apply various statistical methods to data science, tells you how to avoid their misuse, and gives you advice on what's important and what's not.

Many data science resources incorporate statistical methods but lack a deeper statistical perspective. If you’re familiar with the R programming language, and have some exposure to statistics, this quick reference bridges the gap in an accessible, readable format.

With this book, you’ll learn:

• Why exploratory data analysis is a key preliminary step in data science
• How random sampling can reduce bias and yield a higher quality dataset, even with big data
• How the principles of experimental design yield definitive answers to questions
• How to use regression to estimate outcomes and detect anomalies
• Key classification techniques for predicting which categories a record belongs to
• Statistical machine learning methods that “learn” from data
• Unsupervised learning methods for extracting meaning from unlabeled data

Publisher resources

Table of contents

1. Preface
   1. Conventions Used in This Book
   2. Using Code Examples
   3. Safari® Books Online
   4. How to Contact Us
   5. Acknowledgments
   1. Elements of Structured Data
      1. Further Reading
      1. Data Frames and Indexes
      2. Nonrectangular Data Structures
      3. Further Reading
      1. Mean
      2. Median and Robust Estimates
      3. Example: Location Estimates of Population and Murder Rates
      4. Further Reading
      1. Standard Deviation and Related Estimates
      2. Estimates Based on Percentiles
      3. Example: Variability Estimates of State Population
      4. Further Reading
      1. Percentiles and Boxplots
      2. Frequency Table and Histograms
      3. Density Estimates
      4. Further Reading
      1. Mode
      2. Expected Value
      3. Further Reading
      1. Scatterplots
      2. Further Reading
      1. Hexagonal Binning and Contours (Plotting Numeric versus Numeric Data)
      2. Two Categorical Variables
      3. Categorical and Numeric Data
      4. Visualizing Multiple Variables
      5. Further Reading
      1. Random Sampling and Sample Bias
         1. Bias
         2. Random Selection
         3. Size versus Quality: When Does Size Matter?
         4. Sample Mean versus Population Mean
         5. Further Reading
         1. Regression to the Mean
         2. Further Reading
         1. Central Limit Theorem
         2. Standard Error
         3. Further Reading
         1. Resampling versus Bootstrapping
         2. Further Reading
         1. Further Reading
         1. Standard Normal and QQ-Plots
         1. Further Reading
         1. Further Reading
         1. Further Reading
         1. Poisson Distributions
         2. Exponential Distribution
         3. Estimating the Failure Rate
         4. Weibull Distribution
         5. Further Reading
         1. A/B Testing
            1. Why Have a Control Group?
            2. Why Just A/B? Why Not C, D…?
            3. For Further Reading
            1. The Null Hypothesis
            2. Alternative Hypothesis
            3. One-Way, Two-Way Hypothesis Test
            4. Further Reading
            1. Permutation Test
            2. Example: Web Stickiness
            3. Exhaustive and Bootstrap Permutation Test
            4. Permutation Tests: The Bottom Line for Data Science
            5. For Further Reading
            1. P-Value
            2. Alpha
            3. Type 1 and Type 2 Errors
            4. Data Science and P-Values
            5. Further Reading
            1. Further Reading
            1. Further Reading
            1. Further Reading
            1. F-Statistic
            2. Two-Way ANOVA
            3. Further Reading
            1. Chi-Square Test: A Resampling Approach
            2. Chi-Square Test: Statistical Theory
            3. Fisher’s Exact Test
            4. Relevance for Data Science
            5. Further Reading
            1. Further Reading
            1. Sample Size
            2. Further Reading
            1. Simple Linear Regression
               1. The Regression Equation
               2. Fitted Values and Residuals
               3. Least Squares
               4. Prediction versus Explanation (Profiling)
               5. Further Reading
               1. Example: King County Housing Data
               2. Assessing the Model
               3. Cross-Validation
               4. Model Selection and Stepwise Regression
               5. Weighted Regression
               6. Further Reading
               1. The Dangers of Extrapolation
               2. Confidence and Prediction Intervals
               1. Dummy Variables Representation
               2. Factor Variables with Many Levels
               3. Ordered Factor Variables
               1. Correlated Predictors
               2. Multicollinearity
               3. Confounding Variables
               4. Interactions and Main Effects
               1. Outliers
               2. Influential Values
               3. Heteroskedasticity, Non-Normality and Correlated Errors
               4. Partial Residual Plots and Nonlinearity
               1. Polynomial
               2. Splines
               3. Generalized Additive Models
               4. Further Reading
               1. Naive Bayes
                  1. Why Exact Bayesian Classification Is Impractical
                  2. The Naive Solution
                  3. Numeric Predictor Variables
                  4. Further Reading
                  1. Covariance Matrix
                  2. Fisher’s Linear Discriminant
                  3. A Simple Example
                  4. Further Reading
                  1. Logistic Response Function and Logit
                  2. Logistic Regression and the GLM
                  3. Generalized Linear Models
                  4. Predicted Values from Logistic Regression
                  5. Interpreting the Coefficients and Odds Ratios
                  6. Linear and Logistic Regression: Similarities and Differences
                  7. Assessing the Model
                  8. Further Reading
                  1. Confusion Matrix
                  2. The Rare Class Problem
                  3. Precision, Recall, and Specificity
                  4. ROC Curve
                  5. AUC
                  6. Lift
                  7. Further Reading
                  1. Undersampling
                  2. Oversampling and Up/Down Weighting
                  3. Data Generation
                  4. Cost-Based Classification
                  5. Exploring the Predictions
                  6. Further Reading
                  1. K-Nearest Neighbors
                     1. A Small Example: Predicting Loan Default
                     2. Distance Metrics
                     3. One Hot Encoder
                     4. Standardization (Normalization, Z-Scores)
                     5. Choosing K
                     6. KNN as a Feature Engine
                     1. A Simple Example
                     2. The Recursive Partitioning Algorithm
                     3. Measuring Homogeneity or Impurity
                     4. Stopping the Tree from Growing
                     5. Predicting a Continuous Value
                     6. How Trees Are Used
                     7. Further Reading
                     1. Bagging
                     2. Random Forest
                     3. Variable Importance
                     4. Hyperparameters
                     1. The Boosting Algorithm
                     2. XGBoost
                     3. Regularization: Avoiding Overfitting
                     4. Hyperparameters and Cross-Validation
                     1. Principal Components Analysis
                        1. A Simple Example
                        2. Computing the Principal Components
                        3. Interpreting Principal Components
                        4. Further Reading
                        1. A Simple Example
                        2. K-Means Algorithm
                        3. Interpreting the Clusters
                        4. Selecting the Number of Clusters
                        1. A Simple Example
                        2. The Dendrogram
                        3. The Agglomerative Algorithm
                        4. Measures of Dissimilarity
                        1. Multivariate Normal Distribution
                        2. Mixtures of Normals
                        3. Selecting the Number of Clusters
                        4. Further Reading
                        1. Scaling the Variables
                        2. Dominant Variables
                        3. Categorical Data and Gower’s Distance
                        4. Problems with Clustering Mixed Data
                        Show and hide more

                        Product information

                        • Title: Practical Statistics for Data Scientists
                        • Author(s): Peter Bruce, Andrew Bruce
                        • Release date: May 2017
                        • Publisher(s): O'Reilly Media, Inc.
                        • ISBN: 9781491952962

                        You might also like

                        Check it out now on O’Reilly

                        Dive in for free with a 10-day trial of the O’Reilly learning platform—then explore all the other resources our members count on to build skills and solve problems every day.