Advanced Data Analysis from an Elementary Point of View
Free
Description
Contents
Reviews
Language
English
ISBN
Unknown
Introduction
To the Reader
Concepts You Should Know
I Regression and Its Generalizations
Regression Basics
Statistics, Data Analysis, Regression
Guessing the Value of a Random Variable
Estimating the Expected Value
The Regression Function
Some Disclaimers
Estimating the Regression Function
The Bias-Variance Tradeoff
The Bias-Variance Trade-Off in Action
Ordinary Least Squares Linear Regression as Smoothing
Linear Smoothers
k-Nearest-Neighbor Regression
Kernel Smoothers
Exercises
The Truth about Linear Regression
Optimal Linear Prediction: Multiple Variables
Collinearity
The Prediction and Its Error
Estimating the Optimal Linear Predictor
Unbiasedness and Variance of Ordinary Least Squares Estimates
Shifting Distributions, Omitted Variables, and Transformations
Changing Slopes
R2: Distraction or Nuisance?
Omitted Variables and Shifting Distributions
Errors in Variables
Transformation
Adding Probabilistic Assumptions
Examine the Residuals
On Significant Coefficients
Linear Regression Is Not the Philosopher's Stone
Further Reading
Exercises
Model Evaluation
What Are Statistical Models For?
Errors, In and Out of Sample
Over-Fitting and Model Selection
Cross-Validation
Data-set Splitting
k-Fold Cross-Validation (CV)
Leave-one-out Cross-Validation
Warnings
Parameter Interpretation
Further Reading
Exercises
Smoothing in Regression
How Much Should We Smooth?
Adapting to Unknown Roughness
Bandwidth Selection by Cross-Validation
Convergence of Kernel Smoothing and Bandwidth Scaling
Summary on Kernel Smoothing in 1D
Kernel Regression with Multiple Inputs
Interpreting Smoothers: Plots
Average Predictive Comparisons
Computational Advice: npreg
Further Reading
Exercises
Simulation
What Do We Mean by ``Simulation''?
How Do We Simulate Stochastic Models?
Chaining Together Random Variables
Random Variable Generation
Built-in Random Number Generators
Transformations
Quantile Method
Sampling
Sampling Rows from Data Frames
Multinomials and Multinoullis
Probabilities of Observation
Repeating Simulations
Why Simulate?
Understanding the Model; Monte Carlo
Checking the Model
``Exploratory'' Analysis of Simulations
Sensitivity Analysis
Further Reading
Exercises
The Bootstrap
Stochastic Models, Uncertainty, Sampling Distributions
The Bootstrap Principle
Variances and Standard Errors
Bias Correction
Confidence Intervals
Other Bootstrap Confidence Intervals
Hypothesis Testing
Double bootstrap hypothesis testing
Parametric Bootstrapping Example: Pareto's Law of Wealth Inequality
Resampling
Parametric vs. Nonparametric Bootstrapping
Bootstrapping Regression Models
Re-sampling Points: Parametric Example
Re-sampling Points: Non-parametric Example
Re-sampling Residuals: Example
Bootstrap with Dependent Data
Things Bootstrapping Does Poorly
Which Bootstrap When?
Further Reading
Exercises
Weighting and Variance
Weighted Least Squares
Heteroskedasticity
Weighted Least Squares as a Solution to Heteroskedasticity
Some Explanations for Weighted Least Squares
Finding the Variance and Weights
Conditional Variance Function Estimation
Iterative Refinement of Mean and Variance: An Example
Real Data Example: Old Heteroskedastic
Re-sampling Residuals with Heteroskedasticity
Local Linear Regression
For and Against Locally Linear Regression
Lowess
Exercises
Splines
Smoothing by Penalizing Curve Flexibility
The Meaning of the Splines
Computational Example: Splines for Stock Returns
Confidence Bands for Splines
Basis Functions and Degrees of Freedom
Basis Functions
Degrees of Freedom
Splines in Multiple Dimensions
Smoothing Splines versus Kernel Regression
Some of the Math Behind Splines
Further Reading
Exercises
Additive Models
Additive Models
Partial Residuals and Back-fitting
Back-fitting for Linear Models
Backfitting Additive Models
The Curse of Dimensionality
Example: California House Prices Revisited
Closing Modeling Advice
Further Reading
Exercises
Testing Regression Specifications
Testing Functional Forms
Examples of Testing a Parametric Model
Remarks
Why Use Parametric Models At All?
Why We Sometimes Want Mis-Specified Parametric Models
Further Reading
Exercises
More about Hypothesis Testing
Logistic Regression
Modeling Conditional Probabilities
Logistic Regression
Likelihood Function for Logistic Regression
Numerical Optimization of the Likelihood
Iteratively Re-Weighted Least Squares
Generalized Linear and Additive Models
Generalized Additive Models
Model Checking
Residuals
Non-parametric Alternatives
Calibration
A Toy Example
Weather Forecasting in Snoqualmie Falls
Logistic Regression with More Than Two Classes
Exercises
GLMs and GAMs
Generalized Linear Models and Iterative Least Squares
GLMs in General
Examples of GLMs
Vanilla Linear Models
Binomial Regression
Poisson Regression
Uncertainty
Modeling Dispersion
Likelihood and Deviance
Maximum Likelihood and the Choice of Link Function
R: glm
Generalized Additive Models
Further Reading
Exercises
Trees
Prediction Trees
Regression Trees
Example: California Real Estate Again
Regression Tree Fitting
Cross-Validation and Pruning in R
Uncertainty in Regression Trees
Classification Trees
Measuring Information
Making Predictions
Measuring Error
Misclassification Rate
Average Loss
Likelihood and Cross-Entropy
Neyman-Pearson Approach
Further Reading
Exercises
II Multivariate Data, Distributions, and Latent Structure
Multivariate Distributions
Review of Definitions
Multivariate Gaussians
Linear Algebra and the Covariance Matrix
Conditional Distributions and Least Squares
Projections of Multivariate Gaussians
Computing with Multivariate Gaussians
Inference with Multivariate Distributions
Estimation
Model Comparison
Goodness-of-Fit
Exercises
Density Estimation
Histograms Revisited
``The Fundamental Theorem of Statistics''
Error for Density Estimates
Error Analysis for Histogram Density Estimates
Kernel Density Estimates
Analysis of Kernel Density Estimates
Joint Density Estimates
Categorical and Ordered Variables
Practicalities
Kernel Density Estimation in R: An Economic Example
Conditional Density Estimation
Practicalities and a Second Example
More on the Expected Log-Likelihood Ratio
Simulating from Density Estimates
Simulating from Kernel Density Estimates
Sampling from a Joint Density
Sampling from a Conditional Density
Drawing from Histogram Estimates
Examples of Simulating from Kernel Density Estimates
Exercises
Relative Distributions and Smooth Tests
Smooth Tests of Goodness of Fit
From Continuous CDFs to Uniform Distributions
Testing Uniformity
Neyman's Smooth Test
Choice of Function Basis
Choice of Number of Basis Functions
Application: Combining p-Values
Density Estimation by Series Expansion
Smooth Tests of Non-Uniform Parametric Families
Estimated Parameters
Implementation in R
Some Examples
Conditional Distributions and Calibration
Relative Distributions
Estimating the Relative Distribution
R Implementation and Examples
Example: Conservative versus Liberal Brains
Example: Economic Growth Rates
Adjusting for Covariates
Example: Adjusting Growth Rates
Further Reading
Exercises
Principal Components Analysis
Mathematics of Principal Components
Minimizing Projection Residuals
Maximizing Variance
More Geometry; Back to the Residuals
Scree Plots
Statistical Inference, or Not
Example 1: Cars
Example 2: The United States circa 1977
Latent Semantic Analysis
Principal Components of the New York Times
PCA for Visualization
PCA Cautions
Further Reading
Exercises
Factor Analysis
From PCA to Factor Analysis
Preserving correlations
The Graphical Model
Observables Are Correlated Through the Factors
Geometry: Approximation by Linear Subspaces
Roots of Factor Analysis in Causal Discovery
Estimation
Degrees of Freedom
A Clue from Spearman's One-Factor Model
Estimating Factor Loadings and Specific Variances
Maximum Likelihood Estimation
Alternative Approaches
Estimating Factor Scores
The Rotation Problem
Factor Analysis as a Predictive Model
How Many Factors?
R2 and Goodness of Fit
Factor Models versus PCA Once More
Examples in R
Example 1: Back to the US circa 1977
Example 2: Stocks
Reification, and Alternatives to Factor Models
The Rotation Problem Again
Factors or Mixtures?
The Thomson Sampling Model
Further Reading
Nonlinear Dimensionality Reduction
Why We Need Nonlinear Dimensionality Reduction
Local Linearity and Manifolds
Locally Linear Embedding (LLE)
Finding Neighborhoods
Finding Weights
k > p
Finding Coordinates
More Fun with Eigenvalues and Eigenvectors
Finding the Weights
k > p
Finding the Coordinates
Calculation
Finding the Nearest Neighbors
Calculating the Weights
Calculating the Coordinates
Diffusion Maps
Diffusion-Map Coordinates
Fun with Transition Matrices
Multiple Scales
Choosing q
What to Do with the Diffusion Map Once You Have It
Spectral Clustering
Asymmetry
The Kernel Trick
Exercises
Mixture Models
Two Routes to Mixture Models
From Factor Analysis to Mixture Models
From Kernel Density Estimates to Mixture Models
Mixture Models
Geometry
Identifiability
Probabilistic Clustering
Simulation
Estimating Parametric Mixture Models
More about the EM Algorithm
Further Reading on and Applications of EM
Topic Models and Probabilistic LSA
Non-parametric Mixture Modeling
Worked Computating Example
Mixture Models in R
Fitting a Mixture of Gaussians to Real Data
Calibration-checking for the Mixture
Selecting the Number of Components by Cross-Validation
Interpreting the Mixture Components, or Not
Hypothesis Testing for Mixture-Model Selection
Exercises
Graphical Models
Conditional Independence and Factor Models
Directed Acyclic Graph (DAG) Models
Conditional Independence and the Markov Property
Examples of DAG Models and Their Uses
Missing Variables
Non-DAG Graphical Models
Undirected Graphs
Directed but Cyclic Graphs
Further Reading
III Causal Inference
Graphical Causal Models
Causation and Counterfactuals
Causal Graphical Models
Calculating the ``effects of causes''
Back to Teeth
Conditional Independence and d-Separation
D-Separation Illustrated
Linear Graphical Models and Path Coefficients
Positive and Negative Associations
Independence and Information
Further Reading
Exercises
Identifying Causal Effects
Causal Effects, Interventions and Experiments
The Special Role of Experiment
Identification and Confounding
Identification Strategies
The Back-Door Criterion: Identification by Conditioning
The Entner Rules
The Front-Door Criterion: Identification by Mechanisms
The Front-Door Criterion and Mechanistic Explanation
Instrumental Variables
Some Invalid Instruments
Critique of Instrumental Variables
Failures of Identification
Summary
Further Reading
Exercises
Estimating Causal Effects
Estimators in the Back- and Front- Door Criteria
Estimating Average Causal Effects
Avoiding Estimating Marginal Distributions
Propensity Scores
Matching and Propensity Scores
Instrumental-Variables Estimates
Uncertainty and Inference
Recommendations
Further Reading
Exercises
Discovering Causal Structure
Testing DAGs
Testing Conditional Independence
Faithfulness and Equivalence
Partial Identification of Effects
Causal Discovery with Known Variables
The PC Algorithm
Causal Discovery with Hidden Variables
On Conditional Independence Tests
Software and Examples
Limitations on Consistency of Causal Discovery
Further Reading
Exercises
Pseudo-code for the SGS and PC Algorithms
The SGS Algorithm
The PC Algorithm
Experimental Causal Inference
Why Experiment?
Jargon
Basic Ideas Guiding Experimental Design
Randomization
How Randomization Solves the Causal Identification Problem
Randomization and Linear Models
Randomization and Non-Linear Models
Modes of Randomization
IID Assignment
Planned Assignment
Perspectives: Units vs. Treatments
Choice of Levels
Parameter Estimation or Prediction
Maximizing Yield
Model Discrimination
Multiple Goals
Multiple Manipulated Variables
Factorial Designs
Blocking
Within-Subject Designs
Summary on the elements of an experimental design
``What the experiment died of''
Further Reading
Exercises
IV Dependent Data
Time Series
Time Series, What They Are
Stationarity
Autocorrelation
The Ergodic Theorem
The World's Simplest Ergodic Theorem
Rate of Convergence
Why Ergodicity Matters
Markov Models
Meaning of the Markov Property
Autoregressive Models
Autoregressions with Covariates
Additive Autoregressions
Linear Autoregression
``Unit Roots'' and Stationary Solutions
Conditional Variance
Regression with Correlated Noise; Generalized Least Squares
Bootstrapping Time Series
Parametric or Model-Based Bootstrap
Block Bootstraps
Sieve Bootstrap
Trends and De-Trending
Forecasting Trends
Seasonal Components
Detrending by Differencing
Cautions with Detrending
Bootstrapping with Trends
Further Reading
Exercises
Time Series with Latent Variables
Simulation-Based Inference
The Method of Simulated Moments
The Method of Moments
Adding in the Simulation
An Example: Moving Average Models and the Stock Market
Exercises
Appendix: Some Design Notes on the Method of Moments Code
Longitudinal, Spatial and Network Data
V Data-Analysis Problem Sets
What's That Got to Do with the Price of Condos in California?
The Advantages of Backwardness
The Size of a Cat's Heart
It's Not the Heat that Gets You
Nice Demo City, but Will It Scale?
Version 1
Background
Data
Tasks and Questions
Version 2
Data
Problems
Diabetes
Fair's Affairs
How the North American Paleofauna Got a Crook in Its Regression Line
How the Hyracotherium Got Its Mass
How the Recent Mammals Got Their Size Distribution
Red Brain, Blue Brain
Patterns of Exchange
Is This Assignment Really Necessary?
Mystery Multivariate Analysis
Separated at Birth
Brought to You by the Letters D, A and G
Estimating with DAGs
Use and Abuse of Conditioning
What Makes the Union Strong?
An Insufficiently Random Walk Down Wall Street
Macroeconomic Forecasting
Debt Needs Time for What It Kills to Grow In
Appendices
Linear Algebra
Eigenvalues and Eigenvectors of Matrices
Singular Value Decomposition
Square Root of a Matrix
Special Kinds of Matrix
Orthonormal Bases
Orthogonal Projections
Function Spaces
Bases
Eigenvalues and Eigenfunctions of Operators
Further Reading
Big O and Little o Notation
Taylor Expansions
Propagation of Error
Optimization Theory
Basic Concepts of Optimization
Small-Noise Asymptotics for Optimization
Application to Maximum Likelihood
Aside: The Akaike Information Criterion
Constrained and Penalized Optimization
Constrained Optimization
Lagrange Multipliers
Penalized Optimization
Constrained Linear Regression
Statistical Remark: ``Ridge Regression'' and ``The Lasso''
Optimization Methods
Optimization with First- and Second- Derivatives
Gradient Descent
Newton's Method
Newton's Method in More than One Dimension
Stochastic Approximation
Stochastic Newton's Method
Pros and Cons of Stochastic Gradient Methods
Derivative-Free Optimization Techniques
Nelder-Mead, a.k.a. the Simplex Method
Simulated Annealing
Methods for Constraints
R Notes
2 and Likelihood Ratios
Proof of the Gauss-Markov Theorem
Rudimentary Graph Theory
Uncorrelated versus Independent
Information Theory
Programming
Functions
First Example: Pareto Quantiles
Functions Which Call Functions
Sanity-Checking Arguments
Layering Functions and Debugging
More on Debugging
Automating Repetition and Passing Arguments
Avoiding Iteration: Manipulating Objects
ifelse and which
apply and Its Variants
More Complicated Return Values
Re-Writing Your Code: An Extended Example
General Advice on Programming
Comment your code
Use meaningful names
Check whether your program works
Avoid writing the same thing twice
Start from the beginning and break it down
Break your code into many short, meaningful functions
Further Reading
Generating Random Variables
Rejection Method
The Metropolis Algorithm and Markov Chain Monte Carlo
Generating Uniform Random Numbers
Further Reading
Exercises
The book hasn't received reviews yet.
Free
