Study Design and Sampling

Study Design and Sampling

How data is collected determines what conclusions can honestly be drawn from it. In a simple random sample, every member of the population has an equal chance of selection. More generally, probability-based random sampling uses a chance process instead of convenience or self-selection and can support generalizing to the population represented by the sampling frame. A convenience sample draws from whoever is easiest to reach and risks bias. A voluntary response sample only includes those who choose to respond, which tends to overrepresent people with strong opinions.

Only a randomized experiment can support a causal conclusion. An observational study compares existing groups without assigning who goes where, which can reveal an association between two variables but never establishes that one causes the other. Random assignment, not random sampling, is what makes a causal claim possible.

A two-way table organizes data by two categorical variables at once. Each cell where a row and column intersect is a joint frequency, the count of subjects matching both categories. Each row or column total is a marginal frequency, the count for that category alone, regardless of the other variable.

Where this appears as a model: a question about whether a treatment causes an effect. Randomly assigning half of a set of fields to receive a new fertilizer and half to receive none is a randomized experiment, supporting a causal conclusion; simply observing which existing farms already use the fertilizer is only an observational study.

Two different kinds of randomness

Students often blur two ideas that the ACT keeps separate. Each answers a different question.

A study can have one, both, or neither. An experiment on volunteers with random assignment supports a causal conclusion, but only for people like the volunteers. A random survey of a whole city supports generalizing to the city, but it is observational, so it shows association only. To conclude both that a treatment works and that it works for an entire population, a study needs a random sample and random assignment.

Bias in sampling methods

Worked Example. A radio station asks listeners to call in and vote for their favorite song. What kind of sample is this, and what bias does it risk?

Only listeners who choose to call in are included. This is a voluntary response sample. It risks overrepresenting listeners with strong opinions, since people with a mild preference are less likely to call in at all.

When a question asks for the main risk of a survey method, ask who has no chance of being reached. Calling only landline phones leaves out people without landlines, so they are underrepresented. Mailing only to a list leaves out anyone not on the list. A large sample does not cure this problem: a huge sample drawn the wrong way is still biased.

Biased wording

Bias can also come from the question itself. A leading question pushes respondents toward a particular answer. "Don't you agree that the new park is a great addition?" suggests the answer it wants. A neutral version offers balanced choices, such as "How would you rate the new park: poor, fair, good, or excellent?"

Reading a two-way table

A cell where a row and column intersect gives a joint frequency; the count for a category alone, found in a row or column total, is a marginal frequency. A conditional proportion restricts the calculation to just one row or one column instead of the grand total.

Use this pattern: the phrase after "of" or "among" names the group that forms the denominator.

• Proportion of all respondents who approve: $\frac{65}{100} = 0.65$ (the denominator is the grand total).
• Proportion of respondents over 30 who disapprove: $\frac{25}{50} = 0.50$ (the denominator is the Over 30 row total).
• Proportion of those who disapprove who are over 30: $\frac{25}{35} \approx 0.71$ (the denominator is the Disapprove column total).

The last two are different calculations that use the same cell, 25, with different denominators. Read the wording carefully to pick the right one.

What a table cannot tell you

The table above shows that under-30 respondents approved at a higher rate (80%) than over-30 respondents (50%). That is an association between age group and approval. Nobody was randomly assigned to an age group, and no one can be, so there is no experiment here and no causal conclusion. Even with a big difference and a large sample, an observational comparison supports association only.

Strategy checklist

• Identify the sample: who could have been included, and who chose to be?
• Voluntary response and convenience samples risk bias; random samples support generalizing.
• Look for random assignment before accepting any claim that one thing caused another.
• Decide the denominator of a table proportion by finding the group named after "of" or "among."
• Watch for leading wording in survey questions.