Let’s assume we only care about the “stomach” variable. I have 9 days where I ate grains, and 19 days where I ate no grains. On each day I have a number that says how well my stomach felt, on average, during the day. I would like to know if my average stomach feeling is the same or not the same on grains days vs. non-grains days.
(let’s assume that there’s no weirdness where grains makes your stomach feel worse for the next 2 weeks or something. In fact, let’s forget about the fact that these are time series data at all. This might be a minor or a major statistical transgression, I don’t know.)
I think the test to use here is a Student’s t-test (AKA just “t-test”). Particularly, an independent 2-sample t-test with equal variances (assumed) and unequal sample sizes. (side note: I also remembered z-tests and ANOVAs, and it looks like:
- a z-test is a simpler test that you can use when you’re just trying to know if a sample that you’ve taken is significantly different from the group as a whole. For example, if I knew my “stomach” value for every day of my life (or even if I just knew the mean and standard deviation), I could use a z-test to tell if those 19 no-grains days were unusually high/low stomach-value days compared to all 9038 days of my life.
- an ANOVA is a generalization of the t-test. Specifically, the one-way ANOVA is something you can use to compare >2 means. For example, if I tried “no grains” for two weeks, then “no meat” for two weeks, then “no coffee or peanuts” for two weeks (clearly this would be the hardest), I could compare my stomach value for all of those.
- if “ANOVA” wasn’t scary enough, there’s also things called “ANCOVA” and “MANOVA”; the latter makes me snicker every time.)
Okay! Let’s do some t-testing! In one corner, the stomach (, mood, energy) values of my grains days. In the other corner, the stomach (, mood, energy) values of my no-grains days. Which Is Bigger?
t-test on mood before and after grains:
t = -0.78694967152, p = 0.439345514713
t-test on energy before and after grains:
t = -0.961369818324, p = 0.346365085994
t-test on stomach before and after grains:
t = -1.98753017649, p = 0.0588975495116
Again, the p-value is the one that tells you if there’s anything going on. Small p-value means there’s a small chance that this effect could have happened by chance. Looks like I can’t say anything about whether grains effect my mood or energy. But p=0.058 is pretty small! (traditionally 0.05 is the threshold for caring about p-values, at least in psych) And surprisingly so. I couldn’t have told you that from the graph.
Let’s look at the data again: (check it out, I’m learning python string formatting)
stomach with grains: [‘3.10’, ‘2.92’, ‘2.78’, ‘2.83’, ‘2.36’, ‘2.75’, ‘3.29’]
mean = 2.86
stomach without grains: [‘3.33’, ‘3.67’, ‘2.89’, ‘3.37’, ‘3.31’, ‘3.60’, ‘3.00’, ‘2.87’, ‘2.59’, ‘2.80’, ‘2.92’, ‘3.11’, ‘3.22’, ‘3.63’, ‘3.47’, ‘3.29’, ‘2.89’, ‘2.67’]
mean = 3.15
Huh! That is interesting. Now before you jump to conclusions, note a few things:
- this is self reported, not double blind, not even single blind. (although in this case I’m the experimenter and the subject, so single blind = double blind; and it’d be really hard to make this experiment blind.)
- data was gathered “as I feel like it” AKA whenever I use my phone.
- I cut out 3 days' worth of data because they each had only one sample.
- “days” were split at midnight, even though I usually had one or two points after midnight; I should probably split them at about 3 or 4AM.
- I didn’t just cut out grains. I also minimized added sugar (how well? dunno) and added more meat.
- p = 0.058. That’s really borderline significant. It could just be a fluke.
- I didn’t say what I’d look for before doing the experiment. Why is this such a big deal? Well, p=0.058 means that even if grains didn’t matter, 5.8% of the time such an effect could have happened just by chance. Which means that if I tracked 20 variables, I’d find one that “looked significant”.
But hey, exploratory pilot study: super success! I think that my energy and my mood are pretty similar, and I think that maybe grains make my stomach feel worse although I’d need to study it again to tell for sure. Very cool!
blog 2023 2022 2021 2020 2019 2018 2017 2016 2015 2014 2013 2012 2011 2010