I'm hoping someone can help me conceptualize my question better because I have a feeling I'm looking at it the wrong way. I'm looking at a number of Facebook posts and how many engagements each post received. I'm trying to compare the results on engagement of using different words in posts on engagement.

My initial thought was to compare the average engagements of posts that include word X against posts that do not include word X, in other words, one subset against another, mutually exclusive subset. The problem is that it seems to me that given the relatively small number of posts that contain any word X (let's say, 3 posts that do and 97 that don't), there is a good chance that the difference in engagements may just be due to chance. To account for this, I thought about running a two-sample heteroscedastic T-test to compare the means of each subset. My question with this: is it appropriate to use a T-test to compare two subsets of data from the same population? Since I know the population parameters, should I actually be using a Z-test? If so, how can I compare the two subset means and test for significance? Also, if my sample size for any subset is too small, I won't be able to calculate a Z score properly, so does that mean I should use a T-test instead? Or maybe there's a totally different way to look at this.

Any guidance here would be much appreciated!