I'm treating with a data base concerning the comparison of several products with a benchmark. We use 2 methods: t-test & ANOVA to compare the groupes means (means of scores sampled to a product).
The statistic software is developed internally so it's not widely used. il may directly give us the result of 2 analysis but I cannot see how the procedure going on.
By lauching the software, I got 2 series of results corresponding to 2 methods representing the significant difference between those products and the benchmark.
According to the theory, there cannot be too many bias between the 2 methods.
The problem is, there IS too much difference, the software shows 2 conditions:
(a), there is a situation where t-test says there is a sig. dif. but ANOVA says not;
(b), there is also another situation that the 2 methods lead totally same conclusion.
we do 50 tests for each method, the condition (a) occurs 25 times. 50%.
my question :
1, which test we prefer?
2, or we choose different type of test according to different data?
THANKS A LOT!!!