Studies have found that effects of button design characteristics (i.e., button size, button spacing, visual feedback and button shape) on users' touchscreen performance, mental workload and preference (Da, Juan, and Shuang 2018).
We were interested in the effect of color and location of sign in button on the speed of shopping websites users to locate it. We believed that the easiness for users to locate the sign in button will increase the membership tier hence better serve the customer and lead to higher member related revenues.
To observe this, we created an experiment with 169 randomly generated participants across networks and Amazon Mechanical Turk to test how much time they took to locate the "Sign In" button on the assigned webpage. These participants will participate on our designed website at https://weijia.io/Experiment/AB-Login_Button and answer a survey on personal information such as age, gender, educational level and frequency of using online shopping platforms. Then they will enter a randomly assigned webpage with 25% of chance among control, treatment 1,2 and 3. They are given the task to click on sign in button, and the time they took to perform the task will be recorded. For the control group, the "Sign in" button is in the upper left corner of the page. For treatment 1, the "Sign in" button is at the right hand side, together with the user's head-shot. For treatment 2, the "Sign in" button is in the upper left corner of the page, and is highlighted. For treatment 3, the "Sign in" button is at the right hand side, together with the user's head-shot, and is highlighted. Everything except the "Sign in" button is the same for the four groups.
We did not find there is a significant difference across control and tree treatment arms, so we assume that the color and location of "Sign in" button can not effect the users' experience. We think there are two reasons. Firstly, we have a small sample size. According to the power test, we need at least 1571 samples to make this experiment have power, but we only have 169 samples. Secondly, the outcome has a high standard deviation, because we did not find any useful covariate in our dataset.
For further exploration, we have some thoughts. Firstly, we should collect more data to have enough power, and we could collect more information on factors that may have impacts on the engagement time. Then, we could create new treatment arms based on factors mentioned in papers about users' web browsing habits. Lastly, as the factorial analysis has the same result, we can actually create fewer arms. Thus, the number of samples in each group would increase.