Not necessarily, what it means is that you don't have enough data to make a decision. If you observe a 1 point increase in conversion over your next 100 visitors you can't be sure of anything. On the contrary, if you observe a 2 points increase in conversion over your next 1400 visitors, you are now able to make a decision. This tool is here to help you make them.
Yes, I know it can be a little bit confusing. Let me put is this way, this tool only tells you that you can't be absolutely sure that your conversion actually increased.
It means that you will need a larger sample (more visitors), with the same conversion rate, to actually be sure that there is a change.
This is a tricky question and this is exactly why I made this tool. How much should you wait before being sure that you observe a significant difference depends on two things. First it depends on the difference between your two conversions, the greater the gap, the faster you can be sure that there is a gap. Secondly it depends on the number of visitors you used to compute your conversions.
Sure, this tool is actually a simpler version of this one. It leverages statistics and probabilty to ensure you that you make the right decisions. If you really like mathematics, we suggest you read this wiki article and this one.
To my knowledge, this one is the only one giving you an estimate of when will your be significant if it is not alreay. Then this one is more lightweight and not cluttered with dozens trackers waiting to retarget you on Facebook or Linkedin 😉.
My name is Pierre de Wulf. I left my full-time job in 2018 and I try ever since to build profitable buisinesses. You can follow me on Twitter as I continue to document my journey.