Cronbach's alpha is pretty much standard to all pyschometric research, and it can actually be useful both for developing survey instruments and post-hoc validation of the instrument. If you have a mesurement model where you have a few sub scales or "dimensions," which are each manifest by say 5 or 6 questions on the pilot survey, the alpha can help you determine if those questions are truly reflective of their respective dimensions. After a pilot test, where you have actual test subjects take the survey, the alpha can help you decide which questions are important and which can be dropped to streamline the final survey. Some statistical programs (e.g., SAS) will calculate an alpha after dropping each successive question, to show you how much higher alpha would be without that question. An ideal alpha is .7 to .9, but .6 is acceptable. If it's above .9, then your questions are probably redundant and not varied enough in scope for that dimension (they're asking nearly the same thing).
It sounds like you did not actually "pilot test" your survey, but that you did qualitative testing to confirm face validity (the survey looked right and seemed relevant to potential subjects) and perhaps content validity (the questions were correct in scope and depth, at least according to the subjects). A pilot test is usually accomplished by administering the survey to test subjects (n>30) and assessing the results. Sometimes this is out of your scope or resource budget, but it's the best way to catch problems if you're serious about your survey's reliability and validity and you can't afford to make a big mistake with the "live" survey.
Advertise on Talk Stats