The scientific hypothesis is very important. In is a major reason why
the Western world overtook China in the last millennium. 2000 years ago,
the Chinese were more advanced than Europeans. Scientific discovery
stagnated in China, when scholars believed you had to move from
certainty to certainty. Ed has written this in a number of books.
Essentially the hypothesis approach is starting with a conclusion, then
seeing if there is evidence for the conclusion.
For example, a person first hypothesizes that smoking causes lung
cancer. A study takes place in which data is collected and carefully
analyzed, to see whether the statistics support the hypothesis.
This approach has a major weakness though. There is a tendency to
discover what you are looking for. (One of several reasons for this is a process called sensitization, which Ed has written about.)
When a person has a hypothesis, sometimes their ego (or financial interests) can get involved. Experiments can be inadvertently tainted to produce the
desired result. A more blatant case is to
simply repeat the experiment until the desired effect is produced. There is the old story about Pasteur inoculates one dog, 2 dogs are infected they both die. No problem, do the experiment again. This time the inoculated one dies and the other lives. No problem, he did the experiment a third time till he got the result he wanted. (Not sure of the exact details, but its to make a point)
One experiment was done by someone who wanted to prove
oxygen was bad for you. He put some flies in a box which contained pure
oxygen (An absurd artificial situation). These lived longer than the control group (the undesired result)
so he redefined his aim and concluded that oxygen reduced the quality
of life on the flies.
If I wanted to prove "Blondes have more fun", I could influence the
results of a study, by how blonde or fun is defined, how the data is
sampled etc. etc. etc. There are myriads of ways of doing it.
Instead of hypothesizing "smoking causes lung cancer" the question could be asked "Is there a relationship between smoking and lung cancer or not?" A negative result should be regarded as just as valuable as a positive result. This small step could help the experimenter be more objective and get more studies (with negative results) published.
Another limitation is that many things are discovered accidentally, that
is when looking for something else.
My proposal is that an alternative to the hypothesis, is to observe a phenomenon and explore its causes, effects and relationships with other
For example, instead of hypothesizing "smoking causes lung cancer" ask
What are the long term effects of smoking on the lungs? (or the whole
body, if the study was large enough).
This is more neutral and objective. Its also a lot more efficient.
Accidental discoveries need to be noted, also there are approaches that
can increase the probability of finding accidental discoveries. Ed writes about this in his books on creativity.
I can go into more detail, but the post is probably long enough already.