On practical philosophy of research in science and technology

Seppo KarrilaAugust 2016 (2559 Thai)

Executive summary

• Why do we pursue research?• The Magic Spell for organizing (almost) everything• Is “science” getting sick?• What is knowledge, what is science? • How can you approach a given research topic? • Included: some practical hints and tricks that help

you in your research, marked with a star as are the otherwise most important slides.

Why we do research?

• Today’s research may be tomorrow’s technology

• Teachers who don’t follow research might not prepare you for your future!

• You do research to– Demonstrate you know how research is done (MS)– Demonstrate mastery of techniques (MS)– Demonstrate mostly independent contribution to

science (PhD)

PhD: driver’s license to doing science

• Doing a PhD is like a test drive. – You have support from an advisor. Your scope of

research is pre-defined. You need to come up with some “new science” and defend your thesis.

• Once you have your PhD– You are expected to be an independent researcher

who no longer needs an advisor!– Don’t expect your thesis is your career. When you

have your license, you will not stay on the test track, you need to learn about other roads and go to places.

Industry or academic job?

• Academia produces PhD’s who go to academia to produce more PhD’s who go to academia… – It could become a big club, as long as someone

pays for the fun. In the long run this is impossible, academia feeding its own growth and not feeding the outside world.

– Go outside to see what is done in the “real world” with science. Maintain contact with academics, perhaps come back to tell what you learned.

Reality restricts your options

• http://www.theatlantic.com/business/archive/2013/02/how-many-phds-actually-get-to-become-college-professors/273434/

Here is a Magic Spell

• Issue• Significance• Approach• Results• Conclusions

The Magic Spell applies to many things

• A one-page project proposal• A full project proposal• A summary of someone else’s work• Summarizing your project plan• Writing a poster• Giving a presentation• Writing an article

Discussion with the Boss man

• Yes, you wanted to see me. What is this about?– Issue: what is this about

• Well, I have other more important things… – Significance: why this is important

• OK, but what can we do about it?– Approach: actions to take

• So if we do all that, what do we get?– (expected) results: products of the actions

• And what is the value of the results?– Conclusions: effects on decisions, practices, …

Be prepared to see the Magic Spell several times today

• Issue• Significance• Approach• Results• Conclusions

Trust in “science” is suffering

"Ninety-seven percent of original studies had significant results (P < .05). Thirty-six percent of replications had significant results…"

Note that this is rather sick

• If only 36% of reported “statistically significant” results can be replicated, then 64% of the results were NOT replicated as “significant”.

• About TWO OUT OF THREE papers in psychological science are making claims that you can’t trust.

https://www.timeshighereducation.com/news/papers-in-economics-not-reproducible

And it just gets curiouser and curiouser… Why, sometimes I've believed as many as

six impossible things before breakfast.

Depressed?

• Has the World gone bananas? • Did you pick Science because it is the ultimate

in fairness: right is right and wrong is wrong, impersonally and without politicking?

• Hold on, the plague only affects some fields, there will be healthy survivors.

The happy exceptions, solid healthy science• Engineering

– The models used must be correct and reasonably accurate, otherwise your bridge falls, your building collapses, etc.

– The products must deliver on promises! • Math

– Accepts ONLY what has been proven beyond any doubt, in little steps all of which are known to be correct. If a mistake is found later, the result is labeled either “unproven hypothesis” or “conjecture”, or actually proven false with a counterexample.

• Physics & chemistry– Only what can be reproduced and confirmed is worth keeping.

• Common features: – heavy reliance on numbers, logic, and equations – strict rejection of unreliable models, or rules that “seem to work once in a while” –

attempt to record only “absolutely correct” constructs– accountability and discipline enforced, falsehoods resolutely rejected

The Scientific Method

Note that you start with a question and a hypothesis

• The question should have importance– Otherwise any results will also be unimportant. You can become an

expert in unimportant matters… – Importance usually comes from economy: productivity (yield,

processing speed, investment cost, operating cost), or product quality.• Naïve hypotheses are still better than nothing

– By searching through all possible choices, we can find the best alternative. Not “deep science” but straightforward with useful results.

• A “clever” type of hypothesis– Two treatments or technologies have complementary strong and weak

points. Combining them could perhaps give an overall improvement.

We now go back to the fundamentals

• … and discuss knowledge, understanding and science.

• It may seem an abstract and “philosophical” remote aspect, but if more people paid some attention, the trustworthiness of science would be much better than it is today.

Theory of knowledge

• Has been studied for a very long time, called “epistemology”

• Such theories have been unable to reach practical usefulness– The question “how do we know what we know, and do

we really know?” is important– Time travel would also be important… – Companies are not struggling to find and hire

epistemologists

What is “understanding”?• In common language it often just means to sympathize. “I feel for you, I

understand.”• In science there are different levels of understanding

– Forming concepts and nomenclature, and general rules: an apple falls if you let it go

– Naming the ghost behind the action: gravitation. Being able to name the ghost gives a “feeling of understanding it”. • Why does the apple fall – gravitation! This is nonsense and word magic, just another name

that sounds more learned than “it falls because it falls”. – Quantitative understanding: how fast does it fall, how hard does it hit. Can be

calculated accurately with classical mechanics and ITS EQUATIONS FOR GRAVITATION. Now we are getting somewhere! (Some went to the moon, satellites for GPS and communications.)

– Application models based on experiments: when the apple hits the floor, will it break, how much is it damaged, does this affect its price or shelf-life, how should we package apples… These are clearly things you can’t calculate accurately from classical mechanics, you have to do experiments AND MAKE MODELS.

Stages of development• A “young” scientific discipline

– Names and documents things– Creates taxonomies– Is descriptive

• Becomes more quantitative with time, a “teenager”– Finds general rules that are equations, like “conservation of mass”,

thermodynamics of equilibrium• Strives to replace experiments with predictive computations,

becoming an “adult mature discipline”– This transition is currently happening in chemistry…

• In other words a somethingology tries to advance to a somethingonomy, from qualitative to quantitative

“Predictive” is the keyword!• There are theories that explain everything afterwards, but

predict nothing– Pyramidology predicts history accurately but future always poorly– There is a BIG difference between a regression fit and a predictive

equation. Ability to match what is known does not equal ability to predict the unknown.

• Engineering design is based on sufficiently accurate predictions. Design equations may be mandated in official standards. – You can’t know the strength of a concrete mix accurately, so

engineers use safety margins in designs– Still a lot of computations are useful and used in mechanical design,

inaccuracies don’t make a model useless

Training, validation and test data• Here is the correct way to fit predictive models to experimental

data.

• The data are split to three sets, or generated at different times for each of the three– All candidate models are fit to the training data– The validation data is “predicted” with every type of model, and the best

model type is chosen• Now it can be fit to training + validation data

– The chosen model is tested for performance• It has not “seen” the test data before, if you test with previously seen data you

are looking at “fit”, not “prediction ability” !– Now you can pass on the model, and make some claim about its

prediction accuracy

How to make pyramidology look good

• Make predictions about the starting year in January. • Check predictions next January

– Ups, all wrong again• Quickly update the model

– Publish how the new model fits history– Never discuss results from testing old predictions with new data, erase the old

models quickly• Effect: the public model is always an untested fit, there is never

embarrassing numeric evidence about predictiveness (actually complete lack of it)– This game is played all the time. Various companies have forecasters who don’t

want to look incompetent. So does the government. The forecasters want to keep their jobs and maintain credibility, and the purpose of the forecasts is not to be right but to influence decisions. Once the decisions are made, why look back…

Intermediate summary• In the natural sciences, the highest level of understanding is a quantitative

predictive model– Then you can predict effects of choices or actions on future function and

performance, this is called “engineering design”

• Such models must be based on reproducible experiments, the models must be validated, and their accuracies known. The models can only use validated reproducible characterizations whose accuracy is also known. – Characterizing material properties of cement mixes, or steel, or rubber, or plastics –

check the official standards.

• Fundamental theories are not yet good enough– You only learn EQUILIBRIUM thermodynamics– Even “viscosity” becomes difficult with polymeric liquids, so don’t expect accurate

flow calculations in a complicated geometry– Flow mixing reaction rates …

How does science progress?• According to Karl Popper

– there is an accepted “scientific paradigm” that scientist use, until enough evidence accumulates to refine or update it, and that is a “paradigm shift” (like classical mechanics relativity, continuum quanta) • Negative evidence = false predictions ! Without them there will be no

paradigm shift– Various fields like sociology would not fit his definition of science at

all, since they seldom predict anything. There are no testable hypotheses, but an ideology may “explain” anything in past history.

– Thomas Kuhn became popular because he essentially proposed that science is whatever scientists do. This made sociologists and various others happy again, they loved Kuhn.

– Popper’s views are most appropriate in the natural sciences and engineering that have developed to a quantitative stage.

The scientific method!• Make a hypothesis or conjecture• Design experiments that could show it is false

– In fact, the hypothesis should predict something that is extremely unlikely without it being correct, so when that happens it is strong support to the hypothesis

• If the hypothesis survives the tests, keep it. If it also makes useful predictions, others will adopt it and teach it as current wisdom. (If it can’t be used for anything of importance, others will not waste their time. Unless they are political activists.)– It becomes part of the current scientific paradigm

Science is NOT a still image

• Scientists are people who push the boundaries of knowledge, so the coverage tends to expand, except when new knowledge collapses an old theory without a ready replacement. Even then knowledge expanded and we learned an old theory is wrong/insufficient.

• Some parts of science are rock solid. Much of the engineering mathematics of today comes from Gauss, who lived about 200 years ago. But “modern mathematics” is evolving actively.

The big point• Science looks for general truths that summarize

experiences in a useful way, and that allow making predictions.– Instead of learning every sentence you use, you learn grammar.

Summarization and general rules enable learning and doing your own things instead of just copying and repeating.

– Measurements may be needed, but doing measurements is not the same a doing science.

– The most valuable hypotheses have a wide generality. Very specific and restricted hypotheses are “doing the right motions” but amount to little in scientific learning. They provide a data point, not a useful summary. However, they are the necessary less glorious grassroots of science, and that is where most scientists live. It is the big things that are historic with fame.

Other points to note• If the hypothesis is such that nothing can falsify it,

then it only predicts things that would happen anyway (or nothing at all)– It cannot usefully predict anything new and surprising

• But the testing is based on predictions! – A theory that predicts nothing does not enable any

decision, design, or action. These are the only things that give value to scientific theories.

– This is why “theories” that predict nothing and can’t be tested are pseudoscience.

About “mistakes”• Progress of science depends on being wrong

– Falsification of old paradigm is the only way to a new paradigm• If you are very afraid of mistakes, you can’t do anything

– But it hurts less to learn from mistakes of others– Admit mistakes quickly, correct them quickly before they can

take effect, avoid repeating them. It is not honorable to hide a mistake.

– A man who never made a mistake has done nothing• However, don’t study mistakes, study successes

– First, this way there is less to study– Second, repeating a success is better than repeating a mistake

You can never prove that something is true – you can prove something false

• Mathematics is based on axioms, assumed truths. Its proofs are valid IF the axioms are valid. In other words the absolute truths of mathematics are confined within mathematics.– Still, logic and mathematics are the tools of clear thinking. We

want equations, and quantitative predictions!• However many red roses you see, it does not prove that all

roses are red.– It only takes ONE white rose to prove they are not ALL red– This is why a hypothesis must be FALSIFIABLE, that is the best we

can do. You cannot have a PROVABLE general truth, the generality extends only as far as current experiences.

Null hypothesis

• Statistical testing is based on the same ideas.

• You make a NULL HYPOTHESIS “some roses are not red”, that means falsification of the actual hypothesis “all roses are red” (the general truth claimed)

• You show that the null hypothesis is very unlikely to be true

• That supports your actual hypothesis as “significant”, meaning that we can keep it for now

Back to the magic spell…

• Your scientific story is “good” if you have– Issue, significance, approach, results, conclusions– The question must have importance, the conclusions state

effects of your results on theory or on practice!– It is really good if the results are unexpected and surprising

• A technically useful engineering result usually includes a quantitative model– How do you make a model? Where can you start? How is

“science” done in practice?

You are not Newton

• We don’t expect you to come up with new general principles of fundamental science

• But recall that our science is limited, we need experimental models– You have equilibrium thermodynamics, but kinetic

reactions, flows, multiphase materials– Rheology of a polymeric liquid is difficult enough

• Now add solids, perhaps nanomaterials, reactive species, electromagnetic fields, …

– Or just examine if an apple breaks when it hits the floor

For modeling

• Define concepts included in the model– Some need to link to reality through direct

measurements• Prefer physical measurements over industry technical

standards, the former will stay as they are today– Others can be computed intermediate variables• These may come from physics, physical chemistry, etc.• Dimensionless groups! • You should use these in statistical modeling

The problem with indirect measurements

• Does “happiness” relate to “wealth”? – A sociology problem where no concept can be directly measured. Does

wealth include your future inheritance from grandfather, possibly winning lottery, or your relative who works for PTT where you may get a good job? A rich girlfriend? How do you measure “happiness”? Are you feeling 6.3 happy or 8.5 happy, on average this week?

– By adjusting definitions, you can get anything you want. The result is theories that live as long as the professor who started them, and who sits on committees of the National Research Fund while alive. So you better agree with his theory while he still lives, if you want to do related research. Oh, he is also the Editor or on board of all relevant journals…

– Stick with the real sciences, we do things better. I like Karl Popper.

Assume you got a topic from your advisor

• Were you given a hypothesis, or do you need to come up with one?

• Learn the basics, read review articles, find out about techniques used in experimental determinations– For example, nanomaterials are modern and difficult

exactly because they are too small to “see”. Some might also have nasty effects that are slow and delayed, like asbestos fibers, so take precautions. Don’t rush into exciting new things unless you can also measure and detect.

About managing references• Commercial option: EndNote• Free high-quality option: Zotero

• Mostly you will download pdf files. You can feed them to Zotero, which pulls in the “metadata” and formats it in your reference list. You can install an add-in to MS-Word.

• Under your “project” folder, create a subfolder “References”. Keep all your downloaded references in this one place.

• Make sure your project folder gets automatically backed up: Dropbox, OneDrive, Google Drive,… When your computer dies and you get a new one, your years of work can be pulled back from the cloud. (If you know your password…)

Discipline…• Organize your project folder with subfolders

– References– Proposal and planning– Materials and methods

• Raw materials, data sheets, measurement devices, sampling and sample sources

– Experimental designs– Experimental data

• Precious results of experimental time and cost, your family jewels. Guard them and keep them pure.

– Analysis of data • Do NOT mix this with raw data! If you do, soon nobody knows what was

measured and what is estimated or calculated or imputed or corrected… – Reporting

Finding literature• Google:

– (ultrasound OR sonication) AND rice AND filetype:pdf• Google patents

– Selection effect from cost of patenting: economic importance. Patents can be much better for understanding the WHY than academic publications. The motivation is explained explicitly.

• Google scholar– Check a well-known researcher– Or search for references with keywords as usual

• Sometimes useful– View only image results of search (to get chemical structure,

process scheme, etc.)

Hypothesis from review of literature• Are there important gaps in knowledge? – Turn these into hypotheses that seem reasonable

• Can they be studied with YOUR available equipment and techniques? – Can you estimate or guess sizes of effects?

• What are the factors affecting results?– Which ones can you manipulate– Which ones can you observe/measure– Which ones can you limit or select, essentially defining

the scope of your study

Exercise that you can do later• The “outline” feature in MS Word is very good for making a

hierarchical structure, to list factors that affect. • If you enjoy drawing and visuals, you can make a “fishbone

diagram” or a “mindmap”.• Do this with your own project, trying to list everything!

– In plant-based raw materials: variety (of rice, oil palm, rubber tree), age of plant, maturity of fruit, recent weather, use of fertilizer, storage conditions, time delays,…

– Note that you can avoid variance from these factors by pooling and homogenizing an initial batch, then using it across all experiments. Then only your experimental conditions cause differences in results. One key goal in an experimental plan is to reduce or prevent “uncontrolled noise”.

Example: factors influencing gas mileage of a given car

• Weight– Car itself– Passengers

• Number, weight– Cargo loading

• Weight – Gas tank full/empty

• Weight of gas

• Route– Uphill, downhill, level?– Stops at lights, intersections?– Distance to drive– Type or road

• Asphalt• Unpaved dirt road• Muddy• Off-road

– You might have many alternatives…

• Speed• Condition of car

– Old or new car?– Engine runs well?– Maintenance, oil, air filter,

lubrication?– Type and condition of tires, tire

pressure?• Weather

– Rain? – Sunny

• Air conditioning on?– Windy

• Type of gasoline– You might have many

alternatives…

Decisions and actions, once you have your hypothesis

• Select scope, manipulated and observed variables– Do you have the technical skills?– Do you need to design and construct devices?

• Do you need preliminary or “pilot” experiments? – Instead of gambling a long-term plan with uncertainties, can you quickly

check for some effects or phenomena?• Design of experiments, after selecting a minimal set of main

manipulated variables– Can be complicated, better to stay with some standard design (e.g. Plackett-

Burman), otherwise consult a statistician– If task is to optimize something like yield, check out response surfaces and

Box-Behnken design• Statistical analysis of results

– Significant effects AND their effect sizes !

About effect size

• Opening your car windows changes aerodynamics, probably for the worse– The top speed may go down by 3 km/h if you open

the windows– If in experiments you would repeatedly find this is so

(using GPS to measure speed), then this effect is statistically significant

– However, the effect size of 3 km/h is marginal, you don’t need to care. If it dropped the top speed by 40 km/h, that would be practically important.

A key observation

• Statistically significant is not the same as significant!– Statistically significant means, it is likely there is

some detectable difference. A detectable difference can be marginal in size.

– Significant Breakthrough Discovery: with a small effort and at a low cost, you get a large effect (on yield, quality, production rate, …)

– This is why you should pay attention to effect size!

The arts and sciences cherish novelty• But it is not enough. Pay attention to significance!– If I paint with a banana, it is a novelty but nobody will buy

my painting– If I multiply two 50-digit numbers and subtract 1234567,

nobody else ever did that calculation with the same numbers: it is a novelty!• No insight, nothing of interest, can’t be published

– In descriptive sciences, things are published just to document• Asteroid number 123456 photographed through telescope – it

goes to a database as an entry, but nobody really cares. This is their grassroot science, a data point.

Art, science and engineering, a caricature

• When something is done for the first time, it can be art or science

• When it is done repeatedly and efficiently, it can be plagiarism, forgery, or engineering research

• In science, if you know the result you should not do it, because you are looking to expand knowledge

• In engineering, if you don’t know it works, you should not do it, because most likely it will not work.

Engineering and science are intermixed

• The exploration exploitation dilemma in learning– The caricature was about extremes in exploration and exploitation– If you only exploit existing knowledge, that may be convenient and

predictable but you learn nothing new– If you only explore, you are wandering aimlessly and not productive– How much of time or budget should be exploration? – Some companies have been very good at leaving explorations to

others, then quickly doing a technically reliable good job once a new technology has been demonstrated. This is a cost advantage.

– Similar opportunities abound in science, where you can transfer techniques established in another field to your field. You will not be “the first”, but if you are quick you can be the first in a specific context. So keep your eyes open, read widely.

In STEM you should use precise language

• You drive by a field and see a black bull in profile view.

• Layman’s statement: ”See, they have black bulls here!”

• Scientific statement: “In this region there is at least one bull that is black on at least one side.”

On evaluating your own work, or that of others

• Use the magic spell– Issue, significance, approach, results, conclusions– If any of these are missing, thumbs down

• Do this also when you are planning your work– What kind of results do you expect? – How can the results affect theory or practice?

Research proposal

• Surprise:– Issue, significance, approach, expected results, expected

effects (conclusions)

• Now approach needs to include– An experimental design

• How many samples, what measurements, what experiments, how many replications

– Time – how long would it take?– Budget – how much would it cost?– Risks – what can go wrong?

Dealing with risk

• Can you prevent it from happening?– Vaccination may prevent infection– Do you need to check for impurities of raw materials,

sterilize samples, remove suspended solids before optical measurements or chromatography, de-aerate liquids, …

– Start with small doses of an additive, see if trouble arises. Increase dose if all goes well. Large dose first is risky.

• What will you do if the bad thing happens?– Go to doctor or hospital when you get sick– Do you have alternatives or backups, for sources of samples,

for determination techniques, …

Conclusions• Most likely you will run experiments and do some statistical

analysis of the results– Planning of experiments should be based on a hypothesis, the

most “ignorant” just assuming that some factor has an effect• After literature research, you should have a hypothesis and

an experimental approach, convert these to a detailed research proposal

• When reporting results, or checking results of others, statistical significance is worth little, effect size is worth a lot

• The Magic Spell gives structure to any communication, or analysis of communication. If an item in it is missing, then the proposal/presentation/manuscript is incomplete.

One final note and warning on science vs. pseudoscience

• We have not defined science, but it clearly seeks knowledge and understanding that are useful and predictive– The best we can do is accept a useful hypothesis until it has been falsified.

Useful ones predict something, so they can always be tested. • We can identify much of pseudoscience from this already

– Either the hypothesis can’t be falsified ever, by anything– or its proponents bitterly oppose any such test that could falsify it, their

interest is not truth and knowledge but politics• But… everybody wants to have the clout of science. If you point a

finger at pseudoscience, many people will be upset.– It suffices that you know the truth, don’t waste your time in an argument.

Just keep it between you, me, and Sir Karl Popper. And that guy in the toothpaste commercial who wears a white lab coat, he knows, too.

Further reading

• Search for “PLOS ten simple rules”– Lots of valuable advice in short format

• Search for “Hamming you and your research”

• “The art of doing SCIENCE and engineering” by Richard Hamming is a book worth getting and reading