Gaurav Sood writes:
I recently taught a course on ML and reinforced for me how much of an engineering discipline it is.
In 2013, Girshick et al. released a paper that described a technique to solve an impossible-sounding problem—classifying each pixel of an image (or semantic segmentation). The technique that they proposed, R-CNN, combines deep learning, selective search, and SVM. It also has all sorts of ad hoc choices, from the size of the feature vector to the number of regions, that are justified by how well they work in practice. R-CNN is not unusual. Many machine learning papers are recipes that “work.” There is a reason for that. Machine learning is an engineering discipline. It isn’t a scientific one.
You may think that engineering must follow science, but often it is the other way round. For instance, we learned how to build things before we learned the science behind it—we trialed-and-errored and overengineered our way to many still standing buildings while the scientific understanding slowly accumulated. Similarly, we were able to predict seasons and the phase of the moon before learning how our solar system worked. Our ability to solve problems with machine learning is similarly ahead of our ability to put it on a firm scientific basis.
I’ve always said that statistics is a branch of engineering (it could be called “mathematical engineering” or “probability engineering”), so I’m in general agreement with Gaurav on this one. It seems uncontroversial to label machine learning as engineering—it’s typically taught in computer science departments which are in engineering schools, right?—and I’d say the same thing for statistics, even if it happens to be in the science program in a school.
There’s a separate question, though, in Gaurav’s post, which is, Which came first, science or engineering? I guess it depends on the example.
So now I think we should come up with a list of examples, for some of which the science came first and for some of which the engineering came first. Gaurav gave some examples above where the engineering came first. In other examples, the science came first, for example the A-bomb.
Or, it could be neither.
When I was writing the book chapter about correlations and causality, I made the point that there are perfectly fine models of correlation that are predictive and useful, for which I believe there is no hope of untangling the causal structure (“science”). Vice versa, there are lots of “science” with no practical applications.
Not only does this exist, it is 99.999% of what gets published. Then people try to interpret the arbitrary coefficients and end up with a mess of “conflicting” results.
Eg, look at vitamin D and covid. Or really anything and covid. All that happened was a series of fads that eventually faded away.
Just like Meehl described:
https://www3.nd.edu/~ghaeffe/Meehl(1978).pdf
I dunno whats up w that link, but its this paper:
Paul E Meehl. Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology.
Applied and Preventive Psychology
Volume 11, Issue 1, 2004, Page 1,
ISSN 0962-1849, https://doi.org/10.1016/j.appsy.2004.02.001.
It’s behind a paywall. Here’s a link to one that isn’t:
https://drive.google.com/file/d/1PCjXfdbsOukSSlKNa_ipqq-DeAuO2NXm/view?usp=drivesdk
There is also pure experimentation that is neither engineering nor science. Chemistry started this way, as did the pharmaceutical industry. Edison’s light bulb. Engineering is “I need to make a thing that will do X.” The base of knowledge from which that works is sometimes science, sometimes experience and sometimes iterative guessing.
Jonathan:
I would think of pure experimentation—for example, running voltage through lots of different filaments to see what happens—as a form of science.
I see your point, but if you characterize “see what happens” as science, then just about all of existence is science. I think of science more like the atom bomb. The *idea* came from a scientifically (not experientially) derived understanding of the world. To the extent that an understanding of physics narrowed the field of substances through which to pass a current, it’s science. But maximizing the utility of a lightbulb over n different filaments is just trial-and-error (or, generously, engineering, much of which is experientially informed trial-and-error, though some is inspired by scientific understanding)
Think of pharmaceuticals. You have pure serendipity (penicillin). Hard to characterize that as science, right? You have isolation of substances from cultural practices (aspirin) which also doesn’t sound much like science to me. And then you have a scientific theory of, say, cancer and then the engineering of drugs to block some pathway or attack some specific cancer cell. The part where you use the theory is science. At least that’s my take, FWIW.
> I see your point, but if you characterize “see what happens” as science, then just about all of existence is science.
I think I’ve heard it remarked before that science it just writing things down. So if you do those experiments or whatever but don’t document it or share it with others, you’re not doing science. If you write up the details – the design of your experiments and results, etc. – then it becomes science.
I’ve always considered engineering as fundamentally “problem solving” vs. science as “discovery.” Engineers do use science as a toolkit, but the added charge is for the engineer to find a solution given societal constraints.
Not much scientific progress was made on the properties of flint and chert (for points and scrapers) until long after they were obsolete.
But OTOH thinking about early metallurgy makes me question if it’s really true that engineering precedes science. Obviously people had to experiment without knowing everything about the chemical and thermodynamic properties of ores and metals. But OTOH, it seems just as obvious that they practiced some level of science – that is, testing, recording their observations, adjusting parameters and testing again – consistent with their knowledge. So is that engineering or science? Seems like either or both.
With that in mind manufacturing points and scrapers seems like one of the few examples I can think of where the original application was almost exclusively engineering – smashing away at various kinds of rocks in a purely trial and error effort, with only a few concepts about what useful rocks might look like and where they can be found. But even this leads to some concepts or observations and probably experiments about how to choose the best pieces, chip most effectively and quickly.
Even if I think about early agriculture, it seems almost impossible to even conceive it without some sort of rudimentary concept of science. Why would someone plant a seed without having made the observation that it will grow into a plant that produces something useful?
Well…
https://en.wikipedia.org/wiki/Chipmunk
In particular, squirrels are known for planting lots of trees in their territory.
Anoneuoid:
Squirrels and other rodent-like seed-eating creatures don’t “plant” seeds. They store them with every intention to eat them later bcz, as it turns out, seeds have defense mechanisms against being eaten, and predators know which plants rodents like, upping the risk of collecting every seed. So a seed that’s worth getting is a seed that needs to be eaten.
Here’s a wicked book about seeds, their evolution (Thor Hanson) and the ecosystems built up around them:
https://www.barnesandnoble.com/w/the-triumph-of-seeds-thor-hanson/1120254255
The squirrels bury the seeds in a place of their choosing, and many grow into plants. That is effectively “planting”.
It wouldn’t be hard for a human to figure out the plants grew in the same places they tried to store some seeds. Supposedly squirrels haven’t figured that step out, but a primate probably could, and a human surely could.
Also, the plants “want” animals to spread their seeds.
So it’s not planting unless one understands that the seed will grow into a plant and acts with that intention. By these semantics your question
is entirely vacuous!
Somebody:
1) That’s right. “to put or set in the ground for growth”. Note the critical two words “for growth”.
2) No, it isn’t. The fact that you don’t understand something doesn’t make it “vacuous”.
Mull it over for a bit.
Annoueoid:
“The squirrels bury the seeds in a place of their choosing, and many grow into plants. That is effectively “planting”.” No, it isn’t. They aren’t trying to grow a tree. Humans are the only species that “plants” things. At least AFAIK
“It wouldn’t be hard for a human to figure out the plants grew in the same places they tried to store some seeds.”
I’ll buy that, but it’s one of about ten thousand ways they **could** make the connection / understand the relationship between seeds and plants. My point isn’t that it’s hard to find process to observe where making the connection is possible. My point is that humans *did* make the connection, that making such a connection is an early form of scientific reasoning.
You asked “Why would someone plant a seed without having made the observation that it will grow?”
Now you’re stating that if something doesn’t have the foreknowledge that the seed will grow and the intention of growing it, it didn’t actually plant the seed. Hence, your definition precludes the meaning of your question; according to you, nothing can plant a seed without knowing that it’ll grow, so asking “why would something plant a seed without knowing that it’ll grow” is vacuous. It’s like defining manslaughter as an accidental death caused by another person, then asking “why would someone commit manslaughter intentionally?”
I’m pretty sure the problem here is that you’ve already forgotten what it is you wrote that people are responding to, and just reply with a defensive instinct.
All you have to do is walk around in nature for a while in spring and you’re going to see a seed with a sprout coming out of it. It’s pretty trivial to figure out where the plants come from. Similarly if you collect some beans and then store them somewhere you’ll find them sprouted if the conditions were right. Again, pretty trivial for hunter-gatherers to figure out. Most 3rd graders grow plants from a seed as a science project, it’s basically just leave a seed in some wet paper towels, but damp leaves would work fine too.
We think of Engineering as closely related to Science, with Engineers receiving a foundation in science and math, but that’s a recent occurrence. It had it’s origins in the 19th century but is mostly a 20th century phenomenon.
For example, Leonardo da Vinci’s education was that of “engineer”, or artisan, or tradesman broadly defined. He learned through apprenticeships and only mastered tradesman’s mathematics (financial arithmetic basically).
By contrast Newton was on a philosophy tract. He learned Euclid Elements and the beginnings of calculus. His education was generally pie-int-sky academic/philosophical stuff acquired from a University.
Two separate tracts was the norm throughout history. My guess is it took a critical mass of successful science and math before the two could be brought closer together, and before about 1700-1850 or so that critical mass just didn’t exist.
Leonardo may have been an artisan but there were also engineers before the time of Newton. Galileo studied and taught artillery, fortifications and other applications of mathematics and physics. Schools for engineers existed already in the 16th century and paved the way for the development of modern military academies in the late 17th / early 18th centuries.
The distinction between “artisan” and “engineer” wasn’t that clear. Da Vinci did work as a military engineer.
The claim wasn’t that engineers didn’t exist back then, but that it generally was a different educational tract from the “natural philosophers” which evolved into science.
The idea today of engineers and scientists being educated at the same institutions in similar ways and taking the same foundational classes in chemistry, physics, math, and so on is relatively recent.
The tipping point between these two different ways of creating engineers being roughly the 19th century. Give or take.
> Da Vinci did work as a military engineer.
Briefly and relatively late in his life. For many other people at that time it was a career.
I may have misunderstood your comment. My point was that Euclid’s Elements were not just of interest for natural philosophers. Engineers did learn mathematics – and many of the advances in mathematics in that period originated from engineering issues.
You gave the example of the artist/engineer Leonardo and I gave the counter-example of the scientist/engineer Galileo. Tartaglia and Cardano are two other examples of scientists/engineers.
The vast majority of people in ~1500 who were employed doing what we’d call “engineering” didn’t receive university educations and couldn’t do synthetic geometry proofs in the style of Euclid elements.
For that matter, most people in the 1800’s who were employed as “engineers” didn’t know what today are considered bread-n-butter background knowledge for engineers such as Fourier analysis.
I stand by the claim that the relationship between Engineering and Science we take for granted was beginning to building (in some ways) in the 17th and 18th centuries, was in major flux in the 19th century, and only solidified in the 20th century.
And I hypothesize that this lateness grew out of two things:
First, the historical accident that “engineering” grew out of the trades, while “science” grew out of philosophy.
And secondly, except for special cases and isolated examples, it took a lot of success in Science and Mathematics before it was worth Engineer’s time.
In 1700, Newton’s Principia wasn’t required reading for Engineers and most weren’t in a position to read it. In 2000, every undergrad engineer learns Newtonian Mechanics.
> For that matter, most people in the 1800’s who were employed as “engineers” didn’t know what today are considered bread-n-butter background knowledge for engineers such as Fourier analysis.
Interestingly Fourier was a professor at the École polytechnique – an engineering school created a few years after the revolution and converted by Napoleon into a military academy a decade later.
One of Fourier’s student’s was Navier – who would later become a professor there. Navier replaced Cauchy – another former student. Cauchy had replaced Poinsot – also a former student. Fourier was replaced by Poisson – who also had been a student. Jordan and Liouville were also students and later professors. Other notable students include Carnot, Clapeyron (two of the founders of thermodynamics), Coriolis, Fresnel, Gay-Lussac and Poincare. Other notable professors include Ampere, Bertrand, Hermite, Lagrange, and Laplace. Legendre was also associated with the school.
For me it is all trial and error, plus memory. Science (and engineering lore) is the story we come up with to help us remember and teach what we have learned by trial and error. We tell the (multi-part) story until something is seen which disagrees with it, then we revise it.
The story is very useful to us, Biological evolution invented the most remarkable things which we yet know of without a story, but we invent and refine things much faster. (Also we have the ability to do a lot of our trial and error in simulations, e.g., atom bomb development.)
Einstein is reputed to have said:
“Whether you can observe a thing or not depends on the theory which you use. It is the theory which decides what can be observed.”
Objecting to the placing of observables at the heart of the new quantum mechanics, during Heisenberg’s 1926 lecture at Berlin
“it’s typically taught in computer science departments which are in engineering schools, right?”
There’s probably some variation, but at for example Iowa State University there was both a Computer Science dept in the Liberal Arts and Sciences school, AND a Computer Engineering department in the Engineering school.
The CS dept would teach stuff about algorithms, data structures, programming language theory, automata theory, and such. The Computer Engineering school would teach stuff about data busses and timing circuits and instruction sets and microcode and processor design.
the distinction was basically the difference between “more mathy, more universal” (CS) and “more electronicsy, more architecture specific” (CompE)
Same thing I was going to say. Some CS is basically from math and in the same schools as math, Computer Engineering/Software Engineering is in another. And they will have (or not have) different calculus courses.
Epidemiology as engineering came way before the science. We learned how to stop disease outbreaks before we understood the causes. We even learned how to induce an immune response before we understood anything about the immune system.
Trying to create a reliable calendar for farming (and navigation), engineering problems, lead to astronomy. Horticulture and husbandry lead to the theories of evolution and genetics. Masonry and trade are probably the beginning of mathematics. Cooking and baking have been going on for millennia before anything close to chemistry was created. The telescope was invented way before Newton’s theory of optics. And, Galileo reportedly spoke to construction workers to gain an understanding of physics.
It is hard to think of any field of science that at its inception was not a series of engineering problems. But, there may be specific subdomains where the science came first. People have told me that no one could have even conceived of a laser before scientific understanding a photos emerged, but I don’t know if that is true.
” We learned how to stop disease outbreaks before we understood the causes. ”
Is that true? I mean yeah I know that viruses / bacteria weren’t known as the source of many diseases until the late 19th century. But some of the causes were understood, otherwise stopping the transmission of disease would not have been possible. People had a more rudimentary understanding of “the science” then than they do now, but to me they still employed science.
By the same token today there are unknowns about the exact mechanisms by which a given pathogen is communicated from person to person. In this pandemic, we may have finally learned that aerosol transmission is more significant than door-knob or hand-shake transmission. But our “six feet” of social distancing seems like mostly a guess derived from some simple ideas about droplet trajectory, and probably not even very good for aerosols.
IMO throughout our history we have deployed both science and engineering to find the best answers we can to our problems with the information available to us. The quality of that information improves over time, so we acquire more and more knowledge, but we’re still using the same methods to deploy that knowledge to solve problems.
Knowing come diseases are contagious is not the same as knowing the causes. Yes, the distinction between engineering and science is fuzzy, but I was playing along with Andrew’s questions. Most intellectual questions started as very practical questions. Masons were building incredible structures before Euclid, but of course masons had to have some understanding of geometry.
> statistics is a branch of engineering (it could be called “mathematical engineering” or “probability engineering”)
I would say that statistics as a branch of engineering already has a name: operations research. Interestingly that name and subject has a military origin – just like engineering.
On the other hand statistics was originally about the state – a subject also known as “political arithmetic”.
> It seems uncontroversial to label machine learning as engineering—it’s typically taught in computer science departments which are in engineering schools, right?—
It’s somewhat controversial (“Computer Science is no more about computers than astronomy is about telescopes.”). It can also be controversial whether machine learning is part of statistics – or if it’s the other way around. Or whether political science is a branch of economics. Etc.
An important caveat — operations research involves mathematical theories and methods that extend beyond what we would typically think of as statistics.
For example, operations research involves at its core on optimization problems, and among the methods used to solve such optimization problems include linear programming. Now linear programming (in its basic form) is not a methodology that I would tend to think of as part of statistics, per se — there is no notion of probability involved there, no use of analyzing data, etc. It is just solving a series of linear equations with constraints.
Obviously, more complex models and methods used within operations research are often derived from statistics, as well as different branches of mathematics.
That’s true, statistics/probability would be only part of OR – which includes also deterministic things like optimisation and game theory. (But deterministic problems are just a special case of more general stochastic problems!)
While one could also ask if OR isn’t “just” mathematics it seems that statistics got more and more mathematical while OR deliberately remained separate. The name of “dynamic programming”, for example, was chosen to be as far from “mathematical optimisation” as possible.
(Side note: it’s also interesting that mathematics was lumped so happily with science in the context of a discussion about the separation between science and engineering.)
A useful distinction, which complements that between science and engineering, is that between techniques and technologies. Techniques refer to procedures used to solve problems, ranging from simple to highly sophisticated. These techniques can be “right” or “wrong,” and examples include having milk with honey to treat a sore throat, sowing lettuces on a full moon night, or the complicated mixtures used by alchemists in their attempts to make gold.
A technology, on the other hand, is a technique whose validity is supported by science. There is evidence that statistical techniques, such as crude approximations to what we now know as sampling or experimental design, were used historically long before they truly became technologies, i.e., when they were supported by sound mathematics. In that sense, statistics is a technology: a collection of mathematically validated techniques used to solve problems.
However, I am still under the impression that part of what machine learning (ML) consists of is still in its pre-technological stage. Part of the ML literature describes things that seem to work in some contexts without providing a clear justification for their effectiveness. This is not entirely bad, as someday someone may explain to us why these techniques work, as happened when Hastie, Tibshirani, and others (do I recall the story correctly?) explained why the powerful prediction method, boosting, worked well after it had been widely adopted.
In the practice of ML (and I would say in the practice of statistics as well) outside of academia, pre-technological thinking is far more prevalent.
I don’t get it. Is this supposed to be a real distinction, or is Dr. Gelman just trolling which he does in about 25% of his postings*. Is medicine, doctoring, science or engineering? Destroying the pancreas led to diabetes in mice. Science or engineering? The 1923 Nobel prize was given for stuff coming from this, and few Nobels were more deserving. Sickle cell disease is secondary to a mutation in the beta chain of the hemoglobin molecule. About 25% of breast cancers overexpress a receptor called HER/neu, and blocking this receptor is quite useful. Lots of times in medicine we use things we don’t understand fully; are you saying that kind of thing is engineering?
*I don’t intend any disrespect by that remark.
It’s all about definitions (how do you define “science” and what is “engineering”?). The discovery of insulin was pretty much all “science” (“focussed, systematic and monitored experimentation to explore the mechanism of natural phenomena”, one might say) even if there was a typical amount of serendipidity involved (e.g. the identification of the pancreas as the source of an “internal secretion”, the loss of which caused diabetes, was made “by mistake” in 1889).
A better distinction which Carlos refers to in the post above yours is that between “science” and “technology”. e.g the breakthroughs in the discovery of insulin in early 1920’s came largely from new technologies (especially improvements in being able to measure glucose levels in small volumes of blood and in use of alcohol fractionation to purify proteins). **
Often, maybe almost always, the progression is interlinked ; i.e. “development of technologies based on scientific study that themselves facilitate new scientific discoveries – science leads to novel technology leads to new science”. An example is the Human Genome Project that was essentially application of massively scaled-up technologies built on decades of scientific study on the nature and structures of genomes and DNA and a huge toolkit of molecular biology techniques including DNA sequencing. Nothing fundamentally new had to be developed even if in the course of application of these technologies the rate and cost-improvement in genomic DNA sequencing increased dramatically. That application of technology has lead to a huge amount of new science and discovery in human genetics, molecular and genetic medicine and evolution – “science generates technology generates science”
** Michael Bliss “The Discovery of Insulin” is a great account of this IMO.
MRI and CAT scans came from the realization that the Radon transform could make it possible to get images of the body that seemed impossible. Of course then there needed to be engineering to find the best coefficients and find practical patterns of radiation.
“It seems uncontroversial to label machine learning as engineering—it’s typically taught in computer science departments which are in engineering schools, right?—and I’d say the same thing for statistics, even if it happens to be in the science program in a school.”
Andrew, it is worth pointing out that what you state above applies primarily in the US.
In the majority of universities in Canada (including my alma mater, University of Toronto), computer science is usually taught as a department within the science faculty. At the University of Waterloo, computer science is taught within the Faculty of Mathematics.
It’s kind of hard to take the question seriously when it’s based on bizarre nonsense like “we were able to predict seasons and the phase of the moon” being “engineering”.
I mean, sure, if you redefine things so that 90% of actual science is “engineering”, and only (I guess) elaborate mathematical models are “science”, then “engineering” usually precedes “science”.
Otherwise, using sensible definitions of science and engineering, the usual case is going to be that “engineering based on science” was rare prior to the 18th or 19th Century, except insofar as practical empirical knowledge is often feeding into tool-making, building, etc. (“Huh, the wood from this particular tree tends to be pretty strong and spring-y; maybe it would be useful for making bows?”)
A more interesting question might be: when did “theoretical”/”scientific” knowledge lead to practical “engineering” for the wrong reasons? I’m vaguely curious if the accidental discovery/creation of gunpowder by Chinese alchemists was partly the result of (erroneous) theoretical speculations about the best way to create immortality elixirs, for example.