The Unreasonable Ineffectiveness of Machine Learning in Computer Systems Research

[andychu]

I posted this comment with this article. Someone was getting too hyped up about a certain topic :) Let me know if I should submit a pull request or if you want to discuss further.

https://news.ycombinator.com/item?id=16036133

https://www.sigarch.org/the-unreasonable-ineffectiveness-of-machine-learning-in-computer-systems-research/

[andychu]

Another comment I made about the http://langsec.org line of research popped into my head:

https://lobste.rs/s/uyjzjc/science_insecurity_meredith_l_patterson

I heard about this line of work on Hacker News about 5 years ago, and someone was really hyping it up. I read their papers and watched some conference videos -- they could be described as "self-hyping" as well (saying everyone else is "doing it wrong"). Now that I think of it, I'm pretty sure the video in the lobsters is excessively hypey, although I didn't rewatch it.

It's a good line of research but IMO it's flawed, as I describe.

There was a followup paper by a different author I saw a few days ago that I plan to read:

https://news.ycombinator.com/item?id=16004044

It appears he is making their concept of "weird machines" more precise. I was always a bit bothered by their handwavy usage of that term, and I'm glad someone else agreed!

[andychu]

A few more things popped into mind:

This one is from the team/project at Google which has deployed machine learning the longest (early 2000's):

Machine Learning: The High Interest Credit Card of Technical Debt

https://research.google.com/pubs/pub43146.html

Using the framework of technical debt, we note that it is remarkably easy to incur massive ongoing maintenance costs at the system level when applying machine learning.

These links are more general and probably not as good, but might be worth reading:

Geoff Hinton / Yann Lecun on the problems of generalization, and unsupervised learning:

https://www.technologyreview.com/s/608911/is-ai-riding-a-one-trick-pony/

https://medium.com/@Synced/lecun-vs-rahimi-has-machine-learning-become-alchemy-21cb1557920d

http://www.information-age.com/google-deepminds-alphago-victory-not-true-ai-says-facebooks-ai-chief-123461099/

After 6+ years, I think everybody knows that IBM's watson was massively hyped, and didn't live up to ambitions. In case they don't:

https://www.technologyreview.com/s/607965/a-reality-check-for-ibms-ai-ambitions/

This hacker news thread is interesting:

https://news.ycombinator.com/item?id=14980358

It's basically some people working at IBM/Oracle realizing their jobs were somewhat fraudulent :-(

[hwayne]

We definitely need a couple of cold showers on ML and AI. Want to write up some hype/shower/caveats? The High Interested Credit Card might be a solid choice, as a technological shower, and "A Reality Check" (or maybe this one? I like that news thing a lot) as a "it hasn't been working in the real world" shower.

Sorry for throwing all the work onto you; this is waaaaaaaaaay outside of my area of expertise. Thanks so much for your help!

[andychu]

Sure I can write up something short that's consistent what's there. I will send a pull request later.

Another article that is interesting:

What's Worked in Computer Science

https://danluu.com/butler-lampson-1999/

Capability-based security is an interesting case. It's one that I want to work, but I have to admit it's still a "no" like he says.

[hwayne]

IMO we could probably replace this entire repo with a link to danluu.com

[hwayne]

Ping!