Question
How far down the abstraction chain do you need to understand to be a "great" programmer?
Answer
Allen Cheung I think nailed it with "Having to leap "between the layers" of abstraction is pretty much counter to what it means for a software layer to be abstracted."
To be a great programmer doing normal things you don't need to understand more than your "home" abstraction layer. Even things like pointers in C are
not necessary (that's become a ritual hazing interview question, and is
less about practical need-to-know and more a psychology test about "can
you fundamentally do certain tricky kinds of meta-thinking?").
That's almost everything that needs to be said, but there is one very special exception case that I want to mention, because it is hugely important: security-related programming, both defensive and offensive. Greatness at this level naturally leads to greatness at the ultimate level: foundational conceptual advances in computing at basic CS research level. Which means routine computer science depends on "stay in your level" greatness, but basic science advances depend on "multiscale and beyond" greatness.
This leads to the following totem pole of programming greatness:
Let me explain.
What is Multi-Scale Thinking Worth?
Answer: Generally nothing. It is a fun time-waster to satisfy curiosity.
When I was self-learning various kinds of programming (and long before I discovered I had little talent for any of them), I worked through the basics of all the layers primarily because I just didn't like the feeling of working with a magic black-box.
But by the time I got to the lowest hands-on level I've ever been (programming a stepper motor controller at machine code level by entering individual bits into registers on a learning circuit board), and the lowest "armchair theory" level I've been (control of circuit etching chemical processes as homework for a graduate control theory course, pop-science understanding of quantum tunneling as circuits get smaller...), the mystery was gone and along with it the motivation.
I found I could ignore most of it and get 99% of what I needed done within Matlab. That's when I decided I was not destined to be a programmer (great or mediocre), but more of an occasional tinkerer when pen-and-paper wouldn't do.
I think most people WILL follow this trajectory: explore out of curiosity, but once that's sated, you find your sweet spot in the programming universe, settle there and ignore other things/layers.
Except for the hacker-cracker types.
Multi-Scale Thinking in Security
I came away with one clear impression from my multi-scale journeys: cyberwarfare is nearly always a multi-scale thing. Think about the kinds of things crackers use: port-level operations like "sniffing," buffer overflows, taking advantage of unsanitized input fields etc. They live at one level, but the effects show up at a higher level typically. I don't know much about the modern state of the art, but when I went through the usual infatuation period with this stuff in high school (late 80s, early 90s), it was the same sort of thing: interrupt vectors, direct memory access, terminate-and-stay-resident etc. Stuff that didn't show up in my regular homework.
To work with this stuff you have to simultaneously "see" code at the application level at which it was written (i.e. if you are exploiting a hole in Internet Explorer, you have to think like a normal application programmer, about the usual things like UX, data models for the job being done etc.) AND you have to think "one level below" where the target's unexamined assumptions (and therefore vulnerabilities) live.
If you are writing malware, you have to be doubly clever about user psychology compared to product designers ("70% of users would probably click OK on a pop-up box that kinda looks like an anti-virus download box security alert" takes an insight about how users behave around routine things AND how this can be exploited, judo style). AND you have to think at the filesystem level below where viruses/malware do illegal things. Regular developers who obey the rules don't have to think about the filesystem stuff at all and about UX only in non-exploitative ways.
So this means crackers are basically smarter than normal application developer type s/w people, because they can think at more levels, and more moves ahead.
Which means good guys on the defense side have to be EVEN smarter, since they actually have to understand AND code at at least two levels, around more contingencies. So the totem pole for everyday programmers is:
In a way this is merely a manifestation in cyberwarfare of the basic principles of regular warfare: to win, you have to get "inside" the decision cycle of your opponent and operate at a higher tempo and create chaos and entropy. This usually means operating at a level below the abstraction level where your opponent works. For example, the Van Riper episode is a perfect non-programming example.
http://en.wikipedia.org/wiki/Mil...
This is why it can be so scary to be hit by a security problem. Your false sense of security at your level is suddenly violated as somebody takes control at a lower level. You react with anxiety as your world dissolves into chaos in mysterious ways.
Arms races naturally lead to innovation.
From Hacker-Cracker Greatness to Basic Research Greatness
Security attack/defense thinking is actually the amateur cousin of basic research thinking in computer science. If you are REALLY good at spanning multiple levels, the attack/defense game is probably going to be beneath you and bore you.
You have the talents to completely refactor the abstractions at work to come up with fundamentally more powerful new computing architectures. You aren't fighting the battle anymore. You're changing the battlefield and the rules. This is the kind of deep scale-free thinking that leads to amazing new conceptual advances in architecture and programming (say the shift from client/server to P2P or from structured to OO programming). You can't achieve breakthroughs at this level without being able to span all levels.
If the attack/defense creative-destruction arms race is about "thinking one level below" to gain an advantage, innovation is about going meta with respect to the entire abstraction stack. As Einstein said, "The significant problems we have cannot be solved at the same level of thinking with which we created them."
Hackers/crackers think "how can I get under this guys lowest level and control the tempo of the war?" The genius thinks, "how can I create an entirely new abstraction hierarchy that is more powerful?"
Simplest case in point: Alan Kay, one of the founding fathers of modern distributed computing. His main CONTRIBUTIONS were at the level of GUIs and programming languages. But his main INSIGHT was at the level of chips ("you should waste transistors").
So that's why the REAL totem pole is:
Today you can see this happening in quantum computing. These are people thinking about fundamental physics at the lowest level, and about algorithms that come close to threatening NP-completeness at another...
To be a great programmer doing normal things you don't need to understand more than your "home" abstraction layer. Even things like pointers in C are
not necessary (that's become a ritual hazing interview question, and is
less about practical need-to-know and more a psychology test about "can
you fundamentally do certain tricky kinds of meta-thinking?").
That's almost everything that needs to be said, but there is one very special exception case that I want to mention, because it is hugely important: security-related programming, both defensive and offensive. Greatness at this level naturally leads to greatness at the ultimate level: foundational conceptual advances in computing at basic CS research level. Which means routine computer science depends on "stay in your level" greatness, but basic science advances depend on "multiscale and beyond" greatness.
This leads to the following totem pole of programming greatness:
- Foundational CS research: think and code at all levels and about
the possibilities/power AND vulnerabilities of MANY multi-scale contingencies - Great defender: think at 2+ levels, code at 2+ levels, protect against vulnerabilities in MANY contingencies
- Great attacker: think at 2+levels, code at 1 level, attack ONE unprotected weak point
- Regular great programmer who doesn't do security features: think at one level, code at 1 level
Let me explain.
What is Multi-Scale Thinking Worth?
Answer: Generally nothing. It is a fun time-waster to satisfy curiosity.
When I was self-learning various kinds of programming (and long before I discovered I had little talent for any of them), I worked through the basics of all the layers primarily because I just didn't like the feeling of working with a magic black-box.
But by the time I got to the lowest hands-on level I've ever been (programming a stepper motor controller at machine code level by entering individual bits into registers on a learning circuit board), and the lowest "armchair theory" level I've been (control of circuit etching chemical processes as homework for a graduate control theory course, pop-science understanding of quantum tunneling as circuits get smaller...), the mystery was gone and along with it the motivation.
I found I could ignore most of it and get 99% of what I needed done within Matlab. That's when I decided I was not destined to be a programmer (great or mediocre), but more of an occasional tinkerer when pen-and-paper wouldn't do.
I think most people WILL follow this trajectory: explore out of curiosity, but once that's sated, you find your sweet spot in the programming universe, settle there and ignore other things/layers.
Except for the hacker-cracker types.
Multi-Scale Thinking in Security
I came away with one clear impression from my multi-scale journeys: cyberwarfare is nearly always a multi-scale thing. Think about the kinds of things crackers use: port-level operations like "sniffing," buffer overflows, taking advantage of unsanitized input fields etc. They live at one level, but the effects show up at a higher level typically. I don't know much about the modern state of the art, but when I went through the usual infatuation period with this stuff in high school (late 80s, early 90s), it was the same sort of thing: interrupt vectors, direct memory access, terminate-and-stay-resident etc. Stuff that didn't show up in my regular homework.
To work with this stuff you have to simultaneously "see" code at the application level at which it was written (i.e. if you are exploiting a hole in Internet Explorer, you have to think like a normal application programmer, about the usual things like UX, data models for the job being done etc.) AND you have to think "one level below" where the target's unexamined assumptions (and therefore vulnerabilities) live.
If you are writing malware, you have to be doubly clever about user psychology compared to product designers ("70% of users would probably click OK on a pop-up box that kinda looks like an anti-virus download box security alert" takes an insight about how users behave around routine things AND how this can be exploited, judo style). AND you have to think at the filesystem level below where viruses/malware do illegal things. Regular developers who obey the rules don't have to think about the filesystem stuff at all and about UX only in non-exploitative ways.
So this means crackers are basically smarter than normal application developer type s/w people, because they can think at more levels, and more moves ahead.
Which means good guys on the defense side have to be EVEN smarter, since they actually have to understand AND code at at least two levels, around more contingencies. So the totem pole for everyday programmers is:
- Great defender: think at 2+ levels, code at 2+ levels, protect against MANY contingencies
- Great attacker: think at 2+levels, code at 1 level, attack ONE weak point
- Regular great programmer who doesn't do security features: think at one level, code at 1 level
In a way this is merely a manifestation in cyberwarfare of the basic principles of regular warfare: to win, you have to get "inside" the decision cycle of your opponent and operate at a higher tempo and create chaos and entropy. This usually means operating at a level below the abstraction level where your opponent works. For example, the Van Riper episode is a perfect non-programming example.
http://en.wikipedia.org/wiki/Mil...
This is why it can be so scary to be hit by a security problem. Your false sense of security at your level is suddenly violated as somebody takes control at a lower level. You react with anxiety as your world dissolves into chaos in mysterious ways.
Arms races naturally lead to innovation.
From Hacker-Cracker Greatness to Basic Research Greatness
Security attack/defense thinking is actually the amateur cousin of basic research thinking in computer science. If you are REALLY good at spanning multiple levels, the attack/defense game is probably going to be beneath you and bore you.
You have the talents to completely refactor the abstractions at work to come up with fundamentally more powerful new computing architectures. You aren't fighting the battle anymore. You're changing the battlefield and the rules. This is the kind of deep scale-free thinking that leads to amazing new conceptual advances in architecture and programming (say the shift from client/server to P2P or from structured to OO programming). You can't achieve breakthroughs at this level without being able to span all levels.
If the attack/defense creative-destruction arms race is about "thinking one level below" to gain an advantage, innovation is about going meta with respect to the entire abstraction stack. As Einstein said, "The significant problems we have cannot be solved at the same level of thinking with which we created them."
Hackers/crackers think "how can I get under this guys lowest level and control the tempo of the war?" The genius thinks, "how can I create an entirely new abstraction hierarchy that is more powerful?"
Simplest case in point: Alan Kay, one of the founding fathers of modern distributed computing. His main CONTRIBUTIONS were at the level of GUIs and programming languages. But his main INSIGHT was at the level of chips ("you should waste transistors").
So that's why the REAL totem pole is:
- Foundational CS research: think and code at all levels and about the possibilities AND vulnerabilities of many cross-scale dynamics
- Great defender: think at 2+ levels, code at 2+ levels, protect against MANY contingencies
- Great attacker: think at 2+levels, code at 1 level, attack ONE weak point
- Regular great programmer who doesn't do security features: think at one level, code at 1 level
Today you can see this happening in quantum computing. These are people thinking about fundamental physics at the lowest level, and about algorithms that come close to threatening NP-completeness at another...