Back when I was just an intern, spending my summers chilling out in the bay area (and doing work sometimes), I started reading Quora and noticed people debating whether the compensation for software engineering was disproportionate to the difficulty of the job. (I have no idea how long this debate has been going on.) It seemed pretty obvious to me back then the answer was
yes, or, at least, that the actual difficulty of software engineering is much less than people think when they haven’t been exposed to the right opportunities to learn programming. It seemed inevitable that salaries would eventually fall as a result of more people being trained to enter the field.
But I saw a contrary point of view being expressed in the answers sometimes, namely that software engineering is stressful. Some of that, of course, is due to factors that are not universal, such as long hours and on-call rotations. But it was argued that the job is also demanding in other ways, such as the need to constantly learn new technologies and new developments in existing technologies. At the time, I don’t think I had the perspective to be able to understand this.
Now that I’ve been a full-time software engineer for almost three years, I think I’m starting to understand. I’m lucky to be on a team where I’m nearly free from certain stressors that my software engineer friends have to deal with, such as pager duty, daily scrums, and frequent meetings in general. But I also experience job-related stress of a particular kind, and I suspect this experience may be close to universal: it’s that I don’t know what the next problem is that needs to be solved, but I know I’ll have to do everything I can to solve it.
Almost all jobs involve problem-solving to some extent, but occupy different positions on a spectrum. On one end of the spectrum are jobs that involve performing a bounded set of responsibilities in some fixed way, such as working in a call centre where you just follow a script and transfer the caller to a specialist whenever that fails. Those jobs can often be extremely stressful due to the long hours, low pay, and general working conditions, but not due to the nature of the problems to be solved. Many jobs fall somewhere near the middle of the spectrum, consisting mostly of fixed responsibilities with some variability in challenges and some room to invent ideas for how to better perform those responsibilities. I would argue that software engineering stands at the far end of the spectrum, together with other traditionally highly compensated jobs such as that of the doctor or lawyer. For while a lecturer can present the same lecture twice to different audiences and a bus driver can drive along the same route twice on different days, a software engineer should not even write the same ten lines of code twice, or perform the same operations twice—they should be factoring out the code into reusable components and automating the operations.
It follows that if you still have a job as a software engineer, it is only because you are needed in order to deal with a never-ending stream of new problems that are qualitatively different from the ones you have solved before. And that means you have to constantly search for solutions to those problems. You cannot follow a fixed set of steps to try to solve the problems you are faced with, and claim that your responsibilities have been fulfilled. You have to apply your technical and creative skills to their fullest extent.
As I said, this makes software engineering comparable to practising medicine or law. Your job is not to apply some fixed set of techniques but to treat the patient or to win the case (or perhaps settle it favourably, yeah, yeah, I know) and you go to school for a long time so you can learn the knowledge needed to prepare a solution to the problem from first principles as it may be different in significant ways from cases that you have dealt with in the past. Being constantly challenged in this way is very likely to be stressful, particularly when you aren’t sure whether the problem even has a solution, or how you should feel about your own personal inability to solve it.
So I think when people argue that software engineers are disproportionately compensated, they should consider that people whose jobs consist of this kind of generalized problem solving within a broad domain do tend to be well-compensated and that this should not be unexpected when you consider the demands of those jobs (and I hardly hear anyone saying doctors are overpaid).
I wonder whether there are highly compensated jobs that are much closer to the other end of the spectrum, where you do more or less the same thing every day but you get paid a lot of money to do it. I suspect that a job of that type would be very desirable and that it could only continue to be well-compensated if it either had artificial barriers to entry or required skills that are difficult to acquire. If jobs in the former category existed in the past, they have probably mostly disappeared due to globalization and other forces eliminating such barriers. But perhaps there are jobs that fall into the second category.