Why Software Development Has Always Been Difficult ?
I had this interesting debate with one of my NS buddies who is now working as a software project manager. He's a "born" programmer who has a good number of years in software development. Both of us lamented over the fact that good software developers are hard to find nowadays, and outsourcing is not a healthy long-term solution for our local IT environment.
As the conversation progressed, we slowly touched on this argument why software development has always been difficult. The following summarizes some of our arguments.
1. Required Skill Set
Programmers are often faced with this question: "what do I need to know to develop software". My answer is to learn what you need to know in order to develop the software that you are going to develop. There are far too many dimensions to consider.
If we look at technical skills, for programming languages, the commonly used languages are C, C++, Java, Delphi (Pascal), Python, Perl, C#, Ruby, Tcl, ActionScript, JavaScript. Wikipedia has a good list of programming languges. Next, you look at frameworks which may also depend on what platform you are going to develop for. In Windows, Visual .Net is popular; for cross-platforms, you have Qt, wxWindows, J2EE, etc. For web applications, there are even more: ASP.NET, ColdFusion, AJAX, Django, TurboGears, etc.
Other skils to consider is the depth of the software development methodology that you are familiar with. Most academics will advise against waterfall, and promote iterative models. Once again, Wikipedia has listed the common ones. Personally, I would prefer an architecture-centric approach. A good IT practioner must have undergone at least a full SDLC which usually consists of these phases: requirements analysis, design, development, testing and deployment. Every phase by itself will require some level of expertise which (I'm confident to say) can only be acquired through experience and exposure.
2. Software Writing and Design is a Creative Process
In my opinion, software development is a creative process. A system architect can dictate the high-level architecture design and how the components interface with each other. But when it comes to coding, different programmers will produce different codes that fufill the same requirements, but may work and break at different points. Furthermore, it is also difficult to determine how correct each code is; we can easily verify that the code is working but it's often harder to verify where the code breaks and throws exceptions. And the quality of the testing process is dependent upon how skillful the tester is.
3. Software Development Discipline is still a New Field
Software development only begun sixty years ago. Compare this to building architecture which is like almost a thousand years. There are few proven precedents and a lack of real-world activities which software development industry can leverage the base of experience on and to pattern after.
4. Difficult to Model Human/Business Process into Software
Even though we have 4GL (generation languages) which closely resemble English, it is still fundamentally difficult to program a computer to do useful things. For an application with a decent scope, you are looking at converting a good number of business processes and many human interactions into highly specific machine instructions. And most humans don't think in 1s and 0s; there's always a good nested levels of "if we have this condition then we do this else we do that." Most business processes require some level of human intervention to perform some in-house rules and modifications to proceed to next stage.
5. Programmers are often not Domain Expertises
Most schools often train their students to do programming and to acquire a basic understanding of how systems work. Once these students graduate and step into the working society, they can be expected to undertake programming jobs which may involve a set of knowledge that they are totally ignorant of. You can engage a domain expertise to help out in the analysis to straighten out all these issues and pen them down in the specifications, that is if you know how to ask intelligent questions that will bring out all these issues in the first place. Most domain expertises assume that their domain knowledge is "common sense" and expect you to be aware of them as well.
Conclusion
Software programming is an extremely difficult, demanding and torturous process that most people would rather not get involved in. We have been talking about the need to improve and guarantee software quality for ages for more than 40 years, and we still can't find a solution. For now, software programming will continue to be a mind-boggling process that will continue to plague the industry for a good number of years.
