Software engineering

Software engineering (SE) is the profession concerned with creating and maintaining software applications by applying computer science, project management, domain knowledge, and other skills and technologies.

Software applications (including ATMs, compilers, databases, email, embedded systems, graphics, office suites, operating systems, robotics, video games, and the world wide web) embody social and economic value, by making people more productive, improving their quality of life, and enabling them to do things that would otherwise be impossible. SE technologies and practices (including databases, languages, libraries, patterns, platforms, processes, standards, and tools) help developers, by improving productivity and quality.

Software engineering is a community of people. The SE community includes 630,000 practitioners and educators in the U.S. and 1,400,000(?) practitioners in the E.U., Asia, and elsewhere; and is about 60% the size of traditional engineering. American SE pioneers include Kent Beck, Barry Boehm, Fred Brooks, Watts Humphrey, and David Parnas.

There is considerable debate over whether software development should be considered a branch of traditional engineering, a branch of computer science, an independent scientific field, or a non-scientific craft. This article attempts to be neutral on this issue, but errs on the side of being independent to clarify the differences between fields.

As of 2004, in common parlance the term software engineering is used with at least three distinct meanings:

• As the usual contemporary term for the broad range of activities that was formerly called programming or systems analysis;
• As the broad term for the technical analysis of all aspects of the practice, as opposed to the theory of computer programming;
• As the term embodying the advocacy of a specific approach to computer programming, one that urges that it be treated as an engineering profession rather than an art or a craft, and advocates the codification of recommended practices in the form of software engineering methodologies.

Software Engineering today

Software Engineering matters

In the U.S., software drove about 1/4 of all increase in GDP during the 1990s (about $90 billion per year), and 1/6 of all productivity growth (efficiency within GDP) during the late 1990s (about $33 billion per year). Software engineering drove $1 trillion of economic and productivity growth over the last decade. See also software engineering economics.

Software engineering changes world culture, wherever people use computers. Email, the world-wide web, and instant messaging enable people to interact in new ways. Software lowers the cost and improves the quality of health-care, fire departments, and other important social services.

Successful projects where software engineering methods have been applied, include linux, the space shuttle software, and automatic teller machines. When it is cheaper to run a business or agency with software applications than without, businesses and agencies often invest in computers, software, and personnel.

Today's education

People from many different educational backgrounds make important contributions to SE. The fraction of practitioners who earn computer science or software engineering degrees has been slowly rising. Today about 1/2 of all software engineers earn computer science or software engineering degrees. For comparison, about 3/4 of all traditional engineers earn engineering degrees.

Software: About half of all practitioners today have computer science degrees, which are the most relevant degrees that are widely available. A small, but growing, number of practitioners have software engineering degrees. Today in the U.S., about 2,000 universities offer computer science degrees and about 50 universities offer software engineering degrees. Most SE practitioners will earn computer science degrees for decades to come, though someday, this may change.

Domain: Some practitioners have degrees in application domains, bringing important domain knowledge and experience to projects. In MIS, some practitioners have business degrees. In embedded systems, some practitioners have electrical or computer engineering degrees, because embedded software often requires a detailed understanding of hardware. In medical software, some practitioners have medical informatics degrees, or general medical or biology degrees.

Other: Some practitioners have mathematics, science, engineering, or other technical degrees. Some have philosophy, or other non-technical degrees. And, some have no degrees. Note that Barry Boehm earned degrees in mathematics and Edsger Dijkstra earned degrees in physics.

Graduate software engineering degrees have been available from dozens of universities for a decade or so. Undergraduate software engineering degrees are being established at many universities. A new international curriculum for undergraduate software engineering degrees is currently being defined by the CCSE.

Today's practice

Practitioners specialize in many roles in industry (analysts, developers, testers, technical support, managers) and academia (educators, researchers).

Most software engineers work as employees or contractors. Software engineers may work with a profit-seeking corporation (a business), a government agency (civilian or military), or a non-profit agency (a school or .org like Wikipedia). Some software engineers work for themselves as free agents.

There is considerable debate over the future employment prospects for Software Engineers and other IT Professionals. For example, an online futures market called the Future of IT Jobs in America attempts to answer the question as to whether there will be more IT jobs, including software engineers, in 2012 than there were in 2002.

Today's debates

Many debates are raging within SE. As software becomes more pervasive, we all recognize the need for better software, but we disagree on how.

Technologies and Practices: What is the best way to make more and better software? SEs advocate many different technologies and practices, with much disagreement. This debate has gone on for 60 years and may continue forever.

Identity: Is SE a branch of computer science, a branch of traditional engineering, or a field that stands on its own? Recently, software engineering has been finding its own identity and emerging as an important field. Yet, some advocate making SE a part of traditional engineering and others advocate keeping SE a part of computer science.

Professionalism: What will SEs do about professionalism, licensing, and ethics? Licensing is a polarizing issue. Some fiercely advocate it. Others staunchly oppose it.

Success: Is SE a success or a failure? Some look to the enormous economic growth and productivity gains enabled by software and claim that software engineering is a huge success. Others point to the ongoing problems with crashing operating systems and computer viruses and claim that software engineering has failed. How can we reconcile these points of view?

Current directions for software engineering

Aspect-oriented programming and agile methods are important emerging SE technologies and practices.

Aspects help programmers deal with ilities by providing tools to add or remove boilerplate code from many areas in the source code. Aspects describe how all objects or functions should behave in particular circumstances. For example, aspects can add debugging, logging, or locking control into all objects of particular types. Researchers are currently working to understand how to use aspects to design general-purpose code. Related concepts include generative programming and templates.

Agile Methods guide software development projects that evolve rapidly with changing expectations and competitive markets. The heavy, document-driven processes (like CMM and ISO 9000) are fading in importance. Some people believe that companies and agencies export many of the jobs that can be guided by heavy-weight processes. Related concepts include extreme programming and lean software development.

The Future of Software Engineering conference (FOSE) held at the ICSE 2000 documented the state of the art of SE in 2000 and listed many problems to be solved over the next decade. The Feyerabend project attempts to discover the future of software engineering by seeking and publishing innovative ideas.

Evolution

Software engineering has evolved steadily from its founding days in the 1940s until today in the 2000s. Applications have evolved continuously. The ongoing goal to improve technologies and practices, seeks to improve the productivity of practitioners and the quality of applications to users.

1945 to 1965: The origins

The term software engineering first was used in the late 1950s and early 1960s. Programmers have always known about civil, electrical, and computer engineering and debated what engineering might mean for software.

The NATO Science Committee sponsored two conferences on software engineering in 1968 (Garmisch, Germany) and 1969, which gave the field its initial boost. Many believe these conferences marked the official start of the profession.

1965 to 1985: The software crisis

Software engineering was spurred by the so-called software crisis of the 1960s, 1970s, and 1980s, which identified many of the problems of software development. Many software projects ran over budget and schedule. Some projects caused property damage. A few projects caused loss of life. Some used the term software crisis to refer to their inability to hire qualified programmers. The software crisis was originally defined in terms of productivity, but evolved to emphasize quality.

Cost and Budget Overruns: The OS/360 operating system was a classic example. This decade-long project from the 1960s and 1970s eventually produced one of the most complex software systems ever created. OS/360 was one of the first large (1000 programmer) software projects. Fred Brooks claims in Mythical Man Month that he made a multi-million dollar mistake by not developing a coherent architecture before starting development.

Property Damage: Software defects can cause property damage. Poor software security allows hackers to steal identities, costing time, money, and reputations. An expensive European Ariane rocket exploded because of software.

Life and Death: Software defects can kill. Some embedded systems used in radiotherapy machines failed so catastrophically that they administered lethal doses of radiation to patients.

Peter G. Neumann keeps a contemporary list of software problems and disasters at Computer Risks .

The software crisis has been slowly fizzling out, because it is unrealistic to remain in crisis mode for more than 20 years. SEs are accepting that the problems of SE are truly difficult and only hard work over many decades can solve them.

1985 to present: No silver bullet

For decades, solving the software crisis was paramount to researchers. Seemingly, they trumpeted every new technology and practice from the 1970s to the 1990s as a silver bullet to solve the software crisis.

Tools, discipline, formal methods, process, and professionalism were touted as silver bullets. Tools: Especially emphasised were tools. Structured programming, object-oriented programming, CASE tools, Ada, documentation, standards, and UML were touted as silver bullets. Discipline: Some pundits argued that the software crisis was due to the lack of discipline of programmers. Formal methods: Some believed that if formal engineering methodologies would be applied to software development, then production of software would become as predictable an industry as other branches of engineering. They advocated proving all programs correct. Process: Many advocated processes and methodologies like CMM. Professionalism: This led to work on a code of ethics, licenses, and professionalism.

In 1987, Fred Brooks published the No Silver Bullet article, arguing that no individual technology or practice would ever make a 10-fold improvement in productivity within 10 years.

Debate about silver bullets raged over the following decade. Advocates for Ada, components, and processes continued arguing for years that their favorite technology would be a silver bullet. Sceptics disagreed. Eventually, almost everyone accepted that no silver bullet would ever be found. Yet, claims about silver bullets pop up now and again, even today.

Some interpret no silver bullet to mean that SE failed. The search for a single key to success never worked. All known technologies and practices have only made incremental improvements to productivity and quality. Yet, there are no silver bullets for any other profession, either. Others interpret no silver bullet as proof that SE has finally matured and recognized that projects succeed due to hard work.

Major developments

There are a numbers of areas where the evolution of software engineering is notable.

Emergence as a profession: From the mid-1990s to the mid-2000s, software engineering emerged as a bona fide profession, to stand beside computer science and traditional engineering. See also software engineering professionalism.

Role of women: In the 1940s, 1950s, and 1960s, software was often written by women. Men often filled the higher prestige hardware engineering roles. Grace Hopper and many other unsung women filled many programming jobs during the first several decades of software engineering. Today, fewer women work in software engineering than in other professions. Saying that this is sexual discrimination is too simple, because it related directly to individual identity. In one sense, software engineering is the masculinization of programming. The roles women fill in SE continue evolving. Today, more women in software engineering fill the social roles of analysis, training, documentation, and management; and fewer do hard-core technical development.

Processes and Methodology: Processes and methodologies have become big parts of software engineering. Many practitioners resist process, which often treats them impersonally like machines, rather than like people.

Cost of hardware: The relative cost of software versus hardware has changed substantially over the last 50 years. When mainframes were expensive and required large support staffs, software projects could be expensive. Because powerful PCs are cheap, software projects must become cheaper, in comparison.

Comparing related fields

The relationships between software engineering and the fields of programming, computer science, and traditional engineering have been debated for decades. Software engineering resembles all of these fields, but important distinctions exist.

Comparing programming

Both programmers and software engineers work on all sizes of projects: small and large.

Programmers emphasize the task of writing code to produce working software applications, independent of budget and schedule.

Software engineering tries to encompass software projects more completely, including budget and schedule; fits in a large business context with relationships to marketing, sales, production, installation, training, support, and operations; and methods to construct large applications that individual programmers cannot write alone.

Comparing computer science

Many compare software engineering to computer science and information science like they compare traditional engineering to physics and chemistry.

About half of all software engineers earn computer science degrees. Yet on the job, practitioners do applied software engineering, which differs from doing theoretical computer science.

Comparing traditional engineering

Software engineers aspire to build low-cost, reliable, safe products; much like traditional engineers do. Software engineers borrow many metaphors and techniques from traditional engineering disciplines, including requirements analysis, quality control, and project management techniques. Traditional engineers also borrow many tools and practices from software engineers. Yet, there are also many differences between SE and TE.

In the U.S., there are about 10 times as many software engineers as computer engineers. The software engineering community is about 60% as large as the traditional engineering community.

Fighting over issues

The term engineering causes a lot of confusion.

With about 612,000 software engineers in the U.S., and many more around the world, there should be room for many different opinions and approaches. A consensus has yet to emerge.

Right to use the word engineering

The wrangling over the status of software engineering (between traditional engineers and computer scientists) can be interpreted as a fight over control of the word engineering. Traditional engineers question whether software engineers can legally use the term.

Traditional engineers (especially civil engineers and the NSPE) claim that they have special rights over the term engineering, and for anyone else to use it requires their approval. In the mid-1990s, the NSPE sued to prevent anyone from using the job title software engineering. The NSPE won their lawsuit in 48 states.

However, SE practitioners, educators, and researchers ignored the lawsuits and called themselves software engineers anyway. The U.S. Bureau of Labor Statistics uses the term software engineer, too. The term engineering is much older than any regulatory body, so many believe that traditional engineers have few rights to control the term.

The fields of data engineering, knowledge engineering, user interface engineering, and so on have similar concerns about the term engineering. Even smaller or newer fields of biological engineering, safety engineering, and corrosion engineering have these concerns.

Substance versus metaphor

Some believe that the name SE means that practitioners must also be traditional engineers. Others believe that engineering is only a metaphor that SEs should apply appropriately.

Substance: Those who define software engineering as a branch of traditional engineering often believe that SEs apply concepts from traditional engineering to software development. This means that software engineering student should study physics, chemistry, and calculus; practitioners should earn professional licenses; and so on. They believe engineering provides a structured, logical approach, and therefore, a stable final product.

Metaphor: Others are inspired by traditional engineering, but believe that software needs its own solutions. They believe that many traditional engineering concepts cannot apply, because software is fundamentally different than bridges and roads. For example, traditional engineers do not use compilers or linkers to build roads. They believe that students should study computer science and other useful topics, and that practictioners do not necessarily need licenses.

Relation to other terms

Prior to the mid-1990s, most software practitioners called themselves programmers or developers, regardless of their actual jobs. Many people prefer to call themselves software developer and programmer, because we widely agree what these terms mean, while software engineer is still being debated.

The term programmer has often been used as a pejorative term to refer to those who lacked the tools, skills, education, or ethics to write quality software. In response, many practitioners called themselves software engineers to escape the stigma attached to the word programmer. In many companies, the titles programmer and software developer were changed to software engineer, for many categories of programmers.

These terms cause confusion, because some denied any differences (arguing that everyone does essentially the same thing with software) while others use the terms to create a difference (because the terms mean completely different jobs).

Fighting over priorities

Everyone agrees that we want better software, but we disagree on priorities, approaches, and on what an individual should do in a specific circumstance. Everyone seems to advocate a different combination of the following issues. Proponents and methodologists advocate conflicting solutions and often heatedly debate their merits. All subfields mix the following priorities to varying degrees.

Management: Some advocate that software engineering is primarily about the management practices necessary to make reliable budgets and schedules. People at the Software Engineering Institute took this approach and created the CMM.

Formal methods: Some advocate applying rigorous mathematical analysis to computer programming, especially proofs of correctness. They believe that traditional engineering is carried out with mathematical rigor, while programming is an iterative, trial-and-error process. These advocates strive to make programming more rigorous.

Process: Some advocate that software engineers must follow step-by-step processes, much like assembly line workers. This inspired CMM, ISO 9000, RUP, SPICE, and other methods and processes.

Tools: Some advocate that software engineering means tools, especially CASE tools (like Unix tools and IDEs) that emphasize high-level architecture issues. Today's CASE tools emphasize UML.

Ethics: Some advocate that software engineering is mostly about codes of ethics and social responsibility. They sometimes argue that bugs are due to lapses of ethics.

Licenses: Some advocate defining software engineering in terms of professional licenses, like some traditional engineers have. The biggest advocates of this position are from Texas and Canada, where the state governments sponsor licenses for SEs.

Degrees: Some advocate defining SE by college degrees. Most professions have college degrees tailored to the needs of practitioners. Many graduate software engineering degrees are available and undergraduate degrees are becoming available. (Move to SE education?)

Cost and Time versus Quality: Subfields like consumer applications are sensitive to time and cost. Subfields like military and medical applications are sensitive to quality.

Criticisms of software engineering

Critics argue that many of the foundations of software engineering are inherently flawed. The following paragraphs list many criticisms and responses. Note that many of these criticisms apply to other human activities including business and education.

Managing Expectations

Criticism: One key to successful software engineering projects is managing the customer's expectations to something that can be built and delivered. So, the field resembles marketing or sociology more than traditional engineering with its responsibilities to society at large and the perils of legal liability when they fail to protect the public interest.

Response: Every profession manages expectations, including all branches of engineering. Moreover, responsibility to society means meeting the expectations of the general public, which is often a stakeholder.

Poor Requirements

Criticism: The requirements for most SE projects are incomplete or inconsistent. Some clients have little experience writing requirements. Other clients do not know what they want, and say "I'll know it when I see it" (IKIWISI). Even experienced clients who know exactly what they want may not precisely articulate their requirements. Clients often expect much more than they write in the requirements. And, requirement documents can describe applications that have no computable or practical solutions.

Response: One response is to avoid all projects with poor requirements, which would avoid most projects. Another response is using agile development and rapid prototyping to clarify project goals and deliver value (the most important requirements) quickly. Embedded systems are special in that they fit inside products that are designed by other engineers, who can sometimes define the software requirements completely.

Rising Complexity

Criticism: Critics argue that the complexity of requirements and user expectations have only increased. The probability of failure increases with the size, scope and complexity of the project. Technologies and practices have consistently improved over the years, but the gap between what is expected and what is delivered has not improved.

Response: Rising complexity actually shows the success of practitioners, because demand naturally follows supply. When clients demand more, it shows their belief that their demands will be supplied.

Ongoing Change

Criticism: Practitioners continually develop new technologies and practices and use them whenever possible. Critics argue that ongoing change proves that older technologies and practices were failures.

Response: Many view ongoing change as proof that software engineering successfully learns and grows. Traditional engineers also continually develop new technologies and practices.

Ongoing Failure

Criticism: Critics argue that incomplete or poorly designed systems are still too common. The early disasters in the field did not prevent subsequent disasters.

Response: No field that strives to do bigger and better projects has ever avoided all unintentional failures. Traditional engineers also have ongoing failures: automobiles kill 40,000 people every year in the U.S.; Three Mile Island, Chernobyl, and Bhopal Disaster harmed thousands; Space Shuttles Challenger and Columbia blew up; MTBE added to gasoline to reduce air pollution also contaminated drinking water. Although large, reliable software systems can be and have been constructed, software projects that fail during construction or in service are still too common. Ongoing failure is a problem for both traditional engineers and software engineers. Some claim that software engineering is already as predictable and reliable as many fields of engineering, such as space or biological engineering.

Failure to Pinpoint Causes

Criticism: Critics argue that unlike traditional engineers (who analyze failures, find precise causes, and set up guidelines to avoid them in the future), software engineers routinely fail to pinpoint causes of failure or delay precisely enough to avoid repeats in the future.

Response: Debugging is the activity of pinpointing the cause of failures in applications. Process improvement includes the activity of pinpointing the cause of process problems. Software engineers routinely pinpoint causes and then use the results to create better languages, databases, processes, and applications.

Nothing New

Criticism: Critics argue that software engineers created nothing on their own, but merely use what computer scientists already know.

Response: Software engineers developed optimizing compilers, make, cvs, extreme programming, scripting, and bug databases on their own, out of necessity. Regardless of who creates or improves technologies and practices, software engineers are bright enough to adopt the best ones.

Anyone Can Do SE

Criticism: Many bright people from other fields (engineers, scientists, business people) write spreadsheet templates or simulations, and eventually switch to writing large applications, and believe that they do software engineering. So, software engineering is not a special skill.

Response: Software engineering is a skill that is refined through long learning and practice. (If it were real engineering you could simply teach how to do things correctly, omitting the long learning phase!) Software engineers are the ones who get the necessary education and experience, and keep up with evolving technologies, practices, and applications. This is true of every skill.

We Do Not Know What SE Is

Criticism: SE does not yield to the standard ways of categorization, under traditional definitions of engineering (calculus and science). This claim is often made by critics who want to impose their own definitions on everyone else.

Response: We know a lot about what software engineering is. Software engineering is grounded in SE technologies and practices, and applications; and in the community of SE practitioners. Of course, software engineers continue to disagree about many details.

No Software Science

Criticism: Civil, electrical, mechanical, and chemical engineering build on solid results from physics and chemistry. These results enable assembling complex systems in a principled and systematic way. No corresponding results are available for software: With software, we don't know how to systematically decompose complex systems into parts.

Response: Software engineering builds on solid results from computer science and information science. These results enable the building of very sophisticated software systems.

No Theorems About People and Projects

Criticism: No theorems explain why one software engineer is more productive than another. No theorems explain why some software projects succeed and others fail. Without such knowledge no engineering is possible.

Response: No theorems explain why one mechanical engineer is more productive than another. No theorems explain why some some civil projects go over budget and fail, for example why the Big Dig in Boston went way over budget, or why 2 space shuttles blew up. No theorems about people or projects exist anywhere.

Meaning of Success

Criticism: According to a study by the Standish Group (ref?) in 2000, 28 percent of software projects were complete successes (meaning they were executed on time and on budget), and 23% failed outright.

Response: Many engineering projects fail to live up to expectations: many bridges and buildings run over budget or schedule. Consider that 40% of all space shuttles have blown up and the rest have been out of service for years. Almost all custom housing projects runs over budget and schedule. Success rates for software projects are meaningless without context.

Magic

Criticism: When programmers work hard and the system works, customers frequently do not appreciate how difficult the task was.

Response: This is true for every profession.

Software Creation is Inherently Creative

Criticism: In Zeppelins and Jet Planes [1] , Philip Armour argues that software is executable knowledge, which is discovered in a creative process, where trial and error, learning, and the ability to challenge one's assumptions are important. The true "construction" phase of software development is already automated by compilers and linkers. The difficult aspects relate to gathering requirements and designing systems. These are more like a craft than a science or engineering task. So, the potential benefit of software engineering is limited to making it easier for developers to try out ideas, to discover errors earlier, and to give them information about the state of a software system.

Response: Many software engineers use Agile Methods to embody the creative nature of software engineering and to foster learning.

No Consensus

Criticism: In traditional engineering there is a clear consensus how things should be built, which standards should be followed and which risks must be taken care of: If an engineer does not follow these practices and something fails he gets sued. There is no such consensus in software engineering: Everyone promotes their own methods, claiming huge benefits in productivity, usually not backed up by any scientific, unbiased evidence.

Response: This is because SE is a young discipline.

Conclusion

The software engineering profession is big, important, successful (if imperfect), young (yet learning rapidly). SE practitioners continue improving their technologies and practices, and improving applications to better meet the needs of users and the public. SE is now emerging to stand proudly on its own, beside computer science and traditional engineering. See the list of software engineering topics for more details. See Important publications in software engineering for academic achievements in software engineering.