Principles of Requirements Engineering or Requirements Manag

by Adam Sandman on Friday, November 8, 2024

Abstract

Requirements Engineering (RE) is often trivialized as an activity performed by well-meaning analysts before they start doing the real work of specifying a product. In this paper we shall introduce, at the highest levels, the critical processes and procedures used when executing effective Requirements Engineering as part of an overall successful project.

This is a concise look into a discipline which, when explored in full, can and does, fill many books. We shall also highlight how software tools can be leveraged to achieve RE goals that might otherwise be challenging. This paper can be used as a starting point for further research and learning into the many aspects of full Requirements Engineering.

Is Requirements Engineering That Important?

A CIO magazine study concluded, “Analysts report that as many as 71 percent of software projects that fail do so because of poor requirements management” The unfortunate truth is that managers of many of those projects probably believed that they were doing good requirements management.

Although it is rarely stated so simply, the vast majority of software and systems project failures are due in large part to insufficient importance being placed on Requirements Engineering. In September 1999, NASA's $125 million Mars Climate Orbiter probe was destroyed when it tried to enter orbit 100 kilometers too close to Mars. The now notorious error was due to incompatible specifications; the attitude-control system was specified using imperial units but its navigation software used metric units. It cannot be stressed enough that getting requirements right and using them to the fullest extent throughout the project is critical to success. When reviewing reports of project failure, factors such as 'Inadequate understanding of user needs', 'Unclear scope' and 'Changing requirements' can be frequently found, all of which are core aspects of RE or Requirements Management (RM) as it is sometimes called.

A Google search for 'requirements management' results in almost 350 million hits, a testament to the extensive coverage the discipline is now receiving and the work still to be done in providing education in this area.

Even as projects increasingly adopt Agile processes, requirements management remains key. Agile projects may encourage the definition of user needs and wants in different ways than traditional processes, with user stories for example. But it is critical that stories accurately reflect the needs of the stakeholders and allocation of stories to tasks is still performed. Essentially, the stories are the requirements in a Scrum methodology and full access to them is essential to ensure that all team members are working closely together towards a common goal, which is what Agile is all about. Agile projects also promote the fleshing out of requirements as the project progresses rather than all at the beginning, but this does not affect the principles we are addressing here.

Definitions

When it comes to the requirements domain there are plenty of terms scattered throughout the literature. Labels such as requirements management, use-cases, needs, scenarios, stakeholders, constraints and scope are common and each is useful in its own way. For the purpose of this paper we shall define just three key terms: Requirement, Requirements Engineering and Requirements Management.

In some attempts to clarify these terms, definitions get bogged down in the quality of the requirements, i.e. what makes a good requirement? But this is not part of the core definition of a requirement itself; a bad requirement is still a requirement in the same way that a car is still a car even when it breaks down frequently or falls apart. We shall address the quality of requirements later. For now, we can view a requirement as 'any statement of desire or need'. This may seem unhelpfully vague, but it is essentially so. If, for example, the definition were to be more specific by including such things as the source of the requirement, or a capability of a software or systems product, or even the nature of the requirement, then we lose the capacity for requirements to come from many sources or to specify a corporate objective or to be either textual or visual or both. These considerations provide insight into the wide scope and great importance of requirements.

Requirements Engineering and Requirements Management are often used synonymously. However, for our purposes this is unhelpful and so we shall define RE as

the group of activities, both manual and automated, that involve the discovery, manipulation, transformation, analysis and reporting of requirements,

and we shall define RM as

the subset of tasks within RE that involve only the organization and administration of the requirements information, often using a formal requirements management software tool.

What a Requirement is Not

It is sometimes helpful to see requirements as addressing the problem, and not the solution, which emphasizes that a requirement is not an aspect of design nor a suggestion as to how to implement a solution. A requirement should say what we want solved, not how to solve it. Of course, while this definition should be taken seriously, it is not a hard and fast rule; it is sometimes useful to know what solution the user envisions as part of their requirements, and design considerations often produce new or revised requirements.

Further, projects often have constraints that dictate aspects of the solution, such as ensuring compliance with standards or limiting the use of certain technologies. But in stating constraints, we merely put fences around the solution, we do not provide the solution itself.

Requirements Gathering and Elicitation

Requirements do not present themselves fully formed and ready to go. They are often hidden in the subconscious of the customer, trying to be heard amongst the noise of the users' preconceptions, their previous habits, possible personal job dissatisfaction and desire to control the product being developed. Users have to be coaxed, motivated and sometimes manipulated into revealing their true needs rather than their pet solutions. Focus groups, one-on-one discussions, process reviews, observation of existing users at work, problem reports from existing products and questions asked by prospective users in sales situations; all these are useful requirements gathering methods that should be combined as conditions allow to provide a well-rounded understanding of needs.

Many of these requirements elicitation techniques take place in the workplace of the stakeholder or some third party location for group meetings. Gathering this vital information is vital if a proper analysis is to be performed and wherever possible, it is best to enter the information directly into your requirements management tool of choice as the conversation is being had. The software can then help with the requirements analysis and create links between derived user stories or scenarios, and the original words that came straight from the users’ mouth.

While developing bespoke software it can often be useful to find a product champion: someone who can speak for the user community. When working on products with a broader user base, forums such as premier customer meetings that allow disparate users to discuss and sometime prioritize requirements amongst themselves, can help provide a consensus across the user base.

When most people think of requirements the first thought that comes to mind is: what do my customers want? But recognizing that there are different types of users as well as different types of requirements, is an important step toward better requirement engineering as well as more successful product management.

Types of Requirement

Product Managers must appreciate that while end-user requirements are critical, it is also important to listen to the views and needs of other stakeholders. What about the engineers who have to support the product? What about any formal training your organization may do with the product and what about the needs of the sales people? Furthermore, the users are not usually the same people as the customers; someone else is paying for the product and they probably have their opinions too. These are all stakeholders whose needs should be considered.

As mentioned previously, constraints are another type of requirement. Corporate policies, prohibited technologies, regulatory compliance, target platforms and system performance are all examples of requirements that must be added to stakeholder requirements to provide a base set of needs to work from.

These are all considered top-level requirements because they do not derive from other requirements and they must all be considered when initially defining the scope of the project. This is true whether the project uses a traditional waterfall approach or something more agile, where scope may be decided for one iteration, or sprint, at a time.

Requirements Expression

Requirements definitions are more effective when they are not forced into one format. Considering the best way to express each requirement will result in ideas that are easier to understand.

Formal Requirements

Some formal requirements may need to be specified in written language, often called 'shall' statements, e.g. "The ticket dispenser shall dispense a maximum of 10 tickets per transaction." Requirements written this way are sometimes contractually required and may even need to be combined into documents with tightly regulated formats. This is common with government contracts but is far less prevalent in the commercial sector. If you do need to do this, consider carefully the pros and cons of keeping the requirements constantly in a document format versus creating a new version of the document from the requirements as needed.

One of the downsides of 'shall' statements is that they provide all the detail but little sense of the big picture, so collectively, they can take a long time to understand, write, review and make ready for design work. This overhead has led to the impression with some managers that RE is an expensive and time-consuming activity that holds up implementation. Unfortunately, in strict waterfall-type projects, there is some truth to this, and managers need to keep a cool head and trust that time spent at the front end of the project will pay off at the end. For complex, expensive, or life-critical projects, it invariably is.

Use Cases

Where such formal statements are unnecessary, there are other ways to express requirements. Use cases offer a method that provides the details as well as an overall understanding.

Figure 1: A simple UML use case diagram

There is no fixed definition of what can and cannot be in a use case, but as the name suggests, use cases offer small textually described vignettes of user required functionality – like a script. Scenes with a beginning, middle and end, tell a short story of how one part of the system will function for the users. Use cases can also be layered, one containing any number of others with more detail and in this way the overall story can be built up. Because use cases are small examples of how the different part of the system work and interact with one another, they generally easier to review and understand than formal shall statements. As well as textual representations, use cases can also be represented as UML diagrams.

Figure 2a: A textual representation of a use case scenario

There are many diagrams and pictures that can be used to describe requirements, each of which offers a different perspective and so work better when combined, resulting in a more rounded understanding of the problem. Activity diagrams, process flows, state transition diagrams, entity-relationship diagrams and decision trees, are all examples of visual requirements representations that can help individuals, with different skills and different responsibilities on a project, to understand the requirements.

Figure 2b: A process flow graphical representation of a use case scenario

User Stories

User stories appear in most Agile methods and help to express the requirements in a scenario rather like a use case but in a simplified form based on “As a <actor> I want <something> so that <benefit>”:

Figure 3: A typical set of user stories

As with all requirements, it is important to understand the difference between the requirements as originally provided by the stakeholder and the requirements written from the stakeholders perspective. The former is a raw user statement and may contain bias, or an incomplete view of the system as users often state what the system should do and ignore their own part in the process. User stories should be a reflection of stakeholder needs after they have been analyzed collectively and a deeper understanding has been reached of the overall problem. They should include the parts played by both the user and system in the story. This is why stakeholder reviews of the user stories is encouraged; because it is important to ensure that the original intent was not lost in the derived requirements.

Supplementary Information

One of the reasons Word Processors are unsuitable as requirements management tools is that requirements usually have additional information or attributes, such as the source of the requirement, it's priority or it's applicability. Attributes are used to categorize and organize the requirements. Sorting, filtering, group editing and generally keeping track of this supplementary information should be one of the things any RM tool does well. Because of the natural tendency to display attributes in columns against the requirements, there is a temptation to use a spreadsheet to manage requirements, however, this comes with its own drawbacks, which will become evident when we discuss relationships.

Figure 4: Requirement Attributes in a Software RM Tool

Good and Bad Requirements

There are only two primary criteria for a good requirement:

It correctly expresses the need of the stakeholder, and

It is testable.

Naturally, other, lesser criteria can be derived from these. For example, to meet the two primary criteria, a requirement must be understandable and without ambiguity, it must not be overly complex and it must be a true need statement, not a solution statement, (as discussed previously.) A good requirement does not necessarily have to be feasible. User needs should not be dismissed just because it is initially thought that it can't be done; asking for the impossible is the first step toward making the impossible, possible. If necessary, such requirements can be set aside later, once it is determined that they are indeed impractical.

As well as good individual requirements, the quality of the requirement set overall should be considered, and from that it is possible to derive two further criteria. The requirements as a whole must be:

Consistent, and

Complete.

Reviews and Approvals

Reviews are essential to the development of good requirements. For example, looking at the two rules for a good requirement: the first, “It correctly expresses the need of the stakeholder,” requires that the stakeholders review the requirements in order to be sure that this rule has been followed and the second, “It is testable,” would be better pursued with test engineer reviews.

There is a delicate balance to be struck when building reviews into any process. Too few reviews and problems may not be spotted, too many reviews and the project flows like molasses. Using a requirements management tool can help by providing distributed, but controlled, access to the requirements as well as the capability to record reviewers' comments and approval states. The better tools offer help with individualized reviewing and thus help avoid the agonizing 'group review' meeting. The problem of slowed development due to meetings is well understood by agilists such that the majority of Agile processes include clearly defined meetings, usually few in number.

Requirements Decomposition and Relationships

Initial user or system requirements are typically unsuitable as raw input to design. At a minimum, they need to be translated into functional capabilities; a change in perspective from what the user wants to get to what we want the deliverable product to provide, without moving into the solution definition itself.

It is usually the case that requirements will become more detailed as they are decomposed into functional definitions resulting in one-to-many relationships between user and functional requirements. (I shall use the terms relationship and link synonymously for the rest of this paper.) These are sometimes called parent-child relationships. Occasionally, the opposite will happen and a number of user requirements can be consolidated into fewer functional requirements; a many-to-one relationship.

Figure 5a: Link Types

Not surprisingly, most requirements transformations result in a combination of one-to-many, many-to-one and straightforward one-to-one relationships. For reasons that will become clear, it is critical that these complex relationships are recorded and this is where a purpose-built RM tool really comes into its own. Commercial RM products offer a variety of ways to create relationships from the most basic placement of check marks in a 2-dimensional matrix, to easy-to-use drag-and-drop between requirements. Some even provide automatic link creation as a by-product of requirements creation.

Figure 5b: Parent…child relationships illustrated in an RM tool.

When used to their fullest potential, RM tools with support for links are an invaluable asset when it comes to not only software and systems implementation, but acceptance testing and even maintenance. Some tools, such as Inflectra's SpiraTest, combine requirements management and test management, along with the relationships between them, all within a single interface, making the benefits of traceability easier to realize.

Figure 6: Requirements linked to test cases

Taking relationships to the next level, they can be labeled in the same way as requirements, using attributes. They might be labeled based on the meaning of the link, for example: is this a parent-child relationship or requirements from separate sources that have a direct impact on one another? Link attributes can even be used to acknowledge the strength of that impact. Such information supports the analysis of the potential effect of proposed changes far more effectively than relying on unlabeled links. When combined with multiple levels of relationships, this is sometimes called “Rich traceability”.

Lifecycle Traceability & Analysis

While it is valuable to manage relationships between requirements of different types and levels, the benefits are significantly greater if we start linking to data from other phases of the lifecycle. The most widely exploited traceability is that between requirements and tests. Richard Stevens, the mastermind behind what is now IBM Rational DOORS said that “Quality is conformance with requirements,” and the best way to ensure that the deliverables conform to requirements is to use them as the basis of the tests. Ideally, the tests, whether manual or automatic, will be written directly from the requirements creating links in at the same time. Either way, the ultimate the goal is to have every requirement linked to at least one test, even if that test is one of mere observation. This is the best, and perhaps only way, to 'prove' that all requirements have been met.

Other benefits gained from links between requirements and tests include the ability for engineers to focus their efforts on the specific requirements that are not being met when tests fail, and to more effectively review tests by examining them in the context of the requirements they are meant to verify.

Relationships between requirements and tasks can also be useful. Individuals assigned tasks, whether they be for design, or testing, or any other project activity, have a better understanding of the task in hand if they can see the requirements driving that task and so they stand a greater chance of producing quality results.

Figure 7a: Requirements linked to tasks

Once a rich set of relationships has been created, they can be used to analyze the status of a project more effectively. An RM tool should provide the ability to quickly filter requirements that have no associated tests, or artifacts that have a large number of links that may indicate excessive reliance on critical points in the system and that therefore may require a review of the design.

Another relationship that helps to identify problem areas is that between requirements and defects or incident reports. Not only can this help with individual bug fixing but statistics of incidents and their links can, over time, provide a picture of areas that might be less stable. It is not unusual for 20% of the requirements to be related to 80% of the bugs. Finding those problem areas and focusing effort to stabilize the code and reduce risk is essential. It is worth noting that the requirements related to reported incidents may not necessarily be identified through direct defect-requirement relationships but may instead be found through chains of traceability, e.g. the defect is linked to an area of code or design which is in turn already linked back to the requirements.

Figure 7b: Requirements linked to bugs/defects

Agile projects benefit equally from traceability. Iterations may be shorter and take place faster, but it is still important to record relationships between stories and tasks, tasks and code, code and unit tests and stories and acceptance tests. Without these links there is so much we lose, including the impact of later changes, which brings us to change management.

Managing Change

It is common to think of change as being managed by CM (Configuration Management) tools. However, where requirements are concerned, the granularity of data items in CM tools is typically larger than we need. A simple change to the priority of a single requirement can have a big impact on a project and so ideally, as well as managing versions of full requirements, changes to individual attributes of requirements should also be controlled and tracked. The criticality of a product in development dictates the level of access controls required during the development project; not every project needs tight data protection and in an agile project particularly, the administration of detailed access controls will only slow the project down. In such situations, consider role-based controls instead.

Regardless of the level of control imposed, modifications must be recorded. If the priority of a requirement changes, it is valuable to know when it happened, why, and who made the change. As well as offering accountability, this makes it easier to reverse changes if necessary.

Figure 8a: Requirement Change History in a Software RM Tool

A good RM tool should be able to quickly show the knock-on effect of change both before and after a change is made. What will be the impact of a change both on work already completed and work yet to be done? Which artifacts need to be reviewed again after a change is made? This is sometimes called 'suspect links' but is more accurately termed 'suspect objects' as it is the linked items that must be reviewed, less so the links themselves.

Figure 8b: The test case suspect flag was triggered by a requirement change

Requirements level change management along with impact analysis can be beneficial if it becomes necessary to push back against proposed changes. It becomes far easier to demonstrate the reasons for any cost associated directly or indirect with the suggested change. This can save the customer a lot of money and the developers a lot of time. Agile projects are typically able to incorporate changes later in the project than traditional methods, and so understanding the impact of a change affects not only work completed but also work in progress. It is equally important to record changes to user stories in Agile projects for all the reasons we have discussed.

Change tracking at the requirements level can also identify instability. Examining any requirements (or associated data) that may have an unusually high change rate (sometimes called 'churn') can highlight volatility. High volatility when combined with a high level of direct traceability usually indicates an area of higher risk and therefore deserves attention.

A final thought about requirements change management: some sectors, such as those regulated by the U.S. Food and Drug Administration (FDA) (21 CFR Part 11), Aviation (DO-178B) and Finance (Sarbanes-Oxley), require tightly managed control of requirements change, to the extent that an electronic signature may be required for each and every change. While this is excessive for many projects, it is something to consider when implementing formal requirements engineering.

One word of warning: the introduction of formal requirements change control can sometimes create conflict with those responsible for formal configuration management within an organization. In order to create a cooperative relationship between RM and CM practitioners and to avoid duplicating effort, it is prudent to include those responsible for CM in the planning and introduction of any new RM practices.

Team Participation

Successful lifecycle traceability is dependent upon the willing participation of team members across that lifecycle. Whether the cycle is long, as in waterfall-style projects, or short as in agile projects, the more people involved, the more effective is the requirements engineering and traceability in particular. An important factor in getting team members on board is to give easy access to the tools. Choose tools that support distributed access, especially if a project involves contractors or remotely located team members. In these days of instant communication, it is inefficient to wait for responses from other team members, especially in Agile projects where delays can be very costly. When team members are not co-located, instant messaging can help bring them closer together and reduce delays, but be wary of losing vital information in those messages and look for quick and easy ways to capture it.

Some tools provide an alternative approach to distributed teams: the ability to export data for off-line use and then return it with changes. Such functionality is most commonly found in more sophisticated tools and can be complicated to use and not entirely effective.

Requirements Management in Maintenance

There is the temptation to put the requirements into mothballs once the original development project is officially over. But during maintenance, there will almost certainly be the need for defect resolution and the release of updates. RM can play a vital role in identifying, especially for new team members, why the system works the way it does and the potential impact of any proposed 'fixes' or changes, and so prevent 'the fix that breaks something else'!

It is important to allocate effort during maintenance for requirements upkeep. If the requirements are not updated as changes are made and problems resolved, the advantages requirements engineering provides will diminish and eventually it could become part of the problem, not the solution.

Requirements Reuse

Although the details are beyond the scope of this paper, we should mention requirements reuse. Many projects have as their starting point, an existing product or some of its components. It would be time consuming to rewrite the requirements where they are the same or similar for the new project. Instead, effort put into the analysis of the existing requirements can save time and money by identifying not only requirements that can be reused but, through traceability, also design elements, code and even tests that can be applied to the new project.

The simplest approach to reuse is to create a copy of all relevant data for use in the new project. However, difficulties arise when trying to take fixes made to the original product, and roll them into the new one. Another means of requirements reuse is to employ attributes to denote the applicability of requirements to either the existing or the new variant, or both. Problems can occur here when elements that are identical but enter different phases of development or maintenance and so are still the same, but not in the same release cycle. Both these techniques are solutions to parallel development but neither is well supported by RM tools.

The practice of managing multiple product variants is often called 'Product Family Management' and is one of the most problematic areas for organizations looking to become more efficient through increased reuse, and careful planning is required to implement it successfully.

Implementing Requirements Engineering

Introducing formal requirements engineering is not as simple as making the decision and procuring a software tool. As with any change, achieving acceptance can be the biggest battle. It can start a power struggle over who 'owns' the requirements and create higher walls between project members with potential 'not invented here' attitudes. New tools become shelf-ware and silos of differing project activities become stronger, the opposite of what is sought. Formal RE requires a culture change that is more easily achieved by involving a variety of project members from the very start. It is important to ensure that management is on-board and enlist the help of those team members who influence others. Making the implementation of a new process a team effort will increase the likelihood of success and may even yield ideas nobody had previously thought of.

Perhaps it goes without saying that an RM tool should be quick and easy to use, but this is even more apparent in Agile projects where there is the need to maintain a fast pace and not slow things down, which would then defeat the purpose.

Six Typical Requirements Engineering Problems

Common mistakes and traps to avoid:

Scope creep – The introduction of new, 'little' requirements here and there because they seem 'easy' or to make the customer happy. Not only do they soon add up, there is an impact on all the project activities including test, documentation, training, etc.

Gold plating/Over engineering – Doing more than is required to solve a problem or adding code when someone thinks, 'While I'm doing this I may as well do....'. This can put the actual requirements at risk. Like scope creep, it creates a lot more work. A random number generating algorithm doesn't have to incorporate the latest thinking of mathematics Nobel laureates.

Shutting out users – While it is unhelpful to have the users constantly scrutinizing the team's work, it is worse to keep them out. The earlier that user feedback is received, the less costly it is to incorporate.

Unidentified stakeholders – Is is essential that all stakeholders have been identified and consulted. Look for those who are unusually or unexpectedly impacted by the product(s) being built. For example, a paper-saving system may affect the job of an unseen archivist who works in the basement.

Requirements Engineering as an administrative function – When there is resistance to change, the solution often employed is to allow some team members to work as before and have 'administrators' enter information into an RM tool. Not only does this introduce a superfluous overhead, it creates opportunities for human error and denies the project the true benefits of process improvement.

Assuming a tool will solve the problems – Throwing a tool at the problem is like giving a car to a pizza delivery person who cannot drive. It is important to provide adequate training, not only in the tool but also in the process improvements designed to take advantage of the new product.

Summary

The importance of good requirements engineering cannot be overstated. The requirements tell everyone on the project exactly what to do, including the project manager who plans and schedules tasks based on the requirements. It is only through traceability of acceptance tests back to the requirements that proof exists of the correctness of what is delivered. Testing without reference to requirements can only show that the product does something, not whether that something is what is wanted.

It should be no surprise that such a critical aspect of a project can lead to great disasters when it fails. It is only by understanding the requirements engineering process that we can hope to master it and use it to tease out success from development processes that are peppered with traps and pitfalls.