Requirements engineering is the process of eliciting stakeholder needs and desires and developing them into an agreed-upon set of detailed requirements that can serve as a basis for all subsequent development activities. The purpose of requirements engineering methodologies is to make the problem that is being stated clear and complete, and to ensure that the solution is correct, reasonable, and effective.(1)
'Stakeholder needs' seem strange. Sure, we have stakeholders but in the end we only have to make sure that we satisfy the needs of our users. (users are a type of stakeholder). 'Clear and complete … to ensure that the solution is correct …' is another interesting choice of words; I would be reluctant to claim that my requirements are clear and for sure, never as complete. How can we ensure correct? We are not solving a mathematical equation but are designing a solution for a customer who does not yet know what they want, in a technology we don't yet understand in an environment which is changing(2). Regardless of how much you polish your crystal ball, there is always volatility, uncertainty, complexity and ambiguity. Welcome to the VUCA(3) world.
The outcome of RE for software products is documentation. What is the intention of this documentation? It's about codifying knowledge by persisting information. You can read them months or years later and still get same meaning. Well, too bad that writing down and reading is a very lossy process. So, the meaning gets often twisted.
So why do we do it? It's historical. In the past we worked in sequence driven by activities: Planning, Analysis, Design, Implementation, Testing, Release. The first steps '... serve as a basis for all subsequent development activities ... ', they are all about eliciting needs, phrasing them into requirements and then cast them into specifications which were then handed over to the programmers in the IT department (creating and reinforcing silos). The underlying believe is, that we can know everything upfront. Once the project was completed the complaining and finger pointing started. Product discovery and product delivery are clearly separated.
Agile to the rescue (check out the principles in the Agile Manifesto(4))! Instead of being driven by activities, where we plan everything, analyse everything, design everything, …. (interesting that we do all the planning in the beginning even though we know the least - but that is another post) with Scrum we are driven by maximising value for our users. Combining product discovery and product delivery. We slice vertically, we do all of the above activities within one Sprint. Identifying most valuable features (ie. risk, business impact, ...) with analysis, design, implementation, testing all the way to potentially releasable. Instead of creating documentation we create a real working software product which is demoed at the Sprint Review to our stakeholder as well as users, creating real transparency. The important benefit about this approach is creating a shared mental model by having enough conversations with all parties involved: Product Owner, Developers (covering all required skills like: programming, testing, business analysis, ux, …) as well as users if needed. The information we create is often written into User Stories (in case you don’t know, the definition of a User Story: It is a promise for a conversation to take place(5), it is about the talking, less the writing). The conversation creates the cognitive alignment which allows us to pull together into the same direction. Ron Jeffries put it perfect with his Card, Conversation, Confirmation (3Cs) model. (6)
In short: We have conversations with the right people, our findings are written down on a card (or in a ticket; including acceptance tests). Confirmation makes sure it works correctly and is accepted by the users (verification and validation combined). A full vertical slice of product discovery and delivery, just in time - Lean RE.
If you are about to say, 'We need documentation, because …', don't worry, I've good news. For this Scrum has the Definition of Done (DoD). A DoD on top of making sure non that functional requirement (NFR) are omnipresent and good engineering practices are ensured, could also require that a requirements document, specification document, architecture document, test document, operations document, ... is created and kept up to date each Sprint in parallel with the usable product. What's even better, as the documentation is created after the work has been completed, the persisted knowledge is correct and not in need of rework (which usually never happens). This can be even applied in a regulated environment.
It's all about Vision, Value and Validation. (3Vs)(7)
(1) https://www.sciencedirect.com/topics/computer-science/requirement-engineering
(2) Alistair Cockburn
(3) https://en.wikipedia.org/wiki/Volatility,_uncertainty,_complexity_and_ambiguity
(4) http://agilemanifesto.org/principles.html
(5) Alistair Cockburn
(6) https://ronjeffries.com/xprog/articles/expcardconversationconfirmation/
(7) Professional Product Owner, The: Leveraging Scrum as a Competitive Advantage