Joint ISERN-IBM CASCON Celebration Keynote

The Evolution of Empiricism in Software Engineering: A Personal Perspective

By: Victor R. Basili, Professor Emeritus, University of Maryland

Abstract:

Although most scientific and engineering disciplines view empiricism as a basic element of their discipline, that view has not been the tradition in software engineering. There has not been the symbiotic relationship between the development of theories and engineering concepts and empirical studies that test and evolve those theories and concepts.
This talk discusses the evolution of empiricism in the software engineering discipline since the early 70s with respect to the kinds of studies that were being performed, the set of methods being used, the nature of the publications, the community of empirical researchers, the status of replications and meta-analysis, and the role of context variables. It offers a personal, historical perspective of the evolution of empirical studies through a series of example applications that demonstrate the various roles that empiricism can play and what we have learned. The examples are taken from studies in which the speaker was involved. It offers a set of criticisms as to where we have fallen short, suggestions on what we need to do, and the barriers we face in achieving a true engineering discipline that can continue to evolve our knowledge and demonstrate the impact of the research.

Bio:

Dr. Victor R. Basili is Professor Emeritus at the University of Maryland, College Park. He holds a PH.D. in Computer Science from the University of Texas, Austin and is a recipient of honorary degrees from the University of Sannio, Italy (2004) and the University of Kaiserslautern, Germany (2005). He was Founding Director of the Fraunhofer Center for Experimental Software Engineering – Maryland and a founding director of the Software Engineering Laboratory (SEL) at NASA/GSFC. He has worked on measuring, evaluating, and improving the software development process and product using methods that include Iterative Enhancement, the Goal Question Metric Approach (GQM), the Quality Improvement Paradigm (QIP), the Experience Factory (EF), and the GQM+Strategies Approach for aligning organizational goals and strategies through measurement.
Dr. Basili is a recipient of several awards including several NASA Group Achievement Awards, the 2000 Outstanding Research Award from ACM SIGSOFT, the IEEE Computer Society 2003 Harlan Mills Award, and the Fraunhofer Medal. He has authored over 250 journal and refereed conference papers, has served as Editor-in-Chief of the IEEE TSE and the Springer Journal of Empirical Software Engineering. He is an IEEE and ACM Fellow.

Title: Design Science in Software Engineering

Co-Chairs: Per Runeson, Markku Oivo, and Barbara Russo

Abstract: Design science is an empirical research paradigm for conducting, describing and validating research, focusing on research contributions that involve both the understanding of a problem in a particular domain, while proposing a novel or improved solution to address that problem. Design science is under-utilized in software engineering research, and our goal is to understand why. Implicitly much of the research that is done does involve the creation of some artifact to solve a human problem, and thus we assume that the design science paradigm would be a possible frame of reference for the research.

This session aims at creating a basic understanding of what the design science paradigm is, and how it can be applied to SE to help to assess and to communicate research contributions. Further, we aim to discuss what characterizes the research contributions in design science.

Session Goals:

• build a basic understanding of what design science in SE means
• understand how it relates to the concepts of empirical software engineering
• work with conceptual models and checklists to assess the quality of design science contributions

Development of the Session:

• 30-minute introductory lecture
• 40 minutes of group (3-5 people) activity on assessing design science contributions
• 20 minutes plenary discussions from the group work

Background and recommended reading:

A. R. Hevner, S. T. March, J. Park, and S. Ram. Design science in information systems research. MIS Q., 28(1):75–105, Mar. 2004.

R. Wieringa and A. Morali. Technical Action Research as a Validation Method in Information Systems Design Science, pages 220–238. Springer Berlin Heidelberg, Berlin, Heidelberg, 2012.

Expected Outcomes and Plan for Continuing the Work beyond ISERN:

Outcomes:

• feedback on our preliminary work
• input from other related work, which we are not aware of

Beyond ISERN:

• assist researchers in framing their research according to the design science paradigm, and reviewers to assess such research properly.

Title: Preregistration of Software Engineering Studies

Co-Chairs: Stefan Wagner and Martin Solari

Abstract: Preregistration of study protocols and research plans is a way to make research more transparent and repeatable. Several disciplines, such as psychology, have made strong progress in that area. Several journals allow preregistering a study. It means that the study plan or protocol is reviewed before the study is executed. If the protocol is accepted, the full article can then be reviewed faster later on, and such articles often receive something like a „Pre-registered Badge“ to advertise this about the article. Software engineering has so far not tried this means to improve the scientific process. This session will discuss the benefits and drawbacks of the approach and aims at identifying ideas how to implement pre registration in our field.

Session Goals:

Our goal is to establish if pre registration could be useful for software engineering studies. If so, we would like to explore possibilities for its implementation in suitable journals or conferences.

Development of the Session:

• Short talk on pre registration of studies in other disciplines
• Depending on number of participants: plenary or group discussions on pros and cons of pre registration in our field
• Collection and summary of pros and cons
• Group discussion: Development of concrete ideas of how to implement pre registration in SE
• Presentation of ideas and summary

Background and recommended reading:
10.17605/OSF.IO/2DXU5
https://www.youtube.com/watch?v=SWkqdNppL-s

Expected Outcomes and Plan for Continuing the Work beyond ISERN:

Depending on the ideas we develop during the session, we might decide to approach editorial boards or steering committees.

Title: Naming the Pain in Requirements Engineering (NaPiRE): Globally distributed, yearly replicated family of surveys

Co-Chairs: Daniel Méndez, Stefan Wagner, Marcos Kalinowski, Michael Felderer, and Rodrigo Fonseca

Abstract: During last year’s ISERN meetings, we conducted thematic workshops to plan the first globally family of distributed surveys on industrial requirements engineering (RE) practices and problems, called NaPiRE (Naming the Pain in Requirements Engineering). Out of this work, hosted now under the umbrella of ISERN, a project emerged in which, so far, over 50 researchers from 25 countries are engaged. The overall goal is to define the first theory on RE practices and problems. As for today, we completed the data collection and analysis of the first successful replication and are starting the data collection of the next replication. Details about the project can be taken from http://www.re-survey.org

After giving a brief introduction into the current state of the overall project and welcoming new members, we will use the session to jointly

• Discuss in the group of participants the first results from the current data collection and plan for the joint data analysis initiating first steps in group work.
• Plan and introduce further organizational structure.
• Bring in new researchers in participating in future replications.

Given the complexity of the NaPiRE project, we plan to discuss this year current results, but also how to better organize the project with our globally distributed members. It includes, but is not limited to, the project architecture and the infrastructure used.

We will further use the open space event during day 1 to jointly discuss the current results from our surveys as a preparation for the workshop, but also introduce interested ISERN members not previously involved in the project. The outcome of the thematic workshop will be the first set of prepared results from the data analysis and a detailed project plan for the activities of the next year.

Reading Material: Members who already conducted the RE survey in this year are expected to participate. Members not previously involved should read the information provided on the website http://www.re-survey.org as well as the documents listed therein. In particular, we recommend the latest publication

D. Mendez Fernandez, S. Wagner, M. Kalinowski, M. Felderer, P. Mafra, A. Vetrò, T. Conte, M.-T. Christiansson, D. Greer, C. Lassenius, T. Männistö, M. Nayebi, M. Oivo, B. Penzenstadler, D. Pfahl, R. Prikladnicki, G. Ruhe, A. Schekelmann, S. Sen, R. Spinola, J.L. de la Vara, A. Tuzcu, R. Wieringa Naming the Pain in Requirements Engineering: Contemporary Problems, Causes, and Effects in Practice In: Empirical Software Engineering Journal, Springer, 2016 (preprint:https://arxiv.org/abs/1611.1028)

Title: A Factbook for Software Development and Sustainment

Chair: Forrest Shull

Abstract: In 2017, the Software Engineering Institute (SEI) published the US Department of Defense (DoD) Software Factbook, providing an analysis of the most extensive collection of software engineering data owned and maintained by the DoD. The SEI translated these data into information that is frequently sought-after across the DoD. Basic facts are provided about software projects, such as averages, ranges, and heuristics for size, effort, and productivity. Factual, quantitatively derived statements provide easily digestible and usable benchmarks for different system types.

The SEI is searching for collaborators who are interested in replicating these or similar analyses for different domains, especially for safety-critical or defense-related systems in other countries. In the short term, collaborators and we would find value in a comparison of benchmarks or heuristics across countries, industries, or domains. In the long run, we can envision a more general Factbook, jointly published by multiple ISERN members, which would contain data on interesting issues that can help to characterize the more general state of the software industry.

Session Goals:

• Identify potential collaborators and the industries / countries from which they can obtain data
• Refine list of hypotheses / heuristics of interest
• Define path forward

Development of the Session: (How do you expect to organize the session)

• Presentation of the main results from the DoD Software Factbook, and the important factors for which quantitative impact has been assessed; Discussion and feedback on methodology
• Discussion: What similar efforts, current or historical, have been undertaken by the community already
• Discussion: What other benchmarking efforts would be feasible / Who are the interested parties
• Discussion: Brainstorming a publication goal and path forward

Background and recommended reading:

• The US DoD Software Factbook is cleared for public release and will be made available for download within the next 2 weeks; the URL will be published at that time

Expected Outcomes and Plan for Continuing the Work beyond ISERN:

• Identification of collaborators
• Compilation of milestones and synchronization points
• Discussion of data sources and data collection options
• Strategy for meta-analyses and other means of combining results from multiple studies
• Joint publication strategy

Title: Conflicts and Synergies among Quality Requirements

Co-Chairs: Barry Boehm, Xavier Franch, Andreas Jedlitschka and Marcus Ciolkowski (as guest)

Abstract: Quality requirements (QRs; also known as Non-Functional Requirements) have often received little attention compared to functional requirements, even though empirical studies suggest otherwise (e.g., studies with software architects). It has been argued that neglecting QRs is one of the top ten risks of requirements engineering. One of the key issues on QRs is the impossibility to understand their effects in an isolated manner. Synergies and conflicts among QRs are known from long ago both in general and concerning particular types of QRs.

Session Goals:

The purpose of this session is to foster discussions on the elicitation, representation, analysis and knowledge reuse concerning synergies and conflicts among QRs.

Development of the Session: (How do you expect to organize the session)

After a concise introduction, there are leading questions that will be discussed with the audience:

• What type of relationships does exist between QRs?
• How can be these relationships modeled?
• Is it possible to use project data to learn about the relationships among QRs?

Background and recommended reading:

• L. Guzmán, M. Oriol, P. Rodríguez, X. Franch, A. Jedlitschka, M. Oivo. “How Can Quality Awareness Support Rapid Software Development? - A Research Preview”. REFSQ 2017.
• P. Behnamghader, R. Alfayez, K. Srisopha, B. Boehm. “Towards Better Understanding of Software Quality Evolution Through Commit-Impact Analysis”. QRS 2017.
• S. Wagner. Software Product Quality Control. Springer, 2013.

Expected Outcomes and Plan for Continuing the Work beyond ISERN:
During the session, we will try to classify the input from the audience.
After the workshop, the proposers and interested co-workers might collaborate on a shared vision paper.

Title: Investigating Model Inspection with Crowdsourcing

Co-chairs: Stefan Biffl, Marcos Kalinowski, Dietmar Winkler

Abstract:

In (model-based) software engineering, the quality of models representing database designs, software architecture, or mission critical test cases is crucial for the quality of the software artifacts that are (directly) derived from models. Therefore, the early verification of these models is a key capability for software quality assurance. While there are automated model verification approaches (e.g., model checkers), the verification of the correct representation of the reference document content (such as requirement scenarios) in a model requires human expertise that is hard to automate. Software inspection builds on the human expertise in a group of inspectors to detect defects in software artifacts early in the software life cycle and differently from automated approaches. In software model inspection, the

inspectors check whether a conceptual model correctly and completely represents the content of a reference document, such as a requirements specification. Important model types include UML models and Extended Entity Relationship (EER) diagrams for designing software system structure and behavior. A recent push for the need of model quality assurance also in heterogeneous multi-disciplinary environments has come from the Industrie 4.0 initiative in

Germany and from related initiatives on advanced manufacturing based on the Internet of Things in Europe, the North America, and China (Winkler et al., 2017d).

In 2016 we conducted an ISERN thematic session, in which we discussed challenges and potential collaborations for providing systematic method and tool support for teams of inspectors to assess and improve model quality of real and large-scale models.

Results of the 2016 ISERN thematic session were (1) to develop a concept for controlled experiments to allow testing hypotheses on model inspection methods and tool support within the ISERN community and (2) to develop and discuss a protocol of a systematic mapping study on model quality assurance:

• Concept for controlled experiments. In the meantime, we developed an approach for software inspection combined with crowdsourcing tool support (called Crowdsourcing-based Software Inspection – CSI) (Winkler et al., 2017a) conducted several experiment runs to collect data on the feasibility of the approach (Winkler et al., 2017b-c). While the initial results are promising, there are many context variants (e.g., different types and sizes of reference documents and models) and the topic requires further experimental investigations from the community. In the session proposed for ISERN 2017, we want to discuss the experiment concept, benefits and limitations found with the initial approach, and collaboration options with ISERN members.
• Protocol of a systematic mapping study. We developed an initial protocol that we want to share and discuss with interested ISERN members as the foundation for a joint study based on the outcome of the discussion.

Hence, for the ISERN 2017 thematic session, we plan to present the current challenges, explain how the CSI approach intends to address these challenges, report on our recent controlled experiment results from Austria and Brazil, and present experiment variants (e.g., Crowdsourcing tool support, model scoping approaches, reference document types, reference documents, model types, models) to serve as a basis for the discussion with participants to set up further collaborations on the topic.

Session goals:

• Inform the ISERN community on the session topic, including challenges on model inspection, existing work in the ISERN community, and benefits from collaborations in and beyond ISERN.
• Present capabilities from Crowdsourcing (and the CSI approach) as a solution option addressing some of the model inspection challenges.
• Identification of additional alternative solutions, e.g., already available solutions we can build on based on the experience of the participants.
• Present results of the initial experimental studies conducted in Austria and Brazil.
• Present experiment variants to serve as a basis for the discussion with participants to set up further collaborations on the topic.
• Roadmap draft for empirical research in the session context, including concrete next steps and persons willing to lead these.

Development of the session:

• Introduction to the session topic and existing related ISERN collaborations and partners.
• Elicitation of discussion topic candidates relevant for the group regarding the session goals.
• Discussion in 2-3 working groups (max five persons per topic)
• Identification of researchers to carry key topics until the next ISERN meeting.
• Summary of work group inputs to report back to ISERN plenum.

Background and recommended reading:

Background:

• LaToza, T.D., van der Hoek, A. (2016). Crowdsourcing in software engineering: models, motivations, and challenges. IEEE Software, 33(1), pp. 74-80, 2016.
• Mohagheghi, P., Dehlen, V., & Neple, T. (2009). Definitions and approaches to model quality in model-based software development–A review of the literature. Information and Software Technology, 51(12), 1646-1669.
• Rech, J. (Ed.). (2008). Model-Driven Software Development: Integrating Quality Assurance. IGI Global.

Selected related publications by ISERN members

• Genero, M., Fernández-Saez, A. M., Nelson, H. J., Poels, G., & Piattini, M. (2011). Research review: a systematic literature review on the quality of UML models. Journal of Database Management (JDM), 22(3), 46-70.
• Grünbacher, P., Halling, M., Biffl, S., Boehm, B. W. (2004). Integrating collaborative processes and quality assurance techniques: experiences from requirements negotiation. Journal of Management Information Systems, 20(4), 10-30.
• Lange, C. F., & Chaudron, M. R. (2005, September). Managing model quality in UML-based software development. In Software Technology and Engineering Practice, 2005. 13th IEEE International Workshop on (pp. 7-16). IEEE.
• Nelson, H. J., Poels, G., Genero, M., & Piattini, M. (2012). A conceptual modeling quality framework. Software Quality Journal, 20(1), 201-228.
• Nugroho, A., & Chaudron, M. R. (2009). Evaluating the impact of UML modeling on software quality: An industrial case study. In Model driven engineering languages and systems (pp. 181-195). Springer Berlin Heidelberg.
• Pastor, O., & Molina, J. C. (2007). Model-driven architecture in practice: a software production environment based on conceptual modeling. Springer Science & Business Media.
• Travassos, G., Shull, F., Fredericks, M., Basili, V. (1999). Detecting Defects in Object Oriented Designs: Using Reading Techniques to Increase Software Quality, 14th ACM SIGPLAN Conference on Object-oriented programming, systems, languages, and applications (OOPSLA), pp 47-56.
• Winkler D., Sabou M., Petrovic S., Biffl S., Kalinowski M., Carneiro G.: "Improving Model Inspection with Crowdsourcing", In: Proceedings of the 4th International Workshop on Crowdsourcing in Software Engineering, ACM/IEEE International Conference on Software Engineering (ICSE), Buenos Aires, Argentinia, 2017a.
• Winkler D., Sabou M., Petrovic S., Carneiro G., Kalinowski M., Biffl S.: “Investigating Model Quality Assurance with a Distributed and Scalable Review Process”, In: Proceedings of the 20th Ibero-American Conference on Software Engineering, Experimental Software Engineering (ESELAW) Track, Springer, Buenos Aires, Argentina, 2017b.
• Winkler D., Sabou M., Petrovic S., Carneiro G., Kalinowski M., Biffl S.: “Improving Model Inspection Processes with Crowdsourcing: Findings from a Controlled Experiment”, In: Proceedings of the 24th EuroSPI Conference on Systems Software and Service Process Improvement, Ostrava, Czech Republic, September 5-8, 2017c.
• Winkler, D., Wimmer, M., Berardinelli, L., & Biffl, S. (2017d). Towards Model Quality Assurance for Multi-Disciplinary Engineering. In Multi-Disciplinary Engineering for Cyber-Physical Production Systems (pp. 433-457). Springer International Publishing.

Expected outcomes and plan for continuing the work beyond ISERN:

• Set up joint experiments to collect additional evidence on model quality assurance until the next ISERN meeting.
• Set up a systematic mapping study with ISERN collaborators.
• See also the session goals.