Guidelines for performing Systematic Literature review

DISCLAIMER : This information is not my own. I am just summarizing the "

Guidelines for performing Systematic Literature Reviews in Software Engineering by Barbara Kitchenham, Stuart Charters".

Purpose of this page

The information presented here are some basic guidelines for a systematic literature review.

Target Audience

It is aimed primarily at the researchers in software engineering community.

What is Systematic Literature Review?

-> Means of evaluating all the available research relevant to a particular research question.

Importance of Systematic Literature Review:

1. It identifies gaps in the current research and suggests further areas of investigation.

2. It provides reference upon which new research topics can be build upon.

Characteristic of a good systematic review:

1. It should be thorough and fair.

Researchers performing a systematic review should make sure that they report research that does not support their research hypothesis.

Important components of systematic literature review:

1. Systematic review starts by defining a review protocol that specifies the research question being addressed and the method that will be used.

2. Search strategies need to be developed to cover as much information as possible.

3. Search strategies need to be documented.

Nature of Software Engineering Research

Software engineers are human beings so it is more similar in experimentation to social sciences.

The Review Process

Systematic review has three phases:

1) Planning the review

2) Conducting the review

3) Reporting the review

These three stages may be in order or may follow an iteration.

The stages associated with planning the review are:

1. Identification of the need for a review

2. Commissioning a review.

3. Specify a research question

4. Develop a review protocol

5. Evaluate the review protocol

The stages associated with conducting the review are:

1. Identification of the research

2. Selection of primary study

3. Study quality assessment

4. Data extraction and monitoring

5. Data Synthesis

The stages associated with reporting a review are:

1. Specifying dissemination mechanisms

2. Formatting the main report

3. Evaluating the report

The Planning Stage

Why plan?

Planning is necessary to confirm the need of such a review.

The most important activities are::

1. Identify the questions that the review will intend to answers.

2. Producing a review process(plan) defining the basic review process.

Formulating Research Questions

- The search process must identify primary studies that address the research questions

- The data extraction process must extract the data items needed to answer the question.

- The data analysis process must synthesize the data in such a way that the questions can be answered.

A Right Research Question could be :

1. The one that identifies the discrepancies between commonly held beliefs and reality.

2. Is meaningful and important to both practitioners and researchers

3. Will lead to changed in the current practice or increase confidence in the value of current practice.

Review Protocols

Purpose of Review Protocols

Identify methods that can be used to understand a systematic literature review.

Reasons for having review protocols::

--> To prevent bias

A pre-defined protocol is needed to reduce the possibility of researcher's bias.

Otherwise, it is possible that the selection of studies may be driven by researcher's expectations.

Common components of a protocol:

1. Background. The rationale for the survey

2. The research question that the review is intending to answer

3. Strategies used for primary study

4. Study selection criterion

5. Study selection procedures

6. Study quality assessment. Checklist and procedures.

7.Data extraction stratergies

8. Synthesis of extracted data

9. Dissemination stratergy

10. Project time table

Evaluation of Review Protocol

PhD students should present their protocol to their supervisor for review and criticism.

Conducting a Review

Identification of Research

In systematic review, the research process should be rigorous. The researcher should find as many primary studies relating to the research question as possible using unbiased research stratergy.

Generating a search strategy

- Preliminary studies aimed at both identification and systematic review

- Trial searches using various combinations

- Consultation with expert in the field.

- Journals, reference list from review articles, internet can prove to be important sources of evidence

Avoiding Publication Bias

Publication bias means that positive results are more likely to be published than negative results. The concept of positive and negative results sometimes depend upon the viewpoint of the researcher.

Common scenarios of Publication bias

- Sometimes there is publication bias because failure to reject null hypothesis is considered less interesting than a experiment that is able to reject null hypothesis.

- Publication bias occurs when an influential company/group wants to support a particular method/technique.

Searches must be documented and a Bibliography must be maintained

Search process documentation should be fairly detailed so that the reader can go back.

Researchers should specify their rationale for :

- The digital library to be searched

- The journal of conference proceedings to be searched

Study Selection Process

Step 1: selection criteria should be interpreted liberally, so that unless a study identified by the electronic and hand searches can be clearly excluded based on title and abstract, a full copy should be obtained

Step 2: The next step is to apply inclusion/exclusion criterion based on:

- Journal , languages, author, etc.

Step 3: The third step in study selection could be detailed quality criterion.

It is a good idea to keep a list of excluded studies and reasons for exclusion.

Data Extraction

The objective of data extraction is to design data extraction forms to accurately record the information researchers obtained from the primary studies.

Design for Data Extraction forms

The data extraction forms must be designed to collect all the information needed to address review questions and the study quality criterion.

Contents of data collection form

In addition to the question needed to answer the review question and quality evaluation criterion, data collection forms should provide the following information including:-

- Name of the reviewer

- Date of data extraction

- Title, authors, journals

Reporting a Review

Specifying a Dissemination Strategy

The dissemination strategy , should be planned during the dissemination stage ( if any) preparing for systematic review protocol.

Formatting the review report

Systematic review can be reported in two formats:

- In a technical report

- In a journal style conference paper

Where to publish?

"Journal of Information and Software Technology" is a good place to publish literature review.