A faulty literature review can derail a dissertation.
A researcher first needs to understand the literature in the field.
According to Gall, Borg, and Gall (1996) cited in this article : completion of an acceptable dissertation literature review will TAKE BETWEEN THREE AND SIX MONTHS OF EFFORT!!!!!!!!!!!!!!!!!
(As my Scottish Gran used to say "Nay good crying, yee'll just haf to vork harrrder mee girle!"")
Purposes for Writing a Literature Review
1. To demonstrate the Researcher's knowledge about a particular field of study
(vocabulary, theories, key variables and phenomena, methods and history)
2. Getting informed of the influential researchers and research groups in the field
3. The outcome should be "a publishable, scholarly document"
4. Delimits the research problem
5. Seek new lines of enquiry
6. Gain methodological insights
7. Identify recommendations for future research
8. AVOID FRUITLESS SEARCHES
9. Distinguish what has been done and what needs to be done
10. Discover important variables relevant to the topic
11. Synthesize and gain new perspective
12. Identify relationships between the ideas and practises
13. Establish the context of the topic/problem
14 . Acquire the vocabulary of the subject
15. Identify the main methodologies and research techniques in the area
16. Place the research in the historical context and be familiar with the latest developments.
17. Develop the framework to relate new findings to previous findings in the discussion section of the dissertation. In order to be able to discuss how new research advances the previous.
Taxonomy
Cooper’s 5 characteristics of a Literature Review
1. Focus
2. Goal
3. Perspective
4. Coverage
5. Organization
6. Audience
1. Focus :
a. Research Outcomes :
In this case the Literature Review focuses on information analysis and synthesis of the findings, summarizes the substance and draws conclusions. This may help identify where there is information lacking in the area and establishes the justifiable need for a study.
b. Methodological Reviews :
Focus on research methods in the field. Identifies key variables, measures, and methods of analysis. Identifies methodological strengths and weakness, how research practises differ between groups, times, settings, identifies ways in which the method informs the outcome. If shows previous research has been methodologically flawed could justify proposed dissertation.
c. Theory Reviews :
Establish what theories already exists, relationships between them and the degree to which they have been investigated. Required if the dissertation is to propose a new theory. Can establish a lack of, or insufficiencies in existing theories to present rationale for dissertation.
d. Practises / Application Reviews :
Review concentrates on how interventions have been applied or how certain groups carry out certain practises. Dissertation may highlight a practical need not being met.
A Literature Review has a primary purpose but may need to address some or all of the foci. Example given is where an outcomes-orientated review deals with the theories related to the phenomenon and introduces the practical application that will be gained from the dissertation.
2. Goal :
To integrate and generalise the views to present “the map of the research landscape”
a. Generalization of findings across groups, treatments, outcomes, settings
b. Highlight and resolve debates within a filed
c. Bridge language used across a field
Goal may be to critically analyse previous research.
Goal may be to identify central issues or explicate a lien of argument.
A review typically has multiple goals. If it is simply a review could still need to critically analyse, identify central issues and explicate an argument. If the review needs to justify further research the goal should be to critically analyse the literature in order to identify weaknesses and to propose that further research could remedy that weakness.
3. Perspective :
In quantitative reviews, the author typically follows the quantitative tradition and attempts to take a neutral view and reveal only the facts.
In qualitative reviews, the author reveals their own pre-existing views (biases), discusses how these may have affected/influenced the review
4. Coverage :
How wide to cast the net is a critical decision in the review process.
a. Exhaustive
Author proposes to locate and consider all pieces of research on the topic, publish or unpublished. Need to define the population in a bounded manner and ensure volume of articles is manageable. This type of coverage is very time consuming
b. Exhaustive review with selective citation
The author looks at all articles but only from a particular source egg. Journals. The theoretical reason for this must be given.
c. Representative sample
A representative sample of articles is considered and inferences about the population are made. Must show how the sample is actually representative.
d. Purposive sample
Only central/pivotal articles in the field are considered. The author must be convincing as to why these articles are pivotal and the others not
5. Organisation :
a. Historical format
The review gives a chronological order. Used when emphasis is on progression of methods and theories or the changes over time
b. Conceptual format
The review is organised according to the proposition is a research rationale or the theories under review
c. Methodologically
For example in an empirical paper : Introduction, Method, Results, and Discussion.
Meta-analytical reports could mix and match : Introduction, Define the method, present results historically or conceptually, Discuss results
6. Audience :
In the case of a dissertation the supervisor and reviewers are the primary audience. Other scholars may be the secondary audience. Dissertation Literature reviews should not be written for non-academic audiences.
HOW TO CONDUCT A LITERATURE REVIEW
1. Formulate Problem
2. Collect Data
3. Evaluate Data
4. Analyse and Interpret Data
5. Present
These stages mirror the stages in primary research.
The key components of the plan are:
a) a rationale for conducting the review – choosing the appropriate TYPE of Review (Based on Cooper’s Taxonomy)
b) research questions/hypotheses to guide the research
c) explicit plan for collecting data (how articles will be chosen)
d) explicit plan for analysing data
e) plan for presenting data
Components of a Literature review are Articles.
Validity and Reliability issues also apply and the process may be iterative and not follow the order exactly. (No kidding!?)
a) Identify appropriate type of review using Coopers Taxonomy.
b) Problem formation : research questions/hypotheses to guide the research
Decide the question the Literature will answer and determine explicit criteria to determine the inclusion/exclusion of articles. This question guides the literature review.
A Literature Review Question can be answered by reviewing secondary research – but empirical research questions can only be answered through primary research.
The Literature Review will be the primary source of the Empirical Research Question and is influenced by the Goal and Focus of the Review.
(My problem with this is when one has established a Review Question at the outset but through conducting a review of the literature one comes to the conclusion that : it is not an issue, has been dealt with, is a small component of a bigger issue etc. Quite a reiterative step to start again from the beginning at the end of the process!!!!!!!!)
The article does advise that the first step should be to look for reviews that may have already answered the question or questions related but from practical experience these may not be easy to find in the beginning of one’s research as it can take a long time (and many .pdf downloads) before one gets into the landscape and gets to know the major players in the field and what they have produced in terms of research/articles etc)
c) Criteria of Inclusion/Exclusion and Data Collection: explicit plan for collecting data (how articles will be chosen)
Determine the criteria for inclusion and exclusion of articles.
This will be influenced by the reviews Focus, Goals and Coverage.
Criteria should be explicit enough to be able to include or exclude an article based only on these criteria.
It should be reliable (another researcher would include/exclude the same articles based on it)
“This may require considerable trial and error pilot testing” but important to do at the onset before a lot of resources have been expended on collection and analysis.
The goal of the actual Data Collection is to collect the relevant articles both with regards content, number and relevancy, and often begins with electronic search of academic databases.
It is advised that THE DATA COLLECTION PROCEDURE SHOULD BE DESCRIBED IN SUCH DETAIL THAT OTHER RESEARCHERS (same procedure, same conditions) WOULD FIND IDENTICAL SET OF DOCUMENTS.
Information to be kept:
Date of each search, database searched, key words and combinations used, number of records resulting from each search.
Randolph has found that only 10% of articles are found on keyword searches.
The other 90% are found by going through the references in the initial articles, retrieving these references, reading their references, repeating this until no new articles come to light. (Did this intuitively but saturation point has not been met Due to time limitations am just going to have to stop – frustrating).
At this stage Randolph suggests sharing the final list with experts in the field to determine if they detect any missing articles as well as supervisors and reviewers. I contacted Michael Jacobson in a similar manner, this has opened up my eyes to players in the field such as Dede, but also very importantly has changed the focus of my actual research question. This is where the time factor comes in – expecting this process to be completed in just a few weeks does not seem reasonable.
The data collection process stops when the point of saturation has been reached and there is “sufficient evidence to convince” the audience “that everything that can reasonably be done to identify all relevant articles has been diligently undertaken”. If new articles come to light they must only be included if they are critically important.
The next stage in Data Collection is to “cull” the articles.
Separate the potentially relevant from obviously irrelevant.
This may be done by reading them all, only the abstract, only the title or a combination.
Once again this process must be FINELY DOCUMENTED.
From the relevant articles another process of refinement is conducted to arrive at the final subset of articles to be included.
If reliability is critical other reviewers must be consulted as to the process of refinement.
Once final subset of articles have been gathered. Data evaluation stage begins.
d) Data Evaluation and Analysis: explicit plan for analysing data
In this stage the reviewer begins to extract and evaluate the information from the articles that meet with the Inclusion Criteria.
A system must be devised to extract this data based on the Focus and the Goal of the review
DOCUMENT types of data extracted and process used.
This documentation can use a separate coding form and coding book, which is included in the dissertation appendices (or main body). A coding book is an electronic document that documents the type of data to be extracted, process to do so and actual data.
The level of details should be that another researcher would arrive at more or less the same extractions.
This process could reveal other types of data that should be extracted and may result in the coding book being revised and previously reviewed articles being re-reviewed.
Now the researcher attempts to make sense of the extracted data.
e) Presentation plan for presenting data
“Literature reviews are commonly organised historically, conceptually or methodologically”
Formulating and justifying the empirical Research Question/s:
Explain, using evidence, how the dissertation makes a meaningful contribution to knowledge in the field.
It could
- -contribute to an established line of theory and empirical research. In which case make clear what these contributions are and how the study contributes to testing, elaborating or enriching the theoretical perspective.
- -establish a new line of theory. Make clear what the new theory is, how it relates to existing theory, why the new theory is needed and intended scope of its application
- -be motivated my practical concerns. Make clear what these concerns are, why they are important and how the research can address these concerns.
- - be motivated by a lack of information about a problem or issue. Make clear what information is lacking, why it is important and how the research will address the need for information.
Quantitative Literature Reviews
Narrative and Meta-Analytical Reviews
Quantitative Reviews
Emphasised better-designed studies, organised to form a composite picture of state of the knowledge on the topic of the review, each study may be described separately in a few sentences or a paragraph.
Tend to be affected by reviewer’s perspective and thus the same articles could result in different conclusions by different researchers.
(Is this a bad thing? – There is no such thing as absolute truth).
Meta-Analytical Reviews
Reviewer – collects representative or comprehensive sample of articles – codes the articles according to aspects egg quality, type of intervention, outcomes – finds common metric that allows synthesizing – examines how characteristics of the study covary with outcomes of the study.
Qualitative Literature Reviews
Ogawa and Malen’s Method, Randolph’s Method (not included Noblit and Hare’s Method)
Ogawa and Malen’s Method
Step 1 : Create an audit trail – document all steps taken to make clear evidence that supports the finding, where evidence can be found and how evidence was interpreted
Step 2 : Define focus of the review – problem formation, define constructs of the review to determine what is to be included and what not
Step 3: Search for relevant literature, in addition to high qualitative research reports include also non-research artefacts such as memos, newspaper articles etc
Step 4: Classify the documents – according to the types of data they represent
Step 5: Create summary databases – similar to data evaluation stage. Develop coding schemes to reduce information in the relevant documents. Cant read all documents, take casual notes and write a review. Need to develop narrative summaries and coding schemes to take into account all pertinent information. Reiterative process.
Step 6: Identify constructs and hypothesized causal linkages – from summary databases, identify essential themes and create hypotheses about relationships between the themes. Goal is to increase understanding of what is being investigated, not to integrate outcomes and identify factors that covary with outcomes.
Step 7: Search for contrary findings and rival interpretations.
Step 8: Use colleagues or informants to corroborate findings, to confirm the degree to which the review’s conclusions are sound.
Randolph’s Method – The phenomenological method for conducting a qualitative literature review.
To arrive at the essence of the lived experience of a phenomenon Moustakas as quoted in article.
Step 1: Bracketing – identify the phenomenon to be investigated. Researcher brackets their experience with the phenomenon to be investigated by explaining their own experiences with and positions on the phenomenon.
Step 2: Collecting data – collect data about the phenomenon. Primary phenomenological research interviews a set of people who have experienced the phenomenon. Then read the reports of scientists who have done research on the phenomenon using the same criteria for inclusion and definition of the research strategy.
Step 3: Identify meaningful statements – highlight empirical claims about the phenomenon, collect them.
Step 4: Giving meaning – identify meaningful statements and give meaning to those statements maybe putting meaningful statements into categories and paraphrasing them as groups
Step 5: Thick rich description – create such a description of the essence of primary researcher’s experience with the phenomenon as seen through the eyes of the researcher.
MISTAKES COMMONLY MADE IN REVIEWING RESEARCH LITERATURE
1. doesn’t clearly relate findings to the researcher’s own study
2. doesn’t take sufficient time to define best descriptors, best sources
3. relies on secondary sources rather than primary resources
4. uncritically accepts other’s findings and interpretations as valid instead of critically examining all aspects of the research and design
5. does not report (and document) search procedures
6. reports isolated statistical results
7. does not consider contrary findings and alternative interpretations
Meta-Evaluation ones own Literature Review
According to Bootes and Beile’s 5 category rubric for evaluating a literary review:
Coverage, Synthesis, Methodology, Significance, Rhetoric.
