literature review marking criteria

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• How to Write a Literature Review | Guide, Examples, & Templates

How to Write a Literature Review | Guide, Examples, & Templates

Published on January 2, 2023 by Shona McCombes . Revised on September 11, 2023.

What is a literature review? A literature review is a survey of scholarly sources on a specific topic. It provides an overview of current knowledge, allowing you to identify relevant theories, methods, and gaps in the existing research that you can later apply to your paper, thesis, or dissertation topic .

There are five key steps to writing a literature review:

• Search for relevant literature
• Evaluate sources
• Identify themes, debates, and gaps
• Outline the structure
• Write your literature review

A good literature review doesn’t just summarize sources—it analyzes, synthesizes , and critically evaluates to give a clear picture of the state of knowledge on the subject.

Instantly correct all language mistakes in your text

Upload your document to correct all your mistakes in minutes

Table of contents

What is the purpose of a literature review, examples of literature reviews, step 1 – search for relevant literature, step 2 – evaluate and select sources, step 3 – identify themes, debates, and gaps, step 4 – outline your literature review’s structure, step 5 – write your literature review, free lecture slides, other interesting articles, frequently asked questions, introduction.

• Quick Run-through
• Step 1 & 2

When you write a thesis , dissertation , or research paper , you will likely have to conduct a literature review to situate your research within existing knowledge. The literature review gives you a chance to:

• Demonstrate your familiarity with the topic and its scholarly context
• Develop a theoretical framework and methodology for your research
• Position your work in relation to other researchers and theorists
• Show how your research addresses a gap or contributes to a debate
• Evaluate the current state of research and demonstrate your knowledge of the scholarly debates around your topic.

Writing literature reviews is a particularly important skill if you want to apply for graduate school or pursue a career in research. We’ve written a step-by-step guide that you can follow below.

Here's why students love Scribbr's proofreading services

Discover proofreading & editing

Writing literature reviews can be quite challenging! A good starting point could be to look at some examples, depending on what kind of literature review you’d like to write.

• Example literature review #1: “Why Do People Migrate? A Review of the Theoretical Literature” ( Theoretical literature review about the development of economic migration theory from the 1950s to today.)
• Example literature review #2: “Literature review as a research methodology: An overview and guidelines” ( Methodological literature review about interdisciplinary knowledge acquisition and production.)
• Example literature review #3: “The Use of Technology in English Language Learning: A Literature Review” ( Thematic literature review about the effects of technology on language acquisition.)
• Example literature review #4: “Learners’ Listening Comprehension Difficulties in English Language Learning: A Literature Review” ( Chronological literature review about how the concept of listening skills has changed over time.)

You can also check out our templates with literature review examples and sample outlines at the links below.

Download Word doc Download Google doc

Before you begin searching for literature, you need a clearly defined topic .

If you are writing the literature review section of a dissertation or research paper, you will search for literature related to your research problem and questions .

Make a list of keywords

Start by creating a list of keywords related to your research question. Include each of the key concepts or variables you’re interested in, and list any synonyms and related terms. You can add to this list as you discover new keywords in the process of your literature search.

• Social media, Facebook, Instagram, Twitter, Snapchat, TikTok
• Body image, self-perception, self-esteem, mental health
• Generation Z, teenagers, adolescents, youth

Search for relevant sources

Use your keywords to begin searching for sources. Some useful databases to search for journals and articles include:

• Your university’s library catalogue
• Google Scholar
• Project Muse (humanities and social sciences)
• Medline (life sciences and biomedicine)
• EconLit (economics)
• Inspec (physics, engineering and computer science)

You can also use boolean operators to help narrow down your search.

Make sure to read the abstract to find out whether an article is relevant to your question. When you find a useful book or article, you can check the bibliography to find other relevant sources.

You likely won’t be able to read absolutely everything that has been written on your topic, so it will be necessary to evaluate which sources are most relevant to your research question.

For each publication, ask yourself:

• What question or problem is the author addressing?
• What are the key concepts and how are they defined?
• What are the key theories, models, and methods?
• Does the research use established frameworks or take an innovative approach?
• What are the results and conclusions of the study?
• How does the publication relate to other literature in the field? Does it confirm, add to, or challenge established knowledge?
• What are the strengths and weaknesses of the research?

Make sure the sources you use are credible , and make sure you read any landmark studies and major theories in your field of research.

You can use our template to summarize and evaluate sources you’re thinking about using. Click on either button below to download.

Take notes and cite your sources

As you read, you should also begin the writing process. Take notes that you can later incorporate into the text of your literature review.

It is important to keep track of your sources with citations to avoid plagiarism . It can be helpful to make an annotated bibliography , where you compile full citation information and write a paragraph of summary and analysis for each source. This helps you remember what you read and saves time later in the process.

Receive feedback on language, structure, and formatting

Professional editors proofread and edit your paper by focusing on:

• Academic style
• Vague sentences
• Style consistency

See an example

To begin organizing your literature review’s argument and structure, be sure you understand the connections and relationships between the sources you’ve read. Based on your reading and notes, you can look for:

• Trends and patterns (in theory, method or results): do certain approaches become more or less popular over time?
• Themes: what questions or concepts recur across the literature?
• Debates, conflicts and contradictions: where do sources disagree?
• Pivotal publications: are there any influential theories or studies that changed the direction of the field?
• Gaps: what is missing from the literature? Are there weaknesses that need to be addressed?

This step will help you work out the structure of your literature review and (if applicable) show how your own research will contribute to existing knowledge.

• Most research has focused on young women.
• There is an increasing interest in the visual aspects of social media.
• But there is still a lack of robust research on highly visual platforms like Instagram and Snapchat—this is a gap that you could address in your own research.

There are various approaches to organizing the body of a literature review. Depending on the length of your literature review, you can combine several of these strategies (for example, your overall structure might be thematic, but each theme is discussed chronologically).

Chronological

The simplest approach is to trace the development of the topic over time. However, if you choose this strategy, be careful to avoid simply listing and summarizing sources in order.

Try to analyze patterns, turning points and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred.

If you have found some recurring central themes, you can organize your literature review into subsections that address different aspects of the topic.

For example, if you are reviewing literature about inequalities in migrant health outcomes, key themes might include healthcare policy, language barriers, cultural attitudes, legal status, and economic access.

Methodological

If you draw your sources from different disciplines or fields that use a variety of research methods , you might want to compare the results and conclusions that emerge from different approaches. For example:

• Look at what results have emerged in qualitative versus quantitative research
• Discuss how the topic has been approached by empirical versus theoretical scholarship
• Divide the literature into sociological, historical, and cultural sources

Theoretical

A literature review is often the foundation for a theoretical framework . You can use it to discuss various theories, models, and definitions of key concepts.

You might argue for the relevance of a specific theoretical approach, or combine various theoretical concepts to create a framework for your research.

Like any other academic text , your literature review should have an introduction , a main body, and a conclusion . What you include in each depends on the objective of your literature review.

The introduction should clearly establish the focus and purpose of the literature review.

Depending on the length of your literature review, you might want to divide the body into subsections. You can use a subheading for each theme, time period, or methodological approach.

As you write, you can follow these tips:

• Summarize and synthesize: give an overview of the main points of each source and combine them into a coherent whole
• Analyze and interpret: don’t just paraphrase other researchers — add your own interpretations where possible, discussing the significance of findings in relation to the literature as a whole
• Critically evaluate: mention the strengths and weaknesses of your sources
• Write in well-structured paragraphs: use transition words and topic sentences to draw connections, comparisons and contrasts

In the conclusion, you should summarize the key findings you have taken from the literature and emphasize their significance.

When you’ve finished writing and revising your literature review, don’t forget to proofread thoroughly before submitting. Not a language expert? Check out Scribbr’s professional proofreading services !

This article has been adapted into lecture slides that you can use to teach your students about writing a literature review.

Scribbr slides are free to use, customize, and distribute for educational purposes.

Open Google Slides Download PowerPoint

If you want to know more about the research process , methodology , research bias , or statistics , make sure to check out some of our other articles with explanations and examples.

• Sampling methods
• Simple random sampling
• Stratified sampling
• Cluster sampling
• Likert scales
• Reproducibility

 Statistics

• Null hypothesis
• Statistical power
• Probability distribution
• Effect size
• Poisson distribution

Research bias

• Optimism bias
• Cognitive bias
• Implicit bias
• Hawthorne effect
• Anchoring bias
• Explicit bias

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.

There are several reasons to conduct a literature review at the beginning of a research project:

• To familiarize yourself with the current state of knowledge on your topic
• To ensure that you’re not just repeating what others have already done
• To identify gaps in knowledge and unresolved problems that your research can address
• To develop your theoretical framework and methodology
• To provide an overview of the key findings and debates on the topic

Writing the literature review shows your reader how your work relates to existing research and what new insights it will contribute.

The literature review usually comes near the beginning of your thesis or dissertation . After the introduction , it grounds your research in a scholarly field and leads directly to your theoretical framework or methodology .

A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other  academic texts , with an introduction , a main body, and a conclusion .

An  annotated bibliography is a list of  source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a  paper .  

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

McCombes, S. (2023, September 11). How to Write a Literature Review | Guide, Examples, & Templates. Scribbr. Retrieved April 2, 2024, from https://www.scribbr.com/dissertation/literature-review/

Is this article helpful?

Shona McCombes

Other students also liked, what is a theoretical framework | guide to organizing, what is a research methodology | steps & tips, how to write a research proposal | examples & templates, what is your plagiarism score.

How To Structure Your Literature Review

3 options to help structure your chapter.

By: Amy Rommelspacher (PhD) | Reviewer: Dr Eunice Rautenbach | November 2020 (Updated May 2023)

Writing the literature review chapter can seem pretty daunting when you’re piecing together your dissertation or thesis. As  we’ve discussed before , a good literature review needs to achieve a few very important objectives – it should:

• Demonstrate your knowledge of the research topic
• Identify the gaps in the literature and show how your research links to these
• Provide the foundation for your conceptual framework (if you have one)
• Inform your own  methodology and research design

To achieve this, your literature review needs a well-thought-out structure . Get the structure of your literature review chapter wrong and you’ll struggle to achieve these objectives. Don’t worry though – in this post, we’ll look at how to structure your literature review for maximum impact (and marks!).

But wait – is this the right time?

Deciding on the structure of your literature review should come towards the end of the literature review process – after you have collected and digested the literature, but before you start writing the chapter. 

In other words, you need to first develop a rich understanding of the literature before you even attempt to map out a structure. There’s no use trying to develop a structure before you’ve fully wrapped your head around the existing research.

Equally importantly, you need to have a structure in place before you start writing , or your literature review will most likely end up a rambling, disjointed mess. 

Importantly, don’t feel that once you’ve defined a structure you can’t iterate on it. It’s perfectly natural to adjust as you engage in the writing process. As we’ve discussed before , writing is a way of developing your thinking, so it’s quite common for your thinking to change – and therefore, for your chapter structure to change – as you write. 

Need a helping hand?

Like any other chapter in your thesis or dissertation, your literature review needs to have a clear, logical structure. At a minimum, it should have three essential components – an  introduction , a  body   and a  conclusion . 

Let’s take a closer look at each of these.

1: The Introduction Section

Just like any good introduction, the introduction section of your literature review should introduce the purpose and layout (organisation) of the chapter. In other words, your introduction needs to give the reader a taste of what’s to come, and how you’re going to lay that out. Essentially, you should provide the reader with a high-level roadmap of your chapter to give them a taste of the journey that lies ahead.

Here’s an example of the layout visualised in a literature review introduction:

Your introduction should also outline your topic (including any tricky terminology or jargon) and provide an explanation of the scope of your literature review – in other words, what you  will   and  won’t   be covering (the delimitations ). This helps ringfence your review and achieve a clear focus . The clearer and narrower your focus, the deeper you can dive into the topic (which is typically where the magic lies). 

Depending on the nature of your project, you could also present your stance or point of view at this stage. In other words, after grappling with the literature you’ll have an opinion about what the trends and concerns are in the field as well as what’s lacking. The introduction section can then present these ideas so that it is clear to examiners that you’re aware of how your research connects with existing knowledge .

2: The Body Section

The body of your literature review is the centre of your work. This is where you’ll present, analyse, evaluate and synthesise the existing research. In other words, this is where you’re going to earn (or lose) the most marks. Therefore, it’s important to carefully think about how you will organise your discussion to present it in a clear way. 

The body of your literature review should do just as the description of this chapter suggests. It should “review” the literature – in other words, identify, analyse, and synthesise it. So, when thinking about structuring your literature review, you need to think about which structural approach will provide the best “review” for your specific type of research and objectives (we’ll get to this shortly).

There are (broadly speaking)  three options  for organising your literature review.

Option 1: Chronological (according to date)

Organising the literature chronologically is one of the simplest ways to structure your literature review. You start with what was published first and work your way through the literature until you reach the work published most recently. Pretty straightforward.

The benefit of this option is that it makes it easy to discuss the developments and debates in the field as they emerged over time. Organising your literature chronologically also allows you to highlight how specific articles or pieces of work might have changed the course of the field – in other words, which research has had the most impact . Therefore, this approach is very useful when your research is aimed at understanding how the topic has unfolded over time and is often used by scholars in the field of history. That said, this approach can be utilised by anyone that wants to explore change over time .

For example , if a student of politics is investigating how the understanding of democracy has evolved over time, they could use the chronological approach to provide a narrative that demonstrates how this understanding has changed through the ages.

Here are some questions you can ask yourself to help you structure your literature review chronologically.

• What is the earliest literature published relating to this topic?
• How has the field changed over time? Why?
• What are the most recent discoveries/theories?

In some ways, chronology plays a part whichever way you decide to structure your literature review, because you will always, to a certain extent, be analysing how the literature has developed. However, with the chronological approach, the emphasis is very firmly on how the discussion has evolved over time , as opposed to how all the literature links together (which we’ll discuss next ).

Option 2: Thematic (grouped by theme)

The thematic approach to structuring a literature review means organising your literature by theme or category – for example, by independent variables (i.e. factors that have an impact on a specific outcome).

As you’ve been collecting and synthesising literature , you’ll likely have started seeing some themes or patterns emerging. You can then use these themes or patterns as a structure for your body discussion. The thematic approach is the most common approach and is useful for structuring literature reviews in most fields.

For example, if you were researching which factors contributed towards people trusting an organisation, you might find themes such as consumers’ perceptions of an organisation’s competence, benevolence and integrity. Structuring your literature review thematically would mean structuring your literature review’s body section to discuss each of these themes, one section at a time.

Here are some questions to ask yourself when structuring your literature review by themes:

• Are there any patterns that have come to light in the literature?
• What are the central themes and categories used by the researchers?
• Do I have enough evidence of these themes?

PS – you can see an example of a thematically structured literature review in our literature review sample walkthrough video here.

Option 3: Methodological

The methodological option is a way of structuring your literature review by the research methodologies used . In other words, organising your discussion based on the angle from which each piece of research was approached – for example, qualitative , quantitative or mixed  methodologies.

Structuring your literature review by methodology can be useful if you are drawing research from a variety of disciplines and are critiquing different methodologies. The point of this approach is to question  how  existing research has been conducted, as opposed to  what  the conclusions and/or findings the research were.

For example, a sociologist might centre their research around critiquing specific fieldwork practices. Their literature review will then be a summary of the fieldwork methodologies used by different studies.

Here are some questions you can ask yourself when structuring your literature review according to methodology:

• Which methodologies have been utilised in this field?
• Which methodology is the most popular (and why)?
• What are the strengths and weaknesses of the various methodologies?
• How can the existing methodologies inform my own methodology?

3: The Conclusion Section

Once you’ve completed the body section of your literature review using one of the structural approaches we discussed above, you’ll need to “wrap up” your literature review and pull all the pieces together to set the direction for the rest of your dissertation or thesis.

The conclusion is where you’ll present the key findings of your literature review. In this section, you should emphasise the research that is especially important to your research questions and highlight the gaps that exist in the literature. Based on this, you need to make it clear what you will add to the literature – in other words, justify your own research by showing how it will help fill one or more of the gaps you just identified.

Last but not least, if it’s your intention to develop a conceptual framework for your dissertation or thesis, the conclusion section is a good place to present this.

Example: Thematically Structured Review

In the video below, we unpack a literature review chapter so that you can see an example of a thematically structure review in practice.

Let’s Recap

In this article, we’ve  discussed how to structure your literature review for maximum impact. Here’s a quick recap of what  you need to keep in mind when deciding on your literature review structure:

• Just like other chapters, your literature review needs a clear introduction , body and conclusion .
• The introduction section should provide an overview of what you will discuss in your literature review.
• The body section of your literature review can be organised by chronology , theme or methodology . The right structural approach depends on what you’re trying to achieve with your research.
• The conclusion section should draw together the key findings of your literature review and link them to your research questions.

If you’re ready to get started, be sure to download our free literature review template to fast-track your chapter outline.

Psst… there’s more!

This post is an extract from our bestselling Udemy Course, Literature Review Bootcamp . If you want to work smart, you don't want to miss this .

You Might Also Like:

27 Comments

Great work. This is exactly what I was looking for and helps a lot together with your previous post on literature review. One last thing is missing: a link to a great literature chapter of an journal article (maybe with comments of the different sections in this review chapter). Do you know any great literature review chapters?

I agree with you Marin… A great piece

I agree with Marin. This would be quite helpful if you annotate a nicely structured literature from previously published research articles.

Awesome article for my research.

I thank you immensely for this wonderful guide

It is indeed thought and supportive work for the futurist researcher and students

Very educative and good time to get guide. Thank you

Great work, very insightful. Thank you.

Thanks for this wonderful presentation. My question is that do I put all the variables into a single conceptual framework or each hypothesis will have it own conceptual framework?

Thank you very much, very helpful

This is very educative and precise . Thank you very much for dropping this kind of write up .

Pheeww, so damn helpful, thank you for this informative piece.

I’m doing a research project topic ; stool analysis for parasitic worm (enteric) worm, how do I structure it, thanks.

comprehensive explanation. Help us by pasting the URL of some good “literature review” for better understanding.

great piece. thanks for the awesome explanation. it is really worth sharing. I have a little question, if anyone can help me out, which of the options in the body of literature can be best fit if you are writing an architectural thesis that deals with design?

I am doing a research on nanofluids how can l structure it?

Beautifully clear.nThank you!

Lucid! Thankyou!

Brilliant work, well understood, many thanks

I like how this was so clear with simple language 😊😊 thank you so much 😊 for these information 😊

Insightful. I was struggling to come up with a sensible literature review but this has been really helpful. Thank you!

You have given thought-provoking information about the review of the literature.

Thank you. It has made my own research better and to impart your work to students I teach

I learnt a lot from this teaching. It’s a great piece.

I am doing research on EFL teacher motivation for his/her job. How Can I structure it? Is there any detailed template, additional to this?

You are so cool! I do not think I’ve read through something like this before. So nice to find somebody with some genuine thoughts on this issue. Seriously.. thank you for starting this up. This site is one thing that is required on the internet, someone with a little originality!

I’m asked to do conceptual, theoretical and empirical literature, and i just don’t know how to structure it

Submit a Comment Cancel reply

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

• Print Friendly

Purdue Online Writing Lab Purdue OWL® College of Liberal Arts

Writing a Literature Review

Welcome to the Purdue OWL

This page is brought to you by the OWL at Purdue University. When printing this page, you must include the entire legal notice.

Copyright ©1995-2018 by The Writing Lab & The OWL at Purdue and Purdue University. All rights reserved. This material may not be published, reproduced, broadcast, rewritten, or redistributed without permission. Use of this site constitutes acceptance of our terms and conditions of fair use.

A literature review is a document or section of a document that collects key sources on a topic and discusses those sources in conversation with each other (also called synthesis ). The lit review is an important genre in many disciplines, not just literature (i.e., the study of works of literature such as novels and plays). When we say “literature review” or refer to “the literature,” we are talking about the research ( scholarship ) in a given field. You will often see the terms “the research,” “the scholarship,” and “the literature” used mostly interchangeably.

Where, when, and why would I write a lit review?

There are a number of different situations where you might write a literature review, each with slightly different expectations; different disciplines, too, have field-specific expectations for what a literature review is and does. For instance, in the humanities, authors might include more overt argumentation and interpretation of source material in their literature reviews, whereas in the sciences, authors are more likely to report study designs and results in their literature reviews; these differences reflect these disciplines’ purposes and conventions in scholarship. You should always look at examples from your own discipline and talk to professors or mentors in your field to be sure you understand your discipline’s conventions, for literature reviews as well as for any other genre.

A literature review can be a part of a research paper or scholarly article, usually falling after the introduction and before the research methods sections. In these cases, the lit review just needs to cover scholarship that is important to the issue you are writing about; sometimes it will also cover key sources that informed your research methodology.

Lit reviews can also be standalone pieces, either as assignments in a class or as publications. In a class, a lit review may be assigned to help students familiarize themselves with a topic and with scholarship in their field, get an idea of the other researchers working on the topic they’re interested in, find gaps in existing research in order to propose new projects, and/or develop a theoretical framework and methodology for later research. As a publication, a lit review usually is meant to help make other scholars’ lives easier by collecting and summarizing, synthesizing, and analyzing existing research on a topic. This can be especially helpful for students or scholars getting into a new research area, or for directing an entire community of scholars toward questions that have not yet been answered.

What are the parts of a lit review?

Most lit reviews use a basic introduction-body-conclusion structure; if your lit review is part of a larger paper, the introduction and conclusion pieces may be just a few sentences while you focus most of your attention on the body. If your lit review is a standalone piece, the introduction and conclusion take up more space and give you a place to discuss your goals, research methods, and conclusions separately from where you discuss the literature itself.

Introduction:

• An introductory paragraph that explains what your working topic and thesis is
• A forecast of key topics or texts that will appear in the review
• Potentially, a description of how you found sources and how you analyzed them for inclusion and discussion in the review (more often found in published, standalone literature reviews than in lit review sections in an article or research paper)
• Summarize and synthesize: Give an overview of the main points of each source and combine them into a coherent whole
• Analyze and interpret: Don’t just paraphrase other researchers – add your own interpretations where possible, discussing the significance of findings in relation to the literature as a whole
• Critically Evaluate: Mention the strengths and weaknesses of your sources
• Write in well-structured paragraphs: Use transition words and topic sentence to draw connections, comparisons, and contrasts.

Conclusion:

• Summarize the key findings you have taken from the literature and emphasize their significance
• Connect it back to your primary research question

How should I organize my lit review?

Lit reviews can take many different organizational patterns depending on what you are trying to accomplish with the review. Here are some examples:

• Chronological : The simplest approach is to trace the development of the topic over time, which helps familiarize the audience with the topic (for instance if you are introducing something that is not commonly known in your field). If you choose this strategy, be careful to avoid simply listing and summarizing sources in order. Try to analyze the patterns, turning points, and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred (as mentioned previously, this may not be appropriate in your discipline — check with a teacher or mentor if you’re unsure).
• Thematic : If you have found some recurring central themes that you will continue working with throughout your piece, you can organize your literature review into subsections that address different aspects of the topic. For example, if you are reviewing literature about women and religion, key themes can include the role of women in churches and the religious attitude towards women.
• Qualitative versus quantitative research
• Empirical versus theoretical scholarship
• Divide the research by sociological, historical, or cultural sources
• Theoretical : In many humanities articles, the literature review is the foundation for the theoretical framework. You can use it to discuss various theories, models, and definitions of key concepts. You can argue for the relevance of a specific theoretical approach or combine various theorical concepts to create a framework for your research.

What are some strategies or tips I can use while writing my lit review?

Any lit review is only as good as the research it discusses; make sure your sources are well-chosen and your research is thorough. Don’t be afraid to do more research if you discover a new thread as you’re writing. More info on the research process is available in our "Conducting Research" resources .

As you’re doing your research, create an annotated bibliography ( see our page on the this type of document ). Much of the information used in an annotated bibliography can be used also in a literature review, so you’ll be not only partially drafting your lit review as you research, but also developing your sense of the larger conversation going on among scholars, professionals, and any other stakeholders in your topic.

Usually you will need to synthesize research rather than just summarizing it. This means drawing connections between sources to create a picture of the scholarly conversation on a topic over time. Many student writers struggle to synthesize because they feel they don’t have anything to add to the scholars they are citing; here are some strategies to help you:

• It often helps to remember that the point of these kinds of syntheses is to show your readers how you understand your research, to help them read the rest of your paper.
• Writing teachers often say synthesis is like hosting a dinner party: imagine all your sources are together in a room, discussing your topic. What are they saying to each other?
• Look at the in-text citations in each paragraph. Are you citing just one source for each paragraph? This usually indicates summary only. When you have multiple sources cited in a paragraph, you are more likely to be synthesizing them (not always, but often
• Read more about synthesis here.

The most interesting literature reviews are often written as arguments (again, as mentioned at the beginning of the page, this is discipline-specific and doesn’t work for all situations). Often, the literature review is where you can establish your research as filling a particular gap or as relevant in a particular way. You have some chance to do this in your introduction in an article, but the literature review section gives a more extended opportunity to establish the conversation in the way you would like your readers to see it. You can choose the intellectual lineage you would like to be part of and whose definitions matter most to your thinking (mostly humanities-specific, but this goes for sciences as well). In addressing these points, you argue for your place in the conversation, which tends to make the lit review more compelling than a simple reporting of other sources.

Library Guides

Literature reviews: start.

• Criticality

What is a Literature Review?

The literature review is an overview of the significant literature on a topic. The word literature (in 'literature review') broadly refers to the scholarly or scientific writing on a topic. It may include: scholarly articles, books, book chapters, conference papers, theses and other sources of information relevant to your particular area of research.  

It should not be purely descriptive: a literature review needs to include summary and critical evaluation. 

Overall, a literature review should do the following: 

Identify key debates in your field of study 

Present and critically evaluate previous research and ideas 

Highlight gaps in the research 

Types of Literature Reviews

There are four types of literature reviews:

1. A literature review as a standalone assessment

2. A literature review leading to your original research project (dissertation)

3. Annotated bibliographies

4. Critical reviews / Critical summaries

Type 1: Literature Review as Standalone Assessment

A literature review as a standalone assessment  provides an overview of a specific topic/area. This kind of review does not answer a specific research question, but addresses questions like: 

What are the different approaches / theories on...? 

What is the current thinking / state of knowledge about...? 

How did we come to be where we are now? 

Most of the information in this guide can help doing a literature review leading to your original research project (dissertation).

Type 2: Literature Review as Part of Research Project (Dissertation)

A literature review leading to your original research project (dissertation)  prepares the ground for your own study. It involves reading widely to help you to refine your topic and formulate your research question or hypothesis. The purpose of a literature review is to demonstrate familiarity with, and understanding of, research in your field of study before conducting a new investigation. One of your purposes is to establish that you are not repeating research that has already taken place. It will enable you to uncover existing research and to illuminate what is not yet known about your topic. 

Type 3: Annotated Bibliography

An annotated bibliography is a list of selected sources such as articles, books and reports. The list consists of correctly formatted bibliographical references (Cite Them Right Harvard referencing in used most departments at Westminster) as well as a concise description of each text. Some annotated bibliographies also require a critical evaluation of the source.

There are generally two types of annotated bibliography: 

In a  descriptive annotated bibliography, you provide a description of the main arguments of each text without evaluating them (about 100-200 words). This is useful for determining whether a source is relevant to a particular research question or topic. 

In a critical annotated bibliography, your summary of each text should also include a critical analysis that evaluates the strengths and weaknesses of the authors’ arguments (about 100-200 words). You should think about the implications of their work for your field and your research project. 

Tips! 

For all types of annotated bibliography, focus on demonstrating your understanding of the texts as succinctly as possible.  

Visit the Referencing Guide for information on the Cite Them Right Harvard referencing style and how to avoid plagiarism. 

Remember! 

Always consult your module leader's instructions before beginning your annotated bibliography. Their guidance should help you understand the purpose of the assessment, what types of texts to focus on and the marking criteria. The instructions may ask you to produce an introduction and a conclusion to sum up your findings, for instance. 

Type 4: Critical Reviews and Critical Summaries

A Critical review  usually focuses on a single text (e.g. a journal article) and critically examines the approach taken and conclusions reached. 

Usually short and concise - about 1000 words 

Secondary sources can be used to support your critique 

Should contain a brief introduction that establishes the purpose of your review, concisely summarises the text under review, and briefly states your general view of the text 

Should include a brief conclusion at the end 

Should include a reference list for any sources cited

Critique in a critical review

Being critical does not mean 'being negative'. Rather, it means testing each part of a study to see whether the best approach was taken. Each section of a journal article, for instance, should be scrutinised in relation to the purpose it serves. 

Introduction:

• Is the author asking the right question? In the context of the field of study, is the question relevant, useful, insightful?

Literature review

• Have they clearly identified the problem?
• Have they established that their research question has not yet been answered?
• Have they missed any significant pieces of research? 

Methodology

• Have they taken a sensible approach in answering their question? Was the study well-designed? Have they ruled out confounding factors that might produce a misleading result?
• How have they interpreted their data? Is there an alternative interpretation?
• Do their conclusions agree or disagree with prior research? Do they acknowledge and account for differing results?

Always check the marking criteria provided for all assignments to gain a better understanding of what your lecturer expects from your work.

Below are the basic steps to produce an effective literature review: 

Select texts for inclusion, based on relevance to your topic and significance within the field

Evaluate quality of studies to be included  

Plan the structure of your work 

Synthesise information (bring together studies that are similar or complementary)   

These steps are addressed in the tabs of this guide. 

• Next: Research >>
• Last Updated: Nov 18, 2023 10:56 PM
• URL: https://libguides.westminster.ac.uk/literature-reviews

CONNECT WITH US

Literature Review: Assess your Literature Review

• Video Tutorial
• Research Question
• Select Resources
• Search Strategy

Assess your Literature Review

• Sample Literature Reviews
• Resource List
• Quick Links
• Avoid Plagiarism
• Use the rubric below to evaluate the quality of your literature review.  If your instructor has provided you with a rubric, you should use the criteria listed in that course or assignment rubric to ensure that your paper will meet the expectations for the course. ( Download a copy of the rubric.)

Adapted from Education 690: Assessment Rubric/Criteria for Literature Review, retrieved September 29,2010 from http://edweb.sdsu.edu/courses/ed690dr/grading/literaturereviewrubrique.html and Boote, D.N. & Biele, P. (2005). Scholars before researchers: On the centrality of the dissertation literature review in research preparation. Educational Researcher. 34(6) p. 8.

• << Previous: PLAGIARISM
• Next: Sample Literature Reviews >>
• Last Updated: Mar 26, 2024 10:49 AM
• URL: https://research.auctr.edu/literaturereview

Literature reviews

A literature review is a research task that finds, evaluates and discusses information on a particular topic. You need to analyse multiple texts, and discuss key ideas that you find in the reading.

A literature review may also identify gaps for further research. It is not a process of summarising texts separately – that is done in an annotated bibliography .

A good literature review should:

• be well researched, planned and structured
• discuss a relevant issue, problem or practice that relates to your subject
• explain why your chosen topic is important
• identify key ideas or themes in the literature
• synthesise these findings into thematic paragraphs, using sufficient citations and correct referencing
• demonstrate your understanding of the topic and develop your knowledge and evidence-based practice.

Literature reviews differ across disciplines, so use your assessment instructions and marking criteria as your ultimate guide.

7 steps for writing a literature review

1. analyse the task.

Read the assessment instructions and marking rubric carefully. Note how many sources you need to include in your literature review and any guidelines on selecting the literature.

Choose a topic that interests you. A topic that is important to you will help you stay focused.

2. Establish a clear research question

A good research question will help you narrow down your topic so that it is manageable. If your question is too broad, you may be overwhelmed by the reading. If it is too narrow, you won’t find enough literature.

3. Search for relevant literature

Use keywords from your research question to begin searching .

In the research question "What factors impact student learning when on nursing placement?", the keywords are: student, learning, nursing and placement.

4. Read and review the literature, and take notes

When reading the literature, consider the following:

• Is the article relevant to your topic? For journal articles, read the abstract. For other sources, scan quickly and discard if you can’t see anything useful.
• Who is the author/s and what is their expertise?
• Is the source credible and scholarly ? Use any evaluation tools provided by your subject.
• What is the main topic and what themes are discussed?

Pay attention to important information, such as the abstract, introduction, headings/subheadings, graphs/tables and conclusion.

Take notes using the  Cornell method  or a note-taking grid. Keeping notes that help you remember the content and relevance of each source is vital for writing a literature review.

• Example note-taking grid for a literature review [Word - 14KB]

5. Identify common themes or areas for further research

It's important to understand the relationships between the sources you've read. Look out for:

• themes:  what questions, ideas or topics recur across the literature? Where do authors agree or disagree?
• areas for further research:  what is missing from the literature? Are there any weaknesses?

6. Plan the structure of the literature review

Before you start writing, plan how your literature review will be organised.

Literature reviews are usually organised thematically, meaning they discuss one theme after another. You can also organise your ideas chronologically (from past to present) or by methodology (e.g. comparing findings from qualitative and quantitative research).

7. Write, edit, proofread, submit

It's easy to get lost in the reading and not leave enough time for polishing your writing. Use the Assessment Planner to make a clear study plan that includes time for writing, editing and proofreading.

Structure of a literature review

Introduction.

The introduction should include:

• context or background : give a brief summary of the context for your research question and explain why it is important
• purpose (thesis statement): state the purpose of the literature review. This is a statement generated from the research question
• scope (roadmap):  outline the specific themes the literature review will focus on and give the reader a sense of how your writing is organised.

Each body paragraph focuses on a specific theme and draws on several pieces of literature. 

Paragraphs should include:

• topic sentence:  start with the theme of the paragraph
• synthesis of evidence:  make connections between multiple sources by comparing and contrasting their views. Use summaries, paraphrases and quotes , and don't forget to properly reference your sources
• analysis or evaluation: add your own interpretation of the findings and comment on any strengths, weaknesses, gaps or areas for further research in the literature
• link:  end the paragraph by either linking back to your main topic or to the following paragraph.

Conclusions should include:

• restate the purpose of the review
• summary of the main findings : remind your reader of the main points. Make sure you paraphrase your ideas, so you don’t use the same wording as elsewhere in the literature review
• implications of the findings: suggest how the findings might be important for practice in your field
• areas for further research : provide suggestions for future research to address the problem, issue or question.

The conclusion is followed by a Reference list or Bibliography. Consult the Style notes  page of the  Academic Referencing Tool  for examples.

For complete sample literature reviews with further annotations, see the Word and PDF documents below.

• Example Literature Review - Allied Health [PDF 245KB]
• Example Literature Review - Allied Health [Word 69KB]
• Example Literature Review - Education [PDF 283KB]
• Example Literature Review - Education [Word 104KB]

Further resources

• Using a reading to choose a research topic worksheet
• Manchester Academic Phrasebank - Phrases for academic writing and reporting on research
• Turn a stack of papers into a literature review: Useful tool for beginners (Journal article)

Pathfinder link

Still have questions? Do you want to talk to an expert? Peer Learning Advisors or Academic Skills and Language Advisors  are available.

Cant R., Ryan, C., Hughes, L., Luders, E., & Cooper, S. (2021). What helps, what hinders? Undergraduate nursing students’ perceptions of clinical placements based on a thematic synthesis of literature. SAGE Open Nursing, 7, 1-20.  https://doi.org/10.1177/23779608211035845 Adapted and used under CC BY-NC 4.0 license

• << Previous: Annotated bibliographies
• Next: Presentations >>

Literature Review: GRADE System

• Lit Review Types
• GRADE System
• Do a Lit Review
• Citation Justice
• Lit Review Sources

GRADE System for Systematic Reviews

The Grades of Recommendation, Assessment, Development and Evaluation (GRADE) for Systematic Reviews

“ GRADE defines the quality of a body of evidence as the extent to which one can be confident that an estimate of effect or association is close to the quantity of specific interest.” 1

The GRADE system entails an assessment of the quality of a body of evidence for five factors :

• Within-study risk of bias (methodological quality)
• Directness of evidence
• Heterogeneity (any kind of variability among studies)
• Precision of effect estimates
• Risk of publication bias

Levels of quality of a body of evidence in the GRADE approach

The highest quality rating is for randomized trial evidence.

• Randomization sequence was not concealed from the clinicians and researchers
• The people involved in the study discovered which treatments were given to which patients
• Over 50% of their patients were lost for follow-up
• Conversely, observational studies may be upgraded if they produce large effects with no obvious bias

Information Sources:

• 1 Cochrane Handbook for Systematic Reviews of Interventions: http://handbook.cochrane.org/chapter_12/12_2_assessing_the_quality_of_a_body_of_evidence.htm
• Introduction to evidence Based Practice, Duke University Medical Center Library http://guides.mclibrary.duke.edu/ebmtutorial
• << Previous: Lit Review Types
• Next: Do a Lit Review >>
• Last Updated: Feb 16, 2024 8:43 AM
• URL: https://libguides.utoledo.edu/litreview

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• Current issue
• Write for Us
• BMJ Journals More You are viewing from: Google Indexer

You are here

• Volume 19, Issue 1
• Reviewing the literature
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

• Joanna Smith 1 ,
• Helen Noble 2
• 1 School of Healthcare, University of Leeds , Leeds , UK
• 2 School of Nursing and Midwifery, Queens's University Belfast , Belfast , UK
• Correspondence to Dr Joanna Smith , School of Healthcare, University of Leeds, Leeds LS2 9JT, UK; j.e.smith1{at}leeds.ac.uk

https://doi.org/10.1136/eb-2015-102252

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Implementing evidence into practice requires nurses to identify, critically appraise and synthesise research. This may require a comprehensive literature review: this article aims to outline the approaches and stages required and provides a working example of a published review.

Are there different approaches to undertaking a literature review?

What stages are required to undertake a literature review.

The rationale for the review should be established; consider why the review is important and relevant to patient care/safety or service delivery. For example, Noble et al 's 4 review sought to understand and make recommendations for practice and research in relation to dialysis refusal and withdrawal in patients with end-stage renal disease, an area of care previously poorly described. If appropriate, highlight relevant policies and theoretical perspectives that might guide the review. Once the key issues related to the topic, including the challenges encountered in clinical practice, have been identified formulate a clear question, and/or develop an aim and specific objectives. The type of review undertaken is influenced by the purpose of the review and resources available. However, the stages or methods used to undertake a review are similar across approaches and include:

Formulating clear inclusion and exclusion criteria, for example, patient groups, ages, conditions/treatments, sources of evidence/research designs;

Justifying data bases and years searched, and whether strategies including hand searching of journals, conference proceedings and research not indexed in data bases (grey literature) will be undertaken;

Developing search terms, the PICU (P: patient, problem or population; I: intervention; C: comparison; O: outcome) framework is a useful guide when developing search terms;

Developing search skills (eg, understanding Boolean Operators, in particular the use of AND/OR) and knowledge of how data bases index topics (eg, MeSH headings). Working with a librarian experienced in undertaking health searches is invaluable when developing a search.

Once studies are selected, the quality of the research/evidence requires evaluation. Using a quality appraisal tool, such as the Critical Appraisal Skills Programme (CASP) tools, 5 results in a structured approach to assessing the rigour of studies being reviewed. 3 Approaches to data synthesis for quantitative studies may include a meta-analysis (statistical analysis of data from multiple studies of similar designs that have addressed the same question), or findings can be reported descriptively. 6 Methods applicable for synthesising qualitative studies include meta-ethnography (themes and concepts from different studies are explored and brought together using approaches similar to qualitative data analysis methods), narrative summary, thematic analysis and content analysis. 7 Table 1 outlines the stages undertaken for a published review that summarised research about parents’ experiences of living with a child with a long-term condition. 8

• View inline

An example of rapid evidence assessment review

In summary, the type of literature review depends on the review purpose. For the novice reviewer undertaking a review can be a daunting and complex process; by following the stages outlined and being systematic a robust review is achievable. The importance of literature reviews should not be underestimated—they help summarise and make sense of an increasingly vast body of research promoting best evidence-based practice.

• ↵ Centre for Reviews and Dissemination . Guidance for undertaking reviews in health care . 3rd edn . York : CRD, York University , 2009 .
• ↵ Canadian Best Practices Portal. http://cbpp-pcpe.phac-aspc.gc.ca/interventions/selected-systematic-review-sites / ( accessed 7.8.2015 ).
• Bridges J , et al
• ↵ Critical Appraisal Skills Programme (CASP). http://www.casp-uk.net / ( accessed 7.8.2015 ).
• Dixon-Woods M ,
• Shaw R , et al
• Agarwal S ,
• Jones D , et al
• Cheater F ,

Twitter Follow Joanna Smith at @josmith175

Competing interests None declared.

Read the full text or download the PDF:

Academia.edu no longer supports Internet Explorer.

To browse Academia.edu and the wider internet faster and more securely, please take a few seconds to  upgrade your browser .

Enter the email address you signed up with and we'll email you a reset link.

• We're Hiring!
• Help Center

Criteria for Evaluation of Literature Reviews - Rubric

Related Papers

Coaching: An International Journal of Theory, Research and Practice

Mark N K Saunders , Céline Rojon

Exploring and evaluating findings from previous research is an essential aspect of all research projects enabling the work to be set in the context of what is known and what is not known. This necessitates a critical review of the literature in which existing research is discussed and evaluated, thereby contextualising and justifying the project. In this research note we consider what is understood by being critical when reviewing prior to outlining the key attributes of a critical literature review. We conclude with a summary checklist to help ensure a literature review is critical.

Journal of Asian Development

Erni Murniarti

A debatable consensus on and appropriate approaches to literature review function as the theoretical background of the paper. It redefines the literature review substance, synthesis, and procedure to literature matrix, and literature review assessment. In addition, some implications are to interpret the discussion. Finally, finding of the present study is applicable to any study fields at it generally provides matrix for which writing literature review can be easily conducted. Suggestions, identical to the potential of further studies and its application by ongoing researcher and writer, are holistically provided.

Sadruddin Qutoshi

Amanda Bolderston

A literature review can be an informative, critical, and useful synthesis of a particular topic. It can identify what is known (and unknown) in the subject area, identify areas of controversy or debate, and help formulate questions that need further research. There are several commonly used formats for literature reviews, including systematic reviews conducted as primary research projects; reviews written as an introduction and foundation for a research study, such as a thesis or dissertation; and reviews as secondary data analysis research projects. Regardless of the type, a good review is characterized by the author’s efforts to evaluate and critically analyze the relevant work in the field. Published reviews can be invaluable, because they collect and disseminate evidence from diverse sources and disciplines to inform professional practice on a particular topic. This directed reading will introduce the learner to the process of conducting and writing their own literature review.

Rebekka Tunombili

Journal of Clinical Epidemiology - J CLIN EPIDEMIOL

Sara Gesuato

Izet Masic, MD, PhD, FWAAS, FEASA, FIAHSI, FEFMI, FACMI

Psychology Research and Applications

Walter Schumm

PLoS Computational Biology

Marco Pautasso

Reviewing the literature requires the ability to juggle multiple tasks, from finding and evaluating relevant material to synthesising information from various sources, from critical thinking to paraphrasing, evaluating, and citation skills. In this contribution, I share ten simple rules I learned working on about 25 literature reviews as a PhD and postdoctoral student. Ideas and insights also come from discussions with coauthors and colleagues, as well as feedback from reviewers and editors.

RELATED PAPERS

Judith M Read

Marilou Bruchon-Schweitzer

J. Mater. Sci. Mater. Electron. 30, 4880 (2019)

Philippe Ferrandis

Global Spine Journal

pablo andres miranda machado

Alexandre Machado Ferraz

Scientific Reports

Marian Graco

Ashley R Vaughn

Research in Transportation Business & Management

Albert Veenstra

2007 American Control Conference

Vijay Patel

Jurnal Kajian Komunikasi

priyo subekti Subekti

Diacovensia

Stjepan Radić

Abdou Karim KANE

Jurnal Riset dan Aplikasi Mahasiswa Informatika (JRAMI)

mei lestari

Capitolo in volume

Camilla Giantomasso

Margaretha Falensia bagariang

Indonesian Journal of Economics, Social, and Humanities

Abdullahi, Nimota Jibola kadir , Abdulmumin Abubakar Lawal

Cathy Yandell

Chemosphere

Dror Avisar

Jan van Santen

shafiyyah zahari

ahmed abdela

Annual Review of Astronomy and Astrophysics

Phil Marshall

GUNAWAN SILABAN

International Journal of Livestock Research

Dr Manzoor A Mir

See More Documents Like This

•   We're Hiring!
•   Help Center
• Find new research papers in:
• Health Sciences
• Earth Sciences
• Cognitive Science
• Mathematics
• Computer Science
• Academia ©2024

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• PeerJ Comput Sci

Automated language essay scoring systems: a literature review

Mohamed abdellatif hussein.

1 Information and Operations, National Center for Examination and Educational Evaluation, Cairo, Egypt

Hesham Hassan

2 Faculty of Computers and Information, Computer Science Department, Cairo University, Cairo, Egypt

Mohammad Nassef

Associated data.

The following information was supplied regarding data availability:

As this is a literature, review, there was no raw data.

Writing composition is a significant factor for measuring test-takers’ ability in any language exam. However, the assessment (scoring) of these writing compositions or essays is a very challenging process in terms of reliability and time. The need for objective and quick scores has raised the need for a computer system that can automatically grade essay questions targeting specific prompts. Automated Essay Scoring (AES) systems are used to overcome the challenges of scoring writing tasks by using Natural Language Processing (NLP) and machine learning techniques. The purpose of this paper is to review the literature for the AES systems used for grading the essay questions.

Methodology

We have reviewed the existing literature using Google Scholar, EBSCO and ERIC to search for the terms “AES”, “Automated Essay Scoring”, “Automated Essay Grading”, or “Automatic Essay” for essays written in English language. Two categories have been identified: handcrafted features and automatically featured AES systems. The systems of the former category are closely bonded to the quality of the designed features. On the other hand, the systems of the latter category are based on the automatic learning of the features and relations between an essay and its score without any handcrafted features. We reviewed the systems of the two categories in terms of system primary focus, technique(s) used in the system, the need for training data, instructional application (feedback system), and the correlation between e-scores and human scores. The paper includes three main sections. First, we present a structured literature review of the available Handcrafted Features AES systems. Second, we present a structured literature review of the available Automatic Featuring AES systems. Finally, we draw a set of discussions and conclusions.

AES models have been found to utilize a broad range of manually-tuned shallow and deep linguistic features. AES systems have many strengths in reducing labor-intensive marking activities, ensuring a consistent application of scoring criteria, and ensuring the objectivity of scoring. Although many techniques have been implemented to improve the AES systems, three primary challenges have been identified. The challenges are lacking of the sense of the rater as a person, the potential that the systems can be deceived into giving a lower or higher score to an essay than it deserves, and the limited ability to assess the creativity of the ideas and propositions and evaluate their practicality. Many techniques have only been used to address the first two challenges.

Introduction

Test items (questions) are usually classified into two types: selected-response (SR), and constructed-response (CR). The SR items, such as true/false, matching or multiple-choice, are much easier than the CR items in terms of objective scoring ( Isaacs et al., 2013 ). SR questions are commonly used for gathering information about knowledge, facts, higher-order thinking, and problem-solving skills. However, considerable skill is required to develop test items that measure analysis, evaluation, and other higher cognitive skills ( Stecher et al., 1997 ).

CR items, sometimes called open-ended, include two sub-types: restricted-response and extended-response items ( Nitko & Brookhart, 2007 ). Extended-response items, such as essays, problem-based examinations, and scenarios, are like restricted-response items, except that they extend the demands made on test-takers to include more complex situations, more difficult reasoning, and higher levels of understanding which are based on real-life situations requiring test-takers to apply their knowledge and skills to new settings or situations ( Isaacs et al., 2013 ).

In language tests, test-takers are usually required to write an essay about a given topic. Human-raters score these essays based on specific scoring rubrics or schemes. It occurs that the score of an essay scored by different human-raters vary substantially because human scoring is subjective ( Peng, Ke & Xu, 2012 ). As the process of human scoring takes much time, effort, and are not always as objective as required, there is a need for an automated essay scoring system that reduces cost, time and determines an accurate and reliable score.

Automated Essay Scoring (AES) systems usually utilize Natural Language Processing and machine learning techniques to automatically rate essays written for a target prompt ( Dikli, 2006 ). Many AES systems have been developed over the past decades. They focus on automatically analyzing the quality of the composition and assigning a score to the text. Typically, AES models exploit a wide range of manually-tuned shallow and deep linguistic features ( Farag, Yannakoudakis & Briscoe, 2018 ). Recent advances in the deep learning approach have shown that applying neural network approaches to AES systems has accomplished state-of-the-art results ( Page, 2003 ; Valenti, Neri & Cucchiarelli, 2017 ) with the additional benefit of using features that are automatically learnt from the data.

Survey methodology

The purpose of this paper is to review the AES systems literature pertaining to scoring extended-response items in language writing exams. Using Google Scholar, EBSCO and ERIC, we searched the terms “AES”, “Automated Essay Scoring”, “Automated Essay Grading”, or “Automatic Essay” for essays written in English language. AES systems which score objective or restricted-response items are excluded from the current research.

The most common models found for AES systems are based on Natural Language Processing (NLP), Bayesian text classification, Latent Semantic Analysis (LSA), or Neural Networks. We have categorized the reviewed AES systems into two main categories. The former is based on handcrafted discrete features bounded to specific domains. The latter is based on automatic feature extraction. For instance, Artificial Neural Network (ANN)-based approaches are capable of automatically inducing dense syntactic and semantic features from a text.

The literature of the two categories has been structurally reviewed and evaluated based on certain factors including: system primary focus, technique(s) used in the system, the need for training data, instructional application (feedback system), and the correlation between e-scores and human scores.

Handcrafted features AES systems

Project essay grader™ (peg).

Ellis Page developed the PEG in 1966. PEG is considered the earliest AES system that has been built in this field. It utilizes correlation coefficients to predict the intrinsic quality of the text. It uses the terms “trins” and “proxes” to assign a score. Whereas “trins” refers to intrinsic variables like diction, fluency, punctuation, and grammar,“proxes” refers to correlations between intrinsic variables such as average length of words in a text, and/or text length. ( Dikli, 2006 ; Valenti, Neri & Cucchiarelli, 2017 ).

The PEG uses a simple scoring methodology that consists of two stages. The former is the training stage and the latter is the scoring stage. PEG should be trained on a sample of essays from 100 to 400 essays, the output of the training stage is a set of coefficients ( β weights) for the proxy variables from the regression equation. In the scoring stage, proxes are identified for each essay, and are inserted into the prediction equation. To end, a score is determined by estimating coefficients ( β weights) from the training stage ( Dikli, 2006 ).

Some issues have been marked as a criticism for the PEG such as disregarding the semantic side of essays, focusing on surface structures, and not working effectively in case of receiving student responses directly (which might ignore writing errors). PEG has a modified version released in 1990, which focuses on grammar checking with a correlation between human assessors and the system ( r  = 0.87) ( Dikli, 2006 ; Page, 1994 ; Refaat, Ewees & Eisa, 2012 ).

Measurement Inc. acquired the rights of PEG in 2002 and continued to develop it. The modified PEG analyzes the training essays and calculates more than 500 features that reflect intrinsic characteristics of writing, such as fluency, diction, grammar, and construction. Once the features have been calculated, the PEG uses them to build statistical and linguistic models for the accurate prediction of essay scores ( Home—Measurement Incorporated, 2019 ).

Intelligent Essay Assessor™ (IEA)

IEA was developed by Landauer (2003) . IEA uses a statistical combination of several measures to produce an overall score. It relies on using Latent Semantic Analysis (LSA); a machine-learning model of human understanding of the text that depends on the training and calibration methods of the model and the ways it is used tutorially ( Dikli, 2006 ; Foltz, Gilliam & Kendall, 2003 ; Refaat, Ewees & Eisa, 2012 ).

IEA can handle students’ innovative answers by using a mix of scored essays and the domain content text in the training stage. It also spots plagiarism and provides feedback ( Dikli, 2006 ; Landauer, 2003 ). It uses a procedure for assigning scores in a process that begins with comparing essays to each other in a set. LSA examines the extremely similar essays. Irrespective of the replacement of paraphrasing, synonym, or reorganization of sentences, the two essays will be similar LSA. Plagiarism is an essential feature to overcome academic dishonesty, which is difficult to be detected by human-raters, especially in the case of grading a large number of essays ( Dikli, 2006 ; Landauer, 2003 ). ( Fig. 1 ) represents IEA architecture ( Landauer, 2003 ). IEA requires smaller numbers of pre-scored essays for training. On the contrary of other AES systems, IEA requires only 100 pre-scored training essays per each prompt vs. 300–500 on other systems ( Dikli, 2006 ).

Landauer (2003) used IEA to score more than 800 students’ answers in middle school. The results showed a 0.90 correlation value between IEA and the human-raters. He explained the high correlation value due to several reasons including that human-raters could not compare each essay to each other for the 800 students while IEA can do so ( Dikli, 2006 ; Landauer, 2003 ).

E-rater ®

Educational Testing Services (ETS) developed E-rater in 1998 to estimate the quality of essays in various assessments. It relies on using a combination of statistical and NLP techniques to extract linguistic features (such as grammar, usage, mechanics, development) from text to start processing, then compares scores with human graded essays ( Attali & Burstein, 2014 ; Dikli, 2006 ; Ramineni & Williamson, 2018 ).

The E-rater system is upgraded annually. The current version uses 11 features divided into two areas: writing quality (grammar, usage, mechanics, style, organization, development, word choice, average word length, proper prepositions, and collocation usage), and content or use of prompt-specific vocabulary ( Ramineni & Williamson, 2018 ).

The E-rater scoring model consists of two stages: the model of the training stage, and the model of the evaluation stage. Human scores are used for training and evaluating the E-rater scoring models. The quality of the E-rater models and its effective functioning in an operational environment depend on the nature and quality of the training and evaluation data ( Williamson, Xi & Breyer, 2012 ). The correlation between human assessors and the system ranged from 0.87 to 0.94 ( Refaat, Ewees & Eisa, 2012 ).

Criterion SM

Criterion is a web-based scoring and feedback system based on ETS text analysis tools: E-rater ® and Critique. As a text analysis tool, Critique integrates a collection of modules that detect faults in usage, grammar, and mechanics, and recognizes discourse and undesirable style elements in writing. It provides immediate holistic scores as well ( Crozier & Kennedy, 1994 ; Dikli, 2006 ).

Criterion similarly gives personalized diagnostic feedback reports based on the types of assessment instructors give when they comment on students’ writings. This component of the Criterion is called an advisory component. It is added to the score, but it does not control it[18]. The types of feedback the advisory component may provide are like the following:

• • The text is too brief (a student may write more).
• • The essay text does not look like other essays on the topic (the essay is off-topic).
• • The essay text is overly repetitive (student may use more synonyms) ( Crozier & Kennedy, 1994 ).

IntelliMetric™

Vantage Learning developed the IntelliMetric systems in 1998. It is considered the first AES system which relies on Artificial Intelligence (AI) to simulate the manual scoring process carried out by human-raters under the traditions of cognitive processing, computational linguistics, and classification ( Dikli, 2006 ; Refaat, Ewees & Eisa, 2012 ).

IntelliMetric relies on using a combination of Artificial Intelligence (AI), Natural Language Processing (NLP) techniques, and statistical techniques. It uses CogniSearch and Quantum Reasoning technologies that were designed to enable IntelliMetric to understand the natural language to support essay scoring ( Dikli, 2006 ).

IntelliMetric uses three steps to score essays as follows:

• a) First, the training step that provides the system with known scores essays.
• b) Second, the validation step examines the scoring model against a smaller set of known scores essays.
• c) Finally, application to new essays with unknown scores. ( Learning, 2000 ; Learning, 2003 ; Shermis & Barrera, 2002 )

IntelliMetric identifies text related characteristics as larger categories called Latent Semantic Dimensions (LSD). ( Figure 2 ) represents the IntelliMetric features model.

IntelliMetric scores essays in several languages including English, French, German, Arabic, Hebrew, Portuguese, Spanish, Dutch, Italian, and Japanese ( Elliot, 2003 ). According to Rudner, Garcia, and Welch ( Rudner, Garcia & Welch, 2006 ), the average of the correlations between IntelliMetric and human-raters was 0.83 ( Refaat, Ewees & Eisa, 2012 ).

MY Access is a web-based writing assessment system based on the IntelliMetric AES system. The primary aim of this system is to provide immediate scoring and diagnostic feedback for the students’ writings in order to motivate them to improve their writing proficiency on the topic ( Dikli, 2006 ).

MY Access system contains more than 200 prompts that assist in an immediate analysis of the essay. It can provide personalized Spanish and Chinese feedback on several genres of writing such as narrative, persuasive, and informative essays. Moreover, it provides multilevel feedback—developing, proficient, and advanced—as well ( Dikli, 2006 ; Learning, 2003 ).

Bayesian Essay Test Scoring System™ (BETSY)

BETSY classifies the text based on trained material. It has been developed in 2002 by Lawrence Rudner at the College Park of the University of Maryland with funds from the US Department of Education ( Valenti, Neri & Cucchiarelli, 2017 ). It has been designed to automate essay scoring, but can be applied to any text classification task ( Taylor, 2005 ).

BETSY needs to be trained on a huge number (1,000 texts) of human classified essays to learn how to classify new essays. The goal of the system is to determine the most likely classification of an essay to a set of groups (Pass-Fail) and (Advanced - Proficient - Basic - Below Basic) ( Dikli, 2006 ; Valenti, Neri & Cucchiarelli, 2017 ). It learns how to classify a new document through the following steps:

The first-word training step is concerned with the training of words, evaluating database statistics, eliminating infrequent words, and determining stop words.

The second-word pairs training step is concerned with evaluating database statistics, eliminating infrequent word-pairs, maybe scoring the training set, and trimming misclassified training sets.

Finally, BETSY can be applied to a set of experimental texts to identify the classification precision for several new texts or a single text. ( Dikli, 2006 )

BETSY has achieved accuracy of over 80%, when trained with 462 essays, and tested with 80 essays ( Rudner & Liang, 2002 ).

Automatic featuring AES systems

Automatic text scoring using neural networks.

Alikaniotis, Yannakoudakis, and Rei introduced in 2016 a deep neural network model capable of learning features automatically to score essays. This model has introduced a novel method to identify the more discriminative regions of the text using: (1) a Score-Specific Word Embedding (SSWE) to represent words and (2) a two-layer Bidirectional Long-Short-Term Memory (LSTM) network to learn essay representations. ( Alikaniotis, Yannakoudakis & Rei, 2016 ; Taghipour & Ng, 2016 ).

Alikaniotis and his colleagues have extended the C&W Embeddings model into the Augmented C&W model to capture, not only the local linguistic environment of each word, but also how each word subsidizes to the overall score of an essay. In order to capture SSWEs . A further linear unit has been added in the output layer of the previous model which performs linear regression, predicting the essay score ( Alikaniotis, Yannakoudakis & Rei, 2016 ). Figure 3 shows the architectures of two models, (A) Original C&W model and (B) Augmented C&W model. Figure 4 shows the example of (A) standard neural embeddings to (B) SSWE word embeddings.

(A) Original C&W model. (B) Augmented C&W model.

(A) Standard neural embeddings. (B) SSWE word embeddings.

SSWEs obtained by their model used to derive continuous representations for each essay. Each essay is identified as a sequence of tokens. The uni- and bi-directional LSTMs have been efficiently used for embedding long sequences ( Alikaniotis, Yannakoudakis & Rei, 2016 ).

They used the Kaggle’s ASAP ( https://www.kaggle.com/c/asap-aes/data ) contest dataset. It consists of 12.976 essays, with average length 150-to-550 words per essay, each double marked (Cohen’s = 0.86). The essays presented eight different prompts, each with distinct marking criteria and score range.

Results showed that SSWE and the LSTM approach, without any prior knowledge of the language grammar or the text domain, was able to mark the essays in a very human-like way, beating other state-of-the-art systems. Furthermore, while tuning the models’ hyperparameters on a separate validation set ( Alikaniotis, Yannakoudakis & Rei, 2016 ), they did not perform any further preprocessing of the text other than simple tokenization. Also, it outperforms the traditional SVM model by combining SSWE and LSTM. On the contrary, LSTM alone did not give significant more accuracies compared to SVM.

According to Alikaniotis, Yannakoudakis, and Rei ( Alikaniotis, Yannakoudakis & Rei, 2016 ), the combination of SSWE with the two-layer bi-directional LSTM had the highest correlation value on the test set averaged 0.91 (Spearman) and 0.96 (Pearson).

A neural network approach to automated essay scoring

Taghipour and H. T. Ng developed in 2016 a Recurrent Neural Networks (RNNs) approach which automatically learns the relation between an essay and its grade. Since the system is based on RNNs, it can use non-linear neural layers to identify complex patterns in the data and learn them, and encode all the information required for essay evaluation and scoring ( Taghipour & Ng, 2016 ).

The designed model architecture can be presented in five layers as follow:

• a) The Lookup Table Layer; which builds d LT dimensional space containing each word projection.
• b) The Convolution Layer; which extracts feature vectors from n-grams. It can possibly capture local contextual dependencies in writing and therefore enhance the performance of the system.
• c) The Recurrent Layer; which processes the input to generate a representation for the given essay.
• d) The Mean over Time; which aggregates the variable number of inputs into a fixed length vector.
• e) The Linear Layer with Sigmoid Activation; which maps the generated output vector from the mean-over-time layer to a scalar value ( Taghipour & Ng, 2016 ).

Taghipour and his colleagues have used the Kaggle’s ASAP contest dataset. They distributed the data set into 60% training set, 20% a development set, and 20% a testing set. They used Quadratic Weighted Kappa (QWK) as an evaluation metric. For evaluating the performance of the system, they compared it to an available open source AES system called the ‘Enhanced AI Scoring Engine’ (EASE) ( https://github.com/edx/ease ). To identify the best model, they performed several experiments like Convolutional vs. Recurrent Neural Network, basic RNN vs. Gated Recurrent Units (GRU) vs. LSTM, unidirectional vs. Bidirectional LSTM, and using with vs. without mean-over-time layer ( Taghipour & Ng, 2016 ).

The results showed multiple observations according to ( Taghipour & Ng, 2016 ), summarized as follows:

• a) RNN failed to get accurate results as LSTM or GRU and the other models outperformed it. This was possibly due to the relatively long sequences of words in writing.
• b) The neural network performance was significantly affected with the absence of the mean over-time layer. As a result, it did not learn the task in an exceedingly proper manner.
• c) The best model was the combination of ten instances of LSTM models with ten instances of CNN models. The new model outperformed the baseline EASE system by 5.6% and with averaged QWK value 0.76.

Automatic features for essay scoring—an empirical study

Dong and Zhang provided in 2016 an empirical study to examine a neural network method to learn syntactic and semantic characteristics automatically for AES, without the need for external pre-processing. They built a hierarchical Convolutional Neural Network (CNN) structure with two levels in order to model sentences separately ( Dasgupta et al., 2018 ; Dong & Zhang, 2016 ).

Dong and his colleague built a model with two parts, summarized as follows:

• a) Word Representations: A word embedding is used but does not rely on POS-tagging or other pre-processing.
• b) CNN Model: They took essay scoring as a regression task and employed a two-layer CNN model, in which one Convolutional layer is used to extract sentences representations, and the other is stacked on sentence vectors to learn essays representations.

The dataset that they employed in experiments is the Kaggle’s ASAP contest dataset. The settings of data preparation followed the one that Phandi, Chai, and Ng used ( Phandi, Chai & Ng, 2015 ). For domain adaptation (cross-domain) experiments, they followed Phandi, Chai, and Ng ( Phandi, Chai & Ng, 2015 ), by picking four pairs of essay prompts, namely, 1 → 2, 3 →4, 5 →6 and 7 →8, where 1 →2 denotes prompt one as source domain and prompt 2 as target domain. They used quadratic weighted Kappa (QWK) as the evaluation metric.

In order to evaluate the performance of the system, they compared it to EASE system (an open source AES available for public) with its both models Bayesian Linear Ridge Regression (BLRR) and Support Vector Regression (SVR).

The Empirical results showed that the two-layer Convolutional Neural Network (CNN) outperformed other baselines (e.g., Bayesian Linear Ridge Regression) on both in-domain and domain adaptation experiments on the Kaggle’s ASAP contest dataset. So, the neural features learned by CNN were very effective in essay marking, handling more high-level and abstract information compared to manual feature templates. In domain average, QWK value was 0.73 vs. 0.75 for human rater ( Dong & Zhang, 2016 ).

Augmenting textual qualitative features in deep convolution recurrent neural network for automatic essay scoring

In 2018, Dasgupta et al. (2018) proposed a Qualitatively enhanced Deep Convolution Recurrent Neural Network architecture to score essays automatically. The model considers both word- and sentence-level representations. Using a Hierarchical CNN connected with a Bidirectional LSTM model they were able to consider linguistic, psychological and cognitive feature embeddings within a text ( Dasgupta et al., 2018 ).

The designed model architecture for the linguistically informed Convolution RNN can be presented in five layers as follow:

• a) Generating Embeddings Layer: The primary function is constructing previously trained sentence vectors. Sentence vectors extracted from every input essay are appended with the formed vector from the linguistic features determined for that sentence.
• b) Convolution Layer: For a given sequence of vectors with K windows, this layer function is to apply linear transformation for all these K windows. This layer is fed by each of the generated word embeddings from the previous layer.
• c) Long Short-Term Memory Layer: The main function of this layer is to examine the future and past sequence context by connecting Bidirectional LSTMs (Bi-LSTM) networks.
• d) Activation layer: The main function of this layer is to obtain the intermediate hidden layers from the Bi-LSTM layer h 1 , h 2 ,…, h T , and in order to calculate the weights of sentence contribution to the final essay’s score (quality of essay). They used an attention pooling layer over sentence representations.
• e) The Sigmoid Activation Function Layer: The main function of this layer is to perform a linear transformation of the input vector that converts it to a scalar value (continuous) ( Dasgupta et al., 2018 ).

Figure 5 represents the proposed linguistically informed Convolution Recurrent Neural Network architecture.

Dasgupta and his colleagues employed in their experiments the Kaggle’s ASAP contest dataset. They have done 7 folds using cross validation technique to assess their models. Every fold is distributed as follows; training set which represents 80% of the data, development set represented by 10%, and the rest 10% as the test set. They used quadratic weighted Kappa (QWK) as the evaluation metric.

The results showed that, in terms of all these parameters, the Qualitatively Enhanced Deep Convolution LSTM (Qe-C-LSTM) system performed better than the existing, LSTM, Bi-LSTM and EASE models. It achieved a Pearson’s and Spearman’s correlation of 0.94 and 0.97 respectively as compared to that of 0.91 and 0.96 in Alikaniotis, Yannakoudakis & Rei (2016) . They also accomplished an RMSE score of 2.09. They computed a pairwise Cohen’s k value of 0.97 as well ( Dasgupta et al., 2018 ).

Summary and Discussion

Over the past four decades, there have been several studies that examined the approaches of applying computer technologies on scoring essay questions. Recently, computer technologies have been able to assess the quality of writing using AES technology. Many attempts have taken place in developing AES systems in the past years ( Dikli, 2006 ).

The AES systems do not assess the intrinsic qualities of an essay directly as human-raters do, but they utilize the correlation coefficients of the intrinsic qualities to predict the score to be assigned to an essay. The performance of these systems is evaluated based on the comparison of the scores assigned to a set of essays scored by expert humans.

The AES systems have many strengths mainly in reducing labor-intensive marking activities, overcoming time, cost, and improving the reliability of writing tasks. Besides, they ensure a consistent application of marking criteria, therefore facilitating equity in scoring. However, there is a substantial manual effort involved in reaching these results on different domains, genres, prompts, and so forth. Moreover, the linguistic features intended to capture the aspects of writing to be assessed are hand-selected and tuned for specific domains. In order to perform well on different data, separate models with distinct feature sets are typically tuned ( Burstein, 2003 ; Dikli, 2006 ; Hamp-Lyons, 2001 ; Rudner & Gagne, 2001 ; Rudner & Liang, 2002 ). Despite its weaknesses, the AES systems continue to attract the attention of public schools, universities, testing agencies, researchers and educators ( Dikli, 2006 ).

The AES systems described in this paper under the first category are based on handcrafted features and, usually, rely on regression methods. They employ several methods to obtain the scores. While E-rater and IntelliMetric use NLP techniques, the IEA system utilizes LSA. Moreover, PEG utilizes proxy measures (proxes), and BETSY™ uses Bayesian procedures to evaluate the quality of a text.

While E-rater, IntelliMetric, and BETSY evaluate style and semantic content of essays, PEG only evaluates style and ignores the semantic aspect of essays. Furthermore, IEA is exclusively concerned with semantic content. Unlike PEG, E-rater, IntelliMetric, and IEA need smaller numbers of pre-scored essays for training in contrast with BETSY which needs a huge number of training pre-scored essays.

The systems in the first category have high correlations with human-raters. While PEG, E-rater, IEA, and BETSY evaluate only English language essay responses, IntelliMetric evaluates essay responses in multiple languages.

Contrary to PEG, IEA, and BETSY, E-rater, and IntelliMetric have instructional or immediate feedback applications (i.e., Criterion and MY Access!). Instructional-based AES systems have worked hard to provide formative assessments by allowing students to save their writing drafts on the system. Thus, students can review their writings as of the formative feedback received from either the system or the teacher. The recent version of MY Access! (6.0) provides online portfolios and peer review.

The drawbacks of this category may include the following: (a) feature engineering can be time-consuming, since features need to be carefully handcrafted and selected to fit the appropriate model, and (b) such systems are sparse and instantiated by discrete pattern-matching.

AES systems described in this paper under the second category are usually based on neural networks. Neural Networking approaches, especially Deep Learning techniques, have been shown to be capable of inducing dense syntactic and semantic features automatically, applying them to text analysis and classification problems including AES systems ( Alikaniotis, Yannakoudakis & Rei, 2016 ; Dong & Zhang, 2016 ; Taghipour & Ng, 2016 ), and giving better results with regards to the statistical models used in the handcrafted features ( Dong & Zhang, 2016 ).

Recent advances in Deep Learning have shown that neural approaches to AES achieve state-of-the-art results ( Alikaniotis, Yannakoudakis & Rei, 2016 ; Taghipour & Ng, 2016 ) with the additional advantage of utilizing features that are automatically learned from the data. In order to facilitate interpretability of neural models, a number of visualization techniques have been proposed to identify textual (superficial) features that contribute to model performance [7].

While Alikaniotis and his colleagues ( 2016 ) employed a two-layer Bidirectional LSTM combined with the SSWE for essay scoring tasks, Taghipour & Ng (2016) adopted the LSTM model and combined it with CNN. Dong & Zhang (2016) developed a two-layer CNN, and Dasgupta and his colleagues ( 2018 ) proposed a Qualitatively Enhanced Deep Convolution LSTM. Unlike Alikaniotis and his colleagues ( 2016 ), Taghipour & Ng (2016) , Dong & Zhang (2016) , Dasgupta and his colleagues ( 2018 ) were interested in word-level and sentence-level representations as well as linguistic, cognitive and psychological feature embeddings. All linguistic and qualitative features were figured off-line and then entered in the Deep Learning architecture.

Although Deep Learning-based approaches have achieved better performance than the previous approaches, the performance may not be better using the complex linguistic and cognitive characteristics, which are very important in modeling such essays. See Table 1 for the comparison of AES systems.

In general, there are three primary challenges to AES systems. First, they are not able to assess essays as human-raters do because they do what they have been programmed to do ( Page, 2003 ). They eliminate the human element in writing assessment and lack the sense of the rater as a person ( Hamp-Lyons, 2001 ). This shortcoming was somehow overcome by obtaining high correlations between the computer and human-raters ( Page, 2003 ) although this is still a challenge.

The second challenge is whether the computer can be fooled by students or not ( Dikli, 2006 ). It is likely to “trick” the system by writing a longer essay to obtain higher score for example ( Kukich, 2000 ). Studies, such as the GRE study in 2001, examined whether a computer could be deceived and assign a lower or higher score to an essay than it should deserve or not. The results revealed that it might reward a poor essay ( Dikli, 2006 ). The developers of AES systems have been utilizing algorithms to detect students who try to cheat.

Although automatic learning AES systems based on Neural Networks algorithms, the handcrafted AES systems transcend automatic learning systems in one important feature. Handcrafted systems are highly related to the scoring rubrics that have been designed as a criterion for assessing a specific essay and human-raters use these rubrics to score essays a well. The objectivity of human-raters is measured by their commitment to the scoring rubrics. On the contrary, automatic learning systems extract the scoring criteria using machine learning and neural networks, which may include some factors that are not part of the scoring rubric, and, hence, is reminiscent of raters’ subjectivity (i.e., mode, nature of a rater’s character, etc.) Considering this point, handcrafted AES systems may be considered as more objective and fairer to students from the viewpoint of educational assessment.

The third challenge is measuring the creativity of human writing. Accessing the creativity of ideas and propositions and evaluating their practicality are still a pending challenge to both categories of AES systems which still needs further research.

Funding Statement

The authors received no funding for this work.

Additional Information and Declarations

The authors declare there are no competing interests.

Mohamed Abdellatif Hussein conceived and designed the experiments, performed the experiments, analyzed the data, contributed reagents/materials/analysis tools, prepared figures and/or tables.

Hesham Hassan and Mohammad Nassef authored or reviewed drafts of the paper, approved the final draft.

We suggest our customers use the original top-level work we provide as a study aid and not as final papers to be submitted in class. Order your custom work and get straight A's.

Customer Reviews

Emilie Nilsson

Connect with the writers

Once paid, the initial draft will be made. For any query r to ask for revision, you can get in touch with the online chat support available 24X7 for you.

How safe will my data be with you?

Essay service features that matter.

Finished Papers

Online Essay Writing Service to Reach Academic Success.

Are you looking for the best essay writing service to help you with meeting your academic goals? You are lucky because your search has ended. is a place where all students get exactly what they need: customized academic papers written by experts with vast knowledge in all fields of study. All of our writers are dedicated to their job and do their best to produce all types of academic papers of superior quality. We have experts even in very specific fields of study, so you will definitely find a writer who can manage your order.

Customer Reviews

260 King Street, San Francisco

Updated Courtyard facing Unit at the Beacon! This newly remodeled…

Please fill the form correctly

• History Category
• Psychology Category
• Informative Category
• Analysis Category
• Business Category
• Economics Category
• Health Category
• Literature Category
• Review Category
• Sociology Category
• Technology Category

Is buying essays online safe?

Shopping through online platforms is a highly controversial issue. Naturally, you cannot be completely sure when placing an order through an unfamiliar site, with which you have never cooperated. That is why we recommend that people contact trusted companies that have hundreds of positive reviews.

As for buying essays through sites, then you need to be as careful as possible and carefully check every detail. Read company reviews on third-party sources or ask a question on the forum. Check out the guarantees given by the specialists and discuss cooperation with the company manager. Do not transfer money to someone else's account until they send you a document with an essay for review.

Good online platforms provide certificates and some personal data so that the client can have the necessary information about the service manual. Service employees should immediately calculate the cost of the order for you and in the process of work are not entitled to add a percentage to this amount, if you do not make additional edits and preferences.

The writers of PenMyPaper establish the importance of reflective writing by explaining its pros and cons precisely to the readers. They tend to ‘do my essay’ by adding value to both you (enhancing your knowledge) and your paper.

Finished Papers