footnote in essay example

What Are Footnotes and How Do You Use Them?

#scribendiinc

Written by  Scribendi

While reading a book or article, have you ever noticed little numbers placed at the ends of some sentences?

These numbers usually appear as superscripts and correspond with numbers placed at the bottom of the page, next to which appears further information that is both necessary and supplementary. Sometimes this information will come in the form of citations, but sometimes it will simply present additional notes about the topic at hand.

These citations and explanations are called "footnotes" (because they appear in the footer of the page). Take a look at the example below to see where footnotes appear on a page:

We've outlined how to use footnotes below. Check it out!

1. What Are Footnotes?

2. footnotes vs. endnotes, 2.1 should i use footnotes or endnotes, 3. how to do footnote citations, 3.1 in-text citations, 3.2 footnotes, 4. how to use footnotes in essays, 4.1 style guides, 4.1.1 modern language association (mla), 4.1.2  american psychological association (apa), 4.1.3  chicago manual of style (cms), 5. technical guide to using footnotes, 5.1 how to add footnotes in microsoft word, 5.2 how to add footnotes in google docs, 6. final tips and tricks .

Footnotes are notes that are placed at the end of a page and used to reference parts of the text (generally using superscript numbers). Writers use footnotes for several purposes, including  citations , parenthetical information, outside sources, copyright permissions, background information, and more.

Now that you understand what footnotes are, you might be wondering: why use them? The truth is, long explanatory notes can be difficult for readers to trudge through (especially when they occur in the middle of a paper). Providing this information is necessary, but doing so in the main text can disrupt the flow of the writing.

Imagine if every time an author wanted to provide a citation, the entire citation had to be written out at the end of the sentence, like this (Anthony Grafton, The Footnote: A Curious History [Cambridge, MA: Harvard University Press, 1999] 221). Books would become much longer and reading would be much more tedious. That's why footnotes are so useful: they let authors provide the required information without disrupting the flow of ideas.

While footnotes are a great resource for sharing information without clogging up the writing, it's important to note that certain style guides restrict when footnotes can be used. We'll get into that soon!

Unsure how to edit your paper? Contact the Scribendi team for professional proofreading .

Authors can also use endnotes to avoid disrupting their writing with extraneous information. Both serve similar purposes; the main difference lies in their location in your text. Here's a closer look at how both footnotes and endnotes work.

• Identified in the main text with a small superscript number
• Used for citations, parenthetical information, outside sources, copyright permissions, background information, and more
• Provide the correlating notes at the bottom of the same page
• Identified in the main text with a small superscript number (like footnotes)
• Used for citations, parenthetical information, outside sources, copyright permissions, background information, and more (like footnotes)
• Found collectively at the end of an article, chapter, or document (unlike footnotes)

When deciding  whether to use footnotes or endnotes , authors must consider three main factors:

• The style guide being used (as some require either footnotes or endnotes)
• The number of notes being included (as having too many footnotes on each page can be distracting)
• Which option will be more convenient for the reader

To make a footnote citation, label the area of your text that you need to reference with a number (if it's your first footnote, start with "1."). At the bottom of the page, include this number with the citation. When readers see the number in the text, they know they can find the source by looking for the corresponding footnote.

Here's an example of a quoted piece of text using in-text citations vs. footnotes.

"Like the high whine of the dentist's drill, the low rumble of the footnote on the historian's page reassures" ( The Footnote: A Curious History [Cambridge, MA: Harvard University Press], 1999. pg. 1).

"Like the high whine of the dentist's drill, the low rumble of the footnote on the historian's page reassures." 1

[Text continues]

Bottom of the page:

1. The Footnote: A Curious History [Cambridge, MA: Harvard University Press], 1999. pg. 1

The exact format of your footnote depends on   the style guide  you're following. Here are some of the most common style guides for writing papers, as well as the footnote rules for each one.

Of the major style guides, The Chicago Manual of Style (CMS) uses footnotes most often. However, footnotes are occasionally employed in other style guides as well. The main difference is that, while CMS uses footnotes for citation purposes, the Modern Language Association (MLA) and the American Psychological Association (APA) generally rely on them for the provision of additional information.

While MLA style discourages the use of long footnotes or endnotes, the style guide does permit their use for directing readers to other pertinent information on a relevant subject.

The guide recommends that superscript numbers within the text are placed outside any punctuation that might be present (i.e., after a period if the note is at the end of a sentence and after a comma if the note is at the end of a clause). The exception to this is that the superscript numbers should be placed before dashes.

• When a footnote must be placed at the end of a clause, 1 add the number after the comma.
• When a footnote must be placed at the end of a sentence, add the number after the period. 2
• Numbers denoting footnotes should always appear after punctuation, with the exception of one piece of punctuation 3 —the dash.

4.1.2 American Psychological Association (APA)

Like MLA, APA discourages the use of footnotes unless absolutely necessary. Even then, the guide recommends that footnotes only be used to provide content notes (such as providing brief, supplemental information about the text or directing readers to additional information) and to denote copyright permissions. The rules regarding placement of the in-text numbers is the same in APA as in MLA.

4.1.3 The Chicago Manual of Style (CMS)

Of the three main style guides described here, CMS relies on footnotes the most. While CMS does allow the author–date system of in-text referencing (i.e., providing the author's name and the date of publication in parentheses at the end of the phrase, clause, or sentence that references the work), it also offers a citation style in which footnotes or endnotes are employed. In both cases, bibliographies are also required. Whether an author should use the author–date system or footnotes is often decided by the author's professor, journal, or publisher.

As an example, if footnotes are used, the following format should be adhered to when referencing a book in CMS:

Let Us Revise Your Reference Material to Any Style Guide

Try our academic proofreading service , or get a free sample.

To use footnotes in your own book, essay, or article, you must first decide on the most appropriate and logical placement of your footnotes in the text. Add numbers according to your chosen style guide, and be sure to add the numbers directly after the phrase, clause, or sentence to which the corresponding footnote refers.

Most online writing programs (such as Microsoft Word and Google Docs) come with easy-to-use tools for inserting footnotes. Here are step-by-step guides to using footnotes in both these programs.

5.2 How to Add Footnotes in Microsoft Word

Here's how to use footnotes in Microsoft Word 2021:

• Click on the place in the text where you want the first footnote to appear.
• Under the References tab, you'll see the following symbol: AB.1. Beneath this symbol is a button with the words, "Insert Footnote." Click it to create your first footnote.
• After you click that button, two numbers should appear: one number should appear in the main text, and the corresponding number should appear at the bottom of the page.
• Write your citation or additional information next to the number that appears in the footer. Format the information according to the rules of your style guide.
• You can easily return to your place in the text by clicking the number at the beginning of the footnote.

Congrats! You've created your first footnote. You can also adjust the footnote settings (like the numbering) by clicking the arrow beside the Footnotes group. It's really that easy!

Here's how to use footnotes on Google Docs:

• Under the Insert tab, click on "Footnotes."

All you really have to do to create footnotes is click a button—it couldn't be easier!

6. Final Tips and Tricks

To  improve your writing  and avoid cluttering the page, you should use footnotes sparingly and only to provide helpful additions or citations. As previously noted, this information may be considered supplementary, which is why it's best to place it away from the main portion of your writing.

When creating your footnotes, always keep reader convenience in mind, and remember that the footnotes are there to convey helpful information. If your footnotes are excessive or unnecessary, readers are likely to become annoyed—they may even be distracted from the main points of your writing.

Now that you're no longer asking "What are footnotes?" and you know how to use them according to various style guides, footnotes can become a great asset to you as a writer. Be sure to follow the recommendations above, as well as those of your preferred style guide, to ensure that you're using footnotes to their best effect. Don't forget—if you ever need help with writing, our academic articles are here for you!

If you need  professional proofreading , let Scribendi perfect your writing.

Image source: Daria Nepriakhina/Stocksnap.io

Polish Your Writing with Professional Proofreading

About the author.

Scribendi's in-house editors work with writers from all over the globe to perfect their writing. They know that no piece of writing is complete without a professional edit, and they love to see a good piece of writing transformed into a great one. Scribendi's in-house editors are unrivaled in both experience and education, having collectively edited millions of words and obtained numerous degrees. They love consuming caffeinated beverages, reading books of various genres, and relaxing in quiet, dimly lit spaces.

Have You Read?

"The Complete Beginner's Guide to Academic Writing"

Related Posts

How to Cite a Website (and Achieve True Unagi)

How to Create a Bibliography Using Word

Turabian Style: How to Use It

Upload your file(s) so we can calculate your word count, or enter your word count manually.

We will also recommend a service based on the file(s) you upload.

English is not my first language. I need English editing and proofreading so that I sound like a native speaker.

I need to have my journal article, dissertation, or term paper edited and proofread, or I need help with an admissions essay or proposal.

I have a novel, manuscript, play, or ebook. I need editing, copy editing, proofreading, a critique of my work, or a query package.

I need editing and proofreading for my white papers, reports, manuals, press releases, marketing materials, and other business documents.

I need to have my essay, project, assignment, or term paper edited and proofread.

I want to sound professional and to get hired. I have a resume, letter, email, or personal document that I need to have edited and proofread.

 Prices include your personal % discount.

 Prices include % sales tax ( ).

• Link to facebook
• Link to linkedin
• Link to twitter
• Link to youtube
• Writing Tips

How to Use Footnotes and Endnotes

4-minute read

• 5th June 2019

Footnotes and endnotes both let you add extra information in an essay or college paper . But what should you include in these notes? And when should you use them? In this post, we run through everything you need to know about using footnotes and endnotes in academic writing.

What Are Footnotes and Endnotes?

Footnotes appear at the bottom or “foot” of the page. You can therefore put extra information in a footnote, such as source details for a citation, without interrupting the flow of the main text.

To indicate a footnote, you can add a superscript number to the text, such as at the end of this sentence. 1 These numbers then correspond to numbered notes at the bottom of the page.

Endnotes are like footnotes, but they appear together at the end of the document rather than at the bottom of each page. Endnotes are thus less immediately accessible for the reader than footnotes, but they can help ensure that pages with multiple notes don’t become cluttered.

If you are not sure which to use, check your style guide for advice.

Footnotes and Endnotes in Microsoft Word

To insert a footnote or endnote in a Microsoft Word document, you need to:

• Go to References > Footnotes on the main ribbon
• Select either Insert Footnote or Insert Endnote as required
• Type your note in the newly created footnote/endnote

You can also customize the style of footnotes and endnotes by clicking on the arrow in the bottom right of the Footnotes section of the References tab (or by going to Insert > Footnotes in Word for Mac ). This will open a new window where you can select your preferred formatting options.

Find this useful?

Subscribe to our newsletter and get writing tips from our editors straight to your inbox.

When to Use Footnotes and Endnotes

The main uses of footnotes and endnotes are as follows:

• To add a footnote citation in referencing systems such as MHRA and Chicago , with full source information also given in a bibliography at the end of the document. Endnotes are also used for citations in some systems, such as in IEEE or Vancouver referencing, where numbers in the text point to an entry in a reference list at the end of the document.
• To add non-essential commentary on something in the main text of your document. For example, if your research has raised an interesting question that is not directly relevant to your current work, you could mention it in a footnote or endnote. This lets you acknowledge the question – showing the reader that you haven’t simply ignored or failed to notice it – but without interrupting the flow of prose in the main document.

Keep in mind, too, that some referencing systems use in-text parenthetical citations . As such, you should only reference a source in a footnote or endnote if your school has asked you to do it this way.

Do Notes Count Towards the Word Limit?

We’re often asked whether to include footnotes and endnotes in the word count for papers. Different schools have different rules about this, so you will have to check your style guide . However, you should never use these supplementary notes to cheat the word count.

The key here is that essential information should never go in a footnote or endnote. If you do move vital evidence or analysis to a note, the person marking your work may ignore it. And reducing the word count is never more important than putting forward a full, coherent argument.

If you do need to reduce the word count in an essay, you have other options, such as rewriting wordy sentences or cutting repetition. Having your work proofread is a great way to ensure that your writing is always clear and concise, too, so let us know if you’d like any help.

Share this article:

Post A New Comment

Got content that needs a quick turnaround? Let us polish your work. Explore our editorial business services.

9-minute read

How to Use Infographics to Boost Your Presentation

Is your content getting noticed? Capturing and maintaining an audience’s attention is a challenge when...

8-minute read

Why Interactive PDFs Are Better for Engagement

Are you looking to enhance engagement and captivate your audience through your professional documents? Interactive...

7-minute read

Seven Key Strategies for Voice Search Optimization

Voice search optimization is rapidly shaping the digital landscape, requiring content professionals to adapt their...

Five Creative Ways to Showcase Your Digital Portfolio

Are you a creative freelancer looking to make a lasting impression on potential clients or...

How to Ace Slack Messaging for Contractors and Freelancers

Effective professional communication is an important skill for contractors and freelancers navigating remote work environments....

3-minute read

How to Insert a Text Box in a Google Doc

Google Docs is a powerful collaborative tool, and mastering its features can significantly enhance your...

Make sure your writing is the best it can be with our expert English proofreading and editing.

• Skip to main content
• Skip to secondary menu
• Skip to primary sidebar
• Skip to footer

Erin Wright Writing

Writing-Related Software Tutorials

What Are Footnotes and How to Use Them

By Erin Wright

What Are Footnotes?

Footnotes are supplementary pieces of information that support your writing. If you’re following The Chicago Manual of Style (Chicago style, which is the best style guide for general business content), supplementary information includes works cited, suggestions for further research, commentary, quotations, copyright statements, or a combination of any of the above. 1

If you’re following the Publication Manual of the American Psychological Association (APA style) or MLA style from the Modern Language Association, works cited typically appear in the reference list or bibliography; so, footnotes are reserved for commentary, suggestions for further research, or copyright statements. 2

Work cited example based on Chicago style:

1. Lynne Truss, Eats, Shoots & Leaves (New York: Gotham Books, 2003), 89.

Commentary example:

2. This study excluded Groups D and E; therefore, it should not be considered exhaustive.

Suggestion for further research example:

3. Visit www.erinwrightwriting.com for more information about ampersands.

Where Should Footnotes Appear in Formal Documents?

Footnotes usually appear at the bottom of the page. Each footnote is preceded by a number that also appears as a superscript after the corresponding material on that page. Chicago style allows you to use symbols, such as the asterisk or the dagger, instead of numbers if you only have a few footnotes. 3

If you’re following APA style or MLA style, footnotes can appear at the foot of the page or all together at the end of the document. 4 (In Chicago style and MLA style, notes placed at the end of articles, chapters, or books are called endnotes. 5 ) Unlike Chicago style, APA style and MLA style don’t recommend using symbols as footnote identifiers. 6

Where Should Footnotes Appear in General Business Writing?

If you’re publishing less formal content online, such as a blog post or a how-to article, there’s no rule that says you can’t put footnotes at the end of individual sections. I like to call them “floating footnotes” because they float where they’re most needed instead of languishing at the end of a page or document.

In fact, floating footnotes can be more helpful than traditional footnotes for viewers who only need to read a few sections of your content. Floating footnotes can also benefit viewers who don’t want to scroll all the way to the end of a long webpage or ebook.

However, reserve floating footnotes for longer pieces so your content doesn’t become disjointed. If your blog post or article is only a couple of screen lengths, tradition footnotes should work just fine. You can see an example of a floating footnote in the second-to-last section of Three Ways to Add Currency Symbols in Microsoft Word .

Three Tips for Writing Footnotes

1. If your supplementary information is longer than a paragraph, consider using an appendix instead of a footnote.

2. If you’re following Chicago style and your footnotes are taking up too much page space, consider using endnotes instead.

3. Avoid unnecessary footnotes: if they don’t cite your sources or improve your readers’ understanding of the topic, they’re probably not necessary.

Check out these related posts on the differences between bibliographies and reference pages and how to insert footnotes and endnotes in Microsoft Word .

And of course, here are my footnotes for this blog post:

1.  The Chicago Manual of Style , 17th ed. (Chicago: University of Chicago Press, 2017), 14.19, 14.37–40.

2. Publication Manual of the American Psychological Association, 7th ed. (Washington, DC: American Psychological Association, 2020), 2.13; MLA Handbook , 9th ed. (New York; Modern Language Association, 2021), 7.1-7.2.

3. The Chicago Manual of Style , 14.25.

4. Publication Manual of the American Psychological Association , 2.13; MLA Handbook , 7.3.

5. The Chicago Manual of Style , 14.43; MLA Handbook , 7.3.

6. Publication Manual of the American Psychological Association , 2.13; MLA Handbook , 7.3.

Updated January 25, 2022

• Microsoft Word Tutorials
• Adobe Acrobat Tutorials
• PowerPoint Tutorials
• Writing Tips
• Editing Tips
• Writing-Related Resources

Home / Guides / Citation Guides / APA Format / How to do APA footnotes

How to do APA footnotes

Footnotes are a way for the author to provide additional content to their papers without distracting the reader from the text. The information in footnotes is different from the information provided in APA annotated bibliographies . Footnotes can be content based, providing a little more insight on an idea you raise in the text, or they can be used to provide copyright attribution for long quotes and passages.

Properly formatted APA footnotes can be placed at the bottom of the page. Alternatively, you can put them on their own page after the references. This guide on footnotes, end notes, and parentheticals provides information about the differences between these different types of notes. Either way, it’s important to know how to use footnotes properly.

In this guide, students can learn about the different uses for footnotes as well as how to format footnotes according to APA Style. All of the information here comes straight from the 7th edition of the Publication Manual .

Why use footnotes? What information goes into them?

There are two primary reasons why an author would use footnotes:

1. Using a footnote for content

As mentioned above, there are a few different ways to use footnotes. The more common way is when an author wants to provide extra insight on an idea without disrupting the flow of the text. This is called a content footnote.

In this case, you would write a a couple sentences about the extra insight. For example:

______________________

1 This data refers to the situation in 2010, and it includes emissions from industrial processes. Emissions from the latter are released during the physical and chemical transformation of materials like clinker production. Since these industrial production processes are also consumers of energy, here we made the choice to combine them with CO2 emissions from fossil fuel combustion.

2. Using a footnote for copyright attribution

When you are reproducing a portion of a copyrighted work, like an extended passage from a book or journal, it is necessary to provide copyright attribution. This can be done inside a footnote. The footnote is used instead of a parenthetical in-text citation, and you will still need to add the source as an entry in the reference list.

If it is an image or graph you are reproducing, copyright attribution can go in the figure note or table note.

A copyright footnote should start with “ From ” or “ Adapted from ” and the format will change slightly depending on the source.

Here is a template for copyright attribution for a website followed by two examples:

1 From  Webpage title , by Group Author OR Author FirstMiddleName Initials. Author Surname. Year Published, Website Name (URL).

*Note: If the Group Author and Website Name are the same, omit the Website Name slot.

2 From  First images from the James Webb Space Telescope , by National Aeronautics and Space Administration, 2022 (https://www.nasa.gov/webbfirstimages).

3 From  Question of what now for Syria remains as vexed as ever , by M. Chulov. 2022, The Guardian (https://www.theguardian.com/world/2022/jul/19/question-of-what-now-for-syria-remains-as-vexed-as-ever).

Endnotes vs. footnotes: What’s the difference?

According to APA Style, the author may choose to place the footnotes on the bottom of the page on which the callout appears or at the end of the paper on their own page(s).

“Endnotes” is a function on many word processors that inserts callouts and place the notes at the end of the document. While this is the same idea as footnotes, APA calls for a specially-formatted footnotes page.

To place the footnotes at the end of your document, check the preferences of the footnote function. You should be able to select “End of Document” instead of “End of Page.”

How to format APA footnotes

Always use the footnotes function of your word processor to insert footnotes. This will make it much easier to keep track of everything even as page content changes.

How to format footnotes correctly:

• Always use the footnotes function.
• The callout should be in superscript, like this. 1
• The callout should come after the punctuation, like this. 2
• If there’s a dash 3 —the callout comes before the punctuation, not after.
• All callouts should appear in numerical order, like this. 4

APA footnotes example

Now let’s have a look at what properly formatted APA footnotes look like in action.

Here is an example of a concise, relevant, and properly formatted footnote from “The role of renewable energy in the global economy transformation,” published in Energy Strategy Reviews.

. . . A transition away from fossil fuels to low-carbon solutions will play an essential role, as energy-related carbon dioxide (CO2) emissions represent two-thirds of all greenhouse gases (GHG). 1

In this example, the footnotes function automatically created a dividing line at the bottom of the document. It has also reduced the font size by 1pt, which is neither required nor discouraged by APA.

The reason this is a good example, however, is because the footnote provides supplemental information that is both relevant and substantive.  The information would have been too distracting to appear in the main text, but it provides helpful insight on the author’s research method.

Published October 28, 2020.

APA Formatting Guide

APA Formatting

• Annotated Bibliography
• Block Quotes
• et al Usage
• In-text Citations
• Multiple Authors
• Paraphrasing
• Page Numbers
• Parenthetical Citations
• Reference Page
• Sample Paper
• APA 7 Updates
• View APA Guide

Citation Examples

• Book Chapter
• Journal Article
• Magazine Article
• Newspaper Article
• Website (no author)
• View all APA Examples

How useful was this post?

Click on a star to rate it!

We are sorry that this post was not useful for you!

Let us improve this post!

Tell us how we can improve this post?

You can include more than one footnote on the same page in APA style. There is no restriction on the number of footnotes to be included on a page. Depending upon the number of footnotes on the page, the text area of the page will be automatically adjusted to fit the footnotes.

Footnotes in APA are used to provide the reader some additional information about the idea or the element being discussed. Footnotes are used in all types of publications such as journal articles, book chapters, and conference papers.

Two types of footnotes are used in APA style: content footnotes and copyright attribution footnotes. A content footnote provides additional explanation or information about something mentioned in the text, while a copyright attribution footnote provides copyright information for lengthy content that has been reprinted in the text. For both types, the in-text citation remains the same. Remember the following guidelines when you want to cite a footnote:

• Footnotes (whether content footnotes or copyright attribution footnotes) are numbered consecutively in the order in which they appear in the text.
• Use superscript Arabic numerals (1, 2, 3, etc.) to designate a footnote callout.
• This is a footnote. 1
• In this footnote, 2 the author tries to clarify the idea.
• A footnote callout—unlike in-text reference citation 3 —is simple to add.
• You should not add space before the footnote callout.
• If you want to refer to the same footnote again in the text, do not add any superscript Arabic numeral. Instead, write “see Footnote 3.” In this case, the footnote description need not be given again.

Note that a footnote should have only one idea. If you want to add more information, it is advisable to add the content in the text or create an appendix.

APA Citation Examples

Writing Tools

Citation Generators

Other Citation Styles

Plagiarism Checker

Upload a paper to check for plagiarism against billions of sources and get advanced writing suggestions for clarity and style.

Get Started

• Link to facebook
• Link to linkedin
• Link to twitter
• Link to youtube
• Writing Tips

How to Use Footnotes and Endnotes in Essays

• 4-minute read
• 23rd February 2019

Footnotes and endnotes both offer a way to add extra information to an essay . But what should you include in footnotes and endnotes? And when should you use them? In this post, we run through everything you need to know about using footnotes and endnotes in essays.

What Are Footnotes and Endnotes?

Footnotes appear at the bottom or ‘foot’ of the page. This lets you add information to an essay without interrupting the flow of the main text. Usually, this will be a citation or non-essential commentary.

To indicate a footnote, you will need to add a superscript number to the text, such as at the end of this sentence. 1 These numbers then correspond to numbered notes at the bottom of the page.

Endnotes are like footnotes, but they appear together at the end of the document rather than at the bottom of individual pages. This means endnotes are less immediately accessible for the reader than footnotes, but it helps ensure that pages with multiple notes don’t become cluttered. If you are not sure which to use, check your university style guide for advice.

Footnotes and Endnotes in Microsoft Word

To insert a footnote or endnote in a Microsoft Word document, you need to:

• Go to References > Footnotes on the main ribbon
• Select either Insert Footnote or Insert Endnote as required
• Type your note in the newly created footnote/endnote

You can also customise the style of footnotes and endnotes by clicking on the little arrow in the bottom right of the Footnotes section of the References tab (or by going to Insert > Footnotes in Word for Mac ). This will open a new window where you can select your preferred formatting options.

Find this useful?

Subscribe to our newsletter and get writing tips from our editors straight to your inbox.

When to Use Footnotes and Endnotes

The main uses of footnotes and endnotes are as follows:

• To add a footnote citation in referencing systems such as MHRA and Chicago , with full source information also given in a bibliography at the end of the document. Endnotes are also used for citations in some systems, such as in IEEE or Vancouver referencing, where numbers in the text point to an entry in a reference list at the end of the document.
• To add non-essential commentary on something in the main text of your document. For example, if your research has raised a question that is not directly relevant to your essay, you may want to mention it in a footnote or endnote instead. This lets you acknowledge it in your work – showing the reader that you haven’t simply ignored it or failed to notice something – but without interrupting the flow of the main document.

Keep in mind, too, that some referencing systems use in-text parenthetical citations . As such, you should only give references in footnotes or endnotes if your university has asked you to do this.

Do They Count Towards the Word Limit?

We’re often asked whether to include footnotes and endnotes in the word count for an essay. Different universities have different rules about this, so you will have to check your style guide . However, you should never use footnotes or endnotes to try and cheat the word count.

The key here is that only non-essential information should go in footnotes or endnotes. As such, if you move vital evidence or analysis to a footnote, the person marking your work may ignore it. And reducing the word count is never more important than putting forward a full, coherent argument.

If you do need to reduce the word count in an essay, you have other options, such as rewriting wordy sentences or cutting repetition. Having your work proofread is a great way to ensure that your writing is always clear and concise, too, so let us know if you’d like any help.

Share this article:

Post A New Comment

Get help from a language expert. Try our proofreading services for free.

9-minute read

How to Use Infographics to Boost Your Presentation

Is your content getting noticed? Capturing and maintaining an audience’s attention is a challenge when...

8-minute read

Why Interactive PDFs Are Better for Engagement

Are you looking to enhance engagement and captivate your audience through your professional documents? Interactive...

7-minute read

Seven Key Strategies for Voice Search Optimization

Voice search optimization is rapidly shaping the digital landscape, requiring content professionals to adapt their...

How to Ace Slack Messaging for Contractors and Freelancers

Effective professional communication is an important skill for contractors and freelancers navigating remote work environments....

3-minute read

How to Insert a Text Box in a Google Doc

Google Docs is a powerful collaborative tool, and mastering its features can significantly enhance your...

2-minute read

How to Cite the CDC in APA

If you’re writing about health issues, you might need to reference the Centers for Disease...

Make sure your writing is the best it can be with our expert English proofreading and editing.

Turabian Footnote/Endnote Style

Table of Contents: Books E-books Journal Articles (Print) Journal Articles (Online) Magazine Articles (Print) Magazine Articles (Online) Newspaper Articles Review Articles Websites For More Help

The examples in this guide are meant to introduce you to the basics of citing sources using Kate Turabian's A Manual for Writers of Term Papers, Theses, and Dissertations (seventh edition) .  Kate Turabian created her first "manual" in 1937 as a means of simplifying for students The Chicago Manual of Style ; the seventh edition of Turabian is based on the 15th edition of the Chicago Manual . For types of resources not covered in this guide (e.g., government documents, manuscript collections, video recordings) and for further detail and examples, please consult the websites listed at the end of this guide, the handbook itself or a reference librarian .

Whenever you refer to or use another's words, facts or ideas in your paper, you are required to cite the source. Traditionally, disciplines in the humanities (art, history, music, religion, theology) require the use of bibliographic footnotes or endnotes in conjunction with a bibliography to cite sources used in research papers and dissertations. For the parenthetical reference (author-date) system (commonly used in the sciences and social sciences), please refer to the separate guide Turabian Parenthetical/Reference List Style . It is best to consult with your professor to determine the preferred citation style.

Indicate notes in the text of your paper by using consecutive superscript numbers (as demonstrated below). The actual note is indented and can occur either as a footnote at the bottom of the page or as an endnote at the end of the paper. To create notes, type the note number followed by a period on the same line as the note itself. This method should always be used for endnotes; it is the preferred method for footnotes. However, superscript numbers are acceptable for footnotes, and many word processing programs can generate footnotes with superscript numbers for you.

When citing books, the following are elements you may need to include in your bibliographic citation for your first footnote or endnote and in your bibliography, in this order:

1. Author or editor; 2. Title; 3. Compiler, translator or editor (if an editor is listed in addition to an author); 4. Edition; 5. Name of series, including volume or number used; 6. Place of publication, publisher and date of publication; 7. Page numbers of citation (for footnote or endnote).

Books with One Author or Corporate Author

Author: Charles Hullmandel experimented with lithographic techniques throughout the early nineteenth century, patenting the "lithotint" process in 1840. 1

Editor: Human beings are the sources of "all international politics"; even though the holders of political power may change, this remains the same. 1

Corporate Author: Children of Central and Eastern Europe have not escaped the nutritional ramifications of iron deficiency, a worldwide problem. 1

First footnote:

1 Michael Twyman, Lithography 1800-1850 (London: Oxford University Press, 1970), 145-146.

1 Valerie M. Hudson, ed., Culture and Foreign Policy (Boulder: L. Rienner Publishers, 1997), 5.

1 UNICEF, Generation in Jeopardy: Children in Central and Eastern Europe and the Former Soviet Union , edited by Alexander Zouev (Armonk, NY: M. E. Sharpe, 1999), 44.

Note the different treatment of an editor's name depending on whether the editor takes the place of an author (second example) or is listed in addition to the author (third example). 

Subsequent footnotes:

       Method A: Include the author or editor's last name, the title (or an abbreviated title) and the page number cited.

2 Twyman, Lithography 1800-1850, 50.

2 Hudson, ed., Culture and Foreign Policy, 10.

2 UNICEF, Generation in Jeopardy, 48.

       Method B: Include only the author or editor's last name and the page number, leaving out the title.  

2 Twyman, 50.

2 Hudson, ed., 10.

2 UNICEF, 48.

Use Method A if you need to cite more than one reference by the same author.

1. Michael Twyman, Lithography 1800-1850  (London: Oxford University Press, 1970), 145-146.

Ibid., short for ibidem, means "in the same place."  Use ibid. if you cite the same page of the same work in succession without a different reference intervening.  If you need to cite a different page of the same work, include the page number.  For example:   2 Ibid., 50.

Bibliography:

Hudson, Valerie, N., ed. Culture and Foreign Policy . Boulder: L. Rienner Publishers, 1997.

Twyman, Michael. Lithography 1800-1850 . London: Oxford University Press, 1970.

UNICEF.  Generation in Jeopardy: Children in Central and Eastern Europe and the             Former Soviet Union . Edited by Alexander Zouev. Armonk, NY: M. E. Sharpe, 1999.

Books with Two or More Authors or Editors

1 Russell Keat and John Urry, Social Theory as Science, 2d ed. (London: Routledge and K. Paul, 1982), 196.

1 Toyoma Hitomi, "The Era of Dandy Beauties," in Queer Voices from Japan: First-Person Narratives from Japan's Sexual Minorities,  eds. Mark J. McLelland, Katsuhiko Suganuma, and James Welker ( Lanham, MD: Lexington Books, 2007), 157.

For references with more than three authors, cite the first named author followed by "et al." Cite all the authors in the bibliography.

1 Leonard B. Meyer, et al., The Concept of Style , ed. Berel Lang (Philadelphia: University of Pennsylvania Press, 1979), 56.

2 Keat and Urry, Social Theory as Science , 200.

2 Meyer, et al., The Concept of Style , 90.

Keat, Russell, and John Urry. Social Theory as Science , 2d. ed. London: Routledge and K. Paul, 1982.

Hitomi, Toyoma. "The Era of Dandy Beauties." In Queer Voices from Japan: First-Person Narratives from Japan's Sexual Minorities,  edited by Mark J. McLelland, Katsuhiko Suganuma, and James Welker, 153-165.   Lanham, MD: Lexington Books, 2007.

Meyer, Leonard B., Kendall Walton, Albert Hofstadter, Svetlana Alpers, George Kubler, Richard Wolheim, Monroe Beardsley, Seymour Chatman, Ann Banfield, and Hayden White. The Concept of Style . Edited by Berel Lang.  Philadelphia: University of Pennsylvania Press, 1979.  

Electronic Books

Follow the guidelines for print books, above, but include the collection (if there is one), URL and the date you accessed the material.

1 John Rae, Statement of Some New Principles on the Subject of Political Economy (Boston: Hillard, Gray and Company, 1834), in The Making of the Modern World,   http://galenet.galegroup.com/servlet/MOME?af=RN&ae=U104874605&srchtp=a&ste=14  (accessed June 22, 2009).  

2 Rae, Statement of Some New Principles on the Subject of Political Economy .

Rae, John.  Statement of Some New Principles on the Subject of Political Economy. Boston: Hillard, Gray and Company, 1834. In The Making of the Modern World,   http://galenet.galegroup.com/servlet/MOME?af=RN&ae=U104874605&srchtp=a&ste=14  (accessed June 22, 2009).  

PERIODICAL ARTICLES

For periodical (magazine, journal, newspaper, etc.) articles, include some or all of the following elements in your first footnote or endnote and in your bibliography, in this order:

1. Author; 2. Article title; 3. Periodical title; 4. Volume or Issue number (or both); 5. Publication date; 6. Page numbers.

For online periodicals   , add: 7. URL and date of access; or 8. Database name, URL and date of access. (If available, include database publisher and city of publication.)

For an article available in more than one format (print, online, etc.), cite whichever version you used.

Journal Articles (Print)

1 Lawrence Freedman, "The Changing Roles of Military Conflict," Survival 40, no. 4 (1998): 52.

Here you are citing page 52.  In the bibliography (see below) you would include the full page range: 39-56.

If a journal has continuous pagination within a volume, you do not need to include the issue number:

1 John T. Kirby, "Aristotle on Metaphor," American Journal of Philology 118 (1997): 520.

Subsequent footnotes :

2 Freedman, "The Changing Roles of Military Conflict," 49.   

2 Kirby, "Aristotle on Metaphor," 545.

Freedman, Lawrence. "The Changing Roles of Military Conflict."   Survival 40, no. 4 (1998): 39-56.

Kirby, John T. "Aristotle on Metaphor."  American Journal of Philology 118 (1997): 517-554.  

Journal Articles (Online)

Cite as above, but include the URL and the date of access of the article.

On the Free Web

1 Molly Shea, "Hacking Nostalgia: Super Mario Clouds," Gnovis 9, no. 2 (Spring 2009), http://gnovisjournal.org/journal/hacking-nostalgia-super-mario-clouds  (accessed June 25, 2009).

Through a Subscription Database

1 John T. Kirby, "Aristotle on Metaphor," American Journal of Philology 118, no. 4 (Winter 1997): 524, http://muse.jhu.edu/journals/american_journal_of_philology/v118/118.4.kirby.html  (accessed June 25, 2009).

1 Michael Moon, et al., "Queers in (Single-Family) Space," Assemblage 24 (August 1994): 32, http://www.jstor.org/stable/3171189  (accessed June 25, 2009).

Subsequent Footnotes:

2 Shea, "Hacking Nostalgia."

2 Kirby, "Aristotle on Metaphor," 527. 

2 Moon, "Queers in (Single-Family) Space," 34. 

Shea, Molly. "Hacking Nostalgia: Super Mario Clouds," Gnovis 9, no. 2 (Spring 2009), http://gnovisjournal.org/journal/hacking-nostalgia-super-mario-clouds  (accessed June 25, 2009).

Kirby, John T. "Aristotle on Metaphor," American Journal of Philology 118, no. 4 (Winter 1997): 524, http://muse.jhu.edu/journals/american_journal_of_philology/v118/118.4.kirby.html  (accessed June 25, 2009).

Moon, Michael, Eve Kosofsky Sedgwick, Benjamin Gianni, and Scott Weir. "Queers in (Single-Family) Space." Assemblage 24 (August 1994): 30-7, http://www.jstor.org/stable/3171189  (accessed June 25, 2009).

Magazine Articles (Print)

Monthly or Bimonthly

           1 Paul Goldberger, "Machines for Living: The Architectonic Allure of the Automobile," Architectural Digest, October 1996, 82.

1 Steven Levy and Brad Stone, "Silicon Valley Reboots," Newsweek , March 25, 2002, 45.

          2 Goldberger, "Machines for Living," 82.

          2 Levy and Stone, "Silicon Valley Reboots," 46.

Goldberger, Paul.  "Machines for Living: The Architectonic Allure of the Automobile." Architectural Digest, October 1996.

Levy, Steven, and Brad Stone. "Silicon Valley Reboots." Newsweek , March 25, 2002.

Magazine Articles (Online)

Follow the guidelines for print magazine articles, adding the URL and date accessed.

1 Bill Wyman, "Tony Soprano's Female Trouble," Salon.com, May 19, 2001, http://www.salon.com/2001/05/19/sopranos_final/ (accessed February 13, 2017).

1 Sasha Frere-Jones, "Hip-Hop President." New Yorker , November 24, 2008, http://search.ebscohost.com/login.aspx?direct=true&db=aph&AN=35324426&site=ehost-live (accessed June 26, 2009).

Wyman, Bill. "Tony Soprano's Female Trouble." Salon.com, May 19, 2001, http://www.salon.com/2001/05/19/sopranos_final/ (accessed February 13, 2017).

Frere-Jones, Sasha. "Hip-Hop President." New Yorker , November 24, 2008. http://search.ebscohost.com/login.aspx?direct=true&db=aph&AN=35324426&site=ehost-live (accessed June 26, 2009).

Newspaper Articles

In most cases, you will cite newspaper articles only in notes, not in your bibliography. Follow the general pattern for citing magazine articles, although you may omit page numbers.

        1 Eric Pianin, "Use of Arsenic in Wood Products to End," Washington Post , February 13, 2002, final edition.

        1 Eric Pianin, "Use of Arsenic in Wood Products to End," Washington Post , February 13, 2002, final edition, in LexisNexis Academic (accessed June 27, 2009).

Note: In the example above, there was no stable URL for the article in LexisNexis, so the name of the database was given rather than a URL.

Review Articles

Follow the pattern below for review articles in any kind of periodical.

1 Alanna Nash, "Hit 'Em With a Lizard," review of Basket Case, by Carl Hiassen, New York Times , February 3, 2002, http://proquest.umi.com/pqdweb?did=105338185&sid=2&Fmt=6&clientId=5604&RQT=309&VName=PQD (accessed June 26, 2009).  

1 David Denby, "Killing Joke," review of No Country for Old Men , directed by Ethan and Joel Coen,  New Yorker, February 25, 2008, 72-73, http://search.ebscohost.com/login.aspx?direct=true&db=fah&AN=30033248&site=ehost-live (accessed June 26, 2009). 

Second footnote:

2 Nash, "Hit 'Em With a Lizard."

2 Denby, "Killing Joke."

In most cases, you will be citing something smaller than an entire website. If you are citing an article from a website, for example, follow the guidelines for articles above. You can usually refer to an entire website in running text without including it in your reference list, e.g.: "According to its website, the Financial Accounting Standards Board requires ...".

If you need to cite an entire website in your bibliography, include some or all of the following elements, in this order:

1. Author or editor of the website (if known) 2. Title of the website 3. URL 4. Date of access

Financial Accounting Standards Board .  http://www.fasb.org  (accessed April 29, 2009).

FOR MORE HELP

Following are links to sites that have additional information and further examples:

Turabian Quick Guide (University of Chicago Press)

Chicago Manual of Style Online

RefWorks Once you have created an account, go to Tools/Preview Output Style to see examples of Turabian style.

Purdue's Online Writing Lab (OWL) Excellent source for research, writing and citation tips.

Citing Sources Duke University's guide to citing sources. The site offers comparison citation tables with examples from APA , Chicago , MLA and Turabian for both print and electronic works.

How to Cite Electronic Sources From the Library of Congress. Provides MLA and Turabian examples of citing formats like films, photographs, maps and recorded sound that are accessed electronically.

Uncle Sam: Brief Guide to Citing Government Publications The examples in this excellent guide from the University of Memphis are based on the Chicago Manual of Style and Kate Turabian's Manual .

Free All-in-One Office Suite with PDF Editor

Edit Word, Excel, and PPT for FREE.

Read, edit, and convert PDFs with the powerful PDF toolkit.

Microsoft-like interface, easy to use.

Windows • MacOS • Linux • iOS • Android

Select areas that need to improve

• Didn't match my interface
• Too technical or incomprehensible
• Incorrect operation instructions
• Incomplete instructions on this function

Fields marked * are required please

Please leave your suggestions below

• Quick Tutorials
• Practical Skills

How to Add Footnotes in Word? [For Students]

Being a student can be tough. They say it’s one of the best days of your life, but with all the assignments and thesis work, it can definitely take the fun out of it. To excel as a student, you need to ensure you submit your best work. That means your essays need to be convincing, with all the right citations placed correctly. In this article, I’ll show you  how to add footnotes in Word for students so you can properly cite your sources.

Footnotes in APA, MLA and Chicago Format

You haven't truly completed the format if you haven't added the citations and footnotes in the right way. Citations are a crucial component of academic writing, ensuring you give proper credit to sources and maintain scholarly integrity. Each citation style—APA, MLA, and Chicago—has its own specific rules for citing sources and adding footnotes. This can get complicated, especially when you're trying to meet tight deadlines or juggle multiple assignments. Here's what proper citation and footnote placement looks like when you are aiming to meet your academic standards:

APA format:

In APA format, footnotes are used by inserting superscript numbers in the text that correspond to the footnote numbers. Here's how to format footnotes:

Double-space footnotes.

Indent the first line.

Add a space between the superscript number and the note text.

For example, in a research paper, you might cite a book like this:

Antony Grafton, The Footnote: A Curious History (Cambridge, MA: Harvard University Press, 1999), 221.

And a chapter from a book might be cited like this:

W. Jones and R. Smith, 2010, Photojournalism, 21, p. 122. (Copyright 2007 by Copyright Holder. Reprinted with permission.)

These footnotes include detailed citations, including author names, book titles, publication years, and page numbers.

MLA format:

In MLA format, footnotes are used for citing sources within the text. Here's how to format footnotes:

Place superscript numbers within the text to correspond with the footnote numbers.

Include detailed citation information in the footnote.

Single-space entries, with double-spacing between footnotes.

Chicago Format:

In Chicago style, footnotes are used for citing sources within the text. Here's how to format footnotes:

Separate multiple citations with semicolons.

Ensure consistency in citation style throughout the document.

How to Add Footnotes in Your Essay?

Adding footnotes correctly is incredibly important for academic writing, allowing you to reference sources and add explanations or additional information. To ensure you do this right, follow the steps below, designed to be compatible with various devices. To make sure you can follow along on your mobile, Windows, or Mac, I'll use WPS Office for the demo. It's a free office software that's compatible with all Word document versions and can even convert your papers to PDF without losing format.

1.On the References tab

As we move forward in this tutorial, let's address a common query students encounter when working on projects under strict professorial guidelines: how to add footnotes and endnotes in a Word document. Word simplifies this process. By navigating to the "Reference" tab, you can effortlessly insert footnotes and endnotes in your document.

Step 1: Let's launch WPS Writer, a simplified yet advanced writing software, and open our project where we need to insert footnotes.

Step 2: Now, within our document, place the cursor where you want to add the footnote.

Step 3: Next, the option to insert a footnote is located in the "Reference" tab. So, navigate to the Reference tab and click on "Insert Footnote" in the reference ribbon.

Step 4: A subscript will be added next to the text where you placed the cursor, and you will be directed to the bottom of the page where the footnote will appear.

That's how easily footnotes can be added in WPS Writer for your school projects. Another significant reason for using WPS Writer was its user-friendly interface, making it easy for me as a student. Additionally, it is budget-friendly while providing all the necessary tools.

2.Footnotes formatting

Probably the most important thing to keep in mind is the style requested by the instructor to follow: APA, MLA, or any other. Different styles entail different formatting. In this part, I'll show you how to add footnotes in APA style formatting. So, let's open WPS Writer and delve into formatting our footnotes.

Step 1: The first thing to remember is proper footnote referencing; ensure to follow the citation format when adding it to the footnote.

Step 2: To change the numbering format or starting position of your footnotes, right-click on your footnotes and select "Footnote/Endnote" from the context menu.

Step 3: In the Footnote and Endnote dialog box, select the numbering format according to the style in the "Number Format" field.

Step 4: Using the "Start at" field, you can start numbering your footnotes as desired.

Step 5: In APA style, our footnotes should be double-spaced. So, let's select our footnotes and navigate to the Home tab.

Step 6: In the Home ribbon, click on the "Line Spacing" icon and select "2.0" to change the line spacing to double.

With these easy steps, you'll be creating well-structured and formatted footnotes in no time. WPS Writer lets you concentrate on your writing and leaves the technicalities to the software. With a simple and clean interface and powerful tools that support all student needs, WPS Writer is my preferred choice for my writing needs. Plus, there's no bill at the end of each month just for using a writing software!

Use Word, Excel, and PPT for FREE, No Ads.

Edit PDF files with the powerful PDF toolkit.

Microsoft-like interface. Easy to learn. 100% Compatibility.

Boost your productivity with WPS's abundant free Word, Excel, PPT, and CV templates.

1.How to Revise Your Essay Easily?

When you're tackling a long essay, going through every sentence to ensure correct grammar, spelling, and formatting can be quite the challenge. This task can be especially daunting when you're juggling multiple assignments or working under tight deadlines. Thankfully, you don’t have to worry about any of that because with its WPS AI spell check and AI writer functions, you can automatically scan your essay for spelling errors, grammatical mistakes, and formatting inconsistencies.

The AI spell check feature helps you correct typos and other errors in real time, allowing you to focus on refining your ideas rather than hunting for misplaced commas or incorrect word choices. The AI writer function can also help you refine your writing style, offering suggestions for rewording sentences to make them clearer or more impactful. This combination of automated proofreading and writing assistance saves you time and ensures that your essay maintains a high standard of quality, allowing you to submit your work with confidence.

To ensure your thesis/assignment is error-free, let's utilize the WPS AI Spell Check to proofread your document.

Step 1: Open your document in WPS Writer and ensure the "AI Spell Check" toggle is activated in the status bar.

Step 2: Click on any incorrect word or phrase highlighted with a colored dotted underline in your document.

Step 3: This action will open the WPS AI Check pane on the right side of the screen.

Step 4: You will see all suggestions in the "All Suggestions" tab. To view different suggestions, click on each tab and make the correction.

2.How to Convert Word to PDF without Losing Format

Dealing with your thesis or professional essay requires very careful attention to detail, especially when it comes to proper formatting and final submissions. However, converting your essay to PDFwhich is a crucial step for academic or professional submissions—can be a source of frustration, particularly when using Microsoft Word 365, where the process might disrupt your APA or MLA formatting.

Unexpected changes in margins, font sizes, or spacing can turn a polished document into a chaotic one. WPS Office is really helpful in regard to allowing you to convert your essay to PDF while preserving your original formatting. Unlike Word, WPS Office ensures that your APA or MLA style remains intact, with no unexpected shifts in headers, footnotes, or page layout. With just a few clicks, you can convert your document to PDF and be confident that it looks exactly as intended, avoiding last-minute adjustments.

Here is how WPS Writer can help you convert your work with footnotes into PDF in a few easy steps:

Step 1: Open your Word document in WPS Office. Look for the Menu button at the top left corner of the screen.

Step 2: Click on "Save as" in the menu. Then choose "Other formats" .

Step 3: In the options, pick "PDF" from the list. Click "Save" to change your document to a PDF file.

FAQs about adding Footnotes in Word

1. how do you insert multiple footnotes in word.

Here's a straightforward guide on how to insert multiple footnotes in Word:

Step 1: Position your cursor in the main text where you want the footnote number to be displayed.

Step 2: Navigate to the References tab located in the ribbon toolbar.

Step 3: Click on the "Insert Footnote" option. This action will direct you to the bottom of the page, where you can input your footnote text.

Step 4: Enter the content of your footnote according to the required style.

Step 5: Repeat the process for each additional footnote needed. Word will automatically adjust the numbering for you.

2. How do you put two footnotes in one sentence?

According to the Chicago Manual of Style (CMOS), if you have more than one citation relating to the same concept or idea, all relevant citations can be included in a single footnote, each separated by a semi-colon. This method ensures clarity and organization in your references

3. How do you footnote something already footnoted?

Place the Cursor: Click where you want to insert the new footnote.

Insert a Footnote: Use the "Insert Footnote" option, typically in the "References" or "Insert" tab.

Add Reference: In the new footnote, refer to the existing footnote. You can quote, summarize, or mention the original footnote number (e.g., "See footnote 1" ).

Check Footnote Numbering: Ensure that the numbering is correct. Adjust if needed.

Proofread: Confirm that the new footnote is clear and that the document's structure remains intact.

Stop Struggling with Footnotes: Here's the Word Hack You Need

Your essay isn't complete without proper citations, which usually come in the form of footnotes. Once you learn how to add footnotes in Word for students, it's crucial to double-check them to ensure they're correctly formatted and contain all the necessary information. This step is especially important to maintain academic integrity and avoid plagiarism. WPS AI can be a tremendous help in this regard. It can scan your document for errors, suggesting corrections if you've missed a citation or formatted something incorrectly. With WPS AI's assistance, you can confidently complete your footnotes, knowing that you've referenced your sources accurately and consistently. So do yourself a favor and download WPS Writer to make your academic life easier.

• 1. How to Add Page Numbers in Word for Your Papers? [For Students]
• 2. How to insert footnotes in word
• 3. How to Convert PDF to Word for Students
• 4. How to Add a Line in Word [For Students]
• 5. How to Check Word Count for Your Essays in Word [For Students]
• 6. How to Remove Section Breaks in Word? [For Students]

15 years of office industry experience, tech lover and copywriter. Follow me for product reviews, comparisons, and recommendations for new apps and software.

Purdue Online Writing Lab Purdue OWL® College of Liberal Arts

Welcome to the Purdue Online Writing Lab

Welcome to the Purdue OWL

This page is brought to you by the OWL at Purdue University. When printing this page, you must include the entire legal notice.

Copyright ©1995-2018 by The Writing Lab & The OWL at Purdue and Purdue University. All rights reserved. This material may not be published, reproduced, broadcast, rewritten, or redistributed without permission. Use of this site constitutes acceptance of our terms and conditions of fair use.

The Online Writing Lab at Purdue University houses writing resources and instructional material, and we provide these as a free service of the Writing Lab at Purdue. Students, members of the community, and users worldwide will find information to assist with many writing projects. Teachers and trainers may use this material for in-class and out-of-class instruction.

The Purdue On-Campus Writing Lab and Purdue Online Writing Lab assist clients in their development as writers—no matter what their skill level—with on-campus consultations, online participation, and community engagement. The Purdue Writing Lab serves the Purdue, West Lafayette, campus and coordinates with local literacy initiatives. The Purdue OWL offers global support through online reference materials and services.

A Message From the Assistant Director of Content Development 

The Purdue OWL® is committed to supporting  students, instructors, and writers by offering a wide range of resources that are developed and revised with them in mind. To do this, the OWL team is always exploring possibilties for a better design, allowing accessibility and user experience to guide our process. As the OWL undergoes some changes, we welcome your feedback and suggestions by email at any time.

Please don't hesitate to contact us via our contact page  if you have any questions or comments.

All the best,

Social Media

Facebook twitter.

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• My Account Login
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 03 June 2024

Applying large language models for automated essay scoring for non-native Japanese

• Wenchao Li 1 &
• Haitao Liu 2  

Humanities and Social Sciences Communications volume  11 , Article number:  723 ( 2024 ) Cite this article

12 Accesses

1 Altmetric

Metrics details

• Language and linguistics

Recent advancements in artificial intelligence (AI) have led to an increased use of large language models (LLMs) for language assessment tasks such as automated essay scoring (AES), automated listening tests, and automated oral proficiency assessments. The application of LLMs for AES in the context of non-native Japanese, however, remains limited. This study explores the potential of LLM-based AES by comparing the efficiency of different models, i.e. two conventional machine training technology-based methods (Jess and JWriter), two LLMs (GPT and BERT), and one Japanese local LLM (Open-Calm large model). To conduct the evaluation, a dataset consisting of 1400 story-writing scripts authored by learners with 12 different first languages was used. Statistical analysis revealed that GPT-4 outperforms Jess and JWriter, BERT, and the Japanese language-specific trained Open-Calm large model in terms of annotation accuracy and predicting learning levels. Furthermore, by comparing 18 different models that utilize various prompts, the study emphasized the significance of prompts in achieving accurate and reliable evaluations using LLMs.

Similar content being viewed by others

Scoring method of English composition integrating deep learning in higher vocational colleges

ChatGPT-3.5 as writing assistance in students’ essays

Detecting contract cheating through linguistic fingerprint

Conventional machine learning technology in aes.

AES has experienced significant growth with the advancement of machine learning technologies in recent decades. In the earlier stages of AES development, conventional machine learning-based approaches were commonly used. These approaches involved the following procedures: a) feeding the machine with a dataset. In this step, a dataset of essays is provided to the machine learning system. The dataset serves as the basis for training the model and establishing patterns and correlations between linguistic features and human ratings. b) the machine learning model is trained using linguistic features that best represent human ratings and can effectively discriminate learners’ writing proficiency. These features include lexical richness (Lu, 2012 ; Kyle and Crossley, 2015 ; Kyle et al. 2021 ), syntactic complexity (Lu, 2010 ; Liu, 2008 ), text cohesion (Crossley and McNamara, 2016 ), and among others. Conventional machine learning approaches in AES require human intervention, such as manual correction and annotation of essays. This human involvement was necessary to create a labeled dataset for training the model. Several AES systems have been developed using conventional machine learning technologies. These include the Intelligent Essay Assessor (Landauer et al. 2003 ), the e-rater engine by Educational Testing Service (Attali and Burstein, 2006 ; Burstein, 2003 ), MyAccess with the InterlliMetric scoring engine by Vantage Learning (Elliot, 2003 ), and the Bayesian Essay Test Scoring system (Rudner and Liang, 2002 ). These systems have played a significant role in automating the essay scoring process and providing quick and consistent feedback to learners. However, as touched upon earlier, conventional machine learning approaches rely on predetermined linguistic features and often require manual intervention, making them less flexible and potentially limiting their generalizability to different contexts.

In the context of the Japanese language, conventional machine learning-incorporated AES tools include Jess (Ishioka and Kameda, 2006 ) and JWriter (Lee and Hasebe, 2017 ). Jess assesses essays by deducting points from the perfect score, utilizing the Mainichi Daily News newspaper as a database. The evaluation criteria employed by Jess encompass various aspects, such as rhetorical elements (e.g., reading comprehension, vocabulary diversity, percentage of complex words, and percentage of passive sentences), organizational structures (e.g., forward and reverse connection structures), and content analysis (e.g., latent semantic indexing). JWriter employs linear regression analysis to assign weights to various measurement indices, such as average sentence length and total number of characters. These weights are then combined to derive the overall score. A pilot study involving the Jess model was conducted on 1320 essays at different proficiency levels, including primary, intermediate, and advanced. However, the results indicated that the Jess model failed to significantly distinguish between these essay levels. Out of the 16 measures used, four measures, namely median sentence length, median clause length, median number of phrases, and maximum number of phrases, did not show statistically significant differences between the levels. Additionally, two measures exhibited between-level differences but lacked linear progression: the number of attributives declined words and the Kanji/kana ratio. On the other hand, the remaining measures, including maximum sentence length, maximum clause length, number of attributive conjugated words, maximum number of consecutive infinitive forms, maximum number of conjunctive-particle clauses, k characteristic value, percentage of big words, and percentage of passive sentences, demonstrated statistically significant between-level differences and displayed linear progression.

Both Jess and JWriter exhibit notable limitations, including the manual selection of feature parameters and weights, which can introduce biases into the scoring process. The reliance on human annotators to label non-native language essays also introduces potential noise and variability in the scoring. Furthermore, an important concern is the possibility of system manipulation and cheating by learners who are aware of the regression equation utilized by the models (Hirao et al. 2020 ). These limitations emphasize the need for further advancements in AES systems to address these challenges.

Deep learning technology in AES

Deep learning has emerged as one of the approaches for improving the accuracy and effectiveness of AES. Deep learning-based AES methods utilize artificial neural networks that mimic the human brain’s functioning through layered algorithms and computational units. Unlike conventional machine learning, deep learning autonomously learns from the environment and past errors without human intervention. This enables deep learning models to establish nonlinear correlations, resulting in higher accuracy. Recent advancements in deep learning have led to the development of transformers, which are particularly effective in learning text representations. Noteworthy examples include bidirectional encoder representations from transformers (BERT) (Devlin et al. 2019 ) and the generative pretrained transformer (GPT) (OpenAI).

BERT is a linguistic representation model that utilizes a transformer architecture and is trained on two tasks: masked linguistic modeling and next-sentence prediction (Hirao et al. 2020 ; Vaswani et al. 2017 ). In the context of AES, BERT follows specific procedures, as illustrated in Fig. 1 : (a) the tokenized prompts and essays are taken as input; (b) special tokens, such as [CLS] and [SEP], are added to mark the beginning and separation of prompts and essays; (c) the transformer encoder processes the prompt and essay sequences, resulting in hidden layer sequences; (d) the hidden layers corresponding to the [CLS] tokens (T[CLS]) represent distributed representations of the prompts and essays; and (e) a multilayer perceptron uses these distributed representations as input to obtain the final score (Hirao et al. 2020 ).

AES system with BERT (Hirao et al. 2020 ).

The training of BERT using a substantial amount of sentence data through the Masked Language Model (MLM) allows it to capture contextual information within the hidden layers. Consequently, BERT is expected to be capable of identifying artificial essays as invalid and assigning them lower scores (Mizumoto and Eguchi, 2023 ). In the context of AES for nonnative Japanese learners, Hirao et al. ( 2020 ) combined the long short-term memory (LSTM) model proposed by Hochreiter and Schmidhuber ( 1997 ) with BERT to develop a tailored automated Essay Scoring System. The findings of their study revealed that the BERT model outperformed both the conventional machine learning approach utilizing character-type features such as “kanji” and “hiragana”, as well as the standalone LSTM model. Takeuchi et al. ( 2021 ) presented an approach to Japanese AES that eliminates the requirement for pre-scored essays by relying solely on reference texts or a model answer for the essay task. They investigated multiple similarity evaluation methods, including frequency of morphemes, idf values calculated on Wikipedia, LSI, LDA, word-embedding vectors, and document vectors produced by BERT. The experimental findings revealed that the method utilizing the frequency of morphemes with idf values exhibited the strongest correlation with human-annotated scores across different essay tasks. The utilization of BERT in AES encounters several limitations. Firstly, essays often exceed the model’s maximum length limit. Second, only score labels are available for training, which restricts access to additional information.

Mizumoto and Eguchi ( 2023 ) were pioneers in employing the GPT model for AES in non-native English writing. Their study focused on evaluating the accuracy and reliability of AES using the GPT-3 text-davinci-003 model, analyzing a dataset of 12,100 essays from the corpus of nonnative written English (TOEFL11). The findings indicated that AES utilizing the GPT-3 model exhibited a certain degree of accuracy and reliability. They suggest that GPT-3-based AES systems hold the potential to provide support for human ratings. However, applying GPT model to AES presents a unique natural language processing (NLP) task that involves considerations such as nonnative language proficiency, the influence of the learner’s first language on the output in the target language, and identifying linguistic features that best indicate writing quality in a specific language. These linguistic features may differ morphologically or syntactically from those present in the learners’ first language, as observed in (1)–(3).

我-送了-他-一本-书

Wǒ-sòngle-tā-yī běn-shū

1 sg .-give. past- him-one .cl- book

“I gave him a book.”

Agglutinative

彼-に-本-を-あげ-まし-た

Kare-ni-hon-o-age-mashi-ta

3 sg .- dat -hon- acc- give.honorification. past

Inflectional

give, give-s, gave, given, giving

Additionally, the morphological agglutination and subject-object-verb (SOV) order in Japanese, along with its idiomatic expressions, pose additional challenges for applying language models in AES tasks (4).

足-が 棒-に なり-ました

Ashi-ga bo-ni nar-mashita

leg- nom stick- dat become- past

“My leg became like a stick (I am extremely tired).”

The example sentence provided demonstrates the morpho-syntactic structure of Japanese and the presence of an idiomatic expression. In this sentence, the verb “なる” (naru), meaning “to become”, appears at the end of the sentence. The verb stem “なり” (nari) is attached with morphemes indicating honorification (“ます” - mashu) and tense (“た” - ta), showcasing agglutination. While the sentence can be literally translated as “my leg became like a stick”, it carries an idiomatic interpretation that implies “I am extremely tired”.

To overcome this issue, CyberAgent Inc. ( 2023 ) has developed the Open-Calm series of language models specifically designed for Japanese. Open-Calm consists of pre-trained models available in various sizes, such as Small, Medium, Large, and 7b. Figure 2 depicts the fundamental structure of the Open-Calm model. A key feature of this architecture is the incorporation of the Lora Adapter and GPT-NeoX frameworks, which can enhance its language processing capabilities.

GPT-NeoX Model Architecture (Okgetheng and Takeuchi 2024 ).

In a recent study conducted by Okgetheng and Takeuchi ( 2024 ), they assessed the efficacy of Open-Calm language models in grading Japanese essays. The research utilized a dataset of approximately 300 essays, which were annotated by native Japanese educators. The findings of the study demonstrate the considerable potential of Open-Calm language models in automated Japanese essay scoring. Specifically, among the Open-Calm family, the Open-Calm Large model (referred to as OCLL) exhibited the highest performance. However, it is important to note that, as of the current date, the Open-Calm Large model does not offer public access to its server. Consequently, users are required to independently deploy and operate the environment for OCLL. In order to utilize OCLL, users must have a PC equipped with an NVIDIA GeForce RTX 3060 (8 or 12 GB VRAM).

In summary, while the potential of LLMs in automated scoring of nonnative Japanese essays has been demonstrated in two studies—BERT-driven AES (Hirao et al. 2020 ) and OCLL-based AES (Okgetheng and Takeuchi, 2024 )—the number of research efforts in this area remains limited.

Another significant challenge in applying LLMs to AES lies in prompt engineering and ensuring its reliability and effectiveness (Brown et al. 2020 ; Rae et al. 2021 ; Zhang et al. 2021 ). Various prompting strategies have been proposed, such as the zero-shot chain of thought (CoT) approach (Kojima et al. 2022 ), which involves manually crafting diverse and effective examples. However, manual efforts can lead to mistakes. To address this, Zhang et al. ( 2021 ) introduced an automatic CoT prompting method called Auto-CoT, which demonstrates matching or superior performance compared to the CoT paradigm. Another prompt framework is trees of thoughts, enabling a model to self-evaluate its progress at intermediate stages of problem-solving through deliberate reasoning (Yao et al. 2023 ).

Beyond linguistic studies, there has been a noticeable increase in the number of foreign workers in Japan and Japanese learners worldwide (Ministry of Health, Labor, and Welfare of Japan, 2022 ; Japan Foundation, 2021 ). However, existing assessment methods, such as the Japanese Language Proficiency Test (JLPT), J-CAT, and TTBJ Footnote 1 , primarily focus on reading, listening, vocabulary, and grammar skills, neglecting the evaluation of writing proficiency. As the number of workers and language learners continues to grow, there is a rising demand for an efficient AES system that can reduce costs and time for raters and be utilized for employment, examinations, and self-study purposes.

This study aims to explore the potential of LLM-based AES by comparing the effectiveness of five models: two LLMs (GPT Footnote 2 and BERT), one Japanese local LLM (OCLL), and two conventional machine learning-based methods (linguistic feature-based scoring tools - Jess and JWriter).

The research questions addressed in this study are as follows:

To what extent do the LLM-driven AES and linguistic feature-based AES, when used as automated tools to support human rating, accurately reflect test takers’ actual performance?

What influence does the prompt have on the accuracy and performance of LLM-based AES methods?

The subsequent sections of the manuscript cover the methodology, including the assessment measures for nonnative Japanese writing proficiency, criteria for prompts, and the dataset. The evaluation section focuses on the analysis of annotations and rating scores generated by LLM-driven and linguistic feature-based AES methods.

Methodology

The dataset utilized in this study was obtained from the International Corpus of Japanese as a Second Language (I-JAS) Footnote 3 . This corpus consisted of 1000 participants who represented 12 different first languages. For the study, the participants were given a story-writing task on a personal computer. They were required to write two stories based on the 4-panel illustrations titled “Picnic” and “The key” (see Appendix A). Background information for the participants was provided by the corpus, including their Japanese language proficiency levels assessed through two online tests: J-CAT and SPOT. These tests evaluated their reading, listening, vocabulary, and grammar abilities. The learners’ proficiency levels were categorized into six levels aligned with the Common European Framework of Reference for Languages (CEFR) and the Reference Framework for Japanese Language Education (RFJLE): A1, A2, B1, B2, C1, and C2. According to Lee et al. ( 2015 ), there is a high level of agreement (r = 0.86) between the J-CAT and SPOT assessments, indicating that the proficiency certifications provided by J-CAT are consistent with those of SPOT. However, it is important to note that the scores of J-CAT and SPOT do not have a one-to-one correspondence. In this study, the J-CAT scores were used as a benchmark to differentiate learners of different proficiency levels. A total of 1400 essays were utilized, representing the beginner (aligned with A1), A2, B1, B2, C1, and C2 levels based on the J-CAT scores. Table 1 provides information about the learners’ proficiency levels and their corresponding J-CAT and SPOT scores.

A dataset comprising a total of 1400 essays from the story writing tasks was collected. Among these, 714 essays were utilized to evaluate the reliability of the LLM-based AES method, while the remaining 686 essays were designated as development data to assess the LLM-based AES’s capability to distinguish participants with varying proficiency levels. The GPT 4 API was used in this study. A detailed explanation of the prompt-assessment criteria is provided in Section Prompt . All essays were sent to the model for measurement and scoring.

Measures of writing proficiency for nonnative Japanese

Japanese exhibits a morphologically agglutinative structure where morphemes are attached to the word stem to convey grammatical functions such as tense, aspect, voice, and honorifics, e.g. (5).

食べ-させ-られ-まし-た-か

tabe-sase-rare-mashi-ta-ka

[eat (stem)-causative-passive voice-honorification-tense. past-question marker]

Japanese employs nine case particles to indicate grammatical functions: the nominative case particle が (ga), the accusative case particle を (o), the genitive case particle の (no), the dative case particle に (ni), the locative/instrumental case particle で (de), the ablative case particle から (kara), the directional case particle へ (e), and the comitative case particle と (to). The agglutinative nature of the language, combined with the case particle system, provides an efficient means of distinguishing between active and passive voice, either through morphemes or case particles, e.g. 食べる taberu “eat concusive . ” (active voice); 食べられる taberareru “eat concusive . ” (passive voice). In the active voice, “パン を 食べる” (pan o taberu) translates to “to eat bread”. On the other hand, in the passive voice, it becomes “パン が 食べられた” (pan ga taberareta), which means “(the) bread was eaten”. Additionally, it is important to note that different conjugations of the same lemma are considered as one type in order to ensure a comprehensive assessment of the language features. For example, e.g., 食べる taberu “eat concusive . ”; 食べている tabeteiru “eat progress .”; 食べた tabeta “eat past . ” as one type.

To incorporate these features, previous research (Suzuki, 1999 ; Watanabe et al. 1988 ; Ishioka, 2001 ; Ishioka and Kameda, 2006 ; Hirao et al. 2020 ) has identified complexity, fluency, and accuracy as crucial factors for evaluating writing quality. These criteria are assessed through various aspects, including lexical richness (lexical density, diversity, and sophistication), syntactic complexity, and cohesion (Kyle et al. 2021 ; Mizumoto and Eguchi, 2023 ; Ure, 1971 ; Halliday, 1985 ; Barkaoui and Hadidi, 2020 ; Zenker and Kyle, 2021 ; Kim et al. 2018 ; Lu, 2017 ; Ortega, 2015 ). Therefore, this study proposes five scoring categories: lexical richness, syntactic complexity, cohesion, content elaboration, and grammatical accuracy. A total of 16 measures were employed to capture these categories. The calculation process and specific details of these measures can be found in Table 2 .

T-unit, first introduced by Hunt ( 1966 ), is a measure used for evaluating speech and composition. It serves as an indicator of syntactic development and represents the shortest units into which a piece of discourse can be divided without leaving any sentence fragments. In the context of Japanese language assessment, Sakoda and Hosoi ( 2020 ) utilized T-unit as the basic unit to assess the accuracy and complexity of Japanese learners’ speaking and storytelling. The calculation of T-units in Japanese follows the following principles:

A single main clause constitutes 1 T-unit, regardless of the presence or absence of dependent clauses, e.g. (6).

ケンとマリはピクニックに行きました (main clause): 1 T-unit.

If a sentence contains a main clause along with subclauses, each subclause is considered part of the same T-unit, e.g. (7).

天気が良かった の で (subclause)、ケンとマリはピクニックに行きました (main clause): 1 T-unit.

In the case of coordinate clauses, where multiple clauses are connected, each coordinated clause is counted separately. Thus, a sentence with coordinate clauses may have 2 T-units or more, e.g. (8).

ケンは地図で場所を探して (coordinate clause)、マリはサンドイッチを作りました (coordinate clause): 2 T-units.

Lexical diversity refers to the range of words used within a text (Engber, 1995 ; Kyle et al. 2021 ) and is considered a useful measure of the breadth of vocabulary in L n production (Jarvis, 2013a , 2013b ).

The type/token ratio (TTR) is widely recognized as a straightforward measure for calculating lexical diversity and has been employed in numerous studies. These studies have demonstrated a strong correlation between TTR and other methods of measuring lexical diversity (e.g., Bentz et al. 2016 ; Čech and Miroslav, 2018 ; Çöltekin and Taraka, 2018 ). TTR is computed by considering both the number of unique words (types) and the total number of words (tokens) in a given text. Given that the length of learners’ writing texts can vary, this study employs the moving average type-token ratio (MATTR) to mitigate the influence of text length. MATTR is calculated using a 50-word moving window. Initially, a TTR is determined for words 1–50 in an essay, followed by words 2–51, 3–52, and so on until the end of the essay is reached (Díez-Ortega and Kyle, 2023 ). The final MATTR scores were obtained by averaging the TTR scores for all 50-word windows. The following formula was employed to derive MATTR:

\({\rm{MATTR}}({\rm{W}})=\frac{{\sum }_{{\rm{i}}=1}^{{\rm{N}}-{\rm{W}}+1}{{\rm{F}}}_{{\rm{i}}}}{{\rm{W}}({\rm{N}}-{\rm{W}}+1)}\)

Here, N refers to the number of tokens in the corpus. W is the randomly selected token size (W < N). \({F}_{i}\) is the number of types in each window. The \({\rm{MATTR}}({\rm{W}})\) is the mean of a series of type-token ratios (TTRs) based on the word form for all windows. It is expected that individuals with higher language proficiency will produce texts with greater lexical diversity, as indicated by higher MATTR scores.

Lexical density was captured by the ratio of the number of lexical words to the total number of words (Lu, 2012 ). Lexical sophistication refers to the utilization of advanced vocabulary, often evaluated through word frequency indices (Crossley et al. 2013 ; Haberman, 2008 ; Kyle and Crossley, 2015 ; Laufer and Nation, 1995 ; Lu, 2012 ; Read, 2000 ). In line of writing, lexical sophistication can be interpreted as vocabulary breadth, which entails the appropriate usage of vocabulary items across various lexicon-grammatical contexts and registers (Garner et al. 2019 ; Kim et al. 2018 ; Kyle et al. 2018 ). In Japanese specifically, words are considered lexically sophisticated if they are not included in the “Japanese Education Vocabulary List Ver 1.0”. Footnote 4 Consequently, lexical sophistication was calculated by determining the number of sophisticated word types relative to the total number of words per essay. Furthermore, it has been suggested that, in Japanese writing, sentences should ideally have a length of no more than 40 to 50 characters, as this promotes readability. Therefore, the median and maximum sentence length can be considered as useful indices for assessment (Ishioka and Kameda, 2006 ).

Syntactic complexity was assessed based on several measures, including the mean length of clauses, verb phrases per T-unit, clauses per T-unit, dependent clauses per T-unit, complex nominals per clause, adverbial clauses per clause, coordinate phrases per clause, and mean dependency distance (MDD). The MDD reflects the distance between the governor and dependent positions in a sentence. A larger dependency distance indicates a higher cognitive load and greater complexity in syntactic processing (Liu, 2008 ; Liu et al. 2017 ). The MDD has been established as an efficient metric for measuring syntactic complexity (Jiang, Quyang, and Liu, 2019 ; Li and Yan, 2021 ). To calculate the MDD, the position numbers of the governor and dependent are subtracted, assuming that words in a sentence are assigned in a linear order, such as W1 … Wi … Wn. In any dependency relationship between words Wa and Wb, Wa is the governor and Wb is the dependent. The MDD of the entire sentence was obtained by taking the absolute value of governor – dependent:

MDD = \(\frac{1}{n}{\sum }_{i=1}^{n}|{\rm{D}}{{\rm{D}}}_{i}|\)

In this formula, \(n\) represents the number of words in the sentence, and \({DD}i\) is the dependency distance of the \({i}^{{th}}\) dependency relationship of a sentence. Building on this, the annotation of sentence ‘Mary-ga-John-ni-keshigomu-o-watashita was [Mary- top -John- dat -eraser- acc -give- past] ’. The sentence’s MDD would be 2. Table 3 provides the CSV file as a prompt for GPT 4.

Cohesion (semantic similarity) and content elaboration aim to capture the ideas presented in test taker’s essays. Cohesion was assessed using three measures: Synonym overlap/paragraph (topic), Synonym overlap/paragraph (keywords), and word2vec cosine similarity. Content elaboration and development were measured as the number of metadiscourse markers (type)/number of words. To capture content closely, this study proposed a novel-distance based representation, by encoding the cosine distance between the essay (by learner) and essay task’s (topic and keyword) i -vectors. The learner’s essay is decoded into a word sequence, and aligned to the essay task’ topic and keyword for log-likelihood measurement. The cosine distance reveals the content elaboration score in the leaners’ essay. The mathematical equation of cosine similarity between target-reference vectors is shown in (11), assuming there are i essays and ( L i , …. L n ) and ( N i , …. N n ) are the vectors representing the learner and task’s topic and keyword respectively. The content elaboration distance between L i and N i was calculated as follows:

\(\cos \left(\theta \right)=\frac{{\rm{L}}\,\cdot\, {\rm{N}}}{\left|{\rm{L}}\right|{\rm{|N|}}}=\frac{\mathop{\sum }\nolimits_{i=1}^{n}{L}_{i}{N}_{i}}{\sqrt{\mathop{\sum }\nolimits_{i=1}^{n}{L}_{i}^{2}}\sqrt{\mathop{\sum }\nolimits_{i=1}^{n}{N}_{i}^{2}}}\)

A high similarity value indicates a low difference between the two recognition outcomes, which in turn suggests a high level of proficiency in content elaboration.

To evaluate the effectiveness of the proposed measures in distinguishing different proficiency levels among nonnative Japanese speakers’ writing, we conducted a multi-faceted Rasch measurement analysis (Linacre, 1994 ). This approach applies measurement models to thoroughly analyze various factors that can influence test outcomes, including test takers’ proficiency, item difficulty, and rater severity, among others. The underlying principles and functionality of multi-faceted Rasch measurement are illustrated in (12).

\(\log \left(\frac{{P}_{{nijk}}}{{P}_{{nij}(k-1)}}\right)={B}_{n}-{D}_{i}-{C}_{j}-{F}_{k}\)

(12) defines the logarithmic transformation of the probability ratio ( P nijk /P nij(k-1) )) as a function of multiple parameters. Here, n represents the test taker, i denotes a writing proficiency measure, j corresponds to the human rater, and k represents the proficiency score. The parameter B n signifies the proficiency level of test taker n (where n ranges from 1 to N). D j represents the difficulty parameter of test item i (where i ranges from 1 to L), while C j represents the severity of rater j (where j ranges from 1 to J). Additionally, F k represents the step difficulty for a test taker to move from score ‘k-1’ to k . P nijk refers to the probability of rater j assigning score k to test taker n for test item i . P nij(k-1) represents the likelihood of test taker n being assigned score ‘k-1’ by rater j for test item i . Each facet within the test is treated as an independent parameter and estimated within the same reference framework. To evaluate the consistency of scores obtained through both human and computer analysis, we utilized the Infit mean-square statistic. This statistic is a chi-square measure divided by the degrees of freedom and is weighted with information. It demonstrates higher sensitivity to unexpected patterns in responses to items near a person’s proficiency level (Linacre, 2002 ). Fit statistics are assessed based on predefined thresholds for acceptable fit. For the Infit MNSQ, which has a mean of 1.00, different thresholds have been suggested. Some propose stricter thresholds ranging from 0.7 to 1.3 (Bond et al. 2021 ), while others suggest more lenient thresholds ranging from 0.5 to 1.5 (Eckes, 2009 ). In this study, we adopted the criterion of 0.70–1.30 for the Infit MNSQ.

Moving forward, we can now proceed to assess the effectiveness of the 16 proposed measures based on five criteria for accurately distinguishing various levels of writing proficiency among non-native Japanese speakers. To conduct this evaluation, we utilized the development dataset from the I-JAS corpus, as described in Section Dataset . Table 4 provides a measurement report that presents the performance details of the 14 metrics under consideration. The measure separation was found to be 4.02, indicating a clear differentiation among the measures. The reliability index for the measure separation was 0.891, suggesting consistency in the measurement. Similarly, the person separation reliability index was 0.802, indicating the accuracy of the assessment in distinguishing between individuals. All 16 measures demonstrated Infit mean squares within a reasonable range, ranging from 0.76 to 1.28. The Synonym overlap/paragraph (topic) measure exhibited a relatively high outfit mean square of 1.46, although the Infit mean square falls within an acceptable range. The standard error for the measures ranged from 0.13 to 0.28, indicating the precision of the estimates.

Table 5 further illustrated the weights assigned to different linguistic measures for score prediction, with higher weights indicating stronger correlations between those measures and higher scores. Specifically, the following measures exhibited higher weights compared to others: moving average type token ratio per essay has a weight of 0.0391. Mean dependency distance had a weight of 0.0388. Mean length of clause, calculated by dividing the number of words by the number of clauses, had a weight of 0.0374. Complex nominals per T-unit, calculated by dividing the number of complex nominals by the number of T-units, had a weight of 0.0379. Coordinate phrases rate, calculated by dividing the number of coordinate phrases by the number of clauses, had a weight of 0.0325. Grammatical error rate, representing the number of errors per essay, had a weight of 0.0322.

Criteria (output indicator)

The criteria used to evaluate the writing ability in this study were based on CEFR, which follows a six-point scale ranging from A1 to C2. To assess the quality of Japanese writing, the scoring criteria from Table 6 were utilized. These criteria were derived from the IELTS writing standards and served as assessment guidelines and prompts for the written output.

A prompt is a question or detailed instruction that is provided to the model to obtain a proper response. After several pilot experiments, we decided to provide the measures (Section Measures of writing proficiency for nonnative Japanese ) as the input prompt and use the criteria (Section Criteria (output indicator) ) as the output indicator. Regarding the prompt language, considering that the LLM was tasked with rating Japanese essays, would prompt in Japanese works better Footnote 5 ? We conducted experiments comparing the performance of GPT-4 using both English and Japanese prompts. Additionally, we utilized the Japanese local model OCLL with Japanese prompts. Multiple trials were conducted using the same sample. Regardless of the prompt language used, we consistently obtained the same grading results with GPT-4, which assigned a grade of B1 to the writing sample. This suggested that GPT-4 is reliable and capable of producing consistent ratings regardless of the prompt language. On the other hand, when we used Japanese prompts with the Japanese local model “OCLL”, we encountered inconsistent grading results. Out of 10 attempts with OCLL, only 6 yielded consistent grading results (B1), while the remaining 4 showed different outcomes, including A1 and B2 grades. These findings indicated that the language of the prompt was not the determining factor for reliable AES. Instead, the size of the training data and the model parameters played crucial roles in achieving consistent and reliable AES results for the language model.

The following is the utilized prompt, which details all measures and requires the LLM to score the essays using holistic and trait scores.

Please evaluate Japanese essays written by Japanese learners and assign a score to each essay on a six-point scale, ranging from A1, A2, B1, B2, C1 to C2. Additionally, please provide trait scores and display the calculation process for each trait score. The scoring should be based on the following criteria:

Moving average type-token ratio.

Number of lexical words (token) divided by the total number of words per essay.

Number of sophisticated word types divided by the total number of words per essay.

Mean length of clause.

Verb phrases per T-unit.

Clauses per T-unit.

Dependent clauses per T-unit.

Complex nominals per clause.

Adverbial clauses per clause.

Coordinate phrases per clause.

Mean dependency distance.

Synonym overlap paragraph (topic and keywords).

Word2vec cosine similarity.

Connectives per essay.

Conjunctions per essay.

Number of metadiscourse markers (types) divided by the total number of words.

Number of errors per essay.

Japanese essay text

出かける前に二人が地図を見ている間に、サンドイッチを入れたバスケットに犬が入ってしまいました。それに気づかずに二人は楽しそうに出かけて行きました。やがて突然犬がバスケットから飛び出し、二人は驚きました。バスケット の 中を見ると、食べ物はすべて犬に食べられていて、二人は困ってしまいました。(ID_JJJ01_SW1)

The score of the example above was B1. Figure 3 provides an example of holistic and trait scores provided by GPT-4 (with a prompt indicating all measures) via Bing Footnote 6 .

Example of GPT-4 AES and feedback (with a prompt indicating all measures).

Statistical analysis

The aim of this study is to investigate the potential use of LLM for nonnative Japanese AES. It seeks to compare the scoring outcomes obtained from feature-based AES tools, which rely on conventional machine learning technology (i.e. Jess, JWriter), with those generated by AI-driven AES tools utilizing deep learning technology (BERT, GPT, OCLL). To assess the reliability of a computer-assisted annotation tool, the study initially established human-human agreement as the benchmark measure. Subsequently, the performance of the LLM-based method was evaluated by comparing it to human-human agreement.

To assess annotation agreement, the study employed standard measures such as precision, recall, and F-score (Brants 2000 ; Lu 2010 ), along with the quadratically weighted kappa (QWK) to evaluate the consistency and agreement in the annotation process. Assume A and B represent human annotators. When comparing the annotations of the two annotators, the following results are obtained. The evaluation of precision, recall, and F-score metrics was illustrated in equations (13) to (15).

\({\rm{Recall}}(A,B)=\frac{{\rm{Number}}\,{\rm{of}}\,{\rm{identical}}\,{\rm{nodes}}\,{\rm{in}}\,A\,{\rm{and}}\,B}{{\rm{Number}}\,{\rm{of}}\,{\rm{nodes}}\,{\rm{in}}\,A}\)

\({\rm{Precision}}(A,\,B)=\frac{{\rm{Number}}\,{\rm{of}}\,{\rm{identical}}\,{\rm{nodes}}\,{\rm{in}}\,A\,{\rm{and}}\,B}{{\rm{Number}}\,{\rm{of}}\,{\rm{nodes}}\,{\rm{in}}\,B}\)

The F-score is the harmonic mean of recall and precision:

\({\rm{F}}-{\rm{score}}=\frac{2* ({\rm{Precision}}* {\rm{Recall}})}{{\rm{Precision}}+{\rm{Recall}}}\)

The highest possible value of an F-score is 1.0, indicating perfect precision and recall, and the lowest possible value is 0, if either precision or recall are zero.

In accordance with Taghipour and Ng ( 2016 ), the calculation of QWK involves two steps:

Step 1: Construct a weight matrix W as follows:

\({W}_{{ij}}=\frac{{(i-j)}^{2}}{{(N-1)}^{2}}\)

i represents the annotation made by the tool, while j represents the annotation made by a human rater. N denotes the total number of possible annotations. Matrix O is subsequently computed, where O_( i, j ) represents the count of data annotated by the tool ( i ) and the human annotator ( j ). On the other hand, E refers to the expected count matrix, which undergoes normalization to ensure that the sum of elements in E matches the sum of elements in O.

Step 2: With matrices O and E, the QWK is obtained as follows:

K = 1- \(\frac{\sum i,j{W}_{i,j}\,{O}_{i,j}}{\sum i,j{W}_{i,j}\,{E}_{i,j}}\)

The value of the quadratic weighted kappa increases as the level of agreement improves. Further, to assess the accuracy of LLM scoring, the proportional reductive mean square error (PRMSE) was employed. The PRMSE approach takes into account the variability observed in human ratings to estimate the rater error, which is then subtracted from the variance of the human labels. This calculation provides an overall measure of agreement between the automated scores and true scores (Haberman et al. 2015 ; Loukina et al. 2020 ; Taghipour and Ng, 2016 ). The computation of PRMSE involves the following steps:

Step 1: Calculate the mean squared errors (MSEs) for the scoring outcomes of the computer-assisted tool (MSE tool) and the human scoring outcomes (MSE human).

Step 2: Determine the PRMSE by comparing the MSE of the computer-assisted tool (MSE tool) with the MSE from human raters (MSE human), using the following formula:

\({\rm{PRMSE}}=1-\frac{({\rm{MSE}}\,{\rm{tool}})\,}{({\rm{MSE}}\,{\rm{human}})\,}=1-\,\frac{{\sum }_{i}^{n}=1{({{\rm{y}}}_{i}-{\hat{{\rm{y}}}}_{{\rm{i}}})}^{2}}{{\sum }_{i}^{n}=1{({{\rm{y}}}_{i}-\hat{{\rm{y}}})}^{2}}\)

In the numerator, ŷi represents the scoring outcome predicted by a specific LLM-driven AES system for a given sample. The term y i − ŷ i represents the difference between this predicted outcome and the mean value of all LLM-driven AES systems’ scoring outcomes. It quantifies the deviation of the specific LLM-driven AES system’s prediction from the average prediction of all LLM-driven AES systems. In the denominator, y i − ŷ represents the difference between the scoring outcome provided by a specific human rater for a given sample and the mean value of all human raters’ scoring outcomes. It measures the discrepancy between the specific human rater’s score and the average score given by all human raters. The PRMSE is then calculated by subtracting the ratio of the MSE tool to the MSE human from 1. PRMSE falls within the range of 0 to 1, with larger values indicating reduced errors in LLM’s scoring compared to those of human raters. In other words, a higher PRMSE implies that LLM’s scoring demonstrates greater accuracy in predicting the true scores (Loukina et al. 2020 ). The interpretation of kappa values, ranging from 0 to 1, is based on the work of Landis and Koch ( 1977 ). Specifically, the following categories are assigned to different ranges of kappa values: −1 indicates complete inconsistency, 0 indicates random agreement, 0.0 ~ 0.20 indicates extremely low level of agreement (slight), 0.21 ~ 0.40 indicates moderate level of agreement (fair), 0.41 ~ 0.60 indicates medium level of agreement (moderate), 0.61 ~ 0.80 indicates high level of agreement (substantial), 0.81 ~ 1 indicates almost perfect level of agreement. All statistical analyses were executed using Python script.

Results and discussion

Annotation reliability of the llm.

This section focuses on assessing the reliability of the LLM’s annotation and scoring capabilities. To evaluate the reliability, several tests were conducted simultaneously, aiming to achieve the following objectives:

Assess the LLM’s ability to differentiate between test takers with varying levels of oral proficiency.

Determine the level of agreement between the annotations and scoring performed by the LLM and those done by human raters.

The evaluation of the results encompassed several metrics, including: precision, recall, F-Score, quadratically-weighted kappa, proportional reduction of mean squared error, Pearson correlation, and multi-faceted Rasch measurement.

Inter-annotator agreement (human–human annotator agreement)

We started with an agreement test of the two human annotators. Two trained annotators were recruited to determine the writing task data measures. A total of 714 scripts, as the test data, was utilized. Each analysis lasted 300–360 min. Inter-annotator agreement was evaluated using the standard measures of precision, recall, and F-score and QWK. Table 7 presents the inter-annotator agreement for the various indicators. As shown, the inter-annotator agreement was fairly high, with F-scores ranging from 1.0 for sentence and word number to 0.666 for grammatical errors.

The findings from the QWK analysis provided further confirmation of the inter-annotator agreement. The QWK values covered a range from 0.950 ( p  = 0.000) for sentence and word number to 0.695 for synonym overlap number (keyword) and grammatical errors ( p  = 0.001).

Agreement of annotation outcomes between human and LLM

To evaluate the consistency between human annotators and LLM annotators (BERT, GPT, OCLL) across the indices, the same test was conducted. The results of the inter-annotator agreement (F-score) between LLM and human annotation are provided in Appendix B-D. The F-scores ranged from 0.706 for Grammatical error # for OCLL-human to a perfect 1.000 for GPT-human, for sentences, clauses, T-units, and words. These findings were further supported by the QWK analysis, which showed agreement levels ranging from 0.807 ( p  = 0.001) for metadiscourse markers for OCLL-human to 0.962 for words ( p  = 0.000) for GPT-human. The findings demonstrated that the LLM annotation achieved a significant level of accuracy in identifying measurement units and counts.

Reliability of LLM-driven AES’s scoring and discriminating proficiency levels

This section examines the reliability of the LLM-driven AES scoring through a comparison of the scoring outcomes produced by human raters and the LLM ( Reliability of LLM-driven AES scoring ). It also assesses the effectiveness of the LLM-based AES system in differentiating participants with varying proficiency levels ( Reliability of LLM-driven AES discriminating proficiency levels ).

Reliability of LLM-driven AES scoring

Table 8 summarizes the QWK coefficient analysis between the scores computed by the human raters and the GPT-4 for the individual essays from I-JAS Footnote 7 . As shown, the QWK of all measures ranged from k  = 0.819 for lexical density (number of lexical words (tokens)/number of words per essay) to k  = 0.644 for word2vec cosine similarity. Table 9 further presents the Pearson correlations between the 16 writing proficiency measures scored by human raters and GPT 4 for the individual essays. The correlations ranged from 0.672 for syntactic complexity to 0.734 for grammatical accuracy. The correlations between the writing proficiency scores assigned by human raters and the BERT-based AES system were found to range from 0.661 for syntactic complexity to 0.713 for grammatical accuracy. The correlations between the writing proficiency scores given by human raters and the OCLL-based AES system ranged from 0.654 for cohesion to 0.721 for grammatical accuracy. These findings indicated an alignment between the assessments made by human raters and both the BERT-based and OCLL-based AES systems in terms of various aspects of writing proficiency.

Reliability of LLM-driven AES discriminating proficiency levels

After validating the reliability of the LLM’s annotation and scoring, the subsequent objective was to evaluate its ability to distinguish between various proficiency levels. For this analysis, a dataset of 686 individual essays was utilized. Table 10 presents a sample of the results, summarizing the means, standard deviations, and the outcomes of the one-way ANOVAs based on the measures assessed by the GPT-4 model. A post hoc multiple comparison test, specifically the Bonferroni test, was conducted to identify any potential differences between pairs of levels.

As the results reveal, seven measures presented linear upward or downward progress across the three proficiency levels. These were marked in bold in Table 10 and comprise one measure of lexical richness, i.e. MATTR (lexical diversity); four measures of syntactic complexity, i.e. MDD (mean dependency distance), MLC (mean length of clause), CNT (complex nominals per T-unit), CPC (coordinate phrases rate); one cohesion measure, i.e. word2vec cosine similarity and GER (grammatical error rate). Regarding the ability of the sixteen measures to distinguish adjacent proficiency levels, the Bonferroni tests indicated that statistically significant differences exist between the primary level and the intermediate level for MLC and GER. One measure of lexical richness, namely LD, along with three measures of syntactic complexity (VPT, CT, DCT, ACC), two measures of cohesion (SOPT, SOPK), and one measure of content elaboration (IMM), exhibited statistically significant differences between proficiency levels. However, these differences did not demonstrate a linear progression between adjacent proficiency levels. No significant difference was observed in lexical sophistication between proficiency levels.

To summarize, our study aimed to evaluate the reliability and differentiation capabilities of the LLM-driven AES method. For the first objective, we assessed the LLM’s ability to differentiate between test takers with varying levels of oral proficiency using precision, recall, F-Score, and quadratically-weighted kappa. Regarding the second objective, we compared the scoring outcomes generated by human raters and the LLM to determine the level of agreement. We employed quadratically-weighted kappa and Pearson correlations to compare the 16 writing proficiency measures for the individual essays. The results confirmed the feasibility of using the LLM for annotation and scoring in AES for nonnative Japanese. As a result, Research Question 1 has been addressed.

Comparison of BERT-, GPT-, OCLL-based AES, and linguistic-feature-based computation methods

This section aims to compare the effectiveness of five AES methods for nonnative Japanese writing, i.e. LLM-driven approaches utilizing BERT, GPT, and OCLL, linguistic feature-based approaches using Jess and JWriter. The comparison was conducted by comparing the ratings obtained from each approach with human ratings. All ratings were derived from the dataset introduced in Dataset . To facilitate the comparison, the agreement between the automated methods and human ratings was assessed using QWK and PRMSE. The performance of each approach was summarized in Table 11 .

The QWK coefficient values indicate that LLMs (GPT, BERT, OCLL) and human rating outcomes demonstrated higher agreement compared to feature-based AES methods (Jess and JWriter) in assessing writing proficiency criteria, including lexical richness, syntactic complexity, content, and grammatical accuracy. Among the LLMs, the GPT-4 driven AES and human rating outcomes showed the highest agreement in all criteria, except for syntactic complexity. The PRMSE values suggest that the GPT-based method outperformed linguistic feature-based methods and other LLM-based approaches. Moreover, an interesting finding emerged during the study: the agreement coefficient between GPT-4 and human scoring was even higher than the agreement between different human raters themselves. This discovery highlights the advantage of GPT-based AES over human rating. Ratings involve a series of processes, including reading the learners’ writing, evaluating the content and language, and assigning scores. Within this chain of processes, various biases can be introduced, stemming from factors such as rater biases, test design, and rating scales. These biases can impact the consistency and objectivity of human ratings. GPT-based AES may benefit from its ability to apply consistent and objective evaluation criteria. By prompting the GPT model with detailed writing scoring rubrics and linguistic features, potential biases in human ratings can be mitigated. The model follows a predefined set of guidelines and does not possess the same subjective biases that human raters may exhibit. This standardization in the evaluation process contributes to the higher agreement observed between GPT-4 and human scoring. Section Prompt strategy of the study delves further into the role of prompts in the application of LLMs to AES. It explores how the choice and implementation of prompts can impact the performance and reliability of LLM-based AES methods. Furthermore, it is important to acknowledge the strengths of the local model, i.e. the Japanese local model OCLL, which excels in processing certain idiomatic expressions. Nevertheless, our analysis indicated that GPT-4 surpasses local models in AES. This superior performance can be attributed to the larger parameter size of GPT-4, estimated to be between 500 billion and 1 trillion, which exceeds the sizes of both BERT and the local model OCLL.

Prompt strategy

In the context of prompt strategy, Mizumoto and Eguchi ( 2023 ) conducted a study where they applied the GPT-3 model to automatically score English essays in the TOEFL test. They found that the accuracy of the GPT model alone was moderate to fair. However, when they incorporated linguistic measures such as cohesion, syntactic complexity, and lexical features alongside the GPT model, the accuracy significantly improved. This highlights the importance of prompt engineering and providing the model with specific instructions to enhance its performance. In this study, a similar approach was taken to optimize the performance of LLMs. GPT-4, which outperformed BERT and OCLL, was selected as the candidate model. Model 1 was used as the baseline, representing GPT-4 without any additional prompting. Model 2, on the other hand, involved GPT-4 prompted with 16 measures that included scoring criteria, efficient linguistic features for writing assessment, and detailed measurement units and calculation formulas. The remaining models (Models 3 to 18) utilized GPT-4 prompted with individual measures. The performance of these 18 different models was assessed using the output indicators described in Section Criteria (output indicator) . By comparing the performances of these models, the study aimed to understand the impact of prompt engineering on the accuracy and effectiveness of GPT-4 in AES tasks.

Based on the PRMSE scores presented in Fig. 4 , it was observed that Model 1, representing GPT-4 without any additional prompting, achieved a fair level of performance. However, Model 2, which utilized GPT-4 prompted with all measures, outperformed all other models in terms of PRMSE score, achieving a score of 0.681. These results indicate that the inclusion of specific measures and prompts significantly enhanced the performance of GPT-4 in AES. Among the measures, syntactic complexity was found to play a particularly significant role in improving the accuracy of GPT-4 in assessing writing quality. Following that, lexical diversity emerged as another important factor contributing to the model’s effectiveness. The study suggests that a well-prompted GPT-4 can serve as a valuable tool to support human assessors in evaluating writing quality. By utilizing GPT-4 as an automated scoring tool, the evaluation biases associated with human raters can be minimized. This has the potential to empower teachers by allowing them to focus on designing writing tasks and guiding writing strategies, while leveraging the capabilities of GPT-4 for efficient and reliable scoring.

PRMSE scores of the 18 AES models.

This study aimed to investigate two main research questions: the feasibility of utilizing LLMs for AES and the impact of prompt engineering on the application of LLMs in AES.

To address the first objective, the study compared the effectiveness of five different models: GPT, BERT, the Japanese local LLM (OCLL), and two conventional machine learning-based AES tools (Jess and JWriter). The PRMSE values indicated that the GPT-4-based method outperformed other LLMs (BERT, OCLL) and linguistic feature-based computational methods (Jess and JWriter) across various writing proficiency criteria. Furthermore, the agreement coefficient between GPT-4 and human scoring surpassed the agreement among human raters themselves, highlighting the potential of using the GPT-4 tool to enhance AES by reducing biases and subjectivity, saving time, labor, and cost, and providing valuable feedback for self-study. Regarding the second goal, the role of prompt design was investigated by comparing 18 models, including a baseline model, a model prompted with all measures, and 16 models prompted with one measure at a time. GPT-4, which outperformed BERT and OCLL, was selected as the candidate model. The PRMSE scores of the models showed that GPT-4 prompted with all measures achieved the best performance, surpassing the baseline and other models.

In conclusion, this study has demonstrated the potential of LLMs in supporting human rating in assessments. By incorporating automation, we can save time and resources while reducing biases and subjectivity inherent in human rating processes. Automated language assessments offer the advantage of accessibility, providing equal opportunities and economic feasibility for individuals who lack access to traditional assessment centers or necessary resources. LLM-based language assessments provide valuable feedback and support to learners, aiding in the enhancement of their language proficiency and the achievement of their goals. This personalized feedback can cater to individual learner needs, facilitating a more tailored and effective language-learning experience.

There are three important areas that merit further exploration. First, prompt engineering requires attention to ensure optimal performance of LLM-based AES across different language types. This study revealed that GPT-4, when prompted with all measures, outperformed models prompted with fewer measures. Therefore, investigating and refining prompt strategies can enhance the effectiveness of LLMs in automated language assessments. Second, it is crucial to explore the application of LLMs in second-language assessment and learning for oral proficiency, as well as their potential in under-resourced languages. Recent advancements in self-supervised machine learning techniques have significantly improved automatic speech recognition (ASR) systems, opening up new possibilities for creating reliable ASR systems, particularly for under-resourced languages with limited data. However, challenges persist in the field of ASR. First, ASR assumes correct word pronunciation for automatic pronunciation evaluation, which proves challenging for learners in the early stages of language acquisition due to diverse accents influenced by their native languages. Accurately segmenting short words becomes problematic in such cases. Second, developing precise audio-text transcriptions for languages with non-native accented speech poses a formidable task. Last, assessing oral proficiency levels involves capturing various linguistic features, including fluency, pronunciation, accuracy, and complexity, which are not easily captured by current NLP technology.

Data availability

The dataset utilized was obtained from the International Corpus of Japanese as a Second Language (I-JAS). The data URLs: [ https://www2.ninjal.ac.jp/jll/lsaj/ihome2.html ].

J-CAT and TTBJ are two computerized adaptive tests used to assess Japanese language proficiency.

SPOT is a specific component of the TTBJ test.

J-CAT: https://www.j-cat2.org/html/ja/pages/interpret.html

SPOT: https://ttbj.cegloc.tsukuba.ac.jp/p1.html#SPOT .

The study utilized a prompt-based GPT-4 model, developed by OpenAI, which has an impressive architecture with 1.8 trillion parameters across 120 layers. GPT-4 was trained on a vast dataset of 13 trillion tokens, using two stages: initial training on internet text datasets to predict the next token, and subsequent fine-tuning through reinforcement learning from human feedback.

https://www2.ninjal.ac.jp/jll/lsaj/ihome2-en.html .

http://jhlee.sakura.ne.jp/JEV/ by Japanese Learning Dictionary Support Group 2015.

We express our sincere gratitude to the reviewer for bringing this matter to our attention.

On February 7, 2023, Microsoft began rolling out a major overhaul to Bing that included a new chatbot feature based on OpenAI’s GPT-4 (Bing.com).

Appendix E-F present the analysis results of the QWK coefficient between the scores computed by the human raters and the BERT, OCLL models.

Attali Y, Burstein J (2006) Automated essay scoring with e-rater® V.2. J. Technol., Learn. Assess., 4

Barkaoui K, Hadidi A (2020) Assessing Change in English Second Language Writing Performance (1st ed.). Routledge, New York. https://doi.org/10.4324/9781003092346

Bentz C, Tatyana R, Koplenig A, Tanja S (2016) A comparison between morphological complexity. measures: Typological data vs. language corpora. In Proceedings of the workshop on computational linguistics for linguistic complexity (CL4LC), 142–153. Osaka, Japan: The COLING 2016 Organizing Committee

Bond TG, Yan Z, Heene M (2021) Applying the Rasch model: Fundamental measurement in the human sciences (4th ed). Routledge

Brants T (2000) Inter-annotator agreement for a German newspaper corpus. Proceedings of the Second International Conference on Language Resources and Evaluation (LREC’00), Athens, Greece, 31 May-2 June, European Language Resources Association

Brown TB, Mann B, Ryder N, et al. (2020) Language models are few-shot learners. Advances in Neural Information Processing Systems, Online, 6–12 December, Curran Associates, Inc., Red Hook, NY

Burstein J (2003) The E-rater scoring engine: Automated essay scoring with natural language processing. In Shermis MD and Burstein JC (ed) Automated Essay Scoring: A Cross-Disciplinary Perspective. Lawrence Erlbaum Associates, Mahwah, NJ

Čech R, Miroslav K (2018) Morphological richness of text. In Masako F, Václav C (ed) Taming the corpus: From inflection and lexis to interpretation, 63–77. Cham, Switzerland: Springer Nature

Çöltekin Ç, Taraka, R (2018) Exploiting Universal Dependencies treebanks for measuring morphosyntactic complexity. In Aleksandrs B, Christian B (ed), Proceedings of first workshop on measuring language complexity, 1–7. Torun, Poland

Crossley SA, Cobb T, McNamara DS (2013) Comparing count-based and band-based indices of word frequency: Implications for active vocabulary research and pedagogical applications. System 41:965–981. https://doi.org/10.1016/j.system.2013.08.002

Article   Google Scholar  

Crossley SA, McNamara DS (2016) Say more and be more coherent: How text elaboration and cohesion can increase writing quality. J. Writ. Res. 7:351–370

CyberAgent Inc (2023) Open-Calm series of Japanese language models. Retrieved from: https://www.cyberagent.co.jp/news/detail/id=28817

Devlin J, Chang MW, Lee K, Toutanova K (2019) BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, Minneapolis, Minnesota, 2–7 June, pp. 4171–4186. Association for Computational Linguistics

Diez-Ortega M, Kyle K (2023) Measuring the development of lexical richness of L2 Spanish: a longitudinal learner corpus study. Studies in Second Language Acquisition 1-31

Eckes T (2009) On common ground? How raters perceive scoring criteria in oral proficiency testing. In Brown A, Hill K (ed) Language testing and evaluation 13: Tasks and criteria in performance assessment (pp. 43–73). Peter Lang Publishing

Elliot S (2003) IntelliMetric: from here to validity. In: Shermis MD, Burstein JC (ed) Automated Essay Scoring: A Cross-Disciplinary Perspective. Lawrence Erlbaum Associates, Mahwah, NJ

Google Scholar  

Engber CA (1995) The relationship of lexical proficiency to the quality of ESL compositions. J. Second Lang. Writ. 4:139–155

Garner J, Crossley SA, Kyle K (2019) N-gram measures and L2 writing proficiency. System 80:176–187. https://doi.org/10.1016/j.system.2018.12.001

Haberman SJ (2008) When can subscores have value? J. Educat. Behav. Stat., 33:204–229

Haberman SJ, Yao L, Sinharay S (2015) Prediction of true test scores from observed item scores and ancillary data. Brit. J. Math. Stat. Psychol. 68:363–385

Halliday MAK (1985) Spoken and Written Language. Deakin University Press, Melbourne, Australia

Hirao R, Arai M, Shimanaka H et al. (2020) Automated essay scoring system for nonnative Japanese learners. Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pp. 1250–1257. European Language Resources Association

Hunt KW (1966) Recent Measures in Syntactic Development. Elementary English, 43(7), 732–739. http://www.jstor.org/stable/41386067

Ishioka T (2001) About e-rater, a computer-based automatic scoring system for essays [Konpyūta ni yoru essei no jidō saiten shisutemu e − rater ni tsuite]. University Entrance Examination. Forum [Daigaku nyūshi fōramu] 24:71–76

Hochreiter S, Schmidhuber J (1997) Long short- term memory. Neural Comput. 9(8):1735–1780

Article   CAS   PubMed   Google Scholar  

Ishioka T, Kameda M (2006) Automated Japanese essay scoring system based on articles written by experts. Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, Sydney, Australia, 17–18 July 2006, pp. 233-240. Association for Computational Linguistics, USA

Japan Foundation (2021) Retrieved from: https://www.jpf.gp.jp/j/project/japanese/survey/result/dl/survey2021/all.pdf

Jarvis S (2013a) Defining and measuring lexical diversity. In Jarvis S, Daller M (ed) Vocabulary knowledge: Human ratings and automated measures (Vol. 47, pp. 13–44). John Benjamins. https://doi.org/10.1075/sibil.47.03ch1

Jarvis S (2013b) Capturing the diversity in lexical diversity. Lang. Learn. 63:87–106. https://doi.org/10.1111/j.1467-9922.2012.00739.x

Jiang J, Quyang J, Liu H (2019) Interlanguage: A perspective of quantitative linguistic typology. Lang. Sci. 74:85–97

Kim M, Crossley SA, Kyle K (2018) Lexical sophistication as a multidimensional phenomenon: Relations to second language lexical proficiency, development, and writing quality. Mod. Lang. J. 102(1):120–141. https://doi.org/10.1111/modl.12447

Kojima T, Gu S, Reid M et al. (2022) Large language models are zero-shot reasoners. Advances in Neural Information Processing Systems, New Orleans, LA, 29 November-1 December, Curran Associates, Inc., Red Hook, NY

Kyle K, Crossley SA (2015) Automatically assessing lexical sophistication: Indices, tools, findings, and application. TESOL Q 49:757–786

Kyle K, Crossley SA, Berger CM (2018) The tool for the automatic analysis of lexical sophistication (TAALES): Version 2.0. Behav. Res. Methods 50:1030–1046. https://doi.org/10.3758/s13428-017-0924-4

Article   PubMed   Google Scholar  

Kyle K, Crossley SA, Jarvis S (2021) Assessing the validity of lexical diversity using direct judgements. Lang. Assess. Q. 18:154–170. https://doi.org/10.1080/15434303.2020.1844205

Landauer TK, Laham D, Foltz PW (2003) Automated essay scoring and annotation of essays with the Intelligent Essay Assessor. In Shermis MD, Burstein JC (ed), Automated Essay Scoring: A Cross-Disciplinary Perspective. Lawrence Erlbaum Associates, Mahwah, NJ

Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics 159–174

Laufer B, Nation P (1995) Vocabulary size and use: Lexical richness in L2 written production. Appl. Linguist. 16:307–322. https://doi.org/10.1093/applin/16.3.307

Lee J, Hasebe Y (2017) jWriter Learner Text Evaluator, URL: https://jreadability.net/jwriter/

Lee J, Kobayashi N, Sakai T, Sakota K (2015) A Comparison of SPOT and J-CAT Based on Test Analysis [Tesuto bunseki ni motozuku ‘SPOT’ to ‘J-CAT’ no hikaku]. Research on the Acquisition of Second Language Japanese [Dainigengo to shite no nihongo no shūtoku kenkyū] (18) 53–69

Li W, Yan J (2021) Probability distribution of dependency distance based on a Treebank of. Japanese EFL Learners’ Interlanguage. J. Quant. Linguist. 28(2):172–186. https://doi.org/10.1080/09296174.2020.1754611

Article   MathSciNet   Google Scholar  

Linacre JM (2002) Optimizing rating scale category effectiveness. J. Appl. Meas. 3(1):85–106

PubMed   Google Scholar  

Linacre JM (1994) Constructing measurement with a Many-Facet Rasch Model. In Wilson M (ed) Objective measurement: Theory into practice, Volume 2 (pp. 129–144). Norwood, NJ: Ablex

Liu H (2008) Dependency distance as a metric of language comprehension difficulty. J. Cognitive Sci. 9:159–191

Liu H, Xu C, Liang J (2017) Dependency distance: A new perspective on syntactic patterns in natural languages. Phys. Life Rev. 21. https://doi.org/10.1016/j.plrev.2017.03.002

Loukina A, Madnani N, Cahill A, et al. (2020) Using PRMSE to evaluate automated scoring systems in the presence of label noise. Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications, Seattle, WA, USA → Online, 10 July, pp. 18–29. Association for Computational Linguistics

Lu X (2010) Automatic analysis of syntactic complexity in second language writing. Int. J. Corpus Linguist. 15:474–496

Lu X (2012) The relationship of lexical richness to the quality of ESL learners’ oral narratives. Mod. Lang. J. 96:190–208

Lu X (2017) Automated measurement of syntactic complexity in corpus-based L2 writing research and implications for writing assessment. Lang. Test. 34:493–511

Lu X, Hu R (2022) Sense-aware lexical sophistication indices and their relationship to second language writing quality. Behav. Res. Method. 54:1444–1460. https://doi.org/10.3758/s13428-021-01675-6

Ministry of Health, Labor, and Welfare of Japan (2022) Retrieved from: https://www.mhlw.go.jp/stf/newpage_30367.html

Mizumoto A, Eguchi M (2023) Exploring the potential of using an AI language model for automated essay scoring. Res. Methods Appl. Linguist. 3:100050

Okgetheng B, Takeuchi K (2024) Estimating Japanese Essay Grading Scores with Large Language Models. Proceedings of 30th Annual Conference of the Language Processing Society in Japan, March 2024

Ortega L (2015) Second language learning explained? SLA across 10 contemporary theories. In VanPatten B, Williams J (ed) Theories in Second Language Acquisition: An Introduction

Rae JW, Borgeaud S, Cai T, et al. (2021) Scaling Language Models: Methods, Analysis & Insights from Training Gopher. ArXiv, abs/2112.11446

Read J (2000) Assessing vocabulary. Cambridge University Press. https://doi.org/10.1017/CBO9780511732942

Rudner LM, Liang T (2002) Automated Essay Scoring Using Bayes’ Theorem. J. Technol., Learning and Assessment, 1 (2)

Sakoda K, Hosoi Y (2020) Accuracy and complexity of Japanese Language usage by SLA learners in different learning environments based on the analysis of I-JAS, a learners’ corpus of Japanese as L2. Math. Linguist. 32(7):403–418. https://doi.org/10.24701/mathling.32.7_403

Suzuki N (1999) Summary of survey results regarding comprehensive essay questions. Final report of “Joint Research on Comprehensive Examinations for the Aim of Evaluating Applicability to Each Specialized Field of Universities” for 1996-2000 [shōronbun sōgō mondai ni kansuru chōsa kekka no gaiyō. Heisei 8 - Heisei 12-nendo daigaku no kaku senmon bun’ya e no tekisei no hyōka o mokuteki to suru sōgō shiken no arikata ni kansuru kyōdō kenkyū’ saishū hōkoku-sho]. University Entrance Examination Section Center Research and Development Department [Daigaku nyūshi sentā kenkyū kaihatsubu], 21–32

Taghipour K, Ng HT (2016) A neural approach to automated essay scoring. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, Texas, 1–5 November, pp. 1882–1891. Association for Computational Linguistics

Takeuchi K, Ohno M, Motojin K, Taguchi M, Inada Y, Iizuka M, Abo T, Ueda H (2021) Development of essay scoring methods based on reference texts with construction of research-available Japanese essay data. In IPSJ J 62(9):1586–1604

Ure J (1971) Lexical density: A computational technique and some findings. In Coultard M (ed) Talking about Text. English Language Research, University of Birmingham, Birmingham, England

Vaswani A, Shazeer N, Parmar N, et al. (2017) Attention is all you need. In Advances in Neural Information Processing Systems, Long Beach, CA, 4–7 December, pp. 5998–6008, Curran Associates, Inc., Red Hook, NY

Watanabe H, Taira Y, Inoue Y (1988) Analysis of essay evaluation data [Shōronbun hyōka dēta no kaiseki]. Bulletin of the Faculty of Education, University of Tokyo [Tōkyōdaigaku kyōiku gakubu kiyō], Vol. 28, 143–164

Yao S, Yu D, Zhao J, et al. (2023) Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36

Zenker F, Kyle K (2021) Investigating minimum text lengths for lexical diversity indices. Assess. Writ. 47:100505. https://doi.org/10.1016/j.asw.2020.100505

Zhang Y, Warstadt A, Li X, et al. (2021) When do you need billions of words of pretraining data? Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, Online, pp. 1112-1125. Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.acl-long.90

Download references

This research was funded by National Foundation of Social Sciences (22BYY186) to Wenchao Li.

Author information

Authors and affiliations.

Department of Japanese Studies, Zhejiang University, Hangzhou, China

Department of Linguistics and Applied Linguistics, Zhejiang University, Hangzhou, China

You can also search for this author in PubMed   Google Scholar

Contributions

Wenchao Li is in charge of conceptualization, validation, formal analysis, investigation, data curation, visualization and writing the draft. Haitao Liu is in charge of supervision.

Corresponding author

Correspondence to Wenchao Li .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Ethical approval

Ethical approval was not required as the study did not involve human participants.

Informed consent

This article does not contain any studies with human participants performed by any of the authors.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplemental material file #1, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Li, W., Liu, H. Applying large language models for automated essay scoring for non-native Japanese. Humanit Soc Sci Commun 11 , 723 (2024). https://doi.org/10.1057/s41599-024-03209-9

Download citation

Received : 02 February 2024

Accepted : 16 May 2024

Published : 03 June 2024

DOI : https://doi.org/10.1057/s41599-024-03209-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies