bibliography for research paper example

How to Write a Bibliography for a Research Paper

Do not try to “wow” your instructor with a long bibliography when your instructor requests only a works cited page. It is tempting, after doing a lot of work to research a paper, to try to include summaries on each source as you write your paper so that your instructor appreciates how much work you did. That is a trap you want to avoid. MLA style, the one that is most commonly followed in high schools and university writing courses, dictates that you include only the works you actually cited in your paper—not all those that you used.

Academic Writing, Editing, Proofreading, And Problem Solving Services

Get 10% off with 24start discount code, assembling bibliographies and works cited.

• If your assignment calls for a bibliography, list all the sources you consulted in your research.
• If your assignment calls for a works cited or references page, include only the sources you quote, summarize, paraphrase, or mention in your paper.
• If your works cited page includes a source that you did not cite in your paper, delete it.
• All in-text citations that you used at the end of quotations, summaries, and paraphrases to credit others for their ideas,words, and work must be accompanied by a cited reference in the bibliography or works cited. These references must include specific information about the source so that your readers can identify precisely where the information came from.The citation entries on a works cited page typically include the author’s name, the name of the article, the name of the publication, the name of the publisher (for books), where it was published (for books), and when it was published.

The good news is that you do not have to memorize all the many ways the works cited entries should be written. Numerous helpful style guides are available to show you the information that should be included, in what order it should appear, and how to format it. The format often differs according to the style guide you are using. The Modern Language Association (MLA) follows a particular style that is a bit different from APA (American Psychological Association) style, and both are somewhat different from the Chicago Manual of Style (CMS). Always ask your teacher which style you should use.

A bibliography usually appears at the end of a paper on its own separate page. All bibliography entries—books, periodicals, Web sites, and nontext sources such radio broadcasts—are listed together in alphabetical order. Books and articles are alphabetized by the author’s last name.

Most teachers suggest that you follow a standard style for listing different types of sources. If your teacher asks you to use a different form, however, follow his or her instructions. Take pride in your bibliography. It represents some of the most important work you’ve done for your research paper—and using proper form shows that you are a serious and careful researcher.

Bibliography Entry for a Book

A bibliography entry for a book begins with the author’s name, which is written in this order: last name, comma, first name, period. After the author’s name comes the title of the book. If you are handwriting your bibliography, underline each title. If you are working on a computer, put the book title in italicized type. Be sure to capitalize the words in the title correctly, exactly as they are written in the book itself. Following the title is the city where the book was published, followed by a colon, the name of the publisher, a comma, the date published, and a period. Here is an example:

Format : Author’s last name, first name. Book Title. Place of publication: publisher, date of publication.

• A book with one author : Hartz, Paula.  Abortion: A Doctor’s Perspective, a Woman’s Dilemma . New York: Donald I. Fine, Inc., 1992.
• A book with two or more authors : Landis, Jean M. and Rita J. Simon.  Intelligence: Nature or Nurture?  New York: HarperCollins, 1998.

Bibliography Entry for a Periodical

A bibliography entry for a periodical differs slightly in form from a bibliography entry for a book. For a magazine article, start with the author’s last name first, followed by a comma, then the first name and a period. Next, write the title of the article in quotation marks, and include a period (or other closing punctuation) inside the closing quotation mark. The title of the magazine is next, underlined or in italic type, depending on whether you are handwriting or using a computer, followed by a period. The date and year, followed by a colon and the pages on which the article appeared, come last. Here is an example:

Format:  Author’s last name, first name. “Title of the Article.” Magazine. Month and year of publication: page numbers.

• Article in a monthly magazine : Crowley, J.E.,T.E. Levitan and R.P. Quinn.“Seven Deadly Half-Truths About Women.”  Psychology Today  March 1978: 94–106.
• Article in a weekly magazine : Schwartz, Felice N.“Management,Women, and the New Facts of Life.”  Newsweek  20 July 2006: 21–22.
• Signed newspaper article : Ferraro, Susan. “In-law and Order: Finding Relative Calm.”  The Daily News  30 June 1998: 73.
• Unsigned newspaper article : “Beanie Babies May Be a Rotten Nest Egg.”  Chicago Tribune  21 June 2004: 12.

Bibliography Entry for a Web Site

For sources such as Web sites include the information a reader needs to find the source or to know where and when you found it. Always begin with the last name of the author, broadcaster, person you interviewed, and so on. Here is an example of a bibliography for a Web site:

Format : Author.“Document Title.” Publication or Web site title. Date of publication. Date of access.

Example : Dodman, Dr. Nicholas. “Dog-Human Communication.”  Pet Place . 10 November 2006.  23 January 2014 < http://www.petplace.com/dogs/dog-human-communication-2/page1.aspx >

After completing the bibliography you can breathe a huge sigh of relief and pat yourself on the back. You probably plan to turn in your work in printed or handwritten form, but you also may be making an oral presentation. However you plan to present your paper, do your best to show it in its best light. You’ve put a great deal of work and thought into this assignment, so you want your paper to look and sound its best. You’ve completed your research paper!

Back to  How To Write A Research Paper .

ORDER HIGH QUALITY CUSTOM PAPER

• Bipolar Disorder
• Therapy Center
• When To See a Therapist
• Types of Therapy
• Best Online Therapy
• Best Couples Therapy
• Best Family Therapy
• Managing Stress
• Sleep and Dreaming
• Understanding Emotions
• Self-Improvement
• Healthy Relationships
• Student Resources
• Personality Types
• Guided Meditations
• Verywell Mind Insights
• 2023 Verywell Mind 25
• Mental Health in the Classroom
• Editorial Process
• Meet Our Review Board
• Crisis Support

How to Write a Bibliography in APA Format

Kendra Cherry, MS, is a psychosocial rehabilitation specialist, psychology educator, and author of the "Everything Psychology Book."

Emily is a board-certified science editor who has worked with top digital publishing brands like Voices for Biodiversity, Study.com, GoodTherapy, Vox, and Verywell.

• APA Bibliography
• How to Create One
• Why You Need It

Sample Bibliography

An APA format bibliography lists all of the sources that might be used in a paper. A bibliography can be a great tool to help you keep track of information during the research and writing process. In some cases, your instructor may require you to include a bibliography as part of your assignment.

At a Glance

A well-written APA format bibliography can help you keep track of information and sources as you research and write your psychology paper. To create a bibliography, gather up all of the sources that you might use in your paper. Create an APA format reference for each source and then write a brief annotation. Your annotation should be a brief summary of what each reference is about. You can quickly refer to these annotations When writing your paper and determine which to include.

What Is an APA Format Bibliography?

An APA format bibliography is an alphabetical listing of all sources that might be used to write an academic paper, essay, article, or research paper—particularly work that is covering psychology or psychology-related topics. APA format is the official style of the American Psychological Association (APA). This format is used by many psychology professors, students, and researchers.

Even if it is not a required part of your assignment, writing a bibliography can help you keep track of your sources and make it much easier to create your final reference page in proper APA format.

Creating an APA Bibliography

A bibliography is similar in many ways to a reference section , but there are some important differences. While a reference section includes every source that was actually used in your paper, a bibliography may include sources that you considered using but may have dismissed because they were irrelevant or outdated.

Bibliographies can be a great way to keep track of information you might want to use in your paper and to organize the information that you find in different sources. The following are four steps you can follow to create your APA format bibliography.

Start on a New Page

Your working bibliography should be kept separate from the rest of your paper. Start it on a new page, with the title "Bibliography" centered at the top and in bold text. Some people use the title "References" instead, so it's best to check with your professor or instructor about which they prefer you to use.

Gather Your Sources

Compile all the sources you might possibly use in your paper. While you might not use all of these sources in your paper, having a complete list will make it easier later on when you prepare your reference section.

Gathering your sources can be particularly helpful when outlining and writing your paper.

By quickly glancing through your working bibliography, you will be able to get a better idea of which sources will be the most appropriate to support your thesis and main points.

Reference Each Source

Your references should be listed alphabetically by the author’s last name, and they should be double-spaced. The first line of each reference should be flush left, while each additional line of a single reference should be a few spaces to the right of the left margin, which is known as a hanging indent.

The format of each source is as follows for academic journals:

• Last name of first author (followed by their first initial)
• The year the source was published in parentheses
• The title of the source
• The journal that published the source (in italics)
• The volume number, if applicable (in italics)
• The issue number, if applicable
• Page numbers (in parentheses)
• The URL or "doi" in lowercase letters followed by a colon and the doi number, if applicable

The following examples are scholarly articles in academic journals, cited in APA format:

• Kulacaoglu, F., & Kose, S. (2018). Borderline personality disorder (BPD): In the midst of vulnerability, chaos, and awe.  Brain sciences ,  8 (11), 201. doi:10.3390/brainsci8110201
• Cattane, N., Rossi, R., & Lanfredi, M. (2017). Borderline personality disorder and childhood trauma: exploring the affected biological systems and mechanisms.  BMC Psychiatry,   18 (221). doi:10.1186/s12888-017-1383-2

Visit the American Psychological Association's website for more information on citing other types of sources including online media, audiovisual media, and more.

Create an Annotation for Each Source

Normally a bibliography contains only references' information, but in some cases you might decide to create an annotated bibliography. An annotation is a summary or evaluation of the source.

An annotation is a brief description of approximately 150 words describing the information in the source, your evaluation of its credibility, and how it pertains to your topic. Writing one of these for each piece of research will make your writing process faster and easier.

This step helpful in determining which sources to ultimately use in your paper. Your instructor may also require it as part of the assignment so they can assess your thought process and understanding of your topic.

Reasons to Write a Bibliography

One of the biggest reasons to create an APA format bibliography is simply to make the research and writing process easier.

If you do not have a comprehensive list of all of your references, you might find yourself scrambling to figure out where you found certain bits of information that you included in your paper.

A bibliography is also an important tool that your readers can use to access your sources.

While writing an annotated bibliography might not be required for your assignment, it can be a very useful step. The process of writing an annotation helps you learn more about your topic, develop a deeper understanding of the subject, and become better at evaluating various sources of information.

The following is an example of an APA format bibliography by the website EasyBib:

There are many online resources that demonstrate different formats of bibliographies, including the American Psychological Association website . Purdue University's Online Writing Lab also has examples of formatting an APA format bibliography.

Check out this video on their YouTube channel which provides detailed instructions on formatting an APA style bibliography in Microsoft Word.

You can check out the Purdue site for more information on writing an annotated APA bibliography as well.

What This Means For You

If you are taking a psychology class, you may be asked to create a bibliography as part of the research paper writing process. Even if your instructor does not expressly require a bibliography, creating one can be a helpful way to help structure your research and make the writing process more manageable.

For psychology majors , it can be helpful to save any bibliographies you have written throughout your studies so that you can refer back to them later when studying for exams or writing papers for other psychology courses.

American Psychological Association. Publication Manual of the American Psychological Association . 7th Edition. Washington, DC: American Psychological Association; 2020.

Masic I. The importance of proper citation of references in biomedical articles.   Acta Inform Med . 2013;21(3):148–155. doi:10.5455/aim.2013.21.148-155

American Psychological Association. How do you format a bibliography in APA Style?

Cornell University Library. How to prepare an annotated bibliography: The annotated bibliography .

By Kendra Cherry, MSEd Kendra Cherry, MS, is a psychosocial rehabilitation specialist, psychology educator, and author of the "Everything Psychology Book."

• Try for free

How to Write a Bibliography (MLA, APA Examples)

Learn how to easily write a bibliography by following the format outlined in this article.

This resource will help your students properly cite different resources in the bibliography of a research paper, and how to format those citations, for books, encyclopedias, films, websites, and people.

What is a bibliography?

According to Infoplease.com, A bibliography is a list of the types of sources you used to get information for your report. It is included at the end of your report, on the last page (or last few pages).

What are the types of bibliography styles (MLA, APA, etc.)?

The 3 most common bibliography/citation styles are:

• MLA Style: The Modern Language Association works cited page style
• APA Style: The American Psychological Association style
• Chicago Style: The bibliography style defined by the Chicago Manual of Style

We’ll give examples of how to create bibliography entries in various styles further down in this article. 

What sources do you put in a bibliography?

An annotated bibliography should include a reference list of any sources you use in writing a research paper. Any printed sources from which you use a text citation, including books, websites, newspaper articles, journal articles, academic writing, online sources (such as PDFs), and magazines should be included in a reference list. In some cases, you may need or want to cite conversations or interviews, works of art, visual works such as movies, television shows, or documentaries - these (and many others) can also be included in a reference list.

How to get started writing your bibliography

You will find it easier to prepare your MLA, APA, or Chicago annotated bibliography if you keep track of each book, encyclopedia, journal article, webpage or online source you use as you are reading and taking notes. Start a preliminary, or draft, bibliography by listing on a separate sheet of paper all your sources. Note down the full title, author’s last name, place of publication, web address, publisher, and date of publication for each source.

Haven't started your paper yet and need an outline? These sample essay outlines include a research paper outline from an actual student paper.

How to write a bibliography step-by-step (with examples)

General Format: Author (last name first). Title of the book. Publisher, Date of publication.

MLA Style: Sibley, David Allen. What It’s Like to Be a Bird. From Flying to Nesting, Eating to Singing, What Birds Are Doing, and Why. Alfred A. Knopf, 2020.

APA Style: Sibley, D.A. (2020). What It’s Like to Be a Bird. From Flying to Nesting, Eating to Singing, What Birds Are Doing, and Why . Alfred A. Knopf.

Notes: Use periods, not commas, to separate the data in the entry. Use a hanging indent if the entry is longer than one line. For APA style, do not use the full author’s first name.

Websites or webpages:

  MLA Style: The SB Nation Family of Sites. Pension Plan Puppets: A Toronto Maple Leafs Blog, 2022, www.pensionplanpuppets.com. Accessed 15 Feb. 2022.

APA Style: American Heart Association. (2022, April 11). How to keep your dog’s heart healthy. https://www.heart.org/en/news/2022/04/11/how-to-keep-your-dogs-heart-healthy

Online news article from a newspaper site:

APA Style: Duehren, A. (2022, April 9). Janet Yellen faces challenge to keep pressure on Russia. Wall Street Journal. https://www.wsj.com/articles/janet-yellen-faces-challenge-to-keep-pressure-on-russia-while-addressing-global-consequences-11650366000

Print journal articles:

MLA Style: Booch, Grady. "Patterns in Object-Oriented Design." IEEE Software Engineering, vol. 6, no. 6, 2006, pp. 31-50.

APA Style: Booch, G. (2006). Patterns in object-oriented design. IEEE Software Engineering, 6(6), 31–50.

Note: It is suggested that you include a DOI and a webpage address when referencing either a printed journal article, and electronic journal article, or an journal article that appears in both formats. 

MLA Style: Gamma, Eric, and Peter A. Coad. “Exceptions to the Unified Modeling Language in Python Patterns.” IEEE Software Engineering, vol. 2, no. 6, 8 Mar. 2006, pp. 190-194. O’Reilly Software Engineering Library, https://doi.org/10.1006/se.20061. Accessed 26 May 2009.

APA Style: Masters, H., Barron, J., & Chanda, L. (2017). Motivational interviewing techniques for adolescent populations in substance abuse counseling. NAADAC Notes, 7(8), 7–13. https://www.naadac.com/notes/adolescent-techniques

ML:A Style: @Grady_Booch. “That’s a bold leap over plain old battery power cars.” Twitter, 13 Mar. 2013, 12:06 p.m., https://twitter.com/Grady_Booch/status/1516379006727188483.

APA Style: Westborough Library [@WestboroughLib]. (2022, April 12). Calling all 3rd through 5th grade kids! Join us for the Epic Writing Showdown! Winner receives a prize! Space is limited so register, today. loom.ly/ypaTG9Q [Tweet; thumbnail link to article]. Twitter. https://twitter.com/WestboroughLib/status/1516373550415896588.

Print magazine articles:

General format: Author (last name first), "Article Title." Name of magazine. Volume number, (Date): page numbers.

MLA Style: Stiteler, Sharon. "Tracking Red-Breasted Grosbeak Migration." Minnesota Bird Journal, 7 Sept. 2019, pp. 7-11.

APA Style: Jordan, Jennifer, "Filming at the Top of the World." Museum of Science Magazine. Volume 47, No. 1, (Winter 1998): p. 11.

Print newspaper articles:

General format: Author (last name first), "Article Title." Name of newspaper, city, state of publication. (date): edition if available, section, page number(s).

MLA Style: Adelman, Martin. "Augustus Announces Departure from City Manager Post." New York Times, late ed., 15 February 2020, p. A1

APA Style: Adelman, M. (2020, February 15). Augustus announced departure from city manager post. New York Times, A1.

Encyclopedias:

General Format: Encyclopedia Title, Edition Date. Volume Number, "Article Title," page numbers.

MLA Style: “Gorillas.” The Encyclopedia Brittanica. 15th ed. 2010.

APA Style: Encyclopedia Brittanica, Inc. (1997.) Gorillas. In The Encyclopedia Brittanica (15th ed., pp. 50-51). Encyclopedia Brittanica, Inc.

Personal interviews:

General format: Full name (last name first). Personal Interview. (Occupation.) Date of interview.

MLA Style: Smithfield, Joseph. Personal interview. 19 May 2014.

APA Style: APA does not require a formal citation for a personal interview. Published interviews from other sources should be cited accordingly.

Films and movies:

General format: Title, Director, Distributor, Year.

MLA Style: Fury. Directed by David Ayer, performances by Brad Pitt, Shia LaBeouf, Jon Bernthal, Sony Pictures, 2014.

APA Style: Ayer, D. (Director). (2014). Fury [Film]. Sony Pictures.

Featured High School Resources

Related Resources

About the author

TeacherVision Editorial Staff

The TeacherVision editorial team is comprised of teachers, experts, and content professionals dedicated to bringing you the most accurate and relevant information in the teaching space.

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, automatically generate references for free.

• Knowledge Base
• Referencing
• Harvard Style Bibliography | Format & Examples

Harvard Style Bibliography | Format & Examples

Published on 1 May 2020 by Jack Caulfield . Revised on 7 November 2022.

In Harvard style , the bibliography or reference list provides full references for the sources you used in your writing.

• A reference list consists of entries corresponding to your in-text citations .
• A bibliography sometimes also lists sources that you consulted for background research, but did not cite in your text.

The two terms are sometimes used interchangeably. If in doubt about which to include, check with your instructor or department.

The information you include in a reference varies depending on the type of source, but it usually includes the author, date, and title of the work, followed by details of where it was published. You can automatically generate accurate references using our free reference generator:

Harvard Reference Generator

Instantly correct all language mistakes in your text

Be assured that you'll submit flawless writing. Upload your document to correct all your mistakes.

Table of contents

Formatting a harvard style bibliography, harvard reference examples, referencing sources with multiple authors, referencing sources with missing information, frequently asked questions about harvard bibliographies.

Sources are alphabetised by author last name. The heading ‘Reference list’ or ‘Bibliography’ appears at the top.

Each new source appears on a new line, and when an entry for a single source extends onto a second line, a hanging indent is used:

The only proofreading tool specialized in correcting academic writing

The academic proofreading tool has been trained on 1000s of academic texts and by native English editors. Making it the most accurate and reliable proofreading tool for students.

Correct my document today

Reference list or bibliography entries always start with the author’s last name and initial, the publication date and the title of the source. The other information required varies depending on the source type. Formats and examples for the most common source types are given below.

• Entire book
• Book chapter
• Translated book
• Edition of a book

Journal articles

• Print journal
• Online-only journal with DOI
• Online-only journal without DOI
• General web page
• Online article or blog
• Social media post

Newspapers and magazines

• Newspaper article
• Magazine article

When a source has up to three authors, list all of them in the order their names appear on the source. If there are four or more, give only the first name followed by ‘ et al. ’:

Sometimes a source won’t list all the information you need for your reference. Here’s what to do when you don’t know the publication date or author of a source.

Some online sources, as well as historical documents, may lack a clear publication date. In these cases, you can replace the date in the reference list entry with the words ‘no date’. With online sources, you still include an access date at the end:

When a source doesn’t list an author, you can often list a corporate source as an author instead, as with ‘Scribbr’ in the above example. When that’s not possible, begin the entry with the title instead of the author:

Though the terms are sometimes used interchangeably, there is a difference in meaning:

• A reference list only includes sources cited in the text – every entry corresponds to an in-text citation .
• A bibliography also includes other sources which were consulted during the research but not cited.

In Harvard referencing, up to three author names are included in an in-text citation or reference list entry. When there are four or more authors, include only the first, followed by ‘ et al. ’

In Harvard style referencing , to distinguish between two sources by the same author that were published in the same year, you add a different letter after the year for each source:

• (Smith, 2019a)
• (Smith, 2019b)

Add ‘a’ to the first one you cite, ‘b’ to the second, and so on. Do the same in your bibliography or reference list .

To create a hanging indent for your bibliography or reference list :

• Highlight all the entries
• Click on the arrow in the bottom-right corner of the ‘Paragraph’ tab in the top menu.
• In the pop-up window, under ‘Special’ in the ‘Indentation’ section, use the drop-down menu to select ‘Hanging’.
• Then close the window with ‘OK’.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the ‘Cite this Scribbr article’ button to automatically add the citation to our free Reference Generator.

Caulfield, J. (2022, November 07). Harvard Style Bibliography | Format & Examples. Scribbr. Retrieved 15 April 2024, from https://www.scribbr.co.uk/referencing/harvard-bibliography/

Is this article helpful?

Jack Caulfield

Other students also liked, a quick guide to harvard referencing | citation examples, harvard in-text citation | a complete guide & examples, referencing books in harvard style | templates & examples, scribbr apa citation checker.

An innovative new tool that checks your APA citations with AI software. Say goodbye to inaccurate citations!

authorsafterdark.org

research paper writing tips from the top experts

How to Write a Bibliography for a Research Paper: Guide & Samples

A bibliography is undoubtedly one of the most essential parts of every research paper. A research paper without a bibliography equals poorly conducted research. As such, the importance of a bibliography in a research paper cannot be overemphasized.

To produce well-constructed research, you must consult other authors and use their materials to support your argument. This is the only way to give these authors their credit; by listing them out in your bibliography.

Acknowledging the sources used in writing the topic is an effective approach to avoiding plagiarism in your research. It is also a great way to provide and inform other scholars with sources they may not be aware of. This article seeks to provide the format of bibliography and how to write a good bibliography for your research paper.

What Is a Bibliography in a Research Paper?

Simply put, a bibliography is a list of works used in writing a research paper. Every research paper must contain a list of sources the author used in preparing the research paper. Your source can range from books to scholarly papers, speeches, private records, interviews, letters, websites, and other sources.

Your research paper was not created in a vacuum; you must have consulted other authors or sourced to create enriched content. Therefore, at the end of your research, you must attach a list of all the sources used.

How to Write Bibliography for Research Paper

Writing a bibliography isn’t so hard; all you need is a list of sources used and a format for documenting them. To make things easier for you, prepare a draft comprising all of the sources you must have used. Ensure you include the book’s full title, the author, place of publication, publication date, and publisher. All sources must be listed out alphabetically using the authors’ names.

It is also important to note that there are different formats for writing a bibliography. As such, you must decide which format to employ in your research paper. You can format your work in Modern Language Association (MLA) format, American Psychological Association (APA), or Chicago Manual of Style (CMS). Regardless of which you choose, ensure that it is done correctly.

How to Format a Bibliography

Once you have an idea of how to format a bibliography, most of the work is completed. However, when writing it you need to pay close attention to each format and its content to avoid mixing them up. For printed sources, the bibliography of a research paper should provide the following:

• Author’s name
• Title of publication (provide the title of the article if it’s a journal, magazine, or encyclopedia)
• Date of publication
• Place of Publication of a book
• The publishing house of a book
• Volume number of magazine or encyclopedia
• The page numbers

To document sources from a website, you need to document the following:

• The author or editor’s name if there’s any
• Title of the page
• The organization of the webpage
• The URL of the website
• The date the information was gotten from the website

The common formats for writing a bibliography in a research paper are MLA and APA style. A bibliography is known as “Works Cited” in MLA and on the other hand, it is called “References List” in APA. Though both formats contain similar information about the sources used, there are still slight differences in formatting style. Here’s  what MLA and APA bibliography styles should look like.

How to Write a Bibliography APA

Your research paper’s bibliography must be attached at the end of it with the tag “References” at the center. In addition to this, you need to pay attention to the basics of APA style, such as capitalization, abbreviation, punctuation, underlining or italics, hanging indentation, and others. Here’s a research paper bibliography example in APA style:

Author’s last name, first initial. (Publication date). Book title . City of Publication: Publishing company.

• For encyclopedia:

Author’s last name, first initial. (Date). Title of Article. Title of Encyclopedia (Volumes, pages). City of Publication: Publishing Company.

• For magazine and newspaper articles:

Author’s last name, first initial. (Publication Date). Article title. Periodical title, Volume number (issue number if there’s any), page numbers.

How to Write a Bibliography MLA

Bibliography in MLA format is called “Works cited” and must be arranged in alphabetical order according to the author’s name. Here’s the required format for MLA style for varying sources:

“Author’s last name, first name. Book title. Publication City: Publishing company, publication date.”

• Encyclopedia & dictionary:

“Author’s last name, first name. “Title of Article.” Encyclopedia’s Title. Date.”

• Magazine & newspaper:

“Author’s last name, first name. “Article title.” Periodical title volume Date: inclusive pages.”

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

• Grades 6-12
• School Leaders

FREE Poetry Worksheet Bundle! Perfect for National Poetry Month.

How To Write a Bibliography (Three Styles, Plus Examples)

Give credit where credit is due.

Writing a research paper involves a lot of work. Students need to consult a variety of sources to gather reliable information and ensure their points are well supported. Research papers include a bibliography, which can be a little tricky for students. Learn how to write a bibliography in multiple styles and find basic examples below.

IMPORTANT: Each style guide has its own very specific rules, and they often conflict with one another. Additionally, each type of reference material has many possible formats, depending on a variety of factors. The overviews shown here are meant to guide students in writing basic bibliographies, but this information is by no means complete. Students should always refer directly to the preferred style guide to ensure they’re using the most up-to-date formats and styles.

What is a bibliography?

When you’re researching a paper, you’ll likely consult a wide variety of sources. You may quote some of these directly in your work, summarize some of the points they make, or simply use them to further the knowledge you need to write your paper. Since these ideas are not your own, it’s vital to give credit to the authors who originally wrote them. This list of sources, organized alphabetically, is called a bibliography.

A bibliography should include all the materials you consulted in your research, even if you don’t quote directly from them in your paper. These resources could include (but aren’t limited to):

• Books and e-books
• Periodicals like magazines or newspapers
• Online articles or websites
• Primary source documents like letters or official records

Bibliography vs. References

These two terms are sometimes used interchangeably, but they actually have different meanings. As noted above, a bibliography includes all the materials you used while researching your paper, whether or not you quote from them or refer to them directly in your writing.

A list of references only includes the materials you cite throughout your work. You might use direct quotes or summarize the information for the reader. Either way, you must ensure you give credit to the original author or document. This section can be titled “List of Works Cited” or simply “References.”

Your teacher may specify whether you should include a bibliography or a reference list. If they don’t, consider choosing a bibliography, to show all the works you used in researching your paper. This can help the reader see that your points are well supported, and allow them to do further reading on their own if they’re interested.

Bibliography vs. Citations

Citations refer to direct quotations from a text, woven into your own writing. There are a variety of ways to write citations, including footnotes and endnotes. These are generally shorter than the entries in a reference list or bibliography. Learn more about writing citations here.

What does a bibliography entry include?

Depending on the reference material, bibliography entries include a variety of information intended to help a reader locate the material if they want to refer to it themselves. These entries are listed in alphabetical order, and may include:

• Author/s or creator/s
• Publication date
• Volume and issue numbers
• Publisher and publication city
• Website URL

These entries don’t generally need to include specific page numbers or locations within the work (except for print magazine or journal articles). That type of information is usually only needed in a footnote or endnote citation.

What are the different bibliography styles?

In most cases, writers use one of three major style guides: APA (American Psychological Association), MLA (Modern Language Association), or The Chicago Manual of Style . There are many others as well, but these three are the most common choices for K–12 students.

Many teachers will state their preference for one style guide over another. If they don’t, you can choose your own preferred style. However, you should also use that guide for your entire paper, following their recommendations for punctuation, grammar, and more. This will ensure you are consistent throughout.

Below, you’ll learn how to write a simple bibliography using each of the three major style guides. We’ve included details for books and e-books, periodicals, and electronic sources like websites and videos. If the reference material type you need to include isn’t shown here, refer directly to the style guide you’re using.

APA Style Bibliography and Examples

Source: Verywell Mind

Technically, APA style calls for a list of references instead of a bibliography. If your teacher requires you to use the APA style guide , you can limit your reference list only to items you cite throughout your work.

How To Write a Bibliography (References) Using APA Style

Here are some general notes on writing an APA reference list:

• Title your bibliography section “References” and center the title on the top line of the page.
• Do not center your references; they should be left-aligned. For longer items, subsequent lines should use a hanging indent of 1/2 inch.
• Include all types of resources in the same list.
• Alphabetize your list by author or creator, last name first.
• Do not spell out the author/creator’s first or middle name; only use their initials.
• If there are multiple authors/creators, use an ampersand (&) before the final author/creator.
• Place the date in parentheses.
• Capitalize only the first word of the title and subtitle, unless the word would otherwise be capitalized (proper names, etc.).
• Italicize the titles of books, periodicals, or videos.
• For websites, include the full site information, including the http:// or https:// at the beginning.

Books and E-Books APA Bibliography Examples

For books, APA reference list entries use this format (only include the publisher’s website for e-books).

Last Name, First Initial. Middle Initial. (Publication date). Title with only first word capitalized . Publisher. Publisher’s website

• Wynn, S. (2020). City of London at war 1939–45 . Pen & Sword Military. https://www.pen-and-sword.co.uk/City-of-London-at-War-193945-Paperback/p/17299

Periodical APA Bibliography Examples

For journal or magazine articles, use this format. If you viewed the article online, include the URL at the end of the citation.

Last Name, First Initial. Middle Initial. (Publication date). Title of article. Magazine or Journal Title (Volume number) Issue number, page numbers. URL

• Bell, A. (2009). Landscapes of fear: Wartime London, 1939–1945. Journal of British Studies (48) 1, 153–175. https://www.jstor.org/stable/25482966

Here’s the format for newspapers. For print editions, include the page number/s. For online articles, include the full URL.

Last Name, First Initial. Middle Initial. (Year, Month Date) Title of article. Newspaper title. Page number/s. URL

• Blakemore, E. (2022, November 12) Researchers track down two copies of fossil destroyed by the Nazis.  The Washington Post. https://www.washingtonpost.com/science/2022/11/12/ichthyosaur-fossil-images-discovered/

Electronic APA Bibliography Examples

For articles with a specific author on a website, use this format.

Last Name, First Initial. Middle Initial. (Year, Month Date). Title . Site name. URL

• Wukovits, J. (2023, January 30). A World War II survivor recalls the London Blitz . British Heritage . https://britishheritage.com/history/world-war-ii-survivor-london-blitz

When an online article doesn’t include a specific author or date, list it like this:

Title . (Year, Month Date). Site name. Retrieved Month Date, Year, from URL

• Growing up in the Second World War . (n.d.). Imperial War Museums. Retrieved May 12, 2023, from https://www.iwm.org.uk/history/growing-up-in-the-second-world-war

When you need to list a YouTube video, use the name of the account that uploaded the video, and format it like this:

Name of Account. (Upload year, month day). Title [Video]. YouTube. URL

• War Stories. (2023, January 15). How did London survive the Blitz during WW2? | Cities at war: London | War stories [Video]. YouTube. https://youtu.be/uwY6JlCvbxc

For more information on writing APA bibliographies, see the APA Style Guide website.

APA Bibliography (Reference List) Example Pages

Source: Simply Psychology

More APA example pages:

• Western Australia Library Services APA References Example Page
• Ancilla College APA References Page Example
• Scribbr APA References Page Example

MLA Style Bibliography Examples

Source: PressBooks

MLA style calls for a Works Cited section, which includes all materials quoted or referred to in your paper. You may also include a Works Consulted section, including other reference sources you reviewed but didn’t directly cite. Together, these constitute a bibliography. If your teacher requests an MLA Style Guide bibliography, ask if you should include Works Consulted as well as Works Cited.

How To Write a Bibliography (Works Cited and Works Consulted) in MLA Style

For both MLA Works Cited and Works Consulted sections, use these general guidelines:

• Start your Works Cited list on a new page. If you include a Works Consulted list, start that on its own new page after the Works Cited section.
• Center the title (Works Cited or Works Consulted) in the middle of the line at the top of the page.
• Align the start of each source to the left margin, and use a hanging indent (1/2 inch) for the following lines of each source.
• Alphabetize your sources using the first word of the citation, usually the author’s last name.
• Include the author’s full name as listed, last name first.
• Capitalize titles using the standard MLA format.
• Leave off the http:// or https:// at the beginning of a URL.

Books and E-Books MLA Bibliography Examples

For books, MLA reference list entries use this format. Add the URL at the end for e-books.

Last Name, First Name Middle Name. Title . Publisher, Date. URL

• Wynn, Stephen. City of London at War 1939–45 . Pen & Sword Military, 2020. www.pen-and-sword.co.uk/City-of-London-at-War-193945-Paperback/p/17299

Periodical MLA Bibliography Examples

Here’s the style format for magazines, journals, and newspapers. For online articles, add the URL at the end of the listing.

For magazines and journals:

Last Name, First Name. “Title: Subtitle.” Name of Journal , volume number, issue number, Date of Publication, First Page Number–Last Page Number.

• Bell, Amy. “Landscapes of Fear: Wartime London, 1939–1945.” Journal of British Studies , vol. 48, no. 1, pp. 153–175. www.jstor.org/stable/25482966

When citing newspapers, include the page number/s for print editions or the URL for online articles.

Last Name, First Name. “Title of article.” Newspaper title. Page number/s. Year, month day. Page number or URL

• Blakemore, Erin. “Researchers Track Down Two Copies of Fossil Destroyed by the Nazis.” The Washington Post. 2022, Nov. 12. www.washingtonpost.com/science/2022/11/12/ichthyosaur-fossil-images-discovered/

Electronic MLA Bibliography Examples

Last Name, First Name. Year. “Title.” Month Day, Year published. URL

• Wukovits, John. 2023. “A World War II Survivor Recalls the London Blitz.” January 30,   2023. https://britishheritage.com/history/world-war-ii-survivor-london-blitz

Website. n.d. “Title.” Accessed Day Month Year. URL.

• Imperial War Museum. n.d. “Growing Up in the Second World War.” Accessed May 9, 2023. https://www.iwm.org.uk/history/growing-up-in-the-second-world-war.

Here’s how to list YouTube and other online videos.

Creator, if available. “Title of Video.” Website. Uploaded by Username, Day Month Year. URL.

• “How did London survive the Blitz during WW2? | Cities at war: London | War stories.” YouTube . Uploaded by War Stories, 15 Jan. 2023. youtu.be/uwY6JlCvbxc.

For more information on writing MLA style bibliographies, see the MLA Style website.

MLA Bibliography (Works Cited) Example Pages

Source: The Visual Communication Guy

More MLA example pages:

• Writing Commons Sample Works Cited Page
• Scribbr MLA Works Cited Sample Page
• Montana State University MLA Works Cited Page

Chicago Manual of Style Bibliography Examples

The Chicago Manual of Style (sometimes called “Turabian”) actually has two options for citing reference material : Notes and Bibliography and Author-Date. Regardless of which you use, you’ll need a complete detailed list of reference items at the end of your paper. The examples below demonstrate how to write that list.

How To Write a Bibliography Using The Chicago Manual of Style

Source: South Texas College

Here are some general notes on writing a Chicago -style bibliography:

• You may title it “Bibliography” or “References.” Center this title at the top of the page and add two blank lines before the first entry.
• Left-align each entry, with a hanging half-inch indent for subsequent lines of each entry.
• Single-space each entry, with a blank line between entries.
• Include the “http://” or “https://” at the beginning of URLs.

Books and E-Books Chicago Manual of Style Bibliography Examples

For books, Chicago -style reference list entries use this format. (For print books, leave off the information about how the book was accessed.)

Last Name, First Name Middle Name. Title . City of Publication: Publisher, Date. How e-book was accessed.

• Wynn, Stephen. City of London at War 1939–45 . Yorkshire: Pen & Sword Military, 2020. Kindle edition.

Periodical Chicago Manual of Style Bibliography Examples

For journal and magazine articles, use this format.

Last Name, First Name. Year of Publication. “Title: Subtitle.” Name of Journal , Volume Number, issue number, First Page Number–Last Page Number. URL.

• Bell, Amy. 2009. “Landscapes of Fear: Wartime London, 1939–1945.” Journal of British Studies, 48 no. 1, 153–175. https://www.jstor.org/stable/25482966.

When citing newspapers, include the URL for online articles.

Last Name, First Name. Year of Publication. “Title: Subtitle.” Name of Newspaper , Month day, year. URL.

• Blakemore, Erin. 2022. “Researchers Track Down Two Copies of Fossil Destroyed by the Nazis.” The Washington Post , November 12, 2022. https://www.washingtonpost.com/science/2022/11/12/ichthyosaur-fossil-images-discovered/.

Electronic Chicago Manual of Style Bibliography Examples

Last Name, First Name Middle Name. “Title.” Site Name . Year, Month Day. URL.

• Wukovits, John. “A World War II Survivor Recalls the London Blitz.” British Heritage. 2023, Jan. 30. britishheritage.com/history/world-war-ii-survivor-london-blitz.

“Title.” Site Name . URL. Accessed Day Month Year.

• “Growing Up in the Second World War.” Imperial War Museums . www.iwm.org.uk/history/growing-up-in-the-second-world-war. Accessed May 9, 2023.

Creator or Username. “Title of Video.” Website video, length. Month Day, Year. URL.

• War Stories. “How Did London Survive the Blitz During WW2? | Cities at War: London | War Stories.” YouTube video, 51:25. January 15, 2023. https://youtu.be/uwY6JlCvbxc.

For more information on writing Chicago -style bibliographies, see the Chicago Manual of Style website.

Chicago Manual of Style Bibliography Example Pages

Source: Chicago Manual of Style

More Chicago example pages:

• Scribbr Chicago Style Bibliography Example
• Purdue Online Writing Lab CMOS Bibliography Page
• Bibcitation Sample Chicago Bibliography

Now that you know how to write a bibliography, take a look at the Best Websites for Teaching & Learning Writing .

Plus, get all the latest teaching tips and ideas when you sign up for our free newsletters .

You Might Also Like

Generate accurate APA citations for free

• Knowledge Base
• APA Style 7th edition
• APA format for academic papers and essays

APA Formatting and Citation (7th Ed.) | Generator, Template, Examples

Published on November 6, 2020 by Raimo Streefkerk . Revised on January 17, 2024.

The 7th edition of the APA Publication Manual provides guidelines for clear communication , citing sources , and formatting documents. This article focuses on paper formatting.

Generate accurate APA citations with Scribbr

Throughout your paper, you need to apply the following APA format guidelines:

• Set page margins to 1 inch on all sides.
• Double-space all text, including headings.
• Indent the first line of every paragraph 0.5 inches.
• Use an accessible font (e.g., Times New Roman 12pt., Arial 11pt., or Georgia 11pt.).
• Include a page number on every page.

Let an expert format your paper

Our APA formatting experts can help you to format your paper according to APA guidelines. They can help you with:

• Margins, line spacing, and indentation
• Font and headings
• Running head and page numbering

Table of contents

How to set up apa format (with template), apa alphabetization guidelines, apa format template [free download], page header, headings and subheadings, reference page, tables and figures, frequently asked questions about apa format.

Prevent plagiarism. Run a free check.

References are ordered alphabetically by the first author’s last name. If the author is unknown, order the reference entry by the first meaningful word of the title (ignoring articles: “the”, “a”, or “an”).

Why set up APA format from scratch if you can download Scribbr’s template for free?

Student papers and professional papers have slightly different guidelines regarding the title page, abstract, and running head. Our template is available in Word and Google Docs format for both versions.

• Student paper: Word | Google Docs
• Professional paper: Word | Google Docs

In an APA Style paper, every page has a page header. For student papers, the page header usually consists of just a page number in the page’s top-right corner. For professional papers intended for publication, it also includes a running head .

A running head is simply the paper’s title in all capital letters. It is left-aligned and can be up to 50 characters in length. Longer titles are abbreviated .

APA headings have five possible levels. Heading level 1 is used for main sections such as “ Methods ” or “ Results ”. Heading levels 2 to 5 are used for subheadings. Each heading level is formatted differently.

Want to know how many heading levels you should use, when to use which heading level, and how to set up heading styles in Word or Google Docs? Then check out our in-depth article on APA headings .

The title page is the first page of an APA Style paper. There are different guidelines for student and professional papers.

Both versions include the paper title and author’s name and affiliation. The student version includes the course number and name, instructor name, and due date of the assignment. The professional version includes an author note and running head .

For more information on writing a striking title, crediting multiple authors (with different affiliations), and writing the author note, check out our in-depth article on the APA title page .

The abstract is a 150–250 word summary of your paper. An abstract is usually required in professional papers, but it’s rare to include one in student papers (except for longer texts like theses and dissertations).

The abstract is placed on a separate page after the title page . At the top of the page, write the section label “Abstract” (bold and centered). The contents of the abstract appear directly under the label. Unlike regular paragraphs, the first line is not indented. Abstracts are usually written as a single paragraph without headings or blank lines.

Directly below the abstract, you may list three to five relevant keywords . On a new line, write the label “Keywords:” (italicized and indented), followed by the keywords in lowercase letters, separated by commas.

APA Style does not provide guidelines for formatting the table of contents . It’s also not a required paper element in either professional or student papers. If your instructor wants you to include a table of contents, it’s best to follow the general guidelines.

Place the table of contents on a separate page between the abstract and introduction. Write the section label “Contents” at the top (bold and centered), press “Enter” once, and list the important headings with corresponding page numbers.

The APA reference page is placed after the main body of your paper but before any appendices . Here you list all sources that you’ve cited in your paper (through APA in-text citations ). APA provides guidelines for formatting the references as well as the page itself.

Creating APA Style references

Play around with the Scribbr Citation Example Generator below to learn about the APA reference format of the most common source types or generate APA citations for free with Scribbr’s APA Citation Generator .

Formatting the reference page

Write the section label “References” at the top of a new page (bold and centered). Place the reference entries directly under the label in alphabetical order.

Finally, apply a hanging indent , meaning the first line of each reference is left-aligned, and all subsequent lines are indented 0.5 inches.

Tables and figures are presented in a similar format. They’re preceded by a number and title and followed by explanatory notes (if necessary).

Use bold styling for the word “Table” or “Figure” and the number, and place the title on a separate line directly below it (in italics and title case). Try to keep tables clean; don’t use any vertical lines, use as few horizontal lines as possible, and keep row and column labels concise.

Keep the design of figures as simple as possible. Include labels and a legend if needed, and only use color when necessary (not to make it look more appealing).

Check out our in-depth article about table and figure notes to learn when to use notes and how to format them.

The easiest way to set up APA format in Word is to download Scribbr’s free APA format template for student papers or professional papers.

Alternatively, you can watch Scribbr’s 5-minute step-by-step tutorial or check out our APA format guide with examples.

APA Style papers should be written in a font that is legible and widely accessible. For example:

• Times New Roman (12pt.)
• Arial (11pt.)
• Calibri (11pt.)
• Georgia (11pt.)

The same font and font size is used throughout the document, including the running head , page numbers, headings , and the reference page . Text in footnotes and figure images may be smaller and use single line spacing.

You need an APA in-text citation and reference entry . Each source type has its own format; for example, a webpage citation is different from a book citation .

Use Scribbr’s free APA Citation Generator to generate flawless citations in seconds or take a look at our APA citation examples .

Yes, page numbers are included on all pages, including the title page , table of contents , and reference page . Page numbers should be right-aligned in the page header.

To insert page numbers in Microsoft Word or Google Docs, click ‘Insert’ and then ‘Page number’.

APA format is widely used by professionals, researchers, and students in the social and behavioral sciences, including fields like education, psychology, and business.

Be sure to check the guidelines of your university or the journal you want to be published in to double-check which style you should be using.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Streefkerk, R. (2024, January 17). APA Formatting and Citation (7th Ed.) | Generator, Template, Examples. Scribbr. Retrieved April 15, 2024, from https://www.scribbr.com/apa-style/format/

Is this article helpful?

Raimo Streefkerk

Other students also liked, apa title page (7th edition) | template for students & professionals, creating apa reference entries, beginner's guide to apa in-text citation, scribbr apa citation checker.

An innovative new tool that checks your APA citations with AI software. Say goodbye to inaccurate citations!

How to Write a Research Paper: Annotated Bibliography

• Anatomy of a Research Paper
• Developing a Research Focus
• Background Research Tips
• Searching Tips
• Scholarly Journals vs. Popular Journals
• Thesis Statement
• Annotated Bibliography
• Citing Sources
• Evaluating Sources
• Literature Review
• Academic Integrity
• Scholarship as Conversation
• Understanding Fake News
• Data, Information, Knowledge

What is an Annotated Bibliography?

UMary Writing Center

Check out the resources available from the  Writing Center . 

Write an Annotated Bibliography

What is an annotated bibliography?

It is a list of citations for various books, articles, and other sources on a topic. 

An annotation is a short summary and/or critical evaluation of a source.

Annotated bibliographies answer the question: "What would be the most relevant, most useful, or most up-to-date sources for this topic?"

 Annotated bibliographies can be part of a larger research project, or can be a stand-alone report in itself. 

Annotation versus abstracts 

An abstract is a paragraph at the beginning of the paper that discusses the main point of the original work. They typically do not include evaluation comments. 

Annotations can either be descriptive or evaluative. The annotated bibliography looks like a works cited page but includes an annotation after each source cited. 

Types of Annotations: 

Descriptive Annotations: Focuses on description. Describes the source by answering the following questions. 

Who wrote the document?

What does the document discuss?

When and where was the document written? 

Why was the document produced?

How was it provided to the public?

Evaluative Annotations: Focuses on description and evaluation. Includes a summary and critically assess the work for accuracy, relevance, and quality. 

Evaluative annotations help you learn about your topic, develop a thesis statement, decide if a specific source will be useful for your assignment, and determine if there is enough valid information available to complete your project.

What does the annotation include?

Depending on your assignment and style guide, annotations may include some or all of the following information. 

• Should be no more than 150 words or 4 to 6 sentences long. 
• What is the main focus or purpose of the work?
• Who is the intended audience?
• ​How useful or relevant was the article to your topic?
• Was there any unique features that useful to you?
• What is the background and credibility of the author?
• What are any conclusions or observations that your reached about the article?

Which citation style to use?

There are many styles manuals with specific instructions on how to format your annotated bibliography. This largely depends on what your instructor prefers or your subject discipline. Check out our citation guides for more information. 

Additional Information

Why doesn't APA have an official APA-approved format for annotated bibliographies?

Always consult your instructor about the format of an annotated bibliography for your class assignments. These guides provide you with examples of various styles for annotated bibliographies and they may not be in the format required by your instructor. 

Citation Examples and Annotations

Book Citation with Descriptive Annotation

Liroff, R. A., & G. G. Davis. (1981). Protecting open space: Land use control in the Adirondack Park. Cambridge, MA: Ballinger.

This book describes the implementation of regional planning and land use regulation in the Adirondack Park in upstate New York. The authors provide program evaluations of the Adirondack Park Agency’s regulatory and local planning assistance programs.

Journal Article Citation with Evaluative Annotation

Gottlieb, P. D. (1995). The “golden egg” as a natural resource: Toward a normative theory of growth management. Society and Natural Resources, 8, (5): 49-56.

This article explains the dilemma faced by North American suburbs, which demand both preservation of local amenities (to protect quality of life) and physical development (to expand the tax base). Growth management has been proposed as a policy solution to this dilemma. An analogy is made between this approach and resource economics. The author concludes that the growth management debate raises legitimate issues of sustainability and efficiency.

Examples were taken from http://lib.calpoly.edu/support/how-to/write-an-annotated-bibliography/#samples

Book Citation

Lee, Seok-hoon, Yong-pil Kim, Nigel Hemmington, and Deok-kyun Yun. “Competitive Service Quality Improvement (CSQI): A Case Study in the Fast-Food Industry.” Food Service Technology 4 (2004): 75-84.

In this highly technical paper, three industrial engineering professors in Korea and one services management professor in the UK discuss the mathematical limitations of the popular SERVQUAL scales. Significantly, they also aim to measure service quality in the fast-food industry, a neglected area of study. Unfortunately, the paper’s sophisticated analytical methods make it inaccessible to all but the most expert of researchers.

Battle, Ken. “Child Poverty: The Evolution and Impact of Child Benefits.”  A Question of Commitment: Children's Rights in Canada . Ed. Katherine Covell and R.Brian Howe. Waterloo, ON: Wilfrid Laurier University Press. 2007. 21-44.

             Ken Battle draws on a close study of government documents, as well as his own research as an extensively-published policy analyst, to explain Canadian child benefit programs.  He outlines some fundamental assumptions supporting the belief that all society members should contribute to the upbringing of children.  His comparison of child poverty rates in a number of countries is a useful wake-up to anyone assuming Canadian society is doing a good job of protecting children.  Battle pays particular attention to the National Child Benefit (NCB), arguing that it did not deserve to be criticized by politicians and journalists.  He outlines the NCB’s development, costs, and benefits, and laments that the Conservative government scaled it back in favour of the inferior Universal Child Care Benefit (UCCB).  However, he relies too heavily on his own work; he is the sole or primary author of almost half the sources in his bibliography.  He could make this work stronger by drawing from others' perspectives and analyses.  However, Battle does offer a valuable source for this essay, because the chapter provides a concise overview of government-funded assistance currently available to parents.  This offers context for analyzing the scope and financial reality of child poverty in Canada.

Journal Article Example

  Kerr, Don and Roderic Beaujot. “Child Poverty and Family Structure in Canada, 1981-1997.”  Journal of Comparative Family Studies  34.3 (2003): 321-335.

             Sociology professors Kerr and Beaujot analyze the demographics of impoverished families.  Drawing on data from Canada’s annual Survey of Consumer Finances, the authors consider whether each family had one or two parents, the age of single parents, and the number of children in each household.  They analyze child poverty rates in light of both these demographic factors and larger economic issues.  Kerr and Beaujot use this data to argue that. 

Examples were taken from  http://libguides.enc.edu/writing_basics/ annotatedbib/mla

Check out these resources for more information about Annotated Bibliographies. 

• Purdue Owl- Annotated Bibliographies
• University of North Carolina at Chapel Hill- Annotated Bibliographies
• << Previous: Thesis Statement
• Next: Citing Sources >>
• Last Updated: Apr 4, 2024 5:51 PM
• URL: https://libguide.umary.edu/researchpaper

Research Process: Bibliographic Information

• Selecting a Topic
• Background Information
• Narrowing the Topic
• Library Terms
• Generating Keywords
• Boolean Operators
• Search Engine Strategies
• Google Searching
• Basic Internet Terms
• Research & The Web
• Search Engines
• Evaluating Books
• Evaluating Articles
• Evaluating Websites

Bibliographic Information

• Off Campus Access
• Periodical Locator

What is a bibliography?

A bibliography is a list of works on a subject or by an author that were used or consulted to write a research paper, book or article. It can also be referred to as a list of works cited. It is usually found at the end of a book, article or research paper. 

Gathering Information

Regardless of what citation style is being used, there are key pieces of information that need to be collected in order to create the citation.

For books and/or journals:

• Author name
• Title of publication 
• Article title (if using a journal)
• Date of publication
• Place of publication
• Volume number of a journal, magazine or encyclopedia
• Page number(s)

For websites:

• Author and/or editor name
• Title of the website
• Company or organization that owns or posts to the website
• URL (website address)
• Date of access 

This section provides two examples of the most common cited sources: a print book and an online journal retrieved from a research database. 

Book - Print

For print books, bibliographic information can be found on the  TITLE PAGE . This page has the complete title of the book, author(s) and publication information.

The publisher information will vary according to the publisher - sometimes this page will include the name of the publisher, the place of publication and the date.

For this example :  Book title: HTML, XHTML, and CSS Bible Author: Steven M. Schafer Publisher: Wiley Publications, Inc.

If you cannot find the place or date of publication on the title page, refer to the  COPYRIGHT PAGE  for this information. The copyright page is the page behind the title page, usually written in a small font, it carries the copyright notice, edition information, publication information, printing history, cataloging data, and the ISBN number.

For this example : Place of publication: Indianapolis, IN Date of publication: 2010

Article - Academic OneFile Database

In the article view:

Bibliographic information can be found under the article title, at the top of the page. The information provided in this area is  NOT  formatted according to any style.

Citations can also be found at the bottom of the page; in an area titled  SOURCE CITATION . The database does not specify which style is used in creating this citation, so be sure to double check it against the style rules for accuracy.

Article - ProQuest Database

Bibliographic information can be found under the article title, at the top of the page. The information provided in this area is  NOT  formatted according to any style. 

Bibliographic information can also be found at the bottom of the page; in an area titled  INDEXING . (Not all the information provided in this area is necessary for creating citations, refer to the rules of the style being used for what information is needed.)

Other databases have similar formats - look for bibliographic information under the article titles and below the article body, towards the bottom of the page. 

• << Previous: Plagiarism
• Next: Research Databases >>
• Last Updated: Jan 8, 2024 10:20 AM
• URL: https://pgcc.libguides.com/researchprocess

• MJC Library & Learning Center
• Research Guides

Format Your Paper & Cite Your Sources

• Harvard Style
• Citing Sources
• Avoid Plagiarism
• MLA Style (8th/9th ed.)
• APA Style, 7th Edition
• Chicago Style

What is Harvard Style?

What you need to know, harvard style tutorial.

• Other Styles
• Annotated Bibliographies
• How to Create an Attribution

Harvard Style

The Harvard referencing system is known as the Author-Date style . It emphasizes the name of the creator of a piece of information and the date of publication, with the list of references in alphabetical order at the end of your paper.

Unlike other citation styles, there is no single, definitive version of Harvard Style. Therefore, you may see a variation in features such as punctuation, capitalization, abbreviations, and the use of italics. 

Always check with your instructor and follow the rules he or she gives you.

• Harvard Style Guidelines Your class handout
• Harvard Referencing Quick Guide From Staffordshire University

Harvard Style will affect your paper in two places:

• In-text citations in the body of your paper, and
• The reference list at the end of your paper
• All in-text citations should be listed in the reference list at the end of your paper.
• Reference list entries need to contain all the information that someone reading your paper would need in order to find your source.
• Reference lists in Harvard Style are arranged alphabetically by first author.
• Begin your Reference list on a new page after your text and number it consecutively.

Sample References List:

Click on the Links Below to See Additional Examples:

• Sample Paper Paper provided by Kurt Olson
• Harvard Citation Examples Document created by The University of Western Australia

Click on the image below to launch this tutorial that was created by the University of Leeds. The section on Citing in Text is especially useful.

• << Previous: Chicago Style
• Next: Other Styles >>
• Last Updated: Mar 20, 2024 11:36 AM
• URL: https://libguides.mjc.edu/citeyoursources

Except where otherwise noted, this work is licensed under CC BY-SA 4.0 and CC BY-NC 4.0 Licenses .

Purdue Online Writing Lab Purdue OWL® College of Liberal Arts

MLA Sample Paper

Welcome to the Purdue OWL

This page is brought to you by the OWL at Purdue University. When printing this page, you must include the entire legal notice.

Copyright ©1995-2018 by The Writing Lab & The OWL at Purdue and Purdue University. All rights reserved. This material may not be published, reproduced, broadcast, rewritten, or redistributed without permission. Use of this site constitutes acceptance of our terms and conditions of fair use.

This resource contains a sample MLA paper that adheres to the 2016 updates. To download the MLA sample paper, click this link .

Home / MLA Sample Paper

MLA Sample Paper

Mla sample paper #1.

If you’ve been wondering how to produce a research paper that is strong in both formatting and writing, you’ve come to the right place.

Check out our first sample paper below. It is a helpful and clearly labeled visual aid to refer to. Note that while these sample papers do not include MLA abstracts , you should check with your instructor to see if an abstract should be included.

Visual Sample Paper

The example research paper below is one that was written in college for a course on the Inklings. The Inklings were a group of writers in England before WWII, including C.S. Lewis and J.R.R. Tolkien.

The abbreviated MLA paper below (linked here without annotations) is about J.R.R. Tolkien’s Lord of the Rings and how the author used myth, story, and song to link all of his works together. Tolkien is famous for creating a fantasy universe called Middle-earth, which readers can’t truly understand until they read all of the books about Middle-earth ( The Silmarillian, The Hobbit, and The Lord of the Rings ).

Since we’re here to learn how to format an essay, we’ve pointed out some important things about the paper to help you write a correctly formatted essay.

For starters, the essay is in MLA format. That means it follows the style manual of the Modern Language Association, which tells you how to format the paper itself and every source you cite. You’ll also see notes like how long a paragraph should be, how to use commas properly, and how to correctly punctuate a title. Some of these guidelines are different from those in APA format , so be sure to confirm you are using the correct style in your paper.

Pay special attention to the MLA format works cited. We only used one type of source (books), but both citations are correct according to the 9th edition of MLA, published in 2021. When you’re writing your own paper, you need to make sure you always use the most recent edition of the style manual. You’ll also want to check with your instructor to see if you need to include an MLA annotated bibliography with your paper, which contains additional information summarizing and evaluating each source after the regular citation.

Whether you need MLA, APA citations , or Chicago style notes, look up the latest edition before turning in a paper.

MLA Sample Paper #2

See below for an example paper or click below to download it as a Word Document.

The MLA header should be one inch from the top and left margins. The heading and the entire paper should be double spaced.

Eli YaffarabeProfessor Rapheor

28 August 2018

Privatization of Prisons in Texas

              The privatization of governmental services has increased dramatically in the past decade as local, state, and federal agencies have searched for ways to cut costs while still meeting their mandated responsibility to provide various public services. This privatizing trend has particularly affected the criminal justice system. Since the early 1990s, privatized correctional facilities have increased significantly, nationally and statewide. This policy has far-ranging consequences not only within the criminal justice system, but as an instructive example for government officials when considering the costs and benefits of privatization as a public policy option. By 2001, thirty states, the District of Columbia, and Puerto Rico had privately-operated correctional facilities (Austin and Coventry 4). This movement has incited considerable debate and controversy, mainly because prison privatization calls for giving the private sector direct control over the lives of a captive human population.

Surprisingly, there has been little objective and concrete analysis of the privatization of prisons in the United States. This is probably for two reasons: first, ideological arguments on the matter have pushed out substantive research, and second, because this trend has only recently accelerated in the U.S. and mainly on a state level. However, case studies and statistics at the state level are more accessible. With capacity for over 30,000 prisoners in 43 facilities, the state of Texas has privatized more of its prison system than any state in the nation (McDonald and Patten Jr. iv).

Yaffarabe 2

              Public policy concerning the criminal justice system has become more daunting and important in the last decade. The problems in the system are twofold: an overcrowding prison population, mainly due to “three strikes” legislation and reducing early parole; and the costs of operating prisons with this growing population (Austin and Coventry). According to the most recent U.S. Department of Justice survey, slightly over 2.2 million people were incarcerated in correctional facilities in this country in 2003. In comparison, in 1993, 1.37 million people were imprisoned in this country (Beck and Harrison 1).

At the same time, the growth of privately operated correctional facilities has increased significantly in this country. Private prisons now hold 95,522 inmates in this country, which is 6.5 percent of total prisoners (Beck and Harrison 5). In Texas, 16,570 inmates (10 percent of its prison population) are held in private facilities, about 10,000 more than the next highest state. Furthermore, six states had at least 25 percent of their prison population housed in private prisons, led by New Mexico (44%), Alaska (31%), and Montana (29%). These current statistics show that while state governments have been forced to manage and operate overcrowded and over-capacity prisons at considerable costs, many have turned to the private sector to operate prisons (McDonald and Patten Jr.). According to the General Accounting Office, prison operating costs have grown steadily since 1980, increasing almost 550 percent since 1980 based on inflation-adjusted dollars (Austin and Coventry 1).

Prison privatization started in the early 1980s, ostensibly to ease the burden on taxpayers by offering financial relief to private companies to run state prisons. Thomas Beasley founded Corrections Corporation of America in 1983, “the nation’s leader in the construction and management of private prisons” (Darling). That year, Corrections Corporation of America set up the first privately-operated prison in Tennessee. Since then, the number of private

Yaffarabe 3

correctional facility firms has grown to 14 (Austin and Coventry 3). The privatization of prisons occurs in two ways. First, state government can contract out (or outsource) specific services in a correctional facility to a private company after a bidding process. Second, and more radically, private companies build their own privately-managed prisons and contract with state governments to house their inmates. This latter approach, giving private correctional facility firms wide latitude over inmates, is taken in the Texas criminal justice system. In fact, many of these privately operated facilities “have no relationship at all with the state governments in these states, other than an obligation to pay corporate income taxes” (McDonald and Patten Jr. v).

(Due to its length, the remainder of this sample paper is omitted).

Yaffarabe 4

Works Cited Page

Austin, James, and Garry Coventry. Emerging Issues on Privatized Prisons . Bureau of Justice Assistance, Feb. 2001, www.ncjrs.gov/pdffiles1/bja/181249.pdf.

Beck, Allen J., and Paige Harrison. Prisoners in 2003 . Bureau of Justice Statistics, Nov. 2004, www.bjs.gov/content/pub/pdf/p03.pdf.

McDonald, Douglas, and Carl Patten Jr. Governments’ Management of Private Prisons . Abt Associates, 15 Sept. 2003, www.ncjrs.gov/pdffiles1/nij/grants/203968.pdf.

Darling, Michael. “Pitt News: University of Pittsburgh Shouldn’t Lend Its Name to Prison Privatization.” CorpWatch , 15 Nov. 2004, corpwatch.org/article/pitt-news-univeristy-pittsburgh-shouldnt-lend-its-name-prison-privatization.

MLA Formatting Guide

MLA Formatting

• Annotated Bibliography
• Bibliography
• Block Quotes
• et al Usage
• In-text Citations
• Paraphrasing
• Page Numbers
• Sample Paper
• Works Cited
• MLA 8 Updates
• MLA 9 Updates
• View MLA Guide

Citation Examples

• Book Chapter
• Journal Article
• Magazine Article
• Newspaper Article
• Website (no author)
• View all MLA Examples

How useful was this post?

Click on a star to rate it!

We are sorry that this post was not useful for you!

Let us improve this post!

Tell us how we can improve this post?

An in-text citation is a short citation that is placed next to the text being cited. The basic element needed for an in-text citation is the author’s name . The publication year is not required in in-text citations. Sometimes, page numbers or line numbers are also included, especially when text is quoted from the source being cited. In-text citations are mentioned in the text in two ways: as a citation in prose or a parenthetical citation.

Citation in prose

Citations in prose are incorporated into the text and act as a part of the sentence. Usually, citations in prose use the author’s full name when cited the first time in the text. Thereafter, only the surname is used. Avoid including the middle initial even if it is present in the works-cited-list entry. An example of the first citation in prose for a source with one author is given below:

Doug Barry explains the status of the UK.

Parenthetical

Parenthetical citations add only the author’s surname at the end of the sentence in parentheses. An example of a parenthetical citation is given below:

The status of the UK is explained (Barry).

Examples of in-text citations

Here are a few examples of in-text citations for works with various numbers and types of authors:

Use both the first name and surname of the author if you are mentioning the author for the first time in the prose. In subsequent occurrences, use only the author’s surname. Always use only the author’s surname in parenthetical citations.

Citation in prose:

First mention: Stephen George asserts …. (17).

Subsequent occurrences: George argues …. (17).

Parenthetical:

…. (George 17).

Two authors

Use the first name and surname of both authors if you are mentioning the work for the first time in the prose. In subsequent occurrences, use only the surnames of the two authors. Always use only the authors’ surnames in parenthetical citations. Use “and” to separate the two authors in parenthetical citations.

First mention: Kane Williams and Clark Ronald ….

Subsequent occurrences: Williams and Ronald ….

…. (Williams and Ronald).

Three or more authors

For citations in prose, use the first name and surname of the first author followed by “and others” or “and colleagues.” For parenthetical citations, use only the surname of the first author followed by “et al.”

Krishnan Sethu and colleagues…. or Krishnan Sethu and others ….

…. (Sethu et al.).

Corporate author

For citations in prose, treat the corporate author like you would treat the author’s name. For parenthetical citations, shorten the organization name to the shortest noun phrase. For example, shorten the Modern Language Association of America to Modern Language Association.

The Language Literary Association of Canada….

…. (Language Literary Association).

If there is no author for the source, use the source’s title in place of the author’s name for both citations in prose and parenthetical citations.

When you add such in-text citations, italicize the text of the title. If the source title is longer than a noun phrase, use a shortened version of the title. For example, shorten the title Fantastic Beasts and Where to Find Them to Fantastic Beasts .

Endgame explains …. (121).

…. ( Endgame 121).

In MLA style, two types of citations are used to cite a source: a short citation used within the text (called the in-text citation) and a full citation (called the works cited list entry) within the works cited list, which appears at the end of a paper.

The works cited list entry provides the complete details of a source. An in-text citation is a short citation that is placed next to the text being cited. The in-text citation lets the reader know that the information is derived from the cited source, and helps the reader find the full citation within the works cited list.

In order to properly cite a source in MLA style, you must have both citation types in your paper. Every in-text citation has a works cited list entry. Every works cited list entry has at least one (maybe more) corresponding in-text citation.

In-text citations

The basic element needed for an in-text citation is the author’s surname . The publication year is not required in in-text citations. Sometimes, page numbers or line numbers are also included, especially when text is quoted from the source being cited.

First mention: Sian Anderson studies ….

Subsequent occurrences: Anderson analyzes ….

….(Anderson)

or if quoting directly:

…(Anderson 9)

First mention: Paul Fin and Anna Gabriel ….

Subsequent occurrences: Fin and Gabriel ….

….(Fin and Gabriel)

…(Fin and Gabriel 27)

Paul Hill and colleagues…. or Paul Hill and others ….

….(Hill et al.)

…(Hill et al. 138)

Examples of works cited list entries

Below are a few examples of different types of works cited list entries. The examples given are for one author.

Steinman, Louise. The Knowing Body: Elements of Contemporary Performance and Dance . Shambhala Publications, 1986.

Journal article                                      

Barad, K. “Nature’s Queer Performativity.” Qui Parle , vol. 19, no. 2, 2011, pp. 121–58.

Webpage of a website

Midgelow, Vida L. “Experiences and Perceptions of the Artistic Doctorate: A Survey Report.” Artistic Doctorates in Europe,  5 Feb. 2018, www.artisticdoctorates.com/2017/12/28/experiences-and-perceptions-of-the-artistic-doctorate-survey-report/ .

YouTube video

“Behind the Scenes Chili’s Baby Back Ribs Spot.” YouTube , uploaded by Alvin Chea, 11 Sept. 2017, www.youtube.com/watch?v=gTDLh7gNRYA .

MLA Citation Examples

Writing Tools

Citation Generators

Other Citation Styles

Plagiarism Checker

Upload a paper to check for plagiarism against billions of sources and get advanced writing suggestions for clarity and style.

Get Started

This paper is in the following e-collection/theme issue:

Published on 16.4.2024 in Vol 26 (2024)

Adverse Event Signal Detection Using Patients’ Concerns in Pharmaceutical Care Records: Evaluation of Deep Learning Models

Authors of this article:

Original Paper

• Satoshi Nishioka 1 , PhD   ; 
• Satoshi Watabe 1 , BSc   ; 
• Yuki Yanagisawa 1 , PhD   ; 
• Kyoko Sayama 1 , MSc   ; 
• Hayato Kizaki 1 , MSc   ; 
• Shungo Imai 1 , PhD   ; 
• Mitsuhiro Someya 2 , BSc   ; 
• Ryoo Taniguchi 2 , PhD   ; 
• Shuntaro Yada 3 , PhD   ; 
• Eiji Aramaki 3 , PhD   ; 
• Satoko Hori 1 , PhD  

1 Division of Drug Informatics, Keio University Faculty of Pharmacy, Tokyo, Japan

2 Nakajima Pharmacy, Hokkaido, Japan

3 Nara Institute of Science and Technology, Nara, Japan

Corresponding Author:

Satoko Hori, PhD

Division of Drug Informatics

Keio University Faculty of Pharmacy

1-5-30 Shibakoen

Tokyo, 105-8512

Phone: 81 3 5400 2650

Email: [email protected]

Background: Early detection of adverse events and their management are crucial to improving anticancer treatment outcomes, and listening to patients’ subjective opinions (patients’ voices) can make a major contribution to improving safety management. Recent progress in deep learning technologies has enabled various new approaches for the evaluation of safety-related events based on patient-generated text data, but few studies have focused on the improvement of real-time safety monitoring for individual patients. In addition, no study has yet been performed to validate deep learning models for screening patients’ narratives for clinically important adverse event signals that require medical intervention. In our previous work, novel deep learning models have been developed to detect adverse event signals for hand-foot syndrome or adverse events limiting patients’ daily lives from the authored narratives of patients with cancer, aiming ultimately to use them as safety monitoring support tools for individual patients.

Objective: This study was designed to evaluate whether our deep learning models can screen clinically important adverse event signals that require intervention by health care professionals. The applicability of our deep learning models to data on patients’ concerns at pharmacies was also assessed.

Methods: Pharmaceutical care records at community pharmacies were used for the evaluation of our deep learning models. The records followed the SOAP format, consisting of subjective (S), objective (O), assessment (A), and plan (P) columns. Because of the unique combination of patients’ concerns in the S column and the professional records of the pharmacists, this was considered a suitable data for the present purpose. Our deep learning models were applied to the S records of patients with cancer, and the extracted adverse event signals were assessed in relation to medical actions and prescribed drugs.

Results: From 30,784 S records of 2479 patients with at least 1 prescription of anticancer drugs, our deep learning models extracted true adverse event signals with more than 80% accuracy for both hand-foot syndrome (n=152, 91%) and adverse events limiting patients’ daily lives (n=157, 80.1%). The deep learning models were also able to screen adverse event signals that require medical intervention by health care providers. The extracted adverse event signals could reflect the side effects of anticancer drugs used by the patients based on analysis of prescribed anticancer drugs. “Pain or numbness” (n=57, 36.3%), “fever” (n=46, 29.3%), and “nausea” (n=40, 25.5%) were common symptoms out of the true adverse event signals identified by the model for adverse events limiting patients’ daily lives.

Conclusions: Our deep learning models were able to screen clinically important adverse event signals that require intervention for symptoms. It was also confirmed that these deep learning models could be applied to patients’ subjective information recorded in pharmaceutical care records accumulated during pharmacists’ daily work.

Introduction

Increasing numbers of people are expected to develop cancers in our aging society [ 1 - 3 ]. Thus, there is increasing interest in how to detect and manage the side effects of anticancer therapies in order to improve treatment regimens and patients’ quality of life [ 4 - 8 ]. The primary approaches for side effect management are “early signal detection and early intervention” [ 9 - 11 ]. Thus, more efficient approaches for this purpose are needed.

It has been recognized that patients’ voices concerning adverse events represent an important source of information. Several studies have indicated that the number, severity, and time of occurrence of adverse events might be underevaluated by physicians [ 12 - 15 ]. Thus, patient-reported outcomes (PROs) have recently received more attention in the drug evaluation process, reflecting patients’ real voices. Various kinds of PRO measures have been developed and investigated in different disease populations [ 16 , 17 ]. Health care authorities have also encouraged the pharmaceutical industry to use PROs for drug evaluation [ 18 , 19 ], and it is becoming more common to take PRO assessment results into consideration for drug marketing approval [ 20 , 21 ]. Similar trends can be seen in the clinical management of individual patients. Thus, health care professionals have an interest in understanding how to appropriately gather patients’ concerns in order to improve safety management and clinical decisions [ 22 - 24 ].

The applications of deep learning for natural language processing have expanded dramatically in recent years [ 25 ]. Since the development of a high-performance deep learning model in 2018 [ 26 ], attempts to apply cutting-edge deep learning models to various kinds of patient-generated text data for the evaluation of safety events or the analysis of unscalable subjective information from patients have been accelerating [ 27 - 31 ]. Most studies have been conducted to use patients’ narrative data for pharmacovigilance [ 27 , 32 - 35 ], while few have been aimed at improvement of real-time safety monitoring for individual patients. In addition, there have been some studies on adverse event severity grading based on health care records [ 36 - 39 ], but none has yet aimed to extract clinically important adverse event signals that require medical intervention from patients’ narratives. It is important to know whether deep learning models could contribute to the detection of such important adverse event signals from concern texts generated by individual patients.

To address this question, we have developed deep learning models to detect adverse event signals from individual patients with cancer based on patients’ blog articles in online communities, following other types of natural language processing–related previous work [ 40 , 41 ]. One deep learning model focused on the specific symptom of hand-foot syndrome (HFS), which is one of the typical side effects of anticancer treatments [ 42 ], and another focused on a broad range of adverse events that impact patients’ activities of daily living [ 43 ]. We showed that our models can provide good performance scores in targeting adverse event signals. However, the evaluation relied on patients’ narratives from the patients’ blog data used for deep learning model training, so further evaluation is needed to ensure the validity and applicability of the models to other texts regarding patients’ concerns. In addition, the blog data source did not contain medical information, so it was not feasible to assess whether the models could contribute to the extraction of clinically important adverse event signals.

To address these challenges, we focused on pharmaceutical care records written by pharmacists at community pharmacies. The gold standard format for pharmaceutical care records in Japan is the SOAP (subjective, objective, assessment, plan)-based document that follows the “problem-oriented system” concept proposed by Weed [ 44 ] in 1968. Pharmacists track patients’ subjective concerns in the S column, provide objective information or observations in the O column, give their assessment from the pharmacist perspective in the A column, and suggest a plan for moving forward in the P column [ 45 , 46 ]. We considered that SOAP-based pharmaceutical care records could be a unique data source suitable for further evaluation of our deep learning models because they contain both patients’ concerns and professional health care records by pharmacists, including the medication prescription history with time stamps. Therefore, this study was designed to assess whether our deep learning models could extract clinically important adverse event signals that require intervention by medical professionals from these records. We also aimed to evaluate the characteristics of the models when applied to patients’ subjective information noted in the pharmaceutical care records, as there have been only a few studies on the application of deep learning models to patients’ concerns recorded during pharmacists’ daily work [ 47 - 49 ].

Here, we report the results of applying our deep learning models to patients’ concern text data in pharmaceutical care records, focusing on patients receiving anticancer treatment.

Data Source

The original data source was 2,276,494 pharmaceutical care records for 303,179 patients, created from April 2020 to December 2021 at community pharmacies belonging to the Nakajima Pharmacy Group in Japan [ 50 ]. To focus on patients with cancer, records of patients with at least 1 prescription for an anticancer drug were retrieved by sorting individual drug codes (YJ codes) used in Japan (YJ codes starting with 42 refer to anticancer drugs). Records in the S column (ie, S records) were collected from the patients with cancer as the text data of patients’ concerns for deep learning model analysis.

Deep Learning Models

The deep learning models used for this research were those that we constructed based on patients’ narratives in blog articles posted in an online community and that showed the best performance score in each task in our previous work (ie, a Bidirectional Encoder Representations From Transformers [BERT]–based model for HFS signal extraction [ 42 ] and a T5-based model for adverse event signal extraction [ 43 ]). BERT [ 26 ] and T5 [ 51 ] both belong to a type of deep learning model that has recently shown high performance in several studies [ 29 , 52 ]. Hereafter, we refer to the deep learning model for HFS signals as the HFS model, the model for any adverse event signals as All AE (ie, all or any adverse events) model, and the model for adverse event signals limited to patients’ activities of daily living as the AE-L (adverse events limiting patients’ daily lives) model. It was also confirmed that these deep learning models showed similar or higher performance scores for the HFS, All AE, or AE-L identification tasks using 1000 S records randomly extracted from the data source of this study compared to the values obtained in our previous work [ 42 , 43 ] (the performance scores of sentence-level tasks from our previous work are comparable, as the mean number of words in the sentences in the data source in our previous work was 32.7 [SD 33.9], which is close to that of the S records used in this study, 38.8 [SD 29.4]). The method and results of the performance-level check are described in detail in Multimedia Appendix 1 [ 42 , 43 ]. We applied the deep learning models to all text data in this study without any adjustment in setting parameters from those used in constructing them based on patient-authored texts in our previous work [ 42 , 43 ].

Evaluation of Extracted S Records by the Deep Learning Models

In this study, we focused on the evaluation of S records that our deep learning models extracted as HFS or AE-L positive. Each positive S record was assessed as if it was a true adverse event signal, a sort of adverse event symptom, whether or not an intervention was made by health care professionals. We also investigated the kind of anticancer treatment prescription in connection with each adverse event signal identified in S records.

To assess whether an extracted positive S record was a true adverse event signal, we used the same annotation guidelines as in our previous work [ 43 ]. In brief, each S record was treated as an “adverse event signal” if any untoward medical occurrence happened to the patient, regardless of the cause. For the AE-L model only, if a positive S record was confirmed as an adverse event signal, it was further categorized into 1 or more of the following adverse event symptoms: “fatigue,” “nausea,” “vomiting,” “diarrhea,” “constipation,” “appetite loss,” “pain or numbness,” “rash or itchy,” “hair loss,” “menstrual irregularity,” “fever,” “taste disorder,” “dizziness,” “sleep disorder,” “edema,” or “others.”

For the assessment of interventions by health care professionals and anticancer treatment prescriptions, information from the O, A, and P columns and drug prescription history in the data source were investigated for the extracted positive S records. The interventions by health care professionals were categorized in any of the following: “adding symptomatic treatment for the adverse event signal,” “dose reduction or discontinuation of causative anticancer treatment,” “consultation with physician,” “others,” or “no intervention (ie, just following up the adverse event signal).” The actions categorized in “others” were further evaluated individually. For this assessment, we also randomly extracted 200 S records and evaluated them in the same way for comparison with the results from the deep learning model. Prescription history of anticancer treatment was analyzed by primary category of mechanism of action (MoA) with subcategories if applicable (eg, target molecule for kinase inhibitors).

Applicability Check to Other Text Data Including Patients’ Concerns

To check the applicability of our deep learning models to data from a different source, interview transcripts from patients with cancer were also evaluated. The interview transcripts were created by the Database of Individual Patient Experiences-Japan (DIPEx-Japan) [ 53 ]. DIPEx-Japan divides the interview transcripts into sections for each topic, such as “onset of disease” and “treatment,” and posts the processed texts on its website. Processing is conducted by accredited researchers based on qualitative research methods established by the University of Oxford [ 54 ]. In this study, interview text data created from interviews with 52 patients with breast cancer conducted from January 2008 to October 2018 were used to assess whether our deep learning models can extract adverse event signals from this source. In total, 508 interview transcripts were included with the approval of DIPEx-Japan.

Ethical Considerations

This study was conducted with anonymized data following approval by the ethics committee of the Keio University Faculty of Pharmacy (210914-1 and 230217-1) and in accordance with relevant guidelines and regulations and the Declaration of Helsinki. Informed consent specific to this study was waived due to the retrospective observational design of the study with the approval of the ethics committee of the Keio University Faculty of Pharmacy. To respect the will of each individual stakeholder, however, we provided patients and pharmacists of the pharmacy group with an opportunity to refuse the sharing of their pharmaceutical care records by posting an overview of this study at each pharmacy store or on their web page regarding the analysis using pharmaceutical care records. Interview transcripts from DIPEx-Japan were provided through a data sharing arrangement for using narrative data for research and education. Consent for interview transcription and its sharing from DIPEx-Japan was obtained from the participants when the interviews were recorded.

From the original data source of 2,180,902 pharmaceutical care records for 291,150 patients, S records written by pharmacists for patients with a history of at least 1 prescription of an anticancer drug were extracted. This yielded 30,784 S records for 2479 patients with cancer ( Table 1 ). The mean and median number of words in the S records were 38.8 (SD 29.4) and 32 (IQR 20-50), respectively. We applied our deep learning models, HFS, All AE, and AE-L, to these 30,784 S records for the evaluation of the deep learning models for adverse event signal detection.

For interview transcripts created by DIPEx-Japan, the mean and median number of words were 428.9 (SD 160.9) and 416 (IQR 308-526), respectively, in the 508 transcripts for 52 patients with breast cancer.

a SOAP: subjective, objective, assessment, plan.

b S: subjective.

Application of the HFS Model

First, we applied the HFS model to the S records for patients with cancer. The BERT-based model was used for this research as it showed the best performance score in our previous work [ 42 ].

S Records Extracted as HFS Positive

The S records extracted as HFS positive by the HFS model ( Table 2 ) amounted to 167 (0.5%) records for 119 (4.8%) patients. A majority of the patients had 1 HFS-positive record in their S records (n=91, 76.5%), while 2 patients had as many as 6 (1.7%) HFS-positive records. When we examined whether the extracted S records were true adverse event signals or not, 152 records were confirmed to be adverse event signals, while the other 15 records were false-positives. All the false-positive S records were descriptions about the absence of symptoms or confirmation of improving condition (eg, “no diarrhea, mouth ulcers, or limb pain so far” or “the skin on the soles of my feet has calmed down a lot with this ointment”). Some examples of S records that were predicted as HFS positive by the model are shown in Table S1 in Multimedia Appendix 2 .

The same examination was conducted with interview transcripts from DIPEx-Japan. Only 1 (0.2%) transcript was extracted as HFS positive by the HFS model, and it was a true adverse event signal (100%). The actual transcript extracted as HFS positive is shown in Table S2 in Multimedia Appendix 2 .

a S: subjective.

b HFS: hand-foot syndrome.

c All false-positive S records were denial of symptoms or confirmation of improving condition.

Interventions by Health Care Professionals

The 167 S records extracted as HFS positive as well as 200 randomly selected records were checked for interventions by health care professionals ( Figure 1 ). The proportion showing any action by health care professionals was 64.1% for 167 HFS-positive S records compared to 13% for the 200 random S records. Among the actions taken for HFS positives, “adding symptomatic treatment” was the most common, accounting for around half (n=79, 47.3%), followed by “other” (n=18, 10.8%). Most “other” actions were educational guidance from pharmacists, such as instructions on moisturizing, nail care, or application of ointment and advice on daily living (eg, “avoid tight socks”).

Anticancer Drugs Prescribed

The types of anticancer drugs prescribed for HFS-positive patients are summarized based on the prescription histories in Table 3 . For the 152 adverse event signals identified by the HFS model in the previous section, the most common MoA class of anticancer drugs used for the patients was antimetabolite (n=62, 40.8%), specifically fluoropyrimidines (n=59, 38.8%). Kinase inhibitors were next (n=49, 32.2%), with epidermal growth factor receptor (EGFR) inhibitors and multikinase inhibitors as major subgroups (n=28, 18.4% and n=14, 9.2%, respectively). The third and fourth most common MoAs were aromatase inhibitors (n=24, 15.8%) and antiandrogen or estrogen drugs (n=7, 4.6% each) for hormone therapy.

a EGFR: epidermal growth factor receptor.

b VEGF: vascular endothelial growth factor.

c HER2: human epidermal growth factor receptor-2.

d CDK4/6: cyclin-dependent kinase 4/6.

Application of the All AE or AE-L model

The All AE and AE-L models were also applied to the same S records for patients with cancer. The T5-based model was used for this research as it gave the best performance score in our previous work [ 43 ].

S Records Extracted as All AE or AE-L positive

The numbers of S records extracted as positive were 7604 (24.7%) for 1797 patients and 196 (0.6%) for 142 patients for All AE and AE-L, respectively. In the case of All AE, patients tended to have multiple adverse event positives in their S records (n=1315, 73.2% of patients had at least 2 positives). In the case of AE-L, most patients had only 1 AE-L positive (n=104, 73.2%), and the largest number of AE-L positives for 1 patient was 4 (2.8%; Table 4 ).

We focused on AE-L evaluation due to its greater importance from a medical viewpoint and lower workload for manual assessment, considering the number of positive S records. Of the 197 AE-L–positive S records, it was confirmed that 157 (80.1%) records accurately extracted adverse event signals, while 39 (19.9%) records were false-positives that did not include any adverse event signals ( Table 4 ). The contents of the 39 false-positives were all descriptions about the absence of symptoms or confirmation of improving condition, showing a similar tendency to the HFS false-positives (eg, “The diarrhea has calmed down so far. Symptoms in hands and feet are currently fine” and “No symptoms for the following: upset in stomach, diarrhea, nausea, abdominal pain, abdominal pain or stomach cramps, constipation”). Examples of S records that were predicted as AE-L positive are shown in Table S3 in Multimedia Appendix 2 .

The deep learning models were also applied to interview transcripts from DIPEx-Japan in the same manner. The deep learning models identified 84 (16.5%) and 18 (3.5%) transcripts as All AE or AE-L positive, respectively. Of the 84 All AE–positive transcripts, 73 (86.9%) were true adverse event signals. The false-positives of All AE (n=11, 13.1%) were categorized into any of the following 3 types: explanations about the disease or its prognosis, stories when their cancer was discovered, or emotional changes that did not include clear adverse event mentions. With regard to AE-L, all the 18 (100%) positives were true adverse event signals (Table S4 in Multimedia Appendix 2 ). Examples of actual transcripts extracted as All AE or AE-L positive are shown in Table S5 in Multimedia Appendix 2 .

b All AE: all (or any of) adverse event.

c AE-L: adverse events limiting patients’ daily lives.

d All false-positive S records were denial of symptoms or confirmation of improving condition.

Whether or not interventions were made by health care professionals was investigated for the 196 AE-L–positive S records. As in the HFS model evaluation, data from 200 randomly selected S records were used for comparison ( Figure 2 ). In total, 91 (46.4%) records in the 196 AE-L–positive records were accompanied by an intervention, while the corresponding figure in the 200 random records was 26 (13%) records. The most common action in response to adverse event signals identified by the AE-L model was “adding symptomatic treatment” (n=71, 36.2%), followed by “other” (n=11, 5.6%). “Other” included educational guidance from pharmacists, inquiries from pharmacists to physicians, or recommendations for patients to visit a doctor.

The types of anticancer drugs prescribed for patients with adverse event signals identified by the AE-L model were summarized based on the prescription histories ( Table 5 ). In connection with the 157 adverse event signals, the most common MoA of the prescribed anticancer drug was antimetabolite (n=62, 39.5%) and fluoropyrimidine (n=53, 33.8%), which accounted for the majority. Kinase inhibitor (n=31, 19.7%) was the next largest category with multikinase inhibitor (n=14, 8.9%) as the major subgroup. These were followed by antiandrogen (n=27, 17.2%), antiestrogen (n=10, 6.4%), and aromatase inhibitor (n=10, 6.4%) for hormone therapy.

b JAK: janus kinase.

c VEGF: vascular endothelial growth factor.

d BTK: bruton tyrosine kinase.

e FLT3: FMS-like tyrosine kinase-3.

f PARP: poly-ADP ribose polymerase.

g CDK4/6: cyclin-dependent kinase 4/6.

h CD20: cluster of differentiation 20.

Adverse Event Symptoms

For the 157 adverse event signals identified by the AE-L model, the symptoms were categorized according to the predefined guideline in our previous work [ 43 ]. “Pain or numbness” (n=57, 36.3%) accounted for the largest proportion followed by “fever” (n=46, 29.3%) and “nausea” (n=40, 25.5%; Table 6 ). Symptoms classified as “others” included chills, tinnitus, running tears, dry or peeling skin, and frequent urination. When comparing the proportion of the symptoms associated with or without interventions by health care professionals, a trend toward a greater proportion of interventions was observed in “fever,” “nausea,” “diarrhea,” “constipation,” “vomiting,” and “edema” ( Figure 3 , black boxes). On the other hand, a smaller proportion was observed in “pain or numbness,” “fatigue,” “appetite loss,” “rash or itchy,” “taste disorder,” and “dizziness” ( Figure 3 , gray boxes).

This study was designed to evaluate our deep learning models, previously constructed based on patient-authored texts posted in an online community, by applying them to pharmaceutical care records that contain both patients’ subjective concerns and medical information created by pharmacists. Based on the results, we discuss whether these deep learning models can extract clinically important adverse event signals that require medical intervention, and what characteristics they show when applied to data on patients’ concerns in pharmaceutical care records.

Performance for Adverse Event Signal Extraction

The first requirement for the deep learning models is to extract adverse event signals from patients’ narratives precisely. In this study, we evaluated the proportion of true adverse event signals in positive S records extracted by the HFS or AE-L model. True adverse event signals amounted to 152 (91%) and 157 (80.1%) for the HFS and AE-L models, respectively ( Tables 2 and 4 ). Given that the proportion of true adverse event signals in 200 randomly extracted S records without deep learning models was 54 (27%; categories other than “no adverse event” in Figures 1 and 2 ), the HFS and AE-L models were able to concentrate S records with adverse event mentions. Although 15 (9%) for the HFS model and 39 (19.9%) for the AE-L model were false-positives, it was confirmed all of the false-positive records described a lack of symptoms or confirmation of improving condition. We considered that such false-positives are due to the unique feature of pharmaceutical care records, where pharmacists might proactively interview patients about potential side effects of their medications. As the data set of blog articles we used to construct the deep learning models included few such cases (especially comments on lack of symptoms), our models seemed unable to exclude them correctly. Even though we confirmed that the proportion of true “adverse event” signals extracted from the S records by the HFS or AE-L model was more than 80%, the performance scores to extract true “HFS” or “AE-L” signals were not so high based on the performance check using 1000 randomly extracted S records ( F 1 -scores were 0.50 and 0.22 for true HFS and AE-L signals, respectively; Table S1 in Multimedia Appendix 1 ). It is considered that the performance to extract true HFS and AE-L signals was relatively low due to the short length of texts in the S records, providing less context to judge the impact on patients’ daily lives, especially for the AE-L model (the mean word number of the S records was 38.8 [SD 29.4; Table 1 ], similar to the sentence-level tasks in our previous work [ 42 , 43 ]). However, we consider a true adverse event signal proportion of more than 80% in this study represents a promising outcome, as this is the first attempt to apply our deep learning models to a different source of patients’ concern data, and the extracted positive cases would be worthy of evaluation by a medical professional, as the potential adverse events could be caused by drugs taken by the patients.

When the deep learning models were applied to DIPEx-Japan interview transcripts, including patients’ concerns, the proportion of true adverse event signals was also more than 80% (for All AE: n=73, 86.9% and for HFS and AE-L: n=18, 100%). The difference in the results between pharmaceutical care S records and DIPEx-Japan interview transcripts was the features of false-positives, descriptions about lack of symptoms or confirmation of improving condition in S records versus explanations about disease or its prognosis, stories about when their cancer was discovered, or emotional changes in interview transcripts. This is considered due to the difference in the nature of the data source; the pharmaceutical care records were generated in a real-time manner by pharmacists through their daily work, where adverse event signals are proactively monitored, while the interview transcripts were purely based on patients’ retrospective memories. Our deep learning models were able to extract true adverse event signals with an accuracy of more than 80% from both text data sources in spite of the difference in their nature. When looking at future implementation of the deep learning models in society (discussed in the Potential for Deep Learning Model Implementation in Society section), it may be desirable to further adjust deep learning models to reduce false-positives depending upon the features of the data source.

Identification of Important Adverse Events Requiring Medical Intervention

To assess whether the models could extract clinically important adverse event signals, we investigated interventions by health care professionals connected with the adverse event signals that are identified by our deep learning models. In the 200 randomly extracted S records, only 26 (13%) consisted of adverse event signals, leading to any intervention by health care professionals. On the other hand, the proportion of signals associated with interventions was increased to 107 (64.1%) and 91 (46.4%) in the S records extracted as positive by the HFS and AE-L models, respectively ( Figures 1 and 2 ). These results suggest that both deep learning models can screen clinically important adverse event signals that require intervention from health care professionals. The performance level in screening adverse event signals requiring medical intervention was higher in the HFS model than in the AE-L model (n=107, 64.1% vs n=91, 46.4%; Figures 1 and 2 ). Since the target events were specific and adverse event signals of HFS were narrowly defined, which is one of the typical side effects of some anticancer drugs, we consider that health care providers paid special attention to HFS-related signals and took action proactively. In both deep learning models, similar trends were observed in actions taken by health care professionals in response to extracted adverse event signals; common actions were attempts to manage adverse event symptoms by symptomatic treatment or other mild interventions, including educational guidance from pharmacists or recommendations for patients to visit a doctor. More direct interventions focused on the causative drugs (ie, “dose reduction or discontinuation of anticancer treatment”) amounted to less than 5%; 7 (4.2%) for the HFS model and 6 (3.1%) for the AE-L model ( Figures 1 and 2 ). Thus, it appears that our deep learning models can contribute to screening mild to moderate adverse event signals that require preventive actions such as symptomatic treatments or professional advice from health care providers, especially for patients with less sensitivity to adverse event signals or who have few opportunities to visit clinics and pharmacies.

Ability to Catch Real Side Effect Signals of Anticancer Drugs

Based on the drug prescription history associated with S records extracted as HFS or AE-L positive, the type and duration of anticancer drugs taken by patients experiencing the adverse event signals were investigated. For the HFS model, the most common MoA of anticancer drug was antimetabolite (fluoropyrimidine: n=59, 38.8%), followed by kinase inhibitors (n=49, 32.2%, of which EGFR inhibitors and multikinase inhibitors accounted for n=28, 18.4% and n=14, 9.2%, respectively) and aromatase inhibitors (n=24, 15.8%; Table 3 ). It is known that fluoropyrimidine and multikinase inhibitors are typical HFS-inducing drugs [ 55 - 58 ], suggesting that the HFS model accurately extracted HFS side effect signals derived from these drugs. Note that symptoms such as acneiform rash, xerosis, eczema, paronychia, changes in the nails, arthralgia, or stiffness of limb joints, which are common side effects of EGFR inhibitors or aromatase inhibitors [ 59 , 60 ], might be extracted as closely related expressions to those of HFS signals. When looking at the MoA of anticancer drugs for patients with adverse event signals identified by the AE-L model, antimetabolite (fluoropyrimidine) was the most common one (n=53, 33.8%), as in the case of those identified by the HFS model, followed by kinase inhibitors (n=31, 19.7%) and antiandrogens (n=27, 17.2%; Table 5 ). Since the AE-L model targets a broad range of adverse event symptoms, it is difficult to rationalize the relationship between the adverse event signals and types of anticancer drugs. However, the type of anticancer drugs would presumably closely correspond to the standard treatments of the cancer types of the patients. Based on the prescribed anticancer drugs, we can infer that a large percentage of the patients had breast or lung cancer, indicating that our study results were based on data from such a population. Thus, a possible direction for the expansion of this research would be adjusting the deep learning models by additional training with expressions for typical side effects associated with standard treatments of other cancer types. To interpret these results correctly, it should be noted that we could not investigate anticancer treatments conducted outside of the pharmacies (eg, the time-course relationship with intravenously administered drugs would be missed, as the administration will be done at hospitals). To further evaluate how useful this model is in side effect signal monitoring for patients with cancer, comprehensive medical information for the eligible patients would be required.

Suitability of the Deep Learning Models for Specific Adverse Event Symptoms

Among the adverse event signals identified by the AE-L model, the type of symptom was categorized according to a predefined annotation guideline that we previously developed [ 43 ]. The most frequently recorded adverse event signals identified by the AE-L model were “pain or numbness” (n=57, 36.3%), “fever” (n=46, 29.3%), and “nausea” (n=40, 25.5%; Table 6 ). Since the pharmaceutical care records had information about interventions by health care professionals, the frequency of the presence or absence of the interventions for each symptom was examined. A trend toward a greater proportion of interventions was observed in “fever,” “nausea,” “diarrhea,” “constipation,” “vomiting,” and “edema” ( Figure 3 , black boxes). There seem to be 2 possible explanations for this: these symptoms are of high importance and require early medical intervention or effective symptomatic treatments are available for these symptoms in clinical practice so that medical intervention is an easy option. On the other hand, a trend for a smaller proportion of adverse event signals to result in interventions was observed for “pain or numbness,” “fatigue,” “appetite loss,” “rash or itchy,” “taste disorder,” and “dizziness” ( Figure 3 , gray boxes). The reason for this may be the lack of effective symptomatic treatments or the difficulty of judging whether the severity of these symptoms justifies medical intervention by health care providers. In either case, there may be room for improvement in the quality of medical care for these symptoms. We expect that our research will contribute to a quality improvement in safety monitoring in clinical practice by supporting adverse event signal detection in a cost-effective manner.

Potential for Deep Learning Model Implementation in Society

Although we evaluated our deep learning models using pharmaceutical care records in this study, the main target of future implementation of our deep learning models in society would be narrative texts that patients directly write to record their daily experiences. For example, the application of these deep learning models to electronic media where patients record their daily experiences in their lives with disease (eg, health care–related e-communities and disease diary applications) could enable information about adverse event signal onset that patients experience to be provided to health care providers in a timely manner. Adverse event signals can automatically be identified and shared with health care providers based on the concern texts that patients post to any platform. This system will have the advantage that health care providers can efficiently grasp safety-related events that patients experience outside of clinic visits so that they can conduct more focused or personalized interactions with patients at their clinic visits. However, consideration should be given to avoid an excessive burden on health care providers. For instance, limiting the sharing of adverse event signals to those of high severity or summarizing adverse event signals over a week rather than sharing each one in a real-time manner may be reasonable approaches for medical staff. We also need to think about how to encourage patients to record their daily experiences using electronic tools. Not only technical progress and support but also the establishment of an ecosystem where both patients and medical staff can feel benefit will be required. Prospective studies with deep learning models to follow up patients in the long term and evaluate outcomes will be needed. We primarily looked at patient-authored texts as targets of implementation, but our deep learning models may also be worth using medical data including patients’ subjective concerns, such as pharmaceutical care S records. As this study confirmed that our deep learning models are applicable to patients’ concern texts tracked by pharmacists, it should be possible to use them to analyze other “patient voice-like” medical text data that have not been actively investigated so far.

Limitations

First, the major limitation of this study was that we were not able to collect complete medical information of the patients. Although we designed this study to analyze patients’ concerns extracted by the deep learning models and their relationship with medical information contained in the pharmaceutical care records, some information could not be tracked (eg, missing history of medical interventions or anticancer treatment at hospitals as well as diagnosis of patients’ primary cancers). Second, there might be a data creation bias in S records for patients’ concerns by pharmacists. For example, symptoms that have little impact on intervention decisions might less likely be recorded by them. It should be also noted that the characteristics of S records may not be consistent at different community pharmacies.

Conclusions

Our deep learning models were able to screen clinically important adverse event signals that require intervention by health care professionals from patients’ concerns in pharmaceutical care records. Thus, these models have the potential to support real-time adverse event monitoring of individual patients taking anticancer treatments in an efficient manner. We also confirmed that these deep learning models constructed based on patient-authored texts could be applied to patients’ subjective information recorded by pharmacists through their daily work. Further research may help to expand the applicability of the deep learning models for implementation in society or for analysis of data on patients’ concerns accumulated in professional records at pharmacies or hospitals.

Acknowledgments

This work was supported by Japan Society for the Promotion of Science, Grants-in-Aid for Scientific Research (KAKENHI; grant 21H03170) and Japan Science and Technology Agency, Core Research for Evolutional Science and Technology (CREST; grant JPMJCR22N1), Japan. Mr Yuki Yokokawa and Ms Sakura Yokoyama at our laboratory advised SN about the structure of pharmaceutical care records. This study would not have been feasible without the high quality of pharmaceutical care records created by many individual pharmacists at Nakajima Pharmacy Group through their daily work.

Data Availability

The data sets generated and analyzed during this study are available from the corresponding author on reasonable request.

Authors' Contributions

SN and SH designed the study. SN retrieved the subjective records of patients with cancer from the data source for the application of deep learning models and organized other data for subsequent evaluations. SN ran the deep learning models with the support of SW. SN, YY, and KS checked the adverse event signals for each subjective record that was extracted as positive by the models for hand-foot syndrome or adverse events limiting patients’ daily lives and evaluated the adverse event signal symptoms, details of interventions taken by health care professionals, and types of anticancer drugs prescribed for patients based on available data from the data source. HK and SI advised on the study concept and process. MS and RT provided pharmaceutical records at their community pharmacies along with advice on how to use and interpret them. SY and EA supervised the natural language processing research as specialists. SH supervised the study overall. SN drafted and finalized the paper. All authors reviewed and approved the paper.

Conflicts of Interest

SN is an employee of Daiichi Sankyo Co, Ltd. All other authors declare no conflicts of interest.

Performance evaluation of deep learning models.

Examples of S records and sample interview transcripts.

• Global cancer observatory: cancer over time. World Health Organization. URL: https://gco.iarc.fr/overtime/en [accessed 2023-07-02]
• Mattiuzzi C, Lippi G. Current cancer epidemiology. J Epidemiol Glob Health. 2019;9(4):217-222. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Montazeri F, Komaki H, Mohebi F, Mohajer B, Mansournia MA, Shahraz S, et al. Editorial: disparities in cancer prevention and epidemiology. Front Oncol. 2022;12:872051. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lasala R, Santoleri F. Association between adherence to oral therapies in cancer patients and clinical outcome: a systematic review of the literature. Br J Clin Pharmacol. 2022;88(5):1999-2018. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Pudkasam S, Polman R, Pitcher M, Fisher M, Chinlumprasert N, Stojanovska L, et al. Physical activity and breast cancer survivors: importance of adherence, motivational interviewing and psychological health. Maturitas. 2018;116:66-72. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Markman M. Chemotherapy-associated neurotoxicity: an important side effect-impacting on quality, rather than quantity, of life. J Cancer Res Clin Oncol. 1996;122(9):511-512. [ CrossRef ] [ Medline ]
• Jitender S, Mahajan R, Rathore V, Choudhary R. Quality of life of cancer patients. J Exp Ther Oncol. 2018;12(3):217-221. [ Medline ]
• Di Nardo P, Lisanti C, Garutti M, Buriolla S, Alberti M, Mazzeo R, et al. Chemotherapy in patients with early breast cancer: clinical overview and management of long-term side effects. Expert Opin Drug Saf. 2022;21(11):1341-1355. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Cuomo RE. Improving cancer patient outcomes and cost-effectiveness: a Markov simulation of improved early detection, side effect management, and palliative care. Cancer Invest. 2023;41(10):858-862. [ CrossRef ] [ Medline ]
• Pulito C, Cristaudo A, La Porta C, Zapperi S, Blandino G, Morrone A, et al. Oral mucositis: the hidden side of cancer therapy. J Exp Clin Cancer Res. 2020;39(1):210. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bartal A, Mátrai Z, Szûcs A, Liszkay G. Main treatment and preventive measures for hand-foot syndrome, a dermatologic side effect of cancer therapy. Magy Onkol. 2011;55(2):91-98. [ FREE Full text ] [ Medline ]
• Basch E, Jia X, Heller G, Barz A, Sit L, Fruscione M, et al. Adverse symptom event reporting by patients vs clinicians: relationships with clinical outcomes. J Natl Cancer Inst. 2009;101(23):1624-1632. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Basch E. The missing voice of patients in drug-safety reporting. N Engl J Med. 2010;362(10):865-869. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Fromme EK, Eilers KM, Mori M, Hsieh YC, Beer TM. How accurate is clinician reporting of chemotherapy adverse effects? A comparison with patient-reported symptoms from the Quality-of-Life Questionnaire C30. J Clin Oncol. 2004;22(17):3485-3490. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Liu L, Suo T, Shen Y, Geng C, Song Z, Liu F, et al. Clinicians versus patients subjective adverse events assessment: based on patient-reported outcomes version of the common terminology criteria for adverse events (PRO-CTCAE). Qual Life Res. 2020;29(11):3009-3015. [ CrossRef ] [ Medline ]
• Churruca K, Pomare C, Ellis LA, Long JC, Henderson SB, Murphy LED, et al. Patient-reported outcome measures (PROMs): a review of generic and condition-specific measures and a discussion of trends and issues. Health Expect. 2021;24(4):1015-1024. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Pérez-Alfonso KE, Sánchez-Martínez V. Electronic patient-reported outcome measures evaluating cancer symptoms: a systematic review. Semin Oncol Nurs. 2021;37(2):151145. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Patient-reported outcome measures: use in medical product development to support labeling claims. U.S. Food & Drug Administration. 2009. URL: https:/​/www.​fda.gov/​regulatory-information/​search-fda-guidance-documents/​patient-reported-outcome-measures-use-medical-product-development-support-labeling-claims [accessed 2023-11-26]
• Appendix 2 to the guideline on the evaluation of anticancer medicinal products in man—the use of patient-reported outcome (PRO) measures in oncology studies—scientific guideline. European Medicines Agency. 2016. URL: https:/​/www.​ema.europa.eu/​en/​appendix-2-guideline-evaluation-anticancer-medicinal-products-man-use-patient-reported-outcome-pro [accessed 2023-11-26]
• Weber SC. The evolution and use of patient-reported outcomes in regulatory decision making. RF Q. 2023;3(1):4-9. [ FREE Full text ]
• Teixeira MM, Borges FC, Ferreira PS, Rocha J, Sepodes B, Torre C. A review of patient-reported outcomes used for regulatory approval of oncology medicinal products in the European Union between 2017 and 2020. Front Med (Lausanne). 2022;9:968272. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Newell S, Jordan Z. The patient experience of patient-centered communication with nurses in the hospital setting: a qualitative systematic review protocol. JBI Database System Rev Implement Rep. 2015;13(1):76-87. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Yagasaki K, Takahashi H, Ouchi T, Yamagami J, Hamamoto Y, Amagai M, et al. Patient voice on management of facial dermatological adverse events with targeted therapies: a qualitative study. J Patient Rep Outcomes. 2019;3(1):27. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Giardina TD, Korukonda S, Shahid U, Vaghani V, Upadhyay DK, Burke GF, et al. Use of patient complaints to identify diagnosis-related safety concerns: a mixed-method evaluation. BMJ Qual Saf. 2021;30(12):996-1001. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lauriola I, Lavelli A, Aiolli F. An introduction to deep learning in natural language processing: models, techniques, and tools. Neurocomputing. 2022;470:443-456. [ CrossRef ]
• Devlin J, Chang MW, Lee K, Toutanova K. BERT: pre-training of deep bidirectional transformers for language understanding. 2019. Presented at: 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies; June 2-7, 2019;4171-4186; Minneapolis, MN, USA.
• Dreisbach C, Koleck TA, Bourne PE, Bakken S. A systematic review of natural language processing and text mining of symptoms from electronic patient-authored text data. Int J Med Inform. 2019;125:37-46. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Sim JA, Huang X, Horan MR, Stewart CM, Robison LL, Hudson MM, et al. Natural language processing with machine learning methods to analyze unstructured patient-reported outcomes derived from electronic health records: a systematic review. Artif Intell Med. 2023;146:102701. [ CrossRef ] [ Medline ]
• Weissenbacher D, Banda JM, Davydova V, Estrada-Zavala D, Gascó Sánchez L, Ge Y, et al. Overview of the seventh social media mining for health applications (#SMM4H) shared tasks at COLING 2022. 2022. Presented at: Proceedings of the Seventh Workshop on Social Media Mining for Health Applications, Workshop & Shared Task; October 12-17, 2022;221-241; Gyeongju, Republic of Korea. URL: https://aclanthology.org/2022.smm4h-1.54/
• Matsuda S, Ohtomo T, Okuyama M, Miyake H, Aoki K. Estimating patient satisfaction through a language processing model: model development and evaluation. JMIR Form Res. 2023;7(1):e48534. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Yu D, Vydiswaran VGV. An assessment of mentions of adverse drug events on social media with natural language processing: model development and analysis. JMIR Med Inform. 2022;10(9):e38140. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Liu X, Chen H. A research framework for pharmacovigilance in health social media: identification and evaluation of patient adverse drug event reports. J Biomed Inform. 2015;58:268-279. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Nikfarjam A, Sarker A, O'Connor K, Ginn R, Gonzalez G. Pharmacovigilance from social media: mining adverse drug reaction mentions using sequence labeling with word embedding cluster features. J Am Med Inform Assoc. 2015;22(3):671-681. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kakalou C, Dimitsaki S, Dimitriadis VK, Natsiavas P. Exploiting social media for active pharmacovigilance: the PVClinical social media workspace. Stud Health Technol Inform. 2022;290:739-743. [ CrossRef ] [ Medline ]
• Bousquet C, Dahamna B, Guillemin-Lanne S, Darmoni SJ, Faviez C, Huot C, et al. The adverse drug reactions from patient reports in social media project: five major challenges to overcome to operationalize analysis and efficiently support pharmacovigilance process. JMIR Res Protoc. 2017;6(9):e179. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Young IJB, Luz S, Lone N. A systematic review of natural language processing for classification tasks in the field of incident reporting and adverse event analysis. Int J Med Inform. 2019;132:103971. [ CrossRef ] [ Medline ]
• Jacobsson R, Bergvall T, Sandberg L, Ellenius J. Extraction of adverse event severity information from clinical narratives using natural language processing. Pharmacoepidemiol Drug Saf. 2017;26(S2):37. [ FREE Full text ]
• Liang C, Gong Y. Predicting harm scores from patient safety event reports. Stud Health Technol Inform. 2017;245:1075-1079. [ CrossRef ] [ Medline ]
• Jiang G, Wang L, Liu H, Solbrig HR, Chute CG. Building a knowledge base of severe adverse drug events based on AERS reporting data using semantic web technologies. Stud Health Technol Inform. 2013;192(1-2):496-500. [ CrossRef ] [ Medline ]
• Usui M, Aramaki E, Iwao T, Wakamiya S, Sakamoto T, Mochizuki M. Extraction and standardization of patient complaints from electronic medication histories for pharmacovigilance: natural language processing analysis in Japanese. JMIR Med Inform. 2018;6(3):e11021. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Watanabe T, Yada S, Aramaki E, Yajima H, Kizaki H, Hori S. Extracting multiple worries from breast cancer patient blogs using multilabel classification with the natural language processing model bidirectional encoder representations from transformers: infodemiology study of blogs. JMIR Cancer. 2022;8(2):e37840. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Nishioka S, Watanabe T, Asano M, Yamamoto T, Kawakami K, Yada S, et al. Identification of hand-foot syndrome from cancer patients' blog posts: BERT-based deep-learning approach to detect potential adverse drug reaction symptoms. PLoS One. 2022;17(5):e0267901. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Nishioka S, Asano M, Yada S, Aramaki E, Yajima H, Yanagisawa Y, et al. Adverse event signal extraction from cancer patients' narratives focusing on impact on their daily-life activities. Sci Rep. 2023;13(1):15516. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Weed LL. Medical records that guide and teach. N Engl J Med. 1968;278(11):593-600. [ CrossRef ] [ Medline ]
• Podder V, Lew V, Ghassemzadeh S. SOAP Notes. Treasure Island, FL. StatPearls Publishing; 2023.
• Shenavar Masooleh I, Ramezanzadeh E, Yaseri M, Sahere Mortazavi Khatibani S, Sadat Fayazi H, Ali Balou H, et al. The effectiveness of training on daily progress note writing by medical interns. J Adv Med Educ Prof. 2021;9(3):168-175. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Grothen AE, Tennant B, Wang C, Torres A, Sheppard BB, Abastillas G, et al. Application of artificial intelligence methods to pharmacy data for cancer surveillance and epidemiology research: a systematic review. JCO Clin Cancer Inform. 2020;4:1051-1058. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ohno Y, Kato R, Ishikawa H, Nishiyama T, Isawa M, Mochizuki M, et al. Using the natural language processing system MedNER-J to analyze pharmaceutical care records. medRxiv. Preprint posted online on October 2, 2023. [ CrossRef ]
• Ranchon F, Chanoine S, Lambert-Lacroix S, Bosson JL, Moreau-Gaudry A, Bedouch P. Development of artificial intelligence powered apps and tools for clinical pharmacy services: a systematic review. Int J Med Inform. 2023;172:104983. [ CrossRef ] [ Medline ]
• Nakajima Pharmacy. URL: https://www.nakajima-phar.co.jp/ [accessed 2023-12-07]
• Raffel C, Shazeer N, Roberts A, Lee K, Narang S, Matena M, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. J Mach Learn Res. 2020;21(140):1-67. [ FREE Full text ]
• Pathak A. Comparative analysis of transformer based language models. Comput Sci Inf Technol. 2021.:165-176. [ FREE Full text ] [ CrossRef ]
• DIPEx Japan. URL: https://www.dipex-j.org/ [accessed 2024-02-04]
• Herxheimer A, McPherson A, Miller R, Shepperd S, Yaphe J, Ziebland S. Database of patients' experiences (DIPEx): a multi-media approach to sharing experiences and information. Lancet. 2000;355(9214):1540-1543. [ CrossRef ] [ Medline ]
• Lara PE, Muiño CB, de Spéville BD, Reyes JJ. Hand-foot skin reaction to regorafenib. Actas Dermosifiliogr. 2016;107(1):71-73. [ CrossRef ]
• Zaiem A, Hammamia SB, Aouinti I, Charfi O, Ladhari W, Kastalli S, et al. Hand-foot syndrome induced by chemotherapy drug: case series study and literature review. Indian J Pharmacol. 2022;54(3):208-215. [ FREE Full text ] [ CrossRef ] [ Medline ]
• McLellan B, Ciardiello F, Lacouture ME, Segaert S, Van Cutsem E. Regorafenib-associated hand-foot skin reaction: practical advice on diagnosis, prevention, and management. Ann Oncol. 2015;26(10):2017-2026. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ai L, Xu Z, Yang B, He Q, Luo P. Sorafenib-associated hand-foot skin reaction: practical advice on diagnosis, mechanism, prevention, and management. Expert Rev Clin Pharmacol. 2019;12(12):1121-1127. [ CrossRef ] [ Medline ]
• Tenti S, Correale P, Cheleschi S, Fioravanti A, Pirtoli L. Aromatase inhibitors-induced musculoskeletal disorders: current knowledge on clinical and molecular aspects. Int J Mol Sci. 2020;21(16):5625. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lacouture ME, Melosky BL. Cutaneous reactions to anticancer agents targeting the epidermal growth factor receptor: a dermatology-oncology perspective. Skin Therapy Lett. 2007;12(6):1-5. [ FREE Full text ] [ Medline ]

Abbreviations

Edited by G Eysenbach; submitted 25.12.23; peer-reviewed by CY Wang, L Guo; comments to author 24.01.24; revised version received 14.02.24; accepted 09.03.24; published 16.04.24.

©Satoshi Nishioka, Satoshi Watabe, Yuki Yanagisawa, Kyoko Sayama, Hayato Kizaki, Shungo Imai, Mitsuhiro Someya, Ryoo Taniguchi, Shuntaro Yada, Eiji Aramaki, Satoko Hori. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 16.04.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.

To read this content please select one of the options below:

Please note you do not have access to teaching notes, survival of the fittest: the role leaders play in managing change effectively.

Development and Learning in Organizations

ISSN : 1477-7282

Article publication date: 16 April 2024

This paper aims to review the latest management developments across the globe and pinpoint practical implications from cutting-edge research and case studies.

Design/methodology/approach

This briefing is prepared by an independent writer who adds their own impartial comments and places the articles in context.

This paper identified that the mindset of the leader plays a crucial role in determine whether the organization can manage and adapt to change effectively.

Originality/value

The briefing saves busy executives, strategists and researchers hours of reading time by selecting only the very best, most pertinent information and presenting it in a condensed and easy-to-digest format.

• Change management
• Organizational change capability

(2024), "Survival of the fittest: The role leaders play in managing change effectively", Development and Learning in Organizations , Vol. ahead-of-print No. ahead-of-print. https://doi.org/10.1108/DLO-03-2024-0061

Emerald Publishing Limited

Copyright © 2024, Emerald Publishing Limited

Related articles

We’re listening — tell us what you think, something didn’t work….

Report bugs here

All feedback is valuable

Please share your general feedback

Join us on our journey

Platform update page.

Visit emeraldpublishing.com/platformupdate to discover the latest news and updates

Questions & More Information

Answers to the most commonly asked questions here