Readability formulas, programs and tools: Do they work for plain language?

On this page

Posted on 
July 25, 2022

You want to make your documents easy to understand. You want people to follow instructions, make decisions or send information to you. So you want to use plain language.

You’ve possibly been told about reading levels or grade levels. And you’ve probably been encouraged to use a readability formula (such as the one in Word) to get your text down to a certain grade level. Sometimes running a document through a readability tool is all people do to flag plain language issues.

Plain language experts say these tools are actually not useful. And they could even undermine all the other proven techniques you can use.

Wait! What? Is this new?

The idea that readability formulas are problematic goes back a long way. As early as the 1980s, people were researching it and talking about it.

“By the mid-1980s, there was a widespread sense that plain-language advocates had shifted priorities from readability to usability.”

“Research tells us that most readability formulas are outdated methods for assessing text quality.”

– Dr. Karen Schriver

Readability formulas are ineffective and counterproductive

Dr. Schriver described some research on this from Richard Kern, which was published in 1980 (so experts have known about this for a long time).

Kern’s research highlighted important problems with readability scores:

  • (1) Readability formulas cannot match material to readers at targeted grade levels.
  • (2) Rewriting to lower the reading-grade level score does not increase comprehension.
  • (3) Requiring that text be written to satisfy a targeted reading-grade level focuses attention on meeting the score requirement rather than on organizing the material to meet the reader’s information needs.

Dr. Schriver also described how people learn to write simply to get a better score.
But this does not increase reader comprehension.

Readability formulas don’t measure the right things

“Most of what makes a document usable is not included in readability formulas.”

– Dr. Ginny Redish

Dr. Redish describes why the formulas don’t measure the right things.

  • They were not created for technical documents.
  • They assume that short words are always the better words.
  • They don’t work with many documented features of plain language.

Let’s look at the last two in that list because they are easy to follow.

Readability formulas assume that short words are always the better words

Plain language requires you to write for your audience’s needs. So what qualifies as plain writing depends on who the audience is. The goal is to pick words that are familiar to the reader, not to use short words all the time.

Sometimes short words are harder to understand because they are abstract (ennui, writ) or carry a lot of historical meaning (fez, nadir). But a formula will score them as easy words.

“The grade level score from a readability formula is based on the average length of the words and sentences. Though the formulas vary, they generally assume that longer words are harder words and longer sentences are harder sentences. They can’t tell you whether the words you are using are familiar to your readers or whether the sentences you have written are clear and cohesive.”

– Agency for Healthcare Research and Quality, US Department of Health and Human Services

Readability formulas don’t work with many documented features of plain language

Bulleted lists are known to be very helpful to readers (if used to properly chunk information), but readability formulas score such text as harder to read.

“Readability formulas assume that you are writing prose paragraphs. They count sentence length by going from period to period. If you use bulleted lists to chunk your material and lay your text out with white space, readability formulas will say you have long sentences. Yet, usability studies have consistently shown the value of lists and white space as aids to locating and understanding information.”

– Dr. Redish

You can see a demonstration of this point with an example in one of Dr. Redish’s publications.

Dr. Redish also notes these critical techniques that readability formulas do not measure:

  • determining the right content for readers
  • organizing the material for readers
  • using meaningful headings and other tools to guide readers

In her paper (written with Caroline Jarrett), she explains why improving readability scores doesn’t usually correlate with improving comprehension. In fact, she gives a specific example where the version with a readability score that indicated it was easier to understand was actually less understandable to the people who needed to read the document.

The ISO plain language standard excludes such formulas

An international working group made up of experts from 25 countries wrote the ISO plain language standard (expected to be published in 2022). Between them, these experts speak 17 languages and work in a wide range of roles and organizations. Many of these experts have been doing research on plain language for decades.

This group of experts all agreed with this wording in the standard:

“Plain language ensures readers can find what they need, understand it, and use it. Thus, plain language focuses on how successfully readers can use the document rather than on mechanical measures such as readability formulas.”

And there is no mention of readability formulas in the standard besides that explicit exclusion of them.

What is in the standard?

This is the definition that the standard is based on:

A communication is in plain language if the language, structure, and design are so clear that the intended audience can easily

  • find what they need,
  • understand what they find, and
  • use that information.

The four principles in the standard give details on what methods can be used to write a plain language document:

  • Readers get what they need (relevant)
  • Readers can easily find what they need (findable)
  • Readers can easily understand what they find (understandable)
  • Readers can easily use the information (usable)

Where does this leave us on using readability formulas for plain language?

We can see then that since the early 1980s researchers have agreed that readability formulas focus only on a very small part of what makes a communication plain. It’s also clear in 2022 that international experts working on an ISO standard for plain language have decided not to support using readability formulas for assessing the clarity of texts for audiences around the world. You should reconsider your use of the formulas and instead evaluate text quality by testing your communications with intended readers.

View sources

Disclaimer

The opinions expressed in posts and comments published on the Our Languages blog are solely those of the authors and commenters and do not necessarily reflect the views of the Language Portal of Canada.

Get to know Gael Spivak

Gael Spivak

Gael Spivak works in communications for the federal government. She specializes in plain language writing and editing. She’s a former president of the Editors' Association of Canada and the current chair of the Localization and Implementation Committee (for the ISO plain language standard), International Plain Language Federation.

You can:

Leave a comment

Please consult the “Comments and interaction” section on the Canada.ca Terms and conditions page before adding your comment. The Language Portal of Canada reviews comments before they’re posted. We reserve the right to edit, refuse or remove any question or comment that violates these commenting guidelines.

By submitting a comment, you permanently waive your moral rights, which means that you give the Government of Canada permission to use, reproduce, edit and share your comment royalty-free, in whole or in part, in any manner it chooses. You also confirm that nothing in your comment infringes third party rights (for example, the use of a text from a third party without his or her permission).

Comments

Comments are displayed in the language they were submitted.

Read comments

Gael is bang-on about this. I wish everyone comms would let go of this notion, that Grade 8 is some kind of gold standard for general-public communications. If you've never tried, getting most texts down to Grade 8, unless they were written for children or young adults in the first place, is linguistic limbo dancing.
Plain language now focuses on what's appropriate for the particular audience for each particular comms product.
There is no known algorithm that can substitute for the mantra "audience, medium, purpose" and a careful edit with those things in mind.

Yes! Thank you, Gael. Thank you for an additional resource about readability formulas.

I have long asked people to read Mark Hochhauser's great article [What readability expert witnesses should know (US) Issue 54 (pp. 38 – 42)
https://www.clarity-international.org/clarity-journal/page/3/] to provide background. I also came across this article (with quotes from Ginny!): https://www.uxmatters.com/mt/archives/2019/07/readability-formulas-7-rea...

And now I have your article, too, Gael!

I agree with you that current readability formulas do not account for many aspects of plain language, including usability factors.

I think if there is something to blame, it should not be readability as a bottom-up cognitive process. Rather, it should be the formula's inability to get updated to eventually contain usability factors. It is, for one, a mere technical issue to fix. An AI-based readability formula is the solution.

What do you think Gael?

Sorry, Mohammed, I saw your comment only now. Because readability formulas can't test for techniques that actually help make text suitable for readers, I don't see how making an AI version would help. But I am not a researcher. I was just summarizing what researchers found when testing the formulas.

Thank you for bringing this important issue to light! I did want to point out that Readability Tools are still recommended as part of Plain Language Guidelines for other Government of Canada pages (Example: https://www.canada.ca/en/treasury-board-secretariat/services/government-...).

Gael, do you know is these will be updated, or will Readability Stats still be considered "standard practice" despite their flaws?

Great question, Ashley! I have recently been talking to the folks responsible for the Canada.ca Content Style Guide. I emailed them just this week to ask about the possibility of aligning that guide to the ISO plain language standard. They are very interested in this idea. We'll have an in-depth conversation about it soon.

Date modified: