How to Calculate the Relative Influence of an Author

At the end of the first century, Quintilian asked “Is it not sufficient to model our every utterance on Cicero? For my own part, I should consider it sufficient, if I could always imitate him successfully. But what harm is there in occasionally borrowing the vigour of Caesar, the vehemence of Caelius, the precision of Pollio or the sound judgment of Calvus?”

As philologists of the 21st century, we might ask “How often did Roman authors actually borrow phrases from Caesar as opposed to Cicero?”

Caitlin Diddams and I recently published an article in Digital Scholarship in the Humanities which lays out the best practices for determining:

  1. Which phrases shared between two authors did not come from a second possible source
  2. How to measure the relative strength of an “intertextual signal”
  3. How to compare the relative influence of multiple authors on a cross-section of literature

As a test-case, we compared the influence of Cicero and Caesar during the early imperial and late imperial periods.

The methodology we outline in this article can be used on any number of source and target authors, regardless of language. Our formula for calculating the strength of an intertextual signal can be used with any tool for detecting intertextuality (not just Tesserae).

To read the abstract and obtain the full article, visit the Oxford Journals website:

Relative influence in our methodology is compared according to the ‘rate of intertextuality,’ which is a normalized representation of the number of results you get in a Tesserae search. Normalization is necessary because the length of a work influences the number of results obtained. Previous methods of normalization assumed that Tesserae’s scoring algorithm would perform consistently across various authors and genres of literature. We propose that best practice should avoid such assumptions wherever possible.

Our normalization method in brief (the following is excerpted from a pre-print copy of the article):

The number of results of two searches cannot be meaningfully compared until we consider how many results each search could have produced. The number of search results depends on two factors: the level of engagement between the authors and the length of the texts being compared. Longer texts create more sentence-by-sentence comparisons. There are more opportunities for unique intertexts to occur. The number which can be meaningfully compared is not the number of unique results of a Tesserae search, but the ratio of the results found to the results that could have been found. We normalize the number of results according to the following formula:

We define the rate of intertextuality as the number of connected phrases per pair of phrases considered. This is derived by dividing the absolute value of the set of results by the absolute value of the cross-product of the sets of sentences in source and target texts. This cross-multiplication is necessary because Tesserae compares every sentence in a source text to all of the sentences in a target text.6 Therefore the number of possible results in a comparison of any source and target is the product of the number of sentences in the source and the number of sentences in the target.

Measuring the Distinctiveness of Phrases in Latin Epic

Measuring the co-occurrence patterns of words with pointwise mutual information (PMI) can help identify bigram word-pairs that are unusually represented in the work of a given author. By comparing the PMI values of the Latin epic corpus to the PMI values of Vergil, for example, scholars can discover which word pairings are particularly Vergilian. Many of these Vergilian phrases will be obvious, such as pius Aeneas and puer Ascanius. Others, however, invite further investigation. Some word pairings are so unexpected that they may be sufficiently marked for quotation and imitation.

Tesserae is in the process of incorporating PMI data as an option for scoring search results. Tesserae scores currently rate rare words shared between two texts as more likely to constitute an allusion. This is problematic for capturing allusions from Vergil, who is known for combining common words in uncommon ways, in what ancient critics called a new form or affectation (cacozelia). The incorporation of comparative bigram frequencies can more accurately score allusions to Vergilian bigrams, which would otherwise be erroneously demoted. For example, if a search result is particularly indicative of the source author, but not of the corpus or target author, this might indicate that the target author is quoting a recognizable phrase. In this case, the Tesserae score should be increased. If the match is indicative of a target author’s shared language, but not of the corpus or the source author, it is less likely that the target author is trying to evoke the source author. In this case, the Tesserae score should be decreased.

Many studies from the 1990’s on have shown the efficacy of analyzing word co-occurrence patterns in English. In 2000, Rydberg-Cox adapted existing methods for ancient Greek as a basis for philological research. PMI values represent a ratio of “actual” versus “expected” frequency with which two words appear near each.

The actual frequency of a bigram is a measurement of the frequency with which a combination of words x and y occurs. The expected bigram frequency is a measurement of the frequency with which words x and y might have occurred as a bigram based on the frequencies of its constituent unigrams. This represents the bigram frequencies we would see if the distribution of each word were independent of the distribution of the other. In reality, contextual and syntactic relationships change the likelihood that word y will follow word x, and so the actual and expected frequency values diverge. Finally, because PMI overemphasizes low-frequency collocations, it is standard practice to cut off extremely rare words and to log and normalize the results.

Results from the Aeneid and the corpus are then normalized so that PMI values can be meaningfully compared. Normalization translates the scale of the PMI values from Vergil and from Latin epic authors to a range from -1 to 1. Positive PMI values indicate that once you read one word in Vergil, the uncertainty of the next words dramatically shrinks. Negative PMI values indicate that the presence of one word in Latin epic negligibly affects the possibility pool for the next word.

Consider the following example of a high PMI value from Vergil’s Aeneid 12.338: fumantis sudore quatit, miserabile caesis. Fumantis sudore describes horses frothing with sweat, and has a normalized PMI score of 0.684. Since PMI values for Vergil indicate that fumo usually occurs with incense, altars, food and homes, and sudor usually occurs with people, blood, and labor, fumantis sudore is “marked” or unusual phrase in Vergil. Since fumantis sudore is not a high ranking result in the PMI values of the Latin epic corpus, it is further likely to be an example of particularly “Vergilian” language.

The following graph shows the PMI values for Vergilian bigrams that also exist in the epic corpus. Most of the PMIs are positive, indicating strong associations between words. The data with the highest PMI values represents the strongest word associations in Vergil.

The next graph shows the PMI values for bigrams in the epic corpus that also exist in Vergil. Here, the PMIs are mostly negative. This indicates that in epic as a whole, word association is more flexible than in Vergil alone.

These graphs indicate that Vergil’s word associations as different from those in the epic corpus generally. For example, Vergil’s normalized PMI for aequore~toto is about 0.5, occurring 6 times. In the rest of Latin epic, aequore~toto appears 9 times and has a normalized PMI of 0.03. The difference is that in Vergil, aequore expects toto, whereas is in epic generally, aequore does not prime the reader to expect toto.

The data does not tell us which author differs from the corpus more dramatically – other normalization factors will have to be put into place before we can compare, for example, Vergil’s distance from the corpus to Lucan’s distance from the corpus. Beyond its applications for Tesserae, co-occurrence patterns can improve our understanding of what phrases are more striking or marked than others, and of what constitutes the recognizability of an ancient author’s hand.

Appendix to “Measuring the Presence of Roman Rhetoric: An Intertextual Analysis of Augustine’s De Doctrina Christiana IV”

This appendix contains the intertextual parallels that inform the paper “Measuring the Presence of Roman Rhetoric: An Intertextual Analysis of Augustine’s De Doctrina Christiana IV” published in Mouseion Vol. 14 No. 3, Open Digital Corpora of Greek and Latin. The search parameters for these comparisons are listed at the beginning of each file. Please direct any questions to Caitlin Diddams at or James Gawley at

 Vita Washingtonii vs. DDC

Germania vs. DDC IV

Bello Gallico vs. DDC IV

Dialogus vs. DDC IV

Orator vs. DDC IV

Institutio Oratoria vs. DDC IV



This paper examines the intertextual relationship between Augustine’s De Doctrina Christiana IV and Cicero’s Orator. We use quantitative methods to compare Augustine’s level of engagement with Orator against his engagement with other handbooks of classical Latin rhetoric. Our results inform a close reading of the text as body metaphor in DDC 4.13. Augustine incorporates Ciceronian colometry into his presentation of the epistles to demonstrate Paul’s eloquence. We argue that Augustine’s comparatively heavy use of Cicero is an attempt to justify the use of rhetoric in Christian teaching while adapting that rhetoric to Christian purposes.