Chris and I recently submitted an article, “Claudian’s Engagement with Lucan in his Historical and Mythological Hexameters,” based on our presentations in Geneva in November 2012, for inclusion in a conference volume to be published by Winter Verlag. It focuses on Claudian’s creation of intertexts (high-scoring bigram lemma matches found by Tesserae) consisting of phrases that are unique between two of his poems, De Raptu Proserpinae and De Consulatu Stilichonis, and Lucan’s Civil War. The idea is that phrases that are unique to Claudian and Lucan could be of particular interest in their intertextual relationship. “Unique” in this case means that the phrases do not appear in the other authors prior to Claudian in our corpus of Latin poetry at the time the article was produced (this included all canonical poets, but the corpus has since grown). To put Claudian’s intertextuality in context, we also produced similar comparisons of the intertextual relationships between prior epic poets and the Aeneid and Civil War respectively.

The data for these comparisons is available in folders through the following links:

Comparison of later epic poets with Vergil’s Aeneid

Comparison of later epic poets with Lucan’s Civil War

WARNING: the files are large, from 4-70MB.

*** See our updated benchmark data on our recent blog post “Collected Benchmark Sets” ***

Here is the data produced by our two surveys (in 2010 and 2012) of intertexts between Lucan, Bellum Civile 1, and Vergil, Aeneid.

The 2010 spreadsheet lists parallels reported from six different sources: four professional commentaries and two versions of Tesserae. Each parallel was hand ranked by members of our team of graduate student and faculty readers.

This is the source of the data reported in the 2012 TAPA and LLC articles.

The 2012 spreadsheet lists all parallels returned by a Version 3 search of the same two texts, plus any parallels found in the commentaries but not returned by Tesserae. The presence of a given parallel in one or more of the four commentaries is represented by the commentators’ initials. This sheet gives hand ranks from both the 2012 and 2010 tests.

Click to Download:
Tesserae 2010 Benchmark
Tesserae 2012 Benchmark

Please feel welcome to contact us with comments or questions on these data.

We’ve recently fixed a small bug in the scoring system, and you may notice that some scores are higher than they used to be.  Scores are calculated as floating-point values, but displayed as integers in the web interface.  To this point the decimal part of the score has been truncated, so that all scores were effectively rounded down to the next lowest integer.  From now on we will use the more customary rounding rules, so that partial scores equal to and above .5 will be rounded up.  If you compare the results of a search done now to those of the same search before the change, you can expect about half of the scores to be one higher.

As we continue our research on quantifying the literary significance of allusions, scores may change again, and perhaps significantly.  We will post notice and explanations of any such changes here. If for any reason you need access to a previous version of the software in order to replicate older results, please just let us know and we can help you.  Every version of Tesserae, once published on our web site, is archived and can be retrieved.

Harry Diakoff has shared with us a set of Greek synsets—groups of words purported to be mutually synonymous—while Tesserae has an algorithm which is supposed to measure semantic similarity between any two words. While both approaches ultimately are based upon the Perseus XML version of the Liddell-Scott-Jones lexicon, they employ very different methods. We hope that by comparing the two we can improve upon both.

Synsets

I don’t really know how these were created—maybe by translating entries in the English WordNet?  Harry, can you fill in any details here?  The important characteristics of the synonym sets are:

• each set has a unique id
• within a set, all relationships are presumed to be mutual and symmetrical
• words can belong to more than one set

Similarities

Tesserae calculates word similarities using the Python topic modelling package Gensim.  We treat every entry in LSJ as a “document,” which is digested to produce a bag of English words used in defining the Greek headword.  These English words are TF-IDF weighted and used to create a feature vector describing the headword.  Headwords are compared using gensim.similarities.Similarity()—for any query word this returns a score between 0 and 1 for every other word in the corpus.  In addition to this absolute similarity score, we can also sort all results by score and consider the rank position of a given result some measure of its relationship to the query word.

• each pair of words has a unique similarity score
  • some words within a synset can be more alike than others;
  • but homonyms are flattened
• this similarity is symmetrical, but the rank positions aren’t:
  • the rank of result B given query A is not the same as that of A given B

You can see the code I used to calculate these metrics in the synonymy directory of the tess.experiments git repository.  But please be patient—it’s still quite rough; please feel free to improve it…

Current problems

Both of these datasets have their difficulties.  Each set is probably missing some headwords.  The synsets include some false positives and negatives.  The similarity scores can’t be turned into lists of synonyms without a threshold—either a similarity or rank position—that defines synonymy.

Ultimately, we need ground-truthing.  What follows is merely a first attempt to compare the two approaches and figure out to what degree they are in agreement, and to get some ideas about where and in what ways they differ.

A first glance

What I’ve done here is to break Harry’s synsets down into pairwise relationships, and then measure similarity and rank position (in each direction) for all of the pairs that exist in the Tesserae similarity matrix.

Harry sent us 17,342 synsets, which decomposed into 235,702 unique pairs of words.  Of these, 174,816 pairs returned results from a Tesserae similarity query.  In the remaining cases, one or both of Harry’s words didn’t exist in our corpus.  Although we both used the same dictionary, we each had our troubles reading it; more about this in another post.

Initial results look like this:

SIM gives the similarity score for each pair; RANKA gives the rank position of the righthand member among results when the lefthand member is used as the query; RANKB, the rank position of the lefthand member when the righthand one is the query; and SIMSETS gives the id number(s) of synsets in which this pair appears. In the bottom row you see what happens when Tesserae can’t find one or both of the words—in this case it seems that one member of the pair is actually a phrase, although there are other cases where Tesserae can’t find a word that clearly should be in the dictionary.  You can download the full dataset here.

Similarity and synonymy

Given that all the word pairs extracted from the synsets are supposed to be synonyms, and that similarity is supposed to be a measure of synonymy, we might hope that most of the pairs will have high SIM scores.  This didn’t turn out to be the case: while a significant number scored 1, the majority of the pairs scored 0; among the rest, there seemed to be preference for low scores over high.

Rank position and synonymy

On the other hand, rank position did better.  I added together RANKA and RANKB to flatten out weird asymmetries for now, and found that a large majority of word pairs had high ranks:

It seems safe to say we’re not interested in results that ranked 50,000th in an ordered list of most similar words.  Here’s a closeup of just the top-ranked (i.e. furthest left on the x-axis) according to RANKA only.  It does pretty much what we had hoped for:

So it seems on a first pass as though rank is working out of the box, while similarity needs work.  What if we use rank as a filter on similarity?  Here is the distribution of similarity scores among pairs whose combined RANKA + RANKB is less than 100.  Not only are these pairs high ranking, but they’re also relatively symmetrical, given that the two ranks can differ by no more than 99 in each case.  Here, the huge spike at SIM=0 is gone; the spike at 1 is preserved, and the rest form a nice curve around the middle of the similarity spectrum.

Clearly more work to be done here, but this seems to be an exciting start!

Adding texts to Tesserae’s searchable database requires ensuring that every line has a human-readable locus associated with it.  Checking through Perseus TEI and selecting one, consistent numbering system to apply to all the lines of a text is no easy job—as James and his team of interns will readily attest.

One way in which this might be made easier is processing similarly-structured texts in batches.  But TEI is flexible enough that texts with the same structure in print (e.g. Book–Poem–Line, or Book–Chapter–Section) don’t necessarily have the same XML structure.  To take a simple example, some poetic texts enclose each line in <l> tags, with line numbers coded as attributes of the line elements, as in the case of Ovid’s Metamorphoses:

<l>In nova fert animus mutatas dicere formas</l>
<l>corpora; di, coeptis (nam vos mutastis et illas)</l>
<l>adspirate meis primaque ab origine mundi</l>
<l>ad mea perpetuum deducite tempora carmen.</l>
<l n="5">Ante mare et terras et quod tegit omnia caelum</l>
<l>unus erat toto naturae vultus in orbe,</l>
<l>quem dixere chaos: rudis indigestaque moles</l>
<l>nec quicquam nisi pondus iners congestaque eodem</l>
<l>non bene iunctarum discordia semina rerum.</l>

Other texts encode the same structure by interspersing numbered line breaks throughout a block of text, as in Silius Italicus’ Punica:

<lb rend="displayNum" n="1" />Ordior arma, quibus caelo se gloria tollit
<lb rend="displayNum" n="2" />Aeneadum, patiturque ferox Oenotria iura
<lb rend="displayNum" n="3" />Carthago. da, Musa, decus memorare laborum
<lb rend="displayNum" n="4" />antiquae Hesperiae, quantosque ad bella crearit
<lb rend="displayNum" n="5" />et quot Roma uiros, sacri cum perfida pacti
<lb rend="displayNum" n="6" />gens Cadmea super regno certamina mouit
<lb rend="displayNum" n="7" />quaesitumque diu, qua tandem poneret arce
<lb rend="displayNum" n="8" />terrarum Fortuna caput. ter Marte sinistro
<lb rend="displayNum" n="9" />iuratumque Ioui foedus conuentaque patrum
<lb rend="displayNum" n="10" />Sidonii fregere duces, atque impius ensis
<lb rend="displayNum" n="11" />ter placitam suasit temerando rumpere pacem.

When it comes to automatically adding the correct locus to each line of text, these two encodings demand different treatments, as in one case the line number is an attribute of the parent element, whereas in the other the line number is an attribute of a sibling.

I thought it might be interesting to see whether we could automatically classify texts based on the type of XML tags used in encoding them. This could identify which texts would need similar treatment without making assumptions based on the way the print texts were structured.

I decided to try a rough classification of documents based solely on what kinds of nodes they contained and the hierarchical arrangement of those nodes. For example, you can guess which of the following paths occurs in Cicero’s Letters to Atticus, and which in Plautus’ Menaechmi:

TEI.2/text/body/div1[@type='book']/div2[@type='letter']/opener/salute
TEI.2/text/body/div1[@type='act']/div2[@type='scene']/sp/speaker

I generated a list of all unique paths from root to leaf in each text. I only kept attribute values in two cases, the @type of <divn> and the @unit of <milestone>. This is because important information about the structure of the text may be in the attributes here.

In this first experiment I didn’t even bother considering how many instances of each path a text contained; I just set the feature to 1 if the path was present and 0 if not. Each text was ultimately represented by a vector of 1095 binary features, one for each of the unique paths that occurred anywhere in the corpus.

Here we see the texts represented by the first two principal components of those feature vectors. The points have also been colored according to an independent, k-means classification of the original vectors into 8 classes.

For me, three things jump out immediately: first, that drama is set apart from all the other texts; second, that Cicero manages to cover almost the entire feature space; third, that the remaining genres do cluster, but overall tend to show a gradient of characteristics.

Even as we continue to work on a universal text-parsing tool, this line of investigation could potentially speed the addition of Perseus texts.  I think the next step will be add information about who edited the digital text to the feature vector.  This will help move classification from primarily genre-driven, identifying differences we could have predicted, to include TEI coding idiosyncrasies such as the difference in line numbering illustrated above, which we wouldn’t have been able to guess without examining all the files by hand.