- 1 day ago
This episode originally aired May 23, 2024 at youtube.com/@Patristica
Here Mark announces the release of the first publicly available database (SQL/duckdb) of the Diorisis Ancient Greek corpus, containing 820 documents and more than 10 million classical and postclassical Greek word tokens. Mark explains how the database was built and populated, then provides a couple sample queries, with specific comparisons to the Diorisis Search program.
The diorisis duckdb is available at https://zenodo.org/doi/10.5281/zenodo.11261145
Audio-visual content © 2024 Clavis Consulting, LLC (Kansas), unless otherwise noted in the video. All rights reserved. Distributed under a CC BY-NC-ND 4.0 license. Thumbnail background © 2024 K. Lance Lotharp. Used with permission. None of this audio-visual content may be used as data for large language models, neural nets, or machine learning projects without prior written permission from the content creator, his heirs, or designees.
Patreon - https://www.patreon.com/markgbilby
Github - https://www.github.com/mgbilby
ORCID - https://www.orcid.org/0000-0003-0100-6634
Google Scholar - https://www.scholar.google.com/citations?user=dlsBqrAAAAAJ
Qn & Marcion's Gospel - https://www.doi.org/10.5281/zenodo.3927056
Marcion's Apostolos Translated - https://www.doi.org/10.5281/zenodo.8271823
e-Clavis: Christian Apocrypha - https://www.nasscal.com/e-clavis-christian-apocrypha
Here Mark announces the release of the first publicly available database (SQL/duckdb) of the Diorisis Ancient Greek corpus, containing 820 documents and more than 10 million classical and postclassical Greek word tokens. Mark explains how the database was built and populated, then provides a couple sample queries, with specific comparisons to the Diorisis Search program.
The diorisis duckdb is available at https://zenodo.org/doi/10.5281/zenodo.11261145
Audio-visual content © 2024 Clavis Consulting, LLC (Kansas), unless otherwise noted in the video. All rights reserved. Distributed under a CC BY-NC-ND 4.0 license. Thumbnail background © 2024 K. Lance Lotharp. Used with permission. None of this audio-visual content may be used as data for large language models, neural nets, or machine learning projects without prior written permission from the content creator, his heirs, or designees.
Patreon - https://www.patreon.com/markgbilby
Github - https://www.github.com/mgbilby
ORCID - https://www.orcid.org/0000-0003-0100-6634
Google Scholar - https://www.scholar.google.com/citations?user=dlsBqrAAAAAJ
Qn & Marcion's Gospel - https://www.doi.org/10.5281/zenodo.3927056
Marcion's Apostolos Translated - https://www.doi.org/10.5281/zenodo.8271823
e-Clavis: Christian Apocrypha - https://www.nasscal.com/e-clavis-christian-apocrypha
Category
📚
LearningTranscript
00:01Hello, welcome to the 10th or 12th episode of Gospel Open Data Science. I'm your host, Mark G. Bilby.
00:07Today I am officially dropping the Dioris' DuckDB. Many of you may be familiar with the Dioris' Ancient Greek Corpus.
00:17It's an amazing project that came out of the Turing Institute, headed by a team there, including Alessandro Batri and
00:24Barbara McGillivray.
00:25And that corpus is currently available through a Creative Commons license, CC BY 4.0, on Figshare and other places.
00:37But it's all available in XML. It's also available through a search interface that Alessandro built.
00:44So I'm going to show you that briefly, but I'm also going to show you a link to the database
00:50that I compiled using all of the Dioris' data and adding some new features to it
00:55that will allow for a much faster and more sophisticated querying and use cases for the Dioris' data.
01:05Yeah. So let's start walking through that. This is the program I was mentioning. It's called Dioris' Search.
01:14So if you go to Figshare, you can find the Dioris' corpus in TEI XML.
01:19It's about 820 documents, I think, totaling 10 million Greek word tokens, includes morphological tagging, the original word, the word
01:32lemma.
01:33The words tend to be in a beta form, beta code form, rather than Unicode.
01:40In the Dioris' search application, you can actually use either beta code or Unicode UTF-8 to search the Greek
01:47text.
01:48I haven't used it for Windows, partly because the application seemed to raise some security concerns for Windows,
01:55so I haven't used it in a Windows environment, and I can't recommend it one way or the other there.
01:59Because of that, I have used it in a Linux environment and found that it worked pretty well in that
02:05context.
02:05But again, I'm not because I don't want to make any claims.
02:08One way or the other about software and its usefulness, I'll just say that I've used it.
02:12It's worked okay for me, but I'm not going to make any claims or endorsements of the software one way
02:18or the other.
02:19But the software does allow for the selection of authors.
02:23I've gone in, for instance, and defined different corpora or mini corpuses within Greek literature.
02:31So in this case, I can load Greek novels, and then I can build a query and query a specific
02:38word or a specific morphological feature.
02:42Let's say I'm looking for all occasions of the optative mood across Greek novels.
02:48And maybe I also want to include Luke-Acts as part of the comparison along with the novels.
02:55So I've already chosen the morphological form, the optative verb.
03:01So this application does a pretty nice job of querying pretty quickly through these various documents.
03:10And it creates a sortable list, so you can start to see across the Greek novels how often.
03:16So about 0.5 out of every 100 words is an optative mood word according to these tallies.
03:24And Luke-Acts is pretty high.
03:26If we were to throw some more documents in there, we'd actually see that Luke-Acts ranks fairly higher compared
03:33to other documents.
03:35So here's going to be all the Septuagint, for instance, plus Philostratus, 2nd century, Herodotus, you'll get some history, Fabius
03:44de Oceavus, New Testament.
03:46So if you haven't used this, it might be worth checking out.
03:49Again, I'm not going to endorse it one way or the other.
03:51But if we were to run a query on all of these, it's going to take a little bit longer.
03:57But we should see some interesting results from this.
04:02So Alessandro was one of the persons who compiled the, built the DiorSys corpus, and then he developed this application
04:09as a way of accessing it quickly and efficiently.
04:13So we'll let that work.
04:16But the main purpose of this episode is to debut a DuckDB, which is essentially a SQL database of the
04:25DiorSys corpus, which I think I'm fully within my rights to do because the DiorSys corpus is licensed CC BY.
04:33So as long as I attribute the source of the data, which I already have and which I do in
04:38the official documents, it should be good to release it.
04:44So let's see, let me, I'm trying to open up.
04:47Oh, yeah, it's on my Linux box.
04:50Okay.
04:50Well, we can go look at the optative search results real quick, sort them and see that Philostratus, for instance,
04:57is a very high, the highest use of the optative.
05:01Daniel in the Septuagint, Josephus is kind of next, Xenophon of Ephesus, a novelist, writing in the second century, Herodotus,
05:11fairly high, and then Mark, Matthew, Luke is up there, fairly high, but then on the lower end would be
05:19Susanna.
05:22I'm trying to think if there are other texts here.
05:24I don't think so.
05:25I guess that's it.
05:26So it's not, these aren't all texts in the Septuagint, just a subset that I selected that were second century
05:31BC or later texts, but gives us a sense of the range of possibility for various compositions for that particular
05:39morphological feature.
05:40So something, something of a point of view or perspective that we can take.
05:46Let me open up the, I should have a link to this.
05:52Ready?
05:52Let me bring it up real quick.
05:55So if we go to this link here, which I will put in the video description, you can see where
06:03I've deposited the Dursis DuckDB.
06:06So it's a single file.
06:07It's about one gigabyte.
06:09So you should be able to download it.
06:10I won't endorse the data one way or the other, except just to say that I'm making this available.
06:17I have structured the data in some, some new ways.
06:22So it certainly pulls from the data that's already there, but there's also various transformations that happen.
06:29Some of the data is extracted from the TEI header of the 820 XML files, which took, this took a
06:39few days to, to build and extract and transform using a, you know, one of my laptops.
06:45And then the word table uses particular conventions, naming conventions that are used by Bob Gorman and Vanessa Gorman in
06:57their work on syntactical tree banks of Greek.
07:00So I want to be able to correlate the ancient Greek and Latin dependency tree bank data that the Gormans
07:09and Greg Crane and others have produced.
07:12And then especially recently out here, Smockers in the Glaux project has just released 20 million word tokens that are
07:20syntactically tagged in, in Greek.
07:22So he, with the 1400 documents.
07:26So I want to be able to tie those two together.
07:28And Bob Gorman has developed a method for parsing Greek syntactical tree banks.
07:35So I'm, I'm borrowing a few here, uh, the relevant field names that Bob uses, but only, uh, here, those
07:42that would apply to the Dioris's data.
07:44One caveat about the Dioris's data is that the tags are not always conclusive.
07:50So, you know, for word could be morphologically a dative or a genitive or evocative, uh, it, it will list
07:58multiple tags.
08:00So it doesn't necessarily make a decision or a probabilistic, a determination about, or, or a human based determination about
08:08what the actual form is.
08:10Where the Glaux data and the ancient Greek and Latin dependency tree banks data do make a determination, a single
08:16choice about what the morphological form is.
08:19Uh, some of them are somewhat ambiguous in the, uh, in those like medio passive could have a middle sense
08:26or a passive sense,
08:27but it's still a single value that's chosen for each of those fields.
08:31So, you know, with that caveat that there's a little bit of ambiguity in some of the fields in Dioris's,
08:37uh, it's still a lot of highly valuable data, specific words, specific lemmas, and, uh, a variety of morphological features.
08:44Some completely unambiguous, some somewhat ambiguous.
08:47Um, and then, uh, I've also used the Glaux, uh, TLG ID, essentially four digits, hyphen, three digits.
08:57Uh, as a way of, and it comes from the source under Greca, um, with the author coming first and
09:03then a document ID coming next.
09:05But I wanted to be able to tie together the Dioris's data and the Glaux data.
09:09So this is the structure of the, the database.
09:12It's now available.
09:13So you can use this DOI, uh, Zenodo.11261145, uh, to get to, um, this database.
09:24You can download it if you would like, and then you can start to build and run queries on it.
09:29So that's there.
09:30Uh, please take a look.
09:32And, uh, once you download it, uh, and the, uh, the Dioris's search program also can have you download the
09:41corpus just in XML, though, not as one single database file.
09:45But, um, let me just show a little bit of the background of, of how I built the database.
09:50Uh, so DuckDB is essentially a SQL database, um, service.
09:55Uh, it's, it's more and more popular in data science, especially if you're doing big data, work on big data.
10:00It allows for very powerful and quick queries of a lot of SQL data.
10:05So here will be put together a database that includes the entire Dioris's corpus, 10 million words.
10:12Um, it went through various conversion functions, so there's lots of little shorthand references to morphological features, and those need
10:20to be kind of separated out into what they're referring to.
10:23So in the case of, like, ACC, that's going to be a reference to case, dual be a reference to
10:29number, plu would be referenced in pluperfect as a tense, and so on.
10:35So I kind of parse these, including different idioms, which can be present in multiple values in a given field.
10:42Again, Dioris is, might not just have one value, it may have multiple values, or something could reflect multiple Greek
10:49idioms, or multiple Greek voices, or what have you.
10:53And then, um, there's a conversion function that I use, uh, well, that's, that's this, it's pointing back to the
11:01conversion function, but this is the script that populates the database.
11:05It goes basically XML file by XML file, pulls the, the data, parses it, converts it as needed.
11:13Okay, and then here's the, here's a way to read the database itself.
11:19I'm just going to clear out my environment, so you can see that this is all fresh, and I'll make
11:25this a little bit larger.
11:27So this is just going to open up, um, the first 40, 40 values in both the table, a document
11:37table, uh, so it pulls the, uh, the TLG ID,
11:43we have an author value here, all this is in the TI header, the name of the document, genre, which
11:48the Dioris's project provided, which is excellent, and then a sub-genre as well.
11:53And then the date created, I think this is a particularly interesting data point.
11:57I'm not always in agreement with how the Dioris's project tagged dates, particularly regarding New Testament text, because it tends
12:03to take a very conventional or traditional view of the dates of the text,
12:07text, but just having chronological tags for all these can become really interesting when we start to do various analyses.
12:15And these are calculations just based on, uh, sentence counts and words counts that I, I calculated, um, as I
12:22was putting these together.
12:24Um, location isn't filled out currently, but I wanted to put it there because I think there could be a
12:29lot of value in assigning locations to all of these documents as well,
12:33if we wanted to start to create like a geographical map of linguistic patterns.
12:38And then the final one is Glouc's, uh, Glouc's field, which is simply a Boolean true false is the document
12:45in the Glouc's corpus or not.
12:47And the vast majority of the, uh, Dioris's, uh, texts are in the Glouc's corpus, which can allow us to
12:54run queries on both and then cross validate results.
12:59Right.
12:59Uh, just adds another layer of, um, you know, of confirmation, scientific confirmation.
13:05This is just the tail end of, um, of these documents.
13:08You can see there's 820, uh, quite a variety going from 6th, 7th century BCE to 4th, 5th century CE.
13:19And then here's the, uh, the word documents are created a sort of a composite word key using the TLG
13:25ID plus, uh, a unique ID.
13:29That's a sequential ID for the document itself.
13:32Sentence ID.
13:33This is the word within the sentence, the form, uh, the lemma, and I could show you in the Dioris's
13:41itself, what this looks like.
13:43So, you know, the kind of conversion that's going on.
13:45So here's, here's one Dioris's document and, uh, you can see how everything is, you know, we have things like
13:53the creation date.
13:54I mentioned that before genre and sub-genre, those get pulled, but each, each word has its own set of
14:00tags and attributes.
14:02So all this was essentially pulled and parsed and expanded out, um, as needed.
14:09So this is, this is kind of the raw data, you might say.
14:12And then this is the structured, XML is a structured data form, but this is structured in a database format.
14:19Um, that's, you know, clear and unambiguous.
14:22Um, so it has a self part of speech, a self person, this would typically apply only for verbs, uh,
14:30the number, the tense, uh, the mood, the voice, the gender, the case, the degree.
14:37Uh, those are typically just for adjectives, uh, the idioms, and then prosody.
14:42That's like metrical value.
14:45Uh, so that's, that's that.
14:47Um, so now this is all in database format, uh, to my knowledge for the first time.
14:51Uh, and it could be that the Diorsis project folks have already compiled their own databases and, and maybe just
14:58not shared them yet.
14:59Uh, maybe they have shared them and I just don't know about it, but I just wanted to do this
15:03for my own research to be able to run really quick and sophisticated queries of Greek language patterns, um, uh,
15:13to, to assist in research on especially early Christian texts.
15:17And, and where they fit into the broader universe of Greek language patterns and their evolutions.
15:23So, um, this is, uh, we just read through the database.
15:27We can also do just a quick count.
15:28So you can see how many tokens we're dealing with.
15:31Uh, we have 11, 11 million, and this would include, uh, punctuation marks as part of the tokens.
15:37We can inspect, uh, words here.
15:42So, um, here we're just running a query and then it'll output to word tokens.
15:47And we're just pulling the last five words from each document.
15:50So this is just kind of a quality control, uh, measure.
15:55So we can just see what the last, last five, uh, tokens are of any given document.
16:02We're seeing here some punctuation marks that are a little interesting that might not be the best in terms of
16:08lemmas, but that's pretty rare.
16:10I think on the whole, you don't get a lot of those, uh, brackets in the Diorsis data.
16:16Um, so, yep, this all looks pretty good.
16:25And let's see.
16:27Um, and then we can start to do run query.
16:30So this is where it gets really, really interesting where we can start to do much more sophisticated analysis.
16:36So in this case, uh, compiled using chat GPT as, uh, my coding buddy, um, compiled some queries.
16:44In this case that pull from the works of Flavius Josephus, Lucian of Simasida, the New Testament, and the Septuagint.
16:52Um, and I can define the features.
16:56So here I'm just looking for words that have the trigram arc, A-R-chi, or alpha-ro-chi, which
17:04is one I had brought up before in previous presentations on YouTube because it's a particular trigram that occurs quite
17:11frequently in Luke compared to, uh, the Gospel of Marcion.
17:15Uh, usually in verb, verbal form.
17:18So the verb arcami, uh, or arco, uh, occurs quite a bit in canonical Luke and very seldom, if ever,
17:24in Marcion's Gospel.
17:25So I saw that as, as evidence of a later, uh, redactional voice, uh, in canonical Luke.
17:31So if we were to run this query and I'll, again, I'll, I'll just clear this out real quick so
17:35that, you know, I'm pulling from fresh data, pulling freshly from the database.
17:41So, uh, to open a DuckDB database connection, you can use this con, C-O-N language, uh, to open
17:49the connection.
17:50And, uh, so yeah, let's go ahead and run it and see what we get.
17:54Those, you can see how quickly that was.
17:56It was probably like three seconds.
17:57Uh, I could put a timer on it if I wanted to, but, uh, and what it produced is query
18:02results by document.
18:06So, and this is sorted in order of frequency.
18:08So it appears here from the query that we ran, 2nd John actually comes in highest in frequency of the
18:15trigram arc, ar, uh, alpha, ro, chi.
18:18Since it occurs two times in only 245 words.
18:22That's not a huge sample, so it might throw things off.
18:25A larger sample is going to be preferable.
18:27Jude also has a particularly high frequency, as does Hebrews.
18:31That's interesting.
18:33Lucian uses that trigram quite a bit.
18:35Josephus does as well.
18:37And then if we go down through other texts, you can see it diminishes pretty frequently or diminishes pretty significantly
18:43as we go down.
18:44So, uh, there's quite a range in Lucian's own compositions.
18:48There's quite a range in the Septuagint.
18:50Lucian can be both toward the bottom and the, toward the top of this very same list with quite a
18:55bit of flexibility in, in this, uh, range, uh, within the, so maybe, maybe not always a typical author feature
19:04or consistent author feature.
19:06Um, but it still gives us, you know, one characteristic that might be interesting to, to observe.
19:11I wrote another query, and this one is going to do, um, the optative.
19:21So this is the one that I was running before.
19:23So how, how, how frequently does the optative mood occur in these same authors?
19:29So Josephus, I think is part of the query.
19:32Lucian, New Testament text, and the Septuagint.
19:35I could add more.
19:36But, um, let me, again, clear this out so you can see that I'm pulling freshly from the database and
19:43just how, how quick this goes.
19:46So, uh, I'm going to run this.
19:51Oh, I need to correct, uh, a path, a file path.
19:58There we go.
19:59Okay.
19:59So, uh, it's, it's outputs to the same files and, um, we have by document.
20:06So how often does the optative occur?
20:10Looks like Lucian is far and away the most frequent user of the optative in this, uh, sampling.
20:17And it's highly characteristic.
20:19So there is quite a range, but Lucian can consistently, I mean, he's got the top 47 occurrences on this
20:27list.
20:28So optative, frequent optative uses is highly, highly characteristic of Lucian.
20:33Uh, in, among the Septuagint, the highest is fourth Maccabees, then Ruth.
20:38And then Flavius Josephus, the Antiquities, comes next at around 0.5 frequency.
20:46And then we could go down.
20:47Jude is pretty high, actually.
20:49Again, small sample size there.
20:52Here, some of the other epistles in the New Testament, and we could keep going down.
20:56So Marcion's gospel, for instance, has a very low frequency.
20:59Here we can see within New Testament text, Matthew actually has the lowest frequency.
21:03Now it says probably not coincidental that Marcion's gospel in many ways is closer to Matthew
21:09because they're both depending on, uh, a Q, Q source.
21:13So it's a significantly different Q than most scholars think.
21:16Uh, anyway, so that, that we can see very quick queries, but one of the other things we can do
21:22because we have date created, and this is something you won't find in the Diorsa search program,
21:27uh, because Alessandro didn't build in, uh, the ability to look for dates, but we can start now
21:34to group results by date, you know, by specific year.
21:37Essentially, this is something like a Greek engram tool, and this could very easily be turned into
21:43a visualization.
21:45So we can see that the optative in this case occurs very infrequently, uh, in the third century BC
21:52among the samples we have.
21:54It occurs very infrequently in the second century BC samples we have.
21:58So we get in the first century, middle of the first century CE.
22:02It's still pretty infrequent.
22:04It's upping a little bit.
22:05It gets way higher at the end of the first century.
22:08And this is, again, where I might want to quibble with some of the dates that the Diorsa's project
22:13assigned to texts in the New Testament, because, again, they're assuming traditional dates,
22:18like putting Luke around 80, canonical Luke around 80, when canonical Luke was probably finalized
22:23in the 130s, 140s, or 150s.
22:27Um, so to me, it's interesting, but not coincidental that we get a high rate of frequency here in the
22:34late
22:34first century among many texts that are associated with the New Testament, uh, as well as Josephus,
22:40and then a high rate here in the second quarter of the second century.
22:44And that's because I think a lot of these New Testament texts that are traditionally dated
22:48to the late first century probably belong to the second quarter of the second century,
22:53uh, where, where we see this, these, uh, frequencies occurring a lot more among non-Christian
23:00or non-canonical Christian literature.
23:03Um, so we can begin now, uh, using this database to do, like, time mapping,
23:08and chronological mapping of linguistic patterns.
23:11The TLG project allows us to do some of that, but their Ngram viewer tool is not particularly
23:16useful in my experience, nor quick, uh, but the DuckDB allows, again, just like three seconds
23:23to run very sophisticated queries, pulling in dates, grouping by dates, and doing these sorts of things.
23:30So I think this, this could be a very useful resource for people doing computational linguistics
23:35work more broadly, especially in Greek.
23:38Um, and that includes, uh, early Christian texts.
23:41So, um, that's it for today's episode.
23:44I just wanted to show that, point people to where, uh, this resource is, um, where it's living
23:51on, um, uh, the web.
23:57So let me, again, just point that out.
24:01There we go.
24:06I'll conclude with this, just, uh, to go back to that Zenodo site.
24:12Well, let me, uh, look up to the original, uh, here's, here's where the DiorSys project, uh, files
24:18live, if you want to go see that on Figshare.
24:22So you can download the entire corpus, uh, but the DuckDB version of it is now available
24:27as of May 23rd, 2024 on Zenodo.
24:31So please check it out.
24:33Please, uh, use it.
24:34Again, I'm not endorsing.
24:35There's no, like, warranties or, you know, I probably should put in it like a disclaimer.
24:39Um, but it's, this is all pretty simple data.
24:43Um, and it's, it's, uh, essentially just all pulled from, uh, the, the DiorSys XML data,
24:49as I mentioned before.
24:50All right.
24:51Thanks.
24:52That's it.
24:52I hope you have a good night.
24:54Thanks.
24:54Bye.