Documentation for the Robert Graves Diary
author: Martin Holmes (MDH); Joey Takeda (JT)

Documentation for the Robert Graves Diary

Martin Holmes and Joey Takeda, 2018-2020

Introduction

This is the documentation and schema specification for the Robert Graves Diary Project. It covers a brief history of the project, the basics of the encoding practices, some information about special features, some straightforward instructions for anyone who has to maintain the project, and anything else I can think of that might be useful.

Background

The Robert Graves Diary is one of the earliest digital edition projects undertaken by the University of Victoria Humanities Computing and Media Centre, and the UVic Libraries and was a pioneer project in the use of the eXist XML database. The first implementation was completed in 2003, mostly by Spencer Rose, based on TEI P4 encoding. The project got some attention in the TEI and DH communities, and was generally quite well thought-of. It continued to run for many years on an old version of eXist-db running under Tomcat and Cocoon. Updates were done to keep it running between 2005 and 2011 by Martin Holmes and Michael Joyce. Then in 2015 Martin converted the original P4 encoding, which had a number of problems in its structure, into valid TEI P5 XML.

In 2016, a grant application called the Endings Project (see UVic OAC site and GitHub repo), led by Claire Carlin, was funded by the Social Sciences and Research Council of Canada. Its mission is to develop, test and document best practices in the design and implementation of digital edition projects with a view to long-term survival of not only the data but the web application and its main functionality.

The Robert Graves Diary project was a perfect case study for this process, so it has been subjected to the Endings approach. Instead of a web application which ingests XML and generates XHTML on the fly with XQuery and XSLT, a build process was created which takes to P5 XML documents and processes them to create an entirely static website that requires no back-end processing or server support in order to run. This ensures that whatever happens to eXist-db in the long-term, as long as there are web browsers capable of interpreting HTML5, CSS and JavaScript, the site can be browsed and used.

In addition to generating static pages, the site itself was completely reorganized to eliminate the long and confusing query-string URLs that typified the original site, and replace them with simple URLs that can be easily bookmarked.

A static-HTML site, though, does not automatically provide a search functionality. To reproduce the original search, we again deployed eXist as the host for the site, but in a much more straightforward setup where it indexes and serves the HTML rather than the XML, and therefore has less work to do, leading to less maintenance and a simpler configuration. To handle the possible future demise or non-support of eXist, though, other search options have been created: see The four search engines.

The build process

The Endings project has developed and refined an approach to site-building which is based on the following toolset:

- The Subversion version-control system, which stores virtually all of our project data.
- The Jenkins continuous-integration server, which is set up to watch our Subversion repository, and build a new version of the entire project every time anyone commits a change.
- Ant, which manages the build process.
- Saxon, which provides XSLT processing to generate the output website content, as well as various versions of the XML source documents, from the P5 XML encoding.
- The Nu Html Checker (v.Nu), which is used to validate the output web pages that constitute the site.

The two search engines

This site has two search engines, arising out of our work on reliable fallback systems for long-term survival of digital editions projects. These are the two search systems:

- staticSearch. This uses an open-source codebase created as part of Project Endings by Martin Holmes and Joey Takeda. This is a pure-Javascript search system requiring no back-end server-side processing; it just needs a web server. Its static indexes are built at the end of the regular site build process, so each version of the site has its own custom-built index setup that travels with it.
- Google custom search. At the time of writing, Google offers the option of a relatively configurable Google Custom Search page, which uses Google's index to search the collection. This is obviously a fragile function, since Google's APIs and conditions of use change frequently, but it does provide another option which will work if necessary on a static version of the site, provided it is running on its canonical URL of graves.uvic.ca.

Project materials and organization

The project materials are all kept in a Subversion repository at https://hcmc.uvic.ca/svn/graves/. See the section on Using the Subversion repository for detailed instructions on how to get the materials and manage the files. The folder organization is as follows:

- css/graves.css (the CSS style file for the HTML site)
- documentation/ (the latest HTML build of the documentation, along with its own CSS file)
- graphics/ (graphics files used on the site, not including page-images)
- jenkins/ (a log-parsing rules for for the Jenkins CI server build process)
- js/ (a single JavaScript file to provide some site interface features)
- obsolete/ (A folder containing superceded code, failed experiments, and various notes)
- presentations/ (a presentation on early Endings work related to this project)
- utilities/ (various modules and libraries for processing and validation)
- xml/ (the XML-encoded transcriptions and source for site pages)
- xsl/ (the XSLT code which builds the site)
- VERSION (a text file with the current edition number for the project)
- build.xml (the main Ant build file for the project)
- buildSchema.xml (the Ant build file for processing the ODD file into schema and documentation)
- config_staticSearch.xml (configuration for the staticSearch engine)
- google60f99c27b27affc9.html (the file which enables the Google Custom Search page)

Using the Subversion Repository

We keep all the project files in a Subversion Repository. This is a version-control system that ensures that every version of every file can be retrieved if necessary, and prevents one person from inadvertently overwriting changes to a file made by someone else.

Subversion runs on one of our HCMC servers, which is at hcmc.uvic.ca/svn/graves. In order to use it, you will need to install a Subversion client on your computer, and also learn a couple of simple command-line commands. Subversion is usually abbreviated to svn.

Installing a Subversion client

How you will do this depends on which operating system you are using.

Windows

Obtain a command-line client from CollabNet (http://www.open.collab.net/downloads/subversion/). Registration is required to download the program, but there is no cost. Make sure to download the correct version; there are versions for 32-bit and 64-bit Windows. Once the program is downloaded, install it by double-clicking the downloaded installer and following its instructions.

Macintosh

On older versions of Mac OSX (prior to 10.8), the Subversion client was installed by default, so if you're using one of those versions, you don't need to install it. However, it was removed from the default install of more recent versions. The simplest way to install it is to use Homebrew.

Linux

Subversion is installed as part of a regular desktop on most Linux distributions.

Checking out the repository for the first time

Once your subversion client is installed, the first thing we need to do is to check out the repository. To do this, you need to open a terminal window:

- On Windows, click on the Start menu and type cmd into the search box.
- On a Mac, select Terminal from the Utilities folder in Applications.
- On Linux, press Control + Alt + T.

Now we'll check that svn is installed and working. Type svn checkout and press return. If the terminal responses that there are Not enough arguments provided, then svn is working OK.

Now we'll create a directory for our project files:

mkdir graves

And now we've created that directory, we'll navigate into it:

cd graves

And now we'll check out the files from the repository into our directory. Make sure you don't forget the space and period at the end of this command:

svn checkout https://hcmc.uvic.ca/svn/graves .

The server may ask for your netlink user name and password. Then you should see the files download to your drive.

Using SVN in your daily work

These are the basic rules when you're working with svn:

Before you start work, update your local files:

- Open a terminal
- cd graves
- svn update

You can leave the terminal open while you work. Then, when you're ready to commit your changes:

- svn update (again, just in case anything else has been committed by someone else)
- svn commit -m "A message explaining the changes you have made"

That's basically it. If you see any warnings or error messages from svn, check that you're in the right folder in your terminal. You may also see error messages if two people have been editing the same file at the same time, and Subversion needs you to make a decision about whose changes should be kept. Contact Martin immediately if you have any problems with svn; it's best to solve problems quickly so that your work is always up to date and stored safely.

How to do stuff

This is a brief guide to the sorts of tasks you might need to do to maintain the project.

How to get the project materials

All the project materials are stored in our Subversion repository. See Using the Subversion Repository for information on how to check them out and commit changes.

How to open the Oxygen project

All editing should be done with the Oxygen XML Editor. Start Oxygen, then click on Project / Open Project, browse to the project folder, and select graves.xpr. When Oxygen asks you if you want to allow it to make changes to the Oxygen interface, say OK.

How to rebuild the site

As long as our Jenkins server is running (anyone in HCMC should know where it is), you don't actually need to build the site; you can just commit your changes and let Jenkins do it for you. However, in the long term that server may not be there, or the Graves job may no longer be supported on it, so you may have to build the site yourself.

To do this, you will need to be on a computer which has Java and Ant, and ant-contrib will also have to be installed. Search the web for the best ways of doing this on your own system.

Open a terminal in the root folder of the project, and type:

ant -lib utilities

If all goes well, a lot of material should be created in the products folder in the project root, including a diagnostics file (check this to make sure you did not introduce any new errors), and the website itself, in the products/site folder.

<handNote>
@medium
"black"
Written with black ink
"lead"
Written with a lead pencil
"red"
Written with red ink

<rs>
@type
"person"
A string referring to a person.
"organization"
A string referring to an organization.
"place"
A string referring to a place.
"cita"
A string citing a bibliographical item.
"element"
A string referring to an elemental feature such as the Sirocco wind.