An Address Delivered to the New Mexico Association of Museums
By Rick Hendricks, Ph.D.
New Mexico State Historian
I am going to speak today about some of the lessons learned from the launch of the New Mexico Digital History Project in 2005 until mid-2012 when we began a renewal and redesign of the project. I will offer some examples of what worked and what did not work with the old site. I will also talk about where we see a public history site such as ours going in the future, especially how we may be able to collaborate with other sites within the New Mexico state system.
First, a little history: in 2004, a special appropriation from the State Legislature was used to initiate design and construction of the New Mexico History Website. A website team was asked to design a world class site that would assist the Office of the State Historian with his mission “to foster and facilitate an appreciation and understanding of New Mexico history and culture.” A second appropriation from the Legislature in 2005 enabled the Office of the State Historian to continue website development. The website was officially launched at the State Capitol on July 21, 2005.
During the 47th State Legislature’s second session in 2006, Senator Mary Jane M. Garcia introduced Senate Joint Memorial 46. In this document, the New Mexico State Legislature declared the website of the State Historian to be the “Official New Mexico State History Web Site.” They further acknowledged the distinctly unique position of the State Historian to be “charged with serving as New Mexico’s leading advocate and authority on matters relating to the interpretation of New Mexico history; …”
In 2008, our website won the Autry Public History Prize, awarded annually to “media, exhibits, public programs, or written works that contribute to a broader public reflection and appreciation of the past or serve as a model of professional public history practice in the history of the North American West.” The prize is made possible by the generous support of the Autry National Center. This is the website that I call “Big Red.”
The agonizing death of a loved one, such as your very own website, often demands a specialized vocabulary. The second law of thermodynamics, in principle, states that a closed system's disorder cannot be reduced, it can only remain unchanged or increase. A measure of this disorder is entropy. This law also seems plausible for software systems; as a system is modified, its disorder, or entropy, always increases.
Software rot, also known as code rot, software erosion,software decay or software entropy, describes the perceived "rot" which is either a slow deterioration of software performance over time or its diminishing responsiveness that will eventually lead to it becoming faulty, unusable, and/or otherwise called "legacy" and in need of upgrade. Perversely, normal maintenance of software and systems may also cause software rot.
Are you familiar with Software brittleness? Software brittleness refers to the increased difficulty in fixing older software that may appear reliable, but fails badly when presented with unusual data or altered in a seemingly minor way. Patches to fix glitches have probably been issued throughout the years, subtly changing the behavior of the software. In many cases, these patches, while correcting the overt failure for which they were issued, introduce other, more subtle, failures into the system. These subtle failures make subsequent changes to the system more difficult.
Over time, Big Red suffered from all of these conditions.
Here are some symptoms of Big Red’s impending death. One of the first indications that we had a problem came in August of 2010 when we learned that the bibliography was eating the website. Through some unexpected quirk of programming or a devious unauthorized visitor, a feature designed to provide visitors with a useful bibliography of sources on New Mexico history expanded at a frightening pace. With every new article that was posted to the site, the bibliography hungrily fed itself with each and every citation. By the time we discovered this ravenous beast, it was more than 1,300 pages long, impossible to search, shorten, or sever from the site.
Soon we learned that our popular discussion page, rather than a lively exchange of views about some historical matter, was displaying an obscene comment expressing what could be described as the Ottoman interpretation of the 1915 to 1923 Armenian Genocide.
In February 2011, our host reported that we had been hacked again. This time, the site had been used to launch ads for Viagra. The hackers rewrote metadata to make it appear that our site was advertising the popular libido enhancer. They hacked the database to obtain login credentials and then created a backdoor that gave them the ability to do anything they wanted with our data. For some months before we finally located the offending code, Google searches for newmexicohistory.org returned in highest priority a line of text that linked directly to mail order supplies of Viagra, a result that was particularly embarrassing to a state historian and assistant state historian of “a certain age,” who will remain nameless.
One of the more interesting hacks was launched from the University of Texas-Arlington, unless that semester’s class on the history of the Southwest was unusually active. In two breathless days, we got over two million visits to our MP3 files, which, as you might imagine, brought the site down. Our new site will have all of the Centennial Journeys, so we may be in for another onslaught. If you do not know about Centennial Journeys, you should check out the New Mexico Broadcasters Association website until we get the new site up and running.
Being a serial hack victim led to the website being, black listed, earmarked as an “evil site” by Norton, such that potential visitors were warned away lest they suffer dire consequences for landing on newmexicohistory.org. It took a fair amount of doing, and a thorough probing by Norton to get back in its good graces.
The treatment proscribed to fix the ailing site was installation of a new HTML editor, which promised better security. The old HTML editor was determined to be the chink in the armor that was giving hackers access to the inter sanctum of the website’s back end. And please forgive me if that sounds indelicate. If fix sounds too good to be true; it was. We should have recalled the law of unintended consequences and remembered that the cure can be worse than the illness.
Installation complete, firing up the computer and logging on to Big Red became a daily adventure into the unknown. Features came and went. Files completely disappeared. All the tables in the entire website disappeared. This presented us with a real poser. How do we know where to look for the missing tables? Put another way, how do we find something that is not there. About this time, we also gained another distinction, as far as websites go. Our frustrated host proclaimed our website the most problem-ridden and most frequently hacked site of the thousands it hosted, and not worth the relatively minimal hosting fee. It was strongly hinted that no tears would be shed if we were to go away.
Let me turn now to a premature post mortem: almost six months ago, when we embarked on a complete redesign and recreation of newmexicohistory.org, the website doctor said that Big Red was at death’s door, its demise certain and probably a question of days or weeks. But as of this morning, Big Red is still breathing.
As ghoulish as it sounds, we have already begun to autopsy Big Red, looking to see what went right and what went wrong, hoping to avoid a similar round of organ failures with the new site.
In fairness, when Big Red was created, it was probably as well done as possible for a custom design, and as websites go, it is now geriatric. Big Red is a custom-designed content management system. It was designed in such a way that nontechnical people could update it, but the customization allowed the system to become outdated as discussed above. There was no IT team in place with website expertise at our agency and no webmaster in the Office of the State Historian. Without a team of developers to maintain the system, it was doomed to eventual failure. What this meant for us is that as the years have gone by, we have had to rely on the services of a programmer on an ad hoc basis to make even simple changes and implement needed fixes. Our new site relies on Expression Engine, a commercial off-the-shelf software (known in the website game as COTS) with a large community of users and third-party designers that provide increased functionality and support.
Here are a few lessons we have learned from our experience with Big Red. We have learned that as desirable as it may be to have a stable of interns madly adding content to a rapidly growing site to make it “content rich,” and Big Red is nothing if not content rich, the concept has its drawbacks. Many brilliant ideas were never implemented; and text, images, audio and video were loaded in almost every conceivable format.
In the rush to roll out the site, the decision was made to eschew metadata with provenance and complete citations for most images. Many images were used without permission or without a complete record of having obtained permissions, which amounts to the same thing. This is doubly problematic for a division in an agency that includes the state records center and archives. First, because it does not meet archival standards, and second because it means that requests for permission to use images requires needless, often repetitive research, and no small measure of embarrassment and sincere atonement.
The original concept of the site was an invitation to get lost in New Mexico history, to make one’s own connections with the past. It was a place to read scholarly essays and the word of the common man and woman. It was a place to hear the sounds of New Mexico and see the sights. We will strive not to lose that aspect of the site, but research shows that most of our visitors are looking for discrete pieces of information, staying briefly, and leaving the site. Most of our searches are referred directly from Wikipedia. For that reason, we have expended a lot of effort to strengthen the search function of the site, which was not very robust.
Finally, many “things” (to use the technical term), do not work or lead nowhere. I cite the “Circle of Life” as an example. Users report that it is almost impossible to intuit what is meant. In a number of instances, clicking on an image/icon in the wheel returns nothing—no error message, no content. It is sort of a "Ha, Ha, the jokes on you" moment.
Now don't get me wrong, the site has become the go-to site for New Mexico history and is a vital outreach tool for the Office of the State Historian. Please also understand that I do not mean in any way to diminish the accomplishment of my predecessor, Esteban Rael-Gálvez, for his vision in conceiving and delivering on newmexicohistory.org. In my two and two-thirds years on the job, the feedback about the website has been universally positive.
It is indeed heartening to hear from school teachers around the state, many of whom use the website in teaching New Mexico history, as mandated by the state legislature.
All this talk about newmexicohistory.org got me to thinking: Is the Internet destroying history? Are we losing our memory?
The change in gubernatorial administrations in New Mexico that swept in the current administration of Governor Martínez led to loss of websites of the outgoing administration of Governor Richardson. This happened literally from one day to the next. Over the past two years, many if not most state websites have been redesigned.
This reflects a larger trend, a much, much larger trend. There are currently tens of millions of websites proliferating at a rate of 15% to 20% a year. Much of this explosion of electronically accessible information falls into what I will call the three Rs for the twenty-first or perhaps I should say twenty-second century: repeating, repackaging, and repurposing. I would argue that much of the information available on the web is the same information, after having been subjected to the aforementioned three Rs.
The task of systematically collecting and preserving this historical memory will inevitably fall to libraries, archives, and museums. And let’s face it, much of this information will be lost. Internet time capsules such as the Way Back Machine or initiatives by the British Library, which warned in 2009 of a “black hole” in history if website and digital files are not archived, preserve websites by the terabyte, but for a variety of reasons not everything. Australia, one of the world’s great sporting nations bemoaned the loss of its recent Olympic websites, and in England steps were taken to ensure that the 2012 Olympic websites are preserved for posterity. But losing millions of websites may not be so bad. After all, as the director of Peru’s national archive in an earlier era once observed, fire is a historian’s greatest friend.
Only a small percentage of written material has survived from before the age of electronic records, perhaps a mere 1 %. Recall if you will the library of Alexandria destroyed by fire in 48 BCE or the fate of such media as celluloid film. How about the millions of volumes printed on highly acidic paper turning to dust in the world’s libraries?
The Internet is increasing access to treasures of the worlds great libraries and archival repositories. In my own area of interest, there is access to the Archives of the Indies in Seville, Spain now at my fingertips. In 1992, to commemorate the Columbian voyages of discovery, what came to be known as "The Encounter of the Old World and the New," IBM and the Spanish government conceived of the idea of digitizing the millions of documents relating to the history of Spain in the Americas, or at least a few million pages of the documents most in demand. The original concept was to make them available from any computer, then from a couple of sites, and then from one terminal, in, say Chicago. For most of the 90s, the digitized documents were only available in Seville, in the archive reading room. What was the point of preserving the national patrimony of Spain if you could access it from Albuquerque or Cheboygan? Then, quietly, some years ago, Spain launched PARES and opened its digitized repositories to the world, and I suppose to the universe, if there is anyone out there searching. The same can be said for a rapidly growing number of archival repositories. In recent years, online access to historic newspapers and primary documents has created nothing short of a revolution in the way historians can conduct their research if they so choose. The Church of Jesus Christ of Latter-day Saints has delighted legions of genealogists and historians alike by beginning the process of digitizing their collection of millions of rolls of microfilm, which I can now access from any computer anywhere. These changes excite me and get me thinking about the future.
Where to I want to see our site go in the future? What I would like to see is for newmexicohistory.org to continue to be a source for authoritative information about the history of this great state. I have identified areas that are weak on Big Red and need our attention. Some examples are New Mexico’s role in the aerospace industry, the importance of agriculture to the history of the state, the history of mining, the contribution of Native Americans to the state’s history, and that of women. As the only state historian to occupy this office to hail from southern New Mexico, I would hope to persuade folks that there is history worth knowing about south of Socorro, and I do not mean just the Lincoln County War or Billy the Kid.
We are planning to add a significant educational component to assist teachers around the state better utilize our site. We will try to connect with the students who are often turned loose to roam around in the state’s history and culture.
As far as collaborating with entities around the State of New Mexico, let me remind you of the embarrassment of riches we have. Our State Museums are: New Mexico History Museum,
New Mexico Farm and Ranch Heritage Museum, New Mexico Museum of Space History, New Mexico Museum of Art, Museum of Indian Arts and Culture, International Folk Art Museum,
Natural History and Science Museum, National Hispanic Cultural Center, and Indian Arts and Culture Museum. Our state historic sties are at Coronado, Jemez, Lincoln, Fort Selden, Fort Sumner, and El Camino Real International Heritage Center. In a perfect world a visitor to newmexicohistory.org, hoping to find information about a topic such as New Mexico statehood, would find informative essays with links to primary documents in the State Records Center and Archives, to three-dimensional images of artifacts in the New Mexico History Museum Collections, to books in the Fray Angélico Chávez Library–one-stop shopping for the historically curious. I would like for GPS coordinates on a smart phone or tablet to send users with a query about of the place they happen to find themselves to newmexicohistory.org or to one of the websites of the nine museums and six state monuments I just mentioned, and all have websites, in the plural, because there is often more than one. Pretty high tech for a Spanish colonial historian more at home with the flourishes of pen and ink on paper in the seventeenth century than the bits and bytes of the twenty-first century, don’t you think?
But I am also a realist. Although I have had discussions with the director of the archives division at the State Records Center and Archives, and despite the fact that the archives website has recently been upgraded, we are no closer to seeing my dream realized. There is currently no way for use to move seamlessly between articles, images or other media on our site and the archive site. Our IT department has taken a larger role in assisting us with the new site than it did with the old site, but we still do not have a dedicated IT person or persons promoting the success of a site or sites that would have this desired functionality. With the right integrated system, that is not going to fall apart and die because of a lack of technical support, such a system could function successfully for years to come. But the level of coordination, not to mention funding, required to have such a system in place in one relatively small agency is daunting. For such a system to operate statewide, frankly takes my breath away. But I would like to suggest it is certainly worth talking about. Even historians can dream.