BAM2016_Final

Reflections on the BAM Conference

BAM2016_FinalI had the absolute privilege of attending the Big Ancient Mediterranean Conference (#BAM2016) this week. The remarkable projects, enthusiasm for all things digital, and congenial atmosphere was inspiring. Now that the conference has ended, I think it is a good time to organize my thoughts, and perhaps point out some of the common themes that particularly struck me.

1) Our projects are ready talk to each other. This was one of the most exciting revelations of the conference. Many of the digital humanities projects and initiatives represented here not only offer their data in downloadable format (.csv files, JSON dumps, etc), but also feature feature-rich APIs. Even if we are not quite yet to the point where we are using the same meta-data / data standards (more on that later), the use of APIs with permanent URIs allows our data sets to meaningfully interact. The work of Pelagios creates an excellent medium to facilitate such communication, and opens up our data to initiatives that are not limited to studies of the ancient world.

2) Users, users, users, users. We had some spirited and fascinating debate about who the audience is for digital humanities projects, and if it is even possible to create an application that can be effectively used by different audiences (experts, the general public, grad students, etc). I fall squarely on the side of the idea that we engage with multiple audiences by the very nature of a freely-accessible online platform, but our debate revealed a fundamental design question that is often not explicitly addressed: Exactly *who* is a digital humanities project for? Although I may differ with the voices questioning the multi-audience approach, I certainly agree with the position that we need increased usability studies and more robust user information. It is not enough for us to create DH projects that answer our individual questions for ourselves – we need to understand how to communicate with an audience which is used to the visual literacies of web and is less familiar with the conventions of scholarly communication derived from a print medium. The sample edition of Calpurnius from the Digital Latin Library (http://digitallatin.github.io/viewer/editio-2.0.html) is a great model – it captures the information of a textual apparatus free of technical jargon, rendering critical information to a wider audience without a loss of scholarly rigor.

3) Uncertainty. A corollary to the discussion around users is the question of representing uncertainty. There was an interesting question of why we should recognize fuzzy data at all: if an application is directed solely at an academic audience, is it not correct to assume that our users implicitly know that any data or representation of the ancient world is somewhat problematic, and therefore have no problem consuming visual representations that ignore the idea of uncertainty entirely? As I think that our projects need to communicate with non-academic audiences (and indeed academics who may not be as familiar with the inherent uncertainty of the ancient world), I see a very real need to represent the imprecision and uncertainty of our data. Almost all of the projects at BAM grappled with fuzzy data, whether that was geo-spatial (location, assignment to a place), textual (uncertain letter forms, unclear manuscript tradition), or interpretive (multiple archaeological reconstructions, the placement of garrison soldiers at a specific community). Almost every project dealt with uncertainty in a way that reflected the scholarly tradition of their subject area, like placing notes in an apparatus, or describing fuzzy data through text. I see a critical need to establish a common meta-data vocabulary that can, at the very least, alert users (both human and computational) to the presence of uncertainty in our work. I also see room for a common visual literacy for representing uncertainty in maps, social networks, or other visualizations, which is a far more complex issue.

4) Metadata and documentation. For example, even if it proves impossible / impractical / or undesirable to create a common visual literacy surrounding uncertainty, we need to implement a common way of indicating and describing fuzzy data that can be computationally consumed. This returns to my first point: our projects can now talk to each other through computational agents, but we must agree on the vocabulary governing that conversation. Alignment with Pelagios will help in that regard, but I think more attention needs to be paid at all levels of DH projects to metadata standards. For DH projects in the ancient world, the ontology for Linked Ancient World Data offered by LAWD (https://github.com/lawdi/LAWD) should be a staring point.

Much like the slow, often tedious process of generating metadata, creating documentation for DH projects is often overlooked. From comments in code to capturing the design decisions and the entire creative process, DH documentation needs to go beyond the narrative of the research question and capture the entire creative, intellectual, and industrial process of a DH project. The suggestion to look to the hard sciences foe guidance in this process is a fruitful place to start.

5) The use of open-source repositories and the continued importance of institutional support. Most of the projects at BAM had a presence at GitHub, and there was some very interesting discussion around the practicality and usefulness of a non-profit, academically oriented alternative. This debate had as a background the reality that GitHub and other free services are currently a critical component of our work, as many DH projects operate on a shoe-string budget and are dependent on largess from an institution or grants. Such funding is often uncertain; Pleaides, one of the most exemplary projects at BAM, has a 50% success rate at securing NEH funding. For smaller projects this rate may be even lower; some participants indicated that the reward to work ratio of grant applications is not attractive for smaller projects.

There is some good news though, as many institutions have expressed growing interest in the digital humanities as a field. As a digital humanities community we need to build on this interest with a push for institutional backing. The University of Iowa clearly demonstrated the excellent outcomes of a group that is both dedicated to digital humanities and able to provide hosting, archiving, and other technical support.

6) The continued need for face-to-face gatherings. While we have many electronic forums for communication (Twitter, Slack, IRC, site forums, etc), there is still something special that happens when DH scholars are brought together for several days, freed of other distractions, and think about the same issues as a group. For me, the headspace of a conference is entirely different than using Skype in my office; my other projects and papers are out of sight, (largely) out of mind, and my focus is squarely on the discussion.

7) Release the tweets. One place where documentation is somewhat overlooked is at conferences like BAM. Many conferences generate an end-product like proceedings, which while valuable, can not capture the conversation that surrounds each presentation. The incredible use of twitter by BAM attendees, and the use of storify to capture those tweets, can serve as a model for other conference proceedings. Conference organizers should establish an “official” twitter tag, advertise it widely on social media, and ensure that a conference venue offers free wifi-access to the attendees. This expands out the reach of the conference in real-time to attendees and remote presenters who would otherwise be unable to participate in the conversation. A critical component to this is also archiving – for BAM, the use of storify (part 1, part 2) and the support of Iowa libraries ensures that there is a searchable account that can be referenced of the conference and the wider conversation it sparked.

The BAM conference generated a lot of intriguing conversation and displayed a host of excellent projects. If this kind of interest, scholarship, and congeniality can be maintained, the future of DH is bright indeed.

2 of N: Gephi, D3.js, and maps: Success!

gephileafletd3js
A working, geographically accurate map using Gephi, D3.js, and Leaflet. NOTE: Link subject to change.

In my previous post I outlined how I used D3.js to display a “raw” JSON output from Gephi. After some hacking around, I am now able to display my Gephi data on an interactive leaflet map!

This is a departure from other work on the subject for a few reasons:

  1. Not all of my data has geographic information – indeed in many cases a specific longitude / latitude combination is inappropriate and would lend a false sense of permanence to anyone looking at the map. In my case I have names of Greek garrison commanders which have some relation to a place, but it is unclear in some instances if they are actually at a specific place, have dominion over the location, or are mentioned in an inscription for some other reason. Therefore, I need to locate data that has a fuzzy relation to a location (ancient people who may originate, reside, work, and be mentioned in different and / or unknown locations) and locations that may themselves have fuzzy or unknown geography. This is a problem for just about every ancient to pre-modern project, as we do not have a wealth of location information, or even a clear idea of where some people are at any particular moment.
  2. I want to show how social networks form around specific geographic points which are known, and have those social networks remain “reactive” on zooms, changing map states, etc. This can be expanded to encompass epistolary networks, knowledge maps, etc – basically anything that links people together who may not be locatable themselves.
  3. Gephi does not output in GeoJSON, and the remaining export options that are geographically oriented require that *all* nodes have geographic information. As this is not my case (see above), the standard export options will not work for me. Also, as part of my work on BAM, I want to create a framework that is as “plug and play” as possible, so that we can simply take Gephi files and drop them into the system to make new modules. Therefore this work has to be reproducible with a minimum of tweaking.

So, let us get to the code!

First things first: You need to make your html, bring in your javascript,and style some elements. I put the css in the file for testing – it will be split off later.


<!DOCTYPE html>

<head>
<meta name='viewport' content='width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no' />
<!-- Mapbox includes below -->
<script src='https://api.mapbox.com/mapbox.js/v2.2.2/mapbox.js'></script>
<link href='https://api.mapbox.com/mapbox.js/v2.2.2/mapbox.css' rel='stylesheet' />
<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.3/jquery.min.js"></script>
<script src="http://d3js.org/d3.v3.js"></script>
</head>
<meta charset="utf-8">
<!-- Will split off css when done with testing -->

<style>
.node circle {
stroke: grey;
stroke-width: 10px;
}

.link {
stroke: black;
stroke-width: 1px;
opacity: .2;
}

.label {
font-family: Arial;
font-size: 12px;
}

#map {
height: 98vh;
}

#attributepane {
display: block;
display: none;
position: absolute;
height: auto;
bottom: 20%;
top: 20%;
right: 0;
width: 240px;
background-color: #fff;
margin: 0;
background-color: rgba(255, 255, 255, 0.8);
border-left: 1px solid #ccc;
padding: 18px 18px 18px 18px;
z-index: 8998;
overflow: scroll;
}
</style>

<body>

<div id='attributepane'></div>

<div id='map'>
</div>

Next, make a map.

<script>
var map = L.mapbox.map('map', 'yourmap', {
accessToken: 'yourtoken'
});

//set the initial view. This is pretty standard for most of the ancient med. projects
map.setView([40.58058, 36.29883], 4);

Pretty basic so far. Next we follow some of the examples that are already in the wild to initiate D3 goodness:


var force = d3.layout.force()
.charge(-120)
.linkDistance(30);

/* Initialize the SVG layer */
map._initPathRoot();

/* We simply pick up the SVG from the map object */
var svg = d3.select("#map").select("svg"),
g = svg.append("g");

Next, we bring in our json file from Gephi. Again, this is pretty standard:


d3.json("graph.json", function(error, json) {

if (error) throw error;

Now we get into the actual modifications to make the json, D3, and leaflet all talk to each other. The first thing to do is to modify the colors (from http://stackoverflow.com/questions/13070054/convert-rgb-strings-to-hex-in-javascript) so that D3 displays what we have in Gephi:


//fix up the data so it is what we want for d3
json.nodes.forEach(function(d) {
//convert the rgb colors to hex for d3
var a = d.color.split("(")[1].split(")")[0];
a = a.split(",");

var b = a.map(function(x) { //For each array element
x = parseInt(x).toString(16); //Convert to a base16 string
return (x.length == 1) ? "0" + x : x; //Add zero if we get only one character
})
b = "#" + b.join("");
d.color = b;

Next, we need to put in “dummy” coordinates for locations that do not have geography. This is messy and could probably be removed with some more efficient coding later. For the nodes that do have geography, the map.latLngToLayerPoint will translate the values into map units, which places them where they need to go. These are simply lat lon attributes in the Gephi file. I also set nodes that are fixed / not fixed, based on the presence of lat/lon data.


if (!("lng" in d.attributes) == true) {
//if there is no geography, then allow the node to float around
d.LatLng = new L.LatLng(0, 0);
d.fixed = false;
} else //there is geography, so place the node where it goes
{
d.LatLng = new L.LatLng(d.attributes.lat, d.attributes.lng);
d.fixed = true;
d.x = map.latLngToLayerPoint(d.LatLng).x;
d.y = map.latLngToLayerPoint(d.LatLng).y;
}
})

Now to setup the links. As we are keyed on attributes and not an index value, we need to follow this fix:


var edges = [];
json.edges.forEach(function(e) {
var sourceNode = json.nodes.filter(function(n) {
return n.id === e.source;
})[0],
targetNode = json.nodes.filter(function(n) {
return n.id === e.target;
})[0];

edges.push({
source: sourceNode,
target: targetNode,
value: e.Value
});
});

var link = svg.selectAll(".link")
.data(edges)
.enter().append("line")
.attr("class", "link");

Now to setup the nodes. I wanted to do a popup on a mouseclick event, but for some reason this is not firing (mousedown and mouseover do work, however). The following code builds the nodes, with radii, fill, and other information pulled from the JSON file. It also toggles a div that is populated with attribute information from the JSON. There is still some work to do at this part: the .css needs to be cleaned up, images need to be resized, and the attribute information for the nodes should be a configurable option when importing the JSON.


var node = svg.selectAll(".node")
.data(json.nodes)
.enter().append("circle")
//display nodes and information when a node is clicked on
//for some reason the click event is not registering, but mousedown and mouseover are.
.on("mouseover", function(d) {

//put in blank values if there are no attributes
var titleForBox, imageForBox, descriptionForBox = '';
titleForBox = '
<h1>' + d.label + '</h1>

';

if (typeof d.attributes.Description != "undefined") {
descriptionForBox = d.attributes.Description;
} else {
descriptionForBox = '';
}

if (typeof d.attributes.image != "undefined") {
imageForBox = '<img src="' + d.attributes.image + '" align="left">';
} else {
imageForBox = '';
}

var htmlForBox = imageForBox + ' ' + titleForBox + descriptionForBox;
document.getElementById("attributepane").innerHTML = htmlForBox;
toggle_visibility('attributepane');
})
.style("stroke", "black")
.style("opacity", .6)
.attr("r", function(d) {
return d.size * 2;
})
.style("fill", function(d) {
return d.color;
})
.call(force.drag);

Now for the transformations when the map state changes. The idea is to keep the fixed nodes in the correct place, but to redraw the “floating” nodes when the map is zoomed in and out. The nodes that need to be transformed are dealt with first, then the links are rebuilt with the new (or fixed) x / y data.


//for when the map changes viewpoint
map.on("viewreset", update);
update();

function update() {

node.attr("transform",
function(d) {
if (d.fixed == true) {
d.x = map.latLngToLayerPoint(d.LatLng).x;
d.y = map.latLngToLayerPoint(d.LatLng).y;
return "translate(" +
map.latLngToLayerPoint(d.LatLng).x + "," +
map.latLngToLayerPoint(d.LatLng).y + ")";
}
}
);

link.attr("x1", function(d) {
return d.source.x;
})
.attr("y1", function(d) {
return d.source.y;
})
.attr("x2", function(d) {
return d.target.x;
})
.attr("y2", function(d) {
return d.target.y;
});

node.attr("cx", function(d) {
if (d.fixed == false) {
return d.x;
}
})
.attr("cy", function(d) {
if (d.fixed == false) {
return d.y;
}
})

//this kickstarts the simulation, so the nodes will realign to a zoomed state
force.start();
}

Next, time to start the simulation for the first time and close out the d3 json block:


force
.links(edges)
.nodes(json.nodes)
.start();
force.on("tick", update);

}); //end

Finally, time to put a function in to toggle the visibility of the div (from here) and close out our file:


function toggle_visibility(id) {
var e = document.getElementById(id);
if (e.style.display == 'block')
e.style.display = 'none';
else
e.style.display = 'block';
}
</script>
</body>

There you have it- a nice, interactive map with a mix of geographic information and social networks. While I am pleased with the result, there are still some things to fix / address:

  1. The click even not working. This is a real puzzler.
  2. Tweaking the distances of the simulation – I do not want nodes to be placed half a world away from their connections. This may have to be map zoom level dependent.
  3. Style the links according to Gephi and provide popups where applicable. This should be easy enough to do, but simply hasn’t been done in this code.
  4. Tweak the visibility of the connections and nodes. While retaining an option to show the entire network at once, my idea is to have a map that starts out with JUST the locations, and then makes the nodes that are connected to that location visible when you click on it (which would also apply to the unlocated nodes – i.e. you see what they are connected to when you click on them).
  5. Connected to the above point, the implementation of a slider to show nodes in a particular timeframe. As my data spans a period from the 600s BCE to the 200s CE, this would provide a better snapshot of a particular network at a particular time.
  6. Implement a URI based system – you will be able to go to address/someEntityName and that entity will be selected with its information pane and connected neighbors displayed. This will result in an RDF file that will be sent to the Pelagios Project.
  7. Fix up the .css for the information pane.

I will detail further steps in a later post.

1 of N: Gephi, D3.js, and maps

Update (11/12/15): See this post to integrate the following code with leaflet.

After finding no real way to use background maps with SigmaJs, I stumbled on this example of combining leaflet with D3.jshttp://bost.ocks.org/mike/leaflet/. The example is more closely aligned with what I want to achieve, which is using a display library to show a social network that respects / interacts with underlying geography. This would be a very valuable visualization for both TBib/BAM and my own work on garrisons, and completing it will allow me to get back to other tasks, like pounding out Greek inscriptions.

For this work I am not tied to Gephi, but I do like its interface and low learning curve, which is valuable for pedagogical and collaborative use. So, my first order of business is getting a Gephi project to talk nicely with D3.js. There is, of course, a nice example already in the wild: http://bl.ocks.org/susielu/9526340. However, this presented some serious problems, which I will outline to (hopefully!) help others who may be going down this path. So, refer back to http://bl.ocks.org/susielu/9526340 for the code template – what follows below are additions / modifications.

geo-attemprFor this project, I want to recreate the image to the right, which was created in Gephi. If you read my previous post on this topic, this image uses a geo-layout plugin to place locations from Pleiades in their correct geographic placement, then uses other layouts to place the people and other non locatable nodes. The eventual goal is to make an interactive network map above an interactive geographic map, so simply exporting these out as a flat svg file will not provide the functionality I need.

My first attempt to simply plug in my own data met with disaster. First, I got hit with an “Uncaught TypeError: Cannot read property ‘weight’ of undefined” error and absolutely no graph. Looking into it, I noticed that the example assumed that nodes would be referenced by their position in an index, NOT by their own id.


 var links = json.edges.map(function(d){
 return {
 'source': parseInt(d.source),
 'target': parseInt(d.target)
 }
 })

My linkages use a unique ID text attribute, which plays havoc with this function. However, this seems like a simple fix: simply remove the parseInt() function, and the actual linkages should work.


var links = json.edges.map(function(d){
 return {
 'source': d.source,
 'target': d.target
 }
 })

netminusnetGetting closer: I see a network graph….only minus the network. Yikes. So, what is going wrong?

It seems that linking nodes by attribute instead of index is a somewhat common problem in D3.js, with a good solution here: http://stackoverflow.com/questions/23986466/d3-force-layout-linking-nodes-by-name-instead-of-index. Following this example, I modified my code by adding the following:


var edges = [];
links.forEach(function(e) {
// Get the source and target nodes
var sourceNode = nodes.filter(function(n) { return n.id === e.source; })[0],
targetNode = nodes.filter(function(n) { return n.id === e.target; })[0];

// Add the edge to the array
edges.push({source: sourceNode, target: targetNode});
});

...

var force = d3.layout.force()
.nodes(nodes)
.links(edges)

...

var link = svg.selectAll(".link")
.data(edges)

workingFinally, the links show! The nodes, however, are of a uniform size. I want the nodes to reflect their size in Gephi. Luckily this was an easy fix: adding


.attr("r", function(d) { return d.size * 3; })

to

 node.append("svg:circle") 

did the trick. I also wanted to add colors from Gephi – the following code does so (with a conversion from RGB to hex provided by http://stackoverflow.com/questions/13070054/convert-rgb-strings-to-hex-in-javascript) :


var a = d.color.split("(")[1].split(")")[0];
a = a.split(",");

var b = a.map(function(x){ //For each array element
 x = parseInt(x).toString(16); //Convert to a base16 string
 return (x.length==1) ? "0"+x : x; //Add zero if we get only one character
})

b = "#"+b.join("");

 
 return {
 'id' : d.id,
 'x' : d.x,
 'y' : d.y,
 'fixed': true,
 'label' : d.label,
 'size' : d.size,
 'color' : b,
 }
 
 })

and


.style("fill", function (d) { return d.color; })

added to


node.append("svg:circle")

onemoreproblemThis produces a graph that looks correct except for one MAJOR problem: It seems the Y axis is inverted from the original! This is obviously not acceptable if I am trying to capture actual coordinates for a map. All is not lost: I do remember this being a problem in the SigmaJS exporter. A fix is provided here: https://github.com/oxfordinternetinstitute/gephi-plugins/issues/5#issuecomment-22291683. For me, this was as simple as adding the following code:


finalY = -d.y;
return {
'id' : d.id,
'x' : d.x,
'y' : finalY,
'fixed': true,
'label' : d.label,
'size' : d.size,
'color' : b,
}

})

to the

  var nodes = json.nodes.map(function(d)

block.

inorderThe next task will be to finalize some functionality for the D3.js portion of the graph, then on to integrating the whole mess with leaflet. Then, when I have all of this in order, time to re-write it to accept all manner of different inputs / etc for BAM. More on both of these ideas later.

Quick and Dirty Footnotes For Gephi / SigmaJS

Before I begin, I once again want to recognize the excellent SigmaJS Exporter plugin for Gephi. This really does mitigate a lot of the grunt work involved in quickly making a usable, interactive social network graph. However, sometimes you just want another feature or some further refinement – in my case adding workable footnotes to information on each node.

For those of us in the humanities, citations are sine qua non for scholarship. However, there are few good was to maintain linkable citations on the web that are not hardcoded beforehand, or reliant on javascript trickery. What I wanted to do was find a tool or a method to easily move text and citations contained in my dissertation to a description field in a Gephi-based application without manually entering footnotes, footnote numbers, or linking them myself, as I have over 2,000 footnotes to deal with.

What I found is a bit of a hack, and certainly can be improved, but it works. First, you are going to want to have your document in a format that is readable by OpenOffice / LibreOffice / etc. What you need to do is select the bit of text you are interested in, dump it into a new file (making sure to include your footnotes!) and then export that file as XHTML.

export-textOnce this is complete, you will have a lovely, fully encapsulated xml file of your text – including all formatting, footnotes, etc. However, we want to eliminate some of the elements produced by this process. Open this file in your favorite text editor. You will notice that you have code similar to the following at the top of the document:


<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1 plus MathML 2.0//EN" "http://www.w3.org/Math/DTD/mathml2/xhtml-math11-f.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<!--This file was converted to xhtml by LibreOffice - see http://cgit.freedesktop.org/libreoffice/core/tree/filter/source/xslt for the code.-->
<head profile="http://dublincore.org/documents/dcmi-terms/">
<meta http-equiv="Content-Type" content="application/xhtml+xml; charset=utf-8"/>
<title xml:lang="en-US">- no title specified</title>
<meta name="DCTERMS.title" content="" xml:lang="en-US"/>
<meta name="DCTERMS.language" content="en-US" scheme="DCTERMS.RFC4646"/>
<meta name="DCTERMS.source" content="http://xml.openoffice.org/odf2xhtml"/>
<meta name="DCTERMS.issued" content="2015-09-26T21:09:50.283345000" scheme="DCTERMS.W3CDTF"/>
<meta name="DCTERMS.modified" content="2015-09-26T21:23:01.904161000" scheme="DCTERMS.W3CDTF"/>
<meta name="DCTERMS.provenance" content="" xml:lang="en-US"/>
<meta name="DCTERMS.subject" content="," xml:lang="en-US"/>
<link rel="schema.DC" href="http://purl.org/dc/elements/1.1/" hreflang="en"/>
<link rel="schema.DCTERMS" href="http://purl.org/dc/terms/" hreflang="en"/>
<link rel="schema.DCTYPE" href="http://purl.org/dc/dcmitype/" hreflang="en"/>
<link rel="schema.DCAM" href="http://purl.org/dc/dcam/" hreflang="en"/>

This can all be eliminated. Make sure you retain the




<style type="text/css">

tag at the end of the line.

Likewise, delete this from the start of the last line:

</head><body dir="ltr" style="max-width:8.5in;margin-top:0.7874in; margin-bottom:0.7874in; margin-left:0.7874in; margin-right:0.7874in; writing-mode:lr-tb; ">

and this from the end of the last line:

</body></html>

Now, simply paste what is left into a field in your Gephi data.

Gephi with footnotes

Export as usual, and viola! you have your clickable, interactive footnotes.

footnote-gephi

Now, this is good for a quick and dirty solution, but it would require a modification of the json datafile if you ever make a change or wish to add more information (or, even worse, a re-export of your entire network). As such, this solution will not be used for BAM, as we are seeking a more flexible and modifiable code base.

Networks, Geography, and Gephi: Lots of Promise, but Lots of Work to be Done

This post will outline some of my efforts to bring social networks into dialog with geography. Although I have found some interesting plugins and hacks, the results still leave something to be desired.


Screen Shot 2015-10-07 at 1.40.46 PMTo provide some background: From my dissertation I have a nice, interactive map of all garrisons (phrourai, in orange), and garrison commanders (phrourarchoi, in white) from all of Greek sources up to the mid second century C.E. This is all nicely georeferenced, linked to other projects such as Pleiades and Pelagios, and serves its purpose pretty well. However, this provides the location and frequency of garrisons and commanders, and does not really show the social network that developed between commanders, monarchs, and communities. I could perhaps use a clustering strategy to create dynamic markers around specific points, but that seems to be a very unwieldy solution.

Strictly speaking, by modeling people (phrourarchoi, monarchs) with places and abstract communities I am moving beyond a social network and instead looking at an information network, as I am interested in a number of different connections (social, geographic, ideological) that are not traditionally associated with social network analysis.

The first step to get all of my data into Gephi, assign different “types” to my nodes (in my case people, offices, places, phrourarchoi). I then created a network map, ran statistics, assigned the node size based on degree, and ran a force atlas layout. At the same time I also color coded the network based on type. This is all pretty basic Gephi use so far, and produced a perfectly serviceable network graph.

degree
First graph. Pretty basic and serviceable.

Now it was time to experiment with different types of ranking. Betweenness centrality, or the measure of a node’s influence, led to an interesting difference in graphs:

Untitled
Betweenness Graph. Note the increased importance of individuals.

However, this result is somewhat meaningless, as my graph covers a period from the 400s BCE to the 100s CE. Despite any of his wishes to the contrary, Ptolemy VIII did not live forever, yet he is the unquestioned central authority of this graph. All of the other Egyptian monarchs also score highly, underlining their importance in the communications and relationships between different phrourarchoi. This is an interesting yet hardly unsurprising finding – a good portion of the surviving data on phroruarchoi originates from Ptolemaic Egypt, which may inflate the relative importance of the dynasty in this kind of analysis. What this map does show is the enormous influence of individuals – most of whom were not phrourarchoi themselves.

However, I am interested in garrisons as a sustained phenomena across several centuries, so I want to get back to the importance of location and geography on garrisons. In other words: Where are the most important locations for phrourarchoi, and how do those relate to one another?

Running an Eigenvector Centrality measurement produces a graph that somewhat mimics my original map, with physical locations, not people as the most significant authorities. This gives a better impression of what I am looking for – the centrality of a node relative to the whole network, which in my case privileges locations, which often serve as a bridge between different populations of nodes.

egienvector
Eigenvector Centrality

To me this is an interesting graph: It shows the importance of locations, while still highlighting important individuals. Now that I have this graph, I would love to place it on a map. I actually have coordinates for all of the locations, so a simple use of the Gephi GeoLayout plugin puts all of my identified places in a rough geographic layout.

Screen Shot 2015-10-07 at 12.48.40 PM

From here I simply fixed the location of the places, then ran some other layouts to try and make a coherent graph of people and offices that did not have a specific geographic value.The results were generally less than satisfying. The individuals in my dataset are not assigned coordinates because it would make little sense to do so – some phroruarchoi served in multiple locations, and almost all imperial phrourarchoi served outside their place of origin, were buried somewhere else, possibly lived in yet another location, etc.

geo
Force Atlas combined with GeoLayout
circle
Force atlas and Fruchterman-Reingold
geo-attempr
Adjusting the size of the nodes and running force atlas eventually produced  a result that looks more comprehensible, if a bit small.

From this step, I thought I would try out some Gephi plugins to push my data into a format I could drop onto a map. Only a very small percentage of my nodes actually contain geographic information, so the ExportToEarth plugin was not going to help. My first attempt at pushing out a shapefile using Export to SHP initially looked like a success in QGIS:

Screen Shot 2015-10-07 at 1.43.16 PM
This looks promising…

So, I decided to throw in some background, and that is when the trouble started. QGIS does a good job of transforming coordinates, but this was just messy (and not to mention wrong – there certainly were no phrourarchoi in Antarctica!)

Screen Shot 2015-10-07 at 1.35.59 PM
Note how the nodes are now literally all over the map.

So, what happened? If you do not have coordinates already explicitly assigned to your data, Export to SHP actually does not use “geographic” coordinates, and instead uses, in the words of the plugin, “fake geography – that is the current position of the nodes in the Gephi layout”. My thought that this position would line up with correct coordinates fromGeoLayout were false –Export to SHP treats the middle of the map as an origin point (instead of using whatever geographic data is present), and as such it does not match with any projection in QGIS.

This is a bit of a let down. It seems that all of mapping plugins in Gephi need for *ALL* of the nodes to have geographic information already baked in, or they will not export a geographically accurate map. This does make some sense, but it would be nice if you could use GeoLayout to place nodes with actual geographic data, then use force atlas or some other layout to produce a graph, and finally use the location of those nodes as coordinates. In other words, the location of nodes on the graph that have no actual geographic data of their own are located relative to nodes that do have geographic data. I tried the Sigmajs exporter, but the json object also does not use real coordinates, as seen in the fragment below ( lng and lat are the real-world coordinates, while x and are used by SigmaJS):


"label":"Priene/‘Lince’?",
"x":-22.65546226501465,
"y":32.66741943359375,
"id":"Pl_599905",
"attributes":{
...
"lng":"27.297566084106442",
"lat":"37.659724652604169",
...},

So, is there a way around this?

Short of writing a new plugin to do so, it looks like Gephi is simply missing the functionality of assigning geographic points to nodes that do not already have that information, then exporting that graph in a way that makes sense to mapping software. I could export an image and georeference that, but that will not provide the functionality I am looking for either.

What I would like is for a graph produced by Gephi to use coordinates for nodes that have them, and make real world coordinates for nodes that do not. This map could then be placed on Leaflet / OpenLayers / whatever map, providing a level of interaction beyond a static image. As it is impracticable to duplicate the functionality (especially the statistical tools and layouts) of Gephi in a mapping application, this strikes me as something that would be very valuable to visualization and study.

My next idea is to see if R has something close to what I want, which I will detail in a future post.

Code for BAM: Part 1 of N. Gephi and Maps

This is the first in a series of posts where I will be detailing some of the code and development of BAM. Some of these techniques may be old hat for some users or simple hacks, but they might be useful for anyone else who is trying to do similar work.

TBib-select
Terra Biblica with both the social network graph and map displaying information on Jesus.

In this post, I will detail how I got Gephi data (produced by the SigmaJs Exporter) to communicate with an OpenLayers 2 map. When a user clicks on any entity in the network graph the map panel will adjust to show the locations and frequency of that entity in geographic space. At the same time, any clicks on an entity name on the map (provided by a popup) will adjust the social network graph to highlight that entity. This code is built on javascript, PHP, and a PoistGIS backend. At some point in the future BAM may transition to OpenLayers 3, but for now we are sticking with 2 as it formed the basis for À-la-Carte, Digital Strabo, and other digital efforts that BAM builds upon and extends.

For a working demonstration of the final result, see http://awmc.unc.edu/awmc/applications/bam/luke/. All of the code mentioned in this post, and created for BAM, is available at: https://github.com/Big-Ancient-Mediterranean/BAM.

Step 1: Get your data in order!

Before attempting any of this, you need to ensure that the entities that you are using in Gephi and the ones you have in your database have a consistent, unique ID. So, if Andrew has an id of 1234567 in Gephi, you need to associate 1234567 with different locations, texts, etc in your database that are also related to Andrew. Failure to do so will make it VERY difficult, if not impossible, to get all of the different components to talk to each other.

Next, you actually need to build your network in Gephi and export it out. Building the network itself is beyond the scope of this post, but you need to install and familiarize yourself with the excellent SigmaJs Exporter created by Scott Hale at the Oxford Internet Institute. Essentially what we are doing is taking the output of the SigmaJs Exporter, cutting it down, and making it communicate with a dynamic, interactive map on the same webpage.

directoryAfter exporting your network using the SigmaJs Exporter, you should have a directory structure that roughly looks like the screenshot to the right. You want to upload everything but htaccess_exampleweb.config, and index.html to your webserver.

We then need to add this network to an HTML file that already has a map. In our case, we are modifying the code behind Strabo Online and SNAGG. I may detail how to create a map in another post, but there are plenty of resources online to get you going on a basic map.

We are going to mimic the functionality of the index.html file that we excluded in our own html file. First, we need to include the various javascript files and libraries used by the application:


<script src="js/jquery/jquery.min.js" type="text/javascript"></script>
<script src="js/sigma/sigma.min.js" type="text/javascript" language="javascript"></script>
<script src="js/sigma/sigma.parseJson.js" type="text/javascript" language="javascript"></script>
<script src="js/fancybox/jquery.fancybox.pack.js" type="text/javascript" language="javascript"></script>
<script src="js/main.js" type="text/javascript" language="javascript"></script>

<link rel="stylesheet" type="text/css" href="js/fancybox/jquery.fancybox.css"/>
<link rel="stylesheet" href="css/style.css" type="text/css" media="screen" />
<link rel="stylesheet" media="screen and (max-height: 770px)" href="css/tablet.css" />

Now we need to place some divs to hold the content from our social network. These can be styled at your leisure.



<div style="padding-left: 1%;padding-right: 1%;" id="socialNetContainer" class="socialNetContainer">

<div class="sigma-parent">

<div class="sigma-expand" id="sigma-canvas">

<div style="z-index:9994" id="attributepane">

<div class="text">

<div title="Close" class="left-close returntext">

<div class="c cf">
<span>Return to the full network</span>
</div>

</div>


<div class="nodeattributes">

<div class="name"></div>


<div class="data"></div>


<div class="p">Connections:</div>


<div class="link">

<ul>
</ul>

</div>

</div>

</div>

</div>

</div>

</div>

</div>


Now that we have all the functionality of the SigmaJs Exporter in our map, we need to make the components talk to each other. First, we need to identify what node is active on the sigma.js div, and use that information to select the appropriate data for our map. The function nodeActive in SigmaJs identifies what / when a node is active – so we will extend this to pass that information to a variable (for a more detailed explanation on how to extend a javascript function, see http://coreymaynard.com/blog/extending-a-javascript-function/).

We are also going to create a separate function to deal with adjusting the map itself, called tBibPersonConnections, which will be called in our new, extended function:

(function() {
//first copy the old function in the new one
 var old_nodeActive = nodeActive;

//new function with the same name as the old one - this overrides the old function
 nodeActive = function() {

//we are going to build the map from the person_id that is called from the node
// this is a separate function that will be explained below 
 tBibPersonConnections(arguments[0], tBibPeoplelayer);
 activePerson = arguments[0];

// Calls the original function\
 var result = old_nodeActive.apply(this, arguments);

// now return the result
 return result;
 }
})();

tBibPersonConnections is where the work really happens. Lets examine this function slowly.


function tBibPersonConnections(personNameChoice, tBibPeoplelayer)
{
 var dataStringForFeature ='pid=' +personNameChoice +'&amp;amp;amp;amp;amp;amp;start=0';
 tBibPeoplelayer.destroyFeatures();
 tBibfeaturesOnMap =[];

 $.ajax({
 dataType: "json",
 type:'GET',
 data:dataStringForFeature,
 url:'tbib_mapmaker.php',
 success:function(dataJson) {
 for (var i = 0; i &amp;amp;amp;amp;amp;lt; dataJson.features.length; i++){
 var untransformed_feature = geojson_format.read(dataJson, "FeatureCollection");
 //for some reason this is going into an array. Going to hardcode for now
 for (var j = 0; j &amp;amp;amp;amp;amp;lt; dataJson.features.length; j++){
 if (tBibfeaturesOnMap.indexOf(untransformed_feature[j].attributes.pid) &amp;amp;amp;amp;amp;lt; 0){
 tBibPeoplelayer.addFeatures(untransformed_feature[j]);
 tBibfeaturesOnMap.push(untransformed_feature[j].attributes.pid);
}
}
 tBibPeoplelayer.refresh({force:true}); 
 }
 },
 error: function (xhr, ajaxOptions, thrownError) {
 alert(xhr.responseText);
 }
 });

}

The function takes the ID of the person selected and layer that houses all of the feature information as arguments.

The first thing we do is create parameters for the PHP file that will return all of the place / feature information that is associated with an individual person. Do not worry about the “start” parameter for now, as it is only used when resetting the map to an initial state. The lines

tBibPeoplelayer.destroyFeatures();
tBibfeaturesOnMap =[];

first clear the map layer of all features, and then sets up an array to hold all of the new features that we will be adding to the map.

The AJAX call to tbib_mapmaker.php actually queries our database, and returns each feature that is associated with an individual, the number of times the individual is mentioned with the feature, and the geographic location of the feature. While the actual sql calls are specific to this application / database, I will show what we are doing for combining Pleiades data, BAM data, and the map:

$query = "select
pplaces.title, count(pplaces.title), max (pplaces.id) as pleaides_id,
ST_AsGeoJSON(ST_Transform(max(pplaces.the_geom), 3857)) as geom
from pplaces
JOIN
tbib_pleiades
ON
pplaces.id = tbib_pleiades.pleiades_id
JOIN
tbib_network
ON
tbib_pleiades.verse = tbib_network.reference
where
character_1 = '$pidParam' or character_2 = '$pidParam'
GROUP BY
pplaces.title";

We are interested in every occurrence of an individual, so we do not care if the person is the target or the source. Our tbib_network table is exactly the same as the table used to build our Gephi network, and all people are assigned a unique ID that remains consistent across tables.

At the end of the .php file, all of the results are returned in json format:

//make a geojson object
while($row =pg_fetch_assoc($qry_result)){
//resize for map
$sizeForMap = (($row[count] / 10) + 1);

//arrange for map
$arr[] = array(
"type" => "Feature",
"geometry" => json_decode($row[geom]),
"properties" => array(
 "title" =>$row[title],
 "count" =>$sizeForMap,
 "pid" => $row[pleaides_id]
 ),
);
}
//encode into geojson
$geojson = '{"type":"FeatureCollection","features":'.json_encode($arr).'}';
echo $geojson;
?>

In the future, this database work will be mirrored by static json files, to allow for the easy export / import of BAM material.

When the PHP file returns a json string, the function then pulls it apart, creates new OpenLayers features, and then adds them to the map:

 success:function(dataJson) {
 for (var i = 0; i < dataJson.features.length; i++){
 var untransformed_feature = geojson_format.read(dataJson, "FeatureCollection");
 for (var j = 0; j < dataJson.features.length; j++){
 if (tBibfeaturesOnMap.indexOf(untransformed_feature[j].attributes.pid) < 0){
 tBibPeoplelayer.addFeatures(untransformed_feature[j]);
 tBibfeaturesOnMap.push(untransformed_feature[j].attributes.pid);
}
}
 tBibPeoplelayer.refresh({force:true}); 
 }
 },

The result is a layer that changes depending on what person is clicked.

A user selected popup
A user selected popup

That is great for changing the map, but what about changing the nodes on the network graph for when an individual is selected on the map?

As we are displaying people names, not ID as clickable information in our popups, we need a way to translate the names to the IDs used by SigmaJs. This is simply a trivial php script that looks up an ID from a name table. Once the ID is returned, we simply activate the node with a call to the nodeActive function that we extended earlier and to our tBibPersonConnections function.

First, however, we have to listen for the event where the popup on the map is clicked:


//this is the popup listner

$('#popupSnagTable tbody').on( 'click', 'td', function () {
//now to start stripping out to what we need
var columnName = $('#popupSnagTable thead tr th').eq($(this).index()).html().trim();
if (columnName == 'Reference')

{
var ActiveRef = $(this).html().trim();
ActiveRef = ActiveRef.replace('Lk ','');
var ActiveRefSpilt = ActiveRef.split(":");
activeChapter = ActiveRefSpilt[0];
activeVerse = ActiveRefSpilt[1];
getPerseusText($(this).html().trim(), 0);
}
//if the user clicks on a name, then we use this to make an ajax call
if ((columnName == 'Entity 1') || (columnName == 'Entity 2')){
var personNameChoice = $(this).html().trim();

var dataString = 'pid='+personNameChoice;

$.ajax( { type:'GET', data:dataString, url:'bamIdFromNum.php', success:function(data2)

{

//from the sigma.js gephi instance

nodeActive(data2);

//now to add all of the places the entity is on the map. Searching by ID

tBibPersonConnections(data2, tBibPeoplelayer);

}

});

That is all there is to it – just a few listeners and a variable or two. There may be more efficient ways of doing this, but all the components are talking to each other!