Annotating RTI data in 3d and 2d

I’ve been talking to a lot people in recent months about annotation frameworks for RTI and today’s introduction to the #rodeimagingevent (see Hembo’s blog post) has crystalised some of these. I was talking to @kathrynpiquette about annotation and I also tweeted a query to @iipimage about it. @portableant suggested annotorious (something that I know our current MSc student Vassilis Valergas has been examining) and also openCanvas was suggested. Also @aboutgeo mentioned the freeform extension to annotorious developed by @portableant. We started talking about the idea of this being a geospatial problem rather than an image annotation problem and whether frameworks in online GIS were more advanced.

I think at the core of my interest is the potential for novel discoveries to emerge as a consequence of shared annotation in image space. So, as we envisage in our @ahrcrti project multiple annotations would be created online allowing agreement (and disagreement as proposed previously by @LenaKotoula !). We have been discussing annotation in general elsewhere (and as usual very closely with CHI), and the depositing of these annotations within our institutional repository and the Archaeology Data Service, so I will link that in to this post in the future. Here I will concentrate on some thoughts I’ve had around sketching and 3d annotations.

Back in 2010 we published an example of spatially referenced RTI data where we used standard camera matching approaches from graphical modelling to locate an RTI releative to a scanned object. One might also use a photogrammetric approach to cross-reference two or more RTI captures against eachother, with or without the intermediate step of generating three-dimensional geometry. In such an environment one might also interrogate RTI, photogrammetry, laser scan and every other form of data. I have previously proposed an interface like Photosynth for handling this form of interaction.

Still, whatever the end magical framework, I think the key factors are the affordances of the annotation environment. I envisage the following scenario:

Capture

1. Capture RTI
2. Capture a second RTI from a different position, which may simply be an adjustment of focal length
3. Encode the information to derive a relative co-ordinate system in a standardised way within the RTI metadata – this would employ the EXIF metadata for recording the lens properties and IPTC for location and vector orientation; with these metadata one could derive the relative positions
4. Transfer this metadata to all derivatives e.g. cropped RTI and snapshots, in order that these can be tied together.

Annotation

1. Load the RTI into a web or standalone viewer of some kind
2. Generate 2d annotations as usual, but crucially provide keying information to enable viewer settings to link to annotations. For example, you set the specular filter properties and then sketch on some points of interest, perhaps adding point annotations too. You then adjust the filter properties and add some more annotations. These could then be played back and bookmarked.

Integration

1. Calculate a relative co-ordinate system linking these two datasets – note: the accuracy of this is not relevant here, only the ability to quantify (or estimate) the accuracy
2. Optionally also bring in true 3d data
3. Provide a visualisation showing cones in space representing the FOV of each RTI
4. Load an RTI viewer canvas into this plane
5. Visualise extant 2d annotations on this plane
6. Allow annotation in the 3d environment. This would allow a sketch to begin with the vector point snapping to a point on one RTI plane and then jump to another RTI plane. In this case the metadata generated would need to capture the RTI viewer settings on all RTIs involved in this annotation process. So, with every new point you could capture the viewer settings. Interestingly one might choose to capture the viewer settings for the collection of RTIs as a whole that were visible at the point at which the annotation was created since these all have an impact on the annotations created.

Hmmm. Now time to think about what frameworks already exist. We are still working on our web RTI viewer which is structured around @iipimage but there is the great webGL RTI viewer already by Gianpaolo Palma. Anyway, soup has arrived at the #rodeimagingevent here in St Nicholas’ Church in Tallinn and the hacking continues. Not a bad way to spend a Saturday.