My research and scholarship activities follow three distinct paths – the aesthetic, the technological, and the pedagogical. The three paths are intertwined, each contributing to the goals of the other.
As an artist and practitioner, I look for ways in which we can merge technology and aesthetics in order to expand our understanding and interpretation of our world. Our efforts as artists to produce imagery are naturally affected by the technology we use; in some ways we are constrained by technology and in other ways we are freed by it. I am interested in the balance of technology and art; and to the extent to which either one defines and enhances the other.
As a professor, I seek to apply this merger of technology and art not only to the content I put forth in my classes, but in the very methods I use to deliver information. I have thus been involved with concepts such as Augmented Reality, “Flipped Classroom” and other methods of leveraging technology to foster experiential learning on the part of my students, my colleagues and myself.
Technology has always played a part in the creation of art. Historically, we can look at the architecture of the Renaissance for an example. Mathematical and technological discoveries and practices enabled architects like Michelangelo and Alberti to create great cathedrals that offered people an experience of the “Holy Other” – an underscore to the vision of reality favored by the church.
Art historians and critics often debate the concept of “What Is Real?” within the context of painting and other visual media. 17h Century painters Caravaggio and de la Tour, for example, are described as “Romantic Realists”, and are often credited with the inauguration of the realist movement of the mid-20th century. The idea of realism in art underwent further re-definition in the 19th century, represented in part by the work of Jean-Baptiste Corot and Gustave Courbet. Lionello Venturi, the Italian art historian and critic, commented that “Courbet denied beauty in order to stick to reality”. Art movements in the 20th century opened up a new realm of “Existential Realism” (as exhibited in a 1968 exhibit at New York’s Museum of Modern Art titled “The Art of the Real”), wherein minimalist representational artists attempted to re-define realism in terms of the verifiable, physical properties of art objects themselves, rather than the illusionistic properties of the pieces. Abstract Expressionism again sought to redefine the term by setting aside social and symbolic content in favor of pure aesthetics and material representation.
The role of technology in image-making has always been not only a prevalent, but a key factor in our determination of what represents “reality” or “realism”. Photography has sometimes been referenced as the definitive arbiter of what is “real” or “unreal”, while at the same time exploited in the construction of an art-politics platform (“The camera doesn’t lie”) by artists who would build platforms under the banner of absolute accuracy. Cameras, however, are unreliable observers, they are machines that are subject not only to the input and sensibilities of those who wield them, but to the limitations of photographic technology as well – including concepts like exposure, Kelvin sensitivity, color reproduction, and the like. What we photograph is not necessarily what we see; which leads to the distinction between “photo-realism” and “empirical realism”. With the advent of HDR photography comes a jolt to what we believe; the new capability to represent wider dynamic range challenges our understanding of what is “real”, as represented by the photographs we have grown up trusting. The reality we see with our own eyes looks “wrong” to us, because we have been taught to accept the limitations of image-making technology as correct.
In 2012, I engaged in a funded research project with professor Susan Lakin of the School of Photographic Arts and Sciences on the creation of a new collaborative class, one which merged the art and technology of photography and computer based imagery – with the purpose of learning what visual elements create a believable reality, and to exploit those elements in order to make the unreal real. Photographic limitations of exposure, white balance, dynamic range, depth of field, low-light photography and others have been vastly expanded with the development of digital photography; the degree or realism attainable using high-end computer graphics has become sophisticated enough so as to make this kind of collaboration possible.
This research project resulted in a new course taught at RIT, “The Collaborative Composite Image”. During the first run of this class, my collaborator and I began to research new technology, neither entirely photographic nor CG. “Augmented Reality” presented itself as a new way to combine and experience captured and synthetic imagery. We began a relationship with the Memorial Art Gallery in Rochester, NY, to implement our ideas in conjunction with paintings in the permanent collection of the Gallery, resulting in what has become known as “The MAG Project”. This project has been the subject of several talks and presentations, including one on site at the Memorial Art Gallery, at the RIT “Technology Tent” at the LPGA Tournament in Rochester, at the International SIGGRAPH conference in Anaheim, California, at SIGGRAPH Asia in Hong Kong, and at the TCC Online Conference of Honolulu, Hawaii.
The class has run each year since 2012, and is on the schedule for the Fall, 2015 semester as well.
Our investigation into Augmented Reality has led to a number of ideas and investigations into how this technology can be used to achieve aesthetic ends while advancing pedagogic goals as well. In 2014, leading a group of students, I began a project using of this technology (and ancillary tools) designed to enhance engagement among high-school students in STEM education. Working with a publisher of hands-on STEM curriculum, we produced three project concepts designed to achieve these ends. Our work was shown at the RIT Innovation Festival, and has grown into a project for which we are seeking NSF funding. At present, I am working with faculty from CAST, representatives from Sponsored Research, and the publisher of the STEM curriculum on an application for an NSF SBIR/STTR grant in December, 2015. The project for which we are seeking funding has the working title, “The Augmented Textbook”, which will use Augmented Reality and other technologies to make existing textbooks more engaging for the student, and to allow for more frequent updates and content that can change with demographics and location.
In a related research effort, I submitted a proposal for a presentation at SIGGRAPH 2015 based on my observations and thoughts on realism in imagery; this proposal was focused on finding a semantic definition of realism within a context of semiotics, titled “Contrastive Focus Reduplication in Defining Contextual Realism”. The proposal was not selected for inclusion. An example of a positive review for this submission is,
A good idea for a talk on a topic of relevance to a large part of the graphics community. The talk should provoke some useful debate and insights though it may not be graphical in nature per se. Coming from a non-graphical approach would provide a good interdisciplinary insight to this discussion. The topic of talk would be of interest to many. The abstract is well written and compelling.
An example of a negative review is,
Not sure it adds anything to the discussion of photography, physical-based rendering vs. npr and evocative representations and more – all of which have been discussed in many venues. SIGGRAPH has distinguished itself by generally including technical-based information that can contribute more than speculation to this discussion. I don’t see that explicitly listed here.
The general feeling among reviewers was that while the topic was of some academic and critical value, the direction of the talk was not technical enough for the venue. I agree with this evaluation; my plan for this topic is to expand upon the concepts outlined in the abstract and seek publication and/or presentation at a venue with a somewhat broader academic scope with a less technical focus.
Advances in computer technology have led me to another area of research which may seem different, but points to the same end of defining “realism” for creators and consumers alike. Digital sculpture, otherwise known as modeling, has always required a strong mix of technical and aesthetic ability. Over the last two decades, computer modeling has been severely limited by simple processing and memory capability – that only a certain number of polygons could be represented on a screen or within a 3D environment. Sculptors had to find technical workarounds for these limitations which required the dual nature of artist/technician for success in this field. Specifically, the process of digital sculpture has historically been “additive”, beginning with a simple cube with twelve triangles; modelers would have to build up density line by line, face by face, judiciously adding complexity on an as-needed basis, with each line carefully contrived to achieve two ends – one based on the demands and limitations of the software, the other based on sculptural or aesthetic goals.
Today, this twenty-year-old approach is facing serious challenges with the advent and accessibility of sharply increased technological capability of our computer systems. Because our systems can now represent tens of millions of polygons without degradation to user-response or interactivity, we can now take a “top-down” or subtractive approach to modeling. Starting with high-density models, artists can sculpt digitally with almost no regard for technical limitations. Sculptures can be created from within the software itself or imported using consumer or pro-sumer level scanning or photogrammetry methodologies. Until very recently, even these lower-end methodologies were out of reach to students of digital sculpture because of the high cost of purchase and maintenance of these systems.
From a pedagogic point of view, this enables me to separate the aesthetic from the technical, allowing for increased creativity and spontaneity in the classroom and labs; enabling my students to produce multiple design iterations on the fly, and in a separate process apply the requisite technical modifications after the fact.
In 2011, I worked with a graduate student, Scott Riddle, on his thesis project in Medical Illustration. He, too, was interested in the high-polygon count, hyper-accurate representations becoming more accessible through scanning and photogrammetry methodologies; his research took him into the realm of translating MRI data into usable surface data for pre-flight surgical review. His paper, on which I am a co-author, is titled “Retopologizing MRI and Diffusion Tractography Datasets for Real-time Interactivity”, and was published in the Journal of Biocommunications in 2013. This research included Professor James Perkins, then of CIAS School of Art (Medical Illustration) and two neurosurgeons not affiliated with RIT.
Like the photography project above, I began to implement this research into my classes at RIT. I began with some experimental approaches in the spring of 2012. The results were striking, and led to a discussion of concept and methodology presented at the International SIGGRAPH conference in Los Angeles, California in the summer of 2012. The basis of this presentation sparked the interest of Springer Publishing, which led to the publication of “Integrating 3D Modeling, Photogrammetry and Design” in January, 2013, co-authored by my colleague Shaun Foster of the School of Design. The concepts presented in this book became the core curriculum for a class I taught at RIT in the fall of 2013 and 2014 titled “Experimental Digital Workshop: Modeling Strategies”, wherein for the first time we were able to create a new context for understanding old methods, and investigate not only how these new strategic approaches affected modeling outcomes, but learning outcomes as well.
Mr. Foster and I were the recipients of the CIAS “Gitner Family Prize for Outstanding Professional Achievement in Graphic Communication” in 2014.
The next generation of this research is something I am beginning at this writing. With the advent of 3D printing and the variety of materials and methods now available, I am interested in combining the MRI translation done with Riddle in 2011-2012 with the new modeling algorithms that have developed since then. I have begun a series of meetings with a local maxillofacial surgeon, Dr. Joseph Fantuzzo, to explore the research opportunities in creating custom MRI-based prosthetic supports for facial implants using a patient’s MRI data and 3D printing technology.
Currently, my pursuits are also involved with empirical knowledge and experience of the world, investigating the notion of “realism” and whence our knowledge originates. Within the context of the visual reality discussed above, I am collaborating with two colleagues in a project about the experience of gallery space using next-generation technology. Professor Joseph Geigel, from the B. Thomas Golisano College of Computing and Information Sciences, Professor Susan Lakin, from the School of Photographic Arts and Sciences and I are in the process of completing a grant proposal to Microsoft to create a “Virtual/Augmented Gallery” using the company’s new Hololens, which they describe as an “Untethered, see-through holographic computer”.
On a parallel course to the aforementioned research, I am also interested in ways in which technology can be used in the classroom to help separate the huge amount of quantified information my students need to absorb from the interactive, engaging classroom activity that fosters the creative implementation of that information. To this end, I have become part of the Community of Practice at RIT, researching the objectives of the Technology Rich Classroom and the broader topic of the “Flipped Classroom”.
This process has mostly been about selecting specific approaches to image-making using technology, and creating custom, captioned video content to support classroom activity. In addition to having created dozens of videos and making them accessible to students, I’ve used and disseminated my experiences at conferences. In 2012, with my colleague Shaun Foster, I delivered a presentation at the SIGGRAPH conference in Los Angeles titled, “Screencasting Strategies: Heuristics for using Video Content in 3D Computer Graphics Technological and Aesthetic Education”.
In each example of my research listed above, I have sought to focus on the three distinct paths described at the beginning of this document.
In the case of the MAG project, which was largely an aesthetic investigation driven by technology, I expanded this investigation into experimentation and research into how this technology can apply to other experiences in and out of the classroom, which led to the current project for which I am seeking NSF funding, “The Augmented Textbook”. As stated above, this research is based on the use of technology to expand and personalize education in a variety of topics. As a second branch of this research, my colleagues and I are seeking ways in which to enhance, augment and perhaps even re-define the experience of the museum-goer, to facilitate a greater fundamental understanding of the art before him.
In the case of the highly technical research pursued with Scott Riddle et al, the technical concepts translated easily into the fields of design and of pedagogy; this research led to a re-design of an existing course in human anatomical sculpture (“Modeling Strategies”) and an expanded discussion about a new approach to modeling design that was published in 2014. This technical research has also led to a second ongoing project involving maxillofacial reconstruction.
In every case, the research has implications that affect all three of my focal points, aesthetic, technical, and pedagogic; I think these crossovers are at the heart of my research interests. The over-arching principle is always bound up with the way in which we perceive the world; the information that leads to this perception and the language we use to describe it. Our discussion of the concept of the “Real” can be based on our perception, our philosophical understanding or on measurable mathematical precision; my research and scholarship seek not only the answers to these questions, but the questions to which these answers give rise.