What key theoretical insights about multimodality are required to effectively assess multimodal texts?
When considering the assessment of multimodal texts one needs to come back to the basics of literacy and considering the purpose, form and audience. Then layered on top of this is the multimodal aspects that have been chosen to add depth of understanding to the piece of work, the semiotics:
- Linguistic: comprising aspects such as vocabulary, generic structure and the grammar of oral and written language
- Visual: comprising aspects such as colour, vectors and viewpoint in still and moving images
- Audio: comprising aspects such as volume, pitch and rhythm of music and sound effects
- Gestural: comprising aspects such as movement, speed and stillness in facial expression and body language
- Spatial: comprising aspects such as proximity, direction, position of layout and organisation of objects in space.
Finally, it is the overall meaning that is gained and whether or not each semiotic component supports and adds to the purpose and meaning. As has been pointed out by Thomas (2008, 8) in relation to inter-semiotic meanings “Modal complexity occurs when the same narrative meaning is replicated across all modes simultaneously.” Kress and van Leeuwen (2006) also express the importance of the importance of meaning being supported by all aspects of the visual content. Linking it to the similarities that exist in music, it is the composition of a piece of work that is vital to the message delivered. Poor composition, that is semiotic components that are disconnected, provide a meaning that appears confused or lacks the desired power and intensity.
In summary, when assessing a piece of work that is multimodal one must unpack all the layers (semiotic components) and determine whether or not they are supporting the meaning of the work. This requires more than a quick look, it requires a well created measuring tool, and in some cases this might be different for each piece of work assessed. Also a sound understanding of the interplay between semiotics to create the story/meaning is required as they cannot be assessed in isolation.
Kress, G., & van Leeuwen, T. (2006). Reading images: the grammar of visual design.New York: Routledge.
Thomas, A. (2008). Machinima: Composing 3D Multimedia Narratives. In: Unsworth, L. [Ed]. New Literacies and the English Curriculum: Multimodal Perspectives. London, Continuum.