Abstract
1. For many animals, vision represents the primary sensory modality governing a vast diversity of behaviours. Creating 'virtual twins', or visually realistic digital reconstructions of real individuals interacting within their environments, offers a promising approach for understanding how visual information guides an animal's decisions, behaviours, and interactions. Such systems enable the creation of video scenes from any view, including the viewpoint of the animals themselves. This allows receiver information to then be analysed from the perspective of species-specific visual systems, or for the accurate reconstruction of a real animal's movement to provide social stimuli for behavioural experiments.
2. Despite advances in 3D modelling, photorealistic scene rendering, and animal tracking, efficiently creating realistic digital reconstructions of movement remains challenging for many species. In fishes, a continuously-deforming body due to the absence of discrete joints make it particularly difficult to map tracked body keypoints onto articulated digital models, often resulting in unnatural or erratic animations.
3. Here, we introduce DigiFish, an add-on for the open-source animating software Blender, that generates realistic fish animations from tracking data. By integrating anatomical keypoint tracking with spline-based body curvatures, DigiFish reconstructs biologically-realistic swimming postures and movements of fishes within customisable virtual aquatic environments. The system also supports virtual 'point-of-view' cameras that can be placed arbitrarily within a scene, or dynamically driven by eye-tracking data, enabling unlimited perspectives on how individuals, conspecifics, or landmarks appear against complex backgrounds and how these views change with realistic swimming and eye movements.
4. We provide the Blender add-on and user guide and use DigiFish to reconstruct two different contexts: staged predator-prey interactions in the laboratory and natural schooling of wild fish in the field. Using a further reconstruction of fish swimming in an arena, we also use an Elementary Motion Detector model to show that our virtual reconstructions successfully capture perceptually-realistic movements and postures of individual fish within a digitally reconstructed scene.
5. By integrating photo-realistic scenes with biologically-realistic, simulated motion, DigiFish offers a flexible and accessible platform for investigating visually guided behaviour, perception, and cognition in complex and dynamic environments.