Research

Projection mapping that works in bright places

Most of the existing projection mapping systems are used in dark places, such as outdoors at night or indoors with the lights off. One of the reasons for this is that the projected light looks better in a dark environment, but another technical problem is that it is difficult to position the projector in a bright area (prior preparation involving measurement by a camera, etc.). We thought that solving the latter problem would enable projection in bright environments and potentially expand the range of applications of projection mapping technology. In this study, we construct a projection camera system using an event camera that outputs only light changes instead of an ordinary camera. This is because we focused on the wide dynamic range of this camera and its high contrast sensitivity in the high luminance region. In addition, we proposed structured light that blinks at different frequencies for different locations, which is suitable for event cameras. The proposed projector-event-camera system can stably perform calibration and 3D shape measurement in bright environments where ordinary projector-camera systems cannot. As the second step, we are now working on high-speed shape measurement (~1000fps) for animal bodies.

Related achievements: Yuichiro Fujimoto, Taishi Sawabe, Masayuki Kanbara, Hirokazu Kato, “Structured Light of Flickering Patterns Having Different Frequencies for a Projector-Event-Camera System,” The IEEE Conference on Virtual Reality and 3D User Interfaces (IEEE VR), Mar. 2022.

 

Public Speaking Training Using Virtual Reality (VR)

As a part of our research to explore the possibility of interpersonal skill training using virtual reality (VR), we focused on presentation (public speaking) training. This training includes both internal control, such as reducing fear during a presentation, and skill improvement for successful presentation. However, these methods have two problems: (1) many people do not want to watch their own presentation videos, and (2) it is difficult to quantitatively evaluate oneself due to cognitive bias. In order to solve these problems, we propose a retrospective method that applies the viewpoint change by VR. Specifically, the system records your body movements, eye direction, and voice during the presentation, and then reconstructs your presentation in the VR space using a 3D avatar. In this process, we intentionally do not reflect elements that have little relevance to the quality of the presentation, but which tend to be strongly disliked in retrospect (e.g., information on one’s own face). In the second stage, the participants wear a head-mounted display (HMD) to observe from the audience’s point of view, so that they can objectively reflect on themselves from a third person’s point of view.

 As the second step, we are now investigating a real-time biofeedback method in VR space that includes not only movement information but also physiological indicators such as EEG and heartbeat.


Guideline and Tool for Designing an Assembly Task Support System Using Augmented Reality

Augmented reality (AR) systems support complex tasks like assembly by overlaying task-related content onto the real world. In recent years, the effort of designing and developing assembly task support systems in AR decreased with the availability of high potential head-mounted displays and provision of integrated development environments. Nevertheless, problems still arise when companies craft an effective AR task support system, particularly in the difficulty of selecting appropriate techniques and information-presentation methods, and the requirements that vary with each use case. In this study, we formulated a corresponding guideline, developed a selection aid tool that incorporates filtering based on the categorization of subtasks and the degree of freedom of available tracking. We envision our guideline and tool to be accessible as an online web page, assisting AR assembly task support system designers/developers worldwide. Our guideline is available here.

Related achievements: Keishi Tainaka, Yuichiro Fujimoto, Masayuki Kanbara, Hirokazu Kato, Atsunori Moteki, Kensuke Kuraki, Kazuki Osamura, Toshiyuki Yoshitake, and Toshiyuki Fukuoka, “Guideline and Tool for Designing an Assembly Task Support System Using Augmented Reality”, In Proceedings of IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Nov. 2020.

 

Human Detection in Office Environments

Automatic analysis of work types and communication behavior in offices is expected to improve office efficiency. The position of a person at each time is one of the most basic information for this purpose, and continuous automatic detection is desired. In this study, we propose a new method for continuous human detection based on depth data. To address these problems, we propose (1) a method to approximate the cause of missing data and actively use it, and (2) a method to identify a person by combining multiple doll-like features. Furthermore, in order to cover the entire office environment, we developed a system in which multiple Kinects are placed on the ceiling and operate cooperatively to reintegrate the information as height information from the floor. We combined these systems and applied them to several hundred hours of real office data, and obtained practical accuracy.

 

Projection-Mapping for Enhancing the Perceived Deliciousness of Food

The perceived deliciousness of a food item is highly related to its appearance. Image processing has been widely used to make food images more appealing to the public, such as when capturing and posting images on social networking sites. This paper proposes a methodology and a system to enhance the degree of subjective deliciousness perceived by a person based on the appearance of a real food item by changing its appearance in a real environment. First, an online questionnaire survey was conducted to analyze the appearance factors that make food look delicious by using various food images. Based on this knowledge, a prototype system, which projects a computer-generated image onto the food item, was constructed for enhancing its subjective degree of deliciousness based on its appearance at a pixel level. Finally, a user study was conducted in which the subjective degree of deliciousness based on food appearance was compared under various appearance modification conditions. The results show that appropriate chroma and partial-color modifications highly increase this degree of deliciousness, thus implying that the proposed system can successfully be used to improve the appearance of food to make it look more delicious.

 

 

 

 

Geometrically-Correct Projection-Based Texture Mapping onto a Deformable Object

Projection-based Augmented Reality commonly employs a rigid substrate as the projection surface and does not support scenarios where the substrate can be reshaped. This investigation presents a projection-based AR system that supports deformable substrates that can be bent, twisted or folded. We demonstrate a new invisible marker embedded into a deformable substrate and an algorithm that identifies deformation to project geometrically correct textures onto the deformable object. The geometrically correct projection-based texture mapping onto a deformable marker is conducted using the measurement of the 3D shape through the detection of the retro-reflective marker on the surface. In order to achieve accurate texture mapping, we propose a marker pattern that can be partially recognized and can be registered to an object’s surface. The outcome of this work addresses a fundamental vision recognition challenge that allows the underlying material to change shape and be recognized by the system. Our evaluation demonstrated the system achieved geometrically correct projection under extreme deformation conditions. We envisage the techniques presented are useful for domains including prototype development, design, entertainment and information based AR systems.

 

 

 

 

 

 

Relation between Displaying Features of Augmented Reality and User’s Memorization

In this investigation, we verify a hypothesis: “it has positive effects for user’s memorization ability to use features of Augmented Reality (AR)”. The basis of this hypothesis is derived from the following two features. One is a future of AR: “AR can provide information associated with specific locations in the real world”. The other is a future of human memory: “human can easily memorize information if the information
is associated with specific locations”. To verify this hypothesis, we conduct three user studies. As a result, significant differences are found between the situation in which information is associated with the location of the target object in the real world and that in which information is connected with an unrelated location.

Begin typing your search term above and press enter to search. Press ESC to cancel.