Month: October 2023

Note: This blog is organised chronologically.

Intro

For my final major project, I intend to create a VR experience within the realms of cinematography. I want to cover this topic while also tackling it within the context of educational environments and how it could prove beneficial or not for students, as a way of complementing their overall content retention and today’s incorporation of education with emerging technology.

Furthermore, it is true that cinematography tends to leave a huge impact on people, and some of the hobbies or interests we pursue are greatly impacted by media, more specifically movies and shows. Just imagine how this potential could be used in VR and how much more it could expand the way we experience that media.

But for now, I think I will focus more on this question in regards to my dissertation. Since the first submission date isn’t until November, that gives me plenty of time to figure out what I could do.

Concept

Through my FMP, I intend to explore the idea of cinematography as a valuable tool for educational settings and whether or not it has an impact on how students (and learners in general) retain information based on visual data, cues and psychology of film-making strategies.

Here are some examples of useful papers I found:

“VR has shown tremendous potential in educating and rehabilitating children (ages 18 and below) with
neurodevelopmental disorders. A study by Wang and Reid (2011) explores the use of this technology, examining
multiple case studies that have tested the efficacy of VR on participants ranging in age, and gender
disorder. These hypotheses were tested for Attention Deficit Hyperactivity Disorder, Autism and Cerebral Palsy.
As a goal, this comprehensive analysis intended to further explore the potential of VR in addressing
neurological impairments in children and contribute to a better understating of its future implications in VR
education.”

This paper was a valuable resource in evaluating education within different neurodevelopmental spectrums. As
a result, it provided me with a crucial perspective on VR education that accounts for other perspectives aside
from those of neurotypical folk. This could help me develop a more accessible experience across a spectrum of
students.

Links: https://karger.com/ned/article/36/1/2/226406/Virtual-Reality-in-Pediatric-Neurorehabilitation

“When addressing filmmaking, the impact of cinematographic practices can vary considerably based on the
medium used to guide the spectator’s gaze. For one, in VR, since the viewer has complete agency over where to
look throughout the experience, cinematographic principles cannot be applied the same way as in 2D screens.
Although these practices may remain the same in terms of taxonomy, the methods used to apply them as a
way of diverting attention within Cinematic Virtual Reality (CVR) change completely. This is due to the viewer’s
agency and the use of 3D immersive environments that enforce the existence of another spatial dimension,
making it a physical experience. This paired with the existence of multiple points of interest (POI) influences the
viewer’s ability to fully absorb the contents of immersive cinematic pieces. Therefore, the focus lies on finding
the methods that allow the creators to direct focus. Further research by Rothe, Buschek and Hußmann(2019),
demonstrates some of the methods proposed, with novel approaches that address the processes within CVR.”

Links: https://www.mdpi.com/2414-4088/3/1/19

For a better analysis of these ideas, please refer to my thesis posted near the end of this page, where I explore these ideas further and create a more supported basis for my approaches throughout this project.

Update 1

I have been reflecting on what kind of cinematic experience I want to have for my project. I think it is logical that it covers some type of school subject since my goal is to make it an educational experience for regular students (or anyone else who is curious and wants to give it a go).

However, I feel like lately I’ve been having somewhat of an artistic block and I can’t think of what I should make. I’ve been brainstorming and I’ve come up with some potential ideas:

• Language experience
• A historical trip through time eras
• Space exploration
• Geographical exploration (potentially using Google Maps in unreal could be cool)
• Anatomy experience

Sadly, I don’t feel strongly about any of those ideas yet and it seems like I don’t really know what I should do… I’ve talked to Eric about this, and he said I should take my time thinking about this and look for inspiration. Hopefully, at some point, I’ll start feeling more drawn towards something.

Side Story

One week ago, a few girls from 3D Animation came to our class looking for people to collaborate with, from BA VR. They presented their idea, which consisted of a cinematic experience in a non-sensical world inspired by Alice in Wonderland. I immediately became extremely excite with the thought of being able to collaborate with them to create something that was still more or less coherent to my plans. But honestly, I think I rushed it a little bit since I was panicking with the fact that I couldn’t find an idea that pleased me at the moment. So I ended up joining their group and changed my original plan.

However, we had a meeting two days ago. This made me instantly realise that it would be a mistake for 3 reasons:

• They had complete control of the project (because it was theirs to begin with)
• I felt like I was getting sidetracked from my original plan
• I started feeling like I did it out of despair and not really for the idea

So, unfortunately, I had to message them saying I wanted to leave the group. And now I am back to square 1. They were really nice and their ideas were great. I simply cannot afford to jeopardise mine, for the sake of feeling comfortable because of their people’s ideas. But I feel like I should still write about it to show that sometimes things don’t go according to plan and we need to stay confident in our ideas and convictions.

Update 2

I have been getting back on track and looking for inspiration, so I can start moving in one direction. And actually, there is one video that really caught my attention! Meta released it in 2021 and it shows audiences a hypothetical depiction of what MR, AR, XR and VR could look like and how immersive they could be when showing people the wonders of space. I thought it was a really smart choice because it’s nearly impossible for the average human being to experience any sort of space-related experiences, and this might be the only way (for now), that we can bring that reality a bit closer to people, with the help of immersive technologies.

When it comes to other subjects such as history or geography, I can visit physical sites and I’ll be “teleported” to that reality because there are still traces of it (not for everything though). What I mean is, it is possible to immerse myself in those subjects through the real world. But not for space. That’s what made the video so enticing!

I feel like this is finally the direction I want to pursue! A VR experience about space, with the goal of not only educating people but also bringing them closer to such a distant and exclusive experience such as exploring space. Now I need to start developing the concept more and figure what the details of the experience. But at least I am finally feeling good about my FMP idea.

Weeks 3 to 6 – Developing ideas and Prototyping

(Note: I am writing this on week 6 so I already have a very solid idea, drawings, prototypes, etc. So I will kind of just skip to the main idea that I arrived at in the end, the feedback from Eric and Herman and my material)

Throughout these 3 weeks, I started to develop my ideas in a more solid and structured manner. My goal was to captivate the viewer with spatial scenarios and bring them closer to it through mesmerising graphics and interesting facts. So I chose to do an experience with three different themes, to bring variety not only to the visuals but also to the content provided to the viewer.

The 3 scenes will be about:

• Mars
• The Solar System
• Blackholes

I thought these would be a great choice since at least two of them are familiar enough to most people that it’s still recognisable and easy to enjoy while still looking amazing. Whereas the black holes scene would be more of an open-ended question since there is still so much to know about them and how they influence our universe. It would end the experience in an open-ended way and leave the viewer expecting more, and hopefully, feeling inspired to want to learn. Like we sometimes feel after watching a great movie (hence the power of cinematography.

I would still want to have some sort of human connection throughout the experience, so I think having someone guiding the viewer throughout the scenes would be a great way to have them connect in a human way as well, which would further captivate them since they’d be able to establish a connection with a mere virtual character.

For the visuals that precede the 3 main scenes, I want to have two other levels. One that will serve as a level selection lobby, where the viewer can choose which lecture to see first. And another one before which takes place “in our actual world”. What I mean by this is a scene that happens in a regular realistic scenario that everyone experiences, school.

Therefore, the experience will start by subverting people’s expectations into thinking they will have a normal school lesson in a boring-looking classroom, with a normal teacher. Which kind of defeats the purpose of VR. Or at least that’s what I want them to think at first. After that scene, the viewer will be transported into the space lobby and the real experience can start.

Also, it is supposed to be a cinematic experience where I want the viewer to be impacted by the visuals, so I don’t want to sacrifice their ability to enjoy high graphics in VR by having too much movement. That would also jeopardise the experience since I’ll be using Unreal which will really push this project to its performance limits. (I’ll talk more about the software below). So the entire experience will have the viewer sitting down in front of a desk (which makes the transition from the classroom environment to space, a lot smoother, since they will both have desks).

Update – Robot teacher

Why stop at the transition between classroom and space? What if the human teacher became a humanoid robot? In my opinion, it would once again impact the viewer once they made the transition from earth to space. No one would expect it, and there would still be a character creating human connection, but in this context, it would shift to something more futuristic. And the fact that this robot teacher wouldn’t exist in real life (at least for now), makes it more exclusive and memorable to the viewer.

I talked about this idea with Eric and he approved it. However, I think he sees it more like a small robot or more peet-like. In my view, it should be humanoid so as to not lose the human side that I want it to have. Besides, it would be much easier to animate with motion capture than moving a robot around, which could give me an unrealistic load of work.

Concept drawing

Initial storyboard

Quick explanation – It starts with the viewer in an ordinary classroom being taught by a human teacher who introduces the theme of the class (astronomy). It then transitions to the space lobby that replaces the classroom and a robot that replaces the human teacher. They give the viewer 3 options to explore: the solar system, mars or black holes. Upon choosing one option, the player is transported to each respective lecture where they can feel mesmerised by the realistic and immersive environments that correspond to each theme. They have an audio lecture and a few interactions at each level that will further teach them about astronomy. At the end of each lecture, they must return to the lobby. Finally, once satisfied the viewer can click on the finish button in the lobby scene and they will be transported “back” to Earth, where the robot does an extremely sentimental and touching monologue about the wonders of space. They then invite the viewer to lie down on the grass with them while gazing at the stars, as it all fades to black.

Lectures

I want to emphasise this again. Besides being a cinematic experience, this project is directed towards education, so I want to incorporate great visuals with interesting, factual and thought-provoking information.

As a result, I want each level to have a small lecture (which will be given by the robot teacher obviously), where the viewer will get to enjoy the environment, as well as learn more about it.

But I feel that simply audio could be ignored so I want there to be visuals, images, signs, slide perhaps, alongside with the audio and the robot. I think that would complement it very well.

Initially, I thought that I could create these lectures with someone specialising in astronomy, but due to time constraints and overcomplicating my work process, I will be using AI to generate the lectures. This could also be useful when writing my script.

Update – Interaction and UI

I feel like this was always on the back of my head, but I think it’s important to fully commit to this idea, as I develop and explain my project. I think the incorporation of good UI design is crucial for this project, as I will be tackling a type of encounter that is mostly based on user experience and comfortability. Therefore, a clear and modern UI seems like a no-brainer.

This will certainly help the viewer smoothly transition between scenes and interactions, while also getting a sense of comfortable user experience through smooth and easy-to-navigate design.

Scenes and respective Viewer Interactions

Scene 1 – Classroom scene

For the classroom scene, I want to create a “typical” sort of boring environment, reminiscent of the ones we can remember and/or sometimes see portrayed in media and films. I want it to have a tremendous impact on the player upon its transition to the space lobby scene.

Here is an AI-generated image of the aesthetic I am aiming for.

Note: This scene has no physical interaction besides UI.

Scene 2 – Space lobby

For the second scene, I want to create a futuristic environment, where the viewer can already get a perspective from space and how wonderful it can look. This scene will serve as both the transition between Earth and space and as a lobby for the viewer to select which lecture they would like to experience. So it should be aesthetically pleasing without being overwhelming, but also comfortable and spacious.

Here are some AI-generated images I used as inspiration.

Note: This scene has no physical interaction besides UI.

Scene 3 – Solar System Scene

This will be the first lecture. In order to make it truly immersive, it will take place right in front of a 3D depiction of the solar system, which the viewers can admire as they listen to and watch the lecture playing in front of them.

Interaction: After this is done, the viewer will have the possibility to teleport in front of the sun or each planet and explore them up close as they get data about it.

Concept drawing

Scene 4 – Mars Scene

For the Mars scene, the viewer will be teleported to a Martian landscape, where they will also listen to and watch a lecture about the red planet.

Interaction: The viewer will get to choose between learning about the Mars Curiosity Rover and its intricacies, or learning about Mars colonization projects and its hypothetical future. For both elements, the viewer will get 3D models which they will be able to analyse and explore.

Scene 5 – Black hole

Finally comes the black hole scene. Similarly to the other two levels, the viewer will be transported to a depiction of the main theme of this level. Meaning they will be in front of a black hole, as they experience the lecture.

Interaction: This one was actually extremely difficult to figure out since I didn’t know what type of interaction the player could have that would be interesting. After talking to Herman and Eric about it, we arrived at the conclusion that being sucked into a black hole would be an entertaining experience to have at this level. It would also teach the player about the physical effects and implications of black holes in matter.

Scene 6 – Earth Scene

This will be the final scene when the viewer returns to planet Earth. It should seem familiar and peaceful. So I decided that a huge grassy field under the starry night sky would do the job. At the same time not deviating from the theme of space exploration and the wonders it has.

Something like this Image:

Note: This scene has no physical interaction.

Initial Game logic

Inspiration

I had quite a few sources that inspired me for this project. First, and probably the most relevant one would be the Meta video I mentioned above, which not only inspired my overall project direction but also the aesthetic I was trying to go for in terms of UI design. Along with Hyper-reality, which gave me a different, yet equally relevant image of UI design in a hypothetical future of XR.

Then for visuals, namely space-related movies, I had a lot of inspiration from both Interstellar and Gravity, which managed to wow the viewer with their overwhelming yet mesmerising depictions of space and its wonder as well as its mysteries.

Finally, Noun Town and Human Anatomy VR are two great examples of gamification of complex subjects and access to very specific and intricate resources through a merely virtual medium such as VR. These were great references for imagining what kind of interactions I would have in my scenes and how I would develop them so that they spark the interest of the player.

Software

Unreal Engine 5

The software I will use for this project is based on what I am trying to achieve (obviously). Since my goal is to get the best and most realistic graphics while getting great performance, the game engine I’ve selected for my FMP is Unreal Engine 5. This is my main tool for the FMP

Blender

The second software I will be using is Blender. This will be for any adjustments of 3D assets, or modelling of simple ones.

Rokoko

Rokoko will be used to record and export motion capture for the robot’s movements.

Adobe Premier Pro (added later)

This software was added later in the project but I decided to include it in this page.

It was used for trimming the lecture videos and removing the original soundtrack

Adobe Illustrator (added later)

This software was added later in the project but I decided to include it in this page.

This software was used for creating UI design

Collaborations

For this project, there will be at least two or three essential collaborations that will help me create a seamless experience while not overloading me with work that I can get done with the help of other fellow students. I will need:

• A Sound Arts Design Student (for overall experience of sound and dialogues)
• An acting student (for robot’s motion capture acting)
• A UI/UX design student (for the experience’s UI Design)

I already have a rough idea of who I can contact for both sound and acting. However, I will need to establish contact with someone from UI/UX design and hopefully get their help as well.

Week 8 – PowerPoint Presentation

Here is my final presentation. Here are things I will not repeat on my blog, such as my research (which is also incorporated in my thesis), I clear explanation of my solidified ideas and respective reasons, as well as prototype videos.

Note: This is also the point at which I chose the title for my project, “Astro Odyssey”. From Astro, relating to space, the planets, stars, or other objects in space, or to a structure in the shape of a star. and Odyssey for a long journey full of adventures.

Scripts

Below are the scripts for the dialogue of the robot teacher throughout the experience. They were created with a mix of my own inputs and also some AI.

Note: Refer to the Update about lectures at the end of the page.

Final Storyboard

There were a few changes that I made to the final storyboard.

• No more classroom scenes (refer to the last paragraph on this page)
• No robot lying down on the grass (to make the experience more coherent and less prone to player location issues)
• In the end, when fading to black, the experience restarts

Synopsis

Here is the synopsis for a clear summary of my project.

Prototypes for First Submission

Here are the videos for the first prototypes of the space lobby scene, using Unreal Engine 5.

I will not get into too much detail on my process here, as I will explain this further on the next page about “Project Development”.

Note: The VR one is a bit laggy, but later on this was addressed and is now smooth as butter.

Thesis and Research for FMP

As mentioned above, my idea for this project was corroborated and explored in my thesis, where I covered multiple aspects of cinematography and their potential impact on content retention for students. For more information, please refer to the document below.

Update about lectures

There have been a number of issues that prevented me from using originally created lectures. Number one was that the information given to me by chatgpt wasn’t completely correct and was omitting crucial points.

Issue number 2, would be creating UI and or videos that matched the speech and gave the viewer interesting and informative data. To do this, I would have to invest way too much time (that I don’t have) on something that isn’t my field of study and that wouldn’t allow me to focus as much on the quality of the overall project. I could have someone do it, but it seems like it would simply give me too many issues and who knows how long I would have to wait.

I needed to make a quick and solid decision to make sure that 1, the information was undoubtedly correct. And 2, the graphics shown along with the info are pleasing and easily comprehensible.

So I decided to switch the lectures to 3 videos by National Geographic, which I found on YouTube and use their script and the videos themselves as material for my own project. There is no better way to treasure the credibility of the information, than accessing that from big scientific sources.

Naturally, I give them credit by not only maintaining the National Geographic logo but by keeping the credits at the end of the video, incorporated within my project.

Update – No Classroom Scene

There was later another decision I had to make in order to save time and keep the focus on the main concept of this project. I talked to Eric about all the elements I would have in that scene, more specifically the met human character, which would have to have their face expressions animated and sewn with the body animation. And as beautiful as it could look, it was just too much work for something that didn’t really represent the main goal of this project.

So, Eric was the one to suggest I remove the first scene (aka the classroom scene) since it wasn’t essential nor completely relevant. He also pointed out that the people trying on this experience will already be inside an educational institution (LCC building), which is enough to make my point, which I agree with.

That leaves me with 5 scenes:

• Lobby
• Solar System
• Mars
• Black holes
• Earth (Final Scene)