CSE 566 Virtual Reality
Spring 2026
Assignment 4: Advanced VR (20% of final grade)
Proposal Report Due: Mon,
Apr 13, 2026, 11:59 p.m. (Stony
Brook Time)
Final Report Due: Mon,
May 4, 2026, 11:59 p.m. (Stony
Brook Time)
Over the course of this semester, you have built up a strong foundation in Unity, VR scene design, 3D user interfaces, and immersive interaction. For this final assignment, you will bring all of that experience together into a capstone project of your own design. Rather than following a prescribed scenario, you will propose, develop, and demonstrate your own VR application, one that incorporates artificial intelligence as a meaningful component of the experience.
This assignment is deliberately open-ended. You are encouraged to be ambitious and creative. Your application will be evaluated on the quality of your design decisions, the depth of your AI integration, and the polish of your final experience, not on whether it resembles what anyone else built.
Overview
You will design and implement a VR application that meets the following high-level requirements:
● Original concept. Your application must be a new experience, not a direct extension of Assignments 1 or 2. You may reuse assets (terrain, models, scripts) from earlier assignments, but the overall concept and primary interaction loop must be novel.
● AI integration. Your application must incorporate a Large Language Model (LLM) or any other AI system in a way that is meaningful to the user experience - and not as a developer tool used during production. See Section 3 for details and examples.
● VR hardware. Your application must run on a VR-capable device. You may continue to use the Meta Quest 1 headset from earlier assignments, but you may explore other supported platforms (see “VR Platform And Hardware”). Whichever device you choose, the experience must be designed thoughtfully for that hardware's capabilities.
● Interaction design. Your application must include at least two distinct, intentional forms of user interaction. These should feel appropriate to the chosen device and scenario.
Design Proposal (Due: Apr. 13,
2026)
Before beginning
the development, you must submit a short design proposal for the TA’s approval.
The proposal should be no more than one page and must include:
● A title and one-paragraph description of your application concept.
● A brief explanation of how AI will be integrated into the experience (what it does, why it matters to the user).
● The target VR device and platform (e.g., Meta Quest, PC VR via SteamVR, WebXR on mobile, Apple Vision Pro, etc.).
● A list of the two or more interaction modes your application will support.
● Any external assets, APIs, or SDKs you plan to use.
If your proposal is rejected or needs
revision, you will receive feedback and one
opportunity to resubmit. Do NOT begin the development until your
proposal is approved.
AI
Integration Requirement
A key requirement of this assignment is that your application uses an AI system — most likely a Large Language Model (LLM) such as the OpenAI API, Anthropic Claude API, or a local-run open-weight model, in a way that is visible and meaningful to the end user. Calling an LLM only during development (e.g., to help write scripts/code) does NOT satisfy this requirement (Refer to the “Use of AI for Development” Section) . You must have an AI component affecting the game-play of your application, in the run-time. Also, the AI component must play a functional role inside the running application. Some examples of qualifying integrations:
● Scene manipulation. User interaction with the scene is conveyed to the LLM, which triggers actions or movement in response.
● Conversational NPC. A non-player character whose dialogue is generated in real time by an LLM, allowing the user to ask free-form questions and receive contextually appropriate answers.
● AI-driven narration or commentary. The application observes what the user is doing (gaze target, position, action taken) and uses an LLM to generate spoken or displayed narration that reacts to those events.
● Generative environment or puzzle. An LLM or generative model produces unique content at runtime — e.g., clues, riddles, world-building lore, or puzzle configurations — that the user must interact with.
● AI accessibility layer. The application uses an AI system to provide real-time audio descriptions, adaptive difficulty, or natural-language navigation commands for users with different needs.
● AI game master. In a game or simulation, an LLM acts as a rule-enforcing or narrative-guiding agent that responds dynamically to player choices and shapes the course of the experience.
You are not limited to these examples. Whatever form the AI takes, you must clearly document the following in your report: (a) which AI system or API you used, (b) what role it plays during a session, and (c) how a user would notice or benefit from it.
VR Platform and Hardware
You may build for any VR-capable platform. The following are suggested for Unity. Depending on your proposal and instructor/TA approval, we may be able to provide you with access to a different headset than the Meta Quest 1. If you wish to target a platform not listed here, please discuss it with a TA during the proposal stage.
|
Device / Platform |
Build Target |
Unity SDK / Plugin |
Notes |
|
Meta Quest 1 / 2 / 3 / Pro |
Android (ARM64) |
Oculus XR Plugin |
Controller-based 6DoF |
|
PC VR (Valve Index, HTC Vive, Reverb G2) |
Windows (x64) |
OpenXR / SteamVR Plugin |
High-fidelity, tethered |
|
Apple Vision Pro |
visionOS |
PolySpatial / Unity visionOS |
Eye + hand tracking, spatial |
|
WebXR (phone or standalone) |
Browser |
Unity WebGL + WebXR Exporter |
Cardboard / 3DoF, wide reach |
|
Samsung / Android Phone + Cardboard |
Android |
Google Cardboard XDK |
3DoF, gaze-based |
If you choose a platform other than the Meta Quest 1, briefly justify that choice in your report: explain what capabilities of that device you are specifically taking advantage of (e.g., hand tracking, gaze tracking, spatial audio).
Interaction Design Requirements
Regardless of platform, your application must include at least two distinct interaction modes or systems. These should be purposefully designed for your chosen hardware. Below are some examples — you may use these or define your own, subject to proposal approval.
● Navigation. The user can move through the environment (teleportation, physical walking, continuous locomotion, or any other technique appropriate to the device).
● Object interaction. The user can pick up, examine, place, or otherwise manipulate objects in the scene using hand controllers, hand tracking, gaze, or voice.
● AI dialogue. The user can initiate or receive a conversation with an AI-driven character or system, using voice input, a gaze-activated prompt, or a controller button.
● Mode switching. The application has two or more distinct modes (e.g., exploration vs. conversation vs. build mode) and the user can move between them through an intuitive interface.
● Spatial interaction and physics. The user can manipulate objects in the scene by relying on physics-based interactions (e.g., colliders, gravity, etc).
The interactions you include should feel cohesive: they must serve the concept of your application, not exist merely to satisfy a checklist.
Example Project Ideas
The following examples are provided to spark ideas. You are not required to build any of the described below, and you will not receive extra credit for doing so. Original concepts are strongly encouraged.
AI
Campus Tour Guide
Build on the Stony Brook campus scene from Assignment 1 and add a virtual tour guide NPC powered by an LLM. The guide responds to free-form voice or text input from the user, answering questions about campus buildings, giving directions, or telling stories about the university. As the user looks at different buildings, the guide proactively comments on what they are seeing. The experience might run on a Quest 3 with hand tracking, allowing the user to “wave” at the guide to start a conversation.
AI-Driven
Escape Room
Design a small VR escape room in a novel setting (a spaceship, a medieval library, a flooded laboratory). An LLM acts as a cryptic “game master” voice that provides hints, responds to the user saying things like “I’m stuck” or “tell me more about this object,” and dynamically adjusts its clues based on how far along the player is. The AI component means no two playthroughs feel identical. This could run on a PC VR headset to take advantage of higher graphical fidelity.
VR
Storytelling Engine
Build a short interactive narrative in VR where key story beats (e.g., NPC dialogue, environmental descriptions, and branching plot points) are generated dynamically by an LLM based on user choices. The user might play as an explorer arriving in a foreign village, and the villagers they meet respond authentically to whatever the user has done so far. A WebXR build could make the experience easily shareable and accessible without a dedicated headset.
AI-Reactive
Scene
Design a VR environment where the user's physical interactions with objects (e.g. picking something up, moving to a new location, activating a switch) are described to an LLM in real time, which then decides how the scene should respond. For example, moving a chess piece might cause an AI opponent to respond by animating its own move; dropping an object near a character might make them react with surprise; entering a new room might cause the lights and ambient sound to shift based on the LLM's interpretation of the mood. The AI acts as an invisible director, keeping the world feeling alive and responsive without scripted triggers.
Extra credit
Three extra credit functionalities are listed below. Successfully implementing any of them is worth an additional 10% of the assignment value towards the final grade (i.e., up to 2.5% of your final course grade). The extra credit points will be evaluated based on the completeness of your implementation. Please describe any extra credit work clearly in your report.
Persistent AI
Memory Across Sessions
By default, LLMs are stateless, so each time your application starts, the AI has no recollection of previous sessions. For this extra credit task, implement a persistence layer so that the AI retains meaningful memory of past interactions across separate runs of your application. For example, an AI character might remember the user’s name, past decisions, or unresolved story threads from a previous session, and reference them naturally in the current one.
Your implementation must include:
● A storage mechanism for session data (e.g., local JSON file, PlayerPrefs, or a lightweight database) that persists between application runs.
● A strategy for injecting relevant past-session context into the LLM prompt without exceeding context window limits (e.g., summarization, selective retrieval, or embeddings-based search).
● At least one clearly demonstrable moment in your demo video where the AI references something from a prior session.
Multi-User
Networked VR
Extend your application to support two or more simultaneous users in a shared virtual space. Each user should be represented by an avatar visible to the others, and their movements and interactions should be reflected in real time across all connected clients. The AI component must remain functional in the multi-user context; for example, all users can hear or see the AI’s responses, or the AI takes input from multiple users and responds to the group.
You may use any networking solution compatible with Unity, such as Photon PUN, Unity Netcode for GameObjects, or a custom WebSocket server. Your report must document the architecture: how state is synchronized, how the AI is queried (one client acts as host, or independently per client), and any latency or consistency trade-offs you encountered.
Your demo video must show at least two separate client windows or devices running simultaneously and interacting with each other and with the AI.
On-Device /
Local LLM Inference
Rather than invoking a cloud-based LLM API call (e.g., OpenAI, Anthropic), run a language model, locally, on your device. It could be either on your VR headset itself or on the host PC driving the headset. This setup may reduce network latency, connection dependency, and provide improved privacy-protection, but requires you to reason carefully about model size, quantization, and inference performance under real-time constraints.
Suitable approaches include running a quantized model (e.g., Llama 3, Mistral, Phi-3) via llama.cpp, Ollama, or LM Studio, and calling it from Unity via a local HTTP endpoint. On-headset inference on a standalone Quest may require higher effort to integrate and will be weighted accordingly.
Your report must document: the model used, its quantization level, average inference latency observed during your demo, and any quality trade-offs compared to a cloud API. Your demo video must show the application running with no internet connection active.
Use of AI/LLM for Development
The goal of this
assignment is to encourage creativity in designing your own AI-integrated
system within Virtual Reality. Through this process, you will gain hands-on
experience with current AI and LLM trends, while applying core concepts of
Virtual Reality, learned in this course. You may use AI agents for ideation and
collaborative brainstorming. However, using AI to implement your system is strictly prohibited. We want you to fully engage
with and learn from this course, and relying on AI-generated code contradicts
this objective. Please review the Academic
Integrity and Artificial Intelligence Statement and the AI Usage Policy from the Department of
Computer Science for further details on this policy.
Submission
It
is strongly recommended that you test your application before submitting your
work. Please save and zip your Unity project in a folder named “CSE566
Assignments” in Google Drive and
enable the share of this folder rather than sharing the
single project file. In Brightspace submission, please submit the link to your
Google Drive folder.
For
this assignment, include the following for full credit: your
Unity project folder, a report, and a video. Your Unity project folder should
contain your Unity saved scenes and all your Scripts and Assets
that will be required to rebuild your project. Please do not include your
project executable in the folder. You
will also need to upload video recordings for
your application, such as using the Meta Quest (or your device’s) screen
recording functionality.
You
are required to record a 5–10 minute demo video of
your application running on your target device. The video must demonstrate:
● All required interaction modes.
● At least one complete interaction with the AI
component (showing user input and the AI response).
● Any extra features you implemented.
Make sure you show all aspects/functionalities
of the assignment, including but not limited to the scene objects and
the interactions, and the video recording should make it clear that you are actually running VR.
For your report, you should include at
least the following:
- A
title: “CSE 566 Virtual Reality, Spring 2026, Assignment 4: Advanced VR”
- Your
name and Stony Brook ID
- Your
approved design proposal
- Any
extra functionalities/ features that you implemented for this assignment
- Details
on implementation: references to the downloaded 3D models or how you designed
your own model; how the motion paths in your scene were implemented; how the
scene transition was implemented for switching between the modes, to mention
but a few.