CSE 566 Virtual Reality

Spring 2026

Assignment 3: Augmented Reality (20% of final grade)

Prof. Arie Kaufman

Due date: Friday, Apr 10, 2026, 11:59 p.m. (Stony Brook Time)

 

Live help session for Unity’s AR Foundation

The TAs will deliver a tutorial for AR Foundation and hold a help session for this assignment on Monday, March 9, 2026, during the regular course hours. The tutorial will cover from the installation of Unity’s AR Foundation and its necessary packages to the implementation of a basic interactive AR application as well as building to an Android and iOS device. However, feel free to try Unity’s AR Foundation yourself even before the session. We highly recommend you to attend the in-person session.

 

Introduction & Setup

In this assignment, you will be exploring the realm of augmented reality (AR). Similar to your

previous assignments, you are required to build a virtual world, but by “augmenting” the real world on planar surfaces in the real world. However, in this assignment, you will be holding your device with your hand (Hand-held AR) instead of mounting it onto your head. You will be using Unity’s AR Foundation as the platform for AR application development (AR Foundation Installation Guide for Unity 2022.3.25 and 6000.0.62; CodeLabs Tutorial on Building Your First Prototype). You will use your smartphone or tablet for this assignment. Note that the specification of the assignment assumes that you are using our recommended version of Unity (2022.3.25f1 or 6000.0.62f1). Also, we strongly recommend that you meet the following minimum requirements for this assignment to support AR Foundation and its core library (ARCore for Android, ARKit for iOS). For better context of the concepts used in this assignment, you may remind yourself with our course material on Augmented Reality.

 

Assignment Overview

For this assignment, you will design an application in which a user can interactively build an augmented 3D scene on a planar surface in the real world. The concept of the assignment will be about: Recycling and binning. Your AR application should have the following virtual 3D objects: Two types of large bins (in the Utility room) one is a recycle bin and the other is a compost bin, two types of recyclable items, two types of compostable items, and a portable collection box. You can create your own 3D models, or download models from Unity Asset Store or any other sources, provided you have permission to download them and cite each source properly in your documentation.

 

Your application will have two modes: a build mode and a game mode. Therefore, your application should have a simple UI to toggle between each mode.

 

In the build mode, the user should be able to place the aforementioned 3D virtual objects on a planar surface in the real world. For that, the user will have to choose a horizontal plane from their real world scene, such as the bedroom floor or their study desk (Hint: this can be achieved using AR Foundation’s ground-plane detection capability; Markerless AR). After selecting a horizontal surface, your application should prompt a UI that will enable the user to choose a virtual object, manipulate its position and rotation, and confirm its placement in the real world.

 

In the game mode, the user will have to carry out the task of the application – that is, collect the virtual objects and place them in a virtual collection box that must always be fixed to one of the user’s hands (Hint: AR Foundation’s image-tracking capability; Marker-based AR). Once the collection box is full of virtual garbage (e.g., fallen leaves, cans, snacks), your application must inform the user to empty the collection box to the two large bins in your Utility room.

 

Reminder: you will perform some physical movement while you are working/playing with this assignment. Please always be aware of your physical surroundings (e.g., stairs, steps, walls, desk corners) to avoid any injury.

 

Physical-world interactor preparation

To present the users the fascinating experience of AR, your application must provide a number of interactions between the virtual world and the physical world:

 

  1. Physical world setup

Printed markers are often used in AR applications to place virtual objects on absolute physical locations. In such a setup, the application places virtual objects exactly on the marker’s physical location whenever the marker is in the user’s view, nearby. In this assignment, you are required to print and place physical markers to determine the location of the large bins and the collection box (on the hand). A marker can be any image. However, there are a set of rule-of-thumbs for stable virtual object registration. It is recommended to use this “QR-code-like AR marker” generator. Each physical marker must be unique (hit “Regenerate” per marker). Followings are the markers you have to print and place in your application:

 

1.     Two large image markers - These physical markers will be used to mark the two virtual large bins in the Utility room. You should print each marker (e.g., QR code) on a large piece of paper that is no smaller than the A4-sized paper.

2.     One small image marker (Hand-palm size) - This marker will be used to place the virtual portable collection box. It must be printed on a piece of paper that is as small as a palm of a person’s hand so that the user can easily carry the image marker on one hand.

 

  1. Virtual setup

Three virtual objects associated with markers - These marker holders need to be configured so that the application recognizes the physical world markers. The three Image-based markers will be used to detect: the (1) Recycle bin, the (2) Compost bin, and the (3) Portable collection box. These virtual objects need to be rendered on the user’s device whenever the device detects the physical markers. Hint: Each marker gameobject needs to recognize only one physical marker (1-to-1 assignment).

 

Build mode (Placement)

In this mode, the user should be able to select and place 3D virtual objects on a flat surface of the real world. There should be a library of predefined objects similar to Figure 1. When a virtual object from the library is selected, the user should be able to choose a position in the real world, using the view through their AR device. Your application should then place the selected 3D object at the specified position. Your library should include at least the following:

 

●      Six leaves - The color of the six leaves must be distinct. It can only be placed on a horizontal planar surface in the real world and each must have the collider component.

1.     Planar surface 1 - Three of the leaves must be on a physical planar surface

(e.g. Floor).

2.     Planar surface 2 - The other three leaves must be on another planar

surface  (e.g. Desk) different from Planar surface 1.

●      Four soda cans - The four cans can have a uniform texture (material). The cans can only be placed on a horizontal planar surface in the real world and each must have the Collider component.

●      Four bags of chips - The four bags of chips can have a uniform texture. It can also only be placed on a horizontal planar surface in the real world and each must have the collider component.

 

 

Figure 1. Example look of the predefined selectable item library in the Build mode

 

Build mode (Transformation)

When a 3D object from the library is selected and the user taps on the device screen, the user should be provided with a UI similar to Figures 2. For full points, the following minimum functionalities should be implemented:

 

A joystick UI to (1) translate the virtual objects in the x-z direction on the planar horizontal surface and another joystick to (2) rotate them about the Yaw axis (local up-vector) of the surface.

 

The reasoning behind the joystick UI is to allow the user to freely modify the position and the rotation of each virtual object even after the screen tab action. Once satisfied, the user can tap the ‘Finish edit’ button on the screen to finalize the placement.

 

 

Figure 2. Example look of the joystick UI in Build mode

 

Game mode

When a user finishes building the virtual scene augmented on real world surfaces, the user should be able to tap the ‘Mode’ button to go to the Game mode, and interact with the world she/he has built. Unlike previous assignments, this assignment primarily focuses on the interaction between the real-world and the virtual world.

 

●      Surface objects anchoring - It is important in AR that the virtual objects are behaving similar to the physical objects in the real-world. To this end, the virtual objects that were placed on physical surfaces in the Build mode must be “anchored” (hint: Markerless AR) to their set positions. Once in the Game mode, the virtual objects’ visibility is determined by the relative position between the AR camera and the virtual objects. In other words, the virtual objects will disappear in the display of your AR device when they are out of the view and need to be rendered again when the virtual objects are located inside the AR camera’s viewing frustum.

 

●      Image-based marker objects anchoring - Similar to the objects that are anchored to the ground planar surfaces, the application must be able to correctly recognize the two stationary placed physical markers and one marker that may be constantly in motion. Note that the latter marker (Marker that is placed on top of the user’s palm/hand) must be correctly rendered even when the user tilts the hand by a certain degree (e.g. -30 ~ 30 degrees).

 

●      Navigation & View - Navigation in AR requires you to physically walk to the destination. Your application must correctly visualize all the virtual objects in the physical world during navigation. Also, you must let the user keep her/his hand shown while navigating. This is to show the collection box at all times, providing intuitive visualization of the collected objects (showing all the collected virtual garbage in the box). This is demonstrated in Figure 3. However, the user may place this portable marker on the physical surface when the navigation is stopped (when the user is about to collect a virtual object), for the ease of user’s interaction.

 

 

Figure 3. Example look of navigation in Game mode

 

●      Item collection - Users should be able to drag and drop the virtual objects from the placed position to the position of the virtual collection box. Therefore, the user must first physically place the marker (collection box) from their palm to the floor and then tap the virtual object on the device screen to select and “drag-and-drop” it to the collection box. Finally, the collection box must update to correctly show all the collected virtual objects.

 

●      Box emptying - Users must empty the portable collection box when the box is filled with 6 items. However, the user can empty it even before it’s full. In this scenario, the user must bring the portable box and empty it at the Utility room where the two bins are. The way to empty the box can be similar to item collection. You can either place the collection box on the surface (floor) of the Utility room and “drag-and-drop” to pour the collected items to a large bin, or you can tilt the collection box to throw away all the items at once. You do not need to implement a functionality to check if the litters are dumped to the appropriate bin (compostable or recyclable), but the environment is important! Let’s try to only collect the same type of trash, each, before emptying. Box emptying is demonstrated in Figure 4.

 

 

Figure 4. Conceptual illustration of box emptying in Game mode

 

 

Extra Credit

Extra credit of up to additional 10% of the assignment value in the final grade will be given to extra functionalities and/or features that you implemented for this assignment. We will be more excited to see one or more of these functions from you: (1) Applying physics engine (gravity) to virtual garbage box/bins and the garbage (e.g., collected trash falling from your hand to the floor once the bin is tilted; collected trash following the physics inside the portable box), (2) the ability to recognize grabbing hand (e.g., pinching) gestures to collect the virtual garbages, (3) allowing users to choose their dominant hand at the start of the game. This way, the dominant hand (e.g. Right hand) will be used to carry out the interaction and the secondary hand will have the collection box fixed to it, (4) cloud-based anchoring, or any (5) innovative approach for more stable object anchoring. Note that the extra functionalities will be evaluated based on their difficulties.

 

Demonstration Video

You have to show all the functionalities in your demo video for full credit. The recorded demo must include both (1) the screen of the user’s AR device AND the (2) third-person view. You will also need to upload a video recording of your application. To do that, turn on the screen video record on your phone and run the application. Please refer to the example look of the video recording. We advise you to get assistance from a friend to record this video. Please contact the TAs regarding any recording environment limitations. Note that this is a MUST. Any missing material may result in point deduction.

 

Submission

It is strongly recommended that you test your application before submitting your work. Please save and zip your Unity project in a folder named “CSE566 Assignments” in Google Drive and enable the share of this folder rather than sharing the single project file. In Brightspace, please submit the link to your Google Drive folder.

 

For this assignment, include the following for full credit: your Unity project folder, a report, and a video. Your Unity project folder should contain your Unity scene and all your Scripts and Assets that will be required to rebuild your project. Please do not include your project executable in the folder. Any missing materials or disagreeing to follow the instructions may lead to a point deduction.

 

Interact with your application and make sure you show all the aspects of the assignment, including, but not limited to, position/orientation change, build/play mode, and all required AR objects for this scene. Please note that you must demonstrate your application in all angles with varying distances. To elaborate, the objects must remain in the initial placed position with the same orientation unless the user brings the physical AR device closer or rotates the orientation of the device.

 

For your report, you should include at least the following:

-        A title: “CSE 566 Virtual Reality, Spring 2026, Assignment 3: Augmented Reality”

-        Your name and Stony Brook ID

-        Unity and AR Foundation version

-        Hardware used

-        Directory hierarchy

-        Any extra functionalities/ features that you implemented for this assignment

-   Details on implementation: references to the downloaded 3D models or how you design your own model; how the rotations in your scene were implemented; how you designed your own model; how the controls were implemented; how the golf game was implemented; to mention but a few.