Social AR: how to design a gathering in mixed reality (e.g. with Spatial Personas)

Social VR is one of the big trends of virtual reality: people are meeting inside virtual spaces in Rec Room, VRChat, and other similar applications to have fun together or just talk. But now the hype is on mixed reality so, we should also start thinking about how it is meeting people in augmented spaces. We had a taste of it thanks to the recent release of Spatial Personas by Apple, with all the XR influencers sharing videos of them meeting friends in AR in their homes. But is meeting in AR similar to meeting in VR? Does it follow the same rules? And if not, how can we design a gathering in mixed reality? I asked these questions myself and in this article, I want to share with you my thoughts about this topic.

Meeting in Virtual Reality

We are all more or less familiar with gatherings in Virtual Reality: having worked in a company delivering a platform for concerts in virtual reality, I think I have a good understanding of them. When we did a concert in VR, people entering our application were transported into another universe, where they could meet with people from other parts of the world and have fun together. I love VR because it acts as a magical teleport: you put on a headset and you disconnect from reality, going to a totally different place, which may even have rules that are completely different from the ones of the real world (e.g. in VR I can fly). VR gatherings are cool because you can meet with whoever you want wherever you want, being whatever you want (a person, a dog, a milk carton…).

Being a fan of passthrough AR, I’ve always wondered if I could have done the same kind of concert experiences in AR and how social AR could be.

Meeting in Augmented/Mixed Reality

I started investigating this topic a few days ago, and the answer I have given after an analysis of the problem is that no, you can’t have in mixed reality the same experience of a VR gathering. The reason is very simple and goes straight to the meaning of technology: AR/MR is about creating an experience that merges with the environment around you. Mixed Reality should feel like black magic, with the virtual element perfectly blending with the real environment around you, for a result that feels natural and believable.

If we are 10 people in the same social AR experience and all of us are at our own homes, this principle should hold true for all of us, for all our room configurations, which is kinda improbable. Imagine that I have a small room, another player has a huge warehouse, and another one has a narrow corridor. We should all see the virtual elements of the same experience in a way that is at the same time coherent with our environments but that is also coherent between us. Even more, being our avatars virtual elements as well, we should see also each other’s avatars in a way that is believable and consistent, which is basically impossible, because if the guy with the warehouse walks until his faraway walls, we with smaller rooms would see his avatar going beyond our own physical walls, breaking the magic.

But there is even more: even if you launched the experience in rooms with the exact same shape, your furniture would be arranged differently, so someone may appear inside your sofa or your cupboard, breaking again the magic.

Furthermore, as much as it is cool seeing someone appearing inside your room (I tried it and it’s great, it is like a friend of yours came to visit you), it is pretty useless, because he doesn’t see himself inside your room, but inside his room. So you have the sensation that he came to your home, but actually, he has the sensation of having stayed at his home, and this creates a critical difference in how both of you approach the experience.

All of this made me realize that the naive approach of social shared mixed reality just does not work. Mixed Reality is about me living an experience in my own place and this is the same for the other players… making this shared does not work.

At this point, you may wonder “If this doesn’t work, then why all people out there are having a lot of fun with Apple Spatial Personas on Twitter?”. Well, first of all, some social influencers would show themselves enthusiastic even for a potato in mixed reality (Probably making a YouTube video called “THIS MR POTATO CHANGES EVERYTHING” with on the cover the influencer with the mouth wide open and an Apple Vision Pro with a potato in place of the Apple logo). And then because actually there are ways in which you can make a social AR experience feel enjoyable... otherwise I wouldn’t have written this article. It is the naive approach that usually does not work.

Why are you doing social AR?

To solve our conundrum of social mixed reality, we should get back to the roots of every product you build: why are you doing it?

Every product is here to solve a need, and you choose a specific technology because you believe that is the best to fulfill your purpose. Your user, the one with a need to fulfill, also lives in a specific context where your product is being used: an AR headset may for instance be used at home or in the streets, and this difference may influence heavily how you design your product.

Depending on what you are trying to achieve, there are different strategies through which you can craft a good social AR experience. Let’s see some of them.

Events in another world

If you want to do events or gatherings in another world (e.g. on Mars)… well, you use virtual reality. As I’ve said before, this is the right technology to use to fulfill that purpose. MR is about being in your space, if your users should all be in another place, then you have to do virtual reality, or like Apple calls it, “a fully immersive experience”.

But what if your boss wants absolutely you to use MR because “it is the trend of the moment and we need to show that we use it”? Well, in this case, my suggestion is that you still use VR, but then you add some special MR feature. I’m thinking here about Asgard’s Wrath 2: the game was fully VR, but at a certain point, I entered into a time-space-fracture (or something with some other fancy name) and I found myself in my room in MR, where I killed a few monsters, and then I returned to the VR game. Adding a bit of MR, you can make your boss happy and also your users surprised: when I found myself in my room while playing Asgard’s Wrath, I found it to be a pretty cool feature… I wasn’t expecting it.

You just want to see your surroundings

I remember seeing a few times Brad Lynch using VRChat in passthrough mode: basically, he was able to hide all the world environment and see the avatars that were with him inside his physical room. Some of them were beyond the walls, some inside the furniture, but he didn’t care… he just wanted to experiment with the technology.

In other cases, this may be a necessity: maybe someone has just become a father and wants to keep an eye on his baby to see if he’s having any problems. In this case, having passthrough activated in a social VR experience may make a lot of sense, even if the experience is not optimal. Here the purpose is just having awareness of the surroundings.

For me these are not cases of true social AR, it’s just social VR with an AR-passthrough background. As I’ve described in this article about MR camera access and this other one about MR environment understanding, for me mixed reality is about a contextual experience fusing with your environment, not just activating the passthrough.

You want to have a shared MR experience in the same place

In case the users are all in the same physical place, then you have hit the jackpot. In this scenario, you completely fulfill the mission of mixed reality. All users are in the same physical place, living an experience where virtual elements fuse with that physical space, so it makes sense that everyone sees the same real and virtual elements in the same place at the same time. It’s just a matter of creating a coherent experience, using some networking magic (e.g. using the Mirror networking library), and aligning their virtual spaces using Spatial Anchors (which are now present in all AR SDKs). I also advise creating some interactions that involve more than one player at the same time, to create even more sense of commonality.

You want to have a social AR experience about an element in particular

Before I mentioned Apple Spatial Personas and the XR influencers having fun with them. Have you had a look at the kind of social experiences that Apple suggests developers do? If not, let’s have a look at this image together:

As you can see, all the mixed reality experiences are not about a whole environment, but a single element in the scene, may it be real or virtual. If the experience is only about one thing, then it’s easier to create a connection between the users. It’s even easier if this happens with a simple use case like watching a virtual screen (so there is no real element involved and almost no interaction involved), which was the theme of 80% of the social videos of the Vision Pro on X.

If your social mixed reality experience is around a single element, all the users should only gather around an element of that kind in their room and have a common context where to find AR magic. The first good example I had of this kind of interaction has been Demeo: my amazing UX designer Mike invited me to explore Demeo with him and we had a few matches together (which I lost in 3 seconds because I suck at this kind of games… but that’s another story). At first, we were in VR, but then we switched to passthrough augmented reality. Even if I was on Quest 2, it was kinda magical to see him sit at my same table… and he seemed to play his game exactly on my table, as I was doing. The illusion worked because he was at a physical table, I was at a physical table, so we were playing in the same conditions. Plus we were not moving, so we didn’t risk going through walls or things like that. The game created a common physical context, so the MR version translated pretty well. I loved it.

With the AVP it is the same: all people are on a chair watching a virtual screen in front of them. Or they are in front of a table playing virtual chess in MR. The users sit down or anyway don’t move much and they have a common physical context. With these conditions is easier to keep the magic, because the common experience is just happening in a tiny part of the room, with all people having a similar physical context. It’s the same magic of Demeo. It’s more limited than the original idea of having a full shared mixed reality room, but it works… and it is better to do a small thing that works than a big thing that does not.

You can design your experience about one (or more than one) specific attention points as well. Taking the idea of concerts I was talking about before, for instance, you can have the singer on your table, and all the other people being all around your table, all enjoying the same virtual singer around their own physical table. This is a cool way to deliver a social MR experience, even if it feels a bit limited and static (but maybe you can make people move between points of interest).

You want to have a social MR experience that is not fully realistic

The brain has its own rules, and one that is weird but it is true is that the more you give it realism, the more it becomes demanding in terms of realism. If your avatar is a cartoon, the brain is happy with it having 3 fingers instead of 5, but if it is an Epic Metahuman, probably your brain would start noticing that the reflections on the skins are not perfectly realistic. The moment you do something labeled as fantasy work, for the brain everything is fine, but the moment you say something is ultrarealistic, then it will start bitching about every wrong detail (this is the theory of the “uncanny valley”, just that the researchers there didn’t say the word “bitching”).

The Apple Vision Pro uses a trick of this kind, too, to make social MR more believable. The social personas do not have arms, do not have legs, feet, and such (strange that no one is backlashing Apple for it as they did for Meta). They float, which is unrealistic, but is also a good way to escape problems: the avatar of someone who is with you in a room may float on a chair, a sofa, or the floor… in any case, it looks ok-ish. If he’s in the walls it’s a bit stranger, but still, it’s just a ghost.

If in your social MR experience, you don’t need realism, you can play some tricks. For instance, you can play visualization tricks. For instance, you can make people appear as ghosts or clouds, which are particles that can stay anywhere, even behind walls. Or you can play UX tricks: the other people may not have a realistic mapping between their physical position and your physical position. This means that you can choose another person that is in the room with you and his/her avatar teleports to your position so that you can speak by being one in front of the other, without having to physically walk one towards the other, which may be awkward if your rooms have different shapes. Or you can imagine some strange warpings between the various physical spaces (e.g. if my room is 2m x 2m and your room is 4m x 4m, when you do two steps in your physical room, they correspond to one step in mine), but this risks creating a lot of incoherences, so it should be used with caution.

The idea is to bend the rules of reality in your MR experience so that it can become more enjoyable for everyone. But remember, all should be coherent: if you make an experience where everyone is an abstract cloud, probably the whole MR experience should be oniric, abstract, or surreal.

You want people to have a social MR experience at the place of a person in particular

If I want to live a magical MR experience in my home, and I want some friends to enjoy it with me, the natural choice would be to invite them home (in the real world) and enjoy it together. But if they are distant and this is not possible, the best alternative solution in my opinion would be to invite them virtually at home and have the same experience together. So I should scan my home, send the 3D model to them, then enter the MR experience in passthrough, while they enter the same experience but in VR, with their virtual world being my room. This way we could have the same experience together, in the same context, which for me is real and for them is virtual.

I am making some tests on this sense with the Quest, and the Quest 3 is actually able to reconstruct a mesh of the environment of the user, and this can be shared somehow with the other users (you just need a common backend for the upload/download). But the mesh is quite rough, plus has no texture, so people can only enter inside a white reconstruction of your place. On mobile phones, there are instead various photogrammetry applications because object/room scanning was very popular a few months ago. But as far as I remember, the reconstruction usually still needs some manual adjustment. So this solution is theoretically good, but practically the technology is still not ready to support it.

There have already been experiments in this sense: I remember that Boz talked about a similar scenario when Facebook’s mission was “Defy Distance”. On a more practical side, Varjo Reality Cloud had exactly this mission: the main user had a Varjo headset showing him mixed reality and at the same time scanning his room with the cameras of the device. The reconstruction of the room, updated live on the Varjo Cloud, was sent to the headsets of the other users, which could meet with the first user in VR, seeing him inside his own room. The results were pretty rough, but it was an interesting prototype nonetheless.

I’m personally a big fan of this solution. This is the way I can truly invite friends “at home” remotely: I see my home and they see my home, too. We are all in the same space and we can do some activities together: for me, social mixed reality may involve both augmented reality and virtual reality if we want a coherent experience.

These are the solutions I have been able to find to make people meet in social mixed reality and enjoy their experience. In case you have successfully applied other strategies, please let me know in the comments of the post and I will add them to this article. Enjoy social MR development, and as usual, if you need an experienced team to build your MR experience, feel free to get in touch with me…

(Header image by Apple)

Related