Meet ARKit for spatial computing

Last updated on August 31, 2023

Meet ARKit for spatial computing

This video provides several key tips on creating SwiftUI ARKit for visionOS, specfically Hand Tracking and Scene Geometry. Placing object into the real world, that you can interact with using hands.

Overview 2:30

Available in Swift and C
Use any combination of features together
Privacy-first design for secure access to data
3 Components
- Anchors
  - Position and orientation in real world
- DataProvider
  - Observe data like anchor changes
- ARKitSession
  - A set of data providers
Privacy
- Daemon filters data before reaching app
- Full space
- Authoritized ```swift // Privacy // Authorization

session = ARKitSession()

Task { let authorizationResult = await session.requestAuthorization(for: [.handTracking])

for (authorizationType, authorizationStatus) in authorizationResult {
    print("Authorization status for \(authorizationType): \(authorizationStatus)")

    switch authorizationStatus {
    case .allowed:
        // All good!
        break
    case .denied:
        // Need to handle this.
        break
    ...
    }
} } ```

World Tracking 5:30

Allow anchor object into real world
- WorldTrackingProviders
  - Add WorldAnchors for anchoring virtual content
    - WorldAnchor Origin used to hold vitrual content in place
  - Automatic Persistance
    - location is saved along with World Anchor
  - Get the device’s pose relative to the app’s origin
    - Position and orientation of the device relative to the app’s origin
    - Primarily targeted at developers doing their own rendering
    - Querying for the pose is a relatively expensive operation
    - For more see
      - Discover Metal for immersive apps
      - Optimize app power and performance for spatial computing

Scene Understanding 11:50

Inform you about your surroundings
- Plane Detection
  - PlaneDetectionProvider
  - Each plane is provided as a PlaneAnchor
  - Useful for content placement or low-fidelity physics simulations
  - PlaneAnchor
    - alignment: oriention of plane, horizontal or veritical
    - geometry: dimensions of plane
    - classification: type of plane, .wall, .floor, .ceiling, .table, .seat, .window, .door
- Scene Geometry
  - Polygonial mesh the represents the real world
  - SceneReconstructionProvider
    - Mesh geometry is provided as MeshAnchors
    - Useful for content placement or high-fidelity physics simulations
      - geometry
        
        vertices
        
        normals
        
        faces
        
        classifications
        
        .wall, .floor, .ceiling, .table, .seat, .window, .door, .stairs, .bed, .cabinet, .homeAppliance, .tv, .plant
- Image Tracking
  - Detection 2D images in real world
  - ImageTrackingProvider
    - Specify a set of ReferenceImages to detect
      - loadReferenceImages
      - CGImage
      - PixelBuffer
    - Detect images are provided as ImageAnchors
      - estimatedScaleFactor: size of image compares to specified physical size
      - referenceImage -Useful for placing content at known, statically placed images
      - information next to a movie poster.

Hand Tracking 15:22

Skeletal data for your hands
- HandTrackingAnchor is a trackable anchor
  - skeleton:
    - joint
      - parentJoint
      - name
      - localTransform: relative to parent joint
      - rootTransform: relative to root joint
      - isTracked
  - chirality: left hand or right hand
  - transform: wrist transform relative to app
Each hand is provided as a HandAnchor
Useful for content placement or detecting custom gestures
Poll for latest handAnchors or receive HandAnchors when updates are available

Example 18:00

App and view model ```swift // App

@main struct TimeForCube: App { @StateObject var model = TimeForCubeViewModel()

var body: some SwiftUI.Scene {
    ImmersiveSpace {
        RealityView { content in
            content.add(model.setupContentEntity())
        }
        .task {
            await model.runSession()
        }
        .task {
            await model.processHandUpdates()
        }
        .task {
            await model.processReconstructionUpdates()
        }
        .gesture(SpatialTapGesture().targetedToAnyEntity().onEnded({ value in
            let location3D = value.convert(value.location3D, from: .global, to: .scene)
            model.addCube(tapLocation: location3D)
        }))
    }
} } ``` ```swift  // View model

@MainActor class TimeForCubeViewModel: ObservableObject { private let session = ARKitSession() private let handTracking = HandTrackingProvider() private let sceneReconstruction = SceneReconstructionProvider()

private var contentEntity = Entity()

private var meshEntities = [UUID: ModelEntity]()

private let fingerEntities: [HandAnchor.Chirality: ModelEntity] = [
    .left: .createFingertip(),
    .right: .createFingertip()
]

func setupContentEntity() { ... }

func runSession() async { ... }

func processHandUpdates() async { ... }

func processReconstructionUpdates() async { ... }

func addCube(tapLocation: SIMD3<Float>) { ... } } ``` - Session initialization
- runSession() - Hand colliders
- process Hand Updates ```swift  class TimeForCubeViewModel: ObservableObject {
...
private let fingerEntities: [HandAnchor.Chirality: ModelEntity] = [
    .left: .createFingertip(),
    .right: .createFingertip()
]

...
func processHandUpdates() async {
    for await update in handTracking.anchorUpdates {
        let handAnchor = update.anchor

        guard handAnchor.isTracked else { continue }

        let fingertip = handAnchor.skeleton.joint(named: .handIndexFingerTip)

        guard fingertip.isTracked else { continue }

        let originFromWrist = handAnchor.transform
        let wristFromIndex = fingertip.rootTransform
        let originFromIndex = originFromWrist * wristFromIndex

        fingerEntities[handAnchor.chirality]?.setTransformMatrix(originFromIndex,             relativeTo: nil)
    } ``` - Scene colliders

func processReconstructionUpdates() async {
        for await update in sceneReconstruction.anchorUpdates {
            let meshAnchor = update.anchor
            
            guard let shape = try? await ShapeResource.generateStaticMesh(from: meshAnchor)             else { continue }
            
            switch update.event {
            case .added:
                let entity = ModelEntity()
                entity.transform = Transform(matrix: meshAnchor.transform)
                entity.collision = CollisionComponent(shapes: [shape], isStatic: true)
                entity.physicsBody = PhysicsBodyComponent()
                entity.components.set(InputTargetComponent())

                meshEntities[meshAnchor.id] = entity
                contentEntity.addChild(entity)
            case .updated:
                guard let entity = meshEntities[meshAnchor.id] else { fatalError("...") }
                entity.transform = Transform(matrix: meshAnchor.transform)
                entity.collision?.shapes = [shape]
            case .removed:
                meshEntities[meshAnchor.id]?.removeFromParent()
                meshEntities.removeValue(forKey: meshAnchor.id)
            @unknown default:
                fatalError("Unsupported anchor event")
            }
        }
    }

Cubes

class TimeForCubeViewModel: ObservableObject {
  func addCube(tapLocation: SIMD3<Float>) {
      let placementLocation = tapLocation + SIMD3<Float>(0, 0.2, 0)

      let entity = ModelEntity(
          mesh: .generateBox(size: 0.1, cornerRadius: 0.0),
          materials: [SimpleMaterial(color: .systemPink, isMetallic: false)],
          collisionShape: .generateBox(size: SIMD3<Float>(repeating: 0.1)),
          mass: 1.0)

      entity.setPosition(placementLocation, relativeTo: nil)
      entity.components.set(InputTargetComponent(allowedInputTypes: .indirect))

      let material = PhysicsMaterialResource.generate(friction: 0.8, restitution: 0.0)
      entity.components.set(PhysicsBodyComponent(shapes: entity.collision!.shapes,
                                                 mass: 1.0,
                                                 material: material,
                                                 mode: .dynamic))

      contentEntity.addChild(entity)
  }
}

Notes mentioning this note

Discover Metal for immersive apps

Step Three visionOS

Here are all the notes in this garden, along with their links, visualized as a graph.