Evolve your ARKit app for spatial experiences

Evolve your ARKit app for spatial experiences

This video provides several key tips on bringing your iPadOs and iOS ARKit expeerience to Spatial Computing.

Overview 0:00

  • Introduced ARKit in 2017
  • Tracks your device with 6 degrees of freedom
    • Allows for anchoring
  • Scene Understanding
    • Geometry and semantic knowledge for placement
  • Rendering
    • SceneKit -> RealityKit
  • ARKit and RealityKit deeply integrated
  • Built in camera passthrough and hand matting
  • World map persistence is a system service

Prepare your experience

  • Present your app
    • Shared Space
    • Full Space
      • AnchorEntity
      • ARKit
  • Prepare your content

Use RealityView

  • Mix and match 2D and 3D
  • Container for entities
    • Entity component system
  • Interact with SwiftUI Gestures

  • Realistic simulations with surroundings

  • Very Similar to ARView - ARView similar to RealityView - Scene similar to Content - AnchorEntity must ARSession in iOS while RealityView does not need permissions

Bring in your content

  • Shared Space
    • Add entities to your view
    • Relative to space origin
    • Can interact with other entities in the space
  • Full Space
    • AnchorEntity
      • No user permission required
      • Transforms not shared
      • support hands
      • ARKit - placement logic
        • Use ARKit Scene understanding
        • Leverage ARKit anchor data
        • Utilize world anchor
        • Transforms are relative to space origin
        • Requires user permission

Raycasting

  • Ray casting provides a 3D location in physical space that corresponds to a given 2D location on the screen. When you call this function, it succeeds in returning a result when a mathematical ray that ARKit casts outward from the user intersects with any real-world surfaces that ARKit detects in the physical environment.
    • Requires collision components
      • MeshAnchor for each item in the world
    • Raycast with System Gestures or Hand Tracking
      • SpatialTapGesture
      • Hand Anchors
    • Add world anchor
    • Combine the elements
    • Interact with it

ARKit updates

  • Run your session
    • iOS have mutiple tracking configurations
      • AROrientationTrackingConfiguration
      • ARImageTrackingConfiguration
      • ARWorldTrackingConfiguration
      • ARPositional TrackingConfiguration
    • Spatial computing
      • HandTrackingProvider
      • SceneReconstructionProvider
      • ImageTrackingProvider
      • PlaneDetectionProvider
      • WorldTrackingProvider
    • So you have your own selection of anchors you would like to recieve.
  • recieve anchor updates
    • iOS
      • ARSessionDelegate
        • ARFrame
          • Anchors
    • Spatial Computing
      • Providers decoupled from the Frame
        • Anchor updates
  • persist world anchors
    • iOS: multiple steps
      • Application
        • Save map
        • Load map
        • Relocalize
        • Get anchors
        • Show content
    • Spatial computing: Handles the map
      • System
        • Save map
        • Load map
        • Relocalize
      • Application
        • WorldTrackingProviders
        • Get Anchors
        • Show Content

Notes mentioning this note

Step Three visionOS
Step Three visionOS

Here are all the notes in this garden, along with their links, visualized as a graph.

100DaysofSwiftUIAlgorithmsAffirmation TimerBPM ClapperBPM TrainingMetronome Vintage 3DHackingWithSwiftSwiftUI Accessibility Hiding and Grouping DataSwiftUI Accessibility Identifying ViewsSwiftUI Accessibility Read Value ControlsSwiftUI Accessibility Support as NeededSwiftUI AccessibilitySwiftUI Advanced ViewsSwiftUI CGAffineTransformSwiftUI Drawing animatableDataSwiftUI Drawing Special EffectsSwiftUI DrawingSwiftUI ImagePaintSwiftUI MetalSwiftUI PathsSwiftUI ShapesSwiftUI Image AlbumImage GeometryReaderSwiftUI Image InterpolationSwiftUI ImageSwiftUI Intergrate UIKitSwiftUI Basic ViewsSwiftUI ButtonsSwiftUI ColorSwiftUI GradientSwiftUI DatePickerSwiftUI Form ValidationSwiftUI FormSwiftUI SliderSwiftUI StepperSwiftUI TextEditorSwiftUI GridSwiftUI GroupsSwiftUI ListSwiftUI Navigation BarSwiftUI ScrollViewSwiftUI SpacersSwiftUI StacksSwiftUI Views And ModifiersSwiftUI Gestures AdvancedSwiftUI Gestures BasicSwiftUI Gestures CombinedSwiftUI GesturesSwiftUI Custom Row Swipe ActionsSwiftUI HapticsSwiftUI HitTestingSwiftUI InteractionsSwiftUI Searchable ViewsSwiftUI Absolute PositioningSwiftUI AlignmentSwiftUI AlignmentGuideSwiftUI Custom AlignmentSwiftUI GeometryReader BasicsSwiftUI GeometryReader UsageSwiftUI How Layout WorksSwiftUI Layout TechniquesSwiftUI Multiple Views Side by SideSwiftUI Switch View with EnumsSwiftUI Switch View with EnumsSwiftUI NavigationSwift NavigationLinkSwiftUI SheetsSwiftUI TabsSwiftUI BindingSwiftUI Environment WrapperSwiftUI FetchRequest WrapperSwiftUI FocusState WrapperSwiftUI MainActor WrapperSwiftUI ObservableObject WrapperSwiftUI ObservedObject WrapperSwiftUI Property WrappersSwift ObservableObject Manually Publishing ChangesSwiftUI State WrapperSwiftUI StateObject WrapperSwiftUI ViewBuilder WrapperSwiftUI ScenesSwiftUI AlertsSwiftUI Confirmation DialogSwiftUI Context MenuSwiftUI Popup WindowsSwiftUI SheetsCS193p Emoji ArtCS193p Matching GameCS193p Set GameStanford CS193pSwift Basic Data TypesSwift BooleanSwift FloatSwift IntSwift StringSwift ArraySwift ClassSwift Complex Data TypesSwift DictionarySwift EnumSwift SetSwift StructSwift Animating GesturesSwift Animating TransitionsSwift Animations TypesSwift animationsSwift Customize AnimationsSwift URLSessionSwift NetworkingSwift URLSessionSwift Comparable ProtocolsSwift ProtocolsSwift Codable @Published ComformanceSwift CodableSwift Documents DirectorySwift StorageSwift UserDefaultsSwiftSwift App BundleSwift Package DependenciesSwift TimerSwift ToolsSwift Basic TechniquesSwift ClosuresSwift ConditionsSwift ExtensionsSwift FunctionsSwift LoopsSwift OptionalsSwift Variable and ConstantsSwift TechniquesSwift Type AnnotationSwift Unique TypesSwift Result TypeSwift Framework CoreDataSwift Framework CoreImageSwift Framework LocalAuthenticationSwift Framework MLSwift Framework MapKitSwift Framework UNUserNotificationCenterSwift Framework Local NotificationsSwift Framework Remote NotificationsSwift Framework UserNotificationsSwift FrameworksSwiftUI FundamentalsSwiftUI WindowGroupA note about catsConsistency is keyHow to ThinkMove your body every dayYour first seedImage InterpolationCreate accessible spatial experiencesDevelop your first immersive appFundamental Design VisionOSGet started with building apps for spatial...Getting Started visionOSBuild great games for spatial computingCreate a great spatial playback experienceDeliver video content for spatial experiencesDiscover Metal for immersive appsStep Eight visionOSExplore rendering for spatial computingMeet Core Location for spatial computingMeet RealityKit TraceOptimize app power and performance for spatial...Step Five visionOSWhat’s new in Xcode 15Design considerations for vision and motionDesign for spatial inputDesign for spatial user interfacesDesign spatial SharePlay experiencesExplore immersive sound designStep Four visionOSDiscover Quick Look for spatial computingMeet Safari for spatial computingRediscover Safari developer featuresStep Nine visionOSWhat’s new in Safari extensionsBring your Unity VR app to a fully immersive spaceCreate immersive Unity appsExplore App Store Connect for spatial computingStep Seven visionOSExplore materials in Reality Composer ProExplore the USD ecosystemMeet Reality Composer ProStep Six visionOSWork with Reality Composer Pro content in XcodeBuild spatial SharePlay experiencesCreate 3D models for Quick Look spatial...Enhance your iPad and iPhone apps for the Shared...Run your iPad and iPhone apps in visionOSStep Ten visionOSBuilding Spatial Experiences with RealityKitEnhance your spatial computing app with RealityKitEvolve your ARKit app for spatial experiencesMeet ARKit for spatial computingStep Three visionOSElevate your windowed app for spatial computingGo beyond the window with SwiftUIMeet SwiftUI for spatial computingStep Two visionOSTake SwiftUI to the next dimensionTen Steps Overview of visionOS By AppleCreate multiple windows in VisionOSTap and Drag Spatial Gesture in VisionOSVisionOS Basic TutorialsvisionOS Documentation SeriesVisionOS Bear Balloon Reverse Gravity No CollisionVisionOS QuestionsWhy attend WWDCNew to WWDC