PaaS (Phone as a Sensor)

A bit over a year ago I started a course at university for my robotics Masters programme, aimed at combining all that we had learned in the first year, appropriately named: Multidisciplinary Project (MDP). This was going to be great, I thought, a chance to fully develop a robot software stack, combining vision, navigation, and task planning, all nicely integrated in ROS2. A real practical project, instead of the highly theoretical or scoped down practical examples we'd been given up until that point.

The robot we were lent for the project was a MIRTE master, an open source low cost educational robot developed by the TU Delft. It's equipped with a variety of sensors, like a 2D LiDAR, depth camera, IMU, an arm+gripper, and mecanum wheels. A solid base.

After picking up the robot, the first task was to see if we could remotely view the sensor data and start actuating some of the motors. This is where problems started to arise. The camera would be sluggish and not work reliably over the network, the onboard Orange Pi would crash, and the sensors would sometimes stop publishing. To get this fixed, it took a lot of walking back and forth to the MIRTE team to ask what was happening (which was often out of our control). You see, the MIRTE Master was in active development at the time, so our group was being used as beta testers. This ultimately meant a lot of unpredictable bugs, integration errors, or misconfigured ROS paramaters that had to get fixed as the project went along (valuable time lost 😞)

All the hassle of messing around with these camera sensors, IMU's and LiDAR got me to the point of thinking:

And that's what got this project started. The drifting IMU, the potato quality camera, and the hours of debugging not understanding why it wouldn't just work.

I thought, why, if everyone has such a capable device in their pocket at all times, did we have to be limited to this third party hardware that for some frustrating reason kept crashing? If my iPhone has such a great processor, camera, IMU, LiDAR sensor, just let me use that instead! And make it easy to use, so that I can get to the fun stuff instead of just debugging my hardware.

My 3 Requirements

This led me to setup my requirements for this project:

1. It must be "Plug and Play" (from iPhone -> ROS 2)
2. It must have minimal delay or jitter in the transfer of data (ideally wired)
3. It must be able to send depth and camera data (at least)

These three points are what make this project different from others with a similar goal. If you lookup phone sensor bridges for ROS, they usually fall short on one of them.

For example, most existing solutions will exclusively use a wireless connection, which works well for small amounts of data but once you have to send images, depth and other data can add severe delays. Another limitation is that solutions are web based, which limits the access to the iPhone's sensors APIs. This prevents access to Depth data, and limits the specific configuration of sensor outputs.

Where to begin?

I've never built an iOS app before and although I have a bit of experience programming, I am by no means a software developer. That's why I have to give a big shoutout to Claude Code for helping bring my ideas to life. The app obviously won't be properly optimized and there will be some inherent software design faults, but my initial idea is to get this working to see if anybody would actually want to use it (so then why bother to make a "perfect" app?)

SensorStream

After making some prototypes and taking inspiration from some other tethered depth streaming iOS apps, I'm happy to say the app is in a usable state, albeit with some small issues. And I've called it SensorStream.

SensorStream allows you to stream the following data:

• Camera (Rear Wide Lens, 1080p, 30FPS)
• Depth Map (320x180, 30FPS)
• IMU (Up to around 100Hz)
• GPS (Approx 1Hz max, depending on connection)

The screenshots below show some examples of the UI of the app. Each sensor has a live preview window and some configurable parameters.

Unfortunately, looking back at my 3 requirements, the app isn't fully "Plug and Play" once plugged into the computer. It still requires a "bridge", which takes the data from the app and publishes it to ROS 2 topics.

I did however see someone make an app with a similar premise which didn't explicitly require a custom bridge, instead relying on a Zenoh router. I really like this idea as it just feels like a more elegant solution. I might decide to implement something similar later on, but for now I will just keep the app the way it is.

I have a small demo which takes the IMU, Depth, and Camera data and projects it as a Pointcloud and transforms the reference frame based on the IMU orientations. You can see the GIF below how the pointcloud looks.

Where to go from here

There's still a long way to go in optimizing, adding features, and getting the app on the App Store. I don't even know if all the features I have adhere to Apple's App Store guidelines. From what I understand the approval process can be a hassle (and expensive, with the required $99 annual developer account subscription).

But that aside, the things I'd like to implement moving forward are:

• Support multiple devices (more than 1 phone connected)
• Pointclouds directly from iPhone (instead of reconstructing on computer)
• Some numerical tests to validate connection speed / delays / max bandwidth over USB/ etc.
• Add on device processing (offloading computation from the "main" ROS device to the iPhone)

For now, I'll continue to work on the app and hopefully be able to give people a copy on TestFlight to test the app and work out some issues. Also, let me know if anyone is actually waiting for such an app and what their use cases would be, what other features they would want, because I would love to know!

To conclude, I know the MIRTE robot wasn't at fault and this was just an excuse to make a fun little side project, but I'd like to try and go back to the MIRTE and use this (and maybe give it to students so they too could use it in future projects!)