WebGL Interface for Mobile Devices (for Gazebo) – that’s my project for OPW!
My background is in robotics with a focus on mechanics though, so basically everything about this project is new to me, which is very challenging and exciting (I might be saying overwhelming in a few weeks). Let’s start with an overview.
[I’ve been learning only recently, so please forgive me and correct me if I say something wrong 🙂 ]
Gazebo is a free open source robotics simulator partly supported by the Open Source Robotics Foundation (OSRF) – I’m doing my internship with them. Gazebo allows the simulation of multi-robotic systems in various environments with realistic physics. It interfaces very well with the Robotics Operating System (ROS) and runs on Linux.
Gzweb is a client for Gazebo which runs on a web browser. Being a client means that Gzweb provides visualization, but is not performing the main physics computations, which are done by gzserver. Ian Chen, the main Gzweb developer, is one of my mentors during the program. Gzweb is based on WebGL and up to now has been developed with desktop computers in mind. My job is to adapt Gzweb for mobiles in the 3 months to come.
Why web?
Native applications have to be made specifically for a certain operating system, you know, Linux, Windows, MacOS, Android, iOS, and so on. When it comes to the web however, the major players happen to be coming to a consensus and agreeing on standards which result in cross-browser and consequently cross-device compatibility. Therefore, instead of writing native apps for each OS/device in different languages, you can write only one web app. Moreover, the web allows part of your code to be run on external servers, abusing less of the user’s device (but requiring a good connection).
Why mobile?
Mobile devices (phones, tablets…) are becoming a big part of our lives. Soon, you’ll be able to simulate your robotic systems on them too 🙂
Mobile vs desktop/laptop
As awesome as they may be, mobile devices are still more limited in terms of memory, Central Processing Unit (CPU) power and Graphics Processing Unit (GPU) power compared to our good old computers. Their screens can be much smaller and interaction is done mainly through touch screens. Therefore, not only the graphics performance must be optimized, the user interface must be adapted as well.
HTML is one of the web standards people agree on and HTML5 is its latest version. HTML5 brings a lot of new stuff. One of them is the canvas element, which allows graphics to be embedded on the page with JavaScript. Before, this kind of multimedia information depended on external plugins like Flash, which don’t interact well with the rest of the content in a web page. Another new thing which came along HTML5 is Web Socket, which allows for 2-sided communication between client and server (before, the server could only answer whatever the client asked). That’s a great improvement for real-time interactions. Gzweb makes use of canvas and Web Sockets.
WebGL (Web Graphics Library) is a low-level 3D graphics API (Application programming interface) which is used within HTML5‘s canvas. It is based on OpenGL ES 2.0 (Open Graphics Library for Embedded Systems), which is the standard for embedded accelerated 3D graphics. For example, the native apps on mobiles use OpenGL ES. Most browsers (including mobile ones) already support it to different extents. Things are moving so fast that by the end of this internship, maybe all browsers will fully support it.
WebGL support
WebGL is maintained by a consortium of the main browser vendors, including Mozilla, Google, Apple and Opera. However, support for WebGL is still limited in many browsers. I conducted a couple of experiments to check the performance of the current Gzweb version on several browsers. Recall that Gzweb is designed for desktops, so the interface is not supposed to work anyways. What I want to check is the rendering and how many frames per second (fps) can be rendered in different browsers and devices.
The devices and browsers used were:
A) Laptop: Intel Core i5-2410M CPU @ 2.30GHz × 4, Intel® Sandybridge Mobile graphics, Ubuntu 12.04
A1) Mozilla Firefox 24.0
A2) Chromium 28.0
A3) Gazebo (Native. Just for reference)
B) Samsung Galaxy S3: 1.4 GHz quad-core ARM Cortex-A9 CPU, ARM Mali-400 MP GPU, Android 4.1.2
B1) Android browser
B2) Chrome for Android 31.0
B3) Firefox for Android 26.0
C) Samsung Galaxy S4 Mini: 1.7 GHz dual-core Krait 300 CPU, Adreno 305 GPU, Android 4.2.2
C1) Android browser
C2) Chrome for Android 31.0
C3) Firefox for Android 26.0
1) Box moving towards wall: I ran an example from a tutorial which consists of a box with a ray sensor moving towards a wall with a velocity proportional to the distance to the wall. There are only 2 box objects in the scene and only one of them is moving.
2) Kitchen model: Just loading the ‘kitchen + dining room’ model which comes with Gazebo can be a tough task due to the number of objects and textures to be rendered.
The video below shows the results for both experiments, for all browsers. Keep in mind that for all experiments, gzserver was run in the same computer (device A), so the differences seen come from the WebGL side.
There’s a lot going on there. A few pointers:
- On Galaxy S3, only Firefox worked (Chrome says the graphics card doesn’t support WebGL and the Android browser just doesn’t render the canvas). The box example runs with less than 30 fps and the kitchen example with less than 10 fps, presenting a big lag.
- On Galaxy S4 all browsers were able to render the box example with more than 40 fps. For the kitchen example, Chrome barely responded and the other 2 browsers lagged with less than 10 fps.
- On the laptop, the box example worked well on both browsers, but the kitchen lagged on Firefox. The surprise is how poorly Gazebo worked in comparison, lagging a lot with the box and even crashing with the kitchen. There’s obviously some problem with my Gazebo at the moment, which luckily isn’t affecting gzserver.
Since Firefox presented the best performance for the phones, I’ll be using it from now on.
The path to mobile
This project consists of two main parts:
1) Optimizing graphics performance for mobiles
Both examples tested were very simple compared to what’s desired of a multi-robot complex-environment simulator. Thus it is necessary to optimize WebGL for mobiles, a lot. I’m currently reading about different techniques and will be talking about them in the coming posts.
2) Making an interface for mobiles
Once the graphics performance is acceptable, I will start working with my other mentor, Steffi Paepcke, on an interface suitable for mobiles.
In parallel to these two goals, I also plan to learn as much as I can about the workings of Gzweb and Gazebo and to contribute to them as well. I’ve actually learned a lot this week already, but this post is already getting too long, so let’s wait for the next one!
(The screen-shot on the top shows the mobile manipulator from this tutorial on Firefox for the Galaxy S3 😉 )
my Shumi 
Wow, it sounds like a really cool project!