iPhone Voice Controller

Explore how to totally control your iPhone programmatically via voice commands.

Newer mobile phones support voice assistive systems (Ref. 1). APPLE iPhone Voice Control leverages Siri's voice recognition system to provide fine-grained hands-free control of the entire phone at user-level. Voice Control makes possible operation of the phone in a pose that might make manual operation inconvenient or impossible, when for example the phone is face down on a table with the rear camera looking skyward.

This script explores how to generate and use voice commands for programmatic iPhone operation. Similar methods could be applied to Android devices. An example use is the collection of images with iPhone Camera app at arbitrary times with full control over camera settings, and simultaneous collection of sensor, bearing, and other datasets with Sensor Log app (Ref 2). (MATLAB Mobile provides only limited support for image and sensor data collection and no support for managing phone applications.) Use of voice control allows a laptop and iPhone (on an inexpensive tripod) to function as an automated system for astrophotography or a similar process of photo-documentation such as artwork creation. For astrophotography, the sensor data provide phone orientation information, and the image EXIF data (Ref. 3) provides local and universal time as well as GPS, image provenance, and camera setting information. iCloud provides automatic background transfer of images to the laptop and the sensor data can be transferred programmatically to a server or sent via email to the user for analysis. Using voice control, the phone need not be mechanically disturbed, maximum native single camera exposure times (30s) are possible, and immediate sequential exposures may be collected and combined by registration and stacking for still longer exposures.

As is illustrated in the image above, the state of the iPhone screen showing Voice Control command prompts may be viewed remotely on a laptop, and the entire operational history of the phone can be recorded as a movie, by connecting the iPhone to the laptop and using QuickTime App. Thusly, one need not operate the phone blindly (QuickTime App requires a cable connection but 3rd party apps support Bluetooth or WIFI.) and, unlike the use of mirroring, one retains access to the entire laptop.

Voice Control supports a wide variety of system commands (Ref 4) (e.g. 'Go Home' and 'Open Camera' ) and provides names for every control feature in any app. It also supports an item-number-based overlay reference system which is in some respects simpler and used here. In addition, one can record on an iPhone custom command sequences and assign them custom command names as will be described. One can simultaneously use Voice Control and Siri. While the entire dictionary of Siri and Voice Control commands might be translated to voice (Ref. 5) for remote control, we content ourselves with a few examples.

This script may interest students, educators, and others looking for a low-cost mobile phone based data acquisition system or just to have fun, and the script illustrates MATLAB audio manipulation techniques Try this suggestions and Challenges are included for further exploration. Hyperlinks to documentation are provided.