Main Content

Human Activity Recognition Using ESP32 Board and LSM9DS1 IMU Sensor

This example uses:

Open Live Script

This example shows how to classify and predict one of three different physical human activities: Sitting, Standing and Walking based on data acquired using ESP32 board and lsm9ds1. This example uses Classification Learner (Statistics and Machine Learning Toolbox) app to train a machine learning model with the sensor data. To access all the files for this example, click Open Live Script and download the attached files.

Required Hardware

  • ESP32-WROOM-DevKitV1

  • LSM9DS1 IMU sensor

Required MathWorks® Products

  • MATLAB®

  • Statistics and Machine Learning Toolbox

  • Signal Processing Toolbox

  • MATLAB Support Package for Arduino Hardware

Hardware Setup

  • Configure ESP32 hardware for WiFi based workflow. For more information, see Connection over Wi-Fi.

  • Connect SCL and SDA of ESP32 to SCL and SDA of LSM9DS1 IMU.

  • Connect the ground and 3.3V pin of ESP32 to ground and VDD of LSM9DS1 IMU.

  • Place the entire setup in a pocket with sensor facing away from you.

Capture Dataset

You can either use Sitting_raw_data.mat, Standing_raw_data.mat and Walking_raw_data.mat dataset corresponding to the three different types of physical activities: Sitting, Standing and Walking or capture the data manually.

To use the dataset, click Open Live Script and download Sitting_raw_data.mat, Standing_raw_data.mat and Walking_raw_data.mat files.

To capture the data manually, place the setup in your pocket with the sensor facing away from you. Click the checkbox for iscollectData and start capturing the data.

For this example, the dataset contains 3200 samples, as defined by the variable noOfSamples, for each of the three physical human activities. Once the prompt '*** Collecting data for activity = ' Sitting ' ***'' appears on the screen, start capturing data for the Sitting activity. Continue capturing the Sitting data until a new prompt appears, asking you to capture the data for Standing and Walking activities, respectively. At the end of data collection, you will have Sitting_raw_data.mat, Standing_raw_data.mat and Walking_raw_data.mat dataset corresponding to the three different types of physical activities.

% Check this if you want to manually collect the data
iscollectData = false;

% Create activity list
activityList = {'Sitting' , 'Standing', 'Walking'};

% Collect data for all activities
noOfSamples = 3200;
if iscollectData    
    for idx = 1:length(activityList)
        collectData(activityList{idx},noOfSamples);
    end
end

Feature Extraction

Choose a buffer size based on the required granularity of the extracted features. For this example, set bufferSize to 32.

The extractFeatures function is used to extract features from the data acquired using the accelerometer on the LSM9DS1 IMU sensor. The extracted features are stored in the variable features. To use the extractFeatures function, click Open Live Script and download the extractFeatures.m file.

The activityIDList contains the IDs corresponding to the physical activities where activity ID 1 corresponds to Sitting, 2 corresponds to Standing, and 3 corresponds to Walking.

% Initialize buffer size
bufferSize = 32;

features =[];
activityIDList = [];

% Extract features
for idx = 1: length(activityList)
    data = load([activityList{idx} '_raw_data.mat'],'data');
    data = data.data;

    % Calculate the number of groups
    numGroups = floor(size(data,1) / bufferSize);
    for i = 1:numGroups
        startIndex = (i-1) * bufferSize + 1;
        endIndex = i * bufferSize;
        xBuffer = data(startIndex:endIndex,1);
        yBuffer = data(startIndex:endIndex,2);
        zBuffer = data(startIndex:endIndex,3);
        features = [features; extractFeatures(xBuffer, yBuffer, zBuffer)];
    end
    
    activityIDList = [activityIDList;ones(numGroups,1)*idx];
end

Model Training

Use the Classification Learner app to train a classification model. To open the Classification Learner app, type classificationLearner at the MATLAB command line.

Alternatively, in the Apps tab click the arrow at the right of the Apps section to open the gallery. Then, under Machine Learning and Deep Learning, click Classification Learner.

In the app, click the New Session button, and then click From Workspace.

In the Data set pane, click the arrow under Data Set Variable, and then select features. Select From Workspace under Response and select activityIDList. The default option or the Validation Scheme is 5-fold cross-validation, which protects against over-fitting.

Click Start Session.

Classification Learner app loads the data set and plots a scatter plot of the first two features.

On the app toolstrip, in the Train section, click Train All and select Train Selected. When the training is complete, the Models pane displays the 5-fold, cross-validated classification accuracy. For this example the accuracy is 99.3%

To see the Validation Confusion Matrix, in the Plots and Results section of the app toolstrip, click Confusion Matrix (Validation).

On the app toolstrip, in the Export section, and then click Export Model.

In the Export Classification Model dialog box, clear the check box to exclude the training data and export a compact model, and then click OK.

The structure trainedModel appears in the MATLAB® workspace. Save the model using following command.

save('trainedModel.mat','trainedModel');

Prediction

Run this MATLAB code to predict the activity.

% Create an arduino object for ESP32 board over WiFi
aObj = arduino;

% Create IMU sensor object
imuObj = lsm9ds1(aObj);

% Predict the activity for three minutes
tic;
while(toc < 180)
    % Initialize buffer and buffer size
    xBuffer = [];
    yBuffer = [];
    zBuffer = [];
    bufferSize = 32;

    % Get buffered sensor values
    for idx = 1:bufferSize
        % Read the sensor values and calibrate
        accelerationValues = readAcceleration(imuObj);

        % Calibrate the acceleration values
        xBuffer(end+1) = accelerationValues(1);
        yBuffer(end+1) = accelerationValues(2);
        zBuffer(end+1) = accelerationValues(3);
    end

    % Extract features
    features  = extractFeatures(xBuffer', yBuffer', zBuffer');

    % Predict activity
    y = predictActivity(features);
    disp(['Detected activity = ' activityList{y}]);
end
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting
Detected activity = Sitting