How to get mouse position without a click event on appdesigner?

Franck paulin Ludovig pehn Mayo less than a minute ago

Basically, one part of my project is to record the mouse position in appdesigner on an electrovibration tactile screen. I have to use my finger (with 4 haptic motors embedded in a ring).

I am able to record the mouse position when using the mouse itself on the electrovibration tactile screen. But the problem i am facing is that when using my finger i have to tap/click in order to get any feedback from the UIapp and unfortunately it is not required.

I thought at first that it was an issue with the electrovibration tactile screen but figure it out later that no.(I did a test by using "Guide" instead of appdesigner and everything is going just perfectly . I dont need any tapping and i am getting feedback from the Gui.

Below is my main code. It is also attached

(Edited)

  properties (Access = public)
        arduinoObj % Description
        message
    end
    
    properties (Access = public)
        T % Description
        
    end
    
    properties (Access = public)
        
        region1
%         
        v_thick1 % Description
        v_thick2 % Description
        h_thick1 % Description
        h_thick2 % Description
        
        v_thickness_1
        v_thickness_2
        h_thickness_1
        h_thickness_2
        v_or_h_array
        v_or_h
        amplitude_array
        exp_counter
        
    end
    
  
    
    properties (Access = public)
        amplitude % Description
    end
    
    methods (Access = private)
        
      
   function mycallback(app,src,event)
   display(event.SelectedOption);
   end
   
    end
    function startupFcn(app)
    
     clc
            % Read experiment data from a CSV file
            
            
            % Plot patch on uiaxes
            hold(app.UIAxes, 'on')
            
           % region1 = patch(app.UIAxes,[-10 10 10 -10],[-5 -5 -4.4 -4.4],'r','FaceAlpha',1,...
              %'LineWidth',0.01,'LineStyle','-','tag','region1');
            
            load_folder = "C:\Users\student\Desktop\FRANCK\thesis\excel_data\";
            load_name   = "excel_data.xlsx";
            load_addr   = load_folder + load_name;
            
            app.T = readtable(load_addr,'NumHeaderLines',1);
            
            app.exp_counter = 1;
            app.v_thickness_1   = app.T.Var1;
            app.v_thickness_2   = app.T.Var2;
            app.h_thickness_1   = app.T.Var3;
            app.h_thickness_2   = app.T.Var4;
            app.amplitude_array = app.T.Var5;
            app.v_or_h_array    = app.T.Var6;
            
           app.v_thick1 = app.v_thickness_1(app.exp_counter);
           app.v_thick2 = app.v_thickness_2(app.exp_counter);
            
           app.h_thick1 = app.h_thickness_1(app.exp_counter);
           app.h_thick2 = app.h_thickness_2(app.exp_counter);
            
           app.v_or_h = app.v_or_h_array(app.exp_counter);
            
                      
            %Vertical line
            if app.v_or_h == 0
              app.region1 = patch(app.UIAxes, ...
                                [app.v_thick1 app.v_thick2 app.v_thick2 app.v_thick1], ...
                                [-10 -10 10 10],'r', ...
                                'FaceAlpha',1,...
                                'LineWidth',0.01, ...
                                'LineStyle','-','tag','region1'); 
            
            %Horizontal line
            elseif app.v_or_h == 1                
                app.region1 = patch(app.UIAxes,[-10 10 10 -10], ...
                                 [app.h_thick1 app.h_thick1 app.h_thick2 app.h_thick2], ...
                                 'r','FaceAlpha',1,...
                                 'LineWidth',0.01, ...
                                 'LineStyle','-','tag','region1');
            end
           
            % Define pointer behavior for patch
            pm.enterFcn = @(~,~) cursorPositionFeedback(app, app.region1, 'in');
            pm.exitFcn  = @(~,~) cursorPositionFeedback(app, app.region1, 'out');
            pm.traverseFcn = [];
            iptSetPointerBehavior(app.region1, pm)
            % Enable pointer manager for app
            iptPointerManager(app.UIFigure,'enable');
            
            % Create the Arduino serial object
            app.arduinoObj = serialport("COM3", 9600);
            configureTerminator(app.arduinoObj,"CR/LF");
            %flush(app.arduinoObj);
            % 
            for i=1:8 
                app.message = readline(app.arduinoObj);
                disp(app.message)
            end
function cursorPositionFeedback(app, hobj, inout)
% When inout is 'in', change hobj facecolor to green and update textbox.
% When inout is 'out' change hobj facecolor to red, and clear textbox.
% Check tag property of hobj to identify the object.
switch lower(inout)
    case 'in'
        facecolor = 'g';
        txt = 'Motor(s) vibrating';
        pointer = 'fleur';
        writeline(app.arduinoObj, "4&MOTOR_1_2&0!");
%         message = readline(app.arduinoObj);
%         disp(message)
    case 'out'
        facecolor = 'r';
        txt = 'No';
        pointer = 'crosshair';
        writeline(app.arduinoObj, "0&NO_MOTOR&0!");          %% THIS IS THE LINE THAT İS SUPPOSED TO STOP İT
%         message = readline(app.arduinoObj);
%         disp(message)
        
end
hobj.FaceColor = facecolor;
app.TextAreaEditField.Value = txt;
set(app.UIFigure, 'Pointer', pointer)
end  
   % Determine if mouse is within uiaxes
            cp = app.UIFigure.CurrentPoint;
            isInAxes = cp(1) >= app.UIAxes.Position(1) && ...
                cp(1) <= sum(app.UIAxes.Position([1,3])) && ...
                cp(2) >= app.UIAxes.Position(2) && ...
                cp(2) <= sum(app.UIAxes.Position([2,4]));
            if isInAxes
                set(app.CurrentPositionEditField, 'Value',...
                    sprintf('%.2f,  %.2f', app.UIAxes.CurrentPoint(1,1:2)))
            else
                set(app.CurrentPositionEditField, 'Value', '')
            end
            
    function NEXTButton_2Pushed(app, event)
            
            
            uiconfirm(app.UIFigure,'Are You sure?','Confirm Close',...
            'CloseFcn',@(src,event)mycallback(app,src,event));
            
            app.exp_counter = app.exp_counter + 1;
            app.v_or_h = app.v_or_h_array(app.exp_counter);
            
    
            if ishandle(app.region1)
               delete(app.region1);
           end
            
            %Vertical line
            if app.v_or_h == 0
                app.region1 = patch(app.UIAxes,...
                     [app.v_thick1 app.v_thick2 app.v_thick2 app.v_thick1],...
                     [-10 -10 10 10],'r',...
                     'FaceAlpha',1,...
                     'LineWidth',0.01,...
                     'LineStyle','-','tag','region1'); 
           
            %Horizontal line
            elseif app.v_or_h == 1                
                app.region1 = patch(app.UIAxes,[-10 10 10 -10],...
                     [app.h_thick1 app.h_thick1 app.h_thick2 app.h_thick2],...
                     'r','FaceAlpha',1,...
                     'LineWidth',0.01,...
                     'LineStyle','-','tag','region1');
            end
            
              
            % Define pointer behavior for patch
            pm.enterFcn = @(~,~) cursorPositionFeedback(app, app.region1, 'in');
            pm.exitFcn  = @(~,~) cursorPositionFeedback(app, app.region1, 'out');
            pm.traverseFcn = [];
            iptSetPointerBehavior(app.region1, pm);
            % Enable pointer manager for app
            iptPointerManager(app.UIFigure,'enable');
            
            % Create the Arduino serial object
            %app.arduinoObj = serialport("COM6", 9600);
            %configureTerminator(app.arduinoObj,"CR/LF");
            %flush(app.arduinoObj);
            % 
            for i=1:8 
                app.message = readline(app.arduinoObj);
                disp(app.message)
            end
            
            function cursorPositionFeedback(app, hobj, inout)
% When inout is 'in', change hobj facecolor to green and update textbox.
% When inout is 'out' change hobj facecolor to red, and clear textbox.
% Check tag property of hobj to identify the object.
switch lower(inout)
    case 'in'
        facecolor = 'g';
        txt = 'Motor(s) vibrating';
        pointer = 'fleur';
        writeline(app.arduinoObj, "4&MOTOR_1_2&0!");
%         message = readline(app.arduinoObj);
%         disp(message)
    case 'out'
        facecolor = 'r';
        txt = 'No';
        pointer = 'crosshair';
        writeline(app.arduinoObj, "0&NO_MOTOR&0!");             %% THIS IS THE LINE THAT İS SUPPOSED TO STOP İT
%         message = readline(app.arduinoObj);
%         disp(message)
        
end
hobj.FaceColor = facecolor;
app.TextAreaEditField.Value = txt;
set(app.UIFigure, 'Pointer', pointer)
end  

[Adam Danz]

Hi @Franck paulin Ludovig pehn Mayo, I removed the answer you added which just contained a callout to me so that your question appears in the unanswered list.

I won't have time to look into this until after this coming Tuesday (11th) but I can already tell this will be difficult to troubleshoot without being able to reproduce the problem on my end.

Just to be clear, you're saying that you're using the exact same copy-pasted code verbatum from the code block below in Guide and App designer, except for replacing the "app" input, correct? What do you have to click in App Designer? Have you tested that the cursorPositionFeedback function is evoked at all in AppDesigner?

pm.enterFcn = @(~,~) cursorPositionFeedback(app, app.region1, 'in');
pm.exitFcn  = @(~,~) cursorPositionFeedback(app, app.region1, 'out');
pm.traverseFcn = [];
iptSetPointerBehavior(app.region1, pm)
% Enable pointer manager for app
iptPointerManager(app.UIFigure,'enable');

% AND THE cursorPositionFeedback FUNCTION...

[Adam Danz]

By the way, it's better to attach your app than to copy-paste the code so we can immediately open it in AppDesigner.

[Franck paulin Ludovig pehn Mayo]

@Adam Danz Hello adam,

No , what i am trying to say is that i was using the code below in Guide. It is a completely different code that is perfectly working when using Guide. I did it as a test to see wheter the issue was comig from the electrovibration screen settings or coming from appdesigner. Unfortunately, there is no way i can transpose this code in appdesigner. I am using matlab 2021.

function figure1_WindowButtonMotionFcn(hObject, eventdata, handles)
pos = get(hObject, 'currentpoint'); % get mouse location on figure
global x;
global y;
x = pos(1); 
y = pos(2); % assign locations to x and y
set(handles.xloc, 'string', ['x loc:' num2str(x)]); % update text for x loc
set(handles.yloc, 'string', ['y loc:' num2str(y)]); % update text for y loc

[Adam Danz]

Why can't you use this code above in AppDesigner? Or were you refering to some other code that you need but can't implement in AppDesigner?

[Franck paulin Ludovig pehn Mayo]

@Adam Danz Well, i can't use it because it was designed for ''Guide''.

I took the code on the link below.

https://we15hang.blogspot.com/2012/01/matlab-gui-tracking-mouse-locations.html

I basically just need a code that will help tracking the mouse location using my finger and not the mouse itself. I was using a code that you suggested before but it is not working when using my finger . It is getting me lost. I am curious to know what could be the issue as well..

In a nutshell, all the code i have made need me to tap/click on the electrovibration tactile screen when using my finger in order to get any feedback but when using the mouse itself everything just works properly. I dont know if it is a problem coming from appdesigner because when using the "Guide" code(see previous link) everything is just working fine .(I mean when using my finger i am getting the feedback that i want).

[Adam Danz]

Could you provide load_addr and instruction how to use the app?

[Adam Danz]

load_addr is a variable your app produces. It's a filename.

load_folder = "C:\Users\Hp\Desktop\thesis\excel_data\";

load_name = "excel_data.xlsx";

load_addr = load_folder + load_name;

I'm asking that you attach that file or attach a mat file containing the table that is read in from

app.T = readtable(load_addr,'NumHeaderLines',1);

[Franck paulin Ludovig pehn Mayo]

@Adam Danz sorry. My mind was far. Didnt know you were talking about the excel file. My bad

[Adam Danz]

To test the app, I commented-out the arduino stuff in a few lines in startupFcn and the writeline lines in cursorPositionFeedback.

Then I tested the functionality of the pointer behavior and it looks OK to me other than inherent delays that cannot be reduced in MATLAB (GIF below, I resized the app to create a smaller GIF file). So this means the bottleneck or errors or inefficiencies, whichever is causing the problem, is happening in the arduino interface which I cannot troubleshoot. The cursorPositionFeedback function should be as simple and efficient as possible so that it runs as fast as possible.

Also, in your startupFcn, use load_addr=fullfile(load_folder,load_name) rather than load_folder+load_name.

[Franck paulin Ludovig pehn Mayo]

@Adam Danz I will try to rephrase my issue. I am getting the same output as you. This is not the issue i am facing now.

All this project is done on an electrovibration tactile screen where i have to use my finger with a ring around it(vibration motors embeded in the ring). The issue i am facing is that when using my finger i am not getting the same feedback instead i have to tap/click with my finger to get any feedback while when using the mouse itself as you did i am having perfect feeback.

So i thought it was the electrovibration screen issue but it is not coz i have a code we did i GUI (guide) and we are having perfect feedback with the finger . To check what i am saying is that, if your laptop has a tactile screen , instead of using your mouse , use one of your finger and try to see if the band is changing color or you are recording position .

For me, i have to tap on the screen it in order to get any feedaback at that particular spot. I was thinking then it might be a problem with appdesigner.

[Adam Danz]

I see, the touch-screen analogy was helpful to understand the problem. Unfortunately I do not have a touch screen so I cannot test it. I suggest writing to tech support. Make a copy of your app and take out any unnecessary components that are unrelated to the problem so that it just demonstrates the pointer behavior and feedback (bar color, text area). Explain that it works with the mouse but not with touch screen - if that's the case.

I'd be eager to know their response as well.

https://www.mathworks.com/support/contact_us.html

[Franck paulin Ludovig pehn Mayo]

@Adam Danz Thank you Adam i am waiting for their feeback and i will let you know!

[Franck paulin Ludovig pehn Mayo]

also, @s pernot helped me out with this code down saying

"i would suggest to use the following trick from Y. Altman to better tune your GUI app callbacks

several points :

• clock is 50x more efficient on my PC than datetime('now')... so use it or alternatively use cputime instead of datetime('now')
• callbacks keep being triggered so you must control the way they work and exit as soon as possible, you may also prevent reentry if necessary by using uiaxes Interactivity and BusyAction props or similar
• you must tune the refreshing frequency in the callback to match your responsiveness

i hope this will help"

My concern in which callbacks should i place them cos i am getting errors...After the writeline command of running the motor or in another place?

persistent lasttime
if isempty(lasttime)
    lasttime = clock;
end % if lasttime
if all(clock - lasttime >= [0 0 0 0 0 0.04])  % 25 Hz refreshing 
        % do your stuff here
        % xxxx
        % update for next callback entry
        lasttime = clock;
end 
% exit callback as soon as possible here

Answer 1

[Abhishek Chakram]

Hi Franck paulin Ludovig pehn Mayo,

It appears to me that you are facing difficulty in getting mouse position without a click event in the app designer. You can use “WindowButtonMotionFcn” callback for the same.

You can refer to the following video to know more about the “WindowButtonMotionFcn” callback: https://blogs.mathworks.com/videos/2019/08/19/saving-state-in-a-windowbuttonmotionfcn-callback/

Best Regards,

Abhishek Chakram