Eye-tracking algorithm HELP

Hello all :)

I was wondering if anyone could help my team mate and I trouble shoot this code we are trying to run. We are trying to develop an eye-tracking system using a combination of the Timm-Barth and Viola-Jones algorithm. We are trying to modify it to read images live from a cam rather than a saved file, but it keeps on giving an error. In the following line: "imgOrig = imread('img');". We have made an img variable, which is a snapshot from the cam, but it is not being read.

Does anyone suggestions for how to fix it? We are new to MATLAB so any help would be appreciated!

Thank you.

| |

clear all
cam = webcam()
while true % Infinite loop to continuously detect the face

      vid=snapshot(cam);  %get a snapshot of webcam
      vid = rgb2gray(vid);    %convert to grayscale
      img = flip(vid, 2); % Flips the image horizontally

*imgOrig = imread('img');*
G = fspecial('gaussian', [5 5],2);
imgOrig = imfilter(imgOrig,G,'same');
greyScaleImg = rgb2gray(imgOrig);
[gx,gy] = gradient(double(greyScaleImg));
greyScaleImg = 255-greyScaleImg;
greyScaleImg = im2double(greyScaleImg);
[rmax, cmax] = size(greyScaleImg);
gMag = zeros(rmax,cmax);

for y = 1:rmax;
    for x = 1:cmax;
       gMag(y,x) = sqrt((gx(y,x) * gx(y,x)) + (gy(y,x) * gy(y,x)));
    end
end

%Compute the threshold
stdMagGrad = std2(gMag);
meanGMagrad = mean2(gMag);
gradThreshold = 0.3 * stdMagGrad + meanGMagrad;

%Normalise gradients above threshold to unit length, zero everything else
for y = 1:rmax;
    for x = 1:cmax;
        if gMag(y,x) > gradThreshold;
        gx(y,x) = gx(y,x) / gMag(y,x);
        gy(y,x) = gy(y,x) / gMag(y,x);
        else 
            gx(y,x) = 0.00;
            gy(y,x) = 0.00;
        end
    end
end

sqrdDp = zeros(rmax,cmax);
wSqrdDp = zeros(rmax,cmax);

for y = 1:rmax;
    for x = 1:cmax;
        if gMag(y,x) > gradThreshold;
            for cy = 1:rmax;
                for cx = 1:cmax;
                    dx = x-cx;
                    dy = y-cy;
                    magnitude = sqrt((dx * dx) + (dy * dy));
                    dx = dx / magnitude;
                    dy = dy / magnitude;
                    dotProduct = dx * gx(y,x) + dy * gy(y,x);
                    dotProduct = max(0.0,dotProduct);
                    sqrdDp(cy,cx) = sqrdDp(cy,cx) + dotProduct * dotProduct;
                end
            end
        else 
            continue;
        end
    end
end

for y1 = 1:rmax;
    for x1 = 1:cmax;
        wSqrdDp(y1,x1) = sqrdDp(y1,x1) * I2(y1,x1);
    end
end

[maximum, I] = max(wSqrdDp(:));
threshold = 0.9 * maximum;

for y1 = 1:rmax;
    for x1 = 1:cmax;
        if wSqrdDp(y1,x1) < threshold;
            wSqrdDp(y1,x1) = 0.00;
        end
    end
end

[col, row] = ind2sub(size(wSqrdDp),I);
imgOrig = insertMarker(imgOrig, [row, col]);
imshow(imgOrig)
end||

Answer 1

[Image Analyst]

You need to call getsnapshot() instead of imread().

[Nada]

We are still getting an error. When we try specifying the cam as the object.

"Undefined function 'getsnapshot' for input arguments of type 'webcam'."

Just to see if it worked also tried using the variable defined as 'img' as the object. Still has an error:

 "input arguments of type 'uint8'"

[Image Analyst]

Oh, sorry - that's for the Image Acquisition Toolbox which I use. It looks like you're useing the webcam support package. So you already have a snapshot in "img" variable. So to save it in imgOrig, simply do:

imgOrig = img; % Make copy of variable img

[Tasnima Rahman]

Hi Image Analyst,

Thanks for your replies! We have tried the method you have suggested and it now seems to run. However, we are trying to get an output from the code.

I want to display a figure, so have tried using "figure, imshow(imgOrig)" but it does not produce an output. Is my syntax incorrect? I also do not seem to be able to get it to produce any of the values in the command window, when I remove a semi-colon to prevent it from being suppressed.

When I try going through it line-by-line it seems to get stuck around line 78. Do you possibly know why? Perhaps if you were to run it would the same thing happen?

Many thanks.

clear all
cam = webcam()

while true % Infinite loop to continuously detect the face

      vid = snapshot(cam);  %get a snapshot of webcam
      img = flip(vid, 2); % Flips the image horizontally

imgOrig = img;
G = fspecial('gaussian', [5 5],2);
imgOrig = imfilter(imgOrig,G,'same');
greyScaleImg = rgb2gray(imgOrig);
[gx,gy] = gradient(double(greyScaleImg));
greyScaleImg = 255-greyScaleImg;
greyScaleImg = im2double(greyScaleImg);
[rmax, cmax] = size(greyScaleImg);
gMag = zeros(rmax,cmax);

for y = 1:rmax;
    for x = 1:cmax;
       gMag(y,x) = sqrt((gx(y,x) * gx(y,x)) + (gy(y,x) * gy(y,x)));
    end
end

%Compute the threshold
stdMagGrad = std2(gMag);
meanGMagrad = mean2(gMag);
gradThreshold = 0.3 * stdMagGrad + meanGMagrad;

%Normalise gradients above threshold to unit length, zero everything else
for y = 1:rmax;
    for x = 1:cmax;
        if gMag(y,x) > gradThreshold;
        gx(y,x) = gx(y,x) / gMag(y,x);
        gy(y,x) = gy(y,x) / gMag(y,x);
        else 
            gx(y,x) = 0.00;
            gy(y,x) = 0.00;
        end
    end
end

sqrdDp = zeros(rmax,cmax);
wSqrdDp = zeros(rmax,cmax);

for y = 1:rmax;
    for x = 1:cmax;
        if gMag(y,x) > gradThreshold;
            for cy = 1:rmax;
                for cx = 1:cmax;
                    dx = x-cx;
                    dy = y-cy;
                    magnitude = sqrt((dx * dx) + (dy * dy));
                    dx = dx / magnitude;
                    dy = dy / magnitude;
                    dotProduct = dx * gx(y,x) + dy * gy(y,x);
                    dotProduct = max(0.0,dotProduct);
                    sqrdDp(cy,cx) = sqrdDp(cy,cx) + dotProduct * dotProduct;
                end
            end
        else 
            continue;
        end
    end
end

%figure, imshow(imgOrig)

for y1 = 1:rmax;
    for x1 = 1:cmax;
        wSqrdDp(y1,x1) = sqrdDp(y1,x1) * I2(y1,x1);
    end
end

[maximum, I] = max(wSqrdDp(:));
threshold = 0.9 * maximum;

for y1 = 1:rmax;
    for x1 = 1:cmax;
        if wSqrdDp(y1,x1) < threshold;
            wSqrdDp(y1,x1) = 0.00;
        end
    end
end

[col, row] = ind2sub(size(wSqrdDp),I);
imgOrig = insertMarker(imgOrig, [row, col]);
figure, imshow(imgOrig)
end

[Image Analyst]

What kind of "output" were you expecting? What is happening instead?

[Tasnima Rahman]

We expect the code to output an image after processing the image taken from the webcam. This should happen repeatedly because there is an infinite loop resulting in live feedback of the eye being tracked. However, there are some issues affecting this.

I have made a couple of modifications to the code. I have now chosen to use the Viola-Jones cascade detector to detect the face, then detect the eye from a cropped image of the face (with the same detector). I then made a cropped image of the eye to process this section only. I want to use the Timm-Barth algorithm to locate the pupil centre, so have chosen to grey scale the image, and having a smaller image to process is easier which is why I have cropped the eye. This can be confirmed to be working by using imshow (see line 51).

The first issue seems to be that it gets stuck in the loop between lines 81 and 100. I know that it processes each pixel until the entirety of the image has been processed, but it seems to continue cycling through even when cmax has been reached (it goes back to zero and up again). But this doesn't always happen. Sometimes it gives this error:

Index exceeds matrix dimensions.
Error in TimmBarth_Cam_Integration (line 42)
   eyeImage = imcrop(faceImage, bboxeye(1,:));

But how can it only exceed the matrix dimensions sometimes? Placing a marker variable in the middle of the loop shows how it is processed repeatedly hundreds of thousands of times. How can this loop be shortened? I have used a cropped image... Is it supposed to behave this way do you think?

Once this has been passed, the code should provide the coordinates of the centre of pupil (as far as my understanding of the code goes). I assume this is cx and cy, but I am not sure. We need this output to then calculate the change in eye direction if we take a reference coordinate axes (was hoping to use the edge of the Viola-Jones detector as I have seen this done elsewhere). Do you think the cx and cy variables are the centre coordinates? Do you think my method is feasible?

Beyond this, the code also requests an image to be outputted at the end with a marker indicating the centre (I was trying to understand if [row col] indicates the centre) - see lines 118 and 119. I assumed that because it gets stuck on the loop that it cannot output this yet which is why it never displays.

Along with all of this, I realise that although I have an infinite loop placed on the detection processing it does not continually provide live updated image outputs. Do you know how I may be able to add this?

I apologise for the very long description and multiple questions! We are just trying to understand this and also make it work well. I was wondering if you would be able to speak with me directly? There is a lot to ask and I find that it is sometimes easier to speak with someone directly. If you are happy to speak we could use a chat server, for example Team Speak, Ventrilo or Skype. Please let me know and we can make arrangements.

%NOTE: This is run using the Viola-Jones cascade detector - this detector does not
%      work with greyscale images. 

clear all
cam = webcam()

detector = vision.CascadeObjectDetector(); %Create a detector for face using Viola-Jones
detector1 = vision.CascadeObjectDetector('LeftEye'); % Create detector for a single eye

detector1.MergeThreshold = 200;

while true %Infinite loop to continuously detect the face
    vid = snapshot(cam);  %Get a snapshot of webcam
    img = flip(vid, 2); %Flips the image horizontally

     %imgOrig = img;
     G = fspecial('gaussian', [5 5],2); %We can change the size of G by changing [5 5] (maybe smaller = [3 3]?)
     img = imfilter(img,G,'same');
     %greyScaleImg = rgb2gray(img); %%Image was greyscaled later on so that
     %                                  viola-jones can detect the rgb image.
     %[gx,gy] = gradient(double(greyScaleImg));
     %greyScaleImg = 255-greyScaleImg;
     %greyScaleImg = im2double(greyScaleImg);

     bboxFace = step(detector, img); % !img has been changed from grey! %Creating bounding box around face on grey scaled face image using detector 
       if ~ isempty(bboxFace)  %If face exists (~ means opposite OR "is not")
           biggest_box=1;     
           for i=1:rank(bboxFace) %Find the biggest face
               if bboxFace(i,3)>bboxFace(biggest_box,3)
                   biggest_box=i;
               end
           end
           faceImage = imcrop(img,bboxFace(biggest_box,:)); %!CHANGED! % extract the face from the image
           bboxeye = step(detector1, faceImage); % Locations of the left eye using detector
           %eyeInFace = insertObjectAnnotation(faceImage,'rectangle',bboxeye,'Eye'); %Not needed? %Produces image of face with bbox around eye
           %imresize(eyeInFace,0.3);
           %imshow(eyeInFace)  % Only need final image to be outputted
       else
           %imshow(faceImage)  % Only need final image to be outputted
       end

     eyeImage = imcrop(faceImage, bboxeye(1,:));  

     greyScaleImg = rgb2gray(eyeImage); %I put greyscaling onto the eye image so that Timm-Barth can compute better
     [gx,gy] = gradient(double(greyScaleImg));
     greyScaleImg = 255-greyScaleImg;
     greyScaleImg = im2double(greyScaleImg); 
     [rmax, cmax] = size(greyScaleImg);
     gMag = zeros(rmax,cmax);

     imshow(greyScaleImg) %Just a test to see if eye image is cropped and grey - can output image later.

     % This is the Timm-Barth section:

     for y = 1:rmax;
         for x = 1:cmax;
             gMag(y,x) = sqrt((gx(y,x) * gx(y,x)) + (gy(y,x) * gy(y,x)));
         end
     end

     %Compute the threshold
     stdMagGrad = std2(gMag);
     meanGMagrad = mean2(gMag);
     gradThreshold = 0.3 * stdMagGrad + meanGMagrad;
     %Normalise gradients above threshold to unit length, zero everything else
     for y = 1:rmax; %It detects the edge and computes 
         for x = 1:cmax;
             if gMag(y,x) > gradThreshold;
                 gx(y,x) = gx(y,x) / gMag(y,x);
                 gy(y,x) = gy(y,x) / gMag(y,x);
             else
                 gx(y,x) = 0.00;
                 gy(y,x) = 0.00;
             end
         end
     end

     sqrdDp = zeros(rmax,cmax);
     wSqrdDp = zeros(rmax,cmax);

     for y = 1:rmax;
         for x = 1:cmax;
             if gMag(y,x) > gradThreshold
                 for cy = 1:rmax % Detects the centre
                     for cx = 1:cmax
                         dx = x-cx; %Calculates distance between the edge and the centre.
                         dy = y-cy;
                         magnitude = sqrt((dx * dx) + (dy * dy));
                         dx = dx / magnitude;
                         dy = dy / magnitude;
                         dotProduct = dx * gx(y,x) + dy * gy(y,x);
                         dotProduct = max(0.0,dotProduct);
                         sqrdDp(cy,cx) = sqrdDp(cy,cx) + dotProduct * dotProduct;
                     end
                 end
             else
                 continue
             end
         end
         x
     end
     y
     for y1 = 1:rmax;
         for x1 = 1:cmax;
             wSqrdDp(y1,x1) = sqrdDp(y1,x1) * I2(y1,x1);
         end
     end
     [maximum, I] = max(wSqrdDp(:));
     threshold = 0.9 * maximum;
     for y1 = 1:rmax;
         for x1 = 1:cmax;
             if wSqrdDp(y1,x1) < threshold;
                wSqrdDp(y1,x1) = 0.00;
             end
         end
     end
     [col, row] = ind2sub(size(wSqrdDp),I);
     BullsEye = insertMarker(img, [row, col]); %Puts a plus sign on the specified location ([row, col] - is this centre??)
     imshow(BullsEye)
  end

[Amir Dehsarvi]

Any updates on this? IS your toolbox working now and is it available online?