Visibility flag on Keypoint Detector Object

Hello everyone,

I was wondering if anyone could help me understanding how to set up the training data for KeyPoint Detector Object (trainHRNETObjectKeypointDetector).

The help says:

"Keypoint locations, defined in spatial coordinates as an N-by-2 or N-by-3 numeric matrix with rows of the form [x y] or [x y v], respectively, where:

• N is the number of keypoint classes.
• x and y specify the spatial coordinates of a keypoint.
• v specifies the visibility of a keypoint."

I was wondering how to setup the visibility flag v .

Is it 1 when the key point is visible and annotated and 0 when it is not. Or same as Yolo 0 = key point out of view and not labelled; 1 = key point present but not visible; 2 = key point visible ?

Thank you very much for your help

Answer 1

[Vivek Akkala]

As mentioned in the doc, A value of 1 (true) indicates a valid keypoint and 0 (false) indicates an invalid keypoint.

[Marc]

Hi Vivek,

Thanks a lot for your help.

I indeed saw the documentation for the Detect function and tried implenting the same logic to the trainHRNetObjectKeypointDetector. Unfortunately I received the following error message. Also see below the code used.

Error using trainHRNetObjectKeypointDetector>iValidateData (line 222)

Expected input to be nonzero.

Error in

trainHRNetObjectKeypointDetector (line 27)

iValidateData(keypointDetector, trainingData);

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

%Code given in the trainHRNetObjectKeypointDetector function documentation
downloadFolder = tempdir;
dataset = helperDownloadHandPoseDataset(downloadFolder); %with helperDownloadHandPoseDataset as defined in the documentation
data = load(dataset);
handPoseDataset = data.handPoseDataset(1:100,:);
handPoseDataset.imageFilename = fullfile(downloadFolder,"2DHandPoseDataAndGroundTruth","2DHandPoseImages",handPoseDataset.imageFilename);
handPoseImds = imageDatastore(handPoseDataset.imageFilename);

%---Modification to add random visibility flag 0 or 1 to the original training data
trainingData = handPoseDataset(:,2);
keypoints = table2cell(trainingData);
for i = 1:100;
    TD = keypoints{i,1};
    extraDat = [];
    for j = 1:21;
        if rem(j,2) == 1;
            extraDat(j,1) = 0;
        else;
            extraDat(j,1) = 1;
        end;
    end;
    keypoints{i,1} = [TD extraDat];
end;
keypoints = table(keypoints);
%---

handPoseArrds = arrayDatastore(keypoints);
handPoseBlds = boxLabelDatastore(handPoseDataset(:,3));
trainingData = combine(handPoseImds,handPoseArrds,handPoseBlds);
keypointClasses = ["forefinger3","forefinger4","forefinger2","forefinger1", ...
    "middleFinger3","middleFinger4","middleFinger2","middleFinger1", ...
    "pinkyFinger3","pinkyFinger4","pinkyFinger2","pinkyFinger1", ...
    "ringFinger3","ringFinger4","ringFinger2","ringFinger1", ...
    "thumb3","thumb4","thumb2","thumb1","wrist"]';
handKeypointDetector = hrnetObjectKeypointDetector("human-full-body-w32",keypointClasses);
options = trainingOptions("adam", ...
    MaxEpochs=20, ...
    InitialLearnRate=0.001, ...
    MiniBatchSize=16, ...
    LearnRateSchedule="piecewise", ...
    LearnRateDropFactor=0.1, ...
    LearnRateDropPeriod=12, ...
    VerboseFrequency=25, ...
    BatchNormalizationStatistics="moving", ...
    ResetInputNormalization=false);
[trainedHandKeypointDetector,info] = trainHRNetObjectKeypointDetector(trainingData,handKeypointDetector,options);

[Andreas]

Hey Marc, I hope all is well. For the past week I've also ben trying to make the visibility work and I keep getting errors. Did you end up figuring out how to solve it? Or did you use a different method all together? Thanks!

[Vivek Akkala]

Marc, you don’t need to modify the training data to include a visibility flag. Currently, trainHRNetObjectKeypointDetector does not expect a visibility flag, which is why you encounter an error when a keypoint value of 0 is present in the training data. You can train the network directly using the available keypoints without the visibility information. During inference, the network applies a confidence threshold internally. Keypoints with scores below this threshold are discarded, and the remaining keypoints are treated as the visible ones.

[Vivek Akkala]

@Andreas, Could you please share more details about the issues you’re currently facing? Specifically, what do you mean by "making the visibility work"?

[Andreas]

I want to use HRNet to identify keypoints on aircraft, similar to how it is used in the available example with the spacecraft. However, in my dataset not all images have all keypoints labelled, as I want to have a detector that is trained in a variety of angles. Thus I need a way to handle the fact that in some images some keypoints are not visible.

The "trainHRNetObjectKeypointDetector" documentation states the following:

"Keypoint locations, defined in spatial coordinates as an N-by-2 or N-by-3 numeric matrix with rows of the form [x y] or [x y v], respectively, where:

• N is the number of keypoint classes.
• x and y specify the spatial coordinates of a keypoint.
• v specifies the visibility of a keypoint."

Thus, I assumed that I indeed can include images where not all keypoints are visible, however when I try to have a 7x3 matrix in my case, where visible points are assigned a 1 and not visible points a 0, I get a very similar error to Marc above.

Is this just a case of me defining visibility wrong, or is the documentation wrong and you can't train the detector unless all of your images have all of the keypoints labelled?

Thanks!

[Vivek Akkala]

Thanks for the clarification, @Andreas. You can include images where not all keypoints are visible. In such cases, you can assign a very small value (e.g., 1e‑5 or 1e‑4) to the non‑visible keypoints. Ideally, setting them to 0 should work, but it appears that trainHRNetObjectKeypointDetector is not configured to accept zeros. Using a small non‑zero value for missing keypoints should help avoid error mentioned by Marc.

[Marc]

Hi @Andreas,

I have been indeed to find workarounds and have trained my model. This has however been quite a trial-error process for me and here are a few of my obsersations:

1- The 2 main files that were responsible of my issues were: hrnetObjectKeypointDetector.m and trainHRNetObjectKeypointDetector.m, both in the folder matlabroot\...\toolbox\vision\vision

the function hrnetObjectKeypointDetector is called around line 100 in trainHRNetObjectKeypointDetector as the minibatchfcn argument

2- In the trainHRNetObjectKeypointDetector, under the iValidateData function, you can see that the verification below is made on the keypoints including a potential third visibility column (data{2}, being the keypoints table in your datastore)

validateattributes(data{2}{:,:}{:},{'numeric'},{'finite','real','nonzero','positive'})

This means that:

a. If the third column in your keypoint table is a visibility flag, with a 0/1 value, it will trigger an error message

b. If some keypoints are missing in a given frame, they still need to have a valid set of coordinates. A NaN, negative or [0 0] set of coordinates will also trigger an error.

A [1 1] set of coordinates (or upper left corner of your bounding box if using one) could work. However, this can be problematic when training the model. If there is a repetitive and characteristic enough pattern in upper left corner of your camera field of view/bounding box, this will most likely been seen as a valid feature and could greatly impact the quality of the detection.

Otherwise, but not sure it is authorised, another option could be to modify the iValidateData function, commenting that line, and save the trainHRNetObjectKeypointDetector file as a custom function in your own CD. You would however need to be 100% certain that your datastore is completely correct (still no NaN or 0 values), but this would at least allow you to pass a 0/1 flag column to the minibatch function

3- In the hrnetObjectKeypointDetector file, under the preprocessHRNetForTraining function, you can see that a heatmat is generated for every points, regardless of whether or not there is a visibility column. This means that even the points with dummy coordinates are being processed, which as stated above, could lead to training errors. I haven't done a complete test of it, but it should be possible here to modify this part of the function (again if authorised) by reading the flag value and replacing the dummy coordinates by NaN values.

So in summary, if this doesn't have any side effect for training your model, the easiest way, but somehow a still bit uncomfortable, is to allocate dummy but real coordinates to the non-visible keypoints as the visibility column is not taken into account and could trigger errors. If allocating dummy coordinates isn't a solution for you, the only option remaining is to have a visibility column and modify the 2 functions as describe above.

One more time, my apologies is some of these suggestions are not permitted by copiright rules and fingers crossed this will be address in a future release. Otherwise hope my observations are accurate and this helps you