how to create a region of interest in simulink

I am trying to count the number of railroad ties in a video using simulink. The thought is to use sobel edge detection on each video frame. The railroad tie ends up being the background in the binary. Then use bottom-hat to apply a mask to the tie with an area say 15 pixels tall by 60 pixels wide. Then when the complete mask is in the region of interest it detects and counts the tie.

Attached is a frame from the video. I would like to create a rectangular region of interest x1=110, y1=105, x2=110, y2=75, x3=210, y3=75, x4=210, y4=105.

Does anyone know how to create a region of interest? I have looked at the examples in Matlab and I can't seem to figure it out.

Also, I am very new to Matlab/Simulink and not a computer scientist, vision/image scientist. So, if anyone has other suggestions I am very open to suggestions or if anyone thinks what I am doing can't be accomplished through this process then I am all ears as well.

Thanks

Answer 1

[Sean de Wolski]

If you have the Computer Vision System Toolbox then most of the functionality described by ImageAnalyst is available in Simulink blocks.

Alternatively you could do what he recommended by writing the code in MATLAB and then putting this in a MATLAB Function Block or an Interpreted function block if some of the code is not supported for code generation.

For this reason, I would recommend sticking with the CVST blocks:

more

Also, I would take a different approach than IA (live dangerously!). I would identify the right track using the same approach he did but then count the number of ties by identifying the rectangles with the bolts next to the right track and summing up the number of these rectangles.

[Kean]

I do have the computer vision system toolbox and already have a model I have been playing around with in simulink. However, I am new to both Simulink and Matlab. So some of the Simulink blocks don't quite make sense to me...all the parameters and etc you can specify. So, I am hoping to work in both matlab and simulink. By working in the code with Image Analyst I am figuring out what all these functions do and get a final solution. If I can then put it in a function block within simulink then great. But I am learning about the functions and therefore might be able to have a complete automated solution in simulink. But I don't have to have a solution in simulink.

I really appreciate all of this advice.

[Image Analyst]

But Sean, he said that he didn't actually need to count the number of ties per frame (though initially he thought he did). It turned out that what he really needs is the number of ties the train rolls over: "I want to count the ties between mile point 1 and mile point 2". So all we need to do is to recognize when one new tie enters the frame. If we counted them all, then we'd have to too many - like 5 times too many if there were 5 ties per frame, so we'd have to divide by 5 (approximately) to get the number of ties between mile marker 1 and mile marker 2.

[Sean de Wolski]

I missed the hidden comments. If we know how many ties are in a frame and we know how many frames there are per second, Can't we just multiply the

(frames/second)*(ties*/frame)*(seconds/mile)

[Sean de Wolski]

If you are looking for comparing specific frames, use a Unit Delay block in Simulink to compare to frames 1 sample apart looking for when the new tie enters and old ties leave.

Do you have a video of this you can share, Kean?

[Kean]

Here is a 30 second avi clip in zip format. It only shows 211 ties and is only approximately 6 percent of the mile. The issue with making any assumptions with ties per frame and frames per second is that the speed during the video recording was not kept constant though an effort was made to do that. Therefore, I would rather rely on a method that would "detect" and count a tie when it passes a certain part of the frame.

[Sean de Wolski]

The wife wants me to go home so I'll play with this more tomorrow, but here's a start on just comparing frames to each other to identify what's changed (about the simplest possible thing!)

Instead of this operation, you could instead use the operations described by IA, I'll look into this more tomorrow.

[Kean]

I will take a look. Thanks.

[Kean]

Here is what I have come up with so far using Simulink. It is very rough and I was trying out multiple things. But if I think what Image Analyst has talked about (see comments above Image Analyst) could be incorporated into this some how using CVST Blocks I am just not sure how to make it all work.

[Sean de Wolski]

Attached is a file I've thrown together kind of roughly. It finds 209 ties. There is clearly a lot more you could do to this to make it more sensitive to ties and to handle initial/final ties (which is possibly where the discrepancy is coming from). Most of this stuff would be what IA is talking about.

[Kean]

Sean...I haven't had a chance to look at this yet but I really appreciate it.

[Kean]

Sean I looked at the model and I have some questions about the process. First, the first few steps the output is set to uint then it is converted to double. Why not just output double? Second, in the interpolated matlab function the stdfilt(u,ones(9)) was used. Could you explain the "ones(9)"? This has something do to with the Nhood parameter but I am not sure I understand. Then, after the interpolated matlab function you have the relational operator with the constant value set to 5? I apologize for my ignorance with regards to this.

[Kean]

Sean...please see attached

I made some changes. Tell me what you think b/c I am still trying to understand the logic/processes. Here is my understanding of the way it works: Open avi -> change from RGB to intensity -> crop the frame so it only looks between the rails -> use stdfilt() to find standard deviation -> set the threshold to "find" locations and mark 0 or 1 -> mask the found locations -> sum across the rows -> if between rows 0 and 45 there is a mask then count 1 -> sum all counts

[Sean de Wolski]

Close!

1. read image in, convert to grayscale double
2. stdfilt to identify areas of low standard deviation within a 9x9 neighborhood (i.e. the compare to 5). This indicates tracks. All numbers were chosen by trial + error
3. dilate horizontally (rectangle [1x50]) because tracks are horizontal
4. erode vertically to remove noise between tracks
5. Sum each row (recieve a column vector)
6. if there are more than 100 true pixels in a row, it's a track (wooohoo!)
7. Only look at the first row (selector)
8. Identify when the first row goes from being a zero to a one (difference - compare to one).
9. Use the StopCallback to sum up how many ones there are, i.e. the total number of tracks and put this in the workspace as NTies.

Answer 2

[Image Analyst]

I don't know how to do it in Simulink, but if you can write custom MATLAB code in a Simulink block, what I'd do it first sum the image vertically and detect the two rails. Then get rid of those columns - remove them from the image. Then so some sort of texture filter such as stdfilt() or imgradient() to get a texture image. Then sum that horizontally. High sum regions will be gravel/rocks. Low sum regions will be railroad ties. Then just threshold and pass the binary vector into bwlabel to count the ties, or pass into regionprops if you want to determine how many ties and fractions of ties there are. Not too hard. Can you code that up? If not let me know and I'll help you.

[Kean]

The whole image isn't needed. What about cropping the image 130 pixels from each side? Would that take too much processing?

Forgive me if I don't completely understand but would your recommendation provide a total count of ties for the video? I have attached a short avi video in zip format. There are about 211 individual ties in the video. I would have to go back and recount to be sure. However, the tie count doesn't have to start at the first tie. I can set up the test procedure so I begin counting the first whole tie. Example: I want to count the ties between mile point 1 and mile point 2. I can position the camera so the center of the image is just short of the first tie then begin recording the video. Then stop recording when I have passed the last tie when I reach mile point 2.

To answer your question I don't know how to code it but would love to learn.

Thanks so much.

[Image Analyst]

My algorithm counts ties in each frame, not the whole video. You're going to have to do something else to get the whole video. If you just stitch one frame to the earlier and later adjacent frame, do you get a continuous image? Or are the jumps/breaks/discrepancies in it? Is there any tie, or part of a tie, that does not show up in any frame at all? Because there might be a small time lapse between frames where the ties moved along but didn't get an image snapped of them during that inter-frame time.

[Kean]

No you wouldn't get a continuous image. Each frame in the real world measures approximately 5 feet in the vertical. Each frame is only displaced about 4 to 6 inches in the vertical. Or if you see a tie in the bottom of the frame it takes about 12 frames for that tie to disappear out of the picture. I tried to record the video going about 5 to 10 miles per hour. But it was difficult to keep the speed constant. So, throughout the entire video the speed fluctuates a bit.

[Image Analyst]

Well that's the better situation than missing some part. Now to count frames you only need to increment the "tie count" when you have one complete (not partial) tie. And you only need to count the first complete tie that you see at the top of the frame. You do not need to count any below that because they've already been counted on prior frames. This simplifies the code a lot. You don't need to detect partial ties or keep track when a tie comes or goes. You simply need to count it when you have a complete tie, and the tie is in a different place than the prior frame (to account for aliasing). Of course there is a degenerate case where you're snapping a photo at the exact rate where the speed is such that it seems the ties don't move at all. I'm sure you've seen car commercials where it looks like the wheel spokes go backwards or stay still even though the car is moving forward.

[Kean]

That all sounds correct to me. I did not consider aliasing or degenerate. Not sure how to account for that unless I just make sure the video recording procedure is such that it is a non issue.

I am trying to figure out the best way to "detect" that first tie. I am trying to use edge detection and then apply a mask then track/count that mask when it was a in a certain part of the frame. However, due to the fact that the tie is exactly horizontal or fluctuations in the intensity b/c the tie color is not uniform all the time is making a consistent detection possible. BTW- I really appreciate this dialog. Thank you.

[Image Analyst]

Did you do the code to get horizontal and vertical profiles yet? The first step is to find the ties, then the second step is to track/label them. To do that you find the rails and just look inside the rails and crop off the left and right chunks of the image. Then do a texture filter and collape horizontally and look for high texture areas and low texture areas. It should look like a rectangle wave. Then find the low texture areas - those are the ties presumably. Did you do any of that yet?

[Kean]

I am still using Simulink so I don't have to code. I am not a strong programmer. I have cropped the image on the left and right. Now working on the texture filter.

[Kean]

Let me rephrase...I am coding for the example.jpg above in my first post using matlab and then trying to figure out a way to do that with simulink. I am not a strong programmer.

[Image Analyst]

I don't use Simulink and an not sure if you can have a block in Simulink where you execute MATLAB code. If you can, then I can continue to help you, otherwise there's no point.

[Kean]

I don't have to do this in simulink. What I learn from you is helping out a lot.

[Kean]

I used imgradient(), specifically prewitt method, on the image. What do you mean by collapse horizontally?

[Kean]

I have loaded the image, cropped it, changed it to grayscale, and have applied a gradient to it much like this

http://www.mathworks.com/help/images/ref/imgradient.html?searchHighlight=imgradient

When I show figure using montage in the example I get the figure like my 'Igradient.fig'. However, if I just look at Gmax I get the figure 'IGmax.fig" attached. Do you know why they look different. I know the in the Igradient file one side is the max and one is direction. But in the Gmax it is almost blank.

[Image Analyst]

Saving as PNG would be preferable and faster for us to view them.

[Kean]

Sorry about that. Ib.png is the cropped grayscale.

[Kean]

I used the Gmag from imgradient and summed accross each row. I plotted the results...see attached. The ties are the low spots. From the plot you can see that the first tie from the example image is between 40 to 60....so how do I threshold and then ask it to count when a tie passes thought a region of 30 to 70?

[Image Analyst]

It looks good! I'd threshold it around 3000 or 4000. Then you might have to do a little clean up in case you get tiny little extra peaks, like around x=100.

[Kean]

I typed threshold matlab help and it comes up with tons of results. Which function are you referring?

[Image Analyst]

No function, just the operation:

thresholdValue = 3500; % Whatever....
elementsAboveThreshold = signal > thresholdValue;

[Kean]

Here is what I have come up with so far based on our talks. So, when it passes from 0,0,0,0,0,0,0 to 1,1,1,1,1,1,1 it is going from tie to gravel. Nice. Now I need to count when it goes from 1 to 0. Hints?

[Kean]

It's a bit more tricky than counting when it goes from 1 to 0. Because it would be scanning every frame and making duplicates. Would it be possible to detect the 1 to 0 between say rows 30 to 70 and when it rises between rows 30 to 70 it counts a tie. Then make the same detection again between those rows? I am I thinking about this right?

[Kean]

IA i am stumped on how to proceed further. Can you offer up any more hints? thanks.

Answer 3

[Kean]

Sean...in step 6 is there a way to look at the sum of say 10 rows such that if the total was 1000 it would "detect" the tie?

[Sean de Wolski]

Sure, add another Element SUM block in before the comparison.

[Kean]

how do you stop the sum such that only 10 rows are added?

[Sean de Wolski]

Use a selector block to "select" the first 10 elements in the signal before the sum block

[Kean]

Sean...attached is what I have come up with. Not sure if it is doing what I intended by summing the column vector for the first "n" rows then if above a constant return 1. It just doesn't seem to be as sensitive to the that constant. I am getting the correct count of 209. I like your idea of cleaning up the signal using the dialate/erode but I wanted to try something different. Would you mind taking a quick look if I have the logic correct?