Read and detect font from RGB image

Hey,

I try to detect textures from objects by the help of a color changing sensor. Depending on the underlaying texture, the stress of the surface of my sensor changes and thereby the color changes (no stress = red, more stress = green, much stress = blue). As you can see in the uploaded imaged, the texture is more or less "clearly" visible. Now I want to write a code that creates a binary mask which shows the texture. In the example that you can see here, I want to detect the letters "Grol" in the same font as seen on the picture, so if everything would work out as planed I would have a binary mask with black pixels on a white background shaping the word "Grol" in the handwritten-looking font.

I tried different ways to realize this. One by converting the picture into RGB, HSV, or Grayscale and analyzing the edges of the individual image channels (R/G/B or H/S/V). By morphological operations (erosion, dilatation) I wanted to solve my problem. However, several edge detection algorithms (canny, sobel, roberts, prewitt, log, zerocross) did not end up in the results I wanted. Mostly only a part of a letter is recognized, sometimes none are detected. Even after playing around with the parameters of all algorithms (structuring elements, gradients,...), I was not able to solve the problem.

My second approach, to detect only a specific color space and binarize pixels as 1 or 0 belonging to the fact if they are in, or out of the colour space, did also not work. The problem hereby is, that for example the background color on the picture I uploaded is dark green, but also parts of the font (bottom of the letters, have that green tone).

My question now is, whether you know another way to analyze images? Is there a function, or algorithm to detect such textures reliabley?

I mean as a human I find it very easy to read the writing when I look at the picture. I could trace the binary mask very easily if asked to draw it over the image. If it's so easy for me, then there must be a feature that makes the writing stand out. Am I on the wrong track?

If there are some image processing geniuses here, I would be very happy about help and tips. Many thanks for the help in advance.

Best, Raphael

[Image Analyst]

To me the green swirls say "gioli" so I don't think it's as clear as you think. And even then, some of the letters are broken or incomplete. I think you best bet would be some kind of deep learning app. Getting the letters would be hard enough, forget about the font (Arial, Century, Calibri, etc.). I think getting the font would be impossible and I don't see why it would be necessary.

[DGM]

It's not clear to me how the sensor image represents stress. If stress could merely be described as a shift in hue from red to blue, then this would be the stress map:

... but there are regions where it seems obvious that the stylus would have stressed the sensor, but there's apparently an unrelated shift in color. Likewise, the entire (ostensibly) unstressed background of the image is shades of darker green and red, so what does the intensity of the color represent? Are the hues representative of axial strain components or something?

Is there some documentation for this device?

EDIT:

Oh wait a minute. I see what's going on here .. or at least part of it. That's not a written script; that looks like a Grolsch logo impressioned from a bottle cap or something. I was wondering why the R stroke was broken. Still, I don't know what the "sensor" is or how it's supposed to behave. It kind of looks like some sort of film that's just pressed against the part, but I'm not familiar with such a material. Knowing what the logo is makes it a bit easier to see its edges in the image, but the color relationship still makes no sense. It almost looks like its iridescent, or otherwise dependent on viewing angle.

[Raphael Pesch]

Hey @Image Analyst - How did you filter that image as you did. That looks way better than my filtering. If I would be able to create a mask with the quality you did, that would already help me a lot.

@Image Analyst @DGM:

Let me explain the device a little more. Sorry that I did not do it in before, I did not wanted to annoy you with the engineering details behind the imaging process. I am currently writing my Masters thesis at MIT in Boston and my supervisor developed this sensor. Therefore I am more or less one of the first person to implement it in working systems. The paper that explains the physics behin the device will be published in the next weeks, but to sum up the physics behind it briefly: It's working like a mirror that only reflects one wavelength. The more it is streched, the more the reflected wavelength shifts. Therefore the sonsor begins at red light and then shofts through the spectrum (IR -> red -> green -> blue -> UV -> ...). At the moment I try to implement the sensor to a soft robot gripper and detect the texture of objects while gripping it. And congratulations @DGM :D yes, I tried to grip a bottle of "Grolsch" and you see a picture of it in my example.

My goal is to be able to grip different bottles with different textures and by creating masks that show the texture (black where the texture sticks out and white where no texture), sort out which bottle is currently gripped. Of cause this later on could be adapted to different systems, but since I am german, I thought beginning with bottles of beer is a good idea ;-)

If you are more interested in that topic, you can watch a presentation from my supervisor that explains the device on YouTube https://www.youtube.com/watch?v=4KRE6h3sr84 (beginning at minute 36:29).

@Image Analyst It would be a huge help if ypu could explain to me how you masked the image that you uploaded.

[DGM]

So it really is strictly a matter of hue shift, and the intensity information is largely a matter of incidence.

Still, at least with this one image, I'm having a hard time getting a better extraction of the logo from the image. One of my points of attention has been the connecting stroke between the O and L. It's proving difficult to actually pick up that subtle, yet plainly visible detail. Some other areas are a bit difficult to manage due to the reflections of the surroundings.

That difficulty aside, the task of converting a tensile strain map to a binarized representation of the logo seems like it would be a challenge. The strain map is kind of a defacto map of edges and corners. If we consider an idealized edge map, how does one back-calculate the original image from its edge map? You might try to integrate or do something, but in this case, it's not really a consistent edge map to think that would work. Not all edges are highlighted. I'm not sure of a good approach. Then again, for the task of object discrimination, there may be a way to utilize the strain map without the need to actually fully recover the binarized logo. I'm not sure.

If you have a few other image examples, maybe I could poke around some more with those.

[Raphael Pesch]

@DGM First of all, thanks for still answering and trying to help. :-)

Of cause I can share some more pictures (I uploaded a bunch of example pictures in a dropbox file). Here is the link to them: DropBox Images (https://www.dropbox.com/scl/fo/q5hx2ke4kqu91ggttazu7/h?dl=0&rlkey=wfy2pf9myexxtgcmc5mrmpt6b). If you have time for it, feel free to play areound with them.

The reflections of the surroundings are hopefully soon no problem anymore. I am currently working on a process step that enables a anti reflection layer on the material, so that the white apperaing reflection won't/should,'t be a problem anymore.

Another idea than edge detection could be analyzing and filtering the picture with fourier transformation. However, I am have no good results with that approach so far. I still would be interested in how @Image Analyst was able to seperate the letters from the background so good as in his uploaded image. :/