I found an answer to my own problem, thanks to my cluster admin, and @Walter Roberson. And the solution turned out to be not exactly blocking a file from the search path, but adding the system dll / so files to the LD_PRELOAD environment variable before running matlab. So I will change the question title from
"How to block a file from MATLAB search path" to "How to use system libcrypto.so.1.1 instead of MATLAB libcrypto.so.1.1 to avoid error symbol EVP_KDF_ctrl version OPENSSL_1_1_1b not defined in file libcrypto.so.1.1"
Here's the workaround solution that worked for me:
in the linux terminal, do:
before the MATLAB run.
What this does, as it seems to me, is that it prioritizes libraries that it specifies, which is exactly what I needed.