Thanks Andrea for the answer. Here is my code and how I used it.
I created a new shortenFileName method in org.dspace.app.xmlui.utils.XSLUtils
public static String shortenFileName(String string, String middle, int targetLength) { targetLength = Math.abs(targetLength); if (string != null && string.length() > targetLength) {
and then used it in item-view.xsl as follows:
<xsl:value-of select="util:shortenFileName(mets:FLocat[@LOCTYPE='URL']/@xlink:title, ' … ', 20)" />
The file name now looks like this:

source share