Why are Java characters using only a Java character to support Asian characters, even if they are not?

Question

When visualizing a chart with JFreeChart, I noticed a layout problem when chart category labels included Japanese characters. Although the text is rendered with the correct glyphs, the text was placed in the wrong place, apparently because the font metrics were incorrect.

The chart was originally configured to use the Source Sans Pro Regular font for this text, which only supports Latin character sets. The obvious solution is to link the real Japanese .TTF font and ask JFreeChart to use it. This works great because the correct glyphs are used in the output text, and it also decomposes correctly.

My questions

• How did java.awt end up displaying Japanese characters in the first script when using an original font that actually supports nothing but Latin characters? If that matters, I'm testing OS X 10.9 with JDK 1.7u45.

• Is there a way to visualize Japanese characters without tying a separate Japanese font? (This is my final goal!) Although the bundling solution works, I don’t want to add 6 MB of bloat to my application if it can be avoided. Java clearly knows how to display Japanese glyphs somehow even without a font (at least in my local environment) - these, apparently, are only those indicators that were bankrupt. I am wondering if this is related to the "frankenfont" problem below.

• After the JRE performs an internal conversion, why does the Source Sans Pro font tell the caller (via canDisplayUpTo () ) that he can display Japanese characters, even if this is not possible? (See below.)

Edited to clarify:

• This is a server application, and the text that we display will be displayed in the client’s browser and / or in the PDF export. Charts are always rasterized to PNG on the server.

• I have no control over the operating system or server environment, and just as well as using standard Java platform fonts, many platforms have poor font options that are unacceptable in my use case, so I need to bundle my own (at least for latin fonts). Using a platform font for Japanese text is acceptable.

• An application may be asked to display a mixture of Japanese and Latin text without prior knowledge of the type of text. I am ambiguous about which fonts will be used if the string contains mixed languages, if the glyphs are displayed correctly.

More details

I understand that java.awt.Font # TextLayout is smart and that when trying to lay out text, it first asks for the main fonts to see if they can actually display the provided characters. If not, then he apparently swaps another font that knows how to display these characters, but this does not happen here, based on my debugging pretty far from the JRE classes. TextLayout#singleFont always returns a non-zero value for the font, and it goes through the fastInit() constructor.

It's very curious to note that the Source Sans Pro font is somehow forced to tell the caller that he knows how to display Japanese characters after the JRE performs the font conversion.

For example:

 // We load our font here (download from the first link above in the question) File fontFile = new File("/tmp/source-sans-pro.regular.ttf"); Font font = Font.createFont(Font.TRUETYPE_FONT, new FileInputStream(fontFile)); GraphicsEnvironment.getLocalGraphicsEnvironment().registerFont(font); // Here is some Japanese text that we want to display String str = "クローズ"; // Should say that the font cannot display any of these characters (return code = 0) System.out.println("Font " + font.getName() + " can display up to: " + font.canDisplayUpTo(str)); // But after doing this magic manipulation, the font claims that it can display the // entire string (return code = -1) AttributedString as = new AttributedString(str, font.getAttributes()); Map<AttributedCharacterIterator.Attribute,Object> attributes = as.getIterator().getAttributes(); Font newFont = Font.getFont(attributes); // Eeek, -1! System.out.println("Font " + newFont.getName() + " can display up to: " + newFont.canDisplayUpTo(str)); 

The result of this:

 Font Source Sans Pro can display up to: 0 Font Source Sans Pro can display up to: -1 

Please note that the three lines of “magic manipulation” mentioned above are not my own actions; we pass the true source font object to JFreeChart, but it attracts the JRE when drawing glyphs, and this means that the three lines of the "magic manipulation" code above are repeated. The above manipulation is the functional equivalent of what happens in the following sequence of calls:

• org.jfree.text.TextUtilities # drawRotatedString
• sun.java2d.SunGraphics2D # lace
• java.awt.font.TextLayout # (constructor)
• java.awt.font.TextLayout # singleFont

When we call Font.getFont () on the last line of the magic manipulation, we still get the Source Sans Pro font, but the main font2D font font2D is different from the original font, and this single font now claims to know how to display the whole string. What for? It seems that Java is returning to us some kind of "frankenfont" that knows how to display all kinds of glyphs, although it only understands the metrics for glyphs that come in the original source font.

A more complete example showing a JFreeChart rendering example is given here based on one of the JFreeChart examples: https://gist.github.com/sdudley/b710fd384e495e7f1439 The result from this example is shown below.

Example with font Source Sans Pro (incorrectly specified):

Example with the Japanese IPA font (correctly displayed):

Answer 1

I finally figured it out. There were a number of main reasons that were further prevented by the added dose of cross-platform variability.

JFreeChart renders text in the wrong place because it uses a different font object

The layout problem arose because JFreeChart inadvertently calculated metrics for the layout using a different Font object than the one that AWT actually uses to render the font. (For reference, JFreeChart is calculated in org.jfree.text#getTextBounds .)

The reason for the other Font object is the result of the implicit “magic manipulation” mentioned in the question, which is executed inside java.awt.font.TextLayout#singleFont .

These three lines of magical manipulation can be summarized as follows:

 font = Font.getFont(font.getAttributes()) 

In English, this asks the font manager to provide us with a new Font object based on the "attributes" (first name, last name, dot size, etc.) of the supplied font. Under certain circumstances, the Font that it returns to you will be different from the Font that you started from the beginning.

To fix the metrics (and thus fix the layout), to fix it, you need to run the one-line above on your own Font object before installing the font in the JFreeChart objects .

After that, the layout worked fine for me, as did the Japanese heroes. It should also fix the layout for you, although it may not show you Japanese characters. Read below about native fonts to see why.

Mac OS X Font Manager prefers to return the original fonts, even if you download a physical TTF file

The layout of the text has been fixed by the above change ... but why is this happening? Under what circumstances does FontManager actually return a different type of Font object to us than the one we provided?

There are many reasons, but at least on Mac OS X, the reason for the problem is that the font manager seems to prefer to return the native fonts whenever possible.

In other words, if you create a new font from a physical TTF font called "Foobar" using Font.createFont , and then call Font.getFont () with attributes derived from your physical Foobar font ... for so long since OS X already has Foobar font, the font manager will return the CFont object, not the TrueTypeFont object that you expected. This seems to persist even if you register the font through GraphicsEnvironment.getLocalGraphicsEnvironment().registerFont RegisterFont.

In my case, it threw the red herring into the investigation: I already had the font "Source Sans" installed on my Mac, which meant that I got different results from people who did not.

Mac OS X Native Fonts Always Support Asian Characters

The bottom line is that Mac OS X CFont always support Asian character sets. I don’t understand what kind of mechanism allows this, but I suspect that this is some kind of font backup function of OS X itself, not Java. In any case, a CFont always pretends (and indeed can) display Asian characters with the correct glyphs.

This allows us to understand the mechanism that allowed us to create the original problem:

• we created a physical Font from a physical TTF file, which in itself does not support Japanese.
• the same physical font as above was also installed in my Mac OS X font book
• when calculating the chart diagram, JFreeChart specified the Font physical object for the Japanese text metrics. Physical Font could not do this correctly because it does not support Asian character sets.
• when drawing a chart, the magic manipulation in TextLayout#singleFont forced him to get a CFont object and draw a glyph using a native font with the same name compared to the physical TrueTypeFont . Thus, the glyphs were correct, but they were incorrectly positioned.

You will get different results depending on whether you have a font registered and if you have a font installed on your operating system

If you call Font.getFont() with the attributes from the generated TTF font, you will get one of three different results depending on whether the font is registered and if you have the same font installed:

• If you have do font installed on your native platform with the same name as your TTF font (regardless of whether you registered this font or not), you will receive Asian CFont support for the desired font.
• If you registered a TTF Font in a GraphicsEnvironment but don’t have a native font of the same name, calling Font.getFont () will return a TrueTypeFont physical object. This gives you the font you want, but you won’t get Asian characters.
• If you have not registered TTF Font , and you also have your own font with the same name, calling Font.getFont () will return Asian CFont, but it will not be the requested font.

Looking back, none of this is surprising. Leading:

I accidentally used the wrong font

In the production application, I created the font, but I forgot to register it first in the GraphicsEnvironment. If you have not registered the font while performing the magic manipulations above, Font.getFont() does not know how to get it, and instead you will get a backup font. Unfortunately.

On Windows, Mac, and Linux, this fallback font is typically Dialog, which is a logical (composite) font that supports Asian characters. At least in Java 7u72, the default Dialog font has the following fonts for Western alphabets:

• Mac: Lucida Grande
• Linux (CentOS): Lucida Sans
• Windows: Arial

This error was actually good for our Asian users because it meant that their character sets looked as expected with a logical font ... although western users did not get the character sets we wanted.

Since it was rendering in the wrong fonts, and we still needed to fix the Japanese layout, I decided that I better try to standardize one common font for future releases (and thus come closer to the trashgod suggestions).

In addition, the application has font quality rendering requirements that may not always allow the use of certain fonts, so a reasonable solution seemed to be trying to configure the application to use Lucida Sans, which is one physical font that is included by Oracle in all Java instances. But...

Lucida Sans doesn't play well with Asian characters on all platforms

The decision to try Lucida Sans seemed reasonable ... but I quickly found that there were differences in the platform how Lucida Sans is processed. On Linux and Windows, if you ask for a copy of the "Lucida Sans" font, you will get a TrueTypeFont physical object. But this font does not support Asian characters.

The same applies to Mac OS X if you request “Lucida Sans” ... but if you ask for a slightly different name “LucidaSans” (note the lack of space), you will get a CFont object that supports Lucida Sans, as well as Asian characters so you can get your cake and eat it too.

On other platforms, the query "LucidaSans" gives a copy of the standard Dialog font, because there is no such font, and Java returns by default. On Linux, you're in luck because Dialog actually uses Lucida Sans for western text by default (and also uses a decent replacement font for Asian characters).

This gives us a way to get (almost) the same physical font on all platforms and which also supports Asian characters by requesting fonts with these names:

• Mac OS X: "LucidaSans" (inferior to Lucida Sans + Asian fallback fonts)
• Linux: Dialog (Lucida Sans output + Asian fallback fonts)
• Windows: Dialog (Arial + Asian backup fonts are displayed)

I looked at the fonts.properties files on Windows, and I could not find the font sequence that was the default for Lucida Sans, so it looks like our Windows users will have to get stuck with Arial ... but at least it is not which is visually different from Lucida Sans, and the quality of rendering Windows fonts is reasonable.

Where did it all end?

In general, now we are almost just using platform fonts. (I'm sure @trashgod has a laugh now!) Both Mac and Linux servers get Lucida Sans, Windows gets Arial, the rendering quality is good, and everyone is happy!

Answer 2

Although it does not address your question directly, I thought that it could serve as a useful starting point to show the result using a standard platform font in an unvarnished diagram. A simplified version of BarChartDemo1 , source , is shown below.

Due to the vagaries of third-party font metrics, I try to avoid deviations from standard logical fonts , which are selected based on a locale supported by the platform. Logical fonts are mapped to a physical font in configuration files. On Mac OS, the corresponding file is located in $JAVA_HOME/jre/lib/ , where $JAVA_HOME is the result of evaluating /usr/libexec/java_home -v 1.n and n is your version. I see similar results with both version 7 and 8. In particular, fontconfig.properties.src defines the font used to provide variants of Japanese font families. All the mappings seem to use MS Mincho or MS Gothic .

 import java.awt.Dimension; import java.awt.EventQueue; import org.jfree.chart.ChartFactory; import org.jfree.chart.ChartPanel; import org.jfree.chart.JFreeChart; import org.jfree.chart.plot.PlotOrientation; import org.jfree.data.category.CategoryDataset; import org.jfree.data.category.DefaultCategoryDataset; import org.jfree.ui.ApplicationFrame; import org.jfree.ui.RefineryUtilities; /** * @see http://stackoverflow.com/a/26090878/230513 * @see http://www.jfree.org/jfreechart/api/javadoc/src-html/org/jfree/chart/demo/BarChartDemo1.html */ public class BarChartDemo1 extends ApplicationFrame { /** * Creates a new demo instance. * * @param title the frame title. */ public BarChartDemo1(String title) { super(title); CategoryDataset dataset = createDataset(); JFreeChart chart = createChart(dataset); ChartPanel chartPanel = new ChartPanel(chart){ @Override public Dimension getPreferredSize() { return new Dimension(600, 400); } }; chartPanel.setFillZoomRectangle(true); chartPanel.setMouseWheelEnabled(true); setContentPane(chartPanel); } /** * Returns a sample dataset. * * @return The dataset. */ private static CategoryDataset createDataset() { // row keys... String series1 = "First"; String series2 = "Second"; String series3 = "Third"; // column keys... String category1 = "クローズ"; String category2 = "クローズ"; String category3 = "クローズクローズクローズ"; String category4 = "Category 4 クローズ"; String category5 = "Category 5"; // create the dataset... DefaultCategoryDataset dataset = new DefaultCategoryDataset(); dataset.addValue(1.0, series1, category1); dataset.addValue(4.0, series1, category2); dataset.addValue(3.0, series1, category3); dataset.addValue(5.0, series1, category4); dataset.addValue(5.0, series1, category5); dataset.addValue(5.0, series2, category1); dataset.addValue(7.0, series2, category2); dataset.addValue(6.0, series2, category3); dataset.addValue(8.0, series2, category4); dataset.addValue(4.0, series2, category5); dataset.addValue(4.0, series3, category1); dataset.addValue(3.0, series3, category2); dataset.addValue(2.0, series3, category3); dataset.addValue(3.0, series3, category4); dataset.addValue(6.0, series3, category5); return dataset; } /** * Creates a sample chart. * * @param dataset the dataset. * * @return The chart. */ private static JFreeChart createChart(CategoryDataset dataset) { // create the chart... JFreeChart chart = ChartFactory.createBarChart( "Bar Chart Demo 1", // chart title "Category", // domain axis label "Value", // range axis label dataset, // data PlotOrientation.HORIZONTAL, // orientation true, // include legend true, // tooltips? false // URLs? ); return chart; } /** * Starting point for the demonstration application. * * @param args ignored. */ public static void main(String[] args) { EventQueue.invokeLater(() -> { BarChartDemo1 demo = new BarChartDemo1("Bar Chart Demo 1"); demo.pack(); RefineryUtilities.centerFrameOnScreen(demo); demo.setVisible(true); }); } }