Java.lang.OutOfMemoryError: GC upper limit exceeded excel reader

I get java.lang.OutOfMemoryError: exceeding the GC limit exceeded the exception when I try to run the program below. This main program method gains access to the specified directory and iterates over all files containing .xlsx. This works great when I tested it before any other logic. And the method that it calls, xlsx, which basically converts the xlsx file to csv and adds it to an existing file, works fine. But when I put this in a for loop, this is when I get this exception. I suppose this is a conflict, when after he opened xlsx and converted it to csv and its time to open the second, maybe I need to somehow close this line:

File inputFile = new File("C:\\Users\\edennis.AD\\Desktop\\test\\"+nameOfFile); 

This is my only guess right now when this file is interfering, when the second iteration of the loop comes. I am using Apache POI libraries to manage excel files. Thanks at Advance!

 import java.io.File; import java.io.FileInputStream; import java.io.FileNotFoundException; import java.io.FileOutputStream; import org.apache.poi.ss.usermodel.Cell; import org.apache.poi.ss.usermodel.Row; import org.apache.poi.xssf.usermodel.XSSFSheet; import org.apache.poi.xssf.usermodel.XSSFWorkbook; public class ExcelMan { public static void main(String[] args) throws FileNotFoundException { int i =0; File dir = new File("C:\\Users\\edennis.AD\\Desktop\\test\\"); for (File child : dir.listFiles()) { //initializing whether the sheet sent to method is first or not, and //counting iterations for each time the for loop as run boolean firstSheet = true; i++; String nameOfFile = child.getName(); if (nameOfFile.contains(".xlsx")){ System.out.println(nameOfFile); if (i != 0) firstSheet = false; File inputFile = new File("C:\\Users\\edennis.AD\\Desktop\\test\\"+nameOfFile); // writing excel data to csv File outputFile = new File("C:\\Users\\edennis.AD\\Desktop\\test\\memb.csv"); xlsx(inputFile, outputFile, firstSheet); } // } } } static void xlsx(File inputFile, File outputFile, boolean firstSheet) { // For storing data into CSV files StringBuffer data = new StringBuffer(); try { FileOutputStream fos = new FileOutputStream(outputFile, true); // Get the workbook object for XLSX file XSSFWorkbook wBook = new XSSFWorkbook(new FileInputStream(inputFile)); // Get first sheet from the workbook XSSFSheet sheet = wBook.getSheetAt(7); Row row; Cell cell; // Iterate through each rows from first sheet java.util.Iterator<Row> rowIterator = sheet.iterator(); while (rowIterator.hasNext()) { if (firstSheet != true) rowIterator.next(); row = rowIterator.next(); // For each row, iterate through each columns java.util.Iterator<Cell> cellIterator = row.cellIterator(); while (cellIterator.hasNext()) { cell = cellIterator.next(); switch (cell.getCellType()) { case Cell.CELL_TYPE_BOOLEAN: data.append(cell.getBooleanCellValue() + "^"); break; case Cell.CELL_TYPE_NUMERIC: data.append(cell.getNumericCellValue() + "^"); break; case Cell.CELL_TYPE_STRING: data.append(cell.getStringCellValue() + "^"); break; case Cell.CELL_TYPE_BLANK: data.append("" + "^"); break; default: data.append(cell + "^"); } } data.append("\r\n"); } fos.write(data.toString().getBytes()); fos.close(); } catch (Exception ioe) { ioe.printStackTrace(); } } } 

Additional Information:

Below is the stacktrace

  MR.xlsx Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at org.apache.xmlbeans.impl.store.Cur$CurLoadContext.attr(Cur.java:3039) at org.apache.xmlbeans.impl.store.Cur$CurLoadContext.attr(Cur.java:3060) at org.apache.xmlbeans.impl.store.Locale$SaxHandler.startElement(Locale.java:3250) at org.apache.xmlbeans.impl.piccolo.xml.Piccolo.reportStartTag(Piccolo.java:1082) at org.apache.xmlbeans.impl.piccolo.xml.PiccoloLexer.parseAttributesNS(PiccoloLexer.java:1802) at org.apache.xmlbeans.impl.piccolo.xml.PiccoloLexer.parseOpenTagNS(PiccoloLexer.java:1521) at org.apache.xmlbeans.impl.piccolo.xml.PiccoloLexer.parseTagNS(PiccoloLexer.java:1362) at org.apache.xmlbeans.impl.piccolo.xml.PiccoloLexer.parseXMLNS(PiccoloLexer.java:1293) at org.apache.xmlbeans.impl.piccolo.xml.PiccoloLexer.parseXML(PiccoloLexer.java:1261) at org.apache.xmlbeans.impl.piccolo.xml.PiccoloLexer.yylex(PiccoloLexer.java:4808) at org.apache.xmlbeans.impl.piccolo.xml.Piccolo.yylex(Piccolo.java:1290) at org.apache.xmlbeans.impl.piccolo.xml.Piccolo.yyparse(Piccolo.java:1400) at org.apache.xmlbeans.impl.piccolo.xml.Piccolo.parse(Piccolo.java:714) at org.apache.xmlbeans.impl.store.Locale$SaxLoader.load(Locale.java:3439) at org.apache.xmlbeans.impl.store.Locale.parseToXmlObject(Locale.java:1270) at org.apache.xmlbeans.impl.store.Locale.parseToXmlObject(Locale.java:1257) at org.apache.xmlbeans.impl.schema.SchemaTypeLoaderBase.parse(SchemaTypeLoaderBase.java:345) at org.openxmlformats.schemas.spreadsheetml.x2006.main.WorksheetDocument$Factory.parse(Unknown Source) at org.apache.poi.xssf.usermodel.XSSFSheet.read(XSSFSheet.java:138) at org.apache.poi.xssf.usermodel.XSSFSheet.onDocumentRead(XSSFSheet.java:130) at org.apache.poi.xssf.usermodel.XSSFWorkbook.onDocumentRead(XSSFWorkbook.java:286) at org.apache.poi.POIXMLDocument.load(POIXMLDocument.java:159) at org.apache.poi.xssf.usermodel.XSSFWorkbook.<init>(XSSFWorkbook.java:207) at ExcelMan.xlsx(ExcelMan.java:71) at ExcelMan.main(ExcelMan.java:47) 

The excel files are quite large, about 30 or so in the directory, and the largest is about 170 MB, with these file sizes, should I change with the POI?

+7
java
source share
2 answers

What is the size of your excel file? I once had a similar problem creating csv from xls . In my case, I had to switch to an event driven model , take a look at XSSF and SAX (Event API). I also ran out of memory (with -Xmx8g )

Quote from the linked site:

Further HSSF efforts will focus on the following main areas:

  • Performance: POI currently uses large memory for large sheets.
+3
source share

Files do not need to be closed. As long as you do not maintain references to them, they will be GCd, as they go beyond.

The string if (i != 0) will always be evaluated as true, since you increment the variable i at least once before pushing this condition. Thus, firstSheet is always false.

Line

 File inputFile = new File("C:\\Users\\edennis.AD\\Desktop\\test\\"+nameOfFile); 

creates new files. However, you already have an object file for this path represented by child

You always write to the same file that you recreate for the file object and the new FileOutputStream for each time you iterate over the source directories, even if all the entries belong to the same file.

You do not close your FileOutputStream in the finally block, and it may not be right to close the FileOutputStream under error conditions.

Use StringBuilder instead of StringBuffer if you don't need synchronized methods to build a string.

Consider using FileWriter instead of an intermediate StringBuilder. Instead of writing in Builder, use

 PrintWriter writer = new PrintWriter(new BufferedWriter(new FileWriter(outputFile, true)))) 

Instead of using data.append use writer.print or writer.println The PrintWriter and Buffered Writer wrappers are not strictly necessary, but useful.

If you refer to the javadocs XSSFWorkbook for constructor options, you will see that β€œUsing InputStream requires more memory than using a file, so if the file is available, you should instead do something likeβ€œ example follow ” http: // poi .apache.org / apidocs / org / apache / poi / xssf / usermodel / XSSFWorkbook.html # XSSFWorkbook (java.io.InputStream)

Increasing the size of your heap is likely to be a workable solution if all else fails. Assuming you have no potential for significantly larger files than the ones you are currently testing with. Increase heap size in Java

+1
source share

All Articles