Debugging java.lang.OutOfMemoryError issues
-
- Posts: 915
- Joined: Thu May 02, 2019 2:32 pm
Debugging java.lang.OutOfMemoryError issues
Post by chrispitude »
Code: Select all
[exec] [java] INFO LoggingEventListener - Rendered page #4202.
[exec] [java] INFO LoggingEventListener - Rendered page #4203.
[exec] [java] INFO LoggingEventListener - Rendered page #4204.
[exec] [java] Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
[exec] [java] at java.base/java.util.Arrays.copyOf(Arrays.java:3537)
[exec] [java] at java.base/java.io.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:185)
[exec] [java] at com.oxygenxml.chemistry.e.eb.b(Unknown Source)
[exec] [java] at com.oxygenxml.chemistry.OxygenPDFChemistry.b(Unknown Source)
[exec] [java] at com.oxygenxml.chemistry.OxygenPDFChemistry.processInternal(Unknown Source)
[exec] [java] at com.oxygenxml.chemistry.OxygenPDFChemistry$1.b(Unknown Source)
[exec] [java] at com.oxygenxml.chemistry.OxygenPDFChemistry$1.run(Unknown Source)
[exec] [java] at ro.sync.security.SandboxCore.runWithConfirmation(SandboxCore.java:258)
[exec] [java] at ro.sync.security.ConnectionsSandbox.runWithConnectConfirmation(ConnectionsSandbox.java:263)
[exec] [java] at ro.sync.security.Sandbox.runWithConnectConfirmation(Sandbox.java:355)
[exec] [java] at com.oxygenxml.chemistry.OxygenPDFChemistry.process(Unknown Source)
[exec] [java] at com.oxygenxml.chemistry.OxygenPDFChemistry.process(Unknown Source)
[exec] [java] at com.oxygenxml.chemistry.OxygenPDFChemistry.process(Unknown Source)
[exec] [java] at com.oxygenxml.chemistry.OxygenPDFChemistry.main(Unknown Source)
[exec] Error: The following error occurred while executing this line:
[exec] C:\Program Files\Oxygen XML Editor 25 - 2022081006\frameworks\dita\DITA-OT3.x\plugins\com.oxygenxml.pdf.css\build.xml:670: Java returned: 1
BUILD FAILED
C:\Users\chrispy\git\dita-ams\prj\frameworks\dita\dita_project\build_dita_project_preprocessed.xml:25: exec returned: 1
Total time: 52 minutes 21 seconds
Another writer has a Windows 10 laptop with 16GB physical memory, and the transformation always crashes for her even with -Xmx10240m (although we only ran it a few times, so "always" is based on a small dataset).
Part of the difficulty in debugging this is the lack of information. For example,
- Does a deliverable require more memory when published in a set of "All Deliverables" versus published by itself?
- Does a transformation's -X memory limit include what Oxygen itself uses, or do the transformation and Oxygen each have their own limit? (related docs here)
- How do I differentiate between an insufficient -X limit versus the -X limit not getting reached due to other applications consuming memory on the machine?
- Does the -X limit consider available Windows 10 virtual memory, or just physical memory?
As an enhancement idea, perhaps Oxygen could somehow print the highest memory usage for both failed transformations and successful transformations. This would provide visibility into whether the -X limit was reached and thus is insufficient, versus not reached due to unavailable system resources.
I can provide a testcase upon request. Thanks!
-
- Posts: 553
- Joined: Wed Oct 16, 2019 3:47 pm
Re: Debugging java.lang.OutOfMemoryError issues
Post by julien_lacour »
For analysis, could you provide us the part of the console that calls Chemistry, it starts with:
Code: Select all
[exec] pdf-css.chemistry:
[exec]
[exec] pdf-css.chemistry.embedded:
[exec] [echo] =============================
[exec] [echo] Oxygen PDF Chemistry
[exec] [echo] Installation directory: ...
- The transformation using DITA project file should be equivalent to the transformation scenario with the Prefer using the "dita" command option unchecked.
- The -Xmx parameter set is only used by the transformation and not by Oxygen itself.
- The OutOfMemory error mean that the -X limit has been reached, the other applications should not interfere here.
- The -X memory is Java virtual memory, the link between it and Windows memory is a little complicated as the operating system itself manage how the memory is used.
Regards,
Julien
-
- Posts: 915
- Joined: Thu May 02, 2019 2:32 pm
Re: Debugging java.lang.OutOfMemoryError issues
Post by chrispitude »
Here is the requested portion of the log:
Code: Select all
[exec] pdf-css.chemistry.embedded:
[exec] [echo] =============================
[exec] [echo] Oxygen PDF Chemistry
[exec] [echo] Installation directory: C:\Program Files\Oxygen XML Editor 25 - 2022081006\frameworks\dita\DITA-OT3.x\bin\../../../..
[exec] [echo] Processing: C:\Users\chrispy\git\dita-ams\temp\stage1.xml.pp
[exec] [echo] Output file: C:\Users\chrispy\git\dita-ams\out\final\olh_cc\pdf/cc_tclref.pdf
[exec] [echo] =============================
[exec] [java] INFO OxygenPDFChemistry - oXygen PDF Chemistry version 25.0 build 2022-08-10T02:52:53Z
[exec] [java] INFO OxygenPDFChemistry - Using catalogs: [file:/C:/Program%20Files/Oxygen%20XML%20Editor%2025%20-%202022081006/config/catalogs/catalog.xml, file:/C:/Program%20Files/Oxygen%20XML%20Editor%2025%20-%202022081006/frameworks/dita/DITA-OT3.x/catalog-dita.xml, file:/C:/Program%20Files/Oxygen%20XML%20Editor%2025%20-%202022081006/frameworks/docbook/catalog.xml]
[exec] [java] INFO x - CSS: file:/C:/Program%20Files/Oxygen%20XML%20Editor%2025%20-%202022081006/frameworks/dita/DITA-OT3.x/plugins/com.oxygenxml.webhelp.responsive/oxygen-webhelp/app/topic-page-print.css
[exec] [java] INFO x - CSS: file:/C:/Program%20Files/Oxygen%20XML%20Editor%2025%20-%202022081006/frameworks/dita/DITA-OT3.x/plugins/com.oxygenxml.pdf.css/css/print/p-dita.css
[exec] [java] INFO x - CSS: file:/C:/Program%20Files/Oxygen%20XML%20Editor%2025%20-%202022081006/frameworks/dita/DITA-OT3.x/plugins/com.oxygenxml.editlink/pdf-css.css
[exec] [java] INFO x - CSS: file:/C:/Users/chrispy/git/dita-ams/dita/_common/_template/../../../prj/frameworks/dita/css/synopsys.css
[exec] [java] INFO OxygenPDFChemistry - Dumping styled content to: C:\Users\chrispy\git\dita-ams\out\final\olh_cc\pdf\cc_tclref.style.xml
[exec] [java] INFO OxygenPDFChemistry - Dumping FO content to: C:\Users\chrispy\git\dita-ams\out\final\olh_cc\pdf\cc_tclref.fo
[exec] [java] INFO LoggingEventListener - Rendered page #1.
[exec] [java] INFO LoggingEventListener - Rendered page #2.
...omitted...
[exec] [java] INFO LoggingEventListener - Rendered page #4203.
[exec] [java] INFO LoggingEventListener - Rendered page #4204.
[exec] [java] Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
[exec] [java] at java.base/java.util.Arrays.copyOf(Arrays.java:3537)
[exec] [java] at java.base/java.io.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:185)
[exec] [java] at com.oxygenxml.chemistry.e.eb.b(Unknown Source)
[exec] [java] at com.oxygenxml.chemistry.OxygenPDFChemistry.b(Unknown Source)
[exec] [java] at com.oxygenxml.chemistry.OxygenPDFChemistry.processInternal(Unknown Source)
[exec] [java] at com.oxygenxml.chemistry.OxygenPDFChemistry$1.b(Unknown Source)
[exec] [java] at com.oxygenxml.chemistry.OxygenPDFChemistry$1.run(Unknown Source)
[exec] [java] at ro.sync.security.SandboxCore.runWithConfirmation(SandboxCore.java:258)
[exec] [java] at ro.sync.security.ConnectionsSandbox.runWithConnectConfirmation(ConnectionsSandbox.java:263)
[exec] [java] at ro.sync.security.Sandbox.runWithConnectConfirmation(Sandbox.java:355)
[exec] [java] at com.oxygenxml.chemistry.OxygenPDFChemistry.process(Unknown Source)
[exec] [java] at com.oxygenxml.chemistry.OxygenPDFChemistry.process(Unknown Source)
[exec] [java] at com.oxygenxml.chemistry.OxygenPDFChemistry.process(Unknown Source)
[exec] [java] at com.oxygenxml.chemistry.OxygenPDFChemistry.main(Unknown Source)
[exec] Error: The following error occurred while executing this line:
[exec] C:\Program Files\Oxygen XML Editor 25 - 2022081006\frameworks\dita\DITA-OT3.x\plugins\com.oxygenxml.pdf.css\build.xml:670: Java returned: 1
BUILD FAILED
C:\Users\chrispy\git\dita-ams\prj\frameworks\dita\dita_project\build_dita_project_preprocessed.xml:25: exec returned: 1
-
- Posts: 553
- Joined: Wed Oct 16, 2019 3:47 pm
Re: Debugging java.lang.OutOfMemoryError issues
Post by julien_lacour »
Starting with Oxygen 25.0 you should have more information about memory consumption in the DITA Map PDF - based on HTML5 & CSS scenarios.
Regards,
Julien
-
- Posts: 915
- Joined: Thu May 02, 2019 2:32 pm
Re: Debugging java.lang.OutOfMemoryError issues
Post by chrispitude »
Thanks very much for this fix! This will be very helpful the next time I am debugging publishing issues for writers working with some of our larger reference books.
-
- Posts: 915
- Joined: Thu May 02, 2019 2:32 pm
Re: Debugging java.lang.OutOfMemoryError issues
Post by chrispitude »
When I publish in Linux with -Xmx20G set in our global Java options file, I see the following in the log:
Code: Select all
[java] INFO OxygenPDFChemistry - Max memory available: 18204Mb
- image.png (15.94 KiB) Viewed 5388 times
then the log always shows the following:
Code: Select all
[java] INFO OxygenPDFChemistry - Max memory available: 4034Mb
- oxygen_chemistry_memory.zip
- (15.79 KiB) Downloaded 471 times
I hope I am not doing something obviously wrong...
-
- Posts: 39
- Joined: Mon Jul 25, 2022 11:18 am
Re: Debugging java.lang.OutOfMemoryError issues
Post by andrei_pomacu »
In Windows, in the JVM Arguments tab you need to use -DbaseJVMArgLine=-Xmx{memory}.
So instead of a simple -Xmx8g you need to use -DbaseJVMArgLine=-Xmx8g or -DbaseJVMArgLine=-Xmx8192m
Regards,
Andrei
-
- Posts: 915
- Joined: Thu May 02, 2019 2:32 pm
Re: Debugging java.lang.OutOfMemoryError issues
Post by chrispitude »
I confirmed that this works - thank you! This new memory reporting feature is already providing value.
![Smile :)](https://www.oxygenxml.com/forum/images/smilies/icon_smile.gif)
There are many mentions of -Xmx in the Oxygen documentation:
# document(s) found for -Xmx
but no mention of setting it via -DbaseJVMArgLine=. I don't know if it's worth internally modifying a specification when running in Windows versus updating all the documentation instances to mention the Windows requirement.
-
- Posts: 553
- Joined: Wed Oct 16, 2019 3:47 pm
Re: Debugging java.lang.OutOfMemoryError issues
Post by julien_lacour »
Actually this indication is present at the end of the Increasing Memory Allocation for Java topic:
But still I will add a Documentation Task to improve these topics and their display.If the Chemistry PDF CSS processor fails with an Out Of Memory Error, try adding the baseJVMArgLine parameter to the DITA-OT command line. For example:Code: Select all
-DbaseJVMArgLine=-Xmx984m
Regards,
Julien
-
- Posts: 915
- Joined: Thu May 02, 2019 2:32 pm
Re: Debugging java.lang.OutOfMemoryError issues
Post by chrispitude »
Now I am unsure exactly what -Xmx and -DbaseJVMArgLine=-Xmx each control:
- Does -DbaseJVMArgLine=-Xmx require that -Xmx also be specified? (Is it configuring the memory of a launched subprocess?)
- Is -DbaseJVMArgLine=-Xmx specific to PDF Chemistry, or does it apply to other transtypes too?
- Is -DbaseJVMArgLine=-Xmx specific to transformations run from the Oxygen GUI, or does it also apply to batch Publishing Engine jobs?
- Is -DbaseJVMArgLine=-Xmx specific to Windows, or does it apply to linux too?
-
- Posts: 553
- Joined: Wed Oct 16, 2019 3:47 pm
Re: Debugging java.lang.OutOfMemoryError issues
Post by julien_lacour »
- Does -DbaseJVMArgLine=-Xmx require that -Xmx also be specified? (Is it configuring the memory of a launched subprocess?)
The -DbaseJVMArgLine parameter can be set alone, without any additional -Xmx (except the one appearing after =). - Is -DbaseJVMArgLine=-Xmx specific to PDF Chemistry, or does it apply to other transtypes too?
It is specific to Chemistry or any other transtype running a Java sub-process (I don't think there are other other Java sub-processes by default but you can have your own). - Is -DbaseJVMArgLine=-Xmx specific to transformations run from the Oxygen GUI, or does it also apply to batch Publishing Engine jobs?
You can either set the -DbaseJVMArgLine in Oxygen or as a parameter in the dita command. - Is -DbaseJVMArgLine=-Xmx specific to Windows, or does it apply to linux too?
The -DbaseJVMArgLine works fine for both Windows and Linux, for example:Code: Select all
./dita -v -f pdf-css-html5 -DbaseJVMArgLine=-Xmx2048m -i ~/OxygenXMLEditor/samples/dita/flowers/flowers.ditamap -o ~/OxygenXMLEditor/samples/dita/flowers/out/pdf-css-html5
Julien
-
- Posts: 915
- Joined: Thu May 02, 2019 2:32 pm
Re: Debugging java.lang.OutOfMemoryError issues
Post by chrispitude »
This is good information! So the PDF Chemistry transformation internally calls a Java subprocess, and this Java subprocess sometimes needs additional memory provided.
Can you confirm that -Xmx must still provide enough memory for regular DITA-OT preprocessing up through merged map creation?
I think I will need to set both -Xmx and -DbaseJVMArgLine=-Xmx based on experimentation with our largest books, so that neither limit is set too low or unnecessarily high. Hopefully each limit's failure has its own distinctive message somehow. I can try to figure that out in the experimentation too.
-
- Posts: 553
- Joined: Wed Oct 16, 2019 3:47 pm
Re: Debugging java.lang.OutOfMemoryError issues
Post by julien_lacour »
In fact, when transforming DITA to PDF, Oxygen PDF Chemistry is the Java sub-process called by our com.oxygenxml.pdf.css plugin (when reaching the "pdf-css.chemistry" target in the console, to be precise).
I confirm that setting -Xmx may be needed by the regular DITA-OT processing (through the com.oxygenxml.pdf.css plugin) which creates the merged.xml and merged.html files.
You may need to set both for your largest books depending on where the process fails: in the merging steps or after calling Oxygen PDF Chemistry.
Regards,
Julien
-
- Posts: 915
- Joined: Thu May 02, 2019 2:32 pm
Re: Debugging java.lang.OutOfMemoryError issues
Post by chrispitude »
In addition to reporting available memory at the beginning, is it also possible to report peak memory usage at the end? This would be faster (and more reliable!) than deriving values using trial-and-error with very large documents!
I cannot seem to pass this parameter to PDF Chemistry deliverables published from a DITA-OT project. In the following testcase:
- oxygen_chemistry_memory_v2.zip
- (22.4 KiB) Downloaded 465 times
I define the following transformations in a DITA-OT Project framework extension:
- Publish DITA-OT Project (all deliverables) - 6GB, 6GB
- Publish DITA-OT Project (select deliverable) - 6GB, 6GB
Code: Select all
-Xmx6g -DbaseJVMArgLine=-Xmx6g
Code: Select all
[exec] [java] INFO OxygenPDFChemistry - oXygen PDF Chemistry version 25.0 build 2022-09-29T13:05:39Z
[exec] [java] INFO OxygenPDFChemistry - Max memory available: 4034Mb
-
- Posts: 553
- Joined: Wed Oct 16, 2019 3:47 pm
Re: Debugging java.lang.OutOfMemoryError issues
Post by julien_lacour »
The thing is that when running DITA transformation through DITA-OT Project file under the hood a Java process is called (and not the dita command). This is why the Java usage may differ, for this use-cases the dita command is a better choice (maybe through a bat or sh script).
Regarding the Java peak usage, even this can be tricky as we cannot really control when Java Garbage Collector will run and free memory.
Regards,
Julien
-
- Posts: 915
- Joined: Thu May 02, 2019 2:32 pm
Re: Debugging java.lang.OutOfMemoryError issues
Post by chrispitude »
Does this mean that there is no way to control the preprocessing memory limit or PDF Chemistry subprocess memory limit when running DITA transformations from a DITA-OT project file transformation?
- image.png (11.76 KiB) Viewed 5192 times
For a DITA-OT project file transformation, if the DITA-OT transformation is invoked internally, does that mean -DbaseJVMArgLine=-Xmx now controls the preprocessing memory limit (and then the PDF Chemistry subprocess always uses the default)?
I agree that running transformations from a command line provides more flexibility. However,
- Our writers use Oxygen on Windows laptops, and Windows does not provide a sufficient environment for command-line operation.
- We do not have Oxygen Publishing Engine licenses for our writers; they use Oxygen for their local test builds. (We do use Oxygen Publishing Engine in linux for final production publishing.)
- Now that our DITA-OT project files capture all profiling conditions and publication-specific parameters, writers publish more often from DITA-OT project files (instead of from the DITA Maps Manager) for their test builds.
-
- Posts: 553
- Joined: Wed Oct 16, 2019 3:47 pm
Re: Debugging java.lang.OutOfMemoryError issues
Post by julien_lacour »
I added your vote to the issue related to JVM Memory when running DITA-OT Project files (EXM-51733).
The scope of this issue will be to transmit "JVM Arguments" from Scenario Options as ANT_OPTS environment variable.
I will notify this thread as soon as it will be resolved.
Until then you will need to follow these steps:
- Open the OXYGEN_INSTALL_DIR\frameworks\dita\dita_project\build_dita_project.xml file in Oxygen
- Add the ANT_OPTS declaration like follows:
Code: Select all
... <env key="JAVA_HOME" value="${java.home}"/> <env key="ANT_OPTS" value="-DbaseJVMArgLine=-Xmx6G"/> ...
- Set the -Xmx with the needed value, it will be passed to Oxygen PDF Chemistry.
Julien
-
- Site Admin
- Posts: 121
- Joined: Wed Dec 12, 2018 5:33 pm
Re: Debugging java.lang.OutOfMemoryError issues
Post by Cosmin Duna »
As an update, we have released Oxygen 25.1, and a new parameter has been included in the DITA-OT transformation scenarios that allows for setting the JVM memory allocation for publishing deliverables: https://www.oxygenxml.com/doc/versions/ ... l=jvm.args
Best regards,
Cosmin
<oXygen/> XML Editor
http://www.oxygenxml.com
-
- Posts: 915
- Joined: Thu May 02, 2019 2:32 pm
Re: Debugging java.lang.OutOfMemoryError issues
Post by chrispitude »
- Oxygen XML Editor/Author/Developer
- ↳ Feature Request
- ↳ Common Problems
- ↳ DITA (Editing and Publishing DITA Content)
- ↳ SDK-API, Frameworks - Document Types
- ↳ DocBook
- ↳ TEI
- ↳ XHTML
- ↳ Other Issues
- Oxygen XML Web Author
- ↳ Feature Request
- ↳ Common Problems
- Oxygen Content Fusion
- ↳ Feature Request
- ↳ Common Problems
- Oxygen JSON Editor
- ↳ Feature Request
- ↳ Common Problems
- Oxygen PDF Chemistry
- ↳ Feature Request
- ↳ Common Problems
- Oxygen Feedback
- ↳ Feature Request
- ↳ Common Problems
- Oxygen XML WebHelp
- ↳ Feature Request
- ↳ Common Problems
- XML
- ↳ General XML Questions
- ↳ XSLT and FOP
- ↳ XML Schemas
- ↳ XQuery
- NVDL
- ↳ General NVDL Issues
- ↳ oNVDL Related Issues
- XML Services Market
- ↳ Offer a Service