Release memory programmatically
Hello, I have the following use case:
We've created a plugin that among it's features, it needs to create documents in Geneious from sequences obtained from an external database. These databases can be pretty large, e.g.: ~30k sequences.
The plugin works fine with a relatively small amount of sequences, but runs out of memory with larger amounts. We've run a memory profiler and these seem to be the pieces of code that allocate more memory:
This one is used to know if the incoming sequence already exists in Geneious.
And this one is used to create the document from the Genbank file received.
I am not accumulating in memory what's returned by those methods so in theory, the garbage collector should release the memory used by them, but it seems like it's not since usage increases a the plugin runs.
In the UI there's a way to release unused memory which seems like what I need, but I don't see a way to trigger that programatically.
Is there a recommended approach? Is it possible that those methods are storing references internally hence preventing the garbage collector from releasing memory?
PS: I'm aware that I can create a Database Service Plugin, but that doesn't fit our use case.
Thanks!
-
The button in the UI only triggers the garbage collector so probably won't help you because that only frees up objects that are no longer referenced. But if you're running out of memory then the garbage collector has likely already run and failed to free up anything. It is a bit hard to say what the cause is without seeing more of the code.
One possibility is that the search results take up that much memory. If you don't need to work with them all at once then you could switch to using one of the variants of the retrieve method that use a RetrieveCallback. That will allow you to work with each search result one at a time and will avoid holding them all in memory.
If you share the source code with us at support@geneious.com then we can take a look to see what is going on.
Presumably the import is using our standard GenBank importer so I'm quite surprised to see that simply importing the genbank file is jumping up memory to 252.9 GB.
- How much memory was being used prior to the import?
- How large is the genbank file on disk and how many sequences are in it?
If you could share the genbank file we could take a look to see if there is some kind of bug in the importer.
0 -
Hello,
Just to clarify, this is not a single import. The code basically iterates over ~40k sequences (each relatively small, <5kb each) retrieved from an API and imports each sequence as a Genbank. The 252.9 GB that the profiler is showing is the cumulative memory used by the method throughout the whole iteration.
Regarding search results taking too much space, that shouldn't be the case either since that line of code is querying the Geneious database to determine if the current sequence already exists, so the query should only retrieve 1 or 0 results.0 -
Ah OK, I misunderstood the problem you're having. Are you running into OutOfMemoryErrors? Or just observing lots of memory being allocated?
If the later, does clicking on the button in the UI to clear memory decrease what is being used? If it does decrease then that suggests that it is a matter of the garbage collector not yet freeing up the memory (but it could). If you need to you can use System.gc() to suggest the garbage collector run, but most likely it is better to just let it happen by itself.
If you are running into OutOfMemoryErrors then see if you can use your profiler to see what the largest objects are in memory. Then have a look at the gc root for that object. That would probably be the best place to start to track down what is going on.
1
Please sign in to leave a comment.
Comments
3 comments