On non-mac systems, users can set system properties and any JVM options by editing the Geneious.in.use.vmoptions file in the installation directory. On mac systems they will need to edit the plist file within the .app bundle.
However you might want to consider another option since setting system properties can be quite cumbersome from an end user perspective.
One option might be to allow the user to configure it in the GUI via Options for an operation. Or more generally via PluginPreferences. See GeneiousPlugin.getPluginPreferences().