Post

2 followers Follow
0
Avatar

DocumentNote values can (apparently) not be queried on using API

Dear support,

I seem to have to be coming back to this issue again:

I'm developing a plugin for Naturalis that adds document notes to fasta and AB1 documents as it imports them. That works fine. The point is: when I import the exact same document again, the plugin should detect that because one of the document notes functions as a key. The plugin must still import the document, but another document note - the "document version" - must be incremented.

I know Geneious has a revision number itself, but these are the specifications for the plugin, so that's what I need to implement.

So the plugin first does a query on the note field functioning as the key, and then sets the "document version" for the new document as appropriate.

However, this never works. In the log file you can see Geneious executing the right query, but nothing ever comes back.

Attached are two images: the first is the log file (which is actually presented to the user via a Swing TextArea). You can see the query that Geneious executes (it's a toString() of your Query class) and it seems perfectly valid, but the next log messages says: "0 documents found".

The second is the Geneious GUI itself. It shows you the two identifical documents, but still with the same "document version" note. The "Extract ID" column is the key I'm searching on, and as you can see they are the same for the 2 documents.

Strangely, Geneious itself HAS detected the documents are the same - it has put a 2 after the last one's name.

Finally, here you can see the QueryUtils class that does the query (findByExtractID), and the enum, responsible for creating a query field from a DocumentNoteField:

https://github.com/naturalis/sd_java__geneious_plugin/blob/v2_master/importers/src/main/java/nl/naturalis/geneious/note/NaturalisField.java

https://github.com/naturalis/sd_java__geneious_plugin/blob/v2_master/importers/src/main/java/nl/naturalis/geneious/util/QueryUtils.java

Informatiemanagement

Please sign in to leave a comment.

11 comments

0
Avatar

I'm sorry ... I finally figured it out myself. Thanks.

Informatiemanagement 0 votes
Comment actions Permalink
0
Avatar

Actually not, I discovered a mistake in the query, but the problem persists, and I mentioned it in another post: now the query DOES sometimes find the previously imported documents, but sometimes not. It seems like if I execute the action in quick succession, the documents have not been indexed yet.

I really need a way to force Geneious to index the documents. Is there an API for that?

Informatiemanagement 0 votes
Comment actions Permalink
0
Avatar

I believe indexing is normally triggered immediately so you don't need to force it as such. Instead you should wait for indexing to finish using:

WritableDatabaseService.waitForSearchIndexingToComplete()

(on your target database)

Richard Moir 0 votes
Comment actions Permalink
0
Avatar

The point is waitForSearchIndexing returns immediately without doing anything for shared databases (according to the javadocs) precisely because indexing is triggered immediately.

However, if that is so, indexing apparently takes quite some time, certainly enough time for the user to initiate another document operation before the indexing completed.

So then I would need an API like "isStillIndexing". Only after that returns false I would let the document operation allow to do any search

Informatiemanagement 0 votes
Comment actions Permalink
0
Avatar

The javadoc wording is a bit misleading sorry.

"The shared database always indexes documents immediately, so this this method always returns immediately without doing anything"

Indexing isn't run in a separate thread in the shared database, it's synchronous, so the document doesn't even appear in the database until indexing is complete. That's what it means by it "indexes documents immediately".

So if you are getting documents that don't show up in shared database search results then something else is going wrong (or the javadoc is wrong). Can you confirm if that's case before we dig deeper?

Richard Moir 0 votes
Comment actions Permalink
0
Avatar

Yes, I can confirm this. If something would be wrong with my code, I would predicatably get wrong results. But this is not the case.

 

If I import an annotated (say) fasta document in one operation, and then in the next one search for it using the value of one of the document notes, the document sometimes comes back and sometimes not. If I wait more than about 50 seconds, the document is basically guaranteerd to come back. If I do both operations in quick succession the document most of the time does not come back, even with a call to waitForSearchIndexingToComplete.

Informatiemanagement 0 votes
Comment actions Permalink
0
Avatar

Could it be that I make the call to waitForSearchIndexingToComplete within a SwingWorker thread laucnhed from the DocumentOperation subclass, rather than in the DocumentOperation subclass itself?

Also I make the call at the beginning of the DocumentOperation, not at the end. I don't mind users to continuing straight afterwards, but I do mind if they start the next DocumentOperation while indexing is still going on.

Informatiemanagement 0 votes
Comment actions Permalink
0
Avatar

Hello Richard,

 

Have you had a chance to look into my problem?

 

I have now also tried making the call to waitForSearchIndexingToComplete outside the SwingWorker thread. But even if I made it the first (or last) statement in the performOperation method, I still had the same problem.

Informatiemanagement 0 votes
Comment actions Permalink
0
Avatar

Sorry, I've had a chat to the dev team and it turns out we changed the behaviour a while back such that it doesn't index documents straight away, and instead does it in a background thread as you suspected.

Unfortunately the javadoc and implementation for the waitForSearchIndexingToComplete() method have not been updated to accommodate for this.

So there is currently no way to reliably wait for the documents to be indexed in the shared database. I have logged a bug for this but I can't provide an estimate on when it will be fixed.

I've been trying to think of a good workaround but I haven't been able to come up with anything I'm afraid. With that knowledge, maybe you'll have more luck?

Richard Moir 0 votes
Comment actions Permalink
0
Avatar

Hi Richard,

 

OK, thanks. I sure hope your dev team will pick this up quickly, because it is rather critical for us (since obviously documents can now enter the database in a functionally corrupt state - .e.g. the document version note not set correctly).

 

I guess what I'll do now is to force users to wait for a fixed amount of seconds before allowing them to initiate another operation

Informatiemanagement 0 votes
Comment actions Permalink
0
Avatar

BTW I did in the end figure out a (pretty dirty) way od making sure has completed. After I have finished a DocumentOperarion creating or updating some DocumentNotes, I create a "ping document" with a unique value for one of our DocumentNotes. Then, while sleeping every 3 seconds, I try to retrieve that document use that value. If I get it back I take it that indexing of the other documents is now also complete, and delete the pingd ocument.

Informatiemanagement 0 votes
Comment actions Permalink