Issue
We have a Spring Boot application running Hibernate Search 5.10.7.Final with a Lucene backend and are seeing high CPU usage during the same time each day.
A thread dump showed that the threads consuming the CPU corresponded to garbage collection activity by the JVM:
"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007fdef801f800 nid=0x173b runnable
"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007fdef8021800 nid=0x173c runnable
"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007fdef8023800 nid=0x173d runnable
"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007fdef8025000 nid=0x173e runnable
A heap dump shows 2 instances of PeriodicRefreshingReaderProvider
as the largest objects.
There are also 3,763 instances of org.apache.lucene.index.StandardDirectoryReader
.
I am not familiar enough with Hibernate Search or Lucene to determine if this is typical and we just need a larger heap size or something else is awry.
Current max heap size is 12GB (-Xmx). The Apache Lucene indexes only comprise around 8GB of disk space.
Hibernate Search settings
hibernate.search.default.directory_provider = filesystem
hibernate.search.default.indexBase = /var/lucene/indexes
hibernate.search.default.reader.strategy = async
hibernate.search.default.reader.async_refresh_period_ms = 8000
Solution
You're apparently using the async
refresh strategy, and the map that references all opened readers apparently gets very, very large.
No, that is not normal.
You're mentioning the CPU spike happens the same time everyday. What did you set the refresh period to? I.e. what's the value of the configuration property hibernate.search.[default|<indexname>].reader.async_refresh_period_ms
? If it's some really huge value close to 24 hours, that could explain your problem.
Index readers are not usually kept around for such a large amount of time, so I suppose it's possible that they get very big over time. If you are in this situation, try reducing the refresh period to something more reasonable, e.g. 1 minute or 5 minutes: you may get CPU spikes more often, but much smaller ones, and you'll use less memory.
Alternatively, there might be a reader leak somewhere.
I suppose it could be a leak in Hibernate Search proper, but the relevant code has been in use for years and we haven't seen a single report of such a leak, so I find that dubious.
Are you accessing index readers explicitly in your application, by any chance? Through getIndexReaderAccessor()? If so, check that you're correctly closing the readers. If you don't, you're effectively leaking index readers.
Answered By - yrodiere