What to expect from built-in query caching
As I mentioned in another post yesterday, I am curious to know more about the behavior of Convex's built-in query caching. My main question is about the extent to which it's advisable to rely on the built-in caching to achieve a desired level of performance.
Like my function-profiling question, this one has become less pressing because my app's critical query is now running fast enough that I'm not so concerned about cache misses, but I'd still like to get a better understanding of caching behavior and best practices.
Specifically, I'm wondering under what circumstances a cached query might be evicted from the cache without any changes to the query's code or the data it depends on. Is there any behind-the-scenes logic that evicts less frequently / recently used queries? If so, does that run per-instance, or might a less heavily used instance's cached queries be evicted to make room for another instance's entries? In a similar vein, are there things that might cause the cache to be cleared entirely?
In my app, data is read much more frequently than it is written, and the same queries are likely to be requested from different clients, so in a world where the cache weren't actively managed or periodically cleared, I'd expect a very high hit rate. When I was looking at a significant discrepancy between cached and uncached performance, though, I was nervous about relying on cache hits and considering whether I might need to implement some bespoke, app-level caching to ensure consistently good performance.
3 Replies
Hi @Gray, our design intention is that you should never have to do your own caching. We'd rather handle the caching in a strictly consistent way and save developers complicating their apps by doing it manually.
Currently all queries are cached based on the data ranges they read, similarly to how query subscriptions are triggered. If the query doesn't read system time then it will remain in the cache semi-indefinitely and will only drop out if we restart the backend, new code is pushed, or it gets evicted from the cache, which ideally doesn't happen.
There are a few complexities to caching:
1. We currently only cache whole function results predicated on the arguments to that function. This means that if
userId is a function argument we won't cache results between users, since they are independent function runs. We'll improve this over time with sub-query caching.
2. If a query reads system time (Date.now()) then we cache it for at most 5 seconds. This is a tradeoff to provide some caching performance without having results that are consistent but not "fresh".
Since Convex knows exactly what data ranges a query accesses it can perform caching very effectively, assuming you actually want consistent results.
As mentioned, in future we plan to add sub-query caching and also to deploy edge caches close to users.
There is no sharing of cached data between projects/instances and no risk of one project causing data to be evicted from another project's cache.
Lmk if you have any additional questions or caching requirements!Thanks for the detailed explanation, @james!
It's great to have explicit confirmation that your aim is to make caching as plug-and-play and reliable as possible, and to know how that translates to the design.
One follow-up question: is there any difference between dev and prod instances in this regard (e.g., different cache size limits, or differences in the frequency with which the backend might be restarted)? I have thus far only been working in my dev instance, and I thought I was seeing some surprising cache misses, but it's possible that I just pushed code or changed data and forgot.
I will look forward to sub-query caching, as I am currently decomposing my query and taking a small perf hit for an additional round-trip to avoid per-user caching, but this is not a big problem at all.
@Gray no difference yet. but we will introduced bigger cache sizes over time for pro accounts. dev and prod are similar now