Profiling function performance
I have an app that I'm aiming to move from Firebase to Convex. Functionality-wise, the migration is nearly complete.
The Convex version is nicer in every way and there are a bunch of features I'd like to add that I expect will be much easier to implement with Convex than with Firebase.
Before committing, though, I'm trying to satisfy myself that the Convex version of the app will be viable from a performance and cost point of view. I've got a handful of specific questions, which I'll try breaking up into separate support threads.
My app essentially presents a large data set and provides affordances for zeroing in on items of interest via tag-based filtering. The Firebase version, out of necessity, does a lot of manual preprocessing, computing the items returned for combinations of tags and storing them in the database. In the Convex version, I'm hoping to avoid that, instead relying on Convex's close-to-the-db backend logic and query caching to yield good performance while keeping the implementation much simpler.
I'll ask caching questions in another thread, but first I'm wondering if there's any guidance on doing fine-grained performance profiling within a function. My current implementation features highly normalized data; the query driving the main view loops through the base data set, does per-item db reads to retrieve tags, filters items by tag, and finally loads additional fields for the filtered items. Performance in the uncached case is borderline acceptable. I can think of ways to improve it, mainly via different flavors of denormalization, but before going down the path of trying these, I'd ideally like to collect some data on where my execution time is currently being spent.
My naive attempts to collect / log timing data have been thwarted by the limitations of the runtime that guarantee deterministic query execution. Is there any alternative?
9 Replies
@rkbh might have some info here -- we are dumping some profiling data into the new log streams, but not sure it's granular enough to satisfy your needs here
FWIW, I ended up going ahead and trying a reasonably simple form of denormalization and found that it dramatically improved the performance of my function, to the point where I’m no longer concerned. That said, I’m still generally interested in the topic of performance profiling, as I imagine the need could arise again.
Thanks for letting us know @Gray . Finer grained profiling is on our radar.
Thanks, @Indy!
On a related note, with my performance concerns alleviated, I'm now digging into database bandwidth usage and would find some more granular tools helpful there, too. Not sure at what layer database bandwidth is measured, but if it's done per function invocation, even something as minor as adding bandwidth usage to the log view would be very helpful.
That's a great thought! I believe this is in progress. @rkbh who's been working adding more information to log streams will know for sure.
Just saw this, @Gray database read/write and storage read/write bandwidth are being added to log streams very soon! Should be available later today or Monday. The plan is to add it to the
_execution_record log. The docs will be updated here as soon as it's out! https://docs.convex.dev/production/log-streams#function-execution-record-logsLog Streams | Convex Developer Hub
Configure logging integrations for your Convex deployment
Thanks for the update, @rkbh!
@Gray we just introduced getting more granular timing data in functions with
console.time() in convex 1.9. This allows you to see how long things take within a query. release notes: https://news.convex.dev/announcing-convex-1-9/Convex News
Announcing Convex 1.9
Big Import and Export Improvements
When we released ZIP file imports and exports in Convex 1.7, it wasn't long before industrious developers started hitting the limitations. We’ve got some big improvements for you.
* File storage can now be included in Snapshot Export
* Snapshot import via npx convex import
Great—thanks for circling back, @ian!