Searching in Convex
For now full table scan is how I'd go about it. We know that's going to fall down past a certain size of data but it's interesting (well I think so) to push the limits of that approach atm.
There are several potential options, some involving egress to dedicated search infra services. Pragmatically you could denormalize the data, building a search index in Convex (another good candidate for a plug and play library) but we can provide something more convenient before long.
6 Replies
Yep @RJ the best bet you have today is to do a table scan, but quite soon we'll have support for streaming data out of Convex into external systems like Elasticsearch etc. This will also be useful for bulk analytics-type workloads that are a poor fit for an OLTP database (i.e., a low-latency customer-facing database like Convex)
If there's sufficient demand for native search within Convex this is something we could consider implementing internally
Fyi, this is the approach I took and it worked fine, however I ran into query limits. This is where pagination will help a lot once it lands ...
Thank you everyone! That all makes sense to me. I imagine I would really want some native (or library) support for searching in Convex. It would seem a shame to need to pull in a new service/system for fairly basic full text search needs, especially if something like Postgres would have been able to handle the same needs all on its own, at comparable scale.
ah, if it's just the kind of search that you could get from postgres, i.e., a table scan with a
LIKE query rather than an actual search index, then that should likely be fine with convex as-is (with addition of pagination)I was initially asking specifically about
LIKE, but in my previous comment meant Postgres plus modules like fuzzystrmatch (https://www.postgresql.org/docs/current/fuzzystrmatch.html) and pg_trgm (https://www.postgresql.org/docs/current/pgtrgm.html)How is it going with text search? Is there a limit of items when querying with collect()?