Querying the Database | Convex Developer...
Hi folks!
I was reading through the docs\blogs and I have only one question, is this https://docs.convex.dev/using/database-queries#more-complex-queries really the way to go with complex queries?
I imagined Convex to the a better alternative to Firestore, but this looks... confusing to say the least π¦
Any plans for real joins and aggregations?
Also, any plans on revealing the underlying tech? So far, i'm getting the vibes of DynamoDb or smth like Mongo as a storage solution, maybe I'm wrong π€·ββοΈ
Querying the Database | Convex Developer Hub
Push queries down to the database.
31 Replies
Hi @Dima Utkin! Welcome! Glad to see you taking a look. We are leaning into the idea that writing code (js/ts) for your queries is more expressive than recreating a query language. Because the code is running close to the DB - it'll be fast enough for a much larger scale.
That being said, we do want to make it easier to express joins and aggregations, so stay tuned!
Regarding underlying tech, we have some information here in the "Understanding Convex" section https://docs.convex.dev/understanding/convex-fundamentals/. If you have any particular questions, happy to answer. It's not intended to be a secret.
Convex Fundamentals | Convex Developer Hub
We're going to walk through how Convex works, and along the way we'll explore a
We run javascript in a deterministic runtime with "syscalls" back out to our database. The entire javascript function runs within one transaction close to the database.
The database tracks the read set at the given database timestamp in order to make subscriptions easy. We can quickly calculate if a mutation overlaps with any existing query subscriptions and proactively refresh them - thus giving you the behavior in
useQuery
that makes re-rendering in react easy.@nipunn yes, that makes sense to me. The thing that doesn't is that js engines, especially being "embedded" in the DB, can easily get out of memory on a large dataset. And I haven't found any docs on the limits...
But the most important part is that, if i'm not sure if pagination queries will work in this scenario of manual joins π€
our cofounder @james also gave a great talk explaining convex's architecture at the CMU database group here: https://www.youtube.com/watch?v=iizcidmSwJ4
CMU Database Group
YouTube
Convex: Life Without a Backend Team (James Cowling)
CMU Database Group - Vaccination Database Tech Talks - Second Dose (2021)
Speakers: James Cowling (Convex)
November 22, 2021
https://db.cs.cmu.edu/seminar2021-dose2#db12
Sponsors:
OtterTune (https://ottertune.com)
Steven Moy Foundation for Keeping it Real (https://stevenmoyfoundation.org)
here's some more on our limits today: https://docs.convex.dev/understanding/state#limits
Status and Guarantees | Convex Developer Hub
Convex is ready for production use. You can
and yeah, that's a great question around pagination. we currently only support paginating one layer of the relation: if you're querying a table
a
and then joining a one-to-many relation with some table b
, you can paginate the iteration on a
no problem.
so, if the relation is bounded, where there aren't too many b
s for a single a
, this will just work today. more complex multi-level pagination is on the roadmap but we're still working on the right way to express it in the query language. I'd be very curious if you have some example use cases we can talk through!Our docs on the pagination for a single table is here:
https://docs.convex.dev/using/pagination. It can be combined in a one-to-many case like @sujayakar described.
Would love to hear about use cases where that's insufficient.
It's tricky to represent such a multi-table complex pagination well (even in traditional sql) - and want to make sure we're designing for real world use cases.
thanks for clarifying! a roadmap for Convex, at least somewhat accurate, would be crazy helpful!
I'm not working on smth right now, just evaluating the options... And to be honest, EdgeDB blows my mind at the moment. I'm walking through this book of theirs https://www.edgedb.com/easy-edgedb/
Plus, i can clearly see the roadmap for the product in the coming months\years https://www.edgedb.com/roadmap
The only downside is that their cloud solutions is not live yet, plus a lot of code plumbing should be made vs. Convex + React tight integration(maybe Solid is coming, too π
)
You've an awesome tech underneath, but from the outsider it looks like a black box and I'm very afraid of black boxes when it comes to debugging smth π
@Dima Utkin re: solid, https://github.com/jamwt/solid-convex
GitHub
GitHub - jamwt/solid-convex: A simple app demonstrating using Solid...
A simple app demonstrating using SolidJS with Convex - GitHub - jamwt/solid-convex: A simple app demonstrating using SolidJS with Convex
@Dima Utkin if you have some ideas on what would make the box less "black", we're definitely open to them!
also, one challenge with the roadmap is it's pretty dynamic because we learn from developer's usage what's most useful and urgent to build next. based on our experience, it would be pretty bold of us to assume we know the best thing the developers that use Convex much beyond the next 3 months. we have a many-years list of things we intend to build, but the ordering is difficult to have confidence beyond a pretty short window
I can give you a couple of ideas
- once you have limits on some resources, listed on the Pricing page, you might wanna give the user a more detailed explanation on some queries(especially "complex"), like how much docs will be read, how updates will affect the price, what are the best practices to avoid going over the limit, etc.
- provide a way of making some queries non-realtime AND\OR manually invalidating the cache for those on the client side (see tanstack query). Not everything should be always realtime, imho π€
- build some tooling around these realtime updates for the browser, so the developer could see the whole flow, ideally having a way to go "back in time" to re-evaluate some data item, which might have caused an exception during the render, for example
- provide clearer docs on limitations, especially for complex queries like i've mentioned
- provide some options for solving problems with other\external tools, while Convex is not ready for them. Like folks at Firestore are writing tutorials for full-text search, distributed counters, aggregations, etc.
- and yeah, at least some kind of a roadmap is crucial, imho. I've been burned a couple of times on this one. I would never used Firestore years ago if I knew that implementing a
.count()
for queries would take 7-8 years... Or I would never use Elm, if I knew how that's gonna end up π
cool. great feedback, will take some time to digest! and to make more discoverable, since some of these things not being clear more easily is a problem, and some we're just not doing yet at all, which is fair
a few things that might help in the moment with you specifically, but we need to surface better...
not everything reactive is something we've thought about a little bit. I did this awhile ago: https://stack.convex.dev/coping-with-the-web-s-looming-global-reactivity-crisis
Managing Reactivity with useBufferedState
Reactivity has taken a dominant position today within web app development. Our components and app state are all reactive, and the world has adaptedβmo...
curious to know your thoughts on it if you ever get a chance -- reactive UI vs. "an interest in new data available" and how much patterns like I used there are interesting vs. no subscriptions at all. I'm on the fence myself, to be honest
re: pricing, right now pricing is very simple because we don't want to get too granular too early. so the answer is, the pricing doesn't vary depending on how complex a query is
we'd prefer to have predictable pricing for developers early and have the platform bear the costs of variance
we have a lot of ideas about the time travel thing, but none of them are implemented yet. convex's determinism makes things like replay.io patterns but for your entire backend really interesting. glad you're intrigued by this, we are too
external tools, our idea is you'd solve this with airbyte egress connector. we haven't created any good content guides demonstrating this yet, and we should do that soon. point taken
search specifically, we actually have a very interesting answer to that coming soon ( < 2 months)
in general hesitation about lock-in to bad philosophies is a point well taken, and probably the biggest single risk with anything like convex. we can't but agree with that. in some ways you're sort of betting on the team and the philosophy to be aligned with the kinds of things you care about and the kind of tradeoffs you'd like to make in your own architectures, and that's a very hard credibility to earn, but we're going to keep working on it. we'll think through how much an honest if short near term roadmap coming out ahead of time can help create that sense of directional alignment with current and prospective users
thanks again for the really thoughtful reply!
this one is in my "watch later" list π
well, you've an ability to make everything realtime. that's powerful!
but some times, all that we need is an "i'm ready for updates" or "gimme this one fresh". Again, like tanstack query does, refresh on window focus, for example, or manually invalidate the query...
yeah, agree. that's sort of the approach taken in that video, to use a React hook to buffer up the concept of "new data is available" and allow the app to decide when / if it appropriate to pass that through to the end component
to save some time, this is the pertinent hook: https://github.com/jamwt/convex-buffered-state/blob/main/pages/index.tsx#L9
GitHub
convex-buffered-state/index.tsx at main Β· jamwt/convex-buffered-state
Contribute to jamwt/convex-buffered-state development by creating an account on GitHub.
ah, this looks like a Twitter pattern... to prevent the flood of incoming tweets, UI jumps, etc. Nice!
yep
oh, and one more thing. non-reactive\fire-to-get-shaphot calls can really make a difference for SSR apps, that one was easily doable with Firestore and helped me to build a set of SEO-optimized pages in the past π
i guess, i'll need to have a look at convex/browser
yep, this is the sort of "standard" client: https://docs.convex.dev/api/classes/browser.ConvexHttpClient
Class: ConvexHttpClient | Convex Developer Hub
browser.ConvexHttpClient
that doesn't do the reactive stuff
but on the SSR front, we're going to be focusing on a really first class experience for Next SSR / ServerComponents over the next 3 months. right now we don't provide you much help for SSR
great, good to know!
Any chance you'd be willing & able to tease what you mean by this in a bit more detail? π
@alexcole @sujayakar are pretty deep into this right now, they're the right ones to share more
of course! we're working on full text search over your tables in convex, so it'll be super easy to build stuff like chat message search, searching through tasks in a todo app, etc. and, because it's convex, it's fast, fully consistent (no mismatches between your search results and "regular" database queries), and supports reactivity.
@RJ, very curious about example use cases you have in mind, what's important/not important for getting started, and so on. let me know!
Sounds really awesome! So would this first version be simply substring matching (not to belittle substring matchingβthat would be really useful), or would it include any kind of fuzzy-searching? If no fuzzy-searching, how likely would you be to support that in the future?
The use case I've had in mind is searching JSON-ified ProseMirror documents (serialized as strings at the moment because of https://discordapp.com/channels/1019350475847499849/1019350478817079338/1063132877669285948). The most useful and I think simplest thing I'd want to that end is substring matching in a single string field, but more sophisticated stuff would be useful as well of course!
Discord
Discord - A New Way to Chat with Friends & Communities
Discord is the easiest way to communicate over voice, video, and text. Chat, hang out, and stay close with your friends and communities.
yep, weβre starting with just exact word matching to start but will be fast following with prefix and fuzzy matching! the goal is to make something that solves the same needs as algolia or meilisearch.
our design takes a lot of influence from https://docs.meilisearch.com/learn/what_is_meilisearch/philosophy.html
Sounds fantastic, Iβm very excited to see this as it comes!