Pagination, Take on a edge [ents]
I've been browsing and trying to figure out how to handle this for a few days. Basically i have a many to many between wallets and transactions.
The only efficient way to get the entire transaction history is to do something like .table('wallets')
.getX(wallet._id)
.edge('transactions')
But i cant do a pagination or a take or anything on that on a edge... so i assume anything i do is going to be a full read of all results. And can be thousands of transactions.
The problem is that transactions has a array of to's and a array of from's with a walletid/amount pairing. Which would be bad to index and we want to keep transactions as small as possible.
If i'm not missing something painfully simple, then i'm thinking directly accessing the linking (wallet_to_transaction) table would be smarter.
Perhaps then i could do manual pagination or something, or at least get arrays of ids to use?
Thanks
53 Replies
This is a bug / missing feature
https://github.com/xixixao/convex-ents/issues/25
See this thread: https://discord.com/channels/1019350475847499849/1235176712380354590/1235176712380354590
GitHub
many:many edge cannot be paginated · Issue #25 · xixixao/convex-ents
This const result = await ctx .table("messages") .getX(messageId) .edge("tags") .paginate(paginationOpts); should paginate the underlying edge table before loading the ents on t...
@Michal Srb i had seen that before. Back when i wasn't planning on many to manys. I haven't seen anything how to access the linking tables that ents generate from these relationships. But i also couldn't access the table from normal convex either. Is there a trick to get around the guard rails?
You can do something like this: https://labs.convex.dev/convex-ents/setup/config#incremental-adoption-restrict-ctxdb
Or to allow getting around it fully you can just not over-write
db at all, and access it like ctx.db.Configuring Functions - Convex Ents
Relations, default values, unique fields and more for Convex
or Michal's favorite: https://labs.convex.dev/convex-ents/setup/config#exposing-built-in-ctxdb-under-different-name
Configuring Functions - Convex Ents
Relations, default values, unique fields and more for Convex
I am doing this by just accessing the functions directly as its working with all my normal tables, its just like ctx.db is unaware the generated ents table exists. I'll try the skip version as i got that too.
I guess it was more of a typescirpt error with native convex, but is any the solution? await ctx.db
?.query('users_to_members' as any)
.collect()
If you don’t set db to anything it should have types for all the tables. Only if you do the Pick version should it only select a few tables, and it should be defined in the query/mutation for the custom function… not sure why that wouldn’t work. Just don’t cast db to DataModel or anything, it should have the types already- and hopefully within the custom function you could see it autocomplete with db.query(“
@ampp Yes,
ctx.db will not have the type for the many:many edge table, and as any is a fine workaround.Assuming pagination worked on a many to many from what i understand it would never support proper sorting as the linking table has a walletId_transactionId index and a transactionId_walletId index. So i can't get the last 10 transactions(and paginate) here either. Wouldn't this never be possible to implement with the current design constraints?
Last 10 transaction globally or with one of the IDs? Either way you can get it:
- Globally: Use the default _creationTime index (you have to do this manually via ctx.db right now)
- For some id: Use the relevant-direction index (you can use the Ents edge method)
I feel totally lost here, as long as i'm searching using a index i cant sort by creationTime?
ctx.table('wallets').get(wid).edge('walletTransactions') will return a ordered by _creationTime. granted all results. because i cant paginate or take.
as soon as i do this:
or
its unsorted. well first sorted by wallet then transaction then by _creationTime, how can i bypass that?
Ok two things that needs fixing I think:
1. Adding paginate: https://github.com/xixixao/convex-ents/issues/25
2. (this one is harder to work around) Right now there are two indexes, for both directions, but one of them has the other field as well for point lookup. But we probably want 3 indexes, one for each direction and one for lookup.
And you're probably hitting the 2nd case, right?
GitHub
many:many edge cannot be paginated · Issue #25 · xixixao/convex-ents
This const result = await ctx .table("messages") .getX(messageId) .edge("tags") .paginate(paginationOpts); should paginate the underlying edge table before loading the ents on t...
Yeah #2 for sure.
I think in my case i care less about traversing from walletTransactions to wallets as i have two arrays in my transactions table of from and to wallets.
If you swap the order of tables or edge declarations, you might coax the library to declare the other direction index to be the compound one.
Either way I added it to the paginate issue to fix this.
Id be happy with the partial fix of adding the index to the linking table. As far as i understand this is not something i can do.
For now, I think following a many to many relationship should come with a cavate that it will blow up at as soon as the relationship has 4096 records if its left without pagination/take ability
.table('wallets')
.getX(wallet._id)
.edge('walletTransactions')
@ampp both issues fixed, upgrade to
0.9.0
Also let me know if there are any other issues that are not tracked on GitHubAwesome, Thanks for the attention on this. I am getting a error with 0.9.0 on at least one of my many to many:
/facepalm
🙂 it happens lol
fixed 0.9.1
Nice, this week seems especially /facepalm for me, the most annoying one spending several hours trying to get a friends environment up for the 2nd time and ran out of time again. couldn't get any queries to load.
On Windows?
Yeah, with wsl
mutations worked however..
Maybe auth issue? Seems unlikely mutations would work but queries wouldn't
now im getting 400 Bad Request: IndexNotUnique: Indexes must be unique within a table. with 0.9.1
Yeah i tried everything, outside making a new clerk account redid stuff and reviewed things 4 times. made a new convex project too. we've probably done 10 clean slate setups and its only been this one that is trouble. Even queries that run with no login required were blank
Third time's the charm:
convex-ents@0.9.2Yes that did the trick! 20 new indexes 😅
Updated*
site is broken.. im investigating this
its on a many to 1 i think
It's a call to .edge or .edgeX, would be helpful to know which edge and what your schema looks like
something with
And your schema for members?
users to members is a many to many
And members to organization is?
members.edges('users').edge("orgs")
I got a repro, sec
my intent is to go through and optimize these queries, i guess now i do have the ability to be more specific with the new updates
I think members to users is actually our only case where we transverse the many to many in both directions right now for the fun of it 🙃 . A user can have many members within one org, and many orgs tied to the members. But users table is not aware of orgs. It will be a busy lookup.
convex-ents@0.9.3
(that was a bug I just introduced when implementing the pagination)
Getting greatly improved test coverage from all these 🙂Yeah, so far so good, i got to click on a few more pages for all the variations as our tests aren't testing most queries
well this one is harder to track down, still trying to figure it out
Looks like a dangling reference that isn't correctly checked for
Is that the complete stack trace? There should be a line that includes your code
(as opposed to the library code paths)
Yeah i cut off some of it, it might just be my problem, i got to wipe everything, i deleted something that left over data that didnt get a cascade
if i delete a member right now its a problem
Yeah still there should be a better error message
So I'd love to know exactly the scenario
(the rest of the stack trace would be helpful)
im reloading everything, it was referencing a document that was not in my db. It just seemed odd
well my init is throwing this error
our members have many roles with many permissions so its also crazy 😅
Ah, see it.
convex-ents@0.9.4Ok yeah, i just noticed the same one in the users to members relationship, and it appears fixed with 0.9.4
going to init again to see for sure
ya this one was my issue, im fairly sure.
If you have the full stack trace, would still be helpful to see it. The error message should say there's a dangling reference or throw earlier.
I would have to dig through the logs.
im confident it was a dangling reference
interesting, there may be a query issue with rendering the list off a edges, we have the ability to send a transaction to yourself (dont ask 😅 ), i may have to deal with it differently as it creates two transactions. Anyway, after i add this self transaction then when i send other transactions it populates the list that self transaction a new row each time i send as well as a the new transaction. im investigating.
ill first get rid of this extra transaction and see what happens after that
I think i at least managed to confuse the code that pushes these updates, i have yet to turn on pagination here
easier to display it, while im on this page i hit send for 16 and 17 and that 500 grew from 2 to 4 results and if i refresh it the 2 extra entries aren't there.
i don't think we want to use inverse on this table
I think I'm off here. I don't really understand the issue from the description above.
No worries, thanks for everything. I'll post a update to this as i play with the new pagination and such. if it ends up being a thing or not
Thank you!
the problem is easy to resolve, just don't have duplicates in your manytomany linking table. in wallets_walletTransactions i have two entries with matching walletId and walletTransactionId. when that condition exists, every time you add something to those tables additional duplicate entry row is displayed on the UI for every duplicate entry shown. im testing this on non ents right now
Yeah Ents should never create two such edges, but you got into that situation with the last bug
With it fixed, you should not get into that situation again
i needed a check there anyway 😅
convex-ents@0.9.5
There was another bug for symmetrical many:many edges.
The pagination fix caused a lot of issues because the index usage is not strongly typed inside the library implementation (and convex-test wasn't enforcing many of the constraints the real backend did around schema and indexes - I fixed that too now).