Instance taking very long to boot and when it boots it's extremely slow?
If someone can help me troubleshoot why my instance is taking very very long to boot up I'd appreciate
It's using sqlite and local store for files and I'm storing quite a lot of files, database is 4GB~ and the data folder of the container is around 8.3GB
I want to know what's this searchlight step taking upwards of 30 minutes to init, I'm guessing it has something to do with searchIndexes? Is this normal?
6 Replies
I exported the compose volume and loaded it on my personal computer with significantly more hardware and it still has the slow start up but in about 5 minutes it got the ports going, however when I go to check what resources it's using it's chugging on ram pretty hard and doing a lot of IO (my local instance with 0 clients, not even a test client connected to it, no crons, no schedules either)
I still will appreciate if someone has any insight as to what's going on here, (specially what caused it and how to avoid this or any general recommendation) my next step is to wait out on my local machine see if it finishes doing whatever ritual it's doing so that hopefully I can just export the image back into the VPS and call it a day
I found out that when the search folder gets full of these files sometimes you get a crash and then the instance slows down to a cripple, moving these files away and restarting the instance makes it stop being ultra slow;
I'm hosting my application on my own PC for the day and it managed to stop working a few times; Granted, its running on Docker Desktop for Windows which is sub-optimal so it could be Docker's fault there; But it does mold into any crevice of ram that is apparently available (which I think is great, free ram is wasted ram), but it clings to it forever and has no cache eviction configuration options or "planned max ram consumption" anywhere
Not sure if you already tried it but consider migrating over to MySQL or Postgres. When I first started testing Self Hosted Convex I had many performance issues with SQLite, even on a fast system with NVME storage. After importing ~7million rows (took over 2 days), the backend was extremely unreliable and would often crash out of nowhere. The dashboard also became completely unusable, taking upwards of a minute switching between pages.
I then migrated over to Postgres and all my problems went away. I have recently moved again over to MySQL as it performed better when it came to storage efficiency during testing.
You mentioned you are using Docker on windows, I am going to assume it is using the WSL2 engine. That is known for having extremely poor IOPS between the host machines filesystem and the container. Are you mounting your convex data folder as a local volume?
Also, the errors in your first screenshot. I also get them on startup - and have done since I started using Convex. I have just assumed it is caused by some of the internal services being a little overly excited to start up without properly waiting for the backend to complete its initial checks. I have just ignored them and never run into any further problems. If this is a sign of some misconfiguration I would also appreciate a convex pro letting me know. 😁
tyvm for the input, that is very true; I was hosting it on my PC for the friday because I needed the application to be at least somewhat operational right then so my temporary workaround was to SSH tunnel my VPS to my own ports 3210 and 3211, in this way the remote reverse proxy directed traffic to my PC for that temporary workaround
I found out that indeed the Convex backend consumes significantly less ram when configured to use postgres and runs extraordinarily faster, I was having trouble migrating my data to postgres because export was taking an eternity to complete, so I asked the GPT to write me a python script to migrate the data and it worked out of the box (ill append the script here for archive)
Then since export is not working for me rn, I guess I have to manually move the files to the cloud storage, but it looks like it should be quite simple actually
TBH through this process I was kinda scared that I might have chosen the wrong technology, but almost all of my problem is attributed to not configuring an external db earlier