Self-hosted convex backend suddenly crashed (sorry, too many clients already)
Convex Version: 1.29.3
Convex backend image hash:
sha256:b258fc15dcc4ab84edcb36e785e985fbd3eb8dcc509a562fd1ee67d50e0ae821
Convex backend image GitHub version tag: 69a1969a1f307edd7a6e452606925bb004ab3622
Hosting environment: Railway
Connected database: Postgres
Description: The convex backend has been running stably for several days, and it crashed suddenly on Sunday. The last error message in the log seems to indicate that it is caused by some failures in the actions, but we are still investigating if this is the cause. However, as far as I know, actions are run in a separate runtime that should not crash the whole convex backend. The log is flooded by the Postgres error message "sorry, too many clients already". I don't know the implementation of convex and I am not an expert in Postgres so I am not sure what this error means to us.
Attachment is the console log of the self-hosted convex backend runtime.1 Reply
After some basic investigation, it seems it may be caused by exceeding the function argument limitation (16 MiB)
It seems this issue happens whenever the database is handling a large amount of mutations simultaneously.