Self-hosted convex backend suddenly crashed (sorry, too many clients already)
🪲Bug Report
Convex Version: 1.29.3
Convex backend image hash:
Convex backend image GitHub version tag:
Hosting environment: Railway
Connected database: Postgres
Description: The convex backend has been running stably for several days, and it crashed suddenly on Sunday. The last error message in the log seems to indicate that it is caused by some failures in the actions, but we are still investigating if this is the cause. However, as far as I know, actions are run in a separate runtime that should not crash the whole convex backend. The log is flooded by the Postgres error message "sorry, too many clients already". I don't know the implementation of convex and I am not an expert in Postgres so I am not sure what this error means to us.
Attachment is the console log of the self-hosted convex backend runtime.
Convex backend image hash:
sha256:b258fc15dcc4ab84edcb36e785e985fbd3eb8dcc509a562fd1ee67d50e0ae821sha256:b258fc15dcc4ab84edcb36e785e985fbd3eb8dcc509a562fd1ee67d50e0ae821Convex backend image GitHub version tag:
69a1969a1f307edd7a6e452606925bb004ab362269a1969a1f307edd7a6e452606925bb004ab3622Hosting environment: Railway
Connected database: Postgres
Description: The convex backend has been running stably for several days, and it crashed suddenly on Sunday. The last error message in the log seems to indicate that it is caused by some failures in the actions, but we are still investigating if this is the cause. However, as far as I know, actions are run in a separate runtime that should not crash the whole convex backend. The log is flooded by the Postgres error message "sorry, too many clients already". I don't know the implementation of convex and I am not an expert in Postgres so I am not sure what this error means to us.
Attachment is the console log of the self-hosted convex backend runtime.
log.txt4.25KB
