fraktall•2w ago

Hey :hellowave: I'm self-hosting Convex

Hey :hellowave: I'm self-hosting Convex and my server triggered a disk space warning. After investigating, I found that /convex/data/tmp inside the Convex backend container had over 434,000 directories :pepeomg: which massively inflated inode usage and filled the disk. Any advice on what Convex stores there, and how to prevent this buildup?

9 Replies

nipunn•2w ago

Sup. What are you seeing in there? Do you have any more information?

fraktallOP•2w ago

hey, so I have Convex deployed on a server under Coolify management Here's docker compose

services:
  backend:
    image: 'ghcr.io/get-convex/convex-backend:4499dd4fd7f2148687a7774599c613d052950f46'
    volumes:
      - 'data:/convex/data'
    environment:
      - SERVICE_FQDN_BACKEND_3210
      - 'INSTANCE_NAME=${INSTANCE_NAME:-self-hosted-convex}'
      - 'INSTANCE_SECRET=${SERVICE_HEX_32_SECRET}'
      - 'CONVEX_RELEASE_VERSION_DEV=${CONVEX_RELEASE_VERSION_DEV:-}'
      - 'ACTIONS_USER_TIMEOUT_SECS=${ACTIONS_USER_TIMEOUT_SECS:-}'
      - 'CONVEX_CLOUD_ORIGIN=${SERVICE_FQDN_BACKEND_3210}'
      - 'CONVEX_SITE_ORIGIN=${SERVICE_FQDN_BACKEND_3210}/http'
      - 'DATABASE_URL=${DATABASE_URL:-}'
      - 'DISABLE_BEACON=${DISABLE_BEACON:-}'
      - 'REDACT_LOGS_TO_CLIENT=${REDACT_LOGS_TO_CLIENT:-}'
      - 'CONVEX_SELF_HOSTED_URL=${SERVICE_FQDN_CONVEX_6791}'
    healthcheck:
      test: 'curl -f http://127.0.0.1:3210/version'
      interval: 5s
      start_period: 5s
  dashboard:
    image: 'ghcr.io/get-convex/convex-dashboard:4499dd4fd7f2148687a7774599c613d052950f46'
    environment:
      - SERVICE_FQDN_CONVEX_6791
      - NEXT_PUBLIC_DEPLOYMENT_URL=$SERVICE_FQDN_BACKEND_3210
    depends_on:
      backend:
        condition: service_healthy
    healthcheck:
      test: 'wget -qO- http://127.0.0.1:6791/'
      interval: 5s
      start_period: 5s

services:
  backend:
    image: 'ghcr.io/get-convex/convex-backend:4499dd4fd7f2148687a7774599c613d052950f46'
    volumes:
      - 'data:/convex/data'
    environment:
      - SERVICE_FQDN_BACKEND_3210
      - 'INSTANCE_NAME=${INSTANCE_NAME:-self-hosted-convex}'
      - 'INSTANCE_SECRET=${SERVICE_HEX_32_SECRET}'
      - 'CONVEX_RELEASE_VERSION_DEV=${CONVEX_RELEASE_VERSION_DEV:-}'
      - 'ACTIONS_USER_TIMEOUT_SECS=${ACTIONS_USER_TIMEOUT_SECS:-}'
      - 'CONVEX_CLOUD_ORIGIN=${SERVICE_FQDN_BACKEND_3210}'
      - 'CONVEX_SITE_ORIGIN=${SERVICE_FQDN_BACKEND_3210}/http'
      - 'DATABASE_URL=${DATABASE_URL:-}'
      - 'DISABLE_BEACON=${DISABLE_BEACON:-}'
      - 'REDACT_LOGS_TO_CLIENT=${REDACT_LOGS_TO_CLIENT:-}'
      - 'CONVEX_SELF_HOSTED_URL=${SERVICE_FQDN_CONVEX_6791}'
    healthcheck:
      test: 'curl -f http://127.0.0.1:3210/version'
      interval: 5s
      start_period: 5s
  dashboard:
    image: 'ghcr.io/get-convex/convex-dashboard:4499dd4fd7f2148687a7774599c613d052950f46'
    environment:
      - SERVICE_FQDN_CONVEX_6791
      - NEXT_PUBLIC_DEPLOYMENT_URL=$SERVICE_FQDN_BACKEND_3210
    depends_on:
      backend:
        condition: service_healthy
    healthcheck:
      test: 'wget -qO- http://127.0.0.1:6791/'
      interval: 5s
      start_period: 5s

I have 4 Convex crons running (every 1 min, every 5 min and 2x every 30min) After running the service for around 24 hours, I started getting warnings that the available disk space was running low I looked into it and found out the backend container volume had blown up Narrowed it down to /convex/dev/tmp folder

fraktallOP•2w ago

Here's the snapshot of what's inside the folder

fraktallOP•2w ago

Here's what’s in one of those folders that keeps getting created (they’re not actually empty like I said at first)

fraktallOP•2w ago

When I open the modules folder I only see compiled JS from my source files

fraktallOP•2w ago

And cat package.json shows:

{ "type": "module" }

{ "type": "module" }

And yeah it's the same in every folder I set up a cronjob that wipes everything from /convex/data/tmp every 6 hours (except the .tmp* folders) as a temporary fix, but the underlying issue is still there

fraktallOP•2w ago

ordered by date

fraktallOP•2w ago

Around 5 to 10 folders get created every minute hey @nipunn :wave: any advice on the above?

nipunn•5d ago

I'm not really able to to repro what's going on. It is the TMPDIR set, so any of your node actions that write to a temp dir could be writing there. Maybe poke around inside the ones being created to see what's going on. I think you'll have to debug further or another route could be to make a new container and see if the problem reproes there - if not, narrow down the difference between your container and the new one (maybe some of the code you are running, maybe something else?)

Hey :hellowave: I'm self-hosting Convex

Did you find this page helpful?