Hyper•2d ago

Convex Self-Hosted Storage Delete Failure (MinIO/Cloudflare)

Summary: File deletion removes metadata (Convex Dashboard UI updates) but the S3 object remains in MinIO. Relevant Details: * Setup: Self-hosted Convex via Docker, MinIO, with a Cloudflare-proxied HTTPS endpoint. * Works: Uploads/Downloads (GET/POST) are 100% functional. * Fails: Deletion. Files stay in the bucket (unversioned). * Debugging Proof: Manual external DELETE using the exact S3 credentials/endpoint (via Postman) was successful (HTTP 200). This rules out Cloudflare WAF, MinIO, and credential issues. * Problem: Backend logs show NO S3 DELETE request is ever attempted by the cleanup worker, even with RUST_LOG: debug on file_storage. The failure is silent and occurs before connection. Steps to Reproduce: Delete any file in the Convex Files dashboard. Language: Just using dashboard to test before implement Connection Check: Not applicable; the connection works for POST/GET, but the internal DELETE worker fails to fire. Any ideas why the cleanup hook fails silently when the connection is externally proven? 🙏

3 Replies

Convex Bot•2d ago

Thanks for posting in <#1088161997662724167>. Reminder: If you have a Convex Pro account, use the Convex Dashboard to file support tickets. - Provide context: What are you trying to achieve, what is the end-user interaction, what are you seeing? (full error message, command output, etc.) - Use search.convex.dev to search Docs, Stack, and Discord all at once. - Additionally, you can post your questions in the Convex Community's <#1228095053885476985> channel to receive a response from AI. - Avoid tagging staff unless specifically instructed. Thank you!

HyperOP•2d ago

MINIO:

  # ===================================
  # MINIO (S3)
  # ===================================
  convex-minio:
    image: minio/minio:latest
    command: server /data --console-address ":9001"
    environment:
      MINIO_ROOT_USER: minioadmin
      MINIO_ROOT_PASSWORD: MINIO_PASSWORD
      MINIO_BROWSER_REDIRECT_URL: https://ds01.MY_DOMAIN.COM
      MINIO_SERVER_URL: https://s01.my_domain.com/
    volumes:
      - convex-minio-data:/data
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 20s
    networks:
      - convex-net
      - tunnel

  # ===================================
  # MINIO (S3)
  # ===================================
  convex-minio:
    image: minio/minio:latest
    command: server /data --console-address ":9001"
    environment:
      MINIO_ROOT_USER: minioadmin
      MINIO_ROOT_PASSWORD: MINIO_PASSWORD
      MINIO_BROWSER_REDIRECT_URL: https://ds01.MY_DOMAIN.COM
      MINIO_SERVER_URL: https://s01.my_domain.com/
    volumes:
      - convex-minio-data:/data
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 20s
    networks:
      - convex-net
      - tunnel

Backend:

  # ===================================
  # CONVEX BACKEND
  # ===================================
  convex-backend:
    image: ghcr.io/get-convex/convex-backend:latest
    ports:
      - "3210:3210" # API
      - "3211:3211" # Websocket
    environment:
      # --- Instance identity ---
      INSTANCE_NAME: convex-prod
      INSTANCE_SECRET: CONVEX_INSTANCE_SECRET

      # --- Production URLs ---
      CONVEX_CLOUD_ORIGIN: https://convex.MY_DOMAIN.COM
      CONVEX_SITE_ORIGIN: https://convex.MY_DOMAIN.COM

      # --- Postgres ---
      POSTGRES_URL: postgresql://convex:SQL_PASSWORD@convex-postgres:5432

      # --- MinIO (S3) ---
      AWS_REGION: us-east-1
      AWS_ACCESS_KEY_ID: minioadmin
      AWS_SECRET_ACCESS_KEY: MINIO_PASSWORD
      
      AWS_S3_FORCE_PATH_STYLE: "true"
      AWS_S3_DISABLE_SSE: "true"

      S3_ENDPOINT_URL: http://convex-minio:9000
      S3_STORAGE_FILES_BUCKET: convex-files
      S3_STORAGE_MODULES_BUCKET: convex-modules
      S3_STORAGE_SEARCH_BUCKET: convex-search
      S3_STORAGE_EXPORTS_BUCKET: convex-exports
      S3_STORAGE_SNAPSHOT_IMPORTS_BUCKET: convex-snapshot-imports

      # --- Production ---
      DISABLE_BEACON: "true"
      REDACT_LOGS_TO_CLIENT: "true"
      RUST_LOG: info
      DO_NOT_REQUIRE_SSL: "false"

    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3210/version"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 60s

    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
        delay: 10s
        max_attempts: 5
      resources:
        limits:
          cpus: "2.0"
          memory: 4G
        reservations:
          cpus: "1.0"
          memory: 2G
      placement:
        constraints: [node.role == manager]
    networks:
      - convex-net
      - tunnel

  # ===================================
  # CONVEX BACKEND
  # ===================================
  convex-backend:
    image: ghcr.io/get-convex/convex-backend:latest
    ports:
      - "3210:3210" # API
      - "3211:3211" # Websocket
    environment:
      # --- Instance identity ---
      INSTANCE_NAME: convex-prod
      INSTANCE_SECRET: CONVEX_INSTANCE_SECRET

      # --- Production URLs ---
      CONVEX_CLOUD_ORIGIN: https://convex.MY_DOMAIN.COM
      CONVEX_SITE_ORIGIN: https://convex.MY_DOMAIN.COM

      # --- Postgres ---
      POSTGRES_URL: postgresql://convex:SQL_PASSWORD@convex-postgres:5432

      # --- MinIO (S3) ---
      AWS_REGION: us-east-1
      AWS_ACCESS_KEY_ID: minioadmin
      AWS_SECRET_ACCESS_KEY: MINIO_PASSWORD
      
      AWS_S3_FORCE_PATH_STYLE: "true"
      AWS_S3_DISABLE_SSE: "true"

      S3_ENDPOINT_URL: http://convex-minio:9000
      S3_STORAGE_FILES_BUCKET: convex-files
      S3_STORAGE_MODULES_BUCKET: convex-modules
      S3_STORAGE_SEARCH_BUCKET: convex-search
      S3_STORAGE_EXPORTS_BUCKET: convex-exports
      S3_STORAGE_SNAPSHOT_IMPORTS_BUCKET: convex-snapshot-imports

      # --- Production ---
      DISABLE_BEACON: "true"
      REDACT_LOGS_TO_CLIENT: "true"
      RUST_LOG: info
      DO_NOT_REQUIRE_SSL: "false"

    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3210/version"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 60s

    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
        delay: 10s
        max_attempts: 5
      resources:
        limits:
          cpus: "2.0"
          memory: 4G
        reservations:
          cpus: "1.0"
          memory: 2G
      placement:
        constraints: [node.role == manager]
    networks:
      - convex-net
      - tunnel

Dashboard:

  # ===================================
  # CONVEX DASHBOARD
  # ===================================
  convex-dashboard:
    image: ghcr.io/get-convex/convex-dashboard:latest
    ports:
      - "6791:6791"
    environment:
      NEXT_PUBLIC_DEPLOYMENT_URL: https://convex.MY_DOMAIN.COM
      CONVEX_BACKEND_URL: http://convex-backend:3210
      CONVEX_SELF_HOSTED_ADMIN_KEY: convex-prod|CONVEX_INSTANCE_SECRET
    deploy:
      replicas: 1
      resources:
        limits:
          cpus: "0.5"
          memory: 512M
    networks:
      - convex-net
      - tunnel

  # ===================================
  # CONVEX DASHBOARD
  # ===================================
  convex-dashboard:
    image: ghcr.io/get-convex/convex-dashboard:latest
    ports:
      - "6791:6791"
    environment:
      NEXT_PUBLIC_DEPLOYMENT_URL: https://convex.MY_DOMAIN.COM
      CONVEX_BACKEND_URL: http://convex-backend:3210
      CONVEX_SELF_HOSTED_ADMIN_KEY: convex-prod|CONVEX_INSTANCE_SECRET
    deploy:
      replicas: 1
      resources:
        limits:
          cpus: "0.5"
          memory: 512M
    networks:
      - convex-net
      - tunnel

SQL:

  # ===================================
  # POSTGRESQL + pgvector
  # ===================================
  convex-postgres:
    image: pgvector/pgvector:pg17
    environment:
      POSTGRES_USER: convex
      POSTGRES_PASSWORD: SQL_PASSWORD
      POSTGRES_DB: convex_prod
      PG_MAX_CONNECTIONS: 500
      PG_SHARED_BUFFERS: 512MB
      TZ: America/Sao_Paulo
    volumes:
      - convex-postgres-data:/var/lib/postgresql/data
    deploy:
      placement:
        constraints: [node.role == manager]
      replicas: 1
      restart_policy:
        condition: on-failure
        delay: 5s
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U convex -d convex_prod"]
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 30s
    networks:
      - convex-net

  # ===================================
  # POSTGRESQL + pgvector
  # ===================================
  convex-postgres:
    image: pgvector/pgvector:pg17
    environment:
      POSTGRES_USER: convex
      POSTGRES_PASSWORD: SQL_PASSWORD
      POSTGRES_DB: convex_prod
      PG_MAX_CONNECTIONS: 500
      PG_SHARED_BUFFERS: 512MB
      TZ: America/Sao_Paulo
    volumes:
      - convex-postgres-data:/var/lib/postgresql/data
    deploy:
      placement:
        constraints: [node.role == manager]
      replicas: 1
      restart_policy:
        condition: on-failure
        delay: 5s
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U convex -d convex_prod"]
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 30s
    networks:
      - convex-net

HyperOP•22h ago

Apparently, this is not a problem with my implementation or configuration. It is a current limitation of convex https://github.com/get-convex/convex-backend/issues/93

GitHub

logs and file are not being deleted on self-hosted · Issue #93 · ...

Actual blob files are not deleted from storage/files Also with a lot of updates db grows out of control, even deleting all tables, system tables take gigabytes.

Convex Self-Hosted Storage Delete Failure (MinIO/Cloudflare)

Did you find this page helpful?