Leon (Synth)•2mo ago

Help debugging custom JWT auth

Hi, I am trying to use custom JWT auth with my convex database. I have an endpoint that generates a valid JWT (see screenshot below) and I have configured the auth.config.ts in the convex directory. However, the onChange callback for setAuth is always returning false. The convex instance is a local development instance created with npx convex dev --local --once, so should be able to access the well-known? convex/auth.config.ts:

import { config } from "$lib/shared/config";

export default {
    providers: [
        {
            type: "customJwt",
            applicationId: config.domain,
            issuer: "http://localhost:5173",
            jwks: "http://localhost:5173/.well-known/jwks.json",
            algorithm: "RS256"
        }
    ]
};

import { config } from "$lib/shared/config";

export default {
    providers: [
        {
            type: "customJwt",
            applicationId: config.domain,
            issuer: "http://localhost:5173",
            jwks: "http://localhost:5173/.well-known/jwks.json",
            algorithm: "RS256"
        }
    ]
};

setAuth call:

client.setAuth(
    async ({ forceRefreshToken }) => {
        let token = localStorage.getItem(TOKEN_KEY);
        if (!forceRefreshToken && token) {
            return token;
        }

        const response = await api.request(api.v1.auth.token, {
            method: "GET",
            input: {}
        });

        console.log(`Fetch token (force: ${forceRefreshToken}), response:`, response);

        if (!response.ok) {
            console.error("Failed to fetch a new token", response);
            return null;
        }

        token = response.value.token;
        if (token) {
            localStorage.setItem(TOKEN_KEY, token);
        }

        return token;
    },
    (isAuthenticated) => {
        console.log("Auth changed, authenticated:", isAuthenticated); // always false
        authenticated = isAuthenticated;
    }
);

client.setAuth(
    async ({ forceRefreshToken }) => {
        let token = localStorage.getItem(TOKEN_KEY);
        if (!forceRefreshToken && token) {
            return token;
        }

        const response = await api.request(api.v1.auth.token, {
            method: "GET",
            input: {}
        });

        console.log(`Fetch token (force: ${forceRefreshToken}), response:`, response);

        if (!response.ok) {
            console.error("Failed to fetch a new token", response);
            return null;
        }

        token = response.value.token;
        if (token) {
            localStorage.setItem(TOKEN_KEY, token);
        }

        return token;
    },
    (isAuthenticated) => {
        console.log("Auth changed, authenticated:", isAuthenticated); // always false
        authenticated = isAuthenticated;
    }
);

Any help is appreciated!!

No description

25 Replies

Leon (Synth)OP•2mo ago

Checking the authentication tab in the instance dashboard it looks like the provider details from auth.config.ts are missing, but I am unsure why?

No description

Jakov•2mo ago

If the npx convex dev --local --once didn't show an error with any file, it should work @Leon (Synth)

Leon (Synth)OP•2mo ago

Yeah, the dev command is running no problem

No description

Leon (Synth)OP•2mo ago

Checking the websocket messages it is returning an {"type":"AuthError","error":"Could not decode token","baseVersion":0,"authUpdateAttempted":true}, which looks to be this line: https://github.com/get-convex/convex-backend/blob/63efd5913af60fa93dc4e8147d47ccd29c72edb9/crates/authentication/src/lib.rs#L246 Exactly why it failed to decode the token, it doesn't seem to make obvious?

erquhart•2mo ago

That line doesn't look like the same line, do you see an error about AuthHeader?

Leon (Synth)OP•2mo ago

This is all of the messages I see in the websocket for both connection attempts:

No description

Leon (Synth)OP•2mo ago

Interesting point though, that that error doesn't seem to exist on main anywhere with the AuthError type, as far as I can see Is the provider not showing in the dashboard an unrelated bug or could this be causing it?

erquhart•2mo ago

ah yeah, that's strange Is this dev?

Leon (Synth)OP•2mo ago

This is dev I have a self hosted instance but it obviously wouldn't be able to reach the local jwks

erquhart•2mo ago

Can you try with a cloud deployment Betting this is a local hosting bug

Leon (Synth)OP•2mo ago

Before attempting with a cloud deployment, just checked it with a local running verison of the self-hosted images and I get the exact same error. docker.compose.yml:

services:
  db:
    image: postgres:17-alpine
    container_name: scorgportal-db
    restart: always
    ports:
      - "8420:5432"
    volumes:
      - pg-data:/var/lib/postgresql/data
    environment:
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
      - POSTGRES_DB=scorgportal
    networks:
      - dev-services

    healthcheck:
      test: ["CMD-SHELL", "sh -c 'pg_isready -U postgres -d scorgportal'"]
      interval: 10s
      timeout: 3s
      retries: 3

  backend:
    image: ghcr.io/get-convex/convex-backend:6efab6f2b6c182b90255774d747328cfc7b80dd9
    container_name: scorgportal-convex-backend
    stop_grace_period: 10s
    stop_signal: SIGINT
    ports:
      - "8421:3210"
      - "8422:3211"
    volumes:
      - convex-data:/convex/data
    environment:
      - INSTANCE_NAME=scorgportal
      - INSTANCE_SECRET=
      - CONVEX_RELEASE_VERSION_DEV=
      - ACTIONS_USER_TIMEOUT_SECS=
      - CONVEX_CLOUD_ORIGIN=http://127.0.0.1:8421
      - CONVEX_SITE_ORIGIN=http://127.0.0.1:8422
      - DATABASE_URL=
      - DISABLE_BEACON=true
      - REDACT_LOGS_TO_CLIENT=
      - DO_NOT_REQUIRE_SSL=1
      - POSTGRES_URL=postgresql://postgres:postgres@db:5432
      - MYSQL_URL=
      - RUST_LOG=info
      - RUST_BACKTRACE=
      - AWS_REGION=
      - AWS_ACCESS_KEY_ID=
      - AWS_SECRET_ACCESS_KEY=
      - AWS_SESSION_TOKEN=
      - S3_STORAGE_EXPORTS_BUCKET=
      - S3_STORAGE_SNAPSHOT_IMPORTS_BUCKET=
      - S3_STORAGE_MODULES_BUCKET=
      - S3_STORAGE_FILES_BUCKET=
      - S3_STORAGE_SEARCH_BUCKET=
      - S3_ENDPOINT_URL=
    networks:
      - dev-services
    depends_on:
      db:
        condition: service_healthy

    healthcheck:
      test: curl -f http://localhost:3210/version
      interval: 5s
      start_period: 10s

  dashboard:
    image: ghcr.io/get-convex/convex-dashboard:6efab6f2b6c182b90255774d747328cfc7b80dd9
    container_name: scorgportal-convex-dashboard
    stop_grace_period: 10s
    stop_signal: SIGINT
    ports:
      - "8423:6791"
    environment:
      - NEXT_PUBLIC_DEPLOYMENT_URL=http://127.0.0.1:8421
    networks:
      - dev-services
    depends_on:
      backend:
        condition: service_healthy

volumes:
  convex-data:
  pg-data:

networks:
  dev-services:
    name: scorgportal-dev-services

services:
  db:
    image: postgres:17-alpine
    container_name: scorgportal-db
    restart: always
    ports:
      - "8420:5432"
    volumes:
      - pg-data:/var/lib/postgresql/data
    environment:
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
      - POSTGRES_DB=scorgportal
    networks:
      - dev-services

    healthcheck:
      test: ["CMD-SHELL", "sh -c 'pg_isready -U postgres -d scorgportal'"]
      interval: 10s
      timeout: 3s
      retries: 3

  backend:
    image: ghcr.io/get-convex/convex-backend:6efab6f2b6c182b90255774d747328cfc7b80dd9
    container_name: scorgportal-convex-backend
    stop_grace_period: 10s
    stop_signal: SIGINT
    ports:
      - "8421:3210"
      - "8422:3211"
    volumes:
      - convex-data:/convex/data
    environment:
      - INSTANCE_NAME=scorgportal
      - INSTANCE_SECRET=
      - CONVEX_RELEASE_VERSION_DEV=
      - ACTIONS_USER_TIMEOUT_SECS=
      - CONVEX_CLOUD_ORIGIN=http://127.0.0.1:8421
      - CONVEX_SITE_ORIGIN=http://127.0.0.1:8422
      - DATABASE_URL=
      - DISABLE_BEACON=true
      - REDACT_LOGS_TO_CLIENT=
      - DO_NOT_REQUIRE_SSL=1
      - POSTGRES_URL=postgresql://postgres:postgres@db:5432
      - MYSQL_URL=
      - RUST_LOG=info
      - RUST_BACKTRACE=
      - AWS_REGION=
      - AWS_ACCESS_KEY_ID=
      - AWS_SECRET_ACCESS_KEY=
      - AWS_SESSION_TOKEN=
      - S3_STORAGE_EXPORTS_BUCKET=
      - S3_STORAGE_SNAPSHOT_IMPORTS_BUCKET=
      - S3_STORAGE_MODULES_BUCKET=
      - S3_STORAGE_FILES_BUCKET=
      - S3_STORAGE_SEARCH_BUCKET=
      - S3_ENDPOINT_URL=
    networks:
      - dev-services
    depends_on:
      db:
        condition: service_healthy

    healthcheck:
      test: curl -f http://localhost:3210/version
      interval: 5s
      start_period: 10s

  dashboard:
    image: ghcr.io/get-convex/convex-dashboard:6efab6f2b6c182b90255774d747328cfc7b80dd9
    container_name: scorgportal-convex-dashboard
    stop_grace_period: 10s
    stop_signal: SIGINT
    ports:
      - "8423:6791"
    environment:
      - NEXT_PUBLIC_DEPLOYMENT_URL=http://127.0.0.1:8421
    networks:
      - dev-services
    depends_on:
      backend:
        condition: service_healthy

volumes:
  convex-data:
  pg-data:

networks:
  dev-services:
    name: scorgportal-dev-services

auth.config.ts updated to point at host:

export default {
    providers: [
        {
            type: "customJwt",
            issuer: "http://localhost:5173",
            jwks: "http://host.docker.internal:5173/.well-known/jwks.json",
            algorithm: "RS256"
        }
    ]
};

export default {
    providers: [
        {
            type: "customJwt",
            issuer: "http://localhost:5173",
            jwks: "http://host.docker.internal:5173/.well-known/jwks.json",
            algorithm: "RS256"
        }
    ]
};

Will give it a go with a cloud deployment now, just need to get the jwks up somewhere

Leon (Synth)OP•2mo ago

@erquhart can confirm, I even get the error when using convex cloud

No description

Leon (Synth)OP•2mo ago

Leon (Synth)OP•2mo ago

Here is how I am generating the token, if it helps:

const token = jwt.sign(
    {
        user
    },
    env.TOKEN_PRIVATE_KEY.replace(/\\n/g, "\n"),
    {
        algorithm: "RS256",
        issuer: "https://dev.scorgportal.com",
        expiresIn: "6H",
        audience: "scorgportal",
        subject: user
    }
);

const token = jwt.sign(
    {
        user
    },
    env.TOKEN_PRIVATE_KEY.replace(/\\n/g, "\n"),
    {
        algorithm: "RS256",
        issuer: "https://dev.scorgportal.com",
        expiresIn: "6H",
        audience: "scorgportal",
        subject: user
    }
);

But I'd assume if jwt.io says it is a valid token then there is nothing wrong there bump

Leon (Synth)OP•2mo ago

So it looks like I have managed to get past the Could not decode token error by adding the key id header to the JWT. Now, I have seen it successfully authenticated a few times, but it seems to be extremely inconsistent. Most of the time I am now getting this error: Could not validate token Which is further down in the auth method, so, progress!: https://github.com/get-convex/convex-backend/blob/63efd5913af60fa93dc4e8147d47ccd29c72edb9/crates/authentication/src/lib.rs#L304

No description

Leon (Synth)OP•2mo ago

Ok, so... just adding a 1 second timeout before returning the token fromsetAuth seems to have fixed the inconsistencies... 😅

async ({ forceRefreshToken }) => {
    const response = await svelteApi.request(svelteApi.v1.auth.token, {
        method: "GET",
        credentials: "include",
        cache: "no-store"
    });
    if (!response.ok) {
        console.error("Failed to fetch a new token", response);
        return null;
    }

    const token = response.value.token;
    if (dev) {
        console.log(`fetchToken (force: ${forceRefreshToken}):`, token);
    }

    await new Promise((resolve) => setTimeout(resolve, 1000));

    return token;
}

async ({ forceRefreshToken }) => {
    const response = await svelteApi.request(svelteApi.v1.auth.token, {
        method: "GET",
        credentials: "include",
        cache: "no-store"
    });
    if (!response.ok) {
        console.error("Failed to fetch a new token", response);
        return null;
    }

    const token = response.value.token;
    if (dev) {
        console.log(`fetchToken (force: ${forceRefreshToken}):`, token);
    }

    await new Promise((resolve) => setTimeout(resolve, 1000));

    return token;
}

Could this be that the tolerance on the iat claim is too strict?

Leon (Synth)OP•2mo ago

Opened an issue for investigation: https://github.com/get-convex/convex-backend/issues/139

GitHub

Too tight tolerance on customJwt iat claim and kid header seems...

Hi, I spent the most part of yesterday evening and this morning troubleshooting as to why my custom JWT was failing to authenticate with convex. You can find my troubleshooting process in this #sup...

erquhart•2mo ago

Glad you figured all this out - the iat thing is interesting, thanks for opening an issue on it. Waiting a full second is rough, maybe a shorter one will be a working solution until this is actually resolved?

Leon (Synth)OP•2mo ago

I ended up doing the diff between the current time and the issued at time so it isn't always a full second but it seems like it needs to be at least a second since the iat for it to work consistently

erquhart•2mo ago

I haven't really seen this with other jwts, any chance your jwt generation is off a bit? Agree that maybe some tolerance in iat validation would help, but again, haven't seen this in other implementations before.

Leon (Synth)OP•2mo ago

I mean, maybe? This is how I am generating the JWT with jose:

const privateKey = await jose.importPKCS8(env.JWT_PRIVATE_KEY.replace(/\\n/g, "\n"), "RS256");

const issuedAt = new Date();
const token = await new jose.SignJWT({
    userId: event.locals.session.user
})
    .setProtectedHeader({
        alg: "RS256",
        kid: env.JWT_KEY_ID
    })
    .setIssuedAt(issuedAt)
    .setIssuer(env.JWT_ISSUER)
    .setAudience(config.domain)
    .setSubject(event.locals.session.key)
    .setExpirationTime("6h")
    .sign(privateKey);

const privateKey = await jose.importPKCS8(env.JWT_PRIVATE_KEY.replace(/\\n/g, "\n"), "RS256");

const issuedAt = new Date();
const token = await new jose.SignJWT({
    userId: event.locals.session.user
})
    .setProtectedHeader({
        alg: "RS256",
        kid: env.JWT_KEY_ID
    })
    .setIssuedAt(issuedAt)
    .setIssuer(env.JWT_ISSUER)
    .setAudience(config.domain)
    .setSubject(event.locals.session.key)
    .setExpirationTime("6h")
    .sign(privateKey);

Leon (Synth)OP•2mo ago

My system time doesn't seem to be out of sync

No description

Leon (Synth)OP•2mo ago

I don't know what would cause the generation to be off?

erquhart•2mo ago

Is your key generation running in Convex? Same function or different?

Leon (Synth)OP•2mo ago

Key generation is running outside of convex in a normal http endpoint (specifically a sveltekit route)

Convex Community Join

Join the Convex Discord! Explore Convex, the open-source reactive database for app developers.

11KMembers

View on Discord

Did you find this page helpful?