MapleLeaf 🍁
MapleLeaf 🍁•13mo ago

e2e testing with preview deployments

I'm trying to get a setup where I can run e2e tests in Github CI against a preview environment. Here's what I have: 
      - name: Build with Convex deploy
        run: pnpm convex deploy
          --cmd "pnpm run build"
          --cmd-url-env-var-name VITE_PUBLIC_CONVEX_URL
          --preview-name "e2e-${{ github.ref_name }}"
        env:
          CONVEX_DEPLOY_KEY: ${{ secrets.CONVEX_DEPLOY_KEY }}

      - name: Run Playwright tests
        run: pnpm run e2e
        env:
          CLERK_PUBLISHABLE_KEY: ${{ secrets.CLERK_PUBLISHABLE_KEY }}
          CLERK_SECRET_KEY: ${{ secrets.CLERK_SECRET_KEY }}
      - name: Build with Convex deploy
        run: pnpm convex deploy
          --cmd "pnpm run build"
          --cmd-url-env-var-name VITE_PUBLIC_CONVEX_URL
          --preview-name "e2e-${{ github.ref_name }}"
        env:
          CONVEX_DEPLOY_KEY: ${{ secrets.CONVEX_DEPLOY_KEY }}

      - name: Run Playwright tests
        run: pnpm run e2e
        env:
          CLERK_PUBLISHABLE_KEY: ${{ secrets.CLERK_PUBLISHABLE_KEY }}
          CLERK_SECRET_KEY: ${{ secrets.CLERK_SECRET_KEY }}
I want my tests to run seed functions before they go, e.g. 
test.beforeAll(async ({ page }) => {
    await execa("pnpm", ["convex", "run", "seed:characters"], {
        stdio: "inherit",
    })
})

test("navigating characters", async ({ page }) => {
    // etc
})
test.beforeAll(async ({ page }) => {
    await execa("pnpm", ["convex", "run", "seed:characters"], {
        stdio: "inherit",
    })
})

test("navigating characters", async ({ page }) => {
    // etc
})
this doesn't work in CI because the tests are ran outside the convex deploy context and don't have the CONVEX_DEPLOYMENT environment variable I can --preview-run seed:characters, but I want to be able to run specific functions for specific tests to replicate certain setup scenarios without the test going through the UI to do so so my question is: how can I pass the info from convex deploy such that I can run functions during the tests?
17 Replies
MapleLeaf 🍁
MapleLeaf 🍁OP•13mo ago
although maybe I can approach this differently 🤔 I could hit an API route in my app like /setup/characters, which would then run that function in the backend and probably checks some conditional PREVIEW=true env to make sure it can't just be run arbitrarily
MapleLeaf 🍁
MapleLeaf 🍁OP•13mo ago
sshader
sshader•13mo ago
I was about to suggest something similar to what you have (similar thread -- https://discord.com/channels/1019350475847499849/1174084317677371604/1174374607910871082) If we added something like npx convex run "seed:characters" --preview-name "e2e-..." or npx convex run "seed:characters" --url VITE_PUBLIC_CONVEX_URL which required a CONVEX_DEPLOY_KEY, would that work here? Seems like we need to know which deployment to run the function on (either via the name or via the URL) + some proof that you should be allowed to run (internal) functions on this deployment (CONVEX_DEPLOY_KEY seems easiest since it's static across a project)
Discord
Discord - A New Way to Chat with Friends & Communities
Discord is the easiest way to communicate over voice, video, and text. Chat, hang out, and stay close with your friends and communities.
MapleLeaf 🍁
MapleLeaf 🍁OP•13mo ago
If we added something like npx convex run "seed:characters" --preview-name "e2e-..." or npx convex run "seed:characters" --url VITE_PUBLIC_CONVEX_URL which required a CONVEX_DEPLOY_KEY, would that work here?
yeah that would probably work :meowthumbsup: although maybe it could be possible to create an authoritative instance of ConvexHttpClient? one that can run internal mutations, given a deploy key that'd be more straightforward and one less dependency than having to run a child process to run a function
ballingt
ballingt•13mo ago
This exists today but is undocumented/unstable, it might be fine to stabilize it. https://github.com/get-convex/convex-js/blob/32ff1dc1190e35223bda44c41f51a237960da3e0/src/browser/http_client.ts#L83
MapleLeaf 🍁
MapleLeaf 🍁OP•13mo ago
you have no idea how long I've needed this lmao
ballingt
ballingt•13mo ago
It's not likely to go away, a possible behavior change is how it interacts with httpClient.setAuth() since currently you can't use both. The behavior of setting this HTTP header wont' change because we have to support old CLIs. so you're welcome to use this, worst case is you need to make the HTTP request yourself if we remove it. I don't think we will, we just might clean up some language around "admin keys" vs "deploy keys"
MapleLeaf 🍁
MapleLeaf 🍁OP•13mo ago
actually, the issue still remains that I can't get the VITE_PUBLIC_CONVEX_URL variable for the preview after the build step happens, unless I'm missing something? 🤔
ballingt
ballingt•13mo ago
Is that sitting in .env.local?
MapleLeaf 🍁
MapleLeaf 🍁OP•13mo ago
well, that doesn't exist in CI what I mean is, in CI, it runs convex deploy ... to run the build and deploy the functions, but I'm not sure how to get the preview URL from that point to run the tests
ballingt
ballingt•13mo ago
(adding CLI tools for all these things would be really reasonable, let's say we're just figuring out how to unblock you for now) after this runs isn't there a .env.local with the new VITE_PUBLIC_CONVEX_URL in it? and this is GitHub CI, so you could grab this for a future step here?
MapleLeaf 🍁
MapleLeaf 🍁OP•13mo ago
oh I wasn't aware the deploy command generated an env file so if it's not being picked up, then that means I'm not loading the env correctly? 🤔
ballingt
ballingt•13mo ago
I'm not sure, just wondering how the frontend build step gets this populated normally for a preview deploy looking
MapleLeaf 🍁
MapleLeaf 🍁OP•13mo ago
this is the CI command for build: 
        run: pnpm convex deploy
          --cmd "pnpm run build"
          --cmd-url-env-var-name VITE_PUBLIC_CONVEX_URL
          --preview-name "e2e-${{ github.ref_name }}"
        run: pnpm convex deploy
          --cmd "pnpm run build"
          --cmd-url-env-var-name VITE_PUBLIC_CONVEX_URL
          --preview-name "e2e-${{ github.ref_name }}"
ballingt
ballingt•13mo ago
ahhh nm nm ah it just runs the cmd in an environment with that environment variable set so you'd have to build a --cmd that reported the VITE_PUBLIC_CONVEX_URL like --cmd "env | grep VITE_PUBLIC_CONVEX_URL > saved.txt; pnpm run build"
MapleLeaf 🍁
MapleLeaf 🍁OP•13mo ago
ah yeah alright that seems to work, but there's some other stuff I need to fix on my end that I'll take a stab at later thanks for the help! :meowthumbsup:
ballingt
ballingt•13mo ago
would love to hear what would be more convenient here as you figure it out, e2e testing isn't what we led with for this because we want something more cohesive but we can still make it better now

Did you find this page helpful?