Puppeteer in convex

is it possible to use puppeteer in convex actions? I am getting the following error

✖ Error: Unable to push deployment config to https://exuberant-perch-307.convex.cloud
400 Bad Request: UnableToPush: Hit an error while pushing:
Loading the pushed modules encountered the following
    error:
Uncaught Failed to analyze _deps/node/MFSUHX24.js: Class extends value undefined is not a constructor or null

✖ Error: Unable to push deployment config to https://exuberant-perch-307.convex.cloud
400 Bad Request: UnableToPush: Hit an error while pushing:
Loading the pushed modules encountered the following
    error:
Uncaught Failed to analyze _deps/node/MFSUHX24.js: Class extends value undefined is not a constructor or null

18 Replies

Web Dev Cody•2y ago

I’m be curious to hear the answer, do you have use node at the top of the file?

ashuvssut (ashu)OP•2y ago

yes

"use node";
import { v } from "convex/values";
import { action } from "gconvex/_generated/server";
import { cspHeaders } from "gconvex/utils";
import puppeteer from "puppeteer";
 
export const getWebResource = action({
    args: { url: v.string() },
    handler: async (ctx, { url }) => {
        try {
            const browser = await puppeteer.launch();
            const page = await browser.newPage();

            await page.goto(url);
            const bodyHTML = await page.content();
            await browser.close();
            return { statusCode: 200, headers: cspHeaders, body: bodyHTML };
        } catch (error) {
            return {
                statusCode: 500,
                body: JSON.stringify({ message: "Error fetching URL" }),
            };
        }
    },
});

"use node";
import { v } from "convex/values";
import { action } from "gconvex/_generated/server";
import { cspHeaders } from "gconvex/utils";
import puppeteer from "puppeteer";
 
export const getWebResource = action({
    args: { url: v.string() },
    handler: async (ctx, { url }) => {
        try {
            const browser = await puppeteer.launch();
            const page = await browser.newPage();

            await page.goto(url);
            const bodyHTML = await page.content();
            await browser.close();
            return { statusCode: 200, headers: cspHeaders, body: bodyHTML };
        } catch (error) {
            return {
                statusCode: 500,
                body: JSON.stringify({ message: "Error fetching URL" }),
            };
        }
    },
});

basically I want to load webpages in <iframe> in frontend. But some website have a CSP header that helps prevent this. So I am trying to create a proxy server thing using convex actions

ballingt•2y ago

Puppeteer won't work currently (there's work in progress for this) but you should be able to create a proxy with fetch() in either Node.js or normal Convex runtime.

ashuvssut (ashu)OP•2y ago

Alright. no problem. I dont know if i can use fetch or axios because I want everything like css fonts images etc

ballingt•2y ago

Convex will likely support puppeteer before long, your approach of running the site and then grabbing the HTML is useful — just for now a more passthrough approach like a normal HTTP proxy is necessary If you run this in an iframe you'd need to rewrite urls in your proxy in order to request the fonts, scripts etc. in an iframe too. You'd need to rewrite the headers to remove the CSP header. This might work for you but might be challenging depending on the site you're trying to iframe

ashuvssut (ashu)OP•2y ago

yes. makes sense

ashuvssut (ashu)OP•2y ago

I worked on Tom's suggestion and created an action that works like a proxy to fetch the entire webpage with css fonts images and stuff I used absolutify and cheerio to create that action Code: https://github.com/ashuvssut/cookied/blob/dev/apps/convex/webContent.ts#L40

GitHub

cookied/apps/convex/webContent.ts at dev · ashuvssut/cookied

Contribute to ashuvssut/cookied development by creating an account on GitHub.

ashuvssut (ashu)OP•2y ago

Thanks for suggesting the alternative solution!

ballingt•2y ago

Nice! Good to know about absolutify, I hadn't heard of it and it looks super useful

ashuvssut (ashu)OP•2y ago

absolutify can not convert relative urls written in srcset attribute of <img> tag For example: <img srcset="/_next/image?url=%2Fimage1.jpg&w=100 1x, /_next/image?url=%2Fimage1.jpg&w=200 2x"> So you have to look for alternatives like cheerio to do this particular thing The fetch() solution that you mentioned is will not work for scraping JS webapps like React apps. Since, the HTML is only hydrated in client side, I would need puppeteer for this to load the webpage and then do the web scraping thing I ll be waiting for puppeteer support in convex. For now Vercel Serverless seems to do the job Ah! today I saw an issue in Prod. Puppeteer doesn't work with vercel serverless functions in prod build. It just worked locally on my pc in dev mode When I saw the logs (Production), it said

Error: Could not find Chrome (ver. 117.0.5938.92). This can occur if either
 1. you did not perform an installation before running the script (e.g. `npm install`) or
...
...

Error: Could not find Chrome (ver. 117.0.5938.92). This can occur if either
 1. you did not perform an installation before running the script (e.g. `npm install`) or
...
...

Actually its because vercel's 50MB file-size limit For, now I think there's no free CSR webapp scraping serverless solution. Going to stick with fetch

ballingt•2y ago

Ah good to know, thanks!

ashuvssut (ashu)OP•2y ago

haha Just wanted to clarify the misinformation I provided earlier. 😅

ballingt•2y ago

the local dev experience feels great but can bite you when the prod environment is different sadly

ashuvssut (ashu)OP•2y ago

yup. happened many times😅 Hi Tom, I have a question, and it's unrelated to our ongoing thread. This question might sounds a bit silly 😅 I noticed in the recent announcement that Convex will have first-class Python support. I'm curious about what "first-class" means in this context. Does it imply that we'll have fully typed support for both the frontend in TypeScript and the backend in Python? I'm trying to understand how the type system will work with Python functions. Do we need to write specific schemas that the Convex generator understands to generate types for both TypeScript on the frontend and Python functions on the backend?

ballingt•2y ago

In this context "first-class" probably means subscriptions: right now the Convex Python client just does queries and mutations and actions, but not subscribing to a query. But yeah we want types in Python too, this is close! We can't use TypeScript type inference for this, we'll need to use { args: ..., handler: ..., output: ...} We've already limited schemas in such a way that they'll work for Python, so you will not need to change the way you write schemas! All Convex types will work in every language we support. This is part of why you can't use e.g. a v.date() in Convex schema; every type in here needs to be representable in every client language Convex has types / codegen for @ashuvssut (ashu) Are you thinking you'd like to use Python for something? Curious to hear what you'd like here.

ashuvssut (ashu)OP•2y ago

I was discussing with a friend of mine regarding my puppeteer issue He said that In python there is a library called "scrapy" This library's bundle size will be less than 50mb. But implementing web scraping of CSR web app may be a little tricky Whatever it may be. I will try out scrapy when Python support comes up in Convex

ballingt•2y ago

Ah so this would require submitting a Convex function written in Python. That's something we think about, but isn't a concrete plan yet. The announced plan is only to make a first-class Python client.

ashuvssut (ashu)OP•2y ago

Oh... got it. I actually thought that there will be python convex functions lol

Puppeteer in convex

Did you find this page helpful?