oscklm•7mo ago

Issues when installing sharp in action with pnpm

Hey, trying to install sharp for my action. My monorepo is setup usng pnpm as packagemanager. Having trouble trying to install the correct prebuilt binary, so that it can run in the convex javascript runtime?

Error fetching POST  https://hushed-koala-696.convex.cloud/api/push_config 400 Bad Request: InvalidModules: Hit an error while pushing:
Loading the pushed modules encountered the following
    error:
Uncaught Failed to analyze image/actions.js: Could not load the "sharp" module using the linux-arm64 runtime
Possible solutions:
- Ensure optional dependencies can be installed:
    npm install --include=optional sharp
    yarn add sharp --ignore-engines
- Ensure your package manager supports multi-platform installation:
    See https://sharp.pixelplumbing.com/install#cross-platform
- Add platform-specific dependencies:
    npm install --os=linux --cpu=arm64 sharp
- Consult the installation documentation:
    See https://sharp.pixelplumbing.com/install
    at ../../node_modules/sharp/lib/sharp.js (../../../../node_modules/sharp/lib/sharp.js:114:0)
    at __require2 (convex:/user/_deps/node/S732JE4B.js:19:50)
    at ../../node_modules/sharp/lib/constructor.js (../../../../node_modules/sharp/lib/constructor.js:12:0)
    at __require2 (convex:/user/_deps/node/S732JE4B.js:19:50)
    at ../../node_modules/sharp/lib/index.js (../../../../node_modules/sharp/lib/index.js:6:0)
    at __require2 (convex:/user/_deps/node/S732JE4B.js:19:50)
    at <anonymous> (../../functions/image/actions.ts:2:18)

Error fetching POST  https://hushed-koala-696.convex.cloud/api/push_config 400 Bad Request: InvalidModules: Hit an error while pushing:
Loading the pushed modules encountered the following
    error:
Uncaught Failed to analyze image/actions.js: Could not load the "sharp" module using the linux-arm64 runtime
Possible solutions:
- Ensure optional dependencies can be installed:
    npm install --include=optional sharp
    yarn add sharp --ignore-engines
- Ensure your package manager supports multi-platform installation:
    See https://sharp.pixelplumbing.com/install#cross-platform
- Add platform-specific dependencies:
    npm install --os=linux --cpu=arm64 sharp
- Consult the installation documentation:
    See https://sharp.pixelplumbing.com/install
    at ../../node_modules/sharp/lib/sharp.js (../../../../node_modules/sharp/lib/sharp.js:114:0)
    at __require2 (convex:/user/_deps/node/S732JE4B.js:19:50)
    at ../../node_modules/sharp/lib/constructor.js (../../../../node_modules/sharp/lib/constructor.js:12:0)
    at __require2 (convex:/user/_deps/node/S732JE4B.js:19:50)
    at ../../node_modules/sharp/lib/index.js (../../../../node_modules/sharp/lib/index.js:6:0)
    at __require2 (convex:/user/_deps/node/S732JE4B.js:19:50)
    at <anonymous> (../../functions/image/actions.ts:2:18)

12 Replies

oscklmOP•7mo ago

Running this cmd pnpm install sharp --config.arch=x64 --config.platform=linux seemed to install some linux stuff. But still getting the error from convex that the sharp module cannot load in the runtime

sshader•7mo ago

Curious if marking this as an external package would help (https://docs.convex.dev/functions/bundling#external-packages) since this looks like it's a node action

Bundling | Convex Developer Hub

Bundling is the process of gathering, optimizing and transpiling the JS/TS

oscklmOP•7mo ago

Ahh haven't tried that. Will give it a go and report back if it solves it! @sshader After defining the sharp package as external in the convex.json Follwed by installing the sharp package with pnpm add sharp in our monorepo, inside the package where our convex lives. We also tried with pnpm install --config.arch=arm64 --config.platform=linux sharp We are still getting this error. We are going with jimp in the meantime, which doesn't use any native stuff. But would love to hear whether sharp is just not supported by convex or if we are doing something wrong. Since sharp is even used in the docs example, did anyone from convex get it working? I'd love to request it to become supported if possible.

RJ•7mo ago

Hmm, I’m using sharp successfully Are you trying it in a Node action? Oh I see you’re asking about running it in the Convex runtime, specifically

oscklmOP•7mo ago

Its in a convex action I actually did end up on ur previous post on the discord, and was thinking about asking how u got it working. Everything we have tried has failed so far

RJ•7mo ago

Can you maybe post your convex.json and the file with the action in it (or whatever of it you can share)?

oscklmOP•6mo ago

Hey RJ, here is our code. Thanks for wanting to take a look

{
  "functions": "/functions",
  "node": {
    "externalPackages": ["sharp"]
  }
}

{
  "functions": "/functions",
  "node": {
    "externalPackages": ["sharp"]
  }
}

'use node';

import sharp from 'sharp';

import { action } from '../_generated/server';

export const testResizeImage = action({
  args: {},
  handler: async (ctx, args) => {
    // Initialize sharp with a dummy image
    const image = sharp({
      create: {
        width: 100,
        height: 100,
        channels: 4,
        background: { r: 255, g: 0, b: 0, alpha: 1 },
      },
    });

    // Resize the image to 50x50
    const resizedImage = image.resize(50, 50);

    // Save blob in a variable
    const buffer = await resizedImage.toBuffer();

    // Convert buffer to blob
    const blob = new Blob([buffer], { type: 'image/jpeg' });

    // Upload the resized image to storage
    const storageId = await ctx.storage.store(blob);

    return storageId;
  },
});

'use node';

import sharp from 'sharp';

import { action } from '../_generated/server';

export const testResizeImage = action({
  args: {},
  handler: async (ctx, args) => {
    // Initialize sharp with a dummy image
    const image = sharp({
      create: {
        width: 100,
        height: 100,
        channels: 4,
        background: { r: 255, g: 0, b: 0, alpha: 1 },
      },
    });

    // Resize the image to 50x50
    const resizedImage = image.resize(50, 50);

    // Save blob in a variable
    const buffer = await resizedImage.toBuffer();

    // Convert buffer to blob
    const blob = new Blob([buffer], { type: 'image/jpeg' });

    // Upload the resized image to storage
    const storageId = await ctx.storage.store(blob);

    return storageId;
  },
});

So the above, is what results in following error: Error fetching POST [REMOVED] 400 Bad Request: InvalidModules: Hit an error while pushing: Loading the pushed modules encountered the following error: Uncaught Failed to analyze sharp/actions.js: Could not load the "sharp" module using the linux-arm64 runtime Possible solutions: - Ensure optional dependencies can be installed: npm install --include=optional sharp yarn add sharp --ignore-engines - Ensure your package manager supports multi-platform installation: See https://sharp.pixelplumbing.com/install#cross-platform - Add platform-specific dependencies: npm install --os=linux --cpu=arm64 sharp - Consult the installation documentation: See https://sharp.pixelplumbing.com/install at ../../node_modules/sharp/lib/sharp.js (../../../../node_modules/sharp/lib/sharp.js:114:0) at require2 (convex:/user/_deps/node/S732JE4B.js:19:50) at ../../node_modules/sharp/lib/constructor.js (../../../../node_modules/sharp/lib/constructor.js:12:0) at require2 (convex:/user/_deps/node/S732JE4B.js:19:50) at ../../node_modules/sharp/lib/index.js (../../../../node_modules/sharp/lib/index.js:6:0) at __require2 (convex:/user/_deps/node/S732JE4B.js:19:50) at <anonymous> (../../functions/sharp/actions.ts:6:31) And we have tried installing using these commands:

pnpm install --config.arch=x64 --config.platform=linux --config.libc=glibc sharp

pnpm install --config.arch=x64 --config.platform=linux --config.libc=glibc sharp

pnpm install sharp

pnpm install sharp

djbalin•6mo ago

I'll chime in here too and provide you some more info @RJ : Have tried some variations on this in package.json (not 100% sure about the syntax required):

  " "pnpm": {
    "supportedArchitectures": {
      "os": [
        "win32",
        "darwin",
        "linux",
        "ubuntu"
      ],
      "cpu": [
        "x64",
        "arm64"
      ],
      "glibc": "2.26"
    }
  },

  " "pnpm": {
    "supportedArchitectures": {
      "os": [
        "win32",
        "darwin",
        "linux",
        "ubuntu"
      ],
      "cpu": [
        "x64",
        "arm64"
      ],
      "glibc": "2.26"
    }
  },

Importing sharp dynamically: const sharp = require('sharp'); import('sharp').then(async (sharp) => {... And still get the same error as described above 😢

oscklmOP•6mo ago

Issue seems to be that externalPackages option in convex.json does not support mono repos, because of the fact that sharp gets installed in the root node_modules. This results in cli not being able to find sharp, hence it not being installed on the convex server, when running convex dev

hasanaktasTR•5mo ago

Did you find a solution to this? We are using the suggested template and our package manager is yarn. https://www.convex.dev/templates/monorepo We tried these two while installing but they didn't work.

- cd packages/backend

- yarn add sharp

- yarn add sharp --ignore-engines

- cd packages/backend

- yarn add sharp

- yarn add sharp --ignore-engines

Here is my code and the error we got.

"use node";
import { encode, decode } from "blurhash";
import sharp from "sharp";
import { internalAction } from "./_generated/server";
import { v } from "convex/values";

export const encodeImageToBlurhash = internalAction({
  args: {
    imageId: v.id("_storage"),
  },

  handler: async (ctx, args) => {
    const imageUrl = await ctx.storage.getUrl(args.imageId);

    if (imageUrl === null) {
      return null;
    }

    const fetchedImage = await fetch(imageUrl);
    const buffer = await fetchedImage.arrayBuffer();
    const { data, info } = await sharp(buffer)
      .raw()
      .ensureAlpha()
      .resize(32, 32, { fit: "cover" })
      .toBuffer({ resolveWithObject: true });

    const encoded = encode(
      new Uint8ClampedArray(data),
      info.width,
      info.height,
      4,
      4,
    );
    const decoded = decode(encoded, info.width, info.height);

    const image = await sharp(Buffer.from(decoded), {
      raw: {
        channels: 4,
        width: info.width,
        height: info.height,
      },
    })
      .jpeg({
        overshootDeringing: true,
        quality: 40,
      })
      .toBuffer();
    return {
      encoded,
      base64: image.toString("base64"),
    };
  },
});

"use node";
import { encode, decode } from "blurhash";
import sharp from "sharp";
import { internalAction } from "./_generated/server";
import { v } from "convex/values";

export const encodeImageToBlurhash = internalAction({
  args: {
    imageId: v.id("_storage"),
  },

  handler: async (ctx, args) => {
    const imageUrl = await ctx.storage.getUrl(args.imageId);

    if (imageUrl === null) {
      return null;
    }

    const fetchedImage = await fetch(imageUrl);
    const buffer = await fetchedImage.arrayBuffer();
    const { data, info } = await sharp(buffer)
      .raw()
      .ensureAlpha()
      .resize(32, 32, { fit: "cover" })
      .toBuffer({ resolveWithObject: true });

    const encoded = encode(
      new Uint8ClampedArray(data),
      info.width,
      info.height,
      4,
      4,
    );
    const decoded = decode(encoded, info.width, info.height);

    const image = await sharp(Buffer.from(decoded), {
      raw: {
        channels: 4,
        width: info.width,
        height: info.height,
      },
    })
      .jpeg({
        overshootDeringing: true,
        quality: 40,
      })
      .toBuffer();
    return {
      encoded,
      base64: image.toString("base64"),
    };
  },
});

Error

 Error fetching POST  https://blabla.convex.cloud/api/push_config 4
│ 00 Bad Request: InvalidModules: Hit an error while pushing:
│ Loading the pushed modules encountered the following
│     error:
│ Uncaught Failed to analyze imageEncoder.js: Could not load the "sharp" module usi
│ ng the linux-arm64 runtime

 Error fetching POST  https://blabla.convex.cloud/api/push_config 4
│ 00 Bad Request: InvalidModules: Hit an error while pushing:
│ Loading the pushed modules encountered the following
│     error:
│ Uncaught Failed to analyze imageEncoder.js: Could not load the "sharp" module usi
│ ng the linux-arm64 runtime

Monorepo with Next.js and Expo

Fullstack monorepo template feat. Expo, Turbo, Next.js, Convex, Clerk This is a modern TypeScript monorepo template with AI web and native apps featuring: - Turborepo: Monorepo management - Next.js 13: Web app & marketing page - React Native Expo: Mobile/native app - Convex: Backend, database, server functions - [Clerk](ht...

oscklmOP•5mo ago

https://discord.com/channels/1019350475847499849/1255859188232355931

AlphaOmega•2mo ago

if you manually create a node_modules and copy paste sharp files + add it to the convex's app/package pacakge.json as a dependecy it works: documented here: https://github.com/get-convex/convex-js/issues/20

GitHub

externalPackages doesn't work in monorepos · Issue #20 · get-conv...

External Packages Fail in Monorepos Due to Dependency Hoisting - Major DX Issue Description There's a significant issue when using Convex's externalPackages for node envs feature within mon...

Issues when installing sharp in action with pnpm

Did you find this page helpful?