RSS in convex help
Good afteroon, so for my site I want to implement https://www.npmjs.com/package/rss-parser and add some data to my nextjs app.
im trying to think of a way to save the feed data into convex, but how would i use the library on the backend? The only way i know to run convex code like db.insert is in functions but we can use npm. im trying to avoid having to write another rest server just to save rss information. Would i use maybe nextjs api rountes?
npm
rss-parser
A lightweight RSS parser, for Node and the browser. Latest version: 3.13.0, last published: a year ago. Start using rss-parser in your project by running
npm i rss-parser. There are 323 other projects in the npm registry using rss-parser.12 Replies
Basically, i want to run the rss feed in the background and load the data into convex constantly like every hour, and its just loaded onto the client side when new data is in the database
The library has documentation for how to use it in a node environment. I would write an action that runs in the node environment, and have that action fetch the rss feed data and use
ctx.runMutation() to update your convex data. You can have the action schedule itself to run an hour later to keep the updates going moving forward.the library also says it supports browser environment, so it could work in a default-environment action (convex functions do support many npm packages). i would try it and see where it fails. and use
ctx.runMutation to write the data into convex as @erquhart described. maybe run the action on a cron. https://docs.convex.dev/scheduling/cron-jobsCron Jobs | Convex Developer Hub
Convex allows you to schedule functions to run on a recurring basis. For
I think it has to use fetch though
the default convex environment has
fetch in actionsokay thanks guys I will look into this!
Ah, where did I get that fetch only works in node 🤦♂️ I def have convex runtime actions that fetch
should i just use a cron job to run the fetch and mutations over and over? vs scheduling
I would say yes since @lee recommended that approach. I feel like there's some consideration to be had for the right approach at scale, but a cron job is a really straightforward way to start.
Once you start getting real usage, I expect self scheduling will be the best path vs having a cron job run this heavy action that's updating all rss feeds, and having to figure out how to keep that performant. I think self scheduling would just organically scale.
Also makes some future stuff trivial, like allowing users to choose how often an individual rss feed updates.
yeah i understand, thanks for the help
self scheduling from an action is a little sketchy because actions don't guarantee execution. Self scheduling a mutation is fine though, since it does guarantee exactly-once execution. (what @erquhart said, just explaining why)
oh i see thanks
i think for now i will run the mutations mainually
until i need to autoamte them