jamwt•9mo ago

Building a client for a new language

@Tom might have some tips. he was the one who did the Python wrapper

30 Replies

Yeah @ramon your options are 1) wrapping the Rust client and learning how to build binaries for all the architectures you need, 2) limiting yourself to the HTTP API which is very simple to implement, if you can work with JSON and make HTTP calls you're basically set, or 3) reimplement the WebSocket protocol, which we'd appreciate if you did not do. If you can find a guide for compiling Rust to something exposed from GDScript it's not bad! Besides the building and linking which is different for each environment, you need to figure out a way to represent each Convex data type (or a subset if you like) in your language.

ramon•9mo ago

I already have another library compiled to be used within gdscript, so I am acquainted with the process! And I really want the websocket functionality I guess my question is, I know I'm going to go the rust client wrapper route. I just want to know if there are "guidelines" on the functionality I should cover. I can start with the absolute minimum from there

ballingt•9mo ago

@ramon oh got it, hm I think the Python API is reasonable One choice you cna make is to support marching through timestamps of every query currently subscribed (watch_all) or dealing with each subscription separately. I think that's all you really need: subscribe to queries, unsubscribe to them, do something when there's new data, and fire off mutations.

ramon•9mo ago

In the python implementation, does the rust side handle updating values from the subscriptions or is that a python loop?

ballingt•9mo ago

Here's that code more or less https://github.com/get-convex/convex-py/blob/de180ceec1a08ed78cb290f333a50c64ed5f47b6/python/convex/__init__.py#L208-L214 In Python you write your own loop generally to loop over responses and if you want that to happen concurrently with other things you stick it on a thread yourself

ballingt•9mo ago

https://colab.research.google.com/drive/19o5xpraHnfrh-fcOwqiLK-LpoxlWZNYx#scrollTo=k4yspAlvY4hs

Google Colaboratory

ramon•9mo ago

Thanks! In the python implementation, why do you reassign the result (result[k] = result[k]):

    def __next__(self) -> Optional[Dict[SubscriberId, ConvexValue]]:
        result = self.safe_inner_sub().next()
        if not result:
            return result
        for k in result:
            result[k] = result[k]
        return result

    def __next__(self) -> Optional[Dict[SubscriberId, ConvexValue]]:
        result = self.safe_inner_sub().next()
        if not result:
            return result
        for k in result:
            result[k] = result[k]
        return result

this is in https://github.com/get-convex/convex-py/blob/de180ceec1a08ed78cb290f333a50c64ed5f47b6/python/convex/__init__.py#L162 @Tom , I am going over the Python implementation, and the Rust client. In the Rust implementation of QuerySubscription, next hangs until it gets a new message, right? anext implements an async version of it that can be awaited. Is await the only way to fetch this? Is there another method that just checks if there is an item available and returns none otherwise?

ballingt•9mo ago

On the Rust or the Python side? The Python side just implements blocking APIs because that's what seemed natural — but if you want to check for results / poll, on the Rust side I think there are some options?

ballingt•9mo ago

rust docs here https://docs.rs/convex/latest/convex/#modules

convex - Rust

Convex Client

ballingt•9mo ago

It looks like with the high-level async Rust client you do need to await in Rust, but you could build a queue on that side, or a one-slot buffer

ramon•9mo ago

Would that be via poll_next instead of next? I'm trying to understand how this works

ballingt•9mo ago

Yeah that's a way you could do it What's the interface you'd like, maybe we can suggest something. Nipunn who wrote this will be back next week but lots of Rust folks here might have ideas.

ramon•9mo ago

Well, I want to use it within the godot game engine, through gdscript Godot makes update calls to every object every frame, so I think the most natural way would be for me to have a function that checks if there is something available for the subscription, and return it if so, or none if there is none

ballingt•9mo ago

I think poll_next does more than you want

ramon•9mo ago

looking at library that makes it easy to build rust bindings for godot, async calls are not supported yet I could implement all the async plumbing, but then that's a huge detour from what I actually want and may end up just sides stepping convex for now

ballingt•9mo ago

There are two levels of Rust client, the main one which uses Tokio and wires up the WebSocket, and the base client that has the state machine but you're responsible for building a WebSocket and passing messages over it

ballingt•9mo ago

https://docs.rs/convex/latest/convex/base_client/index.html

convex::base_client - Rust

The synchronous state machine for Convex. It’s recommended to use the higher level ConvexClient unless you are building a framework.

ramon•9mo ago

yeap, I would like to use the Tokio version if possible

ballingt•9mo ago

got it, yeah then sounds like you need the async support

ramon•9mo ago

hmmm alright so the options are async support or build the websocket/messaging infra let me check what it takes to get async working on godot

ballingt•9mo ago

if youv'e got a godot WebSocket the sync thing might not be hard to wire up by integrating it into the gameloop

ramon•9mo ago

what do you mean? as in, if godot already implements a websocket, then async should be relatively easy to implement?

ballingt•9mo ago

I mean you don't need async if you use the base client, it's just the state machine. You pass it messages you recieve from teh websocket and it gives you messages it'd like you to send, all sync

ramon•9mo ago

oh gotcha

ballingt•9mo ago

you already said roughly this so I might be repeating

ramon•9mo ago

so you are saying I could use Godot's websocket

ballingt•9mo ago

yeah

ramon•9mo ago

ok that's an option thanks! ohhhhhh I think I get it, so base client is literally just handling "convex specific state machine part". I have to do a higher level wrapper that at each frame looping over all incoming messages, passing them to the base client via receive_message to update the convex state in other words, the base client does the translation parts for how to handle the incoming and outgoing messages, but looping over the messages themselves and sending them is up to me I think I have already something like this set up with my rudimentary websocket system, so this should work continuiing my path here! I need to send the messages of type ClientMessage to the backend myself, but I assume these get serialized in one way or another before being sent through the websocket in all the different clients, right?

ramon•9mo ago

If so, what is the serialization that you normally use? I want to avoid having to convert these [1] messages into their correponding Godot types if I can avoid it. [1] https://docs.rs/convex_sync_types/0.6.0/convex_sync_types/types/enum.ClientMessage.html

ClientMessage in convex_sync_types::types - Rust

API documentation for the Rust ClientMessage enum in crate convex_sync_types.

ramon•9mo ago

        // Send an initial connect message on the new websocket
        let session_id = Uuid::new_v4();
        let message = ClientMessage::Connect {
            session_id: SessionId::new(session_id),
            connection_count,
            last_close_reason,
            max_observed_timestamp,
        };
        let msg = Message::Text(
            serde_json::Value::try_from(message)
                .context("JSONSerializationErrorOnConnect")?
                .to_string(),
        );
        internal.send_worker(msg).await?;

        // Send an initial connect message on the new websocket
        let session_id = Uuid::new_v4();
        let message = ClientMessage::Connect {
            session_id: SessionId::new(session_id),
            connection_count,
            last_close_reason,
            max_observed_timestamp,
        };
        let msg = Message::Text(
            serde_json::Value::try_from(message)
                .context("JSONSerializationErrorOnConnect")?
                .to_string(),
        );
        internal.send_worker(msg).await?;

I found this in the codebase, this implies the messages get serialized as json and then sent as text, right?

Building a client for a new language

Did you find this page helpful?