Skip to content

Home

Jump to bottom
quantumish edited this page Oct 20, 2022 · 3 revisions

𝔄𝔟𝔞𝔫𝔡𝔬𝔫 𝔥𝔬𝔭𝔢 𝔞𝔩𝔩 𝔶𝔢 𝔴𝔥𝔬 𝔢𝔫𝔱𝔢𝔯 𝔥𝔢𝔯𝔢

Welcome to the blatherskite wiki! This should have most of the things you'd need to know if you're contributing to the project.

First off, let me personally apologize. Things were rushed. The codebase is not elegant — although in my opinion it isn't thaat bad.

Architecture

blatherskite is comprised of two main services:

  • chatterbox, a websocket server that handles sending/receiving messages
  • scuttlebutt, an HTTP server that handles all actions besides sending and receiving messages

These both communicate with the Cassandra database that stores everything.

Database

The backing tables mostly match the objects documented in responses.rs:

CREATE TABLE bsk.users (id bigint PRIMARY KEY, name text, email text, hash text);

CREATE TABLE bsk.groups (id bigint PRIMARY KEY, (id bigint PRIMARY KEY, name text, members set<bigint>, is_dm boolean, channels set<bigint>, admin set<bigint>, owner bigint);

CREATE TABLE bsk.channels (id bigint PRIMARY KEY, group bigint, name text, members set<bigint>, private boolean);

CREATE TABLE bsk.messages (channel bigint, id bigint, author bigint, content text, group bigint, thread bigint, PRIMARY KEY (channel, id)) WITH CLUSTERING ORDER BY (id DESC);"

The clustering order in the messages table is so that querying the first N rows of it will give the most recent N messages.

There's also two internal tables used for looking up the groups/dms a user is a part of:

CREATE TABLE bsk.user_groups (id bigint PRIMARY KEY, groups set<bigint>);

CREATE TABLE bsk.user_dms (id bigint PRIMARY KEY, dms set<bigint>);

scuttlebutt

chatterbox

Anatomy of an Endpoint

Let's take DELETE /group as an example:

  #[oai(path = "/group", method = "delete")]
  /// Delete a group. 
  /// 
  /// Only authorized for the owner of a group.
  async fn delete_group(&self, auth: Authorization, id: Query<i64>) -> DeleteResponse {
      use DeleteResponse::*;
      if !self.db.valid_id(IdType::Group, id.0).unwrap() {
          return NotFound(PlainText("Group not found".to_string()));
      } else if self.db.get_group_owner(id.0).unwrap() != auth.0.id {
          return Unauthorized;
      }
      let group = self.db.get_group(id.0).unwrap();
      for member in group.members {
          self.db.remove_user_group(member, id.0).unwrap();
      }
      for channel in group.channels {
          self.db.delete_channel(channel).unwrap();
      }
      self.db.delete_group(id.0).unwrap();
      Success
  }

First off, you might notice this "oai" line lying above all of the API endpoints in main.rs — this is the poem_openapi macro that specifies what the path and method of the endpoint should be.

#[oai(path = "/group", method = "delete")]

Now, let's move onto the actual function signature:

async fn delete_group(&self, auth: Authorization, id: Query<i64>) -> DeleteResponse

Taking an Authorization means that this request needs an authorization header, and poem will automatically run api_checker() before your function in order to parse it into a Claims struct. This object thus contains the ID of the logged in user, which is quite handy. That's what all of those auth.0.ids in the codebase are getting.

Similarly, we see a Query<i64>, which specifies that the response expects an integer in the query. You can accessed this passed value in the function with id.0.

We also see a DeleteResponse type returned from the model: these are defined in responses.rs (alongside the actual User/Group/Channel objects), and enable the generated Swagger documentation to document the response codes of the endpoint. Note that the doc comments here (and in the API endpoint) are actually captured by poem to be used in the final documentation.

#[derive(ApiResponse)]
pub enum DeleteResponse {
    /// The delete operation succeeded
    #[oai(status = 200)]
    Success,
    /// You are not authorized to perform the action
    #[oai(status = 401)]
    Unauthorized,
    /// Invalid ID. Content specifies which of the IDs passed is invalid.
    #[oai(status = 404)]
    NotFound(PlainText<String>),
    /// Internal server error: likely due to a database operation failing
    #[oai(status = 500)]
    InternalError(PlainText<String>),
}

You can then see the function actually returning some of these responses when it does the validation checks:

if !self.db.valid_id(IdType::Group, id.0).unwrap() {
    return NotFound(PlainText("Group not found".to_string()));
} else if self.db.get_group_owner(id.0).unwrap() != auth.0.id {
    return Unauthorized;
}

Note that every endpoint that receives an ID needs to validate it, as all of the database wrappers in scuttlebutt assume that the passed IDs are valid.

Then, finally, we get to the actual logic of the endpoint:

let group = self.db.get_group(id.0).unwrap();
for member in group.members {
    self.db.remove_user_group(member, id.0).unwrap();
}
for channel in group.channels {
    self.db.delete_channel(channel).unwrap();
}
self.db.delete_group(id.0).unwrap();

This is hopefully pretty straightforward. While we've spent this section going through an existing endpoint, these are essentially all of the pieces you need to make your own!

Why did you...

...make this godforsaken monstrosity of a trait?

pub trait Database: Sync + Send {
    fn valid_id(&self, kind: IdType, id: i64) -> Result<bool>;

    fn create_user(&self, id: i64, name: String, email: String, hash: String) -> Result<()>;
    fn update_user(&self, id: i64, name: String, email: String) -> Result<()>;
    fn get_user(&self, id: i64) -> Result<User>;
    fn get_user_hash(&self, id: i64) -> Result<String>;
    fn delete_user(&self, id: i64) -> Result<()>;

    fn create_group(&self, gid: i64, uid: i64, name: String, dm: bool) -> Result<()>;
    fn get_group(&self, id: i64) -> Result<Group>;
    fn update_group(&self, id: i64, name: String) -> Result<()>;
    fn delete_group(&self, id: i64) -> Result<()>;
    
    fn get_group_members(&self, gid: i64) -> Result<Vec<i64>>;
    fn add_group_member(&self, gid: i64, uid: i64) -> Result<()>;
    fn remove_group_member(&self, gid: i64, uid: i64) -> Result<()>;
    
    fn get_group_channels(&self, gid: i64) -> Result<Vec<i64>>;    
    fn add_group_channel(&self, gid: i64, uid: i64) -> Result<()>;
    fn remove_group_channel(&self, gid: i64, uid: i64) -> Result<()>;
    
    fn get_group_admin(&self, gid: i64) -> Result<Vec<i64>>;
    fn add_group_admin(&self, gid: i64, uid: i64) -> Result<()>;
    fn remove_group_admin(&self, gid: i64, uid: i64) -> Result<()>;
    
    fn get_group_owner(&self, gid: i64) -> Result<i64>;

    fn is_group_dm(&self, gid: i64) -> Result<bool>;

    fn create_channel(&self, cid: i64, gid: i64, uid: i64, name: String) -> Result<()>;
    fn get_channel(&self, id: i64) -> Result<Channel>;
    fn update_channel(&self, id: i64, name: String) -> Result<()>;
    fn delete_channel(&self, id: i64) -> Result<()>;

    fn get_channel_members(&self, gid: i64) -> Result<Vec<i64>>;
    fn add_channel_member(&self,cid: i64, id: i64) -> Result<()>;
    fn remove_channel_member(&self, cid: i64, id: i64) -> Result<()>;

    fn is_channel_private(&self, id: i64) -> Result<bool>;
    fn set_channel_private(&self, id: i64, value: bool) -> Result<bool>;

    fn create_user_groups(&self, id: i64) -> Result<()>;
    fn get_user_groups(&self, id: i64) -> Result<Vec<i64>>;
    fn delete_user_groups(&self, id: i64) -> Result<()>;
    fn add_user_group(&self, uid: i64, gid: i64) -> Result<()>;
    fn remove_user_group(&self, uid: i64, gid: i64) -> Result<()>;

    fn create_user_dms(&self, id: i64) -> Result<()>;
    fn get_user_dms(&self, id: i64) -> Result<Vec<i64>>;
    fn delete_user_dms(&self, id: i64) -> Result<()>;
    fn add_user_dm(&self, uid: i64, gid: i64) -> Result<()>;    
    
    fn get_message(&self, id: i64) -> Result<Message>;
    fn get_messages(&self, cid: i64, num: u64) -> Result<Vec<Message>>;
    fn delete_message(&self, id: i64) -> Result<()>;
    fn set_thread(&self, id: i64, cid: i64) -> Result<()>;
}

Okay — I'm sorry. This is technically better than what was there before: raw database calls mixed in with logic that made things impossible to read. What we'd really want is a ORM to generate all of these functions for us behind the scenes, but the only one I could find for Cassandra was this very sketchy undocumented library called catalytic.

Plus, the bonus here is that while there's way too many functions, almost all of the functions in the current codebase are self-documenting since they're short and call clearly-named functions.

...make a Database trait that can only return Cassandra errors?

Wasn't intentional. Please fix — see the corresponding issue.

...shove the ID generation behind a global mutex??

As mentioned earlier, poem doesn't let you have mutable state in your server, which is very sad. I've tried a lot of different strategies, but none work well, so I just took the performance penalty. See issue.

...make it so all messages get sent to every user, then filtered out if they aren't meant to see it after they've received it???

chatterbox was a really rushed job. Ideally we'd like to have a small service that logs open connections then creates specific channels (as in the thread communication primitive) for specific channels (the messaging feature).

Clone this wiki locally