Additional information

• Workshop was part of the RustFest 2019 in Barcelona
• See speaker notes (Press S) for additional information

About me

• Writing rust since 2014
• weiznich at github
• Maintainer of Diesel and creator of Wundergraph

General rules

• Ask Questions
• Interrupt me if you don’t understand something or want to know more

Outline

• Today:
  1. Diesel introduction
  2. Diesel internals
• Tomorrow:
  3. Wundergraph introduction
  4. Wundergraph internals

What is diesel

• Diesel is a query builder with some ORM like features
• Provides the following building blocks:
  • Abstract database interfaces to execute queries
  • Functionality to map query results to rust structs
  • Functionality to map rust structs to query results
  • A query dsl that could be used to construct compile time checked sql queries

What is diesel

table! {
    students {
        id -> Integer,
        name -> Text,
    }
}

#[derive(Queryable)]
struct Student {
    id: i32,
    name: String,
}

let students: Vec<Student> =
    students::table
        .filter(students::id.eq(42))
        .load::<Student>(&conn)?;

diesel::insert_into(students::table)
    .values((
        students::id.eq(43),
        students::name.eq("weiznich"),
    )).execute(&conn)?;

Diesel features

General design assumptions for diesel:

• Diesel does not own the database
• SQL database implementations are different, Diesel should not try to abstract away those differences
• Rust has a strong static type system, try to use that for correctness and performance
• There is a distinction between Rust types and SQL types.

Creating a diesel application

0. Install diesel_cli (A tool to manage migrations/your schema)
1. Setup the database using SQL migrations
2. Create a rust schema file form your database
3. Write code interacting with the database

Diesel CLI

• Small helper cli tool for Diesel
• Provides:
  • a way to manage a database (setup/reset)
  • a way to manage migrations (create/apply/undo)
  • a way to generate Rust schema code from existing databases

The table! marco

table! {
    students(id) {
        id -> Integer,
        name -> Text,
    }
}

• Defines a database schema on rust side
• Generates a bunch of helper types used to construct queries

Select statements

• Diesel provides a query DSL to construct statically known queries
• Generally speaking: DSL maps quite literally to the generated SQL
• Documentation mostly available on the QueryDsl trait

a_table::table
    .select(fields_to_select)
    .filter(some_filter_expression)
    .order_by(some_order_expression)
    .limit(limit_as_i64)
    .offset(offset_as_i64)
    .load::<SomeStructOrTuple>(&conn);

Select statements (Select clause)

a_table::table.select((
    a_table::column_a, // just select a column
    a_table::column_b + a_table::column_b, // an expression
    "abc".into_sql::<Text>() // or a constant
))

• Just a tuple of expressions that should appear in the select clause
• If not given just select the columns of the table in the order given by the table! macro

Select statements (Where clause)

a_table::table
    .filter(a_table::column_a.eq(foo)
        .and(a_table::column_b.like(baz)))
    .or_filter(a_table::column_c.is_null())

• Constructs a where clause of the current SQL select statement
• If not given no where clause is generated
• Calling .filter twice appends the new where clause with an AND to the old one
• See docs *ExpressionMethods traits for methods useful to construct inner expressions

Select statement (Order clause)

a_table::table.order_by((
    a_table::column_a.asc(),
    a_table::column.desc()
))

• Creates a order clause
• Possibility to single order expression or multiple as tuple
• Default sorting order: ASC
• If clause not given no order clause is generated

Result mapping (Queryable)

• A trait that indicates that a given rust type could be the result of a query with a SQL type
• Provided custom derive, mapping is done by order, not by name
• Default implementations for tuples

Query execution

• Different ways to execute a query:
  • load::<U>/get_results::<U>: Returns a list of U
  • get_result::<U>: Returns the first U ignores the rest
  • first::<U>: Returns the first U, attaches a LIMIT 1 clause to the executed query
  • execute: Returns the number of affected columns

Insert statements

diesel::insert_into(a_table::table)
    .values((
        a_table::column_a.eq(value),
        a_table::column_b.eq(other_value)
)).execute(&conn);

• Creates a INSERT INTO statement.

Insert statements

#[derive(Insertable)]
#[table_name = "students"]
struct NewStudent {
    id: i32,
    name: String,
}

diesel::insert_into(students::table)
    .values(&[new_student_1, new_student_2])
    .execute(&conn);

Update statements

diesel::update(a_table::table.filter(a_table::id.eq(1)))
    .set((
        a_table::column_a.eq(some_value),
        a_table::column_b.eq(a_table::column_b + 5.into_sql::<Integer>())
    )).execute(&conn);

• Creates a UPDATE statement
• Similar to insert statements, there is a tuple variant and a variant with a struct

Update statements

#[derive(AsChangeset, Identifiable)]
#[table_name = "students"]
struct ChangeStudent {
    id: i32,
    name: Text,
}

let changed_student: ChangeStudent = get_changed_student();

diesel::update(&changed_student)
    .set(&changed_student).execute(&conn);

Delete statements

diesel::delete(
    a_table::table.filter(filter_expression)
).execute(&conn)'

• Creates a DROP FROM a_table statement

Raw SQL queries

#[derive(QueryableByName)]
#[table_name = "students"]
struct Student {
    id: i32,
    name: String
}

diesel::sql_query("SELECT id, name FROM students WHERE name = $1")
    .bind::<Text, _>("weiznich")
    .load(&conn);

• Raw SQL query interface
• Meant to be used when the query DSL is missing some expressions or failed to express something complex

“Complex” queries

table! {
    students {
        id -> Integer,
        name -> Text,
        supervisor -> Integer,
    }
}

table! {
    teachers {
        id -> Integer,
        name -> Text,
    }
}

allow_tables_to_appear_in_same_query!(students, teachers);
joinable!(students -> teachers (supervisor));

“Complex” queries (naive way)

let teachers = teachers::table.load::<Teacher>(&conn)?;

let teacher_and_students: Vec<(Teacher, Vec<Student>)> =
    teachers.into_iter()
        .map(|teacher| {
            let students = students::table
                .filter(students::supervisor.eq(teacher.id))
                .load::<Student>(&conn)?;
            Ok((teacher, students))
        }).collect()?;

“Complex” queries (joins)

teachers::table.inner_join(students::table)
    .load::<(Teacher, Student)>(&conn);

“Complex” queries (associations)

#[derive(Identifiable, Queryable)]
#[table_name = "teachers"]
struct Teacher {
    id: i32,
    name: String,
}

#[derive(Identifiable, Queryable, Associations)]
#[belongs_to(Teacher, foreign_key = "supervisor")]
#[table_name = "students"]
struct Student {
    id: i32,
    name: String,
    supervisor: i32
}

let teachers = teachers::table.load::<Teacher>(&conn)?;

let students = Student::belonging_to(&teachers)
    .load::<Student>(&conn)?
    .grouped_by(&teachers);

let teachers_with_students = teachers.into_iter()
    .zip(students)
    .collect::<Vec<(Teacher, Vec<Student>)>>();

Common Errors

• Diesel uses the type system to check if
  • a query generates valid SQL
  • a query result matches the structure of the output
  • matching field types are compatible with given query types
• Some of those mistakes result in rather long error messages
• Most of them have a structure that helps you to find the actual problem

Common Errors (Invalid SQL)

students::table.select(teachers::name).load::<Student>(&conn);

error[E0277]: the trait bound `teachers::columns::name: diesel::SelectableExpression<students::table>` is not satisfied
  --> src/main.rs:30:21
   |
30 |     students::table.select(teachers::name).load::<Student>(&conn);
   |                     ^^^^^^ the trait `diesel::SelectableExpression<students::table>` is not implemented for `teachers::columns::name`
   |
   = help: the following implementations were found:

Common Errors (Result missmatch)

#[derive(Queryable)]
struct Student {
    id: i32,
    name: String
}

let s: Vec<Student> = students::table
    .select(students::id)
    .load(conn);

error[E0277]: the trait bound `(std::string::String, i32): diesel::Queryable<diesel::sql_types::Integer, diesel::pg::Pg>` is not satisfied
  --> src/main.rs:34:64
   |
34 |     let s: Vec<Student> = students::table.select(students::id).load(&conn)?;
   |                                                                ^^^^ the trait `diesel::Queryable<diesel::sql_types::Integer, diesel::pg::Pg>` is not implemented for `(std::string::String, i32)`

Common Errors (Type missmatch)

let s: Vec<String> = students::table
    .select(students::id)
    .load(conn);

error[E0277]: the trait bound `*const str: diesel::deserialize::FromSql<diesel::sql_types::Integer, diesel::pg::Pg>` is not satisfied
  --> src/main.rs:34:63
   |
34 |     let s: Vec<String> = students::table.select(students::id).load(&conn)?;
   |                                                               ^^^^ the trait `diesel::deserialize::FromSql<diesel::sql_types::Integer, diesel::pg::Pg>` is not implemented for `*const str`
   = help: the following implementations were found:
             <*const [u8] as diesel::deserialize::FromSql<diesel::sql_types::Binary, DB>>
             <*const str as diesel::deserialize::FromSql<diesel::sql_types::Text, DB>>
   = note: required because of the requirements on the impl of `diesel::deserialize::FromSql<diesel::sql_types::Integer, diesel::pg::Pg>` for `std::string::String`

Current shortcomings of Diesel

• No support for GROUP BY clauses and mixing aggregating and non aggregating expressions
• Diesel expects to know at least the following things about SELECT statements at compile time:
  • Number of returned fields
  • SQL type of each returned field
• Writing code abstracting over Diesel can be challanging

Building a first simple example

• We want to build a simple blog backend
• Today as REST API, Tomorrow as GraphQL API
• Schema consists of 3 Tables:
  • users
  • posts
  • comments
• Project template at https://github.com/weiznich/wundergraph-workshop/

Build a first small example

Bootstrap a simple REST API for the given database schema:

0. Clone the project template
1. Prepare the database by writing sql migrations for the given schema
2. Setup the database using diesel_cli
3. Include the generated schema in your application
4. Expose CRUD operations for all 3 tables as REST API

Custom types

• Diesel supports a set of commonly used types out of the box
• Custom or uncommen SQL types require additional work
• Diesel provides building blocks to easily add support for your own types

Custom types

CREATE TYPE color AS ENUM ('red', 'green', 'blue');

#[derive(SqlType, QueryId)]
#[postgres(type_name = "color")]
struct ColorType;

#[derive(FromSqlRow, AsExrpession)]
#[sql_type = "ColorType"]
enum Color {
    Red,
    Green,
    Blue,
}

Custom types

impl FromSql<ColorType, Pg> for Color {
    fn from_sql(bytes: Option<&[u8]>) -> Result<Self> {
        match bytes {
            Some(b"red") => Ok(Color::Red),
            Some(b"green") => Ok(Color::Green),
            Some(b"blue") => Ok(Color::Blue),
            _ => Err("Unrecognized enum variant".into())
        }
    }
}

impl ToSql<ColorType, Pg> Color {
    fn to_sql<W: Write>(&self, out: &mut Output<W, Pg>) -> Result<IsNull> {
         match *self {
             Color::Red => out.write_all(b"red")?;,
             Color::Green => out.write_all(b"green")?;
             Color::Blue => out.write_all(b"blue")?;
         }
         Ok(IsNull::No)
    }
}

Fundamental traits

• Want to talk about a bit how Diesel queries work internally
• Answer the following questions:
  • How does diesel build SQL from the query dsl?
  • How does diesel check queries at compile time?
  • How does a diesel query compose from simpler query fragments?

Fundamental traits (Backend)

trait Backend {
    type RawValue;
}

• Central definition of a supported database
• RawValue says which how data are represented at protocol level
• Implementations are just zero sized marker types

Fundamental traits (QueryFragment)

trait QueryFragment<DB: Backend> {
    fn walk_ast(&self, pass: AstPass<DB>) -> QueryResult<()>;
}

• Trait indicating that some type could be translated to SQL.
• Used to construct the final query
• Helps to do most of the work at compile time

Fundamental traits (QueryFragment)

struct Eq<L, R>{
    left: L,
    right: R,
}

impl<L, R> QueryFragment<Pg> for Eq<L, R>
where
    L: QueryFragment<Pg>,
    R: QueryFragment<Pg>,
{
    fn walk_ast(&self, mut pass: AstPass<DB>) -> QueryResult<()> {
        self.left.walk_ast(pass.reborrow())?;
        pass.push_sql(" = ");
        self.right.walk_ast(pass.reborrow())?;
        Ok(())
    }
}

Fundamental traits (QueryId)

trait QueryId {
     type QueryId: Any;
     const HAS_STATIC_QUERY_ID: bool;
     fn query_id() -> Option<TypeId> {}
}

impl<L, R> QueryId for Eq<L, R>
where L: QueryId, R: QueryId
{
    type QueryId = Self;
    const HAS_STATIC_QUERY_ID: bool =
       L::HAS_STATIC_QUERY_ID && R::HAS_STATIC_QUERY_ID;
}

Fundamental traits (Expression)

pub trait Expression {
    type SqlType;
}

impl<L, R> Expression for Eq<L, R> {
    type SqlType = Bool;
}

• A marker trait representing a typed SQL fragment
• Used for type checking the final query

Expanded example query

students::table.filter(students::id.eq(42)).select(students::id)

SelectStatement<
    students::table, // The table type, indicating the from clause
    SelectClause<students::id>, // The select clause
    WhereClause<Eq<students::id, Bound<i32, Integer>>>, // The where clause
    // Skipped some more parameters for other clauses
>

Select<
    Filter<students::table, Eq<students::id, i32>>,
    students::id
>

Conditional queries

let mut query = students::table.select(students::name);

if let Some(id) = filter_by_id {
    query = query.filter(students::id.eq(id));
}

• Does not work because the query with filter has a different type than the query without filter

Conditional queries

let mut query = students::table.select(students::name)
     .into_boxed();

if let Some(id) = filter_by_id {
    query = query.filter(students::id.eq(id));
}

• .into_boxed creates a boxed select statement that erases all generic arguments beside of the table name and target backend for the given select statement

Extending our example:

• Add a column to the post table indicating the state of the post.
  • Represented as Enum on rust and postgresql side. Values Draft, Published, Deleted
• Add a route that performs custom filter/order operations based on the query string of the route
  • Something like http://localhost/posts?name="foo"&order="id"

Extending our example

• Add a route that performs allows to pagination on the post table
  • We want http://localhost/posts/page/$page_number?post_count=$postcount to return the number of total pages and the posts of the requested page.

GraphQL

• A query language for API

{
    Teachers {
        teacherName: name
        students {
            name
        }
    }
}

What is wundergraph

• Wundergraph is a crate that helps you with developing a diesel/juniper based GraphQL service in rust
• Enables easy integration of diesel and juniper
• Provides building blocks for writing GraphQL interfaces on top of relational database

Juniper

• Foundational crate for writing GraphQL APIs in rust
• Independ from the actual data source
• Provides foundational building blocks and helper APIs

Juniper

#[derive(GraphQLObject)]
struct Student {
    id: i32,
    name: String,
}

struct Query;

juniper::graphql_object!(Query: Context |&self| {
    field apiVersion() -> &str {
        "1.0"
    }

    field student(&executor, id: i32) -> FieldResult<Student> {
        let context = executor.context();
        context.load_student_with_id(id)
    }
});

Diesel + Juniper (naive way)

#[derive(GraphQLObject, Queryable)]
struct Student {
    id: i32,
    name: String,
}

#[derive(Queryable)]
struct Teacher {
    id: i32,
    name: String,
}

juniper::graphql_object!(Teacher: Context |&self| {
    field id() -> i32 {
        self.id
    }

    field name() -> &str {
        &self.name
    }

    field students(&executor) -> FieldResult<Vec<Student>> {
        let conn = executor.context().pool.get_connection()?;

        let students = students::table
           .filter(students::supervisor.eq(self.id))
           .load::<Student>(&conn)?;

        Ok(students)
    }
})

Problems with the naive way

• N + 1 Query problems
• Loads data not required to answer the request
• Adding options to filter/order/limit non trivial to add

Wundergraph Example

#[derive(Identifiable, WundergraphEntity)]
#[table_name = "students"]
struct Student {
    id: i32,
    name: String,
    supervisor: HasOne<i32, Teacher>,
}

#[derive(Identifiable, WundergraphEntity)]
#[table_name = "teachers"]
struct Teacher {
    id: i32,
    name: String,
    students: HasMany<Student, students::supervisor>,
}

wundergraph::query_object! {
    Query {
        Student,
        Teacher
    }
}

WundergraphEntity

#[derive(WundergraphEntity, Identifiable)]
#[table_name = "student"]
#[primary_key(id)]
/// GraphQL type description
struct Student {
    /// GraphQL field description
    id: i32,
    #[wundergraph(graphql_name = "name")]
    #[column_name = "name"]
    name: String,
    supervisor: HasOne<i32, Teacher>,
    papers: HasMany<Paper, papers::student>,
}

• Marks a type as compatible wundergraph entity
• Controls corresponding GraphQL type
• Field names map directly to corresponding columns in table

query_object!

wundergraph::query_object! {
    /// GraphQL description for query
    Query {
        /// GraphQL description for Student
        #[deprecated(note = "Why")]
        Student,
        #[wundergraph(filter = true, offset = true, order = true)]
        Teacher,
    }
}

Mutation

• Per convention there is a “special” mutation object in GraphQL schemas allowing to mutate data
• Mutations are done by requesting a field on the mutation object and passing mutation data as arguments

mutation CreateStudent {
    createStudent(name: "weiznich", supervisor: 42) {
        id
        name
    }
}

Mutation (implementation)

• Structs implementing …
  • Insertable are automatically usable as insert mutation
  • AsChangeset are automatically usable as update mutation
• Delete mutations are automatically provided via primary keys
• Possible to manual implement custom behaviour by using corresponding traits in wundergraph::query_builder::mutations

Mutation (implementation)

#[derive(Insertable, GraphQLInputObject)]
#[table_name = "students"]
struct NewStudent {
    name: String,
    supervisor: i32,
}

#[derive(AsChangeset, Identifiable, GraphQLInputObject)]
#[table_name = "studens"]
struct StudentChangeset {
    id: i32,
    name: String
}

mutation_object!

wundergraph::mutation_object! {
    /// GraphQL description for mutation
    Mutation {
        Student(insert = NewStudent, update = StudentChangeset, delete = true),
    }
}

Custom Context

• Wundergraph requires you to use a type that implements WundergraphContext as juniper context type
• By default that’s implemented for all types that implement diesel’s Connection trait

QueryModifier

trait QueryModifier<L, DB> {
   fn modify_query(
       &self,
       select: &LookAheadSelection<WundergraphScalarValue>,
       query: BoxedQuery<L, DB, Self>,
   ) -> Result<BoxedQuery<L, DB, Self>>;
}

• A trait that allows you to get access to the final query before execution
• Implemented for the context type, so you are required to use a custom context type if you want to use this trait
• Entities passed as first type parameter

Wundergraph CLI

• Wundergraph CLI is a small CLI helper tool to generate struct definitions from an existing database
• Works similar to diesel CLI
• Generates for each database table:
  • A diesel table! macro call
  • A matching query entity
  • An insert type used for insert mutations
  • A changeset type used for update mutations

Exercise

• Implement the REST API from our first exercise now as GraphQL API:
  0. Add juniper and wundergraph as dependency in your Cargo.toml
  1. Add a juniper GraphQL endpoint to your application
  2. Setup Query/Mutation entities for all three tables
• Note: Project template from yesterday at: https://github.com/weiznich/wundergraph-workshop/

Overview

• Talk a bit about the implementation of wundergraph
• Technical aspects
• How to provide custom implementations

Technical challenges using diesel

• Diesel want’s to know the number of selected columns at compile time
  • Just select constant NULL instead of removing the field
• Arbitrary runtime joining between tables is not possible to implement using diesel
  • Use diesel Associations approach involving a fixed number of queries instead

Technical challenges using diesel

• Dynamic query construction in a generic case
  • BoxedSelectStatement helps there, but writing generic code also required
• Diesel supports primitive types not supported by default as Juniper scalar values
  • Provide own Juniper scalar value implementation

LoadingHandler

pub trait LoadingHandler<DB, Ctx> : HasTable + Sized
where DB: Backend
{
    type Columns;
    type FieldList;
    type PrimaryKeyIndex;
    type Filter;

    const FIELD_NAMES: &[&str];
    const TYPE_NAME: &str;
}

• Central trait for querying GraphQL entities

WundergraphBelongsTo

pub trait WundergraphBelongsTo<Other, DB, Ctx, FK>: LoadingHandler<DB, Ctx> {
    type Key;

    fn resolve(
        glob_args: &[LookAheadArgument<WundergraphScalarValue>],
        look_ahead: &[LookAheadSelection<WundergraphScalarValue>],
        selection: Option<&[Selection<WundergraphScalarValue>],
        keys: &[Option<Self::Key>],
        executor: Executor<Ctx, WundergraphScalarValue>,
    ) -> Result<
        HashMap<
            Option<Self::Key>,
            Vec<Value<WundergraphScalarValue>>
        >>;
}

Filter infrastructure

• Automatically generated for structs implementing LoadingHandler
• Composable out simple filters for each type

trait FilterValue<C> {
    type RawValue;
    type AdditionalFilter;
}

Tuple mappers

• Internally most of the things in wundergraph operate on tuples at a type level to generate things
• Basic work flow:
  • Filter unwanted types for a tuple
  • Depending on the context generate juniper/diesel data structure from the tuple by mapping each “field” to a corresponding data field

Tuple mappers

trait ExtractTableFields {
    type Out;
}

impl<T1, T2> ExtractTableFields for (T1, T2)
where T1: WundergraphValue, (T2,): ExtractTableFields
     <(T2,) as ExtractTableFields>::Out: AppendToTuple<T1>
{
    type Out = <<(T2,) as ExtractTableFields>::Out
        as AppendToTuple<T1>>::Out;
}

impl<C, FK, T2> ExtractTableFields for (HasMany<C, FK>, T2)
where (T2,): ExtractTableFields
{
    type Out = <(T2,) as ExtractTableFields>::Out;
}

Final exercise

• We want to store different versions of the same post.
  • A version is represented as range of starting and ending version
  • Write a SQL function that could be used instead of the posts table in the from clause.
    • SELECT * FROM post_at_version(5);
  • Integrate that in our GraphQL interface