When venturing into Flutter mobile app development, selecting the right tools can significantly impact your app and development experience. At Lollipop, we used a REST API like most companies until our backend developer, Christian, introduced us to Twirp, a simple Remote Procedure Call (RPC) framework. This began our journey to switch from REST to RPC using Twirp.

In this first part of this two-part blog series, we'll discuss three key ingredients that made API development at Lollipop easier:

1. We'll explore Protocol Buffers, Google's method for serialising structured data and enabling efficient communication between your API and mobile app.
2. We'll delve into Twirp, a straightforward RPC framework developed by Twitch.
3. We'll introduce buf.build, a management system for Protocol Buffers that ensures consistency in your schemas.

This blog will provide an overview of how these technologies synergise and work together, shedding light on their integrated workflow. Additionally, we'll explain why and how we chose to use them and discuss alternative options.

Stay tuned for the second part, where we'll provide a step-by-step tutorial on integrating these concepts into a Flutter app. Let's get started!

💬 Protocol Buffers: Structuring the Conversation

When we started to look into Twirp, the first new concept was Protocol Buffers, aka Protobuf. When learning a new tool or language, knowing why it was developed and how it is used today can be helpful. So let's start with a history lesson.

The Origin Story: Why Google Created Protocol Buffers

Google started developing Protobufs in the early 2000s. They were searching for a more efficient way to drive their internal services at scale. The primary goals were to create a simple, smaller format, faster to serialise/deserialise than XML, and language-agnostic.

In 2008, Google decided to open source Protocol Buffers. The first public release, Proto1, supported generating code from Protobuf schemas in several languages. It significantly contributed to the open-source community and became a popular choice for inter-service communication in a microservices architecture.

Over time, Protobufs have become a fundamental component of many systems inside Google and in the broader community, providing a robust and efficient mechanism for data serialisation. They are widely used for communication and storage purposes and have gained popularity in many architectures due to their efficiency and language neutrality.

A quick look at what Protocol Buffers are

So you might think Protobuf is just a simpler, smaller, faster version of XML or JSON. While this is true, it also has another benefit: it is an Interface Definition Language (IDL). This means that while they still define the data structure, they also define the service's functionality, i.e., the methods you can call. To illustrate this, let's look at the following protobuf:

syntax = "proto3";

package icecreamshop;

// The IceCreamService provides operations for managing orders.
service IceCreamService {
  // Creates a new order.
  rpc CreateOrder (IceCream) returns (Order) {}
  
  // Fetches details about an order.
  rpc GetOrderDetails (GetOrderDetailsRequest) returns (Order) {}

  // Marks an order as ready.
  rpc MarkOrderReady (MarkOrderReadyRequest) returns (Order) {}
}

// The IceCream message represents an ice cream in our system.
message IceCream {
  string flavor = 1;
  string size = 2;
  repeated string toppings = 3;
}

// The Order message represents an order in our system.
message Order {
  int32 order_id = 1;
  IceCream ice_cream = 2;
  bool ready = 3;  // Is the order ready for pickup?
}

// The GetOrderDetailsRequest represents a request to fetch details about an order.
message GetOrderDetailsRequest {
  int32 order_id = 1;
}

// The MarkOrderReadyRequest represents a request to mark an order as ready.
message MarkOrderReadyRequest {
  int32 order_id = 1;
}

In this simple example, we have modelled an ice cream shop. The first two lines define the version of proto we are using, i.e., proto3 and the package name icecreamshop used in code generation. Next, we define the service calls, where you can:

• Create an order CreateOrder
• Get the order details GetOrderDetails
• Mark an Order as ready to pick up MarkOrderReady

And finally, we have defined the data types for this service:

• IceCream
• Order
• GetOrderDetailsRequest
• MarkOrderReadyRequest

With this definition, we can generate a client or server using the command protoc in our preferred language. To generate the files in Dart from the .proto file in the same directory, the command would be:

protoc --dart_out=grpc:./ -I=./ ./icecreamshop.proto

brew install protobuf
dart pub global activate protoc_plugin 

This will create the following files:

• icecreamshop.pb.dart

  • Contains the Dart classes corresponding to the messages defined in our .proto file. This includes properties for each field and methods for serialisation and deserialisation to and from binary.

• icecreamshop.pbenum.dart

  • Contains the enums from the .proto file. We don't have any, so this file is empty.

• icecreamshop.pbjson.dart

  • Provides functionality for converting between the Dart message classes and JSON. This allows you to serialise and deserialise the Protocol Buffer messages to and from JSON format, which can be handy when you need the data in a human-readable format.

• icecreamshop.pbgrpc.dart

  • Contains the Dart classes for the gRPC services defined in our .proto file. This includes the client IceCreamServiceClient and the server-side IceCreamServiceBase, allowing you to call or implement the services using gRPC in Dart. This file is generated because we specified gRPC in our protoc command.

We now have a client and service generated from the .proto file and need to implement them. The nice thing about them is that they communicate using a binary format, one of the massive advantages of using an RPC service. Let us see the other advantages Protobuf can bring us.

Advantages of Using Protobuf

The advantages of using Protobuf are numerous. It's highly efficient in size and speed, offering significantly faster serialisation and deserialisation than other formats like JSON. It can be ~3-10 times smaller than XML/JSON and anywhere from 20-100 times faster. It is ideal for Mobile apps as we want to transfer a small data payload as quickly as possible.

It's language-neutral and platform-neutral, meaning you can use it across different programming languages and platforms. If we wanted to create the server in Go, we could regenerate the client and server using the Go plugin for protoc and voilà!; we have the start of the Go Server that can communicate with our Dart client.

Moreover, it offers backward and forward compatibility, ensuring your APIs stay robust even as your data structures evolve. Unlike JSON, it is trivial to rename fields.

Now that we know what protobufs are, the next step in our secret recipe is changing the type of RPC framework we use. The example above used gRPC, but we will explore Twirp and the advantages it brings to the table.

💆 Twirp: gRPC but without the headaches

As we progress in our recipe for success, the next ingredient is Twirp, a simple RPC (Remote Procedure Call) framework created by Twitch. Like gRPC, it provides a structured way for clients and servers to communicate, simplifying the creation of services that your Flutter app can seamlessly interact with. But again, let us look into why it was created.

The Origin story: Why Twitch Created Twirp

The best place to understand why Twitch created Twirp is to look at their 2018 introduction article for Twirp. This article shows that the main reason for Twirp's creation was to address the pain points of gRPC, such as:

• Lack of HTTP 1.1 support - Issue with load balancers and other tools
• Google introducing breaking changes without warning
• Complexities of having an HTTP/2 handler
• Difficulties with binary protobuf for debugging - tho this might be different now that Postman supports gRPC

So, some issues with gRPC at the time caused Twitch enough pain to create its own RPC framework. Also, from the tweet below from Fatih Arslan, that even in 2021, there were some issues with gRPC.

Fatih Arslan showing the success that GitHub had with Twirp!

I do not have a lot of experience in writing the backend code, but it seems that their has been some pain point in using gRPC.

One aspect that gRPC has that Twirp doesn't is full-duplex streams, where the server or client can receive or send streams of data. This may be of benefit to some APIs but for our usecase at Lollipop it was not. There is also an alternative that we will show in the conclusion. Now that we know why Twipr was create lets look at what other benfits it brings.

A quick look at what Twirp is

Advantages of Using Protobuf

Garbage and notes below

buf.build

The final ingredient in our toolkit is buf.build. As a Protobuf management tool, buf.build aids in making your Protobuf schemas clean, consistent, and efficient.

What is buf.build and What is its Purpose?

buf.build is a Protobuf tool that helps developers maintain high-quality Protobuf schemas. It provides features like linting and breaking change detection to keep your schemas clean and consistent.

The Contribution of buf.build to Flutter Apps

In Flutter app development, buf.build can ensure your Protobuf schemas stay clear and error-free, leading to more efficient and reliable apps. It assists with schema management, reduces the risk of breaking changes, and helps enforce best practices.

Benefits of buf.build Usage in Development

buf.build offers several advantages. It promotes Protobuf best practices, detects breaking changes before they become a problem, and encourages consistency across your schemas. It also integrates well with existing CI/CD pipelines, helping to automate and streamline your development process.

Each of these components plays a vital role in Flutter mobile app development. In the next section, we'll explore how they work together to provide a seamless and efficient development process.

By understanding these key components individually, we can better appreciate how they come together to create a robust, efficient, and effective Flutter mobile app development process. But how do they work together? Let's explore that next.

How Twirp, buf.build, and Protobuf Work Together

Understanding how Protobuf, Twirp, and buf.build work in isolation is only half the story. When combined, they make a powerful toolset that can supercharge your Flutter development process.

Overview of the Integration Process

The integration process of Protobuf, Twirp, and buf.build revolves around the idea of using Protobuf to define your application's data structure and services. Protobuf schemas serve as the single source of truth that dictates how your application's services communicate.

Twirp uses these Protobuf definitions to provide a simple, straightforward RPC framework that your application can use to facilitate communication between different services.

Meanwhile, buf.build ensures that these Protobuf schemas remain clean, consistent, and maintainable. It helps enforce best practices, detects breaking changes early, and even assists in automating parts of your development process through CI/CD integration.

Explanation of How Twirp, buf.build, and Protobuf Complement Each Other

Twirp, buf.build, and Protobuf are designed to work together seamlessly:

Protobuf provides a way to define your application's data structures and services clearly and efficiently. Twirp provides an RPC framework that uses these Protobuf definitions, allowing your application's services to communicate effectively and without confusion. buf.build takes care of the management of your Protobuf schemas, ensuring they adhere to best practices, remain consistent, and evolve without introducing breaking changes. This harmony enables the creation of robust, efficient, and scalable Flutter applications.

Details of the Unified Workflow

The unified workflow begins with defining your Protobuf schemas. These schemas outline the structure of the data your application will use and the services that will manipulate this data. Once these definitions are in place, Twirp uses them to facilitate communication between services, ensuring that each service understands exactly how to interact with others.

buf.build enters the workflow by taking on the responsibility of maintaining your Protobuf schemas. It lints your schemas against configurable rules, warns of any breaking changes, and can even publish your schemas to a buf.build registry, making it easier to share them across your team.

Together, Protobuf, Twirp, and buf.build offer a unified and efficient workflow for building APIs form your Flutter mobile apps.

Conclusion

In this first part of our exploration, we have unpacked the what, why, and how of Protobuf, Twirp, and buf.build in the realm of Flutter mobile app development. These powerful tools, when combined, offer an efficient, scalable, and robust way to build superior APIs for mobile applications.

But our journey doesn't end here. In the next part, we will move from theory to practice, showcasing how to put these ingredients into action through a step-by-step guide to building a Flutter mobile app. We'll see you there!

References

A. Links and references to key resources and further reading

• Twirp GitHub
• buf.connect
• buf.connect reddit post
• buf.build introduction blog post for Connect
• RPC in Go using Twitch's Twirp
• buf build
• buf build BSR
• buf build BSR intro
• Buf CLI
• reddit experience with grpc
• reddit Alternatives to gRPC
• reddit using Twirp over gRPC

--- NOTES ---

• Twirp or something like it solves these problems, also ensures forwards and backwards compatibility of APIs through protobuf

• very interesting as a means for having more structured but flexible APIs with less of the faff.

• buf manages our schema and code generators, Twirp is one of those generators, taking the service definitions in our schema and generating client and server stubs

• The automatic documentation is pretty cool and it pull in additional doc comments from the definition file as well if you add some, very discoverable.