sonr/docs/docs/guides/tools/cosmos-proto.md

Protocol Buffers in Cosmos SDK

Overview

The Cosmos SDK uses Protocol Buffers for serialization and API definitions. Generation is handled via a Docker image: ghcr.io/cosmos/proto-builder:0.15.x.

Generation Tools

• Buf: Primary tool for protobuf management
• protocgen.sh: Core generation script in scripts/
• Makefile Commands: Standard commands for generate, lint, format

Key Components

Buf Configuration

1. Workspace Setup

   • Root level buf workspace configuration
   • Manages multiple protobuf directories

2. Directory Structure

   proto/
   ├── buf.gen.gogo.yaml    # GoGo Protobuf generation
   ├── buf.gen.pulsar.yaml  # Pulsar API generation  
   ├── buf.gen.swagger.yaml # OpenAPI/Swagger docs
   ├── buf.lock            # Dependencies
   ├── buf.yaml            # Core configuration
   ├── cosmos/            # Core protos
   └── tendermint/        # Consensus protos
   

3. Module Protos

   • Located in x/{moduleName}/proto
   • Module-specific message definitions

buf.gen.gogo.yaml

buf.gen.gogo.yaml defines how the protobuf files should be generated for use with in the module. This file uses gogoproto, a separate generator from the google go-proto generator that makes working with various objects more ergonomic, and it has more performant encode and decode steps

https://github.com/cosmos/cosmos-sdk/blob/main/proto/buf.gen.gogo.yaml#L1-L9

buf.gen.pulsar.yaml

buf.gen.pulsar.yaml defines how protobuf files should be generated using the new golang apiv2 of protobuf. This generator is used instead of the google go-proto generator because it has some extra helpers for Cosmos SDK applications and will have more performant encode and decode than the google go-proto generator. You can follow the development of this generator here.

https://github.com/cosmos/cosmos-sdk/blob/main/proto/buf.gen.pulsar.yaml#L1-L18

buf.gen.swagger.yaml

buf.gen.swagger.yaml generates the swagger documentation for the query and messages of the chain. This will only define the REST API end points that were defined in the query and msg servers. You can find examples of this here

https://github.com/cosmos/cosmos-sdk/blob/main/proto/buf.gen.swagger.yaml#L1-L6

buf.lock

This is an autogenerated file based off the dependencies required by the .gen files. There is no need to copy the current one. If you depend on cosmos-sdk proto definitions a new entry for the Cosmos SDK will need to be provided. The dependency you will need to use is buf.build/cosmos/cosmos-sdk.

https://github.com/cosmos/cosmos-sdk/blob/main/proto/buf.lock#L1-L16

buf.yaml

buf.yaml defines the name of your package, which breakage checker to use and how to lint your protobuf files.

It is advised to use a tagged version of the buf modules corresponding to the version of the Cosmos SDK being are used.

https://github.com/cosmos/cosmos-sdk/blob/main/proto/buf.yaml#L1-L24

We use a variety of linters for the Cosmos SDK protobuf files. The repo also checks this in ci. A reference to the github actions can be found here

ORM

The Cosmos SDK ORM is a state management library that provides a rich, but opinionated set of tools for managing a module's state. It provides support for:

• type safe management of state
• multipart keys
• secondary indexes
• unique indexes
• easy prefix and range queries
• automatic genesis import/export
• automatic query services for clients, including support for light client proofs (still in development)
• indexing state data in external databases (still in development)

Design and Philosophy

The ORM's data model is inspired by the relational data model found in SQL databases. The core abstraction is a table with a primary key and optional secondary indexes.

Because the Cosmos SDK uses protobuf as its encoding layer, ORM tables are defined directly in .proto files using protobuf options. Each table is defined by a single protobuf message type and a schema of multiple tables is represented by a single .proto file.

Table structure is specified in the same file where messages are defined in order to make it easy to focus on better design of the state layer. Because blockchain state layout is part of the public API for clients (TODO: link to docs on light client proofs), it is important to think about the state layout as being part of the public API of a module. Changing the state layout actually breaks clients, so it is ideal to think through it carefully up front and to aim for a design that will eliminate or minimize breaking changes down the road. Also, good design of state enables building more performant and sophisticated applications. Providing users with a set of tools inspired by relational databases which have a long history of database design best practices and allowing schema to be specified declaratively in a single place are design choices the ORM makes to enable better design and more durable APIs.

Also, by only supporting the table abstraction as opposed to key-value pair maps, it is easy to add to new columns/fields to any data structure without causing a breaking change and the data structures can easily be indexed in any off-the-shelf SQL database for more sophisticated queries.

The encoding of fields in keys is designed to support ordered iteration for all protobuf primitive field types except for bytes as well as the well-known types google.protobuf.Timestamp and google.protobuf.Duration. Encodings are optimized for storage space when it makes sense (see the documentation in cosmos/orm/v1/orm.proto for more details) and table rows do not use extra storage space to store key fields in the value.

We recommend that users of the ORM attempt to follow database design best practices such as normalization (at least 1NF). For instance, defining repeated fields in a table is considered an anti-pattern because breaks first normal form (1NF). Although we support repeated fields in tables, they cannot be used as key fields for this reason. This may seem restrictive but years of best practice (and also experience in the SDK) have shown that following this pattern leads to easier to maintain schemas.

To illustrate the motivation for these principles with an example from the SDK, historically balances were stored as a mapping from account -> map of denom to amount. This did not scale well because an account with 100 token balances needed to be encoded/decoded every time a single coin balance changed. Now balances are stored as account,denom -> amount as in the example above. With the ORM's data model, if we wanted to add a new field to Balance such as unlocked_balance (if vesting accounts were redesigned in this way), it would be easy to add it to this table without requiring a data migration. Because of the ORM's optimizations, the account and denom are only stored in the key part of storage and not in the value leading to both a flexible data model and efficient usage of storage.

Defining Tables

To define a table:

1. create a .proto file to describe the module's state (naming it state.proto is recommended for consistency), and import "cosmos/orm/v1/orm.proto", ex:

syntax = "proto3";
package bank_example;

import "cosmos/orm/v1/orm.proto";

2. define a message for the table, ex:

message Balance {
  bytes account = 1;
  string denom = 2;
  uint64 balance = 3;
}

3. add the cosmos.orm.v1.table option to the table and give the table an id unique within this .proto file:

message Balance {
  option (cosmos.orm.v1.table) = {
    id: 1
  };

  bytes account = 1;
  string denom = 2;
  uint64 balance = 3;
}

4. define the primary key field or fields, as a comma-separated list of the fields from the message which should make up the primary key:

message Balance {
  option (cosmos.orm.v1.table) = {
    id: 1
    primary_key: { fields: "account,denom" }
  };

  bytes account = 1;
  string denom = 2;
  uint64 balance = 3;
}

5. add any desired secondary indexes by specifying an id unique within the table and a comma-separate list of the index fields:

message Balance {
  option (cosmos.orm.v1.table) = {
    id: 1;
    primary_key: { fields: "account,denom" }
    index: { id: 1 fields: "denom" } // this allows querying for the accounts which own a denom
  };

  bytes account = 1;
  string denom   = 2;
  uint64 amount  = 3;
}

Auto-incrementing Primary Keys

A common pattern in SDK modules and in database design is to define tables with a single integer id field with an automatically generated primary key. In the ORM we can do this by setting the auto_increment option to true on the primary key, ex:

message Account {
  option (cosmos.orm.v1.table) = {
    id: 2;
    primary_key: { fields: "id", auto_increment: true }
  };

  uint64 id = 1;
  bytes address = 2;
}

Unique Indexes

A unique index can be added by setting the unique option to true on an index, ex:

message Account {
  option (cosmos.orm.v1.table) = {
    id: 2;
    primary_key: { fields: "id", auto_increment: true }
    index: {id: 1, fields: "address", unique: true}
  };

  uint64 id = 1;
  bytes address = 2;
}

Singletons

The ORM also supports a special type of table with only one row called a singleton. This can be used for storing module parameters. Singletons only need to define a unique id and that cannot conflict with the id of other tables or singletons in the same .proto file. Ex:

message Params {
  option (cosmos.orm.v1.singleton) = {
    id: 3;
  };

  google.protobuf.Duration voting_period = 1;
  uint64 min_threshold = 2;
}

Running Codegen

NOTE: the ORM will only work with protobuf code that implements the google.golang.org/protobuf API. That means it will not work with code generated using gogo-proto.

To install the ORM's code generator, run:

go install cosmossdk.io/orm/cmd/protoc-gen-go-cosmos-orm@latest

The recommended way to run the code generator is to use buf build. This is an example buf.gen.yaml that runs protoc-gen-go, protoc-gen-go-grpc and protoc-gen-go-cosmos-orm using buf managed mode:

version: v1
managed:
  enabled: true
  go_package_prefix:
    default: foo.bar/api # the go package prefix of your package
    override:
      buf.build/cosmos/cosmos-sdk: cosmossdk.io/api # required to import the Cosmos SDK api module
plugins:
  - name: go
    out: .
    opt: paths=source_relative
  - name: go-grpc
    out: .
    opt: paths=source_relative
  - name: go-cosmos-orm
    out: .
    opt: paths=source_relative

Using the ORM in a module

Initialization

To use the ORM in a module, first create a ModuleSchemaDescriptor. This tells the ORM which .proto files have defined an ORM schema and assigns them all a unique non-zero id. Ex:

var MyModuleSchema = &ormv1alpha1.ModuleSchemaDescriptor{
    SchemaFile: []*ormv1alpha1.ModuleSchemaDescriptor_FileEntry{
        {
            Id:            1,
            ProtoFileName: mymodule.File_my_module_state_proto.Path(),
        },
    },
}

In the ORM generated code for a file named state.proto, there should be an interface StateStore that got generated with a constructor NewStateStore that takes a parameter of type ormdb.ModuleDB. Add a reference to StateStore to your module's keeper struct. Ex:

type Keeper struct {
    db StateStore
}

Then instantiate the StateStore instance via an ormdb.ModuleDB that is instantiated from the SchemaDescriptor above and one or more store services from cosmossdk.io/core/store. Ex:

func NewKeeper(storeService store.KVStoreService) (*Keeper, error) {
    modDb, err := ormdb.NewModuleDB(MyModuleSchema, ormdb.ModuleDBOptions{KVStoreService: storeService})
    if err != nil {
        return nil, err
    }
    db, err := NewStateStore(modDb)
    if err != nil {
        return nil, err
    }
    return Keeper{db: db}, nil
}

Using the generated code

The generated code for the ORM contains methods for inserting, updating, deleting and querying table entries. For each table in a .proto file, there is a type-safe table interface implemented in generated code. For instance, for a table named Balance there should be a BalanceTable interface that looks like this:

type BalanceTable interface {
    Insert(ctx context.Context, balance *Balance) error
    Update(ctx context.Context, balance *Balance) error
    Save(ctx context.Context, balance *Balance) error
    Delete(ctx context.Context, balance *Balance) error
    Has(ctx context.Context, account []byte, denom string) (found bool, err error)
    // Get returns nil and an error which responds true to ormerrors.IsNotFound() if the record was not found.
    Get(ctx context.Context, account []byte, denom string) (*Balance, error)
    List(ctx context.Context, prefixKey BalanceIndexKey, opts ...ormlist.Option) (BalanceIterator, error)
    ListRange(ctx context.Context, from, to BalanceIndexKey, opts ...ormlist.Option) (BalanceIterator, error)
    DeleteBy(ctx context.Context, prefixKey BalanceIndexKey) error
    DeleteRange(ctx context.Context, from, to BalanceIndexKey) error

    doNotImplement()
}

This BalanceTable should be accessible from the StateStore interface (assuming our file is named state.proto) via a BalanceTable() accessor method. If all the above example tables/singletons were in the same state.proto, then StateStore would get generated like this:

type BankStore interface {
    BalanceTable() BalanceTable
    AccountTable() AccountTable
    ParamsTable() ParamsTable

    doNotImplement()
}

So to work with the BalanceTable in a keeper method we could use code like this:

func (k keeper) AddBalance(ctx context.Context, acct []byte, denom string, amount uint64) error {
    balance, err := k.db.BalanceTable().Get(ctx, acct, denom)
    if err != nil && !ormerrors.IsNotFound(err) {
        return err
    }

    if balance == nil {
        balance = &Balance{
            Account: acct,
            Denom:   denom,
            Amount:  amount,
        }
    } else {
        balance.Amount = balance.Amount + amount
    }

    return k.db.BalanceTable().Save(ctx, balance)
}

List methods take IndexKey parameters. For instance, BalanceTable.List takes BalanceIndexKey. BalanceIndexKey let's represent index keys for the different indexes (primary and secondary) on the Balance table. The primary key in the Balance table gets a struct BalanceAccountDenomIndexKey and the first index gets an index key BalanceDenomIndexKey. If we wanted to list all the denoms and amounts that an account holds, we would use BalanceAccountDenomIndexKey with a List query just on the account prefix. Ex:

it, err := keeper.db.BalanceTable().List(ctx, BalanceAccountDenomIndexKey{}.WithAccount(acct))

---

sidebar_position: 1

ProtocolBuffer Annotations

This document explains the various protobuf scalars that have been added to make working with protobuf easier for Cosmos SDK application developers

Signer

Signer specifies which field should be used to determine the signer of a message for the Cosmos SDK. This field can be used for clients as well to infer which field should be used to determine the signer of a message.

Read more about the signer field here.

https://github.com/cosmos/cosmos-sdk/blob/e6848d99b55a65d014375b295bdd7f9641aac95e/proto/cosmos/bank/v1beta1/tx.proto#L40

option (cosmos.msg.v1.signer) = "from_address";

Scalar

The scalar type defines a way for clients to understand how to construct protobuf messages according to what is expected by the module and sdk.

(cosmos_proto.scalar) = "cosmos.AddressString"

Example of account address string scalar:

https://github.com/cosmos/cosmos-sdk/blob/e6848d99b55a65d014375b295bdd7f9641aac95e/proto/cosmos/bank/v1beta1/tx.proto#L46

Example of validator address string scalar:

https://github.com/cosmos/cosmos-sdk/blob/e8f28bf5db18b8d6b7e0d94b542ce4cf48fed9d6/proto/cosmos/distribution/v1beta1/query.proto#L87

Example of pubkey scalar:

https://github.com/cosmos/cosmos-sdk/blob/11068bfbcd44a7db8af63b6a8aa079b1718f6040/proto/cosmos/staking/v1beta1/tx.proto#L94

Example of Decimals scalar:

https://github.com/cosmos/cosmos-sdk/blob/e8f28bf5db18b8d6b7e0d94b542ce4cf48fed9d6/proto/cosmos/distribution/v1beta1/distribution.proto#L26

Example of Int scalar:

https://github.com/cosmos/cosmos-sdk/blob/e8f28bf5db18b8d6b7e0d94b542ce4cf48fed9d6/proto/cosmos/gov/v1/gov.proto#L137

There are a few options for what can be provided as a scalar: cosmos.AddressString, cosmos.ValidatorAddressString, cosmos.ConsensusAddressString, cosmos.Int, cosmos.Dec.

Implements_Interface

Implement interface is used to provide information to client tooling like telescope on how to encode and decode protobuf messages.

option (cosmos_proto.implements_interface) = "cosmos.auth.v1beta1.AccountI";

Method,Field,Message Added In

method_added_in, field_added_in and message_added_in are annotations to denotate to clients that a field has been supported in a later version. This is useful when new methods or fields are added in later versions and that the client needs to be aware of what it can call.

The annotation should be worded as follow:

option (cosmos_proto.method_added_in) = "cosmos-sdk v0.50.1";
option (cosmos_proto.method_added_in) = "x/epochs v1.0.0";
option (cosmos_proto.method_added_in) = "simapp v24.0.0";

Amino

The amino codec was removed in v0.50+, this means there is not a need register legacyAminoCodec. To replace the amino codec, Amino protobuf annotations are used to provide information to the amino codec on how to encode and decode protobuf messages.

:::note Amino annotations are only used for backwards compatibility with amino. New modules are not required use amino annotations. :::

The below annotations are used to provide information to the amino codec on how to encode and decode protobuf messages in a backwards compatible manner.

Name

Name specifies the amino name that would show up for the user in order for them see which message they are signing.

option (amino.name) = "cosmos-sdk/BaseAccount";

https://github.com/cosmos/cosmos-sdk/blob/e8f28bf5db18b8d6b7e0d94b542ce4cf48fed9d6/proto/cosmos/bank/v1beta1/tx.proto#L41

Field_Name

Field name specifies the amino name that would show up for the user in order for them see which field they are signing.

uint64 height = 1 [(amino.field_name) = "public_key"];

https://github.com/cosmos/cosmos-sdk/blob/e8f28bf5db18b8d6b7e0d94b542ce4cf48fed9d6/proto/cosmos/distribution/v1beta1/distribution.proto#L166

Dont_OmitEmpty

Dont omitempty specifies that the field should not be omitted when encoding to amino.

repeated cosmos.base.v1beta1.Coin amount = 3 [(amino.dont_omitempty)   = true];

https://github.com/cosmos/cosmos-sdk/blob/e8f28bf5db18b8d6b7e0d94b542ce4cf48fed9d6/proto/cosmos/bank/v1beta1/bank.proto#L56

Encoding

Encoding instructs the amino json marshaler how to encode certain fields that may differ from the standard encoding behaviour. The most common example of this is how repeated cosmos.base.v1beta1.Coin is encoded when using the amino json encoding format. The legacy_coins option tells the json marshaler how to encode a null slice of cosmos.base.v1beta1.Coin.

(amino.encoding)         = "legacy_coins",

https://github.com/cosmos/cosmos-sdk/blob/e8f28bf5db18b8d6b7e0d94b542ce4cf48fed9d6/proto/cosmos/bank/v1beta1/genesis.proto#L23

Another example is a protobuf bytes that contains a valid JSON document. The inline_json option tells the json marshaler to embed the JSON bytes into the wrapping document without escaping.

(amino.encoding)         = "inline_json",

E.g. the bytes containing {"foo":123} in the envelope field would lead to the following JSON:

{
  "envelope": {
    "foo": 123
  }
}

If the bytes are not valid JSON, this leads to JSON broken documents. Thus a JSON validity check needs to be in place at some point of the process.