Dynamic Protobuf in Go: Runtime Schema Hot-Reload via Custom protoc Plugin

Go 项目里 Protobuf 通常是编译期生成静态结构体，但当 schema 要到运行时才能确定——多租户平台、插件架构、或者 schema 频繁变更又不想每次重新部署——就需要一种不依赖预生成代码的动态方案。

本文的路径：先理清 Protobuf 的编译管线（1-3 节），然后基于 dynamicpb + 自定义 protoc 插件实现运行时 schema 加载（第 4 节）。想直接看代码的跳到 4.3。

---

1. Background: Why Protobuf

Protocol Buffers (Protobuf) is a language-neutral, platform-neutral serialization mechanism developed by Google. Compared to text-based formats like JSON and XML, Protobuf offers:

• Compact binary encoding — significantly smaller payloads
• Fast serialization / deserialization — critical for high-throughput systems
• Strong schema contracts — .proto files serve as the single source of truth for data structures
• Cross-language support — generated code available for Go, Java, Python, C++, etc.

These properties make Protobuf the de facto choice for gRPC services, inter-process communication, and high-performance data pipelines.

---

2. The Static Compilation Model and Its Limitations

2.1 How Static Compilation Works

The standard Protobuf workflow: define message types in .proto files, run protoc with a language-specific plugin (e.g., protoc-gen-go), get type-safe structs implementing proto.Message, used with proto.Marshal and proto.Unmarshal. .proto file → protoc → generated Go code → compiled binary.

2.2 Where Static Compilation Falls Short

This model works well when schemas are stable and known at compile time. But it introduces friction in several real-world scenarios:

In all these cases, you need a way to work with Protobuf messages dynamically — without pre-generated Go structs.

---

3. Dynamic Compilation: Key Concepts

Three foundational concepts make dynamic Protobuf possible.

3.1 Dynamic Message

A Dynamic Message is a Protobuf message object whose fields can be accessed and manipulated at runtime, without a pre-generated struct. In Go, this is provided by the dynamicpb package.

You use dynamic messages when:

• The schema is loaded at runtime (e.g., from a file, database, or config center)
• The message type is determined by external input (e.g., a message name in a request header)

3.2 Reflection API

The Protobuf Reflection API allows you to inspect message structure at runtime:

• Descriptors — metadata objects describing fields, types, and the overall structure of messages
• protoreflect package — Go's implementation of the reflection API, providing FileDescriptor, MessageDescriptor, FieldDescriptor, etc.

The key components form a hierarchy: FileDescriptor → MessageDescriptor → FieldDescriptor (name, number, type, label, etc.).

3.3 Dynamic Code Generation via protoc Plugin

In some cases, you need to extract schema metadata from .proto files programmatically. The protoc compiler supports a plugin architecture: it compiles .proto files into FileDescriptorProto objects and streams them to plugins via stdin. Plugins can then process this metadata however they need — including serializing it to JSON for runtime consumption.

---

4. Implementation in Go

4.1 The Core Challenge

Dynamic Protobuf does not require loading generated classes, in Java or Go. Go's dynamicpb builds messages from descriptors; the application uses reflection rather than generating new Go struct types.

The key insight comes from analyzing the Protobuf library's internals. The conversion path from a .proto file to a usable proto.Message:

This gives us two concrete questions to solve:

1. How to obtain a FileDescriptor at runtime (without running protoc at runtime)
2. How to create a proto.Message from a FileDescriptor (without generated structs)

Question 2: Creating Messages from FileDescriptor

The second question is straightforward — dynamicpb handles it directly:

func NewMessages(fd protoreflect.FileDescriptor, msgName string) proto.Message {
    md := fd.Messages().ByName(protoreflect.Name(msgName))
    if md == nil {
        return nil
    }
    return dynamicpb.NewMessage(md)
}

Question 1: Obtaining FileDescriptor at Runtime

The google.golang.org/protobuf runtime does not include a .proto source compiler. A separate compiler can process source at runtime, but this design deliberately compiles offline, then creates a FileDescriptor from FileDescriptorProto:

fdp := new(descriptorpb.FileDescriptorProto)
// Unmarshal from binary or text format...
fd, err := protodesc.NewFile(fdp, nil)

The refined conversion path is the same as the diagram above — with one critical detail: FileDescriptorProto is itself a proto.Message, it can be serialized to binary, JSON, or text format — and deserialized at runtime. The question now is: how do we produce the serialized FileDescriptorProto from a .proto file?

4.2 How protoc Plugins Work

The protoc plugin architecture solves this. When protoc invokes a plugin, it sends a CodeGeneratorRequest via stdin containing the compiled FileDescriptorProto objects. Here is the relevant source code from the Protobuf Go protogen package:

// Simplified plugin protocol, not a complete copy of protogen implementation
func run(opts Options, f func(*Plugin) error) error {
    if len(os.Args) > 1 {
        return fmt.Errorf("unknown argument %q (this program should be run by protoc, not directly)", os.Args[1])
    }
    // Read the compiled binary stream from protoc
    in, err := io.ReadAll(os.Stdin)
    if err != nil {
        return err
    }

    req := &pluginpb.CodeGeneratorRequest{}
    if err := proto.Unmarshal(in, req); err != nil {
        return err
    }
    gen, err := opts.New(req)
    if err != nil {
        return err
    }
    // Execute the plugin's custom processing logic
    if err := f(gen); err != nil {
        gen.Error(err)
    }
    resp := gen.Response()
    out, err := proto.Marshal(resp)
    if err != nil {
        return err
    }
    // Write the response (generated files) to stdout
    if _, err := os.Stdout.Write(out); err != nil {
        return err
    }
    return nil
}

The CodeGeneratorRequest contains the FileDescriptorProto we need:

type CodeGeneratorRequest struct {
    FileToGenerate        []string
    Parameter             *string
    ProtoFile             []*descriptorpb.FileDescriptorProto
    SourceFileDescriptors []*descriptorpb.FileDescriptorProto
    CompilerVersion       *Version
    // ...
}

This means we can build a custom protoc plugin that, instead of generating Go source code, outputs the serialized FileDescriptorProto in a runtime-friendly format.

4.3 Building the Custom Plugin

We choose JSON as the serialization format for FileDescriptorProto because it's human-readable, easy to store in configuration centers, and widely supported.

package main

import (
    "google.golang.org/protobuf/compiler/protogen"
    "google.golang.org/protobuf/encoding/protojson"
    "google.golang.org/protobuf/types/pluginpb"
)

func main() {
    protogen.Options{}.Run(func(gen *protogen.Plugin) error {
        gen.SupportedFeatures = uint64(pluginpb.CodeGeneratorResponse_FEATURE_PROTO3_OPTIONAL)
        for _, file := range gen.Files {
            if !file.Generate {
                continue
            }
            genJsonFile(file, gen)
        }
        return nil
    })
}

func genJsonFile(file *protogen.File, gen *protogen.Plugin) {
    fd := file.Proto
    // Temporarily strip SourceCodeInfo to reduce output size
    sci := fd.SourceCodeInfo
    fd.SourceCodeInfo = nil
    defer func() { fd.SourceCodeInfo = sci }()

    jsonFile := gen.NewGeneratedFile(file.GeneratedFilenamePrefix+".json", ".")
    jsonFile.P(protojson.Format(fd))
}

Why JSON over binary or proto-text?

For systems that prioritize hot-reload via a configuration center, JSON strikes the best balance between readability and practicality.

JSON Output Example

For a .proto file like:

syntax = "proto3";
package tns.search.proto;
option go_package = "./gen;protobuf";

message TnsDemo {
  int64 id = 1;
  int32 status = 2;
  map<string, string> result = 3;
  repeated int32 reasons = 4;
}

For this dependency-free example, the plugin produces a descriptor encoded as JSON, not a JSON Schema validation document:

{
  "name": "protobuf/tns_demo.proto",
  "package": "tns.search.proto",
  "messageType": [
    {
      "name": "TnsDemo",
      "field": [
        {
          "name": "id",
          "number": 1,
          "label": "LABEL_OPTIONAL",
          "type": "TYPE_INT64",
          "jsonName": "id"
        },
        {
          "name": "status",
          "number": 2,
          "label": "LABEL_OPTIONAL",
          "type": "TYPE_INT32",
          "jsonName": "status"
        },
        {
          "name": "result",
          "number": 3,
          "label": "LABEL_REPEATED",
          "type": "TYPE_MESSAGE",
          "typeName": ".tns.search.proto.TnsDemo.ResultEntry",
          "jsonName": "result"
        },
        {
          "name": "reasons",
          "number": 4,
          "label": "LABEL_REPEATED",
          "type": "TYPE_INT32",
          "jsonName": "reasons"
        }
      ],
      "nestedType": [
        {
          "name": "ResultEntry",
          "field": [
            {
              "name": "key",
              "number": 1,
              "label": "LABEL_OPTIONAL",
              "type": "TYPE_STRING",
              "jsonName": "key"
            },
            {
              "name": "value",
              "number": 2,
              "label": "LABEL_OPTIONAL",
              "type": "TYPE_STRING",
              "jsonName": "value"
            }
          ],
          "options": {
            "mapEntry": true
          }
        }
      ]
    }
  ],
  "options": {
    "goPackage": "./gen;protobuf"
  },
  "syntax": "proto3"
}

4.4 Generating the JSON Schema

Build the plugin and run it alongside protoc:

SRC_DIR=$(pwd)

# Build the custom plugin after initializing its module and dependencies
go build -o "$SRC_DIR/protoc-gen-ext"

# Run the descriptor plugin; the output directory must already exist
protoc --proto_path="$SRC_DIR" \
  --plugin=protoc-gen-ext="$SRC_DIR/protoc-gen-ext" \
  --ext_out="$SRC_DIR/protobuf" \
  "$SRC_DIR"/protobuf/*.proto

This exports JSON descriptors without generating static Go code. Add --go_out only when another consumer needs it. Store versioned descriptors in the configuration center for runtime access.

4.5 Using Dynamic Schema at Runtime

With the JSON schema available (e.g., from a config center, database, or file), the runtime usage is straightforward:

package main

import (
    "fmt"
    "google.golang.org/protobuf/encoding/protojson"
    "google.golang.org/protobuf/reflect/protodesc"
    "google.golang.org/protobuf/reflect/protoreflect"
    "google.golang.org/protobuf/types/descriptorpb"
    "google.golang.org/protobuf/types/dynamicpb"
)

func LoadDynamicMessage(jsonSchema []byte, messageName string) (*dynamicpb.Message, error) {
    // Step 1: Deserialize JSON into FileDescriptorProto
    fdp := new(descriptorpb.FileDescriptorProto)
    if err := protojson.Unmarshal(jsonSchema, fdp); err != nil {
        return nil, fmt.Errorf("unmarshal schema: %w", err)
    }

    // Step 2: Create FileDescriptor from FileDescriptorProto
    fd, err := protodesc.NewFile(fdp, nil)
    if err != nil {
        return nil, fmt.Errorf("create file descriptor: %w", err)
    }

    // Step 3: Find the target MessageDescriptor
    md := fd.Messages().ByName(protoreflect.Name(messageName))
    if md == nil {
        return nil, fmt.Errorf("message %q not found in schema", messageName)
    }

    // Step 4: Create a dynamic message instance
    return dynamicpb.NewMessage(md), nil
}

Once you have the dynamicpb.Message, you can use it like any other proto.Message:

// Unmarshal binary Protobuf data into the dynamic message
msg, err := LoadDynamicMessage(jsonSchema, "TnsDemo")
if err != nil {
    log.Fatal(err)
}
if err := proto.Unmarshal(binaryData, msg); err != nil {
    log.Fatal(err)
}

// Access fields via reflection
idField := msg.Descriptor().Fields().ByName("id")
if idField == nil || idField.Kind() != protoreflect.Int64Kind {
    log.Fatal("schema must define int64 id")
}
fmt.Println("id:", msg.Get(idField).Int())

// Marshal back to binary or JSON
jsonBytes, err := protojson.Marshal(msg)
if err != nil {
    log.Fatal(err)
}
fmt.Println(string(jsonBytes))

The loader accepts a top-level message name in a dependency-free file. protodesc.NewFile(fdp, nil) cannot resolve imports. For a complete registry, export transitive dependencies, load the FileDescriptorSet using protodesc.NewFiles, and look up messages by fully qualified name. dynamicpb.NewTypes supplies message and extension resolution for Any and extension-aware decoding. Keep a published registry immutable while requests use it. See the descriptor API and dynamic type registry.

A custom plugin is useful for JSON packaging or additional metadata. To export binary descriptors alone, protoc --descriptor_set_out=schema.pb --include_imports already provides a descriptor set.

4.6 Hot-Reload Architecture

The following diagram shows the intended distribution path; the sample loader does not itself implement a watcher, version store, or atomic publication:

When the .proto schema changes:

1. Re-run protoc with the custom plugin (offline)
2. Update the JSON in your config center
3. Validate dependencies and compatibility, construct a new registry, and atomically publish it for new requests
4. Let in-flight requests retain their original registry; reject invalid updates and keep the last working version

Field descriptors belong to their descriptor graph. Do not cache a FieldDescriptor from an old registry and use it with a message constructed from a newly loaded registry; resolve fields against the message's own descriptor or keep the entire version together.

Schema loading does not implement new business behavior. Preserve field-number compatibility and never reuse deleted tags; changes to application logic still require their own release. See the Protobuf evolution rules.

---

5. Considerations and Trade-offs

Dynamic Protobuf is powerful but comes with trade-offs you should be aware of:

When to use dynamic Protobuf:

• Schema changes frequently and redeployment is costly
• You're building a generic platform that handles arbitrary message types
• You need to decouple schema evolution from application deployment

When to stick with static compilation:

• Schema is stable and known at compile time
• Performance is critical and reflection overhead is unacceptable
• Type safety and developer experience are priorities

---

6. Conclusion

The useful separation is offline schema compilation versus runtime message handling: .proto → FileDescriptorProto → FileDescriptor → dynamicpb.Message. A custom plugin can export JSON descriptors, while standard protoc options can export a binary descriptor set. Combined with dependency validation, versioned registries and atomic publication, this allows generic schema handling to evolve without rebuilding the application for every compatible field change. Business logic still needs its own release process.