Package list golang-github-go-kit-kit / 7e965c1
Update documentation throughout the project Peter Bourgon 5 years ago
11 changed file(s) with 76 addition(s) and 867 deletion(s). Raw diff Collapse all Expand all
33 in large organizations. We solve common problems in distributed systems, so
44 you can focus on your business logic.
55
6 - Website: [gokit.io](https://gokit.io)
67 - Mailing list: [go-kit](https://groups.google.com/forum/#!forum/go-kit)
78 - Slack: [gophers.slack.com](https://gophers.slack.com) **#go-kit** ([invite](https://gophersinvite.herokuapp.com/))
8
9 ## Documentation
10
11 ### Examples
12
13 Perhaps the best way to understand Go kit is to follow along as we build an
14 [example service][examples] from first principles. This can serve as a
15 blueprint for your own new service, or demonstrate how to adapt your existing
16 service to use Go kit components.
17
18 [examples]: https://github.com/go-kit/kit/tree/master/examples
19
20 ### Endpoint
21
22 Go kit primarily deals in the RPC messaging pattern. We use an abstraction
23 called an **[endpoint][]** to model individual RPCs. An endpoint can be
24 implemented by a server, and called by a client. It's the fundamental building
25 block of many Go kit components.
26
27 [endpoint]: https://github.com/go-kit/kit/tree/master/endpoint/endpoint.go
28
29 #### Circuit breaker
30
31 The [circuitbreaker package][circuitbreaker] provides endpoint adapters to
32 several popular circuit breaker libraries. Circuit breakers prevent thundering
33 herds, and improve resiliency against intermittent errors. Every client-side
34 endpoint should be wrapped in a circuit breaker.
35
36 [circuitbreaker]: https://github.com/go-kit/kit/tree/master/circuitbreaker
37
38 #### Rate limiter
39
40 The [ratelimit package][ratelimit] provides endpoint adapters to rate limiting
41 packages. Rate limiters are equally applicable to both server- and client-side
42 endpoints. Use rate limiters to enforce upper thresholds on incoming or
43 outgoing request throughput.
44
45 [ratelimit]: https://github.com/go-kit/kit/tree/master/ratelimit
46
47 ### Transport
48
49 The [transport package][transport] provides helpers to bind endpoints to
50 specific serialization mechanisms. At the moment, Go kit just provides helpers
51 for simple JSON over HTTP. If your organization uses a fully-featured
52 transport, bindings are typically provided by the Go library for the
53 transport, and there's not much for Go kit to do. In those cases, see the
54 examples to understand how to write adapters for your endpoints. For now, see
55 the [addsvc][addsvc] to understand how transport bindings work. We have
56 specific examples for Thrift, gRPC, net/rpc, and JSON over HTTP. JSON/RPC and
57 Swagger support is planned.
58
59 [transport]: https://github.com/go-kit/kit/tree/master/transport
60 [addsvc]: https://github.com/go-kit/kit/tree/master/examples/addsvc
61
62 ### Logging
63
64 Services produce logs to be consumed later, either by humans or machines.
65 Humans might be interested in debugging errors, or tracing specific requests.
66 Machines might be interested in counting interesting events, or aggregating
67 information for offline processing. In both cases, it's important that the log
68 messages be structured and actionable. Go kit's [log package][log] is designed
69 to encourage both of these best practices.
70
71 [log]: https://github.com/go-kit/kit/tree/master/log
72
73 ### Metrics (Instrumentation)
74
75 Services can't be considered production-ready until they're thoroughly
76 instrumented with metrics that track counts, latency, health, and other
77 periodic or per-request information. Go kit's [metrics package][metrics]
78 provides a robust common set of interfaces for instrumenting your service.
79 Bindings exist for common backends, from [expvar][] to [statsd][] to
80 [Prometheus][].
81
82 [metrics]: https://github.com/go-kit/kit/tree/master/metrics
83 [expvar]: https://golang.org/pkg/expvar/
84 [statsd]: https://github.com/etsy/statsd
85 [Prometheus]: http://prometheus.io
86
87 ### Request tracing
88
89 As your infrastructure grows, it becomes important to be able to trace a
90 request, as it travels through multiple services and back to the user. Go
91 kit's [tracing package][tracing] provides enhancements for your endpoints and
92 transport bindings to capture information about requests and emit them to
93 request tracing systems. (Currently, [Zipkin][] is supported; [Appdash][]
94 support is planned.)
95
96 [tracing]: https://github.com/go-kit/kit/tree/master/tracing
97 [Zipkin]: https://github.com/openzipkin/zipkin
98 [Appdash]: https://github.com/sourcegraph/appdash
99
100 ### Service discovery and load balancing
101
102 If your service calls another service, it needs to know how to find it, and
103 should intelligently spread its load among those discovered instances. Go
104 kit's [loadbalancer package][loadbalancer] provides client-side endpoint
105 middleware to solve that problem, whether your organization uses static hosts
106 or IPs, [DNS SRV records][dnssrv], Consul, etcd, or Zookeeper. And if you use
107 a custom system, it's very easy to write your own [Publisher][] and use Go
108 kit's load balancing strategies. (Currently, static hosts, DNS SRV, etcd, Consul
109 and ZooKeeper are supported)
110
111 [loadbalancer]: https://github.com/go-kit/kit/tree/master/loadbalancer
112 [dnssrv]: https://github.com/go-kit/kit/tree/master/loadbalancer/dnssrv
113 [Publisher]: https://github.com/go-kit/kit/tree/master/loadbalancer/publisher.go
1149
11510 ## Motivation
11611
12217
12318 To reach its next level of success, Go needs more than simple primitives and
12419 idioms. It needs a comprehensive toolkit, for coherent distributed programming
125 in the large. Go kit is a set of packages and best practices, leveraging years
126 of production experience, and providing a comprehensive, robust, and trustable
127 platform for organizations of any size.
128
129 In short, Go kit makes Go a viable choice for business-domain microservices.
20 in the large. Go kit is a set of packages and best practices, which provide a
21 comprehensive, robust, and trustable way of building microservices for
22 organizations of any size.
13023
13124 For more details, see
25 [the website](https://gokit.io),
13226 [the motivating blog post](http://peter.bourgon.org/go-kit/) and
13327 [the video of the talk](https://www.youtube.com/watch?v=iFR_7AKkJFU).
13428 See also the
14943
15044 ## Contributing
15145
152 Please see [CONTRIBUTING.md][]. Thank you, [contributors][]!
153
154 [CONTRIBUTING.md]: /CONTRIBUTING.md
155 [contributors]: https://github.com/go-kit/kit/graphs/contributors
46 Please see [CONTRIBUTING.md](/CONTRIBUTING.md).
47 Thank you, [contributors](https://github.com/go-kit/kit/graphs/contributors)!
15648
15749 ## Dependency management
15850
15951 Go kit is a library, designed to be imported into a binary package.
160 Vendoring is currently the best way for binary package authors to ensure reliable, reproducible builds.
161 Therefore, we strongly recommend our users use vendoring for all of their dependencies, including Go kit.
162 To avoid compatibility and availability issues, Go kit doesn't vendor its own dependencies, and doesn't recommend use of third-party import proxies.
52 Vendoring is currently the best way for binary package authors
53 to ensure reliable, reproducible builds.
54 Therefore, we strongly recommend our users use vendoring for all of their dependencies,
55 including Go kit.
56 To avoid compatibility and availability issues,
57 Go kit doesn't vendor its own dependencies,
58 and doesn't recommend use of third-party import proxies.
16359
164 There are several tools which make vendoring easier, including [gb][], [glide][], [gvt][], [govendor][], and [vendetta][].
165 In addition, Go kit uses a variety of continuous integration providers to find and fix compatibility problems as soon as they occur.
166
167 [gb]: http://getgb.io
168 [glide]: https://github.com/Masterminds/glide
169 [gvt]: https://github.com/FiloSottile/gvt
170 [govendor]: https://github.com/kardianos/govendor
171 [vendetta]: https://github.com/dpw/vendetta
60 There are several tools which make vendoring easier, including
61 [gb](http://getgb.io),
62 [glide](https://github.com/Masterminds/glide),
63 [gvt](https://github.com/FiloSottile/gvt),
64 [govendor](https://github.com/kardianos/govendor), and
65 [vendetta](https://github.com/dpw/vendetta).
66 In addition, Go kit uses a variety of continuous integration providers
67 to find and fix compatibility problems as soon as they occur.
17268
17369 ## Related projects
17470
17874
17975 - [gizmo](https://github.com/nytimes/gizmo), a microservice toolkit from The New York Times ★
18076 - [go-micro](https://github.com/myodc/go-micro), a microservices client/server library ★
77 - [h2](https://github.com/hailocab/h2), a microservices framework ★
78 - [gotalk](https://github.com/rsms/gotalk), async peer communication protocol & library
79 - [Kite](https://github.com/koding/kite), a micro-service framework
18180 - [gocircuit](https://github.com/gocircuit/circuit), dynamic cloud orchestration
182 - [gotalk](https://github.com/rsms/gotalk), async peer communication protocol & library
183 - [h2](https://github.com/hailocab/h2), a microservices framework ★
184 - [Kite](https://github.com/koding/kite), a micro-service framework
18581
18682 ### Individual components
18783
205101
206102 ### Web frameworks
207103
104 - [Gorilla](http://www.gorillatoolkit.org)
105 - [Gin](https://gin-gonic.github.io/gin/)
106 - [Negroni](https://github.com/codegangsta/negroni)
107 - [Goji](https://github.com/zenazn/goji)
108 - [Martini](https://github.com/go-martini/martini)
208109 - [Beego](http://beego.me/)
209 - [Gin](https://gin-gonic.github.io/gin/)
210 - [Goji](https://github.com/zenazn/goji)
211 - [Gorilla](http://www.gorillatoolkit.org)
212 - [Martini](https://github.com/go-martini/martini)
213 - [Negroni](https://github.com/codegangsta/negroni)
214110 - [Revel](https://revel.github.io/) (considered harmful)
215111
216112 ## Additional reading
218114 - [Architecting for the Cloud](http://fr.slideshare.net/stonse/architecting-for-the-cloud-using-netflixoss-codemash-workshop-29852233) — Netflix
219115 - [Dapper, a Large-Scale Distributed Systems Tracing Infrastructure](http://research.google.com/pubs/pub36356.html) — Google
220116 - [Your Server as a Function](http://monkey.org/~marius/funsrv.pdf) (PDF) — Twitter
221
0 // Package circuitbreaker implements the circuit breaker pattern.
1 //
2 // Circuit breakers prevent thundering herds, and improve resiliency against
3 // intermittent errors. Every client-side endpoint should be wrapped in a
4 // circuit breaker.
5 package circuitbreaker
0 // Package endpoint defines an abstraction for RPCs.
1 //
2 // Endpoints are a fundamental building block for many Go kit components.
3 // Endpoints are implemented by servers, and called by clients.
4 package endpoint
00 # Examples
11
2 1. [A minimal example](#a-minimal-example)
3 1. [Your business logic](#your-business-logic)
4 1. [Requests and responses](#requests-and-responses)
5 1. [Endpoints](#endpoints)
6 1. [Transports](#transports)
7 1. [stringsvc1](#stringsvc1)
8 1. [Logging and instrumentation](#logging-and-instrumentation)
9 1. [Transport logging](#transport-logging)
10 1. [Application logging](#application-logging)
11 1. [Instrumentation](#instrumentation)
12 1. [stringsvc2](#stringsvc2)
13 1. [Calling other services](#calling-other-services)
14 1. [Client-side endpoints](#client-side-endpoints)
15 1. [Service discovery and load balancing](#service-discovery-and-load-balancing)
16 1. [stringsvc3](#stringsvc3)
17 1. [Advanced topics](#advanced-topics)
18 1. [Creating a client package](#creating-a-client-package)
19 1. [Request tracing](#request-tracing)
20 1. [Threading a context](#threading-a-context)
21 1. [Other examples](#other-examples)
22 1. [addsvc](#addsvc)
23 1. [profilesvc](#profilesvc)
24 1. [apigateway](#apigateway)
25 1. [shipping](#shipping)
26
27 ## A minimal example
28
29 Let's create a minimal Go kit service.
30
31 ### Your business logic
32
33 Your service starts with your business logic.
34 In Go kit, we model a service as an **interface**.
35
36 ```go
37 // StringService provides operations on strings.
38 type StringService interface {
39 Uppercase(string) (string, error)
40 Count(string) int
41 }
42 ```
43
44 That interface will have an implementation.
45
46 ```go
47 type stringService struct{}
48
49 func (stringService) Uppercase(s string) (string, error) {
50 if s == "" {
51 return "", ErrEmpty
52 }
53 return strings.ToUpper(s), nil
54 }
55
56 func (stringService) Count(s string) int {
57 return len(s)
58 }
59
60 // ErrEmpty is returned when input string is empty
61 var ErrEmpty = errors.New("Empty string")
62 ```
63
64 ### Requests and responses
65
66 In Go kit, the primary messaging pattern is RPC.
67 So, every method in our interface will be modeled as a remote procedure call.
68 For each method, we define **request and response** structs,
69 capturing all of the input and output parameters respectively.
70
71 ```go
72 type uppercaseRequest struct {
73 S string `json:"s"`
74 }
75
76 type uppercaseResponse struct {
77 V string `json:"v"`
78 Err string `json:"err,omitempty"` // errors don't JSON-marshal, so we use a string
79 }
80
81 type countRequest struct {
82 S string `json:"s"`
83 }
84
85 type countResponse struct {
86 V int `json:"v"`
87 }
88 ```
89
90 ### Endpoints
91
92 Go kit provides much of its functionality through an abstraction called an **endpoint**.
93
94 ```go
95 type Endpoint func(ctx context.Context, request interface{}) (response interface{}, err error)
96 ```
97
98 An endpoint represents a single RPC.
99 That is, a single method in our service interface.
100 We'll write simple adapters to convert each of our service's methods into an endpoint.
101 Each adapter takes a StringService, and returns an endpoint that corresponds to one of the methods.
102
103 ```go
104 import (
105 "golang.org/x/net/context"
106 "github.com/go-kit/kit/endpoint"
107 )
108
109 func makeUppercaseEndpoint(svc StringService) endpoint.Endpoint {
110 return func(ctx context.Context, request interface{}) (interface{}, error) {
111 req := request.(uppercaseRequest)
112 v, err := svc.Uppercase(req.S)
113 if err != nil {
114 return uppercaseResponse{v, err.Error()}, nil
115 }
116 return uppercaseResponse{v, ""}, nil
117 }
118 }
119
120 func makeCountEndpoint(svc StringService) endpoint.Endpoint {
121 return func(ctx context.Context, request interface{}) (interface{}, error) {
122 req := request.(countRequest)
123 v := svc.Count(req.S)
124 return countResponse{v}, nil
125 }
126 }
127 ```
128
129 ### Transports
130
131 Now we need to expose your service to the outside world, so it can be called.
132 Your organization probably already has opinions about how services should talk to each other.
133 Maybe you use Thrift, or custom JSON over HTTP.
134 Go kit supports many **transports** out of the box.
135 (Adding support for new ones is easy—just [file an issue](https://github.com/go-kit/kit/issues).)
136
137 For this minimal example service, let's use JSON over HTTP.
138 Go kit provides a helper struct, in package transport/http.
139
140 ```go
141 import (
142 "encoding/json"
143 "log"
144 "net/http"
145
146 "golang.org/x/net/context"
147
148 httptransport "github.com/go-kit/kit/transport/http"
149 )
150
151 func main() {
152 ctx := context.Background()
153 svc := stringService{}
154
155 uppercaseHandler := httptransport.NewServer(
156 ctx,
157 makeUppercaseEndpoint(svc),
158 decodeUppercaseRequest,
159 encodeResponse,
160 )
161
162 countHandler := httptransport.NewServer(
163 ctx,
164 makeCountEndpoint(svc),
165 decodeCountRequest,
166 encodeResponse,
167 )
168
169 http.Handle("/uppercase", uppercaseHandler)
170 http.Handle("/count", countHandler)
171 log.Fatal(http.ListenAndServe(":8080", nil))
172 }
173
174 func decodeUppercaseRequest(_ context.Context, r *http.Request) (interface{}, error) {
175 var request uppercaseRequest
176 if err := json.NewDecoder(r.Body).Decode(&request); err != nil {
177 return nil, err
178 }
179 return request, nil
180 }
181
182 func decodeCountRequest(_ context.Context, r *http.Request) (interface{}, error) {
183 var request countRequest
184 if err := json.NewDecoder(r.Body).Decode(&request); err != nil {
185 return nil, err
186 }
187 return request, nil
188 }
189
190 func encodeResponse(_ context.Context, w http.ResponseWriter, response interface{}) error {
191 return json.NewEncoder(w).Encode(response)
192 }
193 ```
194
195 ### stringsvc1
196
197 The complete service so far is [stringsvc1][].
198
199 [stringsvc1]: https://github.com/go-kit/kit/blob/master/examples/stringsvc1
200
201 ```
202 $ go get github.com/go-kit/kit/examples/stringsvc1
203 $ stringsvc1
204 ```
205
206 ```
207 $ curl -XPOST -d'{"s":"hello, world"}' localhost:8080/uppercase
208 {"v":"HELLO, WORLD","err":null}
209 $ curl -XPOST -d'{"s":"hello, world"}' localhost:8080/count
210 {"v":12}
211 ```
212
213 ## Logging and instrumentation
214
215 No service can be considered production-ready without thorough logging and instrumentation.
216
217 ### Transport logging
218
219 Any component that needs to log should treat the logger like a dependency, same as a database connection.
220 So, we construct our logger in our `func main`, and pass it to components that need it.
221 We never use a globally-scoped logger.
222
223 We could pass a logger directly into our stringService implementation, but there's a better way.
224 Let's use a **middleware**, also known as a decorator.
225 A middleware is a function that takes an endpoint and returns an endpoint.
226
227 ```go
228 type Middleware func(Endpoint) Endpoint
229 ```
230
231 In between, it can do anything.
232 Let's create a basic logging middleware.
233
234 ```go
235 func loggingMiddleware(logger log.Logger) Middleware {
236 return func(next endpoint.Endpoint) endpoint.Endpoint {
237 return func(ctx context.Context, request interface{}) (interface{}, error) {
238 logger.Log("msg", "calling endpoint")
239 defer logger.Log("msg", "called endpoint")
240 return next(ctx, request)
241 }
242 }
243 }
244 ```
245
246 And wire it into each of our handlers.
247
248 ```go
249 logger := log.NewLogfmtLogger(os.Stderr)
250
251 svc := stringService{}
252
253 var uppercase endpoint.Endpoint
254 uppercase = makeUppercaseEndpoint(svc)
255 uppercase = loggingMiddleware(log.NewContext(logger).With("method", "uppercase"))(uppercase)
256
257 var count endpoint.Endpoint
258 count = makeCountEndpoint(svc)
259 count = loggingMiddleware(log.NewContext(logger).With("method", "count"))(count)
260
261 uppercaseHandler := httptransport.Server(
262 // ...
263 uppercase,
264 // ...
265 )
266
267 countHandler := httptransport.Server(
268 // ...
269 count,
270 // ...
271 )
272 ```
273
274 It turns out that this technique is useful for a lot more than just logging.
275 Many Go kit components are implemented as endpoint middlewares.
276
277 ### Application logging
278
279 But what if we want to log in our application domain, like the parameters that are passed in?
280 It turns out that we can define a middleware for our service, and get the same nice and composable effects.
281 Since our StringService is defined as an interface, we just need to make a new type
282 which wraps an existing StringService, and performs the extra logging duties.
283
284 ```go
285 type loggingMiddleware struct {
286 logger log.Logger
287 next StringService
288 }
289
290 func (mw loggingMiddleware) Uppercase(s string) (output string, err error) {
291 defer func(begin time.Time) {
292 mw.logger.Log(
293 "method", "uppercase",
294 "input", s,
295 "output", output,
296 "err", err,
297 "took", time.Since(begin),
298 )
299 }(time.Now())
300
301 output, err = mw.next.Uppercase(s)
302 return
303 }
304
305 func (mw loggingMiddleware) Count(s string) (n int) {
306 defer func(begin time.Time) {
307 mw.logger.Log(
308 "method", "count",
309 "input", s,
310 "n", n,
311 "took", time.Since(begin),
312 )
313 }(time.Now())
314
315 n = mw.next.Count(s)
316 return
317 }
318 ```
319
320 And wire it in.
321
322 ```go
323 import (
324 "os"
325
326 "github.com/go-kit/kit/log"
327 httptransport "github.com/go-kit/kit/transport/http"
328 )
329
330 func main() {
331 logger := log.NewLogfmtLogger(os.Stderr)
332
333 var svc StringService
334 svc = stringsvc{}
335 svc = loggingMiddleware{logger, svc}
336
337 // ...
338
339 uppercaseHandler := httptransport.NewServer(
340 // ...
341 makeUppercaseEndpoint(svc),
342 // ...
343 )
344
345 countHandler := httptransport.NewServer(
346 // ...
347 makeCountEndpoint(svc),
348 // ...
349 )
350 }
351 ```
352
353 Use endpoint middlewares for transport-domain concerns, like circuit breaking and rate limiting.
354 Use service middlewares for business-domain concerns, like logging and instrumentation.
355 Speaking of instrumentation...
356
357 ### Instrumentation
358
359 In Go kit, instrumentation means using **package metrics** to record statistics about your service's runtime behavior.
360 Counting the number of jobs processed,
361 recording the duration of requests after they've finished,
362 and tracking the number of in-flight operations would all be considered instrumentation.
363
364 We can use the same middleware pattern that we used for logging.
365
366 ```go
367 type instrumentingMiddleware struct {
368 requestCount metrics.Counter
369 requestLatency metrics.TimeHistogram
370 countResult metrics.Histogram
371 next StringService
372 }
373
374 func (mw instrumentingMiddleware) Uppercase(s string) (output string, err error) {
375 defer func(begin time.Time) {
376 methodField := metrics.Field{Key: "method", Value: "uppercase"}
377 errorField := metrics.Field{Key: "error", Value: fmt.Sprintf("%v", err)}
378 mw.requestCount.With(methodField).With(errorField).Add(1)
379 mw.requestLatency.With(methodField).With(errorField).Observe(time.Since(begin))
380 }(time.Now())
381
382 output, err = mw.next.Uppercase(s)
383 return
384 }
385
386 func (mw instrumentingMiddleware) Count(s string) (n int) {
387 defer func(begin time.Time) {
388 methodField := metrics.Field{Key: "method", Value: "count"}
389 errorField := metrics.Field{Key: "error", Value: fmt.Sprintf("%v", error(nil))}
390 mw.requestCount.With(methodField).With(errorField).Add(1)
391 mw.requestLatency.With(methodField).With(errorField).Observe(time.Since(begin))
392 mw.countResult.Observe(int64(n))
393 }(time.Now())
394
395 n = mw.next.Count(s)
396 return
397 }
398 ```
399
400 And wire it into our service.
401
402 ```go
403 import (
404 stdprometheus "github.com/prometheus/client_golang/prometheus"
405 kitprometheus "github.com/go-kit/kit/metrics/prometheus"
406 "github.com/go-kit/kit/metrics"
407 )
408
409 func main() {
410 logger := log.NewLogfmtLogger(os.Stderr)
411
412 fieldKeys := []string{"method", "error"}
413 requestCount := kitprometheus.NewCounter(stdprometheus.CounterOpts{
414 // ...
415 }, fieldKeys)
416 requestLatency := metrics.NewTimeHistogram(time.Microsecond, kitprometheus.NewSummary(stdprometheus.SummaryOpts{
417 // ...
418 }, fieldKeys))
419 countResult := kitprometheus.NewSummary(stdprometheus.SummaryOpts{
420 // ...
421 }, []string{}))
422
423 var svc StringService
424 svc = stringService{}
425 svc = loggingMiddleware{logger, svc}
426 svc = instrumentingMiddleware{requestCount, requestLatency, countResult, svc}
427
428 // ...
429
430 http.Handle("/metrics", stdprometheus.Handler())
431 }
432 ```
433
434 ### stringsvc2
435
436 The complete service so far is [stringsvc2][].
437
438 [stringsvc2]: https://github.com/go-kit/kit/blob/master/examples/stringsvc2
439
440 ```
441 $ go get github.com/go-kit/kit/examples/stringsvc2
442 $ stringsvc2
443 msg=HTTP addr=:8080
444 ```
445
446 ```
447 $ curl -XPOST -d'{"s":"hello, world"}' localhost:8080/uppercase
448 {"v":"HELLO, WORLD","err":null}
449 $ curl -XPOST -d'{"s":"hello, world"}' localhost:8080/count
450 {"v":12}
451 ```
452
453 ```
454 method=uppercase input="hello, world" output="HELLO, WORLD" err=null took=2.455µs
455 method=count input="hello, world" n=12 took=743ns
456 ```
457
458 ## Calling other services
459
460 It's rare that a service exists in a vacuum.
461 Often, you need to call other services.
462 **This is where Go kit shines**.
463 We provide transport middlewares to solve many of the problems that come up.
464
465 Let's say that we want to have our string service call out to a _different_ string service
466 to satisfy the Uppercase method.
467 In effect, proxying the request to another service.
468 Let's implement the proxying middleware as a ServiceMiddleware, same as a logging or instrumenting middleware.
469
470 ```go
471 // proxymw implements StringService, forwarding Uppercase requests to the
472 // provided endpoint, and serving all other (i.e. Count) requests via the
473 // next StringService.
474 type proxymw struct {
475 ctx context.Context
476 next StringService // Serve most requests via this service...
477 uppercase endpoint.Endpoint // ...except Uppercase, which gets served by this endpoint
478 }
479 ```
480
481 ### Client-side endpoints
482
483 We've got exactly the same endpoint we already know about, but we'll use it to invoke, rather than serve, a request.
484 When used this way, we call it a _client_ endpoint.
485 And to invoke the client endpoint, we just do some simple conversions.
486
487 ```go
488 func (mw proxymw) Uppercase(s string) (string, error) {
489 response, err := mw.uppercase(mw.Context, uppercaseRequest{S: s})
490 if err != nil {
491 return "", err
492 }
493 resp := response.(uppercaseResponse)
494 if resp.Err != "" {
495 return resp.V, errors.New(resp.Err)
496 }
497 return resp.V, nil
498 }
499 ```
500
501 Now, to construct one of these proxying middlewares, we convert a proxy URL string to an endpoint.
502 If we assume JSON over HTTP, we can use a helper in the transport/http package.
503
504 ```go
505 import (
506 httptransport "github.com/go-kit/kit/transport/http"
507 )
508
509 func proxyingMiddleware(proxyURL string, ctx context.Context) ServiceMiddleware {
510 return func(next StringService) StringService {
511 return proxymw{ctx, next, makeUppercaseEndpoint(ctx, proxyURL)}
512 }
513 }
514
515 func makeUppercaseEndpoint(ctx context.Context, proxyURL string) endpoint.Endpoint {
516 return httptransport.NewClient(
517 "GET",
518 mustParseURL(proxyURL),
519 encodeUppercaseRequest,
520 decodeUppercaseResponse,
521 ).Endpoint()
522 }
523 ```
524
525 ### Service discovery and load balancing
526
527 That's fine if we only have a single remote service.
528 But in reality, we'll probably have many service instances available to us.
529 We want to discover them through some service discovery mechanism, and spread our load across all of them.
530 And if any of those instances start to behave badly, we want to deal with that, without affecting our own service's reliability.
531
532 Go kit offers adapters to different service discovery systems, to get up-to-date sets of instances, exposed as individual endpoints.
533 Those adapters are called subscribers.
534
535 ```go
536 type Subscriber interface {
537 Endpoints() ([]endpoint.Endpoint, error)
538 }
539 ```
540
541 Internally, subscribers use a provided factory function to convert each discovered instance string (typically host:port) to a usable endpoint.
542
543 ```go
544 type Factory func(instance string) (endpoint.Endpoint, error)
545 ```
546
547 So far, our factory function, makeUppercaseEndpoint, just calls the URL directly.
548 But it's important to put some safety middleware, like circuit breakers and rate limiters, into your factory, too.
549
550 ```go
551 var e endpoint.Endpoint
552 e = makeUppercaseProxy(ctx, instance)
553 e = circuitbreaker.Gobreaker(gobreaker.NewCircuitBreaker(gobreaker.Settings{}))(e)
554 e = kitratelimit.NewTokenBucketLimiter(jujuratelimit.NewBucketWithRate(float64(maxQPS), int64(maxQPS)))(e)
555 }
556 ```
557
558 Now that we've got a set of endpoints, we need to choose one.
559 Load balancers wrap subscribers, and select one endpoint from many.
560 Go kit provides a couple of basic load balancers, and it's easy to write your own if you want more advanced heuristics.
561
562 ```go
563 type Balancer interface {
564 Endpoint() (endpoint.Endpoint, error)
565 }
566 ```
567
568 Now we have the ability to choose endpoints according to some heuristic.
569 We can use that to provide a single, logical, robust endpoint to consumers.
570 A retry strategy wraps a load balancer, and returns a usable endpoint.
571 The retry strategy will retry failed requests until either the max attempts or timeout has been reached.
572
573 ```go
574 func Retry(max int, timeout time.Duration, lb Balancer) endpoint.Endpoint
575 ```
576
577 Let's wire up our final proxying middleware.
578 For simplicity, we'll assume the user will specify multiple comma-separate instance endpoints with a flag.
579
580 ```go
581 func proxyingMiddleware(instances string, ctx context.Context, logger log.Logger) ServiceMiddleware {
582 // If instances is empty, don't proxy.
583 if instances == "" {
584 logger.Log("proxy_to", "none")
585 return func(next StringService) StringService { return next }
586 }
587
588 // Set some parameters for our client.
589 var (
590 qps = 100 // beyond which we will return an error
591 maxAttempts = 3 // per request, before giving up
592 maxTime = 250 * time.Millisecond // wallclock time, before giving up
593 )
594
595 // Otherwise, construct an endpoint for each instance in the list, and add
596 // it to a fixed set of endpoints. In a real service, rather than doing this
597 // by hand, you'd probably use package sd's support for your service
598 // discovery system.
599 var (
600 instanceList = split(instances)
601 subscriber sd.FixedSubscriber
602 )
603 logger.Log("proxy_to", fmt.Sprint(instanceList))
604 for _, instance := range instanceList {
605 var e endpoint.Endpoint
606 e = makeUppercaseProxy(ctx, instance)
607 e = circuitbreaker.Gobreaker(gobreaker.NewCircuitBreaker(gobreaker.Settings{}))(e)
608 e = kitratelimit.NewTokenBucketLimiter(jujuratelimit.NewBucketWithRate(float64(qps), int64(qps)))(e)
609 subscriber = append(subscriber, e)
610 }
611
612 // Now, build a single, retrying, load-balancing endpoint out of all of
613 // those individual endpoints.
614 balancer := lb.NewRoundRobin(subscriber)
615 retry := lb.Retry(maxAttempts, maxTime, balancer)
616
617 // And finally, return the ServiceMiddleware, implemented by proxymw.
618 return func(next StringService) StringService {
619 return proxymw{ctx, next, retry}
620 }
621 }
622 ```
623
624 ### stringsvc3
625
626 The complete service so far is [stringsvc3][].
627
628 [stringsvc3]: https://github.com/go-kit/kit/blob/master/examples/stringsvc3
629
630 ```
631 $ go get github.com/go-kit/kit/examples/stringsvc3
632 $ stringsvc3 -listen=:8001 &
633 listen=:8001 caller=proxying.go:25 proxy_to=none
634 listen=:8001 caller=main.go:72 msg=HTTP addr=:8001
635 $ stringsvc3 -listen=:8002 &
636 listen=:8002 caller=proxying.go:25 proxy_to=none
637 listen=:8002 caller=main.go:72 msg=HTTP addr=:8002
638 $ stringsvc3 -listen=:8003 &
639 listen=:8003 caller=proxying.go:25 proxy_to=none
640 listen=:8003 caller=main.go:72 msg=HTTP addr=:8003
641 $ stringsvc3 -listen=:8080 -proxy=localhost:8001,localhost:8002,localhost:8003
642 listen=:8080 caller=proxying.go:29 proxy_to="[localhost:8001 localhost:8002 localhost:8003]"
643 listen=:8080 caller=main.go:72 msg=HTTP addr=:8080
644 ```
645
646 ```
647 $ for s in foo bar baz ; do curl -d"{\"s\":\"$s\"}" localhost:8080/uppercase ; done
648 {"v":"FOO","err":null}
649 {"v":"BAR","err":null}
650 {"v":"BAZ","err":null}
651 ```
652
653 ```
654 listen=:8001 caller=logging.go:28 method=uppercase input=foo output=FOO err=null took=5.168µs
655 listen=:8080 caller=logging.go:28 method=uppercase input=foo output=FOO err=null took=4.39012ms
656 listen=:8002 caller=logging.go:28 method=uppercase input=bar output=BAR err=null took=5.445µs
657 listen=:8080 caller=logging.go:28 method=uppercase input=bar output=BAR err=null took=2.04831ms
658 listen=:8003 caller=logging.go:28 method=uppercase input=baz output=BAZ err=null took=3.285µs
659 listen=:8080 caller=logging.go:28 method=uppercase input=baz output=BAZ err=null took=1.388155ms
660 ```
661
662 ## Advanced topics
663
664 ### Threading a context
665
666 The context object is used to carry information across conceptual boundaries in the scope of a single request.
667 In our example, we haven't yet threaded the context through our business logic.
668 But that's almost always a good idea.
669 It allows you to pass request-scoped information between business logic and middlewares,
670 and is necessary for more sophisticated tasks like granular distributed tracing annotations.
671
672 Concretely, this means your business logic interfaces will look like
673
674 ```go
675 type MyService interface {
676 Foo(context.Context, string, int) (string, error)
677 Bar(context.Context, string) error
678 Baz(context.Context) (int, error)
679 }
680 ```
681
682 ### Request tracing
683
684 Once your infrastructure grows beyond a certain size, it becomes important to trace requests through multiple services, so you can identify and troubleshoot hotspots.
685 See [package tracing](https://github.com/go-kit/kit/blob/master/tracing) for more information.
686
687 ### Creating a client package
688
689 It's possible to use Go kit to create a client package to your service, to make consuming your service easier from other Go programs.
690 Effectively, your client package will provide an implementation of your service interface, which invokes a remote service instance using a specific transport.
691 See [package addsvc/client](https://github.com/go-kit/kit/tree/master/examples/addsvc/client)
692 or [package profilesvc/client](https://github.com/go-kit/kit/tree/master/examples/profilesvc/client)
693 for examples.
694
695 ## Other examples
696
697 ### addsvc
698
699 [addsvc](https://github.com/go-kit/kit/blob/master/examples/addsvc) is the original example service.
700 It exposes a set of operations over **all supported transports**.
701 It's fully logged, instrumented, and uses Zipkin request tracing.
702 It also demonstrates how to create and use client packages.
703 It's a good example of a fully-featured Go kit service.
704
705 ### profilesvc
706
707 [profilesvc](https://github.com/go-kit/kit/blob/master/examples/profilesvc)
708 demonstrates how to use Go kit to build a REST-ish microservice.
709
710 ### apigateway
711
712 [apigateway](https://github.com/go-kit/kit/blob/master/examples/apigateway/main.go)
713 demonstrates how to implement the API gateway pattern,
714 backed by a Consul service discovery system.
715
716 ### shipping
717
718 [shipping](https://github.com/go-kit/kit/tree/master/examples/shipping)
719 is a complete, "real-world" application composed of multiple microservices,
720 based on Domain Driven Design principles.
2 For more information about these examples,
3 including a walkthrough of the stringsvc example,
4 see [gokit.io/examples](https://gokit.io/examples).
0 // Package addsvc implements the business and transport logic for an example
1 // service that can sum integers and concatenate strings.
2 //
3 // A client library is available in the client subdirectory. A server binary is
4 // available in cmd/addsrv. An example client binary is available in cmd/addcli.
0 // Package addsvc is an example microservice, useful for education. It can sum
1 // integers and concatenate strings. A client library is available in the client
2 // subdirectory. A server binary is available in cmd/addsrv. An example client
3 // binary is available in cmd/addcli.
54 package addsvc
0 // Package log provides a structured logger.
1 //
2 // Services produce logs to be consumed later, either by humans or machines.
3 // Humans might be interested in debugging errors, or tracing specific requests.
4 // Machines might be interested in counting interesting events, or aggregating
5 // information for offline processing. In both cases, it's important that the
6 // log messages be structured and actionable. Package log is designed to
7 // encourage both of these best practices.
8 package log
0 // Package metrics provides a framework for application instrumentation. All
1 // metrics are safe for concurrent use. Considerable design influence has been
2 // taken from https://github.com/codahale/metrics and https://prometheus.io.
3 package metrics
0 // Package metrics provides an extensible framework to instrument your
1 // application. All metrics are safe for concurrent use. Considerable design
2 // influence has been taken from https://github.com/codahale/metrics and
3 // https://prometheus.io.
40 package metrics
51
62 // Counter is a monotonically-increasing, unsigned, 64-bit integer used to
0 // Package sd provides utilities related to service discovery. That includes
1 // subscribing to service discovery systems in order to reach remote instances,
2 // and publishing to service discovery systems to make an instance available.
3 // Implementations are provided for most common systems.
0 // Package sd provides utilities related to service discovery. That includes the
1 // client-side loadbalancer pattern, where a microservice subscribes to a
2 // service discovery system in order to reach remote instances; as well as the
3 // registrator pattern, where a microservice registers itself in a service
4 // discovery system. Implementations are provided for most common systems.
45 package sd
0 // Package tracing provides helpers and bindings for distributed tracing.
1 //
2 // As your infrastructure grows, it becomes important to be able to trace a
3 // request, as it travels through multiple services and back to the user.
4 // Package tracing provides endpoints and transport helpers and middlewares to
5 // capture and emit request-scoped information. We use the excellent OpenTracing
6 // project to bind to concrete tracing systems.
7 package tracing
0 // Package transport contains bindings to concrete transports.
1 package transport