Go - On building URL strings

Building URL strings in Go may be accomplished in a couple different ways:

url.Parse
string concatenation (using +)
fmt.Sprintf
bytes.Buffer
strings.Builder

You may be asking yourself: which one should I use, then? As always, the answer depends. Let us explore why.

`URL.Parse`

Before we start, a quick refresh on the URL structure:

url

One important fact to be aware is that when building URLs manually, it is easy to forget the percentage-encoding.

The url package implements the URL parsing and query encoding logic, although the api may be easy to misuse, like in the examples below.

Using url.Parse to parse a string:

package main

import (
	"fmt"
	"net/url"
)

func main() {
	u, _ := url.Parse("https://example.org/resource?filter[page]=1")

	// prints https://example.org/resource?filter[page]=1
	fmt.Println(u)
}

url.Parse does not check if the query is properly encoded. However, it is possible to fix it by retrieving the query section and encoding it explicitly:

package main

import (
	"fmt"
	"net/url"
)

func main() {
	u, _ := url.Parse("https://example.org/resource?filter[page]=1")

	// prints https://example.org/resource?filter[page]=1
	fmt.Println(u)

	u.RawQuery = u.Query().Encode()

	// prints https://example.org/resource?filter%5Bpage%5D=1
	fmt.Println(u)
}

Another possible way to encode query string is using url.Values:

package main

import (
	"fmt"
	"net/url"
)

func main() {
	u, _ := url.Parse("https://example.org/foo")

	// prints https://example.org/foo
	fmt.Println(u)

	query := url.Values{}
	query.Set("filter[page]", "1")

	u.RawQuery = query.Encode()

	// prints https://example.org/foo?filter%5Bpage%5D=1
	fmt.Println(u)
}

On query building abstractions, the go-querystring package provides some extra functionality around url.Values, but follows the initial RawQuery usage.

package main

import (
	"fmt"
	"net/url"

	"github.com/google/go-querystring/query"
)

type Options struct {
	Page string `url:"filter[page]"`
}

func main() {
	u, _ := url.Parse("https://example.org/foo")

	// prints https://example.org/foo
	fmt.Println(u)

	q, _ := query.Values(Options{Page: "1"})

	u.RawQuery = q.Encode()

	// prints https://example.org/foo?filter%5Bpage%5D=1
	fmt.Println(u)
}

Something to be aware with url.Parse is that http.NewRequestWithContext signature is:

func NewRequestWithContext(ctx context.Context, method, url string, body io.Reader) (*Request, error)

URL argument type is string, but it will be parsed to url.URL:

func NewRequestWithContext(ctx context.Context, method, url string, body io.Reader) (*Request, error) {
	// omitted code
    // ...

	u, err := urlpkg.Parse(url) //net/url package

    // ...

When using http.NewRequestWithContext, regardless of the method selected to build the URL string, url.Parse is going to be called at least once.

Community Usage

It is always interesting to check how the community tackles this issue to understand if there is a clear preference.

`go-github`

https://github.com/google/go-github/blob/master/github/pulls.go#L165

func (s *PullRequestsService) ListPullRequestsWithCommit(ctx context.Context, owner, repo, sha string, opts *PullRequestListOptions) ([]*PullRequest, *Response, error) {
	u := fmt.Sprintf("repos/%v/%v/commits/%v/pulls", owner, repo, sha)
	u, err := addOptions(u, opts)
	if err != nil {
		return nil, nil, err
	}
    // ...

// https://github.com/google/go-github/blob/master/github/github.go#L242
func addOptions(s string, opts interface{}) (string, error) {
	v := reflect.ValueOf(opts)
	if v.Kind() == reflect.Ptr && v.IsNil() {
		return s, nil
	}

	u, err := url.Parse(s)
	if err != nil {
		return s, err
	}

	qs, err := query.Values(opts)
	if err != nil {
		return s, err
	}

	u.RawQuery = qs.Encode()
	return u.String(), nil
}

In this repo fmt.Sprintf is used in conjunction with go-querystring package.

`consul client`

https://github.com/hashicorp/consul/blob/api/v1.8.1/api/acl.go#L616

func (a *ACL) TokenClone(tokenID string, description string, q *WriteOptions) (*ACLToken, *WriteMeta, error) {
	if tokenID == "" {
		return nil, nil, fmt.Errorf("Must specify a tokenID for Token Cloning")
	}

	r := a.c.newRequest("PUT", "/v1/acl/token/"+tokenID+"/clone")

String concatenation is used in this case.

godo

https://github.com/digitalocean/godo/blob/main/droplets.go#L322

// ListByTag lists all Droplets matched by a Tag.
func (s *DropletsServiceOp) ListByTag(ctx context.Context, tag string, opt *ListOptions) ([]Droplet, *Response, error) {
	path := fmt.Sprintf("%s?tag_name=%s", dropletBasePath, tag)

In this repo fmt.Sprintf is used in conjunction with go-querystring package.

Benchmark

Looking at community usage, anecdotally fmt.Sprintf seems to be the preferred method.

Is there a numeric reason behind it? A simple way to test is benchmark URL building techniques and formats:

All fields are strings: “https://example.com/resource/00000000-0000-0000-0000-000000000000"
At least one field is an int which we need to convert: “https://example.com/resource/314159265"

Running with go test -bench=. -benchtime=10s

BenchmarkBytesString-24                                 91364389               131.1 ns/op           320 B/op          3 allocs/op
BenchmarkBytesStringAndItoa-24                          100000000              100.8 ns/op           128 B/op          3 allocs/op
BenchmarkConcatString-24                                247271304               48.57 ns/op           80 B/op          1 allocs/op
BenchmarkConcatStringAndItoa-24                         150584938               79.36 ns/op           64 B/op          2 allocs/op
BenchmarkSprinfString-24                                56899111               210.9 ns/op           128 B/op          4 allocs/op
BenchmarkSprinfDigit-24                                 54090478               226.2 ns/op            88 B/op          4 allocs/op
BenchmarkSprinfDigitItoa-24                             46432692               256.5 ns/op           112 B/op          5 allocs/op
BenchmarkStringBuilderString-24                         123711208               97.40 ns/op          168 B/op          3 allocs/op
BenchmarkStringBuilderStringAndItoa-24                  123868472               96.78 ns/op           88 B/op          3 allocs/op
BenchmarkURLParseString-24                              14717494               815.6 ns/op           256 B/op          4 allocs/op
BenchmarkURLParseResolveReference-24                     5361052              2236 ns/op            1040 B/op         16 allocs/op
BenchmarkURLQueryEncode-24                               9121402              1312 ns/op             536 B/op         11 allocs/op
BenchmarkURLQueryEncodePackageQueryString-24             6446503              1864 ns/op             984 B/op         17 allocs/op

Looking at benchmark results, string concatenation and strings.Builder seem the fastest and doing fewer allocations.

When not possible to ensure the URL will be correctly encoded, use url.Parse and call Encode explicitly.

While go-querystring usability may be convenient, it infers a performance cost price.

Benchmark code available at https://github.com/jacoelho/url_build_benchmark.