One possible solution is the "encoding/gob" standard package. The gob package creates an encoder/decoder that can encode any struct into an array of bytes and then decode that array back into a struct. There's a great post, here.
As others have pointed out, it's necessary to use a package like this because structs, by their nature, have unknown sizes and cannot be converted into arrays of bytes.
I've included some code and a play.
package main
import (
"bytes"
"encoding/gob"
"fmt"
"log"
)
type P struct {
X, Y, Z int
Name string
}
type Q struct {
X, Y *int32
Name string
}
func main() {
// Initialize the encoder and decoder. Normally enc and dec would be
// bound to network connections and the encoder and decoder would
// run in different processes.
var network bytes.Buffer // Stand-in for a network connection
enc := gob.NewEncoder(&network) // Will write to network.
dec := gob.NewDecoder(&network) // Will read from network.
// Encode (send) the value.
err := enc.Encode(P{3, 4, 5, "Pythagoras"})
if err != nil {
log.Fatal("encode error:", err)
}
// HERE ARE YOUR BYTES!!!!
fmt.Println(network.Bytes())
// Decode (receive) the value.
var q Q
err = dec.Decode(&q)
if err != nil {
log.Fatal("decode error:", err)
}
fmt.Printf("%q: {%d,%d}\n", q.Name, *q.X, *q.Y)
}
Answer from Adam on Stack OverflowOne possible solution is the "encoding/gob" standard package. The gob package creates an encoder/decoder that can encode any struct into an array of bytes and then decode that array back into a struct. There's a great post, here.
As others have pointed out, it's necessary to use a package like this because structs, by their nature, have unknown sizes and cannot be converted into arrays of bytes.
I've included some code and a play.
package main
import (
"bytes"
"encoding/gob"
"fmt"
"log"
)
type P struct {
X, Y, Z int
Name string
}
type Q struct {
X, Y *int32
Name string
}
func main() {
// Initialize the encoder and decoder. Normally enc and dec would be
// bound to network connections and the encoder and decoder would
// run in different processes.
var network bytes.Buffer // Stand-in for a network connection
enc := gob.NewEncoder(&network) // Will write to network.
dec := gob.NewDecoder(&network) // Will read from network.
// Encode (send) the value.
err := enc.Encode(P{3, 4, 5, "Pythagoras"})
if err != nil {
log.Fatal("encode error:", err)
}
// HERE ARE YOUR BYTES!!!!
fmt.Println(network.Bytes())
// Decode (receive) the value.
var q Q
err = dec.Decode(&q)
if err != nil {
log.Fatal("decode error:", err)
}
fmt.Printf("%q: {%d,%d}\n", q.Name, *q.X, *q.Y)
}
I assume you want something like the way C handles this. There is no built in way to do that. You will have to define your own serialization and deserialization to and from bytes for your struct. The binary package will help you encode the fields in your struct to bytes that you can add to the byte array but you will be responsible for specifying the lengths and offsets in the byte array that will hold the fields from your struct.
Your other options are to use one of the encoding packages: http://golang.org/pkg/encoding/ such as gob or json.
EDIT:
Since you want this for making a hash as you say in your comment the easisest thing to do is use []byte(fmt.Sprintf("%v", struct)) like so: http://play.golang.org/p/yY8mSdZ_kf
New to Go. Whats the best way to convert a struct to a byte[] and vice versa? I want to store the struct in cache.
Convert to json string then string to byte[]?
Thanks in advance!!!
In C, for a protocol stack, I'd have a byte array from the network. I'd inspect a couple of bytes to determine what the packet is supposed to be at least, and then cast it to the appropriate struct. Something like this:
bptr = <malloc'd buffer> *protocolStruct = *(ProtocolObject)bptr
In golang, do I use the binary package, the unsafe { }, or what?
Let''s assume I have a struct for an ARP request, something like:
type NativeAddress struct
AddressType uint32
Address [16]byte
}And my ARP request structure is
type ARPRequest struct {
SAP uint16
NSAP uint16
SrcAddress NativeAddres
DstAddress NativeAddress
RequestedAddress uint64
}How does this get encoded with something like Gob and how does it get received on the other end? Wasteful though it is, everything is aligned. Can I just assume it gets encoded linearly ion the wire? In ancient C, I could do a tagged union that worked the same way, but modern C doesn't do that, you really can't count on what two C compilers will generate for structs <-> bytes. Or am I making this harder than it has to be -- should I just encode everything, every packet in protobuf, avro etc? In JVM langues, I have the same problem so I was hoping Golang handled it better. Even with Gob, can two architectures using Golang encode and decode structs reliably and, I still have the problem, after I read the SAP and NSAP bytes, only then do I know what struct it actually is and thus, I can "cast" it, or does Gob take care of that? I can't use something like JSON encoding, this is binary date.
I'd love to just have something that read bytes and said "I just received struct X" or "I have no idea what I just got" if I can't do the C style tricks.
unsafe.Pointer is, well, unsafe, and you don't actually need it here. Use encoding/binary package instead:
// Create a struct and write it.
t := T{A: 0xEEFFEEFF, B: 3.14}
buf := &bytes.Buffer{}
err := binary.Write(buf, binary.BigEndian, t)
if err != nil {
panic(err)
}
fmt.Println(buf.Bytes())
// Read into an empty struct.
t = T{}
err = binary.Read(buf, binary.BigEndian, &t)
if err != nil {
panic(err)
}
fmt.Printf("%x %f", t.A, t.B)
Playground
As you can see, it handles sizes and endianness quite neatly.
I've had the same problem and I solved it by using the "encoding/binary" package. Here's an example:
package main
import (
"bytes"
"fmt"
"encoding/binary"
)
func main() {
p := fmt.Println
b := []byte{43, 1, 0}
myStruct := MyStruct{}
err := binary.Read(bytes.NewBuffer(b[:]), binary.BigEndian, &myStruct)
if err != nil {
panic(err)
}
p(myStruct)
}
type MyStruct struct {
Num uint8
Num2 uint16
}
Here's the running example: https://play.golang.org/p/Q3LjaAWDMh
You have two errors in your code. Firstly, your struct fields should be exported for encoding/binary to see it.
Secondly, [16]string means an array of 16 strings, not a 16-byte string. So your struct should look like this:
type login struct {
Seq uint8
ID uint16
Username [16]byte
Password [16]byte
Unknown1 [16]byte
}
Then it works: https://play.golang.org/p/Nq8fRqgkcp.
You can do that using the unsafe package.
type test struct {
a int
s string
}
v1 := test{
a: 5,
s: "sdaf",
}
fmt.Printf("v1: %#v\n", v1)
b := *(*[unsafe.Sizeof(v1)]byte)(unsafe.Pointer(&v1))
fmt.Printf("bytes: %#v\n", b)
v2 := *(*test)(unsafe.Pointer(&b))
fmt.Printf("v2: %#v\n", v2)
I want to be able to create repeatable byte arrays of structs in Go so I can hash them and then verify that hash at some point.
I am currently following this simple approach to create a byte array from a struct with:
[]byte(fmt.Sprintf("%v", struct))...)This works perfectly until my struct holds an embedded struct with a pointer, for example:
type testEmbeddedPointerStruct struct {
T *testSimpleStruct
}In my tests this creates a different byte array each time, I think it may be because with the pointer the address in memory changes each time?
Is there a way of creating a repeatable byte array digest even if the struct holds a pointer?
I basically want to get the values from the struct and convert them to a byte array
Thanks
You want to use the encode package from go's library. Usually I like the JSON encoding because it's very human readable, but the package supports encoding to/from many formats including binary and gob which is a format designed just for what you are trying to do.
Example from the go documentation to encode to json:
package main
import (
"encoding/json"
"fmt"
"os"
)
func main() {
type ColorGroup struct {
ID int
Name string
Colors []string
}
group := ColorGroup{
ID: 1,
Name: "Reds",
Colors: []string{"Crimson", "Red", "Ruby", "Maroon"},
}
b, err := json.Marshal(group)
if err != nil {
fmt.Println("error:", err)
}
os.Stdout.Write(b)
}
Example from the go documentation to decode from json:
package main
import (
"encoding/json"
"fmt"
)
func main() {
var jsonBlob = []byte(`[
{"Name": "Platypus", "Order": "Monotremata"},
{"Name": "Quoll", "Order": "Dasyuromorphia"}
]`)
type Animal struct {
Name string
Order string
}
var animals []Animal
err := json.Unmarshal(jsonBlob, &animals)
if err != nil {
fmt.Println("error:", err)
}
fmt.Printf("%+v", animals)
}
What about this?
bodyBytes, err := ioutil.ReadAll(response.Body)
You can use this as bytes or simply convert ir to string like this
bodyString := string(bodyBytes)
According to http://golang.org/pkg/encoding/binary/#Read :
Data must be a pointer to a fixed-size value or a slice of fixed-size values.
So you can't use slice []byte in your structure. But you can use fixed size array for it.
Like this:
package main
import (
"bytes"
"encoding/binary"
"fmt"
)
type T struct {
A int16
B int8
C [256]byte
}
func main() {
// Create a struct and write it.
t := T{A: 99, B: 10}
buf := &bytes.Buffer{}
err := binary.Write(buf, binary.BigEndian, t)
if err != nil {
panic(err)
}
fmt.Println(buf)
// Read into an empty struct.
t = T{}
err = binary.Read(buf, binary.BigEndian, &t)
if err != nil {
panic(err)
}
fmt.Printf("%d %d", t.A, t.B)
}
Go playground
I think binpacker will be greate to handle this case:
package main
import (
"bytes"
"fmt"
"github.com/zhuangsirui/binpacker"
)
type T struct {
A uint16
B string
C []byte
}
func main() {
field1 := uint16(1)
field2 := "Hello World"
field3 := []byte("Hello World")
buffer := new(bytes.Buffer)
binpacker.NewPacker(buffer).
PushUint16(field1).
PushUint16(uint16(len(field2))).PushString(field2).
PushUint16(uint16(len(field3))).PushBytes(field3)
t := new(T)
unpacker := binpacker.NewUnpacker(buffer)
unpacker.FetchUint16(&t.A).StringWithUint16Perfix(&t.B).BytesWithUint16Perfix(&t.C)
fmt.Println(t)
}