Go Files

index<--

Go Files

Working with files is an important features implemented in Go using packages. Go has a full set of features to list files and folders, open files, read and write content.

Content Topics:

List files
Walk in directory
Read folder content
Read files
Write in files
Use temprary files

Listing files

In programming, often you have multiple sollutions to do the same thing. I have identified 3 methods to list files in a given folder. Let's see some examples:

ioutil.ReadDir
filepath.Walk
os.File.Readdir

ioutil.ReadDir

ioutil.ReadDir comes from the ioutil package in the Go standard library, you can check the documentation from the official Go Doc website

Signature:


func ReadDir(dirname string) ([]os.FileInfo, error)

Example:

This example will list files and folders found in ./test. If the folder is not found you will get a fatal error using "log" package.

// list files
package main

import (
    "fmt"
    "io/ioutil"
    "log"
)

func main() {
    files, err := ioutil.ReadDir("./test")
    if err != nil {
        log.Fatal(err)
    }

    for _, file := range files {
        fmt.Println(file.Name(), file.IsDir())
    }
}

Try it: list-files

filepath.Walk

filepath.Walk can list files in a directory structure, it also allows us to recursively discover directories and files. Let's study this function:

Signature:


func Walk(root string, walkFn WalkFunc) error

Walk walks the file tree rooted at root, calling walkFn for each file or directory in the tree, including root. All errors that arise visiting files and directories are filtered by walkFn. The files are walked in lexical order, which makes the output deterministic but means that for very large directories Walk can be inefficient. Walk does not follow symbolic links.

WalkFunc is a callback function signature you must create to call Walk() with it:


type WalkFunc func(path string, info os.FileInfo, err error) error

Example

//walk function
package main

import (
    "fmt"
    "os"
    "path/filepath"
)

func main() {
    err := filepath.Walk("./test", 
           func(path string, info os.FileInfo, 
                err error) error {
        if err != nil {
            fmt.Println(err)
            return err
        }
        fmt.Printf("dir: %v: name: %s\n", info.IsDir(), path)
        return nil
    })
    if err != nil {
        fmt.Println(err)
    }
}

Output:

>make run
./main
dir: true: name: ./test
dir: true: name: test/subfolder
dir: false: name: test/subfolder/file21
dir: false: name: test/subfolder/file22
dir: false: name: test/test1
>

Try it: walk-function

os.File.Readdir

os.File object, can be used on folders. With a File object can use Readdir() method. This will return a array slice with all the files in sthe directory associated with the respective file.

Signature:


func (f *File) Readdir(n int) ([]FileInfo, error)

Readdir reads the contents of the directory associated with file and returns a slice of up to n FileInfo values, as would be returned by Lstat, in directory order. Subsequent calls on the same file will yield further FileInfos.

You can use parammeter n: If n > 0, Readdir returns at most n FileInfo structures. In this case, if Readdir returns an empty slice, it will return a non-nil error explaining why. At the end of a directory, the error is io.EOF. If n <= 0, Readdir returns all the FileInfo from the directory in a single slice.

Example:

//readir.go
package main

import (
    "fmt"
    "os"
)

// list files in a folder
func main() {
    //open a folder
    f, err := os.Open("./test")
    if err != nil {
        fmt.Println(err)
        return
    }
    files, err := f.Readdir(0)
    if err != nil {
        fmt.Println(err)
        return
    }
    
    for _, v := range files {
        fmt.Println(v.Name(), v.IsDir())
    }
}

Try it: file-readir

Read / Write

Reading and writing files are basic tasks needed for many Go programs. First we'll look at some examples of reading files.

Example

//reading file
package main

import (
    "bufio"
    "fmt"
    "io"
    "os"
)

// This helper will streamline our error checks below.
func check(e error) {
    if e != nil {
        panic(e)
    }
}

func main() {

    f, err := os.Open("files/dat.txt")
    check(err)

    // Postpone closing 
    defer f.Close()
  
    // Read some bytes
    b1 := make([]byte, 12)
    n1, err := f.Read(b1)
    check(err)
    fmt.Printf("%d bytes: %s\n", n1, string(b1[:n1]))

    // You can jump using "Seek" 
    o2, err := f.Seek(6, 0)
    check(err)
    b2 := make([]byte, 2)
    n2, err := f.Read(b2)
    check(err)
    fmt.Printf("%d bytes @ %d: ", n2, o2)
    fmt.Printf("%v\n", string(b2[:n2]))

    // Read at least 
    o3, err := f.Seek(3, 0)
    check(err)
    b3 := make([]byte, 10)
    n3, err := io.ReadAtLeast(f, b3, 10)
    check(err)
    fmt.Printf("%d bytes @ %d: %s\n", n3, o3, string(b3))

    // Seek(0, 0) accomplishes rewind
    _, err = f.Seek(0, 0)
    check(err)

    // buffered reader
    r4 := bufio.NewReader(f)
    b4, err := r4.Peek(12)
    check(err)
    fmt.Printf("12 bytes: %s\n", string(b4))
}

Try it: read_file

Example

Writing files in Go follows similar patterns.


package main

import (
    "bufio"
    "fmt"
    "os"
)

func check(e error) {
    if e != nil {
        panic(e)
    }
}

func main() {

    // To start, here's how to dump a string (or just bytes) into a file.
    d1 := []byte("hello\ngo\n")
    err := os.WriteFile("tmp/dat1", d1, 0644)
    check(err)

    // For more granular writes, open a file for writing.
    f, err := os.Create("tmp/dat2")
    check(err)

    // It's idiomatic to defer a Close immediately
    // after opening a file.
    defer f.Close()

    // You can Write byte slices as you'd expect.
    d2 := []byte{115, 111, 109, 101, 10}
    n2, err := f.Write(d2)
    check(err)
    fmt.Printf("wrote %d bytes\n", n2)

    // A WriteString is also available.
    n3, err := f.WriteString("writes\n")
    check(err)
    fmt.Printf("wrote %d bytes\n", n3)

    // Issue a Sync to flush writes to stable storage.
    f.Sync()

    // bufio provides buffered writers in addition
    // to the buffered readers we saw earlier.
    w := bufio.NewWriter(f)
    n4, err := w.WriteString("buffered\n")
    check(err)
    fmt.Printf("wrote %d bytes\n", n4)

    // Use Flush to ensure all buffered operations have
    // been applied to the underlying writer.
    w.Flush()

}

Try it: file-writer

Temporary Files

You can use temporary files for intermediate data processing jobs. You can use folder /temp or /tmp but you never know if this folder exist. Sometimes system temporary folder is redirected. Therefore is better to use Go API special design for this purpose.

Example:

// temp_files.go
package main

import (
    "fmt"
    "os"
    "path/filepath"
)

func check(e error) {
    if e != nil {
        panic(e)
    }
}

func main() {

    f, err := os.CreateTemp("", "sample")
    check(err)
    fmt.Println("Temp file name:", f.Name())
    defer os.Remove(f.Name())

    // We can write some data to the file.
    _, err = f.Write([]byte{1, 2, 3, 4})
    check(err)


    // prefer to create a temporary *directory*.
    dname, err := os.MkdirTemp("", "sampledir")
    check(err)
    fmt.Println("Temp dir name:", dname)
    defer os.RemoveAll(dname)

    // Now we can synthesize temporary file names by
    // prefixing them with our temporary directory.
    fname := filepath.Join(dname, "file1")
    err = os.WriteFile(fname, []byte{1, 2}, 0666)
    check(err)
}

Try it: file-writer

References

For advanced programming, you need these references. Learn all the fundtions. If you know a function exists you can check it's signature and examples.

Read next: Concurrency