How can I efficiently upload a large file using Go?

Question

Is there a way to download a large file using Go, which will store the contents directly in the file, and not store it all in memory before writing it to the file? Since the file is so large as to save it all in memory, before writing it to a file, it will use all the memory.

Answer 1

I assume you mean downloading via http (validation errors omitted for brevity):

import ("net/http"; "io"; "os") ... out, err := os.Create("output.txt") defer out.Close() ... resp, err := http.Get("http://example.com/") defer resp.Body.Close() ... n, err := io.Copy(out, resp.Body) 

The body of http.Response is a reader, so you can use any functions that take a reader, for example, read a piece at a time, and not all at once. In this particular case, io.Copy() does the gruntwork for you.

Answer 2

A more descriptive version of Steve M.'s answer

 import ( "os" "net/http" "io" ) func downloadFile(filepath string, url string) (err error) { // Create the file out, err := os.Create(filepath) if err != nil { return err } defer out.Close() // Get the data resp, err := http.Get(url) if err != nil { return err } defer resp.Body.Close() // Writer the body to file _, err = io.Copy(out, resp.Body) if err != nil { return err } return nil } 

Answer 3

The answer chosen above using io.Copy is exactly what you need, but if you are interested in additional features such as resuming broken downloads, automatic file naming, checksum checking or monitoring of multiple downloads, grab checking.

Answer 4

• Here is an example. https://github.com/thbar/golang-playground/blob/master/download-files.go

• I also give you some codes that may help you.

the code:

 func HTTPDownload(uri string) ([]byte, error) { fmt.Printf("HTTPDownload From: %s.\n", uri) res, err := http.Get(uri) if err != nil { log.Fatal(err) } defer res.Body.Close() d, err := ioutil.ReadAll(res.Body) if err != nil { log.Fatal(err) } fmt.Printf("ReadFile: Size of download: %d\n", len(d)) return d, err } func WriteFile(dst string, d []byte) error { fmt.Printf("WriteFile: Size of download: %d\n", len(d)) err := ioutil.WriteFile(dst, d, 0444) if err != nil { log.Fatal(err) } return err } func DownloadToFile(uri string, dst string) { fmt.Printf("DownloadToFile From: %s.\n", uri) if d, err := HTTPDownload(uri); err == nil { fmt.Printf("downloaded %s.\n", uri) if WriteFile(dst, d) == nil { fmt.Printf("saved %s as %s\n", uri, dst) } } }