How to process file uploads in Go

Processing user uploaded files is a common task in web development and it’s quite likely that you’ll need to develop a service that handles this task from time to time. This article will guide you through the process of handling file uploads on a Go web server and discuss common requirements such as multiple file uploads, progress reporting, and restricting file sizes.

In this tutorial, we will take a look at file uploads in Go and cover common requirements such as setting size limits, file type restrictions, and progress reporting. You can find the full source code for this tutorial on GitHub.

Getting started

Clone this repository to your computer and cd into the created directory. You’ll see a main.go file which contains the following code:

main.go
package main

import (
	"log"
	"net/http"
)

func indexHandler(w http.ResponseWriter, r *http.Request) {
	w.Header().Add("Content-Type", "text/html")
	http.ServeFile(w, r, "index.html")
}

func uploadHandler(w http.ResponseWriter, r *http.Request) {
	if r.Method != "POST" {
		http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
		return
	}
}

func main() {
	mux := http.NewServeMux()
	mux.HandleFunc("/", indexHandler)
	mux.HandleFunc("/upload", uploadHandler)

	if err := http.ListenAndServe(":4500", mux); err != nil {
		log.Fatal(err)
	}
}

The code here is used to start a server on port 4500 and render the index.html file on the root route. In the index.html file, we have a form containing a file input which posts to an /upload route on the server.

index.html
<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <meta http-equiv="X-UA-Compatible" content="ie=edge" />
    <title>File upload demo</title>
  </head>
  <body>
    <form
      id="form"
      enctype="multipart/form-data"
      action="/upload"
      method="POST"
    >
      <input class="input file-input" type="file" name="file" multiple />
      <button class="button" type="submit">Submit</button>
    </form>
  </body>
</html>

Let’s go ahead and write the code we need to process file uploads from the browser.

Set the maximum file size

It’s necessary to restrict the maximum size of file uploads to avoid a situation where clients accidentally or maliciously upload gigantic files and end up wasting server resources. In this section, we’ll set a maximum upload limit of One Megabyte and show an error if the uploaded file is greater than the limit.

A common approach is to check the Content-Length request header and compare to the maximum file size allowed to see if it’s exceeded or not.

if r.ContentLength > MAX_UPLOAD_SIZE {
	http.Error(w, "The uploaded image is too big. Please use an image less than 1MB in size", http.StatusBadRequest)
	return
}

I don’t recommended using this method because the Content-Length header can be modified on the client to be any value regardless of the actual file size. It’s better to rely on the http.MaxBytesReader method demonstrated below. Update your main.go file with the highlighted portion of the following snippet:

main.go
const MAX_UPLOAD_SIZE = 1024 * 1024 // 1MB

func uploadHandler(w http.ResponseWriter, r *http.Request) {
	if r.Method != "POST" {
		http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
		return
	}

	r.Body = http.MaxBytesReader(w, r.Body, MAX_UPLOAD_SIZE)
	if err := r.ParseMultipartForm(MAX_UPLOAD_SIZE); err != nil {
		http.Error(w, "The uploaded file is too big. Please choose an file that's less than 1MB in size", http.StatusBadRequest)
		return
	}
}

The http.MaxBytesReader() method is used to limit the size of incoming request bodies. For single file uploads, limiting the size of the request body provides a good approximation of limiting the file size. The ParseMultipartForm() method subsequently parses the request body as multipart/form-data up to the max memory argument. If the uploaded file is larger than the argument to ParseMultipartForm(), an error will occur.

Save the uploaded file

Next, let’s retrieve and save the uploaded file to the filesystem. Add the highlighted portion of code snippet below to the end of the uploadHandler() function:

main.go
func uploadHandler(w http.ResponseWriter, r *http.Request) {
	// truncated for brevity

	// The argument to FormFile must match the name attribute
	// of the file input on the frontend
	file, fileHeader, err := r.FormFile("file")
	if err != nil {
		http.Error(w, err.Error(), http.StatusBadRequest)
		return
	}

	defer file.Close()

	// Create the uploads folder if it doesn't
	// already exist
	err = os.MkdirAll("./uploads", os.ModePerm)
	if err != nil {
		http.Error(w, err.Error(), http.StatusInternalServerError)
		return
	}

	// Create a new file in the uploads directory
	dst, err := os.Create(fmt.Sprintf("./uploads/%d%s", time.Now().UnixNano(), filepath.Ext(fileHeader.Filename)))
	if err != nil {
		http.Error(w, err.Error(), http.StatusInternalServerError)
		return
	}

	defer dst.Close()

	// Copy the uploaded file to the filesystem
	// at the specified destination
	_, err = io.Copy(dst, file)
	if err != nil {
		http.Error(w, err.Error(), http.StatusInternalServerError)
		return
	}

	fmt.Fprintf(w, "Upload successful")
}

Restrict the type of the uploaded file

Let’s say we want to limit the type of uploaded files to just images, and specifically only JPEG and PNG images. We need to detect the MIME type of the uploaded file and then compare it to the allowed MIME types to determine if the server should continue processing the upload.

You can use the accept attribute in the file input to define the file types that should be accepted, but you still need to double check on the server to ensure that the input has not been tampered with. Add the highlighted portion of the snippet below to the FileHandlerUpload function:

main.go
func uploadHandler(w http.ResponseWriter, r *http.Request) {
	// truncated for brevity

	file, fileHeader, err := r.FormFile("file")
	if err != nil {
		http.Error(w, err.Error(), http.StatusBadRequest)
		return
	}

	defer file.Close()

	buff := make([]byte, 512)
	_, err = file.Read(buff)
	if err != nil {
		http.Error(w, err.Error(), http.StatusInternalServerError)
		return
	}

	filetype := http.DetectContentType(buff)
	if filetype != "image/jpeg" && filetype != "image/png" { {
		http.Error(w, "The provided file format is not allowed. Please upload a JPEG or PNG image", http.StatusBadRequest)
		return
	}

	_, err := file.Seek(0, io.SeekStart)
	if err != nil {
		http.Error(w, err.Error(), http.StatusInternalServerError)
		return
	}

	// truncated for brevity
}

The DetectContentType() method is provided by the http package for the purpose of detecting the content type of the given data. It considers (at most) the first 512 bytes of data to determine the MIME type. This is why we read the first 512 bytes of the file to an empty buffer before passing it to the DetectContentType() method. If the resulting filetype is neither a JPEG or PNG, an error is returned.

When we read the first 512 bytes of the uploaded file in order to determine the content type, the underlying file stream pointer moves forward by 512 bytes. When io.Copy() is called later, it continues reading from that position resulting in a corrupted image file. The file.Seek() method is used to return the pointer back to the start of the file so that io.Copy() starts from the beginning.

Handle multiple files

If you want to handle the case where multiple files are being sent from the client at once, you can manually parse and iterate over each file instead of using FormFile(). After opening the file, the rest of the code is the same as for single file uploads.

main.go
func uploadHandler(w http.ResponseWriter, r *http.Request) {
	if r.Method != "POST" {
		http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
		return
	}

	// 32 MB is the default used by FormFile()
	if err := r.ParseMultipartForm(32 << 20); err != nil {
		http.Error(w, err.Error(), http.StatusInternalServerError)
		return
	}

	// Get a reference to the fileHeaders.
	// They are accessible only after ParseMultipartForm is called
	files := r.MultipartForm.File["file"]

	for _, fileHeader := range files {
		// Restrict the size of each uploaded file to 1MB.
		// To prevent the aggregate size from exceeding
		// a specified value, use the http.MaxBytesReader() method
		// before calling ParseMultipartForm()
		if fileHeader.Size > MAX_UPLOAD_SIZE {
			http.Error(w, fmt.Sprintf("The uploaded image is too big: %s. Please use an image less than 1MB in size", fileHeader.Filename), http.StatusBadRequest)
			return
		}

		// Open the file
		file, err := fileHeader.Open()
		if err != nil {
			http.Error(w, err.Error(), http.StatusInternalServerError)
			return
		}

		defer file.Close()

		buff := make([]byte, 512)
		_, err = file.Read(buff)
		if err != nil {
			http.Error(w, err.Error(), http.StatusInternalServerError)
			return
		}

		filetype := http.DetectContentType(buff)
		if filetype != "image/jpeg" && filetype != "image/png" {
			http.Error(w, "The provided file format is not allowed. Please upload a JPEG or PNG image", http.StatusBadRequest)
			return
		}

		_, err = file.Seek(0, io.SeekStart)
		if err != nil {
			http.Error(w, err.Error(), http.StatusInternalServerError)
			return
		}

		err = os.MkdirAll("./uploads", os.ModePerm)
		if err != nil {
			http.Error(w, err.Error(), http.StatusInternalServerError)
			return
		}

		f, err := os.Create(fmt.Sprintf("./uploads/%d%s", time.Now().UnixNano(), filepath.Ext(fileHeader.Filename)))
		if err != nil {
			http.Error(w, err.Error(), http.StatusBadRequest)
			return
		}

		defer f.Close()

		_, err = io.Copy(f, file)
		if err != nil {
			http.Error(w, err.Error(), http.StatusBadRequest)
			return
		}
	}

	fmt.Fprintf(w, "Upload successful")
}

Report the upload progress

Next, let’s add progress reporting of the file upload. We can make use of the io.TeeReader() method to count the number bytes read from an io.Reader (in this case each file). Here’s how:

main.go
// Progress is used to track the progress of a file upload.
// It implements the io.Writer interface so it can be passed
// to an io.TeeReader()
type Progress struct {
	TotalSize  int64
	BytesRead int64
}

// Write is used to satisfy the io.Writer interface.
// Instead of writing somewhere, it simply aggregates
// the total bytes on each read
func (pr *Progress) Write(p []byte) (n int, err error) {
	n, err = len(p), nil
	pr.BytesRead += int64(n)
	pr.Print()
	return
}

// Print displays the current progress of the file upload
// each time Write is called
func (pr *Progress) Print() {
	if pr.BytesRead == pr.TotalSize {
		fmt.Println("DONE!")
		return
	}

	fmt.Printf("File upload in progress: %d\n", pr.BytesRead)
}

func uploadHandler(w http.ResponseWriter, r *http.Request) {
	// truncated for brevity

	for _, fileHeader := range files {
		// [..]

		pr := &Progress{
			TotalSize: fileHeader.Size,
		}

		_, err = io.Copy(f, io.TeeReader(file, pr))
		if err != nil {
			http.Error(w, err.Error(), http.StatusBadRequest)
			return
		}
	}

	fmt.Fprintf(w, "Upload successful")
}
File upload progress reporting

Conclusion

This wraps up our effort to handle file uploads in Go. Don’t forget to grab the full source code for this tutorial on GitHub. If you have any questions or suggestions, feel free to leave a comment below.

Thanks for reading, and happy coding!