How to work with Data Types in Go
This article will take you through the various basic data types in Go, the operations that can be done on these types, and how to convert …
Processing user uploaded files is a common task in web development and it’s quite likely that you’ll need to develop a service that handles this task from time to time. This article will guide you through the process of handling file uploads on a Go web server and discuss common requirements such as multiple file uploads, progress reporting, and restricting file sizes.
In this tutorial, we will take a look at file uploads in Go and cover common requirements such as setting size limits, file type restrictions, and progress reporting. You can find the full source code for this tutorial on GitHub.
Clone this
repository to your
computer and cd
into the created directory. You’ll see a main.go
file which
contains the following code:
package main
import (
"log"
"net/http"
)
func indexHandler(w http.ResponseWriter, r *http.Request) {
w.Header().Add("Content-Type", "text/html")
http.ServeFile(w, r, "index.html")
}
func uploadHandler(w http.ResponseWriter, r *http.Request) {
if r.Method != "POST" {
http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
return
}
}
func main() {
mux := http.NewServeMux()
mux.HandleFunc("/", indexHandler)
mux.HandleFunc("/upload", uploadHandler)
if err := http.ListenAndServe(":4500", mux); err != nil {
log.Fatal(err)
}
}
The code here is used to start a server on port 4500 and render the index.html
file on the root route. In the index.html
file, we have a form containing a
file input which posts to an /upload
route on the server.
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta http-equiv="X-UA-Compatible" content="ie=edge" />
<title>File upload demo</title>
</head>
<body>
<form
id="form"
enctype="multipart/form-data"
action="/upload"
method="POST"
>
<input class="input file-input" type="file" name="file" multiple />
<button class="button" type="submit">Submit</button>
</form>
</body>
</html>
Let’s go ahead and write the code we need to process file uploads from the browser.
It’s necessary to restrict the maximum size of file uploads to avoid a situation where clients accidentally or maliciously upload gigantic files and end up wasting server resources. In this section, we’ll set a maximum upload limit of One Megabyte and show an error if the uploaded file is greater than the limit.
A common approach is to check the Content-Length
request header and compare to
the maximum file size allowed to see if it’s exceeded or not.
if r.ContentLength > MAX_UPLOAD_SIZE {
http.Error(w, "The uploaded image is too big. Please use an image less than 1MB in size", http.StatusBadRequest)
return
}
I don’t recommended using this method because the Content-Length
header can be
modified on the client to be any value regardless of the actual file size. It’s
better to rely on the
http.MaxBytesReader method
demonstrated below. Update your main.go
file with the highlighted portion of
the following snippet:
const MAX_UPLOAD_SIZE = 1024 * 1024 // 1MB
func uploadHandler(w http.ResponseWriter, r *http.Request) {
if r.Method != "POST" {
http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
return
}
r.Body = http.MaxBytesReader(w, r.Body, MAX_UPLOAD_SIZE)
if err := r.ParseMultipartForm(MAX_UPLOAD_SIZE); err != nil {
http.Error(w, "The uploaded file is too big. Please choose an file that's less than 1MB in size", http.StatusBadRequest)
return
}
}
The http.MaxBytesReader()
method is used to limit the size of incoming request
bodies. For single file uploads, limiting the size of the request body provides
a good approximation of limiting the file size. The ParseMultipartForm()
method subsequently parses the request body as multipart/form-data
up to the
max memory argument. If the uploaded file is larger than the argument to
ParseMultipartForm()
, an error will occur.
Next, let’s retrieve and save the uploaded file to the filesystem. Add the
highlighted portion of code snippet below to the end of the
uploadHandler()
function:
func uploadHandler(w http.ResponseWriter, r *http.Request) {
// truncated for brevity
// The argument to FormFile must match the name attribute
// of the file input on the frontend
file, fileHeader, err := r.FormFile("file")
if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
defer file.Close()
// Create the uploads folder if it doesn't
// already exist
err = os.MkdirAll("./uploads", os.ModePerm)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
// Create a new file in the uploads directory
dst, err := os.Create(fmt.Sprintf("./uploads/%d%s", time.Now().UnixNano(), filepath.Ext(fileHeader.Filename)))
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
defer dst.Close()
// Copy the uploaded file to the filesystem
// at the specified destination
_, err = io.Copy(dst, file)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
fmt.Fprintf(w, "Upload successful")
}
Let’s say we want to limit the type of uploaded files to just images, and specifically only JPEG and PNG images. We need to detect the MIME type of the uploaded file and then compare it to the allowed MIME types to determine if the server should continue processing the upload.
You can use the accept
attribute in the file input to define the file types
that should be accepted, but you still need to double check on the server to
ensure that the input has not been tampered with. Add the highlighted portion of
the snippet below to the FileHandlerUpload
function:
func uploadHandler(w http.ResponseWriter, r *http.Request) {
// truncated for brevity
file, fileHeader, err := r.FormFile("file")
if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
defer file.Close()
buff := make([]byte, 512)
_, err = file.Read(buff)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
filetype := http.DetectContentType(buff)
if filetype != "image/jpeg" && filetype != "image/png" { {
http.Error(w, "The provided file format is not allowed. Please upload a JPEG or PNG image", http.StatusBadRequest)
return
}
_, err := file.Seek(0, io.SeekStart)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
// truncated for brevity
}
The DetectContentType()
method is provided by the http
package for the
purpose of detecting the content type of the given data. It considers (at most)
the first 512 bytes of data to determine the MIME type. This is why
we read the first 512 bytes of the file to an empty buffer before passing it to
the DetectContentType()
method. If the resulting filetype
is neither a
JPEG or PNG, an error is returned.
When we read the first 512 bytes of the uploaded file in order to determine the
content type, the underlying file stream pointer moves forward by 512 bytes.
When io.Copy()
is called later, it continues reading from that position
resulting in a corrupted image file. The file.Seek()
method is used to return
the pointer back to the start of the file so that io.Copy()
starts from the
beginning.
If you want to handle the case where multiple files are being sent from the
client at once, you can manually parse and iterate over each file instead of
using FormFile()
. After opening the file, the rest of the code is the same as
for single file uploads.
func uploadHandler(w http.ResponseWriter, r *http.Request) {
if r.Method != "POST" {
http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
return
}
// 32 MB is the default used by FormFile()
if err := r.ParseMultipartForm(32 << 20); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
// Get a reference to the fileHeaders.
// They are accessible only after ParseMultipartForm is called
files := r.MultipartForm.File["file"]
for _, fileHeader := range files {
// Restrict the size of each uploaded file to 1MB.
// To prevent the aggregate size from exceeding
// a specified value, use the http.MaxBytesReader() method
// before calling ParseMultipartForm()
if fileHeader.Size > MAX_UPLOAD_SIZE {
http.Error(w, fmt.Sprintf("The uploaded image is too big: %s. Please use an image less than 1MB in size", fileHeader.Filename), http.StatusBadRequest)
return
}
// Open the file
file, err := fileHeader.Open()
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
defer file.Close()
buff := make([]byte, 512)
_, err = file.Read(buff)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
filetype := http.DetectContentType(buff)
if filetype != "image/jpeg" && filetype != "image/png" {
http.Error(w, "The provided file format is not allowed. Please upload a JPEG or PNG image", http.StatusBadRequest)
return
}
_, err = file.Seek(0, io.SeekStart)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
err = os.MkdirAll("./uploads", os.ModePerm)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
f, err := os.Create(fmt.Sprintf("./uploads/%d%s", time.Now().UnixNano(), filepath.Ext(fileHeader.Filename)))
if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
defer f.Close()
_, err = io.Copy(f, file)
if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
}
fmt.Fprintf(w, "Upload successful")
}
Next, let’s add progress reporting of the file upload. We can make use of the
io.TeeReader()
method to count the number bytes read from an io.Reader
(in
this case each file). Here’s how:
// Progress is used to track the progress of a file upload.
// It implements the io.Writer interface so it can be passed
// to an io.TeeReader()
type Progress struct {
TotalSize int64
BytesRead int64
}
// Write is used to satisfy the io.Writer interface.
// Instead of writing somewhere, it simply aggregates
// the total bytes on each read
func (pr *Progress) Write(p []byte) (n int, err error) {
n, err = len(p), nil
pr.BytesRead += int64(n)
pr.Print()
return
}
// Print displays the current progress of the file upload
// each time Write is called
func (pr *Progress) Print() {
if pr.BytesRead == pr.TotalSize {
fmt.Println("DONE!")
return
}
fmt.Printf("File upload in progress: %d\n", pr.BytesRead)
}
func uploadHandler(w http.ResponseWriter, r *http.Request) {
// truncated for brevity
for _, fileHeader := range files {
// [..]
pr := &Progress{
TotalSize: fileHeader.Size,
}
_, err = io.Copy(f, io.TeeReader(file, pr))
if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
}
fmt.Fprintf(w, "Upload successful")
}
This wraps up our effort to handle file uploads in Go. Don’t forget to grab the full source code for this tutorial on GitHub. If you have any questions or suggestions, feel free to leave a comment below.
Thanks for reading, and happy coding!
Comments
Ground rules
Please keep your comments relevant to the topic, and respectful. I reserve the right to delete any comments that violate this rule. Feel free to request clarification, ask questions or submit feedback.