Blog

Large File Uploads with ColdFusion

February 28, 2024

Uploading large files via HTTP can be a challenge.

It can result in timeouts and frustrated users.

Additionally, large uploads may not be possible if your hosting provider imposes HTTP file upload size restrictions (most do).

The solution is “chunking”.

Chunking involves breaking a single large file upload into multiple smaller files and uploading the smaller files to the server. The chunks are stitched back together to recreate the large file.

We’ll need to create some JavaScript to break the file into chunks and then upload each chunk one by one.

It’s possible to upload more than one chunk simultaneously, but that is beyond the scope of this example.

We’ll start with the HTML form:

<html>
    <head>
        <title>Upload A Large File</title>
    </head>
    <body>
        <form action="path/to/upload/page" id="form">
            <input type="file" name="file" id="file"/>
            <input type="button" value="Upload" id="submit"/>
            <input type="button" value="Cancel" id="cancel"/>
        </form>
    </body>
    <-- JS Here -->
</html>

Now add the JavaScript.

Get references to our HTML buttons:

// Buttons
const submitButton = document.getElementById("submit");
const cancelButton = document.getElementById("cancel");

Setup the tracking variables:

// Array to store the chunks of the file
let chunks = [];

// Timeout to start the upload
let uploadTimeout;

// Boolean to check if a chunk is currently being uploaded
let uploading = false;

// Maximum number of retries for a chunk
const maxRetries = 3;

// Create an AbortController to cancel the fetch request
let controller;

Create some utility functions:

// Function to cancel the upload and abort the fetch requests
const cancelUpload = () => {
    clearUpload();
    if (controller)
        controller.abort();
    enableUpload();
};

// Function to clear/reset the upload varaibles
const clearUpload = () => {
    clearTimeout(uploadTimeout);
    uploading = false;
    chunks = [];
};

// Function to enable the submit button
const enableUpload = () => {
    submitButton.removeAttribute("disabled");
    submitButton.setAttribute("value","Upload");
};

// Function to enable the submit button
const disableUpload = () => {
    submitButton.setAttribute("disabled","disabled");
    submitButton.setAttribute("value","Uploading...");
};

Add the cancel button event listener:

// Event listener for the cancel button
cancelButton.addEventListener("click",(e) => {
    cancelUpload();
});

Add the submit button event listener:

// Event listener for the submit button
submitButton.addEventListener("click",(e) => {
    const fileInput = document.getElementById("file");

    if (fileInput.files.length) {
        disableUpload();

        const file = fileInput.files[0];

        // Break the file into 1MB chunks
        const chunkSize = 1024 * 1024;
        const totalChunks = Math.ceil(file.size / chunkSize);

        // Setup chunk array with starting byte and endbyte that will be sliced from the file and uploaded
        let startByte = 0;
        for (var i = 1; i <= totalChunks; i++) {
            let endByte = Math.min(startByte + chunkSize,file.size);
            // [chunk number, start byte, end byte, uploaded, retry count, error, promise]
            chunks.push([i,startByte, Math.min(startByte + chunkSize,file.size), null, 0, null, null]);
            startByte = endByte;
        }

        // Begin uploading the chunks one after the other
        uploadTimeout = setInterval(() => {
            if (!uploading) {
                uploading = true;  // Prevent the next interval from starting until the current chunk is uploaded

                // Upload the first chunk that hasn't already been uploaded
                for (var i = 0; i < chunks.length; i++) {
                    if (!chunks[i][3] && chunks[i][4] < maxRetries) {
                        // Get the binary chunk (start byte / end byte)
                        const chunk = file.slice(chunks[i][1],chunks[i][2]);

                        const formData = new FormData();

                        formData.append("chunk",chunk);
                        formData.append("chunknumber",chunks[i][0]);
                        formData.append("totalchunks",totalChunks);
                        formData.append("filename",file.name);

                        // Create an AbortController to cancel the fetch request
                        // Need a new one for each fetch request
                        controller = new AbortController();

                        fetch(document.getElementById("form").getAttribute("action"), {
                            method: "POST",
                            body: formData,
                            signal: controller.signal
                        }).then(response => {
                            if (response.status == 413) {
                                cancelUpload();
                                alert("The chunk is too large. Try a smaller chunk size.");
                            } else {
                                response.json().then(data => {
                                    if (data && data.success) {
                                        // Uploaded successfully
                                        chunks[i][3] = true; // Mark the chunk as uploaded
                                        uploading = false; // Allow the next chunk to be uploaded
                                        if (chunks[i][0] == totalChunks) {
                                            // Last chunk uploaded successfully
                                            clearUpload();
                                            enableUpload();
                                            alert("File uploaded successfully");
                                        }
                                    } else {
                                        // Error uplooading
                                        cancelUpload();
                                        alert("Error uploading chunk. Check the browser debug window for details.");
                                        console.error("Unhandled error uploading chunk");
                                    }
                                }).catch(error => {
                                    // Error uploading
                                    cancelUpload();
                                    alert("Error uploading chunk. Check the browser debug window for details.");
                                    console.error("Error:",error);
                                });
                            }
                        }).catch(error => {
                            // Error uploading
                            chunks[i][4]++; // Increment the retry count
                            chunks[i][5] = error; // Store the error
                            uploading = false; // Allow retrying this chunk
                            console.error("Error:",error); // Log the error
                        });
                        break;
                    }
                }
            }
        },1000);            
    } else {
        alert("No file selected.");
    }
});

In the code above we’re storing the start and end bytes of the file chunks in an array that we can loop through. We set our loop to run on an interval in the background so we’re not blocking the browser (we could use Web Workers here, but I’m not very familiar with them). During each interval we check to make sure we’re not uploading a chunk, then we find the next chunk that has not been uploaded. We slice the start and end bytes of the current chunk from the file and upload it.

Now we need to create the ColdFusion server-side code to accept the binary chunks and then stitch them all together into a complete file.

<cfscript>
    if (compareNoCase(cgi.request_method, "post") eq 0) {
        /*
            ColdFusion will take the binary data supplied in the form.chunk variable
            and write it to a temporary file on the server and then provide the path to that
            temp file in the form.chunk variable.
        */
        binaryChunk = fileReadBinary(form.chunk);

        /*
            Make sure to cleanup the temporary file because we are still ultimately going to be uploading a 
            large file to the server.  We don't want the server drive that holds the ColdFusion temp directory
            to fill up
        */
        fileDelete(form.chunk);

        uploadedFile = expandPath("path/to/uploads/#form.filename#");
        if (form.chunknumber eq 1 AND fileExists(uploadedFile))
            fileDelete(uploadedFile);  // Overwrite the file if it already exists

        // Now we can append the binary data to the file we are uploading to the server
        uploadFile = fileOpen(uploadedFile,"append");
    
        // Write the binary data to the file
        fileWrite(uploadFile,binaryChunk);

        // Close the file
        fileClose(uploadFile);
    }
</cfscript>

<cfcontent type="application/json" reset="true"><cfoutput>#serializeJSON({"success":true})#</cfoutput><

On the server-side ColdFusion will accept the binary chunk provided in the form variable and write it to a temporary file in the ColdFusion temp directory. It’s important to be aware of this because we can quickly fill up the servers drive with thousands of 1MG binary files. The code above reads the temporary file then deletes it. The binary data is then appended to the uploaded file.

We return a JSON response with a success flag so the client-side JavaScript will upload the next chunk.

That’s it. Pretty simple.

It’s possible to expand on this example and provide things like progress and status to further enhance the user experience.

Click here to download the source files.

0 Comments

Leave Your Comment

Your email address will not be published. Required fields are marked *


about me

An information technology professional with twenty five years experience in systems administration, computer programming, requirements gathering, customer service, and technical support.