Web APP/JavaScript Download Large File Solution

With the increasing ability of browsers to process data and the popularity of Web APP, many times we will need to download and save the data processed by the browser: such as the report generation CSV, the front-end big data visualization results, the front-end data calculation results, etc. I also wrote in 《JavaScript generated CSV, and Chinese garbled problem》can be solved by Data URLs. But these two days discussed with my friend about JavaScript download CSV suddenly encountered a file size problem.

1. Problem recurrence

Let's look at the following example. In this example, we continue to expand the contents of the txt field, then put it in the URL and simulate clicking to download it.

var a = document.createElement('a');
var txt = '%E5%A7%93%E5%90%8D,%E5%B9%B4%E9%BE%84%0AMofei,18,\n';
for(var i=0; i<15; i++){
  txt = txt + txt;
}
a.href='data:text/csv;charset=utf-8,\uFEFF'+txt;
a.download="text.csv";
a.click()

At first, when the upper limit of i is relatively small, we can download the file smoothly, but when the maximum value of i is adjusted to a relatively large value, we will find such a problem:

Download failed

After continuous testing and scrutiny, we agreed that the problem may be in the length of the URL. After some research, we found such a library [FileSaver.js] (https://github.com/eligrey/FileSaver.js /blob/master/src/FileSaver.js) The important content of this library is only a 5.5k js file. However, the number of Stars has reached 8500+ when I wrote this blog. It is very powerful. He will directly The size extends from the maximum value of the URL to the size of the blob supported by the browser.

2. Solution

Let's see how it is implemented. Read through the entire source code and find that the main solution for this library is to use the browser's Blob and the URL.createObjectURL() method.

2.1 What is Blob?

In the introduction of MDN's [Blob] (https://developer.mozilla.org/en-US/docs/Web/API/Blob) we found such a text introduction:

A Blob object represents a file-like object of immutable, raw data. Blobs represent data that isn't necessarily in a JavaScript-native format. The File interface is based on Blob, inheriting blob functionality and expanding it to support files on the user's system.

Simply put, it is an object of a class file that stores data of a non-JavaScript base type. More Simply put, it is used to store some data that cannot be described by JS.

Its construction method is also relatively simple to use.

var aBlob = new Blob( array[, options]);

For example, if we want to store a html file, we can write:

var aFileParts = ['<a id="a"><b id="b">hey!</b></a>']; 
var oMyBlob = new Blob(aFileParts, {type : 'text/html'});

The first parameter is the text of the Html wrapped in Array, and the second parameter indicates that the file type is HTML(text/HTML).

Blobs are usually used with the File API. Most of the content read by the File API is converted to a Blob object, so the blob size allowed by the browser is naturally much larger than the Url.

2.2 What is URL.createObjectURL()?

The createObjectURL method is derived from the (window.) URL interface, which is primarily used to create object URLs. Simply put, you can convert an object to a URL. Currently, createObjectURL mainly uses a Convert File object or a Blob object. His grammar is also simple:

objectURL = URL.createObjectURL(object);
2.3 Double Swords

With Blob and OjbectURL, we can combine them to achieve larger file downloads. Look directly at our revised content:

var a = document.createElement('a');
var txt = '%E5%A7%93%E5%90%8D,%E5%B9%B4%E9%BE%84%0AMofei,18,\n';
for(var i=0;i<20;i++){
  txt = txt + txt;
}
var t = new Blob([txt], {type : 'application/csv'});
a.href=URL.createObjectURL(t)
a.download="Mofei的CSV.csv";
a.click();

We wrap our txt with new Blob and then convert the blob to the object address via createObjectURL so that the browser can download large files.

3. More

The author of FileSaver.js mainly realizes the possibility of browser downloading large files through Blob and ObjectURL methods, but in fact, the available size of this is often determined by the browser's restrictions on blobs or user memory limits (For most users, the 500M blob size is enough for the front end to use). The author also mentioned another magic library [StreamSaver.js] (https://github.com/jimmywarting/StreamSaver.js) in the project. This library uses service worker to achieve a larger front-end download space. Interested partners can also find out.

152200
  • logo
    • HI, THERE!I AM MOFEI

      (C) 2010-2024 Code & Design by Mofei

      Powered by Dufing (2010-2020) & Express

      IPC证:沪ICP备2022019571号-1