Dealing with large files in Cordova

While it may seem like uploading files in Cordova applications is a trivial task easily completed with cordova-plugin-file-transfer plugin.

But since November 2017 this plugin is deprecated and it is recommended to use XMLHttpRequest to handle this task.

According to Cordova's blog post, we must resolve the file in the local filesystem, read it via FileReader and send it using XMLHttpRequest. But if we try to upload a file that exceeds the size of 30MB (this may vary depending on device) we can get the following console error:

POST https://some/post/url net::ERR_FILE_NOT_FOUND

When we try to execute:

    // Resolve fileLocalUrl to File object
    const resolveFile = (url) => {
      return new Promise((resolve, reject) => {
        window.resolveLocalFileSystemURL(
          url,
          (fileEntry) => {
            fileEntry.file(resolve);
          });
      });
    };

    // Read File's Blob
    const readFile = (file) => {
      return new Promise((resolve, reject) => {
        const reader = new FileReader();
        reader.onloadend = (ev) => {
          if (ev.target.error) {
            return reject(ev.target.error);
          }
          return resolve(new Blob([new Uint8Array(reader.result)]));
        };
        reader.readAsArrayBuffer(file);
      });
    };

    // Get duration of video file
    const getDuration = (blob) => {
      return new Promise((resolve, reject) => {
        const video = document.createElement('video');
        const videoSrc = document.createElement('source');
        video.preload = 'metadata';
        video.appendChild(videoSrc)

        video.onloadedmetadata = videoSrc.onloadedmetadata = () => {
          resolve(Math.round(video.duration));
          if (typeof videoSrc.remove === 'function') {
            videoSrc.remove();
          }
          if (typeof video.remove === 'function') {
            video.remove();
          }
        };
        video.onerror = videoSrc.onerror = reject;

        videoSrc.src = URL.createObjectURL(blob);
        if (typeof video.load === 'function') {
          video.load();
        }
      });
    };

    resolveFile(fileLocalUrl)
      .then(readFile)
      .then(getDuration)
      .then((duration) => {
        console.log('video duration is: ' + duration);
      })
      .catch((err) => {
        console.error(err);
      });

We get an error like this:

GET blob:file:///a06f500c-79b8-4176-8d1a-39bcf8d0b3d4 500 (Internal Server Error)

It appears the problem is with neither XMLHttpRequest, nor URL.createObjectURL.

If we repeat those operations on a small file (< 10MB) – no error appears.

The solution to this problem is surprising. Because WebView has a data transfer limit when working with native code, we need to read the file's blob in chunks. After that, we can join these chunks into a single blob and use it! Here is a replacement readFile function:

    const readFileChunk = (file, start, chunkSize, chunks) {
      return new Promise((resolve, reject) => {
        const reader = new FileReader();
        reader.onloadend = (ev) => {
          if (ev.target.error) {
            return reject(ev.target.error);
          }
          resolve(new Blob([new Uint8Array(reader.result)]));
        }

        let toRead = file;
        if (typeof start !== 'undefined' && typeof chunkSize !== 'undefined') {
          toRead = file.slice(start, start + chunkSize);
          start += chunkSize;
        }

        return reader.readAsArrayBuffer(toRead);
      })
      .then((chunk) => {
        chunks.push(chunk);
        if (start < file.size) {
          return readFileChunk(file, start, chunkSize, chunks);
        } else {
          return new Blob(chunks);
        }
      });
    };

    const readFile = (file) => {
      const chunkSize = 10485760; // 10MB
      const blobs = [];
      return readFileChunk(file, 0, chunkSize, blobs)
    };

However, this approach is memory consuming, because the final blob is assembled and stored in memory. If you’re using very large files, the application may crash. In this case, we don't need to assemble the chunks – we can just send them to a server with some additional data (each chunk's start and end in the result file).