本文为转载文章, 仅用于自己的知识管理收集, 如果涉及侵权,请联系 suziwen1@gmail.com,会第一时间删除
收集该文章,并非代表本人支持文中观点,只是觉得文章内容容易引起思考,讨论,有它自有的价值转载自: https://joji.me/en-us/blog/processing-huge-files-using-filereader-readasarraybuffer-in-web-browser/
The FileReader API in HTML5 allows web browsers to access user files without uploading the files to the web servers. It not only lightens the load of web server but also saves the time of uploading files. It is very easy to use FileReader.readAsText() to process a 300K log file. However, when the file size grows to 1GB, or even 2GB, then the page might crash in browser or not function correctly. This is because readAsText() loads the full file into memory before processing it. As a result, the process memory exceeds the limitation. To prevent this issue, we should use FileReader.readAsArrayBuffer() to stream the file when the web application needs to process huge files, so you only hold a part of the file in the memory.
Test Scenario
Our test scenario is to get the log time range using JavaScript from a given IIS log on local disk.
Sample IIS log:
- 1#Software: Microsoft Internet Information Services 10.0
- 2#Version: 1.0
- 3#Date: 2016-08-18 06:53:55
- 4#Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(Referer) sc-status sc-substatus sc-win32-status time-taken
- 52016-08-18 06:53:55 ::1 GET / - 80 - ::1 Mozilla/5.0+(Windows+NT+10.0;+WOW64;+Trident/7.0;+rv:11.0)+like+Gecko - 200 0 0 476
- 62016-08-18 06:53:55 ::1 GET /iisstart.png - 80 - ::1 Mozilla/5.0+(Windows+NT+10.0;+WOW64;+Trident/7.0;+rv:11.0)+like+Gecko http://localhost/ 200 0 0 3
- 72016-08-18 08:45:34 10.172.19.198 GET /test/pac/wpad.dat - 80 - 10.157.21.235 Mozilla/5.0+(Windows+NT+6.1;+Win64;+x64;+Trident/7.0;+rv:11.0)+like+Gecko - 404 3 50 265
- 82016-08-18 08:46:44 10.172.19.198 GET /test/pac/wpad.dat - 80 - 10.157.21.235 Mozilla/5.0+(Windows+NT+6.1;+Win64;+x64;+Trident/7.0;+rv:11.0)+like+Gecko - 200 0 0 6
Our target is to get the time range of this IIS log
- Start time: 2016-08-18 06:53:55
- End time: 2016-08-18 08:46:44
Implementation using readAsText()
It is quite easy and clear to implement the functionality using readAsText(). After getting the whole content of the file in a string, iterates each line from the beginning and get the string of first 19 characters, then check if it matches the date and time format. If it does, the string is the start time. In the same way, get the end time by iterating each line from the end.
- 1<input type="file" id="file" />
- 2<button id="get-time">Get Time</button>
- 3<script>
- 4 document.getElementById('get-time').onclick = function () {
- 5 let file = document.getElementById('file').files[0];
- 6 let fr = new FileReader();
- 7 fr.onload = function (e) {
- 8 let startTime = getTime(e.target.result, false);
- 9 let endTime = getTime(e.target.result, true);
- 10 alert(`Log time range: ${startTime} ~ ${endTime}`);
- 11 }
- 12 fr.readAsText(file);
- 13 }
- 14 function getTime(text, reverse) {
- 15 let timeReg = /\d{4}\-\d{2}\-\d{2} \d{2}:\d{2}:\d{2}/;
- 16 for (let i = reverse ? text.length - 1 : 0; reverse ? i > -1 : i < text.length; reverse ? i-- : i++) {
- 17 if (text[i].charCodeAt() === 10) {
- 18 let snippet = text.substr(i + 1, 19);
- 19 if (timeReg.exec(snippet)) {
- 20 return snippet;
- 21 }
- 22 }
- 23 }
- 24 }
- 25</script>
The result of processing the sample IIS log (file size: 1K) matches our expectation.
However, the browser crashed if I chose a huge IIS log file (size: 2G) because readAsText() loaded the full file into memory and the process memory exceeds the limitation
Implementation using readAsArrayBuffer()
A File object in JavaScript is inherited from Blob object. We can use Blob.slice() to cut the file into pieces for further processing. The workflow is as below:
- Gets the first 10KB of the file and decode it to text
- Iterates each line from the beginning of the text and checks whether the string of first 19 characters matches the date and time format. If it does, then the string is the start time
- Gets the last 10KB of the file and decode it to text
- In the same way, get the end time by iterating each line from the end
Here is the code:
- 1<input type="file" id="file" />
- 2<button id="get-time">Get Time</button>
- 3<script>
- 4 document.getElementById('get-time').onclick = function () {
- 5 let file = document.getElementById('file').files[0];
- 6 let fr = new FileReader();
- 7 let CHUNK_SIZE = 10 * 1024;
- 8 let startTime, endTime;
- 9 let reverse = false;
- 10 fr.onload = function () {
- 11 let buffer = new Uint8Array(fr.result);
- 12 let timeReg = /\d{4}\-\d{2}\-\d{2} \d{2}:\d{2}:\d{2}/;
- 13 for (let i = reverse ? buffer.length - 1 : 0; reverse ? i > -1 : i < buffer.length; reverse ? i-- : i++) {
- 14 if (buffer[i] === 10) {
- 15 let snippet = new TextDecoder('utf-8').decode(buffer.slice(i + 1, i + 20));
- 16 if (timeReg.exec(snippet)) {
- 17 if (!reverse) {
- 18 startTime = snippet;
- 19 reverse = true;
- 20 seek();
- 21 } else {
- 22 endTime = snippet;
- 23 alert(`Log time range: ${startTime} ~ ${endTime}`);
- 24 }
- 25 break;
- 26 }
- 27 }
- 28 }
- 29 }
- 30 seek();
- 31 function seek() {
- 32 let start = reverse ? file.size - CHUNK_SIZE : 0;
- 33 let end = reverse ? file.size : CHUNK_SIZE;
- 34 let slice = file.slice(start, end);
- 35 fr.readAsArrayBuffer(slice);
- 36 }
- 37 }
- 38</script>
Now we can get the expected result very quickly for a 2GB IIS log after using readAsArrayBuffer.
