![]() But everything changes when each file is larger than 100 MBytes. For all files less than 100 MBytes each everything above is dead accurate and the only thing to think about. And interestingly based on the current client design having 32 slow cores isn't as important to Backblaze as having one or two cores in your CPU that are VERY fast, hopefully 5 GHz or approaching that. If you backup from an SSD, it will help quite a bit. Often this will get better over time as you get to larger files because as I mentioned those are faster to read. You aren't limited by your network, or Backblaze, your CPU or disks locally cannot be read fast enough, and 1 or 2 seconds after the bztrans_thread is launched it completes before another one can be launched. if you only see one or two bztrans_thread (or pretty much anything less than 100) what that means is that your bztransmit parent main thread cannot keep the network "fed". Now here is where it gets interesting to know what is going on. Ok, if you open Task Manager on Windows (Activity Monitor on the Macintosh), count the number of "bztrans_thread" processes you see. All the coordination and logic is in the parent "bztransmit". Now all of the "bztrans64_thread.exe" are children processes doing nothing but the transmitting of the data, so they are small, and pretty simple. Ok, that's the thread doing most of the reading, and most of the encrypting of files currently. ![]() The name of the parent process is bztransmit64.exe on Windows, on Mac it is "bztransmit" without the "64". Ok, so highlighted at the very very bottom of that screenshot it says "main command thread takes 1.5 GBytes of RAM at peak". I have a "todo" to spread that out just a little and push the encryption into the children threads to boost this a little.įor this part, take a look at this screenshot of my desktop: That is a real world backup from Austin Texas on Google fiber (1 Gbit/sec upload capacity from homes) to our datacenters in Sacramento California. The next limit is your CPU which is encrypting the files in only one thread. First of all, If you are reading from a hard drive that can be slow, especially the small files which come first. Now if you are only seeing 300 Mbits/sec, you might be hitting some local limits, but it isn't your bandwidth. So you might see the 300 Mbits/sec rise if you give it a few days. And Backblaze Personal Backup backs up in file size order, small files first. This is due to the fixed amount of time to setup and tear down the HTTPS connections, plus the seek times to write tiny files to slow spinning hard drives in our datacenter. Now, small files MURDER Backblaze's performance. So 100 threads should be able to hit 1 Gbit/sec speeds. Each thread has no idea the other threads on other vaults are even accepting your data at the same time. If you have 100 threads, they can (and do) upload to 100 different servers in the Backblaze datacenter in 100 different racks and aren't even going through a load balancer. The GOOD NEWS is that it is PERFECTLY parallelizable. You can read about the math involved with splitting the file into 17 parts and calculating 3 parity parts here: And you can read about the data durability we (Backblaze) are shooting for with this system of splitting up your data and storing it in different places here: Īll of that chatter and the fact that it is being written to slow spinning drives means 1 thread can't upload faster than around 10 Mbits/sec. Once they have been committed to drive, they respond "we're good" to the server your computer is talking with, and that server ONLY AFTER ALL OF THAT responds to your laptop or desktop saying "The data has been committed safely to the vault". ![]() ![]() You can read about Vaults here: but here is what occurs next: your data is split into 17 parts, 3 parity parts are calculated, and the 20 different "shards" (our terminology) are sent to 20 different computers in 20 different locations in our datacenter, and written to slow spinning drives. What happens is your computer HTTPS POSTS the encrypted data to a single machine in the Backblaze datacenter that is part of what is called a "Vault". The limit I'm mentioning isn't on your side. However, each 1 thread is limited to around 10 Mbits/sec - maybe 15 Mbits/sec maximum based on conditions. I'm just wondering whether there is a speed limit for uploading to Backblaze personal plan? Disclaimer: I work at Backblaze and recently worked on the upload performance and changed the code in certain ways. ![]()
0 Comments
Leave a Reply. |
Details
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |