+- Forums - Open Redstone Engineers (https://forum.openredstone.org)
+-- Forum: Off-Topic (https://forum.openredstone.org/forum-4.html)
+--- Forum: Programming (https://forum.openredstone.org/forum-8.html)
+--- Thread: Time based file compression idea? (/thread-9870.html)
Pages:
1
2
Time based file compression idea? - AltruismAndCake - 05-06-2016
I have this idea for saving bandwidth using a compression(?) idea I've been thinking about, but I really need to hammer out details before I continue. The idea is to use a timing mechanism to cut down the amount of bytes sent.
We start with a 3d box fill with cubes. The exact dimensions can be adjusted, but in this case the box is 65536x16x8. Each cube is a byte of data. The values inside the cube are determined by a pseudo random number generator, with a shared seed. This box will (or should in theory) be identical on both ends. Start with the number 0 at the exact beginning of the minute, and incrementing 1 for each millisecond after the beginning of that minute until you reach 65536, filling data along the way. This interval can be represented as a 16 bit number, 2 bytes.
So if the time is 12:34PM, 56 seconds, 789 milliseconds:
0 means 12:34PM, 56 seconds, 0 milliseconds
55 means 12:34PM, 56 seconds, 55 milliseconds
900 means 12:34PM, 55 seconds, 900 milliseconds
Using another 16 bits, 2 bytes, for selecting rows of 8 byte data, makes a total of 4 bytes of bandwidth used for 8 bytes (most likely scenario) to 128 bytes of the file.
In theory, if you have really bad luck, you could wait days, weeks, months, years before the file finishes downloading. I know this, but without tests, I don't know what the most likely scenario is.
Are there any (other) flaws with the design? I'd like to find out now rather than after i start working on it.
TL;DR download anything over the internet using 1 bit of bandwidth. Just start at 0 and count up. See you next millennium
RE: Time based file compression idea? - VoltzLive - 05-06-2016
Reasons why this is hard and kind of sucks:
- You're trading bandwidth for CPU cycles
- It's very hard to take a chunk of data and create a psudo random seem that is significantly smaller in enough time
A much smarter way is to send the seed, then send a sparse voxel octtree of the bits of the world the player has changed. Though a problem arises when editing the world, as octtrees have a logarithmic rebuild rate, somethine we don't have to worrie about with a byte array implementation like we currently have.
RE: Time based file compression idea? - AltruismAndCake - 05-13-2016
OK, decided to build a demo of my idea to see what kind of speed I would get. To send the first four bytes of data using 2 bytes would take 5 days. Epic fail. I have another idea, but I don't know if the bandwidth savings are worth the effort. It would save 1 byte for every 8 bytes sent. This method would guarantee a lower bounds for speed, worst case scenario, you would get speeds of 2 bytes per second.
So my question is: is making a 9 GB file use only 8 GB of bandwidth worth it if it means you may get speeds of 2 bytes per second (or more depending)?
RE: Time based file compression idea? - PabloDons - 05-14-2016
lolno. bandwidth is not really the problem, it's how fast it downloads that is the real problem. Pretty much nobody pays for a quota of x GB, instead they buy unlimited with x speed
RE: Time based file compression idea? - Chibill - 05-14-2016
But Pablo if the file is smaller it should download fast. (Normally....) Because speed is X Megabits/Second SO if its 1 Gigabyte less then there is that much less data to have to send.
RE: Time based file compression idea? - AltruismAndCake - 05-15-2016
@PabloDons Where I live, it's quite common to pay per GB, it wasn't until a few months ago that something better came up: $60 for 10 GB and unlimited 2G after that. To be quite frank, the previous internet plan was eating us alive at a price of $15/GB with no cap on how much you can download. At one point, my family had to divide a $250 between 4 people. So where do you live that you get unlimited and not throttled?
@Chibill technically the file isn't smaller at all, i just reduce the usage. The protocol is something like any byte sent also represents a bit. This bit equals 0 if it's received the first half a second, 1 if second half. So:
0.25s : fe de ad be
0.57s : ef ee 13
1.11s : 34 22 91
Would be 00001110 (00)<-discarded unless more bytes/bits come
So the whole message is fedeadbeef1334(0e)2291
The thing is this method delays the sending of packets sometimes for the sake on saving bandwidth. I guess I could just drop the idea as I am no longer under a cruel and unusual ISP, just an idea that had me curious.
RE: Time based file compression idea? - PabloDons - 05-15-2016
(05-15-2016, 01:46 AM)AltruismAndCake Wrote: @PabloDons Where I live, it's quite common to pay per GB, it wasn't until a few months ago that something better came up: $60 for 10 GB and unlimited 2G after that. To be quite frank, the previous internet plan was eating us alive at a price of $15/GB with no cap on how much you can download. At one point, my family had to divide a $250 between 4 people. So where do you live that you get unlimited and not throttled?
Oh man, I had no clue paying per GB was common. In norway there is possibly 1 ISP that offers that, but they are mainly a phone company. On phone it is common, but I can't imagine anyone ever downloading large files over 4G as you can just wait until you get home. Well in that case, this does make sense. I think you should keep refining it. Doesn't sound impossible to make it usable to me, though I don't fully understand it.
RE: Time based file compression idea? - Chibill - 05-15-2016
I am stuck with 3 Mb/s but in! limited download.
RE: Time based file compression idea? - AltruismAndCake - 05-15-2016
@PabloDons, I could see if there is some way to speed up already fast downloads, but the current implemention has problems with high speed. I could readjust it, but benefits are still questionable. I'll explain the problem later. The lower your highest ping amount is, the more benefit you get from this. Average ping won't work with this formula, unless your ping is consistently average. 1000/(2*ping)= bits saved per second in the worst case scenario. So a ping of 50ms can download 10 bits per second in addition to whatever you download normally. Technically you can get more than 10 bits but I would need to do a ton of fiddling and tweaking, which in my guess you would get 40 bits per second if lucky. Now that I say it that way, it might not be worth it xD.
Anyway, assuming that the amount you download is consistent (probly not), you could choose when to send packets to save bits. Delaying packets in this example by your ping amount will mean you download the same amount at the end of the second (probly not the case IRL).
Every X bytes of data received, a bit will be saved too, depending on WHEN the packet is received. In the 50ms example, every 50ms means to toggle the value of the "register" bit (0->1->0). Once a bit is saved, it won't be changed, but the register still keeps toggling, saving more bits later on.
I guess saving 10-40 bits per second wouldn't be very ground breaking for someone with a 3Mb/s download or higher though. The whole thing relies on low ping, low speed connections, which is almost never the case. Usually someone has high-low or low-high.
The idea has had a long journey away from what it originally was. It started with me wondering if there was someway to convert satellite video on two ends into digital data, then receive a little bit of data to find a huge chunk of data. But cutting out the satellite video as a middle man proved that its not very efficient.
This new idea is the best I've got so far, 10-40 bits/second... assuming it works and you get lucky. So polish it some more, or drop it and look somewhere else?
RE: Time based file compression idea? - AltruismAndCake - 05-15-2016
@chibill heh, my internet is 3mb/s when on the 4g, its more like 127kB/s on the 2g. Although my upload is magick, it's 10mb/s, high or low speed.