Time based file compression idea?

[AltruismAndCake]

I have this idea for saving bandwidth using a compression(?) idea I've been thinking about, but I really need to hammer out details before I continue. The idea is to use a timing mechanism to cut down the amount of bytes sent.

We start with a 3d box fill with cubes. The exact dimensions can be adjusted, but in this case the box is 65536x16x8. Each cube is a byte of data. The values inside the cube are determined by a pseudo random number generator, with a shared seed. This box will (or should in theory) be identical on both ends. Start with the number 0 at the exact beginning of the minute, and incrementing 1 for each millisecond after the beginning of that minute until you reach 65536, filling data along the way. This interval can be represented as a 16 bit number, 2 bytes.

So if the time is 12:34PM, 56 seconds, 789 milliseconds:
0 means 12:34PM, 56 seconds, 0 milliseconds
55 means 12:34PM, 56 seconds, 55 milliseconds
900 means 12:34PM, 55 seconds, 900 milliseconds

Using another 16 bits, 2 bytes, for selecting rows of 8 byte data, makes a total of 4 bytes of bandwidth used for 8 bytes (most likely scenario) to 128 bytes of the file.

In theory, if you have really bad luck, you could wait days, weeks, months, years before the file finishes downloading. I know this, but without tests, I don't know what the most likely scenario is.

Are there any (other) flaws with the design? I'd like to find out now rather than after i start working on it.

TL;DR download anything over the internet using 1 bit of bandwidth. Just start at 0 and count up. See you next millennium

[VoltzLive]

Reasons why this is hard and kind of sucks:

• You're trading bandwidth for CPU cycles
  
• It's very hard to take a chunk of data and create a psudo random seem that is significantly smaller in enough time
  

Compression only works to a point, this is mathematically a form of compression.

A much smarter way is to send the seed, then send a sparse voxel octtree of the bits of the world the player has changed.  Though a problem arises when editing the world, as octtrees have a logarithmic rebuild rate, somethine we don't have to worrie about with a byte array implementation like we currently have.

[AltruismAndCake]

OK, decided to build a demo of my idea to see what kind of speed I would get. To send the first four bytes of data using 2 bytes would take 5 days. Epic fail. I have another idea, but I don't know if the bandwidth savings are worth the effort. It would save 1 byte for every 8 bytes sent. This method would guarantee a lower bounds for speed, worst case scenario, you would get speeds of 2 bytes per second.

So my question is: is making a 9 GB file use only 8 GB of bandwidth worth it if it means you may get speeds of 2 bytes per second (or more depending)?

[PabloDons]

lolno. bandwidth is not really the problem, it's how fast it downloads that is the real problem. Pretty much nobody pays for a quota of x GB, instead they buy unlimited with x speed

[Chibill]

But Pablo if the file is smaller it should download fast. (Normally....) Because speed is X Megabits/Second SO if its 1 Gigabyte less then there is that much less data to have to send.

[AltruismAndCake]

@PabloDons Where I live, it's quite common to pay per GB, it wasn't until a few months ago that something better came up: $60 for 10 GB and unlimited 2G after that. To be quite frank, the previous internet plan was eating us alive at a price of $15/GB with no cap on how much you can download. At one point, my family had to divide a $250 between 4 people. So where do you live that you get unlimited and not throttled?

@Chibill technically the file isn't smaller at all, i just reduce the usage. The protocol is something like any byte sent also represents a bit. This bit equals 0 if it's received the first half a second, 1 if second half. So:

0.25s : fe de ad be
0.57s : ef ee 13
1.11s : 34 22 91

Would be 00001110 (00)<-discarded unless more bytes/bits come

So the whole message is fedeadbeef1334(0e)2291

The thing is this method delays the sending of packets sometimes for the sake on saving bandwidth. I guess I could just drop the idea as I am no longer under a cruel and unusual ISP, just an idea that had me curious.

[PabloDons]

@PabloDons Where I live, it's quite common to pay per GB, it wasn't until a few months ago that something better came up: $60 for 10 GB and unlimited 2G after that. To be quite frank, the previous internet plan was eating us alive at a price of $15/GB with no cap on how much you can download. At one point, my family had to divide a $250 between 4 people. So where do you live that you get unlimited and not throttled?

Oh man, I had no clue paying per GB was common. In norway there is possibly 1 ISP that offers that, but they are mainly a phone company. On phone it is common, but I can't imagine anyone ever downloading large files over 4G as you can just wait until you get home. Well in that case, this does make sense. I think you should keep refining it. Doesn't sound impossible to make it usable to me, though I don't fully understand it.

[Chibill]

I am stuck with 3 Mb/s but in! limited download.

[AltruismAndCake]

@PabloDons, I could see if there is some way to speed up already fast downloads, but the current implemention has problems with high speed. I could readjust it, but benefits are still questionable. I'll explain the problem later. The lower your highest ping amount is, the more benefit you get from this. Average ping won't work with this formula, unless your ping is consistently average. 1000/(2*ping)= bits saved per second in the worst case scenario. So a ping of 50ms can download 10 bits per second in addition to whatever you download normally. Technically you can get more than 10 bits but I would need to do a ton of fiddling and tweaking, which in my guess you would get 40 bits per second if lucky. Now that I say it that way, it might not be worth it xD.

Anyway, assuming that the amount you download is consistent (probly not), you could choose when to send packets to save bits. Delaying packets in this example by your ping amount will mean you download the same amount at the end of the second (probly not the case IRL).

Every X bytes of data received, a bit will be saved too, depending on WHEN the packet is received. In the 50ms example, every 50ms means to toggle the value of the "register" bit (0->1->0). Once a bit is saved, it won't be changed, but the register still keeps toggling, saving more bits later on.

I guess saving 10-40 bits per second wouldn't be very ground breaking for someone with a 3Mb/s download or higher though. The whole thing relies on low ping, low speed connections, which is almost never the case. Usually someone has high-low or low-high.

The idea has had a long journey away from what it originally was. It started with me wondering if there was someway to convert satellite video on two ends into digital data, then receive a little bit of data to find a huge chunk of data. But cutting out the satellite video as a middle man proved that its not very efficient.

This new idea is the best I've got so far, 10-40 bits/second... assuming it works and you get lucky. So polish it some more, or drop it and look somewhere else?

[AltruismAndCake]

@chibill heh, my internet is 3mb/s when on the 4g, its more like 127kB/s on the 2g. Although my upload is magick, it's 10mb/s, high or low speed.

[Chibill]

Now you see I have my unverse thru a phone line at 3mb/s down and .7 up. Now my phone on the other hand. (Which I can theater to has 100 mb/s down and 50 Mb/s up but I Ave to share my 15 GB of data with my brother.

[AltruismAndCake]

I came up with a way to use the timing mechanism better. Ping was a big limiting factor on saving bandwidth. Someone with 50ms ping could only save 10 bits per second. I just now realized that ping may only so fast. For example, my ping with google is between 100ms to 200ms roughly. So instead of using 200ms toggle, I could instead use 100ms toggle and assume all messages will arrive after 100ms passes. Another example is that a person with 40ms-50ms ping garunteed can use a 10ms toggle, and always assume a message will be delayed by 40ms. By doing that, I just made the bits saved per second 5 times more/faster. Another speed up it that I should measure ping one way, server to client. If I use ping this way, rather than round trip ping, it doubles the speed. Still only 100 bit+100bytes/second but using this method 10x faster than the old method.

The more consistent the ping and the less randomness in the time stuff is receive, the faster the method. Someone with a ping of 100-102ms ping would do much better than someone with 45-50ms ping using this method: the slower ping guy would receive data 2.5x faster compared to the faster ping guy IF they both use the same method.

I feel the method needs refined more, but I'm hoping I can keep pushing the method farther. The data savings are still making the file's bandwidth savings 8/9 the size, but I'll worry about saving more after this method gets faster.

[Chibill]

if you can figure out any of the code this would be coo lto dig into.

[AltruismAndCake]

I'll see if I can start coding it tomorrow. Any certain programming language you want it in?

[Chibill]

I will take anything you can easily do it in. I know a few languages that I can read. (Mostly just want a small working model.)

[AltruismAndCake]

Sorry to say, but I ditched this project after I realised I no longer have a use for it. In fact, soon I plan to upgrade my Cricket Wireless plan, to a plan which no longer has any throttling. Sorry Chibill