The MMM (Triple M)

[SpiritTree]

Introduction

Hello, everyone, it is Tidal_Force in case you are unfamiliar with my forum name.

As builds get more and more ambitious, it has come to my attention that a large, centralized memory system would be useful, as it would allow ORE members to offload large data storage from their plot. It would also serve as an extremely slow, rudimentary network if it has to. However, its protocols will be extremely simple and not explicitly based on real-life systems.

I propose the Massive Memory Module (the MMM for short) to accommodate this need. It is something I have wanted to do for a while now as I will need a large storage system for my upcoming CPU. Below is a list of its protocols and how I will be implementing it.

Basic Operation

The MMM is a system that operates on 8 byte blocks of memory. The memory is stored in queues, which wraparound to provide non-erasing reads. There is a link to the basic structure of the memory system I will be using. It is binary and it approaches 6 blocks/bit.

https://imgur.com/a/zH08e

There will be 2 ^ 11 (2048) addresses. This provides a grand total of 16384 bytes of data that the memory will be able to store. The addresses will only be allocated to a particular user nominally once the system is set up.

Protocol

The MMM will only allow one user access to it at a time. I realize this is simplistic, but I do not anticipate the need for multiple data accesses for an already slow system. If the MMM is already being accessed by a user when another user accesses it, the MMM will deny the request of the second user. It will not hold their request as a pending operation. Each user of the MMM will be identified by a plot address. The basic format of serial data that one needs to send to the MMM is as follows.

[1 Bit: Request Access/Intialization][1 Bit: Operation][11 Bits: Address of Block][Optional 64 Bits: Data][Optional 8 bits: Plot Address]

This means that each access to the MMM will be either 21 bits for reading or 77 bits for writing. If you want to communicate with the MMM, I will provide a serial system for you.

Networking

I will be (if this project is approved) constructing a network that will act as a scaffold for basic operation of the MMM. This network may be replaced when more efficient networks are finished. Each user will communicate with the MMM through the use of a central bus. This bus will be locked in two situations:

• An acknowledged request to the MMM until the MMM is finished processing the request.
  

• A request to the MMM when it is not currently busy.
  

Although this implementation is inherently simplistic, it will make it easier for the project to be completed.

Concerns

• I will need another plot to construct the MMM and its related data buffers.
  

• I am worried about connecting plots together as it may require going through unapproved areas. This is the main problem and if anyone has solutions it would be incredibly helpful.
  

• The access times may be in the order of minutes (600+ ticks).
  
  
  
  Conclusion
  
  
• The MMM will provide for the growing need of massive storage in ORE. It will be able to store 16384 bytes of data and access them in 8 byte blocks. If anyone would like to share suggestions or criticism, I am more than happy to read them.

[Digitalis]

Suggestion: Add a layer of abstraction on top of the main memory controller that buffers requests from different ports and reads them in a round robin fashion to avoid possible collision between users; like hyper-threading but for memory.

[SpiritTree]

Digi, that is a great idea, and I will try to implement that once the basic system is operational.

On another note, upon some basic calculations, I realize that I will have to do two tiers of memory cubes, simply because one tier only allows around 8000 bytes instead of the required 16384.

The height of the entire system will be, if everything goes according to plan, 76 blocks.

[LambdaPI]

Can you implement private storage? So you can't accidentally overwrite someone else's data.

[LordDecapo]

Just gonna throw this in here.... another cross plot like network won't be approved. Small scale implementation where ppl can move there plots near u if they want to participate, or we work out something to give the project a group of plots. But a server wide network won't be approved. Far to many issues come up with it, and the very rare times ppl will use it (so far... like 4 networks made.. ppl get excited say they will build... no one does) it will lag and cause chunk loading issues... 
If 1 person does a WE goof and doesn't catch that it messed up a network wire.. shit could be down for ages till u find that 1 block of dust. 

I like the idea, but keep it local. (Btw every single stack of memory this large I have seen.... drops everyone's FPS to about 2, and kills the server in lag... while it's.. off)

Also... 600+ ticks for 16kB is a lot of time.. I can stack my memory in my GPU and get 16kB in well under 100 ticks (guessed time, scaled up from my last timing count when I estimated 4kB time) and it's not destructive

[Matthew]

16kB............ ok... i'll believe it when i see it. Unless you're doing item based that's not possible... and if you are doing item based you will never get that fast a read. (unless you do a really fast encoding/decoding setup that sacrifices size/compression, but in that case it would also lag to much) so like yeah... 16kB @600ticks isn't happening lol your a few thousand ticks off xD Best i could get was 540 bytes per chest with a read of like 10 mins. not to mention bussing from mem to the destination lol if you drop down to 20 bytes per chest you could get 9 min read times lol. On a real note, being realistic id try 4-8kB of serial hex. The read/write would be less than 150 ticks and the lag may possible be bearable. 4kB of that would tots work, idk about the 8 though. Anyways have fun. also yeah have private storage, use an xor cipher, that'd be neat af

[Zijkhal]

Isn't this type of memory sort of not working 'cause of the weird way redstone signals spread? Or having a redstone wire between the repeaters fixes it?

The way I belive it can be solved is by having a subtract comparator between the repeaters side powered by a repeater. That system can achieve 10 blocks per bit if the control line is shared.
Also, isnt that 12 blocks per bit, if I see correctly each cell is 2x3x2?

[SpiritTree]

Can you implement private storage? So you can't accidentally overwrite someone else's data.

Yes, now that you have mentioned it, I will. It should not be too difficult.

Just gonna throw this in here.... another cross plot like network won't be approved. Small scale implementation where ppl can move there plots near u if they want to participate, or we work out something to give the project a group of plots. But a server wide network won't be approved. Far to many issues come up with it, and the very rare times ppl will use it (so far... like 4 networks made.. ppl get excited say they will build... no one does) it will lag and cause chunk loading issues... 
If 1 person does a WE goof and doesn't catch that it messed up a network wire.. shit could be down for ages till u find that 1 block of dust. 

I like the idea, but keep it local. (Btw every single stack of memory this large I have seen.... drops everyone's FPS to about 2, and kills the server in lag... while it's.. off)

Also... 600+ ticks for 16kB is a lot of time.. I can stack my memory in my GPU and get 16kB in well under 100 ticks (guessed time, scaled up from my last timing count when I estimated 4kB time) and it's not destructive

Alright, I am totally fine with a local implementation. I can understand why a large scale network would not work. If someone does happen to make a W/E mistake it should be fairly simple to locate as I'm separating the memory into blocks. Chunk loading will not be a large problem. If I remember correctly, you can do it with hoppers chained into another chunk.

I'll have to see what I can do about the lag. I am currently building a model, but obviously the FPS and TPS is not affected until the large scale. My estimate of 600 ticks was a complete guess, I'm going to be doing actual calculations of it once I finish the model.

16kB............   ok... i'll believe it when i see it. Unless you're doing item based that's not possible... and if you are doing item based you will never get that fast a read. (unless you do a really fast encoding/decoding setup that sacrifices size/compression, but in that case it would also lag to much) so like yeah... 16kB @600ticks isn't happening lol your a few thousand ticks off xD Best i could get was 540 bytes per chest with a read of like 10 mins. not to mention bussing from mem to the destination lol if you drop down to 20 bytes per chest you could get 9 min read times lol. On a real note, being realistic id try 4-8kB of serial hex. The read/write would be less than 150 ticks and the lag may possible be bearable. 4kB of that would tots work, idk about the 8 though. Anyways have fun.   also yeah have private storage, use an xor cipher, that'd be neat af

Oh ye of little faith, this memory will take up a large area, which will make it totally possible . As for speed, as I mentioned above, I've completely guessed as I'm still working on the small scale model. I've chosen not to use hex because of much, much larger access times and bussing delay/syncing issues. I am implementing private storage.

Isn't this type of memory sort of not working 'cause of the weird way redstone signals spread? Or having a redstone wire between the repeaters fixes it?

The way I belive it can be solved is by having a subtract comparator between the repeaters side powered by a repeater. That system can achieve 10 blocks per bit if the control line is shared.
Also, isnt that 12 blocks per bit, if I see correctly each cell is 2x3x2?

To my knowledge, chunk loaders can fix any problems associated with redstone not being handled correctly in unloaded chunks. I'm not sure your system will work, but I will definitely try it.

You are correct, it is 12 blocks per bit. Derp. Actually less than that now. Here is a picture of my progress on the small scale model.

http://imgur.com/a/bpvIP

If I haven't derped out on the math a second time, each bit takes around 24 blocks.

Did I mention I'm aiming for it to be completely torchless / pistonless?

[Matthew]

Oh ye of little faith, this memory will take up a large area, which will make it totally possible . As for speed, as I mentioned above, I've completely guessed as I'm still working on the small scale model. I've chosen not to use hex because of much, much larger access times and bussing delay/syncing issues. I am implementing private storage.

16KB of base 2........... this will 100% work, no doubt.

If you got 16KB of writable base 2 to work with no tick skipping/data loss/corruption on any address and the lag isn't unbearable i would pay you $100 lol... you know what, if you actually do it in 3 days and meet those standards i will pay lol

[Matthew]

with ur bussing and no decoders i calculated 2.8 mill blocks lol.... i mean its a little rough of an estimate but still ;p just having that all loaded should break shit lol

[LordDecapo]

If you use my serial/hex conversion system, at your scale, it will end up saving u a LOT of space AND speed.
Store data in hex, and right before the write and right after the read, you convert to/from (respecivtly) hex. Allows the density of hex, and the speed + less lag perks of binary transition. Next time ur on do /warp OREboy and look at the memory I'm using, non destructive hex memory that I'm using to pulse serial data in and out. I can explain the stuff I did with it in game or voice if I'd like.
If you used something like that, it would make a lot of stuff easier,, like no complex writeback system, more compact design. And lacks the cons of hex bussing.

With that kinda of system and maybe taking a single plot and putting 4KB on each of the corners of it and use some random part of the address to decide which element data goes to,, (bits 6 and 7 would be pretty good for MC due to my guesses and experiences related to normal program size, packet sizes and such in MC. That spacing could likely allow multiple accesses at once due to the banking) would allow you to do this at 16KB with out AS BAD of lag issues. Or split it up even farther and be even better.

Btw, thanks for understanding about server wide projects. If this gets big, get in touch with me or another admin about moving plots and getting more to have space for this.

[Matthew]

^ that actually has a small chance of working, listen to decrapo.

[SpiritTree]

If you use my serial/hex conversion system, at your scale, it will end up saving u a LOT of space AND speed.
Store data in hex, and right before the write and right after the read, you convert to/from (respecivtly) hex. Allows the density of hex, and the speed + less lag perks of binary transition. Next time ur on do /warp OREboy and look at the memory I'm using, non destructive hex memory that I'm using to pulse serial data in and out. I can explain the stuff I did with it in game or voice if I'd like.
If you used something like that, it would make a lot of stuff easier,, like no complex writeback system, more compact design. And lacks the cons of hex bussing.

With that kinda of system and maybe taking a single plot and putting 4KB on each of the corners of it and use some random part of the address to decide which element data goes to,, (bits 6 and 7 would be pretty good for MC due to my guesses and experiences related to normal program size, packet sizes and such in MC. That spacing could likely allow multiple accesses at once due to the banking) would allow you to do this at 16KB with out AS BAD of lag issues. Or split it up even farther and be even better.

Btw, thanks for understanding about server wide projects. If this gets big, get in touch with me or another admin about moving plots and getting more to have space for this.

Hm, I will think about it. I am guessing you are suggesting to me a hex queue? The memory you have at the warp at the moment is more dense than mine (14 blocks / bit without averaging decoder space). However, the decoder averaging may make it around the same. Since the binary model is practically finished, I will work on a hex model and determine if it invokes the speed increases and lag decreases that you say it will.

My write back system is actually fairly simple. It's entirety is contained in the second linked picture. As for lag, I was unaware that hex caused less of an issue than binary. Do you know this for a fact to be true? I am worried about the higher component counts in a hex based system nullifying any lag decreases we get and possibly making it lag more.

Anyway, thank you for the suggestion, and we will see if a hex model promotes a better overall system.

[Matthew]

If you use my serial/hex conversion system, at your scale, it will end up saving u a LOT of space AND speed.
Store data in hex, and right before the write and right after the read, you convert to/from (respecivtly) hex. Allows the density of hex, and the speed + less lag perks of binary transition. Next time ur on do /warp OREboy and look at the memory I'm using, non destructive hex memory that I'm using to pulse serial data in and out. I can explain the stuff I did with it in game or voice if I'd like.
If you used something like that, it would make a lot of stuff easier,, like no complex writeback system, more compact design. And lacks the cons of hex bussing.

With that kinda of system and maybe taking a single plot and putting 4KB on each of the corners of it and use some random part of the address to decide which element data goes to,, (bits 6 and 7 would be pretty good for MC due to my guesses and experiences related to normal program size, packet sizes and such in MC. That spacing could likely allow multiple accesses at once due to the banking) would allow you to do this at 16KB with out AS BAD of lag issues. Or split it up even farther and be even better.

Btw, thanks for understanding about server wide projects. If this gets big, get in touch with me or another admin about moving plots and getting more to have space for this.

Hm, I will think about it. I am guessing you are suggesting to me a hex queue? The memory you have at the warp at the moment is more dense than mine (14 blocks / bit without averaging decoder space). However, the decoder averaging may make it around the same. Since the binary model is practically finished, I will work on a hex model and determine if it invokes the speed increases and lag decreases that you say it will.

My write back system is actually fairly simple. It's entirety is contained in the second linked picture. As for lag, I was unaware that hex caused less of an issue than binary. Do you know this for a fact to be true? I am worried about the higher component counts in a hex based system nullifying any lag decreases we get and possibly making it lag more.

Anyway, thank you for the suggestion, and we will see if a hex model promotes a better overall system.

ok... listen to him and not me... i see how it is

[LordDecapo]

Well the decode space can be less then what I have. Mine is optimized for its size and to be simulated dual read, could get a mass storage system smaller.

And yes, we do know hex bussing to be significantly more laggy then binary bussing. Especially if ur sending pulses.
I hope to see what u have so far and to talk to u next time we are both on. I'm curious about a few things

[SpiritTree]

I am abandoning this project due to the advent of item memory research.