Forums - Open Redstone Engineers

Hello, everyone, it is Tidal_Force in case you are unfamiliar with my forum name.

As builds get more and more ambitious, it has come to my attention that a large, centralized memory system would be useful, as it would allow ORE members to offload large data storage from their plot. It would also serve as an extremely slow, rudimentary network if it has to. However, its protocols will be extremely simple and not explicitly based on real-life systems.

I propose the Massive Memory Module (the MMM for short) to accommodate this need. It is something I have wanted to do for a while now as I will need a large storage system for my upcoming CPU. Below is a list of its protocols and how I will be implementing it.


The MMM is a system that operates on 8 byte blocks of memory. The memory is stored in queues, which wraparound to provide non-erasing reads. There is a link to the basic structure of the memory system I will be using. It is binary and it approaches 6 blocks/bit.


There will be 2 ^ 11 (2048) addresses. This provides a grand total of 16384 bytes of data that the memory will be able to store. The addresses will only be allocated to a particular user nominally once the system is set up.


The MMM will only allow one user access to it at a time. I realize this is simplistic, but I do not anticipate the need for multiple data accesses for an already slow system. If the MMM is already being accessed by a user when another user accesses it, the MMM will deny the request of the second user. It will not hold their request as a pending operation. Each user of the MMM will be identified by a plot address. The basic format of serial data that one needs to send to the MMM is as follows.

[1 Bit: Request Access/Intialization][1 Bit: Operation][11 Bits: Address of Block][Optional 64 Bits: Data][Optional 8 bits: Plot Address]

This means that each access to the MMM will be either 21 bits for reading or 77 bits for writing. If you want to communicate with the MMM, I will provide a serial system for you.


I will be (if this project is approved) constructing a network that will act as a scaffold for basic operation of the MMM. This network may be replaced when more efficient networks are finished. Each user will communicate with the MMM through the use of a central bus. This bus will be locked in two situations:

• An acknowledged request to the MMM until the MMM is finished processing the request.
  

• A request to the MMM when it is not currently busy.
  

Although this implementation is inherently simplistic, it will make it easier for the project to be completed.

• I will need another plot to construct the MMM and its related data buffers.
  

• I am worried about connecting plots together as it may require going through unapproved areas. This is the main problem and if anyone has solutions it would be incredibly helpful.
  

• The access times may be in the order of minutes (600+ ticks).
  
  
  
  Conclusion
  
  
• The MMM will provide for the growing need of massive storage in ORE. It will be able to store 16384 bytes of data and access them in 8 byte blocks. If anyone would like to share suggestions or criticism, I am more than happy to read them.

Suggestion: Add a layer of abstraction on top of the main memory controller that buffers requests from different ports and reads them in a round robin fashion to avoid possible collision between users; like hyper-threading but for memory.

Digi, that is a great idea, and I will try to implement that once the basic system is operational.

On another note, upon some basic calculations, I realize that I will have to do two tiers of memory cubes, simply because one tier only allows around 8000 bytes instead of the required 16384.

The height of the entire system will be, if everything goes according to plan, 76 blocks.

Can you implement private storage? So you can't accidentally overwrite someone else's data.

Just gonna throw this in here.... another cross plot like network won't be approved. Small scale implementation where ppl can move there plots near u if they want to participate, or we work out something to give the project a group of plots. But a server wide network won't be approved. Far to many issues come up with it, and the very rare times ppl will use it (so far... like 4 networks made.. ppl get excited say they will build... no one does) it will lag and cause chunk loading issues... 
If 1 person does a WE goof and doesn't catch that it messed up a network wire.. shit could be down for ages till u find that 1 block of dust. 

I like the idea, but keep it local. (Btw every single stack of memory this large I have seen.... drops everyone's FPS to about 2, and kills the server in lag... while it's.. off)

Also... 600+ ticks for 16kB is a lot of time.. I can stack my memory in my GPU and get 16kB in well under 100 ticks (guessed time, scaled up from my last timing count when I estimated 4kB time) and it's not destructive

16kB............ ok... i'll believe it when i see it. Unless you're doing item based that's not possible... and if you are doing item based you will never get that fast a read. (unless you do a really fast encoding/decoding setup that sacrifices size/compression, but in that case it would also lag to much) so like yeah... 16kB @600ticks isn't happening lol your a few thousand ticks off xD Best i could get was 540 bytes per chest with a read of like 10 mins. not to mention bussing from mem to the destination lol if you drop down to 20 bytes per chest you could get 9 min read times lol. On a real note, being realistic id try 4-8kB of serial hex. The read/write would be less than 150 ticks and the lag may possible be bearable. 4kB of that would tots work, idk about the 8 though. Anyways have fun. also yeah have private storage, use an xor cipher, that'd be neat af

Isn't this type of memory sort of not working 'cause of the weird way redstone signals spread? Or having a redstone wire between the repeaters fixes it?

The way I belive it can be solved is by having a subtract comparator between the repeaters side powered by a repeater. That system can achieve 10 blocks per bit if the control line is shared.
Also, isnt that 12 blocks per bit, if I see correctly each cell is 2x3x2?

(09-05-2017, 12:47 AM)LambdaPI Wrote: [ -> ]Can you implement private storage? So you can't accidentally overwrite someone else's data.

Yes, now that you have mentioned it, I will. It should not be too difficult.

(09-06-2017, 11:13 AM)LordDecapo Wrote: [ -> ]Just gonna throw this in here.... another cross plot like network won't be approved. Small scale implementation where ppl can move there plots near u if they want to participate, or we work out something to give the project a group of plots. But a server wide network won't be approved. Far to many issues come up with it, and the very rare times ppl will use it (so far... like 4 networks made.. ppl get excited say they will build... no one does) it will lag and cause chunk loading issues... 
If 1 person does a WE goof and doesn't catch that it messed up a network wire.. shit could be down for ages till u find that 1 block of dust. 

I like the idea, but keep it local. (Btw every single stack of memory this large I have seen.... drops everyone's FPS to about 2, and kills the server in lag... while it's.. off)

Also... 600+ ticks for 16kB is a lot of time.. I can stack my memory in my GPU and get 16kB in well under 100 ticks (guessed time, scaled up from my last timing count when I estimated 4kB time) and it's not destructive

Alright, I am totally fine with a local implementation. I can understand why a large scale network would not work. If someone does happen to make a W/E mistake it should be fairly simple to locate as I'm separating the memory into blocks. Chunk loading will not be a large problem. If I remember correctly, you can do it with hoppers chained into another chunk.

I'll have to see what I can do about the lag. I am currently building a model, but obviously the FPS and TPS is not affected until the large scale. My estimate of 600 ticks was a complete guess, I'm going to be doing actual calculations of it once I finish the model.

(09-06-2017, 11:25 AM)Matthew Wrote: [ -> ]16kB............   ok... i'll believe it when i see it. Unless you're doing item based that's not possible... and if you are doing item based you will never get that fast a read. (unless you do a really fast encoding/decoding setup that sacrifices size/compression, but in that case it would also lag to much) so like yeah... 16kB @600ticks isn't happening lol your a few thousand ticks off xD Best i could get was 540 bytes per chest with a read of like 10 mins. not to mention bussing from mem to the destination lol if you drop down to 20 bytes per chest you could get 9 min read times lol. On a real note, being realistic id try 4-8kB of serial hex. The read/write would be less than 150 ticks and the lag may possible be bearable. 4kB of that would tots work, idk about the 8 though. Anyways have fun.   also yeah have private storage, use an xor cipher, that'd be neat af

Oh ye of little faith, this memory will take up a large area, which will make it totally possible . As for speed, as I mentioned above, I've completely guessed as I'm still working on the small scale model. I've chosen not to use hex because of much, much larger access times and bussing delay/syncing issues. I am implementing private storage.

(09-07-2017, 01:01 PM)Zijkhal Wrote: [ -> ]Isn't this type of memory sort of not working 'cause of the weird way redstone signals spread? Or having a redstone wire between the repeaters fixes it?

The way I belive it can be solved is by having a subtract comparator between the repeaters side powered by a repeater. That system can achieve 10 blocks per bit if the control line is shared.
Also, isnt that 12 blocks per bit, if I see correctly each cell is 2x3x2?

To my knowledge, chunk loaders can fix any problems associated with redstone not being handled correctly in unloaded chunks. I'm not sure your system will work, but I will definitely try it.

You are correct, it is 12 blocks per bit. Derp. Actually less than that now. Here is a picture of my progress on the small scale model.

http://imgur.com/a/bpvIP

If I haven't derped out on the math a second time, each bit takes around 24 blocks.

Did I mention I'm aiming for it to be completely torchless / pistonless?

Quote:Oh ye of little faith, this memory will take up a large area, which will make it totally possible . As for speed, as I mentioned above, I've completely guessed as I'm still working on the small scale model. I've chosen not to use hex because of much, much larger access times and bussing delay/syncing issues. I am implementing private storage.

16KB of base 2........... this will 100% work, no doubt.

If you got 16KB of writable base 2 to work with no tick skipping/data loss/corruption on any address and the lag isn't unbearable i would pay you $100 lol... you know what, if you actually do it in 3 days and meet those standards i will pay lol

with ur bussing and no decoders i calculated 2.8 mill blocks lol.... i mean its a little rough of an estimate but still ;p just having that all loaded should break shit lol