create new tag
view all tags

Technology Exploration -- Redis

In this section you will find information on our exploration of the Redis key-value store.

General Information

Evaluation Material

  • Redis is an open-source key-value store. It is lightweight and allows for rapid prototyping. It is not considered as a candidate product for building the final file store as it is mainly targeted at problems where the data set can be kept in memory.
  • We used a Redis backend with its Python driver (redis-py) for building a model of a file and directory hierarchy using the relatively limited expressiveness of a key-value store. It is worth noting that the native data structure supported by Redis as values (strings, hashes, lists, sets, sorted sets) map well the data structures of the Python language and the driver used exposes this.

Using Redis data structures for modeling a file and directory hierarchy

  • The hypothetical file store we implemented contains two types of objects: files and directories. A file is considered a opaque sequence of bytes and a directory is a container for files or other directories. There is no limitation on the number or files or directories a directory can contain. We did not implement symbolic links.
  • Our deliberately simple implementation were inspired by the way the typical Unix file system are implemented. For details refer for instance here and here.
  • Every object in the file store has an identifier: we used Redis' atomic operations for getting and incrementing integers for obtaining identifiers. This is the equivalent in a Unix file system of an i-node number. A Redis key named 'inodeCounter' is associated to an integer value.
  • A key named 'rootDirId' is used to store the identifier of the top directory of the store. Its associated value is therefore an integer. Once the store is created this value should not change.
  • The meta-data of a each file is stored under the key named
    where <id> is the integer identifier of this file. The value associated to the key is a Redis hash (the equivalent of a Python directory) with two entries: 'type' and 'size' which have in turn its associated value. The value for 'type' is in this case the string 'file' and the value for the 'size' key is an string representing the size in bytes of the stored file.
For instance, the file named '/filestore/pictures/cat.jpg', which identifier is 378 would be stored in Redis as the association:
metadata:object:378 --> { 'type' --> 'file', 'size' --> '4567891' }
We could add other metadata fields, such as owner, checksum, creation date, etc. in the same way.
  • A directory's metadata is represented in the same way as a file's metadata, but the value associated to the 'type' key is 'directory'. The value associated to the 'size' key is not present and if present is ignored.
  • The contents of an object of type 'file' is an opaque sequence of bytes, stored in Redis associated with the key
  • The contents of an object of type 'directory' is stored associated with the key 'contents:directory:<id>'. The value associated to this key is a Redis associating a name and an identifier. The name represents an entry in the directory (i.e. the name of a file or another directory contained in this directory) and the value is the identifier of this object within the store.
For instance, the directory '/filestore/pictures', containing files 'cat.jpg', 'fish.jpg' and 'dog.jpg' is represented as a Redis hash as follows:
contents:directory:45 --> { '..' --> '1', 'cat.jpg' --> '123', 'fish.jpg' --> '3897', 'dog.jpg' --> '2359' }
In addition to the files and directories contained in the directory, we store the identifier of the parent directory associated to the entry with name '..' in the contents of a given directory.
  • So, in summary, with a few conventions we can use a key-value store for keeping the necessary information for representing a hierarchical structure composed of directories and files.

-- FabioHernandez - 2011-02-25

Edit | Attach | Watch | Print version | History: r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r2 - 2011-03-01 - FabioHernandez
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2023 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback