IBP - Internet Backplane Protocol: Infrastructure for Distributed Storage (V 0.2)

Wael R. Elwasif, James S. Plank, Micah Beck

Department of Computer Science
University of Tennessee
Technical Report CS-99-430
February 1999

[elwasif, plank, mbeck]@cs.utk.edu
http://web.eecs.utk.edu/~plank/IBP





Introduction:
    In this document, we present a description of IBP v0.2, a client-server tool for remote storage management. We present the IBP client interface along with a detailed description of the semantics involved in every client call. As of the date of this document, IBP has been developed on SUN Solaris OS, with possible ports to other OS's in the future.

The IBP client
   The current implementation of IBP supports only sunchronized client requests, all client IBP calls will block pending completion (or failure) on the server(s) size. It is envisioned that in the future this restriction could be relaxed to allow non-blocking IBP calls to be made. In what follows, we describe the calls that constitute the IBP client interface. We present the C-language prototype of every call along with a detailed description of the data structures, success behavior and error conditions involved in that call. In keeping with UNIX tradition, failure of an IBP client call is indicated by a return value of -1 , or NULL, with a special variable (IBP_errno) set to the appropriate error code.


IBP_allocate()

# include "ibp_client.h"

IBP_cap_set IBP_allocate(char *targetHost, ulong_t size, IBP_attributes attr)

   IBP_allocate() allocates a remote storage area on the host targetHost. The allocated area has a maximum possible size of size bytes, and storage attributes defined by attr.IBP_attributes is typeded to a pointer to struct ibp_attributes defined in "ibp_base.h" and which has the following format

typedef struct ibp_attributes {
        time_t duration;
        int    reliability;
        int    type;
} *IBP_attributes;

Where
      durattion specifies the time at which the allocated storage area will be automatically purged from the pool of storage areas managed by the server. Time is specified in seconds since the epoch (as returned by UNIX's date() command). A value of 0 indicates permanent status for the allocated storage area (it'll be only purged when no more clients have read access to it, otherwise it will be kept alive ccording to the reliability property)

     reliability is a flag that determines how reliable the allocated storage area will be. The current version of IBP supports two level of reliability

    type is a flag that determines the type of storage allocated. The current version of IBP supports two types of storage Return values
   Upon success, IBP_allocate() returns an IBP_cap_set object ,wich is a pointer to struct ibp_cap_set defined in "ibp_base.h", and has the following format.

typedef char* IBP_cap;
typedef struct ibp_cap_set{
      IBP_cap readCap;
      IBP_cap writeCap;
      IBP_cap manageCap;
} *IBP_cap_set;

In the current version of IBP, IBP capability type (IBP_cap) is typedefined to be a simple character string. This however could change in future versions of IBP. The capabilities included in an IBP_cap_set object allow the client read access, write access, and management access to the newly created storage area, respectively.

   Upon failure, IBP_allocate() returns a NULL pointer and sets IBP_errno to one of the following values defined in "ibp_protocol.h"


IBP_store()

# include "ibp_client.h"

int IBP_store(IBP_cap cap, char *data, ulong_t size)

   IBP_store() stores size bytes starting at data at the storage area accessed through the IBP capability cap. For this call to succeed, cap must be a writecap returned by an earlier call to IBP_allocate(), or imported from the client which made the IBP_allocate() call. IBP_store() is a blocking call that only returns when the required size of data is successfully stored at the desired storage area accessed through the IBP capability cap, or an error causes the call to abort prematurely. The call appends data to the end of any previously stored data at the storage area accessed through cap  for storage areas of type IBP_BYTEARRAY and IBP_FIFO. Data written to a storage area of type IBP_BUFFER overwrites any previous data (starting at the beginning of the buffer).

Return values
      Upon success, IBP_store() returns 0. Otherwise it returns -1 and sets IBP_errno to one of the following error codes


IBP_remote_store()

#include "ibp_client.h"

int IBP_remote_store(IBP_cap cap, char *host, ushort_t port, ulong_t size)

      IBP_remote_store() causes the IBP server that hosts the storage area accessed through the IBP capability cap to fetch size bytes from a socket connection to port on host. Data fetched is then written to the storage area accesssed through cap, in a manner similar to that described in the IBP_store() call. It is the responsibility of the client(s) to make arrangements for the specified amount of data to be served at the given port. This call is a blocking call that returns only when data transfer from the remote host to the IBP server is completed successfully, or terminated due to an error condition. For this call to succeed, cap must be a writecap returned by an earlier call to IBP_allocate(), or imported from the client which made the IBP_allocate() call.

Return values
      Upon success, IBP_remote_store() returns 0, otherwise it returns -1 and sets IBP_errno to one of the following error codes


IBP_read()

#include "ibp_client.h"

int IBP_read(IBP_cap cap, char *buf, long size, ulong_t offset)

      IBP_read() reads size bytes, starting at offset, from the storage area accessed through the IBP capability cap, into memory pointed to by buf. For storage areas of type IBP_FIFO, offset is ignored. A size value of -1 causes all currently stored data in an IBP_BYTEARRAY type storage area to be read. For storage areas of type IBP_FIFO, a size value of -1 causes a read operation for the maximum size specified in IBP_allocate() to be initiated. For this call to succeed, cap must be a readcap returned by an earlier call to IBP_allocate(), or imported from the client which made the IBP_allocate() call. IBP_read() is a blocking call that returns only when all required data is read, or the read operation is prematurely terminated due to an error.

Return values

      Upon success, IBP_read() returns 0, otherwise it returns -1 and sets IBP_errno to one of the following error codes


IBP_copy()

#include "ibp_client.h"

int IBP_copy(IBP_cap source, IBP_cap target, long size, ulong_t offset)

      IBP_copy() copies size bytes, starting at offset, from the storage area accessed through the IBP read capability source and writes them to  the storage area accessed through the IBP write capability target. For storage areas of type IBP_FIFO, offset is ignored. A size value of -1 causes all currently stored data in an IBP_BYTEARRAY type storage area accessed through source to be copied. For source storage areas of type IBP_FIFO, a size value of -1 causes a copy operation for the maximum size specified in IBP_allocate() to be initiated. As in other read operations to an IBP_FIFO  type storage area, data read from the storage area will no longer be available for future reads. For this call to succeed, source must be a readcap returned by an earlier call to IBP_allocate(), or imported from the client which made the IBP_allocate() call and target must be a writecap returned by a similar call.

   IBP_copy() is a blocking call that returns only when all required data is successfully copied from the source IBP server to the target IBP server, or the operation is prematurely terminated due to an error.

Return values

      Upon success, IBP_copy() returns 0, otherwise it returns -1 and sets IBP_errno to one of the following error codes


IBP_deliver()

#include "ibp_client.h"

int IBP_deliver(IBP_cap source, char *targetHost, ushort_t port, long size, ulong_t offset)

   IBP_deliver() delivers size bytes from the storage area accessed through the IBP capability source at offest offset to a waiting process running on host targetHostand listening on port port. For source storage areas of type IBP_FIFO, the parameter offset is ignored. A size value of -1 causes all currently stored data in an IBP_BYTEARRAY type storage area accessed through source to be delivered. For source storage areas of type IBP_FIFO, a size value of -1 causes a deliver operation for the maximum size specified in IBP_allocate() to be initiated. As in other read operations to an IBP_FIFO  type storage area, data read from the storage area will no longer be available for future reads. For this call to succeed, source must be a readcap returned by an earlier call to IBP_allocate(), or imported from the client which made the IBP_allocate() call. It is the responsibility of the calling process to ensure the existence of a recepient process on targetHost that is listening on port port (this process will be serving a socket to which the IBP server will connect to initiate the delivery operation.) IBP_deliver() is a blocking call that only returns when the required amount of data is delivered to its destination, or the process is aborted due to an error.

Return values

      Upon success, IBP_deliver() returns 0, otherwise it returns -1 and sets IBP_errno to one of the following error codes


IBP_manage()

#include "ibp_client.h"

int IBP_manage(IBP_cap cap, int cmd, int capType, IBP_status info)

   IBP_manage() allows an IBP client to perform certain management operations on an IBP storage area. Any client that can present the management capability can issue any of the management commands described below. cap is an IBP management capability that is returned in the IBP_allocate() call or imported from the client which made that call. cmd can take one of the following values (defined in the file "ibp_protocol.h")

capType determines the type of the capability affected by the two commands IBP_INCR and IBP_DECR. It can have one of two values, IBP_READCAP and IBP_WRITECAP. It is ignored for the two commands IBP_CHNG and IBP_PROBE.

info is a pointer to a truct of type struct ibp_status (typedefed to IBP_status) The struct has the following format

typedef struct ibp_status{
   int     readRefCount;
   int     writeRefCount;
   int     currentSize;
   ulong_t maxSize;
   struct ibp_attributes attrib;
} *IBP_status;

where readRefCount and writeRefCount hold the reference count for the read and write capabilities respectively (on return from an IBP_PROBE command) and are ignored for other commands. currentSize holds the current size of data stored in the underlying storage area (for storage areas of type IBP_FIFO, it holds the maximum size of the underlying storage area). maxSize holds the maximum size of the storage area, while attrib holds the storage area attributes as defined earlier.

The following table summarizes the use of different parameters with every command.
 
capType
  readRefCount     writeRefCount 
   currentSize 
   maxSize     attrib 
IBP_INCR
In
Not used
Not used
Not used
Not used
 Not used 
IBP_DECR
In
Not used
Not used
Not used
Not used
Not used
IBP_PROBE
 Not used 
Out
Out 
Out 
Out
Out
IBP_CHNG
 Not used 
Not used
Not used
Not used
In
In

Return values

      Upon success, IBP_manage() returns 0, otherwise it returns -1 and sets IBP_errno to one of the following error codes


The IBP server:

     The IBP server manages access to a pool of storage areas that are created, accessed, and managed remotely through the IBP client interface. The IBP server performs no security checks on clients' requests. A client which connects to the IBP server with the proper capability is granted access to the underlying functionality. The only "protected" operations are those performed through the IBP_manage() call. This call requires the client to present the management capability that is returned as part of the IBP_allocate() call before the server can fulfill the client's request.

    The current version of IBP supports two levels of reliable storage, with the client choosing the level of reliability of an allocated storage area.:

      In addition to the aforementioned reliability levels, IBP supports indefinite storage, where storage areas are only reclaimed through client requests or reliability-induced server actions, and time-limited storage, in which the storage area is reclaimed by the IBP server at a certain instance of time specified by a managing IBP client (a managing IBP client is a client that allocates a storage area, or acquires the management capability from the creating client.) See description of the client IBP_allocate() and IBP_manage() calls for ways of controlling properties of storage areas.

IBP server configuration

      The IBP server is configured through the configuration file "ibp.cfg", which should be located in the home directory of the user launching the IBP server. This file contains a list of the form

with one entry per line. The following table lists the currently supported configuration parameters, their names, types, and default values.
 
Parameter name
(case sensitive)
Description
Type
Default value
VOLSIZE
Size of available storage 
for volatile storage areas
(in MegaBytes).
  Integer 
0
VOLDIR
Directory where volatile
storage areas are to be 
stored (absolute path )
String
/tmp/
STABLESIZE
Size of available storage 
for stable storage areas
(in MegaBytes).
Integer
50
STABLEDIR
Directory where stable
storage areas are to be 
stored (absolute path )
String
/tmp/
HOSTNAME
FQDN to be used by clients
to connect to the IBP server
String
Use DNS to try 
retrieving this entry

Default values are used if the corresponding parameter is not specified in the "ibp.cfg" configuration file (or if the file does not exist in the home directory of the user launching the IBP server).


IBP server blocking rules

      The IBP server processes requests on a First Come First Serve basis. Due to the fact that all write operations to storage areas have append semantics, a maximum of one write operation can be actively accessing a storage area at any given time. There is no limit on the number of read processes (for storage areas of type IBP_BYTEARRAY), and an upper limit of one read process for dtorage areas of type IBP_FIFO. If a write request is recieved while another one is active to the same storage area, the new request is queued pending completion of the existing request (the same aplies to multiple read requests to storage areas of type IBP_FIFO).