1. Introduction
2. Adapting the Security Model to the Wide Area
3. A large scale security model
4. Analysis of adapted security model
5. Possible modifications to the large-scale protocols
6. Summary and Conclusion

The standard model for data sharing in PerDiS is a peer-peer based distributed shared memory. Data is shared between PerDiS Daemons (PDs) at a page-based granularity, with each page being fetched 'on demand' from the appropriate PD, as needed by the application.

Such a model is best used in an environment where there are fast and reliable network connections between PDs. For environments where network connections are slow, unreliable, or where disconnected operation may be required, the need for repeated communication with other PDs in undesirable.

The standard PerDiS security model applies security on a peer-to-peer basis. The principal requesting access to a resource and the provider of a resource are in direct contact, and exchange signatures and encrypted data to authenticate each other and ensure data security. A domain based model is used to determine the appropriate level of security to be applied - PDs in the same domain base their authentication on a shared symmetric domain key, whilst PDs in different domains use asymmetric-key based authentication. A security domain is usually determined along organisational boundaries where there is a common security administration. Domains frequently differentiate between local-area and wide-area communications as communication over wide-area networks is generally less trusted than a local private network. However, security domains are orthogonal to any LAN/WAN distinction based on network speed and reliability.

The standard PerDiS model has been expanded to operate in a large-scale environment, in particular to operate across a wide-area network, where exactly the sort of problems described above are likely to occur.

The large-scale component of PerDiS expands the basic DSM model to improve performance when components are communicating over poor network links or are operating in a disconnected mode. Whilst PDs grouped locally continue to use the standard DSM model, communication over a wide area between groups of PDs is conducted using gateways which retrieve whole clusters at a time, which are then cached and made available to the local group of PDs.

There are two issues in the provision of security for the wide area. The first is the changes needed to the security model and implementation to support wide area operation. The second is a consideration of the issues of changes to the trust model, in particular the relaxation of the peer-to-peer security to allow for gateways that are acting on behalf of other principals.

PerDiS documentation usually refers to data as being organised into clusters. The cluster is the application level unit of replication and security. However, at the implementation level, a cluster is composed of several bunches, including meta-data bunches and application data bunches. Each bunch in a cluster is accorded the protection policy of the cluster. Each bunch is composed of many pages - the actual unit of replication within PerDiS.

An application-level access to a cluster is therefore composed of accesses to several bunches, which in turn are composed of accesses to several pages. The following sections discuss the wide-area protocol in terms of pages and bunches - but the same principles can be applied to the cluster as viewed at the application level.

Under the standard PerDiS DSM scheme, when a PD wishes to retrieve data from a cluster, it directly contacts the home site of the cluster as specified in the URL. PerDiS employs a load balancing strategy such that data and the locks associated with it migrate when modified. Hence when the home site is contacted two things may happen. If the data has not migrated, the home site may enter into a negotiation to transfer the data to the requester. If the data has migrated, the home site forwards the request to the PD the data migrated to. Because the data may have migrated several times, the request may be forwarded through several PDs composing the forwarding chain. The request will eventually be forwarded to the current holder of the data and lock, and this PD will contact the original requesting PD to negotiate a data transfer.

Because the peer holding the data cannot be determined in advance, a negotiation occurs once the holder of the data is contacted. As the granularity for such data exchanges is usually that of one page (64K), it is likely that a number of such data requests and returns will occur between the pair of PDs for each cluster accessed. To enhance security performance, a number of measures are taken to reduce the time spent in the security negotiation for subsequent accesses. The 'lazy-crypto' protocol initially signs all requests with a symmetric domain key, shared by PDs within a local domain. If the request is dealt with by a PD outside of this domain, a negotiation occurs whereby each side authenticates the other using their public keys, and then establish a shared session key for future accesses by the same principal to the same cluster.

The security token that accompanies a request specifies the principal making the request, the principal's credentials, and the identity of the PD issuing the request on behalf of the principal. As the PDs are in direct contact with each other, they can guarantee the correct application of security protocols.

Under the standard model, a single network configuration file maps between DNS hostnames (as specified in the cluster URLs) and PerDiS site ids. The wide-area model expands this to also specify a local gateway for each site. An example is given below:

The first time a ULL requests a bunch from its local PD, the PD checks to see if it needs to pre-fetch the bunch. If the gateway for the local PD and the gateway for the home site of the bunch are the same, the bunch is presumed to be available locally, and no pre-fetch occurs. If the two PDs have different gateways, then the data is presumed to be across a wide-area connection, and the local gateway attempts to pre-fetch the whole bunch.

For example, if a ULL attached to PERDIS-PC (S1) wants to retrieve data from a bunch with home site S3, both have a gateway of QMW-GATEWAY, so the access is local. If a ULL on S1 wants to access data on S7, S1's gateway is QMW-GATEWAY and S2's gateway is INESC-GATEWAY, so the access is wide-area.

In the following discussion, the naming terminology is as follows:

• Local PD - the PD to which the ULL requesting data is attached
• Local gateway - the PD acting as a gateway with which the local PD communicates
• Home-site PD - the PD which is the home site for the cluster, outside of the local group
• Home-site gateway - the PD acting as a gateway for the remote group of PDs of which the home-site PD is a member.

When the ULL first tries to open a bunch on its local PD, the local PD checks if the home site for the bunch is a local or remote PD. If it's local, nothing is done. If it's remote, the local PD issues a pre-fetch command to the local gateway.

The local gateway receives the pre-fetch command, and attempts to retrieve a copy of the whole bunch from the home-site PD. It does this via the home-site PD's gateway. It looks up the gateway for the home-site PD in the configuration file, and then makes a request to the home-site gateway. The cache only retrieves a complete copy of the bunch (if available) from the home-site.

The home-site gateway accepts the request, and contacts the storage service at the home-site PD. If the version of the bunch held at the local gateway is the same as the version available from the home site, the home site gateway informs the local gateway that its cached copy is still valid. Otherwise, the home-site gateway retrieves the whole bunch, and returns it to the local gateway, and the local gateway stores the bunch in its local file store.

All future references to the bunch by the local PD are translated into references to the local gateway, which acts as a proxy home site for the bunch.

The section above describes how data is transferred from each home site to the local site at the start of a transaction. We also need to consider what happens when a transaction commits. If the transaction is read-only, then no data needs to be returned to the home sites. However, if the transaction has modified the data, the changes need to be applied to the coherent version of the bunch at the home sites.

At commit time, the local PD hosting the transaction acts as a co-ordinator of a two phase commit (2PC) protocol where each remote home site is a participant. The local site sends the updates to the bunches in a prepare message sent to the home sites. As part of the distributed transaction protocol, each participating home site validates the transaction. If validation succeeds at all participants, the transaction is committed by the co-ordinator sending a commit message to each participant. If one or more participants fail to commit, they each send back to the co-ordinator a suggestion for how the transaction may be renumbered in order to succeed. The co-ordinator uses the suggestions to renumber the transaction, sends out a renumber message to check the renumbering has been successful, and if so, commits. Otherwise the transaction aborts.

If a transaction it committed successfully the transaction/storage manager at the home site can then update its local copy of the bunch, and updates its local file cache using the putFile operation.

Note that at commit time there is direct communication between the participants. The home site gateways are updated explicitly once the transaction has committed, invalidating the cached copy of data if necessary.

There are three components in the large-scale model: the pre-fetching of complete bunches, the proxy serving of bunches by the home gateway, and the wide-area transaction model.

The most important thing to note is that for the local PD all its references to the bunch are made using the standard DSM protocol. All the requests for data it makes on the bunch - will be answered by the local gateway or another local PD. Hence we can continue to use the existing security protocol. The only modification necessary is that the PD should accept as authorised senders of data:

• The home site for the cluster
• A PD that can demonstrate the bunch is offered on the authority of a principal who has write access to the bunch.
• The local gateway

The WAN security operates independently of the standard security model and DSM mechanisms.

The following section describes the protocol in detail.

In the figure above:

• []_P is a signature based on Principal P's private key.
• []_DL is a signature based on the shared domain key for the local domain
• []_SG is a signature based on the session key shared between the two gateways
• []_DH is a signature based on the shared domain key for the home-site domain
• []_H is a signature based on the home site's private key
• {}_SG is a message encrypted with the session key shared between the two gateways.
• REQ is the original request for a page in a bunch
• REP is the complete bunch
• REP' is a page from within the bunch

1. ULL running on behalf of principal P requests a page from a bunch. When the local PD issues a pre-fetch command, it creates a security token, signed with the private key of principal P.

The PD signs into the token the principal's identity, the bunch identifier, a nonce, the PD's site id, and the site id of the local gateway.

2. The request is sent to the local gateway. The request is transmitted over the network, and so is signed in the local domain key. When the gateway receives the request it verifies the signature. The request and token is then re-sent to the home-site gateway.
3. As the request is sent over the network, it is again signed. If the gateways are in the same security domain the domain key will be used again, otherwise the session key negotiated between the gateways for protecting all communication is used. The home-site gateway verifies that the signature is valid. It also checks that the gateway specified in the token and the gateway sending the token match. If the site identifiers do not match the request is rejected as a masquerade attack.
4. The home-site PD receives the request, signed by the home-site gateway in the home-site domain key. The signature is verified. The home-site then verifies the signature authenticates the principal specified in the token, and then performs an access control check to ascertain that the principal does have sufficient security credentials to access the bunch.
5. The whole bunch is returned to the home-site gateway. An authority token is also issued, specifying the identity of the home-site, the bunch identifier, the matching nonce, and the checksum for the bunch. The token is signed with the home-site's private key.
6. The home-site gateway receives the bunch and authority token, signed in the home-site domain key. The signature is validated. The reply and token are then returned to the local gateway, encrypted using the key for the connection between the gateways.
7. The local gateway decrypts the reply and validates the authority token. If the token is signed by the principal specified in the token, and the principal matches the home-site specified in the URL of the bunch, the whole bunch is stored in the local filestore.
8. A single page of the bunch is now returned to the local PD in response to the request. The page is signed using the local domain key. The local PD trusts the local gateway to provide the correct data and to have authenticated the data source appropriately.

The home site is notified of updates to a bunch by the PD hosting the transaction that has made the update sending a prepare message. This means that both participants in the activity have the required security credentials - the PD sending the prepare, commit and renumber messages has the security credentials for the principal running the transaction, and the home site can demonstrate that it is the correct recipient of the data.

A typical commit for a transaction modifying two bunches from two different home sites would therefore need to communicate and co-ordinate between two sites.

The local PD where the transaction is running would then for each home site involved:

1. Create a prepare message listing the read and write sets of the transaction, and a set of changes to the bunch
2. Send the prepare message to the home site.
3. Receive a status message back from the home site, either accepting the update, or suggesting a new global start number

If all the status messages report a success, the local PD would then issue a commit message to each home site.

If a home site reports a failure, it also suggests a new global start number. The local site co-ordinates these suggestions and then if possible issues a renumber message to all the home sites. If success is returned from all the home sites, a commit message is issued to all home sites, otherwise the transaction aborts. The figure below shows that a local PD may communicate with several PDs at each stage of the transaction commit:

There are thus four message types which must be protected - prepare, status, renumber and commit. A suitable security protocol for communication between the local site and each of the home sites is as follows:

[]_P is a message signed with principal P's private key

[]_H is a message signed in the home site's private key.

{}_{C_CONNECTION} is a message encrypted in the connection session key

{}_SESSION is a message encrypted in the established session key

[]_SESSION is a message signed in the established session key

1. The prepare message is formed, along with a generated session key encrypted in the home site's public key. The whole message is signed using the principal's private key. The home site authenticates the principal P's signature on the prepare message and extracts the session key.
2. The local site sends a set of updates to the home site, signed in the newly established session key. The messages will be encrypted at the communication level using the session key established between the two PDs.
3. The home site attempts to perform the modifications required. A status message reporting success or suggesting a new gsn is signed with the session key.
4. The local site validates the signature on the status message.

[Optionally] If the status reports a failure, the local site generates a new gsn and issues a renumber message, signed in the session key.

5. [Optionally] The home site returns a status reporting if the renumber was successful or not, signed in the session key
6. If all home sites accept the update, the local site issues a commit message, signed with the session key.

As there is a direct connection between the two PDs involved in committing the transaction a standard secured channel will be in place. This channel will have a session key established at connection time, authenticating the two PDs to each other and allowing the encryption of data sent between the two PDs to avoid eavesdropping and tampering.

The local PD has already authenticated the identity of the home site as the session key is established on a peer to peer basis, but we must sign the prepare message with the principal's private key in order to authenticate the principal to the home site. At this point we also establish a session key in order to allow future signings to be performed using symmetric cryptography for efficiency. All future communication between the each pair of participants for the transaction is signed with this domain key, protecting the participant from spoofed action messages.

The following sections analyse the adapted security model, describing the modifications made to the trust model and guarantees provided by the security which occur under the new model.

The use of pre-fetching by gateways alters the level of security that can be guaranteed. In the standard model, in the inter-domain case, there is direct communication between the provider of the data and the user of the data. This allows for authentication of identity by both participants of each other, the verification of the requester's right to access the data, and the verification of the data provider to supply the data. This guarantees that if the security protocols are followed properly, the data provider only ever supplies data to a principal authorised to obtain it, and a principal requesting data only ever receives data from a principal authorised to supply it.

This guarantee does have some caveats built into it. Firstly, once a data provider has released a copy of the data to another principal, it trusts that principal to apply the same security controls as it would itself. The PerDiS security model is built on the notion of trust between pairs of principals, which is why all security in the inter-domain configuration is based on pair-wise cryptographic protocols. Secondly, once a principal has been given a copy of data they may modify, they are trusted to handle the data appropriately. Because the PerDiS data model is page based (and does not apply type-based constraints on data access), a set of data may be released to a principal who at the application level may only be allowed to modify some subset of the page. The principal to whom a page is released is therefore trusted to only modify the data they may change. This has a bearing on other principals' interactions with the data too. To avoid overloading the home site of a set of data, control over the data migrates to the last principal to modify the data. Once a page has been modified by a principal, they may pass it on to other principals. To do this, they must demonstrate their authority to do so - this authority stems from their right to modify the page - a principal won't accept a page from a principal without write access to a page. An abuse of the PerDiS security model might be to legally modify a page, then make illegal (at the application level) modifications, and then pass it on to another principal. This would allow a "trojan horse" approach to returning illegally modified data to the home site, under the authority of a different principal. Without resorting to a delta based method of recording changes to pages, the PerDiS security model thus trusts each principal to pass on correct copies of data.

The figure above shows what may happen. The page is released to site A where a principal modifies the two subsets of the page, as shown. The page is then sent from site A to site B, where further modifications are made. The page is then returned to the home site. Neither site B, nor the home site, can determine if the page has been modified appropriately.

The standard security model guarantees that the home site will only ever release data to a principal with the proper credentials to access the data, and the home site only accepts updates to the data from principals with proper credentials. There is an element of trust that a principal will apply proper access control once given a copy of the data, and that a principal will pass on a properly modified copy of a data page to another principal. The basis of the PerDiS security model is this trust between pairs of principals rather than between organisations or system administrators.

The large-scale security model weakens the guaranteed security, mainly because the added levels of indirection add to the number of principals who must be trusted to protect the data properly.

1. The home site receives an authority token along with the request from the local site. The security token can be authenticated as being signed by a principal on the local site. However, the home site cannot determine the path that the request and token have taken. It must trust its home gateway to behave properly, and releases the data to it.

2. The home gateway now has a copy of the data. The home site must trust the administrator of the home gateway to have configured it correctly, so that the data is only passed on to the local gateway, and not to other sites or principals.
3. The home site must trust the home gateway to send the data properly protected, i.e. to employ the appropriate encryption for transmission across the network. The home site must trust that the software is configured correctly, and that keys involved in the encryption have not been compromised.
4. The home site and the local site must both trust the local gateway to be configured correctly and to apply security correctly. This means the home site must trust the principal at the local site, and the administrator of the local gateway. The principal at the home site must trust the local gateway administrator in two ways: to protect the cached copy of the data appropriately, and also not to tamper with the contents of the data.

Point D is of primary importance to the recipient of the data. Firstly, the audit trail will record the recipient principal as having responsibility for the data sent from the home site, yet the recipient must trust his own system administrator to also handle the data properly. Secondly, from leaving the home site to receipt at the local gateway the contents of the data will be protected by a signed checksum. However, because the local site only receives the data in page-size chunks, the local site can not apply any check to ensure the correct content of the data it receives.

The distributed commit protocol has little impact on the security. At commit time, each pair of participants are in direct communication, as they would be for a local commit.

The content of updates is protected when sent over the network by the standard encryption provided by the communication layer. However, as this only authenticates the PDs to each other, rather than the principal involved in the transaction, a session key is established to sign all other messages sent from the principal to the home sites. This allows the home sites to authenticate that each message has originated from a principal with the authority to control the transaction. These signatures therefore protect the commit phase as it occurs over a network.

There is an issue of trusting the PD co-ordinating the transaction to perform the commit correctly. Each home site involved in the transaction relies on the co-ordinator to maintain consistency by only sending a commit message when appropriate. For example, the co-ordinator may send out a prepare message to several home sites, of which some will allow a commit to occur, and some which don't. If renumbering fails the correct operation would be for the co-ordinator to abort the transaction. However, the co-ordinator could issue a commit anyway, causing the transaction to be committed at some sites but not others.

It is clear that the wide-area operation of PerDiS described above reduces the strength of security, requiring greater levels of trust between PDs, and the trust of third parties to manage security correctly. This section considers possible modifications to the wide-area protocols that might improve security.

A local gateway introduces two security issues - users whose PDs operate through the gateway must trust it to protect the information it retrieves using their credentials; users must trust that the information passing through the gateway is not tampered with.

To avoid the security problems introduced, we could run a combined PD/Gateway server where the PD acts as its own gateway. Here the gateway would retrieve any wide-area data using the wide-area protocol as usual, but would only serve the data to any ULLs connected directly to the PD.

Now any data held by the local gateway is there on the authority of a principal whose ULL is using the PD. Hence there is no third party whom we must trust with the data. As the PD receives the whole bunch file, it can also authenticate that the contents have not been tampered with.

The disadvantage of this method is that there is no sharing of information between PDs at a site. The wide-area concept is intended to minimise the traffic over a wide-area link which will most likely be expensive or have low bandwidth. The current wide area protocol allows sharing of data between local PDs with the gateway acting as a central cache. Our restriction loses this advantage, as each individual PD/gateway would contact the home-site gateways directly.

However, the protocol could be adapted, so that the sharing of cached data locally occurs as a cooperative process. The notion of a single local gateway would be replaced with a collection of PDs all acting as combined PD/gateways. Before contacting the home-site gateway to obtain a bunch, a local PD/gateway would first consult all the other local gateways to see if the cluster was already resident locally. If the bunch was available, the data could be accessed from that gateway using the standard DSM protocol. The advantages would be that all the security guarantees regarding the authority of the PD to serve the data could be applied.

It may be possible to restrict home site gateways in the same way, whereby the home site PD and home site gateway are merged. This would obviously be necessary if the restrictions suggested above were applied. Such a restriction would remove a third party to whom we trusted the protection of data.

As discussed in the previous section, the home sites participating in a distributed commit rely on the co-ordinator to only issue a commit order when all sites confirm a commit is possible. The protocol could be expanded to allow communication to occur directly between all participants in the transaction, and the transaction would only commit when each home site had confirmed to every other home site involved in the transaction that the commit may go ahead.

The wide-area protocols impact security in two areas - controlling the reading of data from the store and the sharing of it through a cache, and the updating of the data in the store at transaction commit time.

The impact on the security architecture and trust model by the wide-area commit is minimal. The participants in the commit - the transaction making the commit and the home sites, are in direct communication, as they are in a local commit. Hence all the security information needed by each pair of parties to authenticate each other is available, and no other parties are introduced into the activity.

The pre-fetching of data through gateways introduces two new parties who must be included in the trust model. The home site releases a copy of a bunch to its own gateway, which it must trust to protect properly. The bunch is transferred to the local site's (where the transaction is running) gateway, where again the gateway must be trusted to protect the bunch, and to not tamper with its contents before passing the information on to the local site.

Hence previously the trust model was centred on the trust between the principal requesting the data and the site holding the data, each trusting the other to protect the information properly using correctly configured software. The wide-area protocol requires that the gateway servers at both ends are configured correctly, and that the administrators of the machines may be trusted if data integrity and secrecy is to continue.

In conclusion, users wishing to operate their servers using the wide-area protocols should be aware of the weakening of guarantees that can be made about data security once secondary parties are added to the trust model. However, for many organisations it may be fair for users to trust their local system administrators to run properly configured gateway services. If such trust cannot be given, it is possible for users to continue to use the wide-area protocols in a variety of configurations, but a trade-off must be made between the efficiencies made by the sharing of cached data, and the security than can be afforded to that data.