Distributed File Systems

Distributed File System Architecture

a) b)

The remote access model The upload/download model (used e.g. in FTP)
2

Two Examples
NFS (Sun Network File System)
Developed by Sun for use on its UNIX-based workstations Implemented for many other systems More a model (collection of protocols) than implementation Currently used NFS version 3
NFS version 4 is almost ready

Coda
Support for continuous operations (e.g. laptop connected/ disconnected)
3

NFS

NFS Architecture

The basic NFS architecture for UNIX systems.

5

Communication
Independent of operating system, network architecture, and transport protocol Based on RPC Client ideology: make servers life as simple as possible
One RPC call per file operation
NFS version 4 supports compound procedures

Processes
Traditional client-server architecture Server can be stateless
Abandoned in NFS version 4

Simplicity (no error recovering, etc) Separate (stateful) lock manager

Naming (1)

Directories are exported and mounted Names are relevant to the client
Standardized names
8

Naming (2)

It is not allowed to re-export directories Client has to explicitly mount from the corresponding server

Synchronization Problems
a) On a single processor, when a read follows a write, the value returned by the read is the value just written. In a distributed system with caching, obsolete values may be returned.

b)

10

File Locking in NFS

Operation Lock Lockt Locku Renew Description Creates a lock for a range of bytes Test whether a conflicting lock has been granted Remove a lock from a range of bytes Renew the leas on a specified lock

NFS version 4 operations related to file locking.

11

Client Caching in NFS

NFS implements session semantics (the cached data is flushed back to the server when the client closes the file)

12

Fault Tolerance
No such issue with NFS version 3 and earlier
Stateless protocol does not require recovery

File locking & fault tolerance

A lease is issued on every lock
Lock can be removed even if the client does not crash but the network is partitioned Grace period clients reclaim their locks after the server recovers

NFS version 4 deals with faults better

13

Security

The NFS security architecture.

14

Secure RPCs

Secure RPC in NFS version 4.

15

Access Control (1)

Operation Read_data Write_data Append_data Execute List_directory Add_file Add_subdirectory Delete Delete_child Read_acl Write_acl Read_attributes Write_attributes Read_named_attrs Write_named_attrs Write_owner Synchronize Description Permission to read the data contained in a file Permission to to modify a file's data Permission to to append data to a file Permission to to execute a file Permission to to list the contents of a directory Permission to to add a new file t5o a directory Permission to to create a subdirectory to a directory Permission to to delete a file Permission to to delete a file or directory within a directory Permission to to read the ACL Permission to to write the ACL The ability to read the other basic attributes of a file Permission to to change the other basic attributes of a file Permission to to read the named attributes of a file Permission to to write the named attributes of a file Permission to to change the owner Permission to to access a file locally at the server with synchronous reads and writes

The classification of operations recognized by NFS with respect to access control.

16

Access Control (2)

Type of user Owner Group Everyone Interactive Network Dialup Batch Anonymous Authenticated Service The owner of a file The group of users associated with a file Any user of a process Any process accessing the file from an interactive terminal Any process accessing the file via the network Any process accessing the file through a dialup connection to the server Any process accessing the file as part of a batch job Anyone accessing the file without authentication Any authenticated user of a process Any system-defined service process Description

The various kinds of users and processes distinguished by NFS with respect to access control.

17

Coda

Distributed File System Architecture

a) b)

The remote access model The upload/download model (used e.g. in FTP)
19

Coda
Developed in Carnegie Melon University (CMU) in 1990s Designed to support around 10,000 workstations Aimed at high availability Available in a number of operating system (Linux) Vice file servers Virtue workstations
Every workstation runs a process called Venus (similar to an NFS client)
20

Overview of Coda (1)

The overall organization of Coda

21

Overview of Coda (2)

The internal organization of a Virtue workstation.

22

Communication
By means of RPC
Reliable and more sophisticated RPC2 Supports multicasting Supports multiple RPCs in parallel Supports other features (e.g., side effects: communication using an application-specific protocol on top of RPC)

23

Processes
Clearly distinguishes clients and servers Both are organized as a collection of concurrent threads
User-space implementation + special mechanisms dealing with blocking I/O operations

24

Naming (1)
Analogous to UNIX Files are grouped into volumes
Usually corresponds to a collection of files associated with a user Each file is contained in exactly on volume

Shared files have identical names at all clients

Under UNIX, always mounted under the /afs directory
25

Naming (2)

Clients in Coda have access to a single shared name space.

26

How it Works
When a client opens a file, an entire copy of the file is transferred to the clients machine. Several clients may open the same file for reading Transactional semantics is used to prevent writewrite conflicts:
When a client opens a file for writing, nobody else is allowed to open it for writing When a client closes the file, the file will be transferred back to the server
27

Synchronization Problem
How would transactional semantics work if a server is not reachable? All operations succeed on the clients side Version control is used When the server is again available, updates are transferred to the server in the same order as they took place at the client When a conflict occurs, the updates from the clients session are undone
Manual reconciliation
28

Client Caching (1)

Achieves scalability Higher degree of fault tolerance (clients are less dependent on the server availability) Caches may become invalid Callback promise: the server notifies the client data invalidation
The server keeps track of all clients The client can safely use the local copy until an invalidation notification comes
29

Server Replication (1)

Coda allows volumes to be replicated
Volume storage group (VSG) a group of servers keeping the same volume

A client contacts one of the active VSG in order to get a file When closing a session on an updated file, the client transfers it in parallel to all members of ASVG Conflicts may occur when the network is partitioned
30

Server Replication (2)

Two clients with different AVSG for the same replicated file Each server maintains versions of the files
31

Server Replication (3)

When the network partitioning is healed servers reintegrate their copies of the files
Comparison of versions In many cases conflict resolution may be automated in an application-dependent way In some cases users have to assist (especially, when several users changed the same part of the same file in different ways) In practice, conflicts are rare

32

Fault Tolerance
In NFS (and many other distributed file systems) a client cannot operate unless it can contact a server In Coda it can use the local copy and then rely on conflict recovery
Provided that all needed files are in the cache Coda uses a sophisticated priority mechanism to ensure that useful data is cached Hoarding: filling the cache

33

Disconnected Operation

The state-transition diagram of a Coda client with respect to a volume.

34

Security
Secure channel for authentication & privacy
Same ideas as in Kerberos, different implementation

Access control
For simplicity and scalability, access control lists for directories (not individual files) only Ability to give negative rights to punish misbehaving users faster

35

Access Control
Operation Read Write Lookup Insert Delete Administer Description Read any file in the directory Modify any file in the directory Look up the status of any file Add a new file to the directory Delete an existing file Modify the ACL of the directory

Classification of file and directory operations recognized by Coda with respect to access control.
36

Summary (1)
Issue Design goals Access model Communication Client process Server groups Mount granularity File ID scope Name space NFS Access transparency Remote RPC Thin/Fat No Directory File server Per client Coda High availability Up/Download RPC Fat Yes File system Global Global
37

Summary (2)
Sharing sem. Cache consist. Replication Fault tolerance Recovery Secure channels Access control Session write-back Minimal Reliable comm. Client-based Existing mechanisms Many operations Transactional write-back ROWA Replication and caching Reintegration NeedhamSchroeder Directory operations

38