CS378h: Honors Concurrency

Rust Lab: Two-Phase-Commit

The functional goal of this assignment is to use Rust to implement a simple 2PC (two-phase commit) protocol.

The pedagogical goals of this assignment are two-fold:

To gain exposure to using programming tools that leverage language-level type systems to manage concurrency and provide safety guarantees. Indeed, the primary goal of this assignment is to increase your facility with programming in this kind of idiom, as we feel this is going to be increasingly important in future systems and languages. If you've had no previous exposure to Rust, you will likely find it's type system and approach to concurrency quite restrictive, and learning it will take some time.
To gain some exposure to implementing distributed protocols like 2PC. However, since the primary goal exposure to Rust's concurrency paradigm, the project does not require an implementation of 2PC that works across processes and handles difficult cases like recovery (although you can handle them for extra credit, see below). The project is a 2PC "simulator" because you will implement 2PC in a single process using communicating threads, and the correctness properties we will check are restricted to checking for agreement between coordinator and participants that all transactions are either committed or aborted.

In this assignment, you will write a two phase commit protocol that works with a single coordinator an arbitrary number of concurrent clients (issuing requests to the coordinator) and an arbitrary number of 2PC participants. To restrict the scope of the assignment, your solution may make the following assumptions:

The coordinator does not fail. Recall from lecture that handling this requires additional mechanism such as 3-phase commit.
Participants may fail either by explicitly voting abort in phase one, or by not responding to requests. You may choose to handle failed participants by allowing them to rejoin silently or by rejecting them from future executions of the protocol. You do not need to implement a recovery protocol (although you can for extra credit).
Requests can be represented by a single operation ID. This simplification comes at no loss of generality for the 2PC protocol itself.

Inputs/Outputs

Your program should accept the command line parameters below. If you start with the skeleton code provided, the command line parameter passing and initialization of a number of important bits of infrastructure should be already taken care of for you.

.\target\debug\cs378h-2pc.exe --help

USAGE:
    cs378h-2pc.exe [OPTIONS]

FLAGS:
    -h, --help       Prints help information
    -V, --version    Prints version information

OPTIONS:
    -m [mode]                       mode--"run" runs 2pc, "check" checks logs produced by previous run
    -c [num_clients]                number of clients making requests
    -p [num_participants]           number of participants in protocol
    -r [num_requests]               number of requests made per client
    -S [success_probability_msg]    probability participants successfully sends messages
    -s [success_probability_ops]    probability participants successfully execute requests
    -v [verbose]                    produce verbose output: 0->none, 5->*most* verbose

When you run the project in "run" mode, it should produce commit logs for the participants and coordinator in ~/tmp. When you run the project in "check" mode, it will analyze the commit logs produced by the last run to determine whether your 2PC protocol handled the specified number of client requests correctly.

Using the Skeleton Code

We provide a skeleton implementation that should help get started and insulate you from some of the details of managing the rust build system. The skeleton code provided includes a specification for the cargo build tool (see below) and the simple module decomposition overviewed in the table below. The "Edits Needed?" Column indicates whether we expect you to need to write code in that file or not. The description column summarizes the module functionality and (if appropriate) the changes we expect you to make in that module.

File Edits Needed? Description

Cargo.toml no Build tool input, specifies dependences

src/main.rs yes Starts application, required changes to register/launch clients, participants, coordinator.

src/client.rs yes Implements client-side protocol and state tracking

src/participant.rs yes Implements participant-side protocol and state tracking

src/coordinator.rs yes Implements coordinator-side protocol and state tracking

src/message.rs maybe Provides message format and related enums.

src/tpcoptions.rs maybe Implements command line options and tracing initialization

src/oplog.rs no Implements a commit log API to use in coordinator and participant

src/checker.rs no Implements the correctness check for "check" mode.

File	Edits Needed?	Description
Cargo.toml	no	Build tool input, specifies dependences
src/main.rs	yes	Starts application, required changes to register/launch clients, participants, coordinator.
src/client.rs	yes	Implements client-side protocol and state tracking
src/participant.rs	yes	Implements participant-side protocol and state tracking
src/coordinator.rs	yes	Implements coordinator-side protocol and state tracking
src/message.rs	maybe	Provides message format and related enums.
src/tpcoptions.rs	maybe	Implements command line options and tracing initialization
src/oplog.rs	no	Implements a commit log API to use in coordinator and participant
src/checker.rs	no	Implements the correctness check for "check" mode.

In general, the code in the skeleton is documented to be explicit about what functionality must be implemented and where. For example, the following code fragment from participant.rs documents what the register_participants() function should do, provides specifications for each parameter, a handful of HINTS to keep in mind and an explict block comment where we expect you to need to implement something. While in most cases you have considerably more design freedom than is illustrated by this example, you should find that the skeleton provides reasonably complete documentation of what is expected in each module.

/// 
/// register_participants()
/// 
/// The coordinator needs to know about all participants. 
/// This function should create participants and use some communication 
/// primitive to ensure the coordinator and participants are aware of 
/// each other and able to exchange messages. Starting threads to run the
/// participant protocol should be deferred until after all the communication 
/// structures are created. 
/// 
/// HINT: you probably want to look at rust's mpsc::channel or crossbeam 
///       channels to set up communication. Note that communication in 2PC 
///       is duplex!
/// 
/// HINT: read the logpathbase documentation carefully.
/// 
/// <params>
///     coordinator: the coordinator!
///     n_participants: number of participants to create an register
///     logpathbase: each participant, client, and the coordinator 
///         needs to maintain its own operation and commit log. 
///         The project checker assumes a specific directory structure 
///         for files backing these logs. Concretely, participant log files 
///         will be expected to be produced in:
///            logpathbase/participant_<num>.log
///     running: atomic bool indicating whether the simulation is still running
///     success_prob: [0.0..1.0] probability that operations or sends succeed.
///
fn register_participants(
    coordinator: &mut Coordinator,
    n_participants: i32,
    logpathbase: &String,
    running: &Arc<AtomicBool>, 
    success_prob: f64) -> Vec<Participant> {

    let mut participants = vec![];
    // TODO
    // register participants with coordinator (set up communication channels and sync objects)
    // add client to the vector and return the vector.
    participants
}

OpLog Module

#[derive(Debug)]
pub struct OpLog {
    seqno: i32,
    log_arc: Arc<Mutex<HashMap<i32, message::ProtocolMessage>>>,
    path: String,
    lf: File,
}

impl OpLog {
    pub fn new(fpath: String) -> OpLog {
/// ...

Using cargo build

The skeleton code provided includes a specification for the cargo build tool (see below for more about building) to simplify getting up and running. We expect that with a functional rust installation (if it is not part of your default environment, you can find simple instructions here), building the project should be a matter of typing:

cargo build

When you build the skeleton code, you should get a functional executable right away. You will see a large number of warnings in the skeleton code, reflecting the fact that that the rust compiler does not like the unreferenced variables and modules left by removing logic from our solution to create the skeleton.

Step 1: 90/100 points

Implement the 2PC protocol, handling arbitrary success/failure of operations at participants.

What we will check: We will test your code with various numbers of concurrent clients, participants, and numbers of requests per client, by first running the project in "run" mode, and next running with the same set of parameters in "check" mode. A scenario is illustrated below where we first run with 10 participants, 4 clients issuing 10 requests each, with a 95% probability that each participants successfully executes each request (and thus votes commit in phase 1). Next, we run with the same parameters in "check" mode, causing the program to load the commit logs produced by the first run, and analyze them for correctness. Note that the checker logic is implemented in checker.rs and is provided for you. If you use the message codes and message types specified in message.rs, the checker should "just work" for you.

rossbach@moonstone> ./target/debug/cs378h-2pc -s .95 -c 4 -p 10 -r 10 -m run
CTRL-C!
coordinator:    C:26    A:14    U:0
participant_2:  C:26    A:14    U:0
participant_1:  C:26    A:14    U:0
participant_8:  C:26    A:14    U:0
participant_3:  C:26    A:14    U:0
participant_6:  C:26    A:14    U:0
participant_0:  C:26    A:14    U:0
participant_5:  C:26    A:14    U:0
participant_9:  C:26    A:14    U:0
participant_4:  C:26    A:14    U:0
participant_7:  C:26    A:14    U:0
rossbach@moonstone> ./target/debug/cs378h-2pc -s .95 -c 4 -p 10 -r 10 -m check -v 0
participant_9 OK: C:26 == 26(C-global), A:14 <= 14(A-global)
participant_4 OK: C:26 == 26(C-global), A:14 <= 14(A-global)
participant_3 OK: C:26 == 26(C-global), A:14 <= 14(A-global)
participant_1 OK: C:26 == 26(C-global), A:14 <= 14(A-global)
participant_5 OK: C:26 == 26(C-global), A:14 <= 14(A-global)
participant_6 OK: C:26 == 26(C-global), A:14 <= 14(A-global)
participant_8 OK: C:26 == 26(C-global), A:14 <= 14(A-global)
participant_0 OK: C:26 == 26(C-global), A:14 <= 14(A-global)
participant_2 OK: C:26 == 26(C-global), A:14 <= 14(A-global)
participant_7 OK: C:26 == 26(C-global), A:14 <= 14(A-global)

Step 2: 10/100 points

Extend your solution to support message send failure. This means handling runs that specify "-S" as well as "-s". The former manages send failure probability, while the latter deals only with execution of requests per participant.

Step 3: Extra Credit Options: 10 points each

Extend your solution to work across multiple processes. Rust IPC channel implementations are available, so the bulk of the extension effort here will likely be on creation and teardown of multiple processes, assuming your communication layer can easily encapsulate IPC channels.
Extend your solution to use a fault tolerant log such as commitlog.
Implement a recovery protocol. Allow a failed participant to rejoin.
Support coordinator failure.

Hints

Use the trace/info/debug macros. We have initialized stderrlog for you, so you should have a rich set of tools for instrumenting your application to simplify debugging. We have left many fragments of code that use trace!() and info!() to illustrate how to use them.
Change the default log file path. The default implementation in TPCOptions.rs puts your logs in ~/tmp. This may be just fine for you, but you may want them elsewhere, and if you are developing on a non-UNIXy system, this may fail in non-obvious ways.
Rust documentation is very good. We found The Rust Programming Language and Rust by Example to be invaluable during the development of this programming project.

Deliverables

When you submit your solution, you should include the following:

A written report of the assignment in either plain text or PDF format. Please explain your approach, present performance results, discuss any insights gained, etc.
Your source code, instructions to build it if "cargo build" does not suffice.

Please report how much time you spent on the lab.