CS 395T: Systems Verification and Synthesis Spring 2023

Homework 3: Rosette

Due date: February 20, 10pm
Grading: 5% of your course grade: 3% for Part 1 (DSL), 2% for Part 2 (verification). Part 3 (synthesis) is 1% extra credit.

Rosette is a language for building automated verification and synthesis tools. To be more specific, Rosette is an extension of the Racket programming language that adds support for symbolic values (unknown variables) and compilation to SMT via symbolic execution. These extensions take care of some of the hard and tedious parts of building verifiers and synthesizers. The most common way to use Rosette is to build a domain-specific language (DSL) for your verification or synthesis problem, and then Rosette gives you a verifier and synthesizer for that DSL for free.

In this homework we'll use Rosette to implement a toy version of the Alive tool for verifying LLVM peephole optimizations that we read about in lecture recently. We'll implement a DSL in Rosette for (a subset of) Alive optimizations, and then build a verifier for that DSL. Because Rosette also gives us synthesis tools for DSLs, there is also an extra credit part to extend Alive to support synthesizing new peephole optimizations.

Table of contents

Prerequisites

We'll be working with Rosette in this homework, so the first step is to get Racket and then Rosette set up on your system. Start by installing Racket (at least v8.1). On a Mac, you can get it from Homebrew:

brew install --cask racket

On Linux or Windows (or Mac, if you don't use Homebrew), download and run the appropriate installer from from the Racket website. If you're on Linux, don't get Racket from your package manager—most of them have very outdated versions of Racket that won't work.

We'll need access to the raco command-line tool that comes with Racket. Try running:

raco help

from a terminal. If it works, jump down to installing Rosette below. If not, you need to get the directory you installed Racket into onto your PATH. The Beautiful Racket book has some good instructions on how to do this.

Install Rosette

To install Rosette from the command line:

raco pkg install rosette

You can test that it worked correctly by running something like:

racket -I rosette -e 1

If you just see 1 as output, you're good to go. Otherwise, something went wrong when installing Rosette—try to read back through the output of the installation and debug (or ask for help).

Choosing an IDE

Racket comes with the DrRacket IDE, which might be a good place to start, especially if you've never written Racket before. It comes with a bunch of useful features like highlighting the source of bindings (try mousing over stuff!). On a Mac, DrRacket will be in the /Applications/Racket v8.4 folder.

The Magic Racket extension for Visual Studio Code is also a pretty good option.

Set up the code

We'll be using GitHub Classroom to check out and submit this homework. Follow the GitHub Classroom URL on Canvas to create your private copy of the homework repository, and then clone that repository to your machine. For example, the repository it created for me is called hw3-jamesbornholt, so I would do:

git clone git@github.com:cs395t-svs/hw3-jamesbornholt.git
cd hw3-jamesbornholt

To make sure everything's working, run these two commands:

raco test test-lang.rkt
raco test test-verify.rkt

You should see 16 and 19 failing tests, respectively. You'll know you've finished the homework (other than the extra credit) when all these tests pass!

Part 1: DSL

Recall that Alive is a tool for verifying the correctness of LLVM peephole optimizations. Alive took as input a proposed optimization comprising three parts:

An optimization is correct if, whenever the precondition is satisfied, the before and after programs return the same result. Alive also had some additional correctness criteria around definedness and undefined behavior that we are going to ignore for this homework (concretely, of the three correctness criteria in Section 3.1.2 of the paper, we are only going to consider the third).

Our optimization domain-specific language

The first step to use Rosette is to build a domain-specific language (DSL). In this case, that means building a DSL for specifying optimizations, similar to the one the Alive paper defines. Open the file lang.rkt. This file already defines the syntax for our optimization DSL. For example, here's a complete optimization (taken from test-verify.rkt; the name AddSub:1164 is just a variable name and has no significance):

(define AddSub:1164
  (optimization
   #t
   (program
    (list
     (instruction 't0  (sub (bv 0 16) 'i0))
     (instruction 'ret (add 't0 'i1))))
   (program
    (list
     (instruction 'ret (sub 'i1 'i0))))))

Let's look at the definitions in lang.rkt to break this optimization down. An optimization has three fields—a precondition, the before program, and the after program:

(struct optimization (precondition before after) #:transparent)

In our optimization, the precondition is #t, which is always true. The only other possible precondition in our DSL is a lambda function that takes as input a set of constants; more on that below.

The before program in our optimization is an instance of program, which has only one field for the instructions in the program:

(struct program (instructions) #:transparent)

The instructions field is always a list of instances of instruction. In the case of AddSub:1164, the before program has two instructions in its list, while the after program has only one instruction.

An instruction has two fields—the register it assigns to, and the expression to assign to that register:

(struct instruction (reg expr) #:transparent)

Registers are represented as symbols of the form 't0, 't1, etc. There is a special register 'ret that holds the return value of the program. The return value is how we define correctness—an optimization is correct if the before and after programs return the same value (i.e., assign the same value into the 'ret register). In the case of AddSub:1164, the first instruction stores into the register 't0, and the second stores into the register 'ret. The return value of the before program is therefore the result of the second instruction.

An expression can be one of three things:

(Some of the operation names have a * on the end just to distinguish them from built-in Racket operations). In all three cases, an operand is either a register ('t0, 't1, etc. or 'ret), a primitive value (which is a bitvector, like (bv 0 16)), or a variable. Variables are the inputs to the optimization, and are divided into two classes: inputs 'i0, 'i1, etc., and constants 'c0, 'c1, etc. The correctness criteria is defined in terms of the variables: an optimization is correct if the before and after programs are equivalent for all possible values of the variables that satisfy the precondition. The only distinction between inputs and constants is that the precondition of an optimization can only refer to constants.

In the case of AddSub:1164, the first instruction of the before program evaluates the expression (sub (bv 0 16) 'i0), which subtracts the first input ('i0) from 0, and stores it in the register 't0. The second instruction evaluates the expression (add 't0 'i1), which adds the register 't0 (holding the result of the first instruction) to the second input 'i1, and stores the result in the register 'ret. In other words, the before program computes the expression (0 − i0) + i1. The after program computes the expression i1i0. AddSub:1164 is therefore a valid optimization: no matter the values of i0 and i1, these two expressions are equivalent.

Preconditions

So far we've only seen a precondition #t. Let's look at another example:

(define AddSub:1088
  (optimization
   #t
   (program
    (list
     (instruction 'ret (add 'i0 'c0))))
   (program
    (list
     (instruction 'ret (xor* 'i0 'c0))))))

This optimization replaces adds of the form i0 + c0 (i.e., where the right-hand side is a constant) with XORs. This is clearly not correct—for example, 1 + 1 is 2, but 1 XOR 1 is 0. However, this optimization is correct if the constant 'c0 has only its most-significant bit set—try a few examples on paper if you're not convinced of this. In other words, this optimization is correct if we add a precondition about the constant. We do this by using a lambda:

(define AddSub:1088*
  (optimization
   (lambda (c0) (bveq c0 (bv (expt 2 15) 16)))
   (program
    (list
     (instruction 'ret (add 'i0 'c0))))
   (program
    (list
     (instruction 'ret (xor* 'i0 'c0))))))

The precondition takes as input the constants referred to in the before program. The optimization is verified only with respect to values of the constants that make the precondition true. In this case, the precondition holds only if 'c0 is exactly 215, which as a 16-bit bitvector has only the uppermost bit set.

Restrictions on the DSL

The original Alive DSL required an entire research team and a PLDI paper to build; obviously we're not going to replicate all of that in one homework! Compared to the real Alive DSL, ours has a number of significant simplifications:

If in doubt about what features you need to support, the tests in the test-lang.rkt and test-verify.rkt files are authoritative. Your solution only needs to work correctly on these tests; we won't test it on any others (but please don't do something silly like hardcoding the results!). The intention is for this fragment of Alive to be just big enough to be interesting, but small enough that it shouldn't be a herculean task to implement. If there's still any doubt, please ask.

Task: Implement the DSL semantics

Your first task is to implement the semantics of the optimization DSL. You will need to fill in the interpret-program function in lang.rkt. This function takes as input a program (an instance of the program struct) and a state, executes the program starting from that state, and returns the return value of the program (i.e., the value stored in the 'ret register).

A state is just a dictionary mapping registers, inputs, and constants to their associated values. Encoding dictionaries in Rosette is a little tricky, so lang.rkt already does so for you by providing a state struct. Instances of state implement two methods dict-ref and dict-set! to get and set values, respectively. For example, here's a brief interaction with a state:

(define s (state '()))  ; create an empty state
(dict-set! s 't0 (bv 5 16))  ; set the value of 't0 to 5
(dict-ref s 't0)  ; returns 5
(dict-ref s 't1)  ; throws an error since 't1 is not defined in the state

When implementing interpret-program, you can assume that the state you receive as input already containts values for all program inputs ('i0, 'i1, etc.) and constants ('c0, 'c1, etc.), and contains no other values.

The tests for this step are in test-lang.rkt. I strongly recommend reading those to get a feel for what's going on. Each test case includes a program, a state to run that program from, and an expected return value. You'll know you're done with this step when you can run:

raco test test-lang.rkt

and see 16 passing tests.

Here are a few hints/suggestions for how to go about implementing this function:

Part 2: Verification

Once you have an interpreter for the DSL, we can use it to build a verifier. Open the file verify.rkt. This file contains one function, verify-optimization, which you'll need to fill in. verify-optimization takes as input a candidate optimization (an instance of the optimization struct in lang.rkt), and returns the result of verifying that optimization ((unsat) if the optimization is correct, or else a model of concrete values for which the optimization is incorrect).

Your implementation of verify-optimization needs to do roughly five things:

  1. Construct the state on which to verify the optimization. This state should contain symbolic values for each input and constant used by the optimization.
  2. Establish the precondition for the optimization, which might be either #t or a function you should execute on the values of the constants in the before program.
  3. Execute both the before and after programs using interpret-program, beginning from the same state.
  4. Assert that the return values of those two programs are equal (by comparing them using equal?).
  5. Verify that assertion using Rosette's verify function.

verify-optimization contains a skeleton implementation to guide you to fill in these five parts. The implementation can just return the result of calling verify directly.

You'll know you're done when you can run:

raco test test-verify.rkt

and see 19 passing tests.

Here are a few hints/suggestions for this step:

Part 3: Synthesis

This part is optional and only for 1% extra credit. It's quite a lot of work for only 1 point, and not as well documented or scaffolded as the other parts. Please don't attempt it unless you're comfortable with the rest of the homework. Feel free to skip it, or just read it without doing the work, if you feel like you've spent enough time learning about Rosette.

One of the big advantages of having built our toy Alive implementation in Rosette is that we can now use it to build a synthesis tool that can automatically generate optimizations for us. Building a synthesis tool in Rosette requires doing two things: first, defining a syntactic template (or sketch) that tells us the shape of the programs we want to synthesize, and second, building the actual synthesizer on top of that sketch.

Step 1: Building a sketch

Open synthesize.rkt. The first function we'll fill in here is the make-program-sketch function. This function takes as input two arguments: a length, which specifies how many instructions to use, and a list of variables the instructions can refer to (things like 'i0, 'c2, etc.). The function should return a sketch— an instance of program whose instructions are symbolic.

The sketch should be a list of instructions of the given length. The last instruction should always be an assignment to the register 'ret, while the earlier instructions should assign to the registers 't0, 't1, etc. in order. For example, if length is 1, then the sketch returns a program of one instruction, which assigns to 'ret. If the length is 2, the sketch is a program of two instructions, the first assigning to 't0 and the second to 'ret.

Here are a few hints/suggestions for this step:

Step 2: Implement the synthesizer

Now that you have a way to make sketches, we can implement the synthesize-optimization function. This function takes as input a before program that we want to optimize and a sketch (returned from make-program-sketch) that defines the shape of the output program from the optimization. It returns a concrete program (an instance of program) if one exists, otherwise it returns #f.

Your implementation of synthesize-optimization needs to do a few things broadly similar to what verify-optimization did. The big difference is that it will call Rosette's synthesize instead of verify. You'll need to decide what assertions the synthesize query should include, and what variables it should universally quantify over. Unlike verification, we won't consider preconditions for synthesis: all synthesized optimizations should be correct with a precondition of #t.

You'll know you're done when you can run:

raco test test-synthesize.rkt

and see 8 passing tests. The test cases also run the verifier to make sure the optimizations you synthesize are in fact correct.

Here are a few hints/suggestions for this step:

What to submit

Submit your solutions by committing your changes in Git and pushing them to the private repository GitHub Classroom created for you in the Set up the code step. If you haven't used Git before, Chapters 1 and 2 of the Git book are a good tutorial.

The only files you should need to modify are lang.rkt and verify.rkt (and synthesize.rkt if you do the extra credit). GitHub will automatically select your most recent pushed commit before the deadline as your submission; there's no need to manually submit anything else via Canvas or GitHub.

GitHub Classroom will autograde your submission every time you push, by just running exactly the same raco test commands you've been using by hand (except for Part 3, which it won't test). If you can't complete the entire homework and so the autograder fails, don't worry—I will still be grading manually for partial credit.