Homework 3: Rosette

Due date: February 20, 10pm
Grading: 5% of your course grade: 3% for Part 1 (DSL), 2% for Part 2 (verification). Part 3 (synthesis) is 1% extra credit.

Rosette is a language for building automated verification and synthesis tools. To be more specific, Rosette is an extension of the Racket programming language that adds support for symbolic values (unknown variables) and compilation to SMT via symbolic execution. These extensions take care of some of the hard and tedious parts of building verifiers and synthesizers. The most common way to use Rosette is to build a domain-specific language (DSL) for your verification or synthesis problem, and then Rosette gives you a verifier and synthesizer for that DSL for free.

In this homework we'll use Rosette to implement a toy version of the Alive tool for verifying LLVM peephole optimizations that we read about in lecture recently. We'll implement a DSL in Rosette for (a subset of) Alive optimizations, and then build a verifier for that DSL. Because Rosette also gives us synthesis tools for DSLs, there is also an extra credit part to extend Alive to support synthesizing new peephole optimizations.

Prerequisites

We'll be working with Rosette in this homework, so the first step is to get Racket and then Rosette set up on your system. Start by installing Racket (at least v8.1). On a Mac, you can get it from Homebrew:

brew install --cask racket

On Linux or Windows (or Mac, if you don't use Homebrew), download and run the appropriate installer from from the Racket website. If you're on Linux, don't get Racket from your package manager—most of them have very outdated versions of Racket that won't work.

We'll need access to the raco command-line tool that comes with Racket. Try running:

raco help

from a terminal. If it works, jump down to installing Rosette below. If not, you need to get the directory you installed Racket into onto your PATH. The Beautiful Racket book has some good instructions on how to do this.

Install Rosette

To install Rosette from the command line:

raco pkg install rosette

You can test that it worked correctly by running something like:

racket -I rosette -e 1

If you just see 1 as output, you're good to go. Otherwise, something went wrong when installing Rosette—try to read back through the output of the installation and debug (or ask for help).

Choosing an IDE

Racket comes with the DrRacket IDE, which might be a good place to start, especially if you've never written Racket before. It comes with a bunch of useful features like highlighting the source of bindings (try mousing over stuff!). On a Mac, DrRacket will be in the /Applications/Racket v8.4 folder.

The Magic Racket extension for Visual Studio Code is also a pretty good option.

Set up the code

We'll be using GitHub Classroom to check out and submit this homework. Follow the GitHub Classroom URL on Canvas to create your private copy of the homework repository, and then clone that repository to your machine. For example, the repository it created for me is called hw3-jamesbornholt, so I would do:

git clone git@github.com:cs395t-svs/hw3-jamesbornholt.git
cd hw3-jamesbornholt

To make sure everything's working, run these two commands:

raco test test-lang.rkt
raco test test-verify.rkt

You should see 16 and 19 failing tests, respectively. You'll know you've finished the homework (other than the extra credit) when all these tests pass!

Part 1: DSL

Recall that Alive is a tool for verifying the correctness of LLVM peephole optimizations. Alive took as input a proposed optimization comprising three parts:

an optional precondition
a before program
an after program

An optimization is correct if, whenever the precondition is satisfied, the before and after programs return the same result. Alive also had some additional correctness criteria around definedness and undefined behavior that we are going to ignore for this homework (concretely, of the three correctness criteria in Section 3.1.2 of the paper, we are only going to consider the third).

Our optimization domain-specific language

The first step to use Rosette is to build a domain-specific language (DSL). In this case, that means building a DSL for specifying optimizations, similar to the one the Alive paper defines. Open the file lang.rkt. This file already defines the syntax for our optimization DSL. For example, here's a complete optimization (taken from test-verify.rkt; the name AddSub:1164 is just a variable name and has no significance):

(define AddSub:1164
  (optimization
   #t
   (program
    (list
     (instruction 't0  (sub (bv 0 16) 'i0))
     (instruction 'ret (add 't0 'i1))))
   (program
    (list
     (instruction 'ret (sub 'i1 'i0))))))

Let's look at the definitions in lang.rkt to break this optimization down. An optimization has three fields—a precondition, the before program, and the after program:

(struct optimization (precondition before after) #:transparent)

In our optimization, the precondition is #t, which is always true. The only other possible precondition in our DSL is a lambda function that takes as input a set of constants; more on that below.

The before program in our optimization is an instance of program, which has only one field for the instructions in the program:

(struct program (instructions) #:transparent)

The instructions field is always a list of instances of instruction. In the case of AddSub:1164, the before program has two instructions in its list, while the after program has only one instruction.

An instruction has two fields—the register it assigns to, and the expression to assign to that register:

(struct instruction (reg expr) #:transparent)

Registers are represented as symbols of the form 't0, 't1, etc. There is a special register 'ret that holds the return value of the program. The return value is how we define correctness—an optimization is correct if the before and after programs return the same value (i.e., assign the same value into the 'ret register). In the case of AddSub:1164, the first instruction stores into the register 't0, and the second stores into the register 'ret. The return value of the before program is therefore the result of the second instruction.

An expression can be one of three things:

A unary operation (unaryop operand), where unaryop is either neg or not*.
A binary operation (binop operand operand), where binop is one of add, sub, mul, shl, and*, or*, or xor*.
A comparison operation (icmp operand operand), where icmp can only be eq*

(Some of the operation names have a * on the end just to distinguish them from built-in Racket operations). In all three cases, an operand is either a register ('t0, 't1, etc. or 'ret), a primitive value (which is a bitvector, like (bv 0 16)), or a variable. Variables are the inputs to the optimization, and are divided into two classes: inputs 'i0, 'i1, etc., and constants 'c0, 'c1, etc. The correctness criteria is defined in terms of the variables: an optimization is correct if the before and after programs are equivalent for all possible values of the variables that satisfy the precondition. The only distinction between inputs and constants is that the precondition of an optimization can only refer to constants.

In the case of AddSub:1164, the first instruction of the before program evaluates the expression (sub (bv 0 16) 'i0), which subtracts the first input ('i0) from 0, and stores it in the register 't0. The second instruction evaluates the expression (add 't0 'i1), which adds the register 't0 (holding the result of the first instruction) to the second input 'i1, and stores the result in the register 'ret. In other words, the before program computes the expression (0 − i0) + i1. The after program computes the expression i1 − i0. AddSub:1164 is therefore a valid optimization: no matter the values of i0 and i1, these two expressions are equivalent.

Preconditions

So far we've only seen a precondition #t. Let's look at another example:

(define AddSub:1088
  (optimization
   #t
   (program
    (list
     (instruction 'ret (add 'i0 'c0))))
   (program
    (list
     (instruction 'ret (xor* 'i0 'c0))))))

This optimization replaces adds of the form i0 + c0 (i.e., where the right-hand side is a constant) with XORs. This is clearly not correct—for example, 1 + 1 is 2, but 1 XOR 1 is 0. However, this optimization is correct if the constant 'c0 has only its most-significant bit set—try a few examples on paper if you're not convinced of this. In other words, this optimization is correct if we add a precondition about the constant. We do this by using a lambda:

(define AddSub:1088*
  (optimization
   (lambda (c0) (bveq c0 (bv (expt 2 15) 16)))
   (program
    (list
     (instruction 'ret (add 'i0 'c0))))
   (program
    (list
     (instruction 'ret (xor* 'i0 'c0))))))

The precondition takes as input the constants referred to in the before program. The optimization is verified only with respect to values of the constants that make the precondition true. In this case, the precondition holds only if 'c0 is exactly 2¹⁵, which as a 16-bit bitvector has only the uppermost bit set.

Restrictions on the DSL

The original Alive DSL required an entire research team and a PLDI paper to build; obviously we're not going to replicate all of that in one homework! Compared to the real Alive DSL, ours has a number of significant simplifications:

There is only one type of value in our DSL—a 16 bit unsigned integer, represented using Rosette's 16-bit bitvector type. Alive, by contrast, supports arbitrary widths and both signed and unsigned integers.
Alive supports a broader set of operands to expressions. In particular, it allows instructions like:
```
and %X, C2^(C1&C2)
```
that manipulate constants inline. Our DSL does not support these styles of operands—the only possible operands are inputs 'i0, 'i1, ..., constants 'c0, 'c1, ..., primitive values (16-bit bitvectors like (bv 0 16)), and registers 't0, 't1, ... that have already been assigned earlier in the program.
Our DSL is missing a large number of operations, including most of the comparison operations (ugt, ult, etc.) and a few binary operations (udiv, ashr, etc.). There's nothing precluding us adding these, but none of our tests use them.
Our DSL does not support memory operations like alloca or getelementptr or store.
Our DSL does not support branching (the select instruction).
Our DSL does not include undefined behavior, undefined values, or poison values.
Our DSL does not include flags like nsw or nuw that specify wrap behavior on operations. We consider all operations to be unsigned, and we define overflow to have the wraparound semantics (which is the default for bitvector overflow in SMT).

If in doubt about what features you need to support, the tests in the test-lang.rkt and test-verify.rkt files are authoritative. Your solution only needs to work correctly on these tests; we won't test it on any others (but please don't do something silly like hardcoding the results!). The intention is for this fragment of Alive to be just big enough to be interesting, but small enough that it shouldn't be a herculean task to implement. If there's still any doubt, please ask.

Task: Implement the DSL semantics

Your first task is to implement the semantics of the optimization DSL. You will need to fill in the interpret-program function in lang.rkt. This function takes as input a program (an instance of the program struct) and a state, executes the program starting from that state, and returns the return value of the program (i.e., the value stored in the 'ret register).

A state is just a dictionary mapping registers, inputs, and constants to their associated values. Encoding dictionaries in Rosette is a little tricky, so lang.rkt already does so for you by providing a state struct. Instances of state implement two methods dict-ref and dict-set! to get and set values, respectively. For example, here's a brief interaction with a state:

(define s (state '()))  ; create an empty state
(dict-set! s 't0 (bv 5 16))  ; set the value of 't0 to 5
(dict-ref s 't0)  ; returns 5
(dict-ref s 't1)  ; throws an error since 't1 is not defined in the state

When implementing interpret-program, you can assume that the state you receive as input already containts values for all program inputs ('i0, 'i1, etc.) and constants ('c0, 'c1, etc.), and contains no other values.

The tests for this step are in test-lang.rkt. I strongly recommend reading those to get a feel for what's going on. Each test case includes a program, a state to run that program from, and an expected return value. You'll know you're done with this step when you can run:

raco test test-lang.rkt

and see 16 passing tests.

Here are a few hints/suggestions for how to go about implementing this function:

The values in the state might be symbolic, since later we will build a verifier that reasons about all values rather than a single concrete one. You should be able to mostly ignore this possibility, with one exception covered in the next bullet.
You will probably want to pattern match on expressions to see if they are instances of add/sub/etc. Expressions themselves might be symbolic, so you need to use Rosette's destruct operation to do this. For example:
```
(destruct expr
  [(add lhs rhs) (printf "add ~v to ~v\n" lhs rhs)]
  [(sub lhs rhs) (printf "sub ~v from ~v\n" lhs rhs)]
  [_ (error "unknown expr" expr)])
```
Do not use match for this—it will bite you in weird and hard-to-debug ways later.
Operands can be either registers/inputs/constants (which are symbols you can look up in the state) or primitive values. You can use symbol? to distinguish these two cases: (symbol? 't0) is true, while (symbol? (bv 0 16)) is false.
In lecture we talked about how Racket has for loops but you often don't want to use them. In this homework, you probably do want to use them—in particular, a for loop over the instructions in the program is an easy way to implement interpret-program.
Most of the DSL expressions correspond one-to-one with operations in Rosette's bitvector library. The one exception is eq*, which should return a 16-bit bitvector rather than a boolean. It should return (bv 1 16) if the two arguments are equal, or (bv 0 16) otherwise.
If you need some help getting started, consider first following this Rosette tutorial, which we also used in lecture. The Rosette website also contains pointers to other documentation and tutorials.
- If you need help with Racket, the Racket guide is great, and I also have some lecture notes about Racket and Rosette from my programming languages class.

Part 2: Verification

Once you have an interpreter for the DSL, we can use it to build a verifier. Open the file verify.rkt. This file contains one function, verify-optimization, which you'll need to fill in. verify-optimization takes as input a candidate optimization (an instance of the optimization struct in lang.rkt), and returns the result of verifying that optimization ((unsat) if the optimization is correct, or else a model of concrete values for which the optimization is incorrect).

Your implementation of verify-optimization needs to do roughly five things:

Construct the state on which to verify the optimization. This state should contain symbolic values for each input and constant used by the optimization.
Establish the precondition for the optimization, which might be either #t or a function you should execute on the values of the constants in the before program.
Execute both the before and after programs using interpret-program, beginning from the same state.
Assert that the return values of those two programs are equal (by comparing them using equal?).
Verify that assertion using Rosette's verify function.

verify-optimization contains a skeleton implementation to guide you to fill in these five parts. The implementation can just return the result of calling verify directly.

You'll know you're done when you can run:

raco test test-verify.rkt

and see 19 passing tests.

Here are a few hints/suggestions for this step:

First try to make it work for optimizations that don't involve a precondition, and then extend it to also work with preconditions.
- The tests for optimizations with preconditions all end with *, so if those are the only tests failing, you know you're ready to try adding preconditions.
- Preconditions are the only place where the difference between inputs and constants is relevant.
lang.rkt provides two helper functions you'll want to use: (inputs-for opt) takes an optimization and returns a list of all the inputs referenced by the programs ('i0, 'i1, etc.), while (constants-for opt) does the same but for constants ('c0, 'c1, etc.).
Use (define-symbolic* v (bitvector 16)) (note the *) to create a fresh symbolic 16-bit bitvector called v. There's no good way to make this expression use the actual name you want (like 'i0 or 'c2), but we'll ignore that for this homework—the only thing that matters is that you use the right names when storing these vs into the state dictionary.
You can use the apply function to execute a function on a list of arguments. For example, (foo 1 2 3 4) is equivalent to (apply foo (list 1 2 3 4)). This is useful if you don't know in advance how many arguments you need to apply the function to.
One common issue that can come up when writing Rosette code is that verify returns an empty (model)—this means the verification failed, but it didn't give you a counterexample. Most often this means there is an exception being thrown in your code somewhere during verification. One way to debug this is to use Rosette's error tracer, which will show you exceptions that happened during symbolic execution. Concretely, if you have a failing test, write a new Racket program test.rkt like this:
```
#lang rosette

(require "lang.rkt" "verify.rkt")
(define opt (optimization #t (program ...) (program ...)))
(verify-optimization opt)
```
and then run:
```
raco symtrace test.rkt
```
to see the exceptions generated during symbolic execution.

Part 3: Synthesis

This part is optional and only for 1% extra credit. It's quite a lot of work for only 1 point, and not as well documented or scaffolded as the other parts. Please don't attempt it unless you're comfortable with the rest of the homework. Feel free to skip it, or just read it without doing the work, if you feel like you've spent enough time learning about Rosette.

One of the big advantages of having built our toy Alive implementation in Rosette is that we can now use it to build a synthesis tool that can automatically generate optimizations for us. Building a synthesis tool in Rosette requires doing two things: first, defining a syntactic template (or sketch) that tells us the shape of the programs we want to synthesize, and second, building the actual synthesizer on top of that sketch.

Step 1: Building a sketch

Open synthesize.rkt. The first function we'll fill in here is the make-program-sketch function. This function takes as input two arguments: a length, which specifies how many instructions to use, and a list of variables the instructions can refer to (things like 'i0, 'c2, etc.). The function should return a sketch— an instance of program whose instructions are symbolic.

The sketch should be a list of instructions of the given length. The last instruction should always be an assignment to the register 'ret, while the earlier instructions should assign to the registers 't0, 't1, etc. in order. For example, if length is 1, then the sketch returns a program of one instruction, which assigns to 'ret. If the length is 2, the sketch is a program of two instructions, the first assigning to 't0 and the second to 'ret.

Here are a few hints/suggestions for this step:

You'll want to use the choose* function to create symbolic expressions. For example, (choose* (add 't0 't1) (sub 't0 't1)) is a symbolic expression that can either be an addition or a subtraction. Similarly, (choose* 'i0 'c0) is a symbolic expression that can either be the symbol 'i0 or the symbol 'c0.
- Remember that you can use apply to apply a function like choose* to a list of arguments.
- Despite what the Rosette documentation might suggest, there's no need to use anything from the rosette/lib/synthax library.
Make sure you allow subsequent instructions to reference earlier registers—if the sketch is two instructions, the first instruction can reference any of the variables, but the second instruction can also reference the register 't0 that was assigned by the first instruction.
- You can construct symbols dynamically. For example, (string->symbol (format "t~v" i)) when i is 7 returns the symbol 't7.

You can test make-program-sketch using solve to check that it can generate certain hand-written programs. For example:

(define sketch (make-program-sketch 1 (list 'i0)))
(define prog (program (list (instruction 'ret (add 'i0 'i0)))))
(define M (solve (assert (equal? sketch prog))))  ; should be a model
(evaluate sketch M)  ; should evaluate to prog

Step 2: Implement the synthesizer

Now that you have a way to make sketches, we can implement the synthesize-optimization function. This function takes as input a before program that we want to optimize and a sketch (returned from make-program-sketch) that defines the shape of the output program from the optimization. It returns a concrete program (an instance of program) if one exists, otherwise it returns #f.

Your implementation of synthesize-optimization needs to do a few things broadly similar to what verify-optimization did. The big difference is that it will call Rosette's synthesize instead of verify. You'll need to decide what assertions the synthesize query should include, and what variables it should universally quantify over. Unlike verification, we won't consider preconditions for synthesis: all synthesized optimizations should be correct with a precondition of #t.

You'll know you're done when you can run:

raco test test-synthesize.rkt

and see 8 passing tests. The test cases also run the verifier to make sure the optimizations you synthesize are in fact correct.

Here are a few hints/suggestions for this step:

As with step 1, there's no need to use anything from the rosette/lib/synthax library.
One challenge you might run into is that symbol? doesn't work on symbolic expressions. For example:
```
(define s (choose* 'i0 'c0))
(symbol? s)  ; we expect #t but evaluates to #f
```
You can work around this issue using the for/all form, which tells Rosette to evaluate the body of the form once for each possible value of a symbolic expression:
```
(for/all ([v s]) (symbol? v))  ; evaluates to #t
```
The idea here is that for each possible value of s, for/all binds that value to v and then evaluates the body; once all values have been evaluated, Rosette merges the results back together into a single symbolic expression.
Our synthesizer won't make any attempt to find better programs than the input program—any program that is equivalent to the before program and is a solution to the sketch is a valid result.
- This means our synthesized optimizations aren't necessarily optimizations. The tests try to work around this a little bit by generally using sketches that are smaller than the before programs. But we could do better if we wanted to by, for example, using a cost function to choose the best program (like Chlorophyll, which we'll read about in class).

What to submit

Submit your solutions by committing your changes in Git and pushing them to the private repository GitHub Classroom created for you in the Set up the code step. If you haven't used Git before, Chapters 1 and 2 of the Git book are a good tutorial.

The only files you should need to modify are lang.rkt and verify.rkt (and synthesize.rkt if you do the extra credit). GitHub will automatically select your most recent pushed commit before the deadline as your submission; there's no need to manually submit anything else via Canvas or GitHub.

GitHub Classroom will autograde your submission every time you push, by just running exactly the same raco test commands you've been using by hand (except for Part 3, which it won't test). If you can't complete the entire homework and so the autograder fails, don't worry—I will still be grading manually for partial credit.

CS 395T: Systems Verification and Synthesis Spring 2023