Final Project
The goal in this final project is to deeply engage with and explore the material from the course on your own terms. Your project will have an applications component that aims to apply existing probabilistic programming languages to a problem of your choice, and an implementation component that aims to develop a new probabilistic programming system and benchmark it on some tasks of your choosing.
A successful project will demonstrate mastery of the material that goes beyond what is directly taught in the course. It will involve selecting an interesting problem, executing on the problem in a well-structured and convincing manner, and generating high-quality deliverables that convey some form of significant progress towards the initial project statement.
1 LLM Policy
The use text output by LLMs is prohibited for any written content in this project, including the final report, project proposals, project slides, and milestone report. I will consider unattributed usage of LLM text as plagiarism for these documents. The goal of the writing process is communicate your own ideas and thoughts to the instructor: you will be responsible for each word written in the project. If the instructor suspects the use of LLMs, you may be asked to meet with the instructor to clarify the intended meaning of the text in-person. If you feel inclined to use an LLM to save time or polish your prose for you, simply do not do that: turn in something incomplete or with your own typos in it, I will not take any points off as long as I can understand what you mean. This will help me give you better feedback and succeed on your project.
The usage of LLMs for other components of the project – including researching related work, learning about topics, writing and evaluating code, designing experiments, etc – is encouraged. You are free to use LLMs as you find them useful here, but you are fully responsible for its outputs.
2 General recommendations for success
I have the following general advice on how to choose and execute on a course project like this:
Treat this as an opportunity. This is a chance for you to do something ambitious and fun that you can put on a resume or talk about in an interview. Choose a project with a result you care about, try to make it ambitious but doable and fun.
Build on things you know. Pick application domains or problems you are familiar with from other contexts so that you have a good idea of what success might look like. Draw on prior experiences to pick problems you know might be interesting.
Choose a project that has incremental steps where you can start small and build towards a larger and more cohesive result. Favor incremental progress that can be measured over large end-to-end systems that are difficult to develop in pieces.
Tastefully use LLMs. LLMs let us raise our ambition on course projects dramatically, but they run the risk of automating away too much of the learning and exploration. Treat LLMs as a way to prototype ideas rather than as a way to delegate away the entire project. Remember, you are responsible for justifying every piece of the project to the instructor: too much LLM usage will likely be evident. This is great practice for the real world.
3 Topics and project scope
You should aim to choose an ambitious topic for your project that results in something new and interesting: simply using an off-the-shelf tool on a standard problem is not an interesting project. Projects should have modeling component that involves using a probabilistic programming language to solve an interesting problem, and an inference component that involves implementing a probabilistic programming language. Projects can choose to focus more or less on one particular component, but it is expected that each project do a bit of each. You are free to use an existing probabilistic programming language or build your own for your project.
3.1 Modeling component
The modeling component of the project will involve picking a domain and applying probabilistic programming languages to solve a problem in that domain. Here are some example domains that may be interesting, but you are encouraged to think of your own if you have particular problems you want to explore:
Financial modeling. Many problems in finance involve reasoning probabilistically. For instance, see this book: here.
Systems reliability analysis. Real-life systems have failure probabilities: components break, transistors fail, etc. Can you use probabilistic programs to model and reason about a realistic reliability analysis for an interesting system? We have done a bit of work on this for hardware reliability analysis, see here
There are many problems in the medical and biological domain that are inherently probabilistic, improving disease modeling, drug discovery, medical diagnosis, etc. Can you find and model an interesting problem in this domain?
Reasoning about large language models. There is exciting ongoing work on integrating probabilistic programming languages and LLMs. Here are some papers to read: here, here. Can you come up with new applications of PPLs in this domain, or extend on some of this prior work?
Scientific modeling and simulation. Many problems in science involve testing hypotheses and doing Bayesian reasoning. Can you find an interesting example?
Software verification. Many important software systems have randomness in them, including computer networks, distributed systems, differential privacy, physics simulations, etc. Can you implement an interesting randomized system in a probabilistic programming language and reason about its correctness?
Forecasting and prediction. Prediction markets like Kalshi and polymarket are definitely morally questionable, but they do give us an abundance of prediction problems with well-defined outcomes. Can we use probabilistic programs to perform better predictions, and measure the quality of those predictions against existing strategies?
3.2 Inference component
You are required as part of your project to implement a feature in a probabilistic programming language. This can involve adding a new feature or inference strategy to an existing language, or building your own language from scratch. Ideally, the inference component serves as a complement to your modeling component, enabling a new application or solving a problem you had while you were modeling some interesting problem.
In the second module of the course we will explore the design space of inference algorithms. Here I will include a few possible resources for you to explore if you want to learn more about inference:
4 Deliverables and grading
All project components will be turned in on Gradescope, and the deadlines on Gradescope are the official deadlines you should follow. The final project will have the following phases and deliverables. The rubrics for the deliverables and general descriptions are provided below.
Team formation (5%). You may work by yourself or in a team of two.
Project proposal (5%). Students will make a project proposal outlining their project goals, milestones, and deliverables. Students will meet 1–1 with the course staff to discuss their project proposals and finalize their projects.
Project milestone report (10%). You will provide a written milestone report summarizing project progress and any changes in project scope that have been encountered so far.
Project presentation (30%). You will give a presentation to the class at the end of the semester on your project.
Project report and software artifacts (50%). You will give a written report on your project describing key outcomes, alongside any software artifacts you generate for your project.
5 Project Components
Here we describe in detail the graded components of the final project and the rubrics that will be used to evaluate them.
5.1 Team formation
List team members.
Give a very brief description of what you might want your project to be. This is non-binding. List any ideas you have.
This component will be graded on a pass/no-pass basis. If a no-pass grade is given, students will be given an opportunity to submit a corrected version for full credit.
5.2 Project proposal
You will upload a PDF file to Gradescope with the following information:
Team composition. Who is on the team? If you are working in a team of two, is there a clear division of labor? If team is smaller/larger than recommended, provide a rationale.
Motivation and problem statement. Describe the problem you are trying to address, and provide a formulation of it. Provide an argument for why the problem is interesting to you or to others.
Course relevance. Explain how your project relates to the overall topics of the course. How does it build on or incorporate key course themes? How do ideas from the course contribute to the success of the project?
Measures of success. What is the ideal outcome of the project? What do you expect to show? How will you argue that you have succeeded in the aims of your project? Include a clear description of evaluation methodology that you will apply to measure your success.
Approach. What techniques will you use to achieve the goals of your project? What is the first thing you will try? What key prior ideas will you build on, and what new ideas will you need to explore? What is the programming component of your project, and what code do you aim to create?
Project milestones. Define milestones for your project. Milestones can include learning about certain topics, implementing and evaluating project components, etc. We will not penalize you if you do not achieve your specific milestone. A good milestone (1) is a clear stepping-stone on the way to the project goal; (2) takes about 1-2 weeks to complete; (3) has a clear measure of success upon being completed
Detailed timeline. Give a detailed week-by-week timeline of how you plan to complete your project in terms of our milestones.
Risks and mitigations. Explain what you see as the main risks in successfully completing your project and how you plan to mitigate those risks. What might go wrong, and how will you react if it goes wrong?
Project proposals are expected to be about 3 or 4 pages long. The proposal will be evaluated on a pass/no-pass basis; if the proposal does not pass, then students will be given an opportunity to resubmit for full credit. To pass, a proposal should answer all of the above questions clearly. Students will meet 1–1 with the instructor to discuss their project proposal.
5.3 Milestone Report
You will upload a PDF file to Gradescope with the following information:
Progress overview. What have you done so far? What is the current state of the project? To ensure best use of time, we recommend treating this question as a partial draft of the final report – ideally, if your project does not change, you can write 1-2 pages for this – section then reuse it in the final report.
Milestone progress. Did you achieve your milestone(s)? If not, why not (e.g., some part was challenging because of [insert challenge here] and hence took longer, some other task became irrelevant)? Did you do something extra that you did not mention in your proposed milestone(s)?
Replan. Based on progress so far, do you feel any need to revise your initial milestones or overall project goals? This may involve providing new milestones or a new timeline. If there are significant changes here, reach out to the instructor to meet and discuss.
If you are struggling to meet your milestones or have any issues, reach out to the instructor and attend office hours. The milestone report will be evaluated on a pass/no-pass basis; if the milestone report does not pass, then students will be given an opportunity to resubmit for full credit.
5.4 Project Report
All groups will upload to Gradescope a PDF file containing their file project report and a link to any open-source code that is produced over the course of the project.
The PDF file should contain at a minimum the following information and sections:
Problem description and motivation. Describe the problem on a high level, including some motivation for choosing it
Approach. Explain your technical approach to solving your chosen problem. Give a high-level end-to-end picture of what you did and clearly justify all design decisions. What techniques did you rely on? What related work did you build on?
Evaluation. How did you evaluate success? What justifies your chosen metrics of success? To what extent did your approach succeed or fail according to your evaluation methodology?
Artifact description. Describe your software artifact and how it works. Provide details about its design decisions and justify them. Explain how your artifact can be used to justify conclusions in the report (assume the instructor will follow these instructions to reproduce your results).
Conclusion and discussion Describe key takeaways you had and your reader should have about your project. Explain anything you did not have time to do or would like to do in the future. If relevant, provide a discussion of what you tried that did not succeed, and how it impacted the progression of your overall project.
The project report will be evaluated according to the following rubric:
Motivation quality (30%). To what extent is the problem well-motivated, and how well is this articulated? Are the arguments convincing and well-explained? Is the project well-contextualized within prior work? Is the problem appropriate in ambition and scope?
Validity of conclusions (20%). To what extent are the claims in the report validated by the presented evidence? Is the measure of success well-executed and appropriate for the choice of problem?
Technical clarity and writing quality. (30%) To what extent are appropriate methods used and clearly explained? Is the report self-contained, containing all necessary information for understand results and conclusions? Is it well-written and easy to understand?
Software artifact quality (40%). Is the artifact well-designed and implemented? Is there adequate documentation for running the artifact and reproducing results? Is the artifact well-tested and reliable software?
5.5 Project presentation
Each group will give a presentation at the end of the semester describing their project. The presentation will be approximately 20-30 minutes with 5 minutes for questions (depending on time availability) and must use slides.
The rubric for evaluating the presentation will be similar to the rubric for the final report:
Motivation quality (30%). To what extent is the problem well-motivated, and how well is this articulated? Are the arguments convincing and well-explained? Is the project well-contextualized within prior work? Is the problem appropriate in ambition and scope?
Validity of conclusions (20%). To what extent are the claims in the presentation validated by the presented evidence? Is the measure of success well-executed and appropriate for the choice of problem?
Technical clarity. (20%) To what extent are appropriate methods used and clearly explained? Is the presentation self-contained, containing all necessary information for understand results and conclusions? Is it well-written and easy to understand?
Presentation quality. (30%) Is the presentation clear and easy to follow? Are questions handled well? Are slides well-structured so that information is easy to understand?
References
Adnan Darwiche. Modeling and reasoning with Bayesian networks. 1. Cambridge university press Los Angeles, CA, 2009. |
Daphne Koller and Nir Friedman. Probabilistic graphical models. 165. MIT press Cambridge, 2009. |
Kevin P. Murphy. Probabilistic Machine Learning: Advanced Topics. MIT Press, 2023. http://probml.github.io/book2 |