CS303E Homework 11

Instructor: Dr. Bill Young
Due Date: Friday, November 13, 2026 at 11:59pm

Last modified

Benford's Law

Benford's Law states that in many naturally occurring collections of numbers, the leading digit is likely to be small. Benford's Law is followed most closely when the collection of numbers is reasonably large and there are many unique values in the collection.

Benford's law has been found to apply to population numbers, death rates, lengths of rivers, mathematical distributions given by some power law, and physical constants like atomic weights and specific heats. This law is now often used to detect fraud in lists of socio-economic data submitted in support of public planning decisions and as an indicator of accounting and expenses fraud.

In this programming assignment you will test Benford's law for a data set I found online; it's not a very large data set, but large enough to test the law. The data are from 2019 Population Estimates for Texas counties compared to 2010 census data. It's presented in the form of a csv input file here: 2019PopulationEstimates.csv. You should download this file into your working directory/folder.

Each data line in the file has the following nine fields:

0. FIPS,
1. county name,
2. 2010 census count,
3. population estimate from 7/1/2019,
4. population estimated from 1/1/202,
5. difference 2010 to 7/1/2019,
6. difference 2010 to 1/1/2020,
7. percent change 2010 to 7/1/2019,
8. percent change 2010 to 1/1/2020.

Here are the first few lines of the file:

# FIPS,county,cqr_census_2010_count,july1_2019_pop_est,jan1_2020_pop_est,num_chg_10_19,num_chg_10_20,pct_chg_10_19,pct_chg_10_20
001,Anderson,58458,59025,59120,567,662,1.0,1.1
003,Andrews,14786,19279,19588,4493,4802,30.4,32.5
005,Angelina,86771,90989,90669,4218,3898,4.9,4.5

Your task is to read three values from data lines in the file--census count and two population estimates (fields 2, 3, 4)--and see if Benford's Law applies to these data. Ignore other fields in each line. You'll also keep track of the number of data lines read and the count of unique values encountered.

Note that there's no reason to convert the values in these fields to numbers, and every reason not to. You're really looking at the numbers as ascii strings and asking about the first character in the string.

You'll be printing some messages to the terminal, but also be writing to a separate output file called benford.txt. What's printing to the terminal and the format of this file should match what's shown below.

Here are the steps you should implement:

1. Loop to Accept from the user the name of a file, containing the data. If no file of that name exists, print an error message, and continue in the loop.
2. Create an empty set to store unique values. Note that these are strings.
3. Create a dictionary for leading digit counts, with entries of the form [digit:count]. Note that digit here is a character, not a number. The counts should initially all be 0, for the ten possible leading digits. (In many data sets, no number will start with a '0'. That might not be true here, since we're dealing with population counts, which could be 0.)
4. Loop to read successive lines from the file using readline(); don't read the entire file at once.
5. For each line in the file do the following:
   1. If the line contains "#" anywhere, ignore it; it's a comment line.
   2. Parse the line to extract the relevant fields.
   3. Add those to the set (a value won't repeat if it's already there).
   4. Get the first digit of each and increment the count for that digit in the dictionary. Remember: it's a string, not an integer.
6. Close the data file.
7. Print to the terminal the line: "Output written to benford.txt"
8. Write all other output to an output file named benford.txt. See the examples below. Lines written to the output file are:
   1. the name of the file processed;
   2. count of data lines read;
   3. number of unique values encountered (the size of the resulting set);
   4. a table of resuls for the leading digits.

Your table of results should be formatted as shown below. I used tabs between fields in the table, but you don't have to. (Note that tabs may not properly align the columns; that's OK. You can use format if your columns don't properly align.) Display percentages with one digit after the decimal point.

Each run of the program will overwrite the file. (Hint: you may want to just print to the terminal until you get everything running, and then change to writing to a file.) Be sure to close benford.txt before quitting; otherwise, some output may not be written completely to the file.

There are two reasons why we're using a set to count the number of unique population values: to get you some practice using sets, and because an assumption of Benford's Law is that there are a large enough number of data points for the Law to reasonably apply. You are reporting the number of unique values in the dataset to show that it's large enough for Benford's Law to apply.

Note that you can't get your digit counts from the numbers stored in the set; you need the three values read from each data line. You must count lead digits even from numbers that repeat, and so only appear once in your set of unique values. For example, if 1000 appears ten times among the data, that will only count as one element of your set of unique values, but will increment the initial digit "1" ten times.

This assignment gives you practice in file manipulation (input and output), sets, and dictionaries. You must use all of these to get full credit.

Sample Runs

> python Benford.py
Enter the name of a file of census data: NoSuchFile.csv
File does not exist
> python Benford.py
Enter the name of a file of census data: 2019PopulationEstimates.csv
Output written to benford.txt
>

So how do you know if your program wrote the right thing into file benford.txt? You look| You can do that by opening the file in a text editor. Alternatively, there are commands available from your operating system to display the contents of a file to the terminal. For example, on a Linux or MacOS system, you could do the following, at the operating system command line level:

> cat benford.txt
Processing file: 2019PopulationEstimates.csv

Total lines processed: 255
Unique numbers count: 756
First digit frequency distributions:
Digit	Count	Percentage
0	0	0.0
1	223	29.2
2	137	17.9
3	117	15.3
4	57	7.5
5	67	8.8
6	41	5.4
7	42	5.5
8	42	5.5
9	39	5.1
>

Note that cat is just the Linux/MacOS command to display on the terminal the contents of a file. It was just my way of showing you what ended up in file benford.txt. I believe that type (or sometimes cat) does something similar on Windows. These commands won't work at the level of the Python interactive loop or in IDLE.

Note that fields in the output table are separated by tabs. Round percentages to one decimal place.

Turning in the Assignment:

Your file must compile and run before submission. It must also contain a header with the following format:

# File: Benford.py
# Student: 
# UT EID:
# Course Name: CS303E
# 
# Date:
# Description of Program: 

Programming Tips:

Whenever you start the Python interpreter or an IDE such as IDLE or VSCode, you're really telling the OS to start that software for you. But once you're inside Python or the IDE, you have commands that are appropriate there. It's often said that the OS and the application (Python or the IDE) are at "different levels of abstraction" and that the application is running "under" the OS.

Most often, things just work. But no doubt this semester you've run into some OS-related issues such as when Python can't find your source file, typically because you're running Python in a different directory than the one where the file resides. That's not really a Python issue; it's an OS issue. In a lot of programming languages, you'd have to exit the programming language environment and deal with the OS directly. Unlike many other programming languages, Python has built-in many commands that allow you to execute OS operations from within Python. Many of those are in the os library.

For example, from within Python you can: move to another directory, list all the files in a directory, create a new directory, delete a directory or files, etc. That's very handy because it means you can use Python to write "system programs," which really just means writing code to perform OS-level operations.

Below is some Python code that will show you your current directory and list all of the files in it.

import os
dir = os.getcwd()                    # get current working directory (cwd)
print("Directory is: ", dir )        # print it
myfiles = os.listdir()               # get list of files in cwd
for file in myfiles:                 # print them
    print( file )