Common Lisp: Files and Lists

Andrew J. Pounds, Ph.D.

🎯 Learning Objectives

After completing this exercise, you will be able to:

1 Command-Line Arguments

Command-line arguments let users pass information to a program when they run it. In SBCL, we access them through sb-ext:*posix-argv*.

Seeing All Arguments

Let's start with a simple program that shows us what arguments look like:

; cmdargs.lisp
; This program demonstrates how to access command-line arguments

(defun main ()
  ;; sb-ext:*posix-argv* is a special variable that contains all
  ;; command-line arguments as a list of strings
  (let ((args sb-ext:*posix-argv*))
    
    ;; Print how many arguments we received
    (format t "Number of arguments: ~a~%" (length args))
    
    ;; Loop through each argument and print it with its index
    ;; 'for arg in args' iterates through the list
    ;; 'for i from 0' creates a counter starting at 0
    (loop for arg in args
          for i from 0
          do (format t "args[~a] = ~a~%" i arg))))

(main)

Run it with:

sbcl --script cmdargs.lisp hello world

Output:

Number of arguments: 3
args[0] = sbcl
args[1] = hello
args[2] = world
🔑 Key Point: When using --script, the first argument is always sbcl. Your actual arguments start at index 1.

Getting Just the User's Arguments

To skip the first element (sbcl) and get only what the user typed, use rest:

; user-args.lisp
; This program shows how to extract only the user's arguments

(defun get-user-args ()
  "Return only user-provided command-line arguments.
   We use REST to skip the first element (which is 'sbcl')."
  ;; REST returns all elements except the first one
  ;; So if args is (sbcl hello world), REST returns (hello world)
  (rest sb-ext:*posix-argv*))

(defun main ()
  ;; Get only the user's arguments (not 'sbcl')
  (let ((args (get-user-args)))
    
    ;; Print the arguments as a list
    (format t "Your arguments: ~a~%" args)
    
    ;; Print how many arguments the user provided
    (format t "Number of your arguments: ~a~%" (length args))))

(main)

Run with:

sbcl --script user-args.lisp apple banana cherry

Output:

Your arguments: (apple banana cherry)
Number of your arguments: 3
💡 Tip: The function (rest list) returns everything after the first element. It's the same as (cdr list). Think of it as "skip the first one, give me the rest."

2 Reading from Files

Common Lisp uses with-open-file to safely open and read files. This automatically closes the file when you're done.

Reading Line by Line

Here's how to read a file one line at a time:

; read-file.lisp
; This program reads a file and prints each line

(defun get-user-args ()
  "Return only user-provided command-line arguments."
  (rest sb-ext:*posix-argv*))

(defun main ()
  ;; Get the command-line arguments from the user
  (let ((args (get-user-args)))
    
    ;; Check if user provided a filename
    ;; NULL returns true if the list is empty
    (when (null args)
      (format t "Usage: sbcl --script read-file.lisp <filename>~%")
      ;; RETURN-FROM exits the function early
      (return-from main))
    
    ;; Get the filename (first argument the user provided)
    (let ((filename (first args)))
      
      ;; Check if file exists using PROBE-FILE
      ;; PROBE-FILE returns nil if file doesn't exist
      ;; UNLESS executes its body when the condition is false (file doesn't exist)
      (unless (probe-file filename)
        (format t "Error: File '~a' not found.~%" filename)
        (return-from main))
      
      ;; Open and read the file
      ;; WITH-OPEN-FILE automatically closes the file when done
      ;; :direction :input means we're reading (not writing)
      (with-open-file (stream filename :direction :input)
        
        ;; Read the file line by line
        ;; READ-LINE returns one line at a time
        ;; The two NIL arguments mean: don't error at end-of-file, return NIL instead
        ;; WHILE LINE stops when READ-LINE returns NIL (end of file)
        (loop for line = (read-line stream nil nil)
              while line
              do (format t "~a~%" line))))))

(main)

Create a test file called sample.txt:

Hello World
This is line two
And this is line three

Run with:

sbcl --script read-file.lisp sample.txt
⚠️ Note: The (read-line stream nil nil) call returns nil when it reaches the end of the file. The while line stops the loop when that happens.

Collecting Lines into a List

Often you want to store all lines for later processing:

; read-lines.lisp
; This program reads all lines from a file into a list

(defun get-user-args ()
  "Return only user-provided command-line arguments."
  (rest sb-ext:*posix-argv*))

(defun read-file-lines (filename)
  "Read all lines from a file into a list.
   Returns a list where each element is one line from the file."
  (with-open-file (stream filename :direction :input)
    ;; LOOP with COLLECT builds a list of results
    ;; Each line that is read gets added to the result list
    (loop for line = (read-line stream nil nil)
          while line
          collect line)))  ;; COLLECT adds each line to a list

(defun main ()
  (let ((args (get-user-args)))
    
    ;; Check for required filename argument
    (when (null args)
      (format t "Usage: sbcl --script read-lines.lisp <filename>~%")
      (return-from main))
    
    (let ((filename (first args)))
      
      ;; Verify file exists before trying to read it
      (unless (probe-file filename)
        (format t "Error: File not found.~%")
        (return-from main))
      
      ;; Read all lines into a list
      (let ((lines (read-file-lines filename)))
        
        ;; Now we can work with the lines as a list
        (format t "Total lines: ~a~%~%" (length lines))
        
        ;; Print each line with its line number
        ;; 'from 1' starts counting at 1 (more human-friendly)
        (loop for line in lines
              for i from 1
              do (format t "Line ~a: ~a~%" i line))))))

(main)
💡 Tip: The collect keyword in a loop builds up a list of results automatically. It's very useful for transforming data.

3 Splitting Strings into Words

Tokenization means breaking a string into smaller pieces (tokens). Unlike some languages, Common Lisp doesn't have a built-in split function, so we'll write a simple one.

A Simple Split Function

Here's a straightforward way to split a string on spaces:

; split.lisp
; This program demonstrates how to split a string into words

(defun split-on-spaces (string)
  "Split a string on spaces into a list of words.
   Example: 'hello world' becomes ('hello' 'world')"
  
  ;; We'll build up our list of words
  (let ((words '())           ;; Empty list to store found words
        (current-word "")     ;; String to build current word character by character
        (len (length string))) ;; Length of input string
    
    ;; Loop through each character in the string by index
    (loop for i from 0 below len
          ;; Get the character at position i
          for char = (char string i)
          do 
            ;; Check if this character is a space
            ;; CHAR= compares characters (like = for numbers)
            ;; #\Space is the space character literal
            (if (char= char #\Space)
                
                ;; We found a space - save current word if it's not empty
                (when (> (length current-word) 0)
                  ;; PUSH adds to the FRONT of the list
                  (push current-word words)
                  ;; Reset current-word to start building next word
                  (setf current-word ""))
                
                ;; Not a space - add this character to current word
                ;; CONCATENATE joins strings together
                ;; (STRING char) converts the character to a one-character string
                (setf current-word 
                      (concatenate 'string current-word (string char)))))
    
    ;; Don't forget the last word (no space after it)
    (when (> (length current-word) 0)
      (push current-word words))
    
    ;; PUSH adds to front, so list is reversed
    ;; NREVERSE puts it back in original order
    (nreverse words)))

(defun main ()
  ;; LET* is like LET, but each binding can use previous ones
  (let* ((sentence "The quick brown fox jumps")
         (words (split-on-spaces sentence)))
    
    ;; Display the original sentence
    (format t "Original: ~a~%" sentence)
    
    ;; Show how many words we found
    (format t "Number of words: ~a~%" (length words))
    
    ;; Show the words as a list
    (format t "Words: ~a~%" words)
    
    ;; Print each word individually with its index
    (format t "~%Each word:~%")
    (loop for word in words
          for i from 0
          do (format t "  [~a] = ~a~%" i word))))

(main)

Output:

Original: The quick brown fox jumps
Number of words: 5
Words: (The quick brown fox jumps)

Each word:
  [0] = The
  [1] = quick
  [2] = brown
  [3] = fox
  [4] = jumps
🔑 Key Point: push adds items to the front of a list, so we use nreverse at the end to get them in the original order.

Tokenizing a File

Now let's combine file reading with tokenization:

; tokenize-file.lisp
; This program reads a file and extracts all words from it

(defun get-user-args ()
  "Return only user-provided command-line arguments."
  (rest sb-ext:*posix-argv*))

(defun split-on-spaces (string)
  "Split a string on spaces into a list of words."
  (let ((words '())
        (current-word "")
        (len (length string)))
    (loop for i from 0 below len
          for char = (char string i)
          do (if (char= char #\Space)
                 (when (> (length current-word) 0)
                   (push current-word words)
                   (setf current-word ""))
                 (setf current-word 
                       (concatenate 'string current-word (string char)))))
    (when (> (length current-word) 0)
      (push current-word words))
    (nreverse words)))

(defun main ()
  (let ((args (get-user-args)))
    
    ;; Validate command-line arguments
    (when (null args)
      (format t "Usage: sbcl --script tokenize-file.lisp <filename>~%")
      (return-from main))
    
    (let ((filename (first args)))
      
      ;; Verify file exists
      (unless (probe-file filename)
        (format t "Error: File not found.~%")
        (return-from main))
      
      ;; Read file and collect all words from all lines
      (let ((all-words '()))  ;; This will hold ALL words from the file
        
        (with-open-file (stream filename :direction :input)
          ;; Process each line
          (loop for line = (read-line stream nil nil)
                while line
                do 
                  ;; Split this line into words
                  (let ((words (split-on-spaces line)))
                    ;; Add these words to our master list
                    ;; APPEND joins two lists together
                    (setf all-words (append all-words words)))))
        
        ;; Display results
        (format t "Total words in file: ~a~%" (length all-words))
        (format t "~%First 10 words:~%")
        
        ;; Show first 10 words (or fewer if file is short)
        ;; 'below 10' stops after index 9 (10 iterations)
        (loop for word in all-words
              for i from 0 below 10
              do (format t "  [~a]: ~a~%" i word))))))

(main)

4 Working with Lists

Lists are fundamental to Lisp (the name comes from "LISt Processing"). Here are the essential operations you need to know:

Creating Lists

; Creating lists in Common Lisp

; Method 1: Use quote (') to create a literal list
; The quote prevents Lisp from trying to evaluate the contents
(setf numbers '(1 2 3 4 5))

; Method 2: Use the LIST function
; This evaluates its arguments, so you can use variables
(setf words (list "apple" "banana" "cherry"))

; Empty list - both of these are equivalent
(setf empty '())
(setf also-empty nil)  ;; NIL and '() are the same thing

Accessing Elements

Function Description Example Result
first Get first element (first '(a b c)) a
second Get second element (second '(a b c)) b
third Get third element (third '(a b c)) c
nth Get element at index n (nth 2 '(a b c)) c
rest All but first element (rest '(a b c)) (b c)
length Number of elements (length '(a b c)) 3

Building Lists

; lists-demo.lisp
; This program demonstrates common list operations

(defun main ()
  ;; Start with an empty list
  (let ((my-list '()))
    
    ;; === Adding items with PUSH ===
    ;; PUSH adds items to the FRONT of the list
    ;; This is fast because Lisp doesn't need to traverse the list
    (push "first" my-list)   ;; my-list is now ("first")
    (push "second" my-list)  ;; my-list is now ("second" "first")
    (push "third" my-list)   ;; my-list is now ("third" "second" "first")
    
    (format t "After pushes: ~a~%" my-list)
    
    ;; === Accessing elements ===
    ;; FIRST, SECOND, THIRD get elements by position
    (format t "First element: ~a~%" (first my-list))
    (format t "Second element: ~a~%" (second my-list))
    
    ;; LENGTH tells us how many elements
    (format t "Length: ~a~%" (length my-list))
    
    ;; === Iterating over a list ===
    ;; LOOP FOR ... IN ... iterates through each element
    (format t "~%All elements:~%")
    (loop for item in my-list
          for i from 0
          do (format t "  [~a] = ~a~%" i item))))

(main)

Output:

After pushes: (third second first)
First element: third
Second element: second
Length: 3

All elements:
  [0] = third
  [1] = second
  [2] = first
⚠️ Note: Notice that push adds to the front of the list, so items appear in reverse order. If you want items in the order you added them, use nreverse when you're done, or use collect in a loop instead.

📚 Quick Reference

Task Code
Get user arguments (rest sb-ext:*posix-argv*)
Check if file exists (probe-file filename)
Open file for reading (with-open-file (s file :direction :input) ...)
Read a line (read-line stream nil nil)
Create a list '(1 2 3) or (list 1 2 3)
Add to front of list (push item my-list)
Get first element (first my-list)
Get element at index n (nth n my-list)
Get list length (length my-list)
Reverse a list (nreverse my-list)
Join two lists (append list1 list2)
Convert string to number (parse-integer string)

💻 Programming Exercise: Line Counter

Write a Common Lisp program that reads a text file and reports statistics about it.

Your program should:

  1. Accept a filename as a command-line argument
  2. Display a usage message if no filename is provided
  3. Display an error if the file doesn't exist
  4. Read all lines from the file into a list
  5. Count and display:
    • The total number of lines
    • The total number of words (across all lines)
    • The longest line (and its line number)

Sample Input File (test.txt):

Hello World
This is a longer line with more words
Short
The quick brown fox jumps over the lazy dog

Expected Output:

=== File Statistics ===
File: test.txt

Total lines: 4
Total words: 19

Longest line (#4): The quick brown fox jumps over the lazy dog

📝 Starter Code

Save this as line-counter.lisp and fill in the TODO sections:

; line-counter.lisp
; Line Counter - Starter Code
; 
; This program reads a text file and calculates statistics about it.
; YOU need to complete the TODO sections.

;; ============================================================
;; HELPER FUNCTIONS (these are complete - don't modify them)
;; ============================================================

(defun get-user-args ()
  "Return only user-provided command-line arguments.
   Skips the 'sbcl' that appears as the first argument."
  (rest sb-ext:*posix-argv*))

(defun split-on-spaces (string)
  "Split a string on spaces into a list of words.
   Example: 'hello world' returns ('hello' 'world')"
  (let ((words '())
        (current-word "")
        (len (length string)))
    (loop for i from 0 below len
          for char = (char string i)
          do (if (char= char #\Space)
                 (when (> (length current-word) 0)
                   (push current-word words)
                   (setf current-word ""))
                 (setf current-word 
                       (concatenate 'string current-word (string char)))))
    (when (> (length current-word) 0)
      (push current-word words))
    (nreverse words)))

(defun read-file-lines (filename)
  "Read all lines from a file into a list.
   Returns a list where each element is one line from the file."
  (with-open-file (stream filename :direction :input)
    (loop for line = (read-line stream nil nil)
          while line
          collect line)))

(defun count-words-in-line (line)
  "Count the number of words in a single line.
   Uses split-on-spaces and counts the resulting list."
  (length (split-on-spaces line)))

;; ============================================================
;; MAIN FUNCTION (you need to complete the TODO sections)
;; ============================================================

(defun main ()
  (let ((args (get-user-args)))
    
    ;; ---------------------------------------------------------
    ;; TODO 1: Check if user provided a filename
    ;; If args is empty (null), print a usage message and return
    ;; 
    ;; Hint: Use this pattern:
    ;;   (when (null args)
    ;;     (format t "Usage: ...")
    ;;     (return-from main))
    ;; ---------------------------------------------------------
    
    
    (let ((filename (first args)))
      
      ;; ---------------------------------------------------------
      ;; TODO 2: Check if file exists using probe-file
      ;; If file doesn't exist, print an error and return
      ;;
      ;; Hint: Use this pattern:
      ;;   (unless (probe-file filename)
      ;;     (format t "Error: ...")
      ;;     (return-from main))
      ;; ---------------------------------------------------------
      
      
      ;; Read all lines into a list (this is done for you)
      (let ((lines (read-file-lines filename)))
        
        ;; ---------------------------------------------------------
        ;; TODO 3: Calculate total number of lines
        ;; Store it in a variable called total-lines
        ;;
        ;; Hint: (let ((total-lines (length lines))) ...)
        ;; ---------------------------------------------------------
        
        
        ;; ---------------------------------------------------------
        ;; TODO 4: Calculate total words across all lines
        ;; 
        ;; You need to:
        ;; 1. Create a variable to hold the total (start at 0)
        ;; 2. Loop through each line
        ;; 3. Count words in that line using count-words-in-line
        ;; 4. Add that count to your total
        ;;
        ;; Hint: Use this pattern:
        ;;   (let ((total-words 0))
        ;;     (loop for line in lines
        ;;           do (setf total-words 
        ;;                    (+ total-words (count-words-in-line line))))
        ;;     ... use total-words here ...)
        ;; ---------------------------------------------------------
        
        
        ;; ---------------------------------------------------------
        ;; TODO 5: Find the longest line and its line number
        ;;
        ;; You need to track:
        ;; - The longest line seen so far (start with empty string "")
        ;; - The length of the longest line (start with 0)
        ;; - The line number of the longest line (start with 0)
        ;;
        ;; As you loop through lines, if the current line's length
        ;; is greater than your tracked maximum, update all three.
        ;;
        ;; Hint: Use (length line) to get a line's character count
        ;; Hint: Use 'for line-num from 1' to count from 1
        ;; ---------------------------------------------------------
        
        
        ;; ---------------------------------------------------------
        ;; TODO 6: Print the results
        ;; Use format statements to display:
        ;; - The header "=== File Statistics ==="
        ;; - The filename
        ;; - Total lines
        ;; - Total words
        ;; - The longest line with its line number
        ;; ---------------------------------------------------------
        
        ))))

(main)
💡 Hints for the exercise:

Run your program with:

sbcl --script line-counter.lisp test.txt