255 Parsing text with regular expressions [1d]

Regular expressions extract structured data from text. This lesson teaches you to write, read and test them, and shows where they are the right tool and where they aren't.

Important

Work on this lesson in advisor mode.

Learning goals

• You can read a regular expression and say what it matches: character classes, quantifiers and their greedy and lazy modes, anchors, groups, alternation.
• You can write patterns that extract structured pieces from text, using groups (e.g. named groups) to pull out the parts you need.
• You can apply a pattern repeatedly over a text and process every match.
• You can keep a complex pattern readable, e.g. with the /x flag, comments or nested sub-patterns.
• You can test a pattern interactively, and you know that line breaks and multiline matching behave differently across languages.
• You can explain what a ReDoS attack is and recognize a pattern that is vulnerable to it.
• You can judge when a regular expression is good enough and when a real parser is the right tool.

Resources

Read what's new to you, skim what's familiar, skip what you already master. Stop when you can meet the learning goals.

Your agent can also generate an overview, a tutorial or an explanation for anything here, tailored to what you already know. Just ask.

Learn the basics

• 📄 Ruby docs: Regexp Show archive.org snapshot — the reference: syntax, anchors, groups, quantifiers, options
• 📄 Mastering Ruby regular expressions Show archive.org snapshot — RubyGuides; the friendlier walkthrough
• ▶️ Learn regular expressions in 20 minutes Show archive.org snapshot — Web Dev Simplified; JavaScript syntax, same concepts
• 🎮 RegexOne Show archive.org snapshot — interactive lessons
• 🎮 Rubular Show archive.org snapshot and regex101 Show archive.org snapshot — try an expression against sample text; Rubular speaks Ruby, regex101 explains every token

Ruby specifics, from our cards

• 📄 Regular expressions cheat sheet
• 📄 Quantifier modes — greedy, lazy, possessive
• 📄 Ruby: using named groups
• 📄 Ruby: you can nest regular expressions
• 📄 Making your regular expressions more readable with /x
• 📄 Matching line feeds works differently in every language
• 📄 Testing regular expressions visually
• 📄 Using regular expressions in JavaScript — the differences you'll meet in the browser

ReDoS

• ▶️ Vortrag: ReDoS-Angriffe Show archive.org snapshot — internal talk (in our library), slides; how an inefficient expression becomes a denial-of-service vector
• 🎮 ReDoS checker Show archive.org snapshot — test whether an expression is vulnerable

Exercises

Find words

Write a method second_words(string) that returns the second word of every sentence in the given string.

Tip

You can generate paragraphs of random text on https://loremipsum.de/ Show archive.org snapshot .

Write a regular expression that matches a sentence, then call it multiple times.

Parse Ruby classes

Write a ClassScanner class that parses a .rb file containing a simple Ruby class:

# student.rb

class Student < Person
  attr_reader :first_name, :last_name, :disabled
  attr_accessor :credits

  def full_name
    first_name + ' ' + last_name
  end

  def active?
    !@disabled
  end

end

The ClassScanner should work like this:

# main.rb

code = File.read('student.rb')

scanner = ClassScanner.new(code)
scanner.name # => 'Student'
scanner.superclass # => 'Person'
scanner.own_methods # => [:first_name, :last_name, :disabled, :credits, :credits=, :full_name, :active?]

We're practicing regular expressions here, not implement a fully correct Ruby parser. Here are more details to scope what your implementation does and does not need to do:

• You don't need to implement the parser as a single giant regex. Instead write individual patterns for methods, accessors, etc. Call each patterns until there are no more matches left.
• You only need to parse the given string of Ruby code. You don't need to include methods from the superclass.
• You may assume that the given code contains exactly one class.
• You don't need to support namespaced classes.
• You only need to support method definition through def ... end, attr_reader and attr_accessor. You don't need to support metaprogramming.
• You may assume that keywords like attr_reader, def or end do not appear in any strings.
• You should support the following variants for the same thing:

  attr_reader :one, :two
  

  attr_reader(:one, :two)
  

  attr_reader(:one, 'two')
  

  attr_reader :one
  attr_reader :two
  

• You may assume that all arguments of attr_reader and attr_accessor sit on the same line.
• The superclass may be optional. ClassScanner#superclass should return nil in that case.

Info

In practice we would never parse Ruby code like this. For most file formats there are libraries that parse data correctly, like the parser gem Show archive.org snapshot for Ruby code or Nokogiri Show archive.org snapshot for HTML.

Regular expressions are a blunt tool that happens to be good enough much of the time. Read Parsing Html The Cthulhu Way Show archive.org snapshot for more.