In this lesson you fetch data from external XML and JSON APIs in Ruby, and learn several techniques for testing code that talks to the network without making your test suite slow or flaky.
Important
Work on this lesson in
buildermode.
Learning goals
- You can fetch and parse data from an external API in Ruby — XML and JSON — and give that logic a proper home in your code.
- You can decide when to call an API on demand and when to fetch periodically in the background (e.g. a daily cron job).
- You can handle the failure modes of an external API: errors, no results, ambiguous results.
- You can test code that talks to the network in several ways — real calls, stubs, request-level mocking (e.g. WebMock), recorded responses (e.g. VCR) — and explain the trade-offs.
- You can design a test suite that never talks to the network, stays fast, and doesn't burden unrelated tests with mocking, e.g. by isolating the integration behind a feature flag or a form model.
Resources
Read what's new to you, skim what's familiar, skip what you already master. Stop when you can meet the learning goals.
Your agent can also generate an overview, a tutorial or an explanation for anything here, tailored to what you already know. Just ask.
- 📄 Nokogiri: parsing an HTML/XML document Show archive.org snapshot — the tutorial for the XML exercise
- 📄 Net::HTTP Show archive.org snapshot — Ruby's built-in HTTP client; enough for these exercises
- 📄 TMDB API: getting started Show archive.org snapshot — free for non-commercial use with an API key
- 📄 whenever Show archive.org snapshot — cron jobs written in Ruby
- 📄 Faking and testing the network with WebMock — our card
- 📄 VCR: an alternative way of mocking remote APIs — our card
- 📄 Using feature flags to stabilize flaky E2E tests — our card
Exercises
XML
On the start page of your Movie DB, show the title of a random movie that is coming soon to theaters.
- There's an XML feed for this at https://www.filmjabber.com/rss/rss-upcoming.php Show archive.org snapshot
- Manually parse the feed using Nokogiri, don't use an RSS gem for the purpose of this card.
- Update it once a day (using whenever Show archive.org snapshot )
Tip
Consider the best place to put the new logic. Should it be an existing class or new classes?
JSON
Automatically retrieve the year of a movie for any new movie entered into MovieDB, using the themoviedb.org API Show archive.org snapshot . For the UI this means that there is no longer a "Year" field when creating a movie. It is automatically fetched and stored before the movie is created. When editing a movie there is a "Year" field that can be changed.
Testing: learn the techniques
Write tests for both exercises. Write multiple variants, each using a different approach:
- Just call the real API from the example (skip this variant for the "upcoming movie" exercise)
- Use plain RSpec mocks ("stubs") to replace the HTTP request to the API with scripted behavior. Can you minimize the number of lines of code that now no longer run during tests?
- Mock out the network request to the API using Webmock
- Mock out the network request to the API using VCR. What happens when you change the movie title in the test?
Also remember to test edge cases:
- API responds with an errror
- API has no results
- For year lookup: API has multiple results for a title
Talk to your mentor about the pros and cons of each approach.
Testing: find a balance
Together with your mentor, find out how to optimize your test suite using a combinations of the techniques you learned. Also consider whether form models can help here.
Your final test suite should have the following properties:
- It should not talk to the network while it runs (it's OK to talk to the network once in order to record a VCR cassette).
- It should not record an excessive number of VCR cassettes. In particular, not every test that saves a movie should record a cassette.
- It should run (approximately) as fast as it did before the changes from this card.
- Tests that don't care about the features from the XML (upcoming movie) and JSON (movie year) exercises should not require additional code to mock the network or otherwise silence impractical side effects from the changes in the card.
Implement this in your MovieDB.
Tip
Use feature flags to disable the automatic year lookup in tests. Enable the feature only for tests that actually care about the year lookup.
