In my case [...] the catalog is an XML that contains all kinds of possible products, categories and vendors and it is updated once a month. When you read this file with the Nokogiri default (DOM) parser, it creates a tree structure with all branches and leaves. It allows you to easily navigate through it via css/xpath selectors.

The only problem is that if you read the whole file into memory, it takes a significant amount of RAM. It is really ineffective to pay for a server if you need this RAM once a month. Since I don't need to navigate through the tree structure, but just replicate all the needed data into the database, the best option is to use SAX parser.

When you read very large XML files Nokogiri may explode with this message when creating the tree structure of your file:

Nokogiri::XML::XPath::SyntaxError: FATAL: Memory allocation failed: growing nodeset hit limit

A SAX parser Show archive.org snapshot could be a way to solve this problem.

dev.to

Florian Leinsinger

makandra.de

Say thanks

Last edit

Over 2 years ago

Florian Leinsinger

License

Source code in this card is licensed under the MIT License.

Async control flow in JavaScript: Promises, Microtasks, async/await

Slides for Henning's talk on Sep 21st 2017.

Understanding sync vs. async control flow

Talking to synchronous (or "blocking") API

print('script start')
html = get('/foo')
print(html)
print('s...

Delivering Carrierwave attachments to authorized users only

Preparation

To attach files to your records, you will need a new database column representing the filename of the file. To do this, add a new migration (rails g migration <name>) with the following content:

class AddAttachmentToNotes...

Posted by Florian Leinsinger to makandra dev (2021-12-02 08:10)

Nokogiri: How to parse large XML files with a SAX parser

Related cards:

Understanding sync vs. async control flow

Talking to synchronous (or "blocking") API

Summary

Preparation

Introduction