Who doesn't love a good hallucination?

Books on my shelf

As part of my Little Library project to create AI catalogs from photos, I've encountered an interesting AI challenge: recognizing books based on their spines. LLMs hallucinate a lot when faced with information in a new format or a new alignment. Book spines should be easy for an LLM to read, reference, and match, but they aren't.

You really need to extract books by bounding box, and train to recognize books by matching the title and author exactly. This means labeling books to account for all the ways they can be arranged, obscured, marred by damage, and so on.

It's an excellent baseline problem for Optical Character Recognition (OCR), and it's also agent-friendly, because once you have a good name and title, you can progress to genre, and then enlist the LLM to build more meaning. Currently, I'm relying too heavily on the LLM, and it's not performing well.