Can I read CaLoRAify: Calorie Estimation with Visual-Text Pairing and LoRA-Driven Visual Language Models on EtoBox?

CaLoRAify: Calorie Estimation with Visual-Text Pairing and LoRA-Driven Visual Language Models by Yao, Dongyu; Yao, Keling; Zhou, Junhong; Zhang, Yinghao is a scholarly article available to read on EtoBox.

What is CaLoRAify: Calorie Estimation with Visual-Text Pairing and LoRA-Driven Visual Language Models about?

The obesity phenomenon, known as the heavy issue, is a leading cause of preventable chronic diseases worldwide. Traditional calorie estimation tools often rely on specific data formats or complex pipelines, limiting their practicality in real-world scenarios. Recently, vision-language models (VLMs) have excelled in understanding real-world contexts and enabling conversational interactions, making them ideal for downstream tasks such as ingredient analysis. However, applying VLMs to calorie estimation requires domain-specific data and alignment strategies. To this end, we curated CalData, a 330K image-text pair dataset tailored for ingredient recognition and calorie estimation, combining a large-scale recipe dataset with detailed nutritional instructions for robust vision-language training. Built upon this dataset, we present CaLoRAify, a novel VLM framework aligning ingredient recognition and calorie estimation via training with visual-text pairs. During inference, users only need a single monocular food image to estimate calories while retaining the flexibility of agent-based conversational interaction. With Low-rank Adaptation (LoRA) and Retrieve-augmented Generation (RAG) techni

Author

Yao, Dongyu; Yao, Keling; Zhou, Junhong; Zhang, Yinghao

Published

2024

Language

EN