This self-initiated project stems from my daily exploration of favorite spots in Thailand, including must-visit places and restaurants. Leveraging my experience in property research within Thailand, I’ve developed a tool designed to streamline the extraction of real estate listings from social media. This tool specifically aids in gathering posts by property agents about available leasing opportunities, enhancing my ability to efficiently track and compare property options.
The Idea
The concept originated while scrolling through social media platforms like Facebook and Instagram, where I habitually save posts about intriguing restaurants. However, I noticed that retrieving specific spots when I actually need them was challenging. This sparked the idea of developing a systematic method to organize this information into a database, enabling easy querying and access whenever necessary.
The Challenge
The primary challenge lies in handling the diverse formats and messages of posts shared by users across the internet, which lack any standardized structure. Additionally, navigating the varying types of social media platforms presents another layer of complexity, each with its own way of displaying multimedia elements such as images, text, and Google Maps URLs. This variability requires a flexible and adaptive approach to data extraction and organization.
My Solutions
developed a web scraping tool using Python, which navigates through social media URLs to extract data from platforms like Facebook, Instagram, and even Google Maps URLs. To streamline the process further, I integrated the LINE API and Google Sheets API, enabling me to efficiently manage the URLs I extract from the posts.
An automation program executes my Python script to harvest the required information, which is then processed through the OpenAI API to output structured data in JSON format. This data is enriched with geolocation details using the Google Places API and subsequently stored in my MongoDB database. To enhance accessibility, the results are also pushed to my LINE or Telegram app.
For convenience, the tool is designed to automatically download images and summarize text directly to my computer, creating a comprehensive and user-friendly repository of my favorite spots.
