Skip to content
Donné Alphonse.

Agentic AI2025

AI agent for data scraping

Agentic AI system that scrapes web content and answers questions from the collected data through semantic search.

Role
Python AI developer
Team
Solo
Duration
3 months
Status
Completed

Context

Three-month freelance engagement: automatically collect web content and answer questions from that data, using semantic search and AI agents.

Preview

  • AI agent for data scraping — screenshot 1

Constraints

  • Automated collection of web sources (SeleniumBase).
  • Contextual answers grounded in the collected data.
  • Three-month timeline, solo.

Architecture

7-step flow: Web sources, then Scraping, then Embedding, then Qdrant, then Client, then API, then AI agents. Use the arrow keys to move from one step to the next.

  1. · Pages to collect

    Web content to collect.

  2. · SeleniumBase

    Automated page collection with SeleniumBase.

  3. · Embeddings

    Chunking of collected content and embedding computation.

  4. · Semantic search

    Qdrant vector database queried by similarity.

  5. · Question

    Question asked to the system.

  6. · FastAPI

    FastAPI API exposing the agents.

  7. · LangChain · LangGraph

    LangChain / LangGraph agents querying the vector database and composing a contextual answer.

Web content to collect.

Automated page collection with SeleniumBase.

Chunking of collected content and embedding computation.

Qdrant vector database queried by similarity.

Question asked to the system.

FastAPI API exposing the agents.

LangChain / LangGraph agents querying the vector database and composing a contextual answer.

Two flows: indexing (scraping → embeddings → Qdrant) and answering (API → agents → semantic search).

Results

  • Automated scraping
  • Advanced semantic search
  • Contextual answers

Stack

  • Python
  • FastAPI
  • LangChain
  • LangGraph
  • Qdrant
  • SeleniumBase