edgeRaspberry Pidevops

Edge Crawling with Raspberry Pi 5: Cheap, Distributed, and Privacy-Friendly

ccrawl

2026-01-24

10 min read

Build privacy-friendly distributed micro-crawlers on Raspberry Pi 5 + AI HAT+, coordinate jobs with MQTT/Redis, and ship distilled results to ClickHouse or S3.

Edge Crawling with Raspberry Pi 5: Cheap, Distributed, and Privacy-Friendly

Hook: If your site audit or indexing pipeline is limited by crawl budget, bandwidth, or privacy concerns, deploying distributed micro-crawlers on Raspberry Pi 5 devices (with the new AI HAT+ option) is a cost-effective way to scale targeted crawling, pre-process content on-device, and stream only the distilled results back to central analytics like ClickHouse or S3-compatible object storage.

Why this matters in 2026

Edge compute and on-device AI matured rapidly through late 2024–2025 and into 2026. The Raspberry Pi 5 plus the AI HAT+ (released in late 2025) finally make local inference and efficient parsing practical at the edge for low-cost hardware. At the same time, OLAP systems like ClickHouse have continued to accelerate adoption in analytics stacks — making a hybrid design (edge pre-processing + ClickHouse ingest) compelling for technical SEO teams, developers, and infra owners.

What you'll get from this guide

Architecture patterns for distributed micro-crawlers using Raspberry Pi 5 + AI HAT+
Job queue designs: MQTT vs Redis Streams for edge devices
Practical code snippets (Python asyncio scraper, queue worker, upload to S3/ClickHouse)
Scheduling, orchestration, CI/CD integration, and privacy best practices

High-level architecture

Keep the design simple and resilient. The pattern that scales is:

Coordinator — central service that produces URL jobs (GitHub Actions, web UI, or a small API).
Job Queue — lightweight broker that edge devices subscribe to (MQTT or Redis Streams).
Edge Workers — Raspberry Pi 5 devices running a micro-crawler process; optional AI HAT+ accelerates on-device extraction/classification.
Central Storage — short JSON/NDJSON payloads to S3-compatible object storage and metadata/metrics to ClickHouse.
Monitoring & CI/CD — health checks, Prometheus metrics, and automated deployment of updated crawling rules.

Why Raspberry Pi 5 + AI HAT+?

The Raspberry Pi 5 offers a meaningful CPU uplift and better I/O for edge tasks. The AI HAT+ unlocks local model inference — which you can use to:

Run on-device readability/boilerplate removal models for cleaner text extraction.
Classify or filter content (e.g., skip pages that are clearly login pages or duplicates).
Create embeddings for similarity deduplication before shipping results.

“Do as much work as possible at the edge: fetch, sanitize, classify, and compress. Ship only what you need.”

Design choices: Job queue

Two practical choices for edge-friendly job queues:

MQTT (recommended for low-bandwidth, intermittent connectivity)

Pros: tiny footprint, built for unreliable networks, easy publish/subscribe semantics. Devices can subscribe to per-device topics or capacity topics. Retained messages and QoS levels make job delivery robust.

Cons: Not great for guaranteed exactly-once semantics if you need complex acknowledgement workflows — but you can implement idempotency at the worker.

Redis Streams (recommended if you already run Redis)

Pros: strong ordering, consumer groups, and easy visibility into backlog; works well when a central Redis is available and you want precise consumer-group semantics.

Cons: heavier than MQTT and less optimized for highly intermittent networks.

Micro-crawler: minimal, resilient, privacy-first

Design each edge worker as a tiny process that:

Accepts a job: {url, job_id, max_depth, headers}
Fetches with asyncio/aiohttp and obeys robots.txt and politeness rules
Parses and extracts text using lxml/readability or a local model on the AI HAT+
Computes hashes / embeddings and applies local filters
Compresses and signs payloads, then uploads to S3 and writes metadata to ClickHouse

Python asyncio example (fetch & extract)

import asyncio
import aiohttp
import hashlib
from readability import Document

async def fetch(session, url, timeout=15):
    async with session.get(url, timeout=timeout) as r:
        text = await r.text()
        return r.status, text

async def process(url):
    headers = {"User-Agent": "pi-edge-crawler/1.0 (+https://example.com)"}
    async with aiohttp.ClientSession(headers=headers) as session:
        status, html = await fetch(session, url)
        doc = Document(html)
        content = doc.summary()  # HTML snippet
        text = doc.text()
        sha = hashlib.sha256(html.encode('utf-8')).hexdigest()
        return {"url": url, "status": status, "sha256": sha, "text": text}

# Usage: await process('https://example.com')

Note: use proper robots.txt parsing and throttling — don't blast origin servers. Add retry/backoff and a per-origin concurrency limiter.

On-device ML with the AI HAT+

Use the AI HAT+ to run small models that can:

Detect login/404 pages and skip storing them
Remove PII or sensitive tokens (local regex + model check)
Compute compact embeddings for deduplication

Workflow: after fetching and parsing, pass the cleaned text into a lightweight model (e.g., a quantized transformer) to classify or embed. If the model indicates the page is irrelevant, only metadata is uploaded.

Packaging & upload: S3 + ClickHouse

Edge devices should upload compressed payloads (Gzip/JSONL) to an S3-compatible bucket. Metadata (one row per fetch) is inserted into ClickHouse for fast OLAP queries and diagnostics.

Example ClickHouse table schema

CREATE TABLE IF NOT EXISTS crawler.jobs (
  timestamp DateTime,
  device_id String,
  job_id String,
  url String,
  http_code UInt16,
  sha256 String,
  text_length UInt32,
  s3_path String,
  embedding Array(Float32)
) ENGINE = MergeTree()
PARTITION BY toYYYYMM(timestamp)
ORDER BY (device_id, timestamp);

Insert metadata via HTTP API

# Simple curl example
curl 'http://clickhouse:8123/?query=INSERT%20INTO%20crawler.jobs%20FORMAT%20JSONEachRow' \
  -H 'Content-Type: application/json' \
  --data-binary '[{"timestamp":"2026-01-01 12:00:00","device_id":"pi-01","job_id":"abc123","url":"https://...","http_code":200,"sha256":"...","text_length":1234,"s3_path":"s3://bucket/pi-01/abc123.json.gz","embedding": [0.1,0.02,...]}]'

For high throughput, batch inserts or use ClickHouse’s native clients. ClickHouse’s growth in early 2026 (and large funding rounds) means robust ecosystem tooling for ingestion is widely available.

Upload example to S3 (boto3)

import boto3
import gzip
import json
s3 = boto3.client('s3')

def upload_payload(bucket, key, payload):
    gz = gzip.compress(json.dumps(payload).encode('utf-8'))
    s3.put_object(Bucket=bucket, Key=key, Body=gz)
    return f's3://{bucket}/{key}'

For reliable mobile/edge uploads consider using tested client SDKs and resumable upload patterns — see reviews of client SDKs for reliable uploads.

Job queue worker examples

MQTT worker (paho-mqtt)

import paho.mqtt.client as mqtt

def on_message(client, userdata, msg):
    job = json.loads(msg.payload)
    # process(job)

client = mqtt.Client()
client.on_message = on_message
client.connect('mqtt-broker', 1883)
client.subscribe('crawl/jobs')
client.loop_forever()

Redis Streams worker (redis-py)

from redis import Redis
from redis.exceptions import ResponseError

r = Redis(host='redis')

def worker(group='workers', consumer='pi-01'):
    try:
        r.xgroup_create('crawl:jobs', group, id='$', mkstream=True)
    except ResponseError:
        pass
    while True:
        items = r.xreadgroup(group, consumer, {'crawl:jobs': '>'}, count=1, block=5000)
        if not items:
            continue
        # ack and process each item

Scheduling & CI/CD integration

Use a mix of approaches depending on scale:

Local cron/systemd — simple, reliable scheduling on devices for periodic checks (e.g., cron job to pull new tasks every 5 minutes).
Centralized scheduler — produce job batches centrally (e.g., GitHub Action or Jenkins) and push to the queue for devices to consume.
GitOps — store crawl lists and rules in a Git repo; an action commits a new job file and the coordinator pushes tasks to the queue on push.

Cron example (every 10 minutes)

# /etc/cron.d/pi-crawler
*/10 * * * * pi /usr/local/bin/pi-crawler --pull-jobs >> /var/log/pi-crawler.log 2>&1

CI example: GitHub Action pushes batch jobs

A simple GitHub Action can validate a crawl list, then call your coordinator API to enqueue jobs.

Privacy-first practices

Edge crawling is a chance to improve privacy and compliance:

On-device filtering — redact PII before leaving the device.
Send metadata not raw HTML — only store what’s needed for SEO analysis (HTTP code, title, cleaned text, content hash).
Encrypt in transit — use TLS, sign uploads with device keys, rotate keys regularly.
Retention & purge — apply aging policies centrally (e.g., delete raw payloads after 30 days unless labeled important).

Resilience, metrics & diagnostics

Edge devices are unreliable: design for retries and local buffering.

Use a local queue (SQLite or file-based) to buffer outgoing payloads during network outages.
Expose a small Prometheus metrics endpoint (device uptime, queue depth, last successful sync) and have a central Prometheus or Pushgateway ingest them.
Keep logs minimal and structured (JSON) so central log collection (Fluentbit, Vector) can aggregate when bandwidth allows.

Security & device identity

Provision devices with unique keys/certs at imaging time. Use mutual TLS or signed JWTs for API access. If you use cloud S3, prefer IAM roles or short-lived credentials. Consider hardware-backed keys (secure element) if available.

Cost & performance considerations

Raspberry Pi 5 units are inexpensive; the AI HAT+ adds cost but can reduce cloud compute and egress by pre-processing data. Typical patterns:

Fetch-only jobs: ~1–2s per page (CPU-bound). Multiple devices parallelize effectively.
On-device inference: latency depends on model; use quantized models for speed.
Bandwidth savings: if you send only JSON metadata + compressed extracts, you can reduce egress by 90% compared to shipping raw HTML.

Advanced: embeddings, dedupe, and ClickHouse

Compute compact embeddings on-device, store them in ClickHouse, and run fast similarity queries for duplicate detection or topic clustering. ClickHouse’s ecosystem in 2026 has improved vector support and ingestion tooling, making it a practical central analytics engine for large-scale crawler metadata.

Operational checklist: from 0 to 10 devices

Image Pi 5 devices with a 64-bit OS, Docker or headless Python environment.
Install MQTT/Redis client, boto3, ClickHouse client, and lightweight ML runtime for AI HAT+.
Provision device identity keys and push central config (politeness, rate limits, allowed domains).
Run a test job that fetches one domain, extracts text, computes a hash, uploads to S3, and inserts metadata to ClickHouse.
Set up monitoring (Prometheus node exporter + metrics endpoint) and log forwarding.
Create CI workflow to push new crawl lists and update parsing rules.

Example: End-to-end flow (concise)

Coordinator enqueues URL(s) to MQTT topic /crawl/jobs.
Pi device subscribes, receives job, checks robots.txt, enforces delay.
Device fetches, runs local parser and classifier on AI HAT+.
Device compresses JSON extract, uploads to S3, inserts metadata into ClickHouse.
Coordinator marks job complete; monitoring dashboards update.

Troubleshooting tips

If devices fall behind, check local storage for queue backlog and network outages.
Use idempotent job IDs (job_id = sha256(url + crawl_options)) so retries are safe.
For JS-heavy sites, conditionally route jobs to a heavier headless central crawler — edge devices should avoid running full Chromium unless necessary.
Monitor ClickHouse insert latency; batch small inserts to reduce overhead.

2026 trends & future-proofing

Edge-first crawling maps neatly onto regulatory trends and the push for privacy-preserving analytics. Expect more device-optimized model runtimes (quantized, NNAPI-backed) and richer vector features in ClickHouse and other OLAP engines through 2026. Designing your system to keep sensitive content local and send only structured signals will remain a best practice.

Actionable takeaways

Start with a single Raspberry Pi 5 and one domain. Validate robots, rate limits, and upload flow to S3/ClickHouse.
Use MQTT for flaky networks or Redis Streams if you need strong consumer semantics.
Leverage the AI HAT+ to filter and reduce data shipped off-device — this saves bandwidth and protects privacy.
Instrument devices and ClickHouse with straightforward metrics and a CI workflow to update rules without SSHing into each box.

Further resources & next steps

Open-source crawler examples (look for repos with edge-first patterns, asyncio workers, and MQTT integrations).
ClickHouse docs for bulk inserts and partitioning strategies for time-series metadata.
AI HAT+ runtimes and quantized model guides released in late 2025 — test with small classification models first.

Conclusion & call-to-action

Edge crawling with Raspberry Pi 5 + AI HAT+ is an affordable, privacy-friendly approach to scale targeted crawling while minimizing cloud costs and data egress. Start small: provision one device, validate your job queue and upload pipeline, then incrementally roll out more units. The combination of local pre-processing and ClickHouse-backed analytics offers a fast, queryable way to analyze crawl results across distributed devices.

Ready to build a proof-of-concept? Clone a starter repo, flash one Pi 5, and run the sample MQTT worker. If you want a checklist, example configs, and a tested pipeline that writes to ClickHouse + S3, download the companion repo linked from this article and follow the step-by-step README to get a 3-device cluster running in under an hour.

crawl

Contributor

Senior editor and content strategist. Writing about technology, design, and the future of digital media. Follow along for deep dives into the industry's moving parts.