Building a Web Research Skill with OpenClaw: Competitive Intelligence Automation
Every week I run the same research loop: check competitor websites, scan industry news, look for new product launches, and compile a summary. For months I did this manually — opening tabs, copying notes into a doc, formatting it. It took about 90 minutes per session and I kept forgetting to do it every week.
I automated the entire loop with an OpenClaw skill. Now it runs on a schedule, pulls from 12 sources across the web, compiles a competitive brief, and drops it into a Google Doc. I review it in 10 minutes instead of 90. This guide shows you exactly how to build the same thing.
What a Web Research Skill Does
An OpenClaw web research skill is a reusable automation that combines three capabilities:
- Web fetching: Pull content from competitor pages, blogs, and news sources
- Content extraction: Strip the useful signal from HTML and structure it
- Summary compilation: Combine results into a readable brief or document
The skill runs as a single command. You invoke it, it does the full loop, and returns a structured report. You can also attach it to a hook for scheduled runs — more on that later.
Prerequisites
Before building the skill, make sure you have:
- OpenClaw running — any deployment (Mac Mini, VPS, Raspberry Pi)
- ClawHub access — for fetching prerequisite skills if you want shortcuts
- A Google service account — optional, needed only if you want docs output
- Basic familiarity with OpenClaw skill structure
If you are new to skills entirely, start with the skills guide first — this article assumes you know how skills are structured and focuses on the web research pattern specifically.
Step 1: Define Your Research Targets
A good web research skill starts with a list of targets. These are the URLs, domains, or search queries you want to monitor. Store them in a JSON configuration file inside your skill directory so they are easy to update without touching the skill logic.
Create a directory for your skill:
mkdir -p ~/.openclaw/skills/web-researchCreate a configuration file at ~/.openclaw/skills/web-research/config.json:
{
"targets": [
{
"name": "Competitor A Blog",
"url": "https://competitor-a.com/blog",
"type": "rss",
"max_items": 5
},
{
"name": "Competitor B Changelog",
"url": "https://competitor-b.com/changelog",
"type": "page",
"max_items": 10
},
{
"name": "Industry News (HN)",
"url": "https://news.ycombinator.com",
"type": "search",
"query": "openclaw OR ai agents OR agent automation"
}
],
"output": {
"format": "summary",
"max_word_count": 2500,
"include_timestamps": true
}
}You can add as many targets as you want. Each entry needs a name for the report and a url to fetch from. The type field tells the skill how to process the content — RSS feeds get parsed differently than static pages.
Step 2: Build the Core Skill Script
The skill script is a Python file that OpenClaw loads and calls when the skill is activated. Create ~/.openclaw/skills/web-research/skill.py:
import json
import os
import re
from datetime import datetime
from urllib.request import urlopen
SKILL_DIR = os.path.dirname(os.path.abspath(__file__))
def load_config():
config_path = os.path.join(SKILL_DIR, "config.json")
with open(config_path) as f:
return json.load(f)
def fetch_page(url):
try:
resp = urlopen(url, timeout=15)
html = resp.read().decode("utf-8", errors="replace")
text = re.sub(r"<[^>]+>", " ", html)
text = re.sub(r"s+", " ", text).strip()
return text[:5000]
except Exception as e:
return f"Error fetching {url}: {e}"
def run_research():
config = load_config()
results = []
for target in config["targets"]:
content = fetch_page(target["url"])
results.append({
"source": target["name"],
"timestamp": datetime.now().isoformat(),
"content_preview": content[:2000]
})
return results
def main():
results = run_research()
report_lines = [f"# Web Research Report - {datetime.now().strftime('%Y-%m-%d')}"]
for r in results:
report_lines.append(f"
## {r['source']}")
report_lines.append(f"Scanned: {r['timestamp']}")
report_lines.append(r["content_preview"][:500])
return "
".join(report_lines)
if __name__ == "__main__":
print(main())
This is a minimal working version. It fetches each target URL, strips HTML tags to get plain text, and builds a markdown report. You can extend this with RSS parsing, keyword matching, or AI summarization — more on that in the advanced section.
Step 3: Register the Skill with OpenClaw
OpenClaw loads skills from the skills directory automatically. If the directory has a skill.py or skill.js file with a main() function, it is auto-detected. But for explicit registration, add the skill to your OpenClaw configuration:
{
"skills": {
"entries": {
"web-research": {
"path": "~/.openclaw/skills/web-research/skill.py",
"description": "Fetches competitor content and compiles a research brief",
"timeout": 120
}
}
}
}After adding this entry, restart OpenClaw to pick up the new skill:
openclaw gateway restartVerify the skill is loaded:
openclaw skills listYou should see web-research in the list with the description you provided.
Step 4: Run the Research Skill
Invoke the skill from any OpenClaw session:
openclaw skill run web-researchOr, if you are inside an OpenClaw chat session, you can say:
run the web-research skillOpenClaw routes this to the skill and returns the output. The first run will take 30-60 seconds depending on how many targets you have and how fast they load. Subsequent runs are similar since each call fetches fresh data.
Step 5: Add AI Summarization
Raw text output is useful but a summary is better. Add a summarization step using the gws CLI or an MCP server. If you already have an MCP server configured for an LLM API, you can pipe results through it.
Here is an enhanced version that adds a summarization step using the openclaw mcp call command from within the skill:
import json
import subprocess
def summarize(text, source_name):
prompt = f"Summarize this competitive intelligence data from {source_name}."
cmd = [
"openclaw", "mcp", "call", "llm-mcp-server",
"--input", json.dumps({"prompt": prompt, "context": text[:3000]})
]
result = subprocess.run(cmd, capture_output=True, text=True, timeout=30)
return result.stdout.strip()This approach lets you use any MCP-connected LLM — local or remote — to refine the raw fetched content into something you can actually read.
Step 6: Schedule with Hooks
The real power of a research skill is running it unattended. OpenClaw hooks let you trigger a skill on a schedule. Create a hook configuration in your OpenClaw config:
{
"hooks": {
"entries": {
"weekly-competitive-brief": {
"cron": "0 8 * * 1",
"action": "skill",
"skill": "web-research",
"description": "Monday morning competitive brief"
}
}
}
}This runs the web-research skill every Monday at 8 AM. The output returns to the session that owns the hook — or you can extend the skill to write to a file or a Google Doc.
For more complex hook setups — chaining multiple skills, conditional execution, or error handling — read the advanced hook patterns guide.
Step 7: Output to Google Docs
A research report is most useful when it lands in a shared document. To write results to a Google Doc, you need a Google service account with Docs API access. The gws CLI handles this cleanly:
pip install gws-cli
gws auth service-account --key-file ~/credentials/google-service-account.jsonAdd a docs output step to your skill:
def write_to_doc(report_text):
title = f"Competitive Brief - {datetime.now().strftime('%Y-%m-%d')}"
cmd = ["gws", "docs", "create", "--title", title, "--text", report_text[:40000]]
subprocess.run(cmd, timeout=30)
print(f"Report written to Google Doc: {title}")If you do not need Google Docs, the skill can write to a local markdown file instead — simpler and zero API dependencies.
Advanced: Multi-Source Intelligence
Once the basic skill is working, you can layer on additional capabilities:
- RSS parsing: Use
feedparserto extract structured entries from competitor blog feeds instead of raw HTML scraping - Keyword alerts: Flag specific terms like "pricing", "acquisition", "partnership" and highlight them in the report
- Diff tracking: Cache previous results and highlight what changed since the last run
- Sentiment analysis: Basic polarity scoring on competitor press coverage
- Multiple output channels: Push to Slack, email, or a Discord webhook alongside the Google Doc
Here is an RSS-aware fetch that replaces the simple fetch_page function:
try:
import feedparser
HAS_FEEDPARSER = True
except ImportError:
HAS_FEEDPARSER = False
def fetch_with_rss_support(target):
if target.get("type") == "rss" and HAS_FEEDPARSER:
feed = feedparser.parse(target["url"])
items = []
for entry in feed.entries[:target.get("max_items", 5)]:
items.append(f"- {entry.get('title', '')} - {entry.get('link', '')}")
return "
".join(items)
return fetch_page(target["url"])Troubleshooting
Here are the most common issues and how to fix them.
Skill not found after restart. Check that the path in your OpenClaw config is absolute or uses ~ expansion. OpenClaw does not expand ~ in all config parsers — use the full path like /Users/yourname/.openclaw/skills/web-research/skill.py.
Timeout on fetch. Some competitor sites are slow. Increase the timeout value in your skill registration config from 120 to 300 seconds. Or reduce the number of targets.
HTML remnants in output. The simple regex-based HTML stripper misses some edge cases like script tags. Use pip install beautifulsoup4 and parse with BeautifulSoup for cleaner results.
Hook not firing. Verify the cron expression. 0 8 * * 1 is Monday at 8 AM. A common mistake is using 5-field cron instead of the standard format OpenClaw expects — some configurations use 6-field (with seconds). Check your OpenClaw version docs.
Google Docs output failing. The Docs API has a 50KB per-document write limit. If your report exceeds this, truncate it or split across multiple documents. Also ensure your service account has Docs API enabled in the Google Cloud Console.
FAQ
Can I run this on a Raspberry Pi?
Yes. The skill is lightweight — it uses only standard library modules (plus optionally feedparser). A Raspberry Pi 4 or 5 handles 10-15 targets comfortably. Set longer timeouts since network I/O is the bottleneck, not CPU.
Does the skill need a GPU?
No. Web fetching and text extraction are CPU-bound network operations. If you add AI summarization via a local LLM, a GPU helps but is not required — a CPU-only setup with a quantized model running through Ollama works fine for short summaries.
How do I add more targets without editing the config?
Build a simple CLI wrapper that accepts a URL argument. The skill can accept command-line parameters when invoked with openclaw skill run web-research -- --target-url https://example.com. Parse sys.argv in your main function to support ad-hoc targets.
Can the skill output to Slack?
Yes. Use the Slack MCP server or the Slack webhook API directly. Add a post_to_slack function that sends the report summary to a channel. See the Slack MCP server guide for setup.
What happens if a target site is down?
The fetch_page function catches exceptions and returns an error message instead of crashing. The report still compiles — it just shows which sources failed. You can add a retry with time.sleep(5) for transient failures.
Next Steps
This skill handles the data collection layer of competitive intelligence. Once you have it running, the next step is connecting it to a broader automation pipeline — triaging results, generating alerts for specific keywords, or feeding the brief into a weekly newsletter.
The pattern here — fetch, extract, summarize, output — applies to more than competitive research. Use the same skill structure for monitoring documentation changes, tracking open-source projects, or building a personal news briefing.