Writeup Finder is an automated script that fetches and processes security-related writeups from multiple sources including YouTube channels, Medium, and Pentester.land. The script supports integration with PostgreSQL for storing data and can send notifications to a Discord webhook.
- YouTube Videos: Retrieve and process video data from specified YouTube channels.
- Medium Writeups: Fetch and process writeups from Medium using RSS feeds.
- Pentester.land Writeups: Collect writeups from Pentester.land’s JSON feed.
- PostgreSQL Database: Store data in a PostgreSQL database for efficient management.
- Discord Notifications: Optionally send notifications about new content to a Discord webhook.
- Python 3.x
- PostgreSQL
- Docker (for PostgreSQL)
- Required Python packages:
requests,psycopg2
- Clone the Repository
git clone https://github.com/moeinerfanian/writeup-finder.git
cd writeup-finder- Set Up PostgreSQL with Docker
docker compose up -d- Install Python Dependencies
pip3 install -r requirements.txt- Configure the Script
- Edit
config.pyto include your PostgreSQL database credentials and Discord webhook URL.
- Create Database Tables
- Initialize the database schema with:
python3 main.py db- Run the Script
- Do in order
- Create DB:
python3 main.py db - To skip sending Discord notifications, use the
nodiscordflag:python3 main.py nodiscord - Normally Run:
python3 main.py
- PostgreSQL Integration: Enhanced database handling and connectivity.
- RSS Feed Handling: Improved fetching and parsing of YouTube and Medium RSS feeds.
- Error Handling: Better management of rate limits and parsing errors.
To run the script automatically every 5 hours, add the following line to your crontab:
0 */5 * * * /usr/bin/python3 /path/to/your/repo/main.py- Replace
/path/to/your/repo/with the path to your cloned repository.
Moein Erfanian (Electro0ne)
