Linux Deployment Walkthrough: A Data Collection and Processing Platform from Scraper Scripts to Data Cleaning

Disclaimer: This article is for technical education and demonstration only. It is not professional or financial advice. Any real-world deployment must comply with applicable laws and regulations.

I recently helped a client deploy a data collection and processing system. The server side runs on Linux and ships with built-in collection scripts and an admin panel. I hit plenty of snags along the way, but I also got a real feel for how this thing behaves. I’m writing up the whole process here as a reference for anyone getting into web scraping and data pipelines. One thing up front: collection targets public data sources only, respects each site’s robots protocol, and never touches login sessions or private data. That’s the hard line.

Hands-On Look at the System’s Features

The system has three main parts: the collection side, the cleaning pipeline, and the admin panel. The collector is a Python script with multi-threaded concurrency — five threads by default. In my tests I bumped it to eight, and CPU usage stayed under 60%, running nice and steady. Collection rules live in a JSON config file where you can customize target URLs, field selectors, and pagination logic.

How the Scraper Script Is Scheduled

The script runs under crontab, executing every 10 minutes by default. I changed it to 15 minutes — gentler on the target site and less likely to trigger their rate-limit defenses. There’s a built-in retry mechanism too: requests time out after 30 seconds and get skipped, and after three failed retries the entry goes into a failure log. Logs sit in the logs directory, rotated daily, which makes troubleshooting painless.

The Data Cleaning Pipeline

The cleaning stage is fairly sophisticated. Raw data lands in a Redis queue first, then a cleaning process consumes it: deduplication (based on an MD5 hash of URL plus title), field formatting, null-value removal, and keyword filtering. Cleaned records go into MySQL, and I measured write throughput at roughly 300 rows per second — plenty for this workload. The admin panel also lets you manually re-clean historical data, so when you tweak a rule you don’t have to re-collect everything from scratch.

Deployment Notes and Pitfalls I Hit

My environment was CentOS 7.9 + Nginx + MySQL 5.7 + Python 3.8. The client’s machine was a 2-core, 4 GB box, which turned out to be just right — memory usage hovered around 1.8 GB the whole time.

Installing Dependencies

There’s a gotcha in the Python dependencies: the requests version pinned in requirements.txt is too old and clashes with newer OpenSSL releases. Upgrading to 2.28 or above fixes it. Also make sure gcc and python3-devel are installed, otherwise some C extensions won’t compile. For Redis and MySQL, set proper passwords before starting the services — the default configs aren’t safe even on an internal network.

Nice touch: the admin panel supports switching between Chinese and English, and both field mapping and collection rules can be configured visually in the browser, which makes customization-friendly. The payment-related part is a reserved demo settlement module — handy for demonstrating points-based top-up scenarios. The API docs are complete; just remember to do your integration testing in the sandbox environment before connecting anything real.

Admin Access and Security Hardening

Change the default admin path and port, no exceptions. Put an IP whitelist or basic auth in front of it at the Nginx layer. And don’t use the root account for the database — create a dedicated user with privileges on a single database. All of this takes maybe ten minutes and blocks the vast majority of automated scanners.

Tips for Custom Development

The codebase is well organized — the collector, cleaner, and admin panel are decoupled, so pointing the system at a new data source only requires adding a rule config, not touching core code. If you want alerts pushed to WeCom or DingTalk, just add a webhook call in the retry-failure callback. That took me about half an hour.

Who Is This For?

Teams doing data monitoring, public opinion analysis, or industry information aggregation can pick this up and run with it. Solo developers looking to learn scraper engineering will find the architecture worth dissecting too. Complete beginners should brush up on Linux basics first — otherwise file permissions and firewall rules will stop you cold.

FAQ

Q: What should I do when the scraper gets 403 errors?
A: It’s usually a request rate that’s too high or a user-agent that’s too obvious. Lower the thread count, increase the delay between requests, and set the UA to look like a common browser. Also confirm the target site’s robots protocol actually allows collecting the paths you’re hitting.

Q: How do I handle duplicate records showing up in MySQL?
A: First check that the Redis dedup queue is being consumed properly, then add a unique index on the table as a safety net. I added a unique key on url_md5 so duplicate inserts get ignored outright — problem solved for good.

Q: Can this run on a 2-core, 2 GB machine?
A: Yes, but you’ll need to slim it down. Drop the collection threads to three, set MySQL’s innodb_buffer_pool_size to 512M, and disable any cleaning modules you don’t use. It’ll handle day-to-day operation fine.

Q: Can this system be used commercially?
A: Technically there’s no obstacle, but the collection activity itself must comply with applicable laws and the target site’s terms of use. Stick to public data only, and never use it for anything against the rules.

Overall, this data collection platform is well built, moderately easy to deploy, and decently documented. Get the scheduling frequency and security hardening right, and it runs itself over the long haul. If you run into deployment issues, drop a comment — I’ll reply when I see it.

Disclaimer: This article is for technical education and demonstration only. It is not professional or financial advice. Any real-world deployment must comply with applicable laws and regulations.

#Data Collection #Web Scraping #Linux Deployment #Data Cleaning #Server Setup Tutorial