How to add scraping url for batch in linux

It is effective to use the crontab for the batch schedule. For example, You can create a batch program using the web and register the task in the crontab. In the Linux/Unix environment, the web can be executed through the curl.

Link : How to use curl

shell script for scraping (scraping.sh)

#!/bin/bash

MLIST="55 69 93 94 95 96 100"

echo "-------------------------------------"
echo " Starting..."
echo "-------------------------------------"

for ITEM in $MLIST
do
    echo -n `date '+%Y-%m-%d %H:%M:%S'`" $ITEM : "
    /usr/bin/curl http://yourscrapingurl.com/scrap?id=$ITEM
    echo ""
done

batch scheduler (crontab -e)

##################################################
# scraping by curl (run every 3 hours)
##################################################
0 */3 * * * /home/shl/scraping.sh >> /home/shl/cron_scraping.log 2>&1