# DAT Scraper - Cron Job Setup Guide

This guide explains how to set up the DAT scraper to run automatically every day at 9:00 AM on your server.

## Features

- **Daily Execution**: Runs automatically at 9:00 AM every day
- **Date-based Filenames**: Excel files are named with the date (e.g., `scraped_data_2024-01-15.xlsx`)
- **Headless Mode**: Runs without a display (perfect for servers)
- **Logging**: All output is saved to log files in the `logs/` directory

## Setup Instructions

### 1. Install Dependencies

Make sure all Python dependencies are installed:

```bash
pip3 install -r requirements.txt
```

### 2. Make Scripts Executable

The setup script should have already made the scripts executable, but if needed:

```bash
chmod +x run_scraper.sh setup_cron.sh
```

### 3. Set Up the Cron Job

Run the setup script:

```bash
./setup_cron.sh
```

This will:
- Make the run script executable
- Add a cron job that runs daily at 9:00 AM
- Show you the cron job details

### 4. Verify Cron Job

To view your cron jobs:

```bash
crontab -l
```

You should see a line like:
```
0 9 * * * /home/nitin/Desktop/dat/run_scraper.sh
```

### 5. Test the Setup

You can test the scraper manually:

```bash
# Test in headless mode (like cron will run it)
./run_scraper.sh

# Or test directly with Python
python3 scrape.py --headless
```

## File Structure

After setup, your directory structure will be:

```
dat/
├── scrape.py              # Main scraper script
├── run_scraper.sh         # Wrapper script for cron
├── setup_cron.sh          # Cron setup script
├── requirements.txt       # Python dependencies
├── logs/                  # Log files (created automatically)
│   └── scraper_YYYY-MM-DD_HH-MM-SS.log
└── scraped_data_YYYY-MM-DD.xlsx  # Output files (created daily)
```

## Output Files

- **Excel Files**: Created daily with format `scraped_data_YYYY-MM-DD.xlsx`
  - Contains two sheets: "Origin Details" and "Broker Details"
- **Log Files**: Saved in `logs/` directory with timestamp
  - Format: `scraper_YYYY-MM-DD_HH-MM-SS.log`

## Manual Execution

If you need to run the scraper manually:

```bash
# Run in headless mode (for server)
python3 scrape.py --headless

# Run with browser visible (for testing)
python3 scrape.py
```

## Troubleshooting

### Check Logs

If something goes wrong, check the log files:

```bash
ls -lt logs/ | head -5  # View recent logs
tail -f logs/scraper_*.log  # Follow latest log
```

### Verify Cron is Running

Check if cron service is running:

```bash
sudo systemctl status cron
# or
sudo service cron status
```

### Test Cron Job Manually

You can test the cron job by running it at the current time:

```bash
./run_scraper.sh
```

### Check Permissions

Make sure the scripts have execute permissions:

```bash
ls -l run_scraper.sh setup_cron.sh
```

If they don't have `x` permission:

```bash
chmod +x run_scraper.sh setup_cron.sh
```

### Environment Variables

If you encounter issues with Chrome/Chromium in headless mode, you may need to set:

```bash
export DISPLAY=:0
export PATH="/usr/local/bin:/usr/bin:/bin:$PATH"
```

These are already set in `run_scraper.sh`, but you can verify they're correct for your system.

## Removing the Cron Job

To remove the cron job:

```bash
crontab -e
```

Then delete the line containing `run_scraper.sh` and save.

Or use:

```bash
crontab -l | grep -v run_scraper.sh | crontab -
```

## Schedule Customization

To change the schedule (e.g., run at 8:00 AM instead of 9:00 AM):

1. Edit `setup_cron.sh` and change `0 9` to `0 8` (or your desired time)
2. Run `./setup_cron.sh` again to update the cron job

Cron time format: `minute hour day month weekday`
- `0 9 * * *` = 9:00 AM every day
- `0 8 * * *` = 8:00 AM every day
- `0 9 * * 1-5` = 9:00 AM on weekdays only

