# DAT Website Scraper

A Selenium-based web scraper for the DAT website's search-loads page.

## Setup

1. **Install Python dependencies:**
   ```bash
   pip install -r requirements.txt
   ```

2. **Install ChromeDriver:**
   - The script uses Chrome WebDriver. Make sure you have Chrome browser installed.
   - Install ChromeDriver that matches your Chrome version:
     - Option 1: Download from https://chromedriver.chromium.org/
     - Option 2: Use a package manager:
       ```bash
       # Ubuntu/Debian
       sudo apt-get install chromium-chromedriver
       
       # Or use webdriver-manager (install with: pip install webdriver-manager)
       ```

## Usage

Run the scraper:
```bash
python scrape.py
```

## What it does

1. **Checks for saved session** - If you've logged in before, it will use saved cookies to skip login
2. **Logs in** to the DAT website using the provided credentials (if no valid session exists)
   - Handles two-step login (email, then password)
   - Navigates through MFA verification (clicks "TRY ANOTHER METHOD" → "Recovery code")
   - Waits for you to manually enter the OTP/recovery code
3. **Saves session cookies** after successful login for future use
4. **Navigates** to the search-loads page
5. **Scrapes** load listings from the page
6. **Saves** the data to `scraped_data.json`
7. **Takes** a screenshot and saves it as `search_loads_page.png`

## Output Files

- `scraped_data.json` - JSON file containing scraped load data
- `search_loads_page.png` - Screenshot of the page
- `dat_session_cookies.json` - Saved session cookies (automatically created after login)
- `page_source.html` - HTML source (saved if scraping issues occur)

## Notes

- **Session Persistence**: After the first successful login, the script saves your session cookies. On subsequent runs, it will automatically use the saved session to skip the login process and go directly to the search-loads page. If the session expires, it will prompt you to log in again.
- **Manual OTP Entry**: When the MFA screen appears, the script will navigate to the "Recovery code" option and wait for you to manually enter the OTP/recovery code in the browser. You have up to 5 minutes to complete this.
- The script includes error handling and will save page sources if it encounters issues
- You may need to adjust CSS selectors in the `scrape_search_loads()` function based on the actual page structure
- The browser will stay open for 10 seconds after scraping for inspection
- To run in headless mode, uncomment the headless option in `setup_driver()`

## Troubleshooting

If login fails:
- Check that the email and password are correct
- The script will save the page source to help debug login issues
- You may need to adjust the CSS selectors for the login form fields

If scraping fails:
- The script will save the page source to `error_page_source.html`
- Inspect the HTML to identify the correct selectors for load listings
- Update the `load_selectors` list in the `scrape_search_loads()` function

