Sometimes if you just need some light web scraping, nothing too complicated. Other times, chrome webdriver will just decide to not work, but that’s pretty rare. Or you really need to extract some useful bits of information, but you don’t have selenium already installed handy. Maybe you’re on a borrowed laptop that resets on every checkout.

In these cases, the devtools console will do. No need to take out selenium.

The essence here is you can interact with the page in a few basic, but often good enough ways:

  • document.querySelector, obviously
  • outerHTML on HTMLElement to stringify a DOM node. Do some string manipulation to clean it up.
    • optionally do some string or DOM processing (clone the node first)
    • when done, copy the array to clipboard. Now you have scraped output!
  • simulate user clicking with click() on HTMLElement
    • This might not always work, so your milage may vary.
  • study their code, and emulate some behaviors yourself to “interact” with the page
    • For example, if button fetches HTML and replaces a subtree, you can just do the fetch yourself.
    • Only really possible if it’s using jQuery or simple vanilla JS.
  • for navigation…
    • Replace window.location.href to navigate, but be careful: this wipes out your devtools console. If you really need to do this, spending some time to craft a full script that you can just paste into console will pay off.
    • Modify <a> with target="_blank" to batch open links in new tabs.
  • call setTimeout recursively

…

…

…

Ok I’ll confess, the real reason is chrome webdriver broke on me. As they say: I did this not because it was easy, but because I thought it would be easy. Easier than selenium, which honestly I’m not sure if that’s true. I guess it was kind of fun?