Selenium is a package available for Python that allows Python to directly interact with a web browser. Beyond simply scraping text or images from the page, Selenium allows Python to perform tasks such as: enter text into webform fields, click on radio buttons, or click on text links to go to other pages. Selenium will even allow Python to execute JavaScript and take screenshots.
Getting started
First perform a pip install from the terminal or command prompt (depending on your operating system).
Next you will need to download some drivers. In this blog, we will be using Chrome as the web browser. To control Chrome with Selenium, you will need to download the appropriate Chromedriver. The chrome driver you choose must match or exceed the version of Chrome currently running on your machine. To find your version, go to Help>About in Chrome.
- If you are using Chrome version 113, please download ChromeDriver 113.0.5672.24
- If you are using Chrome version 112, please download ChromeDriver 112.0.5615.49
- If you are using Chrome version 111, please download ChromeDriver 111.0.5563.64
Once downloaded, extract it and place it somewhere you can find it. I put mine on my Desktop.
Open a webpage
Now let’s use selenium to go to a webpage.
While you could just simply import selenium, making sure to import specific elements from the package ensures you are not chewing up RAM with unnecessary drivers and elements. This is just a good programming best practice.
Driver – instantiating the webdriver opens a Selenium controlled Chrome browser.
Now to navigate to a URL, we simply need to pass the URL to the driver.get() method. In the example below, I am passing the URL for Udacity’s home page to the .get() method.
Interact with a page
To demonstrate Selenium’s interactivity, I am going to use another website. Practice Test Automation has a lot of great webpages to practice on. We are going to go to the Practice Test login page.
The Test login page consists of some interactive elements: 2 input boxes (Username and Password), and a Submit button.
Using Selenium, we can have Python pass credentials to this website. First, we must learn how to identify the webpage elements.
Identify Webpage Elements
An easy method is to right click on one of the input boxes and select Inspect from the pop-up menu.
You should be presented with a pop-up on the right side of the web browser showing you the HTML code. (Hint, I often find I must hit inspect twice to get to the proper element.) You will know you are on the proper element when you hover over the HTML code and the input box highlights on the webpage.
In the case of this page, there are 3 elements we are looking to work with. The two input boxes and the submit button. Here is a quick rundown of what we are using here.
<input type> is the actual box we are looking to interact with. Notice Username has an input type of “text” while password has an input type of “password”. That means when you type in the Username box, you will see clear text, and when you type in the password box, you will get masking characters.
The last element is the <button>, this is the Submit button.
Xpath
Once you know the elements you want to work with, one of the simplest ways to identify the elements is through what is known as the Xpath. Xpath stands for XML Path Language, and it was designed to allow users to easily navigate XML style documents.
To get your Xpath, simply right click on one of the elements from the HTML tree, select Copy and then Copy Xpath.
You will get something similar to this: //*[@id=”password”]
This serves as an “address” of sorts for the password input box.
.find_element_by_xpath()
With Xpaths, you can use the .find_element_by_xpath() method. After we find the element, we can use the .send_keys() method to send the username and password.
For the submit button, we gather the Xpath the same way as the input boxes, but now we are going to use the .click() method to interact with it.
If you were successful, you should now see this in your browser.
Take a screenshot
To keep a record of your successful (or unsuccessful login), Selenium makes it easy to take a screenshot.
Summary
Selenium can do much more than what is seen here. With the addition of HTML parsing tools like Beautifulsoup, Selenium can pull text, images, charts, and other elements from webpages. It is great for automated testing and even comes with a headless method that allows Selenium to interact with webpages in the background without ever rendering a GUI browser. For more on what Python and Selenium can do, check out Selenium’s documentation page.
Expand your knowledge
Interested in expanding your Python skills? Udacity offers a variety of Python courses for any skill level. Just starting out, try our Introduction to Programming Nanodegree and learn coding basics. Already have a coding foundation, try our Intermediate Python Nanodegree program to learn more advanced Python topics and skills.
